Patent Publication Number: US-11382702-B2

Title: Medical robotic system providing an auxiliary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. application Ser. No. 15/646,685 (filed Jul. 11, 2017, which is a continuation of U.S. application Ser. No. 14/748,602 (filed Jun. 24, 2015), now U.S. Pat. No. 9,717,563, which is a divisional of U.S. application Ser. No. 12/489,566 (filed Jun. 23, 2009), now U.S. Pat. No. 9,089,256, which is a continuation-in-part of U.S. application Ser. No. 12/163,087 (filed Jun. 27, 2008), now U.S. Pat. No. 10,258,425, each of which is incorporated herein by reference. 
    
    
     FIELD OF THE INVENTION 
     The present invention generally relates to medical robotic systems and in particular, to a medical robotic system providing an auxiliary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide. 
     BACKGROUND 
     Medical robotic systems such as teleoperative systems used in performing minimally invasive surgical procedures offer many benefits over traditional open surgery techniques, including less pain, shorter hospital stays, quicker return to normal activities, minimal scarring, reduced recovery time, and less injury to tissue. Consequently, demand for such medical robotic systems is strong and growing. 
     One example of such a medical robotic system is the da Vinci® Surgical System from Intuitive Surgical, Inc., of Sunnyvale, Calif., which is a minimally invasive robotic surgical system. The da Vinci® Surgical System has a number of robotic arms that move attached medical devices, such as an image capturing device and Intuitive Surgical&#39;s proprietary EndoWrist® articulating surgical instruments, in response to movement of input devices by a surgeon viewing images captured by the image capturing device of a surgical site. Each of the medical devices is inserted through its own minimally invasive incision into the patient and positioned to perform a medical procedure at the surgical site. The incisions are placed about the patient&#39;s body so that the surgical instruments may be used to cooperatively perform the medical procedure and the image capturing device may view it without their robotic arms colliding during the procedure. 
     To perform certain medical procedures, it may be advantageous to use a single entry aperture, such as a minimally invasive incision or a natural body orifice, to enter a patient to perform a medical procedure. For example, an entry guide may first be inserted, positioned, and held in place in the entry aperture. Instruments such as an articulatable camera and a plurality of articulatable surgical tools, which are used to perform the medical procedure, may then be inserted into a proximal end of the entry guide so as to extend out of its distal end. Thus, the entry guide provides a single entry aperture for multiple instruments while keeping the instruments bundled together as it guides them toward the work site. The entry guide may be either rigid or flexible. 
     Since the entry guide generally has a relatively small diameter in order to fit through a minimally invasive incision or a natural body orifice, a number of problems may arise while teleoperating the surgical tools to perform the medical procedure and the camera to view it. For example, because the camera is bundled with the surgical tools, it is limited in its positioning relative to the surgical tools and consequently, its view of the surgical tools. 
     Thus, although the tips of the articulatable surgical tools may be kept in the field of view of the camera, controllable linkages which facilitate the articulatability of the surgical tools may not be in the field of view of the camera. As a consequence, the controllable linkages of the surgical tools may inadvertently collide with each other (or with a link of the camera) during the performance of a medical procedure and as a result, cause harm to the patient or otherwise adversely impact the performance of the medical procedure. 
     Also, since the articulatable camera is generally incapable of viewing its own controllable linkage, operator movement of the camera is especially a concern where collisions with the surgical tool links are to be avoided. Further, when intuitive control is provided to assist the operator in teleoperatively moving the surgical tools and camera, the motions of the linkages required to produce such intuitive motions of the tips of the tools and camera may not be obvious or intuitive to the operator, thus making it even more difficult for the operator to avoid collisions between linkages that are outside the field of view of the camera. 
     Well positioned placements of the entry guide and the articulatable instruments extending out of its distal end allow unencumbered movement and wide range of motion for the instruments so that they may be used to perform a medical procedure at a target site. Due to the restricted view provided by the camera, however, it may be difficult for an operator to determine such a well positioned placement of the entry guide or well positioned placement of the articulatable instruments extending out of its distal end. Further, such a bundled instrument arrangement is prone to getting into non-optimal tool working orientations in ordinary use due in large part to the camera instrument&#39;s abilities to pan and tilt. 
     OBJECTS AND BRIEF SUMMARY 
     Accordingly, one object of one or more aspects of the present invention is a method that provides an auxiliary view to an operator to assist the operator in performing a medical procedure on a patient using a medical robotic system having articulatable instruments extending out of a distal end of an entry guide inserted through a single entry aperture in the patient. 
     Another object of one or more aspects of the present invention is a method implemented in such a medical robotic system that provides a visual indication to an operator that indicates when controllable joints of the articulatable instruments are nearing limitations in their respective ranges of motion. 
     Another object of one or more aspects of the present invention is a method implemented in a medical robotic system that provides a visual indication to an operator that indicates when joints and/or links and/or portions thereof of the articulatable instruments are nearing an undesirable or desirable event or condition. 
     These and additional objects are accomplished by the various aspects of the present invention, wherein briefly stated, the embodiments of the invention are summarized by the claims that follow below. 
     Additional objects, features and advantages of the various aspects of the present invention will become apparent from the following description of its preferred embodiment, which description should be taken in conjunction with the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a top view of an operating room employing a medical robotic system utilizing aspects of the present invention. 
         FIG. 2  illustrates a block diagram of components for controlling and selectively associating device manipulators to left and right hand-manipulatable input devices in a medical robotic system utilizing aspects of the present invention. 
         FIGS. 3-4  respectively illustrate top and side views of an articulatable camera and a pair of articulatable surgical tools extending out of a distal end of an entry guide as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 5  illustrates a perspective view of an entry guide and its four degrees-of-freedom movement as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 6  illustrates a cross-sectional view of an entry guide with passages defined therein that extend between its proximal and distal ends as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 7  illustrates a block diagram of interacting components of an entry guide manipulator as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 8  illustrates a block diagram of interacting components of an articulatable instrument manipulator and an articulatable instrument as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 9  illustrates a flow diagram of a method for providing a computer generated auxiliary view, utilizing aspects of the present invention. 
         FIG. 10  illustrates a data and processing flow diagram to determine instrument link positions and orientations using instrument joint positions and forward kinematics, as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 11  illustrates a data and processing flow diagram to determine instrument joint positions using a sensed instrument tip position and inverse kinematics, as used in a medical robotic system utilizing aspects of the present invention. 
         FIGS. 12-13  respectively illustrate top and side auxiliary views as generated and displayed on a display screen by a method implemented in a medical robotic system utilizing aspects of the present invention. 
         FIG. 14  illustrates top and side auxiliary views as generated and displayed in separate windows on a display screen by a method implemented in a medical robotic system utilizing aspects of the present invention. 
         FIG. 15  illustrates an auxiliary view displayed adjacent to an image captured by the articulatable camera on a monitor in a medical robotic system utilizing aspects of the present invention. 
         FIG. 16  illustrates an auxiliary side view of an articulatable camera having a frustum as generated and displayed by a method implemented in a medical robotic system utilizing aspects of the present invention on a display screen. 
         FIG. 17  illustrates a combined display of an auxiliary view of a pair of articulatable surgical tools from a viewing point of a camera, along with an image captured by the camera, as generated and displayed by a method implemented in a medical robotic system utilizing aspects of the present invention on a display screen. 
         FIG. 18  illustrates a flow diagram of a method for providing auxiliary viewing modes that correspond to device control modes in a medical robotic system, utilizing aspects of the present invention. 
         FIG. 19  illustrates a diagram of a side view of an articulatable instrument extending out of a distal end of an entry guide in a medical robotic system utilizing aspects of the present invention. 
         FIG. 20  illustrates an auxiliary view of articulatable instruments retracted into an entry guide along with indications of range of motion limitations utilizing aspects of the present invention. 
         FIG. 21  illustrates an auxiliary view of articulatable instruments extending out of an entry guide along with indications of range of motion limitations utilizing aspects of the present invention. 
         FIGS. 22-25  illustrate various graphical displays indicating the extension of an articulatable instrument out of a distal end of an entry guide as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 26  illustrates a graphical representation of grippers as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 27  illustrates a graphical representation of an articulatable camera as used in a medical robotic system utilizing aspects of the present invention. 
         FIG. 28  illustrates a simplified auxiliary view of a poorly positioned entry guide with respect to articulatable instruments extending out of its distal end in a medical robotic system utilizing aspects of the present invention. 
         FIG. 29  illustrates a simplified auxiliary view of a repositioned entry guide with articulatable instruments extending out of its distal end in a medical robotic system utilizing aspects of the present invention. 
         FIG. 30  illustrates auxiliary views of articulatable instruments extending out of an entry guide along with an image captured by one of the instruments as displayed on a monitor in a medical robotic system utilizing aspects of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  illustrates, as an example, a top view of an operating room in which a medical robotic system  100  is being utilized by a Surgeon  20  for performing a medical procedure on a Patient  40  who is lying face up on an operating table  50 . One or more Assistants  30  may be positioned near the Patient  40  to assist in the procedure while the Surgeon  20  performs the procedure teleoperatively by manipulating input devices  108 ,  109  on a surgeon console  10 . 
     In the present example, an entry guide (EG)  200  is inserted through a single entry aperture  150  into the Patient  40 . Although the entry aperture  150  is a minimally invasive incision in the present example, in the performance of other medical procedures, it may instead be a natural body orifice. The entry guide  200  is held and manipulated by a robotic arm assembly  130 . 
     As with other parts of the medical robotic system  100 , the illustration of the robotic arm assembly  130  is simplified in  FIG. 1 . In one example of the medical robotic system  100 , the robotic arm assembly  130  includes a setup arm and an entry guide manipulator. The setup arm is used to position the entry guide  200  at the entry aperture  150  so that it properly enters the entry aperture  150 . The entry guide manipulator is then used to robotically insert and retract the entry guide  200  into and out of the entry aperture  150 . It may also be used to robotically pivot the entry guide  200  in pitch, roll and yaw about a pivot point located at the entry aperture  150 . An example of such an entry guide manipulator is the entry guide manipulator  202  of  FIG. 2  and an example of the four degrees-of-freedom movement that it manipulates the entry guide  200  with is shown in  FIG. 5 . 
     The console  10  includes a 3-D monitor  104  for displaying a 3-D image of a surgical site to the Surgeon, left and right hand-manipulatable input devices  108 ,  109 , and a processor (also referred to herein as a “controller”)  102 . The input devices  108 ,  109  may include any one or more of a variety of input devices such as joysticks, gloves, trigger-guns, hand-operated controllers, or the like. Other input devices that are provided to allow the Surgeon to interact with the medical robotic system  100  include a foot pedal  105 , a conventional voice recognition system  160  and a Graphical User Interface (GUI)  170 . 
     An auxiliary display screen  140  is coupled to the console  10  (and processor  102 ) for providing auxiliary views to the Surgeon to supplement those shown on the monitor  104 . A second auxiliary display screen  140 ′ is also coupled to the console  10  (and processor  102 ) for providing auxiliary views to the Assistant(s). An input device  180  is also coupled to the console to allow the Assistant(s) to select between available auxiliary views for display on the second auxiliary display screen  140 ′. 
     The console  10  is usually located in the same room as the Patient so that the Surgeon may directly monitor the procedure, is physically available if necessary, and is able to speak to the Assistant(s) directly rather than over the telephone or other communication medium. However, it will be understood that the Surgeon can also be located in a different room, a completely different building, or other remote location from the Patient allowing for remote surgical procedures. In such a case, the console  10  may be connected to the second auxiliary display screen  140 ′ and input device  180  through a network connection such as a local area network, wide area network, or the Internet. 
     As shown in  FIGS. 3-4 , the entry guide  200  has articulatable instruments such as articulatable surgical tools  231 ,  241  and an articulatable stereo camera  211  extending out of its distal end. Although only two tools  231 ,  241  are shown, the entry guide  200  may guide additional tools as required for performing a medical procedure at a work site in the Patient. For example, as shown in  FIG. 4 , a passage  351  is available for extending another articulatable surgical tool through the entry guide  200  and out through its distal end. Each of the surgical tools  231 ,  241  is associated with one of the input devices  108 ,  109  in a tool following mode. The Surgeon performs a medical procedure by manipulating the input devices  108 ,  109  so that the controller  102  causes corresponding movement of their respectively associated surgical tools  231 ,  241  while the Surgeon views the work site in 3-D on the console monitor  104  as images of the work site are being captured by the articulatable camera  211 . 
     Preferably, input devices  108 ,  109  will be provided with at least the same degrees of freedom as their associated tools  231 ,  241  to provide the Surgeon with telepresence, or the perception that the input devices  108 ,  109  are integral with the tools  231 ,  241  so that the Surgeon has a strong sense of directly controlling the tools  231 ,  241 . To this end, the monitor  104  is also positioned near the Surgeon&#39;s hands so that it will display a projected image that is oriented so that the Surgeon feels that he or she is actually looking directly down onto the work site and images of the tools  231 ,  241  appear to be located substantially where the Surgeon&#39;s hands are located. 
     In addition, the real-time image on the monitor  104  is preferably projected into a perspective image such that the Surgeon can manipulate the end effectors  331 ,  341  of the tools  231 ,  241  through their corresponding input devices  108 ,  109  as if viewing the work site in substantially true presence. By true presence, it is meant that the presentation of an image is a true perspective image simulating the viewpoint of an operator that is physically manipulating the end effectors  331 ,  341 . Thus, the processor  102  may transform the coordinates of the end effectors  331 ,  341  to a perceived position so that the perspective image being shown on the monitor  104  is the image that the Surgeon would see if the Surgeon was located directly behind the end effectors  331 ,  341 . 
     The processor  102  performs various functions in the system  100 . One important function that it performs is to translate and transfer the mechanical motion of input devices  108 ,  109  through control signals over bus  110  so that the Surgeon can effectively manipulate devices, such as the tools  231 ,  241 , camera  211 , and entry guide  200 , that are selectively associated with the input devices  108 ,  109  at the time. Another function is to perform various methods and controller functions described herein. 
     Although described as a processor, it is to be appreciated that the processor  102  may be implemented in practice by any combination of hardware, software and firmware. Also, its functions as described herein may be performed by one unit or divided up among different components, each of which may be implemented in turn by any combination of hardware, software and firmware. Further, although being shown as part of or being physically adjacent to the console  10 , the processor  102  may also comprise a number of subunits distributed throughout the system. 
     For additional details on the construction and operation of various aspects of a medical robotic system such as described herein, see, e.g., U.S. Pat. No. 6,493,608 “Aspects of a Control System of a Minimally Invasive Surgical Apparatus,” and U.S. Pat. No. 6,671,581 “Camera Referenced Control in a Minimally Invasive Surgical Apparatus,” which are incorporated herein by reference. 
       FIG. 2  illustrates, as an example, a block diagram of components for controlling and selectively associating device manipulators to the input devices  108 ,  109 . Various surgical tools such as graspers, cutters, and needles may be used to perform a medical procedure at a work site within the Patient. In this example, two surgical tools  231 ,  241  are used to robotically perform the procedure and the camera  211  is used to view the procedure. The tools  231 ,  241  and camera  211  are inserted through passages in the entry guide  200 . As described in reference to  FIG. 1 , the entry guide  200  is inserted into the Patient through entry aperture  150  using the setup portion of the robotic arm assembly  130  and maneuvered by the entry guide manipulator (EGM)  202  of the robotic arm assembly  130  towards the work site where the medical procedure is to be performed. 
     Each of the devices  231 ,  241 ,  211 ,  200  is manipulated by its own manipulator. In particular, the camera  211  is manipulated by a camera manipulator (ECM)  212 , the first surgical tool  231  is manipulated by a first tool manipulator (PSM 1 )  232 , the second surgical tool  241  is manipulated by a second tool manipulator (PSM 2 )  242 , and the entry guide  200  is manipulated by an entry guide manipulator (EGM)  202 . So as to not overly encumber the figure, the devices  231 ,  241 ,  211 ,  200  are not shown, only their respective manipulators  232 ,  242 ,  212 ,  202  are shown in the figure. 
     Each of the instrument manipulators  232 ,  242 ,  212  is a mechanical assembly that carries actuators and provides a mechanical, sterile interface to transmit motion to its respective articulatable instrument. Each instrument  231 ,  241 ,  211  is a mechanical assembly that receives the motion from its manipulator and, by means of a cable transmission, propagates the motion to its distal articulations (e.g., joints). Such joints may be prismatic (e.g., linear motion) or rotational (e.g., they pivot about a mechanical axis). Furthermore, the instrument may have internal mechanical constraints (e.g., cables, gearing, cams, belts, etc.) that force multiple joints to move together in a pre-determined fashion. Each set of mechanically constrained joints implements a specific axis of motion, and constraints may be devised to pair rotational joints (e.g., joggle joints). Note also that in this way the instrument may have more joints than the available actuators. 
     In contrast, the entry guide manipulator  202  has a different construction and operation. A description of the parts and operation of the entry guide manipulator  202  is described below in reference to  FIG. 7 . 
     In this example, each of the input devices  108 ,  109  may be selectively associated with one of the devices  211 ,  231 ,  241 ,  200  so that the associated device may be controlled by the input device through its controller and manipulator. For example, by placing switches  258 ,  259  respectively in tool following modes “T 2 ” and “T 1 ”, the left and right input devices  108 ,  109  may be respectively associated with the first and second surgical tools  231 ,  241 , which are telerobotically controlled through their respective controllers  233 ,  243  (preferably implemented in the processor  102 ) and manipulators  232 ,  242  so that the Surgeon may perform a medical procedure on the Patient while the entry guide  200  is locked in place. 
     When the camera  211  or the entry guide  200  is to be repositioned by the Surgeon, either one or both of the left and right input devices  108 ,  109  may be associated with the camera  211  or entry guide  200  so that the Surgeon may move the camera  211  or entry guide  200  through its respective controller ( 213  or  203 ) and manipulator ( 212  or  202 ). In this case, the disassociated one(s) of the surgical tools  231 ,  241  is locked in place relative to the entry guide  200  by its controller. For example, by placing switches  258 ,  259  respectively in camera positioning modes “C 2 ” and “C 1 ”, the left and right input devices  108 ,  109  may be associated with the camera  211 , which is telerobotically controlled through its controller  213  (preferably implemented in the processor  102 ) and manipulator  212  so that the Surgeon may position the camera  211  while the surgical tools  231 ,  241  and entry guide  200  are locked in place by their respective controllers  233 ,  243 ,  203 . If only one input device is to be used for positioning the camera, then only one of the switches  258 ,  259  is placed in its camera positioning mode while the other one of the switches  258 ,  259  remains in its tool following mode so that its respective input device may continue to control its associated surgical tool. 
     On the other hand, by placing switches  258 ,  259  respectively in entry guide positioning modes “G 2 ” and “G 1 ”, the left and right input devices  108 ,  109  may be associated with the entry guide  200 , which is telerobotically controlled through its controller  203  (preferably implemented in the processor  102 ) and manipulator  202  so that the Surgeon may position the entry guide  200  while the surgical tools  231 ,  241  and camera  211  are locked in place relative to the entry guide  200  by their respective controllers  233 ,  243 ,  213 . As with the camera positioning mode, if only one input device is to be used for positioning the entry guide, then only one of the switches  258 ,  259  is placed in its entry guide positioning mode while the other one of the switches  258 ,  259  remains in its tool following mode so that its respective input device may continue to control its associated surgical tool. 
     The selective association of the input devices  108 ,  109  to other devices in this example may be performed by the Surgeon using the GUI  170  or the voice recognition system  160  in a conventional manner. Alternatively, the association of the input devices  108 ,  109  may be changed by the Surgeon depressing a button on one of the input devices  108 ,  109  or depressing the foot pedal  105 , or using any other well known mode switching technique. 
       FIGS. 3-4  respectively illustrate, as examples, top and right side views of a distal end of the entry guide  200  with the camera  211  and surgical tools  231 ,  241  extending outward. As shown in a perspective view of a simplified (not to scale) entry guide  200  in  FIG. 5 , the entry guide  200  is generally cylindrical in shape and has a longitudinal axis X′ running centrally along its length. The pivot point, which is also referred to as a remote center “RC”, serves as an origin for both a fixed reference frame having X, Y and Z axes as shown and an entry guide reference frame having X′, Y′ and Z′ axes as shown. When the system  100  is in the entry guide positioning mode, the entry guide manipulator  202  is capable of pivoting the entry guide  200  in response to movement of one or more associated input devices about the Z axis (which remains fixed in space) at the remote center “RC” in yaw ψ. In addition, the entry guide manipulator  202  is capable of pivoting the entry guide  200  in response to movement of the one or more input devices about the Y′ axis (which is orthogonal to the longitudinal axis X′ of the entry guide  200 ) in pitch θ, capable of rotating the entry guide  200  about its longitudinal axis X′ in roll Φ, and linearly moving the entry guide  200  along its longitudinal axis X′ in insertion/retraction or in/out “I/O” directions in response to movement of the one or more associated input devices. Note that unlike the Z-axis which is fixed in space, the X′ and Y′ axes move with the entry guide  200 . 
     As shown in  FIG. 7 , the entry guide manipulator (EGM)  202  has four actuators  701 - 704  for actuating the four degrees-of-freedom movement of the entry guide  200  (i.e., pitch θ, yaw ψ, roll Φ, and in/out I/O) and four corresponding assemblies  711 - 714  to implement them. 
     Referring back to  FIGS. 3-4 , the articulatable camera  211  extends through passage  321  and the articulatable surgical tools  231 ,  241  respectively extend through passages  431 ,  441  of the entry guide  200 . The camera  211  includes a tip  311  (which houses a stereo camera connected to a camera controller and a fiber-optic cable connected to an external light source), first, second, and third links  322 ,  324 ,  326 , first and second joint assemblies (also referred to herein simply as “joints”)  323 ,  325 , and a wrist assembly  327 . The first joint assembly  323  couples the first and second links  322 ,  324  and the second joint assembly  325  couples the second and third links  324 ,  326  so that the second link  324  may pivot about the first joint assembly  323  in pitch and yaw while the first and third links  322 ,  326  remain parallel to each other. 
     The first and second joints  323 ,  325  are referred to as “joggle joints”, because they cooperatively operate together so that as the second link  324  pivots about the first joint  323  in pitch and/or yaw, the third link  326  pivots about the second joint  325  in a complementary fashion so that the first and third links  322 ,  326  always remain parallel to each other. The first link  322  may also rotate around its longitudinal axis in roll as well as move in and out (e.g., insertion towards the work site and retraction from the worksite) through the passage  321 . The wrist assembly  327  also has pitch and yaw angular movement capability so that the camera&#39;s tip  311  may be oriented up or down and to the right or left, and combinations thereof. 
     The joints and links of the tools  231 ,  241  are similar in construction and operation to those of the camera  211 . In particular, the tool  231  includes an end effector  331  (having jaws  338 ,  339 ), first, second, and third links  332 ,  334 ,  336 , first and second joint assemblies  333 ,  335 , and a wrist assembly  337  that are driven by actuators such as described in reference to  FIG. 8  (plus an additional actuator for actuating the end effector  331 ). Likewise, the tool  241  includes an end effector  341  (having jaws  348 ,  349 ), first, second, and third links  342 ,  344 ,  346 , first and second joint assemblies  343 , 345 , and a wrist assembly  347  that are also driven by actuators such as described in reference to  FIG. 8  (plus an additional actuator for actuating the end effector  341 ). 
       FIG. 8  illustrates, as an example, a diagram of interacting parts of an articulatable instrument (such as the articulatable camera  211  and the articulatable surgical tools  231 ,  241 ) and its corresponding instrument manipulator (such as the camera manipulator  212  and the tool manipulators  232 ,  242 ). Each of the instruments includes a number of actuatable assemblies  821 - 823 ,  831 - 833 ,  870  for effectuating articulation of the instrument (including its end effector), and its corresponding manipulator includes a number of actuators  801 - 803 ,  811 - 813 ,  860  for actuating the actuatable assemblies. 
     In addition, a number of interface mechanisms may also be provided. For example, pitch/yaw coupling mechanisms  840 ,  850  (respectively for the joggle joint pitch/yaw and the wrist pitch/yaw) and gear ratios  845 ,  855  (respectively for the instrument roll and the end effector actuation) are provided in a sterile manipulator/instrument interface to achieve the required range of motion of the instrument joints in instrument joint space while both satisfying compactness constraints in the manipulator actuator space and preserving accurate transmissions of motion across the interface. Although shown as a single block  840 , the coupling between the joggle joint actuators  801 ,  802  (differentiated as #1 and #2) and joggle joint pitch/yaw assemblies  821 ,  822  may include a pair of coupling mechanisms—one on each side of the sterile interface (i.e., one on the manipulator side of the interface and one on the instrument side of the interface). Likewise, although shown as a single block  850 , the coupling between the wrist actuators  812 ,  813  (differentiated as #1 and #2) and wrist pitch/yaw joint assemblies  832 ,  833  may also comprise a pair of coupling mechanisms—one on each side of the sterile interface. 
     Both the joggle joint pitch assembly  821  and the joggle joint yaw assembly  822  share the first, second and third links (e.g., links  322 ,  324 ,  326  of the articulatable camera  211 ) and the first and second joints (e.g., joints  322 ,  325  of the articulatable camera  211 ). In addition to these shared components, the joggle joint pitch and yaw assemblies  821 ,  822  also include mechanical couplings that couple the first and second joints (through joggle coupling  840 ) to the joggle joint pitch and yaw actuators  801 ,  802  so that the second link may controllably pivot about a line passing through the first joint and along an axis that is latitudinal to the longitudinal axis of the first link (e.g., link  322  of the articulatable camera  211 ) and the second link may controllably pivot about a line passing through the first joint and along an axis that is orthogonal to both the latitudinal and longitudinal axes of the first link. 
     The in/out (I/O) assembly  823  includes the first link (e.g., link  322  of the articulatable camera  211 ) and interfaces through a drive train coupling the in/out (I/O) actuator  803  to the first link so that the first link is controllably moved linearly along its longitudinal axis by actuation of the I/O actuator  803 . The roll assembly  831  includes the first link and interfaces through one or more gears (i.e., having the gear ratio  845 ) that couple a rotating element of the roll actuator  811  (such as a rotor of a motor) to the first link so that the first link is controllably rotated about its longitudinal axis by actuation of the roll actuator  811 . 
     The instrument manipulator (e.g., camera manipulator  212 ) includes wrist actuators  812 ,  813  that actuate through wrist coupling  850  pitch and yaw joints  832 ,  833  of the wrist assembly (e.g., wrist  327  of the articulatable camera  211 ) so as to cause the instrument tip (e.g., camera tip  311 ) to controllably pivot in an up-down (i.e., pitch) and side-to-side (i.e., yaw) directions relative to the wrist assembly. The grip assembly  870  includes the end effector (e.g., end effector  331  of the surgical tool  231 ) and interfaces through one or more gears (i.e., having the gear ratio  855 ) that couple the grip actuator  860  to the end effector so as to controllably actuate the end effector. 
       FIG. 9  illustrates, as an example, a flow diagram of a method implemented in controller  102  of the medical robotic system  100  for providing a computer generated auxiliary view including articulatable instruments, such as the articulatable camera  211  and/or one or more of the articulatable surgical tools  231 ,  241 , extending out of the distal end of the entry guide  200 . For the purposes of this example, it is assumed that the articulatable camera  211  and surgical tools  231 ,  241  extend out of the distal end of the entry guide  200  and are included in the auxiliary view. However, it is to be appreciated that the method is applicable to any combination of articulatable instruments, including those without an articulatable camera and/or those with an alternative type of image capturing device such as an ultrasound probe. It is further to be appreciated that the method is applicable to articulatable instruments with more or less controllable joints than those described herein. In particular, the method is also applicable to highly jointed or otherwise bendable instruments and/or entry guides such as those that may be used to controllably navigate through various twists and turns in a patient&#39;s body to a target site for performing a medical procedure. 
     In  901 , the method determines whether or not an auxiliary view is to be generated. If the determination in  901  is NO, then the method loops back to periodically check to see whether the situation has changed. On the other hand, if the determination in  901  is YES, then the method proceeds to  902 . The indication that an auxiliary view is to be generated may be programmed into the controller  102 , created automatically or created by operator command. 
     In  902 , the method receives state information, such as positions and orientations, for each of the instruments  211 ,  231 ,  241  and the entry guide  200 . This information may be provided by encoders coupled to the actuators in their respective manipulators  212 ,  232 ,  242 ,  202 . Alternatively, the information may be provided by sensors coupled to joints and/or links of the instruments  211 ,  231 ,  241  and the entry guide manipulator  202 , or the coupling mechanisms, gears and drive trains of the interface between corresponding manipulators and instruments, so as to measure their movement. In this second case, the sensors may be included in the instruments  211 ,  231 ,  241  and entry guide manipulator  202  such as rotation sensors that sense rotational movement of rotary joints and linear sensors that sense linear movement of prismatic joints in the instruments  211 ,  231 ,  241  and entry guide manipulator  202 . Other sensors may also be used for providing information of the positions and orientations of the instruments  211 ,  231 ,  241  and entry guide  200  such as external sensors that sense and track trackable elements, which may be active elements (e.g., radio frequency, electromagnetic, etc.) or passive elements (e.g., magnetic, etc.), placed at strategic points on the instruments  211 ,  231 ,  241 , the entry guide  200  and/or the entry guide manipulator  202  (such as on their joints, links and/or tips). 
     In  903 , the method generates a three-dimensional computer model of the articulatable camera  211  and articulatable surgical tools  231 ,  241  extending out of the distal end of the entry guide  200  using the information received in  902  and the forward kinematics and known constructions of the instruments  211 ,  231 ,  241 , entry guide  200 , and entry guide manipulator  202 . The generated computer model in this example may be referenced to the remote center reference frame (X, Y, Z axes) depicted in  FIG. 5 . Alternatively, the generated computer model may be referenced to a reference frame defined at the distal end of the entry guide  200 . In this latter case, if the orientation and extension of the entry guide  200  from the remote center does not have to be accounted for in the auxiliary view that is being generated by the method, then the position and orientation information for the entry guide  200  may be omitted in  902 . 
     For example, referring to  FIG. 10 , if the state information received in  902  is the instruments&#39; joint positions  1001 , then this information may be applied to the instruments&#39; forward kinematics  1002  using the instruments&#39; kinematic models  1003  to generate the instruments&#39; link positions and orientations  1005  relative to reference frame  1004 . The same process may also be generally applied if the state information received in  902  is sensed states of the joggle coupling and gear mechanisms in the manipulator/instrument interfaces. 
     On the other hand, referring to  FIG. 11 , if the state information received in  902  is the instruments&#39; tip positions  1101  (in the reference frame  1004 ), then this information may be applied to the instruments&#39; inverse kinematics  1102  using the instruments&#39; kinematic models  1003  and the sensor reference frame to generate the instruments&#39; joint positions  1001 . The instruments&#39; joint positions  1001  may then be applied as described in reference to  FIG. 10  to generate the instruments&#39; link positions and orientations  1005  relative to reference frame  1004 . 
     Alternatively, also referring to  FIG. 11 , if the state information provided in  902  is limited to only the camera&#39;s tip position, then the positions of the tips of the surgical tools  231 ,  241  may be determined relative to the camera reference frame by identifying the tips in the image captured by the camera  211  using conventional image processing techniques and then translating their positions to the reference frame  1004 , so that the positions of the camera and tool tips may be applied as described in reference to  FIGS. 10, 11  to generate the instruments&#39; link positions and orientations  1005  relative to the reference frame  1004 . 
     In  904 , the method adjusts the view of the computer model of the articulatable camera  211  and articulatable surgical tools  231 ,  241  extending out of the distal end of the entry guide  200  in the three-dimensional space of the reference frame to a specified viewing point (wherein the term “viewing point” is to be understood herein to include position and orientation). For example,  FIG. 12  illustrates a top view of the articulatable camera  211  and articulatable surgical tools  231 ,  241  extending out of the distal end of the entry guide  200  which corresponds to a viewing point above and slightly behind the distal end of the entry guide  200 . As another example,  FIG. 13  illustrates a side view of the articulatable camera  211  and articulatable surgical tools  231 ,  241  extending out of the distal end of the entry guide  200  which corresponds to a viewing point to the right and slightly in front of the distal end of the entry guide  200 . Note that although the auxiliary views depicted in  FIGS. 12-13  are two-dimensional, they may also be three-dimensional views since three-dimensional information is available from the generated computer model. In this latter case, the auxiliary display screen  140  that they are being displayed on would have to be a three-dimensional display screen like the monitor  104 . 
     The viewing point may be set at a fixed point such as one providing an isometric (three-dimensional) view from the perspective shown in  FIG. 12 . This perspective provides a clear view to the surgeon of the articulatable camera  211  and the articulatable surgical tools  231 ,  241  when the tools  231 ,  241  are bent “elbows out” as shown (which is a typical configuration for performing a medical procedure using the surgical tools  231 ,  241 ). On the other hand, when a third surgical tool is being used (e.g., inserted in the passage  351  shown in  FIG. 6 ), a side view from the perspective of  FIG. 13  may additionally be useful since the third surgical tool may be beneath the articulatable camera  211  and therefore obscured by it in the perspective shown in  FIG. 12 . 
     Rather than setting the viewing point to a fixed point at all times, the viewing point may also be automatically changed depending upon the control mode (i.e., one of the modes described in reference to  FIG. 2 ) that is operative at the time. As an example,  FIG. 18  illustrates a method for automatically changing the auxiliary viewing mode depending upon the control mode currently operative in the medical robotic system  100 . In particular, using this method, a first auxiliary viewing mode is performed in  1802  when the medical robotic system  100  is determined in  1801  to be in a tool following mode, a second auxiliary viewing mode is performed in  1804  when the medical robotic system  100  is determined in  1803  to be in an entry guide positioning mode, and a third auxiliary viewing mode is performed in  1806  when the medical robotic system  100  is determined in  1805  to be in a camera positioning mode. The viewing modes for each control mode are selected so as to be most beneficial to the surgeon for performing actions during that mode. For example, in the tool following and camera positioning modes, either or both the surgical tools  231 ,  241  and camera  211  is being moved at the time and therefore, an auxiliary view of the articulatable camera  211  and articulatable surgical tools  231 ,  241  extending out of the distal end of the entry guide  200 , such as depicted in  FIGS. 12 and 13 , is useful to avoid collisions between links that are out of the field of view of the camera  211 . On the other hand, in the entry guide positioning mode, the articulatable camera  211  and the articulatable surgical tools  231 ,  241  are locked in position relative to the entry guide  200  and therefore, an auxiliary view providing information on other things such as depicted in  FIGS. 16 and 17 , or a computer generated view of the entry guide  200  from a perspective in space, may be useful. 
     Alternatively, operator selectable means for changing the viewing point during the performance of a medical procedure may be provided. For example, the GUI  170  or voice recognition system  160  may be adapted to provide an interactive means for the Surgeon to select the viewing mode and/or change the viewing point of an auxiliary view of the articulatable camera  211  and/or articulatable surgical tools  231 ,  241  as they extend out of the distal end of the entry guide  200 . Buttons on the input devices  108 ,  109  or the foot pedal  105  may also be used for Surgeon selection of viewing modes. For the Assistant(s), the input device  180  may be used along with a GUI associated with the display screen  140 ′ for selection of viewing modes. Thus, the viewing modes that the Surgeon and Assistant(s) see at the time may be optimized for their particular tasks at the time. Examples of such operator selectable viewing modes and viewing angles are depicted in  FIGS. 12-17 and 20-30 . 
     In  905 , the method renders the computer model. Rendering in this case includes adding three-dimensional qualities such as known construction features of the instruments  211 ,  231 ,  241  and the distal end of the entry guide  200  to the model, filling-in any gaps to make solid models, and providing natural coloring and shading. In addition, rendering may include altering the color or intensity of one or more of the instruments  211 ,  231 ,  241  (or one or more of their joints or links or portions thereof) so that the instrument (or joint or link or portion thereof) stands out for identification purposes. 
     Alternatively, the altering of the color, intensity, or frequency of blinking on and off (e.g., flashing) of one or more of the instruments  211 ,  231 ,  241  (or their joints, links, or portions thereof) may serve as a warning that the instrument (or joint or link or portion thereof) is approaching an undesirable event or condition such as nearing a limit of its range of motion or getting too close to or colliding with another one of the instruments. When color is used as a warning, the color may go from a first color (e.g., green) to a second color (e.g., yellow) when a warning threshold of an event to be avoided (e.g., range of motion limitation or collision) is reached, and from the second color to a third color (e.g., red) when the event to be avoided is reached. When intensity is used as a warning, the intensity of the color changes as the instrument (or portion thereof) moves past the warning threshold towards the event to be avoided with a maximum intensity provided when the event is reached. When blinking of the color is used as a warning, the frequency of blinking changes as the instrument (or portion thereof) moves past the warning threshold towards the event to be avoided with a maximum frequency provided when the event is reached. The warning threshold may be based upon a range of motion of the instrument (or portion thereof, such as its joints) or upon a distance between the instrument (or portion thereof) and another instrument (or portion thereof) that it may collide with. Velocity of the instrument&#39;s movement may also be a factor in determining the warning threshold. The warning threshold may be programmed by the operator, using the GUI  170 , for example, or determined automatically by a programmed algorithm in the processor  102  that takes into account other factors such as the velocity of the instruments&#39; movements. 
     Alternatively, the altering of the color, intensity, or frequency of blinking on and off (e.g., flashing) of one or more of the instruments  211 ,  231 ,  241  (or their joints, links, or portions thereof) may serve as an alert that the instrument (or joint or link or portion thereof) is approaching a desirable event or condition such as an optimal position or configuration for performing or viewing a medical procedure. In this case, an alert threshold may be defined so that the color, intensity, and/or blinking of the one or more of the instruments  211 ,  231 ,  241  (or their joints, links, or portions thereof) may change in a similar manner as described previously with respect to warning thresholds and undesirable events or conditions, except that in this case, the change starts when the alert threshold is reached and maximizes or otherwise ends when the desirable event or condition is reached or otherwise achieved. The alert threshold may also be programmed by the operator or determined automatically by a programmed algorithm in a conceptually similar manner as the warning threshold. 
     As an example of such highlighting of an instrument for identification, warning or alerting purposes,  FIG. 15  shows an auxiliary view of the camera  211  and surgical tools  231 ,  241  in a window  1502 , where the camera  211  has been highlighted. As an example of such highlighting of joints of instruments for identification, warning or alerting purposes,  FIG. 12  shows joints of the surgical tools  231 ,  241  that have been highlighted. As an example of highlighting portions of instruments for warning purposes,  FIG. 14  shows a portion  1402  of the surgical tool  241  and a portion  1403  of the camera  211  highlighted to indicate that these portions are dangerously close to colliding. 
     Rendering may also include overlaying the image captured by the camera  211  over the auxiliary view when the viewing point of the auxiliary image is the same as or directly behind that of the camera  211 . As an example,  FIG. 17  illustrates a captured image  1700  of the camera  211  rendered as an overlay to an auxiliary view of surgical tools  231 ,  241  which has been generated from a viewing point of (or right behind) the camera  211 . In this example, the auxiliary view of the surgical tools  231 ,  241  being displayed on the auxiliary display screen  140  (and/or the auxiliary display screen  140 ′) includes portions (e.g.,  1731 ,  1741 ) in the overlaying captured image  1700  and portions (e.g.,  1732 ,  1742 ) outside of the overlaying captured image  1700 . Thus, the portions of the surgical tools  231 ,  241  outside of the captured image  1700  provide the Surgeon with additional information about their respective links or articulating arms that are out of the field of view of the camera  211 . Highlighting of the instrument portions (e.g.,  1732 ,  1742 ) outside of the captured image  1700  may also be done for identification purposes or to indicate a warning or alerting condition as described above. Overlaying the captured image  1700  onto the auxiliary view also has the advantage in this case of showing an anatomic structure  360  which is in front of the surgical tools  231 ,  241  that would not otherwise normally be in the auxiliary view. Although this example shows the captured image  1700  overlaying the auxiliary view on the auxiliary display screen  140 , in another rendering scheme, the auxiliary view may overlay the captured image that is being displayed on the monitor  104 . 
     Rather than overlaying the captured image, rendering may also include using the auxiliary view to augment the image captured by the camera  211  by displaying only the portions of the instruments  231 ,  241  that are not seen in the captured image (i.e., the dotted line portion of the instruments  231 ,  241  in  FIG. 17 ) in proper alignment and adjacent the captured image in a mosaic fashion. 
     In addition to, or in lieu of, overlaying the captured image over the auxiliary view or augmenting the captured image with the auxiliary view, rendering may also include providing other useful information in the auxiliary view. As an example,  FIG. 16  illustrates an auxiliary side view of an articulatable camera  211  with a frustum  1601  rendered on the auxiliary view so as to be displayed on the auxiliary display  140  as emanating from, and moving with, the camera tip  311 . Note that although the frustum  1601  is shown in the figure as a truncated cone, it may also appear as a truncated pyramid to correspond to the captured image that is shown on the monitor  104 . The sides of the frustum  1601  indicate a viewing range of the camera  211  and the base  1602  of the frustum  1601  displays an image  1650  that was captured by the camera  211 . Note that for simplification purposes, the surgical tools  231 ,  241  normally in the auxiliary view have been removed for this example. As another example,  FIG. 14  shows a semi-translucent sphere or bubble  1401  (preferably colored red) which is displayed by the method as part of the rendering process when a warning threshold is reached so as to indicate to the operator that the highlighted portions  1402 ,  1403  of the surgical tool  241  and camera  211  are close to colliding. In this case, the highlighted portions  1402 ,  1403  are preferably centered within the sphere. As yet another example,  FIG. 14  also shows a marker or other indicator  1410  indicating an optimal position for the camera tip  311  for viewing the end effectors of the surgical tools  231 ,  241  as they are being used to perform a medical procedure. The optimal position may be determined, for example, by finding a location where the tips of the end effectors are equidistant from a center of the captured image. 
     In  906 , the method causes the rendered computer model (i.e., the auxiliary view) to be displayed on one or more displayed screens (e.g.,  140  and  140 ′) from the perspective of the selected viewing point. As shown in  FIGS. 12-14 and 16-17 , the auxiliary view is displayed on the auxiliary display screen  140 . As shown in  FIG. 14 , more than one auxiliary view may be displayed at one time (e.g., top and side perspectives may be provided at the same time respectively in windows  1421  and  1422 ). As shown in  FIG. 15 , the auxiliary view may also be displayed on the primary monitor  104  in a window  1502  that is adjacent to an image captured by the articulatable camera  211  which is being shown in another window  1501 . Although the windows  1501  and  1502  appear in this example to be the same size, it is to be appreciated that the position and size of the auxiliary view window  1502  may vary and still be within the scope of the present invention. Also, as previously mentioned, the auxiliary view may be overlayed the captured image in the window  1501  instead of in its own separate window  1502 . In such case, the overlayed auxiliary view may be switched on and off by the Surgeon so as not to clutter the captured image during the performance of a medical procedure. The switching on and off in this case may be performed by depressing a button on one of the input devices  108 ,  109  or depressing the foot pedal  105 . Alternatively, it may be done by voice activation using the voice recognition system  160  or through Surgeon interaction with the GUI  170  or using any other conventional function switching means. 
     After completing  906 , the method then loops back to  901  to repeat  901 - 906  for the next processing cycle of the controller  102 . 
     To assist the operator to make sure that the entry guide  200  and its articulatable instruments are well positioned (i.e., the instruments have wide range of motion during performance of a medical procedure at a target site in the patient), it is useful to provide indications of range of motion limitations in an auxiliary view that is displayed to the operator on one or more of the auxiliary display screens  140 ,  140 ′ and the monitor  104 . 
       FIG. 19  illustrates, as an example, a diagram of the tool instrument  231  from a right side view as it extends out of the distal end of the entry guide  200  with angles, link axes and lengths identified for determining indications of range of motion limitations for the articulatable instrument  231  that may be displayed in the auxiliary view. Due to its joggle joint construction, the instrument&#39;s first and third links  332 ,  336  are maintained in a parallel relationship with each other. Thus, when the first joint  333  is rotated to a maximum angle  1902 , the second joint  335  and wrist joint  337  (respectively at the proximal and distal ends of the third link  336 ) are both at a maximum displacement  1903  from the longitudinal axis  1901  of the first link  332 , which may be calculated as the length of the second link  334  times the sine function of the angle  1902 . If the first link  332  is fully rotatable about its longitudinal axis  1901 , a boundary limit for the third link  336  and consequently, the second joint  335  and wrist joint  337 , may be defined by a cylinder having the maximum displacement  1903  as its radius and a length determined by a maximum extension of the first link  332  out of the distal end of the entry guide  200 . Thus, for a two-dimensional view corresponding to a cross-sectional slice of the cylinder taken at a point along the third link  336  (or at its coupling joints  335 ,  337 ) a boundary limit represented as a circle may be defined for the instrument  231  and similar boundary circles may be defined for each of the other articulatable instruments extending out of the distal end of the entry guide  200 . Although the joint range of motion limits resemble circles in the present example, ellipses and other joint constrained boundary limits may also be accommodated in a similar manner as described herein for boundary circles. 
       FIG. 20  illustrates, as an example, a computer generated auxiliary view  2100  depicting graphical representations of articulatable instruments  211 ,  231 ,  241 ,  251  as the instruments are retracted back into the distal end of the entry guide  200  (from a perspective looking out from and directly behind the distal end from a vantage point along the longitudinal axis X′ of the entry guide  200 ) and indications of range of motion limitations  2011 ,  2031 ,  0241 ,  2051  respectively corresponding to the instruments  211 ,  231 ,  241 ,  251 . 
     The boundary circle  2031  for the tool instrument  231  is determined in this example as described in reference to  FIG. 19 . Boundary circles for the other instruments are determined in a similar fashion. Since the joggle joint constructions for the tool instruments  231 ,  241 ,  251  are the same, their respective boundary circles are of equal size, but displaced from each other so that each is centered along the longitudinal axis of its first link (i.e., in the centers of their respective graphical representations  231 ,  241 ,  251  in  FIG. 20 ). The joggle joint construction of the camera instrument  211 , however, is different in this example so that it results in a smaller boundary circle  2011 . In particular, the camera instrument  211  has either (or both) a smaller maximum angle of rotation for its first joint  323  or a shorter second link  324  than the tool instruments  231 ,  241 ,  251 . The boundary circle  2011 , however, is also centered along the first link  322  of its camera instrument  211 . 
     It is useful to distinguish boundary circles for instruments that are currently being controlled by the operator from boundary circles for instruments that are not currently being controlled by the operator. To this end, boundary circles  2031 ,  2041  are shown as solid circles, because their respective articulatable instruments  231 ,  241  are currently being controlled by input devices  108 ,  109  (i.e., they are in tool following mode) and boundary circles  2011 ,  2051  are shown as dotted circles, because their respective articulatable instruments  211 ,  251  are currently not being controlled by the input devices  108 ,  109 . Alternatively, boundary circles for disassociated instruments may not be displayed at all in the auxiliary view so as not to overly complicate it with unnecessary or unused information. 
     When the association of the input device  109  is switched so that it controls the tool  251  instead of the tool  231 , the boundary circle  2051  will become a solid circle and the boundary circle  2031  will become a dotted circle (or it will not be displayed at all) to indicate the control change. Likewise, when the association of the input devices  108 ,  109  is switched to a camera positioning mode, the boundary circle  2011  corresponding to the camera  211  will become a solid circle and the boundary circles  2031 ,  2041  corresponding to the instruments  231 ,  241  will become dotted circles (or they will not be displayed at all) to indicate the control change. Alternatively to using solid, dotted and invisible circles, control modes may also be indicated by a scheme using different color circles or by other visually distinguishable means such as blinking on and off boundary circles corresponding to instruments that are not being actively controlled at the time. 
       FIG. 21  illustrates, as an example, an auxiliary view  2100  providing additional detail for the articulatable instruments  211 ,  231 ,  241 ,  251  as some of them are shown extending out of the distal end of the entry guide  200  along with their indications of range of motion limitations  2011 ,  2031 ,  2041 ,  2051  corresponding to the instruments. In this example, tool instruments  231 ,  241  are being controlled by the operator in tool following mode using input devices  108 ,  109 , and instruments  251 ,  211  are not being controlled at the time by the operator. In particular, tool instrument  251  is out of use and retracted back to the distal end of the entry guide  200 , and the camera instrument  211  is held fixed in position by its controller  213  after being previously moved to look slightly to the left and downward. Consequently, boundary limits  2031 ,  2041  respectively corresponding to instruments  231 ,  241  are shown as solid circles and boundary limits  2011 ,  2051  respectively corresponding to instruments  211 ,  251  are shown as dotted circles in the auxiliary view  2100 . 
     Conceptually, the auxiliary view  2100  may overlay three cross-sectional slices for each of the articulatable instruments  211 ,  231 ,  241 ,  251  over a cross-sectional slice of the distal end of the entry guide  200 , wherein each of the slices is taken orthogonal to and is registered with the longitudinal axis X′ of the entry guide  200 . The first slice may be taken at each instrument&#39;s first joint (e.g., first joint  333  for tool  231  in  FIG. 19 ), a second slice may be taken at each instrument&#39;s wrist joint (e.g., wrist joint  337  for tool  231  in  FIG. 19 ), and a third slice may be taken at the instrument&#39;s distal tip (e.g., end effector distal tip  338  for tool  231  in  FIG. 19 ). 
     Although cross-sections of the first joint, wrist joint and distal tip for each of the articulatable instruments  211 ,  231 ,  241 ,  251  may be displayed in the auxiliary view  2100 , graphical representations in the form of objects such as circles or ellipses properly positioned where the cross-section slices are taken may be provided instead. In particular, graphical representations of the first joints  323 ,  333 ,  343 ,  353  are shown as circles or ellipses (identified by the same reference numbers as their respective first joints) whose positions in the auxiliary view  2100  indicate locations of their respective first links as they extend out of the distal end of the entry guide  200 ; graphical representations of the wrist joints  327 ,  337 ,  347  are shown as circles or ellipses (identified by the same reference numbers as their respective wrist joints) whose positions in the auxiliary view  2100  indicate articulation of the joggle joints of the instruments  211 ,  231 ,  241 ; and graphical representations of the distal tips  328 ,  338 ,  348  are shown as circles or ellipses (identified by the same reference numbers as their respective distal tips) whose positions in the auxiliary view  2100  indicate their orientations. As an example of determining the orientations of the distal tips, the orientation of the distal tip  338  of the tool  231  in  FIG. 19  is determinable from a roll angle  1907  of the first link  332  about its longitudinal axis  1901  and a pitch angle  1906  between longitudinal axes  1904 ,  1905  respectively of the third link  336  and the end effector  331  of the tool  231 . 
     To clearly distinguish the graphical representations of the distal tips  328 ,  338 ,  348  from those of their respective wrist joints  327 ,  337 ,  347 , the distal tips may be displayed in a different color or a different shade or in another visually distinguishable manner. Alternatively, or additionally, connecting segments may be displayed to identify corresponding first joints, wrist joints and distal tips of the same instrument. For example, a segment  2103  is shown connecting the graphical representation of the first joint  333  to the graphical representation of the wrist joint  337 , and a segment  2104  is shown connecting the graphical representation of the wrist joint  337  to the graphical representation of the distal tip  338  of the tool  231 . Connecting segments  2101 ,  2102  are also shown connecting the graphical representations of the first joint  343 , wrist joint  347  and distal tip  348  of the tool  241  in a similar manner. 
     As indicated by the auxiliary view  2100  of  FIG. 21 , the wrist joint  337  of the tool instrument  231  is close to its boundary limit  2031 . To warn the operator that the wrist joint  337  is nearing its range of motion limitation, a visual indication may be provided such as the color or shade of the graphical representation of the wrist joint  337  changing, the color or shade of a portion  2110  of the boundary limit  2031  closest to the wrist joint  337  changing, and/or the color or shade of one or both of the segments  2103 ,  2104  corresponding to the wrist joint  337  changing. Other visual indications such as blinking, arrows or warning text may also be used. Audio cues or warnings may also be provided along with or in lieu of any such visual indications described herein. 
     In addition to providing indications when the joggle joints are approaching their boundary limits, it is also desirable to provide indications when the articulatable instruments  211 ,  231 ,  241 ,  251  are reaching their maximum extensions out of the distal end of the entry guide  200 . The maximum limit boundaries may be indicated in supplemental auxiliary views such as extension limits  3011 ,  3012  in side supplemental auxiliary views  3001 ,  3002  respectively provided for tools  241 ,  231  on left and right sides of the auxiliary view  2100  in  FIG. 30 , and warnings provided when their respective first links near their extension limit using visual indications such as color or shade or other changes of the first link and/or any other parts of their respective articulatable instrument. 
       FIGS. 22-25  illustrate, as examples, various modifications to graphical representations that may be used in the auxiliary view  2100  for indicating the extent of the extension of the articulatable instrument  231  out of the distal end of the entry guide  200 . Similar modifications to graphical representations of the other instruments  211 ,  241 ,  251  may be used for the same purpose. As shown in  FIG. 22 , the length of rays  2201  emanating from the graphical representation of the wrist joint  337  serve to indicate the extent of the extension (i.e., the length  1909  in  FIG. 19 ) of the first link  332  out of the distal end of the entry guide  200 . Alternatively, or additionally, as shown in  FIG. 23 , the length of rays  2301  emanating from the graphical representation of the distal tip  338  may serve to indicate the extent of the extension of the first link  332  out of the distal end of the entry guide  200 . Alternatively, or additionally, as shown in  FIG. 24 , the relative sizes, colors and/or shades of the graphical representations for the first joint  333 , wrist joint  337  and distal tip  338  may serve to indicate the extent of the extension of the first link  332  out of the distal end of the entry guide  200 . As an example, as the first link  332  extends further out of the distal end of the entry guide  200 , differences in the relative sizes between two or more of the graphical representations for the first joint  333 , wrist joint  337  and distal tip  338  may get increasingly larger. Alternatively, or additionally, as shown in  FIG. 25 , the relative sizes, colors and/or shades of the graphical representations for the segments  2501 ,  2502  may serve to indicate the extent of the extension of the first link  332  out of the distal end of the entry guide  200 . 
     The graphical representations for the distal tips of the instruments may also provide other state information for their tools or camera in addition to displaying graphical representations in the auxiliary view  2100  that indicate joggle joint articulations, extension/retraction of the articulatable instruments  211 ,  231 ,  241 ,  251  and graphical representations of boundaries indicating range of motion limitations for the instruments. As an example,  FIG. 26  illustrates a graphical representation of the distal tip  338  of tool  231  which includes elements  2601 ,  2602  that define an angle  2603  between them that is indicative of how much the jaws  338 ,  339  of the end effector  331  are open or closed. As another example,  FIG. 27  illustrates a graphical representation of the distal tip  328  (including a camera) of the camera instrument  211  which depicts an area  2701  indicative of a field-of-view of the camera instrument  211 . 
     The auxiliary view  2100  may also be used to assist the operator in repositioning the entry guide  200  so that the articulatable instruments are better positioned for performing a medical procedure. 
       FIG. 28  illustrates, as an example, a simplified auxiliary view  2100  of a poor position of the entry guide  200  wherein each of the wrist joints  327 ,  337 ,  347  is near its boundary limit  2011 ,  2031 ,  2041 . To simplify the figure, the tool  251  and graphical representations of the first joints  323 ,  333 ,  343  of the instruments  211 ,  231 ,  241  are omitted so as to not overly complicate it with details. 
     By switching to the entry guide positioning mode as described in reference to  FIG. 2 , the positions of the camera tip  311  of the camera instrument  211  and end effectors  331 ,  341  of the tool instruments  231 ,  241  will be held in place by their respective controllers while the operator repositions the entry guide  200  using one or both of the input devices  108 ,  109 . In particular, the camera tip  311  and end effectors  331 ,  341  are held in place by holding the positions of their wrist joints  327 ,  337 ,  347  and distal tips  328 ,  338 ,  348  in place using their respective controllers while the entry guide  200  is repositioned. The first joints  323 ,  333 ,  343  and boundary limits  2011 ,  2031 ,  2041  of the instruments  211 ,  231 ,  241  move, however, as the entry guide  200  moves. 
       FIG. 29  illustrates, as an example, a simplified auxiliary view  2100  after the entry guide  200  has been repositioned relative to the wrist joints  327 ,  337 ,  347  and distal tips  328 ,  338 ,  348  of the instruments  211 ,  231 ,  241  shown in  FIG. 28  by translating it a distance  2901  so that each of the wrist joints  327 ,  337 ,  347  is better positioned within its boundary limit  2011 ,  2031 ,  2041  for improved range of motion. 
     The auxiliary view  2100  as depicted in  FIGS. 20-29  may be generated by the controller  102  using a computer implemented method such as described in reference to  901 - 905  of  FIG. 9  with modifications for generating and displaying the joggle joint cross-sectional slices and boundary limits from the perspective looking out of the distal end of the entry guide  200 . The computer generated auxiliary view  2100  may then be displayed on the monitor  104  and/or the auxiliary display screens  140 ,  140 ′ alone or in combination with camera captured images and/or other computer generated views such as described in reference to  906  of  FIG. 9 . 
       FIG. 30  illustrates, as an example, a display screen of the monitor  104  in which an image  1501  captured by the camera instrument  211  is shown in a main window, an auxiliary view  2100  of articulatable instruments  211 ,  231 ,  241  extending out of the entry guide  200  is shown in a lower central window, and supplemental auxiliary views  3001 ,  3002  of the tools  241 ,  231  from a different perspective than that of the view  2100  are shown respectively in lower side windows. In this arrangement of views, indications of joggle joint boundary limits may be provided in the lower central window as described in reference to  FIGS. 13-29  and indications of extension limits for the articulatable instruments  241 ,  231  may be provided in the lower side views as previously explained. Visual cues or warnings may also be provided in the auxiliary views as described herein when the articulatable instruments extending out of the distal end of the entry guide  200  are approaching their respective range of motion limitations and/or threatening to collide with one another. 
     Although the various aspects of the present invention have been described with respect to a preferred embodiment, it will be understood that the invention is entitled to full protection within the full scope of the appended claims.