Patent ID: 12201484

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is intended. In the following detailed description of the aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be obvious to one skilled in the art that the embodiments of this disclosure may be practiced without these specific details. In other instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.

Any alterations and further modifications to the described devices, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and/or steps described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and/or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately. For simplicity, in some instances the same reference numbers are used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions of instruments in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom—e.g., roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom).

Referring toFIG.1Aof the drawings, a teleoperational medical system for use in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures, is generally indicated by the reference numeral10. As will be described, the teleoperational medical systems of this disclosure are under the teleoperational control of a surgeon. In alternative embodiments, a teleoperational medical system may be under the partial control of a computer programmed to perform the procedure or sub-procedure. In still other alternative embodiments, a fully automated medical system, under the full control of a computer programmed to perform the procedure or sub-procedure, may be used to perform procedures or sub-procedures. As shown inFIG.1A, the teleoperational medical system10is positioned in a surgical environment11and generally includes a teleoperational assembly12mounted to or near an operating table O on which a patient P is positioned. The teleoperational assembly12may be referred to as a patient side cart. A medical instrument system14and an endoscopic imaging system15are operably coupled to the teleoperational assembly12. An operator input system16allows a surgeon or other type of clinician S to view images of or representing the surgical site and to control the operation of the medical instrument system14and/or the endoscopic imaging system15.

The operator input system16may be located at a surgeon's console, which is usually located in the same room as operating table O. It should be understood, however, that the surgeon S can be located in a different room or a completely different building from the patient P. Operator input system16generally includes one or more control device(s) for controlling the medical instrument system14. The control device(s) may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and the like. In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical instruments of the teleoperational assembly to provide the surgeon with telepresence, the perception that the control device(s) are integral with the instruments so that the surgeon has a strong sense of directly controlling instruments as if present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide the surgeon with telepresence. In some embodiments, the control device(s) are manual input devices which move with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaw end effectors, applying an electrical potential to an electrode, delivering a medicinal treatment, and the like).

The teleoperational assembly12supports and manipulates the medical instrument system14while the surgeon S views the surgical site through the console16. An image of the surgical site can be obtained by the endoscopic imaging system15, such as a stereoscopic endoscope, which can be manipulated by the teleoperational assembly12to orient the endoscope15. The number of medical instrument systems14used at one time will generally depend on the diagnostic or surgical procedure and the space constraints within the operating room among other factors. The teleoperational assembly12may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, generally referred to as a set-up structure) and a teleoperational manipulator. The teleoperational assembly12includes a plurality of motors that drive inputs on the medical instrument system14. These motors move in response to commands from the control system (e.g., control system20). The motors include drive systems which when coupled to the medical instrument system14may advance the medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems may move the distal end of the medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate an articulable end effector of the instrument for grasping tissue in the jaws of a biopsy device or the like. Instruments14may include end effectors having a single working member such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or clip appliers.

The teleoperational medical system10also includes a control system20. The control system20includes at least one memory24and at least one processor22, and typically a plurality of processors, for effecting control between the medical instrument system14, the operator input system16, and other auxiliary systems which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and/or suction systems. The control system20can be used to process the images of the surgical environment from the imaging system15for subsequent display to the surgeon S through the surgeon's console16. The control system20also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. While control system20is shown as a single block in the simplified schematic ofFIG.1A, the system may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent the teleoperational assembly12, another portion of the processing being performed at the operator input system16, and the like. Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the teleoperational systems described herein. In one embodiment, control system20supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

In some embodiments, control system20may include one or more servo controllers that receive force and/or torque feedback from the medical instrument system14. Responsive to the feedback, the servo controllers transmit signals to the operator input system16. The servo controller(s) may also transmit signals instructing teleoperational assembly12to move the medical instrument system(s)14and/or endoscopic imaging system15which extend into an internal surgical site within the patient body via openings in the body. Any suitable conventional or specialized servo controller may be used. A servo controller may be separate from, or integrated with, teleoperational assembly12. In some embodiments, the servo controller and teleoperational assembly are provided as part of a teleoperational arm cart positioned adjacent to the patient's body.

The control system20can be coupled with the endoscope imaging system15and can include a processor to process captured images for subsequent display, such as to a surgeon on the surgeon's console, or on another suitable display located locally and/or remotely. For example, where a stereoscopic endoscope is used, the control system20can process the captured images to present the surgeon with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope.

A surgical environment monitoring system, including one or more monitoring devices such as cameras27a,27b,27c, is positioned in the surgical environment11. The cameras27a-cmay be used to capture images in the surgical environment11outside of the patient P anatomy. For example and as will be described further, the cameras27a-cmay be used to monitor the extremities of the surgeon S during a procedure. The images of the surgeon's hands and feet may be presented to the surgeon through a display at the console16to assist the surgeon during transitions that require movement of the extremities to control operation of the system10. The cameras27a-cmay also or alternatively be used to capture images of the external patient anatomy, the teleoperational assembly, other equipment in the surgical environment, and personnel in the surgical environment. The cameras27a-cmay be mounted in any of a variety of ways including on discrete pedestals or tripods, from the ceiling, on equipment in the surgical environment including the orienting platform53, on the shafts of the instruments14or endoscope15external of the patient anatomy, or on equipment worn by the surgeon S or by other personnel in the surgical environment, such as a head-mounted camera.

A gesture-based interface (GBI)29may also be located in the surgical environment11. The GBI may be a touch-based interface system such as a computer tablet or may be a three-dimensional tracking and interface system such as a Leap Motion system available from Leap Motion, Inc. of San Francisco, California or such as Kinect from Microsoft Corporation of Redmond, Washington. Additionally or alternatively, the GBI may be a wearable device such as a head-mounted device. The GBI29may be used to track two or three-dimensional user inputs from the surgeon S or other surgical personnel.

A patient side interface (PSI)26may be located or locatable near the bedside of the patient. The PSI26may allow the surgeon S to approach the patient and still have access to at least some functionality of the console16or additional inputs not available at the console16. The PSI26may include a display for displaying similar or different images from those displayed at the console16. The PSI26may include a head-mounted display system, a boom-mounted display system, or a dome-style display system that provides primary and peripheral images or 360° images of the surgical environment. The PSI26may also include a user input device such as a computer tablet, trackball, or three-dimensional input system. In some embodiments, the PSI26may include all or some components of the GBI29.

In alternative embodiments, the teleoperational system may include more than one teleoperational assembly and/or more than one operator input system. The exact number of manipulator assemblies will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems may be co-located, or they may be positioned in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.

FIG.1Bis a perspective view of one embodiment of a teleoperational assembly12which may be referred to as a patient side cart. The patient side cart12shown provides for the manipulation of three surgical tools30a,30b,30c(e.g., instrument systems14) and an imaging device28(e.g., endoscopic imaging system15), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The imaging device may transmit signals over a cable56to the control system20. Manipulation is provided by teleoperative mechanisms having a number of joints. The imaging device28and the surgical tools30a-ccan be positioned and manipulated through incisions in the patient so that a kinematic remote center is maintained at the incision to minimize the size of the incision. Images of the surgical environment within the patient anatomy can include images of the distal ends of the surgical tools30a-cwhen they are positioned within the field-of-view of the imaging device28.

The patient side cart12includes a drivable base58. The drivable base58is connected to a telescoping column57, which allows for adjustment of the height of the arms54. The arms54may include a rotating joint55that both rotates and moves up and down. Each of the arms54may be connected to an orienting platform53. The orienting platform53may be capable of 360 degrees of rotation. The patient side cart12may also include a telescoping horizontal cantilever52for moving the orienting platform53in a horizontal direction.

In the present example, each of the arms54connects to a manipulator arm51. Each manipulator arms51may connect to a respective one of the medical tools30a-cor to the imaging device28. The manipulator arms51may be teleoperable. In some examples, the arms54connecting to the orienting platform are not teleoperable. Rather, such arms54are positioned as desired before the surgeon18begins operation with the teleoperative components.

Endoscopic imaging systems (e.g., systems15,28) may be provided in a variety of configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Digital image based endoscopes have a “chip on the tip” design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data. Endoscopic imaging systems may provide two- or three-dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic instruments employ stereo cameras to capture stereo images of the patient anatomy. An endoscopic instrument may be a fully sterilizable assembly with the endoscope cable, handle and shaft all rigidly coupled and hermetically sealed.

FIG.1Cis a perspective view of the surgeon's console16. The surgeon's console16includes a left eye display32and a right eye display34for presenting the surgeon S with a coordinated stereo view of the surgical environment that enables depth perception. The displayed image of the surgical environment may be obtained from an imaging system such as the endoscopic imaging system. Additionally or alternatively, the displayed image of the surgical environment may include images from anatomic models created from pre-operative or intra-operative image data sets. Pre-operative or intraoperative image data set of the patient anatomy may be obtained using external or non-invasive imaging technology such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, or the like. Software alone or in combination with manual input is used to convert the recorded images into segmented two dimensional or three dimensional composite models representing a partial or an entire anatomical organ or anatomical region. An image data set is associated with the composite representation. The images used to generate the composite representation may be recorded preoperatively or intra-operatively during a clinical procedure. The pre-operative or intra-operative image data may be presented as two-dimensional, three-dimensional, or four-dimensional (including e.g., time based or velocity based information) images or as images from models created from the pre-operative or intra-operative image data sets. Images from different imaging modalities may be displayed one at a time (e.g., the surgeon may toggle through the different modality images), may be displayed in parallel (e.g., in multiple windows of a composite display) or one may be overlaid or superimposed on the other.

The console16further includes one or more input control devices36, which in turn cause the teleoperational assembly12to manipulate one or more instruments or the endoscopic imaging system. The input control devices36can provide the same degrees of freedom as their associated instruments14to provide the surgeon S with telepresence, or the perception that the input control devices36are integral with the instruments14so that the surgeon has a strong sense of directly controlling the instruments14. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the instruments14back to the surgeon's hands through the input control devices36. Input control devices37are foot pedals that receive input from a user's foot. Optionally, input control device38may include a touch-based input device such as a computer tablet. Optionally, a gesture-based interface may be included in the console16.

During a teleoperational procedure, entirely virtual and/or augmented reality images may be provided to the surgeon S and/or other surgical personnel to provide a more extensive view of the surgical environment, provide additional information about the patient or the procedure, and/or provide additional controls for use during the procedure. Various systems and methods for providing virtual or augmented reality images during a teleoperational procedure are disclosed in U.S. Pat. No. 8,520,027, filed May 14, 2010, disclosing “Method and System of See-Through Console Overlay;” International Publication Number WO 2014/176403, filed Apr. 24, 2014, disclosing “Surgical Equipment Control Input Visualization Field;” U.S. Pat. No. 8,398,541, filed Aug. 11, 2008, disclosing “Interactive User Interfaces for Robotic Minimally Invasive Surgical Systems;” U.S. Pat. No. 9,788,909, filed Nov. 11, 2013, disclosing “Synthetic Representation of a Surgical Instrument;” and U.S. Pat. No. 9,858,475, filed May 14, 2010 disclosing “Method and System of Hand Segmentation and Overlay Using Depth Data,” which are incorporated by reference herein in their entirety.

FIG.2is a display100illustrating surgical environment image102, which in this example is an interior patient anatomy image obtained by an endoscopic imaging system (e.g., system15,28) augmented with a cursor image104of a real-time image of a user's extremity, which in this example is a right hand of the surgeon. The image of the surgeon's hand may be obtained by a camera27a-cor by the GBI system29and used as a cursor to indicate the current position for interaction with an object in the image102. The endoscopic image102includes an image of an instrument106in the surgical environment. In this example the instrument106is a retractor. When the teleoperational system is in an adjustment mode, the surgeon S may move his right hand, thereby moving the cursor104to interact with the image of the instrument106(e.g., a retractor), to cause an adjustment in the actual position of the retractor in the surgical environment. The cursor image104of the surgeon's hand allows the surgeon to visualize his hand virtually selecting and moving the retractor106. The movement of the retractor in the surgical environment may be generated by the teleoperational system in response to the commanded motion of the surgeon's hand. In various alternative embodiments, a cursor image may be a static (i.e., previously captured) image of the user's extremity (e.g., a hand, finger, foot) or another type of static cursor symbol depicting a portion of a human anatomy (e.g., an image of an eye or a head).

FIG.3illustrates a method150of adjusting a virtual control element with a cursor image of a user's extremity. The method150is illustrated inFIG.3as a set of operations or processes. Not all of the illustrated processes may be performed in all embodiments of method150. Additionally, one or more processes that are not expressly illustrated inFIG.3may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the processes of method150may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system20) may cause the one or more processors to perform one or more of the processes.

At a process152, a virtual control element is displayed. The virtual control element may be a sliding adjuster, a toggle switch, a dial, or other element for controlling a binary system function (e.g. on/off) or a variable system function (e.g., power level, brightness level, sound level, frequency level) of a component of the teleoperational system or an auxiliary equipment component in the surgical environment. At a process154, a cursor image of a user (e.g. the surgeon S) body part (e.g., the hand of surgeon S) used for controlling the virtual control element is displayed. At a process156, a gesture based interface (e.g. GBI29) receives input from the surgeon S by registering movement of the user's hand virtually manipulating the virtual control element as the real-time cursor image of the user's hand interacts with the virtual control element. At a process158, the binary system function or variable system function is changed or adjusted based on the movement of the user's hand. The method150is further illustrated with reference toFIGS.4and5.

With reference toFIG.4A, an image200includes a cursor image of a user's extremity (e.g., a right hand)202interacts with a virtual control element204, which in this example includes a virtual slider206. A graphical element such as a graphical identifier208of the system controlled by the control element204may also be included. In this example, the identifier208is a textual description, but in other examples the identifier208may be a pictorial representation of the system controlled by the control element. In this example, the virtual control element204may control the variable level of power, sound, frequency, or other characteristic of an auxiliary system. The auxiliary system may be, for example, a power generator, a speaker, a display screen, or an irrigation system. As the user's hand moves the slider206to the right, the power level of the generator increases and as the slider moves to the left, the power level decreases. The user's hand may be tracked by a GBI or a touch-based input system. Additionally, the real-time cursor image of the user's hand202may be generated by a camera27a-cto provide the user with spatial awareness of his hand relative to the virtual control element204. Optionally, the images202-208may be superimposed on, integrated with, or otherwise augment an image of either the internal patient anatomy or an image of the surgical environment external of the patient anatomy.

FIG.4Billustrates an image220of the real-time cursor image202interacting with a real-time image of an auxiliary component222in the surgical environment. In this example, the image220captures the surgical environment which includes the auxiliary component222. The image of the auxiliary component in the image220serves as a virtual control element for the actual auxiliary component222in the surgical environment. For example, if the auxiliary component222is a high frequency power generator, the surgeon may gesture with his hand in a predetermined motion toward the auxiliary component222in the surgical environment to change the power level of the generator. An upward hand or finger gesture may correspond to an increase in power level and a downward hand or finger gesture may correspond to a decrease in power level. Alternatively, the auxiliary component222may include a power control knob224, and a clockwise hand gesture toward the knob224may correspond to an increase in power level with a counter-clockwise hand gesture may correspond to a decrease in power level. In alternative embodiments, the auxiliary system may be, for example, a speaker, a display screen, or an irrigation system. The user's hand may be tracked by a GBI or a touch-based input system. Additionally, the real-time cursor image of the user's hand202may be generated by a camera27a-cto provide the user with spatial awareness of his hand relative to the virtual control element.

FIG.5illustrates an image230of a surgical environment with a real-time cursor image232interacting with an information icon234adisplayed over the surgical environment. In this example, the image230captures the surgical environment which includes instruments236a,236b, and236c. Information icon234ais displayed near icon234a. Information icon234bis displayed near icon234b. Information icon234cis displayed near icon234c. As the cursor image232contacts or comes into proximity with one of the information icons234a-b, information about the respective instrument236a-cis displayed in an information cloud. For example, information cloud238is displayed when the cursor image232interacts with the information icon234ato provide information about the tool236a. The provided information may include, for example, the type of instrument, the activation status of the instrument, instructions about operating or trouble-shooting the instrument, and/or buttons activatable by the cursor232to effect the operation of the instrument236a.

In an alternative embodiments, the cursor image of the user's extremity may be a real-time image of the user's foot as it moves between pedal inputs37. The real-time image of the foot and pedals may be obtained by a camera27a-cand may be presented as a separately displayed image, as a picture-in-picture within the current endoscopic image, or behind a semi-transparent current endoscopic image to provide the surgeon with the sense that he is looking through the console16to view his foot as it moves toward a different pedal. In still another alternative embodiment, the image of the user's extremity may be a real-time image of the user's hand as it transitions out of or into engagement with the input control devices36. The real-time image of the hand and control devices36may be obtained by a camera27a-cand may be presented as a separately displayed image, as a picture-in-picture within the current endoscopic image, or behind a semi-transparent current endoscopic image to provide the surgeon with the sense that he is looking through the console16to view his hand as it moves into or out of the control device36. Allowing the surgeon to see his hands or feet as they transition between positions may boost the surgeon's confidence that his hands and feet, which are not directly visible due to the console16blocking the user's view, are making the correct engagement with the input devices36,37.

FIG.6illustrates a method250of patient marking using a gesture based input device. The method250is illustrated inFIG.6as a set of operations or processes. Not all of the illustrated processes may be performed in all embodiments of method250. Additionally, one or more processes that are not expressly illustrated inFIG.6may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the processes of method250may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system20) may cause the one or more processors to perform one or more of the processes.

At a process252, a surgical environment including a virtual marking element is displayed. The virtual marking element may be used, for example, to indicate the surgeon's preferred entry port locations or may mark anatomic features of the patient. At a process254, a cursor image of a user (e.g. the surgeon S) body part (e.g., the hand of surgeon S) used for interacting with the virtual marking element is displayed. At a process256, a gesture-based interface (e.g. GBI29) receives input from the surgeon S by registering movement of the user's hand virtually interacting with the marking element as the real-time cursor image of the user's hand interacts with the virtual marking element. For example, the movement of the user's hand may be used to create a new marker or move a marker from a default location. At a process258, the patient anatomy is marked (e.g. with light, ink, or other marking material) based on the position of the marking element. The method250is further illustrated with reference toFIG.7.

With reference toFIG.7, a surgical environment image300includes an exterior image301of a patient anatomy. A cursor image of a user's extremity (e.g., a right hand)202interacts with virtual marking elements302,304,306. The virtual marking elements may be created by a gesture of the user's hand or may be created in a default location. The user's hand may interact with a virtual marking element306to move the marking element to a different location than where it was originally created. In this example the virtual marking elements302,304,306may be used to mark port locations on the image301of the patient anatomy. The virtual marking elements be virtually dragged relative to the image301of the patient anatomy by the tracked motion of the user's hand. The user's hand may be tracked by a GBI or a touch-based input system. Additionally, the real-time cursor image of the user's hand202may be generated by a camera27a-cto provide the user with spatial awareness of his hand relative to the virtual marking elements302,304,306. After the virtual marking elements302,304,306are established relative to the image301of the patient anatomy, corresponding actual marks may be made with light, ink, or another marking medium on the anatomy of the patient to mark the locations of entry ports prior to a surgical procedure. The visual marking elements may, for example, indicate candidate incision locations which may be evaluated by the control system to provide feedback on the feasibility of the incision locations by evaluating reachable workspace and the likelihood of internal or external collisions. The virtual marking elements and the surrounding areas may be color coded based on the predicted feasibility measures.

FIG.8illustrates a method450of displaying an image of an internal patient anatomy at least partially surrounded by the displayed image of the external patient anatomy. The method450is illustrated inFIG.8as a set of operations or processes. Not all of the illustrated processes may be performed in all embodiments of method450. Additionally, one or more processes that are not expressly illustrated inFIG.8may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the processes of method450may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system20) may cause the one or more processors to perform one or more of the processes.

At a process452, a first image from a first imaging device is received. The first image may be an internal view of the patient anatomy, received, for example, from an endoscopic device. At a process454, a second image from a second imaging device is received. The second image may be an external view of the patient anatomy and/or the surgical environment surrounding the patient anatomy. At a process456, the first image is displayed in spatial context relative to the second image. For example, as shown inFIG.9, a display500includes an image502of the internal patient anatomy at least partially surrounded by the displayed image504of the external patient anatomy. The image502may be obtained by an endoscopic imaging system (e.g., system15,28), and the image504may be obtained by an imaging system such a camera27a-c. The viewing orientation of the internal and external views may be aligned. For example, the external view may be digitally rotated to share the same roll angle as the internal (e.g., endoscopic) view. Additionally or alternatively, the pitch and/or yaw of the external view may be aligned with the viewing direction of the internal view so that internal and external motions can be intuitively controlled from the same reference frame.

One or more elements in embodiments of the invention may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device, The code segments may be downloaded via computer networks such as the Internet, Intranet, etc.

Note that the processes and displays presented may not inherently be related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear as elements in the claims. In addition, the embodiments of the invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.