Abstract:
A teleoperated medical device includes a sensor that senses information that represents a position of a control input out of an operator&#39;s field of view and displays the information to the operator. The displayed information may be combined with an image of a surgical site, so that the operator can locate and operate the control input while continuously viewing the image of the surgical site.

Description:
FIELD 
       [0001]    The present disclosure is directed to systems and methods for surgery, and more particularly to systems and methods for providing clinicians with viewing of a surgical site and a medical equipment control component outside his field of view. 
       BACKGROUND 
       [0002]    It is a primary objective of surgery, whether open or through the use of minimally invasive techniques, to reduce the duration of the procedure and minimize blood loss, the risk of infection, and other bad side effects. With the advent of new surgical equipment, additional equipment and technologies are incorporated into the procedure with the primary purpose of improving the efficacy of the procedure. Incorporation of new equipment and technologies into procedures sometimes complicates things by requiring the surgeon to access additional control components. In open surgery, the surgeon&#39;s field of view typically includes only the surgeon&#39;s hands and not the surgeon&#39;s feet. Alternatively, during minimally invasive surgery, the surgeon&#39;s attention is typically focused on a video display of the surgical site. Here, the surgeon&#39;s own hands and the control components they operate are likely outside his or her field of view. If the surgeon seeks to locate a control component outside his field of view, the surgery must be interrupted, however briefly, to permit the surgeon to turn away from the surgical site. Such interruptions and distractions disturb surgical work flow, and increased procedure duration or even patient harm may result. Improved systems and methods are needed to allow a surgeon to easily access various surgical equipment control components while minimizing impact on surgical work flow. 
       SUMMARY 
       [0003]    The present invention provides systems and methods for medical persons to view surgical equipment control components outside his or her field of view during surgery. The systems provided include an image capture device and a display coupled to the image capture device. The image capture device is configured to capture within its field of view part of a surgical equipment control component. This image captured by the image capture device is output by the display. The display is positioned within the field of view of the medical person operating the surgical equipment control component. 
         [0004]    The methods provided involve capturing an image containing part of a surgical equipment control component and part of the body part of the medical person used to operate the surgical equipment control component. The captured image is then displayed to the medical person operating the surgical equipment control component. 
     
    
     
       BRIEF DESCRIPTIONS OF THE DRAWINGS 
         [0005]    Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. Various features are not drawn to scale. The dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion. 
           [0006]      FIG. 1  is a diagrammatic view of display system components. 
           [0007]      FIG. 2  is a plan view of a telesurgical robotic system. 
           [0008]      FIG. 3  is a front elevation view of a master control console in a telesurgical robotic system. 
           [0009]      FIG. 4  is a front elevation view of a robotic slave device manipulator cart in a telesurgical robotic system. 
           [0010]      FIGS. 5A ,  5 B, and  5 C are perspective views that illustrate a surgeon&#39;s foot on various foot pedals. 
           [0011]      FIG. 6  is an image of the surgical site and an image of the touch-sensitive control interface, output on a single display. 
       
    
    
     DETAILED DESCRIPTION 
       [0012]    This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting—the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the invention. Like numbers in two or more figures represent the same or similar elements. 
         [0013]    Further, this description&#39;s terminology is not intended to limit the invention. For example, spatially relative terms—such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like—may be used to describe one element&#39;s or feature&#39;s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of a device in use or operation in addition to the position and orientation shown in the figures. For example, if a device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes includes various special device positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and/or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components. 
         [0014]    Elements described in detail with reference to one embodiment, implementation, or application may, whenever practical, be included in other embodiments, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions. 
         [0015]    Aspects of the invention are described primarily in terms of an implementation using a da Vinci® Surgical System (specifically, a Model IS3000, marketed as the da Vinci® Si™ HD™ Surgical System), commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments and implementations. Implementations on da Vinci® Surgical Systems (e.g., the Model IS3000; the Model IS2000, commercialized as the da Vinci® S™ HD™ Surgical System) are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. 
         [0016]      FIG. 1  is a diagrammatic view of a display system  100 . Display system  100  includes an image capture device  102  and a display  104  coupled to capture and output moving visual (video) images  106  from image capture device  102 . Image capture device  102  is positioned so that a control component  108  for a surgical equipment device controller  110  is within image capture device  102 &#39;s field of view  112 . Image capture device  102  may be monoscopic or stereoscopic. 
         [0017]    As shown in  FIG. 1 , a medical person  114  (e.g., a surgeon or other medically trained person) operates control component  108 .  FIG. 1  shows person  114  operating control component  108  with a foot  114   a , which is illustrative of various ways to operate the control component (e.g., with a hand, finger, elbow, knee, etc.). Display  104  is positioned so that images  106  from image capture device  102  are output within person  114 &#39;s field of view  114   b.    
         [0018]      FIG. 1  also shows a surgical image capture device  116  positioned to capture and output moving visual images  118  of a surgical site  120 , which is within surgical image capture device  116 &#39;s field of view  116   a . For minimally invasive surgery implementations, surgical image capture device  116  is an endoscope (mono- or stereoscopic). Moving visual images  118  are received and output by display  104 . Consequently, in some aspects person  114  simultaneously views both images  106  and images  118 , as described in more detail below. Where such simultaneous viewing is desired, it should be appreciated that as an alternative to images  106  and images  118  both being received and output by display  104 , an additional display (not shown) positioned within person  114 &#39;s field of view  114   b  may be used. In such aspects, images  106  or images  118  are received and output by display  104 , and the images  106 , 118  not received and output by display  104  are received and output by the additional display (not shown) so that both images  106  and images  118  are output within person  114 &#39;s field of view  114   b.    
         [0019]    In some aspects, surgical equipment device controller  110  controls a feature of a device  122  at the surgical site  120 . For example, device  122  may be an electrosurgical instrument, and so surgical equipment device controller  110  is an electrosurgical energy generator that is coupled to route electrosurgical energy (e.g., for cutting, for cautery, and the like) to device  122 . In this example, control component  108  functions as an on/off control for the electrosurgical generator, so that person  114  can control application of electrosurgical energy at the surgical site. The electrosurgical energy function is merely illustrative, and various other surgical functions can be similarly controlled, such as suction, irrigation, other tissue fusing functions, tissue stapling functions, endoscopic camera control and illumination functions, robotic master/slave control functions, etc. Similarly, and as an alternative by dashed line, surgical equipment device controller  110  may control equipment  124  that does not directly affect the surgical site. Examples of such equipment  124  include operating room environment functions, patient data display functions, and any other function a surgeon may want to directly control via control component  108  while performing surgery. 
         [0020]    Since the operating room environment may include more than one control component  108 , various combinations of one or more image capture devices  102  may be used to capture images of the various control components. For instance, an image of two or more control components  108  may be captured in a single image capture device  102 &#39;s field of view. Alternatively, an image of one or more control components  108  may be captured in one image capture device  102 &#39;s field of view, and another image of one or more additional control components  108  may be captured in another image capture device  102 &#39;s field of view. And, an image of a control component  108  may be captured in two or more image capture devices&#39;  102  fields of view. Consequently, in implementations that include two or more image capture devices  102 , one, two, or more corresponding images can be output at display  104 . 
         [0021]      FIG. 1  further illustrates that person  114  may control device  122  at the surgical site in various ways, represented by dashed line. In one aspect, device  122  is a manually operated surgical instrument that person  114  directly operates. Since at least a portion of device  122  is within image capture device  116 &#39;s field of view  116   a , person  114  sees both device  122 &#39;s motion and the person  114 &#39;s physical relation to control component  108  in display  104 . Alternatively, in other aspects, person  114  operates a telesurgical robotic master control component (not shown), and device  122  is a surgical instrument robotically slaved to follow telesurgical robotic master control component&#39;s motion and other optional control inputs. 
         [0022]    In one aspect, additionally or as an alternative to image capture device  102 , display system  100  includes one or more sensors (e.g., infrared beam detectors) positioned to acquire information about the physical environment of control component  108 . Display system  100  uses information acquired by the one or more sensors to create a computer-generated representation showing control component  108  and its interaction(s) with its immediate surroundings. In one aspect, control component  108  is operated by person  114 &#39;s foot. Infrared beam detectors positioned on and/or around control component  108  detect the position of person  114 &#39;s foot as it moves towards and away from control component  108 . A computer-generated representation of this information is displayed to person  114  on display  104 . 
         [0023]      FIG. 2  is a plan view of a telesurgical robotic system  200  used to perform minimally invasive surgery. The telesurgical robotic system  200  includes a master control console  204  and a robotic slave device manipulator cart  210 .  FIG. 3  is a front elevation view of a master control console  204 . The master control console  204  includes a control console stereoscopic display  104   a . The control console stereoscopic display  104   a  is coupled to a stereoscopic endoscope (not shown; see  FIG. 4 , element  116   b ) that is configured to capture an image of the surgical site. The master control console  204  also includes a left telesurgical robotic master control component  108   a , a right telesurgical robotic master control component  108   b , a foot pedal assembly  108   d , a touch-sensitive display control interface  108   c , and a control component image capture device  102   a . The control component image capture device  102   a  may be monoscopic or stereoscopic. 
         [0024]    During use of the telesurgical robotic system  200 , an operator seated at the master control console  204  operates one or more telesurgical robotic master control component(s)  108   a , 108   b  to control the movement of one or more corresponding surgical instruments robotically slaved to follow the motion of the corresponding telesurgical robotic master control component. The operator&#39;s control of the robotically slaved surgical instruments (not shown; see  FIG. 4 , element  122   a ; see also  FIG. 1 , element  122 ) is facilitated by the video image of the surgical site (not shown; see  FIG. 6 , element  610 ) captured by the stereoscopic endoscope (not shown; see  FIG. 4 , element  116   b ; see also  FIG. 1 , element  116 ) that is received and output to the operator by the control console stereoscopic display  104   a.    
         [0025]      FIG. 4  is a front elevation view of a robotic slave device manipulator cart  210 . The robotic slave device manipulator cart  210  includes an endoscope manipulator  408 , a first instrument manipulator  410 , a second instrument manipulator  420 , and a third instrument manipulator  430 . The endoscope manipulator  408  is configured to allow a stereoscopic endoscope  116   b  (see also  FIG. 1 , element  116 ) to be releasably coupled. The first instrument manipulator  410 , the second instrument manipulator  420 , and the third instrument manipulator  430  are each configured to allow a robotically slaved surgical instrument  122   a  (see also  FIG. 1 , element  122 ) to be releasably coupled and operated. 
         [0026]    During use of telesurgical robotic system  200 , the stereoscopic endoscope  116   b  is configured to capture within its field of view the surgical site and the parts of the robotically slaved surgical instruments  122   a  manipulating tissue at the surgical site. The stereoscopic endoscope  116   b  is coupled to the control console stereoscopic display  104   a , thereby enabling the control console stereoscopic display  104   a  to receive the surgical site image captured by the stereoscopic endoscope  116   b  and output the image to the operator of telesurgical robotic system  200 . 
         [0027]    Referring to  FIG. 3 , as discussed above, the master control console  204  includes a touch-sensitive display control interface  108   c  and a foot pedal assembly  108   d . The touch-sensitive display control interface  108   c  and the foot pedal assembly  108   d  provide the operator of the telesurgical robotic system  200  with access to a variety of control components. Some of these control components may control surgical functions that directly affect the surgical site such as suction, irrigation, tissue fusing functions, tissue stapling functions, endoscopic camera control and illumination functions, robotic master/slave control functions, etc. Other control components may control functions that do not directly affect the surgical site, such as operating room environment functions, patient data display functions, or any other function a surgeon may want to directly control while performing surgery. 
         [0028]    During normal operation of the telesurgical robotic system  200 , the operator of the telesurgical robotic system  200  operates the left telesurgical robotic master control component  108   a  using the left hand and the right telesurgical robotic master control component  108   b  using the right hand to control corresponding robotically slaved instruments  122   a  located at the surgical site. The operator&#39;s actions are facilitated by the surgical site image captured by the stereoscopic endoscope  116   b , which is received by the control console stereoscopic display  104   a  and output to the operator. A consequence of the telesurgical robotic system  200  operator viewing the surgical site image on the stereoscopic display  104   a , however, is that the operator&#39;s field of view is generally limited to the field of view of the stereoscopic endoscope  116   b  capturing the surgical site image. If the operator of the telesurgical robotic system  200  seeks to operate a control component that his or her corresponding operating body part has not yet located (e.g., a hand-operated control component located on the touch-sensitive display control interface  108   c , a foot-operated control component that is not located immediately beneath the operating foot, etc.), he or she generally must look away from the surgical site image output on the control console stereoscopic display  104   a  to locate and actuate this control component. 
         [0029]    It should be appreciated that it is desirable to minimize the extent to which the operator of the telesurgical robotic system  200  is required to divert his attention from the surgical site when he seeks access to control components located on the touch-sensitive display control interface  108   c , on the foot pedal assembly  108   d , or at any other location accessible to the operator while the operator is seated at the master control console  204 . In accordance with some aspects, a control component image capture device  102   a  (mono- or stereoscopic) is coupled to the control console stereoscopic display  104   a  and is configured to capture within its field of view a control component accessible to the operator of the telesurgical robotic system  200 . Both the control component image captured by the control component image capture device  102   a  and the surgical site image captured by the stereoscopic endoscope  116   b  are received and output by the control console stereoscopic display  104   a . It should be appreciated that the control component image and the surgical image may be displayed concurrently on the stereoscopic display  104   a  or alternatively be displayed one at a time. 
         [0030]    In some aspects, the control component image capture device  102   a  is configured to capture within its field of view a control component and the control component&#39;s surroundings. In these aspects, the operator of the telesurgical robotic system  200  is able to view the physical relation between the operator&#39;s body part used to actuate the control component and the control component itself as the operator&#39;s body part approaches and/or moves away from the control component. As discussed earlier, control component image capture device  102   a  may be monoscopic or stereoscopic. If stereoscopic, the corresponding depth sensation from the image displayed to the operator helps the operator position the operating body part on the control component. As an example, the ability of the telesurgical robotic system  200  operator to view on the control console stereoscopic display  104   a  the relation between the operator&#39;s feet and various foot-operated control components located on the footpedal assembly  108   d  facilitates the operator&#39;s repositioning of his feet to actuate the various foot-operated control components during a surgical procedure without looking away from the control console stereoscopic display  104   a , and correspondingly the surgical site image displayed thereupon. 
         [0031]      FIGS. 5A ,  5 B, and  5 C are perspective views showing illustrative control components located on the foot pedal assembly  108   d , including a first right foot control component  108   e  and a second right foot control component  108   f . Also shown is part of the telesurgical robotic system  200  operator&#39;s right foot  114   c .  FIG. 5A  shows the operator&#39;s right foot  114   c  actuating the first right foot control component  108   e .  FIG. 5B  shows the operator&#39;s right foot  114   c  as it moves away from the first right foot control component  108   e  and approaches the second right control component  108   f .  FIG. 5C  shows the operator&#39;s right foot  114   c  actuating the second right foot control component  108   f . In this example, the first right foot control component  108   e  functions as an on/off control for the electrosurgical generator, controlling the application of electrosurgical energy delivered to the tip of a robotically slaved surgical instrument (not shown; see  FIGS. 1 and 4 ) coupled to the left telesurgical robotic master control component  108   a . The second right foot control component  108   f  also functions as an on/off control for the electrosurgical generator, controlling application of electrosurgical energy delivered to the tip of a robotically slaved surgical instrument coupled to the right telesurgical robotic master control component  108   b . As discussed earlier, it should be appreciated that in some aspects, the foot-operated control components control functions other than electrosurgical functions. As discussed above, these foot-operated control components may control surgical functions that directly affect the surgical site such as suction, irrigation, tissue fusing functions, tissue stapling functions, endoscopic camera control and illumination functions, robotic master/slave control functions, etc. Other foot-operated control components may control functions that do not directly affect the surgical site such as operating room environment functions, patient data display functions, or any other function a surgeon may want to directly control while performing surgery. It should also be appreciated that in some aspects, the control component image capture device  102   a  captures within its field of view control components operated by a body part not the operator&#39;s right foot  114   c . In this example, use of a right foot  114   c  to operate a control component is merely illustrative, as the body part used by the telesurgical robotic system  200  operator depends on the physical relation between the operator&#39;s body and the control component. In some aspects, the control component captured within the field of view of the control component image capture device  102   a  is positioned to be actuated by the operator&#39;s left foot, the operator&#39;s hand(s), the operator&#39;s elbow(s), etc. For example, the control component image capture device  102  may be configured to capture within its field of view an image containing one or more telesurgical robotic master control component(s)  108   a , 108   b  and the immediate surroundings, to be displayed by control console stereoscopic display  104   a.    
         [0032]    In accordance with some aspects,  FIG. 6  is an image as viewed from the control console stereoscopic display  104   a , and it includes a surgical site image  610  received from the stereoscopic endoscope  116   b  and a control component image  620  received from the control component image capture device  102   a . As shown in  FIG. 6 , the surgical site image  610  and the control component image  620  may be concurrently output by the control console stereoscopic display for viewing by the operator of the telesurgical robotic system. In this example, the control component image capture device  102   a  is configured to capture within its field of view the touch-sensitive display control interface  108   c  located on the master control console  204 . In this illustrative example display, the touch-sensitive display control interface  108   c  includes an endoscope illumination control component  108   g  to adjust the brightness settings on the endoscope illuminator that is the source of light  116   b  used to illuminate the surgical site. As shown, the control component image  620  includes both the endoscope illumination control component  108   g  and the surroundings of the endoscope illumination control component  108   g . As a result, the operator of telesurgical robotic system  200  is able to locate and operate the endoscope illumination control component  108   g  located on the touch-sensitive display control interface  108   c  by observing the control component image  620  while concurrently observing the effects of his or her actions on the surgical site in real time in the surgical site image  610 . It should be appreciated that use of the endoscope illumination control component  108   g  is merely illustrative of the various control components accessible through the touch-sensitive display control interface  108   c . As discussed above, the touch-sensitive display control interface  108   c  may provide control components for control of various surgical functions that directly affect the surgical site such as suction, irrigation, tissue fusing functions, tissue stapling functions, endoscopic camera control and illumination functions, robotic master/slave control functions, etc. Other control components accessible through the touch-sensitive display control interface  108   c  may control functions that do not directly affect the surgical site such as operating room environment functions, patient data display functions, or any other function a surgeon may want to directly control while performing surgery. It should also be appreciated that in this example, use of the operator&#39;s hand to operate a control component located on the touch-sensitive display control interface  108   c  is merely illustrative of the many body parts of the operator that can be used to operate the many control components accessible to the operator of the telesurgical robotic system  200 . The body part used by the telesurgical robotic system  200  operator depends on the physical relation between the operator&#39;s body and the control component. In some aspects, the control component captured within the field of view of the control component image capture device  102   a  is not the touch-sensitive display control interface  108   c , and is positioned to be actuated by the operator&#39;s foot, the operator&#39;s knee(s), the operator&#39;s elbow(s), etc.