Patent ID: 12185930

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is intended. In the following detailed description of the aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, as would be appreciated by one skilled in the art, embodiments of this disclosure may be practiced without these specific details. In other instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.

Any alterations and further modifications to the described devices, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and/or steps described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and/or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment may be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately. For simplicity, in some instances the same reference numbers are used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions of instruments in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom—e.g., roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom).

Referring now to the drawings,FIGS.1A,1B, and1Ctogether provide an overview of a medical system10that may be used in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures. The medical system10is located in a medical environment11. The medical environment11is depicted as an operating room inFIG.1A. In other embodiments, the medical environment11may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures may take place. In still other embodiments, the medical environment11may include an operating room and a control area located outside of the operating room.

In one or more embodiments, the medical system10may be a teleoperational medical system that is under the teleoperational control of a surgeon. In alternative embodiments, the medical system10may be under the partial control of a computer programmed to perform the medical procedure or sub-procedure. In still other alternative embodiments, the medical system10may be a fully automated medical system that is under the full control of a computer programmed to perform the medical procedure or sub-procedure with the medical system10. One example of the medical system10that may be used to implement the systems and techniques described in this disclosure is the da Vinci® Surgical System manufactured by Intuitive Surgical, Inc. of Sunnyvale, California.

As shown inFIG.1A, the medical system10generally includes an assembly12, which may be mounted to or positioned near an operating table O on which a patient P is positioned. The assembly12may be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, the assembly12may be a teleoperational assembly. The teleoperational assembly may be referred to as, for example, a teleoperational arm cart. A medical instrument system14and an endoscopic imaging system15are operably coupled to the assembly12. An operator input system16allows a surgeon S or other type of clinician to view images of or representing the surgical site and to control the operation of the medical instrument system14and/or the endoscopic imaging system15.

The medical instrument system14may comprise one or more medical instruments. In embodiments in which the medical instrument system14comprises a plurality of medical instruments, the plurality of medical instruments may include multiple of the same medical instrument and/or multiple different medical instruments. Similarly, the endoscopic imaging system15may comprise one or more endoscopes. In the case of a plurality of endoscopes, the plurality of endoscopes may include multiple of the same endoscope and/or multiple different endoscopes.

The operator input system16may be located at a surgeon's control console, which may be located in the same room as operating table O. In some embodiments, the surgeon S and the operator input system16may be located in a different room or a completely different building from the patient P. The operator input system16generally includes one or more control device(s) for controlling the medical instrument system14. The control device(s) may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and other types of input devices.

In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical instrument(s) of the medical instrument system14to provide the surgeon with telepresence, which is the perception that the control device(s) are integral with the instruments so that the surgeon has a strong sense of directly controlling instruments as if present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide the surgeon with telepresence. In some embodiments, the control device(s) are manual input devices that move with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaw end effectors, applying an electrical potential to an electrode, delivering a medicinal treatment, and actuating other types of instruments).

The assembly12supports and manipulates the medical instrument system14while the surgeon S views the surgical site through the operator input system16. An image of the surgical site may be obtained by the endoscopic imaging system15, which may be manipulated by the assembly12. The assembly12may comprise endoscopic imaging systems15and may similarly comprise multiple medical instrument systems14as well. The number of medical instrument systems14used at one time will generally depend on the diagnostic or surgical procedure to be performed and on space constraints within the operating room, among other factors. The assembly12may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, generally referred to as a set-up structure) and a manipulator. When the manipulator takes the form of a teleoperational manipulator, the assembly12is a teleoperational assembly. The assembly12includes a plurality of motors that drive inputs on the medical instrument system14. In an embodiment, these motors move in response to commands from a control system (e.g., control system20). The motors include drive systems which when coupled to the medical instrument system14may advance a medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems may move the distal end of said medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors may be used to actuate an articulable end effector of the medical instrument for grasping tissue in the jaws of a biopsy device or the like. Medical instruments of the medical instrument system14may include end effectors having a single working member such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or clip appliers.

The medical system10also includes a control system20. The control system20includes at least one memory24and at least one processor22for effecting control between the medical instrument system14, the operator input system16, and other auxiliary systems26which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and/or suction systems. A clinician may circulate within the medical environment11and may access, for example, the assembly12during a set up procedure or view a display of the auxiliary system26from the patient bedside.

Though depicted as being external to the assembly12inFIG.1A, the control system20may, in some embodiments, be contained wholly within the assembly12. The control system20also includes programmed instructions (e.g., stored on a non-transitory, computer-readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. While the control system20is shown as a single block in the simplified schematic ofFIG.1A, the control system20may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent the assembly12, another portion of the processing being performed at the operator input system16, and the like.

Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein, including teleoperational systems. In one embodiment, the control system20supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

The control system20is in communication with a database27which may store one or more clinician profiles, a list of patients and patient profiles, a list of procedures to be performed on said patients, a list of clinicians scheduled to perform said procedures, other information, or combinations thereof. A clinician profile may comprise information about a clinician, including how long the clinician has worked in the medical field, the level of education attained by the clinician, the level of experience the clinician has with the medical system10(or similar systems), or any combination thereof.

The database27may be stored in the memory24and may be dynamically updated. Additionally or alternatively, the database27may be stored on a device such as a server or a portable storage device that is accessible by the control system20via an internal network (e.g., a secured network of a medical facility or a teleoperational system provider) or an external network (e.g. the Internet). The database27may be distributed throughout two or more locations. For example, the database27may be present on multiple devices which may include the devices of different entities and/or a cloud server. Additionally or alternatively, the database27may be stored on a portable user-assigned device such as a computer, a mobile device, a smart phone, a laptop, an electronic badge, a tablet, a pager, and other similar user devices.

In some embodiments, control system20may include one or more servo controllers that receive force and/or torque feedback from the medical instrument system14. Responsive to the feedback, the servo controllers transmit signals to the operator input system16. The servo controller(s) may also transmit signals instructing assembly12to move the medical instrument system(s)14and/or endoscopic imaging system15which extend into an internal surgical site within the patient body via openings in the body. Any suitable conventional or specialized servo controller may be used. A servo controller may be separate from, or integrated with, assembly12. In some embodiments, the servo controller and assembly12are provided as part of a teleoperational arm cart positioned adjacent to the patient's body.

The control system20can be coupled with the endoscopic imaging system15and can include a processor to process captured images for subsequent display, such as to a surgeon on the surgeon's control console, or on another suitable display located locally and/or remotely. For example, where a stereoscopic endoscope is used, the control system20can process the captured images to present the surgeon with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope.

In alternative embodiments, the medical system10may include more than one assembly12and/or more than one operator input system16. The exact number of assemblies12will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems16may be collocated or they may be positioned in separate locations. Multiple operator input systems16allow more than one operator to control one or more assemblies12in various combinations. The medical system10may also be used to train and rehearse medical procedures.

FIG.1Bis a perspective view of one embodiment of an assembly12which may be referred to as a patient side cart, surgical cart, teleoperational arm cart, or surgical robot. The assembly12shown provides for the manipulation of three surgical tools30a,30b, and30c(e.g., medical instrument systems14) and an imaging device28(e.g., endoscopic imaging system15), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The imaging device may transmit signals over a cable56to the control system20. Manipulation is provided by teleoperative mechanisms having a number of joints. The imaging device28and the surgical tools30a-ccan be positioned and manipulated through incisions in the patient so that a kinematic remote center is maintained at the incision to minimize the size of the incision. Images of the surgical site can include images of the distal ends of the surgical tools30a-cwhen they are positioned within the field-of-view of the imaging device28.

The assembly12includes a drivable base58. The drivable base58is connected to a telescoping column57, which allows for adjustment of the height of arms54. The arms54may include a rotating joint55that both rotates and moves up and down. Each of the arms54may be connected to an orienting platform53. The arms54may be labeled to facilitate trouble shooting. For example, each of the arms54may be emblazoned with a different number, letter, symbol, other identifier, or combinations thereof. The orienting platform53may be capable of 360 degrees of rotation. The assembly12may also include a telescoping horizontal cantilever52for moving the orienting platform53in a horizontal direction.

In the present example, each of the arms54connects to a manipulator arm51. The manipulator arms51may connect directly to a medical instrument, e.g., one of the surgical tools30a-c. The manipulator arms51may be teleoperatable. In some examples, the arms54connecting to the orienting platform53may not be teleoperatable. Rather, such arms54may be positioned as desired before the surgeon S begins operation with the teleoperative components. Throughout a surgical procedure, medical instruments may be removed and replaced with other instruments such that instrument to arm associations may change during the procedure.

Endoscopic imaging systems (e.g., endoscopic imaging system15and imaging device28) may be provided in a variety of configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Digital image based endoscopes have a “chip on the tip” design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data. Endoscopic imaging systems may provide two- or three-dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic instruments employ stereo cameras to capture stereo images of the patient anatomy. An endoscopic instrument may be a fully sterilizable assembly with the endoscope cable, handle and shaft all rigidly coupled and hermetically sealed.

FIG.1Cis a perspective view of an embodiment of the operator input system16at the surgeon's control console. The operator input system16includes a left eye display32and a right eye display34for presenting the surgeon S with a coordinated stereo view of the surgical environment that enables depth perception. The left and right eye displays32,32may be components of a display system35. In other embodiments, the display system35may include one or more other types of displays.

The operator input system16further includes one or more input control devices36, which in turn cause the assembly12to manipulate one or more instruments of the endoscopic imaging system15and/or medical instrument system14. The input control devices36can provide the same degrees of freedom as their associated instruments to provide the surgeon S with telepresence, or the perception that the input control devices36are integral with said instruments so that the surgeon has a strong sense of directly controlling the instruments. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the medical instruments, e.g., surgical tools30a-c, or imaging device28, back to the surgeon's hands through the input control devices36. Input control devices37are foot pedals that receive input from a user's foot. Aspects of the operator input system16, the assembly12, and the auxiliary systems26may be adjustable and customizable to meet the physical needs, skill level, or preferences of the surgeon S.

During a medical procedure performed using the medical system10, the surgeon S or another clinician may need to manipulate tissue to retract the tissue, expose a target area, inspect a hidden region of tissue, or perform some other action. For example, the surgeon S may need to use the surgical tool30ato retract the tissue but the instrument may be partially or fully outside the field of view of the endoscopic imaging system15. Thus, the surgeon S may be unable to readily observe whether or not movement of the surgical tool30awould cause a collision with the surgical tool30b, the surgical tool30c, or one of the manipulator arms51. Further, the surgeon S may need to contort his or her wrist in order to take control of the surgical tool30a. Thus, it may be desirable to have methods and systems that improve the surgeon S's experience of manipulating tissue during a medical procedure.

The various embodiments described below provide methods and systems that allow the surgeon S to more easily and directly manipulate tissue within the field of view of the endoscopic imaging system15using an instrument (e.g. one of the surgical tools30a,30b, or30c). In one or more embodiments, the display system35may display a tissue control point (e.g. tissue control point200inFIG.2below) over an image representing the field of view of the endoscopic imaging system15. The surgeon S may manipulate the tissue control point200using the operator input system16and the control system20may process this input to thereby control operation of the instrument. The use of the tissue control point200simplifies the steps needed by the surgeon S to manipulate the tissue in a desired manner. Further, using the tissue control point200allows the surgeon S to control operation of the instrument to thereby manipulate tissue in the field of view of the endoscopic imaging system15even when the instrument is partially or fully out of the field of view.

FIG.2is a representational diagram of a tissue control point200that is collocated with the tissue202to be controlled using the tissue control point200. This diagram depicts a boundary203that represents the field of view of the endoscopic imaging system15. This field of view would be displayed as an image of the tissue202in a user interface on the display system35. The image may be an image of or an image representing the field of view of the endoscopic imaging system15.

The diagram further depicts an instrument204, an instrument206, and an instrument208, each of which is engaged with the tissue. Each of the instrument204, the instrument206, and the instrument208may be an example of one type of instrument that may be in a medical instrument system, such as medical instrument system14inFIG.1. For example, in one embodiment, the instrument204, the instrument206, and the instrument208may be surgical tools30a,30b, and30cinFIG.1B. The instrument204may be used to manipulate the tissue202during the medical procedure. The instrument204may be used to perform tasks such as, for example, retraction, countertraction, or a combination thereof. For example, the instrument204may be implemented as a retractor, a grasper, forceps, clamps, or some other type of auxiliary instrument.

The tissue control point200may be movable within the user interface displayed on the display system35inFIG.1Crelative to the image of the tissue and may be movable with a selected number of degrees of freedom. For example, the tissue control point200may be translatable, rotatable, or both. In one or more embodiment, the tissue control point200is implemented as a graphical element (e.g. a movable indicator) that indicates the number of degrees of freedom with which the tissue control point212may be moved. In some embodiments, the tissue control point212may be translated or rotated in any direction relative to the user interface210.

In one embodiment, the tissue control point200is represented using a four-headed arrow cursor. This four-headed arrow cursor indicates that the tissue control point200is movable in four translational directions (e.g. left, right, up, and down). The tissue control point200may be used to control the instrument204. As depicted, the instrument204may be located outside the boundary203representing the field of view of the endoscopic imaging system15. The tissue control point200enables the surgeon S or other clinician to control operation of the instrument204even when the instrument204is not visible in the field of view, and thereby not displayed in the user interface. An offset210is present between the tip212and the tissue control point200. The offset210is represented by a dotted-line that extends between a tip212of the instrument204and the tissue control point200.

FIG.3is the representation diagram of the tissue202fromFIG.7after the tissue202has been manipulated based on movement of the tissue control point200. As depicted, the tissue control point200has been translated upwards. Based on this movement, the control system20operates the instrument204to cause a corresponding movement of the instrument204, which thereby manipulates the tissue.

FIG.4is the representational diagram of the tissue control point200represented as a virtual object400(or virtual fixture) that is collocated with the tissue202to be controlled. InFIG.4, the virtual object400is depicted as a graphical line element. The proxy geometry402of the instrument206may be known to the control system20. The proxy geometry402may indicate the geometry of the jaws of the instrument206, the tip of the instrument206, or some other portion of the instrument206. In this embodiment, the proxy geometry402indicates the geometry of the jaws of the instrument206. A distance404is represented by a dotted-line that extends between the proxy geometry402and the virtual object400. When the proxy geometry402and virtual object400are not in contact, the distance404represents the closest distance between the proxy geometry402and the virtual object400. When the proxy geometry402and the virtual object400are in contact, the distance404represents a penetration depth, which may be used for rendering a restoring force.

FIG.5is a flowchart of a method500for manipulating tissue. The method500is illustrated inFIG.3as a set of operations or processes502through508and is described with continuing reference toFIGS.1A,1B,1C, and2. Not all of the illustrated processes502through508may be performed in all embodiments of method500. Additionally, one or more processes that are not expressly illustrated inFIG.3may be included before, after, in between, or as part of the processes502through508. In some embodiments, one or more of the processes502through508may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system) may cause the one or more processors to perform one or more of the processes.

At process502, the tissue control point200is displayed over the image of the tissue202in the user interface. The tissue control point200may be a graphical indicator that allows a user (e.g. the surgeon S) to control the instrument204engaged with the tissue202of the patient during the medical procedure to thereby manipulate the tissue202. The instrument204may be engaged with the tissue202by being physically associated with the tissue202. For example, the instrument may be touching the tissue202, grasping the tissue202, retracting the tissue202, applying a force to the tissue202, otherwise engaging the tissue202, or a combination thereof. The image of the tissue202may be provided by, for example, the endoscopic imaging system15inFIG.1A. In other words, the image may present the field of view of the endoscopic imaging system15.

The tissue control point200is displayed over the image at a selected location over the tissue202in the image. In this manner, the tissue control point200may be considered virtually collocated with the tissue202. The selected location for the tissue control point200may be established based on a position of the instrument204relative to the tissue. In some cases, the instrument204may be visible in the image. In other cases, the instrument204may not be visible in the image. In other words, the instrument204may be out of a field of view of the endoscopic imaging system15. In cases where the instrument204is out of the field of view, the selected location for the tissue control point200relative to the image may be offset from a position of the instrument204or tip212of the instrument204relative to the tissue202.

At process504, an input is received that moves the tissue control point200within the user interface. This input may be received through the input control device36and processed by the control system20. In some embodiments, the input device208may be a joystick. In other embodiments, the input control device36may include at least one of a touchscreen, a gesture tracking system, a gaze tracking system, a hand control device, a teleoperated device, a mouse, or some other type of input device such as described for input devices36,37.

Based on the received input, the control system20may move the tissue control point200. The tissue control point200may have a selected number of degrees of freedom. The number of degrees of freedom selected may be task-specific based on the types of corresponding movements that need to be provided for the instrument204. The human arm is considered to have seven degrees of freedom. In one embodiment, the number of degrees of freedom selected for the tissue control point200is less than the seven degrees of freedom provided by the human arm to thereby simplify the user interactions needed to achieve the desired movement of the instrument204.

In one embodiment, movement of the tissue control point200may include a left translation, a right translation, an upward translation, a downward translation, an inward translation, an outward translation, a rotation of the tissue control point, or a combination thereof. The translation of the tissue control point200depicted inFIG.3is an example of one type of movement of the tissue control point200based on the received input. In some embodiments, movement of the tissue control point200may include multiple translations in varying directions and/or the same direction. In other embodiments, the tissue control point200may be only translatable, only translatable along one axis, only rotatable, or limited in movement in some other manner. In one embodiment, when the instrument204has jaws that grasp the tissue202, movement of the tissue control point200may be restricted such that the tissue control point200cannot cause the jaws to open and release the tissue202.

At process506, the instrument204, which is physically associated with the tissue202, is operated based on the received input to thereby manipulate the tissue202. In one embodiment, operating the instrument204includes transforming the movement of the tissue control point200into a corresponding movement for the instrument204to thereby manipulate the tissue202. This transformation may take into account factors in addition to the movement of the tissue control point200.

For example, the transformation may include optimizing movement of the instrument204based on one or more secondary objectives. These secondary objectives may include optimizing a speed of movement, ensuring that the movement is within selected range of motion limits for the instrument204, avoiding collision with one or more other instruments or structures, creating sufficient working space between the instrument204and any neighboring instruments, avoiding selected areas or zones (e.g. keep-out zones), or a combination thereof. In one or more embodiments, the control system20determines the position of the instrument204and the manipulator arm (e.g. manipulator arm51) connected to the instrument relative to other instruments and manipulator arms. The control system20may compute a path of movement for the instrument204that both corresponds to the movement of the tissue control point200and prevents interaction of the different instruments and manipulator arms. The path of movement may include any number of translational movements, rotational movements, or combination thereof.

In some embodiments, the control system20may identify operational parameters for the instrument204. These operational parameters may include for example, a geometry, a minimum speed of movement, a maximum speed of movement, a range of motion, a number of degrees of freedom, other types of parameters, or a combination thereof for the instrument204. The control system20may compute a path of movement for the instrument204that takes into account these optional parameters. Further, the control system20may compute a path of movement for the instrument204that ensures that the instrument204does not enter selected areas or zones (e.g. keep-out zones).

In some embodiments, the control system20may impose limits on the amount of force the instrument204is allowed to exert. For example, the instrument204may be generally capable of exerting about 2 pounds of force at the tip212of the instrument204. The control system20, however, may limit the amount of force that can be exerted at the tip212of the instrument204to about 0.5 pounds of force.

In one or more embodiments, operating the instrument204at process506manipulates the tissue202by causing a corresponding movement of the tissue202that achieves both the intended movement of the instrument204as well as the secondary objectives. The corresponding movement of the tissue202may be, for example, a retraction of the tissue202, a translation of the tissue202, a twisting of the tissue202, a rotation of the tissue202, a deformation of the tissue202, or a combination thereof. In this manner, movement of the tissue control point200by the user through the input control device36may be transformed into a corresponding movement of the instrument204that results in the tissue202engaged with or near the instrument204being retracted, twisted, rotated, lifted, pushed down, pulled downwards, raised upwards, moved to the side, moved upwards, deformed, and/or otherwise manipulated.

In some embodiments, the control system20may use imaging data or sensor data to observe movement of the tissue202surrounding the tissue control point200and may update the one or more control laws used in controlling movement of the instrument204based on the movement of the tissue control point200to reduce errors in the observed movement of the tissue202. In other words, the control system20may use feedback in the form of imaging data or sensor data to reduce errors in the actual motion of the tissue202relative to the intended motion of the tissue202based on the movement of the tissue control point200.

At process508, which may be optional, a haptic feedback response is generated in response to operation of the instrument204. The haptic feedback response, which may be also referred to as a haptic communication or a kinesthetic response, may be a physical or mechanical stimulation through the application of forces, vibrations, motion, or a combination thereof to the user. The haptic feedback response may be generated based on a physical effect of the operation of the instrument204on the tissue202and may allow the user to receive information from the control system20in the form of a felt sensation on some part of the body. The haptic feedback response may, for example, allow the user to experience the stiffness, rigidity, or deformability of the tissue202. In some embodiments, the haptic feedback response may allow the user to experience traction and resistance of the tissue202to movement.

The haptic feedback response may be generated using a haptic device that generates tactile sensations that can be felt by the user. The haptic device may include at least one of a teleoperated device, a joystick, gloves, some other type of hand control device, some other type of tactile sensation generating device, or combination thereof. In some embodiments, the haptic device may be the input control device36. For example, the input control device36may reflect forces and torques generated from virtual-physical interactions through physical force. The virtual-physical interactions may be, for example, the encountering of constraints due to contact with virtual surfaces, virtual lines, virtual points, or a combination thereof.

FIG.6is an illustration of a method600for manipulating tissue. The method600is illustrated as a set of operations or processes602through608and is described with continuing reference toFIGS.1A,1B,1C, and2. Not all of the illustrated processes602through608may be performed in all embodiments of method600. Additionally, one or more processes that are not expressly illustrated inFIG.6may be included before, after, in between, or as part of the processes602through608. In some embodiments, one or more of the processes602through608may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system) may cause the one or more processors to perform one or more of the processes.

At process602, an initial location for the tissue control point200is virtually determined. Determining this initial location includes establishing a relationship between the tissue202to be controlled, the tissue control point200, and the instrument204. More specifically, determining this initial location includes establishing a relationship between the tissue202to be controlled, the tissue control point200, and a tip or end effector of the instrument204.

In one embodiment, the initial location for the tissue control point200may be determined or selected by the user. For example, the user may enter initial input that is used to determine the tissue control point200. The user may select the initial location using, for example, the tip212of the instrument204or some other teleoperated instrument or tool to contact a location on the surface of the tissue202and then engage a control. Engaging the control may include, for example, pressing a button on the input control device36. The initial location for the tissue control point may then be determined based on the location on the surface at which the tip212of the instrument204has engaged the tissue202. In some embodiments, the instrument204may be used to grasp the tissue202. The control system20may use the grasping force commanded to the instrument204for grasping the tissue202as a signal to establish and lock an offset between the tip212of the instrument204and the tissue control point200.

In response to the control being engaged, the control system20may then create and display the tissue control point200at a corresponding virtual location over the image. In other embodiments, the user may use the input control device36to manually move the tissue control point200in the user interface into the selected location on the image. In still other embodiments, the input control device36may track or detect the gaze of the user to determine where to position the tissue control point200. For example, the control system20may establish the tissue control point200at a location on the surface of the tissue202at which a gaze of the user is detected as being directed for a selected period of time. Gaze input from the left and right eyes of the user may be triangulated to produce a three-dimensional fixed location.

In one embodiment, the initial location or the tissue control point200may be determined by the control system20. For example, the control system20may generate a sparse or dense three-dimensional surface reconstruction of a surface of the tissue202based on imaging data received from an imaging system (e.g. a laser imaging system). The control system20may then identify the location on the surface of the tissue202connected to or that is engaged by the instrument204(e.g. the tip or end effector of the instrument204). The control system20may identify candidate points on the surface of the tissue202that are, for example, centrally located and most anterior in view and may then select the initial location for the tissue control point200based on the candidate points. The tissue control point200may be selected at a location that is visible in the field of view of the endoscopic imaging system15, is not occluded by other instruments, and is close or optimally centered with respect to controlling the instrument204.

In other embodiments, the three-dimensional surface reconstruction of the surface of the tissue202described above may be used in conjunction with the detected gaze from a single eye. For example, a vector for the gaze detected from the single eye may be intersected with the three-dimensional surface reconstruction to determine the location for the tissue control point200.

In some embodiments, the initial location of the tissue control point200may be computed from a color/depth segmentation of the image provided by the endoscopic imaging system15. For example, the control system20may segment a region of the image using color image segmentation or depth image segmentation. The control system20may then compute a centroid of the segmented region of the image as the initial location for the tissue control point200.

In other embodiments, a two-dimensional location may be determined for the tissue control point200within the image. The two-dimensional location may then be mapped to a three-dimensional location with respect to a field of view of the endoscopic imaging system15or other imaging device that provides the image. For example, stereoscopic images displayed in the left and right eye displays32,32may be processed to determine matching pixel locations in the left and right views. A depth may then be computed from the disparity between the left and right eye pixels and used to determine the three-dimensional location for the tissue control point200. Thus, the initial location for the tissue control point200may be determined with reference to a two-dimensional coordinate system or a three-dimensional coordinate system.

At process604, a selected mode is activated that locks a position of the instrument206with respect to a reference coordinate system. In one or more embodiments, the instrument206and the tissue control point200may be controlled using the same input control device36(e.g. the same joystick). Activating the selected mode configures the input control device36to ensure that the input received is used to control the tissue control point200and not the instrument206. When the selected mode is not activated, input received through the input control device36is used to control the instrument206and not the tissue control point200.

In other words, if a user has activated the selected mode, then the input control device36or some portion of the input control device36that is used to teleoperate the instrument206may be reconfigured such that input received through the input control device36or some portion of the input control device36may be used to control the tissue control point200rather than the instrument206. A user may then use the input control device36to move the tissue control point200, which in turn, causes corresponding operation of the instrument204.

At process606, input is received through the input control device36that moves the tissue control point200within the user interface. The translation of the tissue control point2020depicted inFIG.3is an example of the movement of the tissue control point200that occurs at process606. At process608, movement of the tissue control point200is transformed into a corresponding movement of the instrument204to thereby manipulate the tissue202.

FIG.7is an illustration of a method700for virtually collocating the tissue control point200with the tissue202to be controlled. The method700is illustrated as a set of operations or processes702through708and is described with continuing reference toFIGS.1A,1B,1C, and2. Not all of the illustrated processes702through708may be performed in all embodiments of method700. Additionally, one or more processes that are not expressly illustrated inFIG.7may be included before, after, in between, or as part of the processes702through708. In some embodiments, one or more of the processes702through708may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system) may cause the one or more processors to perform one or more of the processes.

At process702, a position for the instrument204and a position for an imaging device that provides the image of the tissue are computed. In one embodiment, at process702, the position for the instrument204may be a position of a tip212of the instrument204computed using kinematic equations for the manipulator controlling the instrument204. Similarly, the position for the imaging device (e.g. the endoscopic imaging system15) may be a position of the tip of the imaging device computed using kinematic equations for the manipulator controlling the imaging device. A manipulator may be implemented as, for example, the manipulator arm51of the assembly12shown inFIG.1B. The position for the tip212of instrument204may be represented by Trip and the position for the imaging device may be represented by Timd. Ttipand Timdmay be three-dimensional transformation matrices composed of three-dimensional position and three-dimensional rotation components.

At process704, a reference position for the tissue control point200is computed based on the position computed for the instrument204and an offset transformation. The offset transformation may be used to maintain an offset between the tip212of the instrument204and the tissue control point200to thereby allow the tissue control point200to become an extension of the kinematic chain of the manipulator arm51controlling the instrument204. Using the offset transformation allows the tissue control point200to be related to the instrument204in a reference coordinate system for the instrument204or the manipulator arm51controlling the instrument204. The reference position for the tissue control point200may be represented by TTCPand may be three-dimensional.

At process706, the reference position for the tissue control point200is transformed into an imaging device-based position based on the position computed for the imaging device. This imaging device-based position may be in an imaging device coordinate system (e.g. a coordinate system for the endoscopic imaging system15). The imaging device-based position may be in two-dimensions or three-dimensions. The imaging device-based position may be represented by TTCP_IM. At process708, the imaging device-based position is transformed into a display position. The display position may be in a display coordinate system for the user interface. The display position may be represented by TTCP_DC. In some embodiments, the display position may be or may be used to compute the location of the tissue control point200relative to the image displayed in the user interface. Thus, the tissue control point200may be related to the instrument204in a number of different relevant coordinate systems.

FIG.8is an illustration of a method800for generating a haptic feedback response based on movement of the tissue control point200. The method800is illustrated as a set of operations or processes802through824and is described with continuing reference toFIGS.1A,1B,1C, and2. Not all of the illustrated processes802through824may be performed in all embodiments of method800. Additionally, one or more processes that are not expressly illustrated inFIG.8may be included before, after, in between, or as part of the processes802through824. In some embodiments, one or more of the processes802through824may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system) may cause the one or more processors to perform one or more of the processes. In one or more embodiments, the processes802through824may be performed by the medical system10.

At process802, a position is computed for a tip of a first instrument204and a second instrument206. An input control device36may be used to control the second instrument206. In one embodiment, computing the positions at process802includes computing a position of the tips of the two instruments by computing, for example, the forward kinematics for the manipulator arms51connected to these instruments. In one or more embodiments, the input control device36includes, incorporates, or connects to a haptic device that provides haptic feedback. The position for the tip of the first instrument204may be represented by Ttip1and the position for the tip of the second instrument206may be represented by Ttip2. In one or more embodiments, both Ttip1and Ttip2may be three-dimensional positions computed in a manipulator coordinate system using kinematic equations.

At process804, a reference position for the tissue control point200is computed based on the computed position of the tip of the first instrument204and an offset transformation. The offset transformation may be used to maintain an offset between the tip of the instrument204and the tissue control point200to thereby allow the tissue control point200to become an extension of the kinematic chain of the manipulator arm51controlling the instrument204. The reference position for the tissue control point200may be represented by TTCPand may be three-dimensional.

At process806, the reference position for the tissue control point200and the position of the tip of the second instrument206are transformed with respect to a common coordinate system for the medical environment11in which the medical procedure is being performed. In one embodiment, the common coordinate system may be referred to as a world coordinate system. The world position for the tissue control point200within the world coordinate system may be represented as TTCP_WC, while the world position for the tip of the second input control device36within the world coordinate system may be represented as Ttip2_WC. The world position for the tissue control point200may be the world position for the virtual object400representing the tissue control point200.

The proxy geometry402of the second instrument206and the geometry of the virtual object400representing the tissue control point200may be known. The proxy geometry402may be moved such that the proxy geometry402comes into contact with or otherwise engages the virtual object400that represents the tissue control point200. The proxy geometry402may be used to nudge, prod, or otherwise manipulate the virtual object400. The forces exerted on the virtual object400by the proxy geometry402may affect movement of the first instrument204, thereby causing manipulation of the tissue202.

In other examples, the virtual object400may be directly grasped and manipulated using the second instrument206. This type of control provides a natural and direct way of affecting the retracting instrument pose without having to change, for example, a control mode of the telemanipulation interface. In this manner, the user may perceive that they are using the second instrument206to directly move the tissue202.

At process808, the distance404between the world position for the virtual object400and the world position for the proxy geometry402for the instrument206is computed. As depicted inFIG.4, the proxy geometry402may be the geometry of the jaws of the instrument206. At process810, a determination is made as to whether the distance404is less than zero. The distance404is less than zero when there is virtual penetration or interpenetration indicating virtual contact between the proxy geometry402and the virtual object400. When this virtual contact exists, operation of the second instrument206affects the virtual object400, which may, in turn, affect the first instrument204, thereby causing manipulation of the tissue202. When contact between the proxy geometry402and the virtual object400has been lost, the distance404is not less than zero. Without this virtual contact, operation of the second instrument206does not affect the virtual object400and thus, does not affect the first instrument204. When the distance404is less than zero, the distance404may be referred to as a penetration depth. When the distance404is not less than zero, the distance404represents the closest distance between the proxy geometry402and the virtual object400.

Referring again to the process808, if the distance404is not less than zero, the method800returns to the process802, as described above. But if the distance404is less than zero, then at process811, a virtual force/torque is computed. The virtual force/torque may be virtual force/torque of the proxy geometry402on the tissue control point200. Haptic rendering may be used to provide stable virtual contact with the tissue control point200and avoid a slip-through problem that happens when penetration depth exceeds the thickness of the virtual object400.

After process811has been performed, sub-method812and sub-method814are performed. Sub-method812includes processes816-820for manipulating the tissue202via interaction with the tissue control point200and sub-method814includes processes822-826for providing haptic feedback to the user when interacting with the tissue control point200.

In sub-method812, at process816, the computed virtual force/torque is applied to the virtual object400representing the tissue control point200. The computed virtual force/torque is applied as a reaction force that moves the virtual object400. As previously described, the virtual object400geometrically represents the tissue control point200and may be virtually collocated with the tissue202in the user interface. This virtual object400may also be referred to as a simulated tissue control point body or a simulated TCP body.

At process818, the virtual object400transform is updated. This transform may be the transformation that determines how movement of the virtual object400by the proxy geometry402will affect the first instrument204. At process820, the first instrument204is operated based on the updated virtual object transform. Operation of the first instrument204manipulates the tissue202. In particular, at process820, a command for the first instrument204may be generated based the updated virtual object transform and then applied to (e.g. sent to) the first instrument204. In this manner, the second instrument206may be operated to move the proxy geometry402and thereby engage and apply a force/torque on the virtual object400. The force/torque applied to the virtual object400may, in turn, cause movement of the first instrument204, which causes manipulation of the tissue202. The force/torque may be integrated by a mass/damper virtual model to compute the corresponding velocity and change in position.

At process822, the virtual force/torque is transformed with respect to the instrument/manipulator coordinate system. At process824, the virtual force/torque is then transformed from the instrument/manipulator coordinate system to an input coordinate system. The input coordinate system is for the input control device36. At process826, the force/torque is then applied to the input control device36. Applying the force/torque to the input control device36produces a haptic feedback response that may be experienced by the user. For example, the user may feel a physical response to pushing, prodding, nudging, or other motion of the tissue202caused by operation of the first instrument204based on virtual movement of the virtual object400by the proxy geometry402.

In this manner, the tissue control point200may support familiar physical interactions by representing the tissue control point200as both a visual and haptic virtual object402. Movement of the second instrument206may move the proxy geometry402so as to impart forces on the virtual object400representing the tissue control point200. The force applied to the virtual object400may, in turn, induce movement of the first instrument204. The determination regarding the distance404made at process810may ensure that the simulated movement of the virtual object400only occurs while the proxy geometry402is in contact with the virtual object400. This ensures that tissue manipulation is continuously controlled by the second instrument206and ceases upon loss of contact between the proxy geometry402and the virtual object400.

The method800described above provides a way in which the surgeon S or other clinician may control operation of the first instrument204without switching modes on the input control device36that is being used to control the second instrument206. Rather, operation of the second instrument206may be used to virtually contact and impart forces on the virtual object400, which then causes movement or some other type of operation of the first instrument204.

Thus, the embodiments described above provide a method and apparatus for manipulating tissue using the tissue control point200. The control system20establishes a relationship between the tissue control point200and the instrument204such that translational motion of the tissue control point200causes a corresponding movement of the instrument204that also optionally takes into account other degrees of freedom to achieve secondary objectives, such as avoiding collisions with neighboring instruments. The use of the tissue control point200simplifies the steps needed by a surgeon to manipulate the tissue in a desired manner. Further, using the tissue control point200enables the surgeon to control operation of the instrument204to manipulate tissue in the field of view of the endoscopic imaging system15even when the instrument204is partially or fully out of the field of view.

One or more elements in embodiments of the invention may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code segments may be downloaded via computer networks such as the Internet, Intranet, etc.

Note that the processes and displays presented may not inherently be related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear as elements in the claims. In addition, the embodiments of the invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.