Patent Description:
Surgical systems, such as minimally invasive, teleoperated robotic systems, offer patients many benefits, such as reduced trauma to the body, faster recovery, and a shorter hospital stay. Typically, in a surgical system, a surgeon uses a master tool manipulator, sometimes referred to as a master tool, to control movement of a surgical instrument, referred to as a slave surgical instrument.

In one teleoperated surgical system, a master tool grip of the master tool manipulator is specially designed to be both ergonomic and intuitive for controlling the slave surgical instrument. The surgeon holds the master tool grip in a particular way using his/her forefinger and thumb, so that targeting and grasping involves intuitive pointing and pinching motions.

To enable intuitive control of the slave surgical instrument, the master tool grip is ideally aligned in orientation with the slave surgical instrument tip in a view frame of a stereoscopic viewer in the surgical system. The motions of the slave surgical instrument tip ideally follow master tool motions via teleoperation and are consistent in both directions of motion as well as absolute orientation, i.e., the surgical instrument tip motion follows the motion of the master tool grip. If orientation alignment is not achieved between the master tool and the slave surgical instrument tip, the slave surgical instrument tip neither points in the same absolute direction nor rolls along the same axis as the surgeon is pointing with the master tool.

In one aspect, the master tool manipulator uses motors in a gimbal assembly to actively align the orientation axes of the master tool grip in surgeon eye-view coordinates with the associated slave surgical instrument tip in camera view coordinates. This alignment happens automatically before following is engaged between motion of the master tool and motion of the slave surgical instrument.

Specifically, before entering following on the prior teleoperated surgical system, the system tries to align the orientation of the slave surgical instrument tip in a camera frame with the orientation of the master tool grip in an eye-view frame, sometimes referred to as an eye frame. Typically, the teleoperated surgical system performs a master-slave alignment whenever the system transitions from a mode where the orientation alignment between the master tool grip and the slave surgical instrument tip may have been compromised (after a tool change, camera clutch, slave clutch, swapping of arms in a four arm system, etc.).

In the master-slave alignment process, a set of master wrist joint angles are calculated that cause the orientation of the master tool grip in the eye-view frame to match the orientation of the slave surgical instrument tip in the camera view frame, without changing the master tool grip x-y-z position. The master wrist joints are then commanded to match the calculated angles using the motors.

This should result in alignment of the orientations of the master tool grip and the slave surgical instrument tip. However, if a surgeon grasps the master tool grip too firmly while the motors are positioning the wrist joints in the master tool, there may be both displacement and orientation errors between the master tool grip and the slave surgical instrument tip.

After the wrist joints in the master tool grip are commanded to align the orientations of the master tool grip and the slave surgical instrument tip, the teleoperated surgical system again checks that the master and slave orientations match before allowing the user to enter following. If the orientations don't match, a warning message is displayed and the master-slave alignment is attempted again. This process is repeated until the alignment between the orientations of the master tool grip and the slave surgical instrument tip is within an acceptable tolerance, for example, the orientation misalignment between the master and the slave is smaller than ten degrees.

The orientation misalignment is limited to a small angular deviation so that when the master tool is rotated, the slave surgical tip is typically perceived by the surgeon as rolling in the same way. (See <FIG> and the related discussion below, which demonstrates that when the orientation misalignment is larger than the allowed small angular deviation, the slave surgical instrument tip rotates in an unexpected way when the master tool grip is rotated.

When the error in the alignment between the orientations of the master tool grip and the slave surgical instrument tip is smaller than the allowed small orientation misalignment, the master tool grip and the slave surgical instrument tip are considered aligned and the teleoperated surgical system enters following. The requirement to have the master tool grip and the slave surgical instrument tip aligned with such a small orientation misalignment often slows down the surgeon's entry into following.

<CIT> discloses patient-side surgeon interface providing a minimally invasive, teleoperated surgical system. The patient-side surgeon interface has components within the sterile surgical field of the surgery. The components allow a surgeon to control teleoperated slave surgical instruments from within the sterile surgical field. The patient-side surgeon interface permits a surgeon to be in the sterile surgical field adjacent a patient undergoing surgery. Controlling minimally invasive slave surgical instruments from within the sterile surgical field permits minimally invasive surgery combined with direct visualization by the surgeon. The proximity to the patient allows the surgeon to control a teleoperated slave surgical instrument in tandem with controlling manually controlled instruments such as a laparoscopic instrument. Also, the surgeon, from within the sterile surgical field, can use the patient-side surgeon interface to control at least one proxy visual in proctoring another surgeon.

<CIT> discloses a minimally-invasive surgical system including a slave surgical instrument having a slave surgical instrument tip and a master grip. The slave surgical instrument tip has an alignment in a common frame of reference and the master grip, which is coupled to the slave surgical instrument, has an alignment in the common frame of reference. An alignment error, in the common frame of reference, is a difference in alignment between the alignment of the slave surgical instrument tip and the alignment of the master grip. A ratcheting system (i) coupled to the master grip to receive the alignment of the master grip and (ii) coupled to the slave surgical instrument, to control motion of the slave by continuously reducing the alignment error, as the master grip moves, without autonomous motion of the slave surgical instrument tip and without autonomous motion of the master grip.

<CIT> discloses a medical robotic system having a robotic arm holding a medical device, and a control system for controlling movement of the arm according to operator manipulation of an input device. If the medical device is being commanded to a state exceeding a limitation by a threshold amount, then the control system disengages control of the medical device by the input device, servos the arm so that it remains in its current state, servos the input device so that it is set at a position such that a force being applied on the input device remains at its current level, requests the operator to lighten hold of the input device, sets a parameter associated with the input device upon detecting such lightened hold so that the medical device is commanded to a different state that does not exceed the limitation, and reengages control of the medical device by the input device.

<CIT> discloses a control device for controlling a robot system having at least one robot arm to which is attached a surgical instrument which has an end effector, wherein the control device comprises an image acquisition system which records the control specification of at least one hand, evaluates and converts it into corresponding control commands for one or more components of the robot system. It is proposed to provide a control unit which determines the orientation and / or the position and / or the degree of opening of the end effector of a surgical instrument as first size or first sizes, and furthermore the orientation and / or the position and / or the degree of opening of at least one hand are determined as second size or second sizes and in the case of a deviation of one or more a manual control of the end effector of a plurality of the first sizes of the respectively corresponding second size, and in the case of a match of one or more of the first sizes with the respectively corresponding second size, a gesture control releases, so that the surgical instrument can be operated by hand.

The present invention provides a surgical system and a non-transitory computer-readable medium as defined in the appended independent claims. Optional features are defined in the appended dependent claims.

A surgical system includes a slave surgical instrument having a slave surgical instrument tip and a master tool manipulator including a master tool grip. The master tool grip is coupled to the slave surgical instrument tip by a controller. The required orientation alignment between the slave surgical instrument tip and the master tool grip is relaxed so that the controller normally enters following where the motion of the slave surgical instrument tip follows the motion of the master tool trip irrespective of any orientation misalignment between the master tool grip and the slave surgical instrument tip.

Thus, the prior art multiple repetitions of the master-slave alignment process to achieve a small orientation misalignment are either eliminated or significantly reduced. Further, despite any swooping orientation misalignment between the master tool grip and the slave surgical instrument tip just before entering following, the motion of the slave surgical instrument tip intuitively follows the motion of the master tool grip and there is no unexpected motion associated with rotation of the master tool grip during following.

In one aspect, the controller is configured to receive a first command from the master tool manipulator and to send a second command including a desired orientation of the slave surgical instrument tip to the slave surgical instrument. The controller is further configured to generate the desired orientation of the slave surgical instrument tip using an orientation of the master tool grip. The orientation of the master tool grip is included in the first command.

The desired orientation of the slave surgical instrument tip generated by the controller preserves a same perceived rotation between the master tool grip and the slave surgical instrument tip even when there is a swooping orientation misalignment between the master tool grip and slave surgical instrument tip. Here, a same perceived rotation means that if the rotation of the master tool grip is perceived as being about a body-fixed axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed axis of the slave surgical instrument tip. For example, if the rotation of the master tool grip is perceived as being about the body-fixed x-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed x-axis of the slave surgical instrument tip, or if the rotation of the master tool grip is perceived as being about the body-fixed y-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed y-axis of the slave surgical instrument tip, or if the rotation of the master tool grip is perceived as being about the body-fixed z-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed z-axis of the slave surgical instrument tip.

This is in contrast to a prior system where rotation of the master about a space-fixed axis in the eye frame results in slave rotation about the same space-fixed axis in the camera frame. When the orientation offset between the master tool grip and the slave surgical instrument tip is permitted to be large when following is entered, a rotation of the master grip about its pointing direction can result in a circular slave surgical instrument tip motion about the master pointing direction projected onto the slave view frame Thus, the master tool grip and the slave surgical instrument tip, in the prior system, did not have the same perceived motion for swooping orientation misalignments.

In one aspect, the controller includes a memory and an orientation control module. The memory is configured to store a base orientation of the slave surgical instrument tip in a camera frame and to store a base orientation of the eye frame in the master frame. The controller also is configured to receive a first command from the master tool manipulator. The first command from the master tool manipulator includes a current orientation of the master tool grip in the eye frame. The controller further is configured to send to the slave surgical instrument a second command including the desired orientation of the slave surgical instrument.

It should be appreciated that the controller can be made up of one unit, or multiple different units. When the controller is divided up among different units, the units may be centralized in one location or distributed across the surgical system.

The orientation control module is configured to receive the current orientation of the master tool grip in the eye frame and to generate the desired orientation of the slave surgical instrument. The orientation control module also is configured to retrieve the stored base orientation of the master tool grip and the stored base orientation of the slave surgical instrument tip.

The orientation control module is further configured to generate the desired orientation of the slave surgical instrument tip in the camera frame based on the stored base orientation of the slave surgical instrument tip in the camera frame and a relative rotation matrix. The relative rotation matrix represents a relative rotation of the eye frame in the master frame. The orientation control module compensates for any swooping orientation misalignment between the master tool grip and the slave surgical instrument tip in generating the desired orientation of the slave surgical instrument. Thus, the motion of the slave surgical instrument tip intuitively follows the motion of the master tool grip without any unexpected motion.

Herein, a swooping orientation misalignment is an orientation misalignment which in a prior system would result in the tip of the surgical instrument moving in a perceivable circle about an axis of the surgical instrument tip when the master tool grip was rotated about a corresponding axis of the master tool grip. The prior system is characterized by defining the desired orientation of the slave surgical instrument tip as a product of the current orientation of the master tool grip, a base orientation of the slave surgical instrument tip in the camera reference frame, and a transpose of the base orientation of the master tool grip in the eye frame.

In one aspect, the relative rotation matrix is a combination of the current orientation of the master tool grip in the eye frame and the stored orientation of the eye frame in the master frame.

The controller also includes a master-slave alignment module. The master-slave alignment module is configured to send a command to the master tool to move the master tool grip to align the orientation of the master tool grip in the eye frame with the orientation of the slave surgical instrument tip in the camera frame. Also, after the master tool moves the master tool grip, the master-slave alignment module is configured to determine an orientation alignment error between the orientation of the master tool grip in the eye frame and the orientation of the slave surgical instrument tip in the camera frame. Further, the master-slave alignment module is configured to determine whether the orientation alignment error is smaller than or equal to a maximum permitted swooping orientation misalignment. In one aspect, the maximum permitted swooping orientation misalignment is limited so there is no confusion about which axis a rotation is about, e.g., so that pitch and yaw are not confused. As an example, the maximum permitted swooping orientation misalignment is fifty degrees in one aspect.

In one aspect, the master-slave alignment module is further configured to repeat the sending the command, determining the orientation alignment error, and determining whether the orientation alignment error is smaller than a maximum permitted swooping orientation misalignment only if the orientation alignment error is larger than the maximum permitted swooping orientation alignment error. Here, the maximum permitted swooping orientation alignment error is a different way of saying the maximum permitted swooping orientation misalignment.

In one aspect, the controller also includes a storage module coupled to the master-slave alignment module. The storage module is configured to receive the orientation of the master tool grip in the eye frame and the orientation of the slave surgical instrument tip in the camera frame after the master tool moves the master tool grip if the orientation alignment error is smaller than a maximum large alignment error. The storage module is also configured to store, in a memory, the orientation of the master tool grip in the eye frame as a base orientation of the eye frame in the master frame and to store the orientation of the slave surgical instrument tip in the camera reference as a base orientation of the slave surgical instrument tip in the camera reference.

In one aspect, the surgical system includes an endoscope having a distal end. The camera frame is defined with respect to the distal end of the endoscope. The surgical system also includes a surgeon's console including a viewer and the master tool manipulator.

The method for controlling alignment of a slave surgical instrument tip of a slave surgical instrument in a surgical system with alignment of a master tool grip of a master tool manipulator in the surgical system includes generating, by a controller, a desired orientation of the slave surgical instrument tip using an orientation in an eye frame, the desired orientation of the slave surgical instrument tip preserving a same perceived rotation between the master tool grip and the slave surgical instrument tip with a swooping orientation misalignment between the master tool grip and slave surgical instrument tip.

In one aspect, the generating includes generating, by the controller, a desired orientation of the slave surgical instrument tip in a camera frame based on a rotation offset and a current orientation of the master tool grip in an eye frame, the rotation offset being a combination of a stored orientation of the slave surgical instrument tip in the camera frame and a stored orientation of the eye frame in the master frame. The method also includes sending, by the controller to the slave surgical instrument, a command including the desired orientation of the slave surgical instrument tip, and moving, by the slave surgical instrument in response to the command, the slave surgical instrument tip to the desired orientation.

In one aspect, prior to the generating, the method sends a command to the master tool manipulator to move the master tool grip to align the orientation of the master tool grip in the eye frame with the orientation of the slave surgical instrument tip in the camera frame. After the master tool manipulator moves the master tool grip, the method determines an orientation alignment error between the orientation of the master tool grip in the eye frame and the orientation of the slave surgical instrument tip in the camera frame. The method then determines whether the orientation alignment error is smaller than or equal to a maximum permitted swooping orientation misalignment.

In another aspect, a computer-assisted medical system includes a master tool grip, a slave surgical instrument tip, a controller, and a memory. A roll axis of a master frame is defined along a length of the master tool grip. The master frame is associated with an orientation of the master tool grip, and the master frame is a body-fixed master frame.

A roll axis of a slave frame is defined along a length of the slave surgical instrument tip. The slave frame is associated with an orientation of the slave surgical instrument tip, and the slave frame is a body-fixed slave frame;.

The memory contains non-transitory instructions that direct the controller to perform acts including receiving an input corresponding to roll of the master tool grip,, generating, in response to the input a command to roll the slave surgical instrument tip, the command being to rotate the slave surgical instrument tip around the roll axis of the body-fixed slave frame without rotating the slave surgical instrument tip around any axis orthogonal to the roll axis of the body-fixed slave frame, and in response to the command, rolling the slave surgical instrument tip around the roll axis of the body-fixed slave frame.

A method for a state of a computer-assisted medical system in which an orientation misalignment exists between the roll axis of a master frame in an eye frame and a roll axis of a slave frame in a camera frame, the master frame being associated with a master tool grip of a master manipulator, the slave frame being associated with a slave surgical instrument tip, the roll axis of the master tool grip being defined along a length of the master tool grip, the roll axis of the slave surgical instrument tip being defined along a length of the slave surgical instrument tip, the method includes receiving an input corresponding to roll of the master tool grip around a roll axis of the master frame, the master frame being a body-fixed master frame. Generating, in response to the input, a command to roll the slave surgical instrument tip, the command being to rotate the slave surgical instrument tip around a roll axis of a body-fixed slave frame without rotating the slave surgical instrument tip around any axis orthogonal to the roll axis of the body-fixed slave frame, and in response to the command, rolling the slave surgical instrument tip around the roll axis of the body-fixed slave frame.

In the drawings, the first digit of a figure number indicates the figure in which the element with that figure number first appears.

Aspects of this invention anticipate that the prior art master-slave alignment process for a master tool grip and a slave surgical instrument tip in a teleoperated surgical system <NUM>, sometimes referred to a system <NUM>, may result in a swooping orientation alignment error, i.e., a large misalignment, between the master tool grip and the slave surgical instrument tip. To avoid the prior art delays associated with repeating the master-slave alignment process until the orientations of the master tool grip and the slave surgical instrument tip are aligned, controller <NUM> relaxes the criterion for an acceptable alignment error between the orientations of the master tool grip and the slave surgical instrument tip before following can be entered, i.e., a swooping orientation alignment error is acceptable. The relaxed criterion for an acceptable alignment error between the orientations of the master tool grip and the slave surgical instrument tip before following can be entered reduces the number of required iterations of the master-slave alignment process before following can be entered compared to the prior art process that required a more precise alignment between the orientations of the master tool grip and the slave surgical instrument tip.

Herein, a swooping orientation misalignment is an orientation misalignment where in the above described prior system, a rotation of the master tool grip about its pointing direction would result in a circular slave surgical instrument tip rotation about the master tool grip pointing direction projected on the slave view frame. The prior system is characterized by defining the desired orientation of the slave surgical instrument tip as a product of the current orientation of the master tool grip, a base orientation of the slave surgical instrument tip in the camera reference frame, and a transpose of the base orientation of the master tool grip in the eye frame.

Since the prior system limited the orientation error between the slave surgical instrument tip and the master tool grip to a small orientation error, the circular motion of the slave surgical instrument tip was not perceivable to the user, and so for small orientation errors in the prior system, the slave surgical instrument tip and the master tool grip had the same perceived motion. However, when the requirement for the small orientation error is relaxed, the perceived motion of the slave surgical instrument tip and the master tool grip is no longer the same.

As explained more completely below, controller <NUM> stores base orientations of the master tool grip and the slave surgical instrument tip upon completion of the master-slave alignment process, and enters following. As used herein "following" is a state of teleoperated surgical system <NUM> where moving a master tool grip of a master tool in surgeon's console <NUM> by a user of system <NUM> results in controller <NUM> sending a command to a surgical instrument <NUM> to move the slave surgical instrument tip in the same way as the master tool grip moved. Thus, during following, the movement of the slave surgical instrument tip follows the movement of the master tool grip irrespective of the orientation misalignment between the master tool grip and the slave surgical instrument tip when following was started.

During following each commanded movement of the slave surgical instrument tip from the master tool manipulator is rotated by controller <NUM> in a particular way, as described more completely below, to compensate for any alignment error between the master tool grip and the slave surgical instrument tip before following was entered. The result of this rotation is that the motion of the slave surgical instrument tip intuitively corresponds to the motion of the master tool grip despite the initial orientation misalignment between the master tool grip and the slave surgical instrument tip. The motion of the slave surgical instrument tip is what expected by the user based on the movement of the master tool grip because the motion of the master tool grip perceived by a user of system <NUM> is the same as the motion of the slave surgical instrument perceived by the user, i.e., the master tool grip and the slave surgical instrument tip have the same perceived motion.

This is in contrast to prior systems where in some situations, as described more completely below with respect to <FIG>, a large initial misalignment resulted in unexpected motion of the surgical instrument tip when following the motion of the master tool grip. Specifically, the perceived motion of the master tool grip and the slave surgical instrument tip were different, because rotation of the master tool grip about a space-fixed axis in the eye frame resulted in slave surgical instrument tip rotation about the same space-fixed axis in the camera frame. When the orientation offset between the master tool grip and the slave surgical instrument tip is permitted to be large when following is entered, a rotation of the master tool grip about its pointing direction can result in a circular slave surgical instrument tip motion about the master pointing direction projected onto the slave view frame. See <FIG> and the accompanying description below.

More specifically, in one aspect, controller <NUM> (<FIG>) is configured to receive a first command <NUM> from the master tool manipulator (not shown, but see master tool manipulator <NUM> (<FIG>)), and to send a second command <NUM> including a desired orientation of the slave surgical instrument tip to slave surgical instrument <NUM>. Controller <NUM> is configured to generate the desired orientation of the slave surgical instrument tip using an orientation of the master tool grip. The orientation of the master tool grip is included in first command <NUM>.

The desired orientation of the slave surgical instrument tip generated by controller <NUM> preserves a same perceived rotation between the master tool grip and the slave surgical instrument tip even when there is a swooping orientation misalignment between the master tool grip and slave surgical instrument. Here, a same perceived rotation means that if the rotation of the master tool grip is perceived as being about a body-fixed axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed axis of the slave surgical instrument tip. For example, if the rotation of the master tool grip is perceived as being about the body-fixed x-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed x-axis of the slave surgical instrument tip, or if the rotation of the master tool grip is perceived as being about the body-fixed y-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed y-axis of the slave surgical instrument tip, or if the rotation of the master tool grip is perceived as being about the body-fixed z-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed z-axis of the slave surgical instrument tip.

As just explained, this is in contrast to a prior system where when the orientation offset is permitted to be large, a rotation of the master tool grip about its pointing direction can result in a circular slave surgical instrument tip motion about the master pointing direction projected onto the slave view frame. Thus, the master tool grip and the slave surgical instrument tip, in the prior system, did not have the same perceived motion for swooping orientation errors.

In surgical system <NUM>, a user, typically a surgeon, sits at a console <NUM> and grasps a master tool grip (not shown) between the thumb and forefinger so that targeting and grasping involves intuitive pointing and pinching motions. The master tool grip is part of a master tool manipulator, sometimes called a master tool. The motion of the master tool grip is used by controller <NUM>, as described more completely below, to move a tip of slave surgical instrument <NUM>, e.g., move an end effector of a surgical instrument.

Console <NUM> (<FIG> and <FIG>) includes a master display, sometimes referred to as a viewer, which displays at least a stereoscopic image <NUM> (<FIG>) of a surgical site <NUM> of patient <NUM>. Stereoscopic image <NUM> typically includes an image <NUM> of surgical site <NUM>, an image <NUM> of a part of surgical instrument <NUM>, and an image 212T of a tip of slave surgical instrument <NUM>. Console <NUM> also includes one or more foot pedals (not shown).

Console <NUM> (<FIG>) is connected to a controller <NUM> that is turn is connected to a cart <NUM>, which supports a plurality of robotic arms that includes robotic arm <NUM>. Slave surgical instrument <NUM> is held and positioned by robotic arm <NUM>. While it is not shown in <FIG>, an endoscope, held by another of the robotic arms, is typically used to provide stereoscopic image <NUM>.

The surgeon sits comfortably and looks into the master display on console <NUM> throughout surgery. The surgeon performs a medical procedure by manipulating at least master tool grip <NUM> (<FIG>). Master tool grip <NUM> includes two levers <NUM>, <NUM>, sometimes called pinchers, which the surgeon typically grasps between the thumb and forefinger. The master tool provides a master movement command <NUM> including at least a current orientation of master tool grip <NUM>. Master movement command <NUM> is sometimes referred to as a first command.

In response to master movement command <NUM> from the master tool, controller <NUM>, e.g., teleoperation servo controller <NUM> of controller <NUM>, sends a slave movement command <NUM> to slave surgical instrument <NUM>. In response to slave movement command <NUM>, slave surgical instrument <NUM> positions a tip of slave surgical instrument <NUM> as directed in command <NUM>. Slave movement command <NUM> is sometimes referred to as a second command.

Typically, console <NUM> includes at least two master tools and each master tool controls a different surgical instrument. Herein a single master tool with master tool grip <NUM> is considered. In view of this description, the aspects of this invention can be implemented for any desired number of master tools.

The master display is positioned in console <NUM> (<FIG>) near the surgeon's hands so that image <NUM> (<FIG>), which is seen in the master display, is oriented so that the surgeon feels that she or he is actually looking directly down onto surgical site <NUM>. Image <NUM> of tool <NUM> appears to be located substantially where the surgeon's hands are located and oriented substantially as the surgeon would expect tool <NUM> to be based on the position of her/his hand. However, typically, the surgeon cannot see the position or orientation of master tool grip <NUM> while viewing image <NUM>.

The real-time image from the endoscope is projected into perspective image <NUM> such that the surgeon can manipulate a surgical instrument end effector of tool <NUM>, through its associated master tool grip <NUM>, as if viewing the workspace in substantially true presence. By true presence, it is meant that the presentation of an image is a true perspective image simulating the viewpoint of an operator that is physically manipulating the surgical instrument. Thus, controller <NUM> transforms the coordinates of surgical instrument <NUM> to a perceived position so that the perspective image is the image that the surgeon would see if the endoscope were looking directly at surgical tool <NUM> from the surgeon's eye-level during an open cavity procedure.

Controller <NUM> performs various functions in system <NUM>. Controller <NUM> receives the images from an endoscope and generates the stereoscopic image that the surgeon sees. Controller <NUM> uses teleoperation servo controller <NUM> to transfer the mechanical motion of master tool grip <NUM> to an associated slave surgical instrument through control commands <NUM> so that the surgeon can effectively manipulate slave surgical instrument <NUM>. Controller <NUM> and teleoperation controller <NUM> are similar to prior systems with the exception of the aspects described more completely below.

Although described as a controller <NUM>, it is to be appreciated that controller <NUM> may be implemented in practice by any number of modules and each module may include any combination of components. Each module and each component may include hardware, software that is executed on a processor, and firmware, or any combination of the three. Also, the functions and acts of controller <NUM>, as described herein, may be performed by one module, or divided up among different modules or even among different components of a module. When divided up among different modules or components, the modules or components may be centralized in one location or distributed across system <NUM> for distributed processing purposes. Thus, controller <NUM> should not be interpreted as requiring a single physical entity as in some aspects controller <NUM> is distributed across system <NUM>.

The number of surgical instruments used at one time, and consequently, the number of surgical instruments used in system <NUM> generally depends on the medical procedure being performed and the space constraints within the operating room, among other factors. If it is necessary to change one or more of the surgical instruments being used during a procedure, an assistant may remove the surgical instrument, and replace the surgical instrument with a different surgical instrument.

When there are only two master tools in system <NUM>, and when the surgeon wants to control movement of a slave surgical instrument different from the two slave surgical instrument coupled to the two master tools, the surgeon may lock one or both of the two slave surgical instruments in place. The surgeon then associates one or both of the master tools with other slave surgical instruments held by other of the robotic arms and then controller <NUM> becomes active with respect to those surgical instruments.

To facilitate the discussion of the acts performed by controller <NUM>, various frames are used. A frame is a Cartesian coordinate system that includes three spatial axes, e.g., x-, y-, and z-axes, and three orientations, e.g., pitch, yaw, and roll. Each of the orientations is a rotation about one of the spatial axes.

A first frame is referred to as an eye frame. In one aspect, the origin of the eye frame is chosen to correspond with a position where the surgeon's eye is normally located when he or she is viewing surgical site <NUM> in surgeon's console <NUM>. In master movement command <NUM>, the position and orientation of master tool grip <NUM> are specified in the eye frame, in one aspect. Techniques for determining the position and orientation of master tool grip <NUM> in the eye frame are known. See for example, U. Patent No. <CIT>, and disclosing "Camera Referenced Control In a Minimally Invasive Surgical Apparatus").

A second frame is referred to as a master frame. In one aspect, the origin of the master frame is chosen as a point on master tool grip <NUM>.

A third frame is referred to as a camera frame. In one aspect, the origin of the camera frame is chosen to correspond with a position on the distal end of the endoscope. In slave movement command <NUM>, a desired orientation of the salve surgical instrument tip is specified in the camera frame, in one aspect. Techniques for determining the position and orientation of slave surgical instrument tip in the camera frame are known. See for example, U. Patent No. <CIT>.

A fourth frame is referred to as a slave frame. In one aspect, the origin of the slave frame is chosen as a point on the slave surgical instrument tip.

<FIG> is a process flow diagram of a following process that uses relative rotation with the alignment of the master tool and the alignment of the slave surgical instrument tip immediately before the following process is started. The following process is implemented by a following module <NUM> in controller <NUM>.

In this following process, the motion of the slave surgical instrument tip follows the motion of the master tool without any unexpected motion of the slave surgical instrument tip. As described more completely below, a prior following process that also used relative rotation resulted in unexpected rotation of the slave surgical instrument tip when the master tool and the slave surgical instrument tip were misaligned on entering the following process.

In this aspect, a FOLLOWING ENABLED check process <NUM> (<FIG>) determines whether system <NUM> has enabled following between master tool grip <NUM> and the slave surgical instrument tip. In one aspect, FOLLOWING ENABLED check process <NUM> is implemented by following module <NUM> (<FIG> and <FIG>). Following module <NUM> is included in controller <NUM>, and so controller <NUM> determines whether following is enabled. Similarly, each of the other processes and acts described with respect to the following session of <FIG> is performed by teleoperation servo controller <NUM> of controller <NUM>, and so controller <NUM> can be said to perform the processes and acts.

If following is enabled, FOLLOWING ENABLED check process <NUM> passes each command <NUM> from the master tool manipulator to GENERATE NEW SLAVE ORIENATION process <NUM>, and otherwise takes no action. FOLLOWING ENABLED check process <NUM> should not be interpreted as requiring controller <NUM> to continually poll to determine whether following is enabled. FOLLOWING ENABLED check process <NUM> is illustrative only that following cannot be initiated until following is enabled. The determination of whether following is enabled could be done for example, by an event handler which launches the following process, when a following enabled event is received.

GENERATE NEW SLAVE ORIENTATION process <NUM> uses the information in the master tool command and stored orientations <NUM>, e.g., stored orientations mRe' and cRs', to generate a desired orientation cRs_des of the slave surgical instrument tip in a camera frame. Stored orientations mRe' and cRs' are referred as base orientations. GENERATE NEW SLAVE ORIENTATION process <NUM> is implemented by orientation control module <NUM> that is included in controller <NUM>.

Master tool command <NUM> includes an orientation eRm of master tool grip <NUM> in an eye frame. Orientation eRm is sometimes referred to as a current orientation of the master tool grip in the eye frame.

Orientations mRe' and cRs' are stored in memory <NUM> before the following process, sometimes referred to as a following session, is started. Herein, a prime notation on an orientation reference numeral signifies that the orientation is a snapshot of the corresponding orientation state the moment before entering following, and, in one aspect, the stored orientation is a fixed quantity that does not change in the same following session.

Stored orientation mRe' is the last orientation of the eye frame in a master frame before the current following session was started. In this aspect, instead of holding the eye frame stationary and considering the movement of the master tool grip as in prior systems, the master tool grip is assumed to remain stationary and the eye frame is moving. This is an alternative technique for representing the motion of the master tool grip.

Stored orientation cRs' is the last orientation of the slave surgical instrument tip in the camera frame before the current following session was started. Orientations mRe' and cRs' are stored before a following session is started, and so are referred to as base orientations, because they are the orientation on which changes in orientation during following session are based, i.e., the new orientation of the slave surgical instrument tip is determined relative to the stored base orientation cRs' of the slave surgical instrument tip.

GENERATE NEW SLAVE ORIENTATION process <NUM> generates desired orientation cRs_des of the slave surgical instrument tip in the camera frame based on a rotation offset Roffset and current orientation eRm of the master tool grip in the eye frame, e.g., the desired orientation cRs_des of the slave surgical instrument tip in the camera frame is defined as: <MAT>.

Rotation offset Roffset is a combination of the stored orientation cRs' of the slave surgical instrument tip in the camera frame and the stored orientation mRe' of the eye frame in the master frame. In one aspect,
As discussed more completely further below, <MAT> Thus, <MAT>.

As previously stated, the last expression desired orientation cRs_des demonstrates that desired orientation eRs_des of the slave surgical instrument tip in the camera frame is a combination of the two stored orientations and the current orientation of the master tool grip in the eye frame. It is further demonstrated below that (mRe' * eRm) is a transpose of a relative rotation matrix R2. Thus, the desired orientation cRs_des of the slave surgical instrument tip in the camera frame is a product of the stored orientation cRs' of the slave surgical instrument tip in the camera frame and a transpose of relative rotation matrix R2. Relative rotation matrix R2 represents a relative rotation of eye frame <NUM> in master frame <NUM>.

Determining cRs_des of the slave surgical instrument tip in the camera frame using the product of the stored orientation cRs' of the slave surgical instrument tip in the camera frame and a transpose of relative rotation matrix R2 compensates for any orientation alignment error between master tool grip <NUM> and the surgical instrument tool tip when following is entered. Further, as discussed more completely below, this process does not result in any unexpected motion of the surgical instrument tool tip, and so the motion of the surgical instrument tip is said to intuitively follow the motion of master tool grip <NUM>.

After GENERATE NEW SLAVE ORIENTATION process <NUM> generates desired orientation cRs_des of the slave surgical instrument tip in the camera frame, SEND SLAVE MOVE COMMAND process <NUM> sends a slave movement command <NUM>, sometimes referred to as command <NUM>, which includes desired orientation cRs_des to a slave surgical instrument manipulator <NUM> or other slave instrument controller. In response to command <NUM>, slave surgical instrument manipulator <NUM> moves the slave surgical instrument tip to the desired orientation. It is known how a slave surgical instrument responds to a command to change the orientation of the instrument tip, and so the change in orientation of the instrument tip by the slave surgical instrument is not considered in further detail. After sending slave movement command <NUM>, SEND SLAVE MOVE COMMAND process <NUM> returns to FOLLOWING ENABLED check process <NUM>.

The following process of <FIG> is entered when system <NUM> first starts and continues until system <NUM> disables following. When following is disabled by system <NUM>, in one aspect, an SWOOPING ORIENTATION ERROR ACCEPTABLE check process (not shown, but see element <NUM> in <FIG>) in MASTER-SLAVE ALIGNMENT process <NUM> determines whether an orientation misalignment between the orientation of the master tool grip in the eye frame and the slave surgical instrument tip in the camera frame is smaller than or equal to a maximum permitted swooping orientation misalignment.

If the SWOOPING ORIENTATION ERROR ACCEPTABLE check process determines that orientation misalignment between the orientation of the master tool grip in the eye frame and the slave surgical instrument tip in the camera reference is smaller than or equal to a maximum permitted swooping orientation misalignment, STORE ORIENTATIONS process <NUM> receives and stores the current orientation of the slave surgical instrument tip in the camera frame, and receives and stores the transpose of the current orientation of the master tool grip in the eye frame, which is the orientation of the eye frame in the master frame. In one aspect, STORE ORIENTATIONS process <NUM> is implemented by storage module <NUM>. Storage module <NUM> is included in controller <NUM>. Upon completion of process <NUM>, a following session is enabled.

During the initial start-up of system <NUM> and when the SWOOPING ORIENTATION ERROR ACCEPTABLE check process determines that the orientation misalignment is greater than the maximum permitted swooping orientation misalignment, MASTER-SLAVE ALIGNMENT process <NUM> sends a command to the master tool in surgeon's console <NUM> to move master tool grip <NUM> so that the orientation of master tool grip <NUM> in the eye frame matches the orientation of the slave surgical instrument tip in the camera frame. The use of motors in the master tool to change the orientation of master tool grip <NUM> is known, and so is not considered in further detail. Upon completion of the POSITION MASTER TOOL process, the SWOOPING ORIENTATION ERROR ACCEPTABLE check process is repeated.

The larger allowed misalignment at the end of MASTER-SLAVE ALIGNMENT process <NUM> means that in most cases, MASTER-SLAVE ALIGNMENT process <NUM> is performed only once and then following is entered. Thus, any frustration that was previously experienced by the surgeon due to multiple warnings of misalignment is minimized, if not eliminated.

It will be appreciated that controller <NUM> includes at least one, and typically a plurality, of processors which during a following session determine new corresponding positions and orientations of the slave surgical instrument tip in response to master tool movement input commands on a continual basis determined by the processing cycle rate of the controller <NUM>. A typical processing cycle rate of controller <NUM> is about <NUM>. Thus, when the master tool is moved from one position to a next position, the corresponding desired movement of the slave surgical instrument tip is determined at about <NUM>. Of course, controller <NUM> can have any appropriate processing cycle rate depending on the processor or processors used in the controller.

Prior to considering the inventive aspects described above in further detail, specific examples of each of the references frames are presented with respect to <FIG>. Then, to help illustrate the inventive aspects more clearly, prior systems and their short comings with respect to swooping orientation misalignments are discussed. Finally, the inventive aspects described above are considered in more detail and compared with the prior systems.

In <FIG>, a camera frame <NUM> is positioned such that its origin <NUM> is positioned at the viewing end <NUM> of endoscope <NUM>. In this aspect, camera frame <NUM> is a Cartesian coordinate system, e.g., includes an x-axis Xc, a y-axis Ye, and a z-axis Zc. Rotation about z-axis Zc is referred to roll. Rotation about x-axis Xc is referred to as pitch, and rotation about the y-axis Ye is referred as yaw. Naturally, x-axis Xc, and y-axis Yc, are positioned in a plane perpendicular to z-axis Zc. The association of an orientation with a particular axis is illustrative only and is not intended to be limiting. The association between a particular axis and a rotation can be different than what is described herein, so long as the associations are defined consistently in each of the reference frames.

Conveniently, z-axis Zc of camera frame <NUM> extends axially along a viewing axis <NUM> of endoscope <NUM>. While in <FIG>, viewing axis <NUM> is shown in coaxial alignment with a shaft axis of endoscope <NUM>, it is to be appreciated that viewing axis <NUM> also can have an angle with respect to the lengthwise axis of endoscope <NUM>. Thus, endoscope <NUM> also can be an angled endoscope. Endoscope <NUM> is typically angularly displaceable about its lengthwise axis. The x-, y- and z-axes are fixed relative to viewing axis <NUM> of endoscope <NUM> so as to displace angularly about the lengthwise axis in sympathy with angular displacement of endoscope <NUM> about its lengthwise axis.

To enable teleoperation servo controller <NUM> to determine slave position and orientation, a slave frame <NUM> is defined on or attached to slave surgical instrument tip 312T. In the example of <FIG>, slave frame <NUM> has its origin <NUM> at a pivotal connection <NUM>. In this aspect, slave frame <NUM> is a Cartesian coordinate system, e.g., includes an x-axis Xs, a y-axis Ys, and a z-axis Zs. Rotation about z-axis Zs is referred to roll. Rotation about x-axis Xs is referred to as pitch, and rotation about the y-axis Ys is referred as yaw. Naturally, x-axis Xs, and y-axis Ys, are positioned in a plane perpendicular to z-axis Zs.

Conveniently, one of the axes e.g. z axis Zs, of slave frame <NUM> is defined to extend along an axis of symmetry, or the like, of instrument tip 312T. The orientation of slave surgical instrument <NUM> is defined by the orientation of slave frame <NUM> having its origin at pivotal connection <NUM> relative to camera frame <NUM>. Similarly, the position of slave surgical instrument <NUM> is defined by the position of the origin of slave frame <NUM> relative to origin <NUM> of camera frame <NUM>.

An origin <NUM> of eye frame <NUM> (<FIG>) is chosen such that origin <NUM> corresponds with a position <NUM> where the surgeon's eye is normally located when he or she is viewing the surgical site at viewer <NUM> in surgeon's console <NUM>. In this aspect, eye frame <NUM> is a Cartesian coordinate system, e.g., includes an x-axis Xe, a y-axis Ye, and a z-axis Ze. Rotation about z-axis Ze is referred to roll. Rotation about x-axis Xe is referred to as pitch, and rotation about the y-axis Ye is referred as yaw. Naturally, x-axis Xe, and y-axis Ye are positioned in a plane perpendicular to z-axis Ze.

Z-axis Ze extends along a line of sight of the surgeon, indicated by axis <NUM>, when viewing the surgical site through the viewer <NUM>. Conveniently, y-axis Ye is chosen to extend generally vertically relative to viewer <NUM>, and x-axis Xe is chosen to extend generally horizontally relative to viewer <NUM>.

To enable controller <NUM> to determine the position and orientation of master tool grip <NUM> within eye frame <NUM>, a point on master tool grip <NUM> (<FIG>) is chosen which defines an origin <NUM> of a master frame <NUM>. In one aspect, this point is chosen at a point of intersection <NUM> between a first rotational axis <NUM> of master tool manipulator <NUM> and a second rotational axis <NUM> of master tool manipulator <NUM>. (See also point 3A in Fig. 6A of <CIT>).

In this aspect, master frame <NUM> is a Cartesian coordinate system, e.g., includes an x-axis Xm, a y-axis Ym, and a z-axis Zm. Rotation about z-axis Zm is referred to roll. Rotation about x-axis Xm is referred to as pitch, and rotation about the y-axis Ym is referred as yaw. Naturally, x-axis Xm, and y-axis Ym are positioned in a plane perpendicular to z-axis Zm.

Z-axis Zm of the master frame <NUM> on master tool grip <NUM> is chosen to extend along an axis of symmetry of pinchers 231and <NUM>, which extends coaxially along rotational axis <NUM>. The orientation of master tool grip <NUM> within eye frame <NUM> is defined by the orientation of the master frame <NUM> relative to eye frame <NUM>. The position of master tool grip in eye frame <NUM> is defined by the position of origin <NUM> relative to the origin of eye frame <NUM>. Mapping between the master frame and the eye frame is known. See for example, U. Patent No. <CIT>.

Control between master tool grip movement and slave surgical instrument tip movement is achieved using position ePm and orientation eRm of master tool grip <NUM> in eye frame <NUM> and position cPs and orientation cRs of slave surgical instrument tip 312T in camera frame <NUM>. Here, position ePm is a three by one matrix (represented by P in ePm) with the x, y, z position of the origin <NUM> of master frame <NUM> (represented by m in ePm) in eye frame <NUM> (represented by e in ePm). Similarly, position cPs is a three by one matrix (represented by P in cPs) with the x, y, z position of the origin <NUM> of slave frame <NUM> (represented by s in cPs) in camera frame <NUM> (represented by c in cPm). Orientation eRm is a three by three rotation matrix (represented by R in eRm) specifying the orientation of master frame <NUM> (represented by m in eRm) in eye frame <NUM> (represented by e in eRm). Similarly, orientation cRs is a three by three rotation matrix (represented by R in cRs) specifying the orientation of slave frame <NUM> (represented by s in cRs) in camera frame <NUM> (represented by c in eRm).

A prior system tried to align the master tool grip and the slave surgical instrument, as described above in the BACKGROUND. In this system, the orientation alignment mismatch just prior to entering following was limited to a small orientation misalignment. With respect to processing the orientation, the previous system behavior can be best described as "view-frame centric. " Prior to entering following, the prior system stored the orientation eRm' of the master tool grip in the eye frame and stored the orientation cRs' of the slave surgical instrument tip in camera frame. Orientation eRm' and orientation cRs' were stored as rotation matrices.

In the prior system, an offset matrix Roffset_prior was generated from the stored orientations by multiplying the two stored orientations, e.g., by multiplying the rotation matrices, i.e.: <MAT> It is known that sRc = cRsT, which means sRc' = cRs'T, and so the rotation offset is <MAT>.

In following, the master tool grip undergo additional rotation , which is expressed in eye frame: <MAT> Thus, the current orientation eRm is defined as being equal to the product of a relative master rotation matrix R1 in the eye frame and stored orientation eRm'.

In the prior system, the desired slave orientation cRs_des of the slave surgical instrument tip in the camera frames was defined as: <MAT> Substituting the above definition of rotation offset Roffset in the previous expression gives: <MAT> Subtituting the above definition of additional rotation eRm in the previous expression gives <MAT> However, it is known that rotation matrices are orthogonal matrices, i.e., <MAT> Thus, <MAT>.

Effectively, to send a movement command to control movement of the slave surgical instrument when the master tool grip was moved, the prior system first computed relative master rotation matrix R1 in the eye frame, e.g., from above, the relative master rotation matrix was defended as <MAT> <MAT> but <MAT> <MAT> To compute relative master rotation matrix R1 in the eye frame, the prior system multiplied the current rotation matrix eRm by the transpose of the stored orientation eRm'.

To obtain desired slave orientation cRs_des, the prior system then applied the same relative master rotation matrix R1 to the snapshot slave orientation cRs' in camera frame <NUM>. This process of determining desired slave orientation cRs_des is called "view-frame centric," because the system monitors the delta of the motion of the master tool grip in the eye frame, and replicates that delta on the slave surgical instrument in the camera frame. The use of motion relative to the misalignment between the master tool grip in the eye frame and the slave surgical instrument tip in the camera frame creates motion that would confuse the user of the prior system if the system allowed a swooping orientation misalignment just prior to entering following.

<FIG> is a graphic representation of the location Lm of the master tool grip in eye frame <NUM> following the master-slave alignment in the prior system. For ease of discussion, location Lm is in the Xe-Ye plane. Location Lm is saved as (ePm', eRm'). Location Lm is defined in space with respect to eye frame <NUM>, and so in the prior system, frame <NUM> is a space-fixed coordinate system. <FIG> is a graphic representation of the location Ls of the slave surgical instrument tip in camera frame <NUM> following the master-slave alignment. Location Ls is in the Xc-Yc plane. Location Ls is saved as (cpses, cRs'). In this example, location Ls is defined in space with respect to camera frame <NUM>, and so in the prior system, frame <NUM> is a space-fixed coordinate system. At the end of the master-slave alignment, the slave surgical instrument tip misalignment with the master tool grip is larger than that previously allowed by the system.

Upon entering following, the surgeon rotates the master tool grip, as illustrated in <FIG>. The rotation is about space fixed x-axis Xsm in the Xe-Ye plane. Rotation of the master tool grip about space-fixed axis Xsm in the eye frame results in slave surgical instrument tip rotation about the same space-fixed axis Xss in the camera frame. When the orientation offset is permitted to be large, a rotation of the master tool grip about its pointing direction can result in a circular slave surgical instrument tip motion about the master pointing direction projected onto the slave view frame. However, the surgeon is likely to anticipate that the slave surgical instrument tip will rotate about its x-axis Xss in the Xc-Yc plane.

With the swooping orientation misalignment, the prior art system still determines desired slave orientation cRs_des in response to the rotation of the master tool grip as just described. Due to the initial large displacement in orientation, the slave surgical instrument tip does not rotate as expected, because when the relative rotation is applied to the slave surgical instrument tip, the rotation of the slave surgical instrument tip is no longer in the in the Xc-Yc plane. Rather than rotate in the Xc-Yc plane, the slave surgical instrument tip rather rotates along a circular path <NUM> in camera frame <NUM>. In <FIG>, dashed line <NUM> represents the location of slave surgical instrument tip 312T if master tool grip <NUM> and slave surgical instrument tip 312T were aligned. If the two were aligned, rotation of master tool grip about one of its axes would result in slave surgical instrument tip rotating about the same one of its axes with no spatial displacement from the axis. However, due to the swooping orientation misalignment, the path of the rotation of slave surgical instrument is displaced from the axis as show by path <NUM>.

Thus, in this example, the perceived motion of rotating the master tool grip in a plane about an axis Xsm of the master tool grip does not result in a perceived motion of the surgical instrument tip rotating about corresponding axis <NUM> in a plane. Rather, as just described, the movement of the slave surgical instrument tip is along a circular path <NUM>. Circular path <NUM> is in a plane perpendicular to the Xc-Yc plane. Hence, for swooping orientation misalignments in the prior system, the perceived rotation of the master tool grip and the perceived rotation of the slave surgical instrument tip are not the same.

More generally, as described previously, rotation of the master tool grip about a space-fixed axis in the eye frame results in slave surgical instrument tip rotation about the same space-fixed axis in the camera frame. When the orientation offset is permitted to be large, a rotation of the master tool grip about its pointing direction can result in a circular slave surgical instrument tip motion about the master pointing direction projected onto the slave view frame in the prior system.

Consequently, the prior system would result in unnatural rotation of the slave surgical instrument if the maximum alignment error were increased from the small orientation misalignment to a swooping orientation misalignment. Orientation control module <NUM> in teleoperation servo controller <NUM> eliminates the problems with the prior system when there is a swooping orientation misalignment when a following session is initiated.

Orientation control module <NUM> eliminates the problems associated with the prior systems. First, the criterion for ascertaining whether the misalignment between the master tool grip and slave surgical instrument tip is relaxed. A swooping orientation alignment between master tool grip <NUM> and slave surgical instrument tip 312T is allowed when entering is enabled.

The larger allowed orientation misalignment means that in most cases, MASTER-SLAVE ALIGNMENT process <NUM> is run only once and then following is entered. Thus, any frustration that was previously experienced by the surgeon with the prior system due to multiple warnings of misalignment is minimized, if not eliminated.

Second, despite a large initial orientation misalignment, the motion of slave surgical instrument 312T intuitively follows the motion of master tool grip <NUM>. There is no unexpected motion associated with rotation of master tool grip <NUM>. Thus, despite the swooping orientation misalignment on entering following, the perceived rotation of master tool grip <NUM> is the same as the perceived rotation of slave surgical instrument tip 312T. If the rotation of master tool grip <NUM> is about a body-fixed axis of master tool grip <NUM>, rotation of slave surgical instrument tip 312T is about the same body-fixed axis of slave surgical instrument tip 312T so that the motion of the master and slave is intuitive to the user.

To address the behavior of the prior system when a swooping orientation misalignment is allowed, a "manipulator centric" approach is adopted in orientation control module <NUM>. In the manipulator centric approach, any delta, any relative change, in master tool grip motion is defined with eye frame <NUM> moving relative to a master tool grip <NUM> that is fixed in position, i.e., the master frame <NUM> is a body-fixed frame. Specifically, orientation mRe of the eye frame <NUM> in the master frame <NUM> is defined as being equal to the product of relative rotation matrix R2 and stored orientation mRe', i.e., <MAT>.

Note that it is trivial to obtain orientation mRe of eye frame <NUM> as viewed in master tool frame <NUM> by performing a matrix transpose on the orientation eRm of master frame <NUM> as viewed in eye frame <NUM>, i.e.: <MAT> Also, rotation matrices are orthogonal matrices, i.e., <MAT> where I is the identity matrix.

Thus, mRe' = eRm'T. In this aspect, orientation eRm' from MASTER SLAVE ALIGNMENT process <NUM> is transposed and orientation mRe' is stored in STORE ORIENTATIONS process <NUM>.

In this aspect, offset matrix Roffset is generated from the stored orientations by multiplying the two stored orientations, e.g., by multiplying the two stored rotation matrices, i.e.: <MAT>.

In this aspect, the desired slave orientation cRs_des in the camera frame is defined as: <MAT> Substituting the above definition of rotation offset Roffset in expression (<NUM>) gives: <MAT> <MAT> Substituting this definition of additional rotation eRm in expression (<NUM>) gives <MAT> but, <MAT> Thus, <MAT> However, as noted above, rotation matrices are orthogonal matrices, i.e., <MAT> Thus, <MAT>.

Expression (<NUM>) shows that desired slave orientation cRs_des in the camera frame is a combination of relative rotation matrix R2 and the base orientation cRs' of the slave surgical instrument tip in the camera frame. Thus, the desired slave orientation is defined relative to base orientation cRs' of the slave surgical instrument tip in the camera frame.

Expression (<NUM>) is used to determine desired slave orientation cRs_des in the camera frame in response to a message indicating that master tool grip <NUM> moved. Orientation control module <NUM> first computes transposed relative rotation matrix R2T in master frame <NUM>, e.g., from above relative rotation matrix R2 is defined as <MAT> <MAT> but <MAT> <MAT> <MAT> <MAT> <MAT> Thus, to compute the transpose of relative rotation matrix R2 for use in expression (<NUM>), orientation control module <NUM> multiplies the master tool tip current rotation matrix eRm and stored orientation mRe'.

To obtain desired slave orientation cRs_des, orientation control module <NUM> applies the transpose of relative rotation R2 to snapshot slave orientation cRs' in camera frame <NUM>. This process of determining desired slave orientation cRs_des is called "manipulator centric," because the system monitors the delta of the motion of eye frame <NUM> relative to a fixed in position master tool grip, and replicates that delta on the slave surgical instrument <NUM> in camera frame <NUM>. This method for generating desired slave orientation eRs_des eliminates the problem with unexpected motion of the slave instrument (<FIG>) even though the misalignment between master tool grip <NUM> and slave surgical instrument tip 312T can be significantly larger than in the prior system.

As an example, assume that the misalignment between master tool grip <NUM> and slave surgical instrument tip 312T just prior to entering following is the same as described above with respect to <FIG>. Upon entering following, the surgeon rotates the master tool grip <NUM>, as illustrated in <FIG>. The rotation is about body-fixed x-axis Xbm in the Xe-Ye plane. Now, x-axis Xm of the master frame is a body-fixed axis, e.g., rotation is defined with respect to the x-axis that is fixed on the master tool grip. The surgeon is likely to anticipate that the slave surgical instrument tip will rotate about its body-fixed x-axis Xbs in the Xc-Yc plane, i.e., x-axis Xs of the slave frame is fixed and rotation is defined about the x-axis.

Unlike the motion illustrated in <FIG>, with this implementation, system <NUM> effectively describes the relative motion of eye frame <NUM> in master frame <NUM>, because master frame <NUM> is considered a body-fixed frame, and preserves the same relationship on the slave side. The rotation about the body-fixed master x-axis Xbm produces a rotation about the body-fixed slave x-axis Xbs as desired and as illustrated in <FIG>.

As shown in <FIG>, desired orientation of the slave surgical instrument tip cRs_des generated by controller <NUM> preserves a same perceived rotation between master tool grip <NUM> (<FIG>) and slave surgical instrument tip (<FIG>) even when there is a swooping orientation misalignment between master tool grip <NUM> and slave surgical instrument tip 312T. Here, the rotation of the master tool grip <NUM> is perceived as being in the Xe-Ye plane and the rotation of slave surgical instrument 312T is perceived as being in a corresponding Xc-Yc plane so that the perceived motion of master tool grip <NUM> is the same as the perceived motion of slave surgical instrument tip 312T.

More generally, desired orientation of the slave surgical instrument tip cRs_des generated by controller <NUM> preserves a same perceived rotation between master tool grip <NUM> (<FIG>) and slave surgical instrument tip 312T. As explained previously, a same perceived rotation means that if the rotation of the master tool grip is perceived as being about a body-fixed axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed axis of the slave surgical instrument tip. For example, if the rotation of the master tool grip is perceived as being about the body-fixed x-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed x-axis of the slave surgical instrument tip, or if the rotation of the master tool grip is perceived as being about the body-fixed y-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed y-axis of the slave surgical instrument tip, or if the rotation of the master tool grip is perceived as being about the body-fixed z-axis of the master tool grip, the rotation of the slave surgical instrument is perceived as being about a corresponding body-fixed z-axis of the slave surgical instrument tip.

While a rotation about the x-axis is shown in <FIG> and <FIG>, the same behavior is seen in the prior system and in system <NUM> for rotation about the y-axis and for rotation about the z-axis. Thus, the description herein applies to each of pitch, yaw, and roll. However, a description and illustration for each of the y-axis and the z-axis is not presented, because it would be redundant with the description with respect to the x-axis. One knowledgeable in the field understands the application to rotations about the y- and z-axes without repeating the description of <FIG> and <FIG> for each of the axes.

Nevertheless, another example is presented. In the prior art system, prior to entering following there is an orientation misalignment between z-axis Zm of master frame <NUM> in eye frame <NUM> and z-axis Zs of slave frame <NUM> in camera frame <NUM>. Specifically, z-axis Zm of master frame <NUM> is coincident with z-axis Ze of eye frame <NUM>, while z-axis Zs of slave frame <NUM> is displaced from z-axis Zc of camera frame <NUM>. Z-axis Zs of slave frame <NUM> lies in the Yc-Zc plane.

As explained above, in the prior system, the axes of master frame <NUM> and slave frame <NUM> are spaced fixed. Thus, in the prior system, z-axis Zm of master frame <NUM> is spaced fixed axis Zsm, and z-axis Zs of slave frame <NUM> is space fixed z-axis Zss.

Upon entering following, the surgeon rotates the master tool grip, as illustrated in <FIG>. The rotation is about space fixed z-axis Zsm in the Ze-Ye plane. Rotation of the master tool grip about space-fixed axis Zsm in the eye frame results in slave surgical instrument tip rotation about the same space-fixed axis Zss in the camera frame. When the orientation offset is permitted to be large, a rotation of the master tool grip about its pointing direction can result in a circular slave surgical instrument tip motion about the master pointing direction projected onto the slave view frame. However, the surgeon is likely to anticipate that the slave surgical instrument tip will rotate about its z-axis Zss in the Zc-Yc plane.

With the swooping orientation misalignment, the prior art system still determines desired slave orientation cRs_des in response to the rotation of the master tool grip as just described. Due to the initial large displacement in orientation, the slave surgical instrument tip does not rotate as expected, because when the relative rotation is applied to the slave surgical instrument tip, the rotation of the slave surgical instrument tip is no longer in the in the Zc-Yc plane. Rather than rotate in the Zc-Yc plane, the slave surgical instrument tip rather rotates along a circular path <NUM> in camera frame <NUM>. Due to the swooping orientation misalignment, the path of the rotation of slave surgical instrument is displaced from axis Zc as show by path <NUM>.

Thus, in this example, the perceived motion of rotating the master tool grip about a space-fixed axis Zsm of the master tool grip does not result in a perceived motion of the surgical instrument tip rotating about corresponding axis Zc. Rather, as just described, the movement of the slave surgical instrument tip is along a circular path <NUM>. Circular path <NUM> is in a plane perpendicular to the Zc-Yc plane. Hence, for swooping orientation misalignments in the prior system, the perceived rotation of the master tool grip and the perceived rotation of the slave surgical instrument tip are not the same.

Unlike the motion illustrated in <FIG>, with this implementation, system <NUM> effectively describes the relative motion of eye frame <NUM> in master frame <NUM>, because master frame <NUM> is considered a body-fixed frame, and preserves the same relationship on the slave side. The rotation about the body-fixed master z-axis Zbm produces a rotation about the body-fixed slave z-axis Zbs as desired and as illustrated in <FIG>.

As shown in <FIG>, desired orientation of the slave surgical instrument tip cRs_des generated by controller <NUM> preserves a same perceived rotation between master tool grip <NUM> (<FIG>) and slave surgical instrument tip (<FIG>) even when there is a swooping orientation misalignment between master tool grip <NUM> and slave surgical instrument tip 312T. Here, the rotation of the master tool grip <NUM> is perceived as being in the Ze-Ye plane and the rotation of slave surgical instrument 312T is perceived as being in a corresponding Zc-Yc plane so that the perceived motion of master tool grip <NUM> is the same as the perceived motion of slave surgical instrument tip 312T.

<FIG> is a more detailed process flow diagram for one aspect of a process <NUM> performed by controller <NUM>. For ease of discussion, process <NUM> is shown as a linear process. This is illustrative only and is not intended to be limiting. In view of the following disclosure, process <NUM> can be implemented as described below, or alternatively with some or all of the acts being performed in parallel, for example.

When a user, typically a surgeon, first starts a session at surgeon's console <NUM>, MASTER-SLAVE ALIGNMENT process <NUM> is started. MASTER-SLAVE ALIGNMENT process <NUM> is implemented by master-salve alignment module <NUM>, which is included in teleoperation servo controller <NUM>. Teleoperation servo controller <NUM> is included in controller <NUM>. Thus, each of the acts described below with respect to MASTER-SLAVE ALIGNMENT process <NUM> is performed by controller <NUM>, and so the designation of a teleoperation servo controller is optional.

POSITION MASTER TOOL process <NUM> of MASTER-SLAVE ALIGNMENT process <NUM> is equivalent to the process in prior systems. The location (position and orientation) of the slave surgical instrument tip 312T is determined in camera frame <NUM>, and then POSITION MASTER TOOL process <NUM> sends a command to teleoperation servo controller <NUM> to move master tool grip <NUM> to a corresponding desired location (position and orientation) in eye frame <NUM>. See the above description in Related Art of how this was done in one aspect.

However, if the surgeon grasps master tool grip <NUM> too firmly, teleoperation servo controller <NUM> may not move master tool grip <NUM> to the desired location. Thus, in this case, there is a misalignment between the location of master tool grip <NUM> in eye frame <NUM> and the location of slave surgical instrument tip 312T in camera frame <NUM>. Upon completion of POSITION MASTER TOOL process <NUM> processing transfers to SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> in MASTER SLAVE ALIGNMENT process <NUM>.

SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> determines whether any orientation misalignment between slave surgical instrument tip 312T in camera frame <NUM> and master tool grip <NUM> in eye frame <NUM> is smaller than or equal to a maximum permitted orientation misalignment, i.e., smaller than or equal to a maximum permitted swooping orientation error. Unlike the prior art system that required the maximum orientation misalignment to be equal to or smaller than a few degrees, the limitation on the orientation alignment has been significantly relaxed.

Two factors are used to select the maximum permitted swooping orientation misalignment, in one aspect. The first factor is to make the maximum permitted swooping orientation misalignment large enough that most users of surgical system <NUM> will not be forced to wait while MASTER-SLAVE ALIGNMENT process <NUM> is repeated. The second factor is to select the maximum permitted swooping orientation misalignment to avoid confusion between motions about the various axes by the surgeon. For example, if the maximum permitted swooping orientation misalignment were ninety-degrees, yaw motion of the master tool grip would appear as pitch motion of the slave surgical instrument tip, which most likely would be confusing to the surgeon.

The first factor can be determined, for example, by having a number of inexperienced users use surgical system <NUM>, and determine the largest orientation misalignment after MASTER-SLAVE ALIGNMENT process <NUM> is completed. It has been observed that users with less experience tend to grip master tool grip the hardest, which tends to result in a larger orientation misalignment. Experiments have shown that a maximum permitted swooping orientation misalignment of about fifty degrees does not result in confusion between motions about the various axes.

However, the use of fifty degrees is illustrative only and is not intended to be limiting. In view of this disclosure, one knowledgeable in the field can select a maximum permissible orientation misalignment that is suitable for the users of surgical system <NUM>. For example, in some situation, the maximum permissible orientation might be extended to sixty or more degrees, while in a situation where all the users of surgical system <NUM> are experienced users, the maximum permitted swooping orientation might be lowered to thirty degrees or less.

In any case, the maximum permitted swooping orientation error is larger than the maximum permitted small orientation misalignment of the prior system. Herein a swooping orientation misalignment is defined as follows. A swooping orientation misalignment is an orientation misalignment such that when the desired slave orientation cRs_des of the slave surgical instrument tip in the camera frames is defined as: <MAT> Rotation of master tool grip <NUM> about its pointing direction in eye frame <NUM> results in slave surgical instrument tip 312T moving in a circle, which is perceptible to the user, about the master pointing direction projected on the slave view frame. When this definition of desired slave orientation cRs_des is used and the maximum permitted swooping orientation misalignment is limited to the small misalignment of the prior art, any circular motion of surgical instrument tip 312T is not perceived by the user.

Thus, a swooping orientation misalignment is larger than the prior maximum permitted small orientation misalignment. This definition is also consistent with the fact the new maximum permitted swooping orientation misalignment is selected to limit the number of iterations of MASTER-SLAVE ALIGNMENT process <NUM> to obtain a permissible misalignment for entering following. To distinguish the maximum permitted orientation misalignment from the maximum permitted small orientation alignment of the prior system, the maximum permitted orientation misalignment of system <NUM> is referred to herein as a maximum permitted swooping orientation error.

In one aspect, as described above, SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> determines whether any orientation misalignment between master tool grip <NUM> in eye frame <NUM> and slave surgical instrument in camera frame <NUM> is smaller than or equal to the maximum permitted larger orientation error. SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> first determines the orientation misalignment Misalignment Angle as follows. <MAT> <MAT> where acos is the inverse cosine function. If orientation misalignment Misalignment Angle is smaller than or equal to maximum permitted swooping orientation error, SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> transfers processing to STORE LOCATIONS process <NUM>, and otherwise transfers processing to SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> returns processing to POSITION MASTER TOOL process <NUM>. Thus, in one aspect, if orientation misalignment Misalignment Angle is smaller than or equal to fifty degrees, SWOOPING ORIENTATION ERROR ACCEPTABLE check process <NUM> transfers processing to STORE LOCATIONS process <NUM>.

STORE LOCATIONS process <NUM>, sometimes referred to as process <NUM>, includes STORE ORIENTATIONS process <NUM>, in one aspect. In one aspect, STORE LOCATIONS process <NUM> is implemented by storage module <NUM>.

STORE LOCATIONS process <NUM>, in one aspect, first transposes eRm' to obtain mRe', and then stores location Lm(ePm', mRe') of master tool grip <NUM> in memory <NUM> and stores Ls(cPs', cRs') of slave surgical instrument tip 312T in memory <NUM>. Upon completion of STORE LOCATIONS process <NUM>, following is enabled and FOLLOWING process <NUM> is entered.

FOLLOWING process <NUM> is implemented by following module <NUM> which is included in teleoperation servo controller <NUM>, in one aspect. Teleoperation servo controller <NUM> is included in controller <NUM>. Thus, each of the acts described below with respect to FOLLOWING process <NUM> is performed by controller <NUM>.

In FOLLOWING process <NUM>, if following is enabled, FOLLOWING ENABLED check process <NUM> passes each command <NUM> from the master tool manipulator to GENERATE NEW SLAVE POSITION process <NUM>, and otherwise takes no action.

GENERATE NEW SLAVE POSITION process <NUM>, sometimes referred to as process <NUM>, determines new desired position cPs_des of slave surgical instrument tip 312T in camera frame <NUM>. Master command <NUM> includes a position ePm of master tool grip <NUM> in eye coordinate frame <NUM>. To determine a relative position change Δ in the position of master tool grip <NUM>, stored position ePm' is retrieved from memory <NUM> by process <NUM>, and subtracted from position ePm, i.e., <MAT> where Δ is a three by one matrix. The new desired position cPs_des is: <MAT> After determining, new desired position cPs_des, GENERATE NEW SLAVE POSITION process <NUM> transfers to GENERATE NEW SLAVE ORIENTATION process <NUM>.

GENERATE NEW SLAVE ORIENTATION process <NUM>, sometimes referred to as process <NUM>, determines new desired orientation cRs_des of slave surgical instrument tip 312T in camera frame <NUM>. As described above, <MAT> <MAT> Thus, GENERATE NEW SLAVE ORIENTATION process <NUM> retrieves stored orientation mRe' and stored orientation cRs', and then multiplies orientations cRs', mRe', and eRm together to generate new desired orientation cRs_des. As explained with respect to <FIG>, desired orientation of the slave surgical instrument tip cRs_des generated by controller <NUM> preserves a same perceived rotation between master tool grip <NUM> and slave surgical instrument tip 312T even when there is a swooping orientation misalignment between master tool grip <NUM> slave surgical instrument tip 312T. After determining, new desired orientation eRs_des, GENERATE NEW SLAVE ORIENTATION process <NUM> transfers to SEND SLAVE MOVE COMMAND process <NUM>.

After GENERATE NEW SLAVE ORIENTATION process <NUM> generates desired orientation cRs_des of the slave surgical instrument tip in the camera frame, SEND SLAVE MOVE COMMAND process <NUM> sends a slave movement command <NUM>, sometimes referred to as command <NUM>, which includes desired position cPs_des and desired orientation cRs_des to surgical instrument manipulator <NUM>. In response to command <NUM>, slave surgical instrument manipulator <NUM> moves surgical instrument tip 312T of slave surgical instrument <NUM> to the desired location. In this aspect, after sending slave movement command <NUM>, SEND SLAVE MOVE COMMAND process <NUM> transfers to FOLLOWING DISABLED check process <NUM>.

If following is disabled by system <NUM>, FOLLOWING DISABLED check process <NUM> transfers to LARGE ORIENATION ERROR ACCEPTABLE check process <NUM>, and otherwise returns to GENERATE NEW SLAVE POSITION process <NUM>. FOLLOWING DISABLED check process <NUM> should not be interpreted as requiring process <NUM> to poll to determine whether processing has been disabled in every cycle performed by process <NUM>. For example, FOLLOWING process <NUM> could be allowed until an event handler receives a following disabled event, and then the event handler disables FOLLOWING process <NUM>.

The above description and the accompanying drawings that illustrate aspects and embodiments of the present inventions should not be taken as limiting-the claims define the protected inventions. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the scope of this description and the claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail to avoid obscuring the inventive aspects.

Further, this description's terminology is not intended to limit the invention. For example, spatially relative terms-such as "beneath", "below", "lower", "above", "upper", "proximal", "distal", and the like-may be used to describe one element's or feature's relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions and orientations of the device in use or operation in addition to the position and orientation shown in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be "above" or "over" the other elements or features. Thus, the exemplary term "below" can encompass both positions and orientations of above and below.

The singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context indicates otherwise. The terms "comprises", "comprising", "includes", and the like specify the presence of stated features, steps, operations, elements, and/or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups.

All examples and illustrative references are non-limiting and should not be used to limit the claims to specific implementations and embodiments described herein and their equivalents. Any headings are solely for formatting and should not be used to limit the subject matter in any way, because text under one heading may cross reference or apply to text under one or more headings. Finally, in view of this disclosure, particular features described in relation to one aspect or embodiment may be applied to other disclosed aspects or embodiments of the invention, even though not specifically shown in the drawings or described in the text.

While the memory is illustrated as a unified structure, this should not be interpreted as requiring that all memory is at the same physical location. All or part of the memory can be in a different physical location than a processor. Memory refers to a volatile memory, a non-volatile memory, or any combination of the two.

A processor is coupled to a memory containing instructions executed by the processor. This could be accomplished within a computer system, or alternatively via a connection to another computer via modems and analog lines, or digital interfaces and a digital carrier line.

Herein, a computer program product comprises a medium configured to store computer readable code needed for any one or any combination of the methods and/or processes described herein, or in which computer readable code for any one or any for any one or any combination of the methods and/or processes described herein is stored. Some examples of computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes, computer hard drives, servers on a network and signals transmitted over a network representing computer readable program code. A tangible computer program product comprises a medium configured to store computer readable instructions for any one or any combination of the methods and/or processes described herein, or in which computer readable instructions for any one or any combination of the methods and/or processes described herein, is stored. Tangible computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes, computer hard drives and other physical storage mediums.

Claim 1:
A surgical system (<NUM>) comprising:
a master tool (<NUM>) manipulator;
a controller (<NUM>) coupled to the master tool manipulator (<NUM>);
wherein the controller (<NUM>) is configured to: before a following mode is entered:
determine an orientation misalignment, Roffset, between a first orientation of the master tool manipulator (<NUM>) in an eye frame (<NUM>) and an orientation of a slave instrument tip (212T, 312T) in a camera frame (<NUM>), cRs', and
in response to a determination that the following mode is enabled and the orientation misalignment, Roffset, is less than or equal to a maximum permitted orientation error:
store a transpose of the first orientation of the master tool manipulator in the eye frame mRe',
store the orientation of the slave instrument tip in the camera frame, cRs', and
enter the following mode; and
wherein the controller (<NUM>) is further configured to: after the following mode is entered:
receive a command from the master tool manipulator (<NUM>) to move the slave instrument tip (212T, 312T), the command indicating a second orientation of the master tool manipulator in the eye frame, eRm,
generate a new slave orientation, cRs des, based on the second orientation of the master tool manipulator in the eye frame, eRm, the stored transpose of the first orientation of the master tool manipulator in the eye frame, mRe', and the stored orientation of the slave instrument tip in the camera frame, cRs', and
command the system (<NUM>) to orient the slave instrument tip (212T, 312T) based on the new slave orientation, cRs_des, wherein the desired orientation of the surgical instrument in the camera frame is defined as: cRs_des = cRs' * mRe' * eRm.