Patent Description:
Tele-operated surgical systems are often intended to enhance surgeon precision and/or reduce patient trauma during a medical procedure. In such systems, a surgeon interacts with input devices (sometimes referred to as "masters" or "master controllers") to control surgical instruments that are actuated by drive mechanisms such as motors. Because the surgeon is not directly manipulating the surgical instruments, in can sometimes be beneficial to provide haptic feedback at the input devices that indicates or replicates interaction forces (e.g., felt at the surgical instruments, other elements of the surgical system, and/or virtual or synthetic elements/features generated by the surgical system). Note that the force feedback being presented to the user may be the sum of the feedback from a sensor, from an algorithm, from a user interface cue, collision detection, model interaction, etc..

To provide a good user experience, the surgeon would ideally experience a seamless haptic experience through system state and configuration changes. However, this can be difficult to accomplish, for example the actuators used to provide the haptic feedback are have different performance limits or if different actuators reach their performance limits at different times. In such instances, the haptic feedback presented to the user may not properly align with the perceived user experience (e.g., actual force being sensed at the surgical instrument or the visual representation of the interactive object), resulting in a confusing or non-intuitive user experience.

For example, motor torque limits (i.e. saturation limits) are typically imposed in software for robotic interfaces. These performance limits can be created for several reasons including to protect the motor from overheating, to limit the force applied to the surgeon/patient, and/or to keep the motor in it's ideal torque operating range. Limiting the torque at the motor can yield non-isotropic force saturation at the interface (e.g., the handle of the input device or the tip of the instrument). This means when you are trying to present a force at the interface that involves two or more motors, the force direction can be incorrect if one motor is restricted by a motor torque limit.

This can be especially problematic when rendering force to the user (haptic feedback) at the input device. The user may be feeling a force in a given direction, but as the force applied to the user increases and reaches a torque limit for any of the motors associated with render the force to the user, the direction of the force displayed to the user begins to rotate which can be disconcerting and/or confusing for the user.

It is therefore desirable to provide a system for ensuring haptic feedback that is consistent with the force environment at the surgical instrument.

<CIT> discloses a hand controller or 'cursor' for the actuation of slaved apparatus. The cursor includes a handle extending from a line attachment member comprised of four equally spaced legs. The ends of these legs define vertices of a regular tetrahedron. The cursor is supported within a subtending structure by a plurality of tension lines which may be connected to torque motors or to another controlled device. A control computer preferably interfaces the present controller with the slaved apparatus wherein signals representative of the several tension line lengths are translated by the computer into control signals actuating the slaved device. In addition, force transducers on the slaved mechanism generate signals representative of the forces encountered by the slaved device which, in turn, are translated by the computer into torque motor control signals thereby providing interactive cursor force-feedback.

<CIT> discloses a remote control catheterization system comprising: a propelling device, which controllably inserts a flexible, elongate probe into the body of a patient; and a control unit, in communication with the propelling device, and comprising user controls which are operated by a user of the system remote from the patient to control insertion of the probe into the body by the propelling device, wherein the user controls include an intuitive user interface comprising a handle that can be moved longitudinally, forward and back along a longitudinal axis, and also can be moved rotationally, in rotation around the longitudinal axis; the intuitive user interface comprising motion sensors that detect longitudinal motion and rotational motion of the handle and convert them to signals; and signal communication circuitry that communicates the signals to the control unit for commanding the propelling device to move the probe in respective direction and distance as the handle.

<CIT> discloses a robotic surgical system which has a robot arm holding an instrument for performing a surgical procedure, and a control system for controlling movement of the arm and its instrument according to user manipulation of a master manipulator. The control system includes a filter in its forward path to attenuate master input commands that may cause instrument tip vibrations, and an inverse filter in a feedback path to the master manipulator configured so as to compensate for delay introduced by the forward path filter. To enhance control, master command and slave joint observers are also inserted in the control system to estimate slave joint position, velocity and acceleration commands using received slave joint position commands and torque feedbacks, and estimate actual slave joint positions, velocities and accelerations using sensed slave joint positions and commanded slave joint motor torques.

<CIT> discloses an apparatus and method for controlling a robot may scale a motion of a surgical robot based on a type of object gripped by the surgical robot. In the robot controlling method, by scaling the motion of the surgical robot based on the type of object gripped by the surgical robot, the surgical robot may automatically perform the motion on objects using an optimized force although a user does not control a force minutely based on the type of object gripped by the surgical robot.

<CIT> discloses a master motion information obtaining unit obtains at least one or more pieces of master motion information including a position, a posture, a speed, and an angular velocity of a master arm mechanism. A physical information obtaining unit obtains physical information of an operator including an arm weight of the operator. A master motion information correcting unit generates corrected master motion information where an amount of correction of the master motion information is corrected such that heavier the arm weight of the operator included in the physical information, larger a movement of a slave arm. A slave controller controls a slave arm mechanism, according to the corrected master motion information.

To minimize discrepancies between expected force directions and haptic force feedback directions, the outputs of the haptic feedback actuators are scaled whenever one or more of the actuators reaches a predetermined output threshold, thereby maintaining proper haptic feedback directionality when individual actuators would otherwise be commanded to operate outside their accurate performance range. Such scaling may change the overall haptic feedback magnitude, but allows the haptic feedback direction to be appropriately maintained.

In some embodiments, where software limits the output of any actuator, a monitoring process can determine when an output of an actuator would exceed a maximum threshold output (at or below the software-defined output limit), and at that point scale down the output of at least one of the other actuators so that the desired direction of the total output (e.g., force or torque) is maintained. In some embodiments, the monitoring process can additionally or alternatively determine when an output of an actuator would fall below a minimum output threshold (below which the actuator output may be too low to generate accurately), and at that point scale up its output and that of at least one of the other actuators so that the desired direction of the total output (e.g., force or torque) is maintained. In some embodiments, all actuators are scaled when one reaches its threshold output, while in other embodiments, only concurrently active actuators are scaled. In some embodiments, the output thresholds for die actuators are fixed, and in various other embodiments, the output thresholds for the actuators can vary over time or based on actuator and/or system status.

In the following detailed description of the aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be obvious to one skilled in the art that the embodiments of this disclosure may be practiced without these specific details. In other instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention. And, to avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments.

To minimize discrepancies between desired (e.g., sensed or modeled by a surgical system) and actual haptic force feedback directions, the outputs of the haptic feedback actuators are scaled whenever a commanded output of one or more of the actuators falls outside a predetermined working range for that actuator. The predetermined working range can be defined by a maximum output threshold equal to or less than the output limit of the actuator, and/or a minimum output threshold equal to or greater than a minimum accurate output level of the actuator. Such scaling may reduce the overall haptic feedback magnitude, but allows the haptic feedback direction to be appropriately maintained, which is often a more critical aspect of haptic feedback.

<FIG> shows an exemplary method for providing directionally consistent haptic feedback when actuator output limits are exceeded. In a PROVIDE HAPTIC FEEDBACK step <NUM>, a surgical system that allows a user (e.g., surgeon) to control a surgical instrument (and/or other elements of the surgical system, such as a robotic arm, set up structure, or positioning element such as a boom or cart) via an input device(s) (e.g., lever(s), gripper(s), joystick(s), or any other structure capable of receiving user input), and then provides force feedback to that input device based on a desired haptic feedback profile (a set of one or more haptic feedback effects that at least partially reproduce or represent the physical experience of a real or virtual/modeled interaction). The haptic feedback profile can be based on any haptic model input, such as sensed forces at the instrument (e.g., tissue or other instrument interactions) or robotic arm (e.g., aim collisions with structures or staff), user guidance (e.g., haptic detents, fences, or other profiles to provide guidance for the user to move the input device(s) along a desired path or trajectory), and user interface (UI) elements (e.g., presenting a virtual handle or steering wheel to the user). This haptic feedback can be anything from direct replication of the haptic feedback profile, to scaling of the haptic feedback profile, to applying a nonlinear modification of the haptic feedback profile, or any other transformation (e.g., force scaling that varies depending on one or more other factors such as instrument state/speed, viewing magnification, etc.).

The actual force feedback provided at the input device is generated by two or more actuators (e.g., motors, drives, or any other motive elements) that work cooperatively to provide feedback of varying force and direction. For example, an input device having pitch and yaw capabilities may be coupled to a first pair of actuators that apply forces in opposing directions about the pitch axis, and a second pair of actuators that apply forces in opposing directions about the yaw axis. Two or more of the pitch and yaw actuators can then be used simultaneously to provide force feedback that is offset from the pitch and yaw axes.

Because actuators generally do not have performance characteristics that are exactly the same, whether due to inherent performance limitations or due to operational constraints/effects (e.g., heat, mechanical restrictions). For example, one out of the group of haptic feedback actuators will typically reach its maximum output level before the others. Any commanded output beyond that maximum output level will not result in any increased output, and consequently any haptic feedback that involves the maxed-out actuator will likely deviate from the expected force feedback direction. Additionally, the actuators will may begin producing noisier (less precise) outputs as outputs decrease below a certain level, such that haptic feedback below a certain level may also deviate from the expected force feedback direction.

<FIG> graphically depicts an example of this haptic offset, with the outputs of first and second actuators ("Actuator <NUM>" and "Actuator <NUM>", respectively) represented by the graph's horizontal and vertical axes, respectively. Also indicated on the axes of the graph are output limits OL1 and OL2 for the first and second actuators, respectively, representing the output limits for the first and second actuators.

As noted above, the actuators cannot exceed their output limits, which may be static (e.g., defined by inherent performance characteristics of the actuators), or dynamic (e.g., based on a current parameter of an actuator, such as temperature, or a physical restriction such as a kinematic configuration of the driven structure that places the actuator in a mechanically unfavorable position).

The problem thus arises if a commanded output of an actuator exceeds its output limit, such as indicated by desired (commanded) feedback force FD. To generate desired feedback force FD, actuator <NUM> receives a commanded output CO1, and actuator <NUM> receives a commanded output CO2. Commanded output CO1 is less than output limit OL1, and so can be provided by actuator <NUM>. However, because commanded output CO2 is greater than output limit OL2, the actual output of actuator <NUM> will be limited to output level OL2, resulting in an overall feedback force FO that is both smaller than, and offset from, desired feedback force FD. While a slight variability in the magnitude of haptic feedback can generally be accommodated by a user without much difficulty, a deviation in force direction can be quite misleading to a user and can result in improper control actions by the user in response.

Returning to <FIG>, to mitigate this force feedback offset issue, in an ACTUATOR THRESHOLD DETECTION step <NUM>, any commanded output for an actuator that exceeds a maximum output threshold for that actuator is identified. Note that while in some embodiments the maximum output threshold for an actuator can be defined as the output limit for that actuator, in various other embodiments the maximum output threshold could be set at a level below the output limit to provide a buffer for detecting the approach of the actuator limit and/or applying the scaling factor (as described in greater detail below) before any output limits are reached. Note further that as described above, the maximum output thresholds for the haptic feedback actuators can be either static or dynamic, and can be individualized or common across the actuators.

Then, in a GLOBAL ACTUATOR SCALING step <NUM>, a common scaling factor is applied to the commanded outputs for the actuators. The scaling factor is selected to keep the output of the identified actuator (i.e., the actuator identified in step <NUM> having a commanded output greater than its output threshold) less than its output limit. Because the scaling factor is applied to the commanded output for each actuator, the direction of die overall force is maintained, with a reduced overall magnitude. Note that in some embodiments, if multiple actuators would receive commanded outputs that would exceed their output limits, the scaling factor would be based on the commanded output that exceeds its associated actuator's output threshold by the greatest amount - i.e., the scaling factor would be based on the "worst" output discrepancy.

<FIG> graphically depicts an example of this haptic scaling, relative to the same first and second actuator characteristics (output limits OL1 and OL2, respectively) and desired feedback force FD. However, rather than allowing the output limits associated with actuator <NUM> to pull the overall haptic force off the desired force direction as shown in <FIG>, a scaling factor is applied to the commanded outputs CO1 and CO2 to reduce both proportionately to adjusted commanded outputs CO1 and CO2, respectively. The scaling factor is selected such that adjusted commanded output EO2' reduced at least to output limit OL2, although in various other embodiments the scaling factor can be selected to cause adjusted commanded output CO2' to be some increment less than output limit OL2.

In some embodiments, actuators <NUM> and <NUM> could additionally or alternatively exhibit reduced output accuracy at low output levels. In such embodiments, a scaling factor could be applied to commanded outputs CO1 and CO2 if either is less than a minimum output threshold for actuator <NUM> or actuator <NUM>, respectively. The scaling factor would then increase the adjusted commanded outputs CO1' and CO2' above the level at which output accuracy is degraded. Note that as described above, the minimum output thresholds for the haptic feedback actuators can be either static or dynamic, and can be individualized or common across the actuators.

In any event, the application of the scaling factor results in an overall scaled feedback force FS that is aligned directionally with die original desired feedback force FD. As noted above, a consistent haptic experience can be provided even with the change in force magnitude so long as directional consistency of the force feedback is maintained.

Returning to <FIG>, in various embodiments, when it is detected that the commanded output (unsealed) for the actuator identified in step <NUM> would go beyond the output threshold (e.g., exceed the maximum output threshold or fall below the minimum output threshold) for that actuator in an optional ACTUATOR SUB-THRESHOLD DETECTION step <NUM>, the scaling factor applied to all commanded outputs in step <NUM> is removed (or set to <NUM>) in an optional GLOBAL ACTUATOR DE-SCALING step <NUM>. Returning to step <NUM>, unsealed haptic feedback is provided going forward.

As noted previously, in some embodiments, the output thresholds for the haptic feedback actuators can be dynamic - i.e., the specific values can change depending on actuator operational parameters, input device kinematic configuration, or various other factors. In such embodiments, over the course of operation of the surgical system, the output threshold(s) applied in step <NUM> can have different values. In addition, in various other embodiments, different actuators can trigger step <NUM> over the course of operation of the surgical system.

<FIG> and <FIG> shows block diagram of a surgical system <NUM> incorporating haptic feedback at an input device <NUM> and means for providing a consistent haptic experience for the user as described above with respect to <FIG>, <FIG>. Surgical system <NUM> includes an instrument <NUM> for performing a surgical task (e.g., forceps, cutter, retractor, vessel sealer, needle driver, catheter, etc.), an input device <NUM> (e.g., a lever(s), gripper(s), joystick(s), or any other structure capable of receiving user input) for receiving inputs from a user (e.g., surgeon), and a controller <NUM> for receiving input instructions from input device <NUM>, controlling the actions of instrument <NUM> accordingly via a manipulation structure <NUM>, and providing instructions to a haptic feedback actuation mechanism <NUM> to provide haptic feedback to input device <NUM> according to a desired haptic feedback profile. In various embodiments, manipulation structure <NUM> can include any number of systems and structures for maneuvering, positioning, actuating, or otherwise controlling the behavior of instrument <NUM>, including a robotic arm(s)/manipulator(s), set up structure(s), and/or positioning element(s) such as a boom(s) or cart(s), among others. Controller <NUM> can include any combination of hardware, software, firmware, and other modalities for generating, managing, controlling, and effecting the actions described herein. In various embodiments, controller <NUM> can be integrated with instrument <NUM>, input device <NUM>, and/or discrete control hardware (e.g., a standalone processing unit or computing platform).

For exemplary purposes, <FIG> shows an end effector <NUM> at the end of a shaft <NUM> of instrument <NUM> grasping a portion of tissue <NUM> (e.g., retraction). This results in a force FM at end effector <NUM>, which would ideally be delivered as a desired haptic feedback profile force FD at input device <NUM>. The actual haptic feedback delivery is enabled by haptic feedback actuation mechanism <NUM>, which includes multiple actuators that attempt to provide the haptic feedback profile force FD to present the surgeon with a "feel" of the resistance being provided by tissue <NUM> as it is being retracted.

Although the haptic feedback profile force FD is described as being derived from a force FM sensed at end effector <NUM> of instrument <NUM> for exemplary purposes, in various other embodiments, force FM could be sensed at any location for which corresponding haptic feedback at input device <NUM> would be beneficial, such as interactions at shaft <NUM> or any other element of manipulation structure <NUM> (e.g., arm collisions with structures or staff).

In various other embodiments, force FM can be defined according to non-physical parameters, such as guidance or user interface features. For example, in some embodiments, surgical system <NUM> can include a display <NUM> (e.g., a monitor(s), a head-in viewer(s), projections, video glasses/helmet(s), and/or any other graphical presentation element). In various embodiments, display <NUM> can present a virtual or synthetic element <NUM> that can be interacted with via input device <NUM>. In some embodiments, synthetic element <NUM> can be used as a supplemental interface for interacting with a physical component of surgical system <NUM>. For example, as shown in <FIG>, synthetic element <NUM> can be a virtual handle or knob that can be "grasped" and dragged around using input device <NUM> to reposition instrument <NUM> at the surgical site. In other embodiments, synthetic element <NUM> can provide a purely virtual interaction element, such as a dial, toggle, lever, or any other structure for controlling surgical system <NUM>. In any case, by generating a haptic feedback profile based on model forces FM1 associated with interacting with synthetic element <NUM> (e.g., radially outward resistive force produced with grasping a round knob), controller <NUM> can then attempt to provide an appropriate haptic feedback profile force FD at input device <NUM>.

In various other embodiments, surgical system <NUM> may provide guidance to the user with respect to movement of instrument <NUM> and/or input device <NUM>. For example, a desired motion of instrument <NUM> (e.g., a targeted or safe dissection path, a desired retraction movement, or any other beneficial articulation) could optionally be defined as a trajectory <NUM>. By generating a haptic feedback profile based on model forces FM2 associated with maintaining the position of instrument <NUM> along trajectory <NUM> (e.g., inwardly directed forces produced upon deviations from trajectory <NUM>), controller <NUM> can then attempt to provide an appropriate haptic feedback profile force FD at input device <NUM>.

<FIG> shows an exemplary block diagram of actuation mechanism <NUM> that includes multiple actuators <NUM> that apply components of the haptic feedback force to input device <NUM>. Note that while four actuators that drive input device <NUM> via cables or tendons are depicted for descriptive purposes, in various other embodiments, actuation mechanism <NUM> can include any number and type of actuators (e.g., rotary actuators, linear actuators, hydraulics, and/or piezo-electric, vibrotactile, or fluidic actuators) and/or force transmission mechanisms (e.g., direct drive, linkages, gearing, etc.).

The various actuators <NUM> provide actuation outputs (e.g., torque or force) in combination with one another in an effort to produce the desired haptic feedback profile force FD. However, if desired haptic feedback profile force FD requires an output from one of actuators <NUM> that falls outside its working range, attempting to use the commanded outputs for actuators <NUM> without modification would result in an unmodified haptic feedback force FO having both a magnitude and direction different than desired force FD (as described above with respect to <FIG> and <FIG>). As described above, the working range of an actuator can be defined by a maximum output threshold (e.g., at or below die output limit of the actuator) and/or a minimum output threshold (e.g., at or above a minimum reliable output level of the actuator).

Therefore, when controller <NUM> detects that a commanded output for any of actuators <NUM> would go beyond that actuator's defined output threshold (as described above with respect to step <NUM> in <FIG>), it applies a common scaling factor to the commanded outputs supplied to each actuator <NUM> such that the commanded outputs for all actuators <NUM> remain within their predetermined working range (as described above with respect to step <NUM> in <FIG>). This has the effect of changing the output of all actuators <NUM> proportionately, which in turn results in a scaled haptic feedback force FS that, while different in magnitude with respect to the desired feedback force FD, remains directionally aligned with desired feedback force FD (as described above with respect to <FIG>).

Claim 1:
A surgical system comprising:
an input device (<NUM>);
a first actuator (313A) coupled to the input device;
a second actuator (313B) coupled to the input device (<NUM>); and
a controller (<NUM>);
wherein the input device (<NUM>) is capable of receiving input from a user and providing a haptic feedback force to the user;
wherein the first actuator (313A) has a maximum output threshold force and a minimum reliable output level force;
wherein the second actuator (313B) has a maximum output threshold force and a minimum reliable output level force;
wherein a working range of the first actuator (313A) is defined by an upper limit at or below the maximum output threshold force of the first actuator (313A) and a lower limit at or above the minimum reliable output level force of the first actuator (313A);
wherein a working range of the second actuator (313B) is defined by an upper limit at or below the maximum output threshold force of the second actuator (313B) and a lower limit at or above the minimum reliable output level force of the second actuator;
wherein the controller (<NUM>) determines a first commanded output haptic feedback force for the first actuator (313A) to provide to the input device (<NUM>) and a second commanded output haptic feedback force for the second actuator (313B) to provide to the input device (<NUM>);
wherein on a condition in which the first commanded output haptic feedback force would fall outside the working range of the first actuator (313A), the controller (<NUM>) applies a common first scaling factor to the first commanded output haptic feedback force for the first actuator (313A) to provide to the input device (<NUM>) and to the second commanded output haptic feedback force for the second actuator (313B) to provide to the input device (<NUM>);
wherein the first scaling factor adjusts the first commanded output haptic feedback force to be within the working range of the first actuator (313A); and
wherein the first scaling factor adjusts the second commanded output haptic feedback force to be within the working range of the second actuator.