Patent Description:
The present disclosure is directed to systems and computer-readable mediums having instructions stored thereon which, when executed by a processor, cause a control system to perform a method for controlling two or more teleoperational manipulators.

Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions physicians or other staff may insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, or biopsy instruments) to reach a target tissue location. To facilitate control by an operator, techniques have been used to make the control the medical instruments intuitive. However, as more instruments are involved in a given procedure and/or the instruments are not coupled to each other in a known configuration the control of all the instruments becomes more complicated. This is true as well in non-medical contexts, such as in industrial applications, in which multiple manipulators are controlled from a particular perspective.

<CIT> discloses a surgical manipulator system comprising at least one surgical manipulator, at least one guide, a detector, and a drive controller. The surgical manipulator has a surgical device for performing a desired operation. The guide guides the surgical device. The detector detects a position and/or orientation relationship between the surgical device and the guide, and/or a position and/or orientation relationship between the surgical device and another surgical device. The drive controller controls the surgical manipulator such that the surgical device is guided by the guide.

<CIT> discloses a patient-side surgeon interface provides enhanced capabilities in using a minimally invasive, teleoperated surgical system. The patient-side surgeon interface has components within the sterile surgical field of the surgery. The components allow a surgeon to control teleoperated slave surgical instruments from within the sterile surgical field. The patient-side surgeon interface permits a surgeon to be in the sterile surgical field adjacent a patient undergoing surgery. Controlling minimally invasive slave surgical instruments from within the sterile surgical field permits minimally invasive surgery combined with direct visualization by the surgeon. The proximity to the patient allows the surgeon to control a teleoperated slave surgical instrument in tandem with controlling manually controlled instruments such as a laparoscopic instrument. Also, the surgeon, from within the sterile surgical field, can use the patient-side surgeon interface to control at least one proxy visual in proctoring another surgeon.

<CIT> discloses a method and system for telematic control of a slave device. Displacement of a user interface control is sensed with respect to a control direction. A first directional translation is performed to convert data specifying the control direction to data specifying a slave direction. The slave direction will generally be different from the control direction and defines a direction that the slave device should move in response to the physical displacement of the user interface. A second directional translation is performed to convert data specifying haptic sensor data to a haptic feedback direction. The haptic feedback direction will generally be different from the sensed direction and can define a direction of force to be generated by at least one component of the user interface. The first and second directional translation are determined based on a point-of-view of an imaging sensor.

Accordingly, it would be advantageous to provide control systems and methods that enable control of multiple instruments in changeable configurations with respect to each other.

The embodiments of the invention are best summarized by the claims that follow the description.

Consistent with some embodiments, one general aspect includes a teleoperational system including: a first manipulator configured to support an instrument moveable within an instrument workspace, the instrument having an instrument frame of reference, an operator input device configured to receive movement commands from an operator, and a control system configured to implement the movement commands. The control system of the teleoperational system implements the movement commands by comparing an orientation of the instrument with an orientation of a field of view of the instrument workspace to produce an orientation comparison. When the orientation comparison does not meet an orientation criterion set, the control system causes instrument motion in a first direction relative to the instrument frame of reference in response to a movement command. When the orientation comparison meets the orientation criterion set, the control system causes instrument motion in a second direction relative to the instrument frame of reference in response to the movement command, where the second direction differs from the first direction.

Consistent with other embodiments, another general aspect includes a computer readable medium having instructions stored thereon which, when executed by a processor, cause a control system to perform a method of controlling a teleoperational system. The method includes comparing an orientation of a manipulator arm configured to receive an instrument with an orientation of a field of view of a workspace to produce an orientation comparison. When the orientation comparison does not meet an orientation criterion set, the method includes causing instrument motion in a first direction relative to an instrument frame of reference in response to a movement command. When the orientation comparison meets the orientation criterion set, the method includes causing instrument motion in a second direction relative to the instrument frame of reference in response to the movement command, where the second direction differs from the first direction. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Consistent with other examples, there is also disclosed a robotic system. The robotic system includes an operator input system having a first input device, a first robotic platform having a first slave manipulator arm configured to receive a first instrument.

The first slave manipulator arm and the first instrument have at least one reference frame. The robotic system also includes a second robotic platform having a second slave manipulator arm configured to receive an imaging system, where the imaging system provides an imaging reference frame, and a control system that receives movement commands via the first input device. The received movement commands indicate a first direction of movement relative to the imaging reference frame. The control system implements the received movement commands on a platform of the first and second robotic platforms by comparing an orientation of the first instrument in the instrument reference frame with an orientation of the imaging system in the imaging reference frame, determining a mapping to apply to the received movement commands to generate implementable movement commands for the first instrument, and causing instrument motion in a second direction relative to the instrument reference frame in response to the movement command, where the second direction is different than the first direction.

In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

In some instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

This disclosure describes various instruments and portions of instruments in terms of their position, orientation, and/or pose in three-dimensional space. As used herein, the term "position" refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom).

While some embodiments provided herein are discussed primarily with respect to medical procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, human or animal tissue removed from human or animal anatomy and not to be returned to such human or animal anatomy, non-surgical treatment, diagnosis, or cosmetic improvements. The systems, instruments, and methods described herein may also be used for industrial systems and general robotic or teleoperational systems, including those for manipulating or otherwise interacting with work pieces not comprising human or animal tissue.

<FIG> is a diagrammatic view that shows general system components of a computer-assisted, teleoperated system <NUM>. As shown, the system includes a primary slave manipulator <NUM> that moves and controls a first instrument, and an external slave manipulator <NUM> that moves and controls a second instrument. Such manipulators are also called arms or manipulator arms. When implemented as a medical system, the instruments of the system <NUM> may be referred to as medical instruments. The system <NUM> includes a control system <NUM> and at least one input device <NUM> (sometimes also called a "master tool manipulator," "MTM," "master input device," or "master device") that receives inputs from an operator. The control system <NUM> receives the inputs as electronic signals and generates corresponding movement commands that are implemented by the control system <NUM> to move and control the slave manipulators <NUM> and <NUM> and any associated instruments. The system includes an image capture system <NUM> that captures video images of an operational site or work site, such as a surgical site in medical implementations. The image capture system <NUM> optionally includes an image system slave manipulator and camera instrument, including an imaging sensor such as a CCD or CMOS or ultrasonic or fluoroscopic image sensor, that is moved and operated by the image system slave manipulator to change the field of view provided by the camera instrument.

As described, motion of the primary and external slave manipulators, and of the image capture system manipulator and camera instrument, is under teleoperated control corresponding to movement of part or all of the input device <NUM>. One or more hand-held or hand-grasped input control devices may enable the input device <NUM> to permit an operator (e.g., a surgeon, a clinician, or a physician) to control teleoperational slave manipulators <NUM> and <NUM> and, in some embodiments, to view the work site.

The input device <NUM> may be an untethered remote-control type device that communicates wirelessly. Alternatively, the input device <NUM> may be a control device tethered electrically with one or more cables, tethered mechanically via one or more mechanical links or other structure, or tethered both electrically and mechanically. The input device <NUM> may also be optionally tethered, such as be an input device capable of operating wirelessly and under its own power as well as attached to electrical cables or mechanical structures. As a specific example, the input device <NUM> may be disposed at or as part of an operator's console. An example of an operator console <NUM> (sometimes called a "surgeon console" if the operator console is primarily used by a surgeon) including one or more related input mechanisms, that may provide the input device <NUM>, is included in <FIG>.

<FIG> is a front elevation view of an operator console <NUM>, similar to a "surgeon" console of a da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. At the master control console, an operator views video images captured by the image capture system <NUM>. The video may be displayed in a display system <NUM>, which, as shown, includes two separate viewers or displays that align with the eyes of the operator during operation. The display system <NUM> may provide stereoscopic views from the image capture system <NUM> to enable the operator to perceive the field-of-view of the image capture system <NUM> in three dimensions. The three-dimensional view from the image capture system <NUM> may permit the operator to better interact with the environment shown in the display system <NUM>, such as by manipulating structures in the environment. Other display system configurations may be used as the display system <NUM>, in other embodiments.

The operator console <NUM> includes one or more input device <NUM> (two are shown as input device 204A and input device 204B). The exemplary input devices <NUM> may each include a shaft configured to be positioned by at least two fingers of the operator's hand. The input devices <NUM> may be coupled to console <NUM> by kinematic linkages. As the operator moves a input device <NUM> within the range of motion permitted by the operator console <NUM>, the control system <NUM> interprets the movements as commands that can be used to control the slave manipulators <NUM> and <NUM> to cause them to make corresponding movements. Additionally, the input devices <NUM> include a plurality of linked members, linked together by rotational joints, each joint having at least one encoder. Similarly the shafts <NUM> may be rotationally coupled to the input device <NUM> to provide additional input means. For example, rotating one of the shaft 206A or 206B may cause a corresponding rotational movement of an instrument end effector disposed on or integrated into one of the slave manipulators <NUM> or <NUM>. In some embodiments, rotation of the shafts <NUM> may result in other operations. The shafts <NUM> may include buttons or other input means, in some embodiments.

Embodiments of the input devices <NUM> may include or further include any number of a variety of input devices, such as joysticks, trackballs, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, body motion or presence sensors, and/or the like. To provide an operator with a strong sense of directly controlling manipulators <NUM> and <NUM>, the input devices may be provided with the same degrees of freedom as the associated manipulators <NUM> and <NUM>. In this manner, the input devices provide the operator with telepresence or the perception that the input devices, e.g. the input devices <NUM>, are integral with slave manipulators <NUM> and <NUM>. In some embodiments, the input devices <NUM> may have more or fewer degrees of freedom than the associated slave manipulators <NUM> or <NUM> and still provide the operator with telepresence. In some embodiments, the input devices may optionally be manual input devices which move with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaws, applying an electrical potential to an electrode, delivering a medicinal treatment, and/or the like).

Changes in the position of the input device <NUM> and the orientation of the input devices <NUM> and the shafts <NUM> may be interpreted by the control system <NUM> and used to produce movement commands. The control system <NUM> may implement such commands, causing the slave manipulators <NUM> and <NUM> to move as directed by the operator through the input devices <NUM>. The operator console <NUM> may include all or part of the control system <NUM>, or the control system <NUM> may be located elsewhere. The control system <NUM> includes one or more processing devices and memory for storing programmatic instructions and data that can be executed and processed by the one or more processing devices. The operator console <NUM> optionally includes one or more foot pedals (seven are shown in <FIG>) and other user input mechanism, such as switches on the armrest or on the individual master manipulators grasped with the fingers.

As is discussed herein, in some aspects of the disclosure, the field of view of the camera instrument of the image capture system <NUM> is oriented toward a work site, and a secondary instrument (which may be controlled by a primary platform or controlled by a secondary platform) is oriented so that the direction of insertion is toward the image capture system <NUM>. In one implementation, the secondary instrument being inserted toward the camera instrument means the secondary instrument has an insertion direction with a component antiparallel to a viewing direction (also called "line of sight") along the central axis of the camera instrument's field of view. In another implementation, the secondary instrument being inserted toward the camera instrument means the secondary instrument has a component of a particular magnitude (larger than a threshold magnitude) antiparallel to the viewing direction along the central axis of the camera instrument's field of view; this magnitude may be normalized as a ratio of: (<NUM>) component antiparallel to the central axis of the camera instrument's field of view to (<NUM>) component perpendicular to the central axis of the camera instrument's field of view.

In some implementation, the secondary instrument being inserted toward the camera instrument means the secondary instrument has an insertion direction with an angular difference of a certain amount relative to a viewing direction of the camera instrument. For example, in some implementations, the secondary instrument is identified as having an insertion direction toward the camera instrument when the angular difference is equal or greater than <NUM> degrees, <NUM> degrees, <NUM> degrees, etc..

In some embodiments, the tips of the camera instrument and the secondary instrument may be positioned such that a plane can be envisioned between the tips of these instruments with the plane being perpendicular to a central axis of the field of view of the camera instrument. In terms of this plane, one way to think about the relative insertion directions is that the camera instrument is located on one side of the plane, and the secondary instrument controlled by the secondary arm enters the operational site on the opposite side of the plane.

<FIG> is a simplified side view (not necessarily to scale or complete) of some embodiments of the slave manipulators <NUM> and/or <NUM>, referred to as teleoperational arm assembly 300A. The teleoperational arm assembly 300A is depicted as holding a bundled unit <NUM> of medical devices. The bundled unit <NUM> may then be inserted into the patient P through an incision and through a tool guide. The teleoperational arm assembly 300A is mechanically supported by a base B. Links 302A and 302B (generally, links <NUM>) are coupled together and to the base B through horizontal setup joints 304A and 304B (generally, setup joints <NUM>). The setup joints <NUM> in the depicted example are passive joints that allow manual positioning of the assembly 300A when their brakes are released; in other embodiments, they may be active joints that can be controlled via operator input. As depicted, setup joint 304A allows link 302A to be manually rotated about axis 306A, and setup joint 304B allows link 302B to be manually rotated about axis 306B. Although only two links and two setup joints are shown in this example, more or fewer of each may be used as appropriate in this and other teleoperational arm assemblies in conjunction with the present invention.

The teleoperational arm assembly 300A also includes two active joints and transmission system (which can comprise a number of gears or pulleys or cables or belts or other transmission components) driven by motors. A yaw joint <NUM> allows arm section <NUM> (which may include a linkage of multiple arm segments) to rotate around an axis 206C, and a pitch joint <NUM> allows arm section <NUM> to rotate about an axis perpendicular to that of axis 306C and orthogonal to the plane of the drawing. An interface <NUM> comprises mating parts on the carriage <NUM> and the proximal end of the bundled unit <NUM> such as motor driven gears that actuate movement of instruments 320A and 320B (also called tools 320A and 320B, and generally called instruments <NUM>) and an image capturing system <NUM> through conventional joints, cable and pulley systems. In addition, the bundled unit <NUM> is coupled to a carriage <NUM> on the arm section <NUM> which in turn is coupled to a linear drive mechanism to extend or retract the bundled unit <NUM> along its insertion axis 306D.

Although each of the yaw joint <NUM>, pitch joint <NUM> and motor driven gears in the carriage <NUM> may be controlled by an individual joint or gear controller, the controllers may be controlled by a common master/slave control system so that the devices of the bundled unit <NUM> may be controlled through operator manipulation of its associated control device, such as one or both of the input devices <NUM> of <FIG>.

Referring now to <FIG>, shown therein is a perspective view of a distal end of the bundled unit <NUM>. As illustrated, the bundled unit <NUM> includes instruments <NUM> that are removable medical tools for performing a medical procedure, and a removable image capturing system <NUM> for viewing the procedure at a surgical site within a patient. Each of the instruments <NUM> and image capturing system <NUM> extends through a separate lumen formed in an inner core of the bundled unit <NUM>. Replacement of one or both of the instruments <NUM> during or in preparation for performing a medical procedure may then be accomplished by removing the tool that is no longer needed from its lumen and replacing it with a substitute tool or instrument by inserting the substitute tool in the vacated lumen. Alternatively, if unused lumens are available, an additional tool may be inserted through one of those available lumens without removing any other tools already in place.

The image capturing system <NUM>, as shown in <FIG>, includes a stereoscopic pair of cameras 324A and 324B (and/or a single monocular or binocular camera) for three-dimensional imaging of the surgical site and an illuminating device <NUM> such as a light emitting diode (LED) or a fiber optics bundle carrying light from an external source, to enhance visibility of objects in the captured images. Auxiliary image capturing units, such as an ultrasound probe or another optical camera, may also be provided in available lumens of the bundled unit <NUM> for "seeing" into structures, such as anatomic structures for surgical or diagnostic purposes.

In some embodiments, an overtube <NUM> is also included in the bundled unit <NUM> for protecting its inner core and the medical devices (i.e., surgical tools and image capturing units) inserted therethrough. The overtube <NUM> may be rigid. Alternatively, it may be formed of flexible material or comprise actively and/or passively bendable sections so that the bundled unit <NUM> may conform to the shapes of body lumens as it moves therethrough to a surgical site within a patient.

The instruments <NUM> each have a controllably extendable, rotatable, and bendable shaft to which their respective end effectors 330A and 330B are coupled to by wrist mechanisms 332A and 332B, respectively. In the depicted embodiment, the shaft of the instrument 320B comprises three links coupled by distal joints. The most proximal link (the "proximal first link") of the three links is controllably extendable and retractable along an insertion axis, which may be parallel to the insertion axis 306D of the bundled unit <NUM>), and is controllably rotatable about the insertion axis. The middle second link, on the other hand, may be controllably bendable by a distal joint relative to the proximal first link, and the most distal link (the "distal third link") is coupled to the first and second links and bendable by distal joint so that its bend angle is in an opposite direction as that of the middle second link and consequently, keeps distal third link and the proximal first link in parallel alignment.

The shaft of the instrument 320A is similarly constructed as that of the instrument 320B. Additional details for one example of the wrist mechanisms 332A and 332B are provided in commonly owned <CIT> "Surgical Tool Having Positively Positionable Tendon-Actuated Multi-Disk Wrist Joint.

The image capturing system <NUM> may also have a controllably extendable, rotatable, and bendable shaft that facilitates at least insertion/retraction of the image capturing system <NUM> along its insertion axis (which may be parallel to the insertion axis 306D of the bundled unit <NUM>), and that facilitates pitch motion in order to achieve a sufficient elevation of the image capturing system <NUM> "above" the instruments <NUM> so as to properly view them during a surgical procedure. Additional degrees of freedom, such as roll angular movement of the image capturing system <NUM> about its insertion axis, may also be provided in order to facilitate additional positioning and orientation capabilities for the image capturing system <NUM>. For enhanced maneuverability, the shaft of the image capturing system <NUM> may also be bendable such as the controllably bendable, rotatable, and extendable shaft of the instruments <NUM>.

<FIG> is a perspective view of one embodiment of a teleoperated system 300C, according to some aspects of the present invention. The depicted embodiment of <FIG> shows an example of a system 300C including replaceable medical instruments <NUM> that may contain release mechanisms such as disclosed herein. System 300C, which may, for example, include a da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. , employs multiple surgical instruments <NUM>, each of which is mounted in an instrument mount <NUM> on a manipulator arm <NUM> of a robotic system <NUM>. A sterile barrier (not shown in <FIG>) including a drape and instrument adaptors may be between a patient (not shown) and robotic system <NUM> in medical applications. Instruments <NUM> may vary in structure and purpose but may still be interchangeable, so that a user can select and mount various instruments <NUM> in instrument mounts <NUM> of robotic system <NUM> as needed for a particular medical procedure and can swap instruments <NUM> during a medical procedure to provide desired clinical functions. Each instrument <NUM> generally includes an end effector or distal tool tip, an instrument shaft, and a backend. As described herein, different instruments <NUM> may have many different shapes or sizes and may include forceps, graspers, scalpels, scissors, cautery or ablation tools, or needle drivers to name a few possibilities. Instruments <NUM> having different distal tool tips may be mounted on different manipulator arms <NUM> of robotic system <NUM> and may work cooperatively at the same work site. An endoscopic camera, for example, a stereoscopic camera, can also be mounted on a manipulator arm to provide visual information, particularly images of the work site in which distal tool tips of instruments <NUM> may be operating. The instrument mounts <NUM> of robotic system <NUM> may include actuators such as drive motors that provide mechanical power to actuate mechanical structures in instruments <NUM> via drive couplings that connect the actuators through an interface mechanism to inputs of instruments <NUM>.

<FIG> depicts another embodiment of a teleoperated assembly 300D, which includes multiple manipulator arms supported by a column extending from a base. The assembly 300D includes an automated and motorized setup structure that supports projecting manipulator arms, and may include a base <NUM> that rests on the floor, a telescoping support column <NUM> that is mounted on the base, a telescoping boom <NUM> that extends from the support column <NUM>, and an orienting portion as an orienting platform <NUM>. The assembly 300D also includes support beams <NUM>, and several manipulator arms <NUM> that support surgical tools including portions of the image capture system <NUM> of <FIG>. As shown in <FIG>, manipulator arms 360A, 360B, 360C, 360D are instrument arms that support and move the instruments, such as medical instruments used to manipulate tissue. One of these manipulator arms <NUM> may be designated as a camera arm that supports and moves a camera instrument. The camera system may be a stereo endoscope for capturing stereo images of the surgical site and providing the separate stereo images to the display system <NUM> of <FIG>. Other implementations of the imaging system are discussed herein.

Knowledgeable persons will appreciate that the manipulator arms that support the instruments and the camera may also be supported by the base <NUM> or another base, such as a platform (fixed or moveable) mounted to a ceiling or wall, or in some instances to another piece of equipment in the operating room (e.g., an operating table). Likewise, they will appreciate that two or more physically separate bases may be used (e.g., one base supporting one arm).

<FIG> and <FIG> depict embodiments of manipulator assemblies with different platforms. <FIG> is a perspective view of a computer-assisted teleoperated medical system <NUM> that includes two major components (platforms) that interact with a patient. System <NUM> includes a primary teleoperated platform <NUM> (also called primary teleoperated assembly <NUM>) that includes several medical instruments and a camera instrument like the assembly 300A of <FIG>, system 300C of <FIG>, or assembly 300D of <FIG>. For clarity, <FIG> includes a depiction of system 300C of <FIG> as the platform <NUM>, which includes a single manipulator arm. However, other embodiments may include additional manipulator arms, as in assembly 300D of <FIG>. These manipulator arms may each be individually operated by an associated manipulator, such as one of the input devices <NUM> (<FIG>) to perform medical tasks at a work site in the patient P, positioned on the table T. <CIT> and entitled "Wireless Communication in a Robotic Surgical System," depicts some examples of a primary teleoperated platform.

System <NUM> also includes a secondary teleoperated platform <NUM> that may include one or more instruments, each individually operated by an associated manipulator, such as one of the input devices <NUM> (<FIG>). In use, the secondary teleoperated surgical platform <NUM> controls one or more instruments at the surgical site. The secondary teleoperated platform <NUM> may optionally be referred to as being external to the primary teleoperated platform <NUM>. As shown, each of the primary and secondary platforms is mounted on an individual base, like the base B1 and the base B2. The assemblies or platforms may comprise or be mounted to separately movable carts, in some embodiments, so that they can be independently positioned within an operating room or other work environment. In other optional implementations, however, the secondary platform may be located in various other ways, such as permanently fixed to a mechanical ground (such as a floor, ceiling, or wall), permanently or removably mounted to the operating table, or permanently or removably mounted to the primary platform.

As another example, <FIG> is a perspective view of another embodiment of a computer-assisted teleoperated system <NUM>. A wheeled table T having a table surface is positioned on the table base. The table surface can be used to support a work piece such as the patient P or a non-human work piece in other applications. In the example shown in <FIG>, two slave manipulator arms 382A and 382B are positioned for operation on the patient P. The arms 382A and 382B are supported by arm platforms 384A and 384B that may be removably and repositionably attached to a number of different locations along a table rail <NUM>. During operation, the controllable arms 382A and 382B are driven to position tools and instruments. For example, the instrument 388A may be an imaging system and the instrument 388B may be a pair of forceps for positioning tissue. In some implementations, the controllable arms 382A and 382B are teleoperable and include remotely operable powered joints that, when driven, reposition and reorient the tools.

Both <FIG> and <FIG> show configurations in which a manipulator arm and supported instrument on one platform (a primary instrument on a primary platform) may approach a work site from a substantially different side or direction from which an arm and supported instrument on another platform (a secondary instrument on a secondary platform) approaches the work site. For example, while the camera instrument may define a field of view, other instruments may approach the work site from different sides of a plane associated with the field of view. Such a configuration can pose problems for providing intuitive control to an operator that is able to control the arms approaching the site from different sides using a single console, like the console <NUM> of <FIG>.

<FIG> are diagrammatic view of some system components and associated Cartesian frames of reference. <FIG> shows portions from the operator side of the system. <FIG> shows portions from the instrument side of the system. Frames of reference are also often called "reference frames. " Also, frames of reference as discussed in this application are used to explain the different orientations, locations, rotational differences or changes, and translational differences and changes associated with the techniques discussed herein, along with the associated mappings for determining relative locations and orientations, input, and motion commands. Some embodiments implementing the described techniques need not use the exact same reference frames in calculations or in operating the system. For example, the axes of a reference frame may be set differently from what is described in detail in this disclosure. Also, in implementations that use only orientation, the reference frames may only rotate and not translate. Further, other systems may use alternate coordinate systems, such as spherical coordinate systems.

As shown in <FIG>, the imaging system <NUM> has a field of view <NUM>. A work piece <NUM> and a distal portion of instrument <NUM> are positioned within the field of view <NUM>. In this example, the instrument <NUM> includes a joint <NUM> and an end effector <NUM>. A reference frame <NUM> has been noted for the shaft of the instrument <NUM>, proximal to the joint <NUM>, and with insertion of the instrument <NUM> defined as the +Z direction in reference frame <NUM>. The reference frame <NUM> translates and rotates with the translation and rotation of the instrument <NUM>. A reference frame <NUM> has also been noted for the end effector <NUM>, and moves with the end effector <NUM>. If the end effector <NUM> of the instrument <NUM> does not move with respect to the shaft of the instrument <NUM>, such as with rigid instruments having no degree of freedom between the shaft and the end effector, then the reference frames <NUM> and <NUM> can be combined into a single reference frame that describes the position and orientation of the relevant portion of the instrument <NUM> for modeling and/or control.

The field of view <NUM> itself is associated with a reference frame <NUM>, which translates and rotates with the field of view <NUM>. In this example shown, movement of the field of view <NUM> is possible through movement of a joint or other degree of freedom <NUM> between shaft and distal end of the imaging system <NUM>. Thus, <FIG> also shows a shaft-based reference frame <NUM> that translates and rotates with the proximal body of the imaging system <NUM>. If the field of view <NUM> does not move with respect to the proximal body of the imaging system <NUM>, such as when imaging system has no compliance or other degree of freedom between the field of view and the proximal body, then the reference frames <NUM> and <NUM> can be combined into a single reference frame that describes the position and orientation of the relevant portion of the imaging system <NUM> (such as its field of view <NUM>) for modeling and/or control.

Where the physical dimensions of the instrument <NUM> and imaging system <NUM> are known, and where the configuration of their links and joints (e.g. joint angles for rotational joints, linear position for prismatic joints, etc.), can be assumed or determined (e.g. using position or velocity encoders, orientation or position or shape sensors, direct rotation sensors, motor position sensors, etc.), the kinematic relationship between reference frame <NUM> or <NUM> and a reference frame of any other link in the instrument <NUM> can be determined using well-known kinematic calculations. Likewise, the kinematic relationship between reference frame <NUM> or <NUM> and any other link in the imaging system <NUM>. The end effector <NUM> operates within the field of view <NUM>, so mapping between the reference frames <NUM> of the end effector <NUM> and the reference frame <NUM> of the field of view <NUM> will allow control of the end effector <NUM> in the field of view <NUM>.

<FIG> further shows an operator viewing the display system <NUM> and grasping a master input device <NUM>, as shown in <FIG>. The display system <NUM> presents images captured by the imaging system <NUM> of the field of view <NUM>. In some implementations, the display system <NUM> presents the images as captured, without rotation, panning, zooming, or other manipulation of the images; in this case, a direct, identity mapping can be made between the displayed image and the field of view <NUM>'s reference frame <NUM>. In some implementations, the display system <NUM> presents manipulated versions of the images (e.g. rotated by <NUM> degrees, zoomed and panned to a particular part of the field of view, skewed, etc.); in this case, the mapping between the displayed image and the field of view <NUM>'s reference frame <NUM> takes into account the transformations associated with the manipulation of the image. During operation, the operator views the instrument <NUM>'s position and orientation changes on the display system <NUM>, and establishes a relation between reference frames <NUM> and <NUM>. Thus, in some implementations, the operator reference frame is a separate frame that links the reference frames <NUM> and <NUM>. For example, the operator reference frame may be a reference frame based on the eyes, head, torso, other body part, etc. of the operator. As another example, the operator reference frame may be a reference frame based on a console attached to the master input device <NUM>. In some implementations, the operator reference frame is collocated and identical to the reference frame <NUM>, and is essentially the reference frame <NUM>.

As shown in <FIG>, an image <NUM> of the end effector <NUM> is shown on the display system <NUM>. A reference frame <NUM> is associated with display system <NUM>, and a reference frame <NUM> is associated with master input device <NUM>. As the master input device <NUM> is translated and rotated in 3D space, its associated reference frame <NUM> translates and rotates correspondingly relative to the operator reference frame. These translations and rotations of the master input device <NUM> can be sensed or otherwise determined, and thus its reference frame <NUM> can be translated and rotated appropriately.

The translations and rotations of the master input device <NUM> (and its reference frame <NUM>) are mapped (also "transformed") to the instrument <NUM>'s reference frame to provide a control relationship between the master input device <NUM> and the instrument <NUM> by using well-known kinematic calculations.

For example, where the master input device <NUM> is controlling the end effector <NUM>, the translations and rotations of the master input device <NUM> are mapped to the end effector <NUM> using the reference frames <NUM> and <NUM> and kinematic calculations relating the reference frames <NUM> and <NUM>; as the master input device <NUM>'s position or orientation is changed, moving the position or orientation of the reference frame <NUM>, the end effector <NUM>'s position or orientation is changed correspondingly (moving the position or orientation of the reference frame <NUM>), so that end effector <NUM>'s movement is controlled by to the master input device <NUM>'s movement.

As noted earlier, any number of reference frame definitions can be used for control, as long as the reference frame definitions are internally consistent and sufficient for the control system envisioned. In an implementation, the operator's frame of reference defined in Cartesian coordinates and denoted with the subscript "P", -ZP is defined as motion towards the operator, and +ZP is defined as motion away from the operator. In some instances, the Z-axis may be defined according to the line of sight <NUM> of the field of view <NUM> presented to the operator in the display system <NUM>, with +ZP defined as into the display system <NUM>. This line of sight and associated field of view is determined by configurations such as the pose of the imaging system <NUM> (which can be the image capture system <NUM> of <FIG> or the imaging system mounted on a camera arm as shown in <FIG>), any image manipulation or selection (e.g. presenting only a subset of the image) performed by the system, and the like. In this example, -XP is defined as motion toward the operator's left, and +XP is defined as motion toward the operator's right. And, +YP is defined as motion up relative to the operator, and -YP is defined as motion down. These Cartesian axes may be defined differently in other embodiments. For example, one or more of the pairs +Xp/-Xp, +Yp/-Yp, and +ZP/-ZP may be defined oppositely. As another example, the axes may be rotated from what is shown in <FIG> as discussed herein.

<FIG> shows an arrangement of an endoscopic camera comprising the imaging system <NUM>, two primary platform instruments 504A and 504B (collectively "primary platform instruments <NUM>") controlled by the primary platform, and a secondary platform instrument <NUM> controlled by the secondary platform, all positioned around a work site (also called "operational site"). It should be understood that, although much of the following discussion is with a secondary platform instrument <NUM> attached to a secondary platform physically distinct from the primary platform, the same techniques and methods can also be applied to a secondary instrument attached to the primary platform. Further, the primary and secondary platforms may be physically integrated with each other, or physically separated from and moveable relative to each other.

The imaging system <NUM> may comprise an imaging system instrument configured to be supported by a manipulator. <FIG> of the present disclosure show only two primary platform instruments <NUM> and a single secondary instrument <NUM>. It should be understood that embodiments of the present disclosure may include more or fewer primary instruments and more or fewer secondary instruments. In <FIG>, a frame of reference <NUM> for the primary platform is shown. In the frame of reference <NUM> for the primary platform, +ZC is defined as the same direction as the insertion of the camera's field of view (the camera comprising the imaging system <NUM>) (toward the bottom-right of <FIG>). -ZC is defined as the same direction as the withdrawal of the field of view of this camera (toward the top-left of <FIG>). Motion toward the right in the field of view of the camera is defined as +XC (toward the bottom left of <FIG>), and motion toward the left in the field of view of the camera is defined as -XC (toward the top right of <FIG>). Motion "up" in the field of view of the camera is defined as +YC (out of the page in <FIG>), and motion "down" in the field of view of the camera is defined as -YC (into the page in <FIG>).

In this example, the line of sight of the view provided to the operator by the display system <NUM> (e.g. the line of sight <NUM> of <FIG>) comes from the imaging system <NUM> having a line of sight <NUM>. By mapping the movement of instruments <NUM> in the frame of reference for the primary platform to the movement of the input device <NUM> in the operator's frame of reference, the movement of instruments <NUM> is more intuitively related by the operator with the frame of reference shown in <FIG>. Generally, where the full image captured by the imaging system <NUM> is produced on the display system <NUM> without distortion or transformation (e.g. rotation, stretching, cropping, etc.), it is reasonable to model the mapping between the imaging system <NUM> and the view provided to the operator through the display system <NUM> as identity. Where distortion or transformation does occur, the mapping between the field of view of the imaging system <NUM> and the view provided to the operator through the display system <NUM> may be other than identity, and the difference can be accounted for by proper mapping. For example, the difference can be accounted for when mapping the movement of instruments <NUM> in the frame of reference <NUM> for the primary platform to movement of the input device <NUM> in the operator's frame of reference. Thus, the operator can translationally and rotationally move the input devices <NUM> to command corresponding translational and rotational movements of the instruments <NUM> in a more intuitive manner and more naturally relate them through hand-eye coordination.

<FIG> also shows a plane <NUM> that is orthogonal to the line of sight <NUM>. The plane <NUM> is disposed between the imaging system <NUM> and the secondary platform instrument <NUM>. Envisioning this plane <NUM> in cases where the configuration of the system allows can be helpful in understanding relative insertion positions and directions, such as by examining how manipulator arms and their associated instruments approach the plane <NUM>. As shown, the tips of the instruments <NUM>, the tip of the instrument <NUM>, and the tip of the imaging system <NUM> are positioned such that the tips of the instruments <NUM> and the imaging system <NUM> are positioned on a same side of the plane <NUM>. Meanwhile, the tip of the instrument <NUM> is on an opposite side of the plane <NUM>. Thus, in the example, the instruments <NUM> are on one side of the plane <NUM>, while the instrument <NUM> is on the opposite side of the plane <NUM>.

Referring to <FIG>, a simplified view of the work space including the distal end of the imaging system <NUM>, the distal end of the secondary platform instrument <NUM>, and the distal ends of the two primary platform instruments <NUM> are shown. <FIG> also shows a reference frame <NUM> for the secondary platform instrument. And, for ease of explanation, <FIG> also shows the plane <NUM>. Relative to the reference frame <NUM> for the secondary platform instrument, insertion of the secondary platform instrument toward the surgical site (and toward the plane <NUM>) is defined as +ZI, and withdrawal of the secondary platform instrument away from the surgical site and the plane <NUM> is defined as -ZI. Motion of the secondary platform instrument to the left relative to the field of view of the imaging system <NUM> (toward the right in <FIG>) is defined as +XI (the "right") for the reference frame <NUM>, and motion of the secondary platform instrument to the right relative to the field of view of the imaging system <NUM> (toward the left in <FIG>) is defined as -XI (the "left") for the reference frame <NUM>. And, motion of the secondary platform instrument upwards relative to the field of view of the imaging system <NUM> (out of the page) is defined as +YI ("up") for the reference frame <NUM>, and motion of the secondary platform instrument downwards relative to the field of view of the imaging system <NUM> (into the page) is defined as -YI ("down") for the reference frame <NUM>. The control system <NUM> may determine orientations of the instruments <NUM> and <NUM> (and/or their supporting arms) with respect to the work site using any of these reference frames, including the reference frames <NUM>, <NUM>, or another reference frame. As shown, longitudinal axes 602A, 602B, 602C, and 602D, are associated with the imaging system <NUM>, the primary platform instrument 504B, the secondary platform instrument <NUM>, and the primary platform instrument 504A, respectively. As shown in <FIG> and in other figures, the longitudinal axes <NUM> may be within the XI-ZI plane, the XC-ZC plane, or some other plane parallel to these ordinal planes of the instrument reference frame. In other embodiments, the longitudinal axes <NUM> may not lie within, or be parallel to, the XI-ZI plane or the XC-ZC plane. Thus the axes <NUM>, and the instruments associated with the axes <NUM>, may or may not be coplanar, may or may not be in parallel planes, and may or may not be parallel.

As described herein, the control system <NUM> may determine orientational differences between the instruments present in the work site. For example, the control system may calculate an angular orientation difference θ<NUM> between the axis 506A of the imaging system <NUM> and the axis 602B of the primary support instrument 504B, an angular orientation difference θ<NUM> between the axis 602A and the axis 602C, and an angular orientation difference θ<NUM> between the axis 602A and the axis 602D. As shown, the orientation difference θ<NUM> is about <NUM>°, the orientation difference θ<NUM> is about <NUM>°, and the θ<NUM> is about <NUM>°. Before the control system implements movement commands received from one of the input devices <NUM> of <FIG> to move one of the instruments, the control system <NUM> may determine the orientation difference associated with the instrument. The control system <NUM> may then determine the appropriate commands to implement, based on the orientation difference. In some embodiments, the orientation difference θX between the two axes may be determined as the arccosine of the dot product of the axes (e.g. as the inner product of two vectors, each vector representing a direction along one of the two axes). Other methods may also be utilized.

Referring again to <FIG>, it can be seen that motion of one of the primary platform instruments <NUM> is performed in the primary platform's reference frame. The motion of a master input device <NUM> in the operator's (physician's or other user's) reference frame shown in <FIG> is mapped to the motion of that primary platform instrument <NUM> in the platform's reference frame (defined by the field of view of the imaging system <NUM> and what part of the field of view ). Thus, when a master input device <NUM> is moved to the right, the system causes a coupled primary platform instrument, either of 504A and 504B in this example, to move to the right relative to the primary platform's reference frame <NUM>. To provide intuitive control to the user for the secondary platform instrument <NUM> as it appears in the primary platform's reference frame <NUM>, motion of a master input device <NUM> may be mapped to motion of the secondary platform instrument <NUM> in the secondary platform's reference frame <NUM> in a reversed (inverted) manner. This reversal (inversion) may be implemented in a variety of ways. For example, the mapping may include reversing (inverting) motion in at least one axis of the reference frame <NUM>. As another example, a rotation may be applied in determining instrument <NUM> motion such that the instrument <NUM> motion is reversed. As a further example, at least one axis of the reference frame <NUM> may be reversed when the instrument <NUM> is inserted oppositely to the imaging system <NUM>.

Since the secondary instrument <NUM> may be pointed generally toward the imaging system <NUM> and its field of view, if the master input device 204B controls the secondary platform instrument <NUM>, then a "withdraw" motion on the master input device 204B in a -Z direction relative to the operator frame of reference, in accordance with aspects of this invention, correspond to a "withdraw" motion of the secondary platform instrument <NUM> in the platform reference frame <NUM> and an "insert" motion of the secondary platform instrument <NUM> in the instrument frame of reference <NUM>. In other words, in some aspects of this invention, a withdraw motion on the master input device 204B, which moves the input device 204B toward the operator, is mapped by the system to motion of the secondary platform instrument <NUM> toward the imaging system <NUM>, and toward the operator's eyes if viewing the display system <NUM> of <FIG> (assuming the display system <NUM> is showing the image obtained by the imaging system <NUM>). Similarly, a motion to the right on the master input device 204B (relative to the operator) corresponds to a motion to the left of the secondary platform instrument <NUM> (relative to the instrument <NUM>), so that the distal end of the secondary platform instrument <NUM> appears to move to the right in the imaging system <NUM>'s field of view. Likewise, a motion to the left on the master input device 204B (relative to the operator), in accordance with aspects of this invention, correspond to a motion to the right of the secondary platform instrument <NUM> (relative to the instrument <NUM>), so that the distal end of the secondary platform instrument <NUM> appears to move to the left in the imaging system <NUM>'s field of view. In these aspects of the invention, motion up and down of the master input device 204B still corresponds to motion up and down of the secondary platform instrument <NUM> as seen by the operator in the display system <NUM>.

For the user, the perception when viewing the image of the work site is as if the distal tip of the secondary platform instrument <NUM> is grasped and moved by the input device 204B, and the shaft of the secondary instrument <NUM> follows along. This perception is in contrast to the user's perception when moving a primary platform instrument 504A or 504B, in which the user envisions that the hands of the user grasping the input device <NUM> are the end effectors of the primary platform instrument 504A or 504B.

In some embodiments, the shaft of the secondary instrument <NUM> may comprise one or more compliant portions, or may have one or more joints, such that a distal end of the shaft may move differently from a proximal portion of the shaft. For example, in some implementations, a distal end of the shaft of the secondary platform instrument <NUM> moves in the desired direction while the proximal end of the shaft of the secondary platform instrument <NUM> stays stationary, moves parallel to the desired direction to a different amount, move in a direction opposite direction to the desired direction, etc. In other embodiments, the distal and proximal ends of the second platform instrument <NUM> may move in the same direction.

The following table, Table <NUM>, shows exemplary correspondence in the various reference frames. This table assumes that the reference frames are attached to their respective components, and are not redefined relative to their respective components during operation.

It can be seen that the positive and negative directions in the secondary reference frame are defined to correspond to the desired directions of motion in the camera instrument reference frame (and thus with respect to the operator's perspective as facilitated by the display system <NUM> in various implementations). Optionally, insertion-withdrawal and left-right directions in the secondary platform instrument reference frame can be defined to be the same as the directions of the camera instrument reference frame, in which case the signs would be reversed in Table <NUM> for the insertion-withdrawal and left-right directions in the secondary platform reference frame.

Before implementing a movement command received by one of the input devices <NUM> to control the secondary platform instrument <NUM>, the orientation relative to the imaging system <NUM> is determined by the control system <NUM>. This may be done by referring to stored data, such as an entry in a table associated with the secondary platform instrument <NUM>. For example, some instruments may be identified when coupled to a slave manipulator arm as a type of instrument typically utilized in an oppositional configuration. Accordingly, because of the identity or type of the instrument, the control system <NUM> may determine that a transform (also referred to as "mapping") comprising one or more inversions (for example, like that shown in Table <NUM> or another transform relationship) or other adjustments is to be made to properly relate the reference frame of that instrument to the reference frame of the imaging system <NUM>.

In some implementations, the transforms may be implemented as one or more matrices comprising vectors or scalars. These transforms are applied by the control system <NUM> when determining instrument motion based on input device motion information received from the input devices <NUM>. Depending on the implementation, a series of matrices and other calculations may be applied to invert (or otherwise rotate), translate, or scale and generate control signals for controlling the motion of the instruments. Any appropriate transformation mechanism may be used. For example, a rotational transform may be implemented by a rotation matrix (or series of rotation matrices) representing a difference between the orientations of two reference frames.

The orientation differences between instruments may be determined through any appropriate manner. For example, some implementations use image processing, orientation sensors, kinematic information, shape sensing information, some other technique, or a combination of techniques to determine the orientations or orientation differences between instruments. In all cases, comparison with an orientation difference threshold may be used to select an appropriate transform. For example, as shown in <FIG>, the orientation difference between the imaging system <NUM> and the secondary platform instrument <NUM> is about <NUM>°; in this example, the control system <NUM> applies a transform to invert the mapping of the motion of the secondary platform instrument <NUM> relative to the input device <NUM> along at least one axis. In some implementations, orientation differences below <NUM>° may also be assigned the same transform to invert the mapping of the motion of the secondary platform instrument <NUM> relative to the input device <NUM> along at least one axis. In some implementations. For example, orientation differences in a range of about <NUM>° to about <NUM>° may be assigned the same transform as shown in Table <NUM>. In another embodiment, orientation differences in a range of about <NUM>° to about <NUM>° may be assigned a transform shown in table <NUM>. In some implementations, the orientation difference may be defined as an absolute orientation difference such that an orientation difference of <NUM>° is processed the same as an orientation difference of <NUM>°, or an orientation difference of <NUM>° is processed the same as an orientation difference of <NUM>°.

Additionally, some embodiments of the control system <NUM> may apply different transforms depending on the orientation difference. For example, while the transform shown in Table <NUM> may be applied to any instruments having an orientation difference relative to the imaging system ranging from about <NUM>° to about <NUM>°, a second transform may be applied to any instruments having an orientation difference relative to the imaging system ranging from about <NUM>° to about <NUM>°, and a third transform may be applied to any instruments having an orientation difference relative to the imaging system ranging from about <NUM>° to about <NUM>°.

For example, the second transform may associate the +Z direction of a reference frame defined by the imaging system with the +X direction of a reference frame of the secondary platform instrument. The third transform may associate the +Z direction of the imaging system frame with the -X direction of the reference frame of the secondary platform instrument. In some embodiments, the second and third transforms may only be applied to certain types of instruments, and not applied to other types of instruments.

As described herein, the control system <NUM> may adjust an orientation mapping between input device and instrument, when the instrument has an orientation difference relative to the field of view of the work site that exceeds an adjustment threshold. In some implementations, the control system does not further adjust the orientation mapping. In some implementations, the control system <NUM> may also adjust the mapping to include an ergonomic offset to facilitate movement by the operator. An exemplary offset is shown in <FIG> as θE between the axis 602C of the secondary platform instrument <NUM> and an axis <NUM>, representing the axis of the shaft of the input device <NUM> that is to control the secondary platform instrument <NUM>. The ergonomic offset may provide for a certain degree of difference between the orientation of the input device <NUM> assigned to control a particular secondary platform instrument, like the secondary platform instrument <NUM>, and the orientation of that secondary platform instrument. This may be helpful where a mapping without an ergonomic offset is awkward. For example, some implementations use an ergonomic offset that emulates the input device holding the shaft of the instrument <NUM> like a pencil, and enable a pencil-like interaction with the secondary platform instrument <NUM>. As such, the interactions between the input device <NUM> and the secondary platform instrument <NUM> resemble the off-axis interaction between pencil and paper when writing, for example.

Accordingly, with such an ergonomic offset, the operator may move the input device 204B, for example, to cause movement of the second platform instrument <NUM> without bringing the shaft 206B into perfect axial alignment with the secondary platform instrument <NUM>. An angular ergonomic offset ranging from about <NUM>° to about <NUM>° may be utilized by the control system <NUM>, based on factors such as user selectable adjustments, user preferences, the type of instrument comprising the secondary platform instrument <NUM>, etc. In some implementations, this angular ergonomic offset may be implemented as a rotation matrix added to a series of matrices linking the reference frame of the input device 204B and the instrument <NUM>. In some implementations, this is implemented as a rotation of the secondary platform instrument frame <NUM>.

Embodiments of the present disclosure may include a primary instrument platform and a secondary instrument platform that are connected to separate and movable bases that rests on the floor of the operating room or work environment during operation. Embodiments may additionally include primary and secondary instrument platforms that are separate and movable with respect to each other but that are attached to a common structure such as a rail extending alongside a table T. Additional embodiments may also include primary and secondary instruments that are secured to a common base or platform, but that approach an work site of opposite sides of a plane orthogonal to a line of sight provided by an endoscopic instrument, such that controlling the secondary instruments in a natural-feeling way requires adjusting or applying a transform to received input commands.

Embodiments of the present disclosure provide for many different modes of operation. One of these mode of operation is depicted in the flowchart of <FIG>, which illustrates a method 700A of controlling a robotic system including primary and secondary platforms. Several methods are illustrated in the accompanying figures and described herein. Embodiments of the illustrated methods may include additional operations before, after, in between, or as part of the enumerated operations. Additionally, some embodiments of the illustrated methods may omit one or more of the enumerated operations. Moreover, some embodiments of the illustrated methods may include combinations of operations from the depicted methods. Some embodiments of the methods include computer readable, non-transient, tangible media having instructions thereon that when executed by a processor, such as the processing device of the control system <NUM> cause the control system to perform corresponding operations in the context of the described medical systems.

Some embodiments of the method 700A may begin at an operation <NUM> in which a processing device of a robotic control system compares an orientation of an instrument with an orientation of a field of view of the instrument workspace. For example, the control system <NUM> may compare the orientation of the secondary platform instrument <NUM> relative to the orientation of the imaging system <NUM>, which provides a field of view of the workspace to an operator. In some embodiments, the orientations may be compared by comparing the axis 602A with the axis 602C. In some embodiments, the orientation difference is determined in view of planes associated with the imaging system <NUM> and the secondary platform instrument <NUM>, such that only certain components of the axes are compared. As shown, the orientation difference is approximately <NUM>°.

At operation <NUM>, when the orientation comparison does not meet an orientation criterion set, the control system causes instrument motion in a first direction relative to the instrument frame of reference in response to a movement command. For example, when the orientation difference is less than an orientation threshold, such as <NUM>°, <NUM>°, or <NUM>°, a movement command to move the instrument in an insertion direction may be implemented by the control system <NUM> by moving the instrument in the insertion direction defined relative to the imaging system <NUM>. The orientation difference may be a total orientation difference in 3D space, or it may be an orientation difference when the insertion and viewing directions are projected onto a single plane. For example, the orientation difference may be when the directions are projected onto the X-Y plane, the X-Z plane, or the Y-Z plane defined by the imaging system reference frame or the instrument reference frame.

In some embodiments, the operator may select a desired orientation threshold prior to operating the system <NUM>. Other orientation difference thresholds may be utilized in other embodiments. In some embodiments, the orientation difference threshold may be defined by a cone or pyramid extending away from the imaging system <NUM> and centered on a line of sight thereof. Accordingly, the orientation difference threshold may be independent of the role of the imaging system <NUM> in some embodiments. The orientation criterion set may also include factors associated with particular types of instruments. Accordingly, a secondary platform instrument having an ablation end effector may have a different threshold compared to a secondary platform instrument having forceps as its end effector. Additionally, the orientation criterion set may include information to distinguish which instruments are associated with which platform. For example, in some embodiments the orientation criterion set may include an indication that the instrument <NUM> is a secondary platform instrument, in that the instrument <NUM> is coupled to a system that is characterized by the control system <NUM> as a second platform. Combinations of these criteria may be used in comparing an instrument to a criterion set.

At operation <NUM>, when the orientation comparison meets the orientation criterion set, the control system causes the instrument motion in a second direction relative to the instrument frame of reference in response to the same movement command. The second direction differs from the first direction. For example, the second direction may be opposite the first direction. Other examples of different directions are shown in Table <NUM>.

In this way, the control system <NUM> may adjust or apply a movement command mapping (also called "transform") when the orientation difference indicates that the secondary platform instrument is positioned opposite an intermediary plane, like the plane <NUM> shown in <FIG> and <FIG>, such that it is more natural from the operator's perspective to move the secondary platform instrument in a direction different than would be dictated by the reference frame of the secondary platform instrument.

Another exemplary embodiment is shown as a flowchart of a method 700B in <FIG>. The method 700B describes one of many different modes of operation of the system <NUM>, showing a "following mode" of operation, in which a slave instrument follows the motion of a master manipulator. For example, the secondary platform instrument <NUM> may "follow" the motion of the input device 204B. The secondary platform instrument may also be referred to as an external instrument (EI), because it is external to the primary platform in at least some embodiments. <FIG> is a diagram that shows a general flow for generating control signals for the secondary platform instrument (EI).

And the illustrated embodiment of the method 700B may begin at operation <NUM> in which the control system or processing device of the control system to detect that the system is operating in the following mode. At operation <NUM>, the control system detects movement of a master input device in a master workspace. For example, the control system <NUM> may detect movement of the input device 204B within the area in which the input device 204B can move.

At operation <NUM>, the control system generates master signals that are responsive to the movement of the master input device. For example, the control system <NUM> may generate master signals that correspond to the movement of the input device 204B. These master signals may be, comprise, or cause movement commands. At operation <NUM>, the control system maps the motion in the master workspace to motion in a surgical workspace or other work site. For example, the control system <NUM> may map or translate the movement commands received from the input device 204B to an instrument reference frame associated with the instrument, such as the instrument <NUM> that is to be moved based on the movement commands. As described herein, when the orientation of the instrument to be controlled differs substantially from the field of view provided by the imaging system <NUM>, the transform or transforms may be applied to the movement commands in order to map them to the work site so that they can be appropriately implemented by the instrument <NUM>. Implementing movement of the instrument <NUM> may be the same as implementing movement of a manipulator arm supporting the instrument <NUM>.

In order to map the master workspace to the work site, the method 700B may include checks of individual components of the commanded motion. As shown in <FIG>, an X-axis component check is performed at operation <NUM>, a Y-axis component is checked at operation <NUM>, and a Z-axis component is checked at <NUM>. The operations <NUM>, <NUM>, and <NUM> may be performed serially or in parallel by the control system <NUM>. Because commanded motion generally includes components along more than one of the X-axis, Y-axis, and X-axis, the operations <NUM>, <NUM>, and <NUM> may be performed for any and all commanded motions associated with instruments that can be identified by any criteria, such as the orientation criterion set.

At operations <NUM>, <NUM>, and <NUM>, the transformed control signals may be produced by the control system. As shown, at operation <NUM>, based on the check performed at operation <NUM>, the control system generates slave control signals to move the external instrument or secondary platform instrument <NUM> in an inverted X-axis direction in the frame of reference of the secondary platform instrument <NUM>. Similarly, at operations <NUM> and <NUM>, the control system <NUM> may generate a slave control signals to move the secondary platform instrument in an inverted Y-axis or Z-axis, respectfully, in the frame of reference of the secondary platform instrument <NUM>. In some embodiments of operations <NUM>, and/or <NUM>, and/or <NUM>, one or more of the generated control signals may include non-inverted control signals. As shown, the checks at operations <NUM> and <NUM> may result in the generation of non-inverted signals for the Y-axis (at operation <NUM>) and in the generation of inverted signals for the Z-axis (at operation <NUM>. ) The checks performed at operations <NUM>, <NUM>, and <NUM> may rely on a set of criteria, including orientation criteria, to determine whether an inversion modification should be made. The checks perform at operations <NUM>, <NUM>, and <NUM> may indicate that one or more of the control signal components are not to be inverted or otherwise modified. In such instances, one of more of the operations <NUM>, <NUM>, and <NUM> may not be performed, such that one or more of the X-axis, and/or Y-axis, and/or Z-axis will not be inverted by the control system. In some embodiments, an applied transform may modify a movement by an angle that is less than <NUM>°, i.e., that only partially inverts a movement command.

<FIG> is diagrammatic view 800A of as seen from an operator's perspective via the display system <NUM>, displaying video from the imaging system <NUM>. <FIG> shows two primary platform instruments 504A and 504B, and the distal tip of the secondary platform instrument <NUM>. This perspective is representative of the primary platform camera instrument's perspective, because the user's eye reference frame is mapped to the primary platform camera instrument's reference frame.

<FIG> is a diagrammatic view 800B of the distal ends of the primary platform instruments <NUM> and the distal end of the secondary platform instrument <NUM>. This view is provided for purposes of explanation, as the instruments, in some embodiments, are positioned at or near a work site <NUM> within the body of the patient P. <FIG> also shows an exemplary camera reference frame near the depiction of the imaging system <NUM>, and an exemplary instrument frame near the instrument <NUM>. This exemplary instrument frame is associated with the secondary platform instrument <NUM>.

<FIG> are perspective views that show movement M1 of the distal end of the secondary platform instrument <NUM> along the X-axis in the left-right direction, in both the camera reference frame (which may be considered to be the same as the operator's reference frame, in some embodiments) and the secondary platform reference frame. It should be noted that in some implementations, the display system <NUM> may not be directly in front of the operator as shown in <FIG>. For example, the operator may be viewing a display system off to the right side of the operator, such that the operator's head is turned to the right relative to the operator's torso, arms, and hands. The operator's proprioception may enable the operator to move an input device <NUM> to the right of the operator, to cause an instrument to move to the right in the image shown on the display system. However, in other embodiments, other frames of reference may be used. For example, the operator's movement may be detected and/or processed in relation to the display system (thus using a display system-based frame of reference instead of a operator-body-based frame of reference).

As shown in <FIG>, motion M1 to the right in the user/camera reference frame (<FIG>) corresponds to movement M1 to the left in the secondary platform reference frame (<FIG>). The control system <NUM> may effect the movement M1 by receiving a command from the operator, while the operator sees the view 800A in the display system <NUM> of <FIG>. When the operator is controlling the secondary platform instrument <NUM> and the operator desires to move the distal end of the instrument <NUM> to the operator's right, the operator moves the hand-held manipulator (for example, input device 204B, although the input device 204A may also be used). The control system <NUM> modifies the movement commands, adjusting or applying a transform so that appropriate movement commands can be implemented in the instrument reference frame. To implement the movement commands in a way that produces the operator's desired result, the transform (or "mapping") applied by the control system <NUM> causes the movement M1 to be implemented by moving the instrument <NUM> in the -X direction in the reference frame of the secondary instrument <NUM>.

<FIG> are perspective views that show movement M2 of the distal end of the secondary platform instrument <NUM> along the Y-axis or in the up-down direction, in both the camera reference frame and the secondary platform instrument reference frame. Motion M2 up in the camera reference frame (<FIG>) corresponds to movement up in the secondary platform reference frame (<FIG>). The control system <NUM> may effect the movement M2 by receiving a command from the operator, while the operator sees the view 800A in the display system <NUM> of <FIG>. When the operator is controlling the secondary platform instrument <NUM> and the operator desires to move the distal end of the instrument <NUM> toward the top of the operator's view, the operator moves the hand-held manipulator (for example, input device 204B, although the input device 204A may also be used). The control system <NUM> may modify the movement commands, adjusting or applying a transform so that appropriate movement commands can be implemented in the instrument reference frame. To implement the movement commands in a way that produces the operator's desired result, the transform (or "mapping") applied by the control system <NUM> causes the movement M2 to be implemented by moving the instrument <NUM> in the +Y direction in the reference frame of the secondary instrument <NUM>. In some embodiments, the control system <NUM> may determine that no transform needs to be applied to the Y directional component of the movement M2. However, if the movement M2 is not exclusively in the plane of the Y-axis, the control system <NUM> may apply or adjust a transform to the component of the movement M2 that is not in the Y-axis plane.

<FIG> are perspective views that show movement M3 of the secondary platform instrument <NUM> along the Z-axis or in the insertion-withdrawal direction, in both the camera reference frame and the secondary platform reference frame. Insertion motion in the camera reference frame (<FIG>) corresponds to withdrawal motion the secondary platform reference frame (<FIG>). The control system <NUM> may effect the movement M3 by receiving a command from the operator, while the operator sees the view 800A in the display system <NUM> of <FIG>. When the operator is controlling the secondary platform instrument <NUM> and the operator desires to move the distal end of the instrument <NUM> to away from the operator's view, the operator moves the hand-held manipulator (for example, input device 204B, although the input device 204A may also be used) away from the operator's torso, or in the +Z direction in the operator's reference frame, which corresponds to the camera reference frame. The control system <NUM> modifies the movement commands, adjusting or applying a transform so that appropriate movement commands can be implemented in the instrument reference frame. To implement the movement commands in a way that produces the operator's desired result, the transform (or "mapping") applied by the control system <NUM> causes the movement M3 to be implemented by moving the instrument <NUM> in the -Z direction in the reference frame of the secondary instrument <NUM>.

Various forms of the following modes of <FIG>, and shown in other figures, are optionally used in various primary platforms. For example, in an "adjust mode" motion of the master input device 204A or 204B moves the imaging system <NUM> so that the distal end of a camera instrument comprising the imaging system <NUM> correspondingly moves in the camera reference frame and relative to the tissue at the work site <NUM>. In the adjust or adjustment mode motions of the primary instruments are coupled to the camera motion so that the distal ends of the primary platform instruments 504A and 504B are anchored in the camera reference frame and do not move relative to the tissue at the work site. While the distal ends are anchored in the camera reference frame and do not move relative to the tissue at the work site, the proximal ends of the primary platform instruments 504A and 504B may move in order to compensate or adjust for the movement of the proximal end of the camera instrument comprising the imaging system <NUM>. In the adjust mode, the motion of the secondary platform instrument is optionally also coupled to the camera motion so that the distal end of the secondary platform instrument is anchored in the camera reference frame and does not move relative to the tissue at the work site. Anchoring the instruments in space relative to the tissue assists safety, since the instrument distal ends do not move relative to the tissue and so may be grasping tissue or will not be inadvertently moved into contact with tissue. The adjust mode may be used for small changes of camera position or orientation.

<FIG> is a flow diagram that shows a general operation of the secondary platform instrument in the adjust mode. The method <NUM> shown in <FIG> may begin at operation <NUM> in which the control system detects that movement commands are to be implemented according to an adjust mode. When the control system <NUM> detects that the input device <NUM> is moving or has moved (operation <NUM>), the control system may generate master control signals responsive to that movement. These master control signals may be expressed in a frame of reference associated with the input device <NUM>. At operation <NUM>, the control system may map the master workspace to the surgical or other workspace. For example, the control system <NUM> may generate or identify a transform that can be applied to the master control signals, which are expressed in the frame of reference of the manipulator, so that they can be appropriately applied to a selected slave manipulator, like the instrument <NUM> or the arm that supports and positions the instrument <NUM>.

At operation <NUM>, the control system may generate slave control signals to move the primary slave manipulator and/or the secondary or external slave manipulator in the surgical or other workspace without changing the position orientation of the end effectors of the slave manipulators. For example, the control system <NUM> may move the proximal ends of the instruments 504A, 504B, and/or <NUM> in a way that minimizes or eliminates motion at the distal end of these instruments so that the end effectors <NUM> maintain their position and orientation within the work site. This may be done to permit the imaging system <NUM> to be repositioned or to facilitate a desired range of motion of one or more of the instruments present in the work site.

Referring now to <FIG>, shown therein is another exemplary mode of operation as a method <NUM> for a "relocate mode" motion of the master manipulator. During operation of the system <NUM>, the operator may move the primary platform camera instrument in a mode in which the distal end of the camera instrument correspondingly moves in the camera reference frame and relative to the tissue at the work site. In the relocate mode, motions of the other primary instruments may be coupled to the primary platform camera motion so that the distal ends of the primary platform instruments move together with the camera instrument in the camera reference frame. Thus, the camera instrument and the primary platform instrument distal ends all move together relative to tissue at the surgical or other work site. In the relocate mode, the motion of the secondary platform instrument may optionally be coupled to the camera motion so that the distal end of the secondary platform instrument moves in the camera reference frame and moves relative to the tissue at the work site. Similarly, the control system <NUM> applies appropriate transforms to secondary platform instruments so that the secondary platform instruments are also moved relative to the surgical or other work site in a way that maintains the secondary platform instrument end effectors in a fixed relationship to the primary platform camera motion. The relocate mode may be used for large changes of camera position or orientation. <FIG> is a flow diagram of a method <NUM> for general operation of the secondary platform instrument in the relocate mode.

Embodiments of the method <NUM> may begin at an operation <NUM> in which the control system detects that the system is in a relocate mode. At operation <NUM>, the computing system may detect movement of a master input device in a master workspace. For example, the control system <NUM> may detect movement of one of the input devices <NUM> within the area defined by the constraint range of motion of the input devices <NUM>. At operation <NUM> the control system may generate master signals responsive to and corresponding to the movement of the master input device. At operation <NUM>, the control system may map the master workspace to the surgical or other workspace. As described herein, mapping input device motion in the master workspace to instrument motion in the work workspace may include applying or adjusting a transform to the master control signals to translate them from the master workspace, such as the operator's reference frame or camera reference frame, to the instrument reference frame. In doing so, the control system may, at operation <NUM> generate corresponding slave control signals to move a slave manipulator or manipulators in the surgical or other workspace while constraining motion of the secondary or external platform instrument relative to the imaging system and/or the primary instruments. In this way, when the operator moves the imaging system <NUM>, secondary platform instrument <NUM> may be moved within the workspace so that it remains stationary relative to the imaging system <NUM>. The transform (or "mapping") applied to the master control signals is different for the secondary platform instrument <NUM> than for the primary platform instruments.

In operating modes in which motion of the secondary platform instrument <NUM> is coupled to motion of the primary platform camera, motion constraints of the secondary platform instrument <NUM> (e.g., physical joint limits, position along the Z-axis relative to a cannula through which the secondary platform instrument is inserted that may constrain motion of the instrument, and the like) optionally may be used to establish motion limits of the primary platform camera. Such camera instrument motion limits optionally may be absolute, or optionally a user may override a motion limit such as by moving a master manipulator past a virtual, haptic wall (a virtual barrier implemented by force feedback impeding movement past the barrier).

<FIG> is a flow diagram that shows a method <NUM> for a hybrid camera control and secondary platform instrument control mode. In an example hybrid mode, a first master manipulator controls motion of the primary platform camera instrument, and a second master controls motion of the secondary platform instrument. Motion of one of the camera instrument or the secondary platform instrument is prohibited if motion of the other is occurring. In such an operating mode, the control system <NUM> ensures that the correct mapping of master input device motions in the operator (user) reference frame and imaging system (camera) reference frames occurs for either the camera instrument or the secondary platform instrument.

At operation <NUM>, the control system determines that the hybrid control mode is in effect. The determination of the control mode may be obtained by querying state information of the control system <NUM>. At operation <NUM>, the control system <NUM> may detect movement of a first master input device in a master workspace such as the input device 204A. Upon detection of the movement of the input device 204A, the control system may lock out a detected movement of a second master input device, such as the input device 204B, at operation <NUM>. At operation <NUM>, the control system may generate master signals responsive to the movement of the first master input device and then map the master workspace to the surgical or other workspace in order to implement the movement commands included in the master signals, at operation <NUM>. At operation <NUM>, the control system may generate camera control signals to move the camera in the surgical or other workspace. For example, the control system <NUM> may move the imaging system <NUM> according to the reference frame of the imaging system <NUM>. At operation <NUM>, the control system may detect movement of the second master input device in the master workspace. For example, the control system <NUM> may detect movement of the input device 204B. In order to prevent any unwanted interaction between the instrument being controlled by the input device 204B and the instrument being controlled by the input device 204A, the control system <NUM> may lock out any subsequently detected movement of the first master input device, e.g. the input device 204A, at operation <NUM>. At operation <NUM>, the control system may generate master signals responsive to the movement of the second master input device. The control system may map the master workspace to the surgical or other workspace at operation <NUM> in order to appropriately generate slave control signals to move the secondary platform instrument in the surgical or other workspace at operation <NUM>.

<FIG> is a flow diagram of a method <NUM> for a collision avoidance aspect of motion of the secondary platform instrument. If the control system <NUM> determines that movement of the secondary platform instrument <NUM> will result in a collision with a primary platform instrument, either 504A or 504B, the control system may optionally modify the control of one or both of the secondary and primary platform instruments so that the predicted collision will not occur as the secondary instrument moves. For example, when controlling the secondary platform instrument, the user may feel haptic sensation in the master manipulator to "steer around" a potential collision with a primary platform instrument. As another example, the user may observe a primary platform instrument move slightly during motion of the secondary platform instrument to visually communicate the potential collision to the operator. Collision avoidance aspects may be implemented in some or all of various operating modes, such as following, adjust, and relocate modes described above.

Some embodiments of the method <NUM> may begin at operation <NUM> in which the control system detects movement of a master input device in a master workspace, the movement being in a first direction. At operation <NUM>, the control system <NUM> may generate master signals responsive to the movement of the master and put device, which may be the input device 204B as shown in <FIG>. At operation <NUM>, the control system <NUM> may map the master workspace to the surgical or other workspace, identifying relationships between the operator reference frame or camera reference frame and the instrument reference frame of the instrument being controlled by the master input device. At an operation <NUM>, the computing system generate control signals to move the external slave manipulator or secondary platform instrument in the work site. At operation <NUM>, the control system <NUM> uses kinematic or non-kinematic sensor information associated with each of the slave manipulator and their associated instruments to determine whether the movement will cause the secondary platform instrument to collide with one of the primary platform instruments. If so, the control system <NUM> modifies the slave control signals at operation <NUM> either or both of the secondary platform instrument and the primary platform instrument in order to avoid the collision. In some embodiments, the control system <NUM> may modify the slave control signals to cause one slave manipulator to move around the other slave manipulator in order to avoid the collision. This may cause the moving slave manipulator to be moved to the desired location, but via a path that is different than the path dictated by the commanded motion received from the input device 204B. Additionally or alternatively, some embodiments of the operation <NUM> may include moving the slave manipulator that is not associated with the input device 204B, in order to avoid the collision or to alert the operator to enable the operator to avoid the collision. In some implementations, the control system <NUM> may verify a state of the slave manipulator to avoid any additional complications. For example, if the primary slave manipulator has forceps as an end effector, the forceps being closed to grasp tissue, the control system <NUM> may determine that the forceps or closed and/or grasping tissue and prevent responsive movement of the primary slave manipulator that might otherwise be implementable to avoid the collision. The state information may alternatively be able to qualify the movement of the primary slave manipulator, limiting the movement thereof to a wiggle or other motion that is visible to the operator but that will not result in damage to the tissue being grasped.

Returning to the method <NUM>, if the external slave manipulator is determined not to be on a path resulting in collision with the primary manipulator at the determination of operation <NUM>, the control system may move the external slave manipulator and/or primary slave manipulator in the surgical or other workspace, at operation <NUM>, as directed by the movement commands produced by the operator interacting with the input devices <NUM>.

As shown in <FIG>, aspects of the current invention may be implemented in surgical or other systems that include two or more operator's consoles, like the console <NUM>, so that a first operator (user) viewing the work site controls a first master manipulator and a second operator (user) viewing the work site controls a second master manipulator. For example, the first operator may control the primary platform instruments, and the second operator may control the secondary platform instrument. Both the first and second operators view the same video image from the primary platform camera instrument.

<FIG> includes a method <NUM> for controlling robotic manipulators. And the illustrated embodiment of the method <NUM> may begin at operation <NUM> in which the control system or processing device of the control system to detect that the system is operating in the following mode and is communicating with a primary master controller or console and a secondary master controller or console. In some implementations, the control system may verify that both of the consoles have the same state or mode in effect. At operation <NUM>, the control system provides the camera view to the secondary console in addition to the first console. At operation <NUM>, the control system <NUM> may use image analysis and/or kinematic information to determine whether the external instrument or secondary platform instrument is visible in the camera view. When the secondary platform instrument is not visible in the camera view, the control system may abort the following mode at operation <NUM>. When the secondary platform instrument is visible in the camera view, the control system may detect movement have a secondary master input device in a secondary master workspace, at operation <NUM>. For example, the control system <NUM> may detect movement of an input device <NUM> of the second console, like the console <NUM> of <FIG>. The second console may include a master workspace defined by the limits of the motion of the input devices <NUM>.

At operation <NUM>, the control system generates secondary master signals that are responsive to the movement of the secondary master input device. For example, the control system <NUM> may generate master signals that correspond to the movement of the input device 204B of the second console. These master signals may be movement commands. At operation <NUM>, the control system maps the secondary master workspace to the surgical or other workspace. For example, the control system <NUM> may map or translate the movement commands received from the input device 204B of the second console to an instrument reference frame associated with the instrument, such as the instrument <NUM> that is to be moved based on the movement commands. As described herein, when the orientation of the instrument to be controlled differs substantially from the field of view provided by the imaging system <NUM>, the transform or transforms may be applied to the movement commands in order to map them to the work site so that they can be appropriately implemented by the instrument <NUM>. Implementing movement of the instrument <NUM> may be the same as implementing movement of a manipulator arm supporting the instrument <NUM>.

In order to map the master workspace to the work site, the method <NUM> may include checks of individual components of the commanded motion. As shown in <FIG>, an X-axis component check is performed at operation <NUM>, a Y-axis component is checked at operation <NUM>, and a Z-axis component is checked at <NUM>. The operations <NUM>, <NUM>, and <NUM> may be performed serially or in parallel by the control system <NUM>. Because commanded motion generally includes components along more than one of the X-axis, Y-axis, and X-axis, the operations <NUM>, <NUM>, and <NUM> may be performed for any and all commanded motions associated with instruments that can be identified by any criteria, such as the orientation criterion set.

At operations <NUM>, <NUM>, and <NUM>, the transformed control signals may be produced by the control system. The transformed control signals may be based on the checks performed at <NUM>, <NUM>, and <NUM>. As shown, at operation <NUM>, the control system generates slave control signals to move the external instrument or secondary platform instrument <NUM> in an inverted X-axis direction in the frame of reference of the secondary platform instrument <NUM>. Also as shown, the checks at operations <NUM> and <NUM> may result in the generation of non-inverted signals for the Y-axis (at operation <NUM>) and in the generation of inverted signals for the Z-axis (at operation <NUM>. ) Similarly, at operations <NUM> and <NUM>, the control system <NUM> may generate a slave control signals to move the secondary platform instrument in an inverted Y-axis or Z-axis, respectfully, in the frame of reference of the secondary platform instrument <NUM>. The checks performed at operations <NUM>, <NUM>, and <NUM> may rely on a set of criteria, including orientation criteria, to determine whether an inversion modification should be made. For example, the check performed at operation <NUM> may determine that operation <NUM> should not be performed, i.e. that the slave control signals moving the instrument in the Y-axis direction should not be inverted or otherwise altered. In some embodiments, an applied transform may modify a movement by an angle that is less than <NUM>°, i.e., that only partially inverts a movement command.

In another implementation, the secondary platform may include a secondary platform camera instrument, such that two camera instruments are present in a work site. In this situation, the relation of the secondary platform to the primary platform is the same as the relation of the primary platform to the secondary platform as described above. Thus the control system <NUM> maps (or control systems in each of the primary and secondary platforms work together to map) the correct camera reference frame to the correct operator console, and both the first and second operators control the primary and secondary platform instruments as described above, depending on how the primary and secondary platform instruments are coupled to follow motions of the primary and secondary platform master manipulators.

In still another optional implementation, operation in a following mode or a coupled motion mode is restricted on the condition that an instrument or full instrument cluster is present in a camera field of view. Thus the second operator would be prevented from operating an instrument (e.g., operating the second master manipulator to control the secondary platform instrument) if the instrument is not visible in the video image the second operator is viewing. In another optional implementation, operation in a following or coupled motion mode is allowed if the instrument or full instrument cluster is not visible in the video image the second operator is viewing, but a warning is presented to the second operator that informs that the instrument or portion of the full instrument cluster is not visible.

In implementations that include a second operator console, the adjust and relocate modes of operation as described above may be implemented, as shown in the general operating flow in <FIG>. At operation <NUM>, the control system detects from state information that the primary master controller or console is in an adjust mode or a relocate mode as described herein. At operation <NUM>, the control system detects that the secondary master controller or console is in a following mode as described herein. At operation <NUM>, the control system detects movement of a secondary master input device in a secondary master workspace. For example, the control system <NUM> may detect movement of the input device <NUM> of the second console. At operation <NUM>, the control system generates secondary master signals responsive to the movement of the secondary master input device. At an operation <NUM>, the control system maps the secondary master workspace to the surgical workspace, interventional site, or other work site as described herein. At the operation <NUM>, the control system generates slave control signals to move the external slave manipulator/secondary platform instrument in the work site. The slave control signals may be produced by adding or multiplying a transform to the master signals, so that they can be implemented appropriately on the secondary platform instrument.

Embodiments of the present disclosure may facilitate control of secondary platform arms/instruments that have or are likely to have an orientation will difference such that direct implementation of received commands from an operator are likely to result in movements the opposite of those desired by the operator. Embodiments of the present disclosure are applicable to systems having separate bases, systems having different platforms but a common base, and to systems having instruments approaching work site from substantially opposite sides of the plane.

Claim 1:
A teleoperational system (<NUM>) comprising:
a first manipulator (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) configured to support an instrument (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) moveable within an instrument workspace, the instrument (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) having an instrument frame of reference (<NUM>-<NUM>, <NUM>);
an operator input device (<NUM>, <NUM>) configured to receive a movement command from an operator; and
a control system (<NUM>) configured to implement the movement command by:
comparing an orientation of the instrument (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) with an orientation of a central axis of a camera (<NUM>) providing a field of view (<NUM>) of the instrument workspace to determine an orientation difference between the orientation of the instrument (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) and the orientation of the central axis of the camera (<NUM>),
when the orientation difference is less than an orientation difference threshold t, causing instrument motion in a first direction relative to the instrument frame of reference (<NUM>-<NUM>, <NUM>) in response to the movement command, and
when the orientation difference is greater than or equal to the orientation difference threshold, causing instrument motion in a second direction relative to the instrument frame of reference (<NUM>-<NUM>, <NUM>) in response to the movement command, wherein the second direction differs from the first direction.