Patent Description:
Robotic manipulators can be operated to control motion of instruments in a workspace. For example, such manipulators can be used to perform non-medical and medical procedures. As a specific example, teleoperated manipulators can be used to perform minimally invasive surgical procedures. An operator can control the manipulators using a user control system, e.g., connected wirelessly or via a wired connection to the teleoperated manipulators. The user control system can include multiple user input devices such that each of the teleoperated manipulators can be controlled by a distinct user input device of the user control system. The operator can thus independently control each of the teleoperated manipulators using the user input devices.

<CIT> describes robotic surgical systems which allow selectable independent repositioning of an input handle of a master controller and/or a surgical end effector without corresponding movement of the other. In some cases, independent repositioning is limited to translational degrees of freedom. In other cases, the system provides an input device adjacent a manipulator supporting the surgical instrument so that an assistant can reposition the instrument at the patient's side.

<CIT> describes a hyperdexterous surgical system which includes one or more surgical arms coupleable to a fixture and configured to support one or more surgical tools. The system can include an electronic control system configured to communicate electronically with the robotic surgical tools. The control system can electronically control the operation of the surgical tools. The system can include portable handheld controllers actuatable by a surgeon to communicate one or more control signals to the one or more surgical tools via the electronic control system to operate the surgical tools. The portable handheld controllers can provide said control signals from a plurality of locations of an operating arena, allowing a surgeon to be mobile during a surgical procedure and to remotely operate the surgical tools from different locations of the operating arena.

<CIT> describes a system that may include and/or involve a first device, a second device, and logic to effect pairing of the first and second devices upon detection of physical contact between the devices.

<CIT> describes a robotized surgery system comprising at least one robot arm which acts under the control of a control console intended for the surgeon. The console comprises an eye tracking system for detecting the direction of the surgeon's gaze and for entering commands depending on the directions of the gaze detected. The console comprises a screen with at least one zone for viewing the operating field and, among the commands which can be performed depending on the gaze directions, there is an automatic command for enabling or disabling the movement of the robot arm when a gaze direction which falls within or outside of said zone of the screen is detected. Other relevant prior art related to the association between two devices based on motion can be found in <CIT> and <CIT>.

The present invention provides computer-assisted medical system, and a non-transitory computer readable media as defined in the appended independent claims. Optional features are defined in the appended dependent claims. The methods mentioned below are not part of the claimed invention.

In one aspect, a computer-assisted medical system includes teleoperated manipulators, a user input system operable to generate signals to control the manipulators, and a controller configured to execute instructions to perform operations. A portion of the user input system is movable relative to the plurality of manipulators. The operations include, in a pairing mode, associating a first manipulator of the plurality of manipulators with the portion of the user input system based on movement of the portion of the user input system relative to the first manipulator, and, in a following mode, controlling motion of the first manipulator in accordance with an indication generated by the user input system in response to operation of the portion of the user input system by a user.

In another aspect, a method of operating a computer-assisted medical system including a plurality of teleoperated manipulators is featured. The method includes, in a pairing mode, associating a first manipulator of the plurality of manipulators with a portion of a user input system based on movement of the portion of the user input system relative to the first manipulator, and, in a following mode, controlling motion of the first manipulator in accordance with an indication generated by the user input system in response to operation of the portion of the user input system by a user.

In another aspect, a computer-assisted medical system includes teleoperated manipulators, an optical motion detection system operable to generate signals to control the plurality of manipulators, and a controller configured to execute instructions to perform operations. The optical motion detection system is configured to detect motion of hands of a user. The operations include, in a pairing mode, associating a first manipulator of the plurality of manipulators with a first hand of the user based on movement of the first hand relative to the first manipulator, and, in a following mode, controlling motion of the first manipulator in accordance with an indication generated by the optical motion detection system in response to movement of the first hand of the user.

Advantages of the foregoing may include, but are not limited to, those described below and herein elsewhere. For example, associations between portions of the user input system (e.g. one or more user input devices) and the teleoperated manipulators can be formed in an intuitive manner for the operator. Rather than having to interact with lists and information presented on a display that do not provide the operator with an intuitive sense of relative poses of the teleoperated manipulators, an operator can initiate association between a user input device and a particular teleoperated manipulator through physical manipulation of the user input device in an intuitive manner. During the pairing mode, the operator can physically move the user input device relative to the teleoperated manipulators to select a manipulator of the teleoperated manipulators to associate with the user input device. For example, the user input device can be moved toward or to a position proximate a manipulator to which the user input device is to be associated. This process provides an intuitive way for the operator to associate the user input device.

Human-detectable feedback can be provided during the pairing mode so that the operator can be kept apprised of states and processes of devices, e.g., the user input devices and the manipulators. For example, the controller can generate feedback indicative of association states of the portions of the user input system or association states of the manipulators. Based on the feedback, the operator can initiate association processes for the portions of the user input system that have not already been associated, the manipulators that have not already been associated, or both. In some implementations, the controller can generate feedback indicative of a proposed association prior to finalizing an association between a user input device and a manipulator. This can enable the operator to make adjustments to a proposed association, thereby providing the operator with greater control during the association process. In some implementations, human-detectable feedback can be continued or newly provided after an association has been made, and indicate the portion of the user input system that is associated with a particular manipulator, and vice versa. Further, the controller can disassociate a user input device or a manipulator in response to user input or a system event.

Although some of the examples described herein often refer to medical procedures and medical instruments, the techniques disclosed also apply to non-medical procedures and non-medical instruments. For example, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, manipulation of non-tissue work pieces, and/or cosmetic improvements. Other non-surgical applications include use on tissue removed from human or animal anatomies (without return to a human or animal anatomy) or on human or animal cadavers.

Other potential features, aspects, and advantages will become apparent from the description, the drawings, and the claims.

Referring to <FIG>, a system <NUM> in an environment <NUM> includes a manipulator system <NUM> including manipulators 102a, 102b, 102c, 102d (collectively referred to as manipulators <NUM>) that can be teleoperated by an operator <NUM>. The manipulators <NUM> are termed "teleoperated manipulators" because they that can be teleoperated by an operator <NUM> through a physically separate user input system <NUM>. In some implementations, the manipulators <NUM> can also be controlled directly through manual interaction with the manipulators <NUM> themselves. Thus, "teleoperated manipulators" as used in this application include manipulators that can be controlled only through teleoperation, and manipulators that can be controlled through teleoperation and through direct manual control. The manipulators <NUM> include movable portions that can support instruments (not shown), e.g., surgical and medical instruments. The movable portions, for example, correspond to distal ends 112a, 112b, 112c, 112d of the manipulators <NUM>. When the system <NUM> is operated in a following mode, the operator <NUM> can operate a user input system <NUM> to control motion of the manipulators <NUM>.

The operator can view a display system <NUM> that presents imagery representing the instruments mounted on the manipulators <NUM> while the manipulators <NUM> are being controlled by the operator <NUM>. For example, an instrument including an image capture device such as a camera is mounted to one of the manipulators <NUM>. The image capture device generates imagery of the distal ends of other instruments mounted to the other manipulators <NUM> so that the operator <NUM> can monitor poses of the distal ends of the instruments during a surgical operation.

The user input system <NUM> is connected to the manipulators <NUM>, e.g., wirelessly or using a wired connected. The user input system <NUM> includes multiple distinct portions movable relative to the manipulators <NUM> and operable for controlling operations of the manipulators <NUM>. For example, the user input system <NUM> includes user input devices 108a, 108b (collectively referred to as user input devices <NUM>) movable relative to the manipulators <NUM>. The user input system <NUM> can include other user input devices, e.g., keyboards, touchscreens, buttons, foot pedals, etc., in addition to the user input devices <NUM> used to control movement of the manipulators <NUM> in the following mode. These other user input devices can be used to control the display system <NUM> and otherwise control operations of the system <NUM>.

As described herein, in response to movement of the user input devices <NUM> in a pairing mode, a controller <NUM> of the system <NUM> can associate a user input device of the user input system <NUM> with a corresponding one of the manipulators <NUM> during an association process. When associated, the user input device can be operated to control the corresponding manipulator in the following mode to perform an operation, e.g., a medical operation, a surgical operation, a diagnostic operation, etc..

<FIG> shows an example of the manipulator system <NUM>. For simplicity, only the manipulators 102a, 102b of the manipulator system <NUM> are shown. In some implementations, the manipulator system <NUM> includes a single manipulators or includes three or more manipulators, e.g., four manipulators 102a, 102b, 102c, 102d as depicted in <FIG>. In addition, although <FIG> is described with respect to the manipulators 102a, 102b, the manipulators 102c, 102d of <FIG> can include features similar to those presented with respect to the manipulators 102a, 102b. The manipulators 102a, 102b, 102c, 102d may differ in that different instruments may be mounted to the manipulators 102a, 102b, 102c, 102d, and in that the manipulators 102a, 102b, 102c, <NUM> may be supported by an operating table <NUM> at different locations along the operating table <NUM>.

The manipulators 102a, 102b include portions movable about a workspace <NUM>. For example, these portions can correspond to distal ends 112a, 112b of the manipulators 102a, 102b that are movable about the workspace <NUM>. The distal ends 112a, 112b support instruments 116a, 116b such that the instruments 116a, 116b can be moved about the workspace when the distal ends 112a, 112b are moved about the workspace <NUM>. In some implementations, actuation modules 117a, 117b are supportable at the distal ends 112a, 112b of the manipulators 102a, 102b. The actuation modules 117a, 117b are removably mounted to the distal ends 112a, 112b of the manipulators 102a, 102b and include one or more actuators operable to generate insertion and roll motions of the instruments 116a, 116b. The instruments 116a, 116b are insertable through the actuation modules 117a, 117b such that the instruments 116a, 116b are attached to the actuation modules 117a, 117b, which in turn are attached to the distal ends 112a, 112b of the manipulators 102a, 102b.

The manipulators 102a, 102b include powered joints 118a, 118b that can be driven to move the distal ends 112a, 112b of the manipulators 102a, 102b about the workspace <NUM>. Each of the manipulators 102a, 102b includes multiple powered joints 118a, 118b that enable motion of the distal ends 112a, 112b in multiple degrees of freedom, e.g., pitch, yaw, and roll motions of the distal ends 112a, 112b of the manipulators 102a, 102b. The instruments and manipulators described herein can have one or more degrees of freedom that vary in implementations. For example, the one or more degrees of freedom include one or more of a yaw motion of the distal portion of the manipulator, a pitch motion of the distal portion of the manipulator, an insertion motion of the instrument supported by the manipulator, a roll motion of the instrument, a yaw motion of the end effector of the instrument, a wrist motion of an end effector of the instrument, or a jaw or grip motion of the end effector of the instrument.

The system <NUM> is a computer-assisted system. For example, the controller <NUM> can control operation of the system <NUM> or operations of portions of the system <NUM>. In some examples, the controller <NUM> can control operation of the actuators of the powered joints 118a, 118b. In this regard, the distal ends 112a, 112b of the manipulators 102a, 102b, and hence the instruments 116a, 116b, can be moved about the workspace <NUM> when the user input devices <NUM> (shown in <FIG>) are operated by the operator <NUM>. In a following mode, a follower of a manipulator moves in response to movement of a leader. The movement of the follower can emulate the movement of the leader. For a particular manipulator for example, the leader can be one or more of the user input devices <NUM>, and the follower can be one or more components of the manipulator. The follower can be an end effector of the manipulator, a remote center of the manipulator, or some other component of the manipulator. In some examples, in the following mode, the distal ends 112a, 112b are the followers. For example, actuators of the powered joints 118a, 118b can be controlled to generate motion of links of the manipulators 102a, 102b about the powered joints 118a, 118b, thereby repositioning the distal ends 112a, 112b of the manipulators 102a, 102b. The motions of the distal ends 112a, 112b emulate the motions of the user input devices <NUM>. In other examples, the motion of the user input devices <NUM> in the following mode can cause an instrument mounted to the distal end 112a or 112b to be ejected from the distal end 112a or 112b.

Referring to both <FIG> and <FIG>, in some implementations, the system <NUM> is a medical system to perform a medical procedure on a patient <NUM>. For example, the system <NUM> is a diagnostic system that can be used to perform diagnostics on the patient <NUM>. Alternatively or additionally, the system <NUM> is a surgical system that can be used to perform a surgical operation on the patient <NUM>.

A variety of alternative computer-assisted teleoperated instruments 116a, 116b can be used. For example, the teleoperated instruments 116a, 116b can be surgical instruments of different types having differing end effectors. In some cases, the instruments 116a, 116b include multiple DOFs such as, but not limited to, roll, pitch, yaw, insertion depth, opening/closing of jaws, actuation of staple delivery, activation of electro-cautery, and the like. Motion in at least some of such DOFs can be generated by the actuation modules 117a, 117b of the manipulators 102a, 102b to which the instruments 116a, 116b are selectively coupled.

If the instruments 116a, 116b are medical or surgical instruments, possible end effectors include, for example, DeBakey Forceps, microforceps, and Potts scissors include first and second end effector elements that pivot relative to each other so as to define a pair of end effector jaws. Other end effectors, including scalpels and electrocautery probes, have a single end effector element. For instruments having end effector jaws, the jaws will often be actuated by squeezing the grip members of input devices. In some cases, one or more of the instruments 116a, 116b includes an image capture device, such as a camera. The image capture device can capture imagery of other instruments in the workspace, and this imagery can be presented to the operator <NUM> to allow the operator <NUM> to visually monitor positions of other instruments in the workspace.

As depicted in <FIG> and <FIG>, the system <NUM> includes an indicator system <NUM>. The indicator system <NUM> includes one or more indicator devices to generate human-perceptible indications to provide feedback to the operator <NUM> during the association process or during the following mode. For example, the indicator system <NUM> includes indicator devices 124a, 124b. The indicator device 124a provides human-perceptible indications originating in the vicinity of the manipulator 102a, and the indicator device 124b provides human-perceptible indications originating in the vicinity of the manipulator 102b. The indicator devices 124a, 124b can be attached to, positioned on, or positioned proximate to the manipulators 102a, 102b, respectively. The indicator devices 124a, 124b can be located on the actuation modules 117a, 117b and thus can be attached to the manipulators 102a, 102b through the actuation modules 117a, 117b. Alternatively, in some implementations, the indicator devices 124a, 124b are directly attached to or positioned on the manipulators 102a, 102b.

The indicator system <NUM> further includes an indicator device 126a (on the user input device 108a shown in <FIG>) and an indicator device 126b (on the user input device 108b shown in <FIG>). The indicator device 126a provides human-perceptible indications originating in the vicinity of the user input device 108a, and the indicator device 126b provides human-perceptible indications originating in the vicinity of the user input device 108b. For example, the indicator devices 126a, 126b are attached to, positioned on, or positioned proximate to the user input devices 108a, 108b, respectively.

If the indicator devices 124a, 124b, 126a, 126b provide visual feedback, the indicator devices 124a, 124b, 126a, 126b each includes an appropriate visual feedback device. Example visual feedback devices include light-emitting diodes or other light sources, electronic displays such as LCD or OLED displays, etc. In some implementations, the indicator devices 124a, 124b, 126a, 126b emit different colors of light to intuitively indicate to the operator the statuses of the user input devices 108a, 108b and the manipulators 102a, 102b. In some implementations, the indicator devices 124a, 124b, 126a, 126b present different numeric, textual, or graphical patterns to indicate to the operator the statuses of the user input devices 108a, 108b and the manipulators 102a, 102b. For example, when the pairing mode is initiated, a green light or a flashing "O" can indicate that a device is in an unassociated state, and a red light or a steadily presented (not flashing) "O" can indicate that a device is in an associated state, or vice versa. In some cases, as described herein, a yellow light or a flashing or steadily presented "X" can provide a visual warning, such as to the operator <NUM>.

In some implemenations, the indicator devices 124a, 124b, 126a, 126b can provide visual feedback for indicating additional information about the statuses of the user input devices 108a, 108b and the manipulators 102a, 102b. In some implementations, the indicator devices 124a, 124b, 126a, 126b present visual feedback to indicate which user input device recommended to be associated with which manipulator, or to indicate which user input device will be associated with which manipulator upon confirmation. As one example, where the user input device 108a and the manipulator 102a is recommended to be associated with each other, or will be associated with which manipulator upon confirmation, both the user input device 108a and the manipulator 102a can flash matching, similar, or identical: color, number, text, graphics, flashing sequence, or other visual feedback.

In some implementations, the indicator devices 124a, 124b, 126a, 126b present visual feedback to indicate which user input device has been, or is currently, associated with which manipulator. As one example, after the user input device 108a is associated with the manipulator 102a, both the user input device 108a and the manipulator 102a can steadily present (not flashing) similar or identical: color, number, text, graphical pattern, or other visual feedback. In various implementations, this steady presentation of color, number, text, graphics, or other textual feedback can last for the entire duration during which the user input device 108a is associated with the manipulator 102a.

In addition to or as an alternative to providing visual feedback, the indicator devices 124a, 124b, 126a, 126b can provide human-perceptible tactile feedback, aural feedback, or a combination thereof. If the indicator devices 124a, 124b, 126a, 126b provide tactile feedback, the tactile feedback can include vibro-tactile feedback, force feedback, or other forms of feedback associated with a user's sense of touch. The indicator devices 124a, 124b, 126a, 126b can include, for example, a vibration generator. For example, the indicator devices 126a, 126b can generate vibrations that serve as haptic feedback for the operator <NUM> when the operator <NUM> is holding the user input devices 108a, 108b. If the indicator devices 124a, 124b, 126a, 126b provide aural feedback, the indicator devices 124a, 124b, 126a, 126b include, for example, an audio output device such a speaker. In such cases, the indicator devices 124a, 124b, 126a, 126b can narrate audible feedback to the operator <NUM>.

As noted above, the feedback provided by the indicator devices 124a, 124b, 126a, 126b can be indicative of statuses of the manipulators 102a, 102b and the user input devices 108a, 108b before, during, and after the association process. For example, the feedback can be indicative of association states of the manipulators 102a, 102b, association states of the user input devices 108a, 108b, or both. An association state can be indicative of whether a particular manipulator is associated with a user input device or whether a particular user input device is associated with a manipulator.

Alternatively or additionally, the indicator devices 124a, 124b, 126a, 126b can provide feedback during an association process to provide the operator <NUM> with information pertaining to the association process. The feedback can be indicative of a proposed association between a user input device and a manipulator. For example, as described herein, the operator <NUM> can operate the user input devices 108a, 108b to propose an association between a particular user input device and a particular manipulator, and the indicator devices 124a, 124b, 126a, 126b can provide feedback indicative of this proposed association. In some implementations, the feedback can be indicative of a usable or an optimal association between a user input device and a manipulator. For example, as described herein with respect to <FIG> and <FIG>, the controller <NUM> (shown in <FIG>) can determine usable or optimal associations between the user input devices <NUM> and the manipulators <NUM> and provide a signal indicative of the usable or optimal associations.

As shown in <FIG>, the user input device 108a can further include a sensor 128a that can indicate, to the controller <NUM> (shown in <FIG>), movement of the user input device 108a. For example, the sensor 128a can indicate, e.g., generate a signal indicative of, a relative movement between the user input device 108a and the manipulators <NUM>. The controller <NUM>, during the pairing mode, can associate a particular manipulator with the user input device 108a in response to the relative movement between the user input device 108a and the particular manipulator satisfying an association condition.

Similarly, as shown in <FIG>, the user input device 108b can further include a sensor 128b that can indicate, to the controller <NUM> (shown in <FIG>), movement of the user input device 108b. For example, the sensor 128b can indicate, e.g., generate a signal indicative of, a relative movement between the user input device 108b and the manipulators <NUM>. The controller <NUM>, during the pairing mode, can associate a particular manipulator with the user input device 108b in response to the relative movement between the user input device 108b and the particular manipulator satisfying an association condition. Examples of association conditions are described with respect to the process illustrated in <FIG>.

<FIG> and <FIG> illustrate an example of manual operation of the user input devices 108a, 108b to associate the user input devices 108a, 108b with the manipulators 102a, 102b. As shown in <FIG>, if the manipulators 102a, 102b were not previously associated with a user input device, the indicator devices 124a, 124b both provide feedback indicating that the manipulators 102a, 102b are in unassociated states. Similarly, the indicator device 126a also provides feedback indicating that the user input device 108a was not previously associated with a manipulator.

To initiate the association between the user input device 108a and the manipulator 102a, in a pairing mode, the user input device 108a is moved relative to the manipulator 102a. For example, the user input device 108a is moved to a region proximate the manipulator 102a. Based on this movement of the user input device 108a relative to the manipulator 102a, the controller <NUM> (shown in <FIG>) associates the manipulator 102a with the user input device 108a.

The controller <NUM> can control the indicator devices 124a, 126a to provide human-perceptible feedback indicating that the movement of the user input device 108a relative to the manipulator 102a corresponds to a proposed association between the user input device 108a and the manipulator 102a. For example, both the indicator device 124a and the indicator device 126a can provide matching, similar, or identical feedback so that the operator <NUM> can intuitively understand that the proposed association is between the user input device 108a and the manipulator 102a (and, e.g., not between the user input device 108a and the manipulator 102b). In some cases, this feedback is provided as the operator <NUM> moves the user input device 108a. As a result, the operator <NUM> is able to easily determine when the user input device 108a has been appropriately moved to satisfy the association condition to associate the user input device 108a and the manipulator 102a (and, e.g., to not satisfy the association condition to associate the user input device 108a and the manipulator 102b). As described with respect to the process illustrated in <FIG>, in some cases, the operator <NUM> provides confirmation of the proposed association so that the controller <NUM> proceeds with associating the manipulator 102a with the user input device 108a. In some implementations, the indicator devices 124a, 126a can indicate, after the association is made, that the user input device 108a is associated with the manipulator 102a. For example, the indicator devices 124a, 126a provide matching, similar, or identical feedback for part or the entire duration that the user input device 108a is associated with the manipulator 102a. As one example, the indicator devices 124a, 126a can provide the same color, number, text, graphic, or other visual feedback for the entire duration that the user input device 108a is associated with the manipulator 102a.

Referring to <FIG>, in a pairing mode, the user input device 108b is moved relative to the manipulator 102b. For example, the user input device 108b is moved to a region proximate the manipulator 102b. Based on this movement of the user input device 108b relative to the manipulator 102b, the controller <NUM> (shown in <FIG>) associates the manipulator 102b with the user input device 108b. As described herein, the user input device 108b associated with the manipulator 102b is distinct from the user input device 108a associated with the manipulator 102a so that the manipulators 102a, 102b can be independently controlled using the user input devices 108a, 108b of the user input system <NUM>.

The controller <NUM> can control the indicator devices 124b, 126b to provide human-perceptible feedback indicating that the movement of the user input device 108b relative to the manipulator 102a corresponds to a proposed association between the user input device 108a and the manipulator 102a. For example, both the indicator device 124b and the indicator device 126b can provide similar feedback so that the operator <NUM> can intuitively understand that the proposed association is between the user input device 108b and the manipulator 102b (and, e.g., not between the user input device 108b and the manipulator 102a). In some cases, this feedback is provided as the operator <NUM> moves the user input device 108b. As a result, the operator <NUM> is able to easily determine when the user input device 108b has been appropriately moved to satisfy the association condition to associate the user input device 108b and the manipulator 102b (and, e.g., to not satisfy the association condition to associate the user input device 108a and the manipulator 102b). As described with respect to the process illustrated in <FIG>, in some cases, the operator <NUM> provides confirmation of the proposed association so that the controller <NUM> proceeds with associating the manipulator 102b with the user input device 108b. Also, in some implementations, the indicator devices 124b, 126b can indicate, after the association is made, that the user input device 108b is associated with the manipulator 102b. For example, the indicator devices 124a, 126a provide similar or identical feedback for part or the entire duration that the user input device 108b is associated with the manipulator 102b.

Referring to <FIG> and as described herein, an example of the system <NUM> for performing an association process includes the manipulator system <NUM>, the controller <NUM>, and the user input system <NUM>. While described with respect to <FIG>, <FIG>, <FIG>, and <FIG> as including two or four manipulators, in some implementations, as shown in <FIG>, the manipulator system <NUM> can include any number of manipulators. For example, the manipulator system <NUM> includes N manipulators (e.g., Manipulator <NUM> through Manipulator N, collectively referred to as manipulators <NUM>). Similarly, while described with respect to <FIG>, <FIG>, <FIG>, and <FIG> as including two user input devices, in some implementations, as shown in <FIG>, the user input system <NUM> includes any number user input devices. For example, the user input system <NUM> includes M user input devices (e.g., User Input Device <NUM> through User Input Device M, collectively referred to as user input devices <NUM>). Examples of the user input devices <NUM> include: joysticks, touchscreens, gloves, or handheld remotes.

The system <NUM> can further include a sensor system <NUM>. The sensor system <NUM> includes sensors operable to detect movement the user input devices <NUM>. For example, the sensor system <NUM> includes the sensors 128a, 128b (shown in <FIG> and <FIG>). The sensors, in particular, can detect movement of the user input devices <NUM> that satisfy association conditions for associating a particular user input device with a particular manipulator. The sensor system <NUM> can detect poses, e.g., positions, orientations, or both positions and orientations, of the user input devices <NUM> and the manipulators <NUM> in the environment <NUM>. Sensors of the sensor system <NUM> include, for example, infrared sensors, ultrasonic sensors, image capture devices, accelerometers, position encoders, optical sensors, or other appropriate sensors for detecting motion and poses of the manipulators <NUM> and the user input devices <NUM>. Further examples of association conditions and sensors for detecting these association conditions are described with respect to the process illustrated in <FIG>.

The system <NUM> can further include a user output system <NUM> and a memory storage element <NUM>. The user output system <NUM> provides human-perceptible feedback to the operator <NUM>. The user output system <NUM> includes, for example, the indicator system <NUM> or the display system <NUM>. In this regard, the feedback provided by the user output system <NUM> can include feedback provided during an association process or during following mode.

The memory storage element <NUM> can store data indicative of associations formed between the manipulators <NUM> and the user input devices <NUM>. The controller <NUM> can retrieve these stored data to determine whether a user input device or a manipulator is in an associated state or an unassociated state. Referring to <FIG>, the manipulators <NUM> and the user input devices <NUM> are associated so that each user input device <NUM> is associated with a distinct one of the manipulators <NUM>. As a result, the user input devices <NUM> can be controlled by the operator <NUM> so that the associated manipulators can be independently controlled. In some cases, each of the manipulators <NUM> is associated with a corresponding one of the user input devices <NUM>. As a result, each of the manipulators <NUM> can be controlled using the user input devices <NUM>.

Referring to <FIG>, a process <NUM> including an association process and a following process is presented with respect to the system <NUM> described herein. The process <NUM> is performed by the operator <NUM>, the user input system <NUM>, the manipulator system <NUM>, the controller <NUM>, or a combination of the foregoing. At operation <NUM>, a pairing mode is initiated. At operation <NUM>, the association process is performed to associate a manipulator with a user input device. Examples of further operations and sub-operations the operations <NUM> and <NUM> are described with respect to <FIG>, <FIG>, <FIG>, and <FIG>.

At operation <NUM>, a following mode is initiated so that, in a following process, the manipulator can be controlled in response to manipulation of the user input device. In the following mode, the manipulator associated with the user input device at operation <NUM> can be moved in response to manual operation of the user input device by the operator <NUM>. The operator manually manipulates the user input device, which generates a signal or indication received by the controller <NUM>. The controller <NUM> then generates a corresponding signal to move the manipulator of the manipulator system <NUM> with which the user input device is associated. In this regard, the user input device and the manipulator form a leader-follower system in which the user input device is a leader device and the manipulator is a follower device, thereby enabling the manipulator to be teleoperated through operation of the user input device. If the system <NUM> is a surgical system, an instrument supported by the manipulator can be controlled to perform a surgical operation on a patient.

<FIG> illustrates an example of a process <NUM> to associate a user input device of the user input system <NUM> with a manipulator of the manipulator system <NUM>. The process <NUM> is performed, for example, during the operations <NUM> and <NUM> described with respect to the process <NUM>.

Operations <NUM>-<NUM> of <FIG> illustrate an example process of initiating a pairing mode. At operation <NUM> of the process <NUM>, the operator <NUM> operates the user input system <NUM> to initiate the pairing mode. For example, the user input system <NUM> includes a user input device dedicated to initialization of the pairing mode. This user input device can be distinct from the user input devices that are associated to manipulators. At operation <NUM>, the user input system <NUM> transmits a signal to the controller <NUM> to initiate the pairing mode, e.g., in response to the operator <NUM> operating the user input system <NUM> to initiate the pairing mode. At operation <NUM>, the controller <NUM> initiates the pairing mode, e.g., in response to the signal transmitted at the operation <NUM>.

During the pairing mode, the operator <NUM> provides an association intent to associate a particular user input device with a particular manipulator. In addition, feedback is provided to the operator <NUM> so that the operator <NUM> can be kept informed of states of the manipulators of the manipulator system <NUM> and the user input devices of the user input system <NUM>. Operations <NUM>-<NUM> illustrate examples of operations that occur during the pairing mode and in which the foregoing can occur.

In some examples, after the pairing mode is initiated at operation <NUM>, at operation <NUM>, the controller <NUM> transmits signals to provide association indicators to the operator <NUM>. The signals can be transmitted to the user output system <NUM> (shown in <FIG>), e.g., the indicator system <NUM> (shown in <FIG> and <FIG>) or the display system <NUM> (shown in <FIG>). The user output system <NUM> presents the association indicators to indicate an association state of each of the manipulators of the manipulator system <NUM> and an association state of each of the user input devices of the user input system <NUM>.

At operation <NUM>, the operator <NUM> moves a user input device (e.g., one of the user input devices <NUM> described with respect to <FIG>) to provide an association intent. The movement of the user input device corresponds to an intent to associate the user input device with a manipulator of the manipulator system <NUM> (e.g., one of the manipulators <NUM> described with respect to <FIG>).

Based on the movement of the user input device, a signal indicative of the movement of the user input device is transmitted to the controller <NUM>. The signal is transmitted by the sensor system <NUM> (shown in <FIG>) to the controller <NUM>.

At operation <NUM>, based on the signal indicative of the movement of the user input device, the controller <NUM> determines whether the movement of the user input device satisfies an association condition. Association conditions can vary between implementations.

In some examples, the association condition includes contact between a particular user input device and a particular manipulator. The sensor system <NUM> includes a contact sensor, e.g., a capacitive sensor, that detects contact between the user input device and the manipulator intended to be associated with the user input device. The contact sensor can be attached to or positioned on the user input device or the manipulator. In some implementations, the user input device includes a contact sensor, and the manipulator also includes a contact sensor. The association condition is satisfied when the contact sensors detect contact with one another.

In some implementations, the association condition includes proximity between the user input device and the manipulator. For example, the association condition is satisfied when the user input device is move to a region within a predefined distance from the manipulator, e.g., within <NUM>, within <NUM>, within <NUM>, within <NUM>, or within <NUM>. The sensor system <NUM> includes a proximity sensor that detects when the user input device is moved to within the predefined distance. For example, the proximity sensor can include a sensor to detect the distance between the user input device and the manipulator, e.g., a time-of-flight sensor, an infrared sensor, or an ultrasonic sensor. In some examples, the proximity sensor includes a sensor attached to one of the manipulator or the user input device and a detectable tag on the other of the manipulator or the user input device. The sensor can be a radiofrequency detector, and the tag can be a radiofrequency tag.

In some implementations, the association condition includes a motion of the user input device directed toward a region defined by a location of the manipulator in the workspace. For example, each of the manipulators defines a corresponding volume. The volume defined by a manipulator can be within the workspace of the manipulator or can extend outside of the workspace. In some cases, the volume is defined by physical surface of the manipulator. The motion of the user input device includes a direction, e.g., a direction of a velocity or acceleration of the user input device. The association condition for a manipulator is satisfied when the direction of the motion is directed toward the volume defined by the manipulator, e.g., the ray defined by the direction of the motion intersects with the volume. The sensor can be an accelerometer coupled to the user input device. The accelerometer can detect the direction of the motion of the user input device. In implementations in which multiple user input devices are associated with multiple manipulators in the pairing mode, locations of the multiple manipulators define multiple distinct regions. The operator moves the user input device toward a first region to satisfy the association condition for a first manipulator, and the operator moves the user input device toward a second region to satisfy the association condition for a second manipulator.

Contact between the manipulator and the user input device, proximity between the manipulator and the user input device, and direction of movement of the user input device can be detected in other manners. For example, in some implementations, an image capture device of the sensor system <NUM> captures imagery of the environment <NUM> (shown in <FIG>). Contact, proximity, and direction of movement can be determined using image analysis of the captured imagery. The image capture device can be a stationary camera positioned to capture imagery of the manipulator system <NUM> and the user input system <NUM>.

In some implementations, the movement of the user input device in the pairing mode corresponds to ambiguous movement that does not satisfy a single association condition for a single manipulator. In some cases, the association conditions for multiple manipulators are satisfied due to a single motion of the user input device. In such cases, the controller <NUM> does not associate a manipulator with the user input device until an association condition of only one of the manipulators is fulfilled. To guide the operator <NUM> to resolve the ambiguity of multiple association conditions being fulfilled, the controller <NUM> can control the user output system <NUM> to provide feedback to the operator <NUM> indicative of the ambiguity. For example, indicator devices of the manipulators for which the association conditions are satisfied are activated. Multiple indicator devices of the manipulators being activated can be indicative of ambiguity, and the operator <NUM> can adjust movement of the user input device or can generate other motions of the user input device to resolve the ambiguity.

In some cases, the movement of the user input device in the pairing mode corresponds to a movement that does not satisfy an association condition for any manipulator. The controller <NUM> can operate the user output system <NUM> to provide feedback to the operator <NUM>, e.g., to guide the operator to move the user input device in a manner that fulfills an association condition of one of the manipulators. For example, when a movement of the user input device is detected but does not satisfy an association condition for any manipulator, the indicator device for the user input device can provide a warning indication. In some cases, the indicator device emits a yellow light to provide the warning indicator to the operator <NUM>.

After the controller <NUM> determines that the movement of the user input device satisfies the association condition for a manipulator, at operation <NUM>, the controller <NUM> determines an association state of the manipulator. The controller <NUM> determines whether the manipulator is in an unassociated state. For example, the controller <NUM> can access the memory storage element <NUM> (shown in <FIG>) to determine whether an association for the manipulator has been stored on the memory storage element <NUM>. If the manipulator is not in an unassociated state, e.g., in an associated state, the operator <NUM> at operation <NUM> either confirms that a new association is to be provided to the manipulator or indicates that the manipulator should maintain the stored association. If the operator <NUM> confirms that a new association is to be provided, the controller <NUM> can remove the stored association for the manipulator.

If it is confirmed at the operation <NUM> that a new association is to be created for the manipulator or if it is determined at the operation <NUM> that the manipulator in an unassociated state, the controller <NUM> at operation <NUM> requests for confirmation of an association between the user input device and the manipulator. For example, the controller <NUM> transmits data representing the request for confirmation to the user output system <NUM>. In some implementations, the indicator device on the user input device and the indicator device on the manipulator both provide a feedback pattern indicative of a pending association. If the indicator devices are light indicator devices, the indicator devices can both flash a same color (e.g. green), number, text, graphics, or in a same time sequence, or provide some other visual feedback to indicate that the condition for association between the manipulator and the user input device has been satisfied and confirmation from the operator <NUM> is requested.

At operation <NUM>, the operator <NUM> provides the confirmation of the association. In some implementations, the operator <NUM> can provide this confirmation by performing a specific manipulation of the user input device to confirm the association. The specific manipulation can include a movement of the user input device detectable by the sensor system <NUM>, e.g., shaking the user input device up-and-down or directing a movement of the user input device towards the display system <NUM>. The specific manipulation can include a manipulation of the user input device for controlling motion of the manipulator in a following mode. For example, if the user input device includes a joystick, a button, knob, or other user input element, the operator <NUM> can operate the user input element to confirm the association.

At operation <NUM>, after receiving confirmation of the association, the controller <NUM> stores the association, e.g., in the memory storage element <NUM>. The controller <NUM> then provides a success signal at operation <NUM>. For example, the user output system <NUM> is operated to provide a human-perceptible signal indicative of the success of the association between the manipulator and the user input element.

While described with respect to associating a single user input device with a single manipulator, in some implementations, the operations <NUM>-<NUM> can be repeated to associate other user input devices of the user input system <NUM> with other manipulators of the manipulator system <NUM>. The system <NUM> can remain in the pairing mode until the operator <NUM> operates the user input system <NUM> to provide input indicative of initiating the following mode, e.g., initiating operation <NUM>. In the following mode, the user input devices that have been associated with the manipulators can be operated by the operator <NUM> to control movement of the manipulators.

In some implementations, the controller <NUM> can provide recommendations to optimize the associations formed between the manipulators and the user input devices. Process <NUM> of <FIG> illustrates an example process to provide such a recommendation. The process <NUM> is initiated after the pairing mode is initiated. Upon initiation, at operation <NUM>, the user input system <NUM> transmits signals indicative of poses of the user input devices of the user input system <NUM>, e.g., poses of the user input devices in the environment <NUM> (shown in <FIG>). At operation <NUM>, the manipulator system <NUM> transmits signals indicative of poses of the manipulators of the manipulator system <NUM> to the controller <NUM>. At operation <NUM>, the controller <NUM> receives these signals from the user input system <NUM> and the manipulator system <NUM>. In some implementations, the sensor system <NUM> detects the poses of the user input devices, the manipulators, or both and transmits these signals to the controller <NUM>. In addition, the controller <NUM> further receives a signal indicative of the position and the orientation of the image capture device, e.g., on the instrument supported on the manipulator 102c.

The controller <NUM> receives the signals and uses kinematic modeling to determine the positions and orientations of the manipulators 102a, 102b, the positions and orientations of the instruments 116a, 116b, and the position and orientation of the image capture device. In some cases, one or more signals are generated by sensors of the manipulators (e.g., the manipulators 102a, 102b, and the manipulator to which the image capture device is mounted) or sensors of the instruments (e.g., the instruments 116a, 116b, and the image capture device). The sensors of the manipulators include, for example, accelerometers, gyroscopes, encoders, or other sensors associated with joints of the manipulators 102a, 102b. The sensors of the instruments include, for example, shape sensors through shafts of the instruments. Alternatively, the positions and orientations of the manipulators and/or the positions and orientations of the instruments are determined based on one or more signals from optical sensors (e.g., image capture devices). The manipulators or the instruments are equipped with optical fiducials detectable by the optical sensors.

At operation <NUM>, based on the received signals, the controller <NUM> determines usable or optimal associations between the manipulators of the manipulator system <NUM> and the user input devices of the user input system <NUM>. <FIG> illustrates an example in which the manipulators 102a, 102b (shown on the right portion of <FIG>) support the instruments 116a, 116b, respectively (shown on the left portion of <FIG>). The display system <NUM> presents imagery captured by an instrument (not shown) with an image capture device supported by the manipulator 102c.

<FIG> includes a right portion diagrammatically depicting relative positions of the display system <NUM> and the user input devices <NUM>, and a left portion showing a top view of the system <NUM>. As shown in <FIG>, the instrument 116a appears on a left portion of imagery presented on the display system <NUM>, while the instrument 116b appears on a right portion of the imagery. To provide the operator <NUM> with intuitive control of the instruments 116a, 116b as the instruments 116a, 116b appear on the display system <NUM>, the controller <NUM> provides a recommendation to associate the user input device 108a (in the left hand of the operator <NUM>) with the instrument 116a on the left portion of the imagery. Furthermore, the controller <NUM> provides a recommendation to associate the user input device 108b (in the right hand of the operator <NUM>) with the instrument 116b on the right portion of the imagery. The controller <NUM> can determine the relative positions and orientations of the user input devices 108a, 108b and the manipulators 102a, 102b based on the signals indicative of the poses of these devices.

Alternatively, in some implementations, the controller <NUM> determines the positions and orientations of the user input devices 108a, 108b relative to the instruments 116a, 116b supported by the manipulators 102a, 102b. The controller <NUM> further receives a signal indicative of the position and the orientation of the image capture device, e.g., on the instrument supported on the manipulator 102c. The controller <NUM> can determine relative poses of the instruments 116a, 116b as they would appear to the operator <NUM> on the display system <NUM>. The controller <NUM> can determine a recommendation for the associations between the user input devices 108a, 108b and the manipulators 102a, 102b based on these relative poses. In various implementations, the recommendation may include recommended associations for a subset or all of the user input devices (e.g. 108a, 108b) and a subset or all of the manipulators (102a, 102b). Also, in various implementations, the recommendations may indicate degrees of recommendation for a particular association, such as: a more recommended association between a user input device and a manipulator (e.g. between the user input device 108a and the manipulator holding the instrument 116a), a less recommended association between a user input device and a manipulator (e.g. between the user input device 108a and a manipulator holding an instrument not shown in the imagery), or a not recommended association between a user input device and a manipulator (e.g. the user input device 108a and a manipulator holding the instrument 116b).

In some implementations, the controller <NUM> does not receive positions and orientations of the user input devices 108a, 108b for determining the recommendations. The user input devices 108a, 108b can be configured such that the user input devices 108a, 108b have fixed positions and orientations relative to one another. In this regard, the controller <NUM> can provide a recommendation based on the positions and orientations of the manipulators 102a, 102b relative to one another or based on the positions and orientations of the instruments 116a, 116b relative to one another.

After the controller <NUM> determines the optimal associations, the controller <NUM> at operation <NUM> provides a signal to indicate the optimal associations to the operator <NUM>. For example, the controller <NUM> controls the user output system <NUM> to provide an appropriate signal to guide the operator <NUM> to form the optimal associations between the manipulators 102a, 102b and the user input devices 108a, 108b. If the manipulators 102a, 102b and the user input devices 108a, 108b include light indicator devices, the light indicator devices for corresponding devices that the controller <NUM> recommends associating can emit like optical signals, e.g., optical signals having a similar color. These optical signals can be distinct from the optical signals emitted by the light indicator devices of other corresponding devices that the controller <NUM> recommends associating. These optical signals can thus provide guidance to the operator <NUM> to intuitively form the optimal associations, e.g., using the process <NUM>. Alternatively, in some implementations, rather than the process <NUM> being execute to form the optimal associations, the controller <NUM> automatically forms the associations between the manipulators 102a, 102b and the user input devices 108a, 108b based on the determined optimal associations.

In some implementations, prior to initiating the following mode and after the associations between user input devices and manipulators are formed, for each user input device, the controller <NUM> can determine a pose of a portion of the user input device relative to a pose of the manipulator. For example, the portion of the user input device is manually operable by the operator <NUM> to control movement of the manipulator during the following mode. To ensure that the operator <NUM> can control the manipulator <NUM> through its full range of motion using the user input device, the portion of the user input device can be reoriented or repositioned. Process <NUM> of <FIG> is performed to achieve this reorienting or repositioning of a portion of a user input device.

At operation <NUM>, the user input system <NUM> transmits a signal indicative of a pose of a portion of a user input device of the user input system <NUM>. For example, the signal is indicative of the position and orientation of the portion of the user input device relative to the full range of motion of the portion. A position sensor coupled to the portion of the user input device can generate the signal. At operation <NUM>, the manipulator system <NUM> transmits a signal indicative of a pose of a manipulator of the manipulator system <NUM>. Position sensors of the manipulator, e.g., encoders, accelerometers, etc., can generate and transmit the signal. At operation <NUM>, the controller <NUM> receives these signals from the user input system <NUM> and the manipulator system <NUM>.

Based on these signals, at operation <NUM>, the controller <NUM> determines whether a pose of the user input device relative to a full range of motion of the portion matches with a pose of the manipulator relative to a full range of motion of the manipulator. For example, the user input device can have a degree of freedom of motion for controlling yaw motion of the distal end of the manipulator. The controller <NUM> determines whether the position of the user input device within the full range of motion for this degree of freedom of motion matches with the position of the distal end of the manipulator within the full range of motion for its yaw degree of freedom. The controller <NUM> similarly compares the position of the portion of the user input device for each of its other degrees of freedom to the position of the manipulator for its other degrees of freedom.

If the poses of the portion of the user input device and the manipulator do not match, the controller <NUM> at operation <NUM> transmits signals to reorient the portion of the user input device. At operation <NUM>, the user input system <NUM> receives the signals. In some cases, the signals cause automatic motion of the portion of the user input device. For example, the signals drive one or more actuators to move the portion of the user input device. Alternatively, the user input system <NUM> provides feedback to the operator <NUM> to reorient or reposition the portion of the user input device. The user input system <NUM> then at operation <NUM> transmits another signal indicative of the pose of the portion of the user input device, and the controller <NUM> determines again whether there is match between the poses of the portion of the user input device and the manipulator.

When the controller <NUM> determines a match at operation <NUM>, the following mode can be initiated. For example, the success signal can be provided at the operation <NUM> of the process <NUM>, and the following mode can then be initiated.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made.

For example, controllers and any associated components described herein can be part of a computing system that facilitates control of the systems according to processes and methods described herein. <FIG> is a schematic diagram of an example of a computer system <NUM> that can be used to implement a controller, e.g., the controller <NUM> or other controller of the system <NUM>, described in association with any of the computer-implemented methods described herein, e.g., methods including one or more of the processes and operations described with respect to <FIG>. The system <NUM> includes components such as a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. The components <NUM>, <NUM>, <NUM>, and <NUM> are interconnected using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. In some examples, the processor <NUM> is a single-threaded processor, while in some cases, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a user interface on the input/output device <NUM>.

Memory storage for the system <NUM> can include the memory <NUM> as well as the storage device <NUM>. The information can be used by the processor <NUM> in performing processes and methods described herein. In some examples, the memory <NUM> is a computer-readable storage medium. The memory <NUM> can include volatile memory and/or non-volatile memory. In general, the storage device <NUM> can include any non-transitory tangible media configured to store computer readable instructions. Optionally, the storage device <NUM> is a computer-readable medium. Alternatively, the storage device <NUM> may be a floppy disk device, a hard disk device, an optical disk device, or a tape device.

In some cases, the processor <NUM> is in communication with a remote computing system <NUM>. The remote computing system <NUM> includes, for example, a remote server, a cloud computing device, or other computing device remote from the processor <NUM> and its systems. The remote computing system <NUM> includes computing resources remote from the environment of the processor <NUM>, e.g., remote from the surgical environment. In some cases, the remote computing system <NUM> includes one or more servers that establish wireless links with the processor <NUM>. The remote computing system <NUM> includes, for example, a portion of a network-accessible computing platform implemented as a computing infrastructure of processors, storage, software, data access, and so forth accessible by the processor <NUM>.

The system <NUM> includes the input/output device <NUM>. In some examples, the input/output device <NUM> includes a keyboard, a computer mouse, a pointing device, a voice-activated device, a microphone, a touchscreen, etc. In some cases, the input/output device <NUM> includes a display unit for displaying graphical user interfaces.

The features of the methods and systems described in this application can be implemented in digital electronic circuitry, or in computer hardware, firmware, or in combinations of them. The features can be implemented in a computer program product tangibly stored in an information carrier. The information carrier can be, for example, a machine-readable storage device, for execution by a programmable processor. Operations, e.g., of the processes <NUM>, <NUM>, <NUM>, and <NUM>, can be performed by a programmable processor executing a program of instructions to perform the functions described herein by operating on input data and generating output. The described features can be implemented in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program includes a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages. The computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program implements, for example, a fast genetic algorithm (FGA).

Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files. Such devices can include magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for storing the computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

Alternatively, the computer can have no keyboard, mouse, or monitor attached and can be controlled remotely by another computer.

A client and server are generally remote from each other and typically interact through a network.

The processor <NUM> carries out instructions related to a computer program. The processor <NUM> can include hardware such as logic gates, adders, multipliers and counters. The processor <NUM> can further include a separate arithmetic logic unit (ALU) that performs arithmetic and logical operations.

While the manipulators <NUM> are described and shown as being distinct manipulators separately mounted with mounting locations movable relative to each other, e.g., to an operating table, the association processes described herein are also applicable to manipulators that are mounted to a shared base. For example, referring to <FIG>, a manipulator system <NUM> includes manipulators 1102a, 1102b, 1102c, 1102d (collectively referred to as manipulators <NUM>), each of which is mounted to a common base <NUM>. A joint <NUM> can be driven to reorient all of the manipulators <NUM>. The base <NUM> can be mounted to a movable cart portion <NUM>. The movable cart portion <NUM> is, for example, supported above a floor surface by wheels. In this regard, the manipulator system <NUM> is easily movable about an environment.

While the processes have been described as being used for association of the user input devices <NUM> with the manipulators <NUM>, in some implementations, the system <NUM> includes one or more sensors that detect motion and form an association based on the detected motion. For example, the processes described herein are used for association of hands of the operator <NUM> with are described as being handheld. Referring to <FIG>, in some implementations, the system <NUM> includes an optical motion detection system <NUM> including optical sensors 1202a, 1202b. The optical sensors 1202a, 1202b can provide a stereoscopic imagery of the operator <NUM> and can be used to detect motion of the operator <NUM>, in particular, motion of hands 1204a, 1204b of the operator <NUM>. Movements of the hands 1204a, 1204b can be used to control movement of the manipulators <NUM>. For example, the hands 1204a, 1204b are moved in a pattern or sequence in accordance with predefined gestures for controlling the system <NUM>. The predefined gestures can include a gesture for initiating a pairing mode, a gesture for proposing an association between a hand and a manipulator, a gesture for initiating a following mode, or other appropriate gesture to control the system <NUM>. In some implementations, the hands 1204a, 1204b are equipped with gloves detectable by the optical motion detection system <NUM>.

In addition, in the pairing processes described herein, rather than moving the user input devices <NUM> to form associations with the manipulators <NUM>, the operator <NUM> moves hands 1204a, 1204b to form associations between the manipulators <NUM> and the hands 1204a, 1204b. The optical motion detection system <NUM> detects the movements of the hands 1204a, 1204b, and the controller <NUM> determines the association to be formed based on the movements of the hands 1204a, 1204b. The hands 1204a, 1204b can be moved in manners similar to the movements described for the user input devices for satisfying association conditions. When the hands 1204a, 1204b are moved in a manner to satisfy the association conditions, the controller <NUM> forms the associations between the hands 1204a, 1204b and the corresponding manipulators. For example, at operation <NUM>, rather than moving a user input device, the operator <NUM> moves a hand 1204a or a hand 1204b. The hands 1204a, 1204b can then be used in the following mode to control motion of the manipulators.

In some implementations, the one or more sensors for detecting motion include any three-dimensional position sensor, such as a magnetic field sensor or an optical sensor. The one or more sensors can include a motion-gravity sensor such as an inertial motion unit (IMU), a gyroscope, a magnetic field compass, or other appropriate sensor for detecting acceleration.

While the indicator system <NUM> is described as including the indicator devices 124a, 124b, 126a, 126b, the indicator system <NUM> varies in other implementations. In some cases, the indicator system <NUM> includes the display system <NUM>. The display system <NUM> presents visual feedback to the operator <NUM> during the association process. Alternatively or additionally, the indicator system <NUM> includes an augmented reality interface. For example, the operator <NUM> can be equipped with a head-mounted augmented reality system that presents virtual indicators proximate the manipulators <NUM> and the user input devices <NUM> to provide feedback during the association process. The augmented reality system can present an indicator proximate a particular manipulator to indicate an association state of the manipulator, e.g., whether the manipulator is in an unassociated state or an associated state. Such a virtual indicator may present feedback similar to any discussed herein with relation to actual, physical indicators. Also, such a virtual indicator may present feedback particular to augmented reality systems.

The pairing mode can be initiated in response to a particular event. For example, operations <NUM>-<NUM> illustrate a particular example of initiating the pairing mode in response to operation of the user input system <NUM>. In some implementations, the user input device of the user input system <NUM> that is operated to initiate the pairing mode corresponds to a user input device operable to initiate a clutching mode in which the manipulators can be manually repositioned. In the clutching mode, brake systems of the manipulators are disabled or joints of the manipulators are released so that the manipulators can be manually repositioned by the operator. In some examples, the pairing mode is also initiated when the clutching mode is initiated.

In some implementations, the pairing mode can be initiated in response to events that are not associated with operation of user input devices. For example, the controller <NUM> can be configured to initiate the pairing mode when the system <NUM> is initialized. In some cases, the system <NUM> includes an audio input system that detects voice commands issued by the operator <NUM>. The operator <NUM> can utter a voice command, and the controller <NUM> accordingly initiates the pairing mode. Alternatively or additionally, the pairing mode can be initiated when a new operator accesses and operates the user input system <NUM>.

In some implementations, the system <NUM> includes a sensor, e.g., an image capture system or an optical detection system, to detect whether the operator <NUM> is positioned to view the display system <NUM>. The pairing mode can be initiated in response to the sensor detecting that the operator <NUM> is looking away from the display system <NUM>. For example, the sensor can detect that the operator <NUM> is turned away from the display system <NUM>, e.g., toward the workspace where the manipulators <NUM> are located. By turning away from the display system <NUM> toward the workspace, the operator <NUM> can convey intent to associate a user input device with a manipulator. Accordingly, the controller <NUM> initiates the pairing mode in response to the detection of the operator <NUM> turning away from the display system <NUM>. In some cases, the controller <NUM> initiates the pairing mode in this manner after detection that the operator <NUM> has turned away from display system <NUM> for a predefined amount of time, e.g., at least <NUM> seconds, at least <NUM> seconds, at least <NUM> seconds, at least <NUM> minute.

As described herein, a manipulator is associated with the user input device so that the manipulator is movable in response to certain operations of the user input device. Thus, in some implementations, the associated user input device can be used for controlling movement of the manipulator. Further, in some implementations, the associated user input device is operable to control other functions of the manipulator or an instrument mounted to the manipulator instead of, or in addition to, controlling movement of the manipulator. In this regard, at operation <NUM>, when the following mode is initiated, the manipulator is not necessarily moved in response to operation of the associated user input device but, rather, receives a signal to perform a particular function or cause the instrument to perform a particular function. For example, in some implementations where the instrument is an image capture device, the associated user input device is operable to control an image capture function of the image capture device, such as a zoom setting, a lighting setting, a shutter speed setting, or other image capture setting. As another example, in some implementations where the instrument is a suction or irrigation device, the associated user input device is operable to control the application of suction or irrigation. In some implementations where the instrument is an image capture device, the associated user input device is operable to control the image capture device to capture imagery. In some implementations where the instrument is a cauterizing device or other energy application device, the associated user input device is operable to control the energy application device to apply energy to tissue.

In some implementations, in the pairing mode, multiple manipulators are associated with a single portion of the user input system <NUM>. For example, two or more manipulators can be associated with a single one of the user input devices <NUM>. When a single user input device is associated with multiple manipulators, the user input device is operable to generate movement of each of the manipulators. For example, in some implementations, if the operator <NUM> wishes to shift the combined workspace of multiple manipulators or their associated instruments to a different workspace, the operator <NUM> can operate the user input device to shift each of the manipulators to a vicinity of this different workspace. In some implementations, rather than moving each of the manipulators one-by-one to reach the different workspace, the operator <NUM> can associate all of the manipulators to be moved, with a single user input device and operate the single user input device to move the plurality of manipulators, as a group, to the vicinity of the different workspace.

As another example, in some implementations, multiple manipulators can be associated with a single user input device of the user input devices <NUM>, and the single user input device controls only one of the manipulators at a time. In some implementations, an operator selects which one of the manipulators is to be controlled by operating the single user input device via an appropriate method, such as depression of a button, turning of a dial, clicking of a pedal, voice commands, etc. In some implementations, the operator operates a button or pedal is used to cycle through manipulators of the manipulators until the one to be controlled becomes active.

Alternatively or additionally, two or more user input devices can be associated with a single manipulator. For example, in some implementations, one of the user input devices associated with the manipulator is operable to move the manipulator, while the other of the user input devices associated with the manipulator is operable to control a non-movement function of the manipulator or a function of an instrument mounted to the manipulator. In some implementations, the two or more associated user input devices each is operable to control a different degree of freedom or a different set of degrees of freedom of the manipulator. For example, in some implementations, one of the user input devices is manually operable to control a pitch, a yaw, and a roll motion of the manipulator, while the other of the user input devices is manually operable to control movement of the instrument relative to the manipulator along the insertion axis or to control actuation of an end effector of the instrument. As yet another example, in some implementations, a plurality of user input devices are used to enable a multi-handed input. For example, positions, separation distances, direction of motion, speed of motion of the user input devices, relative to each other or a reference, can be used to control the manipulator or an instrument supported by the manipulator.

As a specific example, in an implementation, two user input devices are associated with a single manipulator holding an imaging system such as a camera. An operator holding a user input devices in each hand can control the imaging system with two-handed combination input that simulates manipulation of the work piece relative to the imaging system. For example, in a camera implementation, combined motion of both input devices away from the operator moves the camera away or causes the camera to zoom out, as if the work piece has been pushed away. As another example, in a camera implementation, combined motion of both input devices around a common center rotates the camera field of view, as if the work piece had been rotated. As a further example, in a camera implementation, an increase in the separation distance between the user input devices causes the camera to zoom out, and a decrease in the separation distance between the user input devices causes the camera to zoom in.

Claim 1:
A computer-assisted medical system comprising:
a plurality of manipulators (102a, 102b);
a user input system (<NUM>) operable to generate signals to control the plurality of manipulators (102a, 102b), a portion (108a) of the user input system (<NUM>) being movable relative to the plurality of manipulators (102a, 102b); and
a controller (<NUM>) configured to execute instructions to perform operations comprising:
in a pairing mode, associating a first manipulator (102a) of the plurality of manipulators (102a, 102b) with the portion (108a) of the user input system (<NUM>) based on movement of the portion (108a) of the user input system (<NUM>) relative to the first manipulator (102a) being directed toward a first region defined by a location of the first manipulator (102a), wherein the motion is detected by an accelerometer coupled to the portion (108a) of the user input system (<NUM>), and
in a following mode, controlling motion of the first manipulator (102a) in accordance with an indication generated by the user input system (<NUM>) in response to operation of the portion (108a) of the user input system (<NUM>) by a user,
wherein the system further comprises a sensor system (<NUM>) configured to indicate, to the controller (<NUM>), a relative movement between the portion (108a) of the user input system (<NUM>) and the first manipulator (102a),
wherein associating the first manipulator (102a) of the plurality of manipulators (102a, 102b) with the portion (108a) of the user input system (<NUM>) comprises:
associating the first manipulator (102a) with the portion (108a) of the user input system (<NUM>) in response to the relative movement between the portion (108a) of the user input system (<NUM>) and the first manipulator (102a) satisfying an association condition, and
wherein the association condition further comprises movement of the portion (108a) of the user input system (<NUM>) to a region within a predefined distance from the first manipulator (102a).