Patent Description:
A system of robotic devices can be used to perform a task at a worksite. For example, robotic systems can include robotic manipulators to manipulate instruments for performing the task. The robotic manipulator can include two or more links coupled together by one or more joints. The joints can be active joints that are actively controlled. The joints can also be passive joints that comply with movement of the active joints as the active joints are actively controlled. Such active and passive joints may be revolute or prismatic joints. The configuration of the robotic manipulator may then be determined by the positions and orientations of the joints, the structure of the robotic manipulator, and the coupling of the links.

Robotic systems include industrial and recreational robotic systems. Robotic systems also include medical robotic systems used in procedures for diagnosis, non-surgical treatment, surgical treatment, etc. As a specific example, robotic systems include minimally invasive, robotic telesurgical systems in which a surgeon can operate on a patient from bedside or a remote location. Telesurgery refers generally to surgery performed using surgical systems where the surgeon uses some form of remote control, e.g., a servomechanism, to manipulate surgical instrument movements rather than directly holding and moving the instruments by hand. A robotic medical system usable for telesurgery or other telemedical procedures can include a remotely controllable robotic manipulator. Operators can remotely control motion of the remotely controllable robotic manipulator. Operators can also manually move pieces of the robotic medical system into positions or orientations within its environment.

<CIT> discloses methods for arranging objects in an operating room in preparation for a surgical procedure. The objects are arranged based on surgical procedure information provided to a guidance station. The surgical procedure information dictates the desired placement of the objects. Placement of the objects is then guided according to their desired placement using one or more tracking elements. <CIT> discloses positioning guidance for image acquisition. In order to facilitate positioning of a patient for medical image acquisition, a guiding system is provided. The system comprises a patient detecting device and a patient position prescribing device. The patient detecting device is configured to detect an anatomy of interest of a patient for image acquisition and to detect current spatial information of the anatomy of interest. The patient position prescribing device is configured to provide an initial target position for the detected anatomy of interest, wherein the initial target position is provided as a reference for the image acquisition. The patient position prescribing device is further configured to register the initial target position with the current spatial information, and to determine an adapted target position by adapting the initial target position based inter alia on the current spatial information. <CIT> is a priort art document pursuant to Article <NUM>(<NUM>) EPC and discloses a head mounted display that can be used to visually depict a desired tool path for a tool to follow during manual, semi-autonomous, or autonomous movement of a surgical tool. A navigation controller and/or a manipulator controller can store the tool path and its associated location data. This location data is transmitted to the head mounted display controller, which generates a tool path image visually coinciding with the stored tool path such that the head mounted display displays the tool path image to seemingly be located in an actual bone at the actual locations that the working end of the tool (e.g., a bur) will traverse along the tool path.

The invention is defined by the independent claims, with optional features being defined by the dependent claims.

In one aspect, a computer-assisted medical system includes a user device wearable by an operator. The user device includes a display device configured to present imagery overlaid in an environment physically containing a manipulator assembly, and a sensor configured to detect one or more landmarks physically in the environment. The medical system includes a controller configured to execute instructions to perform operations. The operations include receiving, from the sensor, position or orientation information for the one or more landmarks physically in the environment, directing a manual movement of a portion of the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information; and directing a manual movement, a portion of the medical system distinct from the manipulator assembly relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information; or directing a manual movement, of a patient relative to the manipulator assembly by causing the display device to present second imagery overlaid in the environment based on the received position or orientation information.

In another aspect, a method of setting up a computer-assisted medical system including a manipulator assembly physically in an environment is featured. The method includes receiving, from a sensor of a user device of the computer-assisted medical system, position or orientation information for one or more landmarks physically in the environment. The method further includes directing a manual movement of a portion of the manipulator assembly by causing a display device of the user device to present imagery overlaid in the environment based on the received position or orientation information; and directing a manual movement, a portion of the medical system distinct from the manipulator assembly relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information; or directing a manual movement, of a patient relative to the manipulator assembly by causing the display device to present second imagery overlaid in the environment based on the received position or orientation information.

Advantages of the foregoing may include, but are not limited to, those described below and herein elsewhere. The systems and methods described herein can improve accuracy and precision of manual movement of an object in an environment. An operator wearing a user device can easily see a recommended position, orientation, and configuration for the object relative to an actual position, orientation, and configuration for the object, as the imagery is directly overlaid on the environment.

The systems and methods described herein can also improve workflow efficiency and safety. The user device can present imagery to guide tasks to be performed by the operator without drawing the operator's attention away from the environment. The operator can view and interact with the environment while simultaneously viewing guidance provided by the imagery presented by the user device. In implementations in which multiple user devices for multiple operators are present, the operators can easily collaborate with one another to prepare an environment and objects in the environment for a procedure to perform on a workpiece. For example, the operators can interact with the user devices to collaboratively update information presented on the user devices so that information is efficiently propagated to each of the operators. In addition, the operators can track the progress of tasks that other operators are performing, which can thereby make workflow more efficient.

Other potential features, aspects, and advantages will become apparent from the description, the drawings, and the claims.

Although some of the examples described herein refer to surgical procedures or tools, or medical procedures and medical tools, the techniques disclosed apply to medical and non-medical procedures, and to medical and non-medical tools. For example, the tools, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, and sensing or manipulating non-tissue work pieces. Other example applications involve cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down the system, and training medical or non-medical personnel. Additional example applications include use for procedures on tissue removed from human or animal anatomies (without return to a human or animal anatomy), and performing procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that includes, or does not include, surgical aspects.

Starting with a medical example shown in <FIG>, a computer-assisted medical system <NUM> in an environment <NUM> includes a robotic manipulator assembly <NUM> with a robotic manipulator <NUM>. The medical system <NUM> can be operated to perform a procedure on a workpiece, e.g., to perform a medical procedure on a patient <NUM>. One or more operators (e.g., one or more of surgeons, surgical assistants, nurses, technicians, and other medical practitioners) can operate the medical system <NUM> or portions of the medical system <NUM> to perform the surgery.

A configuration of the manipulator assembly <NUM> can be established in preparation for performing the medical procedure on the patient <NUM>. The manipulator assembly <NUM> or portions of the manipulator assembly <NUM> can be manually moved, e.g., repositioned or reoriented, relative to the patient <NUM> such that the manipulator <NUM> of the manipulator assembly <NUM> can be used to perform the medical procedure on the patient <NUM>. In some examples, the manipulator assembly <NUM> has a recommended configuration, e.g., recommended by a controller of the medical system <NUM> based on data such as presets, input data from sensors or users, etc. An operator <NUM> manually moves the manipulator assembly <NUM> into the recommended configuration. The recommended configuration of the manipulator assembly <NUM> can be defined by positions and orientations of individual components of the manipulator assembly <NUM>. In some examples, the manipulator assembly <NUM> may be manually translated in its entirety across a floor surface <NUM> to reposition the manipulator <NUM> relative to the patient <NUM>. In further examples, the manipulator assembly <NUM> can be manually reoriented to reorient the manipulator <NUM> relative to the patient <NUM>. In further examples, the manipulator <NUM> or a portion of the manipulator <NUM> is translated or reoriented in the environment <NUM>. As described herein, a user device <NUM> of the operator <NUM> can direct the manual movement of the manipulator assembly <NUM>, the manipulator <NUM>, a portion of the manipulator assembly <NUM>, or a portion of the manipulator <NUM> by presenting imagery overlaid in the environment <NUM>.

To control the user device <NUM>, a controller <NUM> (shown in <FIG>) receives information pertaining to one or more landmarks in the environment <NUM> to localize the user device <NUM> worn by the operator <NUM>, e.g., using simultaneous localization and mapping (SLAM) techniques. A landmark can correspond to any physical object in the environment <NUM>, such as an operating table <NUM>, the patient <NUM>, another manipulator in the environment <NUM>, other equipment in the environment <NUM>, a feature on a wall surface in the environment <NUM>, a feature on a floor surface in the environment <NUM>, or other unique features that can be used (as described herein) to localize the user device <NUM> in the environment <NUM>. The controller <NUM> then controls the user device <NUM> to present imagery to the operator <NUM> equipped with the user device <NUM>. The imagery can be presented in manner such that the imagery appears, to the operator <NUM> equipped with the user device <NUM>, overlaid in the environment <NUM> based on the received information. This overlaid information is used to direct manual movement facilitated by the operator <NUM> to move the manipulator <NUM> or a portion of the manipulator <NUM> to a recommended position or orientation.

<FIG> and <FIG> depict an example of imagery overlaid in the environment <NUM>. The controller <NUM> causes a display device <NUM> (shown in <FIG>) of the user device <NUM> to present imagery overlaid with a floor surface <NUM> in the environment <NUM>.

The display device <NUM> presents imagery that indicates a path <NUM> along the floor surface <NUM>. For example, the imagery includes a representation of the path <NUM> that appears, to the operator <NUM> wearing the user device <NUM>, to be overlaid with the environment <NUM>, the floor surface <NUM>, or other portions of the environment <NUM>. The path <NUM> is indicative of a recommended path along which the manipulator assembly <NUM> should be manually moved to arrive at its recommended location. The recommended location and the recommended path can be determined based on various forms of input data, e.g., including input data <NUM> described with respect to <FIG>. In some examples, when the manipulator assembly <NUM> is at the recommended location, the manipulator <NUM> can easily access the patient <NUM>. In cases in which the imagery is used to guide repositioning of the manipulator assembly <NUM> in its entirety, the path <NUM> is indicative of a current location <NUM> of the manipulator assembly <NUM> and a recommended location <NUM> of the manipulator assembly <NUM>. The path <NUM> guides repositioning from the current location <NUM> of the manipulator assembly <NUM> toward the recommended location <NUM> of the manipulator assembly <NUM>. The operator <NUM> manually moves the manipulator assembly <NUM> in its entirety along the path <NUM> from the current location <NUM> to the recommended location <NUM>.

In some implementations, in addition to being indicative of the current location <NUM> and the recommended location <NUM>, the path <NUM> is indicative of multiple waypoints <NUM> along the floor surface <NUM>. These recommended waypoints <NUM> for the manipulator assembly <NUM> are selected such that the manipulator assembly <NUM> is kept away from contacting other objects in the environment <NUM> when the manipulator assembly <NUM> is manually moved along the path <NUM>. In one example, an obstacle, e.g., a chair <NUM>, is located in the environment <NUM> between the manipulator assembly <NUM> and the patient <NUM>. The waypoints <NUM> are selected so that the manipulator assembly <NUM> is maneuvered away from the chair <NUM>. In some implementations, the imagery presented by the user device <NUM> includes an indicator <NUM> that the chair <NUM> is proximate to the path <NUM>. This indicator <NUM> notifies the operator <NUM> of potential obstacles with which the manipulator assembly <NUM> could collide when moved along the path <NUM>. <FIG> depicts four waypoints <NUM>, but fewer or more waypoints can be present in other implementations.

As shown in <FIG>, the imagery includes a map <NUM> indicative of desirable locations and undesirable locations for the manipulator assembly <NUM>. The map <NUM> can be overlaid on the floor surface <NUM> and the environment <NUM> so that the operator <NUM> equipped with the user device <NUM> can easily see where the desirable locations for the manipulator assembly <NUM> are in the environment <NUM>. A region <NUM> of the map <NUM> is indicative of the desirable locations, while a region <NUM> of the map <NUM> is indicative of the undesirable locations. The region <NUM> is positioned proximate the patient <NUM> and the operating table <NUM> and is selected by the controller <NUM> to include locations determined to be easily accessible by the manipulator assembly <NUM>. The region <NUM> corresponds to locations that would be undesirable for the manipulator assembly <NUM>. In some implementations, rather than indicating all undesirable locations of the manipulator assembly <NUM>, the region <NUM> indicates locations that are near the patient <NUM> and the operating table <NUM> but that would be undesirable because, for example, the manipulator <NUM> would be too far from the patient <NUM> and the operating table <NUM> or would be near an obstacle with which the manipulator <NUM> could collide during a procedure.

In some implementations, to produce the imagery including the regions <NUM>, <NUM>, the controller <NUM> determines a desirability value of each potential location for the manipulator assembly <NUM>. The controller <NUM> then designates locations having a desirability value above a predefined threshold as being desirable locations, e.g., corresponding to the locations in the region <NUM>, and designates locations having a desirability value less than or equal to the predefined threshold as being undesirable locations, e.g., corresponding to the locations in the region <NUM>. In some examples, rather than showing two discrete regions, the map <NUM> is a heat map that is indicative of a desirability value for each potential location. The map <NUM> can include a color-code representation of the desirability values. If the map <NUM> is a heat map, the map <NUM> can be indicative of more than two regions, each of the regions being indicative of a different predefined range of desirability values. The desirable locations, the undesirable locations, the waypoints <NUM>, the indicator <NUM>, the map <NUM>, the region <NUM>, and the region <NUM> can be generated based on various forms of input data, e.g., including the input data <NUM> described with respect to <FIG>.

<FIG> depicts an example of the manipulator assembly <NUM> that is movable across the floor surface <NUM> (shown in <FIG> and <FIG>). In addition to including the manipulator <NUM>, the manipulator assembly <NUM> includes a support structure <NUM> that supports the manipulator <NUM> above the floor surface <NUM>. The support structure <NUM> is translatable and orientable relative to the floor surface <NUM>. For example, the support structure <NUM> includes wheels <NUM>, e.g., caster wheels, that enable the operator <NUM> (shown in <FIG> and <FIG>) to manually reposition or reorient the support structure <NUM> relative to the patient <NUM> (shown in <FIG> and <FIG>). The support structure <NUM> is connected to the manipulator <NUM> and supports the manipulator <NUM> at a height above the floor surface <NUM>. In the example process of guiding manual movement of a portion of the manipulator assembly <NUM> described with respect to <FIG> and <FIG>, the portion of the manipulator assembly <NUM> for which movement is guided can correspond to the support structure <NUM>. In particular, the operator <NUM> can manually reposition the manipulator assembly <NUM> in its entirety by manually moving the support structure <NUM>. In other examples, the portion can correspond to the manipulator <NUM>, or another portion of the manipulator assembly <NUM>.

The position or orientation of the manipulator <NUM> can be manually adjusted through other mechanisms. In one example, the height of the manipulator <NUM> above the floor surface <NUM> is adjustable. The manipulator <NUM> can be vertically movable relative to the support structure <NUM>. In another example, the manipulator <NUM> can be reoriented relative to the support structure <NUM>. The support structure <NUM> can include one or more passive joints about which the manipulator <NUM> can be rotated. In the example shown in <FIG>, the support structure <NUM> can include a passive setup arm <NUM> connecting the manipulator <NUM> to a column <NUM> of the support structure <NUM>. The passive setup arm <NUM> includes a series of passive links and joints that can be manually repositioned and reoriented. The passive setup arm <NUM> can be vertically translated relative to the column <NUM>, thereby vertically repositioning the manipulator <NUM> relative to the column <NUM> of the support structure <NUM>.

A base <NUM> of the manipulator <NUM> is connected to the support structure <NUM>, e.g., to the passive setup arm <NUM> of the support structure <NUM>. The manipulator <NUM> includes one or more joints and one or more links that are operable to move an instrument holder <NUM> that is configured to hold an instrument <NUM>. The one or more links of the manipulator <NUM> extend distally from the base <NUM> of the manipulator <NUM>. For example, the manipulator <NUM> includes joints <NUM> and links <NUM>, and one or more of the joints <NUM> are powered joints that can be controlled by a controller <NUM> (shown in <FIG>). In some implementations, one or more of the joints <NUM> are passive joints. By driving the joints <NUM>, the instrument holder <NUM> with the instrument <NUM> can be repositioned relative to the patient <NUM> or the environment <NUM> (shown in <FIG>). In preparation for a procedure, the base <NUM> of the manipulator <NUM> can be repositioned to a desirable location so that desired ranges of motion of the links <NUM> and the joints <NUM> of the manipulator <NUM> can be achieved. In this regard, in certain examples as described herein, the controller <NUM> (shown in <FIG>) can cause the user device <NUM> (shown in <FIG>) to present imagery to direct manual movement of the manipulator <NUM>, a link <NUM>, a joint <NUM>, the base <NUM>, or another portion of the manipulator <NUM>.

<FIG> shows an example of the user device <NUM> worn by the operator <NUM>. The user device <NUM> includes a display device <NUM> and a sensor <NUM>. The display device <NUM> presents imagery to the operator <NUM>. For example, in the example shown in <FIG>, the user device <NUM> is a wearable head-mounted display device that can be worn over eyes of the operator <NUM>. The display device <NUM> includes a see-through display that presents imagery. At least some of the imagery can be overlaid in the environment <NUM> when the user device <NUM> is worn over the eyes of the operator <NUM>. At least some of the imagery can be transparent such that, when overlaid on a portion of the environment <NUM>, this portion of the imagery and the portion of the environment <NUM> are both visible to the operator <NUM>. In some implementations, at least some of the imagery overlaid on a portion of the environment <NUM> can be opaque such that the portion of the environment <NUM> is not visible to the operator <NUM> but this portion of the imagery is visible to the operator <NUM>. A view frame of the display device <NUM> is in front of the eyes of the operator <NUM> so that imagery presented by the display device <NUM> appears overlaid on the portion of the environment <NUM> seen by the operator <NUM>. The operator <NUM> thus simultaneously sees the environment <NUM> as well as any overlaid imagery that is presented on the display device <NUM>.

The sensor <NUM> is configured to detect one or more landmarks in the environment <NUM> to generate information indicative of a position and/or orientation of one or more landmarks in the environment <NUM>. The sensor <NUM> can generate one or more signals in response to detecting the one or more landmarks in the environment, and the one or more signals can be processed and analyzed, e.g., by the controller <NUM>, to generate the information indicative of the position and/or orientation of the one or more landmarks. In some examples, the sensor <NUM> includes an image capture device that captures imagery of the environment <NUM>, including any landmarks in the environment <NUM>. The position and orientation information, using SLAM techniques, can be used to localize the user device <NUM> and hence the display device <NUM> relative to the environment <NUM> and other features within the environment <NUM> such that the imagery presented by the display device <NUM> can be overlaid in a manner that is meaningful to the operator <NUM> wearing the user device <NUM>. In particular, from the perspective of the operator <NUM>, the imagery appears to be overlaid on portions of the environment <NUM> so that the operator <NUM> can easily use the imagery as guidance for interacting with the environment <NUM>.

<FIG> shows an example diagram of the medical system <NUM> that can be used for guiding the manual movement of the manipulator assembly <NUM>. The medical system <NUM> includes the user device <NUM>, manipulator assembly <NUM>, and the controller <NUM>. The controller <NUM> includes one or more computer processors. In some implementations, the controller <NUM> corresponds to a combination of processors of the manipulator assembly <NUM>, the user device <NUM>, and other systems of the medical system <NUM>. The controller <NUM> directs operations of the various systems of the medical system <NUM>.

In some implementations, the medical system <NUM> further includes a user control system <NUM> (also shown in <FIG>). The user control system <NUM> includes a user input system and a user output system. The user input system of the user control system <NUM> is operable by one of the operators <NUM> to control movement of the manipulator assembly <NUM>. In some cases, a user device receives user commands from the operator <NUM> to move the teleoperated manipulator <NUM>. In some cases, the user input system is manually operable such that manual operation of the user input system results in corresponding movement of the manipulator assembly <NUM>. The user input system can include one or more of foot pedals with either or both of toe and heel controls, one or more joysticks, or other manually operable user input devices. In some cases, the user input system includes an image capture device or other sensor that can detect user motion. The user input system generates control signals to control movement of the manipulator assembly <NUM> based on the detected user motion. The user output system of the user control system <NUM> provides imagery of the worksite to the operator operating the user control system <NUM>.

In cases in which the medical system <NUM> includes the user control system <NUM>, the guidance of manual movement of the manipulator assembly <NUM> provided by the controller <NUM> can include guidance for operating the user input system of the user control system <NUM> to move the manipulator assembly <NUM>. For example, if the user input system includes a user input device, the operator <NUM> manually operates the user input device to manually move the manipulator assembly <NUM>. Alternatively, if the user input system includes an image capture device or another sensor that detects operator motion, the controller <NUM> operates the user device <NUM> to present imagery that guides the operator to move in a certain manner that causes the manipulator assembly <NUM> or a portion thereof to move.

The medical system <NUM> includes a sensor system <NUM>, including the sensor <NUM> of the user device <NUM>, that can detect features of the environment <NUM>. The data produced by the sensor system <NUM> can be used with SLAM techniques to localize the user device <NUM> within the environment <NUM>. With the data provided by the sensor system <NUM>, a pose of the user device <NUM>, e.g., a position and an orientation of the user device <NUM>, relative to the environment <NUM> can be determined.

The sensor system <NUM> produces position or orientation information for one or more landmarks extracted from signals generated by the sensor system <NUM>. A landmark is a unique signal or set of signals generated by the sensor system <NUM> that can be distinguished from other signals that could be generated by the sensor system <NUM> as the sensor system <NUM> detects different features within the environment <NUM>. For example, in cases in which the sensor system <NUM> includes the sensor <NUM> of the user device <NUM> and the sensor <NUM> is an image capture device, a landmark can correspond to a unique visual feature in the environment <NUM> that generally does not change position or orientation relative to the environment <NUM>, such as the operating table <NUM>, other equipment in the environment <NUM>, a corner of a room, or another unique visual feature. When such a visual feature is observed by the sensor <NUM>, information received by the controller <NUM> from the sensor <NUM> is indicative of an orientation or a position of the landmark relative to the sensor <NUM>, thus enabling the controller <NUM> to use SLAM techniques to determine a position or an orientation of the sensor <NUM> and hence the user device <NUM> relative to the environment <NUM>.

The sensor system <NUM> can include one or more sensors in addition to the sensor <NUM> of the user device <NUM>. For example, the medical system <NUM> includes one or more of a kinematic sensor <NUM>, a patient sensor <NUM>, an instrument sensor <NUM>, an obstacle sensor <NUM>, or an image capture device <NUM>. Output data produced by the sensors of the sensor system <NUM> can be used by the controller <NUM> to localize the user device <NUM> in the environment <NUM>. In some implementations, the output data can be used to provide other information to the operator <NUM> through the user device <NUM>. In some examples described herein, the output data are used for determining a recommended configuration for the manipulator assembly <NUM> and hence for generating the imagery for guiding the manual movement of the manipulator assembly <NUM> toward the recommended configuration. In other examples, the output data are used to provide information related to a status of a certain subsystem of the medical system <NUM>, a certain operator in the environment <NUM>, the patient <NUM>, or another object in the environment <NUM>.

The kinematic sensor <NUM> can be a kinematic sensor of the manipulator assembly <NUM>. For example, the kinematic sensor <NUM> can detect a pose of the joints <NUM> or the links <NUM> of the manipulator <NUM>. In some cases, the kinematic sensor <NUM> is configured to detect a pose of the instrument holder <NUM> such that a position and orientation of the instrument <NUM> can be determined. The kinematic sensor <NUM> can be an accelerometer, a gyroscope, an encoder, a torque sensor, a force sensor, or other type of sensor that can detect motion of one of the joints <NUM> or the links <NUM> of the manipulator <NUM>. In some examples, the manipulator assembly <NUM> includes a single kinematic sensor, whereas in other implementations, the manipulator assembly <NUM> includes two or more kinematic sensors.

The patient sensor <NUM> is configured to detect a characteristic of the patient <NUM> (shown in <FIG>). For example, the patient sensor <NUM> can be a patient motion sensor that detects when the patient <NUM> moves, e.g., relative to the environment <NUM> or relative to an operating table <NUM> (shown in <FIG>) on which the patient <NUM> is positioned. In some cases, the patient sensor <NUM> includes an image capture device or an optical sensor, e.g., mounted in the environment <NUM> or mounted to the manipulator <NUM>, that detects whether the patient <NUM> is positioned on the operating table <NUM>. In some cases, the patient sensor <NUM> includes an accelerometer or other motion sensor attached to the patient <NUM> that detect movement of the patient <NUM>. In other cases, rather than detecting a motion or position of the patient <NUM>, the patient sensor <NUM> detects another physical characteristic of the patient <NUM>, such as a weight or a size of the patient <NUM>, a blood pressure of the patient <NUM>, a heart rate of the patient <NUM>.

The instrument sensor <NUM> is configured to detect a characteristic of the instrument <NUM> mounted to the instrument holder <NUM>. The instrument sensor <NUM>, for example, is a sensor on the instrument holder <NUM> that detects whether an instrument has been mounted to the instrument holder <NUM>. In some implementations, the instrument sensor <NUM> detects a type of instrument <NUM> mounted to the instrument holder <NUM>. For example, the instrument sensor <NUM> can be a sensor of the manipulator assembly <NUM> that detects an identity indicated in an EEPROM of the instrument <NUM>. In some implementations, the instrument sensor <NUM> detects motion of the instrument <NUM>. For example, the instrument sensor <NUM> includes an accelerometer, a gyroscope, a force sensor, a torque sensor, or other type of sensor mounted to the instrument <NUM> or the instrument holder <NUM> to detect motion of the instrument <NUM>.

The obstacle sensor <NUM> is configured to detect obstacles in the environment <NUM>. The obstacle sensor <NUM>, in some cases, is an optical or acoustic proximity sensor that detects when obstacles are near the manipulator assembly <NUM>. The obstacle sensor <NUM>, for example, is mounted to the manipulator assembly <NUM> and is able to detect when an obstacle within a predetermined distance of the manipulator assembly <NUM>. In some implementations, the obstacle sensor <NUM> is an image capture device mounted in the environment <NUM> and configured to detect when obstacles are moved win the vicinity of the manipulator assembly <NUM>. The obstacles can include objects such as other equipment of the medical system <NUM>, other equipment within the environment <NUM>, persons in the environment <NUM>, or other objects in the environment <NUM>. The obstacle sensor <NUM> can include contact sensors, proximity sensors, optical time-of-flight sensors, and other sensors appropriate for detecting contact with an obstacle or a distance of an obstacle.

The image capture device <NUM> can correspond to one of the image capture devices described with respect to one of the other sensors described herein, e.g., one of the sensors <NUM> of the user devices <NUM>, the kinematic sensor <NUM>, the patient sensor <NUM>, the instrument sensor <NUM>, the obstacle sensor <NUM>, or another sensor. The image capture device <NUM> is positioned within the environment <NUM> to capture imagery of the environment <NUM>. In some cases, one of the user devices <NUM> includes the image capture device <NUM>. In other cases, the image capture device <NUM> is fixed a part of the environment <NUM>, e.g., a wall, a ceiling, or other fixture in the environment <NUM>. The image capture device <NUM> can be an optical sensor, e.g., a camera, or an acoustic sensor.

<FIG> depicts a process 500A for guiding manual movement of a portion of the manipulator assembly <NUM> (shown in <FIG> and <FIG>). As described herein, the portion of the manipulator assembly <NUM> can correspond to the support structure <NUM> of the manipulator assembly <NUM>. The process 500A can be performed by the controller <NUM> described herein. At operation 502A, the controller <NUM> receives position or orientation information for one or more landmarks in an environment. For example, the controller <NUM> receives this information from one or more sensors, such as those described with respect to the sensor system <NUM> or with respect to the sensor <NUM> of the user device <NUM>.

At operation 504A, the controller <NUM> directs manual movement of the portion of the manipulator assembly <NUM>. The controller <NUM> causes the display device <NUM> of the user device <NUM> worn by the operator <NUM> (described with respect to <FIG>, <FIG>, and <FIG>) to present imagery overlaid in the environment <NUM> based on the position or orientation information received at the operation 502A. The imagery can be presented to guide the operator <NUM> to manually move the portion of manipulator assembly <NUM> toward a recommended configuration that is determined based on the position or orientation information received at the operation 502A. The imagery can also be presented such that the imagery is positioned or oriented relative to the environment <NUM> based on the position or orientation information received at the operation 502A. In some examples, the imagery indicates a recommended path of movement for the portion of the manipulator assembly <NUM>, e.g., as described with respect to <FIG> and <FIG>. As described herein, other examples are possible.

<FIG> depicts another process 500B for updating imagery presented to the operator <NUM>. In some cases, the process 500B is executed after the operation 502A of the process 500A is executed such that imagery has already been presented to the operator <NUM> through the user device <NUM> (shown in <FIG>). In the process 500B, at operation 502B, the controller <NUM> receives updated position or orientation information for the one or more landmarks in the environment, e.g., for the one or more landmarks in the environment for which information was received at the operation 502A. This updated position or orientation information can be indicative of movement of the operator <NUM>, and hence the user device <NUM>, relative to the one or more landmarks.

At operation 504B, the controller <NUM> updates the presented imagery based on the updated position or orientation information. For example, if the user device <NUM> has been moved relative to the one or more landmarks, the presented imagery can be updated such that the position or orientation of the imagery relative to the one or more landmarks is maintained even though the user device <NUM> has moved. In this regard, if a portion of the imagery is overlaid on an object in the environment, the presented imagery is updated so that the portion of the imagery remains overlaid on the object when the user device is moved relative to the object.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made to the processes, systems, and mechanisms described herein.

Some implementations described herein are described with respect to medical examples. In other implementations, the medical system <NUM> is a surgical system for performing a surgical procedure on the patient <NUM>. The techniques disclosed herein are also applicable to non-surgical use. For example, they may be used with and improve general or industrial robotic operations, such as those use in manipulating work pieces. These techniques may also be used with and improve medical robotic operations for diagnoses and non-surgical treatment.

The specific examples presented in this disclosure can be applicable to teleoperational robotic systems and remotely controllable arms. The techniques disclosed herein are also applicable to robotic systems that are, in part or in whole, directly and manually moved by operators. For example, these techniques can be applied to robotic systems designed to help steady an instrument held by the manipulator <NUM> while the instrument is manipulated by hand of an operator. As another example, any of the controllable manipulators discussed herein may be configured to allow direct manipulation, and accept operator instruction through input directly applied to a link or a joint of the manipulator. The techniques are also applicable to robotic systems that are, in part or in whole, automatically moved.

An operator can manually move an object, e.g., such as part or all of the manipulator assembly <NUM>, by applying a force directly on the object, e.g., using a hand, a foot, or other body part for applying the force directly on the object. The operator can manually or push the object to reposition or reorient the object. In other examples, the operator can manually move the object by interacting with a user input device that causes the object to move. For example, if the object to be manually moved is the manipulator assembly <NUM>, the wheels <NUM> of the support structure <NUM>, e.g., can be powered wheels that can be controlled by the user input device. In this regard, the manual movement of the manipulator assembly <NUM> directed by the user device <NUM> can correspond to manual movement that is generated in response to manual manipulation of a user input device separate from the manipulator assembly <NUM>.

While the setup arm <NUM> is described as being passive in the above example, in other implementations, the setup arm <NUM> is an active controllable setup arm. For example, the setup arm can be moved in response to operation of the user input system described herein. In some implementations, the setup arm can be backdriven through operation of a powered joint of the manipulator <NUM>. For example, a distal portion of the manipulator <NUM> can be fixed, and the powered joint can be operated to backdrive the setup arm, thereby repositioning or reorienting the setup arm.

The support structure <NUM> is described as including the wheels <NUM>. In some implementations, rather than including wheels, the support structure <NUM> is mounted in the environment <NUM> in a manner that enables the support structure <NUM> to be easily moved in the environment <NUM>. For example, the support structure <NUM> could be directly mounted to the operating table <NUM>, directly mounted to walls of the environment <NUM>, or directly mounted to a ceiling of the environment <NUM>.

In some implementations, the user device <NUM> is worn by the operator <NUM> over the eyes of the operator <NUM>. The user device <NUM> can be a head-mounted user device, and the display device <NUM> of the user device <NUM> can be a see-through display, as described with respect to <FIG>. In other implementations, rather than being a see-through display device, the display device <NUM> can be an opaque display device that is substantially opaque to light. The user device <NUM> as a virtual reality device. To allow the operator <NUM> to view the environment <NUM>, the sensor <NUM> of the user device <NUM> can capture imagery of the environment <NUM>, and the display device <NUM> can present the imagery of the environment <NUM>. In some cases, the controller <NUM> can generate imagery of the environment <NUM> with one or more indicators overlaid on the imagery for guiding the manual movement of the manipulator assembly <NUM>.

While the user device <NUM> is described as being a head-mounted device with a see-through display, the user device can vary in other implementations. In other implementations, the user device <NUM> can be carried by the operator <NUM> in other manners. The user device can be a mobile computing device such as a tablet computer, a smart watch, or a smartphone. For example, the user device <NUM> can be a smart watch worn on the wrist of the operator <NUM>. The mobile computing device is a handheld computing device that the operator <NUM> can easily carry around the environment <NUM> using a hand. Using augmented reality processes, imagery presented on a display device of the mobile computing device can be overlaid on imagery of the environment <NUM>. In this regard, at the operation 504A, 504B, rather than directly overlaying the imagery over the environment <NUM>, the imagery for guiding the manual movement of the manipulator assembly <NUM> is overlaid on imagery of the environment <NUM>. The imagery of the environment <NUM> can be captured by an image capture device of the mobile computing device, or an image capture device of another part of the medical system <NUM>.

While a single user device <NUM> is described with respect to <FIG> and <FIG>, in other implementations, multiple operators with multiple user devices can be present in the environment <NUM>. The environment <NUM> includes any number of operators. Referring back to <FIG>, each of the operators has a corresponding user device <NUM>-<NUM>, <NUM>-<NUM>,. , <NUM>-n (collectively referred to as user devices <NUM>). In some implementations, each of the user devices <NUM> includes a corresponding display device. The multiple user devices <NUM> can each include a corresponding display device <NUM>-<NUM>, <NUM>-<NUM>,. , <NUM>-n (collectively referred to as display devices <NUM>). In some implementations, the user devices <NUM> also include corresponding sensors <NUM>-<NUM>, <NUM>-<NUM>,. , <NUM>-n (collectively referred to as sensors <NUM>). Each of the sensors <NUM> can be similar to the sensor <NUM> described with respect to <FIG>. For example, one or more of the sensors <NUM> can includes an image capture device. In other implementations, the sensors <NUM> can include any one of an accelerometer, a motion sensor, a gyroscope, or another type of sensor. In implementations in which multiple user devices <NUM> are present, the sensor system <NUM> can include any of the sensors <NUM> of the multiple user devices <NUM>. In this regard, the controller <NUM> can receive the data produced by the sensors <NUM> and use the data for the processes described herein. For example, if the one of the user devices <NUM> includes a corresponding sensor <NUM>, output from the sensor <NUM> can be used to provide information that can be used to localize another of the user devices <NUM>, to determine a recommended configuration of the manipulator <NUM>, or to perform other operations described herein.

In the example shown in <FIG>, the environment <NUM> includes four operators <NUM>-<NUM> (i.e., the operator <NUM>), <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> (collectively referred to as operators <NUM>). One or more of the operators <NUM> can carry a user device. For example, in some implementations, only a portion of the operators <NUM> in the environment <NUM> carry user devices. In the example of <FIG>, the operators <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> each carries a corresponding user device <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>. The operator <NUM>-<NUM> operates the user control system <NUM>. One or more of the user devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> can include a corresponding sensor. In some cases, each of the user devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> includes a corresponding sensor, whereas in other cases, one or two of the user devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> includes a corresponding sensor. Similarly, in some cases, each of the user devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> includes a corresponding display device, whereas in other cases, one or two of the user devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> includes a corresponding display device.

In some implementations, imagery presented on one of the user devices <NUM> can correspond to imagery captured by one of the other user devices <NUM>. For example, in implementations in which the medical system <NUM> includes the user device <NUM>-<NUM> carried by the operator <NUM>-<NUM> and the second user device <NUM>-<NUM> carried by the operator <NUM>-<NUM>, in the operations 504A, 504B (shown in <FIG>), imagery presented on the user device <NUM>-<NUM> can correspond to imagery captured by the sensor <NUM>-<NUM> (schematically shown in <FIG>) of the user device <NUM>-<NUM>. If the user device <NUM>-<NUM> is a head-mounted device, the sensor <NUM>-<NUM> of the user device <NUM>-<NUM> can capture imagery of the portion of the environment <NUM> seen by the operator <NUM>-<NUM>. For example, the sensor <NUM>-<NUM> can capture imagery of an equipment table <NUM> in front of the operator <NUM>-<NUM>. In this regard, the display device <NUM>-<NUM> (schematically shown in <FIG>) of the user device <NUM>-<NUM> can present imagery of the operator <NUM>-<NUM> preparing the equipment on the equipment table <NUM>. In some implementations, the operator <NUM>-<NUM> can operate the user device <NUM>-<NUM> to cause the display device <NUM>-<NUM> to present the imagery being captured by the user device <NUM>-<NUM> of the operator <NUM>-<NUM>. This allows the operator <NUM>-<NUM> to easily determine a status of the operator <NUM>-<NUM> or a status of task being performed by the operator <NUM>-<NUM>.

In some cases, the display device <NUM>-<NUM> can present an indicator indicative of the status of the operator <NUM>-<NUM> or the status of the task being performed by the operator <NUM>-<NUM>. For example, the indicator can indicate that the operator <NUM>-<NUM> is moving a particular piece of equipment, is preparing an instrument for use, or is performing some other operating in preparation for the procedure to be performed on the workpiece. While the user device <NUM>-<NUM> is described as presenting imagery captured by the sensor <NUM>-<NUM> (schematically shown in <FIG>) of the user device <NUM>-<NUM>, in other implementations, the display device <NUM>-<NUM> (schematically shown in <FIG>) of the user device <NUM>-<NUM> can also or instead present imagery captured by the sensor <NUM>-<NUM> of the user device <NUM>-<NUM>. Furthermore, in some implementations, the display device <NUM>-<NUM> can present an indicator indicative of a status of the operator <NUM>-<NUM> or a status of task being performed by the operator <NUM>-<NUM>.

In one example, as shown in <FIG>, the operator <NUM>-<NUM> is moving the manipulator assembly <NUM>, and the operator <NUM>-<NUM> is preparing an instrument for use during the procedure on the patient <NUM>. In such cases, if both the user device <NUM>-<NUM> and the user device <NUM>-<NUM> includes corresponding sensors and display devices. The display device <NUM>-<NUM> of the user device <NUM>-<NUM> can be operated to present an indicator indicative of a status of the equipment preparation being performed by the operator <NUM>-<NUM>, and the display device <NUM>-<NUM> of the user device <NUM>-<NUM> can be operated to present an indicator indicative of a status of the manual movement operation being performed by the operator <NUM>-<NUM>.

In some implementations, not all of the user devices <NUM> includes a corresponding sensor or a corresponding display device. For example, the user device <NUM>-<NUM> includes the sensor <NUM>-<NUM> and the display device <NUM>-<NUM>, and the user device <NUM>-<NUM> includes the sensor <NUM>-<NUM> but does not include a display device. In such cases, imagery presented on the display device <NUM>-<NUM> is generated based on data generated by the sensor <NUM>-<NUM> and data generated by the sensor <NUM>-<NUM>. In other examples, the user device <NUM>-<NUM> includes the sensor <NUM>-<NUM> and the display device <NUM>-<NUM>, and the user device <NUM>-<NUM> includes the display device <NUM>-<NUM> but does not include a sensor. In such cases, the imagery presented by the display device <NUM>-<NUM> can be generated based on data collected by sensors of the sensor system <NUM> rather than a sensor on the user device <NUM>-<NUM>.

In some implementations, if both the user device <NUM>-<NUM> and the user device <NUM>-<NUM> include display devices, the imagery presented by the user device <NUM>-<NUM> and the imagery of the user device <NUM>-<NUM> can be similar to one another. The imagery presented by the user device <NUM>-<NUM> and the imagery of the user device <NUM>-<NUM> can both be overlaid on the environment <NUM>. In some cases, because the user devices <NUM>-<NUM>, <NUM>-<NUM> are positioned and oriented differently from one another, the perspective of the imagery presented by the user device <NUM>-<NUM> may differ from the perspective of the imagery presented by the user device <NUM>-<NUM>. The imagery presented by the user device <NUM>-<NUM> and the imagery presented by the user device <NUM>-<NUM> can be positioned and oriented relative to a single global reference frame. The user device <NUM>-<NUM> and the user device <NUM>-<NUM> may both present imagery that indicates the path <NUM> along which the manipulator assembly <NUM> should be moved. The representation of the path <NUM> is overlaid over the same portion of the environment <NUM>, e.g., the same portion of the floor surface <NUM>, in both the imagery presented by the user device <NUM>-<NUM> and the imagery presented by the user device <NUM>-<NUM>.

In some implementations, the user device <NUM>-<NUM> is operated to guide the manual movement of the manipulator assembly <NUM>, and another user device is operated to guide manual movement of another part of the medical system <NUM>. For example, the user device <NUM>-<NUM> carried by the operator <NUM>-<NUM> can direct a manual movement of an auxiliary system170 (shown in <FIG>). In some implementations, the auxiliary system <NUM> is a system that provides vision or image processing capabilities to present imagery of, for example, internal anatomy of the patient <NUM>. The user device <NUM>-<NUM> can present imagery that guides the manual movement of the auxiliary system <NUM> in the environment <NUM>. Other examples of parts for which manual movement can be guided are described herein. The imagery presented on the user device <NUM>-<NUM> can be similar to the imagery described with respect to the user device <NUM> for guiding the manual movement of the manipulator assembly <NUM>.

In implementations in which the user device <NUM>-<NUM> and the user device <NUM>-<NUM> both include sensors, the sensors <NUM>-<NUM>, <NUM>-<NUM> of the user devices <NUM>-<NUM>, <NUM>-<NUM> can both detect landmarks in the environment <NUM> for localizing the user devices <NUM>-<NUM>, <NUM>-<NUM>. For example, as the user device <NUM>-<NUM> is moved about the environment <NUM>, a first landmark can be extracted from data produced by the sensor <NUM>-<NUM>. As the user device <NUM>-<NUM> is moved about the environment <NUM>, a second landmark can be extracted from data produced by the sensor <NUM>-<NUM>. The first landmark extracted can correspond to a physical feature in the environment <NUM> that is distinct from a physical feature corresponding to the second landmark. For example, if the sensors <NUM>-<NUM>, <NUM>-<NUM> are image capture devices, the first landmark could correspond to a portion of imagery representing the user control system <NUM> captured by the sensor <NUM>-<NUM>, and the second landmark could correspond to a portion of imagery representing the operating table <NUM> captured by the sensor <NUM>-<NUM>. In other implementations, the landmarks extracted from the data of the sensors <NUM>-<NUM>, <NUM>-<NUM> can correspond to the same physical feature in the environment <NUM>. While a first landmark and a second landmark are described, in other implementations, each of the sensors <NUM>-<NUM>, <NUM>-<NUM> of the user devices <NUM>-<NUM>, <NUM>-<NUM> can collect data from which multiple landmarks are extracted.

In the operation 502A and the operation 502B, position or orientation information are received so that the controller <NUM> can generate or update the imagery to provide the operator with guidance in manually moving the manipulator <NUM>. A reference frame for the position or orientation information can vary in different implementations. In implementations in which the controller <NUM> determines a pose of the user device <NUM>, the reference frame for the position or orientation information for the user device <NUM> can be the same as the reference frame for the position or orientation information for the one or more landmarks. In other implementations, the reference frame for position or orientation information for a landmark differs from the reference frame for position or orientation information for the user device <NUM>.

In some implementations, other information in addition to the position or orientation information can be used to generate or update the imagery. The controller <NUM> receives, for example, other contextual information pertaining to objects in the environment <NUM> besides the one or more landmarks. <FIG> shows examples of contextual information usable by the controller <NUM> to generate or update imagery presented on the user device <NUM>. The contextual information can include equipment information, operator information, obstacle information, and patient information. For example, input data <NUM> are used by the controller <NUM> to produce an output <NUM>, e.g., signals for causing the user device <NUM> to present imagery to guide manual movement of the manipulator assembly <NUM> or a portion thereof. The input data <NUM> include data loaded into memory associated with the controller <NUM>, user-specified data, data generated by the sensor system <NUM>, etc. The input data <NUM> include, for example, procedure data 600a, equipment data 600b, pose data 600c, operator data 600d, obstacle data 600e, workpiece data 600f, and port data <NUM>.

The data 600a, 600b, 600c, 600e, 600f, <NUM> represent some examples of the data usable by the controller <NUM> to control the medical system <NUM> and to generate the imagery presented on the user device <NUM>. Other types and contents of data may be appropriately used by the controller <NUM> to control the medical system <NUM> or to control the imagery presented on the user device <NUM>. In addition, while described as input data <NUM>, in some implementations, some data of the input data <NUM> are generated from other data of the input data <NUM>. Furthermore, while the operation 502A, 502B are described with respect to generating imagery for directing manual movement of a portion of the manipulator assembly <NUM>, the input data <NUM> can be used for generating imagery to provide other information to the operator <NUM> as described herein.

The procedure data 600a include data indicative of the specific procedure to be performed. For example, in a medical example, the procedure data 600a include data indicative of the medical procedure to be performed on the patient. In some implementations, an operator selects the type of the procedure before the controller <NUM> directs the manual movement of the manipulator assembly <NUM>. The controller <NUM> then generates the imagery presented on the user device <NUM> based on a type of a medical procedure to be performed by the manipulator <NUM> of the manipulator assembly <NUM>.

The procedure data 600a can refer to specific requirements of a workspace, such as an area around the workspace that the instrument <NUM> should be able to access. In one example, the type of the procedure can indicate the specific workspace requirements. The specific type of the procedure may require a predetermined extent of the workspace. In a medical or a surgical example, the workspace can correspond to an area around the patient <NUM> that the instrument <NUM> should be able to access during the surgery, due to the specific medical or surgical procedure to be performed on the patient <NUM>.

In another example, the procedure data 600a are generated when the operator selects the extent of the workspace before the manual movement is performed. The operator can select the extent of the workspace in any number of ways. For example, the operator may input the data by highlighting or tracing, on a representation of a workspace presented on a user interface device, a region representative of the desired extent of the workspace. The operator can indicate a boundary of the desired extent of the workspace.

As another example, the desired extent of the workspace can be indicated by prior operator-directed motion of the manipulator <NUM>, e.g., a distal portion of the manipulator <NUM>. An operator can move the manipulator <NUM> (with or without an instrument being held) to indicate the workspace desired, or by moving a substitute of the instrument <NUM> to indicate the workspace desired. Example substitutes of the instrument <NUM> include a device that represents an average instrument that may be used during the procedure, a device that replicates a proximal portion of the instrument <NUM> but not the entire shaft and end effector, a device that projects a visual indication of locations associated with distal ends of instruments that may be used during the procedure, etc. An image capture device or other sensor can detect the movement of the manipulator <NUM>. If the user interface device corresponds to the user device <NUM>, the sensor <NUM> of the user device <NUM> can detect the movement of the manipulator <NUM>. In some implementations, the controller <NUM> can determine range of motion limits based on signals generated by the sensors associated with the joints <NUM> of the manipulator <NUM> when the operator moves the manipulator <NUM>.

Information about a desired setup configuration of the manipulator assembly <NUM> or the medical system <NUM> can be derived at least in part from such a demonstration. For example, the desired range of motion of the joints of the manipulator <NUM> or the instrument <NUM>, and hence the desired location or orientation for the manipulator <NUM>, can be derived at least in part from such a demonstration. Pose sensors of a sensor system <NUM>, for example, can provide data indicative of configurations of the manipulator <NUM>, configurations of the instrument <NUM>, or other system characteristics detectable during the manual demonstration of the desired workspace. The sensor system <NUM> can thus provide information about the desired range of motion of joints of the manipulator <NUM> or the instrument <NUM>, or of the desired motion envelope. The controller <NUM> or other computing system can then process this sensor information to determine the extent of the workspace demonstrated by the operator.

In some implementations, the procedure data 600a can include a plan indicative of a desired or a recommended setup configuration for the manipulator assembly <NUM> or the medical system <NUM>. For example, the plan can be indicative of positions or orientations of objects in the environment <NUM>. The plan can alternatively or additionally be indicative of desired or recommended locations for one or more objects in the environment <NUM> for which the controller <NUM> would direct manual movement, such as the manipulator <NUM>. Alternatively or additionally, the plan can be indicative of known locations of one or more objects in the environment <NUM>, such as equipment for the procedure. The one or more objects can include equipment, landmarks, obstacles, or other objects in the environment <NUM>. The plan, for example, corresponds to a map produced before the controller <NUM> directs the manual movement of the manipulator <NUM>. An operator can produce the map by interacting with a graphic representation of the environment <NUM>. In a medical example, the operator <NUM> can indicate a location of an operating table, a patient, medical equipment, or another object that will be in the environment <NUM> during a medical procedure. The operator can indicate the location using a user input device, e.g., a mouse, a keyboard, or other appropriate user input device. The graphic representation can be presented through the display device <NUM> of the user device <NUM>. In other implementations, the computing device that the operator uses to produce the plan is independent from the user device <NUM> worn during the procedure.

In some implementations, the procedure data 600a include a stored plan used for setting up a previous procedure, e.g., a procedure similar to the present procedure. For example, during a previous procedure, the plan used for setting up a medical system or a manipulator assembly is stored to be used for setting up another medical system or manipulator assembly for another procedure.

In one example, the plan is established before any operators are in the environment <NUM>. When the operator <NUM> begins the preparation process for a medical procedure, e.g., to place the medical system <NUM> or the manipulator assembly <NUM> into the setup configuration, a representation of the plan can be presented by the user device <NUM>. The operator <NUM> can interact with the user device <NUM> to enter a change into the plan, e.g., updating a location of an object in the environment <NUM>. In examples in which the user device <NUM> can detect gestures, as described herein, the operator <NUM> can perform a gesture to update the plan.

The equipment data 600b include data indicative of specifications of the equipment to be used during the procedure. The equipment data 600b can include data that specify a range of motion for each of the joints <NUM> of the manipulator assembly <NUM>. The range of motion can be a structural or mechanical limitation. In some examples, an initial motion envelope of the instrument <NUM> is estimated based on initial positions (and/or orientations) of the joints <NUM> and the ranges of motion of the joints <NUM> of the manipulator <NUM>. The controller <NUM> determines recommended positions or orientations for the joints <NUM> within the ranges of motion that will enable the instrument <NUM> to achieve a recommended motion envelope for the instrument <NUM>. The controller <NUM> can drive one or more powered joint of the joints <NUM> so that the joints <NUM> move toward the recommended positions or orientations) and the instrument <NUM> is able to move through the recommended motion envelope.

The equipment data 600b can also include information pertaining to the type of the instrument <NUM> mounted to the manipulator <NUM>. This information can be produced, e.g., by the instrument sensor <NUM> (described with respect to <FIG>). The type of the instrument <NUM> may affect, for example, an extent of the workspace and an amount of torque necessary to perform an operation. The type of the instrument <NUM> can be manually inputted by an operator. In some examples, the instrument <NUM> may include a detectable tag that indicates the type of the instrument <NUM>.

The pose data 600c include data indicative of poses of portions of the medical system <NUM>. The pose data 600c include the position and orientation information received at the operations 502A and 502B. In some implementations, the pose data 600c further include position or orientation information for the manipulator assembly <NUM> or portions of the manipulator assembly <NUM>, such as the joints <NUM>, the links <NUM>, the instrument holder <NUM>, or other components of the manipulator assembly <NUM>.

The pose data 600c can be indicative of target positions or orientations or actual positions or orientations of portions of the medical system. In one example, the pose data 600c can include information indicative of an actual configuration of the manipulator <NUM>. The pose data 600c, in such cases, can include the initial pose of each of the joints and/or links of the manipulator <NUM>. Alternatively, the pose data 600c can include information indicative of an actual configuration of the manipulator <NUM> during the guided manual movement of the manipulator <NUM>. The pose of the joints and the links or the configuration of the manipulator <NUM> can be detected by sensors of the sensor system <NUM>.

In another example, the pose data 600c can include information indicative of a target configuration of the manipulator <NUM>. The target configuration of the manipulator <NUM> can be determined based on other data and information described herein, or, in some cases, the target configuration of the manipulator <NUM> can be selected by the operator. In some cases, the target configuration of the manipulator <NUM> can be indicated on the predefined plan described with respect to the procedure data 600a.

The operator data 600d include data pertaining to the operator(s). In a medical example, the operator data 600d includes data pertaining to the medical team, e.g., the operators <NUM>, carrying out the procedure. The operator data 600d include, for example, information related to the capabilities, preferences for equipment layout, levels of experience, levels of skill, and other operator-specific attributes. In some implementations, the operator data 600d include information related to specific roles for the operators on the medical team. For example, the operator data 600d can include information indicating which operators are equipped to perform tasks requiring sterile handling, e.g., handling medical equipment to be placed into the patient <NUM>. In some examples, an operator profile is created for each of the operators before the procedure. A team profile alternatively or additionally is created for a particular team.

The obstacle data 600e include data indicative of poses (e.g. one or more parameters for positions or orientations) of the patient and obstacles in the environment <NUM> relative to the manipulator assembly <NUM>. In some examples, the obstacle data 600e can include a map of the environment <NUM> inputted by the operator. The map can include locations of potential obstacles within the environment <NUM>, such as other pieces of equipment (e.g., of the medical system <NUM>). This map can be similar to the map generated as part of the procedure data 600a.

The obstacle data 600e alternatively or additionally include data from obstacle sensors of the sensor system <NUM>. The obstacle sensor <NUM> can generate signals indicative of positions, orientations, or poses of obstacles within the environment <NUM> before the procedure, or as the manipulator <NUM> moves about the environment <NUM> during the procedure.

While the chair <NUM> is described as one example of a potential obstacle with which the manipulator assembly <NUM> could collide, other examples are possible. In some implementations, potential obstacles include the operating table <NUM>, the auxiliary system <NUM>, another manipulator assembly, a human operator, the user control system <NUM>, or other fixtures in the environment <NUM>.

In some medical contexts, the workpiece data 600f include patient data and include data indicative of patient-specific characteristics. Such patient data can include data indicative of patient body habitus and patient geometry. In some examples, the operator inputs the data indicative of the patient habitus and the patient geometry. In some cases, an imaging device can produce images that can be analyzed by the controller <NUM> (or by a computational system prior or during a procedure) to determine the patient habitus and the patient geometry. The imaging device may include part of the instrument <NUM>. In some examples, the workpiece data 600f can also include data indicative of the pose of the patient relative to the manipulator <NUM> and/or the pose of the operating table <NUM> relative to the manipulator <NUM>. The workpiece data 600f can include pre-operative images, such as x-ray images, x-ray computed tomography images, magnetic resonance imaging scans, and the like. In some cases, the workpiece data 600f includes intraoperative images or surface scans. A portion of the workpiece data 600f can be produced by the patient sensor <NUM>.

The port data <NUM> include data indicative of characteristics of an access port. In medical implementations, the access port corresponds to a device inserted through a body wall of the patient <NUM> through which a portion of the medical system <NUM> can be inserted to access anatomy of the patient <NUM>. The access port, for example, provides a conduit through which the instrument <NUM> is inserted to access the anatomy of the patient <NUM>. In one example, the port data <NUM> can indicate a recommended position or orientation of the access port. The recommended position or orientation of the access port can be indicated on the predefined plan described with respect to the procedure data 600a. In some implementations, the recommended position or orientation of the access port is based on a pose of the manipulator <NUM> when a cannula coupled to the manipulator <NUM> is docked, when an operator indicates readiness for repositioning of the base <NUM>, when an instrument <NUM> is mounted, etc. In some medical implementations, a component such as the instrument <NUM> is inserted through the access port on the patient, and the controller <NUM> can determine the position and orientation of the access port based on signals from sensors on the manipulator <NUM>.

In another example, the port data <NUM> can indicate an actual position or orientation of the access port. The actual position or orientation of the access port can be detected by sensors of the sensor system <NUM>. For example, the sensor <NUM> of the user device <NUM> can detect the position or orientation of the access port. The controller <NUM> can direct the manual movement of the manipulator <NUM> based on the recommended or actual position or orientation of the access port so that an instrument held by the instrument holder <NUM> of the manipulator <NUM> can be easily inserted into the access port.

As described with respect to the procedure data 600a, the operator <NUM> can specify the procedure data 600a or portions of the procedure data 600a before the controller <NUM> directs the manual movement of the manipulator assembly <NUM>. And, as described with respect to the procedure data 600a and the obstacle data 600e, the operator <NUM> can specify locations of objects in the environment <NUM>. In some implementations, the operator <NUM> can specify other types of the input data <NUM>. For example, in some implementations, the operator <NUM> can specify the equipment data 600b by specifying the types of equipment that will be used during the procedure. The operator <NUM> can also specify specifications of the equipment. In another example, the operator <NUM> can specify the characteristics of the operators that would be participating in the procedure and hence specify the operator data 600d.

For portions of the input data <NUM> that are specified by the operator <NUM>, the operator <NUM> can interact with a computing device to specify the data, e.g., a personal computer, a tablet computing device, a mobile phone, the user device <NUM>, or other computing device. In some implementations, the operator <NUM> provides a user input by performing a gesture detectable by the sensor <NUM> of the user device <NUM>. For example, the sensor <NUM> is positioned to detect movement of the operator <NUM>, and the controller <NUM> can direct the manual movement of the manipulator <NUM> in response to the detected movement. The controller <NUM> can be configured to detect predefined operator gestures that cause the controller <NUM> to perform corresponding operations.

In some examples, the imagery presented on the user device <NUM> includes one or more indicators, e.g., indicative of potential positions for objects, indicative of guidance for moving the manipulator <NUM>, indicative of an alert or alarm, or indicative of other information relevant to the operator <NUM>. An indicator in the imagery can be updated, e.g., repositioned in the imagery, cleared from the imagery, adjusted to provide additional details, or adjusted in some other manner, in response to a gesture performed by the operator <NUM>. The controller <NUM> can update the imagery in response to the gesture being proximate to or directed toward the indicator.

In one example, the controller <NUM> causes the user device <NUM> to stop presenting imagery when the operator <NUM> performs a broad swiping motion from one side to another side of the viewing frame of the user device <NUM>.

In another example, the user device <NUM> can present imagery indicative of waypoints <NUM> for a recommended path <NUM> of movement for the manipulator assembly <NUM>. The operator <NUM> can perform a gesture directed toward one of the waypoints <NUM>. For example, the operator <NUM> can extend a hand toward the waypoint and the move the hand in a direction to manually move the waypoint. In particular, in response to the gesture, the user device <NUM> updates the imagery such that the waypoint is repositioned in the direction indicated by the gesture of the operator <NUM>. In some examples, as described with respect to <FIG> and <FIG>, the imagery can include an indicator of a target location for the manipulator assembly <NUM>. Rather than causing waypoint to be repositioned, the operator <NUM> performs a gesture to cause the target location of the manipulator assembly <NUM> to be repositioned at a new target location.

In a further example, the operator <NUM> can perform gestures to rearrange the graphic representation of objects in the environment <NUM> as described with respect to the procedure data 600a. If the graphic representation of the environment <NUM> is presented through a see-through display on the user device <NUM>, the see-through display can present imagery including a representation of an object that would be in the environment <NUM> during the medical procedure. A gesture performed by the operator <NUM> can be indicative of the location of the object during the procedure. The user device <NUM> updates the imagery to indicate the location of the object. In some cases, the location of the object is indicated in the imagery, the gesture performed by the operator <NUM> can cause the user device <NUM> to update the imagery to indicate a new location of the object. In some examples, the objects correspond to the equipment to be used during the procedure. The imagery includes graphic representations of various pieces of equipment, such as a Mayo stand, an accessory cart, a medical imaging system, or other equipment that could be used during the procedure. For pre-operative planning purposes, the operator <NUM> performs gestures to create the plan indicative of the locations of the various pieces of equipment, and, as described herein, the controller <NUM> updates the imagery based on these gestures. The controller <NUM> can further direct the manual movement of the manipulator <NUM> based on this plan.

In some implementations, the user device <NUM> presents imagery to direct the manual movement of the manipulator <NUM>, and the operator <NUM> performs a gesture to indicate a location of a piece of equipment that will be placed in the environment <NUM>. Based on this indicated location, the controller <NUM> can update the imagery so that the guidance provided by the imagery accounts for the piece of equipment to be placed in the environment <NUM>.

While the operator <NUM> can provide user inputs that the controller <NUM> uses to control the imagery presented through the user device <NUM>, in some implementations described herein, sensors of the sensor system <NUM> produce portions of the input data <NUM> used by the controller <NUM> to present the imagery. For example, as described with respect to the pose data 600c, sensors of the sensor system <NUM> can detect positions and orientations of portions of the medical system <NUM>. In some implementations, the controller <NUM> can receive information from both the sensor <NUM> of the user device <NUM> and another sensor of the sensor system <NUM>. The information provided by the other sensor can also be indicative of a position or orientation of the one or more landmarks detected by the sensor <NUM> of the user device <NUM>. Alternatively or additionally, the information can be kinematic information for the manipulator <NUM> or kinematic information for another portion of the medical system <NUM>. The controller <NUM> can cause the user device <NUM> to present the imagery overlaid in the environment based on this received kinematic information.

In one example, the other sensor is a sensor of another user device worn by another operator in the environment <NUM>. For example, the other sensor is a sensor of one of the user device <NUM> worn by one of the operators <NUM>. In another example, the other sensor is a sensor mounted to the environment <NUM>, such as an image capture device mounted to a wall surface, a floor surface, or some other portion of the environment <NUM>. The image capture device can be fixed in the environment <NUM> or can be mounted to a movable object within the environment <NUM>. In some cases, the image capture device is part of equipment of the medical system <NUM>. In further examples, the other sensor is an integrated portion of the medical system <NUM>. For example, the other sensor can correspond to a joint sensor of the manipulator assembly <NUM>.

In implementations in which multiple operators <NUM> are present in the environment <NUM>, the different operators <NUM> can collaboratively provide or update input data <NUM>. For example, if the first operator <NUM>-<NUM> and the second operator <NUM>-<NUM> are both carrying user devices <NUM>-<NUM>, <NUM>-<NUM>, the first user device <NUM>-<NUM> and the second user device <NUM>-<NUM> can both present imagery indicative of the plan (described with respect to the procedure data 600a) for different objects in the environment <NUM>. For example, the imagery can include graphical representations of the objects overlaid in the environment <NUM> such that the operators <NUM>-<NUM>, <NUM>-<NUM> can easily manually move the real world objects to coincide with the graphical representation of the objects. The first and second operators <NUM>-<NUM>, <NUM>-<NUM> can reposition and reorient objects in the environment based on the predefined plan, e.g., by independently moving different objects in the environment <NUM> to their planned positions.

The first user device <NUM>-<NUM> and the second user device <NUM>-<NUM> can provide the guidance for manual movement of the objects as described herein. In further implementations, the first and second operators <NUM>-<NUM>, <NUM>-<NUM> can collaboratively update the plan. For example, the first and second operators <NUM>-<NUM>, <NUM>-<NUM> can operate user input devices as described herein, e.g., operating a touchscreen or performing a gesture, to select a new desired position or orientation of an object in the environment <NUM>. The graphic representation of the object is accordingly repositioned or reoriented to reflect the newly selected position or orientation in both the imagery presented by the first user device <NUM>-<NUM> and the imagery presented by the second user device <NUM>-<NUM>. If the second operator <NUM>-<NUM> selects a new desired position or orientation for the equipment table <NUM>, the graphic representations of the equipment table <NUM> in both the imagery presented by the first user device <NUM>-<NUM> and the imagery presented by the second user device <NUM>-<NUM> are updated.

In the operation 504A and the operation 504B, imagery is presented through the display device <NUM> of the user device <NUM>. As described herein, the imagery can be presented to direct the operator <NUM> to reposition or to reorient the manipulator assembly <NUM> or a portion of the manipulator assembly <NUM>. While the imagery is described as presenting a path <NUM> along which the manipulator assembly <NUM> should be moved, in other examples, the imagery presents other types of indicators to direct the manual movement of the manipulator assembly <NUM>. For example, the imagery can include an indicator indicative of a direction in which the manipulator assembly <NUM> should be manually translated by the operator <NUM>.

In examples in which the operator <NUM> is directed to reorient the manipulator assembly <NUM> or a portion of the manipulator assembly <NUM>, the user device <NUM> can present imagery including an alignment indicator. For example, if the controller <NUM> directs reorientation of the manipulator <NUM>, the alignment indicator be a line, a mark, or other indicator indicative of a target orientation of a link or a joint of the manipulator <NUM>.

The imagery presented by the display device <NUM> is described as being overlaid in the environment <NUM>. If the display device <NUM> is part of a head mounted user device worn over the eyes of the operator <NUM>, the imagery presented on the display device <NUM> is positioned between the environment <NUM> and the display device <NUM> such that the imagery is overlaid on the environment <NUM> as seen by the operator <NUM>. In some implementations, the user device <NUM> presents imagery that portions of the environment overlay. For example, if the user device <NUM> is a tablet computer, a smart phone, or other device that can have an opaque display, the opaque display can present captured imagery of the environment <NUM> with additional imagery or an indicator separate from the captured imagery of the environment <NUM>. The additional imagery may include content similar to the content described with respect to the imagery presented by the user device <NUM>. In some examples, the additional imagery is presented so as to appear overlaid on the captured imagery of the environment <NUM>.

In some implementations, a portion of the captured imagery of the environment <NUM> appears to overlay the additional presented imagery. For example, referring back to <FIG>, the operating table <NUM> is positioned between the operator <NUM> and the auxiliary system <NUM>. If the user device <NUM> presents an indicator indicative of a location of the auxiliary system <NUM>, the user device <NUM> can present the indicator such that the operating table <NUM> appears to be overlaid on the indicator. The indicator can be partially overlaid by the operating table <NUM>.

The imagery can be indicative of information other than guidance for manual movement of the manipulator assembly <NUM> or a portion thereof. <FIG> and <FIG> depict another example of imagery that could be presented by a head mounted user device. Referring to <FIG>, an operator <NUM> wearing a user device <NUM> (e.g., similar to the user device <NUM>) is positioned in an environment <NUM> proximate an operating table <NUM> supporting a patient <NUM>. Manipulators 708a and 708b are mounted on the operating table <NUM>.

<FIG> depicts an example of imagery overlaid in the environment <NUM> that the user device <NUM> presents to the operator <NUM>. The imagery includes status indicators 710a, 710b for the manipulators 708a, 708b. In the example shown in <FIG>, the status indicators 710a, 710b are indicative of whether the manipulators 708a, 708b require manual repositioning. In other implementations, the status indicators 710a, 710b are indicative of whether the manipulators 708a, 708b require manual reorienting or both manual repositioning and manual reorienting. In the example shown in <FIG>, the imagery includes a directional indicator <NUM> indicative of a direction that the manipulator 708b should be moved.

In some examples, the manual movement of the manipulator 708b is before any portion of a medical procedure has performed on the patient <NUM>. For example, the manual movement can be directed before an access port has been inserted into the patient <NUM>, and before instruments 712a, 712b have been inserted into the patient <NUM>. In other examples, the manual movement of the manipulator 708b is directed after a portion of the medical procedure has been performed. For example, the manual movement of the manipulator 708b is directed after the instrument 712b has been inserted into the patient <NUM>. The instrument 712b is retracted from the patient <NUM>, and the user device <NUM> presents the imagery to direct the manual movement of the manipulator 708b to a new position or orientation.

Such repositioning or reorienting after the medical procedure has been initiated but before the medical procedure has been completed can be necessary to complete the medical procedure. For example, a first stage of the medical procedure can require that the instruments 712a, 712b be used on a first region of a target anatomy, and a second stage of the medical procedure can require that the instrument 712a, 712b be used on a second region of the target anatomy. The manual repositioning of the manipulator 708b and manual repositioning of the manipulator 708a can be initiated when the first stage of the medical procedure is complete. The manipulators 708a, 708b can be repositioned or reoriented so that they can access the second region of the target anatomy.

In some implementations, rather than being indicative of whether the manipulators 708a, 708b need to be moved, the indicators 710a, 710b are indicative of other conditions of the manipulators 708a, 708b. For example, the indicators 710a, 710b could be indicative of whether the manipulators 708a, 708b are operational or whether the manipulators 708a, 708b have been properly mounted to the operating table <NUM>. In some examples, the indicators 710a, 710b are indicative of a condition or status of instruments 712a, 712b mounted to the manipulators 708a, 708b. For example, the indicators 710a ,710b could be indicative of types of the instruments 712a, 712b that are currently mounted to the manipulators 708a, 708b, durations that the instruments 712a, 712b have been used, or whether the instruments 712a, 712b should be replaced with new instruments.

The example imagery of <FIG> further includes an anatomy indicator <NUM> indicative of a region corresponding to target anatomy to be accessed by the instruments 712a, 712b. The anatomy indicator <NUM> can include a representation of an internal anatomy of the patient <NUM> such that the operator <NUM> can easily move the manipulators 708a, 708b relative to the internal anatomy that would typically not be visible to the operator <NUM> without a separate medical imaging system. The anatomy indicator <NUM>, particularly in implementations in which the user device <NUM> is worn over eyes of the operator <NUM>, can be generated based on imagery captured by the medical imaging system. For example, the anatomy indicator <NUM> can be a graphic representation of the internal anatomy of the patient <NUM> present in imagery captured by the medical imaging system.

In other implementations, the imagery can include anatomy indicators indicative of regions corresponding to anatomical features not to be accessed by the instruments 712a, 712b. These anatomy indicators can indicate regions that the operator <NUM> should avoid when moving the manipulators 708a, 708b.

In further implementations, the imagery can include a patient indicator <NUM> indicative of a status of the patient <NUM>. For example, the patient indicator <NUM> can indicate a heart rate and a blood pressure of the patient <NUM>. The patient indicator <NUM> can be generated based on the input data <NUM> described with respect to <FIG>, e.g., the workpiece data 600f.

The examples of <FIG> and <FIG> depict one example of depicting guidance of manual movement of the manipulator assembly <NUM>. For example, in <FIG> and <FIG>, the imagery indicates the path <NUM> for directing the manual movement of the manipulator assembly <NUM>. In other implementations, the imagery indicates multiple selectable paths along which the manipulator assembly <NUM> should be moved. Each of the selectable paths directs the manipulator assembly <NUM> from its current location to the desired or recommended location.

In the examples shown in <FIG> and <FIG>, the path <NUM> is shown as including multiple waypoints <NUM>. In some implementations, the path <NUM> corresponds to a region through which the manipulator assembly <NUM> should be moved to move the manipulator assembly <NUM> from its current location to its desired or recommended location. The region can be larger than the area footprint of the manipulator assembly <NUM> or the area footprint of the support structure <NUM> of the manipulator assembly <NUM>. The region defines boundaries within which the support structure <NUM> of the manipulator assembly <NUM> should be kept to avoid obstacles in the environment <NUM>. The manipulator assembly <NUM> could be moved along any one of multiple paths through the region.

In some implementations, in addition to or rather than presenting a representation of a path of desired or recommended movement for the manipulator assembly <NUM>, the user device <NUM> presents an indicator indicative of a desired or recommended direction of motion for the manipulator assembly <NUM>. As the operator <NUM> moves the manipulator assembly <NUM> in the desired or recommended direction, the desired or recommended direction of motion indicated by the indicator is updated so that the operator <NUM> can move the manipulator assembly <NUM> to the desired or recommended location. Additionally or alternatively, the indicator can be indicative of a desired or recommended magnitude of movement for the manipulator assembly <NUM>. For example, the indicator can include text specifying a distance that the manipulator assembly <NUM> should be moved. In examples in which the indicator is a directional arrow, the size of the directional arrow could be indicative of the amount of movement that the manipulator assembly <NUM> should be moved in the indicated direction.

While the indicator <NUM> is described as being presented proximate an obstacle near the path <NUM> in <FIG>, in some implementations, an indicator presented by the user device <NUM> is indicative of a location of the obstacle but is not necessarily proximate to the obstacle. For example, the user device <NUM> can present an annotation indicative of a direction of an obstacle as the operator <NUM> moves the manipulator assembly <NUM> along the path <NUM>. For example, the annotation include an indicator in the form of a directional arrow that indicates that an obstacle is in a particular direction.

In the examples described with respect to <FIG> and <FIG>, the manipulator assembly <NUM> and its manipulator <NUM> are described as being manually moved in accordance with indications presented by the user device <NUM>. The user device <NUM> can present imagery to guide manual movement of other parts of the medical system <NUM> or other portions of the manipulator assembly <NUM> in certain implementations. For example, in some implementations, the user device <NUM> presents imagery to guide manual movement of the base <NUM> of the manipulator <NUM>, the support structure <NUM> of the manipulator assembly <NUM>, or the instrument holder <NUM> of the manipulator <NUM>. In further implementations, manual movement of an intermediate portion of the manipulator <NUM> is guided. For example, the user device <NUM> can present imagery to direct manual movement of a particular one of the joints <NUM> or a particular one of the links <NUM>. In other implementations, manual movement of a distal portion of the manipulator assembly <NUM> or a proximal portion of the manipulator assembly <NUM> is directed. For example, the user device <NUM> can present imagery to guide manual movement of the instrument holder <NUM> to direct manual movement of the distal portion of the manipulator assembly <NUM>, and the user device <NUM> can present imagery to direct manual movement of the column <NUM> or the support structure <NUM> to direct manual movement of the proximal portion of the manipulator assembly <NUM>.

In some examples, the imagery presented includes a representation of a configuration for the manipulator <NUM> recommended by the controller <NUM> based on the input data <NUM>. The representation of the recommended configuration for the manipulator <NUM> can be indicative of a recommended orientation, a recommended position, or both a recommended orientation and a recommended position of one or more of the joints <NUM>, one or more of the links <NUM>, the base <NUM>, or a combination thereof. The representation of the recommended configuration can correspond to imagery overlaid on the environment <NUM>. In some implementations, the representation is an opaque, or a partially transparent, graphic representation of the manipulator <NUM> in the recommended configuration.

In some examples, the imagery provides an animation of a recommended motion of the manipulator <NUM> to the recommended orientation, recommended position, or recommended pose. The animation can be indicative of a sequence of positions or orientations between the current position and orientation of the manipulator <NUM> and the recommend position and orientation of the manipulator <NUM>. In some cases, the animation can provide a representation of a manner of movement of the operator to manually move the manipulator <NUM> into the recommended position or orientation. For example, the animation can represent hands of the operator and the recommended motion of the hands to manually move the manipulator <NUM>. The animation can include one or more transparent graphic representations of the manipulator <NUM> or the operator overlaid on the environment.

In some implementations, rather than directing manual movement of a portion of the manipulator assembly <NUM>, the user device <NUM> presents imagery to direct manual movement of another part of the medical system <NUM>. In some implementations, the imagery is presented to direct manual movement of the instrument <NUM> mounted to the manipulator <NUM> or a cannula through which the instrument <NUM> is inserted. In implementations in which an access port is inserted through a body wall of the patient <NUM>, the imagery presented by the user device <NUM> can be indicative of a desirable or undesirable locations of the access port. The imagery can indicate a recommended port location where the access port should be manually placed by the operator <NUM>. The recommended port location corresponds to a recommendation determined by the controller <NUM> based on the input data <NUM>. In other implementations, the imagery can indicate a keep out region indicative of locations that the access port should not be placed.

The user device <NUM> can direct manual movement of other parts of the medical system <NUM>. In some implementations, the user device <NUM> presents imagery to direct manual movement of another manipulator assembly in accordance with implementations described herein with respect to directing manual movement of the manipulator assembly <NUM> and portions thereof. This other manipulator assembly is distinct from the manipulator assembly <NUM>. In some implementations, this other manipulator assembly is separate from the manipulator assembly <NUM>. The manipulator assembly <NUM> and the other manipulator assembly are standalone assemblies each including its own separate support structure. In other implementations, the manipulator assembly <NUM> and the other manipulator assembly share a common support structure movable about the environment <NUM>.

In other implementations, additionally or alternatively, the user device <NUM> presents imagery to direct manual movement of an auxiliary system of the medical system <NUM>. The auxiliary system includes a resource such as, an illumination source, an energy source, an insufflation source, an instrument storage component, a central processor, or a combination thereof. The auxiliary system can be a standalone system movable about the environment <NUM>. For example, the auxiliary system can correspond to a medical imaging system, the user control system <NUM>, the equipment table <NUM>, an insufflation system (not shown in <FIG>), the operating table <NUM>, or other system in the environment <NUM>. Other auxiliary systems for which manual movement could be directed include trays, tables, imaging systems, user consoles, surgical instruments, medical instruments, diagnostic instruments, or other equipment. The features of the implementations described herein for directing manual movement of the manipulator assembly <NUM> can also be features of implementations in which the imagery presented is for directing manual movement of the auxiliary system. By presenting imagery to direct manual movement of an auxiliary system, the user device <NUM> enables an operator to easily set up equipment relative to the manipulator assembly <NUM> or relative to other equipment in the environment <NUM>.

In some implementations, the user device <NUM> presents imagery to direct movement of the patient <NUM> in the environment <NUM>. For example, the operating table <NUM> on which the patient <NUM> is supported can be moved to move the patient <NUM> about the environment <NUM>. In other implementations, the user device <NUM> presents imagery to direct movement of an operator in the environment <NUM>. For example, in implementations in which the environment <NUM> includes the first operator <NUM>-<NUM> and the second operator <NUM>-<NUM>, the user device <NUM>-<NUM> worn by the first operator <NUM>-<NUM> can direct movement of the first operator <NUM>-<NUM>, the second operator <NUM>-<NUM>, or both. Similarly, the user device <NUM>-<NUM> worn by the second operator <NUM>-<NUM> can direct movement of the first operator <NUM>-<NUM>, the second operator <NUM>-<NUM>, or both. Desirable locations of the first and second operators <NUM>-<NUM>, <NUM>-<NUM> can be determined based on the operator data 600d as well as different types of position and orientation information described herein, e.g., for obstacles, the manipulator assembly <NUM>, and other objects in the environment <NUM>. In implementations in which the operator data 600d include information related to whether an operator is equipped to handle sterile equipment, this information can be used to determine the task that the operator should perform in the environment and a location to perform the task. For example, the information may indicate that the first operator <NUM>-<NUM> is not equipped to handle sterile equipment, and hence the first operator <NUM>-<NUM> is directed to manually move the manipulator assembly <NUM>. In contrast, the operator <NUM>-<NUM> is equipped to handle sterile equipment and hence is directed to prepare instruments for insertion into the patient <NUM>. The operator <NUM>-<NUM> could also be directed to be at the bedside of the patient <NUM> to perform tasks proximate the patient <NUM>, such as pointing out locations for the access port or for insertion of the instrument <NUM> (shown in <FIG>).

As described herein, the techniques, systems, and methods described herein can be used to reposition or reorient a portion of the manipulator assembly or a portion of the medical system before a medical procedure is performed (e.g., pre-operative repositioning or reorienting) or during the medical procedure (e.g., intra-operative repositioning or reorienting). For example, during the medical procedure, these techniques, systems, and methods can be used to reposition or reorient a portion of the manipulator assembly or a portion of the medical system in response to an emergency condition, e.g., a patient condition or an equipment condition that requires the medical system or the manipulator assembly to be placed into a new configuration. In other implementations, the techniques, systems, and methods can be used after the medical procedure to facilitate take-down of the manipulator assembly or the medical system.

The systems (e.g., the medical system <NUM>) and robotic components of the medical systems (e.g., the manipulator <NUM>) described herein can be computer-assisted systems. For example, each of these systems and robotic components can be controlled, at least in part, using one or more computer program products, e.g., one or more computer programs tangibly embodied in one or more information carriers, such as one or more non-transitory machine-readable media, for execution by, or to control the operation of, one or more data processing apparatus, e.g., a programmable processor, a computer, multiple computers, and/or programmable logic components. The systems and robotic components can be controlled using a single computer program product or multiple distinct computer program products for distinct systems and robotic components. A computer program, e.g., for the one or more computer program products, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Operations associated with controlling the medical systems described herein can be performed by a controller (e.g., the controller <NUM>). The controller can include one or more programmable processors executing one or more computer programs to perform the functions described herein. The one or more programmable processors can includes multiple processors that are part of distinct subsystems in the medical system <NUM>, e.g., a processor of the manipulator assembly <NUM>, a processor of the user control system <NUM>, a processor of the auxiliary system <NUM>, and/or another processor of the medical system <NUM>. In other implementations, the one or more programmable processors can include one or more processors that are remote from the environment <NUM>, e.g., a processor of a remote server. Control over all or part of the medical systems described herein can be implemented using special purpose logic circuitry, e.g., an FPGA (field programmable gate array) and/or an ASIC (application-specific integrated circuit).

Generally, a processor will receive instructions and data from a read-only storage area or a random access storage area or both. Elements of a computer include one or more processors for executing instructions and one or more storage area devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, or transfer data to, or both, one or more machine-readable storage media, such as mass PCBs for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Machine-readable storage media suitable for embodying computer program instructions and data include all forms of non-volatile storage area, including by way of example, semiconductor storage area devices, e.g., EPROM, EEPROM, and flash storage area devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

Claim 1:
A computer-assisted medical system (<NUM>) comprising:
a user device (<NUM>, <NUM>-<NUM>, <NUM>-<NUM>) wearable by an operator, the user device (<NUM>, <NUM>-<NUM>, <NUM>-<NUM>) comprising:
a display device (<NUM>) configured to present imagery overlaid in an environment (<NUM>), the environment (<NUM>) physically containing a manipulator assembly (<NUM>), and
a sensor (<NUM>) configured to detect one or more landmarks physically in the environment (<NUM>); and
a controller (<NUM>) configured to execute instructions to perform operations, the operations comprising:
receiving, from the sensor (<NUM>), position or orientation information for the one or more landmarks physically in the environment (<NUM>),
directing a manual movement of a portion of the manipulator assembly (<NUM>) by causing the display device (<NUM>) to present the imagery overlaid in the environment (<NUM>) based on the received position or orientation information; and
directing a manual movement of a portion of the medical system (<NUM>) distinct from the manipulator assembly (<NUM>) relative to the manipulator assembly (<NUM>) by causing the display device (<NUM>) to present the imagery overlaid in the environment (<NUM>) based on the received position or orientation information, or
directing a manual movement, of a patient relative to the manipulator assembly (<NUM>) by causing the display device (<NUM>) to present second imagery overlaid in the environment (<NUM>) based on the received position or orientation information.