Patent Description:
Robotic manipulator assemblies include robotic manipulators that can be operated to control the motion of tools in a workspace. For example, such robotic manipulators can be used to perform non-medical and medical procedures. As a specific example, teleoperated surgical manipulators can be used to perform minimally invasive medical techniques.

It is desirable in medical techniques to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. For example, minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more incisions. Through these natural orifices or incisions, clinicians may insert medical tools to reach a target tissue location. Minimally invasive medical tools include tools such as therapeutic tools, diagnostic tools, and surgical tools. Minimally invasive medical tools may also include imaging tools such as endoscopic tools that provide a user with a field of view within the patient anatomy.

Robotic manipulators may be teleoperated or otherwise computer-assisted. For performing and viewing a robotic procedure at a procedure site (e.g., a surgical site within a patient), two or more slave manipulators may be used for holding and manipulating tools, including for example surgical instrument tools and imaging tools. To manipulate these tools, an operator's control console often includes master control devices which may be selectively associated with the tools and the slave manipulators holding the tools to manipulate them. In such a robotic system, the control of a tool in response to operator manipulation of a master control device may have a number of definable reference frames and corresponding frame transformations to map points in one frame to corresponding points in another frame. When one or more of the position and orientation of the frames and/or frame transformations are unknown, however, precise control of the tools may be difficult to achieve. In such cases, the success rate or accuracy of the procedure may be reduced. In a medical robot context, the safety of a patient being treated at the time by the robotic medical system as well as the successful completion of a procedure being performed on the patient may be jeopardized.

<CIT> discloses a system and method of recentering imaging devices and input controls includes a medical device having one or more end effectors, an imaging device, one or more input controls for teleoperating the one or more end effectors, and a control unit including one or more processors coupled to the end effectors, the imaging device, and the input controls. The control unit suspends teleoperated control of the end effectors by the input controls in response to a recentering request, determines a view recentering move for the imaging device so that the end effectors are contained within a view space of the imaging device, determines one or more input control recentering moves to provide positional and orientational harmony between each of the input controls and a corresponding one of the end effectors, executes the view and input control recentering moves, and reinstates teleoperated control of the end effectors by the input controls.

<CIT> discloses a teleoperated system comprises a display, a master input device, and a control system. The control system is configured to determine an orientation of an end effector reference frame relative to a field of view reference frame, determine an orientation of a master input device reference frame relative to a display reference frame, establish an alignment relationship between the master input device reference frame and the display reference frame, and command, based on the alignment relationship, a change in a pose of the end effector in response to a change in a pose of the master input device. The alignment relationship is independent of a position relationship between the master input device reference frame and the display reference frame. In one aspect, the teleoperated system is a telemedical system such as a telesurgical system.

<CIT> discloses a robotic system includes a camera having an image frame whose position and orientation relative to a fixed frame is determinable through one or more image frame transforms, a tool disposed within a field of view of the camera and having a tool frame whose position and orientation relative to the fixed frame is determinable through one or more tool frame transforms, and at least one processor programmed to identify pose indicating points of the tool from one or more camera captured images, determine an estimated transform for an unknown one of the image and tool frame transforms using the identified pose indicating points and known ones of the image and tool frame transforms, update a master-to-tool transform using the estimated and known ones of the image and tool frame transforms, and command movement of the tool in response to movement of a master using the updated master-to-tool transform.

<CIT> discloses a teleoperational assembly is disclosed which includes an operator control system and a plurality of manipulators configured to control the movement of medical instruments in a surgical environment. The manipulators are teleoperationally controlled by the operator control system. The system further includes a processing unit configured to display an image of a field of view of the surgical environment, determine association information about the manipulators and the operator control system, and display badges near the medical instruments in the image of the field of view of the surgical environment. The badges display the association information for the medical instrument they appear associated with.

According to the invention, a robotic system comprises a display that is viewable by an operator. An operator reference frame is defined relative to the display or the operator viewing the display. The robotic system further comprises an input device movable by the operator and processing unit including one or more processors. The processing unit is configured to present, in the display, a first image of a first tool captured by an imaging device and a marker controlled by the input device. The processing unit is also configured to receive, from the operator, a first indication that the marker is aligned with an alignment target, the alignment target presented in the display and associated with the first image. The processing unit is also configured to in response to the first indication, determine a first alignment relationship based on a second alignment relationship, the first alignment relationship being between the imaging device and the first tool, and the second alignment relationship being between the operator reference frame and the input device.

According to the invention, a method comprises presenting, in a display that is viewable by an operator, a first image of a first tool captured by an imaging device and a marker controlled by an input device movable by the operator. A first indication that the marker is aligned with an alignment target, the alignment target presented in the display and associated with the first image is received from the operator. In response to the first indication, a first alignment relationship is determined based on a second alignment relationship, the first alignment relationship being between the imaging device and the first tool, and the second alignment relationship being between an operator reference frame and the input device. The operator reference frame is defined relative to the display or the operator viewing the display.

According to the invention, a non-transitory machine-readable medium comprises a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method. The method includes presenting, in a display that is viewable by an operator, a first image of a first tool captured by an imaging device and a marker controlled by the input device movable by the operator. A first indication the marker is aligned with an alignment target, the alignment target presented in the display and associated with the first image is received from the operator. In response to the first indication, a first alignment relationship is determined based on a second alignment relationship, the first alignment relationship being between the imaging device and the first tool, and the second alignment relationship being between an operator reference frame and the input device. The operator reference frame is defined relative to the display or the operator viewing the display.

It will nevertheless be understood that no limitation of the scope of the disclosure is intended. In the following detailed description of the aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be obvious to one skilled in the art that the embodiments of this disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.

Any alterations and further modifications to the described devices, tools, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and/or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately. For simplicity, in some instances, the same reference numbers are used throughout the drawings to refer to the same or like parts.

Although some of the examples described herein often refer to surgical procedures or tools, or medical procedures or tools, the techniques disclosed also apply to non-medical procedures and non-medical tools. For example, the tools, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, and sensing or manipulation of non-tissue work pieces. Other example applications involve surgical or nonsurgical cosmetic improvements, imaging of or gathering data from human or animal anatomy, training medical or non-medical personnel, performing procedures on tissue removed from human or animal anatomies (without return to a human or animal anatomy), and performing procedures on human or animal cadavers.

The embodiments below will describe various tools and portions of tools in terms of their state in three-dimensional space. As used herein, the term "position" refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom that can be described using changes in Cartesian X, Y, Z coordinates, such as along Cartesian X, Y, Z axes). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom - e.g., which can be described using roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a portion of an object in at least one degree of translational freedom, and to the orientation of that object or that portion of that object in at least one degree of rotational freedom. For an asymmetric, rigid body in a three-dimensional space, a full pose can be described with six parameters in six total degrees of freedom.

Referring to <FIG> of the drawings, an example robotic system is shown. Specifically, in <FIG>, a computer-aided, robotic medical system that may be teleoperated and used in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures, is generally indicated by the reference numeral <NUM>. As will be described, the teleoperational systems of this disclosure are under the teleoperational control of an operator. In some embodiments, manipulators or other parts of a robotic system may be controlled directly through manual interaction with the manipulators (or the other parts) themselves. Thus, "teleoperated manipulators" as used in this application include manipulators that can be controlled only through teleoperation, and manipulators that can be controlled through teleoperation and through direct manual control. Further, in some embodiments, a non-teleoperational or robotic medical system may be under the partial control of a computer programmed to perform the procedure or sub-procedure. In still other alternative embodiments, a fully automated medical system, under the full control of a computer programmed to perform the procedure or sub-procedure, may be used to perform procedures or sub-procedures.

As shown in <FIG>, the robotic medical system <NUM> generally includes a manipulator assembly <NUM> mounted to or near an operating table O on which a patient P is positioned. The manipulator assemblies described herein often include one or more robotic manipulators and tools mounted thereon, although the term "manipulator assembly" also encompasses the manipulator without the tool mounted thereon. The manipulator assembly <NUM> may be referred to as a patient side cart in this example, since it comprises a cart and is designed to be used next to a patient. A medical tool <NUM> (also referred to as a tool <NUM>) and a medical tool <NUM> are operably coupled to the manipulator assembly <NUM>. Within this disclosure, the medical tool <NUM> includes an imaging device, and may also be referred to as the imaging tool <NUM>. The imaging tool <NUM> may comprise an endoscopic imaging system using optical imaging technology, or comprise another type of imaging system using other technology (e.g. ultrasonic, fluoroscopic, etc.). An operator input system <NUM> allows an operator such as a surgeon or other type of clinician S to view images of or representing the procedure site and to control the operation of the medical tool <NUM> and/or the imaging tool <NUM>.

The operator input system <NUM> for the robotic medical system <NUM> may be "mechanically grounded" by being connected to a base with linkages such as to an operator's console, or it may be "mechanically ungrounded" and not be thus connected. As shown in <FIG>, the operator input system <NUM> is connected to an operator's console <NUM> that is usually located in the same room as operating table O during a surgical procedure. It should be understood, however, that the operator S can be located in a different room or a completely different building from the patient P. The operator input system <NUM> generally includes one or more control device(s) for controlling the medical tool <NUM>. The operator input system <NUM> is also referred to herein as "master manipulators," "master control devices," "master input devices," and "input devices. " The control device(s) may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and the like. In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical tools of the robotic assembly to provide the operator with telepresence; that is, the operator is provided with the perception that the control device(s) are integral with the tools so that the operator has a sense of directly controlling tools as if present at the procedure site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical tools and still provide the operator with telepresence. In some embodiments, the control device(s) are manual input devices which move with six degrees of freedom, and which may also include an actuatable handle for actuating medical tools (for example, for closing grasping jaw end effectors, applying an electrical potential to an electrode, capture images, delivering a medicinal treatment, and the like).

The manipulator assembly <NUM> supports and manipulates the medical tool <NUM> while the operator S views the procedure site through the operator's console. An image of the procedure site can be obtained by the medical tool <NUM>, such as via an imaging system comprising a monoscopic or stereoscopic endoscope, which can be manipulated by the manipulator assembly <NUM> to orient the medical tool <NUM>. An electronics cart can be used to process the images of the procedure site for subsequent display to the operator S through the operator's console. The number of medical tools <NUM> used at one time will generally depend on the medical diagnostic or treatment (e.g. surgical) procedure and the space constraints within the operating room among other factors. The manipulator assembly <NUM> may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place) and a robotic manipulator. The manipulator assembly <NUM> includes a plurality of motors that drive inputs on the medical tools <NUM>. These motors move in response to commands from the control system (e.g., control system <NUM>). The motors include drive systems which when coupled to the medical tools <NUM> may advance the medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems may move the distal end of the medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate an articulable end effector of the tool for grasping tissue in the jaws of a biopsy device or the like. The medical tools <NUM> may include end effectors having a single working member such as a scalpel, a blunt blade, a needle, an imaging sensor, an optical fiber, an electrode, etc. Other end effectors may include multiple working members, and examples include forceps, graspers, scissors, clip appliers, staplers, bipolar electrocautery instruments, etc..

The robotic medical system <NUM> also includes a control system <NUM>. The control system <NUM> includes at least one memory <NUM> and at least one processor <NUM>, and typically a plurality of processors, for effecting control between the medical tool <NUM>, the operator input system <NUM>, and other auxiliary systems <NUM> which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and/or suction systems. The control system <NUM> also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. While control system <NUM> is shown as a single block in the simplified schematic of <FIG>, the system may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent the manipulator assembly <NUM>, another portion of the processing being performed at the operator input system <NUM>, and the like. Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the teleoperational systems described herein. In one embodiment, control system <NUM> supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE <NUM>, DECT, and Wireless Telemetry.

In some embodiments, the control system <NUM> may include one or more servo controllers that receive force and/or torque feedback from the medical tool <NUM> or from the manipulator assembly <NUM>. Responsive to the feedback, the servo controllers transmit signals to the operator input system <NUM>. The servo controller(s) may also transmit signals that instruct the manipulator assembly <NUM> to move the medical tool(s) <NUM> and/ or <NUM> which extends into an internal procedure site within the patient body via openings in the body. Any suitable conventional or specialized controller may be used. A controller may be separate from, or integrated with, manipulator assembly <NUM>. In some embodiments, the controller and manipulator assembly are provided as part of an integrated system such as a teleoperational arm cart positioned proximate to the patient's body during the medical procedure.

The control system <NUM> can be coupled to the medical tool <NUM> and can include a processor to process captured images for subsequent display, such as to an operator using the operator's console or wearing a head-mounted display system, on one or more stationary or movable monitors near the control system, or on another suitable display located locally and/or remotely. For example, where a stereoscopic endoscope is used, the control system <NUM> can process the captured images to present the operator with coordinated stereo images of the procedure site. Such coordination can include alignment between the stereo images and can include adjusting the stereo working distance of the stereoscopic endoscope.

In alternative embodiments, the robotic system may include more than one manipulator assembly and/or more than one operator input system. The exact number of manipulator assemblies will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems may be collocated, or they may be positioned in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.

<FIG> is a perspective view of one embodiment of a manipulator assembly <NUM> that is configured in the form of a cart that is located near the patient during a medical procedure. Thus, this manipulator assembly of <FIG> may also be referred to as a patient side cart. The manipulator assembly <NUM> shown provides for the manipulation of three medical tools 30a, 30b, 30c (e.g., medical tools <NUM>) and a medical tool <NUM> including an imaging device (e.g., medical tool <NUM>), such as a stereoscopic endoscope used for the capture of images of the workpiece or of the site of the procedure (also called "work site"). The medical tool <NUM> may transmit signals over a cable <NUM> to the control system <NUM>. Manipulation is provided by robotic manipulators having a number of joints. The medical tool <NUM> and the surgical tools 30a-c can be positioned and manipulated through incisions in, or natural orifices of, the patient so that a kinematic remote center is maintained at the incisions or natural orifices. Images of the work site can include images of the distal ends of the surgical tools 30a-c when they are positioned within the field-of-view of the imaging device of the medical tool <NUM>.

The manipulator assembly <NUM> includes a movable, lockable, and drivable base <NUM>. The base <NUM> is connected to a telescoping column <NUM>, which allows for adjustment of the height of the arms <NUM> (also called "manipulators <NUM>"). The arms <NUM> may include a rotating joint <NUM> that both rotates and translates parallel to the column <NUM>. The arms <NUM> may be connected to an orienting platform <NUM>. The orienting platform <NUM> may be capable of <NUM> degrees of rotation. The manipulator assembly <NUM> may also include a telescoping horizontal cantilever <NUM> for moving the orienting platform <NUM> in a horizontal direction.

In the present example, each of the arms <NUM> includes a manipulator arm portion <NUM>. The manipulator arm portion <NUM> may connect directly to a medical tool <NUM>. The manipulator arm portion <NUM> may be teleoperatable. In some examples, the arms <NUM> connecting to the orienting platform are not teleoperatable. Rather, such arms <NUM> are positioned as desired before the operator S begins operation with the teleoperative components.

Endoscopic and other imaging systems (e.g., medical tool <NUM>) may be provided in a variety of configurations, including ones having rigid or flexible structures. Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Digital image based endoscopes may have a "chip on the tip" design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data. Endoscopic imaging systems may also utilize other imaging techniques such as ultrasonic, infrared, and fluoroscopic technologies. Endoscopic imaging systems may provide two- or three- dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic tools employ stereo cameras to capture stereo images of the patient anatomy. An endoscopic tool may be a fully sterilizable assembly with the endoscope cable, handle and shaft all rigidly coupled and hermetically sealed.

<FIG> is a perspective view of the operator's console <NUM>. The operator's console <NUM> includes a left eye display <NUM> and a right eye display <NUM> for presenting the operator S with a coordinated stereo view of the surgical environment that enables depth perception. An operator input system <NUM> of the operator's console <NUM> includes one or more input control devices <NUM>, which in turn causes the manipulator assembly <NUM> to manipulate one or more medical tools <NUM> and/or <NUM>. The input control devices <NUM> may be used to, for example, close grasping jaw end effectors, apply an electrical potential to an electrode, deliver a medicinal treatment, or the like. In various alternatives, the input control devices <NUM> may additionally or alternatively include joystick devices, trackballs, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and the like. In some embodiments and for some associated medical tools <NUM>, the input control devices <NUM> will provide the same degrees of freedom as their associated medical tools <NUM> to provide the operator S with telepresence, or the perception that the input control devices <NUM> are integral with the tools <NUM> so that the operator S has a sense of directly controlling the tools <NUM>. In other embodiments, the input control devices <NUM> may have more or fewer degrees of freedom than the associated medical tools and still provide the operator S with telepresence. To this end, position, force, and tactile feedback sensors may be employed to transmit position, force, and tactile sensations from the tools <NUM> back to the operator S's hands through the input control devices <NUM>. An operator input system <NUM> of the operator's console <NUM> may also include input control devices <NUM>, which are foot pedals that receive input from a user's foot.

As shown in <FIG>, in some embodiments, input control devices <NUM> may include one or more of any number of a variety of input devices such as grip inputs <NUM> and trigger switches <NUM>. As illustrated in the example of <FIG>, a master reference frame <NUM> associated with the input control device <NUM>, denoted as m1, is provided. The Z-axis of the master reference frame <NUM> is parallel to an axis of symmetry <NUM> of the input control device <NUM>. The X and Y axes of the master reference frame <NUM> extend perpendicularly from the axis of symmetry <NUM>.

Referring to <FIG>, these figures show an operator reference frame <NUM>, denoted as e1. In various embodiments, the operator reference frame <NUM> may correspond with any appropriate portion of the operating environment, or with the operating environment itself. For example, the origin, the orientation, or both origin and orientation of the operator reference frame <NUM> may correspond with a part of the operator S, or a part of the robotic system. As specific examples, the operator reference frame <NUM> may have an origin based on the position of, and an orientation based on the orientation of, part or all of, the operator S (including, for example, the head, the torso, the entire body, or another body part of the operator). As further examples, the operator reference frame <NUM> may be based on a position of something held or otherwise attached to the operator, such as a head-mounted display, one or more hand-held input device, tracking markers or sensors attached to the operator S, etc. As yet further examples, the operator reference frame <NUM> may coincide with a display screen used to display images of the work space, a sensor system used to detect the operator S (or items attached to the operator S). Such a sensor system may be placed near the operator S or the display screen, such as on top of, below, or next to the operator or display screen. In an embodiment, the operator reference frame <NUM> corresponds to a location and orientation of a viewer of the system through which the operator looks through to the work site.

In the examples of the <FIG>, the operator reference frame <NUM> is defined with both origin and orientation corresponding with the usual or expected position and orientation the operator S's eyes, when the operator S is viewing the surgical site on a display <NUM> of an operator's console <NUM>. As such, this operator reference frame <NUM> is sometimes referred to as an eye reference frame. In the <FIG> example, the Z-axis of the operator reference frame <NUM> extends along a line of sight <NUM> of the operator S, when viewing the surgical site through the display <NUM>. The X and Y axes of the operator reference frame <NUM> extend perpendicularly from the Z-axis at an origin <NUM> of the operator reference frame <NUM>. Forward or inverse kinematic calculations are used to transform between the master reference frame <NUM> m1 and the operator reference frame <NUM> e1. A transformation from the operator reference frame <NUM> to master reference frame <NUM> m1 is denoted as e<NUM>Rm<NUM>.

Referring to the example of <FIG>, illustrated is a robotic system (e.g., a robotic medical system <NUM> of <FIG>) including two manipulator assemblies <NUM> and <NUM> on separate bases <NUM> and <NUM> respectively. The manipulator assembly <NUM> includes a base <NUM>, a structure support <NUM>, and a manipulator <NUM>. In the example of <FIG>, an imaging tool <NUM> is mounted on the manipulator <NUM> and thus the manipulator assembly <NUM> can be considered to further include the mounted imaging tool <NUM>. The imaging tool <NUM> includes a shaft <NUM> and an imaging device <NUM>. The imaging device <NUM> may include for example an optical imager, an ultrasonic imager, an electromagnetic imager such as a fluoroscopic imager, a thermal imager, a thermoacoustic imager, and any other suitable imagers. The imaging device <NUM> has a field of view <NUM>.

As illustrated in <FIG>, the base <NUM> has a reference frame <NUM>, which is also referred to as a camera base reference frame <NUM> and denoted as b1. The imaging device <NUM> has a reference frame <NUM>, which is also referred to as a camera reference frame <NUM> and denoted as c. A transformation from the base reference frame <NUM> to the camera reference frame <NUM> is denoted as b<NUM>Rc, which may be determined based on the forward kinematics of the manipulator assembly <NUM>.

As illustrated in <FIG>, the robotic system also includes a manipulator assembly <NUM>. The manipulator assembly <NUM> includes a base <NUM> that is physically separate and independent from the base <NUM> of the manipulator assembly <NUM>. The manipulator assembly <NUM> includes a structural support <NUM> and a manipulator <NUM>. In the example of <FIG>, a tool <NUM> is mounted on the manipulator <NUM>, and thus the manipulator assembly <NUM> can be considered to further include the mounted tool <NUM>. The tool <NUM> includes a shaft <NUM>, a wrist <NUM> coupled to the distal end of the shaft <NUM>, and an end effector <NUM> coupled to the wrist <NUM>. The base <NUM> has a reference frame <NUM>, which is also referred to as a tool base reference frame <NUM> and denoted as b2. The shaft <NUM> of the tool <NUM> has a reference frame <NUM>, which is also referred to as a shaft reference frame <NUM> and denoted as s. A transformation from the tool base reference frame <NUM> to the shaft reference frame <NUM> is denoted as b<NUM>Rs. The transformation b<NUM>Rs may be determined based on the forward kinematics of the manipulator assembly <NUM>.

In various embodiments, the relative positions and orientations of the bases <NUM> and <NUM> are unknown. As such, the transformation b<NUM>Rb<NUM> from the camera base reference frame <NUM> b1 to the tool base reference frame <NUM> b2 is unknown. In some embodiments, as illustrated in <FIG>, the bases <NUM> and <NUM> are coplanar or on parallel planes, such as located on a horizontal and even surface (herein called plane <NUM>). In those embodiments, a transformation b<NUM>Rb<NUM> from a camera base reference frame <NUM> to a tool base reference frame <NUM> may be described using a single rotational parameter representing a rotation around the Z-axis perpendicular to the plane (e.g., plane <NUM>) since the camera and tool bases reference frames <NUM>, <NUM> are not rotated relative to each other along the X and Y axes defining the plane (e.g., plane <NUM>). Alternatively, in some embodiments, the bases <NUM> and <NUM> are not coplanar, such as located on a ground that is not even. In those embodiments, the transformation b<NUM>Rb<NUM> may be described using multiple rotational parameters representing rotations around the X, Y, and/or Z-axes.

Such an unknown alignment relationship between the bases <NUM> and <NUM> may make intuitive control of a slave tool/end effector by a master control device difficult. To provide an effective control relationship between a master control device and its slave tool/end effector (also referred to as a master-tool alignment), a spatial alignment between the master control device and the tool/end effector is needed. Such a spatial alignment provides a reasonably accurate relationship between the operator's perceived motion of the master control device (e.g., a proprioceptive sense) and the operator's perceived resulting motion of the tool including the shaft and the end effector (e.g., a visual sense). For example, if the operator moves a hand grasping a master control device to the left, the operator expects to perceive the associated slave tool/end effector to move to the left also. If the perceived spatial motions match, then the operator can easily control the slave's movement by moving the master control device. But if the perceived spatial motions do not match (e.g., a master control device movement to the left results in a slave movement up and to the right), then it is difficult for the operator to control the slave's movement by moving the master control device. As described in detail below, an operator-guided registration process may be used to determine the unknown alignment relationship (e.g., b<NUM>Rb<NUM>) between the bases <NUM> and <NUM>, which may then be used to determine the master-tool alignment and a master-tool transformation. The operator-guided registration process may determine the master-tool alignment using registration information provided by the operator-guided registration process and known kinematic relationships and reference frame transforms in the telesurgical system. These relationships are described below in Cartesian terms, although other <NUM>-dimensional coordinate systems may be used.

During an operator-guided registration process, an operator S may move a master control device to align the master control device (e.g., input control device <NUM>) with an alignment target of the manipulator assembly <NUM>. The alignment target may be a manipulator <NUM>, a shaft <NUM>, a wrist <NUM>, a part or the entire end effector <NUM>, another portion of the manipulator assembly <NUM>, and/or a combination thereof. In an example, a portion (e.g., shaft <NUM>, end effector <NUM>) of the tool <NUM> may be used as the alignment target in the operator-guided registration process to increase the accuracy for a transformation between the camera reference frame <NUM> and an end effector reference frame associated with the end effector <NUM>.

After determining the alignment target of the manipulator assembly <NUM>, a transformation cRtarget from the camera reference frame <NUM> (denoted as c) to an alignment target reference frame (denoted as target) associated with the alignment target satisfies the following equation: <MAT> where cRb<NUM> and b<NUM>Rtarget are known transformations that may be determined based on the forward and inverse kinematics of the manipulator assemblies <NUM> and <NUM> respectively.

When a master control device is aligned with an alignment target of the manipulator assembly <NUM> shown in a display of the operator's control console (e.g., a display <NUM> of an operator's console <NUM> of <FIG>), the following condition of equality is satisfied: <MAT>.

Based on equations (<NUM>) and (<NUM>), we have the following: <MAT>.

As such, the unknown base transformation b<NUM>Rb<NUM> may be determined according to equation (<NUM>) because transformations e<NUM>Rm<NUM>, cRb<NUM>, and b<NUM>Rtarget are all known. For example, transformation e<NUM>Rm<NUM> from the eye reference frame <NUM> to the master reference frame <NUM> may be determined based on forward kinematics of the master control device.

For further example, transformation cRb<NUM> may be determined according to cRb<NUM> = (b<NUM>Rc)-<NUM>, where b<NUM>Rc is a transformation from the base <NUM> to the imaging device <NUM>, and may be determined using forward kinematics of the manipulator assembly <NUM>. In the manipulator assembly <NUM>, since the physical dimensions of all mechanical links of the manipulator assembly <NUM> including the imaging tool <NUM> are known, and since all joint angles between these mechanical links can be determined (using direct rotation sensors, motor position sensor, sensors, and the like), the kinematic relationship between the camera base reference frame <NUM> and camera reference frame <NUM> (or a reference frame of any other link in the manipulator assembly <NUM>) may be determined using known kinematic calculations.

Likewise, transformation b<NUM>Rtarget may be determined using forward kinematics of the manipulator assembly <NUM>. In the manipulator assembly <NUM>, since the physical dimensions of all mechanical links of the manipulator assembly <NUM> including the tool <NUM> are known, and since all joint angles between these mechanical links can be determined (using direct rotation sensors, motor position sensor, sensors, and the like), the kinematic relationship between the tool base reference frame <NUM> and a target reference frame (e.g., a shaft reference frame <NUM> or a reference frame of any other portion in the manipulator assembly <NUM>) can be determined using known kinematic calculations.

In some embodiments, the base transformation b<NUM>Rb<NUM> may be described using six parameters in six degrees of freedom, including three rotational parameters (e.g., a yaw (bearing) angle α representing a rotation around the Z-axis, a pitch angle β representing a rotation around the Y-axis, and a roll angle γ representing a rotation around the X-axis) and three translational parameters (e.g., distances x, y, and z along the X, Y, and Z-axes respectively). The transformation b<NUM>Rb<NUM> may be provided as follows: <MAT>.

After the master control device is aligned with the alignment target in the display by the operator S, we have the following equation based on equations (<NUM>) and (<NUM>): <MAT>.

Rotational and/or translational parameters between the camera base reference frame <NUM> and the tool base reference frame <NUM> may be computed based on equation (<NUM>). The transformations e<NUM>Rm<NUM>, cRb<NUM>, and b<NUM>Rtarget may include rotational matrices describing the position and orientation relationships of two reference frames and/or transformation matrices (e.g., a homogeneous transformation matrix H) that describe both the position and orientation relationships of two reference frames.

In some embodiments, relative orientations of the camera base reference frame <NUM> and the tool base reference frame <NUM> without knowledge of relative positions of these reference frames may be sufficient to enable intuitively correct tool operations controlled by an operator using a master control device. That is, even without information about the relative positions of the camera base reference frame <NUM> and the tool base reference frame <NUM>, translations and rotations at the master control device may result in corresponding translations and rotations that feel intuitively correct at the tool <NUM> because the relative orientations are consistent with a first alignment relationship (e.g., e<NUM>Rm<NUM>) between the eye reference frame <NUM> and master reference frame <NUM> and a second alignment relationship (e.g., cRena effector) between the camera reference frame <NUM> and an end effector reference frame associated with the end effector <NUM>. As such, in those embodiments, the operator-guided registration process may determine a transformation b<NUM>Rb<NUM> involving only rotational parameters α, β, and γ (without any translational parameters) for master-tool control with intuitively correct tool operations.

In some embodiments, as shown in the example of <FIG>, the camera base <NUM> and the tool base <NUM> are on a horizontal and even plane <NUM>. As such, the rotational parameters β and γ are zero. In those embodiments, the operator-guided registration process may determine a transformation b<NUM>Rb<NUM> involving only a rotational parameter α (without any translational parameters or rotational parameters β, and γ) for master-tool control with intuitively correct tool operations. In those embodiments, the transformation b<NUM>Rb<NUM> may be described as a Z-axis rotation having a yaw angle α (also referred to as a bearing angle α) between the base frames b1 and b2. The transformation b<NUM>Rb<NUM> may be determined using a single rotational parameter (the bearing angle α) according to equation (<NUM>) as follows: <MAT>.

In an example where the control system determines (e.g., based on forward and inverse kinematics) that <MAT>, and <MAT>, equation (<NUM>) for Z-axis transformation may be rewritten with equation (<NUM>) as follows: <MAT> <MAT>.

Accordingly, three equations in one unknown parameter (the bearing angle α) are obtained as follows: <MAT> <MAT> <MAT>.

In some embodiments, sin(α) and cos(α) are computed using any two of the three equations (<NUM>), (<NUM>), and (<NUM>) (e.g., equations (<NUM>) and (<NUM>), equations (<NUM>) and (<NUM>), or equations (<NUM>) and (<NUM>)). For example, cos(α) may be computed using equations (<NUM>) and (<NUM>) as follows: <MAT>.

Similarly, sin(α) may be computed using equations (<NUM>) and (<NUM>) as follows: <MAT>.

Based on equations (<NUM>) and (<NUM>), the bearing angle α may be determined using an inverse tangent function based on sin(α) and cos(α). To compute sin(α) and cos(α) based on equations (<NUM>) and (<NUM>), the denominators of those equations may not be zero. In other words, in some embodiments where denominators of equations (<NUM>) and (<NUM>) are equal to zero, the bearing angle α may not be computed.

In some embodiments, sin(α) and cos(α) may be computed using three equations (<NUM>), (<NUM>), and (<NUM>). For example, equations (<NUM>), (<NUM>), and (<NUM>) may be rewritten as follows: <MAT>.

A Singular Value Decomposition (SVD) algorithm may be applied to the left-hand side of the equation (<NUM>) for determining sin(α) and cos(α). The bearing angle α may then be determined using an inverse tangent function based on sin(α) and cos(α).

Referring to <FIG>, various systems and methods for performing an operator-guided registration process are described. As discussed above, to satisfy equation (<NUM>), the master control device needs to be aligned with the alignment target of the manipulator assembly <NUM> shown in a display of the operator's control console. In various embodiments, an alignment is considered to be achieved when an amount of misalignment is within a tolerance determined based on the needed motion accuracy. In an operator-guided registration process, an operator may perform an alignment step to move the master control device, and provide an indication to the control system after the operator determines that the master control device is aligned with the alignment target of the manipulator assembly <NUM> shown in a display of the operator's control console. As shown in <FIG>, a marker associated with the master control device is provided on the display to assist the operator in the alignment step. <FIG> illustrates a flowchart of a method for performing an operator-guided registration process where ai marker is provided. Such an operator-guided registration process may use little or none imaging processing, which enables a time-efficient registration process. As such, the operator-guided registration process may be used both before and during an operation (e.g., a medical surgery operation) without causing long interruption to the operation.

In some embodiments, as illustrated in <FIG>, the operator-guided registration process may map a single rotational parameter for a base transformation between the camera base reference frame and the tool base reference frame. In those examples of <FIG>, the alignment target is a shaft <NUM> of the tool <NUM>. Alternatively, in some embodiments as illustrated in <FIG>, the operator-guided registration process may map multiple parameters (translational and/or rotational parameters) for a base transformation between the camera base reference frame and the tool base reference frame. In those examples of <FIG>, the alignment target includes the shaft <NUM>, the wrist <NUM>, and the end effector <NUM> of the tool <NUM>. It is noted that while in <FIG> specific examples are used to describe the alignment target, the alignment target may include any portion of the manipulator assembly <NUM>.

Referring to <FIG>, <FIG>, a marker is shown on a display <NUM> (e.g., display <NUM>) to assist an operator to move the master control device and determine that the master control device is aligned with the alignment target (e.g., the shaft <NUM>). As discussed in detail below, depending on configuration of the operator-guided registration process (e.g., for mapping a single rotational parameter or mapping more than one translational and/or rotational parameters of the base transformation b<NUM>Rb<NUM>, properties (e.g., shape, color, texture) of the alignment target the marker may have various presentation properties including, for example, shape, color, size, translucency, surface patterns, text, any other presentation property, and/or a combination thereof. Such presentation properties of the maker may be used to indicate a direction (e.g., front or back) and an orientation (a roll angle representing a rotation around the X-axis). In an example, the marker (e.g., a cylinder) may not indicate a direction or an orientation. In another example, the marker may use various presentation properties, e.g., a particular shape (e.g., a cone), a color change, a pattern, a symbol, a text, and/or a combination thereof to indicate a direction and/or an orientation associated with the marker.

In the examples of <FIG>, <FIG>, the operator-guided registration process is configured to map a single rotational parameter (e.g., a bearing angle α around Z-axis) for a base transformation (e.g., b<NUM>Rb<NUM>) between the camera base reference frame and the tool base reference frame. In those examples, a single-parameter marker associated with that single rotational parameter may be used to assist the operator S in the alignment step. The single-parameter marker may have a shape (e.g., a line, a cone, a cylinder) with an axis (e.g., an axis of symmetry) associated with that single parameter to be mapped. The operator S may align the single-parameter marker with an alignment target of the manipulator assembly <NUM> by moving the master control device, such that the axis of the single-parameter marker is aligned with the corresponding axis of the alignment target.

A marker with various shapes may be used. For example, the marker may have a one-dimensional shape (e.g., a straight line), a two-dimensional shape (e.g., a triangle, a square, a rectangle, a circle, an oval), and/or a three-dimensional shape (e.g., a cylinder, a pyramid, a prism, a cube, a rectangular prism). In various embodiments, a control system may determine a shape of the marker based on the required alignment accuracy. In the examples of <FIG>, the control system provides a marker <NUM> having a cone shape in the display. In the examples of <FIG>, the control system provides a marker having a shape (e.g., a cylinder) that is the same as that of the shaft <NUM>, which may provide better alignment accuracy than the marker of <FIG>.

Referring to <FIG>, illustrated therein is a display <NUM> (e.g., a display <NUM> of <FIG>) before the operator performs an alignment step of the operator-guided registration process. The display <NUM> shows a portion of the tool <NUM> that is in the field of view <NUM> of the imaging device <NUM>. The portion of the tool <NUM> includes a portion of the shaft <NUM>, the wrist <NUM>, and the end effector <NUM>. The display <NUM> also includes a cone marker <NUM> having a first position <NUM>. The marker <NUM> has an axis <NUM> corresponding to the Z-axis of the master reference frame <NUM> of the master control device.

In various embodiments, the operator S may be instructed to move the master control device to change the position and/or orientation of the marker <NUM> in the display <NUM>, such that the marker <NUM> is aligned with an alignment target (e.g., the shaft <NUM>) in the display <NUM>. In an example, the marker <NUM> is aligned with the shaft <NUM> in the display <NUM> after the axis <NUM> of the marker <NUM> is parallel to an axis <NUM> of symmetry of the shaft <NUM> in the display <NUM>. The instruction may be provided to the operator S using a text message on the display <NUM> or an audio message provided through a speaker or a headphone worn by the operator S, such that the operator S may receive the instruction while viewing the display <NUM>. In an example, the instruction may include an identification of the alignment target (e.g., a shaft <NUM>, an end effector <NUM>, an axis of symmetry of the shaft <NUM>, an axis of symmetry of the end effector <NUM>) to the operator S. In the particular example of <FIG>, an instruction is provided to the operator S identifying the shaft <NUM> as the alignment target.

Referring to the example <FIG>, illustrated therein is a display <NUM> after an operator S has performed an alignment step to move the marker <NUM> of <FIG> at the position <NUM> to position <NUM>, so that the marker <NUM> of <FIG> has an orientation such that the marker <NUM> is aligned with the shaft <NUM> in the display. In the example of <FIG>, the position of the marker <NUM> is moved so that its axis <NUM> is collinear with an axis <NUM> of symmetry of the shaft <NUM>. In an alternative example, the marker <NUM> of <FIG> may remain at the same position as that of <FIG> but has a different orientation such that its axis <NUM> is parallel to the axis <NUM> of symmetry of the shaft <NUM>. In an example, by aligning the axis <NUM> of the marker <NUM> and the corresponding axis <NUM> of the shaft <NUM> of the tool <NUM>, alignment along the Z-axes of the master reference frame <NUM> and the shaft reference frame <NUM> is achieved.

In some embodiments, after the operator S determines that the marker <NUM> is aligned with the shaft <NUM> in the display <NUM> of <FIG>, the operator S provides an indication using the master control device (e.g., using a hand grip, a button, a slider, a foot pedal, a voice recognition device, and the like) to the control system, indicating that the alignment step of the operator-guided registration process is completed. The control system may perform a registration step to compute the parameter to be mapped (e.g. bearing angle α around Z-axis) according to equations (<NUM>) and (<NUM>) or (<NUM>). The control system may then compute the base transformations b<NUM>Rb<NUM> using the bearing angle α according to equation (<NUM>) as discussed above.

Referring to <FIG>, in some embodiments, a marker may be presented on the display based on the alignment target, such that the operator may provide a more accurate alignment between the master control device and the alignment target. For example, the marker may have a shape, color, size, translucency, any other presentation property, and/or a combination thereof that are determined based on the alignment target.

In the example of <FIG>, a display <NUM> includes a portion of the tool <NUM> in the field of view <NUM> of the imaging device <NUM> before an operator performs an alignment step. The portion of the tool <NUM> includes a shaft <NUM>, a wrist <NUM> coupled to the distal end of the shaft <NUM>, and an end effector <NUM> coupled to the wrist <NUM>. In the example of <FIG>, the shaft <NUM> has an axis <NUM> corresponding to the Z-axis of the shaft reference frame <NUM>. The display <NUM> also includes a marker <NUM> at a position <NUM>. The marker <NUM> has an axis <NUM> corresponding to the Z-axis of the master reference frame <NUM>. In the example of <FIG>, the marker <NUM> is a solid (i.e., not transparent/translucent) cylinder. In another example, the marker <NUM> is a semi-transparent or translucent image of the shaft <NUM>. In yet another example, the marker <NUM> is a wire diagram image of the shaft <NUM>. In the example of <FIG>, the operator S may overlay the marker <NUM> with the shaft <NUM> by moving the master control device, which may help the operator S to achieve better alignment accuracy.

Referring to the example of <FIG>, illustrated is the display <NUM> after the operator has performed an alignment step to move the marker <NUM> of <FIG>. As shown in <FIG>, by overlaying the marker <NUM> with the shaft <NUM>, the operator may achieve better alignment accuracy.

Referring to <FIG>, the operator-guided registration process may include two steps, an alignment step and a registration step. Each of the alignment step and registration step has its own best and worst cases regarding the positions and orientations of the tool <NUM> and/or the imaging device <NUM>. <FIG> illustrate the best and worst cases in alignment error sensitivity in the alignment step. <FIG> and <FIG> illustrate the best and worst cases in the registration step. In some embodiments, prior to the alignment step and registration step of the operator-guided registration process, the manipulator assemblies <NUM> and/or <NUM> may be configured (e.g., manually by an operator or controlled using a master control device) such that they do not have configurations corresponding to the worst cases. In some embodiments, the control system may disable its operator-guided registration mode for performing the operator-guided registration process after it determines that the manipulator assemblies <NUM> and <NUM> have worst case configurations for the alignment and/or registration step, and enable the operator-guided registration mode after the manipulator assemblies <NUM> and <NUM> are moved out of the worst case configurations. In the examples of <FIG>, the alignment target is the shaft <NUM> of the tool <NUM>. However, in other examples, the alignment target may be any portion of the manipulator assembly <NUM> controlling the tool <NUM>.

Referring to <FIG>, illustrated therein is the best case configuration in the alignment step of the operator-guided registration process. <FIG> illustrates the relative positions and orientations of the shaft <NUM> and the imaging device <NUM>. As shown in <FIG>, the shaft <NUM> of the tool <NUM> has an alignment target axis <NUM>. The imaging tool <NUM> includes an imaging device <NUM> having an optical axis <NUM>. The tool <NUM> and the imaging tool <NUM> are positioned such that the optical axis <NUM> is perpendicular to the alignment target axis <NUM>. In other words, the shaft <NUM> of the tool <NUM> (the alignment target) lies entirely in a plane that is perpendicular to the optical axis <NUM> of the imaging device <NUM> and parallel to an optical plane <NUM> of the imaging device <NUM>. In the example of <FIG>, the entire length L of the shaft <NUM> is projected on the optical plane <NUM>.

<FIG> illustrates a display <NUM> including an image of the tool <NUM> captured by the imaging tool <NUM> of <FIG>. As shown in <FIG>, because the entire length L of the shaft <NUM> is projected on the optical plane <NUM>, it is the easiest to orient a marker and align the marker to the alignment target axis <NUM> shown in the display <NUM>. As such, such a configuration of relative positions of the tool <NUM> and the imaging tool <NUM> as shown in <FIG> is associated with the best alignment error sensitivity.

Referring to <FIG>, illustrated therein is the worst case configuration in the alignment step of the operator-guided registration process. As shown in <FIG>, the shaft <NUM> of the tool <NUM> has an alignment target axis <NUM>. The imaging tool <NUM> includes an imaging device <NUM> having an optical axis <NUM>. The tool <NUM> and the imaging tool <NUM> are positioned such that the optical axis <NUM> is parallel to the alignment target axis <NUM>. In other words, the axis of the shaft <NUM> of the tool <NUM> lies entirely in a plane parallel to the optical axis <NUM> of the imaging device <NUM>. The alignment target axis <NUM> is perpendicular to the optical plane <NUM> of the imaging device <NUM>, and has an out-of-plane angle θ of <NUM> degrees with regard to the optical plane <NUM>. This is a singularity configuration in which the shaft <NUM> reduces to a circle in the field of view of the imaging device <NUM>, and its alignment target axis <NUM> is projected as a point on the optical plane <NUM>.

<FIG> illustrates a display <NUM> showing an image of the tool <NUM> captured by the imaging tool <NUM> of <FIG>. As shown in <FIG>, while a tip of the end effector <NUM> is visible on the display <NUM>, the shaft <NUM> is not visible. In such a configuration, the operator S may not be able to orient a marker and align the marker to the shaft <NUM>. As such, such a configuration of relative positions of the tool <NUM> and the imaging tool <NUM> as shown in <FIG> is associated with the worst alignment error sensitivity.

Referring to <FIG>, even with an out-of-plane angle θ that is less than <NUM> degrees, it may be difficult for the operator S to align a marker with the shaft <NUM> as the out-of-plane angle θ gets closer to <NUM> degrees. As shown in the example of <FIG>, the shaft <NUM> and the imaging device <NUM> have relative positions and orientations such that an out-of-plane angle θ between an alignment target axis <NUM> of the shaft <NUM> and the optical plane <NUM> of the imaging device <NUM> is greater than zero but less than <NUM> degrees. The projected length <NUM> of the shaft <NUM> on the optical plane <NUM> decreases as the output-of-plane angle θ increases from zero to <NUM> degrees. For example, the projected length <NUM> may be computed using L*cos(θ), where L is the length of the shaft <NUM>. As the projected length <NUM> decreases, it becomes more difficult for the operator S to align the marker to the shaft <NUM>. For an identical translational error, a projected shaft having a shorter length results in a larger angular error. As such, the alignment error sensitivity decreases as the output-of-plane angle θ increases from zero to <NUM> degrees.

In an example, a threshold out-of-plane angle θ<NUM> (e.g., <NUM> degrees) is provided indicating that at the threshold out-of-plane angle θ<NUM>, it becomes difficult for the operator S to align a marker with the shaft <NUM>. An error region <NUM> may be identified. In this example, a swivel angle cone may be identified as an error region <NUM> where the out-of-plane angle θ is between the out-of-plane angle θ<NUM> and <NUM> degrees. Such an error region may identify the bounds of the position/orientation of the shaft <NUM>, where it is difficult for the operator S to align a marker with the shaft <NUM>, or for the system to achieve sufficiently accurate registration given a particular operator-specified alignment. In an example, a control system may disable its operator-guided registration mode for performing the operator-guided registration mode after determining that the shaft <NUM> is in the error region <NUM>. In another example, the control system may provide an error region message to the operator notifying the operator of such worst case configuration. In yet another example, the control system may enable its operator-guided registration mode after determining that the shaft <NUM> is out of the error region <NUM>.

Referring to <FIG>, a curve <NUM> illustrates alignment target axis orientation errors with regard to the out-of-plane angle θ between the alignment target axis <NUM> and the optical plane <NUM> of the imaging device <NUM>. As shown in the curve <NUM>, in a configuration (e.g., the configuration of <FIG>) where the out-of-plane angle θ is zero, the alignment target axis orientation error has a value <NUM> (e.g., about three degrees), which may be caused by a shaft position error (e.g., about <NUM>%). As the out-of-plane angle θ increases from zero to <NUM> degrees, the alignment target axis alignment error increases. In a configuration (e.g., the configuration of <FIG>) where the out-of-plane angle θ is <NUM> degrees, the alignment target axis alignment error is infinity.

In some embodiments, the control system includes a predetermined error threshold <NUM> (e.g., <NUM> degrees) for the alignment target axis orientation error. As shown in <FIG>, the predetermined alignment target axis orientation error threshold <NUM> corresponds to an out-of-plane angle θ<NUM> (e.g., <NUM> degrees). In such embodiments, bounds of an error region (e.g., a swivel angle cone -shaped error region <NUM> of <FIG>) may be identified where the output-of-plane angle θ is between θ<NUM> and <NUM> degrees. In a configuration where a shaft <NUM> is located within such an error region, it is difficult for the operator S to align the marker with the shaft <NUM> and is likely to result in an orientation error that is greater than the predetermined error threshold <NUM>. As such, prior to the operator-guided registration process, the imaging device <NUM> and the shaft <NUM> may be configured (e.g., manually by an operator or controlled using a master control device) such that the shaft <NUM> is not located in the error region. In an example, a control system may disable its operator-guided registration mode for performing the operator-guided registration mode after determining that the shaft <NUM> is in the error region. In another example, the control system may provide an error region message to the operator notifying the operator of such worst case configuration. In yet another example, the control system may enable its operator-guided registration mode after determining that the shaft <NUM> is out of the error region.

Referring to <FIG> and <FIG>, illustrated therein are the best and worst case configurations for the registration step of the operator-guided registration process, where the registration step uses alignment information provided by the alignment step of the operator-guided registration process. <FIG> illustrates the best case configuration for the registration step of the operator-guided registration process, which provides the most alignment information for registration. <FIG> illustrates the worst case configuration for the registration step of the operator-guided registration process, which provides the least alignment information for registration.

Referring to <FIG>, illustrated therein is the best case configuration for the registration step of the operator-guided registration process, where an optical axis (e.g., optical axis <NUM>) of the imaging device <NUM> is aligned with axes (e.g., Z-axes of the camera base frame <NUM> b1 and tool base frame <NUM> b2) associated with the content for registration (e.g., relative bearing angle α around the Z-axes of the camera base reference frame <NUM> and the tool base reference frame <NUM>. In the example of <FIG>, the camera base (e.g., base <NUM> of <FIG>) for the imaging tool <NUM> and the tool base (e.g., base <NUM> of <FIG>) are located on the same horizontal plane <NUM>. As such, the Z-axes of the camera base reference frame <NUM> and the tool base reference frame <NUM> are vertical. The imaging device <NUM> of the imaging tool <NUM> has a position/orientation such that its optical axis <NUM> is parallel to the Z-axes of the camera base reference frame <NUM> and the tool base reference frame <NUM>. The shaft <NUM> has a position/orientation such that its alignment target axis <NUM> is perpendicular to the optical axis <NUM>. Such a configuration provides the maximum alignment information for registration (e.g., associated with the relative bearing angle α). After the alignment step is completed, an angle mismatch between the marker and the alignment target axis <NUM> (e.g., an angle between the Z-axes of the master control device frame and the shaft frame) is identical to the relative bearing angle α between camera base reference frame <NUM> and the tool base reference frame <NUM>.

Referring to <FIG>, illustrated therein is the worst case configuration for the registration step of the operator-guided registration process, where an optical axis (e.g., optical axis <NUM>) of the imaging device <NUM> is perpendicular to the axes (e.g., Z-axes of camera base reference frame <NUM> and the tool base reference frame <NUM>) associated with the content for registration (e.g., relative bearing angle α). In the example of <FIG>, the camera base (e.g., base <NUM> of <FIG>) for the imaging tool <NUM> and the tool base (e.g., base <NUM> of <FIG>) are located on the same horizontal plane <NUM>. As such, the Z-axes of the camera base reference frame <NUM> and the tool base reference frame <NUM> are vertical. The imaging device <NUM> of the imaging tool <NUM> has a position/orientation such that its optical axis <NUM> is perpendicular to the Z-axes of the camera base reference frame <NUM> and the tool base reference frame <NUM>. Such a configuration provides no alignment information content for registration (e.g., relative bearing angle α between the bases <NUM> and <NUM> of <FIG>). In such a configuration, even the best alignment of the marker and the shaft <NUM> during the alignment step may not provide any relevant information regarding the content for registration (e.g., relative bearing angle α between the bases <NUM> and <NUM>). In other words, in this configuration, changing the position of the shaft <NUM> may not improve the relevant information content for registration.

As shown in <FIG> and <FIG>, relevant alignment information regarding the content for registration may reduce as the optical axis <NUM> of the imaging device <NUM> moves away from axes (e.g., Z-axes of the of camera base reference frame <NUM> and the tool base reference frame <NUM>) associated with the content for registration (e.g., the bearing angle α). Such relevant alignment information reaches zero when the optical axis <NUM> of the imaging device <NUM> is perpendicular to the Z-axes of camera base reference frame <NUM> and the tool base reference frame <NUM>. As such, in some embodiments, to avoid the worst case configuration in the registration step, prior to the operator-guided registration process, the imaging device <NUM> may be moved (e.g., manually by an operator) to a position/orientation based on the content for registration. In an example where the content for registration includes the bearing angle α associated with a Z-axis rotation of the bases, prior to the operator-guided registration process, the imaging device <NUM> is moved to a position where its optical axis is not perpendicular to the Z-axes of the bases. In another example where the content for registration includes additional translational parameters and rotational parameters associated with X, Y, and Z-axes of the camera base reference frame <NUM> and the tool base reference frame <NUM>, prior to the operator-guided registration process, the imaging device <NUM> is moved to a position/orientation where its optical axis is not perpendicular to any of the associated X, Y, and Z-axes.

Referring to <FIG>, in some embodiments, the operator-guided registration process may map more than one parameter (translational and/or rotational parameters) of the base transformation b<NUM>Rb<NUM> between the camera base reference frame and the tool base reference frame. In some examples, the bases <NUM> and <NUM> may be on a tilted floor that has an angle with a horizontal plane and/or on a floor that is not even. In those examples, the operator-guided registration process may determine rotational parameters (e.g., a pitch angle β representing a rotation around the Y-axis and a roll angle γ representing a rotation around the X-axis) in addition to the bearing angle α representing a rotation around the Z-axis. In some examples, the operator-guided registration process may determine translational parameters (e.g., distances along the X, Y, and Z-axes) of the base transformation b<NUM>Rb<NUM>, which may improve the master-tool transformation accuracy, thereby improving the operation intuitiveness in controlling the tool using the master control device.

Referring to the example of <FIG>, illustrated is a display <NUM> includes an image of the tool <NUM> captured by the imaging tool <NUM> and a marker <NUM> before an operator S performs an alignment step. The marker <NUM> may be provided using a two-dimensional (2D) or a three-dimensional (3D) model of the tool <NUM>, and thus may represent an actual solid model of the tool <NUM>. The model of the tool <NUM> may be provided using, for example, computer aided design (CAD) data or other 2D or 3D solid modeling data representing the tool <NUM> (e.g., tool <NUM> of <FIG>). In the example of <FIG>, the marker <NUM> is a virtual representation of the tool <NUM>, and includes a virtual shaft <NUM>, a virtual wrist <NUM>, and a virtual end effector <NUM>. When the marker <NUM> is in a virtual representation of a tool, the marker <NUM> is also referred to as a virtual tool <NUM> herein. The virtual tool <NUM> and its position, orientation, and size as shown in the display <NUM> may be determined by the pose of the master control device and the alignment relationship between the master control device and the display <NUM>. In an embodiment, the virtual tool <NUM> is manipulatable at each joint (e.g., at the virtual wrist <NUM>) by the master control device, so that the pose of the tool <NUM> may be mimicked by the virtual tool <NUM> by the operator S using the master control device in the alignment step. In the example of <FIG>, the size of the virtual tool <NUM> is smaller than the actual tool <NUM>. The virtual tool <NUM> may be represented in a number of different ways. In an example, the virtual tool <NUM> is a semi-transparent or translucent image of the tool <NUM>. In another example, the virtual tool <NUM> is a wire diagram image of the tool <NUM>. In yet another example, the virtual tool <NUM> is an image that appears solid (i.e., not transparent/translucent), but such a solid virtual tool <NUM> may make viewing of the actual tool <NUM> in the display <NUM> difficult.

Referring to the example of <FIG>, illustrated is a display <NUM> after the operator S has performed the alignment step to move the master control device to overlay the virtual tool <NUM> with the actual tool <NUM>. The virtual tool <NUM> has been moved based on the change in the alignment relationship between the master control device and the display. In other words, the position, orientation, and size of the virtual tool <NUM> as shown in the display <NUM> of <FIG> correspond to the new pose of the master control device and the new alignment relationship between the master control device and the display after the operator S performs the alignment step.

In some embodiments, after the operator S determines that the virtual tool <NUM> is aligned with (e.g., completely overlaying) the alignment target (e.g., the tool <NUM>) in the display <NUM> of <FIG>, the operator S provides an indication to the control system indicating that the alignment step of the operator-guided registration process is completed. In some embodiments, by overlaying the virtual tool <NUM> completely with the tool <NUM>, alignment information associated with all degrees of freedom is provided for the registration step. As such, the control system may compute one or more parameters (e.g., one or more of the six parameters in six total degrees of freedom in a three-dimensional space) according to equation (<NUM>). In an example, the transformation b<NUM>Rtarget in the equation (<NUM>) is a transformation between the tool base reference frame <NUM> to an end effector reference frame of the end effector <NUM>.

While in the example of <FIG>, a marker <NUM> that is a virtual representation of the tool <NUM> is used for multiple-parameter (e.g., three orientation parameters and the three translation parameters between the two base reference frames) mapping, the marker <NUM> may use various visual indicators (e.g., markings, textures, colors, shapes, text) for such multiple-parameter mapping. In an example, the marker <NUM> may include virsual indicators corresponding to particular features (e.g., flanges, indentations, protrusions) of the tool <NUM>. In that example, an operator S may align the marker <NUM> and the tool <NUM> in both positions and orientations by matching the marker <NUM> with the image of the tool <NUM> in the display. As such, multiple-parameter mapping may be achieved.

<FIG> illustrates a method <NUM> for performing an operator-guided registration process. The method <NUM> is illustrated in <FIG> as a set of operations or processes <NUM> through <NUM>. Not all of the illustrated processes <NUM> through <NUM> may be performed in all embodiments of method <NUM>. Additionally, one or more processes that are not expressly illustrated in <FIG> may be included before, after, in between, or as part of the processes <NUM> through <NUM>. In some embodiments, one or more of the processes may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that, when run by one or more processors (e.g., the processors of a control system such as control system <NUM>), causes the one or more processors to perform one or more of the processes. As shown in the method <NUM>, little or no image processing is needed for such an operator-guided registration process. Therefore, the operator-guided registration process may take much less time than registration processes using computationally expensive image processing, and may be performed during a surgical operation. Furthermore, the operator-guided registration process may be performed without using additional sensors or other hardware in addition to the joint encoders in the manipulators, and may not suffer from sensor related restrictions such as line of sight, magnetic interference, etc..

The method <NUM> begins at process <NUM>, where a control system of a robotic medical system is switched to an operator-guided registration mode. At process <NUM>, the control system may determine whether there is a control loop between a master control device and a tool, and interrupt that control loop if any. In other words, in the operator-guided registration mode, the tool is not controlled by the master control device. In an example, in the operator-guided registration mode, the control system may keep the tool and/or an imaging device stationary. In some embodiments, at process <NUM>, the control system may disable the operator-guided registration mode and/or provide a warning message to an operator after determining that an imaging device and a tool of the robotic medical system have configurations that correspond to the worst case configurations in alignment step and/or registration step as discussed above with reference to <FIG>.

The method <NUM> may proceed to process <NUM>, where an image of a tool captured by an imaging device of the robotic medical system is provided to an operator through a display of an operator's console. In the example of <FIG>, at process <NUM>, an image of a tool <NUM> is captured by an imaging device <NUM> of a robotic medical system. In the example of <FIG>, an image including the field of view of the imaging device <NUM> is provided to an operator S through a display <NUM> of an operator's console. The displayed image may include portions of the tool <NUM> including, for example, shaft <NUM>, wrist <NUM>, and end effector <NUM>.

In some embodiments, the method <NUM> may then proceed to process <NUM>, where an operator S is instructed to align the master control device with an alignment target of the tool <NUM> provided in the display without the assistance of a marker in the display.

Alternatively, in some embodiments, the method <NUM> may proceed to process <NUM>, where a marker is provided to the operator S through the display <NUM> to assist the operator S to perform an alignment step. In such embodiments, a control system may determine the type (e.g., single-parameter marker, multiple-parameter marker) of the marker based on the parameters to be determined by the operator-guided registration process.

In some embodiments, as shown in the examples of <FIG>, <FIG>, the operator-guided registration process is configured to determine a single parameter (e.g., a bearing angle α describing rotation around the Z-axis) of the base transformation b<NUM>Rb<NUM> from the camera base reference frame <NUM> to the tool base reference frame <NUM>. In those embodiments, a single-parameter marker (e.g., marker <NUM> of <FIG>, marker <NUM> of <FIG>) associated with that single parameter (e.g., having an axis associated with the corresponding axis) may be used to assist the operator S to perform the alignment step. The single-parameter marker may have a shape (e.g., a line, a cone, a cylinder) with an axis (e.g., axis <NUM> of <FIG>, axis <NUM> of <FIG>) associated with the single parameter to be determined. In a subsequent process, the operator S may align the single-parameter marker with an alignment target of the tool by moving the master control device, such that the axis of the single-parameter marker is aligned with the corresponding alignment axis of the alignment target.

In some embodiments, as shown in the example of <FIG>, the operator-guided registration process is configured to determine multiple parameters including rotational parameters (e.g., describing rotation about the X, Y, and Z-axes) and/or translational parameters (e.g., describing displacement along the X, Y, and Z-axes) associated with the base transformation b<NUM>Rb<NUM>. In those embodiments, a multi-parameter marker (e.g., virtual tool <NUM> of <FIG>) associated with those multiple parameters may be used to assist the operator S in the alignment step. The multi-parameter marker may have a shape (e.g. a virtual representation of the tool <NUM>) associated with those multiple parameters to be determined. In a subsequent process, the operator S may align the multi-parameter marker with the tool such that the multi-parameter marker and the tool completely overlay in the display by moving the master control device.

At process <NUM>, after determining the marker type, a marker of that determined marker type is shown in the display. The initial position and orientation of the marker on the display may be determined based on the alignment relationship between the master control device and the display and the pose of the master control device.

The method <NUM> may then proceed to process <NUM>, where an operator S is instructed to align the marker with an alignment target of the tool provided in the display. In embodiments where a single parameter is to be determined, the operator S may be instructed to align the marker with an alignment target of the tool along a single axis. In the example of <FIG>, the operator S may be instructed to control the position and orientation of the marker <NUM> using the master control device, such that the marker <NUM> is aligned with the shaft <NUM> of the tool <NUM> along axes <NUM> and <NUM>. In the example of <FIG>, the operator S may be instructed to control the position and orientation of marker <NUM> using the master control device, such that the marker <NUM> overlays the shaft <NUM> in the display <NUM>. In the example of <FIG>, the operator S may be instructed to control the position and orientation of the virtual tool <NUM> using the master control device, such that the virtual tool <NUM> and the tool <NUM> overlay in the display <NUM>.

The method <NUM> may then proceed to process <NUM>, where the control system receives, from the operator S, an alignment completion indication indicating that the master control device is aligned with the alignment target. The alignment completion indication may be provided by the operator S using the master control device (e.g., using a hand grip, a foot pedal, a voice recognition device, and the like). In the example <FIG>, an operator S may provide such an alignment completion indication when the axis <NUM> of the marker <NUM> is parallel to or collinear with the shaft <NUM> of the tool <NUM> in the display <NUM>. In the example of <FIG>, the marker <NUM> has a shape (e.g., a cylinder) that is the same as that of the shaft <NUM>. In that example, an operator S may provide such an alignment completion indication after the marker <NUM> overlays with the shaft <NUM> of the tool <NUM>. In the example of <FIG>, an operator S may provide such an alignment completion indication after the virtual tool <NUM> (including its virtual shaft <NUM>, virtual wrist <NUM>, and virtual end effector <NUM>) overlays with the tool <NUM> (including its shaft <NUM>, wrist <NUM>, and end effector <NUM>).

The method <NUM> may then proceed to process <NUM>, where the control system determines the base transformation b<NUM>Rb<NUM> of the robotic medical system. In the examples of <FIG>, <FIG>, a single parameter (e.g., bearing angle α) may be computed according to equations (<NUM>), (<NUM>), and (<NUM>). In those examples, the base transformation b<NUM>Rb<NUM> may then be determined according to equation (<NUM>) using that single parameter. In the examples of <FIG>, multiple parameters (e.g., rotational parameters and translational parameters) may be computed according to equation (<NUM>). In those examples, the base transformation b<NUM>Rb<NUM> may then be determined according to equation (<NUM>) using those multiple parameters.

The method <NUM> may then proceed to process <NUM>, where the control system switches to a tool control mode, and reconnects the control loop between the master control device and the tool. When operating in the tool control mode, the control system may control the movement of the tool relative to the camera frame in response to movement of a master control device associated with the tool. To effectively move the tool in the camera frame, the control system determines an alignment relationship between the camera reference frame and the end effector reference frame using the base transformation (e.g., b<NUM>Rb<NUM> determined at process <NUM>). For example, the control system may compute a transformation cRend effector from the camera frame c to the end effector reference frame as follows: <MAT> where cRb<NUM> is a transformation from the camera reference frame <NUM> to the camera base reference frame <NUM>, b<NUM>Rend effector is a transformation from the tool base reference frame <NUM> to the end effector reference frame. cRb<NUM> and b<NUM>Rend effector are transformations that may be determined based on the forward and inverse kinematics of the manipulator assemblies <NUM> and <NUM> respectively, and b<NUM>Rb<NUM> is already determined previously at process <NUM> by the operator-guided registration process.

In some embodiments, at process <NUM>, the control system may derive a master-tool transform in response to state variable signals provided by the imaging system, so that an image of the tool in a display appears substantially connected to the master control device. These state variables generally indicate the Cartesian position of the field of view of the imaging device, as supplied by the manipulator supporting the imaging device. The control system may derive the master-tool transform using the base transformation b<NUM>Rb<NUM> determined by the operator-guided registration process, such that the control system may properly control movement of the tool <NUM> relative to the camera frame in response to the movement of the master control device.

In various embodiments, the operator-guided registration process may be performed before or during an operation (e.g., a medical operation). In an medical example, the operator-guided registration process may be performed before the medical operation (e.g. during set-up) outside of the patient or inside the patient. In another example, the operator-guided registration process may be performed during the medical operation. In yet another example, the operator-guided registration process may be performed as a back-up and/or calibration-check registration method where another registration process (e.g., a registration process using sensors on the bases or a registration process using image processing) is the primary registration process. In yet another example, the operator-guided registration process may be used in a robotic system having manipulators on the same base to check and confirm registration of those manipulators with their respective tools.

One or more elements in embodiments of the invention may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a processor-readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device, The code segments may be downloaded via computer networks such as the Internet, Intranet, etc..

Claim 1:
A robotic system (<NUM>) comprising:
a display (<NUM>) that is viewable by an operator, wherein an operator reference frame (<NUM>) is defined relative to the display (<NUM>) or the operator viewing the display (<NUM>);
an input device (<NUM>) movable by the operator;
a processing unit (<NUM>) including one or more processors, the processing unit configured to:
present, in the display (<NUM>), a first image of a first tool (<NUM>, <NUM>) captured by an imaging device (<NUM>, <NUM>);
present, in the display, a marker (<NUM>) controlled by the input device (<NUM>);
receive, from the operator, a first indication that the marker (<NUM>) is aligned with an alignment target (<NUM>, <NUM>), the alignment target (<NUM>, <NUM>) presented in the display and associated with the first image; and
in response to the first indication, determine a first alignment relationship based on a second alignment relationship, the first alignment relationship being between the imaging device (<NUM>, <NUM>) and a first tool (<NUM>, <NUM>, and the second alignment relationship being between the operator reference frame (<NUM>) and the input device (<NUM>).