Patent Description:
Computer-assisted surgical procedures, which may include image guided surgery and robotic surgery, have attracted increased interest in recent years. These procedures include the integration of a "virtual" three-dimensional dataset of the patient's anatomy, typically obtained using pre-operative or intra-operative medical imaging (e.g., x-ray computed tomography (CT) or magnetic resonance (MR) imaging), to the actual position of the patient and/or other objects (e.g., surgical instruments, robotic manipulator(s) or end effector(s) in the surgical area. These procedures may be used to aid the surgeon in planning a surgical procedure and may also provide the surgeon with relevant feedback during the course of surgical procedure. There is a continuing need to improve the safety and ease-of-use of computer-assisted surgical systems.

Document <CIT> forms prior art according to Art. <NUM>(<NUM>) EPC and discloses an apparatus including a cart for a surgical robotic arm having a coupler releasably coupleable to a coupling site on a surgical table. The cart includes a base and a first engagement feature. The base is freely movable on a support surface between a first location remote from the surgical table and a second loeation adjacent the surgical table. The first engagement feature is configured for engagement with a second engagement feature associated with the surgical table such that, when the first engagement feature and the second engagement feature are engaged, the coupler of the robotic arm is disposed in a position in which the coupler of the robotic arm can be engaged by the coupler of the surgical table.

Document <CIT> discloses systems and methods for mounting a robotic arm for use in robotic-assisted surgery including a mobile shuttle. The mobile shuttle includes a support member for mounting the robotic arm that extends at least partially over a gantry of an imaging device.

According to aspects of the present disclosure, a method for transferring a robotic arm from a mounting surface to which the robotic arm is attached during use to a mobile cart for storage and/or transport of the robotic and a system for robot-assisted surgery are provided according to the independent claims. Preferred embodiments are recited in the dependent claims. Various embodiments include methods and systems for performing robot-assisted surgery.

Examples not forming part of the invention include a method for controlling a robotic arm that includes tracking a motion of a handheld device using a motion tracking system, and controlling a robotic arm to adjust at least one of a position and an orientation of an end effector of the robotic arm based on the tracked motion of the handheld device.

Embodiments include a method for transferring a robotic arm from a mounting surface to which the robotic arm is attached during use to a mobile cart for storage and/or transport of the robotic arm, where the method includes tracking the location of the mobile cart relative to the robotic arm using a motion tracking system, and controlling the robotic arm to move the robotic arm into a pose that facilitates transferring the robotic arm from the mounting surface to the mobile cart based on the tracked location of the mobile cart. The mounting surface is located on a carriage that is moveable along a support element. The method further comprises moving the carriage and the robotic arm along the support element to a pre-determined loading/unloading position.

Various embodiments include robotic systems including processors configured to perform operations of the embodiment methods disclosed herein. Various embodiments also include robotic systems including means for performing functions of the embodiment methods disclosed herein. Various embodiments also include non-transitory processor- and server-readable storage media having stored thereon processor-executable instructions configured to cause a processor to perform operations of the embodiment methods disclosed herein.

Other features and advantages of the present invention will be apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings of which:.

<FIG> illustrates a system <NUM> for performing robotically-assisted image-guided surgery according to various embodiments. The system <NUM> in this embodiment includes an imaging device <NUM>, a motion tracking system <NUM> and a robotic arm <NUM> for performing a robotically-assisted surgical procedure. The robotic arm <NUM> may comprise a multi-joint arm that includes a plurality of linkages connected by joints having actuator(s) and optional encoder(s) to enable the linkages to rotate, bend and/or translate relative to one another in response to control signals from a robot control system. The robotic arm <NUM> may be fixed to a support structure at one end and may have an end effector <NUM> at the other end of the robotic arm <NUM>.

The imaging device <NUM> may be used to obtain diagnostic images of a patient <NUM>, which may be a human or animal patient. In embodiments, the imaging device <NUM> may be an x-ray computed tomography (CT) imaging device. The patient <NUM> may be positioned within a central bore <NUM> of the imaging device <NUM> and an x-ray source and detector may be rotated around the bore <NUM> to obtain x-ray image data (e.g., raw x-ray projection data) of the patient <NUM>. The collected image data may be processed using a suitable processor (e.g., computer) to perform a three-dimensional reconstruction of the object. In other embodiments, the imaging device <NUM> may comprise one or more of an x-ray fluoroscopic imaging device, a magnetic resonance (MR) imaging device, a positron emission tomography (PET) imaging device, a single-photon emission computed tomography (SPECT), or an ultrasound imaging device. In embodiments, image data may be obtained pre-operatively (i.e., prior to performing a surgical procedure) or intra-operatively (i.e., during a surgical procedure) by positioning the patient <NUM> within the bore <NUM> of the imaging device <NUM>. In the system <NUM> of <FIG>, this may be accomplished by moving the imaging device <NUM> over the patient <NUM> to perform a scan while the patient <NUM> may remain stationary.

Examples of x-ray CT imaging devices that may be used according to various embodiments are described in, for example, <CIT>, <CIT>, <CIT>, <CIT>, <CIT> and <CIT>. In the embodiment shown in <FIG>, the patient support <NUM> (e.g., surgical table) upon which the patient <NUM> may be located is secured to the imaging device <NUM>, such as via a column <NUM> which is mounted to a base <NUM> of the imaging device <NUM>. A portion of the imaging device <NUM> (e.g., an O-shaped imaging gantry <NUM>) which includes at least one imaging component may translate along the length of the base <NUM> on rails <NUM> to perform an imaging scan of the patient <NUM>, and may translate away from the patient <NUM> to an out-of-the-way position for performing a surgical procedure on the patient <NUM>.

An example imaging device <NUM> that may be used in various embodiments is the AIRO® intra-operative CT system manufactured by Mobius Imaging, LLC and distributed by Brainlab, AG. Other imaging devices may also be utilized. For example, the imaging device <NUM> may be a mobile CT device that is not attached to the patient support <NUM> and may be wheeled or otherwise moved over the patient <NUM> and the support <NUM> to perform a scan. Examples of mobile CT devices include the BodyTom® CT scanner from Samsung Electronics Co. and the O-arm® surgical imaging system form Medtronic, plc. The imaging device <NUM> may also be a C-arm x-ray fluoroscopy device. In other embodiments, the imaging device <NUM> may be a fixed-bore imaging device, and the patient <NUM> may be moved into the bore of the device, either on a surgical support <NUM> as shown in <FIG>, or on a separate patient table that is configured to slide in and out of the bore. Further, although the imaging device <NUM> shown in <FIG> is located close to the patient <NUM> within the surgical theater, the imaging device <NUM> may be located remote from the surgical theater, such as in another room or building (e.g., in a hospital radiology department).

The motion tracking system <NUM> shown in <FIG> includes a plurality of marker devices <NUM>, <NUM>, <NUM> and an optical sensor device <NUM>. Various systems and technologies exist for tracking the position (including location and/or orientation) of objects as they move within a three-dimensional space. Such systems may include a plurality of active or passive markers fixed to the object(s) to be tracked and a sensing device that detects radiation emitted by or reflected from the markers. A 3D model of the space may be constructed in software based on the signals detected by the sensing device.

The motion tracking system <NUM> in the embodiment of <FIG> includes a plurality of marker devices <NUM>, <NUM> and <NUM> and a stereoscopic optical sensor device <NUM> that includes two or more cameras (e.g., IR cameras). The optical sensor device <NUM> may include one or more radiation sources (e.g., diode ring(s)) that direct radiation (e.g., IR radiation) into the surgical field, where the radiation may be reflected by the marker devices <NUM>, <NUM> and <NUM> and received by the cameras. The marker devices <NUM>, <NUM>, <NUM> may each include three or more (e.g., four) reflecting spheres, which the motion tracking system <NUM> may use to construct a coordinate system for each of the marker devices <NUM>, <NUM> and <NUM>. A computer <NUM> may be coupled to the sensor device <NUM> and may determine the transformations between each of the marker devices <NUM>, <NUM>, <NUM> and the cameras using, for example, triangulation techniques. A 3D model of the surgical space in a common coordinate system may be generated and continually updated using motion tracking software implemented by the computer <NUM>. In embodiments, the computer <NUM> may also receive image data from the imaging device <NUM> and may register the image data to the common coordinate system as the motion tracking system <NUM> using image registration techniques as are known in the art. In embodiments, a reference marker device <NUM> (e.g., reference arc) may be rigidly attached to a landmark in the anatomical region of interest (e.g., clamped or otherwise attached to a bony portion of the patient's anatomy) to enable the anatomical region of interest to be continually tracked by the motion tracking system <NUM>. Additional marker devices <NUM> may be attached to surgical tools <NUM> to enable the tools <NUM> to be tracked within the common coordinate system. Another marker device <NUM> may be rigidly attached to the robotic arm <NUM>, such as on the end effector <NUM> of the robotic arm <NUM>, to enable the position of robotic arm <NUM> and end effector <NUM> to be tracked using the motion tracking system <NUM>. The computer <NUM> may also include software configured to perform a transform between the joint coordinates of the robotic arm <NUM> and the common coordinate system of the motion tracking system <NUM>, which may enable the position and orientation of the end effector <NUM> of the robotic arm <NUM> to be controlled with respect to the patient <NUM>.

In addition to passive marker devices described above, the motion tracking system <NUM> may alternately utilize active marker devices that may include radiation emitters (e.g., LEDs) that may emit radiation that is detected by an optical sensor device <NUM>. Each active marker device or sets of active marker devices attached to a particular object may emit radiation in a pre-determined strobe pattern (e.g., with modulated pulse width, pulse rate, time slot and/or amplitude) and/or wavelength which may enable different objects to be uniquely identified and tracked by the motion tracking system <NUM>. One or more active marker devices may be fixed relative to the patient, such as secured to the patient's skin via an adhesive membrane or mask. Additional active marker devices may be fixed to surgical tools <NUM> and/or to the end effector <NUM> of the robotic arm <NUM> to allow these objects to be tracked relative to the patient.

In further embodiments, the marker devices may be passive maker devices that include moiré patterns that may enable their position and orientation to be tracked in three-dimensional space using a single camera using Moiré Phase Tracking (MPT) technology. Each moiré pattern marker may also include a unique identifier or code that may enable different objects within the camera's field of view to be uniquely identified and tracked. An example of an MPT-based tracking system is available from Metria Innovation Inc. of Milwaukee, Wisconsin. Other tracking technologies, such as computer vision systems and/or magnetic--based tracking systems, may also be utilized.

The system <NUM> may also include a display device <NUM> as schematically illustrated in <FIG>. The display device <NUM> may display image data of the patient's anatomy obtained by the imaging device <NUM>. The display device <NUM> may facilitate planning for a surgical procedure, such as by enabling a surgeon to define one or more target positions in the patient's body and/or a path or trajectory into the patient's body for inserting surgical tool(s) to reach a target position while minimizing damage to other tissue or organs of the patient. The position and/or orientation of one or more objects tracked by the motion tracking system <NUM> may be shown on the display <NUM>, and may be shown overlaying the image data. In the embodiment of <FIG>, the display <NUM> is located on a mobile cart <NUM>. A computer <NUM> for controlling the operation of the display <NUM> may also be housed within the cart <NUM>. In embodiments, the computer <NUM> may be coupled to the optical sensor device <NUM> and may also perform all or a portion of the processing (e.g., tracking calculations) for the motion tracking system <NUM>. Alternatively, one or more separate computers may perform the motion tracking processing, and may send tracking data to computer <NUM> on the cart <NUM> via a wired or wireless communication link. The one or more separate computers for the motion tracking system <NUM> may be located on the imaging system <NUM>, for example.

<FIG> illustrates an alternative embodiment in which the optical sensor device <NUM> includes a plurality of cameras <NUM> mounted to an arm <NUM> extending above the patient <NUM> surgical area. The arm <NUM> may be mounted to or above the imaging device <NUM>. The arm <NUM> may also enable the sensor device <NUM> to pivot with respect to the arm <NUM> and/or the imaging device <NUM> (e.g., via one or more ball joints <NUM>). The arm <NUM> may enable a user to adjust the position of the sensor device <NUM> to provide the cameras <NUM> with a clear view into the surgical field while avoiding obstructions. The arm <NUM> may enable the position and/or orientation of the sensor device <NUM> to be adjusted and then locked in place during an imaging scan or surgical procedure. The positioning of the optical sensor device <NUM> on an arm <NUM> may also enable the cameras <NUM> to more easily view and track markers <NUM> (see <FIG>) that may be located on the imaging device <NUM>, such as on the outer surface of the gantry <NUM>, which may be used during automatic registration of patient images, as described further below.

<FIG> also illustrates a display device that may comprise a handheld display device <NUM>. As used herein, "handheld computing device" and "handheld display device" are used interchangeably to refer to any one or all of tablet computers, smartphones, pendant controllers, cellular telephones, personal digital assistants (PDA's), netbooks, e-readers, laptop computers, palm-top computers, wearable computers, and similar portable electronic devices which include a programmable processor and memory coupled to a display screen and may include hardware and/or software to enable display of information, including patient information and/or images, on the display screen. A handheld computing device typically also includes an antenna coupled to circuitry (e.g., a transceiver) to enable wireless communication over a network. A handheld computing or display device may be characterized by a sufficiently compact and lightweight structure to enable a user to easily grasp, maneuver and operate the device using one or both hands.

One or more handheld display devices <NUM> may be mounted to an arm <NUM> extending above the patient surgical area, as shown in <FIG>. The arm <NUM> may also support the optical sensing device <NUM> for the motion tracking system <NUM>, as described above. The one or more display devices <NUM> may be suspended from the arm <NUM>, and the position of a display device <NUM> may be adjustable along the length of the arm <NUM>. The display device <NUM> may be located within a sterile case or holder, such as described in <CIT>. In other embodiments, a handheld display device <NUM> may be mounted to the patient support <NUM> or column <NUM> or to any portion of the imaging system <NUM>, or to any of the wall, ceiling or floor in the operating room, or to a separate cart. One or more handheld display devices <NUM> may be used in addition to or as an alternative to a conventional display device, such as a cart-mounted monitor display device <NUM> as shown in <FIG>.

As shown in <FIG> and <FIG>, the robotic arm <NUM> may be fixed to the imaging device <NUM>, such as on a support element <NUM> (e.g., a curved rail) that may extend concentrically over the outer surface of the O-shaped gantry <NUM> of the imaging device <NUM>. In embodiments, an arm <NUM> to which the optical sensing device <NUM> is mounted may be mounted to the same or a similar support element <NUM> (e.g., curved rail) as the robotic arm <NUM>. The position of the robotic arm <NUM> and/or the arm <NUM> may be adjustable along the length of the support element <NUM>. In other embodiments, the robotic arm <NUM> may be secured to any other portion of the imaging device <NUM>, such as directly mounted to the gantry <NUM>. Alternatively, the robotic arm <NUM> may be mounted to the patient support <NUM> or column <NUM>, to any of the wall, ceiling or floor in the operating room, or to a separate cart. In further embodiments, the robotic arm <NUM> and/or the optical sensing device <NUM> may be mounted to a separate mobile shuttle, as described in <CIT>. Although a single robotic arm <NUM> is shown in <FIG> and <FIG>, it will be understood that two or more robotic arms <NUM> may be utilized.

<FIG> is a process flow diagram that illustrates a method <NUM> of registering patient images. Computer-assisted surgery techniques generally utilize a process of correlating a dataset representing a portion of the patient's anatomy that is to be operated on with the position of the patient at the time of the surgical intervention. The position of the patient may be determined based on a second image dataset which may include real-time camera image(s) from a motion tracking system <NUM> as described above. The correlation between these datasets may be accomplished computationally using software, and may be referred to as "patient registration. " The registration method <NUM> of <FIG> may be implemented using one or more computing devices, such as computer <NUM> shown in <FIG>.

In block <NUM> of method <NUM>, a first image dataset of the patient's anatomy may be obtained using an imaging device, such as the imaging device <NUM> shown in <FIG> and <FIG>. The first image dataset may be a three-dimensional dataset (e.g., a 3D CT tomographic reconstruction, a 3D MRI dataset, etc.) representing at least a portion of the patient's anatomy, including the internal anatomy and/or structure(s) that are to be operated on (i.e., a surgically-relevant portion of the patient's anatomy). The first image dataset may be stored electronically in a memory. The first image dataset may be in any suitable format, such as in a file format that conforms to the Digital Imaging and Communications in Medicine (DICOM) standard.

In block <NUM> of method <NUM>, a second image dataset of the patient and the surrounding patient space may be obtained using a motion tracking system, such as the motion tracking system <NUM> shown in <FIG> and <FIG>. The second image dataset may indicate the current position and/or orientation of the patient. The second image dataset may include at least one image of a marker device that may be obtained using an optical sensing device <NUM> (e.g., cameras <NUM>). The marker device (e.g., reference arc <NUM>) detected by the optical sensing device <NUM> may be in a known fixed relationship with the surgically-relevant portion of the patient's anatomy. The motion tracking system <NUM> may determine the transformation between the marker device <NUM> and the optical sensing device <NUM> (e.g., using well-known triangulation techniques), and may thereby determine the transformation between the sensing device <NUM> (e.g., camera <NUM> position) and the surgically-relevant portion of the patient's anatomy. The motion tracking system <NUM> may similarly determine transformations between each of the other marker devices (e.g., marker devices <NUM> and <NUM> in <FIG>) and the optical sensing device <NUM>. Each of the markers <NUM>, <NUM> and <NUM> being tracked may then be placed within a common coordinate system. In embodiments, the common coordinate system may have an origin or zero point that may be considered to be fixed relative to the surgically-relevant portion of the patient's anatomy, and may also be referred to the patient coordinate system.

In block <NUM> of method <NUM>, the first image dataset may be registered to the common coordinate system as the second image dataset (e.g., the patient coordinate system). This may include performing a rigid transformation to map each pixel or voxel of the first image dataset into corresponding 3D coordinates (i.e., x, y, z coordinates) of the common coordinate system. A number of techniques may be utilized for registering multiple image datasets. In one non-limiting example of a registration process for x-ray CT imaging data, a pre-scan calibration process may be used to precisely calculate (e.g., within <NUM> and/or <NUM>°) the transformation between the isocenter of the x-ray gantry <NUM> and the optical sensing device <NUM>. A set of markers <NUM> (e.g., <NUM> or more, such as <NUM>-<NUM> markers) may be provided on the surface of the gantry <NUM>, as shown in <FIG>. The markers <NUM> may be within the field of view of the optical sensing device <NUM> to enable the gantry <NUM> position to be tracked by the motion tracking system <NUM>. A calibration phantom (not shown for clarity) having a marker device (e.g., similar to marker device <NUM> in <FIG>) fixed thereto may be placed on the patient support <NUM> such that the marker device is also within the field of view of the optical sensing device <NUM>. The motion tracking system <NUM> may determine the transformation between the gantry <NUM> coordinate system defined by the markers <NUM> and the optical sensing device <NUM> coordinate system as well as the transformation between the phantom coordinate system defined by the marker device on the phantom and the optical sensing device <NUM> coordinate system. These transformations may be used to determine the gantry-to-phantom transformation. The phantom may then be scanned using the imaging device <NUM>. A set of elements (e.g., x-ray visible beads) that may be easily identified from the imaging data may be located in the phantom, where the geometry of these elements within the phantom coordinate system may be previously-known. An algorithm may be used to analyze the x-ray image data to identify the x-ray visible elements with respect to the center point of the image data, which corresponds to the isocenter of the gantry <NUM>. Thus, the x-ray visible elements may be located in a coordinate system having an origin at the isocenter of the x-ray gantry <NUM>, and the transformations between the isocenter and the phantom and the isocenter and the markers <NUM> on the gantry <NUM> may be calculated.

During a subsequent scan of the patient <NUM>, the position and orientation of the patient <NUM> with respect to the isocenter of the imaging device <NUM> may be determined (i.e., by tracking the positions of the markers <NUM> on the gantry <NUM>, which are known with respect to the isocenter, and the patient reference arc <NUM>, which is known with respect to the surgically-relevant portion of the patient anatomy). This may enable the image data obtained during the scan to be registered into the patient coordinate system.

In an alternative embodiment, the position of the optical sensing device <NUM> may be known relative to the imaging system <NUM> with sufficient accuracy such that the image dataset of the patient's anatomy obtained using the imaging system <NUM> may be registered in the common coordinate system of the patient without the motion tracking system <NUM> needing to track the position or orientation of the imaging system <NUM>. In embodiments, separate markers <NUM> on the gantry <NUM> of the imaging system <NUM> as shown in <FIG> may not be required or used. In some embodiments, the position of the optical sensing device <NUM> (e.g., the position of each of the cameras <NUM> as shown in <FIG> and <FIG>) may be known relative to the isocenter of the gantry <NUM> of the imaging system <NUM>, such as via a calibration process that may be performed at the factory or during installation or pre-calibration of the system. The gantry <NUM> and/or the optical sensing device <NUM> may include keying features (e.g., high-precision bolt patterns) where the optical sensing device <NUM> attaches to the gantry <NUM> to ensure that the position of the sensing device <NUM> on the gantry <NUM> remains accurately fixed. In embodiments where the camera(s) <NUM> may be movable relative to the gantry <NUM>, high-precision encoders may precisely record and correct for any changes in camera position/orientation relative to the isocenter of the gantry <NUM>. During imaging scans, the optical sensing device <NUM> may track the position and orientation of the patient <NUM> with respect to the camera position, which is in a known, fixed geometric relationship with the isocenter of the imaging device <NUM>. The image data obtained during a scan may thus be registered into the common coordinate system of the patient without needing to first perform a calibration scan on a phantom, as described above.

In block <NUM> of method <NUM>, images of the patient's anatomy from the first image dataset may be displayed with an overlay of one or more features derived from the second image dataset in the common coordinate system. The images may be displayed on a suitable display device, such as display <NUM> shown in <FIG>. The images of the patient's anatomy may include 2D slices of a three-dimensional image dataset (e.g., a tomographic reconstruction) and/or a 3D volume rendering of all or a portion of the image dataset. In embodiments, images obtained using multiple imaging devices or imaging modalities may be fused and displayed in a common coordinate system. For example, the first image dataset of the patient's internal anatomy may be an x-ray CT scan. Another image dataset of the patient's internal anatomy, such as an MRI scan, may be combined with the x-ray CT data and displayed on the display <NUM>. The MRI scan data may be registered into the common coordinate system using a similar registration process as described above. Alternately or in addition, an algorithm for matching landmarks or fiducials identifiable from both image datasets may be used to merge the datasets for display.

The one or more features derived from the second image dataset that may be displayed overlaying the images of the patient's anatomy may include graphical depictions of a tool <NUM>, an end effector <NUM> or another object that is tracked by the motion tracking system <NUM>. The graphical depiction may be based on a known geometry of the tool <NUM>, end effector <NUM> or another object. The graphical depiction may be a rendering of the actual size and shape of the object or may be a depiction of select features of the object, such as a location of a tip end of the object and/or an orientation of the object. The graphical depiction may also indicate a trajectory defined by the object (e.g., a ray extending from a tip end of the object into the patient) and/or a target point within the patient's anatomy that may be defined based on the position and/or orientation of one or more objects being tracked. In various embodiments, the tool <NUM> may be a pointer. The tool <NUM> may also be a surgical instrument, such as a needle, a cannula, dilator, a tool for gripping or cutting, an electrode, an implant, a drill bit, a screw, a screw driver, a radiation source, a drug and an endoscope. In embodiments, the end effector <NUM> of the robotic arm <NUM> may include a hollow tube or cannula that may be configured to hold one or more tools, such as a surgical instrument, and may be used to guide an instrument as it is inserted into the patient's body. Alternately, the end effector <NUM> itself may be or may include an instrument that may be inserted into the patient's body.

The motion tracking system <NUM> may repeatedly acquire new images from the optical sensing device <NUM>, and the relative positions and/or orientations of objects within the field of view of the optical sensing device <NUM> may be updated with each acquisition of new images from the optical sensing device <NUM>. The display <NUM> may be updated to reflect any change(s) in the position and/or orientation of the objects within the common coordinate system (e.g., relative to the patient reference are <NUM>), which may include adding additional graphical elements to depict new objects that are moved within the field of view of the optical sensing device <NUM> and removing graphical depictions of objects when they are no longer within the field of view of the optical sensing device <NUM>. In some embodiments, the optical sensing device <NUM> may include a motorized system to enable the position and/or orientation of the camera(s) <NUM> to move to maintain the surgical area within the center of the field of view of the camera(s) <NUM>.

<FIG> is a component block diagram of an image-guided surgery system <NUM> according to an embodiment. The system <NUM> may be implemented using one or more computing devices, such as computer <NUM> shown in <FIG>. The system <NUM> may be operatively coupled to a first display device <NUM>, which may include a monitor that is fixed to a cart <NUM> or other structure (e.g., wall, ceiling, floor, imaging device, etc.) within the operating suite. The system <NUM> may also be operatively coupled to at least one additional display device <NUM>, which may be a handheld computing device, as described above. The system <NUM> may also include an audio input/output component <NUM>, which may include a speaker or other output component for outputting audible signals (e.g., audio instructions, alerts, etc.) and/or a microphone or other input component for receiving audio inputs (e.g., voice commands) that may be interpreted by the system <NUM>. The system <NUM> may be implemented at least partially in software and may be based on one or more of the Image-Guided Surgery Toolkit (IGSTK), Visualization Toolkit (VTK) and Insight Segmentation and Registration Toolkit (ITK) development frameworks.

The system <NUM> may be configured to receive and store imaging data <NUM> (e.g., DICOM data) collected by an imaging device <NUM>. The imaging data <NUM> may be received directly from the imaging device <NUM> or may be retrieved from another source, such as a remote server. The imaging data <NUM> may be imaging data that is obtained prior to a surgical procedure (e.g., pre-operative image data) and/or imaging data that is obtained during a surgical procedure (e.g., intra-operative image data). In embodiments, the system <NUM> may be configured to display the most-current image data <NUM> collected by the imaging device <NUM>. The image data <NUM> may be registered to a common coordinate system as the tracking data <NUM> from the motion tracking system <NUM> in accordance with a registration method such as method <NUM> described above with reference to <FIG>.

The system <NUM> may also receive tracking data <NUM> from a motion tracking system <NUM>. The system <NUM> may be configured to repeatedly read the tracking data from the motion tracking system <NUM> indicating the current position/orientation of the patient and any other objects tracked by the motion tracking system <NUM>. The system <NUM> may read the tracking data at a frequency (e.g., refresh rate) of greater than <NUM> (e.g., <NUM>). In embodiments, the tracking data from the motion tracking system <NUM> may include data to enable the system <NUM> to identify particular objects from within the tracking data. For example, each marker device (e.g., marker devices <NUM>, <NUM> and <NUM> in <FIG>) may include a unique characteristic (e.g., a unique geometric pattern of reflective markers, a unique flash pattern of active markers, etc.) to enable the marker device to be identified. These unique characteristics of the marker devices may be registered with particular objects or tools (e.g., associated with a particular object or tool in a database) by the system <NUM>. The unique characteristics of the marker devices may be pre-registered in the system <NUM> and/or may be registered to particular objects or tools during the course of a surgical procedure. The system <NUM> may also include a library of graphical elements that may be associated with particular objects or tools (e.g., in a database). The system <NUM> may display graphical elements associated with the objects or tools being tracked by the motion tracking system <NUM> in the common coordinate system with the image data on the display(s) <NUM>, <NUM>.

The system <NUM> may include a user-interface component that may control the display of system information and/or graphical user interface elements on the display(s) <NUM> and <NUM>. The system <NUM> may further process and implement user commands received from user interface devices. A user interface device, may include, for example, a touchscreen user interface which may be integrated with a display device <NUM>, <NUM>. In embodiments, a user interface device may alternately or additionally include one or more of a button, a keyboard, a joystick, a mouse, a touchpad, etc. which may be located on a display device <NUM>, <NUM> and/or on a workstation (e.g., a workstation located on a cart <NUM>). In embodiments, the user interface device(s) may also include a microphone (e.g., audio input/output component <NUM>) that may receive voice commands that may be interpreted by the system (e.g., using voice recognition software). The user commands received via one or more user input devices may enable a user to control various functions of the system <NUM>, such as changing what is shown on the display(s) <NUM>, <NUM> (e.g., displaying different image datasets, displaying different slice(s) and/or different 3D rendering(s) within an image dataset, zooming in or out of an image, displaying different menu options, returning to a home screen, etc.). In embodiments, the user commands may enable a user to set one or more trajectories and/or target locations within the patient's anatomy. The system <NUM> may store the positions and/or orientations of user-defined trajectories or target locations within the common coordinate system, and may display graphical representations of such trajectories or target locations on the display(s) <NUM><NUM>, <NUM>.

The user commands received by the system <NUM> may also include commands for controlling the operation of other components, such as the imaging device <NUM>, the motion tracking system <NUM> and/or a robotic arm <NUM>. For example, for a robotically-assisted surgical procedure, the user command may include an instruction to move a robotic arm <NUM> to a particular position and/or orientation. The instruction to move the robotic arm <NUM> may be based on a user interaction with image data of the patient's anatomy that is displayed on a display device <NUM>, <NUM>. For example, the user may use the display device <NUM>, <NUM> to define a particular trajectory with respect to the patient's anatomy and may send an instruction for the robotic arm <NUM> to move such that that the end effector <NUM> of the robotic arm <NUM> is positioned along the defined trajectory.

A robotic control system <NUM> may control the movement of one or more robotic arms <NUM>. The robotic control system <NUM> may receive sensor data indicating the current parameters of the robotic arm <NUM> (e.g., robot position, joint angles, measured axis forces, motor currents) and may send motor control signals to drive the movement of the arm <NUM>. In embodiments, the motion tracking system <NUM> may track the position of the robotic arm <NUM> (e.g., via marker device <NUM> on end effector <NUM> as shown in <FIG>) to determine the position of the end effector <NUM> within the common coordinate system of the patient. A control loop, which may be executed using the image-guided surgery system <NUM>, the motion tracking system <NUM> and/or the robotic control system <NUM>, may continuously read the tracking data and the robot parameter data and may send instructions to the robotic control system <NUM> to cause the robotic arm <NUM> to move to a desired position and orientation.

In various embodiments, display device <NUM> may be a primary display device (e.g., a monitor) that may be connected to the image-guided surgery system <NUM> by a wired or wireless link. In one embodiment, the system <NUM> may stream video data to the display device <NUM> over a suitable video data interface (e.g., an HDMI interface) and may also exchange other signals with the display device over a separate data connection (e.g., a USB connection).

In various embodiments, display device <NUM> may be a handheld computing device. A handheld display device <NUM> may generally be smaller and lighter than the primary display device <NUM> (e.g., monitor), and may in certain embodiments be referred to as a secondary display device. In some embodiments, display device <NUM> may be a mirror of display device <NUM> and may display all or a portion of the same information as is shown on display device <NUM>. Alternately, display device <NUM> may display different information than is shown on display device <NUM>. In some embodiments, display device <NUM> may be omitted, and handheld display device <NUM> may be the only display device operably connected to the image-guided surgery system <NUM>. In such a case, display device <NUM> may be referred to as the primary display device. Further, although a single handheld display device <NUM> (i.e., a tablet computer) is shown in <FIG>, it will be understood that multiple handheld display devices <NUM> may be simultaneously connected to and used with the system <NUM>.

The handheld display device <NUM> may be coupled to the image-guided surgery system <NUM> by a wired or wireless communication link. In one embodiment, the handheld display device <NUM> may communicate with the system <NUM> over a wireless communication interface. The system <NUM> may stream digital video data (e.g., high-definition video) for display on the handheld display device <NUM>, such as over a wireless local area network (WLAN) connection, including a IEEE <NUM> (e.g., WiFi) connection. The system <NUM> may also exchange other signals with the handheld display device <NUM> (e.g., control signals from the system <NUM> and/or user commands received at a user interface, such as a touchscreen, on the display device <NUM>) over a wireless connection. The system <NUM> and the display device <NUM> may communicate over any suitable wireless protocol or standard, such as over a IEEE <NUM>. 15x (e.g., a BLUETOOTH®) connection.

An image-guided surgical system <NUM> according to various embodiments may provide a plurality of modes for displaying patient information. For example, a first display mode may include displaying a 3D image dataset (e.g., an x-ray CT, MRI, sonogram, PET or SPECT image dataset) in multiple two-dimensional slices corresponding to anatomic planes (e.g., axial, sagittal, coronal planes) transecting the patient. This is illustrated in the screenshot of a display device shown in <FIG>. The display device may be a display device <NUM> (e.g., monitor) as shown in <FIG> or a handheld display device as shown in <FIG> and <FIG>. The display screen <NUM> in this example illustrates four different patient images in four quadrants of the display screen <NUM>. Three of the quadrants (i.e., top left, top right and bottom left quadrants of display screen <NUM>) depict different two-dimensional slices <NUM>, <NUM>, <NUM> of CT image data. A fourth quadrant (i.e., lower left quadrant of display screen <NUM>) includes a 3D volume rendering <NUM> illustrating a "virtual" view of anatomic feature(s) (e.g., bony structures or other discrete internal anatomic features). The two-dimensional slices <NUM>, <NUM>, <NUM> correspond, respectively, to views taken along axial, sagittal and coronal planes through the patient <NUM>.

The display screen <NUM> may also display graphical elements illustrating the relationship of each slice <NUM>, <NUM>, <NUM> relative to the other slices shown on the display screen <NUM>. For example, as shown in <FIG>, the axial slice <NUM> image data may include an overlay of a cross pattern <NUM> showing the intersection of the axial slice <NUM> with the planes corresponding to the sagittal and coronal slices <NUM> and <NUM> shown on the display screen <NUM>. Similar cross patterns <NUM> may be displayed overlaying the display of image data in the sagittal and coronal slices <NUM> and <NUM>. The display screen <NUM> may also include graphical representations or renderings of other objects or tools tracked by the motion tracking system <NUM>. In the example of <FIG>, a graphical representation of a tool <NUM> is shown in the lower right quadrant of the display screen <NUM>. The graphical representation of the tool <NUM> may illustrate the position and orientation of the tool relative to the anatomic features depicted in the 3D volume rendering <NUM>. Similar graphical elements may be displayed in the 2D slice images <NUM>, <NUM> and <NUM> to illustrate the position and/or orientation of one or more objects with respect to the patient.

It will be understood that the four-quadrant view shown in <FIG> is one possible implementation of a display of patient information on a display device <NUM>, <NUM>. Other possible display modes are possible. For example, rather than illustrating multiple different images (e.g., slices) from a patient image dataset (e.g., reconstructed volume), the display screen <NUM> may show only a single image (e.g., a single axial, sagittal or coronal slice <NUM>, <NUM>, <NUM> or a single 3D volume rendering <NUM>). The display screen <NUM> may illustrate only two slices corresponding to different anatomic planes (e.g., axial and sagittal, axial and coronal, or sagittal and coronal slices), or may illustrate a single slice along with a 3D volume rendering. In some embodiments, the display screen <NUM> may illustrate multiple two-dimensional slices corresponding to the same anatomic planes (e.g., multiple axial, sagittal and/or coronal slices taken through different sections of the reconstructed volume) and/or multiple 3D volume renderings viewed from different angles. The different images and display modes of the display screen <NUM> may be customizable based on user selections, which may be made via a user input device and/or user voice commands. In embodiments, the user may be able to select (e.g., scroll through) different patient images, such as sequentially illustrating multiple axial, sagittal and/or coronal slices taken through different sections of the reconstructed volume, or sequentially illustrating multiple 3D volume renderings viewed from different angles. The display screen <NUM> may also display slices along oblique planes taken through the reconstructed volume. The user may also have the capability to control the magnification of images, such as by zooming into or out from a particular portion of an image shown in the display screen <NUM>. The user may control the selection of patient images for display using a user input device, voice commands and/or via a separate tool, such as a pointer device. In some embodiments, the intersection of the three image planes (i.e., axial, sagittal and coronal) shown on the display panel <NUM> may coincide with a target position within the patient's body. The surgeon may use the display panel <NUM> as a "virtual cutting tool" to move through the various slices/views of the patient image volume and to identify and select a target region for a surgical intervention.

The user (e.g., a surgeon) may be able to set one or more target positions and/or trajectories within the patient <NUM>. There may be a variety of ways to set a trajectory or target location. For example, the surgeon may move through different views of the patient image data by manipulating a tool (e.g., a pointer/stylus device and/or an end effector of a robotic arm) over the patient <NUM>, where the tool may define a unique trajectory into the patient. The tool may be tracked within the patient coordinate system using the motion tracking system <NUM>. In some embodiments, an imaginary ray projected forward from the tip end of the tool may define the unique trajectory into the patient, which may be graphically depicted on the display screen <NUM>. A target location along the unique trajectory may be defined based on a pre-determined offset distance from the tip end of the tool. Alternately, the surgeon may directly manipulate and interact with the displayed image data to identify a particular target or trajectory, such as using a workstation computer. A particular target point or trajectory may be set by the system <NUM> in response to an input event, which may include, for example, a voice command, a touch event on a touchscreen interface, and/or an input on a user interface device (e.g., a keyboard entry, a mouse click, a button push, etc.). In embodiments, the surgeon may set a target position and/or trajectory by interacting with image data displayed on a display device, such as display devices <NUM> and/or <NUM>. For example, the surgeon may define a target point and/or trajectory in the patient <NUM> by selecting one or more points on a display screen <NUM> of a display device <NUM>, <NUM> (e.g., marking the points using a stylus, a cursor or mouse pointer, or a touch on a touchscreen user interface). To define a trajectory, for instance, the user may select two or more points in the image data (e.g., a target point and an entrance point on the skin of the patient). In embodiments, the user may be able to make fine adjustments to a selected target point and/or trajectory using any suitable user interface device. Multiple target points and/or trajectories may be set and saved in a memory (e.g., in an image-guided surgery system <NUM> as illustrated in <FIG>), where each target point and/or trajectory may be saved in association with a unique identifier (e.g., file name).

In embodiments, the display screen <NUM> may display graphical element(s) overlaying the image data corresponding to one or more target locations and/or trajectories that are set by the user. For example, defined target locations may be illustrated as identifiable dots or points in the image data, which may be color coded and/or labeled on the display screen <NUM> to enable easy visualization. Alternately or in addition, defined trajectories may be depicted as identifiable lines or line segments in the image data, which may be similarly color coded and/or labeled. As discussed above, the display screen <NUM> may also display graphical elements associated with particular tools or objects, including invasive surgical tools or instruments, that are tracked by the motion tracking system <NUM>. In embodiments, the display screen <NUM> may depict at least a portion (e.g., a tip end) of a surgical instrument as it is inserted into the patient <NUM>, which may enable the surgeon to track the progress of the instrument as it progresses along a defined trajectory and/or towards a defined target location in the patient <NUM>. In some embodiments, the patient images on the display screen <NUM> may be augmented by graphical illustrations of pre-calibrated tools or implants (e.g., screws) that are located within the patient <NUM>.

The at least one robotic arm <NUM> may aid in the performance of a surgical procedure, such as a minimally-invasive spinal surgical procedure or various other types of orthopedic, neurological, cardiothoracic and general surgical procedures. In some embodiments, when the robotic arm <NUM> is pointed along a set trajectory to a target position, the robotic arm <NUM> may maintain a rigid or fixed pose to enable the surgeon to insert an instrument or tool through a cannula or similar guide arranged along a vector that coincides with the predefined trajectory into the body of the patient <NUM>. The cannula may be a portion of the end effector <NUM> of the robotic arm <NUM> or it may be separate component that is held by the end effector <NUM>. The cannula/guide may be positioned by the robotic arm <NUM> such that the central axis of the cannula is collinear with the pre-defined trajectory into the patient <NUM>. The surgeon may insert one or more invasive surgical instrument through the cannula/guide along the trajectory and into the body of the patient to perform a surgical intervention. Alternately, the end effector <NUM> itself may comprise a surgical instrument that may be moved into the body of the patient, such as, without limitation, a needle, a dilator, a tool for gripping, cutting or ablating tissue, an implant, a drill bit, a screw, a screw driver, a radiation source, a drug and/or an endoscope.

Various embodiments include methods and systems for controlling a robotic arm <NUM> to adjust a position and/or orientation of the end effector <NUM> of the robotic arm <NUM>. A robotic arm <NUM> such as shown in <FIG> and <FIG> may be moved autonomously to a particular pose by the robotic control system <NUM> (e.g., in accordance with a robotic motion planning algorithm). For example, in response to a user command for the robotic arm <NUM> to go to a pre-set target position or trajectory, the robotic control system <NUM> may control the robotic arm <NUM> to autonomously move the arm <NUM> to a pose with the tip end of the end effector <NUM> pointing along the pre-set trajectory to the target position. Optionally, the robotic arm <NUM> may also operate in a hand guiding mode in which the movement of the robotic arm <NUM> may be controlled based on a force applied by a user to the arm (e.g., using torque and/or force sensing feedback to the robotic control system <NUM>).

In various examples not forming part of the invention, the robotic arm <NUM> may also operate in a mode in which the robotic arm <NUM> is controlled to adjust a position and/or orientation of an end effector <NUM> of the robotic arm <NUM> based on a tracked motion of a handheld device. A motion tracking system <NUM> such as described above may track the motion of a handheld device (e.g., an instrument <NUM> having a marker device <NUM> attached thereto). The tracked motion of the handheld device may be provided to the robotic control system <NUM> so that the robotic control system <NUM> may control the robotic arm <NUM> to adjust the position and/or orientation of the end effector <NUM> based on the tracked motion. As used herein, this mode of controlling the motion of the robotic arm <NUM> may be referred to as "follow" mode.

<FIG> illustrates a portion of a robotic arm <NUM> including an end effector <NUM>, and a handheld device <NUM> having a marker device <NUM> that enables the handheld device <NUM> to be tracked using a motion tracking system <NUM>. The handheld device <NUM> may be any device that may be held and manipulated by a user. The handheld device <NUM> may be a stylus or pointer device or a surgical instrument (e.g., a needle, screw driver, scalpel, awl, etc.) having an attached marker device <NUM>. Alternately, the handheld device <NUM> may be a dedicated device used only for robot motion control. In examples, the handheld device <NUM> may be a separate component that is not physically connected or coupled to the robotic arm <NUM>.

The system may enter the "follow" mode for controlling the motion of the robotic arm <NUM> in response to an input event from the user. The input event may be, for example, a voice command, a touch event on a display screen, a button push, a mouse/keyboard click, depression of a foot pedal, etc. In some embodiments, the handheld device <NUM> may have a marker device <NUM> with a unique marker pattern such that the system may automatically enter the "follow" mode when the handheld device <NUM> is brought within the field-of-view of the optical sensor(s) <NUM> of the motion tracking system <NUM>.

As described above, the motion tracking system <NUM> may track the motion of the handheld device <NUM> in three-dimensional space, including the translation of the handheld device <NUM> (i.e., x, y and z translation) as well as rotational movement of the handheld device <NUM> (i.e., yaw, pitch and roll rotation). Tracking data corresponding to the motion of the handheld device <NUM> may be provided to the robotic control system <NUM>. The robotic control system <NUM> may perform motion planning based on the received tracking data and send control signals to the robotic arm <NUM> to cause the arm to perform a movement based on the tracked motion of the handheld device <NUM>. In the example of <FIG>, for example, a translation of the handheld device <NUM> in one or more directions (i.e., ± x, y, and/or z directions) may result in a corresponding translation of the end effector <NUM> in the same direction(s). Similarly, a tracked rotational motion of the handheld device <NUM> (i.e., yaw, pitch and/or roll rotation) may result in a corresponding rotation of the end effector <NUM> in the same direction(s). In examples, the end effector <NUM> may "follow" the motion of the handheld device <NUM>.

When operating in "follow" mode, the end effector <NUM> of the robotic arm <NUM> may perform a movement (i.e., translation and/or rotation) corresponding to a relative movement (translation and/or rotation) of the handheld device <NUM>. The handheld device <NUM> may be located and moved in an area that is away from the surgical site, and may avoid obstacles and sterility concerns associated with the surgical site. The user may also freely choose the starting position of their hand when using the handheld device as a control mechanism or "air mouse" for guiding the movements of robotic arm <NUM>.

<FIG> is a process flow diagram illustrating one example of a method <NUM> for controlling a robotic arm based on the tracked motion of a handheld device. Method <NUM>, that is not forming a part of the invention, may be implemented as a control loop on a processor of a robotic control system <NUM>, such as described above with reference to <FIG> and <FIG>. In block <NUM> of method <NUM>, the robotic control system <NUM> enters "follow" mode, which may be in response to a user input event, as described above. In block <NUM>, the robotic control system <NUM> may proceed to a standby state in which the robotic arm is not moved.

In determination block <NUM>, the robotic control system <NUM> may determine whether the system <NUM> is still operating in follow mode. In response to determining that the system <NUM> is still operating in follow mode (i.e., determination block <NUM> = "Yes"), the robotic control system <NUM> may determine whether the robotic arm <NUM> is in an active control state in determination block <NUM>. The system <NUM> may remain in the standby state (i.e., block <NUM>) in response to determining that the robotic arm <NUM> is not in an active control state (i.e., determination block <NUM> = "No").

As used herein, an active control state means that the robotic arm <NUM> is enabled to move in response to control signals received from the robotic control system <NUM>. In some examples, the robotic control system <NUM> may determine whether the robotic arm is in an active control state in determination block <NUM> based on whether or not a user input component is actuated. For example, controlling the motion of the robotic arm may require some form of continuous activation by the user. This may help prevent unintentional movement of the robotic arm <NUM>. The user input component that must be actuated to actively control the robotic arm <NUM> may be a button or similar apparatus (e.g., a foot pedal, pressure-sensitive pad, etc.) that must be pressed or held down to enable movement of the robotic arm <NUM>.

In the example of <FIG>, the user input component may be a button <NUM> on the handheld device <NUM>. The handheld device <NUM> may include circuitry <NUM> configured to detect an input event (e.g., a button push) at the user-interface component and transmit a signal (e.g., a motion-enable signal) that may be received by the robotic control system <NUM>. In some examples, the circuitry <NUM> may include wireless transceiver circuitry <NUM> configured to transmit signals wirelessly using a suitable wireless communication protocol or standard (e.g., an IEEE <NUM>. 15x (BLUETOOTH®) connection or IEEE <NUM> (WiFi) connection). The handheld device <NUM> may include a power supply <NUM> (e.g., battery source) to provide power to electronic components of the device <NUM>. Alternately, the handheld device <NUM> may include a wired link for providing power and/or transmitting signals.

Alternately or in addition, the robotic control system <NUM> may determine whether the robotic arm is in an active control state in determination block <NUM> based on whether the robotic control system <NUM> is receiving tracking data from the motion tracking system <NUM> corresponding to the current position and/or orientation of the handheld device <NUM>. In particular, the robotic arm <NUM> may only operate in an active control state while the robotic control system <NUM> is receiving up-to-date tracking data for the handheld device <NUM>. The robotic arm <NUM> may operate in an active control state when, for example, the handheld device <NUM> is within the field-of-view of the optical sensor(s) <NUM> and the marker device <NUM> is not obstructed. In addition, operation in the active control state may optionally also require the user to actuate a user input component (i.e., continuous activation).

In response to determining that the robotic arm is in an active control state (i.e., determination block <NUM> = "Yes"), the robotic control system <NUM> may plan a movement of the robotic arm <NUM> based on the tracked movement of the handheld device <NUM> in block <NUM>. In various examples, the robotic control system <NUM> may determine a change in position and/or orientation of the handheld device <NUM> between an initial position/orientation and a subsequent position and/or orientation. Based on the change in position and/or orientation of the handheld device <NUM>, the robotic control system <NUM> may then determine a corresponding change in position and/or orientation of the end effector <NUM> of the robotic arm <NUM>. In examples, the robotic control system <NUM> may utilize a motion planning algorithm (e.g., based on the inverse kinematics of the robotic arm) to plan the motion(s) of the robotic arm <NUM> to cause the change in the end effector <NUM> position and/or orientation.

In examples, the robotic control system <NUM> may determine whether a planned robot motion is allowed in optional determination block <NUM>. For example, the robotic control system <NUM> may include a collision model with pre-defined "no go" space(s) in order to prevent the robotic arm <NUM> from colliding with the patient or other objects. A planned robot motion may not be allowed if it would result in the robotic arm <NUM> violating a "no go" space. In response to determining that the planned robot motion is not allowed (i.e., determination block <NUM> = ""No"), the robotic control system <NUM> may return to determination block <NUM>. Optionally, the robotic control system <NUM> may provide feedback to the user (e.g., audio, visual and/or haptic feedback) to indicate that the planned robot motion is not allowed. In response to determining that the planned robot motion is allowed (i.e., determination block <NUM> = "yes"), the robotic control system <NUM> may proceed to block <NUM>.

In block <NUM>, the robotic control system <NUM> may cause the robotic arm <NUM> to move (e.g., via sending control signals to the arm <NUM>) in accordance with the movement planned in block <NUM>. In particular, the robotic control system <NUM> may adjust the position and/or orientation of the end effector <NUM> of the robotic arm <NUM> based on the tracked motion of the handheld device <NUM>. The display screen <NUM> of a display device <NUM>, <NUM> may show a graphical depiction of the end effector <NUM> overlaying patient images as the robotic arm <NUM> is moved.

The method <NUM> may then return to determination block <NUM> to determine whether the robotic control system <NUM> remains in an active control state. If the robotic control system <NUM> determines that it is still in an active control state (i.e., determination block <NUM> = "Yes"), then the robotic control system <NUM> may plan an additional movement of the robotic arm <NUM> based on the tracked movement of the handheld device <NUM> in block <NUM>, determine whether the planned movement is allowed in optional determination block <NUM>, and may control the robotic arm <NUM> to cause the robotic arm <NUM> to make the (allowed) planned movement in block <NUM>. While the robotic arm remains in an active control state (i.e., determination block <NUM> = "Yes"), then the robotic control system <NUM> may repeatedly cycle through the operations of blocks <NUM> through <NUM> to control the robotic arm <NUM> to move the end effector <NUM> based on the detected movement(s) of the instrument <NUM> tracked by the motion tracking system <NUM>.

In response to determining that the robotic arm is no longer in an active control state (i.e., determination block <NUM> = "No"), the robotic control system <NUM> may maintain the robotic arm <NUM> in a standby state in block <NUM>. The robotic arm <NUM> may remain in a standby state until either the robotic control system <NUM> determines that the system <NUM> should exit the follow mode (i.e., determination block <NUM> = "Yes"), or the robotic arm <NUM> again enters an active control state (i.e., determination block <NUM> = "Yes"). The robotic control system <NUM> may determine that the system <NUM> should exit the follow mode (i.e., determination block <NUM> = "Yes") based on a user input event, and may exit the follow mode in block <NUM>.

In examples, the handheld device <NUM> may be tracked using an inertial navigation method as an alternative or in addition to an optical-based tracking method. In the example shown in <FIG>, the handheld device <NUM> includes an inertial measurement unit <NUM> in addition to the marker device <NUM> for an optically-based motion tracking system <NUM>, as described above. In examples, the inertial measurement unit <NUM> may enable redundant motion tracking of the handheld device <NUM>. In particular, the position and/or orientation of the handheld device <NUM> may continue to be tracked when there is a loss of tracking by the optically-based motion tracking system <NUM>, such as when the line of sight between marker device <NUM> and optical sensing device <NUM> is temporarily obscured. A similar inertial measurement unit <NUM> may also be located on the robotic arm <NUM>, such as on or proximate to the end effector <NUM>, to enable inertial motion tracking of the end effector <NUM>.

The inertial measurement unit <NUM> may include a three-axis accelerometer <NUM> and a three-axis gyroscope <NUM>. The accelerometer <NUM> and gyroscope <NUM> may be fabricated utilizing MEMS technology. The accelerometer <NUM> and gyroscope <NUM> may be separate components (e.g., chips) located in the handheld device <NUM> or may be integrated on a single device (e.g., integrated circuit). The handheld device <NUM> may also include circuitry <NUM> coupled to the accelerometer <NUM> and gyroscope <NUM> that may be configured to read output signals from these components <NUM>, <NUM>. The accelerometer <NUM> may output signals measuring the linear acceleration of the handheld device <NUM>, preferably in three-dimensional space. The gyroscope <NUM> may output signals measuring the angular velocity of the handheld device <NUM>, preferably also in three-dimensional space. The signals from the accelerometer <NUM> and gyroscope <NUM> may be processed using a suitable processor, such as a computer <NUM>, to determine the position and orientation of the handheld device <NUM> with respect to an initial inertial reference frame via a dead reckoning technique. In particular, integrating the angular velocity measurements from the gyroscope <NUM> may enable the current orientation of the handheld device <NUM> to be determined with respect to a known starting orientation. Integrating the linear acceleration measurements from the accelerometer <NUM> may enable the current velocity of the handheld device <NUM> to be determined with respect to a known starting velocity. A further integration may enable the current position of the handheld device <NUM> to be determined with respect to a known starting position.

In examples, measurement data from the inertial measurement unit <NUM> may transmitted from the handheld device <NUM> to a separate computing device (e.g., computer <NUM>) via a wired or wireless link. In examples, the data may be transmitted wirelessly using a suitable wireless communication protocol or standard (e.g., an IEEE <NUM>. 15x (BLUETOOTH®) or IEEE <NUM> (WiFi) connection), as described above. The computer <NUM> may perform the inertial navigation calculations to determine the position and orientation of the handheld device <NUM> in three-dimensional space, and preferably within the common, patient-centric coordinate system. The inertial navigation calculations may be initialized with a known initial position, orientation and/or velocity of the handheld device <NUM>, which may be or may be derived from the most recent tracking data from the motion tracking system <NUM>.

In some examples, the robotic control system <NUM> may control the robotic arm <NUM> to provide a pre-determined motion scaling between the movement of the handheld instrument <NUM> detected by the motion tracking system <NUM> and the corresponding movement of the robotic arm <NUM>. As used herein, "motion scaling" refers to the conversion of the tracked movements of the handheld device <NUM> to the movement of a portion of the robotic arm <NUM> (e.g., the end effector <NUM>). The motion scaling may be linear, and may be expressed as a ratio of device <NUM> movement to end effector <NUM> movement (i.e., X:Y, where X is the displacement and/or rotation of the handheld device <NUM> and Y is the corresponding displacement and/or rotation of the end effector <NUM>). In some examples, the relationship between the handheld device <NUM> movement and the end effector <NUM> movement may be non-linear. For example, the ratio X:Y between handheld device <NUM> movement and end effector <NUM> movement may increase as a function of the proximity of the end effector <NUM> to the surgical site. In particular, as the end effector <NUM> is moved closer to the surgical site, the movements of the handheld device <NUM> may result in progressively smaller movements of the end effector <NUM>.

The motion scaling between the instrument <NUM> and the end effector <NUM> may be a fixed parameter. Alternately, the motion scaling may be adjustable by the user. For example, a motion scaling factor applied by the robot control system <NUM> may be adjusted by the user via a user-input event, such as a voice command, a touch event on a display screen, a button push, a mouse/keyboard click, depression of a foot pedal, etc. In one non-limiting example, a first motion scaling factor may be utilized for gross movements of the robotic arm <NUM>, and a second motion scaling factor may be utilized for fine movements of the robotic arm <NUM>. The first motion scaling factor may be useful for moving the robotic arm <NUM> into and out of the surgical area, and the second motion scaling factor may be utilized for making precise adjustments to the position and/or orientation of the end effector <NUM> of the arm <NUM>. In one example, the first motion scaling factor may provide an ~<NUM>:<NUM> ratio of handheld device <NUM> movement to robot end effector <NUM> movement, and the second motion scaling factor may provide a larger ratio (e.g., a <NUM>:<NUM> - <NUM>:<NUM> ratio) of handheld device <NUM> movement to robot end effector <NUM> movement. It will be understood that the robotic control system <NUM> may apply more than two different motion scaling factors to control the movement of the robotic arm <NUM> based on the handheld device <NUM> with varying levels of granularity. Further, in some examples, the motion scaling factor may provide a ratio of handheld device <NUM> movement to end effector <NUM> movement that is less than <NUM>:<NUM>, such that motions of the handheld device <NUM> are amplified in the corresponding motions of the robotic arm <NUM>.

In some examples, while operating in the follow mode, the robotic control system <NUM> may control the movement of the robotic arm <NUM> in response to the detected movement of the handheld device <NUM> so as to limit the speed and/or torque of the robotic arm <NUM>. The robotic control system <NUM> may be also programmed to apply restrictions on the distance and/or speed in which the robotic arm <NUM> may move in response to movements of the handheld device <NUM>. This may provide an important safety function in the case of inadvertent movements of the handheld device <NUM> rapidly or over a long distance. In some examples, the robotic control system <NUM> may also control the movement of the robotic arm <NUM> to smooth out the movements of the handheld device <NUM> and/or to ignore minor tremors or vibrations of the handheld device <NUM> by implementing tremor filtering.

<FIG> illustrates an example of a handheld device <NUM> for controlling the movement of a robotic arm <NUM> in a "follow" mode as described above. The handheld device <NUM> in this example includes a plurality of user input components (i.e., buttons <NUM> and <NUM>). The plurality of user input components <NUM>, <NUM> may be used to control and modify the motion scaling between movements of the handheld device <NUM> and the corresponding movements of the robotic arm <NUM>. For example, while a first button <NUM> is pressed by the user, the robotic control system <NUM> may apply a first motion scaling factor for movement of the robotic arm <NUM> (e.g., to provide gross movements of the arm) and when a second button <NUM> is pressed by the user, the robotic control system <NUM> may apply a second motion scaling factor for movement of the robotic arm <NUM> (e.g., to provide fine movements of the arm). The plurality of user input components <NUM>, <NUM> may be coupled to circuitry <NUM> configured to detect an input event (i.e., button push) at each user input component <NUM>, <NUM> and to transmit signals to the robotic control system <NUM>. In some examples, the circuitry <NUM> may include wireless transceiver circuitry <NUM> configured to transmit the signals wirelessly. Alternately, the handheld device <NUM> may be coupled to the robotic control system <NUM> via a wired link.

<FIG> illustrates a further exampleof a surgical robotic system <NUM> that includes a robotic arm <NUM> having a force sensor <NUM> for detecting a force applied to the robotic arm <NUM> and a handheld device <NUM> having a marker device <NUM> that enables the handheld device <NUM> to be tracked using a motion tracking system <NUM>, as described above. The force sensor <NUM> may be a multi-axis (e.g., six degree of freedom) force and torque sensor that is configured to measure the forces and torques applied to the robotic arm <NUM>. The forces and torques measured by the force sensor <NUM> may be provided to a robotic control system <NUM>, as described above. In examples, the system may operate in a "handguiding" mode in which the robotic control system <NUM> may control the movement of the robotic arm <NUM> in response to the forces and/or torques measured by the force sensor <NUM>. This may enable the user to manually adjust the configuration of the robotic arm, including the position and/or orientation of the end effector <NUM>.

In the examplesof <FIG>, a multi-axis force/torque sensor <NUM> (e.g., transducer) may be mounted within the robotic arm <NUM>. In this example, the force/torque sensor <NUM> is mounted behind the distal-most joint <NUM> of the robotic arm <NUM>, although it will be understood that the sensor <NUM> may be located in another suitable location, such as between the distal-most joint <NUM> of the robotic arm <NUM> and the end effector <NUM>. The sensor <NUM> may be configured to output an electronic signal in response to a force received at the sensor <NUM> (e.g., via a user grasping and applying a force to the distal end of the robotic arm <NUM>). The output signal from the sensor <NUM> may represent the magnitude of the applied force along the x-, y- and z- axes as well as the associated torques about these axes. The output signal from the sensor <NUM> may be provided to the robotic control system via a communication path <NUM>, which may be a wire/cable link extending along the length of the robotic arm <NUM> or a wireless link. The robotic control system <NUM> may plan a movement of the robotic arm <NUM> based on the signal received from the sensor <NUM>. For example, in response to a measured force and/or torque signal, the robotic control system <NUM> may plan and execute a corresponding motion of the robotic arm <NUM> to cause the end effector <NUM> to move in the direction of the applied force/torque. This process may occur repeatedly so that the user may manually move the end effector <NUM> to desired positions and orientations. In some examples, the robotic control system <NUM> may be configured to move the arm in "handguiding" mode only when the applied force/torque measured at the sensor <NUM> exceeds a pre-determined threshold value. In addition, the robotic control system <NUM> may also be configured to compensate for forces due to gravity on the robotic arm <NUM>. The robotic control system <NUM> may also apply a collision model to prevent the robotic arm <NUM> from colliding with the patient or other objects when being moved in handguiding mode.

The system shown in <FIG> may also operate in a "follow" mode in which the robotic control system <NUM> may control the movement of the robotic arm <NUM> in response to the tracked movements of the handheld device <NUM>, as is described above. In examples, the robotic control system <NUM> may alternate between operation in "handguiding" mode and "follow" mode based on an input event from the user (e.g., a voice command, a touch event on a display screen, a button push, a mouse/keyboard click, depression of a foot pedal, etc.). In some examples, while the system is operating in "follow" mode as described above, the robotic control system <NUM> may automatically exit "follow" mode and enter "handguiding" mode in response to a triggering event, such as the force sensor <NUM> measuring a force or torque on the robotic arm <NUM> above a pre-determined threshold. Alternately, while operating in "handguiding" mode, the robotic control system <NUM> may automatically exit "handguiding" mode and enter "follow" mode in response to a triggering event, such as the user bringing a specialized handheld device <NUM> used for moving the robotic arm <NUM> in "follow" mode within the range (e.g., field-of-view) of the motion tracking system <NUM>. Operation in either or both of "handguiding" mode and "follow" mode may require continuous activation by the user, such as holding down a button or footpedal.

In various examples, the "handguiding" mode may be used for gross movements of the robotic arm <NUM>, and the "follow" mode may be used to make precise adjustments to the position and/or orientation of the end effector <NUM>. In both the handguiding mode and the follow mode, the robotic arm <NUM> may be forward driven by the robotic control system <NUM> without requiring any backdriving of the joints.

In certain examples, the handheld device <NUM> may be removably mounted (i.e., docked) to the robotic arm <NUM>, such as within a docking station <NUM> located on the robotic arm <NUM>. The robotic control system <NUM> may be configured to determine whether or not the handheld device <NUM> is mounted within the docking station <NUM> on the robotic arm <NUM>. The robotic control system <NUM> may operate the robotic arm <NUM> in handguiding mode while the handheld device <NUM> is mounted within the docking station <NUM>. When the handheld device <NUM> is removed from the docking station <NUM>, the robotic control system <NUM> may exit handguiding mode and operate the arm in follow mode until the user replaces the handheld device <NUM> within the docking station <NUM>.

In further examples, a force/torque sensor <NUM> as described above may be operatively coupled to a handheld device <NUM> when the handheld device <NUM> is docked in a docking station <NUM> on the robotic arm <NUM>. The force/torque sensor <NUM> may be configured to measure forces and/or torques applied by the user to the handheld device <NUM> docked within the docking station <NUM>, and the robotic control system <NUM> may move the arm in response to these measured forces and/or torques. When the handheld device <NUM> is docked in the docking station <NUM>, it may be used to control the motions of robotic arm in the manner of a joystick or a three-dimensional mouse. When the handheld device <NUM> is removed from the docking station <NUM>, it may be used to control the robotic arm in "follow" mode as described above.

<FIG> illustrate an embodiment of a surgical robotic system <NUM> that includes a robotic arm <NUM> and a cart <NUM> for storing and transporting the robotic arm <NUM> when not in use. As shown in <FIG>, the robotic arm <NUM> may be mounted in a suitable location to enable the robotic arm <NUM> to move throughout the surgical field to assist in the performance of a surgical procedure. In this embodment, the robotic arm <NUM> is mounted to an imaging device <NUM>. In particular, the base of the robotic arm <NUM> is attached to a mounting surface <NUM> that is supported by a support element <NUM> (a curved rail) that extends over the top surface of imaging device <NUM>. The mounting surface <NUM> is a surface of a movable carriage <NUM> that is movable along the length of the support element <NUM>. Alternately, the robotic arm <NUM> may be directly attached to the imaging device <NUM>, such as to the gantry <NUM> or patient support <NUM>, or to another structure in the operating room, such as the wall, ceiling or floor. The robotic arm <NUM> may be attached to a mounting surface <NUM> using bolts or similar mechanical fasteners that may enable the robotic arm <NUM> to be removed when not in use. The robotic arm <NUM> may be removed from the mounting surface <NUM> and stored on or within a cart <NUM> when it is not in use. The cart <NUM> may be a mobile cart to facilitate transport of the robotic arm <NUM>. The cart <NUM> for the robotic arm <NUM> may be a cart <NUM> as described above with reference to <FIG> that may also include, for example, a monitor display, user input(s)/control(s) and system electrical components (e.g., a computer). Alternately, the cart <NUM> for storage/transport of the robotic arm <NUM> may be separate from a cart <NUM> as described with reference to <FIG>.

In some cases, it may be difficult and time-consuming to safely transfer the robotic arm <NUM> between the mounting surface <NUM> to which the robotic arm <NUM> is attached during use and a cart <NUM> used for storage and transport of the robotic arm <NUM>. In the embodiment of <FIG>, the cart <NUM> includes a marker device <NUM> that enables the cart <NUM> to be tracked by a motion tracking system <NUM>, as described above. The robotic control system <NUM> may be configured to control the robotic arm <NUM> to cause the robotic arm <NUM> to move to a transfer position based on the tracked position of the cart <NUM>. The transfer position of the robotic arm <NUM> may be a position that facilitates the transfer of the robotic arm <NUM> from the mounting surface <NUM> to the cart <NUM>.

In <FIG>, the robotic arm <NUM> is shown attached to the mounting surface <NUM> above the gantry <NUM> of the imaging system <NUM> and the cart <NUM> is located a distance away from the imaging system <NUM>, outside of the range (field-of-view) of the motion tracking system <NUM>. The motion tracking system <NUM> may track the robotic arm <NUM> in three-dimensional space via a plurality of markers <NUM> located on the robotic arm <NUM>. A robotic control system <NUM> may control the robotic arm <NUM> to move the arm to a desired pose within the three-dimensional space.

In <FIG>, the cart <NUM> is moved adjacent to the imaging system <NUM> such that the marker device <NUM> is within the range (field-of-view) of the motion tracking system <NUM>. The motion tracking system <NUM> may track the location of the cart <NUM> in the same coordinate system as the robotic arm <NUM>. The cart <NUM> may be pre-calibrated so that the spatial relationship between the marker device <NUM> and a target location <NUM> on or adjacent to the cart <NUM> is known within the common coordinate system. The target location <NUM> may be a location to which the robotic arm <NUM> may be moved to in order to facilitate transfer of the robotic arm <NUM> to the cart <NUM>. In the embodiment of <FIG>, the target location <NUM> is an entrance to a housing <NUM> in the cart <NUM> that is configured to receive and house the robotic arm <NUM>. In other embodiments, the target location <NUM> may be associated with a cradle or other mechanism for securely holding the robotic arm <NUM> for storage and/or transport.

In embodiments, as the cart approaches the robotic arm <NUM>, the motion tracking system <NUM> may track the position of the cart <NUM> and may optionally provide user feedback (e.g., an audio alert, a visual indicator on the robotic arm <NUM> and/or a display screen) when the target location <NUM> is at a location that is suitable for transferring the robotic arm <NUM> to the cart <NUM>. To transfer the robotic arm <NUM> to the cart <NUM>, the robotic control system <NUM> may control the robotic arm <NUM> to move the distal end <NUM> of the arm <NUM> to the target location <NUM>, as shown in <FIG>. This may occur in response to a user input event. The user may then disconnect the robotic arm <NUM> from the mounting surface <NUM> and the arm <NUM> may be lowered into the housing <NUM> as shown in <FIG>.

Alternately, the robotic control system <NUM> may control the robotic arm <NUM> to move the arm <NUM> partially or completely into a holding mechanism on or within the cart <NUM>. This is illustrated in <FIG>, which shows the robotic arm <NUM> moved partially into the housing <NUM> of the cart <NUM>. In this embodiment, the carriage <NUM> to which the robotic arm <NUM> is attached is lowered on the support element <NUM> (i.e., curved rail) to a position on a side of gantry <NUM>. From this position, the robotic arm <NUM> may reach at least partially inside the housing <NUM> of the cart. This process may be fully automated, such that the carriage <NUM> is motorized and configured move to a pre-set loading/unloading position on the support element <NUM>. A quick-connect/disconnect mechanism may be used for mechanically and electrically coupling and decoupling the robotic arm <NUM> from the mounting surface <NUM> on the carriage <NUM>.

The cart <NUM> may optionally include a mechanism (e.g., a platform <NUM> that raises and lowers within the housing <NUM>) that is configured to at least partially lift the robotic arm <NUM> from the housing <NUM> to enable the robotic arm <NUM> to more easily dock to the carriage <NUM>. Once the robotic arm <NUM> is mechanically and electrically connected to the mounting surface <NUM> on the carriage <NUM>, the robotic control system <NUM> may control the robotic arm <NUM> to cause the entire arm <NUM> to move out of the housing <NUM> in the cart <NUM>. The carriage <NUM> may optionally move on the support element <NUM> to position the robotic arm <NUM> in a suitable location for performing robotically-assisted image-guided surgery.

<FIG> is a system block diagram of a computing device <NUM> useful for performing and implementing the various embodiments described above. The computing device <NUM> may perform the functions of an image guided surgery system <NUM> and/or a robotic control system <NUM>, for example. While the computing device <NUM> is illustrated as a laptop computer, a computing device providing the functional capabilities of the computer device <NUM> may be implemented as a workstation computer, an embedded computer, a desktop computer, a server computer or a handheld computer (e.g., tablet, a smartphone, etc.). A typical computing device <NUM> may include a processor <NUM> coupled to an electronic display <NUM>, a speaker <NUM> and a memory <NUM>, which may be a volatile memory as well as a nonvolatile memory (e.g., a disk drive). When implemented as a laptop computer or desktop computer, the computing device <NUM> may also include a floppy disc drive, compact disc (CD) or DVD disc drive coupled to the processor <NUM>. The computing device <NUM> may include an antenna <NUM>, a multimedia receiver <NUM>, a transceiver <NUM> and/or communications circuitry coupled to the processor <NUM> for sending and receiving electromagnetic radiation, connecting to a wireless data link, and receiving data. Additionally, the computing device <NUM> may include network access ports <NUM> coupled to the processor <NUM> for establishing data connections with a network (e.g., LAN coupled to a service provider network, etc.). A laptop computer or desktop computer <NUM> typically also includes a keyboard <NUM> and a mouse pad <NUM> for receiving user inputs.

The foregoing method descriptions are provided merely as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented. Words such as "thereafter," "then," "next," etc. are not necessarily intended to limit the order of the steps; these words may be used to guide the reader through the description of the methods.

In one or more exemplary aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on as one or more instructions or code on a non-transitory computer-readable medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module executed which may reside on a non-transitory computer-readable medium. Non-transitory computer-readable media includes computer storage media that facilitates transfer of a computer program from one place to another. By way of example, and not limitation, such non-transitory computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to carry or store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of non-transitory computer-readable storage media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a machine readable medium and/or computer-readable medium, which may be incorporated into a computer program product.

Claim 1:
A method for transferring a robotic arm (<NUM>) from a mounting surface (<NUM>) to which the robotic arm (<NUM>) is attached during use to a mobile cart (<NUM>) for storage and/or transport of the robotic arm (<NUM>), the method comprising:
tracking a location of the mobile cart (<NUM>) relative to the robotic arm (<NUM>) using a motion tracking system (<NUM>); and
controlling the robotic arm (<NUM>) to move the robotic arm (<NUM>) into a pose that facilitates transferring the robotic arm (<NUM>) from the mounting surface (<NUM>) to the mobile cart (<NUM>) based on the tracked location of the mobile cart (<NUM>);
wherein the mounting surface (<NUM>) is located on a carriage (<NUM>) that is moveable along a support element (<NUM>), the method further comprising:
moving the carriage (<NUM>) and the robotic arm (<NUM>) along the support element (<NUM>) to a pre-determined loading/unloading position.