Patent Description:
Surgical navigation systems are used to track the position of one or more objects during surgery. Surgical robots are useful for holding one or more tools or devices during a surgery, and may operate autonomously (e.g., without any human input during operation), semi-autonomously (e.g., with some human input during operation), or non-autonomously (e.g., only as directed by human input).

From each of <CIT>, <CIT>, <CIT>, and <CIT> a robotic navigation system is known, comprising: a robot base, a robotic arm with a proximal portion secured to the robot base, a distal portion movable relative to the proximal portion, and a tracking marker secured to the robotic arm proximate the distal portion, at least one processor, a navigation system including a tracking marker sensor configured to identify positions of the tracking marker in a first coordinate space; and a memory storing instructions for execution by the at least one processor.

Further robotic navigation systems are known from e.g. <CIT> and <CIT>.

The present invention provides a robotic surgery navigation system with the features according to claim <NUM> and a method of utilizing a time-spaced robotic reference frame with the features according claim <NUM>. Further preferred embodiments of the system and the method are described in the respective dependent claims.

It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and/or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different embodiments of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and/or a medical device.

In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

Navigated robotic procedures often involve reference frames and trackers whose positions are detected by a tracking marker sensor. For example, a navigation system could use a camera as a tracking marker sensor, which can detect optical tracking markers on a reference frame attached to a robotic arm. With this information, the coordinate system of the robotic system can be correlated to the coordinate system of the navigation system. In other circumstances, the information from the tracking marker sensor, which can accurately determine the position and orientation of the robotic arm, can be used to calibrate the robotic system.

Larger reference frames provide better accuracy for navigated systems. However, large reference frames obstruct the operation field, are cumbersome, and are less stable. As disclosed herein, a robotic arm could create a time-spaced reference frame by moving to several different poses, pausing at each pose for a navigation camera or other tracking marker sensor to capture the pose of the robotic arm (e.g., the position of a tracking marker secured to the robotic arm). The points are then stored and combined, and after several movements (e.g., four), the stored points create a "time-spaced" frame-as if the navigation system saw a giant reference frame with reflective spheres or LEDs at those stored points.

In embodiments of the present disclosure, a robotic arm can quickly and accurately move to specified poses within robotic space. To translate these poses in robotic space into points in navigation space, a small reference frame could be attached to the robotic arm. However, for greater accuracy, the robotic arm could trace a "giant" reference frame by moving to multiple (e.g., four) specific locations (reference points), pausing at each location for the navigation camera to "capture" that reference point in navigation space. For example, the end of the robotic arm could have an LED (including, for example, an infrared emitting device), reflective sphere, optical pattern (e.g., a QR code, a bar code), a shape (e.g., a triangle, circle, square) (which may be printed or painted on the arm, applied as a sticker, or provided in any other manner), a color, an emitter of a particular wavelength, a reflector of a particular wavelength, an emitter that pulses at a particular frequency, or any other tracking marker for the camera or other tracking marker sensor to see and track. As the robot moves the robotic arm through the four poses in robotic space, the navigation system detects the tracking marker and thus identifies four corresponding reference points in navigation space, such that those four reference points effectively serve as a giant reference frame linking the robotic and navigations spaces.

Other embodiments may use multiple LEDs, reflective spheres, or other tracking markers on the robotic arm (so that the camera or other tracking marker sensor can track the robotic arm from multiple angles). In addition, the robotic arm could continuously move to and from a central point to each of the designated reference points, with the navigation camera continually tracking the tracking marker and thus the robotic arm, and detecting changes in direction (e.g., a <NUM>-degree change from the arm moving to the reference point vs away from the reference point) to decrease the required time needed to form the time-spaced robotic reference frame.

<CIT>, describes a dynamic snapshot tracking device that may be moved to a plurality of positions within a navigation volume and tracked at each of the plurality of positions. The plurality of positions may be used in concert to define a virtual snapshot tracking device that is larger than a single physical snapshot tracking device, and thus to achieve greater accuracy and less deviation than might otherwise be possible with a single physical snapshot tracking device.

A time-spaced robotic reference frame as described herein (and created using a robot and a tracking marker sensor) may be particularly useful when a position of the robot and is fixed relative to a position of the tracking marker sensor and vice versa. Where the position of the tracking marker sensor relative to the robot is not fixed, a physical reference frame may be used in conjunction with a time-spaced robotic reference frame as described herein. Alternatively, a time-spaced robotic reference frame can be generated in such a way that relative movement (or a chance of relative movement) between the tracking marker sensor and the robot is minimized, such as by capturing the various points of the time-spaced robotic reference frame over a short period of time.

Turning first to <FIG>, a block diagram of a system <NUM> according to at least one embodiment of the present disclosure is shown. The system <NUM> may be used, for example: to carry out one or more aspects of one or more of the methods disclosed herein; for navigation purposes; for registration purposes; for calibration operations; to verify operational integrity of a navigation system (such as the navigation system <NUM>) or of a robot (such as a robot <NUM>); or for any other useful purpose. The system <NUM> comprises a computing device <NUM>, a tracking marker sensor <NUM>, a robot <NUM>, a navigation system <NUM>, a database <NUM>, and a cloud <NUM>. Notwithstanding the foregoing, systems according to other embodiments of the present disclosure may omit any one or more of the computing device <NUM>, the tracking marker sensor <NUM>, the robot <NUM>, the navigation system <NUM>, the database <NUM>, and/or the cloud <NUM>. Additionally, systems according to other embodiments of the present disclosure may arrange one or more components of the system <NUM> differently (e.g., one or more of the tracking marker sensor <NUM>, the robot <NUM>, and the navigation system <NUM> may comprise the components shown in <FIG> as being part of the computing device <NUM>).

The computing device <NUM> comprises at least one processor <NUM>, at least one communication interface <NUM>, at least one user interface <NUM>, and at least one memory <NUM>. A computing device according to other embodiments of the present disclosure may omit one or both of the communication interface(s) <NUM> and the user interface(s) <NUM>.

The at least one processor <NUM> of the computing device <NUM> may be any processor identified or described herein or any similar processor. The at least one processor <NUM> may be configured to execute instructions stored in the at least one memory <NUM>, which instructions may cause the at least one processor <NUM> to carry out one or more computing steps utilizing or based on data received, for example, from the tracking marker sensor <NUM>, the robot <NUM>, the navigation system <NUM>, the database <NUM>, and/or the cloud <NUM>.

The computing device <NUM> may also comprise at least one communication interface <NUM>. The at least one communication interface <NUM> may be used for receiving image data or other information from an external source (such as the tracking marker sensor <NUM>, the robot <NUM>, the navigation system <NUM>, the database <NUM>, the cloud <NUM>, and/or a portable storage medium (e.g., a USB drive, a DVD, a CD)), and/or for transmitting instructions, images, or other information from the at least one processor <NUM> and/or the computing device <NUM> more generally to an external system or device (e.g., another computing device <NUM>, the tracking marker sensor <NUM>, the robot <NUM>, the navigation system <NUM>, the database <NUM>, the cloud <NUM>, and/or a portable storage medium (e.g., a USB drive, a DVD, a CD)). The at least one communication interface <NUM> may comprise one or more wired interfaces (e.g., a USB port, an ethernet port, a Firewire port) and/or one or more wireless interfaces (configured, for example, to transmit information via one or more wireless communication protocols such as <NUM>. 11a/b/g/n, Bluetooth, Bluetooth low energy, NFC, ZigBee, and so forth). In some embodiments, the at least one communication interface <NUM> may be useful for enabling the device <NUM> to communicate with one or more other processors <NUM> or computing devices <NUM>, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

The at least one user interface <NUM> may be or comprise a keyboard, mouse, trackball, monitor, television, touchscreen, button, joystick, switch, lever, and/or any other device for receiving information from a user and/or for providing information to a user of the computing device <NUM>. The at least one user interface <NUM> may be used, for example, to receive a user selection or other user input in connection with any step of any method described herein; to receive a user selection or other user input regarding one or more configurable settings of the computing device <NUM> and/or of another component of the system <NUM>; to receive a user selection or other user input regarding how and/or where to store and/or transfer data received, modified, and/or generated by the computing device <NUM>; and/or to display information (e.g., text, images) and/or play a sound to a user based on data received, modified, and/or generated by the computing device <NUM>. Notwithstanding the inclusion of the at least one user interface <NUM> in the system <NUM>, the system <NUM> may automatically (e.g., without any input via the at least one user interface <NUM> or otherwise) carry out one or more, or all, of the steps of any method described herein.

Although the at least one user interface <NUM> is shown as part of the computing device <NUM>, in some embodiments, the computing device <NUM> may utilize a user interface <NUM> that is housed separately from one or more remaining components of the computing device <NUM>. In some embodiments, the user interface <NUM> may be located proximate one or more other components of the computing device <NUM>, while in other embodiments, the user interface <NUM> may be located remotely from one or more other components of the computer device <NUM>.

The at least one memory <NUM> may be or comprise RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible non-transitory memory for storing computer-readable data and/or instructions. The at least one memory <NUM> may store information or data useful for completing, for example, any step of the methods <NUM> or <NUM> described herein. The at least one memory <NUM> may store, for example, information about one or more predetermined coordinate systems <NUM> (e.g., information about a robotic coordinate system or space, information about a navigation coordinate system or space, information about a patient coordinate system or space); instructions <NUM> for execution by the at least one processor <NUM>, for example to cause the at least one processor <NUM> to carry out one or more of the steps of the method <NUM> and/or of the method <NUM>; and/or one or more algorithms <NUM> for use by the processor in carrying out any calculations necessary to complete one or more of the steps of the method <NUM>, the method <NUM>, and/or of the method <NUM>, or for any other calculations. Such predetermined coordinate system(s) <NUM>, instructions <NUM>, and/or algorithms <NUM> may, in some embodiments, be organized into one or more applications, modules, packages, layers, or engines, and may cause the at least one processor <NUM> to manipulate data stored in the at least one memory <NUM> and/or received from or via another component of the system <NUM>.

The tracking marker sensor <NUM> is operable to detect one or more tracking markers <NUM> (described below). The tracking marker sensor <NUM> may be, for example, an optical camera; an infrared camera; a 3D camera system; a stereoscopic vision system; another imaging device; an electromagnetic system; or any other sensor that can detect one or more tracking markers <NUM>. The tracking marker sensor <NUM> may comprise a dedicated processor for executing instructions stored in a dedicated memory of the tracking marker sensor <NUM>, or the tracking marker sensor <NUM> may simply be configured to transmit data collected therewith to the computing device <NUM> or to another component of the system <NUM>. Although shown in <FIG> as being in communication only with the computing device <NUM>, in some embodiments, the tracking marker sensor <NUM> may be in communication with any one or more of the computing device <NUM>, the robot <NUM>, the navigation system <NUM>, the database <NUM>, and/or the cloud <NUM>. Also, in some embodiments, the computing device <NUM> may comprise the tracking marker sensor <NUM>, while in other embodiments, the navigation system <NUM> may comprise the tracking marker sensor <NUM>. In still other embodiments, the robot <NUM> may comprise the tracking marker sensor <NUM>.

The tracking marker sensor <NUM> may be positioned directly above an operating table or portion thereof, or above and to one side of an operating table or portion thereof, or in another convenient position within an operating room or other room housing the robot <NUM>. The tracking marker sensor <NUM> may be positioned at a location selected to provide the tracking marker sensor <NUM> with a clear and/or unobstructed view of the robotic arm <NUM> of the robot <NUM> (and thus of one or more tracking markers <NUM> fixedly secured to the robotic arm <NUM>) during operation thereof. In some embodiments, the tracking marker sensor <NUM> is fixed, while in other embodiments, the tracking marker sensor <NUM> may be precisely movable (whether manually or automatically) in one or more directions.

In still other embodiments, a position and/or field of view of the tracking marker sensor <NUM> may be used to determine where a robotic arm <NUM> may be moved to generate a time-spaced robotic reference frame as disclosed herein. For example, a surgeon or other medical staff could place the tracking marker sensor <NUM> in a location that seems to work well for the operating room or other space in which the tracking marker sensor <NUM> is being used. The orientation and/or field of view of the tracking marker sensor <NUM> may then be determined (e.g., by taking some shots with the tracking marker sensor <NUM> and recognizing/tracking objects in the field of view, or otherwise), and used to define an area in which the robot <NUM> described below could move the robotic arm <NUM> to create a time-spaced robotic reference frame as described elsewhere herein. Based on constraints in the defined area, the shape and size of the time-spaced robotic reference frame may also be defined.

The tracking marker sensor <NUM> may be configured to capture data regarding one or more sensed tracking markers <NUM> at a plurality of moments in time. For example, where the tracking marker sensor <NUM> is a camera, the tracking marker sensor <NUM> may be configured to capture a sequence of still images comprising one or more tracking markers <NUM>. The tracking marker sensor <NUM> may be configured to capture such data at periodic intervals, or when commanded by a user (e.g., via a user interface <NUM>), or upon a signal (generated either autonomously or in response to user input) from the computing device <NUM>, the robot <NUM>, and/or the navigation system <NUM>.

The tracking marker sensor <NUM> may additionally or alternatively be operable to capture data corresponding to one or more tracking markers <NUM> continuously, in real-time. In such embodiments, the tracking marker sensor <NUM> may provide a stream of real-time sensor data to the computing device <NUM>, which may continuously process the sensor data to detect a changing position of one or more tracking markers <NUM> therein. In some embodiments, the tracking marker sensor <NUM> may be a video camera configured to capture a series of video frames. The processor <NUM> of the computing device <NUM> (or any other processor) may then be used to recognize or otherwise detect the tracking markers <NUM> in each video frame, and/or to compare movement in the different video frames to identify or extrapolate a movement trajectory of the tracking marker <NUM>. In some embodiments, the tracking marker sensor <NUM> may comprise more than one tracking marker sensor <NUM>.

One benefit of continuously or periodically tracking the position of a robotic arm such as the robotic arm <NUM>, and more specifically of a tracking marker sensor <NUM> positioned on such a robotic arm, whether by capturing still images or video of the robotic arm <NUM> and/or of a tracking marker sensor <NUM> positioned thereon, is that a registration of robotic space to navigation space may be continuously or periodically confirmed. This advantageously enhances patient safety by reducing the likelihood that any surgical step will be performed without proper alignment between the robotic space and the navigation space, either or both of which may also be aligned to the patient space. Where a robot is capable of providing independent position information about the robotic arm to a processor <NUM>, the processor <NUM> can compare the independent position information from the robot to position information obtained using the tracking marker sensor <NUM> to provide a redundant safety layer. Moreover, the generation of time-spaced robotic reference frames as disclosed herein may be accomplished while the robot <NUM> (including the robotic arm <NUM>) performs its planned surgical movement, so that no additional movements or time are needed to confirm the registration during the surgery. Rather, the tracking marker sensor <NUM> may be used to capture still images or video (or to otherwise monitor the robotic arm <NUM> and/or the tracking marker sensor <NUM> thereon) during the surgery, as the robot <NUM> moves the robotic arm <NUM> to appropriate poses for carrying out each step of a surgical procedure. Taken together, the respective positions of the tracking marker <NUM> at each of the plurality of poses form a time-spaced robotic reference frame, as further described elsewhere herein. Moreover, in some embodiments, a surgical plan detailing the planned movements of the robot <NUM> (including the robotic arm <NUM> thereof) may be analyzed to identify when the robot <NUM> will be in the most effective poses for capturing a position of a tracking marker <NUM> for use in generating a time-spaced robotic reference frame as disclosed herein.

With reference still to <FIG>, and also to <FIG>, the robot <NUM> may be any surgical robot or surgical robotic system. The robot <NUM> may be or comprise, for example, the Mazor X™ Stealth Edition robotic guidance system. The robot <NUM> may comprise a base <NUM> that supports a robotic arm <NUM>. The robot <NUM> may comprise one or more robotic arms <NUM>. In some embodiments, such as that illustrated in <FIG>, the robotic arm <NUM> may comprise a first robotic arm 144A and a second robotic arm 144B. In other embodiments, the robot <NUM> may comprise more than two robotic arms <NUM>. The robotic arm <NUM> may, in some embodiments, assist with a surgical procedure (e.g., by holding a tool in a desired trajectory or pose and/or supporting the weight of a tool while a surgeon or other user operates the tool, or otherwise) and/or automatically carry out a surgical procedure.

In embodiments of the present disclosure comprising a robot <NUM> with a plurality of robotic arms <NUM> (or a plurality of robots <NUM>, each with one or more robotic arms <NUM>), the robotic arms <NUM> may be used in combination to generate a time-spaced robotic reference frame. For example, each robotic arm <NUM> may assume and/or be moved to a unique pose, and a position of a tracking marker <NUM> on each robotic arm <NUM> may be detected and/or recorded. The resulting positions may then be used to generate a time-spaced robotic reference frame as described elsewhere herein. Alternatively, in some embodiments, one of a plurality of robotic arms may be used to generate a time-spaced robotic reference if or while another of the plurality of robotic arms (or a tracking marker <NUM> secured thereto) cannot be seen by the tracking marker sensor <NUM>.

In some embodiments, a plurality of robotic arms <NUM> may share a common coordinate system or space, while in other embodiments, one or more robotic arms <NUM> may have a coordinate system or space that is different than a coordinate system or space used by another one or more robotic arms <NUM>. In embodiments where the plurality of robotic arms <NUM> use a plurality of coordinate systems, at least one of the plurality of robotic arms <NUM> corresponding to each of the plurality of coordinate systems may be used to generate a time-spaced robotic reference frame that may then be used to map each coordinate system to a navigation coordinate system. Also in some embodiments, a distal end or end effector of one robotic arm <NUM> may be used to contact a designated location (e.g. a specific location on a patient) and another robotic arm <NUM> may be used to form a time-spaced robotic reference frame, thus allowing the reference frame to be used to correlate patient space, robotic space, and navigation space.

Referring still to <FIG> and <FIG>, the robotic arm <NUM> may have three, four, five, six, or more degrees of freedom. The robotic arm <NUM> may comprise one or more segments <NUM>. Each segment <NUM> may comprise a member <NUM> and a joint <NUM> to which the member <NUM> is attached and/or from which the member <NUM> extends. The joint <NUM> may be secured, for example, to the base <NUM> or to the member <NUM> of another segment <NUM>. The joint <NUM> may be any type of joint that enables selective movement of the member <NUM> relative to the structure to which the joint <NUM> is attached. For example, the joint <NUM> may be a pivot j oint, a hinge j oint, a saddle j oint, or a ball- and-socket joint. The joint <NUM> may allow movement of the member <NUM> in one dimension or in multiple dimensions, and/or along one axis or along multiple axes.

In embodiments of the robot <NUM> comprising a robotic arm <NUM> with only one segment <NUM>, the joint <NUM> of the segment <NUM> may be secured to the base <NUM>, and the member <NUM> of the segment <NUM> may comprise a proximal end secured to the joint <NUM> and a distal end supporting an end effector. The end effector may be, for example, a tool (e.g., a drill, saw, screwdriver, imaging device) or a tool guide (e.g., for guiding a biopsy needle, ablation probe, or other tool along a desired trajectory).

In embodiments of the robot <NUM> comprising a robotic arm <NUM> with a plurality of segments <NUM>, such as that illustrated in <FIG>, a first segment <NUM> may comprise a joint <NUM> secured to the base <NUM>, and the member <NUM> of the first segment <NUM> may comprise a proximal end secured to the joint <NUM> and a distal end supporting a joint of a second segment <NUM>. The member <NUM> of the second segment <NUM> may comprise a proximal end secured to the joint <NUM> of the second segment <NUM>, and a distal end supporting a joint <NUM> of a third segment <NUM>, and so on. The member <NUM> of the final segment <NUM> may comprise a distal end that supports an end effector <NUM>, which may be the same as or similar to the end effector described above. In such embodiments, the joints <NUM> of the various segments <NUM> may or may not be of the same type, and the members <NUM> of the various segments <NUM> may or may not be identical. In some embodiments, one or more of the members <NUM> may be configured to telescope or otherwise extend and retract for selective adjustment of a length thereof.

All or some of the joints <NUM> of the segments <NUM> of the robotic arm <NUM> may be powered (so as to be selectively controllable without physical manipulation by a human). Any one or more of electric, pneumatic, hydraulic, and/or other means may be used to selectively control movement of a member <NUM> about the joint <NUM>. For example, each segment <NUM> may comprise a servo for selectively moving the member <NUM> of that segment <NUM> relative to the joint <NUM> of that segment <NUM>.

The robotic arm <NUM> also comprises one or more sensors <NUM>. Each sensor <NUM> may be positioned to detect a position of a member <NUM> of a given segment <NUM> relative to the joint <NUM> of the segment <NUM>. For example, where the joint <NUM> of a given segment <NUM> is or comprises a hinge joint, a sensor <NUM> may detect an angular position of the member <NUM> relative to an axis of the hinge joint. Where the joint <NUM> of a given segment <NUM> is or comprises a rotary joint (e.g., configured to allow rotation of the member <NUM> about an axis that extends through the member <NUM> and the joint <NUM>), the sensor <NUM> may detect an angular position of the member <NUM> relative to the axis that extends through the member <NUM> and the joint <NUM>. Each sensor <NUM> may be, for example, a rotary encoder, a linear encoder, or an incremental encoder.

Data from the sensors <NUM> may be provided to a processor of the robot <NUM>, to the processor <NUM> of the computing device <NUM>, and/or to the navigation system <NUM>. The data may be used to calculate a position in space of the robotic arm <NUM> relative to a predetermined coordinate system. For example, the robot <NUM> may calculate a position in space of the robotic arm <NUM> relative to a coordinate system with an origin at the position where the joint <NUM> of the first segment <NUM> of the robotic arm <NUM> is secured to the base <NUM>. The calculation may be based not just on data received from the sensor(s) <NUM>, but also on data or information (such as, for example, physical dimensions) corresponding to each segment <NUM> and/or corresponding to an end effector secured to the final segment <NUM>. By way of example only, a known location of the proximal end of the robotic arm <NUM> (e.g., where a joint <NUM> of the first segment <NUM> is secured to the base <NUM>), known dimensions of each segment <NUM>, and data from the sensor(s) <NUM> about an orientation of the member <NUM> of each segment <NUM> relative to the joint <NUM> of each segment <NUM> may be used to calculate the path of the robotic arm through space.

Referring still to Figs. 1A-1B, at least one tracking marker <NUM> is fixedly secured to or positioned on the robotic arm <NUM>. The tracking marker <NUM> may be positioned, for example, proximate a distal end of the robotic arm <NUM>, whether on an end effector <NUM> thereof or not. As used herein, "fixedly secured" does not mean "permanently secured," and indeed the tracking marker <NUM> may be detachable from the robotic arm <NUM>. The tracking marker <NUM> may be an infrared emitting diode (IRED) or any other kind of light-emitting diode (LED). In some embodiments, the tracking marker <NUM> may be a reflective sphere, a geometric or optical pattern (such as, for example, a QR code or bar code). The tracking marker <NUM> may further be a shape (e.g., a triangle, circle, square) or a color, either of which may be printed or painted on the arm, applied as a sticker, or provided in any other manner. The tracking marker <NUM> may further be an emitter of a particular wavelength, a reflector of a particular wavelength, an emitter that pulses at a particular frequency, or another item or feature that may be readily distinguished by the tracking marker sensor <NUM>. The tracking marker <NUM> may be configured to be detectable by a tracking marker sensor <NUM> even when covered by a drape or other covering that may be arranged on or over the robotic arm <NUM> to maintain a sterile operating room environment.

Where more than one tracking marker <NUM> is used (e.g., when a plurality of tracking markers <NUM> are secured to a single robotic arm <NUM>, and/or when one or more tracking markers <NUM> are secured to each of a plurality of robotic arms <NUM>), the tracking markers <NUM> may all be identical, or one or more of the tracking markers <NUM> may be different than another one or more of the tracking markers <NUM>. In some embodiments, one or more of the tracking markers <NUM> may be configured to emit light at a first wavelength, and another one or more of the tracking markers <NUM> may be configured to emit light at a second wavelength different than the first wavelength. Also in some embodiments, one or more of the tracking markers <NUM> may be configured to reflect light at a first wavelength, while another one or more of the tracking markers may be configured to reflect light at a second wavelength that is different than the first wavelength. The emitted and/or reflected wavelengths of light of the embodiments described above may be wavelengths within a particular spectrum (e.g., wavelengths corresponding to red light versus wavelengths corresponding to blue light in the visible spectrum, or different wavelengths in the infrared spectrum) as well as wavelengths from different spectrums (e.g., a wavelength in the visible spectrum versus a wavelength in the infrared spectrum). Additionally or alternatively, one or more of the tracking markers <NUM> may be or comprise an emitter that pulses at a first frequency, and another one or more of the tracking markers <NUM> may be or comprise an emitter that pulses at a second frequency different than the first frequency.

In some embodiments, a plurality of tracking markers <NUM> may fixedly secured to, or positioned on, a segment <NUM> of the robotic arm <NUM> that comprises a distal end of the robotic arm <NUM> (but for, e.g., an end effector <NUM> that may be attached thereto). The plurality of tracking markers <NUM> may be arranged so that at least one tracking marker <NUM> is visible (e.g., to the tracking sensor <NUM>) from any one of a plurality of possible orientations of the segment <NUM> and/or the arm <NUM>. For example, in some embodiments, the tracking markers <NUM> may be circumferentially spaced about the robotic arm <NUM> proximate the distal end thereof.

In some embodiments of the present disclosure, a tracking marker <NUM> may be moveably secured to the robotic arm <NUM>, and may further be selectively moveable relative to the robotic arm <NUM>. In such embodiments, the tracking marker <NUM> may be configured to move (or to be moved automatically) from a first position on the robotic arm <NUM> (e.g., proximate a distal end thereof) to a second position on the robotic arm <NUM> (e.g., proximate a proximal end thereof) when the robotic arm <NUM> moves into or out of a certain position or set of positions. The purpose of such movement of the tracking marker <NUM> may be to facilitate maintenance of a line of sight between the tracking marker <NUM> and the tracking marker sensor <NUM>, or to enable a plurality of positions of the tracking marker <NUM> to be recorded by the tracking marker sensor <NUM> without moving the robotic arm <NUM> itself. This may be advantageous, for example, to increase an effective size of a time-spaced robotic reference frame as described elsewhere herein. In such embodiments, the robot <NUM> (and/or another component of the system <NUM>) may be configured to track whether the tracking marker <NUM> is in its respective first position or second position (or other position), and to provide such information to the navigation system <NUM> (or to any other component of the system <NUM>) to enable correlation of a robotic coordinate system with a navigation coordinate system based on a position of the tracking marker <NUM> relative to the robotic arm <NUM> as known by the robot <NUM> (and/or another component of the system <NUM>), and further based on a position of the tracking marker <NUM> as detected by the navigation system <NUM> (e.g., using a tracking sensor <NUM>). The robot <NUM> and/or another component of the system <NUM> may "know" the positions of the tracking marker <NUM> relative to the robotic arm <NUM> as a result of a calibration process, or simply by having stored in a memory thereof information corresponding to the positions of the tracking marker <NUM> relative to the robotic arm <NUM>.

Referring again to <FIG>, the navigation system <NUM> may provide navigation for a surgeon and/or for the robot <NUM> during an operation. The navigation system <NUM> may be any now-known or future-developed navigation system, including, for example, the Medtronic StealthStation™ S8 surgical navigation system. The navigation system <NUM> may include a camera or other sensor(s) for detecting and/or tracking one or more reference markers, navigated trackers, or other objects within an operating room or other room where a surgical procedure takes place. In some embodiments, the navigation system <NUM> may comprise the tracking marker sensor <NUM>. In various embodiments, the navigation system <NUM> may be used to track a position of the robotic arm <NUM> (or, more particularly, of a tracking marker <NUM> attached to the robotic arm <NUM>). The navigation system <NUM> may be used to track a position of one or more reference markers or arrays or other structures useful for detection by a camera or other sensor of the navigation system <NUM>. The navigation system <NUM> may include a display for displaying one or more images from an external source (e.g., the computing device <NUM>, tracking marker sensor <NUM>, or other source) or a video stream from the camera or other sensor of the navigation system <NUM>. In some embodiments, the system <NUM> may operate without the use of the navigation system <NUM>.

The database <NUM> may store information that correlates the position and orientation, or pose, of the robotic arm <NUM> to a position of the tracking marker <NUM>. The database <NUM> may also store information about each of a plurality of detected positions of a tracking marker <NUM>. The database <NUM> may additionally or alternatively store, for example, information about or corresponding to one or more characteristics of the tracking marker <NUM>; one or more surgical plans for use by the robot <NUM>, the navigation system <NUM>, and/or a user of the computing device <NUM> or of the system <NUM>; one or more images useful in connection with a surgery to be completed by or with the assistance of one or more other components of the system <NUM>; and/or any other useful information. The database <NUM> may be configured to provide any such information to the computing device <NUM> or to any other device of the system <NUM> or external to the system <NUM>, whether directly or via the cloud <NUM>. In some embodiments, the database <NUM> may be or comprise part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and/or another system for collecting, storing, managing, and/or transmitting electronic medical records including image data.

The cloud <NUM> may be or represent the Internet or any other wide area network. The computing device <NUM> may be connected to the cloud <NUM> via the communication interface <NUM>, using a wired connection, a wireless connection, or both. In some embodiments, the computing device <NUM> may communicate with the database <NUM> and/or an external device (e.g., a computing device) via the cloud <NUM>.

Turning now to <FIG>, a method <NUM> for utilizing a time-spaced robotic reference frame may be performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) <NUM> of the computing device <NUM> described above. The at least one processor may be part of a robot (such as the robot <NUM>) or part of a navigation system (such as the navigation system <NUM>). A processor other than any processor described herein may also be used to execute the method <NUM>. The at least one processor may perform the method <NUM> by executing instructions stored in a memory, such as the instructions <NUM> of the memory <NUM>. The instructions may correspond to one or more steps of the method <NUM> described below. The instructions may cause the processor to execute one or more algorithms, such as the algorithms <NUM>.

The method <NUM> comprises causing a robotic arm of a robot to move to a plurality of different poses (step <NUM>). The robotic arm of the robot may be the same as or similar to the robotic arm <NUM> of the robot <NUM> described above. The plurality of different poses may be two poses, three poses, four poses, or more poses. In some embodiments, the plurality of different poses may comprise tens or hundreds of poses. The robotic arm of the robot is caused to move to the plurality of different poses in sequence, such that there is a space or period of time (whether measured in milliseconds, seconds, minutes, or otherwise) between a first time at which the robotic arm is in one of the plurality of different poses and a second time at which the robotic arm is in a subsequent one of the plurality of different poses. The space or period of time between any two poses of the plurality of different poses (provided the two poses are adjacent in time) may be identical, or may vary from one pair of adjacent-in-time poses to another pair of adjacent-in-time poses.

The plurality of different poses may be selected or determined to maximize a distance between a tracking marker on the robotic arm in each pose and the tracking marker on the robotic arm in every other pose. The plurality of different poses may alternatively be selected or determined to maximize a distance between the tracking marker on the robotic arm in each pose and the tracking marker on the robotic arm in every other pose while still ensuring that the tracking marker remains within a field of view of a tracking marker sensor, such as the tracking marker sensor <NUM>. In still other embodiments, the plurality of different poses may be selected based at least in part on available poses within a navigation volume in light of one or more obstacles in the navigation volume (e.g., arms and hands of medical staff; tools, instruments, and other devices positioned within the navigation volume). In still further embodiments, the plurality of different poses may be selected based at least in part one or more of the foregoing considerations, and/or based on any other consideration.

In some embodiments, at least one of the plurality of different poses may be a pose in which the robotic arm is in contact with a designated location (e.g., a specific location on the patient). In such embodiments, the resulting time-spaced robotic reference frame may be used to correlate patient space, robotic space, and navigation space.

<FIG> illustrate, by way of example, one sequence of four different poses through which a robotic arm such as the robotic arm <NUM> may be caused to move. The robot <NUM> and robotic arm <NUM> thereof in <FIG> are shown in a field of view of a tracking marker sensor such as the tracking marker sensor <NUM>. In the example of <FIG>, the robotic arm <NUM> is extended to or nearly to a maximum extension thereof in each of the four different poses shown in <FIG>. With the tracking marker <NUM> positioned proximate a distal end of the robotic arm <NUM> in <FIG>, extension of the robotic arm <NUM> to or nearly to a maximum possible extension beneficially enables the time-spaced robotic reference frame created as part of the method <NUM> to be as large as possible, thus facilitating improved accuracy relative to a smaller reference frame. In other embodiments, however, a robotic arm may not be extended to or nearly to a maximum extension thereof, whether in some or in all of the plurality of different poses. Although <FIG> illustrate a robotic arm <NUM> with a single tracking marker <NUM> affixed thereto, in some embodiments of the present disclosure, a plurality of tracking markers <NUM> may be affixed to the robotic arm <NUM>.

In some embodiments, one of the plurality of different poses may be a current pose of the robotic arm, and another one of the plurality of different poses may be a pose of the robotic arm required in connection with a surgical step of a surgical procedure. The causing may comprise a processor (e.g., the processor <NUM>) executing instructions stored in memory (e.g., the instructions <NUM> stored in the memory <NUM>), which instructions may cause the processor to generate one or more signals and transmit the same to a robot such as the robot <NUM>. The signal(s) may be sent, for example, via a communication interface such as the communication interface <NUM>.

Referring again to <FIG>, the method <NUM> also comprises causing a tracking marker sensor to detect a first position, in a first coordinate space, of a tracking marker on the robotic arm when the robotic arm is in a first one of the plurality of different poses (step <NUM>). The tracking marker sensor may be the same as or similar to the tracking marker sensor <NUM>. The tracking marker sensor may be an optical camera, an infrared camera, or any other sensor configured to detect the tracking markers. The tracking marker sensor, in some embodiments, may be part of a robot such as the robot <NUM>, or part of a navigation system such as the navigation system <NUM>, or part of a computing device such as the computing device <NUM>. In some embodiments, the tracking marker sensor may be independent of any of the foregoing components, but may be in electronic communication with one or more of the foregoing components.

The detected position of the tracking marker may be defined based on a coordinate system such as a navigation coordinate system. The detected position of the tracking marker may be stored in a memory such as the memory <NUM>, or in a database such as the database <NUM>. The causing the tracking marker sensor to detect the first position may comprise a processor (e.g., the processor <NUM>) executing instructions stored in memory (e.g., the instructions <NUM> stored in the memory <NUM>), which instructions may cause the processor to generate one or more signals and transmit the signals to a tracking marker sensor such as the tracking marker sensor <NUM>. The signal(s) may be configured to cause the tracking marker sensor to activate (e.g., to take a picture, to open an aperture) so as to detect the first position of the tracking marker. The signal(s) may be timed to cause the tracking marker to activate once the robotic arm has reached one of the plurality of different poses. In some embodiments, the signal(s) may cause the tracking marker sensor to activate periodically for a set number of iterations (e.g., <NUM> times, <NUM> times, <NUM> times) in a given period of time (e.g. <NUM> second, <NUM> seconds, <NUM> seconds, a minute), during which given period of time the robotic arm may be continuously moving through a plurality of different poses (e.g., without pausing at each pose). The signal(s) may be sent, for example, via a communication interface such as the communication interface <NUM>.

The method <NUM> also comprises causing a tracking marker sensor to detect a second position, in the first coordinate space, of the tracking marker on the robotic arm when the robotic arm is in a second one of the plurality of different poses (step <NUM>). The step <NUM> may be identical to or substantially the same as the step <NUM>, but with the robotic arm in a second one of the plurality of different poses.

Although not reflected in <FIG>, in some embodiments, the method <NUM> may comprise causing the tracking marker sensor to detect a third position, a fourth position, a fifth position, and so on of the tracking marker on each of the one or more robotic arms when the one or more robotic arms are in a third one, a fourth one, a fifth one, and so on, respectively, of the plurality of different poses. Each detected position of each tracking marker of the one or more robotic arms may be stored, for example, in a memory such as the memory <NUM> and/or in a database such as the database <NUM>. When considered together, the various tracking marker positions define a time-spaced reference frame capable of having much larger dimensions than might otherwise be possible using, for example, a single, physical reference frame having a plurality of reflective spheres or other tracking markers secured to various fixed arms thereof. In some embodiments, the number of detected positions may be in the hundreds or greater.

<FIG> shows an example of a time-spaced robotic reference frame <NUM> according to embodiments of the present disclosure. The time-spaced robotic reference frame <NUM> comprises four detected tracking marker positions. More specifically, the tracking marker 156a is shown in a first position (corresponding to the pose of the robotic arm <NUM> in <FIG>), which position was detected at a first time. The tracking marker 156b is shown in a second position (corresponding to the pose of the robotic arm <NUM> in <FIG>), which position was detected at a second time after the first time. The tracking marker 156c is shown in a third position (corresponding to the pose of the robotic arm <NUM> in <FIG>), which position was detected at a third time after the second time. The tracking marker 156d is shown in a fourth position (corresponding to the pose of the robotic arm <NUM> in <FIG>), which position was detected at a fourth time after the third time.

In embodiments of the present disclosure in which a robotic arm such as the robotic arm <NUM> comprises a plurality of tracking markers affixed thereto, a time-spaced robotic reference frame generated may be based on the detected positions of the plurality of tracking markers at each of a plurality of different times. Thus, for example, if the robotic arm <NUM> of <FIG> had two tracking markers <NUM> affixed thereto, and a tracking marker sensor were configured to detect both of the tracking markers <NUM> at each of the four different poses of the robotic arm <NUM>, then the resulting time-spaced robotic reference frame would reflect eight tracking marker positions, rather than four tracking marker positions as in the reference frame <NUM>. Inclusion of a plurality of tracking markers on a robotic arm may therefore reduce the number of different poses that a robotic arm must assume to obtain sufficient points for a time-spaced robotic reference frame. For example, if four points are needed for a given time-spaced robotic reference frame, and the robotic arm has two tracking markers affixed thereto, then the time-spaced robotic reference frame may be created by moving the robotic arm to only two different poses. Similarly, if <NUM> points are needed for a given time-spaced robotic reference frame, and the robotic arm has four tracking markers affixed thereto, then the time-spaced robotic reference frame may be created by moving the robotic arm to twenty-five different poses (rather than the fifty different poses that would be required if the robotic arm had two tracking markers secured thereto, or the <NUM> different poses that would be required if the robotic arm had only one tracking marker secured thereto).

The time-spaced robotic reference frame <NUM> corresponds somewhat to a physical reference frame having four reflective spheres or other tracking markers mounted thereto, except that the time-spaced robotic reference frame <NUM> is not constrained by the fixed dimensions of a physical reference frame. Indeed, a robot such as the robot <NUM>, with a robotic arm such as the robotic arm <NUM>, can define a time-spaced robotic reference frame such as the reference frame <NUM> on demand, using a plurality of different poses that may be identical to or different than any previous plurality of different poses. Indeed, a time-spaced robotic reference frame according to embodiments of the present disclosure may be formed by causing a robotic arm to move to a plurality of different poses selected based upon, for example, the position of other objects in the surgical environment. As a result, whereas use of a physical reference frame with fixed dimensions in a crowded surgical environment may require moving one or more tools, instruments, monitors, or other objects out of the way, a time-spaced robotic reference frame may be generated by causing a robotic arm to assume poses that avoid objects in the surgical environment, such that those objects need not be moved. In some embodiments, the poses to which a robotic arm moves to generate the time-spaced robotic reference frame may be selected or other determined, and/or a path between each such pose may be selected or otherwise determined, based on information in (or determined from) a surgical plan or elsewhere about the position of one or more objects in the navigation space. For example, based on a surgical plan that calls for the use of MIS towers in connection with a spinal surgery, the processor <NUM> or another processor may predict the position of the MIS towers based on information in the surgical plan and/or stored information about the dimensions of the MIS towers, and may determine one or more poses and/or paths to, from, and/or between the one or more poses in light of the predicted position of the MIS towers. In other embodiments, a processor <NUM> or another processor may use navigation information about the location of one or more tracked objects in the navigation space when determining one or more poses and/or paths to, from, and/or between the one or more poses for purposes described herein.

Additionally, time-spaced robotic reference frames as described herein need not always have the same pattern or dimensions. For example, one time-spaced robotic reference frame may have dimensions that correspond to a plurality of different poses that extend a robotic arm to or nearly to a maximum extension thereof for each pose, while another time-spaced robotic reference frame may have dimensions that correspond to a plurality of different poses where none of those poses require a maximum or near-maximum extension of the robotic arm.

Although the reference frame <NUM> is shown in two dimensions (e.g., all of the tracking markers 156a to 156d are positioned in the same plane), time-spaced robotic reference frames according to embodiments of the present disclosure may comprise tracking markers positioned in three dimensions. In other words, the plurality of different poses of the robotic arm may comprise poses that cause the tracking marker to be located at different X, Y, and Z positions relative to an origin of a robotic coordinate system, where X, Y, and Z represent orthogonal axes passing through the origin of the robotic coordinate system. Moreover, the tracking marker sensor may be capable of detecting a position of the tracking marker in three dimensions (e.g., in a navigation coordinate system).

A fourth dimension, time, is used to create the time-spaced robotic reference frame <NUM>. However, the reference frame <NUM> itself, once generated by assembling or compiling the detected positions of the tracking marker <NUM> at each of the plurality of different poses corresponding to each of the plurality of different times, does not reflect time.

As evident from the foregoing, embodiments of the present disclosure advantageously eliminate a need for sometimes-bulky reference frames. For example, some navigation systems require certain minimum distances between tracking markers (or that physical references frames for holding tracking markers be a minimum size) to provide accurate navigation within a volume applicable to a given application. Larger volumes in particular may require larger reference frames than smaller volumes. In some instances, the reference frames holding tracking markers for surgeries or other medical procedures can extend from <NUM>,<NUM> (three inches) to <NUM> (ten inches) in multiple dimensions in order to achieve a minimum necessary size. As a result, these reference frames tend to be bulky, are easily bumped (which can, for example, cause undesired movement of the objects to which they are affixed), can get in the way of movement of a surgeon or of any movable object in an operating room environment, and can be difficult to use. The use of a time-spaced robotic reference frame as described herein enables a somewhat bulky object already in the operating room (the robot) to be used, when needed, to create a time-spaced reference frame, thus eliminating the need for dedicated reference frames and the problems associated therewith.

Returning to <FIG>, the method <NUM> also comprises receiving information from the robot corresponding a position of the tracking marker in a second coordinate space when the robotic arm is in each of the plurality of different poses (step <NUM>). Such information may be determined based on a sensed position of the robotic arm, as indicated by one or more sensors such as the sensors <NUM>. Such information may also be determined based on a known relationship of a position of the robotic arm to a position of the tracking marker. Using the system <NUM> for an example, a tracking marker <NUM> may be secured to a precise location (e.g., a location on the robotic arm <NUM> that is precisely known, as reflected by coordinates and/or measurements relative to the robotic arm <NUM> or to a robotic coordinate system) proximate a distal end of a robotic arm <NUM>. The precise location may be establishing during manufacture of the robotic arm <NUM> (e.g., precision tooling may be used to ensure the tracking marker <NUM> is installed precisely in a predetermined location on the robotic arm <NUM>), or during a subsequent calibration process (e.g., the position of the tracking marker <NUM> relative to the robotic arm <NUM> and/or to a robotic coordinate system may be precisely measured after the tracking marker <NUM> has been affixed to the robotic arm <NUM>).

In some embodiments, the information may not comprise a position of each tracking marker corresponding to each pose of the robotic arm, but may instead comprise information sufficient to determine or calculate a position of each tracking marker (in a robotic coordinate system) corresponding to each pose of the robotic arm. Thus, for example, the information may comprise information gathered from one or more sensors such as the one or more sensors <NUM> about the position and orientation of the robotic arm in each of the plurality of different poses, and/or information about the precise location of each tracking marker relative to the robotic arm. Based on such information, a processor such as the processor <NUM> may utilize one or more algorithms such as the algorithms <NUM> to calculate the position of each tracking marker in the robotic coordinate system for each pose of the plurality of different poses of the robotic arm. For example, if the information comprises sensor data sufficient to calculate a pose of one or more arm segments (e.g., segments <NUM>) of a robotic arm such as the robotic arm <NUM>, the calculated poses of the one or more arm segments may then be combined into a calculated pose of the robotic arm as a whole. Once the position of the robotic arm (in a robotic coordinate space) has been determined, then the position of the tracking marker can be determined based on a known position of the tracking marker relative to the robotic arm.

In some embodiments, the information received in the step <NUM> may comprise information stored in a memory such as the memory <NUM>. Such information may include, for example, information about the dimensions, arrangement, range of motion, and/or other characteristics of the segment(s)s of the one or more robotic arms and/or of the one or more robotic arms as a whole.

The method <NUM> also comprises comparing the positions of the tracking marker in the first coordinate space to the positions of the tracking marker in the second coordinate space (step <NUM>). The comparing may comprise registering the first coordinate space to the second coordinate space. The registering may comprise comparing the detected first position and the detected second position (both of which may be, for example, in a navigation coordinate space) to a received or calculated position of each tracking marker in a robotic coordinate space (as contained within or calculated from the received information), and based on the comparison, correlating the robotic coordinate space to the navigation coordinate space or vice versa.

In some embodiments, the registering may further be based on a detected position of a physical reference frame (e.g., not a time-spaced robotic reference frame), which may be secured to, for example, a robot base from which the robotic arm extends. The physical reference frame may be used to ensure that there is no relative movement between the tracking marker sensor and the robot base, whether during a period of time in which the robotic arm is caused to move to the plurality of different poses, during an entire surgical procedure, or during another period of time. Use of a physical reference frame on the robot base may be advantageous when the robot base is mounted on wheels or is otherwise movable. In other embodiments, however, the registering may be accomplished without information about or corresponding to any reference frame other than the time-spaced robotic reference frame. Such embodiments may include embodiments where the robot base and the tracking marker sensor are fixedly mounted relative to each other (e.g., by being fixedly mounted to a wall, a ceiling, an operating table, or other fixed structure). Such embodiments may also include embodiments where the plurality of different positions are reached during a relatively short period of time, such as a period of time equal to or less than five seconds, or equal to or less than three seconds, or equal to or less than two seconds, or equal to or less than one second.

In some embodiments, the registering may be accomplished without information about or corresponding to any reference frame other than the time-spaced robotic reference frame when the points making up the time-spaced robotic reference frame are captured within a predetermined period of time (e.g., within one second or less, or within two seconds or less, or within three seconds or less, or within five seconds or less), and with information about or corresponding to a physical reference frame attached to a robot base (or another physical reference frame) when the points making up the time-spaced robotic reference frame are captured in more than a predetermined period of time (e.g., in more than one second, or more than two seconds, or more than three seconds, or more than five seconds).

In other embodiments, the registering may be accomplished without use of or reliance upon any information corresponding to a physical reference frame (or any reference frame other than the time-spaced robotic reference frame), but only after an initial registration has been completed that is based, at least in part, on information about a physical reference frame. For example, if a physical reference frame is going to be obscured during a surgical procedure, then an initial registration can be accomplished using the physical reference frame, and re-registrations or registration verifications during the procedure may be accomplished using only the time-spaced robotic reference frame.

Also, when the comparison yields a conclusion that the detected tracking marker positions match the tracking marker positions as reflected by or determined using the information, the operational integrity of the robot and of the navigation system can be confirmed. This can be useful during surgical procedures as well as for initial calibration operations for the robotic system. On the other hand, if and when the comparison yields a conclusion that the detected tracking marker positions do not match the tracking marker positions as reflected by or determined using the information, a further conclusion can be reached that one or both of the robot and the navigation system lack operational integrity. Thus, when this occurs, a warning may be displayed to an operator of the robot and/or of the navigation system, and/or an audible sound may be played, via a user interface (such as, for example, the user interface <NUM> of the computing device <NUM>, or a user interface specific to the robot or to the navigation system). Provision of such a warning to an operator of the robot and/or of the navigation system helps to ensure that the suspect operational integrity of the robot and/or of the navigation system can be investigated, and any errors corrected, before the robot and/or the navigation system are used further.

In some embodiments, where the detected tracking marker positions are only slightly different than the tracking marker positions as reflected by or determined using the information, the steps <NUM> through <NUM> may be repeated (e.g., based on a different plurality of different poses of the robotic arm), and the detected tracking marker positions may again be compared to the tracking marker positions as reflected by or determined using the information. If the second tracking marker positions are again only slightly different than the tracking marker positions as reflected by or determined using the information, then an error calculation and/or calibration process may be undertaken to determine an adjustment to be applied to in the future to ensure that the detected tracking marker positions match the tracking marker positions as reflected by or determined using the information, or vice versa. In other words, if an offset between a detected arrangement of the plurality of tracking markers and a corresponding set of tracking marker positions as reflected by or determined using the information can be characterized by a constant or a derived equation, such that the offset can be incorporated into further comparisons, then the operational integrity of the robot and/or of the navigation system may be confirmed.

The present disclosure encompasses embodiments of the method <NUM> that comprise more or fewer steps than those described above. Although aspects of the method <NUM> were described with respect to a single robot comprising a single robotic arm, the method <NUM> may utilize a single robot with a plurality of robotic arms (one or more of which having one or more tracking markers fixedly secured thereto), a plurality of robots each with a single robotic arm having one or more tracking markers fixedly secured thereto, and/or a plurality of robots each with a plurality of robotic arms (one or more of which having one or more tracking markers fixedly secured thereto).

Turning now to <FIG>, a method <NUM> of utilizing a time-spaced robotic reference frame for registration comprises receiving first information about positions of a tracking marker at each of a plurality of different times (step <NUM>). The first information may be received, for example, from a sensor such as the tracking marker sensor <NUM>, or any other sensor suitable for detecting the tracking marker. The first information may be received directly from the sensor or via one or more communication interfaces such as the communication interface <NUM>, and/or via a cloud such as the cloud <NUM>, or via any other network, device, or component.

The plurality of different times may comprise two times, three times, four times, or more times. In some embodiments, the plurality of different times may comprise ten times or <NUM> times. Each of the plurality of different times may fall within a period of time of one second or less, or five seconds or less, or ten seconds or less, or one minute or less, or more than one minute. The tracking marker may be, for example, a tracking marker <NUM>, or any other tracking marker. The tracking marker may be fixedly secured to a robotic arm, which may be any robotic arm described herein, including, for example, a robotic arm <NUM> of a robot <NUM>, or any other robotic arm.

The first information may be or comprise information about positions of the tracking marker in a navigation coordinate system at each of the plurality of times. In some embodiments, the first information may comprise information about a first position of the tracking marker in a navigation coordinate system at at least one time of the plurality of times, and may further comprise information about a second position of the at least one tracking marker relative to the first position. In other words, the first information may comprise information sufficient to determine a position of the tracking marker relative to a navigation coordinate system at each of the plurality of times.

The method <NUM> also comprises receiving second information corresponding to positions of the tracking marker in a robotic coordinate system at each of the plurality of different times (step <NUM>). The second information may describe (or enable calculations sufficient to determine) the position of the tracking marker relative to a robotic coordinate space at each of the plurality of different times. The second information may be obtained, for example, from the robotic arm, which may be a robotic arm <NUM> as described herein or any other robotic arm. The second information may additionally or alternatively be obtained from or via a robot comprising the robotic arm, such as the robot <NUM>, and/or from a memory such as the memory <NUM>, a computing device such as the computing device <NUM> more generally, a database such as the database <NUM>, a cloud such as the cloud <NUM>, or another source.

The second information may be based, for example, on data obtained from one or more sensors in the robotic arm, such as the sensors <NUM>. For example, the second information may comprise sensor data about a detected pose of one or more segments of the robotic arm and/or of the robotic arm as a whole. The second information may additionally or alternatively be based on one or more settings of one or more components of the robotic arm. For example, the second information may comprise data describing a position (whether an actual position or a commanded position) of one or more motors, servos, gears, or other devices or components used to control a pose of the robotic arm and/or one or more segments thereof. The second information may be obtained independently of the first information, and vice versa. The second information may comprise information describing the pose of the robotic arm at each of the plurality of times, or information from which the pose of the robotic arm at each of the plurality of times may be calculated. In the latter instance, one or more algorithms such as the algorithms <NUM> may be used to calculate the pose of the robotic arm at each of the plurality of times.

The second information may also comprise information about the position of the tracking marker relative to the robotic arm, and information about the robotic arm relative to the robotic coordinate system.

The method <NUM> also comprises comparing the robotic coordinate system to a navigation coordinate system based on the first information and the second information (step <NUM>). The comparing may comprise registering the robotic coordinate system to the navigation coordinate system. The registering may be accomplished based on the first information and the second information. In some embodiments, the registering comprises correlating a pose of the robotic arm in the robotic coordinate system at each of the plurality of different times to a detected position of the tracking marker in the robotic coordinate system at each of the plurality of different times. Such correlating may comprise accessing information from a memory such as the memory <NUM>, a database such as the database <NUM>, a robot such as the robot <NUM>, or another storage location. The accessed information may include, for example, information regarding a precise position of the tracking marker on the robotic arm, and/or information about the dimensions, arrangement, range of motion, and/or other characteristics of the segments of the robotic arm and/or of the robotic arm as a whole. Based on such information as well as information about a pose of the robotic arm in the robotic coordinate system (e.g., the second information received in step <NUM>), a position of the plurality of tracking markers in the robotic coordinate system may be calculated. Such calculations may utilize, for example, one or more algorithms such as the algorithms <NUM>. The result of the correlating may be a calculated or otherwise determined position, in the robotic coordinate system, of the tracking marker at each of the plurality of different times.

The registering may comprise determining a relationship between the robotic coordinate system and the navigation coordinate system based on a known position of the tracking marker in the navigation coordinate system (as included in or determined from the first information) at each of the plurality of different times and a known position of the tracking marker in the robotic coordinate system (as included in or determined from, e.g., the second information) at each of the plurality of different times. The registering may utilize one or more algorithms such as the algorithms <NUM> stored in the memory <NUM>.

In some embodiments, the method <NUM> may also comprise registering a patient coordinate system to the navigation coordinate system. Where the robot is connected to the patient (as is sometimes the case during robotic or robotic-assisted surgery), or where the robotic coordinate system has already been registered to a patient coordinate system, registration of the robotic coordinate system to the navigation coordinate system (as described with respect to step <NUM> above) also enables registration of the patient coordinate system to the navigation coordinate system. In some embodiments, the registering comprises correlating the robotic coordinate system to a patient coordinate system (or vice versa), and, based on that registration, correlating the patient coordinate system to the navigation coordinate system. In other words, the registering may comprise determining a relationship between the patient coordinate system and the navigation coordinate system based on a relationship between the patient coordinate system and the robotic coordinate system and a relationship between the robotic coordinate system and the navigation coordinate system. Such registering may utilize one or more algorithms such as the algorithms <NUM> stored in the memory <NUM>.

The method <NUM> also comprises operating the robot based on the comparison (step <NUM>). The operating may comprise a processor such as the processor <NUM> executing instructions (e.g., instructions <NUM>) stored in a memory such as the memory <NUM> that cause the processor <NUM> to transmit one or more signals to the robot <NUM> that cause the robotic arm <NUM> to move in one or more ways to carry out one or more steps of a surgical procedure. The comparison may beneficially enable the robotic arm to be moved to precise positions relative to a patient, using information about the patient such as one or more preoperative images and/or a preoperative surgical plan.

The present disclosure encompasses a number of variations on the method <NUM>. For example, in some embodiments, the second information may comprise position information for the tracking marker (in the robotic coordinate system) at each of the plurality of times, such that the correlating described above with respect to the step <NUM> is unnecessary (as are any calculations to determine a pose of the robotic arm based on the second information, as described above with respect to the step <NUM>). Additionally, while in the method <NUM> described above a position of the tracking marker in the robotic coordinate system at each of the plurality of different times is determined based on the second information about a pose of the robotic arm in the robotic coordinate system, in other embodiments a pose of the robotic arm may be determined based on the first information, and the registering step <NUM> may comprise registering the robotic coordinate system to the navigation coordinate system (or vice versa) based on the position of the robotic arm as determined from the first information and the position of the robotic arm as indicated by or determined from the second information.

The method <NUM> beneficially enables registration of a navigation coordinate system to a robotic coordinate system or vice versa without the use of a reference frame other than a time-spaced robotic reference frame formed (over time) by the robotic arm (including the tracking marker fixedly secured to the robotic arm) itself. The method <NUM> thus avoids the cost of a separate reference frame, the expenditure of the time needed to secure the separate reference frame to the robotic arm, and, in instances where the reference frame would be secured directly to the patient (e.g., to the patient's vertebra or pelvis), the incision that would otherwise be needed to secure the separate reference frame to the patient. Moreover, the method <NUM> removes the need for a snapshot frame that would otherwise be required during registration of a navigation coordinate system to a robotic coordinate system or vice versa.

The present disclosure encompasses embodiments of the method <NUM> with more or fewer steps than those described above.

With reference now to <FIG>, a method <NUM> of utilizing a robotic reference frame comprises receiving information corresponding to a pose of a robotic arm at each of a plurality of different times (step <NUM>). The plurality of different times may comprise a first time and a second time after the first time. In some embodiments, the plurality of different times may further comprise a third time after the second time, a fourth time after the third time, a fifth time after the fourth time, and/or additional times. The plurality of different times may be evenly spaced (e.g., each time of the plurality of different times may be spaced from each adjacent time in the plurality of different times by the same amount of time), or differently spaced. The plurality of different times may all fall within a total time comprising one second less or less, ten seconds or less, one hundred seconds or less, one thousand seconds or less, or any other period of time. The plurality of different times may simply be determined by when the robotic arm reaches each of a plurality of different poses.

The robotic arm may be a robotic arm <NUM> as described herein, or any other robotic arm. A tracking marker is fixedly secured to the robotic arm proximate a distal end thereof. In some embodiments, more than one tracking marker may be fixedly secured to the robotic arm. Also in some embodiments, more than one robotic arm may be used in connection with the method <NUM>. The pose of the robotic arm is different at each of the plurality of different times. The pose of the robotic arm at a first time of the plurality of different times may be such that the robotic arm extends substantially in a first direction, and the pose of the robotic arm at a second time of the plurality of different times may be such that the robotic arm extends substantially in a second direction that is orthogonal to the first direction In some embodiments, the pose of the robotic arm at a third time of the plurality of different times may be such that the robotic arm extends substantially in a third direction that is orthogonal to the first and second directions. In other embodiments, the robotic arm may not extend, in any of the poses corresponding to the plurality of different times, in any direction orthogonal or substantially orthogonal to any direction in which the robotic arm extends in any other pose corresponding to one of the plurality of different times.

In some embodiments, the information corresponding to a pose of the robotic arm at each of the plurality of different times may be information corresponding to a predicted pose of the robotic arm at each of the plurality of different times. For example, the information may correspond to a plurality of planned poses to which the robotic arm will be moved (e.g., in sequence, such that the robotic arm will assume each of the planned poses at a different time), and the plurality of different times may be determined by the times at which the robotic arm is predicted to reach each of the planned poses. The information may correspond to or comprise the results of one or more calculations carried out (e.g., by a robot such as the robot <NUM>, or by a processor such as the processor <NUM>) to determine a predicted position of a tracking marker based on an expected or current pose of the robotic arm to which the tracking marker is fixedly secured. At least one pose of the robotic arm at one of the plurality of different times may be a pose that enables an end of the robotic arm (whether an end effector or otherwise) to be in contact with an object, the position of which needs to be determined or confirmed (whether in a navigation coordinate space or otherwise). The information may comprise information about a position of a tracking marker fixedly secured to the robotic arm relative to a robotic coordinate system or another coordinate system. Where multiple robotic arms are used, each having a tracking marker fixedly secured thereto, the information may comprise information about a characteristic of each tracking marker, such as a wavelength, pulsating frequency, or geometric pattern of each individual tracking marker.

Each tracking marker may be any tracking marker described herein, including, for example, a tracking marker <NUM>. Each tracking marker is secured to a robotic arm.

The method <NUM> also comprises receiving data corresponding to a detected position of a tracking marker on the robotic arm at each of the plurality of different times (step <NUM>). The data may be received, for example, from a tracking marker sensor such as the tracking marker sensor <NUM>. The data may comprise position information of the tracking marker in a navigation coordinate space or another coordinate space. Alternatively, the data may enable calculation of a position of the tracking markers in a navigation coordinate space or another coordinate space. Where multiple tracking markers are used, the data may comprise information about a characteristic of each individual tracking marker, such as a wavelength, pulsating frequency, or geometric pattern of each individual tracking marker.

Notably, the data corresponding to the detected position of the tracking marker is independent of the information corresponding to the pose of the robotic arm, even if such information comprises position information about the tracking marker. In other words, the data is generated without reference to the information, and the information is generated without reference to the data.

The method <NUM> also comprises combining the information and the data to generate a custom, time-spaced reference frame (step <NUM>). The combination may yield, for example, a custom, time-spaced reference frame such as that shown in <FIG>. When the detected position of the tracking marker at each of the plurality of different times (e.g., as reflected in the data) matches the predicted or expected position of the tracking marker (e.g., as reflected in or determined from the information), the exact coordinates of the custom, time-spaced reference frame are known in both a navigation coordinate system and a robotic coordinate system. As a result, the custom, one-time reference frame is usable instead of a separate reference frame that must be secured to or held by the robotic arm or another object or person.

The comparing may comprise using one or more algorithms such as the algorithms <NUM> to translate a position of the tracking markers in one coordinate system (e.g., in a robotic coordinate system) to a position of the tracking markers in another coordinate system (e.g., in a navigation coordinate system). The comparing may also comprise overlaying an image contained in or generated using the data corresponding to the detected positions of the tracking markers at the plurality of different times on a virtual image generated based on the information corresponding to the pose of the robotic arm at each of the plurality of different times to determine whether the tracking markers in both images line up with each other. Other comparison methods may also be used to determine whether the positions of the tracking marker as reflected in the data match the positions of the tracking marker as reflected in or determined from the information.

The method <NUM> also comprises confirming a position of an object in a predetermined coordinate space based on the creation of the custom, one-time reference frame (step <NUM>). As noted above, when the positions of the tracking marker as reflected in or determined from the information match the positions of the tracking marker as reflected in the data, the precise coordinates of the time-spaced robotic reference frame are known. If the position (and, in some embodiments, the orientation) of an object can be detected simultaneously with the detection of the tracking marker at one or more of the plurality of different times, and/or if the robotic arm is in contact with a known surface or feature of the object at one or more of the plurality of different times, then the position (and, in some embodiments, the orientation) of the object may be determined using the custom, one-time reference frame, just as the position (and, in some embodiments, the orientation) of the object may be determined using a reference frame separate from the robotic arm.

Each of the embodiments described herein may utilize a single robotic arm with a single tracking marker secured thereto; a single robotic arm with a plurality of tracking markers secured thereto; a plurality of robotic arms with a single tracking marker secured to each robotic arm; and a plurality of robotic arms with a plurality of tracking markers secured to each robotic arm. As explained above, in embodiments that comprise or utilize a plurality of tracking markers, fewer different poses of the one or more robotic arms may be necessary to achieve a time-spaced robotic reference frame useable for an intended purpose. Similarly, in embodiments that comprise or utilize a plurality of robotic arms, each with at least one tracking marker secured thereto, fewer different poses of the plurality of robotic arms may be necessary to achieve a time-spaced robotic reference frame useable for an intended purpose.

Where a plurality of robotic arms are used in connection with any one or more of the systems, methods, or devices described herein, any information described above as corresponding to one or more poses of a robotic arm may comprise information corresponding to one or more poses of each robotic arm of the plurality of robotic arms. Such information may also be or comprise, in some embodiments, information about a position, in robotic coordinate space, of one or more tracking markers affixed to each of the plurality of robotic arm.

Embodiments of the present disclosure may beneficially increase patient safety during a surgical procedure. More specifically, a time-spaced robotic reference frame as disclosed herein may be used to confirm a registration between, for example, a robot coordinate space and a patient coordinate space, and/or between a navigation coordinate space and a robot coordinate space, and/or between a navigation coordinate space and a patient coordinate space (using a robotic coordinate space as an intermediary). For example, a robot coordinate space may be registered to a patient coordinate space using a relatively small physical reference frame, thus yielding a registration a first degree of accuracy. The robot coordinate space may also be registered to the patient coordinate space using a relatively larger time-spaced robotic reference frame, thus yielding a registration with a second degree of accuracy greater than the first degree of accuracy. As another example, after an initial registration of a robot coordinate space to a patient coordinate space, a time-spaced robotic reference frame as disclosed herein may be used at one or more times throughout a surgical procedure (or, in some embodiments, continuously throughout the surgical procedure) to confirm (using a navigation system comprising a tracking marker sensor) that a robotic arm is in fact in a position that that the robot understands the robotic arm to be in (e.g., through the use of one or more encoders or other sensors in the robotic arm).

Embodiments of the present disclosure enable a trade-off between use of space and time to maintain or improve the accuracy of a registration, re-registration, or registration verification (any one of which may be understood as a registration). For example, where available space is insufficient to utilize a physical reference frame of sufficient size to obtain a desired level of accuracy, a robotic arm having at least one tracking marker secured thereon, as described herein, may be used to generate, over time, a time-spaced robotic reference frame. On the other hand, where available space is sufficient to utilize a physical reference frame of sufficient size to obtain a desired level of accuracy, a single snapshot of such a reference frame may provide the needed information for the registration. The inclusion of multiple tracking markers on a single robotic arm (or of a single tracking marker on multiple robotic arms, or of multiple tracking markers on multiple robotic arms) may be used to reduce the amount of time needed to generate a time-spaced robotic reference frame, while increasing the amount of space needed. As space and/or time constraints change during the course of a surgical procedure, a reference frame suitable for the then-current constraints may be selected and used.

As may be appreciated based on the foregoing disclosure, the present disclosure encompasses methods with fewer than all of the steps identified in <FIG>, <FIG>, and <FIG> (and the corresponding description of the methods <NUM>, <NUM>, and <NUM>), as well as methods that include additional steps beyond those identified in <FIG>, <FIG>, and <FIG> (and the corresponding description of the methods <NUM>, <NUM>, and <NUM>). One or more aspects of the present disclosure may be the same as or similar to one or more corresponding aspects of the devices, systems, and methods disclosed in <CIT>.

Claim 1:
A robotic surgery navigation system (<NUM>), comprising:
a robot base (<NUM>);
a robotic arm (<NUM>; 144A, 144B), comprising:
a proximal portion secured to the robot base (<NUM>);
a distal portion movable relative to the proximal portion; and
a tracking marker (<NUM>) secured to the robotic arm (<NUM>; 144A, 144B) proximate the distal portion;
at least one processor (<NUM>);
a navigation system (<NUM>) including a tracking marker sensor (<NUM>) configured to identify positions of the tracking marker (<NUM>) in a first coordinate space; and
a memory (<NUM>) storing instructions for execution by the at least one processor (<NUM>) that, when executed, cause the at least one processor (<NUM>) to:
cause the robotic arm (<NUM>; 144A, 144B) to move to a plurality of different poses.
wherein at least one of the plurality of different poses corresponds to a maximum extension of the robotic arm (<NUM>; 144A, 144B);
receive information relating to a position of the tracking marker (<NUM>) in a second coordinate space when the robotic arm (<NUM>; 144A, 144B) is in each of the plurality of different poses; and
compare the positions of the tracking marker (<NUM>) in the first coordinate space to the positions of the tracking marker (<NUM>) in the second coordinate space,
wherein the navigation system (<NUM>) is configured to detect a first position of the tracking marker (<NUM>) at a first time when the robotic arm (<NUM>; 144A, 144B) is in a first pose of the plurality of different poses, and to detect a second position of the tracking marker (<NUM>) at a second time when the robotic arm (<NUM>; 144A, 144B) is in a second pose of the plurality of different poses, the second time after the first time and the second position different than the first position, and
wherein the memory (<NUM>) stores additional instructions for execution by the processor (<NUM>) that, when executed, cause the at least one processor (<NUM>) to register the first coordinate space to the second coordinate space based at least on the detected first position, the detected second position, and the information.