SYSTEM AND METHOD TO REGISTER AND CALIBRATE ULTRASOUND PROBE FOR NAVIGATION IN REAL TIME

A system includes an imaging system and a tracking system. The system generates a navigation space based on one or more tracking signals emitted by a transmission device. The system generates a virtual space including at least a portion of a calibration phantom based on one or more images generated by the imaging system. The system identifies a set of coordinates in the virtual space in response to an event in which at least a portion of the tracked device is detected in the one or more images. The system calibrates a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event. Calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

FIELD OF INVENTION

The present disclosure is generally directed to navigation and ultrasound imaging, and relates more particularly to calibrating an ultrasound probe for navigation.

BACKGROUND

Imaging devices and navigation systems may assist a surgeon or other medical provider in carrying out a surgical procedure. Imaging may be used by a medical provider for visual guidance in association with diagnostic and/or therapeutic procedures. Navigation systems may be used for tracking objects (e.g., instruments, imaging devices, etc.) associated with carrying out the surgical procedure.

BRIEF SUMMARY

Example aspects of the present disclosure include:

A system including: a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: generate a navigation space based on one or more tracking signals; generate a virtual space including at least a portion of a calibration phantom based on one or more images, wherein the one or more images are generated in response to one or more signals transmitted by an imaging device; identify a set of coordinates in the virtual space in response to an event in which at least a portion of a tracked device in the calibration phantom is detected in the one or more images; and calibrate a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

Any of the aspects herein, wherein the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a surface of the virtual space at the set of coordinates.

Any of the aspects herein, wherein: calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event; and calibrating the first coordinate system with respect to the second coordinate system is absent pausing a surgical procedure.

Any of the aspects herein, wherein calibrating the first coordinate system with respect to the second coordinate system is based on: beam thickness (e.g., ultrasound beam thickness), beam shape (e.g., ultrasound beam shape), or both of the one or more signals transmitted by the imaging device; pose information of the portion of the tracked device in association with an intersection between the portion of the tracked device and a plane of the virtual space; and one or more properties of the portion of the tracked device.

Any of the aspects herein, wherein the calibration phantom includes: ultrasound conductive material; or a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

Any of the aspects herein, wherein the instructions are further executable by the processor to: output guidance information associated with positioning the imaging device, the tracked device, or both in association with calibrating the first coordinate system with respect to the second coordinate system.

Any of the aspects herein, wherein the tracked device is included in at least a portion of an instrument, and the instructions are further executable by the processor to: detect, in the one or more images, one or more landmarks corresponding to at least a portion of the tracked device, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the one or more landmarks.

Any of the aspects herein, wherein calibrating the first coordinate system with respect to the second coordinate system includes verifying a registration accuracy between the first coordinate system and the second coordinate system.

Any of the aspects herein, wherein the instructions are further executable by the processor to: detect one or more discrepancies between first tracking data corresponding to the tracked device in association with the navigation space and second tracking data corresponding to the tracked device in association with the virtual space; and generate a notification associated with the one or more discrepancies, perform one or more operations associated with compensating for the one or more discrepancies, or both.

Any of the aspects herein, wherein the virtual space corresponds to a field of view of the imaging device.

Any of the aspects herein, wherein the navigation space and the tracked device are associated with at least one of: an optical tracking system, an acoustic tracking system, an electromagnetic tracking system, a radar tracking system, a magnetic tracking system, an inertial measurement unit (IMU) based tracking system, and a computer vision based tracking system.

A system including: an imaging system including an imaging device; a tracking system including: a transmission device; and a tracked device; a calibration phantom; a processor; and a memory storing data that, when processed by the processor, cause the processor to: generate a navigation space based on one or more tracking signals emitted by the transmission device; generate a virtual space including at least a portion of the calibration phantom based on one or more images generated by the imaging system, wherein the one or more images are generated in response to one or more signals transmitted by the imaging device; identify a set of coordinates in the virtual space in response to an event in which at least a portion of the tracked device is detected in the one or more images; and calibrate a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

Any of the aspects herein, wherein the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a surface of the virtual space at the set of coordinates.

Any of the aspects herein, wherein: calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event; and calibrating the first coordinate system with respect to the second coordinate system is absent pausing a surgical procedure.

Any of the aspects herein, wherein the calibration phantom includes: ultrasound conductive material; or a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

Any of the aspects herein, wherein calibrating the first coordinate system with respect to the second coordinate system is based on: beam thickness, beam shape, or both of the one or more signals transmitted by the imaging device; pose information of the portion of the tracked device in association with an intersection between the portion of the tracked device and a plane of the virtual space; and one or more properties of the portion of the tracked device.

A method including: generating a navigation space based on one or more tracking signals; generating a virtual space including at least a portion of a calibration phantom based on one or more images, wherein the one or more images are generated in response to transmitting one or more imaging signals; identifying a set of coordinates in the virtual space in response to an event in which at least a portion of a tracked device in the calibration phantom is detected in the one or more images; and calibrating a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

Any of the aspects herein, wherein the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a plane of the virtual space at the set of coordinates.

Any of the aspects herein, wherein: calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event; and calibrating the first coordinate system with respect to the second coordinate system is absent pausing a surgical procedure.

Any of the aspects herein, wherein the calibration phantom includes: ultrasound conductive material; or a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

Any one or more of the features disclosed herein.

Any one of the aspects/features/implementations in combination with any one or more other aspects/features/implementations.

Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the implementation descriptions provided hereinbelow.

DETAILED DESCRIPTION

It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or implementation, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and/or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different implementations of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and/or a medical device.

In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia Geforce RTX 2000-series processors, Nvidia Geforce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

The terms proximal and distal are used in this disclosure with their conventional medical meanings, proximal being closer to the operator or user of the system, and further from the region of surgical interest in or on the patient, and distal being closer to the region of surgical interest in or on the patient, and further from the operator or user of the system.

In some systems, an ultrasound probe may be calibrated/registered relative to a navigation means (e.g., a tracked sensor, etc.) in association with navigated image acquisition. For example, some systems may establish a transformation matrix that maps the six-dimensional (6D) pose (e.g., position and orientation information) of the tracked sensor to the 6D pose of an ultrasound probe. Some systems may map the 6D pose of the tracked sensor to an image generated by the ultrasound probe or to the ultrasound beam of the ultrasound probe.

In some cases, some calibration methods may be tedious, time consuming, and error prone. In some other cases, some calibration phantoms utilized for calibrating the ultrasound probe to the navigation system are costly and are prone to decay with time (e.g., due to the degradation of hydrogels implemented in some calibration phantoms).

Instances may occur in which a surgical team is unaware that the ultrasound probe has lost calibration with the navigation system. In some cases, even if the surgical team is aware of the loss in calibration, the team may be unwilling to recalibrate the ultrasound probe (e.g., due to lack of time or resources). Undetected or unaddressed loss in calibration during a medical procedure (e.g., due to deformation, tool drop/hit, etc.) may result in surgical errors. In some other cases, metal and other materials present in the environment may cause distortion to an electromagnetic field generated by the navigation system in association with tracking an object, and such distortion may result in surgical errors.

In accordance with aspects of the present disclosure, systems and techniques described herein may support dynamic initial calibration and dynamic recalibration of ultrasound probes (also referred to herein as ultrasonic probes) for navigation. The systems and techniques may incorporate an ultrasound probe connected to a main application/navigation system (also referred to herein as a navigated surgery system). The systems and techniques may include electromagnetic navigation of the ultrasound probe using trackers/sensors coupled to the ultrasound probe and an emitter capable of emitting electromagnetic signals. In some aspects, the systems and techniques may include an electromagnetic tracked device (e.g., an electromagnetic pointer (or stylus)) and a calibration phantom.

In some examples, the calibration phantom may be a phantom with a configuration of rods or wires. In some other examples, the calibration phantom may be a water bath or a tissue phantom, but is not limited thereto. The example calibration phantoms described herein are stable, inexpensive compared to some other calibration phantoms, and electromagnetic friendly. For example, the calibration phantoms may be free of materials that may interfere with electromagnetic navigation and tracking. In some example implementations, the calibration phantom may be a gel (e.g., hydrogel) for ultrasound calibration with optical tracking.

Examples of the techniques described herein may include moving or positioning a tracked device inside the calibration phantom while observing the movement of the tracked device using ultrasound imaging generated by an ultrasound imaging device, in which the ultrasound imaging corresponds to or represents an ultrasound view of the ultrasound imaging device. The techniques may include recording a video file of the ultrasound imaging concurrently with the tracking data and processing the video file (e.g., using a software script). Based on the processing of the video file, the techniques may include identifying a temporal instance at which a portion (e.g., tip) of the tracked device enters the ultrasound view. The systems and techniques may calibrate the ultrasound view with respect to the tracking space. Examples of the tracked device and the tracking space include an electromagnetic tracked device and an electromagnetic tracking space, but are not limited thereto. Aspects of the present disclosure support any type of tracked devices (e.g., sensors) and tracking spaces that may be implemented by a navigation system.

The systems and techniques described herein may support autonomous or semi-autonomous calibration, calibration verification with reduced calibration time compared to some other calibration techniques, and providing or outputting guidance information (e.g., tutorials, user prompts, etc.) for users on how to move or position a device (e.g., ultrasound imaging device, electromagnetic tracked device, an electromagnetic pointer (or stylus), etc.) in association with calibration. For example, the systems and techniques described herein support performing multiple calibrations (e.g., an initial calibration using a water bath, one or more subsequent calibrations using a tissue phantom, etc.), in which a system may perform the calibrations autonomously (or semi-autonomously). The systems and techniques may perform the calibrations continuously (or semi-continuously) and/or in response to trigger criteria, aspects of which are described herein. It is to be understood that as described herein with respect to calibration may be applied to recalibration. The terms “calibration,” “recalibration,” “calibration verification,” and “reregistration” may be used interchangeably herein.

Techniques described herein may be implemented in hardware, software, firmware, or any combination thereof that may automatically detect instrument landmarks on ultrasound images during a medical procedure. The techniques may include detecting landmarks of an instrument (e.g., tip of a needle during placement, distinctive features of navigated catheters, tip of a registration stylus, etc.) during the medical procedure and, using the detected instrument landmarks, automatically calibrating (or adjusting the calibration of) the ultrasound imaging device to the navigation system. The ultrasound imaging device may be, for example, an ultrasound probe.

Aspects of the automatic calibration techniques may provide a time savings for the surgical team, an improved user experience, and increased accuracy over longer portions of medical procedures. Other aspects of the calibration techniques provide cost savings through the use of, as a calibration phantom, an empty container (e.g., an empty box including an electromagnetic friendly material) with cross wires with patterns. The calibration phantom may be filled with water for the calibration procedure, thereby resulting in cost savings compared to other materials (e.g., gels, silicones, etc.). In some additional aspects, as water does not decay with time (e.g., compared to gels and silicones), using water in the calibration phantom may provide increased durability.

The electromagnetic tracking and calibration solutions described herein support directly using electromagnetic tools implemented in some existing medical procedures. In some aspects, direct use of such existing electromagnetic tools may provide increased accuracy due to accurate tracking of electromagnetic tools by some navigation systems.

In some examples, the calibration techniques described herein may be implemented using actual tissue of a subject as a calibration phantom. For example, the calibration techniques described herein may be implemented during a medical procedure associated with the actual tissue. In an example, if medical personnel is inserting a device (e.g., an ablation antenna, cardiac catheter, etc.) using ultrasound guidance provided by an ultrasound imaging device, the device will be visible in the ultrasound view. The calibration techniques and calibration software described herein may include using (e.g., automatically, or in response to a user request, etc.) the ultrasound images and corresponding electromagnetic tracking information to recalibrate the registration between the ultrasound space and the electromagnetic navigation space (also referred to herein as recalibrating the ultrasound tracking registration), without interrupting the medical procedure.

The systems and techniques described herein support recalibrating the ultrasound imaging system in the background based on automatic detection of target objects (e.g., instruments, tools, etc.) in the ultrasound images using AI/machine learning computer vision algorithms and object detection. The systems and techniques support automatic registration which may be implemented continuously, based on each event in which a tracked instrument or tracked device is detected in the ultrasound view, and/or periodically (e.g., based on a temporal trigger).

The systems and techniques support automatic registration in response to other trigger criteria (e.g., in response to detection of a target instrument in an ultrasound image) at any point during a medical procedure. In an example, the systems and techniques may include continuously verifying the registration accuracy between the ultrasound imaging system and the navigation system anytime the target instrument (e.g., surgical instrument, electromagnetic pointer, etc.) is detected in the ultrasound imaging. The systems and techniques support alerting the user and/or taking corrective actions in response to registration discrepancies. In an example, after outputting an alert (e.g., an audible alert, a visible alert, a haptic alert, etc.) to the user, the system may provide the user with a list of corrective actions for improving calibration. Non-limiting examples of corrective actions may include real-time actionable feedback for users to move the target instrument in association with registration.

In some other aspects, the systems and techniques may support dynamically and automatically detecting distortion in the navigated volume due to discrepancies between expected navigation and imaging data. In an example, the discrepancies may be between pose information of a tracked object as indicated by the navigation system and pose information of the tracked object as indicated by the imaging data. The systems described herein may support techniques for alerting (e.g., providing a notification to) a user of the discrepancies and compensating for the discrepancies. The systems and techniques may include calibrating the navigation data to the imaging data (e.g., calibrating a navigation space to an ultrasound space) while compensating for the discrepancies.

Aspects of the present disclosure support integration of a calibration phantom (e.g., water bath, hydrogel phantom, etc.) into the structure of a patient tracker. The integration of the calibration phantom may support user recalibration of a navigated ultrasound probe before or during a medical procedure. According to example aspects of the recalibration techniques described herein, the temporal duration associated with recalibration is reduced compared to some other recalibration techniques.

Implementations of the present disclosure provide technical solutions to one or more of the problems associated with other navigation systems and calibration techniques. For example, the systems and techniques described herein provide time savings, improved user experience, cost savings, and increased accuracy in comparison to other registration and calibration techniques. The systems and techniques described herein support continuous registration during a surgical procedure and continuous registration verification, in which registration and registration verification may be autonomous or semi-autonomous.

Aspects of the systems and techniques described herein support time efficient and cost-effective utilization of ultrasound images during surgery to navigate and display to medical personnel the locations of surgical devices (e.g., instruments, surgical tools, robotic end effectors, etc.) with respect to the patient anatomy. The systems and techniques described herein provide a reliable accuracy of the calibration between imaging devices (e.g., an ultrasound image probe, other imaging probes, etc.) and a navigation system, and the reliable accuracy may support accurate navigation of images (e.g., ultrasound images, etc.) that are generated based on data captured by the imaging devices.

In some aspects, different imaging probes may be different based on manufacturer, configuration, probe type, and the like, and such imaging probes may require new calibration and can lose calibration during a medical procedure. Aspects of the calibration techniques described herein are relatively time efficient, cost effective, user friendly, and provide increased accuracy compared to other techniques for calibrating or recalibrating imaging probes. The time efficiency, cost effectiveness, user friendliness, and increased accuracy supported by the systems and techniques described herein may provide improved confidence for a surgeon in a navigated ultrasound space and support a reduction in surgical errors. In some alternative and/or additional aspects, the registration and calibration techniques described herein may be implemented autonomously (e.g., without input from medical personnel) or semi-autonomously (e.g., with partial input from medical personnel).

Aspects of the present disclosure relate to navigated and robotic surgery and to any type of surgery that may be associated with intra-surgical ultrasound imaging. Aspects of the present disclosure support implementing any of the techniques described herein to any medical procedure (e.g., cranial, spinal, thoracic, abdominal, cardiac, ablation, laparoscopic, minimally invasive surgery, robotic surgery, etc.) associated with the use of intra-surgical ultrasound imaging. In some other aspects, the systems and techniques described herein may be implemented in association with initiatives related to data analytics, artificial intelligence, and machine learning, for example, with respect to data analytic scenarios for procedure and device optimization.

In some cases, the techniques described herein may be implemented as a standalone application that uses a calibration phantom (e.g., a water bath, etc.) and an imaging system (e.g., ultrasound imaging, optical or electromagnetic tracking, 3D rendering software, and calibration software) or as an application integrated with an imaging system or navigation system. The examples described herein with reference to the following figures may support multiple types, geometries, configurations, and sizes of calibration phantoms other than the examples illustrated and described herein.

FIG.1Aillustrates an example of a system100that supports aspects of the present disclosure.

The system100includes a computing device102, one or more imaging devices112, a robot114, a navigation system118, a database130, and/or a cloud network134(or other network). Systems according to other implementations of the present disclosure may include more or fewer components than the system100. For example, the system100may omit and/or include additional instances of one or more components of the computing device102, the imaging device(s)112, the robot114, navigation system118, the database130, and/or the cloud network134. In an example, the system100may omit any instance of the computing device102, the imaging device(s)112, the robot114, navigation system118, the database130, and/or the cloud network134. The system100may support the implementation of one or more other aspects of one or more of the methods disclosed herein.

The computing device102includes a processor104, a memory106, a communication interface108, and a user interface110. Computing devices according to other implementations of the present disclosure may include more or fewer components than the computing device102. The computing device102may be, for example, a control device including electronic circuitry associated with controlling any components of the system100.

The processor104of the computing device102may be any processor described herein or any similar processor. The processor104may be configured to execute instructions stored in the memory106, which instructions may cause the processor104to carry out one or more computing steps utilizing or based on data received from the imaging devices112, the robot114, the navigation system118, the database130, and/or the cloud network134.

The memory106may be or include RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer-readable data and/or instructions. The memory106may store information or data associated with completing, for example, any step of the process flow500described herein, or of any other methods. The memory106may store, for example, instructions and/or machine learning models that support one or more functions of the imaging devices112, the robot114, and the navigation system118. For instance, the memory106may store content (e.g., instructions and/or machine learning models) that, when executed by the processor104, enable image processing120, segmentation122, transformation124, and/or registration128. Such content, if provided as in instruction, may, in some implementations, be organized into one or more applications, modules, packages, layers, or engines.

Alternatively or additionally, the memory106may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor104to carry out the various method and features described herein. Thus, although various contents of memory106may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and/or machine learning models. The data, algorithms, and/or instructions may cause the processor104to manipulate data stored in the memory106and/or received from or via the imaging devices112, the robot114, the navigation system118, the database130, and/or the cloud network134.

The computing device102may also include a communication interface108. The communication interface108may be used for receiving data or other information from an external source (e.g., the imaging devices112, the robot114, the navigation system118, the database130, the cloud network134, and/or any other system or component separate from the system100), and/or for transmitting instructions, data (e.g., image data, tracking data, navigation data, calibration data, registration data, etc.), or other information to an external system or device (e.g., another computing device102, the imaging devices112, the robot114, the navigation system118, the database130, the cloud network134, and/or any other system or component not part of the system100). The communication interface108may include one or more wired interfaces (e.g., a USB port, an Ethernet port, a Firewire port) and/or one or more wireless transceivers or interfaces (configured, for example, to transmit and/or receive information via one or more wireless communication protocols such as 802.11a/b/g/n, Bluetooth, NFC, ZigBee, and so forth). In some implementations, the communication interface108may support communication between the device102and one or more other processors104or computing devices102, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

The computing device102may also include one or more user interfaces110. The user interface110may be or include a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and/or any other device for receiving information from a user and/or for providing information to a user. The user interface110may be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Notwithstanding the foregoing, any required input for any step of any method described herein may be generated automatically by the system100(e.g., by the processor104or another component of the system100) or received by the system100from a source external to the system100. In some implementations, the user interface110may support user modification (e.g., by a surgeon, medical personnel, a patient, etc.) of instructions to be executed by the processor104according to one or more implementations of the present disclosure, and/or to user modification or adjustment of a setting of other information displayed on the user interface110or corresponding thereto.

In some implementations, the computing device102may utilize a user interface110that is housed separately from one or more remaining components of the computing device102. In some implementations, the user interface110may be located proximate one or more other components of the computing device102, while in other implementations, the user interface110may be located remotely from one or more other components of the computer device102.

The imaging device112may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and/or other aspects of patient anatomy to yield image data (e.g., image data depicting or corresponding to a bone, veins, tissue, etc.). “Image data” as used herein refers to the data generated or captured by an imaging device112, including in a machine-readable form, a graphical/visual form, and in any other form. In various examples, the image data may include data corresponding to an anatomical feature of a patient, or to a portion thereof. The image data may be or include a preoperative image, an intraoperative image, a postoperative image, or an image taken independently of any surgical procedure. In some implementations, a first imaging device112may be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device112may be used to obtain second image data (e.g., a second image) at a second time after the first time.

The imaging device112may be capable of taking a 2D image or a 3D image to yield the image data. The imaging device112may be or include, for example, an ultrasound scanner (which may include, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device utilizing X-ray-based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermographic camera (e.g., an infrared camera), a radar system (which may include, for example, a transmitter, a receiver, a processor, and one or more antennae), or any other imaging device112suitable for obtaining images of an anatomical feature of a patient. The imaging device112may be contained entirely within a single housing, or may include a transmitter/emitter and a receiver/detector that are in separate housings or are otherwise physically separated.

In some implementations, the imaging device112may include more than one imaging device112. For example, a first imaging device may provide first image data and/or a first image, and a second imaging device may provide second image data and/or a second image. In still other implementations, the same imaging device may be used to provide both the first image data and the second image data, and/or any other image data described herein. The imaging device112may be operable to generate a stream of image data. For example, the imaging device112may be configured to operate with an open shutter, or with a shutter that continuously alternates between open and shut so as to capture successive images. For purposes of the present disclosure, unless specified otherwise, image data may be considered to be continuous and/or provided as an image data stream if the image data represents two or more frames per second.

The robot114may be any surgical robot or surgical robotic system. The robot114may be or include, for example, the Mazor X™ Stealth Edition robotic guidance system. The robot114may be configured to position the imaging device112at one or more precise position(s) and orientation(s), and/or to return the imaging device112to the same position(s) and orientation(s) at a later point in time. The robot114may additionally or alternatively be configured to manipulate a surgical tool (whether based on guidance from the navigation system118or not) to accomplish or to assist with a surgical task. In some implementations, the robot114may be configured to hold and/or manipulate an anatomical element during or in connection with a surgical procedure.

The robot114may include one or more robotic arms116. In some implementations, the robotic arm116may include a first robotic arm and a second robotic arm, though the robot114may include more than two robotic arms. In some implementations, one or more of the robotic arms116may be used to hold and/or maneuver the imaging device112. In implementations where the imaging device112includes two or more physically separate components (e.g., a transmitter and receiver), one robotic arm116may hold one such component, and another robotic arm116may hold another such component. Each robotic arm116may be positionable independently of the other robotic arm. The robotic arms116may be controlled in a single, shared coordinate space, or in separate coordinate spaces.

The robot114, together with the robotic arm116, may have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arm116may be positioned or positionable in any pose, plane, and/or focal point. The pose includes a position and an orientation. As a result, an imaging device112, surgical tool, or other object held by the robot114(or, more specifically, by the robotic arm116) may be precisely positionable in one or more needed and specific positions and orientations.

The robotic arm(s)116may include one or more sensors that enable the processor104(or a processor of the robot114) to determine a precise pose in space of the robotic arm (as well as any object or element held by or secured to the robotic arm).

In some implementations, reference markers (e.g., navigation markers) may be placed on the robot114(including, e.g., on the robotic arm116), the imaging device112, or any other object in the surgical space. The reference markers may be tracked by the navigation system118, and the results of the tracking may be used by the robot114and/or by an operator of the system100or any component thereof. In some implementations, the navigation system118can be used to track other components (e.g., imaging device112, surgical tools, instruments145(later described with reference toFIG.1B), etc.) of the system and the system can operate without the use of the robot114(e.g., with the surgeon manually manipulating the imaging device112and/or one or more surgical tools, based on information and/or instructions generated by the navigation system118, for example).

The navigation system118may provide navigation for a surgeon and/or a surgical robot during an operation. The navigation system118may be any now-known or future-developed navigation system, including, for example, the Medtronic StealthStation™ S8 surgical navigation system or any successor thereof. The navigation system118may include one or more cameras or other sensor(s) for tracking one or more reference markers, navigated trackers (e.g., tracking devices140, etc.) or other objects within the operating room or other room in which some or all of the system100is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some implementations, the navigation system118may include one or more tracking devices140(e.g., electromagnetic sensors, acoustic sensors, etc.).

In some aspects, the navigation system118may include one or more of an optical tracking system, an acoustic tracking system, an electromagnetic tracking system, a radar tracking system, an inertial measurement unit (IMU) based tracking system, and a computer vision based tracking system. The navigation system118may include a corresponding transmission device136capable of transmitting signals associated with the tracking type. In some aspects, the navigation system118may be capable of computer vision based tracking of objects present in images captured by the imaging device(s)112.

In various implementations, the navigation system118may be used to track a position and orientation (e.g., a pose) of the imaging device112, the robot114and/or robotic arm116, and/or one or more surgical tools (e.g., instrument145, etc.) (or, more particularly, to track a pose of a navigated tracker attached, directly or indirectly, in fixed relation to the one or more of the foregoing). In some examples, the instrument145may be an electromagnetic pointer (or stylus). The navigation system118may include a display for displaying one or more images from an external source (e.g., the computing device102, imaging device112, or other source) or for displaying an image and/or video stream from the one or more cameras or other sensors of the navigation system118.

In some implementations, the system100can operate without the use of the navigation system118. The navigation system118may be configured to provide guidance to a surgeon or other user of the system100or a component thereof, to the robot114, or to any other element of the system100regarding, for example, a pose of one or more anatomical elements, whether or not a tool is in the proper trajectory, and/or how to move a tool into the proper trajectory to carry out a surgical task according to a preoperative or other surgical plan.

The processor104may utilize data stored in memory106as a neural network. The neural network may include a machine learning architecture. In some aspects, the neural network may be or include one or more classifiers. In some other aspects, the neural network may be or include any machine learning network such as, for example, a deep learning network, a convolutional neural network, a reconstructive neural network, a generative adversarial neural network, or any other neural network capable of accomplishing functions of the computing device102described herein. Some elements stored in memory106may be described as or referred to as instructions or instruction sets, and some functions of the computing device102may be implemented using machine learning techniques.

For example, the processor104may support machine learning model(s)138which may be trained and/or updated based on data (e.g., training data144) provided or accessed by any of the computing device102, the imaging device112, the robot114, the navigation system118, the database130, and/or the cloud network134. The machine learning model(s)138may be built and updated based on the training data144(also referred to herein as training data and feedback).

The neural network and machine learning model(s)138may support AI/machine learning computer vision algorithms and object detection in association with automatically detecting, identifying, and tracking target objects (e.g., instruments, tools, etc.) in one or more images153or a multimedia file154.

The database130may store information that correlates one coordinate system to another (e.g., one or more robotic coordinate systems, an ultrasound space coordinate system, a patient coordinate system, and/or a navigation coordinate system, etc.). The database130may additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information about a target and/or image information about a patient's anatomy at and/or proximate the surgical site, for use by the robot114, the ultrasound space coordinate system, the navigation system118, and/or a user of the computing device102or of the system100); one or more images153useful in connection with a surgery to be completed by or with the assistance of one or more other components of the system100; and/or any other useful information.

The database130may be configured to provide any such information to the computing device102or to any other device of the system100or external to the system100, whether directly or via the cloud network134. In some implementations, the database130may include information associated with a calibration phantom149associated with a calibration procedure. In some implementations, the database130may be or include part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and/or another system for collecting, storing, managing, and/or transmitting electronic medical records including image data.

In some aspects, the computing device102may communicate with a server(s) and/or a database (e.g., database130) directly or indirectly over a communications network (e.g., the cloud network134). The communications network may include any type of known communication medium or collection of communication media and may use any type of protocols to transport data between endpoints. The communications network may include wired communications technologies, wireless communications technologies, or any combination thereof.

Wired communications technologies may include, for example, Ethernet-based wired local area network (LAN) connections using physical transmission mediums (e.g., coaxial cable, copper cable/wire, fiber-optic cable, etc.). Wireless communications technologies may include, for example, cellular or cellular data connections and protocols (e.g., digital cellular, personal communications service (PCS), cellular digital packet data (CDPD), general packet radio service (GPRS), enhanced data rates for global system for mobile communications (GSM) evolution (EDGE), code division multiple access (CDMA), single-carrier radio transmission technology (1×RTT), evolution-data optimized (EVDO), high speed packet access (HSPA), universal mobile telecommunications service (UMTS), 3G, long term evolution (LTE), 4G, and/or 5G, etc.), Bluetooth®, Bluetooth® low energy, Wi-Fi, radio, satellite, infrared connections, and/or ZigBee® communication protocols.

The Internet is an example of the communications network that constitutes an Internet Protocol (IP) network consisting of multiple computers, computing networks, and other communication devices located in multiple locations, and components in the communications network (e.g., computers, computing networks, communication devices) may be connected through one or more telephone systems and other means. Other examples of the communications network may include, without limitation, a standard Plain Old Telephone System (POTS), an Integrated Services Digital Network (ISDN), the Public Switched Telephone Network (PSTN), a Local Arca Network (LAN), a Wide Area Network (WAN), a wireless LAN (WLAN), a Session Initiation Protocol (SIP) network, a Voice over Internet Protocol (VOIP) network, a cellular network, and any other type of packet-switched or circuit-switched network known in the art. In some cases, the communications network may include of any combination of networks or network types. In some aspects, the communications network may include any combination of communication mediums such as coaxial cable, copper cable/wire, fiber-optic cable, or antennas for communicating data (e.g., transmitting/receiving data).

The computing device102may be connected to the cloud network134via the communication interface108, using a wired connection, a wireless connection, or both. In some implementations, the computing device102may communicate with the database130and/or an external device (e.g., a computing device) via the cloud network134.

The system100or similar systems may be used, for example, to carry out one or more aspects of any of the process flow500described herein. The system100or similar systems may also be used for other purposes.

FIG.1Billustrates an example of the system100that supports aspects of the present disclosure. Aspects of the example may be implemented by the computing device102, imaging device(s)112, robot114(e.g., a robotic system), and navigation system118.

In some aspects, the navigation system118may provide navigation information based on an electromagnetic field generated by a transmission device136. The navigation information may include tracking information167(also referred to herein as tracking data) as described herein. For example, the transmission device136may include an array of transmission coils capable of generating or forming the electromagnetic field in response to respective currents driven through the transmission coils. The navigation system118may include tracking devices140capable of sensing the electromagnetic field. Aspects of the navigation system118described herein may be implemented by navigation processing129.

The system100may support tracking objects (e.g., an instrument145, imaging device112, etc.) in a trackable volume150using an electromagnetic field produced by the transmission device136. For example, the transmission device136may include a transmitter antenna or transmitting coil array capable of producing the electromagnetic field. The system100may track the pose (e.g., position, coordinates, orientation, etc.) of the objects in the tracking volume150relative to a subject141. In some aspects, the system100may display, via a user interface of the computing device102, icons corresponding to any tracked objects. For example, the system100may superimpose such icons on and/or adjacent an image displayed on the user interface. The terms “tracking volume,” “trackable volume,” “navigation volume,” and “volume” may be used interchangeably herein.

In some aspects, the transmission device136may be an electromagnetic localizer that is operable to generate electromagnetic fields. The transmission device136may drive current through the transmission coils, thereby powering the coils to generate or form the electromagnetic field. As the current is driven through the coils, the electromagnetic field will extend away from the transmission coils and form a navigation domain (e.g., volume150). The volume150may include any portion (e.g., the spine, one or more vertebrae, the brain, an anatomical element, or a portion thereof, etc.) of the subject141and/or any portion of a calibration phantom149-a. The transmission coils may be powered through a controller device and/or power supply provided by the system100.

The tracking devices140may include or be provided as sensors (also referred to herein as tracking sensors). The sensors may sense a selected portion or component of the electromagnetic field(s) generated by the transmission device136. The navigation system118may support registration (e.g., through registration128) of the volume150to a virtual space155. The navigation system118may support superimposing an icon representing a tracked object (e.g., an instrument145, a tracking device140-b, a tracking device146, etc.) on the image. The system100may support the delivery of tracking information associated with the tracking devices140and/or tracking device146to the navigation system118. The tracking information may include, for example, data associated with magnetic fields sensed by the tracking devices140.

The tracking devices140may communicate sensor information to the navigation system118for determining a position of the tracked portions relative to each other and/or for localizing an object (e.g., instrument145, tracking device146, etc.) relative to an image153. The navigation system118and/or transmission device136may include a controller that supports operating and powering the generation of electromagnetic fields.

In the example ofFIG.1B, the system100may generate a navigation space119based on one or more tracking signals transmitted by the transmission device136. The navigation space119may correspond to environment142or a portion thereof. For example, the navigation space119may correspond to a subject141(e.g., a patient) included in the environment142or an anatomical element (e.g., an organ, bone, tissue, etc.) of the subject141. The environment142may be, for example, an operating room, an exam room, or the like. The tracking signals are not limited to electromagnetic tracking signals, and it is to be understood that the example aspects described with reference toFIG.1Bmay be implemented using other types of tracking signals (e.g., optical tracking signals, acoustic tracking signals, etc.).

The system100may generate a virtual space155based on (e.g., in response to) signals transmitted by imaging device112. The virtual space155may correspond to a field of view159of the imaging device112. In an example, the system100may generate images153in response to signals transmitted by imaging device112, and the images153may correspond to the field of view159of the imaging device112. In the example ofFIG.1B, the imaging device112is an ultrasound probe transmitting ultrasound signals, and the images153may be ultrasound images.

The images153may be static images or video images. In some aspects, the images153may be stored as a multimedia file154that includes video (or video and sound). The imaging device112and the example signals transmitted by the imaging device112are not limited thereto, and it is to be understood that the example aspects described with reference toFIG.1Bmay be implemented using other types of imaging devices112(e.g., X-ray, CT scanner, OCT scanner, etc.) and imaging systems described herein. The system100may support acquiring image data to generate or produce images (e.g., images153, multimedia file154, etc.) of the subject141.

In an example, using the imaging device112and the transmission device136, the system100may detect or track the calibration phantom149-aand other objects (e.g., tracking devices140, instruments145, tracking device146, etc.) included in the volume150and the virtual space155. For example, at least a portion of the calibration phantom149-amay be located in the volume150(as generated by the navigation system118) and the virtual space155(as generated by the computing device102). In the example ofFIG.1B, the calibration phantom149-ais a tissue phantom, but is not limited thereto.

The system100may register and calibrate the imaging device112with respect to the navigation system118. In the example ofFIG.1B, for an image153in which the calibration phantom149-ais detected, the system100may identify a set of coordinates157in the virtual space155in response to an event in which the system100detects at least a portion of the instrument145in the image153. In an example, the system100may detect that the portion of the instrument145is located in the calibration phantom149-aand intersects a surface of the virtual space155at the set of coordinates157. For example, the system100may detect that the portion of the instrument145intersects the surface at an angle perpendicular to the surface. In some examples, the portion of the instrument145may be a tracking device146(e.g., an electromagnetic antenna of the instrument145).

In some example implementations, the virtual space155may be a 2D virtual space generated based on 2D images (e.g., ultrasound images, CT images, etc.) captured by the imaging device112, and the surface may be a plane of the virtual space155. In some other example implementations, the virtual space155may be a 3D virtual space (e.g., a volume), and the surface of the virtual space155may be a planar surface or non-planar surface.

According to example aspects of the present disclosure, the system100may calibrate the virtual space155to the navigation space119in response to the event in which the system100detects the instrument145(or at least a portion of the instrument145) in the image153. For example, in response to the event, the system100may calibrate a coordinate system160associated with the virtual space155with respect to a coordinate system165associated with the navigation space119. In an example, the system100may calibrate the coordinate system160with respect to the coordinate system165based on the set of coordinates157and temporal information156associated with the event in which the instrument145(or portion of the instrument145) intersects a surface (e.g., a plane, a volume, etc.) of the virtual space155at the set of coordinates157. Further, for example, the system100may calibrate the coordinate system160with respect to the coordinate system165based on tracking information associated with the tracking device146, the tracking device140-b, and the tracking device140-a.

For example, the instrument145may be registered to the navigation system118such that the navigation system118may track and determine pose information161of the instrument145based on tracking information167associated with the tracking device146, the tracking device140-a, and/or the tracking device140-band temporal information166corresponding to the tracking information167. Further, for example, the navigation system118may track and determine pose information161of the imaging device112based on tracking information167associated with the tracking device140-aand temporal information166corresponding to the tracking information167. Accordingly, for example, the system100may calibrate the coordinate system160with respect to the coordinate system165based on the tracking information167(e.g., associated with the tracking device146, the tracking device140-b, and the tracking device140-a), the temporal information166associated with the tracking information167, the temporal information156associated with the event, and the coordinates157associated with the event.

The system100may detect, in an image153(or multimedia file154), one or more landmarks corresponding to the instrument145, a portion of the instrument145, or the tracking device146. The landmarks may correspond to distinctive features (e.g., a tip, a shape of the tip, etc.) of the instrument145or the tracking device146. The system100may calibrate the coordinate system160with respect to the coordinate system165based on the one or more landmarks.

Aspects of the present disclosure support calibrating the virtual space155to the navigation space119(e.g., calibrating the coordinate system160associated with the virtual space155to the coordinate system165associated with the navigation space119) in response to one or more criteria. For example, the system100may calibrate the virtual space155to the navigation space119in response to each occurrence of the event. In another example, the system100may calibrate the virtual space155to the navigation space119in response to each nthoccurrence of the event (e.g., each third occurrence of the event, each fifth occurrence, etc.). In some other examples, the system100may calibrate the virtual space155to the navigation space119in response to each nthoccurrence of the event within a temporal duration (e.g., each third occurrence of the event, in which the third occurrence is X seconds or less after a first occurrence of the event (where X is an integer value)). Accordingly, for example, aspects of the present disclosure support automatic registration during a surgical procedure, in which the registration is continuous or semi-continuous.

The system100may support calibrating the coordinate system160with respect to the coordinate system165without pausing a medical procedure (e.g., surgical procedure). In the example in which the calibration phantom149-ais a tissue phantom inside the body of the subject141, the system100may calibrate the coordinate system160with reference to the coordinate system165(e.g., recalibrate the registration between the virtual space155and the navigation space119) in the background while medical personnel performs a medical procedure on the subject141, without interrupting the medical procedure. That is, for example, the system100may calibrate the coordinate system160with reference to the coordinate system165during the medical procedure, without prompting the medical personnel to pause the medical procedure, such that the medical personnel may proceed with the medical procedure without waiting for calibration to be completed. In some aspects, the system100may calibrate the coordinate system160with reference to the coordinate system165without prompting the medical personnel to participate in a separate calibration operation.

In some aspects, the system100and techniques described herein may support calibrating the coordinate system160with respect to the coordinate system165based on any of: properties (e.g., beam thickness, beam shape, signal frequency, etc.) of signals transmitted by the imaging device112, pose information of the instrument145(or pose information of the tracking device146) in association with an intersection between the instrument145(or tracking device146) and the surface of the virtual space155, and properties (e.g., shape, etc.) of the tracking device146, example aspects of which are later described with reference toFIG.3.

Aspects of calibrating the virtual space155to the navigation space119(e.g., calibrating the coordinate system160associated with the virtual space155to the coordinate system165associated with the navigation space119) include verifying a registration accuracy between the coordinate system160and the coordinate system165. For example, in response to an occurrence of the event as described herein, the system100may calculate a registration accuracy between the coordinate system160and the coordinate system165and compare the registration accuracy to a target accuracy value. In an example, in response to a comparison result in which the registration accuracy is less than the target accuracy value, the system100may perform one or more operations described herein in association with recalibrating the virtual space155to the navigation space119. For example, the system100may autonomously recalibrate the coordinate system160to the coordinate system165. In another example, the system100may generate and output a notification including user guidance information175(e.g., tutorials, user prompts, corrective actions, real-time actional feedback, etc.) regarding how to move or position a device (e.g., the imaging device112, the instrument145, etc.) in association with the calibration process. In some examples, the notification may include a visual notification, an audible notification, a haptic notification, or a combination thereof.

Other additional and/or alternative aspects of calibrating the virtual space155to the navigation space119include automatically detecting distortion in the navigated volume150due to discrepancies between navigation data (e.g., tracking information167provided by navigation system118) and imaging data (e.g., images153, multimedia file154, etc.). In an example, the discrepancies may be between pose information161of a tracked object (e.g., tracking device140-b, instrument145, tracking device146, etc.) as indicated by the navigation system118and pose information162of the tracked object as determined by the computing device102from the imaging data.

In an example implementation, in response to an occurrence of the event as described herein, the system100may calculate the discrepancy and compare the discrepancy to a target discrepancy threshold value. In an example, in response to a comparison result in which the discrepancy is greater than the discrepancy threshold value, the system100may perform one or more operations described herein (e.g., autonomous recalibration, outputting a notification including user guidance information175, etc.) in association with recalibrating the virtual space155to the navigation space119. In some aspects, the system100may calibrate the navigation data to the imaging data (e.g., calibrate the navigation space119to the virtual space155) while compensating for the discrepancies.

In an example, the system100may identify that tracking device146(e.g., electromagnetic antenna of an instrument145) is intersecting the plane of an ultrasound imaging field158at a point147of intersection inside a circle148, example aspects of which will later be described with reference toFIG.2B.

The techniques described herein may provide continuous automatic registration, continuous semi-automatic registration, or a combination thereof, and the registration techniques may be implemented during a medical procedure.

FIG.1Cillustrates an example of the system100that supports aspects of the present disclosure. Aspects of the example inFIG.1Cinclude like aspects described with reference toFIG.1B. Referring to the example ofFIG.1C, the calibration phantom149-bmay be an ultrasound transmitting volume (e.g., a water bath) implemented using an empty container (e.g., an empty box including an electromagnetic friendly material). In some aspects, the calibration phantom149-bmay include ultrasound conductive material.

For example, the container may be formed of low magnetic or non-magnetic materials so as to minimize distortion to electromagnetic fields. In an example, the container may be formed of a material having a magnetic permeability of about 1.0 to about 1.1 (relative), and the material may have a relatively low electrical conductivity (e.g., an electrical conductivity less than a threshold value). In some examples, the material may be a stainless steel alloyed with different metallic elements associated with obtaining specific properties (e.g., temperature and corrosion resistance, fracture tolerance, etc.). Non-limiting examples of the material include Nickel/Chromium alloys (e.g., Series 300 alloys, type 304 stainless steel (annealed condition only), type 316 stainless steel), Cobalt/Chromium alloys (e.g., L605, MP35N), and Titanium alloys (e.g., Ti6Al4V), plastics, and wood.

In an example, one or more surfaces of the container may include cross wires with patterns, and the container is full of water.

In some examples, the calibration phantom149-bmay be integrated into the structure of a patient tracker. In some other aspects, the calibration phantom149-bmay be included in the environment142as a standalone structure that is separate from an operating table associated with the subject141. Example aspects of the water bath are later described with reference toFIG.2A.

According to example aspects of the present disclosure, the system100may support calibrating the virtual space155to the navigation space119using the calibration phantom149-band the techniques as described herein, in which the calibration phantom149-bis substituted for the calibration phantom149-adescribed with reference toFIG.1B.

In some example implementations, the system100may support calibrating the virtual space155to the navigation space119using both the calibration phantom149-aand the calibration phantom149-b. For example, the system100may support calibration outside the subject141using the calibration phantom149-b(e.g., water bath) and further calibration (e.g., recalibration, calibration adjustment, etc.) using the calibration phantom149-a(e.g., tissue of the subject141), and the combination may provide an increase in accuracy compared to other calibration techniques. An example implementation of using both the calibration phantom149-aand the calibration phantom149-bin association with a calibration process is later described with reference toFIG.4.

FIG.2Aillustrates an example implementation200of the system100. Referring toFIG.2A, an imaging device112(e.g., an electromagnetic tracked ultrasound probe) may be inserted in calibration phantom149-b(e.g., a water bath). A transmission device (e.g., an electromagnetic emitter) may be positioned inside or within a threshold distance of the calibration phantom149-b. In the example view ofFIG.2A, the transmission device is positioned outside of (e.g., behind) the calibration phantom149-b.

A tracking device146may be positioned in an ultrasound imaging field158(later illustrated atFIG.2B) associated with the imaging device112at multiple points by moving the tracking device146through the ultrasound imaging field. In the example implementation200, the ultrasound imaging field158corresponds to the field of view159of the imaging device112. A navigation space119corresponding to the calibration phantom149-bmay be generated by the navigation system118as described herein, and the system100may display a virtual representation201of the navigation space119and the virtual space155via, for example, a user interface110. The tracking device146may be referred to as a navigated instrument. In the example ofFIG.2A, the tracking device146is a navigation pointer (e.g., stylus).

Example aspects of the virtual representation201of the navigation space119and the virtual space155are later described with reference toFIG.2B.

FIG.2Billustrates an example of the virtual representation201of the navigation space119and the virtual space155.

The virtual representation201may include a multi-dimensional representation corresponding to the volume of the calibration phantom149-b. The virtual representation201may include the imaging device112, the instrument145, the instrument145(or portions of the instrument145), and/or tracking device146. In some aspects, the virtual representation201may include a pattern (e.g., represented by lines202and/or dots) that correspond to the patterns described with reference to the container used for implementing the calibration phantom149-b.

Features of the virtual representation201may be described in conjunction with a coordinate systems203(e.g., coordinate system203-a, coordinate system203-b). Each coordinate system203, as shown inFIG.2B, includes three-dimensions including an X-axis, a Y-axis, and a Z-axis. Additionally or alternatively, coordinate system203-amay be used to define surfaces, planes, or volume of the calibration phantom149-band/or the navigation space119. The virtual representation201may include coordinate system203-bthat corresponds to the imaging device112.

The planes of each coordinate system203(e.g., coordinate system203-a, coordinate system203-b) may be disposed orthogonal, or at 90 degrees, to one another. While the origin of a coordinate system203may be placed at any point on or near the components of the navigation system118, for the purposes of description, the axes of the coordinate system203are always disposed along the same directions from figure to figure, whether the coordinate system203is shown or not. In some examples, reference may be made to dimensions, angles, directions, relative positions, and/or movements associated with one or more components of the imaging device112and/or the navigation system118with respect to a coordinate system203.

Referring to the virtual representation201illustrated inFIG.2B, a tracking device146(e.g., electromagnetic antenna of an instrument145) is intersecting the plane of the ultrasound imaging field158at a point147of intersection inside a circle148. In an example, the point147may be an echogenic dot. In some aspects, the virtual representation201may include a window204displaying the imaging field158and information (e.g., point147, circle148, etc.) associated with the imaging field158. The system100may display guidance information175indicating pose information of the tracking device146with respect to the ultrasound imaging field158. In some aspects, the user guidance information175may include an indication whether the tracking device146is outside the plane of the ultrasound imaging field158(e.g., ‘Out of Plane’) or intersecting the plane of the ultrasound imaging field158(e.g., at a point147). In some aspects, the guidance information175may include distance information (e.g., ‘Tip to Plane: 1.69 cm’) of the tracking device146with respect to the ultrasound imaging field158.

It is to be understood that the aspects described herein with reference to the plane of the ultrasound imaging field158support implementations applied to any surface of a virtual space155(e.g., a plane of the virtual space155for cases in which the virtual space155is a 2D virtual space, a planar surface or non-planar surface of the virtual space155for cases in which the virtual space155is a 3D virtual space, etc.). It is to be understood that the aspects described herein may be applied to an electromagnetic antenna, a navigation stylus, a pointer, or any navigated tools having a geometry and location that is defined, known, and trusted by the system100.

The system100may record all navigation data (e.g., electromagnetic data), imaging data (e.g., ultrasound data), and corresponding temporal information (e.g., temporal information156, temporal information166) in a multimedia file154. In an example, the multimedia file154may be a movie file, and the system100may record timestamps corresponding to the navigation data and the imaging data. Based on the navigation data, the imaging data, and the temporal information, the system100may identify when the tip of the tracking device146enters the ultrasound field of view159and intersects the plane of the ultrasound imaging field158. Based on the identification of when the tip of the tracking device146enters the ultrasound field of view159and intersects the plane of the ultrasound imaging field158, the system100may verify the calibration of the imaging device112with the electromagnetic navigation of the navigation system118.

FIG.3Aillustrates example views300and301of an ultrasound beam305transmitted by the imaging device112when viewed from different perspective views. Referring to the example view300, the ultrasound beam305is relatively thick (or wide) with respect to the Y-axis. In contrast, referring to the example view301, the ultrasound beam305is relatively narrow with respect to the Z-axis. In the example views300and301of the ultrasound beam305, thickness varies in depth, and the shape and focal point of the ultrasound beam305may be based on parameters (e.g., power, frequency, etc.) of the ultrasound beam305.

According to example aspects of the present disclosure, referring to example view300, the system100may calibrate the virtual space155to the navigation space119based on instances in which the instrument145(or tracking device146) intersects a cross-sectional area310(e.g., an area in the XY plane) of the ultrasound beam305in a direction along the Z axis. That is, for example, the system100may perform the calibration for instances in which the instrument145or tracking device146intersects a portion of the cross-sectional area310(e.g., the length of the instrument145or tracking device146is perpendicular (or near perpendicular) the cross-sectional area310), while factoring in values of parameters (e.g., thickness, depth, shape, focal point, etc.) of the ultrasound beam305.

In another example, referring to example view301, the system100may calibrate the virtual space155to the navigation space119for instances in which the instrument145(or tracking device146) intersects the ultrasound beam305in the direction along the Z axis. That is, for example, the system100may perform the calibration for instances in which the instrument145or tracking device146intersects the ultrasound beam305, while incorporating values of parameters (e.g., thickness, depth, shape, focal point, etc.) of the ultrasound beam305.

FIG.3Billustrates example elements320(e.g., element320-a, element320-b) that may be implemented at instrument145(e.g., at a tip325of the instrument145, at a tip325of tracking device146, etc.) in association with calibrating the virtual space155to the navigation space119. In an example implementation, element320-amay be a gel that is cylinder shaped. In another example implementation, element320-bmay be a gel that is sphere shaped.

FIG.3Cillustrates example shapes330(e.g., shapes330-athrough330-c) that may be implemented at a tip325of the instrument145, at a tip325of tracking device146, or the like. Aspects of the present disclosure may include implementing any of the shapes330, and the shapes330may support tip-image center alignment. That is, for example, each shape330may be symmetrical with respect to a center of the shape330, which may support alignment of the center of the shape330and a center335of an ultrasound beam305emitted by the imaging device112, an example of which is illustrated atFIG.3D.

FIG.4illustrates an example of a process flow400in accordance with aspects of the present disclosure. In some examples, process flow400may implement aspects of a computing device102, an imaging device112, a robot114, and a navigation system118described with reference toFIGS.1through3.

In the following description of the process flow400, the operations may be performed in a different order than the order shown, or the operations may be performed in different orders or at different times. Certain operations may also be left out of the process flow400, or other operations may be added to the process flow400.

It is to be understood that any of the operations of process flow400may be performed by any device (e.g., a computing device102, an imaging device112, a robot114, navigation system118, etc.) of the system100described herein.

At405, the system100may generate a navigation space119as described herein. In the example of the process flow400.

At410, the system100may generate a virtual space155based on images captured by the imaging device112as described herein. For example, the system100may generate the virtual space155based on images representing the inside of calibration phantom149-b(e.g., water bath).

At420, the system100may initiate a calibration process401in accordance with aspects of the present disclosure. For example, at420, the system100may initiate calibration of the coordinate system160associated with the virtual space155(and imaging device112) with respect to the coordinate system165associated with the navigation space119(and navigation system118).

At425, the system100may provide user guidance information175(e.g., tutorials, user prompts, corrective actions, real-time actional feedback, etc.) described herein regarding how to move or position a device (e.g., the imaging device112, the instrument145, etc.) in association with the calibration process. The terms “guidance information” and “calibration guidance information” may be used interchangeably herein. Aspects of the present disclosure support implementations with or without providing user guidance information175.

At430(Event Detected?= ‘Yes’), the system100may determine, from an image153(or multimedia file154), whether an event has occurred in which the instrument145(or portion of the instrument145) intersects a surface (e.g., a plane, a volume, etc.) of the virtual space155. In another example, at430(Event Detected?= ‘No’), the system100may analyze subsequent images153(or multimedia files154) until the system100detects an event in which the instrument145(or portion of the instrument145) intersects a surface (e.g., a plane, a volume, etc.) of the virtual space155. In some aspects, the system100may return to425and provide additional user guidance information175that prompts a user to position or orient the imaging device112and/or the instrument145to trigger such an event.

At435, the system100may identify, from the image153(or multimedia file154) the set of coordinates157at which the instrument145(or portion of the instrument145) intersects a surface (e.g., a plane, a volume, etc.) of the virtual space155. In some aspects, at435, the system100may identify the temporal information156associated with when the instrument145intersected the surface of the virtual space155. In some cases, the system100may identify pose information162(in the virtual space155) of the instrument145that corresponds to the temporal information156.

At440, the system100may calibrate the coordinate system160with respect to the coordinate system165based on the set of coordinates157and the temporal information156as described herein. In an example, the system100may calibrate the coordinate system160with respect to the coordinate system165based on the set of coordinates157, the temporal information156, the pose information161(of the instrument145in the virtual space155) corresponding to the temporal information156, and pose information162(of the instrument145in the navigation space119) corresponding to the temporal information156.

At455through457, the system100may determine whether to repeat the calibration process401.

For example, at455, the system100may determine whether a user input requesting recalibration has been received. In response to receiving a user input requesting recalibration, the system100may determine that recalibration is to be performed (e.g., Recalibrate Based on User Input?= ‘Yes’).

In another example, at456, the system100may determine whether a temporal duration (e.g., recalibration every X hours, every day, etc.) associated with performing recalibration has elapsed. In response to identifying that the temporal duration has elapsed, the system100may determine that recalibration is to be performed (e.g., Recalibrate Based on Temporal Duration?= ‘Yes’).

In some other examples, at457, the system100may detect for any losses in calibration between the imaging device112and the navigation system118(e.g., the navigation system118is unable to track the imaging device112) has occurred. In response to detecting a loss in calibration between the imaging device112and the navigation system118(e.g., the navigation system118is unable to track the imaging device112), the system100may determine that recalibration is to be performed (e.g., Recalibrate Based on Calibration Loss?= ‘Yes’).

According to example aspects of the present disclosure, based on decisions by the system100at any of455through457, the system100may repeat the calibration process401, beginning at any operation (e.g., generating the virtual space155at410, initiating calibration at420, etc.) of the calibration process401. In an example, in response to a ‘Yes’ decision at any of455through457, the system100may return to410and generate the virtual space155, but while navigating the calibration phantom149-a(e.g., tissue phantom) with the imaging device112. For example, in repeating the calibration process401, the system100may regenerate the virtual space155based on images captured by the imaging device112as described herein, but the images may be associated with or include calibration phantom149-a(e.g., tissue phantom).

In an example implementation, after repeating the calibration process401, the system100may again return to410in response to a ‘Yes’ decision at any of455through457and generate the virtual space155, while imaging the calibration phantom149-a(e.g., tissue phantom) with the imaging device112.

In an alternative or additional example, in response to a ‘No” decision at any of455through457, the system100may continue to provide navigation information (e.g., tracking information167, etc.).

While providing imaging information (e.g., images153, etc.) and navigation information (e.g., tracking information167, etc.), the system100may monitor for one or more events in which the instrument145(or portion of the instrument145) intersects a surface (e.g., a plane, a volume, etc.) of the virtual space155as described herein. In an example, at460, the system100may detect an event in which the instrument145(or portion of the instrument145) intersects a surface of the virtual space155(Event Detected?= ‘Yes’).

At465, the system100may determine, from the event detected at460, whether recalibration is to be performed.

In an example implementation, the system100may detect the amount of distortion in the navigated volume150(e.g., discrepancies between navigation data associated with the instrument145and imaging data associated with the instrument145) based on the event detected at460. Based on the amount of distortion detected by the system100, the system100may return to435(e.g., for recalibration) or refrain from returning to435(e.g., abstain from performing recalibration).

For example, the system100may determine that the amount of distortion is greater than a threshold distortion value, and at465(e.g., Recalibrate?= ‘Yes’), the system100may return to435and repeat the calibration as described with reference to440. In another example, the system100may determine that the amount of distortion is less than the threshold distortion value, and at465(e.g., Recalibrate?= ‘No’), and the system100may continue to provide imaging information and/or navigation information (e.g., tracking information167, etc.) while monitoring for any of the events described with reference to455through460.

As supported by aspects of the present disclosure, the system100may determine (at465) whether to perform recalibration, at any occurrence of an event detected at460. For example, the system100may perform recalibration at each occurrence of an event detected at460, at each nthoccurrence of the event, or at each nthoccurrence of the event within a temporal duration.

As illustrated and described herein, the example aspects of the process flow400described herein support automatic and continuous (or semi-continuous) recalibration by the system100.

FIG.5illustrates an example of a process flow500in accordance with aspects of the present disclosure. In some examples, process flow500may implement aspects of a computing device102, an imaging device112, a robot114, and a navigation system118described with reference toFIGS.1through3.

In the following description of the process flow500, the operations may be performed in a different order than the order shown, or the operations may be performed in different orders or at different times. Certain operations may also be left out of the process flow500, or other operations may be added to the process flow500.

It is to be understood that any of the operations of process flow500may be performed by any device (e.g., a computing device102, an imaging device112, a robot114, navigation system118, etc.) of the system100described herein.

At505, the process flow500may include generating a navigation space based on one or more tracking signals.

At510, the process flow500may include generating a virtual space including at least a portion of a calibration phantom based on one or more images, wherein the one or more images are generated in response to transmitting one or more imaging signals.

In some aspects, the calibration phantom includes an ultrasound conductive material. For example, the calibration phantom may include an ultrasound transmitting volume (e.g., water bath). In some aspects, the calibration phantom includes a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

In some aspects, the virtual space corresponds to a field of view of the imaging device.

At515, the process flow500may include identifying a set of coordinates in the virtual space in response to an event in which at least a portion of a tracked device in the calibration phantom is detected in the one or more images.

In some aspects, the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a plane of the virtual space at the set of coordinates.

In some aspects, the navigation space and the tracked device are associated with at least one of: an optical tracking system, an acoustic tracking system, an electromagnetic tracking system, a magnetic tracking system, a radar tracking system, an inertial measurement unit (IMU) based tracking system, and a computer vision based tracking system.

At520, the process flow500may include calibrating a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

In some aspects, calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event. In some aspects, calibrating the first coordinate system with respect to the second coordinate system is absent pausing the surgical procedure.

In some aspects, calibrating the first coordinate system with respect to the second coordinate system is based on: beam thickness (e.g., ultrasound beam thickness), beam shape (e.g., ultrasound beam shape), or both of the one or more signals transmitted by the imaging device; pose information of the portion of the tracked device in association with an intersection between the portion of the tracked device and a plane of the virtual space; and one or more properties of the portion of the tracked device.

In some aspects, the tracked device is comprised in at least a portion of an instrument, and the process flow500may include detecting, in the one or more images, one or more landmarks corresponding to at least a portion of the tracked device, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the one or more landmarks.

In some aspects, calibrating the first coordinate system with respect to the second coordinate system includes verifying a registration accuracy between the first coordinate system and the second coordinate system.

At525, the process flow500may include outputting guidance information associated with positioning the imaging device, the tracked device, or both in association with calibrating the first coordinate system with respect to the second coordinate system.

In some aspects, the process flow500may include detecting one or more discrepancies between first tracking data corresponding to the tracked device in association with the navigation space and second tracking data corresponding to the tracked device in association with the virtual space. In some aspects, the process flow500may include generating a notification associated with the one or more discrepancies, performing one or more operations associated with compensating for the one or more discrepancies, or both.

The process flow500(and/or one or more operations thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s)104of the computing device102described above. The at least one processor may be part of a robot (such as a robot114) or part of a navigation system (such as a navigation system118). A processor other than any processor described herein may also be used to execute the process flow500. The at least one processor may perform operations of the process flow500by executing elements stored in a memory such as the memory106. The elements stored in memory and executed by the processor may cause the processor to execute one or more operations of a function as shown in the process flow500. One or more portions of the process flow500may be performed by the processor executing any of the contents of memory, such as image processing120, a segmentation122, a transformation124, and/or a registration128.

As noted above, the present disclosure encompasses methods with fewer than all of the steps identified herein (and the corresponding description of respective process flows), as well as methods that include additional steps beyond those identified in the figures and process flows described herein). The present disclosure also encompasses methods that include one or more steps from one method described herein, and one or more steps from another method described herein. Any correlation described herein may be or include a registration or any other correlation.

The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, implementations, and/or configurations for the purpose of streamlining the disclosure. The features of the aspects, implementations, and/or configurations of the disclosure may be combined in alternate aspects, implementations, and/or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, implementation, and/or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred implementation of the disclosure.

Example aspects of the present disclosure include:

A system including: a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: generate a navigation space based on one or more tracking signals; generate a virtual space including at least a portion of a calibration phantom based on one or more images, wherein the one or more images are generated in response to one or more signals transmitted by an imaging device; identify a set of coordinates in the virtual space in response to an event in which at least a portion of a tracked device in the calibration phantom is detected in the one or more images; and calibrate a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

Any of the aspects herein, wherein the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a surface of the virtual space at the set of coordinates.

Any of the aspects herein, wherein: calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event; and calibrating the first coordinate system with respect to the second coordinate system is absent pausing a surgical procedure.

Any of the aspects herein, wherein calibrating the first coordinate system with respect to the second coordinate system is based on: beam thickness, beam shape, or both of the one or more signals transmitted by the imaging device; pose information of the portion of the tracked device in association with an intersection between the portion of the tracked device and a plane of the virtual space; and one or more properties of the portion of the tracked device.

Any of the aspects herein, wherein the calibration phantom includes: a water bath; or a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

Any of the aspects herein, wherein the instructions are further executable by the processor to: output guidance information associated with positioning the imaging device, the tracked device, or both in association with calibrating the first coordinate system with respect to the second coordinate system.

Any of the aspects herein, wherein the tracked device is included in at least a portion of an instrument, and the instructions are further executable by the processor to: detect, in the one or more images, one or more landmarks corresponding to at least a portion of the tracked device, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the one or more landmarks.

Any of the aspects herein, wherein calibrating the first coordinate system with respect to the second coordinate system includes verifying a registration accuracy between the first coordinate system and the second coordinate system.

Any of the aspects herein, wherein the instructions are further executable by the processor to: detect one or more discrepancies between first tracking data corresponding to the tracked device in association with the navigation space and second tracking data corresponding to the tracked device in association with the virtual space; and generate a notification associated with the one or more discrepancies, perform one or more operations associated with compensating for the one or more discrepancies, or both.

Any of the aspects herein, wherein the virtual space corresponds to a field of view of the imaging device.

Any of the aspects herein, wherein the navigation space and the tracked device are associated with at least one of: an optical tracking system, an acoustic tracking system, an electromagnetic tracking system, a radar tracking system, an inertial measurement unit (IMU) based tracking system, and a computer vision based tracking system.

An imaging system including an imaging device; a tracking system including: a transmission device; and a tracked device; a calibration phantom; a processor; and a memory storing data that, when processed by the processor, cause the processor to: generate a navigation space based on one or more tracking signals emitted by the transmission device; generate a virtual space including at least a portion of the calibration phantom based on one or more images generated by the imaging system, wherein the one or more images are generated in response to one or more signals transmitted by the imaging device; identify a set of coordinates in the virtual space in response to an event in which at least a portion of the tracked device is detected in the one or more images; and calibrate a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

Any of the aspects herein, wherein the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a surface of the virtual space at the set of coordinates.

Any of the aspects herein, wherein: calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event; and calibrating the first coordinate system with respect to the second coordinate system is absent pausing a surgical procedure.

Any of the aspects herein, wherein the calibration phantom includes: a water bath; or a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

Any of the aspects herein, wherein calibrating the first coordinate system with respect to the second coordinate system is based on: beam thickness, beam shape, or both of the one or more signals transmitted by the imaging device; pose information of the portion of the tracked device in association with an intersection between the portion of the tracked device and a plane of the virtual space; and one or more properties of the portion of the tracked device.

A method including: generating a navigation space based on one or more tracking signals; generating a virtual space including at least a portion of a calibration phantom based on one or more images, wherein the one or more images are generated in response to transmitting one or more imaging signals; identifying a set of coordinates in the virtual space in response to an event in which at least a portion of a tracked device in the calibration phantom is detected in the one or more images; and calibrating a first coordinate system associated with the virtual space with respect to a second coordinate system associated with the navigation space in response to the event, wherein calibrating the first coordinate system with respect to the second coordinate system is based on the set of coordinates and temporal information associated with the event.

Any of the aspects herein, wherein the set of coordinates are identified in the virtual space in response to at least the portion of the tracked device intersecting a plane of the virtual space at the set of coordinates.

Any of the aspects herein, wherein: calibrating the first coordinate system with respect to the second coordinate system is in response to one or more occurrences of the event; and calibrating the first coordinate system with respect to the second coordinate system is absent pausing a surgical procedure.

Any of the aspects herein, wherein the calibration phantom includes: a water bath; or a tissue phantom included in a body of a subject, wherein the tissue phantom and the subject are associated with a surgical procedure.

Any one or more of the features disclosed herein.

Any one of the aspects/features/implementations in combination with any one or more other aspects/features/implementations.