Patent ID: 12207888

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

An Electromagnetic Tracking (EMT) system (also called a magnetic tracking system) can be used in various environments, such as medical settings, to track an object (e.g., a tracked object). For example, in a surgical setting, the EMT system can be used to track medical equipment (e.g., a surgical tool) for one or more purposes (e.g., endoscopic surgery), thereby allowing the three-dimensional position (e.g., location) and the orientation of the object to be known to a medical professional (e.g., a surgeon) during a medical procedure. Generally, the magnetic tracking system is configured to track objects inside a body to assist the medical professional with an operation performed by the medical professional.

FIG.1Ashows a diagram of an environment including an example of a magnetic tracking system100for calibrating the magnetic tracking system using a calibration device140.FIG.1Bshows a diagram of an environment including an example of a magnetic tracking system100for tracking a tracked object142. The magnetic tracking system100includes a magnetic signal detector130(also called receiver130) coupled to a non-magnetic signal detector110(such as a camera) in a calibration device140. The magnetic tracking system100is configured to determine a location of one or more objects142(also called tracked objects), shown inFIG.1B. The location is relative to a transmitter assembly112configured to emit a magnetic signal132in an environment of the magnetic tracking system.

Generally, the calibration device140is configured to detect the magnetic signal132emitted by the transmitter assembly112and determine a position and orientation (called a pose) of the calibration device140in the environment. Based on the position and orientation of the calibration device140in the environment, distortions in the magnetic signal can be mapped in the environment for subsequent tracking of the tracked object142. The calibration device140determines, using non-magnetic signals, a location in the environment for the calibration device and determines, for that location, a distortion to the magnetic signal generated by the transmitter112. A distortion map is generated based on measured distortions at a set of locations in the environment.

Generally, the magnetic signal can be distorted in the environment by objects (called non-tracked objects120) in the environment other than the tracked objects. The non-tracked objects120can include objects such as operating tables, chairs, shelves, or any other metal or magnetically responsive object in the environment. Generally, the magnetic tracking system100is calibrated to update the determined pose of the calibration device140based on the distortions of the magnetic signal132known to be caused by the non-tracked objects120. The calibration can be performed prior to operation of the magnetic tracking system100to track the tracked objects142. The calibration generates a distortion correction model that indicates, for any location in the environment, a distortion to the magnetic signal132caused by the non-tracked objects120.

For correcting the distortions in the magnetic signal132caused by the non-tracked objects120, the magnetic tracking system100uses a non-magnetic signal that is a part of the calibration device140. The magnetic tracking system100uses the non-magnetic signal (e.g., a visual signal) to determine a location of the calibration device140in the environment. The magnetic tracking system100determines, using the distortion correction model generated during the calibration of the magnetic tracking system100, a distortion in the magnetic signal132caused by the non-tracked objects120at the location of the calibration device140. The magnetic tracking system100performs a distortion correction to remove the distortion caused by the non-tracked objects120to generate data representing a corrected magnetic signal. The magnetic tracking system100determines a position of the tracked object142using the corrected magnetic signal.

The magnetic tracking system100uses a non-magnetic device to determine a pose of the calibration device140in the environment. The non-magnetic device (e.g., non-magnetic signal detector110) is used for calibration of the magnetic tracking system100by generation of the distortion correction model as previously described. The non-magnetic signal detector110receives the non-magnetic signal (e.g., a visual signal). The calibration device140receives a magnetic signal132from the transmitter112for the same location in the environment as for the visual signal. The computing device102, based on the visual signal (e.g., including representations of markers108) and the corresponding magnetic signal132for that location, estimates a pose of the calibration device140. The magnetic tracking system100uses a first pose estimate from the non-magnetic signal (e.g., visual signal detectible by a camera110) and a second pose estimate from the magnetic signal132together to generate final pose estimate for the magnetic tracking device140. When this is performed in real time, it can be referred to as a fused pose estimate because the magnetic tracking system100combines the pose estimates from the magnetic and non-magnetic signals. This would typically be done using a Kalman filter based Simultaneous Localization and Mapping (SLAM) process to perform fusion. An example of this can be found in U.S. Patent Application Pub. No. 2019/0242952, filed on Feb. 8, 2019, and is incorporated in entirely by reference in this document.

Another approach is to collect either 1) a pose estimate from the non-magnetic signal (e.g., a visual signal from camera110) and a second pose estimate from the received magnetic signal132together or 2) the pose estimate from the non-magnetic signal (e.g., from the camera110) and the distortions in the magnetic signal132caused by the non-tracked objects120. When the poses are collected for both non-magnetic and magnetic signals, a compensation method that corrects the distorted magnetic pose into non-magnetic pose information can be performed. This correction is computed offline but applied in real time to the magnetic pose estimate to perform improved EM tracking. When the non-magnetic pose and magnetic signals are collected, the non-magnetic pose is converted to virtual magnetic signals, and the difference between these and the magnetic signals are used to map distorted magnetic signals into the virtual non-magnetic signals. This mapping is computed offline but applied in real time to the magnetic signals to perform improved EM tracking.

To acquire the non-magnetic signal, the non-magnetic signal detector110is moved around the environment200to various locations. InFIG.1A, an example of the non-magnetic signal detector110is a camera. However, other non-magnetic signals can be used to determine a ground-truth location of the calibration device140in the environment. The camera (and other components of the magnetic tracking system100) is described in greater detail with respect toFIG.2. Other examples of non-magnetic signal detectors110can include an ultrasonic device, a radio device, an infrared device, an acoustic device, and so forth configured to receive a signal and localize the non-magnetic signal detector110using the received non-magnetic signal.

The non-magnetic signal detector110can include a camera. The camera can be configured to view markers106a-e,collectively markers106, and marker array108, positioned in the environment in order to determine the pose of the calibration device140(which includes the camera) in the environment. The markers106and marker array108can be optical markers. As the magnetic tracking system100(and the camera) are moved around the environment, the field of view144of the camera passes across the markers106a-eand/or marker array108. The orientation of the markers with respect to one another and to the camera provide information about where the camera is located in the environment. For example, the locations of the each of the markers106a,106b,106c,and106d,106e,and the array of markers108in the environment are known. When the camera field of view144records one of the markers106a-eor marker array108in an image or video, the position and orientation of the marker relative to the camera are discernable from the images or video captured by the camera. The absolute pose of the camera can be discerned using data regarding the positions of each of the markers106a-eor marker array108in the environment and the relative pose of the camera relative to the markers.

In some implementations, the marker array108is placed at or near the location of where the tracked object will be during operation of the magnetic tracking system100. For example, the marker array can be placed on an operating table, operating chair, a floor under the tracked object142, a ceiling above the tracked object142, and so forth. In some implementations, the markers of marker array108can be placed in a regular pattern, shown inFIG.1A. In some implementations, the markers of the array108can be arranged in irregular patterns that can indicate where in the array the camera is viewing when only a portion of the array is visible. In some implementations, the markers of the array108are each unique in one or more aspects, such as the pattern, shape, size, or in another aspect.

The magnetic tracking system100can use various localization processes for determining the pose of the camera relative to the markers106a-eand/or marker array108. For example, the magnetic tracking system100can use a Simultaneous Localization and Mapping (SLAM) process. The SLAM process can use the markers106a-eand marker array108as landmarks. In some implementations, each of the markers106a-eand markers of marker array108are unique such that each is distinguishable from the other markers.

Once the magnetic tracking system100has a pose estimate using the non-magnetic signal, the magnetic signal receiver130can be used to measure the magnetic signal132at the same pose of the calibration device140for which that pose estimate was generated. The magnetic tracking system100generates another pose estimate using the magnetic signal132. The pose estimate generated from the magnetic signal132is corrected using the pose estimate from the non-magnetic signal. This process can be repeated as the calibration device140is moved around the environment.

As shown inFIGS.1A and1B, the markers106a-eand marker array108can be placed in a pattern near where the tracked object142is to be tracked in the environment. The calibration device140can view the markers106a-eand/or marker array108. In some implementations, the markers106a-dcan be placed on and around the non-tracked object120in the environment. In some implementations, one or more markers, such as marker106e,can be positioned on the transmitter112. The calibration device140can determine a relative pose with respect to the transmitter112by viewing the marker(s)106eon the transmitter assembly112.

In some implementations, the calibration device140can include a transmitter (such as transmitter112) and the receiver130can be fixed in place in the environment while the calibration device140is moved around the environment. This also enables the calibration device140to determine its position and orientation in the environment by magnetic means.

In some implementations, the magnetic signal132can be transformed directly into pose data from which pose compensation (e.g., distortion compensation) is performed. The pose data from the magnetic signal134are compared to the optically determined pose of the calibration device140for generating the distortion correction map for the environment.

For simplicity, an example non-magnetic signal detector110including a camera is described in detail as an example embodiment throughout this disclosure. However, other non-magnetic signal detector110configurations are possible. For example, the non-magnetic signal detector110can include a radio signal receiver. Radio beacons (or other radio signal emitters) can be placed around the environment to emit signals that can be used to localize the non-magnetic signal detector110in the environment. Other frequencies in the electromagnetic spectrum are also possible.

The processing of the magnetic signal132and the non-magnetic signal can be performed by a computing device102. The computing device102can include one or more processors, as subsequently described in relation toFIG.7. For example, the computing device102is configured to receive data representing the magnetic signal132and the non-magnetic signal and perform the distortion correction processing for determining the pose of the calibration device140and the tracked object142.

A user interface104is configured to be in communication with the computing device102. The user interface104is configured to present a representation of the pose of the calibration device140, the tracked object142, and so forth. The user interface104can be configured to display a three dimensional representation of the calibration device140and tracked device142in the environment. Other such representations of the calibration device140and the tracked object142, and other components of magnetic tracking system100, are possible. For example, the user interface104can display a text representation of the pose of the calibration device140or other objects in the environment according to a coordinate system (such as a Cartesian system, polar coordinate system, etc.).

The user interface104can be configured to present a representation of the distortion correction model determined by the computing device102for the environment. For example, a heat map overlay can show which regions in the environment have greater distortions (e.g., with greater heat map values).

The user interface104can include a display for reporting the position and orientation of the tracked object to a user of the magnetic tracking system100. The position and orientation that are reported to the user can be useful for assisting the user in one or more applications, such as performing a medical operation. For example, the user interface can report the position and orientation as a visual representation of the tracked object with respect to a portion of the magnetic tracking system100, report coordinates of the tracked object, superimpose the tracked object in images captured by the camera110, and so forth.

The user interface104can be configured to control operation of the transmitter assembly112. The user interface104can include one or more controls (software controls, hardware controls, etc.). The controls can be configured to enable the user to turn the transmitter assembly112off or on, change the frequency of operation of one or more of the transmitter assemblies, cause a transmitter assembly to upload calibration data, and so forth.

In some implementations, the computing device102is configured to send or transmit distortion correction data to one or more other devices. For example, in a medical environment, the computing device102can be configured to transmit distortion correction data to one or more other surgical instruments, magnetic tracking devices, visualization generation devices, and so forth.

Turning toFIG.1B, the magnetic tracking system100is shown during tracking of the tracked object142. The tracked object142is moved in the environment of the magnetic tracking system100. The computing device102mapped the distortions based on the data measured by the calibration device140. The location of the tracked object142relative to the transmitter112is determine with the distortion effects from the non-tracked objects120removed to improve the accuracy of determining where the tracked object142is in the environment. The tracked object142includes items such as surgical tools or tool tips that are being tracked in the environment. Generally, the calibration device140need not be present during tracking of the tracked object142. While the markers108can still be present in the environment, the distortions have already been mapped by the calibration device140. The markers can be moved from the environment if necessary, such as away from a surgical table or operating area, if the markers interfere with one or more tasks performed during tracking by the magnetic tracking system100.

Turning toFIG.2, an example of the magnetic tracking system200is shown. In some implementations, magnetic tracking system200can be similar to magnetic tracking system100ofFIG.1AandFIG.1B. As previously described, the magnetic tracking system200includes a computing device202(which can be similar to computing device102ofFIG.1AandFIG.1B), a user interface204(similar to user interface104), a camera device210(similar to camera device110), and a transmitter assembly212, similar to transmitter112. The magnetic tracking system200also includes include amplifiers232a-b,and the magnetic receiver230(also called a receiver assembly230or receiver230). Various embodiments of the components of the magnetic tracking system200are now described.

The magnetic tracking system200is configured to track the position(s) and orientation(s) of one or more tracked objects (not shown) that are in the environment of the magnetic tracking system200. In medical contexts, the tracked object generally includes a medical device or a portion of a medical device. For example, the magnetic tracking system200can be used to track items such as surgical instruments, probes, endoscopes, catheters, and so forth when they are inside a human body. In some implementations, the magnetic tracking device240(similar to calibration device140ofFIG.1A) is the tracked object. In some implementations, the tracked object is another object, and the magnetic tracking device240is used for distortion correction only. The magnetic tracking device240includes the receiver130that senses the magnetic signal132from the transmitter assembly212. The magnetic tracking system200can determine the position of the receiver230of the tracked object based on these signals, as subsequently described in greater detail.

The magnetic tracking system200is configured to emit a magnetic signal (e.g., a magnetic field) from the transmitter assembly212. The receiver230is configured to measure the magnetic field and send the measured signal to the computing device102. In some implementations, an amplifier232is included to amplify the signal that is measured by the receiver230. The amplifier232generally has a positive gain configured to amplify any analog signals received from the receiver230so that the computing device202can receive the amplified signal as an input. The receiver230can include one or more elements for measuring the magnetic signal emitted by the transmitter assembly212, such as a magnetometers, coils, etc. For performing tracking of the tracked object, the position and orientation of the receiver230is calculated by computing device202using measurements of the fields from the transmitter assembly212. The position and orientation of the transmitter assembly212are known by observing the transmitter assembly212with cameras210a-bor equivalent non-magnetic sensor.

The transmitter assembly212includes a transmitting element configured to generate a magnetic signal, such as a transmitter coil. In some implementations, multiple distinct transmitter assemblies can be used. The transmitter assemblies can be slightly different (or even unique) from one or more of the other transmitter assemblies. For example, the transmitter assembly212can be configured to transmit a magnetic signal using a different modulating frequency. In some implementations, each coil has slightly different magnetic properties. The computing device202can store information characterizing the magnetic properties of the transmitter assembly212for calibration purposes. In some implementations, the transmitter assembly212includes markers226(similar to markers206) that can be used for optically determining the location of the transmitter assembly relative to the camera210or tracking device240.

During operation of the magnetic tracking system200, the transmitter assembly212is configured to emit a magnetic signal that can be measured by the receiver230.

In some implementations, the transmitter assembly212is configured to be identifiable by a non-magnetic means to determine the position and orientation the transmitter assembly with respect to the magnetic tracking device240. The non-magnetic means can include one or more of an optical means, an ultrasonic means, a radio means, acoustic means, infrared means, and so forth. Generally, one or more of the markers206a-b(which can be similar to markers106a-eand/or marker array108), such as an optical or fiducial marker, can be included on or near the transmitter assembly212. The computing device202is configured to recognize the marker, distinguish the marker from other markers, and determine a potion and orientation of the marker (and thus the position and orientation of a transmitter assembly) from images of the marker. While markers206aand206bappear as groups of markers inFIG.2, each marker206aand206bcan be an individual marker.

While the markers206a-bcan be placed on the transmitter itself, the markers can also be placed throughout the environment. One or more of the markers206a-bcan include unique patterns such that each marker is distinct from other markers in the environment. Generally, the markers206a-bare positioned such that, at any given pose used for distortion correction, the camera210a-bfield of view includes at least one marker. The magnetic tracking system200is configured to optically determine the pose of the magnetic tracking device240(which includes the camera210, the computing device202, and the receiver230). The optically determined pose is used to determine the distortions that are present in the magnetic signal132received by the receiver, as previously described. For example, in a medical context, the camera210is placed above the patient and the markers206a-b(e.g., including106a-eand/or marker array108) are placed around where the patient surgery will be (e.g., a region where distortions are being mapped). The camera110is configured to capture images of the markers206a-b(e.g., including markers106a-eand/or marker array108) to measure, for each pose of the magnetic tracking device240, a corresponding magnetic signal132. This measured magnetic signal132can be compared to an ideal magnetic signal that would be present without distortions. The difference in the magnetic signals can be used to generate, for each pose in the region, a distortion value. The distortion values are determined for each pose in the region with a resolution that is acceptable for the application being performed (e.g., a surgery). For example, the magnetic tracking device240can be moved an inch, two inches, five inches, etc. between measurements. For example, the magnetic tracking device240can be rotated 1 degree, 2 degrees, 5 degrees, etc. as needed between measurements.

In some implementations, the camera210is removed (or not included) for performing magnetic tracking during operation of the magnetic tracking device240. For example, during a surgery, the camera210can be disconnected from the magnetic tracking device240.

In some implementations, the camera210can be a stereoscopic camera including two cameras, such as camera210a,and camera210bthat are displaced from one another. The cameras210a,210bare configured to capture images from different angles with respect to the markers206a-b.The images from the stereoscopic cameras210a,210bcan be used to determine not only planar “x, y” position data and yaw of the magnetic tracking device240, but also the depth “z” position of magnetic tracking device240and the roll and pitch of the magnetic tracking device with respect to the markers206a-b.In some implementations, a single camera is used to determine depth data using multiple markers206on a same object (e.g., spaced at a known distance from each other).

The data representing the relative positions and orientations of the magnetic tracking device240are used by the computing device202to determine how to interpret the magnetic signals received from the receiver230. The magnetic signals indicate a position and orientation of the tracked device relative to the transmitter assembly212. To determine the absolute position and orientation of the tracked device, the positions and orientations of the markers are determined. Generally, each of the markers has a known position and orientation with respect to the transmitter assembly212. The coordinate system established by these markers is the global coordinate system (e.g., the Cartesian coordinate axes shown inFIG.1AandFIG.1B).

The receiver230is configured to measure the magnetic signals transmitted by the transmitter assembly212to determine the position and orientation of the tracked object with respect to the transmitter assemblies. The position of the tracked object can be measured as relative to any global reference point, such as the transmitter assembly212. The computing device202is configured to convert the measured magnetic signals into position and orientation data. In some implementations, position data can be expressed as a position vector of position coordinates (e.g., x, y, z coordinates). In this example, the receiver230uses a Cartesian coordinate system (with x, y, and z coordinates) to represent a location in space; however, other types of coordinate systems (e.g., cylindrical, spherical, etc.) may be utilized.

The orientation of the tracked object refers to a direction the tracked device is facing with respect to the global reference point (e.g., the transmitter assembly212), and can be expressed similarly by using a coordinate system and represented, for example, as a vector of orientation coordinates (e.g., azimuth (Ψ), altitude (θ), and roll (φ) angles). The magnetic tracking system200operates to be an up to six degree of freedom (6 DoF) measurement system that is configured to allow for measurement of position and orientation information related to a forward/back position, up/down position, left/right position, azimuth, altitude, and roll. For example, if the receiver230includes a single receiving coil, a set of minimum of at least five transmitter assemblies212can provide five degrees of freedom (e.g., without roll). In an example, if the receiver230includes as least two receiving coils, a minimum of at least six transmitter assemblies212can provide enough data for all six degrees of freedom to be determined. Additional transmitter assemblies or receiving coils can be added to increase tracking accuracy or allow for larger tracking volumes.

The computing device202comprises one or more processors and is configured to receive data representing the magnetic signal132and the non-magnetic signal for estimating the distortion of the magnetic signal in the environment. The computing device202receives the magnetic signal from the receiver230and converts the magnetic signal into position data and orientation data of the magnetic tracking device240. The computing device202can include input and output ports for sending and receiving both analog and digital data. The computing device202can include a waveform generator (not shown) for driving the transmitter assembly212. Aspects and examples of the computing device102are further described in relation toFIG.7.

The computing device202can include circuitry to drive the transmitter assembly212and control the operation of the transmitter assembly212. For example, the computing device202can include a controller that is configured to control the transmitter assembly212. The transmitter assembly212can be configured to emit magnetic signals at different times or frequencies in a measurement cycle. For example, the computing device202can be configured to control the transmitter assembly212to transmit a magnetic signal at a particular time in a measurement cycle, transmit magnetic signal in a particular sequence, etc. The receiver230measures each of the magnetic signals. If a timing regime is used, the computing device202can associate the received magnetic signal with a particular transmitter assembly based on when the magnetic signal is received by computing device202from the receiver230. The controller can control the transmitter assembly212using time-slice multiplexing, frequency multiplexing, and so forth.

The transmitter assembly212can be calibrated individually. In some implementations, the transmitter assembly212has one or more physical properties that are distinct. For example, the transmitter assembly212can be calibrated with modeled parameters, mapping of magnetic fields from the magnetic signals, spherical harmonics, closed form solutions, and so forth. The calibration data can be stored by the computing device202. The calibration data for a transmitter assembly can be stored locally with that respective transmitter assembly on a local storage. In an aspect, the calibration data can be sent (e.g., wired or wirelessly) to the computing device202to assist the computing device202for determining the position and orientation of the transmitter assembly. As stated previously, the transmitter assembly212can be configured to operate on a different frequency from one another, turn on at different times, and so forth.

A transmitter coil (not shown) is configured to produce a magnetic signal that is received by the receiver230. The coil can emit a signal (e.g., magnetic field) that is unique to the particular transmitter assembly212. For example, the transmitter assembly212may modulate the magnetic signal with a particular frequency. The coil can be a single or multi-turn coil. The coil can include any geometric shape capable of generating a magnetic field when supplied with an electric current. The coil can be a part of circuitry of the transmitter assembly212(e.g., a printed circuit board (PCB), or the coil can be separately attached to the transmitter assembly.

The transmitter assembly212may not include active circuitry. The coil can be driven from a remote source, such as a waveform generator of the computing device202. The transmitter assembly212can be configured to be plugged into the computing device102(or another device) to drive the magnetic signal. In some implementations, the transmitter assembly212can be configured to connect to one or more other transmitter assemblies, such as in a daisy-chain formation. In another example, the transmitter assembly212can be connected in parallel with one or more other transmitter assemblies.

The computing device202can be configured to determine the position and orientation of the tracked object in a variety of ways. For example, a least-squares solution can be used to determine the position and orientation of the receiver230with respect to the transmitter assembly212. In another example, a Kalman filter or one or more other numerical methods can be used to determine the positions and orientations of the receiver230with respect to the transmitter assembly212.

The computing device202, the transmitter assembly212, the camera210, and the user interface204can communicate with each other through either wired or wireless connections. For example, the transmitter assembly212can be wired into ports of the computing device202. In such a configuration, the computing device202can provide a power signal to drive the transmitter assembly212, and the transmitter assemblies can each include passive electronics. In another example, the transmitter assembly212can be equipped with a data transceiver configured to wirelessly transmit data (e.g., calibration information) to the computing device and receive data (e.g., control signals) from the computing device202.

The markers206a-bcan be rearranged during for distortion correction for the magnetic tracking system200, and the new positions and orientations of each of the makers can be determined by the computing device202using camera210data. For example, the markers206a-b(e.g., including106a-eand/or marker array108ofFIG.1A) can be moved around during calibration to ensure that one or more of the markers is within several inches of where the tracked object (e.g., a catheter) will be while the tracked object is moved around the environment (e.g., inside the patient). Once the markers206a-bare in place in the environment, the calibration device140is moved around the environment to calibrate the distortion correction model for the region.

In some implementations, the markers206a-beach include adhesive patches which can be affixed to one or more surfaces of the environment of the magnetic tracking system200. For example, the markers206a-bcan be affixed to a subject (e.g., a patient) of a medical operation.

The combination of the receiver230and the transmitter212can be a distortion indicator or distortion correction system for the environment. The magnetic tracking system200can use a determined difference between optical determination of the optical pose of each of assemblies and the EM determination of pose as an indication of distortion present in the environment. The magnetic tracking system200can thus perform distortion compensation. For example, data collected by the receiver assembly230, including magnetic field measurements and pose, distorted data, optical data (truth), and gradient data (estimated by differences in fields as sensor move) can be used to make corrections for distortion. The distortion can be modeled using physical models. These models can include curve-fitting (e.g., for magnetic signals and for pose solutions), splines, triangulations, radial basis functions, and using machine learning methods.

Turning toFIG.3, an example of an environment300is shown for distortion correction of the magnetic signal132. Markers306a-g(collectively markers306) are placed throughout the environment300. The markers306can be arranged or rearranged depending on wherein the environment300the distortion correction is being performed, similar to markers106and206, previously described. Generally, the markers306are placed such at least one is visible to the camera310(which can be similar to camera110ofFIG.1Aand/or camera210ofFIG.2) of a magnetic tracking device340(similar to magnetic tracking devices140and/or240) at any pose of the magnetic tracking device.

In an aspect, the one or more markers306can include any visual mechanism for distinguishing the markers from each other. The markers306are each be distinct from one another so that the computing device302(which can be similar to computing devices102and202, previously described) can discriminate between the markers when determining the pose of the magnetic tracking device340visually. The positions of the markers306relative to one another indicate to the computing device302what the position and orientation of the transmitter assembly312is relative to the camera310, receiver330(e.g., similar to receiver130and/or receiver230, previously described), or other portion of the magnetic tracking system300. For example, if marker306bappears in a positive direction along a x-axis with respect to icon306c,the computing system can determine that the transmitter assembly300is rotated at a particular yaw value (e.g., planar rotation with respect to the camera310). Alternatively, computer vision or machine learning methods may be used to track the transmitter assembly300, as is known in the art. Each of the markers306can be selected from a library of icons which the computing device302is configured to recognize and assign to different locations in the environment300. The computing device302associates a received magnetic signal with a position determined based on what marker306is seen and what its orientation is. This assists in position and orientation calculations. In other words, the computing device302uses the markers306to know which magnetic pose is associated with which visual pose. The computing device302can then determine what the ideal magnetic signal would be at the ground truth location and determine a magnetic distortion value.

In an aspect, the camera device310can include stereoscopic cameras which provide distance information to the computing device302. The computing device302can use the distance information (whether determined at the camera device310or calculated at the computing device302) to determine the position of the transmitter assembly300in three dimensions. The camera device310can include a single camera which can determine distance based on the use of multiple markers306a-hand their known geometry.

As stated previously, the markers306a-hare generally configured to be recognized for computer vision recognition (similar to markers106and206and marker array108). The markers can include any images, such as bar codes, QR codes, symbols, icons, and so forth. In the example of magnetic tracking system300, the markers306a-hare pixelated symbols. While fewer than 8 symbols are included in this example for each marker306, additional symbols can be added for determining pose. In addition to being symbols, a marker can include a retroreflector configured for infrared excitation or another kind of landmark that is detectable by a non-magnetic means.

In some implementations, the markers306a-hcan vary, as long as the configuration is capable of providing enough information for the magnetic tracking system300to generate an estimate of the visual pose. For example, the markers can include ArUco or ChArUco patterns. For example, the markers can include passive retroreflectors that respond to infrared (IR) excitation from an IR source near the camera aperture(s)110aand110b.The markers306a-fcan include active beacons that transmit light signals, radio frequency data, ultrasonic signals, or other pose information. In some implementations, the shapes of the markers306athemselves can be used for pose determination.

In an aspect, each marker306a-fcan include an adhesive layer which is applied to a side. The adhesive layer is configured to adhere the marker306to another surface, such as a wall, ceiling, non-tracked object, and so forth. The surface is not required to be flat or regular. The adhesive layer can include an adhesive gel, glue, suction cups, or other adhesive surface. The adhesive layer is generally configured to be removable and reapplied to a surface repeatedly. The adhesive layer does not obscure the side of the marker306that includes the symbols.

Turning toFIGS.4A-4B, a calibration environment with a scaffold400is shown. The scaffold allows the magnetic tracking device440(e.g., similar to magnetic tracking devices140,240, and/or340) to move in a known region of the environment with respect to the markers406(similar to markers106,206, and306, previously described). For example, the scaffold400allows planar motion of the magnetic tracking device440with respect to the markers406. A number of planes P1, P2, P3, P4, P5(and so forth) can be provided in the scaffold400. At each plane, the magnetic tracking device440can be positioned and a magnetic signal132measured. The pose of the magnetic tracking device440is known with a high precision due to the fixed nature of the scaffold400. Shelves (or other support means) can be added or removed to allow the magnetic tracking device440to move relative to the markers406. Various measurements of the magnetic signal132can be taken at each of the planes. A distortion correction model can then be generated as previously described. For example, inFIG.4A, the magnetic tracking device440is at position P1. InFIG.4B, the magnetic tracking device440is at position P2. The planes can be positioned close together to increase a resolution of the distortion correction model as needed. The magnetic tracking device440can be moved around the plane to capture a large number of magnetic signal measurements at the plane.

Turning toFIG.5, a calibration environment with a scaffold500is shown. The scaffold includes stackable layers504that can be built up or broken down quickly, such as over a patient on a surgical table. The scaffold500is portable and allows the magnetic tracking device502(e.g., similar to magnetic tracking devices200,300) to move in a known region of the environment with respect to the transmitter112. For example, the scaffold500allows planar motion of the magnetic tracking device502in the environment. Rather than including markers, the scaffold can be formed with precise dimensions such that the absolute position of the magnetic tracking device502is known in a longitudinal direction. An encoder (e.g., a laser-based encoder) can be used to track where the magnetic tracking device502moves around the plane. For example, a computer mouse coupled to a receiver (similar to receivers130,230, and/or330) can be used to measure the magnetic signal132at each location on a plane. For each plane, the magnetic tracking device502can be iteratively positioned across various points on the plane, and the magnetic signal132is measured at each point. The pose of the magnetic tracking device502is known with a high precision due to the fixed nature of the scaffold500. Shelves (or other support means) can be added or removed to allow the magnetic tracking device502to move in additional dimensions. A distortion correction model can then be generated as previously described. The magnetic tracking device502can be moved around the plane to capture a large number of magnetic signal measurements at the plane.

Turning toFIG.6A, a flow diagram of a process600for performing magnetic tracking with the magnetic tracking systems and transmitter assemblies ofFIGS.1-5is shown. The process600represents how a magnetic tracking system (e.g., magnetic tracking systems ofFIGS.1-5) is configured for determining an object pose of a tracked object in an environment of the magnetic tracking system.

A computing system is configured to receive (602), from a magnetic tracking device, a measurement of a non-magnetic signal and a corresponding measurement of the magnetic signal. The non-magnetic signal can include a visual signal, such as an image or video. The computing system estimates (604), based on the measurement of the non-magnetic signal, a non-magnetic pose of the magnetic tracking device in the environment for the location. The computing system is configured to estimate (606), based on the measurement of the magnetic signal, a magnetic pose of the magnetic tracking device in the environment for the same location in the environment. In some implementations, the magnetic pose and the visual pose are estimated in parallel. In some implementations, estimated poses can be saved to buffers for pipelining the generation of the distortion correction model. For example, the computing system determines (608) a difference between the magnetic pose estimate and the non-magnetic pose estimate to produce an error value. As the difference is being determined, the computing system can be configured to measure a next iteration of magnetic signals and non-magnetic signals to estimate a second magnetic pose and a second non-magnetic pose.

The computing system is configured to determine (610) a magnetic distortion correction value for associating with the particular location in the environment based on the determined difference in the pose values. Once distortion values are calculated for a plurality of locations, the computing system can build or generate (612) a distortion correction model may be formed with distortion models, spherical harmonics, mathematical functions and the like and can be performed using a pre-computed correction or as evaluating a model in real time, including the distortion values and output a representation of the distortion correction model. The distortion correction model can be stored in a memory such that, as the magnetic tracking system is moved around an environment, the distortion correction can be retrieved to correct a magnetic pose that is determined by the magnetic tracking system. In other words, because the distortion value for a given location in the environment is known, the magnetic tracking system can refine the magnetic pose estimate of the tracked object once object tracking is performed. During tracking, the computing device is configured to output a representation of the tracked object pose, such as to a user interface.

In some implementations, the marker comprises one of an ArUco pattern, a ChArUco pattern, an infrared retroreflector, a light source, an ultrasonic source, a radio signal source, and an outer shape of the transmitter assembly.

FIG.6Bshows a process620for distortion correction for electromagnetic fields using inside-out tracking. Process620illustrates a function fitting process where a non-magnetic signal is a function of the magnetic signal. For example, the x position of the magnetic signal may be compensated by a polynomial comprised of powers of the magnetic signal such that it equals the non-magnetic signal. These functions take many forms, such as polynomials, radial basis functions, spherical harmonics, kriging, neural networks, splines, etc. The function fitting process is typically performed off-line after all the signal data is captured. The magnetic signal may be in the measurement domain (the magnetic fields) ad/or the tacking (solution or pose) domain.

A computing device (such as computing devices102,202, or302ofFIGS.1-3) is configured to receive (622) a measurement of the non-magnetic signal and a corresponding measurement of the magnetic signal. The computing device is configured to generate (624) a mapping function that relates the magnetic signal to the corresponding non-magnetic signal. The computing device is configured to store (626) the mapping function for use during magnetic tracking.

FIG.6Cshows a process630for distortion correction for electromagnetic fields using inside-out tracking. Process630illustrates the process of using the function generated by process620during EM-only tracking. A magnetic signal is received, and that signal is used in the function to generate a corrected magnetic signal. The corrected magnetic signal may be in the measurement domain (the magnetic fields) and/or the tacking (e.g., solution or pose) domain. If the corrected magnetic signal is in the measurement domain, the corrected magnetic signal is then run through the standard tracking algorithm to generate pose data.

A computing device (such as computing devices102,202, or302ofFIGS.1-3) is configured to receive (632) a measurement of the magnetic signal during a tracking process. The computing device is configured to apply (634) a mapping function generated from a calibration process ofFIG.6Bto the received signal. The computing device is configured to output (636) a corrected magnetic signal indicating the true location and orientation of the tracked device.

FIG.7is a block diagram of an example computer system700. The computing device102ofFIGS.1A-1Bmay be an example of the computer system700described here. The system700can include a processor710, a memory720, a storage device730, and an input/output device740. Each of the components710,720,730, and740can be interconnected, for example, using a system bus750. The processor710is capable of processing instructions for execution within the system700. The processor710can be a single-threaded processor, a multi-threaded processor, or a quantum computer. The processor710is capable of processing instructions stored in the memory720or on the storage device730. The processor710may execute operations such as causing the EMT system100to determine the position and/or the orientation of tracked device102.

The memory720stores information within the system700. In some implementations, the memory720is a computer-readable medium. The memory720can, for example, be a volatile memory unit or a non-volatile memory unit.

The storage device730is capable of providing mass storage for the system700. In an aspect, the storage device730is a non-transitory computer-readable medium. The storage device730can include, for example, a hard disk device, an optical disk device, a solid-date drive, a flash drive, magnetic tape, or some other large capacity storage device. The storage device730may alternatively be a cloud storage device, e.g., a logical storage device including multiple physical storage devices distributed on a network and accessed using a network. In some implementations, the information stored on the memory720can also or instead be stored on the storage device730.

The input/output device740provides input/output operations for the system700. In some examples, the input/output device740includes one or more of network interface devices (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 10 port), and/or a wireless interface device (e.g., a short-range wireless communication device, an 602.11 card, a 3G wireless modem, or a 4G wireless modem). Generally, the input/output device740includes driver devices configured to receive input data and send output data to other input/output devices, e.g., a keyboard, a printer, and display devices. In some implementations, mobile computing devices, mobile communication devices, and other devices are used.

The system700can include a microcontroller. A microcontroller is a device that contains multiple elements of a computer system in a single electronics package. For example, the single electronics package could contain the processor710, the memory720, the storage device730, and input/output devices740.

Although an example computer system has been described inFIG.7, implementations of the subject matter and the functional operations described above can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier, for example a computer-readable medium, for execution by, or to control the operation of, a processing system. The computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter effecting a machine readable propagated signal, or a combination of one or more of them.

The term “computer system” may encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. A processing system can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program (also known as a program, software, software application, script, executable logic, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile or volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks or magnetic tapes; magneto optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the subject matter described herein. Other such embodiments are within the scope of the following claims.