Patent Description:
In an invasive surgical procedure, including a minimally invasive procedure, it is typically necessary to track a surgical device, such as a catheter, within a patient that is not directly visible to a physician performing the procedure. Typically, a computerized tomography (CT) image, such as from fluoroscopy or magnetic resonance imaging (MRI), of the patient is available to the physician.

<CIT> describes an apparatus, including a patient tracker, attached to a subject, having magnetic field sensors and optical landmarks with known spatial relationships to each other. A camera acquires a 3D optical image, in a first frame of reference (FOR), of the subject's face. A magnetic radiator assembly generates magnetic fields at the subject's head, thereby defining a second FOR. <CIT> also describes a processor which: processes field sensor signals to acquire location coordinates of the sensors in the second FOR; segments a tomographic image of the subject, having a third FOR, to identify the subject's face in the third FOR; computes a first transformation between the third and first FORs to map the tomographic face image to the 3D optical image; maps the optical landmarks to the third FOR; maps the respective location coordinates of the sensors to the first FOR; and computes a second transformation between the second FOR and the third FOR.

One method used for tracking surgical devices within a patient is the TruDi™ electromagnetic image-guided navigation system, produced by Acclarent, Inc. , of <NUM> Technology Drive, Irvine, CA <NUM> USA. In this system, alternating magnetic fields are transmitted, from fixed transmitters external to the patient, so as to pass through the patient. If a procedure is being performed on a patient's head, such as ear, nose and throat (ENT) procedure, then the fixed magnetic field transmitters may be placed around the patent's head. As part of the procedure, the surgical device is placed in the middle of the calibration chamber, and three orthogonal fields may be applied. A sensor, typically a single or multiple axis coil, is attached to the surgical device and inserted into the patient and takes voltage measurements. A processor records the currents generated by the fields traversing the sensor. The processor analyzes the currents so as to determine both the position and the orientation of the sensor in an electromagnetic frame of referenced defined by the fixed transmitters.

Current magnetic based location detection systems may also utilize flexible sensors located on the surgical device in combination with an algorithm or processor to estimate the position, shape, and size of the surgical device, based on voltage measurements taken by the sensors. Flexible sensors typically consist of single-axis sensors (SAS) with <NUM> transmitters and <NUM> sensor, for a total of <NUM> measurements taken at a single point. A tri-axial sensor (TAS), comprising <NUM> sensors, with <NUM> transmitters and <NUM> coils, collects a total of <NUM> voltage measurements at a single point. When the surgical device is navigated or advanced, additional points may be collected within a short amount of time.

The registration of the CT image with a three-dimensional (3D) camera image is a preliminary step in the TruDi™ navigation procedure. Registration involves the localization of the surgical device relative to the registered CT image. In other words, by registering or matching up, the CT image with a 3D image and magnetic coordinates, the location of a surgical device being tracked by an electromagnetic tracking system may be accurately determined and displayed on said 3D camera image, CT image, or combination thereof. In some instances, it may be beneficial to provide an operator with a 3D view of surface of anatomical structures in the head of a patient. After completing the registration of the 3D image with the CT image and the 3D image with the magnetic coordinates, the CT image may then be registered with the magnetic coordinates. Current systems for registering the CT image with the 3D camera image require equipment such as a patient tracker to be attached to the patient's head and certain measures must be taken to account for the patient tracker during registration. It would be desirable to have a streamlined system and method for registering the CT image with the 3D camera image.

Systems, methods, and devices for registering a 3D image of a patient with magnetic coordinates are disclosed. A tri-axial sensor (TAS) is added to the 3D camera. The location and orientation of the TAS sensor may be determined based on the known magnetic fields that are applied by a magnetic field transmitter. The camera coordinate system may then be transferred to the magnetic coordinate system. After completing the registration of the 3D image with a CT image and the 3D image with the magnetic coordinates, the CT image may then be registered with the magnetic coordinates. By registering the CT image with the magnetic coordinates, the location of a catheter being tracked by the electromagnetic tracking system may be accurately shown on a display of the CT image, optical image, or combination thereof.

<FIG> is a schematic illustration of a registration system <NUM>, according to an embodiment of the present disclosure.

The medical procedure undergone by the patient is assumed to comprise tracking of an surgical device, such as a catheter, which is inserted into the patient by a medical professional <NUM>. The tracking is provided by an electromagnetic tracking system <NUM>, described in more detail below.

The electromagnetic tracking system comprises a magnetic radiator assembly <NUM>, which is positioned around the patient's head. Assembly <NUM> comprises magnetic field transmitters <NUM>, which are fixed in position and transmit alternating sinusoidal magnetic fields into a region <NUM>, where the head of the patient <NUM> is located. By way of example, magnetic field transmitters <NUM> of assembly <NUM> are arranged in an approximately horseshoe shape around the head of the patient <NUM>. However, alternate configurations for the radiators of assembly <NUM> will be apparent to those having ordinary skill in the art, and all such configurations are assumed to be comprised within the scope of the present invention.

A magnetic sensor, herein assumed to be a coil, is attached to the surgical device being tracked within the patient <NUM>. The attached coil generates electrical signals in response to the alternating magnetic fields traversing the coil, and these signals are transferred to a system processor <NUM>. The processor <NUM> is configured to process the signals so as to derive location and orientation values for the sensor. Other elements of the system <NUM>, including magnetic transmitters <NUM>, are controlled by the system processor <NUM>.

The TruDi™ system referred to above uses a tracking system similar to that described herein for finding the location and orientation of a coil in a region irradiated by magnetic fields.

The processor <NUM> uses software stored in a memory <NUM> to operate system <NUM>. The software may be downloaded to the processor <NUM> in electronic form, over a network, for example, or it may, additionally or alternatively, be provided and/or stored on non-transitory tangible media, such as magnetic, optical or electronic memory. The processor <NUM> uses the software to analyze the signals received from the magnetic sensors. Software for a registration algorithm <NUM> in implementing registration system <NUM>, which executed by the processor <NUM>, is also stored in memory <NUM>. Registration algorithm <NUM> is described in more detail below.

The processor <NUM> may be mounted in a console <NUM>, which comprises operating controls <NUM> that typically include a keypad and/or pointing device such as a mouse or trackball. The console <NUM> connects to the radiators via a cable <NUM> and/or wirelessly. The medical professional <NUM> may use operating controls <NUM> to interact with the processor while performing the medical procedure described above. While performing the procedure, the processor may present results of the procedure on a screen <NUM>. The presentation of the results of the procedure on the screen <NUM> allows for a medical professional <NUM> using the system to visualize the precise location of surgical device, such as a catheter, relative to a CT image of the patient.

As described above, the electromagnetic tracking system <NUM> is able to track the position and orientation of a magnetic sensor in region <NUM>, by virtue of the magnetic fields transmitted into the region from magnetic transmitters <NUM>. It will be understood that the position and orientation derived for system <NUM> is with reference to a frame of reference (FOR) of the magnetic system, as defined by the positions of magnetic transmitters <NUM>. In order for the tracking of the sensor to be useful, the magnetic system FOR needs to be registered with the FOR for an image of the patient <NUM> that is stored in memory <NUM>. Subsets <NUM> and <NUM> of image <NUM>, described further below, are also stored in memory <NUM>.

While the CT image may typically comprise a magnetic resonance imaging (MRI) image or a fluoroscopic image, in the description herein the image is assumed to comprise, by way of example, a fluoroscopic CT image.

The medical professional <NUM> uses a three-dimensional (3D) camera <NUM> to capture a 3D optical image of the face of patient <NUM>. In some embodiments, the camera <NUM> is a RealSense 3D camera, produced by Intel Corporation of Santa Clara, California. The 3D camera <NUM> may comprise at least one optical sensor. In some embodiments, the 3D camera <NUM> may comprise two, separate optical sensors. The 3D optical image comprises a set of optical voxels, each voxel having three Cartesian coordinates as well as color, typically red, green, blue (RGB) values. The set of optical voxels is herein also termed a 3D scatter plot <NUM>, and the optical voxels of scatter plot <NUM> are stored in memory <NUM>.

For the registration implemented by the system <NUM>, a patient tracker <NUM> is positioned on patient <NUM>. The patient tracker <NUM> is described with reference to <FIG>, described below.

<FIG> are schematic figures illustrating the patient tracker <NUM> according to an embodiment. Patient tracker <NUM> is formed as a substantially planar sheet, and <FIG> illustrates a view of the tracker, as seen by the camera <NUM>. i.e., after the tracker has been positioned on the patient <NUM>. <FIG> is an exploded view of the tracker.

In some embodiments, the patient tracker <NUM> is constructed of five laminar sheets 80A, 80B, 80C, 80D, and 80E, all sheets having substantially the same shape, and being bonded together. Sheet 80A is an upper sheet, also shown in <FIG>, and incorporated in the sheet is a plurality of optically identifiable landmarks <NUM>. By way of example, sheet 80A comprises three optical landmarks <NUM>. However, other embodiments may comprise other numbers of landmarks.

Sheet 80C is an intermediary laminar sheet, typically formed from a flexible insulating material, upon which are formed, typically by printing, planar conducting coils <NUM> in the form of conductive spirals. Coils <NUM> act as electromagnetic sensors. These are the same number of coils <NUM> as landmarks <NUM>, and each coil is located on sheet 80C so that it is in a known spatial relationship with respective landmark <NUM>. By way of example, each coil <NUM> is located to be directly aligned with a respective landmark <NUM> when the sheets of the tracker are bonded together. However, other embodiments may be formed with different known spatial relationships between the coils and the landmarks. For example, coils and landmarks may be offset by known spatial amounts.

A cable <NUM> (illustrated in <FIG>) connects coils <NUM> to processor <NUM>. Connections of coils <NUM> to the cable are not shown in <FIG> for simplicity.

Sheet 80E is a lower laminar sheet formed from biocompatible adhesive, and it is this sheet that contacts patient <NUM> during operation of system <NUM>.

Sheets 80B and 80D are intermediate laminar sheets, formed of conductive material, so as to act as electrical shields for coils <NUM>. Within sheets 80B are non-conductive regions <NUM> aligned with coils <NUM>. The presence of the non-conductive regions <NUM> enable the coils to operate correctly. In some embodiments, the non-conductive regions <NUM> are openings.

<FIG> is a flowchart diagram of a method <NUM> for executing registration algorithm <NUM>, according to an embodiment. Method <NUM> may be performed on the registration system <NUM> illustrated in <FIG>.

At <NUM>, the electromagnetic tracking system <NUM> is activated, and the head of patient <NUM> is placed within region <NUM> of the system. Patient tracker <NUM> is attached to the forehead of the patient, using biocompatible adhesive sheet 80E, and so that optical landmarks <NUM> are uppermost and are visible. Cable <NUM> is connected between the patient tracker and processor <NUM>, and the processor <NUM> may be activated to acquire signals conveyed by the cable from coils <NUM>. The processor analyzes the signals to calculate the positions of the coils in the FOR defined by magnetic transmitters <NUM>. If the calculated positions are found to be within an expected part of region <NUM>, processor <NUM> may provide an indication that the electromagnetic tracking system <NUM> is operating correctly to medical professional <NUM>. An example indication of the electromagnetic tracking system <NUM> operating correctly is the processor sending a notification that is displayed on screen <NUM>.

At <NUM>, the processor <NUM> may analyze a CT image of the head of the patient stored in memory <NUM>. In some embodiments, the processor <NUM> may analyze the image to identify a subset of CT voxels of the stored image corresponding to surface features of the head of the patient, and the subset may be stored as surface subset <NUM>.

At <NUM>, medical professional <NUM> activates the 3D camera <NUM> to acquire a 3D optical image of the face of patient <NUM>, and the acquired image is stored as scatter plot <NUM> in memory <NUM>. It will be understood that the image acquired by the 3D camera <NUM> includes an image of patient tracker <NUM> that is on the face of the patient.

Any suitable algorithm may be used to find the transformation that best maps surface subset of CT voxels <NUM> to the optical voxels of 3D scatter plot <NUM>. For example, any cloud point matching algorithm, such as robust point matching and kernel correlation, may be used. In some embodiments, an Iterative Closest Point (ICP) algorithm may be used. However, up to <NUM>, there is a known difference in the two sets of voxels, since an image of the patient tracker is present in scatter plot <NUM> but is not present in CT voxel subset <NUM>.

At <NUM>, the absence of an image of the patient tracker in CT voxel subset <NUM> is compensated for by adding an image of the patient tracker to the CT voxel subset. The addition may be implemented by presenting an image of the CT voxel subset to the medical professional <NUM> on screen <NUM>, allowing the professional to overlay an image of the patient tracker on the presented image, and storing the combined image as an adjusted CT voxel subset <NUM>.

Alternatively, at <NUM>, adjusted subset <NUM> is derived from CT voxel subset <NUM> by professional <NUM> selecting portions of subset <NUM> that do not include the patient tracker image. The medical professional <NUM> may perform the selection on an image of subset <NUM> presented on screen <NUM>, and the selected portions are stored as adjusted CT voxel subset <NUM>.

At <NUM>, the processor <NUM> maps adjusted CT voxel subset <NUM> to the voxels of scatter plot <NUM>. If at <NUM> the adjusted CT subset includes an image of the patient tracker, then the mapping may be performed for all the voxels of the two sets. Alternatively, if <NUM> is implemented by selecting portions of subset <NUM> that do not include the patient tracker image, the processor <NUM> makes a corresponding selection in the voxels of scatter plot <NUM>, and the mapping is performed between the selected sets of voxels.

The mapping provides a registration between the FOR of the CT image of the patient <NUM> and the FOR of the optical image of the patient. The processor <NUM> may quantify the registration as a first transformation matrix M[CT-OPT] which may be used to transform entities in one of the frames of reference to the other FOR.

At <NUM>, the processor <NUM> uses the known spatial relationship between optical landmarks <NUM> and coils <NUM> to perform a mapping between the locations of the landmarks in the optical 3D scatter plot <NUM> and the positions of the coils in the FOR of electromagnetic tracking system <NUM>, as found at <NUM>. The mapping provides a registration between the FOR of the electromagnetic tracking system and the FOR of the optical image, and this may be quantified as second transformation matrix M[MAGN-OPT].

At <NUM>, the processor <NUM> combines the two registrations, produced at <NUM> and <NUM>, to produce a third registration between the FOR of the electromagnetic tracking system <NUM> and the FOR of the CT image. The resulting registration may be quantified as a third transformation matrix M[CT-MAGN] and it will be understood that matrix M]CT-MAGN] may be generated from matrices M[MAGN-OPT] and M[CT-OPT].

As noted above, the method of <FIG> may be performed on the registration system <NUM> of <FIG> comprising the patient tracker <NUM>. In some embodiments, the registration system <NUM> does not comprise patient tracker <NUM>. Instead, an additional sensor may be added to the 3D camera <NUM> to register a 3D image of a head of a patient <NUM> with magnetic coordinates of the electromagnetic tracking system <NUM>. The additional sensor may determine a location and orientation of the 3D camera <NUM>. The location and orientation of the 3D camera <NUM> may be used to find registration between the electromagnetic system and CT image, as described in more detail below.

According to the claimed invention, the additional sensor is a tri-axial sensor (TAS). A TAS includes <NUM> sensors, with <NUM> transmitters and <NUM> coils, collects a total of <NUM> voltage measurements at a single point (3x9=<NUM>). The three sensors of the TAS may provide simultaneous measurements in three orthogonal directions. The 3D camera may be tracked and navigated to match the real position (magnetic location and orientation) of the TAS. The location and orientation of the TAS sensor may be determined based on the known magnetic fields that are applied by a magnetic field transmitter. The 3D camera <NUM> may be tracked and navigated using the TAS sensor, which reads the configuration of the field to search for the location and orientation of the 3D camera <NUM>. The addition of the TAS sensor to the 3D camera <NUM> provides for the ability to register the 3D camera image with the magnetic coordinates, which enhances the accuracy of the registration of the CT image with the magnetic coordinates.

In other embodiments, the additional sensor is a single-axis sensor (SAS) or dual-axis sensor (DAS). A SAS collects a total of <NUM> voltage measurements (1x9=<NUM>). A DAS collects <NUM> voltage measurements (2x9=<NUM>).

<FIG> is a flowchart diagram of a method <NUM> for executing a registration algorithm according to an embodiment. Method <NUM> may be performed on the registration system <NUM> illustrated in <FIG>, however, method <NUM> does not require the use of patient tracker <NUM>. Instead of using patient tracker <NUM>, 3D camera <NUM> further comprises, in addition to the at least one optical sensor, at least one magnetic sensor. This allows for the location and orientation of the 3D camera to be determined and used to register the 3D image with the CT image. According to the claimed invention, the magnetic sensor is a TAS, as discussed above. The TAS may provide simultaneous measurements in three orthogonal directions. In other embodiments, a SAS or DAS may be used instead of a TAS.

As noted above, the 3D camera <NUM> also comprises at least one optical sensor. In some embodiments, the 3D camera <NUM> may comprise two, separate optical sensors. The 3D optical image comprises a set of optical voxels, each voxel having three Cartesian coordinates as well as color, typically red, green, blue (RGB) values. The set of optical voxels is herein also termed a 3D scatter plot <NUM>, and the optical voxels of scatter plot <NUM> are stored in memory <NUM>.

At <NUM>, the electromagnetic tracking system <NUM> is activated, and the head of patient <NUM> is placed within region <NUM> of the system. When the electromagnetic tracking system <NUM> is activated, magnetic field transmitters <NUM>, which are fixed in position and transmit alternating sinusoidal magnetic fields into a region <NUM>, are placed where the head of the patient <NUM> is located.

At <NUM>, medical professional <NUM> activates camera <NUM> to acquire a 3D optical image of the face of patient <NUM>. The processor <NUM> stores the acquired 3D image as 3D scatter plot <NUM> in memory <NUM>.

At <NUM>, the processor <NUM> may be activated to analyze the voltage measurements of the TAS. The processor <NUM> is configured to process the signals so as to derive location and orientation values for the sensor. The processor <NUM> is configured to process the signals so as to derive location and orientation values for the sensor. Other elements of the system <NUM>, including magnetic transmitters <NUM>, are controlled by the system processor <NUM>. Other elements of the system <NUM>, including magnetic transmitters <NUM>, are controlled by the system processor <NUM>. The processor <NUM> is configured to process the signals so as to derive location and orientation values for the sensor, and thereby the camera. Other elements of the system <NUM>, including magnetic transmitters <NUM>, are controlled by the system processor <NUM>.

In some embodiments, <NUM> and <NUM> are performed simultaneously. In further embodiments, time stamps may be taken at both <NUM> and <NUM> and cross-referenced to ensure that they match (i.e., to confirm that the time the 3D image of the patient acquired at <NUM> is the same time at which the camera location and orientation was determined at <NUM>).

At <NUM>, the processor <NUM> transfers the optical 3D scatter plot <NUM> to the magnetic coordinate system of the electromagnetic tracking system. The location and orientation of the 3D camera <NUM> determined at <NUM> may be used to map the 3D scatter plot <NUM> to the magnetic coordinates. Any suitable location algorithm may be used to map the 3D scatter plot <NUM> to the magnetic coordinates. For example, a cloud point matching algorithm, such as ICP or robust point matching, may be used. The mapping may provide a registration between the optical image and the magnetic coordinates.

At <NUM>, the processor may register the 3D image with a CT image of the patient <NUM>. A CT image of the head of patient <NUM> may be retrieved from memory <NUM>. The processor <NUM> may analyze the image to identify CT voxels of the stored image corresponding to surface features of the head of the patient <NUM>. The CT voxels may be mapped to the optical voxels of the 3D scatter plot <NUM>. Any suitable algorithm may be used to find the transformation that best maps the optical voxels of the 3D scatter plot <NUM> to a surface of CT voxels. In some embodiments, an ICP algorithm may be used. This mapping may provide a registration between the optical image and the CT image.

In some embodiments, after completing the registration of the 3D image with the CT image and the 3D image with the magnetic coordinates, the CT image may then be registered with the magnetic coordinates. For example, the processor <NUM> may combine the registration between the optical image and the magnetic coordinates and the registration between the optical image and the CT image to produce another registration between magnetic coordinates and the CT image. Registering the 3D image with the magnetic coordinates first may enhance the accuracy of the registration of the CT image with the magnetic coordinates.

<FIG> is a schematic illustration of a 3D scatter plot <NUM> corresponding to the 3D optical image of a patient <NUM> positioned with a region <NUM> of a registration system, according to an embodiment. The registration system may comprise the registration system <NUM> illustrated in <FIG>, with the patient tracker <NUM> being optional. In <FIG>, the 3D scatter plot <NUM> of the acquired 3D image is overlaid onto the face of the patient <NUM> for illustrative purposes.

<FIG> are schematic illustrations of the mapping the 3D scatter plot <NUM> of the acquired 3D image to the CT coordinate system <NUM> of the electromagnetic tracking system, according to an embodiment. This mapping may occur in <NUM> of the method describes with respect to <FIG>. <FIG> illustrates the 3D scatter plot <NUM> and the CT voxels <NUM> when they are separate and FIG. <FIG> are schematic illustrates of the 3D scatter plot <NUM> being mapping to the CT voxels <NUM>. In some embodiments, the location and orientation of the 3D camera <NUM> when the 3D image was acquired is used in an algorithm to map the 3D scatter plot to the magnetic coordinates, as discussed above.

Claim 1:
A method comprising:
activating an electromagnetic tracking system (<NUM>) positioned around a head of a patient;
receiving a three-dimensional, 3D, image of a surface of the head of the patient and storing the 3D image as a scatter plot (<NUM>) in a memory (<NUM>), wherein the 3D image is acquired using a 3D camera comprising at least one optical sensor, characterised by the 3D camera further comprising at least one magnetic sensor, wherein the at least one magnetic sensor is a tri-axial sensor, TAS, and by the method further comprising:
determining a location and an orientation of the 3D camera when the 3D image was acquired via the at least one magnetic sensor based on known magnetic fields applied by one or more transmitters of the electromagnetic tracking system;
registering the scatter plot to magnetic coordinates of the electromagnetic tracking system using the determined location and orientation of the 3D camera, wherein the registered scatter plot is stored in the memory.