Patent Description:
There are several commonly applied methods for treating various maladies affecting organs including the liver, brain, heart, lung and kidney. Often, one or more imaging modalities, such as magnetic resonance imaging, ultrasound imaging, computed tomography (CT), as well as others are employed by clinicians to identify areas of interest within a patient and ultimately targets for treatment.

An endoscopic approach has proven useful in navigating to areas of interest within a patient, and particularly so for areas within luminal networks of the body such as the lungs. To enable the endoscopic, and more particularly the bronchoscopic, approach in the lungs, endobronchial navigation systems have been developed that use previously acquired MRI data or CT image data to generate a three dimensional rendering or volume of the particular body part such as the lungs. In particular, previously acquired images, acquired from an MRI scan or CT scan of the patient, are utilized to generate a three dimensional or volumetric rendering of the patient.

The resulting volume generated from the MRI scan or CT scan is then utilized to create a navigation plan to facilitate the advancement of a navigation catheter (or other suitable device) through a bronchoscope and a branch of the bronchus of a patient to an area of interest. Electromagnetic tracking may be utilized in conjunction with the CT data to facilitate guidance of the navigation catheter through the branch of the bronchus to the area of interest. In certain instances, the navigation catheter may be positioned within one of the airways of the branched luminal networks adjacent to, or within, the area of interest to provide access for one or more medical instruments.

A fluoroscopic imaging device is commonly located in the operating room during navigation procedures. The standard fluoroscopic imaging device may be used by a clinician to visualize and confirm the placement of a tool after it has been navigated to a desired location. However, although standard fluoroscopic images display highly dense objects such as metal tools and bones as well as large soft-tissue objects such as the heart, the fluoroscopic images have difficulty resolving small soft-tissue objects of interest such as lesions or tumors. Further, the fluoroscope image is only a two dimensional projection. In order to be able to see small soft-tissue objects in three dimensional space, an X-ray volumetric reconstruction is needed. Several solutions exist that provide three dimensional volume reconstruction of soft-tissues such as CT and Cone-beam CT which are extensively used in the medical world. These machines algorithmically combine multiple X-ray projections from known, calibrated X-ray source positions into three dimensional volume in which the soft-tissues are visible.

<CIT> discloses a device and method for determining the location of a catheter in set of x-ray images in which the intersection of a rays is used.

The disclosure relates to a system, apparatus, and method of detecting a catheter in a series of fluoroscopic images, determining the position of the catheter based on the detection, and updating the displayed position of the catheter in an electromagnetic navigation system for surgical procedures. The disclosure utilizes a combination of shallow neural network operators and deep neural network operators to detect catheter candidates in a fluoroscopic data set. Additionally, false-positive candidate detections are eliminated according to the methods described herein. The position data of the catheter is acquired from the fluoroscopic data and is used as a correction factor for the displayed electromagnetically tracked position of the catheter.

The system of the disclosure constructs a fluoroscopic-based 3D construction of a target area which includes the catheter and a target (e.g., soft-tissue object, lesion, tumor, etc.) in order to determine the location of the catheter relative to the target. In particular, the system identifies the position, orientation, angle, and distance of the catheter relative to the target in each fluoroscopic frame of the fluoroscopic data. This relative location data is used to update a displayed electromagnetic position of the catheter over a CT-based rendering, for example, of a patient's luminal network. With this updated display, a clinician is able to more accurately navigate and confirm placement of the catheter and other surgical tools relative to a target during an electromagnetic navigation procedure.

Aspects of the disclosure are described in detail with reference to the figures wherein like reference numerals identify similar or identical elements. As used herein, the term "distal" refers to the portion that is being described which is further from a user, while the term "proximal" refers to the portion that is being described which is closer to a user.

According to one aspect of the disclosure, a method for detecting a catheter in fluoroscopic data is provided. The method includes acquiring fluoroscopic data from a fluoroscopic sweep of a target area. The target area may be, for example, within a patient's luminal network. The fluoroscopic data includes 2D fluoroscopic frames of the target area captured from different perspectives. The method further includes performing an initial catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data, performing a secondary catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data. Additionally, the method includes eliminating false-positive catheter tip candidates of the secondary catheter detection by reconstructing a 3D position of the catheter tip and finding an intersecting point of rays corresponding to each 2D frame, and reweighing the catheter tip candidates of the secondary catheter detection based on a distance of the catheter tip candidate from a projected 3D point.

The initial catheter detection may include applying a shallow neural network operator and the secondary catheter detection may include applying a deep neural network operator. The secondary catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data may include considering the catheter tip candidates of the initial catheter detection. Reweighing the catheter tip candidates of the secondary catheter detection based on a distance of the catheter tip candidate from a projected 3D point may include decreasing a weight of a pixel corresponding to a candidate when the distance of the catheter tip candidate is far from the projected 3D point.

Additionally, the method may further include iteratively repeating eliminating false-positive detections by reconstructing a 3D position of the catheter tip and finding an intersecting point of rays corresponding to each 2D frame. Additionally, or alternatively, the method may include displaying a user interface for manually selecting the catheter tip in a 2D fluoroscopic frame of the fluoroscopic data prior to performing an initial catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data.

In another aspect, a method for detecting a catheter in fluoroscopic data during a surgical navigation procedure is provided. The method includes tracking an electromagnetic position of a catheter using electromagnetic coordinates during a navigation procedure of the catheter to a target area, displaying the tracked electromagnetic position of the catheter on a display of a 3D rendering, and acquiring fluoroscopic data from a fluoroscopic sweep of the target area. The fluoroscopic data includes 2D fluoroscopic frames of the target area captured from different perspectives. The target area may be, for example, within a patient's luminal network and the 3D rendering may be, for example, a 3D rendering of the patient's luminal network. The method further includes constructing fluoroscopic-based three dimensional volumetric data of the target area from the acquired fluoroscopic data including a three-dimensional construction of a soft-tissue target in the target area, acquiring, for each 2D frame of the fluoroscopic data, data of a position of the catheter relative to the three-dimensional construction of the soft-tissue, and registering the acquired data of the position of the catheter relative to the three-dimensional construction of the soft-tissue with the electromagnetic position of the catheter. Additionally, the method includes displaying the position of the catheter on the display of the 3D rendering based on the registration of the acquired data of the position of the catheter relative to the three-dimensional construction of the soft-tissue with the electromagnetic position of the catheter.

The method may further include performing at least one catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data. An initial catheter detection may include applying a shallow neural network operator and a secondary catheter detection may include applying a deep neural network operator. Additionally, or alternatively, the method may further include eliminating false-positive catheter tip candidates of the secondary catheter detection by reconstructing a 3D position of the catheter tip and finding an intersecting point of rays corresponding to each 2D frame. The secondary catheter tip detection may consider the candidates identified in the initial catheter detection. Additionally, or alternatively, the method may include reweighing the catheter tip candidates of the secondary catheter detection based on a distance of the catheter tip candidate from a projected 3D point.

In another aspect, a system for performing an electromagnetic surgical navigation procedure is provided. The system includes an electromagnetic tracking system having electromagnetic tracking coordinates, a catheter including a sensor configured to couple to the electromagnetic tracking system for detecting a position of the catheter in the electromagnetic coordinates, and a computing device operably coupled to the electromagnetic tracking system and the catheter.

The computing device is configured to display a navigation path to guide navigation of the catheter to a target area, display the position of the catheter in the electromagnetic coordinates on a 3D rendering, and acquire fluoroscopic data from a fluoroscopic sweep of the target area. The target area may be, for example, within a patient's luminal network and the 3D rendering may be, for example, a 3D rendering of the patient's luminal network. The fluoroscopic data includes 2D fluoroscopic frames of the target area captured from different perspectives. Additionally, the computing device is configured to perform an initial catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data, perform a secondary catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data, eliminate false-positive catheter tip candidates of the secondary catheter detection by reconstructing a 3D position of the catheter tip and finding an intersecting point of rays corresponding to each 2D frame, and reweigh the catheter tip candidates of the secondary catheter detection based on a distance of the catheter tip candidate from a projected 3D point.

The computing device may be configured to perform an initial catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data by applying a shallow neural network operator and perform a secondary catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data by applying a deep neural network operator. Additionally, or alternatively, the computing device may be configured to construct fluoroscopic-based three dimensional volumetric data of the target area from the acquired fluoroscopic data. The fluoroscopic-based three dimensional volumetric data includes a three-dimensional construction of a soft-tissue target in the target area.

In an aspect, the computing device is further configured to acquire, for each 2D frame of the fluoroscopic data, data of a position of the catheter relative to the three-dimensional construction of the soft-tissue, register the acquired data of the position of the catheter relative to the three-dimensional construction of the soft-tissue with the electromagnetic position of the catheter, and display the position of the catheter on the display of the 3D rendering based on the registration of the acquired data of the position of the catheter relative to the three-dimensional construction of the soft-tissue with the electromagnetic position of the catheter.

Various aspects and embodiments of the disclosure are described hereinbelow with references to the drawings, wherein:.

In order to navigate tools to a remote soft-tissue target for biopsy or treatment, both the tool and the target should be visible in some sort of a three dimensional guidance system. The majority of these systems use some X-ray device to see through the body. For example, a CT machine can be used with iterative scans during procedure to provide guidance through the body until the tools reach the target. This is a tedious procedure as it requires several full CT scans, a dedicated CT room and blind navigation between scans. In addition, each scan requires the staff to leave the room. Another option is a Cone-beam CT machine which is available in some operation rooms and is somewhat easier to operate, but is expensive and like the CT only provides blind navigation between scans, requires multiple iterations for navigation and requires the staff to leave the room.

Accordingly, there is a need for a system that can achieve the benefits of the CT and Cone-beam CT three dimensional image guidance without the underlying costs, preparation requirements, and radiation side effects associated with these systems.

The disclosure is directed to a system and method for catheter detection in fluoroscopic data and constructing local three dimensional volumetric data, in which small soft-tissue objects are visible, from the fluoroscopic data captured by a standard fluoroscopic imaging device available in most procedure rooms. The catheter detection and constructed fluoroscopic-based local three dimensional volumetric data may be used for guidance, navigation planning, improved navigation accuracy, navigation confirmation, and treatment confirmation.

<FIG> depicts an Electromagnetic Navigation (EMN) system <NUM> configured for reviewing CT image data to identify one or more targets, planning a pathway to an identified target (planning phase), navigating an extended working channel (EWC) <NUM> of a catheter guide assembly <NUM> to a target (navigation phase) via a user interface, and confirming placement of the EWC <NUM> relative to the target. One such EMN system is the ELECTROMAGNETIC NAVIGATION BRONCHOSCOPY® system currently sold by Medtronic PLC. The target may be tissue of interest identified by review of the CT image data during the planning phase. Following navigation, a medical instrument, such as a biopsy tool or other tool, may be inserted into the EWC <NUM> to obtain a tissue sample (or perform any treatment) from the tissue located at, or proximate to, the target.

As shown in <FIG>, EWC <NUM> is part of a catheter guide assembly <NUM>. In practice, the EWC <NUM> is inserted into bronchoscope <NUM> for access to a luminal network of the patient "P. " Specifically, EWC <NUM> of catheter guide assembly <NUM> may be inserted into a working channel of bronchoscope <NUM> for navigation through a patient's luminal network. A distal portion of the EWC <NUM> includes a sensor <NUM>. The position and orientation of the sensor <NUM> relative to the reference coordinate system, and thus the distal portion of the EWC <NUM>, within an electromagnetic field can be derived. Catheter guide assemblies <NUM> are currently marketed and sold by Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, or EDGE™ Procedure Kits, and are contemplated as useable with the disclosure. For a more detailed description of the catheter guide assemblies <NUM>, reference is made to commonly-owned <CIT>, <CIT>, and <CIT>. System <NUM> and its components are described in greater detail below.

The following description of <FIG> provides an exemplary workflow of using the components of system <NUM>, including the computing device <NUM> and the fluoroscopic imaging device <NUM>, to accurately detect a catheter in the fluoroscopic data (for example, by eliminating any false-positive detections). Additionally, the following description of <FIG> provides an exemplary workflow of using system <NUM> to construct local, fluoroscopic-based, three dimensional volumetric data of a desired region of interest, register the position of the detected catheter relative to a target with an electromagnetic coordinate system tracking the catheter, and update a displayed position of the catheter relative to the target using the registration. The systems and methods described herein may be useful for visualizing a particular target region of a patient utilizing imaging devices which are commonly located within a surgical setting during EMN procedures, thereby obviating the need for subsequent MRI or CT scans, and confirming the placement of the catheter relative to the target by providing an updated and more accurate display of the catheter in a 3D rendering of a patient's luminal network.

<FIG> illustrates a method for detecting a catheter in fluoroscopic data and eliminating false-positive detections in conjunction with a system such as the system <NUM> described in <FIG> which will now be described with particular detail. Although the methods illustrated and described herein are illustrated and described as being in a particular order and requiring particular steps, any of the methods may include some or all of the steps and may be implemented in any order. Additionally, any or all of the steps of any of the methods described herein may be carried out by computing device <NUM> of system <NUM> or any other component or combination of components of system <NUM>.

Method <NUM> begins with step <NUM>, where a catheter (or any surgical tool) is navigated to a region of interest within a patient's luminal network. The navigation in step <NUM> may utilize the electromagnetic tracking and pathway plans described above. In step <NUM>, a fluoroscopic sweep of the target area is performed to acquire fluoroscopic data of the target area with the catheter positioned therein. In particular, in step <NUM>, a fluoroscope is positioned about the patient such that fluoroscopic images of the target area may be captured along the entire sweep (e.g. ranging from the angles of -<NUM> degrees and +<NUM> degrees).

After the fluoroscopic data is acquired, in step <NUM> an initial catheter detection is performed. In the initial catheter detection, computing device performs an operator in each 2D frame of the fluoroscopic data to detect possible catheter tip candidates in each 2D frame. In one aspect, step <NUM> includes utilizing a shallow neural network operator (e.g., four layers). Utilizing the shallow neural network operator provides the advantage of fast completion (approximately <NUM> second for every <NUM>2D frames of the fluoroscopic data), but simultaneously provides the disadvantage of identifying many false-positive catheter tip detections. In particular, <FIG> illustrates an example 2D frame <NUM> of fluoroscopic data before a shallow neural network operator is applied in step <NUM>, and <FIG> illustrates the resulting image <NUM> after the shallow neural network operator is applied in step <NUM> showing all of the catheter tip candidates detected. As seen in <FIG>, while the true catheter tip <NUM> in the frame has been detected as a candidate, many false-positive <NUM> candidate detections have been made as well.

In step <NUM>, a secondary catheter detection operation is performed to detect catheter tip candidates in each 2D frame of the fluoroscopic data. In one aspect, step <NUM> includes utilizing a second neural network operator, in this case a deep neural network operator (e.g., eleven layers). Utilizing the deep neural network operator takes a longer period of time to complete when compared to the shallow neural network operator, but results in fewer false-positive candidate detections. Additionally, the deep neural network operator alone, though not providing as many false-positive candidate detections as the shallow neural network operator, in some instances fails to detect the actual catheter tip as a candidate. <FIG> illustrates an example 2D frame <NUM> of fluoroscopic data before a deep neural network operator is applied in step <NUM>, and <FIG> illustrates the resulting image <NUM> after the deep neural network operator is applied in step <NUM>. As seen in <FIG>, the true catheter tip <NUM> in the image has been detected as a candidate and only few false-positive <NUM> candidate detections have been made.

In view of the above, one benefit of the disclosed method is that utilizing only one of the initial catheter detection in step <NUM> or the secondary catheter detection in step <NUM> could be insufficient for accurately identifying the catheter tip in the fluoroscopic data. As explained above, if the shallow neural network operator of step <NUM> were performed alone, the catheter will be detected in almost all of the 2D frames, but many false-positives will be detected as well, leading to an inaccurate catheter detection. Additionally, if the deep neural network operator were performed alone, then few false positives will be detected, but some actual catheter detections will be missed, also leading to an inaccurate catheter detection. Accordingly, method <NUM> includes applying both the shallow neural network operator and the deep neural network operator to ensure an accurate catheter detection. Additionally, if the deep neural network has a misdetection in a fluoro video frame, the catheter won't be detected in that frame (even if it was found in the initial catheter detection). Thus, one thing that prevents misdetection of the catheter in this case are valid detections from other frames, as described in further detail below. Another benefit of utilizing two separate neural networks is that splitting the detector to an initial network and a second deeper network is may enhance runtime optimization as described above.

In one aspect, in step <NUM>, the deep neural network operator considers the candidates from the initial catheter detection of step <NUM>. That is, in utilizing the candidates detected in the initial catheter detection, it can be ensured that the actual catheter will not be missed in the deep neural network operator and will always be identified as a catheter candidate after performance of the secondary catheter detection. In particular, the initial catheter detection outputs a "catheter probability" for each pixel in each fluoroscopic frame. Only pixels with high probability (above some fixed threshold) are considered as catheter candidates after the initial catheter detection. An image patch is extracted around each pixel candidate (having the catheter probability exceeding the threshold) and is input to the second catheter detection (e.g., the deep neural network).

In step <NUM>, false-positive detections are eliminated by reconstructing 3D position of the catheter tip. In particular, with reference to <FIG>, the intersecting point <NUM> of the rays 503a, 503b, 503c. 503n extending from the candidates of each 2D frame 501a, 501b, 501c. 501n is the 3D reconstructed position of the catheter tip. In step <NUM>, when the number of rays intersecting at the same point for a particular catheter candidate exceeds a preconfigured threshold, it is determined that the candidate is not a false-positive and is actually the catheter tip. In particular, in order to confirm that the candidate is not a false-positive detection, a certain numbers of rays must be intersecting at the same intersection point. When the number of rays intersecting at the same point for a particular catheter candidate is below a preconfigured threshold, then it is determined that the candidate is a false-positive detection and not the catheter tip. Method <NUM> may optionally also include step <NUM>, where the process of eliminating false-positive candidate detections is iteratively repeated. Iteratively repeating the elimination of false-positive candidate detections in step <NUM>, enables the preconfigured threshold for intersecting rays to be set at an optimal value, leading to more reliable and accurate elimination of false-positive candidate detections.

In step <NUM>, the catheter candidate detections are reweighed according to their distance from the projected 3D point in step <NUM>. In particular, with reference to <FIG> illustrates an example image <NUM> after steps <NUM>-<NUM> are completed and <FIG> illustrates an image <NUM> after the reweighing of step <NUM> in performed. The image <NUM> in <FIG> includes as catheter tip candidates, the actual catheter tip <NUM> and a false-positive <NUM>. The image <NUM> in <FIG> includes only the actual catheter tip <NUM> as a candidate and does not include any false-positives. In step <NUM>, each voxel is assigned a weight according to the number of rays intersecting through the voxel and the distance of the voxel from the 3D position of the catheter tip identified in step <NUM>. In step <NUM>, any candidate that is far in distance from the 3D position of the catheter tip identified in step <NUM> will be assigned a lower weight and will be identified as a false-positive candidate detection and will be removed. On the other hand, any candidate that is close in distance to the 3D position of the catheter tip identified in step <NUM> will be assigned a higher weight and will be identified as a catheter tip candidate detection and will remain.

<FIG> illustrates a method for performing an electromagnetic surgical navigation procedure in conjunction with a system such as the system <NUM> described in <FIG> utilizing method <NUM>, which will now be described with particular detail and will be referred to as method <NUM>. Although the methods illustrated and described herein are illustrated and described as being in a particular order and requiring particular steps, any of the methods may include some or all of the steps and may be implemented in any order. Additionally, any or all of the steps of any of the methods described herein may be carried out by computing device <NUM> of system <NUM> or any other component or combination of components of system <NUM>.

Method <NUM> begins at step <NUM> where the electromagnetic position of a catheter is tracked during navigation of the catheter to a target area within a patient's luminal network. In step <NUM>, the tracked electromagnetic position of the catheter is displayed during the navigation on a display of a CT-based 3D rendering of the patient's luminal network. Displaying the electromagnetic position of the catheter on the 3D rendering of the patient's luminal network assists the clinician in navigating to the region of interest or target area. Step <NUM> may additionally include displaying a predetermined route or path to follow through the patient's luminal network to navigate to the target area. Such a display of the pathway or route enables a clinician to identify when the catheter has deviated from the desired path, the progress of the navigation, and when the target is reached.

Once the catheter is navigated to the target area as displayed in step <NUM>, it is possible that the actual position of the catheter within the patient's luminal network is not exactly as displayed in step <NUM>, because the displayed position of the catheter is based only on the electromagnetically tracked position of the catheter's sensor and is displayed over an old CT-data set. Targets within the target area, and other such anatomical features, may have changed since the time of acquisition of the CT data and the electromagnetic tracking is subject to interference and unreliability. Additionally, the display in step <NUM> merely shows a representation of the catheter, and not the actual catheter within the luminal network relative to structures therein. To more accurately display the actual position of the catheter within the patient's actual luminal network, method <NUM> proceeds to step <NUM>.

In step <NUM>, a fluoroscopic sweep of the target area is performed to acquire fluoroscopic data of the target area with the catheter positioned therein. In particular, in step <NUM>, a fluoroscope is positioned about the patient such that fluoroscopic images of the target area may be captured along a sweep (e.g. ranging from the angles of -<NUM> degrees and +<NUM> degrees). In step <NUM>, the catheter tip is detected throughout the frames of the fluoroscopic data. In one aspect, step <NUM> utilizes some or all of the steps of method <NUM> for the catheter detection.

In step <NUM>, a fluoroscopic-based 3D rendering of the target area with the catheter positioned therein is constructed. Such a fluoroscopic-based 3D construction enables visualization of objects that are not otherwise visible in the fluoroscopic data itself. For example, small soft-tissue objects, such as tumors or lesions, are not visible in the fluoroscopic data, but are visible in the fluoroscopic-based 3D rendering of the target area. Further details regarding the construction of step <NUM> may be found in <CIT>, entitled System and Method for Navigating to Target and Performing Procedure on Target Utilizing Fluoroscopic-Based Local Three Dimensional Volume Reconstruction, and <CIT>, entitled System and Method for Local Three Dimensional Volume Reconstruction Using a Standard Fluoroscope.

In step <NUM>, position data of the catheter relative to the target in the target area in each frame of the fluoroscopic data is acquired. In particular, the construction of the fluoroscopic-based 3D rendering in step <NUM> enables the visibility of the target (e.g., soft tissue object, lesion, tumor, etc.) and the location of the target to be determined or otherwise ascertained. As described above, prior to the construction of the fluoroscopic-based 3D rendering in step <NUM>, the soft-tissue objects (e.g., target, tumor, lesion, etc.) are not visible and the location of the object(s), relative to the catheter cannot be determined. In one aspect, in step <NUM>, the position, orientation, and distance of the catheter relative to the target are determined for each slice of the fluoroscopic-based 3D rendering. Additionally, the position data acquired in step <NUM> may be correlated to the fluoroscopic data acquired in step <NUM>.

In step <NUM>, the position data acquired in step <NUM> is registered to the electromagnetic tracked coordinates of the catheter. In one aspect, in step <NUM>, the system displays the catheter on the CT data by employing an "Antenna-to-CT" registration, where the catheter position (and optionally orientation) in antenna coordinates is continuously calculated by the electromagnetic localization algorithms. The position is then transformed to CT coordinates. This registration may be based on the first bifurcations of the airway tree (main carina and bronchi). Due to this (and other factors, such as patient sedation, and different patient pose during CT and bronchoscopic procedure) the registration is less accurate in the periphery of the lungs. The registration is corrected for the target area by composing one or more of the following registrations: <NUM>) Antenna-to-fluoro, where the catheter position is known in both antenna coordinates (using electromagnetic localization) and fluoro coordinates (automatic catheter detection described in the preceding steps), and is used as a basis for the registration; and/or <NUM>) Fluoro-to-CT, where the target's position is marked on the CT data pre-operatively and then also marked on the fluoro-based 3D reconstruction inter-operatively. After performing calculations for the above registrations, a new catheter position in antenna coordinates is transformed to fluoro coordinates by employing the "Antenna-to-fluoro" registration, which is then transformed to CT coordinates by employing the "Fluoro-to-CT" registration.

In step <NUM>, the displayed position of the catheter on the CT-based 3D rendering of the patient's luminal network is updated using the position data acquired in step <NUM>. In particular, the position, orientation, and distance of the catheter relative to the target in the fluoroscopic-based 3D data set are compared to the position, orientation, and distance of the displayed catheter relative to the target in the CT-based 3D rendering. In one aspect, the orientation, position, and distance of the displayed catheter relative to the target in the CT-based rendering is updated to correspond or match with the orientation, position, and distance of the catheter relative to the target in the fluoroscopic-based 3D data set.

Referring back to <FIG>, EMN system <NUM> generally includes an operating table <NUM> configured to support a patient "P;" a bronchoscope <NUM> configured for insertion through the patient's "P's" mouth into the patient's "P's" airways; monitoring equipment <NUM> coupled to bronchoscope <NUM> (e.g., a video display, for displaying the video images received from the video imaging system of bronchoscope <NUM>); a tracking system <NUM> including a tracking module <NUM>, a plurality of reference sensors <NUM> and a transmitter mat <NUM>; and a computing device <NUM> including software and/or hardware used to facilitate identification of a target, pathway planning to the target, navigation of a medical instrument to the target, and confirmation of placement of an EWC <NUM>, or a suitable device therethrough, relative to the target.

A fluoroscopic imaging device <NUM> capable of acquiring fluoroscopic or x-ray images or video of the patient "P" is also included in this particular aspect of system <NUM>. The fluoroscopic data (e.g., images, series of images, or video) captured by the fluoroscopic imaging device <NUM> may be stored within the fluoroscopic imaging device <NUM> or transmitted to computing device <NUM> for storage, processing, and display. Additionally, the fluoroscopic imaging device <NUM> may move relative to the patient "P" so that images may be acquired from different angles or perspectives relative to the patient "P" to create a fluoroscopic video from a fluoroscopic sweep. Fluoroscopic imaging device <NUM> may include a single imaging device or more than one imaging device. In embodiments including multiple imaging devices, each imaging device may be a different type of imaging device or the same type. Further details regarding the imaging device <NUM> are described in <CIT>.

Computing device <NUM> may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions stored on the storage medium. The computing device <NUM> may further include a database configured to store patient data, CT data sets including CT images, fluoroscopic data sets including fluoroscopic images and video, navigation plans, and any other such data. Although not explicitly illustrated, the computing device <NUM> may include inputs, or may otherwise be configured to receive, CT data sets, fluoroscopic images/video and other data described herein. Additionally, computing device <NUM> includes a display configured to display graphical user interfaces.

With respect to the planning phase, computing device <NUM> utilizes previously acquired CT image data for generating and viewing a three dimensional model of the patient's "P's" airways, enables the identification of a target on the three dimensional model (automatically, semiautomatically, or manually), and allows for determining a pathway through the patient's "P's" airways to tissue located at and around the target. More specifically, CT images acquired from previous CT scans are processed and assembled into a three dimensional CT volume, which is then utilized to generate a three dimensional model of the patient's "P's" airways. The three dimensional model may be displayed on a display associated with computing device <NUM>, or in any other suitable fashion. Using computing device <NUM>, various views of the three dimensional model or enhanced two dimensional images generated from the three dimensional model are presented. The enhanced two dimensional images may possess some three dimensional capabilities because they are generated from three dimensional data. The three dimensional model may be manipulated to facilitate identification of target on the three dimensional model or two dimensional images, and selection of a suitable pathway (e.g., route to be following during navigation) through the patient's "P's" airways to access tissue located at the target can be made. Once selected, the pathway plan, three dimensional model, and images derived therefrom, can be saved and exported to a navigation system for use during the navigation phase(s). One such planning software is the ILOGIC® planning suite currently sold by Medtronic PLC.

With respect to the navigation phase, a six degrees-of-freedom electromagnetic tracking system <NUM>, e.g., similar to those disclosed in <CIT>,<CIT>, and published <CIT> and <CIT>, or other suitable positioning measuring system, is utilized for performing registration of the images and the pathway for navigation, although other configurations are also contemplated. Tracking system <NUM> includes a tracking module <NUM>, a plurality of reference sensors <NUM>, and a transmitter mat <NUM>. Tracking system <NUM> is configured for use with a sensor <NUM> of catheter guide assembly <NUM> to track the electromagnetic position thereof within an electromagnetic coordinate system.

Transmitter mat <NUM> is positioned beneath patient "P. " Transmitter mat <NUM> generates an electromagnetic field around at least a portion of the patient "P" within which the position of a plurality of reference sensors <NUM> and the sensor element <NUM> can be determined with use of a tracking module <NUM>. One or more of reference sensors <NUM> are attached to the chest of the patient "P. " The six degrees of freedom coordinates of reference sensors <NUM> are sent to computing device <NUM> (which includes the appropriate software) where they are used to calculate a patient coordinate frame of reference. Registration, as detailed below, is generally performed to coordinate locations of the three dimensional model and two dimensional images from the planning phase with the patient's "P's" airways as observed through the bronchoscope <NUM>, and allow for the navigation phase to be undertaken with precise knowledge of the location of the sensor <NUM>, even in portions of the airway where the bronchoscope <NUM> cannot reach. Further details of such a registration technique and their implementation in luminal navigation can be found in <CIT>.

Registration of the patient's "P's" location on the transmitter mat <NUM> is performed by moving sensor <NUM> through the airways of the patient's "P. " More specifically, data pertaining to locations of sensor <NUM>, while EWC <NUM> is moving through the airways, is recorded using transmitter mat <NUM>, reference sensors <NUM>, and tracking module <NUM>. A shape resulting from this location data is compared to an interior geometry of passages of the three dimensional model generated in the planning phase, and a location correlation between the shape and the three dimensional model based on the comparison is determined, e.g., utilizing the software on computing device <NUM>. In addition, the software identifies non-tissue space (e.g., air filled cavities) in the three dimensional model. The software aligns, or registers, an image representing a location of sensor <NUM> with a the three dimensional model and two dimensional images generated from the three dimension model, which are based on the recorded location data and an assumption that sensor <NUM> remains located in non-tissue space in the patient's "P's" airways. Alternatively, a manual registration technique may be employed by navigating the bronchoscope <NUM> with the sensor <NUM> to pre-specified locations in the lungs of the patient "P", and manually correlating the images from the bronchoscope to the model data of the three dimensional model.

Following registration of the patient "P" to the image data and pathway plan, a user interface is displayed in the navigation software which sets forth the pathway that the clinician is to follow to reach the target. One such navigation software is the ILOGIC® navigation suite currently sold by Medtronic PLC.

Once EWC <NUM> has been successfully navigated proximate the target as depicted on the user interface, the EWC <NUM> is in place as a guide channel for guiding medical instruments including without limitation, optical systems, ultrasound probes, marker placement tools, biopsy tools, ablation tools (i.e., microwave ablation devices), laser probes, cryogenic probes, sensor probes, and aspirating needles to the target.

With the above-described updated position of the catheter relative to a target in a target area, a clinician is able to more accurately navigate to a region of interest or target within a patient's luminal network. Errors in the electromagnetically tracked and displayed location of the catheter in the CT-based 3D rendering of the patient's luminal network are corrected based on near real-time position data extracted from fluoroscopic-based 3D data of target area.

From the foregoing and with reference to the various figure drawings, those skilled in the art will appreciate that certain modifications can also be made to the disclosure without departing from the scope of the same. For example, although the systems and methods are described as usable with an EMN system for navigation through a luminal network such as the lungs, the systems and methods described herein may be utilized with systems that utilize other navigation and treatment devices such as percutaneous devices. Additionally, although the above-described system and method is described as used within a patient's luminal network, it is appreciated that the above-described systems and methods may be utilized in other target regions such as the liver. Further, the above-described systems and methods are also usable for transthoracic needle aspiration procedures.

Detailed embodiments of the disclosure are disclosed herein. However, the disclosed embodiments are merely examples of the disclosure, which may be embodied in various forms and aspects. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the disclosure in virtually any appropriately detailed structure.

As can be appreciated a medical instrument such as a biopsy tool or an energy device, such as a microwave ablation catheter, that is positionable through one or more branched luminal networks of a patient to treat tissue may prove useful in the surgical arena and the disclosure is directed to systems and methods that are usable with such instruments and tools. Access to luminal networks may be percutaneous or through natural orifice using navigation techniques. Additionally, navigation through a luminal network may be accomplished using image-guidance. These image-guidance systems may be separate or integrated with the energy device or a separate access tool and may include MRI, CT, fluoroscopy, ultrasound, electrical impedance tomography, optical, and/or device tracking systems. Methodologies for locating the access tool include EM, IR, echolocation, optical, and others. Tracking systems may be integrated to an imaging device, where tracking is done in virtual space or fused with preoperative or live images. In some cases the treatment target may be directly accessed from within the lumen, such as for the treatment of the endobronchial wall for COPD, Asthma, lung cancer, etc. In other cases, the energy device and/or an additional access tool may be required to pierce the lumen and extend into other tissues to reach the target, such as for the treatment of disease within the parenchyma. Final localization and confirmation of energy device or tool placement may be performed with imaging and/or navigational guidance using a standard fluoroscopic imaging device incorporated with methods and systems described above.

Claim 1:
A system for performing an electromagnetic surgical navigation procedure, the system comprising:
an electromagnetic tracking system (<NUM>) having electromagnetic tracking coordinates;
a catheter including a sensor configured to couple to the electromagnetic tracking system for detecting a position of the catheter in the electromagnetic tracking coordinates; and
a computing device operably coupled to the electromagnetic tracking system and the catheter, the computing device configured to:
display a navigation path to guide navigation of the catheter to a target area;
display the position of the catheter in the electromagnetic coordinates on a 3D rendering;
acquire fluoroscopic data from a fluoroscopic sweep of the target area, the fluoroscopic data including 2D fluoroscopic frames of the target area captured from different perspectives;
perform an initial catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data;
perform a secondary catheter detection for catheter tip candidates in each 2D frame of the fluoroscopic data;
reconstruct a 3D position of the catheter tip and finding an intersection point of rays extending from the candidates of each 2D frame (501a, 501b, 501c, 501n) ; (503a, 503b, 503c, 503n)
eliminate as false-positive catheter tip candidates (<NUM>) of the secondary catheter detections whose number of rays determined to intersect the intersection point fall below a preconfigured threshold.