Patent ID: 12208220

DETAILED DESCRIPTION

Catheters and catheter like devices, such as endoscopes, are used in a myriad of medical procedures. These flexible devices are typically used to navigate through luminal networks of the body including the vasculature, airways, and digestive systems. In accordance with the disclosure, to aide in navigating to a specific location, the distal tip of the catheter can be articulated, deflected, or rotated by a user through controls on the catheter proximal end outside the body. These manipulations allow the tip to point towards and enter branching structures. Upon arrival at the desired anatomic location a medical procedure may be performed such as lesion observation, heart valve replacement, stent or pacemaker deployment, radio frequency or microwave ablation, placement of chemotherapy drugs, and a variety of others.

In accordance with the disclosure, a 3D model of a patient's lungs or another suitable portion of the anatomy, may be generated from previously acquired scans, such as CT or MRI scans. The 3D model and related scan data are used to identify targets, e.g., potential lesions for biopsy or treatment, and to generate a pathway plan through the anatomy to reach the targets.

Once the pathway plan is generated and accepted by a clinician, that pathway plan may be utilized by a navigation system to drive a catheter or catheter like device along the pathway plan through the anatomy to reach the desired target. The driving of the catheter along the pathway plan may be manual or it may be robotic, or a combination of both. Manual systems include the ILLUMISITE navigation system sold by Medtronic PLC, robotic systems include the ION system sold by Intuitive Surgical Inc. and the MONARCH system sold by Auris Health, Inc. In a single procedure planning, registration of the pathway plan to the patient, and navigation are performed to enable a medical device, e.g., a catheter to be navigated along the planned path to reach a target, e.g., a lesion, so that a biopsy or treatment of the target can be completed.

FIG.1is a perspective view of an exemplary system for facilitating navigation of a medical device, e.g., a catheter to a soft-tissue target via airways of the lungs. System100may be further configured to construct fluoroscopic based three-dimensional volumetric data of the target area from 2D fluoroscopic images to confirm navigation to a desired location. System100may be further configured to facilitate approach of a medical device to the target area by using Electromagnetic Navigation (EMN) and for determining the location of a medical device with respect to the target. One such EMN system is the ILLUMISITE system currently sold by Medtronic PLC, though other systems for intraluminal navigation are considered within the scope of the disclosure including shape sensing technology which detect the shape of the distal portion of the catheter and match that shape to the shape of the luminal network in a 3D model.

One aspect of the system100is a software component for reviewing of computed tomography (CT) image scan data that has been acquired separately from system100. The review of the CT image data allows a user to identify one or more targets, plan a pathway to an identified target (planning phase), navigate a catheter102to the target (navigation phase) using a user interface on computing device122, and confirming placement of a sensor104relative to the target. The target may be tissue of interest identified by review of the CT image data during the planning phase. Following navigation, a medical device, such as a biopsy tool or other tool, may be inserted into catheter102to obtain a tissue sample from the tissue located at, or proximate to, the target.

As shown inFIG.1, catheter102is part of a catheter guide assembly106. In one embodiment, catheter102is inserted into a bronchoscope108for access to a luminal network of the patient P. Specifically, catheter102of catheter guide assembly106may be inserted into a working channel of bronchoscope108for navigation through a patient's luminal network. The he catheter102may itself include imaging capabilities and the bronchoscope108is not strictly required. A locatable guide (LG)110(a second catheter), including a sensor104may be inserted into catheter102and locked into position such that sensor104extends a desired distance beyond the distal tip of catheter102. The position and orientation of sensor104relative to a reference coordinate system, and thus the distal portion of catheter102, within an electromagnetic field can be derived. Catheter guide assemblies106are currently marketed and sold by Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, or EDGE™ Procedure Kits, and are contemplated as useable with the disclosure.

System100generally includes an operating table112configured to support a patient P, a bronchoscope108configured for insertion through patient P's mouth into patient P's airways; monitoring equipment114coupled to bronchoscope108or catheter102(e.g., a video display, for displaying the video images received from the video imaging system of bronchoscope108or the catheter102); a locating or tracking system114including a locating module116, a plurality of reference sensors18and a transmitter mat120including a plurality of incorporated markers; and a computing device122including software and/or hardware used to facilitate identification of a target, pathway planning to the target, navigation of a medical device to the target, and/or confirmation and/or determination of placement of catheter102, or a suitable device therethrough, relative to the target.

In accordance with aspects of the disclosure, the visualization of intra-body navigation of a medical device, e.g., a biopsy tool, towards a target, e.g., a lesion, may be a portion of a larger workflow of a navigation system. A fluoroscopic imaging device124capable of acquiring fluoroscopic or x-ray images or video of the patient P is also included in this particular aspect of system100. The images, sequence of images, or video captured by fluoroscopic imaging device124may be stored within fluoroscopic imaging device124or transmitted to computing device122for storage, processing, and display. Additionally, fluoroscopic imaging device124may move relative to the patient P so that images may be acquired from different angles or perspectives relative to patient P to create a sequence of fluoroscopic images, such as a fluoroscopic video. The pose of fluoroscopic imaging device124relative to patient P while capturing the images may be estimated via markers incorporated with the transmitter mat120. The markers are positioned under patient P, between patient P and operating table112and between patient P and a radiation source or a sensing unit of fluoroscopic imaging device124. The markers incorporated with the transmitter mat120may be two separate elements which may be coupled in a fixed manner or alternatively may be manufactured as a single unit. Fluoroscopic imaging device124may include a single imaging device or more than one imaging device.

Computing device122may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions stored on the storage medium. Computing device122may further include a database configured to store patient data, CT data sets including CT images, fluoroscopic data sets including fluoroscopic images and video, fluoroscopic 3D reconstruction, navigation plans, and any other such data. Although not explicitly illustrated, computing device122may include inputs, or may otherwise be configured to receive, CT data sets, fluoroscopic images/video and other data described herein. Additionally, computing device122includes a display configured to display graphical user interfaces. Computing device122may be connected to one or more networks through which one or more databases may be accessed.

With respect to a planning phase, computing device122utilizes previously acquired CT or MM image data for generating and viewing a three-dimensional model or rendering of patient P's airways, enables the identification of a target on the three-dimensional model (automatically, semi-automatically, or manually), and allows for determining a pathway through patient P's airways to tissue located at and around the target. More specifically, CT images acquired from previous CT or MM scans are processed and assembled into a three-dimensional volume, which is then utilized to generate a three-dimensional model of patient P's airways. The three-dimensional model may be displayed on a display associated with computing device122, or in any other suitable fashion. Using computing device122, various views of the three-dimensional model or enhanced two-dimensional images generated from the three-dimensional model are presented. The enhanced two-dimensional images may possess some three-dimensional capabilities because they are generated from three-dimensional data. The three-dimensional model may be manipulated to facilitate identification of target on the three-dimensional model or two-dimensional images, and selection of a suitable pathway through patient P's airways to access tissue located at the target can be made. Once selected, the pathway plan, three-dimensional model, and images derived therefrom, can be saved and exported to a navigation system for use during the navigation phase(s).

With respect to the navigation phase, a six degrees-of-freedom electromagnetic locating or tracking system114, or other suitable system for determining position and orientation of a distal portion of the catheter102, is utilized for performing registration of the images and the pathway for navigation. Tracking system114includes the tracking module116, a plurality of reference sensors118, and the transmitter mat120(including the markers). Tracking system114is configured for use with a locatable guide110and particularly sensor104. As described above, locatable guide110and sensor104are configured for insertion through catheter102into patient P's airways (either with or without bronchoscope108) and are selectively lockable relative to one another via a locking mechanism.

Transmitter mat120is positioned beneath patient P. Transmitter mat120generates an electromagnetic field around at least a portion of the patient P within which the position of a plurality of reference sensors118and the sensor104can be determined with use of a tracking module116. A second electromagnetic sensor126may also be incorporated into the end of the catheter102. The second electromagnetic sensor126may be a five degree-of-freedom sensor or a six degree-of-freedom sensor. One or more of reference sensors118are attached to the chest of the patient P. Registration is generally performed to coordinate locations of the three-dimensional model and two-dimensional images from the planning phase, with the patient P's airways as observed through the bronchoscope108, and allow for the navigation phase to be undertaken with knowledge of the location of the sensor104.

Registration of the patient P's location on the transmitter mat120may be performed by moving sensor104through the airways of the patient P. More specifically, data pertaining to locations of sensor104, while locatable guide110is moving through the airways, is recorded using transmitter mat120, reference sensors118, and tracking system114. A shape resulting from this location data is compared to an interior geometry of passages of the three-dimensional model generated in the planning phase, and a location correlation between the shape and the three-dimensional model based on the comparison is determined, e.g., utilizing the software on computing device122. In addition, the software identifies non-tissue space (e.g., air filled cavities) in the three-dimensional model. The software aligns, or registers, an image representing a location of sensor104with the three-dimensional model and/or two-dimensional images generated from the three-dimension model, which are based on the recorded location data and an assumption that locatable guide110remains located in non-tissue space in patient P's airways. Alternatively, a manual registration technique may be employed by navigating the bronchoscope108with the sensor104to pre-specified locations in the lungs of the patient P, and manually correlating the images from the bronchoscope to the model data of the three-dimensional model.

Though described herein with respect to EMN systems using EM sensors, the instant disclosure is not so limited and may be used in conjunction with flexible sensor, ultrasonic sensors, or without sensors. Additionally, the methods described herein may be used in conjunction with robotic systems such that robotic actuators drive the catheter102or bronchoscope108proximate the target.

In accordance with the disclosure, the catheter102and its articulation and orientation relative to a target is achieved using a catheter drive mechanism300. One example of such a drive mechanism can be seen inFIG.3Awhich depicts a housing including three drive motors to manipulate a catheter extending therefrom in 5 degrees of freedom (e.g., left, right, up, down, and rotation). Other types of drive mechanisms including fewer or more degrees of freedom and other manipulation techniques may be employed without departing from the scope of the disclosure.

As noted above,FIG.3depicts a drive mechanism300housed in a body301and mounted on a bracket302which integrally connects to the body301. The catheter102connects to and in one embodiment forms an integrated unit with internal casings304aand304band connects to a spur gear306. This integrated unit is, in one embodiment rotatable in relation to the housing301, such that the catheter102, internal casings304a-b, and spur gear306can rotate about shaft axis “z”. The catheter102and integrated internal casings304a-bare supported radially by bearings308,310, and312. Though drive mechanism300is described in detail here, other drive mechanisms may be employed to enable a robot or a clinician to drive the catheter to a desired location without departing from the scope of the disclosure.

An electric motor314R, may include an encoder for converting mechanical motion into electrical signals and providing feedback to the computing device122. Further, the electric motor314R (R indicates this motor if for inducing rotation of the catheter102) may include an optional gear box for increasing or reducing the rotational speed of an attached spur gear315mounted on a shaft driven by the electric motor314R. Electric motors314LR (LR referring to left-right movement of an articulating portion317of the catheter102) and314UD (referring to up-down movement of the articulating portion317), each motor optionally includes an encoder and a gearbox. Respective spur gears316and318drive up-down and left-right steering cables, as will be described in greater detail below. All three electric motors314R, LR, and UD are securely attached to the stationary frame302, to prevent their rotation and enable the spur gears315,316, and318to be driven by the electric motors.

FIG.3Bdepicts details of the mechanism causing articulating portion317of catheter102to articulate. Specifically, the following depicts the manner in which the up-down articulation is contemplated in one aspect of the disclosure. Such a system alone, coupled with the electric motor314UD for driving the spur gear1216would accomplish articulation as described above in a two-wire system. However, where a four-wire system is contemplated, a second system identical to that described immediately hereafter, can be employed to drive the left-right cables. Accordingly, for ease of understanding just one of the systems is described herein, with the understanding that one of skill in the art would readily understand how to employ a second such system in a four-wire system. Those of skill in the art will recognize that other mechanisms can be employed to enable the articulation of a distal portion of a catheter and other articulating catheters may be employed without departing from the scope of the disclosure.

To accomplish up-down articulation of the articulating portion317of the catheter102, steering cables319a-bmay be employed. The distal ends of the steering cables319a-bare attached to, or at, or near the distal end of the catheter102. The proximal ends of the steering cables319a-bare attached to the distal tips of the posts320a, and320b. As shown inFIG.12, the posts320aand320breciprocate longitudinally, and in opposing directions. Movement of the posts320acauses one steering cable319ato lengthen and at the same time, opposing longitudinal movement of post320bcauses cable319bto effectively shorten. The combined effect of the change in effective length of the steering cables319a-bis to cause joints a forming the articulating portion317of catheter102shaft to be compressed on the side in which the cable319bis shortened, and to elongate on the side in which steering cable319ais lengthened.

The opposing posts320aand320bhave internal left-handed and right-handed threads, respectively, at least at their proximal ends. As shown inFIG.13housed within casing304bare two threaded shafts322aand322b, one is left-hand threaded and one right-hand threaded, to correspond and mate with posts320aand320b. The shafts322aand322bhave distal ends which thread into the interior of posts320aand320aand proximal ends with spur gears324aand324bb. The shafts322aand322bhave freedom to rotate about their axes. The spur gears324aand324bengage the internal teeth of planetary gear326. The planetary gear326also an external tooth which engage the teeth of spur gear318on the proximal end of electric motor314UD.

To articulate the catheter in the upwards direction, a clinician may activate via an activation switch (not shown) for the electric motor314UD causing it to rotate the spur gear318, which in turn drives the planetary gear326. The planetary gear326is connected through the internal gears324aand324bto the shafts322aand322b. The planetary gear326will cause the gears324aand324bto rotate in the same direction. The shafts322aand322bare threaded, and their rotation is transferred by mating threads formed on the inside of posts320aand320binto linear motion of the posts320aand320b. However, because the internal threads of post320aare opposite that of post320b, one post will travel distally and one will travel proximally (i.e., in opposite directions) upon rotation of the planetary gear326. Thus, the upper cable319ais pulled proximally to lift the catheter102, while the lower cable319bmust be relaxed. As stated above, this same system can be used to control left-right movement of the end effector, using the electric motor314LR, its spur gear316, a second planetary gear (not shown), and a second set of threaded shafts322and posts320and two more steering cables319. Moreover, by acting in unison, a system employing four steering cables can approximate the movements of the human wrist by having the three electric motors314and their associated gearing and steering cables319computer controlled by the computing device122.

Though generally described above with respect to receiving manual inputs from a clinician as might be the case where the drive mechanism is part of a hand-held catheter system, the disclosure is not so limited. In a further embodiment, the drive mechanism300is part of a robotic system for navigating the catheter102to a desired location within the body. In accordance with this disclosure, in instances where the drive mechanism is part of a robotic catheter drive system, the position of the distal portion of the catheter102may be robotically controlled.

The drive mechanism may receive inputs from computing device122or another mechanism through which the surgeon specifies the desired action of the catheter102. Where the clinician controls the movement of the catheter102, this control may be enabled by a directional button, a joystick such as a thumb operated joystick, a toggle, a pressure sensor, a switch, a trackball, a dial, an optical sensor, and any combination thereof. The computing device responds to the user commands by sending control signals to the motors314. The encoders of the motors314provide feedback to the control unit24about the current status of the motors314.

In accordance with the disclosure, and as outlined in greater detail below, the drive mechanism300receives signals derived by the computing device122to drive the catheter102(e.g., extend and retract pull-wires) to maintain the orientation of the distal tip of the catheter102despite extension of a tool such as a biopsy needle or ablation catheter or movements caused by respiration and cardiac cycles.

As described in connection withFIGS.3A and3B, catheter102is operated on its proximal end through a collection of controls for rotation and distal tip deflection. In contrast, to the embodiment described in connection withFIGS.3A and3B, a manually advanced catheter102may not include the motor314R, relying instead on manual manipulation for rotation of the catheter102. Alternatively, the drive mechanism may include only a single wire319, or a single pair of wires319a,319b. In such an embodiment, articulation is enabled in a single or in a pair of wires in opposite directions. One or more knobs or levers or wheels on the proximal handle control or energize the energize the respective motor314to enable for distal tip articulation. Rotation and advancement/extraction of the catheter102are controlled directly by the user's hand pushing, pulling and rotating the catheter102within the patient. As described in connection withFIGS.3A and3B, any or all of these manual controls can be removed, and users indirectly control the catheter operation through an interface to the motors such as a joystick. Navigation may also be fully automatic with user oversight.

Following planning, registration, and then navigation of a catheter102proximate the target, a user interface (UI)200on computing device122may depict an image as seen inFIG.2. UI200depicts a rear perspective view of a virtual distal tip202of catheter102. The position of the virtual distal tip202relative to the virtual target204is displayed in the UI200. This displayed position is based on the detected position of the catheter102, and more particularly the sensors104,126relative to the location of the target as identified in the pre-procedure scan from which the 3D model as derived. This relative position relies on the registration to provide accuracy. However, whether manual or robotic, as noted above, the pathway plan and 3D model developed from the pre-procedure scan data must be registered to the patient before navigation of the catheter to a target within the anatomy can being.

Accordingly, a local registration process operating on computing device122may be employed. In accordance with the local registration process, once catheter102has been successfully navigated proximate, as shown inFIG.2, the target202a local registration process400may be performed for each target to reduce the CT-to-body divergence. An initial step402a sequence of fluoroscopic images is captured via fluoroscopic imaging device124for example from about 30 degrees on one side of the AP position to about 30 degrees on the other side of the AP position. At step404a fluoroscopic 3D reconstruction may be then generated by the computing device122. The generation of the fluoroscopic 3D reconstruction is based on the sequence of fluoroscopic images and the projections of structure of markers incorporated with transmitter mat120on the sequence of images. Following generation of the fluoroscopic 3D reconstruction, two fluoroscopic images are displayed on computing device122at step406. At step408the distal tip of the catheter102is marked in each of these images. The two images are taken from different portions of the fluoroscopic 3D reconstruction. The fluoroscopic images of the 3D reconstruction may be presented on the user interface in a scrollable format where the user is able to scroll through the slices in series if desired.

Next at step410the target needs to be identified in the fluoroscopic 3D reconstruction. In one example, the clinician will be asked to mark the target in two different perspectives of the 3D reconstruction. At step412, the entire fluoroscopic 3D reconstruction may be viewed to ensure that the target remains within an identified location, such as a circle placed on the target throughout the fluoroscopic 3D reconstruction. At step414the local registration can be accepted. At step416the relative position of the virtual catheter202in the 3D model relative to the virtual target204is updated to display the actual current relative position of the end of the catheter102and the target calculated by the local registration process400. By the local registration process the offset between the location of the target and the tip of the catheter102is determined as they are observed in the fluoroscopic 3D reconstruction. The offset is utilized, via computing device122, to correct any errors in the original registration process and minimize any CT-to-body divergence. As a result, the location and or orientation of the navigation catheter on the GUI with respect to the target is updated. At this point the clinician has a high degree of confidence in the position of the catheter102relative to the target as displayed in the UI200.

By the process described above the relative positions of the catheter102and the target are marked in the 3D fluoroscopic reconstruction and the offset determined. In addition, the position of the catheter102is always being sensed either in the EM field to provide EM coordinates of its position, or in robotic coordinates if a robot is employed. The offset can then be used to update the position of the virtual distal tip202in UI200relative to the virtual target204. Thus improving the registration and providing a clearer indication of the relative positions of the virtual distal tip202and virtual target204, that more closely depicts the actual relative positions of the distal tip of the catheter102and the target in the patient.

While far improved, the local registration process400does not account for movement of the catheter102or the target caused by respiration or heartbeat. Further, after the local registration process, the clinician or robot will typically remove the LG110with sensor104from the catheter102and insert a medical device in the catheter102and advance the medical device towards the target. As will be appreciated, in addition to the effects of respiration and heartbeat the relative positions of the catheter102and the target can be affected by the removal of the LG and the insertion of other tools. Thus, while the local registration is an improvement and overcomes the CT to body divergence, it is necessarily a static update to the relative positions of the catheter102and the target.

As noted above, maintaining distal tip location and orientation to the target is difficult as the patient's body is moving due to pulmonary and cardiac activity along with muscle motion and disturbances cause by external forces such as the surgical staff moving the patient. Additional motion can be caused by motion of equipment around the patient such as surgical bed or anesthesia tube motion. These motions cause external forces that can alter the amount of distal tip articulation of the catheter102, cause the catheter102to move forward or backward, or impart rotational forces on the catheter102. If the catheter102is used as a tool to interact with the target location such as a lesion or is used to deploy a tool through a lumen in the catheter, the target anatomy can distort or move due to pressure from the tool deployment. As an example, deployment of a needle to perform biopsy on a lesion adjacent to a bronchial tube can result in movement of the catheter102. The action of the needle perforating the stiffer, cartilage-like airway can distort the softer lung tissue behind causing both the lesion and the catheter102to move and the needle miss. Thus, the 3D model and the sensed position of the catheter102, as depicted inFIG.2, even after a local registration process400cannot be relied upon to confirm that the tip of the catheter102is oriented towards (i.e., pointed at a target) at any particular time, and further that the tip of the catheter102remains in this orientation when tools are inserted therethrough for biopsy and treatment of the target.

A further aspect of the disclosure is directed to the use of an inertial measurement unit (IMU) within catheter102. For some of these applications the IMU328may be configured as a small microchip measuring as little as 2 mm×2 mm in size and less than 1 mm in thickness. As will be appreciated, such a small size makes IMUS very useful in medical navigation applications in accordance with the disclosure. However, other size devices may be employed without departing from the scope of the disclosure.

An IMU328is a sensor typically including one or more accelerometers, and one or more gyroscopes. Additionally, an IMU328may include one or more of a magnetometer, a pressure sensor, and other types of sensors. An IMU328provides individual velocity and acceleration measurements in the X, Y, and Z directions as well as roll about the X, Y, and Z axes. Using trigonometric operations, these measurements can be converted into a directional vector showing which way the IMU238is moving. Combining two vectors allows for calculation of distance traveled. While the effects of gravity need to be compensated for at all times, the gravity vector can be used to identify the orientation of the sensor. Identification of the orientation of the IMU328provides an indication of the orientation of the catheter102. Thus, this in addition to using the local registration process400to determine the location of the distal portion of the catheter102relative to the target, the IMU328can be utilized to determine an orientation of the distal tip of the catheter102.

FIG.5depicts a catheter102as it may appear on a UI500following navigation proximate a target. By including an IMU328in the distal portion of the catheter102, the plane502defined by the opening504in the end of the catheter102can be determined using an application running on the computing device122. The plane502which coincides with the opening504allows for determination of a vector506which as depicted inFIG.5is normal to a plane502transverse to the length of the catheter102. The vector506depicts the path that a tool, such as a biopsy or treatment tool, extended from the opening504at the distal end of the catheter102would follow if extended from the distal tip of the catheter102. As can be seen inFIG.5, the vector506does not coincide with a vector from the catheter102to the target508, thus a tool extended from the catheter would miss the target508. The angle θ represents the 3D angle of change in orientation of the distal tip of the catheter102required to have plane502oriented such that its orientation is such that a vector510extending normal to the plane502intersects the target508.

A further aspect of the use of an IMU328is that its movement can be tracked throughout its navigation through the luminal network. As such, the effects of physiological forces from respiration and cardiac function can be detected and isolated from the overall movement of the catheter102. In a sense, the IMU328can act as both a respiration rate monitor and a heart rate monitor. In some instances, this may require a pause to the navigation process to allow for collection of movement data while the catheter102is not otherwise being moved.

With collection of the physiological data such as heart rate and respiration rate, the movement of the distal portion of the catheter102caused by these physiological functions can be determined and tracked by the application running on computing device122. Tracking can occur throughout the navigation process and the estimates of the movement caused by each force applied to the catheter102(e.g. motor force, hand manipulation, tissue responsive forces, and physiological forces) can be calculated and updated throughout navigation. And once proximate a target, as depicted inFIG.5the movement caused by the physiological data can be further refined and utilized as described in greater detail below.

As will be understood by those of skill in the art, even with motorized control as depicted inFIGS.3A and3B, known systems have difficulty in maintaining the orientation of the distal tip of the catheter102with respect to the target. In fact, known systems are generally not concerned with orientation of the catheter102, but rather focus on maintaining the position and location or the articulation of the catheter. This is typically done through the use of sensors such as EM sensors or shape sensors. These known systems detect changes in the position or articulation of the distal tip and seek to counteract that movement through use of the motors to return the catheter102to the desired shape or to return the catheter102a desired position. However, such designs do not account for anatomic changes where the target's location in space is moving independent of any movement of the distal tip of the catheter102. Rather these systems seek to maintain the position of the catheter102as determined in, for example, EM coordinates. But typically, there are no EM coordinates for the target, and more importantly there is no ability to utilize EM coordinates to track movement of the target. Still further, the entire patient, or at least a significant portion of them is also moving meaning that not only is the location of the target moving because of the physiological forces, but in fact the entire or a substantial portion of the entire patient.

A further aspect of the disclosure is directed to a method of addressing the above-identified shortcoming of prior catheter navigation systems. As described above, the method600may optionally start at step602with navigating a catheter102into a luminal network of a patient. This navigation may be manual or robotically driven and may follow a pre-planned pathway through a 3D model derived from pre-procedure images such as CT images. Further, prior to the onset of navigation the 3D model may be registered to the patient as described above. Those of skill in the art will recognize that method600may also begin after navigation and commence with step604wherein IMU328associated with the catheter102collects data related to movement of the catheter. This data regarding movement of the catheter102, and particularly the IMU328enables determination of both respiration and heart rate data and the movements caused by the physiological forces. As an optional step606, a UI on a display of computing device122may prompt the user to “Stop movement” of the catheter102, allowing the IMU to collect just the physiological movement data (e.g., that caused by heartrate, breathing, and other internal muscular contractions of the patient).

After reaching a desired proximity to a target508at step608, for example between 2 and 5 CM from the target for example based on detection of the position of the catheter via the EM sensor104, a fluoroscopic sweep is acquired of the end of the catheter102and the target508at step610. Following the process described above, the end of the catheter102and the target508are marked in images derived from the fluoroscopic sweep at step612. At step614the orientation of the distal tip of the catheter102is determined based on the data received from the IMU328. At step616the application determines whether the orientation of the distal tip of the catheter102is such that a vector perpendicular to a plane502defined by the opening504in the distal tip is pointed at the target508. If the answer is no, at step618the application running on computing device122can signal the motors314to adjust the articulation of the catheter102to achieve the correct orientation based a 3D angle between the vector506that does not intersect the target508and a vector510that would intersect the target508. As an alternative the computing device122may provide an output in a UI on a display of the computing device of the movement necessary to achieve an orientation of the opening504such that the vector traverses the target. If the answer is yes, the method proceeds to step620, where the movement data collected by the IMU relating the heart rate and respiration (as well as other muscle movements) is utilized by the application to provide drive signals to the motors314of the catheter102to maintain the orientation of the opening504relative to the target.

At step622, if other operations are undertaken such as removal of the LG or insertion of a biopsy needle, the position and orientation of the catheter102is maintained by the application running on computing device122. The IMU328outputs signals relating to its orientation which is associated with the orientation of the plane502at the of opening504of the catheter102. When movement of the IMU328is detected, based on these user activities the application running on the computing device122generates signals to the motors314to adjust the shape of the distal portion of the catheter102to correct any change in orientation relative to the target.

Further, as part of step622the movements of the catheter102and particularly the IMU328caused by respiration and heartbeat can be detected. Because these are highly cyclical and repeated movements the application running on the computing device and generate signals which are sent to the motors314to adjustment the position of the distal portion of the catheter102such that opening504remains oriented at the target500throughout the cardiac and respiratory cycles.

The maintaining of the orientation of the distal end of the catheter102may be occur automatically, with drive signals being generated by computing device122and delivered to the motors314to counteract the movements of the distal tip of the catheter102. Alternatively, the movement of the distal portion of the catheter102can be manually controlled either via switches, levers, or a wheel associated with the motors314to achieve the desired orientation of the opening504towards the catheter. The need for, the magnitude and the direction of these movements may be presented on a UI200.

Whether manually initiated or controlled or partially controlled via computing device122the amount and speed of the movements the catheter102caused by the motors314can be gated based on the proximity to the target. Thus, when navigating the central airways or larger lumens larger movements of the end of the catheter102are possible. As the catheter102approaches a target500, particularly a target located closer to the periphery of the lung or other bodily organ, the range of available movements of the distal portion of the catheter102may be reduced. This reduction can increase the safety of the movements particularly in the periphery of organs like the lungs. The determination of proximity of these locations may be based on the tracking system114employing the EM sensor104or a shape sensor, as described above.

Similarly, the amount of drift or error in orientation of the opening504in the catheter104towards the target500can be adjusted as desired by the user or to the limits of the processing power and speed of the computing device122. Where a user desires continual movement of the catheter102to confirm orientation the amount of acceptable drift can be reduced. Alternatively, if fewer movements are desired, for example due to the tissue being navigated, the acceptable drift can be increased, and the signals will be sent from the computing deice122to the motors314less frequently.

Still further, because of the cyclical nature of these physiological forces, the application running on the computing device can anticipated the application of forces on the catheter102and begin application of compensating signals at a timing to ensure proper orientation at all times.

In one embodiment, the representations of the virtual catheter tip202and target204can be seen to move in the images displayed in the UI200in concert with the physiological signals. The images may be live fluoroscopic images, 3D model images from the pre-procedure imaging and pathway plan, or other imaging modalities. In an embodiment where the UI200displays 3D model images the viewer is afforded a virtual image of the reality of the movements occurring within the patient. In the UI200in this embodiment the distal portion of the virtual catheter tip202may be seen to flex to maintain the orientation towards the virtual target204.

In a further embodiment, the UI200may include an indicator. For example, a portion of the UI200may have a portion which either changes brightness or changes color (e.g., from red to yellow to green). Alternatively, movements caused by respiration and cardiac function can be graphed over time, and portions of the graph may include a color indicator or include a color band. The indicator can be timed relative to the cardiac and respiratory forces applied to the catheter102. Despite the articulation of the catheter102to maintain the orientation of the opening504towards the target500, in some circumstances either the user may find it desirable to or the application running on computing device122can be configured to limit the timing of the use of tools to only those portions of the cyclical movement caused by the physiological forces that correspond to the approximate timing in the respiratory and cardiac cycle of when the fluoroscopic images were acquired. By performing this gating, a further confirmation that despite the movement of the target and the catheter102, and even though the IMU328has been utilized to articulate the catheter102, the catheter102and the target at these portions of the cardiac and respiratory cycle are now substantially at the same locations (e.g., in the same phase of the cardiac and respiratory cycle) as when the original orientation confirmation was acquired.

In a further aspect of the disclosure, the imaging system for identification of the catheter102and the target could be an ultrasound system, computed tomography (CT) system, cone beam CT system, MRI system or another capable system of imaging both the catheter102and the target simultaneously. Further, though described herein as occurring just once, the external imaging (e.g., fluoroscopy, ultrasound,) may be repeated at intervals during the procedure to ensure continued orientation of the catheter tip and the target. Still further, thought described herein as being identified by a user viewing the fluoroscopic or ultrasound images to determine the location and orientation of the catheter102relative to the target and the location of the target, these systems and methods described herein may additionally or alternatively employ automatic identification methods. Such automatic methods may use image processing software to automatically identify the distal tip and its orientation from the imaging system output.

Still further, in embodiments of the disclosure the maintaining of orientation of the distal tip of the catheter102and the target is a feature that can be turned on and off by the user or automatically whenever the catheter is greater than a preset distance from the target. Still further, the UI200may include an indicator to alert the user if the orientation lock between the catheter102and the target is ever lost. Such a feature may also indicate that the external imaging needed to be renewed so that the method600can be started a new. As will be appreciated, this orientation lock may be turned may be selectively turned on or off as desired by the user or by the application running on computing device120.

Reference is now made toFIG.7, which is a schematic diagram of a system700configured for use with the methods of the disclosure including the method ofFIG.6. System700may include a workstation701, and optionally an imaging device715(e.g., a fluoroscope or an ultrasound device). In some embodiments, workstation701may be coupled with imaging device715, directly or indirectly, e.g., by wireless communication. Workstation701may include a memory702, a processor704, a display706and an input device710. Processor or hardware processor704may include one or more hardware processors. Workstation701may optionally include an output module712and a network interface708. Memory702may store an application718and image data77. Application718may include instructions executable by processor704for executing the methods of the disclosure including the method ofFIG.6.

Application718may further include a user interface716. Image data714may include the CT scans, the generated fluoroscopic 3D reconstructions of the target area and/or any other fluoroscopic image data and/or the generated one or more slices of the 3D reconstruction. Processor704may be coupled with memory702, display706, input device710, output module712, network interface708and imaging device715. Workstation701may be a stationary computing device, such as a personal computer, or a portable computing device such as a tablet computer. Workstation701may embed a plurality of computer devices.

Memory702may include any non-transitory computer-readable storage media for storing data and/or software including instructions that are executable by processor704and which control the operation of workstation701and, in some embodiments, may also control the operation of imaging device715. Imaging device715may be used to capture a sequence of fluoroscopic images based on which the fluoroscopic 3D reconstruction is generated and to capture a live 2D fluoroscopic view according to this disclosure. In an embodiment, memory702may include one or more storage devices such as solid-state storage devices, e.g., flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memory702may include one or more mass storage devices connected to the processor704through a mass storage controller (not shown) and a communications bus (not shown).

Although the description of computer-readable media contained herein refers to solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor704. That is, computer readable storage media may include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information, and which may be accessed by workstation1001.

Application718may, when executed by processor704, cause display706to present user interface716. User interface716may be configured to present to the user a single screen including a three-dimensional (3D) view of a 3D model of a target from the perspective of a tip of a medical device, a live two-dimensional (2D) fluoroscopic view showing the medical device, and a target mark, which corresponds to the 3D model of the target, overlaid on the live 2D fluoroscopic view. An example of the user interface716is shown, for example, inFIG.2. User interface716may be further configured to display the target mark in different colors depending on whether the medical device tip is aligned with the target in three dimensions.

Network interface708may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and/or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and/or the Internet. Network interface708may be used to connect between workstation701and imaging device715. Network interface708may be also used to receive image data714. Input device710may be any device by which a user may interact with workstation701, such as, for example, a mouse, keyboard, foot pedal, touch screen, and/or voice interface. Output module712may include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial busses (USB), or any other similar connectivity port known to those skilled in the art. From the foregoing and with reference to the various figures, those skilled in the art will appreciate that certain modifications can be made to the disclosure without departing from the scope of the disclosure.

While detailed embodiments are disclosed herein, the disclosed embodiments are merely examples of the disclosure, which may be embodied in various forms and aspects. For example, embodiments of an electromagnetic navigation system, which incorporates the target overlay systems and methods, are disclosed herein; however, the target overlay systems and methods may be applied to other navigation or tracking systems or methods known to those skilled in the art. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the disclosure in virtually any appropriately detailed structure.