Patent Description:
Computed tomography (CT) images are commonly used to identify objects, such physiological structures as well as medical instruments, in a patient's body. Various CT scans may be performed before and/or during a medical procedure to identify such objects and to monitor progress of the medical procedure. However, the objects may not always be detectable, in part or in whole, based solely on CT images. Further, interference with CT scans may be caused by various sources, and mitigation of such interference is not always possible. An example of such a system using CT data is shown in <CIT>, which shows a display device, a computing device, a processor and memory with instructions to process image data received from a patient's body. Described hereinbelow is an improved system for identifying objects, and particularly medical instruments, in CT images.

Provided in accordance with embodiments of the present disclosure is a system for identifying a percutaneous tool in image data. In an aspect of the present disclosure, an illustrative system which is able to perform steps including receiving image data of at least a portion of a patient's body, identifying an entry point of a percutaneous tool through the patient's skin in the image data, analyzing a portion of the image data including the entry point of the percutaneous tool through that patient's skin to identify a portion of the percutaneous tool inserted through the patient's skin, determining a trajectory of the percutaneous tool based on the identified portion of the percutaneous tool inserted through the patient's skin, identifying a remaining portion of the percutaneous tool in the image data based on the identified entry point and the determined trajectory of the percutaneous tool, and displaying the identified portions of the percutaneous tool on the image data.

In another aspect of the present disclosure, the system further includes the ability to perform the steps of receiving characteristic data of the percutaneous tool, and identifying the remaining portion of the percutaneous tool in the image data is further based on the characteristic data of the percutaneous tool.

In a further aspect of the present disclosure, the characteristic data of the percutaneous tool includes one or more of a length of the percutaneous tool, a diameter of the percutaneous tool, and a flexibility metric of the percutaneous tool.

In another aspect of the present disclosure, determining a trajectory of the percutaneous tool based on the identified portion of the percutaneous tool inserted through the patient's skin includes determining an angle of insertion of the identified portion of the percutaneous tool inserted through the patient's skin, and determining a trajectory of the percutaneous tool based on the angle of insertion of the identified portion of the percutaneous tool inserted through the patient's skin.

In yet another aspect of the present disclosure, the system further includes the ability to perform the steps of identifying a target location in the image data, determining a path from the entry point to the target location, determining whether the trajectory of the percutaneous tool corresponds to the path, and displaying the identified portions of the percutaneous tool, the trajectory, and the path on the image data.

In a further aspect of the present disclosure, if it is determined that the trajectory of the percutaneous tool does not correspond to the path, the method further includes determining a difference between the trajectory and the path, and displaying guidance for adjusting an angle of the percutaneous tool based on the determined difference between the trajectory and the path.

In another aspect of the present disclosure, the percutaneous tool is an ablation needle, and the system further includes the ability to perform the steps of receiving configuration settings for an ablation procedure, identifying a position of a radiating portion of the percutaneous tool in the image data, determining a projected ablation zone based on the configuration settings and the identified position of the radiating portion of the percutaneous tool, and displaying the projected ablation zone on the image data.

In a further aspect of the present disclosure, the system further includes the ability to perform the steps of receiving an indication that the radiating portion of the percutaneous tool has been activated, determining a progress of an ablation procedure based on the configuration settings and a time during which the percutaneous tool has been activated, and displaying an estimated ablated zone based on the determined progress of the ablation procedure.

In another aspect of the present disclosure, the system further includes the ability to perform the steps of identifying a distal portion of the percutaneous tool in the image data, determining a line in the image data between the entry point and the distal portion of the percutaneous tool, and displaying the determined line on the image data.

In a further aspect of the present disclosure, the distal portion of the percutaneous tool is identified based on characteristic data of the percutaneous tool.

In yet a further aspect of the present disclosure, the distal portion of the percutaneous tool is identified based on an electromagnetic sensor included in the percutaneous tool.

In another aspect of the present disclosure, identifying the remaining portion of the percutaneous tool includes analyzing the image data to identify high intensity areas along the determined line, and including portions of the high intensity areas along a length of the determined line and within a radius of the determined line, and the radius is determined based on a diameter characteristic of the percutaneous tool.

In a further aspect of the present disclosure, identifying the remaining portion of the percutaneous tool further includes excluding portions of the high intensity areas along the length of the determined line and outside of the radius of the determined line.

Provided in accordance with embodiments of the present disclosure is a system for identifying a percutaneous tool in image data.

Provided in accordance with embodiments of the present disclosure is a non-transitory computer-readable storage medium storing a program for identifying a percutaneous tool in image data.

Provided as a further example, which however is not claimed, is a method for identifying a percutaneous tool in image data. In an aspect of the present disclosure, an illustrative method includes receiving image data of at least a portion of a patient's body, identifying a potential distal portion of a percutaneous tool in the image data, identifying a potential shaft portion of the percutaneous tool within a predetermined distance from the identified potential distal portion of the percutaneous tool, determining a line from the identified potential distal portion of the percutaneous tool through the identified potential shaft portion of the percutaneous tool, identifying a potential remaining portion of the percutaneous tool in the image data based on the line, and displaying the identified potential distal, shaft, and remaining portions of the percutaneous tool on the image data.

The method, which is not claimed, further includes receiving characteristic data of the percutaneous tool, and identifying the remaining portion of the percutaneous tool in the image data is further based on the characteristic data of the percutaneous tool.

In a further example, the characteristic data of the percutaneous tool includes one or more of a length of the percutaneous tool, a diameter of the percutaneous tool, and a flexibility metric of the percutaneous tool.

In another example, the method further includes determining whether the identified potential distal, shaft, and remaining portions of the percutaneous tool correspond to a valid percutaneous tool.

In yet another example, the method further includes identifying a target location in the image data, determining a path from the entry point to the target location, determining a trajectory of the percutaneous tool based on an entry point and angle of insertion of the percutaneous tool into the patient's body, determining whether the trajectory of the percutaneous tool corresponds to the path, and displaying the trajectory and the path on the image data.

In a further example, if it is determined that the trajectory of the percutaneous tool does not correspond to the path, the method further includes determining a difference between the trajectory and the path, and displaying guidance for adjusting an angle of the percutaneous tool based on the determined difference between the trajectory and the path.

In another example, the percutaneous tool is an ablation needle, and the method further includes receiving configuration settings for an ablation procedure, identifying a position of a radiating portion of the percutaneous tool in the image data, determining a projected ablation zone based on the configuration settings and the identified position of the radiating portion of the percutaneous tool, and displaying the projected ablation zone on the image data.

In a further example, the method further includes receiving an indication that the radiating portion of the percutaneous tool has been activated, determining a progress of an ablation procedure based on the configuration settings and a time during which the percutaneous tool has been activated, and displaying an estimated ablated zone based on the determined progress of the ablation procedure.

In another example, the method further includes identifying an entry point of the percutaneous tool into the patient's body in the image data, determining a line in the image data between the entry point and the identified potential distal portion of the percutaneous tool, and displaying the determined line on the image data.

In yet another example, identifying the potential remaining portion of the percutaneous tool includes analyzing the image data to identify high intensity areas along the determined line, and including portions of the high intensity areas along a length of the determined line and within a radius of the determined line, and the radius is determined based on a diameter characteristic of the percutaneous tool.

In a further example, identifying the potential remaining portion of the percutaneous tool further includes excluding portions of the high intensity areas along the length of the determined line and outside of the radius of the determined line.

In another example, the distal portion of the percutaneous tool is identified based on characteristic data of the percutaneous tool.

In yet another example, the distal portion of the percutaneous tool is identified based on an electromagnetic sensor included in the percutaneous tool.

Provided as a further example, which however is not claimed, is a system for identifying a percutaneous tool in image data. In an aspect of the present disclosure, an illustrative system includes a percutaneous tool, a display device, and a computing device including a processor, and a memory storing instructions which, when executed by the processor, cause the computing device to receive image data of at least a portion of a patient's body, identify a potential distal portion of a percutaneous tool in the image data, identify a potential shaft portion of the percutaneous tool within a predetermined distance from the identified potential distal portion of the percutaneous tool, determine a line from the identified potential distal portion of the percutaneous tool through the identified potential shaft portion of the percutaneous tool, identify a potential remaining portion of the percutaneous tool in the image data based on the line, and display the identified potential distal, shaft, and remaining portions of the percutaneous tool on the image data.

Provided as a further example, which however is not claimed, is a non-transitory computer-readable storage medium storing a program for identifying a percutaneous tool in image data. In an aspect of the present disclosure, the program includes instructions which, when executed by a processor, cause a computing device to receive image data of at least a portion of a patient's body, identify a potential distal portion of a percutaneous tool in the image data, identify a potential shaft portion of the percutaneous tool within a predetermined distance from the identified potential distal portion of the percutaneous tool, determine a line from the identified potential distal portion of the percutaneous tool through the identified potential shaft portion of the percutaneous tool, identify a potential remaining portion of the percutaneous tool in the image data based on the line, and display the identified potential distal, shaft, and remaining portions of the percutaneous tool on the image data.

Various aspects and features of the present disclosure are described hereinbelow with references to the drawings, wherein:.

The present disclosure generally relates to systems for identifying and segmenting medical instruments, such as ablation needles, in radiographic images. In particular, by determining a line based on a trajectory of a medical instrument inserted through a patient's skin and/or a line extending from a distal portion of an instrument, high intensity areas identified in radiographic images within a predetermined distance from the line may be segmented as part of the medical instrument, and high intensity areas more than the predetermined distance from the line may be excluded.

Radiographic images, such as computed tomography (CT) images, magnetic resonance imaging (MRI) images, cone beam computed tomography (CBCT) images, two-dimensional (2D) and/or three-dimensional (3D) X-ray images, 2D and/or 3D ultrasound images, and/or various other imaging modalities may be obtained during a medical procedure to identify placement of medical instruments, such as ablation needles, in a patient's body, and particularly, about a treatment site. While placement of medical instruments may be confirmed via visual inspection during open and/or laparoscopic surgical procedures, such visual inspection is often not possible during percutaneous procedures. As such, radiographic imaging techniques are used to guide and confirm placement of medical instruments. However, identification of the medical instruments in the radiographic images is not a perfect process, and, due to limited resolution and clarity of radiographic images, structures and/or objects other than the medical instruments may be misidentified as part of the medical instruments. Further, medical instruments inserted at an angle that crosses multiple imaging planes and/or image slices or traverse an area of significant interference, may be hard to identify in the radiographic images.

Identification of medical instruments in radiographic images may be significantly improved when characteristics, such as length, diameter, point of insertion, and/or trajectory of the medical instrument are taken into account when attempting to identify the medical instruments in the radiographic images. For example, by determining a line based on a trajectory of a medical instrument inserted through a patient's skin, it may be determined which high intensity areas identified in image data about the medical instrument's location should be included as part of the medical instrument, and which high intensity areas should be excluded. Then, when the position of the medical instrument, guidance may be provided for navigating the medical instrument to a target location. Once the medical instrument is placed at the target location, a projected ablation zone may be determined based on the location of a radiating portion of the ablation needle, and the projected ablation zone may be displayed to the clinician so that the clinician may visualize the ablation zone relative to the radiographic images to determine whether the ablation zone encompasses the entirety of the area the clinician is seeking to treat.

Methods for automated identification and segmentation of a percutaneous tool in radiographic images, such as CT images, providing guidance for navigating the percutaneous tool to a target location, as well as monitoring a progress of a treatment procedure, such as an ablation procedure, may be implemented as part of an electromagnetic navigation (EMN) system. Generally, in an embodiment, the EMN system may be used in planning and performing treatment of an area of the patient's body, such as the patient's lungs, by determining a path to a target location, such as a treatment location, inserting an ablation needle into the patient's body, and positioning the ablation needle proximate the target location. The EMN system may be configured to display various views of the patient's body, including the radiographic images and/or a three-dimensional (3D) model of the patient's body generated based the radiographic images.

With reference to <FIG>, there is shown a system <NUM> usable for automated identification and segmentation of an ablation needle in radiographic images. System <NUM> may include a display device <NUM>, a table <NUM> including an electromagnetic (EM) field generator <NUM>, a treatment tool <NUM> including a distal radiating portion <NUM>, an ultrasound sensor <NUM> connected to an ultrasound workstation <NUM>, a peristaltic pump <NUM>, and a computing device <NUM> attached to or in operable communication with a microwave generator <NUM>. Display device <NUM> is configured to output instructions, images, and messages relating to the performance of the medical procedure.

Table <NUM> may be, for example, an operating table or other table suitable for use during a medical procedure, which includes EM field generator <NUM>. EM field generator <NUM> is used to generate an EM field during the medical procedure and forms part of an EM tracking system that is used to track positions of medical instruments within the patient's body, such as by tracking a position of one or more EM sensors included in and/or coupled to treatment tool <NUM>. EM field generator <NUM> may include various components, such as a specially designed pad to be placed under, or integrated into, table <NUM> or a patient bed. An example of such an EM tracking system is the AURORA™ system sold by Northern Digital Inc.

Treatment tool <NUM> is a medical instrument for percutaneously accessing and diagnosing and/or treating tissue at a target location. For example, treatment tool <NUM> may be an ablation needle having a microwave ablation needle or antenna that is used to ablate tissue. In other embodiments, treatment tool <NUM> may be a biopsy tool for obtaining a tissue sample at the target location. Those skilled in the art will recognize that various other types of percutaneous tools, including, for example, cannulas for guiding catheters or other tools to a treatment site may also be used without departing from the scope of the present disclosure. In embodiments where treatment tool <NUM> is an ablation needle, treatment tool <NUM> includes distal radiating portion <NUM>, and may further include or be coupled to one or more EM sensors enabling the EM tracking system to track the location, position, and orientation (also known as the "pose") of treatment tool <NUM> inside the body of the patient. As explained in further detail below, treatment tool <NUM> may be described as having various portions. For example, when treatment tool <NUM> is inserted into a patient's body, treatment tool <NUM> may be described as having a portion inserted into the patient's body, and a portion external to the patient's body. The portion of treatment tool <NUM> inserted into the patient's body may further be divided into a portion inserted through the patient's skin-that is, the portion of treatment tool <NUM> that is in contact with the various layers of the patient's skin-and a remaining portion inserted into the patient's body-that is, the rest of treatment tool <NUM> inserted into the patient's body excluding the portion that is in contact with the various layers of the patient's skin. Likewise, the remaining portion of treatment tool <NUM> inserted into the patient's body may further be divided into a distal portion (which may include distal radiating portion <NUM>), and a proximal portion. The distal portion may be the portion of treatment tool <NUM> inserted the furthest into the patient's body.

Peristaltic pump <NUM> may be configured to pump fluid through treatment tool <NUM> to cool treatment tool <NUM> during the medical procedure. Microwave generator <NUM> may be configured to output microwave energy to treatment tool <NUM> via distal radiating portion <NUM>. Computing device <NUM> may be, for example, a laptop computer, desktop computer, tablet computer, or other similar device. Computing device <NUM> may be configured to control microwave generator <NUM>, peristaltic pump <NUM>, a power supply (not shown), and/or any other accessories and peripheral devices relating to, or forming part of, system <NUM>. In some embodiments, microwave generator <NUM> controls the operation of peristaltic pump <NUM>. While the present disclosure describes the use of system <NUM> in a surgical environment, it is also envisioned that some or all of the components of system <NUM> may be used in alternative settings, for example, an imaging laboratory and/or an office setting.

In addition to the EM tracking system, the surgical instruments used during the medical procedure, such as treatment tool <NUM>, may also be visualized by using CT and/or ultrasound imaging. Ultrasound sensor <NUM>, such as an ultrasound wand, may be used to image the patient's body during the medical procedure to visualize the location of the surgical instruments, such as treatment tool <NUM>, inside the patient's body. Ultrasound sensor <NUM> may have an EM tracking sensor embedded within or attached to the ultrasound wand, for example, a clip-on sensor or a sticker sensor. Ultrasound sensor <NUM> may be positioned in relation to treatment tool <NUM> such that treatment tool <NUM> is at an angle to the ultrasound image plane, thereby enabling the clinician to visualize the spatial relationship of treatment tool <NUM> with the ultrasound image plane and with objects being imaged. Further, the EM tracking system may also track the location of ultrasound sensor <NUM>. In some embodiments, one or more ultrasound sensors <NUM> may be placed inside the patient's body. EM tracking system may then track the location of such ultrasound sensors <NUM> and treatment tool <NUM> inside the patient's body. Ultrasound workstation <NUM> may be used to configure, operate, and/or view images captured by ultrasound sensor <NUM>. Likewise, computing device <NUM> may be used to configure, operate, and/or view images captured by ultrasound sensor <NUM>, either directly or relayed via ultrasound workstation <NUM>.

Various other surgical instruments or surgical tools, such as LIGASURE® devices, surgical staplers, etc., may also be used during the performance of a medical procedure. In embodiments where treatment tool <NUM> is a microwave ablation needle, the microwave ablation needle is used to ablate a lesion or tumor (hereinafter referred to as a "target location") by using microwave energy to heat tissue in order to denature or kill cancerous cells. The construction and use of a system including such an ablation needle is more fully described in co-pending <CIT>, <CIT>, and <CIT>.

As noted above, the location of treatment tool <NUM> within the body of the patient may be tracked during the medical procedure. An example method of tracking the location of treatment tool <NUM> is by using the EM tracking system, which tracks the location of treatment tool <NUM> by tracking sensors, such as EM sensors, coupled to or incorporated in treatment tool <NUM>. Various types of sensors may be used, such as a printed sensor, the construction and use of which is more fully described in co-pending <CIT> A percutaneous treatment system similar to the above-described system <NUM> is further described in co-pending <CIT>.

While the above-described system <NUM> uses a microwave generator <NUM> to provide microwave energy to treatment tool <NUM>, those skilled in the art will appreciate that various other types of generators and tools may be used without departing from the scope of the present disclosure. For example, radio frequency (RF) ablation tools receiving RF energy from RF generators may be substituted for the microwave generators and ablation tools described above.

With reference to <FIG>, there is shown a simplified block diagram of computing device <NUM>. Computing device <NUM> may include at least one memory <NUM>, one or more processors <NUM>, a display <NUM>, a network interface <NUM>, one or more input devices <NUM>, and/or an output module <NUM>. Memory <NUM> may store an application <NUM> and/or image data <NUM>. Application <NUM> may, when executed by processor <NUM>, cause display <NUM> to display a user interface <NUM>. Application <NUM> may also provide an indication of the location of treatment tool <NUM> in relation to the target location, as well as the size, shape, and location of an ablation zone, as described further below.

Memory <NUM> may include any non-transitory computer-readable storage medium for storing data and/or software that is executable by processor <NUM> and which controls the operation of computing device <NUM>. In an embodiment, memory <NUM> may include one or more solid-state storage devices such as flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memory <NUM> may include one or more mass storage devices connected to processor <NUM> through a mass storage controller (not shown) and a communications bus (not shown). Although the description of computer-readable media included herein refers to a solid-state storage device, it should be appreciated by those skilled in the art that computer-readable storage media may be any available media that can be accessed by processor <NUM>. That is, computer readable storage media may include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device <NUM>.

Network interface <NUM> may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and/or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and/or the internet. Input device <NUM> may be any device by means of which a user may interact with computing device <NUM>, such as, for example, a mouse, keyboard, foot pedal, touch screen, and/or voice interface. Output module <NUM> may include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial busses (USB), or any other similar connectivity port known to those skilled in the art.

Turning now to <FIG>, there is shown a flowchart of an exemplary method <NUM> of automated identification and segmentation of an ablation needle in radiographic images, and providing guidance for navigating the ablation needle to a target location, according to an embodiment of the present disclosure. Method <NUM> may be performed, for example, by using system <NUM>, described above. In particular, application <NUM>, executing on computing device <NUM>, may be used to perform, or cause other components of system <NUM> to perform, the steps of method <NUM>. While the various steps of method <NUM> are described below in an exemplary sequence, those skilled in the art will recognize that some or all of the steps may be performed in a different order, repeated, and/or omitted without departing from the scope of the present disclosure.

Starting at step S302 of <FIG>, application <NUM> receives image data of a patient's body. The image data may be radiographic image data, such as CT image data, MRI image data, CBCT image data, X-ray image data, ultrasound image data, etc. For exemplary purposes, CT image data will be used in the description provided below. The image data may be received directly from an imaging device, such as a CT scanner, and/or may have previously been stored in memory <NUM> of computing device <NUM>. The image data may be received at the start of the medical procedure, and/or during the performance of the medical procedure. For example, multiple sets of image data may be received at various times during the medical procedure when identification of an ablation needle is requested, as described further below.

Additionally, the image data may be used for pre-procedural purposes, such as for identifying the patient's body in the image data at step S304, and generating a 3D model of the patient's body at step S306. The 3D model of the patient's body may include one or more portions of the patient's body, and particularly, may include the portion of the patient's body where the medical procedure will be performed, e.g. where the target location is. In the example described hereinbelow, the image data include a portion of the patient's chest, and thus the 3D model generated at step S306 is of a portion of the patient's chest.

After generating the 3D model, or concurrently therewith, application <NUM>, at step S308, processes the image data to identify a target location. In embodiments, the clinician provides input, such as via input device <NUM> of computing device <NUM> to identify the target location. For example, the clinician may review the image data received at step S302 and/or the 3D model generated at step S306 and select or mark one or more target locations. Application <NUM> may further determine one or more recommended entry points where treatment tool <NUM> should be inserted through the patient's skin to enable access to the target location identified at step S308. Application <NUM> may then cause display device <NUM> and/or display <NUM> to display guidance for inserting treatment tool <NUM> through the patient's skin.

Thereafter, at step S310, application <NUM> determines whether treatment tool <NUM> has been identified in the image data. If it is determined that treatment tool <NUM> has not been identified ("No" at step S310), processing proceeds to step S350 (described below with reference to <FIG>. ) Alternatively, if it is determined that treatment tool <NUM> has been identified ("Yes" at step S310), processing proceeds to step S312.

Turning now to <FIG>, at step S350 application <NUM> selects an ablation needle detection algorithm to use. Various algorithms may be used to detect an ablation needle in image data. For purposes of the present disclosure, two ablation needle detection algorithms will be described. For example, steps S352-S358 and S370-S376 may correspond to a first exemplary algorithm, while steps S362-S368 and S370-S376 may correspond to a second exemplary algorithm. However, those skilled in the art will recognize that various other ablation needle detection algorithms may also be used without departing from the scope of the present disclosure.

Processing of a first exemplary algorithm may start at step S352 where application <NUM> receives characteristic data of treatment tool <NUM>. The characteristic data may include a type of treatment tool <NUM>, such as an ablation needle, being used, a length of treatment tool <NUM>, a diameter of treatment tool <NUM>, a flexibility metric (such as Young's modulus) of treatment tool <NUM>, a location of one or more EM sensors included in treatment tool <NUM>, a location of distal radiating portion <NUM> in treatment tool <NUM>, locations of radiolucent fiducial markers and/or features designed to be visible under ultrasound imaging, etc. The characteristic data may be accessed by application <NUM> from memory <NUM>, may be inputted by the clinician via input device <NUM>, and/or may be provided to application <NUM> by treatment tool <NUM> and/or generator <NUM>.

Thereafter, at step S354, application <NUM> identifies one or more potential distal portions of treatment tool <NUM>. For example, application <NUM> may process the image data received at step S302 and/or additional image data received subsequently and throughout the medical procedure, to identify one or more distal portions of treatment tool <NUM>, such as based on the characteristic data of treatment tool <NUM> received at step S352. In embodiments, application <NUM> may process only a portion of the image data that includes the patient's body, an area proximate the target location determined at step S308, and/or an area proximate the recommended entry points determined at step S308. For example, application <NUM> may identify one or more areas of high intensity pixels having a shape similar to an eclipse as potential distal portions of treatment tool <NUM>. In embodiments, application <NUM> may determine a depth that treatment tool <NUM> is inserted into the patient's body, and may then seek to identify potential distal portions of treatment tool <NUM> that are about a corresponding distance from the recommended entry points determined at step S308. The depth that treatment tool <NUM> is inserted into the patient's body may be determined based on lines and/or markers on treatment tool <NUM> (not shown in <FIG>), and/or one or more EM sensors included in treatment tool <NUM>. Additionally or alternatively, application <NUM> may identify the distal portion of treatment tool <NUM> based on the location of the one or more EM sensors, radiolucent fiducial markers, and/or other radiopaque elements included in treatment tool <NUM>. For example, the position of the one or more EM sensors and/or radiolucent fiducial markers included in treatment tool <NUM> relative to the distal portion of treatment tool <NUM> may be known, and thus the location of the distal portion of treatment tool <NUM> may be determined based on a detected position of the one or more EM sensors and/or radiolucent fiducial markers.

Next, at step S356, application <NUM> identifies one or more potential shaft portions of treatment tool <NUM> proximate the potential distal portions of treatment tool <NUM> identified at step S354. For example, application <NUM> may further process the image data to identify areas of high intensity pixels within a predetermined distance of the identified potential distal portions of treatment tool <NUM>. In embodiments, application <NUM> may identify areas of high intensity pixels within <NUM> millimeters (mm) of the identified potential distal portions of treatment tool <NUM>. In further embodiments, application <NUM> may identify areas of high intensity pixels more than <NUM> but less than <NUM> from the identified potential distal portions of treatment tool <NUM>.

Thereafter, at step S358, application <NUM> determines a line extending from the potential distal portions of treatment tool <NUM> identified at step S354 through the potential shaft portions of treatment tool <NUM> identified at step S356. In embodiments, application <NUM> may determine a line extending from each of the identified potential distal portions of treatment tool <NUM> through each of its corresponding identified potential shaft portions of treatment tool <NUM>. In further embodiments, application <NUM> may only determine a line extending from each of the identified potential distal portions of treatment tool <NUM> through corresponding identified potential shaft portions of treatment tool <NUM> that are in line with at least one of the recommended entry points identified at step S308. For example, as shown in <FIG>, application <NUM> may determine a line <NUM> in the image data extending from an identified potential distal portion <NUM> through potential identified shaft portions to a recommended entry point <NUM>. Alternatively or additionally, application <NUM> may determine a plurality of lines <NUM> in the image data corresponding to outlines of tool <NUM> (representing treatment tool <NUM>), based on the image data and/or the characteristic data.

Next, at step S370, application <NUM> identifies high intensity areas in the image data along each determined line <NUM>. For example, application <NUM> may identify bright spots and/or areas in the image data along a length of each determined line <NUM>. High intensity areas may be indicative of metallic objects, such as treatment tool <NUM>. Application <NUM> then, at step S372, includes portions of the high intensity areas within a radius of each determined line <NUM> as part of an identified potential treatment tool <NUM> in the image data. The portions of the high intensity areas within the radius of each determined line <NUM> may correspond to potential remaining portions of treatment tool <NUM>. For example, application <NUM> may include all high intensity areas within a radius determined based on the characteristic data (received at step S352) as part of the identified potential treatment tool <NUM>. In an embodiment where application <NUM> determines a plurality of lines <NUM> at step S370, application <NUM> may include only high intensity areas within the area included in the outlines of treatment tool <NUM>, as indicated by the determined lines <NUM>, as part of the identified potential treatment tool <NUM>.

Likewise, application <NUM>, at step S374, excludes portions of the high intensity areas outside of the radius of each determined line <NUM> from the identified potential treatment tool <NUM> in the image data. In an embodiment where application <NUM> determines a plurality of lines <NUM> at step S370, application <NUM> may exclude all high intensity areas not within the area included in the outlines of treatment tool <NUM>, as indicated by the determined lines <NUM>, from being part of the identified potential treatment tool <NUM>. Application <NUM> may further fill in any gaps or omissions in the area within the radius from each determined line <NUM>, and thus may be expected to be included in the identified potential treatment tool <NUM>.

Next, at step S376, application <NUM> determines whether each identified potential treatment tool <NUM> is a valid or invalid treatment tool <NUM>. For example, as noted above, a line is determined from each of the identified potential distal portions of treatment tool <NUM> through corresponding identified potential shaft portions of treatment tool <NUM>. Thus, a plurality of potential treatment tools <NUM> may be identified. Application <NUM> may further process the image data based on the characteristic data received at step S352 to determine automatically and/or via input from the clinician, which, if any, of the potential treatment tools <NUM> identified in the image data is a valid treatment tool <NUM>. In embodiments, multiple treatment tools <NUM> may be inserted into the patient's body and identified concurrently, thus in some embodiments, application <NUM> may determine at step S376 that multiple potential treatment tools <NUM> are valid treatment tools <NUM>. After all potential treatment tools <NUM> are analyzed and determined to be valid or invalid, processing returns to step S310, where it is again determined whether treatment tool <NUM> has been identified in the image data.

Processing of a second exemplary algorithm may start at step S362 where application <NUM> receives characteristic data of treatment tool <NUM>. As with the first exemplary algorithm, the characteristic data may include a type of treatment tool <NUM>, such as an ablation needle, being used, a length of treatment tool <NUM>, a diameter of treatment tool <NUM>, a flexibility metric (such as Young's modulus) of treatment tool <NUM>, a location of one or more EM sensors included in treatment tool <NUM>, a location of distal radiating portion <NUM> in treatment tool <NUM>, locations of radiolucent fiducial markers and/or features designed to be visible under ultrasound imaging, etc. The characteristic data may be accessed by application <NUM> from memory <NUM>, may be inputted by the clinician via input device <NUM>, and/or may be provided to application <NUM> by treatment tool <NUM> and/or generator <NUM>.

Thereafter, application <NUM> may receive additional image data of the patient's body and, at step S364, process the additional image data to identify a portion of treatment tool <NUM> inserted through the patient's skin. For example, application <NUM> may analyze image data of an area proximate the recommended entry points determined at step S308 to identify a portion of treatment tool <NUM> inserted through the patient's skin. Application <NUM> may further determine an angle of insertion of treatment tool <NUM> through the patient's skin based on the identified portion of treatment tool <NUM>, and may thus determine a trajectory of treatment tool <NUM> based on the entry point of treatment tool <NUM> through the patient's skin and the angle of insertion. For example, as shown in <FIG>, application <NUM> may analyze image data of a patient's body ("P") to identify a portion <NUM> of a tool <NUM> inserted through the skin of body P.

Next, at step S366, application <NUM> identifies potential distal portions of treatment tool <NUM> in the image data. Similar to step S354, application <NUM> may process the image data received at step S302 and/or additional image data received subsequently and throughout the medical procedure, to identify one or more distal portions of treatment tool <NUM>, such as based on the characteristic data of treatment tool <NUM> received at step S362. In embodiments, application <NUM> may process only a portion of the image data that includes the patient's body, an area proximate the target location determined at step S308, and/or an area proximate the entry point determined at step S364. For example, application <NUM> may identify one or more areas of high intensity pixels having a shape similar to an eclipse as potential distal portions of treatment tool <NUM>. In embodiments, application <NUM> may determine a depth that treatment tool <NUM> is inserted into the patient's body, and may then seek to identify potential distal portions of treatment tool <NUM> that are about a corresponding distance from the entry point determined at step S364. The depth that treatment tool <NUM> is inserted into the patient's body may be determined based on lines and/or markers on treatment tool <NUM> (not shown in <FIG>), and/or one or more EM sensors included in treatment tool <NUM>. Additionally or alternatively, application <NUM> may identify the distal portion of treatment tool <NUM> based on the location of the one or more EM sensors, radiolucent fiducial markers, and/or other radiopaque elements included in treatment tool <NUM>. For example, the position of the one or more EM sensors and/or radiolucent fiducial markers included in treatment tool <NUM> relative to the distal portion of treatment tool <NUM> may be known, and thus the location of the distal portion of treatment tool <NUM> may be determined based on a detected position of the one or more EM sensors and/or radiolucent fiducial markers.

Thereafter, at step S368, application <NUM> determines a line between the entry point determined at step S364 and each of the potential distal portions of treatment tool <NUM> identified at step S366. For example, as shown in <FIG>, application <NUM> may determine a line <NUM> in the image data extending along a central axis of tool <NUM> (representing treatment tool <NUM>) from a portion <NUM> of treatment tool <NUM> identified in the patient's skin to an identified potential distal portion of treatment tool <NUM>, based on the characteristic data of treatment tool <NUM> received at step S362. An angle and/or trajectory of line <NUM> may be based on the angle and/or trajectory of treatment tool <NUM> inserted through the patient's skin as determined at step S364. Alternatively or additionally, application <NUM> may determine a plurality of lines <NUM> in the image data corresponding to outlines of tool <NUM> (representing treatment tool <NUM>), based on the image data and/or the characteristic data. Thereafter, processing proceeds to step S370, which is performed as described above in the description of the first exemplary algorithm.

Returning now to <FIG>, at step S312, application <NUM> determines a path from the entry point where treatment tool <NUM> is inserted though the patient's skin (as identified at step S308 and/or step S364) to the target location identified at step S308. For example, as shown in <FIG>, application <NUM> may determine a path <NUM> from the entry point to a target <NUM>.

Thereafter, at step S314, application <NUM> determines whether treatment tool <NUM> is following the path determined at step S312. In embodiments, application <NUM> may receive further image data of one or more portions of the patient's body, and may determine an angle of insertion of treatment tool <NUM> through the patient's skin, and thus a trajectory of treatment tool <NUM>. Application <NUM> may then compare the trajectory of treatment tool <NUM> with the path determined at step S312 to determine whether the trajectory of treatment tool <NUM> corresponds to the path determined at step S312, and thereby determine whether treatment tool <NUM> is following the path determined at step S312. For example, as shown in <FIG>, application <NUM> may determine a trajectory <NUM> of tool <NUM> (representing treatment tool <NUM>). Application <NUM> may further determine a difference between the trajectory <NUM> of tool <NUM> and the path <NUM>. If it is determined that treatment tool <NUM> is not following the path ("No" at step S314), processing proceeds to step S316, where application <NUM> generates and causes display device <NUM> and/or display <NUM> to display guidance for adjusting the position and/or angle of treatment tool <NUM>. For example, as shown in <FIG>, application <NUM> may generate instructions to guide the clinician on how to adjust the position of treatment tool <NUM> and/or navigate treatment tool <NUM> to the target location (represented by target <NUM>). In the example shown in <FIG>, application <NUM> causes display device <NUM> and/or display <NUM> to display guidance <NUM> instructing the clinician to adjust the angle of treatment tool <NUM> by <NUM> degrees in the direction shown by an arrow <NUM>, and to insert treatment tool <NUM><NUM> further into the patient's body. Thereafter, processing returns to step S314, where application <NUM> again determines if treatment tool <NUM> is following the path determined at step S312. If it is determined at step S314 that treatment tool <NUM> is following the path ("Yes" at step S314), processing proceeds to step S318.

At step S318, application <NUM> determines whether treatment tool <NUM> has reached the target location. For example, application <NUM> may receive further image data of the portion of the patient's body, and may again identify treatment tool <NUM> in the image data, as described above, to determine whether treatment tool <NUM> has been placed at the target location. Additionally or alternatively, application <NUM> may receive input from the clinician, such as via input device <NUM> of computing device <NUM>, indicating that treatment tool <NUM> has been placed at the target location. If application <NUM> determines that treatment tool <NUM> has not reached the target location ("No" at step S318), processing returns to step S314. Alternatively, if application <NUM> determines that treatment tool <NUM> has reached the target location ("Yes" at step S318), processing proceeds to step S320.

Turning now to <FIG>, at step S320, application <NUM> may receive configuration settings for an ablation procedure. In some embodiments, the configuration settings are received earlier in the medical procedure or are preconfigured prior to the start of the medical procedure. The configuration settings may include a location of the ablation procedure, identified anatomical structures proximate the location of the ablation procedure, a duration of the ablation procedure, a wattage that will be output by treatment tool <NUM> during the ablation procedure, modeled ablation procedure performance, etc. The configuration settings may be preconfigured, such as included in or based on a treatment plan configured by a clinician prior to the start of the medical procedure, and/or may be input by the clinician at the start of, or during, the medical procedure, such as by using input device <NUM> of computing device <NUM>.

Next, at step S322, application <NUM> identifies radiating portion <NUM> of treatment tool <NUM>. For example, application <NUM> may determine a location of radiating portion <NUM> based on the characteristic data of treatment tool <NUM> received at step S321. Thereafter, at step S324, application <NUM> determines a projected ablation zone. The determination of the projected ablation zone may be based on the identified location of radiating portion <NUM> and the configuration settings for the ablation procedure received at step S320. Application <NUM> may then, at step S326, cause display <NUM> of computing device <NUM> and/or display device <NUM> to display the projected ablation zone. The projected ablation zone may be displayed on the image data. Alternatively or in addition, the projected ablation zone may be displayed on the 3D model generated at step S306. For example, as shown in <FIG>, a projected ablation zone <NUM> may be displayed as centered on a radiating portion <NUM> (representing radiating portion <NUM>) of tool <NUM>. As will be appreciated by those skilled in the art, projected ablation zone <NUM> may be selectively displayed at any point during the medical procedure, and is not necessarily limited to being displayed only after treatment tool is placed at the target location. Thus, as shown in <FIG>, projected ablation zone <NUM> is displayed while tool <NUM> is being navigated to target <NUM>, such that the clinician may see the area of tissue that is within the projected ablation zone based on a current location of treatment tool <NUM> and the configuration settings for the ablation procedure.

Thereafter, at step S328, it is determined whether radiating portion <NUM> of treatment tool <NUM> has been activated. For example, treatment tool <NUM> and/or generator <NUM> may notify computing device <NUM>, and thus application <NUM>, that a button, trigger, and/or activation switch has been activated allowing microwave energy to be emitted from radiating portion <NUM> of treatment tool <NUM>. If it is determined that radiating portion <NUM> has not been activated ("No" at step S328), processing proceeds to step S334.

Alternatively, if it is determined that radiating portion <NUM> has been activated ("Yes" at step S328), processing proceeds to step S330 where application <NUM> determines a progress of an ablation procedure. The determination of the progress of the ablation procedure may be based on the configuration settings received at step S320, the projected ablation zone determined at step S326, and/or an elapsed time since the radiating portion was activated. Thereafter, at step S332, application <NUM> may display an estimated progress of the ablation procedure. For example, application <NUM> may cause display <NUM> of computing device <NUM> and/or display device <NUM> to display the estimated progress of the ablation zone. Similar to the projected ablation zone <NUM> displayed at step S326, the estimated progress of the ablation procedure may be displayed on the image data and/or the 3D model.

Next, at step S334, application <NUM> determines whether the ablation procedure has been completed. The determination whether the ablation procedure has been completed may be based on the estimated progress of the ablation procedure determined at step S332 and/or the configuration settings received at step S320. If it is determined that the ablation procedure has not been completed ("No" at step S334), processing returns to step S328. Alternatively, if it is determined that the ablation procedure has been completed ("Yes" at step S334), processing ends.

Turning now to <FIG>, there is shown an exemplary graphical user interface (GUI) <NUM> that may be displayed by computing device <NUM> and/or display device <NUM> at various times during the above-described medical procedure. GUI <NUM> may include, or be based on, the image data received at step S302 and/or the 3D model generated at step S304 of method <NUM> of <FIG>. GUI <NUM> may show physiological structures <NUM> and identified portions <NUM> of treatment tool <NUM>. Physiological structures <NUM> may be any physiological structures identifiable in the image data and/or 3D model that are relevant to the medical procedure, and may be selectively displayed based on the clinician's preference.

Claim 1:
The system (<NUM>) for identifying a percutaneous tool (<NUM>) in image data comprising:
a display device (<NUM>); and
a computing device (<NUM>) including:
a processor (<NUM>); and
a memory (<NUM>) storing instructions (<NUM>) which, when executed by the processor, cause the computing device to perform the following steps:
receiving (S302) image data of at least a portion of a patient's body;
identifying (S364) an entry point of a percutaneous tool through the patient's skin in the image data;
analyzing (S366) a portion of the image data including the entry point of the percutaneous tool through that patient's skin to identify a portion of the percutaneous tool inserted through the patient's skin;
determining (S368) a trajectory of the percutaneous tool based on the identified portion of the percutaneous tool inserted through the patient's skin;
identifying (S370) a remaining portion of the percutaneous tool in the image data based on the identified entry point and the determined trajectory of the percutaneous tool; and
displaying the identified portions of the percutaneous tool on the image data.