Motion correction of angiography images for 3D reconstruction of coronary arteries

Systems and methods for computing a transformation for correction motion between a first medical image and a second medical image are provided. One or more landmarks are detected in the first medical image and the second medical image. A first tree of the anatomical structure is generated from the first medical image based on the one or more landmarks detected in the first medical image and a second tree of the anatomical structure is generated from the second medical image based on the one or more landmarks detected in the second medical image. The one or more landmarks detected in the first medical image are mapped to the one or more landmarks detected in the second medical image based on the first tree and the second tree. A transformation to align the first medical image and the second medical image is computed based on the mapping.

TECHNICAL FIELD

The present invention relates generally to motion correction of angiography images, and more particularly to motion correction of angiography images for 3D reconstruction of coronary arteries.

BACKGROUND

Coronary heart disease is caused by a blockage or narrowing (stenosis) of the arteries that supply blood to the heart, typically due to the accumulation of cholesterol plaque on the arterial walls. X-ray coronary angiography is an imaging modality for diagnosis and guidance of therapeutic procedures for coronary heart disease. X-ray coronary angiography is popular due to its ability to assist in both diagnosis and therapy of coronary heart disease, as well as its high spatio-temporal resolution. However, accurately reconstructing 3D coronary arteries from angiography images remains a challenge due to the loss of information from the projectional radiography. In particular, respiration motion must be accounted for in order to accurately reconstruct 3D coronary arteries.

BRIEF SUMMARY OF THE INVENTION

In accordance with one or more embodiments, systems and methods for computing a transformation to correct motion between a plurality of medical images are provided. One or more landmarks are detected in a first medical image and a second medical image. A first tree of the anatomical structure is generated from the first medical image based on the one or more landmarks detected in the first medical image and a second tree of the anatomical structure is generated from the second medical image based on the one or more landmarks detected in the second medical image. The one or more landmarks detected in the first medical image are mapped to the one or more landmarks detected in the second medical image based on the first tree and the second tree. A transformation to align the first medical image and the second medical image is computed based on the mapping.

In one embodiment, the first tree comprises the one or more landmarks detected in the first medical image, the second tree comprises the one or more landmarks detected in the second medical image, and the mapping is performed by, for each respective landmark of the one or more landmarks in the first tree, computing a set of candidate mappings between the respective landmark and the one or more landmarks in the second tree, filtering the set of candidate mappings to remove candidate mappings where a descendant of the respective landmark is not mapped to a descendant of a particular landmark of the candidate mapping in the second tree, and selecting a candidate mapping from the filtered set of candidate mappings based on a distance associated with each candidate mapping. The set of candidate mappings may comprise all possible mappings between the respective landmark and the one or more landmarks in the second tree.

In one embodiment, the transformation is computed by projecting the one or more landmarks detected in the first medical image to respective epipolar lines of the one or more landmarks in the second medical image, determining a transformation of the second medical image to move the one or more landmarks in the second medical image towards a closest point of its respective epipolar line, applying the transformation to the second medical image to move the one or more landmarks in the second medical image, and repeating the projecting, the determining, and the applying until a stopping condition is satisfied.

In one embodiment, the first tree is generated to include the one or more landmarks detected in the first medical image between a first start point and a first end point selected by a user and the second tree is generated to include the one or more landmarks detected in the second medical image between a second start point and a second end point selected by the user.

In one embodiment, the anatomical structure is a coronary artery and the one or more landmarks are detected by detecting one or more bifurcations of the coronary artery in the first medical image and the second medical image. The first medical image and the second medical image may be different views of the anatomical structure and may be x-ray angiography images.

In one embodiment, the one or more landmarks may be detected in one or more additional medical images of the anatomical structure, a tree of the anatomical structure may be generated for each respective image of the additional medical images based on the landmarks detected in the respective image, the landmarks detected in the first medical image may be mapped with the landmarks detected in the second medical image and the landmarks detected in the additional medical images, and the transformation may be computed to align the first medical image, the second medical image, and the additional medical images based on the mapping.

DETAILED DESCRIPTION

The present invention generally relates to methods and systems for motion correction of angiography images for 3D reconstruction of coronary arteries. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems for motion correction of angiography images for 3D reconstruction of coronary arteries. A digital image is often composed of digital representations of one or more objects (or shapes). The digital representation of an object is often described herein in terms of identifying and manipulating the objects. Such manipulations are virtual manipulations accomplished in the memory or other circuitry/hardware of a computer system. Accordingly, is to be understood that embodiments of the present invention may be performed within a computer system using data stored within the computer system.

Further, it should be understood that while the embodiments discussed herein may be discussed with respect to motion correction of angiography images for 3D reconstruction of coronary arteries, the present invention is not so limited. Embodiments of the present invention may be applied for aligning images for any application.

FIG. 1shows a workflow100for compensating motion between x-ray angiography images, in accordance with one or more embodiments. In workflow100, x-ray angiography images102and104are input into an artificial intelligence (AI) system106trained for landmark detection. X-ray angiography images102and104are different views of a same coronary artery of a patient, and are not aligned due to respiratory motion of the patient during image acquisition. AI system106outputs heat map images108and110identifying bifurcations112and114(or other anatomical landmarks) of the coronary artery in x-ray angiography images102and104, respectively. A mapping116between bifurcations112and114is determined and an in-plane motion matrix118is computed. In-plane motion matrix118represents a transformation to align x-ray angiography images102and104to spatially correspond with each other to thereby compensate for motion of the patient. In-plane motion matrix118may be used to transform bifurcations114of x-ray angiography image104to be closer to projected epipolar lines of bifurcations112of x-ray angiography image102.

FIG. 2shows a method200for aligning medical images, in accordance with one or more embodiment. The steps of method200may be performed by any suitable computing device, such as, e.g., computer702ofFIG. 7. Method200will be described with reference to workflow100ofFIG. 1.

At step202, a first medical image and a second medical image of an anatomical structure are received. In one embodiment, the anatomical structure is a coronary artery of a patient, however the anatomical structure may be any suitable anatomical structure of the patient. In one embodiment, the first medical image and the second medical image received at step202are x-ray angiography images102and104ofFIG. 1depicting a coronary artery.

In one embodiment, the first medical image and the second medical image depict different views of the anatomical structure. For example, the first medical image and the second medical image may be acquired at a same time or at different times at different positions with respect to the anatomical structure (e.g., with a certain separation angle between the acquisition of the first medical image and the second medical image). In another embodiment, the first medical image and the second medical image depict a same view of the anatomical structure acquired at different times, but at different states of deformation of the anatomical structure, e.g., due to motion (e.g., respiratory motion) of the patient.

In one embodiment, the first medical image and the second medical image are x-ray angiography images, however it should be understood that the first medical image and the second medical image may be of any suitable modality, such as, e.g., x-ray, magnetic resonance imaging (MRI), ultrasound (US), single-photon emission computed tomography (SPECT), positron emission tomography (PET), or any other suitable modality or combination of modalities. The first medical image and the second medical image may be received directly from an image acquisition device, such as, e.g., image acquisition device714ofFIG. 7(e.g., an x-ray scanner, etc.), used to acquire the medical images. Alternatively, the first medical image and the second medical image may be received by loading medical images previously stored on a memory or storage of a computer system (e.g., a picture archiving and communication system, PACS) or by receiving medical image data via network transmission from a remote computer system.

At step204, one or more landmarks are detected in the first medical image and the second medical image. In one embodiment, for example where the anatomical structures is a coronary artery, the landmarks comprise corresponding bifurcations of the coronary artery detected in both the first medical image and the second medical image. Detecting such bifurcations is advantageous as the bifurcations define the geometry of the underlying coronary artery and generally coexist across different views of the coronary artery. The detected landmarks may be identified in any suitable form, such as, e.g., heat maps, binary maps, etc. In one embodiment, the landmarks detected at step204are bifurcations112and114identified on heat maps108and110detected from x-ray angiography images102and104, respectively, inFIG. 1. In one embodiment, the landmarks comprise a catheter tip or stenosis. It should be understood that the landmarks may be any suitable landmark representing anatomically meaningful locations on organs, bones, blood vessels, etc.

In one embodiment, the landmarks are detected using a machine learning network. Such machine learning network is illustratively represented as AI system106inFIG. 1. In one example, the machine learning network may be a fully convolutional network (FCN) having an encoder-decoder structure, as shown inFIG. 3. However, it should be understood that the machine learning network may have any suitable design or architecture, and is not limited to the network architecture shown inFIG. 3.

FIG. 3shows a network architecture300for training an FCN, in accordance with one or more embodiments. As shown inFIG. 3, the FCN has an encoder-decoder structure comprising encoder304and decoder306. During a training or offline stage, as shown inFIG. 3, encoder304receives as input a training image302(e.g., an x-ray angiography training image) depicting a coronary artery and codes training image302into a code that is substantially less in size than that of training image302. Decoder306decodes the code to generate heat map310identifying bifurcations of the coronary artery depicted in training image302. Layer308is the last layer of feature tensors before the final output layer. All the intermediate information generated in encoder304is shared with decoder306so that no information is lost in the encoding process. A training loss314is defined between heat map310and ground truth annotated image312of bifurcations in training image302. The locations of the bifurcations in ground truth annotated image312are diffused with Gaussian blurring to account for the uncertainty of annotators and to ease the training. During an online or inference stage, the trained FCN may be applied to detect landmarks in the first medical image and the second medical image at step204ofFIG. 2, in accordance with one embodiment. In particular, the trained FCN receives one or more input images (e.g., the first medical image and the second medical image received at step202ofFIG. 2) and outputs a heat map for each input image identifying landmarks in that input image. The heat map has the same resolution and size as the input image. The location (e.g., coordinates) of the landmarks may be determined by applying image process techniques, such as, e.g., thresholding and independent component analysis. In one embodiment, in addition to the one or more input images, temporally neighboring frames are also input into the trained FCN. In one embodiment, the input images include a channel comprising a centerline of the coronary artery in the input images, which may serve as an attention map and improve the overall detection performance.

At step206ofFIG. 2, a first tree of the anatomical structure is generated from the first medical image based on the one or more landmarks detected in the first medical image and a second tree of the anatomical structure is generated from the second medical image based on the one or more landmarks detected in the second medical image. The trees comprise a plurality of points representing a path from a start point on the anatomical structure to one or more end points in the first medical image and the second medical image. For example, where the anatomical structure is a coronary artery, the trees comprise a plurality of points representing a path from a root of the coronary artery to one or more leaves of the coronary artery.

The start and end points are defined in the first medical image and the second medical image based on input received from a user (e.g., a clinician). For example, a user may interact with a computing device (e.g., using a mouse) to select seeds defining the start and end points of the anatomical structure in the first medical image and in the second medical image. The first tree is generated based on the first medical image, the landmarks detected in the first medical image, and the start and end points defined in the first medical image. The second tree is generated based on the second medical image, the landmarks detected in the second medical image, and the start and end points defined in the second medical image. The first tree and the second tree are generated to include points corresponding to the detected landmarks in the first medical image and the second medical image, respectively. The first tree and the second tree may be automatically constructed based on, e.g., tracking-based methods, graph-based methods, or any other suitable method. In one embodiment, the first tree and the second tree are generated according to the method disclosed in U.S. Pat. No. 10,206,646, entitled “Method and System for Extracting Centerline Representation of Vascular Structures in Medical Images via Optimal Paths in Computational Flow Fields,” the disclosure of which is incorporated herein by reference in its entirety.

At step208, the one or more landmarks detected in the first medical image are mapped to the one or more landmarks detected in the second medical image based on the first tree and the second tree.FIG. 1illustratively shows mappings116between landmarks, in accordance with one embodiment. The mapping may be represented as a non-surjective injective mapping function M that maps each landmark in the first tree (i.e., each landmark detected in the first medical image) to a same corresponding landmark in the second tree (i.e., a same corresponding landmark detected in the second medical image). Formally, given the first tree T1(V1, E1) having vertices V1 representing points in tree T1 and edges E1 connecting the vertices V1 and the second tree T2(V2, E2) having vertices V2 representing points in tree T2 and edges E2 connecting the vertices V2, mapping function M matches a landmark n1 εT1 to a landmark M(n1)εT2. In one embodiment, an optimal mapping M* of landmarks in the first tree to corresponding landmarks in the second tree is determined according method400ofFIG. 4. In one embodiment, the mapping may be determined for all points in the first tree and the second tree, and is not limited to the landmarks.

FIG. 4shows a method400for mapping a landmark in a first tree to a same corresponding landmark in a second tree, in accordance with one or more embodiments. The steps of method400may be performed at step208ofFIG. 2for each respective landmark in the first tree.

At step402, a set of candidate mappings for the respective landmark in the first tree is computed. The set of candidate mappings for the respective landmark in the first tree represents all possible mappings between the respective landmark in the first tree and landmarks in the second tree. If the first tree and the second tree have a different number of landmarks, a mapping of N points is performed, where N is the number of landmarks in the tree having the fewest landmark.

At step404, the set of candidate mappings for the respective landmark in the first tree is filtered to remove candidate mappings that are ancestry-violating. A candidate mapping for a respective landmark n1 is ancestry-violating if a descendant of the respective landmark n1εT1 is not mapped to a descendant of a landmark M(n1) εT2 of the candidate mapping. A descendent of a landmark is any point further down the tree, from the start point to the end point. A candidate mapping that is not ancestry-violating is considered to be ancestry-respectful.

At step406, a candidate mapping is selected for the respective landmark in the first tree from the filtered set of candidate mappings for the respective landmark of the first tree. In one embodiment, the candidate mapping that has a minimum cost is selected from the set of candidate mappings. For example, in one embodiment, the candidate mapping having a minimum cost may be the candidate mapping associated with a shortest distance (e.g., Euclidean distance metric) to epipolar lines. Specifically, in one embodiment, for each candidate mapping, a landmark P1in image A is mapped to a landmark P2in image B. As the landmark P1in image A is projected to an epipolar line L1in image B, the Euclidean distance may be calculated between L1and P2as the cost. The sum of all Euclidean distances between landmarks in image B and the projected epipolar lines from image A is the overall cost for this candidate mapping. Among all candidate mappings, the one that has the minimum cost will be the final optimal one. In another embodiment, the candidate mapping having a minimum cost may be determined based on the position of the candidate mapping relative to the epipolar line. Specifically, the epipolar line separates the images into two regions and different costs may be associated with the candidate mapping based on which region the candidate mapping is located. It should be understood that the cost may be any other suitable metric.

In one embodiment, method400is not performed for each landmark in the first tree. Instead the quality of the mapping is compared for a different number of landmarks, not necessarily all landmarks in the first tree. This will make the mapping more robust to bifurcation false positive detection. In another embodiment, method400is performed for all points in the first tree and is not limited to the landmarks in the first tree.

At step210ofFIG. 2, a transformation to align the first medical image and the second medical image is computed based on the mapping. In one embodiment, the transformation is a motion compensation transformation for compensating for motion (e.g., respiratory motion) of the patient. In one embodiment, the transformation may be an in-plane motion matrix, as illustratively shown as in-plane motion matrix118inFIG. 1, however the transformation may be in any suitable form. The transformation may be computed according to method500ofFIG. 5, in accordance with one embodiment.

FIG. 5shows a method500for determining a transformation to align a first medical image and a second medical image, in accordance with one or more embodiments. The steps of method500may be performed at step210ofFIG. 2.

At step502, landmarks in the first medical image are projected to respective epipolar lines of the landmarks in the second medical image. An epipolar line in the second medical image represent possible points in the second medical image where a particular landmark depicted in the first medical image may be located.

Referring toFIG. 6, a schematic diagram600is shown illustratively depicting epipolar geometry of a landmark visualized in different imaging planes, in accordance with one or more embodiments. Schematic diagram600shows imaging plane602of a landmark P610acquired by image acquisition device A1606and imaging plane604of landmark P610acquired by image acquisition device A2608, which may be the same or different than image acquisition device A1606. In one embodiment, imaging plane602is the first medical image, imaging plane604is the second medical image, and landmark P1612is the respective landmark in method500ofFIG. 5. It should be understood that landmark P610may be located between image acquisition devices A1606and A2608and imaging planes602and604, respectively, where, e.g., image acquisition devices A1606and A2608are x-ray image acquisition devices.

Landmark P610is captured in imaging plane602by image acquisition device A1606along line of sight616as point P1612and in imaging plane604by image acquisition device A2608along line of sight618as point P2614. When projecting point P1612in imaging plane602to imaging plane604, point P1612may be located along any point of line of sight616that is visible in imaging plane604, such as exemplary candidate points624. The portion of line of sight616visible in imaging plane604is referred to as an epipolar line620.

At step504ofFIG. 5, a transformation X of the second medical image is determined to move the landmarks in the second medical image towards a closest point of its respective epipolar line. The transformation X may be any transformation, such as, e.g., rigid or affine transformations, depending on the number of landmarks. The closest point may be determined based on the Euclidean distance or any other suitable distance metric. As shown inFIG. 6, a transformation X is determined that transforms imaging plane604to move point P2614towards point626.

At step506, the transformation X is applied to the second medical image to move the landmarks in the second medical image.

At step508, it is determined whether a stopping condition is satisfied. In one embodiment, the stopping condition is satisfied when the transformation X converges (i.e., is close to an identity matrix). In another embodiment, the stopping condition is satisfied after a predetermined number of iterations. Other criteria for the stopping condition are also contemplated. If the stopping condition is not satisfied at step508, method500returns to step502for another iteration. If the stopping condition is satisfied at step508, method500ends at step510. The transformation determined after one or more iterations of method500represents the transformation to align the first medical image and the second medical image.

At step212ofFIG. 2, the transformation is output. For example, the transformation can be output by displaying the transformation on a display device of a computer system, storing the transformation on a memory or storage of a computer system, or by transmitting the transformation to a remote computer system.

Advantageously, embodiments of the present invention provide for an automatic landmark detection and motion correction method. In one embodiment, the embodiments of the present invention may be applied to correct motion between the first medical image and the second medical image for 3D reconstruction of a coronary artery.

It should be understood that while method200ofFIG. 2is described for aligning a first medical image and a second medical image, the present invention may be applied to align any plurality of medical images. For example, the one or more landmarks may be detected in one or more additional medical images of the anatomical structure, a tree of the anatomical structure may be generated for each respective image of the additional medical images based on the landmarks detected in the respective image, step208may be performed by mapping the landmarks detected in the first medical image with the landmarks detected in the second medical image and the landmarks detected in the additional medical images, and step210may be performed by computing the transformation to align the first medical image, the second medical image, and the additional medical images based on the mapping.

Systems, apparatus, and methods described herein may be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor that is connected to a network communicates with one or more client computers via a network. A client computer may communicate with the server via a network browser application residing and operating on the client computer, for example. A client computer may store data on the server and access the data via the network. A client computer may transmit requests for data, or requests for online services, to the server via the network. The server may perform requested services and provide data to the client computer(s). The server may also transmit data adapted to cause a client computer to perform a specified function, e.g., to perform a calculation, to display specified data on a screen, etc. For example, the server may transmit a request adapted to cause a client computer to perform one or more of the steps or functions of the methods and workflows described herein, including one or more of the steps or functions ofFIGS. 2 and 4-5. Certain steps or functions of the methods and workflows described herein, including one or more of the steps or functions ofFIGS. 2 and 4-5, may be performed by a server or by another processor in a network-based cloud-computing system. Certain steps or functions of the methods and workflows described herein, including one or more of the steps ofFIGS. 2 and 4-5, may be performed by a client computer in a network-based cloud computing system. The steps or functions of the methods and workflows described herein, including one or more of the steps ofFIGS. 2 and 4-5, may be performed by a server and/or by a client computer in a network-based cloud computing system, in any combination.

A high-level block diagram of an example computer702that may be used to implement systems, apparatus, and methods described herein is depicted inFIG. 7. Computer702includes a processor704operatively coupled to a data storage device712and a memory710. Processor704controls the overall operation of computer702by executing computer program instructions that define such operations. The computer program instructions may be stored in data storage device712, or other computer readable medium, and loaded into memory710when execution of the computer program instructions is desired. Thus, the method and workflow steps or functions ofFIGS. 2 and 4-5can be defined by the computer program instructions stored in memory710and/or data storage device712and controlled by processor704executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform the method and workflow steps or functions ofFIGS. 2 and 4-5. Accordingly, by executing the computer program instructions, the processor704executes the method and workflow steps or functions ofFIGS. 2 and 4-5. Computer702may also include one or more network interfaces706for communicating with other devices via a network. Computer702may also include one or more input/output devices708that enable user interaction with computer702(e.g., display, keyboard, mouse, speakers, buttons, etc.).

Processor704may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of computer702. Processor704may include one or more central processing units (CPUs), for example. Processor704, data storage device712, and/or memory710may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and/or one or more field programmable gate arrays (FPGAs).

Input/output devices708may include peripherals, such as a printer, scanner, display screen, etc. For example, input/output devices708may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to the user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to computer702.

An image acquisition device714can be connected to the computer702to input image data (e.g., medical images) to the computer702. It is possible to implement the image acquisition device714and the computer702as one device. It is also possible that the image acquisition device714and the computer702communicate wirelessly through a network. In a possible embodiment, the computer702can be located remotely with respect to the image acquisition device714.

Any or all of the systems and apparatus discussed herein may be implemented using one or more computers such as computer702.

One skilled in the art will recognize that an implementation of an actual computer or computer system may have other structures and may contain other components as well, and thatFIG. 7is a high level representation of some of the components of such a computer for illustrative purposes.