Patent Description:
Overall prevalence of heart failure with preserved ejection fraction (HFpEF) has been known to be <NUM>-<NUM>% in the general population and is typically related to diastolic dysfunction. It is further known that the analysis of the mitral valve annulus (MVA) throughout the cardiac cycle might act, amongst others, as a predictor for HFpEF.

Currently, the diagnosis of heart failure, especially with preserved ejection fraction, remains extremely challenging due to the complicatedness of the disease and its early subtle effects on the motion of the heart and the mitral valve, especially. The interplay between motion and flow remains unsatisfactorily understood. It is possible to extract motion related parameters of the heart from a echo-doppler acqusitions and/or MR images. These approaches are however time-consuming and do not provide satisfactory results.

The publication of <NPL> describes a method for motion tracking on cardiac MRI images, which employs deep neural networks.

The publication of <NPL> describes a method to segment heart structures in cardiac MRI images based on two connected neural networks.

Accordingly, a need exists to improve the determination of motion parameters of the heart.

According to a first aspect a method for determining a motion parameter of the heart is provided wherein the method comprises the step of determining a sequence of cardiac MR images showing a time resolved motion of the heart. Furthermore, a subset of the sequence of cardiac MR images is applied as a first input to a first trained convolutional neural network, which is configured to determine, as first output, a probability distribution of at least two anatomical landmarks in the subset. The sequence of cardiac MR images is cropped and aligned based on the at least two anatomical landmarks in order to determine a reframed and aligned sequence of new cardiac MR images, wherein all the new images of the reframed and aligned sequence show the same orientation of the heart. The reframed and aligned sequence of new cardiac MR images is applied to a second trained convolutional neural network which is configured to determine, as a second output, a further probability distribution of the at least two anatomical landmarks in each new MR image of the reframed and aligned sequence. Finally, the motion parameter of the heart is determined based on the second output.

The proposed method provides a robust and fully automated algorithm for the detection of the motion parameter. The first trained convolutional neural network is configured to identify regions of interest including the two anatomical landmarks such as the mitral valve, wherein the second trained convolutional neural network extracts the landmark in the time resolved images of the heart using the identified regions of interest. The movement of the landmark can then be used to determine the required motion parameters.

Furthermore, the corresponding entity is provided configured to determine the motion parameter wherein the entity comprises a memory and at least one processing unit which is configured to operate as discussed above or as discussed in further detail below.

Furthermore, a computer program comprising program code to be executed by at least one processing unit of the entity is provided, wherein the execution of the program code causes the at least one processing unit to carry out a method as discussed above or as discussed in further detail below.

Additionally a carrier comprising the computer program is provided wherein the carrier is one of an electronic signal, optical signal, radio signal, and computer readable storage medium.

The foregoing and additional features and effects of the invention will become apparent from the following detailed description when read in conjunction with the accompanying drawings in which like reference numerals refer to like elements.

Rather, the various elements are represented such, that their function in general purpose becomes apparent to a person with skill in the art. Any connection or coupling between functional blocks, devices, components of physical or functional units shown in the drawings and described hereinafter may be implemented by an indirect or direct connection. A coupling between components may be established over a wired or wireless connection. Functional blocks may be implemented in hardware, software, firmware, or a combination thereof.

In the following, a fully automated algorithm for detecting the mitral valve annulus is disclosed, however it should be understood that any other valve of the heart may be used. The MR images can be a <NUM> chamber (2CHV), and/or <NUM> chamber (4CHV) MR images such as CMR (Cardiac Magnetic Resonance) images. The system discussed below initially detects the mitral valve region of interest before extracting the time resolved landmarks of the mitral valve annulus. This information is then used to extract the motion related parameters including displacements, velocities and diameters. The system performance is analyzed based on pre-annotated data sets, which were annotated by experts marking the desired region of interests. Thereafter, the motion parameters were extracted retrospectively on N=<NUM> unlabeled data sets. The system may automatically calculate motion related parameters such as the mitral valve velocities, mitral valve plane motion, mitral valve diameters and how these parameters evolve over time during the heartbeat. These parameters have shown to be clinically important for the automatic assessment of heart failure, especially for heart failure with preserved ejection fraction which is typically related to diastolic dysfunction.

<FIG> shows a schematic view of an MR imaging system <NUM>, which comprises a magnet <NUM> generating the magnetic field B0. The patient or object under examination <NUM> lying on a table <NUM> is moved into the center of the MR imaging system <NUM>, where the MR signals can be detected after excitation by RF pulses using coils <NUM>. By applying RF pulses and magnetic field gradients, the nuclear spins of object <NUM>, especially the part located in the receiving coils are excited and location coded currents induced by relaxation can be detected. The way how MR images, especially CINE images are generated and how the MR signals are detected, using a sequence of RF pulses and a sequence of magnetic field gradients, is known in the art, so that a detailed explanation thereof is omitted. The MR system may furthermore comprise shim coils <NUM> which are used to correct in-homogeneities of the magnetic field B0.

The MR imaging system <NUM> comprises a control module <NUM> which is used for controlling the MR imaging system. The control module <NUM> comprises a gradient control unit <NUM> for controlling and switching the magnetic field gradients, an RF control unit <NUM> for controlling and generating RF pulses for the imaging sequences. The image sequence control unit <NUM> is provided to control the sequence of the applied RF pulses and magnetic field gradients and thus is also configured to partly control the gradient control unit <NUM> and the RF control unit <NUM>. In a memory <NUM>, computer programs needed for operating the MR imaging system and the imaging sequences necessary for generating the MR images can be stored together with the generated MR images. The MR images and any further information can be displayed on a display <NUM> wherein a human machine interface <NUM> is provided, which can be used by an operator of the MR imaging system to control the MR imaging system. Furthermore a machine learning module <NUM> is provided which comprises a first trained neural network <NUM> and a second trained neural network <NUM>. The machine learning module with the two convolutional neural networks <NUM> and <NUM> is configured, as will be explained below to generate and output a likelihood distribution of certain anatomical landmarks of the heart such as the mitral valve annulus. A central processing unit <NUM> can coordinate the operation of the different functional units shown in <FIG> and can comprise one or more processors, which can carry out instructions stored on the memory <NUM>. The memory can include program code to be executed by the processing unit <NUM>.

<FIG> shows a schematic view of module or device <NUM> which is configured to automatically determine heart related parameters such as valve valocities or displacements or diameters.

The MR system <NUM> generates a time series of MR images of the heart, by way of example a two-chamber, <NUM> CHV, or a four-chamber view, <NUM> CHV, the two image options being schematically shown as image <NUM> and <NUM> in <FIG>. For each of the images a time series of images is present, a sequence of cardiac MR images. The first trained neural network <NUM> is a convolutional neural network and is configured to detect the landmark such as the mitral valve. The network <NUM> can be a 2D network in which a single image is input such as the first image of the time series of MR images. The output of the first neural network <NUM> is a heat map, such as the heat map <NUM> shown in <FIG>, in which the two landmarks, the mitral valves are shown. Based on the two landmarks, the image size is adjusted by cropping the image (e.g. to generate a square image size) as shown and the image is rotated, so that all images of the time series now have the same orientation, here the apex of the heart is oriented in the upward direction. Accordingly, the first image in the series is passed through the first neural network <NUM> in order to determine a more detailed view of the region of interest and to provide a defined orientation for all of the images of the time series.

It is possible to interpolate the sequence of cardiac MR images, such as <NUM> time frames, so that always the same number of MR images is present and can be used as input for the second network <NUM>. Both networks are chained convolutional neural networks and are both trained to detect landmarks based on a heat map regression task. The first network can be a residual 2D Unet as described inter alia in <NPL>. The network identifies the mitral valve annulus in both, the four-chamber view and/or the two-chamber view by regressing three landmarks on the first timeframe of each series. The third landmark can be the apex of the heart. After rotation as shown by the image <NUM> and after cropping and a pixel space interpolation, a series of reframed and aligned images is generated such as image <NUM>. These images are input into the second neural network a 3D UNet e.g. <NPL>), wherein the second network extracts the time resolved heat maps of both landmarks.

A postprocessing step as shown in image <NUM> fits a defined distribution such as a Gaussian distribution to refine the final landmark coordinates.

The result is a time series of images <NUM> or <NUM> in which the landmark such as the mitral valve annulus is indicated and marked. Based on the evolution of the landmarks it is possible to determine different motion parameters such as the mitral annulus velocities, the atrioventricular plane displacement, the atrioventricular plane velocities or the mitral valve annulus diameter. Accordingly, it is possible to determine clinically relevant parameters of interest such as the mitral annulus tissue velocity, the time resolved atrioventricular plane displacement and peak displacement or slice tracking of image slices based on atrioventricular plane displacement. Furthermore, it is possible to determine the time resolved atrioventricular plane velocity curves and early diastolic velocity, an indication for the systolic or diastolic function. Furthermore parameters such as the end systolic long axis mitral annular diameter can be determined or the mitral annular total motion quantification, such as the accumulation over the cardic cycle of the displacement for every landmark in millimeters. Furthermore, it is possible to determine the maximum minus the minimum displacement, such as the distance traveled by the lateral annulus from the end diastole to the end systole. Furthermore, it is possible to determine the mitral valve contraction , e.g. in mm such as the diameter contraction defining the time resolved difference between the maximum and the minimum diameter.

In the following training of the data used to train network <NUM> and <NUM> is discussed in more detail.

Ground truths annotated images from <NUM> subjects were provided which were generated at <NUM> and <NUM> Tesla, wherein the images showed two-chamber views and four-chamber views. The data included semi-automatically annotated landmarks showing the mitral valve annulus, MVA throughout the cardiac cycle. The mean in-plane resolution was <NUM> ± <NUM>.

Training: The model was trained from scratch using the Adaptive Wing Loss on the heatmaps while decreasing the heatmaps satndard deviation exponentially thoughout training epochs $$$ σ_{ep}=<NUM> \cdot <NUM>,<NUM>^{ep} $$$. The networks were trained using Adam optimizer with momentum of $$$\beta\ =\ <NUM>$$$ and learning rate $$$\labmda\ =\ <NUM>$$$\ with weight decay regularization. Online data augmentation was performed using random rotation, contrast enhancement, translation, maximum clipping, blurring and noise addition.

<FIG> shows a more detailed view of the feature extraction convolutional blocks. <FIG> shows the residual blocks <NUM> corresponding to the first neural network and the convolution blocks <NUM> corresponding to the second neural network. The blocks contain convolutions, batch normalizations as well as activation functions such as the rectified linear unit (ReLU) or variants thereof. <NUM> and <NUM> further illustrate the typical residual block addition and skip connections, respectively.

<FIG> shows a more detailed view of the first neural network <NUM> and <FIG> shows the second neural network <NUM> comprising the layer composition. The networks <NUM> and <NUM> comprise feature extraction 2D residual and 3D convolutional blocks. Each residual block comprises a spatial convolution (CONV) <NUM> x <NUM>, batch normalization, BN, and Leaky Rectified Linear Units, LReLU, activation layers. The 3D block comprises double spatial and temporal CONV (<NUM>*<NUM>*<NUM>) -BN-LReLU operations. The architecture of Network <NUM> is based on the 2D Unet with <NUM> encoder-decoder blocks. In network <NUM> for down sampling asymmetrical max-pooling layers were applied into temporal and spatial dimensions.

<FIG> shows a schematic view of a heat map or probability distribution as generated by the two networks. In the heat map or probability distribution <NUM>, two landmarks such as the landmarks <NUM> and <NUM> are generated.

In <FIG> the different extracted parameters are shown as predicted by the networks (predicted curve) versus the ground truth data (GT), wherein the time resolved motion curves are extracted from the neural networks and the following parameters are calculated:.

Analysis: Network accuracy was evaluated by the root mean square difference between ground truth and detected landmarks as well as by a Bland-Altmann analysis (<FIG>) on extracted motion parameters. On <NUM> unlabeled datasets acquired on <NUM>. 5T systems, successful inference was assessed by detecting unambiguous outliers. Every tracked series whose plane displacement is not temporally smooth (mean standard deviation) at any cardiac phase is discarded. Finally, motion parameters were extracted from this data (<FIG>).

One of the curve shows the ground truth data wherein the other curve shows the output result as calculated based on the output from the neural networks.

Furthermore, the <NUM> networks were used for unlabeled images as shown in <FIG> wherein <FIG> show extracted parameters from <NUM> unlabeled data sets in which data are shown for the two and four-chamber view respectively. The bars represent the standard deviation over datasets in each plot.

<FIG> shows a Bland-Altmann analysis. Landmark coordinate mean errors of <NUM> ± <NUM> (2CHV) and <NUM> ± <NUM> (4CHV) were achieved as compared to ground truth, manually annotated datasets.

The Bland-Altmann analysis of <FIG> revealed the following mean agreement values: MVAPD-PD: <NUM> ± <NUM>, MVAPV-e': <NUM> ± <NUM>/s,, VAD: <NUM><NUM><NUM>, MAVL-e': <NUM> ± <NUM>/s, MAMD: <NUM> ± <NUM>, MADC: <NUM> ± <NUM>.

The localization network fails to locate the ROI in less than <NUM>% of unlabeled datasets and at least one time-frame was not smoothly tracked in <NUM>,<NUM>%.

From the above said some general conclusions can be drawn.

First of all a sequence of cardiac MR images is determined which shows the time resolved motion of the heart. Then a subset of images is applied to the first trained convolutional neural network <NUM>. The subset of the sequence of cardiac MR images can comprise a single MR image of the sequence of cardiac MR images, preferably a first timeframe in the sequence of cardiac MR images and the first trained convolutional neural network is a 2D convolutional neural network.

Preferably, the second trained convolutional neural network is a 3D network which is able to process the time series of MR images.

Different motion parameters can be determined based on the identified landmarks, such as the plane displacement of the mitral valve annulus, the plane velocity of the mitral valve annulus, the total motion of the annulus, the septal or lateral velocity of the mitral vavle annulus, the evolution of the diameter of the mitral valve annulus, a mitral annular tissue velocity, a time resolved artioventricular plane velocity or the end systolic long axis mitral annular diameter.

Furthermore, a pixel space interpolation is applied to the images before the reframed and aligned sequence is applied to the second trained convolutional neural network.

Furthermore, it is possible to carry out a fitting, for each of the at least two landmarks in which a defined probability distribution such as Gaussian distribution is fitted to the further probability distribution as output by the second network, wherein the maximum of the fitted distribution is used as the final position of the at least two landmarks, which is used to determine the motion parameter.

The method may be repeated with two sequences of MR images having different slice orientations so that at least four anatomical landmarks are obtained and the motion parameter is determined based on the at least four anatomical landmarks. By way of example when the image plane is rotated by <NUM>°, four different landmarks on the annulus may be obtained.

The two convolutional neural networks <NUM> and <NUM> may be both trained with the same training data in which the at least two landmarks were indicated as ground truths.

The two convolutional neural networks may be both trained based on a heat map regression.

The first neural network <NUM> can comprise a residual Unet and the second network can comprise a residual Unet.

Claim 1:
A computer-implemented method for determining a motion parameter of a heart, the method comprising:
- determining a sequence of cardiac MR images (<NUM>,<NUM>) showing a time resolved motion of the heart,
- applying a subset of the sequence of cardiac MR images as a first input to a first trained convolutional neural network (<NUM>) configured to determine, as first output a probability distribution of at least <NUM> anatomical landmarks in the subset,
- cropping and aligning the sequence of cardiac MR images based on the at least <NUM> anatomical landmarks, in order to determine a reframed and aligned sequence of new cardiac MR images (<NUM>), wherein all new images of the reframed and aligned sequence show the same orientation of the heart,
- applying the reframed and aligned sequence of new cardiac MR images (<NUM>) to a second trained convolutional neural network (<NUM>) configured to determine, as second output, a further probability distribution of the at least <NUM> anatomical landmarks in each new MR image of the reframed and aligned sequence,
- determining the motion parameter of the heart based on the second output.