# Extraction of volumetric indices from echocardiography: which deep learning solution for clinical use?

Hang Jung Ling, Nathan Painchaud, Pierre-Yves Courand, Pierre-Marc Jodoin, Damien Garcia, Olivier Bernard

## ► To cite this version:

Hang Jung Ling, Nathan Painchaud, Pierre-Yves Courand, Pierre-Marc Jodoin, Damien Garcia, et al.. Extraction of volumetric indices from echocardiography: which deep learning solution for clinical use?. Functional Imaging and Modeling of the Heart, Jun 2023, Lyon, France. hal-04087268v2

HAL Id: hal-04087268

https://hal.science/hal-04087268v2

Submitted on 8 May 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Extraction of volumetric indices from echocardiography: which deep learning solution for clinical use?

Hang Jung Ling¹[0000-0003-0475-9121], Nathan

Painchaud¹,²[0000-0001-8269-5704], Pierre-Yves

Courand¹,³,⁴[0000-0003-3199-7977], Pierre-Marc Jodoin²[0000-0002-6038-5753],

Damien Garcia¹[0000-0002-8552-1475], and Olivier Bernard¹[0000-0003-0752-9946]

¹ Univ Lyon, INSA-Lyon, Université Claude Bernard Lyon 1, UJM-Saint Etienne, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621, Lyon, France

² Dept. of Computer Science, University of Sherbrooke, Sherbrooke, QC, Canada

³ Cardiology Dept., Hôpital Croix-Rousse, Hospices Civils de Lyon, Lyon, France

⁴ Cardiology Dept., Hôpital Lyon Sud, Hospices Civils de Lyon, Lyon, France

**Abstract.** Deep learning-based methods have spearheaded the automatic analysis of echocardiographic images, taking advantage of the publication of multiple open access datasets annotated by experts (CAMUS being one of the largest public databases). However, these models are still considered unreliable by clinicians due to unresolved issues concerning i) the temporal consistency of their predictions, and ii) their ability to generalize across datasets. In this context, we propose a comprehensive comparison between the current best performing methods in medical/echocardiographic image segmentation, with a particular focus on temporal consistency and cross-dataset aspects. We introduce a new private dataset, named CARDINAL, of apical two-chamber and apical four-chamber sequences, with reference segmentation over the full cardiac cycle. We show that the proposed 3D nnU-Net outperforms alternative 2D and recurrent segmentation methods. We also report that the best models trained on CARDINAL, when tested on CAMUS without any fine-tuning, still manage to perform competitively with respect to prior methods. Overall, the experimental results suggest that with sufficient training data, 3D nnU-Net could become the first automated tool to finally meet the standards of an everyday clinical device.

**Keywords:** Ultrasound · cardiac segmentation · temporal segmentation · deep learning · CNN.

# 1 Introduction

Echocardiographic imaging has undergone major advances in recent years thanks to artificial intelligence, especially the deep learning (DL) paradigm. In particular, the automated extraction of clinical indices from the segmentation of cardiac structures has been the subject of intense research leading to major breakthroughs. A key component of these advances has been the publication
of open access annotated datasets, including CETUS (45 patients, 3D images annotated at End-Diastole - ED and End-Systole - ES) [1], CAMUS (500 patients, 2D images annotated at ED and ES in apical two-chamber - A2C - and apical four-chamber - A4C - views) [5], EchoNet-Dynamic (10,036 patients, 2D sub-sampled images annotated at ED and ES in A4C view) [7], HMC-QU (109 patients, 2D sequences annotated in A4C view) [2] and TED (98 patients from the CAMUS dataset, 2D sequences annotated in A4C view) [8].

These datasets allowed effective and fair comparisons of methods, whether they are generic image segmentation models [5,6] or were specifically designed to process echocardiographic images [11,9]. Thus, the performance of current state-of-the-art (SOTA) methods on the CAMUS dataset confirmed the dominance of the DL-based methods, which finally achieved inter- and intra-observer variability for most of the geometric (Dice score, Hausdorff distance, mean absolute distance - MAD) and clinical metrics (ejection fraction - EF, volumes at ED/ES).

Although these results are extremely promising and represent a crucial step towards the automation of echocardiographic image analysis, they are not sufficient to justify confidence in fully automated methods in a clinical context. Indeed, two crucial challenges on the path to the practical application of these algorithms remain understudied in the field: i) the frame-by-frame temporal consistency of the predictions, and ii) the generalization of the methods across datasets. Based on this observation, we propose the following contributions:

1. We study the performance of two generic architectures based on common temporal data processing techniques on 2D echocardiography sequences, and compare them to current SOTAs in the same field;

2. We present a new private dataset called CARDINAL (240 patients, 2D sequences annotated in A4C and A2C views), and report the performance impact of training our methods exclusively on CARDINAL and testing on CAMUS.

# 2 Benchmarked Methods

CAMUS is currently the only dataset where an evaluation platform has been established to effectively compare the performance of segmentation methods<fnref n="5" />. We therefore relied on this dataset to select the methods we retained in this study.

## 2.1 2D DL Methods

The currently best performing method on the CAMUS dataset exploits the nnU-Net formalism [6]. This model is based on the U-Net architecture and implements several successful DL tricks, such as a patch-wise approach to preserve image resolution, data augmentation during both training and inference to enforce

<footnote n="5">⁵ https://www.creatis.insa-lyon.fr/Challenge/camus/results.html</footnote>
generalization and automatic hyperparameter search of the U-Net architecture to increase accuracy. Note that the 2D version uses only one U-Net model [4].

Recently, Sfakianakis et al. [9] developed a DL solution called GUDU based on three key aspects. First, they proposed to use data augmentations tailored to ultrasound acquisition, i.e. variation of the contrast between the myocardial tissue and the left ventricular (LV) cavity, random rotation from the origin of the sectorial shape to mimic different probe positioning, and perspective transformations to simulate probe twisting. Inspired by ensemble models, the authors also trained 5 U-Nets with different architectures and averaged their outputs during inference in order to compute the final prediction. Finally, a new loss function was proposed that takes into account the relative position of the cardiac structures with respect to each other. The authors demonstrated the usefulness and complementarity of each contribution in an ablation study.

## 2.2 2D+t DL Methods

Despite the success of 2D methods in producing accurate segmentations for individual echocardiographic frames, they often fail to maintain temporal consistency between frames [8]. Prior to the publication of HMC-QU and TED, there were no publicly available datasets to train and compare methods that incorporate the temporal dimension in 2D+time echocardiography. This explains why so few papers have focused on this topic.

Due to the lack of 2D+time annotated datasets, Wei et al. [11] proposed a method to leverage the limited ED/ES annotated frames and propagate them to unannotated frames. This was achieved by training a 3D U-Net designed to predict both the deformation fields between each pair of consecutive frames and the segmentation masks at each frame of the sequence. The deformation fields are used to propagate the ED/ES reference annotations forward and backward in time through the sequence. The corresponding propagated masks are then used as targets for self-supervised segmentation of the entire sequence. This encourages the model to learn consistent temporal dynamics to find the best match between the predicted segmentation masks and the propagated annotations.

More recently, Smistad et al. [10] and Hu et al. [3] added convolutional long-term memory blocks to each layer of the encoder of a 2D U-Net (this type of model is hereafter referred to as U-Net LSTM). Thus, instead of processing a single frame, these methods take a series of frames as input and store the extracted features over time to produce the final segmentation of the entire sequence. Results show that such a strategy tends to reduce segmentation shifts from one frame to another.

By their very nature, echocardiographic sequences exhibit regular properties along the time axis. Therefore, it seems logical to consider 2D ultrasound sequences as complete volumes containing coherent 3D shapes and to extract 3D features using 3D convolutional layers to promote temporal consistency. Thus, in this paper, we propose to train a 3D nnU-Net to segment the complete cardiac sequences in a single run. We hypothesize that this model will inherently learn temporal consistency while maintaining a high level of segmentation accuracy.
# 3 Experimental Setup

## 3.1 CARDINAL Dataset

**Acquisition Protocol:** The proposed dataset consists of clinical examinations of 240 patients, acquired at the University Hospital of Lyon (Croix-Rousse Lyon Sud, France) under the regulation of the local ethics committee of the hospital. The complete dataset was acquired with GE ultrasound scanners. For each patient, 2D A4C and A2C view sequences were exported from the EchoPAC analysis software. Each exported sequence corresponds to a set of B-mode images expressed in polar coordinates. The same interpolation procedure as used for the CAMUS dataset was applied to express all sequences in Cartesian coordinates with a single grid resolution of 0.31 mm². Each sequence in the CARDINAL dataset corresponds to a complete cardiac cycle defined as the interval between peaks of maximal LV cavity surface area.

**Reference Annotations:** To tackle the total number of frames to be annotated, an experienced observer first delineated the different contours using semi-automatic tools to ensure temporal consistency of the segmented shapes. Each corresponding output was then checked/corrected by two other experienced observers. To identify the ED/ES frames in the sequence, the ED frames correspond (by definition) to the beginning and end of each sequence, and the ES frame corresponds to the frame where the LV cavity surface is smallest.

## 3.2 Implemented DL Methods

For a fair comparison, we implemented the 2D nnU-Net, U-Net LSTM, and 3D nnU-Net described in section 2 using the same Python library called ASCENT<fnref n="6" />. These models shared the following training hyperparameters: batch size of 2, SGD optimizer with a learning rate of 0.01 coupled with a polynomial decay scheduler, and 1000 training epochs. The 2D and 3D nnU-Net used a patch-wise approach to avoid resizing the images, thus preserving the native image resolution. To train the U-Net LSTM, the input images were resized to 256 x 256 and 24 consecutive frames were randomly selected and fed to the model to produce the corresponding segmentations. For inference, the sliding window approach with a Gaussian importance map was used. The prediction was given by the average of the softmax probabilities of all windows. To improve segmentation accuracy, the final prediction was obtained by averaging the predictions of the original and mirrored images along different axes. More implementation details for each model can be found in table 1.

# 4 Results

We evaluate the methods described in section 2 using three types of measures to get a complete picture of their performance in terms of segmentation accuracy

<footnote n="6">⁶ https://github.com/creatis-myriad/ASCENT</footnote>
<table><caption>Table 1: Details of the implementation of the three methods evaluated in this study. *Lowest resolution*: Size of the lowest resolution of feature maps in pixels. *Optimization scheme*: Optimizer + initial learning rate + learning rate scheduler used. *Training duration (hours)*: number of hours required to train each model for 1000 epochs. The configurations shared between models are only shown once in their respective rows.</caption><thead><tr><th>Configurations</th><th>3D nnU-Net</th><th>2D nnU-Net</th><th>U-Net LSTM</th></tr></thead><tbody><tr><td>Patch size (pixels)</td><td>320 × 256 × 24</td><td>640 × 512</td><td>256 × 256 × 24</td></tr><tr><td>Batch size</td><td></td><td>2</td><td></td></tr><tr><td>Nb. feature maps</td><td></td><td>32 ↓ 480 ↑ 32</td><td></td></tr><tr><td>Lowest resolution</td><td>10 × 8 × 6</td><td>5 × 4</td><td>8 × 8 × 24</td></tr><tr><td>Downsampling scheme</td><td colspan="3">Stride pooling</td></tr><tr><td>Upsampling scheme</td><td colspan="3">Deconvolution</td></tr><tr><td>Normalization scheme</td><td colspan="3">Instance normalization</td></tr><tr><td>Optimization scheme</td><td colspan="3">SGD + 0.01 + polynomial decay</td></tr><tr><td>Loss function</td><td colspan="3">Cross entropy + Dice</td></tr><tr><td>Number of parameters</td><td>41.3 M</td><td>30.4 M</td><td>49 M</td></tr><tr><td>Training duration (hours)</td><td>22.8</td><td>8</td><td>69.5</td></tr></tbody></table>

(table 2), extraction of clinical indices (table 3) and temporal consistency (table 4). In each of these tables, we group the methods according to the datasets on which they were trained and tested (CARDINAL is abbreviated as CL and CAMUS is abbreviated as CS) to make it easier to observe the change in performance when generalizing to a new dataset.

## 4.1 Geometric and Clinical Accuracy

Table 2 shows the segmentation accuracy computed from the CARDINAL and CAMUS datasets for the 5 algorithms described in section 2. The values in bold correspond to the best scores for each metric for a given training/test dataset setup. From the results on the CARDINAL dataset (CL/CL case), we can see that the 3D nnU-Net has the best segmentation scores for all metrics, for both ED and ES. It is also interesting to note that the two temporal consistency methods (3D nnU-Net and U-Net LSTM) produce better results than the 2D nnU-Net method. This can be explained by the fact that the reference segmentation has regular properties along the temporal axis due to the annotation process. Methods that integrate the temporal dimension into their architecture are therefore more likely to produce segmentation results that are closer to the manual references.

It is worth mentioning that methods trained and tested on the same dataset (sections CL/CL and CS/CS in Table 2) get overall better results up to 1.7x for the Hausdorff and MAD metrics. One reason for such an improvement is the larger amount of annotated training images for CARDINAL (18,793 im-
ages from 190 training/validation patients, reference frames for the full cardiac cycle in A2C and A4C views) than for CAMUS (1,800 images from 450 training/validation patients, reference frames at ED and ES in A2C and A4C views).

Table 3 reports the clinical metrics for the 5 methods. As in Table 2, the methods enforcing temporal consistency gets the best results on CARDINAL, especially for the ejection fraction for which temporal consistency is essential (mean correlation score of 0.917). Furthermore, the best models trained on CARDINAL or CAMUS produce similar results for volume estimation (average correlation of 0.978), revealing a limit reached by these approaches, certainly due to the resolution of the imaging systems.

<caption>Table 2: LV segmentation accuracy of the benchmarked methods, on different subsets of frames. The columns All, ED and ES indicate results averaged over all frames, only ED frames, and only ES frames, respectively. Since CAMUS only provides annotation for ED/ES frames, results over all frames are not available when testing on it.</caption>

(CL:CARDINAL, CS:CAMUS)

<table><thead><tr><th rowspan="2">Methods</th><th rowspan="2">Train/test</th><th colspan="3">Dice</th><th colspan="3">Hausdorff (mm)</th><th colspan="3">MAD (mm)</th></tr><tr><th>All</th><th>ED</th><th>ES</th><th>All</th><th>ED</th><th>ES</th><th>All</th><th>ED</th><th>ES</th></tr></thead><tbody><tr><td>3D nnU-Net</td><td></td><td><strong>.969</strong></td><td><strong>.968</strong></td><td><strong>.960</strong></td><td><strong>2.3</strong></td><td><strong>2.7</strong></td><td><strong>2.5</strong></td><td><strong>0.7</strong></td><td><strong>0.8</strong></td><td><strong>0.7</strong></td></tr><tr><td>2D nnU-Net</td><td>CL/CL</td><td>.957</td><td>.961</td><td>.942</td><td>2.9</td><td>3.1</td><td>3.1</td><td>0.9</td><td>1.0</td><td>1.1</td></tr><tr><td>U-Net LSTM</td><td></td><td>.964</td><td>.964</td><td>.956</td><td>2.5</td><td>2.8</td><td>2.6</td><td>0.8</td><td>0.9</td><td>0.8</td></tr><tr><td>3D nnU-Net</td><td></td><td>-</td><td><strong>.939</strong></td><td><strong>.926</strong></td><td>-</td><td>5.2</td><td><strong>4.6</strong></td><td>-</td><td><strong>1.6</strong></td><td><strong>1.5</strong></td></tr><tr><td>2D nnU-Net</td><td>CL/CS</td><td>-</td><td>.934</td><td>.921</td><td>-</td><td><strong>4.9</strong></td><td>4.6</td><td>-</td><td>1.8</td><td>1.6</td></tr><tr><td>U-Net LSTM</td><td></td><td>-</td><td>.925</td><td>.903</td><td>-</td><td>6.0</td><td>5.8</td><td>-</td><td>2.1</td><td>2.1</td></tr><tr><td>2D nnU-Net</td><td></td><td>-</td><td><strong>.952</strong></td><td><strong>.935</strong></td><td>-</td><td><strong>4.3</strong></td><td><strong>4.2</strong></td><td>-</td><td><strong>1.3</strong></td><td><strong>1.3</strong></td></tr><tr><td>CLAS</td><td>CS/CS</td><td>-</td><td>.947</td><td>.929</td><td>-</td><td>4.6</td><td>4.6</td><td>-</td><td>1.4</td><td>1.4</td></tr><tr><td>GUDU</td><td></td><td>-</td><td>.946</td><td>.929</td><td>-</td><td>4.7</td><td>4.7</td><td>-</td><td>1.4</td><td>1.4</td></tr></tbody></table>

## 4.2 Integration of Temporal Consistency

Table 4 allows a better investigation of the temporal performance of the methods by providing additional information on the number/percentage of frames considered temporally inconsistent w.r.t. their neighboring frames. As expected, the methods incorporating temporal persistence produced fewer temporal errors. Looking at the number of sequences with at least one temporally inconsistent frame, the 3D nnU-Net clearly outperforms U-Net LSTM, with only 4 inconsistent sequences over 100 compared to 98 sequences for U-Net LSTM. This result illustrates the greater ability of features computed from 3D convolutional layers to extract relevant spatio-temporal information. The few remaining temporal errors for the 3D nnU-Net are more an indication that the metrics we
<table><caption>Table 3: Clinical metrics of the benchmarked methods. The ED/ES volumes were computed from both the predicted and reference masks using Simpson's biplane method. *Corr.*: Correlation between the ejection fraction (EF) derived from the predicted/reference segmentation. *MAE*: Mean Absolute Error between the EF derived from predicted/reference segmentation.</caption><thead><tr><th rowspan="2">Methods</th><th rowspan="2">Train/test</th><th colspan="2">EF</th><th colspan="2">Volume ED</th><th colspan="2">Volume ES</th></tr><tr><th>Corr.</th><th>MAE (%)</th><th>Corr.</th><th>MAE (ml)</th><th>Corr.</th><th>MAE (ml)</th></tr></thead><tbody><tr><td>3D nnU-Net</td><td rowspan="3">CL/CL</td><td>.913</td><td>2.9</td><td><strong>.978</strong></td><td><strong>3.3</strong></td><td><strong>.974</strong></td><td><strong>2.7</strong></td></tr><tr><td>2D nnU-Net</td><td>.850</td><td>3.8</td><td>.967</td><td>4.4</td><td>.957</td><td>3.2</td></tr><tr><td>U-Net LSTM</td><td><strong>.922</strong></td><td><strong>2.7</strong></td><td>.973</td><td>3.4</td><td>.969</td><td>2.8</td></tr><tr><td>3D nnU-Net</td><td rowspan="3">CL/CS</td><td><strong>.869</strong></td><td><strong>5.3</strong></td><td><strong>.974</strong></td><td><strong>9.6</strong></td><td><strong>.976</strong></td><td><strong>4.9</strong></td></tr><tr><td>2D nnU-Net</td><td>.810</td><td>7.0</td><td>.970</td><td>12.8</td><td>.959</td><td>6.2</td></tr><tr><td>U-Net LSTM</td><td>.822</td><td>11.1</td><td>.879</td><td>15.9</td><td>.903</td><td>8.2</td></tr><tr><td>2D nnU-Net</td><td rowspan="3">CS/CS</td><td>.857</td><td>4.7</td><td><strong>.977</strong></td><td><strong>5.9</strong></td><td><strong>.987</strong></td><td><strong>4.0</strong></td></tr><tr><td>CLAS</td><td><strong>.926</strong></td><td><strong>4.0</strong></td><td>.958</td><td>7.7</td><td>.979</td><td>4.4</td></tr><tr><td>GUDU</td><td>.897</td><td><strong>4.0</strong></td><td><strong>.977</strong></td><td>6.7</td><td>.981</td><td>4.6</td></tr></tbody></table>

used are (overly) strict on the temporal smoothness. Indeed, the 3D nnU-Net temporal "inconsistencies" appear invisible to the expert eye. As a qualitative evaluation, fig. 1 illustrates in detail the temporal consistency of each of our own method on one patient from the CARDINAL test set. To complement this, we also provide in the supplementary material examples of temporally consistent and inconsistent segmentation results obtained by the 3D nnU-Net method for the CARDINAL and CAMUS datasets.

## 4.3 Generalization Across Datasets

The ability to generalize across datasets is crucially important to gauge the capacity of a method to properly analyze data affected by a distributional shift. To this end, the models trained on CARDINAL were also evaluated on the CAMUS test set without any fine-tuning. The results are reported in the "CL/CS" sections of tables 2 to 4. Among the methods evaluated, 3D nnU-Net is the undisputed best. It even produces competitive geometric and clinical scores compared with SOTA methods trained directly on CAMUS. Thanks to the integration of temporal consistency, the 3D nnU-Net trained on CARDINAL also produces one of the best correlation scores for the EF calculated on the CAMUS dataset, even when compared to SOTA methods trained directly on CAMUS. In view of these results, and considering that the annotation process between the two databases was not identical and was carried out by different experts (which inevitably introduces a bias during the learning phase), the generalization capacity of the 3D nnU-Net model seems remarkable.
<table><caption>Table 4: Temporal consistency of the benchmarked methods, as defined in [8]. Nb of seq. w/ err.: number of sequences (out of the 100 testing sequences) where at least one frame is temporally inconsistent. % of frames w/ err.: percentage of frames that are inconsistent in the sequences with at least one temporally inconsistent frame. Err. to thresh. ratio: average ratio between the measure used to identify temporal inconsistencies and the threshold for temporal inconsistencies. A lower value indicates “smoother” temporal segmentations.</caption><thead><tr><th>Methods</th><th>Train/test</th><th>Nb of seq. w/ err.</th><th>% of frames w/ err.</th><th>Err. to thresh. ratio</th></tr></thead><tbody><tr><td>3D nnU-Net</td><td rowspan="3">CL/CL</td><td>4</td><td>4</td><td>.045</td></tr><tr><td>2D nnU-Net</td><td>100</td><td>30</td><td>.210</td></tr><tr><td>U-Net LSTM</td><td>98</td><td>13</td><td>.110</td></tr><tr><td>3D nnU-Net</td><td rowspan="3">CL/CS</td><td>28</td><td>12</td><td>.095</td></tr><tr><td>2D nnU-Net</td><td>85</td><td>21</td><td>.162</td></tr><tr><td>U-Net LSTM</td><td>83</td><td>16</td><td>.114</td></tr></tbody></table>

## 5 Conclusion

We evaluated the ability of different methods to accurately segment echocardiographic images, with a focus on temporal consistency and cross-dataset generalization. To this end, we introduced a new private database called CARDINAL, with annotations from an expert on the full cardiac cycle for each sequence. The results show that 3D nnU-Net and U-Net LSTM produce the best geometric and clinical scores on the CARDINAL dataset due to the integration of temporal persistence. Regarding the temporal consistency metrics, 3D nnU-Net performed significantly better than U-Net LSTM with only four sequences (instead of 98) out of 100 having at least one image that was temporally inconsistent. As far as cross-dataset generalization is concerned, 3D nnU-Net is also the best performing method. When trained on CARDINAL and tested on CAMUS, it achieved comparable geometric and clinical scores to the best methods both trained and tested on CAMUS. All these results clearly show that 3D nnU-Net is a serious candidate to become the first automated tool to meet the requirements of routine clinical examinations.

## Acknowledgment

This research was funded, in whole or in part, by l'Agence Nationale de la Recherche (ANR), project ANR-22-CE45-0029-01. The authors gratefully acknowledge financial support of the MEGA doctoral school (ED 162), the NSERC Canada Graduate Scholarships-Doctoral Program, the FRQNT Doctoral Scholarships Program, and the LABEX PRIMES (ANR-11-LABX-0063) of Université de Lyon, within the program "Investissements d'Avenir" operated by the French
<figure><img src="image_2.png" /><figcaption>Fig. 1: Visualization of the temporal consistency of the segmentations on one patient from the CARDINAL test set. (left) Frames sampled between ED and ES, with segmentation masks from our own methods + reference. (right) Curves of the LV and myocardium surfaces w.r.t. frame in the sequence.</figcaption></figure>

ANR. The authors also thank GENCI-IDRIS for providing access to HPC resources (Grant 2022-[AD010313603]). For the purpose of open access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript (AAM) version arising from this submission.

## References

1. Bernard, O., Heyde, B., Alessandrini, M., Barbosa, D., Camarasu-Pop, S., Cervenansky, F., Valette, S., Mirea, O., Galli, E., Geleijnse, M.L., Papachristidis, A., Bosch, J.G., d'Hooge, J.: Challenge on Endocardial Three-dimensional Ultrasound Segmentation (CETUS). In: MICCAI CETUS. pp. 1-8 (Sep 2014)

2. Degerli, A., Zabihi, M., Kiranyaz, S., Hamid, T., Mazhar, R., Hamila, R., Gabbouj, M.: Early Detection of Myocardial Infarction in Low-Quality Echocardiography. IEEE Access 9, 34442-34453 (2021)

3. Hu, J., Smistad, E., Salte, I.M., Dalen, H., Lovstakken, L.: Exploiting temporal information in echocardiography for improved image segmentation. In: 2022 IEEE International Ultrasonics Symposium (IUS) (2022)

4. Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203-211 (Feb 2021)

5. Leclerc, S., Smistad, E., Pedrosa, J., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Jodoin, P.M., Grenier, T., Lartizien, C., D'hooge, J., Lovstakken, L., Bernard, O.: Deep Learning for Segmentation Using an Open Large-Scale Dataset in 2D Echocardiography. IEEE Transactions on Medical Imaging 38(9), 2198-2210 (Sep 2019)
6. Ling, H.J., Garcia, D., Bernard, O.: Reaching intra-observer variability in 2-D echocardiographic image segmentation with a simple U-Net architecture. In: 2022 IEEE International Ultrasonics Symposium (IUS) (2022), https://hal.science/ hal-03979523

7. Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heidenreich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A., Zou, J.Y.: Video-based AI for beat-to-beat assessment of cardiac function. Nature 580(7802), 252–256 (Apr 2020)

8. Painchaud, N., Duchateau, N., Bernard, O., Jodoin, P.M.: Echocardiography Segmentation With Enforced Temporal Consistency. IEEE Transactions on Medical Imaging 41(10), 2867–2878 (Oct 2022)

9. Sfakianakis, C., Simantiris, G., Tziritas, G.: GUDU: Geometrically-constrained Ultrasound Data augmentation in U-Net for echocardiography semantic segmentation. Biomedical Signal Processing and Control 82, 104557 (Apr 2023)

10. Smistad, E., Salte, I.M., Dalen, H., Lovstakken, L.: Real-time temporal coherent left ventricle segmentation using convolutional LSTMs. In: 2021 IEEE International Ultrasonics Symposium (IUS) (2021)

11. Wei, H., Cao, H., Cao, Y., Zhou, Y., Xue, W., Ni, D., Li, S.: Temporal-Consistent Segmentation of Echocardiography with Co-learning from Appearance and Shape. In: MICCAI. pp. 623–632 (2020)