## 2 Evolution of CMR Segmentation Performance (2009-2021)  
We motivate the need to shift from a focus on accuracy, as the main performance criterion, towards other criteria, i.e., reliability and robustness, by studying the evolution of CMR segmentation methods' accuracy over approximately a decade. To this end, we focus on fully-automated cardiac segmentation methods from short-axis (SA) CMR acquisitions. SA CMR segmentation has been widely studied, thanks to the large number of labelled SA CMR datasets available through multiple segmentation challenges and within the UK Biobank [12], a large-scale biomedical database containing in-depth genetic and health information from half a million participants. We analyze the performance of 50 CMR segmentation methods, published since 2009, the year where the Sunnybrook Cardiac MR Left Ventricle Segmentation Challenge ([[https://www.cardiaclas.org/studies/sunnybrook-cardiac-data/](https://www.cardiaclas.org/studies/sunnybrook-cardiac-data/)]([https://www.cardiaclas.org/studies/sunnybrook-cardiac-data/](https://www.cardiaclas.org/studies/sunnybrook-cardiac-data/)), accessed on 7 April 2022). took place. This challenge is the first ever reported CMR segmentation challenge. A large number of the here-reported works were developed in the context of this and four other CMR segmentation challenges. In chronological order, these are: the LV Segmentation Challenge ([[http://www.cardiac.org/challenges/lv-segmentation-challenge](http://www.cardiac.org/challenges/lv-segmentation-challenge)]([http://www.cardiac.org/challenges/lv-segmentation-challenge](http://www.cardiac.org/challenges/lv-segmentation-challenge)), accessed on 7 April 2022) in 2011 [13], the Right Ventricle (RV) Segmentation Challenge ([[https://rvsc.projets.litislab.fr](https://rvsc.projets.litislab.fr)]([https://rvsc.projets.litislab.fr](https://rvsc.projets.litislab.fr)), accessed on 7 April 2022) in 2012 [14], the Automated Cardiac Diagnosis Challenge ([[https://www.creatis.insa-lyon.fr/Challenge/acdc](https://www.creatis.insa-lyon.fr/Challenge/acdc)]([https://www.creatis.insa-lyon.fr/Challenge/acdc](https://www.creatis.insa-lyon.fr/Challenge/acdc)), accessed on 7 April 2022) in 2017 [11] (ACDC), and the Multi-Centre, Multi-Vendor & Multi-Disease Cardiac Image Segmentation Challenge ([[https://www.ub.edu/mms](https://www.ub.edu/mms)]([https://www.ub.edu/mms](https://www.ub.edu/mms)), accessed on 7 April 2022) in 2020 [15] (M&Ms). Table 1 presents the SA CMR segmentation methods considered in our study and specifies the cardiac structures each method extracts, i.e., the left ventricle (LV), the right ventricle (RV) and left ventricular myocardium (MYO). Figure 1 presents SA CMR segmentation methods' progress in performance measured with the Dice Score Coefficient (DSC). The methods are discriminated per segmented cardiac structure (LV, RV and MYO). Furthermore, we differentiate between DL-based (blue) and non-DL methods (orange). We observe that, up to 2015, methods were exclusively not DL-based, mostly focused on LV segmentation, and with an important performance gap between the LV and the RV and MYO. The latter may be explained by the LV's relatively lower variability in shape than the other cardiac structures. In 2015, in the context of the Kaggle Second Annual Data Science Bowl ([[https://www.kaggle.com/c/second-annual-data-science-bowl](https://www.kaggle.com/c/second-annual-data-science-bowl)]([https://www.kaggle.com/c/second-annual-data-science-bowl](https://www.kaggle.com/c/second-annual-data-science-bowl)), accessed on 7 April 2022), the top-performing methods relied on deep learning technologies ([[https://github.com/wosbialex/diagnose-heart](https://github.com/wosbialex/diagnose-heart)]([https://github.com/wosbialex/diagnose-heart](https://github.com/wosbialex/diagnose-heart)), accessed on 7 April 2022). After this milestone, the scientific community shifted quickly towards DL. After 2016, only one non-DL CMR segmentation method [19] has been reported.