## 2 Evolution of CMR Segmentation Performance (2009-2021)
An immediate consequence of this change of techniques is the jump in performance for all cardiac structures. This is more evident for MYO and RV, which had the lowest DSCs, improving from average DSCs of 0.71 and 0.64, respectively before 2015, to both achieving \begin{table}
\begin{tabular}{c l l l l l} \hline \hline
**No.** & **Ref.** & **Challenge** & **No.** & **Ref.** & **Challenge** \\ \hline
1 & [PERSON] et al. [16] & LV & 25 & [PERSON] et al. [17] & ALL \\
2 & [PERSON] et al. [18] & LV & 26 & [PERSON] and [PERSON] [19] & ALL \\
3 & [PERSON] et al. [20] & LV & 27 & [PERSON] et al. [21] & MYO \\
4 & [PERSON] et al. [22] & RV & 28 & [PERSON] et al. [23] & ALL \\
5 & [PERSON] et al. [24] & MYO & 29 & [PERSON] et al. [25] & ALL \\
6 & [PERSON] et al. [26] & MYO & 30 & [PERSON] et al. [27] & ALL \\
7 & [PERSON] et al. [28] & LV & 31 & [PERSON] et al. [29] & ALL \\
8 & [PERSON] et al. [30] & RV & 32 & [PERSON] and [PERSON] [31] & ALL \\
9 & [PERSON] et al. [32] & LV & 33 & [PERSON] et al. [33] & ALL \\
10 & [PERSON] et al. [34] & LV & 34 & [PERSON] et al. [35] & ALL \\
11 & [PERSON] et al. [36] & RV & 35 & [PERSON] et al. [37] & ALL \\
12 & [PERSON] and [PERSON] [38] & LV & 36 & [PERSON] et al. [39] & ALL \\
13 & [PERSON] et al. [40] & LV & 37 & [PERSON] et al. [41] & ALL \\
14 & [PERSON] et al. [42] & LV & 38 & [PERSON] and Tziritas [43] & ALL \\
15 & [PERSON] et al. [44] & LV & 39 & [PERSON] et al. [45] & ALL \\
16 & [PERSON] [46] & ALL & 40 & [PERSON] [47] & ALL \\
17 & [PERSON] et al. [48] & LV & 43 & [PERSON] et al. [49] & ALL \\
18 & [PERSON] et al. [50] & ALL & 42 & [PERSON] et al. [51] & ALL \\
19 & [PERSON] et al. [52] & MYO & 43 & [PERSON] et al. [53] & ALL \\
20 & [PERSON] et al. [54] & ALL & 44 & [PERSON] et al. [55] & ALL \\
21 & [PERSON] et al. [56] & ALL & 45 & [PERSON] and [PERSON] [57] & ALL \\
22 & [PERSON] et al. [58] & ALL & 46 & [PERSON] et al. [59] & ALL \\
23 & [PERSON] et al. [60] & ALL & 47 & [PERSON] et al. [61] & ALL \\
24 & [PERSON] et al. [6] & ALL & 48 & [PERSON] et al. [62] & ALL \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Fully automated SA CMR segmentation methods published between 2009 and 2021 with the segmented structure of interest (LV, RV or MYO). ALL denotes that a method segments the three cardiac sub-structures. 0.85 after 2015. LV segmentation reports an improvement from 0.88 average DSC to 0.91. Since then, the number of methods has exploded. However, performance improvements have stalled and, in some cases, deteriorated. This is the case of the general performance in the M&Ms Challenge [15], which assessed how well methods could cope with changes in the properties of the input images (e.g., different origins, scanner vendors and protocols). The result was a drop in the performance, as observed from the RV trend line or the very low performing methods (e.g., point 34) in Figure 1.  
Finally, while most DL-based methods in Figure 1 report a very high accuracy, close to the inter-observer variability, [PERSON] et al. [11] demonstrated that DL-based methods, even the best performing ones [25], produced CMR segmentations with implausible anatomical configurations. The authors go then to suggest the adoption of new performance evaluation metrics that are more resilient to abnormalities. In the following, we show that the problems here identified, i.e., performance drops or implausible segmentations, can be addressed by accounting for reliability and robustness.