# Deep Learning Based Cardiac MRI Segmentation: Do We Need Experts?
## 4 Results and Discussion  
The first set of results are laid out in Table 1. It corresponds to standard geometrical metrics, i.e., the Dice score and the Hausdorff distance (HD) for the left ventricular (LV) cavity (Table 1), the myocardium (MYO) (considering the endocardial and epicardial of the left ventricle) (Table 2) and the cavity of the right ventricle (RV) (Table 3). It also contains the end-diastolic volume (EDV) as well as the ejection fraction (EF) for the LV and the RV and the myocardial mass error. For all three tables, the networks (U-Net, Attention U-Net and ENet) were trained on the ACDC training set and tested on the ACDC testing set and the M&Ms training set. Results for the ACDC testing set reveal that for the LV (Table 1) the networks trained on the non-expert annotations (Non-Expert 1 as well as Non-Expert 2) manage to achieve performance that is statistically indistinguishable from that of the expert. This is true regardless of the training loss (CE+Dice vs MAE+Dice) and the metric (Dice, HD, and EF). The only exception is for the EF error for Non-Expert 1 with loss function MAE+Dice. The situation, however, is more fuzzy for the MYO and the RV. In both cases, we can see that results for the Non-Expert 1 are almost always worse than that of the expert, especially for the CE+Dice loss. For example, there is a Dice score drop of 12% on the myocardium. In addition, the clinical results on the RV (Table 3) show a clear gap between the Non-Expert 1 and the other annotators. However, we can appreciate how the MAE+Dice loss improves results for both non-experts. Overall, for the MYO and the RV, results for the Non-Expert 2 are very close (if not better) than that of the expert. This is obvious when considering the average myocardial mass error in Table 2. Although, one recurrent result  
Figure 1: ACDC Annotation differences between Expert, Non-Expert 1 and Non-Expert 2.  
from our experiments is the hit-and-miss performance of all the evaluated networks on the M&Ms data set, where in a number of cases, the output segmentation is completely degenerated as shown in Figure 2. Moreover, the difference in segmentation performance between all the annotators is similar regardless of the segmentation network (U-Net, Attention U-Net or ENet) used, although Attention U-Net shows the best performance overall, which is to be expected, given its larger capacity. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**U-Net** & & & & & & & \\ \hline \hline
**Test Set** & \begin{tabular}{c} **ACDC** \\ **Trainset** \\ \end{tabular}  & **Loss** & **Avg. DSC** & **Avg. HD** & **Avg. EF (\%)** & **Avg EF Err. ** & **Avg. EDV (mL)** \\ \hline \multirow{3}{*}{ACDC} & **Expert** & CE+Dice & \(0.92\pm 0.08\) & \(11.84\pm 11.43\) & \(47.57\pm 19.62\) & \(3.59\pm 3.17\) & \(175.70\pm 69.03\) \\  & Non-Expert 1 & \(\left\{\begin{array}{ll}\text{CE+Dice}&0.92\pm 0.08\) & \(12.15\pm 10.63\) & \(50.27\pm 20.00\) & \(4.37\pm 3.69\) & \(173.05\pm 67.06\) \\ MAE+Dice & \(0. Further analysis of the segmentation performance on the different sections of the heart (Figure 3), namely the base, the middle and the apex, show that the differences between the non-experts and the expert annotations lie heavily on the two ends of the heart. The performance gap is more pronounced on the apex for the three anatomical structures. In parallel, when we look at the performance from the disease groups (Figure 4), we can distinguish a relative similarity in the Dice score between the different annotators and disease groups. Our experiments also reveal some interesting results on the M&Ms data set, a data set with different acquisition settings than ACDC. In that case, we see that the gap in performance between the expert and the non-expert decreases substantially.