# An Empirical Study on Ensemble of Segmentation Approaches
## 3 Results
### Experiments
#### 3.3.2 Ablation Studies  
The first ablation study is related to the evaluation of different loss functions to increase the diversity of the models and improve the performance of the ensemble. In Table 3, the performance of RN101, with the different loss functions here proposed, is reported and compared with the dice loss as the baseline and DA1 as the data augmentation method. For the sake of space, for the polyp and skin datasets we report only the average performance value among the test sets. Then, the stand-alone networks are fused, always using the sum rule, in some ensembles:  
* _ELoss101(10)_ is an ensemble, combined by the sum rule, of 10 RN101, each coupled with data augmentation DA1 and a given loss function. The final fusion is given by: \(2\times\ L_{\text{GD}}+2\times\ L_{\text{T}}+2\times\text{Comb1}+2\times\text{Comb2 }+2\times\text{Comb3}\); where with \(2\times\ L_{\text{GD}}\) we mean two RN101 trained using LGD loss function;
* _ELossMix(10)_ is an ensemble similar to the previous one, but here data augmentation is used to increase diversity: the networks coupled with the loss used in _ELoss101(10)_ (LGD, LT, Comb1, Comb2, Comb3) are trained one time using DA1 and another time using DA2 (i.e., 5 networks each trained two times, so we have an ensemble of 10 networks);
* _ELossMix2(10)_, it is similar to the previous ensemble, but using LDiceBES instead of LT
* _ELossLarge(10)_ is an ensemble of 10 networks: the networks trained using (LGD, LT, Comb1, Comb2, Comb3) use DA2 as the augmented training set, while the networks trained considering the new loss functions tested in this work (LSTR, LBoundExps, LDiceBES, LMS, LCS) are coupled with DA1. The results reported in Table 3 show that the proposed new loss functions gain performance similar to dice and can be considered a useful starting point for the design of an ensemble. In fact, the good performances of _ELoss101_ and _ELossLarge_ with respect to _ERN101(10)_ prove that the including networks trained by different loss functions are useful for the creation of an ensemble: this observation is even more evident considering that it is validated on very different problems. We should not overlook the value of altering the training set, in this case through the use of different data augmentation: this strategy seems to be the winner when combined with different loss functions (_ELossMix_). _ELossMix2_ seems to be more stable than _ELossMix_. Finally, we can notice that _ELossMix_ and _ELossLarge_ obtain a similar performance. \begin{table}
\begin{tabular}{c c c c c c c} \hline  & **POLYP** & **SKIN** & **LEUKO** & **BFLY** & **EMICRO** & _AVG_ \\ \hline _RN18_ & 0.806 & 0.865 & 0.897 & 0.960 & 0.908 & 0.887 \\ _RN50_ & 0.802 & 0.871 & 0.895 & 0.968 & 0.909 & 0.889 \\ _RN101_ & 0.808 & 0.871 & 0.915 & 0.976 & 0.918 & 0.898 \\ _ERN18(10)_ & 0.821 & 0.866 & 0.913 & 0.963 & 0.913 & 0.895 \\ _ERN50(10)_ & 0.807 & 0.872 & 0.897 & 0.969 & 0.918 & 0.893 \\ _ERN101(10)_ & **0.834** & **0.878** & **0.925** & **0.978** & **0.919** & **0.907** \\ \hline \end{tabular}
\end{table} 
Table 2: Performance (dice) of the proposed ensembles in the five benchmark datasets; the last column _AVG_ reports the average performance. Best performance is marked in bold font. The second ablation study is related to the evaluation of different architectures. The performance of the above cited methods coupled with different data augmentation strategies are reported in Table 4. The names DA1 and DA2 refer to the strategies explained in Section 2, while DA1/2 denotes that the given ensemble is obtained by the fusion of networks based on DA1 and networks based on DA2. HardNet-MSEG is trained with two different optimizers (SGD denoted as H_S and Adam denoted as H_A), and the ensemble FH is the fusion of HarDNet-MSEG trained with different optimizers. PVT is trained using the AdamW optimizer (as suggested in the original paper where PVT has been proposed). The loss functions for both HarDNet-MSEG and PVT are the same as the original papers.