# An Empirical Study on Ensemble of Segmentation Approaches
## 3 Results
### Experiments
#### 3.3.3 Comparison with the Literature
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Kvasir**} & \multicolumn{2}{c}{**ClinicaDB**} & \multicolumn{2}{c}{**ColonDB**} & \multicolumn{2}{c}{**ETIS**} & \multicolumn{2}{c}{**CVC-T**} & _Average_ \\ \cline{2-11}  & **IoU** & **Dice** & **IoU** & **Dice** & **IoU** & **Dice** & **IoU** & **Dice** & **IoU** & **Dice** & **IoU** & **Dice** \\ \hline _FH_(2) + 2 \(\times\)_PVT_(2) & **0.874** & **0.920** & 0.894 & 0.937 & **0.751** & **0.826** & **0.717** & **0.787** & 0.842 & 0.904 & **0.816** & **0.875** \\ Ensemble in [24] & 0.871 & 0.917 & 0.886 & 0.931 & 0.697 & 0.769 & 0.663 & 0.740 & 0.829 & 0.901 & 0.790 & 0.852 \\ HarDNet-MSEG [6] & 0.857 & 0.912 & 0.882 & 0.932 & 0.66 & 0.731 & 0.613 & 0.677 & 0.821 & 0.887 & 0.767 & 0.828 \\ PraNet (from [6]) & 0.84 & 0.898 & 0.849 & 0.899 & 0.64 & 0.709 & 0.567 & 0.628 & 0.797 & 0.871 & 0.739 & 0.801 \\ SFA(from [6]) & 0.611 & 0.723 & 0.607 & 0.700 & 0.347 & 0.469 & 0.217 & 0.297 & 0.329 & 0.467 & 0.422 & 0.531 \\ U-Net++ (from [6]) & 0.743 & 0.821 & 0.729 & 0.794 & 0.41 & 0.483 & 0.344 & 0.401 & 0.624 & 0.707 & 0.570 & 0.641 \\ U-Net (from [6]) & 0.746 & 0.818 & 0.755 & 0.823 & 0.444 & 0.512 & 0.335 & 0.398 & 0.627 & 0.710 & 0.581 & 0.652 \\ SETR [21] & 0.854 & 0.911 & 0.885 & 0.934 & 0.69 & 0.773 & 0.646 & 0.726 & 0.814 & 0.889 & 0.778 & 0.847 \\ TransUnet [54] & 0.857 & 0.913 & 0.887 & 0.935 & 0.699 & 0.781 & 0.66 & 0.731 & 0.824 & 0.893 & 0.785 & 0.851 \\ TransFuse [9] & 0.870 & **0.920** & **0.897** & **0.942** & 0.706 & 0.781 & 0.663 & 0.737 & 0.826 & 0.894 & 0.792 & 0.855 \\ UACANet [10] & 0.859 & 0.912 & 0.88 & 0.926 & 0.678 & 0.751 & 0.678 & 0.751 & **0.849** & **0.910** & 0.789 & 0.850 \\ SANet [55] & 0.847 & 0.904 & 0.859 & 0.916 & 0.670 & 0.753 & 0.654 & 0.750 & 0.815 & 0.888 & 0.769 & 0.842 \\ MSNet [56] & 0.862 & 0.907 & 0.879 & 0.921 & 0.678 & 0.755 & 0.664 & 0.719 & 0.807 & 0.869 & 0.778 & 0.834 \\ PVT [7] & 0.864 & 0.917 & 0.889 & 0.937 & 0.727 & 0.808 & 0.706 & **0.787** & 0.833 & 0.900 & 0.804 & 0.869 \\ SwinE-Net [57] & 0.870 & **0.920** & 0.892 & 0.938 & 0.725 & 0.804 & 0.687 & 0.758 & 0.842 & 0.906 & 0.803 & 0.865 \\ AMNet [58] & 0.865 & 0.912 & 0.888 & 0.936 & 0.690 & 0.762 & 0.679 & 0.756 & - & - & - & - \\ \hline \hline \end{tabular}   
In the BFLY, many approaches have been tested (see [59]), and the two best methods reported in the literature are:  
1. Ref. [59] that reports an IoU of 0.950;
2. Ref. [60] that reports an IoU of 0.945. Our suggested ensemble (i.e., \(FH(2)+2\times PVT(2)\)) strongly outperforms previous state-of-the-art methods obtaining an IoU of 0.970. Clearly, the ensemble boosts the performance of the best stand-alone network (PVT combined with DA2). The main cons of this approach are that the best ensemble is composed by six networks, and this means 6\(\times\) RAM requirements and 6\(\times\) inference time. Furthermore, the inference time is very low also using an ensemble, and with the current GPU architectures it is not a problem in many issues (obviously it could be a problem in some applications such as autonomous drive, but not in the segmentation problems faced in this paper). E.g., a single HarDNet-MSEG runs at 86.7 images/second on a GeForce RTX 2080 Ti GPU. We have performed a further experiment to select the optimal set of models to be included in the final ensemble. We have extracted a validation set to select the best set of networks: we have considered only the two problems including many test sets, i.e., polyp and skin segmentation. In the polyp problem, the Kvasir test set has been chosen as a validation set; in the skin application problem, the ECU test set has been used as validation set. We have used a sequential forward floating selection (SFFS) [61] for retaining the subset of networks that maximize the dice performance indicator in the validation set. The performance of both ensembles were lower than we expected and in both datasets our best approach (i.e., \(FH(2)+2\times PVT(2)\)) gained a higher performance.