## 2 Materials and Methods
### ReViMS Version 2.0
#### 2.2.2 Averaging Different 2D/3D Segmentations  
One of the major problems when assessing quality performances of segmentations tools is the lack of a robust Ground Truth for the considered dataset. Typically, manual segmentation by an expert annotator can be accounted as \"Ground Truth\". However, the inter- and intra-rater reliability in the segmentation procedure is a well-known problem in several fields, ranging from radiology to histology [20]. Different experts clearly differ in segmenting the same object, and the variability is empathised in case of blurry images such as those typical of fluorescence microscopy.  
In order to obtain a robust Ground Truth, improving and comparing 2D/3D manual segmentations obtained by different experts becomes fundamental, but these are challenging tasks. Practically, today there is not a standard procedure for \"averaging\" different 2D curves and 3D surfaces, even excluding outliers. Logically, the term \"averaging\" is not meant in the strict analytical sense because there is not a standard definition of average between curves.  
In this scenario, the expectation-maximisation algorithm for _Simultaneous Truth and Performance Level Estimation_ (_STAPLE_) is an interesting offer for \"averaging\" 2D curves [21]. The algorithm considers a collection of binary segmentations and computes a probabilistic estimate of the true segmentation and a measure of the performance level represented by each segmentation. In particular, the probabilistic estimate of the true segmentation is formed by estimating an optimal combination of the segmentations, weighting each segmentation on the estimated performance level. Besides proving the quality of their approach on several datasets, the authors also provide an open-source version of the algorithm for several programming languages, including a _MATLAB_ version working with 2D binary segmentations ([[https://bit.ly/2](https://bit.ly/2) Su6 jLJ]([https://bit.ly/2](https://bit.ly/2) Su6 jLJ))).  
In order to \"average\" different 2D curves and 3D surfaces created by different human annotators to obtain a more robust Ground Truth to assess quality performances of automatic segmentation tools, we have included in _ReViMS_ version 2.0 an extended version of _STAPLE_ working with 3D segmentations. Briefly, the 3D \"averaging\" segmentation approach now included in _ReViMS_ can be summarised as follows:  
1. Calculation, for each 3D object \(k_{1}\) segmented by a first expert annotator, the list of the overlapping objects \(k_{j}\) (\(j=2,\ldots,n\); \(n=\) number of annotators) in the segmentations created by the other experts;
2. Selection, among the listed, of the objects \(k_{j}\) with volume spatially overlapping more than 50% with \(k_{1}\);
3. Selection, among the objects \(k_{j}\) with volume spatially overlapping more than 50%, of the 2D \(z\)-sections which overlap more than 50% with the corresponding \(z\)-section of \(k_{1}\);
4. \"Averaging\" of the segmentations exploiting the standard _STAPLE_ algorithm, by considering one-by-one each selected \(z\)-section.  
Source-code and standalone versions of _ReViMS_ version 2.0, including the above described 3D \"averaging\" segmentation algorithm derived from _STAPLE_, are available at the website: [[http://sourceforge.net/p/revims](http://sourceforge.net/p/revims)]([http://sourceforge.net/p/revims](http://sourceforge.net/p/revims)).