## 2 Review of the USOVA3D Public Database and the Evaluation Protocol
### Evaluation Protocol  
A general evaluation protocol for the unbiased assessment of detection algorithms was also developed in [7] as a supplement to the USOVA3D database. This evaluation returns an overall score of the detection algorithm, \(\xi_{alg}\), which takes into account several aspects of detection performance and accuracy (5 metrics) on all testing data. At the same time, a detection that deviates more strongly from the average detection performance on the testing set is considered in the overall score with a smaller weight. The advantage of our protocol is that we obtain a single effectiveness estimate for each method over all metrics, all raters, and all testing data, and, therefore, the methods do not need to be re-ranked according to each of the individual metrics.  
The overall algorithm score is determined as described in this sequel. An individual detection algorithm is validated on the USOVA3D testing set in our research. Detection effectiveness and accuracy are first evaluated using 5 metrics separately for each of the 19 entries in the testing set. The following metrics are calculated: (i) Product of sensitivity and precision ((1) in [7]), (ii) Product \(\rho_{1}\rho_{2}\), (iii) The ratio of the total volume of correctly detected objects (follicles or ovary) and the total volume of all the referential objects ((2) in [7]), (iv) The mean Euclidean distance (in voxels) between the surfaces of correctly detected and referential objects ((3) in [7]), and (v) The mean absolute difference (in voxels) between the diameters of equivalent spheres that have the same volumes as the detected and referential objects ((4) in [7]). The first three metrics have values between 0 and 1, while the last two metrics are normalised to the interval \([0,1]\). All five metrics are then summed and multiplied by 20 to give the so-called combined score, i.e., a value between 0 and 100 for each entry (volume) in the testing set.  
The USOVA3D database was annotated manually by 2 raters. The segmentation result for each entry in the testing set is, therefore, compared to the annotations of both raters. The metrics for those objects in the volume where the raters agreed in the annotations are considered with greater weight than for objects where there was disagreement between both the raters. The final score, \(\xi_{pol}\), for an individual entry (volume) in the testing set is calculated from the combined scores in this way. There are currently 19 entries in the USOVA3D testing set, which means we get 19 final scores \(\xi_{pol}\). Finally, the overall algorithm score, \(\xi_{alg}\), is determined by statistical analysis of the 19 final scores \(\xi_{vol}\), calculated for all volumes in the USOVA3D testing set.  
An overall algorithm score was introduced primarily for ranking the detection algorithms in respect of their performance. The overall score \(\xi_{alg}\) equal to 100 would have a perfect detection algorithm, while \(\xi_{alg}=0\) would be assigned to the worst detection algorithm possible. The evaluation protocol summarised here is explained in detail in [7]. Auxiliary routines that implement this evaluation protocol are available on the USOVA3D portal.  
This evaluation protocol can also be used with a small extension to assess inter-rater variability. In the case of the USOVA3D database, at first we take the annotations of the first rater as 'ground truth' and the annotations of the second rater as the detection algorithm results, and then calculate the overall score (\(\xi_{rater2}\) score, that measures the variability of rater 2's annotations with respect to rater 1). Afterwards we reverse the raters' roles and repeat the calculation (\(\xi_{rater1}\) score). The inter-rater variability and reliability of the USOVA3D database has been analysed carefully in [7].