## 3 Experimental Methods
### Empirical Toy Problem  
This toy problem was constructed to test the performance of the proposed method in a controlled environment by simulating the annotation process for a single two-dimensional landmark. In this problem, the canonical ground truth landmark \(Y_{G}\) is defined and thus can be used to compare our method of deriving ground truth regression values with that of using the global mean value. The following describes the method of simulating annotator selections and applying Algorithm 1 to these data. To represent the typical distribution of annotators, we selected \(g=3\) groups of \((N_{exp}=4)\)_expert_, \((N_{crowd}=10)\)_crowd-sourced_ and \((N_{adv}=2)\)_adversarial_ annotators. Each annotator from each group identifies the same landmark \(K=3\) times, where the objective of the simulated exercise is to find a single location that is the closest approximation of the canonical \(Y_{G}\). \(K=3\) was selected to provide some consensus within annotator selections. The simulated landmark selections were generated by first randomly selecting\(Y_{G}=(20,30)\); this value was generated using a random number generator with no other considerations made. The estimation error \(\varepsilon(a)\) (see Equation (3)) for each annotator was determined by randomly selecting offset values to apply to \(Y_{G}\) with the addition of some random noise. As the _expert_ group was the most precise, the random offset value in each the \(x\) and \(y\) direction was selected between \(\kappa_{exp}\in[-2,4]\). An offset range with a non-zero mean was selected to represent ambiguity in the annotator's estimation and to maintain the unknown nature of \(Y_{G}\). The random offset range for the _crowd-sourced_ group was \(\kappa_{crowd}\in[-6,8]\), while the _adversarial_ group, actively disregarding the instructions of the task, had a range of \(\kappa_{adv}\in[60,90]\). These value ranges were selected as a representation of ability and care, but also to allow for the occurrence of overlapping values in the case of the _expert_ and _crowd-sourced_ groups, and to disallow any overlap with the _adversarial_ group. These overlaps, as described above, represent the ability of some of the _crowd-sourced_ members to perform comparably to some _expert_ annotators and vice-versa. These landmark types are shown in Figure 1.  
\[\varepsilon(a) =\kappa(a)+\eta(0,1) \tag{3}\] \[\kappa(a) \subseteq\{\kappa_{g}^{0},\kappa_{g}^{1},...,\kappa_{g}^{n}|x_{ g},n\in\mathbb{N}\} \tag{4}\]  
where:  
* Annotator \(a\) is a member of one of the three groups (\(g\)): _expert_, _crowd_ and _adversarial_;
* \(\kappa_{g}^{0}\) to \(\kappa_{g}^{n}\) is the range of offset integers available for random selection in both \(x\) and \(y\) directions for the corresponding group \(g\);
* \(\eta(0,1)\) is a random sample from a normal distribution with mean 0 and standard deviation 1. As each annotator identified the landmark multiple times, additional variance for each selection \(\delta(a)\) (see Equation (5)) was generated by selecting random values for each annotator's standard deviation, again representative of effort and skill: \(\sigma_{exp}=[1,11]\)  
Figure 1: Simulated annotations. Simulated annotations for the toy problem, demonstrating the selections for the _expert_, _crowd-sourced_ and _adversarial_ groups, as well as the canonical ground truth \(Y_{G}\). \(\sigma_{crowd}=[1,15]\) and \(\sigma_{adv}=[10,20]\), and combining them with each of the \(K=3\) normally generated random values \(\eta(0,1)\):  
\[\delta_{r}(a) =\sigma(a)\times\eta(0,1) \tag{5}\] \[\sigma(a) \subseteq\{\sigma_{g}^{0},\sigma_{g}^{1},...,\sigma_{g}^{n}| \sigma_{g},n\in\mathbb{N}\} \tag{6}\]  
where:  
* Annotator \(a\) is a member of one of the three groups (\(g\)): _expert, crowd_ and _adversarial_;
* \(\sigma_{g}^{0}\) to \(\sigma_{g}^{n}\) is the range of standard deviation values available for random selection in both \(x\) and \(y\) directions;
* \(\eta(0,1)\) is a sample from a normal distribution with mean 0 and standard deviation 1.