## 2 Landmark Identification Using Annotator Precision  
Inspired by the work of [PERSON] and [PERSON], we propose an iterative method for determining the final ground truth labels that considers the precision (see Equation (2)) of each annotator separately for each landmark or position to be labelled. \[\delta_{n}(a) =\sqrt{(X_{nKD}(a)-\overline{\mathbf{X_{n}}})^{2}} \tag{1}\] \[p_{n}(a) =\frac{1}{\frac{1}{k}\Sigma\delta_{n}(a)} \tag{2}\]  
where:  
* \(K\) is the number of replicates selected for each landmark by each annotator;
* \(X_{nKD}(a)\in\Re^{KD}\) are all the selections by annotator \(a\) for landmark \(n\) in \(D\) dimensions;
* \(\overline{\mathbf{X_{n}}}\in\Re^{D}\) is the global mean position for landmark \(n\);
* \(p_{n}(a)\in\Re^{D}\) is the precision of annotator \(a\) for landmark \(n\). Using Algorithm 1, the performance of each annotator is taken into account, as well as their ability to consistently identify the position of each landmark. If an annotator is more accurate at locating one landmark over another, the more accurate annotations will contribute more significantly to the overall result for that specific landmark. Similarly, this applies when comparing the annotators' performances to each other. For each landmark \(n\), the precision of each annotator \(a\) is computed in both the \(x\) and \(y\) directions. The cumulative precision value \(P\) for each annotator is then used to determine how much of the annotator's selections contribute to the ground truth value for that specific landmark. The use of annotator precision as a metric of evaluation is a key component to the proposed method as it quantifies the ability of each annotator to locate each landmark independently. It allows the performance of each annotator to be compared during each iteration, as well as not conflating an annotator's performance locating one landmark with another. This method is applied repeatedly to update the ground truth estimate based on the annotator's precision with respect to the most recent update. In order to improve the quality of the estimation, the least precise annotator is removed from the set on each iteration. At the start of each new iteration, the current ground truth estimate of each landmark is re-initialised using the average value after removing an annotator from the set, as opposed to carrying over the previous value. This ensures that there is no bias that accumulates between iterations, preventing the history of previous iterations from biasing some annotators over others. For each iteration, the ground truth estimate is updated using the weighted average based on annotator precision. This is a key component of the proposed method as it is the means by which more precise annotators contribute more to the position of the landmark. Other methods, such as a non-weighted average, were considered but deliberately excluded from this method as they would still consider all annotators in the group equally, which, as described previously, is one of the issues with the current global mean approach. Returning to our primary use case of determining the location of canonical ground truth landmarks in images, the described process is completed independently for each landmark in each image. ```
Data: Manually annotated labels for a single data sample \(X\in\Re^{AKD}\), by \(A\) annotators. A maximum number of iterations \(M\) and an acceptable tolerance of deviation \(T\).