## 5 Discussion  
As discussed in the Introduction section, one of the key challenges met during this study was how to select ground truth landmark positions in problems of high subjectivity, where it may not be possible to determine a 'true location'. How can one position be right or wrong if the selection is a matter of opinion? This was one of the key drivers for the use of the _toy problem_ in this study, as it provides a mechanism for delivering a single correct location while also representing the performance of each annotator. The synthetic data are a very close approximation to producing a globally known and agreed upon ground truth landmark in a regression problem. As such, these data provide a mechanism for objectively comparing our method of landmark selection with that of the global mean. As shown in Table 1, for each group of annotators--_expert_, _crowd_ and _adversarial_--the proposed method was able to estimate the ground truth location with a greater accuracy than computing the global mean in both iterations of the toy problem. Other than achieving an improved performance, the method also demonstrated a degree of robustness to differing the parameter selection as shown by the second iteration of the toy problem. This is further supported by the path of the landmark estimate through each of the iterations as shown in Figure 4, for each of the annotator groups, on each iteration tracking towards the simulated ground truth position and finally residing closer to the final 'known' position. When applying our method of landmark determination to a set of actual annotations, we can again see an improvement in the final position location comparing Figures 5 and 6  
Figure 6: Facial landmark ground truth localisation. This plot shows the location of the mean landmark location (shown in green) as well as the derived ground truth (red crosses). The lines joining the green and red crosses show the path taken from the initial mean location to the derived ground truth location. to the instructed positions in Figure 3. In comparison with the _toy problem_, an overall objective position cannot be objectively determined due to the subjective nature of the task; however, by visually comparing the final selected positions, shown by the red crosses in Figure 6, we can see improvements in the final selections. Landmarks around the border of the face and eyes are more accurately positioned on the boundaries, while positions with strong locating features such as the corner eyes exhibit less improvement. Two particularly effective examples are shown by the two images on the right of Figure 6 as they provide examples with extreme facial expressions as well as poses in the image. It is also interesting to note that the proposed method is invariant to the order in which the annotations are presented due to the use of the global mean as the starting position seed. With the same starting position, the iterative loop will always compute the same precision for each annotator and thus will eliminate annotators in the same order. This is the only time in the process where all annotators are considered equal and it ensures that the process is repeatable. Using seed positions that vary for each execution may result in different starting locations of \(x_{0}\) and thus could lead to differing outcomes. Further study would be required to determine if this would lead to improved results or not. When considering if the annotator precision method of ground truth derivation is suitable, it would be beneficial for the reader to assess or estimate the intra-annotator variance for each landmark of interest. One of the primary advantages of the proposed method is the ability to weight more accurate annotators higher in determining the final position, even when all of the annotators are considered 'experts'. In situations where each of the annotators are comparable in performance, the additional complexity of this method may not be worthwhile; such as the results produced by the _toy problem_.