# Appearance-Based Gaze Estimation as a Benchmark for Eye Image Data Generation Methods
## 6 Results
### Data Augmentation Study  
The results of the data augmentation study are presented in Figure 4 as boxplots of the gaze estimation improvement as a percentage. The improvement was computed as a change in performance relative to the baseline of the sparse real training set once the generated images were mixed into it. Every data point corresponds to a per-subject mean over 5 test folds. One subject turned out to be an outlier in the performance of the best method and was separated from this analysis, which is explained in more detail below. The mean gaze estimation error achieved on the sparse training set of real images was 1.01\({}^{\circ}\). The use of the full real training set resulted in the gaze error of 0.45\({}^{\circ}\). Note that the results were normalized to be in the percentage range on a per-subject basis. Therefore, any single data point at 0% does not correspond to the gaze estimation error in 1.01\({}^{\circ}\), as this value depends on the baseline performance of the corresponding subject. Accordingly, if all data points of a given method are at 100%, only then their mean gaze estimation error is equal to 0.45\({}^{\circ}\). We found that both methods that we applied directly as designed by their original author--EyeGAN and the base StarGANv2--are unsuitable for data augmentation purposes as they degraded the gaze estimation error on average by 8.99% and 11.77%, respectively. In both cases, our proposed changes led to a substantial improvement. Using RITNet masks for EyeGAN training allowed us to achieve the change in gaze estimation error that is positive on average (+1.34%). However, the use of EyeGAN-RITNet data in the training set was still detrimental to the gaze estimation performance for 15 out of 39 subjects. The addition of SPADE normalization into the base StarGANv2 turned out to be much more impactful. StarGANv2-SPADE+AdaIN is the only method that has proved to be a reliable tool for appearance-based gaze estimation. It degraded the performance for only one subject, which was removed from this analysis. For reporting completeness, the gaze estimation results for the excluded subject were the following: 0.84\({}^{\circ}\) on the sparse real set, 0.42\({}^{\circ}\) on the full real set; the use of Blender data resulted in the +0.28% gaze error improvement, EyeGAN \(-\)31.4%, EyeGAN-RITNet \(-\) -38.87%, StarGANv2 \(-\) -0.08%, StarGANv2-SPADE+AdaIN \(-\) -31.32%. For the remaining 39 subjects, our best method achieved the mean gaze estimation increase of 11.22% with a maximum per-subject improvement of 19.75%. Moreover, it is the method with the smallest deviation from the mean performance. The performance difference between Blender, EyeGAN-RITNet, and StarGANv2-SPADE+AdaIN was statistically significant, as shown in Table 11. The other two methods were excluded from the statistical analysis, as they proved to be mostly harmful for data augmentation. The most promising contribution of our StarGANv2-SPADE+AdaIN method is that it can be used to generate useful data even for novel subjects. Figure 5 shows that there was no significant degradation in performance for test subjects that were not included in the training set. However, this claim is supported by data from only 8 people, which may not be enough for far-reaching conclusions. Aside from collecting more data, the easiest way to improve the reliability of this study is to repeat it in a 5-fold cross-validation manner, splitting all 40 available subjects into disjoint test sets. Blender is another method that achieved similar performance for the train and test subjects, which was an expected outcome, as it always generates data from the same single identity. There are two most surprising observations from the data augmentation study results. First, while the mean IoU value in Table 1 for the EyeGAN-based methods was the highest, that did not translate to good performance when assessed in an end-to-end learning scenario.