# Appearance-Based Gaze Estimation as a Benchmark for Eye Image Data Generation Methods
## 6 Results
### Content Consistency  
The content consistency was evaluated by computing the mIoU between the segmentation masks of the input and the output of every data generation method on a per-class basis and for all classes combined, as reported in Table 1. The background class typically occupies the majority of the eye image and is the easiest and at the same time the least important for us to get right, as it includes everything not related to the eyeball itself. Therefore, it is misleading to judge the eye image content consistency by a single combined mIoU, as it will be somewhat artificially inflated by the inclusion of the large mIoU for the background class. Moreover, the per-class scores may provide additional insight into the priorities of future work. Blender delivered the ideal content consistency, as the segmentation masks obtained via the rendering pipeline are pixel-perfect, and the \"input\" and the \"output\" are essentially the same mask. (Note that the original Blender masks lacked the pupil label that was added with the help of RITNet, which means that the mIoU for the pupil class, and, in turn, the combined mIoU, were very close but not technically equal to 1.0.) These Blender masks were used as input to all other data generation methods and were used as a ground-truth in computing their mIoU metrics. The missing output segmentation masks were computed by segmenting the generated images using RITNet. Our hypothesis that the use of RITNet masks to train EyeGAN would lead to improved content consistency was proved to be true. EyeGAN-RITNet significantly outperformed EyeGAN across all segmentation classes and achieved the best mIoU of 0.87 among the tested methods. We also hypothesized the same outcome from the use of SPADE in the StarGANv2 generator, and it turned out to be confirmed as well. Incorporating SPADE led to an increased mIoU of 0.57 with the base StarGANv2 achieving only 0.50. Nevertheless, the StarGANv2-SPADE+AdaIN method failed to match the mIoU scores of both EyeGAN-based methods. We propose two possible reasons to explain this result. First, EyeGAN was designed to have ground-truth mask-image pairs available for training, while StarGANv2 solved a much harder task of an unsupervised domain translation with no requirement of the paired training data. Second, StarGANv2 refined a synthetic render, rather than generating an image starting with its segmentation mask. The former approach was also utilized in SimGAN, and it was already shown to be inferior to the latter in terms of the resulting content consistency with the EyeGAN introduction. Therefore, we suggest that the StarGANv2 results may be further improved by borrowing the EyeGAN approach to image generation and exploring the way to leverage only a partial availability of ground-truth image-mask pairs in the training set.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multirow{2}{*}{**a. FID (4.7)**} & \multicolumn{2}{c}{**b. mIoU (1.0)**} \\  & & All (Sclera/Iris/Pupil) & \multirow{2}{*}{**c. Appearance Control**} \\ \hline Blender & 262.95 & \(\approx\)1.0 (1.0/1.0/\(\approx\)1.0) & \(-\) \\ \hline EyeGAN & 56.87 & 0.71 (0.48/0.66/0.78) & \(-\) \\ \hline EyeGAN-RITNet & 58.43 & 0.87 (0.69/0.88/0.93) & \(-\) \\ \hline StarGANv2 & 51.55 & 0.50 (0.43/0.45/0.16) & \(+\) \\ \hline StarGANv2-SPADE+AdaIN & 51.08 & 0.57 (0.53/0.56/0.23) & \(+\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: The comparison of all tested synthetic data generation methods using **a**. Frechet Inception Distance (FID) as a metric of image photorealism (lower is better, 4.7 is best); **b**. Mean Intersection-over-Union (mIoU) as a segmentation mask consistency metric, provided separately for sclera, iris and pupil segmentation classes (higher is better, 1.0 is best); **c**. whether it allows control over visual subject appearance in generated images.