# Appearance-Based Gaze Estimation as a Benchmark for Eye Image Data Generation Methods
###### Abstract  
Data augmentation is commonly utilized to increase the size and diversity of training sets for deep learning tasks. In this study, we propose a novel application of an existing image generation approach in the domain of realistic eye images that leverages data collected from 40 subjects. This hybrid method combines the benefits of precise control over the image content provided by 3D rendering, while introducing the previously lacking photorealism and diversity into synthetic images through neural style transfer. We prove its general efficacy as a data augmentation tool for appearance-based gaze estimation when generated data are mixed with a sparse train set of real images. It improved the results for 39 out of 40 subjects, with an 11.22% mean and a 19.75% maximum decrease in gaze estimation error, achieving similar metrics for train and held-out subjects. We release our data repository of eye images with gaze labels used in this work for public access. eye-tracking; gaze estimation; machine learning; generative adversarial networks; data augmentation; style transfer +
Footnote †: journal:  
most relevant in modern applications [11]. The next key component of the pipeline is the method used to estimate the gaze from camera image(s). These methods are commonly categorized as model-based, appearance-based, and hybrid between the two [12]. Overall, every method has a step of extracting the relevant features from the eye image and the main difference is in the way these are used for the gaze estimation. The first category usually involves a parametric eye model that is designed with a certain level of realism. In the purest model-based case, the extracted features are used to estimate the set of model parameters that is sufficient to make a direct gaze inference [13]. A fully appearance-based method is typically not backed by an explicit eye model and aims to produce the gaze directly from an image. With the emergence of desktop and embedded hardware for high-resolution image capture and powerful deep learning techniques for image processing [14; 15; 16], the appearance-based gaze estimation became the most common in modern ET solutions [17]. These solutions are mostly data-driven, and the representative feature distributions and large quantity of their training data are essential in achieving good performance. Data augmentation is a valuable tool in general machine learning tasks for increasing the size and diversity of the collected data set. It is commonly seen in studies that work with visual or audio information, where cropping, affine transformations, or adding random noise are applied to artificially increase the number of data samples [18; 19; 20]. These methods can usually prevent overfitting of deep learning models to the training set, but they are unlikely to help in achieving better robustness in real-world deployment of the model where it can experience completely unseen samples yet from the same domain. For complex supervised learning tasks, it is unfeasible to collect the labeled training set that completely represents the distribution of the data used for model inference. As random sub-sampling of the existing data would not help to close the gap, it may be possible to introduce the lacking variability by the synthetic generation of the new data. It is challenging not only to produce a realistic extension of the data yet also to preserve the label consistency in the generated output; otherwise the learned model has a high chance of showing poor generalization in testing scenarios. Synthetic data generation is especially appealing in ET research. The reason for this is two-fold. First, it is expensive time- and resource-wise to collect labeled data that exhibit all potential real-world variability, such as sensor positions, lighting conditions, and subject appearance. Second, it provides great research flexibility for novel ET sensors that have not yet been built in consumer-grade hardware [21; 22].