# Advancements in Gaze Coordinate Prediction Using Deep Learning: A Novel Ensemble Loss Approach
## 2 Related Works  
In recent times, there are broadly two methods for gaze prediction: The first method is the model-based approach, and the second is the appearance-based approach. The model-based approach predicts gaze by modeling physical features and shapes such as facial rotation and eye information. It utilizes a 3D model of the head and eyes to calculate the movement vector of the pupil based on the motion information of these two features. However, it has the drawback of being limited in terms of applicable environments and equipment. Next, the appearance-based approach extracts features (such as eyes and face) from input face images to predict gaze. This method allows gaze prediction from images acquired with relatively cost-effective equipment. In recent years, advances in hardware performance have made it possible to build large datasets (MPIIGaze, Gaze360, GazeCapture, etc. ), making it possible to study eye tracking using deep learning. As a result, the research on utilizing deep learning-based appearance-based models for gaze prediction has been active [18; 19; 20; 21]. First, [PERSON] et al. [6] proposed an architecture that employs a convolutional neural network(CNN)-based VGG model for predicting gaze using solely images. The inputs consist of images of the face, both eyes, and head pose, whereas the outputs use gaze angle vectors. Their proposed technique demonstrated an accuracy of 13.9\({}^{\circ}\) on the MPIIGaze dataset and 10.5\({}^{\circ}\) on the Eyediap dataset [22], based on mean errors. Although this study introduced the MPIIGaze data, the model's performance itself is not high. Next, FAZE [9] takes face images and gaze direction vectors as input and utilizes meta-learning to perform gaze prediction using the latent vectors of the encoder-decoder. For this method, they performed calibration on person ID with a small number of images, around three. Without the calibration on GazeCapture data, the mean error is approximately 3.49\({}^{\circ}\), but with three calibration samples, it is 3.18 degrees, which is the state-of-the-art performance at the time. For MPIIGaze, the mean error is 5.23\({}^{\circ}\) without calibration, which is lower than [6], but with calibration, the error decreased to 3.42\({}^{\circ}\). However, the disadvantage of this study is that while it can be used on personal devices, it is difficult to use it in situations where the calibration step is difficult to apply, such as the problem of tracking an unspecified number of eyes. The same difficulty appears in another dictionary study, Sage [10]. iTracker [7] is a study on predicting the gaze point on a mobile device, a smartphone, or tablet PC in a 2D plane using only the bilateral eye and face images in the input video. The study introduces a new dataset, GazeCapture, which consists of approximately 2.5 M images from 1450 identities. The study used a CNN model based on AlexNet, which is a lightweight model that can achieve 10-15 fps that can be performed in real-time. Based on the mean absolute error without calibration, the accuracy was 1.71 cm on a mobile phone and 2.53 cm on a tablet PC. With calibration, the accuracy improved to 1.34 cm and 2.12. For this model, the input dataset was obtained from input image and the output performance was satisfactory. Therefore, we used this model as one of the backbone models to apply the ensemble loss proposed in our study. An adaptive feature fusion network (AFF-Net) [23] is a model that has shown the best performance on the problem of predicting X and Y coordinates in a 2D plane [5]. As input, it takes a photo of a face and both eyes, and the coordinate values of the face and both eyes. In the prediction phase, we used a single, fully connected layer to infer the outputs, which are the X and Y coordinates, via regression. Using the GazeCapture dataset, they achieved a mean absolute error of 1.62 cm on smartphones and 2.30 cm on tablets. For the MPIIFaceGaze dataset, they reported a mean absolute error of 3.9 cm.