# Video-Based Gaze Detection for Oculomotor
## 2 Materials and Methods  
### Gaze Detection Model  
The gaze detection model used in this study was developed based on existing published NN models for video-based eye detection, eye segmentation, and gaze detection [22; 28], with the addition of information regarding visual stimuli target locations, as depicted in Figure 1.  
The initial input--namely, an image containing the subject's face--is first processed using the MediaPipe neural network. The MediaPipe Face model [29] detects face landmarks from still images or video; specifically, landmarks around the eyes, such as the iris and the eyelid boundaries. These landmarks are used to extract the eye position in each image. Then, using the eye position information, two smaller images of the left and right eye are cropped from the original image, of which one is horizontally flipped, as is common practice in the field [20], in order to optimize training by assuming symmetry of the extracted features. The cropped eye images, along with the eye position in the image, are then passed into the NeuralLight Gaze Estimation Model (NLGEM), the general architecture of which is shown in Figure 2.  
The NeuralLight Gaze Estimation Model is a deep neural network (DNN) composed of three distinct modules: (1) A Convolutional Layers Module, including two equally weighted networks (for the two eyes), which extracts relevant features from eye images using convolutional layers; (2) Fully connected layers (Eye Position Data Module), which generate features based on eye position data (i.e., this module processes spatial information); (3) Integration layers (Combination Module), which utilize fully connected layers to combine the features extracted from the image (module 1) and spatial information (module 2). The output is mapped into the target PoR co-ordinates for each eye, denoted as X.  
Figure 1: A schematic description of the gaze detection model configuration, where the only input data are the camera-captured image and the stimuli target location on the display. The output Y is the calibrated gaze point of regard (PoR) on the display. The NLGEM model was trained using 109 subjects as a part of a large healthy subject cohort [30] examined in 925 sessions, with 153,119 frame samples. The NLGEM training cohort included 70 males and 39 females, with a mean age of 41 and a standard deviation of 18.5 years and with the IR eye tracker readings as the training labels. The NLGEM output gaze point co-ordinates of the two eyes are averaged into a single generic gaze estimation PoR, X, represented with two co-ordinates, which are passed along with the target location into the Calibrated Gaze Model (CGM), which predicts the calibrated gaze PoR, Y, as described in Equations (1)-(3), for multiple frames. The calibrated gaze matrix \(\mathbf{Y}\) and generic gaze matrix \(\mathbf{X}\) are given as follows:  
\[\mathbf{X}= \begin{bmatrix}x_{h}\\ x_{v}\end{bmatrix} ,\ \ \mathbf{Y}= \begin{bmatrix}y_{h}\\ y_{v}\end{bmatrix}  \tag{1}\]  
where \(\mathbf{x}_{h}\) and \(\mathbf{x}_{v}\) are the NLGEM gaze estimation vectors in the horizontal and the vertical axes, respectively, and \(\mathbf{y}_{h}\) and \(\mathbf{y}_{v}\) are the CGM calibrated gaze estimation vectors in the horizontal and vertical axes, respectively. For each of N known stimulus target points, a group of calibrated point matrices \(\mathbf{Y}_{i=1...N}\) and their corresponding NLGEM generic gaze estimation matrices \(\mathbf{X}_{i=1...N}\) are extracted, and the calibrated gaze vector is approximated by solving a regression problem defined according to the following equations. \[y_{i,h}=E_{h}(\mathbf{X}_{i})\ ;\ y_{i,v}=E_{v}(\mathbf{X}_{i}) \tag{2}\]  
where \(\mathbf{E}_{h}\) and \(\mathbf{E}_{v}\) are second-order polynomial regression functions [31; 32; 33] of the following forms:  
\[