SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After the pupil centers are obtained, we leverage the polynomial regression [46] to estimate the PoGs on the screen. Given the coordinate of the pupil center  $c = (x_c, y_c)$ , the corresponding PoG is obtained by  $n$  order polynomial transformations:
$$
x _ {G a z e} = a _ {0} + \sum_ {u = 1} ^ {n} \sum_ {v = 0} ^ {u} a _ {(v, u)} x _ {c} ^ {u - v} y _ {c} ^ {v}, \tag {6}
$$
where  $x_{Gaze}$  is the estimated horizontal coordinate of the PoG on the screen;  $n$  is the polynomial order;  $a_{(v,u)}$  and  $b_{(v,u)}$  are the coefficients. The vertical coordinate  $y_{Gaze}$  can be obtained following the same protocol. Due to the subject heterogeneity, e.g., different kappa angles and cornea radius of human eyes, the coefficients are obtained through a subject-dependent calibration.
5 Evaluation on EV-Eye Dataset
In this section, we implement two benchmarking approaches for evaluating the EV-Eye dataset: our proposed method and the model-based method introduced in EVBEYE [25].
5.1 Implementation and Evaluation Metrics
We implement the benchmarking methods with pytroch 3.8. When training the DL-based segmentation network, Adam [60] optimizer and ReduceLROnPlateau [61] scheduler are adopted. The initial learning rate is set to  $1e^{-3}$ , then it decays by multiplying a scale factor of 0.1 when the dice coefficient is not improved after two consecutive epochs. The batch size is set to 8. Both training and testing are implemented on an NVIDIA GeForce RTX 3090 GPU with 24GB VRAM.
Four metrics are adopted for the dataset evaluation (the detailed description can be found in Appendix Section C.1): Intersection over union (IoU) is a widely used metric for pupil region segmentation [62], which corresponds to the overlap between the estimated and groundtruth pupil region. Dice coefficient (F1 Score) is another commonly used metric for eye segmentation tasks [55, 63], which measures the similarity between the estimated and groundtruth pupil region. Pixel error (PE) of eye tracking [8, 64, 65] is the localization accuracy of eye tracking demonstrated as Euclidean distance in pixels between the estimated and groundtruth pupil centers. Difference of direction (DoD) in gaze tracking is the difference between the estimated and reference gaze directions to show the performance of gaze tracking. This metric is similar to the "gaze estimation error" on existing eye tracking literature [41, 52, 51].
5.2 Evaluation on frame-based pupil segmentation
Both our method and EVBEYE [25] contain the frame-based pupil segmentation component. The difference is that we adopt DL-based pupil segmentation with post-processing (DL-based method) rather than model-based method. The 9,011 manually-segmented images are used for the user-independent evaluation: in each round, the manually labeled images from one subject are selected for testing and those from the rest 47 subjects are used for training. The IoU, F1 score, and PE of each subject for two different methods i.e., DL-based method and model-based method are reported.
IoU and F1 score. The IoUs and F1 scores of different subjects obtained by the two methods are presented in Figure 5(a) and Figure 5(b), respectively. The DL-based method achieves significantly higher IoU and FI score for all the subjects compared with the model-based method. The average of IoU value is 0.9187 and 0.8360 for DL-based method and model-based method, respectively; while the average F1 score of these two methods are 0.9560 and 0.9075, respectively. In average, the IoU and F1 score of our DL-based method are  $8.27\%$  and  $4.85\%$  higher than the model-based method in pupil segmentation task.
(a) IoUs
(b) F1 scores
Figure 5: IoUs (a) and F1 scores (b) on frame-based pupil segmentation.
PE of frame-based pupil segmentation. The PEs of the two methods are shown in Figure 6(a). Our DL-based method achieves significantly lower PE than model-based method for all subjects. The average PE of DL-based method and model-based method is  $0.64\mathrm{px}$  and  $1.3\mathrm{px}$ , respectively. Therefore, DL-based method achieves a significant  $50.7\%$  improvement in frame-based pupil segmentation over model-based method.
(a) Frame-based Pe
(b) Event-based Pe
Figure 6: The pixel error of frame-based (a) and event-based (b) pupil tracking.
5.3 Evaluation on event-based pupil tracking
Below, we compare and evaluate the accuracy of our template matching-based method and EVB-EYE [25]'s model-based method on event-based pupil tracking.
PE of event-based pupil tracking. As we are not able to obtain the event-wise groundtruth, we consider the 9,011 labeled frames as reference to assess the accuracy of event-based pupil tracking. Specifically, the two methods start with obtaining the pupil region of the last image before the labeled one. Then, event-based pupil tracking module runs through the events between the two images. The last pupil center obtained by the event-based module is compared with the groundtruth of the labeled image to obtain the tracking accuracy. The number of events for each event-based update is set to be 20 (same to that used in EVBEYE [25]). The PE of the two methods for each subject is shown in Figure 6(b). Our matching-based method achieves significantly lower PE for all subjects compared with model-based method. The average PE over all subjects is reduced by about  $6.5 \times$ , i.e., from 7.7px to 1.2px.
5.4 Evaluation on update frequency
(a) CDF
(b) PDF
Figure 7: CDF and PDF of instantaneous tracking frequency achieved by our method.
We evaluate the temporal resolution of pupil tracking using EV-Eye by calculating the cumulative distribution function (CDF) and probability density function (PDF) of the tracking frequency. We update the pupil location as soon as a candidate points subset with 20 events is accumulated. The time difference between the first and last event is  $T_{interval}$  and the instantaneous tracking frequency is defined as  $F = 1 / T_{interval}$ . The CDF and PDF of tracking frequencies are presented in Figure 7. The tracking frequency of our method is dynamic because the time of generating 20 events are adaptive to pupil movement. Moreover, the peak tracking frequency  $F_{peak}$  is up to  $38.4\mathrm{kHz}$ . This indicates our method is capable of capturing ultra-high-speed saccadic movement.
Adaptive update strategy: the tracking frequency of the benchmarking approaches is not at a fixed rate: it is determined by how long it takes to accumulate 20 events caused by the movement of the pupil region. Therefore, the tracking frequency is adaptive and proportional to the movement speed.  $38.4\mathrm{kHz}$  is the maximum tracking frequency calculated in our dataset (in saccade state). In fact, the extremely-high tracking frequency is an inherent property of an event camera which is high in temporal resolution (around tens of microseconds). Another benefit of the adaptive update strategy is its energy and computation efficiency: it is not necessary to work at very high frequency all the time. For example, when eyes are in a fixed or slow-moving state, the generated events are relatively few, leading to low updating frequency. This property can better utilize the limited resources on computation, energy, and bandwidth.
5.5 Evaluation on gaze tracking
The polynomial regression model (discussed in Section 4.3) is used to map the pupil centers in image domain to the PoGs for gaze tracking. As groundtruth is not available, we utilize the gaze references provided by Tobii for calibration following the similar protocol in [25]. We show the DoDs of the two methods for different subjects in Figure 8, since it's a general indicator on gaze estimation task [52, 41, 5, 66]. From the results we can observe, our method (Ours) achieves significantly lower DoDs for all subjects than model-based method from EVBEYE. The average DoD of Ours and model-based method for the whole field of view is  $4.71^{\circ}$  and  $9.72^{\circ}$  respectively, which indicates the gaze estimation results of our method are significantly closer to the commercialized eye tracker than model-based method proposed in EVBEYE.
Figure 8: DoDs of model-based method vs. ours with respect to the gaze references. The average DoD of our method and model-based method for the whole field of view is  $4.71^{\circ}$  and  $9.72^{\circ}$  respectively.
The collected multi-modal dataset, EV-Eye, and the proposed benchmark, as presented and evaluated above, offer accessible resources for research in developing new eye tracking algorithms with high frequency. These resources support applications that require the analysis of dynamic eye movement in high temporal resolution, such as the diagnosis of mental disorders. However, further efforts are required in the future to enhance the usability of the dataset. Firstly, additional labeling work is necessary to improve its usability. Currently, approximately 9,000 sparsely selected and labeled images are available to evaluate the accuracy of pupil region segmentation. To support a wider range of applications, a significantly larger number of images with diverse types of labels is required. For instance, we plan to label specific periods of different eye movement statuses, including saccade, fixation, smooth pursuit, blinks, etc. Furthermore, more texture details of the eye, such as regions of the pupil, iris, eyelids, and eyebrows, need to be labeled. Secondly, the dataset lacks ground truth for gaze. Obtaining ground truth for gaze tracking is challenging in real-world settings. Therefore, we utilized the commercial gaze tracker, Tobii Glasses3, to provide gaze references. However, a number of studies[67, 68] have shown that eye-trackers generally are less accurate for large view angles and this seems to be a common issue in many other eye-tracking systems such as ETH-XGaze, MPIIGaze and Angelopoulos et al [51, 52, 25]. Hence, methods to acquire more trustworthy and accurate gaze references need further exploration. Thirdly, regarding subject diversity, we are aware that at the moment the dataset is collected solely from an academic institution, which may introduce slight bias in some sense (e.g. no very diverse ethnicities). We plan to make every effort to expand our dataset to include more diverse participants. Lastly, in the context of the current work, we only select three major types of gaze behavior in natural viewing scenarios, i.e., fixation, saccade and smooth pursuit. However, we realize that considering more diverse scenarios in the real-world could enrich the gaze behavior in the dataset and thus broaden community use. In the next stage, we plan to use video stimuli such as CRCNS, DIEM [41], and Gaze360 [69] in the wild environment to capture more diverse types of eye movements.
In this paper, we introduce the most diverse and largest event-based multimodal dataset EV-Eye for high-frequency eye tracking, collected from 48 subjects with different devices. Frames and events from two DAVIS346 can describe the eye movement in extremely high temporal resolution and a commercialized eye tracker can provide densely distributed gaze references for cross modality comparison. Then we propose a novel hybrid frame-event eye-tracking approach to uncover the potential of the multi-modal dataset to achieve a tracking frequency of up to  $38.4\mathrm{kHz}$ . The extensive evaluations on EV-Eye demonstrate our method achieves significantly higher accuracy and is more robust to the diverse dataset than the state-of-the-art hybrid frame-event eye tracking method.
This work is supported by Shandong Provincial Natural Science Foundation, China, Grant No.2022HWYQ-040.
[1] Eye tracking. https://en.wikipedia.org/wiki/Eye_tracking.
[2] Maria K Eckstein, Belén Guerra-Carrillo, Alison T Miller Singley, and Silvia A Bunge. Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience, 25:69-91, 2017.
[3] Andrew T Duchowski. A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers, 34(4):455-470, 2002.
[4] Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu. Appearance-based gaze estimation with deep learning: A review and benchmark. arXiv preprint arXiv:2104.12668, 2021.
[5] Carlos H. Morimoto and Marcio R. M. Mimica. Eye gaze tracking techniques for interactive applications. Comput. Vis. Image Underst., 98(1):4-24, apr 2005.