SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guangrong Zhao $^{1}$ , Yurun Yang $^{1}$ , Jingwei Liu $^{1}$ , Ning Chen $^{1}$ , Yiran Shen $^{1*}$ , Hongkai Wen $^{2}$  and Guohao Lan $^{3}$
$^{1}$ School of Software, Shandong University, China
$^{2}$ Department of Computer Science, University of Warwick, UK
$^{3}$ Department of Software Technology, Delft University of Technology, NL
{guangrong.zhao, yiran.shen}@sdu.edu.cn
hongkai.wen@warwick.ac.uk
g.lan@tudelft.nl
In this paper, we present EV-Eye, a first-of-its-kind large-scale multimodal eye tracking dataset aimed at inspiring research on high-frequency eye/gaze tracking. EV-Eye utilizes the emerging bio-inspired event camera to capture independent pixel-level intensity changes induced by eye movements, achieving submicrosecond latency. Our dataset was curated over two weeks and collected from 48 participants encompassing diverse genders and age groups. It comprises over 1.5 million near-eye grayscale images and 2.7 billion event samples generated by two DAVIS346 event cameras. Additionally, the dataset contains 675 thousand scene images and 2.7 million gaze references captured by a Tobii Pro Glasses 3 eye tracker for cross-modality validation. Compared with existing event-based high-frequency eye tracking datasets, our dataset is significantly larger in size, and the gaze references involve more natural and diverse eye movement patterns, i.e., fixation, saccade, and smooth pursuit. Alongside the event data, we also present a hybrid eye tracking method as a benchmark, which leverages both the near-eye grayscale images and event data for robust and high-frequency eye tracking. We show that our method achieves higher accuracy for both pupil and gaze estimation tasks compared to the existing solution.
Eye tracking is a technique that continuously measures the movement of the eyes [1] and has shown great promise in a wide range of scientific fields [2] and everyday applications [3]. Traditional eye tracking systems in current mainstream leverage traditional CCD/CMOS cameras to capture the appearance of eyes for tracking [4, 5, 6, 7, 8]. Unfortunately, constrained by the frame rate and the limited bandwidth of CCD/CMOS camera, the tracking frequency of conventional eye tracking systems is often bounded by a few hundred hertz, e.g., the latest Tobii Pro Glasses 3 eye tracker [9] has a tracking frequency of  $100\mathrm{Hz}$ . Similarly, Pupil Labs [10] achieves up to  $200\mathrm{Hz}$  gaze estimation. Although such a modest frequency is enough for many common use cases, e.g., human-computer interaction [11, 12] and activity recognition [13, 14], it stands in the way of enabling game-changing applications that may require over kilohertz tracking frequency, such as the diagnosis of mental disorders [2, 15], and gaze-based user authentication [16, 17]. The peak angular speed of the human eye in saccade can reach up to  $700^{\circ}/s$  [18] and the eye's acceleration is up to  $24,000^{\circ}/s^{2}$  [19]. The high-frequency eye tracking can reveal latent information encoded in high-speed and irregular
eye movements. For example, during the diagnosis of mental disorders, an increase in delays for visually guided saccades can help to characterize ADHD [20]. Unfortunately, achieving accurate eye tracking beyond kilohertz frequency requires a significant increase in camera bandwidth, which becomes a fundamental barrier for mainstream CCD/CMOS camera-based systems. Some high-end eye tracking systems (costing tens of thousands of US dollars), such as the EyeLink 1000 [21], can provide eye tracking at 1KHz by utilizing high-speed cameras. However, the high frame rate presents a considerable computational burden on downstream processing.
This challenge has inspired the use of the emerging bio-inspired dynamic vision sensor [22, 23], also known as the event camera, for eye tracking [24, 25]. In contrast to traditional CCD/CMOS cameras, which acquire information in a frame-based principle with a fixed frame rate, an event camera perceives the scene by capturing independent pixel-level light-intensity changes and producing asynchronous event streams that indicate the locations and polarity of the intensity changes. Due to its asynchronous nature and cost-saving readout, an event camera can achieve sub-microsecond latency [22]. Moreover, an event camera operates adaptively: the faster the targeted motion, the more events are generated per second, and vice versa. In a near-eye scenario, the light-intensity changes induced by eye movement are sparse in both time and space [25]. Thus, the event camera can adapt to the density of events based on the speed of eye movements and utilizes the camera bandwidth more efficiently than traditional cameras. These properties make event cameras ideal for high-frequency eye tracking.
While event-based eye tracking has demonstrated significant advantages over its frame-based counterpart, it is still in its early stage and requires substantial research efforts and public resources, such as datasets. First, similar to traditional camera-based eye tracking [26, 8, 27, 28], the success of event-based eye tracking heavily relies on the availability of large-scale event-based datasets. However, as detailed later in Section 3, the only existing dataset, EBVEYE [25], provides sparse gaze references with a limited range of eye movements (mainly fixation) collected from a limited number of subjects. This limitation poses a risk of high sensitivity to subject and tracking condition changes. Secondly, the state-of-the-art event-based eye tracking solution adopts the model-based approach [25], which is not robust to subject diversity and sensor noise. This work aims to shed light on the existing challenges and advances the field of event-based eye tracking by making the following contributions:
- We introduce the largest and most diverse multi-modal frame-event dataset for high-frequency eye tracking in the literature (over 170Gb in total).<sup>2</sup> To the best of our knowledge, ours is the largest multi-modal dataset aimed at high-frequency eye tracking in the literature.
- We propose a novel hybrid frame-event eye-tracking benchmarking approach tailored to the collected dataset, capable of tracking the pupil at a frequency up to  $38.4\mathrm{kHz}$ . As demonstrated using our dataset, the proposed approach significantly outperforms the existing solution [25] in both pupil and gaze estimation by a large margin.
Applications of Event Camera. The methods of event-based applications normally start by converting the asynchronous event streams into formatted representations. For example, image-like representations convert event streams into event-based "images" by calculating the distributions of the events and their timestamps. Image-like representations have been widely used for object tracking [29], gait recognition [30], and optical flow estimation [31]. To better preserve the temporal information of event streams, graph-based representations [32, 33], point cloud-based representations [34] and voxel-grid-based representations [35] have been proposed. Convolutional neural networks, graph-based convolution, and PointNet [36] can be applied to the corresponding representations for various vision tasks, including object/ action classification [32], gait recognition [33], gesture recognition [34], frames interpolation [37] etc.
Conventional Eye Tracking. Current solutions for eye-tracking are either model-based [38, 39] or appearance-based [5, 26]. The model-based methods rely on corneal reflection or pupil shape to infer the Point of Gaze (PoG) from parametric features, such as pupil centers and iris. However,
2The EV-Eye dataset and the implementation of our benchmarking methodologies can be found at: https://github.com/Ningreka/EV-Eye.
these methods are sensitive to ambient light conditions and require high-resolution images to ensure good tracking accuracy. By contrast, appearance-based methods leverage neural networks to map images of eyes to the PoG, which have shown significant improvement in gaze estimation accuracy and system robustness [5, 26]. However, limited by the frame rate of conventional cameras, the eye tracking frequency of existing methods is usually constrained to around  $200\mathrm{Hz}$ . In addition, existing conventional eye-tracking datasets rarely contain videos, and when they do, they are only of participants gazing at fixed points [40]. While there are some datasets including natural eye movement patterns such as saccades, fixations, and smooth pursuits [41, 42, 43], they do not provide data in high temporal resolution and dense gaze references.
Event-based Eye Tracking. There are also works that leverage event streams for eye tracking. For instance, Feng et al. [44] propose an event-driven eye segmentation method for eye tracking. However, the event streams were generated by the event simulator rather than collected in real-world experiments. Ryan et al. [45] propose an event-based neural network for real-time face and eye detection. However, their focus is primarily on the detection and localization of eyes, rather than pupil tracking. More recently, Stoffregen et al. [24] introduce a coded differential lighting method to track eye movement at a frequency of 1KHz. Angelopoulos et al. [25] is the most relevant work to ours, in which the authors propose a hybrid frame-event approach that achieves up to 10KHz eye tracking. However, our method exhibits greater robustness across user diversity and achieves higher accuracy in eye tracking. Additionally, our dataset consists of 48 subjects, which is twice that of EBVEYE. Lastly, our dataset includes dense gaze references for multiple eye movements i.e., fixation, saccade, and smooth pursuit (the definition can be found in Section 3.2 Eye Tracking Model), while EBVEYE only provides sparse references during fixation status.
3 The EV-Eye Dataset
3.1 Dynamic Vision Sensors
To make the paper self-contained, we will brief the principal background of event cameras. Unlike traditional RGB cameras, event cameras do not produce synchronous video frames at a fixed rate, but asynchronous event streams. Specifically, pixels of the event camera work independently, to detect the change in the light intensity of the scene as,
$$
\left| \log I (x, y, t _ {\text {n o w}}) - \log I (x, y, t _ {\text {p r e v i o u s}}) \right| <   C \tag {1}
$$
where  $I(x,y,t)$  is the intensity value of pixel  $(x,y)$  at time  $t$ . When the change of intensity at the pixel is over the threshold  $C$ , an event will be released immediately. An event stream is a collection of events over time and is represented as a stream of quadruplet  $\{x,y,t,p\}$ . When the event corresponds to a positive change, the polarity  $p$  is  $+1$  otherwise it is  $-1$ . Compared with traditional RGB cameras, event cameras possess several unique characteristics. As an event is launched as soon as a change is detected without global synchronization, the event streams are high in temporal resolution and low in response latency (in the order of microseconds). Event cameras save sensing energy and bandwidth as they produce events only when changes are detected. The high dynamic range (140 dB vs. 60 dB of traditional RGB cameras) enables them to work greatly under challenging lighting conditions. These characteristics make event cameras have great potential for high-speed motion capture and working on resource-constrained devices.
Eye Tracking Model. In this paper, we consider a common eye-tracking model. It starts by localizing the centroids of the pupil area in the image domain. Subsequently, the corresponding Point of Gazes (PoGs) are determined, representing where the users are looking in realworld, through polynomial regression [46]. Eye movements are various and can be vastly categorized into three major types in natural viewing scenarios [41], i.e., fixation, saccade, and smooth pursuit. fixation is the status of eyes when users stare at a fixed point; saccade represents the fast eye movement towards a point of interest; smooth pursuit is the eye movement when the users follow a smoothly moving object in a predictable route.
Eye Tracking Modalities. As shown in Figure 1(a), EV-Eye adopts three different sensing modalities. These modalities include near-eye grayscale images and event streams captured by two sets of DAVIS346 event cameras [47], and gaze references provided by Tobii Pro Glasses 3 [9].
(a) Collection setup
Figure 1: Illustration of dataset collection setup (a) and collected samples (b). The near-eye grayscale images overlaid with  $40\mathrm{ms}$  of events.
(b) Collected samples
Event Streams: The event streams are collected by two sets of DAVIS346 event cameras with a resolution of  $346 \times 240$ . They comprise positive and negative events triggered by intensity changes resulting from eye movements, winks, and other subtle motions. This setup provides high temporal resolution, enabling high-frequency eye tracking.
Grayscale Images: The DAVIS346 cameras also record near-eye grayscale image sequences at a frame rate of 25fps. They complement the event streams by providing rich semantic information about the eyes and facilitating robust pupil segmentation in the image domain.
Gaze References: As obtaining ground truth for Points of Gaze (PoGs) during eye movement is not feasible, we adopt a commercialized device, Tobii Pro Glasses 3 [9], to acquire gaze references. It provides the PoGs and pupil diameters of the users at  $100\mathrm{Hz}$ . Tobii Glasses have a field of view  $(\mathrm{FoV})$  of  $95^{\circ} \times 63^{\circ}$ , and it can achieve 0.6 angular error in gaze estimation in the central area of the virtual screen.
3.3 Dataset Curation
Data Collection Setup. The setup of the data collection is illustrated in Figure 1(a). During the collection, the subject's head is securely fixed on an ophthalmic headrest to prevent any unintended movement. Two DAVIS346 event cameras are positioned closely to record the movement of each eye. The aperture and exposure time of DAVIS346 are adjusted to prevent overexposure. We leverage a 32-inch monitor with a resolution of  $1920 \times 1080$  to display the visual stimulus to guide the gaze movement of the subject. The distance between the monitor and the subject is about  $33\mathrm{cm}$ , which leads to a field of view (FoV) of  $95^{\circ} \times 63^{\circ}$ . The visual stimulus is a solid red circle displayed on the monitor, with a diameter of  $60\mathrm{px}$ . In our setup, the subject wears Tobii Glasses to obtain reference PoGs. Additionally, the scene camera of Tobii records scene images throughout the experiments. All devices are synchronized before each data collection session.
Acquisition Protocol. We recruited 48 participants (28 male and 20 female) $^3$  aged between 21 and 35 years. They have various vision corrections, and each of them participates in four sessions of data collection. For the first two sessions, the screen is equally divided into  $11 \times 11$  grid, resulting in 121 squared blocks. The stimulus appears in the center of one block in a random sequence with each appearance lasting for 1.5s. The subject quickly finds the stimulus and focuses on it when it is stationary. The random sequence for each subject is the same, and each session ends when all blocks are visited. These two sessions allow us to trigger and capture both saccade and fixation states of the eye movement. For the last two sessions, we ask the subject to fixate on a smoothly moving stimulus across the screen. The stimulus first moves horizontally from the lower-right to the upper-left corner of the screen then it moves vertically from the upper-left to the lower-right. The movement of the stimulus is a predictable squared wave trajectory. The space between the adjacent horizontal trajectories is 54px, which results in 20 horizontal paths. Similarly, the space between adjacent vertical trajectories is 96px and leads to 20 vertical paths. The moving pattern of the stimulus allows us to record eye movement in smooth pursuit.
3Our data collection was approved by our local IRB council.
(a) EV-Eye (ours)
(b) EBVEYE [25]
Figure 2: Distributions of the gaze references of datasets, the gaze references provided in our dataset are significantly more dense and involve all states.
Data Annotation. We leverage the VGG Image Annotator [48] to label the pupil region of 9,011 near-eye images selected uniformly across the whole image dataset. Normally, the pupil region is regarded as an ellipse. Therefore, we label the region by adjusting the major axis, minor axis, tilt angle, and center of the ellipse [49]. Then inpolygon function [50] is applied to generate binarized masks  $G$  as the ground truth according to the region of the ellipse. Besides the data collected by DAVIS346, our dataset also includes 2.7 million PoGs estimated by the Tobii Pro eye tracker and 675 thousand images recorded by its scene camera. As the ground truth of gaze is impossible to obtain, data collected by Tobii is regarded as a reference for gaze estimation.
3.4 Data Characteristics
Our dataset includes multi-modal data collected from two DAVIS346 cameras and one Tobii Pro Glasses 3. The two DAVIS346 cameras produce 1.5 million near-eye grayscale images and more than 2.7 billion events. Figure 1(b) shows some samples of near-eye grayscale images from nine subjects. The images are overlaid with  $40\mathrm{ms}$  accumulated events between two consecutive frames.
Figure 2(a) demonstrates the distributions of the gaze references provided in our dataset. From the figure, we can observe PoGs of our dataset are densely distributed across a two-dimensional space with approx.  $95^{\circ}$  in yaw and  $63^{\circ}$  in pitch directions. However, as shown in Figure 2(b), EBVEYE [25] only provides very sparse gaze references. The red dots are the locations of the appearance of the stimulus for fixation state and the squared wave lines are the trajectory of the stimulus during smooth pursuit. EBVEYE assumes the human gaze can follow the guidance of the stimulus, though it is often not the case in practice. By comparing the two datasets, the gaze references provided in our dataset are significantly more dense and involve all states, i.e., fixation, saccade, and smooth pursuit. Then Tobii Glasses eye tracker can provide richer temporal information to enable researching on gaze estimation and eye movement dynamics [41]. It is worth noting that, in addition to the dynamic gaze references provided by Tobii, EV-Eye also provides the sparse coordinates of PoGs on the monitor screen during fixation states same as those "ground truth" provided by EBVEYE and static image-based datasets such as ETH-XGaze [51] and [52], the sparse coordinates are recorded by the scene camera of Tobii Pro Glasses 3 eye tracker synchronously[53].
4 Benchmarking Methodologies
In this section, we will describe our proposed benchmarking approach, which leverages both the near-eye grayscale images and the event streams generated by the event camera for accurate and high-frequency eye tracking. The overview of our method is shown in Figure 3.
4.1 Frame-based Pupil Segmentation
DL-based Pupil Segmentation. We employ U-Net [54] for pupil segmentation, which has been proven to achieve state-of-the-art accuracy and adopted as the backbone of several DL-based eye tracking works [55, 56, 57, 58]. The detailed settings of the model can be found in Appendix B. The pupil segmentation component outputs a binarized mask to extract the pupil area.
Figure 3: Overview of the proposed benchmarking approach of EV-Eye.
Post-processing. After obtaining the binarized mask  $\mathcal{M}$ , we adopt morphological closings [59] to remove additional noise, such as glint, in the segmented pupil area. Then, we consider the centroid of the segmented mask as the pupil center  $c$  and employ an edge detector to find the pupil boundaries  $Q$  as the pupil template for the following high-frequency tracking.
4.2 High-Frequency Event-based Pupil Tracking
Candidate Points Subset. After the pupil area is segmented out, we devise a method for selecting a subset of event points for high-frequency pupil tracking. This enables us to filter out noisy events caused by the movement of eyelashes and eyelids, which are unrelated to the actual pupil movement.
An example of the event points selection is shown in Figure 4(a). First, for each pixel on the boundary  $Q$  of the pupil template, we calculate its distance  $\gamma$  to the template center  $c$ . We denote the averaged distance for all pixels on  $Q$  as  $\bar{\gamma}$ . Then, we select a set of  $N$  event points to form a candidate point set  $P$  with  $|P| = N$  by the following rule:
$$
\lambda_ {1} \bar {\gamma} <   | | (x, y) - c | | <   \lambda_ {2} \bar {\gamma}, \tag {2}
$$
where  $(x,y)$  is the coordinate of the current event point on the image;  $||(x,y) - c||$  is the distance from the current event point to the template center;  $0 < \lambda_1 < 1$  and  $\lambda_{2} > 1$  are two scale factors (we set  $\lambda_{1} = 0.8$ ,  $\lambda_{2} = 1.2$ , respectively in our experiment). As shown in Figure 4(a), we accumulate  $N$  event points lying between the two concentric circles to form the event points subset generated by pupil movement.
(a) Candidate subset
(b) Before matching
(c) After matching
Figure 4: An example of selected candidate point subset (a): the event points lying between the two concentric circles form the candidate point subset and an example of points-to-edge matching(b), the candidate point subset guides pupil updating, the blue arrow is the moving direction of the pupil.
Points-to-edge Matching. Next, we propose a points-to-edge matching approach to update the template center based on the accumulated candidate points set  $P$ .
We first calculate the translation of the current pupil boundary  $Q$  using the candidate events set  $P$ . In our case, the events are generated due to the pupil movement in the horizontal and vertical directions in the camera space. As an example shown in Figure 4(b), when the pupil moves toward the upper-left corner, some of the iris pixels change to pupil pixels and generate red events with negative polarity (intensity decreases); similarly, some of the pupil pixels become iris pixels and produce blue events with positive polarity (intensity increases). Thus, the goal of the points-to-edge matching approach is
to find the optimal translation that optimizes the  $\ell_2$ -norm error  $E$ :
$$
\min  E (T) = \min  \frac {1}{N} \sum_ {i = 1} ^ {N} \| q _ {i} - \left(p _ {i} + T\right) \| ^ {2}, \tag {3}
$$
where  $T$  is the optimal translation;  $q_{i}$  and  $p_i$  are samples in  $Q$  and  $P$ , respectively. More specifically,  $T$  can be obtained in three steps. First, for each event sample  $p_i \in P$ , we find its closest pixel  $q_{i*}'$  in  $Q$  by the nearest neighbor search and obtain  $Q' = \{q_1', \dots, q_{i*}', \dots, q_N'\}$ , which is the set of nearest neighbors for all event points in  $P$ . Meanwhile, for each  $p_i \in P$ , we obtain its horizontal distance  $\Delta T_{xi}$  and vertical distance  $\Delta T_{yi}$  to its nearest pixel  $q_{i*}'$  in  $Q$ . In the second step, we shift each  $p_i \in P$  along the horizontal and vertical axes by  $\Delta \bar{T}_x$  and  $\Delta \bar{T}_y$ , respectively, where  $\Delta \bar{T}_x = \frac{1}{N} \sum_{i=1}^{N} \Delta T_{xi}$  and  $\Delta \bar{T}_y = \frac{1}{N} \sum_{i=1}^{N} \Delta T_{yi}$ . Finally, we repeat the first two steps and update the translation  $T$  by:
$$
T = T + \bar {\Delta} T, \tag {4}
$$
where  $\bar{\Delta T} = \{\bar{\Delta T}_x,\bar{\Delta T}_y\}$ . The iteration terminates when  $\bar{\Delta T}$  diminishes, i.e.,  $\bar{\Delta T} /T < 0.01$ .
Template Center Updating. After obtaining the optimal translation  $T$ , we use the  $N$  candidate events in  $P$  to update the template center by:
$$
c ^ {t + 1} = c ^ {t} - T, \tag {5}
$$
where  $c^t$  is the last template center. By updating the center of the pupil template with a few events, the eye movement can be tracked in high frequency.
4.3 Gaze Estimation