OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild
Authors: Hongyu Qu
Jianan Wei
Xiangbo Shu
Yazhou Yao
Wenguan Wang
Jinhui Tang
Venue: NeurIPS 2025 poster ()
Keywords: 3D Gaze Estimation
In-The-Wild Generalization
Semi-supervised Learning
Primary area: applications
TL;DR: We propose OmniGaze, a semi-supervised learning framework, which utilizes large-scale unlabeled data and reward-driven pseudo-labeling strategy to effectively generalize gaze estimation in the wild.

Abstract: Current 3D gaze estimation methods struggle to generalize across diverse data domains, primarily due to $\textbf{i)}$ $\textit{the scarcity of annotated datasets}$, and $\textbf{ii)}$ $\textit{the insufficient diversity of labeled data}$. In this work, we present OmniGaze, a semi-supervised framework for 3D gaze estimation, which utilizes large-scale unlabeled data collected from diverse and unconstrained real-world environments to mitigate domain bias and generalize gaze estimation in the wild. First, we build a diverse collection of unlabeled facial images, varying in facial appearances, background environments, illumination conditions, head poses, and eye occlusions. In order to leverage unlabeled data spanning a broader distribution, OmniGaze adopts a standard pseudo-labeling strategy and devises a reward model to assess the reliability of pseudo labels. Beyond pseudo labels as 3D direction vectors, the reward model also incorporates visual embeddings extracted by an off-the-shelf visual encoder and semantic cues from gaze perspective generated by prompting a Multimodal Large Language Model to compute confidence scores. Then, these scores are utilized to select high-quality pseudo labels and weight them for loss computation. Extensive experiments demonstrate that OmniGaze achieves state-of-the-art performance on five datasets under both in-domain and cross-domain settings. Furthermore, we also evaluate the efficacy of OmniGaze as a scalable data engine for gaze estimation, which exhibits robust zero-shot generalization on four unseen datasets.

OMNIGAZE: Reward-inspired Generalizable Gaze Estimation in the Wild
Hongyu Qu' Jianan Wei Xiangbo Shu Yazhou Yao', Wenguan Wang"t, Jinhui Tang
"Nanjing University of Science and Technology "Zhejiang University
Nanjing Forestry University
https://github. com/ htap///uhoggyuu///ee /uugonggyy/mmiiaaee OmniGaze
Abstract
Current 3D gaze estimation methods struggle generalize across diverse data do- mains, primarily due to i) the scareity of annotated datasets, and ii) the insufficient diversity of labeled data. In this work, we present OMNIGAZE, seme-super framework for 3D gaze estimation, which utilizes large-scale unlabeled data col- lected from diverse and unconstrained real-world environments to mitigate domain bias and generalize gaze estimation in the wild. First, we build a diverse col- lection of unlabeled facial images, varying in facial appearances, background environments, illumination conditions, head poses, and eye occlusions. In order to leverage unlabeled data spanning broader distribution, OMNIGAZE adopts a stan- dard pseudo-labeling strategy and devises reward model to assess the reliability of pseudo labels. Beyond pseudo labels as 3D direction vectors, the reward model also incorporates visual embeddings extracted by off- the-shehlf visual encoder and semantic cues from gaze perspective generated by prompting a Multimodal Large Language Model to compute confidence scores. Then, these scores are utilized select high-quality pseudo labels and weight them for loss computation. Extensive experiments demonstrate that OMNIGAZE achieves state-of-the of-the-artttttt performance on five datasets under both in-domain and cross-domain settings. Furthermore, also evaluate the efficacy of OMNIGAZE scalable data engine for gaze estimation, which exhibits robust zero-shot generalization on four unseen datasets.
Introduction
Eye gaze provides human with means for evaluating individual's interest in their internal and external environments [1, 2], which is subtle but informative. 3D gaze estimation, as a crucial topic in the field of gaze signal analysis, aims directly predict gaze direction from face images, which serves as the foundational representation in various applications, such as virtual reality [3, 4, 5], human-computer interaction [6, 8), medical diagnosis [9, 10), and driver monitor systems [11, 12).
Due to the variants of subject appearance, background environments, image quality, shooting angle and illumination across existing datasets [13, 14, 15, 16]. the performance of gaze estimation methods [17, 18] trained on a single dataset suffer from performance degradation when testing on new, unseen datasets. This limitation motivates recent research [19, 20, 21, 22, 23, 24, 25, 26] to focus on cross- cross-domain generalization for gaze estimation, seeking to bridge inter-dataset discrepancies. Though effective, these methods still constrained by the limited diversity of labeled training data, restricting their applicability for real-world applications. In contrast, enormous face images can be easily accessed by crawling from Internet [27] or synthetic generation using generative models [28].
Equal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MPuuaac
EyeDiap
Ours EyeDiap
Gone
Head
Illumination (Dright)
Eye ecclusion
MPliiaace
GvGaze
Gaze360
GMM
-
E
Appearance
Illumination (low-light)
Resolution
In-domain
Cross-domain
domain
Zero-shot
Figure 1: Left: By making efficient use of large-scale diverse unlabeled datasets via reward- pseudo lahel selection, our OMINGAZE can estimate high-quality 3D gaze directions for in-the-wild images in diverse conditions, e.g., extreme head poses, varying lighting conditions, and appearance, etc. Red and yellow arrows represent predictions from OMNIGAZE and base model. Right: our OMINGAZE achieves state-of-the-art performance on five datasets under three settings, i.e., in-domain, cross-domain, and zero-shot generalization.
To generalize gaze estimation with large-scale unlabeled face datasets, there are two main strands of research: weakly supervised learning and unsupervised learning. Regarding to weakly supervised learning, previous works enhance 3D gaze estimation with weak social gaze interaction labels (e.g., mutual-gaze [29] and gaze-following [30]). while [31] generates gaze pseudo-annotation by leveraging 3D eye region geometry. However, their effectiveness is constrained by reliance on labeled datasets from gaze-related domains. In the context of unsupervised learning, self-supervised pre-tra
traininiiig stands out as the leading paradigm, which endeavors to learn a robust gaze representation via dellndes pretext tasks, e.g., eye-consistent image reconstruction [32, 33, 34], masked image restoration [35] and gaze redirection [36]. Nevertheless, these pretext tasks exhibit weak semantic relevance to gaze estimation, resulting in inefficient utilization of unlabeled face images.
In light of these limitations, we find that there remains a notable void for supervupd frameworks capable of effectively harnessing both labeled data and large-scale unlabeled datasets in the gaze esti- mation community. Then, we propose OMNIGAZE a semi-supervised learning framework (cf. Fig. 1), which employs pseudo-labeling strategy generalize gaze estimation in the wild with large-scale unlabeled face datasets. Concretely, OMNIGAZE implements standard SSL. three-phas, training protocol: i) teacher model is trained via supervised learning on annotated datasets; ii) this model utilized to generate pseudo-labels for unlabeled samples and high-quality instances are selected to enhance training data; iíi) a generalized student model is optimized by integrating both annotated and pseudo-labeles data. However, applying this strategy to 3D gaze estimation is confronted with three critical challenges: existing threshold-based
labeling methods [37, 38, 39], specifically tailored for classification tasks, are inapplicable for regression output; @ pseudo-label generated by a teacher model trained on labeled datasets with limited diversity suffer from domain bias [40], which leads to difficulty in utilizing the pseudo labels; ② learning robust gaze representations demands training data with rich diversity [41, 30, 35] to capture the wide variability across individuals.
Faced with challenge O, OMNIGAZE devises a dedicated reward model that utilizes unlabeled images paired with pseudo labels to assess the reliability of these pseudo labels. To learn reliable reward scores, propose two advancements: direction vector, thereby enabling a geometry-aware representation and enhancing alignment with natural gaze behaviors; ii) To harness the enormous knowledge stored in large-scale pretrained language models, we extract visual embeddings of unlabeled images via an off-the-shelf visual encoder and define prompt to guide the Multimodal Large Language Model to generate scene- specific gaze descriptions for unlabeled images; These linguistic descriptions are encoded via the text encoder of CLIP and combined with visual features to construct multi-modal gaze representation. Thus, the reward model can capture the nuanced nature of gaze for robust confidence assessments.
Each pseudo gaze label is interpolated into 3D gaze
As a response to challenge 2, OMNIGAZE adopt two strategies: i) utilize confidence scores to filter out unreliable pseudo labels and reweight the importance of different high-quality samples for loss computation; ii) establish a loop for mutual boosting between the student model and reward model training, enabling continuous refinement of pseudo labels to progressively enhance both gaze estimation accuracy and pscudo-label quality in OMNIGAZE.

---

To tackle challenge ① and fuel the proposed semi-supervised data engine, we curate a diverse collection of unlabeled face images from six publicly available sources, exhibiting wide variability in terms of facial appearance, lighting conditions, head poses, imaging environments, etc (Table 1).
Through embracing scaling up data well effective reward-inspired pseudo label selection, our OMEIGAZE surpasses all top-leading solutions on five datasets under both in-domain and cross- domain settings ($4.2). Furthermore, we demonstrate the efficacy of OMNIGAZE engine for generating reliable gaze annotations for facial images under diverse conditions. Without any tonettuiig. OMNIGAZE exhibits robust zero-shot generalization across four unseen datasets, evidencing its great potential for deployment in wild-scene applications ($4.3).
scalable data
2 Related Work
Appearance-based Gaze Estimation. Appearance- bbuedd gaze estimation aims to regress 3D gaze from 2D face images captured by web cameras. Early methods develop their algorithm using scene-restricted datasets and attempt to enhance generalizability through strategies such extracting gaze-correlated face features [42, 15, 43, 44] or integrating geometric constraints [45, 46, 47]. Though effective enough for certain subjects, they suffer from performance degradation in unconstrained environments, e.g., free head motion and profile faces of subjects positioned further from the camera.
To track this issue, subsequent studies endeavor to construct datasets [13, 14, 15, 16] for gaze estimation in more physically unconstrained settings. Though they employed various methods to simulate real-world scenarios, such as using panoramic cameras record multiple participants once [I [14] or multi-view photogrammetry to simulate gaze variations under extreme head poses [13]. these approaches still rely on pre-defined assumptions that inherently simplify real-world complexity, and remain difficult to scale compared to web-crawled [27], crowd-sourced [15] or synthetic data [28].
Gaze Estimation. The scarcity of diverse labeled training data in appearance based gaze estimation leads current fully-supervised methods to achieve strong within domiinn performance but suffer from poor generalization in cross-domain scenarios. To address this challenge, recent efforts for gaze estimation can be categorized into two paradigms: domain adaptation and domain generalization. Domain adaptation approaches primarily try to minimize the domain discrepancy between source domain and known target domain via strategies, e.g., adversarial learning [14, 48), collaborative learning [49], contrastive learning [22], and consistency learning [23, 24]. In contrast, recent endeavors in domain generalization address a more realistic scenario without access to target samples, focusing on learning Nomain-invariaat features through methods, e.g., self-adversarial learning to preserve gaze information [25] or data augmentation based on gaze-irrelevant factors [26]. However, given the inherent diversity of face images (e.g., s., illumination, head orientation, and eye occlusion), these methods remain constrained by their reliance on the coverage of labeled source- domain data, which hinders their performance in the unconstrained real-world environments.
Semi-supervised Learning. The goal of semi-supervised learning (SSL) is to enhance model's performance under the limited availability of labeled data by leveraging unlabeled data. The two mainstream methods are entropy minimization [37,38 38] and consistency regularization 50,50, 51, 52]. The former is proposed based on the manifold assumption or the smoothness assumption, i.e., the model should output similar predictions regardless of input perturbations, which necessitates well- designed data augmentations based on the prior of specific tasks. The latter encourages the model itself to output confident predictions on unlabeled data, leading to the main problem of SSL: how to efficiently select high-quality pseudo labels? ? For classification tasks, FixMatch [53] uses fixed confidence threshold to filter out uncertain samples, while FlexMatch [54] enhances this strategy with class-aware thresholds. Furthermore, SemiReward [55] introduces a reward score based on cosine similarity between pseudo and groundtruth labels to evaluate the quality of pseudo labels. Despite these advances, the SSI. for gaze estimation still remain to be explored.
3 OMNIGAZE
3.1
Semi-supervised Training Pipeline
We first formulate the semi-supervised framework in 3D gaze estimation. Let D,, = = W and Ny denotes the labeled and unlabeled datasets, where z! and z/ are the labeled and
Du Ne
3

---

Where the looking? Descmbe the goze
Gaze Descriptions
Image
The person looking upward, with her eyes directed toward the ceiling. Her direction vertically upword, honizontally
GT Label
InstructBLIP
Labeled Data
Shared
Prediction
Image Encoder
2
Pseudo Lobel
Feature Extracter
Multimedal Cue Integration
Label Scorer
carge-scale
Student Model
Filterring
Unlabeled Data
0,1 02
Pseudo Label
Reweighting
cos(e)) sin(0)
sin(e) couls) cos(
d Teacher
Reward Model
Pseudo
Gaze Label
Final
Rewards
Pseudo 6aze Label
3D Direction Vector
(a) Training Strategy
(b) Reward Model Scorer
Figure 2: Overview of the proposed semi-supervised learning framework. (a) OMINGAZE jointly trains on both labeled data and large-scale unlaheled data, utilizes a reward model to select and reweight high-quality pseudo labels for unlabeled data. (b) The reward model evaluates the reliability of pseudo labels by jointly reasoning over visual appearance, beene-specific gaze descriptions, and geometric gaze directions.
unlabeled face images, and y is the 3D gaze ground truth label. To make of all training data, the training pipeline of OMNIGAZE can be divided into three phases: i) Pseudo-label generation. Following previous self-training strategies [37, 38], teacher model Or pre-trained on Dz via supervised learning is applied to generate pseudo labels y for Drr; ii) Reward-driven pseudo-label selection. To measure the reliability of continuous pseudo labels, a reward model is proposed to predict confidence scores, which then used to filter out low-quality pseudo labels and reweight the contribution of high-quality samples in the loss calculation ($3.4); iii) Student model self-training, These high-quality samples along with labeled data utilized train robust student model hs ($3.5). A brief illustration of training pipeline is shown in Fig. 2.
3.2 Learning Labeled Face Images
Model Architecture. Our gaze estimation model adopts Vision Transformer (ViT) architecture [56]. Specifically, given an input image x, the model first extracts gaze representation via a transformer encoder, and then employs lightweight MLP to regress the gaze direction yaw and pitch angles.
Supervised Loss. Following [57, 58, 44], we adopt an angular loss to optimize our gaze estimator:
Σ NL Il villz
(I)
N₂
where ; lur (x)) is the estimation result.
3.3
Unleashing the Power of Unlabeled Face Images
Unlabeled Face Image Collection. Due to impoverished labeled data, current gaze estimators struggle generalize diverse real-world conditions, e.g., different facial appearance, extreme head poses, varying lighting conditions, and broader gaze distributions. To learn robust gaze representations, attempt to harness the power of large-scale unlabeled face images, which are widely available through online repositories and publicly curated datasets [59, 60, 28, 61, 62, 27] originally designed for facial analysis tasks. Concretely, we compile face images from six public datasets construct large-scale unlabeled dataset compassing over 1.4 million images, which covers diverse head poses, lighting conditions, appearance, etc. Table provides a detailed breakdown of this dataset.
Pseudo-label Generation. Given an unlabeled dataset, we first developed a high- nigh-performing teacher model to automatically generate pseudo gaze labels. Specifically, we make full use of existing labeled datasets to train this teacher model ht in supervised manner. Then, we utilize ho to assign pseudo gaze labels on unlabeled images:
Du (13) u Nej hr(e)), Du
(2)

---

Table 1: Key characteristics unlabeled training datasets used in OMINGAZE. total, our OMINGAZE
trained on labeled images and 1.4M unlabeled facial images jointly.
Dataset
Appearance Diversily
Scene
Illumination
Head Pose
Eye Occ.
Face Res.
Size
CelchA [59]
Altributes, makeup, age
Studi--ike
Controlled
Mostly Trontal
Law
~777K
VGGFace2 [60]
Identity, age, ethnicity Synthetic with variation
Real-world
Varied
Wide range
Varying
4.89K
FaceSynthetics [28]
Synthetic
Controlled
Wide range
High
86K
SFIIQ-T21 [61]
Broad demographic
Synthetic Real-worid
Varied
Wide range
Hlgh
~ 120K
VFHQ [62]
High-fidelity ethnicity
Varied
Wide range
High
~ 210K
WebFace [27]
Indoor
Controlled
Mostly frontal
Medium
3354
where Du is the pseudo labeled dataset. Then combine the labeled dataset and pseudo labeled dataset as a new training dataset 'D = Dr U Dy to jointly train a student model h.s.
Pseudo-label Selection. Pseudo-labels generated by teacher models pre-trained on limited annota- tions are susceptible to confirmation bias. The effective utilization of noisy pseudo-labels during training remains persistent challenge in self-training paradigms. Prior research [53, 54] has pre- dominantly addressed this by filtering out low-quality pseudo-labels through dynamic or handcrafted thresholding strategies. Though effective, these strategies are oriented for classification tasks while ill-suited for regression tasks like gaze estimation, where 3D gaze labels are continuous signals. In this work, accompany the student model hg with an auxiliary reward model hg that generates confidence scores to assess the reliability of pseudo-gas -azz labels. By learning to distinguish between reliable and unreliable pseudo labels, h& select high-quality samples from Du, thereby enhancing the utilization of unlabeled data and improving the efficacy of self-training for the student model.
3.4 Empowering Reward Model with Multimodal Cues
Our reward model ha (ef. Fig. 2b) evaluates the reliability of pseudo labels by reasoning of multimodal cues: geometric gaze directions, visual feature, and semantic context. By leveraging these cues, ha can capture the nuanced and context-dependent nature of gaze for robust confidence assessments.
Multimodal Cues Integration. To improve the generalization capability of the reward model for in-the-wild samples, we integrate multimodal cues, egg., visual and linguistic cues, into the reward model, enabling it to distinguish visually similar but semantically different gaze patterns. First, we extract the visual cues by encoding the input image I& E D via the visual encoder of CLIP [63]:
fx = [fass fr, f", flr] =
(3)
where M denotes the number of patch in the image. Second, we further obtain the linguistic descriptions by questioning MLLMs, e.g., InstructBLIP [64], on the imput image with a pre-defined prompt: 3D space, where is the person looking, including details about horizontal (leftriight direction, vertical (up/down) direction, and forward/backward relative to the viewer? Then, these descriptions are converted into linguistic embeddings fl. via the text encoder of CLIP. Finally, we adopt cross-altention to aggregate f and fú. resulting in an semantic-aware gaze representation:
f% AvgPool A (LI[(Crosstttt(f f" fh)),
(4)
where AvgPool is the average pooling. LN is the standard layer normalization. CrossAttn denotes the standard -ross-attention operation, where visual features f is the query, and text features fg is the key and value. As such, our reward model can dynamically attend to relevant semantic cues when interpreting visual features, enhancing its ability to disambiguate subtle gaze variations. We provide more examples for generating scene-specific gaze descriptions in the Appendix.
Reward Model as Scorer. Given a pseudo label yı= = (0k, thx), we first interpolate it into a 3D direction vector. Compared to angular representations such yaw and pitch, 3D direction vectors provide a more expressive and continuous formulation for modeling gaze behavior, which facilitates more precise alignment with semantic-aware gaze representations. Specifically, we convert the pseudo gaze label into a 3D direction vector ta using a Spherical-to-Cartesii coordinate transformation:
Us = (cos((12)) sin( cos()))) ). cos(s))) (0s)).
(5)
Leveraging the obtained semantic- aware gaze representation ft and 3D gaze vector Uk. the reward model predicts confidence scores via a cross-attention followed by an MI.P
vx))).
(6)
where ' E [0. 1] and Sigmo d denotes the Sigmoid function. Such confidence score allows the reward model to assess the consistency between visual appearance, semantic cues, and gaze labels.
5

---

To further strengthen the evaluation capability of reward model, we feed the confidence score ř3. and the cosine similarity score (between the student model prediction dA and the pseudo label y&) into a label scorer implemented via lightweight MLP to obtain final confidence scores E 1/, yielding a holistic and reliable measure of pseudo-lahel quality:
Tk = Sigmo (MLPodM(MMM(Mi sim( yy y+))
(7)
These confidence scores are then utilized to filter out unreliable samples and reweight high- qualiiy ones (see $3.5), thus enhancing the stability and effectiivnnes of student model self-training.
3.5 Training with Pscudo Labels
Training of Reward Model lg. We train reward model he by jointly using the labeled and unlabeled dataset (i.e., D,, and Dty), where we treat ground-truth gaze labels of labeled data as trust pseudo- labels. Formally, given confidence scores {
No+ No for Dr. and Dv. the reward model lig is
supervised via binary classification loss:
_ - No.tNo (cs log(rx) + (1 (x) - rx)),
(8)
where C E (0, 1} is a binary observability mask indicating the label source. Specifically, c = denotes that yı is ground-truth label, while cj 0 indicates that yı is pseudo label. By this means, the reward model gradually learns to distinguish between reliable and unreliable pseudo labels.
Training of Student Model hss Meanwhile, our student model hs receives the confidence scores from the reward model lig for Du, which indicate the reliability of the corresponding pseudo labels. These scores are used modulate the learning of the student model hs in two ways: i) filtering out low-confidence pscudo labels (i.e., Ty <), and ii) adaptively reweighting the contribution of the remaining pseudo labels. In other words, the reward model fGG guides the student model to attach more attention to correct labels and ignore erroneous labels. Formally, given Dy {#3) M/ N and corresponding confidence scores (ry {r= the unsupervised loss on unlabeled data can be defined as:
No
Nu L" ""=E, j=1 -, 2)
(1x)(=") - 1/,
(9)
where [.] is the indicator function for threshold filtering, and 0.5 is a confidence threshold. Specifically, if hG considers a pseudo label unreliable (i.e., < r, < <..5), the unsupervised loss L encourages hg to increase its prediction gap from that pseudo label. In contrast, when hg highly trusts a pseudo label (i.e., T3 1), L" enforces stronger alignment between the student prediction and pseudo label. We also explored confidence-bel yielded limited practical benefits, leaving deeper analysis for future work. Finally, overall training objective of the student model hg is an average combination of L (Eq. 1) and L" (Eq. 9).
soft weighting a [0, 1] for labeled samples, but
Pseudo-Label Update Strategy. To ensure training stability and robustness, we adopt a periodic pseudo-label update strategy, where the frozen teacher model's parameters are periodically refreshed with the student model's weights every K epochs to regenerate pseudo-labels (ablation study in Table 7 of Appendix). This interval mitigates the risk of early-stage overfitting to noisy labels while allowing the student model to progressively benefit from improved predictions over time.
Experiment
4.1 Experimental Settings
Training. OMNIGAZE is trained with batch size of 512. The training of OMNIGAZE can be divided into two stages: i) The teacher model is trained on labeled datasets for 50 epochs. We utilize the Adam optimizer [65] with an initial learning rate of 0.005, and a weight decay of 0.05. ii) We train the student model and reward model on both labeled and unlabeled data for 40 epochs with a base learning rate of 0.001 and 0.0001, respectively. Hyper-parameter K empirically set 10.
Testing, Following previous works [13, 58, 31], we use one input image scale of 224 x 224 without any data augmentation for the sake of fairesss. Note that, during model deployment, our OMNIGAZE does not bring any change to network architecture of the student model or additional computation cost. The reward model hG is directly discarded after network training.
Evaluation Metric. Following the conventions [66, 67, 68, 44], we use the angular error for evaluation, where lower values indicate better performance.

---

Table Quantitative in-domain gaze estimation results ($4.2) on five benchmarks [43, 16, 44, 14, 12].
Method
MPIlFaaceGzee [43]
EycDiap [T6]
RT-Gene [44]
Gaze360 [14]
IVGaze [12]
FullFace
[43]
[CNPSWW1
4.93
6.53
10.00
14.99
13.67
RCNN
68
4.10
5.31
10.30
11.23
Gaze360
μOCVIS}
4.06
5.36
7.06
11.04
8.15
RT-Gene
[6CCV18]
4.66
6.02
8.60
12.26
X.aaee
13
[ECCV22]
4.80
6.50
12.00
7.06
CANet 58
[MAID]]]
4.27
5.27
8.27
11.20
TazcTR
[C2922))
4.00
5.17
6.55
10.62
7.33
AGE-Net Net [70]
[CIP24]
3.61
4.78
3DGazeNet [3]]
(8CCV24)
4,00
9.60
OMNIGAZE (Ours)
2009
=.07=0.15
5.400221
9.12sa11
6.72+a15
4.2
OMNIGAZE as Generalized Gaze Estimator
We first pretrain OMNIGAZE on our curated training dataset, and then fine-tune it on downstream gaze estimation tasks to examine OMNIGAZE as the weight initialization. We investigate the effectiveness of OMNIGAZE. on two settings: 1) in-domain gaze estimation, i.e., training and testing on the same dataset, and 2) cross-domain gaze estimation, Le., training one dataset, testing In this part, we approach our algorithm on ViT-B encoder pre-trained on ImageNet-218 K. dataset [69].
unseen one.
Dataset. We curate comprehensive training dataset by combining labeled gaze datasets with six public unlabeled face datasets [59, 60, 28, 61, 62, 27] (Table 1). Note that in in-domain gaze estimation, labeled gaze datasets comprises ETH-XGaze [13] along with specific evaluation training datasets; In cross-domain gaze estimation, we only use source datasets as labeled gaze datasets. For evaluation, beyond the test split of Gaze360 [14], further assess OMNIGAZE four widely used benchmarks: MPIIFaceGaze [43], EyeDiap [16], RT-Gene [44], and IVGaze [12].
In-domain Gaze Estimation. We first compare OMNIGAZE with top-leading solutions on five widely used gaze estimation benchmarks under the in-domain evaluation setup.As shown in Ta- ble 2, OMNIGAZE surpasses recent state-of-the-art gaze estimation algorithms by solid margins. In particular, it yields 0.64", 0.71%, and 1.15 reductions in angular error on MPliFaceGaze [43], EyeDiap [16], and RT-Gene [44] respectively. It is particularly impressive considering the fact that the improvement solely achieved by our semi-supervised training scheme, without any network architectural modification. Notably, we adopt the basic network design for in-domain gaze estimation, and we believe our results can be further enhanced equipped with more advanced architectures.
Cross-domain Gaze Estimation. Next we investigate the cross- domain generalization of OM-
Table 3: Quantitative cross-domain gaze estimation results ($4.2). DE. Do. DM. and DD denote ETH-XGaze [13], Gaze360 [14], MPI- IFaceGaze [43] and EyeDiap [16] datasets.
NIGAZE under the cross-domain set-
Method
DR + DD DE - Dr
Dr
DDr
Dr - DD
ting. Like previous studies [26,
FullFFace [43] (cvesom
11.13
14.42
12.35
30.15
71, 72, 73] that report results for models trained on ETH-XGaze [13]
CA.Net [58] IMAMD))
27.13
31.41
PureGaze [25]
[XAM02))
7.08
7.48
9.28
9.32
or Gaze360 [14], we evaluate OM- NIGAZE trained
RAT [23
[cve92]]
7.40
6.91
7.69
7.08
Gaze-Consisten [26]
(MAME5)
6.50
7.44
7.55
9.03
the same la-
AGG [7]
[CV9R24)
5.91
6.75
7.87
7.90
beled gaze dataset for the sake fairness. Table 3 reports
CLIP-Gaze [22
7.51
6.89
7.06
LG-Gaze [73 (EcCv24
6.45
7.22
6.83
6.86
com-
parison results with SOTA domain- generalization methods ETH-XGaze [13] and Gaze360 [14]. We can observe that our training algorithm always improves generalization on any dataset. This proves the effectiveness making use of labeled datasets and large-scale diverse unlabeled datasets.
OMNIGAAZE(UuS)
5.07
4.84
4.95
5.7516m
4.3 OMNIGAZE as Automatic Data Engine
A fundamental challenge in gaze estimation lies in the limited availability of diverse and well- annotated training data. To address this, we propose to leverage OMNIGAZE as a general-purpose automatic data engine-2 system capable of generating reliable gaze annotations for face images under diverse conditions. Thus, we comprehensively validate the zero-shot gaze estimation capability of OMNIGAZE, i.e., directly estimating gaze directions unseen datasets or in-the-wild images.
Dataset. We curate comprehensive training dataset by combining two public labeled 3D gaze datasets (i.e., Gaze360 [14] and ETH-XGaze [13]) with six public unlabeled face datasets (see Table 1
7

---

Visual comparison results (84.3) on four unseen datasets (left) and in the wild (right). Red and yellow arrows represent gaze estimation predictions from OMNIGAZE and base model trained only on labeled datasets. Four datasets from left right: MPllFaceGaze [76], EyeDiap [16], RT-Gene [44], and IVGaze [12].
Figure 3:
Table 4: Quantitative zero-shot generalization results ($4.3) on MPIIFaceGaze [43], EyeDiap [16]. RT-
Gene [44], and IVGaze [12]. Note that all methods in the first block follow the in-domain evaluation. We provide three model scales, based on ViT-S (24.8M), ViT-B (97.5M), and ViT-L (335.3M), respectively.
Method
Zero-shot
MPllFaceGaze [43]
EyeDiap [T6]
RT-Gene [44] IVGaze
FullFFuce
43
[CNPSWNU7)
4.93
6.53
10.00
13.67
Gaze360
14
[CCV19]
x
4,06
5.36
7.06
8.15
CANet
[58]
[AAADI]]
X
4.27
5.27
8.27
3DGazeNet
[CCV24]
X
4.00
DINO-B
6.52
8.50
21.09
19.82
ViT-B [56]
6.17
8.85
20.73
18.97
FaRL-B
[75
[CV902]]
6.09
8.12
19.80
18.06
OMNIGAZE-Smaa
(Ours)
3.70gmms
4.85_am
10.0220M
11.87
OMNIGAZE-BBan (Ours)
3,44
4.31=0.2
9.09#021
10.81=022
OMNIGAZE-Large (Ours)
3,03
4.15
9.01 20.1
10.43
for more details), comprising over 2.4M data samples in total. We assess OMNIGAZE on four unseen gaze estimation benchmarks, i.e., MPIIFaceGaze [43], EyeDiap [16], RT-Gene [44], and IVGaze [12].
Network Architecture. principle, our semi-spervised learning scheme can be applied into any feature encoder. In our experiments, we approach our algorithm on VIT-S, ViT-B, and ViT-L encoder.
Quantitative Zero-shot Benchmark Evaluation. Table 4 summarizes the zero-shot generalization comparison results on four representative unseen datasets[43, 16, 44, 12]. Note that we also adopt certain powerful ViT-based models (DINO-B [74], ViT-B [56], and FaRL-B [75]) as the gaze feature encoder to fine-tune tune on labeled datasets, and directly evaluate unseen datasets. DINO-B and ViT-B have general semantic representations, while FaRL-B is designed for face analysis tasks. As seen, both with ViT-B encoder, OMNIGAZE consistently outperforms other ViT-based models on diverse scenes, proving the high versatility of our algorithm. It is also worth noting that, though previous SOTA methods in the first block uses the corresponding training images (not zero-shot anymore), our OMNIGAZE still evidently superior to them on certain datasets, e-g., 4.00 - 3.44 on MPIlFaceGaze. This highlights the remarkable potential of our model as an automatic data engine.
Qualitative Zero-shot Results. We visualize predicted gaze directions by OMNIGAZE and base model (i.e., train on labeled datasets and directly evaluate unseen datasets) on four unseen datasets in Fig. (left). We observe that, after considering data scale-up and filtering out noisy pseudo labels, OMNIGAZE demonstrates generalization improvements with lower angular error on each dataset.
Qualitative Results in the Wild. In Fig. 3 (right), we provide additional qualitative results on in-the- wild images, to resemble the practical zero-shot application in real-world conditions. Compared to OMNIGAZE
predict gaze direction accurately in unseen diverse environments, e.g., extreme head poses, challenging lighting conditions, and diverse appearances. We respectfully refer the reviewer to the appendix (SF) for more qualitative results.
the base model trained only labeled datasets,
4.4 Diagnostic Analysis
For in-depth analysis, conduct ablative studies using ViT-B encoder under zero-shot setting ($4.3).
Key Component Analysis. In Table 5a, we first examine the efficacy of essential components in our algorithm. The s row reports the result of the baseline model, which only trains on labeled datasets. For 2nd row, through jointly training on labeled datasets and large-scale diverse unlabeled data in semi-supervised manner, observe consistent and modest improvements against the baseline
8

---

Table 5: A set of ablative studies ($4.4) multiple datasets (MG: MPIlFaceGaze [76], ED: Eye-
Diap [16], RG: RT-Gene [44] under the zero-shot setting.
Labeled data Unlabeled data Pscudo-labels selection
MG
ED
RG
Contidence score
MG
ED
RG
6.17
8.45
20.73
w/o reward
4.97 5.42
13.75
4.97
5.42
13.75
(Eq.
3.71 4.68
9.96
3.44
4.31
9.09
(Eq. (Ours)
3.44 4.31
9.09
(a) Core components in OMNIGAZE
(b) Confidence score
Evaluator Component
MG
RG
Filtering Reweighting
MG
RG
BASELINE
4.52
5.19
12.01
4.97
5.42
13.75
+ Scene-specific Gaze Descriptions only
3.69
4.78
10.23
4.18
5.02
11.71
+ 3D Dircction Vector only
4.03
4.93
10.56
3.84
4.78
10.39
Gaze Deseriptions Direction Vector (Ours)
3.44
4.31
9.09
3.44
4.31
9.09
(c) Network design for reward model
(d) Filtering strategy
Table 6: Comparison results with the same backbone ($4.4) on MPIIFaceGaze [76], EyeDiap [16] and
RTGene [12) under the in-domain setting.
Method
Backbone
MPIIFacoGazz [76]
EyeDiap [16]
RI-Gene [12]
FullFFace [43]
AlexNet
4,93
6.53
10.00
Gaze360 [14]
ResNet-18
4.06
5.36
7.06
XGaze [13]
ResNet-50
4.80
6,50
12.00
CANet [58]
ResNet-50
4.27
5.27
8.27
GazeTR [57]
ResNet-18
4.00
5.17
6.55
AGE-Net [70]
ResNet-34
3.61
4.78
NeGazeNet [31]
ResNet-18
4.00
BASELINE
ResNet-18
4.48
5.56
7.45
OminGaze (Ours)
ResNet-18
3.46
4.37
5,89
on each dataset (e.g., 8.45 5.422 on EyeDiap [16].. This supports our claim that large-scale unlabeled face images provides significantly high level of data diversity, thus enhancing zero-shot generalization of our method. Furthermore, after assessing and filtering out low-quality pseudo-labels via the reward model, the performance boosts 3.44 and 4.31 on MPIIFaceGaze and EyeDiap, respectively. This suggests that scaling up datasets and further selecting high-quality samples can work in a collaborative manner, confirming the effectiveness of our overall algorithmic design.
Network Design for Reward Model. We investigate the impact of scene-specific gaze descriptions and gaze label interpolation (cf. Eq.5) in the reward model ($3.4), which is summarized in Table 5c. We construct a BASELINE model that directly predicts confidence scores based on the visual appear- ance and gaze labels. First, upon aggregating visual appearance and scene-specific gaze descriptions, all datasets observe notable improvements (e.g., 4.52 1..2*+3.66 3.69 on MPIIFaceGazze [76]). This verifies the effectiveness of learning semantic-aware gaze representation. Second, after interpolating gaze labels into 3D direction vectors, we also achieve significant angular error reductions, revealing the value of capturing the underlying geometry of gaze behaviors. Finally, our full reward model delivers the best performance across all datasets, validating the joint effectiveness of our network design.
Pseudo-label Filtering Strategy. We further probe the influence of different pseudo-label filtering strategies ($3.5). As outlined Table 5d, by filtering out low-quality pseudo-labels, the method has slight improvement of 0.79 and 0.40 angle emrrr MPllFaceGaze [76] and Eye [16], respectively. addition, using confidence scores to reweight the importance of different samples, the model also exhibits improvements in angle error, achieving 3.84 and 4.78º on MPIlFaceGaze and EyeDiap, respectively. Outstandingly, OMNIGAZE achieves the highest performance on all datasets by integrating both confidence-based pseudo-label filtering and reweighting. The empirical evidence proves that our design facilitates gaze representation learning of the student model.
Student Prediction in Confidence Evaluation. To evaluate the contribution of student model and pseudo label prediction similarity sim (i: yv) in final scores T& (ef. Eq. 7), we remove the similarity sim( V1k) from the final confidence scores (cf. Eq. 6) and report the results in Table 5b. The absence of additional information about gaze estimation results in a slight performance decline, suggesting that this can serve complementary cue for the assessment ability of the reward model.
Comparison with Same Backbone. To ensure a fair comparison and address potential concerns regarding backbone capacity, we present additional experiments using the same lightweight ResNet- 18 backbone as several SOTA methods [57, 70, 31] under the in-domain setting. The comparison results are summarized in Table 6. As shown, our OMNIGAZE achieves consistently better per-

---

formance than both the baseline and existing SOTA methods (e.g., GazeTR [57], AGE-Net [70], and 3DGazeNet [31]) when adopting the similar backbone architecture. This confirms that the improvements stem from our proposed semi-supervisse learning strategies rather than model scales.
5 Conclusion
In this work, present OMNIGAZE, novel semi-supervised framework effectively generalize gaze estimation the wild via harnessing the power of large-scale unlabeled data. To achieve this, we carefully construct diverse collection of unlabeled face images, varying in head poses, illumination conditions, facial appearances, etc, and devise a reward model to filter out noisy pseudo labels in unlabeled data. The reward model jointly reasons over visual appearance, semantic gaze context, and geometric gaze labels to predict confidence scores for accurate pseudo-labe assessments. Extensive empirical analysis demonstrates that OMNIGAZE sets new SOTAs on both in-domain and cross-domain settings, and also exhibits excellent zero-shot generalization ability.
Acknowledgement. This work supported by the National Natural Science Foundation of China (Grant No. U25A20442, 62222207, 62427808, 62472222), Fundamental Research Funds for the Central Universities 222-2022500000 LD25F020001), the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No. VRLAB2025A02), the Major Research Program of Jiangsu Province (Grant No. BG2024042), the Postgraduate Research & Practice Innovation Program of Jiangsu Province (No. KYCX25_0755), and the Natural Science Foundation of Jiangsu Province (No. BK20240080).
Zhejiang Provincial Natural Science Foundation of China (No.
References
[1] Nathan Emery. The eyes have it: the neuroethology. function and evolution of social gaze. Nearoscience & biobehavioral reviews, 24(6):581-604, 2000.
[2] Lifeng Fan, Wenguan Wang, Siyuan Huang, Xinyu Tang, and Song Chun Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In ICCV, pages 5724 5733, 2019.
[3] Katerina Mania, Ann McNamara, and Andreas Polychronakis. Gaze-aware displays and interaction. In SIGGRAPH, pages 1-67. 1021.
[4] Yun Suen Pai, Benjamin Tag, Benjamin Outram, Noriyasu Vontin, Kazunori Sugiura, and Kai Kunze. Gazesim: simulating foveated rendering using depth in eye gaze for vr. In SIGGRAPH, pages 1-2, 2016.
[5] Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. Towards foveated rendering for gaze-tracked virtual reality, ACM Transactions On Graphics, 35(6):111,, 2016.
[6] Myungguen Choi, Daisuke Sakamoto, and Tetsuo Ono. Kuiper belt: Utilizing the "out-of-natural angle" region in the eye-gaze interaction for virtual reality. In CHI, pages 1-17, 2022.
[7] Carlos Elmadjian and Carlos H Morimoto. Gazebar: Exploiting the midas touch gaze interaction. In CHI. pages 1-7, 2021.
Sean Andrist, Xiang Zhi Tan, Michael Gleicher, and Bilge Mutlu. Conversational gaze aversion for humanlike robots. In CHI, pages 25-32, 2014.
[9] Zhuoqing Chang, J Matias Di Martino, Rachel Aiello, Jeffrey Baker, Kimberly Carpenter. Scott Compton, Naomi Davis, Brian Eichner, Steven Espinosa, Jacqueline Flowers, et al. Computational methods to measure patterns of gaze in toddlers with autism spectrum disorder. JAMA pediatrics, 175(8):827-836 2021.
[10] Sam Perochon, J Matias Di Martino, Kimberly LH Carpenter, Scott Compton, Naomi Davis, Brian Eichner, Steven Espinosa, Lauren Franz, Pradeep Raj Krishnappa Babu, Guillermo Sapiro, al. Early detection of autism using digital behavioral phenotyping. Nature Medicine, 29(10):2489 2497, 2023.
[11] Muhammad Qasim Khan and Sukhan Lee. A comprehensive survey of driving monitoring and assistance systems. Sensors, 19(11),5574 2019.
10

---

[12] Yihua Cheng, Yaning Zhu, Zongji Wang, Hongquan Hao, Yongwei Liu, Shiqing Cheng, Xi Wang, and Hyung Jin Chang. What do you see in vehicle? comprehensive vision solution for in-vehicle gaze estimation. In CVPR, pages 1556-1565, 2024.
[13] Xucong Zhang. Sconwook Park, Thaho Beeler, Derek Bradley, Siyu Tang, and Otmar Hilliges. Eth-xgaze: A large scale dataset for gaze estimation under extreme head pose and gaze variation. In ECCV, pages 365-381, 2020.
[14] Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba. Gaze360: Physically unconstrained gaze estimation the wild. In ICCV, pages 6912-6922200 02216 2019.
[15] Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojeiech Matusik, and Antonio Torralba. Eye tracking for everyone. CVPR, pages 2176 2184, 2016.
[16] Kemneth Alberto Funes Mora. Florent Monay, and Jean-Mare Odobez. Eyediap: A database for the development and evaluation of gaze estimation algorithms from ryb and rgb-d cameras. In Proceedings of the sympusium on eye tracking research and applications, pages 255-258. 2014.
[17] Hengfei Wang, Jun O Oh. Hyung Jin Chang, Jin Hee Na, Minwoo Tae, Zhongqun Zhang, and Sang- Choi. Gazecaps: Gaze estimation with eel-aattention-ruueed capsules. In CVPR, pages 2669-2677, 2023.
[18] Jun O Oh, Hyung Jin Chang, and Sang-Il Choi. Self-attention with convolution and deconvolution for efficient eye gaze estimation from full face image. In CVPR, pages 4992-5000, 2022.
[19] Isack Lee. Jun-Seok Yun, Hee Hyeon Kim, Youngju Na, and Seok Bong Yoo. Latentgaze: Cross-domain gaze estimation through gaze-aware analytic latent code manipulation. In ACCV, pages 3379-3395, 2022.
[20] Xin Cai, Jiabei 7eng, Shiguang Shan, and Xilin Chen. Source-free adaptive gaze estimation by uncertainty reduction. In CVPR, pages 22035-22045, 2023.
[21] Jiawei Qin, Takuru Shimoyama, and Yusuke Sugano. appearance-base based 3d gaze estimation. In CVPR, pages Seaming-hy-nooel-veeeeee
for full-face
2022.
[22] Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng
Li. Contrastive regression for domain adaptation on gaze estimation. In CVPR, pages 19385, 2022.
[23] Yiwei Bao, Yunfei Liu, Haofei Wang, and Feng Lu. Generalizing gaze estimation with rotation consistency. In CVPR, pages 4207-4216, 2022.
[24] Yoichiro Hisadome, Tianyi Wu, Jiawei Qin, and Yusuke Sugano. fusion for multi-view appearance- based gaze estimation. In WACV, pages 5985 5594, 2024.
cross-view feature
[25] Yihua Cheng, Yiwei Bao, and Feng Lu. Puregaze: Purifying gaze feature for generalizable gaze estimation. In AAAI, volume 36, pages 436-443, 2022.
[26] Mingjie Xu, Haofei Wang. and Feng Lu. Learning generalized gaze estimator from gaze-consistent feature. In AAAI, volume 37, pages 3027-3035, 2023.
(27] Dong Yi, Zhen Lei, Shengeai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint 141 2014.
[28] Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio. Thomas J Cashman, and Jamie Shotton. Fake till you make it: face analysis in the wild using synthetic data alone. In ICCV, pages 3681-3691. 2021.
[29] Rakshit Kothari, Shalini De Mello, Umar Iqbal, Wonmin Byeon, Seonwook Park, and Jan Kautz. Weakly- supervised physically unconstrained gaze estimation. In CVPR, pages 9980-9989, 2021.
[30] Pierre Vuillecard and Jean-Mare Odobez. Enhancing 3d gaze estimation in the wild using weak supervision with gaze following labels. arXiv preprint arXiv:2502. 2024449902..0249, 2025.
[31] Evangelos Ververas, Polydefkis Gkagkos, Jiankang Deng, Michail Christos Doukas, Jia Guo, and Stefanos Zafeiriou. 3dgazenet: Generalizing 3d gaze estimation with weak-supervision from synthetic views. In ECCV, pages 387-404, 2024.
[32] Yunjia Sun, Jiabei Zeng, Shiguang Shan, and Xilin Chen. Cross-encoder for unsupervised gaze representa- tion learning. In ICCV, pages 3702-3711, 2021.
[33] Yiwei Bao and Feng Lu. Unsupervised gaze representation learning from multi-view face images. In CVPR, pages 1419-1428, 2024.
11

---

[34] Yaoming Wang, Jin Li, Wenrui Dai, Bowen Shi, Xiaopeng Zhang, Chenglin Li, and Hongkai Xiong. Bootstrap autoencoders with contrastive paradigm for self-supervised gaze estimation. In ICML, 2024.
[35] Jiawei Qin, Xucong Zhang, and Yusuke Sugano. Unigaze: Towards universal gaze estimation via large-scale pre-training. arXiv preprint 0xxx:2502.02300,, 2025.
[36] Yu Yu and Jean-Marc Odobez. Unsupervised representation learning for gaze estimation. In CVPR, pages 7314-7324, 2020.
[37] Zhi-Hua Zhou and Ming Li. Semi-supervised learning by disagreement. Knowledge and Information Systems, 24:415-439, 2010.
[38] Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. CVPR, pages 10687-10698, 2020.
[39] Jiwon Kim, Youngjo Min, Dachwan Kim, Gyuseong Lee, Junyoung Seo, Kwangrok Ryoo, and Seungryong Kim. Commatch: supervieee learning with confidence-guider consistency regularization. In ECCV. pages 674-690. Springer, 2022.
[40] 1Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint 1905:0990500054: 2019.
[41] Lihe Yang. Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pages 10371-10381, 2024.
[42] Timo Schneider, Boris Schauerte, and Rainer Stiefelhagen. Manifold alignment for person independent appearance-basee gaze estimation. In ICPR, pages 1167-1172, 2014.
[43] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. It's written all over your face: Full-face appearance-base gaze estimation. In CVPRW, pages 51- 60, 2017.
[44] Tobias Fischer, Hyung Jin Chang, and Yiannis Demiris. Rt-gene: Real-time eye gaze estimation in natural environments. In ECCV, pages 252- 2018
[45] Wangjiang Zhu and Haoping Deng. Monocular free-head 3d gaze tracking with deep learning and geometry constraints. In ICCV, pages 3143-311 3152, 2017.
[46] Sconwook Park, Adrian Spurr, and Otmar Hilliges. Deep pictorial gaze estimation. In ECCV, pages 721-738, 2018
[47] Rajeev Ranjan, Shalini De Mello, and Jan Kautz. Light-weight head pose invariant gaze tracking. In CVPRW, pages 2156-2164, 2018.
[48] Kang Wang, Rui Zhao, Hui Su, and Qiang Ji. Generalizing eye tracking with bayesian adversarial lcarning. In CVPR, pages 1190--19166 2019.
[49] Yunfei Liu, Ruicong Liu, Haofei Wang, and Feng Lu. Generalizing gaze estimation with outlier-giided collaborative adaptation. In ICCV, pages 3835-3844, 2021.
[50] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang. Marios Savvides, Takahiro Shinozaki. Bhiksha Raj, al. Freemateh: adaftid thresholding for semi-supervised learning. arXiv preprint arXiv:2205.00 07246. 2022.
[51] Junbo Yin, Jin Fang, Dingfu Zhou, Liangjun Zhang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang- Semi-supervised object detection with proficient teachers. In ECCV, pages 727-743. Springer, 2022.
[52] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Logic-indueed diagnostic reasoning for semi- supervised semantic segmentation. /CCV, pages 16197-16208, 2023.
[53] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi cemi-supervised learning with consistency and confidence. In NeurIPS, pages 596-608, 2020.
[54] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pscudo labeling. In NeurIPS, pages 18408-18411) 2021.
[55] Siyuan Li. Weiyang Jin, Zedong Wang. Fang Wu, Zicheng Liu, Cheng Tan, and Stan Z Li. Semireward: A general reward model for semi-supervised learning. In ICLR, 2023.
12

---

[56] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[57] Yihua Cheng and Feng Lu. Gaze estimation using transformer. In ICPR, pages 3341-3331 3341-3347, 2022.
[58] Yihua Cheng, Shiyao Huang, Fei Wang, Chen Qian, and Feng Lu. A coarse-to-iine toneine adaptive network for appearance-based gaze estimation. In AAAI, pages 10623-10630, 2020.
[59] Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3738. 2015.
[60] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In FG, pages 67-74, 2018.
[61] David Beniaguev. Synthetic faces high quality - text 2 image (sfhq-12i) datasct, 2024.
[62] Lianghin Xic, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: high-quality dataset and benchmark for video face super-resolution. In CVPR, pages 657-666, 2022.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748 276312021
[64] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs vision- lssounlanguaae models with instruction tuning. arXiv, 2023.
[65] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2014.
[66] Mingjie Xu and Feng Lu. Gaze from origin: learning for generalized gaze estimation by embedding the gaze frontalization process. In AAAI, pages 6333-6341, 2024.
[67] Huan Liu, Julia Qi, Zhenhao Li, Mohammad Hassanpour, Yang Wang. Konstantinos N Plataniotis, and Yuanhao Test-time personalization with meta prompt for gaze estimation. AAAI, pages 3621-3629, 2024.
[68] Cristina Palmero, Javier Sclva, Mohammad Ali Bagheri, and Sergio Escalera. Recurrent enn for 3d gaze estimation using appearance and shape cues. In BMVC, 2018.
[69] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor, Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2 104. 10972, 2021.
[70] Yichen Shi, Feifei Zhang, Wenming Yang, Guijin Wang, and Nan Su. Agent-guided gaze estimation network by two-eye asymmetry exploration. ICIP, pages 2320-2326, 2024.
[71] Yiwei Bao and Feng Lu. From feature to gaze: A generalizable replacement of linear layer for gaze estimation. In CVPR, pages 449-1418, 2024.
[72] Pengwei Yin, Guanzhong Zeng. Jingjing Wang, and Di Xie. Clip-gaze: towards general gaze estimation via visual-linguistic model. In AAAI, pages 6729-6737, 2024.
[73] Pengwei Yin, Jingjing Wang, Guanzhong Zeng, Di Xie, and Jiang Zhu. Lg-gaze: Learning geometry-aware continuous prompts for language-guided gaze estimation. In ECCV, pages 1-17, 2024.
[74] Mathilde Caron. Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9650-9660, 2021.
[75] Yinglin Zheng, Hao Yang, Ting Zhang. Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in a visual-linguistic manner. In CVPR, pages 18697-18709, 2022.
[76] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In CVPR, pages 4511-4520, 2015.
[77] Shreya Ghosh. Munawar Hayat, Abhinav Dhall, and Jarrod Knibbe. Migls: Multi-task gaze estimation with limited supervision. In WACV. pages 3223-3234 2022.
[78] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?(and dataset of 230,000 3d facial landmarks). In ICCV, pages 1021-1030. 2017.
13

---

[79] MA FISCHLER AND. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM. 24(6):381-395, 1981.
[80] Xucong Zhang. Yusuke Sugano, and Andreas Bulling. Revisiting data normalization for appearance-based gaze estimation. In ETRA. pages 1-9, 2018.
[81] Peng Wu, Xiankai Lu, Hao Hu, Yonggin Xian. Jianbing Shen, and Wenguan Wang. Logiczsl: Exploring logic-induced representation for compositional zero-shot learning. In CVPR, pages 30301-30311, 2025.
[82] Hongyu Qu, Rui Yan, Xiangbo Shu, Hailiang Gao, Peng Huang, and Guo-Sen Xie. Myp-shot: Multi- velocity progressive-alinneet framework for few-shot action recognition. IEEE Transactions Multime- dia, 2025.
[83] Hongyu Qu, Jianan Wei, Xiangho Shu, and Wenguan Wang. Learning clustering-based prototypes for compositional zero-shot learning. In ICLR, 2025.
[84] Ling Xing, Hongyu Qu, Rui Yan, Xiangbo Shu, and Jinhui Tang. Locality-aware cross-modal correspon- dence learning for dense audio-visual events localization. arXiv preprint arXiv:2409.07967, 2024.
[85] Pengpeng Li, Xiangho Shu, Chun-Mei Feng, Yifei Feng, Wangmeng Zao, and Jinhui Tang. Surgical video workflow analysis via visual-language learning. npj Health Systems, 2(1):5, 2025.
[86] Ling Xing, Alex Jinpeng Wang, Rui Yan, Xiangbo Shu, and Jinhui Tang. Vision-centric token compression in large language model. NeurlPS, 2025.
[87] Jianan Wei, Tianfei Zhou, Yang, and Wenguan Wang. Nonverbal interaction detection. In ECCV, pages 277-295, 2024.
[88] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Hydra-sgg: Hybrid relation assignment for one-stage scene graph generation. ICLR, 2025.
189] Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role-playing large language models. In NeurlPS, pages 132238-132288 2024.
[90] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. In NeurIPS, pages 21 2115-111171, 2023.
[91] Liulei Li. Wenguan Wang, and Yang. Human-object interaction detection collaborated with large relation-driven diffusion models. In NeurlPS, pages 23655-23678, 2024.
[92] Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, and Jianbing Shen, Cascaded human-object interaction recognition. In CVPR, pages 4263 2263-4272, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of this work is discussed and the related details can be found in Appendix G.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: This paper does not include theoretical results. Instead, we provide compre- hensive ablation study on provided method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of this paper and the data used in this paper is publicly available (see $3 and $4).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code of this paper will be released later while the data used in this paper is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpiccccceeceeecl heabblicgguidee CodeeSbbm ssi aasionPoiicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details in $4 understand the experimental results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see the experimental results method in Table 2, Table and Table 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We conduct all experiments on four NVIDIA RTX 3090 GPUs which is provided $4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttpps:///eurre http/nnuripp..e/ cc hctppp/bllc./hcccccciicellllllll
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte:///euurppa cccpccss.......ocoooooooooo wttpcccccppcccceccccccccc hppppli//tthis/////iieepppppppee cthissullellnnss
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this work in Appendix G.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have correctly and respectfully cited the original paper that produced the dataset used in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: About the code and model of this paper, they will be publicly available as the paper is published.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology and experimental pipeline of this study do not involve the use of any large language models (LLMs).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatpps///neurip...m haarripp..cc/ooteeeec///////0250 for what should or should not be described.
21

---

Summary of the Appendix
This supplementary document provides additional details for the main paper, titled "OMINGAZE: Reward-inspieed Generalizable Gaze Estimation in the Wild". The appendix is organized as follows:
AA provides additional dataset analysis.
S presents more quantitative results.
SC provides more implementation details of OMINGAZE.
SD provides the pscudo-code of the reward model.
E shows generated scene-specific gaze descriptions along with corresponding face images.
SF offers more qualitative results.
$G discusses our limitations, broader impact, future work, and ethical considerations.
A Additional Dataset Analysis
A.1 Unlabeled Dataset Details
CelebA [59] is large-scale dataset containing variety of facial attributes, including 40 attributes such as makeup, age, and gender. This dataset consists over 200,000 images collected from several celebrities in screen-based gaze target setup. The head poses of these face images are mostly frontal, and there are minimal facial occlusions. To reduce redundancy and filter out samples with extreme head poses, we sub-sample about 177,000 images from CelebA.
VGGFace2 [60] is diverse face dataset that includes images from people of various ages, ethnicities, and identities around the world. Unlike CelebA, VGGFace2 focuses on real-world images that capture variety of environmental conditions, backgrounds, and lighting variations. This dataset features wide range of head poses, including frontal, profile, and other angles, providing greater challenges for gaze estimation. There are also varying levels of occlusion present in the images. To reduce redundancy while maintaining diversity, we select 55 images per identity from VGGFace2.
FaceSynthetics [28] is synthetic face image dataset designed simulate real-world variations in facial features, age, gender, and expression. This dataset includes wide range of head poses, from frontal to profile. As the data is synthetic, various levels occlusion are introduced for experimental purposes, Additionally, these face images are of high quality, with high resolution and rich detail.
SFHQ-T2I is synthetic face dataset that covers broad demographic range, including variations in age, gender, ethnicity, and facial expressions. The dataset consists of around 120,000 images generated in various environments, including different lighting settings and background variations. The head poses are diverse, covering wide range of angles, from frontal to profile and other perspectives. The image quality is high, with detailed and clear facial features.
VFHQ [62] is high-fidelity face video dataset with focus on generating highly realistic images. VFHQ contains various facial features and expressions, wide head poses and gaze variations. To reduce redundancy while maintaining diversity, we sub-sample every 20 frames from VFHQ.
WebFace [27] is large-scale real-world face dataset collected from various online sources. It includes a diverse range of demographics, e.g., different races, ages, genders, and facial expressions. Due to the data from the internet, WebFace features wide variety of conditions, such as varying lighting, backgrounds, and facial poses. To reduce redundancy and filter out samples with extreme head poses, we sub-sample about 354,000 images from WebFace.
A.2 Dataset Diversity
We provide additional examples from each dataset in Fig. can be observed that each dataset covers narrow range of visual conditions, e.g., specific capturing environments (e.g., indoor [13], outdoor [14], synthetic [28, 61], or in-vehicle [12] scenes), controlled or studio-like illumination, specific facial appearance, and frontal head poses. Due to the domain-specifee bias, gaze estimation models trained on one single dataset suffer from performance degradation when testing on new, unseen datasets. To overcome this limitation, OMNIGAZE not only combines multiple labeled
22

---

ETH-XGaze [13]
Gaze360
SKI
VGGFace2 [60]
SHHQ-121661
A
PaoeSynlhetics 28
VFHQI621 62]
GVGaee [22
[S9)
Figure 4: Additional examples of each dataset($A.2). Our curated training dataset (i.e., labeled datasets and scale unlabeled data) exhibits wide variability in terms of capturing environments (e.g., indoor, outdoor, synthetic, in-vehicle scenes), facial appearance, lighting conditions, head poses, etc.
datasets but also incorporates a diverse collection of larges-scale unlabeled datasets, varying in facial appearance, capturing environments, illumination conditions, head poses, eye occlusions, etc. By effectively harnessing both labeled data and targe-scale unlabeled datasets, OMNIGAZE mitigates domain bias and achieves robust, high-quality 3D gaze estimation in the wild.
B More Quantitative Results
Pseudo-label Update Strategy. We next probe the effectiveness of the periodic pseudo-label update strategy and the choice of update interval under the zero-shot setting, which is summarized in Table 7. The 1s row, which removes the pseudo-label update mechanism during training, results in consistent performance drop on each dataset, confirming that noisy pseudo labels hinder zero- shot gaze estimation generalization of the student model. We further investigate the impact of the pseudo-label update interval K. As outlined Table 7, our OMINGAZE yields the best performance with moderate update interval (i.c., K = 10). Too frequent updates regarding pscudo-labels (i.e.,
23

---

Table 7: Analysis of pseudo-label update strategy and update interval K ($B) MPalFaceGaze [43],
EyeDiap [16] and RTGene [12] under the zero-shot setting. The adopted network designs marked in red.
Update Strategy
Update Interval
K
MPlieacceGaze [43]
HyeDiap [16]
RTGine [12]
3.77
4,98
10.15
3.89
5.19
11.17
3.52
4.38
9.27
3.44
4.31
9,09
20
3.59
4.55
9.78
K= may introduce instability due to noisy early predictions, while overly sparse updates regarding pscudo-labels (i.e., K 20) limit the model's ability to correct its own mistakes, thus significantly increasing the training complexity. The empirical evidence proves that updating pseudo-labels every K=10 epochs mitigates the risk of early-stage overfitting to noisy labels while allowing the student model progressively benefit from improved predictions over time.
Unlabeled Data Size. We study the impact of the unlabeled data size under the zero-shot set- ting in Fig. 5. We randomly sample subsets from each component dataset to create 25%, 50%, and 75% subsets of the full unlabeled data. Note that 0% subset means our OM- NIGAZE is trained only on labeled data with- out any unlabeled data. When jointly train- ing our OMNIGAZE on both labeled and un- labeled data, we observe that OMNIGAZE gains stable improvements (e.g., 6.17 - 4.32 on MPIIFaceGazee [76] and 20.73 16.98
22.5
MPIlFaceCGaze
20.0
EyeDiap
17.5
RT-Gene
Angular Error (
15.0
12.5
10.0
7.5
5.0
RT-Gene [44]) as the size of unlabeled data grows (e.g., 0% subset - 50% subset). When more than 75% subset, further increasing the unlabeled data size gives marginal performance gains. We speculate this is because our per- formance is primarily driven by data diversity, which appears sufficient at this scale. This study confirms our motivation to make full use of large-scale and diverse unlabeled data for predicting gaze direction accurately across various domains.
2.5
0.0
25
50
75
100
Percentage Unlabeled Data (%)
Figure 5: The impact of unlabeled data size ($B).
Effect of Scene-specifie Gaze Descriptions. We investigate the effect of only using scene-specific gaze descriptions to train the reward model, without incorporating other semi-supervised strategies proposed in OMNIGAZE (e.g., 3D direction vector pseudo labels, pseudo label filtering and reweight- ing, periodic pseudo-label updating). We report the results in Table 8. As seen, the performance improvement brought by using only scene-specific gaze descriptions is limited, e.g., 0.42% gains on MPIlFaceGaze [76]. The empirical evidence proves that the major gains of OMNIGAZE come from the ensemble of our semi cumi-supervised training strategies rather than the use of MLLM for calibration.
Table 8: Effect of scene-specific gaze descriptions (SB) MPIlFaceGaze [76], EyeDiap [16] and RTGene [12]
under the zero-shot setting.
Method
MPllfaceGaze [76]
EycDiap [16]
GiGene [12
BASELINE
4.62
5.24
12.95
+scene-spess spennccpeccficccccc gaze descriptions OMNIGAZE (Ours)
4.20
4.91
12.03
3.44
4.31
9.09
Efficiency Analysis. Table 9 reports the inference speed comparison between OMNIG AZE and the baseline under different backbones. Note that, OMNIGAZE introduces the reward model only during training for pseudo-label assessment and selection, and discards the reward model during the testing phase. Thus, as shown in Table 9, OMNIGAZE neither introduces additional computational overhead nor architectural modification to the base model during testing compared the base model.
Comparison with Methods Utilizing Large-scale Unlabeled Data. To further validate the effec- tiveness of OMNIGAZE in leveraging unlabeled data, we conduct comprehensive comparison with
24

---

Table 9: Inference speed comparison ($B) between haseline and OMNIGAZE under different backbones.
Method
Backbone
Inference Speed (ms)
MPIFFaceaaze [76]
FyeDiap [16]
RTCiee [12]
Baseline
ResNet-11
4.48
5.56
OMNIGAZE (Ours)
ResNet-18
3,46
4.37
5,89
Baseline
ViT-B
10.9
4.62
5.24
12.95
OMNIGAZE (Ours)
VIT-B
10.9
3.44
4.31
9.09
several state-of-the-ar methods that utilize large-scale unlabeled facial datasets in Table 10. As shown, OMNIGAZE consistently outperforms all the competitors trained with large-scale unlabeled data. This verifies the effectiveness of our model design and training strategy.
Table 10: Comparison results with methods that utilize large-scale unlabeled data (SB) MPI- IFaceGaze [76], EyeDiap [16] and Gaze360 [14] under the nn-oomiin setting.
Method
Backhone
MPllFacoGaze [76]
EyeDiap [16]
Gaze360 [14]
3DGazeNet [31]
ResNet-18
4.00
9.60
MTGLS [77]
ResNet-50
4.07
12.83
UniGaze [35]
ViT-B
4.75
5.52
9.64
ST-WSGE [30]
VIT-B
6.40
8.20
13.20
OMNIGAZE (Ours)
VIT-B
2.97
4.07
9.12
More Cross-domain Results. To further evaluate the generalization ability of OMNIGAZE, we train our OMNIGAZE on ground truth datasets that are more limited (e-g.. MPMFFaceaaze [76] or GazeCap [15]), and test our model on testing datasets that have large diversity (e.g.. Gaze360 [14]).
We compare our OMNIGAZE with LAEO [29] and 3DGazeNet [31], and provide the cross-domain comparison results in Table 11. As observed, OMNIGAZE consistently outperforms these methods under the cross-domain setting, demonstrating effectiveness as generalized gaze estimator.
Table 11:
Quantitative cross-domain gaze estimation results ($B). Note that we train our OMNIG on ground truth datasets that are more limited, and test our model on testing datasets that have large diversity.
Method
MPilFaceGaze [76] + Gaze360 [14]
GazeCapture [15] + Gaze360 [14]
LABO [16]
27.2
3DGazeNet [14]
17.6
17.6
OminGaze (Ours)
13.8
14.2
Reward Model Training Strategy. We study the impact different reward model training strategies in Table 12. Here we pre-train the reward model by jointly using the labeled and unlabeled datasets, and then evaluate the performance our model with the pre-trained reward model under the zero- shot setting. As seen, the performance of the pre- pre-trained reward model is inferior to that of the daline-trained counterpart. We hypothesize that this because the pre- trained model tends to overfit to the initial distribution of pseudo labels, and thus struggles to assess the reliability of pseudo labels at different training stages.
C Implementation Details
Unlabeled Data Pre-processing, We first detect facial landmarks [78] and estimate the 3D head pose through the Perspective-n-Poine (PnP) algorithm [79]. Based on the estimated pose, we apply data normalization [80] to crop face images, so to align each face image to canonical coordinate system. Specifically, five key landmarks (i.e., eye centers, nose tip, and mouth corners) are aligned to pre-defined facial templates. Such alignment procedure is crucial for reducing pose variation and improving the generalization of gaze estimation models. Moreover, we filter out samples with extreme head poses for unlabeled datasets.
Training. OMNIGAZE is trained with batch size of 512. All face images are the size of 224x after the data normalization process. The training of OMNIGAZE can be divided into two stages: i) The teacher model is trained on labeled datasets for 50 epochs. We utilize the Adam optimizer [65] with an initial learning rate of 0.005, and weight decay of 0.05. ii) We train the student model and reward model on both labeled and unlabeled data for 40 epochs with a base learning rate of 0.001 and 0.0001, respectively. Both labeled and unlabeled datasets are balanced in minibatch to ensure each
25

---

Table 12:
Ablation studies different reward model training strategies ($B) on MPIlFaceGaze [76].
EyeDiap [16] and RTGene [12] under the zero-shot setting.
Reward Model Traiiing
MPLiFaceGaze [76]
EyeDiap [16]
RTGene [12]
Pre training
3.71
5.03
10.24
Online training (Ours)
3.44
4.31
9.09
dataset accounts for an almost equal ratio. During training, we do not apply any image augmentation.
The pseudo-label updating interval is empirically set to 10.
Reproducibility. MZNIGAZE implemented in PyTorch, and trained on on 4 NVIDIA RTX 3090 GPUs with 24GB memory per card.
D Pseudo Code
Algorithm SI provides the pseudo-code of the reward model. To guarantee reproducibility, code and pre-trained models will be made publicly available.
Algorithm S1 Pseudo-code for the reward model of OMNIGAZE in a PyTorch-like style.
ing: input face image
gaze_des: des: acene-specific gaze descriptions peeudo label: pseudo label pre_label: student model prediction
RewardNodel (img, gaze_des, psoudo_label, pre_label):
Encode images using CLIP
feats EncodeImage (ing) CLIP's visual encoder and MLP, Eq.
Tokenize encode gaze descriptions CLIP
feat_ EncodeText (Tokenizeegaaeeeeeeeeocc
Kultimodal feature integration cross-attention img_eni enfeats
(1mgg text_feats)) Eq. 4
# Convert pseudo gaze label to 3D direction vector theta, psi pseudo_labed label dir_vector (p ,os (psi) cos (tbbta)) Eq.
Obtain initial confidence scores via the reward model cross_feat CrossAttn (img. enfeats, dir_vector)
Initial confidence in [0,1], Eq.
Compute cosine similarity betveen pseudo label and prediction sim_score DosineSimilarittyppeddda (pseudo_label, pre_label)
# Define Final confidence scores
final_input (aint__sco, sim_score])
final_scors (M (MLP(fine) (MIP((inll__nnuut)) Final confidence score, Eq. 7
return acole
E Scene-specific Gaze Descriptions
The detailed, crene-specific descriptions are obtained by questioning MLLMs, e.g., InstructBLIP [64]. on the input image with a pre-defined prompt: In 3D space, where is the person looking. including details about horizontal (left/right) direction, vertical (up/down) direction, and forwand/buckward relative to the viewer? We provide several examples of scene-specific gaze descriptions in Fig. 6. As seen, scene-specific gaze descriptions complement low-level visual features by providing additional high-level spatial semantics, which are often ambiguous underdetermined in raw image space.
26

---

Face Image
Gaze Description
Face Image
Gaze Description
The person is looking slightly to their left and upward. The person not looking directly forward; their gaze is angled slightly away from the viewer.
The person is looking straight ahead and downward, with no significant left or right deviation. The gaze is angled downward, it is away from the viewer.
The person is looking slightly to the left and at eye level. The person maintains a direct gaze towards the viewer.
The person is looking straight ahead, with minimal deviation to the left or right. The gaze is eye level. The gaze is looking forward.
The baby is looking slightly to the left relative to the viewer and directed upwards. The baby is looking forward towards the viewer.
The person is looking almost straight ahead, with minimal deviation the right. The person is looking forward towards the viewer.
Figure 6: Examples of the generated scene-speciin cene-specific gaze descriptions along with face images ($E).
F
More Qualitative Results
We provide more qualitative results on in-the-wild images in Fig. 7, to resemble the practical zero- shot application in real-world conditions. We use a pre-trained facial landmark detector [78] to normalize input images for gaze estimation, and de-normalize the gaze direction predictions for visualization the original image space. We observe that our OMNIGAZE can predict gaze directions accurately in unseen diverse environments, e.g., extreme head poses, challenging lighting conditions, background environments, and diverse appearances, based on large-scale diverse unlabeled datasets and reward-driver pseudo label selection.
G Discussion
Limitation. One limitation of our algorithm is that it needs a large amount of unlabeled data, varying in facial appearances, illumination conditions, head poses, and eye occlusions, which may be time-consuming to collect. However, in practice, enormous face images can be easily accessed by crawling from the Internet [27] or synthetic generation using generative models [28]; we compile face images from six public datasets [59, 60, 28, 61, 62, 27] to construct a large-scale unlabeled dataset encompassing over 1.4 million images, which covers significant data diversity. In the future, we will attempt to collect face images from more sources to train more capable student model for generalizable gaze estimation in the wild. Additionally, though the reward model evaluates the reliability of pseudo labels and selects high-quality ones, OMNIGAZE still requires repeating the teacher model and reward model several times (i.e., pseudo-label update strategy) to refine pseudo labels like many previous semi-supervised learning algorithms, which incurs extra computational costs. Therefore, an important consideration in future research the balance between computational cost and the quality of pscudo labels provided by the teacher model. Moreover, though the reward
27

---

Figure 7: Visual comparison results ($F) on in-the wild images. Red yellow arrоws represent gaze estimation predictions from our OMNIGAZE and base model trained only on labeled datasets.
model utilizes VLMs [63] and MLLMs [64] offline extract visual features and generate scene- specific descriptions, respectively, it still requires extra computational budget for pseudo-label assessment during the training phase. Note that directly discard the reward model during the testing phase without any network architectural modification or extra inference cost.
Broader Impact. This work introduces OMNIGAZE, a powerful semi-supervised framework for generalizable gaze estimation in the wild via harnessing both labeled data and large-scale unlabeled datasets, which overcomes the limitations of previous solutions struggling generalize across diverse data domains due to the searcity and insufficient diversity of annotated datasets. Like every coin has two sides, using our framework will have both positive and negative impacts. On the positive side, OMNIGAZE pushes the boundary of gaze estimation algorithms,, particularly under the cross- domain and acro-shot/few-shol settings [81, 82, 83] that are common real-world scenarios. This advancement can significantly contribute to number of potential real-world applications, e.g., virtual reality [3, 4, 5]. human computer interaction [6, 7, 8], video understanding [84, 85], and autonomous driving [1 [11, 12]. For potential negative social impact, the reward model in OMNIGAZE relies heavily reinforcement
on VLMs [63] and MLLMS [64, 86] for pseudo-label assessment, thus leading of biases and inequalities inherent in the data used during their large-scale pre-training stage. In addition, is essential to ensure that gaze algorithms do not invade the privacy of people by adhering to ethical standards and legal regulations, so to avoid potential negative societal impacts.
Future Work. Our OMNIGAZE aims estimate high-quality 3D gaze directions for in-the-wi till images in diverse conditions by making efficient use of large-scale unlabeled datasets and reward- driven pseudo label selection. It is also interesting to extend the idea of our algorithm to develop a scalable data engine for other visual tasks, which might improve data engineering techniques for producing reliable supervision. Moreover, the design our reward model, which reasons over multimodal cues for pseudo-label assessments, stands for an early attempt to select high-quality pseudo-labels in gaze estimation and deserves generalize this framework to broader domains, e.g., nonverbal communication understanding [87] and relational reasoning [88, 89, 90, 91, 92] tasks.
further explored. In the future, we plan to
Ethical Considerations. Our research utilizes existing facial and gaze datasets, and does not generate any new face images. In accordance with ethical guidelines, we assume that these datasets are originally collected and published in compliance with relevant ethical and data protection standards. Our experimental protocols only focus on image content, ensuring that no personally identifiable information or links to other personal data are involved.
28


Reviews:
[{"review_id": "vj8rm6aW3q", "reviewer": "Reviewer_rGpP", "summary": "This submission presents a scheme to scale the training of 3D gaze direction estimation models such as to best take advantage of both labeled data as well as unlabeled face image data. The proposed OmniGaze method adopts a pseudo-labeling scheme, which is not only simple to adopt but is also proven to be effective in object recognition tasks in particular. The pseudo-labeling scheme of OmniGaze is unique in that an independent reward model is designed to assess the utility of the pseudo-labels, in a progressive manner. The reward model leverages an InstructBLIP VLM to generate a textual description of gaze, then compares this description with the pseudo-label as well as CLIP-encoded image features to assess the reliability of the pseudo-label. Furthermore, these confidence values are leveraged during student model training via a filtering and weighting approach. OmniGaze shows significant performance improvements in in-domain and cross-domain gaze estimation, out-performs self-supervised baselines in a \"zero-shot\" setting, and each of its design decisions are tested in a comprehensive ablation study.", "questions": "I do not have any specific questions that the authors should answer in a rebuttal.", "limitations": "The authors do not discuss the limitations of their proposed work nor the potential societal impact.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This paper proposes many clever ideas to allow large-scale training of gaze estimation models by selectively distilling information from labeled datasets such as Gaze360 and ETH-XGaze. I am generally impressed by the conceptual simplicity of the proposed solution, the clever use of InstructBLIP to semi-objectively assess pseudo-label reliability, the practical use of cosine-similarity to subjectively assess pseudo-label reliability, and the sensible use of filtering (instead of loss weighting only) and periodic pseudo-label update strategy to allow more effective learning over time. The results speak for themselves and the reader finds oneself nodding in agreement with the authors. This paper was a true pleasure to read.\n\nCertain results are too-good to be believable, for instance the values in Table 2 show 2.97-deg for MPI, 4.07-deg for EyeDiap - which are significant improvements compared to the state-of-the-art. While I do not doubt the authors' integrity, a brief inspection (especially into MPIIFaceGaze) would be appreciated. Did previous methods have particular failure groups that OmniGaze happens to avoid by leveraging the prior knowledge of InstructBLIP?", "was_revised": true, "final_justification": "I have carefully reviewed the paper, my original review, and the reviews of my peers as well as the authors' responses. My initial understanding and assessment of the paper remains unchanged and thus I am happy to retain my rating.", "rebuttal": "Thank you for your encouraging review and strong support! We also appreciate the reviewer for pointing out the novelty of the idea and model design, the clear clarity of our paper, and the comprehensiveness of our experiments. We provide point-to-point response below.\n\n----\n\n**Q1: Experimental results verification.**\n\n**A1:** Thanks for your careful review. We have carefully verified the reported results multiple times, and confirmed that all comparisons follow the same protocol as prior works. To further enhance reproducibility, we had promised to release our code (L20 and L238).\n\n**Q2: Clarify whether OmniGaze avoids prior failure groups during testing via the integration of InstructBLIP.**\n\n**A2:** We would like to clarify that InstructBLIP is discarded in the inference stage. Instead, we make use of the knowledge of InstructBLIP during training to assess the reliability of pseudo labels. This enables our OminGaze to select reliable pseudo labels from large-scale unlabeled data with wide variability for training. In this way, our OminGaze avoids the failure groups caused by limited diversity in labeled data (see Fig. 1 Left), not because InstructBLIP directly avoids them, but because **its knowledge helps us better judge which pseudo-labels are trustworthy**.\n\n**Q3: Limitations and potential societal impact.**\n\n**A3:** We would like to clarify that we had discussed the limitations of our OminGaze and the potential societal impact below in the Appendix Sec. G. Thank you.\n\n> **Limitation**. One limitation of our algorithm is that it needs a large amount of unlabeled data, varying in facial appearances, illumination conditions, head poses, and eye occlusions, which may be time-consuming to collect. However, in practice, enormous face images can be easily accessed by crawling from the Internet [6] or synthetic generation using generative models [3]; we compile face images from six public datasets [1,2,3,4,5,6] to construct a large-scale unlabeled dataset encompassing over 1.4 million images, which covers significant data diversity. In the future, we will attempt to collect face images from more sources to train a more capable student model for generalizable gaze estimation in the wild. Additionally, though the reward model evaluates the reliability of pseudo labels and selects high-quality ones, OmniGaze still requires repeating the teacher model and reward model several times (i.e., pseudo-label update strategy) to refine pseudo labels like many previous semi-supervised learning algorithms, which incurs extra computational costs. Therefore, an important consideration in future research is the balance between computational cost and the quality of pseudo labels provided by the teacher model. Moreover, though the reward model utilizes VLMs [18] and MLLMs [19] to offline extract visual features and generate scene-specific descriptions, respectively, it still requires extra computational budget for pseudo-label assessment during the training phase. Note that we directly discard the reward model during the testing phase without any network architectural modification or extra inference cost introduced to the base model.\n\n> **Broader Impact**. This work introduces OmniGaze, a powerful semi-supervised framework for generalizable gaze estimation in the wild via harnessing both labeled data and large-scale unlabeled datasets, which overcomes the limitations of previous solutions struggling to generalize across diverse data domains due to the scarcity and insufficient diversity of annotated datasets. Like every coin has two sides, using our framework will have both positive and negative impacts. On the positive side, OmniGaze pushes the boundary of gaze estimation algorithms, particularly under the cross-domain and zero-shot settings that are common in real-world scenarios. This advancement can significantly contribute to a number of potential real-world applications, e.g., virtual reality [20,21,22], human-computer interaction[23,24,25], and autonomous driving [26,9]. For potential negative social impact, the reward model in OmniGaze relies heavily on VLMs [19] and MLLMs [18] for pseudo-label assessment, thus leading to the reinforcement of biases and inequalities inherent in the data used during their large-scale pre-training stage. In addition, it is essential to ensure that gaze algorithms do not invade the privacy of people by adhering to ethical standards and legal regulations, so as to avoid potential negative societal impacts."}, {"review_id": "iMuNsygYv9", "reviewer": "Reviewer_uapy", "summary": "Existing gaze estimation models face two main challenges: (1) a lack of large annotated datasets, and (2) limited diversity in labeled data. To address these issues, this paper proposes a semi-supervised framework called OMNIGAZE for 3D gaze estimation, which leverages large-scale unlabeled data. OMNIGAZE employs pseudo-labeling and introduces a reward model to evaluate the reliability of pseudo labels. This reward model incorporates visual embeddings and a multimodal large language model (LLM) to extract semantic cues. Extensive experimental results demonstrate the effectiveness of the proposed approach.", "questions": "1. I would like to see analysis for novelty.\n2. the use of a ViT-based architecture in training raises concerns about fairness in comparison. Most existing methods use lightweight models such as AlexNet or ResNet-18. I would like to see a fair comparison.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths: In gaze estimation, there is a lack of both unlabeled data and diversity in existing datasets. Current methods often rely on self-supervised or weakly supervised learning to address this issue, but the results remain unsatisfactory. This paper introduces a reward-based strategy utilizing a large language model (LLM) to overcome these limitations. A teacher model, trained on labeled data and enhanced with LLM guidance, generates pseudo-labels—though leveraging these pseudo-labels effectively remains challenging. To address this, a reward model assigns confidence scores to refine the learning process. With iterative training, the model's performance improves significantly. Extensive experiments demonstrate the effectiveness of the proposed strategy.\nWeaknesses: 1. First, I have serious concerns about the novelty of the proposed method. The reward-based framework closely resembles existing approaches such as “SemiReward,” and the paper does not clearly articulate what sets it apart.\n2. Second, the performance improvements seem largely attributable to the use of a large language model (LLM) for calibration. As we know, LLMs like CLIP are pretrained on massive datasets and are inherently powerful. Therefore, relying on them raises further doubts about the novelty of the contribution.\n3. Although the paper aims to address the scarcity of annotated data, it appears to rely heavily on labeled datasets during training. This contradicts its core motivation, and I do not see a clear distinction between OMNIGAZE and prior work.\n4. Finally, the use of a ViT-based architecture in training raises concerns about fairness in comparison. Most existing methods use lightweight models such as AlexNet or ResNet-18, making the reported improvements potentially misleading.\nBased on the above points, I am unable to recommend this paper for acceptance.", "was_revised": true, "final_justification": "Thanks for answering my questions, I have reviewed the authors' rebuttal and other reviewers' comments, and the authors have provided more evidences for novelty. I increase my score to Borderline accept.", "rebuttal": "Thank you so much for the valuable time and constructive feedback for us to improve the paper. We provide point-to-point response below.\n\n----\n\n**Q1: Novelty of our OminGaze.**\n\n**A1:** We respectfully disagree. We argue that our work represents **a novel contribution in gaze estimation filed** far beyond a simple reuse of prior techniques such as SemiReward [52]. Our proposed framework is **well-motivated by the application domain (i.e., gaze estimation), and the motivation and reasoning behind it are clearly articulated**. In addition, our method also introduces **concrete technical innovations** that go beyond prior efforts. Below, we clarify the key distinctions between our work and existing approaches in terms of the **motivation** and **technical design**.\n\n**1) Motivation**\n\nLimitations of current generalized gaze estimation works: **i)** Weakly supervised methods rely on gaze-related annotations (e.g., social interaction [28,29] or 3D eye geometry [30]), restricting their ability to leverage abundant unlabeled data. **ii)** Unsupervised pretext tasks exhibit weak intrinsic semantic relevance to gaze estimation, resulting in inefficient utilization of unlabeled face images.\n\nData availability in gaze estimation: **i)** Existing gaze datasets​​ exhibit limited variants in subject appearance, image quality, and illumination. **ii)** Large-scale facial image collections can be readily accessed through internet crawling or synthesized via generative modeling techniques. \n\nThis motivates us to **develop a semi-supervised framework based on pseudo-label generation to effectively utilize both labeled and large-scale unlabeled data in the gaze estimation community**. This direction, to the best of our knowledge, has not been explored in the gaze estimation community.\n\n**2) Challenge**\n\nApplying pseudo-label based semi-supervised learning to gaze estimation poses significant challenges:\n- **C1**: Threshold-based pseudo-labeling methods, tailored for classification tasks, are inapplicable to regression outputs.\n- **C2**: Teacher models trained on limited-diversity datasets generate biased pseudo-labels.\n- **C3**: Robust gaze representation learning demands training data with rich variability to capture inter-individual differences.\n  \n**3) Technical design**\n\n**To tackle C1**, we adopt a reward model akin to that employed in SemiReward [52] to ​​assess the quality of regression pseudo-labels​​. **Notably**, the semi-supervised learning framework with the vanilla reward model only achieves performances of **4.52/5.19/12.01 for MG/ED/RG** (see Table 5c). \nTo learn reliable reward scores, we propose: **i) 3D direction vector pseudo labels** enabling geometry awareness, and **ii) MLLM-generated scene-specific gaze descriptions**. Owing to these modifications, our model achieves significant performance improvements of **1.08/0.88/2.92 on MG/ED/RG** (see Table 5c). We will provide a discussion in the related work, to highlight **the difference between our reward model design and SemiReward**.\n\n**To tackle C2**, we adopt **two training strategies (i.e., pseudo label filtering and reweighing, and periodic pseudo label updating)** to ensure high-quality pseudo-labels used for student model training, both yielding substantial performance gains (see Table 5d and S1 in Appendix). \n\n**To tackle C3**, we **curate diverse unlabeled face images from six public sources**, exhibiting wide variability in terms of facial appearance, lighting conditions, head poses, imaging environments (see L11-39 in Appendix). \n\nIn summary, driven by the distinct motivation of \"**harnessing large-scale unlabeled data to generalize gaze estimation in the wild**\", we adopt a semi-supervised learning paradigm. By systematically analyzing the key challenges of applying semi-supervised learning to gaze estimation, we propose **a set of principled and effective technical designs covering dataset construction, model innovation, and training strategies**, which significantly set our method apart from previous approaches such as SemiReward. Beyond prior works, we believe our work offers valuable **new insights** and **technical contributions** to the field of gaze estimation.\n\n**Q2: Usage of LLM.**\n\n**A2:** We respectfully disagree. **First**, an increasing number of recent works [ref1,ref2,ref3,ref4,ref5] have explored strategies to effectively transfer knowledge from large foundation models (e.g., LLM and VLM), rendering this a non-trivial endeavor. For instance, [ref1,ref2] leverage pre-trained CLIP to formulate gaze estimation as a vision-language alignment task, while [ref3] transfers knowledge from foundation models (i.e., DINO) for gaze target estimation. Therefore, it seems unfair to treat this as a specific limitation of our approach.\n\n**Second**, unlike previous methods using LLM in other filed, we employ scene-specific gaze descriptions generated by an LLM as additional linguistic cues to train the reward model for pseudo-label assessment and selection. **Notably**, the reward model along with LLM-generated gaze descriptions are removed during testing without any network modification to the base model. \n\n**Third**, the performance improvements of our method stems not largely attributable to the use of MLLM for calibration, but rather stems from the ensemble of our semi-supervised training strategies (e.g., 3D direction vector pseudo labels, pseudo label filtering and reweighing, periodic pseudo-label update strategy). Below are results using ​​only​​ \"scene-specific gaze descriptions\" for reward model training without other strategies proposed in our work. As seen, the performance improvement brought by using only scene-specific gaze descriptions is limited, e.g., 0.42 gains on MPIIFaceGaze.\n\n|Method|MPIIFaceGaze|EyeDiap|RT-Gene|\n|:-|:-|:-|:-|\n|Baseline|4.62|5.24|12.95|\n|+scene-specific gaze descriptions|4.20|4.91|12.03|\n|OminGaze (**Ours**)|**3.44**|**4.31**|**9.09**|\n\nThe discussion and results above will be integrated into the Appendix Sec. B. \n\n[ref1] CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model. AAAI 2024.\n[ref2] LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation. ECCV 2024.\n[ref3] Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders. CVPR 2025.\n[ref4] PointLLM: Empowering Large Language Models to Understand Point Clouds. ECCV 2024.\n[ref5] GSVA: Generalized Segmentation via Multimodal Large Language Models. CVPR 2024.\n\n**Q3: Concern about relying on Labeled data.**\n\n**A3:** Sorry for this confusion. In fact, the gaze estimation field is **not lacking in labeled data**, e.g., large-scale datasets like ETH-XGaze containing 750K samples. However, the number of face image datasets with gaze estimation is limited (i.e., **the scarcity of annotated datasets**), and each dataset is constrained by its specific devices and data collection scenarios, resulting in **insufficient diversity of labeled data** in terms of subject appearance, head poses, and lighting conditions. Consequently, models trained on these datasets often exhibit performance degradation when deployed in real-world settings (as stated in L27-30). In light of this, the motivation of our work is to **address the limited diversity of current datasets rather than the scarcity of annotated data**. To address this challenge, we curate a diverse collection of unlabeled face images from six publicly available sources, which exhibit wide variability in subject appearance, background environments, image quality, and illumination. \n\nMoreover, in Sec. 4.2, our method achieves SOTA results in both the in-domain and cross-domain gaze estimation **across a wide range of labeled training dataset scales**, ranging from the small-scale MPIIFaceGaze **(10K)** to the large-scale ETH-XGaze **(750K)**. Thus, the assertion that our method relies heavily on labeled datasets during training is unfounded, and our methodology and experimental results remain consistent with our core motivation. Thank you.\n\n**Q4: Distinction between OMNIGAZE and prior work.**\n\n**A4:** We would like to clarify that **our OminGaze demonstrates distinct differences from prior gaze estimation work**:\n- **Learning paradigm**. As outlined in L35-44, unlike previous weakly-supervised and unsupervised methods, we adopt a semi-supervised learning framework that aligns well with the data availability in gaze estimation (see **Q1 Motivation**).\n- **Method framework**. Our reward-inspired architecture introduces new modifications and strategies for effectively leveraging unlabeled data, none of which have been explored in prior gaze estimation works.\n- **Experimental results**. Our model demonstrates superior performance across in-domain, cross-domain, and zero-shot scenarios compared to previous approaches, as evidenced by comprehensive evaluations.\n\n\n**Q5: Comparison with the same backbone.**\n\n**A5:** To solidly address your concern, we present additional results under the in-domain setting that utilizing the similar lightweight ResNet-18 backbone as SOTA methods in Table 2, as follows:\n\n|Method|Backbone|MPIIFaceGaze|EyeDiap|RT-Gene|\n|:-|:-|:-|:-|:-|\n|FullFace|AlexNet|4.93|6.53|10.00|\n|Gaze360|ResNet-18|4.06|5.36|7.06|\n|XGaze|ResNet-50|4.80|6.50|12.00|\n|CANet|ResNet-50|4.27|5.27|8.27|\n|GazeTR|ResNet-18|4.00|5.17|6.55|\n|AGE-Net|Resnet-34|3.61|4.78|-|\n|3DGazeNet|ResNet-18|4.00|-|-|\n|Baseline|ResNet-18|4.48|5.56|7.45|\n|OminGaze (**Ours**)|ResNet-18|**3.46**|**4.37**|**5.89**|\n\nAs seen, our model also **demonstrates better performance than the baseline and SOTA methods (e.g., GazeTR, AGE-Net, and 3DGazeNet) under the similar backbone**. We will add the above results into Sec. 4.2.\n\nMoreover, we would like to respectfully clarify that our motivation for adopting ViT-based backbone is to **explore the upper bound performance of our framework and enable more generalizable gaze estimation in the wild**, not to gain unfair advantage over previous methods. Thanks."}, {"review_id": "VekAFO9M5j", "reviewer": "Reviewer_6xkW", "summary": "This paper proposes a semi-supervised learning framework for 3D gaze estimation, which leverages unlabeled face images from six publicly available sources to achieve generalizable gaze estimation in the wild. It first generates pseudo labels for unlabeled images, and uses a reward model to assess the reliability of these pseudo labels. Experiments are conducted under in-domain and cross-domain settings, and exhibit robust zero-shot generalization performance on four unseen datasets.", "questions": "1)\tThis paper utilizes a large amount of unlabeled face data to assist gaze estimation. This seems somewhat unfair when comparing to existing methods, especially since it only improves performance by less than 1% (as shown in Table 2). This makes it difficult to judge the effectiveness of the proposed method. It seems more like an improvement brought about by data augmentation.\n\n2)\tThis paper trains a teacher model on labeled Gaze360 and ETH-Xgaze datasets, and uses it to assign pseudo gaze labels on unlabeled images, which seems to utilize significantly more data than the labeled datasets used by existing methods in Table 2. Is this comparison similarly unfair?\n\n3)\tThis paper appears to be a combination of existing semi-supervised methods (pseudo labels generation) and MLLM (Multimodal Large Language Model) for more cues, which has already been done in other fields, demonstrating limited innovation.\n\n4)\tIn the right of Figure1, this paper demonstrates that the better generalization ability exists under different settings. Could it be because this paper utilized more data?", "limitations": "The authors should analyze the dependence of the proposed work on unlabeled data types and the impacts of inference speed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1)\tThis paper is well written and organized.\n\n2)\tThis paper achieves extensive experiments on five datasets under both in-domain and cross-domain settings, and exhibits robust zero-shot generalization performance on four unseen datasets.\n\nWeaknesses:\n\n1)\tThis paper appears to be a combination of existing semi-supervised methods (pseudo labels generation) and MLLM (Multimodal Large Language Model) for more cues, which has already been done in other fields, demonstrating limited innovation.\n\n2)\tThis paper demonstrates that the better generalization ability exists under different settings. Could it be because this paper utilized more data?", "was_revised": true, "final_justification": "I have thoroughly reviewed the authors' rebuttal and other reviewers' comments, I concur with Reviewers uapy and JjdM that the methodological innovation appears incremental. I therefore maintain my original rating.", "rebuttal": "We are gratified by the kind words from you, regarding the quality of presentation, and the comprehensiveness of our experiments. Thank you for your insightful review. We provide point-to-point response below.\n\n----\n**Q1: Combination of existing semi-supervised methods and MLLM.**\n\n**A1:** We respectfully disagree that our work is a combination of existing semi-supervised methods and MLLM. Our proposed framework is **well-motivated by the application domain of gaze estimation, with clear articulation of both the motivation and the underlying reasoning**. In addition, our method also introduces concrete technical innovations that go beyond prior efforts. Below, we clarify the key distinctions between our work and existing approaches in terms of the **motivation** and **technical design**.\n\n**1) Motivation**\n\nLimitations of current generalized gaze estimation works: **i)** Weakly supervised methods rely on gaze-related annotations (e.g., social interaction [28,29] or 3D eye geometry [30]), restricting their ability to leverage abundant unlabeled data. **ii)** Unsupervised pretext tasks exhibit weak intrinsic semantic relevance to gaze estimation, resulting in inefficient use of unlabeled face images.\n\nMeanwhile, we observe two important trends about data availability in gaze estimation: **i)** Existing gaze datasets​​ exhibit limited variants such as subject appearance and illumination. **ii)** Large-scale facial images are readily obtainable from public sources. \n\nThese observations motivate us to **develop a semi-supervised gaze estimation framework that leverages pseudo-label generation to fully utilize both labeled and large-scale unlabeled data**. This direction, to the best of our knowledge, has NOT been explored in the gaze estimation community.\n\n**2) Challenge**\n\nIntroducing pseudo-label semi-supervised learning into gaze estimation is **non-trivial** due to the following challenges:\n- **C1**: Existing threshold-based pseudo-labeling methods, tailored for classification tasks, are inapplicable to regression outputs.\n- **C2**: Teacher models trained on limited-diversity datasets generate biased pseudo-labels.\n- **C3**: Robust gaze representation learning demands training data with rich variability to capture inter-individual differences.\n\n**3) Technical design**\n\n**To tackle C1**, We adopt a reward model (inspired by SemiReward [52]) to assess regression-based pseudo-label quality in gaze estimation.  To further enhance reward quality, we propose two key improvements: (1) **3D direction vector pseudo labels** to incorporate geometry awareness. (2) **MLLM-generated scene-specific gaze descriptions** for richer semantic cues, which is inspired by prior work [ref1] in reinforcement learning that utilizes preferences generated by an off-the-shelf LLM as a substitute for human feedback. Different from previous methods using MLLM in other fields, **such MLLM-generated gaze descriptions serve as additional linguistic cues to train the reward model for pseudo-label assessment and selection**. **Notably**, the reward model is removed during testing without any network modification to the base model. These technical designs lead to significant gains of **1.08/0.88/2.92 on MG/ED/RG** (Table 5c). \n\n**To tackle C2**, we adopt **two robust training strategies** to ensure high-quality pseudo-labels for student model training, both yielding substantial performance gains (see Table 5d and S1).\n\n**To tackle C3**, we **curate diverse unlabeled face images from six public sources**, exhibiting wide variability in terms of facial appearance, lighting conditions, and head poses (see L11-39 in Appendix). \n\nIn summary, motivated by the goal of \"**harnessing large-scale unlabeled data to generalize gaze estimation in the wild**\", we propose **a set of principled and effective technical designs that includes dataset construction, model innovation, and training strategies** tailored for this task, which goes far beyond a simple combination of existing semi-supervised approaches and MLLMs. Thus, we believe our work offers valuable **new insights** and **technical contributions** to the field of gaze estimation.\n\n[ref1] Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv 2023.\n\n**Q2: Reason for better generalization.**\n\n**A2:** Sorry for this confusion. **First**, we would like to clarify that we adopt **the same labeled dataset** with other methods under the cross-domain settings (See Sec. 4.2) for a fair comparison. For the in-domain setting, we follow the protocol of the SOTA method GazeTR [54]. To further explore the potential of our OminGaze as a data engine, we extend the zero-shot experiments on our curated training dataset (See Sec. 4.3).\n\n**Second**, to better address your concern, we **apply our curated unlabeled training dataset to existing SOTA methods** (e.g., Gaze360 [13], UniGaze [ref2] and GazeTR [54]) using a self-training strategy under the in-domain setting below for a fair comparison. We also provide more comparisons with methods (e.g., 3DGazeNet [30], UniGaze [ref2], and ST-WSGE [ref3]) that utilize large-scale unlabeled data within an unsupervised or weakly-supervised manner below. Related results will be added in Appendix Sec. B.\n\n|Method|Backbone|MPIIFaceGaze|EyeDiap|Gaze360|\n|:-|:-|:-|:-|:-|\n|3DGazeNet|ResNet-18|4.00|-|9.60|\n|UniGaze|ViT-B|4.75|5.52|9.64|\n|ST-WSGE|ViT-B|6.40|8.20|13.20|\n|Gaze360†|ResNet-18|3.95|5.03|10.57|\n|GazeTR†|ResNet-18|3.55|4.72|10.04|\n|UniGaze†|ViT-B|4.68|5.68|9.61|\n|OminGaze **(Ours)**|**ViT-B**|**2.97**|**4.07**|**9.12**|\n\n \"†\" denotes retraining these methods using our collected training data.\n\nAs seen, **our method outperforms these methods with large-scale unlabeled face data**. This verifies **the effectiveness of our reward model and training strategies**. This is consistent with the findings in Table 5a Line 2-3.\n\nIn summary, **by ruling out the benefit of merely having more training data, we attribute the success of OminGaze to two key factors**: (1) The student model assigns more accurate and reliable pseudo-labels with the assistance of the reward model. (2) The reward model plays a crucial role in evaluating the reliability of these pseudo-labels and influences their contribution to the loss calculation, as detailed in Eq. 9.\n\n[ref2] Unigaze: Towards universal gaze estimation via large-scale pre-training. Arxiv 2025.\n\n[ref3] Enhancing 3d gaze estimation in the wild using weak supervision with gaze following labels. CVPR 2025.\n\n**Q3: Unfair comparison with different data size.**\n\n**A3:** We have provided comparison results with the same training data **in the table of Q2**. The results show that, **for models with similar unlabeled training data, our OminGaze achieves better performance than SOTA methods, verifying the effectiveness of our model design**. To further rule out the benefit of merely adding more data, we had provided ablation studies in Table 5a where we trained both our OminGaze and baseline using the same data. \n\nIn summary, we would like to clarify that utilizing large-scale unlabeled face data via semi-supervised learning in gaze estimation is **non-trivial** due to biased pseudo-labels. In contrast, **our OminGaze designs reward-based training strategy to leverage such unlabeled data effectively**.\n\nMoreover, we **respectfully disagree that OminGaze results are marginal**. **First**, the limited diversity in a single dataset leads to performance saturation among models in the in-domain setting, e.g., only 1.32 gains from FullFace (2017) to AGE-Net (2024), which motivates the subsequent research toward robust gaze estimators in the wild. Despite this near-saturation, **our method achieves consistent improvements across all datasets in the in-domain setting**. **Second**, our OminGaze achieves notable performance in the more challenging cross-domain and zero-shot setting, e.g., **1.86** and **1.88** gains on ETH-XGaze→EyeDiap and Gaze360→MPIIGaze. This is a quite big score for cross-domain gaze estimation. Thanks.\n\n**Q4: Unfair comparison in Table 2.**\n\n**A4:** Sorry for this confusion. **First**, we would like to clarify that in Table 2, we only used the ETH-XGaze dataset along with specific evaluation datasets as the labeled data source, which is identical to some SOTAs, such as GazeTR[54]. We also had provided such data usage details in L246-247. Thus, we believe we render **a fair comparison to existing work**. **Second**, we have presented additional experiments by training competitors on our curated training data **in the table of Q2**. Thanks.\n\n**Q5: Limitations.**\n\n**A5:** *Unlabeled data types*: Thanks for your suggestion. We agree that analyzing the impact of unlabeled data types is an important direction. However, due to the lack of explicit type annotations such as illumination, and eye occlusion in existing datasets, we try to coarsely categorize the unlabeled data into three broad groups: (1) Expert-curated datasets (CelebA, VGGFace2), (2) Synthetic datasets (FaceSynthetics, SFHQ-T2I), and (3) Web-crawled datasets (WebFace, VFHQ). Then we control each dataset size and evaluate the effect of each data type. Due to limited rebuttal period, we will include the results in the final version. Moreover, we have provided Figure S2 in Appendix examines the effect of unlabeled data size. \n\n*Inference speed*: Sorry for this confusion. Our semi-supervised learning framework OminGaze only introduces the reward model during training for pseudo-label assessment and selection. Note that **we discard the reward model during testing without any network modification or extra inference cost introduced to the base model**. Below is the inference time comparison between our OminGaze and baseline with different backbone. We will add the above discussion and results into the Appendix Sec. B.\n\n|Method|Backbone|Inference speed|\n|:-|:-|:-|\n|Baseline|Resnet-18|2.7ms|\n|Ours|Resnet-18|**2.7ms**|\n|Baseline|ViT-B|10.9ms|\n|Ours|ViT-B|**10.9ms**|"}, {"review_id": "vf64GCuEUY", "reviewer": "Reviewer_JjdM", "summary": "The paper present a method for improved gaze estimation in unconstrained real-world environments, by utilizing large-scale unlabeled face images and MLLMs to extent the data diversity of existing public gaze datasets. In particular the authors adopt a pseudo-labeling strategy using a teacher model trained with ground truth gaze labels and devise a reward model to assess the reliability of pseudo labels, while pseudo labels are extracted by the teacher model. Then, a generalized student model is trained on both ground truth and pseudo-labeled data, while the pseudo-labels and the reward model are also updated during the training process. The proposed method, OmniGaze demonstrate improved performance on within-dataset and cross-dataset experiments, while the effectiveness of the method as a zero-shot gaze predictor is also evaluated.", "questions": "Questions:\n- From my understanding, the whole pipeline is trained end-to-end, including the reward model. What is the motivation behind it? Have you tried pre-training the reward model?\n\n- In lines 192-195 you mention that an additional similarity score between the student output and the pseudo-labels is employed together with the output of the reward model. How crucial is this addition to the effectiveness of the reward strategy?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a novel method based on MLLMs to harness the vast distribution of unlabeled face data for gaze estimation. The authors propose a reward strategy to filter pseudo-labelled data and employ the high-quality ones for training, which is interesting and makes use of the extensive representation power of recent linguistic MLLMs.\n\n- The method demonstrates improvements over previous SOTA in cross-domain gaze estimation.\n\n- The paper is well written and the concepts easily-understood. \n\nWeaknesses:\n- The size of the encoder used by OmniGaze  is quite larger compared to most of the compared methods, e.g Gaze360[13], 3DGazeNet[30]. A comparison with the model sizes, or with an encoder similar to these methods, would offer better understanding of the effect of the proposed training strategy.\n\n- It should be highlighted that in the within-dataset and cross-dataset experiments none of the compared methods employ the data used by OmniGaze. Therefore, it would be beneficial if the authors could provide comparisons with methods that utilize the Omnigaze data (unlabeled) within an unsupervised or semi-supervised scheme. For example MTGLS [*] uses pseudo-labels from unconstrained face images to improve gaze estimation.\n[*] Ghosh S, Hayat M, Dhall A, Knibbe J. MTGLS: Multi-task gaze estimation with limited supervision. In Proceedings of the IEEE/CVF winter conference on applications of computer vision 2022 (pp. 3223-3234).\n\n- Cross-domain results include only experiments in which the ground truth datasets include large diversity in gaze directions and head poses. As the method is presented as generalized gaze estimation, experiments with more diverse test datasets should have been included. For example, how does the large-scale unlabeled data corpus improves accuracy in cases where the ground truth data are more limited (e.g. MPIIFaceGaze) and the test set is large (e.g Gaze360)? This is an important comparison missing from the current work but included in similar works such as [28] and [30].", "was_revised": true, "final_justification": "I thank the reviewers for their rebuttal. After reading their response I am happy to raise my rating to accept. The authors have provided quantitative evaluations and details about their method which have eliminated my initial concerns.", "rebuttal": "We sincerely thank the reviewer for the positive feedback and great suggestions! We are glad that you found our work to have novel methodological contributions and a clear presentation. We provide point-to-point response below.\n\n----\n\n**Q1: Comparison with the similar backbone.**\n\n**A1:** Great suggestion! For a fair comparison, we provide the comparison between our OminGaze with Resnet-18 backbone and SOTA methods under the in-domain setting below.\n\n|Method|Backbone|MPIIFaceGaze|EyeDiap|RT-Gene|\n|:-|:-|:-|:-|:-|\n|FullFace|AlexNet|4.93|6.53|10.00|\n|Gaze360|ResNet-18|4.06|5.36|7.06|\n|XGaze|ResNet-50|4.80|6.50|12.00|\n|CANet|ResNet-50|4.27|5.27|8.27|\n|GazeTR|ResNet-18|4.00|5.17|6.55|\n|AGE-Net|Resnet-34|3.61|4.78|-|\n|3DGazeNet|ResNet-18|4.00|-|-|\n|Baseline|ResNet-18|4.48|5.56|7.45|\n|OminGaze **(Ours)**|ResNet-18|**3.46**|**4.37**|**5.89**|\n\nAs seen, **for models with similar backbones, OminGaze achieves better performance than the baseline and SOTA methods** (e.g., GazeTR, AGE-Net, and 3DGazeNet). Thus, the above results along with Table 5a show that our proposed training strategy is effective with different backbones (ViT-B and Resnet-18). The above results will be incorporated into Sec. 4.2 for clarity.\n\n**Q2: Comparison with methods that utilize unlabeled data.**\n\n**A2:** Thanks for the suggestion. **First**, we would like to clarify that **our compared 3DGazeNet[ref1] in Table 2 also utilize large-scale unlabeled face data, but most of the compared methods do not leverage unlabeled data**. Thus, we will explicitly highlight this point in Sec. 4.2 to avoid potential misunderstandings. **Second**, following your suggestion, we provide more comparisons with methods (e.g., 3DGazeNet[ref1], UniGaze[ref2], ST-WSGE[ref3], and MTGLS [ref6]) that utilize large-scale unlabeled data within an unsupervised or weakly-supervised manner below. In addition, we also retrain existing SOTA methods (e.g., Gaze360[ref4], UniGaze[ref2] and GazeTR[ref5]) using our curated training dataset. The detailed results are summarized below.\n\n|Method|Backbone|MPIIFaceGaze|EyeDiap|Gaze360|\n|:-|:-|:-|:-|:-|\n|3DGazeNet|ResNet-18|4.00|-|9.60|\n|MTGLS|ResNet-50|4.07|-|12.83|\n|UniGaze|ViT-B|4.75|5.52|9.64|\n|ST-WSGE|ViT-B|6.40|8.20|13.20|\n|Gaze360†|ResNet-18|3.95|5.03|10.57|\n|GazeTR†|ResNet-18|3.55|4.72|10.04|\n|UniGaze†|ViT-B|4.68|5.68|9.61|\n|OminGaze **(Ours)**|**ViT-B**|**2.97**|**4.07**|**9.12**|\n\n† denotes retraining these SOTA methods using our collected unlabeled data.\n\nAs seen, **compared with these methods using large-scale unlabeled face data, OminGaze still achieves better performance**. This verifies the effectiveness of our model design and training strategy. Related discussion and statistics will be added in Appendix Sec. B. Thanks.\n\n[ref1] 3dgazenet: Generalizing 3d gaze estimation with weak-supervision from synthetic views. ECCV 2024.\n\n[ref2] Unigaze: Towards universal gaze estimation via large-scale pre-training. Arxiv 2025.\n\n[ref3] Enhancing 3d gaze estimation in the wild using weak supervision with gaze following labels. CVPR 2025.\n\n[ref4] Gaze360: Physically unconstrained gaze estimation in the wild. ICCV 2019.\n\n[ref5] Gaze estimation using transformer. ICPR 2022.\n\n[ref6] Mtgls: Multi-task gaze estimation with limited supervision. WACV 2022.\n\n**Q3: More cross-domain results.**\n\n**A3:** Good comment! Following your suggestion, we train our OminGaze on ground truth datasets that are more limited (*e.g.*, MPIIFaceGaze or GazeCapture), and test our model on testing datasets that have large diversity (*e.g.*, Gaze360). The comparison results with LAEO[ref7] and 3DGazeNet[ref8] are as follows:\n\n|Method|MPIIFaceGaze $→$ Gaze360|GazeCapture $→$ Gaze360|\n|:-|:-|:-|\n|LAEO|-|27.2|\n|3DGazeNet|17.6|17.6|\n|OminGaze **(Ours)**|**13.8**|**14.2**|\n\nAs seen, **our OminGaze also demonstrates better performance under such settings**. This verifies the effectiveness of our model as a generalized gaze estimator. We will incorporate the above results into Appendix Sec. B. Thanks.\n\n[ref7] Weakly supervised physically unconstrained gaze estimation. CVPR 2021.\n\n[ref8] 3dgazenet: Generalizing 3d gaze estimation with weak-supervision from synthetic views. ECCV 2024.\n\n**Q4: Motivation for training reward model online.**\n\n**A4:** Apologize for any confusion. As stated in L220-223, we adopt a periodic pseudo-label update strategy, where the frozen teacher model's parameters are periodically refreshed with the student model's weights every K epochs to regenerate pseudo-labels. Therefore, **a fixed pre-trained reward model would fail to distinguish increasingly high-quality pseudo labels during training**. To improve clarity, the following updates have been made to the manuscript:\n\n> L220-223: To ensure training stability and robustness, we adopt a periodic pseudo-label update strategy, where the frozen teacher model's parameters are periodically refreshed with the student model's weights every *K* epochs to regenerate pseudo-labels (ablation study in Table S1 of Appendix). This interval mitigates the risk of early-stage overfitting to noisy labels while allowing the student model to progressively benefit from improved predictions over time.\n\nIn addition, we also **pre-train the reward model by jointly using the labeled and unlabeled datasets**. We then evaluate the performance of our model with the pre-trained reward model under the zero-shot setting. The results are as follows:\n\n|Reward model training|MPIIFaceGaze|EyeDiap|RTGene|\n|:-|:-|:-|:-|\n|Pre training|3.71|5.03|10.24|\n|Online training **(Ours)**|**3.44**|**4.31**|**9.09**|\n\nAs seen, the performance of pre-trained reward model is inferior to that of the online-trained counterpart. We hypothesize that this is because the pre-trained model tends to overfit to the initial distribution of pseudo labels, and thus struggles to assess the reliability of pseudo labels at different training stages. The results above will be integrated into the Appendix Sec. B. Thanks.\n\n\n**Q5: Cosine similarity scores in Eq.7.**\n\n**A5:** Good question! We incorporate cosine similarity scores between the student output and pseudo-labels as auxiliary information to enhance the reliability of the reward scores. This design is motivated by the observation that cosine similarity reflects the directional agreement between predicted and pseudo gaze vectors, which is particularly meaningful in gaze estimation [ref8, ref9]. By integrating it into the final reward scores, we ensure more stable pseudo-label assessment. Notably, even without this cosine similarity scores, our reward-based training strategy still achieves strong results, but the introduction of cosine similarity scores provides further reward stability and performance gains. The effectiveness of this strategy had been empirically validated through ablation studies below in Table 5b. \n\n|Confidence Score|MPIIFaceGaze|EyeDiap|RTGene|\n|:-|:-|:-|:-|\n|*w/o* reward|4.97|5.42|13.75|\n| $\\hat{r}_k$ (Eq. 6)|3.71|4.68|9.96|\n| $r_k$ (Eq. 7) **(Ours)** |**3.44**|**4.31**|**9.09**|\n\n\nAs seen, by adding the similarity scores into final confidence scores (Eq. 7), our model results in a slight performance increase (e.g., 9.96 → 9.09 on RTGene). We will include this discussion in the revised manuscript.\n\n[ref8] Lg-gaze: Learning geometry-aware continuous prompts for language-guided gaze estimation. ECCV 2024.\n\n[ref9] CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model. AAAI 2024."}]

Author rebuttal:

Meta-review: The paper introduces OmniGaze, a semi-supervised framework for 3D gaze estimation. Initial reviewer concerns centered on the incremental nature of the methods and the limited analysis of the model and dataset effects.

In their rebuttal, the authors provided clear clarifications that addressed these points, convincing most reviewers of the work’s contribution. Reviewer 6xkW maintained concerns regarding the incremental solutions and unclear motivations, but while the authors supplied further responses, the reviewer did not follow up. After reviewing the conversation exchange, the AC concluded that the authors’ responses adequately addressed the concerns.

Given that three of the four reviewers recognized the merit of the work, and in light of the significant performance improvements demonstrated across diverse datasets, the AC recommends acceptance (poster), as the proposed gaze estimation method is an important computer vision tool with potential real-world impact.
Decision: Accept (poster)