Title: OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Gaze Estimation

In-The-Wild Generalization

Semi-supervised Learning
Abstract: Current 3D gaze estimation methods struggle to generalize across diverse data domains, primarily due to $\textbf{i)}$ $\textit{the scarcity of annotated datasets}$, and $\textbf{ii)}$ $\textit{the insufficient diversity of labeled data}$. In this work, we present OmniGaze, a semi-supervised framework for 3D gaze estimation, which utilizes large-scale unlabeled data collected from diverse and unconstrained real-world environments to mitigate domain bias and generalize gaze estimation in the wild. First, we build a diverse collection of unlabeled facial images, varying in facial appearances, background environments, illumination conditions, head poses, and eye occlusions. In order to leverage unlabeled data spanning a broader distribution, OmniGaze adopts a standard pseudo-labeling strategy and devises a reward model to assess the reliability of pseudo labels. Beyond pseudo labels as 3D direction vectors, the reward model also incorporates visual embeddings extracted by an off-the-shelf visual encoder and semantic cues from gaze perspective generated by prompting a Multimodal Large Language Model to compute confidence scores. Then, these scores are utilized to select high-quality pseudo labels and weight them for loss computation. Extensive experiments demonstrate that OmniGaze achieves state-of-the-art performance on five datasets under both in-domain and cross-domain settings. Furthermore, we also evaluate the efficacy of OmniGaze as a scalable data engine for gaze estimation, which exhibits robust zero-shot generalization on four unseen datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper present a method for improved gaze estimation in unconstrained real-world environments, by utilizing large-scale unlabeled face images and MLLMs to extent the data diversity of existing public gaze datasets. In particular the authors adopt a pseudo-labeling strategy using a teacher model trained with ground truth gaze labels and devise a reward model to assess the reliability of pseudo labels, while pseudo labels are extracted by the teacher model. Then, a generalized student model is trained on both ground truth and pseudo-labeled data, while the pseudo-labels and the reward model are also updated during the training process. The proposed method, OmniGaze demonstrate improved performance on within-dataset and cross-dataset experiments, while the effectiveness of the method as a zero-shot gaze predictor is also evaluated.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper proposes a novel method based on MLLMs to harness the vast distribution of unlabeled face data for gaze estimation. The authors propose a reward strategy to filter pseudo-labelled data and employ the high-quality ones for training, which is interesting and makes use of the extensive representation power of recent linguistic MLLMs.\n\n- The method demonstrates improvements over previous SOTA in cross-domain gaze estimation.\n\n- The paper is well written and the concepts easily-understood. \n\nWeaknesses:\n- The size of the encoder used by OmniGaze  is quite larger compared to most of the compared methods, e.g Gaze360[13], 3DGazeNet[30]. A comparison with the model sizes, or with an encoder similar to these methods, would offer better understanding of the effect of the proposed training strategy.\n\n- It should be highlighted that in the within-dataset and cross-dataset experiments none of the compared methods employ the data used by OmniGaze. Therefore, it would be beneficial if the authors could provide comparisons with methods that utilize the Omnigaze data (unlabeled) within an unsupervised or semi-supervised scheme. For example MTGLS [*] uses pseudo-labels from unconstrained face images to improve gaze estimation.\n[*] Ghosh S, Hayat M, Dhall A, Knibbe J. MTGLS: Multi-task gaze estimation with limited supervision. In Proceedings of the IEEE/CVF winter conference on applications of computer vision 2022 (pp. 3223-3234).\n\n- Cross-domain results include only experiments in which the ground truth datasets include large diversity in gaze directions and head poses. As the method is presented as generalized gaze estimation, experiments with more diverse test datasets should have been included. For example, how does the large-scale unlabeled data corpus improves accuracy in cases where the ground truth data are more limited (e.g. MPIIFaceGaze) and the test set is large (e.g Gaze360)? This is an important comparison missing from the current work but included in similar works such as [28] and [30].\n\nQUESTIONS:\nQuestions:\n- From my understanding, the whole pipeline is trained end-to-end, including the reward model. What is the motivation behind it? Have you tried pre-training the reward model?\n\n- In lines 192-195 you mention that an additional similarity score between the student output and the pseudo-labels is employed together with the output of the reward model. How crucial is this addition to the effectiveness of the reward strategy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a semi-supervised learning framework for 3D gaze estimation, which leverages unlabeled face images from six publicly available sources to achieve generalizable gaze estimation in the wild. It first generates pseudo labels for unlabeled images, and uses a reward model to assess the reliability of these pseudo labels. Experiments are conducted under in-domain and cross-domain settings, and exhibit robust zero-shot generalization performance on four unseen datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1)\tThis paper is well written and organized.\n\n2)\tThis paper achieves extensive experiments on five datasets under both in-domain and cross-domain settings, and exhibits robust zero-shot generalization performance on four unseen datasets.\n\nWeaknesses:\n\n1)\tThis paper appears to be a combination of existing semi-supervised methods (pseudo labels generation) and MLLM (Multimodal Large Language Model) for more cues, which has already been done in other fields, demonstrating limited innovation.\n\n2)\tThis paper demonstrates that the better generalization ability exists under different settings. Could it be because this paper utilized more data?\n\nQUESTIONS:\n1)\tThis paper utilizes a large amount of unlabeled face data to assist gaze estimation. This seems somewhat unfair when comparing to existing methods, especially since it only improves performance by less than 1% (as shown in Table 2). This makes it difficult to judge the effectiveness of the proposed method. It seems more like an improvement brought about by data augmentation.\n\n2)\tThis paper trains a teacher model on labeled Gaze360 and ETH-Xgaze datasets, and uses it to assign pseudo gaze labels on unlabeled images, which seems to utilize significantly more data than the labeled datasets used by existing methods in Table 2. Is this comparison similarly unfair?\n\n3)\tThis paper appears to be a combination of existing semi-supervised methods (pseudo labels generation) and MLLM (Multimodal Large Language Model) for more cues, which has already been done in other fields, demonstrating limited innovation.\n\n4)\tIn the right of Figure1, this paper demonstrates that the better generalization ability exists under different settings. Could it be because this paper utilized more data?\n\nLIMITATIONS:\nThe authors should analyze the dependence of the proposed work on unlabeled data types and the impacts of inference speed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting gaze estimation models face two main challenges: (1) a lack of large annotated datasets, and (2) limited diversity in labeled data. To address these issues, this paper proposes a semi-supervised framework called OMNIGAZE for 3D gaze estimation, which leverages large-scale unlabeled data. OMNIGAZE employs pseudo-labeling and introduces a reward model to evaluate the reliability of pseudo labels. This reward model incorporates visual embeddings and a multimodal large language model (LLM) to extract semantic cues. Extensive experimental results demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: In gaze estimation, there is a lack of both unlabeled data and diversity in existing datasets. Current methods often rely on self-supervised or weakly supervised learning to address this issue, but the results remain unsatisfactory. This paper introduces a reward-based strategy utilizing a large language model (LLM) to overcome these limitations. A teacher model, trained on labeled data and enhanced with LLM guidance, generates pseudo-labels—though leveraging these pseudo-labels effectively remains challenging. To address this, a reward model assigns confidence scores to refine the learning process. With iterative training, the model's performance improves significantly. Extensive experiments demonstrate the effectiveness of the proposed strategy.\nWeaknesses: 1. First, I have serious concerns about the novelty of the proposed method. The reward-based framework closely resembles existing approaches such as “SemiReward,” and the paper does not clearly articulate what sets it apart.\n2. Second, the performance improvements seem largely attributable to the use of a large language model (LLM) for calibration. As we know, LLMs like CLIP are pretrained on massive datasets and are inherently powerful. Therefore, relying on them raises further doubts about the novelty of the contribution.\n3. Although the paper aims to address the scarcity of annotated data, it appears to rely heavily on labeled datasets during training. This contradicts its core motivation, and I do not see a clear distinction between OMNIGAZE and prior work.\n4. Finally, the use of a ViT-based architecture in training raises concerns about fairness in comparison. Most existing methods use lightweight models such as AlexNet or ResNet-18, making the reported improvements potentially misleading.\nBased on the above points, I am unable to recommend this paper for acceptance.\n\nQUESTIONS:\n1. I would like to see analysis for novelty.\n2. the use of a ViT-based architecture in training raises concerns about fairness in comparison. Most existing methods use lightweight models such as AlexNet or ResNet-18. I would like to see a fair comparison.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission presents a scheme to scale the training of 3D gaze direction estimation models such as to best take advantage of both labeled data as well as unlabeled face image data. The proposed OmniGaze method adopts a pseudo-labeling scheme, which is not only simple to adopt but is also proven to be effective in object recognition tasks in particular. The pseudo-labeling scheme of OmniGaze is unique in that an independent reward model is designed to assess the utility of the pseudo-labels, in a progressive manner. The reward model leverages an InstructBLIP VLM to generate a textual description of gaze, then compares this description with the pseudo-label as well as CLIP-encoded image features to assess the reliability of the pseudo-label. Furthermore, these confidence values are leveraged during student model training via a filtering and weighting approach. OmniGaze shows significant performance improvements in in-domain and cross-domain gaze estimation, out-performs self-supervised baselines in a \"zero-shot\" setting, and each of its design decisions are tested in a comprehensive ablation study.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper proposes many clever ideas to allow large-scale training of gaze estimation models by selectively distilling information from labeled datasets such as Gaze360 and ETH-XGaze. I am generally impressed by the conceptual simplicity of the proposed solution, the clever use of InstructBLIP to semi-objectively assess pseudo-label reliability, the practical use of cosine-similarity to subjectively assess pseudo-label reliability, and the sensible use of filtering (instead of loss weighting only) and periodic pseudo-label update strategy to allow more effective learning over time. The results speak for themselves and the reader finds oneself nodding in agreement with the authors. This paper was a true pleasure to read.\n\nCertain results are too-good to be believable, for instance the values in Table 2 show 2.97-deg for MPI, 4.07-deg for EyeDiap - which are significant improvements compared to the state-of-the-art. While I do not doubt the authors' integrity, a brief inspection (especially into MPIIFaceGaze) would be appreciated. Did previous methods have particular failure groups that OmniGaze happens to avoid by leveraging the prior knowledge of InstructBLIP?\n\nQUESTIONS:\nI do not have any specific questions that the authors should answer in a rebuttal.\n\nLIMITATIONS:\nThe authors do not discuss the limitations of their proposed work nor the potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}