Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
Authors: Anupam Pani
Yanchao Yang
Venue: NeurIPS 2025 poster ()
Keywords: VLM
Eye-gaze
Primary area: deep_learning
TL;DR: Regularize the model attention in a transformer (VLM) using human gaze to enhance egocentric activity prediction and understanding.

Abstract: Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it a powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks: fine-grained future event prediction and current activity understanding.
Unlike prior approaches that rely solely on visual inputs or use gaze as an auxiliary input signal , our method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing it to generalize across multiple VLM architectures that utilize attention.
Experimental results show that our approach improves semantic prediction scores by up to 11$\%$ for future event prediction and around 7$\%$  for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided training in improving the accuracy and robustness of egocentric VLMs. Overall, this work establishes a foundation for using human gaze to enhance the predictive capabilities of VLMs in real-world scenarios like assistive robots and human-machine collaboration.
Code and additional information is available at: https://github.com/anupampani/Gaze-VLM

Gaze- VLM: Bridging Gaze and VLMs via Attention Regularization for Egocentric Understanding
Anupam Pani
Yanchao Yang!
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong Department of Electrical and Electronic Engineering, The University of Hong Kong apani3@connect. hku. h yanchaoy@hku. hk
Abstract
Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocen- tric understanding tasks: fine- ome-grained future event prediction and current activity understanding. Unlike prior approaches that rely solely on visual inputs or use gaze as auxiliary input signal, method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing across multiple VLM architectures that utilize attention. Experimental results show that our approach improves semantic prediction scores by up to 1 11% for future event prediction and around 7% for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided training in improving the accuracy and robustness of egocentric VLMs. Overall, this work establishes foundation for using human gaze to enhance the predictive capabilities of VLMMs in real-world sce- narios like assistive robots and numan-maching collaboration. Code and additional information is available at: https: tiub......... com/ anupampani/Caze-We
generalize
Introduction
Vision-Language Models (VLMs) are foundation models that jointly process visual and textual inputs to understand and generate multi-modal information. Foundational works such as ViLBERT, LXMERT, and CLIP have demonstrated their versatility across tasks like image captioning, visual question answering, and multi-mode retrieval (Lu et al. 2019a Tan and Bansal 2019 Radford et al. 2021). Beyond these benchmarks, VLMs hold great promise for real-world applications requiring human-machine collaboration, such assistive robotics (Li al. 2024), accessibility tools (Zhao jet al. 2024b), and autonomous driving (Zhou et al.) (2024)). In such domains, temporally grounded tasks, like future action prediction and activity understanding, are critical. Accurate short-term predictions enable systems to anticipate user needs and provide timely, context-aware assistance.
While coarse activity predictions (e.g., "brewing coffee") provide general context, fine-grained descriptions (c.g., "reaching for the coffee capsule in the top-right cabinet") offer actionable detail (Goyal and Durrett 2021). Achieving this level of understanding requires models reason about short-term goals as well as visual focus. We posit that eye gaze is a powerful supervisory signal in this setting: it reflects a person's 's attention, often precedes interaction, and encodes both spatial and temporal cues (Frischen et al. 2007 Tipper 2010).
To leverage these insights, we introduce gaze-regularized framework that uses gaze only during training to guide attention, enabling deployment with standard image frames. Our method incorpo- rates a gaze-regularized attention block after the visual encoder, aligning attention maps with gaze
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

distributions from spatial heatmaps using a KL divergence loss. The framework is modular and compatible with various VLMs which use transformer-basee architectures.
We validate our approach using several open-source VLM backbones on an egocentric dataset annotated with gaze and fine-grained captions. Our model improves future action prediction by 11% and activity understanding by 7%, compared to baselines trained without gaze. To summarize, our contributions are: 1) A modular gaze-regularized VLM framework that improves egocentric activity understanding and generalizes across multiple architectures; 2) A gaze-guided attention mechanism that aligns model attention with human gaze during training: 3) An occlusion-awand -waree filtering strategy that improves the reliability of aggregated spatial heatmaps; 4) A comprehensive evaluation across tasks and models, demonstrating strong performance even without gaze inference.
2 Related Work
Attention mechanisms have become fundamental in vision tasks for identifying salient features, with gaze information serving valuable cue in egocentric settings. Prior research has leveraged gaze for tasks such as next-active object prediction (Thakur et al. 2023) and task-relevant information extraction (Hayhoe al. 2003). Recent mansformer-basee approaches, including global-local transformers for gaze prediction (Lai et al. 2024) and joint gaze-action modeling (Huang et al. 2020).
have demonstrated the effectiveness of capturing gaze dynamics.
The idea of leveraging human gaze to guide computational models is well established. Min and Corso (2020) integrated gaze into attention mechanism for egocentric activity recognition. They modeled gaze distributions using a variational autoencoder and used to modulate features, demonstrating that gaze is powerful signal even when not available at test time. A key distinction lies in the use of optical flow: their two-stream architecture requires flow as a model input during both training and inference, whereas we use flow solely during pre-processing for occlusion-aware gaze filtering, making it unnecessary at test time. While sharing the high-level goal of using gaze as a training- time signal, our work focuses on large-scale VLMs for generating fine-grained textual descriptions. Furthermore, our method presents a simple, modular attention regularization loss that can be plugged into various transformer- baseed VLMs.
There has been growing interest in aligning VLM behavior with human attention. For instance, Voila-A (Yan et al.) 2023) aligns model cross-attention with user gaze during training and inference to steer predictions for static image tasks, requiring gaze input at test time. contrast, our method uses gaze only during training, making it applicable to standard VLM inference. Other works have explored generating attention maps from human-object interactions (Zhou et al. 2024b) or, in the context of static image classification, using teacher models within contrastive learning frameworks to produce spatial-attention labels for improved robustness (Yao et al. (023). Our work situated in the dynamic, temporally groundedd setting of egocentric videos, where introduce pipeline for aggregating and filtering raw gaze signals to create robust supervisory heatmaps for regularizing VLM attention during training.
We extend this research by introducing gaze-regularized attention mechanism that integrates human gaze into VLMSs. Rather than predicting gaze, our method uses it during training to shape model attention, aligning computational focus with human viewing behavior. This approach embeds biologically inspired visual priors directly into the attention structure of VLMs, extending gaze applications beyond recognition to multi-modal reasoning and prediction.
Activity understanding and prediction are central to egocentric behavior modeling, with applications in assistive robotics and wearable AI. Recent advances have evolved from LSTM- --se- methods (Furnari and Farinella 2019) to hierarchical, intention coniiiiionoddiiiedddcccoooe designs (Zhao et al.) 2024a Mascaro et al. 2024) and transformer-basee et al. 22024). Building on these foundations, our work incorporates gaze-based attention regularization to enhance context-aware prediction of both current and future activities in egocentric video.
models that capture fine-grained spatiotemporal dependencies (Roy
3 Method
In this section, we present our approach for enhanced training of VLMS by incorporating human eye gaze data for egocentric activity understanding and future action prediction.

---

Input Seit.
Apgreganed Feetures
Visuaa Tokeess
Vision Encoder
Transformer Encoder
Language Decoder
goilng place the Bosue.
Training with Gaze Regularization
Heatmap
Divargence
Attantion
Regularized Attention Over Time
lessellation
K
Overview of the Proposed Scheme. During inference (top), egocentric frames are processed by vision encoder to extract visual tokens, which flow through transformer encoder to produce aggregated features that are decoded into descriptive text. During training (bottom), we add gaze regularization: human gaze heatmaps are converted to patch-wise distributions matching the transformer's 's attention granularity. The model's attention distribution is then guided toward human visual focus patterns through KL-divergence minimization, enhancing feature aggregation without requiring gaze data during inference.
Figure 1:
Our method builds upon the well-established observation that human gaze patterns reveal crucial information about attention allocation and intention. Rather than treating gaze as merely an addi- tional input feature, we employ it principled regularization signal during training to guide the transformer's ' attention mechanisms. This approach effectively aligns the model's s computational attention with human visual focus, creating more biologically plausible processing mechanism that leverages the correlation between gaze patterns and intentional behavior.
Figure provides an overview of the proposed training pipeline. The system processes egocentric video frames through vision encoder to extract visual features, which are then modulated through our gaze-regularized attention blocks. During training, the attention mechanism is explicitly regularized to align with human gaze patterns using Kullback-Leibler divergence. The resulting attention- modulated features enable the model to generate descriptive text of current activities or predictions of forthcoming actions, while requiring only standard visual input during inference.
3.1 Gaze Representation with Temporal Aggregation
We construct gaze supervision signals for the regularizer by first transforming individual gaze points spatially and then aggregating them temporally with occlusion handling. The gaze points in the dataset are originally represented in text form, and construct the gaze supervision signals from the coordinates by first transforming them into spatial heatmaps.
Spatial Transformation For each gaze point g: (o.'9) E [1...., w| x [1...., h] at time f, we first generate a spatial heatmap m, Ighxw through Gaussian smoothing:
m, (G, (g:)),
where (gg) is an indicator function with value at position 9r and elsewhere, G, is a Gaussian kernel with standard deviation denotes convolution, and represents normalization so the heatmap sums to This process transforms discrete gaze coordinates into 2D probability distributions that reffect the spatial allocation of visual attention.
Temporal Aggregation with Occlusion Handling Human visual attention involves fixations lasting approximately 200ms (Rayner) (2099), interspersed with rapid saccadic movements. Since saccades reflect transitional eye movements rather than focused attention, relying on individual gaze points, particularly those captured during saccades, can introduce noise into the supervision signal. To mitigate this, we propose aggregating spatially transformed gaze maps over temporal window. This temporal integration captures the stable structure of visual attention and yiclds more informative and robust supervisory signal for training. This aggregation over a temporal window is done via:
H of
Σ or (fr+t m.)) T=1-8
(
3

---

where defines the temporal window (e.g.. 200ms) and fr is a warping function (e.g.. optical flow) computed from RGB frames that represents pixel motion from time to t, and is used transport m, to the current frame at time L. Furthermore, considering that point in the scene fixated upon may become occluded due to camera movement (e.g., out of frame) or a change in the environment (e.g., dynamic objects), including such occluded points would create misleading supervision signals. Therefore, we perform an oc- clusion check using bidirec- tional optical flow consistency between frames (Hur and Roth, 2017), resulting in o, that indi- cates the validity of gaze points to facilitate the temporal aggre- gation. If the occlusion is signif- icant and major, the gaze point related to the image frame is considered invalid and is not used for the temporal aggrega- tion. The above procedure en- sures that our supervision signal only incorporates gaze directed at visible elements at the current time step. Please refer to Figure 2 for an illustration this process. The resulting aggregated heatmaps (H₃) serve as robust supervision targets for the proposed attention regularization framework (more details are provided in Section A.3)
f F (Bachwars)
Aggregated Heatmoo
12
DAR aneeserrrtes ctecs
X
Sr (Marp to Frome 0
Recorded Gaze Paints
S (Forward)
Figure 2: Temporal Aggregation of Gaze. Top: Input egocentric frames. Bottom: Gaze points transformed into heatmaps. filtered using bidirectional optical flow (fr-+t, f(-r) to remove occluded points, then warped and aggregated into temporally coherent supervision signal.
3.2 Problem Formulation
We address egocentric behavior understanding through vision-language approach that enables both interpreting current activities and anticipating future ones. These capabilities are essential for applications like assistive technologies and human-robot collaboration, where machines must comprehend ongoing human actions and predict subsequent behavior.
Given sequence of egocentric video frames {4t] spanning an observation window of To seconds, our model generates textual descriptions that capture human activities. This framework supports two complementary tasks:
Activity Understanding: Generating descriptions of activities occurring within the observation window To seconds.
Future Activity Prediction: Generating descriptions of activities likely occur in the upcoming seconds based on the observed video frames.
Formally, our model learns to generate appropriate textual descriptions conditioned on the observed image sequence:
<(144)))) = p((4))))
(3)
where l represents either activity descriptions or future activity predictions, depending on the task.
A key aspect of our approach is that the model operates standard visual inputs (RGB frames) during both training and inference. Gaze information is used exclusively during training as a regularization signal to guide the model's attention mechanism, not input feature. This design ensures practical deployability in scenarios where eye-tracking data might be unavailable during inference.
3.3 Graze-Regularizer Attention Mechanism
Transformer- VLMS typically process visual inputs through three main stages: feature extraction via a vision encoder, feature refinement through attention mechanisms, and text generation via language decoding. Our approach targets the attention mechanisms specifically, leaving the vision encoder and language decoding components largely unchanged.
Most VLMs extract visual features Φ {V1, Vi...., .... sin) from input frames using pre-trained encoders (such as VIT), where each v/s E Rd represents a feature vector (token) corresponding to an image patch. These features are then used to compute the keys and values the attention module. For the query, derive global query across the input sequence, capturing the overall scene context and

---

activity information, rather than using frame- frame-specific queries that risks being overfitted to the image photometrics. The global query attends over all spatial patches within an imput image to compute attention weights. These weights are later compared against human gaze patterns via the gaze-based regularization. By introducing gaze regularization to the attention computation, we guide the model to focus on regions that humans naturally attend to when performing activities or interactions. Figure illustrates the training pipeline of the proposed gaze-regularized VLMs as well as the regularization process, which shows how the model's attention distribution gets more aligned with the human gaze distribution over time. Next, we elaborate on the details.
Attention Computation Our approach utilizes standard attention mechanisms found in transformer architectures. In these implementations, attention is computed as:
QKT VTe
Attention(Q2. Q, V) softmax
V AV.
where Q, K, and V are query, key, and value derived from visual features, d, is the dimension of the key vectors, and A represents the attention weights. As mentioned previously, we utilize global query obtained from the input sequence, whereas the keys and values are derived from individual frames. Using the gaze-based regularization, which is introduced next, we aim to align the model's attention weights more closely with human gaze patterns.
Gaze Regularization Before aligning the model's 's attention with human visual focus, we need to transform the pixel-wise gaze heatmap He into patch-wise distribution that matches the granularity of transformer attention. This transformation necessary because vision transformers operate on image patches rather than individual pixels, and the attention mechanism computes weights with keys from image patches.
We thus divide the domain Ω of the (aggregated) heatmap H, into a grid of P patches that correspond the same spatial partition used by the vision encoder (e.g., UP 52 and p; np = 0), and compute the gaze score for each patch as:
H
Σ
Hr(z,y),
(5)
(2.8)Ep:
where i E (1,2...., P) is the patch index, and Z = Ex,y He( y) is a normalization constant ensuring that E He, =.
These patch-wise gaze scores H. can now be directly compared to the model's attention weights A,, which operate at the same patch level. The key to the proposed simple yet effective approach is adding a regularization term during training that encourages the model's 's attention weights to align with this human gaze distribution:
AL. H
DKL (AI(A = Σ Ars log
(6)
where DrL is the Setlbackk-eibbee divergence between the two patch-wise distributions. This regularization guides the model attend to regions that humans focus when performing activities, while still allowing it to learn task-relevant attention patterns from the training data.
Training Objective The overall training objective combines the standard cross-entropy loss for text generation with our gaze- based attention regularization term:
Lintal = LCF + X Σ (A.((l).....
where LCR = - -x(() log( 1) is the cross-entropy loss between predicted and ground- truth descriptions, and À hyperparameter controlling the strength of the regularization.
This formulation ensures that the model learns to generate accurate textual descriptions while developing attention patterns that align with human visual focus. By adjusting the hyperparameter X, we can control the influence of human gaze patterns on the model's attention mechanism, allowing us to systematically evaluate the impact of gaze regularization egocentric behavior understanding tasks, which we discuss in the following section.
5

---

Future Frames
Ground Truth
Predictionsbbase)
Predictions (ours)
The camers going comerase with the seated fron with u dock of cards and mobile phone sereen vo blg in teee them. indiutiull ls goingnt readtthe baoki
The incivduual TV ramone with vos on the tabli and indivddutt in front. The cauted parson in hankgguuddd
goiigg talk the seatted person with mobile phone
The
viskile The souted porso la going'o roodia bookl
garr
The cemers weonee in going to pan plantic packast with paper and decit cards roude and the pareon bitu atort will move room, duolls gotngg iemove
The inciviuual opening
The indi- open cards
living room with mariion in teedigrundd
blue adhirt well moves inside the thving rom.
The incividual is remooving e paper invilda
The compoe wourcr is to
remove cards from plastio.
Aninduuduss in handwwll muahh troweds a pasti choonlate
pppesse gainng giab
hand will meah for pack chooolate the kitchen island. indirduual duai got to goo the chooolatee snack placedion the
screen seenbsthed indivddull
drrikk from kitchenn inlandd
chocolate nuts and will look at the har code acktnee which inen
the lube, Bhs paoles/whill dag dlink en tablo.
Figure 3: Qualitative Results of Future Activity Prediction. Examples comparing predictions from the base model and our rezzereegulrizzed model against ground-truth annotations and actual future frames. The gaze-regularized model correctly predicts specific objects and actions (e.g., "picking up chocolate") while the base model makes less accurate predictions (e.g., incorrectly predicting "a drink"). Key words in the predictions are highlighted to emphasize differences in prediction specificity and accuracy.
4 Experiments
We conduct a comprehensive set of experiments to evaluate the effectiveness and generalization of our proposed guze-regul framework for egocentric behavior understanding. Our experiments utilize Ego4D clips with gaze annotations, which provide the necessary ground truth for both training and evaluation. The primary experimental comparison contrasts our sazzeregglarized models with a base model that uses only RGB inputs and standard attention, without any gaze supervision or alignment during training. To further analyze the effects of gaze regularization, we perform ablation studies by varying strength, evaluate performance across different temporal anticipation horizons, and examine runtime efficiency for practical deployment. For future prediction tasks, we use an observation window of T, = 5s and a prediction horizon Ta = 2s, while for current 3s. Our evaluation methodology is activity understanding, we use observation window of MiiiencceBBRT--e BERT- based transformer (Reimers) based on semantic similarity scores computed using a Sentence- and Gurevych 2019), which rewards semantically coherent predictions and penalizes irrelevant or incoherent outputs. In addition to the experiments provided in the main paper, we also provide extensive ablation which can be found in Section A.5 of the Appendix.
4.1 Dataset Construction
We adapt the Ego4D dataset (Grauman et al. 2022) to construct a training set suitable for egocentric activity understanding and future prediction using gaze-regularized VLMs. Ego4D provides ego- centric video clips with synchronized eye-tracking data, which we leverage through the following processing steps:
1. Temporal Sampling: To reduce computational load while preserving temporal structure, we downsample all videos to one frame per second.
Spatial Heatmap Formation: Raw gaze coordinate annotations (in pixel format) are converted into spatial heatmaps through Gaussian filtering, which transforms sparse gaze points into contin- uous attention distributions that better reflect the spread and uncertainty human visual focus. These heatmaps serve soft supervisory signals during training, allowing the gaze regularizer to guide the model's attention toward regions that align with human perceptual patterns.
3. Textual Description Generation: We use GPT-4V OOpenAl, 2023) to generate fine- ine-grained textual descriptions that capture the egocentric activities. To ensure temporal consistency and contextual relevance, we provide GPT-4V with the image sequences and iteratively refine the prompts. The final prompt template, obtained after multiple rounds of feedback and validation, guides GPT-4V generate descriptions that include objects being manipulated, ongoing actions, and spatial movement cues. This process results in the formation of finer-grann grained annotations which we utilize train the model for the egocentric behavior understanding task.
Together, these steps yield a training dataset consisting of synchronized RGB frames, gaze heatmaps, and rich textual annotations. This curated dataset provides the necessary visual, attentional, and

---

Input Frames
Attention w/o gaze
Attention w gaze
Annotations (for reference)
The camen converse whi sested
cards mobile froet either. The audivdual a rotor red book in of
Amaenton table
table frems (deak, book] and mobile phone
The weare ganglo plastlc pocket with peper deck cards insit living room thehndibbat goging ginng remove from plastic packet.
blue shirt
Amentoon On
objortipacket
pack chooolatee cooted counter wrte wowan can be sen betind #nieewedaillgginngoo pick up the chocolate nuts ondwi look in codo soarnor which on labal the the drinni on the table.
Attention: On human-and soft
Attention: chec bapt
Figure 4: Comparison of Attention Maps. Attention distributions from base model (without gaze regularization) versus gaze-regularized model. Our approach produces more semantically meaningful attention: Top: Our model focuses on specific objects (cards, book, phone) while the base model attends broadly to the table. Middle: Our model attends both the person and plastic packet, compared to the base model's person-only focus. Bottom; Our model correctly attends to chocolate and drink on the counter, while the base model attends to less relevant areas. This demonstrates how gaze regularization guides the model to focus on objects crucial for understanding human activities.
semantic supervision to effectively train our gaze-regularized vision- language models for both current activity understanding and future event prediction in egocentric settings. More information about the dataset can be found in Section A.2 of the Appendix.
4.2 Gaze-Regularizer Model Evaluation Across Architectures
We evaluate the effectiveness of our gaze-regularized framework across five transformer-based VLM architectures: OpenFlaamino,, modified OpenFlamingo without the Perceiver Resampler, LaViLa's Narrator module (Zhao et al., (2022), InternVL (2.5-1B) (Chen et al. 2024). and OpenLLaVa VA (LLaVA- VerT-Vienaa-BB.. (ByLinn and Long 2024). Each model is trained on the same Ego4D- -aseed dataset and evaluated using only inputs for two tasks: future event prediction and current activity understanding. Our framework integrates gaze-regularized attention block between the visual encoder and the language decoder, applying gaze supervision exclusively during training. This modular design makes it applicable to any architecture that utilizes attention mechanisms, while maintaining practical inference by eliminating any dependency on gaze data at test time.
As reported in Table I our method consistently improves performance across all architectures. For future prediction, we observe absolute gains ranging from 8.9% 10.5%, with OpenFlamingo and its modified variant showing the strongest improvements. In current activity understanding, see improvements between 4.9 % and 6.9%, with LaViLa achieving the largest increase. These results demonstrate that gaze-guine gaze-guided regularization meaningfully enhances temporal and contextual reasoning in egocentric vision tasks. Figure 3 shows the annotations predicted by the models, while Figure visualizes attention maps for qualitative comparison.
4.3 Sensitivity to Gaze Regularization Scale
To assess the impact of the rez-aaegulaiization term, conduct an ablation study by systematically varying the regularization strength parameter À during training. When λ=0, the model is trained without any gaze supervision, using only the cross-entropy loss, though the rest of the architecture remains unchanged.
As shown in T moderate regularization strength (A=100) consistently yields the best results across both tasks and all architectures. The most significant performance drop occurs in the unregu- larized setting (A=0), underscoring the value of gaze supervision in improving attention allocation and prediction quality. While increasing to 1000 still leads to competitive results, it introduces a marginal performance decline in some cases compared to the setting (A=100).
7

---

Table 1: Performance comparison of baseline models versus our raze-regularized approach across five VLM architectures. Results show semantic similarity scores for future prediction and activity understanding tasks Ego4D. All models RGB-only inputs at inference time, demonstrating consistent performance gains (4.9-10.5%) with our method across all architectures and tasks.
FUTURE PREDICTION
ACTIVITY UNDERSTANDING
MODEL
BASE
OURS
GAIN
BASE
OURS
GAIN
OPENFLAMINGO
0.6525
0.7505
+9.8%
0.7176
0.7848
+6.7%
MODIFIED OPENFLAMINGO
0.6155
0.7200
+10.5%
0.6677
0.7300
+6.2%
LAVILA NARRATOR
0.6030
0.6924
+8.9%
0.6576
0.7268
+6.9%
INTERNVL
0.6323
0.7318
+9.9%
0.6831
0.7404
+5.7%
OPENLLAVA
0.5872
0.6771
+9.0%
0.6539
0.7027
+4.9%
Table 2: Effect of regularization strength (A) on semantic performance across VLM architectures.
FUTURE PREDICTION
ACTIVITY UNDERSTANDING
MODEL
x=
==100
A=1000
x=0
A=100
>=1000
OPENFLAMINGO
0.6317
0.7505
0.7354
0.6917
0.7848
0.7721
MODIFIED OPENFLAMINGO
0.6032
0.7200
0.7072
0.6603
0.7300
0.7194
LAVILA NARRATOR
0.6033
0.6924
0.6765
0.6310
0.7268
0.7123
INTERNVI.
0.6342
0.7318
0.7190
0.6867
0.7404
0.7287
OPENLLAVA
0.5882
0.6771
0.6628
0.6523
0.7027
0.6895
These findings suggest that while gaze-based alignment substantially benefits model performance, excessive regularization may constrain the model's ability to attend to regions that are task-relevant not explicitly captured by human gaze. We systematically explored different scaling values to determine the range in which the regularizer is most effective, though further investigation may be needed precisely calibrate optimal regularization strength across different tasks and architectures.
4.4 Impact of Anticipation Window Length Predictive Accuracy
We further evaluate the ro- bustness and temporal gener- alization of our approach by varying the anticipation win- dow Tax testing the model's ability predict future activi- ties at different time horizons. Specifically, we compare per- formance at 2-second and 5- second intervals, using fixed observation window T, 5s across both settings. This ex- periment allows us to assess whether the benefits of gaze regularization persist as the prediction task extends further into the future, where the increasing temporal gap between observation and target introduces greater uncertainty.
Table 3: Performance across anticipation horizons for future event prediction with two different variations.
2s
5s
MODEL
BASE
OURS
BASE
OURS
OPENFLAMINGO
0.6525
0.7505
0.6297
0.7315
MOD. OPENFLAMINGO
0.6155
0.7200
0.5911
0.7094
LAVILA NARRATOR
0.6030
0.6924
0.5888
0.6713
INTERNVI
0.6323
0.7318
0.6172
0.7004
OPENLLAVA
0.5872
0.6771
0.5491
0.6492
As shown in Table our gaze-regularized model consistently outperforms the baseline at both horizons. The performance advantage is particularly notable because human gaze patterns can provide early indicators of intent that help bridge the temporal gap between current observations and future actions. Nevertheless, we observe that performance for both models declines when the prediction horizon increases from to 5 seconds. This decline is expected due to the inherent challenges of longer-horizon prediction, where the available context from the observation window becomes less informative and the range of possible future actions expands, making accurate prediction increasingly difficult.

---

Table 4: Out-of-distribttion generalization on EGTEA+ Gaze. All models are trained on our dataset and evaluated using RGB-only inputs during inference.
FUTURE PREDICTION
ACTIVITY UNDERSTANDING
MODEL
BASE
OURS
GAIN
BASE
OURS
GAIN
OPENFLAMINGO
0.6501
0.7305
+8.0%
0.6963
0.7527
+5.6%
MODIFIED OPENFLAMINGO
0.6071
0.6797
+7.3%
0.6326
0.6924
+5.0%
LAVILA NARRATOR
0,5804
0.6632
+8.3%
0.6397
0.7003
+6.1%
INTERNVL
0,6230
0.7107
+8.8%
0.6712
0.7265
+5.5%
OPENLLAVA
0.5714
0.6209
+4.9%
0.6338
0.6615
+2.8%
4.5
Evaluating Generalization on nut-of-Distribution Egocentric Data
To assess how well our gaze-regularized framework generalizes beyond the training distribution, we evaluate it on the EGTEA+ Gaze dataset (Li et al., 2020a This setting allows to test whether the attention alignment learned from Ego4D transfers effectively to new egocentric domains.
We evaluate both the base and gaze-regularized models on this new dataset without any additional fine-tuning, using only models trained on our original Ego4D-derried dataset. As shown Table 4 the gaze-regularized models maintain their performance advantages, achieving improvements of 7-9% future prediction and 5-6% activity understanding across most architectures. While we observe performance variations across models, these can partly be attributed to the adaptations required to make each architecture compatible with our input pipeline. These results suggest that our method produces attention patterns that generalize beyond the training domain and remain effective in novel egocentric settings. We note that these comparisons serve primarily as a sanity check rather than definitive ranking of model capabilities across datasets.
4.6 Runtime-Performance Analysis of Gaze Regularization
Inference efficiency is essential in real-world applications such assistive robotics, where la- tency and computational con- straints must be carefully bal- anced. To evaluate this critical performance-efficienng trade-off, we measure both the semantic ac- curacy and runtime characteris- tics of different OpenFlaminge variants under our gaze-regularization framework, as OpenFlamingo consistently demonstrated the strongest overall performance in our experiments.
Table 5: Runtime performance of various model variants from the openflamingo architecture.
MODEL VARIANT
SCORE
GAIN
RUNTIME (S)
BASE (NO REG.)
0.6525
1.7
GAZE-REs (w/O OCCL.)
0.7298
+7.7%
2.3
GAZE-REG (W/ OCCL.)
0.7505
+9.8%
2.3
During training, the inclusion of occlusion filtering adds computational overhead, as it requires identifying and discarding unreliable gaze inputs to improve the quality of supervision. However, at inference time, all models operate using only RGB inputs-ni no gaze data, heatmaps, or occlusion computation is required. This design choice ensures that our approach maintains practical deployment efficiency while benefiting from guided learning during the training phase.
As shown in Table 5] our best- best-performing model (Gaze-Reg w/ occlusion) achieves a 9.8% gain in semantic performance over the base model, with only a minor increase in inference time (2.3s vs 1.7s). These measurements reflect forward pass time only and exclude pre-processing or data loading steps. All tests conducted using an NVIDIA A800 GPU ensure consistent evaluation conditions.
4.7 Reducing Visual Hallucinations
further investigate whether gaze regularization mitigates visual hallucinations known issue where VI.Ms generate descriptions containing objects actions absent from the visual input. Standard benchmarks for evaluating hallucinations, such as HallusionBench (Guan et al.] 2024), provide established metrics for this phenomenon but lack the gaze annotations required for method.

---

Since standard hallucination benchmarks lack gaze annotations, we conducted a controlled human evaluation. We selected 200 examples from our dataset and generated outputs from both the baseline and gaze-regularized models. These responses were randomly shuffled and presented to evaluators blind to the model source, who were shown the ground-truth video context and instructed to flag any instances of clearly hallucinated or unrelated content. Taking inspiration from the metric in HallusionBench, we report the C1 score, which is the ratio of model responses containing clearly hallucinated instances to the total number of responses evaluated. The results, summarized in Table 6 show that our method meaningfully reduces hallucination. The Cr score dropped from 0.205 to 0.140. This supports the hypothesis that by focusing the model's attention risually-grounde human- attendd regions, our approach discourages the model from "inventing" details based on linguistic priors alone, leading to more reliable and trustworthy responses.
Table 6: Human evaluation of visual hallucination rates, Our gaze-regularized model reduces the occurrence of hallucinated content, as measured by the Cj score.
Model Variant
Hallucinated Cases
Total Samples
C₁
Base Model
41
200
0.205
Gaze-Regularized Model
28
200
0.140
4.8 Quantifying Attention-Gaze Alignment
A core premise of our work is that aligning a model's attention with human gaze leads to better task performance. To quantitatively validate that gaze-regularized model indeed learns to approximate human attention patterns, we measured the spatial overlap between the model's final-layer attention maps and the ground-truth human gaze heatmaps the test set. Specifically, we computed the top-10 10 overlap, which measures the proportion of the model's top-10 10 attended image patches that fall within the human gaze distribution.
The results confirm an improvement in alignment when comparing the base model and the gaze- regularized model. The gaze-regularized model achieved 68% top- 10 overlap, which is an increase from the 42% overlap observed in the base model which does not use gaze regularization. This improvement provides concrete evidence that regularization loss successfully steers the model's focus toward regions that humans find important. This learned alignment with human priors is key factor behind the consistent performance improvements observed across all tasks and architectures.
5 Conclusion
In this work, we have demonstrated that incorporating human gaze data as training- -ime supervi- sory signal significantly enhances the performance of VLMs in egocentric behavior understanding, improving both current activity recognition and future event prediction. By aligning model attention with human visual focus through gaze-based regularization, our approach captures subtle, temporally grounded cues that static attention mechanisms often miss. Crucially, it requires no gaze input at infer- ence time, making it practical for potential world deployment in assistive systems, wearable AI, and human-robot collaboration. The framework modular and integrates seamlessly with attention- based VLMS such as OpenFlamingo, La VaLa, Intern VL. and OpenLLaVA. Across all models and tasks, observe improvements up 11 % in future prediction and 8 % in activity understanding demonstrating the effectiveness and generalization of our method. Furthermore, this attention alignment substantially improves output reliability, reducing the rate of visual hallucinations based on human evaluation. This suggests that human gaze provides a spatial prior that effectively grounds language generation, reducing reliance on potentially misleading textual associations. Broader adop- tion of augmented learning depends on access to srgeesccale, high-quality datasets with reliable and calibrated gaze annotations. Training on such data with task-specific labels could further enhance performance. Future work may explore joint modeling of gaze and action, as well as extending the framework to other temporally grounded tasks beyond egocentric understanding. We will release our code and dataset to support continued research and encourage further exploration of gaze as powerful supervisory signal for aligning vision-based models,,ncludie VI.Ms- more closely with human attention and intent.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the Early Career Scheme of the Research Grants Council (RGC) grant # 27207224, the HKU-100 Award, a donation from the Musketeers Foundation, and in part by the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust. We would like to extend our sincere gratitude to Siyan Dong, Qihang Fang, Ruizhe Liu and Qian Luo (from HKU IDS) for their invaluable technical discussions and implementation support throughout this project. We also like to thank Andrew Lin from UC Irvine for helping with annotation work and feedback during the experimental phase. Finally, appreciate the constructive feedback from our reviewers, which helped strengthen this paper and will pave the way for future projects.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
Reuben M. Aronson, Thiago Santini, Thomas C. Kübler, Enkelejda Kasneci, Siddhartha Srinivasa, and Henny Admoni. Eye-hand behavior in human-robot shared manipulation. In 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 4-13, 2018.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo: An open-source framework for training large autoregressive vision-language models, 2023.
Abdishakour Awale and Duygu Sarikaya. Human gaze guided attention for surgical activity recogni- tion, 2022.
Daniel Bolya, Po- Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception The best visual embeddings are not at the output of the network, 2025. URL https //arxiv.org/abs/
encoder:
13181
Dutliff Boshoff, Raphael Elikplim Nkrow, Bruno Silva, and Gehard P. Hancke. Gait device pairing in unconstrained environments with signal morphing. IEEE Transactions on Consumer Electronics, 2024. ennirtroimeddieiioonmeeen
org. :::
Christian Braunagel, Wolfgang Rosenstiel, and Enkelejda Kasneci. Ready for take-over? a new driver assistance system for an automated classification of driver take-over readiness. IEEE Intelligent Transportation Systems Magazine, 9(4): 10-):00-2, 2017. doi: 20.10999/1SS..217277431.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener. Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly- scaled multilingual sanguugge-magge model, 2023. URL httts::///axiiii.. www.absg////o/oooooeo/..otaaaert 2209 06794]
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- Interna- fional Journal of Computer Vision (IJCV), 130:33-55, 2022. URL, /ttp:////oo.oo/11100001//
s1 1112630221-15533 01531-2

---

Alexandra Frischen, Andrew P. Bayliss, and Steven P. Tipper. Gaze cueing of attention: Visual attention, social cognition, and individual differences. Psychological Bulletin, 133(4):694-724. July 2007. ISSN 0033-2909, doi: 10. 10.1037/00033-20001333.444 10. 00.1037700333 20000000000000...00 133.433.44 694
URL URLhttp:: //dx. doi org/
Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with Dolling-unrollinn Istms and modality attention, 2019.
Tanya Goyal and Greg Durrett. Annotating and modeling fine-grained factuality in summarization, 2021. URL httpstt:::// http///axvvvvv/ org/ abs ab04. 04302
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie. Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi. Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar. Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park. James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani. Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2022.
Michelle R. Greene, Benjamin J. Balas, Mark D. Lescroart, Paul R. MacNeilage, Jennifer A. Hart, Kamran Binaee, Peter A. Hausamann, Ronald Mezile, Bharath Shankar, Christian B. Sinnott, Kaylie Capurro, Savannah Halow, Hunter Howe, Mariam Josyula, Annie Li, Abraham Mieses, Amina Mohamed, Ilya Nudnou, Ezra Parkhill, Peter Riley, Brett Schmidt, Matthew W. Shinkle, Wentao Si, Brian Szekely, Joaquin M. Torres, and Eliana Weissmann. The visual experience dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video, 2024. https:/ts:://attt
org/ abs/ 0abs/2400 2404. 18934
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang. Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models, 2024. https: org/ abs/2310. 22100 14566
Mary M Hayhoe, Amurag Shrivastava, Ryan Mruezek, and Jeff B Pelz. Visual memory and motor planning in a natural task. J. Vis., 3(1):49-63 63, 2003.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- large language models via over-trust penalty and cnoospcctionaallcaaiin 2024. URL http:::////arrte
org/ ab/////s///1313
17911
Yifei Huang, Minjie Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato. Mutual context network for jointly estimating egocentric gaze and action. IEEE Transactions on Image Processing, 29:7795-7806, 2020. ISSN 1941-0042. doi: 111.:09/ip.2220300777 URL URLhhtpp::/dae /attp://dx..o doi dow.:diii.ogg/ii 10. 1109/
TIP. 2020 3007841
Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, and Yu Qiao. Egoexolearn: A dataset for bridging asynchronous ego- and exo-centric view of procedural activities in real world, 2025. (RtLaatss://arriiii 2403. 16182
Junhwa Hur and Stefan Roth. Mirrorflow: Exploiting symmetries in joint optical flow and occlusion estimation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
12

---

Bruno Laeng, Ilona M. Bloem, Stefania D' Ascenzo, and Luca Tommasi. Scrutinizing visual images:
The role of gaze in mental imagery and memory. Cognition. 131(2):263 283,, 2014. ISSN 0010- 0277. doi: htmp://iiaeaaaattt....//./ffffo.
.003. BtRlhttps::/vaer www.sccienceirecct
Bolin Lai, Miao Liu, Fiona Ryan, and James M. Rehg. In the eye of transformer: Global-local correlation for egocentric gaze estimation, 2024. URL https://arxiis /atp:////axvvv. orgoorgg//. /a//abb//22223 04464
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- language understanding and generation, 2022. URL https://arxiv org//bss/22 12086
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang. Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators, 2024. URL // /txxivxivv.o /rgg//b////11.013///
Yin Li, Miao Liu, and James M. Rehg. In the eye of the beholder: Gaze and actions in first person video, 2020a.
Yin Li, Miao Liu, and James M. Rehg. In the eye of the beholder: Gaze and actions in first person video, 2020b. URL httpt:tttt//xitii org/ hts//bbb//00000 0066
Chen Lin and Xing Long. Open-llava-next: An open-source implementation of llava-next series for facilitating the large multi-modal model community, httpa//tc..g///tttooo/oooooooooa Open LaLVL-LaVAAANN- 2024.
com/xiaoachen988
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL httpps//arxiivorr/absee http://abb/234.044//
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-languaa tasks, 2019a. URL http:://aaaiii..rrgg//////////// wwwp/r/g/abb//////// 1908 02265
Minlong Lu, Danping Liao, and Ze-Nian Li. Learning spatiotemporal attention for egocentric action recognition. In 2019 IEEE/CVF International Conference Computer Vision Workshop (ICCWW), pages 4425 4434, 2019b. doi: 10. 10.1109/CC...20119.0 9.00433.
Esteve Valls Mascaro, Hyemin Ahn, and Dongheui Lee. Intention-connittioned long-term human egocentric action forecasting, 2024.
Kyle Min and Jason Corso. Integrating human gaze into attention for egocentric activity recognition. 11 2020.
OpenAl. Gpt-4v(ision), 2023. URL httpp:tss:/:: hatpaa//occcooootrbboooo/o/////
pdf
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https
//arxiv. orag/vi.rg//e abs///b3.//1300
Keith Rayner. The 35th sir frederick bartlett lecture: Eye movements and attention in reading, scene perception, and visual search. Quarterly journal of experimental psychology, 62(8):1457-1506, 2009.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019. URL htttttt/t/aaiiiiii org/ absa 1908 10084
Debaditya Roy, Ramanathan Rajendiran, and Basura Fermando. Interaction region visual transformer for egocentric action anticipation, 2024. URL Utt./ttts://aaxx org/ abtpg//bb///......///// 12111
Ali Shafti, Pavel Orlov, and A. Aldo Faisal. Gaze-based, context-aware robotic system for assisted reaching and grasping. In 2019 International Conference on Robotics and Automation (ICRA), pages 863-869, 2019. doi: 10. 10.11990
13

---

Swathikiran Sudhakaran and Oswald Lanz. Attention is all we need: Nailing down object- attention for egocentric activity recognition, 2018.
Hao Tan and Mohit Bansal. Lxmert: Learning modaltyy encoder representations from trans- wttp://www..rg/aa////..... 07490
formers, 2019. URL
Zachary Teed and Jia Deng. Raft: Recurrent all all-pairs field transforms for optical flow, 2020. URL
https://arxiv, wwtp:/ii.orroeeoooo org/ wwwgaaa/////...ooog
12039
Sanket Thakur, Cigdem Beyan, Pietro Morerio, Vittorio Murino, and Alessio Del Bue. Enhancing
next active object- based egocentric action anticipation with guided attention, 2023. URL https
//arxiv. org/ /bbb//2/555 12953
Steven P. Tipper. Eps mid mid-career award 2009 From observation to action simulation: The role of
attention, eye-gaze, emotion, and body state. Quarterly Journal of Experimental Psychology, 63
(11):2011-2105, November 2010. ISSN 1747-0226. doi: 10.1080/1747021 100000220000020002222 URL
http://dx.co wwg. 1080 1T0007007770110002222222
Daniel Weber, Thiago Santini, Andreas Zell, and Enkelejda Kasneci. Distilling location proposals of unknown objects through gaze information for human-robot interaction. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 11086-11093 2020.
doi: 10.1 10.1109//R0045443. 000005444533.00...999 200.9999922......9..
Yana Wei, Liang Zhao, Kangheng Lin, En Yu, Yuang Peng, Runpei Dong, Jianjian Sun, Haoran Wei, Zheng Ge, Xiangyu Zhang, and Vishal M. Patel. Perception in reflection, 2025. URL
https://arxiv. org/ 3bs/2504.0716
Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, and Shuai Ma. Voila-a: Aligning vision- language models with user's gaze attention, 2023. URL htttp:://aaxiiii orgooogg//aaoooo ha///b//2000 2401 09454
Fengyu Yang. Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang. Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, and Alex Wong. Binding touch to everything: Learning unified multimodal tactile representations, 2024. URL htttps///axivi org/ abs/
2401. 18084
Yushi Yao, Chang Ye, Junfeng He, and Gamaleldin F. Elsayed. Teacher-generater spatial-attention labels boost robustness and accuracy of contrastive models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23282-23291, 2023, doi: 10.1109/
CVPR52272.2023.00233
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama- adapter: Efficient fine-tuning of language models with zero-init attention, 2023. URL URL httppt//aaxxii...oo org/ abs/ 2303. 16199
Qi Zhao, Shijie Wang. Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-term action anticipation from videos?, 2024a. htttttsss//aiiii. org/ abs / 2307. 16368
Yi Zhao, Yilin Zhang, Rong Xiang, Jing Li, and Hillming Li. Vialm: A survey and benchmark of visually impaired assistance with large models, 2024b. URL httpss://axxi..orrttt absp///b//222 01735.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models, 2022. URL https://axxiz /ttp//abb22222.501111
Xingeheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C. Knoll. Vision language models in autonomous driving: A survey and outlook, 2024a.
URL https:://aattt wwwaaaivvvvvorgaaaea
arg/abs/2310 14414
Yuchen Zhou, Linkai Liu, and Chao Gou. Learning from observer gaze:zero-sher attention prediction oriented by human-object interaction recognition, 2024b. URL https: http//xxxvvvv.ro/. abs// 2405
09931
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes the limitations are mentioned the appendix
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not include any theorems formal theoretical results. It is focused on methodological design and empirical evaluation gaze-regularized vision- language framework.
Justification:
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper details how construct the dataset well how fit the gaze- regularized block in a VLM architecture
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Dataset construction steps and annotation protocols are well documented in the main paper and the appendix.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and evaluation details are described in the appendix
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experimental results are detailed and provided with several ablations to ensure significance of the method and experiment
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Yes information is provided in the appendix well.
15

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Stt.ghg.gg.....oou.....oo//oo//t
Answer: [Yes]
Justification: We utilise publicly available dataset with gaze annotations (e.g., Ego4D) and does not involve any new data collection from human subjects. adheres to ethical use of data and model outputs.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses deployment in assistive robotics and human-maching collaboration.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The models and datasets do not pose a high risk for misuse. The study does not release generative models or web-scraped content.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce gaze-augmented, annotated subset of Ego4D, and details the annotation procedure, prompt design, and filtering steps. Release of the dataset is planned and methodology is reproducible.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study does involve any new data collection or interaction with human subjects crowdworkers.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research uses publicly available datasets that already obtained IRB ap- proval. No new user studies or interventions were conducted.
16. Declaration of LLM usage
16

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?
Answer: [Yes]
Justification: We use GPT-4V to generate fine-grained annotations in automated manner since manual annotation all frames was not possible.
17

---

A Appendix
This appendix presents supplementary material to support our study, We provide a table notations to aid the readers, an expanded discussions of related work, further details the dataset curation and experimental prompts, and results from supplementary ablation studies. We conclude with comprehensive model training details to facilitate replication of our work.
Short table of Notations without scouring through the text, we provide a table of notations for quick reference where the symbols are accompanied by short description.
To ensure that the readers can directly refer to table for certain notations
Table 7: Summary of key notations used in temporal gaze aggregation and occlusion filtering.
Symbol Description
Timestamp of current frame
Temporal aggregation window size (e.g., 200ms)
4
RGB image frame time
9t
Gaze point (pixel coordinate) at time t
m
Spatial heatmap generated from It via Gaussian smoothing Aggregated gaze heatmap at time after occlusion filtering
H.
f
Forward optical flow from frame T, to It
f
Backward optical flow from I to L. Designated pixel location in image 1, Translated pixel location in /, using forward flow Discrepancy between forward and backward flow vectors Proportion of pixels with flow discrepancy >
p
P
A
Mobserved
Threshold for pixel-level flow discrepancy Threshold for deciding major occlusion (set to 60%)
0r
Binary occlusion validity flag for frame T,
A.1 More Related Work
Vision Language Models VLMs take in as imput images and text together (image, texf) and produce a text output. VLMs learn a mapping function (image, text) + text where the task varies from Visual Question Answering (VQA) tasks to generating text based on image and text provided. The VLMs output text condition on the image text sequence and several models exists such as BLIP, LLaVa Flamingo etc. Liu et al. (2023), Li et al. 2022) Zhang et al.) (023) Alayrac et al. (2022)
) Chen et al. )2023). In this study, we mitially employ the open-source version of the Flamingo model Awadalla ct al. (2023), built upon the foundations of the original Flamingo described in Alayrac et al. (2022).
To show that approach can be implemented and generalized to other architectures, we also utilise ViLa's Narrator module (Zhao et al. 2022), and adapted versions of Intern VL'Chen et al. 2024) and OpenLLaVA (Lin and Long] 2024) in experiments. These models were chosen for their of attention mechanisms, which are central to our -our framework explicitly aims to modulate attention to better align with human visual focus. In addition traditional input modalities, such as RGB images, recent advancements have highlighted the potential of integrating various modalities beyond vision, including gaze, gait, and tactile sensors (Boshoff al.) 2024 Yang al., 2024). Building on this premise, our approach incorporates eye gaze as an additional signal.
Dataset The Ego4D and EPIC-Kitchent datasets consist of egocentric videos of camera- performing daily activitics in semi-controlled environments (Grauman et al. 2022 Damen et al.. 2022). Additional relevant datasets include the EGTEA+ Gaze dataset, with 28 hours of cooking- centric content paired with gaze (Li et al. 2020b), and the Visual Data Experience (VDE) dataset, which provides approximately 240 hours of everyday activity recordings with synchronized gaze and head tracking (Greene et al. 2024).
While some datasets provide coarse labels, the auzeeaugeented subset of Ego4D used in our work lacks fine-grained annotations. To address this, we supplement it with descriptive captions generated via GPT-4V. Future work may extend this setup by incorporating VDE or other multimodal datasets.
18

---

Additionally, Huang et al. (025) introduce dataset combining egocentric and exocentric perspectives for procedural activities. This dual view setup which aligns with an "observe first, imitate next" paradigm may enhance downstream modeling but the inclusion of this study is left for future work.
Attention and gaze-augmented models: Attention-based models are widely used to identify im- portant features and improve performance across various domains, including autonomous driving (Braunagel et al. 2017), action prediction, and human-computer interaction (Weber et 2020 Shatti et al. 2019 Aronson et al. 2018). These models enhance interpretability by directing focus to the most relevant regions of an image or sequence.
For instance, class activation maps have been employed to leverage pooling layers in deep networks, generating class saliency maps (Sudhakaran and Lanz 2018) that highlight key objects associated with future actions. Guided-attention mechanisms have further been used to model next-active object interactions, improving action anticipation (Thakur et al. 2023). The Spatiotemporal Attention Module (STAM) (Lu et al., (209b) incorporates eye gaze as supervision, training a network to predict attention maps for activity recognition. Other studies have similarly used gaze guide attention maps in activity recognition tasks (Min and Corso 2020
Awale and Sarikaya 2022)..
Beyond artificial systems, human perception and action have long been studied in relation to gaze. Eye movements provide task task-specific visual cues, enabling more efficient action execution (Hayhoe) et al. 2003). In work, explore how eye gaze data can be integrated into Vision-Language Models (VLMs) to improve egocentric behavior understanding, conditioning text annotations on gaze data to enhance future activity prediction.
Recent works such as OPERA (Huang et al. 2024) and Perception in Reflection (Wei et al., 2025) operate on the cross-attention maps during text generation and apply retrospective corrections if and when the model focuses on irrelevant regions. For instance, OPERA penalizes captions that mention objects like "pencil" if the cross-attention never highlighted any pencil in the image. Perception in Reflection similarly evaluates whether the model has missed out on finer details by reflecting on generated answers using multi-turn dialogue dataset.
In contrast, our work focuses on egocentric video understanding, where the goal to model human behavior and intent in dynamic scenes. In this context, human gaze offers a strong and temporally grounded prior for what the person doing or about to do as humans naturally fixate on task relevant objects during interaction. Rather than relying post-hoc attention correction, our method introduces gaze alignment to shape the visual representation passed into the language module, helping the model focus on semantically meaningful content from the start.
To summarize, inn our project, we study various ways utilize eye gaze data in VLM setting, building gaze-regularized attention mechanism. Our model conditions predictions on eye gaze data as an input signal to output fine- -re-grdi text annotations of events happening in the near future, as well as providing detailed annotations of the current activity.Our approach is compatible with various transformer-bat based VLMs and does not require gaze at inference time.
A.2 Dataset Creation
This section provides details on the dataset creation process for model training and testing, including the utilization of an occlusion filtering mechanism to create aggregated spatial heatmaps.
A.2.1 Dataset Construction and Prompt Design
The Ego4D dataset comprises of egocentric video clips along with supplementary data such as audio, text annotations, eye gaze data, and additional metadata (Grauman et al.) (2022). For our project, we focused on the subset of video clips that include eye gaze data, containing approximately 33.3 hours of egocentric videos recorded from 80 participants. The eye gaze data is provided in numerical form, containing canonical timestamps and the pixel coordinates of gaze points. We transform gaze points to images represent important visual regions, aligning with how humans perceive spatial information (Laeng et al. 2014). Due to the minute differences between consecutive images in the original videos, we perform downsampling to one image per second to reduce computational requirements while remaining effective. Since our focus is fine-grained egocentric behavior understanding, we modify existing data with detailed textual descriptions to enhance human-machine interaction. We leverage GPT-4V to generate annotations for video frames by processing sequence of images and
19

---

User Prompt
Sample Image
Descrbbe what happening in the image sequence and output the text descriptions.
Goal: To get annotations for the image sequence.
P
Refined Prompt
For each image the sequence, provide ennotation which desciibes the action taking
Language Model
GPT 4V
Final GPT Prompt
Language model prompt fenditasis and initial user promet, suggeat new prompt which would sariefy requiaments.
Foreach imageina sequendo oofles, providi direct description, Ensum one description par imagle eapturing otpects, actiorn and descriations
procented (outost should coreopond to image 1) Imtgine thanyou nead to proetde theow descriptara to robot that they can comatsumntely sooter you.
Human evaluation
User Feedback
Annotations
whitto Annotation
only annotation the whole sequence, We went and actions undertalan in the sequence.
The process of brewing cup coffee taking place image sequance. person can be seen talking to the
the drip tray coffee that dispensing hot beverage.
annotation
Figure GPT-4V Annotation Workflow Left. We illustrate our annotation pipeline, where sequences of egocentric images are provided to GPT-4V along with an iteratively refined prompt. Human feedback is used improve the prompt over multiple rounds, ensuring greater accuracy and contextual coherence in the generated captions. The use of the entire sequence helps maintain temporal consistency, allowing GPT-4V to better capture object interactions and ongoing actions. Right. We show sample image from the dataset, the final refined prompt, and the resulting annotation. This process results in finer-grained text annotations which we utilise our experiments.
prompting it to describe each frame. This method ensures contextual coherence across frames. After obtaining initial descriptions, we evaluate their quality and provide feedback to refine the output. This feedback is used to modify the prompt, which is then fed back into GPT-4V with the image frames. We conduct prompt selection and refinement using multiple sequences from different video clips ensure generalization. This iterative process continues until we establish an optimal prompt template that consistently yields accurate and contextually appropriate annotations.
Specifically, we start by passing small set of images (validation set) to GPT-4V with basic prompt like: "Describe what happening the image sequence and output the text descriptions." The initial output was manually evaluated, and feedback prompt, was passed to ChatGPT to refine the prompt for generating accurate and meaningful text annotations. The manual evaluation process ensured that the generated annotations met the following criteria:
provided. This feedback, along with the original
clear description of the objects being manipulated or focused on in the scene.
detailed account of the actions being performed by both the camera wearer and other individuals present in the images.
3. Information about any trajectories movements that take place within the scene.
4. Clear, fine-grained annotations that fulfill these criteria in way that is easily understandable by both humans and machines, such as robots that may use these instructions for task execution.
After several iterations of refining the prompt and evaluating the results, identified suitable template for generating high-quality text annotations. This final template was then used to annotate the image sequences using GPT-4V. More details the prompt design process be found in Figure 5) and for a corresponding high level view on the algorithm, please refer to Algorithm |
We also provide additional details regarding the dataset used in our work. To reiterate, the purpose of using a third-party captioning process was to enhance the granularity of the usual action describing annotations i. make it more fine-grained. We had approximately 33 hours of data and since the
20

---

images are sampled at every second, we had a little over 108000 images.The The images were divided into 'chunks' each of about size 10 such that when they are provided GPT-4V, GPT-4V aware of the full context. Due to connection issues, sometimes we would get an error and so for such cases, we would discard the chunk. In addition, certain annotations also contained the word 'blurred' or 'unclear', which is not useful for egocentric behavior understanding tasks and we tried to avoid such annotations-image pair as much as possible. Annotations typically ranged from 15-20 words and terms like 'individual" and "camera-wearer were common. Their overuse might have contributed to some annotation inaccuracies.
Two human evaluators conducted comprehensive quality assessment throughout the annotation process. During prompt refinement, they evaluated 500 image-caption pairs across multiple iterations. For the final dataset, we implemented a rigorous quality control protocol: each sequence of frame- wise captions was reviewed, and sequences where more than 20% of frames contained flagged annotations (e.g., object/action mismatches, spurious hallucinations, or overly generic descriptions) were discarded. This resulted in an 80% quality acceptance threshold the sequence level.
To summarize, the quality assessment occurred in two stages: (I) during prompt refinement, evaluators focused on identifying systematic issues with annotation quality across multiple clips; (2) for the final dataset where generated sequences underwent manual review using explicit flagging criteria. Annotations were flagged they contained: object/action mismatches, spurious hallucinations, overly generic descriptions, lack of temporal grounding, or terms like 'unclear' / /'blureed'. Only sequences with at least 80% unflagged frame-level captions were retained for training, ensuring high qualiiy supervision signals while maintaining scalability.
Prompt Refinement Examples We arrived our final, optimized prompt through an iterative process designed to enforce frame-wise specificity and rich scene description. The key stages of this evolution were follows:
Initial Prompt: "Describe what is happening in these images." " Output: "A person is making coffee in a kitchen and then talking to another individual nearby." Feedback: Too generic and summary- like; lacks per-frame grounding and temporal consistency.
"Describe what the person is doing in each frame one by one." Output: "Frame No person can be seen in the frame. Frame The person moves toward the counter. Frame 3: The person is talking to another individual." Feedback: Overly focused on person visibility rather than comprehensive scene understanding.
Intermediate Prompt:
"For each image in the sequential series, provide a direct description. Ensure one description per image, capturing the objects, actions, and their spatial relationships. Provide the descriptions in the order presented (output 1 should correspond to image I). Imagine that robot assistant." " Output: "Frame 1: A white cup placed
Final Optimized Prompt:
you need to provide these descriptions on the drip tray under the coffee machine, which is brewing and dispensing a hot beverage. Frame 2: The individual has moved closer to the coffee machine and brewing a cup of coffee. Frame
3: A person in a black shirt can be seen talking to the camera wearer in the kitchen near the coffee machine."
A.2.2 Annotation-Gazz Alignment
To quantify annotation-gaze alignment, we conducted a separate small scale study where human evaluators judged whether GPT-4V descriptions accurately referenced objects/actions in gaze-attended regions. We provided the evaluators with the original image, as well as gaze-overlaid image where the gaze occupied regions (heatmap) were overlaid onto the original image, along with the annotation generated. Under strict matching criteria (i.e the gaze regions accurately match the object/actions in the annotations, and are relevant). , 74% of annotations showed precise alignment. The remaining cases included partial alignments (describing relevant context but missing specific gaze- fixztfi objects) and approximate alignments (capturing general scene context). This conservative estimate demonstrates the overall reliability of our supervision signal, particularly given that our method leverages temporal sequences where gaze-semantic alignment naturally improves across frames.
A.3 Occlusion-Chedd Process
The input image sequence can exhibit dynamic changes due to movement in the environment or the movement of the camera wearer. The method for aggregating gaze points is suitable only when the
21

---

frames within the å interval for which aggregation is done, shows moderate movement. However, preventing movement in a dynamic environment challenging. If we aggregate gaze points in the [t (, /] interval to construct the aggregated heatmap H, and there is major occlusion between the earlier frames [t- 5. f) and the final frame timestamp i, it becomes impractical to include gaze points from the occluded frames. In such cases, collecting gaze points from occluded frames may lead to inaccurate or misleading representations in the heatmap.
To alleviate this issue, we perform an occlusion check between each frame in the 14-6, 6) interval and the final frame at time to ensure appropriate gaze aggregation. In the case of significant occlusion or a drastic change in the scene, gaze points corresponding to the earlier frames should not be collected for the aggregation and subsequent formation of the heatmap H₁.
Using a method similar to the consistency check with optical flow presented by Hur and Roth (2017) explicitly exclude gaze points that occluded in the current frame. If a pixel is correctly translated and there is no major occlusion, then the difference between the forward optical flow displacement of pixel p and the displacement of the translated pixel p with backward optical flow should be close to zero.
For RGB image I at time 6, we gather the image frames } for all E [t 8,t). Let the forward optical flow between images I, and I, in the horizontal direction be denoted by fr and the backward optical flow by fi Let the coordinates of designated pixel be p. The new coordinates of the translated pixel in the subsequent frame, using optical flow, are computed as follows:
P = P + (P)
Next, we calculate the distance moved by this designated pixel in the horizontal and vertical directions according to the following equations:
A= 1= (p) + (p))
If the observed proportion of pixels 7losserved exceeding the distance discrepancy is more than predefined threshold y7. we conclude that a major occlusion has occurred; otherwise, the occlusion is minor. The observed proportion of such pixels Mobserved is calculated as:
HxW
1
Nobservvd
H W Σ
1())
(10)
We disregard the gaze points for frames } where there is major occlusion with respect to the image frame 4 and we represent this filtering using the function O, If the occlusion is minor, the appropriate gaze points }g-} for all T E [t 8, are then translated into their new coordinates using the forward optical flow, and collected for the formation of heatmap H, and subsequently.
As mentioned above, the idea is that if there is major occlusion, the difference between the distance traversed by a pixel during forward optical flow,and the distance traversed by the translated pixel during backward optical flow will be significantly greater than in cases where the occlusion is minor. Optical flow was calculated using the implementation of the RAFT model developed by Teed and Deng 2020)). The hyperparameter is the threshold distance, which was set to 20, whereas v) the threshold proportion of pixels that have exceeded the occlusion limit, set to 0.60. A brief overview of the process can also be seen in Algorithm
2 While optical flow has been employed in prior work for egocentric vision tasks (Min and Corso 2020), our approach fundamentally differs in both implementation and computational requirements. Previous methods typically integrate optical flow computation directly into their model architecture, requiring flow estimation during both training and inference phases. In contrast, we leverage optical flow exclusively during the proccscens stage to enable temporal aggregation of gaze signals across adjacent frames. This allows us to construct more robust gaze heatmaps by compensating for minor head movements and aligning fixations across time. Crucially, by confining optical flow to pre-processing, our method eliminates the need for computationally expensive flow models like RAFT during inference, resulting in significantly faster deployment and reduced computational overhead at test time.
22

---

Algorithm 1: Iterative Prompt Refinement for Fine-Grained Annotation
Validation image subset V (V,....Pe Vn Full image set I (I,....,Zy ]
Input:
Initial prompt Po:
GPT-4V model; ChatGPT interface
Output:
Final refined prompt p
Annotations ( T
P Po:
repeat
Generate annotations Tv GPT4V(VV P);
Manually evaluate TV for:
Object relevance and accuracy
Actions of camera wearer and others
Movements or spatial cues
Clarity and fine granularity
Provide feedback F on Ty;
Refine prompt: <- ChatGPT(P, F);
until annotations V are satisfactory;
P PP;
// Final prompt after refinement on validation set
for each batch I (j = 1 to k) do
Generate final annotations:
T, + GP, P*);
return P*, (Ti,.... Tk)
Algorithm 2: Occlusion-Aware Gaze Point Aggregation
Input:
Image sequence (7>) for E 6, Gaze points {9>} at each timestamp T,
Optical flow model (RAFT),
Thresholds = 20, r7 = 0.60
Output: Aggregated heatmap H+
Initialize H, as empty
for r = to do
Compute forward flow fr-t and backward flow fi-ri for pixel p I, do A p f (p):
fc + +
Mark p occluded if A
Compute occlusion ratio: 1Jbbeerved
Hxw Σ e);
if 1observed 7) then
10
Discard gaze point g- (major occlusion);
else
12
Translate to gr-+e using fr
13
Add Gaussian heatmap from gr to Ho;
Normalize Hú;
15 return H,
A.4
Model Overview and Components
In our study, we initially employ the open-source version of the Flamingo model (Awadalla et al.) 2023), based the original Flamingo architecture proposed by Allayrac et al, (002)). Flamingo is VLM designed to process interleaved image-text inputs, featuring pre-trained vision encoder, trainable Perceiver Resampler for constructing fixed-length visual tokens, and cross-attention
23

---

Ego-centric video clip
A person's hands is holding drink... An individual's hands is grabbing a drink from the kitchen island.
Base Model
Ground-truth text annotations
Predictions
individual's hand reaching
towards a pack chocolate coated nuts kitchen island counter while drink and black screen be seen behind. An Individual has picked up the chocolate nuts and is looking at the bar code scanner which is on the label of the packet while the drink on the table.
Gaze Regularized Model
A hand is reaching for pack of chocolate on the kitchen island. Individual is grabbing the chocolate snack placed on the kitchen island counter near the drink.
Heatmaps for training
Figure 6: Simple Illustration of Activity Understanding. The model receives sequence of egocentric frames input and generates fine-grained prediction of the current activity. While the base model misidentifies the object being picked up ('a drink'), ) the gaze-regularized model correctly predicts "chocolate' by leveraging attention aligned with human gaze during training. Importantly, spatial gaze heatmaps are used only to guide attention during training and are not used at inference. Predicted annotations are shown on the right, with ground-truth descriptions below.
language decoder. To extend its capabilities for egocentric video understanding, we introduce gaze- regularized attention mechanism that is placed before the Perceiver Resampler, enabling alignment between model attention and human gaze distributions.
Building on this design, generalize our gaze-regularization framework other transformer based VLMs, including LLaVA, InternVI and La LaViLa, by modifying their input pipelines as required to accept a sequence of image frames instead single frame. Our core idea remains consistent across models: a gaze-based attention regularization mechanism is applied prior to fusing visual tokens with language decoder, modulating attention to emphasize gaze- paze-informed spatial regions.
is important to note that while we adopt the architectures of models like LLaVA and Intern we do not their original pretraining paradigms loss functions. As such, the performance reported for these models should not be interpreted as a ranking of their full potential. Rather, our intention is demonstrate the generalizability and impact of gaze regularization on different VLM architectures under a consistent training setup.
Inspired in part by the transformer- bassod egocentric gaze modeling approach of Lai al. 2024) method uses gaze heatmaps only during training to guide attention. During inference, the models rely solely on RGB image inputs, making them suitable for deployment scenarios where gaze data is unavailable. A simple illustration this setup, along with example outputs for activity understanding. shown in 66
To further ensure robustness, we incorporate an occlusion-check module based on optical flow consistency that filters unreliable gaze points during temporal aggregation. This improves the fidelity of gaze supervision and ensures that only visually relevant regions guide the model's attention. The resulting framework enables gaze-guided fine-grained predictions and current activity understanding across multiple VLMs, while maintaining modularity and scalability.
A.5 Other Experiments
In this section, we present series of ablation studies that explore the core design choices in gaze-regularized framework. We begin by defining the evaluation metrics used to measure semantic alignment between predicted and ground-truth activity descriptions. Next, conduct a sanity-check experiment to evaluate whether existing pretrained VLMs are capable future activity prediction in egocentric settings.
We then analyze how the size and temporal density of gaze points affect model performance by varying size and number of gaze points used for heatmap aggregation. To disentangle performance

---

gains due to architecture versus gaze supervision, we augment the base model with additional self- attention layers but without gaze regularization. We also test whether providing gaze coordinates in textual form (rather than as spatial heatmaps) yields comparable results.
Finally, we examine the effect of the observation horizon. Most of the ablations conducted using the OpenFlamingo architecture as the base model, as it was the model initially used to get some preliminary results and intuition about how to proceed further. Once the framework was validated, we generalized it to other architectures, as discussed in the main paper.
A.5.1 Architectures Used
We tested our method using a set of established open-source models, including OpenFlamingo, La LaVila Intern VL 2.5-IB, and Ope (LLaVA-NeXT-Viemm) strong and diverse foundation for our evaluation. Their clear documentation and straightforward implementation made them practical choices for our experiments.
These models provided
A.5.2 Evaluation Metrics
For evaluation, we propose using a semantic transformer (Reimers and Gurevych 2019 quantifiable score that compares the generated output with the ground-truth action text, along with the ROUGE-L scores for some tables. This scoring system designed ensure that the model not penalized for semantically equivalent phrasings. For example, 'Football is being played by people' and PPoplee are playing football' convey the same meaning and should yield similar scores. At the same time, the system should penalize outputs with nonsensical or incoherent word order. While some ablation tables in the appendix report both semantic and ROUGE-L scores for completeness, omitted ROUGE-L from the main paper due to space constraints and because it exhibited highly correlated trends with the semantic scores. This allowed us to present a more concise evaluation while still conveying the key performance improvements.
provide
A.5.3 Evaluating Pretrained VLMs for Future Activity Prediction
Pretrained VLMs such as Vn and OpenFlamingo have demonstrated impressive performance on a variety tasks, including visual question answering, image captioning, and multi-moda reasoning. However, these models have not been explicitly designed or optimized for future activity prediction. To assess whether such models can be directly applied to short-horizon future prediction, we conduct sanity-chect experiment: provide each VLM with sequence of image frames and evaluate its ability to generate predictions about future actions.
Table 8: Evaluation pretrained VLMS for activity prediction
MODEL
SEMANTIC SCORE (t)
INTERN LL L-2-1B BB INTERN VEE L-2-2B
0.1572
0.1601
INTERNVI.-2.5-1
0.1596
OPENFLAMINGO-3B
0.1810
OPENFLAMINGO-4B
0.1878
OUR METHOD
0.7505
While this evaluation is not entirely fair,given differences in training objectives, data domains, and task formulation jit offers a useful diagnostic of how well existing models transfer to our setup. As shown in Table [ current VLMs exhibit limited performance on our test set. This may stem from domain shift the lack of temporal modeling their pretraining. Nonetheless, the results highlight promising opportunity for future work to extend VLM capabilities toward temporally grounded tasks like short-horizon future activity prediction.
A.5.4 Impact of Gaussian Overlay Size in Singular vs. Aggregated Gaze Models
In our exploration of gaze-regularized models, we investigated two primary strategies for representing gaze: (i) the singular gaze model, which uses single gaze point per frame to form spatial heatmap.
25

---

Future Frames
Ground Truth
Predictions(base)
Predictions (ours)
going fron dack and mobile phane screen vaibbe fronr them The indiriiit is going read the rod book in tront hes, poosinay oftee neing usiedd by camert weare.
laginng
bi on the sable individuall tront.
sested parson with the mobile phone
The souted personia ii the mnwttakggouu in front the cartigeme gurm with bosk.
visible The personi gringto to read
nad book wfiich near dackk.
The weener's
piassii pasket with and a deck of canda instt and the personiin blus shart will move the Usiin room The inchetust la go to removeia deskkot cords plastic pasket
The indvécust speniing
The individ,
deck
Biving noomn with moto on in the traibar-gyruud
blun shirt will mov inskde thes livingromm
The indivicat la removinng paper iraide
The currera wearet la going
plastie
remove cards from plastio..
An in hamdiwot neach temaadte e pack of chocolatte
will
reach for of
indanduatie hand diink from the kits islandd
the letchen Island. Inchiicual goiiing
seen behindsAA An indivddual la goiing chocotute nuts and will took the bar code acanner writch lxin the lbbll the parter while the diine in the tablis
grab the chorolate snoch placed on
titthen island courner near the drinki.
the carrass meaeer parzs cards the parsar the player cards the table..
Apperseels continuus will tablo.
Ther
person with headeett The caed came condiuueo player will place card the table.
The rehurtiants hand is going to neach a brown container in
The indiicuee handwwll neach for container bettom will grab the
The indiuttud hanclee meaching for brrww contai bellw glansen
front the wittts cup bel seree giocoss bettom shatf ponsitty Theindivtuulle going gros the browes conttiierr
glossos The Incul hand
the cobiieet The divii estel
shhil.
will toweerds kitthhn with spoon in handd. The inckviuual logiing use the spoon to get
The coreea weerer Wlll reassh for contotier
The Inciudial movingg the kdtchen Ialand a spoon The tndivetuar will
on the kiti hes counter, The indivdualwil will
sore sugar from container on the sobio full of enoolo ondio
grab seme enaells the tabile and
greb the sugar from container using spoon neev wfw. efleooeeeed
be
beverage antte
Figure 7: Extended Qualitative Results of Future Activity Prediction. Examples comparing predictions from the base model and our gaze-regularived model against ground-truth annotations and actual future frames. The gaze-regularized model correctly predicts specific objects and actions (e.g., "picking up chocolate") while the base model makes less accurate predictions (e.g.. incorrectly predicting "a drink"). In addition, the gaze regularized model also gives finer-grained predictions by correctly predicting the color (e.g., "brown container"), objects surrounding the main object (e.g.. "below some glasses"), which the base model fails to do in the predictions. By highlighting such properties, giving instructions to other humans or machines becomes easier, especially if there are multiple such objects involved. Key words in the predictions are highlighted to emphasize differences in prediction specificity and accuracy.
and (ii) the aggregated gaze model, which combines gaze points from temporal window with occlusion filtering.
To assess the role of spatial spread in these representations, we varied the standard deviation (σ) of the Gaussian kernel used to generate the gaze heatmap for both models. In the singular gaze model, increasing which expands the influence region around each gaze point led to modest performance improvements. This suggests that slightly broadening the gaze-induced attention might help the model focus semantically meaningful areas rather than overly localized pixels.
In contrast, increasing the Gaussian smoothing the aggregated gaze model had minimal impact. As shown in Table 9 its performance remained nearly unchanged. This indicates that the strength of the aggregated model stems from its temporal diversity and occlusion-awand gaze selection, rather than from the spatial extent of individual gaze contributions.
These experiments were initially conducted using the OpenFlamingo architecture, which served as our primary reference during early ablation studies.
Overall, these findings highlight an important distinction:
while spatial smoothing can benefit isolated gaze representations, temporally aggregated gaze signals already encode spatial redundancy via multiple temporally aligned points. Future work may investigate how the interplay between Gaussian spread (σ), image resolution, and aggregation window size affects performance.
A.5.5 Human Evaluation of Visual Hallucinations
To assess whether our gaze regularization method mitigates visual hallucination in VLMs, we conducted controlled human evaluation. This analysis was motivated by the fact that standard
26

---

Table 9: Effect of the size of the gaze point overlays model accuracy
MODEL
a
SCORE(()
F-SCORE
SINGULAR
20
0,6211
0.4327
SINGULAR (LARGER OVERLAYS)
50
0,6512
0.4463
AGGREGATED
20
0.7505
0.5173
AGGREGATED (LARGER OVERLAYS)
50
0.7436
0.5034
hallucination benchmarks (e.g., HallusionBench (Guan et al., 2024)) do not contain gaze annotations surrogate attention signals, making direct application of our method infeasible these benchmarks.
We randomly selected 200 examples from our test set and generated textual outputs using both the base model and our gaze-regularized model. These responses were then randomly shuffled and presented to human evaluator who was blind to the model source. For each example, the evaluator was shown the corresponding ground-truth video context and annotations, enabling reliable identification hallucinated content.
The evaluator was instructed to flag any instance where the model's output mentioned objects or actions that were either:
1. Clearly absent from the visual input, or Unrelated to the actual scene content
Following the evaluation protocol from HallusionBench, we computed the C₁ score, defined as the ratio of annotations with clearly hallucinated instances to the total number of annotations:
Table 10: Human evaluation results for visual hallucination. Our gaze-regularized model shows
significant reduction in hallucinated content.
Model Variant
Hallucinated Cases
Total Samples
C₁ 0.205
Base Model Gaze-Regularized Model
41
200
28
200
0.
This improvement suggests that aligning the model's attention with human gaze patterns helps ground the language generation in visually present content, reducing the tendency to hallucinate objects or actions not supported by the visual evidence. While this evaluation was conducted only on our dataset, the findings indicate the potential of gaze-guided training for addressing hallucination in VLMs more broadly. Future work could explore generating proxy gaze signals to enable evaluation on standard hallucination benchmarks.
A.5.6 Impact of Using More Gaze Points in Temporal Aggregation
To explore the impact of using a larger number of points (or a longer aggregation duration), trained an aggregated gaze model with the openflamingo architecture utilizing 12 frames instead of (and hence the aggregation time 8 is 400 ms). The results of this experiment are provided Table. [T] This minimal improvement is likely due to the occlusion checks already filtering out less informative points, reducing the marginal utility of including more frames and limiting the number of usable gaze points. In cases with minimal occlusion, the slight decrease in performance can be compared to that observed in an aggregated gaze model with larger overlays, where performance also decreased slightly. This result suggests that maybe utilizing an excessive number of gaze points could confuse the model. However, as this explanation is currently based on intuition rather than extensive analysis, this experiment was not included in the main paper.
Table 11: Comparison when more aggregated points are used for the gaze-regularized model
GAZE POINTS
SCORE(t)
F-SCORE(1)
6
0.7505
0.5173
12
0.7398
0.4971
27

---

A.5.7 Impact of Occlusion Filtering on Model Performance
In the aggregated gaze model, gaze points are collected within a specified time interval 8 (200 ms). However, in dynamic environments, aggregating gaze points from the interval [t - 6, t] may introduce inaccuracies due to changes in the scene or camera movement. To mitigate this, we introduced an occlusion check to ensure that only relevant gaze points are aggregated. More details about the occlusion check method can be found in Sec. A.3 in the Appendix. To evaluate the impact of this adjustment, we conducted experiments comparing models with and without the occlusion check in the future prediction task. As shown in Table 12 the openflamingo implementation of our framework incorporating the occlusion check slightly outperforms the one without it. The difference in the evaluation metrics can be attributed the fact that only relevant and accurate gaze points considered, which reduces noise and prevents the model from being confused by irrelevant data.
Table 12: Effect of using occlusion filtering during training
MODEL
ScoRE
F-SCORE (t)
GAZE-REG (W/O OCCL.) GAZE-REG (W/ OCCL.)
0.7298
0,4800
0.7505
0.5173
A.5,8 Can Gaze Coordinates alone improve Model Performance?
In our studies, we converted gaze data from coordinate text form into heatmaps, which were then utilised by our gaze regularizer during training time. This transformation allows the regularizer to highlight important visual regions and aligns more closely with how humans perceive spatial information (Leng et al., 2014). To conduct a sanity check and assess whether using gaze data in visual form is more suitable than using gaze in text form we trained a gaze-regularized model that utilizes gaze coordinates as text input. Our results indicated that using gaze data in text form improved performance compared to the base model without gaze. However, it still fell short of the performance achieved by our model (as shown in Table (3). This highlights the importance of not just including gaze data, but representing it in a specific format such that it can be used aligned to modulate the spatial nature of model attention.
Table 13: Effect of using gaze information in text form and comparison with other models
MODEL
SCORE(t)
F-SCORE (t)
BASE
0.6525
0,4318
AGGREGATED GAZE(IN TEXT FORM) AGGREGATED GAZE
1
0,4630
0.7505
0.5405
A.5.9 Addition of Self Attention Blocks to the Base Model
To ensure a fair comparison, we evaluate whether the performance improvements in our gaze- regularized model stem solely from its architectural enhancements. specifically, the extra attention layers that operate on global query derived from the full image sequence. To test this, we augment the base OpenFlamingo model by adding two self-attention layers and providing it with the same global query, but without applying any gaze-hased regularization. This allows us to isolate the effect of architectural changes from the influence of gaze supervision. As shown Table 14 the modified base model does show improved performance compared to the original baseline. However, it still falls short of the performance achieved by the gaze-regularized models. This suggests that while incorporating global queries and self- attention blocks enhances the model's 's capacity to aggregate contextual information, it is the gaze-guie gaze-guided regularization that ultimately drives stronger alignment with human attention and increases model performance.
A.5.10 Training Base VLM with Gaze-Embedded Images
As sanity check, we evaluate what happens when baseline models are trained on gaze-embedded RGB images, where gaze heatmaps directly superimposed onto the original RGB frames. This
28

---

Table 14: Comparison of base model with attention block against gaze regularized models
MODEL
SCORE(1)
F-SCORE(1)
BASE
0,6525
0,4318
BASE (W SELF-ATTENTION)
0,6701
0,4393
GAZB-RBOULARIZZE
0.7505
0.5173
setup allows us to assess whether performance improvements could arise simply from exposing the model like visual cues, without any explicit use of gaze during training inference,as is the case in our gaze-regularized framework, which does not take gaze as input, All models were trained under the same conditions, using identical data splits. As shown in Table using gaze-embedded inputs yields a modest performance gain over standard RGB inputs. However, this improvement remains well below the performance achieved by gaze-regularized models. These results suggest that the benefits of our approach stem from the explicit use of gaze as a training-tinn -imiingg-mmeeee regularizer, rather than from simply incorporating gaze like patterns in the input images.
Table 15: Comparison for egocentric event prediction when gaze-overlaid images are provided to base VLM.
MODEL
SCORE(1)
F-SCORE
BASE W RGB IMAGE
0.6525
0.4318
BASE W GAZE- EMBEDDED IMAGE
0.6873
0.4435
GAZE REGULARIZED
0.7505
0.5173
A.5.11 Effect of Temporal Gaze Aggregation on Performance
As described in the main paper, construct gaze heatmaps by first transforming individual gaze points into spatial maps and then aggregating them over a short temporal window. This aggregation is accompanied by occlusion filtering ensure that only visually valid gaze points contribute the final heatmap used for attention regularization.
To understand the specific contribution of temporal aggregation, compare performance against baseline that uses only single gaze point at each rifeeee to as the singular gaze model. In this case, the gaze heatmap H, at time / is defined as:
(11) is an indicator function with value 1 at position 9: and 0 elsewhere, G, is a Gaussian kernel with standard deviation a, denotes convolution, and π represents a normalization so the heatmap sums to 1.
H: = m, (G 1(g:))
Our experiments show that aggregating gaze points over a short interval (e.g., 200 milliseconds) consistently improves performance over the singular gaze variant. This improvement arises be- cause temporal aggregation captures the stable and temporal structure of visual attention across frames, whereas frames,whereas fleeting fixations. addition, over-regulariizatio of attention a small space (pertaining to a singular overlaid point) might not always be beneficial, as shown by the performance degradation as compared the baseline model where no gaze regularization is used.
single gaze points can be often noisy, particularly if sampled during saccades or
By integrating gaze information from multiple frames (and by including an occlusion filtering mechanism), the model gains richer and more reliable supervisory signal that better aligns its attention with human intent. This temporal consistency enables more accurate understanding of ongoing actions and better prediction of upcoming behavior. A simplified illustration comparing singular and temporally aggregated gaze heatmaps is shown in Figure 8
A.5.12 Change in Observation Horizon for Future Prediction
In the main paper, we reported results for future prediction tasks using an anticipation horizon of Ta 2 seconds and observation horizon of To 5 seconds. To further analyze the model's 's behavior,
29

---

Table 16: Semantic performance on future prediction and activity understanding tasks comparing singular and aggregated gaze models. Gains indicate the absolute improvement from using aggregated gaze.
FUTURE PREDICTION
ACTIVITY UNDERSTANDING
MODEL
SINGLE
AGGREGATED
OPENFLAMINCR
0.6512
0.7505
+9.9%
0.6913
0.7848
+9.4%
MODIFIED OPENFLAMINGO
0.7200
0.6631
0.7300
LAVILA NARRATOR
0,6037
0.6924
+8.9%
0.6364
0.7268
+9.0%
0.6813
OPENLLAVA
0,5814
0.6771
49.6%
0.6500
0.7027
+5.3%
Aggregated
Singular
Figure 8:
Illustration of singular and aggregated gaze heatmaps. For the singular gaze heatmap. only the gaze point associated with the final heatmap is utilized. On the other hand, for the aggregated gaze heatmap, gaze points over an interval 8 200ms are collected to form the aggregated heatmap.
conducted an additional experiment by reducing the observation horizon to T, = seconds i.e., using shorter input sequence to predict the same future window. Interestingly, this reduced observation window led slightly improved performance for some models, while maintaining consistent overall gains when gaze regularization is used as opposed to a baseline model, shown in Table 17
This suggests that even with limited visual context, gaze-regularized models can effectively anticipate future actions. Interestingly, while humans leverage visual working memory for short-term planning, this memory may not effectively simulated over longer temporal horizons in our current framework. As result, tracking the visual cues may become harder as the anticipation window grows, leading to reduced predictive accuracy. While this remains a speculative explanation, it motivates further investigation into integrating memory inspired mechanisms into our gaze- anse-inspired framework. Accordingly, we present these exploratory results in the appendix rather than the main paper as the explanation is intuitive and not experimentally investigated.
Table 17: Semantic performance on future prediction task with reduced observation horizon (3 seconds). The aggregated gaze model continues to outperform the base model across all architectures.
MODEL
SINGLE
GAIN AGGREGATED
OPENFLAMINGO
0.6616
0.7565
+9.5%
MODIFIED OPENFLAMINGO
0.6200
0.7281
+10.8%
LAVILA NARRATOR
0.6144
0.6966
+8.2%
INTERNVL
0.6301
0.7279
+9.8%
OPENLLAVA
0.5851
0.6731
+8.8%
A.5.13 Activity Understanding for Static Image
We examine whether a static scene, independent of sequential context, is sufficient for the gaze- regularized models understand an ongoing activity, noting that VLMs are often used static-scene setting. While single image lacks temporal context, it contains visual cues and object affordances that hint at probable actions. Unlike future prediction, which benefits from evolving gaze patterns, static scene understanding relics mainly visible objects. If gaze is beneficial, it should highlight key
30

---

regions that aid activity understanding. To test this, we compare the base and gaze-regularized models in generating fine-grained descriptions from single images, isolating the impact of gaze without temporal cues. Results from Table T8 that the gaze-regularized model achieves modest 1.3 % improvement over the base model, indicating that gaze has moderate impact on scene understanding in single-image settings. However, its full potential is realized in tasks where evolving gaze patterns provide richer contextual cues, Hence for the activity understanding tasks in the main paper, we provide long enough observation horizon = 3s such that the activity can be understood, as well as to utilize the evolving temporal gaze patterns.
Table 18: Comparison between base model and gaze-regularized models (with different regularization coefficients) for activity understanding tasks.
MODEL
SCoRE
F-SCORE
BASE
0.6897
0,4432
RAGLAREG
0.7014
0.4611
A.5.14 Effect of Using Sequence-Level vs. Frame-Level Queries for Attention Computation
Table 19: Impact of using sequence-level queries semantic prediction for activity understanding. Using global queries from the entire input sequence improves performance on both in-distribution and
data.
MODEL VARIANT
SCORE (TEST)
SCORE (OOD)
REZZERREGLARRZEDD (w/o GLOBAL QUERY) DAZE-REGULARIZEL (WITH GLOBAL QUERY)
0.7284
0.6902
0.7505
0.7305
In our main framework, we compute a global query using the entire input image sequence, rather than generating separate queries for each individual frame. This choice supports better temporal generalization. If frame-level queries are used, the model processes each image independently, without awareness of the broader activity context. As a result, the attention mechanism may become overly dependent on visual patterns specific to individual frames.
This becomes particularly problematic in tasks like future prediction, where the same frame could occur in different activity sequences. Without access to temporal context, a frame-level query cannot distinguish between these scenarios, increasing the risk that the model memorizes attention patterns instead of also relying learning task-relevant cues. In contrast, a sequence-level (global) query captures the dynamics of the entire clip, encouraging the model to focus on features that reflect the underlying activity rather than frame-specific objects. From Table the results support our intuition: incorporating global queries not only improves semantic scores but also results in smaller performance drop on out- -ot-of-diitribttion data compared to using frame-wise queries.
A.5.15 Generalization to Other Egocentric Tasks
Challenges with Native Ego4D Benchmarks Our method requires gaze supervision during train- ing, which limits us to the subset of Ego4D videos containing gaze annotations. While Ego4D provides extensive benchmarks for activity classification, video-text retrieval, and forecasting, these benchmarks primarily use videos without gaze data. Furthermore, the gaze-annotated subset lacks the structured labels required for these benchmarks (e.g., verb/noun labels, bounding boxes, or precise temporal segments). This inherent dataset limitation prevented direct application of our method to standard Ego4D) benchmarks. However, we provide couple of additional experiments below to highlight the efficacy of our method, by using pseudo-gaze or predicted gaze signals (using a third-party pre-trained gaze prediction module) even for the videos without gaze but the other structured labels, as well as coarse narration prediction task- utilizing the original coarse narrations present in the subset for gaze data to show the improvements made by the gaze regularized model over the base model.
Coarse Narration Prediction Task this experiment, we leveraged the narrations available in the gaze-annotated subset. These narrations provide coarse, first-person descriptions of activities
31

---

Input Frames
Attention gaze
Annotations (for reference)
cameen going converse
person
deck of cards and mobile phone visilein front them. The individual goingto road red book Infront passibly being by
mobile
The camere wearer goingto open a plastic packet with paper and cards person living room. going remove deck of cards from plastic packet.
Attention: On and objestiposket with cards)
individ mach towerds pack of
kitchen island counter while An going pick up the chocolate nuts and wid look the bar coda scannerwtichh on labal ohe packer the drink is on the table.
Attention: On chocolate tapt the
drink
The individual's hand going to reach for a brown container in front the white cup and bolow some glassos the bottom shelf of the cabinet. The individuals going to grab brown container possialy sugar from
Attention conteiner
The camera a spoon in hand. The indviduuull going use the spoon to get some sugar from container on the table full enaccs table.
towards the kitchen island with
beverage
Attestoo:: container and drink
Figure 9: Extended Comparison Attention Maps. This figure compares attention distributions from the base model (without gaze regularization) and our gaze-regularized model. Our method produces attention maps that are more semantically aligned with human gaze, leading better focus on task-relevant regions. For instance, in the second-to-last and the container-objeets emphasized in the ground truth annotation. the final row, attention is correctly directed toward the hand and the container, improving the model's ability to interpret and predict the ongoing activity.
the model attends to both the glasses
recorded by camera wearers. We defined coarse future narration prediction task to evaluate whether our approach generalizes to scenarios with less detailed supervision.
As shown in Table 20, our gaze-regularized model achieves modest but consistent improvements over the baseline, with 3. 3.9% increase in semantic score and 2.2% in F-score. The smaller gains compared to fine-grained prediction tasks (which showed 7-10% improvements) support our core hypothesis: gaze supervision is particularly effective for capturing subtle spatial and object details that coarse labels often overlook.
Table 20: Performance coarse future narration prediction task. Gains modest compared to fine-grained tasks, supporting our claim that gaze helps capture subtle details.
Model Variant
Semantic Score
F-Score
Base Model
0.7232
0.4982
Gaze ReazuaRegul Model
0.7621
0.5200
Generalization with Predicted Gaze without human gaze, we applied it to the Ego4D Short-Term Object Interaction Anticipation (STA) benchmark using predicted gaze heatmaps. We employed an off-the-shelf gaze prediction model to
To test our method's applicability to standard benchmarks
32

---

generate pseudo-gaud gaze signals, which were then used for attention regularization during training via our standard KL-divergence loss.
As shown in Table 21 our approach improves performance on this established benchmark, with absolute gains of 2.88 mAP for noun prediction and 2.16 mAP for verb+noun prediction. These results demonstrate that our gaze regularization framework can generalize to tasks without human gaze annotations, suggesting that even approximate attention signals can provide beneficial supervision.
Table 21: Results Ego4D Short-Term Anticipation (STA) benchmark using predicted gaze. Our
method shows consistent improvements despite using surrogate gaze signals.
Model Variant
mAP (Noun) 1
mAP (Noun Verb)
Base Model
22.13
11.29
Razz-Regularized Model
25.01
13.45
These experiments collectively demonstrate the versatility of our approach: it provides substantial benefits for fine-gre one-grained understanding tasks with detailed annotations, modest improvements for coarse prediction tasks, and remains effective even when using predicted gaze signals on standard benchmarks.
Our Model
Camera appears blurry. Image is moving fast and blurred.
Camera wearer which contains items, most probably ketchup bottle. The scene appears to be moving fast and blurred.
refrigerator
activity majority input sequences blurred.
Ground Truth (Future Prediction) Camera wearer going to talk to people and look at the snacks on the white table counter. Camera wearer will move to a blurred scene which looks like kitchen with
Example of ground truch annotations containing the word 'blur' which is not useful
real life deployment for prediction and hence had to be removed.
Camera placing white cup waiting for beverage. Camera wearer has placed white cup under coffee machine for beverage.
Camera wearer still waiting with white cup placed under stainless- steel coffee machine
Potential examples where second Interval actions change
drastically
Figure 10: Example failure case and qualitative observations. Top: When the majority frames in the input sequence significantly blurred, even gaze-regularized models struggle produce accurate predictions. Although the ground-truth annotations preserve contextual intent, the lack of clear visual input limits the model's ability to of image-text pairs that were excluded during dataset construction because they contained words like "blur" or "unclear," which offer little utility for activity understanding or future prediction. We aimed to minimize the inclusion of such samples. Bottom: A case where the actions remain largely consistent across the sequence, making the activity easier infer regardless the modeling approach.
about ongoing actions. Middle: Examples
Training and Evaluation details Both the base model and the gaze-regularized model were trained using two NVIDIA A800 80GB GPU cards. For initial experiments, we used the OpenFlamingo architecture develop and evaluate our approach. The base OpenFlamingo model required approxi- matcly 36-38 hours to train, while the corresponding gaze-regularized version took around 50 hours.
33

---

This additional time is due to the added regularization computation and the insertion of gaze- -azze-l attention blocks. Training was conducted with batch size of 32 and learning rate of 7 x 10-5 over 10 epochs. The vision encoders were kept frozen and pre-trained. To accelerate training, we employed Fully Sharded Data Parallel (FSDP), which efficiently distributes model parameters and gradients across GPUs, improving memory usage and speed. Data loading was managed using the WebDataset loader, with datasets converted to .tar format for compatibility with both WebDataset and FSDP. After validating our method with OpenFlamingo, we extended the same training pipeline to other architectures such as InternVL, La VaLa, and OpenLLa VA. In each case, the integration of our gaze-regularized component followed the same principle: it was inserted immediately after the visual encoder and before the language decoder, allowing for modular modulation of attention without disrupting the rest of the model architecture.
To give comparison in terms of model complexity, the base model for openflamingo has approxi- mately 944 million parameters. The gaze-regularized model with 2 additional attention blocks, which the best performing model, has approximately 966 million parameters. For evaluation, models were assessed using the semantic transformer (SBERT) developed by Reimers and Gurevych| (2019), along with ROUGE-L scores. Regarding the runtime, we would like to clarify that the evaluation was conducted using RGB images for the base model and the gaze-regularized model. On average, the base model took approximately 1.7-1.8 seconds to process a sequence while the latter took 2.2 2.3 seconds. However, this runtime does not include the time taken for image loading pre-processing overhead. The testing was also conducted on a NVIDIA A800 80GB GPU card.
A.6 Discussion and Limitations
In the following section, we briefly discuss some limitations of our current framework and outline promising directions for future work.
While our gaze-regularized framework offers strong improvements in egocentric understanding tasks, certain constraints remain. To maintain computational efficiency, we downsample videos frame per second. Although this preserves high-level temporal patterns, it may limit sensitivity to rapid or transient actions. Nonetheless, we find that the current performance and predictions capture egocentric behavior understanding in majority of the scenarios.
A common challenge in egocentric vision is the presence of motion blur or poor visual quality. In sequences where most frames are heavily blurred, even razeeregularized models struggle to generate accurate predictions, as both visual and gaze cues become less informative. Figure T0 (llluustrates such a case. Even though we tried to identify most of such ground truth annotations and mitigate them, some cases still persist and this one of the limitations.
While our occlusion- -wware gaze filtering improves robustness, the reliability of gaze supervision still depends on accurate calibration. Misalignment in eye-tracking data can lead to suboptimal heatmaps, which in turn may misguide the attention regularizer. For training the models from scratch using custom data, it is necessary to ensure that the gaze calibration used for collecting training data is in tune with the camera wearer. Our findings offer encouragement and also point towards the need for deeper exploration of how gaze aligns with visual representations in VLMs. For example, recent observations in [Bolya et al. (2025) suggest that the final-layer features of vision encoders may not always be ideal for all downstream tasks. This raises an interesting parallel: just as feature selection matters in model design, it is worth asking whether certain stages of gaze processing e.g., early fixations vs. late context scanning ,aiggn better with specific types of reasoning, such as classification, visual search, or question answering.
This leads to promising direction for interdisciplinary collaboration. Psychological research has long shown that gaze behavior is task- maskdeepnddnt: humans scan scenes differently when searching for object versus answering question. Extending this insight to VLMs could help define how attention should behave in task-specific settings. By incorporating gaze collection protocols tailored distinct tasks, we may be able to design models that not only align better with human attention but also adapt their internal focus in a task-aware manner.
In summary, our framework provides a strong foundation for integrating gaze as a training signal in egocentric VLMs. Future work can build on this by exploring task-specific attention modeling, leveraging insights from both vision science and cognitive psychology, and investigating how different forms of gaze data interact with visual representation choices across diverse downstream tasks.
34


Reviews:
[{"review_id": "W48cF6tcis", "reviewer": "Reviewer_qTVN", "summary": "This paper proposes a modular gaze-regularized attention mechanism to enhance VLMs for egocentric behavior understanding, particularly future prediction and current activity recognition. The authors leverage eye gaze information during training to align model attention with human visual focus using KL divergence loss. The method is simple yet effective and broadly applicable to a range of transformer-based VLMs. Extensive experiments across five models and two egocentric datasets show consistent performance gains in both in-distribution and out-of-distribution settings.", "questions": "Could the authors clarify which attention layers are regularized, and whether any comparative analysis was conducted to motivate this selection?", "limitations": "No limitations are mentioned.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strength:\n1. The idea of using gaze information as a training-time regularizer is interesting.\n2. The designed modular attention block that can be easily integrated into different VLM architectures.\n3. This framework gains higher performance on a suite of models on two benchmarks.\n\n# Weakness:\n1. Ablations: The novelty of the proposed gaze-regularized attention hinges on several key components, such as the temporal aggregation and occlusion-aware filtering of gaze signals (Section 3.1). However, the current ablation analysis is limited. It would strengthen the paper to include more comprehensive ablations that isolate the contribution of each component. In particular, since the model relies on additional eye-tracking annotations, it would be informative to compare performance when using raw gaze data directly versus using the proposed/simplest aggregation (average or max).\n2. Model generalization: It is unclear whether gaze alignment also mitigates visual hallucinations, which is an important concern in VLMs. Could you please evaluate the proposed model on hallucination benchmarks like hallusionbench[1], pope[2] or others?\n3. Related Work on VLM Attention: The paper could better contextualize its contribution by discussing prior work that analyzes or manipulates attention in VLMs[3][4]. \n[1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models, CVPR 2024\n[2]Evaluating Object Hallucination in Large Vision-Language Models, EMNLP 2023\n[3] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024\n[4] Perception in Reflection, ICML 2025", "was_revised": true, "final_justification": "The idea of using gaze information as a training-time regularizer is interesting. The proposed modular attention block is well designed and can be easily integrated into different VLM architectures. This framework achieves higher performance across a suite of models on two benchmarks.\n\nAlthough the work only focuses on the egocentric perspective, it presents a relatively solid treatment of VLM attention, supported by extensive ablations. The method is applied to VLMs, and extending it to a broader range of perception tasks could further strengthen the paper. Overall, I maintain my support.", "rebuttal": "We thank reviewer qTVN for the clear summary and constructive feedback. We appreciate your positive assessment of the idea of gaze-based training-time regularization, our modular attention block design, and the performance improvements across various architectures. Below, we address the specific comments and questions.\n\n**(1) Component-level ablations (temporal aggregation, occlusion filtering, gaze representation)**: \n\nThank you for highlighting the importance of isolating these components. We share the same insight and provide comprehensive ablations in the appendix  (Sec. 6.5) including:\n\nImpact of temporal aggregation vs. singular gaze (Table 14): Aggregated gaze consistently outperforms single-frame gaze. This happens since single gaze points can often be noisy especially during saccades or fleeting fixations, whereas the aggregation captures the temporal structure of human attention across frames.\n| Model Variant   | Future Prediction ↑ | Activity Understanding ↑ |\n| --------------- | ------------------- | ------------------------ |\n| Singular Gaze   | 0.6512              | 0.6913                   |\n| Aggregated Gaze | 0.7505              | 0.7848                   |\n\nOcclusion-aware filtering (Table 10): Leads to an improved score by filtering unreliable gaze points during motion. These improvements likely result from filtering out unreliable gaze points, reducing noise and helping the model focus on relevant information.\n| Model Variant            | Semantic Score ↑ | F-Score ↑ |\n| ------------------------ | ---------------- | --------- |\n| Gaze-Reg (w/o occlusion) | 0.7298           | 0.4800    |\n| Gaze-Reg (w/ occlusion)  | 0.7505           | 0.5173    |\n\n\nEffect of Gaussian smoothing (σ) (Table 8): We found that the smoothing can benefit isolated gaze representations, it does not have a huge impact on temporally aggregated gaze points, as shown in the table below:\n| Model Variant             | Semantic Score ↑ | F-Score ↑ |\n| ------------------- | ---------------- | --------- |\n| Singular (σ = 20)   | 0.6211           | 0.4327    |\n| Singular (σ = 50)   | 0.6512           | 0.4463    |\n| Aggregated (σ = 20) | 0.7505           | 0.5173    |\n| Aggregated (σ = 50) | 0.7436           | 0.5034    |\n\n\nWe also provide an experiment when a larger temporal window was used for aggregation in the gaze regularized model (Table 9). From the experiment, we observed that aggregation over a larger time window (400ms) does not lead to further improvements compared to a temporal window of 200ms.\n\n| Aggregation Window | Semantic Score ↑ | F-score ↑ |\n| ---------------- | ------- | --------- |\n| 400ms               | 0.7398  | 0.4971    |\n| 200ms                | 0.7505  | 0.5173    |\n\n\nMoreover, gaze in text form vs. spatial heatmaps (Table 11): Textual gaze (providing raw gaze data in the form of a text prompt during training and inference) provides some benefit, but not as much as the structured spatial attention regularization. This is also included in the appendix which was submitted as part of the supplementary material:\n| Model Variant                   | Semantic Score ↑ | F-Score ↑ |\n| ------------------------- | ---------------- | --------- |\n| Base (no gaze)            | 0.6525           | 0.4318    |\n| Gaze in Text Form         | 0.7021           | 0.4630    |\n| Aggregated Gaze (Heatmap) | 0.7505           | 0.5405    |\n\nThanks for your suggestions, and we will more clearly reference these tables in the main paper to have a better articulation of the ablations that we have carried out. We hope these ablations can help address your questions.\n\n\n**(2) Evaluation on hallucination benchmarks:**\n\nThank you for this valuable suggestion. Existing hallucination benchmarks such as HallusionBench and POPE do not include gaze annotations or surrogate attention signals, making it difficult to directly apply our method.\n\nThat said, we recognize the importance of understanding whether our method reduces hallucination. To evaluate this, we conducted a human study comparing hallucination rates between the base and gaze-regularized models.\n\nWe selected 200 examples from our dataset and generated outputs from the baseline model and the gaze-regularized model. These responses were randomly shuffled and presented to a human evaluator who was blind to which model produced which output. For each example, the evaluator was also shown the ground truth annotation, enabling a reliable comparison between the predicted output and the actual scene content. The evaluator was instructed to flag any instances where the model mentioned objects or actions that were either absent from the visual input or clearly unrelated.\nTaking inspiration from the metric in HallusionBench, we provide the $\\mathcal{C}_I$ score which is the ratio of the annotations with clearly hallucinated instances divided by total number of annotations. \n| Model Variant    | Hallucinated Cases | Total Samples | $\\mathcal{C}_I$ Score |\n| ---------------------- | ------------------ | ------------- | --------------------- |\n| Base model             | 41                | 200           | 0.205                 |\n| Gaze-regularized model | 28                | 200           | 0.140                |\n\nThis shows that our proposed gaze regularized model produces an improvement by reducing the hallucination rate. We agree that extending our method to hallucination benchmarks is promising, and we will explore incorporating proxy gaze signals or human attention annotations for such benchmarks in future work.\n\n**(3) Related work on attention manipulation in VLMs:**\n\nWe appreciate the suggestion to better contextualize our work with respect to recent attention-manipulation studies in multimodal models. \nRecent works such as OPERA and Perception in Reflection operate on the cross-attention maps during text generation and apply retrospective corrections if and when the model focuses on irrelevant regions. For instance, OPERA penalizes captions that mention objects like “pencil” if the cross-attention never highlighted any pencil in the image. Perception in Reflection similarly evaluates whether the model has missed out on finer details by reflecting on generated answers using a multi-turn dialogue dataset. \n\nIn contrast, our work focuses on egocentric video understanding, where the goal is to model human behavior and intent in dynamic scenes. In this context, human gaze offers a strong and temporally grounded prior for what the person is doing or about to do as humans naturally fixate on task-relevant objects during interaction. Rather than relying on post-hoc attention correction, our method introduces gaze alignment to shape the visual representation passed into the language module, helping the model focus on semantically meaningful content from the start.\n\nOur approach is compatible with various transformer-based VLMs and does not require gaze at inference time. We will expand Section 2 or Appendix Sec. 6.1 to include these recent VLM attention regularization works.\n\n**(4) Which attention layers are regularized:**\n\nWe apply gaze regularization to the final attention layer of a dedicated transformer block, which we insert after visual feature extraction (e.g., ViT) and before the fusion with the language decoder. This design ensures that gaze supervision directly shapes the final visual representation passed to the language module, guiding the model to focus on semantically relevant content. We chose this location to keep the gaze module architecturally modular and compatible across different VLMs. We also experimented with other placements and attention depths (Appendix Sec. 6.5.7), but found this setup offered the best trade-off between performance and flexibility.\n\nWe thank the reviewer again for the helpful feedback and hope these clarifications strengthen your confidence in our submission. Please let us know if you have more questions."}, {"review_id": "y6xQuGXJC4", "reviewer": "Reviewer_mYKH", "summary": "This paper proposes a method for incorporating human gaze supervision into vision-language models via attention regularization, to improve egocentric video understanding. The authors focus on two tasks: current activity recognition and future activity prediction. Instead of relying on gaze at inference time, they propose to use gaze heatmaps only during training to guide model attention toward human-attended regions. They construct a dataset by combining the Ego4D gaze subset with GPT-4V-generated textual labels. The method shows consistent improvements over strong VLM baselines like OpenFlamingo and LaViLa, with additional gains in zero-shot generalization to the EGTEA dataset.", "questions": "Please refer to the above weaknesses. If addressed appropriately, I would be open to raising my score.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.  Incorporating human gaze into VLM training for egocentric understanding addresses an important challenge in aligning models with human attention and cognition.\n2. The attention regularization strategy is simple, interpretable, and effective. It requires no additional overhead during inference and can be applied to various VLMs.\n3. Multiple baselines (OpenFlamingo, LaViLa, InternVL, etc.) are used for comparison.\n4. Ablation studies and qualitative visualizations support the claims.\nWeaknesses:\n1. The paper relies entirely on GPT-4V-generated captions as ground truth labels, without human annotation or verification. While the authors mention prompt tuning through multiple iterations, no manual review of the generated labels was conducted. This raises concerns about the reliability of the supervision signal. Furthermore, it is unclear why more quantitative and objective tasks based on native Ego4D annotations (e.g., activity classification or retrieval) were not considered, which would provide stronger empirical validation.\n2. The discussion of related work is insufficient. Several recent works on attention- and gaze-augmented models that leverage gaze information to enhance vision-language models (VLMs) should be discussed. These approaches share conceptual similarities with the current paper, though this work is positioned in the era of large-scale VLMs. Notable examples include:\n- [NeurIPS 2024] Voila-A: Aligning Vision-Language Models with User’s Gaze Attention\n- [CVPR 2024] Learning from observer gaze: Zero-shot attention prediction oriented by human-object interaction recognition\n- [CVPR 2023] Teacher-generated spatial-attention labels boost robustness and accuracy of contrastive models\n A comparison with these works would help better situate the novelty and contribution of this paper.\n3. More transparency on prompt design. If list more actual prompts and failure examples, it would increase reproducibility.\n4. Could some gaze regions be incorrectly described by GPT-4V? Can you quantify this with a small human study?", "was_revised": true, "final_justification": "This paper proposes a method for incorporating human gaze supervision into vision-language models via attention regularization, to improve egocentric video understanding. The authors focus on two tasks: current activity recognition and future activity prediction. After the authors' detailed responses, most of my concerns have been addressed. I suggest that the authors include all meaningful experiments and discussions of related work from the rebuttal period in the revised version. I am willing to optimistically increase my score to 4.", "rebuttal": "We sincerely thank reviewer mYKH for the valuable suggestions and constructive feedback. We especially appreciate your recognition of the paper’s strengths, including our focus on aligning VLMs with human attention through gaze supervision and the simplicity and effectiveness of our training-only regularization strategy. Below, we address the concerns and comments you raised.\n\n**(1) GPT-4V-generated captions and label quality**:\n\nWe appreciate the reviewer’s insight regarding the supervision signal quality. \nWe would like to clarify that manual review and human validation were indeed performed during dataset construction. \nAs described in Appendix Sec. 6.2.1 and Fig. 5, we employed an iterative prompt refinement process involving human evaluators. Each iteration involved two human evaluators reviewing 50 image-caption pairs from 10 different clips — totaling 500 pairs per iteration. To assess caption quality during prompt-refinement, we adopted a flag-based strategy: if either annotator judged a caption to be substantially incorrect or semantically off-topic (e.g., object/action mismatch or spurious hallucination or overly generic descriptions or lack of temporal grounding), it was flagged. We applied a conservative quality threshold to accept only those prompts whose outputs demonstrated consistent accuracy across sampled image sequences. Since our design goal was not to create a new benchmark, but to ensure a reliable and semantically detailed supervisory signal for training gaze-regularized attention, our manual filtering and prompt refinement shall ensure that the final dataset reflects this objective while remaining scalable.\n\nWe will make this process clearer in the final version and include the quality control criteria explicitly to avoid confusion, on top of the information found in the appendix. We hope this resolves concerns around label validity and helps refocus attention on our primary contribution.\n\n**(2) Use of native Ego4D classification/retrieval tasks**:\n\nEgo4D contains both gaze-annotated and non-gaze-annotated videos. Our method requires gaze supervision during training, so we are limited to the subset of videos that contain gaze data. However, for these gaze-annotated videos, Ego4D does not provide task-specific labels for benchmarks such as activity classification, video-text retrieval and standard forecasting tasks (e.g verb/noun labels, bounding boxes). As a result, we are unable to apply our method directly to these evaluations.\n\nHowever, the subset of gaze-annotated videos are provided with narration annotations - coarse, short descriptions in the dataset provided as first person narrations of activities during recording. While these narrations are not suitable for the Ego4D forecasting benchmark that require structured outputs like bounding boxes or verb-noun labels, we define a coarse future narration prediction task. This allows us to test whether our model can predict future coarse narrations describing activities at a high level, given an input image sequence in egocentric settings.\n\nAs shown below, our gaze-regularized model still outperforms the base model on this task which only requires coarse labels. Note that the performance gain is modest compared to when finer-grained annotation prediction is required, supporting our core claim that gaze supervision is especially effective in capturing subtle details that tasks with coarse labels often dismiss.\n|Model Variant| Semantic Score ↑ | F-Score ↑ |\n| - | -| - |\n| Base Model | 0.7232  | 0.4982|\n| Gaze regularized model | 0.7621  | 0.5200|\n\nIn addition, to test whether our gaze-regularization approach generalizes to benchmarks that do not include human gaze annotations such as the Ego4D Short-Term Object Interaction Anticipation (STA) benchmark, we use a gaze prediction model to generate heatmaps. These predicted gaze maps are then used to regularize attention during training via our KL-divergence loss, similar to our main setting.\n\nRecent STA models adopt a transformer-based setup, which allows our approach to integrate naturally by applying gaze regularization to the attention distribution obtained. Performance is evaluated using the standard mean average precision (mAP) metric.\nThese preliminary results suggest that predicted gaze or a surrogate gaze for training time regularization may also improve prediction tasks like STA.\n| Model Variant| mAP (Noun) | mAP (Noun + Verb)|\n| - | - | - |\n| Base Model           | 22.13           | 11.29    |\n| Gaze regularized model | 25.01           | 13.45    |\n\nWe appreciate your question and suggested experiments that help further validate the effectiveness of our proposed gaze-regularization mechanism. It also shows that adapting our gaze regularization approach using surrogate attention signals is a promising direction, and we will highlight its potential in future work.\n\n**(3) Related work**:\n\nRecent works such as Voila-A, Learning from Observer Gaze, and Teacher-Generated Spatial Attention have explored incorporating attention into vision-language models. While they share the high-level goal of aligning model attention with human cues, they differ from our work in important ways.\nVoila-A : uses gaze at inference time to steer predictions on static image tasks like COCO captioning by using voila perciever resampler. It assumes gaze is available at test time and directly influences decoding. In contrast, our model uses gaze only during training and can be a modular component to several transformer based VLMs. Moreover, we focus on temporally grounded egocentric video tasks, such as activity recognition and future event prediction, where gaze reflects task-driven intent such as objects being manipulated or regions of upcoming interaction.\nLearning from Observer Gaze :  predicts attention maps from human-object interactions in static scenes, aimed at zero-shot attention transfer. However, it does not integrate gaze into the training of VLMs for downstream tasks. Our work instead uses gaze as a supervisory signal via KL-regularization to align VLMs attention with human focus.\nTeacher-Generated Spatial Attention: generates pseudo-attention maps via teacher models to guide contrastive learning. We differ as our objective is not to predict gaze but to use gaze as a regularization signal to enhance egocentric behaviour understanding in VLMs.\n\nWe thank the reviewer for pointing us to these related works, and we will revise the related work section accordingly to place our contributions more delicately within this growing area.\n\n**(4) Prompt design and reproducibility:**\n\nThank you for raising this important point. Our prompt design process is described in Appendix Sec. 6.2.1 and illustrated in Fig. 5 found with the supplementary material, and also in section (1) GPT-4V-generated captions and label quality.\nIn addition, we also provide brief examples of some prompts below to show the differences.\n\nFailure case 1: Describe what is happening in these images.\nOutput: A person is making coffee in a kitchen and then talking to another individual nearby.\nFeedback: Too generic and summary-like; lacks per-frame grounding\n\nFailure case 2: Describe what the person is doing in each frame one by one.\nOutput: Frame 1: No person can be seen in the frame. Frame 2: The person moves toward the counter. Frame 3: The person is talking to another individual.\nFeedback:Improvement in structure, but overly focused on person visibility rather than full scene understanding.\n\nFailure case 3: Describe what is happening in the image sequence and output the text descriptions for each image in the sequence, capturing the objects and actions taking place.\nOutput: Frame 1: The process of brewing a cup of coffee is taking place in the sequence.Frame 2: The person can be seen talking to the camera wearer in the kitchen.\nFeedback: Adds more detail, but doesn't consistently describe each frame or maintain sequence alignment\n\nFinal Prompt: For each image in the sequential series, provide a direct description. Ensure one description per image, capturing the objects, actions, and their spatial relationships. Provide the descriptions in the order presented (output 1 should correspond to image 1). Imagine that you need to provide these descriptions to a robot assistant.\nOutput: Frame 1: A white cup is placed on the drip tray under the coffee machine, which is brewing and dispensing a hot beverage Frame 2: The individual has moved closer to the coffee machine and is brewing a cup of coffee.  Frame 3: A person in a black shirt can be seen talking to the camera wearer in the kitchen near the coffee machine.\n\nWhile previous outputs were somewhat reasonable, they didn’t meet our requirements for fine-grained, frame-wise annotations. The final prompt, however, produced detailed annotations for each frame and maintained the correct sequence. We will include more failure cases and corresponding feedback along with the outputs in the final version of the paper.\n\n**(5) Could some gaze regions be incorrectly described by GPT-4V?**\n\nTo quantify the potential misalignment, we conducted a small human study. Two annotators were shown images with gaze heatmaps overlaid on the input images, along with the GPT-4V annotation. For each sample, they were asked to judge whether the annotation mentioned the object(s) or action(s) located in the gaze region.\nWe found that in approximately 74% of the samples, the annotations were judged to be well-aligned with the gaze regions. That is, the annotation referenced the semantically relevant content that the human was fixating on. The remaining cases included instances of partial or missing alignment.\n\nWe thank the reviewer once again for the thoughtful and constructive feedback. We hope our responses and additional results adequately address your concerns. Please let us know if you have more questions."}, {"review_id": "m6i1qRhRCT", "reviewer": "Reviewer_bTnR", "summary": "This paper investigates the task of egocentric understanding. To enhance VLMs for this task, it presents a novel gaze-regularized attention mechanism that aligns the model's focus with human visual gaze. In addition, an occlusion filtering mechanism is proposed to remove irrelevant gaze points in the temporal dimension. Experimental results on different VLMs validate that the proposed mechanisms improve accuracy in both semantic prediction and future event prediction tasks.", "questions": "1.  The manuscript only evaluates the proposed gaze-regularized mechanism on two tasks. How well would the method generalize to other egocentric video tasks, such as EgoVQA?  In my opinion, the current evaluation is insufficient to convincingly demonstrate the effectiveness and generalizability of the proposed gaze-regularized supervision mechanism.\n\n2. The evaluation relies solely on semantic similarity, which may be insufficient to evaluate the model in some challenging scenarios. Would more comprehensive metrics better capture model performance?\n\n3. The Transformer encoder in Figure 1 is not specified, which affects clarity and reproducibility.  Could the authors clarify the detailed parameters, such as the number of encoder layers?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The introduction of gaze information is novel in the field of egocentric video understanding. The experimental results also validate the effectiveness of the gaze-regularized supervision mechanism.\n\n- The proposed occlusion-aware filtering strategy is a useful module for aggregating gaze signals.\n\n- The authors use GPT-4V to generate fine-grained textual descriptions that capture egocentric activities and evaluate the performance of various VLMs.\n\n\nWeaknesses\n\n- The proposed module is only validated on two tasks, future prediction and activity understanding, limiting the overall evaluation of the gaze-regularized mechanism.\n\n- The compared models are not specified with their versions, such as InternVL in Tables 2, 3, and 4.\n\n- The evaluation metric (semantic similarity) used for the future prediction and activity understanding tasks is relatively simple and may not fully capture the complexity of these tasks or accurately reflect the model's performance in more challenging scenarios.", "was_revised": true, "final_justification": "Thanks for the authors' response. The majority of my concerns have been addressed.\n\nHowever, I still think the overall evaluation (only two tasks) of the gaze-regularized mechanism is not enough.\n\nTherefore, I am inclined to keep my initial rating.", "rebuttal": "We thank reviewer bTnR for the thoughtful review and constructive suggestions. We appreciate your recognition of the novelty of incorporating gaze into VLMs for egocentric understanding, the usefulness of the occlusion-aware temporal aggregation strategy, and the demonstrated improvements across models and tasks. Below, we address the concerns raised in the review:\n\n**(1) Task evaluation:**\n\nWe focus on fine-grained current activity understanding and future activity prediction since they are important for real-world use cases like assistive robotics, where it is critical to know what a person is doing and what they are likely to do in detail.\n\nCoarse labels like “picking up the bottle” or \"(Pick),(Bottle)\" don’t capture important details such as where the bottle is or which one is being picked up. For example, knowing the difference between “picking up the bottle” and “reaching for the red bottle on the top-left shelf” requires the model to understand spatial layout and context. These are precisely the types of spatial details human gaze tends to prioritize during task execution. \n\nBy using human gaze during training, we aim to guide the model toward these regions, which we observe can improve fine-grained activity understanding and future prediction.\n\nAs for the additional egocentric tasks (e.g., activity classification, video-text retrieval and forecasting) in Ego4D, we observe that the subset of videos in Ego4D on which these tasks are benchmarked do not have gaze annotations, and hence directly applying our method is difficult. \nHowever, to understand generalizability of our method to tasks whose benchmarking data does not have human gaze annotations, we use a gaze prediction model to estimate gaze heatmaps. On the Ego4D Short-Term Object Interaction Anticipation (STA) benchmark, we apply these predicted heatmaps during training to regularize the model’s attention using the same KL-divergence loss as before. This lets us test whether gaze-guided supervision can still help even when real gaze is not available. Preliminary results on the Ego4D v2 test split (measured using the standard metric - mean average precision mAP) show that the gaze-regularized model outperforms the baseline\n| Model Variant                   | mAP (Noun) | mAP (Noun + Verb)|\n| ------------------------- | ---------------- | --------- |\n| Base Model           | 22.13           | 11.29    |  |\n| Gaze regularized model | 25.01           | 13.45    |\n\nThese early results suggest that using predicted gaze to guide attention during training could be a promising direction, and we will highlight its potential for extending our approach to a wider range of benchmarks in future work.\n\n**(2) Model version clarity:**\n\nThank you for this helpful suggestion. We will update all relevant result tables to explicitly specify the versions of the models used. \nWe would also like to clarify that our goal is not to compare models against each other, but rather to demonstrate that gaze-regularized training consistently improves performance over each respective base model. Each model serves as an instantiation to validate the generality and plug-and-play nature of our proposed approach.\n\n**(3) Evaluation metrics:**\n\nWe use sentence transformers for semantic scoring to ensure that models are not penalized for surface-level paraphrasing, while still discouraging incoherent or off-topic predictions. This metric is particularly robust for capturing meaning similarity in egocentric captions, which often vary in phrasing.\nTo complement this, we also report standard NLP metrics such as ROUGE-L and F-score in Appendix Tables 8–15. These metrics show consistent trends with the semantic score and further support our core claims. We omitted them from the main paper due to space constraints.\n\nAdditionally, we conducted a small human hallucination study to assess whether our gaze-regularized model produces fewer unsupported or spurious descriptions. Since benchmarks like HallusionBench [1] do not contain gaze annotations, we designed a custom evaluation: paired outputs from the base and gaze-regularized models were shown to a human reviewer who flagged hallucinated objects or actions.\n\nWe report the $\\mathcal{C}_I$ score which is the number of hallucinated instances divided by the total number of annotations (lower is better).\n| Model Variant                  | $\\mathcal{C}_I$ Score  |\n| ---------------------- | ------------------------ |\n| Base model             | 0.205                    |\n| Gaze-regularized model | 0.140                    |\n\n[1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models, CVPR 2024 \n\n**(4) Missing information about the transformer encoder:**\n\nWe appreciate this request for architectural clarity. While our framework is modular and applies across multiple VLMs, we provide a concrete reference using OpenFlamingo in the appendix. In this setup, the visual encoder is a ViT-L/14 backbone with 24 transformer layers and this is frozen during training. The patch size is 14 x 14 in the ViT L/14 with a token sequence length of 256 visual tokens, and the hidden size (d) as 1024. In this setting, we inserted transformer layers after the ViT and regularized the attention weights obtained to enhance the visual features before they are passed into the language decoder.\n\n**(5) Formatting:**\n\nWe also acknowledge your note on inconsistent symbol formatting, and will correct these in the final version, and update the table of symbols in the appendix accordingly.\n\nWe thank the reviewer for highlighting key strengths and for suggesting improvements. We believe our task justifications, evaluation extensions, and clarifications around architecture and metrics directly address all concerns and further strengthen the case for our gaze-regularized framework. Please let us know if you have more questions or suggestions."}, {"review_id": "6BpvOHO6TO", "reviewer": "Reviewer_cjzU", "summary": "The paper presents a new regularization loss using gaze data to align attention maps in VLMs with gaze. The paper presents and demonstrate the hypothesis that steering attention maps to match gaze will lead to improvements in egocentric video understanding tasks. The paper evaluates this hypothesis on two tasks: future action prediction and activity understanding. The paper also presents a gaze temporal aggregation handling method to form a high quality training signal used for regularization.", "questions": "* What are p and “\\Phi” in eq 3 L133?\n* L83: “The gaze points in the dataset are originally represented in text form” - Is this just a text string of x,y locations? Or is it a textual description of the gaze (i.e “top left corner of the image“ for example?)\n* L315: “As shown in Table 5, our best-performing model (Gaze-Reg w/ occlusion) achieves a 9.8% gain in semantic performance over the base model, with only a minor increase in inference time ” - where is the inference time increase coming from given that the proposed method only impacts the training pipeline? If no gaze information is provided at inference time, shouldn’t the number of FLOPs be the same?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper’s hypothesis and experiments are sound and clear. The method presented is simple yet effective. Reported results highlight the effectiveness of the method and validate the tested hypothesis (Tab. 1).\n* The paper shows great zero-shot performances on the EGTEA Gaze+ dataset (Tab4) with a similar performance increase when training with gaze-regularization in the in-distribution evaluation setting. This strengthen the claim that gaze-regularization improves performances on egocentric video understanding task.\n* The paper provides nice qualitative examples of attention heatmaps on target objects. This highlights another aspect of the paper which is: the proposed method leads to a more interpretable solution which could be valuable for certain applications.\n\n\n\nWeaknesses:\n\n* The paper lacks additional theoretical discussion on whether it makes sense to align internal attention maps to gaze. VLMs models are huge networks with >10B parameters. Internal features are deep with an exponentially increasing receptive field and although there’s some level of spatial consistency in the generated features, it is not clear to me why the model needs to focus on features highlighted by gaze. I understand that results tend to justify this argument, but, it would be valuable to add more discussion in the paper about this. \n\n* The paper lacks justification on whether gaze can be estimated from egocentric inputs. From the paper we can read:\nL31 “We posit that eye gaze is a powerful supervisory signal in this setting: it reflects a person’s attention, often precedes interaction, and encodes both spatial and temporal cues” - I think this is a valid hypothesis, but in the context of egocentric video, are we confident enough that a model can infer the user’s gaze? It would strengthen the paper to cover this point.\n\n* The idea of learning a gaze distribution during training and sampling gaze (spatio-temporal) from that distribution during inference has been tested before. (“ Kyle Min Jason J. Corso Integrating Human Gaze into Attention for Egocentric Activity Recognition, WACV 2021“ ). I believe the paper should mention about this related work. \nThe paper should acknowledge this work and refine the claims focusing on transformer-based VLM and its attention layers.\n\n* The paper is missing a few ablation experiments that would strengthen the claim:\n*   (1) Missing ablation experiments showing the impact of the /sigma std of the Gaussian smoothing during gaze heatmap creation. \n*   (2) Missing ablation experiments showing the impact of the Occlusion Handling method on the final results.\n*   (3) Given that the method already relies on the Ego4D data, it would strengthen the paper to show numbers on the Forecasting benchmark of the Ego4D suite (https://ego4d-data.org/docs/benchmarks/forecasting/).\n\n* The paper lacks results showing whether gaze is actually being learned during training. This could be tested by extracting the attention maps from the attention layers at inference time and compare with GT gaze.\n\n* The paper lacks information on whether the gaze-regularization is applied on all attention layers? (I’m assuming multiple blocks using attention in a VLM model). This is a crucial design choice that should be mentioned in the paper.\n\n* The paper should clarify about the following: Gaze fixation is of 5fps (~200ms - L92: “Human visual attention involves fixations lasting approximately 200ms“) but the frame sampling rate of the created data is 1 FPS (L208 ”Temporal Sampling: To reduce computational load while preserving temporal structure, we downsample all videos to one frame per second.“). Therefore is the section on Temporal Aggregation with Occlusion Handling“ obsolete?", "was_revised": true, "final_justification": "I thank the authors for taking the time to address my questions. All my concerns have been addressed:\n\n(1) Ablation results showing the effect of the gaze distribution estimation in the attention layer\n\n(2) More in-depth related work and comparison with proposed method. Although, the presented hypothesis has been tested in different context and architecture, the results presented in the context of a VLM model is valuable and is worth being shared to the community.\n\nI trust the authors to incorporate the additions in the manuscript. I am increasing my rating.", "rebuttal": "We thank reviewer cjzU for the insightful and thorough review. We are grateful for your recognition of the clarity of our hypothesis, the simplicity and effectiveness of our method, and its ability to generalize across multiple VLMs. Below, we address the key concerns you raised.\n\n**(1) Theoretical justification for aligning internal attention with gaze:**\n\nThank you for raising this important point. We agree that it’s worth thinking carefully about why aligning a model’s internal attention with human gaze is useful.\n\nIn our work, we use human gaze as a signal to guide the model’s visual attention toward regions that people tend to focus on during a task. These regions typically include key objects, interaction points, and important parts of the scene that provide spatial and contextual cues.\nThis kind of guidance is especially useful for temporally grounded tasks like activity recognition and future prediction. These tasks require interpreting visual details over time to understand what someone is doing and what they are likely to do next. Human gaze often captures these cues naturally, such as looking at an object before reaching for it or glancing toward a location where an action is about to happen. We apply gaze guidance using a KL-divergence loss. This is controlled by a tunable weight (\\lambda), and so gaze is therefore used as a helpful prior, not a hard constraint. Importantly, this alignment shapes the visual features that are passed into the language decoder, helping it attend to the most relevant parts of the input when generating predictions. In doing so, gaze-guided training improves the quality of the model’s outputs and potentially improves interpretability by aligning internal representations with semantically relevant regions. We believe this offers an intuitive and practical way to inject useful human priors into vision-language models.\n\n**(2) Can gaze be inferred from egocentric inputs?**\n\nWe appreciate this thoughtful question. While predicting gaze directly from RGB images is challenging, our objective is not to directly predict gaze. Instead, we use human gaze during training to guide the model towards more human-aligned attention patterns.\n\nTo assess whether the model infers this alignment, we measure the top-k spatial overlap between model attention maps and ground-truth gaze heatmaps on the test set. Specifically, we compute top-10 overlap between the model’s 1×256 attention map (corresponding to 16×16 image patches) and the human gaze distribution. This metric quantifies how often the model’s most attended regions coincide with where humans actually looked.\n\nIn this evaluation, the base model showed an average overlap of 42%, while the gaze-regularized model achieved 68%, indicating significantly stronger spatial alignment with human attention. These results support our claim: gaze regularization improves attention alignment, functioning as an inductive bias rather than a hard constraint. We will add this result to the appendix upon revision.\n\n**(3) Additional ablations:**\n\nWe conducted ablation studies to better understand the contribution of individual design choices in our gaze representation pipeline. Results are presented in the Appendix and are summarized below.\n\nImpact of temporal aggregation vs. singular gaze (Table 14): Temporally aggregated gaze outperforms single-frame gaze on both activity understanding and future prediction. This supports our hypothesis that human attention over time provides richer, more stable guidance than momentary singular fixations, which could be noisy.\n| Model Variant   | Future Prediction ↑ | Activity Understanding ↑ |\n| --------------- | ------------------- | ------------------------ |\n| Singular Gaze   | 0.6512              | 0.6913                   |\n| Aggregated Gaze | 0.7505              | 0.7848                   |\n\nOcclusion-aware filtering (Table 10): Filtering out gaze points during motion blur or occlusion improves performance by removing noisy supervision, helping the model focus on more reliable gaze regions.\n| Model Variant            | Semantic Score ↑ | F-Score ↑ |\n| ------------------------ | ---------------- | --------- |\n| Gaze-Reg (w/o occlusion) | 0.7298           | 0.4800    |\n| Gaze-Reg (w/ occlusion)  | 0.7505           | 0.5173    |\n\n\nEffect of Gaussian smoothing (σ) (Table 8): Smoothing improves results for single-frame gaze, but has limited impact for aggregated gaze, suggesting that temporal aggregation already helps stabilize attention supervision.\n| Model Variant             | Semantic Score ↑ | F-Score ↑ |\n| ------------------- | ---------------- | --------- |\n| Singular (σ = 20)   | 0.6211           | 0.4327    |\n| Singular (σ = 50)   | 0.6512           | 0.4463    |\n| Aggregated (σ = 20) | 0.7505           | 0.5173    |\n| Aggregated (σ = 50) | 0.7436           | 0.5034    |\n\nThese results are useful to understand the value of using temporally aggregated and occlusion-filtered gaze heatmaps as training signals.\n\n**(4) Additional related work:** \n\nThank you for pointing out this relevant work. Prior work by Kyle Min and Jason J. Corso (Integrating Human Gaze into Attention for Egocentric Activity Recognition, WACV 2021) models gaze using a probabilistic framework with a VAE, and directly predicts gaze during inference to enhance egocentric activity recognition. This line of research helped shape our project. However, our approach differs in that we use gaze only during training as a supervisory signal and do not require it at inference time. Additionally, our method is designed as a modular component compatible with vision-language models. While we referenced this work in the appendix, we will expand our discussion of it in the final version.\n\n\n**(5) Temporal aggregation at 1 fps:**\n\nWhile the input video is downsampled to 1 frame per second for the model, we retain access to the original high-frame-rate video and gaze recordings (at ~30 fps). For each selected input frame, we aggregate gaze points from the preceding 200ms window (typically 6 frames). We also compute forward and backward optical flow during this window to perform occlusion-aware filtering (see Sec. 6.3), ensuring that only reliable gaze points are retained. This allows us to generate temporally coherent and spatially grounded gaze heatmaps, preserving the value of temporal aggregation even at reduced model input rates.\n\n**(6) Forecasting benchmark and task generality:**\n\nWe agree that future forecasting is important. \nTo clarify, the standard Ego4D forecasting benchmark requires structured annotations such as verb/noun labels and bounding boxes, which are not available for the gaze-annotated subset. The gaze-annotated dataset contains narrations which are coarse, first person descriptions about the activity narrated by the camera-wearer for the recording.  These narrations are not suitable for the Ego4D forecasting benchmark since it requires specific outputs in the dataset like bounding boxes and verb-noun labels.\nAs a compromise, we define a separate narration prediction task : a proxy that evaluates whether the model can predict narrations in an egocentric setting, given an input image sequence. Since the narrations are coarse, the performance gain is modest compared to tasks with fine-grained captions. We believe this result supports our claim that gaze supervision is especially helpful for capturing subtle object-action details that are present in fine-grained labels but often overlooked in coarse labels.\n\n| Model Variant                   | Semantic Score ↑ | F-Score ↑ |\n| ------------------------- | ---------------- | --------- |\n| Base (no gaze)            | 0.7232           | 0.4982    | \n| Aggregated Gaze (Heatmap) | 0.7621           | 0.5200    |\n\nAdditionally, to test generalization to datasets without human gaze, we use a gaze prediction model to generate estimated heatmaps and apply our method on the Ego4D Short-Term Object Interaction Anticipation (STA) benchmark. Since the models used for STA are transformer-based, our gaze regularization approach is naturally compatible. Preliminary results on the Ego4D v2 test split (mean average precision) show improvements over the baseline.\n| Model Variant | mAP (Noun) | mAP (Noun + Verb)|\n| -| - | -|\n| Base Model| 22.13 | 11.29|\n| Gaze regularized model | 25.01| 13.45|\n\nThanks for your question. We hope it is clear that these findings highlight the potential for extending our approach to forecasting and other tasks using surrogate gaze signals, even when real gaze data is unavailable.\n\n**(7) Clarifications on attention layer placement:**\n\nWe regularize the final layer of a dedicated attention block inserted between the visual encoder and the language module (e.g., prior to the Perceiver Resampler in OpenFlamingo). This choice ensures that human gaze influences the final aggregated features passed to the language decoder, and it keeps the component modular for different VLMs. We include a design explanation and ablations on the number of blocks in the appendix (Sec. 6.5.7).\nRegarding symbols: in Eq. 3 (L133), p denotes the probability of generating the target text sequence conditioned on the sequence of image inputs, as modeled by the vision-language model \\phi.\n\n**(8) Other Clarifications :**\n\n“Gaze in text form” refers to the original ego4D annotations where raw (x, y) gaze coordinates are provided along with the timestamps.\nInference time increase is due to the added attention module that computes global queries across the visual sequence while using individual frames to get key and values for the attention module. The attention weight then obtained from the final layer is then used during regularization.\n\nWe thank the reviewer again for these thoughtful questions and hope our clarifications reinforce the value and clarity of our contributions. Please feel free to let us know if you have more suggestions or questions."}]

Author rebuttal:

Meta-review: The paper presents a gaze-regularized framework for VLMs on egocentric vision tasks. While one reviewer bTnR maintains a "borderline reject" (3) due to concerns about evaluation being limited to two tasks, the authors provided an explanation that the lack of gaze-annotated datasets inherently restricts broader task validation. The other reviewers found the rebuttal convincing, addressing concerns about metrics and model details. Therefore, my initial recommendation is acceptance.
Decision: Accept (poster)