Voila-A: Aligning Vision-Language Models with User's Gaze Attention
Authors: Kun Yan
Zeyu Wang
Lei Ji
Yuntao Wang
Nan Duan
Shuai Ma
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Vision Language Model
Human Gaze
Multimodal
Controlled Generative Model
AR/VR
Primary area: human-AI_interaction
TL;DR: Aligning Vision-Language Models with User's Gaze Attention

Abstract: In recent years, the integration of vision and language understanding has led to significant advancements in artificial intelligence, particularly through Vision-Language Models (VLMs). However, existing VLMs face challenges in handling real-world applications with complex scenes and multiple objects, as well as aligning their focus with the diverse attention patterns of human users. In this paper, we introduce gaze information, feasibly collected by ubiquitous wearable devices such as MR glasses, as a proxy for human attention to guide VLMs. We propose a novel approach, Voila-A, for gaze alignment to enhance the effectiveness of these models in real-world applications. First, we collect hundreds of minutes of gaze data to demonstrate that we can mimic human gaze modalities using localized narratives. We then design an automatic data annotation pipeline utilizing GPT-4 to generate the VOILA-COCO dataset. Additionally, we introduce a new model VOILA-A that integrate gaze information into VLMs while maintain pretrained knowledge from webscale dataset. We evaluate Voila-A using a hold-out validation set and a newly collected VOILA-GAZE testset, which features real-life scenarios captured with a gaze-tracking device. Our experimental results demonstrate that Voila-A significantly outperforms several baseline models. By aligning model attention with human gaze patterns, Voila-A paves the way for more intuitive, user-centric VLMs and fosters engaging human-AI interaction across a wide range of applications.

Voila-A: Aligning Vision-Language Models with User's Gaze Attention
Kun Yan',, Zeyu Wang?; Lei Ji Yuntao Wang?,
Nan Duan
Shuai Ma f
SKLSDE Lab, Beihang University
2 Key Laboratory of Pervasive Computing, Tsinghua University Microsoft Research
kunyanEbuaa, edu. cn, wang 2y2ngmay2mmals tsinghua. edu. cn mashuai @buaa, edu.
Abstract
In recent years, the integration of vision and language understanding has led significant advancements in artificial intelligence, particularly through Vision- Language Models (VLMs). However, existing VLMs face challenges handling real-world applications with complex scenes and multiple objects, well aligning their focus with the diverse attention patterns of human users. In this paper, we introduce gaze information, feasibly collected by AR VR devices, as proxy for human attention to guide VLMs and propose a novel approach, Voila-A, for gaze alignment enhance the interpretability and effectiveness of these models in real-world applications. First, we collect hundreds of minutes of gaze data demonstrate that can mimic human gaze modalities using localized narratives. We then design an automatic data annotation pipeline utilizing GPT- to generate the VOILA-COCO dataset. Additionally, we innovate the Voila Perceiver modules to integrate gaze information into VLMs while preserving their pre-trained knowledge. We evaluate Voila-A using hold-out validation and a newly collected VOILA-GAZE test set, which features real-life scenarios captured with gaze-tracking device. Our experimental results demonstrate that Voila-A significantly outperforms several baseline models. By aligning model attention with human gaze patterns, Voila-A paves the way for more intuitive, user-centrie VLMs and fosters engaging human-AI interaction across wide range of applications. Our code is available at httpss//github..com//guuoo/oo
Introduction
The integration vision and language understanding has witnessed significant advancements in recent years, particularly through the development of Vision-Language Models (VLMs). These models have demonstrated remarkable performance in various tasks, such as visual question answering, image captioning, and visual storytelling, among others. Although VI.Ms exhibit strong performance various tasks, their applicability in everyday scenarios is hindered by their limited alignment with human users' focus. This misalignment leads to suboptimal performance and decreased user satisfaction. Current VLMs' inability to process these intentional modalities results imprecise and unhelpful responses. As demonstrated in Figure T user's intent can be communicated through
equal contribution corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

spoken language, multimodal expressions, or even be concealed. The users' attention can clarify vague expressions, meanwhile, uncovering hidden intentions more challenging.
Most recent VLMs (31 28 [] primarily focus on learning alignment between vision input and text tokens for LLMs designing learnable interaction layers to attend the vision input to the frozen LLM layers. The importance of aligning Al with human attention has been high- lighted in previous research, which demonstrates that incorporating visual attention can lead to improved user experience [26] 45 39]. Additionally, there has been growing interest in grounded VLMs, which investigate the fine- greig grounding capability between region-text pairs instead of image-text pairs and further conduct dense regional prediction tasks 60 18 61
The representation of visual regions within computational models can be achieved through various methodolo-
ulone reclaser te cree bosk
Octer
gies, such as bounding boxes [63] discrete points [34], or continuous
youraa pes cxacge with bank
I paot wit my herdo?f
traces 140 [53]
For the integration
e pom a 1- pes the songa using your sanci Hovever in eniight be eas or Lam
VILLA
of such regional data into models, researchers have developed multiple approaches. These include the con- catenation of cropped image patches with the original textual or visual in- puts (59 4. the application of masks or Gaussian maps to highlight user- specified areas [29 30), and the in-
The radi son the for the inarg oree our Hewviier
the ateettove feoe terezek and other
food The
Tedeetleeeti taar and Five our selories it wo The
Koomas 2
sager
VELA
Lenaklu faul oolonroee will cul tox f durtorcch uomudo hove ail di lu calorie an not reterrander anded reguir cona momonn, as they conersure palomm, capensing
leille sasi leae aelland
corporation of positional encodings to represent points, boxes, or traces 23 [TT Despite the extensive explo- ration of bounding boxes and segmen- tation techniques within VLMs, their application remains suboptimal for end-users to generate input signals.
Figure 1:
AR and VR scenarios usually involve complex scenes with multiple objects. Users may interested in only one specific object and gaze is the most natural way to inter- act with the device.
We propose to introduce novel approach that leverages gaze tracking as a more intuitive and interactive method for defining visual regions, particularly within augmented reality (AR) and virtual reality (VR) environments. Zhang [58] provides an overview of gaze-related research, outlining process that begins with collecting human gaze data (further discussed in [B3), followed by building models to predict human attention distribution (i.e., saliency models, as discussed in [.4), and culminating in samaan-azz--ssisttee AI. They conclude that Al agents capable of perceiving and understanding human gaze behaviors can better infer user needs and assist in daily tasks. However, they also note that research in this final direction is still limited. Our work aims to advance this area further. Prior research in gaze-based visual representation includes gazeedireetted visual grounding [1 and the implementation of eye-gaze within vision transformers (33]. Nonetheless, these methodologies have encountered obstacles in terms of scalability and adaptability. Despite significant progress, the seamless integration of gaze information into large-scale VLMS is still a formidable challenge.
To tackle this issue, we demonstrate that mouse trace data can be a proxy for gaze behavior modeling and leverage trace data from Localized Narratives [40] to annotate instructional data using GPT- 4 (36]. Another critical aspect of the challenge is maintaining the integrity of the pre-trained knowledge within VLMs while effectively assimilating gaze data. We further design Voila-A's attention mechanism to incorporate gaze information while not forgetting pre-trained knowledge. We evaluate Voila-A through a hold-out validation set and newly collected test set VOILA-GAZE, featuring real-life scenarios with a gaze-tracking device
In this paper, we make the following contributions:
We propose Voila-A, a novel approach for aligning VLMs with user's gaze attention, and design innovative mechanisms to integrate gaze information into VLMs while preserving pre-trained knowledge.
2

---

We leverage trace data from Localized Narratives to annotate instructional data using GPT-4, generating the VOILA-COCO dataset with 72k QA pairs, and demonstrate the scalability of this method.
We evaluate Voila-A through hold-out validation set and a newly collected VOILA- AAALE test set of real gaze samples, demonstrating that our approach significantly outperforms several baselines.
By aligning model attention with human gaze patterns and leveraging state-of-the-art techniques, we make a step forward in the development of more intuitive and user-centric VLMs, paving the way for more effective and engaging human-AI interaction wide range of applications
2 Leveraging Trace Data as an Alternative Approach to Align VLMs with Gaze Attention
In this section, we discuss the potential of trace data as a proxy for gaze data and propose method for transforming trace data to make it more gaze-like, ultimately enabling the effective use of trace data for aligning VLMs with user gaze attention.
END vs Sample Rate
Earth Mover's Distance
0.17
Obtaining gaze data for training VLMs can be challenging, as it is difficult to annotate and expensive acquire. We propose that an alternative approach can be employed to align VLMs with user gaze attention: utiliz- ing trace data, such as mouse traces. Localized Narratives (LN), a prior work, has anno- tated 849,000 images with mouse traces that are aligned with each word of the descriptions. The project involved 156 professional annotators who worked full-time, with annotator managers ensuring high-quality annotations through man- rates. ual inspections and an automatic quality control mechanism. After discarding 23.5% of annotations, the remaining ones demonstrated a semantic accuracy of 98.0% for nouns and verbs. The accuracy of mouse traces in relation to object locations was also analyzed, revealing that most trace points were within the correct bounding box.
0.15
0.15
0.14
0.13
20
25
30
35
40
Sample Rete
Figure EMD between the mean heatmaps of Ik gaze and trace samples with varying sampling
To demonstrate the similarities between gaze and mouse traces, we first collect hundreds of minutes of gaze data samples as described in C and then we further collect mouse trace annotations on those samples. The mouse trace collection was done using the Bubbleview (21] interface, where the annotators were instructed to verbally repeat the questions and register their eye fixations with mouse clicks. The resulting CC (cross-correlation) score between real gaze fixations and mouse traces was 0.82, and the NSS (normalized scanpath saliency) score was 2.57. These scores are comparable to those reported in the Bubbleview study [21] and are consistent with the performance of SOTA saliency models (71, indicating that gaze and mouse traces exhibit similarities, as users tend to fix their gaze on the target object when asking questions, a behavior also observed with mouse traces. However, there are minor differences between the two, specifically in terms of gaze fixation continuity and the presence of noise points outside the target object at the end of a query. In the case of mouse traces, points that fell outside the bounding box were attributed to two factors: annotators often circled around objects, causing the traces to be near but not inside the box, and some annotators started moving the mouse before describing the object or vice versa. These observations provide valuable insights for properly leveraging trace data into the alignment process and understanding the relationship between gaze attention and language description.
In order to utilize trace data as a substitute for gaze data, we introduce a method to transform mouse traces, thereby reducing the discrepancies between the two data types and making the trace data more gaze-like. We first address the inherent noise in both trace points and gaze points by converting them into 2D heatmaps using Gaussian blur:
3

---

2
(z, y)
(I)
2
where (z, y) represents the heatmap value at position (ar y). and is the standard deviation of the Gaussian kernel.
Since mouse traces more continuous than gaze fixations, downsample the trace data to better resemble the distribution of gaze data. We investigate the Earth Mover's Distance (EMD) between the mean heatmaps of 1k gaze and trace samples while varying the sampling rate from to 40:
E. ((()) (( Σ 21=1 (P)
EMD(P (ID((P.O
P.Q) (P.
(2)
where P and Q are the distributions of the gaze and trace heatmaps, F; denotes the cumulative distribution function, and n is the number of bins.
We observe that the EMD has local minimum value around sampling rate of 25 as shown in Figure 2 By selecting this optimal sampling rate, we can approximate the trace heatmap as an alternative to the real gaze heatmap from statistical perspective. Consequently, this transformation mitigates the differences in reaeroelationshhss, compactness, and noisiness between the trace and gaze data.
Method
3.1
Automatic Data Annotation For LN-COCO
The automatic data annota- tion process for Voila-A driven by the motivation to develop more intuitive and user-centric VLM by align- ing model attention with human gaze patterns. As shown in Figure 3 this pro- cess aims to create effec- tive and engaging human- AI interaction experience across various applications. To achieve this, we have de- signed an innovative anno- tating approach that lever- ages the capabilities GPT- 4 as a visual assistant to an- notate trace-aligned instruc- tional data to simulate the user's s gaze attention. The data annotation process fol- lows design principles to en-
FEELINE INTIALSAT
GPT-4
HALIAU
FORNAA
DIVERAT
Marull Fronet
timas
100 Seed Sample
System Prompt
Context Example
Grounded QA
LN-COCO
System Prompt
Context Example
25k Imaaze
QA Fain
VOILA-
COCO
Localoed QA Parrs
ALTERING
RUTERING
Grounded QA Parrs
Figure 3: Automatic Data Annotation Pipeline
sure accurate, relevant, and consistent annotations. These include: focusing on referable sentences and appropriate tags, 2) using conversational format with specific and general questions, 3) address- ing various visual content aspects with definite answers, incorporating complex questions while avoiding uncertainty, and 5) offering detailed, well-organized explanations.
As illustrated in Figure the automatic data annotation pipeline comprises three stages.
Stage Prompt Design Iteration. The first stage focuses on refining the prompt design. Let S = (T, Nr, T,. 1 be a set of 100 samples from the LN-COCO dataset, where I; represents image, N; the localized narrative, T; the corresponding trace, and C, the set five captions from COCO-caption. We initiate the process with basic system prompt, instructing GPT-4 to generate direct questions QP and indirect questions Q, and corresponding answers As,; that specifically reference the localized narratives while considering COCO-caption background information. The

---

referring portions are annotated with a unique marker M for trace matching during post-processing. We also provide two in-context examples to guide the model in generating helpful. well-formatted, diverse, and visually grounded QA pairs. Throughout each iteration k, we manually evaluate the quality of the generated grounded QA pairs and adjust the prompt to enhance their helpfulness, formatting, diversity, and visual relevance. After K = 10 iterations, we find the quality of most pairs to be satisfactory, and subsequently, we freeze the prompt to initiate the pipeline.
Stage 2: Data Sampling. In the second stage, we sample N = 25, 000 image pairs from the LN-COCO dataset and obtain approximately M 75, 000 QA pairs.
Stage 3:
Post-processing. The third stage involves post-processing the raw grounded QA pairs. This includes further filtering based on set of keywords IC ("prompt", "this picture", "reference caption", ..... We define a filtering function Fs (Qigi Ar,;, that identifies and removes QA pairs containing meta deseriptions of the prompt. We note that this issue may be further resolved by using GPT-4V, which was not available during our submission date. Additionally, we identify cases where answers are unhelpful, such as "I don't know" or "It's hard to tell." We find that these types of answers have low reward scores, so we further examine all pairs using a reward model (1) and filter the dataset by setting a minimum reward threshold т. We define a filtering function (. Ai,;; that removes QA pairs with reward scores below T, Finally, we segment each localized narrative into temporally aligned segments with respect to the special marker M. Each segment comprises a grounded fact, a corresponding trace, a direct and indirect question, and an answer. This forms the final VOILA-COCO dataset, denoted as D {F., Ti, QP, Qf. As,)]. It is worth noting that we did not utilize all localized narratives, leaving room for future exploration. We annotate the COCO subset of localized narratives, resulting in the Voila-COCO dataset, with statistics presented in Table
The finalized prompt can be found in E We also visualize a sample of our annotated data in Figure Data quality analysis can be found in Section D By adhering to these design principles, the automatic data annotation process ensures that the resulting dataset is of high quality and effectively aligns the VLM's attention with that of a human user.
14
3.2 VOILA-GAZE: Real-life gaze-QA pairs
To further demonstrate the effectiveness of our method in aligning VLMs with real-life users' gaze attention. we conduct experiments in two everyday scenarios, encompassing a variety of question types details can be found in Table 5
Table I: Statistics of Voila-COCO and Voila-Gaze Datasets, SR refers to Survival Rate from raw data after filtering
In addition to the recorded gaze trajectory, video, and transcription, each participant is instructed to annotate the key elements of their questions, formulate clear questions based on their interpre- tations, and choose the best answer from three candidate answers generated by GPT-4 according to their annotations. The experiment includes
DATASET
SPLIT
ALMAGES
#QUESTIONS
SR
VOILA-COCO VOILA-COCE VOILA-COCO VOILA-GAZE
TRAINING
20000
93.5%
VALIDATION
100
550
71.1%
TEST
500
1900
75.7%
REAL-LIFE
200
200
18.2%
16 participants (8 per scenario) with an equal gender distribution, aged between 20 and 30 (with a standard deviation of 2.06). Each participant takes approximately 240 minutes to complete the study, After applying post-filtering and manual checking, we curate set of 200 QA pairs as our real-life benchmark, VOILA-GAZE. The curation process is conducted by two individuals sequentially, with the second person double-checking the following aspects: 1. The question is related and aligned with gaze. 2. The answer is meaningful and can be considered a proper response to the gaze- directed question. 3. The question not related to specific brands, prices, or any other objects beyond general knowledge. 4. The question type is not biased towards few simple patterns. This two-step process ensures the quality and relevance of the curated data while minimizing potential biases and maintaining focus on general knowledge. Samples of VOILA-GAZE are shown in Figure
3.3 Model Design
In developing our design, a critical consideration the adherence to the established architecture of current VLMs. It is essential to avoid introducing significant number of new parameters
5

---

Volla Percstiver Block NN
Pretralted and Frezen
FPW
finetuned
Attention
2ed Stage Finstuned
It StF
V
Q []
UM Layer
Vos'a Perrabur cemompeer
Coted Cress Arte intention Layer
Tamp
X
I.
Latare Quarre ee
lirear Layers
Vidio Freader
LM Layor
Lincar Layers
Vision Encoder
Gated Ameentinn
Frocueette
Drwael Usn (Fawscelweee are butto to stoe bear? OPE.
Goot Pastmag
Vilan ioput
Figure 4: Architecture of the VOILA Model: On the left, gaze fixation is transformed into heatmap, which is subsequently processed through linear layers to encode the visual attention. This encoded data is then segmented into discrete patches that are spatially correlated with corresponding image patches. These gaze patches are further refined into key embeddings, which undergo modulation by a gating mechanism, designed to incrementally integrate gaze data. The resulting gaze and image key embeddings are then combined and subjected to a self-attention mechanism, which synthesizes the information into cohesive set of latent perceiver embeddings. On the right, the figure delineates the integration pathway where the gaze heatmap and the image concurrently enter the VOILA Perceiver. This integrated input is subsequently directed through gated cross-attention modules before progressing into the language model layers, culminating in a unified output that encapsulates the interplay between visual attention and linguistic processing.
or making extensive structural modifications. This constraint is due to the limitations of the current gaze dataset, which does not support large-scale pretraining. Additionally, we must be vigilant in preventing catastrophic forgetting during the fine-tuning process. Therefore, we aim to implement only those modifications that are proven to be necessary and beneficial through our ablation studies. These changes are carefully selected to enhance performance without compromising the stability and efficiency of the model.
We employ the model architecture from OpenFlamingo, as illustrated on the right side of Figure This framework consists of pre-trained vision encoder, language decoder, and gated cross-attention layers, offering flexibility for multi-image and multi-turn conversational interactions. The primary challenge lies in incorporating gaze instructional signals into a pre-trained VLM. To tackle this issue, we initially developed several potential solutions, which are discussed See 4.3.2 and Fig T6
Based on empirical evidence, we ultimately confirm the effectiveness of the Voila Perceiver Resam- pler solution. The Voila Perceiver Resampler(VPP) processes the input image features and gaze information, and then feeds them into series of Voila Perceiver Blocks (VPB):
VPR(X, G) =LN((VBB, (X, G)), VPB, G) = VPB(X, To. G), VPB, (X, =VP,(X, (X,
for i = 1.2,...,
The input hidden states of VPR are denoted as X E R Bxxxx XD L E BBxHXXxxXXXX x and G e B Lx where B is the batch size, L,andd, = I. are the lengths of latent tokens and image/heatmap patches, respectively. H is the number of attention heads, and D is the hidden size. In which X represents the image features, latent features, which are introduced from the original Perceiver as small set of latent units that forms an attention bottleneck through which the inputs must pass. To obtain the gaze information G E R B we first divide the gaze heatmap G' E IR Bxhx into patches. Then, we apply a linear transformation followed by layer normalization. The process can be represented by the following equation:
the gaze heatmap embedding patches. I. denotes the
= LN (LLiinar(patcch(C"))
The VPR comprises a series of Voila Perceiver Blocks (depicted on the left side of Figure 4 This mechanism leverages gaze information to enhance visual feature perception. Our design adheres

---

to the principle that the gaze serves an information aggregator in the attention process without disrupting the original learned distribution. The Voila Perceiver Block(VPB) is defined as follows:
= LN(L + FF(L + GA(X, G)))
(5)
The feed-forward network, FF((), is sequence of layer normalization, linear transformation, GELU activation, and another linear transformation. The attention mechanism, GA(X, L, G), is computed as follows:
= L V = (X @ L), K= = (X @ L) + W& (G lanh(g) + P)
QK T )
(6)
GA(X, L, G) Softmax
VD
Here. @ denotes concatenation along the feature dimension, and P is zero padding with the same shape as L. And g is a learnable gating parameter attributed to each block for fading in gaze guidance gradually. W. are the QKV matrices of corresponding input.
3.4 Training
Our approach utilizes the OpenFlamingo training paradigm to train the Voila model, building upon the pre-trained weights of the Otter model, which incorporates an MPT-7B [7 language encoder and a CLIP ViT-L/14 [42] vision encoder. To avoid overfitting and maximize the benefits of pre-trained knowledge, initially freeze both encoders. As shown in Figure 4 we then train only the linear layers directly related to gaze input at the first stage for one epoch before fine-tuning the entire Perceiver resampler module, the cross-attention layers integrated into the language encoder, and the input/output embeddings of the language encoder in the second stage for an additional epoch. This process results in roughly 1.3 billion trainable parameters for the Otter model. Note that the g each VPB are initialized as O.
During training, adhere specific format for preparing our training data. This format combines an image, user instruction, GGPT"-genneran answers 1, and a unique token known as the [endofchunk] token. We arrange the training data as follows:
<context> [image]
GPTr(answers) <answer>,[endoochumr
Here, the [image]. [answer], [fixation), and [endofchunk] tokens are distinct and serve particular functions. We adopt chatbot-like format enhance the costucctionfollowiinn capabilities and conversational generalizability of our model. The [image] and [endofchunk] tokens originate from the OpenFlamingo training paradigm, while the [answer] token is new addition by Otter. The [answer] token separates answers from instructions, allowing us to mask all tokens following the [answer] token during training and designate them as the model's prediction objectives. We also introduce the [fixation] token to direct the language model to utilize gaze information. We train our model using cross-entropy loss function.
Experiment
4.1 Evaluation metrics
GPT-4 RANKING We utilize GPT-4 RANKING our primary automated evaluation metric to assess model performance through a one-to-one comparison. The GPT Ranking represents the language model's 's evaluation of the quality of the generated response. This score signifies the extent to which the response aligns with the ground truth image description and answer while demonstrating the model's language proficiency. Factors such as grammar, semantics, and fluency are taken into account when comparing the response to that of another model. It is important to note that GPT-4 exhibits sequence ordering bias. To mitigate this issue, we implement dual-setting approach that reverses the order of the models, ensuring that the order does not influence the outcome. The prompt and evaluation procedure can be found in Figure 15
Reward Score Given that our dataset is automatically annotated using GPT-4, it is crucial to mitigate any potential system bias during model evaluation. this end, we incorporate human preference by utilizing reward model score as auxiliary metric. The reward model, which assesses
7

---

the human-like quality of response, is trained using human feedback to predict the superiority of generated answer relation a given question from human perspective [T This approach allows us to achieve more balanced and robust evaluation process, ensuring that our model's performance aligns with human expectations and preferences.
4.2
Main Results
4.2.1 VOILA Exhibits a Balanced Capability Between Helpfulness and Fact Grounding
In Figure we observe a notable su- periority of VOILA over both Otter and Kosmos-2 on the VOILA-COCO- TESTSET. Regarding the grounding capability, both VOILA and Kosmos- 2 trained with fine-grained grounded facts outperform Otter model to large extent. Besides, VOILA sur- passes Kosmos-2 marginally. With respect helpful capability, Otter de- livers significantly more helpful re- sponses than Kosmos-2. Since Otter is trained top of Openflamingo with instructiio-follooingg dataset, it can provide more helpful response, especially for informative queries while Kosmos-2 tends to answer visual observation from the input image. In addition, VOILA trained gaze dataset demonstrates stronger helpful capabilities over all models.
Overnd
Helpltd
Grounding
Creeat
Helgld
Ch. deounding
Helpodi
Croumaling
Figure 5: GPT-RANKING ON VOILA-COCO-TTetsee
In real gaze scenarios, illustrated Figure VOILA outperforms the two baseline models as well. These scenarios differ substantially from collected COCO images and present more challenging questions, necessi- tating higher degree of accurate in- tent understanding and reasoning. Es- pecially from the comparison of Otter vs Kosmos-2, we found that there are many more Tie results due the hard- ness of the real cases. Despite these increased demands, VOILA continues to surpass both models, further showcasing its balanced proficiency in both helpfulness and fact grounding.
Hespftl
Ormunting
Oherall
Helpoel
Obs. Oomanting
Overall
Halprial
Romoa
Figure 6: GPT-RANKING ON VOILA-GAZE
4.3
Ablation studies
4.3.1
Query types has a significant impact on Response Quality
Table investigates the varying per- formance of different question types,
Table 2: Ablation on query types, WR means Winning Rate Over Otter-base
specifically direct and implicit/coref- erence queries. As the base model Openflamingo was pre-trained on di- rect queries, both Otter and VOILA performed better in this category, as expected. In addition, it is natural for humans to communicate with coref- erence queries. VOILA maintained
METHOnS
QUESTION TYPE5
WR
LRR
REWARD ScorE
OTTER-BASE
CORIFERENCE QUERY
-1.91
OTTER-DASE
DIRECT QUIRRY
0.51
0.1
0.02
VOILA
CORIFERENCE QUERY
0.41
0.15
-0.79
VOILA
DIRECT QUERY
0.62
0.15
0.14
VOILA
IN CONTEXT PRGKIPT COXPPERENCE QUERY
0.46
0.15
0.02
VOILA
IN CONTEAT PROMP1 DURECT QUERY
0.77
0.12
0.20
strong performance when handling coreference queries with the gaze as guidance while the Otter model greatly decreased. Furthermore, appended in-context QA pairs prior to the current query and observed that the examples further improved the quality of the responses. real-life situations, multi-turn conversations are involved in most interactions with many coreference queries. The design

---

of an in-context prompt can assist VOILA to demonstrate a superior ability, This improvement is evident across both direct and coreference query types.
4.3.2 Heatmap is better way to incorporate gaze
To establish the effectiveness of our approach, we implemented several alternative methods for incorporat- ing gaze data into VLMs. These methods include: converting gaze se- quences into discrete position tokens for LLMs, using the bounding box position of trace trajectories as addi- tional patch tokens concatenated to VIT image feature token lists, and con- verting the bounding box coordinates into discrete tokens. We illustrate these methods in Figure T6 However, all these methods failed to outperform the gaze heatmap approach, as shown in Table
Table Ablation on Methods of Integrating Gaze Data, WR means Wining Rate over Otter-base
METHODS
WR
LR
REWARD ScoRE
OUTUR--HASI
-1.91
AS DISCRETE
0.25
GAZE fownding bot AS IMAGE PATCH
0.36
0.20
-1.26
GAZE hounding box AS DISCRETE: POSITION TOKINS
0.21
0.22
-1.72
VOILAIGAZE AS HEATMAP)
0.41 0.18
-0.79
4.3.3 Gradual Unfreezing of Parameters Yields Better Results
Table 4 presents empirical findings that demonstrate the effectiveness of gradually unfreezing model param- eters, Instead of directly unfreez- ing the vision perceiver and cross- attention layers, or using LORA to fine-tune the entire model, we first fine-tuned the gaze-related weights and then fine-tuned the other parts of the perceiver and attesnattntiinn lay- ers, which yielded better results. We hypothesize that this improvement is due to the newly added gaze component needing to adapt to the distribution of the pre-trained layers first. This adaptation process can further help mitigate the issue of catastrophic forgetting.
Table Ablation on Training Procedure, WR means Wining Rate over Otter-base
LAYERS TUN-DU
WR
LR
REWARD SCORE
OTTER BASE FROZEN ALL
-1.91
0.24
-1.78
VOILA WEIGHT
-1.52
VOILA GAZE: WHIGHT-LORRA
0.21
VOILA GAZE WEIGHT MERCIEVER+CROSS ATTENTION
0.41
0.18
-0.79
4.4 Qualitative studies
We conducted qualitative studies on randomly selected cases and demonstrated the results of several representative examples in Appendix Figure [8] According to the analysis, the conclusions can be summarized as follows: 1) Existing models are able to generate reasonable results for explicit queries. In the 1st row, the object cakes and the attributes color are explicitly mentioned in the query, and the three models can answer (partially) correctly; 2) Regarding coreference queries, the model Otter hard to understand the pronouns like i/ without spatial guidance as shown in the 2nd row. This requires further context or generates the answer based on the salient object like plane instead of the actual human attention; 3) The Kosmos-2 model can take the bounding box for grounding as spatial guidance, it is sometimes not accurate compared to the heatmap used in VOILA. As shown in the 3rd row, the bounding box is too coarse and made the model focus on the object plane instead of the actual human attention sky; 4) Besides, we found that Kosmos-2 tends to describe the detailed visual content and sometimes lacks the Instruction-followe responses Keyboard depicted in the bounding box ignoring the actual query intention; Finally, 5) There are still further challenges for all models to deal with. For instance, counting for objects requires intensive fine-grained recognition of the visual content demonstrated in the last row.
capability: In the 4th row, the Kosmos-2
5 Conclusion
In this study, we presented Voila-A, a cutting-edye approach that aligns Dision-Language Models with user gaze attention. Voila-A can be implemented in HMD AR/VR devices as an egoview copilot, benefiting wide range of users, including visually impaired individuals who rely on their gaze to communicate their intent. This method surpasses the capabilities of similar mobile apps that necessitate users to lift their phones for scene capture. We successfully utilized trace data to create the

---

VOILA-COCO dataset, showcasing Voila-A's superior performance in two benchmarks. Our research lays the foundation for more engaging human-Al interactions and encourages further exploration of Voila-A's integration with various modalities and tasks in the realm of multimodal Al systems.
6 Acknowledgements
This work was conducted during Kun Yan's 's internship at Microsoft Research Asia and supported in part by NSFC No. 61925203, U22B2021, NSFC No. 62472244 and No. 62132010.
References
[1] Openssiantoowwddmmmmdddddee-e--e
https://nugginnfaac. co/
Opeukssisstattrraxx---ooee----- debeeea-lddderr-a-eeee v3--arge- 2023.
[2] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in 2022.
Neural Information Processing Systems,
[3] A. Awadalla, I. Gao, J. Gardner, J. Hessel, Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre,
S. Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01338 2023.
[4] L.. Bracha, E. Shaar, A. Shamsian, E. Fetaya, and G. Chechik. Disclip: Open-vocabulary referring expression generation. arXiv preprint arXiv:2305. 2990000055/ 2023.
[5] K. Chen, Z. Zhang, W. Zeng, Zhang, F. Zhu, and Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint 2306 2023.
[6] M.-C. Chen, R. Anderson, and M.-H. Sohn. What can mouse cursor tell us more?: correlation of eye/mouse movements on web browsing. CHI '01 Extended Abstracts on Human Factors in Computing Systems, 2001.
[7] S. Chen, N. Valliappan, S. Shen, X. Yc, K. Kohlhoff, and J. He. Learning from unique perspectives: User-aware saliency modeling. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2701-2710, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society.
[8] Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-languuae tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[9] W. Dai, Li, Li, A. M. H. Tiong, Zhao, W. Wang, B. Li, P. Fung, and Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[10] A. Das, H. Agrawal, L. Zitnick, D. Parikh, and D. Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? ArXiv, hes/1606.035566 2016.
[11] T. Gong, C. Lyu, S. Zhang, Y. Wang, M. Zheng, Q. Zhao, K. Liu, W. Zhang, P. Luo, and K. Chen. Multimodal-gne gptt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.0 047900 2023.
[12] Q Guo and E. Agichtein. Towards predicting web searcher gaze position from mouse move- ments. CHI '10 Extended Abstracts on Human Factors in Computing Systems, 2010.
[13] E. J. Hu, Y. Shen, P. Wallis, Z. Low-rank adaptation large language models. arXiv preprint arXiv:2106 09685, 2021.
Allen-Zhi Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora:
[14] J. Huang, R. W. White, and G. Buscher. User see, user point: gaze and cursor alignment in web search. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2012.
10

---

[15] J. Huang, R. W. White, and S. T. Dumais. No clicks, no problem: using eursor movements to understand and improve search. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011.
[16] M. Ilaslan, C. Song. J. Chen, D. Gao, W. Lei, Q. Xu, J. Lim, and M. Shou. Gazevqa: A video question answering dataset for multiview eye-gaze task-oriented collaborations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pages 10462-10479, 2023.
[17] M. Jiang, S. Huang, J. Duan, and Q. Zhao. Salicon: Saliency in context. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[18] W. Jin, S. Mukherjee, Y. Cheng, Y. Shen, W. Chen, A. H. Awadallah, Jose, and X. Ren. Grill: Grounded vision-language pre-training via aligning text and image regions. arXiv preprint arXiv:2305. /4676, 2023.
[19] Judd, K. A. Ehinger, F. Durand, and A. Torralba. Learning to predict where humans look. 2009 IEEE 12th International Conference on Computer Vision, pages 2106-21 111, 2009.
[20] W. Kienzle, Wichmann, Scholkopf, and M. O. Franz. nonparametric approach to bottom-up visual saliency. In Neural Information Processing Systems, 2006.
[21] N. W. Kim, Z. Bylinskii, M. A. Borkin, K. Z. Gajos, A. Oliva, F. Durand, and H. Pfister. Bubbleview: an interface for crowdsoureing image importance maps and tracking visual attention. ACM Transactions on Computer-Human Interaction (TOCHI). 24(5):36, 2017.
[22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.
[23] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao. S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint 1.Xiv:23304.264333 2023.
[24] C. Koch and S. Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. Human neurobiology, 4 4:219-27, 1985.
[25] M. Kümmerer, T. S. A. Wallis, and M. Bethge. Deepgaze i:: Reading fixations from deep features trained on object recognition. ArXiv, abs/1610.01563, 2016.
[26] M. F. Land. Eye movements and the control of actions in everyday life. Progress in retinal and eye research, 25(3):296-322 2006.
[27] B. Li, Y. Zhang, L. Chen, J. Wang. J. Yang. and Z. Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023.
[28] Li, D. Li, Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
[29] Z. Lin, Z. Zhang, L.-Z. Chen, M.-M. Cheng, and S.-P. Lu. Interactive image segmentation with first click attention. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1339-1 2020.
[30] Z. Lin, Z. Zhang, L.-H. Han, and S.PP. Lu. Multi-mode interactive image segmentation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 905-914, 2022.
[31] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
[32] J. Liu, L. Wang, and M.-H. Yang. Referring expression generation and comprehension via attributes. In Proceedings of the IEEE International Conference Computer Vision, pages 1856-4864, 2017.
[33] C. Ma, L. Zhao, Y. Chen, S. Wang, L. Guo, T. Zhang, D. Shen, X. Jiang, and T. Liu. Eye- gaze-guided vision transformer for rectifying shorteut learning. IEEE Transactions on Medical Imaging, 2023.
11

---

[34] A. Mani, N. Yoo, W. Hinthorn, and O. Russakovsky. Point and ask: Incorporating pointing into visual question answering. arXiv preprint arXiv:2011.13681, 2020.
[35] E. Niebur and C. Koch. Control of selective visual attention: Modeling the where pathway, In Neural Information Processing Systems, 1995.
[36] OpenAl. Gpt-4 technical report, 2023.
[37] J. Pan, E. Sayrol, X. G. i Nieto, K. McGuinness, and N. E. O'Connor. Shallow and deep convolutional networks for saliency prediction. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 598 606, 2016.
[38] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[39] R. Piening, R. Piening, K. Pfeuffer, A. Esteves, T. Mittermeier, S. Prange, P. Schröder, and
F. Alt. Looking for info: Evaluation of gaze based information retrieval in augmented reality. IFIP TC13 International Conference on Human Human-Computte Interaction, 2021.
[40] Pont-Tuset, Uijlings, S. Changpinyo, Soricut, and V. Ferrari. Connecting vision and language with localized narratives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 647-664. Springer, 2020.
[41] K. Qian, Z. Zhang, W. Song, and J. Liao. Gvgnet: Gaze-directed visual grounding for learning spder-specified object referring intention. IEEE Robotics and Automation Letters, 2023.
[42] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[43] E. Sood, F. Kögel, F. Strohm, P. Dhar, and A. Bulling. Vqa-mhug: A gaze dataset to study multimodal neural attention in vqa. In Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 27-43. Association for Computational Linguistics, 2021.
[44] Y. Sugano and A. Bulling. Seeing with humans: Gaze-assisted neural image captioning. ArXiv, abs/1 (68.6000 05003, 2016.
[45] V. Tanriverdi and R. J. Jacob. Interacting with eye movements in virtual environments. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 265- 272, 2000.
[46] H. R. Tavakoli, F. Ahmed, A. Borji, and J. T. Laaksonen. Saliency revisited: Analysis of mouse movements versus fixations. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
[47] M. N. Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023. Accessed: 2023-05-05.
[48] M. Tonsen, K. Baumann, and K. Dierkes. A high-level description and performance evaluation of pupil invisible. arXiv preprint arXiv:2009.000 00508, 2020.
[49] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Ilama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[50] A. B. Vasudevan, D. Dai, and L. V. Gool. Object referring in videos with language and human gaze. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4129-4138. 2018
[51] P. Voigtlaender, S. Changpinyo, J. Pont-Tuset, R. Soricut, and V. Ferrari. Connecting vision and language with video localized narratives. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2461-2471, 2023.
12

---

[52] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-ser te-euuence learning framework, In International Conference on Machine Learning, pages 23318-233444 PMIR, 2022.
[53] K. Yan, Ji. H. Luo, M. Zhou, N. Duan, and S. Ma. Control image captioning spatially and temporally. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-1 2014-2025, 2021.
[54] Y. Yao, Chen, A. Zhang, W. Ji, Z. Liu, T.-S. Chua, and M. Sun. Pevl: Position-enhanced pre-training and prompt tuning for vision-language models. arXiv preprint arXiv:2205. 2022.
[55] A. L. Yarbus. Eye movements and vision. In Springer US, 1967.
[56] Q. Ye, Xu, Xu, J. Ye, M. Yan, Zhou, Wang, Hu, P. Shi, Y. Shi, al. mplug- owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[57] R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao. Llama- adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16/99, 2023.
[58] R. Zhang, A. Saran, B. Liu, Y. Zhu, S. Guo, S. Niekum, D. H. Ballard, and M. M. Hayhoe. Human gaze assisted artificial intelligence: A review, IJCAI proceedings of the conference. 2020:4951-4958, 2020.
[59] S. Zhang, P. Sun, S. Chen, M. Xiao, W. Shao, W. Zhang, K. Chen, and P. Luo. Gpt4roi: Instruction uning large language model on region-of-intered arXiv e-prints, pages arXiv-2307, 2023.
[60] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li. et al. Regionclip: Region- based language-image pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16793-16803, 2022.
[61] Q. Zhou, C. Yu, S. Zhang, S. Wu, Z. Wang, and Wang. Regionblip: A unified multi-modal pre- training framework for holistic and regional comprehension. arXiv preprint urXiv:2308.02299) 2023.
[62] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing lision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 230.::2200..0092, 2023.
[63] Y. Zhu, O. Groth, M. Bernstein, and L.. Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004, 2016.
13

---

A Limitation and Discussion
A.1
Future work
Despite its promising potential, there are limitations and future directions to consider. These include enhancing inference efficiency for real-time responses, integrating voice modalities for seamless interaction, and supporting higher resolutions for OCR and screen or UI understanding.
A.2 Hallucination
During inference, our method occasionally exhibits hallucinations related to image content. This issue may stem from the limited number of training samples and the imperfect integration of visual hidden distributions into the language decoding process during vision language pretraining. Recognizing the potential for further scaling of our method, we consider addressing these hallucinations as a future research direction.
A.3 Comparison to GPT-4VV
With the release of GPT-4-V, which features vision capabílities, during the submission of this paper, we find it pertinent to include a brief discussion on the relationship between our work and GPT-4-V. Although GPT-4-V demonstrates remarkable visual capabilities in demos, surpasses our method in terms of visual understanding abilities. Nonetheless, our work remains valuable as it presents an effective approach to incorporating user sensory information for generating more relevant responses.
As reported, GPT-4-V occasionally struggles to accurately capture a user's intent when referencing specific elements within an image, prompting the design of an interface that allows users directly draw sketches for highlighting purposes. Our method has the potential to enhance this user experience and can be extended more dynamic scenarios such as virtual reality and augmented reality.
B Related Work
B.1 Multimodal Large Language Model
Recent research works for multimodal vision and language tasks relied multimodal large language models(MLLLM) and demonstrated superior performance. One line is to learn alignment between the vision input and the text token for LLM. LLaVA [1] directly feeds visual features to the LLM using
only a learnable fully connected layer. BLIP-2 (28 proposed Q-Former to extract a fixed number of query features from visual features, which are aligned with the text in frozen LLM. Another direction is to design learnable interaction layers to attend the vision input to the frozen LLM layers. Flamingo 2 adopts a perceiver resampler module to convert visual features into visual tokens and interleave the tokens in plain text at the locations of vision inputs. Besides, Flamingo performed learnable cross attention attend the visual tokens of the image that appeared just before in the interleaved sequence, rather than to all previous images and built a large-scale interleaved image-text dataset. On top of these designs, recent works mainly focus on improving mustruction-follovin ability. LLaMA- ALaMteAdapters [57] aims to adapt LLaMA 49 into an followiiinnn model with an additional adapters module and multi-modal prompts. Mini-GPT4 622 mPLUG-OWL [56]. and InstructBLII [9] adopt the Q-Former on various language models for instruction following capability. Besides, MultiModal-GPP [II fine-tuned OpenFlamingo (3) using Low-rank Adapter (LoRA) [3] and Otter (77 introduced Multl-Modal In-Context Instruction Tuning (MIMIC-IT) dataset following three heuristics, both of which demonstrate improved instruction-folloiim ability, Simultancously.
Grounded MILM Inspired by the success of MLLM, recent works focus on investigating the fine-grain grounding capability between region-text pairs instead of image-text pairs, and further conduct dense regional prediction tasks. One research line is to learn regional alignment between the image regions with the corresponding text tokens. RegionCLIP [60] extends CLIP with pseudo image regions and textual concept pairs. Grill [18] proposes to replace the referring words with the corresponding visual patches to align text and image Regions. RegionBLIP [61] takes position- assisted regional objects soft prompts for L.M on image-region-text data. Another research focus to unleash the grounding ability a multimodal large language model. VL-T5 [8] converts the visual grounding task into regional ROI box feature conditioned text generation to predict the box
14

---

id. OFA [52], PEVL 54 and KOSMOS-2 [8 reformulate continuous corner coordinates of the object to discrete position tokens. Shikra [3] handles spatial coordinate inputs and outputs in natural language without introducing extra vocabulary or position encoders. The works S 38 59 also perform Instruction tuning and convert the position of regional objects into language descriptions. Although the gaze is flexible and interactive, it is easy for humans to understand the gaze's semantic representation but hard for Al agents.
B.2 Region Representation for Large Language Models
The visual region can be represented as bounding boxes 63 points
34 traces
40
Existing approaches usually leverage Fast-RCNN to detect bounding boxes which limits the pre- defined or recognized objects in the bounding box and hard to scale out. Points are flexible but are too line-grained and require a large number of points to represent large regions precisely. Trace is more natural way to input by using the mouse trace coordinates and is most similar to human's gaze. In AR and VR scenarios, although trace is applicable with gesture, we propose to use gaze more conveniently and interactively. The two works [5] 34 take bounding boxes or points as region input for visual question answering and are the most similar work. Different from them, we take gaze as regional inputs.
Region Inputs In order input the regional information the model, several methods 50 directly concatenate cropped image patches with the original text/image as model input. Another method [2 30] uses 0/1 mask Gaussian map input with the original image emphasize the area of user interest. Additionally, other methods [23] 51 first encode points, boxes, or trace to positional encodings then add them intermediate features or learned queries. Specifically for gaze, 41 propose a gaze-directed visual grounding and fuse the gaze feature through multi-modal fusion module. EG- ViT (33] propose a eye-gaze gaze-guided vision transformer which takes the masked image patches within the gaze interest.
B.3 Gaze and Cursor as a proxy for Attention
Cursor-based techniques, including approaches like SALICON [17] and Bubble [21], have emerged as affordable, nonintrusive, and scalable alternatives traditional eye-tracking methods for collecting human attentional data. Empirical evidence from prior work has established strong connections between eursor like signals and gaze positions. Studies focusing on web browsing and search tasks have found high correlation between cursor and gaze locations, with better alignment along the vertical dimension [15] 14 12 6). These findings support the motivation use cursor- based techniques as proxy for attention.
Despite their success, existing cursor-based studies have limitations, such the need for complex post-processing of mouse movement data and evaluations limited to simple aggregate comparisons with eye-tracking data [21]. Furthermore, while prior work serves as a solid foundation from data-centric perspective, it lacks a demonstration of whether modern applications aiming to assist users using gaze, such as vision language models (VLMs), can be trained from cursor data and later adapt to gaze signals, especially when transitioning from 2D planar images on screens to ego view scenes in head- mounted display (HMD) scenarios. Our work aims to directly tackle this problem, as we believe it is the optimal time to close the entire visionary loop of understanding and utilizing the gaze modality to ultimately achieve smart, in situ personal assistants.
B.4 Saliency models on modeling gaze attention
551 proposed that tasks could be decoded from fixation patterns, receiving mixed support in subse- quent research. Early computational models of visual attention focused on bottom-up approaches, representing pre-attentive selection processes from visual input 24). Later, the saliency map concept
emerged [35]. Initially, models were trained on fixation data from eye-tracking experiments 20, 19]
but collecting large datasets proved difficult. The SALICON dataset [77 addressed this challenge by using mouse movements to simulate natural viewing behavior, leading to state-of-the-art performance in saliency models
11 37 46). As deep learning advanced, saliency modeling improved (25).
enabling more complex gaze pattern modeling in vision- languae tasks 44 10 50)
15

---

Can mxnrriss brain?
tool iro wire??
Yes. the presence a Rublk's cube in the suguesis that puzzle garee exercae brein.
No you can not use a screwdnwvrr to cut the iron ware. You can use cut plar instearl
VESIA
V
Can peel it with my hands??
Recommend some dishes to cook
it ponsible ta peel orange using your hands. However. might de eesier skin from the orange.
Some of the dishes you can cook with Chinese cabbage, cabbage eagplant indicde styrgtying. steaming They are versanla vagetables the can he eusily combined with other ingredients creaue delicious and healthy meals.
V
peeter
sautiing
V
Figure 7: Different Scenarios in the Future
Saliency models aim to approximate the human visual system by predicting eye fixations images (21). Unlike traditional saliency models, our approach takes ground truth gaze data, image, and natural language inputs to generate contextually relevant responses, presenting novel challenge. Recent work, such as 44. leverages gaze signals to enhance captioning tasks but does not accommodate dynamic user queries beyond captioning. Additionally, their LSTM-based method falls short compared to contemporary large transformer baselines. [43] introduces gaze data to the visual question-answering (VQA) task, but their analysis remains limited comparing human and neural attentive strategies learned by VQA models. Besides, a recent attempt [I6] leverages gaze for video question-answering answering in industrial tasks such as assembly and disassembly and proposed a discriminate model for multi-choice answer prediction. With the development of large vision-language models, we believe our work provides valuable contribution to modern applications by effectively tackling the new challenge.
C Gaze Data Collection
In order to examine the gaze patterns of users, we conducted a preliminary study where participants engaged in one of three daily situations: grocery shopping, visiting museum, or engaging in domestic activities. Participants were instructed to perform queries specific to each scenario as if the system could provide the desired information. Our research involved 21 participants (8, 5, and for each respective scenario), consisting of 13 males and 8 females, ranging in age from 19 to 30 years old (with standard deviation of 18). The study took each participant between and 150 minutes to finish, and they were compensated at a rate of 15 USD per hour for their involvement. As a result, we get 548 minutes of gaze recording. The Pupil Labs Invisible 4% a gaze-tracking smart glasses system that has been widely used for research purposes. It is equipped with gaze sensors, an egocentric camera, a microphone, and an inertial measurement unit (which was not used in this work). Participants were asked to wear the Pupil Labs Invisible glasses without any headwear that could obstruct the sensors on the glasses. Since the Invisible glasses require a connection to a mobile phone for operation, instructed participants keep the phone in their pockets minimize potential distractions. Data was continuously recorded as participants engaged in their chosen scenario.
D Voila COCO diversity and quality
We conduct a comprehensive analysis of the VOILA-COCO dataset to evaluate the quality and diversity of the automatically generated QA pairs. Our examination included several aspects:
16

---

Qualitative Case Study
Original
Gaze Hentmap
Groud Truth Query: What colors are the cakes?
Otter
Kosmos-2
Voila
There are two cakes in the image. The cake is white, and the second cake has red color.
The cakes
The cakes the lable are red, brown, and orange
Answer: The takes table in red, brown, and orange col-
orated with red and black colors
Yes, is possible poel the orange using How- ever, it might be eas- poeler remove the skin from the orange.
Question: Can I poel my An- swer: Yes, can peel orange your hands...
I'm sorry. but I couldn't understand the Can you please rephrase or peo- vide context?.
Yes. you can peel orange with your hands
your
What's the color? Answer: The
Question:
color plane is white.
The color of the planes in the image is blue
background blue color.
color of the sky in the background blue
Question: How ducc eye fatigue? An- eye fatigue while using a regular breaks, follow the 20- 20-20 rule adjust brightnss and contrast,
To minimize eye fa-
swer:
most common
rigue working laptop, consider following steps:
laptop,
cause of eye fatigue overexposure to bright light
Keyboard
1. Adjust the screen's brightuess to com- fortable level...
Question: How many figures are there this painting? An- swer: According what see, the paint- ing seems to contain 23 figures.
There are two people standing the paint- ing.
There figures in this image.
There figures in the painting.
We show successful predictions all models. Middle: We demonstrate the problems of bascline models compared with VOILA including coreference queries, gaze grounding methods, etc. Bottom: We display hard challenges for all models.
Figure 8: Qualitative Case Study: Top:
Supermarket shopping Task
Domestic living Task
Comparison
Appliance Malfunction
Completing Recipe
Activity & Health
Recommend
Snack & Fruits
Knowledge
Dressing Advice Entertainment
Decision Making
Strengthen Decision
Small Talk
Table 5: Guiding for User in VOILA- Collection, note this guide aims to facilitate and inspire users to generate questions related to data collection, rather than imposing strict limitations on the scope of their inquiries.
17

---

Question: Please recommend fruits for losing
Queition: Which better for making dumplings? Potato better making fillngs,
Quediio: why is working?
possible
conditioner
variety malfunctioning thermesta,, or exzet cause, best to consult your user monual contast professional HYAE techiicion for proper diegnosh and repait.
incorporating
apples, oranges,
cumpling adcitional Ingrediant to add color and flavor but using potatoes as the main ingredient will reull a more satafying and traditonaa dumpling experience.
refetgeant issues. To decermine
into your diet. Apples calories, which can help you feel full longer Oranges provide a good source vitamin C and fizes, while lemons can sid digestion and boost metaboliom. Remember consume these fruits in moderation and part of belenard diet for optimal resulls..
Figure 9; Data sample from VOILA-GAZE
Utilizing Word Cloud visualizations, we have assessed the range of vocabulary employed within the dataset.
Vocabulary Richness:
Sentence Length: We have analyzed the length of sentences for direct questions, indirect questions, and answers
realistic distribution that reflects typical conversational patterns.
Topic Distribution: By applying Latent Dirichlet Allocation (LDA) models, we have investigated the topical variety present within the dataset across the different categories of questions and answers.
D.1
Direct Questions
See Figure 10
Wond Count
view
1002
L27
200
(a) Wordeloud
(b) Length Distribution
Figure 10: Visualizations of VOILA-COCO: direct questions
D.2 Indirect Questions
See Figure []
D.3
Answers
See Figure I2
18

---

An sestoaal Dellinden Duedtees
1200
Color
Aot Couo
(a) Wordeloud
(b) Length Distribution
Figure 11: Visualizations of VOILA-COCO: indirect questions
View
140
Scene
500
Aic Goed
(a) Wordeloud
(b) Length Distribution
(c) LDD Topic Model Distribution
Figure 12: Visualizations of VOILA-COCO: answers
E
Details for Automatic Data Annotation
Figure shows the system prompt and in-context examples for Automatic Data Annotation
F
Implementation Details
In this section, we describe the implementation details of our model. The model architecture is derived from Otter, which combines text model and vision model. The text model is an instance of MPTForCausalI.A . 7B and the vision model is based on the CL.IP ViT-Large model with a patch size of 14.
Text Model Configuration: The text model has 32 layers, each with attention heads. The model has hidden size and dmodet of 4096. The attention mechanism uses multi-head attention with torch implementation and alibi attention. The model uses learned position embeddings and low-precisin layer normalization. The model is trained with maximum sequence length of 2048 and vocabulary size of 50432. The tokenizer used is EleutherAA/gDt-neoxx-222 The model's torch data type is set to bfloat16.
19

---

Prompt for Automatic Referring Annotation
an Al assistant, your task to andlyze single as you are directly observing You will provided background and referable seniences describing the view, but your responses should be hused on your observations rather than the descripions. First, anmolate the referable tag number, emphasize the section you want discuss. Note: DO NOT anmotate background sentences. Then, enguge in conversation with person inquiring about the view, focusing on the lagged elements while considering the background information. Use the questions. conversation should resemble the tone visual assistant examining the view and answering questions. Include a variety of questions and their corresponding answers. Your questions should cover various aspects the visual content, such as object types, object counts, actions, locations, and relative positions berween objects. Fasure your questions have definite answers: (I) the view clearly displays the content questicn, allowing for confident response; or (2) can be confidently determined from the view that content not present. Avoid questions that cannot be answerod with certainty. Try your best to incorporate complex questions relevant to the view's content, including background information about the objects discussions about eveats occarring within the view. Refrain from asking about uncertain details. When using < Q4t,, follow the original question with un indireci question, as the questioner is looking at the urea of interesi; use promouns to refer to the relevant part, and omit context in the original question if it be inferred from the user's 's eye guze lixation. When answering complex questions, provide detailed responses, including examples or reasoning steps, to create a convincing and well-organized explanation. Use multiple paragraphs necessury. Make the answer comprehensive and informative. Throuphout the conversation, maintain the impression that you directly observing the view rather than reading descriptions. Do not mention any words like "image", "picture", or "text" in the question and answer, as if you are equipped in an HMD and sharing the same view with the
System Prompt:
sentincce using <QA> =</Q#> lug, where
<Qt>> tag for questions related specific tugs in the sentence, and <Q> tag
context Example:
Role:
Role: assistant
Background: There is movie theater that displays the show times above the doors.A hotel for dogs the
Refer=<Q1 > red fire hydrant=</Q1>> >> is <<Q2>> deep in the now-</Q2>.0Q5 The fire hydrant in the snow near receatly plowed sidewalk. </Q3> <Q4-<0044/000 This city has had very hard winter with snow, K/Q4
Question:
<Q1> What the red thing?
winter.
Referable: red fire hydrant is deep in snow. The fire
Indirect Question:
What is it?
hydrant is in the snow neur recently plowed sidewalk. This city has had a very hard winter with snow.
Answer:
It is a red lire hydrant.
Question:
<Q2><s the fire hydrant fully covered by snow or partially visible?
=
Indirect Question:
Is it fully covered by snow partially visible?
Answer:
The fire hydrant is partially visible, with significant portion of it covered by the deep snow.
Question:
<Q3> the condition of the sidewalk near the hydrant?
Indirect Question:
How's the condition of the sidewalk there?
Answer:
The sidewalk near the fire hydrant hus been recently plowed, so it relatively clear of snow compared to the surrounding area.
Question:
< Q44 What does this scene suggests about the winter season the city?
Indirect Question:
the winter here?
Answer:
It suggests that the winter scason in the city has been quite harsh and challenging. The significant accumulation seen in the view indicates that the city has experienced heavy snowfall. which can lead to various difliculties and discuptions for its residents. For instance, the deep snow covering the fire hydrant could potentially hinder its accessibility case of emergencies, which poses safety concerns. The recently plowed sidewalk implies that the city's 's maintenance crews have been actively working to keep walkways
and safe pedestrians, but the sheer amount snow fall might continuous snow removal efforts.
difficult to keep up with the
Furthermore, such extreme winter conditions can impact transportation, leading to traffic issues, delays in public transit, increased risks of accidents due slippery roads. It can also cause problems for businesses and schools, people might face challenges commating to work or allending clusses. In conclusion, red hydrant deep the snow the recently plowed sidewalk suggest the city faced particularly severe winter season, with substantial snowfall that has likely caused various challenges and distuptions for its residents and infrastructure."
Figure 13: Prompt for Automatic Referring Annotation
20

---

Caption: Bottom side of
Caption: Behind
Caption: Rottom right side od the image person standing is skateboard.
Caption: Behind him wall.
Image man is doing skateboarding and jumping.
fencing. Behind the fencing there is water.
Question: What behind the person standing the bottom right? Indirect Question: What's behind that person?
What skatebourder In the bettom left doing?
What behind
the bottom sici view? Indirect Quection: What's there on the buttom right side?
Question: What
jumping skatebourder?
indirect Question: What is he doing? The skateboarder bettom left performing trick, jumping into the air with his skateboard.
Indirect Question: What's behind
Answer: Bahind the person standiing
right, there
Answer: Behind the jumping skatehsarder, there is fencing, and water.
bottom
the view, there a person standingg
and stateboard present near
Caption: the ground.
beside
Caption: On the right, there is
Caption: On the right side the woman walking
man who lyynn his bag.
yellow colour
Question:: is there any barrier
Question:
doing
Caption: the
wearing black colour bag along with her.
the
on the bench?
Image can see the grass in green
Question: Can you describe the woman walking right side?
cattle's grazing area?
Indirect Question: What doing?
color.
Indirect Question: there
Question:
grass?
Indirect Question: What does she look like
separation between them? Answer: Yes, there is fance beside the ground, providing barrier between
Answer: The man lying on his possibly taking resting.
Indirect Question: What color is it?
right side wearing yellow shirt and carrying black bag with her. She seems be going about possibly shopping or commuting.
walking
Answer: grass green color.
cattle's grazing area.
Figure 14: Annotated Example of VOILA-COCO
Vision Model Configuration: The vision model has 24 hidden layers and a hidden size of 1024. uses 16 attention heads and an intermediate size of 4096. The activation function is quick_gelu. The model uses an image size of 224 and 3 input channels. The patch size is 14, and the projection dimension is 512. The layer normalization epsilon is set to 10-
The architecture uses cross attention every layers and only attends to previous layers. The model's s torch data type is set to float32. Media placement augmentation is enabled during training.
Voila Configuration:
For initialization, we use the Kaiming normal method with fan-in mode, ReLU nonlinearity, and standard deviation of 0.02.
For optimization, employ the AdamW optimizer [22 with starting learning rate of 1e-5 and batch size of 4. We train Voila for three epochs, scheduling the learning rate using a cosine annealing scheduler. To prevent exploding gradients, apply gradient clipping with threshold of 1.0.
G GPT-RANK
Figure 15 presents our prompt and evaluation procedure for GPT-RANKING
21

---

GPT Ranking
Overall System Prompt:
a question along ground deseription amswer imuge, evaluaae two provided candidate answers. Deternine which unswer is factually accurate, logical, and helpful to the user. il you think answer is betler, respond with -1, answer is better respond with if you result tie, oulgut Only respond
'-1" 'O'
your choice.
Helpfulness System Prompt: Given question along ground description answer an image, evaluaae provided candidate unswers. Deternine which unswer is solving the user's 's problem and is more helpful the user. if you think answer I is betler, respund with -1, if answer 2 respond 1, if you
tie, ouiput Only respand
your
Fact Grounding System Prompt:
question along ground description answer image, evaluate two provided candidate unswers. Determine which answer is factually grounded to the Fact provided. if you think answer I is betler, respord with 1, answer is betler respond with you think the result tie, outpot Only respond either 'I" to indicate your choice.
for all key in keys do answerl t- modell [ ressonse"]] [rrs]"rsppos"""""
answer2 model2[key]] ressonnse""
fact, gt_ ss_answe,, question <- dataset[key]
prompt1 -- CreateProomii(qusstinn. fact, answwr answerl, answer2)
response_ 1 - CalIGPT(promptl)
prompt2 f- CheatePrommiiqqusttonn, fact, gt answer, answer2, answerl)
response_2 + CallGPT((rompp2)
score <- ComppueeScorerrsssnne_...l response_2)
Append scores with score
end for
Figure 15: GPT-RANKING Procedure
H Ablations on how to incorporate gaze
Figure shows different approaches exist for incorporating gaze data.
I
Generalizability of VOILA
We conduct an evaluation using the VizWiz dataset to validate the generalizability of our model. Given that VizWiz is a widely recognized benchmark in the Visual Question Answering (VQA) domain, we were able directly apply our model in this context. However, it is important note that the standard VizWiz VQA dataset does not provide positional information, which is critical for evaluating the distinetion between conventional VLMs and those enhanced with gaze input. To address this gap, we evaluated our model on the validation split of the VizWiz grounding dataset, employing a metric analogous that used in VizWiz VQA but incorporating grounding points as proxy for gaze input. Our model achieved an accuracy of 49.6, outperforming the baseline model, which scored 33.2. We believe these zero-shot results convincingly demonstrate both the effectiveness and generalizability of our method.
Impact Statements
Our experiments have been reviewed and approved by the Institutional Review Board (IRB) at our institute. All participants of the VOILA-GAZE collection process have provided informed consent, agreeing that their data will be used as part of a publicly available dataset. To ensure privacy, all personal information has been properly anonymized. Additionally, review of any potential privacy issues before releasing the dataset to the public. We are committed to upholding ethical standards in research and protecting the privacy of study participants.
will conduct thorough
Voila-A can be implemented HMD AR/VR devices as an egoview copilot, benefiting a wide range of users, including visually impaired individuals who rely on their gaze to communicate their intent.
22

---

Pretrained and Frozen
Pretroined and Frozen
Finetuned
Outeut mustroom can
Finetured
Outpot tert
The mushroom con be
LM Layer
LM Loyer
Perosiver Resampler
Gared Cross Attn
Parcelver Besompier
Gated Cross Attn
Visiart Encoder
LM Layer
Viston Encoder
LM Loyer
Gated Cross Aftn
Gated Cross Atto
Processed
Precessed
[Image] Usen betfer stou
[image] (ataalettoooo.......... -orooo---ooo
Pretralned and Frozzn
Finstuned
Prefralined and Frozen
Fineluned
mushrocn
Layer B
LM Loyer
Peraslvar Resampler
Gated
Parcniar Resampler
Liwer Lager
Goted Cross Attn
Kigt 20.5
187
LM Loyer
Enooder
LM Layer
Cross
Goted Cross Attn
Praceseed [krege]
Processed text
[Image]
[eak Beudeedaeeeeabtiee afear beer GPT:
GPI:
Figure 16: Different model design on how to incorporate gaze. Top-left: Otter-base, Top-right: Gaze as discrete position tokens. Bottom-left: Gaze bounding box as image patch Bottom-right bounding box as discrete position tokens
This method surpasses the capabilities of similar mobile apps that necessitate users lift their phones for scene capture.
K Reproducibility Statement
In the interest of promoting transparency and facilitating further research within the community, we are committed to providing comprehensive resources alongside the publication of our work. To this end, we will release the VOILA-COCO and VOILA-GAZE datasets, as well as the annotation pipeline, training code, and model weights. By making these materials publicly available, we aim to ensure that our methods can easily replicated and built upon by other researchers. Our commitment to reproducibility aligns with the broader scientific goal of fostering collaboration and the development of novel ideas and techniques the field.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Each contribution is discussed in our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine to include aspirational goals as motivation long clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please find see
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or a speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please sec
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Please see sec
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hatpsssccppulli/uuuudeeee httpa///gguiiee//ddeuuubiiisssee CodeSubmisss ssssSuboiiiiiooooioooc.oo. for more details.
The authors should provide instructions on data access and preparation, including how to access the data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bar is not applicable to our metric.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the emorr bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar the standard deviation or the standard error of the mean.
26

---

is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please sec
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental
well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We fully adhere the NeurIPS Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special considera- tion due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We disscussed in section J
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, it legitimate to point out that improvement in the quality of generative models could be used to generate
27

---

deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please find in see K
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All assets are properly cited and licensed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswi thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer: [Yes]
Justification: Please find in our method section.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Please find in see C
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our experiments have been reviewed and approved by the Institutional Review Board (IRB) at our institute. All participants have provided informed consent, agreeing that their data will be used as part of a publicly available dataset. To ensure privacy, all personal information has been properly anonymized. Additionally, we will conduct thorough review of any potential privacy issues before releasing the dataset to the public. We are commit to upholding ethical standards in research and protecting the privacy of study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "yqLo2RbLO4", "reviewer": "Reviewer_Hary", "summary": "The paper aims to improve the integration of VLMs in real-world applications such as AR/VR by making the interaction seamless and user-centric. This is achieved by incorporating users’ gaze information into the VLM for a more natural conversation and a frictionless experience. First, the LN-COCO dataset is proposed as a benchmark to train models for this task, after experimentally validating that mouse traces can be used as a proxy for gaze behavior. Furthermore, the authors collect and annotate VOILA-GAZE with real gaze data to be used as another test set. Second, the authors introduce the VOILA architecture which takes an image, a gaze heatmap, and a related question as input, and provides an answer as output. This model is compared against two popular VLMs on the two benchmarks using two newly introduced performance metrics. Ablation studies are also conducted to validate model design choices and training schedules.", "questions": "- L25-27: needs a reference or experiment to back it up. How is the limited alignment with human gaze detrimental for VLMs?\n- The resolution of the word clouds in Figures 10, 11, and 12 should be increased\n- Why is the (key of the) gated gaze feature added to (key of the) image features instead of multiplication?\n- Why is the VPR using an attention between latent embeddings, and a concatenation of latent embeddings and image tokens (+gaze tokens) instead of the more natural cross-attention from the latent embeddings to the image+ gaze tokens?\n- Is the learnable gating parameter $g$ a scalar or vector?\n- L274: What is meant by “the input and output embeddings of the language encoder”?\n- Is the model trained for 2 (L272-274) or 3 (L737) epochs?\n- Related to the previous point, the color of the Voila Perceiver Resampler module in figure 4 is inconsistent with the text. The figure says that the Perceiver is fine-tuned in the first stage,  but the text says otherwise (L273-274 and L379-381). Which one is it?\n- Is the learnable $g$ part of the “gaze-related weights” that are fine-tuned in the first step?\n- What are the “linear layers” used to encode the gaze heatmap? Is it a linear projection on each patch token? or an MLP?\n- Can you elaborate more on the role of [fixation] token?", "limitations": "The authors properly address the limitations of their work together with useful discussions, an impact section, and a reproducibility statement.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is really well-written, easy to read and sufficiently illustrated. The motivation is thoughtfully outlined, and the contribution is clearly stated and properly contextualized relative to prior works.\n- The paper addresses an important and original research topic that is severely underexplored despite being very relevant and holding significance for certain applications.\n- The introduced benchmarks VOILA-COCO and VOILA-GAZE are novel, and will be useful to the broader community working on this topic.\n- Overall, the proposed Voila model seems to outperform other baselines (Otter and Kosmos-2) on both datasets according to the chosen metrics.", "weaknesses": "- The biggest problem I have with the paper concerns the main experiments. The authors compare Voila to Otter and Kosmos-2, however important information about the evaluation is missing or unclear, which makes the interpretation of the results difficult:\n    - It is not clear whether the baseline models are trained on VOILA-COCO or simply evaluated.\n    - It is not explained how the authors deal with direct vs indirect questions during training and evaluation. In figures 5 and 6, are they counted as two separate instances? Or is only one of them selected? In this case, which one?\n    - Voila takes the gaze information as input, but it is not clear whether Otter and Kosmos-2 are fed that information as well. The comparison would be unfair in the latter case, especially since we have coreference queries with pronouns like “it” and no location information to disambiguate the intent. In L393-394, it is mentioned that Kosmos-2 can take bounding boxes as input, but does this mean it is used this way during evaluation to consider gaze input? Also, even for Otter which is not specifically designed to include location information, it would be possible to inject it in the text prompt (similar to the design in Figure 16 top-right and bottom-right)\n- Some quantitative results are suspiciously surprising. In table 2, the winning rate, which I assume is based on the GPT4 Ranking metric, is only 1% better for Otter with direct queries (51%) compared to indirect ones. How is this possible if Otter doesn’t have access to gaze information to disambiguate the question? I would expect many answers to be completely incorrect for any question that is not focused on an obviously salient part of the image. Also, all variants of Voila (including in-context prompt) underperform the base otter on indirect queries according to the same metric, even though Voila explicitly uses gaze. The numbers for the reward score, on the other hand, are more reasonable.\n- There is limited novelty in the architecture itself, since all the components are exactly like the Flamingo model. The contribution of the paper is in the introduction of the gated gaze heatmap tokens as an additive soft bias to the keys K computed from the image tokens within the Perceiver module. However, as far as I’m concerned, the value of the paper lies more in investigating and establishing the groundwork for an important research direction by proposing benchmarks, protocols and baselines for more works to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your support of our work, for your meticulous review, and for your thoughtful suggestions that have helped us improve.\n\n**W1**\n- Both Kosmos-2 and Otter are finetuned on the same training set of VOILA-COCO.\n- During the data generation process, we generate one direct question and one indirect question for each query content. During the training process, we randomly select one question from each question pair, i.e., the direct questions and indirect questions are expected to be balanced during the training stage. This training question composition remains unchanged for all tables and figures in the Experiment session.\nDuring the validation process, by default, we use only indirect questions (Figures 5 and 6, Tables 3 and 4). The only exception is Table 2, which shows the ablation results on query types, row 1/3/5 use indirect questions and row 2/4/6 use direct questions as validation set.\n- During evaluation, kosmos-2 takes bounding boxes as input, following the same format as in their paper and code repo. The way we preprocess bounding boxes for kosmos-2 is consistent with VOILA's variant in Table 3 (row 3 and 4). Specifically, we calculate the minimum bounding box that includes the gaze trace of each question. \nFor Otter evaluation, we do not directly inject gaze traces or bounding box data because Otter hasn't seen such kind of data during training. As shown in Figure 16 you've mentioned, the top-right and bottom-right setting maintains the same model architecture as Otter. These settings add special trainable tokens for the gaze trace/fixation bounding box, which can be considered as a location information-injected version of Otter. As shown in Table 3, our VOILA also surpasses these Otter variants.  \nIn Figures 5 and 6, although we compared VOILA to no location information-injected version of Otter, it is because the main goal of our paper is to (1) examine the necessity of additional signals for generating appropriate responses in everyday scenarios, and (2) evaluate whether gaze signals offer a superior representation of user intent compared to other modalities such as bounding boxes, instead of establishing a traditional benchmark for competitive model architectures based on specific metrics.\n\n**W2**\nThe GPT-4 ranking has 3 outcomes, that is, win, tie, or lose. As shown in Figure 5 \"Voila vs Kosmos-2\" Grounding column, approximately 40% winning rate already surpass its competitor. Therefore, in Table2, comparing row 1/3/5, VOILA surpasses Otter in indirect questions, and comparing row 2/4/6, VOILA surpasses Otter in direct questions. Also, comparing row 1/2, 3/4, 5/6, both model variants and prompt strategy performs better on direct questions compared to indirect ones. We will update tie/loss rate into the table in later revision.\n\n**W3**\nAs you kindly point out, instead of chasing technical novelty, our paper aims to implement modifications and prove them to be necessary and beneficial, establishing the groundwork for gaze-incorporated VLMs. \nIn developing VOILA, a critical consideration is the adherence to the established architecture of current VLMs, making it easy to follow and generally applicable. It is essential to avoid introducing a significant number of new parameters or making extensive structural modifications. This constraint is due to the limitations of the current gaze dataset, which does not support large-scale pretraining. Additionally, we must be vigilant in preventing catastrophic forgetting during the fine-tuning process.\n\n**Q1**\nThanks for pointing out and we will update our statement accordingly: Our study on Otter demonstrate that current VLMs fails on various daily usecases because of disalignment with users' intention.  Also, recent  HCI research like GazeGPT[1] incorporates only one gaze point as location information to GPT-4v in the form of bounding boxes, illustrating the improved alignment and user preference after the incorporation of gaze. \n[1] GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear\n**Q2**\nWill update high resolution illustrations.\n\n**Q3**\nFor better numerical stability, we choose addition instead of multiplication.\n\n**Q4**\nThis architecture, drawing inspiration from Perceiver and Flamingo, seamlessly integrates self-attention within latent features and cross-attention between latent and visual features in a unified step.\n\n**Q5**\nThe vector $g$ is a trainable multi-channel variable with dimensions corresponding to (Number of Heads, Dimensionality).\n\n**Q6**\nThat means word embedding and final projection after the last layer of language model.\n\n**Q7,8,9,10**\nSorry for the mismatch and we will revise it. Here is the fact: In the first stage colored in pink, we train the MLPs and gate $g$ of gaze inputs for 1 epoch, then tune together with Perceiver resampler module for 1 epoch.  In the second stage colored in orange, we finetune MLPs, $g$, Perceiver, Cross attention, word embeddings together for 1 epoch. In total, that's three epoch as stated in L737.\n\n**Q11**\nTo maintain the model's versatility in handling scenarios without trace input, the inclusion of a special token like 'fixation' is crucial for ensuring the model's robustness across diverse situations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxzWnMqPCZ", "reviewer": "Reviewer_3dvo", "summary": "This paper proposes Voila-A, an instruction-tuned VLM that integrates “gaze” information. The instruction data is constructed using tracing data from “Localized Narratives” and GPT-4 assistance, then converted to gaze-like data using the Bubbleview method. The overall approach is evaluated on newly collected datasets: Voila-COCO with 1900 questions (in the test set) and Voila-GAZE with 200 questions. A GPT-4 aided ranking of Q&A is employed to compare Voila-A against Otter (image input only) and Kosmos-2 (also takes bbox inputs), where Voila-A is shown to out-perform the baselines on both the synthetic and real-world evaluation datasets.", "questions": "- What is “1k gaze” (mentioned on page 3)?\n- It is mentioned in Sec. 3.4 that “the g in each VPB are initialized as 0”. How do these gating values look post-training? Does the value increase in later layers, or earlier layers? Why not initialize to random or non-zero-constant values?\n- Would Voila-A perform well in referring expression comprehension tasks such as RefCOCO, in comparison to Kosmos-2?", "limitations": "The authors briefly discuss the limitations of their work in Appendix A.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors motivate their case for a gaze-guided instruction-tuned VLM well. Understandably, collecting sufficient real-world gaze data can be challenging, so the authors propose to convert trace data from the Localized Narratives dataset to pseudo-scanpaths using Bubbleview - this is a creative and realistic solution to their problem. In addition, the authors collect real-world data using the Pupil Labs Invisible eye tracker from 21 participants and show that their proposed Voila-A out-performs contextual (Otter) and bbox-guided (Kosmos-2) approaches as assessed by GPT-4.", "weaknesses": "The proposed method in terms of the Voila Perceiver Resampler and Block (VPR and VPB) generally make sense. However, the decision on how to define the K, Q, and V are insufficiently motivated. Ideally, it would be good to see at least a simple ablation study to justify the design decisions that depart from the original Flamingo architecture.\n\nThe main results are easy to understand, especially when viewed with the Appendix in mind. However, it is a purely GPT-based evaluation and while it is a good effort, one wonders whether there could be a more objective and quantitatively distill-able way of measuring performance. For example, one could evaluate referring tasks or use a user survey to evaluate the methods against each other. A user study could be particularly helpful, as it would evaluate the quality of the VOILA-GAZE dataset simultaneously (and it only consists of 200 questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewr 3dvo, we sincerely thank you for your support of our work and appreciate your thoughtful suggestions that have helped us improve.\n\n**W1**\nRegarding the Perceiver's choices for Key (K), Query (Q), and Value (V), we can delve into a clearer explanation here. The primary function of this component is to condense visual information into compact latent representations, which can subsequently be utilized for cross-attention with language models. In this setup, the Q are derived from the latent representations, while the K and V originate from the visual inputs. However, calculating K and V from the latents as well allows for an efficient self-attention mechanism within the latents themselves.  In section 4.3.2 and Figure 16, we compare more design choices beyond Flamingo structure.\n\n**W2**\nThank you for your kind suggestion. Our primary goal is to establish a reproducible automated benchmark, and as such, most of our evaluations are conducted without the inclusion of supplementary user surveys. To guarantee that the question-answer pairs are amenable to automatic evaluation, each reference answer has undergone meticulous review by our dedicated annotator and has been subject to a rigorous double-checking process during the selection of the final test cases. We acknowledge the potential benefits of a user study on the voila-gaze results and will take into consideration the incorporation of such a study to further validate the dependability of our automated metrics.\n\n**Q1**\nNote that Section 2's motivation is to demonstrate the similarity between gaze trace and mouse trace so that training with VOILA-COCO can make sense. Therefore, we collected some mouse trace data on the VOILA-GAZE dataset to compare with its gaze trace. For this comparison, the gaze trace contains 1k sample of gaze points.\n\n**Q2**\nWe observe that the value of 'g' is larger in the median layers, whereas it is relatively smaller in both the later and the earlier layers. We initialize 'g' to zero to guarantee that the output matches that of the original model at the start of training.\n\n**Q3**\nRefCOCO's  referring expression comprehension task requires models to predict a bounding box with a given referring phrase, however, our VOILA model does not possess the ability to generate bounding boxes as it is not trained on such data. Instead, we were able to test VOILA's overall referring expression comprehension ability using VizWiz-VQA-Grounding dataset (as shown in Appendix I). VizWiz dataset is specially collected for answering visual question for blind people and the majority of VizWiz-VQA-Grounding dataset contains pronoun usage in the questions. Therefore, to accurately answer VizWiz's question, the model should understand what the pronoun refers to in the image.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EYw5U8w0yU", "reviewer": "Reviewer_CTT5", "summary": "In their paper, Voila-A: Aligning Vision-Language Models with User’s Gaze Attention, the authors introduce a dataset, a new model architecture Voila-A, which is a cognitively-enhanced VLM. After motivating the research and introducing both the datasets and model design, the authors cover a lot of experiments on the new open-source benchmark dataset. After evaluating their model and some baselines, they perform an ablation study and conlude the paper.", "questions": "What does Voila-A stand for?\nWhy didn't you compare to more baseline models?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "New open-source dataset in a low-resource modality (gaze, trace).\nNew open-source benchmark constructed from the dataset mentioned above.\nNew open-source VLM model (including training procedure etc.), based on OpenFlamingo that integrates both gaze and trace. \nIn detail description of the model and each layer as well as experimental details. \nMostly nicely written.\nExtensive ablation study.", "weaknesses": "Missing comparison to previous VQA/VLM models, e.g. Sood et al. [43], only one baseline mdel (Otter-Base).\nUnclear how hyperparameter were tuned. \nPresentation needs some work e.g. 223, 249, 276, replace \\cite{ by ~\\cite{ for more readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your support of our work and for your attentive reading and thoughtful suggestions.\n\n**Baseline models**\n\nNote that except for Otter, we also have Kosmos-2 as baseline, as shown in Figure 5 and 6. For reference [43], this is a paper published in 2020 and in their experiment description, they stated that all the models they used were published before 2019, which VLMs haven't emerged back then, therefore comparing our Voila to their models may be unfair and liekly to have obvious outcome.\n We choose Otter and kosmos-2 for several reasons:\n(1) these two models each represents a typical VLM architecture. Current dominat VLM architectures are either cross attention based where visual feature is injected to each layer of language backbone, or token based where visual tokens were treated similarly to language tokens in an autoregressive manner. For the baselines we've chosen, Otter and Kosmos-2 are both fully open-sourced and received hundreds of citations, representing each type of VLM architecture mentioned above.\n(2) Compared to other Open-flamingo based VLMs, Otter is tuned on a Multi-modal in-context instruction tuning dataset (MIMIC-IT), which we believe that building upon Otter should be the best choice for a user-centered QA manner like Voila, because Voila's use case also involved in-context querying and need to follow user's instruction.\n(3) Compared to other VLMs that have similar architecture to Llava, Kosmos-2 has grounding ability by inputing bounding-boxes, which is the most common strategy of incorporation location information currently, thus forms a comparable baseline against our location injection strategy.\n\n**Presentation Issue**\n1. Hyperparameter settings can be find in Appendix F.\n2. We will proof read again and fix format issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JBy2ckvFSD", "reviewer": "Reviewer_qe53", "summary": "This work proposes an approach to align Vision-Language Models (VLMs) with users' gaze. To develop such a method, the project first creates a mock dataset of human gaze from image captions, known as Voila-coco. It then develops a model to integrate gaze data into VLMs. Additionally, a new dataset called Viola-gaze has been introduced.", "questions": "- Why textual descriptions are not enough in the provided application scenarios, for example directly include orange in the question instead of using it with gaze point? \n- What are the benefits of incorporating gaze? \n- Why not use an eye tracker to directly collect gaze instead of using BubbleView? \n- What do Otter and Kosmos2 refer to? Why are they chosen for comparison?\n- Why not just directly use human preference to evaluate the performance of VOILA? What's the performance of the reward-scoring network?\n- Similarly, what would be GPT-ranking results compared to human preference?", "limitations": "The authors have addressed the limitations to a very limited extent. There is no discussion related to the gaze modality and the limited amount of data for example.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "This paper proposes an interesting idea of aligning the attention of vision-language models to human attention indicated by fixation locations. It contains several components ranging from data collection to model development.", "weaknesses": "- Motivation: The motivation for aligning VLMs to gaze attention is to enhance the models' interpretability and effectiveness (L8). However, if I didn't miss anything, the experiments do not evaluate interpretability. More importantly, it is not clear to me why textual descriptions are not enough in the provided application scenarios and what the benefits of incorporating gaze are. \n- Clarity: I find it difficult to follow and understand what exactly has been done in the project. For example, Sec. 2 explains that this project uses BubbleView to collect gaze-like data. However, the motivation to do so is not clear. It is also not clear how good the transformation of trace data to pseudo gaze sequences is. Overall, the write-up and general presentation can be improved. \n- Limited technical contribution: I understand the challenge when training with limited data, however, this also limits the design of the model. The encoding of heatmap generated from gaze sequences seems to have limited technical contribution.   \n- Evaluation: first of all, it is not clear what Otter and Kosmos2 refer to and why they are chosen for comparison. It is also not clear what the GPT-4 ranking results mean. Similarly, why not just directly use human preference to evaluate the performance of VOILA? What's the performance of the reward-scoring network? It seems to me that both evaluation methods need to be evaluated first.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your critique of our work and will take your comments into consideration for future revisions. We would like to clarify a few points that may have been misunderstood, hoping this will enable you to re-evaluate our work more accurately.\n\n**W1 & Q1, Q2 on Motivation**\nWe envision that gaze-facilitated visual question answering may take place on future wearable head devices, such as VR/AR and smart glasses. During such an interaction process, users may ask contextual questions that have proven to be ambiguous by many human-computer interaction (HCI) researches, such as using pronouns to refer to surrounding objects. Therefore, using \"it\" instead of the object name, which is the indirect question in our case, is very likely to happen in real-life scenarios.  Furthermore, with advanced devices like the Apple Vision Pro utilizing gaze as a crucial interface for spatial computing, the integration of gaze tracking into Vision-Language Models (VLMs) presents an important yet unexplored challenge.\n\nAnother important use case of incorporating gaze is visual question answering for visually impaired people. The VizWiz-VQA-Grounding, which we also evaluated VOILA on in Appendix I, presents questions collected with blind people and the majority of these questions have at least one pronoun in them.  \n\n**W2 & Q3 on the process of gaze collection**\nThe main experiment in this paper is the gaze-incorporated VLM, which refers to Voila. To train Voila, one challenge lies in the training data, where it is hard to collect a large-scale gaze VQA dataset. Alternatively, we used a mouse trace dataset called Localize Narrative as a substitution and constructed visual questions using GPT-4, thus we derived a trace-augmented VQA dataset, which we refer to as VOILA-COCO in this paper. The above is the main workflow of our project. \nHowever, we need to prove that mouse trace can be an approximation for gaze trace, even though this approximation has been used for many HCI research already. This is the reason why BubbleView is used in Section 2: we want to illustrate one basic belief of our work before we dive into it.  To illustrate this, the VOILA-GAZE dataset we collected already contains gaze trace data because it is collected by gaze-sensing smart glasses, then we use BubbleView to derive mouse trace annotation on the same dataset. By comparing the two trace modalities, we provide strong evidence for using VOILA-COCO as the training dataset for a gaze-incorporated VLM.\n\n**W4 & Q4, Q5 on Evaluation**\nWe choose Otter and kosmos-2 for several reasons:\n1. these two models each represents a typical VLM architecture. Current dominat VLM architectures are either cross attention based where visual feature is injected to each layer of language backbone, or token based where visual tokens were treated similarly to language tokens in an autoregressive manner. For the baselines we've chosen, Otter and Kosmos-2 are both fully open-sourced and received hundreds of citations, representing each type of VLM architecture mentioned above.\n2. Compared to other Open-flamingo based VLMs, Otter is tuned on a Multi-modal in-context instruction tuning dataset (MIMIC-IT), which we believe that building upon Otter should be the best choice for a user-centered QA manner like Voila, because Voila's use case also involved in-context querying and need to follow user's instruction.\n3. Compared to other VLMs that have similar architecture to Llava, Kosmos-2 has grounding ability by inputing bounding-boxes, which is the most common strategy of incorporation location information currently, thus forms a comparable baseline against our location injection strategy.\n\nThere are 2 major concerns about using human preference:\n1. For the majority of questions in VOILA-COCO, the correctness of an answer can be judged with a reference answer and ground truth description given, while human preference is necessary for tasks that can not be judged objectively such as writing or image generation.\n2. The goal of this paper is to establish a groundwork for gaze-facilitated VLMs, therefore reproducibility is very important. This yields the demand for an autonomous evaluation pipeline instead of human preference, which results may vary depending on multiple factors, such as region/religion/education/gender/etc.\nHowever, as a user intention incorporated VLM, reflecting user preference during the evaluation process is indeed a matter. We believe the GPT-4 ranking metric can reflect user preference since it is trained with tons of user preference data through RLHF, further, we believe that the user preference data used in GPT-4 should generally be more unbiased. This has also been demonstrated in many other works such as GPTScore[1], GPTRank[2]. \n\n[1] Fu, Jinlan, See-Kiong Ng, Zhengbao Jiang and Pengfei Liu. “GPTScore: Evaluate as You Desire.” ArXiv abs/2302.04166 (2023): n. pag.\n[2] Liu, Yixin, Alexander R. Fabbri, Pengfei Liu, Dragomir R. Radev and Arman Cohan. “On Learning to Summarize with Large Language Models as References.” ArXiv abs/2305.14239 (2023): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Most of the reviewers appreciate the proposed method and dataset for aligning gaze and the VLM model. Although there are co-current works as I can see from arXiv, the proposed dataset and method have significant contributions to the community. One reviewer raised multiple concerns about the paper, and the author replied with a detailed rebuttal. 
I suggest to accept this paper.
Decision: Accept (spotlight)