Toward Semantic Gaze Target Detection
Authors: Samy Tafasca
Anshul Gupta
Victor Bros
Jean-marc Odobez
Venue: NeurIPS 2024 poster (2024)
Keywords: gaze following
dataset
deep learning
computer vision
Primary area: machine_vision
TL;DR: We extend the gaze following task, which is currently focused on localization alone, to also incorporate the class label of the gaze target. We propose new benchmarks and a design a new architecture that outperforms existing methods.

Abstract: From the onset of infanthood, humans naturally develop the ability to closely observe and interpret the visual gaze of others. This skill, known as gaze following, holds significance in developmental theory as it enables us to grasp another person’s mental state, emotions, intentions, and more. In computer vision, gaze following is defined as the prediction of the pixel coordinates where a person in the image is focusing their attention. Existing methods in this research area have predominantly centered on pinpointing the gaze target by predicting a gaze heatmap or gaze point. However, a notable drawback of this approach is its limited practical value in gaze applications, as mere localization may not fully capture our primary interest — understanding the underlying semantics, such as the nature of the gaze target, rather than just its 2D pixel location. To address this gap, we extend the gaze following task, and introduce a novel architecture that simultaneously predicts the localization and semantic label of the gaze target. We devise a pseudo-annotation pipeline for the GazeFollow dataset, propose a new benchmark, develop an experimental protocol and design a suitable baseline for comparison. Our method sets a new state-of-the-art on the main GazeFollow benchmark for localization and achieves competitive results in the recognition task on both datasets compared to the baseline, with 40% fewer parameters

Toward Semantic Gaze Target Detection
Samy Tafasca Idiap Research Institute
Anshul Gupta Idiap Research Institute
École Polytechnique Fédérale de Lausanne stafasca idiap. ch
École Polytechnique Fédérale de Lausanne agupta@ i
d diaip.p.... .p.ch
Victor Bros
Jean-Mare Odobez
Idiap Research Institute
Idiap Research Institute
École Polytechnique Fédérale de Lausanne vbros@idde wiie@idipp...o
École Polytechnique Fédérale de Lausanne odobe 0 di idiap ch
Abstract
From the onset of infanthood, humans naturally develop the ability to closely observe and interpret the visual gaze of others. This skill, known as gaze following, holds significance in developmental theory it enables to grasp another person's ss mental state, emotions, intentions, and more [6]. computer vision, gaze following is defined the prediction of the pixel coordinates where person in the image is focusing their attention. Existing methods in this research area have predominantly centered on pinpointing the gaze target by predicting gaze heatmap gaze point. However, a notable drawback of this approach is its limited practical value in gaze applications, as mere localization may not fully capture our primary interest understanding the underlying semantics, such as the nature of the gaze target, rather than just its 2D pixel location. To address this gap. extend the gaze following task, and introduce a novel architecture that simultancously predicts the localization and semantic label of the gaze target. We devise a pseudo-annotation pipeline for the GazeFollow dataset, propose a new benchmark, develop an experimental protocol and design suitable baseline for comparison. Our method sets a new state-of- oftte-of-the-are art on the main GazeFollow benchmark for localization and achieves competitive results the recognition task on both datasets compared the baseline, with 40% fewer parameters.
Introduction
Gaze is an important marker in non- verbal communication that is indicative of a person's visual attention. also proxy measure of cognition and can be used to evaluate subject's intentions, preferences, emotions, among others. Consequently, it received lot of attention over the years from different research communities such as neuroscience 13 psychology
[7] cognitive science 44
robotics 43 and education [37].
In computer vision, the analysis and understanding of attention was formulated through different tasks. One research direction focuses on predicting a gaze direction representing the 3D line of sight from a frontal image of face (34]. Another one tries to estimate the visual focus of attention (VFOA), i.e. the gaze target of person, given 3D information about the subject (e.g. body, head, eyes) and the scene (e.g. layout, object positions) [45 Beyond predicting gaze as standalone signal, several research efforts focused understanding gaze dynamics in the context of social communication
such as inferring mutual gaze [35
15 20 joint attention [4 20 or gaze aversion 15
This paper focuses the task of gaze following [2), which extends the idea of Visual Focus of Attention (VFOA). Gaze following aims to predict the 2D pixel coordinates where a person in an
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

image is looking, The major benefit of this formulation is that it makes no assumptions about the scene and doesn't 't require additional equipment, such as wearable devices. However, a notable drawback is that solely predicting the pixel location of the gaze target often falls short for real-world applications that demand additional information, such as object class or social gaze class.
One possibility to address this limitation is to post-process the output of gaze following method by verifying a gaze point falls within the bounding box of a detected object [10]. However, this multi-stage process entails additional computation, often requiring the use of additional pre-trained models, leading to inefficiency and less than optimal results. Furthermore, pre-trained detectors typically ignore uncountable objects (e.g. wall, sea) which are often possible gaze targets. Finally, unlike object detection, in gaze following we predict heatmaps and not boxes, which makes applying separate object detector afterwards challenging. In such case, how do we match a gaze heatmap to the right object box? We could consider the gaze point (i.e. arg max) as mentioned before, but what if the point falls within multiple boxes? And what if the heatmap is multimodal, and the arg max happens to land on the wrong target? The joint training of the gaze heatmap and the gaze target class, aside from being the more natural formulation, allows the model to learn the best way to dynamically make sense of the heatmap in order to infer the right class.
An alternative approach is frame the problem as a Human-Ohjectt-neeren task, where looking serves as an interactive action. However, many existing HOI datasets lack consistent and systematic labeling of the looking behavior. To the best of our knowledge, V-COCO [22] is the only dataset doing so, but it is limited to 80 object classes, which may not encompass the full spectrum of potential gaze targets found in images, including cases where individuals look at locations categorized as stuff semantic classes. Moreover, other datasets focus on a limited number of classes (e.g., watching TV or a cell phone), thereby biasing the looking task towards specific objects. Consequently, utilizing an HOI verb-object task formulation with existing HOI datasets might result in learning spurious correlations, wherein the detection of particular object (e.g. TV) strongly suggests a specific verb (e.g. watching). This is well known issue in HOI and has fostered the development of benchmark datasets specifically designed to evaluate HOI methods based on evidence rather than relying on dataset-specific correlation priors (25. Given these limitations, there a clear need to explore the task in novel manner and develop new datasets and protocols to address this research topic effectively.
(HOI) recognition
In this paper, we propose an end-to-end architecture that predicts both the localization and the class label of the gaze target, addressing the limitations above. Furthermore, we frame the recognition part as a visual-text alignment task, which offers the benefit of generalizing to other classes beyond the training vocabulary. To this end, we make the following contributions
We address, for the first time, the semantic gaze following problem by devising a visual- language architecture that efficiently tackles both localization and target class categorization tasks simultancously.
We introduce novel benchmarks, new baseline for comparison, and experimental protocols for investigating the extended task, drawing from datasets within the gaze following and
HOI communities.
Our architecture sets a new of in gaze target localization on the main GazeFollow benchmark dataset, and demonstrates strong categorization performance compared to more complex and computationally intensive baselines.
2
Related Work
Gaze Target Detection. Traditional methods for estimating the Visual Focus of Attention (VFOA)
[4] 39 [8, 43 [ 36) were limited by their reliance on specific scene or activity priors, which hindered their generalization to more arbitrary settings where such priors could not be provided. To overcome these limitations, Recasens et al. [41] proposed novel formulation of the problem, aiming to infer the 2D image coordinates that correspond to the scene target being observed by a person in the image. Standard methods for gaze following adopt a two-branch architecture, comprising a scene branch for saliency detection and a head analysis branch for the person of interest to infer a gaze direction. Information from the two branches is then fused to predict the final gaze heatmap. This type of architecture has demonstrated robust performance in gaze following, with the ability to also
predict if the person is looking within the image outside p 32
48
49 26
16 5 27
Semantic Gaze Following, Building on this foundation, we introduce an architecture that not only estimates gaze target localization but also identifies the class label of the gaze target. To the best

---

Heatmops
Image Encoder
Gaze Decoder
Embeds
Scene Image
Liab
Gate Heatmmpp
Bactbone
Upacolerr
Gaze Encoder
Text Encoder
HLP
Gode rabel Embadds
Head
Bbok
Crops bbox
Mocoabution
Gaze Decoder
Overall Architecture
Gaze Encoder
+ Aciiition
Fical-woiie Pat-Product
Image
Tokenn
Fimbeeking
Figure 1: Overview of our architecture. [A] The scene image is passed through Image Encoder to produce image tokens (blue squares). [B] The head crops and head box coordinates are processed by Gaze Encoder generate gaze tokens (orange squares). [C] The image and gaze tokens are fed to Gaze Decoder to predict both the gaze heatmaps and (normalized) gaze label embeddings (yellow squares) through series cross-attention operations. [D] The text encoder computes (normalized) class embeddings (green squares) based on predefined vocabulary of concept classes. [E] Finally, we compute similarity scores between the predicted gaze label embeddings and vocabulary embeddings.
the first address this task. The closest work in this direction is from Tonini et al. [50] which proposed model that simultaneously predicts the gaze heatmap and detects objects in the scene. Their approach involves primary branch that identifies both the head and object bounding boxes within a scene, followed by feature extraction and gaze cone estimation to predict the best candidate object of focus the cone and its corresponding heatmap. is important to note that object detection here only serves an auxiliary task to improve gaze localization performance and is not meant to produce a class label. In fact, the target object is often not detected, and only bounded vocabulary set is supported. Our work seeks to address this limitation by adopting a weakly supervised contrastive objective aiming align vision and
of knowledge,
Architecture
Inspired by the idea of promptable segmentation [30]. design architecture (See Figure T for promptable gaze following where the scene image is processed separately from people. We decode gaze outputs by prompting the encoded image representation using person-specific information (i.e. head crop and box coordinates). This is achieved by lightweight transformer decoder that can process multiple people the same time separately for the expensive scene encoding operation to be performed only once in a person-agnostic manner. Then, after we obtain the gaze tokens of people, their gaze targets (i.e. location and label) can be decoded efficiently. Intuitively, we expect the image representation to identify gaze target candidates (i.e. salient objects), and for the decoding step act as a filtering mechanism that selects the right gaze target based on the query person. We empirically verify this hypothesis in the qualitative analysis section. is important following approaches [TO 16 27 21 that fuse the person and image representations early on. This makes them inefficient for multi-person inference because the expensive encoding repeated for each individual. We provide more details about each component below.
batch. The rationale behind this design is
that this design in stark contrast to most previous two-stream gaze
3.1 Image Encoder
Given an input image XC we employ a transformer encoder to produce image tokens xing E NDD, where N = w h is the number of patches, and the dimension of the transformer. We can also use a convolutional backbone the image encoder, assuming we equip the output feature map with positional information.
t W, p. P is the patch size,

---

3.2 Gaze Encoder
The goal of the gaze encoder is to process the head crop of the target person at a higher resolution (e.g. 224 x 224), as well as its location in the image in order to produce position seattiin-eenstivee embedding capturing directional gaze information. Given an input head crop haop E R # X we pass through a backbone followed by a 2-layer MLP to output a gaze token x E R Additionally, we project the corresponding head bounding box hobox E R+ to D using linear layer and add it to the gaze token to make it aware of the person's 's position in the image. If we process Np persons, we get the final gaze tokens x= E R. Please note that we can batch process multiple people in different images by merging the image and person dimensions the batch dimension. However, this requires the use of padding or truncation of people to ensure Np is the same across all images.
3.3
Gaze Decoder
The goal of the gaze decoder to combine information from the scene and the people in order to predict the gaze heatmaps and label embeddings. It is composed of a transformer with N blocks, an upscaler module and two MLPs for the final predictions.
Once the image is encoded, the image tokens x x and gaze tokens x go through the transformer decoder blocks, which are composed of two cross-attention operations and a ferd-forward module, with residual connections and normalization layers in-between. The gross-atteniin operations go in two ways: one where the gaze tokens x generate the queries, while the image tokens xiee generate the keys and values (we call this x + x or gaze-to-image cross-attention), and one where image tokens x!m_ generate the queries while the gaze tokens xtaze generate the keys and values (we call this x -> xeaee or image-to- -ozaaz cross-attention). This two-way design allows image tokens to incorporate information from gaze tokens, and vice-versa. This helps them better align for the final dot-produee operation that will produce the predicted gaze heatmap.
After the N blocks, we transform the gaze tokens one last time using a final x8 > xig cross- attention layer. The output is updated version of the gaze and image tokens, which we also call x E IR ,2,x xD and xingg E I NxD to ease notation. In Figure | x x!ngg flows through the decoder following the blue arrows, while xaae flows through the orange arrows.
To obtain the final predictions, we use the task-specific MLPs in the following manner: (i) the gaze passed through Gaze Label MLP composed of 6 layers to predict the gaze label embeddings glah E R N,,,,, where Dj is the dimension of the output gaze label embedding, (ii) the gaze tokens xaae fed to Gaze Heatmap MLP composed of layers project them to a lower dimensional embedding xaae E R N₂ xd where d < D. At the same time, the image tokens are rearranged into spatial feature map xigg E <RixwxD D and upscaled to the resolution of the output heatmap while reducing the number of channels ximgg E R Him XW xd The Upscaler module is composed of two blocks of one interpolation followed by one convolutional layer each. Finally, for each projected gaze token, we apply a dot-product with every spatial position of the upscaled image tokens to obtain the final heatmaps H. E R N, Man x Whm, Essentially, this dot- tro-prrduct acts as a filtering mechanism that localizes the gaze target of the subject thanks to the alignment done by the decoder between the gaze token and the image tokens the right spatial positions.
tokens x E R &
Additionally, the gaze decoder uses learnable token called the not gaze token whose role is to expand the range of the x - x cross cross-attention attention. Recall that this operation updates the image representation with information from the gaze tokens i.e. each image token gets updated with a weighted sum of the (values of the) gaze tokens. Conceptually, the not gaze token allows tokens of image regions where nobody focuses their attention to be updated by a different "null" token instead of forcing a weighted sum on the existing gaze tokens. We verify this hypothesis in a later section.
3.4
Text Encoder
Since we frame the prediction of the gaze label as a text-vision alignment task, our architecture also requires a text encoder to convert the vocabulary of class labels into text embeddings. We chose CLIP 40 as our text encoder, and we kept it frozen.
During training, the predicted gaze label embedding is aligned with the available class label em- beddings. During inference, we first build a vocabulary of class labels, pass them through the text encoder to get their embeddings, then compare the predicted gaze label embedding with each one of them and select the class label with the highest similarity score.
4

---

3.5 Losses
Heatmap Loss (Creg). is the standard pixel-wise MSE between the GT and the predicted heatmaps:
W Ha
2
Label Loss (Ciob). The label loss follows a similar formulation to the multimodal contrastive InfoNCE used in [40] with two main differences: 1) our loss is not symmetric (i.e. we only consider the image to text loss component), and since our language labels are (pseudo-) classes and not full captions, there will inevitably be class redundancies within each batch (i.e. two or more samples having the same gaze label), so we consider the unique ground-truth gaze labels in each batch.
pred and their associated Formally, given a batch of N predicted visual gaze label embeddings I = Blab,
ground-truth class embeddings T; Biab,s and ground- truth classes y/, the loss be expressed
N
exp (s(Li, T.)/T) Ejeu exp (s( 12)/T)
Llub
log
N
1=1
where 7 is a learnable temperature parameter, s(a. b) is the similarity between a and 6, and U is minimum subset of indices in [1. NJ which identifies all classes in the batch, i.e. {y,, E U4) {v., i = ....N .N and Y(j,j') E 22, + yj.. This loss aims to maximize the cosine similarity between each predicted visual gaze embedding and the corresponding class embedding, while minimizing the similarity between the negative pairs. This formulation bears resemblance to the idea of supervised contrastive learning presented in [29]
Angular Loss (Cang). Optionally, we can append a second MLP head to the backbone in the gaze encoder in order predict a normalized gaze direction vector from the input head crop. This vector can be supervised using the angular loss, which is defined based on the cosine of the angle between the predicted and ground truth gaze vectors according to:
Lang 1- <g
gvee Oves
where < a, b > denotes the inner product between a and b. While the anguler loss doesn't influence the final performance, we found it be very useful in interpreting model predictions. For example, it helps to understand whether a failure mode is due to the head processing part (e.g. when the face is not visible), or the target selection part. Furthermore, it can be informative in real- sol-w applications when the gaze heatmap is not reliable (e.g. when the person is looking outside the frame).
Global loss. The global loss is given by: L Atm Crm + AtabCiab bongCang
4 Datasets
4.1
GazeFollow
In order to train our end-to-end architecture, we need annotations for the semantic label of the gaze target. Unfortunately, there is gaze following benchmark that provides annotations for both the gaze target position and class label. In order to solve this problem, we design a pseudo-annotation pipeline to automatically infer the semantic label of the ground-truth gaze target. Considering that gaze following datasets come with point annotations for gaze targets, our approach is to first segment the image, then match the gaze point with the predicted semantic class of the underlying pixel. Since we need to segment the images entirely, it is important to ensure that the segmentation we perform incorporates an open vocabulary that is able to describe any object encountered the dataset.
To this end, we use the GazeFollow dataset [42), and leverage two open-source projects that implement open-wond segmentation using various foundational vision and language models. The first method, known as SIMM-Grounddd-... SAM" generates precise masks and accurate semantic labels. However, has a tendency to overlook many regions in the image. To fill this gap and pseudo-annotate the missed gaze instances, we utilize a second method called provides more comprehensive coverage of the image, but it comes with the trade-off of introducing noisy labels and oversegmentation where many small segments are constituents of larger object. We provide a comparison of the two segmentation methods in the supplementary.
This tool
httpp:///githhbb.c bhttss//gituuh.cmm//EAAAA haunss//gghhubbooom//nnnazzg///e wtwpr/annnniiieggetttnneniieieee Anything
Segment-Anythen
5

---

Running this pipeline on our dataset produces ~ 5000 unique labels. We perform a series of text processing operations to clean these labels (i.e. correct incomplete words, remove stopwords. duplicates, adjectives, and prepositions), eventually leading to reduced vocabulary of ~ 3700 pseudo-labels.
Finally, in order to reliably test our models, we manually annotate the test set of GazeFollow with target classes. This annotation process is not restricted to a predefined set of objects, but we ensure that the labels are consistent (e.g. by avoiding synonyms when possible). Also, since the area where a person is looking, represented as a heatmap, often includes multiple objects, we also annotated other possible gaze targets whenever possible. For example, person cutting a cake is probably looking at both the cake and the knife. This multi-label annotation approach also helps to deal with the ambiguity related to object hierarchy (e.g. a person looking at the wheel of a car is also looking at the car, so it makes sense to assign both to the gaze instance). At the end, we obtained a test set ground-truth truth vocabulary of 346 classes (cf. the supplementary for word cloud).
4.2 GazeHOI
Aside from GazeFollow, we also introduce new benchmark for the simultancous localization and recognition of the gaze target. Since the assignment of a class label is predicated on determining the location of the gaze target first, it is very difficult manually annotate new dataset from scratch. This is because we can not define a vocabulary of object classes that we want to annotate in advance, and ensure people are only looking those objects.
To circumvent this problem, we repurpose existing human-object interaction datasets (i.e. annotations for person box, object box, object class and interaction verb) to create GazeHOI. The process is described as follows: (i) we combine 5 HOI datasets (V-COCO [22], HICO-DET (7). HCVRD (53) SWiG-HOI (51), HOI-A R (ii) for each dataset, we manually select a subset of verbs from the top 200 frequent ones where the person is also likely looking at the object (e.g. cud or repair, but not carry). (iii) we use an off-the-shelf head detector detect head bounding boxes and match them to the annotated person's 's bounding box, then we filter out instances without detected heads, (iv) for each head-object HOI instance, we draw their bounding boxes on the corresponding image, (v) team of annotators looks at these images, and answers yes or no based on whether or not the bounding boxes are correct, and the person is looking at the object they are interacting with, and (vi) discard no instances. This process solves our problem, and has the benefit of reducing the usually expensive manual annotation to a simpler manual verification.
At the end, we obtain total of 43808 images and 58146 instances after discarding about 50% of images in the verification step. Each instance is annotated with the person's 's body and head boxes, the object box, the object class, and the interaction verb. The vocabulary has 985 object classes, from which we isolate 522 rare classes (i.e. less than 10 instances each) into a separate split. This can be used for future research on open-vocabulary semantic gaze following, similar to the practice adopted in ppn-vocabulary object detection 19] The final dataset used in our experiments features vocabulary of 463 object labels and 55995 gaze instances which we split into 47214/3781, /7700000000000 /5000 for the train, val and test sets. Last but not least, we also run a deduplication pipeline to ensure that the validation and test sets contain no images from the training sets of GazeHOI or GazeFollow since both of them are based on popular overlapping vision benchmarks. In Figure samples from GazeHOI (cf. the supplementary for a word cloud of the vocabulary).
2
we show a few
5 Experiments
5.1 Baseline
Aside from comparing with previous gaze target localization methods from the literature, we also propose a strong baseline to assess our gaze target recognition performance given that no such work was attempted before. To this end, we design a 2-stage method as follows: first, we freeze our proposed gaze model and use it to predict gaze heatmap. Next, we use this heatmap to condition the original image by emphasizing the focused area. Then, we apply the CLIP's 40] pretrained vision model on the resulting image to produce the gaze label embedding. The idea is to leverage the alignment between CLIP's vision and text encoders. Finally, in terms of conditioning the input image based on the predicted heatmap, we consider three variants: masking, blurring and cropping. Figure
3 shows an overview of the baseline as well as the different conditioning variants.

---

Cell phone
Cow
Book
Figure Samples from GazeHOI, show the head box (white) and the object's box (red).
Masked Image
Vocabutary
Vikae Modil
CUP
Figure 3: [Left]] Overview of the proposed baseline architecture using masked conditioning. The similarity scores S, are computed from the gaze label embedding I, and the class embeddings T,- [Right] Comparison of the conditioning variants based the predicted gaze heatmap: original image (top left), masking (top right), blurring (bottom left), and cropping (bottom right).
As an additional benefit, since our model's predicted label is dependent on the predicted gaze location, using our own gaze model to generate the heatmap for the baseline allows us to control for the localization factor, enabling an unbiased comparison recognition performance.
5.2 Comparison with the State-of-the-art
We summarize our quantitative results on the GazeFollow benchmark in Table (the experimental protocol is provided in the supplementary). In terms of localization performance, our architecture sets a new state-of of-the-art across both metrics, surpassing the second best method Tafasca er al. by 4.4% and third best of Jin al. by 8.5% the Avg. Dist metric. In terms of recognition, method outperforms all 3 variants of the frozen baseline by a significant margin, i.e. more than 20% flat accuracy points compared to the second best (i.e. crop variant).
Since CLIP has probably not been trained on many images that are heavily masked, blurred or cropped, we decided to add two more variants by fine-tuning CLIP's vision encoder: (1) we directly fine-tune the baseline (crop) on GazeFollow. During training, the cropping is based on the ground- the
truth heatmap instead of predicted one. instead of feeding CLIP modified images, original ones and apply the conditioning at the feature level. Specifically, we perform a weighted average of the output image tokens (i.e. from the CLIP vision encoder applied to the input image) based on the (downscaled) heatmap (i.e. predicted by the gaze model), followed by a projection. Since the heatmap might not cover the entire object, this last variant allows the model have access to the surrounding context. The performance of these two variants is much closer to our method, where the cropping one slightly worse, and the heatmap weighting is slightly better.
It is important to remember two key facts about the design of our baselines: (i) They build upon our model's localization ability (i.e. the most accurate to date) which contributes to recognition performance. To verify this, we replaced the gaze heatmap prediction part in the best version of the baseline by the model from [10] and found that performance drops and becomes worse than proposed model (refer Table Baseline ). (ii) They add entire vision transformer that
7

---

Localization
Recognition
Method
Avg. Dist. +
Min. Dist.
Acc@1
Acc@31 f
MultiAcc@ 1
Random
0.471
0.391
0.002
0,010
0,003
Bias Majority
0.295
0.229
0.010
0.015
0.011
Chong al. T0
0.137
0.077
Fang et al. [6
0.124
0.067
Jin
[27
0.118
0.063
Bao et al. [
0.122
Tafasca et al. 48
0.125
0.064
Jin al. 26
0.126
0.076
Tafasca et al.
0.113
0.057
Baseline (Mask)
0.108
0.051
0.124
0.253
0.147
Baseline (Blur)
0.108
0.051
0.190
0.362
0.222
Baseline (Crop)
0.108
0.051
0.239
0.428
0.278
Baseline (Crop) fine-tuned
0.108
0,051
0.437
0,588
0,504
Baseline (heatmap weight)
0.108
0.051
0.466
0.653
0.542
Baseline (heatmap weight)
0.137
0.077
0.442
0.620
0.514
Ours
0.108
0.051
0.447
0.642
0,516
Table I: GazeFollow dataset. The best scores given in bold, while the second best underlined.
All baselines use our own model for gaze heatmap prediction, except for Baseline! which uses (10).
is pretrained on large-scale dataset specifically designed for such semantic recognition tasks, In terms of parameter count, our model features 116M while the baseline has 200M. This is because our model only uses an MLP head (3.3M parameters) for label prediction while the baseline uses ViT (86.6M parameters) to encode the semantics of the image separately. That is a decrease of 42% in the total number of parameters (ef. the supplementary material for comparison of FLOPS). We later show in our ablations that performance drops significantly if try use the same CLIP vision encoder to do both localization and recognition, which emphasizes the need for the baseline have two specialized ViT encoders. Finally, upon manual inspection of test set samples where our model and the best performing them are failures due to hierarchy or semantic similarity (e.g. prediction of ball then racket when ground-truth is racket). Please refer to the supplementary material for some qualitative examples to illustrate this comparison.
saetmmap-weigtte baseline don't 't agree, we noticed that most of
On GazeHOI (ef. Table 2). observe similar trend with our method out- performing the zero-shot baselines by large margin, while both fine-tuned vari- ants outperform
Localization
Recognition
Method
GazeAce f
Acc@1 1 Acc@3
model recogni- tion scores. We note that the pre-trained GazeFollow model is already able to per- form the localization task (i.e. 65% vs. 72% for Gaze Accuracy). However, the zero-shot performance is much lower than the fine-tuned counterpart (i.e. 30% vs. 58%). This is probably due to the mis- match between the pseudo-labels seen in GazeFollow and the vocabulary of Gaze- HOI (i.e. about 150 classes are new).
Random
0.166
0.002
0.006
Bias / Majority
0.352
0.082
0.083
Baseline (Mask)
0.723
0.197
0.298
Baseline (Blur)
0.723
0.306
0.458
Baseline (Crop)
0.723
0.388
0.546
Baseline (Crop FT)
0.723
0.617
0.707
Baseline (Hm weight)
0.723
0.646
0.748
Ours
0.652
0.306
0.463
Ours
0.723
0.583
0.706
Table Results our model and baselines on the Gaze- HOI dataset. The best scores are given in bold, while the second best underlined. The sign means the model was trained on GazeFollow and evaluated on GazeHOI without fine-tuning.
Finally, we need to emphasize that unlike the baselines, our gaze label prediction head is trained from scratch, so we be- lieve that the small scale of the datasets also plays an important role in limiting recognition performance (i.e. 100K vs. 400M instances).

---

5.3 Model Analysis & Ablations
Image Encoder. We hypothesized previously that our architecture design should steer the image encoder into capturing possible gaze target candidates. In order to verify this claim, we visualize in the second row of Figure ttg attention maps from the last layer of the image encoder. It is very clear that the model is highlighting specific regions that correspond to salient objects (e.g. heads, hands, documents, tables, signboard, etc.).
Batch Size. Contrastive learning approaches are known to benefit from larger batch sizes vary our batch size from 48 to 300 and find that performance plateaus at the top end (cf. Table 3).
40 We
Losses. We also perform ablation to assess the impact of the different losses. Based on Table 3 we note that training recognize the gaze target's ' label without explicit localization downgrades Ace@ performance by flat 3.4%. On the other hand, adding the label loss does not seem to influence localization performance. However, even if it is not reflected on the current datasets and metrics, we believe that supervising using semantic information could improve localization since the task of select- ing what a person looks at in a given context, after a field of view has been determined, is inherently a semantic problem.
Experiment Avg. D. Acc@1 t Acc@3
BS 0.109
0.405
0.601
BS = 144
0.109
0,437
0.623
BS 216
0.108
0.450
0.636
BS 300
0.108
0.447
0.642
0.107
Lrab L Liab
0.417
0.604
0.108
0.451
0.640
Lim Liah/ Lung
0.108
0.447
0.642
Np 5
0.113
0.437
0.619
N₂ 3
0.112
0.433
0.620
Number of People. We note that increasing the number of people during training progressively de- grades performance (cf. Table Given our late- fusion approach, a higher N₂ puts more pressure on the decoder to accommodate multiple gaze to- kens. Then, the image tokens need to be updated in a way that selectively aligns with all gaze tokens simultaneously at different spatial positions.
Np 0.108
0.450
0.636
bbox
0.132
0.353
0.562
head crop
0.133
0.357
0.565
bbox & head crop
0.107
0.386
0.592
Table 3: Ablation results GazeFollow for batch size, losses, number of people during training (BS = 216), and gaze encoder (BS = 72 and resolution of 448 x 448).
Gaze Encoder. To verify whether our elaborate gaze encoder is necessary, we experiment with an encoder that only processes the box coordinates, and another one that only takes the head crop with- out location. To compensate for the high resolution of head crops, we conduct this ablation with an image resolution of 448 x 448 and batch size of 72 because of hardware constraints. In the first case, the burden shifts towards the image encoder which now has to capture salient objects, and also understand people and their gaze so that querying with the head position retrieves the answer immediately. On the other hand, training with only the head crop will probably force the model to infer the head location by matching head features to image features. Based on Table we see that both versions degrade localization performance significantly.
Not Gaze Token. Quantitatively speaking, this extra token seems to improve performance, especially for the localization metrics (i.e. Avg Dist of 0.112 vs. 0.115 for models trained with N, = 3 and batch size of 216). To better understand what's 's happening, we visualize the ime - aaee cross- attention map from the last decoder block. Since the input is single person, each image token the cross-attention operation is updated based on a weighted average of the gaze token and the not guze token. The last row of Asows the weight value w of the gaze token (meaning that the weight of the learnable token is w). The red areas indicates image tokens that gave more importance to the gaze token, and vere-versa for the blue. It is clear from the samples that the decoder is using the not gaze token as placcholder for the background, or any area not being looked at by the input person, allowing for a better discrimination of gaze-relevant image regions.
Discussion
Limitation and Future Work. While our model has shown good performance in recognizing the gaze target class, the current design is not equipped to deal with the multi-label nature of the problem. Specifically, multiple objects are highlighted by the gaze heatmap. would be desirable for the model to predict them all (e.g. knife, oignon, and chopping board). Since the model is limited to

---

Figure 4: Qualitative samples from our model on images from the internet. The top row shows gaze point predictions for all people. The second row show the last attention map from the image encoder. The third row shows a predicted heatmap of single person. The last row shows the weight of the gaze tokens the last image gaze cross-altention the decoder for the same person.
predicting one gaze label embedding, we think it will likely produce some weighted average of those objects' embeddings. Furthermore, while the GazeFollow dataset is relatively small, there significantly more objects in the images than annotated gaze instances. These additional semantic pseudo-labels (from the segmentation) can also be leveraged through auxiliary losses to align spatially localized image content with their textual counterpart (a similar approach was used in (31) in order enhance recognition performance. We leave the investigation these ideas future work.
Societal Impact. Gaze following methods can bring tremendous value in many real-world applica- tions that foster positive change in society (e.g. screening neurodevelopmental disorders). However, care must be taken when deploying this technology in order to avoid privacy violations, and mit- igate risks of malfunction sensitive applications (e.g. surveillance systems). We encourage the community to use these models responsibly.
Conclusion
In conclusion, our study represents step forward in gaze target detection by extending the traditional gaze following formulation to incorporate the class label of the target. Our proposed architecture has successfully integrated semantic understanding into the task while maintaining wate-of-the-aa performance in terms of localization. To this end, we leveraged weakly supervised training regime based on pertinent pseudo-labels derived from open-world segmentation pipelines. Naturally, we employed contrastive learning objective to align the visual embedding representing the gaze target with its textual counterpart, thereby allowing for some flcxibility to build specialized vocabularies during inference based the application. We hope that our code, datasets, model checkpoints and research insights will pave the way for future research on semantic gaze following.
Ackmowledgement. This research has been supported by the A14Autism project (Digital Phenotyping of Autism Spectrum Disorders Children, grant agreement number CRSII5 202235/1) of the the Sinergia interdisciplinary program of the SNSF.
10

---

References
[1] S.O. O. Ba and J.-M. Odobez. A study on visual focus of attention recognition from head pose in meeting room. In S. Renals, S. Bengio, and J. G. Fiscus, editors, Machine Learning for Multimodal Interaction, Lecture Notes in Computer Science, pages 75-87. Springer. 2.
[2] S. O. Ba and J.-M. Odobez. Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Transactions on Pattern Analysis and Machine Intelligence. 33(1):101- 116, 2010.
[3] R. Bachmann, D. Mizrahi, A. Atanov, and A. 7amir. Multimae: Multi-modal multi-task masked autoencoders, In Computer Vision-EECCV 2022: 17th European Conference, Tel Aviv, Israel October 23-27, 2022, Proceedings, Part XXXVII, pages 348-367, Springer, 2022.
[4] C. Bai, S. Kumar, J. Leskovec, M. Metzger, F. Nunamaker Jr. and V. Subrahmanian. Predicting the visual focus attention in multi-person discussion videos. In IJCAI, pages 4504-4510, 2019.
[5] Bao, B. Liu, and J. Yu. Escnet: Gaze target detection with the understanding of 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14126-144 14135, 2022.
[6] R. Brooks and A. N. Meltzoff. The development of gaze following and relation to language. Developmental science, 8(6):535-543, 2005.
[7] Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng. Learning to detect human-object interactions. In 2018 ieee winter conference on applications of computer vision (wacv), pages 381-389. IEEE, 2018.
[8] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[9] E. Chong, N. Ruiz, Y. Wang, Zhang, A. Rozga, and J. Rehg. Connecting gaze, scene, and attention: Generalized attention estimation via joint modeling of gaze and scene saliency. 9.
[10] E. Chong, Y. Wang, N. Ruiz, and J. M. Rehg. Detecting attended visual targets in video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5396-5406, 2020.
[11] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18100:/810.0480:: 2018.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 1 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[13] N. J. Emery. The eyes have it: the neuroethology, function and evolution of social gaze. Neuroscience & biobehavioral reviews, 24(6):581 2000.
[14] L. Fan, Y. Chen, P. Wei, W. Wang, and S.-C. Zhu. Inferring shared attention in social scene videos. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6460-6468, 2018.
[15] L. Fan, W. Wang. S. Huang. X. Tang, and S.-C. Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[16] Y. Fang. J. Tang. W. Shen, W. Shen, X. Gu, L. Song. and G. Zhai. Dual attention guided gaze target detection in the wild. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11390-1111 11399, June 2021.
[17] A. Frischen, A. Bayliss, and P. Tipper. Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychological bulletin, 133(4):694, 2007.

---

[18] S. Gorga and K. Otsuka. Conversation scene analysis based on dynamic bayesian network and imsee gaze detection. In International Conference on Multimodal Interfaces and the Workshop Machine Learning for Multimodal Interaction, ICMI-MLMI '10, pages 1-8. Association for Computing Machinery. 4.
[19] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulane object detection via vision and language knowledge distillation. arXiv preprint arXtv:2104.139211 2021.
[20] A. Gupta, S. Tafasca, N. Chutisilp, and J.-M. Odobez. A unified model for gaze following and social gaze prediction. In IEEE International Conference on Automatic Face and Gesture Recognition (FG). 2024.
[21] A. Gupta, S. Tafasca, and J.-M. Odobez. A modular multimodal architecture for gaze target pre- diction: Application to privacy-sensitiee settings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5041-5050, 2022.
[22] S. Gupta and J. Malik, Visual semantic role labeling. arXiv preprint arXiv:1505.0444 2015.
[23] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[24] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05400 2018.
[25] H. Jiang. X. Ma, W. Nie, Z. Yu, Y. Zhu, and Anandkumar. Bongard-hoo hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pages 19056-19065, 2022.
[26] T. Jin, Z. Lin, S. Zhu, W. Wang, and S. Hu. Multi-person gaze-following with numerical coordinate regression. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 01-08. IEEE, 2021.
[27] T. Jin, Q. Yu, S. Zhu, Z. Lin, J. Ren, Y. Zhou, and W. Song. Depth-aware gaze-following via auxiliary networks for robotics. Engineering Applications of Artificial Intelligence, 113:104924, 2022.
[28] P. Kellnhofer, A. Recasens, S. Stent, W. Matusik, and A. Torralba. Gaze360: Physically unconstrained gaze estimation in the wild. Proceedings of the IEEE/CVF international conference computer vision, pages 6912-6921, 2019.
[29] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu,, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-1866 2020.
[30] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643; 2023.
[31] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, al. Grounded language-ing image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[32] D. Lian, Z. Yu, and S. Gao. Believe it or not, we know what you are looking at! 10.
[33] Y. Liao, S. Liu, F. Wang, Y. Chen, C. Qian, and J. Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 482 490, 2020.
[34] G. Liu, Y. Yu, K. A. Mora, and Odobez. A differential approach for gaze estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),, 4303)11092-1099, 2021.
[35] M. J. Marin-Jimenez, V. Kalogeiton, P. Medina-Suarez, and A. Zisserman. Laeo-net: revisiting people looking at each other in videos. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3477-3485, 2019.
12

---

[36] B. Massé, S. Ba, and R. Horaud. Tracking gaze and visual focus of attention of people involved in social interaction. IEEE transactions on pattern analysis and machine intelligence. 40(1 1):2711-27244 2017.
[37] M. Morales, P. Mundy, and Rojas. Following the direction of gaze and language development in 6-month-olds. Infant Behavior and Development, 21(2):373-317 --7737777, 1998.
[38] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini 1eXXi:2304.0719: 2023.
[39] K. Otsuka, J. Yamato, Y. Takemae, and H. Murase. Conversation scene analysis with dynamic bayesian network basedon visual head tracking. In 2006 IEEE International Conference on Multimedia and Expo, pages 949-952. 3.
[40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[41] A. Recasens, A. Khosla, C. Vondrick, and A. Torralba. Where they looking? In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
[42] A. Recasens", A. Khosla", C. Vondrick, and A. Torralba. Where are they looking? In Advances in Neural Information Processing Systems (NIPS), 2015. indicates equal contribution.
[43] S. Sheikhi and J.-M. Odobez. Combining dynamic head posc-gaze gaze mapping with the robot conversational state for attention recognition in human-robot interactions. Pattern Recognition Letters, 66:81-90, 2015.
[44] S. V. Shepherd. Following gaze: Frontiers in integrative neuroscience, 4:5, 2010.
behavior as a window into social cognition.
[45] R. Siegfried and J.-M. Odobez. Robust unsupervised gaze calibration using conversation and manipulation attention priors. ACM Transactions on Multimedia Computing, Communications, and Applications, 18(1):1-27, Jan. 2022.
[46] K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857-16867, 2020.
[47] R. Stiefelhagen, M. Finke, J. Yang, and A. Waibel. From gaze to focus of attention. In D. P. Huijsmans and A. W. M. Smeulders, editors, Visual Information and Information Systems, Lecture Notes in Computer Science, pages 765-772. Springer. 1.
[48] S. Tafasca, A. Gupta, and J.-M. Odobez. Childplay: new benchmark for understanding children's s gaze behaviour. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 20935-20946, 2023.
[49] S. Tafasca, A. Gupta, and J.-M. Odobez. Sharingan: A transformer architecture for multi-person gaze following. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2008-2017, 2024.
[50] Tonini, N. Dall' Asen, C. Beyan, and E. Ricci. Object-aware gaze target detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21860- 21869, 2023.
[51] S. Wang, K.-H. Yap, H. Ding, J. Wu, J. Yuan, and Y.-P. Tan. Discovering human interactions with lorge-vocabulary objects via query and multi-scale detection. In Proceedings of the IEEF//VF International Conference on Computer Vision, pages 13475-13484, 2021.
[52] Zhang, X. Zhu, Dai, M. Ye, and C. Zhu. Distribution- coordinate representation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7093-7102, 2020.
[53] B. Zhuang, Q. Wu, C. Shen, I. Reid, and A. van den Hengel. Hevrd: A benchmark for large- scale human-centered visual relationship detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
13

---

A Supplementary Material
A.1 Qualitative Evaluation
We show multiple qualitative samples on the GazeHOI dataset, both correct predictions and failure cases in Figures a respectively.
Overall, common classes like book, cellphone, bicycle are usually well localized and correctly identified. It's interesting to see that the model performs well in spite of the diversity the samples in terms of scenes, people, activities, interactions, distance from the camera, viewing angle and lighting conditions.
ST Blephont
Celiphone Pred: Cellphoon,, Hoos, Inlephonn
Aimlone Predi Aipians, findhrennt Kit
Prect Elephant, danatt,, Zobra
Predi Palc, Railing
Frost
GT. Airplono Predi Airp ano. Engine Car
ET. Book Preci Book, Coliphone Booklet
ST. Bear Prest Bear, Sall Occan
6T. Bicycle
Pred Blcycle Mottrrocc Wheal
GP Book Prect Beok, Rotiilet, Magazine
GT: Preel Book, Magneite, 3008
Beat Predi Boott Orean, Sa
GT: Boxing glove Fredi Boxing mwn Sert, Grove
Figure 5: Samples with correct predictions on GazeHOL. The bounding boxes in white represent the ground-truth, the predicted gaze point is shown in blue. We also show the 3 top predicted class labels, and the corresponding ground-trute label.
Looking at the failure cases, some are simply difficult because the object is barely visible (e.g. the bag in the second image of the last row). Even if it's correctly localized, if there is occlusion, or the object is too small, the model will naturally tend to predict the class of the closest larger object that it knows. In this case, tries to predict the animal deer, but since it's not the vocabulary, it retrieves giraffe and zebru instead. In other cases, the target label is correct when the location is off. This can happen when the image features multiple instances of the same object class, and the model localizes the wrong one (e.g. bird in second image of the third row). Finally, some failure cases simply due
14

---

Backoack Prest Person Frisbes, Book:
GT. Prest Doli Toy, Bed
GT. Collphane
6T. Prect Brush Stissors, Fabric
Fredi Car, Windotie d, Parking meter
Prodt Person, Bng. Envalope
Prodt Person, Ca mara, Car seat
Fredi Porson, Food, D ning table
ST. Slowtrrch Fred Poster, Paiithgg Paintorush
Bird Prect Bind, Ohickenn Fish
Pridt Cooddig can Fivingons
GloElock
Predt Person Fitt, Boiinnggooee
Pred: Giafff, Zebra, Cow
Pred: Box, Block Carchoard
Pred: Cup, Coffee, Soup
Figure 6: Samples with incorrect predictions on GazeHOI (iee. either the class label or the gaze point is not matching the ground-truth). The bounding boxes in white represent the ground-truth, the predicted gaze point is shown in blue. We also show the top predicted class labels, and the corresponding ground-truth label.
to the multi-label nature of the problem. For example, in the first image of the third row, the model predicts poster and painting, while the ground-truth is blowtorch. Technically both the painting and the blowtorch are in the vicinity of the person. We observe similar patterns on qualitative samples from the FozeFollow dataset in Figure
A.2 Baseline Comparisons & Limitations
Due the localization task being framed as heatmap prediction, our model's design is well- equipped to dynamically infer the correct target label. This means that it will not necessarily output the object class corresponding to the area of the highest intensity in the heatmap when the heatmap multimodal. The baseline however, pushes the class label to match the main peak of the heatmap by design. In Figure 8 (top), we illustrate this behavior using different samples where the predicted heatmap is bimodal, with the right target corresponding to the smaller peak. In this case, model
15

---

Cu
GT: Ball
GT: Ball
GT Bunk Megioni
GT. Microscope
Gl: Track
Pred: Dall Goff club Playing field
Predt Minssops Marocake Lablen
Barin
GT: Fruit
GT. Fristre
GT: Harse
GT: Cace
Prect Fruit Baraara so Broccot
Predi Horse, Horse jump, Person
Hredi Cakes Lakke serwers Candte
GT: Screon
GieTee
GT: Computer
GT. Computer
Predi Screen, Television, Window
Predi Computeer Desk, Screen
Preck Cumputter, Refi igeretir, Voting booth
GT: Skatabcard
GT: Vacuum Tolettran
Prodt Trao Wood Wood
(Predi Markeet Storctront, Stall
Bowi,
Figure 7: Example predictions on GazeFollow. Annotations are shown in white and predicted points in blue. We also provide the 3 top predicted class labels, and the corresponding ground-truth label below each image. [Toop] correct predictions, [Bottom] failures cases.
decides predict the class object corresponding to the second peak, while the baseline inevitably predicts the class of the object matching the higher peak. For example, the rightmost image shows a heatmap that mainly highlights the easel while second (and weaker) peak points to the elephant. Our model correctly predicts Elephant while the baseline incorrectly outputs Easel.
Since the baseline's recognition performance is slightly better than model, we analyzed set of samples where the two models didn't 't agree on a gaze target class. In Figure [ (bottom), we show several samples from GazeFollow where the baseline matches the ground-truth while our model is not, despite the latter producing a reasonable prediction. For example, since Ball and Balloon are both included in the vocabulary, the model might be predicting one when the ground corresponds to the other. This ambiguity is introduced by the semantic overlap in certain classes of the vocabulary (e.g. Wave and Water).
A.3 Experimental Protocol
Datasets. We use both the GazeHOI and GazeFollow [42] datasets. GazeFollow is an image- based datasct annotated with head boxes and 2D gaze points. It has around 130K annotated instances in 122K images. The test set comprises 4782 gaze instances, each of which is labeled by multiple annotators.
Metries. For GazeFollow, we report the standard distance metrics (i.e. min and avg) following previ- ous protocol [10 to assess gaze target localization, and propose Accuracy @ 1 and Accuracy @3
16

---

GT: Cake
GT: Fireworks
GT: Elephant Ours: Elephant
Ours: Cake
Ours: Fireworks
Baseline: Flower
Baseline: Tree
Baseline: Easel
GT: Ball Ours: Balloon, Baseline: Ball, balloon
GT: Rock wall Ours: Rock, rock wall Baseline: Rock wall, mountain
GT: Floor Broom, floor Baseline: Floor, broom
GT: Water Ours: Wave, water
Baseline: Water, waterfall
Figure 8: Comparison with the baseline. We show the annotated person in white, the predicted point in blue, and the predicted heatmap overlaid. [Top] cases where gaze location is incorrect, so the baseline's 's predicted class is also incorrect, but consistent with the location. However, our model's predicted label is correct. Notice the bimodal heatmap on the image with the elephant. [Bottom] Ambiguous cases where the baseline is correct, but our model is incorrect despite being semantically accurate.
for gaze target recognition. Additionally, we introduce a Multi MultiAccuracy(e @ score to account for the multiple gaze labels that can be assigned to the same instance. This metric checks the top predicted label belongs to the set of ground-truth labels. For GazeHOI, we report a gaze accuracy instead of the distance since we don't it have access ground-truth point. This metric checks whether the predicted gaze point falls within the ground-truth bounding box of the object. In terms of recognition performance, we use the same Accuracy @ and Accuracy @3 as before.
Implementation Details. Our architecture processes the input scene image at a resolution of 256 256, and the head crop at 224 224, to produce an output heatmap 64 x and label embedding of 512. The dimension d inside the decoder is set to 96. The ground-truth truth heatmap uses gaussian
placed around the gaze point. We use CLIP as the text encoder and keep frozen. We set the number of blocks in the gaze decoder to 2, and the number of people during training ResNet- pretrained on Gaze360 28), while
to N, 1. The backbone in gaze encoder the image encoder is ViT-base model 12 initialized from multimodal MAE (3] Finally, our experiments on GazeHOI are initialized from a model trained on GazeFollow.
Training. For the main experiments on GazeFollow, we use the Adam W optimizer with learning rate 2: and weight decay of 0.003. We train for 20 epochs with a warmup of epochs, and cosine annealing schedule. The batch size is set to 300, and the loss coefficients for A Atat and Aang. are set to 1000, 1, and respectively. We also use stochastic weight averaging (24 for better generalization. On GazeHOL, the only change is to remove the warmup, and reduce the learning rate
17

---

Figure 9: Comparison of the outputs of the segmentation methods. Left: RAM-Grounded SAM.
Right: Semantic-Segment Segment-Anttinng.
persor
plweshnt playarg
Lood skateboard
te
felevision
by
paperr
dOg
FLOOF
cake
food
anit
Sure ZUE
PELL
Fite
Frisbee
ce
bird
phones
kmni
tree
PIOES
b
cell
phone
Laptop
Patta hot
dog
Computer
giral
FOOD
P
Computer
pel
Siscyclll
son
sports
Mis BoNL
Bieycl
pall
ound
couple
table
aptop
auna
grass
book
surt de
P
Snoy
document
g
curde
Lapt
kite
pel
son
HOT
computer
ball
Lepaant dining tabtablee
D
bOX
UPP
skateboard
Frishee
paper Feleilis
Showboard
Figure 10: Word clouds of the vocabulary of gaze labels. Left: pseudo labels of GazeFollow. Middle:
ground-truth labels of the test set of GazeFollow. Right: ground-truth labels of GazeHOI.
to 2e - 5. All experiments are done on either a single RTX 3090 (24Gb of memory) or H100 (80Gb of memory) depending on memory requirements, and last for 2 to 10 hours each.
When training on GazeHOI, we use the center point the object's box as a proxy for the gaze target point. However, for larger objects this strategy is not optimal. For instance, the center point of person's box will around the waist area while the actual gaze point will typically be located on the head (e.g. bottom right sample of figure 2 To overcome this issue, when object boxes are larger than a threshold, we use a prediction from a model trained on GazeFollow as the ground-trute point as long it falls within the object's 's ground-trutt box, otherwise, default back to the center point. Validation. GazeFollow doesn't 't propose any validation split, we use the train/val splits proposed in 48 The best models on the validation set are selected based on the distance score for GazeFollow and the validation loss for GazeHOI.
Inference. To perform inference with our models, we first build vocabulary of classes (or use one from the datasets), pass them through the text encoder to obtain the text embeddings, and discard the text encoder. The label predictions can now be made by comparing the predicted gaze label embedding to the text embeddings and selecting the top-k based on the similarity score.
Coordinate Decoding. Vision tasks where the goal is to predict pixel coordinates on an image are typically formulated as a heatmap prediction (e.g. facial keypoints detection, pose estimation, gaze following). For computational reasons, the heatmaps are often set to a much lower resolution than the input. This introduces quantization errors when considering the arg max of the heatmap as the predicted point. In this paper, we use the DARK 52 coordinate decoding scheme, which accounts for the distribution of the heatmap when estimating the continuous coordinate values.
Extra People. Generally speaking, both datasets annotate a single person per image, even when there are more. In the experiments where we need to train with more than one person, we use an off-the-shed head detector to find other extra people. Since they don't 't have associated gaze points, we only backpropagate the loss from the person that is annotated. We use padding or truncation of people's heads and box coordinates as necessary to ensure that N, is fixed during training.
18

---

A.4 Annotation Protocols
A.4.1 Label Generation
In Figure 9
we provide comparison of the outputs of the open-world segmentation methods used to derive the pseudo-labels for GazeFollow. Please note that in case the pixel corresponding to the annotated gaze point isn't segmented by any of the two methods, we simply use the closest segmented pixel from the second approach (i.e. Seeaatti-SSemmen-AAnttiiin
In Figure I0 provide a word cloud of the vocabulary frequencies of the pseudo-labels of GazeFollow (left), the ground-truth labels of the test set of GazeFollow (middle), and the ground-truth labels of GazeHOI (right).
A.4.2 Annotation Teams
The test of GazeFollow was annotated by single person ensure consistency of class labels. For GazeHOI, the manual verification step was done by 3 people.
Experiment
Avg. D.
Min. D. J
Acc@1 t
Acc@3 f
CLIP Vision
0.148
0.087
0.341
0,485
Supervised
0.129
0.069
0.437
0.628
MAE
0,112
0.056
0,442
0,621
DinoV2
0.113
0.055
0.465
0.658
MultiMAE
0.108
0.051
0,450
0.636
Random
0.107
0.052
0.459
0.630
BERT
0.109
0.052
0.449
0.632
MPNet
0.110
0.054
0,422
0.615
CLIP Text (FT Proj)
0.108
0.052
0.439
0.633
CLIP Text (Frozen)
0.108
0.051
0.447
0.642
N
0.109
0.053
0,444
0.647
N 2
0.108
0.051
0.447
0.642
N 0.107
0.051
0.447
0.634
Table 4: Ablation results on GazeFollow for the image encoder's pretraining, the text encoder's pretraining, and the number of decoder blocks (in order).
A.5
More Ablations
Vision Pretraining. In this section, we assess the influence of the pretrained weights of the image encoder on the final performance of the model. To this end, aside from our MultiMAE weights, try CLIP's pretrained vision encoder, a ViT trained on ImageNet, standard MAE 23 and DinoV2 38
Please note that Dino uses image resolution of 224 224 and patch size of 14 instead of resolution of 256 x 256 and patch size of 16. However, the number of output tokens remains unchanged. Also, for memory restrictions, are shown in Table 4 (top row). First, we note that using CLIP's vision weights to perform both localization and recognition degrades performance significantly (i.e. 0.148 vs. 0. 0.108 Avg Dist). This serves to prove that the design of our baseline required CLIP's vision part to only perform the recognition task, not localization. Aside from CLIP, both MAE and Dino offer performance that is similar to our MultiMAE. In other words, our architecture is not very sensitive to the pretraining of the image encoder.
use batch size of 216 instead of 300. The results
Text Pretraining. We also perform an ablation to evaluate the influence of the text embeddings on the final performance (cf. Table 4 second row). For this experiment, we try random projections, BERT [II, MPNet [46], CLIP's text encoder where we fine-tune the final projection layer, and our version with the frozen CLIP. The performance for both localization and recognition seems unaffected by the type of text encoding used for the vocabulary. Surprisingly, even random projections seem to perform on par with the rest of the methods. However, random projections severely limit the generalization ability of the model as they can only work on the dataset and labels they were trained
19

---

140
120
100
PFLOPS
80
60
10
50
20
Baseline Ours
0
3
13
15
Number of People Np
Figure 11: Comparative analysis of FLOPS number of people.
To demonstrate this, perform cross-dataset evaluation GazeHOI with these random projections as class embeddings. We obtain a similar localization performance for both methods, but get Acc@1 of 26% and Ace@ 3 of 39% compared to frozen CLIP text variant which obtains 31% and 46%, respectively (these are the same values reported Table 2 of the main paper).
Gaze Decoder. The final ablation is on the number of blocks used in the transformer of the gaze decoder. From Table (third row), we can see that performance seems to optimal for N although the differences not significant.
A.6 Model Efficiency
We mentioned previously that the baseline has a larger parameter count than our proposed model. In this section, we provide FLOPS comparison as function of the number of people in the image to illustrate the efficiency and graceful scaling of our architecture's 's design. Aside from our model and baseline, we also show [TO], which popular two-stage method like ours, and (50) which is single-stage model (i.e. it simultaneously predicts all people's ' heads and gaze heatmaps at the same time, instead of using a separate head detector). To ensure fair comparison with (50), we include the cost of head detection in all other methods. Please note that we use the RGB-only variant of (50), instead of the RGB+Depth variant (i.e. the best performing one). Using latter would increase the cost even
to account for depth extraction.
50 is constant as expected, while 10] shoots up since the early fusion of the architecture means that the expensive scene encoding is repeated for each person. On the other hand, our model displays a more modest increase due to the lightweight decoder, and the scene encoding being executed only once. We also observe that the baseline surpasses model by about 20 GFLOPS due to the extra ViT (which is also ran once per image). is worth noting that even at Np 15, our model is still largely more efficient than the constant one-stage [50]
As we can see Figure IT
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the listed contributions correspond exactly what proposed in the paper, i.e. a new task, new benchmarks, a new architecture, and new evaluation protocol.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we provide separate section to discuss the limitations of our method and future research directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
The paper is applied nature and doesn't 't contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper describes the experimental protocol in details. We also plan to share the code, model checkpoints and datasett/annotations with the community for reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
22

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The experimental protocol provides sufficient information to reproduce the results. We also plan to make the code, datasets, and model checkpoints available the community.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public/ guides Caa.ddee///odeSuulmi ss ssionPolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips httpsscccccbbcccccccccccccc wwabbbic/uuiiii ides/ CoodeS ubmi ss ssssssinn ciiionooiioo icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper includes enough information reproduce the results. But will also share the code and model checkpoints.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: unfortunately, it would be computationally expensive for to report error bars in our study.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about the type of hardware used (and by extension memory), and the typical duration of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httphh/pppppp//eeree......////ap
Answer: [Yes]
Justification: We have read the code of ethics, and believe our paper conforms to the guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have a separate section for that.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our models are not high risk, so we think designing safeguards is not necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification; We use pretraining weights and datasets licensed for research use. We properly credit and cite the resources we used. The licenses for the constituent datasets can be found in their respective webpages.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papelpappssss wwwthhocee com/ cataaaaassessee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide enough details about how data was collected and annotated.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Our research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This question doesn't 't apply to our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "MtlDyiL7ut", "reviewer": "Reviewer_roY1", "summary": "The authors propose an architecture capable of predicting both the gaze target location and the semantic class of a person's gaze target in an image, representing an advancement over traditional methods that only predict the pixel coordinates of gaze fixations. They introduce new benchmark datasets and experimental protocols, leveraging tasks from gaze following and human-object interaction. The task of gaze target detection is particularly valuable for human-centric AI.", "questions": "1. Please address the concerns raised in the Weaknesses \n2.Additionally, will the proposed datasets and models be made publicly released soon?", "limitations": "The authors have indeed addressed the limitations of their work to some extent in the manuscript.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The integration of semantic analysis into gaze following is a significant step forward.\n2.The development of benchmarks provides a foundation for future research, assuming they are publicly released.", "weaknesses": "1.Compared to \"Object-aware gaze target detection\" from ICCV 2023, the improvements and extensions are limited. Although the model outputs both a gaze heatmap and its class, the paper fails to convincingly justify the necessity for detecting the object class, especially when a conventional object detection or classification model could achieve similar results after obtaining the heatmap.\n2.The category labels for the GazeFollow training set and the head crops for GazeHOI are generated using existing models. This reliance on pseudo-labels potentially limits the data's value, despite manual re-checks for the head crops. \n3.The paper primarily showcases examples from the GazeHOI dataset, with few examples and details from the GazeFollow dataset.\n4.The proposed model does not outperform the baseline in localization and is surpassed by the Baseline (heatmap weight) in recognition, as demonstrated in Tables 1 and 2. This suggests that the sophisticated model structure presented in Section 3 has limited effectiveness.\n5. The references have numerous formatting errors, such as those in [1, 46].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Limited improvement compared to Tonini et al. (2023) [46]**\n\nWe respectfully disagree with the reviewer. Here are the significant differences\n- [46] does not predict a gaze target category (L87-89), it merely uses general object detection as an auxiliary task to improve gaze target localization\n- [46] is limited to the 80 categories of COCO whereas our weakly-supervised training learns about significantly more semantic concepts (eg. 463 in GazeHOI)\n- Learning general object detection alongside gaze following (which are two mostly unrelated tasks) using the same backbone is difficult, and significantly penalizes the accuracy of both. In fact, in most cases, [46] can’t even detect people’s heads, let alone predict their gaze. We provide several samples from GazeFollow’s test set in Figure 1 (right panel) of the PDF where [46] can’t properly localize heads, even easy ones (the predicted gaze locations seems random or based on statistical priors). The GT person in these samples is not detected at all, which raises questions about what their reported performance values actually mean. In fact, we ran [46] on 200 images from the test set of GazeFollow, and found 15% of cases where the GT person’s head is not overlapping with any predicted box, and this is not including other issues.\n- The architecture design is very different: [46] uses a DETR and passes the embeddings of detected persons and objects to a standard transformer decoder. Our model follows a more principled approach by framing the task as promptable gaze following, where a scene encoder highlights gaze candidate regions and a novel decoder (recognized as interesting by reviewer **KgpK**) decodes the heatmap and gaze embedding jointly by aligning representations. The insights on scene encoding and gaze decoding are supported by evidence (L313-329 and L331-340) and make the architecture conceptually more intuitive, and what the model is learning easier to explain (not to mention the largely better localization performance).\n- Our model is significantly more efficient (cf. Figure 1, left panel in the PDF)\n- [46] proposed an incremental architecture for an already established task. **Our paper is proposing a new task, new datasets and annotations, a novel architecture achieving SOTA localization (outperforming previous SOTA by a large margin) and strong recognition, evaluation protocols and performance metrics.** The positioning of the two papers is not the same, we kindly ask the reviewer to keep this in mind.\n\n**Necessity of detecting the gaze target class**\n\nAs stated in L36-37, most gaze following applications do not particularly care about the pixel location where a person is looking, but rather the underlying semantics. For example, in autism screening, clinicians mostly look for specific gaze behavioral patterns that can not be identified without the semantic class (e.g. eye-contact or alternating between looking at a toy and the clinician). The semantics can also enable context-aware responses. For example, if a person is looking at a potentially dangerous object, the system can trigger appropriate alerts or actions.\n\nFrom a technical perspective, predicting both location and class can help disambiguate the intent, especially in scenes where multiple objects are in the field of view of the person. Also, since the area covered by the heatmap often includes multiple objects, the semantic class can help select a target based on more than just visual saliency, and refine its location.\n\nThe importance of the semantic component was also argued in [45], where authors propose a metric to assess models semantically, stating that localization metrics alone can be misleading (ie. even smaller distances can translate to different objects while larger distances can reflect the same object if it’s big enough).\n\n**Reliance on pseudo-labels potentially limits the data’s value**\n\nA) Regarding the class labels of the training set of GazeFollow, we agree that pseudo-labeling is not perfect, however, please keep in mind that: \n- Annotating a gaze class from scratch is very difficult as explained in L206-209\n- Many papers today leverage pseudo-annotation and weakly-supervised training with great success (e.g. [37])\n- The large vocabulary of pseudo-labels makes it possible for models trained on GazeFollow to acquire a rich semantic understanding, and generalize to other semantically close vocabularies. For example, we can distinguish between “man”, “woman”, and “child” instead of the generic “person” category found in most vision-based vocabularies. Also, note in Table 2 that the cross-dataset evaluation (ie. no fine-tuning) on the vocabulary of GazeHOI, shows comparable performance to the zero-shot baselines relying on the large-scale semantic pre-training of CLIP.\n\nB) As for the head boxes of GazeHOI, the statement that they are pseudo-annotations is simply not true. A model-based annotation that is verified by a human annotator is, in fact, a ground-truth (and the default approach nowadays, e.g. [28]). It is perhaps worth noting that GazeHOI initially contained about 100k images, more than 50% of which were discarded during that manual verification step. Furthermore, from our past experience with this specific case, we can confidently say that a model-assisted approach works better than human annotation because the pre-trained head detector we use is extremely accurate and tightly delimits the heads. A human annotation from scratch at this scale often results in looser and inaccurate bounding boxes as can be observed in GazeFollow’s original annotations.\n\n**Qualitative samples from the GazeFollow dataset**\n\nWe have focused on GazeHOI because it was an entirely new dataset. For the sake of completeness, we will also add an extra page of qualitative samples (annotations, good predictions, and failure cases) from GazeFollow to the final version of the paper. In the meantime, we provide many samples in Figure 2 of the attached PDF for reference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MeLFswsbxR", "reviewer": "Reviewer_DJCe", "summary": "In this work, the authors propose an enhancement to classical gaze-following prediction, which typically operates on a 2D coordinate level, by integrating semantic label prediction. To this end, they introduce a novel architecture inspired by promotable segmentation, incorporating a multi-input transformer. This architecture uniquely combines scene images and head-related attributes (bounding coordinates and head crops) through a disjoint fusion process. This allows the gaze encoder to process multiple user pairs without additional processing load, making the model uniquely applicable to a wide range of scenarios. \n\nFurthermore, using the GazeFollow dataset, the authors present a robust pseudo-annotation pipeline and introduce a new benchmark for the community's benefit.  \n\nThe paper is well-written, with a coherent narrative that provides ample evidence of the method's effectiveness through a series of ablation studies.", "questions": "A typo in line 161, where instead of constrastive, please fix it to contrastive.  \n\nDo I understand correctly, that semantic label embeddings have the form of natural language words,e.g. cup, bad, sock, etc or some other form of encoding/transformation was adopted here? It is relevant for the label loss function.  \n\nIs it planned to release the source code?", "limitations": "Those provided failure cases are indeed very relevant, therefore, I recommend additionally considering additional forms to represent data like 3d to omit those failures.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Extending the existing gaze-following task with semantic cues enhances model explainability by introducing a semantic gaze-following task. Instead of merely predicting 2D coordinates on an RGB image, the authors incorporate semantic components, significantly improving overall performance. \n\n2. The introduction of the label loss function involves calculating the cross-entropy of the cosine similarity between predicted and true visual gaze label embeddings. This label loss follows the contrastive InfoNCE formulation, a key component in self-supervised models that learn meaningful representations from negative and positive pairs. The analogy is applied here by comparing positive and negative segmentation labels. \n\n3. The combination loss includes an angular loss, which provides additional interpretability for the predictions. \n\n4. A supportive benchmark is proposed to evaluate model performance in an unbiased manner, allowing the community to build upon it. \n\n5. Empirical evaluations confirm the model's performance, demonstrating state-of-the-art results on the main GazeFollow task. The provided ablation study helps readers understand the contribution of individual components, such as the loss function.", "weaknesses": "For the proposed baseline, where the pipeline is robust, I could consider several additional data augmentation techniques, such as flipping and distorting image patches. This could further enhance the pipeline. \n\nAccording to the supplementary material, the model performs poorly in scenes with specific angles and relationships between the subject and object. To further improve overall performance, this problem should be considered in a 3D space, requiring a different dataset with fully labeled scenes. This approach would help mitigate partial occlusions by incorporating depth information. \n\nA dataset is a crucial component of this work. Therefore, I suggest including some statistical comparisons to thoroughly track the pseudo changes in the GazeFollow dataset. Currently, the approach seems to be at the level of omitting synonyms, among other methods. However, I assume this was not entirely feasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the valuable suggestions. Here are our comments:\n\n**Data augmentation for the baseline**\n\nWe agree with the reviewer. The baseline variants that are fine-tuned already use data augmentation techniques like flipping and color jittering.\n\n**Use of depth information to model the 3D scene**\n\nThis is a great idea. In fact, the earlier versions of our architecture used depth information (extracted from the RGB image) to reconstruct the 3D scene point cloud and infer a 3D visual field of the person from a predicted 3D gaze direction. This field map was used to filter out image areas where the person could not be looking (in 3D) given their head and eye direction. As expected, this addition improved localization performance. Ultimately, we decided to drop this component for 2 reasons: \n1. Using depth to improve performance is already well documented in the literature [5, 16, 20, 26, 45, 46]\n2. Between the new task, the datasets, the evaluation protocols, the baselines, and the new architecture, our paper contains numerous contributions as it is, and we deemed it preferable to keep the focus on the topic of semantic gaze following without adding unnecessary complexities.\n\n**More statistics on GazeFollow’s pseudo-labels and cleaning steps**\n\nWe appreciate the suggestion. We will expand the section on GazeFollow’s pseudo-annotations to provide more details in the final version of the paper. The released annotations will also include the original pseudo-labels from both segmentation methods used in the paper in case other researchers want to process or combine them differently.\n\n**Clarification on semantic label embeddings**\n\nThe semantic label embedding of the gaze target (or simply gaze label embedding) is a vector of size 512 that captures semantic information about the visual area the person is looking at. This embedding is trained to match the text embedding (produced by the text encoder) of the ground-truth class via a contrastive loss. At test time, we use a vocabulary of classes (e.g. person, tree, table) that is converted into class embeddings by the text encoder. Then, given a predicted gaze label embedding, we find the closest class embedding (using cosine similarity) to get the predicted object class (or gaze label) from the vocabulary.\n\n**Release of source code**\n\nOf course! The source code, datasets and model checkpoints will be made publicly available upon acceptance. Our hope is that this will attract more researchers to work on this topic, and encourage the community to build upon our work.\n\n**Typo**\n\nThank you for flagging the typo, we have corrected it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4onJTK2VX", "reviewer": "Reviewer_9bSU", "summary": "This paper introduces a novel approach to semantic gaze target detection, extending the traditional gaze following task to include not just localization of where a person is looking, but also identification of what they are looking at. The authors address the limitation of existing gaze following methods that only predict pixel coordinates, proposing to also identify the semantic label of the gaze target. They introduce an end-to-end architecture that simultaneously predicts both the location and class label of the gaze target, where the recognition part is framed as a visual-text alignment task. They also create new benchmark datasets through pseudo-annotation of existing datasets. The proposed method demonstrated superior performance, especially in the localization accuracy of the gaze target, surpassing existing methods.", "questions": "- Please clearly explain the advantages of simultaneously estimating semantics (whether the improvement in accuracy truly stems from this problem setting).\n- If there are any experimental evidence on computational efficiency within the paper or appendix, please indicate them.", "limitations": "It appears that a reasonable argument is presented in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is written clearly overall, and the motivation is also evident.\n- The method design is also clear, and it is interesting that this approach improves localization accuracy.", "weaknesses": "- While interesting, it is not entirely clear why training semantic label prediction simultaneously improves localization accuracy, and this point requires further detailed discussion. Whether it is appropriate to simply interpret that the simultaneous training approach has achieved state-of-the-art performance still needs to be investigated.\n- The computational efficiency benefits of simultaneous estimation are understandable, but there are few experimental details reported on this point. The abstract mentions \"with 40% less computation,\" but it was unclear how this number was derived.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**More experimental evidence on computational efficiency. How did we get the 40% decrease in parameters?**\n\nIn terms of parameter count, our model features 116M while the baseline has 200M. This is because our model only uses an MLP head (3.3M params) for label prediction while the baseline uses a ViT encoder (86.6M params). That’s a decrease of 42% in the total number of parameters. \n\nTo be more thorough, we also provide a FLOPS comparison as a function of the number of people in the image to illustrate the efficiency and graceful scaling of our architecture’s design (cf. Figure 1 left panel of the PDF). Aside from our model and baseline, we also show Chong et al. [10] which is a popular two-stage method like ours, and Tonini et al. [46] which is a single-stage model (ie. it simultaneously predicts all people’s heads and gaze heatmaps at the same time, instead of using a separate head detector). To ensure a fair comparison with [46], we include the cost of head detection in all other methods. Please note that we use the RGB-only variant of [46], instead of the RGB+Depth variant (i.e. the best performing one). Using the latter would increase the cost of [46] even more to account for depth extraction.\n\nAs we can see from the graph, [46] is constant as expected, while [10] shoots up since the early fusion of the architecture means that the expensive scene encoding is repeated for each person. On the other hand, our model displays a more modest increase due to the lightweight decoder and the scene encoding being executed only once. We also observe that the baseline surpasses our model by about 20 GFLOPS due to the extra ViT (which is also ran once per image). It is worth noting that even at 15 people, our model is still largely more efficient than the constant one-stage [46].\n\nWe will add a section specifically dedicated to the efficiency dimension to the final version of the paper.\n\n**Motivation for joint detection of heatmap and object class. Does joint training improve localization?**\n\nWe refer the reviewer to our detailed reply to this comment in the *overall rebuttal* section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6PaJsIrqu2", "reviewer": "Reviewer_kgqK", "summary": "This paper proposes simultaneously detecting a person's gaze location and recognizing the class label of the gaze target.  To complete the task, a new architecture is designed. The architecture is efficient when multiple people are present in the image because it processes the scene image once and detects each person's gaze by decoding the correlations between their head information and the scene image tokens. To evaluate the method, this paper annotates the target label in the existing GazeFollow dataset and collects GazeHOI based on several existing datasets. On benchmark GazeFollow and GazeHOI, the proposed architecture performs better in localizing the gaze target than the existing gaze-following method.", "questions": "Please refer to the weakness.", "limitations": "The authors have adequately addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  This paper simultaneously detects the location and semantic label of a person's gaze target in a scene image. The task is meaningful and has been explored rarely. \n2.  Benchmark is collected with both gaze target location and semantic label. \n3.  The component of gaze decoder is interesting. The proposed method is efficient when multiple people are present in the image because it processes the scene image once and detects each person's gaze by decoding the correlations between their head information and the scene image tokens.  It avoids the computation burden in previous methods that merge the scene images and head images before the image encoder.  \n4. The proposed method outperforms other method in localizing the gaze target.", "weaknesses": "1. According to the experimental results (Tab 1&2), the proposed method does not show advantages in recognizing the label of gaze target rather than the baselines. The baselines also show equivalent performance in localizing the gaze target. Therefore, the baseline (heatmap) is shown to be the best solution in simutaneously detecting the location and semantic label of a person's gaze target. \n\n2. The introducing of L_lab in the proposed method is not necessary. First, adding L_lab does not improves the ability the gaze target localization experimentally (Tab. 3). Second, L_lab mechanism enpowers the architecture to recognize the semantic label of gaze target, but it could be designed in a simpler and more effective way, as shown by the baselines in Tables 1 and 2.\n\n3. Details of the gaze decoder (the third and fourth paragraph in Sec 3.3) is not presented clearly. It could be clearer if the intermediate x_gaze and x_img are marked in Fig. 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Necessity of $L_{lab}$. It doesn’t improve localization, and the semantics can be predicted in a simpler and more effective way by the baseline**\n\nThe $L_{lab}$ loss is necessary for joint training (cf. our reply in the *overall rebuttal* on the motivation for joint training). It is not meant to improve localization. However, even if it is not reflected on the current datasets/metrics, we believe it could improve localization since the task of selecting what a person looks at in a given context after a field of view has been determined, is inherently a semantic problem. \n\nRegarding the comparison, we fail to see how adding and fine-tuning an entire ViT (86.6M params) is “simpler” than an extra MLP head (3.3M params). Please check our reply to reviewer **9bSU** about efficiency. As for effectiveness, we refer the reviewer to our answer on the comparison with the baseline to better interpret the numbers (cf. *overall rebuttal*).\n\n**More explanation of the decoder**\n\nWe wish to thank the reviewer for recognizing the value of our decoder’s design. We will take steps to improve the description and annotate the main figure with $x_{gaze}$ and $x_{img}$ for better readability. Below, we further explain paragraphs 3 and 4 of section 3.3 as requested. \n\nAt first, $x_{img}$ (which represents image tokens) and $x_{gaze}$ (which represents gaze tokens, one per person) go through a set of transformer blocks composed of a series of cross-attention and feed-forward operations combined with residual connections. These cross-attention operations go in two ways, one where the gaze tokens $x_{gaze}$ generate the queries, while the image tokens $x_{img}$ generate the keys and values (we call this $x_{gaze}$ → $x_{img}$ or gaze to image cross-attention), and one where image tokens generate the queries and gaze tokens generate the keys and values (we call this $x_{img}$ → $x_{gaze}$ or image to gaze cross-attention). The two-way design allows image tokens to incorporate information from gaze tokens, and vice-versa. This helps them better align for the final dot-product operation that will produce the final heatmap.\n\nOn Figure 1 of the paper, $x_{img}$ flows through the decoder following the blue arrows until the *Upscaler* module, after which it becomes $\\hat{x}_{img}$ (L138-139), which is simply a spatially upscaled version of the image feature map representation with a lower number of channels. \n\nOn the other hand $x_{gaze}$ flows through the decoder by taking the path denoted by the orange arrows until the two output MLP heads. The bottom MLP predicts the gaze label embedding (one per person), while the top one produces $\\hat{x}_{gaze}$ (L136-137). \n\nTo obtain the final heatmap, we simply perform a dot-product between $\\hat{x}_{img}$ \n\nand $\\hat{x}_{gaze}$ (L141-144).\n\nFinally, the *not gaze token* is simply an extra learnable token, much like the CLS token in standard transformers. However, unlike the CLS token which aggregates information due to the final supervision, our extra token is not supervised and plays a different role. Conceptually, it serves to improve the image to gaze cross-attention. Without this token, this operation will update each image token with a weighted sum of the value embeddings derived from gaze tokens. But what if nobody is looking at the area represented by that image patch? We need to allow the model to flexibly choose a different “background” token in that weighted sum, instead of forcing only information from those people’s gaze tokens to update that image patch. The *not gaze token* plays the role of this “background” placeholder, a hypothesis that we verify in Figure 4 (last row) and L331-340.\n\n**Comparison with the baseline**\n\nPlease check our detailed reply to this comment in the *overall rebuttal* section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our gratitude to the reviewers for their thoughtful feedback. We address below the common concerns, and will incorporate the discussion in our final version.

As a reminder, the goal of this paper is to establish the foundation for the novel, significant and challenging task of semantic gaze following by introducing benchmarks, evaluation protocols, and model architectures achieving SOTA for gaze following. 

**roY1, 9bSU    Motivation for joint detection of heatmap and object class. Why not use a separate object detector? Does joint training improve localization?**

We have motivated the joint training in L38-54 using efficiency (which **9bSU** acknowledged) as our main argument. Given the importance of this question, perhaps we should elaborate more:
- First, combining localization and recognition is the more natural formulation. Otherwise, why not also decouple object detection into object localization and image classification with two networks?
- There is no object detector or classifier that can recognize all classes of our annotations. Plus, we also have to deal with uncountable objects (ie. Stuff classes).
- Joint training means doing both tasks using the same backbone, which is preferable when they are tightly related like in our case. Decoupling means adding an entire separate network instead of our MLP head, which is unnecessarily inefficient.
- Unlike object detection, in gaze following we predict heatmaps and not boxes, which makes applying a separate object detector afterwards more challenging than **roY1** implies. How do you match a heatmap to the right box? Should you consider the argmax of the heatmap as the location of the object and match it with the boxes? What if the point falls within multiple boxes? Or instead, should you consider the energy of the heatmap contained in each box? How do you deal with the different scales of overlapping boxes then? Also, what if the heatmap is multimodal, how do you make your selection? We would need to craft rules and heuristics tailored to each use-case, and will almost inevitably be suboptimal. **Joint training allows the model to learn the best way to dynamically make sense of that heatmap in order to infer the right class.** For example, we found several instances where the predicted gaze label is correct despite the location being incorrect. This can happen when the heatmap is multimodal and the model decides to select the class of the second peak instead. In this case, using a separate model to match the predicted argmax location will surely lead to the wrong class (cf. Figure 3 left panel of the PDF).

Finally, we achieve SOTA in gaze following localization performance due to our novel architecture (promptable gaze following formulation and decoder design). We do not claim in our paper that joint training improves localization. In fact, it doesn’t, at least not on our datasets. Instead, we argue that it’s difficult to do the extended task any other way that is as efficient, equally performant, and more natural. That being said, joint training definitely improves recognition (+3.4% flat points on Acc@1, cf. Table 3).

**roY1, kgqK    Comparison with the baseline. Localization is the same as the baseline, and recognition is slightly worse**

There seems to be a misunderstanding. The baseline uses a two-step process: first, a frozen gaze model predicts a gaze heatmap; then, a second network performs class recognition using the original image fused with the heatmap.

In our experiments, **we use our own novel gaze model to generate the heatmap for the baseline, so the localization performance is the same by design** (L234-237). This allows us to control for the localization factor, enabling an unbiased comparison of recognition performance (L242-244).

As for recognition accuracy, while the heatmap variant is slightly better, it is important to consider that the design of our baselines is unfair to our model:
- All baselines use our own model’s localization, which is the current SOTA by a fair margin (L261-262). Thus, the baseline literally needs our model to achieve that recognition accuracy. To verify this, we swap our gaze model with the one from [10] and find that Acc@1 and Acc@3 drop from 0.466 and 0.653 to 0.442 and 0.620 respectively on GazeFollow. Under this setting, the baseline becomes worse than our model.
- The baseline uses an extra CLIP encoder instead of our MLP head, making it computationally more expensive (L262). The heatmap variant is also fine-tuned separately, which means extra training costs. **Attempting to use the same CLIP vision encoder to do both localization and recognition significantly degrades performance** (cf. Table 4, first row)
- The baseline benefits from the large-scale (400M samples) semantic pre-training of CLIP (L262-263), whereas our recognition head is trained from scratch only on 100K samples (L275-277)
- The baseline (esp. the heatmap variant) is novel, and provides interesting insights. It was designed from scratch, and should be considered a research contribution on its own. This was recognized by reviewer **DJCe. In hindsight, maybe the term “baseline” was not the right one to use here**
- As explained in L265-268, an error analysis revealed that many cases where our method and the heatmap baseline didn’t agree were ambiguous, either due to the hierarchy or semantic similarity of the predicted and GT classes. We show a few samples in Figure 3 (right panel) of the PDF

**roY1, DJCe    Release of source code and datasets**

As stated in the checklist, we confirm that the source code, datasets and model checkpoints will be made publicly available upon acceptance. We are eager to see how the community will build upon our work.

**roY1   Reference formatting**

Thank you, they have been corrected.

We hope that our arguments provide a compelling case, and kindly ask all reviewers to contextualize the numbers and consider all contributions when assessing the merits of our paper.

Meta-review: The paper introduces a two stage technique to perform both gaze localization and classification.  It also introduces a benchmark dataset.  It receives 4 reviews, including one accept, two borderline accepts , and one borderline reject .  Overall, reviewers like this work . They believe adding semantic labels to gaze following improves its explainability, and that  using pseudo labels and self-supervision loss reduce data dependence and improves its generalization.   The main concerns include marginal technical novelty and marginal performance improvement, computational complexity of the method , and  missing some  technical details .  The authors rebuttal addresses some of the concerns.
Decision: Accept (poster)