MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction
Authors: Anshul Gupta
Samy Tafasca
Arya Farkhondeh
Pierre Vuillecard
Jean-marc Odobez
Venue: NeurIPS 2024 poster (2024)
Keywords: gaze following
social gaze prediction
multi-task learning
Primary area: machine_vision
TL;DR: We propose a new architecture and dataset for jointly modelling gaze following and social gaze prediction

Abstract: Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference as an ad-hoc post-processing of the gaze following task. Furthermore, the vast majority of gaze following approaches have proposed models that can handle only one person at a time and are static, therefore failing to take advantage of social interactions and temporal dynamics. In this paper, we address these limitations and introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. It comprises (i) a temporal, transformer-based architecture that, in addition to frame tokens, handles person-specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, built from multiple gaze following and social gaze datasets by extending and validating head detections and tracks, and unifying annotation types. We demonstrate that our model can address and benefit from training on all tasks jointly, achieving state-of-the-art results for multi-person gaze following and social gaze prediction. Our annotations and code will be made publicly available.

A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction
MTGS:
Anshul Gupta Samy Tafasca Arya Farkhondeh Pierre Vuillecard Jean-Mare Odobez Idiap Research Institute, Martigny, Switzerland École Polytechnique Fédérale de Lausanne. Switzerland fagupta, stafasca, af afarkhondeh, pvuillecard, odobez))idiay
Figure I: Results of our multi-person person and temporal transformer architecture for joint gaze following and social gaze prediction, namely Looking at Humans (LAH), Looking at Each Other (LAEO), and Shared Attention (SA). For each person, the social gaze predictions are listed with the associated person ID (e.g. in frame 1, person 2 is in SA with person 4). More qualitative results can be found in the supplementary G
Abstract
Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference Furthermore, the vast majority of gaze following approaches have proposed models that can handle only one person at time and static, therefore failing take advantage of social interactions and temporal dynamics. this paper, we address these limitations and introduce novel framework jointly predict the gaze target and social gaze label for all people in the scene. It comprises (i) a temporal, transformer-based architecture that, in addition to frame tokens, handles person- specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, built from multiple gaze following and social gaze datasets by extending and validating head detections and tracks, and unifying annotation types. We demonstrate that our model can address and benefit from training on all tasks jointly, achieving state-of-the-an results for multi-person gaze following and social gaze prediction. Our annotations and code will be made publicly available.
ad-hoc pros-processing of the gaze following task.
Introduction
Social interaction plays pivotal role in our daily lives and is influenced by an array behavioral elements, encompassing not only verbal communication but also non-verbal cues like gestures (53) or body language. In particular, the ability decode people's gaze, including communicative behaviors like eye contact and shared attention with or learn from others (44). In contrast, the absence or impairment these skills is often indicative of developmental disorders such autism (42). Thus, designing social gaze prediction algorithms
particular object, is highly related to our capacity to connect
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and systems has attracted considerable attention from different communities, ranging from medical diagnosis to human-rober interactions (42) 43 36).
In this work, we investigate whether we can build unified framework to infer from video data the gaze target and social gaze label in one stage for all people in the scene. This requires: (i) A new
architecture capable of jointly modeling these tasks, (ii) A large-scale dataset with annotations for all the tasks. Specifically, we focus on the LAH, LAFO and SA social gaze tasks (illustrated in Fig. D.
Methods for social gaze prediction in the literature adopt one of two approaches. The first one focuses on the design of dedicated networks to process pairs of head crops and potentially other scene information (31 (30 55 12 26). While their specialization makes them effective, they offer little room for generalization to other gaze-related tasks. The second one first address the gaze following tasks (40), defined as predicting the 2D location people's 's gaze targets, and then uses the predicted gaze points to infer social gaze through ad-hoc post- post-processing schemes. For instance, combining gaze following heatmaps from multiple people to predict shared attention (9).
However, gaze following itself is a challenging task. Besides geometric aspects, the task requires understanding and establishing a correspondence between top-down information related to the person's state, activity, cognitive intent, and bottom-up saliency related to the scene context (salient items like objects or talking people). Furthermore, gaze following performance, measured by distance, does not always translate to similar social semantic performance (e.g. when evaluating if the predicted gaze point falls on a person's head not (48))
Motivation. Existing methods for gaze following suffer from several drawbacks. First, most of the methods perform prediction for single person (40) 41 9 14
48) requiring multiple inference passes on the same image to process multiple people in the scene. In contrast, a multi-person gaze following architecture processes the image only once and has to capture salient items for all people in the scene, while maintaining the ability to infer the gaze target of each individual. This is inherently more complex and challenging. Another drawback of the single-perse pegle-person formulation is that it does not explicitly model people's s interactions, thereby preventing the possibility of jointly inferring people's gaze target and social gaze attributes. Secondly, the majority of proposed models for gaze following are static, using only a single image at time. This is partly due to the absence of large and diverse video datasets, and the difficulty of leveraging large-scale static ones like GazeFollow (41). This is a limitation, as temporal information can capture head and gaze coordination patterns 443 which can help gaze direction inference, especially when the eyes are not completely visible
Finally, none of these methods have investigated learning the gaze following and social gaze prediction tasks jointly. Thus, it remains a research question whether such formulation can improve performance by having social cues inform the gaze following task and vice-versa, or if performance would degrade as we try to accommodate multiple tasks, datasets, and people within the same framework.
Contributions. Given these motivations, we propose a new, unified framework for gaze following and social gaze prediction with the following contributions:
A novel temporal and multi-person architecture for gaze following and social gaze prediction (Sec.B). Our approach posits people as specific tokens that can interact with each other and the scene content (i.e. frame tokens). This based for the modeling of (i) temporal information at multiple levels (from 2D gaze direction to 2D gaze target level). (ii) the joint prediction of the gaze target and social gaze label.
representation allows
A new dataset, VSGaze, that unifies annotation types across multiple gaze following and social gaze datasets (Sec. I.I.Ip.
New social gaze protocols and metrics for better evaluating semantic gaze following perfor- mance (Sec. 4.3)
Results. In our experiments (Sec. 5) we show that our architecture achieves sfate-of-the-arr results for multi-person gaze following, while also performing competitively against single-person models. is also able to leverage our proposed VSGaze dataset to jointly tackle gaze following and social gaze prediction, achieving competitive performance compared to methods trained individual tasks. In particular, our experiments further demonstrate that the performance benefits from this joint prediction, i.e. adding the social loss. improves gaze following performance, and vice-versa. Finally, the new social gaze metrics provide complementary information to the standard distance- based metrics, helping assessing model performance from the social interaction perspective.
is worth noting that our architecture is easily extendable and allows for the integration of auxiliary -rrson-specffic information that can influence the final predictions. In the supplementary [ explore this aspect by integrating people's 's speaking status in the person tokens to improve the results.

---

2
Related work
Gaze Following. Typical methods for this task exploit a two-branch architecture: one for processing the scene and the other for processing the person of interest (40) 9 14
17
22 23 26). They have
17 23), pose (I7), and objects (20), or by potentially leveraging scene geometry (19 23 (4). However, only few
distinguished themselves by the addition of other relevant modalities like depth (14
efforts have addressed the multi-person case. (22) first proposed a simple architecture relying on a scene backbone to get person-agnostic image representation that is subsequently fused with the head crop features of each individual obtained using another backbone. While this reduces computation, the model does not account for person interactions, as each head is processed separately. In another architecture to perform multi-person gaze following.
direction, (52) 5 rely on a transformer-basee Their methods borrow from DETR (6), taking the image as input and simultaneously predicting the head bounding box and gaze target for every person in the scene. While these methods can implicitly model person interactions, an important limitation is that they compute performance on detected heads which are matched to the ground truth. Given that both the head detection and matching steps are prroo, precludes comparing their results to others. We provide examples in the supplementary (Fig. (8) where (51) makes incorrect head detections.
Temporal gaze estimation. Temporal information has proven effective for 3D gaze estimation. Previous research developed models to learn from various inputs, including face, eyes, and facial landmarks using multi-streant recurrent CNN (77; : eyes and visual stimuli or raw RGB frames from in-the-wild settings using convolutional RNNs or LSTMs (38) (25;: and the temporal coordination of gaze, head, and body orientations using LSTMs (34) However, the use of such methods for the gaze following task in arbitrary scenes has been underexplored. The only exceptions are (9) who introduced a convolutional L.STM block at the bottleneck of the heatmap prediction architecture, and (32) who leveraged temporal attention over aggregated frame-level features. However, both approaches only showed a slight improvement compared to their static versions, highlighting the challenge of exploiting temporal information for this task. Conceptually, the methods did not model 2D gaze direction dynamics, and can not be extended for multi-person gaze inference.
Social gaze prediction. Several research papers in the literature are dedicated to the study of looking at each other (LAFO) and shared attention (SA) tasks. For LAFO, most methods rely on processing the head crops to obtain some gaze directional information, and then combining it with 2D inferred 3D geometric information to predict the LABO label (31 30; 5). Drawbacks include processing pairs of persons independently and, as they only process heads and do not address gaze following, lacking global image context and not extending easily to other social tasks like SA. Similarly to (52 51) a recent paper (15) proposed encoder-decoder transformer architecture to predict heads and the LAEO labels, and while achieving good results, suffers from the same drawbacks.
Regarding shared attention, the first method to address it in the wild was (12), which framed the problem as tasks: the binary classification of whether SA occurs in a frame, and the inference of the location of the SA target object. Their method combined predicted 2D gaze cones of people in the scene with a heatmap of object region proposals, while others (46) directly inferred SA from the raw image. Since then, several methods leveraged combining gaze following heatmap predictions of all people (9) (52) and improved performance. Nevertheless, the above task formulation (12) used by all papers suffers from two main issues: (i) cannot distinguish between multiple SA instances if they occur in the same frame; (ii) it does not determine which specific people are sharing attention. Our work solves both problems by framing the task binary classification between pairs of people. This formulation is more natural and has the benefit of extending to other social gaze tasks.
Finally, none of the previous works performed both social gaze prediction tasks. There are three notable and interesting exceptions. First, (13) who addressed the inference of gaze communication activities (atomic and events, including LAEO and SA) using a graph-based approach with 12 dimensional tokens. However, as it predicts single gaze 'state" for each person, it is problematic as it does not allow for simultaneous LAEO and SA. It also does not identify the other person involved in the social gaze interaction. The second is (7) who addressed dyadic communication by proposing a gaze following style 2-branch architecture processing order-dependent pairs of people. However, inference using 7 is inefficient because the model needs N₂ (N- forward passes to consider all pairwise relationships for given scene with N, people. Also both methods do not address the gaze
following task. More recently, (16) did address all tasks using a graph approach, but there was no temporal modeling nor joint training on all tasks.
3

---

Ding
Parceer
Phorgte
Layarr
CPPT
Frame Toxeno
Tokensjb
Coss
Pecple
In-Out
Inbout
Person
Takensio
Glst
Vasiee
Interaction
Block
I
LAH
alie
Person Module
Interaction Module
Prediction Module
Kean Riouus
Figure 2: Proposed architecture for multi-person temporal gaze following and social gaze prediction.
See approach overview Section
3 Architecture
As motivated Section designing a temporal architecture for gaze following is challenging given the lack of large scale video datasets. Since the broader scene tends to remain relatively static for short temporal window, we focused on modeling person-level temporal information to simplify the learning problem. In particular, transformers, with a temporal transformer for processing the person tokens. At the same time, facilitates interactions between the scene and person tokens via cross-attention.
architecture processes person and scene tokens through separate
Approach overview. Our approach is illustrated in Fig. 2 It takes as input a sequence of t= 1...7 frames, well as the head bounding box tracks ,,177 and corresponding head crops h 1, which assumed to have been extracted for each of the 1... Np persons. The outputs are the sequence of gaze heatmaps A and in-out gaze labels 1, for each person i, as well as the sequence of per-frame pair-wise social gaze labels for each task and pair i, E {1... Np):: e.-j.. for LAH, e.++.11:2 for LABO, and C..j.::7 for SA.
cmg
The model proceeds as follows. First, each frame f is processed by standard VIT tokenizer to produce the set patch-wise frame tokens fr, resulting a sequence of frame tokens fir parallel, the Person Module processes the sequence of head crops from each person using the Temporal Gaze Processor, and the resulting sequence outputs are then tokenized at each frame along with the bounding box locations to produce the sequence of person token Pi.1:7. Secondly, the Interaction Module jointly processes the frame and person tokens, iteratively updating them at each time step through person-scene cross-attention interaction components and scene ViT self-attention, and in time through person spatio-temporal social interaction components. Finally, the Prediction Module processes each time step the resulting frame and person tokens (from multiple blocks) infer the sequence of gaze heatmaps and in-out gaze labels for each person, as well as pair-wise social gaze labels. We detail the three modules in the next sections.
3.1 Person Module
This module aims model person-specific information relating to gaze and head location.
Temporal Gaze Processor. It aims to capture all gaze-related information (direction, dynamics).
First, individual head crops crop are processed by a Gaze Backbone Gsta to produce gaze embeddings according to stat = Ssm (ha
(h Then, to model the gaze dynamics of a person, we rely on a Temporal Gaze Encoder Gremp to process the sequence g of gaze embeddings plus learnable temporal position embeddings x1.7 and obtain their temporal counterparts: g., Gremp + X1:T). Giemp is implemented as single Transformer layer with self-attention. Finally, supervise the learning of relevant gaze embeddings, we attach a Gaze Vector Decoder that predicts person's 2D gaze vector at each time step. g., = Gvs jemp), where Guse implemented as a 2-layer MLP.
Person tokenization. The person tokens are obtained by projecting the temporal gaze embeddings and normalized 4d head box locations using learnable linear layers (Pgaee and Prox respectively) to tokens of same dimension than frame token, and adding them together:

---

Pt. = P gaze (..z)(B..t)) (g. + Pbox (bop)) ).
3.2 Interaction Module
The Interaction module aims modeling the exchange of information between persons and the scene at each time step, as well as the spatio-temporal social interactions between people. One important goal of this process is align the person and frame token representations so that (i) person-specific gaze heatmaps can be predicted from the set of output frame tokens and each person output token; (ii) in-out gaze and social gaze prediction can be made from the person tokens.
To do so, we designed the module to consist of B blocks, each comprising Person-Scene Interaction and Spatio-Temporal Social Interaction components. The input to the first block is the set of person tokens P1.N..1:TT from the Person Module, and the frame tokens fir,. Each block then processes the set of outpur" person tokens 0.b- P1: and output frame tokens 1:7" from the previous block, and returns updated tokens after a series of self/cross-attention layers through the components.
Person-Scene Interaction. This component models the interactions between people and the scene and can capture inferring gaze scene objects or body parts like hands exploit some global context. It is inspired by ViT-Adaptor (8) which has shown good performance for dense prediction tasks when relying on pretrained models and small amounts of data for the target task. It proceeds in 3 steps:
(i) People-to-Scemt Encoder Ip. it updates the frame tokens with person information relevant to gaze by processing the frame tokens f.h-1 and frame-level person tokens pob according to t.th (f.h)) ...... It is implemented as a single Transformer layer with cross-attention, where ro generate the queries and Pi.N,.. generate the keys and values.
(ii) The updated frame tokens fath pass through the standard set of of ViT layers Vo process the scene information, resulting the output frame tokens for the block b: rea = (ff*))
(iii) Scene-to-Peopee People Encoder Ztp it updates the person tokens that they capture location informa- tion related to the salient items they are probably looking at. It works by processing the frame-level 0.b-1 person tokens PI:N,.. and obtained frame tokens ro.b according to: Pi:Ng.t = Ib PI:N,... ro.b). It is
abll also implemented as a single Transformer layer with cross-attention, where the set Pi.N.. generates the queries and r. generates the keys and values.
Spatio-temporal Social Interaction. This component allows the sharing of information between people and of the alignment of their representations for social gaze prediction. This also include modeling the temporal evolution of individual tokens. To achieve this, a Social Encoder 7hp first s.b processes and updates the frame-level person tokens Pi:N,.t to capture interactions between people each frame, according to: pi.NN.. - ( (Di.NN...) It is followed by Temporal Person Encoder Zbi that processes the updated person token sequences p of each person and updates them to capture temporal patterns of attention, resulting in the output person tokens for the block: P = Ih (pfh. Both Zo and Za are implemented as a single Transformer layer with self-attention. pp
3.3 Prediction Module
ob The Prediction Module processes the set of output person (PI:N,.1.7) 1.)) and frame (f tokens from all Interaction Module blocks to predict the person-specific gaze heatmaps and in-out labels, as well as the pair-wise social gaze labels.
Gaze Heatmap Prediction. Here, we follow the model introduced in (49) which takes inspiration from the DPT decoder (39) for dense prediction tasks, and adapts it to handle multiple heatmap predictions from the same ViT outputs. This is performed by conditioning the decoding on each person's 's token. The standard DPT decodes the image features from multiple layers of a VIT in a Feature Pyramid Network (27) style. It works by fusing at block level the feature maps from level b +1 after an upsampling stage, and the feature maps computed by reassemble stage from the VIT output of block b. We aim to apply this approach to the frame tokens (frth 1 B).
Note that the superscript o, p and do not represent indices, but intermediate token updates within block b.
5

---

but conditioned on a specific person. In our model, this is achieved through a modification in the reassemble stage, in which the image feature maps produced by the standard reassemble stage are multiplied every location (using a Hadamard product) with the projected person token Pi, of that same block level. The gaze heatmap Ai, for each person at each frame is thus obtained as:
o.b
A D(((F,)) .P..). b B))
(2)
where D denotes this conditional DPT. See (49) and supplementary H for details.
Social Gaze Prediction. This decoder processes the person tokens from all B Interaction Module blocks to predict the social gaze label for every pair of people in every frame. In practice, the B tokens single person in frame are linearly projected and concatenated to
{P .... Pit u.B corresoondngg produce multi-scale person token P Then, to predict social gaze label, pairs of these tokens are concatenated and processed by the decoders for LAH and for SA (illustrated through the Pairwise Instance Generator in Fig. 2 Their outputs are the predicted LAH score e,++j. for person i looking at j. and the predicted SA score Ct,j, for j.
ji = E(p,, pj.1) and Cij.t C(P ms P,..)
E' and are implemented as 3-layer MLPs with residual connections. For LAEO, both people j.j need to be looking at each other for positive label, and either one can be looking away for negative label. Hence, we simply compute the LAEO score ji as min(e,-jj,t= @)-i,)).
In-Out Prediction. This decoder O processes the multi-scale person tokens p,, to predict at every frame whether people are looking inside the frame or outside the frame, O Op where O implemented as a 5-layer MLP with residual connections.
3.4 Losses
The total loss a linear combination of the gaze heatmap loss LHM, gaze vector loss Lvec, social gaze losses LLAH, LSA and the in-out loss L1o:
L = AHMCHM + AVECCVEC + ALAHCLAH + ASACSA + AιoCio
C is applied at each time step per person for LHM: Lvec: Lio, and per pair for LAAH LSA. All losses are standard: Λим is defined as the pixel-wise MSE loss between the GT and predicted heatmaps, Lvec the cosine loss, and the social gaze and in-out losses as binary cross-yntropy losses. Since LAEO is inferred from LAH predictions (Sec 3.3. we do not have any LAEO loss.
4 Experiments
4.1 Datasets
We perform experiments on multiple gaze following and social gaze datasets.
GazeFollow (41) is large-scale static dataset for gaze following, featuring 122K images. Most images are annotated for single person with their head bounding box and gaze target point. The test set contains gaze point annotations by multiple annotators. Despite lower quality images and annotations, given its rich diversity, it remains a good dataset to use for pre-training.
(VAT) (9) is video dataset, annotated with head bounding boxes, gaze points, and inside vs outside frame gaze for a subset the people in the scene. It contains 1331 clips collected from 50 shows YouTube.
ChildPlay (48) is a recent video dataset for gaze following. annotated with head bounding boxes, gaze points, and label indicating 7 overlaaplg gaze classes including inside frame, outside frame and gaze shifts. It contains 401 clips from 95 YouTube videos, and features children playing and interacting with other children and adults.
VideoCoAtt (12) is a video dataset for shared attention estimation, containing 380 videos or 492k frames from TV shows. When shared attention behavior occurs (i.e. about 140k frames), the relevant frames are annotated with the bounding box of the target object, as well as the head bounding boxes of the people involved.
UCO-LAEO (11) a video dataset for LAEO estimation, annotated with head bounding boxes, and a label indicating whether two heads are LAEO. It contains 22,398 frames from 4 TV shows.

---

Datasel
Gaze Points
LAH
LABO
SA
GazeFollow
41
118k
27k/493k
VAT
109k
74k/729K
13/461k
16k/94k
ChildPlay
48
217k
59k/682k
7k/35Ik
4k/S5k
VideoCoAI 12
367k
2906/1551K
U
40019188
ECO-LAEO 31
21k
21k/36k
10k/54K
VSGaze
714k
444K/2998k
30k/866k
420l//067k
Table Person-wise gaze point and pair-wise social gaze annotation (positive/negative) statistics for our datasets. VSGaze unifies annotation types across VAT, ChildPlay, VideoCoatt and UCO-LAEO.
Two other interesting datasets with annotations for multiple social gaze behaviours are VACA- TION (I3) and Gt-static
(7). However, VACATION does not provide annotations for all social gaze
behaviours when they occur simultaneously (examples in supplementary Fig. 77. This annotation
scheme is problematic and is linked to their method design as described Sec. On the other hand,
GP-static only considers dyadic interactions and ís not publicly available.
4.1.1 VSGaze dataset
limitation of the above datasets is that they only contain annotations for gaze following or specific social gaze tasks. Hence, we propose the Video dataset with Social gaze and Gaze following annotations or VSGaze dataset. VSGaze extends head track annotations and unifies annotation types across VAT, ChildPLay, VideoCoAtt and UCO-LAEO. This allows for joint training of gaze following and social gaze, and provides new tasks and metrics for evaluating performance on the component datasets. The construction of VSGaze is described below.
Extending Head Track Annotations. As each dataset contains annotations for only a subset of people the scene, we detect all missing heads using the pre-trained Yolov5 head detection model (24) used by Tafasca el al. (48), and track them using the ByteTrrack algorithm (54). We further manually verified the accuracy of the obtained tracks. This step is vital obtain positive and negative social gaze pairs. Consider a scene with 3 people. i, k where i is looking at j. If only i is annotated, the positive LAH pair i + j would be missed. The negative LAH pair - k would also be missed.
Unifying Gaze Following and Social Gaze Annotations. Given the extended set of head bounding box annotations, well as existing gaze following and social gaze annotations, we then process these amnotations to obtain gaze following and social gaze labels across all the datasets. The gaze target for UCO-LAEO and VideoCoAtt set as the center of the LAEO and SA target bounding box respectively, LAH pairs are obtained by checking if the gaze target falls inside another person's 's head box. For LABO we check the reverse as well. SA pairs are obtained by checking if the gaze targets for both people fall in the same head box. We provide the detailed protocol in the supplementary B Annotation statistics are summarized in Table T Overall, VideoCoatt is the largest source of annotations except for LAEO. We also see that the pair-wise annotations are skewed towards negative cases. The statistics further provide insight into the content of the datasets. As VAT, VideoCoAu and UCO-LAEO contain clips from TV shows, there are many more instances of looking at other people and at each other. On the other hand for ChildPlay. LAH mainly occurs when the supervising adult looks at a child and there is limited LAEO.
4.2 Training and Validation
We follow standard practice 9 14 [7) by first training the static version of our model (i.e. with no temporal attention Geemp- Ihe on GazeFollow. It is trained for 20 epochs with a learning of rate of 1 x 10-4, 4, The resulting weights then serve as initialization for our proposed temporal model. We freeze the VIT V and train the temporal model on VSGaze for another 20 epochs with a learning rate of 10-6 8 For validation, we use the provided splits for UCO-LAEO. VideoCoAtt and ChildPlay, and the splits proposed by Tafasca et al. (48) for GazeFollow and VAT. For all our experiments, we use = 5 frames with a temporal stride of 3. To allow for batch training, we randomly sample up to N, 4 people in scene (padding in case there are less) During testing, we evaluate per sample and consider all people. The Interaction Module consists of = blocks.. interacting with at layers
{2, 5, 8, 11}. Additional implementation details provided in the supplementary
7

---

Method
PP
Dist. 4.
AP1o
FlAAn
FlAARO 1
APsa T
Chong
0.121
0.918
0.778
0.562
0.288
Chonge ?
0.130
0.956
0.764
0.529
0,331
Gupta" 1+
0.119
0.784
0.590
0.335
Ours-moGi moGi
0,738
0.579
0,515
0.111
0.598
0.339
Ours
0.107
0.940
0,795
0.590
0.576
Ours: PP
0.107
0,940
0.812
0.603
0.352
Table 2: Comparison against gaze following methods on VSGaze. All models were trained on
VSGaze. PP indicates social gaze predictions from post-processing gaze following outputs (v) vs predictions from decoders (X). Best results are bold, second best results are underlined.
4.3 Evaluation
Gaze Following, We use the standard metrics:
AUC: for GazeFollow, the predicted heatmap is compared against a binary GT map with value at annotated gaze point positions, to compute the area under the RÔC curve.
Distance (Dist.): the arg of the heatmap provides the gaze point. We can then compute the 1.2 distance between the predicted and GT gaze point on a 1 x square. For GazeFollow, compute Minimum (Min.) and Average (Avg.) distance against all annotations.
In-Out AP (AP it is the Average Precision (AP) of the In-Out gaze prediction scores.
Social Gaze. We propose an evaluation protocol for LAH, well as new pair-wise evaluation protocol for SA as motivated in Sec.
Social gaze decoders: For the LAH task we compute an Fl score. A sample is individual person i, and at inference, it is assigned the person j for which the ez-jj,t is the largest. Hence, for GT positive case, the prediction will considered
true positive if matches the GT target AND ey+j. e is above 0.5. Otherwise, the prediction is false negative. Similarly, a GT negative is true negative if e. is below 0.5, otherwise it is a false positive. For LAEO, as with LAH, we compute an F1 score. A sample is a pair of people j, and for person i, we consider Ozeaj,r with the highest
score. We then set Gje+jæ to 0 Vj * before computing the performance. We also the reverse for j. i. For SA, we compute standard AP score by considering sample pair of people, and thresholding predicted scores at different values to compute the area under the Precision- Recall curve. Gaze following methods: Social gaze predictions are obtained by post-processing the predicted gaze points. For LAH, we check whether the predicted gaze point for a person falls inside the target person's 's head box. For LAEO we check the reverse as well. We compute Fl scores for both. For SA, we check if the distance between two people's 's predicted gaze points is within a set of thresholds and compute an AP score.
5 Results
5.1
Comparison against the State-of-The Art
We compare against recent SoTA methods addressing either social gaze tasks or gaze following. In addition, for fairness and to evaluate the impact of the VSGaze dataset, we also re- re-trained on this dataset the static image based models of Chong (9) (Chongs *) and Gupta (I7) (Gupta*), as well as the temporal model of (9) (Chongr*), the only temporal gaze following model with available code.
VSGaze. The results on VSGaze are given in Table 2 Note that regarding our approach, for social gaze, we compute the scores by leveraging either the predictions from the respective task decoders (Ours), or by post-processing the gaze following outputs of our model (Ours-PP).
Compared to the baselines, we observe that our model achieves the best performance for all tasks except for in-out gaze prediction. In particular, we achieve sigmificant gains in the distance and APsA metrics when leveraging the predictions from the SA decoder. The latter highlights the importance of modeling SA as a classification task compared to post-processing gaze following outputs, which struggles to capture whether the gaze points for a pair of people falls on the same semantic item.
In addition, we note that better gaze following performance does not always translate to better social gaze performance. For instance, although Chongs has a better distance score compared to Chongy performs worse for shared attention. This effect even more pronounced on ChildPlay (Supp/C). and suggests the benefit of considering social gaze metrics for better characterizing the performance

---

Method
Multi
AUC
Avg.Dist.
Min.Dist.
Method
Multi
Dist.
AF1o 1
Fang [4)
x
0.922
0.124
0.067
Fang 14
X
0.108
0.896
Tomani 550
x
0.927
0,141
Tonini (50)
X
0.129
Jin 26
x 0.920
0.118
0.063
Jin
0,897
41
0.928
0.122
Bаo 41
X 0.120
0.669
Huu 21
X
0.923
0,128
0.069
Tafasca
0.936
Talasca
48
X
0,109
0.834
Chong
(
X
0.921
0.137
0.077
Chong
X
0.134
0.853
Gupta
X
0.933
0.134
0.071
Gapta
0.134
0.864
Jin 22
0.919
0.126
0.076
Jin 2
0.134
0.880
Ours static
0.929
0.116
0.059
0,869
Ours
0.105
0.869
(a) Results on GazeFollow (41
(b) Results on VAT (9).
Method
Multi
Dist.
APto
Method
Dist.
APIARO
Tafasca 33
X
0.107
0.986
Doesti [I]
Jiminez 33]
0.762
0.795
Gupta
X
0.113
Jiminez 50
0.867
Ours
0.117
0.994
Ourst
0.113
0.993
Ourst
0.019
0.974
(c) Results on ChildPlay (48).
(d) Results on UCO-LAEO (31).
Table 3: Comparison against task specific methods fine-tuned on individual datasets. Best multi- person results are in bold, overall best results are underlined. Multi indicates multi-person (V) vs single-person (X) gaze following methods. Ours initialized from training on GazeFollow, while Ourst is initialized from training on VSGaze.
of gaze following models, especially its semantic performance. In the supplementary C we provide breakdown of performance on each of the component datasets VSGaze
State-of-the-art comparison: fine tuning individual datasets. Table compares our model against task specific methods. For GazeFollow, we use our static model (Ours-static) that was trained on GazeFollow and used to initialize model trained VSGaze. For the video datasets, as SoTA methods were trained (or finetuned) on individual datasets, for fairness we also fine-tune our model on these datasets, investigating two initialization alternatives: either from the model trained on GazeFollow (Ours), or from the model trained on VSGaze (Ourst). Note that we are unable to compare against previous results for VideoCoatt due to new pair-wise evaluation protocol that better captures SA performance (Sec 4.3).
On GazeFollow and VAT, our model outperforms the only other comparable multi-person gaze following model of Jin (22). It also achieves competitive or better results to single-person methods, even those leveraging auxiliary modalities such as depth (14 50 23 21 48). Importantly, on the social LAEO task, we set the new state of the art on UCO-LAEO, far outperforming methods
designed specifically for LAEO 31 30
We also note that fine-tuning using the VSGaze model initialization can improve results compared the standard protocol of fine-tuning after training on GazeFollow (ex. distance on ChildPlay and APLAEO on UCO-LAEO). This suggests that training on VSGaze can leverage the complementary knowledge provided by the different tasks and datasets, which follows observations made in other works addressing multi-task training (IO).
5.2 Analysis
Impact of Architecture. Comparing the performance of our model with no social gaze losses (Ours-noSoc) against the basclines (Table 2 we see that it already performs on par or better than them while being much more efficient as it processes the image only once for all people in the scene. also serves as a strong gaze following baseline to compare performance against.
Impact of Social Gaze Loss. Our architecture can further benefit from the social gaze losses, showing improved gaze following performance and social gaze prediction (Ours and Ours-PP, Table 2 In particular, observe significant gains for the SA compared to Ours-noSoc. Interestingly, the addition of the social gaze losses also better aligns the gaze following outputs for social gaze prediction. Comparing Ours-PP and Oursano,ocial, we that performance for all social gaze tasks is improved. Impact of Gaze Following Loss. We additionally train our model without the standard gaze following losses: heatmap, gaze vector and in-out (Ours-noGF, Table 2). Across VSGaze, we see that

---

performance for all social gaze tasks drops, which indicates that the gaze following and social gaze losses provide complementary information, and using both can give improved performance.
Impact of VSGaze. When comparing the performance of the models trained on VSGaze (Supp Table against their versions fine-tuned on individual datasets (Table we see that the fine-tuned models always perform better. This is because fine-tuning allows the models to learn dataset specific priors (ex. more LAH cases in VAT, Table IT For instance, on VAT, Gupta* has a distance score of 0.138 when trained on VSGaze, compared to score of 0.134 when directly fine-tuned on VAT. Also, our model has a distance score of 0.112 when trained on VSGaze, and score of 0.105 when fine-tuned on VAT. This highlights the challenge in leveraging multiple datasets: while we may expect better performance by having more data, the different priors and statistics bring additional difficulties. Nevertheless, our model trained on VSGaze is able to achieve strong performance across all datasets.
Impact of temporal information. Comparing the static and temporal versions of our model trained on VSGaze, we observe improvements in performance for shared attention (0.555 vs 0.576, Tab. 5 supplementary), and similar or slightly improved performance for other metrics. This is in contrast with Chongr, which often has lower performance than Chongs (for distance, LAH and LAEO metries in Table These results follow observations from prior work (Section ( and highlight the challenge in leveraging temporal information for gaze following, We provide a detailed analysis in supplementary D
The supplementary (overview in provides additional ablations and discussions.
6 Conclusion
We propose a new framework for multi-person, temporal gaze following and social gaze prediction comprising of novel architecture and dataset. Through series of experiments, we show that model can effectively learn from a mix of video-based datasets with different statistics, to perform gaze following and social gaze prediction without sacrificing performance on any of them. The trained model can then be further fine-tuned on individual datasets to improve performance towards a specific scenario or task.
We hope that our proposed framework opens new directions for modeling people's 's gaze behavior, and are keen to see applications of the proposed dataset and social gaze losses in other methods. In the future, we intend to further investigate the benefits of temporal information and other auxiliary signals, including new ways of incorporating them into the architecture. We also plan to expand the VSGaze dataset to include more samples and annotations.
Acknowledgement. This research has been supported by the AI4Autism project (Digital Phenotyping of Autism Spectrum Disorders in Children, grant agreement number CRSII5 202235/1) of the the Sinergia interdisciplinary program of the SNSF.
References
[1] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. (2021)
[2] Association., A.P.: Diagnostic and statistical manual of mental disorders (5th ed.) (2013), https://doi... http:////oo....//..../ 117.//ppp.. books. 9780890425596
[3] Bachmann, R., Mizrahi, D., Atanov, A., Zamir, A.: Multimae: Multi-modal multi-tast masked autoencoders. In: Computer
2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXVII. pp. 348-367. Springer (2022)
[4] Bao, J., Liu, B., Yu, J.: Escnet: Gaze target detection with the understanding of 3d scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14126-14135 (2022)
[5] Cantarini, G., Tomenotti, F.F., Noceti, N., Odone, F.: Hhp-net: A light heteroscedastic neural network for head pose estimation with uncertainty (2021)
[6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko. S.: End-to-end object detection with transformers. In: Computer ViCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. pp. 213-221 Springer (2020)
10

---

[7] Chang, F., Zeng, J., Liu, Q. Shan, S.: Gaze pattern recognition in dyadic communication. In: Proceedings of the 2023 Symposium on Eye Tracking Research and Applications. pp. 1-7 (2023)
[8] Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., Qiao, Y.: Vision transformer adapter for dense predictions. In: The Eleventh International Conference on Learning Representations (2022)
[9] Chong, E., Wang. Y.. Ruiz, N., Rehg. J.M.: Detecting attended visual targets in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5396-5406 (2020)
[10] Ci. Y. Wang, Y., Chen, M.. Tang, S., Bai, L., Zhu, F., Zhao, R., Yu, F., Qi, D., Ouyang, W.: Unihep: unified model for human-centric perceptions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17840-17852 (2023)
[11] Doosti, B., Chen, C.H., Vemulapalli, R., Jia, X., Zhu, Y., Green, B.: Boosting image-based mutual gaze detection using pseudo 3d gaze. Proceedings of the AAAI Conference on Artificial Intelligence 35(2), 1 1273-1281 (May 2021). https://doi.org/100 /00. 1609/aaai vtt.aa.aiiv352..1166 16215, https //ojs. aaai org/ / index.ndxxxiie php/ phpp:AAI//////pppp http///arrceeeeeee12222122
[12] Fan, L., Chen, Y., Wei, P., Wang, W., Zhu, S.C.: Inferring shared attention in social scene videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1660-6466 (2018)
[13] Fan, L., Wang, W., Huang, S., Tang, X., Zhu, S.C.: Understanding human gaze communication by spatio-temporal graph reasoning. In: Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) (October 2019)
[14] Fang, Y., Tang, J., Shen, W., Shen, W., Gu, X., Song, L., Zhai, G.: Dual attention guided gaze target detection in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11390-11399 (June 2021)
[15] Guo, H., Hu, Z., Liu, J.: Mgtr: End-to-end mutual gaze detection with transformer. In: Proceed- ings of the Asian Conference Computer Vision. pp. 1590-16600 1605 (2022)
[16] Gupta, A., Tafasca, S., Chutisilp, N., Odobez, J.M.: unified model for gaze following and social gaze prediction. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG) (2024)
[17] Gupta, A., Tafasca, S., Odobez, J.M.: A modular multimodal architecture for gaze target predic- tion: Application to privacy-sensitive settings. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 5041-5050 (2022)
[18] Gupta, A., Vuillecard, P., Farkhondeh, A., Odobez, J.M.: Exploring the zero-shot capabilities of vision-language models for improving gaze following. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 615-624 (2024)
[19] Hu, 7.,, Yang, D., Cheng, S., Zhou, I.., Wu, S., Liu, J.: We know where they are looking at from the rgb-d camera: Gaze following in 3d. IEEE Transactions on Instrumentation and Measurement (2022)
[20] Hu, 7., Zhao, K., Zhou, B., Guo, H., Wu, S., Yang, Y., Liu, Gaze target estimation inspired by interactive attention. IEEE Transactions on Circuits and Systems for Video Technology 32(12), 8524 8536 (2022)
[21] Hu, Z., Zhao, K., Zhou, B., Guo, H., Wu, S., Yang, Y., Liu, J.: Gaze target estimation inspired by interactive attention. IEEE Transactions Circuits and Systems for Video Technology 32(12), 3524-8536 (2022)
[22] Jin, T., Lin, Z., Zhu, S., Wang, W., Hu, S.: Multi-person goll-following with numerical coordinate regression. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). pp. 01-08. IEEE (2021)
11

---

[23] Jin, T., Yu, Q. Zhu, S., Lin, Z., Ren, J., Zhou, Y., Song, W.: Depth-aware mollowolg via auxiliary networks for robotics. Engineering Applications of Artificial Intelligence 113, 104924 (2022)
[24] Jocher, G., Chaurasia, A., Stoken, A., Borovec, J., NanoCode012, Kwon, Y., TaoXie, Michael, K., Fang, J., imyhxy, Lorna, Wong, C., Yifu, Z., V, A., Montes, D., Wang. Z., Fati, C., Nadar, J., Laughing, UnglvKitDe, tkianai, yxNONG, Skalski, P., Hogan, A., Stro- bel, M., Jain, M., Mammana, I.,, xylieong: ultralytiesfollov. v6.2 YOLOv5 Classifi- cation Models, Apple M1, Reproducibility, ClearMI. and Deci.ai integrations (Aug 2022). https://doi.or//10..3 /ttp//00o.////eooooooooo www.enondd..00005900 https://doi. org/ w281/22eodo..ooo 7002879
[25] Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., Torralba, A.: Gaze360: Physically unconstrained gaze estimation in the wild. In: IEEE International Conference Computer Vision (ICCV) (October 2019)
[26] Lian, D., Yu, Z. Gao, S.: Believe or not, we know what you are looking at! In: Asian Conference Computer Vision. pp. 35-50. Springer (2018)
[27] Lin, T.Y., Dollár, P., Girshick. R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings the IEEE conference on computer vision and pattern recognition. pp. 2117-2125 (2017)
[28] Loshchilov, I., Hutter, F. Decoupled weight decay regularization. In: International Conference on Learning Representations (2018)
[29] Maenner, M., et al.: Prevalence and characteristics of autism spectrum disorder among children aged 8 years autism and developmental disabilities monitoring network, 11 sites, united states, 2020. Morbidity and mortality weekly report. CDC surveillance summaries/ / Centers for Disease Control 72(2) (2023)
[30] Marín-Jiménez, M.J., Kalogeiton, V., Medina-Suárez, P., Zisserman, A.: LAEO-Net+H revisiting people Looking At Each Other in videos. IEEE Transactions on Pattern Analysis and wttp:///w/...11.//P.......
Machine Intelligence (2021).
[31] Marin-Jimenez, M.J., Kalogeiton, V., Medina-Suarezz P., Zisserman, A.: Laco-net: Revisiting people looking at each other in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
[32] Miao, Q., Hoai, M., Samaras, D.: Patch-level gaze distribution prediction for gaze following. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 880-889 (2023)
[33] Min, K., Roy, S., Tripathi, S., Guha, T., Majumdar, S.: graphs for active speaker detection. In: European Conference on Computer Vision. pp. 371-387, Springer (2022)
Learning long-tern spatial- apatial-temporal
[34] Nonaka, S., Nobuhara, S., Nishino, K.: Dynamic 3d gaze from afar: Deep gaze estimation from temporal eye-head-bor body coordination. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2192-2201 (June 2022)
[35] Otsuka, K., Takemae, Y., Yamato, J., Murase, H.: A probabilistic inference of multiparty- conversation structure based on markov-switching models of gaze patterns, head directions, and utterances. In: Inter. Conf. on Multimodal Interfaces. pp. 191-198 (2005)
[36] Otsuka, K., Kasuga, K., Kohler, M.: Estimating visual focus of attention in multiparty meetings using deep convolutional neural networks. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction. pp. 191-199 (2018)
[37] Palmero, C., Selva, J., Bagheri, M.A., Escalera, S.: Recurrent CNN for 3d gaze estimation using appearance and shape cues. In: British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK. September 3-6, 2018, p. 251. BMVA Press (2018)
[38] Park, S., Aksan, F.,, Zhang, X., Hilliges, O.: Towards end-to-end video-based eye-tracking. In: European Conference on Computer Vision (ECCV) (2020)
12

---

[39] Ranftl, R., Bochkovskiy, A., Koltun, V: Vision transformers for dense prediction. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12179-12188 (2021)
[40] Recasens", A., Khosla", A., Vondrick, C., Torralba, A.: Where are they looking? In: Advances in Neural Information Processing Systems (NIPS) (2015), indicates equal contribution
[41] Recasens, A., Vondrick, C.. Khosla, A., Torralba, A.: Following gaze in video. In: Proceedings of the IEEE International Conference Computer Vision. pp. 1435-1444 1443 (2017)
[42] Senju, A., Johnson, M.H.: Atypical eye contact in autism: models, mechanisms and develop- ment. Neuroscience & Biobehavioral Reviews 33(8), 1204-1214 (2009)
[43] Sheikhi, S., Odobez, J.M.: Combining dynamic head pose-gaze mapping with the robot conversational state for attention recognition human-robot interactions. Pattern Recognition Letters 66, 81-90 (2015)
[44] Shepherd, S.V.: Following gaze: saze-follooing behavior window into social cognition. Frontiers in integrative neuroscience 4, 5 (2010)
[45] Stiefelhagen, R., Yang, J., Waibel, A.: Modeling focus of attention for meet-
ing indexing based multiple cues. IEEE Trans. Neural Networks 13(4), 928-938 (2002). https://doi.org/10.. 109/TNN. 1021893, https:://dii.rgg//// 11tp://....111///111111 1109/TNN..TT 2002
1021893
[46] Sumer, O., Gerjets, P., Trautwein, U., Kasneci, E.: Attention flow: End-to-end joint attention estimation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3327-3333 (2020)
[47] Tafasca, S., Gupta, A., Kojovic, N., Gelsomini, M., Maillart, T., Papandrea, M., Schaer, M., Odobez, J.M.: The ai4autism project: A multimodal and interdisciplinary approach to autism diagnosis and stratification. In: Companion Publication of the 25th International Conference on Multimodal Interaction. pp. 414-425 (2023)
[48] Tafasca, S., Gupta, A., Odobez, J.M.: Childplay: new benchmark for understanding children's gaze behaviour. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20935-20946 (2023)
[49] Tafasca, S., Gupta, A., Odobez, J.M.: Sharingan: A transformer architecture for multi-person gaze following. In: Int. Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[50] Tonini, F., Beyan, C., Ricci, E.: Multimodal across domains gaze target detection. In: Proceed- ings of the 2022 International Conference on Multimodal Interaction. pp. 420-431 (2022)
[51] Tonini, F., Dall' Asen, N., Beyan, C., Ricci, E.: Object-aware gaze target detection. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 21860-21869 (October 2023)
[52] Tu, D., Min. with transformers. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2192-2200. IEEE (2022)
X., Duan, H., Guo, G., Zhai, G., Shen, W.: End-to-end human-gaze-targee detection
[53] Vuillecard, P., Farkhondeh, A., Villamizar, M., Odobez, J.M.: Cedb-hg: Novel annotations and gaze-aware representations for head gesture recognition. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG) (2024)
[54] Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: Multi-object tracking by associating every detection box (2022)
13

---

A Supplementary Material
The supplementary material is organized as follows:
details the construction of VSGaze;
breakdown of performance on VSGaze by component dataset;
ablations on temporal window length and novel architecture components;
implementation details;
experiments on incorporating auxiliary information (speaking status);
qualitative analysis of predictions;
details on our DPT based gaze heatmap decoder;
discussion on limitations;
discussion on broader impact.
Unless specified, all experiments and visualizations in the supplementary use LAH and LAEO predictions obtained via the post-processing strategy, and SA predictions from the corresponding decoder. This is because post- post-processing gaze following outputs for LAH and LAEO ensures that outputs for these three tasks align. We also observe slightly better performance for LAH and LAEO using the post-processing approach compared to using the predictions from their respective decoders (Table 2 Ours-PP vs Ours).
B VSGaze Construction Details
We provide the detailed protocol for obtaining gaze following and social gaze annotations on VSGaze: Gaze Target Point. For people sharing attention VideoCoAu (12), we compute their gaze points as the center of the SA object's ' bounding box. Similarly, for a person pair LAEO in UCO-LAEO (3I). we compute their gaze points as the center of the other person's head bounding box.
LAH. We generate LAH annotations for all datasets. To do so, similarly to Tafasca et al. (48), we check whether the gaze point for an annotated person falls inside any other person's head bounding box. For the GazeFollow test set, at least 2 of the annotated gaze points should fall inside another person's head bounding box.
LAEO. We use the provided annotations for UCO-LAEO. For VAT and ChildPlay, we generate LAEO annotations by using the LAH annotations, checking whether the LAH target for a pair of people corresponds to the other person. We cannot obtain LAEO for GazeFollow as most images are annotated for single person, and for VideoCoAtt because if a person is SA target, they are not in the set of people sharing attention and their gaze is not annotated.
SA. We use the provided annotations for VideoCoAtt. For VAT and ChildPlay, generate novel SA annotations from the LAH annotations, checking whether two person share their attention to the same third person. We cannot obtain SA for GazeFollow as most images annotated for a single person, and for UCO-LAEO as a pair of people annotated with LAEO cannot be sharing attention.
C Breakdown of Results on VSGaze
We provide breakdown of results VSGaze by component dataset in Table 4 Note that following the results in the main paper, LAH and LAFO results for Ours-noGF and Ours are obtained from their respective decoders. We can observe that performance trends individual datasets can differ from the aggregated results on VSGaze. For instance, although we have a small improvement for LAH compared the baselines across VSGaze, we perform significantly better on ChildPlay. In general, as VideoCoAtt represents the highest number of samples in VSGaze, it also has the highest impact.
Also on ChildPlay, once again we see that better gaze following performance does not translate to better social gaze performance. Although Gupta has better distance score compared to Chongs performs significantly worse for all social gaze tasks.
D Ablations
D.I Temporal Window Length
We compare performance of our model for different temporal window lengths on VSGaze in Table Note that T corresponds to static model. We observe that incorporating temporal information
14

---

Dataset
Method
PP
Dist.
AP1o
FlAAH
FlAAOO
Al's
Chongs
0.132
0.798
0.785
0.486
0.288
VAT W
Chongy
0.137
0.843
0.783
0.479
0.332
Gupta*
0.138
0.766
0.518
0.300
Ours-noCF noGF
0.766
0.503
0.435
Ours
0.112
0.845
0.791
0.526
0.521
Ours:
0.112
0.845
0.825
0.548
0.497
Chongs"
0.123
0.973
0.597
0.470
0.154
ChildPlay
Chongr*
0.137
0.165
Gupta 17
0.119
0.979
0.571
0,428
0.132
Ours noG
0.609
0.404
0,207
Ours noo
0.118
0.994
0.620
0.412
0.188
Ours
0.113
0.993
0.682
0.426
0.179
Ours-PP
0.113
0.993
0.651
0.436
0.216
Chongs
0.120
0.793
0.290
VideoCoAt
1p
Chong
0.126
0.790
0.337
Gupta
0.1
0.815
0. 0.347
Ours-noGF
0.733
0.524
Ours-noSoc
0.107
0.822
0.335
0.106
Ours-PP
0.106
0.825
0.345
Chonga
0.031
0.986
UCO-LABO 31
Chongy 9
0,064
0.941
0.774
Gupla*
Ours-noGF
0.989
0.939
Ours-noSoc
0,043
0.978
0.840
Ours
0.027
0.990
0.888
Ours-PP
0.027
0.994
0.870
Chongs 0
0.121
0.918
0.778
0,562
0.288
VSGaze
Chong
0,130
0.956
0.764
0.529
0.331
Gupta* 1b
0.119
0.929
0.784
0.590
0.335
Ours-noGF
0.738
0.579
0.515
Ours-noSoc
0.111
0.945
0.802
0.339
Ours
0.107
0.940
0.795
0.590
0.576
0.107
Table 4: Comparison against gaze following methods on VSGaze and its component datasets: VAT (9).
ChildPlay (48), VideoCoAtt (I2) and UCO-LAEO (3I). All models were trained on VSGaze. PP indicates social gaze predictions from post-processing gaze following outputs (V) predictions from decoders (x), Best results are in bold, second best results are underlined.
can improve performance, especially in the case of shared attention. For the other metrics performance remains comparable. As a temporal window of 9 does not necessarily give better performance than temporal window of use T for our experiments.
We note that these observations are in contrast to those from the static (Chongs) and temporal models (Chongr) of (.). As seen in Table 2 Chongr often performs worse than Chongs, with especially lower scores for the distance and LÁEO metrics. This follows prior observations from the state of the art regarding temporal modelling for gaze following (Section 2 and illustrates the challenge in leveraging temporal information.
While architecture design may be reason for the lack of greater improvement in performance, the data itself is an important factor. Firstly, despite the larger number of samples in VSGaze compared to standard video based gaze datasets, there is high redundancy between frames so data diversity not comparable to that of GazeFollow. Secondly, the moments where temporal information is important, such as during gaze shifts, only form small percentage of total instances. Hence, improvements for predictions these moments not reflected in overall metrics. For instance, gaze shifts form less than 10% total instances in ChildPlay (48). However, in our qualitative analysis (Section G) we can situations where temporal information helps. In future work we plan to investigate new metrics for evaluating the performance temporal models.
D.2 Architecture Components
We systematically remove different novel components of our architecture to analyse their impact and provide results T
Interaction Module. Removing the Person-to-Scene Interaction encoder IB (Ours-nolps) has the
15

---

T
Dist. 4
APIo
FhAA
Fharro
Al'sA
0.108
0.946
0.806
0.599
0.555
0.107
0.940
0.812
0.603
0.576
0.106
0.811
Table 5: Ablations for different temporal window lengths T on VSGaze. Best results are in bold, second best results are underlined.
Method
4
AP1o t
Fl t
FlaABo
APsa
pur
0.108
0.937
0.806
0.612
0.547
Ours-tolep
0.581
Ours
0.112
0.939
0.799
0.605
0,538
Ours-noDPT
Ours
0.107
0.940
0.822
0.603
0.576
Table 6: Ablations on different novel components of our architecture on VSGaze. ppp refers to the Temporl Social Interaction component, Isp refers to the Scene-to-Person encoder, Ips refers to the Person-to-Sen -o-onne encoder and DPT refers to the gaze heatmap decoder. Best results are in bold, second best results are underlined.
largest impact on performance, especially for distance, LAH and SA. Without this encoder, the frame tokens cannot access the person tokens, so encoding their gaze relevant salient items is much harder. Removing the Seene-to-Pesson Interaction encoder Itg (Ours-nol,p) decreases LAEO and SA performance. Without this encoder, the person tokens cannot access the frame tokens, so they cannot capture the locations of gazed salient items. As the frame tokens may be able adapt to this change, gaze following performance is not impacted negatively. Finally, removing the Spatio- Temporrl Social Interaction component Th Th (Ours-nolppe) decreases LAH and SA performance. Without this component, there is no interaction between person tokens, so identification of social dynamics is hindered. Interestingly,
see boost in LAEO performance. However, a major portion of LAEO positives come from the UCO-LAEO dataset (see Table D which consists mainly of two person scenes, so capturing social interactions may be less important.
DPT Decoder. We replace our proposed modified DPT decoder for gaze heatmap prediction (Section H with simpler decoder (Our)-noDPT). This decoder projects the frame and person tokens from the last Interaction block, performs a dot product between them, and then upscales the output to the heatmap resolution. Using the simpler decoder results in drops in performance for the distance, LAEO and SA metrics. Unlike the DPT, it lacks multi-scale representations which impacts heatmap prediction and supervision of tokens.
Overall, we observe that removing the components tends to impact shared attention performance. This is similar the observation in the previous section regarding temporal information. Unlike LAH and LAEO where the target is always another person, SA is more challenging as the shared attention target can be any person or object/point. Hence, this task may be benefitting more from additional information architecture components.
E
Implementation Details
The Gaze Backbone Ssm is ResNet18 pre-trained on Gaze360 (25), and processes the head crops at resolution of 224 x 224. We use a VīT-base model (I V initialized with MultiMAE weights (3) to process the scene at 224 224. The temporal position embedding x pero-initialized, while all other layers are randomly initialized. For training, we use the Adam optimizer (28) with warmup and cosine annealing. The loss coefficients are set as λHm = 1000, AvEc = 3, Àro = 2 and ÂLAH ASA 1. To counter the class imbalance in positive negative social gaze labels, we weight positive samples for the social gaze loss by 2.
All models were trained on an internal cluster using single Nvidia RTX 3090 GPU with 24GB memory. Training time for a single experiment is approximately hours on both GazeFollow and VSGaze. Total compute across all experiments is approximately 140 GPU hours.
Incorporating Auxiliary Information
Previous studies on analyzing conversations during meetings have shown that people usually look at the other speaking participants (45), and such cues be exploited for gaze target selection
16

---

Dutasel
Method
Dist. +
AP1o 1
Flaan 1
Flaaso
APss
Sprs-spk
0.107
0.938
0.810
0.588
0.590
Ours
0.107
0.940
0.812
0.603
0.576
Table Performance for incorporating people's 's speaking information in our model. Best results are in bold.
LAH: 0.95
LAH 0.77
Figure 3: An illustration of the few cases where the predicted gaze point does not match with the predicted LAH label. The uncertainty in the gaze target is reflected in the heatmap, while the uncertainty in the LAH target is reflected in the LAH scores.
(35). Hence, expect that identifying speaking persons provide better understanding for gaze following, and help recognize attentiveness towards people, especially speakers. The latter is especially important in autism diagnosis, eye contact closely monitored by the clinician when they call out to the tested child (2).
F1 Experiments and Results
To incorporate speaking information in our model, we adapt the Person Module (Section Specifi- cally, we first obtain speaking scores for each person using active speaker detection model (33) The model retrained detect speakers using the video alone and with no audio. The obtained speaking scores s;, for each person are linearly projected to the token dimension using Pspk. and added to the person token. Hence, the new person token is obtained as:
Pi, = Paaze
(g,,t temp + Papx ($i.,) Poox (h. box).
(5)
We note that this formulation can easily be extended incorporate other kinds of person-speciffe auxiliary information such gestures. We explore this aspect in follow up work where we extract gestures and other cues using vision-language models (8)
We provide results for incorporating speaking information in Table 7 On VSGaze, once again we see improvements for SA while other metrics remain similar. As the speaker detection model tends to fail in cases with side-view faces or for children in the case of ChildPlay, we may further benefit from better speaker detection model. In addition, investigating other ways to better capture and incorporate such auxiliary information in our model is direction for future research.
G
Qualitative Analysis
G.1 Qualitative Results and Comparisons
We provide qualitative results from our models in Figure 5 We observe that all models perform well, accurately capturing people's 's gaze targets and social gaze behaviour. We also see that incorporating temporal and speaking information can help improve predictions.
In the first sequence, the static model occasionally makes an error for person 1, and picks person 2's hands as the predicted gaze target. This is because it cannot distinguish blinking from when person lowers their gaze. On other hand, the temporal models recognize blinking and maintain the target as person 2's face. In the second sequence, the static model misses shared attention between persons 1,2.3 in the first frame and persons 2,3 in the third frame. This sequence challenging due to the presence of subtle head motions which the temporal models can better capture. In the third sequence,
17

---

Trastary
Fuace
Tranabiee
Pasurarcks
Transors
Pasmurcha
Puaie
Trandfree
Head
(a)
Figure 4: The standard DPT (a, taken from (39) and proposed person person-conditioned re-assemble
stage (b). This transformed DPT is used for predicting gaze heatmaps for each person in the scene.
both the static model and our proposed model make an error when predicting person 3's gaze target frame However, our model with speaking information recognizes person as the target given their high speaking score.
In addition, we provide qualitative comparisons of our model against other methods Figure We see that our model performs better overall, accurately inferring people's gaze target and social gaze behaviour despite the complexity of the scenes with multiple salient targets, obscured eyes, varied settings and age groups.
G.2 Alignment Between Gaze Following Outputs and Social Gaze Decoders
We perform analysis to understand the difference in performance between post-processing gaze following outputs or using the predictions from the task specific decoders. For LAH, we find that predictions for the two schemes align 87% the time. On visualizing outputs, we identify that cases where they don't 't match are usually where the model confused between two potential targets. So the arg max of the gaze heatmap for obtaining the gaze point picks one target, while the arg max of the LAH scores picks the other target. This confusion in target selection is illustrated for person in Figure 3 We see that the predicted gaze heatmap highlights both person and 3, and similarly, the LAH scores for looking at both persons 2 and 3 high.
H Gaze Heatmap Prediction Details
As mentioned in Section 3
we rely on the standard DPT (39) decoder that has been developed for dense prediction, and propose an interesting way transform it performing person-conditioned gaze heatmap prediction.
Similar to an FPN (27), the DPT assembles the set of ViT image tokens into image-like feature representations at various resolutions. The feature representations the final dense prediction. Specifically, the DPT decoder contains two stages: 1) A Re-assemble stage to construct feature maps at specific resolutions at each block. and 2) Fusion stage where the feature maps across consecutive blocks are upscaled and combined.
then progressively fused into
To include the person specific information, represented by the person token in architecture, modify the re- re-assemble stage to filter only the information relevant for the given person (Figure 4 More precisely, following the standard DPT, we first project the input frame tokens f. at level to a lower token dimension using 1 conv layer, followed by spatial upsampling downsampling using a strided 3 x transposed conv layer or conv layer respectively;
DPTT Split Resample" (ConnplalCCCont(((eee (Concat(f.")) (FF
(6)
where Split is the reverse of the spatial concatenation operation, Concat, and Resample is the spatial uppampiing/downsampiii
operation. To condition on person, we then perform a token-wise
hadamard product of the projected frame tokens with the the projected person token (using projection layer P from the corresponding block b.
PDP-cc.b = PDPT Ph P (p:b)
where denotes the hadamard product. The standard Fusion stage then follows obtain the predicted gaze heatmap. Note that superscripts DPT, DPT-c denote updates the frame tokens.
18

---

I Limitations
As discussed Supp. gaze following outputs and predictions from the social gaze decoders do not always align (13% cases). One possible solution is to post-process gaze following outputs for LAH and LAEO as done the case of Ours-PP. This ensures that outputs for gaze following. LAH and LAEO align. However, further aligning the outputs with SA is more challenging as post-processing for SA does not account for whether the gaze points fall on the same semantic item. Ensuring that outputs for all social gaze tasks and gaze following align is challenging and interesting direction for future research. Potential directions include more refined post-processing techniques, or the addition of task consistency losses and regularization. The latter particularly interesting may further increase the benefit of the social gaze losses.
Secondly, discussed in supplementary D temporal information does not bring large improvements in performance, with improvements observed mainly for SA. Investigating new architectures, datasets and metries to improve training and evaluation of temporal information is another important and interesting direction for future research.
J Broader Impact
Gaze following and social gaze prediction has strong potential for positive societal impact. Indeed, eye-contact and shared attention are monitored during the ADOS autism diagnostic test for children ( as marker for social communication. Given the time consuming nature of clinical tests and large scale prevalence of autism in 36 children (29),, automatic tools for autism screening based on gaze can help flag potential autism cases and reduce the burden on doctors (47).
Nevertheless, as with any medical application, such screening tools must used carefully to avoid missing cases, for instance through human-in-the-loop systems. It is also important to ensure that gaze algorithms do not invade the privacy of people. They should be deployed with appropriate consent and only for specific applications.
19

---

Ours spk
Ours
Ours static
(a) Strs-static fails recognise person blinking frames 2.4
Oura spk
Ours
Ours statle
(b) Ours static misses shared attention behaviour frames 1.3
Outtaspi
Ours
Ours dtati
(c) Ours-spk captures the right target for person 3 in frame
Figure 5: Qualitative results of our proposed model (Ours), our model with speaking information (Ours-spk) and our model without temporal information (Ours-static). When the target predicted to be inside the frame, we display the predicted gaze point and the social gaze tasks with the associated person id(s)
20

---

323
12345
12345
12395
Ours
Chongs
ChongT
Gupta
Figure Qualitative comparison of our model against other methods Chongs, Chongr Gupta (17). Our model performs better overall, outperforming other methods in complex scenes with obscured eyes, multiple salient targets, varied settings and age groups.
SA
Figure 7: Samples from VACATION, Person 3 in both cases is missed as the associated person is already anno- tated with social gaze 'state".
Figure Results from Tonini al. (2023) demonstrating incorrect head de- tections. Annotations in white.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim three major contributions: novel architecture described in Section 3 a new dataset described in Section 4.1.1 and new social gaze protocols and metrics described in Section 4.3 Our experimental results are provided in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have provided a discussion on limitations in Supplementary
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe architecture in Section and our dataset construction in Section 4.1.I with further details in Supplementary
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibilityy may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the data and code upon cleanup.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ priips.cc/ubbliccguu.eee CoodeSumeeieeeeee ssi iessonPolli mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details including data splits, hyperparameters and type of optimizer are provided in Section 4.2 with additional details in Supplementary B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We are unable to compute error bars due to constraints in terms of access to computational resources and training time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on computer resources in Supplementary
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have read and observed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential positive and negative societal impacts of the work have been discussed in Supplementary []
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not think our data and models high risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers for leveraged datasets, code and models have been cited. Appropriate licenses will be included when releasing code and data.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets have been currently released with the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "0h8XERGBd8", "reviewer": "Reviewer_7cdN", "summary": "This paper focuses on social gaze prediction in videos. An approach based on ViT has been proposed, combining three modules including person module, interaction module, and prediction module. The authors also summarised the current shortcomings of the existing datasets and have introduced a new dataset comprising social gaze interactions such as looking at someone, looking each other, and shared attention.", "questions": "1. Could you please demonstrate that the differences between the obtained values are significant?\n\n2. Could you please discuss your contributions with respect to the existing transformer-based approaches for gaze following and social gaze estimation?\n\n3. Could you please explain why you selected these methods for comparison specifically, and why you chose the datasets like GazeFollow and VideoAttention, most of which do not even have annotations for social gaze cues?", "limitations": "The authors have discussed the limitations and broader impact satisfactorily. In addition, they have provided detailed visualisations to offer insight into failure cases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strengths of the paper are:\n\n- Focusing on video-level gaze prediction, particularly on events like shared attention, which goes beyond gaze target estimation in images.\n- Introducing a new dataset for the target problem.\n- Overall, the paper is well-written and well-organized.", "weaknesses": "The main weakness of the paper is the results. If you refer to Tables 2 and 3, the differences in terms of distance and other metrics are very small, making it difficult to assess the benefits of the proposed approach. In some cases, the differences in terms of distance range between 0.01 and 0.001. This makes the results questionable in my view. The authors may need to run statistical tests to demonstrate that such differences are significant, or these improvements may be by chance given that the differences are negligible in most cases.\n\nRegarding the methodology, I am not sure if papers like [1] and [2] were available at the time of submitting this paper, but it is hard to see how the proposed approach is competitive compared to these aforementioned approaches in terms of both methodological novelty and performance.\n\nOn page 3, the authors mention \".. both methods do not address the gaze following task.\" However, the proposed method and dataset do not address the gaze following task either. Indeed, the new dataset only includes a subset of the gaze communication cues presented by Fan et al. at ICCV 2019. Why hasn't this dataset been considered in the evaluations given its relevance?\n\n[1] Sharingan: A Transformer Architecture for Multi-Person Gaze Following\n[2] A Unified Model for Gaze Following and Social Gaze Prediction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and for raising valuable discussion elements. We answer their comments and questions below.\n\n**Significance of improvements**\n\nWe agree with the reviewer that differences of 0.001 can be negligible, however, differences in 0.01 are significant as discussed below:\n- Distance: This metric for gaze following computes the distance between the ground truth and predicted gaze point on a normalized 1x1 grid. For an HD image of size 1080x720, a difference of 0.01 results in a difference of up to 11 pixels in image space on average (and often more). This can result in selecting a completely different target, ex. a nearby face.\n- F1: This metric for social gaze computes the harmonic mean of the precision and recall scores. A difference of 0.01 can correspond to a difference of 3% in either precision or recall.\n\nFollowing the suggestion of the reviewer, we also re-trained our model with 5 different seeds on VSGaze, and obtained a standard deviation of 0.0006 for distance, 0.0017 for $AP_{IO}$, 0.0007 for $F1_{LAH}$, 0.0048 for $F1_{LAEO}$ (LAEO has significantly less positives, Table 1) and 0.0020 for $AP_{SA}$. We further include qualitative examples in the attached pdf with the overall response, where we see that our model outperforms the baselines in various complex scenes.\n\n**Comparison to existing transformer based architectures**\n\nWe discuss existing transformer based architectures in L93-100, where we mention that they follow a DETR style approach for simultaneously predicting people’s head bounding box and gaze target. While their approach is interesting, they are prone to head detection errors (examples in attached pdf with overall response). And as their evaluation relies on matching detected and annotated heads (and performance is only reported on the matched heads), it is difficult to compare performance against them. Also, these methods do not address social gaze prediction. \n\nWe thank the reviewer for references [1, 2], which are interesting and contemporary to our work. These were not available at the time of our submission. Nevertheless, our work differs notably from these studies.\n\nFirstly, [1] focuses solely on the gaze following task. Their architecture is based on a ViT and treats person and frame tokens equally in the self-attention layers, which as noted in their paper, can be limiting. In contrast, our architecture allows the processing of person and frame tokens through separate transformers, facilitating interactions via cross-attention. In particular, this separation also allows for temporal processing of person tokens at multiple levels of the architecture.\n\n[2] extends [1] by leveraging a frozen gaze following model, and adding graph layers to model interactions and predict social gaze. However, this approach has several drawbacks:\n- As shown in [2], the graph layers provide limited benefit, which the authors attribute to over-smoothing, a known issue in graph neural networks. Our transformer-based architecture avoids this problem and benefits from the Interaction Module components, as demonstrated in our ablations in Section D.2.\n- Since [2] freezes the gaze following backbone during training, the gaze following and social gaze tasks cannot complement each other during training. Our analysis in L393-403 demonstrates that these tasks provide mutual benefits, and jointly training our architecture with both gaze following and social gaze losses yields the best performance.\n\nIn terms of performance, our method performs comparably to [1] for gaze following on VAT (Dist. 0.105 vs 0.107) and improves over [2] for LAEO on UCO-LAEO (AP: 0.974 vs 0.946). We are unable to compare performance for other tasks against [2] as they use different performance metrics. We leave a detailed quantitative comparison for future work.\n\n**Method does not address gaze following, reasoning behind choice of datasets and compared methods**\n\nWe would like to clarify that **our method *does* address the gaze following task**. We discuss the gaze heatmap decoding step in L218-230 and provide results for gaze following using the distance metric (abbreviated Dist. in the tables). Indeed, our method **achieves the state of the art multi-person gaze following performance on standard benchmarks as indicated in Tables 3a,b,c**. \n\nWe compare performance against other gaze following methods whose predictions can also be post-processed for social gaze. Fan et al. (2019) and Chang et al. (2023) do not address gaze following, which is an important task to identify the target of shared attention, and improves performance for social gaze prediction when trained jointly (L393-403). Further, recent methods for gaze following [3] have been shown to outperform task-specific social gaze models (L126-127). \n\nAlso, Fan et al. (2019) predict a single social gaze ‘state’ for a person, not allowing for simultaneous social gaze behaviors ex. LAEO and SA (L137-139). This issue extends to their dataset, which annotates a single social gaze state for a person at a given moment (L276-279). In the attached pdf, we show examples from the dataset where this annotation protocol fails. These issues were also reported in [2,4]. \n\nBesides this serious issue, VSGaze significantly differs from Fan (2019). **Content wise, it is much more diverse**, including scenes from TV shows, daily activities and childcare settings. It is also **more complex: in terms of number of people** in the scene (3.43 vs 2.14) and **size** (5.8x larger).\n\n[1] Tafasca et al. (2024). Sharingan: A Transformer Architecture for Multi-Person Gaze Following. CVPR.\n\n[2] Gupta et al. (2024). A Unified Model for Gaze Following and Social Gaze Prediction. FG.\n\n[3] Chong et al. (2020). Detecting attended visual targets in video. CVPR.\n\n[4] Belen et al. (2023). Temporal Understanding of Gaze Communication with GazeTransformer. Gaze Meets ML Workshop at NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MYdgIE8gi2", "reviewer": "Reviewer_5XsF", "summary": "This paper presents a novel framework for multi-person temporal gaze following and social gaze prediction. The authors propose an architecture that jointly predicts gaze targets and social gaze labels for all people in a scene. It uses a transformer-based model that processes both frame tokens and person-specific tokens to capture gaze information and interactions. They also build a dataset called VSGaze, which unifies and extends annotations across multiple existing gaze following and social gaze datasets. This allows joint training on multiple tasks and datasets. The proposed model achieves state-of-the-art results for multi-person gaze following and competitive performance on social gaze prediction tasks.", "questions": "It seems challenging to clearly state whether the training of models combining multiple tasks is useful even with a small amount of data based on the current results. Is there anything we can say about the relationship between data quantity and joint training? Since the effect of large-scale training with multiple datasets itself is not the core argument of this paper, I think it would be more insightful if this aspect can be carefully separated from the observations.", "limitations": "The discussion in the appendix seems to be conducted appropriately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, this is a very solid study, and the comprehensive approach of organizing the data and training a large-scale unified model is quite logical.\n- There is a noticeable improvement in performance across multiple tasks. Experiments are being conducted thoroughly from various angles.", "weaknesses": "- Conversely, the network's structure is relatively straightforward, combining existing approaches, and its novelty as a method is not necessarily significant.\n- The model's generalization capability to unknown domains and datasets has not been evaluated. While preparing the data is challenging, it is particularly crucial for methods involving humans.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and for highlighting our paper as a very solid study. We answer their questions and comments below. \n\n**Novelty of method**\n\nWe refer the reviewer to our discussion on the novelty of our architecture in the overall response.\n\n**Generalization to new domains/datasets**\n\nWe agree with the reviewer that having a model that generalizes well to different domains and datasets is important. Indeed, this was a main motivation behind creating VSGaze, which extends and combines datasets containing scenes from diverse settings such as daily activities (VideoCoAtt), talk shows (VAT), childcare (ChildPlay) etc. We show that our model trained on VSGaze performs well on each of its constituent datasets in Table 4. On the other hand, training on a specific domain results in poorer generalization to other domains. For instance, we evaluated our VAT trained model on ChildPlay and found that it generalizes poorly (Dist: 0.119, $AP_{IO}$: 0.991, $F1_{LAH}$: 0.624, $F1_{LAEO}$: 0.363, $AP_{SA}$: 0.194) compared to the model trained on VSGaze (Dist: 0.113, $AP_{IO}$: 0.993, $F1_{LAH}$: 0.651, $F1_{LAEO}$: 0.436, $AP_{SA}$: 0.216). We leave the investigation of generalization to unseen domains and datasets for future work.\n\n**Data quantity vs joint training**\n\nThis is an interesting and important question that we discuss under ‘Impact of VSGaze’ in L404-410. We find that fine-tuning models on specific datasets (Table 3) typically results in better performance compared to training on VSGaze (Table 4). This is because the model can learn dataset specific priors, ex. more LAH cases in VAT compared to ChildPlay (Table 1). For instance, on VAT, Gupta [1] has a distance score of 0.138 when trained on VSGaze, compared to a score of 0.134 when directly fine-tuned on VAT. Also, our proposed model has a distance score of 0.112 when trained on VSGaze, and a score of 0.105 when fine-tuned on VAT. Hence, while we may expect models to benefit from more data, accounting for different priors and statistics across datasets (especially given VSGaze’s diversity as discussed above) brings additional challenges. We will detail this aspect more in the final version of the paper.\n\n[1] Gupta et al. (2022). A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings. CVPRW.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2NupS6SD04", "reviewer": "Reviewer_poG5", "summary": "Paper proposes a novel framework which solves multiple gaze prediction tasks, gaze heatmap, in-out frame classification, social gaze classification for multi-person in one-pass simultanously. It also contributes a new dataset by extending existing datasets annotations. Comprehensive experiments were conducted for multiple datasets (GazeFollow, VideoAttentionTarget, ChildPlay, VideoCoAtt, UCO-LAEO which shows superior results compare to existing methods.", "questions": "One of the paper's claimed contribution is the temporal aspect of gaze. However, in the proposed model, there is no clear design of this beyond using a simple sequence of frame tokens and temporal information is aggregated with standard attention mechanism. The main discussion of temporal information is relegated to the Appendix. Will the author consider reframing the importance of temporal information in the title and abstract, or to move the temporal information discussion to the main paper?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Paper's experimental setup is comprehensive and supports the central claim of the paper of unifying several gaze prediction/classifcation tasks.\n\n2. Sufficient ablation studies were also conducted to investigate the contribution and importance of the various submodules.\n\n3. The extension of existing datasets with more annotations is also a good contribution to the community.", "weaknesses": "Paper's technical contribution is incremental. The framework design, while novel, does not have any significant improvements over prior works.\n\nThe contributed dataset is also an extension of existing datasets. However, as pointed out in the rebuttal, the effecient construction process of the dataset is also a strength of this paper.\n\nThis is a good paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and for noting our submission as a good paper. We answer their questions and comments below.\n\n**Novelty of method**\n \nWe refer the reviewer to our discussion on the novelty of our architecture in the overall response. \n\n**Dataset is an extension of existing datasets**\n\nWe appreciate the reviewer highlighting our dataset as a good contribution to the community. We would like to clarify however, that while VSGaze directly benefits from existing datasets (e.g. by having gaze following annotations), it is not just a concatenation and straightforward extension of these datasets.\n\nIndeed, annotating videos with gaze following and social gaze labels is a time-consuming process, which is why existing datasets typically annotate for a subset of these labels and are small in size. Hence, we wanted to come up with a **scalable method to build a dataset with *all of these labels*, that was the *largest* of its kind, and *diverse* in terms of scene content**. A key insight was that people’s head bounding boxes could be used as a semantic entity to unify annotations across multiple gaze datasets. However, the considered **datasets only annotated a subset of people** in the scene, whereas **all people’s head bounding boxes are required** to obtain all possible positive and negative social gaze labels. Hence, we leveraged a strong head detector to extend the head bounding box annotations in these datasets and manually verified the detections. This extended set of head bounding boxes was then coupled with the existing gaze annotations to extend and unify them across datasets. The detailed process for the construction of VSGaze is given in L280-308 and supplementary B. \n\nFurther, the method for constructing VSGaze can be extended in the future to obtain more gaze annotations. For instance, we focus on people’s heads as a semantic entity for unification, but could leverage strong segmentation methods such as SAM [1] to unify gaze annotations using other semantic entities such as people’s hands. Also, future dataset annotation efforts can focus on a subset of gaze annotations such as gaze following labels, and can then follow our method to extend them with social gaze annotations. \n\n**Temporal information is a simple aggregation of frame tokens**\n\nWe would like to clarify that temporal information is **incorporated in our architecture by aggregating *person tokens* across time using self-attention**. While self-attention itself is fairly standard and has shown strong performance for temporal tasks such as action recognition [2], designing how to use it for a specific task and domain is a research question. Previous works on temporal gaze following [3,4] performed a frame level aggregation of temporal information, achieving limited to no success. On the other hand, **we hypothesized that the broader scene tends to remain relatively static for a short temporal window**, and instead focused on **modeling person-level temporal information** to account for **gaze direction as well as gaze target dynamics**. To do this, we incorporate temporal aggregation of person tokens at multiple levels of the architecture. As discussed in the paper (supplementary D.1) and the overall response, our architecture benefits from the addition of temporal information, with most improvements for shared attention in particular (Table 5). It also learns to account for behaviors such as blinking (Figure 5) which are not captured by current metrics. Nevertheless, more research, datasets and metrics are needed to fully exploit this information.\n\n**Moving discussion on temporal information to the main paper**\n\nWe moved the discussion on temporal information to the supplementary due to space limitations. Based on the reviewer’s suggestion, we plan to move the discussion to the main paper by instead moving Section 4.2 (training and validation details) and parts of Section 4.1 (VSGaze construction details) to the supplementary.\n\n[1] Kirillov et al. (2023). Segment Anything. ICCV.\n\n[2] Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS.\n\n[3] Chong et al. (2020). Detecting attended visual targets in video. CVPR.\n\n[4] Miao et al. (2023). Patch-level gaze distribution prediction for gaze following. WACV.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqtcFpsogX", "reviewer": "Reviewer_FWeu", "summary": "This paper handles the problem of predicting human social labels and gaze heatmap simultaneously for all people in the input image sequence, which improve the accuracy of gaze following based on human social cues. Specifically, they first calculate person tokens through a person module, and then design an interaction module to extract interactive relationship between humans and scene. The author proves the effectiveness of the proposed model on multiple datasets, and verifies the role of each part of the model through ablation experiments. And a new dataset suitable for both subtasks is also annotated.", "questions": "1. Change the name of orange box in the Person Module of fig2 to Temporal Gaze Processor may be better to match the description in the essay.\n2. In \"$I_{ps}^b$ It is implemented as a single Transformer layer with cross-attention\", \"it\" may be a redundant word.\n3. Does  $f_t^{o,b}=V_b(f_t^{p,b})$ in line 198 denote the output frame tokens for the block b, rather than the block B in Person-Scene Interaction?\n4. Where is \"temporal transformer architecture\" in Fig.1?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The experimental section of this article is very detailed, and good results have been achieved in comparison with different methods on multiple datasets, proving the universality of this method.\n\n2. The author for the first time incorporates human social cues into gaze prediction tasks to obtain interaction relationships between different individuals, which is an interesting topic.", "weaknesses": "1. The author lacks an introduction to the Pairwise Instance Generator module in the model. What is its function? There is also a lack of explanation of the number of Interaction Blocks. What impact does the change of B have on the effect of the model?\n2. The symbols used in the article are confusing and difficult to understand. For example, “Each block then processes the set of output person tokens $P_{1:N_n,1:T}^{o,b-1}$  and output frame tokens $f_{1:T}^{o,b-1}$  from the previous block”, why not indicate the range of b in this sentence? And too many superscripts and subscripts in formulas and expressions can easily confuse readers in the introduction to Interaction Module.\n3. Lack of visual results to prove in which scenarios the proposed model has an advantage over the comparison methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and for their positive evaluation of our paper. We answer their questions and comments below.\n\n**Clarification of elements**\n\nThe Pairwise Instance Generator was drawn in the figure to illustrate that the social gaze decoders take a pair of person tokens as input (L234-235). We will clarify this in the text. $f_t^{o,b}$ indeed refers to the output tokens for the specific block $b$. We use $B=4$ Interaction Module blocks as defined in L318-319. Our Interaction Module is inspired from ViT-Adaptor which found this value to give the best performance. We did not conduct additional ablations for different values of $B$. We included multiple notations for completeness when describing the Interaction Module, but will work on simplifying the notation so that it is easier to follow. The visualizations in Figure 1 are from our proposed multi-person temporal transformer model (Ours in results). We also thank the reviewer for pointing out the typos, we will correct them in the final version of the paper. \n\n**Qualitative comparisons**\n\nWe provide qualitative comparisons of our method against other methods in the attached pdf with the overall response. We see that our model performs better overall, accurately capturing people’s gaze target and social gaze behavior. This is despite the complexity of the scenes with obscured eyes, multiple salient targets, varied settings (indoor, outdoor) and different age groups (children, adults). We will include these examples in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their feedback on our paper, which presents the following contributions:
- A novel **temporal, multi-person architecture** that jointly models gaze following and social gaze prediction.
- A new **dataset, VSGaze**, which extends and unifies annotations across multiple gaze following and social gaze datasets. As by far the largest and most diverse dataset of its kind, it opens new avenues for gaze modeling.
- New evaluation protocols and metrics for assessing semantic gaze following and social gaze performance.

We are pleased that the reviewers found our central premise of unifying several gaze prediction tasks into a single framework logical (5XsF). They appreciated our novel architecture that, for the first time, encodes human interaction relationships for gaze tasks (FWeu). They also appreciated our extensive experiments (5XsF, poG5, FWeu), which demonstrated that 
(1) Our architecture can successfully model all tasks jointly, and that **this new architecture and joint training improved performance across multiple tasks and datasets compared to other methods**; (2) the importance and contribution of the different sub-modules in the overall performance.

They further noted the value of our proposed dataset (7cdN, poG5) as a good contribution to the community and appreciated our social gaze metrics for characterizing semantic gaze performance (7cdN). Lastly, they described our paper as well-written and organized (7cdN)

A common concern among reviewers (7cdN, 5XsF, poG5) was the novelty of our architecture. We would like to emphasize that our architecture was developed to address several complex research questions:

**(1) How to account for multiple people in the scene?** As discussed in lines 45-47, this challenge is inherently more complex than single-person gaze following, as our architecture has to process the scene only once, and to capture salient items for all individuals while retaining the ability to decode each person's gaze target. Among the cited gaze-following papers, only [1] presents a comparable multi-person architecture, but as it processes each person separately, it fails to account for interactions (see discussion in lines 89-100).
 
In contrast, by employing specific person tokens to encode individuals and an Interaction Module, our architecture is able to capture interactions between people and the scene (encoded as frame tokens), and has achieved state-of-the-art results for multi-person gaze following (Tables 3a, 3b, 3c).

**(2) How to jointly model multiple gaze tasks?** Predicting the gaze target for each individual while simultaneously predicting social gaze between pairs of people presents a significant challenge, as gaze targets are represented as heatmaps, whereas social gaze is a binary label. Jointly modeling these diverse tasks has not been previously attempted. Our architecture overcomes this challenge thanks again to our token-based representation. It leverages person and frame tokens for gaze heatmap prediction through the conditional DPT decoder, and pairs of person tokens for social gaze prediction through the social gaze decoders. This approach achieves strong performance across all tasks without compromising on any of them (Table 2).

**(3) How to incorporate temporal information?** Incorporating temporal information is particularly challenging due to the small size of gaze datasets. Only two other gaze-following methods [2, 3] have attempted this, with limited success and without accounting for gaze direction dynamics. In contrast, our architecture addresses this challenge by incorporating temporal information at multiple levels, from gaze direction dynamics to gaze heatmap prediction. Inspired by ViT-Adaptor [4], we freeze the ViT layers during training on VSGaze, allowing frame tokens to adapt through interactions with person tokens. This approach leverages temporal information and improves performance, especially for shared attention tasks (Table 5). Our qualitative examples (Figure 5) demonstrate that our temporal model captures behaviors such as blinking, but unfortunately such interesting behaviors are not accounted for by current metrics. Despite these advancements, as discussed in supplementary section D.1, further research, datasets, and metrics are needed to fully harness the potential of temporal information.

In the supplementary F, we show that our architecture also supports incorporating person-level auxiliary information such as their speaking status. We will include this discussion and all other feedback from reviewers in the final version of the paper. 

[1] Jin et al. (2021). Multi-person gaze-following with numerical coordinate regression. FG.

[2] Chong et al. (2020). Detecting attended visual targets in video. CVPR.

[3] Miao et al. (2023). Patch-level gaze distribution prediction for gaze following. WACV.

[4] Chen et al. (2022). Vision transformer adapter for dense predictions. ICLR.

Meta-review: This paper proposes a framework for multi-person temporal gaze following and social gaze prediction. All the reviewers recommended acceptance of this paper. The AC checked the author response and the discussions, and agreed to accept this paper.
Decision: Accept (poster)