MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction

Authors: anshul  gupta
samy  tafasca
arya  farkhondeh
pierre  vuillecard
jean-marc  odobez
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:35:16

Abstract: Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference as an ad-hoc post-processing of the gaze following task. Furthermore, the vast majority of gaze following approaches have proposed models that can handle only one person at a time and are static, therefore failing to take advantage of social interactions and temporal dynamics. In this paper, we address these limitations and introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. It comprises (i) a temporal, transformer-based architecture that, in addition to frame tokens, handles person-specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, built from multiple gaze following and social gaze datasets by extending and validating head detections and tracks, and unifying annotation types. We demonstrate that our model can address and benefit from training on all tasks jointly, achieving state-of-the-art results for multi-person gaze following and social gaze prediction. Our annotations and code will be made publicly available.

Main paper: 
 introduction: 
 Social interaction plays a pivotal role in our daily lives and is influenced by an array of behavioral elements, encompassing not only verbal communication but also non-verbal cues like gestures (53) or body language. In particular, the ability to decode people's gaze, including communicative behaviors like eye contact and shared attention on a particular object, is highly related to our capacity to connect with or learn from others (44) . In contrast, the absence or impairment of these skills is often indicative of developmental disorders such as autism (42) . Thus, designing social gaze prediction algorithms and systems has attracted considerable attention from different communities, ranging from medical diagnosis to human-robot interactions (42; 43; 36) . In this work, we investigate whether we can build a unified framework to infer from video data the gaze target and social gaze label in one stage for all people in the scene. This requires: (i) A new architecture capable of jointly modeling these tasks, (ii) A large-scale dataset with annotations for all the tasks. Specifically, we focus on the LAH, LAEO and SA social gaze tasks (illustrated in Fig. 1 ). Methods for social gaze prediction in the literature adopt one of two approaches. The first one focuses on the design of dedicated networks to process pairs of head crops and potentially other scene information (31; 30; 11; 5; 12; 46) . While their specialization makes them effective, they offer little room for generalization to other gaze-related tasks. The second one first address the gaze following tasks (40) , defined as predicting the 2D location people's gaze targets, and then uses the predicted gaze points to infer social gaze through ad-hoc post-processing schemes. For instance, combining gaze following heatmaps from multiple people to predict shared attention (9) . However, gaze following itself is a challenging task. Besides geometric aspects, the task requires understanding and establishing a correspondence between top-down information related to the person's state, activity, cognitive intent, and bottom-up saliency related to the scene context (salient items like objects or talking people). Furthermore, gaze following performance, as measured by distance, does not always translate to similar social semantic performance (e.g. when evaluating if the predicted gaze point falls on a person's head or not (48) ). Motivation. Existing methods for gaze following suffer from several drawbacks. First, most of the methods perform prediction for a single person (40; 41; 9; 14; 17; 48) , requiring multiple inference passes on the same image to process multiple people in the scene. In contrast, a multi-person gaze following architecture processes the image only once and has to capture salient items for all people in the scene, while maintaining the ability to infer the gaze target of each individual. This is inherently more complex and challenging. Another drawback of the single-person formulation is that it does not explicitly model people's interactions, thereby preventing the possibility of jointly inferring people's gaze target and social gaze attributes. Secondly, the majority of proposed models for gaze following are static, using only a single image at a time. This is partly due to the absence of large and diverse video datasets, and the difficulty of leveraging large-scale static ones like GazeFollow (41) . This is a limitation, as temporal information can capture head and gaze coordination patterns (43) which can help gaze direction inference, especially when the eyes are not completely visible (34) . Finally, none of these methods have investigated learning the gaze following and social gaze prediction tasks jointly. Thus, it remains a research question whether such formulation can improve performance by having social cues inform the gaze following task and vice-versa, or if performance would degrade as we try to accommodate multiple tasks, datasets, and people within the same framework. Contributions. Given these motivations, we propose a new, unified framework for gaze following and social gaze prediction with the following contributions:• A novel temporal and multi-person architecture for gaze following and social gaze prediction (Sec. 3). Our approach posits people as specific tokens that can interact with each other and the scene content (i.e. frame tokens). This token-based multi-person representation allows for the modeling of (i) temporal information at multiple levels (from 2D gaze direction to 2D gaze target level), (ii) the joint prediction of the gaze target and social gaze label. • A new dataset, VSGaze, that unifies annotation types across multiple gaze following and social gaze datasets (Sec. 4.1.1). • New social gaze protocols and metrics for better evaluating semantic gaze following performance (Sec. 4.3). Results. In our experiments (Sec. 5), we show that our architecture achieves state-of-the-art results for multi-person gaze following, while also performing competitively against single-person models. It is also able to leverage our proposed VSGaze dataset to jointly tackle gaze following and social gaze prediction, achieving competitive performance compared to methods trained on individual tasks. In particular, our experiments further demonstrate that the performance benefits from this joint prediction, i.e. adding the social loss, improves gaze following performance, and vice-versa. Finally, the new social gaze metrics provide complementary information to the standard distance-based metrics, helping assessing model performance from the social interaction perspective. It is worth noting that our architecture is easily extendable and allows for the integration of auxiliary person-specific information that can influence the final predictions. In the supplementary F, we explore this aspect by integrating people's speaking status in the person tokens to improve the results. 
related work: 
 Gaze Following. Typical methods for this task exploit a two-branch architecture: one for processing the scene and the other for processing the person of interest (40; 9; 14; 17; 22; 23; 26) . They have distinguished themselves by the addition of other relevant modalities like depth (14; 17; 23) , pose (17) , and objects (20) , or by potentially leveraging scene geometry (19; 23; 14) . However, only a few efforts have addressed the multi-person case. (22) first proposed a simple architecture relying on a scene backbone to get a person-agnostic image representation that is subsequently fused with the head crop features of each individual obtained using another backbone. While this reduces computation, the model does not account for person interactions, as each head is processed separately. In another direction, (52; 51) rely on a transformer-based architecture to perform multi-person gaze following. Their methods borrow from DETR (6) , taking the image as input and simultaneously predicting the head bounding box and gaze target for every person in the scene. While these methods can implicitly model person interactions, an important limitation is that they compute performance on detected heads which are matched to the ground truth. Given that both the head detection and matching steps are error-prone, it precludes comparing their results to others. We provide examples in the supplementary (Fig. 8 ) where (51) makes incorrect head detections. Temporal gaze estimation. Temporal information has proven effective for 3D gaze estimation. Previous research developed models to learn from various inputs, including face, eyes, and facial landmarks using a multi-stream recurrent CNN (37) ; eyes and visual stimuli or raw RGB frames from in-the-wild settings using convolutional RNNs or LSTMs (38; 25) ; and the temporal coordination of gaze, head, and body orientations using LSTMs (34) . However, the use of such methods for the gaze following task in arbitrary scenes has been underexplored. The only exceptions are (9) who introduced a convolutional LSTM block at the bottleneck of the heatmap prediction architecture, and (32) who leveraged temporal attention over aggregated frame-level features. However, both approaches only showed a slight improvement compared to their static versions, highlighting the challenge of exploiting temporal information for this task. Conceptually, the methods did not model 2D gaze direction dynamics, and can not be extended for multi-person gaze inference. Social gaze prediction. Several research papers in the literature are dedicated to the study of looking at each other (LAEO) and shared attention (SA) tasks. For LAEO, most methods rely on processing the head crops to obtain some gaze directional information, and then combining it with 2D or inferred 3D geometric information to predict the LAEO label (31; 11; 30; 5) . Drawbacks include processing pairs of persons independently and, as they only process heads and do not address gaze following, lacking global image context and not extending easily to other social tasks like SA. Similarly to (52; 51), a recent paper (15) proposed an encoder-decoder transformer architecture to predict heads and the LAEO labels, and while achieving good results, suffers from the same drawbacks. Regarding shared attention, the first method to address it in the wild was (12) , which framed the problem as 2 tasks: the binary classification of whether SA occurs in a frame, and the inference of the location of the SA target object. Their method combined predicted 2D gaze cones of people in the scene with a heatmap of object region proposals, while others (46) directly inferred SA from the raw image. Since then, several methods leveraged combining gaze following heatmap predictions of all people (9; 52) and improved performance. Nevertheless, the above task formulation (12) used by all papers suffers from two main issues: (i) it cannot distinguish between multiple SA instances if they occur in the same frame; (ii) it does not determine which specific people are sharing attention. Our work solves both problems by framing the task as a binary classification between pairs of people. This formulation is more natural and has the benefit of extending to other social gaze tasks. Finally, none of the previous works performed both social gaze prediction tasks. There are three notable and interesting exceptions. First, (13) who addressed the inference of gaze communication activities (atomic and events, including LAEO and SA) using a graph-based approach with 12 dimensional tokens. However, as it predicts a single gaze 'state' for each person, it is problematic as it does not allow for simultaneous LAEO and SA. It also does not identify the other person involved in the social gaze interaction. The second is (7) who addressed dyadic communication by proposing a gaze following style 2-branch architecture processing order-dependent pairs of people. However, inference using (7) is inefficient because the model needs Np! (Np-2)! forward passes to consider all pairwise relationships for a given scene with N p people. Also both methods do not address the gaze following task. More recently, ( 16) did address all tasks using a graph approach, but there was no temporal modeling nor joint training on all tasks.  
architecture: 
 As motivated in Section 1, designing a temporal architecture for gaze following is challenging given the lack of large scale video datasets. Since the broader scene tends to remain relatively static for a short temporal window, we focused on modeling person-level temporal information to simplify the learning problem. In particular, our architecture processes person and scene tokens through separate transformers, with a temporal transformer for processing the person tokens. At the same time, it facilitates interactions between the scene and person tokens via cross-attention. Approach overview. Our approach is illustrated in Fig. 2 . It takes as input a sequence of t = 1 . . . T frames, as well as the head bounding box tracks h box i,1:T and corresponding head crops h crop i,1:T which are assumed to have been extracted for each of the i = 1 . . . N p persons. The outputs are the sequence of gaze heatmaps A i,1:T and in-out gaze labels o i,1:T for each person i, as well as the sequence of per-frame pair-wise social gaze labels for each task and pair i, j ∈ {1 . . . N p }: e i→j,1:T for LAH, e i↔j,1:T for LAEO, and c i,j,1:T for SA. The model proceeds as follows. First, each frame t is processed by a standard ViT tokenizer to produce the set of patch-wise frame tokens f t , resulting in a sequence of frame tokens f 1:T . In parallel, the Person Module processes the sequence of head crops from each person i using the Temporal Gaze Processor, and the resulting sequence outputs are then tokenized at each frame along with the bounding box locations to produce the sequence of person token p i,1:T . Secondly, the Interaction Module jointly processes the frame and person tokens, iteratively updating them at each time step through person-scene cross-attention interaction components and scene ViT self-attention, and in time through person spatio-temporal social interaction components. Finally, the Prediction Module processes at each time step the resulting frame and person tokens (from multiple blocks) to infer the sequence of gaze heatmaps and in-out gaze labels for each person, as well as pair-wise social gaze labels. We detail the three modules in the next sections.This module aims to model person-specific information relating to gaze and head location. Temporal Gaze Processor. It aims to capture all gaze-related information (direction, dynamics). First, individual head crops h crop i,t are processed by a Gaze Backbone G stat to produce gaze embeddings according to g stat i,t = G stat (h crop i,t ). Then, to model the gaze dynamics of a person, we rely on a Temporal Gaze Encoder G temp to process the sequence g stat i,1:T of gaze embeddings plus learnable temporal position embeddings x 1:T and obtain their temporal counterparts: g temp i,1:T = G temp (g stat i,1:T + x 1:T ). G temp is implemented as a single Transformer layer with self-attention. Finally, to supervise the learning of relevant gaze embeddings, we attach a Gaze Vector Decoder that predicts a person's 2D gaze vector at each time step, g v i,t = G vec (g temp i,t ), where G vec is implemented as a 2-layer MLP. Person tokenization. The person tokens are obtained by projecting the temporal gaze embeddings and normalized 4d head box locations using learnable linear layers (P gaze and P box respectively) to tokens of same dimension than frame token, and adding them together: p i,t = P gaze (g temp i,t ) + P box (h box i,t ).(1)The Interaction module aims at modeling the exchange of information between persons and the scene at each time step, as well as the spatio-temporal social interactions between people. One important goal of this process is to align the person and frame token representations so that (i) person-specific gaze heatmaps can be predicted from the set of output frame tokens and each person output token; (ii) in-out gaze and social gaze prediction can be made from the person tokens.To do so, we designed the module to consist of B blocks, each comprising Person-Scene Interaction and Spatio-Temporal Social Interaction components. The input to the first block is the set of person tokens p 1:Np,1:T from the Person Module, and the frame tokens f 1:T . Each block then processes the set of output1 person tokens p o,b-1 1:Np,1:T and output frame tokens f o,b-1 1:T from the previous block, and returns updated tokens after a series of self/cross-attention layers through the components.Person-Scene Interaction. This component models the interactions between people and the scene and can capture inferring gaze to scene objects or body parts like hands or exploit some global context. It is inspired by ViT-Adaptor (8) which has shown good performance for dense prediction tasks when relying on pretrained models and small amounts of data for the target task. It proceeds in 3 steps: The Prediction Module processes the set of output person {p o,b 1:Np,1:T } and frame {f o,b 1:T } tokens from all Interaction Module blocks to predict the person-specific gaze heatmaps and in-out labels, as well as the pair-wise social gaze labels. Gaze Heatmap Prediction. Here, we follow the model introduced in (49) which takes inspiration from the DPT decoder (39) for dense prediction tasks, and adapts it to handle multiple heatmap predictions from the same ViT outputs. This is performed by conditioning the decoding on each person's token. The standard DPT decodes the image features from multiple layers of a ViT in a Feature Pyramid Network (27) style. It works by fusing at block level b the feature maps from level b + 1 after an upsampling stage, and the feature maps computed by a reassemble stage from the ViT output of block b. We aim to apply this approach to the frame tokens {f o,b 1:T , b = 1 : B}, but conditioned on a specific person. In our model, this is achieved through a modification in the reassemble stage, in which the image feature maps produced by the standard reassemble stage are multiplied at every location (using a Hadamard product) with the projected person token p o,b i,t of that same block level. The gaze heatmap A i,t for each person at each frame is thus obtained as:EQUATIONwhere D denotes this conditional DPT. See (49) and supplementary H for details. Social Gaze Prediction. This decoder processes the person tokens from all B Interaction Module blocks to predict the social gaze label for every pair of people in every frame. In practice, the B tokens {p o,1 i,t . . . p o,B i,t } corresponding to a single person in a frame are linearly projected and concatenated to produce a multi-scale person token p ms i,t . Then, to predict a social gaze label, pairs of these tokens are concatenated and processed by the decoders E for LAH and C for SA (illustrated through the Pairwise Instance Generator in Fig. 2 ). Their outputs are the predicted LAH score e i→j,t for person i looking at j, and the predicted SA score c i,j,t for i, j.e i→j,t = E(p ms i,t , p ms j,t ) and c i,j,t = C(p ms i,t , p ms j,t ).E and C are implemented as 3-layer MLPs with residual connections. For LAEO, both people i, j need to be looking at each other for a positive label, and either one can be looking away for a negative label. Hence, we simply compute the LAEO score e i↔j,t as min(e i→j,t , e j→i,t ).In-Out Prediction. This decoder O processes the multi-scale person tokens p ms i,t to predict at every frame whether people are looking inside the frame or outside the frame, as o i,t = O(p ms i,t ), where O is implemented as a 5-layer MLP with residual connections.The total loss L is a linear combination of the gaze heatmap loss L HM , gaze vector loss L VEC , social gaze losses L LAH , L SA and the in-out loss L IO :EQUATIONL is applied at each time step per person for L HM , L VEC , L IO , and per pair for L LAH , L SA . All losses are standard: L HM is defined as the pixel-wise MSE loss between the GT and predicted heatmaps, L VEC as the cosine loss, and the social gaze and in-out losses as binary cross-entropy losses. Since LAEO is inferred from LAH predictions (Sec 3.3), we do not have any LAEO loss. 
datasets: 
 We perform experiments on multiple gaze following and social gaze datasets. GazeFollow ( 41) is a large-scale static dataset for gaze following, featuring 122K images. Most images are annotated for a single person with their head bounding box and gaze target point. Two other interesting datasets with annotations for multiple social gaze behaviours are VACA-TION ( 13) and GP-static (7) . However, VACATION does not provide annotations for all social gaze behaviours when they occur simultaneously (examples in supplementary Fig. 7 ). This annotation scheme is problematic and is linked to their method design as described in Sec. 2. On the other hand, GP-static only considers dyadic interactions and is not publicly available.A limitation of the above datasets is that they only contain annotations for gaze following or specific social gaze tasks. Hence, we propose the Video dataset with Social gaze and Gaze following annotations or VSGaze dataset. VSGaze extends head track annotations and unifies annotation types across VAT, ChildPLay, VideoCoAtt and UCO-LAEO. This allows for joint training of gaze following and social gaze, and provides new tasks and metrics for evaluating performance on the component datasets. The construction of VSGaze is described below.Extending Head Track Annotations. As each dataset contains annotations for only a subset of people in the scene, we detect all missing heads using the pre-trained Yolov5 head detection model ( 24) used by Tafasca et al. (48) , and track them using the ByteTrack algorithm (54) . We further manually verified the accuracy of the obtained tracks. This step is vital to obtain positive and negative social gaze pairs. Consider a scene with 3 people, i, j, k, where i is looking at j. If only i is annotated, the positive LAH pair i → j would be missed. The negative LAH pair i → k would also be missed.Unifying Gaze Following and Social Gaze Annotations. Given the extended set of head bounding box annotations, as well as existing gaze following and social gaze annotations, we then process these annotations to obtain gaze following and social gaze labels across all the datasets. The gaze target for UCO-LAEO and VideoCoAtt is set as the center of the LAEO and SA target bounding box respectively. LAH pairs are obtained by checking if the gaze target falls inside another person's head box. For LAEO we check the reverse as well. SA pairs are obtained by checking if the gaze targets for both people fall in the same head box. We provide the detailed protocol in the supplementary B. Annotation statistics are summarized in Table 1 . Overall, VideoCoatt is the largest source of annotations except for LAEO. We also see that the pair-wise annotations are skewed towards negative cases. The statistics further provide insight into the content of the datasets. As VAT, VideoCoAtt and UCO-LAEO contain clips from TV shows, there are many more instances of looking at other people and at each other. On the other hand for ChildPlay, LAH mainly occurs when the supervising adult looks at a child and there is limited LAEO.We follow standard practice (9; 14; 17) by first training the static version of our model (i.e. with no temporal attention G temp , I b pt ) on GazeFollow. It is trained for 20 epochs with a learning of rate of 1 × 10 -4 . The resulting weights then serve as initialization for our proposed temporal model. We freeze the ViT V and train the temporal model on VSGaze for another 20 epochs with a learning rate of 3 × 10 -6 . For validation, we use the provided splits for UCO-LAEO, VideoCoAtt and ChildPlay, and the splits proposed by Tafasca et al. (48) for GazeFollow and VAT. For all our experiments, we use T = 5 frames with a temporal stride of 3. To allow for batch training, we randomly sample up to N p = 4 people in a scene (padding in case there are less). During testing, we evaluate per sample and consider all people. The Interaction Module consists of B = 4 blocks , interacting with V at layers {2, 5, 8, 11}. Additional implementation details are provided in the supplementary E. Gaze Following. We use the standard metrics:• AUC: for GazeFollow, the predicted heatmap is compared against a binary GT map with value 1 at annotated gaze point positions, to compute the area under the ROC curve. • Distance (Dist.): the arg max of the heatmap provides the gaze point. We can then compute the L2 distance between the predicted and GT gaze point on a 1 × 1 square. For GazeFollow, we compute Minimum (Min.) and Average (Avg.) distance against all annotations. • In-Out AP (AP IO ): it is the Average Precision (AP) of the In-Out gaze prediction scores. Social Gaze. We propose an evaluation protocol for LAH, as well as a new pair-wise evaluation protocol for SA as motivated in Sec. 2. Social gaze decoders: For the LAH task we compute an F1 score. A sample is an individual person i, and at inference, it is assigned the person ĵ for which the e i→j,t is the largest. Hence, for a GT positive case, the prediction will be considered as a true positive if ĵ matches the GT target AND e i→ ĵ,t is above 0.5. Otherwise, the prediction is a false negative. Similarly, a GT negative is a true negative if e i→ ĵ,t is below 0.5, otherwise it is a false positive. For LAEO, as with LAH, we compute an F1 score. A sample is a pair of people i, j, and for person i, we consider e i↔ ĵ,t with the highest score. We then set e i↔j,t to 0 ∀j ̸ = ĵ before computing the performance. We also do the reverse for j, i. For SA, we compute a standard AP score by considering a sample as a pair of people, and thresholding predicted scores at different values to compute the area under the Precision-Recall curve. Gaze following methods: Social gaze predictions are obtained by post-processing the predicted gaze points. For LAH, we check whether the predicted gaze point for a person falls inside the target person's head box. For LAEO we check the reverse as well. We compute F1 scores for both. For SA, we check if the distance between two people's predicted gaze points is within a set of thresholds and compute an AP score. 
comparison against the state-of-the art: 
 We compare against recent SoTA methods addressing either social gaze tasks or gaze following. In addition, for fairness and to evaluate the impact of the VSGaze dataset, we also re-trained on this dataset the static image based models of Chong (9) (Chong S *) and Gupta (17) (Gupta*), as well as the temporal model of (9) (Chong T *), the only temporal gaze following model with available code. VSGaze. The results on VSGaze are given in Table 2 . Note that regarding our approach, for social gaze, we compute the scores by leveraging either the predictions from the respective task decoders (Ours), or by post-processing the gaze following outputs of our model (Ours-PP). Compared to the baselines, we observe that our model achieves the best performance for all tasks except for in-out gaze prediction. In particular, we achieve significant gains in the distance and AP SA metrics when leveraging the predictions from the SA decoder. The latter highlights the importance of modeling SA as a classification task compared to post-processing gaze following outputs, which struggles to capture whether the gaze points for a pair of people falls on the same semantic item. In addition, we note that better gaze following performance does not always translate to better social gaze performance. For instance, although Chong S * has a better distance score compared to Chong T *, it performs worse for shared attention. This effect is even more pronounced on ChildPlay (Supp C), and suggests the benefit of considering social gaze metrics for better characterizing the performance of gaze following models, especially its semantic performance. In the supplementary C, we provide a breakdown of performance on each of the component datasets of VSGaze.State-of-the-art comparison: fine tuning on individual datasets. Table 3 compares our model against task specific methods. For GazeFollow, we use our static model (Ours-static) that was trained on GazeFollow and used to initialize our model trained on VSGaze. For the video datasets, as SoTA methods were trained (or finetuned) on individual datasets, for fairness we also fine-tune our model on these datasets, investigating two initialization alternatives: either from the model trained on GazeFollow (Ours), or from the model trained on VSGaze (Ours †). Note that we are unable to compare against previous results for VideoCoatt due to our new pair-wise evaluation protocol that better captures SA performance (Sec 4.3). On GazeFollow and VAT, our model outperforms the only other comparable multi-person gaze following model of Jin (22) . It also achieves competitive or better results to single-person methods, even those leveraging auxiliary modalities such as depth (14; 50; 4; 23; 21; 48) . Importantly, on the social LAEO task, we set the new state of the art on UCO-LAEO, far outperforming methods designed specifically for LAEO (31; 30; 11) . We also note that fine-tuning using the VSGaze model initialization can improve results compared to the standard protocol of fine-tuning after training on GazeFollow (ex. distance on ChildPlay and AP LAEO on UCO-LAEO). This suggests that training on VSGaze can leverage the complementary knowledge provided by the different tasks and datasets, which follows observations made in other works addressing multi-task training (10).Impact of Architecture. Comparing the performance of our model with no social gaze losses (Ours-noSoc) against the baselines (Table 2 ), we see that it already performs on par or better than them while being much more efficient as it processes the image only once for all people in the scene.It also serves as a strong gaze following baseline to compare performance against. Impact of Social Gaze Loss. Our architecture can further benefit from the social gaze losses, showing improved gaze following performance and social gaze prediction (Ours and Ours-PP, Table 2 ). In particular, we observe significant gains for the SA compared to Ours-noSoc. Interestingly, the addition of the social gaze losses also better aligns the gaze following outputs for social gaze prediction.Comparing Ours-PP and Ours-noSocial, we see that performance for all social gaze tasks is improved. Impact of Gaze Following Loss. We additionally train our model without the standard gaze following losses: heatmap, gaze vector and in-out (Ours-noGF, Table 2 ). Across VSGaze, we see that performance for all social gaze tasks drops, which indicates that the gaze following and social gaze losses provide complementary information, and using both can give improved performance. Impact of VSGaze. When comparing the performance of the models trained on VSGaze (Supp Table 4 ) against their versions fine-tuned on individual datasets (Table 3 ), we see that the fine-tuned models always perform better. This is because fine-tuning allows the models to learn dataset specific priors (ex. more LAH cases in VAT, Table 1 ). For instance, on VAT, Gupta* has a distance score of 0.138 when trained on VSGaze, compared to a score of 0.134 when directly fine-tuned on VAT. Also, our model has a distance score of 0.112 when trained on VSGaze, and a score of 0.105 when fine-tuned on VAT. This highlights the challenge in leveraging multiple datasets: while we may expect better performance by having more data, the different priors and statistics bring additional difficulties. Nevertheless, our model trained on VSGaze is able to achieve strong performance across all datasets. Impact of temporal information. Comparing the static and temporal versions of our model trained on VSGaze, we observe improvements in performance for shared attention (0.555 vs 0.576, Tab. 5 in supplementary), and similar or slightly improved performance for other metrics. This is in contrast with Chong T , which often has lower performance than Chong S (for distance, LAH and LAEO metrics in Table 2 ). These results follow observations from prior work (Section 2) and highlight the challenge in leveraging temporal information for gaze following. We provide a detailed analysis in supplementary D.The supplementary (overview in A) provides additional ablations and discussions. 
conclusion: 
 We propose a new framework for multi-person, temporal gaze following and social gaze prediction comprising of a novel architecture and dataset. Through a series of experiments, we show that our model can effectively learn from a mix of video-based datasets with different statistics, to perform gaze following and social gaze prediction without sacrificing performance on any of them. The trained model can then be further fine-tuned on individual datasets to improve performance towards a specific scenario or task. We hope that our proposed framework opens new directions for modeling people's gaze behavior, and are keen to see applications of the proposed dataset and social gaze losses in other methods. In the future, we intend to further investigate the benefits of temporal information and other auxiliary signals, including new ways of incorporating them into the architecture. We also plan to expand the VSGaze dataset to include more samples and annotations. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Potential positive and negative societal impacts of the work have been discussed in Supplementary J.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not think our data and models are high risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original papers for leveraged datasets, code and models have been cited. Appropriate licenses will be included when releasing code and data.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: No assets have been currently released with the paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The supplementary material is organized as follows:• B: details on the construction of VSGaze;• C: breakdown of performance on VSGaze by component dataset;• D: ablations on temporal window length and novel architecture components;• E: implementation details;• F: experiments on incorporating auxiliary information (speaking status);• G: qualitative analysis of predictions;• H: details on our DPT based gaze heatmap decoder;• I: discussion on limitations;• J: discussion on broader impact. Unless specified, all experiments and visualizations in the supplementary use LAH and LAEO predictions obtained via the post-processing strategy, and SA predictions from the corresponding decoder. This is because post-processing gaze following outputs for LAH and LAEO ensures that outputs for these three tasks align. We also observe slightly better performance for LAH and LAEO using the post-processing approach compared to using the predictions from their respective decoders (Table 2 , Ours-PP vs Ours).We provide the detailed protocol for obtaining gaze following and social gaze annotations on VSGaze: Gaze Target Point. For people sharing attention in VideoCoAtt ( 12), we compute their gaze points as the center of the SA object's bounding box. Similarly, for a person pair LAEO in UCO-LAEO (31) , we compute their gaze points as the center of the other person's head bounding box. LAH. We generate LAH annotations for all datasets. To do so, similarly to Tafasca et al. (48) , we check whether the gaze point for an annotated person falls inside any other person's head bounding box. For the GazeFollow test set, at least 2 of the annotated gaze points should fall inside another person's head bounding box. LAEO. We use the provided annotations for UCO-LAEO. For VAT and ChildPlay, we generate LAEO annotations by using the LAH annotations, checking whether the LAH target for a pair of people corresponds to the other person. We cannot obtain LAEO for GazeFollow as most images are annotated for a single person, and for VideoCoAtt because if a person is an SA target, they are not in the set of people sharing attention and their gaze is not annotated. SA. We use the provided annotations for VideoCoAtt. For VAT and ChildPlay, we generate novel SA annotations from the LAH annotations, checking whether two person share their attention to the same third person. We cannot obtain SA for GazeFollow as most images are annotated for a single person, and for UCO-LAEO as a pair of people annotated with LAEO cannot be sharing attention.We provide a breakdown of results on VSGaze by component dataset in Table 4 . Note that following the results in the main paper, LAH and LAEO results for Ours-noGF and Ours are obtained from their respective decoders. We can observe that performance trends on individual datasets can differ from the aggregated results on VSGaze. For instance, although we have a small improvement for LAH compared to the baselines across VSGaze, we perform significantly better on ChildPlay. In general, as VideoCoAtt represents the highest number of samples in VSGaze, it also has the highest impact. Also on ChildPlay, once again we see that better gaze following performance does not translate to better social gaze performance. Although Gupta* has a better distance score compared to Chong S *, it performs significantly worse for all social gaze tasks.We compare performance of our model for different temporal window lengths on VSGaze in Table 5 . Note that T = 1 corresponds to a static model. We observe that incorporating temporal information can improve performance, especially in the case of shared attention. For the other metrics performance remains comparable. As a temporal window of 9 does not necessarily give better performance than a temporal window of 5, we use T = 5 for our experiments. We note that these observations are in contrast to those from the static (Chong S ) and temporal models (Chong T ) of ( 9). As seen in Table 2 , Chong T often performs worse than Chong S , with especially lower scores for the distance and LAEO metrics. This follows prior observations from the state of the art regarding temporal modelling for gaze following (Section 2), and illustrates the challenge in leveraging temporal information.While architecture design may be a reason for the lack of greater improvement in performance, the data itself is an important factor. Firstly, despite the larger number of samples in VSGaze compared to standard video based gaze datasets, there is high redundancy between frames so data diversity is not comparable to that of GazeFollow. Secondly, the moments where temporal information is important, such as during gaze shifts, only form a small percentage of total instances. Hence, improvements for predictions in these moments are not reflected in overall metrics. For instance, gaze shifts form less than 10% of total instances in ChildPlay (48) . However, in our qualitative analysis (Section G) we can see situations where temporal information helps. In future work we plan to investigate new metrics for evaluating the performance of temporal models.We systematically remove different novel components of our architecture to analyse their impact and provide results in ) decreases LAEO and SA performance. Without this encoder, the person tokens cannot access the frame tokens, so they cannot capture the locations of gazed at salient items. As the frame tokens may be able to adapt to this change, gaze following performance is not impacted negatively. Finally, removing the Spatio-Temporal Social Interaction component I b pp , I b pt (Ours-noI ppt ) decreases LAH and SA performance. Without this component, there is no interaction between person tokens, so identification of social dynamics is hindered. Interestingly, we see a boost in LAEO performance. However, a major portion of LAEO positives come from the UCO-LAEO dataset (see Table 1 ) which consists mainly of two person scenes, so capturing social interactions may be less important. DPT Decoder. We replace our proposed modified DPT decoder for gaze heatmap prediction (Section H) with a simpler decoder (Ours-noDPT). This decoder projects the frame and person tokens from the last Interaction block, performs a dot product between them, and then upscales the output to the heatmap resolution. Using the simpler decoder results in drops in performance for the distance, LAEO and SA metrics. Unlike the DPT, it lacks multi-scale representations which impacts heatmap prediction and supervision of tokens. Overall, we observe that removing the components tends to impact shared attention performance. This is similar to the observation in the previous section regarding temporal information. Unlike LAH and LAEO where the target is always another person, SA is more challenging as the shared attention target can be any person or object/point. Hence, this task may be benefitting more from additional information or architecture components.The Gaze Backbone G stat is a ResNet18 pre-trained on Gaze360 (25) , and processes the head crops at a resolution of 224 × 224. We use a ViT-base model (1) V initialized with MultiMAE weights (3) to process the scene at 224 × 224. The temporal position embedding x is zero-initialized, while all other layers are randomly initialized. For training, we use the AdamW optimizer (28) with warmup and cosine annealing. The loss coefficients are set as λ HM = 1000, λ VEC = 3, λ IO = 2 and λ LAH = λ SA = 1. To counter the class imbalance in positive vs negative social gaze labels, we weight positive samples for the social gaze loss by 2. All models were trained on an internal cluster using a single Nvidia RTX 3090 GPU with 24GB memory. Training time for a single experiment is approximately 8 hours on both GazeFollow and VSGaze. Total compute across all experiments is approximately 140 GPU hours.Previous studies on analyzing conversations during meetings have shown that people usually look at the other speaking participants (45) , and such cues can be exploited for gaze target selection . Hence, we expect that identifying speaking persons can provide better scene understanding for gaze following, and help recognize attentiveness towards people, especially speakers. The latter is especially important in autism diagnosis, as eye contact is closely monitored by the clinician when they call out to the tested child (2).To incorporate speaking information in our model, we adapt the Person Module (Section 3). Specifically, we first obtain speaking scores for each person using an active speaker detection model (33) .The model is retrained to detect speakers using the video and with no audio. The obtained speaking scores s i,t for each person are linearly projected to the token dimension using P spk , and added to the person token. Hence, the new person token is obtained as:EQUATIONWe note that this formulation can easily be extended to incorporate other kinds of person-specific auxiliary information such as gestures. We explore this aspect in a follow up work where we extract gestures and other cues using vision-language models (18) . We provide results for incorporating speaking information in Table 7 . On VSGaze, once again we see improvements for SA while other metrics remain similar. As the speaker detection model tends to fail in cases with side-view faces or for children in the case of ChildPlay, we may further benefit from a better speaker detection model. In addition, investigating other ways to better capture and incorporate such auxiliary information in our model is a direction for future research.We provide qualitative results from our models in Figure 5 . We observe that all models perform well, accurately capturing people's gaze targets and social gaze behaviour. We also see that incorporating temporal and speaking information can help improve predictions.In the first sequence, the static model occasionally makes an error for person 1, and picks person 2's hands as the predicted gaze target. This is because it cannot distinguish blinking from when a person lowers their gaze. On the other hand, the temporal models recognize blinking and maintain the target as person 2's face. In the second sequence, the static model misses shared attention between persons 1,2,3 in the first frame and persons 2,3 in the third frame. This sequence is challenging due to the presence of subtle head motions which the temporal models can better capture. In the third sequence, both the static model and our proposed model make an error when predicting person 3's gaze target in frame 2. However, our model with speaking information recognizes person 1 as the target given their high speaking score.In addition, we provide qualitative comparisons of our model against other methods in Figure 6 . We see that our model performs better overall, accurately inferring people's gaze target and social gaze behaviour despite the complexity of the scenes with multiple salient targets, obscured eyes, varied settings and age groups.We perform analysis to understand the difference in performance between post-processing gaze following outputs or using the predictions from the task specific decoders. For LAH, we find that predictions for the two schemes align 87% of the time. On visualizing outputs, we identify that cases where they don't match are usually where the model is confused between two potential targets. So the arg max of the gaze heatmap for obtaining the gaze point picks one target, while the arg max of the LAH scores picks the other target. This confusion in target selection is illustrated for person 1 in Figure 3 . We see that the predicted gaze heatmap highlights both person 2 and 3, and similarly, the LAH scores for looking at both persons 2 and 3 are high.As mentioned in Section 3, we rely on the standard DPT (39) decoder that has been developed for dense prediction, and propose an interesting way to transform it for performing person-conditioned gaze heatmap prediction. Similar to an FPN (27) , the DPT assembles the set of ViT image tokens into image-like feature representations at various resolutions. The feature representations are then progressively fused into the final dense prediction. Specifically, the DPT decoder contains two stages: 1) A Re-assemble stage to construct feature maps at specific resolutions at each block, and 2) a Fusion stage where the feature maps across consecutive blocks are upscaled and combined.To include the person specific information, as represented by the person token in our architecture, we modify the re-assemble stage to filter only the information relevant for the given person (Figure 4 ). More precisely, following the standard DPT, we first project the input frame tokens f o,b t at level b to a lower token dimension using a 1 × 1 conv layer, followed by spatial upsampling or downsampling using a strided 3 × 3 transposed conv layer or conv layer respectively;EQUATION) where Split is the reverse of the spatial concatenation operation, Concat, and Resample is the spatial upsampling/downsampling operation. To condition on a person, we then perform a token-wise hadamard product of the projected frame tokens with the the projected person token (using projection layer P b DPT ) from the corresponding block b.EQUATION) where * denotes the hadamard product. The standard Fusion stage then follows to obtain the predicted gaze heatmap. Note that superscripts DPT, DPT-c denote updates to the frame tokens.As discussed in Supp. G, gaze following outputs and predictions from the social gaze decoders do not always align (13% cases). One possible solution is to post-process gaze following outputs for LAH and LAEO as done in the case of Ours-PP. This ensures that outputs for gaze following, LAH and LAEO align. However, further aligning the outputs with SA is more challenging as post-processing for SA does not account for whether the gaze points fall on the same semantic item. Ensuring that outputs for all social gaze tasks and gaze following align is a challenging and interesting direction for future research. Potential directions include more refined post-processing techniques, or the addition of task consistency losses and regularization. The latter is particularly interesting as it may further increase the benefit of the social gaze losses. Secondly, as discussed in supplementary D, temporal information does not seem to bring large improvements in performance, with improvements observed mainly for SA. Investigating new architectures, datasets and metrics to improve training and evaluation of temporal information is another important and interesting direction for future research.Gaze following and social gaze prediction has strong potential for positive societal impact. Indeed, eye-contact and shared attention are monitored during the ADOS autism diagnostic test for children (2) as a marker for social communication. Given the time consuming nature of clinical tests and large scale prevalence of autism (1 in 36 children ( 29)), automatic tools for autism screening based on gaze can help flag potential autism cases and reduce the burden on doctors (47) . Nevertheless, as with any medical application, such screening tools must be used carefully to avoid missing cases, for instance through human-in-the-loop systems. It is also important to ensure that gaze algorithms do not invade the privacy of people. They should be deployed with appropriate consent and only for specific applications. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will release the data and code upon cleanup. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).38th Conference on Neural Information Processing Systems (NeurIPS 2024).Note that the superscript o, p and s do not represent indices, but intermediate token updates within a block b. 


Decision: This paper proposes a framework for multi-person temporal gaze following and social gaze prediction. All the reviewers recommended acceptance of this paper. The AC checked the author response and the discussions, and agreed to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper focuses on social gaze prediction in videos. An approach based on ViT has been proposed, combining three modules including person module, interaction module, and prediction module. The authors also summarised the current shortcomings of the existing datasets and have introduced a new dataset comprising social gaze interactions such as looking at someone, looking each other, and shared attention.
strength_weakness: strength_weakness: The main weakness of the paper is the results. If you refer to Tables 2 and 3, the differences in terms of distance and other metrics are very small, making it difficult to assess the benefits of the proposed approach. In some cases, the differences in terms of distance range between 0.01 and 0.001. This makes the results questionable in my view. The authors may need to run statistical tests to demonstrate that such differences are significant, or these improvements may be by chance given that the differences are negligible in most cases.

Regarding the methodology, I am not sure if papers like [1] and [2] were available at the time of submitting this paper, but it is hard to see how the proposed approach is competitive compared to these aforementioned approaches in terms of both methodological novelty and performance.

On page 3, the authors mention ".. both methods do not address the gaze following task." However, the proposed method and dataset do not address the gaze following task either. Indeed, the new dataset only includes a subset of the gaze communication cues presented by Fan et al. at ICCV 2019. Why hasn't this dataset been considered in the evaluations given its relevance?

[1] Sharingan: A Transformer Architecture for Multi-Person Gaze Following
[2] A Unified Model for Gaze Following and Social Gaze Prediction
questions: questions: 1. Could you please demonstrate that the differences between the obtained values are significant?

2. Could you please discuss your contributions with respect to the existing transformer-based approaches for gaze following and social gaze estimation?

3. Could you please explain why you selected these methods for comparison specifically, and why you chose the datasets like GazeFollow and VideoAttention, most of which do not even have annotations for social gaze cues?
limitations: limitations: The authors have discussed the limitations and broader impact satisfactorily. In addition, they have provided detailed visualisations to offer insight into failure cases.
paper_summary: paper_summary: This paper presents a novel framework for multi-person temporal gaze following and social gaze prediction. The authors propose an architecture that jointly predicts gaze targets and social gaze labels for all people in a scene. It uses a transformer-based model that processes both frame tokens and person-specific tokens to capture gaze information and interactions. They also build a dataset called VSGaze, which unifies and extends annotations across multiple existing gaze following and social gaze datasets. This allows joint training on multiple tasks and datasets. The proposed model achieves state-of-the-art results for multi-person gaze following and competitive performance on social gaze prediction tasks.
strength_weakness: strength_weakness: - Conversely, the network's structure is relatively straightforward, combining existing approaches, and its novelty as a method is not necessarily significant.
- The model's generalization capability to unknown domains and datasets has not been evaluated. While preparing the data is challenging, it is particularly crucial for methods involving humans.
questions: questions: It seems challenging to clearly state whether the training of models combining multiple tasks is useful even with a small amount of data based on the current results. Is there anything we can say about the relationship between data quantity and joint training? Since the effect of large-scale training with multiple datasets itself is not the core argument of this paper, I think it would be more insightful if this aspect can be carefully separated from the observations.
limitations: limitations: The discussion in the appendix seems to be conducted appropriately.
paper_summary: paper_summary: Paper proposes a novel framework which solves multiple gaze prediction tasks, gaze heatmap, in-out frame classification, social gaze classification for multi-person in one-pass simultanously. It also contributes a new dataset by extending existing datasets annotations. Comprehensive experiments were conducted for multiple datasets (GazeFollow, VideoAttentionTarget, ChildPlay, VideoCoAtt, UCO-LAEO which shows superior results compare to existing methods.
strength_weakness: strength_weakness: Paper's technical contribution is incremental. The framework design, while novel, does not have any significant improvements over prior works.

The contributed dataset is also an extension of existing datasets. However, as pointed out in the rebuttal, the effecient construction process of the dataset is also a strength of this paper.

This is a good paper.
questions: questions: One of the paper's claimed contribution is the temporal aspect of gaze. However, in the proposed model, there is no clear design of this beyond using a simple sequence of frame tokens and temporal information is aggregated with standard attention mechanism. The main discussion of temporal information is relegated to the Appendix. Will the author consider reframing the importance of temporal information in the title and abstract, or to move the temporal information discussion to the main paper?
limitations: limitations: NA
paper_summary: paper_summary: This paper handles the problem of predicting human social labels and gaze heatmap simultaneously for all people in the input image sequence, which improve the accuracy of gaze following based on human social cues. Specifically, they first calculate person tokens through a person module, and then design an interaction module to extract interactive relationship between humans and scene. The author proves the effectiveness of the proposed model on multiple datasets, and verifies the role of each part of the model through ablation experiments. And a new dataset suitable for both subtasks is also annotated.
strength_weakness: strength_weakness: 1. The author lacks an introduction to the Pairwise Instance Generator module in the model. What is its function? There is also a lack of explanation of the number of Interaction Blocks. What impact does the change of B have on the effect of the model?
2. The symbols used in the article are confusing and difficult to understand. For example, “Each block then processes the set of output person tokens $P_{1:N_n,1:T}^{o,b-1}$  and output frame tokens $f_{1:T}^{o,b-1}$  from the previous block”, why not indicate the range of b in this sentence? And too many superscripts and subscripts in formulas and expressions can easily confuse readers in the introduction to Interaction Module.
3. Lack of visual results to prove in which scenarios the proposed model has an advantage over the comparison methods.
questions: questions: 1. Change the name of orange box in the Person Module of fig2 to Temporal Gaze Processor may be better to match the description in the essay.
2. In "$I_{ps}^b$ It is implemented as a single Transformer layer with cross-attention", "it" may be a redundant word.
3. Does  $f_t^{o,b}=V_b(f_t^{p,b})$ in line 198 denote the output frame tokens for the block b, rather than the block B in Person-Scene Interaction?
4. Where is "temporal transformer architecture" in Fig.1?
limitations: limitations: Yes

Comments: title: reply: Rebuttal by Authors
Additional Remarks
comment: Thank you for the response! We appreciate your feedback and positive evaluation of our work.
Thanks for your response. I will keep my rating.
Thank you for the response! We appreciate you raising your score and noting our submission as a well executed paper.
I have raised my score to 'weak accept' after considering all the feedback and responses. However, as another reviewer also noted, I am still not fully convinced about the significant methodological novelty of this architecture compared to previous ones. Nonetheless, this is a well-executed paper, and introducing a new dataset is a plus.
Dear reviewers, I hope our response answered your questions and comments, please let us know if you have any additional concerns. We appreciate your time, thank you!