AL-GTD: Deep Active Learning for Gaze Target Detection

Authors: lorenzo  vaquero
cigdem  beyan
francesco  tonini
nicola  dall'asen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 14:07:35

Abstract: Gaze target detection aims at determining the image location where a person is looking.
While existing studies have made significant progress in this area by regressing accurate gaze heatmaps, these achievements have largely relied on access to extensive labeled datasets, which demands substantial human labor.
In this paper, our goal is to reduce the reliance on the size of labeled training data for gaze target detection. To achieve this, we propose AL-GTD, an innovative approach that integrates supervised and self-supervised losses within a novel sample acquisition function to perform active learning (AL).
Additionally, it utilizes pseudo-labeling to mitigate distribution shifts during the training phase. AL-GTD achieves the best of all AUC results by utilizing only 40-50% of the training data, in contrast to state-of-the-art (SOTA) gaze target detectors requiring the entire training dataset to achieve the same performance. 
Importantly, AL-GTD quickly reaches satisfactory performance with 10-20% of the training data, showing the effectiveness of our acquisition function, which is able to acquire the most informative samples.
We provide a comprehensive experimental analysis by adapting several AL methods for the task. AL-GTD outperforms AL competitors, simultaneously exhibiting superior performance compared to SOTA gaze target detectors when all are trained within a low-data regime.
Code is available at https://github.com/francescotonini/al-gtd.

Main paper: 
 introduction: 
 Human communication relies on a range of multimodal cues, including speech and gestures. Among these cues, the gaze holds significant importance as it reveals a person's visual focus of attention, enabling us to comprehend the interests, intentions, or (future) actions of individuals [21] . Gaze analysis has been extensively utilized across various fields, including human-computer interaction [6, 49] , neuroscience [16, 52] , social robotics [1] , and social and organizational psychology [9, 19] . Automated gaze behavior analysis has been addressed through two tasks: gaze estimation and gaze target detection. Gaze estimation involves discerning the direction of a person's gaze, usually within a 3D space, when provided with a cropped human head image as input. This encompasses estimating the horizontal and vertical angles of the gaze, as well as determining the depth or distance at which the gaze is focused [11, 31, 36] . On the other hand, gaze target detection (or gaze-following) is the process of identifying the specific point or area within a scene that a person is looking at [15, 33, 60] . This involves analyzing visual cues such as head orientation, and other contextual information to determine the focal point of a person's gaze [4, 14, 15, 23, 34, 41, 47, 53, [59] [60] [61] .In this study, we focus on the gaze target detection task. Our multimodal gaze network, namely GTN, integrates the head crop of the person of interest with RGB and depth data. We employ two separate backbones for processing scene RGB and depth data to extract meaningful features indicating areas of interest. A third backbone is used for head processing to predict an attention map projected into the scene features, which identifies image areas where the person is more likely to look. Then, the fusion of head, scene and depth features generates the final gaze heatmap, centered on the person's visual focus of attention.The training and evaluation of the gaze target detection task are carried out using manually human-annotated data. In certain AL-SSL [20] Tonini et al. [59] Tonini et al. [60] Tu et al. [61] Figure 1 : The performance of our AL-GTD compared to counterpart active learning approach AL-SSL [20] and SOTA gaze target detectors: Tu et al. [61] and Tonini et al. [59, 60] on the GazeFollow dataset [54] . Our method consistently performs better than competitors and achieves SOTA AUC performance with half of the training data, proving the effectiveness of our acquisition function.instances, collecting a gaze target detection dataset proves to be challenging, leading to low consensus among annotators, as shown in [47, 60] . Moreover, manual labeling is a highly tedious and timeconsuming process. Some studies have reported labor times ranging from ten seconds to one minute for every second of gaze data [3, 17] .Another study [22] revealed that an experienced human annotator may require two to more than ten times the duration of a video to accurately label the gaze. Unfortunately, the performance of stateof-the-art (SOTA) gaze target detectors, especially those relying on a Transformer architecture [60, 61] , mainly depends on the size of the labeled training set.In this paper, our objective is to explore solutions for reducing reliance on the size of labeled training data in gaze target detection. In essence, given our method GTN, we aim to find ways to achieve accurate performance even with a smaller amount of labeled training data. The key is to effectively select a subset of the data that optimizes model training. In this vein, researchers have delved into various strategies for choosing the most informative samples in a dataset for labeling, a methodology commonly referred to as Active Learning (AL). Our study focuses on AL's pioneering application in gaze target detection. We opt to keep our gaze detector relatively simple (e.g., we do not include body pose and scene point clouds [4] that can potentially improve the results, or perform extra tasks like retrieving the gazed-object's location [60] ), yet we still prove our GTN's effectiveness, particularly in a low data regime.Usually, AL is achieved by developing a scoring function, i.e. the AL acquisition function, that selects the most informative samples, which are the ones contributing the most to the effectiveness of the model. This function, for example, can prompt the labeling of samples for which the network exhibits the highest uncertainty, indicating its lowest confidence in predictions [5, 26, 38, 58, 65] .Other popular AL acquisition functions are based on diversity [18, 56] or ensemble of multiple learning models [5] . Instead, we present a novel AL acquisition function, which is integrated into our GTN.Our method, named AL-GTD, utilizes three metrics to assess the informativeness of the network predictions on unlabeled samples. These scores measure (a) the discrepancy and (b) scatteredness between the network's intermediate attention map and predicted gaze heatmap, as well as (c) the scene content described by an object detector. To further enhance the performance, we incorporate a selfsupervised learning (SSL) strategy that measures the consistency of network predictions against image augmentations. Furthermore, inspired by [20] , we include pseudo-labeling in our pipeline, and automatically annotate unlabeled samples using the prediction of GTN. Pseudo-labeling enables us to expand the training dataset without incurring additional labeling costs while also handling distribution shifts resulting from the acquisition function's objective.Given that this study marks the initial endeavor of integrating gaze target detection within AL, we also incorporate several existing AL acquisition functions applied to other tasks to conduct a comparative study to evaluate the effectiveness of our AL-GTD. To this end, we leverage the gaze target detection datasets GazeFollow [53] and VideoAttentionTarget [15] to benchmark the AL methods: Entropy [57] , MC-Dropout [26] , Learning Loss [65] , VAAL [58] , AL-SSL [20] and our AL-GTD. Additionally, we refine the uncertainty definition introduced in UnReGa [8] , which is used to improve source-free gaze estimation by minimizing uncertainty, and utilize these refinements as an AL acquisition function, establishing another novel AL methodology for gaze target detection.Experiments show that AL-GTD can reduce the reliance on the size of labeled training data. For example, it can perform SOTA AUC results with only 50% of the GazeFollow [53] dataset's training split. Notably, it demonstrates promising results, i.e., nearing SOTA AUC, even when trained on only 20% of the full training set, indicating its proficiency in selecting highly informative training samples. Furthermore, our AL-GTD remarkably surpasses the SOTA gaze target detectors [59] [60] [61] when all are trained within a low-data regime (see Fig. 1 reflecting all these results). Comprehensive analyses confirm that our AL-GTD consistently outperforms other AL approaches and the ablation study proves the importance of each component of the proposed method.The main contributions can be summarized as follows.(1) This paper is the first attempt to achieve effective gaze target detection under the condition of limited training data.(2) We introduce a novel deep AL approach for gaze target detection, outperforming its competitors and yielding robust results with fewer labeled training data. Our method achieves SOTA AUC performance by utilizing significantly less labeled training data, which is a performance level reached by [59] [60] [61] only when trained on the complete training dataset.(3) We benchmark AL for gaze target detection by re-purposing several SOTA AL methods. We also present an additional AL acquisition function different from the proposed method by adjusting the approach in [8] . Such evaluations can be used as a reference by researchers interested in tackling the same task in the future. OD extracts relevant objects 𝑂 in the scene from 𝐼 𝑅𝐺𝐵 , while 𝐺𝑇 𝑁 processes both 𝐼 𝑅𝐺𝐵 and 𝐼 𝐷 of the scene and the crop of the head of the person of interest 𝐼 𝐻 . From 𝐺𝑇 𝑁 , the attention map 𝑀 𝐴 and gaze heatmap 𝐻 𝐺 are obtained. The outputs of OD and 𝐺𝑇 𝑁 are used to build the acquisition function (Eq. 6), composed of the objectness Γ, the scatteredness Σ, and the discrepancy Δ scores. The oracle annotates the most informative samples, while those with the lowest scatteredness (Eq. 7) are pseudo-labeled. Both the manually labeled samples by the oracle and the pseudo-labeled samples are added to the pool 𝐿 𝑖+1 , and 𝐺𝑇 𝑁 is trained on the updated set. This process is repeated for a fixed number of iterations 𝑁 until the exhaustion of the labeling budget 𝛽. 
related work: 
 Gaze Target Detection. This task aims to determine the location that a person is looking at in a scene captured from a third-person perspective. The typical output of this task is a 2D heatmap, indicating the probability of where the person is looking in the scene while the gaze target is determined by the pixel coordinates corresponding to the highest value in the heatmap. To this end, [53] introduced the GazeFollow dataset and proposed a two-pathway approach. In that approach, one CNN-based pathway concentrates on extracting features from the head crop of the individual whose gaze is to be detected, while the other CNN-based pathway extracts features from the RGB scene image, with subsequent fusion of these features. The aforementioned two-pathway methodology is further advanced in [15] , integrating spatiotemporal modeling for video-based gaze target prediction as well as introducing VideoAttentionTarget dataset. This concept is refined in subsequent works [4, 23, 34, 47, 59] , some of which introduce a third pathway to incorporate the depth map of the scene image, which is estimated using a monocular depth estimator [23, 34, 47, 59] . Recently, Transformer-based methods are introduced [60, 61] . Such methods can simultaneously detect the gaze targets of multiple individuals in a scene. However, these models rely on larger training datasets, leading to performance degradation in low-data scenarios. To tackle this issue, we introduce AL-GTD, an innovative multimodal (RGB + depth) gaze target detection model specifically designed to facilitate Deep Active Learning in settings constrained by the amount of labeled training data.Deep Active Learning. AL aims to optimize labeling costs by iteratively annotating (i.e., querying a human) [48] only those most informative samples from a large pool of unlabeled data. In recent years, special attention has been given to AL for a variety of tasks, with works addressing image classification [5, 10, 18, 25, 26, 28, 37-39, 48, 56, 58] , object detection [13, 20, 35, 45, 55, 63] , semantic segmentation [7, 27, 30, 37, 64] , pedestrian detection in videos [2] , person re-identification [43, 62] , human hand pose estimation [29] , video action detection [50] , and temporal action localization [32] .These works can be categorized based on their acquisition function as: i) uncertainty-based, ii) diversity-based, or iii) committeebased, with also the possibility of hybrid methods [13, 20, 45, 47] . Uncertainty-based methods primarily depend on entropy [13, 20, 32, 45, 63, 64] . A clustering-based selection criterion was proposed in [50] to ensure diversity across samples while [65] introduced a Learning Loss module to predict the losses of unlabeled samples, which was later refined in subsequent works [10, 37] . On the other hand, Gal et al. [25, 26] presented deep Bayesian AL, which involves training a model with dropout layers and employing Monte Carlo dropout to approximate the sampling from the posterior. This method later became a mainstream baseline, as evidenced in [5, 13, 18, 20, 28, 38, 39, 45, 48, 56, 58] . Furthermore, Sinha et al. [58] integrated a Variational AutoEncoder (VAE) and a discriminator to establish an AL metric. Committee-based models, where ensembles are typically employed for sample set selection, also utilize a separate task classifier trained in a fully supervised manner [5, 29, 39, 48] . Recently, [20, 45] introduced a robustness score measuring an image's consistency and its augmented version. Using pseudo-labels is also very beneficial, as seen in [20] . However, the primary focus of such solutions lies in the classification head, overlooking regression problems such as gaze target detection. It is imperative to highlight that the domain of AL for gaze target detection remains unexplored. Gaze target detection aims to predict the image coordinates where a person is looking in a scene captured from a third-person perspective. As outlined in previous works [23, 34, 47, 59 ], multimodal information (e.g. the depth map) substantially enriches the gaze target detection performance. To accomplish this, we define a model 𝐺𝑇 𝑁 that is fed with an input I = {𝐼 𝑅𝐺𝐵 , 𝐼 𝐷 , 𝐼 𝐻 }, where 𝐼 𝑅𝐺𝐵 ∈ R 𝑊 ×𝐻 ×3 and 𝐼 𝐷 ∈ R 𝑊 ×𝐻 represents the RGB and depth image of the scene, and 𝐼 𝐻 ∈ R 𝑤×ℎ×3 denotes a 𝑤 ×ℎ crop centered on the person's head whose gaze is subject to be predicted. The output 𝐻 ∈ R 𝑊 ×𝐻 is the predicted gaze heatmap, where higher activation areas correspond to the area where the person is looking.Our gaze target detector 𝐺𝑇 𝑁 works within active learning such that, given an unlabeled dataset 𝑈 and a labeling budget 𝛽, the objective is to iteratively select the most informative samples from 𝑈 and annotate them using an oracle (i.e., human) to create a new labeled set 𝐿 that maximizes the performance of 𝐺𝑇 𝑁 . Initially, a small random set of samples 𝐿 0 ∈ 𝑈 are labeled and are used to train a prior model 𝐺𝑇 𝑁 0 . Following this, 𝐿 is progressively populated over 𝑁 AL iterations. Thus, for each iteration 𝑖 ∈ [1, 𝑁 ], 𝐺𝑇 𝑁 𝑖 -1 processes 𝑈 𝑖 -1 = 𝑈 \ 𝐿 𝑖 -1 and the predictions are evaluated by an acquisition function to determine the most informative samples, which are then labeled by an oracle. Meanwhile, we also pseudolabel the samples with the highest confidence, assuming that they are less informative samples or, in other words, the ones our 𝐺𝑇 𝑁 can perform accurate gaze predictions. Both pseudo-labeled and human-annotated samples are added to the labeled set 𝐿 𝑖 . Finally, a new model 𝐺𝑇 𝑁 𝑖 is trained on 𝐿 𝑖 , and the process is repeated until the labeling budget 𝛽 is exhausted. This method, namely, AL-GTD, integrates supervised and self-supervised losses within a novel AL acquisition function to enable the usage of our gaze target detector under low-data regimes. An overview of AL-GTD is depicted in Fig. 2 , along with the associated pseudocode. Detailed explanations of its components, including 𝐺𝑇 𝑁 (Fig. 3 ), are provided below. 
our gaze target detector: 
 To make the model more robust and impose a self-supervised consistency, which is used during AL, we also define a set of augmented inputs of I as A = A (I), with A being standard augmentations (e.g. random cropping, horizontal flipping, contrast/brightness changes, etc...). I is processed by different branches which interact with each other to generate the output 𝐻 . The scene branch S processes the image 𝐼 𝑅𝐺𝐵 while the depth branch D processes the depth map 𝐼 𝐷 . Both inputs are enriched with the head mask 𝐼 𝑀 , which encodes the head's position in the image. The crop of the head of the person of interest 𝐼 𝐻 is processed by a separate head branch H , which extracts high dimensionality features 𝑓 𝐻 about the head pose and gaze of the person. Furthermore, a separate module 𝐷 𝐴 projects the concatenation of 𝑓 𝐻 and 𝐼 𝑀 into the attention map 𝑀 𝐴 , which encodes the probability of areas of the scene being gazed at by the person with only the head features and its position in the image. The RGB and depth features of the scene, denoted as 𝑓 𝑆 and 𝑓 𝐷 , respectively, are combined with 𝑀 𝐴 to prioritize areas where the gaze is likely to occur. These features are then processed by parallel encoders 𝐸 𝑆 and 𝐸 𝐷 , projecting scene and head cues into two latent spaces. Finally, a decoder 𝐷 𝐻 fuses the latent spaces above and produces the gaze heatmap 𝐻 𝐺 focused on the person's gaze point.Our AL pipeline comprises a novel acquisition function (Sec. 3.2.1), which evaluates the objectness determined by leveraging an object detector, the scatteredness of heatmap activations, and the discrepancy between 𝑀 𝐴 and 𝐻 𝐺 . Additionally, we incorporate pseudo-labeling (Sec. 3.2.2) and self-supervised learning (Sec. 3.2.3) into our approach.Recall that the objective of an acquisition function is to identify which samples are the most informative and can enhance the network's predictions. AL literature [13, 20, 32, 45, 63, 64] shows that high entropy samples (representing lowconfidence) are expected to provide more information than low ones. However, in the context of gaze target detection, highly confident low entropy predictions can result in incorrect gaze heatmaps, particularly in complex scenes where multiple people and objects exist. Conversely, images with low confident predictions and high entropy may indicate poor model robustness and, if labeled, they might not bring new information to the existing network. Therefore, an acquisition function for gaze target detection must address several perspectives. We propose three properties that determine the informativeness of images to be labeled, described as follows.Objectness. Inspired by studies demonstrating that individuals often gaze at living or non-living objects during social and physical interactions [12, 40, 46] , we incorporate the use of an object detector OD under the assumption that it could enhance the informativeness of samples for gaze target detection within AL. Due to how the scene branch S is pre-trained, elements in the scene can influence 𝐷 𝐻 to predict the gaze heatmap onto objects. This behavior potentially leads to poor generalization, especially when many objects are visible but the person is looking elsewhere. We empirically find that objects in the foreground shift the activation of the gaze heatmap toward the object's center, ignoring cues provided by the attention map. This detector takes the scene image as input and predicts a set of objects 𝑂 = OD (𝐼 𝑅𝐺𝐵 ). Each object prediction 𝑜 ∈ 𝑂 consists of a bounding box 𝑏 = 𝑐 𝑥 , 𝑐 𝑦 , 𝑤, ℎ, 𝑘 , with 𝑐 𝑥 , 𝑐 𝑦 , 𝑤, ℎ denoting the center coordinates, width, and height of the object, respectively, while 𝑘 ∈ R 𝐾 represents the probability distribution across the 𝐾 classes considered. To discourage the network from taking shortcuts into foreground objects, we detect objects being targeted by the gaze heatmap and calculate the max confidenceEQUATIONwhere 𝑂 𝐻 ⊆ 𝑂 is the set of objects such that the peak activation 𝑃 𝐻 of the heatmap 𝐻 𝐺 lies inside their bounding boxes, 𝑐 𝑜 is the confidence of the object, and 1 is the indicator function of 𝑂 𝐻 . The objectness score is calculated as the maximum confidence across multiple augmentations of the same scene, i.e.:EQUATIONwhere (•) 𝑎 represents the prediction for sample 𝑎.Heatmap activation scatteredness estimation. The peak activation points of the gaze heatmap 𝐻 𝐺 should ideally be densely positioned on a small region, with values progressively increasing as we reach the peak of the heatmap. Conversely, sparse peak activation points indicate network uncertainty. Let 𝜌: R 𝑛×𝑛 → R 𝑛 2 ×2 be the function that takes a positive-defined matrix and ranks in descending order its elements and returns the coordinates of the cells. We discretize the heatmap into 𝐵 bins, obtain the heatmap peaks coordinates in descending order as 𝜋 = 𝜌 (•), and compute the scatteredness of the activations as:EQUATIONwhere 𝜋 1 is the maximum activation point of the heatmap and 𝜋 𝑝+1 are the coordinates of the 𝑝-most highest and farthest activation point from 𝜋 1 . The scatteredness function iteratively finds the highest 𝑃 activation points farthest from the peak and calculates their distance from it. The final scatteredness score is calculated as:EQUATIONwith high values indicating network uncertainty in at least one of the augmentations 𝑎.Discrepancy between attention map and gaze heatmap. The difference between the attention map 𝑀 𝐴 and gaze heatmaps 𝐻 𝐺 is a useful measure of model uncertainty. In effective gaze target detection networks, 𝑀 𝐴 and 𝐻 𝐺 must agree in the direction (orientation) and location of gaze, with the latter being a more refined version of the former. To estimate the differences between them, we compute the distance among the peak activation points 𝛿 (𝑀 𝐴 , 𝐻 𝐺 ) = ||𝑃 𝐴 -𝑃 𝐻 || 2 , where 𝑃 𝐴 and 𝑃 𝐻 are the positions in the 2D image space of the highest activation point of the attention map and gaze heatmap, respectively.To address model robustness, we estimate the disagreement across multiple augmentations of the same scene and calculate the maximum distance among augmentations as:EQUATIONInformative samples have a high value for Δ(A), and labeling them allows the network to reduce the discrepancy between the saliency map and gaze heatmap.The three above criteria are aggregated to identify those samples whose attention maps differ from the predicted gaze heatmap, that contain objects with high confidence included in the heatmap, and whose activation points are sparse. The final 𝑆𝑐𝑜𝑟𝑒 of a sample is calculated as:EQUATIONwhere 𝜆 1 , 𝜆 2 , and 𝜆 3 are the learnable weights.After assigning scores to all samples in the unlabeled set, we label the top 𝛽/𝑁 samples with the highest scores. The process is repeated for 𝑁 active learning iterations.We contend that it is essential for the gaze network to come across representative and easily detectable samples in order to avoid distribution shifts. At the same time, our goal is to avoid labeling confident samples to save labeling resources. As a solution, we suggest utilizing a pseudo-labeling technique, where the network trained in the previous AL cycle generates pseudo-labels for the network currently undergoing training. We calculate a labeling score for each unlabeled sample asEQUATIONwhere 𝐻 𝑃 𝐻 𝐺 is the peak value of the gaze heatmap, and we pseudolabel samples with the highest scores up to a fixed percentage per iteration.The network is endto-end trained on both labeled and pseudo-labeled samples by minimizing the Mean Squared Error loss between the predicted heatmap 𝐻 𝐺 and H𝐺 ∼ N ( P 𝐻 , Σ), with P 𝐻 being the ground-truth/pseudolabeled gaze point and Σ being a positive covariance matrix. In addition to this supervised loss, we incorporate also a self-supervised loss to ensure consistency across multiple augmentations A of the input. The self-supervised learning (SSL) between augmentations of the same image allows us to increase the robustness of the network and reduce the probability of assigning a high 𝑆𝑐𝑜𝑟𝑒 (A) to easy, non-informative samples, whose discrepancy and scatteredness scores are high because of poor network generalization. We define the consistency loss 𝐿 𝑐 among predictions as 𝐿 𝑐 (𝑎, 𝑎 ′ ) = ||𝐻 𝑎 𝐺 -A -1 (𝐻 𝑎 ′ 𝐺 )|| 2 where A -1 is the inverse augmentation function that aligns the predictions and allows us to effectively calculate the localization error among augmentations. After predicting the gaze heatmap 𝐻 𝐺 for all 𝑎 ∈ A, we compute the supervised loss with the ground-truth gaze heatmap H𝐺 as𝐿 ℎ (𝑎) = ||𝐻 𝑎 𝐺 -H𝑎 𝐺 || 2 .We then compute the total loss as:EQUATION)Random MC-Dropout [26] LL4AL [65] UnReGa [8] AL-SSL [20] Entropy [57] VAAL [58] Figure 4 : Comparisons among AL methods on the GazeFollow [53] dataset. Left: Area Under the Curve (AUC) of the predicted gaze heatmap w.r.t. the ground truth (GT). Center and right: average and minimum distance between GT and the predicted point. Our method, AL-GTD, consistently surpasses random sampling and other AL methods, demonstrating superior performance even with a small initial training dataset (3.7K samples, ∼3% of the original train split).Random MC-Dropout [26] LL4AL [65] UnReGa [8] AL-SSL [20] Entropy  
experiments: 
 We evaluate all methods using the standard gaze target detection GazeFollow [53] and VideoAttentionTarget [15] . Gaze-Follow is a large-scale image dataset comprising over 122K images, featuring annotations for gaze and head locations for more than 130K individuals. On the other hand, VideoAttentionTarget consists of YouTube video clips, each up to 80 seconds long, with 110K frame gaze annotations and their corresponding head locations. For this dataset, we follow the common practice of sampling one image every 20 frames [4, 15, 23, 34, [59] [60] [61] . To measure the performance of the algorithms, we use the standard evaluation metrics for gaze target detection [14, 15] . The Area Under the Curve (AUC) evaluates the confidence of the predicted gaze heatmap w.r.t. the ground truth (GT), while Distance (Dist.) represents the L 2 distance between the GT gaze point and the location on the predicted gaze heatmap with maximum confidence. For GazeFollow, it is customary to report both the minimum and average distances, while for VideoAttentionTarget, only the average distance is reported due to the presence of a single GT point during evaluation.The scene S and depth D branches consist of a ResNet50 pretrained on RGB and depth scene classification [66] , respectively.The object detector OD is based on DETR [67] and pretrained on the COCO [42] dataset. Following earlier works [4, 23, 34, 59, 60] , we employ a monocular depth estimation network [51] to extract depth maps from the input images. The head branch H is built upon a ResNet50 backbone pretrained on the Eyediap dataset [24] and includes a feature projector 𝐷 𝐴 to produce the attention map 𝐻 𝑐 from head features 𝑓 𝐻 . We train S, D, and H , as well as 𝐷 𝐴 , 𝐸 𝑆 , 𝐸 𝐷 , and 𝐷 𝐻 with an Adam optimizer and a 2.5 × 10 -4 learning rate while keeping OD frozen. On GazeFollow [53] , we bootstrap our prior model 𝐺𝑇 𝑁 0 with 3.7K randomly sampled images from the training set (∼3% of the original training split). Subsequently, we conduct five AL iterations using AL-GTD, where we label 2.5K images (∼2% of the original training split) and pseudo-label 2.5K additional images per iteration. On VideoAttentionTarget [15] , we bootstrap 𝐺𝑇 𝑁 0 with 932 randomly selected images and label 932 frames per iteration. We perform four AL iterations and use the pseudo-labeling setup of the GazeFollow experiments.Other Methods. We compare AL-GTD with random sampling as well as AL methods such as Entropy [57] , LL4AL [65] , VAAL [58] , AL-SSL [20] , MC-Dropout [26] , and the committee-based UnReGa [8] . The implementation details of the baselines are given in Supp. Mat. All baselines are trained and evaluated using the same configurations employed for AL-GTD, including the optimizers, learning rate, AL budget, and the initial subset randomly selected for training. At the start of each AL iteration, we reload the initial pre-trained weights to encourage the model to generalize on the updated data distribution, following common implementation, e.g., [13, 20, 45] .The main objective of this work is to decrease reliance on extensive amounts of labeled training data for gaze target detection. Our method, AL-GTD, accomplishes SOTA results of 92.2% AUC for GazeFollow dataset [53] using only 50% of the total training samples, rather than utilizing the entire training dataset, as in [59] [60] [61] . This is also illustrated in Fig. 1 in detail. Furthermore, this percentage decreases to 40% of the training set for VideoAttentionTarget dataset [15] , in which AL-GTD achieves 93.5% AUC, once again Table 1 : Ablation study on GazeFollow [53] in terms of AUC (best in bold) for the effect of SSL, and the components of the AL acquisition function: objectness (Γ), discrepancy (Δ), and scatteredness (Σ). We also report the results of random sampling and AL-SSL [20] for the reader's reference. Note that the cycle with 3.7K samples represents the initial training, where no AL is applied. Therefore, the results are equal for all (𝐴𝑈 𝐶 = 82.64 ± 0.00). AL-GTD against its counterparts and random sampling for GazeFollow [53] and VideoAttentionTarget [15] , respectively. These results together with the number of parameters and the elapsed times for training and inference, are available in tabular format in the Supp. Mat. The results demonstrate that AL-GTD consistently outperforms the random sampling and all other methods even from the initial cycle of AL in which the used training data is really small. Overall, the second-best performing method is AL-SSL [20] . While it is challenging to pinpoint a clear winner between AL-SSL [20] and other methods, AL-SSL [20] tends to outperform them as the size of the training data increases. However, it never manages to surpass our method, as evidenced in Fig. 1 . On average, the worst performing methods are VAAL [58] and LL4AL [65] .The contributions of the components of our AL acquisition function (i.e., discrepancy Δ, scatteredness Σ, and objectness Γ) and SSL are reported in Tab. 1 in terms of AUC for the GazeFollow dataset. Refer to the Supp. Mat. for the Dist. results, which back up the conclusions drawn from the AUC-based analysis. Our first objective is to understand the effect of SSL by removing it from our pipeline (Row 1 vs. Row 6), which results in a decrease in performance (sometimes even slightly more than 2% AUC) in every AL cycle. We then assess the impact of each acquisition component by removing them one at a time while keeping SSL (Rows 2-4). In these instances, the results reveal that there is no clearly superior criterion among the three. However, on average, the removal of the objectness (Γ) criterion results in a slightly greater decrease in performance compared to the removal of the others one at a time. Consequently, we also tested retaining the SSL and Γ criteria while removing all other criteria (Row 5). Also, in this case, performance drops in every AL cycle. Therefore, we can conclude each criterion contributes importantly and using them together yields the best gaze target detection results. Furthermore, it is noteworthy that all combinations still outperform the runner-up method [20] and random sampling.The effect of pseudo-labeling. Tab. 2 presents the performance of AL-GTD across various numbers of samples pseudolabeled. These results correspond to 13.7K manually annotated samples from the GazeFollow dataset [15] , while the percentages for pseudo-labeling are relative to the full size of the training data. One can observe that pseudo-labeling overall contributes to performance enhancement. For instance, with 0.1% or 2% pseudo-labeling, improvements are noticeable compared to not using pseudo-labeling. Conversely, increasing the percentage of pseudo-labeling, such as up to 5%, may lead to performance decreases in the low data regime training. Based on these findings, we choose to conduct our main experiments with 2%, which is equal to the amount of data we request human annotators to manually label. Furthermore, as explained in Sec. 3.2.2 and Eq. 7, the samples to be pseudo-labeled are chosen based solely on the Σ criterion, which has been empirically found to be the most effective in lowdata regimes. The corresponding results are provided in Tab. 3, indicating that when we use a 𝑆𝑐𝑜𝑟𝑒 (A) equivalent to the criterion for selecting samples to be manually labeled or replace Σ in Eq. 7 with Γ or Δ, the AUC values decrease.The effect of SSL. We injected our SSL to Entropy [57] , Un-ReGa [8] and MC-Dropout [26] and compare them with SSL-based methods: AL-GTD and AL-SSL [20] in Tab. 4 in terms of AUC. One can observe that it is possible for SSL-based Entropy, MC-Dropout and UnReGa to surpass AL-SSL [20] while AL-GTD still performs the best out of all. The results from other metrics and AL cycles consistently confirm the performance of AL-GTD (see Supp. Mat.). The corresponding results demonstrate that Transformer-based models [60, 61] consistently exhibit remarkably lower performance across all metrics and subset sizes compared to AL-GTD. Such differences are attributed to the high volume of training data required by Transformers. Instead, CNN-based SOTA [59] demonstrates better performances compared to [60, 61] , although it still falls short of AL-GTD's performances. We assert that while all other methods may encounter challenges with limited training data, AL-GTD's SOTA AUC performance with 50% of the training data represents an important success. Such achievement is attainable by others only with access to the full training dataset. In terms of Dist. metrics AL-GTD trained with 50% of the training data surpasses [59] 's performance when trained on the full training set. However, all methods require more data to achieve significantly low Dist. scores.We also investigate the quality of the data selected by AL-GTD by comparing the performance of [59] on an equal amount of randomly selected data (shown with ★ in Tab. 5). Such results demonstrate that the data selected by AL-GTD is more informative for [59] , showing that AL-GTD can be used for training data curation. 4.2.6 Qualitative Results. We visualize gaze heatmaps generated by AL-GTD, the best-performing AL counterpart [20] , and random selection in Fig. 6 . For additional qualitatives, refer Supp. Mat.  
discussions and conclusions: 
 We have presented AL-GTD, a novel multimodal gaze target detection model that incorporates active learning (AL). Our proposal achieves SOTA AUC performance with just 40-50% of the usual training data and is competitive w.r.t. the current best-performing models even when trained on a mere 10-20% of the full training dataset. This highlights the ability of AL-GTD to effectively choose highly informative samples during training. AL-GTD integrates supervised and self-supervised losses (SSL) within a novel AL acquisition function while also employing pseudo-labeling to effectively mitigate potential distribution shifts during training. It demonstrates superior performance over all SOTA AL methods, even when those AL methods are combined with the same SSL strategy and pseudo-labeling employed by ours. This shows the effectiveness of our primary technical novelty, the AL acquisition function, in enabling rapid learning.Despite setting a new SOTA in gaze target detection within a reduced labeled data setting, we still believe that the Distance metric can be improved. One way to tackle this is by integrating a loss function specifically optimized for calculating the distance between the ground truth and predicted gaze points. An effective strategy could entail estimating the peak from the predicted heatmap in a differentiable manner [44] and computing the 𝐿1 distance w.r.t. the ground truth. Furthermore, Transformer-based gaze target detectors have shown limited effectiveness in low-data scenarios, as demonstrated in our study. Surprisingly, there are currently no Deep AL methods that utilize Transformer-based backbones. A potential avenue for further research could involve replacing the 𝐺𝑇 𝑁 in AL-GTD with a Transformer-based gaze target detector, incorporating additional modules to enhance the Transformer's performance in low-data environments. Lastly, while our gaze target detection model incorporates both scene features and depth map features equally to establish the scatteredness and discrepancy criteria, we aim to further capitalize on the consistencies between these modalities. We will investigate the definition of explicit rules that govern the relationships across these different modalities, aiming to incorporate these rules into our AL acquisition function for improved performance. 
 
 Appendix:  


Decision: All the reviewers are satisfied with the response. The final paper should incorporate the author's valid responses. I am delighted to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: The paper presents a novel approach to gaze target detection that aims to reduce reliance on extensive labeled datasets. The proposed method, AL-GTD, integrates supervised and self-supervised learning with a novel sample acquisition function for active learning. The approach also employs pseudo-labeling to handle distribution shifts during training. AL-GTD achieves state-of-the-art performance using only 40-50% of the training data required by existing methods. The paper provides a comprehensive experimental analysis, demonstrating that AL-GTD outperforms other active learning methods and traditional gaze target detectors in low-data regimes.
strength_weakness: strength_weakness: 1. This work is a meaning attempt to achieve effective gaze target detection under the condition of limited training data.
2. AL-GTD achieves SOTA performance using significantly less labeled data compared to existing methods, demonstrating efficiency in data usage.
3. The paper is well-written and easy to follow. The diagrams illustrate motivations and the details well.
limitations: limitations: 1. Please provide a detailed explanation of the structure of GTN in the Method section, such as the specific network architecture of the three backbones S, H, and D.
2. It is recommended to include more gaze heatmap visualizations and qualitative analyses to validate the effectiveness of the proposed method and compare it with other SOTA methods. These analyses could be included in the main text or supplementary materials.
3. Please cite some of the latest gaze / visual attention studies from 2024.
[AAAI2024] Gaze Target Detection by Merging Human Attention and Activity Cues 
[CVPR2024] Learning from Observer Gaze: Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition
4. Some formulas lack punctuation, such as formulas (2), (3), (4), (5), and (6).
If the authors address my concerns, I would be happy to raise my score.
paper_summary: paper_summary: This paper proposes a novel active learning strategy to reduce label dependency on training data for gaze estimation. The proposed framework utilizes pseudo labelling technique to mitigate the distribution shift in the data.
strength_weakness: strength_weakness: The active learning framework seems promising as it has the potential to address the label based noise usually present in the gaze data.

The experiments have been conducted on GazeFollow and VideoAttentioTarget datasets. The proposed algorithm reaches SOTA with only 50% of the training data.

The quantitative analysis and ablation studies are thorough. It shows that the ssl strategy, discrepancy, scatterness and objectiveness of the proposed pipeline plays crucial role in overall performance.
limitations: limitations: Just a minor point, figure 6 is difficult to understand. Please put a gaze vector or any other way to show the concerned subject and subject specific target.
paper_summary: paper_summary: The paper introduces AL-GTD, a novel approach for gaze target detection that reduces reliance on large labeled datasets by integrating supervised and self-supervised losses within an innovative sample acquisition function for active learning. AL-GTD achieves superior performance compared to state-of-the-art gaze target detectors, using only 40-50% of the training data, and quickly reaches satisfactory results with just 10-20% of the data, highlighting the effectiveness of its acquisition function in selecting the most informative samples.
strength_weakness: strength_weakness: The proposed methodology addresses the challenge of relying on large labeled datasets by combining supervised and self-supervised losses within an innovative sample acquisition function for active learning. This approach is significant because obtaining large amounts of training data is often impractical. While this method is relevant to various vision problems, it would be nice to see (a discussion or explanation on broader impact) how this concepts would translate to other CV challenges.
limitations: limitations: 1. Although the author highlighted their contributions in three major points, the core contribution is the use of limited datasets to achieve or surpass the state-of-the-art (SOTA).

2. There are inconsistencies in the paper:

    - What is **V** in line 502 and Equation (2)?
    - The use of vague terms like **reaches satisfactory performance** (Abstract Line 20-21) and other parts of the paper. What is satisfactory performance? How is this defined?
    - In Sec 3.2.1, Lines 425-428, the author should cite relevant evidence or provide an explanation for why **highly confident low entropy predictions can result in incorrect gaze heatmaps**.

3. It is unclear how scatterdness is computed. How do you define heatmap peaks given there could be multiple regions of interest?

4. In Figure 4, the experimental results (AUC curve) show that the error margin for AL-GTD is reasonably higher for a 6K sample size but becomes considerably consistent or slimmer for UnReGa. This behavior could imply inconsistency in the proposed method. Is there any study regarding such variability? Is it due to the fact that some random samples can have complex scenes?

5. In Table 1, the ablation study shows that for the limited sample size (a strong aspect of the proposed work), objectness, discrepancy, and scatterdness are indifferent, suggesting that 3.7K samples are too few for the proposed method, and notable differences appear at 6.2K. Is there any study indicating the optimal limited size for this proposed method?

6. From the complex scenes provided in the supplementary material, the method seems best suited for simpler scenes. However, there might be chances for improvement by training with larger datasets to achieve better results. Is there any ablation study on qualitative results for complex scenes comparing models trained with an optimally limited data size versus 50% or 100% of the data? Are we sacrificing the model's performance on complex scenes?
paper_summary: paper_summary: This paper focuses on the task of gaze target detection which aims to locate the target region where a person is looking in image or video scenarios. To tackle the issue of current methods relying on extensive labeled data, this paper proposes an active learning method that integrates supervised and self-supervised losses within a novel sample acquisition function. In addition, this paper also introduces pseudo-labeling to tackle the distribution shifts in the training phase. Experiments on the benchmark datasets validate the effectiveness of the proposed method when trained within a low-data regime.
strength_weakness: strength_weakness: - The utilization of active learning to realize gaze target detection in limited training data is novel.
- Extensive experiments are conducted on the benchmark dataset to validate the effectiveness of active learning for gaze target detection.
- This paper is well organized and easy to read.
limitations: limitations: - The paper utilizes pseudo-labeling to augment the training dataset but lacks a detailed explanation of the protocol for generating these pseudo-labels and how to avoid distribution shifts.
- The standard augmentations (e.g., random cropping, horizontal flipping) are adopted to augment the inputs. However, these augmentation techniques are typically used for image recognition tasks, and their applicability and effectiveness specifically for gaze target detection may not be optimal. Could the authors discuss whether any specialized augmentation strategies are tailored to the unique challenges of gaze target detection?