Single-Stage Visual Query Localization in Egocentric Videos
Authors: Hanwen Jiang
Santhosh Kumar Ramakrishnan
Kristen Grauman
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Query Localization
Egocentric Video
Spatial-Temporal Correspondence
Episodic Memory
TL;DR: We introduce a single-stage Visual Query Localization method that achieves state-of-the-art results and is ten times faster during inference.

Abstract: Visual Query Localization on long-form egocentric videos requires spatio-temporal search and localization of visually specified objects and is vital to build episodic memory systems. Prior work develops complex multi-stage pipelines that leverage well-established object detection and tracking methods to perform VQL. However, each stage is independently trained and the complexity of the pipeline results in slow inference speeds. We propose VQLoC, a novel single-stage VQL framework that is end-to-end trainable. Our key idea is to first build a holistic understanding of the query-video relationship and then perform spatio-temporal localization in a single shot manner. Specifically, we establish the query-video relationship by jointly considering query-to-frame correspondences between the query and each video frame and frame-to-frame correspondences between nearby video frames. Our experiments demonstrate that our approach outperforms prior VQL methods by $20$% accuracy while obtaining a $10\times$ improvement in inference speed. VQLoC is also the top entry on the Ego4D VQ2D challenge leaderboard.

Single-Stage Visual Query Localization in Egocentric Videos
Hanwen Jiang'
Santhosh Kumar Ramakrishnan' UT Austin
Kristen Grauman'
FAIR, Meta"
Abstract
Visual Query Localization long-form egocentric videos requires spatio-temporal search and localization of visually specified objects and vital to build episodic memory systems. Prior work develops complex multi-stage pipelines that leverage well-established object detection and tracking methods to perform VQL. However, each stage is independently trained, and the complexity of the pipeline results in slow inference speeds. We propose VQLoC, a novel single-stage VQL framework that is end-to-end trainable. Our key idea is first build holistic understanding of the query-video relationship and then perform spatio-temporal localization in single shot manner. Specifically, establish the query-video relationship by jointly considering query-to-frame correspondences between the query and each video frame and frame-to-frame correspondences between nearby video frames. Our experiments demonstrate that our approach outperforms prior VQL methods by 20% accuracy while obtaining a 10x improvement in inference speed. VQLoC is also the top entry on the Ego4D VQ2D challenge leaderboard. Project page: https://hwjiang 510. wwt....tbbb.o///..o..e io/VQLoC/
1 Introduction
Episodic memory, a specific type of long-term memory in humans, facilitates the retrieval of our past experiences such their time, location, related context, and activities [46]. This form of memory is pivotal to the progressive evolution of embodied intelligence due to its foundational role in enabling long-horizon reasoning [4, 19]. Nevertheless, human memory can falter in accurately recalling the details of daily activities-such as where we left our keys, or who we met while going for a run. This perceptual overload can be mitigated through the utilization of head-worn cameras AR glasses to capture our past experiences and the development of systems that can retrieve information. To this end, the episodic memory benchmark [13] recently proposed, which aims to develop systems that enable super-human human memory by allowing humans to query about their past experiences recorded on head- worn devices. These systems also have the potential to serve as a conduit for learning from past. experiences and data [60, 44], promoting long-term robot learning [25].
We focus on 2D Visual Query Localization (VQL), vital task in the episodic memory benchmark that aims to answer the question "where did I last see object X?". Specifically, given long-form egocentric video representing past experiences from a human Gamera-weared the goal search and localize a visual query object, specified as visual image crop, in space-time (see Fig. 1, left). Addressing VQL is of paramount importance, query object localization serves as prerequisite for downstream object-centric reasoning tasks.
VQL presents myriad of challenges existing computer vision systems. For example, current object detection systems localize pre-defined object categories on well-curated internet-style images [40, 5]. In contrast, VQL requires localizing open-set of objects specified as visual queries. Current object tracking systems expect to be initialized with a bounding box of the object next to its appearance in the video [54, 31]. However, the visual query crop of VQL originates from an image outside
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Videos
Video & Reporse
Spatio-Tumporal C Corresoonennees
Eoapple
Queey
Figure 1: Visual Query Localization (VQL): (Left) The goal is to localize visual query object within a long-form video, as demonstrated by the response track marked by yellow bounding boxes. The complexity of this task arises from the need to accommodate an open-set of object queries with diverse scales, viewpoints, and states as they appear in the vidco. (Right) Our method VQLoC first establishes a holistic understanding of the query-video relationship by jointly reasoning about tuerytto-frmee (spatio) and frame-to- trame-to-frame (temporal) correspondences, before localizing the response in single-stage and end-to-end trainable manner.
of the video being queried; that is, there may not be an exact match and "close frame" for the visual query. Complicating matters further, the egocentric nature of the VQL task presents its unique challenges. Compared with the visual query, the object may appear in the video under significantly varying orientations, sizes, contexts, and lighting conditions, and it may experience motion blur and occlusions. Finally, egocentric videos can span several minutes, hours, or days in real-world applications, while the object itself may appear only for a few seconds, resulting in a "needle-ing in-the-hheeee heeeehhystcc"" problem.
Prior work has attempted to address VQL through a bottom-up framework with three stages [13, 52, 53]: i) in each video frame, detect all objects and conduct pairwise comparisons with the visual query to obtain the proposal that is most similar to the query; ii) identify the similarity score peaks throughout the video; and iii) perform bidirectional tracking around the most recent peak to recover the spatio-temporal response. Although this framework is grounded in well-established object detection and tracking approaches, it relies heavily on the first stage to detect the object by independdnntyy looking each frame. While this may be possible if the object appears clearly in the video, it is often not the case due to the egocentric nature of images. Errors in frame-level object detection can potentially cause the entire system fail since the framework is not end-to-end differentiable and errors in carlier stages may not be rectifiable in later stages. Moreover, the methods suffer from a slow inference speed due to the high complexity of pairwise comparison with redundant object proposals.
To address these limitations, we propose VQLoC (Visual Query Localization from Correspondence), novel single-stage framework for VQL. Our key insight in VQLoC is that holistic understanding of the query-video relationship is essential to reliably localize query objects in egocentric videos. Accordingly, VQLoC jointly models the query-to-frame relationship between the query and each video frame and andframe- frume-to-frame relationships across nearby video frames (see Fig. 1, right), and then performs spatio-temporal localization in a single-stage and end-to-end trainable manner. Specifically. we establish the query-to- to-m-faefee relationship by extracting image features for the visual query and each video frame using a ViT backbone pretrained with DINO [37] and using a cross-attention transformer module [48] to establish correspondences between the image regions in the query and video frame. We then propagate these correspondences over time using a self alf-attention transformer module [48] that exploits the frame-to came-to-frame relationships resulting from the temporal continuity of videos to capture the overall query-video relationship. Finally, we use a convolutional prediction head that performs spatio-temporal localization by utilizing the query-video relationship to make frame-level predictions. Critically, our model operates in a single stage, i.e., there are no intermediate localization outputs with dedicated post-processing steps, and it is end-to-end trainable since it uses only differentiable modules to obtain the final prediction.
VQLoC has several benefits when compared with the prior stage-wise methods. Unlike prior work that explicitly generates object proposals in the video frame and compares them with the visual query, VQLoC implicitly establishes the query- query-frame relationship by performing iitention-based reasoning between the visual query features and the video frame features. This approach effectively utilizes image regions of the background and non-query objects as contextual information for inference. Additionally, our implicit query-frame relationships are significantly faster to compute than explicitly generating proposals and performing pairwise comparisons, which is critical for real-world episodic memory applications. Finally, VQLoC is end-to-end trainable, which results in much better performance compared to prior work.
2

---

We evaluate VQLoC on the Ego4D Visual Query 2D Localization benchmark. VQLoC achieves the state-of-the-ee on this benchmark, outperforming previous methods by 20%. and it is also ranked first on the public Ego4D VQ2D challenge leaderboard." Furthermore, VQLoC achieves real-time inference with 36 Frames Per Second (FPS), which 10x faster than prior work. We are committed to releasing our code upon publication for future research.
2 Related Work
Object detection. Prior work has made impressive progress in building state-of-the-ar object detectors for localizing objects in images [26, 12, 40, 5]. These methods typically detect objects with pre-specified object categories and are evaluated internet-stlle and human-curated image datasets [27, 10, 56, 38). While these methods relied significantly on having sufficient training data, recent work has extended them incorporate long-tail object categories [14, 43] and free-form language-basee based detection [58]. Prior work has also proposed methods for one-shot detection, which build on the success of object detection methods [17, 34, 57]. They leverage objectness priors by calculating similarities between visual queries and region proposals. However, all of the methods work on well-curated images, e.g. the COCO dataset [27]. Our work focuses on the VQL task, which is a challenging version of one-shot detection on long-form egocentric videos. Unlike image-based based one-shot detection where the object is assumed to be present in the image, video-based one-shot detection requires us to identify in which frame the object appears for -patio-temporal localization.
Object Tracking. There is a rich literature on object tracking methods that track a static or dynamic object throughout given video [8, 33, 54, 31, 22, 36, 35]. The object is typically specified bounding box in a video frame. and the goal is to track it through the subsequent frames in the same video [24, 11]. Short-term tracking methods are developed to track objects till they completely disappear from the video frame [47, 30J, while long-tern tracking methods are capable of recovering the object track when it re- re-appears in the video [50, 32, 54]. Tracking methods are initialized with an object bounding box close to its appearance in the video, while visual queries in VQL may originate from outside the video. This can pose challenge for tracking methods. We compare with a state-of-the-arr long-term tracker in our experiments [54]. Technically, VQLoC differs from tracking- based methods in using apar transformer, temporal reasoning within local windows, and anchor-based prediction heads.
Visual Query Localization (VQL). The recently proposed episodic memory benchmark introduces the VQL task on egocentric videos, where the goal is apatio-temporally localize query object specified as visual image crop [13]. Prior work has tackled VQL using a three-stage detection and tracking framework. First introduced as baseline Ego4D, this approach performs frame-level detection, identifies the most recent "peak" in the detections across time, and performs bidirectional tracking [3] to recover the complete temporal extent of the response [13]. Subsequent research in this paradigm improves the frame-level detector by reducing false positive predictions through negative frame sampling [52] and proposes using background objects as context [53]. However, these multi-stage approaches independently perform spatial and temporal reasoning in modular fashion without end-to-end training, which results in compounding errors across the stages. Furthermore, they incur significant computational costs and time for training and inference, which limits their applicability to real-time episodic memory applications. Our proposed method, VQLoC, distinguishes itself by first establishing holistic spatio-temporal correspondences and then localizing the object with an end-to-end paradigm. This leads to state-of-the-ar results with fast inference speeds.
Visual Correspondence. Establishing correspondence is a core problem in computer vision [15]. The advancement of visual correspondence models has significantly contributed to the success of various computer vision tasks, including image matching and retrieval [45, 42], tracking and flow estimation [2, 16). 3D vision [29, 21], and representation learning [51, 49]. Recent developments in vision foundation models have demonstrated remarkable capabilities to establish correspondence between multi-modal [39] and spatio-temporal [6] inputs, empowering new applications [23, 20).
The design of model architectures plays critical role in finding correspondence. Specifically, transformer models [48, 1, 59], which propagate information by establishing similarity- cor- respondence, are widely used. For example, TubeDETR [55], which performs visual question answering, concatenates the text query and frame features and finds their correspondence using self-attention transformers. STARK [54] employs a similar architecture and extends it to visual templates for visual object tracking. These methods do not differentiate the query features and the
'Ego4D VQ2D challenge leaderboard: attpe://eval. ai/ htt///ebbbballengee..eeeeeeeeeee challenge-ppeeeeeeec page/ / 1843/eesderboard
3

---

Video y
Video Features
Query- Frome Comssonniace
Query-Videa Correponndnce Festures
Predictions
Timoo
Features
Bounding boxes
Enc.
Spatio- Temporal TX
Spatal
Pred. Head
Query
Stared
TX
Foreground scores
Query Features q
Wy
Enc.
Timee
Figure 2: Visual Query Localization from Correspondence (VQLoC): We address the VQL problem by first establishing the overall query-vides relationship and then inferring the response to the query. VQLoC extracts image features for both the query and video frames independently. It establishes query-to- tu-ry-to-frame between the query features q and each video frame feature v; using a transformer module (Spatial TX), and obtains the correspondence features f for identifying potential query-relevant regions in the frames. VQLoC then propagates these frame-level correspondences through time using another transformer module (Spatio- Temporal TX), which leverages the frame-to tore-tto-rameee correspondence between nearby video frames and obtains the query-video correspondence features v'. Finally, VQLoC uses the query-vides correspondence prediet the per-frame bounding box and the probability that the query ohject occurs in the frame.
frame correspondence
frame features after the concatenation, making the features lose spatial correlation with the input frames. In contrast, VQLoC utilizes the cross-attention transformer to establish the query-to-frame correspondence, updating the frame features by incorporating query features. This maintains the spatial correlation between frame-level features and the frames, providing strong priors for object localization.
3
Method
We propose a novel end-to-end framework for localizing visual queries in egocentric videos. We design model based on the insight that holistic understanding of the query-video relationship is crucial for reliably localizing query objects appearing in the video. To this end, we propose to first capture both the query-to-frame relationship for each video frame and the frame-to-frame relationship across nearby frames. We then predict the results based on the captured query-video relationship in a single shot. Next, will describe the formulation of the VQL task in Sec. 3.1, introduce model architecture in Sec. 3.2, and describe our approach for training and inference in Sec. 3.3.
3.1 VQL Task Formulation
The VQL task is formulated as open-set localization in egocentric videos, answering questions like "where did last see my wallet?". Formally, given an egocentric video V = {w1, er) consisting of T frames and a visual query Q specified an image crop of the query object?, the goal is to spatio-temporally localize the most-recent occurrence of the query object in the video. The localization result a response track indices, respectively, and r, is bounding box containing the query object on the 7t frame.
(ra, ".,e), where and are start and end frame
3.2 VQLoC Architecture
Our proposed method VQLoC is illustrated in Fig. 2. VQLoC employs a shared image encoder to extract visual features for both the video frames and the visual query. We utilize the DINO [6] pre-trained VIT [9], which can capture object-level semantic correspondence under varying camera viewpoints [18]. First, VQLoC leverages spatial transformer find the query-to no-frame correspon- dence between each video frame and the visual query. The spatial transformer identifies the regions in the frames that are similar the query in semantics, handling the potential large appearance variation. VQL.C then uses a spatio-temporal transformer that propagates the query-to-frame correspondence over time by leveraging the frame-to-frame correspondence arising from temporal continuity in the video, and establishes the query-video correspondence. This process is performed within a local temporal window, handling the fast head motion of egocentric videos. Finally, uses the holistic understanding of the query-video relationship to predict the response track. The response prediction
"The image crop of the query object is sampled from outside the ground-truun response track.

---

Query- Frame
Query Video Corespondeaces
Frame Features vi
Correpoonencee
(thw ()
Features Ea
Sgatio-Tempone TX Block (xN)
Features y
Mask
thw)
Pred. Head
K HADDE
Reg. Head
Prob. Head
Mullt-Hadd
Self-
tteatonn
TX
@ Anchoes
Padlan
A Anchor-lere level Preds.
Train
Test
hnz
Label
Top-1
ATred Not stiod
Confider.
Amchorr
Figure 3: Architecture for spatio-temporal transformer (left): add the downsampled query-to- to-frme correspondence features fa with 3-D spatio-temporal positional embedding, and flatten into tokens (fz). We use several transformer layers to propagate the frame-level correspondences over time hy leveraging frame- to-frame correspondences between nearby vidco frames within local temporal window, and obtain the overall query-video correspondence features v". Architecture for the prediction head (right): After establishing the query-video relationship. we use it predict anchor-level results each video frame We perform label association during training and pick the refined anchor with the highest probability as the prediction for the frame in testing.
head is an anchor- based model, which deals with the small scales of query objects in the video. Next, we describe these components detail.
Visual Encoder. Our visual encoder consists a fixed DINO pre- trained VIT followed by several learnable convolution layers. We independently extract visual features for each of the T video frames and the visual query and obtain the video features v E T xC and query features q EE Hx H, W signify the spatial resolution of the features, and C is the channel size.
Spatial Transformer. The spatial transformer establishes query-to-frame correspondences between the visual crop and each video frame using cross-attention [48] between their visual features. Formally, let U R # x be the flattened video feature v; for video frame and let IR HWxC be the flattened query feature The query-tor query-to-frame correspondence feature f; is obtained as follows:
f FFN (rooCCCstttttnnionn(n ();,
(1)
where FFN is feed-forward neural network. Intuitively, the frame features vs updated by incorporating the corresponding query features q, where the gross-attention computes their feature similarity the fusion weights. The transformer output is reshaped to # denoted as f, Here, the updated features f, retain the spatial arrangement and maintain the spatial cues for performing accurate localization. This process is repeated for all video frames obtain the final query-to-frame correspondence features e XxxXx
Transformer. The spatio-temporal transformer is designed to propagate and refine the query-to- soerytto-frame correspondence features from Eqn. over time, by establishing frame- to-frame correspondences between nearby video frames. It leverages the temporal continuity of videos. We illustrate the architecture Fig. 3. We first decrease the spatial resolution of the feature maps f using shared convolution layers, where the output is Fa R xxxxxxxx h, w and are spatial resolutions and channel size of the downsampled feature maps. We add 3-D spatio-temporal positional embedding p E IR. xhxuxe to fa, and flatten it into 1-D tokens f.::
f Flatten + p) E T Thwxe
where the positional embedding p is learnable and initialized as zeros.
Next, we use the anntio-temporal transformer module to establish the query-video relationship. It consists of multiple layers of locally-windowe propagation within a local time window. In particular, feature element belonging to time t only attends to feature elements from time steps in the range [A- w. (+w). L- and ++ w are the start and end of the local time window with temporal length + We found that this was beneficial since the locations of query objects nearby frames are highly correlated, providing strong priors for feature refinement. We achieve locally windowed attention via masking attention weights. After performing
temporal self-attention, i.e., we restrict the feature
5

---

feature refinmentt using the apatio-tempordl transformer module, we reshape the 1-D tokens back into the original 3-D shape to get the query-video correspondence features
xxxxuxx,
Prediction Head. Instead of directly predicting a single bounding box for an image, we first define anchor boxes B E xxxxxxxx on the feature maps [26]. We found that this was advantageous because the sizes and locations of query objects are diverse. The multi-scale anchor boxes can provide a strong prior on query object sizes and locations. For each frame, we define n anchor boxes for each of the spatial elements of the feature map (h- w elements in total). Each anchor box defined on the feature map space can be mapped back to its corresponding location in the pixel space for inferring predictions. For each frame from V,, we obtain the predictions by passing the query-video correspondence feature V through two heads:
P.: = ProbHead (v)) E TR hxxxx and Ad; =ReHHeddy vv)) E Rhxuxin
where each head consists of several convolution blocks. Pe is the anchor-level query object occurrence probability and Ab; is the regressed bounding box refinement. We then reshape Ab; to Rixuxxxd denoted AB, The final refined anchor boxes for the jth frame are B, = B + AB,. We perform the same operation on all frames.
3.3 Training and Inference
In both training and inference, we chunk the untrimmed video into fixed-length clips. During training, we ensure that the clip covers at least a part of the response track. During testing, we concatenate the predictions on all clips as the final result. We introduce the training and inference details follows.
Training. VQLoC is trained with the loss Limg Libow X2 Lprob all clip frames for supervising bounding box regression and query object occurrence probability prediction. For each refined anchor box b E B, we calculate the intersection of union (IoU) between the ground-truth query object box and the original corresponding anchor box b. If the IoU is higher than threshold 0, it will be assigned as a positive box, denoting the anchor box region captures the query object well. We apply the bounding box loss as Lbboz (B, b) = Lregg (b, b), where b is the ground box for the query object, and B C B is the set of positive boxes. Specifically, Treg a combination of the T, loss and the generalized loU (GloU) loss [41]:
(reg(b, b) = [b.- bell + [b, b, |l + [|b - bw Agiou Lgiou b).
where be by and bw are the center coordinate, height, and width of the bounding box, and Agion is weight that balances the losses [5]. To get the query object probability loss, we assign the probability labels to the positive and negative anchor boxes with and 0 respectively. We denote the assigned probability labels as P. The query object occurrence probability loss is defined as Lprob = Luce( (P, P), which is the binary cross-entropy (BCE) loss.
Since the target videos are long, false positive prediction, where the model wrongly identifies other objects as the query object, is one of the bottlenecks that limit the temporal accuracy of the model. To prevent this, we perform hard negative mining (HNM) (similar to [13, 53, 52]). Given a mini-batch of N videos, we diversify the negative anchors for each batch element n by treating frames from every other video * occurrence probability for all the negative anchors using Eqn. 3 and sample the top K anchors with the largest BCE loss as hard negatives (i.e., negatives which our model thinks are positives). We then train the model using the positives from the same batch element and the hard negatives sampled across all batch elements with the BCE loss. We keep the ratio between positives and negatives as 1: during training.
negative. After expanding the pool of negatives, we calculate the query
Inference. During inference, we get the prediction for each frame by choosing the refined anchor box with the highest query object probability. We reject low-confidence predictions by applying threshold to to the predicted query object probability. See supplementary for more details.
Experiments
First, we describe our implementation details, the dataset and evaluation metrics. We then quantita- tively compare VQLoC with prior methods, and present ablation studies that examine the impact of various design choices in VQLoC.

---

Table 1: VQL results Ego4D VQ2D benchmark: We compare our method VQLoC against the baselines. The test results are obtained from the challenge leaderboard.
Our--ViT-B
20
Cacoformer
VITS
15
Bsseline
Validation
Test
Reoddime
Cun-VTT--2244
tAP2s
stAPas
1oc%
Succ.
stAP2s
FPS
SimmRCNN [13)
0.22
0.15
32.92
43.24
0.20
0.13
3
NFM [52]
0.26
0.19
37.88
47.90
0.24
0.17
3
STAR
CocoPormer [53]
0.26
0.19
37.67
47.68
0.25
0.18
3
50
300
150
Speed (FPS)
STARK [54]
0.10
0.04
12.41
18.70
33
Figure 4: Accuracy-speed trade-off: The
VQLoC (ours)
0.31
0.22
47.05
55.89
0.32
0.24
36
radius of circles notify model sizes.
4.1 Experimental Setup
Implementation Details. We train the model with 448 x 448 resolution videos obtained by resizing the source videos along the longer edge and applying zero-padding to the shorter edge for each frame. We set the loss coefficients A = 1 and Agion. = 0.3. The positive anchor box threshold is 0 = 0.2 due the small scales of query objects in videos.. VQLoC is trained for 60, 000 iterations with batch size 24, utilizing the AdamW optimizer [28] with peak learning rate of 0.0001 and a weight decay of 0.05. A linear learning rate scheduler and warmup of 1, 000 iterations are employed. We adopt DINOv2's ViT-B14 as the backbone and train on clips of 30 frames with a frame rate of 5 fps. The clip length selected keep the balance between positive and negative frames. Clips are randomly selected to cover at least a portion of the response track. We define anchor boxes on the feature map with spatial resolution of 16 x 16, each element has 12 anchor boxes with 4 base sizes (16², 322, 642, 1282) and 3 aspect ratios (0.5, 2). During training, we apply random data augmentations to the clips, such as color jittering, horizontal flipping, and random resized cropping. We train and evaluate RTX 6000 GPUs.
Dataset. We train and evaluate our model on the Ego4D dataset [13]. Ego4D is large-scale egocen- tric video dataset designed for first-person video understanding and contains open-world recordings of day-to-day activities performed in diverse locations. We use the VQ2D task annotations from the episodic memory benchmark in Ego4D, which consists of 13.6k 13.6k/4.5k// /4.5k/4.44 queries annotated over 262/87/ 87/884 hours of train / val / test videos. The average target video duration is ~ 140 seconds and the average response track length is ~ 3 seconds, making this a challenging benchmark for the VQL task. Ego4D is, to the best of our knowledge, the only publicly available dataset for VQL.
Metrics. In our evaluation, we adhere to the official metrics defined by the Ego4D VQ2D bench- mark. We report two average precision metrics: temporal AP (1AP25) and sputio-temporal AP (stAP25), which assess the accuracy of the predicted temporal and spatio-temporal extents of response tracks comparison to the ground-truth using an Intersection over Union (IoU) threshold of 0.25. Additionally, report the Recovery (rec%) metric, which quantifies the percentage of predicted frames where the bounding box achieves at least 0.5 IoU with the ground-truth. Lastly, we report the Success metric, which measures whether the IoU between the prediction and the ground-truth exceeds 0.05. Apart from VQ2D evaluation metrics, we also report the inference FPS of each method measure of computational efficiency.
Baselines. We compare our approach against the following baselines. All baselines are trained on the Ego4D dataset for a fair comparison.
SiamRCNN [13]: This is the official baseline for VQL introduced in Ego4D. It employs
three-stage pipeline with frame-level detection [50], latest peak detection, and object tracking [3].
NFM [52]: It employs the three-stage pipeline introduced in SiamRCNN and improves the frame-level detector training by sampling background frames as negative to reduce false positives.
This was the winning entry of the Ego4D VQ2D challenge held at CVPR 2022.
CocoFormer [53]: improves the frame-level detection architecture from SiamRCNN by
reasoning about all object proposals in frame jointly (as a set) using transformer module.
STARK [54]: It is an end-to-end method for long-term visual object tracking. We adapt the method with object occurrence prediction head to enable it to work on long- forg-form videos.
7

---

Q uerr
Response Track (Zoom-in with blue box)
Probability
150
1030
1250 1902
Figure 5: Qualitative analysis of VQLoC: In each row, we visualize an example prediction from VQLoC. For each example, we show the visual query, four frames from the predicted response track, and temporal plot of the predicted object-occurrence probabilities (y-axis) for each video frame (x-axis). We highlight the ground-truth and predicted boxes green and yellow, respectively. We also provide the zoom-in view of the predicted response on the top-right comer of cach frame. The temporal plot additionally shows the detected peaks in blue and the ground tround-truth response window (which is only annotated for the latest appearance of query objects) in red. VQLoC is able to localize query objects under challenging conditions such as clutter (row 1), soriial-o--err-cmmplee occlusions (row 2), and motion blur with significant viewpoint variations (row 3). We show a failure case in row 4 where VQLoC fails distinguish between two cables with similar appearances. Note that temporal plot in the last column shows the object occurrence probability on the entire video, while the ground-trum truth is only annotated for the last occurrence.
4.2 Experimental results
The results are shown in Table 1. Our method VQLoC outperforms prior works on all metrics and sets the saate-of-the-ar for this task. VQLoC relatively improves over the next-best baseline (CocoFormer) by 19% tAP25. 16% stAP25. 25. 25% recovery, and 17% success. Furthermore, the rumning speed of VQLoC 10x faster than the three-stage VQL methods. Besides, we also demonstrate that directly applying tracking methods, i.e. STARK, doesn't 't work well for VOL. We conjecture the reason is that STARK concatenates the query features and frame features without differentiating them. In this manner, the concatenated features, which used regress the localization results, lose the spatial correlation with input frames. While this design might work for normal videos where the camera viewpoint changes slowly and smoothly, its problem is exacerbated in egocentric videos where the camera moves dramatically and the object specified by the visual query is quite different from its appearance in the video. As shown in Fig. 4, VQLoC shows a reasonably good balance on the speed-accurace trade-off. At 80 FPS, we match the best baseline performance; at 33 FPS, we outperform the best baseline by healthy margin. We present a qualitative analysis in Fig. 5.
4.3 Ablation Study
We present an ablation study on each block of VQLoC, examining and evaluating their impact within the structure of our model.
Backbone Size and Input Resolution. We explore the impact of using different backbone sizes and input spatial resolutions. We test the following alternatives: using smaller ViT-s14 backbone with 448 x 448 resolution, and ii) using the original ViT-b14 backbone but smaller 224 x 224 resolution. Specifically, the input with spatial resolutions 448 x 448 and 224 x 224 result in features with spatial size 32² and 162, respectively. As shown in Table 2, the model with the largest input size and the largest backbone achieves better results. However, experiments demonstrate that using a larger backbone only brings marginal gains (comparison between the last two rows). In contrast, when inputting the model with low resolution, the performance drops dramatically. Specifically, the ratio of tAP25 stAP25 drops from 66% to 45%. This decrease reflects the huge degeneration of

---

Table 2: Ablation on the backbone size choices. Res. denotes the spatial resolution of the inputs.
Table 3: Ablation of finding frame-query query correspon- dence. X-TX denotes our spatial transformer.
Backbone
Res.
tAP25
stAP25
rec%
Suec.
Fusion
Layers
tAP25
StAP25
rec%
Succ.
VIT-b14
224
0.22
0.10
32.18
45.84
Conv
0.22
0.13
32.27
44.95
ViT-s14
448
0.28
0.18
37.71
52.18
X-TX
1
0.29
0.19
38.58
53.12
ViT-b14
448
0.29
0.19
38.58
53.12
X-TX
3
0.29
0.20
37.91
53.74
Table 4: Ablation of spatio-temporal transformer.
Table 5: Ablation on the prediction head and loss function. FL denotes focal loss.
Win. is the window size, and glb. means global.
ST Refine.
Win. Layers tAP25
stAP25
rec% Succ.
Anchor
Loss
tAP25
stAP25
rec% Succ.
ST-TX
glb.
3
0.08
0.03
12.49 15.73
X
BCE
0.17
0.10
34.23
47.60
ST-TX
9
3
0.27
0.18
37.98 51.08
BCE
0.12
0.07
40.23
42.27
ST-TX
7
3
0.27
0.19
36.97
50.21
FL
0.29
0.19
38.58 53.12
ST-TX
5
3
0.29
0.19
38.58
53.12
BCE+HNM
0.31
0.22
47.05
55.89
localization accuracy. Since the query objects in the videos are usually small, the low-resolution inputs and feature maps may not capture their details clearly. This matches our intuition that accurate localization requires relatively high-resolution features.
Query-Franc Frame Correspondence. We study different methods for establishing the correspondence between the query and the frame. Specifically, we compare against model variant that aggregates the spatial information from the query using convolutions. In detail, we apply downsample convolutions on the query features to obtain 1-D feature vector (with 1 x spatial resolution). then tiles this feature to obtain a H x W feature map and concatenates it with the frame features. Finally, we use convolutions to obtain the our Spatial TX module for establishing this correspondence. As shown in Table 3, the performance using this naive fusion method is much worse (rows vs. 2). highlighting the value of establishing correspondences using spatial transformer. We also experiment with using more transformer layers in our spatial TX module (rows 2 vs. 3 in Table 3), but this provides minimal gains, likely due to the strength of the image backbone features.
correspondence features. Importantly, this does not use
Spatio-Temppraa Temporal Transformer Designs. Next, we study the role of using local temporal windows in our spatio-temporal TX block in Table 4. We experiment with global window, and local windows of sizes (5, 7, with 5 being our default choice. Using a global window (row 1) significantly reduces the performance since the object's location, pose and appearance change dramatically over time in egocentric videos, making it hard to establish long-range dependencies. Instead, focusing on the local temporal window with several consecutive frames makes the feature propagation easier, establishing useful short-horizon dependencies. Our experiments show that the window size 5 works best, where 5 consecutive frames span 1 second for the 5 FPS videos.
Prediction head and Loss Function. Finally, we study the design of using anchors in the prediction head and the loss functions used to supervise the anchor probabilities in Table 5. Not using anchors (row 1) performs worse than our models which use both anchors and advanced losses. Our model with anchors performs poorly with the BCE loss due to the data imbalance between positive and negative anchor boxes (row 2). The reason is that even positive frames will introduce hundreds of negative anchors, which makes the anchor-level ratio between negatives and positives much higher than the ratio in frame-level. Using focal loss or hard negative mining improves significantly over using only the BCE loss by accounting for the data imbalance (rows 3 and 4 vs. row 2). We observed that hard-negative mining (HNM) works better than focal loss (rows 3 vs. 4). Unlike HNM that only focuses on hard negatives, focal loss additionally focuses hard positives, where the object is annotated in the training data but may be difficult to recognize due to effects like severe occlusions (e.g., only 10% of the object is visible). Therefore, these hard positives may provide harmful training signals since they are not reliably distinguishable from negatives.
Backbone Choice. We also ablate the backbone choice by replacing the DINOv2 encoder with a -.P--pretaineed ViT [39]. Note that the highest resolution that CLIP supports is 336 x 336. Experiments demonstrate with 0.28 tAP25. 0.22 stAP 44.67 rec% and 53.80 Suce. The results are comparable with using DINOv2 backbone, showing the versatility of VQLoC to the backbone choice.

---

Table 6: Model performance on different scales of objects in the videos. and stands for small (1941 samples) and large object (2336 samples), respectively.
Method
Scale
tAP25
stAP25
rec%
Succ.
CocoFormer [53]
0.206
0.127
40.804
32.583
VQLaC
5
0.213
0.138
44.719
33.738
CocoFormer [53]
0.338
0.271
16.164
40.737
VQLoC
[
0.454
0.387
67.680
53.635
5 Conclusion
We propose VQLoC, a single-stage and end- and-to-end trainable method for the Visual Query Localization (VQL) problem in long-horizon videos. VQLoC first builds holistic understanding of the query- video relationship, and leverages this understanding to localize the query in a single shot. Our key insight lies in how we establish the spatio- correspondence between the query and video features. Specifically, we build the query-to-frame correspondence for each video frame and then propagate these over time to nearby frames by leveraging the temporal smoothness of videos. Compared with prior stage-wise methods, VQLoC not only demonstrates hetter spatio- temporal localization accuracy but also improves the inference speed significantly by avoiding explicit comparison between region proposals and the query. VQLoC also achieves the top performance on the Ego4D VQ2D challenge leaderboard. In future work, we plan develop systems perform in-depth object-level understanding based on our model's VQL predictions.
Limitation. Similar to prior works, VQLoC is trained in a supervised manner, requiring large number of annotations for training. Besides, the hyperparameters, i.e. local window length, might need to be tuned for training or inference other datasets with different FPS.
Acknowledgement. UT Austin is supported in part by the IFML NSF AI Institute, Kristen Grauman is paid as a research scientist by Meta.
References
[1] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference Machine Learning, 2021.
[2] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Fully- convolutional siamese networks for object tracking. In ECCV Workshops, 2016.
[3] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Know your surroundings: Exploiting scene information for object tracking. In European Conference on Computer Vision, 2020.
Joydeep Biswas and Manuela M. Veloso. Episodic non-markov localization: Reasoning ahout short-term and long-term features. 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3969-3974, 2014.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-on End-to-end object detection with transformers. ArXiv, abs/2005.1288 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[7] Ding-Jic Chen, He-Yen Hsieh, and Tyng-Lahh Liu. Adaptive image transformer for one-shot object detection. 202/ IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12242-12251, 2021.
[8] Yutao Cui. Cheng Jiang. Limin Wang, and Gangshan Wu. Mixformer: End-to-eud tracking with iterative mixed attention. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-13618, 2022.
[9] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv. abs/2010. 11929, 2020. 2020.
10

---

[10] Mark Everingham, Lue Van Gool, Christopher K. L. Williams, John M. Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338, 2010.
[11] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Deng. Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large- single object tracking. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5369-5377 2018.
[12] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[13] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Q. Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K. Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Z. Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer. Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina González, James M. Hillis, Xuhua Huang. Yifei Huang. Wenqi Jia, Weslie Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar, Federico Landini. Chao Li. Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu. Jonathan Munro. Tullie Murrell, Takumi Nishiyasu. Will Price. Paola Ruiz Puentes, Merey Ramazamova, Leda Sari, Kiran K. Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pahlo Arbeléez, David J. Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishma Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani. Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3.000 hours egocentric video. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18973-18990, 2021.
[14] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. Lvis: A dataset for large vocabulary instance segmenta- tion. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5351-5359 2019.
[15] Tal Hassner and Ce Liu. Dense Image Correspondences for Computer Vision. Springer, 2016.
[16] Berthold K. P. Horn and Brian G. Schunck. Determining optical flow. In Other Conferences, 1981.
[17] Ting-I Hsich, Yi-Chen Lo. Hwann- Tzong Chen, and Tyng-Luh Liu. One-shot object detection with co-attention and co-excitation. In Neural Information Processing Systems, 2019.
[18] Vingdong Hu, Renhao Wang. Kaifeng Zhang, and Yang Gao. Semantic-aware fine- ane-grained correspondence. In European Conference on Computer Vision, 2022.
[19] Yang Hua, Alahari Karteek, and Cordelia Schmid. Occlusion and motion reasoning for long-term tracking. In European Conference Computer Vision, 2014.
[20] Krishna Murthy Jatavallabhula, Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer. Soroush Saryazdi, Nikhil Varma Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso M. de Melo, M. Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion: Open set multimodal 3d mapping. ArXiv, abs/2302.07241, 2023.
[21] Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. ArXiv, abs/2212.04492, 2022.
[22] Yang Jin, Yongzhi Li, Zehuan Yuan, and Yadong Mu. Embracing consistency: A onc-stage approach for spatio-temporal video grounding. ArXiv, abs/2209. 13306, 2022.
[23] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. ArXiv, abw/2303.095..t 009553, 2023.
[24] Bo Li, Wei Wu, Qiang Wang. Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4277-4286, 2018.
[25] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12) 2947, 2017.
[26] Tsung Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Transactions Pattern Analysis and Machine Intelligence, 42:318-322 2017.
11

---

[27] Tsung- Yi Lin, Michael Maire, Serge J. Belongie. James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
[28] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[29] Bruce D. Lucas and Takeo Kanade, An iterative image registration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence, 1981.
[30] Alan Lukezic, Tomas Vojir, Luka Cehovin Zaje, Jiri Matas, and Matej Kristan. Discriminative correlation filter with channel and spatial reliability. In Computer Vision and Patter Recognition, 2017.
[31] Fan Ma, Mike Zheng Shou, Linchao Zhu, Haoqi Fan, Yilei Xu, Yi Yang, and Zhicheng Yan. Unified transformer tracker for object tracking. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8771-8780, 2022.
[32] Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool. Learning target candidate association keep track of what not to track. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13444 13454, 2021.
[33] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 8834-8844, 2021.
[34] Claudio Michaelis, Ivan Ustyuzhaninov. Matthias Bethge, and Alexander S. Ecker. One-shot instance segmentation. ArXiv, abs/181 1. 1557, 2018
[35] WonJun Moon, Sangeek Hyun, Sang shin Paldal-gu Suwon-city Park, Dongchan Park, and Jae-Pil Heo. Query dependent video representation for moment retrieval and highlight detection. CVPR, pages 23023-23033, 2023.
[36] Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-global video-text interactions for temporal grounding. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 10807-10166 2020.
[37] Maxime Oquab, Timoth' ee Dareet, Th' co Moutakanni, Huy Q. Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-No Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russ Howes, PaoYao Huang. Shang- Weeng-WennJn Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. ArXiv, abs/2304.07199. 2023.
[38] Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alexander Sorkine-Homung Homung, and Luc Van Gool. The 2017 davis challenge on vidco object segmentation. ArXiv, abs//704.00675 2017.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Muchine Learning, 2021.
[40] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-enn: Towards real-timo object detection with region proposal networks. IEFE Transactions on Pattern Analysis and Machine Intelligence, 39:1 137-1149, 2015.
[41] Seyed Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. 2019 JEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 658 0836666,77 2019.
[42] Paul- Edouard Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Supergluc: Learning feature matching with graph neural networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019.
[43] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8429-8438, 2019.
[44] Sainbayar Sukhbaatar, Arthur D. Szlam, Jason Weston, and Rob Fergus. End-to -on-o---d memory networks. In NIPS, 2015.
12

---

[45] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector- free local feature matching with transformers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9918-8927, 2021.
[46] Endel Tulving and Wayne Donaldson. Organization of memory. American Journal of Psychology, 86:675, 1973.
[47] Jack Valmadre, Luca Bertinetto, João E. Henriques, Andrea Vedaldi, and Philip H. S. Torr. End-to-end representation learning for correlation filter based tracking. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5000 5008, 2017.
[48] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
[49] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
[50] Paul Voigtlaender, Jonathon Luiten, Philip S. Torr, and B. Leibe. Siam r-cmn: Visual tracking by re-detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6577-65 77776657,222 2019.
[51] X. Wang and Abhinav Kumar Gupta. Unsupervised learning of visual representations using videos. 2015 IEEE International Conference on Computer Vision (ICCV), pages 2794-2802, 2015.
[52] Mengmeng Xu, Cheng- Yang Fu, Yanghao Li, Bernard Ghanem, Juan-Manuel Pérez-Rúa, and Tao Xiang. Negative frames matter in egocentric visual query 2d localization. ArXiv, abs/2208. 0e..2220001999,,,. 2022.
[53] Mengmeng Xu, Yanghao Li, Cheng-Yaue Fu, Bernard Ghanem, Tao Xiang, and Juan-Manuel Pérez-Rúa. Where is my wallet? modeling object proposal sets for egocentric visual query localization. ArXiv, abs/2211.10528. 2022.
[54] Bin Yan, Houwen Peng. Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pages 10428-
10437, 2021.
[55] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: temporl video grounding with transformers. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recogni- tion (CVPR), pages 16421-16432, 2022.
[56] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2633-2642, 2018.
[57] Tengfei Zhang. Yue Zhang, Xian Sun, Hao Sun, Menglong Yan, Xue Yang. and Kun Fu. Comparison network for onc-shot conditional object detection. ArXiv, abs/1904.02317, 2019.
[58] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krahenbuhl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference on Computer Vision, 2022.
[59] Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühh. Global tracking transformers. 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8761-8770, 2022.
[60] Linchao Zhu and Yi Yang. Inflated episodic memory with region self attention for long-tailes visual recognition. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4352 2020.
13

---

Appendices
A
Broader Impact
Visual Query Localization (VQL) has broad benefits to AR and robotics applications. For example, VQL is the basis for developing super-humm human episodic memory systems that can retrieve information from past human experiences. It has the potential to assist and strengthen the long-term reasoning capability of humans. Furthermore, can be vital for embodied agents/robots to learn from past experiences or perform relevant tasks with long-term dependency.
B
Model Details
Model Architecture. We provide the workflow of our model as follows. The details of each block, ie. Encoder, Spatial Transformer,
Transformer, and Prediction Heads included in the main text. Specifically, T is the length of clip, and N is the number of anchors. We use 30 and N 8 8- 12 768. We provide the code of the model class in the supplementary.
Stage
Configuration
Output
0
Vidco Frames
x 448 x 448 x
0
Visual Query
448 x 448 x
Encoder
Frame features
T' x 32 x 32 x 256
Query features
32 x 32 x 256
Spatial Transformer
2
Updated frame features
1 x 32 x 32 x 256
Temporal Transformer
3
Downsampled frame features
T' x x 8 256
3
Propagated frame features
T x x 8 256
Prediction Heads
4
Anchor refinement
T' N x
4
Anchor scores
T x N
inference
5
Top-1 anchor
T x
5
Top-1 score
Table 7: Model workflow.
Inference Details. We provide more inference details here. For the predicted object occurrence scores on all frames, we first smooth the scores with a median filter with kernel size Then we apply peak detection the smoothed scores. We find the peak with the highest score and use 0.8 as the threshold to filter non-confident peaks. We then select the response track that corresponds to the last peak as the results. To find the start and end time steps of the last response track, threshold the occurrence scores with the value 0.7 8p, where 8p the score of the last peak.
C More Results
Visualization. Please see our supp video for more visualization results of the retrieved response track with bounding boxes. Moreover, we provide the visualization feature affinity between the query and frame. As shown below, the query point (in blue) has high feature similarity with the same object the frame. We note that we use the pixel in the visual query as the source feature in this visualization, while we use the pixel in the target frame the source feature in our spatial transformer. The reason that if we visualize the feature similarity the latter manner, the response will cover the entire visual query, which is not informative.
14

---

0.8
0.6
0.4
0.2
Figure 6: Visualization feature similarity between the query and frame. The query point (in blue) high feature similarity with the target object in the frame.
Leaderboard Result. Our model VQLoC achieved state-of-the-art performance on the public leaderboard of Ego4D VQ2D benchmark.
LAP
Last submission
Rank Partioipent team
loU-D.25
1oU=0025 (1)
STAP
IAP (1)
sucoess
recoovrry %.
Meta Altribules
(1)
(1)
(1)
VQLOG (VQLeC)
0326
0:3228
00971
6943
42.107
hours
HostL
(e020.te
0.1966
02883
00021
15000
47 6575
30 29.8229
2mmrrhhe
Vanw
The visnncaatoon
1911
0,288
00758
1432
1262
43,2426
thi ago
Hoar_5970 Tesr Teem
(VCP1 haseline)
0.3340
00818
5820
34.0335
manihs
Masuraaattro
0209
0.1091
0.0056
07799
30
4152
11.5047
10 days ago
View
Figure 7: Ego4D VQ2D benchmark leaderboard the time of submission.
Error Bar. As shown in Table 8, provide the average value and the performance variance by running the model 3 times. The results demonstrate that VQLoC consistently achieves reasonable performance.
Table 8: Model performance and variances with multiple
Seed
tAP
2tAP
rec%
Succ.
42
0.31
0.22
47.05
55.89
0
0.30
0.23
45.64
57.32
0.31
0.22
47.73
55.42
mean
0.31
0.22
46.81
56.21
2c-5
2c-5
0.76
0.65
15


Reviews:
[{"review_id": "f7ywZZjfSS", "reviewer": "Reviewer_59Vz", "summary": "The paper proposes VQLoC, an end-to-end method for egocentric visual query localization based on a holistic understanding of the query-video relationship. The key component is a spatio-temporal transformer, which can effectively model the relationships between the query and frames. The presented approach can perform single-stage inference and achieve the winning entry on the leaderboard. Moreover, the method produces a set of models to balance the speed and balance, while the top-performing one is more advanced from both perspectives than the state-of-the-art. Moreover, the visualization shows that VQLoC is excellent at the visual query 2D detection task and can even deal with challenging cases such as clutter, occlusions, and motion blur.", "questions": "1. In the ablation study L299, using high-resolution frames (448 × 448) leads to way better results than the low-resolution ones (224 × 224). Is it possible to further increase the resolution for better performance, even if we can slightly sacrifice the clip length?\n\n2. It seems that the prediction head only gives framewise bounding box locations and confidence scores, but the VQ2D task requires a response track of the query object. Is there any mechanics in the head to make the predicted boxes consistent with the predictions from the temporal neighboring frames? \n\n3. According to Table 1, the recovery ratio and success rate improved significantly, but the AP only raised a little. Is it because the single-stage detection pipeline is weak in predicting precise bounding boxes?", "limitations": "The limitation is already discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper positively impacts the egocentric community, especially in episodic memory tasks. Unlike Moment Query and Language Query, the visual query task is more complicated in spatial-temporal localization, and the existing solutions are redundant. The paper provides a single-stage solution, and the chosen model is more accurate and efficient than the state-of-the-art. This will encourage more researchers to work on the valued research problem.\n\n2. The proposed architecture for the spatio-temporal transformer is able to effectively model the relationships between the query to each frame as well as between the frames. Therefore, the model can leverage rich semantics during feature embedding. Locally-windowed temporal self-attention is applied to improve the model efficiency without losing too much information. \n\n3. Code and the architecture details are attached to the paper, which helps people to re-implement the code. Also, the high-quality video in the supplementary material makes the paper clear and easy to follow.", "weaknesses": "1. My main concern is that although the paper aims to solve the Visual Query Localization task, all the experiments are conducted in the VQ2D setting. According to the definition of the Ego4D [13] paper, visual query localization can be done in the video domain (VQ2D) or the real-world coordinate (VQ3D). Therefore, it is not precise to only validate this method on VQ2D, and evaluating the proposed method on the 3D setup is highly recommended. Otherwise, the paper should revise the task as visual queries 2D localization.\n\n2. Further experiments could be conducted to improve the model further. For instance, the authors find the local windows\nof size 5 works the best in the set {5, 7, 9}, then they should experiment with smaller windows for the local optima of their hyper-parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive comments, and your acknowledgment of our potential impact to the community! We will address your questions as follows.\n\n1. **[Weakness Q1]** Our study intentionally focuses on VQ2D due to its unique challenges, e.g. the ‘needle-in-the-haystack’ problem, and the large variation between visual queries and their appearance in the video, as we discussed in L33-43 of the submission. While VQ3D is interesting,  it poses unique challenges that *are orthogonal to* VQ2D [1], i.e. difficulties in camera pose estimation, which should be solved separately and is beyond the scope of our work. However, we expect our improvements in VQ2D to propagate to VQ3D since 2D localization is a precursor to 3D localization. We will revise the task as visual query 2D localization as you suggested to make it more clear.\n\n[1] Mai, Jinjie et al. “Estimating more camera poses for ego-centric videos is essential for VQ3D.” ArXiv 2022.\n\n---\n\n2. **[Weakness Q2]** We experiment with smaller window sizes 3 and 1, where 1 means there is no temporal reasoning within the model. As shown below, we observe that window size 5 works the best. Moreover, without temporal reasoning (i.e., window size = 1), the performance drops dramatically, showing the effectiveness of the spatial-temporal transformer module we proposed.\n\n| Window Size | tAP | stAP | rec% | succ |\n|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Size 5 | **0.31** | **0.22** | **47.05** | **55.89** |\n| Size 3 | 0.27 | 0.17 | 42.26 | 48.14 |\n| Size 1 | 0.17 | 0.11 | 33.56 | 42.83 |\n---\n\n3. **[Questions 1]** While higher resolution could theoretically enhance the performance, our computation resources restrict us to the 448x448 resolution. We are currently training with batch size 3 on each GPU. If we intend to increase the resolution further, e.g. to 896x896, the batch size should be decreased to one-fourth, as the memory requirement of attention-based models grows quadratically. And in this case, we cannot even run on a batch size of 1. If there are GPUs with larger memory available, we believe it is possible to further increase the resolution. Moreover, we are running on a clip length of 30 frames. As the average response track is 3 seconds (15 frames), this clip length leads to a balanced ratio between positive and negative frames during training. Thus, decreasing the clip length in an effort to accommodate higher-resolution images may make training unbalanced, and the smaller batch size may also negatively influence the performance.\n---\n\n4. **[Questions 2]** Note that the prediction of each frame *is not isolated from nearby frames*. The proposed spatial-temporal transformer with local-windowed attention allows the information to be propagated across consecutive frames by establishing frame-to-frame correspondence, which implicitly makes the prediction smooth (L177-179 in the paper). Our experiments (Table 4 and our response to WQ2) and visualization in the original supplementary video demonstrate the effectiveness of using the spatial-temporal transformer to make the performance better and smooth.\n---\n\n5. **[Questions 3]** We note that our method demonstrates significant and consistent *relative* improvement in all metrics: 19% tAP, 16% stAP, 24% rec%, and 17% succ., when compared to the best baseline method. Among all the metrics, the stAP is the most challenging one, as it requires high precision in both temporal and spatial understanding. Thus, the absolute value of stAP is relatively low for all methods, as are the absolute differences in stAP between methods.\n\n"}, {"review_id": "RseuFPH3F2", "reviewer": "Reviewer_bR3g", "summary": "This paper tackles the problem of query localization of visually specified objects in egocentric human-activity videos. The main technical contribution for the solution is to design a single-stage transformer-based architecture to model the query-to-frame correspondence matching and frame-to-frame correspondence propagation. Ablation studies verify the effectiveness of the proposed components and comparisons show the superiority of the method in performance and speed with respect to prior works.", "questions": "None", "limitations": "YES", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The network is single-stage and end-to-end trainable, which alleviates the issues in existing multi-stage approaches that different stages are separate from each other for conducting non-differentiable stage-wise predictions. \n2) Model designs, including the frame-wise spatial transformer and the subsequent spatio-temporal transformer, are technically sound and easy to follow.\n3) Good performance is achieved by the proposed method and speed-accuracy trade-off is also taken into consideration.\n", "weaknesses": "1)\tThe technical contributions in terms of the model designs are short of novelty, and most of the building blocks can be viewed as simple adaptations from existing techniques used in methods for similar tasks (e.g., visual tracking, temporal sentence grounding, spatio-temporal video grounding, etc., can refer to STARK[1], LGI[2], STCAT[3], QD-DETR[4]). Apart from that, the unique challenges in the task of VQL, such as the egocentric characteristics presented in the videos, are less considered and are scarcely reflected by the model designs.\n2) A powerful pre-trained model DINOv2 is adopted as the visual backbone, which plays a crucial role for ensuring the success of the query-to-frame correspondence matching. However, no related discussions are included in the manuscript. What if the method gets rid of (by replacing it with a weaker but common one) the pre-trained DINOv2 model? Would it simply fail or drop significantly in performance? And it’s unclear how much gains are brought by the proposed architecture except for the utilization of a stronger feature backbone.\n3) According to the ablation studies, the proposed model heavily relies on a high input resolution to achieve decent localization results. This could be a negative factor hindering the VQL model from scaling up to a larger amount of training data.\n4) Some literal expressions, such as the writing of Section 3.3 could be reorganized to make it clearer.\n5) As shown by the temporal plot of predicted object-occurrence probabilities in Figure 5, there are still a lot of spurious responses with large magnitude (some are even larger than responses within GT) outside the target temporal region, which implies the current model’s deficiency in accurate temporal localization to some extent.\n\n[1] Bin Yan, et al. \"Learning spatio-temporal transformer for visual tracking.\" In ICCV, 2021.\n[2] Mun, Jonghwan, et al. \"Local-global video-text interactions for temporal grounding.\" In CVPR, 2020.\n[3] Jin, Yang, et al. \"Embracing consistency: A one-stage approach for spatio-temporal video grounding.\" In NeurIPS, 2022.\n[4] Moon, WonJun, et al. \"Query-dependent video representation for moment retrieval and highlight detection.\" In CVPR, 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments. We will address your concerns as follows.\n\n1. **[Novelty and related work]** We respectfully disagree with the comment. Our method is not a simple adaptation of the existing methods. Its novelty is *appreciated by other reviewers*, and we reiterate our contributions here. We *take the challenges and unique characteristics of egocentric videos into consideration* and propose three ways to address them: i) The **spatial transformer** finds the correspondence between the frame-query to identify regions that are similar to the query. This is designed to **handle the large appearance variation** of the query object with its appearance in the egocentric video; ii) The **spatial-temporal transformer** performs temporal reasoning within locally windowed consecutive frames to propagate spatial correspondences identified earlier. This local window is necessary to **handle the fast head motion** of the egocentric videos. iii) The **anchor-based bounding box prediction with hard negative mining** during training is designed to **handle the small scale and rapid appearance of the objects** in the egocentric videos. These egocentric characteristics and proposed techniques differentiate our task/model from previous visual tracking/grounding tasks and solutions. And our performance without these components drops 35%, 80%, and 40%, respectively. In contrast, for frame-query feature fusion, STARK and STCAT use concatenation; LQI uses Hadamard product. For temporal reasoning, STARK updates the template; LGI, STCAT, and QD-DETR perform reasoning on the entire clip. For bounding box (or target frame) prediction, all mentioned works use non-anchor-based loss without hard negative mining. Since STARK is a SOTA tracker and can be directly used to localize the visual queries, it is tested as a baseline and demonstrates degenerated performance on the VQL task (Table 1). We will add more discussion and cite these papers in the future version as you suggested.\n---\n\n2. **[Backbone]** Good question. Our method **does not fail** when we replace DINOv2 with a common CLIP backbone, which is widely used in video-related [1] or non-video [2] tasks.  The performance with the CLIP backbone is considerably better than the baseline models and demonstrates that our model still works with alternate backbones. Note that large-scale pre-training is a valuable component of most existing methods [3] (whether it is CLIP or DINOv2), and it is not surprising that it plays an important role here. However, the choice of DINOv2 as the particular backbone for this task is also a part of our contribution (see L155-157). As contrastive-learning-based backbones, including DINOv2, demonstrate good *semantic correspondence properties* [4], it is useful for performing the visual query task, especially for handling *large visual differences* between the query and its appearance in the videos. Besides, the backbone is not the only important aspect of our model. Even with a strong DINOv2 backbone, we demonstrate in our ablations that the performance deteriorates significantly without our other proposed components, as illustrated in the reply to your first question. In short, each component of our model is designed to work well for the VQL task, and i) the performance cannot be solely attributed to the backbone and ii) our CLIP test demonstrates some versatility to the backbone.\n\n| Backbone | Resolution | tAP | stAP | rec% | succ |\n|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n|DINOv2|448| 0.31 | 0.22 | 47.05 | 55.89|\n| CLIP | 336 | 0.28 | 0.20 | 44.67 | 53.80 |\n\n*Note that the highest resolution that CLIP supports is 336. \n\n[1] Lin, Ziyi et al. “Frozen CLIP Models are Efficient Video Learners.” ECCV 2022.\n[2] Shen, Sheng, et al. \"How much can clip benefit vision-and-language tasks?.\" ICLR 2022.\n[3] Wang, Yi et al. “InternVideo: General Video Foundation Models via Generative and Discriminative Learning.” ArXiv 2022.\n[4] Hu, Yingdong et al. “Semantic-Aware Fine-Grained Correspondence.” ECCV 2022.\n\n---\n\n3. **[Input resolution]** We would like to clarify that our model is **not dependent on high-resolution inputs**. For comparison, we note that baselines, i.e. SiamRCNN, NFM, and CocoFormer, work on original video resolution, which is 1200 pixels. In contrast, **our method works on downsized resolution** of 448 pixels, yet outperforms the baselines. This indicates that our method is effective even though it uses lower resolution and is not over-reliant on high-resolution inputs. Besides, the performance drop with using a smaller resolution (i.e. 224) is expected and is not unique to our method [5], as the resolution of visual features will also be halved. Specifically, with 224 resolution, the average object bounding box size is about 35 pixels (i.e., 15% of the image length), which is challenging to identify even for humans. We will better emphasize this point in our revised version.\n\n[5] Bertasius, Gedas et al. “Is Space-Time Attention All You Need for Video Understanding?” ICML 2021.\n\n---\n\n4. **[Writing]** We will revise Sec.3.3 to improve clarity.\n---\n\n5. **[Spurious responses in Fig.5]** It is crucial to note that VQL aims to identify the ‘*latest*’ appearance track of the query object. This means that the ground-truth is annotated only with the latest appearance of the query object and ignores all prior occurrences in the video. Thus, there is only a single peak in the visualized ground-truth. However, since the query object itself may appear multiple times earlier in the video, our model will attempt to identify all of them (not just the latest) — these correspond to the other peaks (or “spurious responses” as noted) and is *the expected and correct behavior* of the model. To better demonstrate the point, we include additional visualization (in the rebuttal pdf), which shows all identified peaks. The results indicate that our method can identify the query object consistently."}, {"review_id": "BQSOZ60IN5", "reviewer": "Reviewer_YaTy", "summary": "The paper introduces a new approach to address the visual query localization problem in egocentric videos. The major contribution is that the proposed method, a single-stage model, simplifies the previous multi-stage frameworks and eliminates the need for off-the-shelf object detectors, tracking, and similar components by utilizing an end-to-end trainable model. Notably, the proposed method has achieved the top position in the Ego4D VQ2D challenge as of the submission date, demonstrating its effectiveness. Besides, the method demonstrates a considerable enhancement in inference speed, further highlighting its advantages.", "questions": "Please refer to the previous section. ", "limitations": "The authors addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel and technically sound method that formulates the visual query localization task as a unified framework. By taking a video and a template query image as input, the proposed method can directly produce the track of bounding boxes, eliminating the need for off-the-shelf object detectors and tracking methods. This is achieved by decoupling the task into spatial reasoning (finding per-frame responses) and temporal reasoning (consultation within the clip), followed by a prediction head (box&score) upon the learned features. In summary, the idea behind the method is both original and rational.\n\n2. The proposed method demonstrates notable computational efficiency, a crucial aspect for real-world applications. By simplifying previous multi-stage frameworks into a one-stage process, the method achieves a remarkable 10x improvement in speed while maintaining a satisfactory level of performance. This advancement represents a significant stride towards practical applications.\n\n3. The paper is well-written. Readers with a relevant background will have no difficulty following and comprehending the content.\n\n4. The paper presents solid experimental results, achieving the top position on the Ego4D VQ2D challenge leaderboard at the date of submission.", "weaknesses": "1. L12-L13: \"Our experiments demonstrate that our approach outperforms prior VQL methods 12 by 20% accuracy while obtaining a 10× improvement in inference speed\". Without delving into the entire paper, a glance at Table 1 and Figure 4 might cause confusion regarding the reported results, as the speed does not appear to be 10x faster than STARK [51] (if my understanding is correct, this is an adapted tracking approach upon the proposed framework). It would be beneficial to include some explanation and details in the caption to prevent such confusion.\n\n2. It is unclear how the number of frames (T=30) per clip is determined? Is this choice based on the limit of computational resources or driven by experimental performance considerations. If the latter, it would be helpful to know if any ablation studies have been conducted to investigate its aspect.\n\n3. Discussion on the long videos. The input is fixed to a maximum of 30 frames (~6 seconds with a frame rate of 5fps). To handle longer videos, the proposed solution is to chunk the untrimmed video into fixed-length clips. In this case, is there an additional step to ensure smooth predictions between these clips, such as incorporating tracking between them? Alternatively, another solution to address long videos is to use a streaming mode. Suppose we have processed the past 30 frames $\\{ t_{i-29}, …, t_{i-1}, t_i \\}$, we would only need to process the new incoming frame $t_{i+1}$. Spatial transformer naturally lends itself to this scenario since it is computed at frame level. The spatio-temporal transformer might be more complex. But it operates within a bounded window, which suggests that it might not require significant re-computation or adjustments for the streaming mode. It would be valuable to have the authors' insights and comments on this matter.\n\n4. L177-L193: The symbol $w$ is used ambiguously, representing both the spatial width and the temporal window.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your insightful comments and your suggestions on the paper details. We will address your questions as follows.\n\n1. **[Speed comparison]** We apologize for the confusion. The speed improvement was not in comparison to STARK but the prior state-of-the-art VQL methods (SiamRCNN / NFM / CoCoFormer). We will clarify this in our writing.\n---\n\n2. **[Clip length]** The clip length (T=30) was chosen based on the computational resource limits and the average response track length (~15 frames). This selection offers a *balanced ratio* between positives and negatives within each clip during training, ensuring steady training. Since we use local-windowed attention to handle fast head motion in egocentric videos, extending the clip length beyond T=30 would not theoretically enhance performance. For example, with a window size of 5 and 3 layers of local-windowed attention, frames more than approximately 7 frames away will not be attended to. We will clarify this in the later version.\n---\n\n3. **[Long videos]** We appreciate the insightful comments. As we use local-windowed attention to perform spatial-temporal reasoning, the smoothness of predictions within the several consecutive frames is implicitly learned. As you suggested, it is still possible that the prediction is not smooth enough between the two clips, e.g. the last frame of a clip, and the first frame of the following clip. The streaming mode you suggested is interesting and may further improve the smoothness at the intersection of clips. Further, the local-windowed attention can be used in the streaming mode with slight modification. It would be interesting to explore this in future work. Besides, we think inference in a streaming mode is especially important for egocentric vision, as it usually requires real-time inference. We will add more discussion in the later version.\n---\n\n4. **[Writing]** Thanks for pointing this out. We will fix it."}, {"review_id": "PVPxknCkPo", "reviewer": "Reviewer_8x9A", "summary": "The author proposes a new single-stage end-to-end method for visual query localization. The proposed method builds a holistic understanding of query-video relationship, then spatial-temporal localization is performed. The proposed achieves major performance and speed gain over previous methods on major benchmark datasets.", "questions": "Please mainly see the weaknesses section for details.", "limitations": "I think the author has adequately addressed the limitations. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general, the proposed method is simple and performs well on the major egocentric Ego4D dataset. The proposed method utilizes simple pipelines with cross attention/Transformer, however, achieves state-of-the-art results. In general, I am satisfied with the paper. ", "weaknesses": "However, I have some concerns about the paper.\n\n1. The author only uses the Ego4D dataset as the benchmark for egocentric dataset. Since it is a new task, it is OK to only have few baselines/datasets cause it is a brand new task. However, It would be better if the author's method can be applied to other tasks/sub-tasks like visual tracking datasets (or modify the tracking datasets \n\n2. More experiment details should be reported. Because it is an end-to-end framework. Though the author reports the FPS of the model, I think the author could report the more detailed metric like FPS/GPU memory. I think the proposed method could have larger GPU memory consumption/larger FLOPs while with fast execution time. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your helpful feedback, and we will address your concerns as follows.\n\n1. **[Benchmark]** Thank you for your valuable suggestion. In this paper, we focus on **the special properties of the visual query localization task on egocentric videos** (L33-43), e.g. drastic head motion, large variation between the query object and its appearances in the video, open-set queries, and long videos. Visual tracking, on the other hand, is a different task compared with VQL. Specifically, visual tracking usually requires the model to work on shorter videos with slower camera viewpoint changes, as well as small differences between the visual query and their future appearance in the video. Thus, your idea to apply or adapt our method to visual tracking datasets is intriguing, but we are unaware of appropriate datasets that satisfy the above properties. To the best of our knowledge, Ego4D is the only dataset that has these properties to test on. We will add these considerations to our discussion to highlight potential applications and adaptations.\n---\n\n2. **[More details]** Thank you for your suggestions to provide the FPS/GPU memory. In detail, our method observes 2 FPS/GPU per GB. We note that it's difficult to compute this FPS/GPU metric for the baselines, as their GPU usage is different for multiple stages of the baselines, i.e. detection, query comparison, and tracking. For an apples-to-apples comparison, we verified that the current FPS is evaluated with similar “*maximum*” GPU usage between our method and the baselines. Moreover, our method is *more parallelizable* than the baselines. If we are provided with larger GPU memory, our FPS will also increase accordingly for a given (video, query) pair.  This is not the case for the baselines as some stages like tracking cannot be parallelized (i.e., tracking has to happen sequentially from one frame to the next).\n"}, {"review_id": "nW2vmOp8en", "reviewer": "Reviewer_4CTN", "summary": "The paper proposes VQLoC, an end-to-end trainable framework for Visual Query Localization (VQL) on long-form egocentric videos. Compared with Ego4D's multi-stage approaches, VQLoC proposes a single-stage process that efficiently localizes visually specified objects. It establishes both query-to-frame and frame-to-frame relationships for spatial-temporal localization. The proposed method achieves new SOTA performance on the public leaderboard with a notable improvement.", "questions": "(P6 L207) Why not trim complete negative video clips to augment the data for the training?\n \n(P6 L228) Are the top K anchors selected from the entire video clip n' and mixed with anchors from n? I need clarification here since the loss is frame level, but the text is about video level.\n \nEgo4D Episodic Memory also has similar challenges like VQ3D, NLQ, and MQ. Is it possible to adopt similar strategies for these tasks by changing the query encoder and predictor? It will be quite interesting to develop a unified framework for video query tasks.", "limitations": "I don't think the authors can provide more ablations for a large benchmark like VQ2D within the rebuttal date. Overall, this work has pushed VQ2D task forward with a remarkable step. However, more complete ablations will make this paper stronger in the future. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "VQLoC can streamline the VQ2D localization process. The method is end-to-end trainable and achieves faster inference speed, as shown in Figure 4.\n \nThe results are verified on the public server. Furthermore, from the quantitative ablation study, it seems like the boost comes from the model design rather than parameter tuning or training tricks, which is good.\n \nThe related work covers the field well. And the authors made comparisons to previous baselines in terms of size, speed, and performance.", "weaknesses": "My major concern is the completeness of the ablation study. This paper can be stronger if more ablations can be done.\n \nFor the spatio-temporal transformer, Tab. 4 only shows the window size = 5 gives the best performance among the numbers greater than 5. It's not convincing 5 is the best choice. How about a window size smaller than 5? It's possible that if you don't do any frame-to-frame correspondence, i.e., a window size of 1/0 gives even better performance.\n \n Also, there are multiple thresholds in the methodology. Are the results sensitive to the selection of those hyperparameters? For example,  the peak response plot in Figure 5 looks noisy, so it's not sure whether an empirical choice of φ makes sense. The same also applies to the anchor box threshold theta.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed and helpful suggestions, and we are glad to see the highly positive comments. We will address your questions below. \n\n1. **[Local window size ablation]** We experiment with smaller window sizes 3 and 1, where 1 means there is no temporal reasoning within the model. As shown below, we observe that window size 5 is better than size 3. Moreover, without temporal reasoning, the performance drops dramatically, showing the effectiveness of the spatial-temporal transformer module we proposed.\n\n| Window Size | tAP | stAP | rec% | succ |\n|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Size 5 | **0.31** | **0.22** | **47.05** | **55.89** |\n| Size 3 | 0.27 | 0.17 | 42.26 | 48.14 |\n| Size 1 | 0.17 | 0.11 | 33.56 | 42.83 |\n\n---\n\n2. **[Selection of hyperparameters - anchor assignment threshold]** Among the other hyperparameters,  the anchor box threshold influences the training a lot. The reason is that it is highly related to the assignment of positive and negative anchors. As the objects are usually **small** on egocentric videos (1/7 of the image resolution on average), anchor box IoU threshold 0.5 (commonly used in detection papers) does not work well since it is harder for the anchor boxes to intersect with small objects with such a high IoU. This is the reason for picking a small IoU of 0.2. We will discuss it in the later version.\n---\n\n3. **[\"Noisy\" temporal plot in Fig. 5]** The peak response in Fig. 5 looks “noisy” as it shows the results on very long videos, e.g. a video of 1000 frames. Note: VQL aims to identify the ‘*latest*’ appearance track of the query object, and the ground-truth is annotated accordingly (and ignores earlier appearances of the visual query in the video). Thus, there is only a single peak in the visualized ground-truth. However, the query object itself **may appear multiple times earlier** in the video, and our model will attempt to identify all of them (not just the latest) — these correspond to the other peaks and *is the expected and correct behavior* of the model. To better demonstrate the point, we include additional visualization (in the rebuttal pdf), which shows the all identified peaks. And the results indicate that our method can identify the query object consistently.\n---\n\n4. **[Selection of hyperparameters - post-processing threshold]** We select the threshold based on the validation performance following the approach from the baseline methods (SiamRCNN and CoCoFormer). We agree that this is a limiting factor and can be improved in the future, but this is orthogonal to our core contributions. Moreover, our results demonstrate that this threshold determined on the validation set can generalize to the test set without a performance gap (see Table 1).\n---\n\n5. **[Use negative clips]** We generally believe trimming negative clips can be a promising technique to avoid false positives.  However, this is currently not possible on the Ego4D VQL dataset since *it annotates only the latest object appearance* (i.e., to answer “where did I *last* see [query]?”), while ignoring the earlier appearance of the object in the video. This means that the clips before the ground-truth response track may not all be negatives.\n---\n\n6. **[Anchor selection]** The loss $L_{img}$ (L210) is computed per anchor of the image, not over an image as a whole. We will better clarify it in the next version. The top K hard negative anchors are chosen from frames in video $n$ as well as frames from every other video $n’$ (see L229-231).\n---\n\n7. **[Unified framework]** Yes, we completely agree that these egocentric vision tasks can share similar challenges, caused by the characteristics of egocentric videos, and it is likely that these tasks have synergy. For example, NLQ and MQ require similar localization capabilities. VQ3D requires both 2D and 3D understanding of objects and scenes. Joint training with multiple tasks might improve the performance of each task. The key to the unified framework is how we handle inputs with different modalities and propose a jointly trainable end-to-end approach. It is also necessary to handle the unique properties of each task, e.g. camera pose estimation is the bottleneck of VQ3D [1]. This is an interesting and promising research direction. However, note that this is orthogonal to our core contribution of efficiently localizing a specific query object instance within the video.\n\n[1] Mai, Jinjie et al. “Estimating more camera poses for ego-centric videos is essential for VQ3D.” ArXiv 2022."}]

Author rebuttal:
We appreciate the insightful feedback and detailed suggestions from the reviewers. It’s great to see the highly positive comments, including “positively impacts the egocentric community” (R-59Vz), “The paper introduces a novel and technically sound method” (R-YaTy), “is simple and performs well” (R-8x9A), and “VQLoC can streamline the VQ2D localization process” (R-4CTN). We will address the questions of each reviewer separately. Besides, we add additional visualization in the rebuttal pdf to answer the "noisy" peak problem.

Meta-review: This work proposes a novel single-stage solution for visual query localization. Through the single-stage design, the proposed method achieves a 10x speedup while maintaining satisfactory performance.

In the process of rebuttal and discussion, reviewers raise multiple questions regarding 2D/3D settings,  pre-trained backbone, detailed parameters and more ablation studies. All reviewers agree that the rebuttal well addressed their concerns.

After the discussion period, this submission receives 3 “Accept”, 1 ”Weak Accept” and 1 “Borderline Accept”.

It is recommended for acceptance and the authors should take the comments into consideration to refine the paper in the camera-ready version.
Decision: Accept (poster)