SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training. VQLoC is trained with the loss  $L_{img} = L_{bbox} + \lambda_p \cdot L_{prob}$  on all clip frames for supervising bounding box regression and query object occurrence probability prediction. For each refined anchor box  $\hat{\mathbf{b}} \in \hat{\mathcal{B}}$ , we calculate the intersection of union (IoU) between the ground-truth query object box and the original corresponding anchor box  $\mathbf{b}$ . If the IoU is higher than a threshold  $\theta$ , it will be assigned as a positive box, denoting the anchor box region captures the query object well. We apply the bounding box loss as  $L_{bbox}(\hat{\mathcal{B}}, \mathbf{b}) = \Sigma_{\hat{b} \in \hat{\mathcal{B}}^p} L_{reg}(\hat{\mathbf{b}}, \mathbf{b})$ , where  $\mathbf{b}$  is the ground-truth box for the query object, and  $\hat{\mathcal{B}}^p \subseteq \hat{\mathcal{B}}$  is the set of positive boxes. Specifically,  $L_{reg}$  is a combination of the  $L_1$  loss and the generalized IoU (GIoU) loss [41]:
$$
L _ {r e g} (\hat {\mathbf {b}}, \mathbf {b}) = \| \hat {\mathbf {b}} _ {\mathbf {c}} - \mathbf {b} _ {\mathbf {c}} \| + \| \hat {\mathbf {b}} _ {\mathbf {h}} - \mathbf {b} _ {\mathbf {h}} \| + \| \hat {\mathbf {b}} _ {\mathbf {w}} - \mathbf {b} _ {\mathbf {w}} \| + \lambda_ {g i o u} \cdot L _ {g i o u} (\hat {\mathbf {b}}, \mathbf {b}),
$$
where  $\mathbf{b_c}$ ,  $\mathbf{b_h}$  and  $\mathbf{b_w}$  are the center coordinate, height, and width of the bounding box, and  $\lambda_{giou}$  is a weight that balances the losses [5]. To get the query object probability loss, we assign the probability labels to the positive and negative anchor boxes with 1 and 0, respectively. We denote the assigned probability labels as  $\mathcal{P}$ . The query object occurrence probability loss is defined as  $L_{prob} = L_{bce}(\hat{\mathcal{P}},\mathcal{P})$ , which is the binary cross-entropy (BCE) loss.
Since the target videos are long, false positive prediction, where the model wrongly identifies other objects as the query object, is one of the bottlenecks that limit the temporal accuracy of the model. To prevent this, we perform hard negative mining (HNM) (similar to [13, 53, 52]). Given a mini-batch of  $N$  videos, we diversify the negative anchors for each batch element  $n$  by treating frames from every other video  $n' \neq n$  as a negative. After expanding the pool of negatives, we calculate the query occurrence probability for all the negative anchors using Eqn. 3 and sample the top K anchors with the largest BCE loss as hard negatives (i.e., negatives which our model thinks are positives). We then train the model using the positives from the same batch element and the hard negatives sampled across all batch elements with the BCE loss. We keep the ratio between positives and negatives as  $1:3$  during training.
Inference. During inference, we get the prediction for each frame by choosing the refined anchor box with the highest query object probability. We reject low-confidence predictions by applying a threshold  $\varphi$  to the predicted query object probability. See supplementary for more details.
First, we describe our implementation details, the dataset and evaluation metrics. We then quantitatively compare VQLoC with prior methods, and present ablation studies that examine the impact of various design choices in VQLoC.
Table 1: VQL results on Ego4D VQ2D benchmark: We compare our method VQLoC against the baselines. The test results are obtained from the challenge leaderboard.
Validation	Test	FPS
tAP25	stAP25	rec%	Succ.	tAP25	stAP25
SiamRCNN [13]	0.22	0.15	32.92	43.24	0.20	0.13	3
NFM [52]	0.26	0.19	37.88	47.90	0.24	0.17	3
CocoFormer [53]	0.26	0.19	37.67	47.68	0.25	0.18	3
STARK [54]	0.10	0.04	12.41	18.70	-	-	33
VQLoC (ours)	0.31	0.22	47.05	55.89	0.32	0.24	36
Figure 4: Accuracy-speed trade-off: The radius of circles notify model sizes.
Implementation Details. We train the model with  $448 \times 448$  resolution videos obtained by resizing the source videos along the longer edge and applying zero-padding to the shorter edge for each frame. We set the loss coefficients  $\lambda_{p} = 1$  and  $\lambda_{giou} = 0.3$ . The positive anchor box threshold is  $\theta = 0.2$  due to the small scales of query objects in videos.. VQLoC is trained for 60,000 iterations with batch size 24, utilizing the AdamW optimizer [28] with a peak learning rate of 0.0001 and a weight decay of 0.05. A linear learning rate scheduler and a warmup of 1,000 iterations are employed. We adopt DINOv2's ViT-B14 as the backbone and train on clips of 30 frames with a frame rate of 5 fps. The clip length is selected to keep the balance between positive and negative frames. Clips are randomly selected to cover at least a portion of the response track. We define anchor boxes on the feature map with spatial resolution of  $16 \times 16$ , each element has 12 anchor boxes with 4 base sizes  $(16^{2}, 32^{2}, 64^{2}, 128^{2})$  and 3 aspect ratios (0.5, 1, 2). During training, we apply random data augmentations to the clips, such as color jittering, horizontal flipping, and random resized cropping. We train and evaluate on RTX 6000 GPUs.
Dataset. We train and evaluate our model on the Ego4D dataset [13]. Ego4D is a large-scale egocentric video dataset designed for first-person video understanding and contains open-world recordings of day-to-day activities performed in diverse locations. We use the VQ2D task annotations from the episodic memory benchmark in Ego4D, which consists of  $13.6k / 4.5k / 4.4k$  queries annotated over 262/87/84 hours of train / val / test videos. The average target video duration is  $\sim 140$  seconds and the average response track length is  $\sim 3$  seconds, making this a challenging benchmark for the VQL task. Ego4D is, to the best of our knowledge, the only publicly available dataset for VQL.
Metrics. In our evaluation, we adhere to the official metrics defined by the Ego4D VQ2D benchmark. We report two average precision metrics: temporal AP  $(\mathrm{tAP}_{25})$  and spatio-temporal AP  $(\mathrm{stAP}_{25})$ , which assess the accuracy of the predicted temporal and spatio-temporal extents of response tracks in comparison to the ground-truth using an Intersection over Union (IoU) threshold of 0.25. Additionally, we report the Recovery (rec%) metric, which quantifies the percentage of predicted frames where the bounding box achieves at least 0.5 IoU with the ground-truth. Lastly, we report the Success metric, which measures whether the IoU between the prediction and the ground-truth exceeds 0.05. Apart from VQ2D evaluation metrics, we also report the inference FPS of each method as a measure of computational efficiency.
Baselines. We compare our approach against the following baselines. All baselines are trained on the Ego4D dataset for a fair comparison.
- SiamRCNN [13]: This is the official baseline for VQL introduced in Ego4D. It employs a three-stage pipeline with frame-level detection [50], latest peak detection, and object tracking [3].
- NFM [52]: It employs the three-stage pipeline introduced in SiamRCNN and improves the frame-level detector training by sampling background frames as negative to reduce false positives. This was the winning entry of the Ego4D VQ2D challenge held at CVPR 2022.
- CocoFormer [53]: It improves the frame-level detection architecture from SiamRCNN by reasoning about all object proposals in a frame jointly (as a set) using a transformer module.
- STARK [54]: It is an end-to-end method for long-term visual object tracking. We adapt the method with our object occurrence prediction head to enable it to work on long-form videos.
Response Track (Zoom-in with blue box)
Figure 5: Qualitative analysis of VQLoC: In each row, we visualize an example prediction from VQLoC. For each example, we show the visual query, four frames from the predicted response track, and a temporal plot of the predicted object-occurrence probabilities (y-axis) for each video frame (x-axis). We highlight the ground-truth and predicted boxes in green and yellow, respectively. We also provide the zoom-in view of the predicted response on the top-right corner of each frame. The temporal plot additionally shows the detected peaks in blue and the ground-truth response window (which is only annotated for the latest appearance of query objects) in red. VQLoC is able to localize query objects under challenging conditions such as clutter (row 1), partial-to-near-complete occlusions (row 2), and motion blur with significant viewpoint variations (row 3). We show a failure case in row 4 where VQLoC fails to distinguish between two cables with similar appearances. Note that the temporal plot in the last column shows the object occurrence probability on the entire video, while the ground-truth is only annotated for the last occurrence.
The results are shown in Table 1. Our method VQLoC outperforms prior works on all metrics and sets the state-of-the-art for this task. VQLoC relatively improves over the next-best baseline (CocoFormer) by  $19\% \mathrm{tAP}_{25}$ ,  $16\% \mathrm{stAP}_{25}$ ,  $25\%$  recovery, and  $17\%$  success. Furthermore, the running speed of VQLoC is  $10\times$  faster than the three-stage VQL methods. Besides, we also demonstrate that directly applying tracking methods, i.e. STARK, doesn't work as well for VQL. We conjecture the reason is that STARK concatenates the query features and frame features without differentiating them. In this manner, the concatenated features, which are used to regress the localization results, lose the spatial correlation with input frames. While this design might work for normal videos where the camera viewpoint changes slowly and smoothly, its problem is exacerbated in egocentric videos where the camera moves dramatically and the object specified by the visual query is quite different from its appearance in the video. As shown in Fig. 4, VQLoC shows a reasonably good balance on the speed-accuracy trade-off. At 80 FPS, we match the best baseline performance; at 33 FPS, we outperform the best baseline by a healthy margin. We present a qualitative analysis in Fig. 5.
We present an ablation study on each block of VQLoC, examining and evaluating their impact within the structure of our model.
Backbone Size and Input Resolution. We explore the impact of using different backbone sizes and input spatial resolutions. We test the following alternatives: i) using a smaller ViT-s14 backbone with  $448 \times 448$  resolution, and ii) using the original ViT-b14 backbone but a smaller  $224 \times 224$  resolution. Specifically, the input with spatial resolutions  $448 \times 448$  and  $224 \times 224$  result in features with spatial size  $32^2$  and  $16^2$ , respectively. As shown in Table 2, the model with the largest input size and the largest backbone achieves better results. However, experiments demonstrate that using a larger backbone only brings marginal gains (comparison between the last two rows). In contrast, when inputting the model with low resolution, the performance drops dramatically. Specifically, the ratio of  $tAP_{25} : stAP_{25}$  drops from  $66\%$  to  $45\%$ . This decrease reflects the huge degeneration of
Table 2: Ablation on the backbone size choices. Res. denotes the spatial resolution of the inputs.
Backbone	Res.	tAP25	stAP25	rec%	Succ.
ViT-b14	224	0.22	0.10	32.18	45.84
ViT-s14	448	0.28	0.18	37.71	52.18
ViT-b14	448	0.29	0.19	38.58	53.12
Table 4: Ablation of spatio-temporal transformer. Win. is the window size, and glb. means global.
ST Refine.	Win.	Layers	tAP25	stAP25	rec%	Succ.
ST-TX	glb.	3	0.08	0.03	12.49	15.73
ST-TX	9	3	0.27	0.18	37.98	51.08
ST-TX	7	3	0.27	0.19	36.97	50.21
ST-TX	5	3	0.29	0.19	38.58	53.12
Table 3: Ablation of finding frame-query correspondence. X-TX denotes our spatial transformer.
Fusion	Layers	tAP25	stAP25	rec%	Succ.
Conv	-	0.22	0.13	32.27	44.95
X-TX	1	0.29	0.19	38.58	53.12
X-TX	3	0.29	0.20	37.91	53.74
Table 5: Ablation on the prediction head and loss function. FL denotes focal loss.
Anchor	Loss	tAP25	stAP25	rec%	Succ.
×	BCE	0.17	0.10	34.23	47.60
✓	BCE	0.12	0.07	40.23	42.27
✓	FL	0.29	0.19	38.58	53.12
✓	BCE+HNM	0.31	0.22	47.05	55.89
localization accuracy. Since the query objects in the videos are usually small, the low-resolution inputs and feature maps may not capture their details clearly. This matches our intuition that accurate localization requires relatively high-resolution features.
Query-Frame Correspondence. We study different methods for establishing the correspondence between the query and the frame. Specifically, we compare against a model variant that aggregates the spatial information from the query using convolutions. In detail, we apply downsample convolutions on the query features to obtain a 1-D feature vector (with  $1 \times 1$  spatial resolution). It then tiles this feature to obtain a  $H \times W$  feature map and concatenates it with the frame features. Finally, we use convolutions to obtain the query-to-frame correspondence features. Importantly, this does not use our Spatial TX module for establishing this correspondence. As shown in Table 3, the performance using this naive fusion method is much worse (rows 1 vs. 2), highlighting the value of establishing correspondences using our spatial transformer. We also experiment with using more transformer layers in our spatial TX module (rows 2 vs. 3 in Table 3), but this provides minimal gains, likely due to the strength of the image backbone features.
Spatio-Temporal Transformer Designs. Next, we study the role of using local temporal windows in our spatio-temporal TX block in Table 4. We experiment with a global window, and local windows of sizes  $\{5,7,9\}$ , with 5 being our default choice. Using a global window (row 1) significantly reduces the performance since the object's location, pose and appearance change dramatically over time in egocentric videos, making it hard to establish long-range dependencies. Instead, focusing on the local temporal window with several consecutive frames makes the feature propagation easier, establishing useful short-horizon dependencies. Our experiments show that the a window size 5 works best, where 5 consecutive frames span 1 second for the 5 FPS videos.
Prediction head and Loss Function. Finally, we study the design of using anchors in the prediction head and the loss functions used to supervise the anchor probabilities in Table 5. Not using anchors (row 1) performs worse than our models which use both anchors and advanced losses. Our model with anchors performs poorly with the BCE loss due to the data imbalance between positive and negative anchor boxes (row 2). The reason is that even positive frames will introduce hundreds of negative anchors, which makes the anchor-level ratio between negatives and positives much higher than the ratio in frame-level. Using focal loss or hard negative mining improves significantly over using only the BCE loss by accounting for the data imbalance (rows 3 and 4 vs. row 2). We observed that hard-negative mining (HNM) works better than focal loss (rows 3 vs. 4). Unlike HNM that only focuses on hard negatives, focal loss additionally focuses on hard positives, where the object is annotated in the training data but may be difficult to recognize due to effects like severe occlusions (e.g., only  $10\%$  of the object is visible). Therefore, these hard positives may provide harmful training signals since they are not reliably distinguishable from negatives.
Backbone Choice. We also ablate the backbone choice by replacing the DINOv2 encoder with a CLIP-pretrained ViT [39]. Note that the highest resolution that CLIP supports is  $336 \times 336$ . Experiments demonstrate with  $0.28\mathrm{tAP}_{25}$ ,  $0.22\mathrm{stAP}_{0.20}$ ,  $44.67\mathrm{rec\%}$  and  $53.80\mathrm{Succ}$ . The results are comparable with using DINOv2 backbone, showing the versatility of VQLoC to the backbone choice.
Table 6: Model performance on different scales of objects in the videos.  $s$  and  $l$  stands for small (1941 samples) and large object (2336 samples), respectively.
Method	Scale	tAP25	stAP25	rec%	Succ.
CocoFormer [53]	s	0.206	0.127	40.804	32.583
VQLoC	s	0.213	0.138	44.719	33.738
CocoFormer [53]	l	0.338	0.271	56.164	40.737
VQLoC	l	0.454	0.387	67.680	53.635
We propose VQLoC, a single-stage and end-to-end trainable method for the Visual Query Localization (VQL) problem in long-horizon videos. VQLoC first builds a holistic understanding of the query-video relationship, and leverages this understanding to localize the query in a single shot. Our key insight lies in how we establish the spatio-temporal correspondence between the query and video features. Specifically, we build the query-to-frame correspondence for each video frame and then propagate these over time to nearby frames by leveraging the temporal smoothness of videos. Compared with prior stage-wise methods, VQLoC not only demonstrates better spatiotemporal localization accuracy but also improves the inference speed significantly by avoiding explicit comparison between region proposals and the query. VQLoC also achieves the top performance on the Ego4D VQ2D challenge leaderboard. In future work, we plan to develop systems to perform in-depth object-level understanding based on our model's VQL predictions.
Limitation. Similar to prior works, VQLoC is trained in a supervised manner, requiring a large number of annotations for training. Besides, the hyperparameters, i.e. local window length, might need to be tuned for training or inference on other datasets with different FPS.
Acknowledgement. UT Austin is supported in part by the IFML NSF AI Institute. Kristen Grauman is paid as a research scientist by Meta.
[1] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, 2021.
[2] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Fully convolutional siamese networks for object tracking. In ECCV Workshops, 2016.
[3] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Know your surroundings: Exploiting scene information for object tracking. In European Conference on Computer Vision, 2020.
[4] Joydeep Biswas and Manuela M. Veloso. Episodic non-markov localization: Reasoning about short-term and long-term features. 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3969-3974, 2014.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ArXiv, abs/2005.12872, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[7] Ding-Jie Chen, He-Yen Hsieh, and Tyng-Luh Liu. Adaptive image transformer for one-shot object detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12242-12251, 2021.
[8] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-13618, 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
Visual Query Localization (VQL) has broad benefits to AR and robotics applications. For example, VQL is the basis for developing super-human episodic memory systems that can retrieve information from past human experiences. It has the potential to assist and strengthen the long-term reasoning capability of humans. Furthermore, it can be vital for embodied agents/robots to learn from past experiences or perform relevant tasks with long-term dependency.
Model Architecture. We provide the workflow of our model as follows. The details of each block, i.e. Encoder, Spatial Transformer, Spatio-Temporal Transformer, and Prediction Heads are included in the main text. Specifically,  $T$  is the length of a clip, and  $N$  is the number of anchors. We use  $T = 30$  and  $N = 8 \cdot 8 \cdot 12 = 768$ . We provide the code of the model class in the supplementary.
Stage	Configuration	Output
0	Video Frames	T × 448 × 448 × 3
0	Visual Query	448 × 448 × 3
Encoder
1	Frame features	T × 32 × 32 × 256
1	Query features	32 × 32 × 256
Spatial Transformer
2	Updated frame features	T × 32 × 32 × 256
Spatio-Temporal Transformer
3	Downsampled frame features	T × 8 × 8 × 256
3	Propagated frame features	T × 8 × 8 × 256
Prediction Heads
4	Anchor refinement	T × N × 4
4	Anchor scores	T × N
If inference
5	Top-1 anchor	T × 4
5	Top-1 score	T
Table 7: Model workflow.
Inference Details. We provide more inference details here. For the predicted object occurrence scores on all frames, we first smooth the scores with a median filter with kernel size 5. Then we apply peak detection on the smoothed scores. We find the peak with the highest score  $s$  and use  $0.8 \cdot s$  as the threshold to filter non-confident peaks. We then select the response track that corresponds to the last peak as the results. To find the start and end time steps of the last response track, we threshold the occurrence scores with the value  $0.7 \cdot s_p$ , where  $s_p$  is the score of the last peak.
Visualization. Please see our supp video for more visualization results of the retrieved response track with bounding boxes. Moreover, we provide the visualization of feature affinity between the query and frame. As shown below, the query point (in blue) has high feature similarity with the same object in the frame. We note that we use the pixel in the visual query as the source feature in this visualization, while we use the pixel in the target frame as the source feature in our spatial transformer. The reason is that if we visualize the feature similarity in the latter manner, the response will cover the entire visual query, which is not informative.
Figure 6: Visualization of feature similarity between the query and frame. The query point (in blue) has high feature similarity with the target object in the frame.
Leaderboard Result. Our model VQLoC achieved state-of-the-art performance on the public leaderboard of Ego4D VQ2D benchmark.
Figure 7: Ego4D VQ2D benchmark leaderboard at the time of submission.
Error Bar. As shown in Table 8, we provide the average value and the performance variance by running the model 3 times. The results demonstrate that VQLoC consistently achieves reasonable performance.
Table 8: Model performance and variances with multiple runs.
Seed	tAP25	stAP25	rec%	Succ.
42	0.31	0.22	47.05	55.89
0	0.30	0.23	45.64	57.32
1	0.31	0.22	47.73	55.42
mean	0.31	0.22	46.81	56.21
var	2e-5	2e-5	0.76	0.65