SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanwen Jiang $^{1}$  Santhosh Kumar Ramakrishnan $^{1}$  Kristen Grauman $^{1,2}$  UT Austin $^{1}$  FAIR, Meta $^{2}$
Visual Query Localization on long-form egocentric videos requires spatio-temporal search and localization of visually specified objects and is vital to build episodic memory systems. Prior work develops complex multi-stage pipelines that leverage well-established object detection and tracking methods to perform VQL. However, each stage is independently trained, and the complexity of the pipeline results in slow inference speeds. We propose VQLoC, a novel single-stage VQL framework that is end-to-end trainable. Our key idea is to first build a holistic understanding of the query-video relationship and then perform spatio-temporal localization in a single shot manner. Specifically, we establish the query-video relationship by jointly considering query-to-frame correspondences between the query and each video frame and frame-to-frame correspondences between nearby video frames. Our experiments demonstrate that our approach outperforms prior VQL methods by  $20\%$  accuracy while obtaining a  $10\times$  improvement in inference speed. VQLoC is also the top entry on the Ego4D VQ2D challenge leaderboard. Project page: https://hwjiang1510.github.io/VQLoC/
Episodic memory, a specific type of long-term memory in humans, facilitates the retrieval of our past experiences such as their time, location, related context, and activities [46]. This form of memory is pivotal to the progressive evolution of embodied intelligence due to its foundational role in enabling long-horizon reasoning [4, 19]. Nevertheless, human memory can falter in accurately recalling the details of daily activities—such as where we left our keys, or who we met while going for a run. This perceptual overload can be mitigated through the utilization of head-worn cameras or AR glasses to capture our past experiences and the development of systems that can retrieve information. To this end, the episodic memory benchmark [13] is recently proposed, which aims to develop systems that enable super-human memory by allowing humans to query about their past experiences recorded on head-worn devices. These systems also have the potential to serve as a conduit for learning from past experiences and data [60, 44], promoting long-term robot learning [25].
We focus on 2D Visual Query Localization (VQL), a vital task in the episodic memory benchmark that aims to answer the question "where did I last see object  $X$ ?". Specifically, given a long-form egocentric video representing past experiences from a human camera-wearer, the goal is to search and localize a visual query object, specified as a visual image crop, in space-time (see Fig. 1, left). Addressing VQL is of paramount importance, as query object localization serves as a prerequisite for downstream object-centric reasoning tasks.
VQL presents a myriad of challenges to existing computer vision systems. For example, current object detection systems localize pre-defined object categories on well-curated internet-style images [40, 5]. In contrast, VQL requires localizing an open-set of objects specified as visual queries. Current object tracking systems expect to be initialized with a bounding box of the object next to its appearance in the video [54, 31]. However, the visual query crop of VQL originates from an image outside
Figure 1: Visual Query Localization (VQL): (Left) The goal is to localize a visual query object within a long-form video, as demonstrated by the response track marked by yellow bounding boxes. The complexity of this task arises from the need to accommodate an open-set of object queries with diverse scales, viewpoints, and states as they appear in the video. (Right) Our method VQLoC first establishes a holistic understanding of the query-video relationship by jointly reasoning about query-to-frame (spatio) and frame-to-frame (temporal) correspondences, before localizing the response in a single-stage and end-to-end trainable manner.
of the video being queried; that is, there may not be an exact match and "close frame" for the visual query. Complicating matters further, the egocentric nature of the VQL task presents its own unique challenges. Compared with the visual query, the object may appear in the video under significantly varying orientations, sizes, contexts, and lighting conditions, and it may experience motion blur and occlusions. Finally, egocentric videos can span several minutes, hours, or days in real-world applications, while the object itself may appear only for a few seconds, resulting in a "needle-in-the-haystack" problem.
Prior work has attempted to address VQL through a bottom-up framework with three stages [13, 52, 53]: i) in each video frame, detect all objects and conduct pairwise comparisons with the visual query to obtain the proposal that is most similar to the query; ii) identify the similarity score peaks throughout the video; and iii) perform bidirectional tracking around the most recent peak to recover the spatio-temporal response. Although this framework is grounded in well-established object detection and tracking approaches, it relies heavily on the first stage to detect the object by independently looking at each frame. While this may be possible if the object appears clearly in the video, it is often not the case due to the egocentric nature of images. Errors in frame-level object detection can potentially cause the entire system to fail since the framework is not end-to-end differentiable and errors in earlier stages may not be rectifiable in later stages. Moreover, the methods suffer from a slow inference speed due to the high complexity of pairwise comparison with redundant object proposals.
To address these limitations, we propose VQLoC (Visual Query Localization from Correspondence), a novel single-stage framework for VQL. Our key insight in VQLoC is that a holistic understanding of the query-video relationship is essential to reliably localize query objects in egocentric videos. Accordingly, VQLoC jointly models the query-to-frame relationship between the query and each video frame and frame-to-frame relationships across nearby video frames (see Fig. 1, right), and then performs spatio-temporal localization in a single-stage and end-to-end trainable manner. Specifically, we establish the query-to-frame relationship by extracting image features for the visual query and each video frame using a ViT backbone pretrained with DINO [37] and using a cross-attention transformer module [48] to establish correspondences between the image regions in the query and video frame. We then propagate these correspondences over time using a self-attention transformer module [48] that exploits the frame-to-frame relationships resulting from the temporal continuity of videos to capture the overall query-video relationship. Finally, we use a convolutional prediction head that performs spatio-temporal localization by utilizing the query-video relationship to make frame-level predictions. Critically, our model operates in a single stage, i.e., there are no intermediate localization outputs with dedicated post-processing steps, and it is end-to-end trainable since it uses only differentiable modules to obtain the final prediction.
VQLoC has several benefits when compared with the prior stage-wise methods. Unlike prior work that explicitly generates object proposals in the video frame and compares them with the visual query, VQLoC implicitly establishes the query-frame relationship by performing attention-based reasoning between the visual query features and the video frame features. This approach effectively utilizes image regions of the background and non-query objects as contextual information for inference. Additionally, our implicit query-frame relationships are significantly faster to compute than explicitly generating proposals and performing pairwise comparisons, which is critical for real-world episodic memory applications. Finally, VQLoC is end-to-end trainable, which results in much better performance compared to prior work.
We evaluate VQLoC on the Ego4D Visual Query 2D Localization benchmark. VQLoC achieves the state-of-the-art on this benchmark, outperforming previous methods by  $20\%$ , and it is also ranked first on the public Ego4D VQ2D challenge leaderboard. Furthermore, VQLoC achieves real-time inference with 36 Frames Per Second (FPS), which is  $10\times$  faster than prior work. We are committed to releasing our code upon publication for future research.
Object detection. Prior work has made impressive progress in building state-of-the-art object detectors for localizing objects in images [26, 12, 40, 5]. These methods typically detect objects with pre-specified object categories and are evaluated on internet-style and human-curated image datasets [27, 10, 56, 38]. While these methods relied significantly on having sufficient training data, recent work has extended them to incorporate long-tail object categories [14, 43] and free-form language-based detection [58]. Prior work has also proposed methods for one-shot detection, which build on the success of object detection methods [17, 34, 7, 57]. They leverage objectness priors by calculating similarities between visual queries and region proposals. However, all of the methods work on well-curated images, e.g. the COCO dataset [27]. Our work focuses on the VQL task, which is a challenging version of one-shot detection on long-form egocentric videos. Unlike image-based one-shot detection where the object is assumed to be present in the image, video-based one-shot detection requires us to identify in which frame the object appears for spatio-temporal localization.
Object Tracking. There is a rich literature on object tracking methods that track a static or dynamic object throughout a given video [8, 33, 54, 31, 22, 36, 35]. The object is typically specified as a bounding box in a video frame, and the goal is to track it through the subsequent frames in the same video [24, 11]. Short-term tracking methods are developed to track objects till they completely disappear from the video frame [47, 30], while long-term tracking methods are capable of recovering the object track when it re-appears in the video [50, 32, 54]. Tracking methods are initialized with an object bounding box close to its appearance in the video, while visual queries in VQL may originate from outside the video. This can pose a challenge for tracking methods. We compare with a state-of-the-art long-term tracker in our experiments [54]. Technically, VQLoC differs from tracking-based methods in using a spatial-aware transformer, temporal reasoning within local windows, and anchor-based prediction heads.
Visual Query Localization (VQL). The recently proposed episodic memory benchmark introduces the VQL task on egocentric videos, where the goal is to spatio-temporally localize a query object specified as a visual image crop [13]. Prior work has tackled VQL using a three-stage detection and tracking framework. First introduced as baseline in Ego4D, this approach performs frame-level detection, identifies the most recent "peak" in the detections across time, and performs bidirectional tracking [3] to recover the complete temporal extent of the response [13]. Subsequent research in this paradigm improves the frame-level detector by reducing false positive predictions through negative frame sampling [52] and proposes using background objects as context [53]. However, these multi-stage approaches independently perform spatial and temporal reasoning in a modular fashion without end-to-end training, which results in compounding errors across the stages. Furthermore, they incur significant computational costs and time for training and inference, which limits their applicability to real-time episodic memory applications. Our proposed method, VQLoC, distinguishes itself by first establishing holistic spatio-temporal correspondences and then localizing the object with an end-to-end paradigm. This leads to state-of-the-art results with fast inference speeds.
Visual Correspondence. Establishing correspondence is a core problem in computer vision [15]. The advancement of visual correspondence models has significantly contributed to the success of various computer vision tasks, including image matching and retrieval [45, 42], tracking and flow estimation [2, 16], 3D vision [29, 21], and representation learning [51, 49]. Recent developments in vision foundation models have demonstrated remarkable capabilities to establish correspondence between multi-modal [39] and spatio-temporal [6] inputs, empowering new applications [23, 20].
The design of model architectures plays a critical role in finding correspondence. Specifically, transformer models [48, 1, 59], which propagate information by establishing similarity-based correspondence, are widely used. For example, TubeDETR [55], which performs visual question answering, concatenates the text query and frame features and finds their correspondence using self-attention transformers. STARK [54] employs a similar architecture and extends it to visual templates for visual object tracking. These methods do not differentiate the query features and the
Ego4D VQ2D challenge leaderboard: https://eval.ai/web/challenges/challenge-page/1843/leaderboard
Figure 2: Visual Query Localization from Correspondence (VQLoC): We address the VQL problem by first establishing the overall query-video relationship and then inferring the response to the query. VQLoC extracts image features for both the query and video frames independently. It establishes query-to-frame correspondence between the query features  $\mathbf{q}$  and each video frame feature  $\mathbf{v}_i$  using a transformer module (Spatial TX), and obtains the correspondence features  $\mathbf{f}$  for identifying potential query-relevant regions in the frames. VQLoC then propagates these frame-level correspondences through time using another transformer module (Spatio-Temporal TX), which leverages the frame-to-frame correspondence between nearby video frames and obtains the query-video correspondence features  $\mathbf{v}^*$ . Finally, VQLoC uses the query-video correspondence to predict the per-frame bounding box and the probability that the query object occurs in the frame.
frame features after the concatenation, making the features lose spatial correlation with the input frames. In contrast, VQLoC utilizes the cross-attention transformer to establish the query-to-frame correspondence, updating the frame features by incorporating query features. This maintains the spatial correlation between frame-level features and the frames, providing strong priors for object localization.
We propose a novel end-to-end framework for localizing visual queries in egocentric videos. We design our model based on the insight that a holistic understanding of the query-video relationship is crucial for reliably localizing query objects appearing in the video. To this end, we propose to first capture both the query-to-frame relationship for each video frame and the frame-to-frame relationship across nearby frames. We then predict the results based on the captured query-video relationship in a single shot. Next, we will describe the formulation of the VQL task in Sec. 3.1, introduce our model architecture in Sec. 3.2, and describe our approach for training and inference in Sec. 3.3.
3.1 VQL Task Formulation
The VQL task is formulated as open-set localization in egocentric videos, answering questions like "where did I last see my wallet?" Formally, given an egocentric video  $\mathcal{V} = \{v_1,\dots ,v_T\}$  consisting of  $T$  frames and a visual query  $\mathcal{Q}$  specified as an image crop of the query object<sup>2</sup>, the goal is to spatio-temporally localize the most-recent occurrence of the query object in the video. The localization result is a response track  $\mathcal{R} = \{r_s,r_{s + 1},\dots ,r_e\}$ , where  $s$  and  $e$  are start and end frame indices, respectively, and  $r_i$  is a bounding box containing the query object on the  $i^{th}$  frame.
3.2 VQLoC Architecture
Our proposed method VQLoC is illustrated in Fig. 2. VQLoC employs a shared image encoder to extract visual features for both the video frames and the visual query. We utilize the DINO [6] pre-trained ViT [9], which can capture object-level semantic correspondence under varying camera viewpoints [18]. First, VQLoC leverages a spatial transformer to find the query-to-frame correspondence between each video frame and the visual query. The spatial transformer identifies the regions in the frames that are similar to the query in semantics, handling the potential large appearance variation. VQLoC then uses a spatio-temporal transformer that propagates the query-to-frame correspondence over time by leveraging the frame-to-frame correspondence arising from temporal continuity in the video, and establishes the query-video correspondence. This process is performed within a local temporal window, handling the fast head motion of egocentric videos. Finally, it uses the holistic understanding of the query-video relationship to predict the response track. The response prediction
2The image crop of the query object is sampled from outside the ground-truth response track.
Figure 3: Architecture for spatio-temporal transformer (left): We add the downsampled query-to-frame correspondence features  $\bar{\mathbf{f}}_d$  with 3-D spatio-temporal positional embedding, and flatten it into 1-D tokens  $(\mathbf{f}_d)$ . We use several transformer layers to propagate the frame-level correspondences over time by leveraging frame-to-frame correspondences between nearby video frames within a local temporal window, and obtain the overall query-video correspondence features  $\mathbf{v}^*$ . Architecture for the prediction head (right): After establishing the query-video relationship, we use it to predict anchor-level results for each video frame  $i$ . We perform label association during training and pick the refined anchor with the highest probability as the prediction for the frame in testing.
head is an anchor-based model, which deals with the small scales of query objects in the video. Next, we describe these components in detail.
Visual Encoder. Our visual encoder consists of a fixed DINO pre-trained ViT followed by several learnable convolution layers. We independently extract visual features for each of the  $T$  video frames and the visual query and obtain the video features  $\mathbf{v} \in \mathbb{R}^{T \times H \times W \times C}$  and query features  $\mathbf{q} \in \mathbb{R}^{H \times W \times C}$ .  $H, W$  signify the spatial resolution of the features, and  $C$  is the channel size.
Spatial Transformer. The spatial transformer establishes query-to-frame correspondences between the visual crop and each video frame using cross-attention [48] between their visual features. Formally, let  $v_{i} \in \mathbb{R}^{HW \times C}$  be the flattened video feature  $\mathbf{v}_{i}$  for video frame  $i$  and let  $q \in \mathbb{R}^{HW \times C}$  be the flattened query feature  $\mathbf{q}$ . The query-to-frame correspondence feature  $f_{i}$  is obtained as follows:
$$
f _ {i} = \operatorname {F F N} (\text {C r o s s A t t e n t i o n} (v _ {i}, q)), \tag {1}
$$
where FFN is a feed-forward neural network. Intuitively, the frame features  $v_{i}$  are updated by incorporating the corresponding query features  $q$ , where the cross-attention computes their feature similarity as the fusion weights. The transformer output  $f_{i}$  is reshaped to  $\mathbb{R}^{H\times W\times C}$ , denoted as  $\mathbf{f}_i$ . Here, the updated features  $\mathbf{f}_i$  retain the spatial arrangement and maintain the spatial cues for performing accurate localization. This process is repeated for all video frames to obtain the final query-to-frame correspondence features  $\mathbf{f}\in \mathbb{R}^{T\times H\times W\times C}$ .
Spatio-Temporal Transformer. The spatio-temporal transformer is designed to propagate and refine the query-to-frame correspondence features  $\mathbf{f}$  from Eqn. 1 over time, by establishing frame-to-frame correspondences between nearby video frames. It leverages the temporal continuity of videos. We illustrate the architecture in Fig. 3. We first decrease the spatial resolution of the feature maps  $\mathbf{f}$  using shared convolution layers, where the output is  $\overline{\mathbf{f}}_d \in \mathbb{R}^{T \times h \times w \times c}$ ,  $h, w$  and  $c$  are spatial resolutions and channel size of the downsampled feature maps. We add 3-D spatio-temporal positional embedding  $\mathbf{p} \in \mathbb{R}^{T \times h \times w \times c}$  to  $\overline{\mathbf{f}}_d$ , and flatten it into 1-D tokens  $\mathbf{f}_d$ :
$$
\mathbf {f} _ {d} = \operatorname {F l a t t e n} (\bar {\mathbf {f}} _ {d} + \mathbf {p}) \in \mathbb {R} ^ {T h w \times c}, \tag {2}
$$
where the positional embedding  $\mathbf{p}$  is learnable and initialized as zeros.
Next, we use the spatio-temporal transformer module to establish the query-video relationship. It consists of multiple layers of locally-windowed temporal self-attention, i.e., we restrict the feature propagation within a local time window. In particular, a feature element belonging to time  $t$  only attends to feature elements from time steps in the range  $[t - w, t + w]$ .  $t - w$  and  $t + w$  are the start and end of the local time window with temporal length  $2w + 1$ . We found that this was beneficial since the locations of query objects in nearby frames are highly correlated, providing strong priors for feature refinement. We achieve locally windowed attention via masking attention weights. After performing
feature refinement using the spatio-temporal transformer module, we reshape the 1-D tokens back into the original 3-D shape to get the query-video correspondence features  $\mathbf{v}^{*}\in \mathbb{R}^{T\times h\times w\times c}$
Prediction Head. Instead of directly predicting a single bounding box for an image, we first define anchor boxes  $\mathcal{B} \in \mathbb{R}^{h \times w \times n \times 4}$  on the feature maps [26]. We found that this was advantageous because the sizes and locations of query objects are diverse. The multi-scale anchor boxes can provide a strong prior on query object sizes and locations. For each frame, we define  $n$  anchor boxes for each of the spatial elements of the feature map ( $h \cdot w$  elements in total). Each anchor box defined on the feature map space can be mapped back to its corresponding location in the pixel space for inferring predictions. For each frame from  $V_i$ , we obtain the predictions by passing the query-video correspondence feature  $\mathbf{v}_i^*$  through two heads:
$$
\hat {\mathcal {P}} _ {i} = \operatorname {P r o b H e a d} \left(\mathbf {v} _ {i} ^ {*}\right) \in \mathbb {R} ^ {h \times w \times n} \quad \text {a n d} \quad \Delta \hat {b} _ {i} = \operatorname {R e g H e a d} \left(\mathbf {v} _ {i} ^ {*}\right) \in \mathbb {R} ^ {h \times w \times 4 n}, \tag {3}
$$
where each head consists of several convolution blocks.  $\hat{\mathcal{P}}_i$  is the anchor-level query object occurrence probability and  $\Delta \hat{b}_i$  is the regressed bounding box refinement. We then reshape  $\Delta \hat{b}_i$  to  $\mathbb{R}^{h\times w\times n\times 4}$ , denoted as  $\Delta \hat{\mathcal{B}}_i$ . The final refined anchor boxes for the  $i^{th}$  frame are  $\hat{\mathcal{B}}_i = \mathcal{B} + \Delta \mathcal{B}_i$ . We perform the same operation on all frames.
In both training and inference, we chunk the untrimmed video into fixed-length clips. During training, we ensure that the clip covers at least a part of the response track. During testing, we concatenate the predictions on all clips as the final result. We introduce the training and inference details as follows.