SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Tang $^{1}$ , Kevin J Liang $^{1}$ , Kristen Grauman $^{1,2}$ , Matt Feiszli $^{1,*}$ , Weiyao Wang $^{1,*}$
FAIR, Meta $^{1}$ , UT Austin $^{2}$ , Equal Contribution*
{haotang, kevinjliang, grauman, mdf, weiyaowang}@meta.com
Visual object tracking is key to many egocentric vision problems. However, the full spectrum of challenges of egocentric tracking faced by an embodied AI is underrepresented in many existing datasets, which tend to focus on short, third-person videos. Egocentric video has several distinguishing characteristics from those commonly found in past datasets: frequent large camera motions and hand interactions with objects commonly lead to occlusions or objects exiting the frame, and object appearance can change rapidly due to widely different points of view, scale, or object states. Embodied tracking is also naturally long-term, and being able to consistently (re-)associate objects to their appearances and disappearances over as long as a lifetime is critical. Previous datasets under-emphasize this re-detection problem, and their "framed" nature has led to adoption of various spatiotemporal priors that we find do not necessarily generalize to egocentric video. We thus introduce EgoTracks, a new dataset for long-term egocentric visual object tracking. Sourced from the Ego4D dataset, EgoTracks presents a significant challenge to recent state-of-the-art single-object trackers, which we find score more poorly on our new dataset than existing popular benchmarks, according to traditional tracking metrics. We further show improvements that can be made to the STARK tracker to significantly increase its performance on egocentric data, resulting in a baseline model we call EgoSTARK. We publicly release our annotations and benchmark (https://github.com/EG04D/episodic-memory/tree/main/EgoTracks), hoping our dataset leads to further advancements in tracking.
First-person or "egocentric" computer vision aims to capture the real-world perceptual problems faced by an embodied AI; it has drawn strong recent interest as an underserved but highly relevant domain of vision, with important applications ranging from robotics [67, 18] to augmented and mixed reality [2, 70, 29]. Visual object tracking (VOT), long a fundamental problem in vision, is a core component to many egocentric tasks, including tracking the progress of an action or activity, (re-)association of objects in one's surroundings, and predicting future states of the environment. Yet, while the VOT field has made many significant advancements over the past decade, tracking in egocentric video remains underexplored. This lack of attention is in large part due to the absence of a large-scale egocentric tracking dataset for training and evaluation. While the community has proposed a number of popular tracking datasets in recent years, including OTB [81], TrackingNet [60], GOT-10k [33], and LaSOT [21], we find that the strong performance that state-of-the-art trackers achieve on these benchmarks does not translate well to egocentric video, thus establishing a strong need for such a tracking dataset.
We attribute this performance gap to the many unique aspects of egocentric views compared to the more traditional third-person views of previous datasets. In contrast to intentionally "framed" video, egocentric videos are often uncurated, meaning they tend to capture many attention shifts between
Figure 1: A video from the proposed EgoTracks dataset, with yellow clip segments marking when the object (blowtorch) is visible. Note the frequent disappearances and reappearance of the object over an 8 minute video, with lengthy absences, necessitating re-detection to track accurately without false positives. The egocentric nature of the video includes the camera-wearer interacting with the object (Occurrence 2), resulting in significant hand occlusions and dramatic changes in pose.
activities, objects, or locations. Due to the first-person perspective, large head motions from the camera wearer often result in objects repeatedly leaving and re-entering the field of view; similarly, hand manipulations of objects [68] leads to frequent occlusions, rapid variations in scale and pose, and potential changes in state or appearance. Furthermore, egocentric video tends to be long (sometimes representing the entire life of an agent or individual), meaning the volume of the aforementioned occlusions and transformations scales similarly. These characteristics all make tracking objects in egocentric views dramatically more difficult than scenarios commonly considered in prior datasets, and their absence represents an evaluation blindspot.
Head motions, locomotion, hand occlusions, and temporal length lead to several challenges. First, frequent object disappearances and reappearance causes the problem of re-detection within egocentric tracking to become especially critical. Many previous tracking datasets primarily focus on short-term tracking in third-person videos, providing limited ability to evaluate many of the challenges of long-term egocentric tracking due to low quantity and length of target object disappearances. As a result, competent re-detection is not required for strong performance, leading many recent short-term trackers to neglect it, instead predicting a bounding box for every frame, which may lead to rampant false positives or tracking the wrong object. Additionally, the characteristics of short-term third-person video have also induced designs relying on gradual changes in motion and appearance. As we later show (Section 5.2), many of the motion, context, and scale priors made by previous short-term tracking algorithms fail to transfer to egocentric video.
Notably, re-detection, occlusions, and longer-term tracking have long been recognized as difficult for
VOT as a field, leading to recent benchmark construction efforts [54, 11, 58, 73, 34, 75] emphasizing these aspects. We argue that egocentric video provides a natural source for these challenges at scale while also representing a highly impactful application for tracking, therefore constituting a significant opportunity. We thus present EgoTracks: a large-scale long-term egocentric visual object tracking dataset for training and evaluating long-term trackers. Seeking a realistic challenge, we source videos from Ego4D [29], a large-scale dataset consisting of unscripted, in-the-wild egocentric videos of daily-life activities. The result is a large-scale dataset to evaluate the tracking and re-detection ability of SOT models, with more than 22,028 tracks from 5708 average 6-minute videos. This
Figure 2: EgoTracks is an order of magnitude larger than past long-term VOT datasets, with significantly more tracks (circle area) and object disappearances/appearances in longer videos.
constitutes the first large-scale dataset for visual object tracking in egocentric videos in diverse settings, providing a new, significant challenge compared with previous datasets.
We perform a thorough analysis of our new dataset and its new characteristics relative to prior benchmarks, demonstrating its difficulty and the need for further research to develop trackers capable of handling long-term egocentric vision. Our experiments reveal remaining open problems and insights towards promising directions in egocentric tracking. Leveraging these intuitions, we propose multiple simple yet effective changes, such as adjustment of spatiotemporal priors, egocentric data finetuning, and combining multiple templates. We apply these proposed strategies on a state-of-the-art (SOTA) STARK tracker [84], training a strong tracker dedicated to long-term egocentric tracking: EgoSTARK. We hope Ego-STARK can serve as a strong baseline and facilitate future research.
1. We present EgoTracks, the first large-scale long-term object tracking dataset with diverse egocentric scenarios. We analyze its uniqueness in terms of evaluating the re-detection performance of trackers.
2. We conduct comprehensive experiments to understand the performance of many state-of-the-art trackers on the EgoTracks validation set and observe that due to the biases and evaluation blindspots of existing third-person datasets, they tend to struggle.
3. We perform an analysis of what makes a good tracker for long-form egocentric video. Applying these learnings to the STARK tracker [84], we produce a strong baseline we call EgoSTARK, which achieves significant improvements (+15% F-score) on EgoTracks.
2.1 Visual object tracking datasets
Visual object tracking studies the joint spatial-temporal localization of objects in videos. From a video and a predefined taxonomy, multiple object tracking (MOT) models simultaneously detect, recognize, and track multiple objects. For example, MOT [57] tracks humans, KITTI [26, 53] tracks pedestrians and cars, and TAO [15] tracks a large taxonomy of 833 categories. In contrast, single object tracking (SOT) follows a single object via a provided initial template of the object, without any detection or recognition. Thus, SOT is often taxonomy-free and operates on generic objects. The community has constructed multiple popular benchmarks to study this problem, including OTB [81], UAV [59], NfS [38], TC-128 [48], NUS-PRO [43], GOT-10k [33], VOT [40], and TrackingNet [60].
While these SOT datasets mainly consist of short videos (e.g. a few seconds), long-term tracking has been increasingly of interest. Tracking objects in longer videos (several minutes or more) poses unique challenges, e.g. significant transformations, displacements, disappearances, and reappearance. On top of localizing the object when visible, the model also must produce no box when the object is absent, and then re-localize the same object when it reappears. OxUvA [73] is one of the first to benchmark longer videos (average 2 minutes), with 366 evaluation-only videos. LaSOT [21] scales this to 1400 videos with more frequent object reappearance. Concurrently, VOT-LT [39] includes frequent object disappearances and reappearance in 50 purposefully selected videos.
Our EgoTracks focuses on long-term SOT and presents multiple critical and unique attributes: 1) significantly larger scale, with 5708 videos of an average 6 minutes (Figure 2); 2) more frequent disappearances & reappearance (avg. 17.7 times) happening in natural, real-world scenarios; 3) data sourced from egocentric videos shot in-the-wild, involving unique challenging situations, such as large camera motions, diverse perspective changes, hand-object interactions, and frequent occlusions.
2.2 Single object tracking methodologies
Many modern approaches use convolutional neural networks (CNNs), either with Siamese network [45, 76, 44] or correlation-filter based [13, 3, 8, 56, 4] architectures. With recent successes in vision tasks like classification [17] and detection [5], Transformer architecture [74] for tracking have also become popular. For example, TransT [6] uses attention-based feature fusion to combine features of the object template and search image. More recently, several works utilize Transformers as direct predictors to achieve a new state of the art, such as STARK [84], ToMP [55] and SBT [82]. These models tokenize frame features from a ResNet [31] encoder, and use a Transformer to predict the bounding box and object presence score with the feature tokens. These methods are often developed on short-term SOT datasets and assume that the target object stays in the field of view with minimum occlusions. On the other hand, long-term trackers [75, 34, 11] are designed to cope with
Table 1: Object tracking datasets comparison. In addition to larger scale than previous datasets, the scenarios captured by EgoTracks represent a significantly harder challenge for SOTA trackers, suggesting room for improved tracking methodology.
Dataset	Video Hours	Avg. Length (s)	Ann. FPS	Ann. Type	Egocentric	SOTA(P/AO)*
ImageNet-Vid [66]	15.6	10.6	25	mask	No
YT-VOS [83]	5.8	4.6	5	mask	No	- / 83.6 [32]
DAVIS 17 [64]	0.125	3	24	mask	No	- / 86.3 [7]
TAO [15]	29.7	36.8	1	mask	No
UVO [79]	2.8	10	30	mask	No	- / 73.7 [61]
EPIC-KITCHENS VISOR [14]	36	12**	0.9	mask	Yes	- / 74.2 [61]
GOT-10k [33]	32.8	12.2	10	bbox	No	- / 75.6 [9]
OxUvA [73]	14.4	141.2	1	bbox	No
LaSOT [21]	31.92	82.1	30	bbox	No	80.3/- [9]
TrackingNet [60]	125.1	14.7	28	bbox	No	86/- [9]
TREK-150 [19, 20]	0.45	10.81	60	bbox	Yes	- / 50.5 [19, 20]
EgoTracks (Ours)	602.9	367.9	5	bbox	Yes	45/54.1
*: P: Precision, AO: average overlap (J-Score for mask-based datasets). **: Original video is 720s.
the problem of re-detecting objects in their reappearances. Designed to be aware of potential object disappearances, these approaches search the whole image for its reappearance.
2.3 Tracking in egocentric videos
Multiple egocentric video datasets have been introduced in the past decades [12, 29, 41, 71, 63, 23], offering a host of interesting challenges, many of which require associating objects across frames: activity recognition [36, 46, 85, 80, 27], anticipation [22, 24, 28], video summarization [16, 41, 42, 52], human-object interaction [14, 50], episodic memory [29], visual query [29], and camera-wearer pose inference [35]. To tackle these challenges, tracking is leveraged in many methodologies [29, 14, 51, 42, 50], yet few works have been dedicated to this fundamental problem on its own. [19, 20] have started to recognize the challenges of egocentric object tracking and might be the most related work to ours. The major difference, however, is the scale of the dataset: [19, 20] contain 150 tracks intended only for evaluation, while EgoTracks is  $100 \times$  larger (see Table 1), containing 20k tracks with training and evaluation splits. Also, while past efforts have sourced videos from the kitchen-heavy EPIC-KITCHEN [12], EgoTracks sources videos from Ego4D [29], which has more diverse scenarios. EgoTracks provides a unique, large-scale testbed for developing tracking methods dedicated to egocentric videos; our improved baseline EgoSTARK also serves as a potential plug-and-play module to solve other tasks where object association is desired.
In egocentric video understanding, Ego4D [29] and EPIC-KITCHENS VISOR [14] are closely related. Ego4D contains the largest collection of egocentric videos in-the-wild; EgoTracks is annotated on a subset of Ego4D. In addition, Ego4D proposes many novel tasks, such as Episodic Memory, with tracking identified as a core component. VISOR was introduced concurrently, annotating short-term (12 sec on average) videos from EPIC-KITCHENS [12] with instance segmentation masks. We believe EgoTracks offers multiple unique values complementary to EPIC-VISOR: long-term tracking (6 min vs. 12 sec), significantly larger-scale (5708 video clips vs. 158), and more diversified video sources (80+ scenes vs. kitchen-only; see Figure 4).
3 The EgoTracks dataset
We present EgoTracks: a large-scale long-term egocentric single object tracking dataset, consisting of a total of 22028 tracks from 5708 videos. We follow the same data split as the Ego4D Visual Queries (VQ) 2D benchmark (see Appendix A for details).
3.1 Ego4D visual queries (VQ) benchmark
Ego4D [29] is a massive-scale egocentric video dataset, consisting of 3670 hours of diverse daily-life activities of consenting participants in an in-the-wild format; the videos have personally identifiable information removed and were screened for offensive content. The dataset is accompanied by multiple benchmarks, but the most relevant task for our purposes is episodic memory's 2D VQ task: Given an egocentric video and a cropped image of an object, the goal is to localize when and where the object was last seen in the video, as a series of 2D bounding boxes in consecutive frames. This task is closely related to long-term tracking: finding an object in a video given a visual template is identical
Figure 3: EgoTracks tracklet attribute examples. Left: Micropipette on a bench (top) versus actively used (bottom). Middle: A paint can (top) is opened (bottom). Right: A blowtorch (top) requiring context from other frames to identify due to distance and motion blur (bottom).
Table 2: Track attributes in train/val sets.
Total number	Percentage
All Tracks	17593	100%
is-active	3963	22.52%
istransformed	1080	6.13%
is_recognizable	17557	99.79%
Table 3: Evaluation protocols visualization.
to the re-detection problem in long-term tracking. Moreover, Ego4D's baselines rely heavily on tracking methods: Siam-RCNN [75] and KYS [4] for global and local tracking, respectively.
Shortcomings. While highly related, the VQ dataset is not immediately suitable long-term tracking. In particular, the VQ annotation guidelines were roughly the following: 1) identify three different objects that appear multiple times in the video; 2) annotate a query template for each object, which should contain the entire object without any motion blur; 3) annotate an occurrence of the object that is temporally distant from the template. Thus, these annotations are not exhaustive over time (they are quite sparse), limiting their applicability to tracking. On the other hand, the selection criteria result in a strong set of candidate objects, which we leverage to build EgoTracks.
3.2 Annotating VQ for long-term tracking
We thus start with the VQ visual crop and response track, asking annotators to first identify the object represented by the visual crop, the response track, and object name. From the video's start, we instruct the annotators to draw a bounding box around the object each time it appears. Because annotators must go through each video in its entirety, which contain an average of  $\sim 1800$  frames at 5 frames per second (FPS), this annotation task is labor-intensive, taking roughly 1 to 2 hours per track. An important aspect of this annotation is its exhaustiveness: the entire video is densely annotated for the target object, and any frame without a bounding box is considered as a negative. Being able to reject negatives examples is an important component to re-detection in real-world settings, as false positives can impact certain applications as much as false negatives.
Quality Assurance. All tracks are quality checked by expert annotators after the initial annotations. To measure the annotation quality, we employ multi-review on a subset of the validation set. Three independent reviewers are asked to annotate the same video. We find the overlaps between these independent annotations are high ( $>0.88$  IoU). Further, since EgoTracks has a focus on re-detection, we check the temporal overlap of object presence and find it to be consistent across annotators. In total, the entire annotation effort represented roughly 86k worker-hours of effort.
3.3 Tracklet attributes
In addition to the bounding box annotations, we also label certain relevant attributes to allow for different training strategies or deeper analysis of validation set performance. We annotate the following three attributes per occurrence (see Figure 3 for examples and Table 2 for statistics):
- is.active: In Ego4D, the camera wearer often interacts with relevant objects with their hands. Objects in the state of being handled pose a challenge for tracking algorithms due to frequent occlusion and rapid changes in pose.
- is_transformed: Objects in Ego4D may undergo transformations, such as deformations and state changes. Such instances require being able to quickly adapt to the tracked object having a new appearance.
- is_recognizable: Due to occlusions, motion blur, scale, or other conditions, some objects in Ego4D can be extremely difficult to recognize without additional context. We thus annotate if the object is recognizable solely based on its appearance, without using additional context information (e.g. other frames).
Figure 4: EgoTracks is a large-scale egocentric dataset of diverse scenarios (left) and objects (right).
Figure 5: Qualitative results of different trackers. EgoTracks presents significant challenges for all trackers, due to drastic viewpoint changes, occlusions, changes in scale, head motion etc.
3.4 Data diversity
Figure 4 shows the diversity of EgoTracks in terms of both the actions the videos contain and name of the object tracks. Ego4D [29] videos depict a wide variety of in-the-wild activities (Figure 4, left), captured in locations all around the world. As such, they represent a far more diverse set of scenes than other egocentric datasets [12], thus requiring models to learn more general representations.
There was no fixed taxonomy for which objects to track. Instead, annotators were asked to choose objects that are "interesting" [29], which were named freely, resulting in considerable object diversity. We extracted the nouns from these object names, resulting in around  $1000+$  unique names. Figure 4, right shows the wide array of objects tracked in EgoTracks.
4 Analysis of state-of-the-art SOT trackers
We compare the performance of several off-the-shelf tracking models on EgoTracks's validation set. Identifying STARK [84] as the one with the best performance, we conduct further ablation studies under different settings using STARK to further understand its behavior.
4.1 Evaluation protocols and metrics
Evaluation Protocols. We introduce several evaluation protocols for EgoTracks, consisting of different combinations of the initial template, evaluated frames, and the temporal direction in which the tracker is run. For the initial template, we consider two choices:
Table 4: EgoTracks performance comparison. Off-the-shelf, all trackers perform poorly, demonstrating the new challenges of EgoTracks. Higher performance from tracking by detection methods + Oracle imply that instance association, not detection, is one of the primary challenges.
Method	AO	F-score	Precision	Recall	FPS
KYS [4]	16.09	13.09	12.50	13.74	20
DiMP [3]	16.45	11.84	10.31	13.91	43
GlobalTrack [34]†	23.63	20.40	31.28	15.14	6
LTMU [11]†	29.33	27.46	37.28	21.74	13
ToMP [55]	30.93	20.95	19.63	22.46	24.8
Siam-RCNN [75]†	37.48	35.38	52.80	26.67	4.7
MixFormer (MixViT-L, ConvMAE) [9, 10]	27.93	25.54	28.30	23.27	10
STARK [84] - Res50	35.99	30.48	34.70	27.17	41.8
STARK [84] - Res101	35.03	30.18	35.30	26.35	31.7
Tracking by Detection†
Mask R-CNN [30]+Oracle	60.00	-	-	-
GGN [78]+Oracle	75.92	-	-	-
GGN+InstEmb	15.19	9.92	11.75	8.58
†: trackers with re-detection
- Visual Crop Template (VCT): The visual crop images were specifically chosen to be high-quality views of the target and served as our annotators' references for identifying the object throughout the videos. Thus, they make ideal candidates for initializing a tracker.
- Occurrence First Frame Template (OFFT): The tracker is initialized with the first frame of each occurrence (see  $\overrightarrow{OO}$  below). While this may result in a lower quality view of the object, temporal proximity to subsequent frames means it may be closer in appearance.
Note that we exclude the template frame from the calculation of any evaluation metrics. We also consider several choices for the evaluated frames and temporal direction:
- Video Start Forward ( $\overline{\mathbf{VS}}$ ): The tracker is evaluated on every frame of the video in causal order, starting from the first frame. This represents a tracker's ability to follow an object through a long video.
- Visual Crop Forward/Backward  $(\overleftrightarrow{\mathbf{VC}})$ : The tracker is run on the video twice, once starting at the visual crop frame and running forward and time, and a second time running backwards. This represents an alternative way of covering every frame in the video, but with closer visual similarity between VCT initialization and the first frames encountered by the tracker.
- Occurrences Only Forward ( $\overrightarrow{\mathbf{OO}}$ ): The tracker is only evaluated on the object occurrences, when the object is visible. This simplifies the tracking task and allows us to dis-entangle the challenge of re-detection from that of simply tracking in an egocentric clip.
We specify protocols by concatenating the appropriate descriptors. We primarily consider VCT-  $\overrightarrow{\mathbf{VS}}$  VCT-VC, VCT-OO, and OFFT-OO (Table 3) in our experiments.