SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metrics. We adopt common metrics in object tracking, including F-score, precision, and recall; details can be found in [54]. Trackers are ranked mainly by the F-score. We additionally consider average overlap (AO), success, precision, and normalized precision as short-term tracking metrics [72].
4.2 SOT trackers struggle on EgoTracks
We compare the performance of select trackers on EgoTracks with the VCT-  $\overrightarrow{\mathbf{V}}\overrightarrow{\mathbf{S}}$  evaluation protocol. Given the breadth of tracking algorithms, we do not aim to be exhaustive but select high-performing representatives of different tracking principles. KYS [4] and DiMP [3] are two short-term tracking algorithms that maintain an online target representation. ToMP [55], STARK [84] and MixFormer [9, 10] are three examples of the SOTA short-term trackers based on Transformers. GlobalTrack [34] is a global tracker that searches the entire search image for re-detection. LTMU [11] is a high performance long-term tracker that combines a global tracker (GlobalTrack) with a local tracker (DiMP). Siam R-CNN [75] leverages dynamic programming to model a full path of history for long-term. The performance of these trackers on EgoTracks are summarized in Table 4. AO in this table is equivalent to recall at the probability threshold 0. Qualitative results are shown in Figure 5.
Table 5: Comparing tracker initializations. (Left) Comparison of trackers initialized from the first frame in each occurrence and tracking only that single occurrence (oracle re-detection). (Right) STARK whole-video performance, starting from video start frame vs. the visual crop frame.
Method	AO	Success	Pre	PreNorm
KYS [4]	33.92	34.87	31.22	34.87
DiMP [3]	34.80	35.70	32.13	38.98
ToMP [55]	45.17	45.93	41.74	47.88
STARK [84]	50.01	50.64	45.76	51.91
Method	AO	F-score	Precision	Recall
\( \mathbf{STARK - VCT-VS} \)	35.99	30.48	34.70	27.17
\( \mathbf{STARK - VCT-VC} \)	40.01	34.02	38.31	30.60
Table 6: OFFT-OO AO of standard STARK model [84] for each attribute.
Attribute	True	False
is.active	49.65	55.73
is_transformed	49.19	55.31
is_recognizable	55.52	46.65
Table 7: Performance of trackers finetuned on EgoTracks.
Method	AO	F-score	Precision	Recall
ToMP	36.13	28.11	29.01	27.26
Siam-RCNN	45.67	41.41	56.11	32.81
STARK	44.25	38.20	42.06	34.99
We highlight several observations. First, the object presence scores from most short-term trackers are not very useful, as can be seen from the low precision of KYS (12.5), DiMP (13.91), and ToMP (19.63), while long-term trackers like GlobalTrack, DiMP_LTMU and Siam R-CNN achieve higher precisions at 31.28, 37.28 and 52.8. This is expected as long-term trackers are designed to place more emphasis on high re-detection accuracy, though there clearly is still room for improvement. STARK achieves the second highest precision at 34.70, which is an exception as it has a second training stage to teach the model to classify whether the object is present. Second, more recent works such as MixFormer and STARK achieve better F-score than previous short-term trackers. This could be partially due to advances in training strategies, more data, and Transformer-based architectures. Surprisingly, we found recent MixFormer [10] does not outperform STARK, despite achieving new SOTA on its training dataset. This highlights a potential difficulty in generalization.
We also include results using the principle of Tracking by Detection [62, 1]: a detector proposes 100 bounding boxes, and we select the best using cosine similarity of box features. We observe that an open-world detector GGN [78] trained on COCO [49] generalize reasonably well with oracle matching, achieving 75.92 AO. However, the association problem is very challenging, bringing down the AO to 15.19. Implementation details can be found in Appendix B.
4.3 Re-detection and diverse views are challenging
We perform additional EgoTracks experiments following alternative evaluation protocols to gain further insights on tracker performance (Table 5). To decouple the re-detection problem from other egocentric aspects of EgoTracks, we evaluate with the OFFT-  $\overrightarrow{\mathbf{OO}}$  protocol, which ignores the negative frames of the video, thus obviating the need for re-detection. Unsurprisingly, all trackers do significantly better, emphasizing the challenging nature of re-detection in EgoTracks. We also run experiments in the VCT-  $\overleftrightarrow{\mathbf{VC}}$  setting, where the initial template is temporally adjacent to the first tracked frames. Here we see a  $3 - 4\%$  improvement to AO, F-score, precision, and recall compared to the VCT-  $\overrightarrow{\mathbf{VS}}$  protocol, illustrating that trackers like STARK are designed to expect gradual transitions in appearance. Both these experiments illustrate that the re-detection problem is a significant challenge for tracking and the need for better long-term benchmarks.
4.4 Attributes capture hard scenarios for tracking
We use the validation set tracklet attribute annotations described in Section 3.3 to further understand performance on our evaluation set. For each attribute, we split the tracklets into two groups, corresponding to the attribute being true and false. We then use a standard STARK tracker [84] and report AO for each group of tracklets using the OFFT-OO evaluation protocol in Table 6. As might be expected, we find that when objects are being actively used by the user or in the midst of a transformation, AO tends to be lower, by roughly  $6\%$ , likely due to occlusions or changes in appearance. Additionally, STARK tends to have a harder time when the object is hard to recognize in the image, whether due to occlusions, blur, scale, or other conditions.
Table 8: Evaluating EgoTracks on EPIC-VISOR [14].
Method	AO	F1	Precision	Recall
Siam-RCNN	51.0	42.4	37.7	48.4
Siam-RCNN (finetuned on EgoTracks)	52.7	43.5	38.4	50.3
STARK	59.2	46.4	39.4	56.6
STARK (finetuned on EgoTracks)	61.4	48.5	41.0	59.4
EgoSTARK	64.7	50.6	42.6	62.3
Table 9: Train/test-time hyperparameters comparison.
Method	AO	F-score	Precision	Recall
Data	STARK	35.99	30.48	34.70	27.17
STARK - ft on VQ	38.94	33.53	39.13	29.33
STARK - ft on EgoTracks	44.25	38.20	42.06	34.99
Augmentation	STARK - ft on VQ	38.94	33.53	39.13	29.33
STARK - ft + multiscale	48.44	41.92	42.65	41.30
Search window	search_size = 320	35.99	30.48	34.70	27.17
search_size = 480	48.21	39.69	43.95	36.19
search_size = 640	52.09	42.39	46.23	39.15
search_size = 800	54.08	43.74	47.60	40.45
Table 10: STARK with different context ratios. Bold row is the default setting. CR: context ratio, SRR: search region ratio, SIS: search image size (in image resolution).
Setting	Method	AO	F-score	Precision	Recall
CR	SRR	SIS
Same SIS	1x	2.5x	320	28.22	26.81	28.68	25.16
2x	5x	320	38.94	33.53	39.13	29.33
3x	7.5x	320	44.70	36.03	40.28	32.59
4x	10x	320	43.19	34.32	37.98	31.31
Same SRR	1x	5x	640	41.50	31.09	30.31	31.91
3x	5x	208	39.87	35.36	41.54	30.79
Same CR	2x	7.5x	480	48.21	39.69	43.95	36.19
2x	10x	640	52.09	42.39	46.23	39.15
2x	12.5x	800	54.08	43.74	47.60	40.45
4.5 Evaluating EgoSTARK on EPIC-VISOR
To demonstrate egocentric object tracking generalization from training on EgoTracks, we perform further quantitative evaluations on the EPIC-VISOR dataset [14], an egocentric dataset focused on active objects in kitchen scenes. We observe that training with EgoTracks leads to both Siam R-CNN and STARK showing improvements in tracking on EPIC-VISOR (Table 8), demonstrating the value of EgoTracks as a large-scale pre-training dataset. By additionally incorporating the improvements we propose in Section 5, EgoSTARK achieves further improvements on EPIC-VISOR.
5 Egocentric tracking design considerations
Observing that existing trackers do not perform well on EgoTracks, we perform a systematic exploration of priors and other design choices for egocentric tracking. Though not specifically designed for long-term tracking, Section 4 suggests STARK [84] to be the most competitive tracker on EgoTracks. We focus on this tracker for additional analysis, suggesting improvements to egocentric performance.
5.1 Egocentric fintuning is essential
We first demonstrate how various trackers trained on third-person videos can significantly benefit from finetuning on EgoTracks. As shown in Table 7, all methods gain improvement on F-score ranging from  $6\% - 10\%$ . In addition, as shown in Table 9, finetuning on the VQ response track subset improves the F-score from  $30.48\%$  to  $33.53\%$ , while using the full EgoTracks annotation further improves the F-score by  $4.67\%$  to  $38.2\%$ . This demonstrates that: 1) finetuning with egocentric data helps close the exocentric-egocentric domain gap; 2) training on full EgoTracks provides further gains, showing the value of our training set.
5.2 Third-person spatiotemporal priors fail
Modern SOTs find certain assumptions on object motion, appearance, and surroundings helpful on past datasets, but some of these design choices translate poorly to long-term egocentric videos.
Search window size. An example is local search. Many trackers assume the tracked object appears within a certain range of its previous location. Thus, for efficiency, these methods often search within a local window of the next frame. This is reasonable in high FPS, smooth videos with relatively slow motion, commonly in previous short-term tracking data, but in egocentric videos, the object's pixel coordinates can change rapidly (frequent large head motions), and re-detection becomes a key problem. Therefore, we experiment with expanded search regions beyond what are common in past methods. As we expand search size from 320 to 800, we see dramatic improvements (Table 9): STARK is able to locate objects that were previously outside search window due to rapid movements.
Multiscale augmentations. The characteristics of egocentric video also affect common SOT assumptions of object scale. Many trackers are trained with the assumption that an object's scale is consistent with the template image and between adjacent frames. However, large egocentric camera
motions, locomotion, and hand interactions with objects (e.g. bringing an object to one's face, as in eating) can translate to objects rapidly undergoing large changes in scale. We thus propose adding scale augmentations during training, randomly resizing the search image by a factor of  $s \in [0.5, 1.5]$ . While simple, we find this dramatically improves performance on EgoTracks, improving STARK's AO by nearly  $10\%$  and F-score by more than  $8\%$  (Table 9).
Context ratio. Past SOT works have found that including some background can be helpful for template image feature extraction, with twice the size of the object being common. We experiment with different context ratios to see if this rule of thumb transfers to egocentric videos. Because of the local window assumption, the sizes of the template and search images are related:  $\frac{\text{Search Image Size(SIS)}}{\text{Search Region Ratio(SRR)}} = \frac{\text{Template Image Size}}{\text{Context Ratio(CR)}} = \text{Object Scale}$ . The template image size is set to a fixed size  $128 \times 128$ . When changing the context ratio, we carefully control the other parameters for a fair comparison. The results are shown in Table 10. Among all three parameters - CR, SRR, and SIS, the search region size (determined by SRR and SIS) has the highest impact on the F-score. This is expected because there are frequent re-detections, which require the tracker to search in a larger area for the object, rather than just within the commonly used local window. Varying the CR has mixed results so we adhere to the common practice of using a CR of 2.
We present EgoTracks, the first large-scale dataset for long-term egocentric visual object tracking in diverse scenes. We conduct extensive experiments to understand the performance of state-of-the-art trackers on this new dataset, and find that they struggle considerably, possibly in part due to overfitting to some of the simpler characteristics of existing benchmarks. We thus propose several adaptations for the egocentric domain, leading to a strong baseline that we call Ego-STARK, which has vastly improved performance on EgoTracks. By publicly releasing this dataset, we hope to encourage advancements in the field of long-term tracking and draw more attention to the challenges of long-term and egocentric videos.
Challenges and future directions. Based on our experiments in Tables 4 and 5, we find re-detection to be a key challenge of long-term tracking, especially in egocentric video, where objects frequently go in and out of view, or are exposed to high motion blur. We see a few promising future directions:
- Stronger features: The sizable gap in performance between the Oracle and InstEmb variants of Tracking by Detection (Table 4) illustrates the impact of insufficiently discriminative features. Better features, for example with geometric keypoints, optical flow, or long-term trajectories [77], would allow improved association of objects and thus re-detection.
- Leveraging spatial signals: Camera trajectories can provide powerful spatiotemporal priors to trackers. For example, knowledge of camera pose can help re-localize a object whose position does not change while out of view.
- Global, multi-view object representations: Egocentric videos, with their diverse camera trajectories and tendency to capture interactions with objects, often offer significantly more varied viewpoints of objects than traditional third-person tracking datasets. In the latter, object appearances tend to be more constant, so prior tracking methods can often get away with a single image template. The aforementioned characteristics of egocentric video necessitate more robustness, and a challenging egocentric tracking dataset like EgoTracks represents an opportunity to develop more global, view-variant object representations learned in an online fashion. A simple version can be found in Appendix D, where we augmented EgoSTARK by fusing multiple templates, which we find indeed improved EgoTracks results.
[1] S. Avidan. Ensemble tracking. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 2, pages 494-501 vol. 2, 2005. 8
[2] Ronald T Azuma. A survey of augmented reality. Presence: teleoperators & virtual environments, 6(4):355-385, 1997. 1
[3] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182-6191, 2019. 3, 7, 8
A EgoTracks data details
A.1 Data splits
We followed the same data split as [29] for train, val and test sets. The exact number for the number of videos and number of tracks may not be the same as [29], because annotators were instructed to ignore videos and objects that they may not know how to annotate due to reasons like poor video resolution, failure to understand what object to track and so on.
Table 11: Data split for EgoTracks. We follow the same split as Ego4D VQ benchmark[29].
Split	# of videos	# of tracks
Train	3433	13140
Val	1154	4459
Test	1121	4429
Total	5708	22028
A.2 Object size distribution
Figure 6 shows the distribution of object sizes in EgoTracks. To account for the difference in video resolution in Ego4D, we use normalized object size instead of absolute pixel size to plot the distribution. The normalized object size is defined as the object size divided by image resolution:  $\sqrt{obj_w \times obj_h} / \sqrt{img_w \times img_h}$ .
Figure 6: Object size distribution of EgoTracks. The object size is normalized by video resolution to account for differences in resolution in Ego4D.
B Tracking by detection
In Section 4.2 of the main paper, we introduced a few baselines leveraging the paradigm of Tracking-by-Detection. We detail our designs and implementations in this section.
A Tracking-by-Detection framework contains two modules:
- A Detector that generates bounding box proposals. Each proposed bounding box contains an instance that potentially corresponds to the tracking target. Since SOT focuses on generic objects, this detector module should be as generic as possible, not limited to specific types of objects.
- An Association module that links proposed bounding boxes to template. This module takes proposals from the detector and chooses the one that most likely corresponds to the template object.
B.1 Detector models
We benchmark two off-the-shelf models: a baseline Mask R-CNN [30] and an open-world instance segmentation model GGN [78]. Both models are trained on COCO2017 [49] dataset with 80 object categories. The models are trained in a class-agnostic fashion (object vs. background) since it generalizes better on generic objects [79]. We choose an open-world model since we would like a detector that can detect objects regardless of their categories, one of the main advantage of openworld detectors. GGN uses a similar architecture as Mask R-CNN but achieves SOTA on multiple benchmarks through new training strategies dedicated to open-world. We use the bounding boxes from the box head as bounding box proposals. To evaluate detectors, we use an Oracle to select the best box matched to groundtruth (by maximum overlap with groundtruth).
Table 12: Comparing detectors on EgoTracks. Open-world detector GGN strongly outperforms baseline Mask R-CNN. @k means we select top-k detecting results.
Method	AO@10	AO@30	AO@100
Mask R-CNN	24.27	42.76	60.00
GGN	47.59	64.01	75.92
Both model achieve reasonable performance (Table 12). The AO@100 is much higher than tracking baselines. The open-world GGN model achieves much stronger performance on both low number of proposals (AO@10) and high number of proposals (AO@100). This shows that the ability to detect generic objects regardless of their categories is indeed an important attribute of EgoTracks. In addition, this also proves that EgoTracks contains many objects out of the popular COCO taxonomy, as Mask R-CNN performs significantly worse.
B.2 Associating bounding boxes
We use bounding box (instance) embedding to select from proposed bounding boxes as tracking output. Specifically, we compute embedding features on the cropped images from proposed bounding boxes and the template image. Then we compute cosine similarities between the template feature and all bounding box features. We select the bounding box with highest cosine similarity as tracking output, and the cosine similarity is output as tracking confidence.
We consider two models to extract embedding. (1) We use a standard ResNet-50 trained on ImageNet as a baseline. (2) To enforce discrimination across different instances, we use a ResNet-50 model trained with additional instance discrimination task: InstEmb. On top of a standard cross-entropy loss similar as the baseline model, we add a supervised contrastive loss [37] by contrasting across instances (both same categories and different categories). To generate more instance examples, we add Objects365 [69] to training data by cropping images based on bounding box annotations.
Table 13: Performance of different embedding models. We observe InstEmb model performs significantly better, but still has a large gap with the oracle in Table 12 on AO. This shows that temporal association is very challenging in EgoTracks.
Method	AO	F1	Precision	Recall
Baseline	2.5	1.2	0.8	2.5
InstEmb	15.2	9.9	11.8	8.6
We use the top-100 bounding boxes from GGN model. We summarize the two models in Table 13. We observe that using the baseline ImageNet features do not generate good association between detection results and template object. The InstEmb model achieves much stronger performance, showing the importance of instance discrimination from the additional instance contrastive loss. Despite its stronger performance, there is a significant gap compared to the oracle and most tracking methods. This shows that the temporal association problem is very challenging in EgoTracks.
C Effect of visual template for short-term tracking
We study the effect of different STARK template initializations to demonstrate how the template affects the tracking. More specifically, we experiment with two protocols: VCT-  $\overrightarrow{\mathbf{O O}}$  and OFFT-  $\overrightarrow{\mathbf{O O}}$ .
Table 14: Comparing initializing STARK with different templates.
Method	AO	Success	Pre	PreNorm
STARK - OFFT-OO	48.18	48.86	45.76	51.90
STARK - VCT-OO	51.10	51.83	47.78	53.96
Table 15: STARK with different numbers of templates.
Templates	AO	F-score	Precision	Recall
1	32.97	25.42	25.80	25.04
3	34.76	26.84	28.84	25.57
5	35.47	28.03	29.82	26.45
7	34.81	27.83	30.77	25.40
9	33.92	26.89	30.36	24.12
The difference between the two protocols is the template used for initializing the tracker: VCT-  $\overrightarrow{OO}$  uses the visual crop, while OFFT-  $\overrightarrow{OO}$  uses the first frame for each occurrence. To ensure a fair comparison between the two protocols, we ignore both the visual crop and the first frame for each occurrence when evaluating. Results are shown in Table 14. Please note the STARK - OFFT-  $\overrightarrow{OO}$  in Table 14 uses the same model as STARK in Table 5. The difference in performance is due to ignoring the visual crop in evaluation in Table 14. Even though the visual crop is not adjacent temporally to the start of each occurrence, VCT-  $\overrightarrow{OO}$  performs  $3\%$  better to OFFT-  $\overrightarrow{OO}$  in terms of AO because of the better template used.
D Multiple templates can improve tracking
Transformer-based architectures can encode arbitrary length inputs, making it straightforward to consume features from an arbitrary number of templates. The original STARK design encodes two templates: the initialization and a single dynamically updated template. A natural extension is to include more templates, which may expose the transformer to different views of the object (particularly relevant in egocentric video), though low-quality views may compromise performance [47].
What's the right trade-off? We experiment with different numbers of templates for a basic STARK model. Motivated by potential applications where a user can take a short video of an object from different angles [65], we extend the single visual crop to a visual clip of templates by incorporating additional frames from the same occurrence where the visual crop appears as the template. We adopt a simple template sampling method: uniformly sampling 3, 5, 7, or 9 templates from the visual crop's occurrence. Uniformly sampling the videos temporally can be a simple yet effective heuristic to gather diverse views from an occurrence. We summarize the results in Table 15. While we observe improvements across all metrics using up to 5 templates, performance declines with more. We hypothesize that increasing the number of templates does increase the knowledge available to STARK for tracking, but after a certain point it may dilute the information in the templates and make it difficult for the transformer to synthesize. This highlights the importance of template selection and multi-view fusion mechanisms, which inspires promising directions.
E Re-detection success rate
To more directly capture re-detection rate of trackers on EgoTracks, we pose it as a per-occurrence detection problem, where we consider that the tracker has successfully re-detected an object during an occurrence if there is at least one frame during the occurrence where the tracker produced a bounding box above some IOU threshold with the ground truth. We report the average proportion of occurrences for which the tracker meets this criterion below, at two IOU thresholds: 0.5 and 0.75. Table 16 confirms that recall (F-score) is consistent with the success rate of re-detection: methods with higher recall (therefore F-Score) also have higher re-detection success rate.
Table 16: Success rate of re-detection rate for different methods on EgoTracks.
Method	Recall@IoU 0.5	Recall@IoU 0.75
KYS	21.90	16.54
DiMP	22.35	16.77
GlobalTrack	42.72	35.55
LTMU	41.18	34.67
ToMP	39.85	34.81
Siam-RCNN	47.73	44.70
MixFormer	37.57	25.61
STARK - Res50	43.31	39.00
STARK - Res101	42.40	38.34
EgoSTARK	66.83	59.95
F Additional visualizations