SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Activity. An action describes what happens in a short period, while an activity can be seen as a collection of actions with clear causal relationships. A new activity is usually accompanied by a scene switch or an explicit change of the third-party object. Compared with the former action, if an action is preferred to be the beginning (i.e., reason) of a new event rather than an ending (i.e., result) of an old event, it can be regarded as a starting point of an activity. As shown in Figure 4 and Figure 1 (D1-D3), the first four actions describe a complete causality (the target approaches the third party, they fight, and cause the third party to be insensible), while the 5th action starts a new event (start examining the unconscious third-party and conduct a second fight when he wakes up). Therefore, the 4th and 5th actions can be divided into two different activities.
(a) Topics
(d) Temporal Continuity(e) Spatial Continuity
Figure 5: Statistical analysis of key aspects in MGIT. (a-b) Distribution of topics and length of sequences. (c) Distribution of activities and actions. The bubble diameter is in proportion to the length of a sequence, the vertical coordinate and the horizontal coordinate represent the total activities and actions of this sequence. (d-f) Distribution of temporal continuity, spatial continuity, and narrativity. (g) The word cloud of semantic descriptions.
(b) #Frames
(f) Narrativity
(c) Activities, Actions and Frames
(g) Word Cloud
Story. Story is a high-level description. To avoid boring narrativity, we do not stack the existing actions and activities, but use some words (e.g., first, then, after that, finally, etc.) to guide the content, making the temporal and causal more precise.
Based on the data collection process and the multi-granular annotation strategy, we construct MGIT with 2.03 million frames, and provide detailed annotation with 150 stories, 621 activities, and 982 actions. The semantic descriptions contain 77,652 words with 921 non-repetitive words, and more detailed analyses have been illustrated in Figure 5.
4.1 Datasets and Evaluation Methods
Datasets. We select OTB-Lang [1], TNL2k [3], LaSOT [2], and MGIT as experimental environments. Several variants of LaSOT are also concerned: (1)  $\mathrm{LaSOT_{Ext}}$  [17] is a complement of LaSOT [2] with 150 newly added video sequences. (2) Figure 1 indicates that several semantic descriptions in LaSOT are ambiguous. Thus, 22 ambiguous and 20 unambiguous sequences are selected to
Figure 6: Evaluation mechanisms of visual-based and multi-modal based trackers. (A) Traditional multi-modal tracking mechanism (i.e., only initialize a tracker with BBox and simple semantic information in the first frame). (B-D) Tracking with semantic information updates (i.e., initialize a tracker with BBox and semantic information in the first frame, then update the semantic information in each new interval). (E) Traditional one-pass evaluation (OPE) mechanism (i.e., only initialize a tracker with BBox in the first frame).
form the  $\mathrm{LaSOT_{sub}}$ , aiming to better analyze tracking performance with different kinds of natural language information. (3)  $\mathrm{LaSOT_{NLC}}$  is a subset of  $\mathrm{LaSOT_{sub}}$ , which is formed by the 20 unambiguous sequences, and we have carefully checked all the semantic and visual information in this subset.
Evaluation Methods. As shown in Figure 6, various mechanisms are designed to evaluate tracking precision (PRE) and success rate (SR). We use  $F_{t}$  to represent the  $t$ -th frame. (1) Precision is calculated based on the center distance between the predicted BBox  $p_{t}$  and the ground truth BBox  $g_{t}$  (i.e.,  $d_{t} = \| c_{p} - c_{g}\|_{2}$ , where  $c_{p}$  and  $c_{g}$  represent center points). By calculating the proportion of frames where  $d_{t} \leq \theta_{d}$  and plotting curves at different thresholds, we can generate a precision plot. PRE is common to use  $\theta_{d} = 20$  as the criterion to rank trackers. (2) Furthermore, researchers [19] provide the normalized precision (N-PRE) to eliminate the effect of target size. When trackers have a predicted center outside the ground-truth, an additional penalty term, represented by  $d_{t}^{p}$ , is included to account for the shortest distance between the center point  $c_{p}$  and the edge of the ground-truth. The final result is then normalized to a range of 0 to 1 (i.e.,  $N(d_{t}) = \frac{d_{t}^{\prime}}{\max(\{d_{i}^{\prime}|i\in F_{t}\})}$ , where  $d_{t}^{\prime} = d_{t} + d_{t}^{p}$ ). Similarly, the normalized precision plot is generated by plotting statistical outcomes derived from various  $\theta_{d}^{\prime}$  values. (3) Besides, frames with the intersection over union (IoU)  $\Omega(p_{t},g_{t}) = \frac{p_{t} \bigcap g_{t}}{p_{t} \bigcup g_{t}} \geq \theta_{s}$  can be regarded as successful tracking, and the SR measures the percentage of successfully tracked frames. Drawing the results based on various  $\theta_{s}$  is the success plot. For more details on the evaluation metrics, please refer to Section B.1 in the Appendix.
Table 1: Results on different multi-modal benchmarks (based on mechanism A in Figure 6).
Tracker	OTB-Lang [1]	TNL2k [3]	LaSOT [2]	LaSOTExt [17]	LaSOTSub	LaSOTNLC	MGIT
PRE	SR	PRE	SR	PRE	SR	PRE	SR	PRE	SR	PRE	SR	PRE	SR
SNLT [46]	0.848	0.666	0.081	0.100	0.475	0.459	0.306	0.262	0.527	0.495	0.513	0.483	0.004	0.036
VLT_SCAR [42]	0.898	0.739	0.556	0.497	0.677	0.630	0.503	0.428	0.670	0.633	0.659	0.633	0.124	0.177
VLT_TT [42]	0.931	0.764	0.583	0.539	0.714	0.670	0.549	0.465	0.707	0.660	0.721	0.662	0.324	0.474
JointNLT [18]	0.856	0.653	0.598	0.552	0.640	0.607	0.457	0.398	0.624	0.583	0.707	0.651	0.433	0.603
4.2 Comparison with Other Multi-modal Benchmarks (Mechanism A)
We select several SOTA multi-modal trackers as baseline models and evaluate them on various benchmarks (as shown in Table 1). To fairly compare the tracking performance on MGIT and other datasets, we only allow trackers to use the semantic information of the first action in this experiment. Results show that: (1) most trackers perform worst on MGIT, which means it is a more complex environment with more challenges. (2) By comparing the tracking results on  $\mathrm{LaSOT}_{\mathrm{sub}}$  and  $\mathrm{LaSOT}_{\mathrm{NLC}}$ , we can find that most trackers perform worse on  $\mathrm{LaSOT}_{\mathrm{sub}}$ , showing that ambiguous
semantic information may introduce external interferences. Thus, we avoid this problem via the scientific annotation and check process for MGIT construction.
4.3 Experimental Results on MGIT
Tracking by NL&BBox (Mechanism B-D). As shown in Figure 6 (B-D), both visual information (BBox of the first frame) and semantic information (natural language description) can be used for multi-modal trackers. Specifically, different granularities have various lengths, while most trackers have a maximum limit of the input semantic information. JointNLT [18] sets 50 as a maximum limit and truncates the excess information. This truncation occurs for both activity (C) and story (D). Similarly, the VLT [42] series limits the semantic length but can avoid truncation by adjusting the parameters. Thus, we set the semantic length to 80 for the activity and 200 for the story, with zero padding as necessary. From Table 2, we can draw the following conclusions: (1) SNLT, VLT_SCAR, and VLT_TT perform well when using longer semantic information like activity and story. This indicates that the semantic information processing modules (BERT [47]) used in these trackers can effectively handle long text. Besides, their good performances in activity indicate that as an intermediate granularity, activity accomplishes a balance between the amount of information and the number of semantic description updates. (2) On the contrary, JointNLT performs well on action rather than levels with longer descriptions, suggesting that truncated semantic information leads to poorer performance. Therefore, to obtain better multi-modal information processing capabilities, algorithms should first ensure that long texts can be processed rather than truncated directly.
Table 2: Results of different trackers on MGIT.
Tracker	Architecture	Initialize	Mechanism	PRE	N-PRE	SR
SiamCAR [11]	SNN	BBox	0.116	0.378	0.183
SiamRCNN [10]	SNN	BBox	0.512	0.707	0.591
PrDiMP [12]	SNN+CF	BBox	0.296	0.602	0.453
KeepTrack [13]	SNN+CF	BBox	E	0.373	0.695	0.519
TransT [39]	Transformer	BBox	0.447	0.670	0.539
MixFormer [14]	Transformer	BBox	0.526	0.775	0.629
OSTrack [15]	Transformer	BBox	0.476	0.706	0.583
GRM [16]	Transformer	BBox	0.500	0.718	0.597
SNLT [46]	SNN	NL&BBox	Action (B)	0.004	0.226	0.036
Activity (C)	0.004	0.234	0.038
Story (D)	0.005	0.230	0.040
VLT_SCAR [42]	SNN	NL&BBox	Action (B)	0.116	0.354	0.167
Activity (C)	0.124	0.382	0.180
Story (D)	0.127	0.403	0.184
VLT_TT [42]	Transformer	NL&BBox	Action (B)	0.318	0.602	0.468
Activity (C)	0.325	0.627	0.485
Story (D)	0.322	0.616	0.480
JointNLT [18]	Transformer	NL&BBox	Action (B)	0.445	0.786	0.610
Activity (C)	0.441	0.780	0.605
Story (D)	0.433	0.773	0.600
By comparing results under mechanisms A and D, we can find that in this complex environment, well-designed trackers (i.e., trackers with suitable long input processing ability) can perform better via longer descriptions than only relaying a short description (SNLT:  $0.036 \rightarrow 0.040$ , VLT_SCAR:  $0.177 \rightarrow 0.184$ , VLT_TT:  $0.474 \rightarrow 0.480$  in SR). The above experiments indicate two key points: (1) Richer semantic information (mechanism D based on story) can improve the tracking perfor
mance than a simple sentence (mechanism A based on information for the first action), which can also verify the accuracy and necessity of the proposed multi-granularity semantic annotation strategy. (2) Only providing a simple description for multi-modal trackers is unreasonable for MGIT. Thus, initializing the tracking process with longer and more specific sentences, or updating the semantic information periodically throughout the sequence, has been found to be more effective in accurately locating targets within complex scenes.
Tracking by BBox Only (Mechanism E). We mainly evaluate SOTA visual-based trackers under mechanism E. As shown in Table 2, by comparing with other trackers, the transformers-based trackers have emerged as the predominant approach and achieved SOTA performance. Besides, it is worth noting that visual-based trackers usually outperform multi-modal trackers. Although we hope that more modal information can improve the tracking performance, the current multi-modal approaches cannot better align different modalities, resulting in the multi-modal information not being fully utilized. In contrast, visual-based methods have been well developed over the past decades and can better use visual features to accomplish good tracking performance. This result (i.e., current multi-modal trackers are worse than visual-based trackers) can also be found in other works, highlighting the significant room for improvement in multi-modal tracking. More detailed experimental results and analyses can be found in Section B.3 of the Appendix.
4.4 Visualization and Bad Case Analysis
LaSOT bird-2 sequence: white bird walking on the among other birds
the among other birds
LaSOT zebra-16 sequence: zebra running on dry grass with other zebras
MGIT 012 sequence (mechanism A): A pink cartoon pig wearing red clothes talks to her family members on the grassland.
GroundTruth
JointNLT (CVPR23)
Figure 7: Bad cases of representative multi-modal trackers on LaSOT [2] and MGIT. (A-B) Ambiguous semantic annotations on LaSOT lead trackers to locate at similar objects. (C-D) The mechanism A used in existing multi-modal SOT benchmarks is unable to adapt to complex scenes like MGIT.
VLT_TT (NeurIPS22)
MGIT 006 sequence (mechanism A): A skateboard is slid by a man in black on the playground.
SNLT (CVPR21)
We further analyze the bottlenecks of the multi-modal algorithms through the bad cases shown in Figure 7. The first two examples are selected from LaSOT [2], demonstrating that ambiguous semantic information may introduce noise, leading algorithms to wrongly focus on similar objects – this emphasizes the importance of accurate semantic annotations. The latter two examples are chosen from MGIT, demonstrating that the experimental environment constructed by MGIT presents complex spatio-temporal and causal relationships, posing challenges to multi-modal algorithms. Specifically, the complexity of MGIT results in significant differences between the target appearance and background environment in the initial frame and subsequent states. Besides, MGIT is selected from the recently released VideoCube [19] benchmark, which has a higher image resolution, posing challenges for trackers to relocate the target after failure. Additionally, using only the first action information (mechanism A) is applied in all other multi-modal SOT benchmarks. However, it is not applicable to visual object tracking in complex scenes like MGIT (Figure 7(C-D)). Therefore, the proposed multi-granularity annotation strategy offers a more reasonable solution. Multi-modal trackers who want to perform better on MGIT need a more well-designed semantic information processing module to accurately extract useful information described by semantic labels. Nevertheless, existing trackers have not made specialized designs for this aspect, which can be further improved.
Summary. Accuracy target tracking is the foundation for accomplishing high-level tasks like long video understanding, and introducing natural language into visual object tracking is a possible way to increase tracking ability. Different from existing multi-modal benchmarks that mainly consisted of short sequences with simple or even ambiguous descriptions, we (1) propose a new multi-modal benchmark named MGIT with 150 long video sequences, and (2) design a multi-granular annotation strategy for generating scientific semantic information. On the one hand, MGIT is a challenging and complex environment for visual tracking and video understanding research (i.e., trackers should process the spatio-temporal and causal relationships coupled with longer narrative content to accomplish better performance). On the other hand, the multi-granular annotation strategy models the human cognitive enhancement process, which may provide a step-by-step "learning" environment for generating human-like trackers. The experimental results demonstrate that MGIT is a more complex environment, and our proposed strategy is a feasible solution for coupling human understanding into semantic labels. Besides, existing trackers still have a large room for development, like improving the capability for processing long text and aligning multi-modal information. Conclusioally, we hope this work can help researchers to conduct further research in object tracking and video understanding.
Limitations. Some limitations here can be further enhanced in future work. First, we can expand MGIT with more types of videos to provide a more complicated environment for data-driven algorithms. Besides, we can design a better comprehensive evaluation system to measure visual tracking and video understanding ability. Finally, we can add more types of tasks based on the benchmark, and try to test algorithms for tasks like video caption and action recognition.
This work was supported in part by the National Key R&D Program of China (No.2022ZD0116403); the National Natural Science Foundation of China (No.61721004); the Strategic Priority Research Program of Chinese Academy of Sciences (No.XDA27000000).
[44] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021.
[45] Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60, 2014.
[46] Qi Feng, Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5847-5856, 2021.
[47] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[48] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944-10956, 2021.
A Dataset Information
A.1 Basic Information
In this work, we propose a new multi-modal global instance tracking benchmark named MGIT. It consists of 150 long video sequences with a total of 2.03 million frames, aiming to fully represent the complex spatio-temporal and causal relationships coupled in longer narrative content.
This work further expands our former work, which was accepted by IEEE TPAMI in 2022. We proposed a global instance tracking task in the previous work and released an online evaluation platform (URL: http://videocube.aitestunion.com). We hope the online platform can help researchers use our proposed resources better and conduct more fair comparisons via our real-time evaluation server.
In the past year, our platform has evaluated 380 algorithms and received more than  $287\mathrm{k}$  IP visits from 130 countries (statistics by Jan 04, 2024). However, all submitted algorithms do not show significant performance improvement on the GIT task, and their tracking performances are significantly degraded on the GIT task compared to other representative single object tracking benchmarks (i.e., short-time tracking and long-time tracking). This phenomenon shows the limitations of a single visual modality for long video understanding of complex narrative relationships. Thus, we conduct this work to introduce semantic information, which aims to help the algorithms better cope with the challenges posed by complex narrative relationships for target tracking and long video understanding.
Since the motivation for this work is closely inherited from our former work, and the existing online platform has received considerable attention worldwide, we select to release the MGIT benchmark via the same online platform. The proposed benchmark, experimental results, and toolkit will be released gradually on http://videocube.aitestunion.com/ (Figure A1).
Our dataset has been uploaded to OneDrive and Baiduun disk, and the online evaluation platform is maintained by dedicated staff, which will ensure the stability of the dataset.
We declare that we bear all responsibility in case of violation of rights, etc., and confirm the data license. Our work is licensed under CC BY-NC-SA 4.0. Users are free to use the dataset for research purposes.
Figure A1: Our online platform and currently updated download links with related instructions.
A.2 Data Selection
The 150 sequences in MGIT are carefully selected from the original VideoCube [19] benchmark. In the video selecting part, we thoroughly consider the consistency between the new dataset (MGIT)
and the original dataset (VideoCube) in various dimensions (6D principle), while also taking into account the consistency of difficulty.
The specific process is as follows. (1) We assess the similarity in the distribution of the selected MGIT dataset to the original VideoCube dataset across various dimensions, including object class, scene category, motion mode, and more, in accordance with the 6D principle. (2) Besides, it is essential for the selected dataset to maintain a similar level of difficulty as the original VideoCube dataset. Regarding the difficulty level, we select three state-of-the-art trackers (MixFormer [14], KeepTrack [13], and SiamRCNN [10]) with different architectures as the basis for our selection criteria. The success scores (based on IoU) of these algorithms on the original 500 sequences are calculated and ranked to measure sequence difficulty. (3) By considering both the distribution across the 6D principle and the difficulty level, we carefully choose 150 representative sequences to construct the MGIT dataset, aiming to maintain consistency with the distribution of the original VideoCube dataset.
A.3 Semantic Annotation
A.3.1 Semantic Annotation Deficiencies of Existing Multi-modal SOT Benchmarks
To better show the semantic annotation deficiencies of existing multi-modal SOT benchmarks, we conduct statistical analyses on OTB-Lang [1], LaSOT [2], LaSOT $_{\text{Ext}}$  [17], and TNL2k [3] from two aspects:
1. Ambiguity of semantic labeling: A random sampling inspection is conducted to address the ambiguity of semantic annotation. Specifically, 10 sequences are randomly selected from each dataset for inspection. To ensure random and fair selection, all sequences in each dataset are alphabetically sorted first, and samples are taken at equal intervals.
2. Completeness of grammatical structures: Semantic descriptions of high quality typically necessitate complete grammatical structures. Hence, Stanford CoreNLP [45] is utilized to analyze the semantic labels in all four datasets, and statistics those that adhered to the criteria of complete sentences.
Table A3: The statistics of semantic annotation quality in four representative datasets.
Benchmark	OTB-Lang [1]	LaSOT [2]	LaSOTExt [17]	TNL2k [3]
Statistical Analysis 1: Inspection Pass Rate (Non-ambiguous Semantic Descriptions)	30%	70%	60%	60%
Statistical Analysis 2: Complete Sentences Rate (Checked by Stanford CoreNLP, Including Complete Grammatical Structures)	9%	63%	36%	20%
Figure A2: Statistical analysis about the completeness of grammatical structures, based on Stanford CoreNLP [45] toolkit. (NP: noun phrase; FRAG: fragment; S: simple declarative clause; SQ: inverted yes/no question, or main clause of a wh-question. Only S and SQ satisfy the completeness of the grammatical structure.)
As shown in Table A3 and Figure A2, the current datasets exhibit deficiencies regarding ambiguity and completeness. However, our proposed MGIT benchmark considers these factors during construction and avoids the previously mentioned issues, thus possessing higher-quality semantic annotations.
A.3.2 Annotation Process
We chose an elite annotation team instead of crowdsourcing to carry out this work and ensured quality through multiple efforts.
A.3.3 Annotation File
We propose a multi-granular annotation strategy to generate the semantic description, and use JSON format to save the natural language annotation for each video sequence. Here we illustrate an example to show the JSON file structure for video sequence 001 in the MGIT benchmark, as shown in Listing 1. Due to the limited space, we only illustrate some representative information, while the remaining information with similar structure is indicated by ellipses. Please download and check the dataset for more detailed annotation about each video sequence.