SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shiyu Hu<sup>1,2</sup> Dailing Zhang<sup>1,2</sup> Meiqi Wu<sup>3</sup> Xiaokun Feng<sup>1,2</sup>
Xuchen Li $^{4}$  Xin Zhao $^{1,2}$  Kaiqi Huang $^{1,2,5}$
$^{1}$ School of Artificial Intelligence, University of Chinese Academy of Sciences
$^{2}$ Institute of Automation, Chinese Academy of Sciences
$^{3}$ School of Computer Science and Technology, University of Chinese Academy of Sciences
$^{4}$ School of Computer Science, Beijing University of Posts and Telecommunications
$^{5}$ Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences
{hushiyu2019, zhangdailing2023, fengxiaokun2022}@ia.ac.cn, wumeiqi18@ mails.ucas.ac.cn,
xuchenli@bupt.edu.cn, {xzhao,kqhuang} @ia.ac.cn
Tracking an arbitrary moving target in a video sequence is the foundation for high-level tasks like video understanding. Although existing visual-based trackers have demonstrated good tracking capabilities in short video sequences, they always perform poorly in complex environments, as represented by the recently proposed global instance tracking task, which consists of longer videos with more complicated narrative content. Recently, several works have introduced natural language into object tracking, desiring to address the limitations of relying only on a single visual modality. However, these selected videos are still short sequences with uncomplicated spatio-temporal and causal relationships, and the provided semantic descriptions are too simple to characterize video content. To address these issues, we (1) first propose a new multi-modal global instance tracking benchmark named MGIT. It consists of 150 long video sequences with a total of 2.03 million frames, aiming to fully represent the complex spatio-temporal and causal relationships coupled in longer narrative content. (2) Each video sequence is annotated with three semantic grains (i.e., action, activity, and story) to model the progressive process of human cognition. We expect this multi-granular annotation strategy can provide a favorable environment for multi-modal object tracking research and long video understanding. (3) Besides, we execute comparative experiments on existing multi-modal object tracking benchmarks, which not only explore the impact of different annotation methods, but also validate that our annotation method is a feasible solution for coupling human understanding into semantic labels. (4) Additionally, we conduct detailed experimental analyses on MGIT, and hope the explored performance bottlenecks of existing algorithms can support further research in multi-modal object tracking. The proposed benchmark, experimental results, and toolkit will be released gradually on http://videocube.aitestunion.com/.
Single object tracking (SOT) is an important computer vision task that aims to locate an arbitrary moving target in a video sequence, and can be regarded as the foundation for high-level tasks like video understanding. In the past decade, researchers have proposed numerous high-quality benchmarks [4, 5, 6, 7, 8, 9] for the visual-based SOT task, and a series of trackers [10, 11, 12, 13, 14,
OTB-Lang Liquor sequence: brown liquor bottle
LaSOT airplane-1 sequence: white airplane landing on ground
TNL2k Arrow_Video_ZZ04_done sequence: the second arrow from left to right
Action 1: A male secret agent wearing a black suit walks in the washroom.
Activity 1: A male secret agent wearing a black suit walks in the washroom, and stands near a man wearing a light grey suit. They fight, then the male secret agent wins, and lifts the insensible grey-suit man to the washroom cubicle.
Story: A male secret agent wearing a black suit walks in the washroom, and stands near a man wearing a light grey suit. They fight, and the male secret agent wins. He then lifts the insensible grey-suit man to the washroom cubicle. The male secret agent crouches in the washroom cubicle and checks the insensible grey-suit man. Suddenly, the grey-suit man wakes up, and they fight together again in the washroom. Eventually, the male secret agent wins the fight. After the male secret agent talks with a woman wearing a brown suit, he again lifts the insensible grey-suit man to the washroom cubicle. Finally, the male secret agent lefts the washroom after talking with the brown-suit woman.
Figure 1: Comparison of MGIT and other multi-modal object tracking benchmarks. (A-C) Examples of video content and semantic descriptions on OTB-Lang [1], LaSOT [2], and TNL2K [3]. The green bounding box (BBox) indicates ground truth, while the red dashed BBox indicates other objects that satisfy the semantic description. These benchmarks have short sequences with simple narrative content. Besides, their semantic labels mainly describe the first frame, which may misguide algorithms. (D1-D3) An example of the multi-granular annotation strategy used by MGIT. Compared to existing benchmarks, MGIT contains longer sequences with more complex narratives, and the multi-granular annotation provides more prosperous and flexible information to portray long videos.
15, 16] have demonstrated good tracking capabilities in these environments, especially in short video sequences ranging from hundreds to thousands of frames. However, researchers noticed that most trackers always perform poorly in longer videos with more complicated narrative content. Besides, only relying on a single visual modality also limits the application scenarios. Thus, several works have begun to offer additional semantic annotations for SOT task.
As the first multi-modal SOT benchmark, OTB-Lang [1] provides a language description for the classic OTB [5] benchmark, hoping to provide a more natural human-machine interaction method. The long-term tracking benchmark LaSOT [2, 17] also supplies a semantic annotation for each sequence, desiring to utilize linguistic features to improve the tracking performance. TNL2k [3] wants to achieve more flexible and accurate tracking ability with more explicit information (e.g., location information) in the semantic description. Although these multi-modal benchmarks have introduced semantic information into visual object tracking, they still face the following problems. (1) Short sequences with uncomplicated spatio-temporal and causal relationships: Existing works mainly focus on videos with hundreds to thousands of frames (the average sequence lengths of OTB-Lang, LaSOT, and TNL2k are 590 frames, 2,502 frames, and 622 frames), while shorter video sequences are always insufficient to reflect complex narrative content. (2) Simple semantic descriptions: The quality of semantic information is critical to multi-modal trackers' performance, while incorrect or ambiguous semantic information may misguide algorithms in tracking interference [18]. However, the semantic labels in existing works mainly describe the state in the first frame, but lack the portrayal
of the complete sequence. For example, the brown liquor bottle description of Figure 1 (A) cannot distinguish the object from the interference (another brown liquor bottle). In Figure 1 (B), white airplane landing on ground may also misdirect trackers to locate another airplane that has parked on the right ground. In Figure 1 (C), the second arrow from left to right only satisfies to represent the object state at the beginning of the sequence; as the object moves, the position constraint contained in the semantic information will become misleading. Consequently, a better way to construct a multi-modal benchmark is not to provide a simple natural language description for short videos, but to design a scientific way to couple human understanding of long videos into semantic labels.
Therefore, we should first select suitable long videos with rich narrative relationships to compose a complex environment. VideoCube [19] is a high-quality benchmark recently released for the global instance tracking (GIT) task (i.e., search an arbitrary user-specified instance in a video without any assumptions about motion consistency), which can be regarded as expanding the definition of traditional SOT task (i.e., tracking a target in single-camera and single-scene) to success model the human visual tracking ability in a complex environment. Thus, we selected 150 representative long video sequences from VideoCube to form a new multi-modal benchmark named MGIT. The proposed new benchmark is consistent with the distribution of the original VideoCube in all dimensions (e.g., length, scene categories, object classes, motion modes, spatio-temporal consistency, and difficulty). Besides, we carefully check the content of each sequence to ensure that the selected data contain as many different types of video narratives as possible. Figure 1 (D1-D3) illustrates an example in MGIT. Compared with other works, sequences in MGIT include more complex content (i.e., the spatial-temporal variation and causal relationship are more complicated).
Besides, we design a multi-granular annotation strategy to provide scientific natural language information. On the one hand, existing research has indicated that complex narrative content can be perceived as several components and their relations, which is consistent with cognitive intuition [20]. On the other hand, the process of human comprehension and cognitive development is progressive as well [21, 22]. Therefore, designing a hierarchical structure to represent the video content is a reasonable annotation method. As shown in Figure 1 (D1-D3), each sequence in MGIT is annotated with three semantic grains (i.e., action, activity, and story). We hope this method can provide a step-by-step "learning" environment for multi-modal trackers, in which they can first learn multi-modal information at a fine-grained level (action), then gradually develop to a morea comprehensive level (activity), and finally understand the complex video narrative at a story level like humans.
Contributions. (1) We propose a new multi-modal benchmark named MGIT. It consists of 150 long videos with a total of 2.03 million frames, and the average length of a single sequence is  $5 \sim 22$  times longer than existing multi-modal benchmarks. We hope this new benchmark fully represents the complex spatio-temporal and causal relationships coupled in longer narrative content (Section 3.1). (2) We design a multi-granular annotation strategy for providing scientific semantic information. Via this strategy, MGIT can provide a favorable environment for multi-modal object tracking research and long video understanding (Section 3.2). (3) We execute comparative experiments on other benchmarks. Experimental results explore the impact of different annotation methods, and validate that the proposed strategy is a feasible solution for coupling human understanding into semantic labels (Section 4.2). (4) We conduct detailed experimental analyses on MGIT. Results indicate that existing methods still have significant room for improvement in multi-modal tracking (Section 4.3). The proposed benchmark, experimental results, and toolkit will be released gradually on http://videocube.aitestunion.com/.
Benchmarks with Visual Information. Standard SOT trackers are always initialized in the first frame by a target's bounding box (BBox), then continuously locating it in the video sequence. Since 2013, many benchmarks represented by OTB [4, 5] and VOT [6, 23] have been released, and these standardized datasets with scientific evaluation mechanisms promote the SOT research. With the development of deep learning techniques, these short-term and small-scale benchmarks have struggled to support data-driven trackers. Thus, several researchers have started to design larger-scale datasets like GOT-10k [9] and TrackingNet [8], while others have tried to collect data with longer videos and proposed long-term tracking benchmarks like OxUvA [24] and VOT-LT [25, 26]. Recently, some researchers have noticed that short-term and long-term tracking tasks include a continuous motion assumption in their definitions, resulting in the experimental environments being restricted to
single-camera and single-scene. Therefore, they propose the global instance tracking task [19] with a new benchmark named VideoCube to track an arbitrary moving target in any type of video.
Benchmarks with Visual and Semantic Information. Unlike numerous visual benchmarks that have evolved over a decade, multi-modal benchmarks combining visual and semantic information have only received attention lately. OTB-Lang [1] is the first multi-modal SOT benchmark, which provides additional natural language description for sequences in OTB100 [5] benchmark. However, the limited dataset scale has prevented the multi-modal SOT task from receiving widespread attention. After that, a large-scale and long-term tracking benchmark LaSOT [2, 17] is released with multimodal annotations. In the same year, researchers propose the TNL2k [3] benchmark to achieve more flexible and accurate object tracking with natural language. These two benchmarks have provided a prosperity of data and have facilitated the generation of various multi-modal trackers.
Figure 2: Comparison between MGIT with other SOT benchmarks, including visual-based (e.g., OTB50 [4], OTB100 [5], GOT-10k[9], TrackingNet [8], OxUvA [24], and VideoCube [19]) and multi-modal SOT benchmarks (e.g., OTB-Lang [1], LaSOT [2], and TNL2k [3]). The bubble diameter is in proportion to the total frames of a benchmark, and the vertical coordinate represents the average sequence length of each benchmark. Obviously, the proposed MGIT includes longer videos with multi-modal information.
As shown in Figure 2, existing works either focus on visual modality, or concentrate on multimodality but lack longer videos with complex content. Besides, Figure 1 indicates a more scientific annotation strategy is also needed for providing high-quality semantic information. These limitations prompt us to propose MGIT, hoping to construct a more complex and flexible environment for research.
Algorithms with Bounding Box. Visual-based trackers always utilize the target's appearance and motion information to accomplish the tracking process, including the correlation filter (CF) based trackers [27, 28], Siamese neural network (SNN) based trackers [29, 30, 31, 32, 33, 34, 11, 35, 36, 10], the combination of CF and SNN [37, 38, 12, 13], and the transformer-based trackers [39, 14, 15, 16]. Before 2021, SNN-based trackers are the prevalent methods. Recently, transformer-based trackers have demonstrated exemplary performance and gradually become the dominant architecture.
Algorithms with Bounding Box and Natural Language. Tracking a moving target with visual
and semantic information is a new task for SOT research; thus, representative works are mainly released in recent two years. AdaSwitcher [3] is released with the TNL2k benchmark, which proposes a switcher that utilizes natural language to alternate search mechanism (i.e., switch between the global search visual grounding module and the local visual tracking module). GTI [40] decomposes the visual language tracking task into three sub-tasks: tracking, grounding, and integration, and verifies the performance of each sub-module. SINT [41] proposes a semantic information fusion module that can be utilized across various SNN-based trackers. VLT [42] introduces a modality mixer named ModaMixer with asymmetric ConvNet search, which aims to demonstrate pure ConvNet models can achieve comparable results to state-of-the-art (SOTA) transformer-based algorithms. Besides, the proposed ModaMixer can further improve performance when directly applied to transformer-based trackers. JointNLT [18] unifies visual grounding and tracking as a coherent task (i.e., locating referred objects based on visual-language references). It employs the transformer-based architecture to model the relation between natural language and visual information.
3 Construction of MGIT
We propose a new multi-modal benchmark named MGIT and design a multi-granular annotation strategy for generating scientific semantic information. On the one hand, we have carefully selected 150 longer video sequences to form MGIT (please refer to Section A.2 in the Appendix for more details), hoping this complex environment can promote visual tracking and video understanding research. On the other hand, we hope this multi-granular annotation strategy can provide a step-by
step "learning" environment for multi-modal trackers. Like humans can increase their comprehension by gradually increasing the learning difficulty, trackers can first learn at a fine-grained level (action), then to a more comprehensive level (activity), and finally accomplish a story level understanding of long video sequences. A well-trained elite annotation team is selected to execute this task instead of crowdsourcing, and the annotation quality is ensured through various efforts. The detailed workflow has been outlined in Section A.3.2 of the Appendix.
3.1 Data Collection
Story: A pink cartoon pig wearing red clothes talks to her family members on the grassland. Today, the red-clothes pig and her family aim to visit a castle. They go to the castle in a red car, and the red-clothes pig sits in the back. They stop the vehicle nearby the foothills and walk to the castle. At the entrance of the castle, they meet a white cartoon pig wearing gray armor. The red-clothes pig first talks with the gray-armor pig, then they are invited to visit the castle. The red-clothes pig walks with her family into the castle and sits beside a blue-clothes pig on the chair. After that, they have a meal in the castle's living room, and the red-clothes pink pig gets a gift from a yellow-clothes pig after the meal. Finally, the red-clothes pig walks with her family members on the stairway, and then stands at the top of the tower.
Story: A black gorilla holding a lady in white crouches on a gray building, and some airplanes attack them. He then walks and climbs to the top of the grey building. After that, he stands atop the grey building, hits an airplane, fights with a gray soldier in the other airplane, and finally crouches on the gray building.
Story: A black motorcycle is checked by a man with orange and white clothes in the yard; then, the man rides this black motorcycle in the yard. As an obstacle race, the black motorcycle first bounces across obstacles in the playground, then bounces across obstacles in the street. After that, it bounces across obstacles near the pool and across obstacles in the stream. After a brief break, the black motorcycle bounces across obstacles in the playground, then across obstacles near the pool, and finally across obstacles in the stream.
Story: A small basketball is played by a boy with a grey t-shirt and black shorts, and then inflated by a man with a red t-shirt and black pants in the skatepark. After that, the basketball is played by the boy, and then played by the man. After they practice, the basketball is holden by the boy from the skatepark to outdoors; then it is played by the boy outdoors. Finally, The basketball then is carried away by the boy.
Story: A brown cello is played by a man with white shirt and black Story: A red cap is worn by a man with a gray t-shirt on the soccer pants in the room. court.
Figure 3: The representative data of MGIT. Here we illustrate six sequences with different aspects (e.g., narrativity, topics, virtuality, object classes, spatio-temporal continuity, and total frames).
MGIT follows the recently proposed large-scale benchmark VideoCube [19] to conduct the data collection. VideoCube refers to the film narrative (i.e., a chain of causal relationship events occurring in space and time) and proposes the 6D principle for benchmark construction. In this work, we divide the 6D principle into two parts. Four dimensions (i.e., object class, spatial continuity, temporal continuity, and total frame), together with narrativity and topic, form the new sequence-level selection criterion. The other two dimensions (i.e., motion mode and scene category) will be refined during fine-grained semantic annotation. Therefore, we first regard the original VideoCube as the candidate samples, then add the additional examination of narrativity and topic, and finally select 150 video sequences to form the MGIT. Particularly, the proportions of the train/val/test subsets are the same as the original VideoCube. Thus, sequences in each subset are  $105 / 15 / 30$  in MGIT. Taking Figure 3 as an example, here we present several dimensions considered in the data collection process:
Topic and Narrativity. We have divided the main video topics into six categories, which are cartoons, movies & TV shows, outdoor sports, regular sports, performances, and documentaries. Among them, cartoons and movies & TV shows usually have a high narrativity (i.e., the video content contains a solid causal relationship, as shown in Figure 3 A and B). Outdoor sports and regular sports contain rich patterns of motion, and these motions can be linked chronologically into a story, but the narrativity is usually simple than in cartoons and movies (Figure 3 C and D). Compared to other topics, performances and documentaries usually record one action with low narrativity (Figure 3 E and F). However, these examples are classifications for most cases; it is worth noting that some performances (e.g., sketches with explicitly narrative content on stage) and some documentaries (e.g., documentaries with causal teaching steps) also belong to high narrativity.
Spatial Continuity and Temporal Continuity. Temporal continuity means the video content is developed according to the normal time flow (i.e., without fast-forwarding, fast-receding, or interpolation). Spatial continuity means the video content takes place in a fixed space.
**Virtuality.** Virtuality refers that this video is computer-generated, like cartoons or games. The same content in virtuality videos can be very different from videos sampled from the real world; thus, virtuality videos can present a new challenge for object tracking and long video understanding.
3.2 Natural Language Annotation
Figure 4: An example of action annotation. We label the target, motion pattern, third-party object, and scene for each action. The target to be tracked is determined in the first frame and does not change during the entire video sequence. A change in any of the other three elements will end the current action and proceed into the following action.
In this work, we design a multi-granular annotation strategy to provide scientific natural language information. Video content is annotated by three grand (i.e., action, activity, and story, as shown in Figure 1 D1-D3). This hierarchical structure to represent the video content is motivated by existing works in computer vision [20, 43] and human cognitive [21, 22], such as a recent method [43] decouples the video content into multiple granularities for the visual question-answering task [44], aiming to help algorithms better understand video information like humans.
Action. As shown in Figure 4, we use the following critical narrative elements to portray an action: tracking target (who), motion (what) and third-party object (if present), location (where), and time interval (when). On the one hand, the above elements are necessary to portray narrative content. On the other hand, these elements are also essential grammatical components to form complete sentences. In particular, we use Stanford CoreNLP [45], a widely used natural language processing toolkit, to check the semantic annotations of other multi-modal datasets. We find that more than half of these semantic descriptions are only annotated at the phrase level, lacking the necessary grammatical structure (the detailed statistic result has been shown in Section A.3.1 of the Appendix). Thus, compared with existing works, MGIT can describe more detailed narrative content.