A Multi-modal Global Instance Tracking Benchmark (MGIT): Better Locating Target in Complex Spatio-temporal and Causal Relationship
Authors: Shiyu Hu
Dailing Zhang
Meiqi Wu
Xiaokun Feng
Xuchen Li
Xin Zhao
Kaiqi Huang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Visual Object Tracking
Multi-modal Tracking
Long Video Understanding
Benchmark and Evaluation

Abstract: Tracking an arbitrary moving target in a video sequence is the foundation for high-level tasks like video understanding. Although existing visual-based trackers have demonstrated good tracking capabilities in short video sequences, they always perform poorly in complex environments, as represented by the recently proposed global instance tracking task, which consists of longer videos with more complicated narrative content. 
Recently, several works have introduced natural language into object tracking, desiring to address the limitations of relying only on a single visual modality. However, these selected videos are still short sequences with uncomplicated spatio-temporal and causal relationships, and the provided semantic descriptions are too simple to characterize video content.
To address these issues, we (1) first propose a new multi-modal global instance tracking benchmark named MGIT. It consists of 150 long video sequences with a total of 2.03 million frames, aiming to fully represent the complex spatio-temporal and causal relationships coupled in longer narrative content. 
(2) Each video sequence is annotated with three semantic grains (i.e., action, activity, and story) to model the progressive process of human cognition. We expect this multi-granular annotation strategy can provide a favorable environment for multi-modal object tracking research and long video understanding. 
(3) Besides, we execute comparative experiments on existing multi-modal object tracking benchmarks, which not only explore the impact of different annotation methods, but also validate that our annotation method is a feasible solution for coupling human understanding into semantic labels. 
(4) Additionally, we conduct detailed experimental analyses on MGIT, and hope the explored performance bottlenecks of existing algorithms can support further research in multi-modal object tracking. 
The proposed benchmark, experimental results, and toolkit will be released gradually on  http://videocube.aitestunion.com/.

A Multi-modal Global Instance Tracking Benchmark (MGIT): Better Locating Target in Complex Spatio-temporal and Causal Relationship
Shiyu Hu?.2 Dailing Zhang 1.2 Meiqi Wu
Xiaokun Feng!.2 1.2
Xuchen Li Xin Zhao Kaiqi Huang
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences
Sccooo of Computer Science and Technology. University of Chinese Academy of Sciences
Schooo of Computer Science, Beijing University of Posts and Telecommunications
Centter for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences (hushiyu2019, zhangdailing2023, fengxiaokun2022) @ia.ac.en, wumeiqi 8 @ mails. ac.s.ac.en, xuchenli @ bupt.edu.cn, (xzhao.kqhuang| @ @ia.ac.cn
Abstract
Tracking an arbitrary moving target in a video sequence is the foundation for high-level tasks like video understanding. Although existing visual-based trackers have demonstrated good tracking capabilities in short video sequences, they always perform poorly in complex environments, represented by the recently proposed global instance tracking task, which consists of longer videos with more compli- cated narrative content. Recently, several works have introduced natural language into object tracking, desiring to address the limitations of relying only single visual modality. However, these selected videos are still short sequences with uncomplicated spatio-temporal and causal relationships, and the provided semantic descriptions are too simple to characterize video content. To address these issues, we (1) first propose a new multi-modal global instance tracking benchmark named MGIT. consists 150 long video sequences with a total of 2.03 million frames, aiming to fully represent the complex spatio-temporal and causal relationships coupled longer narrative content. (2) Each video sequence is annotated with three semantic grains (i.e., action, activity, and story) to model the progressive process of human cognition. We expect this multi-granular annotation strategy can provide favorable environment for multi-modal object tracking research and long video understanding. (3) Besides, we execute comparative experiments on existing multi-modal object tracking benchmarks, which not only explore the im- pact of different annotation methods, but also validate that our annotation method is feasible solution for coupling human understanding into semantic labels. (4) Additionally, we conduct detailed experimental analyses on MGIT, and hope the ex- plored performance bottlenecks of existing algorithms can support further research in multi-modal object tracking. The proposed benchmark, experimental results, and toolkit will be released gradually on http:///ideocuuee.aatttttiioonoi
Introduction
Single object tracking (SOT) an important computer vision task that aims locate an arbitrary moving target in a video sequence, and can be regarded as the foundation for high-level tasks like video understanding. In the past decade, researchers have proposed numerous high-quality benchmarks 4
9 for the visual-based SOT task, and a series of trackers [T0]
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

OTB-Lang Liquor sequence: brown liquor bottle
Laso 1' airplann-1 sequence: white airplane landing on ground
TNL Arrow Video 2204 done sequence: second BnroW from left night
Action 1: A malc socret agont woating black suit w walks in the washroom.
Activity
Actlan
Activity I: male secret agent weening black suit walks in the washroom, and stands near man wearing a light grey suit. They fight, then the male secret agent wins, and lifts the insensible grey man washroom cubicle.
Actiun
Actton
Story
Activlts
Actan
Story: A male secret agent wearing black suit walks in the washroom, and stands near man wearing light grey suit. They fight, and the male secret agent wins. He then lifts the insensible grey-suil man
Ardan
washmom cubicle. washroom cubicle and chocks the insensible grey grey-suit mam. Suddealy, the grov-suit man wakes up, and they fight together again the washroom. Eventually, the male secret agent wins fight. A again lifts the insensible grey-suit suit man to the washroom cubicle. Finally, the male secret agent lefts the washroom after talking with the brown- suit woman.
male secret agent crouches
Activitt
Artee
male socref agent talks with woman wearing brown suit, he
Arten
Figure 1: Comparison of MGIT and other multi-modal object tracking benchmarks. (A-C) Examples of video content and semantic descriptions on OTB-Lang [1 LaSOT (2), and TNL2K [3] The green bounding box (BBox) indicates ground truth, while the red dashed BBox indicates other objects that satisfy the semantic description. These benchmarks have short sequences with simple narrative content. Besides, their semantic labels mainly describe the first frame, which may misguide algorithms. (D1-D3) An example of the to existing benchmarks, MGIT contains longer sequences with more complex narratives, and the multi-granule annotation provides more prosperous and flexible information to portray long videos.
annotation strategy used by MGIT. Compared
15 [6 have demonstrated good tracking capabilities in these environments, especially in short video sequences ranging from hundreds thousands of frames. However, researchers noticed that most trackers always perform poorly in longer videos with more complicated narrative content. Besides, only relying on a single visual modality also limits the application scenarios. Thus, several works have begun to offer additional semantic annotations for SOT task.
As the first multi-modal SOT benchmark, OTB-Lang 11 provides language description for the classic OTB (5) benchmark, hoping to provide more natural human-machine interaction method. The long-term tracking benchmark LaSOT 2 17 also supplies semantic annotation for each sequence, desiring to utilize linguistic features to improve the tracking performance. TNI.2K (3] wants to achieve more flexible and accurate tracking ability with more explicit information (e.g.. location information) in the semantic description. Although these multi-modal benchmarks have introduced semantic information into visual object tracking, they still face the following problems. (1) Short sequences with uncomplicated spatio-temporal and causal relationships: Existing works mainly focus on videos with hundreds to thousands of frames (the average sequence lengths of OTB-Lang LaSOT, and TNL2k are 590 frames, 2,502 frames, and 622 frames), while shorter video sequences are always insufficient to reflect complex narrative content. (2) Simple semantic descriptions: The quality of semantic information is critical to multi-modal trackers' performance, while incorrect or ambiguous semantic information may misguide algorithms tracking interference [8]. However, the semantic labels in existing works mainly describe the state in the first frame, but lack the portrayal

---

of the complete sequence. For example, the brown liquor bottle description of Figure 1 (A) cannot distinguish the object from the interference (another brown liquor bottle). In Figure I (B), white airplane landing on ground may also misdirect trackers to locate another airplane that has parked on the right ground. In Figure (C)) the second arrow from left to right only satisfies to represent the object state at the beginning of the sequence; as the object moves, the position constraint contained in the semantic information will become misleading. Consequently, a better way to construct a multi-modal benchmark is not to provide simple natural language description for short videos, but to design a scientific way to couple human understanding of long videos into semantic labels.
Therefore, we should first select suitable long videos with rich narrative relationships to compose a complex environment. VideoCube [19] is high-quality benchmark recently released for the global instance tracking (GIT) task (i.e., search an arbitrary user-specified instance in a video without any assumptions about motion consistency), which can be regarded as expanding the definition of traditional SOT task (i.e., tracking target in single-camera and single-scene) to success model the human visual tracking ability in a complex environment. Thus, we selected 150 representative long video sequences from VideoCube to form new multi-modal benchmark named MGIT. The proposed new benchmark is consistent with the distribution of the original VideoCube in all dimensions (e.g., length, scene categories, object classes, motion modes, spatio-temporal consistency, and difficulty). Besides, we carefully check the content of each sequence to ensure that the selected data contain as many different types of video narratives as possible. Figure (D1-D3) illustrates example in MGIT. Compared with other works, sequences in MGIT include more complex content (i.e., the spatial-temporal variation and causal relationship are more complicated).
Besides, we design multi-granular annotation strategy to provide scientific natural language information. On the one hand, existing research has indicated that complex narrative content can be perceived as several components and their relations, which is consistent with cognitive intuition [20]. On the other hand, the process of human comprehension and cognitive development is progressive as well 22 22]. Therefore, designing a hierarchical structure to represent the video content is a reasonable annotation method. As shown in Figure I (D1-D3), each sequence in MGIT is annotated with three semantic grains (i.e., action, activity, and story). We hope this method can provide a step- by-step "learning" environment for multi-modal trackers, in which they can first learn multi-modal information at a fine-grained level (action), then gradually develop to a morea comprehensive level (activity). and finally understand the complex video narrative at story level like humans.
Contributions. (1) We propose a new modti benchmark named MGIT. It consists of 150 long videos with total of 2.03 million frames, and the average length of single sequence is 5~ 22 times longer than existing multi-modal benchmarks. We hope this new benchmark fully represents the complex spatio-temporal and causal relationships coupled in longer narrative content (Section (.1] (2) We design a aulti-granular annotation strategy for providing scientific semantic information. Via this strategy. MGIT can provide favorable environment for multi-modh object tracking research and long video understanding (Section(3.2). (3) We execute comparative experiments on other benchmarks. Experimental results explore the impact of difffrent annotation methods, and validate that the proposed strategy is a feasible solution for coupling human understanding into semantic labels (Section 4.2). (4) We conduct detailed experimental analyses on MGIT. Results indicate that existing methods still have significant room for improvement in multi-modal tracking (Section 4.3) The proposed benchmark, experimental results, and toolkit will be released gradually on http://iddeouubee.aaietttouooooi
2 Related Work
Benchmarks with Visual Information. Standard SOT trackers are always initialized in the first frame by target's bounding box (BBox). then continuously locating it in the video sequence. Since 2013, many benchmarks represented by OTB 4 S and VOT 6 23 have been released, and these standardized datasets with scientific evaluation mechanisms promote the SOT research. With the development of deep learning techniques, these short-term and small-scale benchmarks have struggled to support data-driven trackers. Thus, several researchers have started to design larger-scale datasets like G0T-10k [9] and TrackingNet 8 while others have tried to collect data with longer videos and proposed long-term tracking benchmarks like OxUvA 24 and VOT-LT [25] 26 Recently, some researchers have noticed that short-term and long-term tracking tasks include a continuous motion assumption in their definitions, resulting in the experimental environments being restricted to
3

---

single-camert and single-scene. Therefore, they propose the global instance tracking task [19] with a new benchmark named VideoCube to track an arbitrary moving target in any type of video.
Benchmarks with Visual and Semantic Information. Unlike numerous visual benchmarks that have evolved over decade, multi-modal benchmarks combining visual and semantic information have only received attention lately. OTB-Lang [I] is the first multi-modal SOT benchmark, which provides additional natural language description for sequences in OTB 100 [5] benchmark. However, the limited dataset scale has prevented the multi-modal SOT task from receiving widespread attention.
After that, large-scale and long-term tracking benchmark LaSOT modal annotations. In the same year, researchers propose the TNL2k benchmark to achieve more flexible and accurate object tracking with natural language. These two benchmarks have provided a prosperity of data and have facilitated the generation of various mulli--odal trackers.
17 is released with multi-
As shown in Figure 2 existing works either fo- cus on visual modality, or concentrate on multi- modality but lack longer videos with complex content. Besides, Figure [ entific annotation strategy is also needed for pro- viding high-quality semantic information. These limitations prompt us to propose MGIT, hoping to construct a more complex and flexible envi- ronment for research.
More Modalities
18000
VISUAL
SEAALISERANTIC
16000
VideoCube
MGIT
Iindicates a more sci-
14000
Longer Video Sequences
12000
10000
Averoge
OxUNA
TradkingNet
OTB100
LaSOT
Algorithms with Bounding Box. Visual-based trackers always utilize the target's appearance and motion information to accomplish the track- ing process. including the correlation filter (CF) based trackers [27 28 Siamese neural network
GOT-1Ck
OmT-Lang
OTBS0
TrackingNet
GOT-1ok
TNLI
DTB100 MGIT
OMUhA
VideoCube
a
Lasor
Figure 2: Comparison between MGIT with other SOT benchmarks, including visual-based (e.g.,
(SNN) based trackers
29
30
31
32
3
34
OTB50 4 OTB100 [5], GOT-IOK[)), Track-
1 35 36 10]. the combination of CF and
ingNet 8 OxUvA [24]. and VideoCube (19) and multi-modal SOT benchmarks (e.g.. OTB-
SNN B7
38
12
13 and the transformer-base
[6]. Before 2021, SNN- based trackers are the prevalent methods. Re- cently, transformer-base
trackers 39
14
15
Lang M. LaSOT 22) and TNL2k (]. The bubble diameter is in proportion the total frames of a benchmark, and the vertical coordinate represents the average sequence length of each benchmark. Obviously, the proposed MGIT includes longer videos with multi-modal information.
trackers have demon- strated exemplary performance and gradually become the dominant architecture.
Algorithms with Bounding Box and Natural Language. Tracking a moving target with visual and semantic information new task for SOT research; thus, representative works are mainly released in recent two years. AdaSwitcher (3) is released with the TNL2k benchmark. which proposes switcher that utilizes natural language to alternate search mechanism (i.e., switch between the global search visual grounding module and the local visual tracking module). GTT (40) decomposes the visual language tracking task into three sub-tasks: tracking, grounding, and integration, and verifies the performance of each sub-module, SINT [1] proposes semantic information fusion module that can be utilized across various based trackers. VLT 42 introduces a modality mixer named ModaMixer with asymmetric ConvNet search, which aims to demonstrate pure ConvNet models can achieve comparable results to state-of-the-ar (SOTA) transformer-basse
algorithms. Besides, the proposed ModaMixer can further improve performance when directly applied to transformer-ban trackers. JointNLT [18] unifies visual grounding and tracking as a coherent task (i.e., locating referred objects based on visual- languag references). It employs the transformer- based architecture to model the relation between natural language and visual information.
3 Construction of MGIT
We propose a new strategy for generating scientific semantic information. On the one hand, we have carefully selected 150 longer video sequences to form MGIT (please refer to Section A.2 in the Appendix for more details), hoping this complex environment can promote visual tracking and video understanding research. On the other hand, we hope this multi-granular annotation strategy can provide a step-by-
benchmark named MGIT and design a multi-granular annotation

---

step "learning" environment for multi-mout modal trackers. Like humans can increase their comprehension by gradually increasing the learning difficulty, trackers can first learn fine-grained level (action). then to a more comprehensive level (activity), and finally accomplish a story level understanding of long video sequences. well-trained elite annotation team selected to execute this task instead of crowdsourcing, and the annotation quality is ensured through various efforts. The detailed workflow has been outlined in Section A.3.2 the Appendix.
3.1 Data Collection
Story: A pink cartoon pig wearing red clothes talks to her family members on the grassland. Today, the red- rel-cclthees pig and her family and the rod-clothss pig and walk castle, At entrance of the castl, they meet whita cartoon pig weaning gray armor. The red-clote cee-cclotes pig first talks with the gray- ammor pig, then they are invited to visit the castle, The red-clothes pig walks with her family into the castle and sits beside calue-clothes
to visit castle. They go the castle in
the back. They stop the vehicle nearby the foothills
the chair. A Afeer that, they have meal in the castle's living mom, and the red-clothes pink pig gets gift from stands the top
yellow-clothes pig of the tower.
meal. Finally, the clothes pig walks with her (amily members on the stainway,
Story: A black gorilla holding a lady in white crouches on gray bailding, and some airplanes attack them.. He then walks and climas to the top of the grey building. After that, be stands stop the grey building, hits an airplane, lights with gray soldier in the other aimlane, and fimally crouches the gray building.
Story: A blaca motorcycle is abecked by man with orange and white clothes the yard; then, the man rides this black motnrcycle in the yard, obstacle race, black motorcycle first bounces across obstacles in the playground, then bounces across obstaclee les in the streot, A fler that, it bounces across obslacles near the pool and obstacles in the stream. fter brief motorcycle bounces across obstacles in the playground, then across obstacles near the pool, and finally across obstacles in the stream.
Story: A small basketball hall is play by a boy with a grey b-bitt and black shorts, and then inflated by man with red s-iht and black pants in the skatepark.
fler that, the basketball by the hov, and then played by the man. A fter they practice, the besket kesketbal is holden by the boy from the skaloperk to ouldoors; then played the boy outdoors. Finally, The basketball then carridd away by boy.
played
black
Story: A brown cello is played by man with white shirt and black Story: A red cap is worn by man with a gray t-shit on the soccer count.
pants in the room.
Spoiall
T Tempoorl
Spatial Contimuity
temperaal
Sputko
Narrativity
Topics
V Virtuality
Object
Continuity
#trames dAction 4t cSooyty
High
B
High
Mostes&TV TV Shows
F
Gorila
6016
12
Middle
Sports
Meteacylle
9264
Cumpen
Narrotire
D
Middle
Regular Sports
F
Basketball
8336
Low
6793
Low
Documes/aries
F
Cap
23710
Canu Seistinnhhp
selection eitaron
Aspects relieed groined cnnotition
Figure 3: The representative data of MGIT. Here we illustrate six sequences with different aspects (e.g., narrativity, topics, virtuality, object classes, spatio-temporal continuity, and total frames).
MGIT follows the recently proposed large-scale benchmark VideoCube 19 to conduct the data collection. VideoCube refers to the film narrative (i.e., chain of causal relationship events occurring in space and time) and proposes the 6D principle for benchmark construction. In this work, we divide the 6D principle into two parts. Four dimensions (i.e., object class, spatial continuity, temporal continuity, and total frame), together with narrativity and topic, form the new sequence-level selection criterion. The other two dimensions (i.e., motion mode and scene category) will be refined during fine-gne tin-ggrined semantic annotation. Therefore, we first regard the original VideoCube as the candidate samples, then add the additional examination of narrativity and topic, and finally select 150 video sequences to form the MGIT. Particularly, the proportions of the train/val/test subsets are the same as the original VideoCube. Thus, sequences each subset are 105/15/30 in MGIT. Taking Figure an example, here we present several dimensions considered in the data collection process:
5

---

Topic and Narrativity, We have divided the main video topics into six categories, which are cartoons, movies & TV shows, outdoor sports, regular sports, performances. and documentaries. Among them, cartoons and movies & TV shows usually have high narrativity (i.e., the video content contains solid causal relationship. as shown Figure 3 A and B). Outdoor sports and regular sports contain rich patterns of motion, and these motions can be linked chronologically into story, but the narrativity is usually simple than in cartoons and movies (Figure C and D). Compared to other topics, performances and documentaries usually record one action with low narrativity (Figure and F). However, these examples are classifications for most cases; it is worth noting that some performances (e.g., sketches with explicitly narrative content stage) and some documentaries (e.g.. documentaries with causal teaching steps) also belong to high narrativity.
Spatial Continuity and Temporal Continuity. Temporal continuity means the video content developed according to the normal time flow (i.e., without fast-forwarding, fast-receding, or interpolation). Spatial continuity means the video content takes place in fixed space.
Virtuality. Virtuality refers that this video computer-generatee like cartoons or games. The same content in virtuality videos can be very different from videos sampled from the real world; thus, virtuality videos can present challenge for object tracking and long video understanding.
3.2 Natural Language Annotation
male secret agent wearing black suit stand witt
walk
fight with
lift
wearing light grey sult.
washroom
ction 1: male secret agent weering black walks the washroom.
A clion A male souros
Aclion 3: A male secrol.
Action 4 A male secret agent
agent wearing black stands wearing a light grey suit
agent wearing a black
wearing bl black lifts insensible
suit fights with
man wearing light greg suit
wearing light grey suit
washroom cubicle.
in the washrom.
in the washroo..
male secrel agent wear wearing black suit check
crouchh
light with
wearing light grey suit
washroom cubicle
washroom
Action St male secret agent wearing a black suit crouches in the Action 6: A male secret agent wearing black suit fights
washmom cubicle, and checks wearing light suit.
man weaning a light grey suit in the washmom.
made secret agent wearing black suit
talk with
lift
talk with
wearing brown suit
wearing
sui washroom
woman wearing brown suit
A ction 7: A male secret agent weering black suit talks with a woman wearing
A ction & A male secret agent weering black suit
Action A male secret agent wearing black suit talks with woman wearing brown suit in the washroom.
brown suitin in the washroom.
inseusilee man
weering a light grey suit to the washmom cubicle.
@Target otMotion AT AThird-party @Location
Figure 4: example action annotation. We label the target, motion pattern, third- saird-patty object, and scene for each action. The target to be tracked is determined in the first frame and does not change during the entire video sequence. A change any of the other three elements will end the current action and proceed into the following action.
In this work, we design multi-granular annotation strategy provide scientific natural language information. Video content is annotated by three grands (i.e., action, activity, and story, as shown in Tigure[[DI-D]) This hierarchical structure to represent the video content is motivated by existing works in computer vision 20 43 and human cognitive [2] 22 such as a recent method 43
decouples the video content into multiple granularities for the visual question-answerin task aiming to help algorithms better understand video information like humans.

---

Action. As shown in Figure 4 we use the following critical narrative elements to portray action: tracking target (who), motion (what) and third-pus -arty object (if present), location (where). and time interval (when). On the one hand, the above elements are necessary to portray narrative content. On the other hand, these elements are also essential grammatical components to form complete sentences. In particular, we use Stanford CoreNLP 45 widely used natural language processing toolkit, to check the semantic annotations of other multi-modal datasets. We find that more than half of these semantic descriptions are only annotated at the phrase level, lacking the necessary grammatical structure (the detailed statistic result has been shown in Section (...T of the Appendix). Thus, compared with existing works, MGIT can describe more detailed narrative content.
Activity. An action describes what happens in short period, while an activity can be seen as a collection of actions with clear causal relationships. A new activity is usually accompanied by scene switch or an explicit change of the third-party object. Compared with the former action, an action is preferred to be the beginning (i.e., reason) of a new event rather than an ending (i.e., result) of an old event, can be regarded as a starting point of an activity. As shown in Figure and Figure (D1-D3), the first four actions describe a complete causality (the target approaches the third party, they fight, and cause the third party to be insensible), while the 5th action starts a new event (start examining the unconscious third-party and conduct second fight when he wakes up). Therefore, the 4th and 5th actions can be divided into two different activities.
Documentaris
21
Regulor Sports
(0,5000]
(5006.10000)
(10000, 15000)
Carbec
20000)
25
Outdoor
20000]
Outdoe Sport
Topics
(b) #Frames
(0) Activities, Actions and Frames
Pink
Walks
Street
Black
shir
Woman
#F
Blue
Crey
Brown
High
Table
B
S
M
Man
preve
Girl
Green
Dog
Yello W. Wearing
Runs
Forest
BlActor
(d) Temporal Continuity (e) Spatial Continuity
(i) Narrativity
@
Word Cloud
Figure 5: Statistical analysis of key aspects in MGIT. (a-b) Distribution of topics and length of sequences. (c) Distribution of activities and actions. The bubble diameter is in proportion to the length a sequence, the vertical coordinate and the horizontal coordinate represent the total activities and actions of this sequence. (d-f) Distribution of temporal continuity, spatial continuity, and narrativity. (g) The word cloud semantic descriptions.
Story. Story is high-level description. To avoid boring narrativity, do stack the existing actions and activities, but use some words (e.g., first, then, after that, finally, etc.) to guide the content, making the temporal and causal more precise.
Based on the data collection process and the multi-granular annotation strategy, we construct MGIT with 2.03 million frames, and provide detailed annotation with 150 stories, 621 activities, and 982 actions. The semantic descriptions contain 77,652 words with 921 non-repetitive words, and more detailed analyses have been illustrated in Figure 5
4 Experimental Results
4.1 Datasets and Evaluation Methods
Datasets. We select OTB-Lang [1. TNL2k 33 LaSOT 2 and MGIT as experimental environments. Several variants of LaSOT are also concerned: (1) LaSOT 17 is complement of LaSOT 2 with 150 newly added video sequences. (2) Figure indicates that several semantic descriptions in LaSOT are ambiguous. Thus, 22 ambiguous and 20 unambiguous sequences are selected to
7

---

Action 1
Actionn
Action
Actioon
Acton
Action
Adion
Adtion
Adion!!
2403
Adtent
2403
3300
0053
Adliil Vol
1796
1875
2403
3999
6292
0033
9033
1247 1756
3875
2401 3999
6232
7157
7833
0033
Figure 6: Evaluation mechanisms of visual- based and multi- based trackers. (A) Traditional multi-modal tracking mechanism (i.e., only initialize a tracker with BBox and simple semantic information in the first frame). (B-D) Tracking with semantic information updates (i.e., initialize a tracker with BBox and semantic information in the first frame, then update the semantic information in each new interval). (E) Traditional one-pass evaluation (OPE) mechanism (i.e., only initialize a tracker with BBox in the first frame).
form the LaSOT aiming to better analyze tracking performance with different kinds of natural language information. (3) LaSOTwLe is a subset of LaSOT which formed by the 20 unambiguous sequences, and we have carefully checked all the semantic and visual information in this subset.
Evaluation Methods. As shown in Figure various mechanisms designed evaluate tracking precision (PRE) and success rate (SR). We use F; to represent the 1-th frame.
(1) Precision is calculated based on the center distance between the predicted BBox Pt and the ground truth BBox gt (i.e., d, c|cp Cgll2 where Cp and o represent center points). By calculating the proportion of frames where de I Oa and plotting curves at different thresholds, we can generate precision plot, PRE is common to use 6. = 20 as the criterion to rank trackers. (2) Furthermore, researchers [19] provide the normalized precision (N-PRE) to eliminate the effect of target size. When trackers have predicted center outside the ground-truth, an additional penalty term, represented by de", included to account for the shortest distance between the center point ep and the edge of the
ground-truth. The final result is then normalized to a range of to (i.e., N(d,) max((d; (iFF)))
where di de + de"). Similarly, the normalized precision plot is generated by plotting statistical outcomes derived from various 0g values. (3) Besides, frames with the intersection over union (IoU) ((p:.e)) = Pe 0g. 2 0g can be regarded as successful tracking, and the SR measures the percentage of successfully tracked frames. Drawing the results based on various 02 is the success plot. For more details on the evaluation metries, please refer to Section B.1 the Appendix.
il Us.
Table 1: Results on different mulli--ooaa benchmarks (based on mechanism A in Figure 6
Tracker
OTB-Lang
TNL22 3
PRE
LaSOT [2]
SR
LaSOTaa 1
LaSOTsa SR
LaSOTsic
MGIT
SR
SNLT 46
0.848
0.666
0.081
0.100
0.475
0.459
0.306
0.262
0.527
0.495
0.513
0.483
0.004
0.036
SCAR
0.898
0.739
0.556
0.497
0.677
0.630
0.633
0.659
0.124
0.177
VLT TT 42
0.931
0.764
0.583
0.539
0.714
0.670
0.549
0.465
0.707
0.660
0.721
0.662
0.324
0.474
JoinNNLT
0.85
0.653
0.598
0.552
0.640
0.607
0.477
0.398
0.624
0.583
0.707
0.433
0.603
4.2 Comparison with Other Multi-modal Benchmarks (Mechanism A)
We select several SOTA multi- modal trackers as baseline models and evaluate them on various benchmarks (as shown in Table I To fairly compare the tracking performance on MGIT and other datasets, we only allow trackers to use the semantic information of the first action in this experiment. Results show that: (1) most trackers perform worst on MGIT, which means it is more complex environment with more challenges. (2) By comparing the tracking results on LaSOT and LaSOTNLe, can find that most trackers perform worse on LaSOT Sub, showing that ambiguous
8

---

can perform better via longer descriptions than semantic information may introduce external interferences. Thus, we avoid this problem via the scientific annotation and check process for MGIT construction.
4.3 Experimental Results on MGIT
Tracking by NL& (Mechanism B-D). As shown in ( both visual information (BBox of the first frame) and semantic information (natural language description) can be used for multi-modal trackers. Specifically, different granularities have various lengths, while most trackers have maximum limit of the input semantic information. JointNLT [8]] sets 50 as maximum limit and truncates the excess information. This truncation occurs for both activity (C) and story (D). Similarly, the VLT [42] series limits the semantic length but can avoid truncation by adjusting the parameters. Thus, we set the semantic length to 80 for the activity and 200 for the story, with zero padding as necessary. From Table we can draw the following conclusions: (1) SNLT, VLT -CC_,, and VLT-TV perform well when using longer semantic information like activity and story. This indicates that the semantic information processing modules (BERT 47)) used in these trackers can effectively handle long text. Besides, their good performances in activity indicate that as an intermediate granularity, activity accomplishes a balance between the amount of information and the number of semantic description updates. (2) On the contrary, JointNLT performs well on action rather than levels with longer descriptions, suggesting that truncated semantic information leads to poorer performance. Therefore, to obtain better multi-modal information processing capabilities, algorithms should first ensure that long texts can be processed rather than truncated directly.
Table 2: Results of different trackers on MGIT.
By comparing results un- der mechanisms A and D, we can find that in this complex environment, well-designed trackers (i.e.,
Tracker
Architecture
Initialize
Mechanism
PRE
N-PRE
SR
SiamCAR 11
SNN
BBox
0.116
0.378
0.183
SlamRCNN
10
SNN
BBox
0.512
0.707
0.591
PrDiMP 1
SNN+CF
BBox
0.296
0.602
0.453
KeepTrack
13
SNN+CF
BBox
TransT 399
Transformer
BBox
E
0.447
0.670
0.539
trackers with suitable long
MixFormer 4
Transformer
BBox
0.526
0.775
0.629
input processing ability)
OSTrack 15
Tranffrmer
BBox
0.476
0.706
0.583
GRM 16
Transfomerr
BBox
0.500
0.718
0.597
Action (B)
0:004
0.226
0.036
SNLT 46
SNN
BLx
Activity (C)
0.004
0.234
0.038
only relaying a short de- scription (SNLT: 0.036 - 0.040, ILT_SCAR: 0.177 + 0.184, VLT TT: 0.474 + 0.480 in SR). The above experiments indicate two
Story (D)
0.005
0.230
0.040
Action (B)
0.354
0.167
VLT SCAR 12
SNN
NL&BBox
Activity (C)
0.124
0.382
0,180
Story (D)
Action (B)
0.318
0.602
0.468
VLT TT +
Transformer
BBxx
Activity (C)
0.325
0.627
0.485
key points: (1) Richer se- mantic information (mecha- nism D based on story) can improve the tracking perfor- mance than a simple sentence (mechanism A based on information for the first action), which can also verify the accuracy and necessity of the proposed multi granuiataatl semantic annotation strategy. (2) Only providing simple description for multi- modal trackers is unreasonable for MGIT. Thus, initializing the tracking process with longer and more specific sentences, or updating the semantic information periodically throughout the sequence, has been found to be more effective in accurately locating targets within complex scenes.
Story (D)
0.322
0.616
0.480
Action (B)
0.445
0.786
0.610
JointNIT 18
Transformer
Bo&
Activity (C)
0.441
0.780
0.605
Story (D)
0.433
0.773
0.600
Tracking by BBox Only (Mechanism E). We mainly evaluate SOTA visual-based trackers under mechanism E. As shown in Table 2 by comparing with other trackers. the transformers-basee trackers have emerged as the predominant approach and achieved SOTA performance. Besides, it is worth noting that visual-based trackers usually outperform multi-modal trackers. Although we hope that more modal information can improve the tracking performance, the current multi-modal approaches cannot better align different modalities, resulting in the multi-modal information not being fully utilized. In contrast, visual-based methods have been well developed over the past decades and can better use visual features to accomplish good tracking performance. This result (i.e., current multi- modal trackers are worse than visual-based trackers) can also be found in other works, highlighting the significant room for improvement multi-modal tracking. More detailed experimental results and analyses can be found in Section B.3 of the Appendix.

---

4.4 Visualization and Bad Case Analysis
bird-2 sequence: bird walking the among other birds
LaSO'T zebra- sequence: running dry grass with other zebras
MGII 012 sequence (mechanism A): A pink cartoon pig wearing red alothes talks to her fomily members on the grassand. GroundTruhh JoinNNL (CVPR23)
MGIT 006 sequence (mechanlsm A): A skateboard slid by man in black on the playground.
WVLT_TT TT (NeurIPS22)
VLT SCAR (Neur1P522)
SNLT (CVPR21)
Figure 7: Bad cases of representative multi-modal trackers on LaSOT 2 and MGIT. (A-B) Ambigu- semantic annotations on LaSOT lead trackers to locate similar objects. (C-D) The mechanism used in existing multi-modal SOT benchmarks is unable to adapt to complex scenes like MGIT.
We further analyze the bottlenecks of the multi-modal algorithms through the bad cases shown in Figure 7 The first two examples are selected from LaSOT (2), demonstrating that ambiguous semantic information may introduce noise, leading algorithms wrongly focus on similar objects -hhis emphasizes the importance of accurate semantic annotations. The latter two examples are chosen from MGIT, demonstrating that the experimental environment constructed by MGIT presents complex spatio-temporal and causal relationships, posing challenges to multi-moda algorithms. Specifically, the complexity of MGIT results in significant differences between the target appearance and background environment in the initial frame and subsequent states. Besides, MGIT is selected from the recently released VideoCube [99 benchmark, which has a higher image resolution, posing challenges for trackers to relocate the target after failure. Additionally, using only the first action information (mechanism A) is applied all other multi-modal SOT benchmarks. However, it is not applicable to visual object tracking in complex scenes like MGIT (Figure C-D)). (Figure[[-D)). Therefore, the proposed multi granularity annotation strategy offers a more reasonable solution. Multi-modal trackers who want to perform better on MGIT need more well well-designed semantic information processing module to accurately extract useful information described by semantic labels. Nevertheless, existing trackers have not made specialized designs for this aspect, which can be further improved.
5 Conclusions
Summary. Accuracy target tracking is the foundation for accomplishing high-level tasks like long video understanding, and introducing natural language into visual object tracking a possible way to increase tracking ability. Different from existing multi-modal benchmarks that mainly consisted of short sequences with simple or even ambiguous descriptions, we (1) propose new multi-modal benchmark named MGIT with 150 long video sequences, and (2) design multi- granular annotation strategy for generating scientific semantic information. On the one hand, MGIT is challenging and complex environment for visual tracking and video understanding research (i.e., trackers should process the spatio-temporas and causal relationships coupled with longer narrative content to accomplish better performance). On the other hand, the multi-granular annotation strategy models the human cognitive enhancement process, which may provide a step-by- step "learning" environment for generating human-like trackers. The experimental results demonstrate that MGIT is a more complex environment, and our proposed strategy is a feasible solution for coupling human understanding into semantic labels. Besides, existing trackers still have large room for development, like improving the capability for processing long text and aligning multi-moda information. Conclusionally, we hope this work can help researchers conduct further research in object tracking and video understanding.
Limitations. Some limitations here can be further enhanced in future work. First, we can expand MGIT with more types of videos to provide a more complicated environment for data-driven algo- rithms. Besides, we can design better comprehensive evaluation system to measure visual tracking and video understanding ability. Finally, we can add more types of tasks based on the benchmark, and try test algorithms for tasks like video caption and action recognition.
10

---

Acknowledgments and Disclosure of Funding
This work supported in part by the National Key R&D Program of China (No.20227D01 16403); the National Natural Science Foundation of China (No.61721004) the Strategic Priority Research Program of Chinese Academy of Sciences (No.XDA27000000)
References
[1] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. Tracking by natural language specification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6495-6503, 2017.
[2] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5369 5378, 2019.
[3] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13763-13773, 2021.
[4] Yi Wu, Jongwoo Lim, and Ming- Hsuan Yang. Online object tracking: A benchmark. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2411- 1-
2418,7 2013.
[5] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1834918884- 2015.
[6] Matej Kristan, Roman Pflugfelder, Ale Leonardis, Jiri Matas, Fatih Porikli, Luka Cehovin, Georg Nebehay, Gustavo Fernandez, Tomá Vojir, Adam Gatt, Ahmad Khajenezhad, Ahmed Salahledin, Ali Soltani-Farani, Ali Zarezade, Alfredo Petrosino, Anthony Milton, Behzad Bozorgtabar, Bo Li, Chee Seng Chan, Cherkeng Heng, Dale Ward, David Kearney, Dorothy Monekosso, Hakki Can Karaimer, Hamid R. Rabice, Jianke Zhu, Jin Gao, Jingjing Xiao, Junge Zhang, Junliang Xing, Kaiqi Huang, Karel Lebeda, Lijun Cao, Mario Edoardo Maresca, Mei Kuan Lim, Mohamed El Helw, Michael Felsberg, Paolo Remagnino, Richard Bowden, Roland Goecke, Rustam Stolkin, Samantha Yueying Lim, Sara Maher, Sebastien Poullot, Sebastien Wong, Shin Ichi Satoh, Weihua Chen, Weiming Hu, Xiaoqin Zhang, Yang Li, and Zhiheng Niu. The visual object tracking vot2013 challenge results. 2013 IEEE International Conference on Computer Vision Workshops, pages 08-111,23 2013.
[7] Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmark and simulator for uav tracking. In European conference on computer vision, pages 445-461. Springer, 2016.
[8] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Track- ingnet: A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pages 300-317, 2018.
[9] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1562-1577 2021.
[10] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-can: Visual tracking by re-detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6578-6588, 2020.
[11] Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, and Shengyong Chen. Siamcar: Siamese fully convolutional classification and regression for visual tracking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6269-6277, 2020.
[12] Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7183-7192, 2020.
11

---

[13] Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool. Learning target candidate association to keep track of what not to track. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13444-13454, 2021.
[14] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-1333 13608813188,.1 2022.
[15] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking; A one-stream framework. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 341-357. Springer, 2022.
[16] Shenyuan Gao, Chunluan Zhou, and Jun Zhang. Generalized relation modeling for transformer tracking, arXiv preprint arXiv:2303 2303::2300.1688000 2023.
[17] Heng Fan, Hexin Bai, Liting Lin, Fan Yang. Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juehuan Liu, Yong Xu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129(2):439-461, 2021.
[18] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23151-23166 June 2023.
[19] Shiyu Hu, Xin Zhao, Lianghua Huang, and Kaiqi Huang. Global Instance Tracking: Locating Target More Like Humans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):576-592 592, January 2023.
[20] Zhang Zhang, Tieniu Tan, and Kaiqi Huang. An extended grammar system for learning and recognizing complex visual events. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(2):240-255, 2011.
[21] Magali Bovet. Piager's Theory of Cognitive Development and Individual Differences, pages 269-279. Springer Berlin Heidelberg, Berlin, Heidelberg, 1976.
[22] Richard C Atkinson and Richard M Shiffrin. Human memory: A proposed system and its control processes. In Psychology of learning and motivation, volume 2, pages 89-195, Elsevier, 1968.
[23] Matej Kristan, Jiri Matas, Aleš Leonardis, Tomáš Vojif, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay, Fatih Porikli, and Luka Čehovin. A novel performance evaluation methodology for single-target trackers. IEEE transactions on pattern analysis and machine intelligence,
38(11):1137-2155 2016.
[24] Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Amoll WM Smeulders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A benchmark. In Proceedings of the European conference on computer vision (ECCV), pages 670-685, 201 8.
[25] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka "Cehovin Zaje, Tomas Vojir, Goutam Bhat, Alan Lukezic, Abdelrahman Eldesokey, et al. The sixth visual object tracking vo12018 challenge results. Proceedings of the European Conference Computer Vision (ECCV) Workshops, pages 0 0-0, 2018.
[26] Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, Roman Pflugfelder, Koni-Kristian Kamarainen, Luka CChovvin Zajc, Ondrej Drbohlav, Alan Lukezic, Amanda Berg, et al. The seventh visual object tracking vot2019 challenge results. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[27] João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. High- tracking with kernelized correlation filters. IEEE transactions on pattern analysis and machine intelligence, 37(3):583-566 2014.
12

---

[28] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg, Eco: Efficient convolution operators for tracking. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6638-6646, 2017.
[29] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. European conference on computer vision, pages 850-865. Springer, 2016.
[30] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu, High performance visual tracking with siamese region proposal network. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8971-8980, 2018.
[31] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. In Proceedings of the European conference on computer vision (ECCV), pages 101-117, 2018.
[32] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang. Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEÉ/CVF Conference Computer Vision and Pattern Recognition, pages 4282-42991 2019.
[33] Bin Yan, Haojie Zhao, Dong Wang. Huchuan Lu, and Xiaoyun Yang. 'skimming- perusal perusal'tracking: tracking: A framework for real-time and robust long-term tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[34] Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4591-4600, 2019.
[35] Yinda Xu, Zeyu Wang. Zuoxin Li, Ye Yuan, and Gang Yu. Siamfe++: Towards robust and accu- rate visual tracking with target estimation guidelines. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[36] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-aware anchor-free tracking. In European Conference on Computer Vision, pages 771-787. Springer, 2020.
[37] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Atom: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4660-4669, 2019.
[38] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEF/CVF international conference on computer vision, pages 6182-6191, 2019.
[39] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8122 8221,111,222 2021.
[40] Zhengyuan Yang, Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. Grounding- Buiciing-ntterationn IEEE Transactions on Circuits and Systems for Video Technology. 31(9):34333344 3443,,443,2222 2021.
[41] Ran Tao, Efstratios Gavves, and Arnold WM Smeulders. Siamese instance search for tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1420-1429) 2016.
[42] Mingzhe Guo, Zhipeng Zhang, Heng Fan, and Liping Jing. Divert more attention vision- language tracking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 4446 4460. Curran Associates, Inc., 2022.
[43] Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video as conditional graph hierarchy for multi-granular question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 2804-2812 2022.
13

---

[44] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[45] Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60, 2014.
[46] Qi Feng, Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5847-5856, 2021.
[47] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 180.0408/0 2018.
[48] Yu Huang. Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 10944-10956, 2021.
14

---

Appendix
A Dataset Information
A.1 Basic Information
In this work, we propose multi-modal global instance tracking benchmark named MGIT. It consists of 150 long video sequences with a total of 2.03 million frames, aiming to fully represent the complex spatio-temporal and causal relationships coupled in longer narrative content.
This work further expands our former work, which was accepted by IEEE TPAMI in 2022. We proposed global instance tracking task in the previous work and released an online evaluation platform (URL: use our proposed resources better and conduct more fair comparisons via our real-time evaluation server.
wtw.deoouub..atstttiio... com). We hope the online platform can help researchers
In the past year, our platform has evaluated 380 algorithms and received more than 287k IP visits from 130 countries (statistics by Jan 04, 2024). However, all submitted algorithms not show significant performance improvement on the GIT task, and their tracking performances are significantly degraded on the GIT task compared to other representative single object tracking benchmarks (i.e., short-time tracking and long-time tracking). This phenomenon shows the limitations of a single visual modality for long video understanding of complex narrative relationships. Thus, we conduct this work to introduce semantic information, which aims to help the algorithms better cope with the challenges posed by complex narrative relationships for target tracking and long video understanding.
Since the motivation for this work is closely inherited from our former work, and the existing online platform has received considerable attention worldwide, we select to release the MGIT benchmark via the same online platform. The proposed benchmark, experimental results, and toolkit will be released gradually on
(Figure(AT].
Our dataset has been uploaded to OneDrive and Baiduyun disk, and the online evaluation platform is maintained by dedicated staff, which will ensure the stability of the dataset.
We declare that we bear all responsibility in case of violation rights, etc., and confirm the data license. Our work is licensed under CC BY-NC-SA 4.0. Users are free to use the dataset for research purposes.
Bounloads
VideoCube
Download Dats
plcder
Figure Al: Our online platform and currently updated download links with related instructions.
A.2 Data Selection
The 150 sequences in MGIT are carefully selected from the original VideoCube [1 benchmark. In the video selecting part, thoroughly consider the consistency between the new dataset (MGIT)
15

---

and the original dataset (VideoCube) in various dimensions (6D principle), while also taking into account the consistency of difficulty.
The specific process is as follows. (1) We assess the similarity in the distribution of the selected MGIT dataset to the original VideoCube dataset across various dimensions, including object class, scene category, motion mode, and more, in accordance with the 6D principle. (2) Besides, it is essential for the selected dataset maintain similar level difficulty the original VideoCube dataset. Regarding the difficulty level, we select three state-oftthe-an trackers (MixFormer [14, KeepTrackk [3], and SiamRCNN [TO] with different architectures the basis for our selection criteria. The success scores (based on IoU) of these algorithms on the original 500 sequences are calculated and ranked to measure sequence difficulty. (3) By considering both the distribution across the 6D principle and the difficulty level, we carefully choose 150 representative sequences to construct the MGIT dataset, aiming to maintain consistency with the distribution of the original VideoCube dataset.
A.3 Semantic Annotation
A.3.1
Semantic Annotation Deficiencies of Existing Multi-modal SOT Benchmarks
To better show the semantic annotation deficiencies of existing multi-modal SOT benchmarks, we conduct statistical analyses on OTB-Lang LaSOT 2, LaSOTe [7 and TNL22 3 from two aspects:
1. Ambiguity of semantic labeling: A random sampling inspection is conducted to address the ambiguity of semantic annotation. Specifically, sequences are randomly selected from each dataset for inspection. To ensure random and fair selection, all sequences in each dataset are alphabetically sorted first, and samples are taken at equal intervals.
2. Completeness of grammatical structures: Semantic descriptions of high quality typically
necessitate complete grammatical structures. Hence, Stanford CoreNLP [45] is utilized analyze the semantic labels in all four datasets, and statistics those that adhered to the criteria of complete sentences.
Table A3: The statisties of semantic annotation quality four representative datasets.
Benchmark
OTB-LLng
LaSOT 2
LaSOTea
17
TNL22
3
Statistical Analysis Inspection Pass
30%
70%
60%
60%
(Non-ambiguous Semantic Descriptions)
Statistical Analysis 2: Complete Sentences Rate
(Checked by Stanford CoreNLP, Including Complete Grammatical Structures)
9%
63%
36%
20%
5, 5%
4%
1.1%
11,
457, 33%
308, 22%
25, 17%
40, 27%
45, 2%
401, 20%
210, 15%
29, 19%
1554, 78%
79, 80%
37%
425, 30%
NP
FRAG S SQ (c LaSOT-oxt
@ OTB-Lang
(b) LaSOT
(d) TNL2K
Figure A2: Statistical analysis about the completeness of grammatical structures, based on Stanford CoreNLP 45 toolkit. (NP: noun phrase; FRAG: fragment; S: simple declarative clause; SQ: inverted yes/no question, or main clause of wh-question. Only S and SQ satisfy the completeness of the grammatical structure.)
16

---

As shown in Table|A3 Figure A2 the current datasets exhibit deficiencies regarding ambiguity and completeness. However, our proposed MGIT benchmark considers these factors during construction and avoids the previously mentioned issues, thus possessing higher-quality semantic annotations.
A.3.2 Annotation Process
We chose an elite annotation team instead of crowdsourcing carry out this work and ensured quality through multiple efforts.
1.
Task Decomposition. We first decompose the task to ensure standardized workflow for execution. For instance, we begin by annotating the finest granularity (action), and subsequently continue with the annotations of activity and story. This approach ensures accuracy and consistency in the fundamental content throughout various levels of granularity. Given that action is the finest granularity and its annotation quality may affect activity and story, we refer to film narrative literature and English grammar materials to further decompose the description an action. This decomposition involves identifying the tracking target (who), the motion (what), the presence of a third-party object (if applicable), the location (where), and the time interval (when). By obtaining these specific details, annotators can attain a standardized and comprehensive description of the action.
2. Annotator Selection. Considering the difficulty of controlling annotation quality in crowd- sourcing, we chose highly cognitive graduate students with experience in dataset annotation to form an annotation team. Team members not only have experience in annotating vision- based datasets represented by VideoCube but also have experience in annotating image datasets in visual psychology- They have solid foundation in dataset construction in fields such as computer vision and cognitive psychology. All team members undergo standardized training before formal annotation to ensure their understanding of task characteristics and annotation rules. Additionally, the training session includes 10 video examples of different types, requiring annotators comprehend the annotation process and details.
3. Annotation Workflow. (1)
The formal annotation process begins, wherein annotation personnel is grouped based on video types, including cartoons, movies, TV shows, sports, performances, and documentaries. Any issues requiring discussion will be documented, followed by a comprehensive discussion among all personnel, and then the annotation process for that particular sequence will commence. (2) For instance. in the case of a sniper rifle as the target, which term should be employed: "gun" or "sniper rifle"? Through consulting relevant materials and conducting discussions, we have concluded that the fundamental principle of annotation is to incorporate human comprehension into semantic labels. As there is no second firearm in the sequence and the term "gun" encompasses "sniper rifle" semantically, the usage of "ggu"" aligns with commonplace terminology. Nevertheless, if second gun appears in the sequence, "sniper rifle" may be employed to underscore the target's distinctiveness.
(3) Furthermore, to enhance the standardization of annotations, we refer to WordNet to construct verb and noun lists. Initially, annotators will choose candidate terms from the current vocabulary lists to depict the essential elements of the scene, aiming to maintain consistency in the portrayal of actions across varying sequences to the greatest extent possible. If there are no appropriate terms found in the candidate list, annotators will employ new vocabulary to depict the elements and subsequently incorporate them into the candidate list, supplemented with relevant examples for future annotation reference.
4. Quality Review. After completing the annotation for all sequences, we will review the content to ensure its quality. Additionally, we utilize the Stanford CoreNLP [45), a natural language processing toolkit, to examine the semantic annotations and ensure the grammatical structure's 's completeness.
A.3.3 Annotation File
We propose multi-granular annotation strategy to generate the semantic description, and use ISON format to save the natural language annotation for each video sequence. Here we illustrate an example to show the ISON file structure for video sequence 00/ in the MGIT benchmark, as shown in Listing [ Due to the limited space, we only illustrate some representative information, while the remaining information with similar structure is indicated by ellipses. Please download and check the dataset for more detailed annotation about each video sequence.
17

---

{
"action": {'action_1", "start_frame" 0, "end_frame" 1246,
"length": 1247,
"object_class" "male secret agent", "appearance"; "black suit",
"action_1"; "walk", "prep_1": NaN, jord__object_"", NaN, "action_2": NaN, "prep_2": NaN, "3rd_object_2" NaN, "scene": "washroom", "description": "A male secret agent wearing black suit - walks in the washroom"),
"action_2": {..},
"action_3": (...}.
"action_4": 4": {...},
"action_5": (...).
"action_6" 6": (...),
21
"action_T" 7": (...),
"action_8" (...),
"action_g": f..,,,,,
" "activity": 1" {
estart_frames 0,
" lend_framee 2400,
"length": 2401,
28
"description": male secret agent wearing black suit walks in the washroom, and stands near man wearing 4 a light grey suit. They fight, then male secret agent wins, and lifts the insensible grey-suit man to the washroom cubicle."}, "activity_2": {...},
30
"activity_3"; {...,,>, "story": ("story_1": { "start_frame" 0,
32
9032,
34
"length": 9033,
"description": "A male secret agent wearing black suit walks in the washroom, and stands near a man wearing light grey suit. They fight, and the male secret agent wins. He then lifts the insensible grey-suit man the washroom cubicle. The male secret agent crouches in the washroom cubicle and checks the insensible grey-suit man. Suddenly, the grey-suit man wakes up, and they fight together again in the male secret agent wins the fight. After the male secret agent talks with woman wearing brown suit, he again lifts the insensible grey-suit man to the washroom cubicle.
washroom. Eventually,
Finally, the male secret agent lefts the washroom after talking with the brown-suit woman. "}}}
Listing 1: The JSON file about the semantic information of video sequence 007.
18

---

Pull Phll
Way
Fly
Dancelnteras Warch
Weaa
Stream
Basketball Court
Circus
Son
Fielde
Fielde
Castle
aik
See Mounter
City
Sky
Sta
DuR
buis
Fall
Crouch
otdoor
Scuet Bentingt Weweo werado
Showfield Woods
Road
Market
Chase
KeeiBounce
Help
e
Ridebriaa
bodminton Drivel
Outo
Room
Speak
Table
Earchh
Stage
Store
StorTennis &
Court
HoldR
Fook
Control
Carrary Pick
Blowarken
SitFightt
Cave Cave
Stadium
Corader
Gr
Floor
Alley Slope!
Poots
Savve
Vilke
(a) The doad action verbs
(b) The The doad wwm.gra
Figure A3: The word cloud of action verbs and scene categories on MGIT.
1. Action: For each action, save the following information in the JSON file:
(a) start_ frame:: The starting frame of the action. Note that the original VideoCube is labeled with the first frame starting from O. Therefore, in the JSON file, we use 0 to represent the first frame (Note that in the figures of the main paper, we the same format as the other datasets to show the starting point as for ease of understanding).
(b) mud_frama The ending frame of the action.
(c) length: Length of the time interval.
(d) object_class The object class of the tracking target.
(e) appearance: The appearance of the tracking target. We ensure that the description of the target's s appearance is unique in the entire sequence.
(f) action 1: The first action of the target. The word cloud of action verbs is illustrated in Figure A3 (a).
g prep_ The preposition of action (if present).
(h) 3rd li__objec_____ I: The interaction object of the first action (if present).
(i) action_2: The second action of the target (if present).
() prep_2: The preposition of action (if present).
(k) 3rd_ohe rrd_ojject22: The interaction object of the second action (if present.
()) scene: The scene category. The word cloud of action verbs is illustrated in Figure A3 (b).
(m) description: The natural language description.
2. Activity: For each activity, we save the following information in the JSON file:
(a) start start_frame: The starting frame of the activity.
(b) end_frame The ending frame of the activity.
(c) length:
Length of the time interval.
(d) description: The natural language description.
3. Story: For each story, save the following information in the JSON file:
(a) start_frame: The starting frame of the story.
(b) end_frame: The ending frame of the story.
(c) length: Length of the time interval.
(d) description: The natural language description.
A.4 Dataset Structure
The MGIT benchmark includes 150 long video sequences (344G) with detailed annotations. We add semantic information based on multi-granularity annotation strategy while retaining the detailed annotation information of the original VideoCube dataset, aiming to help multi-modal methods better understand the narrative content of long videos.
The dataset download and file organization process is as follows:
Download three subsets (rrain/valless) and the info data. Please click on the hyperlink to visit our dataset (choose the link that works best for you).
(a) Train Data (229G, 105 Sequences):
19

---

val
005/
frame_008
000000.jpg'
--- 016891.jpp
029/
362/
Listing The dataset structure of vol subset.
-- MGIT/
2
data/
train/
002/
6
480/
val/
005/
362/
test/
001/
498/
train_list.te
val_list.txx
fest_list.oo
attribute/
absent/
color_coostancy_ttra
description/
shotcut/
Listing 3: The dataset structure of the MGIT benchmark.
OneDrive
11.
Baiduyu Disk (The extraction code is cube.)
(b) Validation Data (37G, 15 Sequences):
OneDrive
11.
Baiduyu Disk (The extraction code is cube.)
(c) Test Data (78G, 30 Sequences):
OneDrive
ii.
Baiduyu Disk (The extraction code is cube.)
(d) Info Data (89.16M, 15 Attributes):
OneDrive
ii.
Baiduyu Disk
(The extraction code is cube.)
2. Check the number of files in each subset and run the unzipping script. To facilitate trans- mission and downloading, the very long video sequences in the dataset are divided into smaller segments during the packaging process. Each segment is compressed and kept under
20

---

4GB. For instance. in the train set, the sequence 013 is divided into three compressed files:
frame 013 split.z01, frame 013 ____sspi.. and frame 013 sps_ttiitzipp Before unzipping:
(a) The train subset should include 129 files (128 data files and an unzip_trame train bash).
(b) The val subset should include 22 files (21 data files and an unzip_val bash).
(c) The test subset should include files (40 data files and an unzip_test bash).
3. Run the unzipping script in each subset folder, and delete the script after decompression.
4. Taking val subset of full version example, the folder structure is listed Listing 2
5. Unzip attribute.zij in info data. Attention that we only provide properties files for train and val subsets. For ground-truti truth files in the test subset, we only offer small number of annotations for restart frames to support the essential funetion of the R-OPE mechanism (For detailed information about the R-OPE mechanism, please refer to the TPAMI paper [19] about GIT task and the VideoCube benchmark. Note that we only use the OPE mechanism for MGIT evaluation process, while the R-OPE mechanism is supported for visual-based trackers.). The annotations of other frames in the test subset have been set as zero. Please upload the final results the server http:///ieeocubbaaattttt..ccccc
for evaluation.
6. Rename and organize folders as Listing 3
Note that the semantic information (saved as JSON file) for the MGIT benchmark is saved in the description folder.
B Experiment Information
B.1 Evaluation Metrics
Assume an experiment dataset E (e.g., MGIT) comprises |E] sequences, with |-| representing the cardinality. In the sequence L, use F, represent the 1-th frame. We assume that pt denotes the predicted position by the tracker T, and refers to the ground-truth position. Notably, a frame does contain the target (i.e., full-occlusion
oft-of-view). it is considered an empty set (1.c.. = σ) and thereby excluded from the evaluation process. The precision score and success score of frame F; are calculated through the following formulas:
d, 1|cp - coll2:
Pe ng
(1)
8; ((pm 91)
Pc Une
where de represents the distance between the center points cp and cg. while 22(-) denotes the intersec- tion over union.
Recently, the normalized precision score (19) is proposed to eliminate the impact of the target size and frame resolution. In the case where trackers have a predicted center outside the ground-truth, an additional penalty item di" is included, representing the shortest distance between the center point Cp and the edge of the ground-truth. If the center point of a tracker falls within the ground -ruth, the
center distance di is equal to the original precision de, resulting in d," = O:
di
(dd)) max (di i i F.H) d, d, + di".
(2)
The precision P(E), normalized precision N(F.), and success S(F) of environment E. are defined as follows:
21

---

18T
P(E) =
{t: di s datl.
E
12
{ t::N (d, s 561
N(E)
L
1 E
S(F)
L [t 8t 2031
The precision plot is generated by calculating the proportion of frames with a distance de less than equal to 0g and plotting the statistical results across different 0₂ values. In most cases, existing benchmarks use 0 = 20 as standard threshold to rank trackers.
The normalized precision plot is generated similarly by plotting statistical results obtained from varying Oa values within the range of [0,1]. However, directly assigning specific 02 value to rank trackers can introduce subjective biases. Therefore, the ranking of trackers is based on the proportion of frames in which the predicted center cp successfully falls within the ground-truth rectangle gr.
The success plot is generated by plotting the results obtained from different overlap thresholds, 0,. on curve. In this plot, the mAO (mean average overlap) is commonly utilized to rank trackers.
B.2 Baseline Information
All experiments are performed on a server with 4 NVIDIA TITAN RTX GPUs and a 64 Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz. Detailed information about the baselines are illustrated in Table A4
we use the parameters provided by the original authors.
Table A4: Table:
The model architectures and URLs of open-sourced algorithms used in this work.
Tracker
Architecture
Initializa
URL
SiamCAR CAR [1]
SNN
BBox
https://gtthhh.com//
SiamRCNN [10
SNN
BBox
https://github.com/
R-CN
PrDiMP 12
SNN
BBox
KeepTTrack 13
SNN
BBox
TransT 39
Transformer
BBox
hupp://gituub.com//ceennnn Trannss
MixFormer 14
Transformer
BBox
https//ggttuh.ccm/M/CCCA/UUUUU
OSTrack 16
Transformer
BBox
htpps//gtthuu..ommm/oooooo000115
GRM [16]
Transformer
BBox
https:/ggttauu.comm/.itttPPPfffo
SNLT [46
SNN
NL&BBox
https://uttuuuccomm///////g/////
42
SNN
NI &BB
VLT TT 42
Transformer
NL&BBox
httpss//gutuub..om//uuuuaDDDDiet
JointNLT
18
Transformer
NL&BBox
https//gttttb..oomm//roooss////t
Note: SNN-Siamese Neural Network. NL-Natural Language. BBox- BBox-Bounding Box.
B.3 More Detailed Experimental Results MGIT
82
64) Precisioa plots MGIT
@6) Normallzed precision plats MGIT
6d) Succes plats on MG (bused on IoU)
Figure A4: The precision plot (a), normalized precision plot (b), and the success plot (c) on MGIT.
22

---

Figure A4 illustrates the precision plot (a), normalized precision plot (b), and the success plot (c). The performance of trackers indicates that the multi-modal trackers still exhibits a certain gap when compared to visual-based trackers.
Specially, the multi-modal tracker SNLT [46 performs poorly in MGIT. The possible reasons behind the poor results of SNLT are as follows: (1) SNLT is based on the local search, which exhibits a performance gap to the global search trackers. Experiments reveal that local search trackers may encounter a more severe tracking drift problem in the MGIT task (this method tracks by cutting out the search area from the original image, while the high image resolution in MGIT will challenge it). Besides, SNLT's 's weaker tracking ability is more prone to failure. The errors generated as result will further misguide, thus creating a negative loop. For example, when it loses the target or drifts towards a similar object, it will persist in tracking failure until the target reemerges within its search area with more recognizable visual information. (2) In addition, there are some issues within SNLT's open-source code, and although we fix them during our reproduction and evaluation, SNLT has some limitations in terms of project completeness compared to other better- maintained multi-modal open-sourced trackers, which may also contribute to its poor performance.
Besides, some other challenges may also influence the tracking performance. Here, we discuss the challenges trackers face from two perspectives: by comparing different multi-modal information provisioning mechanisms, and from limitations between single-modal and multi-modal approaches.
1. Comparison of different multi-modal evaluation mechanisms: (1) First, compared to mechanism A, which only provides semantic information for the first action, multi-modal methods do not show a significant performance decrease in mechanisms B, C, and D. Except JointNILT [18], the success rate scores of other multi-modal trackers in mechanism D are superior to mechanism A. (2) Furthermore, most algorithms performed the worst in mechanism A because they only obtained semantic descriptions for the first action, and this semantic information is not updated in the subsequent process, which may introduce noise to the tracker the sequence progresses. (3) Mechanisms B and C both regularly update the semantic information during tracking. However, C has a moderate frequency of semantic information updates, and each update provides a moderate length of semantic information, which may better leverage the capabilities of trackers. (4) Mechanism D, similar to mechanism A, only provides semantic information in the initial frame but offers story information that can cover the entire video. However, current modii-modal trackers lack well-designed semantic understanding modules for handling long texts, making challenging to align semantic information with visual information
2. Comparison between single- modal and multi-modal approaches: Semantic modalities can provide information beyond superficial features such as appearance and location com- pared to pure visual trackers. However, achieving better correlation and fusion between modalities still needs to be solved. Previous research on other multi-modal tasks has experi- mentally and theoretically demonstrated that multi-modaa approaches information, resulting in improved algorithms [48]. However, in the field of SOT research, the performance of multi-modal algorithms still lags behind that of magle-modal algorithms. The main reason may be related to the need for more high-quality benchmarks existing multi-modal benchmarks have significant limitations in the completeness of semantic infor- mation and video complexity, making it challenging to provide a favorable experimental environment for multi-modal trackers. Additionally, these multi-modal benchmarks do not adopt a multi-granula annotation strategy, resulting in an evaluation system that only involves mechanism A. As a result, they cannot thoroughly explore the current methods' bottlenecks like our work.
introduce more
Therefore, the emergence of MGIT can provide high-quality experimental environment for research, and help the algorithms quickly identify bottleneck issues under various evaluation mechanisms, thereby accelerating the development of efficient multi mooti trackers.
Furthermore, the multi-modal trackers demonstrate superior performance in terms of the normalized precision plot (Figure A4 (b)). We attribute this to the integration of the semantic modality, which enables the multi-mode modal tracker to effectively recognize the target's position. This observation aligns with intuition, but there is room for further improvement in localization accuracy.
23

---

B.4 A Comparison Experiment about Using Various Granularities
Action
Actionn
Action
Actioon
Action
Action
Action
Action
Action
1207
1796
1875
2401
3909
6392 7157
7613
9033
1247
3796
1875
2403
3999
7157
7613
Figure A5: An illustration of mechanism B, mechanism B, and their combination.
Table A5: Results of mechanism B, mechanism B, and their combination.
Tracker
Architecture
Initialize
Mechanism
PRE
N-PRE
SR
Action (B)
0.004
0.226
0.036
SNLT 46
SNN
Blx
Story (D)
0.005
0.230
0.040
Combination (B+D)
0.004
0.229
0.037
Action (B)
0.116
0.354
0.167
VLT_SCAR 42
SNN
&lx
Story (D)
0.127
0.403
0.184
Combination (B+D)
0.107
0.367
0.165
Action (B)
0.318
0,602
0.468
VLT 42
Transformer
NL&
Story (D)
0.322
0.616
0.480
Combination (B+D)
0.327
0.612
0.477
Action (B)
0.445
0.786
0.610
JointNI.T
18
Transformer
NI&
Story (D)
0.433
0.773
0.600
Combination (B+D)
0.443
0.783
0.607
Considering that integrating information from different granularities may further benefit the algo- rithms, we here take mechanisms and D as an example to explore whether multiple granularities of information are more effective for algorithms. As shown in Figure A5 we combine mechanisms and D using the following strategy: taking mechanism as the main component, we replace its semantic information from the first frame with story information (mechanism D). The experimental results are shown in Table A5
Since most current multi-modal trackers perform target feature matching frame by frame, when the first frame receives story information, the semantic information will be replaced by new action information after reaching the next action. Therefore, the effective range of story information only covers the initial action. As result, for most algorithms, the score difference between combination (B+D) and the original mechanism B is insignificant.
worth noting that this experiment only provides a simple and direct approach evaluating tracking performance with the combination of multi-granularity information. The direct reason for insignificant improvement lies the limitations of existing trackers (lacking a well-designed semantic processing module and the poor multi-modal alignment capability). If future multi-modal tracking algorithms can design a stronger semantic information processing module to comprehensively represent the video content (i.e., hierarchically constructing video content based graph 43)), perhaps a more powerful tracking capability can obtained than simply using single granularity of information.
24


Reviews:
[{"review_id": "dUlMLn09TV", "reviewer": "Reviewer_dttb", "summary": "", "questions": "", "limitations": "Overall, there are some technical details unclear in current version as described above.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) This paper provides a complex dataset with long-sequences video segments, which is clearly a more challenging task.\n(2) This paper proposes a multi-granular annotation strategy, which can be divided into three parts, i.e., action, activity, and story. This strategy aims at coupling human understanding into semantic labels.\n(3) This dataset is challenging for previous models, which perform poorly on this dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "forgclMgW0", "reviewer": "Reviewer_uBXK", "summary": "", "questions": "", "limitations": "1. The author did not specify who is responsible for annotating natural language and dividing motion, third-party, and location in actions. If it is done by a crowdsourcing team, has the author employed any means to ensure the accuracy of semantic descriptions and avoid semantic ambiguity caused by improper division?\n2. If the author could design other meters to evaluate these three mechanisms, it would make the construction of MGIT more convincing.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper proposed a challenging multi-modal global instance tracking benchmark.\n2. The multi-granular annotation strategy is interesting and can provide more dense semantic descriptions for multi-modal GIT task. Such a strategy can provide a step-by-step learning power like human being for multi-modal trackers. \n3. Previous language-based tracking tasks often suffered from problems such as semantic ambiguity or incomplete description of the overall state of the sequence objects. However, the introduction of MGIT has brought about new paradigms and trends in addressing long-term multi-modal tracking, which is highly encouraging. \n4. This article is written fluently and smoothly. I can read it very well. The exquisite illustrations help me better understand the processes of data collection, natural language annotation, experiments, etc.\n5. In different multimodal benchmark test results, it is shown that MGIT is more challenging, and a scientific annotation strategy can avoid the interference of ambiguous information. Meanwhile, comparisons are also made with newer types of transform trackers.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "frbm1g1f0M", "reviewer": "Reviewer_ZBXv", "summary": "", "questions": "", "limitations": "The authors address the limitations of existing works by introducing a new benchmark for multi-modal tracking. They are upfront about the limitations of their work. However, the experiment in Table 2 is very confusing. The results of SNLT are too worse. Natural language knowledge should be auxiliary information that can improve the performance of the tracker. However, the results of the SNLT show that natural language knowledge completely impairs the effect of tracking, and the accuracy is extremely low. The results require further explanation.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThis paper contributes a new benchmark for multi-modal tracking (MGIT). It consists of 150 long video sequences with a total of 2.03 million frames. Each video sequence is annotated with three semantic grains (i.e., action, activity, and story). \n\n2.\tThe authors execute comparative experiments to explore the impact of different annotation methods and validate that the proposed annotation method is a feasible solution.\n\n3.\tDetailed experimental analyses are conducted on MGIT, supporting further research in multi-modal object tracking.\n\n4.\tThe motivation for dataset construction is strong and well-presented. The logic and structure of the paper are clear, and the quality of this benchmark is further enhanced by sufficient analysis. This work can help researchers to conduct further research in object tracking and video understanding.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "U81L2cljjP", "reviewer": "Reviewer_u9AZ", "summary": "", "questions": "", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well-structured, and the diagrams effectively demonstrate the multi-granular annotations and data statistics of the dataset. Furthermore, the detailed and well-crafted tables underscore the value of the proposed dataset.\n\n2. The paper commendably proposes a challenging multi-modal global instance tracking benchmark with multi-granular annotation.\n\n3. Previous language-based tracking tasks have often been constrained by the precision of the description statement of the object in the first frame, as well as inaccuracies in the language description due to significant state changes of the object in long videos. MGIT addresses these issues by introducing multi-granular annotations, which break down the problem and provide a more accurate language description.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors present MGIT, a multi-modal benchmark for long video understanding. It comprises 150 sequences with over 2 million frames annotated with multi-granular semantics - action, activity, and story. This mimics human cognition and aims to address issues with previous benchmarks and advance multi-modal tracking and video understanding. Detailed experiments on current methods demonstrate the challenging nature of this benchmark and validate the usefulness of this dataset to the field. 

Pros: 
- High-quality, challenging multimodal video benchmark to advance tracking and video understanding consisting of much longer videos than previous benchmarks.
- Proposes multi-granular annotation approach (action, activity and story) addressing issues with previous language-based tracking benchmarks.
- Provides extensive evaluation of existing SOTA tracking models demonstrating large room for improvements and stimulating new research. 
- Well written and structured paper with helpful illustrations and easy to parse tables. Detailed and helpful appendix incorporating feedback by the reviewers. 

Cons: 
- (Minor) As mentioned in the paper’s Limitations section adding additional video categories could improve the usefulness of the dataset even further. 

Unanimous support by the reviewers in favor of accept with two good or higher and two borderline (one rating was increased after authors addressed concerns). AC agrees with the reviewers and recommends Accept.
Decision: Accept (Poster)