SlowGuess/NeurIPS_2023_OCR huggingface dataset:
"action": {"action_1": { "start_frame": 0, "end_frame": 1246, "length": 1247, "object_class": "male secret agent", "appearance": "black suit", "action_1": "walk", "prep_1": NaN, "3rd_object_1": NaN, "action_2": NaN, "prep_2": NaN, "3rd_object_2": NaN, "scene": "washroom", "description": "A male secret agent wearing a black suit  $\rightarrow$  walks in the washroom"}, "action_2": {...}, "action_3": {...}, "action_4": {...}, "action_5": {...}, "action_6": {...}, "action_7": {...}, "action_8": {...}, "action_9": {...}, }, "activity": {"activity_1": { "start_frame": 0, "end_frame": 2400, "length": 2401, "description": "A male secret agent wearing a black suit  $\leftrightarrow$  walks in the washroom, and stands near a man wearing  $\leftrightarrow$  a light grey suit. They fight, then the male secret  $\leftrightarrow$  agent wins, and lifts the insensible grey-suit man  $\leftrightarrow$  to the washroom cubicle.", "activity_2": {...}, "activity_3": {...}, }, "story": {"story_1": { "start_frame": 0, "end_frame": 9032, "length": 9033, "description": "A male secret agent wearing a black suit  $\leftrightarrow$  walks in the washroom, and stands near a man wearing  $\leftrightarrow$  a light grey suit. They fight, and the male secret  $\leftrightarrow$  agent wins. He then lifts the insensible grey-suit  $\leftrightarrow$  man to the washroom cubicle. The male secret agent  $\leftrightarrow$  crouches in the washroom cubicle and checks the  $\leftrightarrow$  insensible grey-suit man. Suddenly, the grey-suit  $\leftrightarrow$  man wakes up, and they fight together again in the  $\leftrightarrow$  washroom. Eventually, the male secret agent wins the  $\leftrightarrow$  fight. After the male secret agent talks with a  $\leftrightarrow$  woman wearing a brown suit, he again lifts the  $\leftrightarrow$  insensible grey-suit man to the washroom cubicle.  $\leftrightarrow$  Finally, the male secret agent lefts the washroom  $\leftrightarrow$  after talking with the brown-suit woman."}}
Listing 1: The JSON file about the semantic information of video sequence 001.
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32   
33   
34   
35
(a) The word cloud of action verbs
(b) The word cloud of scene categories
Figure A3: The word cloud of action verbs and scene categories on MGIT.
1. Action: For each action, we save the following information in the JSON file:
(a) start_frame: The starting frame of the action. Note that the original VideoCube is labeled with the first frame starting from 0. Therefore, in the JSON file, we use 0 to represent the first frame (Note that in the figures of the main paper, we use the same format as the other datasets to show the starting point as 1 for ease of understanding).
(b) end_frame: The ending frame of the action.
(c) length: Length of the time interval.
(d) object_class: The object class of the tracking target.
(e) appearance: The appearance of the tracking target. We ensure that the description of the target's appearance is unique in the entire sequence.
(f) action_1: The first action of the target. The word cloud of action verbs is illustrated in Figure A3 (a).
(g) prep_1: The preposition of action (if present).
(h) 3rd_object_1: The interaction object of the first action (if present).
(i) action_2: The second action of the target (if present).
(j) prep_2: The preposition of action (if present).
(k) 3rd_object_2: The interaction object of the second action (if present.
(1) scene: The scene category. The word cloud of action verbs is illustrated in Figure A3 (b).
(m) description: The natural language description.
2. Activity: For each activity, we save the following information in the JSON file:
(a) start_frame: The starting frame of the activity.
(b) end_frame: The ending frame of the activity.
(c) length: Length of the time interval.
(d) description: The natural language description.
3. Story: For each story, we save the following information in the JSON file:
(a) start_frame: The starting frame of the story.
(b) end_frame: The ending frame of the story.
(c) length: Length of the time interval.
(d) description: The natural language description.
A.4 Dataset Structure
The MGIT benchmark includes 150 long video sequences (344G) with detailed annotations. We add semantic information based on a multi-granularity annotation strategy while retaining the detailed annotation information of the original VideoCube dataset, aiming to help multi-modal methods better understand the narrative content of long videos.
The dataset download and file organization process is as follows:
1. Download three subsets (train/val/test) and the info data. Please click on the hyperlink to visit our dataset (choose the link that works best for you).
(a) Train Data (229G, 105 Sequences):
Listing 2: The dataset structure of val subset.
-- val/  
| -- 005/  
| | -- frame_005/  
| | | -- 000000.jpg/  
| | | ...  
| | | -- 016891.jpg/  
| | -- 029/  
| | ...  
| | -- 362/
Listing 3: The dataset structure of the MGIT benchmark.
-- MGIT/  
| -- data/  
| | -- train/  
| | | -- 002/  
| | | | ...  
| | | -- 480/  
| | | -- val/  
| | | | -- 005/  
| | | | ...  
| | | | -- 362/  
| | | -- test/  
| | | | -- 001/  
| | | | ...  
| | | | -- 498/  
| | | -- train_list.txt  
| | | -- val_list.txt  
| | | -- test_list.txt  
| | -- attribute/  
| | | -- absent/  
| | | -- color_constancytran/  
| | | ...  
| | | -- description/  
| | | ...  
| | | -- shortcut/
i. OneDrive
ii. Baiduu Disk (The extraction code is cube.)
(b) Validation Data (37G, 15 Sequences):
(c) Test Data (78G, 30 Sequences):
(d) Info Data (89.16M, 15 Attributes):
i. OneDrive
ii. Braiduyu Disk (The extraction code is cube.)
2. Check the number of files in each subset and run the unzipping script. To facilitate transmission and downloading, the very long video sequences in the dataset are divided into smaller segments during the packaging process. Each segment is compressed and kept under
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24
4GB. For instance, in the train set, the sequence 013 is divided into three compressed files: frame_013_split.z01, frame_013_split.z02, and frame_013_split.zip. Before unzipping:
(a) The train subset should include 129 files (128 data files and an unzip_train bash).
(b) The val subset should include 22 files (21 data files and an unzip_val bash).
(c) The test subset should include 41 files (40 data files and an unzip_test bash).
3. Run the unzipping script in each subset folder, and delete the script after decompression.
4. Taking val subset of full version as an example, the folder structure is listed as Listing 2.
5. Unzip attribute.zip in info data. Attention that we only provide properties files for train and val subsets. For ground-truth files in the test subset, we only offer a small number of annotations for restart frames to support the essential function of the R-OPE mechanism (For detailed information about the R-OPE mechanism, please refer to the TPAMI paper [19] about GIT task and the VideoCube benchmark. Note that we only use the OPE mechanism for MGIT evaluation process, while the R-OPE mechanism is supported for visual-based trackers.). The annotations of other frames in the test subset have been set as zero. Please upload the final results to the server (http://videocube.aitestunion.com/) for evaluation.
6. Rename and organize folders as Listing 3. Note that the semantic information (saved as JSON file) for the MGit benchmark is saved in the description folder.
B Experiment Information
B.1 Evaluation Metrics
Assume an experiment dataset  $E$  (e.g., MGIT) comprises  $|E|$  sequences, with  $|\cdot|$  representing the cardinality. In the sequence  $L$ , we use  $F_t$  to represent the  $t$ -th frame. We assume that  $p_t$  denotes the predicted position by the tracker  $T$ , and  $g_t$  refers to the ground-truth position. Notably, if a frame does not contain the target (i.e., full-occlusion or out-of-view), it is considered an empty set (i.e.,  $g_t = \phi$ ) and thereby excluded from the evaluation process. The precision score and success score of frame  $F_t$  are calculated through the following formulas:
$$
d _ {t} = \| c _ {p} - c _ {g} \| _ {2},
$$
$$
s _ {t} = \Omega \left(p _ {t}, g _ {t}\right) = \frac {p _ {t} \bigcap g _ {t}}{p _ {t} \bigcup g _ {t}}, \tag {1}
$$
where  $d_{t}$  represents the distance between the center points  $c_{p}$  and  $c_{g}$ , while  $\Omega(\cdot)$  denotes the intersection over union.
Recently, the normalized precision score ([19]) is proposed to eliminate the impact of the target size and frame resolution. In the case where trackers have a predicted center outside the ground-truth, an additional penalty item  $d_t^p$  is included, representing the shortest distance between the center point  $c_p$  and the edge of the ground-truth. If the center point of a tracker falls within the ground-truth, the center distance  $d_t'$  is equal to the original precision  $d_t$ , resulting in  $d_t^p = 0$ :
$$
N \left(d _ {t}\right) = \frac {d _ {t} ^ {\prime}}{\operatorname* {m a x} \left(\left\{d _ {i} ^ {\prime} \mid i \in F _ {t} \right\}\right)}, \tag {2}
$$
$$
d _ {t} ^ {\prime} = d _ {t} + d _ {t} ^ {p}.
$$
The precision  $\mathrm{P}(E)$ , normalized precision  $\mathrm{N}(E)$ , and success  $\mathrm{S}(E)$  of environment  $E$  are defined as follows:
$$
\mathrm {P} (E) = \frac {1}{| E |} \sum_ {l = 1} ^ {| E |} \frac {1}{| L |} \left| \left\{t: d _ {t} \leq \theta_ {d} \right\} \right|,
$$
$$
\mathrm {N} (E) = \frac {1}{| E |} \sum_ {l = 1} ^ {| E |} \frac {1}{| L |} \left| \left\{t: N \left(d _ {t}\right) \leq \theta_ {d} ^ {\prime} \right\} \right|, \tag {3}
$$
$$
\mathrm {S} (E) = \frac {1}{| E |} \sum_ {l = 1} ^ {| E |} \frac {1}{| L |} | \{t: s _ {t} \geq \theta_ {s} \} |.
$$
The precision plot is generated by calculating the proportion of frames with a distance  $d_t$  less than or equal to  $\theta_d$  and plotting the statistical results across different  $\theta_d$  values. In most cases, existing benchmarks use  $\theta_d = 20$  as a standard threshold to rank trackers.
The normalized precision plot is generated similarly by plotting statistical results obtained from varying  $\theta_{d}^{\prime}$  values within the range of [0,1]. However, directly assigning a specific  $\theta_{d}^{\prime}$  value to rank trackers can introduce subjective biases. Therefore, the ranking of trackers is based on the proportion of frames in which the predicted center  $c_{p}$  successfully falls within the ground-truth rectangle  $g_{t}$ .
The success plot is generated by plotting the results obtained from different overlap thresholds,  $\theta_{s}$  on a curve. In this plot, the mAO (mean average overlap) is commonly utilized to rank trackers.
B.2 Baseline Information
All experiments are performed on a server with 4 NVIDIA TITAN RTX GPUs and a 64 Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz. Detailed information about the baselines are illustrated in Table A4, we use the parameters provided by the original authors.
Table A4: Table: The model architectures and URLs of open-sourced algorithms used in this work.
Tracker	Architecture	Initializa	URL
SiamCAR [11]	SNN	BBox	https://github.com/ohhhyeahhh/SiamCAR
SiamRCNN [10]	SNN	BBox	https://github.com/VisualComputingInstitute/SiamR-CNN
PrDiMP [12]	SNN	BBox	https://github.com/visionml/pytracking
KeepTrack [13]	SNN	BBox	https://github.com/visionml/pytracking
TransT [39]	Transformer	BBox	https://github.com/chenxin-dlut/TransT
MixFormer [14]	Transformer	BBox	https://github.com/MCG-NJU/MixFormer
OSTrack [15]	Transformer	BBox	https://github.com/botaoye/OSTrack
GRM [16]	Transformer	BBox	https://github.com/Little-Podi/GRM
SNLT [46]	SNN	NL&BBox	https://github.com/fredfung007/snlt
VLT_SCAR [42]	SNN	NL&BBox	https://github.com/JudasDie/SOTS
VLT_TT [42]	Transformer	NL&BBox	https://github.com/JudasDie/SOTS
JointNLT [18]	Transformer	NL&BBox	https://github.com/lizhou-cs/JointNLT
Note: SNN-Siamese Neural Network. NL-Natural Language. BBox-Bounding Box.
B.3 More Detailed Experimental Results on MGIT
(a) Precision plots on MGIT
(b) Normalized precision plots on MGIT
(c) Success plots on MGIT (based on IoU)
Figure A4: The precision plot (a), normalized precision plot (b), and the success plot (c) on MGIT.
Figure A4 illustrates the precision plot (a), normalized precision plot (b), and the success plot (c). The performance of trackers indicates that the multi-modal trackers still exhibits a certain gap when compared to visual-based trackers.
Specially, the multi-modal tracker SNLT [46] performs poorly in MGIT. The possible reasons behind the poor results of SNLT are as follows: (1) SNLT is based on the local search, which exhibits a performance gap to the global search trackers. Experiments reveal that local search trackers may encounter a more severe tracking drift problem in the MGIT task (this method tracks by cutting out the search area from the original image, while the high image resolution in MGIT will challenge it). Besides, SNLT's weaker tracking ability is more prone to failure. The errors generated as a result will further misguide, thus creating a negative loop. For example, when it loses the target or drifts towards a similar object, it will persist in tracking failure until the target reemerges within its search area with more recognizable visual information. (2) In addition, there are some issues within SNLT's open-source code, and although we fix them during our reproduction and evaluation, SNLT has some limitations in terms of project completeness compared to other better-maintained multi-modal open-sourced trackers, which may also contribute to its poor performance.
Besides, some other challenges may also influence the tracking performance. Here, we discuss the challenges trackers face from two perspectives: by comparing different multi-modal information provisioning mechanisms, and from limitations between single-modal and multi-modal approaches.
1. Comparison of different multi-modal evaluation mechanisms: (1) First, compared to mechanism A, which only provides semantic information for the first action, multi-modal methods do not show a significant performance decrease in mechanisms B, C, and D. Except JointNLT [18], the success rate scores of other multi-modal trackers in mechanism D are superior to mechanism A. (2) Furthermore, most algorithms performed the worst in mechanism A because they only obtained semantic descriptions for the first action, and this semantic information is not updated in the subsequent process, which may introduce noise to the tracker as the sequence progresses. (3) Mechanisms B and C both regularly update the semantic information during tracking. However, C has a moderate frequency of semantic information updates, and each update provides a moderate length of semantic information, which may better leverage the capabilities of trackers. (4) Mechanism D, similar to mechanism A, only provides semantic information in the initial frame but offers story information that can cover the entire video. However, current multi-modal trackers lack well-designed semantic understanding modules for handling long texts, making it challenging to align semantic information with visual information
2. Comparison between single-modal and multi-modal approaches: Semantic modalities can provide information beyond superficial features such as appearance and location compared to pure visual trackers. However, achieving better correlation and fusion between modalities still needs to be solved. Previous research on other multi-modal tasks has experimentally and theoretically demonstrated that multi-modal approaches can introduce more information, resulting in improved algorithms [48]. However, in the field of SOT research, the performance of multi-modal algorithms still lags behind that of single-modal algorithms. The main reason may be related to the need for more high-quality benchmarks – existing multi-modal benchmarks have significant limitations in the completeness of semantic information and video complexity, making it challenging to provide a favorable experimental environment for multi-modal trackers. Additionally, these multi-modal benchmarks do not adopt a multi-granular annotation strategy, resulting in an evaluation system that only involves mechanism A. As a result, they cannot thoroughly explore the current methods' bottlenecks like our work.
Therefore, the emergence of MGIT can provide a high-quality experimental environment for research, and help the algorithms quickly identify bottleneck issues under various evaluation mechanisms, thereby accelerating the development of efficient multi-modal trackers.
Furthermore, the multi-modal trackers demonstrate superior performance in terms of the normalized precision plot (Figure A4 (b)). We attribute this to the integration of the semantic modality, which enables the multi-modal tracker to effectively recognize the target's position. This observation aligns with intuition, but there is room for further improvement in localization accuracy.
B.4 A Comparison Experiment about Using Various Granularities
Figure A5: An illustration of mechanism B, mechanism B, and their combination.
Table A5: Results of mechanism B, mechanism B, and their combination.
Tracker	Architecture	Initialize	Mechanism	PRE	N-PRE	SR
SNLT [46]	SNN	NL&BBox	Action (B)	0.004	0.226	0.036
Story (D)	0.005	0.230	0.040
Combination (B+D)	0.004	0.229	0.037
VLT_SCAR [42]	SNN	NL&BBox	Action (B)	0.116	0.354	0.167
Story (D)	0.127	0.403	0.184
Combination (B+D)	0.107	0.367	0.165
VLT_TT [42]	Transformer	NL&BBox	Action (B)	0.318	0.602	0.468
Story (D)	0.322	0.616	0.480
Combination (B+D)	0.327	0.612	0.477
JointNLT [18]	Transformer	NL&BBox	Action (B)	0.445	0.786	0.610
Story (D)	0.433	0.773	0.600
Combination (B+D)	0.443	0.783	0.607
Considering that integrating information from different granularities may further benefit the algorithms, we here take mechanisms B and D as an example to explore whether multiple granularities of information are more effective for algorithms. As shown in Figure A5, we combine mechanisms B and D using the following strategy: taking mechanism B as the main component, we replace its semantic information from the first frame with story information (mechanism D). The experimental results are shown in Table A5.
Since most current multi-modal trackers perform target feature matching frame by frame, when the first frame receives story information, the semantic information will be replaced by new action information after reaching the next action. Therefore, the effective range of story information only covers the initial action. As a result, for most algorithms, the score difference between combination  $(\mathrm{B} + \mathrm{D})$  and the original mechanism B is insignificant.
It is worth noting that this experiment only provides a simple and direct approach to evaluating tracking performance with the combination of multi-granularity information. The direct reason for insignificant improvement lies in the limitations of existing trackers (lacking a well-designed semantic processing module and the poor multi-modal alignment capability). If future multi-modal tracking algorithms can design a stronger semantic information processing module to comprehensively represent the video content (i.e., hierarchically constructing video content based on graph [43]), perhaps a more powerful tracking capability can be obtained than simply using a single granularity of information.