SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pha Nguyen<sup>1</sup>, Kha Gia Quach<sup>2</sup>, Kris Kitani<sup>3</sup>, Khoa Luu<sup>1</sup>
$^{1}$  CVIU Lab, University of Arkansas  $^{2}$  pdActive Inc.  $^{3}$  Robotics Institute, Carnegie Mellon University
$^{1}$  {panguyen, khoaluu}@uark.edu  $^{2}$ kquach@ieee.org  $^{3}$ kkitani@cs.cmu.edu
uark-cviu.github.io/Type-to-Track
Figure 1: An example of the responsive Type-to-Track. The user provides a video sequence and a prompting request. During tracking, the system is able to discriminate appearance attributes to track the target subjects accordingly and iteratively responds to the user's tracking request. Each box color represents a unique identity.
One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions. We present a new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework (MENDER) using the third-order tensor decomposition. The experiments in five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to  $14.7\%$  accuracy and  $4\times$  speed faster.
Tracking the movement of objects in videos is a challenging task that has received significant attention in recent years. Various methods have been proposed to tackle this problem, including deep learning techniques. However, despite these advances, there is still room for improvement in intuitiveness and responsiveness. One potential way to improve object tracking in videos is to incorporate user input into the tracking process. Traditional Visual Object Tracking (VOT) methods typically require
Table 1: Comparison of current datasets. # denotes the number of the corresponding item. Bold numbers are the best number in each sub-block, while highlighted numbers are the best across all sub-blocks.
Datasets	Task	NLP	#Videos	#Frames	#Tracks	#AnnBoxes	#Words	#Settings
OTB100 [8]	SOT	X	100	59K	100	59K	-	-
VOT-2017 [9]	SOT	X	60	21K	60	21K	-	-
GOT-10k [10]	SOT	X	10K	1.5M	10K	1.5M	-	-
TrackingNet [11]	SOT	X	30K	14.43M	30K	14.43M	-	-
MOT17 [12]	MOT	X	14	11.2K	1.3K	0.3M	-	-
TAO [13]	MOT	X	1.5K	2.2M	8.1K	0.17M	-	-
MOT20 [14]	MOT	X	8	13.41K	3.83K	2.1M	-	-
BDD100K [15]	MOT	X	2K	318K	130.6K	3.3M	-	-
LaSOT [6]	SOT	✓	1.4K	3.52M	1.4K	3.52M	9.8K	1
TNL2K [7]	SOT	✓	2K	1.24M	2K	1.24M	10.8K	1
Ref-DAVIS [16]	VOS	✓	150	94K	400+	-	10.3K	2
Refer-YTVOS [17]	VOS	✓	4K	1.24M	7.4K	131K	158K	2
Ref-KITTI [18]	MOT	✓	18	6.65K	-	-	3.7K	1
GroOT (Ours)	MOT	✓	1,515	2.25M	13.3K	2.57M	256K	5
users to manually select objects in the video by points [1], bounding boxes [2, 3], or trained object detectors [4, 5]. Thus, in this paper, we introduce a new paradigm, called Type-to-Track, to this task that combines responsive typing input to guide the tracking of objects in videos. It allows for more intuitive and conversational tracking, as users can simply type in the name or description of the object they wish to track, as illustrated in Fig. 1. Our intuitive and user-friendly Type-to-Track approach has numerous potential applications, such as surveillance and object retrieval in videos.
We present a new Grounded Multiple Object Tracking dataset named GroOT that is more advanced than existing tracking datasets [6, 7]. GroOT contains videos with various types of multiple objects and detailed textual descriptions. It is  $2 \times$  larger and more diverse than any existing datasets, and it can construct many different evaluation settings. In addition to three easy-to-construct experimental settings, we propose two new settings for prompt-based visual tracking. It brings the total number of settings to five, which will be presented in Section 5. These new experimental settings challenge existing designs and highlight the potential for further advancements in our proposed research topic.
In summary, this work addresses the use of natural language to guide and assist the Multiple Object Tracking (MOT) tasks with the following contributions. First, a novel paradigm named Type-to-Track is proposed, which involves responsive and conversational typing to track any objects in videos. Second, a new GroOT dataset is introduced. It contains videos with various types of objects and their corresponding textual descriptions of 256K words describing definition, appearance, and action. Next, two new evaluation protocols that are tracking by retrieval prompts and caption prompts, and three class-agnostic tracking metrics are formulated for this problem. Finally, a new transformer-based eMbed-ENcoDE-extRact framework (MENDER) is introduced with third-order tensor decomposition as the first efficient approach for this task. Our contributions in this paper include a novel paradigm, a rich semantic dataset, an efficient methodology, and challenging benchmarking protocols with new evaluation metrics. These contributions will be advantageous for the field of Grounded MOT by providing a valuable foundation for the development of future algorithms.
2.1 Visual Object Tracking Datasets and Benchmarks
Datasets. To develop and train VOT models for the computer vision task of tracking objects in videos, various datasets have been created and widely used. Some of the most popular datasets for VOT are OTB [19, 8], VOT [9], GOT [10], MOT challenges [12, 14] and BDD100K [15]. Visual object tracking has two sub-tasks: Single Object Tracking (SOT) and Multiple Object Tracking (MOT). Table 1 shows that there is a wide variety of object tracking datasets in both types available, each with its own strengths and weaknesses. Existing datasets with NLP [6, 7] only support the SOT task, while our GroOT dataset supports MOT with approximately  $2 \times$  larger in description size.
Benchmarks. Current benchmarks for tracking can be broadly classified into two main categories: Tracking by Bounding Box and Tracking by Natural Language, depending on the type of initialization.
Table 2: Comparison of key features of tracking methods. Cls-agn is for class-agnostic, while Feat is for the approach of feature fusion and Stages indicates the number of stages in the model design incorporating NLP into the tracking task. NLP indicates how text is utilized for the tracker: assist (w/ box) or can initialize (w/o box).
Approach	Task	NLP	Cls-agn	Feat	Stages
GTI [27]	SOT	assist	X	concat	single
TransVLT [28]	SOT	assist	X	attn	single
TrackFormer [4]	MOT	-	X	-	-
MDETR+TFm	MOT	init	✓	attn	two
TransRMOT [18]	MOT	init	✓	attn	two
MENDER	MOT	init	✓	attn	single
* Statistics from the official site, including objects other than human.
** Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License
Table 3: Statistics of GroOT's settings.
Datasets	#Videos	#Frames	#Tracks	#AnnBoxes	#Words	Parts
MOT17**	Train	7	5,316	546*	112,297*	3,792	(1)
Test	7	5,919	785*	188,076*	5,757	(2)
Total	14	11,235	1,331*	300,373*	9,549
TAO**	Train	500	764,526	2,645	54,639	19,222	(3)
Val	993	1,460,666	5,485	113,112	39,149	(4)
Test	914	2,221,846	7,972	164,650	-
Total	2,407	4,447,038	16,089	332,401	58,371
MOT20**	Train	4	8,931	2,332*	1,336,920*	-	(5)
Test	4	4,479	1,501*	765,465*	-	(6)
Total	8	13,410	3,833*	2,102,385*	-
GroOT**	nm	1,515	2,249,837	13,294	2,570,509	21,424	all
syn	1,515	2,249,837	13,294	2,570,509	53,540	all
def	1,515	2,249,837	13,294	2,570,509	99,218	all
cap	1,507	2,236,427	9,461	468,124	67,920	w/o MOT20 uses (4)
retr	993	1,460,666	1,952	-	13,935
all uses (1, 2, 3, 4, 5, 6) and w/o MOT20 uses (1, 2, 3, 4).
Previous benchmarks [20, 19, 8, 9, 21, 22, 22, 23] were limited to test videos before the emergence of deep trackers. The first publicly available benchmarks for visual tracking were OTB-2013 [19] and OTB-2015 [8], consisting of 50 and 100 video sequences, respectively. GOT-10k [10] is a benchmark featuring 10K videos classified into 563 classes and 87 motions. TrackingNet [11], a subset of the object detection benchmark YT-BB [24], includes 31K sequences. Furthermore, there are long-term tracking benchmarks such as OxUvA [25] and LaSOT [6]. OxUvA spans 14 hours of video in 337 videos, comprising 366 object tracks. On the other hand, LaSOT [6] is a language-assisted dataset consisting of 1.4K sequences with 9.8K words in their captions. In addition to these benchmarks, TNL2K [7] includes 2K video sequences for natural language-based tracking and focuses on expressing the attributes. LaSOT [6] and TNL2K [7] support one benchmarking setting with their provided prompts, while our GroOT dataset supports five settings. Ref-KITTI [18] is built upon the KITTI [26] dataset and contains only two categories, including car and pedestrian, while our GroOT dataset focuses on category-agnostic tracking, and outnumber the frames and settings.
A similar task with a different nomenclature to the Grounded MOT task is Referring Video Object Segmentation (Ref-VOS) [16, 17], which primarily measures the overlapping area between the ground truth and prediction for a single foreground object in each caption, with less emphasis on densely tracking multiple objects over time. In contrast, our proposed Type-to-Track paradigm is distinct in its focus on responsively and conversationally typing to track any objects in videos, requiring maintaining the temporal motions of multiple objects of interest.
2.2 Grounded Object Tracking
Grounded Vision-Language Models accurately map language concepts onto visual observations by understanding both vision content and natural language. For instance, visual grounding [29] seeks to identify the location of nouns or short phrases (such as a black hat or a blue bird) within an image. Grounded captioning [30, 31, 32] can generate text descriptions and align predicted words with object regions in an image. Visual dialog [33] enables meaningful dialogues with humans about visual content using natural, conversational language. Some visual dialog systems may incorporate referring expression recognition [34] to resolve expressions in questions or answers.
Grounded Single Object Tracking is limited to tracking a single object with box-initialized and language-assisted methods. The GTI [27] framework decomposes the tracking by language task into three sub-tasks: Grounding, Tracking, and Integration, and generates tubelet predictions frame-by-frame. AdaSwitcher [7] module identifies tracking failure and switches to visual grounding for better tracking. [35] introduce a unified system using attention memory and cross-attention modules with learnable semantic prototypes. Another transformer-based approach [28] is presented including a cross-modal fusion module, task-specific heads, and a proxy token-guided fusion module.
2.3 Discussion
Most existing datasets and benchmarks for object tracking are limited in their coverage and diversity of language and visual concepts. Additionally, the prompts in the existing Grounded SOT benchmarks do not contain variations in covering many objects in a single prompt, which limits the application of existing trackers in practical scenarios. To address this, we present a new dataset and benchmarking
(a) Our MOT17 [12] subset sample with captions in both action and appearance types.
(a) Our MOT17 [12] subset.
(b) Our TAO [13] subset samples with captions. Best viewed in color and zoom in.
Figure 2: Example sequences and annotations in our dataset.
Figure 3: Some words in our language description.
(b) Our TAO [13] subset.
metrics to support the emerging trend of the Grounded MOT, where the goal is to align language descriptions with fine-grained regions or objects in videos.
As shown in Table 2, most of the recent methods for the Grounded SOT task are not class-agnostic, meaning they require prior knowledge of the object. GTI [27] and TransVLT [28] need to input the initial bounding box, while TrackFormer [4] need the pre-defined category. The operation used in [27] to fuse visual and textual features is concatenation which can only support prompts describing a single object. A Grounded MOT can be constructed by integrating a grounded object detector, i.e. MDETR [36], and an object tracker, i.e. TrackFormer [4]. However, this approach is low-efficient because the visual features have to be extracted multiple times. In contrast, our proposed MOT approach MENDER formulates third-order attention to adaptively focus on many targets, and it is an efficient single-stage and class-agnostic framework. The scope of class-agnostic in our approach is constructing a large vocabulary of concepts via a visual-textual corpus, following [37, 38, 39].
3 Dataset Overview
3.1 Data Collection and Annotation
Existing object tracking datasets are typically designed for specific types of video scenes [40, 41, 42, 43, 44, 2]. To cover a diverse range of scenes, GroOT was created using official videos and bounding box annotations from the MOT17 [12], TAO [13], and MOT20 [14]. The MOT17 dataset comprises 14 sequences with diverse environmental conditions such as crowded scenes, varying viewpoints, and camera motion. The TAO dataset is composed of videos from seven different datasets, such as the ArgoVerse [45] and BDD [15] datasets containing outdoor driving scenes, while LaSOT [6] and YFCC100M [46] datasets include in-the-wild internet videos. Additionally, the AVA [47], Charades [48], and HACS [49] datasets include videos depicting human-human and human-object interactions. By combining these datasets, GroOT covers multiple types of scenes and encompasses a wide range of 833 objects. This diversity allows for a wide range of object classes with captions to be included, making it an invaluable resource for training and evaluating visual grounding algorithms.
We release our textual description annotations in COCO format [50]. Specifically, a new key 'captions' which is a list of strings is attached to each 'annotations' item in the official annotation. In the MOT17 subset, we attempt to maintain two types of caption for well-visible objects: one describes the appearance and the other describes the action. For example, the caption for a well-visible person might be ['a man wearing a gray shirt', 'person walking on the street'] as shown in Fig. 2a. However,  $10\%$  of tracklets only have one caption type, and  $3\%$  do not have any captions due to their low visibility. The physical characteristics of a person or their personal accessories, such as their clothing, bag color, and hair color are considered to be part of their appearance. Therefore, the appearance captions include verbs 'carrying' or 'holding' to describe personal accessories. In the TAO subset, objects other than humans have one caption
describing appearance, for instance, [‘a red and black scooter’]. Objects that are human have the same two types of captions as the MOT17 subset. An example is shown in Fig. 2b. These captions are consistently annotated throughout the tracklets. Fig. 3 is the word-cloud visualization of our annotations.
3.2 Type-to-Track Benchmarking Protocols
Let  $\mathbf{V}$  be a video sample lasts  $t$  frames, where  $\mathbf{V} = \left\{\mathbf{I}_t \mid t < |\mathbf{V}|\right\}$  and  $\mathbf{I}_t$  be the image sample at a particular time step  $t$ . We define a request prompt  $\mathbf{P}$  that describes the objects of interest, and  $\mathbf{T}_t$  is the set of tracklets of interest up to time step  $t$ . The Type-to-Track paradigm requires a tracker network  $\mathcal{T}(\mathbf{I}_t, \mathbf{T}_{t-1}, \mathbf{P})$  that efficiently takes into account  $\mathbf{I}_t$ ,  $\mathbf{T}_{t-1}$ , and  $\mathbf{P}$  to produce  $\mathbf{T}_t = \mathcal{T}(\mathbf{I}_t, \mathbf{T}_{t-1}, \mathbf{P})$ . To advance the task of multiple object retrieval, another benchmarking set is created in addition to the GroOT dataset. While training and testing sets follow a One-to-One scenario, where each caption describes a single tracklet, the new retrieval set contains prompts that follow a One-to-Many scenario, where a short prompt describes multiple objects. This scenario highlights the need for diverse methods to improve the task of multiple object retrieval. The retrieval set is provided with a subset of tracklets in the TAO validation set and three custom retrieval prompts that change throughout the tracking process in a video  $\{\mathbf{P}_{t_1=0}, \mathbf{P}_{t_2}, \mathbf{P}_{t_3}\}$ , as depicted in Fig. 1(a). The retrieval prompts are generated through a semi-automatic process that involves: (i) selecting the most commonly occurring category in the video, and (ii) cascadingly filtering to the object that appears for the longest duration. In contrast, the caption prompts are created by joining tracklet captions in the scene and keeping it consistent throughout the tracking period. We name these two evaluation scenarios as tracklet captions cap and object retrieval retr. With three more easy-to-construct scenarios, five scenarios in total will be studied for the experiments in Section 5. Table 3 presents the statistics of the five settings, and the data portions are highlighted in the corresponding colors.
3.3 Class-agnostic Evaluation Metrics
As indicated in [51], long-tailed classification is a very challenging task in imbalanced and large-scale datasets such as TAO. This is because it is difficult to distinguish between similar fine-grained classes, such as bus and van, due to the class hierarchy. Additionally, it is even more challenging to treat every class independently. The traditional method of evaluating tracking performance leads to inadequate benchmarking and undesired tracking results. In our Type-to-Track paradigm, the main task is not to classify objects to their correct categories but to retrieve and track the object of interest. Therefore, to alleviate the negative effect, we reformulate the original per-category metrics of MOTA [52], IDF1 [53], HOTA [54] into class-agnostic metrics:
$$
\begin{array}{l} \text {M O T A} = \frac {1}{| C L S ^ {n} |} \sum_ {c l s} ^ {C L S ^ {n}} \left(1 - \frac {\sum_ {t} \left(\mathrm {F N} _ {t} + \mathrm {F P} _ {t} + \mathrm {I D S} _ {t}\right)}{\sum_ {t} \mathrm {G T} _ {t}}\right) _ {c l s}, \text {C A - M O T A} = 1 - \frac {\sum_ {t} \left(\mathrm {F N} _ {t} + \mathrm {F P} _ {t} + \mathrm {I D S} _ {t}\right) _ {C L S ^ {1}}}{\sum_ {t} \left(\mathrm {G T} _ {C L S ^ {1}}\right) _ {t}} (1) \\ \mathrm {I D F 1} = \frac {1}{| C L S ^ {n} |} \sum_ {c l s} ^ {C L S ^ {n}} \left(\frac {2 \times \text {I D T P}}{2 \times \text {I D T P} + \text {I D F P} + \text {I D F N}}\right) _ {c l s}, \quad \text {C A - I D F 1} = \frac {(2 \times \text {I D T P}) _ {C L S ^ {1}}}{(2 \times \text {I D T P} + \text {I D F P} + \text {I D F N}) _ {C L S ^ {1}}} (2) \\ \mathrm {H O T A} = \frac {1}{| C L S ^ {n} |} \sum_ {c l s} ^ {C L S ^ {n}} \left(\sqrt {\operatorname {D e t} \mathrm {A} \cdot \operatorname {A s s} \mathrm {A}}\right) _ {c l s}, \quad \mathrm {C A - H O T A} = \sqrt {\left(\operatorname {D e t} \mathrm {A} _ {C L S ^ {1}}\right) \cdot \left(\operatorname {A s s} \mathrm {A} _ {C L S ^ {1}}\right)} (3) \\ \end{array}
$$
where  $CLS^n$  is the category, set size  $n$  is reduced to 1 by combining all elements:  $CLS^n \to CLS^1$ .
Given the image  $\mathbf{I}_t$  and the request prompt  $\mathbf{P}$  describing the objects of interest, which can adaptively change between  $\{\mathbf{P}_{t_1},\mathbf{P}_{t_2},\mathbf{P}_{t_3}\}$  in the retr setting, and  $K$  is the prompt's length  $|\mathbf{P}| = K$ , let enc(\cdot) and  $\text{emb}(\cdot)$  be the visual encoder and the word embedding model to extract features of image tokens and prompt tokens, respectively. The resulting outputs,  $\text{enc}(\mathbf{I}_t) \in \mathbb{R}^{M \times D}$  and
emb  $(\mathbf{P})\in \mathbb{R}^{K\times D}$ , where  $D$  is the length of feature dimensions. A list of region-prompt associations  $\mathbf{C}_t$ , which contains objects' bounding boxes and their confident scores, can be produced by Eqn. (4):
$$
\mathbf {C} _ {t} = \operatorname {d e c} _ {\gamma} \left(\operatorname {e n c} (\mathbf {I} _ {t}) \bar {\times} \operatorname {e m b} (\mathbf {P}) ^ {\intercal}, \operatorname {e n c} (\mathbf {I} _ {t})\right) = \left\{\mathbf {c} _ {i} = \left(c _ {x}, c _ {y}, c _ {w}, c _ {h}, c _ {\text {c o n f}}\right) _ {i} \mid i <   M \right\} _ {t} \tag {4}
$$
where  $(\bar{\times})$  is an operation representing the region-prompt correlation, that will be elaborated in the next section,  $dec(\cdot ,\cdot)$  is an object decoder taking the similarity and the image features to decode to object locations, thresholded by a scoring parameter  $\gamma$  (i.e.  $c_{conf}\geq \gamma$ ). For simplicity, the cardinality of the set of objects  $|\mathbf{C}_t| = M$ , implying each image token produces one region-text correlation.
We define  $\mathbf{T}_t = \left\{\mathbf{tr}_j = (tr_x, tr_y, tr_w, tr_h, tr_{conf}, tr_id)_j \mid j < N\right\}_t$  produced by the tracker  $\mathcal{T}$ , where  $N = |\mathbf{T}_t|$  is the cardinality of current tracklets.  $i, j, k,$  and  $t$  are consistently denoted as indexers for objects, tracklets, prompt tokens, and time steps for the rest of the paper.
Remark 1 Third-order Tensor Modeling. Since the Type-to-Track paradigm requires three input components  $\mathbf{I}_t$ ,  $\mathbf{T}_{t - 1}$ , and  $\mathbf{P}$ , an auto-regressive single-stage end-to-end framework can be formulated via third-order tensor modeling.
To achieve this objective, a combination of initialization, object decoding, visual encoding, feature extraction, word embedding, and aggregation can be formulated as in Eqn. (5):
$$
\mathbf {T} _ {t} = \left\{ \begin{array}{l l} \text {i n i t i a l i z e} (\mathbf {C} _ {t}) & t = 0 \\ d e c \left(\mathbf {1} _ {D \times D \times D} \times_ {1} \operatorname {e n c} (\mathbf {I} _ {t}) \times_ {2} \operatorname {e x t} (\mathbf {T} _ {t - 1}) \times_ {3} \operatorname {e m b} (\mathbf {P}), \operatorname {e n c} (\mathbf {I} _ {t})\right) & \forall t > 0 \end{array} \right. \tag {5}
$$
where  $\text{ext}(\cdot)$  denotes the visual feature extractor of the set of tracklets,  $\text{ext}(\mathbf{T}_{t-1}) \in \mathbb{R}^{N \times D}$ ,  $\mathbf{1}_{D \times D \times D}$  is an all-ones tensor has size  $D \times D \times D$ ,  $(\times_{n})$  is the  $n$ -mode product of the third-order tensor [55] to aggregate many types of token $^1$ , and initialize( $\cdot$ ) is the function to ascendingly assign unique identities to tracklets for the first time those tracklets appear.
Let  $T \in \mathbb{R}^{M \times N \times K}$  be the resulting tensor  $T = \mathbf{1}_{D \times D \times D} \times_1 \text{enc}(\mathbf{I}_t) \times_2 \text{ext}(\mathbf{T}_{t-1}) \times_3 \text{emb}(\mathbf{P})$ . The objective function can be expressed as the log softmax of the positive region-tracklet-prompt triplet over all possible triplets, defined in Eqn. (6):
$$
\theta_ {e n c, e x t, e m b} ^ {*} = \arg \max  _ {\theta_ {e n c, e x t, e m b}} \left(\log \left(\frac {\exp \left(T _ {i j k}\right)}{\sum_ {l} ^ {K} \sum_ {n} ^ {N} \sum_ {m} ^ {M} \exp \left(T _ {l n m}\right)}\right)\right) \tag {6}
$$
where  $\theta$  denotes the network's parameters, the combination of the  $i^{th}$  image token, the  $j^{th}$  tracklet, and the  $k^{th}$  prompt token is the correlated triplet.
In the next subsection, we elaborate our model design for the tracking function  $\mathcal{T}(\mathbf{I}_t,\mathbf{T}_{t - 1},\mathbf{P})$  named MENDER, as defined in Eqn. (5), and loss functions for the problem objective in Eqn. (6).
4.2 MENDER for Multiple Object Tracking by Prompts
The correlation in Eqn. (5) has the cubic time and space complexity  $\mathcal{O}(n^3)$ , which can be intractable as the input length grows and hinder the model scalability.