SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 2 Correlation Simplification. Since both  $\mathrm{enc}(\cdot)$  and  $\mathrm{ext}(\cdot)$  are visual encoders, the region-prompt correlation can be equivalent to the tracklet-prompt correlation. Therefore, the region-tracklet-prompt correlation tensor  $T$  can be simplified to lower the computation footprint.
To design that goal, the extractor and encoder share network weights for computational efficiency:
$$
\operatorname {e x t} \left(\mathbf {T} _ {t - 1}\right) _ {j} = \operatorname {e x t} \left(\left\{\mathbf {t r} _ {j} \right\} _ {t - 1}\right) = \left\{\operatorname {e n c} \left(\mathbf {I} _ {t - 1}\right) _ {i}: \mathbf {c} _ {i} \mapsto \mathbf {t r} _ {j} \right\}, \text {t h e r e f o r e} \left((T _ {: j:}) _ {t - 1} = (T _ {i: :}) _ {t}\right): \mathbf {c} _ {i} \mapsto \mathbf {t r} _ {j} ^ {2} \tag {7}
$$
where  $T_{j:}$  and  $T_{i:}$  are lateral and horizontal slices. In layman's terms, the region-prompt correlation at the time step  $t - 1$  is equivalent to the tracklet-prompt correlation at the time step  $t$ , as visualized in Fig. 4(a). Therefore, one practically needs to model the region-tracklet and tracklet-prompt
1 implemented by a single Python code with Numpy: np.einsum('ai, bj, ck -> abc', P, I, T).
2 If  $\mathbf{P}$  changes, the equivalence still holds true, see Appendix for the full algorithm.
(a) Because the tracklet set  $\mathbf{T}_{t - 1}$  pools visual features of the image  $\mathbf{I}_{t - 1}$ , the region-prompt is equivalent with tracklet-prompt (only need to filter unassigned objects).
Figure 4: The auto-regressive manner takes advantage of the equivalent components. Simplifying the correlation in (a) turns the solution to MENDER in (b), and reduces complexity to  $\mathcal{O}(n^2)$  where  $n$  denotes the size of tokens.
(b) The structure of our proposed MENDER. It employs a visual backbone to extract visual features and a word embedding to extract textual features. We model the tracklet-prompt correlation  $\text{ext}(\mathbf{T}_{t-1}) \bar{\times} \text{emb}(\mathbf{P})^{\top}$  instead of the region-prompt to avoid unnecessary computation caused by no-object tokens [56]. Best viewed in color and zoom in.
correlations which reduces time and space complexity from  $\mathcal{O}(n^3)$  to  $\mathcal{O}(n^2)$ , significantly lowering computation footprint. We alternatively rewrite the decoding step in Eqn. (5) as follows:
$$
\mathbf {T} _ {t} = \operatorname {d e c} _ {\gamma} \left(\left(e n c \left(\mathbf {I} _ {t}\right) \bar {\times} e x t \left(\mathbf {T} _ {t - 1}\right) ^ {\top}\right) \times \left(e x t \left(\mathbf {T} _ {t - 1}\right) \bar {\times} e m b (\mathbf {P}) ^ {\top}\right), e n c \left(\mathbf {I} _ {t}\right)\right) \quad \forall t > 0 \tag {8}
$$
Correlation Representations. In our approach, the correlation operation  $(\bar{\times})$  is modelled by the multi-head cross-attention mechanism [57], as depicted in Fig. 4(b). The attention matrix can be computed as:
$$
\sigma (\mathbf {X}) \bar {\times} \sigma (\mathbf {Y}) = \mathcal {A} _ {\mathbf {X} | \mathbf {Y}} = \operatorname {s o f t m a x} \left(\frac {\left(\sigma (\mathbf {X}) \times W _ {Q} ^ {\mathbf {X}}\right) \times \left(\sigma (\mathbf {Y}) \times W _ {K} ^ {\mathbf {Y}}\right) ^ {\intercal}}{\sqrt {D}}\right) \tag {9}
$$
where  $\mathbf{X}$  and  $\mathbf{Y}$  tokens are one of these types: region, tracklet, prompt.  $\sigma (\cdot)$  is one of the operations  $enc(\cdot),emb(\cdot),ext(\cdot)$  as the corresponding operation to  $\mathbf{X}$  or  $\mathbf{Y}$ . Superscript  $W_{Q}$ ,  $W_{K}$ , and  $W_{V}$  are the projection matrices corresponding to  $\mathbf{X}$  or  $\mathbf{Y}$  as in the attention mechanism.
Then, the attention weight from the image  $\mathbf{I}_t$  to the prompt  $\mathbf{P}$  are computed by the matrix multiplication for  $\mathcal{A}_{\mathbf{I}|\mathbf{T}}$  and  $\mathcal{A}_{\mathbf{T}|\mathbf{P}}$  to aggregate the information from two matrices as in Eqn. (8). The result is the matrix  $\mathcal{A}_{\mathbf{I}|\mathbf{T}\times \mathbf{T}|\mathbf{P}} = \mathcal{A}_{\mathbf{I}|\mathbf{T}}\times \mathcal{A}_{\mathbf{T}|\mathbf{P}}$  that shows the correlation between each input or output. Then, the resulting attention matrix  $\mathcal{A}_{\mathbf{I}|\mathbf{T}\times \mathbf{T}|\mathbf{P}}$  is used to produce the object representations at time  $t$ :
$$
\mathbf {Z} _ {t} = \mathcal {A} _ {\mathbf {I} | \mathbf {T} \times \mathbf {T} | \mathbf {P}} \times \left(e m b (\mathbf {P}) \times W _ {V} ^ {\mathbf {P}}\right) + \mathcal {A} _ {\mathbf {I} | \mathbf {T}} \times \left(e x t \left(\mathbf {T} _ {t - 1}\right) \times W _ {V} ^ {\mathbf {T}}\right) \tag {10}
$$
Object Decoder  $dec(\cdot)$  utilizes context-aware features  $Z_{t}$  that are capable of preserving identity information while adapting to changes in position. The tracklet set  $\mathbf{T}_t$  is defined in the auto-regressive manner to adjust to the movements of the object being tracked as in Eqn. (8). For decoding the final output at any frame, the decoder transforms the object representation by a 3-layer FFN to predict bounding boxes and confidence scores for frame  $t$ :
$$
\mathbf {T} _ {t} = \left\{\mathbf {t r} _ {j} = \left(t r _ {x}, t r _ {y}, t r _ {w}, t r _ {h}, t r _ {\text {c o n f}}\right) _ {j} \right\} _ {t} ^ {t r _ {\text {c o n f}} \geq \gamma} \operatorname {F F N} \left(\mathbf {Z} _ {t} + \operatorname {e n c} \left(\mathbf {I} _ {t}\right)\right) \tag {11}
$$
where the identification information of tracklets, represented by  $tr_{id}$ , is not determined directly by the FFN model. Instead, the  $tr_{id}$  value is set when the tracklet is first initialized and maintained till its end, similar to tracking-by-attention approaches [4, 58, 59, 60].
4.3 Training Losses
To achieve the training objective function as in Eqn. (6), we formulate the objective function into two loss functions  $L_{\mathbf{I}|\mathbf{T}}$  and  $L_{\mathbf{T}|\mathbf{P}}$  for correlation training and one loss  $L_{GIOU}$  for decoder training:
$$
\mathcal {L} = \gamma_ {\mathbf {T} | \mathbf {P}} L _ {\mathbf {T} | \mathbf {P}} + \gamma_ {\mathbf {I} | \mathbf {T}} L _ {\mathbf {I} | \mathbf {T}} + \gamma_ {G I o U} L _ {G I o U} \tag {12}
$$
where  $\gamma_{\mathbf{T}|\mathbf{P}}$ ,  $\gamma_{\mathbf{I}|\mathbf{T}}$ , and  $\gamma_{GIoU}$  are corresponding coefficients, which are set to 0.3 by default.
Alignment Loss  $L_{\mathbf{T}|\mathbf{P}}$  is a contrastive loss, which is used to assure the alignment of the ground-truth object feature and caption pairs ( $\mathbf{T}, \mathbf{P}$ ) which can be obtained in our dataset. There are two alignment losses used, one for all objects normalized by the number of positive prompt tokens and the other for all prompt tokens normalized by the number of positive objects. The total loss can be expressed as:
$$
\begin{array}{l} L _ {\mathbf {T} | \mathbf {P}} = \\ - \frac {1}{| \mathbf {P} ^ {+} |} \sum_ {k} ^ {| \mathbf {P} ^ {+} |} \log \left(\frac {\exp \left(e x t (\mathbf {T}) _ {j} ^ {\intercal} \times e m b (\mathbf {P}) _ {k}\right)}{\sum_ {l} ^ {K} \exp \left(e x t (\mathbf {T}) _ {j} ^ {\intercal} \times e m b (\mathbf {P}) _ {l}\right)}\right) - \frac {1}{| \mathbf {T} ^ {+} |} \sum_ {j} ^ {| \mathbf {T} ^ {+} |} \log \left(\frac {\exp \left(e m b (\mathbf {P}) _ {k} ^ {\intercal} \times e x t (\mathbf {T}) _ {j}\right)}{\sum_ {l} ^ {N} \exp \left(e m b (\mathbf {P}) _ {k} ^ {\intercal} \times e x t (\mathbf {T}) _ {l}\right)}\right) \tag {13} \\ \end{array}
$$
where  $\mathbf{P}^{+}$  and  $\mathbf{I}^{+}$  are the sets of positive prompts and image tokens corresponding to the selected enc( $\mathbf{I}$ )_i and  $emb(\mathbf{P})_{k}$ , respectively.
Objectness Losses. To model the track's temporal changes, our network learns from training samples that capture both appearance and motion generated by two adjacent frames:
$$
L _ {\mathbf {I} | \mathbf {T}} = - \sum_ {j} ^ {N} \log \left(\frac {\exp \left(e x t (\mathbf {T}) _ {j} ^ {\intercal} \times e n c (\mathbf {I}) _ {i}\right)}{\sum_ {l} ^ {N} \exp \left(e x t (\mathbf {T}) _ {j} ^ {\intercal} \times e n c (\mathbf {I}) _ {l}\right)}\right), \text {a n d} L _ {G I o U} = \sum_ {j} ^ {N} \ell_ {G I o U} \left(\mathbf {t r} _ {j}, \mathbf {o b j} _ {i}\right) \tag {14}
$$
$L_{\mathbf{I}|\mathbf{T}}$  is the log-softmax loss to guide the tokens' alignment as similar to Eqn. (13). In the  $L_{GIOU}$  loss,  $\mathbf{obj}_i$  is the ground truth object corresponding to  $\mathbf{tr}_j$ . The optimal assignment between  $\mathbf{tr}_j$  or  $\mathbf{obj}_i$  to the ground truth object is computed efficiently by the Hungarian algorithm, following DETR [56].  $\ell_{GIOU}$  is the Generalized IoU loss [61].
Experimental Scenarios. We create three types of prompt: category name nm, category synonyms syn, category definition def. One tracklet captions cap scenario is constructed by our detailed annotations and one more objects retrieval retr scenario is given in our custom request prompts as described in Subsec. 3.2. The dataset contains 833 classes, each has a name and a corresponding set of synonyms that are different names for the same category, such as [man, woman, human, pedestrian, boy, girl, child] for person. Additionally, each category is described by a category definition sentence. This definition makes the model deal with the variations in the text prompts. We join the names, synonyms, definitions, or captions and filter duplicates to construct the prompt. Trained models use as the same type as testing. We annotated the raw tracking data of the best-performant tracker (i.e., BoT-SORT [62] at  $80.5\%$  MOTA and  $80.2\%$  IDF1) at the time we constructed experiments and used it as the sub-optimal ground truth of MOT17 and MOT20 (parts (2, 4) in Table 3). That is also the raw data we used to evaluate all our ablation studies.
Datasets and Metrics. RefCOCO+ [63] and Flickr30k [64] serve as pre-trained datasets for acquiring a vocabulary of visual-textual concepts [37]. The  $ext(\cdot)$  operation is not involved in this training step. After obtaining a pre-trained model from RefCOCO+ and Flickr30k, we train and evaluate our model for the proposed Type-to-Track task on all five scenarios on our GroOT dataset and the first-three scenarios for MOT20 [14]. The tracking performance is reported in class-agnostic metrics CA-MOTA, CA-IDF1, and CA-HOTA as in Subsec. 3.3 and mAP50 as defined in [13].
Tokens Production.  $emb(\cdot)$  utilizes RoBERTa [65] to convert the text input into a sequence of numerical tokens. The tokens are fed into the RoBERTa-base model for text encoding using a
Table 4: Ablation studies. sim indicates whether the correlation is the simplified Eqn. (8) or the Eqn. (5). See 5.1 for the abbreviations. The two first settings get only one word for the request prompt, therefore, tensor  $T$  is an unsqueezed matrix, resulting in no difference in  $\mathbf{nm}(X)$  vs  $(\checkmark)$ , and  $\mathbf{syn}(X)$  vs  $(\checkmark)$ .
P	sim	CA-MOTA	CA-IDF1	MT	IDs	mAP	FPS
GroOT - MOT17 Subset
nm	X/√	67.00	71.20	544	1352	0.876	10.3
syn	X/√	65.10	71.10	554	1348	0.874	10.3
def	X	67.00	72.10	556	1343	0.876	5.8
√	67.30	72.40	568	1322	0.877	10.3
cap	X	58.20	53.20	289	1751	0.674	3.4
√	59.50	54.80	201	1734	0.688	7.8
GroOT - TAO Subset
nm	√	27.30	37.20	3523	4284	0.212	11.2
syn	√	25.70	36.10	3212	5048	0.198	11.2
def	X	15.20	27.30	2452	6253	0.154	6.2
√	16.80	27.70	2547	6118	0.158	10.5
cap	X	20.30	31.80	2943	5242	0.188	4.3
√	20.70	32.00	3103	5192	0.184	8.7
retr	X	32.40	38.40	630	3238	0.423	7.6
√	32.90	39.30	645	3194	0.430	11.5
GroOT - MOT20 Subset
nm	X/√	72.40	67.50	823	2498	0.826	7.6
syn	X/√	70.90	65.30	809	2509	0.823	7.6
def	X	72.90	67.70	823	2489	0.826	4.3
√	72.10	67.10	812	2503	0.825	7.6
Table 5: Comparisons to the two-stage baseline design. In each dataset, the from-top-to-bottom scenarios are syn, def, cap and retr. Best viewed in color.
Approach	CA-MOTA	CA-IDF1	MT	IDs	mAP	FPS
GroOT - MOT17 Subset
MDETR + TFm	62.60	64.70	519	1382	0.793	2.2
MENDER	65.10	71.10	554	1348	0.874	10.3
MDETR + TFm	62.60	64.70	519	1382	0.793	2.2
MENDER	67.30	72.40	568	1322	0.877	10.3
MDETR + TFm	44.80	45.20	193	1945	0.619	2.1
MENDER	59.50	54.80	201	1734	0.688	7.8
GroOT - TAO Subset
MDETR + TFm	21.30	33.20	2945	5834	0.184	3.1
MENDER	25.70	36.10	3212	5048	0.198	11.2
MDETR + TFm	14.60	21.40	1944	6493	0.137	3.1
MENDER	16.80	27.70	2547	6118	0.158	10.5
MDETR + TFm	15.30	23.60	2132	6354	0.156	3.0
MENDER	20.70	32.00	3103	5192	0.182	8.7
MDETR + TFm	25.70	26.40	513	3993	0.387	3.1
MENDER	32.90	39.30	645	3194	0.430	11.5
GroOT - MOT20 Subset
MDETR + TFm	61.20	60.40	784	2824	0.732	1.9
MENDER	70.90	65.30	809	2509	0.823	7.6
MDETR + TFm	68.00	66.30	763	2975	0.783	1.9
MENDER	72.10	67.10	812	2503	0.825	7.6
12-layer transformer network with 768 hidden units and 12 self-attention heads per layer. enc(\cdot) is implemented using a ResNet-101 [66] as the backbone to extract visual features from the input image. The output of the ResNet is processed by a Deformable DETR encoder [67] to generate visual tokens. For each dimension, we use sine and cosine functions with different frequencies as positional encodings, similar to [68]. A feature resizer combining a list of (Linear, LayerNorm, Dropout) is used to map to size  $D = 512$  for all token producers.
Comparisons in Different Scenarios. Table 4 shows comparisons in the performance of different prompt inputs. For MOT17 and MOT20, the category name is 'person', while category definition is 'a human being'. Since the prompt by category definition is short, it does not differ much from the nm setting. However, the syn setting shuffles between some words, resulting in a slight decrease in CA-MOTA and CA-IDF1. The cap setting results in prompts that contain more diverse and complex vocabulary, and more context-specific information. It is more difficult for the model to accurately localize the objects and identify their identity within the image, as it needs to take into account a wider range of linguistic cues, resulting in a decrease in performance compared to def (59.5% CA-MOTA and 54.8% CA-IDF1 vs 67.3% CA-MOTA and 72.4% CA-IDF1 on MOT17).
For TAO, the def setting has a significant number of variations and many tenuous connections in the scene context, for example, 'an aircraft that has a fixed wing and is powered by propellers or jets' for the airplane category. Therefore, it results in a decrease in performance (16.8% CA-MOTA and 27.7% CA-IDF1) compared to cap (20.7% CA-MOTA and 32.0% CA-IDF1), because the cap setting is more specific on the object level than category level. The best performant setting is nm (27.3% CA-MOTA and 37.2% CA-IDF1), where names are combined.
Simplified Attention Representations. Table 4 also presents the effectiveness of different attention representations of the full tensor  $T$  (denoted by  $X$ ) and the simplified correlation (denoted by  $\checkmark$ ). The performance is reported with frame per second (FPS), which is self-measured on one GPU NVIDIA RTX 3060 12GB. Overall, the performance of simplified correlation is witnessed with a superior speed of up to  $2 \times$  (7.8 FPS vs 3.4 FPS of cap on MOT17 and 11.5 FPS vs 7.6 FPS of retr on TAO), resulting in and a slight increase in accuracy due to attention stability and precision gain.
Table 6: Comparisons to the state-of-the-art approaches on the category name nm setting.
Approach	Cls-agn	CA-IDF1	CA-MOTA	CA-HOTA	MT	ML	AssA	DetA	LocA	IDs
ByteTrack [69]	✘	77.3	80.3	63.1	957	516	52.7	55.6	81.8	3,378
TrackFormer [4]	✘	68.0	74.1	57.3	1,113	246	54.1	60.9	82.8	2,829
QuasiDense [70]	✘	66.3	68.7	53.9	957	516	52.7	55.6	81.8	3,378
CenterTrack [71]	✘	64.7	67.8	52.2	816	579	51.0	53.8	81.5	3,039
TraDeS [72]	✘	63.9	69.1	52.7	858	507	50.8	55.2	81.8	3,555
CTracker [73]	✘	57.4	66.6	49.0	759	570	45.2	53.6	81.3	5,529
MENDER	✓	67.1	65.0	53.9	678	648	54.4	53.6	83.4	3,266
5.3 Comparisons with A Baseline Design
Due to the new proposed topic, no current work has the same scope or directly solves our problem. Therefore, we compare our proposed MENDER against a two-stage baseline tracker in Table 5. We use current SOTA methods to develop this approach, i.e., MDETR [36] for the grounded detector, while TrackFormer [4] for the object tracker. It is worth noting that our MENDER relies on direct regression to locate and track the object of interest, without the need for an explicit grounded object detection stage. Table 5 shows our proposed MENDER outperforms the baseline on both CA-MOTA and CA-IDF1 metrics in all four settings category synonyms, category definition, tracklet captions and object retrieval (25.7% vs. 21.3%, 16.8% vs. 14.6%, 20.7% vs. 15.3% and 32.9% vs. 25.7% CA-MOTA on TAO), while can maintain up to  $4 \times$  run-time speed (10.3 FPS vs 2.2 FPS). The results indicate that training a single-stage network enhances efficiency and reduces errors by avoiding separate feature extractions for both detection and tracking steps.
5.4 Comparisons with State-of-the-Art Approaches
The category name nm setting is also the official MOT benchmark. Table 6 is the comparison of our result on the category name setting on the official leaderboard of MOT17, compared with other state-of-the-art approaches, including ByteTrack [69] and TrackFormer [4]. Note that our proposed MENDER is one of the first attempts at the Grounded MOT task, not to achieve the top rankings on the general MOT leaderboard. In contrast, other SOTA approaches benefit from the efficient single-category design in their separate object detectors, while our single-stage design is agnostic to the category and for flexible textual input. Compared to TrackFormer [4], our proposed MENDER only demonstrates a marginal decrease in identity assignment (67.1% vs 68.0% CA-IDF1). The decrease in the CA-MOTA stems from our detector's design which integrates flexible input.
We have presented a novel problem of Type-to-Track, which aims to track objects using natural language descriptions instead of bounding boxes or categories, and a large-scale dataset to advance this task. Our proposed MENDER model reduces the computational complexity of third-order correlations by designing an efficient attention method that scales quadratically w.r.t the input sizes. Our experiments on three datasets and five scenarios demonstrate that our model achieves state-of-the-art accuracy and speed for class-agnostic tracking.
Limitations. While our proposed metrics effectively evaluate the proposed Type-to-Track problem, they may not be ideal for measuring precision-recall characteristics in retrieval tasks. Additionally, the lack of the question-answering task in data and problem formulation may limit the algorithm to not being able to provide language feedback such as clarification or alternative suggestions. Additional benchmarks incorporating question-answering are excellent research avenues for future work. While the performance of our proposed MENDER may not be optimal for well-defined categories, it paves the way for exploring new avenues in open vocabulary and open-world scenarios [74].
Broader Impacts. The Type-to-Track problem and the proposed MENDER model have the potential to impact various fields, such as surveillance and robotics, where recognizing object interactions is a crucial task. By reformulating the problem with text support, the proposed methodology can improve the intuitiveness and responsiveness of tracking, making it more practical for video input support in large-language models [75] and real-world applications similar to ChatGPT. However, it could bring potential negative impacts related to human rights by providing a video retrieval system via text.
Acknowledgment. This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), and Google Initiated Research Grant. We also thank Utsav Prabhu and Chi-Nhan Duong for their invaluable discussions and suggestions and acknowledge the Arkansas High-Performance Computing Center for providing GPUs.
[71] Xingyi Zhou, Vladlen Koltun, and Philipp Krahenbuhl. Tracking objects as points. In European Conference on Computer Vision, pages 474-490. Springer, 2020. 10
[72] Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12352-12361, 2021. 10
[73] Jinlong Peng, Changan Wang, Fangbin Wan, Yang Wu, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 145-161. Springer, 2020. 10
[74] Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open-vocabulary multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5567-5577, 2023. 10
[75] OpenAI. Gpt-4 technical report. arXiv, 2023. 10