Type-to-Track: Retrieve Any Object via Prompt-based Tracking
Authors: Pha Nguyen
Kha Gia Quach
Kris M. Kitani
Khoa Luu
Venue: NeurIPS 2023 poster (2023)
Keywords: Grounded Object Tracking
Multiple Object Tracking
Vision Language

Abstract: One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions. We present a new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework (MENDER) using the third-order tensor decomposition. The experiments in five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7\% accuracy and $4\times$ speed faster.

Type-to-Track: Retrieve Any Object via Prompt-based Tracking
Pha Nguyen', Kha Gia Quach', Kris Kitani Khoa Luu'
CVI Lab, University of Arkansas pdActive Inc.
Robotics Institute, Carnegie Mellon University "paanguyen khoaluu)@uark. edu "kquach@ieee. "kitaai cmu edu
uark-cviu uark-cviu.githii github. io/ tooType-to-Traae
Feel and back all people
szze
Fellow men suit
Fecss rean the aditios
cup
(a) Appearance on fine grained scale
(b) and on a class agnostic scale.
Figure 1: An example of the responsive Type-to-Track. The user provides video sequence and prompting request. During tracking, the system able to discriminate appearance attributes track the target subjects accordingly and iteratively responds to the user's tracking request. Each box color represents a unique identity.
Abstract
One of the recent trends in vision problems is to use natural language captions describe the objects of interest. This approach overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces novel paradigm for Multiple Object Tracking called Type-to- Track, which allows users to track objects in videos by typing natural language descriptions. We present new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models transformer-bant based third-order tensor decomposition. The experiments five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7% accuracy and 4x speed faster.
framework (MENDER) using the
Introduction
Tracking the movement of objects in videos is challenging task that has received significant attention in recent years. Various methods have heen proposed to tackle this problem, including deep learning techniques. However, despite these advances, there is still room for improvement in intuitiveness and responsiveness. One potential way to improve object tracking in videos is to incorporate user input into the tracking process. Traditional Visual Object Tracking (VOT) methods typically require
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Comparison of current datasets. # denotes the number of the corresponding item. Bold numbers the
best number in each sub-block, while highlighted numbers the best across all sub-blocks.
Datasets
Task
NLP
#Videos
#Frames
#Tracks
#AnnBoxes
#Words
#Settings
OTB100 [8]
SOT
100
59K
100
59K
VOT-2017 [9]
SOT
60
21K
60
21K
GOT-10k [10]
SOT
10K
1.5M
10K
1.5M
TrackingNet [11]
SOT
X
30K
14,43M
30K
14.43M
MOT17 [12]
MOT
14
11.2K
1.3K
0.3M
TAO [13]
MOT
1.5K
2.2M
8.1K
0.17M
MOT20 [14]
MOT
8
13.41K
3,83K
2.1M
BDD100K [15]
MOT
2K
318K
130.6K
3.3M
LaSOT [6]
SOT
1.4K
3,52M
1.4K
3.52M
9.8K
TNL2K [7]
SOT
2K
1.24M
2K
1.24M
10.8K
Ref-DAVIS [16]
VOS
150
94K
400+
10.3K
Refer-YTVOS [17]
VOS
4K
1.24M
7.4K
131K
158K
Ref-KITTI [18]
MOT
18
6.65K
3.7K
GroOT (Ours)
MOT
1,515
2.25M
13.3K
2.57M
256K
5
users to manually select objects in the video by points [1]. bounding boxes [2, 3], or trained object detectors [4, 5]. Thus, in this paper, we introduce new paradigm, called Type-to-Track, to this task that combines responsive typing input to guide the tracking of objects in videos. It allows for more intuitive and conversational tracking, as users can simply type the name or description of the object they wish to track, as illustrated in Fig. 1. Our intuitive and user- friendliryy Type-to- Track approach has numerous potential applications, such as surveillance and object retrieval videos.
We present a new Grounded Multiple Object Tracking dataset named GroOT that is more advanced than existing tracking datasets [6, 7]. GroOf' contains videos with various types of multiple objects and detailed textual descriptions. It is 2x larger and more diverse than any existing datasets, and it can construct many different evaluation settings. In addition three easy-to-construct experimental settings, we propose two new settings for prompt-based visual tracking. It brings the total number of settings to five, which will be presented in Section 5. These new experimental settings challenge existing designs and highlight the potential for further advancements in our proposed research topic.
In summary, this work addresses the use of natural language to guide and assist the Multiple Object Tracking (MOT) tasks with the following contributions. First, a novel paradigm named Type-to-Track is proposed, which involves responsive and conversational typing to track any objects in videos. Second, a new GroOT dataset is introduced. It contains videos with various types of objects and their corresponding textual descriptions of 256K words describing definition, appearance, and action. Next, two new evaluation protocols that are tracking by retrieval prompts and caption prompts, and three class-agnostic tracking metrics are formulated for this problem. Finally, new transformer-based Subeed-ENcoDE.
framework (MENDER) is introduced with third-order tensor decomposition as the first efficient approach for this task. Our contributions in this paper include novel paradigm, rich semantic dataset, an efficient methodology, and challenging benchmarking protocols with new evaluation metrics. These contributions will be advantageous for the field of Grounded MOT by providing valuable foundation for the development of future algorithms.
2 Related Work
2.1 Visual Object Tracking Datasets and Benchmarks
Datasets. To develop and train VOT models for the computer vision task of tracking objects in videos, various datasets have been created and widely used. Some of the most popular datasets for VOT are OTB [19, 8]. VOT [9], GOT [10], MOT challenges [12, 14] and BDD100K [1S]. Visual object tracking has two sub-tasks: Single Object Tracking (SOT) and Multiple Object Tracking (MOT). Table shows that there is a wide variety of object tracking datasets in both types available, each with its own strengths and weaknesses. Existing datasets with NLP [6, 7] only support the SOT task, while GroOT dataset supports MOT with approximately 2x larger in description size.
Benchmarks. Current benchmarks for tracking can be broadly classified into two main categories: Tracking by Bounding Box and Tracking by Natural Language, depending the type of initialization.

---

Table 3: Statistics of GroOT's sellings.
Table 2: Comparison of key features of tracking methods. Cls-agu is for class-agnostic, while Feat is for the approach of feature fusion and Stages indicates the number of stages in the model design incorporating NLP into the tracking task. NLP indicates how text is utilized for the tracker: assist (w/ box) or can initialize (w/o box).
Dotasets
rVidens
#Fruttes
FTracks
#Words
Parts
Truis
;
5,36
546'
112,29
37792
(1)
MOTIT
Yost
5,919
7851
188,076'
5,757
(7)
Total
14
11,355
1,531
300.373
0,54p
Tran
764.526
2.655
(b
TAO
2,321,346
7.972
154,50
(4)
Total
2,407
4,467,058
16,080
332,401
56.371
Tram
8.911
2,331
1.376
(5)
Approach
Task
NLP
Cls-agn
Stages
MOT20
Test
4,479
1,503"
765.465
(6)
(27)
SCT
asaist
x
concat single uttn
Total
13,410
3,33"
2,502.355
TransVLT [28]
SOT
assist
X
single
1,555
1315
2,249,857
2,340.335
13.284
2,500.00
2,570.50
21,424
$3,540
aV
TrackFormer [4]
MOT
GroUT
def
1.515
2,205.837 837
13.294
2.571.500
99.218
o7
cup
1.307
2,236 427
9.661
458.124
67.920
volo MO720
MDETR-TTm
MOT
init
/
attn attn
two
retr
995
1,000000
1.99
139955
aoes (4)
TrasRMOT [18]
MOT
init
V
two single
alf uses 2, 4, 5,
w/o MOT20 uses (1. 3, 4).
MENDER
MOT
init
Statisties from the official site, including ohjects other than human.
Creative Commons Attribution- NonCommercial www.cnial--Saaaralliic.. 3.0 License
Previous benchmarks [20, 19, 8, 9, 21, 22, 22, 23] were limited to test videos before the emergence
of deep trackers. The first publicly available benchmarks for visual tracking were OTB-2013 [19] and OTB-2015 [8], consisting of 50 and 100 video sequences, respectively. GOT-1OK [10] is
benchmark featuring 10K videos classified into 563 classes and 87 motions. TrackingNet [11], a
subset of the object detection benchmark YT-BB [24], includes 31K sequences. Furthermore, there
are long-term tracking benchmarks such as OxUvA [25] and LaSOT [6]. OxUvA spans 14 hours of video in 337 videos, comprising 366 object tracks. On the other hand, LaSOT [6] is language- assisted dataset consisting of 1.4K sequences with 9.8K words in their captions. In addition to these benchmarks, TNL.2K [7] includes 2K video sequences for natural sanguage-based tracking and focuses on expressing the attributes. LaSOT [6] and TNI.2K [7] support one benchmarking setting with their provided prompts, while our GmOT dataset supports five settings. Ref-KITTI [18) is built upon the KITTI [26] dataset and contains only two categories, including car and pedestrian while our GroOT dataset focuses on -agnoorya tracking, and outnumbers the frames and settings.
A similar task with a different nomenclature to the Grounded MOT task is Referring Video Object Segmentation TRef-VOS) [16, 17]. which primarily measures the overlapping area between the ground truth and prediction for single foreground object in each caption, with less emphasis on densely tracking multiple objects over time. In contrast, our proposed Type-to- -Tackk paradigm is distinct in its focus on responsively and conversationally typing to track any objects in videos, requiring maintaining the temporal motions of multiple objects of interest.
2.2 Grounded Object Tracking
Grounded Vision-Language Models accurately map language concepts onto visual observations by understanding both vision content and natural language. For instance, visual grounding [29] seeks to identify the location of nouns or short phrases (such as a black hat or blue bird) within an image. Grounded captioning [30, 31, 32] can generate text descriptions and align predicted words with object regions in an image. Visual dialog [33] enables meaningful dialogues with humans about visual content using natural, conversational language. Some visual dialog systems may incorporate referring expression recognition [34] to resolve expressions in questions or answers.
Grounded Single Object Tracking is limited to tracking a single object with nox-initialized and language-assisee methods. The GTI [27] framework decomposes the tracking by language task into three sub-tasks: Grounding, Tracking, and Integration, and generates tubelet predictions frame-by- frame. AdaSwitcher [7] module identifies tracking failure and switches to visual grounding for better tracking. [35] introduce unified system using attention memory and cross-attention modules with learnable semantic prototypes. Another transformer based approach [28] is presented including a cross-modal fusion module, task-specific heads, and a proxy token-guided fusion module.
2.3 Discussion
Most existing datasets and benchmarks for object tracking are limited in their coverage and diversity of language and visual concepts. Additionally, the prompts in the existing Grounded SOT benchmarks do not contain variations in covering many objects a single prompt, which limits the application of existing trackers in practical scenarios. To address this, we present new dataset and benchmarking
3

---

the street
person drined shirt
black S
shirt
paverant
man
pit persor the streeth
pers
sue
sonson
Wonan
Fromt
we
standing walking
standing
Kingsins idewaa lk
(a) Our MOT17 subset sample with captions both action and appearance types.
white
(a)
MOT17 [12]
subset.
Call
person
LAI
shirt
wearn
O
ing
OO
wome
CAGLE BIA
Landang
a Kblue
(b) Our TAO [13] subset.
(b) Our TAO [13] subsct samples with captions. Best viewed in color and zoom in. Figure 2: Example sequences and annotations in our dataset.
Figure 3: Some words in our language description.
metrics to support the emerging trend of the Grounded MOT, where the goal is to align language descriptions with une-grained regions or objects videos.
As shown in Table 2, most of the recent methods for the Grounded SOT task are not class-agnostic, meaning they require prior knowledge of the object. GTI [27] and TrT [28] need input the initial bounding box, while TrackFormer [4] need the pre-defined category. The operation used in [27] to fuse visual and textual features concatenation which can only support prompts describing single object. A Grounded MOT can be constructed by integrating a grounded object detector, i.e. MDETR [36], and an object tracker, i.e. Frrmerormer [4]. However, this approach is low-efficient because the visual features have to be extracted multiple times. In contrast, our proposed MOT approach MENDER formulates third-order attention to adaptively focus on many targets, and it an efficient single-stage and class-agnostic framework. The scope of class-agnostic in our approach is constructing a large vocabulary of concepts via a visual-textual corpus, following [37, 38, 39].
3 Dataset Overview
3.1 Data Collection and Annotation
Existing object tracking datasets are typically designed for specific types of video scenes [40, 41, 42, 43, 44, 2.. To cover a diverse range of scenes, GroOT was created using official videos and bounding box annotations from the MOT17 [12]. TAO [13], and MOT20 [14]. The MOT17 dataset comprises 14 sequences with diverse environmental conditions such as crowded scenes, varying viewpoints, and camera motion. The TAO dataset is composed of videos from seven different datasets, such as the Argo Verse [45] and BDD [15] datasets containing outdoor driving scenes, while LaSOT [6] and YFCC100M [46] datasets include in-the-will thle internet videos. Additionally, the AVA [47]. Charades [48], and HACS [49] datasets include videos depicting numan-human and human-object interactions. By combining these datasets, GroOT covers multiple types of scenes and encompasses wide range 833 objects. This diversity allows for a wide range object classes with captions to be included, making it an invaluable resource for training and evaluating visual grounding algorithms.
We release our textual description annotations in COCO format [50]. Specifically, a new key 'captions' which is list of strings is attached to each 'annotations" item in the official annotation. In the MOT17 subset, we attempt maintain two types of caption for well-visible objects: one describes the appearance and the other describes the action. For example, the caption for a well-visible person might be [ man wearing gray shirt', 'person walking on the street'] '] as shown in Fig. 2a. However, 10% of tracklets only have one caption type, and 3% do not have any captions due to their low visibility. The physical characteristics of a person or their personal accessories, such as their clothing, bag color, and hair color are considered to be part of their appearance. Therefore, the appearance captions include verbs 'carrying' or 'holding' to describe personal accessories. In the TAO subset, objects other than humans have caption

---

describing appearance, for instance, I'a red and black scooter']. Objects that are human have the same two types of captions as the MOT17 subset. An example is shown Fig. 2b. These captions are consistently annotated throughout the tracklets. Fig. 3 is the word-cloud visualization of our annotations.
3.2 Type-to-Track Benchmarking Protocols
Let V be video sample lasts t frames, where V I 1 IV]] [
and I be the image sample at particular time step t. We define a request prompt that describes the objects of interest, and T, is the set of tracklets of interest up to time step f. The Type-to-Track paradigm requires tracker network TI, 1, that efficiently take into account L, T and to produce T, = T(,, T,- P). To advance the task of multiple object retrieval, another benchmarking set is created in addition to the GroOT dataset. While training and testing sets follow One-to- to-One scenario, where each caption describes single tracklet, the new retrieval set contains prompts that follow One-to Many scenario, where a short prompt describes multiple objects. This scenario highlights the need for diverse methods to improve the task of multiple object retrieval. The retrieval set provided with subset of tracklets the TAO validation set and three custom retrieval prompts that change throughout the
tracking process in video =o Pt.. Jt, as depicted Fig. I(a). The retrieval prompts are generated through a semi- automaaiti process that involves: (i) selecting the most commonly occurring category in the video, and (ii) cascadingly filtering to the object that appears for the longest duration. In contrast, the caption prompts created by joining tracklet captions in the scene and keeping consistent throughout the tracking period. We name these two evaluation scenarios as trackler captions cap and object retrieval retr With three more easy-to-construct scenarios, five scenarios in total will be studied for the experiments in Section 5. Table presents the statistics of the five settings, and the data portions are highlighted in the corresponding colors.
3.3 Class-agnostic Evaluation Metrics
As indicated in [51], long-tailed classification a very challenging task in imbalanced and large-scale datasets such as TAO. This is because it is difficult to distinguish between similar fine- -me-inni classes, such as bus and van, due the class hierarchy. Additionally, is even more challenging to treat every class independently. The traditional method of evaluating tracking performance leads to inadequate benchmarking and undesired tracking results. In our Type-to-Tree -rackk paradigm, the main task is not to classify objects to their correct categories but to retrieve and track the object of interest. Therefore, to alleviate the negative effect, we reformulate the original per-category metrics of MOTA [52], IDFI [53]. HOTA [54] into class-agnostic metries:
MOTA
1
CLS"
Σ
(1- E. (FN, + FP, + IDS.)
CA-MOTA
E. (FN, FP, IDS.)
CCLS
Σ, GT,
dls
E. (GTCLs))
(1)
CL8"
1
2 x IDTP
(2 IDTP)
IDF1 =
ICLS"
2 x IDTP + IDFP + IDEN
CA-IDFI
= x2x IDTP + IDFP + IDFN)
cla
(2)
CLS
HOTA
DetA ASSA
CA-HOTA
V (DetAcLst) (AssACusI)
(3)
CLSS
where CLS" is the category, set size is reduced to 1 by combining all elements: CLS" + CLS!.
4 Methodology
4.1 Problem Formulation
Given the image I, and the request prompt P describing the objects of interest, which can adaptively change between (Pr,- Prz. Ptg] in the retr setting, and K is the prompt's length [P| K, let enc(-) and emb(-) be the visual encoder and the word embedding model extract features of image tokens and prompt tokens, respectively. The resulting outputs, enc(I:) E RMX and
5

---

emb(P) E RKXD, where D is the length of feature dimensions. A list of region-prompt prompt associations C₁. which contains objects' bounding boxes and their confident scores, can be produced by Eqn. (4):
C₂ === dec -=eecencte (enc(T.))em((P))) (ene()))))) {c=
(Cz, Cuy Cwy Cha Ceon ) < M).
(4)
where (x) is an operation representing the pegion-prompt correlation, that will be elaborated in the next section, dec(';) +) is an object decoder taking the similarity and the image features to decode to 7 object locations, thresholded by a scoring parameter y (i.e. Geonf 2 7). For simplicity, the cardinality of the of objects |Crl M, implying each image token produces one region-text correlation.
{ try = (Irz, bry, www tra, frooff trid)j 1j < <N}, produced by the tracker T, where N = [Tyl is the cardinality of current tracklets. j, k, and i are consistently denoted as indexers for objects, tracklets, prompt tokens, and time steps for the rest of the paper.
We define T, =
Remark 1 Third-order Tensor Modeling. Since the Type-to- Sxpe-to-Track paradigm requires three input com- ponents L. T,-1, and P, an auto-regressive single-stage end-to-end framework can be formulated via third-order tensor modeling.
To achieve this objective, a combination of initialization, object decoding, visual encoding, feature extraction, word embedding, and aggregation can be formulated as in Eqn. (5):
(imitialize(C) (Ct) (imii
t= 0
dec LnxDXD X1 enc(L.) eT (nc(()
1L 0
(5)
where ezz(.) denotes the visual feature extractor of the set of tracklets, eT-( RNX,, DDDDD is an all-ones tensor has size D x D x D, ( (x, ) is the n-mode product of the third- order tensor [55] to aggregate many types of token', and initialize()) is the function to ascendingly assign unique identities to tracklets for the first time those tracklets appear.
Let T E R be the resulting tensor T = DxDxD X1 enc(I,) X2 cet((,,)) X3 emb(P). The objective function can be expressed as the log softmax of the positive region-trackket-promp triplet over all possible triplets, defined in Eqn. (6):
exp (Tss)) EM exp(Tium))
Cenc.exteem arg
log
(6)
N
where 0 denotes the network's 's parameters, the combination of the hth image token, the j tracklet, and the 4thh prompt token is the correlated triplet.
In the next subsection, we claborate our model design for the tracking function T(L, T,-1, P). named MENDER, as defined in Eqn. (5), and loss functions for the problem objective in Eqn. (6).
4.2 MENDER for Multiple Object Tracking by Prompts
The correlation in Eqn. (5) has the cubic time and space complexity Onz3), which can be intractable as the input length grows and hinder the model scalability.
Remark 2 Correlation Simplification. Since both ene(-) and ext(.) are visual encoders, the region-prompt correlation can be equivalent the tracklet-prompt correlation. Therefore, the tegioon-rruckle--romm correlation lensor T can be simplified to lower the computation foolprint.
To design that goal, the extractor and encoder share network weights for computational efficiency:
ext ((rr))-1) = {(nn((L-1)s: e, tr, therefore ((T,)))- = (Tā:)) - try² (7)
where Ty: and Ta: are lateral and horizontal slices. In layman's 's terms, the region- eegiontprompt correlation at the time step 1- 1 is equivalent to the tracklet-promp correlation at the time step I, as visualized in Fig. 4(a). Therefore, one practically needs to model the region-tracklet and tracklet-prompt
implemented by single Python code with Numpy: mp. einaum (aii, bj. ck abc', P, I, T). 2 I changes, the equivalence still holds true, see Appendix for the full algorithm.

---

trccktt condictee
Spatial Puriloud Emteddinng
equiilest conmetions
Estoder
Temponall Poont Embeeding
Follow suit
shared
X
DETR Greake
Transformer Decoder
Decoder
Tr-1
cmoo P)
Wool unbedding
"Follow men suit"
X2
X
P
MENDER
T, visual features and word embedding extract textual features. model the tracklet-promp correlation ext(TT, 1) exmm(()) insteud of region-prr prompt avoid unpcessary computation caused ao-object
(b) The structure our proposed MENDER. employs a visual backhome to
(a) Because the tracklet set T pools visual features image It-1. pooppppoppt equivalent with tracklet-prompt (only need filter unassigned objects).
takens [56]. Best viewed in color and zoom in.
Figure The auto-regressive manner takes advantage of equivalent components. Simplifying correlation (a) turns the solution to MENDER in (b), and reduces complexity to (nz²) where n denotes the size tokens.
correlations which reduces time and space complexity from Onn²) to O(n)), significantly lowering computation footprint. We alternatively rewrite the decoding step in Eqn. (5) as follows:
T. dec (cnn)I))ccc((T-.)) x (esd()=)))ennd())) enc(I.))
AL 0
Correlation Representations. our approach, the correlation operation (x) is modelled by the multi-head cross tross-attention mechanism [57], depicted in Fig. 4(b). The attention matrix can be computed as:
(a(X) W x
o (X) = AxiY softmax
(9)
VD
where X and tokens are one of these types: region, tracklet, prompt. (-)) one of the operations enc(-), emb(-), ezr(.) as the corresponding operation to X Y. Superscript We. WK. and Wy are the projection matrices corresponding to X or Y as in the attention mechanism.
Then, the attention weight from the image I, to the prompt P are computed by the matrix multiplica- tion for AIT and Arip to aggregate the information from two matrices as in Eqn. (8). The result the matrix AςTxTTI ArT x Ari that shows the correlation between each input output. Then, the resulting attention matrix AFTxTI is used to produce the object representations at time t:
Z. = AITTT (emb(P) x w+) + AITT x (eat(T==1) x W
(10)
Object Decoder dec(-) utilizes context-award features Z, that are capable of preserving identity information while adapting to changes position. The tracklet set T, defined in the auto-regressive manner to adjust to the movements of the object being tracked in Eqn. (8). For decoding the final output at any frame, the decoder transforms the object representation by a 3-layer FFN to predict bounding boxes and confidence scores for frame
{ [tr, (tra, try; br trh, tr
FFN + enc(I.))
(11)
where the identification information of tracklets, represented by Iria, is not determined directly by the FFN model. Instead, the tris value set when the tracklet is first initialized and maintained till its end, similar tracking by-attention approaches [4, 58, 59, 60].

---

4.3 Training Losses
To achieve the training objective function as in Eqn. (6), we formulate the objective function into two loss functions LITT and LTIP for correlation training and one loss LGI for decoder training:
L = Triel + " + NGIoU L.GIOU
(12)
where PTJP, T1/T, and TGr are corresponding coefficients, which are set 0.3 by default.
Alignment Loss LTIP is a contrastive loss, which is used to assure the alignment of the ground ground-truth truth object feature and caption pairs (T, P) which can be obtained in our dataset. There are two alignment losses used, one for all objects normalized by the number of positive prompt tokens and the other for all prompt tokens normalized by the number of positive objects. The total loss can be expressed as:
LTP
exp
ezr( x emb(P)s)
exp
[emb(P) x ext(T)))
log
Σ
log
P
exp
cxx((T)] emb(P)))
T
exp
(emb(P)) eze((T)))
(13)
where P+ and I+ are the sets of positive prompts and image tokens corresponding to the selected enc(T), and emb(P)k, respectively.
Objectness Losses. To model the track's temporal changes, our network learns from training samples that capture both appearance and motion generated by two adjacent frames:
exp (e))(T)] x enc(T).)
L1TT = - - log
and LGIOU Σ (aro obj,) (14)
Σ> exp (ert(T)] x eno(1).)
LIfT the log-softmax loss to guide the tokens' alignment as similar Eqn. (13). In the LGIØU loss, obj, is the ground truth object corresponding to try. The optimal assignment between tr, or obj; the ground truth object is computed efficiently by the Hungarian algorithm, following DETR [56].
fGIoU is the Generalized IoU loss [61].
5 Experimental Results
5.1 Implementation Details
Experimental Scenarios. We create three types of prompt: category name category synonyms syn category definition def One tracklet captions cap scenario is constructed by our detailed annotations and one more objects retrieval retr scenario is given in our custom request prompts as described in Subsec. 3.2. The dataset contains 833 classes, each has name and corresponding set of synonyms that are different names for the same category, such as [man, woman, human, pedestr trritarnnn inn, boy, girl, child] for person. Additionally, each category is described by category definition sentence. This definition makes the model deal with the variations the text prompts. We join the names, synonyms, definitions. captions and filter duplicates to construct the prompt. Trained models use as the same type as testing. We annotated the raw tracking data of the best best-performant tracker (i.e., BoT-SORT [62] 80.5% MOTA and 80.2% IDF1) the time we constructed experiments and used it as the sub-optimal ground truth of MOT17 and MOT20 (parts (2, 4) in Table 3). That is also the raw data we used evaluate all our ablation studies.
Datasets and Metrics. RefCOCO+ [63] and Flickr30k [64] serve as pre-trained datasets for acquiring
vocabulary of visual-textual concepts [37]. The ext(-) operation is not involved in this training step. After obtaining a pre-trained model from RefCOCO+ and Flickr30k, we train and evaluate model for the proposed Sope-to-Track task on all five scenarios
GroOT dataset and the
first-three scenarios for MOT20 [14]. The tracking performance is reported in alass-agnostic metrics CA-MOTA, CA-IDF1, and CA-HOTA in Subsec. 3.3 and mAP50 defined in [13].
Tokens Production. emb(.) utilizes RoBERTa [65] to convert the text input into a sequence of numerical tokens. The tokens are fed into the RoBERTa-bee model for text encoding using

---

Table 4: Ablation studies. sim indicates whether the correlation is the simplified Eqn. (8) or the Eqn. (5). See 5.1 for the abbreviations. The two first settings get only one word for the request prompt. therefore, tensor T is an unsqueezed matrix, resulting in no difference in nm (X) vs
Table 5: Comparisons to the two-stage baseline design. In each dataset, the from-top-to-botten tottot scenarios syn def cap
and
retr Best viewed in color.
Approach
CA-MOTA
CA-IDFI
MT
mAP
FPS
GroOT
- MOT17 Subset
(V), and syn (X) vS (
MDETR+ + TFm
62.60
64.70
519
1382
0.793
2.2
P stm
MOTA
CA
IDFI
MT
IDs
MAP
FPS
MENDER
65.10
554
1348
0.874
10.3
GroUT
MOT17
Subset
MDETR. TPm MENDER
62.60
64.70
519
1382
0.793
2.2
nm
RV
67.00
71.30
544
1352
0.876
10.3
67.30
72.40
568
1322
0.877
19m
XV
65.10
71.10
554
1348
0.874
10.3
MDETR TFm
44,80
45.20
193
1945
0.619
2.1
def
67.00
72.10
556
1343
0.876
5.8
MENDER
59,50
$4.80
201
1734
0.688
7.8
07.30
72.40
568
1322
0.877
10.3
GroOT
Subset
cap
58.20
53.30
289
1751
0.674
3.4
59.50
54,80
201
1734
0.688
7.8
MDETR TFm
21.30
33.20
2945
5834
0.184
3.1
GniOT
TAO
Subset
MENDER
25.70
36.10
3212
5048
0.198
11.2
27.30
37.20
3523
4284
0.212
11.2
MDETR+ MENDER
14.60
1944
6493
0.137
3.1
25.70
36.10
3212
5048
0.196
11.2
16,80
27.70
2547
6118
0.158
10.5
15.20
2452
6253
0.154
MDETR TFm MENDER
15,30
23.60
2132
6354
0.156
3.0
20.70
32.00
3103
5192
0.182
8.7
cap
20.30
20.70
3103
5192
MIDTR + 1FFm
25.70
26.40
513
3993
0,387
3.1
MENDER
32.90
0.430
retr
3238
0.423
32,90
39.30
3194
0.430
GroOT
MO120 Subset
GroOT
MOT20
Subset
MDETR TFm
61.20
60.40
784
2824
0.732
1.9
am
XV/
12:40
67.50
823
2498
0.826
7.6
MENDER
70.90
65.30
809
2509
0.823
7.6
XV
70.90
65.30
809
2509
0.825
7.6
MDETR TFm
68.00
66. 30
763
2975
0.783
1.9
def 72.90
67.70
823
2489
0.826
4.3
MENDER
72.10
67.10
812
2503
0.825
7.6
72.10
67.10
813
2503
0.825
7.6
12-layer transformer network with 768 hidden units and 12 self-attention heads per layer. enc(.) is implemented using 101 [66] as the backbone to extract visual features from the input image. The output of the ResNet is processed by a Deformable DETR encoder [67] to generate visual tokens. For each dimension, we sine and cosine functions with different frequencies as positional encodings, similar to [68]. A feature resizer combining a list of (Linear, LayerNorm, Dropout) is used map to size D = 512 for all token producers.
5.2 Ablation Study
Comparisons in Different Scenarios. Table 4 shows comparisons in the performance of different prompt inputs. For MOT17 and MOT20, the category name is 'person', 2 while category definition is human being'. Since the prompt by category definition short, it does not differ much from the nm setting. However, the syn setting shuffles between some words, resulting in a slight decrease in CA-MOTA and CA-IDFI. The cap setting results in prompts that contain more diverse and complex vocabulary, and more context-specific information. It is more difficult for the model to accurately localize the objects and identify their identity within the image, as it needs to take into account wider range of linguistic cues, resulting in decrease in performance compared to def (59.5% CA-MOTA and 54.8% CA-IDFI vs 67.3% CA-MOTA and 72.4% CA-IDFI on MOT17).
For TAO, the def setting has a significant number of variations and many tenuous connections in the scene context, for example, 'an aircraft that has fixed wing and is powered by propellers or jets' for the airplane category. Therefore, it results in a decrease in perfor- mance (16.8% CA-MOTA and 27.7% CA-IDF1) compared to cap (20.7% CA-MOTA and 32.0% CA-IDFI), because the cap setting is more specific on the object level than category level. The best performant setting is nm (27.3% CA-MOTA and 37.2% CA-IDFI), where names are combined.
Simplied Attention Representations, Table 4 also presents the effectiveness of different attention representations of the full tensor 7 (denoted by X) and the simplified correlation (denoted by v/. The performance reported with frame per second (FPS), which is self-measured on one GPU NVIDIA RTX 3060 12GB. Overall, the performance of simplified correlation is witnessed with a superior speed of up to 2x (7.8 FPS vs 3.4 FPS of cap on MOT17 and 1 11.5 FPS vs 7.6 FPS of retr on TAO), resulting in and slight increase in accuracy due to attention stability and precision gain.

---

Table 6: Comparisons to the state-of-the-art approaches on the category name nm setting.
Approach
Cls-gn
CA-IDFI
CA-MOTA
CA-HOTA
MT
ML
AssA
DetA
LocA
IDs
ByteTrack [69]
80.3
957
52.7
55.6
81.8
3,378
TrackFormer [4]
68.0
74.1
57.3
1.113
246
54.1
60.9
82.8
2,829
QuasiDense [70]
66.3
68.7
53.9
957
516
52.7
55.6
81.8
3,378
CenterTrack [71]
64.7
67.8
52.2
816
579
51.0
53.8
81.5
3,039
TraDeS [72]
63.9
69.1
52.7
858
507
50.8
55.2
81.8
3,555
CTracker [73]
57.4
66.6
49.0
759
570
45.2
53.6
81.3
5,529
MENDER
67.1
65.0
53.9
678
54.4
53.6
83.4
3,266
5.3
Comparisons with A Baseline Design
Due to the proposed topic, current work has the same scope directly solves our problem. Therefore, we compare our proposed MENDER against two-stage baseline tracker in Table 5. We use current SOTA methods to develop this approach, i.e., MDETR [36] for the grounded detector, while TrackFormer [4] for the object tracker. It is worth noting that our MENDER relies on direct regression locate and track the object of interest, without the need for an explicit grounded object detection stage. Table 5 shows our proposed MENDER outperforms the baseline on both CA-MOTA and CA-IDFI metrics in all four settings category synonyms. category definition, tracklet captions and object retrieval (25.7% vs. 21.3%. 16.8% vs. 14.6%, 20.7% vs. 15.3% and 32.9% vs. 25.7% CA-MOTA on TAO), while can maintain up to 4x run-time speed (10.3 FPS 2.2 FPS). The results indicate that training a single-stage network enhances efficiency and reduces errors by avoiding separate feature extractions for both detection and tracking steps.
5.4 Comparisons with State-of-the-Art Approaches
The category name nm setting is also the official MOT benchmark. Table 6 is the comparison of our result on the category name setting on the official leaderboard of MOT17, compared with other state-of-the-art approaches, including BrteTrack [69] and TrackFormer [4]. Note that proposed MENDER is one of the first attempts at the Grounded MOT task, not to achieve the top rankings the general MOT leaderboard. In contrast, other SOTA approaches benefit from the efficient single-category design in their separate object detectors, while our single-stage design agnostic to the category and for flexible textual input. Compared to TrackFormer [4], proposed MENDER only demonstrates a marginal decrease in identity assignment (67.1% vs 68.0% CA-IDFI). The decrease in the CA-MOTA stems from our detector's design which integrates flexible input.
Conclusion
We have presented novel problem of Type-to- -Tack,, which aims to track objects using natural language descriptions instead of bounding boxes or categories, and large-scale dataset to advance this task. Our proposed MENDER model reduces the computational complexity of third-ordes correlations by designing an efficient attention method that scales quadratically w.r.t the input sizes. Our experiments three datasets and five scenarios demonstrate that model achieves state-of- the-art accuracy and speed for class-agnostic tracking.
Limitations. While our proposed metrics effectively evaluate the proposed Type-to-Track problem, they may not be ideal for measuring precision-recall characteristics in retrieval tasks. Additionally, the lack of the question -eestion-answering task in data and problem formulation may limit the algorithm to not being able to provide language feodback such as clarification or alternative suggestions. Additional benchmarks incorporating question-answering, answering are excellent research avenues for future work. While the performance our proposed MENDER may not be optimal for well-defined categories, paves the way for exploring new avenues in open vocabulary and open-world scenarios [74].
Broader Impacts. The tone-to-Track problem and the proposed MENDER model have the potential to impact various fields, such as surveillance and robotics, where recognizing object interactions is crucial task. By reformulating the problem with text support, the proposed methodology can improve the intuitiveness and responsiveness of tracking, making it more practical for video input support in large-language models [75] and weal-world applications similar to ChatGPT. However, it could bring potential negative impacts related to human rights by providing video retrieval system via text.
10

---

Acknowledgment. This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), and Google Initiated Research Grant. We also thank Utsav Prabhu and Chi-Nhan Duong for their invaluable discussions and suggestions and acknowledge the Arkansas High- Hiigh-Perforance Computing Center for providing GPUs.
References
[1] Pha Nguyen, Thanh- Dat Truong, Miaoqing Huang. Yi Liang, Ngan Le, and Khoa Luu, Self- Sele-super domain adantation in crowd counting. In 2022 IEEE. International Conference on Image Processing (ICIP),
pages 2786-2790. TEFE, 2022.
[2] Kha Gia Quach, Pha Nguyen, Huu Lc, Thanh-Dat Truong, Chi Nhan Duong, Minh- TrietTrict Tran, and Khoa Luu. Dyglip: A dynamic graph model with link prediction for accurate multi-camera multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13784-13793, 2021. 2,
[3] Kha Gia Quach, Huu Le, Pha Nguyen, Chi Nhan Duong, Tien Dai Bui, and Khoa Luu. Depth perspective- aware multiple object tracking. arXiv preprint arXiv:2207.04551, 2022.. 2022. 2
[4] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8844-8854, 2022. 2, 3, 4, 7, 10
[5] Pha Nguyen, Kha Gia Quach, John Gauch, Samee U Khan, Bhiksha Raj, and Khoa Luu. Utopia: Unconstrained tracking objects without preliminary examination via cross-domain adaptation. arXiv preprint 2rv: 09613, 2023. 2
[6] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu,, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 5374-5383, 2019. 2, 4
[7] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. To- wards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 13763-13773 June 2021. 2,3
[8] Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9): 1834-18884888 2015. 2,
[9] Matej Kristan, Jiri Matas, Aleš Leonardis, Tomas Vojir, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay, Fatih Porikli, and Luka Cehovin. A novel performance evaluation methodology for single-target trackers. IEEE Transactions on Pattern Analysis and Machine Intelligence, (111))221-2215: Nov 2016. 3,3
[10] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-1Ok: A large high-diversitg benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. 2, 3
[11] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), pages 300-317, 2018. 2,3
[12] A. Milan, L. Leal- Taixé, L Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-object tracking. arXiv:1603 00831 [cs]. March 2016. arXiv: 1603.00831. 2, 4
[13] Achal Dave. Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale benchmark for tracking any object. Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 436-454. Springer, 2020. 2, 4,
[14] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Laal-T Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003,0 09003, 2020. 4,8
[15] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangehen Liu, Vashisht Madhavan, and Trevor Darrell, BDDIO0K: A diverse driving dataset for heterogeneous multitask learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020. 2, 4
11

---

[16] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions, In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Pupers, Part IV 14, pages 123-141. Springer, 2019. 2,3
[17] Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 208-223. Springer, 2020. 2,
[18] Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong. Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14633 14642, 2023. 3,
[19] Yi Wu, Jongwoo Lim, and Ming Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pottern Analysis & Machine Intelligence, 37(9):1834, 2015. 2.3
[20] Pengpeng Liang, Erik Blasch, and Haibin Ling. Encoding color information for visual tracking: Algorithms and benehmark. IEEE Transactions on Image Processing, 24(12):5630 064000556444444 2015. 3
[21] A Li, M Lin, Y Wu, MH Yang, and S Yan. NUS-PRO: A New Visual Tracking Challenge. IEEE Transactions on Puttern Analysis and Machine Intelligence, 38(2):335-349 349, 2016.
[22] Siyi and Dit-Yan Yeung. Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. 3
[23] Hamed Kiani Galoogahi, Ashton Fagg, Chen Huang, Deva Ramanan, and Simon Lucey. Need for speed:
A benchmark for higher frame rate object tracking. In Proceedings of the IEEE International Conference on Computer Vision, pages 1125-1134, 2017. 3
[24] Esteban Real, Jonathon Shlens. Stefano Mazzocchi. Xin Pan, and Vincent Vanhoucke. Youtube- boundinghoxes: A large high-precision human-annotated data set for object detection in video. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5296-5305, 2017.
[25] Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Aroold WM Smeulders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A benchmark. In Proceedings of the European Conference on Computer Vision (ECCV), pages 670-685, 2018. 3
[26] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robutics Research, 32(1 11:1111:233123331 2013. 3
[27] Zhengyuan Yang, Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. -rrckiningtrackkngg integration. IEEE Transactions on Circuits and Systems for Video Technology, 31(9):3433 3443, 2020. 3,
[28] Haojie Zhao, Xiao Wang, Dong Wang, Huchuan Lu, and Xiang Ruan. Transformer vasion-language tracking via proxy token guided cross-modal fusion. Pattern Recognition Letters, 2023. 3,4
[29] Fenglin Liu, Xian Wu, Shen Ge, Xuancheng Ren, Wei Fan, Xu Sun, and Yuexian Zou. Dimbert: learning vision-language grounded representations with disentangled colliioddl-aattentinn ACM Transactions on Knowledge Discovery from Data (TKDD), 16(1):1-19, 2021. 3
[30] Wenqiao Zhang, Haochen Shi, Siliang Tang, Jun Xiao, Qiang Yu, and Yueting Zhuang. Consensus graph representation learning for better grounded image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3394-3402, 2021.3
[31] Wenhui Jiang, Minwei Zhu, Yuming Fang, Guangming Shi, Xiaowei Zhao, and Yang Liu. Visual cluster grounding for image captioning. IEEE Transactions Image Processing, 31:3920-3934, 2022. 3
[32] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang- Grit: A generative region-to-ter texext transformer for object understanding. arXiv preprint arXiv:2212.002880 2022. 3
[33] Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and generalization in a 2d world. In International Conference on Learning Representations, 2018. 3
[34] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 787-798, 2014.
12

---

[35] Yihao Li. Jun Yu, Zhongpeng Cai, and Yuwen Pan. Cross-modal target retrieval for tracking by natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4931-4940, 2022. 3
[36] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Modtr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEFF/CVF
International Conference on Computer Vision, pages 1780-1790, 2021. 4, 10
[37] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14403) 2021. 4,8
[38] Muhammad Maaz, Hanoona Bangalath Rasheed, Salman Hameed Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming- ding-Hsuan Yang. Multi- modal transformers excel at class-agnostic object detection. arXiv, 2021. 4
[39] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end lask-agnostic vision-language architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16399-16409, 2022. 4
[40] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Haaiiinn,..unn... and Luc Van Gool. The 2017 DAVIS Challenge on Video Object Segmentation. arXi, 2017. 4
[41] Linjie Yang. Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019.
[42] Ning Xu, Linjie Yang. Yuchen Fan. Dingeheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. YouTube-VOS: Large-Scale Video Object Segmentation Benchmark. arXiv, 2018. 4
[43] Jiyang Qi. Yan Gao. Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benehmark. International Journal of Computer Vision, 130(8):0022-200) 2022. 4
[44] Namdar Homayounfar, Justin Liang, Wei-Chiu Ma, and Raquel Urtasun. Videoclick: Video object segmentation with single click. arXiv preprint XIi1:2101.06555. 2021. 4
[45] Ming-Fane Fang Chang, John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnttt, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, Argoverse: 3d tracking and forccasting with rich maps. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 8748-8757, 2019. 4
[46] Bart Thomee, David A Shamma. Gerald Friedland. Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 73, 2016. 4
[47] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li. Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 6047-6056, 2018.
[48] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part I 14, pages 510-526. Springer, 2016.
[49] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8668-8678, 2019. 4
[50] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECC ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014, 4
[51] Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E Huang, and Fisher Yu. Tracking every thing in the wild. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 498-515. Springer, 2022. 5
[52] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metries. EURASIP Journal Image and Video Processing, 2008:1-10, 2008. 5
13

---

[53] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and data set for multi-target, multi-camera tracking. In European conference on computer vision, pages 17-35. Springer, 2016. 5
[54] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura I Teixe and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision, (29(():54855))) -788 2021. 5
[55] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 510)):555-000
2009. 6
[56] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020. 7.8 8
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 7
[58] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In European Conference on Computer Vision (ECCV), 2022. 7
[59] Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Son Lam Phung, Ngan Le, and Khoa Luu. Multi- camera multi-object tracking on the move via single-stage global association approach. arXiv preprint DoXiv:2211.09663 2022.7 7
[60] Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Ngan Le, Xuan Bac Nguyen, and Khoa Luu, Multi-camera multiple 3d object tracking on the move for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2569-25777 2022. 7
[61] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 8
[62] Nir Aharon. Roy Orfaig, and Ben Zion Bobrovsky. Bot-sort: Robust associations multi- munti-peddsttian tracking. arXiv preprint 2206 14651, 2022. 8
[63] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-4, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016. 8
[64] Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo. Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phen to-ratssc correspondences for richer image-to- sentence models. IJCV, 123(1):74 99, 2017.
[65] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arxi: 1692, 2019. 8
[66] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[67] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable (detr): Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. 9
[68] Yuqing Wang. Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8741-8750) 2021. 9
[69] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference Computer Vision (ECCV), 2022. 10
[70] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 164 1744173,200 2021. 10
14

---

[71] Xingyi Zhou, Vladlen Koltun. and Philipp Krähenbühhl Tracking objects points. In European Conference Computer Vision, pages 474-490. Springer, 2020. 10
[72] Jialian Wu, Jiale Cao, Liangehen Song, Yu Wang, Ming Yang. and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12352-12361, 2021.
[73] Jinlong Peng, Changan Wang, Fangbin Wan, Yang Wu, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Chained-tracker Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part IV 16, pages 145 161. Springer, 2020. 10
[74] Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open- vocabulary multiple object tracking. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5567 2023. 10
[75] OpenAL Gpt-4 technical report. arXiv, 2023. 10
15


Reviews:
[{"review_id": "gZJsCl5hjf", "reviewer": "Reviewer_HDeT", "summary": "\nThe authors have created a new dataset by combining existing Tracking datasets and adding language prompt annotations. They also proposed a method by extending and modifying MDETR and Trackformer. \n", "questions": "1. How is this dataset different from Refer-Youtube-VOS, Ref-DAVIS, and Ref-KITTI? \n2. How did you evaluate MOT17 and MOT20 datasets? The test ground truth is not public for these datasets. What is MOT17 subset in Tab. 4 and 5?\n3. Why is there  no unifying evaluation metric. The authors report evaluation separately on all the different datasets they worked on. A new dataset needs to be either compatible with previous ones, or propose a new metric. Without a single metric, new line of researchers would find it difficult to adapt the dataset.\n4. How many human hours was needed to annotate the dataset?\n\n\n\n\n\n\n", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The dataset quality looks sound, diverse, and sufficiently large enough to train Deep networks.\n2. The dataset is well documented and the task is clearly defined.\n3. The proposed solution is simple and should be easy to extend by new researchers.\n", "weaknesses": "1. The authors claim that Type-to-Track is a novel paradigm. This is incorrect as there exists Datasets, e.g., Refer-Youtube-VOS, Ref-DAVIS, Ref-KITTI, having the same challenge.  \n2. The related work in incomplete. The authors fail to compare their proposed dataset with existing datasets from Video Instance Segmentation(VIS) literature. Note that, MOTS and VIS are the same task with different nomenclature.\n3. The modeling part is unnecessarily written in a complicated way. It is a simple extension of DETR architecture to model the specific solution. Instead of complicating the method with notation, The authors should try to simplify the method section. \n4. There are too many evaluation metrics. There are 5 different sub-categories of the same task with separate evaluations on the 3 base datasets. So there are in total, around 12*6=72 different evaluation metrics. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer **HDeT** for your valuable and thoughtful comments. We are glad that the Reviewer agrees with the positive aspects of our work, including the acknowledgment of the dataset's quality through documentation and the clear definition of the task. Moreover, we are pleased to know that the proposed solution is perceived as simple and easily extendable. We will take all suggestions into careful consideration in our revision.\n\n**Q1:** The authors claim that Type-to-Track is a novel paradigm. This is incorrect as there exists Datasets, e.g., Refer-Youtube-VOS, Ref-DAVIS, Ref-KITTI, having the same challenge.\n\n**A1:** Our proposed Type-to-Track paradigm is distinct in its focus on **responsive and conversational** typing to track **any objects** in videos, requiring **maintaining the temporal motions of multiple objects of interest**. In contrast, the Video Object Segmentation (VOS) challenge **primarily measures the overlapping area between the ground truth and prediction** for **a single foreground object in each caption**, with less emphasis on densely tracking multiple objects over time. And the Ref-KITTI [A] challenge is **not agnostic to the category** and **not able to change the prompt responsively**. Therefore, our statement on the novel paradigm is still valid, as stated in lines 35-36 in the main paper.\n\n**Q2**: The authors fail to compare their proposed dataset with existing datasets from Video Instance Segmentation(VIS) literature.\n\n**A2**: We acknowledge the importance of comparing our proposed dataset with existing datasets from the Referring Video Object Segmentation literature (Refer-Youtube-VOS, Ref-DAVIS), which involves **a single object along with a single referring expression**, in the same category as Single Object Tracking (SOT). The mentioned VIS task from the Reviewer may be a typo that typically deals with multiple objects without considering Referring Expressions. We appreciate your feedback and we added these two datasets to the comparison in Table 1.\n\n**Q3:** The modeling part is unnecessarily written in a complicated way. It is a simple extension of DETR architecture to model the specific solution.\n\n**A3:** We want to emphasize that our method **goes beyond a simple extension and presents an efficient solution to the tracking problem**. Our approach is specifically designed to address **the complex problem of multi-modality**, where we handle three types of input components in a single tracker network. It involves modeling the interactions between video frames, textual prompts, and object features, which is **a unique challenge not addressed by standard DETR**. In our paper, we take great care to present **a clear and detailed formulation of our approach**. We provide **step-by-step explanations** in both mathematical formulas (from Eqn. 2 to Eqn. 3) and visual representations to help readers **understand the motivation and the key components of our method** (Eqn. 5 and Fig. 4a), **enabling the audience can reproduce our algorithm and conduct further research** in this area.\n\n**Q4:** There are too many evaluation metrics.\n\n**A4:** The reason for employing different evaluation metrics is to **ensure a comprehensive assessment** of the proposed method. Our decision to employ a range of evaluation metrics is also driven by **the nature of the problem and the standard protocol for tracking tasks**. As pointed out by Reviewer **RiBQ**, relying solely on metrics like MOTA and IDF1 can not fully capture the performance of a tracker. Similarly, the 5 different sub-categories explore various settings of the prompt, as detailed in our ablation study.\n\n**Q5:** How is this dataset different from Refer-Youtube-VOS, Ref-DAVIS, and Ref-KITTI?\n\n**A5:** Please refer to **KP1**, **KP2** and **A1**. As also indicated by Reviewer **h9ub**, our dataset stands out from previous tracking datasets due to **its diverse range of prompts**.\n\n**Q6:** How did you evaluate MOT17 and MOT20 datasets? What is MOT17 subset in Tab. 4 and 5? \n\n**A6:** The MOT17 subset in Table 4 and Table 5 is evaluated on the sub-optimal test set ground truth. Please refer to **KP3**.\n\n**Q7:** Why is there no unifying evaluation metric.\n\n**A7:** Please refer to **A4**.\n\n**Q8:** How many human hours was needed to annotate the dataset?\n\n**A8:** To ensure high-quality annotations, we engaged a team of ten professional annotators, each dedicating 15 hours per week to the task. The annotation work commenced in January and continued until the end of July. Haft of the time is spent to be cross-checked manually to maintain accuracy and consistency. As of now, the final version of the dataset is under assessment to ensure that it meets all the necessary criteria and adheres to the established quality standards."}, {"review_id": "YT7HZlr56k", "reviewer": "Reviewer_h9ub", "summary": "The paper introduces a novel task and dataset that involves tracking multiple objects in videos using language prompts. It proposes two evaluation protocols: retrieval prompts, where the prompts can change during the tracking process, and caption prompts, which remain consistent. The paper also presents class-agnostic evaluation metrics by modifying MOTA and IDF1. Additionally, a novel framework called MENDER is introduced to address the grounded MOT task.", "questions": "As discussed in the weaknesses, I hope the authors would supplement some dataset comparisons and experimental validations.", "limitations": "Yes, the authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The significance of the proposed task and dataset is well-defined, as existing tracking datasets using language prompts have been focused on single-object tracking (SOT) tasks.\n2. The evaluation protocol for the proposed dataset consists of five distinct protocols, each with clear explanations of their purpose. This dataset stands out from previous tracking datasets due to its diverse range of prompts.\n3. The proposed MENDER framework demonstrates efficient and effective performance in addressing the grounded MOT task.\n4. The paper is overall well-written and easy to follow.", "weaknesses": "1. Referring video segmentation datasets like Ref-YTVOS and Ref-DAVIS are also related to this grounded MOT task since they also aim at tracking a referred target in a video. I think these datasets should also be mentioned in the paper as a comparison benchmark.\n2. Moreover, I think Refer-KITTI [1] could also be discussed in the paper as it also utilizes referring expressions for multi-object tracking.\n3. The experimental comparisons in the newly proposed dataset are limited. It would be beneficial to include additional methods such as TransRMOT [1] and popular MOT baselines like ByteTrack [2] for a more comprehensive evaluation of the dataset's performance. Adding these comparisons would provide stronger validation for the dataset.\n\n[1] Referring Multi-Object Tracking, https://arxiv.org/pdf/2303.03366.pdf\n\n[2] ByteTrack: Multi-object tracking by associating every detection box, ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer **h9ub** for your thoughtful and constructive comments. We are pleased to see that the significance of our proposed task and dataset has been acknowledged. Furthermore, we are glad to receive positive feedback on the efficiency and effectiveness of our proposed MENDER framework. We are grateful for the reviewer's valuable input, and we will carefully consider all suggestions to enhance the quality in the upcoming revision.\n\n**Q9:** Referring video segmentation datasets like Ref-YTVOS and Ref-DAVIS are also related to this grounded MOT task since they also aim at tracking a referred target in a video.\n\n**A9:** Please refer to **KP1** and **A2**. We added them to the comparison in Table 1 in the attachment.\n\n**Q10:** Moreover, I think Refer-KITTI [A] could also be discussed in the paper as it also utilizes referring expressions for multi-object tracking.\n\n**A10:** Please refer to **KP2**. We added them to the comparison in Table 1 in the attachment.\n\n**Q11:** The experimental comparisons in the newly proposed dataset are limited.\n\n**A11:** We appreciate Reviewer **h9ub** for suggesting another comprehensive comparison. We submitted our prediction to the official leaderboard for a convenient and direct comparison with other SOTA approaches on the `category_name` setting. We will try to test SOTA’s performance on other settings. Please refer to **A26**."}, {"review_id": "2FHB2vDniU", "reviewer": "Reviewer_9rR3", "summary": "This paper delves into language-grounded multi-object tracking, a highly important task that has been underexplored due to the lack of suitable datasets.  Firstly, the authors propose a novel dataset that extends existing Multiple Object Tracking (MOT) datasets with additional captions describing appearance or action of each tracklet, as well as retrieval prompts describing main content of a scene. To evaluate language-grounded multi-object tracking, the authors consider five scenarios with different types of text input (prompts) and suggest class-agnostic MOTA and IDF1 that decouple the classification accuracy. Also, the authors introduce MENDER, an efficient single-stage model that autoregressively tracks objects relevant to the prompt. The effectiveness of this model is validated through a comparison with a two-stage baseline model consisting of MDETR and TrackFormer.", "questions": "Major concerns are raised in the weakness section.\n\nHere are some minor questions:\n* Why only text prompt for every frame exists, but there is no prompt-tracklet paired annotations in the MOT files?\n* In Table 3, dataset statistics for MOT17 videos are not correct. As far as I know, there are 7 unique videos for the training and test subsets, respectively. Also, your released dataset seems using the GT tracklets from the unique videos.\n* How do you have the mot17_test annotation file? Is the tracklet obtained from GT or just prediction value?\n* TAO dataset is the most important one as the proposed dataset is built on top of this one, but why it is not included in Table 1?\n* Why do you design the MENDER as online tracking model that runs autoregressively, instead of predicting multiple spatio-temporal tubes at once?\n* In Table 4 and 5, is mAP the absolute value or percentage value?\n* In Table 4, some bolded values are not the best in the metric.", "limitations": "The limitations and future works are well addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* This paper addresses a significant problem, aiming to retrieve multiple objects relevant to an input prompt from a video. This setting is novel compared to existing spatio-temporal video grounding methods (e.g., TubeDETR[1]), which are limited to retrieving only a single object. The proposed task can not only serve as a user-friendly video analysis application but also work as a video perception foundation model for fine-grained video understanding.\n* The proposed dataset is significant to further research on video-language model.\n* The proposed diverse scenarios for evaluating grounded MOT are interesting and valid.\n\n[1] A. Yang, et al., TubeDETR: Spatio-Temporal Video Grounding with Transformers", "weaknesses": "## Inconsistent annotations\n  * In the released dataset, some inconsistent annotations are observed. How quality assurance is achieved in the manual annotation process? Is there criteria to accept or reject the captions from the annotator?\n  * In Sec 3.1, L116 states that the appearance-related caption and action-related caption are provided in order if the tracked object is person. However, in the annotations, the captions are randomly ordered. For example, in mot17_train file, object whose instance_id is 170 has captions of [\"person walking on sidewalk\", \"person wearing black jacket\"], where action-related caption is the first in the caption list. \n  * Also, object whose instance_id is 9 has three captions: [\"a person sitting on a bench\", \"child in stroller\", \"baby stroller\"]. And, in TAO dataset, there are some instances with six captions. What is the reaons of such variance in caption annotations?\n  * I would rather suggest to have separate field for appearance described caption and action described caption.\n\n## Heavy dependence on TAO dataset\n* MOT Challenge datasets include more challenging tracking cases caused by complex motion and occlusion, compared to TAO dataset. However, its scale is very small (e.g., only 7 videos for training). \n* I think captioning other human-centric MOT datasets with complex motion would improve the importance of the dataset. For example, human-centric datasets with dense FPS video (e.g., DanceTrack[2] and HiEve[3]) are the possible candidates.\n\n## Unclear evaluation metrics\n* In the paper, I cannot find the details of matching GT-prediction pairs which are necessary to compute FN and FP.\n* What is the reason of cls exists in Eq. 1 even though we are not considering class of tracklets? Isn't that the class of tracklet do not exist in the case of grounded MOT?\n\n## Lack of experimental details\n* Unclear details about implementing the two-stage baseline model which is based on the MDETR and TrackFormer. TrackFormer jointly performs object detection and tracking, but how MDETR is integrated into TrackFormer to achieve grounded object detection?\n\n[2] P. Sun, et al., DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion, CVPR 2022\n[3] W. Lin, et al., Human in events: A large-scale benchmark for humancentric video analysis in complex events, arxiv 2020.05", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to Reviewer **9rR3** for your thoughtful and valuable feedback. We are pleased to see that the significance of our work has been acknowledged.\n\n**Q12:** How quality assurance is achieved in the manual annotation process? Is there criteria to accept or reject the captions from the annotator?\n\n**A12:** Similar to [D] presented in NeurIPS 2022 Datasets and Benchmarks, during the annotation process, we conduct regular quality reviews by batch to identify any errors. Regarding acceptance or rejection, captions are evaluated based on the cross-checked voting by all other annotators. We provide clear guidelines and instructions to the annotators to choose **the most distinct feature of the targets in the videos**.\n\n**Q13:** What is the reaons of such variance in caption annotations?\n\n**A13:** Note that the version v0.1 we released on Feb 17th, 2023 **long ago before the submission deadline**. Therefore it is just an initial version for testing the annotation procedure and before the cross-checked step. We published a newer version v0.2 on Jul 8th, 2023 (before this rebuttal) which has solved some inconsistent instances. The data demonstration and results are shown at the version v0.6. The first official ready version, v1.0, is currently under assessment to ensure the quality and consistency of the annotations. We ensure the maintenance of the annotations in the long term, and we will cooperate with the MOTChallenge organizers upon this paper’s acceptance and their approval.\n\n**Q14:** I would rather suggest to have separate field for appearance described caption and action described caption.\n\n**A14:** We appreciate your suggestion. We will consider organizing the data format in this way.\n\n**Q15:** Heavy dependence on TAO dataset\n\n**A15:** We acknowledge **the unique characteristics of each dataset**, including the wide range of categories in TAO, the distinct human appearance in MOT17, and the dense population in MOT20. Although the MOT17 and MOT20 have a limited scale, we decided to combine all datasets to **diversify our scenarios**. We appreciate the reviewer’s suggestion on captioning DanceTrack and HiEve datasets. We will consider contributing to these aspects in future work.\n\n**Q16:** In the paper, I cannot find the details of matching GT-prediction pairs which are necessary to compute FN and FP.\n\n**A16:** We followed the original formulas of MOTA and IDF1 for all the components FN, FP, IDS, IDTP, IDFP, and IDFN of the metrics. The score threshold required for a TP match is 0.5, following the TrackEval implementation.\n\n**Q17:** What is the reason of cls exists in Eq. 1 even though we are not considering class of tracklets?\n\n**A17:** Originally, we derived the per-category form of MOTA and IDF1 and their class-agnostic form (i.e. CA-MOTA and CA-IDF1), and included the $cls$ notation to show the difference in using category. However, we had to remove the original forms due to the page limit and added a description to explain the $cls$ notation. In the revision upon acceptance, we will show the derivation of both forms as below:\n\n$\\text{MOTA} = \\frac{1}{|CLS|}\\sum_{cls}^{CLS}\\Big(1 - \\frac{\\sum_{t}{(FN_{t} + FP_{t} + IDS_{t})}}{\\sum_{t}{GT_{t}}}\\Big)_{cls}$\n\n$\\text{CA-MOTA} = 1 - \\frac{\\sum_t\\sum_{cls}{(FN_{t} + FP_{t} + IDS_{t})}}{\\sum_t(\\sum_{cls}{GT_{cls}})_{t}}$\n\n$\\text{IDF1} = \\frac{1}{|CLS|}\\sum_{cls}^{CLS}\\Big(\\frac{2\\times IDTP}{2\\times IDTP + IDFP + IDFN}\\Big)_{cls}$\n\n$\\text{CA-IDF1} = \\frac{\\sum_{cls}{(2\\times IDTP)}}{\\sum_{cls}{(2\\times IDTP + IDFP + IDFN)}}$\n\n**Q18:** Unclear details about implementing the two-stage baseline model which is based on the MDETR and TrackFormer.\n\n**A18:** The TrackFormer’s source code has a mode to receive public detections as input. We generated grounded detections from MDETR and saved them to a compatible format, then imported them to the public detection mode of TrackFormer.\n\n**Q19:** Why only text prompt for every frame exists, but there is no prompt-tracklet paired annotations in the MOT files?\n\n**A19:** We follow the original design of the TAO dataset's annotations for the consistency of our annotation system working on two datasets. That decision was made to maintain compatibility and consistency and ensure simplicity and ease of use, enabling users to leverage a wide range of existing COCO API readers for data analysis and processing.\n\n**Q20:** In Table 3, dataset statistics for MOT17 videos are not correct.\n\n**A20:** Thank you for pointing this out. We corrected Table 3 and also the statistics in Table 1 in the attachment.\n\n**Q21:** How do you have the mot17_test annotation file? Is the tracklet obtained from GT or just prediction value?\n\n**A21:** The tracklet identities are the prediction value. Please see **KP3**.\n\n**Q22:** TAO dataset is the most important one as the proposed dataset is built on top of this one, but why it is not included in Table 1?\n\n**A22:** We did have the TAO dataset’s statistics in Table 1 and Table 3 in the first version, but we removed it in Table 1 due to the page limit. Please refer to the updated Table 1 in the attachment. We will re-add the TAO dataset in the revision upon acceptance.\n\n**Q23:** Why do you design the MENDER as online tracking model that runs autoregressively, instead of predicting multiple spatio-temporal tubes at once?\n\n**A23:** Online is a preferred practice as widely accepted by the community specifically for the tracking problem. We will add this discussion of the online tracking practice compared with the spatiotemporal tube practice defined by TubeDETR.\n\n**Q24:** In Table 4 and 5, is mAP the absolute value or percentage value?\n\n**A24:** It is the absolute value.\n\n**Q25:** In Table 4, some bolded values are not the best in the metric.\n\n**A25:** Thank you for pointing out the typos. We corrected them in Table 4.\n\n[D] Doersch, Carl, et al. \"Tap-vid: A benchmark for tracking any point in a video.\" Advances in Neural Information Processing Systems 35 (2022): 13610-13626."}, {"review_id": "62hNJUBPlK", "reviewer": "Reviewer_RiBQ", "summary": "This paper systematically tackles the prompt-based multiple object tracking problem by (1) introducing a dataset for the Grounding Multiple Object Tracking Task (GroOT), (2) introducing a new tacking paradigm (type to track), and (3) developing a new language-based multiple object tracking model (MENDER).", "questions": "I find the paper well-written and relevant to the community. However, the weaknesses listed above somewhat hinder the contributions of this paper.\n\nIn my opinion:\n1. a comparison on the `category_name` setting to SOTA tracking-by-detection methods on GroOT is essential for this paper\n2. retrieval-specific metrics would complement the contribution of the dataset, which as of now does not allow directly assessing the retrieval task performance\n3. the paper would benefit from ablations on the tokenizer choice.\n\nI would consider increasing my rating and supporting this paper if my current concerns are addressed, in particular points 1 and 2.", "limitations": "The authors have adequately pointed out limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper systematically tackles the prompt-based multiple object tracking problem with a variety of relevant contributions. We list the strengths of the paper below, expanding each paragraph with comments on the relevance of each contribution.\n\n## A new tracking paradigm (type to track)\nThis paper introduces the type to track paradigm, which consists in prompting models with text descriptions and obtaining all the tracklets corresponding to it. This paradigm is of great practical importance, as it allows users to query videos with text prompts to retrieve the desired objects.\n\n## A dataset for the Grounding Multiple Object Tracking Task (GroOT)\nThis paper introduces GroOT, a dataset which extends MOT17 and TAO with a variety of text captions to learn models suitable for the type to track task.\n\nGroOT supports 5 settings, including tracking by :\n- category name\n- category synonyms\n- category definition\n- retrieval prompts: 3 retrieval prompts per video are provided. the prompts are sequential and each prompt could refer to multiple tracklets at the same time (one-to-many).\n- tracklet captions: one caption here corresponds to one tracklet, where the caption consistently describes the object throughout the tracklet. In the TAO subset, the caption describes the object appearance. In the MOT17 subset, 2 types of captions are provided for each object, one describing the action and the other the appearance.\n\n## Class-agnostic evaluation metrics\nTo evaluate the type to track performance on GroOT, this paper introduces CA-MOTA and CA-IDF1, two class-agnostic tracking metrics.\n\n## A new language-based multiple object tracking model (MENDER)\nThe authors also introduce MENDER, a proof-of-concept tracker capable of retrieving tracklets based on text prompts.\n\nMENDER is composed of an image encoder, a text encoder, and a transformer decoder.\nThe **image encoder** encodes in parallel the image at time $t$ and the cropped bounding boxes for tracked instances at $t-1$.\nThe **text encoder** is a RoBERTa encoder that encodes the text prompts.\nCross-attention between image features, track embeddings and text embeddings produces the inputs to the transformer decoder, which is trained with a GIoU loss to detect the objects based on the track and text queries.\nThe proposed model is simple enough and addresses the type to track task effectively. ", "weaknesses": "## Missing baselines\nThe `category_name` setting could be tackled by any multiple object tracking algorithm. The current version of the paper lacks an absolute estimate of the tracking performance of MENDER in comparison to state-of-the-art tracking algorithms. It is possible that the current design produces suboptimal tracking performance, and only empirical results can show whether this is true or not. The paper would thus benefit from a comparison of MENDER to SOTA tracking-by-detection methods on the `category_name` setting.\n\n## Class-agnostic evaluation metrics\n**Evaluation of retrieval performance.** The proposed metrics focus more on evaluating the tracking performance of retrieved tracklets rather than the retrieval performance itself. Although the limitations section already indicates this as a limitation, I find this to be a weakness since a benchmark for tracking retrieval should specifically provide suitable metrics for the evaluation of such task. \n\n**CA-HOTA.** Although MOTA and IDF1 are established tracking metrics, the community has reached an agreement that they cannot solely represent the performance of a tracker. The HOTA metrics (DetA, AssA, and HOTA) disentangle different components of the tracking pipeline and it would be beneficial for the proposed benchmark to also introduce a class-agnostic version of such metrics.\n\n\n## Missing ablation on the tokenizer and the text encoder\nThe authors adopt the RoBERTa tokenizer and text encoder. However, they do not provide an explanation for choosing it over other popular architectures, e.g. CLIP. The paper would benefit from a comparison to other text encoder architectures and initializations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to Reviewer **RiBQ** for providing constructive feedback and recognition of our contributions. We are delighted to see that the reviewer has recognized the multiple relevant contributions of our paper. We will continue to refine and enhance our work per your suggestions.\n\n**Q26:** Missing baselines.\n\n**A26:** We appreciate Reviewer **RiBQ** for suggesting the performance comparison with SOTA approaches. The `category_name` is also the official benchmark, we published our result of the `category_name` setting on the official leaderboard of MOT17 (please find the method MENDER on the Private detection track, we are not allowed to attach any external links). It allows **convenient and direct comparison with other approaches**, including ByteTrack [B]. Note that the proposed MENDER is **one of the first attempts at the Grounded MOT task**, not to achieve the top rankings on the general MOT leaderboard. Other SOTA approaches benefit from the efficient single-category design in their separate object detectors, while our design is **agnostic to the category** and for **flexible textual input**. Below is the official comparison of some methods:\n\n|Method           |MOTA   |IDF1   |HOTA   |AssA   |DetA   |\n|-----------------|-------|-------|-------|-------|-------|\n|MENDER           |65.0   |67.1   |53.9   |54.4   |53.6   |\n|TrackFormer      |74.1   |68.0   |57.3   |54.1   |60.9   |\n|ByteTrack        |80.3   |77.3   |63.1   |62.0   |64.5   |\n\n**Q27:** Evaluation of retrieval performance.\n\n**A27:** As mentioned in the limitations section of our paper, this is indeed a recognized limitation of our work. We set a concrete goal in this work to lay the groundwork for future research in this emerging field and to inspire further investigations. As we continue to refine and expand the benchmark, we value the reviewer's feedback and we will explore additional metrics that can better capture the retrieval performance aspects.\n\n**Q28:** CA-HOTA\n\n**A28:** We appreciate Reviewer RiBQ for suggesting another class-agnostic metric for the comprehensive evaluation. As we stick with the idea of ignoring the negative effect of classification, we formulate the CA-HOTA as below:\n\n\n$\\text{HOTA} = \\frac{1}{|CLS|}\\sum_{cls}^{CLS}\\Big(\\sqrt{DetA\\cdot AssA}\\Big)_{cls}$\n\n$\\text{CA-HOTA} = \\sqrt{(DetA_{CLS^1})\\cdot(AssA_{CLS^1})}$, where the category set $CLS$ with size $n$ is reduced to $1$ by combining all elements: $CLS^n\\rightarrow CLS^1$\n\nWe also report the additional metrics namely CA-HOTA, AssA a DetA for the TAO dataset on the `tracklet_captions` setting as follows. We will organize and include them in other settings.\n\n|Method           |CA-HOTA   |AssA   |DetA   |\n|-----------------|----------|-------|-------|\n|MDETR + Tfm      |14.9      |19.3   |11.7   |\n|MENDER           |19.5      |26.0   |15.8   |\n\n**Q29:** Missing ablation on the tokenizer and the text encoder.\n\n**A29:** We appreciate Reviewer **RiBQ** for suggestions on the ablation of the text encoder. In our initial study, we opted for the RoBERTa tokenizer and text encoder based on its widespread usage and proven performance in various grounded baselines, including MDETR. We were initially constrained by the significant resources needed for further evaluations. As part of our commitment to continuously improve our work, we plan to conduct a comprehensive analysis of various text encoder architectures and initialization methods in our future iterations."}, {"review_id": "bYek4H1DJy", "reviewer": "Reviewer_j89C", "summary": "This paper provides a new paradigm for MOT, using natural language descriptions to track objects of interest in videos. To accomplish it, the authors collected and annotated various object and scene descriptions. Meanwhile, the new evaluation protocols are also formulated following the evaluation of the original MOT. The authors also provide a new method, showing good performance in experiments.", "questions": "See the weakness section.", "limitations": "The authors have already provided the limitations in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work has an intuitive and good motivation to solve MOT using natural language. \n* This paper is well organized, which contains collecting new benchmarks, designing effective methods, and conducting experiments.\n* More importantly, the authors provide many visualization results, which are interesting.\n", "weaknesses": "* First of all, the main idea of ‘Type-to-Track’ has a similar idea to Referring Multi-Object Tracking [1] (released at Arxiv2023.03, accepted at CVPR2023), which both use language description to prompt multiple objects. It is important to discuss and compare the related work.\n* I’m curious about the annotation details of the descriptions. Are they manually annotated? Or they are annotated by the current techniques of GPT?\n* In Table 3, does the number of words represent the number of descriptions? Or the words are words of descriptions?\n* The evaluation of caption prompts confuses me. The crucial words of created captions have one-to-one object matching with the grounded objects. So each object is evaluated individually?\n* This paper is not easy to follow and sometimes overclaimed on some details, e.g., third-order tensor modeling (using three inputs), and correlation simplification (using separate attention). Besides, the authors define many new concepts but lack details and explanations, e.g., the single-stage end-to-end framework. How to deal with new entrance or exit objects?\n* How about the visualization performance of the proposed method?\n\n[1] Referring Multi-Object Tracking, CVPR2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to Reviewer **j89C** for providing valuable feedback. We are pleased to know that the reviewer acknowledges the intuitive and compelling motivation behind our work. We are encouraged by the reviewer's positive remarks, and we continue to address the reviewer’s concerns below:\n\n**Q30:** First of all, the main idea of ‘Type-to-Track’ has a similar idea to Referring Multi-Object Tracking [A] (released at Arxiv2023.03, accepted at CVPR2023), which both use language description to prompt multiple objects.\n\n**A30:** We appreciate Reviewer **j89C** for suggesting the Ref-KITTI dataset. Please refer to **KP2** and the attachment for the updated Table 1, and **A1** for our responsively class-agnostic tracking paradigm.\n\n**Q31:** I’m curious about the annotation details of the descriptions. Are they manually annotated? Or they are annotated by the current techniques of GPT?\n\n**A31:** All the descriptions are manually annotated by human annotators. The annotation process began before the release of the first GPT technique supporting visual input (GPT-4) on Mar 13th, 2023, and our first initiated annotation was already released on Feb 17th, 2023. Note that the current visual GPT techniques mainly focus on images or videos, while our task involves tracking objects, which demands **the temporal consistency of each instance** throughout the entire length of the videos. This distinction highlights the unique challenges of our task, which **can not be fully addressed by existing visual GPT techniques**.\n\n**Q32:** In Table 3, does the number of words represent the number of descriptions? Or the words are words of descriptions?\n\n**A32:** It is the number of words in descriptions. We counted other datasets in the same criteria.\n\n**Q33:** The evaluation of caption prompts confuses me. The crucial words of created captions have one-to-one object matching with the grounded objects. So each object is evaluated individually?\n\n**A33:** The term \"one-to-one\" in the evaluation of caption prompts refers to the fact that **each caption is matched to one specific object**. However, it **does not mean that objects are evaluated one-by-one**. In the evaluation process, all text prompts are **passed into a single network pass**, and the performance is measured by the **multiple object tracking metrics**. These MOT metrics **heavily penalize the inaccuracies in identity assignment**, especially as the number of objects increases, and **can not be evaluated individually**.\n\n**Q34:** This paper is not easy to follow and sometimes overclaimed on some details, e.g., third-order tensor modeling (using three inputs), and correlation simplification (using separate attention). Besides, the authors define many new concepts but lack details and explanations, e.g., the single-stage end-to-end framework. How to deal with new entrance or exit objects?\n\n**A34:** We appreciate your concerns and have taken them into account. Allow us to clarify the points you mentioned:\n\n- Third-order tensor modeling: The use of a third-order tensor refers to our specific approach to handle three types of input components (video frames, textual prompts, and object features) in a single tracker network. **A third-order tensor [E]** refers to **a three-dimensional tensor** that allows us to model the interactions and dependencies between these components effectively.\n\n- Correlation simplification: The correlation simplification process is detailed in the derivation from Eqn. 3 to Eqn. 6, where we show **the step-by-step development of the simplified correlation approach**. It involves streamlining the correlation computation to enhance efficiency and performance.\n\n- Single-stage end-to-end framework: The term \"single-stage end-to-end framework\" in our paper refers to **the joint detection-and-tracking approach that is fully trainable**. We compared this framework with **another two-stage framework, where each stage needs to be trained separately**.\n\n- New entrance or exit objects: In the Supplementary material, we provided a detailed description of how we handle such cases. For new entrance objects, the initialization process occurs from **line 13 to line 14** of our algorithm. When a new object appears in the video, we create a new tracklet for it and initialize its features based on the first appearance. On the other hand, for exit objects, we handle deprecations by removing inactive tracklets from further consideration which exceed a tolerant window time, as described in **line 24** of the algorithm.\n\n**Q35:** How about the visualization performance of the proposed method?\n\n**A35:** The visualization performance of our proposed method is presented in the supplementary material, where we provide a video demonstration and visualizations to showcase the accuracy, effectiveness, and failed cases of our approach. Some more cases are in the attachment.\n\n[E] Kolda, Tamara G., and Brett W. Bader. \"Tensor decompositions and applications.\" SIAM review 51.3 (2009): 455-500."}]

Author rebuttal:
We thank all reviewers for their valuable comments and suggestions. The reviewers highlight the strong aspects of the paper, including the high-quality and diverse dataset, the well-defined task, and the paper's clarity and organization. Reviewers **h9ub** and **RiBQ** are inclining towards acceptance (**Accept** and **Weak Accept**) due to the task's significance and the paper's well-structured presentation. Similarly, Reviewers **HDeT** and **j89C** lean towards acceptance with **two Borderline Accept** ratings. In contrast, Reviewer **9rR3** has assigned a **Borderline Reject** rating, mainly stressing the need for some additional details. We will address the common key points (**KP**s) first. Then the individual answers will be given in the following comments.

**KP1: Dataset comparison with Video Object Segmentation (VOS):** We thank reviewers **HDeT** and **h9ub** for suggesting Refer-Youtube-VOS, and Ref-DAVIS datasets. We added these two datasets to the comparison Table 1 with the specification of the VOS task and handling **a single object for each expression**. However, we want to clarify the importance and difference that makes our dataset stand out from these two datasets is the ability to construct the Grounded MOT task in **five different types of setting**, including **a setting that covers multiple objects in a single expression**. These five settings give us the full capability of evaluating different input types, which is also agreed upon by Reviewer **h9ub**.

**KP2: Dataset comparison with Ref-KITTI:** We thank reviewers **HDeT**, **h9ub**, and **j89C** for suggesting the Ref-KITTI [A] dataset. We included that dataset in Table 1. Note that our dataset focuses on **responsively tracking**, **agnostic to category**, and currently **outnumbers the frames and settings**, while the Ref-KITTI contains only two categories which are cars and pedestrians. At the time we submitted this paper on May 11th, 2023, we were unaware of the availability of the Ref-KITTI dataset. According to the comparison policy of NeurIPS 2023, we consider the Ref-KITTI [A] dataset as a concurrent work to our submission because the first time it appeared is Mar 11th, 2023 on arXiv and officially on proceedings of CVPR 2023 is June 2023.

Please find the updated Table 1 in the attachment. We will discuss the VOS task (Refer-Youtube-VOS, Ref-DAVIS) and the Ref-KITTI in the main text of the revision accordingly.

**KP3: Ours MOT17 test annotation is a sub-optimal ground truth:** We appreciate reviewers **HDeT** and **9rR3**’s inquiry, as it allows us to provide additional details to our experiments. We annotated the raw tracking data of the best-performant tracker at the time we constructed experiments as the sub-optimal ground truth (i.e., BoT-SORT [C] at 80.5% MOTA and 80.2% IDF1). That is also the raw data we used to evaluate all our ablation studies. The MOT17 subset in Table 4 on the `category_name` setting has **a similar performance** to our official result (please find the method MENDER on the Private detection track of the MOT17 challenge, we are not allowed to attach any external links): 67.00% MOTA and 71.20% IDF1 vs. 65.0% MOTA and 67.1% IDF1. We would cooperate with the MOTChallenge organizers for the official test prompts upon their approval. We had this detail in the main paper in the first version but removed it due to the page limit. We will re-add this detail to the revision upon acceptance.

[A] Wu, Dongming, et al. "Referring Multi-Object Tracking." CVPR 2023.

[B] Zhang, Yifu, et al. "Bytetrack: Multi-object tracking by associating every detection box." ECCV 2022.

[C] Aharon, Nir, Roy Orfaig, and Ben-Zion Bobrovsky. "BoT-SORT: Robust associations multi-pedestrian tracking." arXiv preprint arXiv:2206.14651 (2022).

Meta-review: This paper received accept, weak and borderline accept recommendations. There is a rebuttal.
Reviewer RiBQ has concerns around the dataset quality, which are shared with Reviewer 9rR3. The authors attempt to address these in the rebuttal. Reviewer 9rR3's concerns with dataset quality are addressed by the authors in the rebuttal and 9rR3 is willing to raise their score. Similarly, Reviewer h9ub's concerns around dataset quality have been sufficiently addressed in the final version of the dataset. Further h9ub thinks the performance of MENDER falls short of other SOTA MOT models but is sufficient for the purpose of this paper. Reviewer HDeT shares the dataset concerns and feels that the paper presentation is too complex; therefore remains their borderline accept rating. Reviewer j89C recommends borderline accept and the rebuttal resolves most of their concerns.
The AC agrees with the reviewers' opinion towards accepting this work. The authors are strongly encouraged to include the additional feedback into the paper and the improved dataset, as well as including the results from the rebuttal in the camera-ready version. Specifically, the authors should  conduct a comparison and discussion with RMOT in the revised version, including the performance comparison of MENDER to other SOTA trackers in the main paper.
Decision: Accept (poster)