Title: Type-to-Track: Retrieve Any Object via Prompt-based Tracking
Venue: NeurIPS 2023
Accepted: yes
Keywords: Grounded Object Tracking

Multiple Object Tracking

Vision Language
Abstract: One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions. We present a new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework (MENDER) using the third-order tensor decomposition. The experiments in five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7\% accuracy and $4\times$ speed faster.

Type-to-Track: Retrieve Any Object via Prompt-based Tracking

Abstract

One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces a novel paradigm for Multiple Object Tracking called Type-toTrack , which allows users to track objects in videos by typing natural language descriptions. We present a new dataset for that Grounded Multiple Object Tracking task, called GroOT , that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework ( MENDER ) using the third-order tensor decomposition. The experiments in five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7% accuracy and 4 × speed faster.

1 Introduction

Tracking the movement of objects in videos is a challenging task that has received significant attention in recent years. Various methods have been proposed to tackle this problem, including deep learning techniques. However, despite these advances, there is still room for improvement in intuitiveness and responsiveness. One potential way to improve object tracking in videos is to incorporate user input into the tracking process. Traditional Visual Object Tracking (VOT) methods typically require

users to manually select objects in the video by points [1], bounding boxes [2, 3], or trained object detectors [4, 5]. Thus, in this paper, we introduce a new paradigm, called Type-to-Track , to this task that combines responsive typing input to guide the tracking of objects in videos. It allows for more intuitive and conversational tracking, as users can simply type in the name or description of the object they wish to track, as illustrated in Fig. 1. Our intuitive and user-friendly Type-to-Track approach has numerous potential applications, such as surveillance and object retrieval in videos.

We present a new Grounded Multiple Object Tracking dataset named GroOT that is more advanced than existing tracking datasets [6, 7]. GroOT contains videos with various types of multiple objects and detailed textual descriptions. It is 2 × larger and more diverse than any existing datasets, and it can construct many different evaluation settings. In addition to three easy-to-construct experimental settings, we propose two new settings for prompt-based visual tracking. It brings the total number of settings to five, which will be presented in Section 5. These new experimental settings challenge existing designs and highlight the potential for further advancements in our proposed research topic.

In summary, this work addresses the use of natural language to guide and assist the Multiple Object Tracking (MOT) tasks with the following contributions. First, a novel paradigm named Type-to-Track is proposed, which involves responsive and conversational typing to track any objects in videos. Second, a new GroOT dataset is introduced. It contains videos with various types of objects and their corresponding textual descriptions of 256K words describing definition, appearance, and action. Next, two new evaluation protocols that are tracking by retrieval prompts and caption prompts , and three class-agnostic tracking metrics are formulated for this problem. Finally, a new transformer-based eMbed-ENcoDE-extRact framework ( MENDER ) is introduced with third-order tensor decomposition as the first efficient approach for this task. Our contributions in this paper include a novel paradigm, a rich semantic dataset, an efficient methodology, and challenging benchmarking protocols with new evaluation metrics. These contributions will be advantageous for the field of Grounded MOT by providing a valuable foundation for the development of future algorithms.

2 Related Work

2.1 Visual Object Tracking Datasets and Benchmarks

Datasets. To develop and train VOT models for the computer vision task of tracking objects in videos, various datasets have been created and widely used. Some of the most popular datasets for VOT are OTB [19, 8], VOT [9], GOT [10], MOT challenges [12, 14] and BDD100K [15]. Visual object tracking has two sub-tasks: Single Object Tracking (SOT) and Multiple Object Tracking (MOT). Table 1 shows that there is a wide variety of object tracking datasets in both types available, each with its own strengths and weaknesses. Existing datasets with NLP [6, 7] only support the SOT task, while our GroOT dataset supports MOT with approximately 2 × larger in description size.

Benchmarks. Current benchmarks for tracking can be broadly classified into two main categories: Tracking by Bounding Box and Tracking by Natural Language , depending on the type of initialization.

all uses (1, 2, 3, 4, 5, 6) and w/o MOT20 uses (1, 2, 3, 4).

Previous benchmarks [20, 19, 8, 9, 21, 22, 22, 23] were limited to test videos before the emergence of deep trackers. The first publicly available benchmarks for visual tracking were OTB-2013 [19] and OTB-2015 [8], consisting of 50 and 100 video sequences, respectively. GOT-10k [10] is a benchmark featuring 10K videos classified into 563 classes and 87 motions. TrackingNet [11], a subset of the object detection benchmark YT-BB [24], includes 31K sequences. Furthermore, there are long-term tracking benchmarks such as OxUvA [25] and LaSOT [6]. OxUvA spans 14 hours of video in 337 videos, comprising 366 object tracks. On the other hand, LaSOT [6] is a languageassisted dataset consisting of 1.4K sequences with 9.8K words in their captions. In addition to these benchmarks, TNL2K [7] includes 2K video sequences for natural language-based tracking and focuses on expressing the attributes. LaSOT [6] and TNL2K [7] support one benchmarking setting with their provided prompts, while our GroOT dataset supports five settings. Ref-KITTI [18] is built upon the KITTI [26] dataset and contains only two categories, including car and pedestrian , while our GroOT dataset focuses on category-agnostic tracking, and outnumbers the frames and settings.

A similar task with a different nomenclature to the Grounded MOT task is Referring Video Object Segmentation (Ref-VOS) [16, 17], which primarily measures the overlapping area between the ground truth and prediction for a single foreground object in each caption, with less emphasis on densely tracking multiple objects over time. In contrast, our proposed Type-to-Track paradigm is distinct in its focus on responsively and conversationally typing to track any objects in videos, requiring maintaining the temporal motions of multiple objects of interest.

2.2 Grounded Object Tracking

Grounded Vision-Language Models accurately map language concepts onto visual observations by understanding both vision content and natural language. For instance, visual grounding [29] seeks to identify the location of nouns or short phrases (such as a black hat or a blue bird) within an image. Grounded captioning [30, 31, 32] can generate text descriptions and align predicted words with object regions in an image. Visual dialog [33] enables meaningful dialogues with humans about visual content using natural, conversational language. Some visual dialog systems may incorporate referring expression recognition [34] to resolve expressions in questions or answers.

Grounded Single Object Tracking is limited to tracking a single object with box-initialized and language-assisted methods. The GTI [27] framework decomposes the tracking by language task into three sub-tasks: Grounding, Tracking, and Integration, and generates tubelet predictions frame-byframe. AdaSwitcher [7] module identifies tracking failure and switches to visual grounding for better tracking. [35] introduce a unified system using attention memory and cross-attention modules with learnable semantic prototypes. Another transformer-based approach [28] is presented including a cross-modal fusion module, task-specific heads, and a proxy token-guided fusion module.

2.3 Discussion

Most existing datasets and benchmarks for object tracking are limited in their coverage and diversity of language and visual concepts. Additionally, the prompts in the existing Grounded SOT benchmarks do not contain variations in covering many objects in a single prompt, which limits the application of existing trackers in practical scenarios. To address this, we present a new dataset and benchmarking

(a) Our MOT17 [12] subset.

(b) Our TAO [13] subset.

personstandincp

woomtandiwearwomaninredshirt

personinthestreet

person ina strinedshirt

personwearing

woman

peoplepink

man

pavement

aptapersoniin the streeth

per son

blue

yellow

Posr

red

background ]a Cket

green

holding

wearing

parking

womansuit

bag

street

sitting

singunbrella

carrying

pants

standing

plaidstrelle.

Walkingsidewalk

Ragteimei

gray"

white

boyholding apaddle

light

person

hairpeoplefield

holding

Laptop

ired

MO

the dog is black

wearing

e

open

bottle

shirt

oung

hander

elephantbag

ande

Q

smara

woman

play

book

long

ean

e

arelookinglarge chair

womaninredsweatshirtandjeans

blackblue

baby

girl

table bear

playing

zebra

girlwearing a pink cma

a black and white dog

metrics to support the emerging trend of the Grounded MOT, where the goal is to align language descriptions with fine-grained regions or objects in videos.

As shown in Table 2, most of the recent methods for the Grounded SOT task are not class-agnostic, meaning they require prior knowledge of the object. GTI [27] and TransVLT [28] need to input the initial bounding box, while TrackFormer [4] need the pre-defined category. The operation used in [27] to fuse visual and textual features is concatenation which can only support prompts describing a single object. A Grounded MOT can be constructed by integrating a grounded object detector, i.e. MDETR [36], and an object tracker, i.e. TrackFormer [4]. However, this approach is low-efficient because the visual features have to be extracted multiple times. In contrast, our proposed MOT approach MENDER formulates third-order attention to adaptively focus on many targets, and it is an efficient single-stage and class-agnostic framework. The scope of class-agnostic in our approach is constructing a large vocabulary of concepts via a visual-textual corpus, following [37, 38, 39].

3 Dataset Overview

3.1 Data Collection and Annotation

Existing object tracking datasets are typically designed for specific types of video scenes [40, 41, 42, 43, 44, 2]. To cover a diverse range of scenes, GroOT was created using official videos and bounding box annotations from the MOT17 [12], TAO [13], and MOT20 [14]. The MOT17 dataset comprises 14 sequences with diverse environmental conditions such as crowded scenes, varying viewpoints, and camera motion. The TAO dataset is composed of videos from seven different datasets, such as the ArgoVerse [45] and BDD [15] datasets containing outdoor driving scenes, while LaSOT [6] and YFCC100M [46] datasets include in-the-wild internet videos. Additionally, the AVA [47], Charades [48], and HACS [49] datasets include videos depicting human-human and human-object interactions. By combining these datasets, GroOT covers multiple types of scenes and encompasses a wide range of 833 objects. This diversity allows for a wide range of object classes with captions to be included, making it an invaluable resource for training and evaluating visual grounding algorithms.

We release our textual description annotations in COCO format [50]. Specifically, a new key 'captions' which is a list of strings is attached to each 'annotations' item in the official annotation. In the MOT17 subset, we attempt to maintain two types of caption for well-visible objects: one describes the appearance and the other describes the action . For example, the caption for a well-visible person might be ['a man wearing a gray shirt', 'person walking on the street'] as shown in Fig. 2a. However, 10% of tracklets only have one caption type, and 3% do not have any captions due to their low visibility. The physical characteristics of a person or their personal accessories, such as their clothing, bag color, and hair color are considered to be part of their appearance. Therefore, the appearance captions include verbs 'carrying' or 'holding' to describe personal accessories. In the TAO subset, objects other than humans have one caption

describing appearance, for instance, ['a red and black scooter'] . Objects that are human have the same two types of captions as the MOT17 subset. An example is shown in Fig. 2b. These captions are consistently annotated throughout the tracklets. Fig. 3 is the word-cloud visualization of our annotations.

3.2 Type-to-Track Benchmarking Protocols

Let V be a video sample lasts t frames, where V = { I t | t < | V | } and I t be the image sample at a particular time step t . We define a request prompt P that describes the objects of interest, and T t is the set of tracklets of interest up to time step t . The Type-to-Track paradigm requires a tracker network T ( I t , T t -1 , P ) that efficiently take into account I t , T t -1 , and P to produce T t = T ( I t , T t -1 , P ) . To advance the task of multiple object retrieval, another benchmarking set is created in addition to the GroOT dataset. While training and testing sets follow a One-to-One scenario, where each caption describes a single tracklet, the new retrieval set contains prompts that follow a One-to-Many scenario, where a short prompt describes multiple objects. This scenario highlights the need for diverse methods to improve the task of multiple object retrieval. The retrieval set is provided with a subset of tracklets in the TAO validation set and three custom retrieval prompts that change throughout the tracking process in a video { P t 1 =0 , P t 2 , P t 3 } , as depicted in Fig. 1(a). The retrieval prompts are generated through a semi-automatic process that involves: (i) selecting the most commonly occurring category in the video, and (ii) cascadingly filtering to the object that appears for the longest duration. In contrast, the caption prompts are created by joining tracklet captions in the scene and keeping it consistent throughout the tracking period. We name these two evaluation scenarios as tracklet captions cap and object retrieval retr . With three more easy-to-construct scenarios, five scenarios in total will be studied for the experiments in Section 5. Table 3 presents the statistics of the five settings, and the data portions are highlighted in the corresponding colors.

3.3 Class-agnostic Evaluation Metrics

As indicated in [51], long-tailed classification is a very challenging task in imbalanced and large-scale datasets such as TAO. This is because it is difficult to distinguish between similar fine-grained classes, such as bus and van, due to the class hierarchy. Additionally, it is even more challenging to treat every class independently. The traditional method of evaluating tracking performance leads to inadequate benchmarking and undesired tracking results. In our Type-to-Track paradigm, the main task is not to classify objects to their correct categories but to retrieve and track the object of interest. Therefore, to alleviate the negative effect, we reformulate the original per-category metrics of MOTA [52], IDF1 [53], HOTA [54] into class-agnostic metrics:

where CLS n is the category, set size n is reduced to 1 by combining all elements: CLS n → CLS 1 .

4 Methodology

4.1 Problem Formulation

Given the image I t and the request prompt P describing the objects of interest, which can adaptively change between { P t 1 , P t 2 , P t 3 } in the retr setting, and K is the prompt's length | P | = K , let enc ( · ) and emb ( · ) be the visual encoder and the word embedding model to extract features of image tokens and prompt tokens, respectively. The resulting outputs, enc ( I t ) ∈ R M × D and

emb ( P ) ∈ R K × D , where D is the length of feature dimensions. A list of region-prompt associations C t , which contains objects' bounding boxes and their confident scores, can be produced by Eqn. (4):

where ( ¯ × ) is an operation representing the region-prompt correlation, that will be elaborated in the next section, dec γ ( · , · ) is an object decoder taking the similarity and the image features to decode to object locations, thresholded by a scoring parameter γ (i.e. c conf ≥ γ ). For simplicity, the cardinality of the set of objects | C t | = M , implying each image token produces one region-text correlation.

We define T t = { tr j = ( tr x , tr y , tr w , tr h , tr conf , tr id ) j | j < N } t produced by the tracker T , where N = | T t | is the cardinality of current tracklets. i , j , k , and t are consistently denoted as indexers for objects, tracklets, prompt tokens, and time steps for the rest of the paper.

Remark 1 Third-order Tensor Modeling. Since the Type-to-Track paradigm requires three input components I t , T t -1 , and P , an auto-regressive single-stage end-to-end framework can be formulated via third-order tensor modeling.

To achieve this objective, a combination of initialization, object decoding, visual encoding, feature extraction, word embedding, and aggregation can be formulated as in Eqn. (5):

where ext ( · ) denotes the visual feature extractor of the set of tracklets, ext ( T t -1 ) ∈ R N × D , 1 D × D × D is an all-ones tensor has size D × D × D , ( × n ) is the n -mode product of the thirdorder tensor [55] to aggregate many types of token 1 , and initialize ( · ) is the function to ascendingly assign unique identities to tracklets for the first time those tracklets appear.

Let T ∈ R M × N × K be the resulting tensor T = 1 D × D × D × 1 enc ( I t ) × 2 ext ( T t -1 ) × 3 emb ( P ) . The objective function can be expressed as the log softmax of the positive region-tracklet-prompt triplet over all possible triplets, defined in Eqn. (6):

where θ denotes the network's parameters, the combination of the i th image token, the j th tracklet, and the k th prompt token is the correlated triplet.

In the next subsection, we elaborate our model design for the tracking function T ( I t , T t -1 , P ) , named MENDER , as defined in Eqn. (5), and loss functions for the problem objective in Eqn. (6).

4.2 MENDER for Multiple Object Tracking by Prompts

The correlation in Eqn. (5) has the cubic time and space complexity O ( n 3 ) , which can be intractable as the input length grows and hinder the model scalability.

Remark 2 Correlation Simplification. Since both enc ( · ) and ext ( · ) are visual encoders, the region-prompt correlation can be equivalent to the tracklet-prompt correlation. Therefore, the region-tracklet-prompt correlation tensor T can be simplified to lower the computation footprint.

To design that goal, the extractor and encoder share network weights for computational efficiency:

where T : j : and T i :: are lateral and horizontal slices. In layman's terms, the region-prompt correlation at the time step t -1 is equivalent to the tracklet-prompt correlation at the time step t , as visualized in Fig. 4(a). Therefore, one practically needs to model the region-tracklet and tracklet-prompt

(a) Because the tracklet set T t - 1 pools visual features of the image I t - 1 , the region-prompt is equivalent with tracklet-prompt (only need to filter unassigned objects).

region-tracklet correlation

tracklet-prompt correlation

SpatialPositionalEmbedding

region-prompt correlation

equivalent connections

DETR

Encoder

:TemporalPositionalEmbedding

Bac

Followmenin suit

enc(It)

encl·

ext(-)

emb(-);

P

X2

00

×3

Pooling

Backbone

DETR

Encoder

Transformer

×

×

Decoder

Tt-1

ert(

Decoder

ect(T

emb(P)

embedding

+

+

"Followmen!

Word

×2

X3

in suit'

P

emb(P)

MENDER

Tt

X

correlations which reduces time and space complexity from O ( n 3 ) to O ( n 2 ) , significantly lowering computation footprint. We alternatively rewrite the decoding step in Eqn. (5) as follows:

Correlation Representations. In our approach, the correlation operation ( ¯ × ) is modelled by the multi-head cross-attention mechanism [57], as depicted in Fig. 4(b). The attention matrix can be computed as:

where X and Y tokens are one of these types: region, tracklet, prompt. σ ( · ) is one of the operations enc ( · ) , emb ( · ) , ext ( · ) as the corresponding operation to X or Y . Superscript W Q , W K , and W V are the projection matrices corresponding to X or Y as in the attention mechanism.

Then, the attention weight from the image I t to the prompt P are computed by the matrix multiplication for A I | T and A T | P to aggregate the information from two matrices as in Eqn. (8). The result is the matrix A I | T × T | P = A I | T ×A T | P that shows the correlation between each input or output. Then, the resulting attention matrix A I | T × T | P is used to produce the object representations at time t :

Object Decoder dec ( · ) utilizes context-aware features Z t that are capable of preserving identity information while adapting to changes in position. The tracklet set T t is defined in the auto-regressive manner to adjust to the movements of the object being tracked as in Eqn. (8). For decoding the final output at any frame, the decoder transforms the object representation by a 3-layer FFN to predict bounding boxes and confidence scores for frame t :

where the identification information of tracklets, represented by tr id , is not determined directly by the FFN model. Instead, the tr id value is set when the tracklet is first initialized and maintained till its end, similar to tracking-by-attention approaches [4, 58, 59, 60].

4.3 Training Losses

To achieve the training objective function as in Eqn. (6), we formulate the objective function into two loss functions L I | T and L T | P for correlation training and one loss L GIoU for decoder training:

where γ T | P , γ I | T , and γ GIoU are corresponding coefficients, which are set to 0 . 3 by default.

Alignment Loss L T | P is a contrastive loss, which is used to assure the alignment of the ground-truth object feature and caption pairs ( T , P ) which can be obtained in our dataset. There are two alignment losses used, one for all objects normalized by the number of positive prompt tokens and the other for all prompt tokens normalized by the number of positive objects. The total loss can be expressed as:

where P + and I + are the sets of positive prompts and image tokens corresponding to the selected enc ( I ) i and emb ( P ) k , respectively.

Objectness Losses. To model the track's temporal changes, our network learns from training samples that capture both appearance and motion generated by two adjacent frames:

L I | T is the log-softmax loss to guide the tokens' alignment as similar to Eqn. (13). In the L GIoU loss, obj i is the ground truth object corresponding to tr j . The optimal assignment between tr j or obj i to the ground truth object is computed efficiently by the Hungarian algorithm, following DETR [56]. ℓ GIoU is the Generalized IoU loss [61].

5 Experimental Results

5.1 Implementation Details

Experimental Scenarios. We create three types of prompt: category name nm , category synonyms syn , category definition def . One tracklet captions cap scenario is constructed by our detailed annotations and one more objects retrieval retr scenario is given in our custom request prompts as described in Subsec. 3.2. The dataset contains 833 classes, each has a name and a corresponding set of synonyms that are different names for the same category, such as [man , woman , human , pedestrian , boy , girl , child] for person . Additionally, each category is described by a category definition sentence. This definition makes the model deal with the variations in the text prompts. We join the names, synonyms, definitions, or captions and filter duplicates to construct the prompt. Trained models use as the same type as testing. We annotated the raw tracking data of the best-performant tracker (i.e., BoT-SORT [62] at 80.5% MOTA and 80.2% IDF1) at the time we constructed experiments and used it as the sub-optimal ground truth of MOT17 and MOT20 (parts (2, 4) in Table 3). That is also the raw data we used to evaluate all our ablation studies.

Datasets and Metrics. RefCOCO+ [63] and Flickr30k [64] serve as pre-trained datasets for acquiring a vocabulary of visual-textual concepts [37]. The ext ( · ) operation is not involved in this training step. After obtaining a pre-trained model from RefCOCO+ and Flickr30k, we train and evaluate our model for the proposed Type-to-Track task on all five scenarios on our GroOT dataset and the first-three scenarios for MOT20 [14]. The tracking performance is reported in class-agnostic metrics CA-MOTA, CA-IDF1, and CA-HOTA as in Subsec. 3.3 and mAP50 as defined in [13].

Tokens Production. emb ( · ) utilizes RoBERTa [65] to convert the text input into a sequence of numerical tokens. The tokens are fed into the RoBERTa-base model for text encoding using a

def

,

cap and

retr

.

Best viewed in color.

12-layer transformer network with 768 hidden units and 12 self-attention heads per layer. enc ( · ) is implemented using a ResNet-101 [66] as the backbone to extract visual features from the input image. The output of the ResNet is processed by a Deformable DETR encoder [67] to generate visual tokens. For each dimension, we use sine and cosine functions with different frequencies as positional encodings, similar to [68]. A feature resizer combining a list of (Linear , LayerNorm , Dropout) is used to map to size D = 512 for all token producers.

5.2 Ablation Study

Comparisons in Different Scenarios. Table 4 shows comparisons in the performance of different prompt inputs. For MOT17 and MOT20, the category name is 'person' , while category definition is 'a human being' . Since the prompt by category definition is short, it does not differ much from the nm setting. However, the syn setting shuffles between some words, resulting in a slight decrease in CA-MOTA and CA-IDF1. The cap setting results in prompts that contain more diverse and complex vocabulary, and more context-specific information. It is more difficult for the model to accurately localize the objects and identify their identity within the image, as it needs to take into account a wider range of linguistic cues, resulting in a decrease in performance compared to def (59.5% CA-MOTA and 54.8% CA-IDF1 vs 67.3% CA-MOTA and 72.4% CA-IDF1 on MOT17).

For TAO, the def setting has a significant number of variations and many tenuous connections in the scene context, for example, 'an aircraft that has a fixed wing and is powered by propellers or jets' for the airplane category. Therefore, it results in a decrease in performance (16.8% CA-MOTA and 27.7% CA-IDF1) compared to cap (20.7% CA-MOTA and 32.0% CA-IDF1), because the cap setting is more specific on the object level than category level. The best performant setting is nm (27.3% CA-MOTA and 37.2% CA-IDF1), where names are combined.

Simplied Attention Representations. Table 4 also presents the effectiveness of different attention representations of the full tensor T (denoted by ✗ ) and the simplified correlation (denoted by ✓ ). The performance is reported with frame per second (FPS), which is self-measured on one GPU NVIDIA RTX 3060 12GB. Overall, the performance of simplified correlation is witnessed with a superior speed of up to 2 × (7.8 FPS vs 3.4 FPS of cap on MOT17 and 11.5 FPS vs 7.6 FPS of retr on TAO), resulting in and a slight increase in accuracy due to attention stability and precision gain.

5.3 Comparisons with A Baseline Design

Due to the new proposed topic, no current work has the same scope or directly solves our problem. Therefore, we compare our proposed MENDER against a two-stage baseline tracker in Table 5. We use current SOTA methods to develop this approach, i.e., MDETR [36] for the grounded detector, while TrackFormer [4] for the object tracker. It is worth noting that our MENDER relies on direct regression to locate and track the object of interest, without the need for an explicit grounded object detection stage. Table 5 shows our proposed MENDER outperforms the baseline on both CA-MOTA and CA-IDF1 metrics in all four settings category synonyms , category definition , tracklet captions and object retrieval (25.7% vs. 21.3%, 16.8% vs. 14.6%, 20.7% vs. 15.3% and 32.9% vs. 25.7% CA-MOTA on TAO), while can maintain up to 4 × run-time speed (10.3 FPS vs 2.2 FPS). The results indicate that training a single-stage network enhances efficiency and reduces errors by avoiding separate feature extractions for both detection and tracking steps.

5.4 Comparisons with State-of-the-Art Approaches

The category name nm setting is also the official MOT benchmark. Table 6 is the comparison of our result on the category name setting on the official leaderboard of MOT17, compared with other state-of-the-art approaches, including ByteTrack [69] and TrackFormer [4]. Note that our proposed MENDER is one of the first attempts at the Grounded MOT task, not to achieve the top rankings on the general MOT leaderboard. In contrast, other SOTA approaches benefit from the efficient single-category design in their separate object detectors, while our single-stage design is agnostic to the category and for flexible textual input. Compared to TrackFormer [4], our proposed MENDER only demonstrates a marginal decrease in identity assignment (67.1% vs 68.0% CA-IDF1). The decrease in the CA-MOTA stems from our detector's design which integrates flexible input.

6 Conclusion

We have presented a novel problem of Type-to-Track , which aims to track objects using natural language descriptions instead of bounding boxes or categories, and a large-scale dataset to advance this task. Our proposed MENDER model reduces the computational complexity of third-order correlations by designing an efficient attention method that scales quadratically w.r.t the input sizes. Our experiments on three datasets and five scenarios demonstrate that our model achieves state-ofthe-art accuracy and speed for class-agnostic tracking.

Limitations. While our proposed metrics effectively evaluate the proposed Type-to-Track problem, they may not be ideal for measuring precision-recall characteristics in retrieval tasks. Additionally, the lack of the question-answering task in data and problem formulation may limit the algorithm to not being able to provide language feedback such as clarification or alternative suggestions. Additional benchmarks incorporating question-answering are excellent research avenues for future work. While the performance of our proposed MENDER may not be optimal for well-defined categories, it paves the way for exploring new avenues in open vocabulary and open-world scenarios [74].

Broader Impacts. The Type-to-Track problem and the proposed MENDER model have the potential to impact various fields, such as surveillance and robotics, where recognizing object interactions is a crucial task. By reformulating the problem with text support, the proposed methodology can improve the intuitiveness and responsiveness of tracking, making it more practical for video input support in large-language models [75] and real-world applications similar to ChatGPT. However, it could bring potential negative impacts related to human rights by providing a video retrieval system via text.

Acknowledgment. This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), and Google Initiated Research Grant. We also thank Utsav Prabhu and Chi-Nhan Duong for their invaluable discussions and suggestions and acknowledge the Arkansas High-Performance Computing Center for providing GPUs.

References

Pha Nguyen, Thanh-Dat Truong, Miaoqing Huang, Yi Liang, Ngan Le, and Khoa Luu. Self-supervised domain adaptation in crowd counting. In 2022 IEEE International Conference on Image Processing (ICIP) , pages 2786-2790. IEEE, 2022. 2

Kha Gia Quach, Pha Nguyen, Huu Le, Thanh-Dat Truong, Chi Nhan Duong, Minh-Triet Tran, and Khoa Luu. Dyglip: A dynamic graph model with link prediction for accurate multi-camera multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13784-13793, 2021. 2, 4

Kha Gia Quach, Huu Le, Pha Nguyen, Chi Nhan Duong, Tien Dai Bui, and Khoa Luu. Depth perspectiveaware multiple object tracking. arXiv preprint arXiv:2207.04551 , 2022. 2

Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multiobject tracking with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8844-8854, 2022. 2, 3, 4, 7, 10

Pha Nguyen, Kha Gia Quach, John Gauch, Samee U Khan, Bhiksha Raj, and Khoa Luu. Utopia: Unconstrained tracking objects without preliminary examination via cross-domain adaptation. arXiv preprint arXiv:2306.09613 , 2023. 2

Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5374-5383, 2019. 2, 3, 4

Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13763-13773, June 2021. 2, 3

Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37(9):1834-1848, 2015. 2, 3

Matej Kristan, Jiri Matas, Aleš Leonardis, Tomas Vojir, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay, Fatih Porikli, and Luka ˇ Cehovin. A novel performance evaluation methodology for single-target trackers. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(11):2137-2155, Nov 2016. 2, 3

Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019. 2, 3

Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 300-317, 2018. 2, 3

A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-object tracking. arXiv:1603.00831 [cs] , March 2016. arXiv: 1603.00831. 2, 4

Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale benchmark for tracking any object. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16 , pages 436-454. Springer, 2020. 2, 4, 8

Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé. Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003 , 2020. 2, 4, 8

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020. 2, 4

Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part IV 14 , pages 123-141. Springer, 2019. 2, 3

Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16 , pages 208-223. Springer, 2020. 2, 3

Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14633-14642, 2023. 2, 3

Yi Wu, Jongwoo Lim, and Ming Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis & Machine Intelligence , 37(9):1834, 2015. 2, 3

Pengpeng Liang, Erik Blasch, and Haibin Ling. Encoding color information for visual tracking: Algorithms and benchmark. IEEE Transactions on Image Processing , 24(12):5630-5644, 2015. 3

A Li, M Lin, Y Wu, MH Yang, and S Yan. NUS-PRO: A New Visual Tracking Challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(2):335-349, 2016. 3

Siyi Li and Dit-Yan Yeung. Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017. 3

Hamed Kiani Galoogahi, Ashton Fagg, Chen Huang, Deva Ramanan, and Simon Lucey. Need for speed: A benchmark for higher frame rate object tracking. In Proceedings of the IEEE International Conference on Computer Vision , pages 1125-1134, 2017. 3

Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtubeboundingboxes: A large high-precision human-annotated data set for object detection in video. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5296-5305, 2017. 3

Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Arnold WM Smeulders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A benchmark. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 670-685, 2018. 3

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013. 3

Zhengyuan Yang, Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. Grounding-trackingintegration. IEEE Transactions on Circuits and Systems for Video Technology , 31(9):3433-3443, 2020. 3, 4

Haojie Zhao, Xiao Wang, Dong Wang, Huchuan Lu, and Xiang Ruan. Transformer vision-language tracking via proxy token guided cross-modal fusion. Pattern Recognition Letters , 2023. 3, 4

Fenglin Liu, Xian Wu, Shen Ge, Xuancheng Ren, Wei Fan, Xu Sun, and Yuexian Zou. Dimbert: learning vision-language grounded representations with disentangled multimodal-attention. ACM Transactions on Knowledge Discovery from Data (TKDD) , 16(1):1-19, 2021. 3

Wenqiao Zhang, Haochen Shi, Siliang Tang, Jun Xiao, Qiang Yu, and Yueting Zhuang. Consensus graph representation learning for better grounded image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 3394-3402, 2021. 3

Wenhui Jiang, Minwei Zhu, Yuming Fang, Guangming Shi, Xiaowei Zhao, and Yang Liu. Visual cluster grounding for image captioning. IEEE Transactions on Image Processing , 31:3920-3934, 2022. 3

Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280 , 2022. 3

Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and generalization in a 2d world. In International Conference on Learning Representations , 2018. 3

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 787-798, 2014. 3

Yihao Li, Jun Yu, Zhongpeng Cai, and Yuwen Pan. Cross-modal target retrieval for tracking by natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4931-4940, 2022. 3

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021. 4, 10

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14393-14402, 2021. 4, 8

Muhammad Maaz, Hanoona Bangalath Rasheed, Salman Hameed Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Multi-modal transformers excel at class-agnostic object detection. arXiv , 2021. 4

Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16399-16409, 2022. 4

Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 DAVIS Challenge on Video Object Segmentation. arXi , 2017. 4

Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV , 2019. 4

Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark. arXiv , 2018. 4

Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benchmark. International Journal of Computer Vision , 130(8):2022-2039, 2022. 4

Namdar Homayounfar, Justin Liang, Wei-Chiu Ma, and Raquel Urtasun. Videoclick: Video object segmentation with a single click. arXiv preprint arXiv:2101.06545 , 2021. 4

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8748-8757, 2019. 4

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016. 4

Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6047-6056, 2018. 4

Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 510-526. Springer, 2016. 4

Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8668-8678, 2019. 4

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 4

Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E Huang, and Fisher Yu. Tracking every thing in the wild. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII , pages 498-515. Springer, 2022. 5

Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing , 2008:1-10, 2008. 5

Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In European conference on computer vision , pages 17-35. Springer, 2016. 5

Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision , 129(2):548-578, 2021. 5

Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review , 51(3):455-500, 2009. 6

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020. 7, 8

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 7

Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In European Conference on Computer Vision (ECCV) , 2022. 7

Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Son Lam Phung, Ngan Le, and Khoa Luu. Multicamera multi-object tracking on the move via single-stage global association approach. arXiv preprint arXiv:2211.09663 , 2022. 7

Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Ngan Le, Xuan-Bac Nguyen, and Khoa Luu. Multi-camera multiple 3d object tracking on the move for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2569-2578, 2022. 7

Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 8

Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651 , 2022. 8

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69-85. Springer, 2016. 8

Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-tosentence models. IJCV , 123(1):74-93, 2017. 8

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. 8

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 9

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable {detr}: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations , 2021. 9

Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8741-8750, 2021. 9

Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 10

Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 164-173, 2021. 10

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In European Conference on Computer Vision , pages 474-490. Springer, 2020. 10

Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12352-12361, 2021. 10

Jinlong Peng, Changan Wang, Fangbin Wan, Yang Wu, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16 , pages 145-161. Springer, 2020. 10

Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Openvocabulary multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5567-5577, 2023. 10

OpenAI. Gpt-4 technical report. arXiv , 2023. 10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a new paradigm for MOT, using natural language descriptions to track objects of interest in videos. To accomplish it, the authors collected and annotated various object and scene descriptions. Meanwhile, the new evaluation protocols are also formulated following the evaluation of the original MOT. The authors also provide a new method, showing good performance in experiments.\n\nSTRENGTHS:\n* This work has an intuitive and good motivation to solve MOT using natural language. \n* This paper is well organized, which contains collecting new benchmarks, designing effective methods, and conducting experiments.\n* More importantly, the authors provide many visualization results, which are interesting.\n\nWEAKNESSES:\n* First of all, the main idea of ‘Type-to-Track’ has a similar idea to Referring Multi-Object Tracking [1] (released at Arxiv2023.03, accepted at CVPR2023), which both use language description to prompt multiple objects. It is important to discuss and compare the related work.\n* I’m curious about the annotation details of the descriptions. Are they manually annotated? Or they are annotated by the current techniques of GPT?\n* In Table 3, does the number of words represent the number of descriptions? Or the words are words of descriptions?\n* The evaluation of caption prompts confuses me. The crucial words of created captions have one-to-one object matching with the grounded objects. So each object is evaluated individually?\n* This paper is not easy to follow and sometimes overclaimed on some details, e.g., third-order tensor modeling (using three inputs), and correlation simplification (using separate attention). Besides, the authors define many new concepts but lack details and explanations, e.g., the single-stage end-to-end framework. How to deal with new entrance or exit objects?\n* How about the visualization performance of the proposed method?\n\n[1] Referring Multi-Object Tracking, CVPR2023.\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nThe authors have already provided the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically tackles the prompt-based multiple object tracking problem by (1) introducing a dataset for the Grounding Multiple Object Tracking Task (GroOT), (2) introducing a new tacking paradigm (type to track), and (3) developing a new language-based multiple object tracking model (MENDER).\n\nSTRENGTHS:\nThis paper systematically tackles the prompt-based multiple object tracking problem with a variety of relevant contributions. We list the strengths of the paper below, expanding each paragraph with comments on the relevance of each contribution.\n\n## A new tracking paradigm (type to track)\nThis paper introduces the type to track paradigm, which consists in prompting models with text descriptions and obtaining all the tracklets corresponding to it. This paradigm is of great practical importance, as it allows users to query videos with text prompts to retrieve the desired objects.\n\n## A dataset for the Grounding Multiple Object Tracking Task (GroOT)\nThis paper introduces GroOT, a dataset which extends MOT17 and TAO with a variety of text captions to learn models suitable for the type to track task.\n\nGroOT supports 5 settings, including tracking by :\n- category name\n- category synonyms\n- category definition\n- retrieval prompts: 3 retrieval prompts per video are provided. the prompts are sequential and each prompt could refer to multiple tracklets at the same time (one-to-many).\n- tracklet captions: one caption here corresponds to one tracklet, where the caption consistently describes the object throughout the tracklet. In the TAO subset, the caption describes the object appearance. In the MOT17 subset, 2 types of captions are provided for each object, one describing the action and the other the appearance.\n\n## Class-agnostic evaluation metrics\nTo evaluate the type to track performance on GroOT, this paper introduces CA-MOTA and CA-IDF1, two class-agnostic tracking metrics.\n\n## A new language-based multiple object tracking model (MENDER)\nThe authors also introduce MENDER, a proof-of-concept tracker capable of retrieving tracklets based on text prompts.\n\nMENDER is composed of an image encoder, a text encoder, and a transformer decoder.\nThe **image encoder** encodes in parallel the image at time $t$ and the cropped bounding boxes for tracked instances at $t-1$.\nThe **text encoder** is a RoBERTa encoder that encodes the text prompts.\nCross-attention between image features, track embeddings and text embeddings produces the inputs to the transformer decoder, which is trained with a GIoU loss to detect the objects based on the track and text queries.\nThe proposed model is simple enough and addresses the type to track task effectively.\n\nWEAKNESSES:\n## Missing baselines\nThe `category_name` setting could be tackled by any multiple object tracking algorithm. The current version of the paper lacks an absolute estimate of the tracking performance of MENDER in comparison to state-of-the-art tracking algorithms. It is possible that the current design produces suboptimal tracking performance, and only empirical results can show whether this is true or not. The paper would thus benefit from a comparison of MENDER to SOTA tracking-by-detection methods on the `category_name` setting.\n\n## Class-agnostic evaluation metrics\n**Evaluation of retrieval performance.** The proposed metrics focus more on evaluating the tracking performance of retrieved tracklets rather than the retrieval performance itself. Although the limitations section already indicates this as a limitation, I find this to be a weakness since a benchmark for tracking retrieval should specifically provide suitable metrics for the evaluation of such task. \n\n**CA-HOTA.** Although MOTA and IDF1 are established tracking metrics, the community has reached an agreement that they cannot solely represent the performance of a tracker. The HOTA metrics (DetA, AssA, and HOTA) disentangle different components of the tracking pipeline and it would be beneficial for the proposed benchmark to also introduce a class-agnostic version of such metrics.\n\n\n## Missing ablation on the tokenizer and the text encoder\nThe authors adopt the RoBERTa tokenizer and text encoder. However, they do not provide an explanation for choosing it over other popular architectures, e.g. CLIP. The paper would benefit from a comparison to other text encoder architectures and initializations.\n\nQUESTIONS:\nI find the paper well-written and relevant to the community. However, the weaknesses listed above somewhat hinder the contributions of this paper.\n\nIn my opinion:\n1. a comparison on the `category_name` setting to SOTA tracking-by-detection methods on GroOT is essential for this paper\n2. retrieval-specific metrics would complement the contribution of the dataset, which as of now does not allow directly assessing the retrieval task performance\n3. the paper would benefit from ablations on the tokenizer choice.\n\nI would consider increasing my rating and supporting this paper if my current concerns are addressed, in particular points 1 and 2.\n\nLIMITATIONS:\nThe authors have adequately pointed out limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into language-grounded multi-object tracking, a highly important task that has been underexplored due to the lack of suitable datasets.  Firstly, the authors propose a novel dataset that extends existing Multiple Object Tracking (MOT) datasets with additional captions describing appearance or action of each tracklet, as well as retrieval prompts describing main content of a scene. To evaluate language-grounded multi-object tracking, the authors consider five scenarios with different types of text input (prompts) and suggest class-agnostic MOTA and IDF1 that decouple the classification accuracy. Also, the authors introduce MENDER, an efficient single-stage model that autoregressively tracks objects relevant to the prompt. The effectiveness of this model is validated through a comparison with a two-stage baseline model consisting of MDETR and TrackFormer.\n\nSTRENGTHS:\n* This paper addresses a significant problem, aiming to retrieve multiple objects relevant to an input prompt from a video. This setting is novel compared to existing spatio-temporal video grounding methods (e.g., TubeDETR[1]), which are limited to retrieving only a single object. The proposed task can not only serve as a user-friendly video analysis application but also work as a video perception foundation model for fine-grained video understanding.\n* The proposed dataset is significant to further research on video-language model.\n* The proposed diverse scenarios for evaluating grounded MOT are interesting and valid.\n\n[1] A. Yang, et al., TubeDETR: Spatio-Temporal Video Grounding with Transformers\n\nWEAKNESSES:\n## Inconsistent annotations\n  * In the released dataset, some inconsistent annotations are observed. How quality assurance is achieved in the manual annotation process? Is there criteria to accept or reject the captions from the annotator?\n  * In Sec 3.1, L116 states that the appearance-related caption and action-related caption are provided in order if the tracked object is person. However, in the annotations, the captions are randomly ordered. For example, in mot17_train file, object whose instance_id is 170 has captions of [\"person walking on sidewalk\", \"person wearing black jacket\"], where action-related caption is the first in the caption list. \n  * Also, object whose instance_id is 9 has three captions: [\"a person sitting on a bench\", \"child in stroller\", \"baby stroller\"]. And, in TAO dataset, there are some instances with six captions. What is the reaons of such variance in caption annotations?\n  * I would rather suggest to have separate field for appearance described caption and action described caption.\n\n## Heavy dependence on TAO dataset\n* MOT Challenge datasets include more challenging tracking cases caused by complex motion and occlusion, compared to TAO dataset. However, its scale is very small (e.g., only 7 videos for training). \n* I think captioning other human-centric MOT datasets with complex motion would improve the importance of the dataset. For example, human-centric datasets with dense FPS video (e.g., DanceTrack[2] and HiEve[3]) are the possible candidates.\n\n## Unclear evaluation metrics\n* In the paper, I cannot find the details of matching GT-prediction pairs which are necessary to compute FN and FP.\n* What is the reason of cls exists in Eq. 1 even though we are not considering class of tracklets? Isn't that the class of tracklet do not exist in the case of grounded MOT?\n\n## Lack of experimental details\n* Unclear details about implementing the two-stage baseline model which is based on the MDETR and TrackFormer. TrackFormer jointly performs object detection and tracking, but how MDETR is integrated into TrackFormer to achieve grounded object detection?\n\n[2] P. Sun, et al., DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion, CVPR 2022\n[3] W. Lin, et al., Human in events: A large-scale benchmark for humancentric video analysis in complex events, arxiv 2020.05\n\nQUESTIONS:\nMajor concerns are raised in the weakness section.\n\nHere are some minor questions:\n* Why only text prompt for every frame exists, but there is no prompt-tracklet paired annotations in the MOT files?\n* In Table 3, dataset statistics for MOT17 videos are not correct. As far as I know, there are 7 unique videos for the training and test subsets, respectively. Also, your released dataset seems using the GT tracklets from the unique videos.\n* How do you have the mot17_test annotation file? Is the tracklet obtained from GT or just prediction value?\n* TAO dataset is the most important one as the proposed dataset is built on top of this one, but why it is not included in Table 1?\n* Why do you design the MENDER as online tracking model that runs autoregressively, instead of predicting multiple spatio-temporal tubes at once?\n* In Table 4 and 5, is mAP the absolute value or percentage value?\n* In Table 4, some bolded values are not the best in the metric.\n\nLIMITATIONS:\nThe limitations and future works are well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel task and dataset that involves tracking multiple objects in videos using language prompts. It proposes two evaluation protocols: retrieval prompts, where the prompts can change during the tracking process, and caption prompts, which remain consistent. The paper also presents class-agnostic evaluation metrics by modifying MOTA and IDF1. Additionally, a novel framework called MENDER is introduced to address the grounded MOT task.\n\nSTRENGTHS:\n1. The significance of the proposed task and dataset is well-defined, as existing tracking datasets using language prompts have been focused on single-object tracking (SOT) tasks.\n2. The evaluation protocol for the proposed dataset consists of five distinct protocols, each with clear explanations of their purpose. This dataset stands out from previous tracking datasets due to its diverse range of prompts.\n3. The proposed MENDER framework demonstrates efficient and effective performance in addressing the grounded MOT task.\n4. The paper is overall well-written and easy to follow.\n\nWEAKNESSES:\n1. Referring video segmentation datasets like Ref-YTVOS and Ref-DAVIS are also related to this grounded MOT task since they also aim at tracking a referred target in a video. I think these datasets should also be mentioned in the paper as a comparison benchmark.\n2. Moreover, I think Refer-KITTI [1] could also be discussed in the paper as it also utilizes referring expressions for multi-object tracking.\n3. The experimental comparisons in the newly proposed dataset are limited. It would be beneficial to include additional methods such as TransRMOT [1] and popular MOT baselines like ByteTrack [2] for a more comprehensive evaluation of the dataset's performance. Adding these comparisons would provide stronger validation for the dataset.\n\n[1] Referring Multi-Object Tracking, https://arxiv.org/pdf/2303.03366.pdf\n\n[2] ByteTrack: Multi-object tracking by associating every detection box, ECCV 2022\n\nQUESTIONS:\nAs discussed in the weaknesses, I hope the authors would supplement some dataset comparisons and experimental validations.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors have created a new dataset by combining existing Tracking datasets and adding language prompt annotations. They also proposed a method by extending and modifying MDETR and Trackformer.\n\nSTRENGTHS:\n1. The dataset quality looks sound, diverse, and sufficiently large enough to train Deep networks.\n2. The dataset is well documented and the task is clearly defined.\n3. The proposed solution is simple and should be easy to extend by new researchers.\n\nWEAKNESSES:\n1. The authors claim that Type-to-Track is a novel paradigm. This is incorrect as there exists Datasets, e.g., Refer-Youtube-VOS, Ref-DAVIS, Ref-KITTI, having the same challenge.  \n2. The related work in incomplete. The authors fail to compare their proposed dataset with existing datasets from Video Instance Segmentation(VIS) literature. Note that, MOTS and VIS are the same task with different nomenclature.\n3. The modeling part is unnecessarily written in a complicated way. It is a simple extension of DETR architecture to model the specific solution. Instead of complicating the method with notation, The authors should try to simplify the method section. \n4. There are too many evaluation metrics. There are 5 different sub-categories of the same task with separate evaluations on the 3 base datasets. So there are in total, around 12*6=72 different evaluation metrics.\n\nQUESTIONS:\n1. How is this dataset different from Refer-Youtube-VOS, Ref-DAVIS, and Ref-KITTI? \n2. How did you evaluate MOT17 and MOT20 datasets? The test ground truth is not public for these datasets. What is MOT17 subset in Tab. 4 and 5?\n3. Why is there  no unifying evaluation metric. The authors report evaluation separately on all the different datasets they worked on. A new dataset needs to be either compatible with previous ones, or propose a new metric. Without a single metric, new line of researchers would find it difficult to adapt the dataset.\n4. How many human hours was needed to annotate the dataset?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}