Dense Video Object Captioning from Disjoint Supervision
Authors: Xingyi Zhou
Anurag Arnab
Chen Sun
Cordelia Schmid
Venue: ICLR 2025 Spotlight (2025)
Keywords: object captioning
video
tracking
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We propose a new task and model for dense video object captioning -- detecting, tracking and captioning trajectories of objects in a video. This task unifies spatial and temporal localization in video, whilst also requiring fine-grained visual understanding that is best described by natural language. We propose a unified model, and demonstrate how our end-to-end approach is more accurate and temporally coherent than a multi-stage pipeline combining state-of-the-art detection, tracking, and captioning models. Moreover, we propose a training strategy based on a mixture of disjoint tasks, which allows us to leverage diverse, large-scale datasets which supervise different parts of our model. Although each pretraining task only provides weak supervision, they are complementary and, when combined, result in noteworthy zero-shot ability and serve as strong initialization for additional finetuning to further improve accuracy. We carefully design new metrics capturing all components of our task, and show how we can repurpose existing video grounding datasets (e.g. VidSTG and VLN) for our new task. We show that our model improves upon a number of strong baselines for this new task. Furthermore, we can apply our model to the task of spatial grounding, outperforming prior state-of-the-art on VidSTG and VLN, without explicitly training for it. Our code is available at https://github.com/google-research/scenic.

Published as a conference paper at ICLR 2025
DENSE VIDEO OBJECT CAPTIONING FROM DISJOINT SUPERVISION
Xingyi Zhou" Anurag Arnab" Chen Sun
Cordelia Schmid
Google DeepMind
ABSTRACT
We propose a new task and model for dense video object captioning detecting, tracking and captioning trajectories of objects in a video. This task unifies spatial and temporal localization in video, whilst also requiring fine- fine-grained visual under- standing that is best described by natural language. We propose a unified model, and demonstrate how our end-to-end approach more accurate and temporally coherent than multi-stage pipeline combining state-of-the art detection, track- ing, and captioning models. Moreover, we propose a training strategy based on a mixture of disjoint tasks, which allows us to leverage diverse, large-scale datasets which supervise different parts of our model. Although each pretraining task only provides weak supervision, they are complementary and, when combined, result in noteworthy zero-shot ability and serve as strong initialization for additional finetuning further improve accuracy. We carefully design new metrics captur- ing all components of our task, and show how we can repurpose existing video grounding datasets (e.g. VidSTG and VLN) for our new task. We show that our model improves upon a number of strong baselines for this new task. Furthermore, we can apply our model to the task of spatial grounding, outperforming prior tfat-oof-the-ar on VidSTG and VLN, without explicitly training for it. Our code is available at http://ggiibb.oommggggllrreeeeee
INTRODUCTION
Powered by gigantic datasets and models, language is becoming the output modality of the most
capable artificial intelligence models (Team et al. 2023 Alaaraa et al.
2022] Ouyang et al.) 2022]
Li et al. 2023; Liu et al.) 2023 Tong et al. 2024; 4; Li et al. 2024a). Language unifies different
2023a). is more descriptive tasks with the same output space (Rafell et al., 2020 Chen et al.
than discrete class labels (Wu et al. 2022a Long et al.) 2023). and naturally facilitates zero-shot prediction of novel tasks (Radford et al.) 202 Brown al. 2020). Inspired by advances in natural language understanding, the vision community has explored language in a number of tasks including image captioning (Chen et al., 2015), dense image captioning (Krishna al.) 2017b), question answering (Antol et 2015). video captioning (Monfort et al. 2021) and representation learning (Radford et al. (202)). However, likely due the scarcity of large-scale, aligned training data, we are not aware of any existing single vision-language model that unifies both fine-grained spatial- (by detecting objects) and temporal- (by reasoning across time in videos) understanding.
In this paper, we propose a new task and model for dense video object captioning (Dense VOC) the task of generating captions of trajectories of all objects from video (Fig. I]. Dense VOC requires understanding across space, time, and language (Fig. 2). and is therefore a superset of
existing vision tasks, namely object detection (Everingham et al. 2015 Lin al., b, multi-object tracking (Dendorfer et al., (221]] Dave et al.) (2020) and captioning (Chen et al. 2015). It offers broad range of applications, such as sports analysis, wildlife monitoring, and behavioral analysis.
A prominent challenge for training our model is that datasets with captioned trajectories are scarce. However, annotations for each sub-task, or even each combination of the sub-tasks, are abundant. For example,
train object proposal component using image-level object detection labels from COCO (Lin et al. 2014), and the captioning component from video-level captioning datasets like SMiT (Monfort et al.) 20211. These disjoint training tasks are complementary, and in combination supervise our entire model. This enables us to perform our Dense VOC task in a zero-shot manner, and we show that we can achieve noteworthy performance despite not having access to any full,
Equal contribution. (zhouxy, aamah) @ @google.com

---

Published as a conference paper at ICLR 2025
child gray bounces basketball outdoors
There is bench in the background
child blue clothes plays basketball with another child
A brown basketball
Figure 1: Overview of the dense video object captioning (Dense VOC) task. Given a video, we predict object trajectories (identities denoted by colors) and their natural language description. We show a video from the VidSTG (Zhang et al. 2020) validation set.
captioned object trajectories during training. Furthermore, this pretraining serves as a powerful initialization for finetuning on the full Dense VOC task, where limited annotations are available.
Another challenge in our task is to produce holistic and consistent captions for objects across frames. Note that a baseline of applying a strong, dense image caption- ing model per-frrame, and then linking objects together is poorly suited to this scenario: the captions at each frame are likely be different due subtle appearance changes across frames. This motivates our end-to-end trained model, which includes a novel end-to-end track- ing algorithm that aggregates features of the same object across time, enabling the subsequent captioner to leverage global features to produce coherent captions.
Cbjject dotection Space
Object tracking
Object captionee
Dense vod
Video recognitiin
Siee
Lengage
Image Captioning
Video captioning
Although we are the first to our knowledge to study Dense VOC, we can still repurpose existing video grounding datasets for evaluation and domain-specific finetuning. We use VidSTG (Zhang et al., (220) and VLN (Voigtlaender ct al. 2023), originally designed for spatiotemporal sen- tence grounding: Instead of finding object tube given sentence query (grounding). we predict object trajectories directly and use the sentence queries the ground truth captions. In addition, we show that our generative model trained for Dense VOC can perform grounding by simply selecting the bounding boxes with the maximum likelihood of producing the query sentence. We also develop a new metric that jointly measures captioning, detection and tracking accuracy by extending HOTA (Luiten et al., 2021), the most popular metric for multi-object tracking.
Figure 2: Overview of Dense VOC. Our problem involves understanding across space, time, and language, and thus en- compasses other vision tasks, which typ- ically consider one or two of these axes. We show these subtasks are complemen- tary, and pretraining on them enables zero-shot generalization to Dense VOC.
Experiments show that our end-to-end trained Dense VOC model outperforms baselines consisting of strong, per-task models by a substantial margin, producing more accurate and inherently temporally consistent captions. Moreover, we achieve significant improvements from our disjoint, multi-dataset training. We additionally show how we can readily apply our model to related domain datasets: by finetuning our model on a recent person tracking and captioning dataset, BenSMOT (Li ct al. (024b), we outperform prior work by 18.2 points. Furthermore, by applying our generative captioning model to the discriminative grounding task, we are able to outperform dedicated spatial grounding models on both VidSTG and VLN. In summary, we propose the following contributions:
1. We propose the new task of Dense Video Object Captioning. We propose novel evaluation metrics, and repurpose existing grounding datasets for evaluation.
2. We design end-to-end architecture for our task, with a novel tracking algorithm and feature aggregator that ensures temporally consistent captions. Unlike conventional offline trackers, our tracker is trained end-to-end with the model and produces long-tem term-termm trajectory features for subsequent captioning.
3. We show our model can be trained without full annotations for the task, with mixture of disjoint datasets which supervise different parts of our model.
4. We further show how our models generalize to downstream video grounding tasks, achieving state-of -fe-----thhrrr- art results on two datasets, without explicitly being trained for grounding.
5. Moreover, we significantly improves the state-of-the-art on the BenSMOT dataset [i et al. (024b) for Semantic Multi-Objeet Tracking.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Image captioning (Chen et al., 2015; Anderson ct al., 2018 Xu et al., 2015 Rennie et al. describes the content of an image with language. State-of- oh-the-art art methods map the input image to
2017
output text by using multi-
models (Jiang et al. 2020 Desai & Johnson, 2021 [i et al.
2020
Zhang et al., 2021a a Li et al. [023] Yu et 2l.222022) pretrained on large datasets (Sharma et al.
2018
Radford et al., 2021). ). For example, GIT (Wang al., 2022) simple forwards vision tokens from
ViT encoder Dosovitskiy et al. 202T) to an auto-regressive language decoder (Vaswani et al.
2017
Devlin et al. 2019). Similar ideas apply to video captioning (Xu et al., [016)] Zhou et al. 2018
Monfort et al. 222TT) by concatenating (Wang et al. 2022) or pooling (Yan et al. (022) features
from each frame, before feeding them an auto-regressive text decoder. Our work builds on existing captioning architectures (Wang et al. 2022), and extends them to trajectory captioning using our
end-to-end model and weak supervision (Monfort et al. 2021 Krishna et al. [017b] Lin et al. 2014).
Dense object captioning in contrast, detects objects in an image and describes them with text (John- son et al. 2016 Li et al., 2019;; Shao et al. 2022; Wu al., 2022a). It was popularized by the 201 dataset, which contains full annotations for the task. Early work, DenseCap (Johnson et al. 2016) used one-stage detector (Redmon et al. 2016) followed by
Visual Genome Krishna et al.
an LSTM text decoder (Hochreiter & Schmidhuber] 1997) on dense feature maps. Most recently.
GRiT (Wu et al. 2022a) built upon the state-of-the-ee image captioning architecture of GIT (Wang
et al. 2022), and generated object captions, also with a transformer decoder (Vaswani et al. 2017). from Rol-pooled (He et al., 2017) image features. Our model advances architectures like GRiT
to videos and incorporates end-to-end tracking. We also note that dense video captioning in the literature refers to the task of localizing and captioning multiple events temporally in videos (Krishnal al. 2017a; Zhou et al.) 20118: Wang al. [02 Yang et al.) (2023)). Our task, in contrast, involves tracking and captioning objects in a video, and therefore requires spatial localization, which is why we name our task "dense video object captioning".
Multi-object tracking detects objects and track them with a consistent identity label. The predomi- nant approach is racking-after-deter
(Bewley et al. 2016 Zhang al.] [021b] Du et al. 2021).
i.e. first running detectors on each frame and then using a separate tracker to link them. While this works well for existing benchmarks with only few classes (Dendorfer al. 202T Geiger et al.
2012; Yang et al., 2019), it is more challenging in our case: we need tracks before captioning to have a single, consistent textual output for the whole trajectory. Thus, our work follows end-to-end multi-object tracking (Cheng et al. 2022, Li et al. 2022a Wang et al. 2021c| /Zhou et al. 2022b). We adopt global tracker GTR (Zhou et al. 2022b). which casts tracking as pairwise association among all objects within a video. Whilst GTR applies a sliding- window-based identity association algorithm during inference as a procepsoc step, we design an efficient algorithm to perform this process end-to-end. This is necessary for our task, since our trajectory features are used by a subsequent captioning module which is trained jointly. We are not aware of prior work which efficiently assigns object identities and corresponding features tracks, and trains end-to- and-to-end through this process. Finally, note that video object tracking and segmentation (Yang et al. 2021 (223bb Yang& & Yang 2022 Cheng & Schwing. 2022 Cheng et al. 2024) focuses on following only single object which is given in the first frame (Perazzi et al. |2166] Xu et al.) 2018). This is therefore a different setting from our task of detecting, tracking and captioning multiple objects.
Video object grounding (Zhang et al., 2020; Voigtlaender et al., (023) finds a spatio-temporal
tube given a video and query sentence as inputs. Existing, discriminative methods (Zhang al. 2020: Yang et al. 2022 Jin et al. 2022, Su et al. 2021) co-embed visual and text inputs, and use the sentence feature to find the corresponding object. In contrast, we use our generative language model for this task by selecting the object with the highest likelihood of producing the query. To our knowledge, we are the first work to explore the alternate paradigm of generative models for this task. Finally, we note that these tasks are also related to video-referring segmentation (Bellver cet al. 2020 Wu et al. 2022b, Yu et al. (0166) which grounds textual queries to segmentation masks. Segmentation, however, is not the focus of our work.
Concurrent to our work, BeyondMOT (Li et al., 2024b) proposes an video object tracking and captioning benchmark and model. We highlight two differences: 1. et al. (2024b) uses frame- by-frame tracker similar to our basclines (Tab. (2) and we propose a novel end-to-end tracker. 2. Our work aims to track and caption all objects in the video, while Li et al. (2024b) handles only
3

---

Published as a conference paper at ICLR 2025
Fubyrt
Augglon
Tracking
Trajectory captioning
Class-egnoslic detector
H (Sec: 3.2)
Grouping (Aig
Aogregator (Sec 3.5)
Language Decoder
F
Roll features
Association matrix
Identity
Trajectory features
Object
Video
9
y: dog phokting up toy
92
32
Figure 3: Overview of our model. Our end-to-end model has three modules: First it produces object proposals per-frame using class- class-agnostic detector (left, trained with detection loss, Lobject). These object proposals are then passed to an end-to-end tracking module that groups objects into trajectories (middle, trained with association loss, Laxsoc). The identities produced by the tracking module are used to aggregate features which are then fed to language decoder to produce the final caption (right, trained with caption loss Leaption). Our model can be trained end-to-end with partial supervision on different and disjoint datasets to provide zero-shot Dense VOC capabilities.
persons. As result, task is much more challenging, and show our model yields superior performance on their benchmark. OW- VISCpp (Choudhuri et al. 2024) on the other hand augments video segmentation model, Cheng et al. (0222), with language model (OPT with 2.7 billion parameters (Zhang et al. (2022a)) head for video segmentation and captioning. In contrast, our model trained flexibly using our disjoint pretraining, which enables us achieve better detection and tracking performance whilst still using a substantially smaller model.
METHOD
As shown in Fig. S our end-to-end model consists of interlinked heads for object proposal, tracking and captioning the resulting trajectories. Before introducing our novel components, we review prior techniques for captioning and dense object captioning in images (Wu et al. 2022a Wang 2022).
3.1 BACKGROUND
Image captioning maps input image, I E #xxx3 to caption c= (y1, 3/.......................... ................................ Dn,) which is sequence of up to n, text tokens from a given vocabulary. The minimal set of components is an image encoder, followed by text decoder VVasaani et al.] 2017). The encoder maps the input image I, to feature representation E IR Xd consisting of nt, tokens with dimensionality d. The subsequent text decoder is auto-regressive (Graves 2013 predicts the next text token, 2/., as a function of both the image features, f, and previously generated text tokens, Vo--1. denoted by y! Decode No.-i). seginning-of-sentenc Note that the first step of decoding begins with yo BOS, special beginning-of of-sentence token, and the caption ends when the ond-of-sentence token, EOS, is output by the model. This simple image captioning model has been demonstrated be effective and scalable by GIT (Wang et al., 2022). achieving fftt--of-the-art results across number of captioning datasets.
GRiT (Wu al., 2022a) extends the approach further to dense object captioning of images: Here, the authors use an object proposal network (Zhou et al. 2019) to produce a set of K class-agnostic bounding boxes, b,. b2. b2..... bx. Features corresponding to each of these objects are obtained using RolAlign (He et al. (017). resulting in localized feature. fa grxrxd where r 7 is the output resolution of RolAlign. Each of these grid features flattened into fre IRr and decoded independently by the text decoder, as done in GIT. Therefore, the loss used to train a GRiT model consists of L Lobject + Leaption where Leaption is cross-entropy loss over all text tokens in the vocabulary, and Lobject consists of bounding box regression and objectness terms, standard in object detection literature (Zhou et al., 2019; Ren et al., 2015 Lin al., 2017).
We now describe how we extend object captioning to videos by tracking object proposals over time (Sec. (3.2 and aggregating trajectory features and captioning them (Sec. in an end-to-end fashion. Section 3.4 explains how we train our model, whilst Sec. 3.5 describes how we apply our model directly to video object grounding tasks.
3.2 END-TO-END TRACKING
As shown in Fig. (left), we first produce object proposals separately for each frame. Tracking then aims to assign each object in each frame unique trajectory identity 5 N. We define fl E Rr"x as the ROI feature of object proposal & in frame F [fi the concatenation of all object
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: Identity assignment from association matrix. This greedy algorithm can be implemented efficiently on accelerators, enabling end-to-end training.
Input
Association Matrix A E TKTTT // I': num. frames. K: num. objects per frame. Hyperparameters: Association score threshold
Output
Identities for each object 5 E N7'K
M - T-Tx K
// Number of total objects. // Preprocess A to ensure object pairs in the same frame have score of O. // Binary matrix for possible merges. I/ Initialize output identities. shape (M, ) Initialize IID count.
A < preprocess(A)
A < (A 2 a).yytyp(loool
8 < zeros(M
id_ count + 0
while A.any() 0 do
track len l
I/ Number of objects in each merge. Find the longest track to merge. // Create a new identity. Assign the current track new ID using A., // Remove merged indices. "I"" is logical
f track srack_len.argmax/
id count +- id count + 8 8+ id count Å. A A A.
end return 8
features in the video. Let M = [F] = Σi= K, as the total number of objects in all frames, where K, is the number of object proposals at the hh frame. Thus, we have F Mx
From these object features, F, we predict global association matrix, A MxM, where A.j = if the objects denoted by the 7th row and jth column are from the same trajectory (Fig. 3 middle). Otherwise, Aij O means that they are from different trajectories, or one of them is the background.
We use transformer module, H, with two self-attention layers, similar to Zhou et al. (2022b), to predict the association matrix A (H(F)), where is the sigmoid activation. Given the object trajectory annotations, we construct the ground truth association matrix A for A, where A.j = 1 if and only if row and column j of A are matched to the same ground truth trajectory using an Intersection over Union (IoU) criteria of 0.5. The training loss Lassoc for this module is then a binary cross entropy between A and A, Lassoe
M Ei BCE(Ar..
After constructing our association matrix, A, we need to aggregate object-level features according to identities 5 = [ot1==1.e11 generate trajectory-level captions for the next captioning stage. Here, 5% denotes the identity of the k--th object proposal in the t-th frame. We design a greedy grouping algorithm (Alg. [ operating on A to obtain 5. Concretely, we greedily extract the longest trajectory from untracked objects, until there are no possible associations left (indicated by the association score being above a threshold 0). This guarantees each trajectory has most one object in each frame. This algorithm can be implemented efficiently on accelerators, allowing us to backpropagate through it.
As aforementioned, prior trackers (Zhng et al. 2021b Zhou et al., 2020 2022a) do not explicitly perform identity assignment within the model, but rather as a is the final output for such methods. Our work efficiently assigns object identities to tracks in an end-to-end trainable network, which enables us to perform joint trajectory-level captioning training as described next.
step since tracking
3.3 TRAJECTORY CAPTIONING
Our end-to-end tracking module produces object features, fy (we omit the frame index below for clearer notation), paired with their identities, 8g, which denote their correspondence over time. We now describe two methods for aggregating features along this trajectory in order to caption it.
Soft aggregation. A straightforward way to leverage object features over time is to compute a weighted sum to combine them into a single, global trajectory feature. We observe that the association matrix, A (Sec. 3.2) already serves summation weight. Specifically, we set G [ F,
A
where denotes matrix multiplication, and || || normalizes A by rows. Each row of G. g& R xd therfore denotes an aggregated feature over its trajectory for object k.
Hard aggregation. An alternative weighted temporal averaging is concatenate and construct new trajectory features. Let f, = (fxx ) be the set of all object features with identity 7. We note f, can be as long as the entire video, and thus it may be expensive to directly use f,. Therefore, uniformly
5

---

Published as a conference paper at ICLR 2025
Dataset
Annotation type
Train set size (10*) Lobject Losson Leaption
COCO (Lin et al. 2014)
Image detection
1 118
VG (Krishna al. 20TTb)
Image object captioning
70
SMiT (ontorr et al.
[2021]
Video captioning
480
Aug- Sag-CoCO (in et al. 2014)
Video object tracking
118
Table 1: Datasets for pretraining. We supervise different losses based available annotations.
sample subset of object features from the trajectory, denoted gr UniformSample(ff, m), where g. E R inspired by Wang et al. (2022) m is the number sampled frames, and we set m = following ablations in Appendix C.2
The vrajectory-aggregate features for each object, gk. are then autoregressively decoded into output captions for each object, Ys- This follows Sec. 3.1 where Vs. Decode(ga- Decooee(Bx..eeDx=-1 Ye,0.i-)) Note that the language decoder has the same parameters as in single-frame object captioning, but processes more input tokens. Therefore, train it in the same manner with softmax cross-entropy loss over the vocabulary of text tokens, denoted by Leaption.
3.4 PRETRAINING WITH DISJOINT SUBTASKS
As shown in Fig. 3. our model is trained with the loss function, L = Lobject Lassoc + Leaption When we have full Dense VOC annotations, which supervise each component of our model we can train our entire model end-to-end. However, to leverage more weakly-labeled data, we can also decompose Dense VOC into subtasks, and use each subtask to supervise the relevant part of our model using the available annotations as shown in Tab. IT This approach also enables to perform our final task in a zero-shot manner (i.e. without training on any full Dense VOC annotations).
Object detection. Using detection datasets for images, with Lobject. We use COCO (Lin et al. 2014) as it is the most popular dataset for this task.
train the object proposal generator
Dense captioning in images. Dense object captioning datasets of images allow to train both the object proposal generator and the text decoder, by supervising Lobject and Leaption Here, we use Visual Genome (Krishna al. 2017b), the largest dataset for this task.
Global video captioning. Video captioning datasets help us to reduce the domain gap to our final task by also training on video. In particular, we use Spoken Moments in Time (SMiT) (Monfort
et al. 2021) which is the largest dataset for this task and contains narrations for short clips (roughly 3 seconds). As there no object annotations, but only video-level captions, we construct an object proposal from the entire frame and caption that with our text decoder, applying L Leaption This
approach is inspired by prior work on weakly-supervise object detection (Zhou al. 2022a
Bilen
& Vedaldi 2016, Arnab et al.) 2020).
Tracking. Training the tracking module of our network (Sec. 3.2) requires annotations that associate detections of an object identity throughout the video. We found that existing tracking datasets either have too limited vocabularies for general objects (MOT (Dendorfer et al. (021)) l), KITTI (Geiger et al. 2012), YouTube VIS (Yang et al. 2019)). or are too small (TAO (Dave et al. 2020) and UVO (Wang ct al. 2b2 label 600 and 000 videos respectively), and thus giving unsatisfactory results in our setting (Appendix C.3). As result, following existing work (Zhang et al. (2021b] Zhou et al.) 2020), we instcad augment image datasets into tracking ones by applying two different data augmentations to the same image, and then linearly interpolating the frames in between form pseudo-video. In particular, we augment COCO (referred to as Aug-COCO (Zhou et al. 2020)). This enables us to apply Lassoc and Lonject when training our model.
3.5 APPLICATION TO VIDEO OBJECT GROUNDING
The task of video object grounding (Zhang et al. 2020 Voigtlaender et al. 2023) consists of two inputs: a video, V, and sentence query, a. The output is a sequence of bounding boxes, [6*, b.+t1 ...., 66), corresponding to the sentence query, where and are the indices of the start and end frames respectively.
Our model, however, generates captions, the output, rather than requiring it as an input. To apply our model to grounding, we follow an analogous approach to prior works that performed closed-set image classification with captioning models (Alayrac al., 2022] Chen et al. (023b): we evaluate the likelihood (i.c., exponential negative cross-entie cross-entropy loss) of the sentence query, c, for each of the object trajectories produced by model. In practice, we find that instead of just taking the

---

Published as a conference paper at ICLR 2025
object trajectory with the highest sentence- liweence-i,e,hhodd, achieve higher accuracy by weighting the likelihood by the detection score, Sk, from our object proposal module. Thus, given bounding boxes, trajectory features and detection scores, {(bf sk 8k) )= we choose the bounding boxes with the highest weighted likelihood:
k" arg max (sk exp(p Leaption (Decode)))n)))
8' b²= bl.-
4 EXPERIMENTAL EVALUATION
As we are proposing a new task, there is no dedicated dataset or evaluation metric for dense video object captioning for all objects. Fortunately, existing video grounding datasets (Zhang al. 2020 Voigtlaender ct al. 2023) have annotations for object trajectories and their captions, allowing us to repurpose them for Dense VOC, as defined next. We also report results on concurrent person-focused video object tracking and captioning benchmark, BenSMOT(Li et al., 2024b).
4.1 DATASETS
VidSTG (Zhang et al. 2020) was originally created for spatio-temporal sentence grounding, but can be used for Dense VOC: Each video annotates multiple textual queries and their corresponding spatio-temporal tubes. By aggregating these across all videos, we obtain the paired trajectory-caption annotations that we need for training and evaluating our model.
VidSTG has exhaustive trajectory (i.e. bounding box and tracking) annotations for all objects (hang al. 2019). but not all objects are used in grounding, and thus not all objects have captions. We account for this fact in both training and testing. Specifically, we do not compute Leaption on objects without caption annotations, and also exclude them during evaluation (see Sec. [.2]. In particular, when a prediction is matched to ground truth without caption annotations, we do not evaluate its captioning metrics, but still evaluate detection metrics. The dataset contains 5,436 training videos and 602 validation videos, with each video being at most 200 frames long. We use the declarative annotations from the dataset containing 19,000 captioned trajectories for training, and report results on the declarative validation set.
Video Localized Narratives (VLN) /Voiglaenner et al., 2023) augments existing datasets by narrating the "actors" in video. We therefore these narrations our target captions. We use the subset from the UVO dataset (Wang et al. 2021b) as UVO has exhaustive detection and tracking annotations for all objects. Like VidSTG, the captions are not exhaustive for all objects, so we exclude objects without captions in both training and evaluating the captioning module. Each video has bounding box annotations for sparsely sampled frames, and thus we train and evaluate these frames. The dataset contains a total of 5,136 training and 2,451 validation videos.
BenSMOT Li al. 2024b) contains person bounding boxes trajectories and their manually- annotated captions for 3292 YouTube videos. The dataset has in average 2.2 trajectories per video.
4.2 EVALUATION METRICS
Captioned-HOTA (CHOTA). Our primary metric, CHOTA, builds on Higher Order Tracking Accuracy (HOTA) Luiten et al. 202T) which is now the most popular metric in multi-object tracking - by adding a captioning term. HOTA decomposes tracking into two subproblems: detection and association, with the final score being the geometric mean of detection accuracy (DetA) and (TP)
Association Accuracy (AssA): HOTA = VDetA AssA. Here, DetA TP and AssA
averages the "Association IoU" over true-positives, AssA [TP ((wyerr Ass-loU(x, y)), where (a, 3) are the matched prediction-groumm truth box pairs in each frame. Note that HOTA computes the DetA and AssA for each detection in each frame, rather than for each trajectory, as the overall trajectory performance is implicitly measured by the association of detections over time. Moreover, it considers all possible trajectory matches that can be made simultaneously (Sec. 7 of Luiten et al. )2
Our task consists of captioning, detection and association. Therefore, we also define an additional "Captioning Accuracy" (CapA) term as:
CapA
1
(METFOR(Z,)) v) + CIDEr(Z, v) + SPICE(,, v)), (E.JETP
Σ
(2)
3|TP"
7

---

Published as a conference paper at ICLR 2025
CHOTA
DetA
AssA
CapA
Consistent captions
Per-frame cap. w. IOU tracker
49.9
64,4
52.2
37.1
x
Per-frame cap. w. FairMOT (Zhang et al. 2021b
51.2
63.4
57.2
37.0
3 Per-frame cap. w. ByteTracck (Zhang el al. 2022b)
52.3
64.2
60.2
37.1
X
4 Middle-frame cap. w. ByteTrack (Zhang ct al. 2022bb)
50.7
64.2
60.2
33.8
5 Ours, soft aggregation
54.6
64,4
65.9
38.4
6 Ours, hard aggregation
54.9
64.2
65.9
39.1
Table 2: Comparison of our end-to-end model to per-fask baselines on VidSTG validation. Our models are based on #2 of Tab. right. The image dense captioning models used in the baselines (rows # Iff4)) are trained on the same datasets, and run off-the- trackers as post-processing. Our end-to-end approach improves across all metrics, and produces temporally consistent captions.
which uses three popular image-captioning metrics (Chen et al. 2015). and TP' are the true- que-positive detection pairs that have caption annotations (as discussed in Sec.|4.] Note that for compatibility with HOTA, we follow DetA and AssA and compute CapA separately per-object on each frame. The final metric is then CHOTA = y DDetAA AssA CapA, effectively adding captioning term to the HOTA metric. We include further details and code in Appendix B along with results using the image dense object captioning metrics, MAP-METEOR (Appendix
4.3 IMPLEMENTATION DETAILS
Our implementation is based on the public release of GRiT (Wu et al. 2022a). GRiT uses a ViTDet- Base (Li al. 2022b) backbone (initialized with CLIP (Radford et al. (021)). ). CenterNet (Zhou
et al. 2019) object proposal network and Rol Head, and a vandomly-initalized text decoder.
We first train our model for general Dense VOC on large-scale disjoint datasets (Sec. 3.4). During disjoint pretraining, we sample batches from different datasets with an even ratio, (1: I: 1), thus avoiding additional hyperparameters. For video datasets, we sample 8 frames for a video and use a local batch size of 1. For image datasets, we use local batch size of 8. We train our model on 32 GPUs, which means we have an effective batch size of 256 images or 32 videos.
We evaluate the models on the two fully-annotated datasets (Sec. 4.1) in zero-shot and hull-finetuning setups. For VidSTG, we sample 16 frames during training, and then run on all 200 frames during testing. For VLN, we use all 3 annotated frames in both training and evaluation. In both cases, we use an input size of 384 x 384. During inference, we threshold the outputs of our object proposal module with score of 0.5, and only track the remaining objects. We include exhaustive implementation details and hyperparameters in Appendix B.2 with the full code. We report runtime in Appendix B33
4.4 ANALYSIS OF END-TO-END TRACKING
We first study the benefits of our end-to-end model in Tab. 2 We do this by comparing to multiple, strong baseline models running in sequence. Concretely, we use the state-of-the-ar image-dense object captioning model |w et al. (2022a) followed by tracking as post-processing step. We use trackers ranging from a simple lo based tracker /Wu et al. (2019) to more recent, sophisticated methods like FairMOT (Zhang et al.) 2021b), and ByteTrack (Zhang al. 2022b).
As the baseline predicts captions independently on each frame, the caption is not consistent over the entire trajectory. Therefore, we consider additional baseline where we only use the caption from the middle frame of the trajectory. Finally, note that as our baseline captioner is pretrained on Visual Genome, and then finetuned on individual frames of VidSTG, it has been trained on identical data to our model, allowing us to make fair comparisons.
As shown in Tab. 2. per-frame captioners followed by offline trackers produce temporally inconsistent captions (#1 #33). Naively selecting the caption from the middle frame as the trajectory-level caption produces temporally consistent captions, but comes at the cost of captioning accuracy, as a single frame may not be representative of the entire event (#4). Both variants of our model (45 and #6) improve tracking quality substantially, as shown by their large improvement on AssA, demonstrating the benefits of end-to-end training and incorporating temporal information. Our model improves on CapA too, showing that improved object trajectories provide better features for subsequent captioning. Finally, we note that the quality of the initial detections at each frame, measured by DetA, does not really change between the baselines and our method. This does, however, show that training our model jointly with multiple loss functions does not compromise performance on individual tasks.
Overall, our end-to-eni model (#6) improves the CHOTA by 2.6 points over the best baseline (#3). As hard aggregation performs slightly better, we use in our following experiments.
8

---

Published as a conference paper at ICLR 2025
#COCO VG SMIT
Aug
VidSTG (zem-shot)
VIN (zero-shot)
VidSTG (finetumed)
VIN (finetuned) CHOTA DetA AssA CapA
COCO CHOTA DetA AssA CapA CHOTA DetA AssA CapA
CHOTA DetA AssA CapA
47.8
54.6
57.8
34.5
29.7
35.3
85.4
8.7
48.9
27.8
52.3
64.9
63.0
34,9
31.
43.9
88.7
8,2
17.8
7.8
12.1
7.4
54,9
64.2
65.9
39.1
40.6
45.1
88.4
16.7
45.4
51.9
56.9
31.6
37.4
41.2
87.7
14.5
/
19.1
8.5
14.3
8.5
55.2
64.0
67.1
39.2
41.0
44.2
88.4
17.8
/
49.9
8.1
28.0
7.8
55.6
65.7
68.9
38.4
40.9
44.1
88.8
17.4
50.4
4.9
28.7
7.5
54.4
64.9
63.9
38.8
35.6
43.7
88.5
11.6
/
29.9
56.5
65.8
40.1
41.1
44,2
17.7
31.1
51.4
59.6
9.8
29.2
29.1
88.0
9.7
56.9
65.8
70.4
39.7
41.3
44.3
89.5
17.7
Table 3: Zero-shot (left) and finetuning (right) evaluation of our disjoint trained models with
varying datasets. We show results on VidSTG (Zhang et al. 2020) and VLN (Voigtlaender et al. 2023). Each row is model pretrained on the specified datasets for zero-shot evaluation and then finetuned on the downstream datasets. #0 is finetumed from a CLIP checkpoint. For models without tracking supervision (#1-7), we cannot report their zero-shot association accuracy (AssA). Our full model (#8) gains full Dense VOC ability from disjoint training, and shows good performance on all metrics with or without finetuning, on both datasets. Detailed captioning metrics are in Appendix C.4
4.5 ANALYSIS OF DISJOINT TRAINING
Zero-shot evaluation. We first pretrain on multiple disjoint datasets (Sec. 3.4). and evaluate zero-shot on our target datasets, VidSTG and VLN, without training on them in Tab. leet.. Zero-shot evaluation simple to perform for captioning models compared to classification, thanks to their open vocabulary.
As mentioned in Tab. and Fig. 3 each dataset supervises different parts of our model. For example, model that is only trained on COCO (#1 in Tab. 3. is only trained with Lobject. meaning that it only produces object proposals which we can evaluate with the Detection Accuracy component of our CHOTA metric. Visual Genome (VG) can supervise both the object proposal and captioning heads of our model. However, there is a large domain gap between the captions in VG and our target datasets, since the captions in VG are for single images and focus on very different vocabularies. Furthermore, VG tends to annotate bounding boxes around object parts rather than entire objects. Consequently, our zero-shot DetA is low when training only on VG (#2). To mitigate the differences in the type of bounding boxes annotated by VG, we ignore Lobject it when using it in conjunction with COCO. Note that we cannot evaluate a model trained only on SMiT, as it does not produce bounding boxes.
We observe in Tab. 3 (left) that the different datasets have complementary properties: Adding COCO improves detection accuracy (#2 to #5, #4 to #7), and adding SMiT improves the captioning accuracy (#2 to #4, #5 #7) even though SMiT only captions a video-level. Finally, training with Aug- COCO allows us to also supervise Lasson and thus the tracking module of our model. A model trained on all the datasets (#8) can therefore perform the full Dense VOC task, and shows good performance on all individual metrics compared to models trained on fewer datasets. Notably, we observe our final model with tracking improves captioning ability (CapA) without adding captioning training data. Similar to Tab. 2 the improvements are likely from our ability to leverage temporal information.
Finetuning evaluation. We now finetune each of the pretrained models from Tab. left and show results in the right. We also include a baseline (#0) which initializes from only CLIP-pretrained checkpoint, observing that this model performs poorly. Once again, we observe that different pretraining datasets are complementary, as adding either SMiT or COCO (#2 to #4, #2 to #5, #1 to #6) improves results further. Adding more pretraining datasets improves results further (#7), and we achieve the best results with our model pretrained on all pretraining datasets (48), which outperforms the best single-dataset pretrained model by 2.0 CHOTA on VidST'G and 0.7 CHOTA on VLN. The improvement over only CLIP-pretrained checkpoint is even larger, by 9.1 CHOTA and 11.6 CHOTA on the two respective datasets. Qualitative visualizations are shown in the supplement.
4.6 COMPARISON TO CONCURRENT WORKS BENSMOT AND OW-
We compare to the concurrent work Li et al. 2024b), which focuses on person category rather than all classes like our task. We finetune our model on the training set using the same our VidSTG experiments. Our full model achieved 90.19 HOTA and 0.254 CIDEr on this benchmark, significantly outperforming Li et al.) (2024b). There are two major advantages of our model: our disjoint pretraining, and our use of larger backbone (ViT-B vs. DLA-34 in BeyondMOT). We further break down the improvements by removing these two components in Tab. 4 The results show: 1. Our pretraining provides consistent gains on Li et al. (2024b) benchmark, improving especially captioning metrics. 2. With a small backbone and no pretraining, our model still outperforms Li et al.) (2024b) on tracking and captioning metrics, showing the advantages of our end-to-end architecture.
as

---

Published as a conference paper at ICLR 2025
HOTA
DetA
AssA
CIDEr
CHOTA
DetA
AssA
CapA
Li et al.
2024b)
71.98
80.79
73.71
0.087
OW- Cis
53.0
60.1
54.0
43.9
Ours
90.19
90.79
89.59
0.254
Ours
56.9
65.8
70.4
39.7
Pretrain 88.56
89.38
87.74
0.150
Table 5: Compare with concurrent work OW-
Backbone 86.55
84.33
89.19
0.129
VisCap (Choudhuri et al. 2024) on VidSTG.
Table 4: ffate-oftthe-ar comparison BenS- MOT (Li et al. (024b). Our model outperforms comparable setting (no extra data, small backbone), and our full model im- proved 18.2 HOTA and 0.167 CIDEr.
The results are from Choudhuri et al. (2024) Tab. 2 and our Tab. 3 #8 under the same setting. Our model are better at detection and tracking, with lower captioning accuracy due to smaller langauge head (46M vs. 2.7B params.).
Li al. (2024b)
Backbone
Rec. & Prec.>> 0.5
Finetuned
Zero-shot
ReferFormer
ResNet50 ViT-B
48.3
STVGBert 47.3
GRiT
62.1
TubeDETR
59.0
STCAT 61.7
Ours
ViT-B
65.1
Ours
61.9
54.1
Table 6: State-of-the-art comparison of spatial grounding on VLN Location-QA Voigtlaender et [023). We report the official metric, which evaluates il bounding box recall and precision are both above 0.5. We compare the ReferFormer
Table 7: State-of-the-art compari-
son of spatial grounding on the Vid- STG with STVGBert, [Su et al. (2021), TubeDETR Yang
(2022). and STCAT Jin et al. (2022). All models use ground truth temporal localization.
baseline Voigtlaender et al. (2023), GRiT Wu et al. (2022a),
and our model (#8 of Tab. left).
We compare to OW-VIs which uses a Mask2Former architecture with video object queries. Tab. 5 shows improved overall performance in CHOTA. Our largest improvement is in Association Accuracy, showing that our end-to-end tracking module (Sec. (.2) outperforms the Mask2Former counterparts. OW-VisCap gets higher captioning accuracy they used substantially larger, 2.7 billion parameter OPT language decoder (Zhang et al. (222a)), whilst we used a smaller, 46 million parameter language model in GIT (Wang& et al. 2022).
4.7 STATE-OF-THE-ART COMPARISON ON VIDEO GROUNDING
As introduced in Sec. Sec.B.5 3.5 Dense VOC models can be directly used for sentence grounding, by finding the proposals with the maximum likelihood of generating the query. We evaluate spatial grounding on the VLN Location-QA (Voigtlaender et al. 2023) and VidSTG (Zhang et al., 2020 benchmarks.
VLN Location-QA consists of questions starting with "Where is", and requires the model produce a bounding box at each frame in the video. The task is therefore effectively a sentence grounding problem, and indeed, the ReferFormer (Wu al. 20226) baseline used by Voigtlaender al. (023) performs sentence grounding after removing "W Where is" from the question. We also remove this prefix before grounding following Sec. 3.5 for both our final model, and an additional GRiT baseline.
In this dataset, only one annotated frame (unknown at inference time) is evaluated. and this benchmark therefore effectively does not involve temporal localization. As the annotation of this dataset based on mouse traces instead of bounding boxes, the evaluation metric considers bounding box coverage (recall) and precision (full details Voigtlaender al. (2023). As shown in Tab, (6] we improve substantially over ReferFormer and our GRiT (Wu al. 2022a) baseline.
VidSTG requires producing a sequence bounding boxes for a given sentence query. The evaluation metric is the average of the Intersection over Union (IoU) at each frame, between the predicted and ground truth bounding boxes for the target object. We compare to other prior works on this dataset in Tab. 7 assuming that the input video is already trimmed temporally to the objects of interest. Our model achieves the best loU, outperforming models designed specifically for grounding, thereby showing that our generative framework task. We also evaluate zero-shot without training on VidSTG, and still perform competitively, This emphasizes the efficacy of our disjoint pretraining. We provide more results in Appendix C.5
be used effectively in the discriminative grounding
CONCLUSION
We proposed the new task of dense video object captioning. Although this task requires expensive annotations across space, time and language, show that train model on existing larger- scale datasets for disjoint subtasks. We show our proposed end-to-end architecture is important for producing more accurate and coherent captions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for imgge captioning and visual question answering. In CVPR, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In 2015.
2015.
Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia Schmid. Uncertainty-aware weakly supervised action detection from untrimmed videos. In ECCV, 2020.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshops, 2005.
Miriam Bellver, Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i Nieto. Refvos: A closer look referring expressions for video object segmentation. arXiv:2010.00263, 2020.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In ICIP, 2016.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In CVPR, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. In arXiv preprint 2rXiv:2305.18566 2023a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. In ICLR, 2023b.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitiick. Microsoft coco captions: Data collection and evaluation server 1504000500.00255, 2015.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
Ho Kei Cheng and Alexander Schwing. XMem: Long-term video object segmentation with an atkinson-shifrinn memory model. In ECCV, 2022.
Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon- Joon-Young Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In CVPR, 2024.
Anwesa Choudhuri, Girish Chowdhary, and Alexander Schwing, Ow-viscap: Open-world world video instance segmentation and captioning. NeurIPS, 2024.
Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A
large-scale benchmark for tracking any object. In ECCV, 2020.
Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé. Motchallenge: A benchmark for single-camera multiple target tracking. IJCV, 2021.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yuming Du, Wen Guo, Yang Xiao, and Vincent Lepetit. 1st place solution for the uvo challenge video-based open-world segmentation 2021. arXiv:21 10. 11661, 2021.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.08 0850, 2013.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach, Women also snowboard: Overcoming bias captioning models. ECCV, 2018.
Sepp Hochreiter and Jtirgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Leamed-Miller, and Xinlei Chen. In defense of grid features for visual question answering. In CVPR, 2020.
Yang Jin, Zehuan Yuan, Yadong Mu, et al. Embracing consistency: one-stage approach for spatio-temporal video grounding. In NeurIPS, 2022.
Justin Johnson, Andrej Karpathy, and Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, 2016.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017a.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017b.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv:2408.0332 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Xiangtai Li, Wenwei Zhang, Jiangmiao Pang, Kai Chen, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. Video k-net: A simple, strong, and unified baseline for video segmentation. In CVPR, 2022a.
Xiangyang Li, Shuqiang Jiang, and Jungong Han. Learning object context for dense captioning, In AAAI, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantic aligned pre-training for vision-language tasks. In ECCV, 2020.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. ECCV, 2022b.
Yunhao Li, Hao Wang, Qin Li, Xue Ma, Jiali Yao, Shaohua Dong, Heng Fan, and Libo Zhang. Beyond mot: Semantic multi-object tracking. ECCV, 2024b.
12

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In ICCV, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. In NeurIPS, 2023.
Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang. Capdet: Unifying dense captioning and open- open-world world detection pretraining. In CVPR, 2023.
Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. IJC,, 2021.
Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In CVPR, 2021.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Workine-Hornung A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, 2017.
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. In ICMR, 2019.
Zhuang Shao, Jungong Han, Demetris Marnerides, and Kurt Debattista. Region-object relation- aware dense captioning via transformer. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018
Rui Su, Qian Yu, and Dong Xu. Stygbert: A linguissi transformer based framework for spatio-temporal video grounding. In ICCV, 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Dean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk. Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv:2312.1 1805, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1 1: fully open, vision-centric exploration of multimodal lmss. In NeurUPS, 2024.
13

---

Published as a conference paper at ICLR 2025
Generalized Intersection Over Union. A metric and loss for bounding box regression. In CVPR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Paul Voigtlaender, Soravit Changpinyo, Jordi Pont-Tusee Tuset, Radu Soricut, and Vittorio Ferrari. Con- necting vision and language with video localized narratives. In CVPR, 2023.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: generative image-to-text transformer for vision and language. TMLR, 2022.
Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding, In ICCV, 2021a.
Weiyao Wang, Matt Feiszli, Heng Wang. and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In ICCV, 2021b.
Yuqing Wang. Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2021c
Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv:2212.00288) 0028800 2022a.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In CVPR, 2022b.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. 2019.
https: http:///hhh.bbcooooooooooeeeeeee
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: eqquence-to-sequenne video object segmentation. In ECCV, 2018
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv:22/2.04979 2022.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio- temporal video grounding with transformers. In CVPR, 2022.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In CVPR, 2023a.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019.
Zongxin Yang and Yi Yang. Decoupling features in hierarchical propagation for video object segmentation. In NewrIPS, 2022.
Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In NewrIPS, 2021.
Zongxin Yang, Xiaohan Wang, Jiaxu Miao, Yunchao Wei, Wenguan Wang, and Yi Yang. Scalable video object segmentation with identification mechanism. arXiv preprint arXiv:2203. / /442, 2023b.
14

---

Published as a conference paper at ICLR 2025
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinvI: Revisiting visual representations in vision-language models. In CVPR, 2021a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022a.
Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification multiple object tracking. IJCV, 2021b.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In ECCV, 2022b.
Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: temporall video grounding for multi-fome sentences. In CVPR, 2020.
Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In ICCV, 2021.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv:1904.07850 2019.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In ECCV, 2020.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. ECCV, 2022a.
Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl, Global tracking transformers. In CVPR, 2022b.
15

---

Published as a conference paper at ICLR 2025
Abrown dog chesing an adult on the grnex
An ecult woman while holds while bell
An wnul woman grey horlde yel bag
blue clotess pulis
dog
goet another
dlng cundoom
Thwre is brown alwar chossing
while gost on the goound
Figure 4: Qualitative results VidSTG. Our model captures motion (1st row) and handles crowded scenes (2nd row). However, it may misrecognize objects (2nd row, "dog" should be "goat") and action boundaries (2nd row, "chasing" before occurs).
APPENDICES
We present further qualitative results (App. A additional experimental details (App. B additional experimental analysis (App. C). and broader impact and potential negative impact (App...
A QUALITATIVE RESULTS
We show example qualitative visualizations Fig. 4ad discuss typical failure cases.
B ADDITIONAL EXPERIMENTAL AND IMPLEMENTATION DETAILS
B.
CODE
Our code is available
Our CHOTA evaluation code is in file "code / cchttat........... py"" This evaluation code is based on the The original code under an MIT license.
official HOTA implementation P
B.2 FULL TRAINING DETAILS
As mentioned in Sec. 4.3 our model based on GRiT Wu al, (2022a). The original GRiT code released under an MIT license. Following GRiT, we use ViTDet-Basse Dosovitskiy et al. (02) Liet al. (2022b) backbone, a CenterNet Zhou et al. (2019) region proposal network and Rol Head, and aundomly-nittializee text decoder following that of GIT Wang al. (2022). The text decoder consists self-attention layers with casual feature masks Wang et al. (2022). All model architecture parameters follow the defaults from GRiT Wu et al. (2022a).
The original GRiT uses an MAE pretrained checkpoint, while in our case we found a CLIP pretrained task. To fit more frames into memory for
checkpoint Radford al. (2021) performs better both training and evaluation, we use a 384 x 384 input size instead of the original 1024 x 1024. This choice moderately decreases dense image captioning performance Visual Genome (from 17.3 to 15.7 APM)
During disjoint multi-dataset pretraining, we sample batches from different datasets in an even ratio (L 1: 1 1). For image datasets, batch is composed of different images; for video datasets, we put the time dimension batches and always guarantee images the same mini-batch are from the same video. We use a local batch size of either video (consisting of 8 sampled frames), or 8 images. As we use GPUs, this means that our global batch size either 32 videos or 256 images. We use the AdamW optimizer with learning rate of 2 x 10-4 weight decay of 0.05, and a layerwise learning rate decay of 0.7 Li et al. (202bb): Wu et al. 2002a We train for 22.5 x 10 iterations per dataset, decreasing the learning rate by factor of 10 after 90% and 97.9 97.5% of the training schedule Wu et al.
lhttp://ggihub.coo/loooo witen/ Tvac
http:://gtthuhhccm//aaaaaa//CC
16

---

Published as a conference paper at ICLR 2025
(022a). For pretraining all the 4 datasets in Sec. 3.4 this corresponds to total of 90 103 iterations, which took approximately 20 hours on 32, 16GB V100 GPUs.
For VidSTG Zhang al. (2020) finetuning, sample 16 frames in training, and run all 200 frames in testing. For VLN Voigtlaender et al. (023) finetuning, we use the 3 annotated frames in both training and evaluation. For finetuning experiments on both datasets, we use video batch size 16 and train for 11.25 x 10 iterations, with a learning rate 10-5, weight decay of 0.05, and layerwise-leaning decay of 0.7 Li et al. 2022b). The finetuning took approximately 6 hours 16, 16GB GPUs for VidSTG, and about 2 hours on 16, 16GB GPUs for VLN. Inference on VidSTG requires 32GB of GPU memory fit 200 frames.
Training losses. Training our model involves a detection loss Lobject, tracking loss Lassocs and a captioning loss Leaptions that is
Lobject + Lassoc + Leaption
For completeness, we detail these three terms next:
The detection loss Zhou et al. (2019) involves a center heatmap loss, a bounding box regression loss, and classification and bounding box refinement loss the Rol head:
Lobject Lheatmap + Lreg + Lroi-cls + -Lroi-reg
The heatmap loss is defined on the predicted heatmap Y E RRH and the ground truth heatmap Y E NoW;
Y.y) log(Yss)
Lheatmay (V, (X, (1 - (Mi)) log(1 Yiy)
(5)
otherwise,
where is the number of objects in the image, a 2 and B 4 are the focal loss weights Lin et al. (017))
Lreg a gloU loss Union 2019
- (IoU(B,, B.) (C((BBB)
Lreg (B, B)
(6)
[Ca|
where B and B are the predicted and the ground truth bounding boxes of the annotated objects, C, is the enclosing convex hull of B; and B, and computes the area.
Lroi-cls is softmax classification loss on each Rol box, defined on the predicted class logits e R² and the ground truth label ē E {0, 1}. Here we only have foreground or background classification.
Lroi els (c, ē) = log voftmax(()a:
Lroi is an LI loss between the predicted boxes B and the ground truth boxes B,
Tyroi-reg B, B) B B|-
The tracking loss is per-element binary cross-entropy loss between the predicted association matrix and the ground truth binary matrix A:
Trassoc
(As log Aij (1 A.,) log (1 Au))) M
I
(9)
The captioning loss is softmax on each predicted word over the entire vocabulary, with label smoothing co-efficient of 0.1 following GIT Wang al. (2022).
Leaption
CE(Decode(f (..:coode(f.....11.....
(10)
where is the ground truth caption, I. is the groud-truth sentence length, and f is the object feature.
B.3 RUNTIME
We report the runtime on 16 GB V100 GPU below for a 64-frame video in Tab. g Our model takes 2.36 seconds to process 64-frame video, which means 27.1 frames per second (amortizing the global trajectory captioning time over all frames). The majority of the runtime is in the autoregressive text decoding which is inherently sequential process.
17

---

Published as a conference paper at ICLR 2025
Detection on all frames
Tracking on all frames
Captioning all trajectories
Total
Run time (ms)
406
4
1950
2360
Frames per second (fps)
157.6
16000
32.8
27.1
Table 8: Wall-clock runtime of each stage of model. The numbers measured 16GB V100
GPU for a 64-frame video.
Aug COCO
VidSTG (zero-shot)
VLN (zero-shot)
VidSTG (finetuned)
VLN (finetuned)
#COCO VG SMIT
APM
APM
APM
APa
0
54.1
35.1
1 /
69.1
36.3
17.1
9.9
68.7
45.9
3
54.8
38.0
18.2
12.7
68.9
47.2
5 /
37.4
19.7
70.8
46.1
6 /
36.7
18.1
69.4
41.3
7 /
38.2
19.0
71.2
48.2
39,5
20.1
71.5
48.2
Table 9: Zero-shot (left) and finetuning (right) evaluation of our disjoint trained models with varying datasets using image dense captioning metric A We show results on VidSTG Zhang et al. (020) and VLN Voigtlaender et al. (2023). Each row is a model pretrained on the specified datasets for zero-shot evaluation and then finetuned on the downstream datasets following Tab. 3 The results are consistent with the CHOTA metric: our models trained on joint datasets perform the best.
C ADDITIONAL EXPERIMENTAL ANALYSIS
C.I APMEVALUATION,
mAP-METEOR is the official evaluation metric used in Visual Genome Krishna et al. (2017b) dataset for dense image object captioning. This metric evaluates predictions in each frame separately, without evaluating the tracking output.
MAP-METEOR based on the Average Precision used in object detection Lin al, (2014), Ever
ingham et al. (0155). but includes caption similarity criteria for determining true positives: i.e. prediction is a true positive if the Intersection over Union (IoU) with the ground truth bounding box above threshold, and if the METEOR score Banerjee & Lavie (2005) is above another threshold. We follow the same implementation and thresholds as the Visual Genome datase"" i.e. IoU thresholds of (0.3, 0.4, 0.5, 0.6, 0.7) and METEOR thresholds of (0.0, 0.05, 0.1, 0.15, 0.2).
In our case, some objects in the datasets Zhang et al. (2020) Voigtlaender et al. (2023 only have bounding box annotations and caption annotations (Sec. 4.1). For these objects without caption annotations, we allow any caption prediction (and therefore ignore it) by setting its METEOR score the maximum of 1, For brevity, abbreviated this metric APM. We report AP xy following Tab. in Tab. 9. The improvements are consistent with CHOTA.
C.2 ABLATION OF HARD TRACKING SAMPLING.
We analyze the effect of the number of sampled frames, m, in hard-aggregation (Sec. 3.3 in Tab. TO With hard-aggregation, the captioning accuracy benefits from larger number of frames m. thanks to longer input-sequence length. However, this also costs more GPU memory in both training and testing. We use = 6 in our ablation experiments (Tab. 2 as it achieves the best accuracy. It also follows the default number of frames used in the GIT Wang et al. (2022) video captioning model.
C.3 USING THE UVO DATASET FOR DISJOINT PRETRAINING
For the disjoint pretraining of our model (Sec. 3.4) we used Augmented COCO as tracking dataset. Another alternative would have been to use UVO Wang al. (2021b), which contains real-world videos, but is relatively small at only 5000 videos.
18

---

Published as a conference paper at ICLR 2025
TN CHOTA(") DetA(1) AssA(t) CapA(1) (Px(11)
53.6
64.3
66.3
36.1
68.7
54.1
37.3
4
54.4
64.3
65.7
37.8
68.7
6
54.9
64.2
65.9
39.1
69.1
8
54.6
64.3
66.1
38.4
69.1
Table 10: Hyper-parameter sweep for number of sampled frames, m, for hard-tracking, We show results on VidSTG Zhang et al. (2020) validation. The models are based on #2 of Tab. Tab.B on VidSTG. Results with hard- hart-feature aggregation get improved with more frames and get saturated with frames.
Tracking dataset
CHOTA
DetA
AssA
CapA
AM
Zero-shot
UVO
24.8
41.2
52.1
7.1
33,8
Aug-COCO
31.1
51.4
59.6
9.8
39.5
Finetuning
UVO
50.9
65.2
53.4
37.9
70.4
Aug-COCO
56.9
65.8
70.4
71.5
Table 11: Results using UVO Wang et al. b the tracking dataset. We show both zero-shot results (top) and finetuning results (bottom) on VidSTG datasets. For reference, we also include our results of using Aug-COCO (#8 of Tab. Aug-COCO performs better both settings, motivating our choice.
Table 11 compares Aug-COCO and UVO under the setting of Tab. 3 #8, both using a default multi- dataset sampling ratio 1 1. We observe that disjoint pretraining with Aug-COCO consistently performs better than UVO in both zero-shot and finetuning scenarios, thus motivating our choice to use Aug-COCO for our experiments in the main paper.
C.4 DETAILED CAPTIONING RESULTS
The Captioning Accuracy (CapA) component of CHOTA metric is the average of the CIDEr, METEOR and SPICE metrics. For completeness, we report each of these captioning metrics individually in Tabs. 12 and 13, for zero-shot and full-finetning evaluation, respectively.
C.5 VIDSTG SPATIO-TEMPORAL GROUNDING EVALUATION
Table of the main paper compared to prior methods on the spatial-video grounding task VidSTG (where the input videos were assumed to already be temporally trimmed). In Tab. [4] we report results for spatial-, temporal- and spatio-temporal grounding by reporting the Spatial IoU (sloU), Temporal IoU (toOU) and Video IoU (vIoU) respectively.
The sloU assumes the video is already temporally trimmed before evaluation, thus evaluating spatial localization. Similarly, the tloU assumes that the video is already cropped spatially around the object of interest. and only the temporal extent of the query sentence needs be determined, thereby evaluating temporal localization. The vloU evaluates both spatial and temporal localization.
Our model was designed for the Dense VOC task, and not grounding, and we were able to perform grounding by selecting the bounding boxes with the highest likelihood of generating the target sen- tence (Sec, 3.5). As shown in Tab. 14 this approach works well for apatial-grounding outperforming prior works in terms of the sloU. However, our model first generates object trajectories, without taking the input query into account, it struggles more at temporal localization. Nevertheless, it still achieves competitive results compared to prior works for both the tloU and vloU, although model was not designed specifically for this task like the other methods in Tab. 14 which include explicit temporal localization modules within the network.
19

---

Published as a conference paper at ICLR 2025
VidSTG COCO CapA CIDEr METEOR SPICE CapA CIDEr METEOR SPICE
VLN
ECOCO
2.7
8.5
4.4
7.7
134
8.5
3.1
8.7
13.8
8.1
4.0
7.2
13.0
7.8
3.1
7.8
12.4
7.5
3.7
9.1
5.2
8.3
13.7
9.0
3.9
9.2
13.9
9.8
7.0
13.3
9.7
4.6
14.6
Table 12: Detailed captioning metries of our zero-shot evaluation (Tab. 3 left), We show the
individual captioning metrics CIDEr, METEOR, and SPICE for each row on both datasets.
COCO VG SMiT
Aug- COCO
VidSTG
VLN
CapA CIDEr METEOR SPICE CapA CIDEr METEOR SPICE
8.7
3.9
34.8
41.7
27.5
35.5
8.2
6.7
14.0
3.9
31.6
33.8
27.2
33.6
14.5
9.6
19.6
14.2
39.2
49.9
30.3
37.5
17.8
13.9
21.4
17.9
38.4
48.3
30.0
36.9
17.4
15.1
20.5
16.5
40.1
51.5
30.8
38.1
17.7
14.6
21.5
17.1
39.7
51.0
37.7
17.7
14.3
21.5
17.5
Table 13: Detailed captioning metrics of our finefuning evaluation (Tab. (lab.3rrise right). We show the
individual captioning metrics CIDEr, METEOR, and SPICE for each row both datasets.
Validation set
Test set tlU
sIoU tloU vIoU
sloU
vloU
STGRN Zhang et al.
20200)
38.0
48.5
19.8
STVGBert Su (221))
47.3
24.0
TubeDETR Yang et al.
(2022)
56.4
47.2
28.7
59.0
48.1
30.4
STCAT Jin al. (2022
61.7
50.8
33.1
Ours (zero-shot)
51.8
40.0
22.0
54.1
40.2
22.5
Ours
58.7
41.8
25.9
61.9
41.1
26.3
Table 14: State-of-the-are comparison of spatial-temporal grounding on VidSTG. "." means the numbers are not reported the paper. Our model performs competitively this task, although was not actually designed for it. As our model generates object trajectories without conditioning on the input query, struggles at temporal localization, denoted by the tloU. The spatial localization performance, denoted by the sloU, outperforms dedicated methods for this task.
D LIMITATIONS
Currently, our model produces single caption for each trajectory, and in future work, we aim to caption potentially multiple action segments within trajectory. Also, we repurposed existing grounding datasets for our task. as annotating new captioning dataset can be subjective. We leave annotating Dense VOC dataset with rigorous protocols and richer captioning as future work.
E BROADER IMPACT AND POTENTIAL NEGATIVE IMPACT
Our work presents a new task and model dense video object captioning. This task represents general technology with wide range of potential applications. Whilst are unaware of all potential applications of such models, it is important to be cognizant that each application has its own merits and societal implications depending on the intentions of the individuals building and using the system. For example, we believe that the Dense VOC models can be used as part of systems to improve video search and retrieval, though they could also be used in video surveillance systems too. Additionally. we note that training datasets, especially for captioning Hendricks et al. (2018); Zhao et al. (2021). contain biases that may render models trained them unsuitable for deployment.
20


Reviews:
[{"review_id": "bKZrRPRWDf", "reviewer": "Reviewer_szRA", "summary": "This paper introduces a novel task towards the field of video understanding—dense video object captioning—with the aim of advancing the comprehension of temporal and spatial information within videos. In the absence of training data for this task, the authors innovatively proposes an end-to-end approach that integrates various datasets to train different modules. The superiority of the proposed methodology is demonstrated through extensive zero-shot and full fine-tune experiments.", "questions": "Beyond the questions in the weaknesses, I have an additional curiosity regarding the approach. In the Spoken Moments in Time (SMiT) dataset, captions are provided at the video level. The approach of using entire frames as detection proposal and generating captions contrasts significantly with the method of using detectors on the Visual Genome dataset to identify objects and then generating captions, resulting in a noticeable domain discrepancy. I am uncertain whether this discrepancy should be addressed, as the examples provided suggest that the generated results for single targets also include interaction information with other objects, such as \"A brown dog chasing an adult on the grass.\" Given that the input to the text decoder is supposed to be a single-track feature and the model does not explicitly model interactions between targets, could this intriguing phenomenon be attributed to the way the SMiT dataset is utilized in training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Clear and Precise Definition of a New Benchmark**: The paper introduces a novel task aimed at achieving more comprehensive video understanding, thereby setting higher standards for a single model's capability to interpret videos. It also establishes well-defined evaluation metrics that are appropriate for this new benchmark.\n\n2. **Design of a Concise and Effective Training Framework**: In response to the proposed task, the paper presents a streamlined, end-to-end trainable framework that effectively integrates multiple tasks such as detection, tracking, and description generation. The authors ensure seamless end-to-end training by efficiently constructing ground truth for the tracking task.\n\n3. **Innovative Data Organization Method**: To Address the lack of a dedicated dataset for the proposed task, the authors decompose the task and combine or generate data from various sources to train different modules effectively.\n\n4.  **Comprehensive Experimental Evaluation**: The authors conducted extensive experiments, demonstrating the advantages of their approach over simply concatenating state-of-the-art solutions for individual tasks. They highlight the effectiveness of combining multiple datasets, the superior generalization ability of the trained model, and its outstanding performance on relevant datasets. The open source of the code further strengthens the credibility of these findings.", "weaknesses": "1. **Clarification of Task Significance**: Although the task imposes higher demands on deep models for video understanding, the paper does not clearly articulate the associated benefits. It would be advantageous to illustrate the task's relevance in more challenging or representative application scenarios, such as sports commentary or intelligent animal monitoring. This would better underscore its significance and research value.\n\n2. **Examination of Method Generalizability**: The authors successfully modularize the training process by breaking down the overall task into distinct sub-tasks and aligning them with suitable training datasets, as demonstrated by the experimental results. However, a more in-depth discussion on the domain differences between datasets and the rationale for task decomposition could provide a stronger basis for the broader applicability of this approach. Such an analysis would significantly enhance the paper's contribution to the research community.\n\n3. **Correction of Typos and Formatting Inconsistencies**: The paper contains several typos and formatting inconsistencies that require attention. For example, in Section 3.5, the symbols in Equation (1) do not correspond with the preceding descriptive content. Additionally, there are inconsistencies in citation formats in the paper, for example, the format of citations in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2rCo3wV17", "reviewer": "Reviewer_o3Kj", "summary": "This work introduces a new task and model for dense video object captioning that unifies spatial and temporal localization by detecting, tracking, and captioning object trajectories in videos. The proposed end-to-end model combines visual understanding and language generation, outperforming multi-stage pipelines in accuracy and temporal coherence. Leveraging a novel training strategy that uses disjoint, large-scale datasets for pretraining, the model achieves impressive zero-shot capability and serves as a robust base for fine-tuning. Additionally, new metrics for task assessment and successful adaptation of video grounding datasets further demonstrate the model's effectiveness in video spatial grounding tasks.", "questions": "Can you provide a detailed breakdown of common failure cases observed during the evaluation? Understanding these could clarify the model's limitations and areas for improvement.\n\nHow does your model perform with unseen objects? Providing results or insights on this aspect could strengthen the claims of generalizability.\n\nCould you elaborate on the rationale behind the chosen evaluation metrics? A more comprehensive justification would help clarify their relevance and effectiveness in measuring performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a novel task—dense video object captioning with unified spatial and temporal localization—bridging video understanding and natural language description in a unique way.\n\nThe end-to-end model presented is robust, outperforming multi-stage pipelines by integrating detection, tracking, and captioning into a cohesive approach that achieves notable zero-shot capabilities.\n\nThe authors clearly outline the model architecture, training strategy, and metrics, making complex components understandable and well-motivated.\n\nBy developing metrics and reusing datasets for this new task, the work contributes valuable tools and benchmarks to the video understanding community, advancing video spatial grounding tasks and surpassing previous state-of-the-art results.", "weaknesses": "The paper primarily focuses on a few datasets, which may not fully represent the diversity of real-world scenarios. Expanding evaluations across a broader range of benchmarks could strengthen the validity of the results.\n\nWhile quantitative metrics are important, including qualitative evaluations—such as human assessments of captioning quality—could enrich the understanding of model performance and highlight potential areas for improvement.\n\nThe paper does not provide an in-depth analysis of failure cases, which limits understanding of specific scenarios where the model struggles, such as occlusions, fast motion, or complex interactions between objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6F4pIyVh52", "reviewer": "Reviewer_zAgu", "summary": "This paper proposes the new task of detecting, tracking and captioning objects in a video. The authors propose evaluation metrics for this task, a baseline method and a training strategy involving disjoint supervision.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work tackles an important problem that was missing from the literature. The paper is well-written and easy to follow. Extensive experiments have been performed.", "weaknesses": "My only concern is that the end-to-end tracking algorithm (listed as a contribution) seems to be naive and not novel enough. There are other methods that perform identity association within the model (like MinVIS[1], CAROQ [2], trackformer [3]). The authors have explored some other ways of integrating temporal information/tracking in Table 2, but how about different ways of association within the model (e.g., query vector propagation in trackformer [3])?\n\n\n[1] MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training, Huang et al., NeurIPS 2022.\n\n[2] CAROQ: Context-Aware Relative Object Queries to Unify Video Instancd and Panoptic Segmentation, Choudhuri et al., CVPR 2023.\n\n[3] TrackFormer: Multi-Object Tracking with Transformers, Meinhardt et al., CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0kqK5dXt7b", "reviewer": "Reviewer_58ev", "summary": "The paper proposes a new task called Dense Video Object Captioning (Dense VOC), which involves detecting, tracking, and generating captions for objects in a video. The model aims to unify spatial and temporal localization while providing a natural language description of object trajectories. The approach combines detection, tracking, and captioning into a single end-to-end model, which is shown to be more accurate and temporally consistent compared to traditional multi-stage pipelines. The authors also propose a novel training strategy using disjoint supervision from different datasets, enabling zero-shot generalization and strong performance after fine-tuning. New metrics are introduced to evaluate the combined tasks, and the model achieves state-of-the-art results on tasks like video grounding without explicit training for those tasks.", "questions": "1. The use of disjoint datasets and diverse forms of supervision makes the training process complex and resource-intensive, posing challenges for reproduction and practical deployment. Is it possible to unify this process within a robust code infrastructure to facilitate easier incremental implementation and application?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed approach integrates detection, tracking, and captioning, which ensures more temporally consistent results and holistic video understanding compared to multi-stage pipelines.\n2. The use of disjoint tasks for training allows the model to generalize well even without specific full annotations for the task, showcasing impressive zero-shot capabilities.\n3. The model outperforms strong baselines and even specialized models on tasks like video grounding and multi-object tracking, demonstrating its effectiveness across various video-related tasks.\n4. The introduction of new evaluation metrics (e.g., CHOTA) tailored for Dense VOC ensures a more comprehensive assessment of model performance, capturing detection, tracking, and captioning jointly.\n5. The model is successfully applied to related tasks such as video grounding and person tracking, proving its versatility.", "weaknesses": "1.\tThe framework demands significant computational resources, requiring more than 32 GPUs, and the specific GPU model was not provided. While separate training offers the potential for lightweight product applications, the experiments lack consideration of this aspect.\n2.\tThe experimental section lacks an analysis and comparison of inference efficiency across different stages of the model as well as with existing backbones.\n3.\tThe model heavily relies on datasets that do not fully cover Dense VOC, which could limit the effectiveness of pre-training. Although the authors address this by using multiple disjoint tasks, the absence of specialized, annotated data for Dense VOC remains a challenge.\n4.\tThe term “Dense” is somewhat confusing and seems ambiguous in the context of the task, which unifies spatial and temporal localization in video with fine-grained visual understanding through natural language. Perhaps a more precise term could better capture the essence of this process.\n5.\tThe caption tracking process is event-oriented. Is there an analysis of different event types in this task?\n\nIf the authors can address my concerns, I am inclined to raise my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)