AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
Authors: Wenhao Chai
Enxin Song
Yilun Du
Chenlin Meng
Vashisht Madhavan
Omer Bar-Tal
Jenq-Neng Hwang
Saining Xie
Christopher D Manning
Venue: ICLR 2025 Poster (2025)
Keywords: Video Captioning
Benchmark
Multimodel Large Language Model
Primary area: datasets and benchmarks

Abstract: Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.

Published as a conference paper at ICLR 2025
AURORACAP: EFFICIENT, PERFORMANT VIDEO DETAILED CAPTIONING AND A NEW BENCHMARK
Wenhao Chai 1,2,+ Enxin Song Yilun Du
Chenlin Meng 2.3 Vashisht Madhavan 2
Omer Bar- Tal 2
Jenq-Neng Hwang
Saining Xie 5
Christopher D. Manning 3
1 University of Washington 2 Pika Labs
Stanford
Harvard New York University
Link: Leaderbeard Video Detailed Caption Model AuroraCap
ABSTRACT
Video detailed captioning a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. this paper, we propose AURORACAP, a video captioner based large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results various in little performance loss. AURORACAS shows superior performance video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of few dozen words, which limits research this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. addition, we propose a new L.-M-assisted metric VDC- SCORE for bettering evaluation, which adopts divide-and-conquus strategy to transform long caption evaluation into multiple short question-anae answerr pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.
INTRODUCTION
The task of video detailed captioning involves generating comprehensive and coherent textual descriptions of video content, capturing not only the primary actions and objects but also intricate details, contextual nuances, and temporal dynamics. It has emerged as critical area of research in computer vision and natural language processing, with significant implications for the fields of robotics (Yang et al., 2023b; Du et al., 2024), ego-centric perception (Grauman et al., 2022; 2023), embodied agents (Zhao et al., 2023; 2024c;d; Deng et al., 2023; 2024; Zhao et al., 2024b), and video editing (Chai et al., 2023) and generation (Bar-Tal et al., 2024). The challenges of video detailed captioning compared to past problems include the limited detailed caption data for training and evaluation, and also the lack of good evaluation metric.
Before the emergence of Large Language Models (LLMs), previous models could only generate very short and rough descriptions of videos (Wang et al., 2022; Xu et al., 2023; Yan et al., 2022; Yang et al., 2023a). Although these models have been trained on web-scale video-text datasets (e.g., HowTo100M (Miech al., 2019) and VideoCC3M (Nagrani al., 2022)), their capabilities remain limited due to their smaller scale and the lack of world knowledge possessed by LLMs. Recently, researchers start build more powerful large multimodal models (LMMs) upon pretrained LLMs (e.g., LLaVA (Liu et al., 2024b), InstructBlip (Dai et al., 2024b), Intern VL (Chen et al., 2023b)). These models typically use intermediate components (e.g.. Q-Foreer (Li al., 2023a) or an MLP) to connect the pre-trained vision transformer (ViT) (Dosovitskiy et al., 2020) and the LLM. Expanding from image-based LMMS to video-based LMMs is natural progression, as videos can be viewed as sequences of frames. While most LMMs start with loading the pre-trained weights from image models and are further fine-tuned with video-text data, we find that LLaVA-like models can be easily adapted video one without any additional parameters but only with high-quality video-text instruction data for fine-tuning.
Equal contributions.

---

Published as a conference paper at ICLR 2025
Table I: Benchmark comparison for video captioning task. Ave. Length indicates the average
number of words per caption. Compared to the existing benchmarks, VDC has the longest captions.
Dataset
Theme
# Video
# Clip
# Caption
Word
Vocab.
Ave. Length
MSVD (Chen& & Dolan, 2011)
1,970
1,970
70,028
607,339
13,010
8.67
MSR-VTT (Xu al., 2016)
7,180
10,000
200,000
1,856,523
29,316
9.28
Open
ActivityNet (Krishna et al., 2017a)
20,000
100,000
100,000
1,340,000
15,564
13.40
S-MIT (Monfort et al., 2021)
515,912
515,912
515,912
5,618.064
50,570
10.89
M-VAD (Torabi et al., 2015)
48,986
55,905
519,933
18,269
9.30
Movie
MPII-MD (Rohtbach ct al., 2013)
94
68,337
68,375
653,467
24,549
9,56
Youcook2 (Zhou et al., 2018)
Cooking
2,000
15,400
15,400
121,418
2,583
7.88
Churades (Sigurdsson el al., 2016)
9,848
10,000
27,380
607,339
13,000
22.18
Human
VATEX (Wang et al., 2019)
41,300
41,300
413,000
4994,768
44,103
12.09
VDC (ours)
Open 1,027
1,027
1,027
515,441
20,419
500.91
However, naive treatment of videos as a series of image frames can result in significant computational overhead and may cause generalization of the length problem ( (Wang et al., 2024b). To address these concerns, more specifically, reduce the number of visual tokens, Video-LLaMA (Li al., 2023f) adapts the video Q-former, MovieChat (Song et al., 2023) uses memory bank, LLaMA-VID (Li al., 2023g) simply uses global pooling, and FastV (Chen et al., 2024b) drops visual tokens by attention rank within LLM layers. In this paper, we present AURORACAP, adapting a simple yet efficient method called Token Merging (Bolya et al., 2022), which is proved to be effective in image and video classification and editing tasks (Li et al., 2023e). To be specific, we gradually combine similar tokens transformer layer using bipartite soft matching algorithm to reduce the number of visual tokens. Following this pattern, visual tokens compared to the original tokens generated by ViT with a marginal performance drop in various benchmarks. With this technique, it is easier to support higher-resolution and longer video sequence inputs for training and inference.
experiments show that we can only 10% to 20%
We present results on several widely used benchmarks, but find that existing video understanding benchmarks are either question-answert anwser--wwwweeebbeeeee (Song et al., 2023; Chen & Dolan, 2011; Caba Heilbron al., 2015; Xu et al., 2016; Xiao et al., 2021; ;; Fu al., 2024; Wu al., 2024), which cannot demonstrate detailed descriptive abilities, or they provide descriptions that are too short, with only a few words (Xu et al., 2016; Caba Heilbron al., 2015) as shown Table 1. Some large-scale datasets focus on specific domains such as ego-centric (Grauman et al., 2023) or contain low-quality videos and annotations (Bain al., 2021). Therefore, we construct the VDC (Video Detailed Captions) benchmark, which contains over one thousand high-quality video-caption pairs spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. We utilize GPT-4 4o as our recaption assistant with a hierarchical prompt design. To preserve as much information as lrense-frans extraction
possible from the videos and maintain temporal consistency, we implement strategy. Using the dense frames as input, despite description of the whole video, we also generate high-quality captions from different aspects, including objective facts, backgrounds, camera angles and movements. Manual quality inspection is employed to ensure the quality of the video captions. While existing video-caption datasets (Chen al., 2024d; Ju et al., 2024; Li et al., 2024b) offer structured captions, VDC is the first benchmark focused on detailed video captioning providing significantly longer and more detailed captions than others as shown in the Table I and Section D.
We also introduce a novel evaluation metric specifically designed for detailed captioning task. Traditional metrics like METEOR (Banerjee & Lavie, 2005), CIDEr (Vedantam al., 2015), and BLEU (Papineni et al., 2002), designed for machine translation or short captions, fail to evaluate detailed captions which contain rich semantic information. On the other side, an --seed evaluation metric is commonly used in visual question answering benchmarks (Maaz et al., 2023; Wu et al., 2024), especially for those generated by VLMs (Song et al., 2024; 2023). However, we observe that the LLM- evaluation metric still struggles to differentiate the quality of detailed captions and tends to give lower scores. To address these challenges, we propose VDCSCORE, a novel captioning evaluation metric that leverages the reliability of large language models (LLMs) by evaluating short visual question-anse answer pairs. We first decompose the ground-trutt caption into a set of concise question- answer pairs using the LLM, then generate corresponding responses from the
2

---

Published as a conference paper at ICLR 2025
predicted caption. Finally, the LLM is used to assess the accuracy of each response provide an overall score. In particular, our paper makes the following contributions:
In Section 2, we illustrate how imput before injecting into LLM with marginal performance drop. Using these insights, we pro- pose AURORACAP, video understanding base model with improved captioning performance, which also demonstrates advanced performance on existing benchmarks.
reduce the number of tokens used for image video
In Section 3, we present VDC, the first benchmark for detailed video captioning, featuring over one thousand videos with significantly longer and more detailed captions. We comprehensively evaluate proprietary and open-source models using our proposed VDCsCORE metric. We also conduct extensive ablation studies based on AURORACAP.
2
AURORACAP: A VIDEO DETAILED CAPTIONING BASELINE
2.1 ARCHITECTURE
Large multimodal model. To effectively leverage the capabilities of both the pre-trained LLM and visual model, which is typically CLIP (Radford et al., 2021) DINO (Oquab et al., 2023), LLaVA adapt simple multi-layer perceptron (MLP) as projection layer connect each patch tokens of image features into the word embedding space. The original LLaVA model is trained by two-stage ensruuctio-tuuing procedure, which first pretraining projection layer for feature alignment and then finetuning end-to-end while freeze the visual encoder. Recent works like Prismatic VLMs (Karamcheti et al., 2024) and Idefics2 (Laurençon et al., 2024) further explore the design space of LLaVA architecture. We adapt some conclusion from those works for training the model.
Token merging. To increase the throughput of existing ViT models, Token Merging (Bolya et al., 2022) is proposed to gradually combines similar tokens in a transformer to reduce the number of tokens passing through VIT models. Token Merging has been proven to be effective on image and video classification tasks even without the need for training. Token Merging applied between the attention and MLP within each transformer block as:
1. Alternatively partition the tokens into two sets A and B of roughly equal size.
For each token in set A, calculate the token similarity with each token in set based on cosine similarity of the Key features in attention block.
3. Use bipartite soft matching and then select the most similar pairs.
4. Merge the tokens using weighted average, record the token size.
Concatenate the two sets & and back together again.
Once the tokens have been merged, they actually carry the features for more than one input patch.
Therefore, the merged tokens will have less effect in softmax attention as
QKT vd
A softmax
logs
where is the number of patches the token represents after token merging. We conduct frame-wise token merging in AURORACAP, the visualization of token merging can be found in Appendix E.
2.2 TRAINING RECIPE
Building upon the exploration in works like Prismatic VLMs (Karamcheti et al., 2024), Idefics2 (Laurençon et al., 2024), and InternVL (Chen et al., 2023b), we further adopt three- stage training strategy, which can noted as Pretraining stage, Vision stage and Language stage. The training data used in each stage are shown in Table G6, Table G7 and Table G8. More training details selection and data preprocessing operation can found in Appendix G.
including
Pretraining stage. Similar LLaVA, first align visual features from the vision encoder ViT with the word embedding space of LLMss. To achieve this, we freeze the pretrained ViT and LLM, training solely the vision-language connector. Consistent with -L- (Li al., 2024a), we employ a two-layer MLP as the projection layer and pretrain on 1.3M image-caption pairs. To optimize performance, we explore various combinations of the pre-trained VIT and LLM in Appendix F.
3

---

Published as a conference paper at ICLR 2025
Table Comparison of AURORACAP with LLM-based SoTA methods on image-captioning bench- marks under zero-shot setting, except k-shot where k is marked as a superscript. The GPT-4V CIDEr results are from (Team, 2024), the only reference we know of, but seem oddly low. * marks fine-tuned results. Scores in bold indicate the best performance under zero-shot setting.
Flickr (51,784)
NoCaps (4,500)
COCO Cap (5,000)
Model
CI-
BLEU
BLBU
Met-
ROU-
BLEEU
BLEU
Met- 6ox
ROU-
CI-
BLLEX
BLDD
Met-
ROU-
DE:
0.4
6h4
DBr
081
InstUCiBLIP
Ins00c110L1P-13000 et 20286)
116.3
135.0"
82.2*
43.4
78.6
LLaNA 73 at.
68.4
69.6
26.6
34.8
25.9
54.5
99.9
67.7
25.5
ULINA etal 2024)
66.6
65.2
24.2
22.2
48.8
08.7
34.0
25.4
54,9
101.8
62.2
24.5
MiniPPM V-38
2013)
66.5
25.1
27.2
89.9
79.1
33.2
29.7
55.8
94.2
69.8
23,9
28.3
UeCap (L 20250
56.7
21.3
21.8
42.7
91.2
24.7
25,0
(Alayme 2022)
67.
84.5
Chameloon- 548 (liss, 2004)
94.72
120
GPT
55.5"
78.5
Gemini 1.5
K2.2"
99.3
AURORACAP-78
.8.9
75.6
328
26.7
55.4
111.
85.6
44,4
20.0
69.6
120.8
78.0
35,3
28.6
57.2
Table 3: Comparison of AURORACAP with SoTA methods on existing video captioning benchmarks
under zero-shot setting. AURORACAP substantially outperforms other recent methods.
MSR-VTI VTT (1,000)
VATEX (1,000)
Model
CI-
BLEU
BLEU
Met-
ROU-
CI-
BLEU @1
BLEU
Met-
ROU-
DEr
@1
@4
GE
DEr
@4
cor
GE
ZeroCap (Tewel ol., 2022)
9.6
2.9
16.3
35.4
DeCup (Li et al. 2023d)
18.6
14.7
20.4
18.7
13.1
15.3
PaL1-3 (Chen et al., 2022)
21.3
Ma al. (Ma et al. 2024)
22.1
3.5
17.3
28.7
23.9
2.8
14.1
23.5
LLaVA-78 (l et al,, 2024b)
16.9
Video-LLaMA (Zhang et al. 2023a)
2.3
4.9
16.8
3.8
4.3
16.3
21.8
AURORACAP-7B
33.1
58.6
21.0
23.9
49.5
33.8
$7.1
18.4
19.0
40.8
Vision stage. Unlike LL,VA, we next unfreeze the pretrained VIT while freezing the L.LM during vision stage and train with the public data among various computer vision tasks (e.g., captioning, object identification, classification, reasoning, VQA, etc.) to get better generalization (Huh et al., 2024). The motivation for doing this is that CLIP ViT usually performs poorly in aspects such as Orientation and Direction, Positional and Relational Context, Quantity and Count (Tong et al., 2024b). However, since the most of the collected datasets lack high-quality and detailed corresponding language descriptions, the labels often consist of only few words short phrase when converted to text. Therefore, unfreezing the language model at this stage is risky, as it may lead to a degradation in the performance of the language model.
Language stage. Finally, we conduct end-to-end training, which means all the components are trainable, with the most high-quality public data during language stage training. We mix all the data, including images and videos, captions and instructions, into each mini-batch for training. To improve video captioning performance, we duplicate the video captioning datasets twice. We remove all the video training data for training image-based AURORACAP as well for image captioning task.
2.3 EVALUATION
In this section we evaluate AURORACAP on various tasks including image captioning, video caption- ing, and video question answering. Appendix H show detailed evaluation settings.
Image Captioning. We evaluate AURORACAP using CIDEr (C), BLEU-4 (B@4), BLEU-1 (B@) 1), METEOR (M), and ROUGE-L (R) metric on Flickr (Plummer et al., 2015), NoCaps (Agrawal et al., 2019), and COCO-Cap (Lin al., 2014) benchmarks and compare it with LLM-based otate-of-the-art methods as shown in Table 2. We show the performance of image based AURORACAA under zero-shot settings. Notice that these benchmarks all contain short captions consisting of single sentence, so they only partially reflect the model's performance. The performance mentioned in the rest of this paper refers to video-based AURORACAP.
Video Captioning. Although the current video captioning benchmarks only contain one-sentence captions, to compare with prior work, we similarly evaluate on these benchmarks. We evaluate Au- RORACAP using CIDer (C), BLEU-4 (B@4), BLEU-1 (B @ 1), METEOR (M), and ROUGE-L (R) metric on MSR- VTT (Xu al., 2016), VATEX (Wang al., 2019) as shown in Table
4

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of AURORACAP with SoTA methods on video question answering and classifi- cation benchmarks under zero-shot setting. The pretrained LLM size is 7B for all listed models.
ANet Score
MSVD
MSR- VTT
iVQA
Model
Ace
Acc
Score
Ace
Score
Acc
Just Ask (Yang et al., 2021)
12.2
FrozenBiLM (Yang et al., 2022)
24.7
32.2
16.8
26.8
Video-LLaMA (Zhang et al., 2023a)
12.4
1.1
51.6
29.6
1.8
VideoChat (Li et al., 2023b)
26.5
2.2
56.3
2.8
45.0
2.5
Video-ChaGGPT (Maaz al., 2023)
35.2
2.7
64.9
3.3
49.3
2.8
LLaMA-VID (Li et al., 2023g)
47.4
3.3
69.7
3.7
57.7
3.2
Video-LLaVA (Lin et al., 2023a)
45.3
3.3
70.7
3.9
59.2
3.5
VA (Wu, 2024)
51.2
3.5
73.8
4.1
60.0
3.5
LLavA-NeXT-Viide (Liu et al., 2024a)
53.5
3.2
MovieChat (Song et al., 2023)
45.7
3.4
75.2
3.8
52.7
2.6
MovicChal+ (Song al., 2024)
48.1
3.4
76.5
3.9
53.9
2.7
AURORACAP-7B
61.8
3.8
62.6
3.6
43.5
2.9
55.2
Video Question Answering. We evaluate AURORACAP on MSVD-QA (Xu et al., 2017). ActivityNet-QA (Yu al., 2019), MSRVTT-QA (Xu al., 2017), and iVQA (Yang et al., 2021) for video question answering tasks shown in Table 4. Although AURORACAP is primarily captioning model, it achieves competitive performance in some VQA datasets (ANet, iVQA). For others (MSVD, MSR-VTT) performance is more modest, but still not bad. In some failure cases observed the model, we found that prompting the model to generate a comprehensive caption for the video input can lead to outputs that include the correct answer. This phenomenon may be attributed to disruption in the model's instruction-following capabilities during the training.
VDC: A VIDEO DETAILED CAPTIONING BENCHMARK
3.1 BENCHMARK DATASET CURATION
3.1.1 VIDEO COLLECTION AND PROCESSING
To ensure the reliability of the benchmark, is crucial to maintain high video quality, balanced data distribution, and content complexity. Panda-70M (Chen et al., 2024e) offers a high-resolution, open- domain YouTube video dataset with diverse one-minute clips across wildlife, cooking, sports, news, TV shows, gaming, and 3D rendering, ideal for studying complex real-world scenarios. Additionally, a large volume of aesthetically appealing videos from user-uploaded platforms like Mixkit (Mixkit, 2024), Pixabay (Pixabay, 2024), and Pexels (Pexels, 2024) provides scenic views and visually pleasing human activities with minimal transitions and simpler events. Ego4D (Grauman et al., 2022) complements the video source by focusing on ego-centric human activities and auto-driving scenarios, ensuring comprehensive coverage of real-world scenes. To mitigate content homogeneity among these candidate videos and maintain diversity the final dataset, inspired by ShareGPT4Vide (Chen et al., 2024d), we building VDC upon those various video sources. Note that the videos used in VDC construction are not included the training data of AURORACAP, To ensure balanced data distribution, we allocate equal proportions of videos from Panda-70M (Chen et al., 2024e), Ego4D (Grauman et al., 2022), Mixkit (Mixkit, 2024), Pixabay (Pixabay, 2024), and Pexels (Pexels, 2024). We first split the video into clips and apply dense frame extraction.
3.1.2 STRUCTURED DETAILED CAPTIONS CONSTRUCTION PIPELINE
We believe that a comprehensive detailed video caption benchmark should encompass various aspects, including main objects, camera movements, and background. However, most existing benchmarks (Li et al., 2024b; Chen et al., 2024d) provide only a single caption for the entire video with less structured details. Therefore, develop structured detailed captions construction pipeline to generate extra detailed descriptions from various perspectives, significantly extending the length and enhancing the richness compared previous benchmarks. Following (Ju et al., 2024), the structured captions
5

---

Published as a conference paper at ICLR 2025
Table The video source distribution of the proposed VDC benchamrk including diverse settings such as natural landscapes, human activities, and animal activities. Videos from different sources have a similar proportion in VDC, reducing the data bias.
Video Source
Sample
Proportion
Duration (sec.)
Ave. Length (sec.)
Ave. Keyframe
Panda-70k 70M (Chen et al,, 2024e)
229
22.25%
5,714
24.95
7.18
Ego4D (Grauman 2022)
196
19.05%
10.935
55.79
19.46
Mixkit (Mixkit, 2024)
197
19.14%
3,261
16.55
6.58
Pixahay (Pixahay, 2024)
199
19.34%
4,748
23.86
8.99
Pexels (Pexels, 2024)
20.21%
4,343
20.88
7.99
Total
1,027
29,001
28.18
10.43
40s-50s
30s-40s
7%
4%
Short Caption: Ave. Length 26.5
20s-30s
40
Background Caption: Ave. Length = 80.4
50s-60s
18%
Main Object Caption: Ave. Length = 90.4
21%
Percentage
Camera Caption: Ave. Length = 86.6
Detalled Caption: Ave. Length 500.9
10
10s
13%
38%
-0s-20s
50
100
350
400
450
500
550
600
050
700
Caption Length (words)
Figure 1: Video length statistics.
Figure 2: Distribution structured caption length.
in VDC encompass not only short and detailed captions but also three additional categories: (1) main object caption, offering comprehensive analysis of the primary subjects' actions, attributes, interactions, and movements across frames, including variations in posture, expression, and speed; (2) background caption, providing detailed descriptions of the background, such objects, location, weather, time, and dynamic elements; and (3) camera caption, which details the camera work, including shot types, angles, movements, transitions, and special effects.
To generate detailed, fine-grained, and accurate captions, we leverage GPT-4o to produce video descriptions. We utilize the dense video frames to obtain captions. We observed that generating all captions in single conversation round often introduces hallucinations in the detailed captions. To address this, we design a hierarchical prompt strategy to efficiently obtain accurate structured captions and detailed captions in two conversation rounds: (1) structured captions generation and (2) detailed captions integration. In the first round, the prompt briefly introduces the differences between structured captions and uses the dense video frames as input to generate the short caption, main object caption, background caption, camera caption, and the detailed caption. the second round, the generated captions serve as the reference. The second-round prompt guides GPT-4o 4o to enhance the detailed caption based on the initial captions, ensuring consistency without introducing new entities or relations, and producing a vivid, engaging, and informative description. The whole prompt template for the structured detailed captions construction pipeline can be found in Appendix K. Finally, we conduct manual review correct captions with hallucinations and supplement omitted visual elements. The refined detail structured captions are then used as the ground truth for evaluation.
3.1.3 COMPARISON ON NUMERICAL STATISTICS
Based on the hierarchical scheme, VDC can capture a rich variety of details of the video and reduce hallucinations. The visual representation in Figure demonstrates the video duration distribution of VDC. Over 87% of the videos exhibit duration ranging from 10K to 12K frames, while 1% of videos extending beyond 60 seconds. Only 13% of videos have duration less than 10 seconds. As illustrated in Table 1, the average length of detailed descriptions in VDC is significantly longer than in previous benchmarks. Figure shows the length distribution structured captions in VDC, with detailed captions averaging over 500 words. Appendix M present more statistics.

---

Published as a conference paper at ICLR 2025
GT caption
generated caption
Draise
Who is the main character of this vided?
@ask
The video showcases an exhilarating moment as snowboarder soars through the air, executing a stunning trick. Dressed in bold red and white jacket, black pants, and protective helmet. The backdrop to this action-packed scene is breathtaking snowy mountain landscape. The mountain's peak is visible overall composition suggests high-speed descent down the mountsin...
The video captures a thrilling moment anowboarder in mid-air, performing an impressive
3get
Snowboarder
@check
Oet
snowboarder, clad vibrant red and black jacket, black pants, and protective helmet, The snowboarder is holding onto a rope with one hand, su suggssting that they are being pulled up mountain by a snowmobile, a common practice snowboarding gain speed momentum
Snowboarder
Drais
Who is the main character of this
aask
video?
distance. The
Øget
video
red and white
@Scheck
Oget
red and black
Figure Evaluation pipeline with VDCsCORE, Like when humans take reading comprehension tests, we transform the matching between two paragraphs into set of question-answer pairings. We first generate some question-ansee answer pairs based on the ground truth captions, then derive corresponding answers one by one from the generated captions, and finally perform matching. The process is automatically evaluated with the L.M involvement each step.
3.2
EVALUATION METRIC DESIGN AND LEADERBOARD
3.2.1
VDCSCORE: EVALUATING DETAILED CAPTIONS WITH LLMS
Evaluating video captions requires not only assessing the quality of the captions but also flexibly evaluating the alignment between the video and the caption. While metrics such as BLEU (Papineni al., 2002), CIDEr (Vedantam et al., 2015), and ROUGE-L (Lin, 2004) have been employed for caption evaluation, these metrics are predominantly designed for short captions and rely heavily on word-level frequency-ban based alignment. Given the advanced semantic understanding capabilities of large language models (LLMs), Video-ChatGPP (Li et al., 2023b) proposes using LLM as an evaluation assistant to directly judge the correctness of the whole predicted captions and assign scores. However, demonstrated in Table 6, our experiments indicate that when dealing with detailed captions, the direct application of LLM struggles to accurately distinguish the correctness of various predicted captions, fails to effectively evaluate the precision of detailed descriptions, and exhibits tendency to assign disproportionately lower scores. Therefore, we introduce VDCSCORE, a novel quantitative metric that utilizes LLMs evaluate the similarity between predicted and ground-truth detailed captions through divide-and-congue approach.
The core idea of VDCSCORE is decompose long detailed captions into multiple short question- answering pairs, avergae the evaluation of each pair as the final result. We elaborate the design of VDCSCORE the following parts: (1) ground-truth question-answe
pairs extraction, (2) responsed answers generation and (3) answers matching. As illustrated in Figure 3, we first employ GPT-40 generate question-ansee -nsweer pairs from the detailed ground -trtt caption. To ensure that the generated question-answer pairs capture as much information as possible from the original caption and facilitate accurate evaluation by LLMs, we constrain the number of generated pairs and impose specific guidelines: the questions must be open-ended, and the answers should be concise, and directly relevant to the questions. For a fair comparison and to mitigate potential variability arising from generating different question-anse annweer pairs for the same caption, pre-generate standardized set of question-ansee answer pairs for all captions in VDC, as depicted Figure 3. The used prompt templates used along with additional examples, are provided in Appendix I..
VDCSCORE subsequently analyzes the predicted captions by leveraging ground-truth question- answer pairs. We prompt GPT-4o read the detailed predicted captions and generate answers based solely on these captions. To mitigate biases arising from discrepancies in the length between ground-truth and predicted answers, also impose constraints ensuring that responses limited concise sentences or phrases, Consequently, for each pair of ground-truth and predicted captions, we
gpt-40-20:4 20040-0-----8 10--008--50 06, the latest 4o version in Septemher, 2024.

---

Published as a conference paper at ICLR 2025
41.7
Gemini-1.5 Pro
38
36
34
AuroraCap-7B
InternVL-2-88
VDCScore
32
LLaVA-OV-78
ShareGPT4Video-BB
LLaVA-Next-V7B
30
LongVA-7B
Video- Video-LLaVA-BB
28
VILA-7B
Video-ChatGPT-78
26
MovieChat-7B
LLaMA-VID-7B
24
0
1000 2000
3000
4000
5000
6000
7000
8000
# visual tokens
Figure 4: Comparison between various models with different number of visual tokens input VDC. For Gemini- 1.5-Pro, we only report the performance. We manage the number of visual tokens by managing token merging for AURORACAP, and manage the number of frames for others. AURORA- CAP achieves a much better VDCsCORE than all other models given a certain compression in the number of visual tokens kept and indeed approaches the performance of Gemini-1.5-PPp
Table 6: Comparison of AURORACAP with LLM-based baseline methods on VDCsCORE and other evaluation metrics under zero-shot setting. For each evaluation metric, we report the average value of the five structured captions in VDC. Note that VDD, CIDEr, and BLEU are only the average of background and main object caption, since the values of the others are closed to zero.
VDCsCORE
VDD Score
CI-
BLEU
BLEU
Met-
ROU-
Model
Elo
Acc
Score
Acc
DRrr
@
684
GF
Gemini-1.5 Pro (Reid al., 2024)
41.73
2.15
49.68
3.07
5.97
29.72
2.63
21.21
20.19
1.171
LLaMA-VID (Li al., 2023()
30.86
1.62
4.63
1.63
1.48
17.74
1.46
8.07
17.47
859
ChaGGP77 (Mazz
2023)
MovieChut-7B (Song al,, 2023)
31.92
1.64
10.24
1.86
5.14
14.33
3.17
13.60
14.98
890
VILA-7B (Lin et al., 2023h)
32.61
1.70
16.27
2.02
8.20
19.13
2.11
5.62
16.63
1.073
Video-LLaVA-7B (Lin
2023a)
32.80
1.72
14.14
2.00
4.43
17.20
2.32
10.36
17.53
1,007
LLaVA-I-5-7B (Liu et al., 2023)
33.98
L.76
26.71
2.33
6.63
29. 80
2.54
22.79
20.36
825
LongVA-7B (Zhang al, 2024h)
LLaVA-1 (Liu ot al., 2023)
34.78
1.80
28.26
2.36
3.90
20.43
2.02
26.37
17.87
943
LLaVA-NeXT-V17B (Zhang al. 2024d)
35.46
1.85
25.62
2.34
2.66
20.18
2.33
28.17
17.51
1.022
LLaVA-1.6-7B (Liu al., 2024a)
35.70
1.85
40.16
3.09
17.36
1.59
24.23
17.08
846
LLaVA-1.6-13B (Liu et al. 2024a)
35.85
1.85
34.55
2.51
5.55
29.23
2.50
20.26
19.96
728
ShaxeGPT4Video-8B
2024d)
36.17
1.85
1.102
LLaVA-OV-7B (Li al., 2024s)
37.45
1.94
41.83
2.70
4.09
28.34
2.84
23.98
19.59
1.155
InteraVL-2-8B (Chen et al., 2023b)
37.72
1.96
48.99
3.03
5.59
15.75
2.48
10.76
17.63
1.081
AURORACAP-7B
19.09
1,267
obtain a set of <question correct answer, predicted answer> triplets. Following Caleo-ChatGPT (Li al., 2023b), then ask GPT-4o output two scores for each triplet: one for answer correctness and another for answer quality. The final accuracy and score are calculated by averaging the correctness score and quality score respectively. When using same captions as input, VDCSCORE, returns an accuracy of 100%, demonstrating the feasibility and reliability.
3.2.2 BENCHMARKING VIDEO DETAILED CAPTIONING
To our knowledge, no standard evaluation benchmarks have been established for detailed video captioning. To advance this field, we assess several baselines on our proposed VDC. As illustrated in Table present quantitative comparison between our AURORACAP with existing state- of-the-art LMMs. We compare the VDCSCORE with both rule-based and model-based caption metrics with AURORACAP performing well. BLEU (Papineni et al., 2002), CIDEr (Vedantam et al.,
8

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of AURORACAP with LLM-based bascline methods on VDCsCORE under zero- shot structured captions setting. We consider the VDCsCORE of detailed captions, short captions, background captions, main object captions and camera captions. We also test the vision-blind case suggesting by (Chen et al., 2024c; Tong et al., 2024a).
Camera Acc/ / Score
Short Acc/ / Score
Background Acc / Score
Main Object Acc/ / Score
Detailed Acc/ / Score
Model
Vicuna-v1.5-7B (Chiang et al., 2023)
21.68/ / 1.12
23.06 / 1.17
22.02/ / 1.15 19.52/ 1.10
22.64 / 1.16 19.57 / 1.10
23.09 / 1.20
Llama-3.1-88 (Dubey et al., 2024)
17.83 / 1,00
17.90 / 1.02
20.10/ 1.22
Gemini- 1.5 Pro (Reid et al., 2024)
38.68 2.05
35.71 1,88
4..84/2223
47.32/2.41
43.11 1011/2.:3 /2.22
LLaMA- (Li al., 2023f)
39.47 22.47/2.10 2.
29.92 (
28.01 / 1.45
31.24 1.559
25.67 1,38
Video CPee-Cha (Maaz et al., 202.)
37.46/ 2.00
1.56 29.36
33.68 1.70
30.47/ 1.60
24.61 /1266
MovieChat-7B (Song el al., 2023)
37.25/ 1.98
32.55 1.559
28,99
31.97 1.66
28.82/
VILA-7B (Lin et al., 2023b)
34.33/ 1.83
30,40 1.55
35.15/ 1.80
3 1.72
29.78 1.58
Video-LLaVA-7B (Lin et al., 2023a)
37.48/ 1.97
30.67 1.63
32.50/ 1.70
36,01/ 1.85
27.36 / 1.43
LLaVA-1.5-7B (Liu et 202.3)
38.38/2.00
28.61 1,51
34.86/ 1.79
34.62/1.7
33.43 1.73
LongVA-7B (Zhang et al., 2024b)
/
31.94/1.66 1.63
36.39/ 1.85
40.95/2.11
27.91 1.48
LLaVA-1.5-13B (Liu et al., 2023)
38.97
30.89 6.0
34.79/ 1.78
36.27 1.84
33,00 1.74
LLaVA-NeXT--77 (Zhang et al., 20244)
39.73/2.10
30.63 1.60
36.54/ 1.88
36,54/ 1.88
33.84 1.77
LLaVA-1.6-220 (Liu et ul. 2024a)
36.50/ 1.93
31,91 1.65
37.58/ 1.92
36.03/1.85
36.47 1.89
LLaVA-1.6-13B (Liu et al., 2024a)
35.61 1.86
31.90 1.66 1.66
38.90 1.99
36.65 1.87
36.18 1.89
ShareGPT4Vidco-SBB (Chen et 2024d)
33,28 1.76
39.08 1.94
35.77
1.81
37.12/ 1.89
35.62
1.84
LLaVA-OV-7B (l. ct al., 2024a)
3.082/2.02
32.58 / 1.70
37.43/ 1.92
38.21 / 1.96
41.20 2.
IntemnV1-2-88 (Chen et al., 2023h)
39.08/2.11 /2.11
33.02 / 1.74
37.47 1.89
44.16 2,22
34.89 1.82
AURORACAP-7B
43.50
32.07 /1.68
35.92/ 1.84
39.02/1.97
41.30/2.18
Table 8: Comparison of AURORACAP with LLM-based baseline methods on VDD (Liu et al., 2024a) under zero-shot structured captions setting. We consider the VDD (Liu et al., 2024a) of detailed captions, short captions, background captions, main object captions and camera captions.
Camera Ace / Score
Short Acc/ Score
Background Ace / Score
Main Object Acc / Score
Detailed Acc/ / Score
Model
1emini-1.5 Pro (Reid et al., 2024)
18.89 2.115
16.91 / 1.572
57.73 3.263
41.64 2.886
2.581 / 0.330
LLaMA-VID (Li al., 2023()
28.28 2.513 2.513
1.034/ 1,042
6,198 1.895
3,063 1,366
2,046 / 0,304
Video-ChatGPT-TB (Muaz et al., 2023)
16.00/2.175
4.173/ 1.032
14.14/ 14.14/2.273
3.001 / 1.423
1.038/ 0,192
VILA-7B (Lin et al. 2023b)
4.005 1.751
2.087 0.233
2,385 22.45
10.10/
1,015 0.262
Video-LLaVA-7B (Lin et al., 2023a)
20.00/2.336
3.193 1.064
17.17/2.253
11.11/ L.765
3.130 0.316
LLaVA-1.5-7BB (Liu el al., 2023)
26.88 26.88/2.515
1.222 / 0.793
36.50/2.72
16.92 1.937
0.694 / 0.276
LongVA-7B (Zhang et al., 2024b)
7.200/
0.794
50.00/3.200
15.31 22196
0.002 0.002/0.220 0.247
I.LVA-11.5-338 (l.iu al., 2023)
32.65/2.666
2.836 0,922
37.55/2.749
18.98 1.978
0.688 0.275
LLaVA-NeXT-VV7 (Zhang et al., 2024d)
29.81 2.664
1.913 0.957
3.,23/2.692
18.02 1.999
0.887 / 0.279
LLaVA- (Liu et al., 2024a)
21.11 2.2229
8.696 1.146
53.31/3.105
27.01 2.2286
1.282 0.279
I.L.V- 1.3- (l.iu et al., 2024a)
21.56/ 2.199
9.798 1,206
39.37
29.73 2.329
1,287 0.271
ShareGPTwWideo-BBB (Chen
2024d)
33,28 1.768
4.908 0,986
37.12/
3.213 0.752
LLaVA-OV-7B (Li al., 2024a)
17.11 / 2.086
11.15/ 1.277
55,82 /3
27.84/ 2.258
1.372 0,249
InternVVL.2-888 (Chen et al,, 2023b)
29.00 / 2,545
4.041 / 1,079
23.64/3.444
34.34 2.627
3.032/ / 0.394
AURORACAP-7B
49.40 3.144
3.813/08886
/..52/3.261
37.14 2,533
1,275 /
2015), ROUGE-L (Lin, 2004) and METEOR (Banerjee & Lavie, 2005) are included as representative rule-based metrics. For model-based metrics, we also consider Video Detailed Description (Li et al., 2023b) (VDD), which employs ChatGPT as an evaluation assistant to compare full captions.
Table 7 presents VDCSCORE performance across various sections of structured captions within VDC. Following (Chen al., 2024c), we also incorporate vision-blind baselines. Furthermore, Figure illustrates schematic diagram of the performance and efficiency of different video training models. Since the comparison of model inference time under different architectures, models, deployment frameworks, and output lengths is unfair, so we used the number of visual tokens as a representation of efficiency. AURORACAP achieves superior performance in video detailed captioning while utilizing significantly fewer visual tokens than other models, fully highlighting the efficiency of AURORACAP. We also show additional experimental results with VDD metric as shown in Table 8. We also perform human study using Elo ranking to supplement assessment of AURORACAP" performance. As depicted in Figure 5, VDCsCORE shows better correlation with human evaluation results than VDD and ROUGE metric.
evaluation and provide a more intuitive

---

Published as a conference paper at ICLR 2025
Pea rson Correlation Conflicient - 0.86
Pearson Correlation Confficient (- 0.66
Pearson Coefficient
45
VDCscore
40
ROUGE-L
2
35
900
1000
1100 12:00
900
1000 1100
1200
900
1000
1100
1200
Score
Figure 5: Pearson correlation analysis among three evaluation (Lin, 2004), VDD (Liu et al., 2024a)-and human Elo rankings for video models. VDD reflects the score for detailed captions. VDCsCORE demonstrates the highest consistency with expert judgments, thereby reinforcing the reliability. The detailed settings are provided the Appendix O.
ROUGE-
Captions
Accuracy Shart Captiors
Background Captions
32.5
33.5
$1.0
36
30.5
30.0
25.5
1.0
0.8
0.6
0.4
0.2
0.0
0.4
2.2
0.0
1.0
5.8
0.5
0.4
0.0
Pal VINAA
toieen best
Tadio situal Isken lorpt
Rabio of stucal token bapt
Acouracy Main Captions
Detailed Captions
VDC
37
40.5
88
36
40.0
18
39
39.5
39.0
1.0
0.8
0.0
2.0
0.4
0.0
Radoo of Visual tolenn kept
Radio wisus taken Se
Ratio of visuul token kegt
Figure 6: Ablation study of token merging on VDC. We found that token merging significantly reduces the number of tokens while maintaining minimal performance drop, and even showing improvement in some aspects. We highlight the token merging ratio when achieving 90% and 80% performance with the dash line and filled area.
strategy of AURORACAP, token merging plays a significant role in reducing the number of visual tokens. We further study how the video detailed captioning capability is influenced by token merge ratio. We define the performance percentage as the proportion between the highest and lowest values on the entire performance curve. As shown in Figure most models maintain satisfactory performance (> 80%) even with only 0.2 of visual token kept ratio. Since AURORACAP focuses on spatial visual token merging, the temporal features introduce additional complexity explore the token merging laws, resulting in the optimal performance may occurs at a middle level of visual token kept ratio.
Ablation study on token merging ratio. As
4 CONCLUSION
In this paper, we first introduce AURORACAP, efficient video detailed captioner based on large multimodal model. By leveraging the token merging strategy, we significantly reduce the computa- tional overhead without compromising performance. We also present VDC, novel video detailed captioning benchmark designed to evaluate comprehensive and coherent textual descriptions of video content. For better evaluating, We propose VDCSCORE new LLM-assister metric with divide- and-conquer strategy. Our extensive evaluation on various video and image captioning benchmarks demonstrated that AURORACAt achieves competitive results, even outperforming models in some tasks. We also conduct thorough ablation studies to validate the effectiveness of token merging and other aspects of our model. We found that the current model performs poorly in terms of the trade-off between performance and the scale of input tokens. Additionally, there is still room for improvement in camera handling and detailed captioning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We acknowledge Pika Labs for the computing resources support. We are also thankful to Xtuner", Imms-eval", and SGLang for their well-developed and user-friendly codebase. We would like to thank Enxin Song for her contribution to the benchmark processing.
ETHICS STATEMENT
This research on video captioning utilizes publicly available datasets, ensuring that all data complies with privacy regulations. We acknowledge the potential biases that can arise in automatic caption generation, particularly concerning gender, race, other characteristics. We have taken measures to evaluate and minimize such biases, while remaining committed further improvements. Additionally, we recognize the potential risks of misuse, such as generating misleading captions, and have checked the training dataset with safeguards against such applications.
REPRODUCIBILITY STATEMENT
We have made several efforts to ensure the reproducibility of our work. All the key implementation details, including the architecture of our model, the training procedures, and hyperparameter set- tings, are described in supplementary meterial Section G. The introduction of the used evaluation benchmarks and settings are in Section H. The introduction of the used evaluation benchmarks and settings are in Section H. Prompt template of VDC generation is in Section K. Question-anse answer pairs generation prompt template of VDCSCORF is in Section I.. And calculation details of Elo ranking in Section O. We have also outlined any hardware configurations and computation requirement used for experiments in Figure F8 further support reproducibility. The code for model training, evaluation, deployment, as well as model weight, benchmark, and training dataset of all the training stages will be released with the paper.
"GitHlub Repository: httpps/s//kkaabb.cm///eeettttttt
"GitHub https:
leval
'GitHub Repository: http:///w.pppp...tt////.///////e
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee. and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4, 4,32,33 37
2019.
Lean-Baptise Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-23777 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016. 23
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016. 34
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021. 2
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005. 2,
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 2024.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07728 2024. 23
Ali Furkan Biten, Ruben Tito, Andres Maila, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019. 34
Daniel Bolya, Cheng-Yane Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In The Eleventh International Conference on Learning Representations, 2022. 2,3
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015. 2, 23
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.174533 2024. 23
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23040-23000 2023.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pp. 190-200, 2011. 2, 23
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.1 1684, 2024a. 34
12

---

Published as a conference paper at ICLR 2025
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and- aang-and-play inference acceleration for large vision- language models. arXiv preprint a0Xiv:2403.06764 2024b.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpi4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279) 2023a. 34
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are the right way for evaluating large vision-languag, models? arXiv preprint arXiv:2403.20330 2024c.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint M4iv:2240.004325,, 2024d. 2, 8, 9, 30, 34
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13320-13331, 2024e. 5,6
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209 arXiv:2209.06794 2022.
Xin Chen, Yu Zhu, Hua Zhou, Liang Diao, and Dongyan Wang. Chinesefoodnet: A large-scale image dataset for chinese food recognition. arXiv preprint arXiv: 02743, 2017. 34
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijic Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternV:: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint 23..:::312..122555 2023b. 1,
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint ar arXiv:2406.07470 2024. 23
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http::/vvicuaa Imsys. org (accessed 14 April 2023), 2(3)6, 2023. 9
Dawei Dai, YuTang Li, YingGe Liu, Mingming Jia, Zhang YuanHui, and Guoyin Wang. 15m mul- timodal facial image-text dataset, 2024a. URL hwwpt:ttp///c. http://axvvvv///// org/ http:/ab///2200555555 08515. 34
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024b. 1,
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 326-335, 2017. 34
Jie Deng, Wenhao Chai, Jianshu Guo, Qixuan Huang, Wenhao Hu, Jenq-Neng Hwang, and Gaoang Wang. Citygen: Infinite and controllable 3d city layout generation. arXiv preprint arXiv:2312.01.508, 2023.
Jie Deng, Wenhao Chai, Junsheng Huang. Zhonghan Zhao, Qixuan Huang, Mingyan Gao, Jianshu Guo, Shengyu Hao. Wenhao Hu, Jenq Hwang, et al. Citycraft: A real crafter for 3d city generation, arXiv preprint arXiv:2406.0498. 2024.
13

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Zhengfang Duanmu, Wentao Liu, Zhongling Wang, and Zhou Wang. Quantifying visual image quality: A bayesian view. Annual Review of Vision Science, 7:437-464, 2021. 34
Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783. 2024. 9
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. arXiv preprint
2016. 34
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprini arXiv:2405.21073 2024.
Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? Dataset and methods for multilingual image question. Advances in neural information processing systems, 28, 2015. 34
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.0279) 2024. 24
Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9904-6915 2017. 32, 34, 37
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 18995-19012, 2022. 1, 5, 6, 26
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.1825,, 2023. 1,2
Mingfei Han, Xiaojun Chang, Heng Wang, and Linjie Yang. Shot2story A new benchmark for comprehensive understanding of multi-shot videos. arXiv preprint arXiv:2312. 1030000 2023. 24
Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint ar0iv:2308. 12038, 2023. 4
Ting-Hao Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pp. 1233-1239, 2016. 34
Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019. 29, 32, 37
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint 27X. 2024. 4
14

---

Published as a conference paper at ICLR 2025
Vladimir lashin and Esa Rahtu. A better use of audio-visudl cues: Dense video captioning with bi-modal transformer. arXiv preprint arXiv:2005.08277, 2020. 23
Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal
large language models. arXiv preprini wwxiv:2312.14223: 2023. 34
Peng Jin. Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified
visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13700-13710, 2024. 23,20 24
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017, 34
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprini arXiv:2407.06330 2024. 2, 5, 34, 39
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of risually-conditionn language models. arXiv preprint arXiv:2402.07866. 2024. 3
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski. Cornelius Emde, Virginic Do. Zeynep Akata, and Thomas Lukasiewicz. e-vil: A dataset and benchmark for natural language explanations in lasgon-angguage tasks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1254, 2021. 34
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023. 27
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. hierarchical approach for generating descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 317-325, 2017. 34
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp.
706-715, 2017a. 2
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017b. 27
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint SxXiv:2405.022446 2024.
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge- based visual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3108-3120, 2022. 34
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a. 3, 8, 9, 34
Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, et al. Wolf: Captioning everything with a world summarization framework. arXiv preprint arXiv:2407.1 08440078900 18908, 2024h. 2,5
15

---

Published as a conference paper at ICLR 2025
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping eeguuage-image pre- pre-training with frozen image encoders and large language models. In International conference machine learming. pp. 19730-19742. PMLR, 2023a. 1
Junyan Li, Delin Chen, Tianle Cai, Peihao Chen, Yining Hong, Zhenfang Chen, Yikang Shen, and Chuang Gan. Flexattention for efficient high-resolution vision-language models, arXiv preprint arXiv:2407.20228, 2024c. 23
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023b. 5, 7, 8, 9, 23
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang. Jingjing Xu, Xu Sun, et al. M3IT: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint 2306003306.44887,, 2023c. 34
Qingyun Li, Zhe Chen, Weiyun Wang. Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, et al. Omnicorpus: An unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406. 08418, 2024d. 24
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. arXiv preprint arXiv:2303.03032, 2023d.
Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, and Ling-Yu Yu Duan. Densefusion- Im: Merging vision experts for comprehensive multimodal perception. arXiv preprint arXiv:2407.0832 00300,, 2024e. 34
Xirong Li. Weiyu Lan, Jianfeng Dong, and Hailong Liu. Adding chinese captions images. In Proceedings of the 2016 ACM on international conference on multimedia retrieval, pp. 271-275, 2016. 34
Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu. Coco- en for cross-lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia, 21(9):247-23600 2019. 34
Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. arXiv preprint arXiv:2312. 10656, 2023e. 2
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311. 2023f. 2, 8, 9, 23
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language
models. arXiv preprint urXiv:2311.1704. 2023g. 2,5
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023h. 37
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023a. 5, 8, 9, 23
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004. 7, 10, 23
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533. 2023b. 8, 9,23
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Vurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014. 4, 27, 32, 34, 37
Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 861 1-8619, 2018. 33, 35
16

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 4, 8, 9,
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024a. URL https: //llava-vl. githu.. io ht/ollgg//000 2004--222422222 01-30--3 aalvavaveettt 4,5, 8, 9, 10, 23, 34
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in
neural information processing systems, 36, 2024b. 1, 4, 24, 34
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.01011, 2017. 37
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song-Chum Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022. 34
Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narrator via equal distance to visual tokens. arXiv preprint arXiv:2312.08870, 2023. 23
Yunchuan Ma, Laiyun Qing, Guorong Li, Yuankai Qi, Quan Z Sheng, and Qingming Huang. Retrieval enhanced zero-shot video captioning. arXiv preprint arXiv:2405. 07046,, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2, 5, 8, 9, 23, 24, 45
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212-46244, 2023. 24, 25
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019. 34
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. 37
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEFE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021. 34, 37
Antoine Miech, Dimitri Zhukov, Jean-Baptist Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2630-2640, 2019. 1
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019. 27, 34
Mixkit. Free stock video clips, motion graphics, & music tracks, 2024. URL htl.https://mixxt
co/. Accessed: 2024-08-24 5,
Mathew Monfort, Sou Young Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14871-14881, 2021. 2
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In European Conference on Computer Vision pp. 407-426. Springer, 2022.
17

---

Published as a conference paper at ICLR 2025
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002. 2, 7, 23
Pexels. Free imageds and video clips, 2024. URL https: httpps://wwww./ wwwewppxeeeee... ctt.eal..cccoooooomommmmmmeeeeee 2aom/zh-ent Accessed: 2024-08-24. 6
Pixabay. Stunning free images & royalty-free stock, 2024. URL
com/
Accessed: 2024-08-24, 6
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting eegion-to-prrass correspondences for richer image-to toseeenc-ee models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015. 4, 37
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-88 8763. PMLR, 2021. 3
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. In Synthetic Data for Computer Vision Workshop@ CVPR 2024, 2024. 24
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055330 2024. 8,9
Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. Testa: Temporal-spatial token aggregation for fong-form video-language understanding. arXiv preprint arXiv:2310. 19060, 2023. 23
Marcus Rohrbach, Wei Qiu, Ivan Titov, Stefan Thater, Manfred Pinkal, and Bernt Schiele. Translating video content to natural language descriptions. In Proceedings of the IEEE international conference computer vision, pp. 433 440, 2013. 2
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115:211-252, 2015. 34
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022. 34
Yuzhang Shang. Mu Cai,, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 153888 2024. 23
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pp. 758-758. Springer, 2020. 34.3% 37
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer ECsV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 510-526. Springer, 2016. 2
18

---

Published as a conference paper at ICLR 2025
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 8317-8326. 2019. 34
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprini urXiv:2307.16444 2023. 2, 5, 8, 23, 24
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat4: Question-aware sparse memory for long video question answering. arXiv preprint
arXiv:2404.17176. 2024. 2, 5, 23
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223, 2017. 34
Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long llm. arXiv preprint 0rXiv:2404.043446 2024. 23
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 13878-13888, 2021. 34
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. 2024. 4
Qwen team. Qwen2-vl. 2024. 23
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022. 4
Tristan Thrush. Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 2022. 34
Yunjie Tian, Tianren Ma, Lingxi Xie, Jihao Qiu, Xi Tang, Yuan Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Chatterbox: Multi-round multimodal referring and grounding. arXiv preprint arXiv:2401.13307 2024. 34
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: : A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024a. 9, 34
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint 2xXiv:2401.06200 2024b.
Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. Using descriptive video ser- vices to create large data source for video annotation research. arXiv preprint arXiv: 2015. 2
Ramakrishna Vedantam, C Lawrence Zimick, and Devi Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015. 2, 7, 8, 23, 29
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint 1601:0601.00144 2016. 29, 34
Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via milm. arXiv preprint arXiv:2403. 16558, 2024a. 23
19

---

Published as a conference paper at ICLR 2025
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-ters transformer for vision and language. arXiv preprint 2rXhv:2205.14100 2022.
Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404. /2224, 2024b.
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. 34
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A sarge-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4581-4591, 2019. 2, 4, 26, 29, 33, 35, 37,42 42
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024. 2
Penghao Wu and Saining Xie. V#: Guided visual search core mechanism in multimodal Ilms.
arXiv preprint arXiv:2312.14111 2023. 34
Wenhao Wu. Freeva: Offline milm training-free video assistant. arXiv e-prints, pp. arXiv-2405, 2024. 5, 23
Junbin Xiao, Xindi Shang. Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021. 2
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference Multimedia, pp. 1645-1653, 2017. 5, 533 35, 37
Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. International Conference on Machine Learning, pp. 31728-388888 387488. PMLR, 2023. 1
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016. 26, 29, 33, 35, 37, 42
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXhv:2404.16994, 2024a. 23
Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.158// 2024b. 29, 30
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5045- 2022. 24
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:22/2.04999, 2022. 1
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1686-1697, 2021. 5.
20

---

Published as a conference paper at ICLR 2025
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124-14: 141, 2022.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10714-10726, 2023a. 1,23
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023b.
Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. End-to-end multi- modal fact-checking and explanation generation: A challenging dataset and models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2733-2743, 2023. 34
Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal Ilms. arXiv preprint urKiv:2410.07177 2024. 24
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Compulational Linguistics, 2:67-78, 2014. 32
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qu: dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019. 5, 29, 33, 35, 37
Xiang Yuc, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-disciphn multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023.
32, 37
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 6720-6731, 2019. 34
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language
model for video understanding. arXiv preprint arXiv:2306.00 02855, 2023a. 4, 5, 23
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. Internin-xcooppose-2255 A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03322 2024a. 23
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint 24Xi::2406.16852, 2024b. URL https:://aaxiy/ org http//ab///4... 8.9
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. J6199, 2023b. 23
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng. Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmano, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint urXiv:2404.01258, 2024c. 23, 24, 34
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2019. 23
21

---

Published as a conference paper at ICLR 2025
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024d. URL 1o github...
8,
Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, et al. Distilling vision-language models on millions of videos. arXiv preprint arXiv:2401.06129, 2024a. 23
Zhonghan Zhao, Wenhao Chai, Xuan Wang. Li Boyi, Shengyu Hao, Shidong Cao, Tian Ye, Jenq- Neng Hwang, and Gaoang Wang. See and think: Embodied agent in virtual environment. arXiv preprint arXiv:2311. 15209, 2023.
Zhonghan Zhao, Wenhao Chai, Xuan Wang, Ke Ma, Kewei Chen, Dongxu Guo, Tian Ye, Yanting Zhang, Hongwei Wang, and Gaoang Wang. Steve series: Step-by-step construction of agent systems in minecraft. Computer Vision and Puttern Recognition Workshop, 2024b. 1
Zhonghan Zhao, Kewei Chen, Dongxu Guo, Wenhao Chai, Tian Ye, Yanting Zhang, and Gaoang Wang. Hierarchical auto-organizing system for open-ended multi-agent navigation. arXiv preprint arXiv:2403.08282, 2024c.
Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang. Kewei Chen, Dongxu Guo, Yanting Zhang, Hongwei Wang, and Gaoang Wang. Do we really need complex agent system? distill embodied agent into single model. arXiv preprint arXiv:2404.0466 14669, 2024d. 1
Yinhe Zheng, Guanyi Chen, Xin Liu, and Jian Sun. Mmchat: Multi-modal chat dataset social media. arXiv preprint arXiv:2108.0744 2021. 34
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 2, 23
Xingyi Zhou, Anurag Arna,, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. arXiv preprint 40X172240.01197, 2024. 23
22

---

Published as a conference paper at ICLR 2025
Supplementary Material
The supplementary material is structured as follows:
literature review about the related works in Section A.
More results on long video benchmarks in Section B.
N- N-gram based evaluation metrics in Section C.
Benchmark comparison between VDC and other video captioning benchmarks in Section D.
The calculation and the visualization examples of token merging in Section E.
More ablation studies for AURORACAP in Section F.
The training settings for AURORACAP in Section G.
The introduction of the used evaluation benchmarks and settings in Section H.
More limitations of AURORA ACAP at current stage Section I.
Additional evaluation metrics analysis in Section J.
Prompt template of VDC generation in Section K.
Question-ans answer pairs generation prompt template of VDCSCORE in Section L.
More statistics information of VDC in Section M.
More statistics information of VDCsCORE in Section N.
Calculation details of Elo ranking in Section O.
Case studies among several baselines in Section P.
Prompt template of predicted answers extraction in Section Q.
Prompt template of correctness evaluation in Section R.
A RELATED WORKS
Video Captioning The general goal of video captioning is understanding a video and describing it with natural language. Unlike image captioning, video captioning requires the description also the temporal dimension (e.g., human action, camera and object movement). The current datasets including videos in the domain of human activities (Caba Heilbron al., 2015), cooking scene (Zhou et al., 2018; lashin & Rahtu, 2020), movie (Song et al., 2023), and open domain (Chen & Dolan, 2011). In Table 1, we show the summary of current video captioning benchmarks. Most of them are with short caption, which is not suitable for evaluating video detailed captioning task. There are several widely used metrics to evaluate the correctness of generated caption, such as BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016), and BERT (Zhang et al., 2019). Dense Video Captioning (Yang et al., 2023a; Zhou et al., 2024) is also a captioning task but further requires localizing the events temporally. In this paper, we focus on the detailed description of a short video clip, where there are no scene changes or camera switches.
Large Multimodal Models for Video With the develop of LLMs and LMMs (Maaz et al., 2023; Zhang et al., 2023b; Li et al., 2023f; Song et al., 2023; Zhang et al., 2023a; Song et al., 2024; Zhao al., 2024a; Zhou et al., 2024; team, 2024; Jin et al., 2024; Beyer et al., 2024; Shang al., 2024; Cai et al., 2024; Cheng et al., 2024; Li et al., 2024c), many recent works have explored adapting them into video understanding field (e.g.. Video-LLaMA (Li et al., 2023f), Video-LLaVA (Lin et al., 2023a), VideoChat (Li et al., 2023b), Vista-LLaMA (Ma et al., 2023), LLaVA-Hound (Zhang et al., 2024c), Koala (Tan et al., 2024), Elysium (Wang al., 2024a), and MovieChat (Song et al., 2023; 2024)). Thanks to this flexible design, the models can combine pretrained knowledge with minimal trainable parameters. Instead of requiring large-scale training, using only a small amount of high-quality training data can even achieve better results. Most of the existing models use additional parameters for temporal modeling. There are also some interesting observations that we can actually build advancing LMMs without additional parameters for temporal modeling (Xu et al., 2024a; Lin et al., 2023b; Liu et al., 2024a; Zhang et al., 2024a; Ren et al., 2023) or even without further training with video-text data (Wu, 2024). Recent works (e.g., Free VA (Wu, 2024), V-a (Liu et al., 2024a), VLIA (Lin et al., 2023b), and PLLaVA (Xu et al., 2024a)) also find that the vanilla LLaVA-like model pretrained on high- nigh-quality image instruction data also be a strong video understanding model. Free VA further observe that using existing video instruction tuning data like Video- Chte-- 100K (Maaz al., 2023) to tune LMMs may not necessarily lead to improvements. As the concurrent work,
23

---

Published as a conference paper at ICLR 2025
also observe this phenomenon and proceeded to develop video detailed captioning baseline training based on the LLaVA architecture. Chat-UniVa UniVi (Jin et al., 2024) employ a set of dynamic visual tokens to uniformly represent images and videos. However, for current video imputs, when sample at 1 FPS, the frames are already sufficiently sparse; frame-to-frame similarity is quite low except in static scenes. On the other hand, if Under these conditions, we believe that strong temporal merging should be avoided, as it is likely to impair the model's s core capabilities, especially since the CLIP vision encoder trained on images.
keyframes input, the frame similarity will be even lower.
As for the benchmark, inspired by LLaVA (Liu et al., 2024b), Video-ChatGPT (Maaz et al., 2023) introduces 100K video clips with text instructions with the first vLMMs benchmark evaluation system powered by LLMs. MovieChat-Ik IK (Song et al., 2023) and CinePile (Rawal et al., 2024) are crestton--asseriigg based benchmark et al., 2023) comprises videos with 2 to shots each sourced for our dataset from the public video benchmark HDvila100M (Xue et al., 2022). However, currently there is no video benchmark available to evaluate video detailed captioning tasks like IIW (Garg al., 2024) and other datasets (Li et al., 2024d) did image captioning field. Other works (Ye et al., 2024) focus on the specific domain like Ego. In this paper, work fills this gap. As for metric, uses similar dlvide-and-conueee strategy to build preference dataset, ensuring that instructional data remains factually consistent with detailed captions. Our goal for VDCsCORE, however, is to evaluate the accuracy of generated captions through QA pairs generated from ground truth.
long-form video understanding. Shot2Story20K (Han
(Zhang et al., 2024c)
B MORE RESULTS ON LONG-FORM VIDEO BENCHMARKS
We take MovieChat- 1K (Song al., 2023), Egoschema (Mangalam et al., 2023) as the representative long video benchmark, and compare with existing models. As shown in the following Table B2 and Table B1, AURORACAP achieves comparable performance even without training on long videos and special design. We also plot the visual token kept ratio curve EgoSchema as Figure B1, demonstrating the stability and effectiveness of our token merging method for longer video sequences.
Table Bl: Model performance Egoschema (Mangalam al., 2023) benchmark.
Model
Design For Long Video
Accuracy
Random Choice
20.0
FrozenBiLMM
x
26.9
mPLUG-Owl
31.1
TimeChat
33.0
Video-LLAVA
38.4
LLAMA-VID
X
38.5
ALAVA-NeXT-Viide
43.9
VideoLLAMA2
X
51.7
MovieChat
53.5
Human Performance
76.2
AURORACAP (ours)
46.0
Table B2: Model performance MovieChat-IPy (Song et al., 2023) benchmark.
Model
Degign for Long Video
Breakpoint
Global
39.1
51.7
VideoChat
X
46.1
57.8
TimeChat
46.1
73.8
VideoChatGPT
X
48.0
47.6
MovieChat
48.3
62.3
MovieChat+
49.6
71.2
AURORACAP (ours)
52.6
59.7
24

---

Published as a conference paper at ICLR 2025
Accuracy on Egoschema
48,0
46
45.6
Accuracy
40
38
1.0
0.8
0.6
0.4
0.2
0.0
Ratio of visual token kept
Figure B1: Token merging ablation curve on Egoschema (Mangalam et al., 2023). The best perfor- mance is set range from 0.6 to 0.7 of visual token kept ratio.
C N-GRAM BASED EVALUATION RESULTS
A variant of VDCSCORE could be developed to assess final triplets (<question, correct answer, predicted answer>) by employing a n-gram based evaluation approach as shown in Table C3 with correlation shown in Figure C2.
Table C3: Evaluation results of various models on VDC based on ROUGE for answer matching.
Model
Avg.
Camera
Short
Background
Main Object
Detailed
LLaMA-VID
6.66
9.53
5.12
5.30
6.07
7.29
Bale--ChaGGPT-BB
6.85
9.20
5.37
6.40
6.29
6.97
ViLA-7B
6.68
7.88
5.16
6.29
6.35
7.74
Video-LLAVA-7B
6.79
8.67
5.25
6.09
6,64
7.28
LLAVA-1.5-7B
6.34
8.31
5,40
5.80
5.52
6.65
LongVA-7B
6.70
8.37
4.99
6.11
6.79
7.23
LLAVA-1.5-13B
6.34
8.64
5,55
5.57
5.15
6.79
LLAVA-NeXT-V7B
6.69
8.61
5.30
6.35
6,27
6,93
LLAVA-1.6-7B
6.45
8.08
5.38
5.79
5.82
7.20
LLAVA-1.6-13B
6.18
7.31
5.28
6.00
5.29
7.01
ShareGPT4Video-BB
6.78
8,46
5.73
5.82
6,54
7.36
LLAVA-OV-7B
6.59
8.09
5.66
5,86
5.58
7.75
InterVL--2-85
6.82
8.14
5.63
5.95
6,53
7,86
AURORACAP-7B
7.60
9.55
5.66
6.79
7.42
8.58
Gemini- 1.5 Pro
7.59
8.78
6.25
7.29
7.70
7.94
Pearson Correlation Coefficient
VDCScore ROUGE
7.0
900
1000
1100
1200
Elo Score
Figure C2: Correlation between the n-gram evaluation scores and human elo ranking the VDC.
25

---

Published as a conference paper at ICLR 2025
D BENCHMARK COMPARISON
We compare our proposed VDCwith some examples from several video captioning benchmarks as shown in Figure D3, Figure D4, and Figure D5. The corresponding captions are shown as followings:
Figure D3: Video example of MSR-VTT (Xu al., 2016) benchmark.
Figure D4: Video example of VATEX (Wang al., 2019) benchmark.
Figure D5: Video example of VDC benchmark sampled from Ego4D (Grauman et al., 2022).
Benchmark Caption (for VDC we show only the detailed caption)
MSR-VTT D3
(4 words) Teams are playing soccer.
VATEX D4 (13 words) woman instructs and demonstrates how to remove the insides a pumpkin.
VDC DS
(618 words) The video opens with an intimate close-up of a surface adorned with vibrant green moss and intricate lichen, initially evoking the serene beauty a natural landscape. imagery quickly transitions, revealing that the mossy surface is actually part of motorcyele or vehicle's 's engine compartment, creating a striking contrast hetween the lush textures of nature and the cold, hard lines of mechanical components. As the camera angle shifts, the viewer is drawn deeper into the engine compartment, where the interplay of moss overgrowth various machinery introduces a fascinating blend organic life industrial elements, highlighting the unexpected coexistence of nature and technology.
organic
The perspective then zooms in, accentuating the rich details of the mossy growth, which clings tenaciously to the metallic surfaces, while dark cavity bencath hints the complexity of the machinery. The reflective metallic surfaces glint in the light, further enhancing the visual contrast and inviting the viewer to explore this unique juxtaposition. Suddenly, the scene shifts dramatically, with rapid camera motion creating a vibrant blur of colors and shapes, transforming the previously detailed views into chaotic whirlwind, suggesting swift movement through the intricate landscape of the engine compartment.
As the motion blur begins dissipate, the viewer is presented with clearer image a light- colored, textured surface, where blurred mechanical components can be discerned, indicating a deccleration in movement. The camera stabilizes, revealing rough-textured floor or ground that suggests indoor or industrial environment, characterized by sense of organized chaos. The scene transitions detailed examination a cluttered workspace filled with tangled wires, casings, and components a variety of colors, emphasizing the disorganized state of electronic or mechanical internals, possibly during a maintenance or repair process.
The perspective shifts once more, showcasing darker, textured surfaces juxtaposed against lighter insulating materials, with hidden metallic elements peeking through, suggesting another angle within this same cluttered interior space. human hand enters the frame, reaching out to interact with the components, signaling an active workspace filled with purpose. As the scene expands, additional hands join the fray, actively manipulating various objects within the crowded environment, signifying ongoing task or collaborative effort amidst the complex array of components and materials.
26

---

Published as a conference paper at ICLR 2025
The atmosphere is imbued with of urgency and engagement, the camera captures the dynamic interactions the individuals working together. The camera work remains fluid and dynamic, featuring a mix of close-up shots that highlight the intricate details of the components and wider angles that provide context to the bustling environment. The slightly shaky nature of the shots adds layer of realism and immersion, drawing the viewer into the heart of the action. The low light conditions create a moody ambiance, with shadows dancing enhancing the visual depth and interest of the scene. Overall, the video encapsulates vivid portrayal of the intersection between nature and machinery, as well as the collaborative spirit of those engaged in the intricate task of maintenance and repair within this unique setting.
the surfaces,
E TOKEN MERGING VISUALIZATION
Managing and Trouhleshonting Networks
Figure E6: Token merging visualization. From top to bottom, the image IDs are COCO:COCO- 0ain2014-000000447000 VG:2331508, SA-IB:sa-393200, OCR-VQA:717891 From left to right, the number of tokens representing the images are 490, 154, 18, and 6.
The number of tokens merged in each layer of the transformer in ViT is calculated by the following formula:
(1-ratio) (W H/PP)/L
where W. H is the width and the height of the input image, P is the patch size, is the number of layers in the transformer, ratio is the proportion of tokens to be kept. The merging is evenly distributed across all layers of the transformer.
We show token merging visualization examples from COCO (Lin et al., 2014), VG (Krishna et al., 2017b), SA-IB (Kirillov al., 2023), and OCR-VQA (Mishra et al., 2019) in Figure E6, since only conduct frame-wise operation. Features extracted by CLIP ViT-H model". We conduct image captioning for the examples given final token numbers by 7B AURORACAP as following, We remove line breaks in generated caption for better reading experience.
Huggingface Model:
https: sttps://huggtt ng wtt.c/..c//ppp///eeeeee
27

---

Published as a conference paper at ICLR 2025
# Token Describe this image in detail. (Figure E6 Row 1)
490 In the heart of a winter wonderland, a skier takes flight, suspended in mid-air against the backdrop of clear blue sky. The skier, clad in a vibrant red and white jacket, is captured in moment of pure exhilaration. The skier's s body angled towards the left side the image, with their skis crossed in an impressive display of skill and control. The skier's helmet, a stark contrast to the surrounding snow, is adorned with a pink and white design, adding a touch of color to the otherwise monochrome landscape. The skier's s goggles, reflecting the bright sunlight, are testament to the intensity of the sport. Beneath the skier, the snow-covered mountain slopes stretch out, dotted with trees that stand silent spectators to the skier's 's daring feat. The image is a freeze-frame of action and adventure, encapsulating the thrill and beauty of winter sports.
154 In the midst of a clear blue sky, a skier is captured in a moment of thrilling action. The skier, clad in a vibrant red and white jacket, is suspended in mid-air, performing an impressive trick. The skier's s body is parallel to the ground, creating sense of motion and excitement. The skier's 's skis, a striking combination of red and white, are crossed in front of them, adding the dynamic nature of the image. The skier's 's helmet, a necessary safety measure, is white and contrasts with the colorful jacket. The backdrop to this action- srcion-packed scene is a breathtaking snowy mountain landscape. The mountain's peak is visible in the distance, adding depth to the image. The overall composition of the image suggests high-speed deseent down the mountain, with the skier skillfully navigating the terrain. This image encapsulates the exhilarating sport of skiing, showcasing not only the skill of the skier but also the stunning beauty of the mountain landscape.
18 In the heart of snowy mountain, daring skier takes flight. Dressed a vibrant red and white jacket, black pants, and a protective white helmet, the skier is captured mid-air, performing an impressive trick. The skier's 's body is parallel to the ground, arms outstretched, as if embracing the thrill of the moment. The skier's skis, a striking combination of red and white, are pointed upwards towards the clear blue sky. The skier's poles, held firmly in their hands, add the dynamic nature of the scene. The backdrop to this exhilarating moment is breathtaking snowy mountain landscape. The mountain's 's peak, blanketed in snow, stretches out into the distance. The clear blue sky above is devoid of clouds, adding to the serene beauty of the scene. This image captures not just moment, but story of adventure, thrill, and the sheer joy of skiing.
6 The image captures thrilling moment of snowboarder in mid-air, performing impressive trick. The snowboarder clad in a vibrant red and white jacket. black pants, and a protective helmet, is the central figure this dynamic scene. The snowboarder is holding onto a rope with one hand, suggesting that they are being pulled up the mountain by a snowmobile, a common practice in snowboarding to gain speed and momentum. The rope extends from the top left corner of the image, creating sense of movement and direction. The backdrop of the image is breathtaking snowy mountain landscape. complete with trees and other natural elements. The snow snowboarder is positioned in the center of the image, drawing the viewer's 's attention to their daring stunt. The image is a testament to the exhilarating sport of snowboarding and the skill and courage of the athletes who participate in it. It's freeze-frame of adrenaline-fuedd fueled moment, encapsulating the thrill and excitement of snowhoarding.
# Token What is the number in the picture? (Figure E6 Row 2)
490 The number in the picture is "45", It is displayed on the back of the jersey that the person is wearing.
154 The number in the picture is "45",
18 The number in the picture is "45". It is displayed on the back of the baseball jersey worn by the player in the image.
6 The number in the picture is "45". It is displayed in a large, bold font on the jersey of the baseball player in the image.
# Token What is the relative position of the white and red cars in the picture? Which one in the left? (Figure E6 Row 3)
490 The white car is positioned to the left of the red car in the image. The white car is closer to the foreground and is stopped at the traffic light, while the red car is further back, also stopped at the same traftic light. The relative position of the cars is such that the white car is in the left lane and the red car is in the right lane, both facing the same direction of traffic.
154 The white car is positioned to the left of the red car in the image. The red car is further back on the road, indicating it is behind the white car. This is determined by the perspective of the image, where the white car is closer to the foreground and the red car is further away, both in the same lane and moving the same direction.
18 The white is the left side of the image, and the red car is the right side. The white car closer to the foreground, while the red car is further away, positioned behind the white car.
6 The white car is on the left side of the image, while the red car is the right side. The white car positioned closer to the foreground, while the red car is further back near the intersection.
28

---

Published as a conference paper at ICLR 2025
F ABLATION STUDIES
In this section, we conduct the ablation studies in several aspects including token merging, slowfast inference, efficiency, training strategy, and backbone selection.
Token merging. As a core strategy of AURORACAP, token merging plays significant role in reducing the number of visual tokens. We conduct extensive ablation studies explore the impact of the token kept ratio Retk in terms of performance across multiple tasks including image captioning. visual question answering, video captioning, and video question answering as shown in Figure F9 and Figure F10. We define the performance percentage the proportion between the highest and lowest values on the entire performance curve. We identify the minimum retention thresholds for achieving 90% and 80% performance. As shown in Figure F9, while the performance of AURORACAP generally declines with fewer visual tokens across most benchmarks, it remains relatively stable at higher retention levels. Most models maintain satisfactory performance (> 80%) even with only 0.4 of Retk. highlighting the efficiency of our approach. Visual token retention thresholds vary by task complexity, with more visually demanding tasks needing higher retention of visual tokens. For instance, CIDEr (Vedantam et al., 2015) on COCO-Cap (Veit et al., 2016) maintains over 90% performance with an Rytk of 0.3, whereas accuracy on GQA (Hudson & Manning, 2019) drops to 90% when the Rytk is reduced to 0.8. Unlike image understanding, the optimal performance across most video understanding benchmarks occurs at relatively low Reck as depicted in Figure F10. And for MSR-VE VTT (Xu et al., 2016), VATEX (Wang et al., 2019), and ActivityNet-QA (Yu et al., 2019), even achieve better results extremely low Retk (< 0.1). indicates that comparing to image, video input have higher redundancy. Note that AURORACAt focuses on spatial visual token merging, while the temporal features introduce additional complexity to explore the token merging laws. Appendix shows more calculation details and the visualization results of token merging.
Slowfast inference. Inspired by Slowfast-LLave (Xu et al., 2024b), we explore whether combining frames with low and high Rutk can enhance performance. In practice, we don't conduct token merging in the first frame and concatenate them with the merged tokens from subsequent frames. We apply this strategy to both video captioning and video question answering tasks, comparing performance with and without the inclusion of full first-frame visual tokens. As illustrated in Table F4, slowfast inference brings marginal performance improvement in video question answering tasks or even drop in video captioning tasks but with more computing cost. Therefore, by default, we don't 't using slowfast inference for video detailed captioning.
We also present the performance curve with and without the inclusion of full first-frame visual tokens as the visual token kept ratio varies during inference across multiple video understanding tasks. As illustrated in Figure Fl] I and Figure F12, despite the inclusion of full first-frame visual tokens, slowfast inference does not consistently result in significantly positive effects on performance. In some cases, the incorporation of full first-frame visual tokens even worsens performance degradation as the kept ratio decreases, particularly in video captioning tasks.
Table F4: Ablation on slowfast inference for AURORACAP-7B. We present the average performance among different token merging ratio various video understanding benchmarks. We show that slowfast inference brings marginal performance improvement in video question answering tasks or even drop in video captioning tasks but with more computing cost.
MSR VTT
VATEX B@4
ANet
MSVD
MSRVTT
Setting
C
B@4
M
C
B@ I
M
Acc
Ace
Acc
w/o slowfast
26.72
53.01
17.58
21.25
46.78
28.03
52.28
15.22
16.95
38.30
58.55
56.45
37.26
w/ slowfast
26.18
51.68
17.00
21.20
46.16
28.07
52.16
15.12
16.95
38.27
59.66
55.65
38.22
-0.54
-1.33
-0.58
-0.05
-0.62
+0.04
-0.12
-0.10
-0.01
-0.03
-0.80
+0.96
Efficiency. To assess the inference speed, we utilize the inference time per video question- -aswerin pair in seconds (TPV) as an evaluative metric. SGLang is an accelerated serving framework for LLMs and multimodal LLMs. We consider four settings including with or without token merging and SGLang. Figure F7 indicates the minimum TPV achievable in each settings across seven video understanding datasets. Reducing the visual tokens and using SGLang result in excellent inference times per video question-answering pair while all the datasets with short video and question inputs.
29

---

Published as a conference paper at ICLR 2025
Setting A
Setting a
Setting
Setting
1.591
1.635
1.119
1.975
2.201
1.892
7.454
100%
80%
60%
20%
0%
MSR-VTT
VATEX
ANet
MSVD
MSRVTT
IVQA
VDC
Figure F7: Comparison between different inference settings: A: Rosk 1.0, without SGLang, B: Rark 0.1, without SGLang, C: Rotk 1.0, with SGLang, D: Rock 0.1, with SGLang. The number indicates the maximum inference time seconds for each benchmark.
In contrast, maintaining full visual tokens or omitting the use of SGLann results in comparatively slower performance, demonstrating the superior inference efficiency of AURORACAP. For each input, we process the video at a resolution 378 378 and sample frames using single H100 GPU.
Training strategy. Alternative training strategies for the language stage of AURORACAP are less frequently explored, which is the primary focus of this section. For fair comparison, we use the same training datasets across all settings and maintain consistent hyper-parameters. The following training settings are explored:
Setting End-to-end training and set Rotk 1.0.
Setting B: Following Slowfast-LLaVA (Xu et al., 2024b), we retain full visual tokens in the first frame and concatenate with merged tokens using Roek of 0.1.
Setting C: End-to-end training and set Pysk 0.1.
Setting D: Most videos in the training data have no more than 8 key frames, while a subset (mainly from ShareGPT4Vide (Chen al., 2024d)) contains significantly more. We first exclude this subset from end-to-end training and then use it to train solely the LLM, enhancing its ability handle multi-frame inputs.
Setting E: Training solely the LLM match the performance Setting
Roth at 0.1.
We implement these training strategies, track training costs in H100 hours, and evaluate across various video understanding tasks. As shown Figure F8, while training with RR of 1.0 improves performance, it significantly increases training time. Surprisingly, mixing lower and higher visual token ratios during training offers no significant advantage. Training only the LLM under the two settings results in performance drop, indicating that enhancing long video understanding still requires collaboration with the finetuning the visual encoder. Therefore, we choose Setting C as the final training strategy.
We use the the training loss among the last ten iterations in original LLaVV alignment pretraining stage to guidance the ViT and LLM backbones selection as shown Table F5.
Backbone selection.
G DETAILED TRAINING SETTINGS
as the initialization of AURORACAP-7B. Training hyper- parameters for both stages are shown in Table G9. For visual data preprocessing, we resize each image video frame to the short side of 378 while keeping original aspect. Instead of doing center crop, we conduct bilinear position embedding interpolation for each training sample. For vidco data, extract frames at 2 FPS uniformly. For token merging, use constant schedule for each blocks.
We use CLIP ViT-H ", Vicuna-1.5..0
Huggingface Model: httpss///ogiiggaacc..o////pppppp
Huggingface Model:
-
30

---

Published as a conference paper at ICLR 2025
Setting A
Setting 2
Setting
Setting
Setting E
33.1
39.5
62.9
67.7
45.4
56.2
1120.0
100%
80%
60%
20%
0%
MSR-VTT
VATEX
ANet
MSVD
MSRVTT
IVQA
GPU Hours
Figure F8: Comparison between different training strategy in Language stage. We take Accuracy for Question-Answering tasks and CIDEr for captioning tasks the evaluation metric and present the performance percentage. We choose Setting as the final training strategy as shown with The number shows the maximum value for each benchmark.
Table F5: Final training loss during pretraining stage with original LLaVA pretraining data.
VIT
ViT Size
LLM
LLM Size
Loss
facehook/dinov2 giantt
1,136M
pnicrosoft/phi-2
2.7B
3,3021
openaiiclip vit-larg--aachh parge-patch11e 3336
Oween/Owen1.500.5BB
3.1001
openai/clip vit-large-patch 14-366
428M
microsoft/phi-2
2.7B
2.8067
laion/CLIP VTT BeT-bigG-11 14-la41n2B------- (aannnBB-99555 39B b160k
1,845M
microsolt/phi-2
2.7B
2.7124
Sreehook/linov2-giin
1,136M
mansys/vicuma-13ba.
13b-v1.5
138
2.3895
laion/CLIP- SWW.CCLP--VT--iiGG... -TT-bigGG-144aiin22---333
1,845M
intern
2,3437
laion/CIIIP Moon/CIIP-VVT-BBAe hi----ih---- 14 Ia-oo----------3-3333- laion2B-39B b160k
1,845M
mnternlm/inneemmm22eoo chat-mm--chh2222 chat-
20B
2.2745
laion/C1.IP. ViT-higii- hoi.lliinnnnnnnbll1l1111
1,845M
seeppeek-aideepssee... haae/deeppeccc-----6---hhhh 67b-chat
67B
2.1572
1,845M
mistralaiiMiistrale 7---staal-7----stttt-----0
7B
2.1569
openaticlip- www.-aaarr-ppcchhh4511
patch1
mistralai/Mixtnal 8x7B Instruct-vO.1
632M
Imsy s/viuuai 165
13B
2,0443
1,845M
2,0365 7B
1,845M
8x7B
1.9889
spemaicdip--iii-lngeeppcchhhhhhi
Imsy imsys/vicuma-
1,845M
13B
1,9708
laiou/CLIP-/T--biee Bii.Le--TTTbii-------oo--------
1,845M
Insys/vicuna- 13b-v1.5
13B
1.9412
apll:DENSB-CLII-VVIH-EL
632M
imsys/iccma-7--v155-666 7b-v1.5-16k
7B
1.8679
H EVALUATION BENCHMARKS AND SETTINGS
We list all the hyper-parameters and prompt used for evaluation as shown in Table H10. For our proposed VDC, we show the settings following with the max number of the tokens of 1,024:
Type Prompt used for evaluation
Camera Describe any camera zooms, pans, angle changes.
Background Summarize the background setting of the video based on these frames.
Main Object Describe the main subject, including their attributes and movements throughout the video.
Detail Imagine the video from these frames and describe it in detail.
31

---

Published as a conference paper at ICLR 2025
Flickr
BQ4 Flickr
on Flickr
85-
80
75
70
26
0.8
0.2
Radio
token
58
ROUGE-L on Flickr
CIDEr
E:44 NoCaps
54-
ROUGE-
C1DTY
300
50-
40-
0.8
3.2
2/4
Radio siuual
Rarto socull bokke leps!
Rallo tolken
METEDR NoCaps
ROUGE-L on NoCops
COCD-Cap
63-
METEOR
335
50
28
55
105
25
1.0
0.8
1.0
Rado vauua
tosen kecp
Ratio
BB4 on CO
Cap
ROUGE COCO
34-
33-
NETICA
ROUGE
27
83
32
33
20-
0.8
Radio Viuual
tooen beit
Rai tolknn
Accuracy on GOA
Accuracy NMMU
Accuracy on VQAw2
36
335
20
92
Ratto visuull taken kept
visual
Figure F9: Ablation study of token merging on image captioning on Flickr (Young et al., 2014), NoCaps (Agrawal et al., 2019), COCO-Cap (Lin al., 2014), visual question answering in GQA (Hud- son & Manning, 2019), MMMU (Yue et al., 2023), VQAv2 (Goyal et al., 2017). We found that token merging significantly reduces the number of tokens while maintaining minimal performance drop. and even showing improvement in some tasks. We highlight the token merging ratio when achieving 90% and 80% performance with the dash line and filled area.
32

---

Published as a conference paper at ICLR 2025
on
on
32
30-
96
25
24
14-
1.0
0.8
0.2
Rado vasual
tooen
Rat tolcen
24
23
20
3.0
0.8
0.6
0.6
0.4
0.2
0.0
Radlo visual son
Rado viuul tosson Nest
Rot viauaa bolcon ku
B81
54-
5:4
4.-
14
44-
0.9
0.2
Razo visual
tosen
Raat tolcen
Accuracy ActivityNet-OA
Accuracy
42.5
40-
40.0
38
15.0
32.5
34
10.0
Rall Iosken
Score on ActivitylettQe KAt-QA
Accuracy QA
Accuracy
3.5
3.4
3.5
Figure F10: Ablation study of token merging on video captioning on MSRVTT (Xu et al., 2016), VATEX (Wang et al., 2019), video question answering on MSRVTT-QA (Xu et al., 2016), ActivityNet- QA (Yu et al., 2019), MSVD-QA (Xu et al., 2017), iVQA (Liu et al., 2018). We found that token merging significantly reduces the number of tokens while maintaining minimal performance drop. and even showing improvement in some tasks. We highlight the token merging ratio when achieving 90% and 80% performance with the dash line and filled area.
33

---

Published as a conference paper at ICLR 2025
Table G6: Summary of datasets used for training AURORACAP in Pretraining stage.
Tusk
Sample
Dataset
(Liu el al,, 2024b), ShareGPT4V (Chen et al., 2023a), MaLLaAA-Cppion-LAIOO (Chen et al., 2024a), VaLLVA-Capttin--V VFLAN- (Chen al., 2024a), DenseFusion (Li et al., 2024e)
Image Captioning
1.3M
Table G7: Summary of datasets used for training AURORACAA ACAP in Vision stage. For classification, Reasoning, VQA, and Generation tasks, we adopt the dataset processed by MSIT (Li et al., 2023c) to fit the training objective of language models.
Task
# Sample
Dataset
ShareGPT4V-PT (Chen al., 2023a). TextCaps (Sidorov 2020). Image- Paragraph-Catiinn Captioning (Krause al., 2017)
Captioning
1,925K
Object-centric
438K
COST (Jain el al., 2023), ChatterBox (Tian et al., 2024), V (Wu & Xie, 2023) COCO-GO1 (Lin ct al., 2014), COCO-Text (Veit et al., 2016), ImageNet (Rus- sukovsky et al., 2015), COCO-ITM (Lin ul., 2014),
Classification
238K
+-SLL-VE (Kayser et al., 2021), Mocheg (Yau al., 2023), IQA (Duanmu
al., 2021)
CLEVR (Johnson et al., 2017) NLVR (Suhr et al., 2017), VCR (Zellers et al., 2019), VisualMRC (Tanaka et al., 2021), Winoground (Thrush et al., 2022)
Reasoning
100K
VQA v2 (Goyal al., 2017), Shapes VQA (Andreas et al., 2016),
DocVQA (Mathew et al., 2).1). OK-VQA (Marino et al., 2019).
Text-VQA (Singh al. 2019). OCR-VQA (Mishra et al., 2019). A-OK-
VQA
3,518K
VQA (Schwenk al., 2022). ScienceQA (Lu et 2022))
ST-VQA (Biten et al. 2019), VIQuAE (Lemer et al., 2022), LLaVA-
OneVision (Li ct al., 2024a)
Visual Sturytelling (Huang al., 2016), Visual Dialog (Dus al., 2017),
Generation
145K
Multi30k (Elliot et al., 2016)
COCO-Caption CN (Li al., 2019), Flickr-tt-Caption (Li et al., 2016),
Chinese
193K
multimodal Chat (Zheng al., 2021), FM-IQA (Gao al., 2015), ChineseFood-
Net (Chen 2017)
Total
6.6M
For datasets, we uniformly sample without duplication.
Table G8: Summary of datasets used for training AURORACAP in Language stage.
Task
# Sample
Dataset
ShareGPTaV (Chen et al., 2023a), ALLaVV--Cption-LAION--. (Chen al., 2024a), ALLLVV- Caption-VFLAA VFL.----AAA (Chen et al., 2024a), DenseFusion (Li ct al., 2024c), FaceCaption (Dai et al., 2024a)
Image Captioning
1,779K
MiraData
al., 2024). LLaVA-Hound (Zhang al. 2024c). ShareGPT4Video (Chen et al., 2024d), Private Data
Video Captionin
1,639K
LLaVA-Mix-665K (Liu et al., 2023), LVIS- VVSs-nnstruetAV (Wang et al., 2023),
Ler.V---nstut--AAONN.... (Chen al,, 2024a), ALLaVA-Isttmct-VILE VFLLN. (Chen al., 2024a), Cambrian (Tong al., 2024a), M4-Instruct (Liu al.,
Image Instruction
9.742K
2024a)
Video Instruction
268k
LA Hound (Zhang et al., 2024c), ShareGPT4Vidde (Chen et al,, 2024d)
Language-only
143K
Evol- Antruut-GGT4-Tuunne 143K (Chen et al., 2024a)
Total
15.0M
We duplicate video captioning and instruction datasets twice.
34

---

Published as a conference paper at ICLR 2025
CIPC on NSS-VTT
Sigh en MISVVTT
VETEON on MINVIT
213
19
12.5
Parta Of Ciusl/ooie Solln Aeg
CEDE on
Sgod on
214
Parta of Siaudfolon Soclen Beg:
Fa dtsad Soher Beg!
Rado-d Fraid orfen
Accuracy
Eegt
Regt
Scars
Accuracy AcDrkyMey-
Actetomoy
2.0-
3.5
2.5
2.4
0.2
Figure F11: Comparison between performance with and without the inclusion of full first-frame visual tokens during inference on video captioning on MSRVTT (Xu et al., 2016), VATEX (Wang al., 2019), video question answering on MSRVTT-QA (Xu al., 2016), ActivityNet-QU (Yu et al.. 2019), MSVD-QU DAVDD-QAA (Xu et al., 2017), iVQA (Liu et al., 2018). As the visual token kept ratio varies, the blue curve indicates performance with slowfast inference, while the red curve represents performance without slowfast inference.
35

---

Published as a conference paper at ICLR 2025
Accoracy on Camure Caston
Secre on Camera Coption
Accurscy on Short Castinn
1477 Of REALD Salan
F003-C dial SiArr degt
Shar Cantion
Aecunny on Boclggued
Scor Backgmunt Caprion
1.75
2.0
153
33
Parta Todue Beyt
lccuracy Man Orject Centien
Main dowee Caption
Acturayy Caprion.
2.30
1.5
35
362-
Eagt
Caption
Scarc
2.3
Eegt
SAAT AEgT
Figure F12: Comparison between performance with and without the inclusion of full first-frame visual tokens during inference on VDC. As the visual token kept ratio varies, the blue curve indicates performance with slowfast inference, while the red curve represents performance without slowfast inference.
36

---

Published as a conference paper at ICLR 2025
Table G9: Training
for AURORACAP.
Hyper-parae parameters
Pretraining stage
Vision stage
Language stage
trainable parameters
MLP
VIT MLP
VIT + MLP + LLM
warmup schedule
linear
warmup start factor
Ie-5
warmup ratio
0.03
learning rate schedule
cosine decay
optimizer
AdamW (Loshchilov Hutter, 2017)
optimizer hyper- seper-parameters
B1, 82 (0.9, 0.999)
weight decay
0.1
max norm
epoch
peak learning rate
2e-4
le-4
2e-5
total equivalent batch size
512
6,144
768
token keep propurtion
100%
100%
10%
Table H10: Evaluation settings summary for each benchmarks. For all benchmarks we set temperature, top p. number of beams to 0, respectively.
Benchmark
# Sample
# Takens
Prompt
Flickr (Plummer 2015)
31,784
Provide
caption for provided image. Provide one-sentence caption for the provided image.
NoCaps (Agrawal et al., 2019)
4,500
64
COCO-Cap Cap0 (1 al., 2014)
5,000
Provide one sentence caption for the provded image.
ChariQA (Masry al., 2022)
question with single word.
VQA (Mathew et al., 2021)
5,349
32
Answer the question using single word phrase.
TextCaps (Sidorov al., 2020)
3,166
64
Provide one une-sentence caption for the provided image.
GQA (Hodson Manning, 2019)
12,578
Answer question using single word or phrase.
POPE (1. al., 202.3h
9,000
128
Answer the question using single word phrase.
Answer with the option letter from the given choices directly.
MMMU (Yue ol.,
900
Answer question using single word or phrase.
VQAzZ (Goyal 2017)
214.354
Answer the question using single word phrase.
MSR-VTT (Xu al,, 2016)
1,000
64
Provide one une-sentence caption for the provided video.
VATEX (Wang et al,, 2019)
4.478
Provide hrief single sentence caption the last video helow.
MSVD-QA (Xu al., 2017)
question using single word or phrase.
ActivityNetQQA (Yu et al., 2019)
8,000
Answer the question using single word or phrase.
MSRVTT-QA (Xu al,, 2017)
6,513
Answer the question using a single word or phrase.
(Yang 2021)
question using single
plarase.
37

---

Published as a conference paper at ICLR 2025
I
LIMITATIONS
Table 111:
Comparison AURORACAP with LLM-based SoTA methods on visual question answering
benchmarks under zero-shot setting.
MMMU (900)
GQA (12,578)
POPE (9,000)
VQAv2 (214,354)
Model
LLM
Acc
Acc
F1
Acc
LLaVA-1.5-7B
Vicuma-1.5-77
35.30
61.97
85.87
76.64
LLaVA-I.5-13B
Vicuma-I.554 1.5-13B
34.80
63.24
85.92
78.26
LLaVA-I.6-7B
Vicuma-1.5-7B
35.10
64.23
86.40
80.06
LLAVA-1.6-13B
Vicuma-1.5-13B
35.90
65.36
86.26
80.92
AURORACAP-7B
Vicuma-1.5-78
36.11
59.72
83.31
75.85
Table 112: Limitation terms of OCR capability compared with LLaVA models. Appendix shows
the introduction and metries of each benchmark.
ChariQA (2,500)
DocVQA (5,349)
TexiCaps (3,166)
Model
LLM
Acc
Acc
Acc
LLaVA-1.5-7B
Vicuna- 1.5-7B
18.24
28.08
98.15
1LaVA-1.5-138
Vicuna- 1.5-138
30.29
103.92
LLaVA-L.6-BB
Vicuna-1.55 1,5-7B
54.84
74.35
71.79
LLaVA-1.6-13B
Vicuna-1.5-11
62.20
77.45
67.39
AURORACAP-7B
Vicuma-1.5-7B
25.88
34.60
93.33
We evaluate AURORACAP on various visual question answering benchmarks shown in Table I11. Since the performance of the VQA task heavily depends on the performance of the LLM,, we chose the same I.LM for a fair comparison. Also, due the the limitation of OCR-related samples in training dataset, AURORACAP does not perform well in OCR at current stage as shown in Table 112.
J ADDITIONAL EVALUATION METRICS ANALYSIS
In this section, we make the analysis about more evaluation metrics on VDC other than VDCSCORE. As shown Figure JI3, we found that the correlation between CLIP score and human ratings is low (0.39), which may be due to the fact that, in lengthy caption, there may be only one two fine-grained errors that the CLIP text encoder lacks the capability to detect. Meanwhile, we test the impact of different numbers of QA pairs the results when calculating VDCsCORE. As shown in the Figure J13, more QA pairs produced more robust test results. VDCSCORE shows over 0.86 of pearson correlation with human when using 20 QA pairs
Correlation Confficient
Correlation Confficlent
Correlation
7.5
7.4
0.40
VOEScore
7.0
vorSsore
7.2
0.35
Cip
900
1000
1100
1200
900
1000
1100
1200
900
1000
1100
1200
Elo Score
Bo Score
Fio Score
Figure J13: Correlation between the CLIP evaluation scores, different number of QA pairs used in VDCsCORE and human clo ranking VDC.
38

---

Published as a conference paper at ICLR 2025
K VDC GENERATION PROMPT TEMPLATE
Following (Ju et al., 2024), we utilize LLM to generate structured detailed captions. Given an input video, LLM return five detailed captions, including camera caption, short caption, background caption, main object caption and detailed caption for the entire video guided by our designed prompt template. The complete prompt is shown as followings:
Type Prompt
SYSTEM You are describing the video. Please provide detailed captions of the video from different aspects.
User
Please provide detailed and comprehensive captions for the following content: Short Caption: Summarize the video in one detailed sentence, capturing key actions and the overall mood. 2. Background Caption: Provide detailed description of the background. including objects. location. weather, time, and any dynamic elements such movements in the environment. 3. Main Object Caption: Give a thorough description of the main subject's actions, attributes, interactions, and movements throughout the video frames, including changes in posture, expression, or speed. 4. Camera Caption: Describe the camera work in detail, including shot types, angles, movements, transitions, and any special effects used to enhance the video. 5. Reference Caption: Generate detailed dense caption for the video that is at least 300 words long. The caption should capture all visible actions, environmental details, and the overall emotional atmosphere in depth. Describe in detail the interactions between the main subjects and their environment, including subtle nuances of their movements or expressions. Elaborate on the sounds, textures, and other sensory experiences depicted in the video. Discuss the camera techniques used extensively, including shot types, angles. movements, and transitions. Highlight the mood and tone of the video throughout, creating rich narrative that connects viewers emotionally to the scene. Include comprehensive descriptions of background elements that add context and depth, such as weather conditions, time of day, and cultural or historical settings. Make sure to provide a vivid portrayal that is engaging, informative. and rich enough for Al to re-generate the video content.
No need to provide summary content. Do not describe each frame individually, Avoid using phrases like 'first frame'. The description should be rich enough for Al to re- the video. Please generate the response as a Python dictionary string with keys like 'short caption'. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. These the frames from the video: [VIDEO KEYFRAME INPUT]
GPT-4o Short Caption: In sunny outdoor setting, two workers engage collaborative effort to prepare gravel base for construction project, showcasing teamwork and focus.
Camera Caption: The camera captures the scene from a low angle, providing a unique perspective that emphasizes the workers' actions and the surrounding environment. The shots are primarily medium close-ups, focusing on the workers' upper bodies and hands as they interact. The camera occasionally shifts to wider angle, revealing more of the workspace and background elements. There smooth transitions between shots, maintaining a steady flow that enhances the viewer's 's engagement with the unfolding narrative. The use of a slightly fish-eye lens effect adds depth to the scene, making the environment feel more immersive. Background Caption: The scene is set in residential area with a well- malltanint lawn and a brick wall partially constructed in the background. The ground is covered with white plastic sheet, and scattered gravel is visihle. A few wooden planks lie the ground, and tools are scattered around, including a black water bottle and a power tool.
Main Object Caption: The main subjects are two workers, one wcaring a gray sweatshirt and gloves, and the other dressed in casual work attire. They are actively engaged in conversation, with the first worker gesturing with his hands emphasize points while explaining the task at hand. His posture is relaxed yet focused, indicating a collaborative atmosphere. The second worker listens attentively, occasionally nodding and responding with gestures, showcasing a dynamic interaction. Both workers exhibit sense of camaraderie and professionalism as they prepare for the next steps in their project. Reference Caption: The video unfolds... ... (300 words)... ... through them.
User
The video has been describe from the following aspects: 1. short caption: short caption, 2. background caption: background caption, 3. main object caption: main object caption, 4. camera caption: camera caption, 5. Refeerence Caption: reference caption Please generate detailed dense caption for the vidco that is pretty long. You should expand the reference caption upon the information provided in the short caption, background caption, main object caption, and camera caption. Ensure that the detailed caption does not introduce any new entities relationships that were not mentioned in the previous captions. Make sure to provide vivid portrayal that is engaging, informative, and rich enough for Al to re-generate the video content. "Avoid using phrases like 'first frame'. 'short caption', 'background caption', "main object caption', and 'camera caption'. The description should be rich enough for Al to re- re-generate the video.
GPT-4o Detailed Caption: The video opens with... (500 words) literature that surrounds them.
User [Manually check and refine]
39

---

Published as a conference paper at ICLR 2025
L QUESTION-ANSWER PAIRS GENERATION PROMPT TEMPLATE OF VDCSCORE
To decompose the ground-truth structured detailed captions in VDC, we utilize 40 as the LLM assistant to generate numerous short question-answer pairs for subsequent evaluation. The full prompt and example cases are presented as followings:
Type Prompt
SYSTEM
You intelligent chatbot designed for generating 20 question-ansee -anseer pairs given detailed description of video or image. You are describing the video.
Here's how you can accomplish the task: INSTRUCTIONS: Cover the main objects and actions in the video or image.
The questions should be open-ended and start with 'What', 'Who', 'Where', 'When', 'Why', 'How', etc.
TTee answer should be a short sentence or phrase.
Generate 20 question-answer pairs.
User Please generate question-answer pairs given a detailed description video or image: detailed description: The camera work is dynamic, featuring a mix of close-up shots and wider angles that provide context to the environment. The shots are slightly shaky, adding a sense of realism and immersion the viewer. Transitions between shots are smooth, with the camera often panning tilting to follow the subject's movements. The use of low light creates moody atmosphere, with shadows enhancing the visual interest the scene
Please generate the response in the form of a Python list of tuple with the question and the correspond- ing answer. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python list of tuple. For example, your response should look like this: [(the question the answer 1), (the question 2, the answer 2). J.
Go I"question": "What is the style of camera work in the video?", "answer": "dynamic", "question"" "What is the effect of shaky shots in the video?", "answer": "realism and immersion", "question": "What type of transitions are used in the video?", "answer": "smooth", "question": "Why is the moody atmosphere created in the video?", "answer": "through low light". "question": "What type of camera movement achieved through panning and tilting?", "answer": "following the subject's movements". "question": "What the effect of the camera movement on the viewer?", "answer": "sense of realism and immersion". .
User [Manually check and refine]
M MORE STATISTICS INFORMATION OF VDC
Figure M14 indicates the word distribution of the structured captions in VDC. Table M13 illustrates the distribution of the question categories we provided for VDCsCORE calculation.
transit tion
sppriiite protagonist Vibrant wor-Fo adventure int interniine
enhancing
Bit sound
enhancio
SELLITE
Ceating
JKV
forus
SENVE
Freatin
Orrien
Captull Jesierr
protagon ist lenbarks Journey filled Laughter unexpected twists
S cene Set
camera work
atmospher phere
br-ihht
(a) Camera Caption
(b) Short Caption
(c) Background Caption
pos ture & main fo expression
main subject
STV FONTEO WWW.ERRRRTOOVOOO MOSAn
scene eeveewrr
S C movementt
Camera whorksoocssinn LDDE
rerfeeetinnn Lictingg
Light
hand sense
Stam
Video
Vlewer
creating menemmveeen
hallt
video
Sho
V videeshoowashh
moment create nature
ephanc
(d) Main Object Caption
(c) Detailed Caption
(f) Total
Figure M14: Word cloud of different structured captions in VDC, showing the diversity.
40

---

Published as a conference paper at ICLR 2025
Table M13: Distribution of the question categories in VDCSCORE.
Type
Background
Camera
Short
Main object
Detailed
Environment Scene
3,464
2,543
4,401
3,097
3,549
Character & Object Intent Outcomes
6,272
3,779
4,834
6,035
6,440
4,585
5,105
5,067
4,474
4,640
Action
2,861
1,198
1,857
3,192
1,932
Attributes & Relationship
944
334
1,388
1,478
1,280
N MORE STATISTICS INFORMATION OF VDCSCORE
We generate total of 96,902 question-answer pairs for VDC, with average of 18.87 pairs per detailed caption. As depicted Figure N15, each section of the structured captions includes similar number of question-anse answer pairs. Additionally, Figure N16 presents the distribution of question types generated for VDC. To enhance the evaluation of detailed captions, we configure all questions open-ended. Envir ronment. environment, atmosphere, scene, and time. Character & Ob Objettt focuses on entities within the caption, such as people, animals, plants, and objects, while Attribute amines their attributes and interrelations. Intent & Outcomes addresses deeper interpretative questions regarding methods, purposes, reasons, and outcomes. We further analyze the distribution of generated question types within structured captions. Main object captions predominantly fea- ture Charas aarterter Ob ject questions, whereas Environment & Scene questions are more prominent in background captions, and Came ra questions constitute a larger proportion in camera.
Scene encompasses inquiries about location,
Relation ex-
Main Object Caption 19.83%
Attribute & Relation
Environment & Scene 15.22%
4.84%
Camera Caption
9,85%
19.72%
Camera 6.78%
Character
Short Caption
Object 24.42%
20.14%
Detailed Caption 20.44%
19.77% Background Caption
28.72% Intent Results
10.17% Emotion
Figure N15: Question-answen pairs propor- taion in structured captions.
Figure N16: Distribution of question type in VDCSCORE.
O CALCULATION OF ELO RANKING
In this section, we present the methodology for evaluating and ranking AURORACAP and various models using Elo rating system as shown Figure 017. The parameters used in the simulation are summarized in Table 014.
For all models participating the ELO ranking, we collecte the output captions for each video. As shown in Figure O17, we develop a frontend tool to randomly select captions generated by different models for the same video. Without revealing the model IDs, human evaluators then chose the caption they found better.
3. We record comparison results and calculated the ELO values based on the parameters in Table 014.
4. In Figure 5, we calculate the Pearson correlation between different metrics and human ELO values, ultimately showing that proposed VDC metric the closest to human evaluations.
41

---

Published as a conference paper at ICLR 2025
Table 014: Elo parameter setting.
Parameter
Number
initial Elo mean
1,000
Elo standard deviation base of logarithm scaling factor
300
10
400
K-factor
32
minimum Elo rating
700
number of simulated matches
2,778
Caption Comparison Interface
Log
the dhes ei person seen room with couch dia 1ab% The perion ahe gingiia broom This standing
The vites featune person in grey awentah insercting with brown paper bag The incldees couch with patte cover table
cowerd
with brige blortct the thore ii paper with some you on but contentis contents The oveelll carting Auggedts canual, cocaaliiddi indber emaiao timentt.
cooux indior enstroment The perti'' actions are for indicdinng that
Choose this
Save Record
Figure 017: GUI screen for Elo ranking.
P CASE STUDY
We perform an extensive case study of AURORACAP on a variety of videos for video detailed captioning. As shown as followings, AURORACAP is capable of providing excellent detailed captions regarding the camera motion, background and main object with less hallucination.
Figure P18: Video example of MSR-VTT (Xu et al., 2016) benchmark.
Figure P19: Video example of VATEX (Wang et al., 2019) benchmark.
42

---

Published as a conference paper at ICLR 2025
Figure P20: Video example.
Figure P21: Video example.
Figure P22: Video example.
Figure P23: Video example.
Figure P24: Video example.
Figure P25: Video example.
Example Caption
Figure P18
[short caption] A with mustache and woman with ponytail are sitting at a table. [short caption] A man plays the drums in recording studio.
Figure P19
Figure P20 sandy shore, guzing out towards the ocean. She is wearing black beanie and pink jacket, adding a pop of color the otherwise muted scene. The heach, expanse of sand, stretches out
[detailed caption] The video depicts serene beach scene where young woman stands on the
43

---

Published as a conference paper at ICLR 2025
in front of her, meeting the ocean at the horizon. The ocean, a vast body of water, is visible the background. The beach bathed in a soft, diffused light, creating dreamy atmosphere. The girl's gaze is directed towards the horizon, suggesting a sense of wonder or contemplation. The image is slightly blurred, adding a dreamy quality to the seene. The woman's 's position on the beach, coupled with the gentle waves of the ocean, suggests moment of contemplation or admiration. The relative positions of the objects suggest peaceful day at the beach, with the girl possibly enjoying the serene view the ocean. The colors are mostly muted, with the girl's 's pink jacket standing out against the sandy beach and the blue ocean. The blurred background and the out-of-focus elements, such as the ocean and the sky. contribute to the sense of tranquility and text present in the video, and the colors are muted, with the
focus on the woman. There exception of the pink jacket, which stands out against the more subdued tones of the surroundings. [detailed caption] The video features man with beard and long hair, lying on the ground with his head resting on tree trunk. He wearing a colorful shirt with a mix of orange and green patterns. The man's 's face is partially obscured by the tree trunk, but his eyes are visible, looking upwards. He appears to be smiling or laughing. with his mouth open. The man's 's arms are crossed over his chest, suggesting relaxed playful posture. The background is blurred, but it appears be a natural outdoor setting, possibly forest park. In the background, another person partially visible, their presence subtly hinted at by a hand reaching out from the left side of the frame. The colors in the video are warm, with the man's 's shirt standing out against the muted background. The man's 's hair and beard are dark, and he has light complexion. The overall mood of the video is lighthearted and carefree. There are no visible texts other objects in the video, and the relative positions of the objects remain constant with the man in the foreground and the second person in the background.
Figure P21
Figure P22 [background caption] The background features a race track with visible tire marks and barriers. surrounded by grassy areas and a few scattered trees. The track is set in a rural or semi-rural location, with hills in the distance and cloudy sky overhead, suggesting overeast weather conditions. The track itself appears well- well-maintained with a smooth surface, designed for high- speed racing. The weather, indicated by the cloudy sky, contributes the overall ambiance of the scene, enhancing the sense of a cool, possibly early morning or late aftermoon setting.
Figure P23 [camera caption] The camera maintains steady, frontal view throughout the video, capturing the woman's 's expressions and the intricate details of the bookshelves. The camera starts at the right edge of the bookshelf, moving across the shelf, and ends at the left edge of the bookshelf. The framing is consistent, focusing the woman and the bookshelves, with the camera positioned medium distance to the subject. This movement provides comprehensive view of the bookshelf, showcasing the variety of books and their arrangement on the shelves. The camera occasionally pans to reveal the depth of the library, showcasing the rows of books and the inviting atmosphere. The use of natural light enhances the visual appeal, creating a warm and inviting tone throughout the video.
Figure P24
[camera caption] The view shot remains relatively statie, focusing the children playing in the backyard. The camera angle is at eye level, capturing the scene from a distance that allows both children to visible. There is minimal camera movement, maintaining a steady focus on the children and their activities. The sequence of video frames suggests a continuous moment of play without significant changes in shooting angles or camera movement.
Figure P25
[main object caption] The main subject in the video, a black car, is seen driving down a street that appears to be in state of disarray. The car moves steadily forward, navigating around obstacles such blue car parked on the side of the road. The car's 's movement is smooth and continuous, suggesting it either in motion or has just come to a stop. The environment around the car chaotic, with debris scattered across the road and signs of destruction, indicating a recent event or disaster. The car's position remains central in the frame, with the camera angle focused on it from a slightly elevated perspective, possibly from a vehicle or structure above.
Q PREDICTED ANSWER EXTRACTION PROMPT TEMPLATE
Given the question-answer pairs based on the ground truth caption, we utilize GPT-4o to extract predicted answers based on the generated caption by our designed prompt template. The complete prompt is shown as followings:
Type Prompt
SYSTEM You are an intelligent chatbot designed for providing accurate answers to questions related to the content based on a detailed description of video or image. Here's how you can accomplish the task:"
44

---

Published as a conference paper at ICLR 2025
#HINSTRUCTIONS
Read the detailed description carefully.
Answer the question only based on the detailed description.
The answer should be short sentence or phrase.
User Please provide accurate answers to questions related to the content based on a detailed description of
vidco or image:
detailed description: The setting is urban street with distinctive red and white painted bike lane, which is part of larger roadway. The lane is marked with white lines and the word "WALK" painted in large, bold letters, indicating a pedestrian crossing area. The pavement and grey tiles, with curb separating the bike lane from the adjacent road. The weather appears to be clear and sunny, casting sharp shadows the ground, suggesting it is daytime. There are no other vehicles or pedestrians in sight, and the road empty, which could imply quiet time of day less busy area. The presence a yellow circular object on the handlebar of the bicycle notable, possibly bell or a light.
mix red brick
question: What objects are the cyclist's 's hands interacting with? DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION, Only provide short but
accurate answer.
GPT-4o The cyclist's hands interacting with the bicycle handlebars.
R CORRECTNESS EVALUATION PROMPT TEMPLATE
Following (Maaz et al., 2023), we evaluate the correctness and score of the predicted answers with the assistant of GPT-4o. Given the question, correct answer, and predicted answer from the generated caption, GPT-4o o should return the True False judgement and relative score (0 to 5). The complete prompt is shown as followings:
Type Prompt
SYSTEM You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-
answer pairs.
Your task is compare the predicted with the correct answer and determine if they match meaningfully, Here's how you can accomplish the task:
#INSTRUCTIONS:
Focus on the meaningful match between the predicted answer and the correct answer.
Consider synonyms or paraphrases valid matches.
-Evaluate the correctness of the prediction compared to the answer.
User Please evaluate the following video- based question question-answer answer pair:
Question: What type of surface can be seen covering floor?
Correct Answer: carpet
Predicted Answer: Carpeted surface.
Provide your evaluation only yes/no and where the score is an integer value between 0 and 5. with indicating the highest meaningful match.
Please generate the response in the form of a Python dictionary string with keys 'pred' and "score", where value of pred'' is a string of "yes" or 'no' and value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string.
For example, your response should look like this: ('pred': 'yes', 'score': 4.8). GPT-40 ('pred': 'yes", 'score': 5}
45


Reviews:
[{"review_id": "0ukflQZIFC", "reviewer": "Reviewer_K52W", "summary": "Towards the video detailed captioning task, this paper conducts comprehensive research including the introduction of a new model called AuroraCap, a novel dataset VDC and a novel evaluation metric VDC-Score. For AuroraCap, authors apply the token merging strategy on large models. Extensive experiments have been conducted to verify the effectiveness of AuroraCap, and the necessity of VDC and VDCScore. This work contributes significantly to advancing the field of detailed video understanding.", "questions": "1. How many GPU hours have been consumed on this research?\n2. Will you publish the data, the pre-trained model with weights and also the training recipes?\n3. In the future, I believe the VDC benchmark will facilitate more research on the video, including classification, video generation, etc.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThis paper is well-written.\n2.\tThe proposed benchmark VDC is large-scale and diverse, which contains short, background, main object, camera and detailed captions. \n3.\tThe average length of captions in VDC exceeds previous datasets, to facilitate the research of understanding and also generation of long videos. \n4.\tVery extensive experiments.", "weaknesses": "1. The paper is not very easy to follow, owing to many tables and figures. \n2. For all the compared captioning models, are they trained under the same schema? If true, which one setting is chosen among different settings presented in Figure D5 and Figure D6?\n3. In Table 6 – Table 8, what does the accuracy mean? \n4. For VDCScore, it is hard to evaluate which score is better among VDCScore, VDD, CIDEr and maybe CLIP-based score, RefCLIP-S[1] and RefPAC-S[2].\n\n[1] Hessel, Jack, et al. \"Clipscore: A reference-free evaluation metric for image captioning.\" arXiv preprint arXiv:2104.08718 (2021).\n\n[2] Sarto, Sara, et al. \"Positive-augmented contrastive learning for image and video captioning evaluation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PGcDY8X9gr", "reviewer": "Reviewer_7VHc", "summary": "The paper introduces AuroraCap, an LLM-based video captioning model that uses token merging to reduce visual tokens. AuroraCap is evaluated across multiple image and video captioning datasets. To assess models' ability to generate detailed captions, the paper also proposes a new benchmark, Video Detailed Captioning (VDC), along with an LLM-assisted metric, VDCscore. VDC includes over 1,000 annotated structured captions, while VDCscore transforms long captions into multiple short question-answer pairs. Various existing models are evaluated on the VDC benchmark.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The presentation of the paper is clear. It's easy to follow.\n* The proposed VDC benchmark is a valuable contribution to the community. The detailed video captioning capability of video LLMs is an area that has been relatively underexplored, and introducing VDCscore adds a meaningful component to the evaluation pipeline. The evaluation process of VDCscore aligns well with human reasoning process.\n* Extensive experiments are conducted.", "weaknesses": "* Certain aspects of AuroraCap are not fully explored. For instance, while it performs well on the ANet dataset in video QA tasks, it performs poorly on MSVD, which is typically an easier benchmark. What might be causing this discrepancy?\n* Unfair comparison:\n  * Table 2 refer to the baseline models as the SoTA methods on image captioning benchmarks under zero-shot setting. However, I believe there are other LLM-based models with stronger captioning performance. For example, the BLIP family—BLIP2 [1], which achieves a 121.6 CIDEr score on NoCaps (val set), and InstructBLIP [2], which is even stronger. It would be beneficial to clarify the criteria used to select these baselines.\n  * Similar concerns apply to the video captioning and video QA task comparisons.\n* I notice that the VDCscore relies heavily on the use of GPT-4o:\n  * As also pointed out by other work [3], different versions of GPT can affect the results of GPT assisted evaluation heavily. Although I may have missed it, I did not see which version of GPT-4o was used for the evaluation in the paper. Standardizing the evaluation method is essential for consistency in future studies.\n  * Given the multiple variables that can influence GPT-assisted evaluation, and since VDCscore already uses phrased answers, what are the potential drawbacks of using an automatic metric based on, e.g. n-gram matching, to evaluate the final score? Could a variant of VDCscore be devised that assesses final triplets (i.e., <question, correct answer, predicted answer>) without relying on GPT?\n* minor: Several instances of \"BELU\" should be corrected to \"BLEU\"; small grammar errors like in line 203: \"are only contains\" --> \"only contain\"\n\n\n\n[1] Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023.\n\n[2] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. NeurIPS 2023.\n\n[3] Wenhao Wu. Freeva: Offline mllm as training-free video assistant. arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lo86RABi04", "reviewer": "Reviewer_BpiW", "summary": "1.This paper proposes AURORACAP, a video captioner based on a large multimodal model. AURORACAP follows a simple architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, it implements a token merging strategy, reducing the number of input visual tokens with little performance loss. AURORACAP shows superior performance on various video and image captioning benchmarks. \n2.Existing video caption benchmarks only include simple descriptions, so the authors develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. \n3.The authors also propose a new LLM-assisted metric VDCSCORE for better evaluation, which transforms long caption evaluation into multiple short question-answer pairs.", "questions": "the questions are listed above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.the proposed AURORACAP shows good performance. Using bipartite soft matching method to merge similar token is novel and show potential on token reduction.  \n2.VDC with detailed video caption is good contribution for video captioning research community, and authors provide the detailed description of dataset curation. \n3.the proposed metric VDCscore is a good improvement for captioning task evaluation with novelty. \n4.Paper is in good writing for understanding.", "weaknesses": "1.AURORACAP: there is no description of the AURORACAP's 1.3M pretraining data. We could't conclude the reseasons for the good peroformance of AUROREACAP. Could authors provide details on:\n\na. The sources and types of data included in the 1.3M pretraining dataset\n\nb. Any preprocessing or filtering steps applied to this data\n\nc. How this pretraining data compares to datasets used by other models? \n\nThis information would help readers better understand the factors contributing to AURORACAP's performance.\n\n2.VDCSCORE: the potential stability issue of the metric.\n\na. Is there a fixed number of question-answer pairs generated for each caption, or does it vary?\n\nb. If it varies, how do the authors ensure consistency in the metric across captions of different lengths or complexities?\n\nc. Did the authors conduct any experiments to test the stability of the metric with varying numbers of question-answer pairs?\n\n\n3.VDCSCORE: miss detailed information in the Elo Ranking. Could authors provide details on:\n\na. The specific dataset used for the Elo Ranking comparison, including its size and composition\n\nb. The criteria used for selecting this dataset for the comparison\n\nc. How this dataset relates to or differs from the VDC benchmark?\n\n\n4.VDCSCORE：some quality problems of caption are about the repetition or grammar errors. Does this metric could evaluate these types of quality?\n\na.How does VDCSCORE handle linguistic aspects of caption quality, such as repetition and grammatical errors?\n\nb.Were any specific measures incorporated into the metric to detect and penalize such issues?\n\nc.Could the authors provide examples or experiments demonstrating how the metric performs on captions with these types of linguistic problems compared to human evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwMMksE8i0", "reviewer": "Reviewer_gKDb", "summary": "This paper proposes AURORACAP, a video captioner based on a large multimodal model that utilizes a token merge strategy to support lengthy video sequences. The authors also introduce VDC, a video detailed captioning benchmark featuring over one thousand carefully annotated structured captions. Finally, the authors validate the effectiveness of their model through a divide-and-conquer approach across multiple benchmarks, including the newly introduced VDC and various metrics, including human Elo ranking.", "questions": "1. Why do the authors train their model on general tasks while naming it a captioner? It seems more reasonable to refer to it as a general model while emphasizing its captioning capabilities.\n2. Why does GPT-4V have the lowest CIDEr score on the Flickr benchmark? I believe the authors need to verify whether a fair comparison was conducted, including aspects such as the prompts used during evaluation. I do not consider this to be a reasonable outcome.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work introduces the application of a token merging strategy for video understanding, demonstrating through extensive experiments that it can utilize only 10% to 20% of the visual tokens compared to the original tokens generated by ViT, with only a marginal drop in performance across various benchmarks.\n2. The proposed VDC benchmark addresses dimensions that are lacking or insufficient in existing benchmarks, such as rich world knowledge, object attributes, camera movements, and, crucially, detailed and precise temporal descriptions of events.\n3. This paper is well-structured and includes intuitive figures that effectively illustrate the work.", "weaknesses": "1. There is a lack of novelty regarding the token merging strategy, as similar attempts have already been made in Chat-UniVi[1], which validated the effectiveness of this approach on video understanding benchmarks. Furthermore, the divide-and-conquer validation of the model essentially constructs Video QA based on video detail captioning, a method that has also been explored in [2].\n2. It would be beneficial to evaluate the token merging method's stability on longer video sequence benchmarks, such as Egoschema[3] and VideoMME[4].\n\n[1] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. https://arxiv.org/abs/2311.08046\n[2] Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward. https://arxiv.org/abs/2404.01258\n[3] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. https://egoschema.github.io\n[4] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. https://arxiv.org/pdf/2405.21075", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JjUcC49rez", "reviewer": "Reviewer_ZSxM", "summary": "The paper introduces AuroraCap, a video captioning model based on LMM, along with VDC, a detailed captioning benchmark, and VDCScore, an evaluation metric designed for long captions. AuroraCap handles video sequences with lower visual token numbers by the token merge strategy, outperforming state-of-the-art models, even including GPT-4V and Gemini-1.5 Pro.  VDC addresses the limitations of existing video captioning benchmarks by providing over one thousand videos with detailed, structured captions. since existing metrics are insufficient to evaluate detailed captioning performance, they develop a new LLM-assisted metric VDC-SCORE, that breaks down long captions into short question-answer pairs, ensuring better alignment with human judgments.", "questions": "1. **Training Strategy Clarity:** Was the token merging strategy used during AuroraCap’s training phase? How does varying the token merge ratio during training impact inference performance?\n2. **Performance in Video Question Answering:** What factors limit AuroraCap’s performance in video question answering compared to captioning tasks?\n3. **Token Merge Curve on VDC** As shown in Figure 6, why does retaining all visual tokens not lead to the highest VDCScore?\n\nI will consider raising my score if the authors can further address my concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Despite adopting a simple architecture design consistent with LLaVA, AuroraCap achieves remarkable results on captioning tasks with its three-stage training recipe.\n2. AuroraCap combines the Token Merging strategy, significantly reducing the number of visual tokens input and thereby improving inference efficiency.\n3. This paper also presents VDC, a detailed video description benchmark, which includes over one thousand structured descriptions, providing detailed annotations for videos from multiple perspectives.\n4. To accurately evaluate the model’s detailed captioning capabilities, the paper introduces VDCScore. This metric uses a divide-and-conquer strategy to break long captions into short QA pairs and employs LLM assistance for scoring. This approach avoids the challenges traditional metrics face with long captions and mitigates potential hallucination issues associated with using LLM evaluations.\n5. Compared to existing models, AuroraCap achieves excellent performance in both image captioning and video captioning tasks.\n6. The paper also explores the impact of the visual token kept ratio on model performance, demonstrating that with the token merging strategy, fewer visual tokens can still maintain high performance.", "weaknesses": "1. **Spatial Token Merging:** AuroraCap only performs spatial token merging within frames and does not implement temporal token merging between frames, which could further enhance inference efficiency.\n2. **Static Token Merging:** It seems that for different visual inputs, the number of visual tokens remains consistent by setting a visual token kept ratio. However, this ‘static’ token merging strategy is unreasonable for tokens with varying levels of informativeness.\n3. **Unclear Training Strategy:** It is not clear whether the token merging strategy was used during training. Additionally, the impact of using different token merge ratios during training on inference performance is not clear.\n4. **Performance in Video Question Answering:** Although AuroraCap demonstrates leading performance in captioning tasks, its advantages in video question answering tasks are not apparent, which is perplexing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)