CaReBench: A Fine-grained Benchmark for Video Captioning and Retrieval
Authors: Yifan Xu
Xinhao Li
Yichun Yang
Desen Meng
Rui Huang
Limin Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Model
Fine-grained Video Retrieval
Video Detailed Captioning
Primary area: datasets and benchmarks

Abstract: Video understanding, including video captioning and retrieval, is still a great challenge for video-language models (VLMs). The existing video retrieval and caption benchmarks only include short descriptions, limits their ability of detailed video understanding evaluation. To address this problem, we present CaReBench, a testing benchmark for fine-grained video Captioning and Retrieval with 1,000 high-quality pairs of videos and human-annotated detailed captions. Uniquely, it provides manually separated spatial annotations and temporal annotations for each video. Based on this design, we introduce two evaluation metrics, ReBias and CapST, specifically tailored for video retrieval and video captioning tasks, respectively. These metrics enable a comprehensive investigation into the spatial and temporal biases inherent in VLMs. In addition, to handle both video retrieval and video captioning tasks in a unified framework, we develop a simple baseline based on a Multimodal Language Model (MLLM). By implementing a two-stage Supervised Fine-Tuning (SFT), we fully unlock the potential of MLLM, enabling it not only to generate detailed video descriptions but also to extract video features. Surprisingly, experimental results demonstrate that, compared to the CLIP-based models designed for retrieval and the popular MLLMs skilled in video captioning, our baseline shows competitive performance in both fine-grained video retrieval and video detailed captioning.

Published as a conference paper at ICLR 2026
CAREBENCH: A FINE-GRAINED BENCHMARK FOR VIDEO CAPTIONING AND RETRIEVAL
Yifan Xu', Xinhao Li', Yichun Yang', Desen Meng', Rui Huang', Limin Wang!.2.
1 State Key Laboratory for Novel Software Technology, Nanjing Univerisity Shanghai AI Laboratory yi yifanxu@smail edu. en, Imwangenju. edu.
ABSTRACT
Video understanding, including video captioning and retrieval, is still a great chal- lenge for video- Sadoulla models (VLMs). The existing video retrieval and caption benchmarks only include short descriptions, limits their ability of detailed video understanding evaluation. To address this problem, we present CAREBENCH, a testing benchmark for fine-grained video Captioning and Retrieval with 1,000 high-quality pairs of videos and human-annotated detailed captions. Uniquely, it provides manually separated spatial annotations and temporal annotations for each video. Based on this design, we introduce two evaluation metrics, Re- Bias and CapST, specifically tailored for video retrieval and video captioning tasks, respectively. These metries enable a comprehensive investigation into the spatial and temporal biases inherent in VLMs. In addition, to handle both video retrieval and video captioning tasks in unified framework, we develop simple bascline based
Multimodal Language Model (MLLM), By imple- menting a two-stage Supervised Fine- Tineinuning (SFT), we fully unlock the poten- tial of MILM, enabling it not only to generate detailed video descriptions but also to extract video features. Surprisingly. experimental results demonstrate that, compared to the CLIP-based models designed for retrieval and the popular MLLMs skilled in video captioning, our baseline shows competitive performance in both fine-grained video retrieval and video detailed captioning. Project page: github.io
attps::/careee
INTRODUCTION
Video captioning (Wang et al. 2022 Xu al. 2023 Wang et al. 2024a Chai et al. 2024) and video retrieval (Radford et al. 2021; Luo et al. 2022 Ma et al. [022 Zhou et al. [024 Wang et al., 2024d, Zhu al.) 2024 Zhang et al.) 2024a) are two main tasks video- language understanding. Captioning requires perception and description of the main objects, events and actions in the video, while retrieval aims at finding the most relevant video/text based on the text/video query. These two tasks intuitively reflect the alignment and comprehension ability of Video-Language Models (VLMs), serving as critical evaluations of VLM capabilities.
However, existing retrieval and captioning benchmarks still struggle to evaluate fine-grained under- standing of VLM. Traditional benchmarks (Xu et al., 2016 Chen & Dolan 2011 Hendricks et al.) 2017) have short and rough annotations, assessing general and coarse coarse-grained video understanding of VLMS due to brief descriptions. Recent works (Zhang et al.) 2024a] Yang et al. 2024 Chai et al., 2024 use powerful VLMs like GPT-4o (OpenAI (023) for auto-annotation. which inevitably in- troduces hallucinations and potential biases. DREAM-IK (Wang et al., (024a)) has more accurate human annotations, yet it lacks hierarchical captions and truly comprehensive focus on both objects
and events.
In addition, designing effective metrics for video captioning also poses a challenge. Traditional n-gram metrics (Vedantam et al. 2015) are difficult to evaluate fine-grained captions (Wang et al. 2024a Chai et al., (2024). while LLM LLM-ba evaluations (e.g. AutoDQ (Wang et al. (024a)), lack comprehensive consideration of both static objects and dynamic actions.
"Correponding author: Limin Wang.

---

Published as a conference paper at ICLR 2026
Table 1: Statisties of retrieval and captioning benchmarks. Traditional benchmarks, namely MSR-VTT (Xu et al. 2016), MSVD (Chen & Dolan 2011), DiDeMo (Hendricks et al., (017)) and NetivityNet (Heilbron al, 2015) have very short captions. Detailed captioning benchmarks (Wang) et al. 2024a| CCai et al. (024)) have longer and detailed captions, but they are either annotated by GPT or fail to focus on both static objects and dynamic actions.
Benchmark
Sample Avg. Len. Avg. Words Annotator Hierarchical Anno. Static Object Dynamic Action
MSR-VTT al. 2016
1,000
15.01s
9,41
Human
DiDeMo (Hemarikks or 2017
1,037
53,946
29.11
Human
x
MSVD CCien Dolan
670
10.04s
7.01
Human
Activity (Neitbeun et al 2015
5,044
36.00s
13.48
Human
X
X
X
DREAM-I IK [Wang al.
1,000
8.9s
59.3
Human
VDC (Chai al 2024
1,000
28.18s
500.91
GPT
/
CAREBENCH
1,000
14.35s
227.95
Human
To address these issues, we present CAREBENCH, a fine-grained Benchmark for video Captioning and Retrieval. It contains 1,000 videos with human- annaattnd detailed captions. Unlike images, video understanding tasks require models to understand both static scenes and dynamic actions. So we apply a hierarchical annotation scheme with each annotation covering four aspects: an overall summary, static object descriptions, dynamic action descriptions, and misc descriptions (c.g., filming style, camera movement, etc.). Such design ensures each caption has sufficient details, challenging models to capture sine-graneed information. Furthermore, to evaluate models spatiotemporally. each caption is manually separated into spatial and temporal parts. Based this, construct ReBias and CT, two novel metrics for video retrieval and captioning, respectively. Due to our benchmark and metries design, this work brings the community some new insights about spatiotemporal biases of state-of-the-ar VLMs that other benchmarks may fail to reveal.
During the evaluation on both video retrieval and captioning tasks, realize that previous works often treat retrieval and captioning as separate tasks, leading to the development of specialized mod- els for each. Specifically, CLIP-based dual-encoder models have been well advanced for video retrieval, while Multimodal Large Language Models (MLLMs) have been tailored for video cap- tioning. However, we discover that the two tasks can be unified and formulated as mapping from the pixel space to a high-dimensional space: Ø IR x W xC - I (either vocabulary space RD, or embedding space R D.). ). This finding renders it feasible to address the gap between video retrieval and captioning.
Taking advantage of the unified architecture of MILMs, we develop CARE, simple and unified baseline for both detailed video captioning and fine- fine-grained video retrieval. Specifically, our method involves a two-stage supervised fine-tuning (SFT). This makes it possible to generate video captions and discriminate video contents using only model. The first stage aligns the model output to a fine-grained text space. by training the model using mixed LLaVA- Video-1788 178k (Zhang et al.] 2024cb and Tarsier (Wang al.) (024a) recaptioned data. In the second stage, text-only contrastive learning approach (Jiang et al. 2024bb is adopted to enable the MLLM to effectively perform cross- modal representations. As shown in Figure D our experiments indicate that, compared to CLIP- based retrieval models and MLLM captioning models, CARE achieves superior performance on CAREBENCH.
In summary, we make the following contributions:
(1) We introduce novel fine-grained benchmark named CAREBENCH. It designed for video retrieval and captioning, comprising 1,000 videos with high-quality that provide sufficient video details. Each video features hierarchical descriptions ensuring compre- hensive coverage, and manually split spatial/temporal captions. Based on this, we construct ReBias and CapST, two novel metrics designed for the video retrieval and captioning tasks, respectively. Such designs reveal new insights about spatiotemporal biases of VLMs that other benchmarks may ignore.
descriptions
(2) We present CARE, simple baseline for fine-grained video retrieval and captioning. By applying two-stage Supervised Fine- Tine-Tuniggaa (SFT), we enable CARE to not only generate detailed video de- scriptions but also to extract video features. Our experiments show that, compared to the CLIP-based models designed for retrieval and the popular MLLMs skilled in video captioning, our baseline has competitive performance in both fine-grained video retrieval and detailed video captioning.
2

---

Published as a conference paper at ICLR 2026
General Retrieval (Ava.
Retriee R@11)
Retroival
Capton (Avg.
100
Parforanca
80
61.5
63.3
67.6
78 78:077 07.6
60
47.1
47.146.6
46.3
52.1
40
33.1
33.9
23.8
29.9
25.426.8
33.4
20-
0.0
0.0
0.0
8/16
Long<LiP Netiooaal Modll
Int InternVide22 Retrivall Modil
LL Captaning Ridds
Owen2-V1
Calfe (Durrs Unifee Moda
Rociloval Rodel
Capeloning Modll
Figure 1: CAREBENCH performance of popular models and CARE. The results on MILMS are reported on their public version without contrastive training. The CLIP-based models have achieved excellent performance video retrieval tasks, but they lack the ability to describe videos. On the other hand, MLLMs can deseribe videos in detail, but their retrieval performance is very poor. In contrast, CARE not only shows outstanding retrieval performance but also has a strong capability to describe videos. Features are extracted from MLLMs using EOL prompt (Jiang et al. 2024b).
2 RELATED WORK
Video captioning aims describe videos using natural language. Traditional cap- tioning benchmarks, such as ActivityNet (Heilbron et al. (2015), MSVD (Chen & Dolan 2011), and MSR-VTT VTT (Xu et al. 2016). typically use single sentence to describe video, which is insufficient to convey the full visual contents. As a result, they can no longer effectively stress-test modern MLLMs, as these models can output semantically richer descriptions than reference captions. To address these issues, new benchmarks have been proposed. DREAM-IK 1K (Wang et al. (024a) anno- tates five categories of videos with rich action content and introduces a novel automatic evaluation method called AutoDQ, which assesses the accuracy and recall of actions and events in captions. Similarly, VDC (Chai et al. (2024) employs hierarchical prompting with GPT-4o for structured and detailed captions, followed by manual correction. However, lacks explicit focus human actions and motion. In this paper, we explore new fine-grained video captioning benchmark focusing not only on objects but also actions to comprehensively evaluate VLMs.
Video Caption,
Video Retrieval. Video retrieval aims to find the most relevant video/text based on the text/video
query. Traditional methods (Wang al. 2024d] Ma al., [022] Luo et al.) [022] Zhang et al.] 2024a Li et al.) 2023, Girdhar et al.) 2023) focus on using dual encoders based on CLIP (Radford al. 2021) to extract features. But most of them are limited by the 77-token context length in- herited from CLIP, hindering long-caption understanding (Zhou et al. (2024). While long-text and fine-grained video retrieval becomes important. Long-CLIP (Zhang et al. 2024a) addresses this problem by extending context to 248 tokens for long-text retrieval. But the benchmark used by it are annotated by LLMs, which may contain coarse coarse-graine -grained, uncertain and wrong descriptions. In this paper, further explore the model training and the benchmark design for fine-grained video retrieval.
Multimodal Large Language Model. Due great advancements in LLMs (Devlin et al. 2019
Brown et al. 2020 Wei et al., 2022) (Cowwhery et al.) 2023), their multimodal counterparts (MLLMs) (Li et al. 2025 Chen
2023 Yao al. [024]] Zhang et al. 2024bb Wang et al. 2024b) are receiving significant attention, particularly for their capability to perform various visual lasks using straightforward instructions. Recent works like VideoChat (Li et al. 2025), are trained on large-scale datasets such as (Wang et al. 2024c and demonstrate outstanding performance on multimodal benchmarks (Fu et 2l.]]2024] Li et 2024). But these models are restricted to generat- ing responses based solely on user instructions and lack the capability to represent videos, images, and text. In this paper, we construct a unified baseline for both video retrieval and video captioning.
Multimodal Embedding. CLIP (Radford et al.) 2021) I learns image and text representations by aligning them with contrastive learning. However, Mind the Gap (Liang et al. 2022) notes that different data modalities are embedded with gaps in their shared representation space. To address
this issue, recent works like VISTA (Zhou et al.) 2024) and E5-V (Jiang et al. 2024b) explore unified representation. They find that MLLMs provide a unified multimodal framework to unify cross-min modsl representations without gaps. We regard it as promising method and will further explore unified MI.LM representation video retrieval.
3

---

Published as a conference paper at ICLR 2026
-03 3.4%
200-
258-308 5.4%
20e-25g 7.7%
Frequency
15s-2da 20.1%
10a-15e 34.8%
150
200
250
300
250
450
Caption Longth (word)
(a) Video length distribution.
(b) Caption length distribution.
Figure 2: Statistics of CA REBENHH. In our benchmark, most videos range from 5-20 seconds and most captions fall between 150 and 300 words in length.
3 CAREBENCH: A FINE-GRAINED BENCHMARK
3.1 VIDEO COLLECTION
We manually select 1,000 videos from FineAction (Liu et al. 2022) with 10-20 videos in each subcategory. FineAction a video dataset for temporal action localization with 106 subcategories and 4 major categories: personal care, socializing & relaxing, sports & exercise, and household activities. Videos in each subcategory share similar scenes and actions, which poses a challenge to the models' ability understand and discriminate similar videos.
3.2 TWO-STAGE ANNOTATION PIPELINE
The annotation pipeline consists of two stages. In stage one, annotators describe videos in detail, covering four key aspects of each video. Subsequently, they are guided to separate the annotations into temporal and spatial descriptions. To ensure high quality and minimize bias, each video is independently captioned by two annotators and subsequently refined and merged by our experts. Refer to Figure and Appendix A for annotation pipelines, data examples and the case study.
3.2.1 STAGE-I: DETAILED ANNOTATION
In Stage-I, annotators provide detailed video descriptions limited to 150-300 words. Each descrip- tion can be divided into four parts: a general overview, an action description, a object description, and misc. description, outlined below:
General Overview provides a one-sentence summary of the entire video. For example, this video shows person slicing watermelon.
Object Description focuses on static objects with attributes like position, color, shape, and other visual details. It contains primary and secondary objects, background, their relative positions, inter- actions, and even watermarks.
Action Description captures the actions occurring in the video, detailing the event sequences (e.g., first..., (hen...) and providing specific details of each action (e.g., rotating the watermelon clockwise). It also includes the style of the actions (e.g., culting fruit quickly, climbing the tree clumsily).
Misc. Description about 2-4 sentences in length. It covers different aspects, such as the viewpoint (e.g., third-person perspective) and the overall type of the video (e.g., delightful and relaxing).
3.2.2 STAGE-II: SPATIO-TEMPORAL SEPARATION
Stage-Il refines the initial annotations by separating spatial and temporal elements. It removes ac- tion texts from object descriptions create pure spatial captions, and eliminates static references
4

---

Published as a conference paper at ICLR 2026
from action descriptions to form pure temporal captions. This design ensures precise evaluation of VLMs' spatiotemporal modeling capabilities by preventing interference between dynamic and static elements.
Spatial Description provides a comprchensive view, beginning with a general overview and then detailing main objects, secondary objects, and the backgrounds. It ensures that spatial descriptions can differentiate between similar videos within the same subcategory.
Temporal Description begins with general overview, then focuses actions and their order. Spatial-specifii details are excluded. It ensures temporal descriptions uniquely identify each video within its subcategory.
3.3 COMPARISON ON STATISTICS
The captions in CAREBENCH are human-annotated, providing detailed and comprehensive descrip- tions of the videos. Consequently, its statistics differ significantly from those of traditional bench- marks. As shown in Table 1 our benchmark is similar in size MSR-VTT (Xu
2016).
DiDeMo (Hendricks et al. 2017). 7). but the average number of words per caption is 24.2x x higher than that of MSR-VIN (Xu et 2 7.82x higher than DiDeMo (Hendricks al. 2017). and 32.5x
higher than MSVD (Chen & Dolan 2011). The chart in Figure shows the video length distribu- tion of CAREBENCH. Since excessively long video durations significantly increase the difficulty for annotators to provide detailed descriptions, our benchmark focuses on videos ranging from 5 to 20 seconds length, with over 80% of the videos falling within this range. Only 5.8% are shorter than 5s extends beyond 30s. Figure 2b demonsttats how the caption length distributes. Most captions in CAREBENCH contain between 175 and 275 words.
3.4 METRICS DESIGN
CAREBENCH contains manually annotated highspatial-tempore captions. This design enables us to identify biases in the model's understanding of static objects and dynamic actions by analyzing the imbalance in spatio-temporal performance across video retrieval and captioning tasks. To quan- tify the spatio-temporal perfomance and bias, introduce two novel metrics for video retrieval and video captioning, respectively: ReBias and CapST. These two metrics comprehensively reveal VLMs' performance and inherent biases by separately evaluating spatial tasks and temporal tasks.
3.4.1 REBIAS
Evaluating spatial and temporal captions separately reveals the model's performance across both di- mensions. We introduce ReBias, a metric that measures spatiotemporal Retrieval Bias. It measures a model's bias towards its focus on static objects versus dynamic actions by showing how far the temporal-to-ppaate recall ratio deviates from (lower is better). It can be formulated as follows:
Reemporat
B
Rspatiat
where Reemporal and Rapatial denotes the average recall on temporal/spatial retrieval, respectively.
3.4.2 CAPST
Existing video captioning metrics face limitations: traditional n-gram methods, like CIDEr (Vedan- tam al., 2015). struggle with long captions (Chai al.) 2024 Wang al.) (2024)), while LLM- based metrics (Chai et al.] 22244) Wang et al. (024a) lack comprehensivenese in evaluating both ob- jects and actions. For example, Veore (Chai al. 2024) evaluates predictions by querying both ground truth and prediction details to compute recall, but it ignores precision which is critical for as- sessing hallucinations; AutoDQ (Wang et al., (024a) only focuses on evaluating actions/events and neglects objects. To overcome these issues, we propose CapST, video Captioning metric jointly evaluating Spatial objects and Temporal events. Similar to Wang et al. (2024a). a powerful LLM extracts events from temporal captions and objects from spatial captions and computes the Natural Language Inference (NLI) relationship between the ground truth Dyt and the predictions D prred- Specifically, we compute the recall and precision score of sample according to Equation and and
5

---

Published as a conference paper at ICLR 2026
Stage-l: Fine-Greined Caption Caption
Stage-ll: Retrieval Adaptation
Scace ShR
Vocabulryy Space ji
Enteedingg Space 2
0.4766
drcgent mediiloo cound ritting
Large Language Model
Large Language Model
Vision Encoder
Discribe the video in detafl:
The singers cover songs from the past well and the musiciens good at improvising.
one words
Video
Prompt
Prompt
Figure 3: Training recipe of CARE. Stage-I aligns CARE outputs to a fine-grained text space for detailed video descriptions, while Stage-ll contrastively trains CARE to extract features, shifting the output space from vocabulary (IR P.) to embedding space PR))
report average recall and precision of all samples in benchmark. More details about quantitative and human-ale aligged validation on different metries can seen in Appendix C
R N(Dgt
entail
Eprea)
Expred)
P N(D pred
entail
Egt)
N(Egt)
where Epred and Egz denote elements (either objects or events) extracted from predictions and ground truth captions, respectively. (Epree)) and (Egtt) is the number of elements extracted from D prod and Dgt. respectively. (Dgt gt Dyt, and N
entuil, Eyrea) refers to the number of Epred entailed by Ege) means the number of Eg entailed by Dpred-
entail
ppreed pred
Specially, when multiple attributes are combined in single description (c.g.. (an elderly man wear- ing glasses and a blue suif"), NLI evaluation tends penalize partially matching predictions, even when those predictions correctly identify some valid characteristics. To address this issue, we in- struct the LLM to split attributes during extraction. For instance, the aforementioned description would be divided into "an elderly man wearing glasses" and "an elderly man wearing blue suit." This design allows more precise evaluation of the model's performance describe objects with multiple attributes.
"an
CARE: A UNIFIED VIDEO MODEL
Previous works treat video retrieval and captioning as separate tasks, fostering specialized models like CLIP-based dual-encoders for retrieval and MLLMs for captioning. However, we find that these tasks can unified into a single framework, formulated
mapping from the pixel space to digh-dimensional space: Φ R xx**xxW'>> XxXxC + E (either vocabulary space IR or embedding space RPPc) To bridge this gap, we introduce CARE, a unified baseline built on Qwen2-VI. (Wang et al. 2024b), trained via two-stage progressive SFT to achieve both robust video captioning and strong video representation. The training pipeline is shown in Figure]
4.1 STAGE-I: FINE-GRAINED CAPTION ADAPTATION
MLLMSs excel in general video understanding but often miss key video details. To align the model with ine-grained video understanding and provide a robust backbone for Stage-II, we train CARE with high-quality video-caption pairs. Specifically, we set finetuning prompt to "Descr ibe the video in detail.. and train our model using video-text pairs from Tarsier Recap (Wang al.) 2024a), emphasizing action-rice descriptions, and LLaVA- Video- 178k (Zhang et al.] (024c). focus- ing on short videos with details. With fine-gna grained caption adaptation, the model output is aligned with fine- mne-griieed text space and can better focus on detailed actions and objects when describing videos.

---

Published as a conference paper at ICLR 2026
Table 2: Video caption performance of popular models on CAREBENCH (Events).
We report
F1 Fl/Recall/Precision for each category. # Params denotes the number of LLM parameters.
CAREBENCH Caption (Events)
Model
Params
Personal Care Social & Relax Sports Excercise
Household
Overall
GPT-4o mini
32.9/24.9/48.
34.7/26.2/55
44.3/38.0/33u
34,2/26 9146. 16.8/2
LLaVA NeXT Video ZZang et al. 2024bb
7B
27.5/20. 1
25.0/17.4444.1
29.4/21.144..
24.3/16.2/48.
26.6/18. 7/45,
InternVL2 Chen
22.2/18.4/233
23.0/17.9/332
27.9/23.4/34.5
18.4/14.7/2488
23 3/3.8/330.7
InteraVL2.5
Clea et 2024
7B
22.0/15. 1/41.1
24.0/16.8/4.6
34.0/26.1488
22.3/15.
26.0/18.6/46
InternVL2.55
Clic
24.6/16.7/46.
25,9/18. 3/444
36.0/27.8/51 8/51.
24.9/17.5/43
28 28.2/20.//46.
MiniCPM-V
26
Yai (a0
m
2024
7B
30.2/21 M552.0
25.9/18.6/48.8
38. 1/29.///333
28.
28.5/20.0/49.5
31, 1/22./22//511
Tarsier (Wang
20240
25.4/16.5/55.0
2615/18 0/50.4
32.0/22 8/53.3
22.8/15. 3/44477
27.1/18.//51.1
Qwen2-Vis VL. Waing er 2024h
7B
4/23 9/34,9
27.5/20.8/40.
33 6/.3//6.6/43.6
2/35. 28. 28.8/22.9/3900
Qwen2-VL
6/22. 1/45.0
1/20 6/44,2
37.3/28.5/53.
26.4/18 6/45 30.5/22.644.,
Qwen2.5 VL.
Palt 2l. 2125
7B
30.0/21 2/51//
29. .3/8..9
36. 1/2.000080088
27.2/19:4/45.
31 31.1/22.749..2
32.4/24.0049.8 4/.9025.//5..8
42.8/33.7/58.5
31.5/24.4/44.7
35.3/26.9/51.3
CARE
7B
34.4/25.6/52.6
32.2/24.0/48.88
42 3/.3/33.1/88.1
30.9/23.4/4.. /5..//2.6/51.4
Table 3: Video caption performance of popular models on CAREBENCH (Objects). We report F1/Recall/Precision for each category. # Params denotes the number of LLM parameters.
CAREBENCH Caption (Objects)
Params
Personal Care Social & Relax Sports & Excercise
Household
Overall
GPT-4o mini
29.2/21.2/47.22
34.2/26.5/48.
36.1V27.4/526
35. 1/.7/62.648.2
31.8/25.8499.
LLaVA NeXT
Video (Zhnng al. 2024b
7B
21.7/15.5/36.2
14.1/33.3/39.
26.8/19.66 6/423 6/42.3
26.3/19.5/40.
24.7/17.9/9..8
InteraVL2
Chen al 2025
7B
20.4/15.1/31.6
23.1/17.3/344
3433.7
22.7/17.1/35.8
22.9/17.1/34.9
IntemVL2.
Can 2024
7B
26.4/20.4/37.2
28.4/22.7/37.9
31.6/26.4/39.4
29.6/24.4/37.7
29.1/23.5/38.2
1
Lgg 2024
72B
28.7/22.444000
28.6/23.3/57.3
34.0/28.2442 2/42.7
30.8/25.7/38.5
30.5/24.839.5
MiniCPM-V
2,0 Yao
2024
7B
28. 1/.73/9
0/4888
72.9/33.7/49.3
2/2222..//2..
9/550/
Tarsier Wang dr. 20246
30.0/22.224
30.022
33.A/24.9/500
31.2/23.9/45.1
31.1/23.4/6..
Qwen2-VL Walne 3l.
2024b
7B
23.7/15.8447.7
23.0/15.144 1/4.8
24.9/16 2/5/1.////3.1
24.8/16 8/47.2
24.0/15.9/99.
Qwen2-VL. Wang
20240
72B
24.5/16.
22.5/14.7/47.8
8/56/668/56.3
26.5/17
4/55,7
24.2/15.8/511
Qwen2.5..V VI.
2025
7B
32.8/25.34666
32.7/25.9/44.2
34.8/27.6/7.3
14.0/27.7744.1
33.5/26.5/45.5
CARE
7B
1/25.6/555.3
31.3//2.2/53..1
33,2/23. 2/582
33.6/23.8/5 8/57.1 32.4/22.9/55.7
CARE
7B
30.5/21.155..2 31/5/
31.8/1.3/6.66
32.6/23.0/5.88
31.7/2188/5..8
4.2 STAGE-II: RETRIEVAL ADAPTATION
After Stage-I, CARE achieves precise alignment between pixel space and fine-grained text space. To shift the model output from the vocabulary space i. to the embedding space RDD we use similar method as Jiang et al. 2202bb at, employing an Explicit One-word Limitation (EOL) prompt to extract embeddings from CARE. Specifically, there two steps: (1) given an EOL prompt: "<sent> Summary of the above sentence in one word:". the model is instructed summarize the sentence 8; in the next token; (2)
the hidden states in the next token generation step as the final embeddings ft- Then, we train the model on an NIJ dataset (Gao et al.) 2021 where each sample contains a sentence st, its positive s. and its hard negative s, Since there are no video inputs during Stage-II, we freeze the vision encoder and train the LLM only. Our training objective is given as:
1/7
C = -llog
(4)
Σ/=1 (co( caa(te)))) M/)
where /ú, f.+ Ji denote the embeddings of the sentence its positive s and its hard negative 8,, respectively. cos(-) is the cosine similarity function. J is the temperature hyperparameter.
EXPERIMENTS
In this section, we present the experiments on CAREBENCH. Section 5.1 shows the experiment settings. Section 5.2 and 5.3 analyze the results video captioning and retrieval. In section 5.5 we conduct ablations to show the effectiveness of our methods. Additional experiments on other benchmarks and smaller MI.LMs are included in Appendix
7

---

Published as a conference paper at ICLR 2026
Table Video retrieval performance of some state-of-the-arts methods on CAREBENCH. All the results are reported in zero-shot setting.
CAREBENCH General Retrieval
Model
Text-to-Video
Video-to- Text R@5
@
R@5
R@ 10
R@D
R@
CLIP-based Models
CLIP B/16 Radford et al. 2021
45.7
79.6
89.1
48,4
82.4
90.8
CLIP L/14 (Radford et al.
51.2
83.4
90.6
54.7
86.9
93.6
LanguageBmn (Zhu el al. 2024
91.0
95.0
Long-CLin B/14 Zhang ct al
2024a
59.2
85.3
92.1
55.8
84.7
92.9
Long-CLIP L/14 (Zhang cl al
2024a
62.7
88.8
95.7
60.3
88.8
94.9
Intern Video22tget
IB Wang ei al.
2024d
72.5
93.7
97.3
69.5
94.6
97.8
MLLMs
LLaVA NeXT Video 7B (Zhang al. 2024b
22.4
51.5
65.3
25.2
54.4
67.7
MiniCPM-V 2.6 Yao al. 2024
8.2
26.9
38.4
16.7
39.9
55.8
In2ernVL22 Chen al. 2023
34.6
67.1
80.2
35.1
68.5
82.0
Tarsier 7B (Wang et al. 2024a
83.5
32.3
84.4
Qwen2-VI. 7B (Wang et al, 2024b
30.9
64.7
79.1
32.9
69.6
82.7
Contrastively trained MILMs
LLaVA NV 7B (Zhang el al. 2024b
66.9
89.4
96.0
62.7
89.2
95.4
MiniCPM- 2.6 Yao et al. 2024
71.0
92.2
97.0
69.3
92.8
97.1
VL2 8B (Chen al. 2023
72.1
92.6
96.8
73.6
93.4
97.4
Tarsier 7B (Wang
2024a
71.0
93.8
97.8
70.6
94.2
98.0
Qwen2-VI. 7B (Wang
2024b
76.6
95.3
98.7
77.4
95.6
98.7
Qwen2.5-VL 7B [Ba et al.
2025
75.5
95.8
98.7
CARE
77.0
95.6
98.7
79.0
96.8
99.1
Table 5: Spatiotemporal retrieval results of video retrieval on CAREBENCH. LLaVA NV 7B is short for LLaVA NeXT Video 7B. We train all the MILMMs contrastively NLII dataset enable them to generate video embeddings. All the results are reported zero-shot setting.
CAREBENCH Spatial Retrieval
CAREBENCH Temporal Retrieval
Model
Text-to-Video
Video-to-Text
Text-toViddo
Video-to-Text
ReBias'
R@1 1 R@s RØ 10 R@1 R@5 R@10 R@1 RéS R@10 R&1 R@5 R 10
CLIP-based Models
RRdffrrd
79.0
47.6
CLIP 1/14 Khotord er mi 2021
49.0
81.9
91.4
55.4
85.6
93.0
33.5
70.3
84.0
39.7
76.2
87.9
16.52
LanguageBond Bund iZh al.
2024
64.7
90.8
61.0
87.2
94.5
39.8
77.3
90.5
42.2
77.6
91.7
18.10
Long-Cli CLIP 8/14
2024a
62.5
86.0
92.7
53.8
84.1
92.7
32.0
65.4
79.3
29.7
67.3
84.1
31.88
Long. CLIP 1/14
ZZhang el al.
20:40
65.6
90.9
96.0
61.0
88.3
94.4
33.2
68.8
81.6
34.5
71.9
86.6
31.77
Intern Video2
Wang
2024d
94.2
62.7
80.8
82.5
92.5
MLLMs
LLaVA NV 7B ZZang et 2024b
34.1
63.1
76.0
31.1
63.7
75.1
18.6
48.1
62.4
20.7
47.1
62.4
32.32
MiniCPM-V 2.6
25.2
13.3
11.8
47.4
64.4
24.41
Intern 8B Chen a 2044
40.4
72.9
83.8
40.3
73.0
85.7
29.3
62.5
77.4
27.1
59.8
75.9
19.31
7B Wang
75.0
26.8
64.6
83.5
32.3
84.4
Qwen2 VL. 7B Wang 2024b
28.1
61.3
76.1
31.6
65.6
80.4
24.3
61.5
78.4
26.4
59.2
76.1
5.28
Contrastively trained MLLMs
LLaVA Zhang 2024b
92.0
65.0
90.0
43.3
88.9
88.7
22.69
MiniCPM-X 2.6 10TTa al 2024
71.7
93.6
98.0
67.6
92.3
97.7
50.5
82.9
92.1
46.1
80.9
93.3
16.89
8B Chen
74.3
Tasserr 7B Wang et 20:4
70.2
94.0
98.2
67.4
93.5
97.4
50.1
84.1
92.8
50.0
84.7
94.9
14.04
Qwen2.V1. Wing cr
95.0
98.1
Qwen2.5-VL [Bi et
1025
77.0
95.6
98.8
72.6
94.8
97.8
51.4
84,1
93.9
50.5
83.1
93.5
17.56
CARR
76.8
96.3
98.7
78.1
95.8
99.3
50.7
85.3
94.4
53.4
86.3
94.0
17:53
Intern Vnterniideo2tteed is tested without match header for faimess.
5.1 SETTINGS
In Stage-I, train Qwen2-VL VL Wang al. 2024b) for about 400 GPU hours with learning rate of 2e-5, batch size of 64, max pixel 460,800, and 16 input frames. For Stage-II, CARE is initialized from Stage-T and trained NLII dataset with the video backbone frozen. Due text-only
8

---

Published as a conference paper at ICLR 2026
contrastive learning, Stage-Il -I only requires 24 GPU hours. We set epoch, batch size, and warmup ratio to 2, 768, and 0.2, respectively, and fully fine-tune CtREstag--1 with learning rate of 2e-4.
5.2 VIDEO CAPTIONING
In Table 2 and Table 3 present quantitative comparison of the video captioning task on CAREBENCH between CARE and popular VLMs. Results are reported in zero-shot setting fol- lowing our CapST metric. We use DeepSeek-V3 (DeepSeek-Al et al., (024) to serve as the LLM judge. The number of input frames are set to 32. The default prompt is "Descri be the vi detail unless the official research (Zhang et al. 2024b) recommends a specific one.
As illustrated in T 3] our model has demonstrated superior performance across all the categories, surpassing all existing open-source models currently available. Considering the disparity between the models' parameters and their performance, even the most powerful MLLM, Qwen2-VI L 72B, exhibits a significant performance gap when compared to our 7B CARE. This indicates that all current models still lack the ability to provide highly detailed, comprehensive, and fine- mne-graiedd video descriptions. Additionally, it can be observed that whether the model has undergone stage training does not affect its captioning performance. These promising results demonstrate that even small- scale 7B model is capable of understanding the details within videos, including dynamic actions and static object elements and can have outstanding captioning and retrieval abilities simultaneously.
5.3 VIDEO RETRIEVAL
We compare CLIIP-baeed models, contrastively trained MILMs and our CARE on CAREBENCH, following the setting of 32 input frames. Table 4 and Table present the general retrieval perfor- mance and spatiotemporal retrieval performance annotations, while spatial and temporal retrieval leverage spatial captions and temporal captions from second-stage. All tasks employ Recall at Rank K (R @ K, higher is better) in zero-shot set- ting. The following observations can be concluded according to our analysis:
CAREBENCH. General retrieval uses first-stage
MLLMs perform better than CLiP-based models video retrieval. CLIP-based models have long dominated retrieval performance benchmarks. However, as demonstrated in Table 4 MLLMs trained with contrastive learning exhibit significantly enhanced retrieval capabilities, surpassing their predecessors in performance. Our CARE yields the most favorable results, surpassing CLIP, Long- CLIP, LanguageBind, Intern InternVide22 Video2 and all the other MLLMs.
VLMs have inherent biases in their spatiotemporal understanding and excel at leveraging spa- tial shorteuts for video understanding. According to Table 5 all the models exhibit imbalance in spatiotemporal understanding, with spatial retrieval performance significantly outperforming tem- poral retrieval performance. When we switch from general retrieval to spatial retrieval. the perfor- mance drops for VLMs is small (Arg. R@ /: Qwen2- VI. -0.20, Tarsier -2.00, MiniCPM-V 2.6 -0.50). In contrast, the drop is substantial for Temporal Retrieval (Aug. R@ I: Qwen2-VL. VI -24.70, Tarsier 200.75, MiniCPM-V 2.6 -21.85). Even when most action-related cues are removed from captions, VLMs still maintain comparable performance. This indicates that these VLMs rely on scene cues as a shorteut rather than using the detailed action information when they describe videos. Such bias highlights the need for improved methods to enhance temporal understanding capabili- ties in video understanding tasks.
5.4 VIDEO QA
To demonstrate CARE'S generalization capability on out-of-domain tasks, we conduct video QA evaluation on MVBench (Li et al. 2024) and TVBench (Cores et al.] 2024). Results for models other than CARE are from the official reports of benchmarks. The results in Table [ show that CARE demonstrates strong generalization on general video understanding benchmarks.
5.5 ABLATION STUDY
In this section, conduct experiments further investigate the effect of our proposed two- stage SFT. Using the same setting as mentioned in Section [3. and building upon the Qwen2-Ve VL model (Wang et al. 2024b), we perform quantitative analysis evaluate the impact of different

---

Published as a conference paper at ICLR 2026
Table 6: Performance of Video QA on MVBench (Li et 2024) and TVBench (Cores et al. 2024
Results for models other than CARE are from the official reports of benchmarks.
Models
MVBench (Li et al. 2024)
TVBench (Cores et al. 2024)
GPT-4o
47.8
35.8
VideoChat2
51.0
33.0
PLLaVA-7B
46,6
34.2
Gemini 1.5 Pro
60.5
46.5
CARE
60.4
50.1
Table 7: Effect of the two-stage training. Four model settings are included: the baseline, CARE with fine- ine-grained caption adaptation only, CARE with retrieval adaptation only, and CARE with full two-stage SFT. The evaluation metrics include Avg. R@ which denotes the average text-to- video and video-to-text @@ on CAREBENCH General Retrieval, and Avg. F1, which represents the average action/object Fl on CAREBENCH. Unified Score the average of R R@) and FI.
Retrieval Avg. R@
Caption Avg. Fl
Overall Unified Score
Setting
Baseline
25.6
26.8
26.2
+ -Fine-Grree Grained Caption Adaptation
17.6(-8.0)
33.8(+7.0)
25.7(-0.5)
+Retrieval Adaptation
77.0(+51.4)
28.2(+1.4)
52.6(+26.4)
Frne-Grained Caption Adaptation & Retrieval Adaptation
78.0(452.4)
33.4(+6.6)
55.7(+29:55
stages on the model's performance in video captioning and retrieval tasks, as shown in Table 7 Our baseline model, Qwen2-VL (Wang al. 2024b), shows strong captioning skills (Avg. Fl 26.8) but struggles with retrieval tasks (Avg. R@ 25.6) without retrieval adaptation. Adding fine- fine-graine graindd slight cost to retrieval performance (Avg. R@ 1 -8.0). On the other hand, using only retrieval adaptation gives the model excellent retrieval capabilities (Avg. R +51.4), which a big improvement over the baseline. After both training stages, our model not only excels in detailed video description but also achieves top-level retrieval performance. Interestingly, we have uncovered evidence that video retrieval and captioning tasks can mutually enhance each other: retrieval adaptation improves the baseline's s video captioning performance by +1.4 (Avg. F1 26.8 + 28.2), and the high-quality fine- tuning of fine- -he-ggained caption adaptation further boosts the retrieval adapted model by +1 (Avg.
caption adaptation greatly improves the model's captioning ability (Avg. F1 +7.0)
R@I 77.0 + 78.0).
6 CONCLUSION
In this work, we present CAREBENCH, a fine-grained benchmark for video captioning and retrieval, featuring 1,000 videos with high-quality human-annotated descriptions. Fach caption structured hicrarchically to cover four key aspects: overall summary, static object descriptions, dynamic action descriptions, and miscellancous details such filming styles. We also propose ReBias and CapST, two novel metrics for assessing retrieval and captioning performance. Additionally, we develop CARE, unified baseline for both tasks, leveraging two-stage supervised fine-tuning approach to generate detailed captions and extract video features. Experiments show that CARE outperforms specialized models in both fine-grained retrieval and captioning. Our work highlights the potential of unifying video captioning and retrieval tasks under single framework, challenging the traditional methods. However, our model doesn't 't address problems about VLMs' spatiotemporal bias. Look ahead, future research could explore further integration of both tasks and try to develop more balanced model.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by the Basic Research Program of Jiangsu (No. BK20250009) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
REPRODUCIBILITY STATEMENT
We provide comprehensive materials to reproduce our results. Model and training details (including loss functions, hyperparameters, and training settings) are in Section 4 and 5.1 Dataset sources,
licenses, and annotation steps are documented in Section 3 and Appendix
ETHICAL STATEMENT
(1) Human annotators. We pay human annotators above the legally mandated minimum wage in accordance with the laws where the research is conducted.
(2) Biases in benchmark annotations. The authors are aware of the potential for bias in the anno- tations of our benchmark. These annotations may inadvertently reflect the annotators' perspectives and biases. We have tried to minimize the bias during the expert refinement and each annotation is cross-checked by two human experts.
REFERENCES
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaobai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar Xix::2502.11925 2025.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Wenhao Chai, Enxin Song, Yilun Du Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng- Neng Hwang, Saining Xie, and Christopher D. Manning, Auroracap: Efficient, performant video detailed captioning and a new benchmark. CoRR, abs/2410.03051, 2024.
David L. Chen and William B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, pp. 190-200. The Association Computer Linguistics, 2011.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- lisual-linguistic tasks. CoRR, Abs/2312.14233. 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin

---

Published as a conference paper at ICLR 2026
Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Samkaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240: 113, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arxiv preprint arXiv:2507.0626. 2025.
Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, and Yuki M. Asano. Tvbench: Redesigning video-language evaluation, 2024.
Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang. Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URL https:
1/shareggt4o..ithheee github.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li. Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang. Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen. R. L. Jin, Ruiqi Ge. Ruisong Zhang. Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu. Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L.. Xiao, and Wangding Zeng. Deepseek-v3 technical report. CORR, abs/2412.19437 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pp. 4171-4186. As- sociation for Computational Linguistics, 2019.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. CoRR, abs/2405.210055. 2024.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP (1), pp.
Association for Computational Linguistics, 2021.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In CVPR, pp. 15180- 15190. IEEE, 2023.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pp. 961-970. IEEE Computer Society, 2015.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Rus- sell. Localizing moments in video with natural language. In ICCV, pp. 5804-5813, IEEE Com- puter Society, 2017.
12

---

Published as a conference paper at ICLR 2026
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. In EMNLP (Findings), pp. 3182-3 3196. Association for Computational Linguistics, 2024a.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: universal embeddings with multimodal large language models. CoRR, abs/2407.12580 2024b.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, pp. 19891-19903, IEEE, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- video under- standing benchmark. In CVPR, pp. 22195-2112 22206. IEEE, 2024.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: chat-centric video understanding. Sci. China Inf. Sci., 68(10), 2025.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022.
Yi Liu, Limin Wang, Yali Wang. Xiao Ma, and Yu Qiao. Fineaction: A fine one-grained video dataset for temporal action localization. IEEE Trans. Image Process., 31:6937-6950, 2022.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of CLIP for end end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-CLIP: end-to- end multi- graite contrastive learning for video-text retrieval. ACM Multimedia, pp. 638 647. ACM, 2022.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus based image description evaluation. In CVPR, pp. 4566-4575. IEEE Computer Society, 2015.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Trans. Mach. Learn. Res., 2022, 2022.
Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models. CoRR, abs/2407.0066 .0634, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception the world at any resolution. CoRR, abs/2409.12199 12191, 2024b.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. ICLR. OpenRe- view.net, 2024c.
13

---

Published as a conference paper at ICLR 2026
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He. Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang. Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Internvide02: Scaling foundation models for multimodal video understanding. In ECCV (85), volume 15143 of Lecture Notes in Computer Science, pp- 396-416. Springer, 2024d.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In ICLR. OpenReview.net, 2022.
Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang. Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jingren Zhou. mplug-2: 2: A modularized multi modal foundation model across text, image and video. In ICML, volume 202 of Proceedings of Machine Learning Research, pp.
PMLR, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging IEEE Computer Society, 2016.
video and language. In CVPR. pp.
Dongjie Yang. Suyuan Huang. Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. Advances Neural Information Processing Systems, 37:57240-57266 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li. Zhiyuan Liu, and Maosong Sun. Minicpm-v: A GPT-4V level MILM on your phone. CoRR, abs/2408.01 800, 2024.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability CLIP. In ECCV (51), volume 15109 of Lecture Notes in Computer Science, pp. 310-325. Springer, 2024a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruc- tion tuning with synthetic data, 2024c. URL http:::/aaxi...oooottt wttp://www....a/////...... 02713
Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. VISTA: visualized text em- bedding for universal multi-modal retrieval. In ACI. (1), pp. 3185-3200. Association for Compu- tational Linguistics, 2024.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, Caiwan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending vadeo-language pretraining to n-modality by aaguage-bassed semantic alignment. In ICLR. OpenReview.net, 2024.
14

---

Published as a conference paper at ICLR 2026
Traditional Coarse-Fine Caption
CaReBench Caption
Cheese being sliced.
Summary This video showcases the sparring process between two individuels using samurai swords. e Spatial Description The person on the loft is dressed in black protective helmet and gloves, gripping attire, sword his right hand, sporting black and boots. The individual the rightt black and white. The setting is indoors, featuring a black floor and light beige walls, One side of the wall has a wooden door, while the ather side displays hocks adomed with various styles of be
boots. similar autfif, but boots
swards and other equipment. few spectators
Temppral Description During their exchange, the sift-iide con- testant takes the initistive to attack. using his sword to deflect the right-side contestant's blade from below, then switty transi- tioning to defensive block. After successfully parrying, he attempts to strike at the chest first-person perspective person view, reenacting the aforementioned actions. transitions back to the first-person persgective for the second attack where the swords clash back and forth several times. The right-side contestant then moves forward for strike, but the left-side contestant dodges, elegantly sliding his sword from right right-side contestant, although misses
GPT-4o Generated Caption
sequence of frames showcasing various stationary ob jects placed on a surface. The frame composition remains largely consistent throughout the video, festuring white water bottle equip- ped with black cap and carabiner, a fork laid on purple cloth, knit green object, lattel-ptteerred fabric, and a blue mobile phone in case. The background displeys well with noticeable peeling paint, indicating slight viear and tear. At one point. the camera captures an angled view of calling light colorca hortzontal surface before returning to the original scene. The video concludes focusing back the stationary objjets. The transitions smooth with minimal move- maintaining stable frame structure except for brief diversions.
of the right-side contestant.
The video then shifts
third-
across the the The video altemates between first person person perspectives, vividly illustrating the viewpoints of both the contestanis during their sparring and the spectators watching the action unfold.
Misc (Filming Style etc.) Overall the entire video can di- vided into segments, effectively showcasing the dynamic nature their practico sessions.
Coerso-grained Doseription
Unsertain Desoription
Wronn Desemption
Comparision of captions between MSR-VTT(Xu et al., 2016), GPT-4o generated Cata(Cui et al.] 2024) and CAREBENCH. The caption in the upper left corner is from MSR- XT et al.) 2016). It only contains short-text coarse descriptions. The annotation located in the lower left corner is generated by GPT-4o sourced from ShareGPT-4o(Cui et al., 2024). It has some coarse- granse-,, uncertain and wrong descriptions. The fine- fineggrained caption on the right is selected from CAREBENCH and is created by our human annotator following the pipeline. The green sentences are fine-grained descriptions and the brown words show the temporal sequences in the video.
Figure 4:
A CASE STUDY
Benchmarks like MSRVTT (u al. 2016) rely on brief short captions. As shown in Figure 4 the MSRVTT caption in the upper-left corner overlooks key details, such as the contents of the kitchen and the attire of the man. Captions annotated by LLMs may have coarse-grained, uncertain and wrong descriptions. As shown in Figure 4 GPT-4o erroneously identifies the slipper beneath the phone as a phone case and describes the camera's violent shaking as "minimal movement." The fine-grained caption on the right is selected from CA CAREBENCH and is created by human. The green sentences are fine- traingg descriptions and the brown words show the action sequences in the video. For more sample of CAREBENCH, see the end of the appendix.
B ADDITIONAL EXPERIMENTS
B. EXPERIMENTS ON TRADITIONAL BENCHMARKS
We compare CLIP-based models, MLLMs, and CARE traditional retrieval benchmarks. All the experiments follow the setting of 32 input frames. Table and Table 9 present the retrieval performance of all the models on MSR-VTT (Xu et al. 2016). MSVD (Chen & Dolan 2011 and
DiDeMo (Hendricks et al. 2017]. All the retrieval results are reported in zero-shot setting. Table I0
illustrates the captioning performance of popular models DREAM-1K (Wang et al. 2024a)
15

---

Published as a conference paper at ICLR 2026
Table 8: Video retrieval performance on MSR-VTT (Xu al. 2016) and MSVD
CCen &
Dolan 2011).
MSR-VTT (Xu et al. 2016
MSVD (Chen & Dolan 201
Model
Text-to- Text-to-Vidoo
Video-to-Ter Text
Text-to- Text-to-Video
Video-to- Text
R@1 R@5 R@10 RØ1 R@5 R@10
R@1
Rãs R@10 R@1 R@s R@10
CLIPP-based Models
CLIP B/16 (Radford et al.
2021
33.8
56.1
66.6
30.5
53.8
65.5
37.0
64.2
74.1
60.5
79.9
87.5
CLIP (Kadlord
66.2
85.5
91.8
LanguageBind Zhi al.
M124
42.1
65.9
75.5
40.1
65.4
73.9
50.0
77.7
85.6
75.1
90
94.2
Long-CLIP B/1
Zhang al
2024a
38.7
62.3
57.7
68.2
40.4
77.7
63.4
81.6
87.8
Long.CT.PP CI 1/14 Zhang
2014n
40.9
65.5
74.6
36.2
62.2
71.5
46.5
73.5
82.0
69.3
86.0
90.3
ViteraVideo22aagr IB Wang el al.
2024d
44.2
70.1
78.1
40.5
66.9
76.3
53.0
79.1
87.2
74.6
88.5
93.4
Contrastively Trained MILMS
NeXT
(Zang et
2024b
47.3
74.3
InterVV22 8B Chen 2024
44.6
69.3
77.4
40.8
66.6
76.5
47.7
75.9
83.9
64.2
81.3
87.2
MiniCPM-V ea u. 2024
44.7
69.7
77.8
68.7
77.6
50.5
78.7
85.8
84.6
90.2
Tasser 7B (Wang 20248
43.4
69.2
77.0
35.8
62.5
72.3
52.1
79.7
86.5
67.8
88.8
93.1
Qwen2-VL 78 Wang et a. 2024a
46.9
69.2
79.7
43.4
69.2
78.8
53.3
79.7
86.5
73.7
89.6
92.4
CARE
43.9
67.0
75.7
41.7
68.1
76.2
52.6
74.6
87.9
92.4
Intern VideZiddeeaateed without match header faimess.
Table Video retrieval performance on DiDeMo (Hendricks al. 2017).
DiDeMo
Model
Text-to-Video
Video-to-Text
R@1
R@5
R@
R@D
R@5
R @ 10
CLIP-based Models
CLIP B/16 (Radford et al. 2021
23.5
46.3
55.2
22.2
43.8
54.0
CLIP L/14 (Radford al. 2021
24.1
48.0
58.2
23.8
44.9
54.0
LanguageBind (Zhu al. 2024
35.6
63.6
71.7
35.6
62.8
Long-CIIP B/14 (Zhang el al.
2024a
30.3
52.4
63.7
24.8
52.8
63.4
Long-CLIP I/14 (Zhang al.
2024g
32.4
56.2
65.2
28.5
54.1
64.7
Intern Vntem\iddoo2ttug
IB
Wang et al.
2024d
35.0
63.7
74.1
35.5
60.7
70.7
Contrastively Trained MLLMs
LLaVA NeXT Video 7B (Zhang al.
2024b
36.0
62.3
71.7
31.4
58.0
68.0
VL2 Chen et 20233
39.7
65.6
74.1
35.5
64.0
72.2
MiniCPM-V 2.6 Yao et al 2024
40.6
65.2
74.2
35.7
61.6
70.1
Tarsier 7B WWag al. 2024a
42.1
68.2
77.1
39.5
64.6
73.7
Qwen2-VL 7B Wang al.
2024a)
46.1
69.6
77.6
42.1
66.1
76.3
CARE
41.4
68.5
77.1
39.1
66.0
75.8
Intern Video2 is tested without match header for fairness.
B.2 EXPERIMENTS ON SMALLER MLLMS
We additionally benchmark small contrastively-trained MILMS (1B/2B) on CAREBENCH and MSR-VTT (Xu et al.) 2016). The training follows the setting of Stage-Il and consumes only 2.26 GPU hours and 6.4 GPU hours for IB and models, respectively. Table [2 Re- call@ 11,5,10) for both text-to-video and video-to-text retrieval. As shown, competitive InternVL. 2.5 IB/2B surpass Long-CLIP and narrow much the gap to earlier 7B MLLMs -LLMss-hggliighing the importance of training objectives and data over parameter count alone-while our 7B model still achieves the strongest overall results. All methods follow the same input preprocessing and evaluation settings ensure comparability.
16

---

Published as a conference paper at ICLR 2026
Table 10: Video caption performance of popular models DREAM-IK (Wang al. 2024a).
We report F1/Recall/Preciiion for each category. # Params denotes the number of LLM parameters.
Model
Parums
DREAM-IK
LLte-Action Movies Animation Movies Stock Videos
Youtube Videos Stp- Sbort
Overall
GPT-4o mini
34.3/32.7/30.6
28, 28.9/260022.66
37.9/38003799
35.5/30,2/57.5
24.7/2924424
24.031../57.44
LLaVA NeXr Video ZZhnne et 21240
78
V1.2
KChnn a 20233
7B
27. .127 1/74
20.6/18 1/33.
269//42//002
25/7/21
26 269/1447722
L3cs er
2024
71
IntenVL12.5
2014
728
MiniCPM-V
2024
7B
30,5127 7/33
24.8/22.5227
35.4/35 0/35.8
29.5/28.0/31
31,6/26
5/38,
30.5/27.9/5
Tursi::
29,5/25
33.0/28./39.11
33.
01.6230,22 222002240..3
Qua:2.VT
7244 2/92./
2/18 4/38.0
37.0/38 0/36.1
30.3/27.0/35.5
8/37./
1/33
6226,
Wang
32.0/270//8811
CARE.
32.0/30/004.2
34.5/32.//7..1
32,3/33.7441.7
38.1/36.88 86.66
Table 11: Video retrieval performance of small MILMs on CAREBENCH. All the results are reported in zero-shot setting.
CAREBENCH General Retrieval
Model
Text-to- Video
Vixeoo R@S
Video-to-Text
@
R@
R@D
R @ 10 R R@5
CLIP- Models
CLIP B/16 Radford al.
2021
45.7
79.6
89.1
48.4
82.4
90.8
CLIP 1/14 (Radford et al. 2021
90.6
54.7
86.9
93.6
LanguageBind (Zhu ct al. 2024
64.3
91.0
96.3
59.5
88.0
95.0
Long-CIII B/14 (Zhang
2024a
59.2
85.3
92.1
55.8
84.7
92.9
Long-CLIP L/14 Zhang ct al.
2024a
62.7
88.8
95.7
60.3
88.8
94,9
Intern Video2
1B (Wang et al.
2024d
72.5
93.7
97.3
69.5
94.6
97.8
Contrastively trained MLLMs
LLaVA NV 7B
Zhang et al. 2024b
66.9
89.4
96.0
62.7
89.2
95.4
MiniCPM-V 2.6
Yao
2024
71.0
92.2
97.0
69.3
92.8
97.1
Intern 1B
(Chen ef al.
2024
66.3
92.6
97.3
63.6
90.2
96.1
Intern V1.2 8B (Chen el al. 2023
72.1
92.6
96.8
73.6
93.4
97.4
VL2.5 2B (Chen
2024
73.4
93.9
97.9
73.4
92.9
97.4
Tarsier 7B (Wang et al. 2024a
71.0
93.8
97.8
70.6
94.2
98.0
Qwen2-VL 7B
Wang al. 2024b
76.6
95.3
98.7
77.4
95.6
98.7
CARE
77.0
95.6
98.7
79.0
96.8
99.1
C QUANTITATIVE AND HUMAN-ALIGNED VALIDATION ON METRICS
We conduct additional quantitative and human-aligned validation to show that CapST reflects fine- grained caption quality, correlates with human judgment compared n-gram metrics and remains stable across different LLM judges.
C.I N-GRAM METRICS ON CAREBENCH
N-gram metrics CAREBENCH including BLEU @4 METEOR, ROUGE-L and CIDEr are shown in [6] Due to the extremely high richness of the vocabulary in fine-grained captions, sentences with similar semantics differ greatly at the token level. Consequently, n-gram-based metrics already close to zero and no longer meaningful.
C.2 HUMAN-ALIGNED VALIDATION ON CAPST METRIC
To provide human-aligned validation CapST, invite 10 human experts and follow the Elo settings Table 17 to perform "which-is-better" evaluations on CAREBENCH and show the Elo scores in Table 18
17

---

Published as a conference paper at ICLR 2026
Table 12: Video retrieval performance of small MLLMs on MSR-VTT (Xu al. 2016). All the
results are reported in zero-shot setting.
MSR- VTT
Model
Text-to- Video
Vidco-to-Text
R@1
R@5
R@10
R@D
R@5
R@ 10
CLIP-based Models
CLIP B/16 (Radford al.
2021
33.8
56.1
66.6
30.5
53.8
65.5
CLIP L/14 Radtord ct al.
2021
36.7
58.8
68.0
32.8
54.7
66.2
LanguageBine (Zhu ct al. Long-CLIP B/14
2024
42.1
65.9
75.5
40.1
65.4
73,9
Zhang et al.
2024a
38.7
62.3
70.6
34.4
57.7
68.2
Long-CLIP L/14 Zhang el al.
2024a
40.9
74.6
62.2
InternVideo2steg2
IB Wang et al.
2024d
44.2
70.1
78.1
40.5
66.9
76.3
Contrastively trained MLLMs
LLaVA NeXT Video 7B (Zhang ct al.
2024b
40.3
64.9
74.1
30.5
58.0
69.0
Intern VL2. 1B
(Chen et al. 2024
41.3
64.4
73.8
36.3
61.6
70,9
Intern 8B
Chen
2023
44.6
69.3
77.4
40.8
66.6
76.5
In1ernV1.2.5 2B
(Chen el
2024
41.9
75.7
39.7
65.8
MiniCPM-V 2.6
Yao et al. 2024
44.7
69.7
77.8
41.6
68.7
77.6
Tarsier 7B WWnn 20243
43.4
69.2
77.0
35.8
62.5
72.3
Qwen2-VL Wang al. 2024a)
46.9
69.2
79.7
43.4
69.2
78.8
CARE
43.9
67.0
75.7
41.7
68.1
76.2
Table shows the pearson correlation coefficients between these metries and Elo scores (e.i. human preference). The results indicate that: (1) All the t--gram-based metrics are significantly uncorrelated with the Elo score. (2) Action and Object Fl demonstrate significant correlations with the Elo score, suggesting they better capture human preferences.
C.3 LLM JUDGE STABILITY CAPST METRIC
To show the stability of CapST across different LLM judges, evaluated all the models using an identical prompt on Deepseek V3 (DeepSeek-AA AT et al. 2024) (our implementation), Gemini 2.5 Pro/Comanici al. (2025) and GPT 4.1 O
For a intuitive comparison, we computed the correlations between these LLM judges. All inter- judge correlations exceed 0.96, indicating that CapST remains highly stable across different judges. The results are shown in Table
Table [4
Table 15
Table Video caption performance popular models on CAREBENCH using GPT 4.1 OOpe- nAI 2023) as LLM judge. Only overall scores are reported.
Models
Params
Action
Action
Action
Object
Object
Object
GPT-4o mini
36.1
30.5
44.2
34.0
26.7
46.6
LLaVA NeXT Video
7B
25,9
20.2
36.2
24.5
18.1
37.6
InternVL.2 InternVL 2.5
7B
23.9
21.4
27.2
23.5
18.3
32.7
7B
27.3
21.0
38.8
30.1
24.9
37.9
InternVL 2.5
72B
30.0
22.9
43.6
31.6
26.3
39.5
MiniCPM-V 2.6
7B
32,7
25.0
47.1
30.6
22.4
48.0
Tarsier
7B
27.2
19.0
48.1
32.0
24.2
47.4
Qwen2-VL
7B
28.5
24.6
33.9
22.9
15.8
41.7
Qwen2-Vr
72B
29.9
23.0
42.7
23.2
15.6
45.5
CAREstage Estage-1
7B
35,3
26.9
51.0
32.5
23,0
53.2
CARE
7B
34.4
26.1
50.3
32.3
23.0
54.2
18

---

Published as a conference paper at ICLR 2026
Table 14: Video caption performance of popular models CAREBENCH using Gemini 2.5
Pro (Comanici et al., 2025) as LLM judge. Only overall scores are reported.
Models
# Params
Action F1
Action
Action
Object
Object
Object P
GPT-4o mini
41.5
31.7
59.9
39.9
30.7
57.0
LLaVA NeXT Video
7B
29.0
19.9
53.2
30.1
21.5
50.1
LL2
7B
28.6
23.1
37,4
28.5
21.4
42.8
InternVI. 2.5
7B
30.4
21.1
53.9
35.6
28.7
47.1
InternVL. 2.5
72B
33.4
23.4
58.4
36.8
29.5
48.9
MiniCPM-V 2.6
7B
35.3
24.7
62.2
36.4
26.1
60.2
Tarsier
7B
32.1
21.5
63.6
36.4
27.1
55.4
Qwen2-VL
7B
32.8
25.1
44.0
27.7
18.4
56.2
Qwen2-VL.
72B
34.8
26.1
52.1
28.1
18.4
60.0
CAREgtage-
7B
41.2
31.5
59.6
37.1
26.0
64.8
CARE
7B
40.3
30.2
60.7
36.4
25.1
66.6
Table Inter-judge consistency analysis via Pearson correlation. We report the Pearson corre- lation coefficients (and corresponding p-values) between evaluation results produced by DeepSeek (DeepSeek-Al al. 2024), GPT 4.1 (OpenAl 2023), and Gemini 2.5 Pro (Comanici et al.
2025
DeepSeek-V3 V3 vs GPT-4.1
DeepSeek-V3 V3 vs Gemini-2.5-Pro
GPT-4.1 vs Gemini-2.5-Pro
Metric
Corr.
P-val
Metric
Corr.
P-val
Metric
Corr.
P-val
Action Fl
0.97
5.6e-07
Action
0.98
1.5e-07
Action Fl
0.98
3.5c-07
Action R
0.96
3.6e-06
Action R
0.97
5.4e-07
Action R
0,92
7.7e-05
Action P
0.94
1.8e-05
Action P
0.95
7.9e-06
Action P
0.92
4.9e-05
Object Fl
0.99
1.3e-08
Object Fl
0.98
1.5e-07
Object Fl
0.99
3.5e-09
Object
0.99
2.4e-09
Object
0.99
2.4e-08
Object
0.99
1.2e-09
Object
0.96
5.6e-06
Object
0.99
3.1c-10
Object
0.96
2.7c-06
Avg
0.97
Avg
0.98
Avg
D LIMITATIONS
Although CARE demonstrates strong generalization ability on fine- fine-grained video retrieval and cap- tioning tasks, it exhibits a certain degree of performance drop compared to the baseline (i.e con- stratively trained Qwen2-VL.) on coarse-grained, traditional datasets such as MSR-VTT (Xu et al.] 2016), as shown in Table 8 and Table 9 To further investigate this phenomenon, we provide in this section the results of CARE versus contrastively trained Qwen2-VL (Wang et al. 2024b) on MSVD (Chen & Dolan, 2011), MSR-VTT (Xu et al., 2016), DiDeMo (Hendricks al.) 2017). VDC (Chai et al. 2024) short captions, DREAM-IK (Wang et al. 2024a), ShareGPT-4o (Cui et al., 2024 and CAREBENCH, together with explicit, quantitative analysis of how performance varies with caption length and action-object complexity across benchmarks.
D.1 PERFORMANCE OF CARE AND QWEN2-VL. 7B ACROSS DIFFERENT BENCHMARKS
Table 20 shows the results of CARE and contrastively trained Qwen2-V VL difference benchmarks.
To ensure fair comparison model performance across datasets and to eliminate the influence of absolute Avg Recall values, we use the relative gap (percentage) in Equation (5). The results of performance gaps across difference benchmarks are illustrated in Table 2T
ReCRre RQwen
G
(5)
RQwen
where ReARE Avg Recall of CARE and Rowen is Avg Recall of contrastively trained Qwen2-VI..
19

---

Published as a conference paper at ICLR 2026
Table 16: CAREBENCH results with BLEU@4, METEOR, ROUGE-L, and CIDEr.
Models
BLEU@4
METEOR
ROUGE-I
CIDEr
LLaVA Video 7B (Zhang et 2024b)
0.030
0.155
0.185
0.019
MiniCPM-V 2.6 Yao et al.
2024)
0.022
0.139
0.163
0.025
Intern VL. 2.5 8B (Chen
2024)
0.015
0.120
0.156
0.000
Tarsier 7B (Wang ct al.
2024a)
0.012
0.093
0.151
0,000
Qwen2-VL 7B (Wang cl al. 2024b)
0.013
0.114
0.128
0.009
Qwen2-VE VL 72B
(Wang al.
2024b)
0.018
0.115
0.167
0.000
CARE
0.012
0.100
0.153
0.000
Table 17: Elo setting.
Parameter
Number
initial Elo mean
1,000
Elo standard deviation
300
base of logarithm scaling factor K-factor
10
400
32
minimum Elo rating
700
D.2 PEARSON CORRELATION ANALYSIS BETWEEN PERFORMANCE GAP AND BENCHMARK STATISTICS
To visualize how the relative performance gap numerically relates to each benchmark statistic, we performed Pearson correlation analysis; the results are shown in Table 22
According to the results illustrated above, we can come to the conclusions that:
1. CARE has clear performance advantage on long-text, fine-grained tasks especially those containing a large number of actions and objects. Pearson correlation analysis shows that the Relative Gap between model and Qwen2-VL is strongly and positively correlated with Avg. Words and Avg. Objects, and moderately positively correlated with Avg. Ac- tions. The more detailed the benchmark and the more objects and actions it contains, the more pronounced CARE'S advantages become.
2. CARE underperforms Qwen2-VL on coarse-grained retrieval benchmarks with simple ob- jects and actions within the 7-to-60-word range. In contrast to Qwen2-VL, CARE incor- porates an additional Fine-grained Alignment SFT. Although more ablation studies are needed to verify how the SFT design affects performance, this extra training reduces the model's generalization on coarse coarse-grained tasks.
tentatively conclude that
E OBJECT LEAKAGE IN TEMPORAL CAPTION
We'v 've tried our best to reduce the object leakage (i.e. having unnecessary objects) in temporal caption. We here provide statistics and examples illustrating the degree of static/object leakage in temporal annotations. We extract the objects in spatial captions and temporal captions using Deepseek V3. The avg. objects in spatial caption is 10.76 and avg. objects in temporal caption 6.82. The following shows two examples.
Example 1:
[video] 0_00000004_3..1 mp4
[category] Personal Care [subcategory] apply_eyebrows
[spatial_st [gaatial_objetss]
20

---

Published as a conference paper at ICLR 2026
Table 18: Elo scores of popular models CAREBENCH.
Model
Elo Score
Intern VI.2 (Chen et 2023
889.31
LLaVA NeXT Video 7B Zhang et al.
2024b
914.27
Qwen2- VL. Wang et al. 2024b
959.34
MiniCPM-V 2.6 Yao et al.
2024
1051.14
Tarsier 7B (Wang el al. 2024a
1061.59
CARE
1124.34
Table 19: Pearson correlation coefficients between metrics and Elo scores.
Metrics
Elo Score
BLEU@4
-0.41
METEOR
-0.54
ROUGE-L
-0.19
CIDEr
-0.14
Action F1
0.81
Object FI
0.71
woman with long black hair right eyebrow with brow dye soft orange eyeshadow with silver glitter pale pink lips
pink wall with a black grid pattern an upper garment with a blue base upper garment with white floral patterns small bottle of yellowish-brown brow dye small white mirror
a mirror with a yellow-edged border mirror with silver design in the lower right corner a brush with a black bristle head brush with yellowish-brown stick brush with silver-white text branding.
[temporal_objects]
person
brow gel
right eyebrow
mirror
small brush
Example
[video] v_00007656_11mmp
[category] Sports, Excercise [subcategory] drlp_golf
[spatial_objectss]
boy in light blue short-sleeve shirt a boy in khaki shorts
boy wearing white baseball cap
a boy wearing white-brown athletic shoes black golf club
boy in a red short-sleeve shirt boy in khaki-striped shorts
21

---

Published as a conference paper at ICLR 2026
Table 20: Performance of CaRe and Qwen2-VL 7B across different datasets.
Model
ShareGPT-44
CaReBench
MSVD
VDC short captions
MSR-VTT
DIDeMo
DREAM-IK
CaRe
87.38
91.03 90.38
78.88
80.19
62.10
61.32 62.97
92.93
Qwen2-VL. VL. 7B
87.03
79.20
81.00
64.53
94.68
Metrics are computed the average of T2V R@ 1,2,5 and V2T R@ 1,2,5.
Table 21: Performance gaps, average words and actions/objects complexity of different benchmarks.
Benchmark
Avg. Words
Relative Gap (%)
Avg. Objects
Avg. Actions
MSVD
7.0
-0.404
2.34
1.56
MSR-VTT
9.4
-3.766
2.34
1.89
DiDeMo
29.1
-2,620
4.07
3.69
VDC short captions
32.8
-1.000
4.67
3.18
DREAM-IK
59.3
1.817
5.79
6.16
ShareGPT-40 125.7
0.402
8.31
5.78
CaReBench
228.0
0.719
10.03
6.94
Objects and actions extracted by Deepseek V3 (DeepSeck-Al et al.
2024
boy wearing a red baseball cap boy wearing white-brown athletic shoes white glove
black golf club
vast expanse of grass pushcart on the left some clothing on the pushcart
deep blue bag on the grass
railing enclosing the golf course several yellow-green trees white house white signboard on the ground
[temporal_object
boy in red shirt, boy in a blue shirt, golf club, grassy field, ball
F LOGITS VISUALIZATION
To explore how CARE works, feed its output embedding of a video featuring chef is cutting tomatoes in the kitchen into the last linear layer (i.c. Im Im.head). It projects the embedding into the vocabulary space. By decode the output logits, an embedding. can be discovered that tokens with high logits constitute the essential semantics of the input video, as shown in Figure 5cl describing the main visual objects and actions of the video
easily visualize the semantic components of
Table 22: Pearson correlation coefficients (r) between relative gap and different variables.
Item
Pearson (p-value)
Rel Gap vs Avg. Words
+0.706 (p 0.076)
Rel Gap vs Avg. Objects
+0.718 (p = =0.069)
Rel Gap vs Avg. Actions
+0.523 (p 0.229)
22

---

Published as a conference paper at ICLR 2026
such kitchen, cutting, tomatoes and chef, while the tokens in Figure 5b contain many subwords and irrelevant tokens like dice, car and pizza. It can be inferred that the semantic distribution the next token space is hugely changed by two-stage SFT, allowing the main semanties to be the components of the embedding.
(a) The input video.
14.
Logit
lahen
(b) Top 50 tokens decoded from the output embeddings of Qwen2-VL
16-
Loght Vanu
12
20
(c) Top 50 tokens decoded from the output embeddings of CARE.
Figure Top 50 tokens decoded from the output embeddings of Qwen2- VL and CARE. Qwen2-VL is the baseline model of CARE without any SFT. Compared to Dwen2-VIL, two-stage SFT makes the semantic components of CARE embedding much more related to the input video featuring chef is cutting tomatoes in the kitchen.
G ANNOTATION GUIDELINES
To inform our annotators the key points that they need to pay attention to, we design guideline to teach them how describe videos accurately. The guideline is shown below.
Annotation Guideline (Stage 1)
Task
Your task to descrihe videos detail and hierarchically within 150-300 words. We provide two examples and some points you may need to know.
Example I: Cutting a Watermelon
(A video about cutting a watermelon provided.)
Summary This video shows man cutting a watermelon.
Object Description The man is wearing a green T-shirt and black apron, with black mesh hat on his head. His left hand is wearing gray glove, while his right hand, holding fruit knife, wearing a transparent glove. He stands at the corner of the countertop, with white cutting board in front of him, holding watermelon. To his left, there is sink containing another uneut watermelon.
Action Description The man first cuts off both ends of the watermelon. Then, he places the watermelon upright and rotate it clockwise, slicing off the rind piece by
23

---

Published as a conference paper at ICLR 2026
Stage-l: Detailed Annotttin
disbys "okce Wood" while apper the
penson
bibeled Body Postiont The commentary tioughout the video conducted in Engiish The zaimmer dressed a sert and wearing a whis and red swin cap necgate though posl. camera istally captures
1p or
from They kidk efficeetty. hext, the shot shits to a dide view the swlmmer, During tht exnsiten, the bike watermrt the loner right comner disapsoors fiss, lollowdd walermrk 'iht remaining background vanchh Meanemils, swimeress movements...
breathe,
coo
Bib
Worker Annotation
Expert Correction
upper comerr new water
appears watermarks
end the weee. provides
Stage 2: Spatial-Tempora Part Separation
move baricc descriptions annotation
5 pesial Annotation
This video showcases person su miiing o (in conserion the to In the upper left comer of Bie screen ddoplyg the "Voce of Ken Word while the aceam of the fram labolos 'Body Pettinn: The -enaaeeiiaao--oooouuoooo description)
Geoughoui- -i------- Wees- conducws- i- Eegiih gremove
Spettal-eeporaa CaapporllAnntaaann Annotation 5 Searrtioo
weering
thrugg the pool in loser right comer scries, the nime aas Schipper" snows. (add subttle description missing Stage 1),
Temporal Annotation
camera Hnlillly captures attiete from behind, (weto semming in center using
Expert Correction
effcientty Rest shot stitss bue watermark in loeer right comeer disspoears tist. followed the watermanc the upper let corres es A new walamark than appears the lower iihtt remariiy whible undi the and iideo, when nalama is wards Heananlla, backgroudd audie wwwvides--eeeee wow.-rentaryyy----oooo un- description)
transiton,
Figure 6: An overview of the annotation pipeline. In Stage-I, workers are asked to describe videos hierarchically in detail. In Stage-II, workers need to separate spatial descriptions with temporal descriptions.
piece. He uses the knife to push the rind into a trash bin on his right. Next, he takes light green tray from his right and place it next to the cutting board. After peeling the watermelon, he cuts it into pieces and slides them onto the light green tray.
Misc Description The video is filmed from behind the man, showing a quick and efficient process of cutting the watermelon. With impressive speed, he slices through the fruit, showing his expertise.
Example 2: Cutting a Tomato
(A video about cutting tomato is provided.)
Summary In the footage, someone is holding a knife and cutting tomato on cutting board.
The person is wearing black clothes, with watch his left wrist. On the cutting board, there are four previously cut tomatoes and one sliced green fruit. On the table, there is bag of uneut tomatoes and small knife. In the top left corner of the video, there is a "luxeat" watermark, and the text "NOWT'VE SEEN EVERYTHING" is written in the bottom left corner.
Object Description
Action Description While cutting the tomato, the person first slices it forcefully with one cut, then speeds up the chopping frequency, quickly slicing the tomato into neat pieces.
Mise Description The video is filmed from a third-person person perspective, showcasing clean and efficient vegetable-cutting. The person's 's motions are skillful and confident.
Key Points for Descriptions
Object Description Describe the entire frame in as much detail as possible. Focus on the objects visible in the frame, clearly describing their positions, appearances, and interactions (e.g "left hand" "right hand" "on the left" "on the right" "above" "below" "upside-down" "holding" "wearing" etc.). This part should follow the de- scription order outlined below: (1) describe the main object in the frames: for exam- ple, "The person is wearing a green T-shirt and a black apron, with black mesh hat their head. His left hand is wearing gray glove, while his right hand, holding fruit knife, is wearing a transparent glove." (2) describe the secondary objects in the frames: for example, "The person is standing at corner of metal countertop. In
24

---

Published as a conference paper at ICLR 2026
front of him is white cutting board with watermelon on it. To his left, there is sink containing another uncut watermelon."
Action Description Clearly describe the actions performed by the main subject, noting the sequence events (e.g first do X, then do Y). Include details about the nu- ances of the actions (e-g rotating the watermelon clockwise, flipping it upside-down) and the style of execution (e.g cutting fruit very quickly, climbing tree clumsily).
Misc Description Describe the video's filming perspective (e.g firsstpprsoon"" "third-person," "off-site footage of a competition") and provide a brief summary of the overall style and impression conveyed by the actions (e-g orderly and fast watermelon cutting, sharp and efficient movements, clumsy actions, or dangerous behaviors). This part should be concise, within 2-4 sentences.
Annotation Guideline (Stage 2)
Task
In this stage, your task to separate the original hierarchical descriptions into two parts: spatial descriptions (which do not include any descriptions about movements) and temporal descriptions (which do not include any object descriptions). Camera movements, such as zoom-ins, zoom-outs, etc should be included in temporal descriptions. Key Points for Descriptions
The spatial description should cover the key objects, secondary objects, and the en- vironment the frame. must ensure that, based on the spatial description alone, the videos in the assigned subcategory can be differentiated from one another.
The temporal description should exclude any obvious static object descriptions that help distinguish different videos. Only the details and sequence of actions should be kept, and it must ensure that, based on the temporal description alone, the videos in the assigned subcategory can be differentiated from one another.
All the contents of spatial and temporal descriptions should come from the Stage descriptions, and no additional details should be added. Both spatial and temporal descriptions should begin with a summary.
H LICENSE INFORMATION
Datasets.
Below the datasets used in this paper that have known license information:
DiDeMo (Hendricks et al. 2017) (BSD 2 Clause License), Activity-Net (Heilbron et al. 2015)
(MIT License), DREAM-IK (Wang al. (2024)) (Apache-2.0 License), VDC (Chai et al,, 2024)
(Apache-2.0 License), Please note that CAREBENCH will be released with MIT License in the future.
Models. Below are the models used in this paper that have known license information: In-
VL2 (Chen al. 2023) (MIT License), InterVVL22. (Chen et al., 2024) (Qwen License), LLaVA NeXT Video (Zhang et al. 2024b) (Llama 2 Community License), Qwen2-VL. (Wang al. 2024b) (Apache-2.0 Licenss,, Tarsier (Wang et al. 2024a) (Apache-2.0 License), LanguageBind (Zhu al,
2024
(MIT License), Long-CLIP (Zhang et al.) 2024a) (Apache-22 2.0 License). Intern Video2
(Wang
ct al.
2024d) (Apache-2.0 License).
25

---

Published as a conference paper at ICLR 2026
Video
Caption
Annotation: This video showcases heartwarming scene amusement park where man holding little girl. The man is dressed blue top. revealing only head, neck, part sleeveless blue top adorned with plenty of sequins on the front. Around her neck, she wears several strands of pink beaded necklaces. Surrounding them other children and adults, with person Peppa mascot features pink pig small fenced area made of wood. Behind them is white wall that has blackboard with green and pink patterns drawn on it. The girl is leaning against the man's right being held high by him, with her left hand resting his neck and her right hand look back, releasing her left hand from
upper body. The little
golden hair wearing
costume standing behind them. children
body. costumed character interacting and waving
neck. The man mouths something to her, and the girl faces the camera again, cheerfully raising her right hand and waving towards The Peppa Pig mascot behind embrace sameone in front before turning to the right to keep waving. The video captures this scene from the viewpoint the characters, and their smiles, along with those of the nearby onlookers, bright and joyous, showcasing delightful atmosphere.
hanging down beside her. She then turns around
left
resting on its belly
continuously waving with right hand, even stopping briefly to
holding young girl, The dressed blue top, revealing only his head, neck, and part of his upper body. The little girl has golden hair and wears blue sleeveless top adorned with numerous sequins the front. Around her neck. she sports necklace made of several pink high
Spatial Annotation: This showcases
park
ground. Her left hand rests the man's neck, beads. right hand hangs naturally by her side. Surrounding them are other children and adults, and in the background, there's person dressed costume resembling Peppa Pig, with
girl leaning against man's right arm,
pig head and blue body. This costumed character is standing person small enclosed area made wooden fencing. interacting and greeting the children outside. Behind him white wall featuring small blackboard decorated with
and pink patterns.
Temporal Annotation: This video showcases scene in an amusement park where man holding little girl in his arms. The girl turns her head look back, releasing her left hand from the man's neck while he says something to her. She then straightens up to face the camera and happily waves her right hand it. Behind them, a Peppa Pig plush toy stands with hand resting on its belly and its right hand waving enthusiastically. Al one point, it brielly hugs the person in front before turning to the right continue waving.
Video
Caption
chairs This video showcases woman styling her hair. She dressed in white blouse and has long hair. On her right wrist, she wears round brush and holds black hairdryer. front of her white table, which has two black towels gray chair, bahind there several mirrors. she uses the round brush in her left hand to curl section her hair on the left side while simuttaneoully using the hairdryer in her right hand to blow dry those strands. Afterward, she continues to use the round brush to style her hair securing the ends while also using the hairdryer with her left hand to blow dry the hair. The entire video filmed from frontal perspective, showcasing her expertise and technique.
Annotation:
wetch, while her left hand combs drapad
alongside as numerous
seated
tables
awayy
adomed
beginning the
showrases woman drying she wears a watch, while her left hand grips a round hairbrush and holds black hairdryer, In front of her there is white table adorned with two black towels, on which verious combs are placed. The woman is seated in gray chair, with a row tables and chairs facing away from her behind. The tables are stocked with numerous bottles. Additionally the wall behind har features several mirrors hanging prominently.
Spatial Annotation:
top
long hair. her right wrist,
section with her left hand to combo through her hair, securing the brush at the end, and then she uses the hairdryer in her hand to finish styling those sections of hair.
showcases woman styling her Temporal Annotation: simultaneously blow-drying with hairdryer her right hand. After that, continues left
starts by using round
her hand
26

---

Published as a conference paper at ICLR 2026
Video
Caption
The video captures heartwarming moment woman embracing dog. Set outdoors under brilliant sun, it features brown-haires hairee woman wearing a black tank top, holding her black dog close. in the background, there's red and white vehicle adomed with paw print decals. Initially, she gazes down the side profile of her dog, while the other gently strokes its fur. As the camera rotates clockwise, the dog playfully sticks out its tongue, attempting to lick her, She closes har eyes and tums
Annotation:
wrapped around
wearing blissful expression, while both hamds continue caress the dog's neck and Later on, lifts dog's front paws towards camera while scratching its neck.
moment, another person's arm appears on the right side of the frame, gently rubbing the dog's chin. The woman plants kiss on the dog's forehead, then leans her head closaly against using her left hand. She then embraces the dog tightly once more, tenderly stroking the on its chin with her right index finger A man's hand reaches in from the right side of the frame to give the dog some affectionate scratches on its head As the camera gradually puils back, the woman continues stroke the dog's back with her left hand while nuzzling her The video is shot from a third-person perspective, with the camera positioned very close to the woman and her dog. The scene filled with the warmth of their embrace, creating wonderfully intimate atmosphere.
small pup. The tilts its head outward, prompting her to start playing with its front paws
against
Spatial Annotation: The video captures the moment woman embraces her dog. Set outdoors in glorious sunshine, the scene features hrawn-haireed woman wearing black tank top, holding her black dog close. In the background, there red white vehicle adomeed with paw print patterns.
The video captures tender moment woman embracing her dog. At first, she gazes down dog's side profile, with one hand wrapped around the dog and the other gently stroking it. As the camera rotates clockwise, the dog eagerly sticks out tongue, attempting to lick her, but she closes her eyes and turns away, using both hands to dog's neck head. Later lifts dog's two
Temporal Annotation:
the paws to face the camera while continuing to scratch its neck. At this paint, another person's arm appears on the right side of the video, reaching out to pet the puppy's chin. The woman kisses the dog's forehead and then presses begins to manipulate its frant paws with her left hand. She then pulls the dog in tightly, continuing to pet it and gently brushing her right index finger along its chin fur. Just outside the frame the right, scratching its As camera gradually zooms out, woman uses her left bottom, while also nuzzling her head against its.
head closely against the small dog's. The dog tilts its head outward,
woman
extends his hand pet the dog. strake puppy's back from top to
Video
Caption
Annotation: This video showcases the fencing competition between athletes from the Arab Republic of Egypt and South Korea. the flags of both countries, their respective abbreviations, and the names of the
At the bottom of the videa, you competitors. The match progresses through rounds Republic of Egypt, while on the right is South Korean fencer CHOI B. During the match, the Egyptian fencer has their left leg forward
3. On the left side, we have A. ABOUELKASSEM representing the Arab
the sword their right hand. Both athletes are clad in fencing uniforms and black helmets, with the South Korean fencer standing out in red shoes. As the match unfolds, they begin by cautiously probing each other before the Korean fencer suddenly lunges forward, striking the Egyptian athlete on leg. response, the Egyptian fencer leaps upward to evade the blow but loses their balance upon landing and falls to the left. The second part of the video features slaw-motion replay of this action. The entire video is competition area, vividly illustrating various dynamics of the match.
hand, while the
their right leg forward and wields the sword their
filmed from the side
Spatial Annotation:
This video showcases the competition between athletes from the Arab Republic of Egypt and South Korea on fencing arena.
bottom of the videa, you can see the flags both countries, their abbreviated names, names of the athletes. The match is in rounds 1-3. On the left is A ABOUELKASSEM representing the Arab Republic of Egypt. while the right CHOI B. from South Korea. Throughout the competition, both athletes dressed fencing attire and wearing black helmets. Notably, the South Korean athiete wearing red shoes. The Egyptian athlete has their left legg forward and holds the sword in their left hand, while the South Korean athlete has their right leg forward with the sward held in their right hand.
Temporal Annotation: This video showcases the competition between the athletes from the Arab Republic of Egypt and South Korea on the fencing arena. During the match, the two players initially engaged in careful testing of each other's defenses. Suddenly, the South Korean fencer lunged forward with swift thrust, striking the leg of the athlete from the Arab Republic of Egypt. In response, the Egyptian fencer jumped up. but unfortunately, lost balance upon landing and fell to the left. The second part the video features slow-mation replay of this sequence events.
27


Reviews:
[{"review_id": "5ZEtgP1xab", "reviewer": "Reviewer_X3xc", "summary": "This paper introduces CaReBench, a new fine-grained benchmark for video captioning and retrieval, comprising 1,000 videos with human-annotated hierarchical captions. Each caption includes spatial and temporal components, allowing a detailed evaluation of models’ understanding of static objects and dynamic actions.\n\nThe authors also propose two novel metrics:\nReBias, for quantifying spatiotemporal bias in retrieval tasks, and\nCapST, for evaluating captions with both precision and recall over spatial and temporal elements.\n\nAdditionally, the paper presents CARE, a unified baseline model built upon Qwen2-VL, trained via two-stage SFT to handle both captioning and retrieval tasks simultaneously.", "questions": "1. How stable are CapST scores across different LLM evaluators (e.g., GPT-4o vs DeepSeek-V3)?\n\n2. Have you explored joint multi-task training (caption + retrieval simultaneously) instead of the sequential two-stage pipeline?\n\n3. What are the main annotation challenges in separating spatial and temporal components, and how is annotator consistency ensured?\n\n4. Given the rapid evolution of multimodal large language models (MLLMs), have you considered adding comparisons with more recent MLLMs (for example, Qwen2.5-VL) to demonstrate how your benchmark and unified model stack up against the latest models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Benchmark Design:**\nCAREBENCH introduces a uniquely structured dataset with hierarchical annotations that explicitly separate spatial and temporal descriptions, filling a clear gap in current benchmarks.\n\n2. **New Evaluation Metrics:**\nReBias and CapST are well-motivated and address the shortcomings of existing metrics (e.g., CIDEr, AutoDQ, VDCScore), providing more interpretable and fine-grained evaluations.\n\n3. **Unified Framework:**\nThe CARE model elegantly unifies retrieval and captioning within one architecture using a two-stage fine-tuning pipeline, demonstrating potential task synergy.\n\n4. **Comprehensive Experiments:**\nThe paper includes extensive quantitative results across state-of-the-art models and thorough ablation studies on the two-stage fine-tuning process.", "weaknesses": "1. **Reliance on LLM-Based Evaluation:**\nCapST depends on an LLM (DeepSeek-V3) as the evaluator, which may introduce bias. The paper lacks inter-rater consistency checks or human alignment experiments.\n\n2. **Theoretical Depth:**\nThe paper mainly focuses on empirical contributions. The “unified mapping” idea (ϕ: RT×H×W×C → RD) is intriguing but not theoretically explored.\n\n3. **Incomplete Bias Mitigation:**\nWhile ReBias reveals spatiotemporal imbalance, the proposed model still shows clear temporal bias, and the paper does not attempt to mitigate it.\n\n4. **Ablation Detail and Visualization:**\nSome analyses, such as the effects of video length, motion complexity, or per-category performance variance, are missing and would add valuable insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NlpsPy5Sm0", "reviewer": "Reviewer_uxnZ", "summary": "The paper introduces CAREBENCH, a new benchmark of 1,000 videos with rich human-written captions designed to evaluate fine-grained video understanding. Each video has hierarchical, multi-aspect annotations (overall summary, static object details, dynamic action details, and miscellaneous context). Spatial and temporal descriptions are manually separated to probe how models handle static scene vs. dynamic action information. Based on this data, the authors propose two evaluation metrics: ReBias, which measures a model’s bias toward spatial vs. temporal retrieval, and CapST, an LLM-based captioning score combining recall/precision of objects and events. They also present CARE, a unified video-language model (built on Qwen2-VL) trained via a two-stage fine-tuning (first to improve caption detail, then contrastively for retrieval). Experiments (zero-shot evaluation on CAREBENCH and other datasets) show CARE achieves very strong performance in both detailed captioning and video retrieval, outperforming specialized retrieval models (CLIP variants) and powerful captioning MLLMs. The paper carefully discusses related work, conducts ablations of the two-stage training, and analyzes a consistent spatiotemporal bias (models do much better on spatial cues than temporal actions).", "questions": "1. How sensitive is the CapST metric to the choice of LLM judge or prompt? Have you validated its consistency using alternative models like GPT-4 or through prompt variations?\n\n2. While CARE shows strong zero-shot retrieval results on MSR-VTT, MSVD, and DiDeMo, could you provide more experimental details? How does it perform on captioning tasks for these datasets, and what are its limitations on out-of-domain videos?\n\n3. Given CAREBENCH's scale of 1,000 videos, do you recommend it primarily for evaluation or also for fine-tuning? What are the risks of overfitting and how might training on it affect model robustness?\n\n4. Will the benchmark be publicly released? Are there licensing restrictions from FineAction, and how will reproducibility be ensured?\n\n5. Given the spatial bias identified by ReBias, have you explored any mitigation strategies, such as data augmentation or balanced training? What approaches would you suggest to reduce this bias?\n\n6. How does CapST handle paraphrases or partial omissions? Could you provide quantitative correlation between CapST scores and human judgments to further validate the metric?\n\n7. Could you elaborate on how conflicts between annotators were resolved during expert refinement? What guidelines ensured consistency, and was any inter-annotator agreement measured?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. CAREBENCH demonstrates notable strengths across key areas. Its originality lies in the novel hierarchical annotation schema—dividing video content into summary, objects, actions, and miscellaneous categories—which addresses a critical gap in video understanding evaluation. The introduction of specialized ReBias and CapST metrics further showcases innovation, leveraging the dataset's structure to examine biases and caption quality.\n\n2. The work exhibits strong methodological rigor through its careful human annotation process involving dual annotators and expert refinement. The two-stage training approach is well-designed, with Stage I focusing on fine-grained caption alignment and Stage II on contrastive retrieval training, a design validated through ablation studies.\n\n3. The paper is clearly written and logically structured, with informative figures and tables that effectively support the technical content. The comprehensive related work discussion demonstrates deep scholarly engagement.\n\n4. In terms of significance, CAREBENCH provides valuable tools that reveal how current VLMs often rely on spatial shortcuts while struggling with dynamic actions. By exposing these biases, the work should stimulate development of more balanced models, with the CARE baseline demonstrating the feasibility of unified modeling approaches.", "weaknesses": "1. The work has several limitations that warrant consideration. While the paper effectively identifies significant spatiotemporal biases in models, it stops at measurement and does not propose methods to mitigate these biases. This focus on benchmarking over developing corrective techniques somewhat limits its immediate practical impact on improving model design.\n\n2. The generalization capability of the proposed CARE model remains uncertain. Its evaluation on standard benchmarks beyond CAREBENCH shows mixed results, particularly on MSR-VTT retrieval where it fails to clearly outperform established baselines. More comprehensive analysis is needed to understand how well the unified approach transfers across diverse domains.\n\n3. The CapST metric relies exclusively on DeepSeek-V3 as the sole LLM judge for evaluating objects and actions, which raises potential concerns about robustness and inherent biases in the assessment. Although some validation against human judgment is provided, a more extensive evaluation using multiple LLMs, varied prompts, or additional human raters would strengthen confidence in the metric's reliability and general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2OE9wrIqs", "reviewer": "Reviewer_DL3n", "summary": "The paper introduces CaReBench, a fine-grained benchmark for video captioning and retrieval with human-annotated spatial and temporal descriptions, along with two new evaluation metrics, ReBias and CapST, to analyze spatiotemporal bias. It further proposes CARe, a unified two-stage model that jointly handles both tasks, achieving competitive performance compared to CLIP-based and MLLM-based baselines.", "questions": "1. Could the authors clarify the motivation for integrating video captioning and video retrieval into a single unified model? It would be helpful to explain whether these two tasks bring any mutual benefits or complementary effects in practice.\n2. Regarding the use of LLMs for NLI judgment, could the authors elaborate on its reliability? For instance, would the evaluation results vary significantly if a different LLM were used? \n3. It would be interesting to know whether the EOL-based embedding extraction method can effectively distinguish long texts that are semantically similar but not identical.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposes CaReBench, a fine-grained benchmark with detailed spatial and temporal annotations and new metrics ReBias and CapST for analyzing spatiotemporal bias.\n2. Introduces a unified model CARe that jointly handles video captioning and retrieval through a two-stage training framework.\n3. Demonstrates strong experimental results and clear analysis, supported by high-quality human annotations and well-presented methodology.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uAJM2aj5NZ", "reviewer": "Reviewer_54Wg", "summary": "This paper propose CAREBENCH, a novel evaluation benchmark for fine-grained video captioning and retrieval with manually separated spatial annotations and temporal annotations. New evaluation metrics ReBias and CapST are introduced for a deeper investigation into the spatial and temporal biases inherent in VLMs. Comprehensive experiments are conducted with developed unified baseline method CARE for address both tasks.", "questions": "1.\tSee weakness.\n\n2.\tI appreciate the dedicated efforts on the proposed benchmark and its experiment results. However, I still have some doubts about the positioning of this job as a benchmark or dataset. To begin with, as a benchmark, its targeted video captioning and retrieval are relatively old video understanding tasks, the name of the proposed method should require more innovative new concepts of video understanding. Meanwhile, many compared benchmarks in Table 1 are the test set of a larger video understanding dataset, which includes more training samples for the cross-validation of its effectiveness, but this work only proposes a testing benchmark instead of a complete dataset for fine-grained video captioning and retrieval. In summary, I would give a much higher score if this work proposed a complete dataset with both a training set and a testing benchmark. If not, I would consider raising my score if the author integrates these two tasks to provide convincing and innovative concepts of video understanding.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tFine-grained video captioning and retrieval are worth studying and have certain commonalities. This work proposes a new benchmark for unified evaluation on both tasks.\n\n2.\tThe experiment result is solid with a newly developed baseline method for both video captioning and retrieval.\n\n3.\tConvincing visualization examples are provided to prove the effectiveness of the proposed method.", "weaknesses": "1.\tAlthough video captioning and retrieval tasks have a certain duality, the name of the proposed fine-grained video captioning and retrieval benchmark in this article seems to simply blend these two tasks, appearing to be an improvement and integration of previous work, lacking some innovative new concepts of video understanding.\n\n2.\tThe developed baseline method is encouraged to be tested on some other related video understanding benchmarks to demonstrate its generalization ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)