IF-VidCap: Can Video Caption Models Follow Instructions?
Authors: Shihao Li
Yuanxing Zhang
Jiangtao Wu
Zhide Lei
Yiwen He
Runzhe Wen
Chenxi Liao
Chengkang Jiang
An Ping
Shuo Gao
Suhan Wang
Zhaozhou Bian
Zijun Zhou
Jingyi Xie
Jiayi Zhou
Jing Wang
Yifan Yao
Weihao Xie
Yingshui Tan
Yanghai Wang
Qianqian Xie
Zhaoxiang Zhang
Jiaheng Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Video Caption
Instruction-following
Benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce IF-VidCap, a new benchmark for evaluating controllable video captioning, which contains 1,400 high-quality samples.

Abstract: Although Multimodal Large Language Models (MLLMs) have demonstrated proficiency in video captioning, practical applications require captions that follow specific user instructions rather than generating exhaustive, unconstrained descriptions. 
Current benchmarks, however, primarily assess descriptive comprehensiveness while largely overlook instruction-following capabilities.
To address this gap, we introduce IF-VidCap, a new benchmark for evaluating controllable video captioning, which contains 1,400 high-quality samples.
Distinct from existing video captioning or general instruction-following benchmarks, IF-VidCap incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness.
Our comprehensive evaluation of 26 prominent models reveals a nuanced landscape: despite the continued dominance of proprietary models, the performance gap is closing, with top-tier open-source solutions now achieving near-parity.
Furthermore, we find that models specialized for dense captioning underperform general-purpose MLLMs on complex instructions, indicating that future work should simultaneously advance both descriptive richness and instruction-following fidelity.

Published as a conference paper at ICLR 2026
IF-VIDCAP: CAN VIDEO CAPTION MODELS FOLLOW INSTRUCTIONS?
Shihao Li"e, Yuanxing Zhang "aaaang",JJannaee .Jiangtoo Wu'n, Zhide Lei' Yiwen He' Runzhe Wen', Chenxi Liao', Chengkang Jiang', An Ping'. Shuo Gao', Suhan Wang', Zhaozhou Bian', Zijun Zhou', Jingyi Xie*, Jiayi Zhou', JJing Wang', Yifan Yao' Weihao Xie Yingshui Tan Yanghai Wang", Qianqian Xie Zhaoxiang Zhang", Jiaheng Liu NNanjing University, Kuaishou Technology, SCASIA, 'Shanghai University, MM-A-P Ssshihhe sma n mnju.ddr cn liujiahengenji. wew.nnjjeeddc......
Use an onnordered list starting with 'o describe mare events chronological order. Each must (15-20 words)
Cc.
Event L: Attakk
1. masked Yiga Clan member raises sickle weapon, preparing strike down defenseless Princess Zelda.
2. Suddenly. Link appears to rescue her, standing protectively over Zelda after defeating one of the Yiga attackers.
3. Surrounded more approaching enemies, determined Link stands ready defend the princess in the desert.
Event 2: Wotedd
1. A character in a mask throws hook Into the sky.
2. Two characters, one kneeling and one standing, face off against two masked figures in desert.
3. The standing character turns around, revealing blue tunic and golden hair.
A masked character in red armor raises sickle high into the air. A woman in blue and gold armor looks up at sky. The sickle is seen flying through the and then disappearing. The woman closes her eyes and then opens them, looking determined, A man in blue and gold armor stands up with a sword, ready for battle. woman
Surrounded
tarsier
The faces characters approaching them, while the woman watches. The woman looks concerned as she continues to observe the situation. The man lurns his head look woman, who watching approaching characters.
ground
masked
Figure 1: Differences in Controlled Video Captioning Capabilities among MLLMs. In our case, constraint types are color-coded, with each color corresponding to a category in Figure 3d and sentences highlighted in red indicate incorrect event captions.
ABSTRACT
Although Multimodal Large Language Models (MLLMs) have demonstrated pro- ficiency video captioning, practical applications require captions that follow specific user instructions rather than generating exhaustive, unconstrained descrip- tions. Current benchmarks, however, primarily assess descriptive comprehensive- ness while largely overlook eestructionffolloinng capabilities. To address this gap, we introduce IF- VidCa,, a new benchmark for evaluating controllable video cap- tioning, which contains 1, 1,400 high-quality samples. Distinet from existing video captioning or general instruction-ful folllwiinn-ollowingg benchmarks, IF-VidCap incorporates systematic framework that assesses captions on two dimensions: format cor- rectness and content correctness. Our comprehensive evaluation of 26 prominent models reveals a nuanced landscape: despite the continued dominance of propri- etary models, the performance gap is closing. with top-tier open-source solutions now achieving near-parity. Furthermore, we find that models specialized for dense captioning underperform -prrpose MILMS on complex instructions, indi- cating that future work should simultaneously advance both descriptive richness and instruction-follooiing fidelity.
Equal contribution. 1 Corresponniig Author. hhtps://if-viccap.ittbb..

---

Published as a conference paper at ICLR 2026
INTRODUCTION
High-quulity, controllable video captions are crucial for range of downstream tasks, including structured captions for video generation (Ju et al. 2024), targeted descriptions for video edit- a stylistically appropriate copy for content creation Ye al. 2024). While recent Multimodal Large Language Models (MLLMs) excel general video description Yuaa al., Wang et al. 2025a), they often struggle to adhere to fine-grained instructions (Lian et al., 2025 that impose specific stylistic or formatting constraints. As illustrated in Figure [] this lim- itation highlights critical gap between their strong perceptual abilities and their weaker fidelity to complex user directives. We argue this gap stems from the fact that such tasks require not just perception, but sophisticated synthesis of reasoning and constrained generation.
ing QJang et a (025). and
2025
Existing evaluation for multimodal understanding, which primarily consists of question- answering (Fu et al. 2025 Zhou et al. 2025; Li et al., 2024) and captioning tasks (Chai et al.,
2025 Wang et al.; 2024 Chen et al. (025). lacks the sophisticated instruction instruction-folloiing following assess- ments common in language-only benchmarks (Zhou et al. 2023 Zhang et al., 2025c). Specifically, benchmarks on video captioning traditionally prioritize descriptive accuracy and comprehensive- ness, often neglecting practical constraints such as output format, length, and adherence specific content requirements or prohibitions. As controllable video captions for diverse downstream ap- plications become more prevalent, assessing a MLLM's capacity to fulfil varied structural and semantic instructions within a single generation has emerged as a critical challenge.
To evaluate fine- grriiraindd instruction following in video captioning, we introduce IF-Vad new benchmark comprising 1,400 instructions actoss over 13 video categories. These instructions in- corporate 27 distinct constraint types-such -ussh as output formatting, aspect restrictions, and element focus-ane and feature high compositional complexity, with an average of six constraints each. We im- plement the benchmark with an automated evaluation protocol that leverages high-quality annota- tions to assess both instruction fidelity and semantic quality, illustrated by Figure 2 Experiments on 26 state-of-the-an models demonstrate the benchmark's 's difficulty and discriminative power, re- vealing significant performance gap in which dlosed-soure models generally outperform their open-source counterparts. The main contributions are summarized as follows:
video captioning benchmark. We propose a new, high- quality benchmark, IF- VidCap, featuring 1,400 complex, compositional instructions aligned with realworld downstream applications. IF- VidCap could push the boundaries instruction- following evaluation for video captioning.
The first instruction-followe
A robust evaluation protocol to evaluate instruction fidelity and semantic quality. IF- VidCap is attached with a robust, multi-dimensional evaluation protocol that combines both rule-based and L.M-bbssed checks. The annotations on all samples within the evaluation have been carefully checked for coverage, correctness, and stability.
Discrimination of mainstream MILMSs and insights on the instruction-following video captioning. We observe significant performance disparities among current MLLMs, par- ticularly highlighting the poor performance of specialized captioning models when faced with instructional constraints. These findings suggest that the integration of captioning and instructionnfollooiing abilities could be a critical direction for future research.
A Training Dataset for Instruction-Follovinng To enable fine-grained, instruction-based control, introduce new training dataset that we have specifically curated. We use this data to fine-tune the Dwen2.5-VL-7B--Im achieves improved insttrcction-oolloiin performance over the base model.
model. The resulting model, IF-Captioner-Qwer
RELATED WORKS
The evaluation of instruction-ffllooinng for Large Language Models (LLMs) has matured significantly, evolving alongside the models' capabilities (Ouyang al. 2022 Early benchmarks focused on general task performance (Jiang et al. 2024: Qin et al.) (024b) or pro- grammatic verifiability (Zhou et al. 2023). More recently, the focus has shifted towards assessing adherence to complex, compositional instructions (Zhang et al. 2025c Wen et al. 2024) and spe-
cialized domain constraints (He et al.
2024 Qin et al. 2024a). This emphasis critical, training
2

---

Published as a conference paper at ICLR 2026
Rule-based Checklist
unordered list starting with
two main clothing items)
TRUE
Open-ended Checklist
the protagonist's doothing color and carredd items, without referring other attributes?
Does the response
Ci attire color
the unordered attempt static attributes
the main character (Including attire color??
shirt?
cheracter in the video usina unardered list starting with
static
Green, Blue, C. Brown, D. Cannot be determined
Focus (Cx ottine color) and (C. carnied items). For attire color, record CG. two main clothing items)
Q3
character's pants?
Pink, Gray, C. Brown, D. Cannot be deternined
items)
Instruction
unordered attempt list static attributes the main character (including carried items)?
attire color: light green top, dark gray pants carried items: bag with glowing blue part
What Item
have?
kettle with blue pattorn, A dagger with blue gem, A bag or board with blue light. D. None of the above
Generated Caption
Sample data in IF- VidCCp. The constraint types are color-coded, with colors correspond- ing to the categories in Figure 3d Our checklist divided into two types based on the checking method: rule-based items checked by LLM with rule scripts and open-ended items checked by LLM. The rule-based items cover format correctness, while the open-ended items cover semantic and content correctness.
Figure 2:
on such intricate instructions demonstrably improves model fidelity (Sun et al. 2024 Mukherjee cet al. 2023). However, this rigorous evaluation paradigm has remained largely confined to text-only tasks. Existing multimodal benchmarks, by contrast, tend to prioritize broad cross-tase task generaliza- tion (Bitton et al. 2023) over fine-grate fine-grained instructional diversity within a single domain. IF-Vid directly addresses this gap by introducing systematic, instruction-centric evaluation framework for the foundational task of video captioning.
Video Captioning Benchmarks. Traditional video captioning benchmarks designed to evalu- ate the semantic quality of generated text, not a model's adherence to arbitrary instructions. They typically fall into several categories: those targeting fine-grained descriptions (Chen et al.) 2025 Wang et al.] [2024] Zhang et al.) 2025b). those assessing structured or scripted outputs (Ju et al. 2024 Yang et al. (2024), and those focused on specific applications like content retrieval (Xu al. 2025 or long-form video understanding (Ren et al.) 2024). While crucial for advancing descrip- tive capabilities, these benchmarks share a common limitation: they evaluate against fixed set of quality criteria, such as accuracy or detail, rather than dynamic user-defined constraints. In contrast, Vi-Vid is the first benchmark in video captioning designed to systematically measure a model's ability to comprehend and execute diverse and compositional instructions.
IF-VIDCAP
To systematically address the challenge of controllable video captioning. our first step was analyze the instructional requirements of diverse downstream applications like video editing and content creation. From this analysis, we distilled comprehensive constraint framework encompassing 27 distinct types (Figure 3d), which serves as the blueprint for our benchmark and ensures its relevance to practical needs.
Based on this framework, IF-Vid is built Video- Iwwo--nsstutoonChheeeee high-quality videos, we employed hybrid generation cewruttioo-nnnd-eee reffrieeeeen pipeline to create the dataset. Each checklist features two components for evaluation: nule-based items to assess hard constraints like format and structure, and open-endee QA to probe semantic fidelity. This paradigm provides a comprehensive assessment of model's controllability.
triplet. After curating

---

Published as a conference paper at ICLR 2026
3.1 DATA COLLECTION
3.1.1 VIDEO COLLECTION
To construct high-quality evaluation benchmark, selected 350 videos build the test test set, began by assembling a large, copyright-free video pool from academic datasets and public platforms. Each video was then subjected filtering process, disqualifying content based on technical deficiencies (e.g., resolution below 480p. duration outside the 2-60s range), poor visual composition (e.g.. ex- cessive clutter), or data integrity issues (e-g., duplicates). The resulting collection offers extensive variety-from animation to natural aandsanpe--enuuring the benchmark is reliable, and free from common data quality pitfalls.
3.1.2 ANNOTATION PIPELINE
Our annotation pipeline employs two-stage workflow that combines automated generation with expert validation to achieve both scale and q
Stage 1:
Automated Draft Construction. An Instruction Generator produces cnstructtio-checklit pairs for each video. Subsequently, a suite of Response Generators creates multiple candidate cap- tions, which are assessed by an Automated Evaluator to provide preliminary judgments.
Stage 2:
Human Refinement and Validation. A team of professionally trained annotators metieu- lously refines the automatically generated draft, resulting in 83.6% modification rate. Consensus among all three annotators was necessary for a sample's 's acceptance; any conflicts in opinions were resolved by senior supervisor. This process yielded the final 1,400 high-quality samples.
3.2
DATASET STATISTICS
3.2.
OVERALL STATISTICS
Statistical analysis of the IF-VidCap dataset confirms its suitability comprehensive benchmark, distinguished by diverse duration, content coverage, and instructional complexity (Figure 33).. The dataset features uniform distribution of video durations, which are, on average, longer than those existing short-video benchmarks (Figure Ba). Its broad content diversity, spanning numer- ous categories (Figure 3b), facilitates robust testing of cross-domain generalization. Furthermore, instruction complexity ranges from standard prompts (3-8 constraints) to highly challenging cases (over 10 constraints), specifically designed probe compositional reasoning (Figure Be). Together, these characteristics establish IF-VidCap as a next-gen testhed for evaluating MI.LMs.
3.2.2 COMPARISON WITH OTHER BENCHMARKS
We adopt IFEval (Zhou al. 2023 CELLO (He al. 2024). InfoBench (Qin et al.] 2024b)
FollowBench (Jiang et al.] 2024). SysBench (Qin et al. 2024a)
CFBench (Zhang et al. 2025c)
and ConplexBench (Wen al. 2024) the instruction-follone
baselines. captioning. CapsBench (Liu et al. 2024), Dream-1k 1k (Wang et al., 2024) and CaReBench (Xu et al. 2025).As As
shown in Table IF- VidCap advances both Instruction Following and Captioning benchmarks. Un- like prior text-only IF datasets, it introduces the video modality while offering greater scale (1,400 avg. constraints), and broader content diversity. For video captioning, pioneers a shift from dense description to fine-grained instruction following, featuring more chal- lenging longer content with longer videos (20.5s). By bridging these domains, IF- VidCap provides more rigorous benchmark for evaluating controllable generation in multimodal models.
samples), complexity
3.3 EVALUATION PROTOCOL
3.3.1
EVALUATION CRITERIA
IF-VidCap evaluates two core dimensions-iinstuution following and video description qual- ity-employing LLM-driven methods ensure both flexibility and scalability.
"The detailed process for constructing the set is provided in Appendix D
4

---

Published as a conference paper at ICLR 2026
45-455
3-5
Objject
Arey
Crotered
12.09%
27.72%
Jummn Commmaa
Plain Text
Unordered
JOH
Frozidee
Comparatice
Excuate
20.57%
Exclude
Abstra casraction
Ste structura
Keyword
21.71%
Include
(a) Video duration.
Table
Focus
Praming
Tate
Grand
Perspective
Consent
Centent
Cinematic
Format
Markdown
Highight
Sparts,
Action
Element
Italis
Event
Stylotic
14.%%
film & TV
Attribtte
Marker
Delimiter
Video category,
Eneiyy
Language
Count
Profx
Suftkk
Length
Number Instruatioss
Care
Englidh
Character
Chimbe
Lower
Sensence
Word
Tebe
100
(c) Constraint count.
(d) Overview of IF- IF-VidCap constraint categories.
Figure 3: Dataset statistics for IF- VidCap. (a-c) show distributions for video duration, category, and constraint count, respectively. (d) provides an overview of the constraint categories.
Table Comparison of Instruction Following and Video Captioning Benchmarks. "#Size", "#Types", "#Const.", "Vid. Len.". and "#QAs" denote the number of samples, constraint types, average constraints per instruction, average video length, and average QA pairs per caption, respec- tively. "Mod." indicates the input modality (text/image/video), while "Evaluation" specifies the evaluation method (Rule-based, LLM-as-Judge, or a combination).
Benchmark
#Size
#Types
#Const.
Vid. Len.
#QAs
Mod.
Evaluation
Instruction Following Benchmarks
IFEval
541
25
1.54
Text
Rule
CELLO
523
4
2.18
Text
Rule
InfoBench
500
5
5.93
Text
LLM
FollowBench
944
5
3.0
Text
LLM Rule
SysBench
500
2.38
Text
L.M
CFBench
1,000
10-25
4.24
Text
LLM
ComplexBench
1,150
4-19
4.61
Text
LLM+Rule
Captioning Benchmarks
CapsBench
200
12.36
Image
LLM
VidCapBench
643
10.25s
16.55
Video
LLM
D
1,000
8.9s
Viden
LIM
CaReBench
1,000
14.35s
Video
LLM
IF-VidCap (Ours)
1,400
27
6.00
20.5s
16.95
Video
LLM+Rule
Instruction Following Evaluation: The evaluation paradigm has shifted toward using LLMs as
automated judges. Common strategies include pairwise comparison for estimating win rates, direct assessment for computing pass rates, and rule- alle-uugentted systems(Wen et al.) (024). The rule- augmented approach, which combines LLM--based semantic extraction with deterministic checks, is
5

---

Published as a conference paper at ICLR 2026
particularly effective for complex instructions, leveraging both the nuanced generalization of LLMs and the reliability of sule-based verification.
Video Captioning Evaluation: Evaluation criteria have evolved toward semantic fidelity. While structured matching (Chai et al. 2025 Wang et al.) 2024) and rubric scoring (Ren et al., 2025) outperform traditional metrics, they often fall short in capturing fine-grained details. In contrast, Question-Answering (QA) provides a more rigorous framework to validate such nuances.
Building the above insights, IF-Vid adopts composite mechanism for constraint satisfac- tion evaluation(Figure 2 For rule-checkable constraints, we combine an LLM with rule seripts: the L.M serves content extractor while the rule scripts act the verification executor. This integrates the LLM's adaptability to complex text processing with the determinism of rule execu- tion. For open-ended constraint checking, we design retrieval-based QA pairs, enabling the LLM to answer using the video caption context. Specifically, the checklist uses true/false questions to let the LLM directly judge the semantic correctness of the description, and multiple-choice questions to have the LLM select facts inferable from the video description. All answers provided by the LLM are compared against the ground truth, and statistics are aggregated at the constraint level (a single constraint may include multiple QA pairs to enable atomic checking and control over granularity).
3.3.2 EVALUATION METRICS
Our evaluation metrics are based on the following two main instruction-followigg evaluation met- rics:Constraint Satisfaction Rate (CSR) and Instruction Satisfaction Rate (ISR).
CSR
ISR
1=1
2=1
where of = if the j-th constraint of the i-th instruction satisfied, otherwise 0; 8; 1 if all con- straints of the i-th instruction are satisfied, otherwise O; m denotes the total number of instructions, and n, represents the total number of constraints for the i-th instruction. Additionally, based on the evaluation system of IF- IFdCC,, propose fine-grained metries:
Rule-Based CSR/ISR: Only considers format-related constraints, which reflects the model's ability to control output format.
Open-Ended CSR/ISR: Only considers content-related constraints, which reflects the model's ability to understand and describe multimodal content.
3.4 MODEL TRAINING
Furthermore, we introduce fine-tuning dataset designed to teach models generalizable instruction- following skills. Critically, its generation process is intentionally distinct from our test set's. We began by sourcing 11K high-quality video-caption pairs from datasets including Vript (Yang) cet al.) 2024), ShareGPT4Video (Chen et al.) 2024), VideoUFO-Gemini (Wang & Yang 2025). and ShareGPT-40 (Cui et al. 2024). For each pair, adopted "response-to-instruee treating the existing caption as a textual proxy for the video's ' content, we prompted DeepSeek- V3.1 to synthesize diverse instructions based on constraint framework This leverages the LLM's pre-trained knowledge to mirror varied, real-world task preferences. To prevent stylistic overfitting, the prompts used in this process entirely different from those for our test
approach:
Ultimately, this process yields 1 11K curated video-caption pairs, from which we construct a training of 46K video-instructioo-resppss triplets to fine-tune the Qwen-225-VLL-7-IInstte model.
'The complete details of the constraint framework are available in Appendix

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 MAIN RESULTS
We evaluate 26 popular models including Cemmni-2255-Prr/Fiie
TTam 2025
GPT-4o Ope-
nAI 2024), Intern IntemVL-33.5 (Wang et al. 2025a),
Dwwn2.55VVL-Instiee (Bai et al. 2025). VideoL-
LaMA3 (Zhang et al. 2025a), MiniCPM-VV4.5
Yao et al. 2024). Llama-3.2 Vision Instruct (eee
et al. 2025). LLAVA- V1.6- Vicuna (Liu et al.
2023), Video-LLaVe (Lin et al., 2024). ARC-
Hunyuan- Video KKng et al. 2025), MiMo- Team et al. 2025a), Kimi-VL, (Team et al., 2025b).
LLaVA- Video (Zhang et al.) (025d), VideoChat2 (Li et al.; 2024),IIneea Video2.5 (Wang et al.
2025b), GLM-4.IV (Team et al.[22222) and Tarsier (Yuan et al. (025).
The main results in Table 2 lead to the following key observations: (1) Performance scales with model size within the same family. (2) Top open-source models now rival closed-source counter- parts, (3) The "Thinking" mode's superior performance underscores the necessity of reasoning for complex tasks. (4) Instruction Satisfaction Rate (ISR) is inherently lower than Constraint Satis- faction Rate (CSR), it requires concurrently meeting all constraints. (5) Models exhibit stronger control over format (rule-based checks) than content (open-ended checks), likely because content re- quires complex multi-modal reasoning. whereas format is often text-based. (6) The human baseline exhibits distinct behavioral differences compared to advanced models. Since the human reference relies on 10 untrained undergraduates (whose independent results were merged), their performance in open-ended content description slightly trails behind top-tier models. Conversely, driven by hu- man mechanisms of careful checking and self-reflection, they demonstrate superior performance in rule-based format control, substantially surpassing all models. Interestingly, the positive effect of such reflection on enforcing format constraints is mirrored "Thinking" models, as their Chain-of- Thought (CoT) often incorporates explicit final check and correction of the format.
Additionally, we trained an IF-Caaptioner-Qua LL-7B-Instruct on our self-constructe datase(" We observe that Saptioner-Qwer outperforms Qwen2.5-VL-7B-Instte Instruuct by a large margin on both ISR and CSR metrics.
model, which was created by fine-tuning Qwen2.5-
4.2 FURTHER ANALYSIS
Effect of Different Instruction Metrics. We analyze the evaluation results of four representative models to investigate how Constraint Satisfaction Rate (CSR) and Instruction Satisfaction Rate (ISR) vary with instruction complexity, which we define jointly by constraint count and prompt length. Experiments are conducted on curated test set of 1021 high-quality instances, expert-selected from the full benchmark. In these instances, constraints exhibit dependencies (e.g., chaining, nesting, choice) rather than being mutually independent. This selection ensures that constraint count serves as a reliable proxy for overall complexity. The results (Figure 4a and 44b) clearly demonstrate that model's ' ability to satisfy constraints and adhere to instructions degrades as instruction complexity increases, confirming that models' instruction followinnn capabilities are significantly challenged by more intricate and difficult commands.
Effect of Different Video Parameters. We analyze the sensitivity of the Qwen2.5-VL. 2--2255---L--B-BB- VL-7B-Insttruc model to different input video parameters. Fixing the resolution at 224x224 and varying frame counts (8, 16, 32, 64, 128), ISR and CSR generally increased with more frames, peaking at 64 before dropping at 128 likely due to the model's limited capacity for very long sequences (Figure 4c). With constant f2-frame sequence, increasing resolution from 168x168 to 784x784 consistently improved both metrics (Figure 4d). Overall, the model benefits from richer temporal context and higher spatial fidelity, though excessively long sequences can harm performance.
Agreement Evaluation. To assess the alignment between our evaluation method and human judg- ment, use the professional annotations obtained in Section 3.1.2 as the human reference. Across the entire test set, we measured the agreement between automated evaluations from three differ- ent assessor models and the human evaluations. The assessor models include GPT-5-mini (OpenAl 2025). DeepSeek- V3. 13.1-NoTThin (DeepSeek-Al et al. 2025) and Qwen3-32B (Yang et al. 2025). In the overall agreement of the automated evaluation is strong particularly for the advanced
Table 3
*The detailed training process and hyperparameter settings are provided in Appendix
7

---

Published as a conference paper at ICLR 2026
Table 2: Results of different models. ? indicates that models support the "thinking" mode, and the scores split by "/" denote the performance of thinking and non-thinking modes, respectively.
Models
Overall
Rule-based
Open-ended
ISR
CSR
ISR
CSR
ISR
CSR
75.57
93.64
33,93
Closed Source Large Multimodal Models
Gemini-22.5PPP
27.83
74.53
74,35
87.81
35.22
59,00
Gemini-2.55 5.-aan
72.63
84,51
GPT-4o
22.90
70.74
69.20
85.12
30.94
53,91
Gemini-2.0-Flan
18.19
67.45
63.04
82.06
26.86
50.39
Open-Soure Large Multimodol
Qwen3 VI-Insimect
2350
26.41
71.65
67.16
84,14
36.79
57.12
71.17
83.21
VL-3.9
38B
20.71 / 15.43
68.30
64.76
39.43//5..0
00.17/78.92
31.79/ 24,93
54.42 48.20
Mimo- VL-SFT
17.72
66.33
56.2
78.14
28.96
$2.54
wwen2.5-VL-lnstten
72B
17.50
67.28
64.29
83.22
25.71
48.65
ILtemmV-33.59
/9.93
65.90/56.45
15.90/566 56.45
60.32 /48.14
79.95 /7.68
26.84 16.98
49.50 38.65
Qwen2.5- 5----tosttt
328
15.16
64,04
53,66
76.95
26.72
48,94
Lade9
12.21 10.63
57.38
/7.11
48.64
7169/71.22 77.21
71.69/
19.93
18.46
40.65 40.75
MiniCPM-V-4.5
11.75 /8.57
61.67
59.23
58.09
56.07
79.35
77.62
18.05
14.64
40.97
37.73
GLM-4.1V
11.46
$7.88
47.64
70.48
20.06
43.14
Qwen2.5-VL-lnstreer
7B
10.92
$8.12
52.51
73.81
18.75
39,65
Kimi-VT.
53.94
63.73
LLaVA, Viileo Vileo-Qwent Qwen2
7B
8.93
53.43
41.86
65.59
17.64
39.22
VideoChat2 Hii---DDD-g--iiiii -aattr-Misttal
Internvide22.5
7B
7.42
$1.34
41.43
65,57
13.54
35.16
Qwen2,5-VL Meen2.5-VL-lostruue
6.54
51,74
43,46
66,50
13.15
34,47
IJama-33 3.2. 3.2-Visoon Visisiom-Instmctt
90B
5.80
45.18
36.03
59.56
11.03
28.36
Llama-3, 2--Viison Instruct
11B
4.00
39.87
31.29
53.24
7.71
24.25
LLaVA-VI V...iia Vicuna
3.54
43.92
35.84
60.09
7.30
25.02
Vidco-LLaVA
7B
3.13
38.74
26.53
51.27
7.73
24.05
ARC-Hunyuan-
Tarsier2
7B
1.40
26.05
9.30
27.75
9.91
24.04
IV-Captiomer-Oe (Ours)
79.03
Score (%)
Score (%)
Comiini-22 5-mm
35-
Clwwe225 328 Iisouct
Owan22 V. 328 Instruct
cal tarta 2-118- don-indmet
Loma 3,2 Visin instuctt
20
Number Convrraatts
1-7
Instruction Length
(a) Constraint Count
(b) Instruction Length
65.0
12.0
60.0
62.5
CSR (h)
1%)
10.0
BR (%)
7.0
60.0
6.5
45.0
5.0
50.0
128
448 Resolutions
Prames
(c) Frame Count
(d) Resolutions
Figure 4: The Impact of Constraint Count, Instruction Length, Frame Count, and Resolutions Instruction hnstuuction-Following Capability (CSR and ISR).
proprietary evaluator (GPT-5-mini) which reaches 96.33% thereby validating the reliability of our evaluation pipeline. We also provide a scored-basee based consistency analysis fo CSR/ISR in Table 4 both open and closed-source models demonstrate high degree of consistency. For open-source
8

---

MILMMs in fine-grained descriptive tasks as Published as a conference paper at ICLR 2026
models, the agreement is slightly weaker yet still considerable, indicating that the evaluation method is broadly applicable across different types of models.
Table Agreement between automated evaluation and human evaluation across different models.
Model
Overall Agreement
Rule-based.
Open-ended.
GPTT-5-mini Been3-235B-A22--nssae
96.33
96,90
96.08
94.78
95.35
94.24
DeepSeek- V..1....11.... I-NoThink
92.18
93.55
91.58
Qwen3-32B
92.03
92.10
92.00
Table 4: Agreement between automated evaluation and human evaluation across different models with ISR/CSR
Overall
Rule-based
Open-ended
Model
ISR
CSR
ISR
CSR
ISR
CSR
GPT-5-mini
23.86
69.45
63.57
82.92
33.79
55.92
2reB--235BAA22B-- Instruct
22.53
69.36
59.39
81.25
36.35
58.02
DeepSeek- V3. -hoTink
22.50
68.95
56.21
78.04
35.36
57.28
Qwen3-32B
17.25
65.10
54.27
77.68
28.01
50.02
Human
23.89
69.57
63.00
82.64
33.93
55.82
Constraint Type Analysis. Our statistical analysis of the Constraint Satisfaction Rate (CSR) for a set of representative models across the entire benchmark (see Figures 5) reveals that current mul- timodal models exhibit distinct preferences when adhering to different types of constraints.
Overall, general- purposs models excel at format control but are weaker on content control, as the latter presents dual challenge of both multimodal comprehension and output regulation. In con- trast, specialized captioning models like Tarsier struggle with both format and content, tending to describe video indiscriminately. Notably, the primary performance gap between models of different scales stems not from formatting, but from their weaker handling content constraints. This sug- gests that the key bottleneck for improving instruction-followiii in general- purepoe models is the deep comprehension of multimodal information.
Furthermore, regarding format constraints, observe that JSON structure and length control con- stitute the primary hurdles for all MLLMs, with such precise and rigid constraints effectively prob- ing the boundaries of model capabilities. In terms of content constraints, Cinematic and Focus requirements prove similarly challenging: this highlights the visual models' deficiency in knowl- edge regarding camera movements and cinematography, as well their struggle to adhere to strict focus requirements. Notably, Video Detail Caption (VDC) models fail to achieve superior perfor- mance on our Detail constraints compared to their caneral-purpose This stems from the fact that our benchmark emphasizes detailed descriptions of specific attributes, actions, or events rather than holistic video descriptions, reveal- ing that VDC models still lag behind opposed to mere event enumeration.
counterparts (e.g., Tarsier2-77 7B
versus ---.2.5---7B-mmme
Error Analysis. Our analysis of model responses reveals several primary error categories. In terms of format constraints, frequent violations include those related to (1) length, (2) complex JSON structures, (3) incomplete typographic emphasis (e.g.. bold, italics), and (4) item counts. As for content constraints, common failures involve (1) difficulty with negative constraints (excluding keywords) positive ones, (2) mislocalization misidentification of attributes, (3) missing key information or inaccurate descriptions, and (4) incorrect camera perspective recognition. We also find that smaller-scale models are more prone to uncontrolled generation when processing samples containing multiple objects or continuous actions, highlighting a control deficit in handling complex video data. Figure shows two typical failure cases
Mooe concrete examples provided in Appendix

---

Published as a conference paper at ICLR 2026
16.1
11.1
12.8
12.3
01.0
76.7
66.3
11.8
76.2
75.0
18.
12.3
S 2764
16.4
X6.
16.3
7A8
22.3
710
754
174
16.
214
783
72.
162
96.1
VV.
71A
RCB
FAL
P8A
F64
FAT
164
76
RCD
PAR
KGS
62.8
10.5
PL-
M6.1
25.5
544
PL.
S2.0
15.6
60.0
64.3
55.2
06.5
46.5
454
SL7
LL
18.4
27.3
48.4
764
283
45.
11.
21.
22.6
25.7
16.1
424
527
11.1
17.4
Figure 5: CSR performance of different models on different constraint types.
Instruction
Instraction
Teeryibe the twn pedestrians waiking on the aldewacii
Cineertto a JSON army of people who took signiffcant action nahename and appearance.
50-70 words Bold thai cost colors co.. "Larososse"!
describe thiir actions dotai.
separate
physical featuree. exclusing the vorida "black" or "fat"
shouls descrics thar
Caption
Caption
Scene: peoksitianss wall um orte sidcnok
Choni' Tbey""
Hegluate cost and the rese dons g00y ceat
oppeorence" boy with spicy black heir, werring o grase shirt ")
Walk, sirkk wolk, Walk, Mork Wolk Walk Wark Wils Wark wookk.... Deattpple rxpatiiioe contect cnntee
"appasnannn": lnngesstaturad warning ortage tonk hop ")
"ppccnnnce"" girl weering blke slecediess dreak")
Error Analysia
Error Analysis
Fallure scone cesor pi wis
Worsi it wort "tlad" requined "nakename"" desciprart.
Insorvect
stort
X2 incomplete Bolding: I: ae to beic the word "ray" X3 Repstitive Langrage: loopped when descrting
3 Whing Subject Type: micudds get who pareen who iificrr
Figure 6: Examples of generated captions that do not meet the constraints.
5 CONCLUSION
This paper introduces IF-VidCap the first benchmark designed to systematically assess the ability of video captioning models adhere complex, compositional instructions. Our comprehensive evaluation of leading Multimodal Large Language Models (MLLMs) reveals a differentiated land- scape of their capabilities, showing that performance varies significantly across different types of constraints. We find that models specialized for descriptive captioning falter on constrained gen- eration, thus we suggest that rich description and strict control are capabilities to be developed in tandem, rather than as separate objectives. To catalyze progress, we release a new instruction- -uuning dataset and demonstrate that targeted fine-tuning is viable path to enhance instruction adherence.
ACKNOWLEDGEMENTS
This work was supported by the Beijing Major Science and Technology Project (No. Z251100000 1000084255)) This work was supported in part by the National Natural Science Founda- tion of China (No. 62506161).
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research was conducted in strict accordance with ethical standards for academic research. All data utilized in this study were sourced from publicly available datasets or collected with the in- formed consent of participants. To ensure participant privacy and confidentiality, all personally identifiable information was removed, and sensitive data were anonymized prior to analysis.
We affirm our commitment to fairness, transparency, and integrity, having carefully considered and mitigated potential biases in our data, modeling, and evaluation processes. The research intended solely for academic and scientific purposes, with no foreseeable harm to individuals, communities, or the environment.
REPRODICIBILITY STATEMENT
We strive to ensure the reproducibility of our research. All datasets used in this study are publicly available or can be shared upon request, and the data preprocessing steps are described in detail. The algorithms, model architectures, training procedures, and evaluation protocols are fully documented to allow independent replication.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng. Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: /ttp///axvvv./.. org/ abs abs/2502.22. /2s.//202..13 13923
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision-language instruc- tion following inspired by real-world use, 2023. URL aattps://arxiia wtt.o/gggabb////aaaeeeeag/
06595
Wenhao Chai, Enxin Song, Yilun Du Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D. Manning, Auroracap: Efficient, performant video detailed captioning and new benchmark, 2025. URL htttt:tt//// arxiv. org/ abtp://bs/22411 /2000.
03051
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving vidco understanding and genera- tion with better captions. arXiv preprint arXiv:2406.04325 2024.
Xinlong Chen. Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chen- gru Song, Qiang Liu. Di Zhang, and Tieniu Tan. Vidcapbench: A comprehensive benchmark of video captioning for controllable text-to-video generation, 2025. JTR.hhttss://arxiiti
abs/2502. 12782
Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang. Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URL https:
/ //sharegpt4o.ggr wtt.:ppt.....gttu...... github.
DecpSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li. Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo. Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan
11

---

Published as a conference paper at ICLR 2026
Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang. Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma. Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang. Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong. Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang. Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL https: /wtp/aaaivvvv..///////e../a.aav. org/ abs 2412 2124111.9993 19437
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever compre evaluation benchmark of multi-modal llms in video analysis, 2025. URL
http::://axxiv... https://axxiv.or//aap/ abs 2405.2200 21075
Qianyu He, Jie Zeng. Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang. Yuncheng Huang. Haoning Ye, Zihan Li. Shisong Chen, Yikai Zhang, Zhouhong Gu, Jiaqing Liang, and Yanghua Xiao. Can large language models understand real-world complex instructions?, 2024. URLptttss: / www.. crg/ abs / 2309. C9150
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints follow- ing benchmark for large language models, 2024. URL https: wwtp:iiww.rg/bb//231110 20410
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. 2025. BtLlhttpss//arxii/ org/ abs//bas/./////////////// 07598
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions, 2024.
/ww.vv/../v.v///////a//a./..aav. org/oogg//......ooooo...e 2407 00000000 06358
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, An- dong Wang. Changlin Li, Duojun Huang, Fang Yang. Hao Tan, Hongmei Wang. Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhen- tao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Song- tao Liu, Dax Zhou, Hongfa Wang. Yong Yang, Di Wang. Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025. URL http:s://arxiv.ooo wwwpwiiwwwggg.oo///oo/o 2412. 03603
Jewon Lee, Ki-Ung Song, Seungmin Yang, Donguk Lim, Jaeyeon Kim, Wooksu Shin, Bo-Kyeong Kim, Yong Jae Lee, and Tae-Ho Kim. Efficient 1lama-3.2-vision by trimming cross-attended visual features, 2025. URL httph:tttt/stttt
Kunchang LI, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modas video under- standing benchmark, 2024. URL htt.hhtts::/ pattps://arxiin wttp:/wwwwwr///////////// 17005
12

---

Published as a conference paper at ICLR 2026
Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video caption- 2025.
ing. arXiv preprint 2Xiv:2504.16017
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2024. URL https://arxiv: org/ hoe/abs//e 2311. 10122
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to- image alignment with deep-fusion large language models, 2024. URL http::tpps//// httss///xxxvvorr///
abs 0003/2409.00 10695
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL https://arxxn org/ abs 2304.0000000 08485
Wei Liu, Yancheng He, Hui Huang, Chengwei Hu, Jiaheng Liu, Shilong Li, Wenbo Su, and Bo Zheng. Air: Complex instruction generation via automatic iterative refinement, 2025. URL
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
(tt/ssts:://arx org/ www.b///..... 02707
OpenAl. Gpt-4o system card, 2024. URL http:://arrii..ogg//a/////////
OpenAl. Gpt-5
system card, 2025.
URL https://cdn.openn comee/
gpt- 50c-5-systee
pd
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL htt.:httt::/seett //apxvv////vvvvvo/ org/ abs 2203. 02155
Yanzhao Qin, Tao Zhang, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, and Bin Cui. Sysbench: Can large language models follow system messages?, 2024a. URL
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models, 2024b. https::s::////// /ttp///aavvvvvaa///a//a./aaaa wwwpxiiv.orraeeoooo /ttp:/bb///2233353
Weiming Ren, Huan Yang, Jie Min, Cong Wei, and Wenhu Chen. Vista: Enhancing long-duration and high-resolution video understanding by video spatiotemporal augmentation, 2024. URL
00927
Yiming Ren, Zhiqiang Lin, Yu Li, Gao Meng, Weiyun Wang, Junjie Wang, Zicheng Lin, Jifeng Dai, Yujiu Yang, Wenhai Wang, and Ruihang Chu. Anycap project: A unified framework, dataset, and benchmark for controllable omni-modal captioning, 2025. https://arxxiv
abs/ abs/2507, 2507.1284I 12841
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained fnslowctionnfollooinng ability of large language models, 2024. URLpttps: //
22.44444...0 0244 02823
Core Team, Zihao Yue, Zhenru Lin, Yifan Song. Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song. Zhenbo Luo, Yue Yu, Yudong Wang. Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang, Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong, Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen,
13

---

Published as a conference paper at ICLR 2026
Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang. Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, and Bingquan Xia. Mimo-vl technical report, 2025a. URL httph:tttt//aaae http//awwww....raa/a.oo/ wtw.oogg//bbba///oeo 2506. 03569
Gemini 2.5 Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodallity, long context, and next generation agentic capabilities, 2025. URL
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung. Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang. Hongcheng Gao, Huabin Zheng. Jiaming Li. Jianlin Su. Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng. Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He,, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingeheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen. Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du Yuxin Wu, Yuzhi Wang. Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang. Zhiqi Huang. Zihao Huang, Zijia Zhao, Ziwei Chen, and Zongyu Lin. Kimi-vl technical report, 2025b. URLttttss: / rxiv. org/ abs 0/1///55444049.99
Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi. Junhui Ji, Lihang Pan. Shuaiqi Duan, Weihan Wang, Yan Wang. Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang. Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng. Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang. Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang. Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and glm-4. 1-ttinkinn: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025c. URL https: //tp:vaaavvvvv//a org/ abs 2207 01006
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models, 2024. URL 00634
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang. Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo. Wenwei Zhang, Songyang Zhang, Maosong Cao. Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu. Huanze Tang, Rui Wang. Haijun Lv, Wanli Ouyang. Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internv13.5: Advancing open- multimodal models in versatil- ity, reasoning, and efficiency, 2025a. UtL
htcp//bs/25550 18265
Wenhao Wang and Yi Yang. Videoufo: A million-sscale user-foouuee dataset for text-to-video gen- eration, 2025. URL http://s//aaaii.orr////////////0
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and
14

---

Published as a conference paper at ICLR 2026
Limin Wang. Internvide02.5: Empowering video milms with long and rich context modeling, 2025b. URL htttttts//aaiiii.. org/ abs/ 1238
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. URL hnttps://arxiv.rrrrt
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction-followwe
with multiple constraints composition, 2024. URL https://
arxiv.org/ hbsp:/bb//240000 03978
Yifan Xu, Xinhao Li, Yichun Yang, Desen Meng, Rui Huang, and Limin Wang. Carebench: A fine- grained benchmark for video captioning and retrieval, 2025. URL htttp:://axiv..or
abs/2501 00513
An Yang. Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025, URL wttttttss//aaatte org/
09388
Dongjie Yang. Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video worth thousands words, 2024. URL https://arxiy: org/ abs/ 2406 06040
Yuan Yao, Tianyu Yu, Ao Zhang. Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm your phone, 2024. URL
abs/2400 01800
Zilyu Ye, Jinxiu Liu, JinJin Cao, Zhiyang Chen, Ziwei Xuan, Mingyuan Zhou, Qi Liu, and Guo- Jun Qi. Openstory: large-scale open-domain dataset for subject-driven visual storytelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7953-7962, 2024.
Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large vision-language models from detailed video description comprehensive video understanding. 2025. URL http:t/s///aav..oooooo////////// 07888
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Vidcollama 3: Frontier multimodal foundation models for image and video under- standing, 2025a. https: //wpvv.a.vvvv//////////.a./aivvv org/ abs/2201.13166 13106
Shi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li, Xiaobin Zhu, and Xu-Cheng Yin. Vcaps- bench: large-scale fine-grained benchmark for video caption quality evaluation, 2025b. URL
https::/arxiv.orgg/ wttp://w.wrra/bb////o/o
23484
Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, and Zenan Zhou. Cfbench: A comprehensive fonstraints-folooiing benchmark for 1lms, 2025c. URL https:
7/arxiv. www.vo.rggaaa abs 01122
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava- video: Video instruction tuning with synthetic data, 2025d. URL https: htttps://axxiv //trrii..voo.. abs/ /21//410000 0..00.00 02713
15

---

Published as a conference paper at ICLR 2026
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovee evaluation for large language models, 2023. URL https:
/arsiv..rrg/ab//31111.00 07911
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi-task long video understanding, 2025. URL /tttps://arxiv.ott abs
04264
16

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE DIRECTIONS
While our automated evaluation protocol shows high agreement with human judgment, it can stum- ble on semantic nuances, particularly with highly abstract or subjective constraints. Future work could explore more advanced evaluation methods to better capture these subtleties.
Furthermore, we acknowledge a limitation regarding the data distribution. Our training set is constructed using "caption-co-instrattoo"" paradigm. While this method-ccoceeptualll conceptually analogous to established text-domain frameworks like Self-Instruct (Wang et al. 2023 and AIR (Liu al. 2025) is pragmatic and necessary approach given the current absence of large- scale authentic user queries in the video captioning domain, there remains a potential distributional gap between synthetic data and real-world user instructions. To mitigate this, we leveraged advanced generative models (DeepSeek-V33) V3.1) to simulate diverse user personas and implemented rigorous human quality control process, where our data achieved a high naturalness score (4.63/5.0) Despite these efforts to ensure linguistic diversity and plausibility, a purely synthetic approach may not fully capture the long-tail distribution of complex human requests. We view our current work as critical foundational step, paving the way for future research to utilize real user logs once such resources become more accessible to the community.
Moreover, our benchmark primarily focuses short videos to ensure that the evaluation of the model's s constrained descriptive capabilities is not conflated with the challenges of video duration. Nevertheless, extending this framework to longer video understanding- where temporal reasoning and summarization under constraints become even more critical-is undeniably a promising direc- tion. We hope that IF- VidCaa will encourage the development of next-generation MLLMs that are not only powerful perceivers but also faithful and controllable assistants.
B REAL-WORLD APPLICATIONS OF THE INSTRUCTION CAPTION
The advancement of multimodal video understanding models has spurred the adoption of instruction-following video captioning. a task that involves generating textual descriptions tailored to specific, predefined constraints. Unlike holistic video summarization, this targeted task critical for range of applications. We elaborate six key real-world applications:
Video Editing: For tasks like object addition, deletion, or modification upon reference images, captions must precisely describe only the intended change prevent erroneous alterations to the scene. Otherwise, unexpected items would be impacted on the edited video. In this case, the video caption must not imply redundant content.
Video Generation: Models require aspect-specific descriptions gain granular control over static attributes, such as object appearance, which generic event-level captions cannot provide. In this case, caption must be comprehensive from the specifie aspects.
Script Generation: Captions must follow a structured, chronological, and scene- scene-by-sceme by-nee format ensure narrative coherence. In this case, both the format and comprehension are necessary for the video caption.
Anomaly Detection: In surveillance, captions selectively focus on predefined subjects of interest, omitting irrelevant activity to reduce signal noise and improve the detection of salient events. In this case, only the caption on the key elements is acceptable.
Autonomous Driving: Systems require concise environmental descriptions facilitate rapid situational awareness and decision-making, where verbose details would introduce processing overhead. this case, short caption on key elements desired.
Industrial Pipelines: For automated content analysis, captions must adhere strict data schema (e.g., JSON) to ensure interoperability and prevent parsing failures in the workflow. In this case, the format of the caption must be strictly followed.
17

---

Published as a conference paper at ICLR 2026
C PROMPTS
C.I
TEST
Initial Prompt for Model Query.
As professional video describer, your task is to provide accurate controlled video caption based on instructions and imput from the video. The video sampled a rate of 2 frames second. Strictly follow the instructions without adding any opening remarks, closing statements, or additional explanations. Here are your instructions:
This text serves as the initial prompt for the model and will placed the very beginning of the input query. The frame sampling rate specified in the prompt may vary depending the model, and all video input parameters, including frame sampling, are provided Appendix G
C.2 JUDGE
Content extraction for rule-based check items and question answering for open-ended check items are both performed using OpenAl's gpt-5 mini-mm025--0--0000 The specific prompts are as follows:
The prompt for based content extraction.
You are highly accurate and strictly rule-following content extraction program. You need to extract the correct content to be checked for a given rule-based checking function. Your sole task is: based on the requirements and related instructions of the given checkitem, extract content from the input response, and configure the content parameter in the checkitem. Input Information:
You will receive a JSON object containing the following two key pieces of information:
response: string, the source from which extract content.
checki tem: JSON object, which specifies the check type and content for which you need to perform the content extraction.
Checkltem Structure Description:
The format of the CheckItem you receive is as follows:
scheck_id* "string", // The unique econstraintt_d"" "string", // The constraint type ID "check_descripticn" description" : "string", // The check description
the check item
"check
"parameters":
"content": mull, Awaiting content extraction // ... Copy other dynamic specific parameters
constraint.id Type Description:
The following explains each constraint type and the execution logic of the corresponding rule script:
text: :liittext:: Plain text check. The script determines if content contains special structural symbols.
plain
json.object: JSON object check. The script determines if content conforms the JSON structure specified by the schema.
json array: JSON array check. The script determines if content conforms to the JSON structure specified by the schema.
unordered isst: Unordered list check. The script determines if, after splitting content by newlines, each element is paragraph starting with the symbol.
18

---

Published as a conference paper at ICLR 2026
ordered_11 i st:: Ordered list check. The script determines if, after splitting tent by newlines, each element is a paragraph starting with the symbol and its in- crementing counterpart.
table: Table check. The script determines if content satisfies the table syntax and has the column names specified by col_ _ame.
keyword: Keyword check. The script checks whether the content includes or excludes specific fixed keyword string.
ma markdown: Markdown decoration syntax check. The script determines if the prefix and suffix of content satisfy the specified markdown decoration syntax (e.g., for bold).
prefix_suffin Prefix and suffix check. The script determines if the prefix and suffix of content satisfy the specified parameters.
del immmee:: Delimiter check. The script determines if content contains the de- limiter specified in the parameters.
length: Length check. The script determines the length of content, as divided by the unit specified in the parameters (character, word, sentence, paragraph), meets the specified range.
count: Count check. The script determines if the number of objects enclosed in () in content meets the specified range.
case:
Case check. The script determines if content meets the specified case (e.g., uppercase).
language: Language check. The script determines if content meets the specified language.
Special Notes:
1. content is a list of strings, where each element treated as an object for one rule check execution. The script will execute the check corresponding the con- straint_id for each element in content and ultimately fill the result parameter with the logical "AND" of all check results. This design is handle multiple pieces of content for the same check parameters that are not continuous in the response (interrupted by irrelevant content other than newlines).
2. For the count constraint check, each element of the content list (i.e., representing a check on one continuous piece of content) should be string containing multiple objects separated by '0' and '', for example, content:[ (nntentt"((a,,bb)....) (c), where a, and с are all objects as required by the prompt, which could be a word, a few words, or even sentence.
3. Note: When extracting the count content, semantic alignment must be considered. If the content in the response is clearly inconsistent with the semantics required by the count constraint, then the extraction should be left empty. At the same time, the number of extracted items should reflect the actual quantity present in the response, rather than the parameters specified in the checkitem. For instance, the response contains 5 items but the checkitem expects 4, all 5 items should still be extracted for an accurate evaluation.
4. content must be fully and accurately extracted from the response to be checkable by the script. The corresponding script checking mechanism must be fully con- sidered during extraction. For example: if checking for bold, the **** prefix and suffix should be included (unless they are absent); if checking the language, special characters (like (}) should not be included; checking for prefixes and suffixes, be careful not to add content to the beginning or end of content that was not in the response.
19

---

Published as a conference paper at ICLR 2026
5. When extracting content, pay attention the applicable scope of the checks. For example, if checking the length or language of a list item's content, avoid extracting the list's starting symbol (e.g., -, A. prevent it from affecting the inspection.
Output Specification:
You must return a single, valid JSON object, which is the content you actually extracted. must not contain any additional explanatory text.
"content": ["string"]
// The content you actually extracted
Positive and Negative Case Comparison Example I: response: "Here the video descrip- tions: n n n A. Screens on posts change from green to red. (. Car flips over." checkitem:
check_id"; "rulc-002",
constraint_idd" "ordered
shheck_description.. "Describe the two distinet state + changes shown on the screens of the yellow posts using ordered 1ist starting with 'A.'." "parameters";
"content"; null, "symbol": "A."
Correct Extraction
"content"; ["A. Screens on posts change from green to red. AnB. Car flips over."]
Incorrect Extraction
"content": ["A. Screens on posts change from green to 4 red.", ", "3. Car flips over.":
Reason for Error: ordered list check should treat the entire continuous ordered list as single element, not as multiple elements from the list. Example 2: response: "jjon' nn "title": "Link rumning towards distant volcano, Breath of the Wild scene.". An "tags": (n "character attire": "Green tunic, beige pants", A'n "action": "Running", "landmark": "Volcano" (n),n" checkitem:
"check_ion ;d"; "rule-001",
"constraint_ion id"; "ison "ison_object",
"check deseriptiinn "Output JSON object that must 4 contain two keys: 'title' and "tags".. ",
"parameters";
"content"; null, " "schema":
"type": "object",
"properties":
"title":
"type"; "string"
"tags": {
"type": "object",
20

---

Published as a conference paper at ICLR 2026
"properties":
"character_attire: "type": "string"
" "action": "type": "string"
b "landmark"; {
"type": "string"
"required: [
" "characti "character_attiree"
"action", " "Landmark"
required"; "title", "tags"
Correct Extraction:
"content": ["(nn \"title)"; \"Link running towards
S distant volcano, Breath of the Wild scene. An
7 \"tagsi": (\n ("character_attieey \"Green
tunic, beige pants\", \"ction)"";
("Running) An \"landmark)"; \"Volcano) "In
Ann)"]
Incorrect Extraction:
"content": ["title, tags"]
Reason for Error: A JSON type check must extract the entire relevant content completely and then hand over to the script for checking. You only need to identify which content is relevant.
Example 3: response: "Here are the video descriptions: p h Carkn \n Screens on posts change from green red. i B. Car flips over." checkitem:
"check_id"; "rule-001",
"constraint_id". "count",
"chheck_@escriptiinn" "pescribe the two distinct state changes shown on the screens of the yellow posts + using an ordered list starting with 'A.
"parameters":
" "content": mull, "min_count": 2,
"max_count";
Correct Extraction:
21

---

Published as a conference paper at ICLR 2026
"content": [" (Screens on posts change from green to F red.), (Car flips over.)
Incorrect Extraction:
"content": I"A. Screens posts change from green to red. An Car flips over.")
The check type is count, which focuses on the 'two distinct state changes' in the constraint description, not the structural check of the ordered list. Example 4: respone: "The video begins with a view of a drawer containing various items. hand picks up white power bank labeled 'WOPOW" from the drawer. The power bank is shown up close with animated hearts around it, emphasizing its cute design. The back of the power bank opened reveal the charging cable, which then pulled out. Four power banks in different colors (green, white, beige, and purple) are displayed on table. The video shows the power bank's small size and 10,000mAh capacity, comparing tissue box. The power bank's 10,000mAh capacity is reiterated with close-up of the label. Finally, the power bank shown charging two smartphones simultancously" checkitem:
Reason for Error:
"check_ioe _ "rule-001",
" "constraint_id": "count",
"check_dess deteecckddscrrptiinn" "The table must contain exactly + rows (excluding header) "parameters":
" "content": null, "min_count"; 3, "max_count";
Correct Extraction:
"content": [""]
Incorrect Extraction:
"content": (a white power bank Labeled WOPOW ) (a 4 built-in charging cable), , (10, 000mAh capacity)
The check type is count, and the check description requires locating rows in table. However, since the original description contains table all, the target cannot be identified, and an empty extraction should be performed. Example 5: respone:"person person B, person C, person D" checkitem:
Error cause:
"check_id"; "rule-001",
"constraint_iie _id;: "count",
" "check_deer descripticn": "describe 3 person", "parameters":
"content": null, "min_count"; 3, "max :ount"ount";
22

---

Published as a conference paper at ICLR 2026
Correct Extraction:
"content": + D)
(person A), (person B), (person C), (person
Incorrect Extraction:
"content": [" (person (person B), (person C) "]
Reason for Error: When extracting, all items should be retrieved, disregarding the require-
ments of min and max count, in order ensure accurate evaluation. Next, carefully read the actual input below and perform the processing that conforms to the conventions and reasonable:
The prompt for open-ended content evaluation.
Roles and Goals:
You are a Video Caption Evaluator. You are required to perform specified checks on a provided caption. Based on the specific question and the reference description of the video's content, you will judge the response in two types. Input Information:
You will receive a JSON object containing the following four key pieces of information:
prompt: A string, providing instructions to generate response from the model.
response: A string, which is the caption from the model being evaluated.
question: string, which is the question that needs to be answered.
options: A list of strings, which are the available choices for you to select from (single choice).
Task Description:
You need to answer the question based on the actual content of the response (choose the answer that is closest the options). After providing your answer, you fill in the result_exy, anaatinionnn ion field with explanation and the result.confi fence field with confidence score (on scale of [1-5]).
Output Specification:
You must return single and valid ISON object, which your completed answer and expla- nation. It must not contain any additional explanatory text.
"answer": "string", // The result you provide. 4 'ptiions'
is multigle-choice question, this will be A, B, C, or D. If
it is true/false question, it will be 'yes' 'no'. "result__eplaantiion" "string", Your explanation. "integer" // Your confidence score.
"result_confideee
Next, carefully read the actual input below and perform the processing that conforms the conventions and is reasonable:
23

---

Published as a conference paper at ICLR 2026
C.3 CONSTRUCTION PROMPTS FOR THE TEST SET
In this section, each actual prompt consists of the prompt shown below plus the constraint framework table in Appendix E serving the actual content of the Core Knowledge Base.
The prompt for rule-based checklist generation.
Roles and Goals:
You extremely rigorous and logical checklist constructor. Based the user-provided prompt, constraints _used, and anference_captionn you need to generate a detailed, precise evaluation checklist JSON object that fully adheres all the rules below.
Core Knowledge Base:
You must strictly follow the knowledge base named 'constraint framework'. This knowledge base defines the technical details and parameter tables for all constraint.ids.
Input Information:
You will receive a JSON object containing the following two key pieces of information:
prompt: A string, which contains the constraint instructions for the model under test.
constraint s_used: An array of strings, which lists the unique IDs of all rule- based atomic constraints used to generate this prompt.
Core Task:Generate Checklist
You must strictly follow the process to generate the content for the www.daaaeeehheeeeee sec- tion.
1. Iterate through the input constraints used list. For each formatting or structural constraint
(e.g., jaonjssojeeeeee length, unordered etc.), create check item.
2. ID and Description:
check_id: Sequentially generate a unique ID, starting from "rule-001".
constraint.id: Copy this directly from the constraints list.
deecck.description Precisely extract the instructional description related to the current constraint id from the prompt.
3. Parameter Extraction
Locate:
First, locate the descriptive sentence or phrase in the prompt text that directly corresponds to the current constraint
Extract: Then, precisely extract the parameter values from this located phrase. For example, for length constraint, you should first locate the phrase "no more than 150 words" and then extract "max": 150, 'unit": "word" from it.
Assign: Assign the extracted values to the corresponding parameters. The value for the content key must always be null. For JSON objects and lists, their schema parameter must conform to the JSON Schema specification.
Output Specification:
You must return single, valid ISON object that represents the complete structure for ruled based check. It must not contain any additional explanatory text.
"ruled base_a__eecdkk"" 4 items
// List of rule-based check
check id": "string", // Unique for the rule
a check, e.g., "rule-001"
"constraint_id" "string", // The corresponding
4 original format constraint ID, e.g., "Length"
"check_ debeekkdescrrption "string", // Description 4 of the check item
24

---

Published as a conference paper at ICLR 2026
"parameters"; // Parameters to be checked, 4 extracted from the prompt "content"; null, // Content is left null, to be
+ extracted by the Check model |/ ether dynamic and specific parameters, e.g., "max": 150, "unit": "word"
Concrete Example:
The following is complete example, including input and expected output, for your refer- ence. Input:
"prompt":
"Please output JSON object, which must contain the keys 'summary' and 'key _ey__actios'' The
value for the 'sumnary' key should be a video summary of no more than 30 words. The value for
the 'key_actions' key should be an unordered list using '-r as the bullet point, listing all key 4 actions the main character.",
cconstraints__sed""
" "isson_bjjett,, "Length", "unordered. list"
Expected Output:
baseedd_based_che
"check id"; "rule-001",
"constraint_id"; "json_object", cchecc_desse check_description" "Output JSCN object,
4 which must contain the keys 'sumarry' and + "cyyonccions'.",
"parameters":
"content": mull,
"schema":
"type": "object",
"properties"; " "summary": { "type": "string"
b "key actions": [ "type": "string"
"required": [ " "summary" "key actions"
"chec_iid", "rule-002", "oonstraintjid.: "1ength",
25

---

Published as a conference paper at ICLR 2026
"check "check_descriin : "The value for the ''ummary' key should be a video summary + of no more than 30 words.",
"parameters":
" "content"; null, "unit": "word", mmxx_enn": 30
"chec_idd" "rule-003",
"constraintiid" "unordered_list
Scheck_descriptionn "The value for the
'key_actions' key should be unordered 4 1ist using f-r as the bullet point.", parameters";
"ccntent"; mull, "symbol"; n_"
Please strictly adhere to the Checklist construction philosophy and complete the task based on the input.
The prompt for open-ended checklist generation.
Roles and Goals:
You are an expert in evaluation questionnaire design, proficient in evaluation methodologies. Based on the original video and the user-provided prompt and ruled_ seedd__aee__eee _cecd,chck,, you need to generate a detailed JSON object for an evaluation checklist that fully adheres to all the rules below. This checklist will include a series of discriminative tions to assess a caption generated by the model under test, based on the requirements of the prompt.
comprehension ques-
Input Information:
You will receive two inputs: Original Video: The visual basis for fact-checking. 2. A JSON object, which includes:
prompt: A string containing the constraint instructions for the model being tested.
ru ruled ed based check: A list of constraint items from the prompt that have al- ready been verified by rules. You do not need to check these items again. (The beed_bbaed_chhe covers all format, structure, and keyword issues that can be au- tomatically judged by code. Your task is to focus on aspects that require semantic understanding and factual judgment in conjunction with the video content.)
Core Task:Generate Checklist
You must strictly follow the process to generate the content for www.eeeddddeeheeeeee 1. Con- tent Decomposition: First, break down all content and semantic requirements from the prompt into multiple independent check_content. Each check_content should be a concise summary specific task, for example:" of the protagonist." And single check content can only check one constraint item, and can- combine multiple independent constraint items. For each decomposed check content, you must strictly follow the decision-making process below to determine which check_jtent jtems
video summary" or "List all key actions
26

---

Published as a conference paper at ICLR 2026
generate: 1. Is attempt check needed? (The attempt type is yes/no question that only cares if the model tried to execute the instruction, not whether the content is correct.)
Condition: Check the core intent of the constraint has been fully covered in ruled based check.
Yes: If it is covered (e.g., the prompt requires a JSON object, and its at- tribute correctness and structural validity are already checked by json_object in ruled bale__based_chekk then do not generate an attempt check item.
No If it is not covered (c.g., the prompt asks to "generate a summary," which is non-rule-based requirement). then you must generate an attempt check item. The question should focus on whether the model attempted to execute the instruction, for example: the content appear to video summary?"
Is correctness check needed? (The correctness type multiple-choice question aimed at examining the degree to which the model describes video facts according to the prompt's requirements.)
Condition: Check if the correctness of the constraint can be objectively judged directly based on the response.
Yes: it can be judged (e.g., the prompt requires "do not mention color"), then do not generate correctness check item.
No: If it requires combining specific facts from the video, generate fine-grained questions.
3.question Design Principles
Atomicity Principle: Each question can only test single, minimal, inde- pendent fact.
Correct Example: "Who the protagonist in the video?"
Incorrect Example: "Who is the protagonist, where they, and what they doing?" (This question contains multiple scoring points).
Opt-out Principle: For most questions, in addition to one correct option
and two reasonable incorrect options, you can set "None of the above" or "Can- not determined" opt-out option.
The content of the questions should be based on the requirements of the generated prompt. Check the following indicators of the de- scription:
Granularity Design:
Correctness: The content of the description should be accurately present in the video and meet the prompt's descriptive requirements.
Completerse emppeetenes:: When the prompt requires a complete set, design questions to confirm whether all necessary elements are included. For example: "Which of the following options fully lists all the actions?"
iiccyssiiity: When the prompt requires a focused set, design reverse ques- tions check whether unnecessary elements are excluded. The specific steps are as follows: 1. Create distractors: Based on the video content, create several facts that exist
reasonable but do not meet the prompt's requirements as distractor options. 2. Design the question: Your question should be something like: "Which of the following objects/actions (depending the check content) is mentioned?" 3. Design the answer: The correct answer should be "None the above are men- tioned."
Note that the above checks also apply granularity control (for example, for sum- maries, check whether unnecessary details are included, and for detailed descrip- tions, check whether important details are omitted).
4. ID and Sorting
27

---

Published as a conference paper at ICLR 2026
check_id
_ick d Sequentially and uniquely increment the ID starting from "open-001" across all www..ended_cheeeeeeeeeeeeeeeeeee checks.
Internal Sorting : Within any single check_content, both attempt and cor- rectness check items exist, the attempt item must precede the correctness item(s).
Output Specification:
You must return a single valid JSON object that is the complete structure of the Checklist, without any additional explanatory text.
open ended c
"string", // The check content, 4 extracted from the Prompt "check_items":
"check "check_id"; id": "string", // Unique for the ( open-ended check, e.g., "open-001" "check_type"; "attempt correctness", // + Intention/Acouracy check
"question" : "string",
The check question // A list of options for 9 ccrrecttness', or ['yes', 'no'] for 4 'attempt'
"options":
"A. Option text 1",
"B. Option text 2",
"c. Option text 3",
"D. Option text 4"
"correct_answer" "string" // The correct 9 answer, A, B, or D, or yes or no
Specific Example:
The following complete example including input and expected output for reference. Input: Assumed Video Content: A man with beard, wearing blue apron, is making coffee in kitchen. The video shows close-ups of following steps: first pours coffee beans into an electric grinder. Then he uses espresso machine to make shot of espresso, pouring into a white mug. Finally, he steams milk with a steam wand and pours into the coffee, creating simple heart-shaped latte art. Throughout the process, the man has focused and satisfied expression.
"prompt": "Please generate briet summary for this video, no than 50 words. Then, in unordered 1ist, + list all the main tools used by the protagonist 4 the video. Ensure the output does not contain the 4 word 'beverage".",
" "ruled_based_chec""
"Please generate brief summary for this video,
4 more than words", "In an unordered list, list all the main tools used by 4 the protagenist in the video", "Does not contain the word 'beverage'"
Expected Output:
28

---

Published as a conference paper at ICLR 2026
"open_ _ndee_
"check_content "Generate video summary", "check_items"
beneck_id"" "open-001",
"check_tyye :ypeck "attempt", "question"; "Docs the description look
4 like a video summary?", "options": "yes", "no"
pnprreett_nsweer "yes"
"check_id"; "open-002", "check_type"; "correctness",
"question"; "Who is the protagonist of ( the video?",
"options": "A. A man",
"B. Coffee",
"c. A mug",
"D. Cannot be determined"
"correcttanswer"" "A"
"check id" "open-003",
"check "check_type"; "correctness", "question": "Which of the following
+ descriptions of the protagonist's
4 attire is correct?", "eptions":
"A. He is wearing blue apron",
"B. He is wearing a blue T-shirt",
"c. He is wearing a white lab coat",
"D. His attire is not shown in the
4 video"
"correct_ansver", "A"
"reeckccnttett; "In an unordered list, 1ist all 4 the main tools used by the protagonist", "check_itens"
"check_id" "open-004",
"check_type") "attempt", "question": "Are the items listed in the < 1ist tools?",
"options": "yes",
"no"
}, "correct_answer": "yes"
29

---

Published as a conference paper at ICLR 2026
"check_id": "open-005",
"check_type"" "correctness", "question"; "Based on the video
b description, which of the
4 following options most completely
9 lists all the main tools used by 4 the protagonist?",
"options";
"A. Grinder, espresso machine, mug",
"B. Espresso machine, mug",
"C. Grinder, French press, mug",
"D. tools are mentioned in the description"
"cerrect answer": "A"
"check_id", "open-006", rcheck_type"" "correctness", " "question": "According to the video
9 description, which of the 4 following tools was mentioned?",
"options":
"A. Coffee canister",
"B. Milk carton",
"c. electric kettle",
"D. None of the above were ment m
ccrrect _nswerr;_ "D"
Please strictly adhere the Checklist construction philosophy and complete the task accord- ing to the input.
The prompt for prompts generation.
ROLE and GOAL:
You are creative and meticulous multimodal AI evaluation expert. Your core task is to generate 12 diverse and challenging video description task packages based on a given video and constraint knowledge base, to evaluate the instruction-followinn capabilities of large models.
Knowledge Base:
You must strictly adhere to the constraint system knowledge base named "Constraint Frame- work." When constructing tasks, you must:
Atomic Constraints: Only use the atomic constraint items defined in the knowledge base and combine them according to the specified constraint relation-
ships.
Record IDs:
You must record all used constraint items by their unique IDs.
30

---

Published as a conference paper at ICLR 2026
Des gn for Quant buantifiabl Fiable Eva luation: Understand the check items corresponding to each constraint to design description prompts that can be quanti- tatively evaluated.
TASK:
The given video sampled at 5fps. For the input video, you need to: 1. Deep Video Thoroughly and comprehensively analyze the video content, identifying all key entities, attributes, actions, events, relationships, and potential emotional and causal chains. This forms the basis for generating high-quality tasks and reference caption. 2. Create 12
Analysis:
Diverse Prompts:
Field Distribution: Based on four general domains - For Understanding, For Generation, For Retrieval, For Communication and specialized domains For Sports Analytics and For Instructional - select four domains according the actual content of the video, and create three distinct instructions for each selected domain.
For each field, the three prompts should belong to different progressive difficulty tiers:
Level 1: A total of 4-5 constraints.
Level 2: A total of 6-7 constraints.
Level 3: Very Difficult Instructions. Difficulty can be increased in the following aspects:
Structural Control Complexity: Deep and complex multi-leved struc- tural constraints, nesting, and the mixing and dependency of different types or re- quirements of structural constraints.
Mu meldmmmmall Perceptual Compl eemplexity Integration and comparison of information across timelines, capture of non-focal information, and understanding of spatial relationships.
Mult t iimodal Reasoning Complexity: Requiring the "description" of causal intentional relationships, and experiential descriptions based on first- person perspective.
Instruct ion Comprehension Complexity: Complex combinations of constraints, such strict execution order, complex logical combinations, and strict conditional branching.
Prompt Design Principles:
Focus on Description: Prompts should focus on "describing" the video content and avoid being framed as questions reasoning tasks that go beyond the video's content.
pecific and quantifiable: Constraints must be explicit and quantifiable (e.g.."list "list 3 objects," "no more than 50 words"), avoiding vague, qualitative de- scriptions (e.g., "in a humorous style").
Align with the actual video content: The proposed prompts should be meaningful and accurately reflect the real content of the video. Encour- age the creation of deseription instructions specifically designed for unique content in the video. Avoid prompts that are incorrect unrealistic (it is necessary to pre-validate whether a response would be reasonable).
Do Not Plagiarize Examples: You must not copy or closely imitate any examples from the knowledge base.
Constraint Type Balance: The number of format constraints should not exceed the number of content constraints.
31

---

Published as a conference paper at ICLR 2026
Output Specification:
You must return the answer as single valid JSON array containing exactly 12 objects, with- any explanatory text. Each object the array must conform the following structure:
"field": "string", // Must be one of the application
S domains from the knowledge base: For Understanding,
For Generation, For Retrieval, For Communi 4 For Sports Analytics, For Instructional
"prompt_id"" id"; "string", // The unique ID of the promtp, + starting with "01", followed by "02", "03", etc. "generatee__rropt""" "string", // The instruction you 4 generated
"constraints_used"
"string" // An array of unique ID strings for all 4 constraints from the knowledge base used in this task.
Example:
"Eield": "For Understanding", "prompt_id": "01",
creference_captionn "Based on the video content, please generate JSON array record all moving entities. Baen entity should be JSON object with two fields: ""type"" (person/an'mail /aim/m/ct///) and 4 \"descriptions (a description of it appearance). ",
" Wconstraints_used"
"json_array",
"json object"
"entities attributes"
Now, please analyze the provided video and generate these tasks according to the above instructions.
C.4 CONSTRUCTION PROMPTS FOR THE TRAINING SET
The prompt for training set generation consists of the prompt shown below, with the constraint framework table Appendix D serving as the actual content the Core Knowledge Base.
The prompt for training set generation.
Constrained Instruction Construction Specification: Role and Goal:
You are logical and creative Constrained Instruction Construction Engineer. Your core task is: Based on the original video caption, and referencing a comprehensive knowledge base of constraint framework, construct rigorous prompts that align with the caption, along with corresponding target responses, according given set constraints.
Core Principles:
Factual Singularity: The sole source of truth for all generated content is the video caption itself. Do fabricate, infer, or introduce any information not explicitly stated in the caption.
32

---

Published as a conference paper at ICLR 2026
Constraint Supremacy: The input constraint list is the supreme guideline for generating the prompt and response. Apply all constraints strictly and precisely.
Non-ConfLicting Constraints: You can apply constraints in separate scopes if needed (e.g., apply JSON Object constraint first, then ordered list straint).
"losed-Loop Consistency: The refcoence.response must be a perfect and flawless answer to the generated prompt. The two must correspond perfectly in terms of facts and constraints.
Knowledge Base:
Strictly adhere to the "Constraint Framework", The constructed prompt must not exceed the scope defined by this knowledge base.
Input Information:
video caption: The original video caption from which all facts must be de- rived.
constraints: One or more sets of strings representing constraints apply to each training sample. Each set of constraints will generate a separate training sample.
Task Workflow:
1. I."eeoontruut Facts: Extract factual content directly from the video." "Prse Constraints:
Understand all constraints apply them correctly." 3. "Construct Prompt: Extend the seed instruction 'describe this video caption' by incorporating all specified constraints. Ensure clarity and unambiguity." "Generate Exemplary Response: Produce 'refer- ence response' that perfectly satisfies the generated prompt, based solely on the video cap- tion content."
Output Specification:
You must output **only a JSON array**, Each JSON object JSON array must contain exactly two keys: "generated _promppt and "reference_resppeeeeeeeeeee response". **Requirements:**
1. **Output type:** raw JSON array only - not a string representation. characters, whitespace, or line breaks** outside the array structure. 3. **No explanations or * of any kind. 4. **The JSON array must be directly parsahle** by any program without further processing. 5. **The ISON array must contain exactly element per set of input constraints, no more, no less.
**No extra
** Important emphasis;**
This is m#madatoory**, not suggestion.
Outputting anything other than raw ISON array is considered invalid.
Always ensure strict compliance with this format.
All values in the JSON must be non-empty; no null or empty strings are allowed.
D CONSTRUCTION OF THE TEST SET
Video Collection
Automated Draft Construction
Supervise Annotation
ingsrightt losms
Impeare intractiin quality
DreamiK
Roweetine than
Deration
Cleck euadel gravrated cheeldiio.
VidCapBeench
Vinai conpattiin
Respoase Generation
Online Platform
Castuat onvers diverv sipto
deplicueed
Automated Evaluation
Anotation Team
Manually corvect lenttiiee
350 high-quality videos
1,400 raw samples
1,400 high high-quality samples
Figure 7: Test Set Construction Workflow
33

---

Published as a conference paper at ICLR 2026
To construct high-gue high-quality test set, designed reproducible pipeline from raw videos finalized samples, as illustrated in Figure 7
Videos collected from academic datasets Drram Ik Wang al.
2024 VidCapBench Chen
ct al. )025)) and public platforms, then filtered based on task relevance, resolution, duration, com- position, and duplication. Non-compliant or low-quality samples were removed, resulting in sistent candidate pool.
Annotation is carried out in two-stage process automatic generation followed by human refine- ment. In the automatic stage, Gemini 2.5 Pro generates instruction-checkiss pairs for each video (e.g., instruction: "Please list the static attributes of the main character in the video using ordered list starting with '-'. Focus on attire color and carried items. For attire color, record two main clothing items." " checklist item unordered list; only one checklist item shown). We then elicit candidate responses (e.g.. "- Green tunic; Black pants; Carrying sword: Wearing belt with pouches") from multiple models under evaluation (e.g.. Gemini 2.0 Flash), and conduct automated quality assessment using GPT-5-mini as the judge (e.g., automated verdict: "correct").
In the human evaluation stage, to ensure high-quality data and its practical utility for downstream applications, annotators were assigned the following tasks:
Verifying that each instruction aligns with the video content and corresponds the prede- fined instruction type;
2. Confirming the correctness and compliance of the model-generated checklists with the specified requirements;
3. Assessing the accuracy of the judgments made by the model evaluator;
4. Manually correcting any issues identified in steps (1) and (2).
Importantly, to enhance the consistency between model evaluations and human judgments, the prompts used for the evaluation model were iteratively refined based on annotator feedback.
To better demonstrate how our annotation team carries out its work, we provide a screenshot of the annotation interface, shown in Figure which illustrates how annotators interact with the system, conduct labeling tasks, and ensure both consistency and quality throughout the process.
aproche poociie
Ppin Focerr Dricks
Figure 8: A screenshot of the human evaluation system, illustrating how annotators interact with the platform to conduct labeling and quality assessment.
E CONSTRAINT FRAMEWORK
The Table D presents the complete constraint framework of IF-VIDCAP in a systematic manner. worth noting that, in order to provide clearer and more concise representation of the examples
34

---

Published as a conference paper at ICLR 2026
and definitions within the constraint system, several individual constraints that originally appeared separately in the Figure have been merged and organized in the table. This restructuring not only intuitive understanding of the logical relationships
improves readability but also facilitates and hierarchical structure among the constraints.
Table The Table of constraint framework
Group
Constraint
Explanation
Example
Organize information using a Markdown syntax table.
Use a Markdown table to record the items appearing in the video, with attributes for name, color, and size.
Table
Requires the mandatory use exclusion of specified keywords.
The description must not contain the word "is".
Keyword
The output description must use ordered symbols to organize information into list. The required valid formats are explicitly defined, and the starting numbering is restricted to: A., a., 1., I., and i. only.
Please describe the protagonist's three key actions in chronological order using an ordered list starting with $1.'
Ordered list
Structural
Requires the output description to use symbols like organize information into list (only one type of bullet point should used throughout the entire list).
Please use an unordered list starting with "-' to list all the vehicles appearing in the video.
Unordered list
Requires the output description to be an array (list) that conforms to the JSON specification.
Please list all actions performed by the characters the video in the form of ISON array.
Jsonaarray
Requires the output description be one or more key-value pairs that conform the JSON specification.
Please output the core entities and their attributes from the video in JSON object format, where the entity is the key and its attributes are the value.
Json Json_object
Requires the output description to be a natural language text without any special structure or markup.
Please describe this video in paragraph.
Plain text
35

---

Published as a conference paper at ICLR 2026
Specifies the output language, including using specified language for all or part of the output.
Please describe the appearing in the video in English, and simultaneously provide Chinese translation for
Language
Specifies the case format for English output, including UPPERCASE, lowercase, and Title Case.
Please describe the video in UPPERCASE.
Case
Limits the length of the output in units of character, words, sentences, or paragraphs. The limitation methods include range limits and exact specifications.
Please summarize the video content in 50 to 60 words. or Please describe this scene in no more than 3 sentences.
Length
Stylistic
Imposes limit the number of described elements that are semantically defined by content, such as objects or actions. The limitation can be range or specific number.
Please describe the features of the three characters in the video.
Count
Adds specified strings before and after the output text.
Please describe the video. The description must start with 'Video Summary:' and end with -Endd'.
Prefix Suffix
Specifies that particular content within the output description should use designated Markdown syntax for emphasis or organization (headings (#text), bold (text), highlight (=-ext==), italics (text), code block (text)).
Please summarize the video content by scene, bold the names of the characters, and make each scene level-two heading in separate paragraph.
Markdown
Requires the output description to use specific symbols strings (e.g".. ::, 'I', ":' --) separate different pieces of information.
Example: List the characters in the video and their actions, using "_' to separate each character-action pair.
Delimiter
Describe the camera movements, shooting techniques, and editing skills the video.
Describe the cinematic language of this clip. including the main camera movements and changes shot scale.
Cinematic
Describe the specific actions occurring in the video.
Describe in detail the entire process of the boy feeding the puppy.
Action
Element
Describe the key events occurring in the video.
List the key events in the video.
Events
Identify key entities in the video.
Deseribe the key entities in the video.
Entities
Identify the static or dynamic attributes entities in the video.
Describe the appearance of the red car the video.
Attribute
36

---

Published as a conference paper at ICLR 2026
Describe the visual content in detailed and objective manner.
Describe in detail the appearance of all characters in the video, their actions and interactions, and the changes in camera focus during the process.
Detail
Provide high-level summary and condensation of the video's content.
Summarize the main of this video in one sentence.
Abstraction
Summary
Goes beyond surface-level description to infer and connect the intentions, emotions, causal relationships of characters, but the inference must be based the video content.
Based on the character's expression and behavior in the video, infer their current mood and its cause.
Inference
Explicitly requires the description include certain content or specific wording.
Describe the video and include the key actions.
Comparative
Explicitly requires that certain specific visual elements not be mentioned in the description
Describe the video, but do not mention the blue car.
Exclude
Explicitly requires that certain specific visual elements must be mentioned in the description.
Describe the video, must include the blue car.
Include
Framing
Specifies that the model should focus on particular aspects of the video, including specific aatitics/arcas, specific senses (c.g., auditory/olfactory inference), etc.
Describe only the activities of the girl in the yellow dress, and infer the sounds she might be hearing.
Focus
Specifies the narrative perspective for the generated description.
As the cat the video, describe your day the first person.
Perspective
F
DATASET SAMPLES
To better present our dataset, this section provides three specific examples, including video frames, instructions and the checklist.Each checklist includes rule-based checks and open-ended checks.The former verifies formatting constraints, while the latter ensures content accuracy. demonstrate our rigorous evaluation of constraint adherence and video captions.
Thess These examples
37

---

Published as a conference paper at ICLR 2026
Data Sample
Prompt:
Create storyboard script three sequential steps: Describe the woman's appearance, using italics for details of her dress and hair. 2. List 3 sounds she might perceive while flying (e.g., wind, rustling pages) in JSON array. 3. Write her internal thoughts as she flies, connecting the floating books her purpose. Thoughts must reference at least 2 distinct book colors, and steps must be labeled "Step 1:", "Step 2:", "Step 3:" respectively.
RULE-BASED CHECKS
Rule-001: italics for words describing clothing and hairstyle only
Constraint:
markdown
Parameters: "md type:" "italic"
List 3 sounds she might perceive while flying (e.g., wind, rustling pages) in a JSON array.
Rule-002:
json array Parameters: "Schema": ("type": "array", "items": "type: string", "minltems": "3", "maxltems": "3"}
Constraint:
OPEN-ENDED CHECKS
Open-001 Step
Describe the woman's appearance
Does Step 1 include description of the woman's appearance?
A. Yes
B.No
Correct Answer: Yes
Which detail about the woman's 's dress (in italics) matches the video?
A. Black dress with gold trim
B. White dress with red accents
C. Blue dress with white polka dots
D. No dress details described
Correct Answer:
Which detail about the woman's hair (in italics) matches the video?
A. Long ponytail
B. Short bob
C. Braided hair
D. No hair details described
Correct Answer: A
38

---

Published as a conference paper at ICLR 2026
Open-002: Step List sounds in JSON array
Does Step 2 present the sounds in a JSON array format (c.g., ,"oound"....])"
A. Yes
B. No
Correct Answer: Yes
Which of the following is a plausible sound the woman might hear while flying with books?
A. Alarm clock ringing
B. Car horn honking
C. Wind whistling
D. None of the above
Correct Answer:
Open-003: Step 3: Write internal thoughts connecting books to purpose (2 book colors)
Does Step 3 contain internal thoughts about flying and the floating books?
A. Yes
No
Correct Answer: yes
Do the internal thoughts mention at least two distinct book colors (e.g,, red, blue, green)?
A. Yes
No
C. Only color
D. Cannot be determined
Correct Answer: A
Do the internal thoughts explain a purpose related the floating books (e.g., sharing knowl- edge, exploration)?
A. Cannot be determined
B. No, purpose is unclear
C. No, no purpose mentioned
D. Yes, purpose is clear
Correct Answer:
39

---

Published as a conference paper at ICLR 2026
Data Sample
Prompt:
Start with "Family Pienic Moments:". Describe three key actions (one per adult, one child) in 20-25 words each. Then, infer the overall mood of the group and list two visual cues sup- porting this. End with " "-End of Description Total word count must be 80-90 words.
RULE-BASED CHECKS
Rule-001: Start with 'Family Picnic Moments:'. End with 'End of Description'.
Constraint: prefix suffix fix:" "nEn of Description
Parameters: "prefix:" "Family Picnic Moment:"; "suf-
Rule-002: Describe three key actions (one per adult, one child).
Constraint: count
Parameters:
"min count:" "3"; "max count:" "3"
Rule-003: Then, infer the overall mood of the group and list two visual cues supporting this.
Constraint: count
Parameters: "min count:" "2"; "max count:" "2"
Rule-004: Describe three key actions (one per adult, one child) in 20-25 words each.
Constraint: length "25"
Parameters:
"unit:" "word"; "min len:" "20:"; "max len:"
Rule-005: Total word count must be 80-90 words.
Constraint: length
Parameters: word:"" min_len:" 80 max_len:" 90"
40

---

Published as a conference paper at ICLR 2026
OPEN-ENDED CHECKS
Describe three key actions (one adult male, one adult female, one child) with factual accuracy.
Open-001:
Does the caption include descriptions of three key actions (one adult male, one adult female, one child)?
A. Yes
B. No
Correct Answer: Yes
Which of the following accurately describes the adult male's key action (per the caption)?
B. He sets up the pienic blanket the grass A. He distributes food (e.g., bananas, bread,
apples) family members
C. He plays a game with the children
None of the above
Correct Answer: A
Which of the following accurately deseribes the adult female's key action (per the caption)?
A. She reads storybook to the children
B. She helps arrange food and interacts warmly with the children
C. She prepares picnic food in kitchen
None of the above
Correct Answer: C
Which of the following accurately describes the child's key action (per the caption)?
A. She runs around the park playing tag
She receives food from the adult male and smiles happily None of the above
She packs picnic basket
Correct Answer:
Open-002 Infer the group's overall mood and list two supporting visual cues
Does the caption include an inferred overall mood and two visual cues supporting it?
A. Yes
B. No
Correct Answer: Yes
Which of the following best matches the inferred mood?
A. Angry
B. Sad
C. Joyful
D. None the above
Correct Answer:
Which following matches inferred supporting cues?
A. Smiling faces, shared food interaction
B. Frowning expressions, distant scating
C. Arguing voices, crossed arms
D. None of the above
Correct Answer: A
41

---

Published as a conference paper at ICLR 2026
Data Sample
Prompt:
Create a first-pesson diary entry from the rusty robot's 's perspective. Include 2 inferred sounds (bused visuals), I emotional reflection, and end with '-End'. Use Title Case for all sentences.
RULE-BASED CHECKS
Rule-001: End With "_End-"
Constraint: prefix_sufii suffix
Parameters: "prefix:" " "Null"; "suffix:" "end"
Rule-002: Use Title Case For All Sentences
Constraint: case
Parameters: "case_type:" "title"
OPEN-ENDED CHECKS
Open-001: Include 2 inferred sounds based on visuals
Does the diary include at least two inferred sounds based on visuals?
Yes
Correct Answer: Yes
Which the following is an inferred sound mentioned in the diary (based on video visuals)
Birds chirping
Metallic clinking robot joints
Water splashing
Cannot be determined
Correct Answer:
Which of the following is another inferred sound mentioned in the diary (based video visuals)?
Steam hissing from the glass bulb C. Children laughing
Wind whistling through windows None of above
Correct Answer:
Which the following irrelevant sounds were mentioned diary?
A. Thunder rumbling
Telephone ringing
C. Dog barking
D. None of the above were mentioned
Correct Answer:
42

---

Published as a conference paper at ICLR 2026
Open-002: Include emotional reflection
Does the diary include at least one emotional reflection?
A. Yes
B. No
Correct Answer: Yes
What emotional reflection is expressed in the diary (based the rusty robot's demeanor)?
A. Feeling sad and isolated
B. Feeling excited and energetic
C. Feeling angry and resentful
D. No emotional reflection is expressed
Correct Answer: A
G EVALUATION SETTINGS
We provide the detailed settings of our evaluated open-source models(Table Most models are tested under default Cettings.Close--ee models are accessed via API calls, using the default configuration.
Table 6: Evaluation metrics for locally deployed open-source models.The "Frame" column rep- resents the frame rate (float) fixed frame number(integer). The "None" in the table means dis- abled. "Auto" means determined by the model's default configuration. ? indicates that models sup- port the "thinking" mode, and the scores split by "f" denote the performance of thinking and non- thinking modes, respectively.
Models
Params Resolution
Frame
Temperature
Top-p
Repetition Penalty
Open-Source Large Multimodal Models
InternV--33
241B
448*448
10
0.1
0.001
1.05
InterVL-3355
38B
448*448
1.0
0.6/0.1
0.001
1.05
InternVL-33.
8B
448*448
1.0
0.6/0.1
0.001
1.05
Owen3-VL-Instrrct
235B
Auto
2.0
0.1
0.001
1.05
Qwen2:5-VI--Intruuc
32B
Auto
2.0
0.1
0.001
1.05
Dwen2.5-VL-Inntnun
32B
Auto
2.0
0.1
0.001
1.05
Swen2.5-VL-Innttrne
7B
Auto
2.0
0.1
0.001
1.05
Qwen2.5-VL-Inne
3B
Auto
2.0
0.1
0.001
1.05
MiniCPM-V-4.59
448+448
2.0
0.6/0.1
1.05
VideoLLLaM33
7B
Auto
2.0
0.1
0.9
1.05
3...-Visvioon--strrue
90B
448+448
1.0
0.1
Auto
Auto
11B
448*448
1.0
0.1
Auto
Auto
LlaVA-V1.6- 6-Vieuna
7B
448*448
32
None
None
Auto
Video-LLaVA
7B
224+224
8
0.1
Auto
Auto
ARC-Hunyuan- Video
7B
Auto
1.0
None
None
Auto
Tarsier2
448+448
1.0
0.7
0.7
1.05
GLM-4.IV
Auto
Auto
0.1
Auto
Auto
Vide22
448+448
32
0.0
Auto
Auto
Kimi-Vi VL
Auto
32
Auto
Auto
Auto
LLaVA-Vidde Video
Auto
32
Auto
Auto
MiMo-VL
Auto
32
0.3
0.95
Auto
VideoChat2
224x224
16
Auto
Auto
Auto
IF Qwen(Ours)
7B
Auto
2.0
0.1
0.001
1.05
H TRAINING SETTINGS
Our model, IF-Captioner- Qwenn,, developed by fine-tuning the pre-trained Qwen-2.5-Vi Instruct-7B model. The fine-tuning process was conducted for total of 2 epochs. We employed the A optimizer with peak learning rate of 10-6 which managed by cosine annealing
43

---

Published as a conference paper at ICLR 2026
scheduler. A linear warmup phase applied for the initial 3% of the total training steps to ensure stable convergence. To fit the model within our hardware constraints, we configured a per-device batch size of 4 and utilized gradient accumulation steps, resulting in effective global batch size of 16. To enhance computational efficiency and reduce memory footprint, we leveraged b (bf16) mixed-precision training throughout the procedure.
I ERROR ANALYSIS
Example
Instruction: 30-40 words, describe only the person's 's action in the video using simple En- glish. Avoid technical terms and focus what they are doing with their hands and tools.
Generated Caption: The person, wearing welding helmet, uses tool to adjust fix some- thing on black appliance in a kitchen setting.
Error Analysis: The response violates the word-count requirement, containing only 22 words instead of the specified 30-40. It is also imprecise and omits key details- for example, that the subject was holding flashlight flashlihtt-comprrne compromising completeness and clarity.
Example 2
Instruction: Start with "Retrieval Keywords:" and output 5 pipe-separated keywords (each 1-3 words long) for the video, excluding the word "bathroom".
Generated Caption: Retrieval Keywords: mirror, sink, bathroom, woman, reflection
The response fails to comply with two key instructions. First, it includes the prohibited keyword "bathroom", which should have been excluded. Second, the keywords are separated by commas instead of the required pipe symbol.
Error Analysis:
Example
44

---

Published as a conference paper at ICLR 2026
Instruction: Based on the video, identify the cap color of each swimmer in every lane.
Generated Caption: Lane 1: Yellow cap; Lane 2: Blue cap; Lane 3: Red cap; Lane 4: White
cap: Lane 5: Green cap; Lane 6: Black cap; Lane 7: Purple cap; Lane 8: Orange cap.
The lane-to-cap color mapping is inaccurate. Specifically, Lane is incor- rectly described as "yellow cap" when should be "black cap" and Lane 3 mistakenly given as "red cap" instead of "white cap". These errors lead to incorrect overall description of the race outcome.
Error Analysis:
Example 4
Instruction: According to the video, how many kinds of ingredients are in the filling one type of flower pastry, what are they?
The filling of one type of flower pastry contains three kinds of ingredients: rose petals, sugar, and lemon juice.
Generated Caption:
The response is incomplete: it fails to list all six ingredients for the flower pastry filling. It provides only subset and omits several items and descriptive notes. This likely stems from relying on single segment rather than synthesizing evidence across the entire video. A correct answer should aggregate information from all relevant sections and explicitly enumerate all six ingredients, each accompanied by brief description.
Error Analysis:
Example 5
Instruction: Describe the video content in chronological order using 3 simple sentences. Focus on the player's actions and the rabbit- like creatures' reactions.
The player uses a water gun to chase and capture fluffy white rabbits in snowy landscape. The rabbits scatter and hide behind rocks as the player aims and shoots. The scene shifts to a forest where more rabbits appear, some turning pink when hit. The player continues to shoot, capturing several rabbits while the environment changes slightly.
Generated Caption:
The instructions required three-sentence response, but the answer included four. In addition, the description was inaccurate for example, the rabbit hiding behind the rock but on top of it in the video.
Error Analysis:
45


Reviews:
[{"review_id": "XppOiOve8l", "reviewer": "Reviewer_z26H", "summary": "This paper introduces IF-VIDCAP, the first benchmark to systematically evaluate instruction-following in video captioning models, shifting focus from traditional descriptive accuracy to fine-grained, compositional constraint adherence (e.g., format, content, style, reasoning). It comprises 1,400 high-quality video-instruction-checklist triplets (27 constraint types, averaging 6 constraints per instruction) and proposes a hybrid evaluation protocol (rule-based checks + LLM-as-Judge) for format correctness and semantic fidelity. Experiments on 20+ models show that while proprietary models lead, top open-source variants are catching up, and general-purpose MLLMs surpass specialized captioners on complex instructions. The authors further release a training dataset and their fine-tuned IF-Captioner-Qwen, which demonstrates substantial gains in instruction-following.", "questions": "1. Do you plan to extend IF-VIDCAP to longer videos (e.g., >1 min) or multi-scene narratives that require temporal summarization or causal reasoning?\n2. Are there plans to support multi-turn instruction-following, where users refine their requests iteratively?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. IF-VidCap is the first benchmark to explicitly evaluate instruction-following in video captioning (27 constraint types), addressing a critical gap beyond traditional accuracy/fluency metrics.\n2. 1,400 video-instruction-checklist triplets via a two-stage pipeline (auto-generation + human refinement), with 83.6% modification rate and consensus-based validation.\n3. Combines rule-based checks (deterministic) + LLM-as-Judge QA (semantic), achieving 96.33% human-agreement for reliable assessment.", "weaknesses": "1. 1,400 samples is relatively small compared to text-only instruction-following benchmarks (e.g., IFEval, CFBench). And videos average 20.5s and max out at 60s — does not test long-form temporal reasoning or multi-scene narratives.\n2. Evaluation focuses on compliance, not quality. Does not assess fluency, coherence, or creativity of generated captions.\n3. Training data distribution gap: Uses a \"caption-to-instruction\" generation method, which may not reflect real user instruction distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3KPC3C9mq", "reviewer": "Reviewer_mY9K", "summary": "This paper introduces IF-VidCap, a new benchmark for instruction-following video captioning that evaluates a model's ability to adhere to diverse, multi-constraint user instructions. Comprising 1,400 video-instruction pairs with complex constraints, the benchmark reveals through extensive evaluation that even top models like GPT-4o achieve only modest instruction fidelity, with specialized captioning models struggling significantly. The work also provides a 46K-pair training dataset and shows that fine-tuning can improve performance, positioning IF-VidCap as a key driver for future research in controllable video description.", "questions": "1. Will the authors release the IF-VidCap dataset, annotation tools, prompts, and evaluation scripts? Making these public is crucial for adoption.\n\n2. Can the authors clarify how the “thinking” mode is applied across models? Is it literally GPT-style CoT prompting, or something else? Providing the exact prompts or settings would help reproducibility.\n\n3. How do models perform on each of the 27 constraint types? Are there systematic failures on particular formats (e.g. JSON vs Markdown) or content categories (e.g. spatial vs temporal constraints)? Understanding this could guide future improvements.\n\n4. The training instructions were generated from captions using DeepSeek-V3.1. How natural and diverse are these instructions? Do they cover the same linguistic structures as the test instructions? Has any human evaluation been done on the quality of these synthetic instructions?\n\n5. For context, how do these models perform on standard video captioning metrics (BLEU, CIDEr, CLIPScore) on the same videos? It would be informative to see the drop in performance when moving from free captioning to constrained captioning.\n\n6. The paper relies on automated scores for the main results. Was any human evaluation of model outputs on IF-VidCap done (even a small sample) to check alignment with CSR/ISR? If so, what were the results?\n\n7. Have the authors considered how real-world users might specify instructions beyond the benchmark’s templates? For example, could models handle more open-form instructions (“Describe the video as a news report”) that go beyond fixed constraints?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work demonstrates significant strengths through its creation of IF-VidCap, the first benchmark systematically evaluating instruction-following in video captioning with complex, real-world constraints. The benchmark is built on high-quality, carefully curated data and features a comprehensive, human-validated evaluation protocol. Its extensive experiments across ~20 diverse models yield clear insights into scaling effects and model capabilities, while the accompanying training dataset proves practically useful for model improvement. The analysis is both thorough and accessible, supported by effective visualizations, making a strong case for the benchmark's value in advancing controllable video captioning research.", "weaknesses": "The benchmark has several limitations, including its focus on short videos which excludes long-form content and constrained summarization tasks. Its evaluation, while efficient, relies on automated LLM judgments that may miss nuanced errors and depends on proprietary models, raising reproducibility concerns. Although fine-tuning demonstrates improvement, the absolute performance gains remain modest, and the analysis lacks a deeper investigation into the underlying reasons. Furthermore, the paper provides limited implementation details for key techniques like \"thinking mode\" and offers few qualitative examples to illustrate model failures, which could hinder comprehensive understanding and diagnosis of current shortcomings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Qsnml2INk", "reviewer": "Reviewer_h647", "summary": "The paper introduces IF-VidCap, a new benchmark designed to evaluate whether multimodal large language models can generate video captions that adhere to specific user instructions. It proposes a structured evaluation framework that measures both format correctness and content correctness, combining LLM-based and rule-based checking. The benchmark includes 1,400 samples and an associated fine-tuning dataset to enhance instruction-following abilities. Through evaluation of over 20 models, the authors provide a comprehensive analysis of current MLLMs’ capabilities and limitations in controlled video captioning.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.  Valuable and timely evaluation benchmark — the proposed dataset fills a significant gap in assessing instruction-following behavior for video captioning models. \n2. Covers a wide range of different settings, including multiple constraint types, compositional tasks, and diverse video sources. \n3.  Includes a fine-tuning dataset, enabling reproducibility and extension for future research.\n4. Two-format setting (rule-based vs. open-ended checking) is well-designed and helps assess both structural and semantic capabilities. \n5. Strong methodological contribution: The combination of LLM-based and rule-based evaluation is novel and well-justified. \n6. Reliability verification — the authors confirm the stability and consistency of their evaluation metrics. \n7. Comprehensive analysis of model capabilities — Figure 5 and related results provide meaningful insights into the strengths and weaknesses of current models.\n8. Clear structure and thorough experiments: The benchmark is well-documented, with strong empirical validation across multiple models and metrics.", "weaknesses": "1. Lack of detail on video selection and preprocessing: It’s unclear how the 350 base videos were chosen and filtered beyond general quality criteria. The authors should provide a full list or dataset summary for reproducibility. \n2. Limited discussion on annotation consistency: Although human refinement is mentioned, inter-annotator agreement or quality control statistics are not detailed. \n3. Benchmark scope limitation: The dataset focuses primarily on short or medium-length videos (2–60 seconds), leaving longer temporal reasoning largely unexplored. \n4. Complexity vs. accessibility trade-off: The multi-step evaluation protocol may limit broader adoption unless supported by easily usable code or interfaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZvUZPBg3I6", "reviewer": "Reviewer_sjFE", "summary": "This paper proposes IF-VidCap, the first benchmark explicitly targeting controllable, instruction-following video captioning. IF-VidCap consists of 1,400 high-complexity samples—each composed of a video, a detailed, compositional set of user instructions (averaging six constraints across 27 types), and an evaluation checklist grounded in both rule-based and open-ended question assessments. The benchmark enables systematic study of MLLM video captioning models on both content accuracy and their ability to adhere to fine-grained, user-defined instructions. Detailed experimental results and analyses highlight shortcomings in current models and demonstrate the strength of instruction-specific tuning.", "questions": "1. Could you add part of human-written captions under constraints to measure Human-ISR/CSR and a scripted case for format-only ceilings; otherwise, hardness is anecdotal.\n\n2. Could you quantify template overlap with IF-VidCap constraints (n-gram/Jaccard/TF-IDF) and provide cross-benchmark results to show that IF-Captioner-Qwen’s gains persist on external IF-video evaluations?\n\nOther concerns please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**1. Novelty and Relevance:**\nThe paper identifies a clear gap in current video captioning evaluation: lack of systematic assessment for instruction-following. : Shifts video captioning evaluation from “describe everything” to controllable generation with compositional constraints, a real need for editing, generation, and content ops. The benchmark enumerates 27 constraint categories with clear coverage (format, stylistic, content; Fig. 3d), which is more granular than prior video captioning benchmarks.\n\n**2. Evaluation Metrics Design:**\nThe rule+LLM hybrid judging (LLM extracts; scripts verify) for format, complemented with retrieval-based QA for content, is sensible and scalable. Inclusion of both CSR/ISR and their rule/open-ended breakdowns is helpful for diagnosing failure modes.", "weaknesses": "**1. Insufficient Baseline:**\nWhile Table 2 compares many models, some critical vision MLLM baselines from the missing related works (particularly methods leveraging procedural or hierarchical modeling and dynamic storyline composition) are not included or discussed, limiting the ability to guarantee current models are being fairly and comprehensively evaluated, such as LLaVA-Video, VideoChat, InternVideo, KiMi-VL, Keye-VL, MiMo-VL, GLM-4.1V and so on.\n\n**2. Unclear Difficulty Calibration:**\nBest ISR ≈ 27.8% (Gemini-2.5-Pro), with open-ended CSR ≈ 35%–36% across leaders. The paper interprets this as “difficult,” but doesn’t calibrate human or near-oracle ceilings. \n\n**3. Narrow Analysis of IF-Captioner-Qwen:**\nGains are shown only on IF-VidCap. Need other video captioning benchmark (e.g., VidCapBench, Dream-1K and so on) to demonstrate transfer and rule out overfitting to this benchmark’s constraint taxonomy.\n\n**4. Figures Need Deeper Interpretation:**\nSeveral figures are highly informative but under-discussed in the text. For example, Figure 5’s constraint-type heatmap is comprehensive, but the main text lacks deeper discussion of which constraint categories most differentiate current MLLMs, and how these signal bottlenecks for the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)