AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
Authors: Xinlong Chen
Yue Ding
Weihong Lin
Jingyun Hua
Linli Yao
Yang Shi
Bozhou Li
Qiang Liu
Yuanxing Zhang
Pengfei Wan
Liang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: caption
audio-visual
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present **AVoCaDO**, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities. We propose a two-stage post-training pipeline: (1) **AVoCaDO SFT**, which fine-tunes the model on a newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) **AVoCaDO GRPO**, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC benchmark under visual-only settings. The model will be made publicly available to facilitate future research in audiovisual video understanding and generation.

Published as a conference paper at ICLR 2026
AVOCADO: AN AUDIOYISUAL VIDEO CAPTIONER DRIVEN BY TEMPORAL ORCHESTRATION
Xinlong Chen2.1.1, Yue Ding"), Weihhnn Lin', Jingyun Hua', Linli Yao", Yang Shi", Bozhou Li,, Qiang Li Yuanxing Zhang', Pengfei Wan', Liang Wang"
'lling Team, Kuaishou Technology
New Laboratory of Pattern Recognition (NIPR)).
Institute of Automation, Chinese Academy of Sciences (CASIA)
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences
*Peking University
Project webpage: https://avocado- http:://awvaad..capp.ooee...... io/
ABSTRACT
Audiovisual video captioning aims generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present AVoCaDO, pow- erful Audio video Captioner Driven by the temporal Orchestration between audio and visual modalities. We propose two-stage post-training pipeline: (1) AVoCaDO SFT, which fine-tunes the model on newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) AVoCaDO GRPO, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms exist- ing spencsource models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC and DREAM-1K K benchmarks under visual-only settings.
INTRODUCTION
In the era of multimodal large language models (MLLMs), video captioning plays critical role in advancing video understanding. In addition to facilitating the alignment of multimodal represen- tations during pretraining (Xu et al., 2021; Li et al., 2024), also functions as a key mechanism for injecting semantic knowledge into downstream video understanding and generation tasks (Sun et al., 2019; Hong et al., 2022; Zhang et al., 2025b). Recent studies (Chen et al., 2024; 2025c; Zhang et al.; Wang et al., 2025b) have shown that training with higher-quality video captions not only im- proves captioning performance, but also yields consistent improvements across broad spectrum of downstream applications. Therefore, advancing the capabilities of vidco captioning models offers foundational pathway toward building more powerful video understanding and generation systems.
Despite notable progress in recent video captioning models (Xu et al., 2024; Chai et al., 2024; Yuan et al., 2025; Shi et al., 2025b; Ren et al., 2024; Shen et al., 2023), most existing approaches remain predominantly vision-centric, often overlooking the rich semantic cues embedded in audio signals. In practice, auditory elements, such as dialogues, voiceovers, and background music, are indispensable for achieving holistic and contextually grounded understanding of video content. A truly comprehensive captioning model should therefore integrate and reason jointly over both visual and auditory modalities. A common workaround for vision-only models to generate an independent audio caption via a separate audio model and concatenate it to the visual description. However, such a decoupled strategy inherently fails to model fine- -raiindd temporal alignment and causal interplay between audiovisual events, limiting its reliability in practical applications.
"This work was conducted during the author's internship at Kling Team, Kuaishou Technology Corresponning author: giang.liue liu@nlpria..c.cm

---

Published as a conference paper at ICLR 2026
Lasce Coopseal Mezzn Browaing,
As M.Gane Cogs vocalin, there cuuse
(Siraing) Loubville slugaet they hell
vucliie in the Unial Marine Corps
20 a0-zdiennce repueseon Muiine Coops
Separate AV caption
Joint caption
Video captios
Chi Audio caption
The video with clusoup of black microplonc..... woman's yourd clossspp
begianing (ocased ol's black mphope .. Nest, woman ... Behind her a drum sct and a dack
theee closoop.
The begine with fenall
speai
"Lasce Corporal Megan Browning...
vouse.... saying. "Lance Corporal
sacalist, we these cusuee rulience. spoiks,, "As Marine some of woman, new idenilied by lower dhird graplic is "LCpl Mopan Browning.... Shhe is oming in chair in from ot drum and dark blze Santeer that reals "NAVAL SCHOOL OF MUSIC She is still
MMegin Browning.
than
shot
blue benner with cimular emblem and the wonds "NAVAL SCHOOL OF ML SIC" White text
esplaits her role: "As Marine Carps vocallet, we there
ileutifees 'LCg!
engaag suchenee and represent the
Magin Reowning
speaking. Audio. "..and wide of Reowning singing country- rrack sung... TTo Louisilll slagger..
unilton
the Marise
wider shot womenn standing and... She
countyy mok song... lynics sing
ggsstires with her free hand as she sings
Jore: "lo Louisville suggper...
7 Question
Performance Comparation on the Dally-Omal Benchmark
Which sadio sorment occomad in 1.pl Brow- ning's rume and aile were diplaaed en screen foot of the Naval School of Musce' flag? A. Courcy musk per formance with byica reforescing leedigint
Separaa
.4E
.5.B
Gemini
B. A vocalist explaning oxfience eapapeoent respossiilttes is the Mor Corpo
N-27A
.0-5
C. Fonnal self aluntiiiation inclading neme and miliary hrndh
n. Orbostal mucic spprinpood iert about hustoicall vocalis miletooes
NAA
25
X
AV Event
Campa- ratie
Conkea Undeestand.
Ever: Sequince
Infermnc
Remoning
Avg
Alignment
Figure 1: Schematic illustration of the pilot experiment. In this example, naively concatenating captions from the video and audio modalities fails to yield a correct answer to the corresponding question. In contrast, jointly processing both modalities to generate time-aligned caption provides sufficient information, as indicated by the underlined text.
To validate the importance of audiovisual alignment, we conduct a pilot experiment on Daily- Omni (Zhou et al., 2025). Using Gemini-2.5-Pro (Comanici al., 2025), we generate two types of captions: one by processing visual and audio inputs separately and then concatenating their re- sulting captions; and the other by jointly processing both modalities to produce temporal-aligned caption. A judge model (also Gemini-2.5-Pro) is then tasked with answering questions based solely on the textual captions. As shown in Fig. 1, the joint approach yields a significant performance improvement, with an average accuracy gain of 15.8%. This gap is even more pronounced in the "AV Event Alignment" category. where reaches 27.8%. underscoring the critical necessity of au- diovisual temporal alignment in captions for comprehensively understanding the video content.
Based on the above analysis, we propose AVoCaDO, an audiovisual video captioner that effectively integrates visual and auditory events in temporal synchrony. Built upon Qwen2.5-Omm (Xu et al., 2025a), which aligns visual and audio signals via interleaved token sequences, AVoCaDO fur- ther enhanced through two-stage post-training pipeline: (1) AVoCaDO SFT, where we collect and construct a dataset 107K high-quality audiovisual video-caption pairs for supervised fine-tuning, with particular emphasis temporal alignment between visual and audio events during caption generation; (2) AVoCaDO GRPO, where we introduce a reward function based on key event align- ment to optimize the temporal coherence of audio and visual information. Additionally, we design two auxiliary rewards to further enhance dialogue accuracy, reduce repetition collapse and regulate caption length. Collectively, these optimizations tailor AVoCaDO to generate captions that are not only semantically rich but also temporally aligned with audiovisual inputs. Extensive experiments demonstrate that AVoCaDO significantly outperforms existing open-source models across multiple audiovisual captioning benchmarks, and achieves competitive performance on the VDC Detailed subset (Chai et al.. 2024) and DREAM-IK (Wang et al., 2024), which evaluate captions in visual- only settings. Our contributions can be summarized as follows:
We propose AVoCaDO, powerful audiovisual video captioner that effectively integrates visual and auditory events with a strong emphasis on temporal alignment. This model will be open- source facilitate future research in more video understanding and generation tasks.
We design a two-stage post-training pipeline for AVoCaDO: (1) AVoCaDO SFT, leverages a 107K high-quality audiovisual caption dataset enhance temporal alignment; and (2) AVoCaDO
2

---

Published as a conference paper at ICLR 2026
GRPO, which employs carefully designed reward functions improve temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse.
Extensive experiments show that AVoCaDO outperforms all existing open-source audiovisual models and even surpasses the commercial Gemini-2.5-Pro UGC- (Wu al., 2025). It also achieves competitive performance under visual-only settings.
2 RELATED WORKS
2.1 VIDEOLLMS FOR VIDEO CAPTIONING
Recent advances in Video Large Language Models (VideoLLMs) (Zhang et al.; OpenBMB, 2025; Zhang et al., 2025a; Shi et al., 2025a;c) have substantially enhanced progress in video captioning. captioners (Ren et al., 2025; Xue et al., 2025; Yao et al., 2024) typically
These VideoLLM-basee employ a video encoder capture video semantics and then bridge them with an LLM to generate high-quality captions. To further describe fine-grained video cues, OwlCap (Zhong et al., 2025) and Tarsier series (Wang et al., 2024; Yuan et al., 2025) construct large-scale, high-quality SFT datasets to enable the generation of detailed captions that balance dynamic motion and static detail.
However, most of these efforts are vision-centiie while neglecting audio content, which plays a vital role in forming comprehensive understanding of video content. Although several recent audiovi- sual VideoLLMs (Cheng et al., 2024; Geng et al., 2025; Liu et al., 2025b; Sun et al., 2024; Hua et al., 2025) have incorporated both modalities, they are not specifically optimized for the captioning task. Concurrent to our work, video- SAoeo---LOON--22 (Tang et al., 2025) and VGG-VideoCaptioner (Wu et al., 2025) have also explored audiovisual video captioning. Nevertheless, the former requires computationally intensive post-training involving six rounds of DPO with sample pairs selected solely based atomic event metrics, while the latter is limited to short-form user-generated videos. In contrast, our AVoCaDO achieves precise temporal alignment of audiovisual events through relatively lightweight training process guided by more holistic audiovisual considerations, and is capable of generating temporally synchronized, high-quality captions across diverse scenarios.
2.2 REINFORCEMENT LEARNING FOR VIDEOLLMS
Reinforcement Learning (RL) (Christiano al., 2017) has attracted increasing attention in Vide- oLLMs for enhancing complex reasoning through explicit thinking chains and verifiable reward designs. Video-R1 (Feng et al., 2025b), VerIPO (Li et al., 2025c), and LongVILA-RI (Chen et al., 2025d) adopt GRPO (Shao et al., 2024) with rule-based rewards to improve performance on general video understanding tasks. Similarly, Time-RI (Wang al., 2025c), TAR-TVG (Guo et al., 2025), and Tempo-RO (Yue et al., 2025) introduce loU-related rewards to advance temporal grounding.
However, these task-specific approaches are not well-suited for detailed video captioning. Verifying long video descriptions remains challenging, as they are prone to visual omissions and hallucina- tions. At present, only a few RL-based methods explicitly target video captioning. VideoChat-RI (Li et al., 2025b) leverages event-recall rewards to improve caption quality. VersaVid-RI Vid-R1 (Chen et al., 2025a) balances the accuracy and completeness of captions through meticulously designed reward mechanism. VideoCap-RI (Meng et al., 2025) decomposes captioning into structured thinking and caption generation stages, integrating thinking and captioning scorers to improve output quality.
In summary, these studies focus on only specific aspects of visual-only captioning. By contrast, our work proposes a holistic reward design to enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse, which is tailored for audiovisual video captioning, resulting in significant gains in fine-grained caption quality across multiple dimensions.
AVOCADO
AVoCaDO is powered by a carefully designed post-training pipeline tailored specifically for audio- visual video captioning. This pipeline consists of two sequential stages: the AVoCaDO SFT stage, followed by the AVoCaDO GRPO stage. We select Qwen2.5-Omml as the base model for its built-in ability to align video frames and audio signals using interleaved token sequencing.
3

---

Published as a conference paper at ICLR 2026
The vidco opens with close-up shot of man... 1 His expression is serions and concemed as he spenks, gesturing with his hands emphasis. The camera pulls back... The second man, with short dark hair... ( He leans forward, smiling and gesturing with his hands as be speaks. The camera then shills woman... ( She looks intently
Qualified
Video frame cuption
Quality Checker
person she is speaking (off-camern) and pestures with her hands as she milks..
Unsuitable length
J man speaks with skeptical tonc, "If we kill him, won't they..." @ Another man immediately chimes in with a single, first man continues... "And then the same thing huppens." ? A woman then interjects... "Uh. actually, we're aware a small faction..."
|Repetition collapse
Audio caption
emphatic word, "Exacily""
Keypolnt omission
AAddlviiuall video caption (Fusing video frame caption audio caption with temporal order)
video opens with a close-up shot of man... T The man.... "If him, won't they...
expression sctious and concemed
he speaks, gesturing with his hands for emphasis The camera polls back to... 2 He leans forward... "Exaetly." " his voice full of agreement. The first man continues his thought, .... "Annd then the sume thing huppens." The camera then shifts to the woman, who now the facus... X She looks intently two man (off-camern) and gestures with her hands she interjects.... "Uh, actually, we're aware of..."
The pipeline for generating high-quality eemporally-alinnee audiovisual video captions. For clarity, corresponding audio-visual events before and after fusion are marked with circled num- bers and underlined for reference.
Figure 2:
3.1 AVOCADO SFT
In this stage, we train the base model using 107K high-quality audiovisual video-caption pairs cu- rated by us. The dataset is constructed by collecting videos from diverse sources and pairing them with meticulously generated captions. The curation procedure is described below.
To enhance the model's capability in describing complex audiovisual interactions, we collect short- form videos rich in auditory elements, including mixed speech, background music, and sound ef- fects. Specifically, we source 24K videos from TikTok- (Company, 2025) and 18K from Short- Video (Shang et al., 2025), both of which offer dense, real-world audiovisual scenarios ideal for audiovisual understanding. To further strengthen the model's ' grasp of multi-scene dynamics and cinematic transitions, we randomly sample 20K videos from Shot2Story (Han et al., 2023). Additionally, we incorporate 29K samples from Fine Video (Farré et al., 2024), 11K from Siluu----ooonsss (Pierre-Care 2024), and 5K from CinePile (Rawal et al., 2024) to further im- prove the model's generalization performance across diverse audiovisual contexts.
Although the pilot experiment confirms the importance of audiovisual joint captioning, we observe that directly generating such joint captions may sometimes lead to information omissions from either the audio or visual stream (see App. D.1 for details). To obtain semantically rich and temporally aligned captions, we adopt two-stage captioning strategy, as illustrated in Fig. 2. First, we utilize Gemini-2.5-Pro to generate modality-specific captions separately from the video frames and the audio track. These separate captions, along with the original video, are then fed back into Gemini- 2.5-Pro to be synthesized into a temporally coherent multimodal caption by aligning events across modalities according to the temporal structure of the video. Finally, quality checker is employed to ensure high data quality. Initially, clearly low-quality captions, such as those with inappropriate length or repetitive patterns, are filtered out. The remaining samples then undergo completeness assessment, where both the pre- and post-synthesis captions are presented to GPT-4.1' for scoring on 1-5 scale based on synthesis completeness. Only samples scoring 4 or above are retained, thereby reducing the risk of critical information loss during multimodal fusion.
3.2 AVOCADO GRPO
To further enhance the model's capabilities in audiovisual video captioning, we adopt the Group Relative Policy Optimization (GRPO) algorithm (Shao et al., 2024), training the model on a ran- domly selected subset of 2K samples from our SFT dataset. As shown in Fig. we design three complementary reward functions to guide the optimization process: (1) a checklist- reward that promotes comprehensive coverage of audiovisual keypoints; (2) dialogue- based reward that
hhttp::/ppltfoomm oopeoppooaa
4

---

Published as a conference paper at ICLR 2026
Ground-trurh caption
Modee-gencrated esption
pentlc sightly megpical rruncal scoes festumn exingo and piane, plays deorpbost sdidl libelief.
cloee-pp
plases sges
speaks
oegenis,
"Vu hon the foom futeg "Ieanw
The chiftito mellin
wortark voice
phont who thei word".
bees scacling, but Daads somuching you lane undertand." de sevay,
Drcompase into keypoinnt
Extruct dialogues
Keypoints
dialogue ( (Woman, "It's just mo.")
generated dialogue
Length-regolarizttonn
Crass-modal Narrative Legie: The wortas's sntionn and cactposed gace lixes inselly on the aaas witte rameuny vake, carriing losing
① (Man with glasses""Iss "I'S just me.")
Z (Man, "I thought I'd llvv
2 (Min with glasses thought have hurdred things sayy. one drive")
1B
hundred things say,
(Women,, "Lat's there enuy ghosts who
other with expresiian sedaca allccion.
Dysami Actionn Interattio:: They gor oach
kept their word."
x
(Weeman with carly "You say you know throe ghosts keeping their words.)) ")
Aadiory Alemenn: A pomte prgguar, and siigtly magkal mosial soor plas thooughee the serre
2 (Women, "James, I I'now.... there's someching you have undorstand"" ")
3 (Min with glasses, "I know you'se seardiing,
2012 Lengh ponddgggmmndd rephinn kotlla)
Medel-goneratee caption
Compute Recall
Compute Precision
Checkllst-based reward Rc
Dialogue-based reward
F1 score
Length-regullrized reward
Figure 3: Illustration of the three rewards Rc, RD, and R2, which are specifically designed for improving the quality of audiovisual video captioning.
targets the ASR fidelity and speaker identification accuracy of dialogues, critical component of audiovisual content; and (3) length-regularized reward that mitigates repetition collapse and reg- ulates caption length. These reward functions complement each other and work synergistically to optimize various critical aspects for enhancing the overall captioning quality.
3.2.1 GROUP RELATIVE POLICY OPTIMIZATION
GRPO significantly reduces both training time and GPU memory usage by eliminating the need for a separate critic model in Proximal Policy Optimization (PPÓ). Specifically, GRPO works by sampling group of G responses f0r, 02 ... OG } for each question from the old policy model Toa.. then computing their corresponding rewards fri, Tr,,"?,.., rG} to derive the advantage function A; for response og:
Fi mean(fr1, F2, (f........................ rGh)
A
(1)
std([r1, 72, rG})
The current policy model no is then optimized using the following objective function:
1 G
To( (q)
I (0) = E. 10 ~TT (0:14)
Σ
min
Too (oslg) Ass
G
i=1
(2)
clip To( ④4)
- 3 DRI. (mel lral|nae (Tref) )
1 E, + )
To (o:1g)
3.2.2 CHECKLLITTBBASD REWARD
To enhance the overall completeness of audiovisual video captioning, we propose a checklist-based reward R. grounded in fine-grained content decomposition. Specifically, each ground-truth caption Sg is pre-decomposed by GPT-4o into structured inventory of keypoints K = (k1, .2...., kn with n | indicating the inventory size. These keypoints are organized according to a compre- hensive taxonomy spanning five core dimensions tailored to audiovisual caption:
High-level coherence in which auditory and visual modalities mutually explain, complement, or guide each other to reveal underlying intent or storyline; explicit temporal alignment between modalities is required.
Cross-modal Narrative Logic:
Dynamic Action Interaction: Motions, events, and pairwise or group interactions among entities, capturing the evolving narrative dynamics of the scene.
5

---

Published as a conference paper at ICLR 2026
Auditory Elements: All sound-related content, including speech, music, and ambient diegetic sound effects, which essential for holistic multimodal comprehension.
Spatio-temporal Cinematography: Structural and stylistic features, such as scene transitions, temporal progression, and camera techniques that shape perceptual and narrative flow.
Static Entity Description: Attributes and spatial configurations of relatively stationary entities, including persons, objects, and environmental elements.
During GRPO training, for generated caption Sgen. the checklist-based reward R. defined as:
|
Re(SSgen gen
K)
Judge(Sgen- k;)
(3)
K
where (yen k:)) {0, 1) is the matching score assigned by judge model, specifically, GPT- 4.1, indicating whether Sgen correctly mentions keypoint k;.
3.2.3 PIALOGUE-BASER REWARD
In parallel, dialogue serves as a critical component audiovisual content. Therefore, we further design dialogue-basee based reward R ensure the ASR fidelity and speaker identification accuracy of a dialogue in captions.
As shown in Fig. 3, we first extract and structure dialogues from captions as list using Gemini-2.55 2.5- Pro, where each entry consists of a speaker and their corresponding spoken content. Let the model- generated dialogue sequence be denoted D Dgen [(s gon). (sgmm c
:...., (. (s
NN
and the ground-trute dialogue sequence as Dst = represents the speaker, ct is the spoken content of the i-th dialogue unit, and M and N are the lengths of the two sequences, respectively.
[( (5g .2 )...., (5 gt [ where st
To compute RD, need to simultaneously consider the speaker similarity Sppeakr and content similarity Scoment between Dgen and Dg. To this end, we adopt two-step strategy: first, we match dialogue units based content similarity; then, we verify speaker consistency for the matched pairs.
For any dialogue content pair (ctTh where i E [1 and [L, MJ, their content similarity Sim qi .g)) 2t) is measured using the edit distance? between the two strings, calculated as:
gen =1 edit distance (c.en
Sim
(4)
max (Ien(c. (cn)
where len(-) denotes the string length. Our goal is to identify subsequence of dialogue units from gen that matches positionally with a subsequence of the same length from Dat- such that the content similarity Sim(-) of each aligned pair exceeds predefined threshold 7, and the total content similarity Scumtent is maximized.
The search for this optimal subsequence is analogous to the classical Longest Common Subsequence (LCS) problem and can be solved via dynamic programming. Let Fi,j represent the maximum total content similarity achievable from the first dialogue units of D Deen and the first j dialogue units of Dat- The transition equation is defined as follows:
if or
Fij
max
Fi-1d Fis
if > 0, > Sim
gen
m1
max
FI-1,j FLg Fi-15-1 Sim c .g}}
if 0.j > 0, Sim
gen
2
where the similarity threshold y is set to 0.6.
After identifying the optimal matched subsequence based on the dialogue content, we further sess speaker consistency (assigned as 0 or 1) for each matched pair based on the video content
"https://en. http:///wwwwkiiedda...o http://wiiiiiiitdtttttteeee

---

Published as a conference paper at ICLR 2026
using Gemini-2.5-PHr and the total number of correctly matched speaker pairs serves the speaker similarity Sspeake-- The final similarity Scombined between the two sequences then calculated as:
combined
(Sspeaker
Scontem)
/
From a physical interpretation, Scombined represents the proportion of correct dialogue units in D gens which takes values in the range [0, min (M, N).. The recall and precision are then computed as:
Rec
Scombined / M,
Prec
Seumbined / N
(6)
The final dialogue- based reward RDD defined as the Fl score:
R.D = 2 Prec
Rec
(Prec + Rec)
3.2.4 LENGTH-REGULARTAN REWARD
For video captioning, output repetition collapse remains a frequently observed issue (Li al., 2023; Yao et al., 2025a). Moreover, in practical deployment scenarios, it is essential to balance inference efficiency with caption quality, which often necessitates maintaining moderate output length.
To mitigate the rate of repetition collapse and enhance inference efficiency, design length- regularized reward RS that encourage complete captions while penalizing excessive length. The thresholds and 72 are set 2048 and 4096 respectively, which is analyzed in App. D.2.
1.0,
if len(S <
len (Sgen) - T1 i S len (Sen) <
R. =
1
(8)
72
0.0,
otherwise
During GRPO training, use the of the aforementioned three rewards the final reward R.
R = RC + RD + Rr,
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
4.1.1 BASELINES
First, we consider two concurrent works focusing on audiovisual video captioning, video- SALMONN-2 and UGC-VideoCaptionn
important baselines. Additionally, evaluate several popular general-purpose audio-visual understanding models, covering both open-source (Qwen- Omni series (Xu et al., 2025a;b), HumanOmniV2 (Yang al., 2025), ARC-Hunyuan- Video (Ge et al., 2025), MiniCPM-o-2.6 (OpenBMB, 2025)) and commercial options (Gemini-2.5 series). To further assess the importance of audio modality, we compare against some strong vision-only mod- els, including Qwen2.5-VL. (Bai et al., 2025), InternVL3.5 (Wang et al., 2025a).
4.1.2 BENCHMARKS
For audiovisual video captioning, we evaluate models on video-SALMONN-2 testset, UGC- VideoCap, Daily-Omni and WorldSense (Hong et al., 2025). The former two benchmarks evaluate caption quality directly, while the latter two are originally designed for audiovisual video question- answering (QA). To adapt these QA-oriented benchmarks for caption evaluation, target model to generate caption for each video, and then utilize a judge model (Gemini-2.5-PPr) answer questions solely based the textual captions. To mitigate answer guessing when the caption lacks necessary information, we instruct the judge model to refrain from answering such questions, which then marked as incorrect samples. Additionally, we evaluate models on the "detailed" subset of the VDC and DREAM-IK benchmarks under visual-only settings.
first the
hhttps:/pplttfomme openai wtt.://cccccccccccttctettccee
S-turbo

---

Published as a conference paper at ICLR 2026
video-SALMONN-2 testset
UGC-VideoCap
Model
Size
Modality
Miss 4
Hall 4
Total +
Audio 1
Visual
Detail f
Avg. 1
Gemini-2.5-Pro
A+V
18.1
13.3
31.3
69.5
74.7
73.7
72.6
Gemini-2.5-Plass
13.9
33.3
74.0
73.0
V1..
25.5
79.4
64.8
59.5
57.4
Qwen2.5-VL
V
40.5
17.0
57.5
46.6
69.1
62.3
59.3
HumanOmniV2
7B
A +V
49.2
12.3
61.6
45.6
66.3
59.5
57.1
ARC-Hunyuan-Ving Video
7B
A+V A+V
45.7
12.5
58.2
52.7
56.0
55.8
$4.8
Qwen2.5-0mni
A+V
41.7
15.4
57.1
46.9
66.1
60.0
57.7
MiniCPM-o-2.6
42.2
14.3
56.5
38.6
68.5
57.7
$4,9
CGC-VideoCaptionge
3B
A AV
31.6
17.0
48.6
61.4
58.4
57.5
59.1
video-SALMONN-22
7B
A
21.2
17.6
38.8
61.8
71.4
68.5
67.2
Owen3-Omni- Instric
30B-A3B
A+V
32.0
13.6
45.6
67.5
74.8
72.3
71.5
Oven3-Ommi-Cuptionn
30B-A3B
A+V
47.6
69.0
75.5
72.3
72.5
AVoCaDO (Ours)
37.3
73.2
Model performance on the audiovisual video captioning benchmarks. "A" and "V" refer to the audio and visual modalities, respectively. The results presented above are reproduced using the official code. Note that the HideoSAALMONN model, which occasionally led to misjudgments. To ensure more reliable evaluation, we uniformly replaced it with GPT-4.1. "Concurrent works with us.
Table I:
testset originally employed GPT-3.5* as the judge
4.2
EXPERIMENTAL RESULTS
4.2.1 DIRECT CAPTION EVALUATION
We first evaluate the audiovisual video captioning performance the Hide--SALMONN- testset and the UGC- VideoCap benchmark, which employ different metrics to directly assess caption qual- ity. As shown in Tab. AVoCaDO achieves state-of-the-art performance among all open-source models on both benchmarks.
Notably, while open-source models, such as HumanOmniV2, exhibit a slightly lower Hallucination rate compared AVoCaDO the Siteo-SALMONN-2 testset, this is because these models are not specifically optimized for detailed captioning and tend to produce overly brief descriptions that fail to convey the full content of the video, leading to signif- icantly higher Miss rate and weaker perfor- mance UGC-VideoCap. In contrast, AVo- CaDO strikes better balance between com- prehensiveness and accuracy, ultimately out- performing all open-source models in both the Total metric the Lideo-SALMMON-- testset and the average score on UGC- CideoCap.
Daily- Omni
World-
Model
Size
Sense
Gemini-2.5-Pro
60.2
33.8
Gemini-2.5-Flash
55.3
31.0
HumannmmniV2
7B
8.2
6.6
ARC-Hunyuan- Video MiniCPM-o-2.6 Qwen2.5-Omi
7B
8.6
8.7
8B
9.8
7.2
7B
13.4
8.6
UGC- NGCC-iddoCaaptioner Viden-SAlMONN
3B
17.0
11.2
7B
29.9
18.2
Owen3-Ommi-lnsteee Qwen3-Omni- Omnn---nnn-a
30B-A33
17.5
12.7
30B-A3B
27.2
AVoCaDO (Ours)
7B
50.1
25.7
Table QA performance by Gemini-2.5-PP based textual captions. To mitigate answer guessing when the caption lacks necessary infor- mation, the model instructed to refrain from an- swering such questions, which are then marked as incorrect samples.
Moreover, compared to the latest large-scale MoE-based omni model, Qwen3-Omni, AVo- CaDO still demonstrates better performance. Remarkably. AVoCaDO even surpasses the Gemini-2.5 series on UGC-VideoCap, high- lighting its strong capability in audiovisual video captioning.
4.2.2 QA QA-BASED CAPTION EVALUATION
The Daily-Omni and WorldSense benchmarks feature challenging questions that require compre- hension of either one or both modalities, along with their temporal relationships. To assess caption quality, employ judge model (Gemini-2.5-Pro) that answers these questions based solely on the
8

---

Published as a conference paper at ICLR 2026
textual captions. To reduce speculative answers when the caption lacks essential information, in- struct the judge model to refrain from answering such questions, which are then marked as incorrect.
As shown in Tab. 2. AVoCaDO significantly outperforms existing open-source models comparable size, well the latest large- scale MoE-based Qwen3-Omni se- ries, achieving performance im- provements of 20.2% on Daily- Omni and 7.5% on Worldsense over the strongest baseline models.
VDC Detailed
DREAM-IK
Model
Acc
VDCscore
FI score
VideoL
7B
33.4
1.9
30.5
ShareGiPT4 Video
8B
35.6
1.8
19.5
AuroraCap
41.3
2.2
20.8
Qwen2.5-VL
44.5
2.4
30.1
Qwen2.5 5- Omni Aico-SAALMON-2
7B
39.7
2.2
31.6
7B
46.1
2.5
34.4
AVoCaDO (Ours)
7B
47.4
2.5
35,9
Additionally, we further evaluate models the VDC Detailed sub- set and DREAM-IK, two bench- marks that are specifically designed to measure the captioning perfor- mance for visual-only videos. As reported in Tab. 3, AVoCaDO also demonstrates competitive performance in this setting.
Table Model performance on the VDC Detailed subset and DREAM-1K, which evaluate captions in visual-only settings.
Reward
video-SALMONN- testset
Daily-Omni by caption
Model
R.D.
Re
R₁
Total
Dig. Fl
RepCol (%)
Avg. 1
Dig. Fl
RepCol (%)
Qwen2.5-Omni
57.1
7.1
7.1
13.4
16.9
8.1
AVcCaDO-SFT
=
41.4
74.4
3.5
48.1
73.6
5.1
ANoCaDO-SFT-2K
43.0
74.1
2.9
48.5
74.8
5.3
41.3
76.5
2.4
49.5
76.1
6.0
ANoCaDO-GRPO
/
37.3
75.9
3.9
49.5
75.2
4.9
4 37.3
76.9
0.4
50.1
76.2
1.0
Table 4: Ablation study post-training pipeline. "Dlg. F1" represents the metric of dialogue quality, computed as in Eq. 7. "RepCol" indicates the ratio of generations exhibiting repetition collapse. AVoCaDO-SFT-2K* refers the model further fine-tuned AVoCaDO-SFT using the same 2K samples employed during the GRPO phase.
4.2.3 ABLATION STUDIES
In Tab. 4, we conduct an in-depth analysis of each component within our post-training pipeline.
First, the AVoCaDO-SHF stage significantly enhances the model's 's overall performance across three key dimensions: benchmark scores, dialogue quality, and the reduction of repetition collapse in cap- tions. These improvements consistent both the Mideo-AALMMON-2 testset, where captions are evaluated directly, and the Daily-Omni benchmark, which assesses caption quality through QA task. This uniform improvement underscores the effectiveness of our SFT data construction strategy.
In the AVoCaDO-GRPO stage, incorporating the dialogue-based reward R.D improves the dialogue Fl-score by over 2% on both benchmarks. Additionally, the accuracy Daily-Omni is also en- hanced by 1.4%, which is attributed to the model's improved ability to generate detailed and precise dialogue content for answering specific questions. Concurrently, the checklist- basedd reward Re significantly reduces the total error rate on the video-SAL.MON- testset, underscoring its effec- tiveness in capturing key audiovisual events. Finally, the length-regularizer reward R, not only markedly alleviates repetition collapse but also boosts performance across other metrics, highlight- ing its dual benefit ensuring conciseness and quality.
To further validate the contribution of these tailored rewards, we additionally fine-tune AVoCaDO- SFT on the same 2K data used in GRPO, yielding NoCaaD--FFT225 However, the model shows no significant performance gains and even exhibits notable degradation on the SALOON. 2 testset. These results suggest that the performance gains stem from the curated reward functions rather than the data volume, confirming their efficacy in advancing audiovisual captioning.

---

Published as a conference paper at ICLR 2026
Caption generated by AVoCaDO
A static, medium shot shows two people, older woman and a younger girl... The audio begins with sharp, percussive sound effect of hands slapping the table. The younger girl speaks in friendly, clear voice. "I'm Aubrey." The woman replies equal cheerfulness. "I'm Amy." They then speak together an enthusiastic, presentational tone, "And you're watching Food Mania Review!"... The seene transitions to a title card... followed
upbeat, energetic, and slightly retro-sounding musical jingle with female vocalist singing "Food Mania Review." The video then returns the original shot of the while the music fading. The girl begins, her voice bright and excited. "Today, we are trying Cheez-Its.", she picks up the "DUOZ" box and gestures fowards it while talking. The woman interjects. her tone conversational and little hesitant, "Two new, well not, well new us..... The with her hands as she speaks. The camera remains static throughout the scene.
and girl the table
smiles and gestures
Figure 4:
An illustration of a video caption generated by AVoCaDO, featuring both precise audio- visual temporal alignment and accurate dialogue rendering.
4.2.4 QUALITATIVE ANALYSIS
Fig. shows a caption generated by AVoCaDO, highlighting strong capabilities in audiovisual temporal alignment and precise representation of dialogues. More cases can be found in App. F.
5 CONCLUSION
This work concentrates on the task of audiovisual video captioning. Initially, we highlight the sig- nificant role of temporal alignment between visual and audio events. Informed by this observation, introduce AVoCaDO, audiovisual video captioner driven by the temporal alignment between audio and visual modalities. Building upon Qwen2.5-0mni, AVoCaDO enhanced through a two- stage post-training strategy: AVoCaDO SFT, which fine-tunes the model on 107K high-quality audiovisual caption dataset emphasizing temporal alignment, and AVoCaDO GRPO, which lever- ages tailored reward functions to further boost temporal coherence and dialogue accuracy while reducing repetition collapse and regulating caption length. Experimental results demonstrate that AVoCaDO substantially outperforms existing open-source models four audiovisual video cap- tioning benchmarks and delivers competitive results on the VDC Detailed subset and DREAM-IK -K, which focus on visual-only video captioning. Ablation studies validate the effectiveness of each component in our training pipeline, underscoring the overall effectiveness of our approach.
ACKNOWLEDGEMENTS
This work is supported by the Strategic Priority Research Program of Chinese Academy of Sci- ences (XDA0480102) and National Natural Science Foundation of China (62576339, 92570204, 62236010).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 13923, 2025.
Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. arXiv preprint arXiv:2410.0305: 2024.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and genera- tion with better captions. Advances in Neural Information Processing Systems, 37:1 17:19472-19995, 2024.
Xinlong Chen, Yuanxing Zhang, Yushuo Guan, Bohan Zeng. Yang Shi, Sihan Yang, Pengfei Wan, Qiang Liu, Liang Wang, and Tieniu Tan. Versavid-rl: A versatile video understanding and rea- soning model from question answering to captioning tasks. arXiv preprint Nxiv:2500609079, 2025a.
Xinlong Chen, Yuanxing Zhang, Qiang Liu, Junfei Wu, Fuzheng Zhang, and Tieniu Tan. Mixture of decoding: An attention-inspired adaptive decoding strategy to mitigate hallucinations in large vision-language models. arXiv preprint arXiv:2505. 17061, 2025b.
Xinlong Chen, Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chen- gru Song, Qiang Liu, Di Zhang, and Tieniu Tan. Vidcapbench: A comprehensive benchmark of video captioning for controllable text-to-video generation. arXiv preprint arXiv:2502. 12782, 2025c.
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling rl to long videos. arXiv preprint arXiv:2507.07966. 2025d.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temport modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.074766 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mareel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261, 2025.
The Data Company. Tiktok-10m: A large-scale short video dataset for video under- standing, 2025. URL https: happ:://uuginggacc. co/ Taseets/The-aatteee company/ TikTok-1CM A dataset of 10 million TikTok posts for multimodal learning and social media analysis.
Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, et al. Omnisift: Modality-asymment token compression for efficient omni-modal large language models. arXiv preprint arXiv:2602.04488 04804, 2026.
Miquel Farré, Andi Marafioti, Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Finevideo. wtt.a/aee/eaeeeeiiieeeeee/ https: http///tggggaaaa//e http://nwaaaaaaaecaeeeeeeeeeeeee htt//daaaets///ee wtw.eeetttt/ggiiiiggneg.egee http://ttssteteeeonnnnggggon//ee
2024.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo. Yibing Wang. Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint arXiv:2503.21776, 2025a.
11

---

Published as a conference paper at ICLR 2026
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng. Junfei Wu, Xiaoying Zhang, Benyou Wang. and Xiangyu Yue. Video-rl: Reinforeing video reasoning in milms. arXiv preprint arXiv:2503.21776 2025b.
Yuying Ge, Yixiao Ge, Chen Li, Teng Wang. Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al. huwyuan-video- Structured video comprehension of real-world shorts. arXiv preprint urXiv:2507.20 20939, 2025.
Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang. Jinming Duan, and Feng Zheng. Long- vale: www.addoiiiugee-----
benchmark towards time-aware omni-modal perception of long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 18959-18969, 2025.
Chaohong Guo, Xun Mo, Yongwei Nie, Xuemiao Xu, Chao Xu, Fei Yu, and Chengjiang Long. Tar- tvg: Enhancing vims with timestamp anchor-constrained reasoning for temporal video grounding. arXiv preprint urXiv:2508.07663 2025.
Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos. arXiv preprint wrXv:2332.100300 2023.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for texi-to-video generation via transformers. arXiv preprint arXiv:2205. 2022.
Daili Hua, Xizhi Wang, Bohan Zeng, Xinyi Huang, Hao Liang, Junbo Niu, Xinlong Chen, Quanqing Xu, and Wentao Zhang. Vabench: A comprehensive benchmark for audio-video generation. arXiv preprint arXiv:2512.09299 2025.
Bozhou Li, Xinda Xue, Sihan Yang, Yang Shi, Xinlong Chen, Yushuo Guan, Yuanxing Zhang, and Wentao Zhang. The unseen bias: How norm discrepancy in pre-norm milms leads visual information loss. arXiv preprint arXiv:2512.08374 2025a.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu, Learning to answer questions in dynamic audio-visual scenarios. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 19108-19118, 2022.
Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. Advances Neural Information Processing Systems, 36:72888-72903. 2023.
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text video. arXiv preprint arXiv:2410.06166, 2024.
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforce- ment fine-tuning, arXiv preprint arXiv:2504. 06958, 2025b.
Yunxin Li, Xinyu Chen, Zitao Li, Zhenyu Liu, Longyue Wang, Wenhan Luo, Baotian Hu, and Min Zhang. Veripo: Cultivating long reasoning in video-llms via criiieer-uuied iterative policy optimization. arXiv preprint arXiv:2505.19000 2025c.
Qiang Liu, Xinlong Chen, Yue Ding, Bowen Song, Weiqiang Wang, Shu Wu, and Liang Wang. Attention guuded self-reflection for zero-shot hallucination detection in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 11016-21032. 2025a.
Xiulong Liu, Zhikang Dong, and Peng Zhang. Tackling data bias in music-avqa: Crafting balanced dataset for unbiased dresstion-nsweerig In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4478 4487, 2024.
12

---

Published as a conference paper at ICLR 2026
Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328, 2025b.
Desen Meng, Rui Huang, Zhilin Dai, Xinhao Li, Yifan Xu, Jun Zhang, Zhenpeng Huang, Meng Zhang, Lingshu Zhang, Yi Liu, et al. Videocap-rl: Enhancing milms for video captioning via structured thinking. arXiv preprint 2rXtv:2506.01700 2025.
OpenBMB. Miniepm-o 2.6: A gpt-4o level milm for vision, speech, and multimodal live streaming on your phone. https: /ttp:://ghtht/// com Ctte///ennBB//ee BwwtaBBMMMMMiiCe /MMniiMM-ni..i// 2025.
Langlais Pierre-Carl. Releasing youtube-commons: a massive open corpus for con-
versational and multimodal data. https://hugging:e
coutuube-cmmmne 2024.
Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313-14323, 2024.
Yiming Ren, Zhiqiang Lin, Yu Li, Gao Meng, Weiyun Wang. Junjie Wang, Zicheng Lin, Jifeng Dai, Yujiu Yang. Wenhai Wang, et al. Anycap project: unified framework, dataset, and benchmark for controllable omni-modal captioning. arXiv preprint 2rXiv:2507.12841 2025.
Yu Shang, Chen Gao, Nian Li, and Yong Li. A large-scale dataset with behavior, attributes, and con- tent of mobile short-video platform. In Companion Proceedings of the ACM on Web Conference 2025, pp. 793-796, 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint ar axiv:2402.033300 2024.
Xuyang Shen, Dong Li, Jinxing Zhou, Zhen Qin, Bowen He, Xiaodong Han, Aixuan Li, Yuchao Dai, Lingpeng Kong, Meng Wang, al. Fine-grained audible video description. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10585-10596, 2023.
Yang Shi, Yuhao Dong. Yue Ding, Yuran Wang, Xuanyu Zhu, Sheng Zhou, Wenting Liu, Haochen Tian, Rundong Wang, Huanqian Wang, al. Realunify: Do unified models truly benefit from unification? a comprehensive benchmark. arXiv preprint 2xXiv:2509.24897 2025a.
Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang. Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi tranularrttl video representation for multimodal large language model. arXiv preprint arXiv:2504. 10068, 2025b.
Yang Shi, Huanqian Wang, Wulin Xie, Huanyao Zhang, Lijie Zhao, Yi-Fan Zhang, Xinfeng Li, Chaoyou Fu, Zhuoer Wen, Wenting Liu, et al. Mme-videooce Evaluating baseb capabilities of multimodal Ilms in video scenarios. arXiv preprini arXiv:2505.2 1333, 2025c.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF interna- fional conference computer vision, pp. 7464-7473, 2019.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visua large language models. arXiv preprint arXiv:2406. 15704, 2024.
Changli Tang. Yixuan Li. Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: Captioning-enhann enhancced mudio-visual large language models. arXiv preprint arXiv:2506. 20ii:220.6./220. 2025.
13

---

Published as a conference paper at ICLR 2026
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.006.3 2024.
Weiyun Wang. Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open- open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint 2rXiv 2025a.
Xiao Wang, Jingyun Hua, Weihong Lin, Yuanxing Zhang, Fuzheng Zhang, Jianlong Wu, Di Zhang. and Liqiang Nie. HAIC: Improving human action understanding and generation with better cap- tions for mudti-modal large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10158-10181, 2025b.
Ye Wang, Ziheng Wang. Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, al. Time-rl: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503. 133V:2503..337,,, 2025c.
Peiran Wu, Yunze Liu, Zhengdong Zhu, Enmin Zhou, and Shawn Shen. Uge-vidcocaptiooee: An omni uge video detail caption model and new benchmarks. arXiv preprint arXiv:2507. 1133, 2025.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. [tt::2099.1484., 2021.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.202/5 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi. Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint urXiv:2509.17765, 2025b.
Yifan Xu, Xinhao Li. Yichun Yang. Desen Meng, Rui Huang, and Limin Wang. Carebench: fine- grained benchmark for video captioning and retrieval. arXiv preprint arXiv:2501.00513, 2024.
Zihui Xue, Joungbin An, Xitong Yang. and Kristen Grauman. Progress-aware video frame cap- tioning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13639- 13650, 2025.
Pinci Yang. Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu, Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM Inter- national Conference Multimedia, pp. 1480-34491 2022.
Qize Yang, Shimin Yao, Weixuan Chen, Shenghao Fu, Detao Bai, Jiaxing Zhao, Boyuan Sun, Bowen Yin, Xihan Wei, and Jingren Zhou. Humanomniv2: From understanding umni-modal reasoning with context. arXiv preprint arXiv:2506.21277, 2025.
Junchi Yao, Shu Yang, Jianhua Xu, Lijie Hu, Mengdi Li, and Di Wang. Understanding the repeat curse in large language models from a feature perspective. arXiv preprint arXiv:2504.142. 2025a.
Linli Yao, Yuanmeng Zhang, Ziheng Wang, Xinglin Hou, Tiezheng Ge, Yuning Jiang, Xu Sun, and Qin Jin. Edit you wish: Video caption editing with multi gralied user control. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1924-1933. 2024.
Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. Timechat-online 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference Multimedia, pp- 10807-10816, 2025b.
Liping Yuan, Jiawei Wang. Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large lanion-lagguage models from detailed video description to comprehensive video understanding. arXiv preprint arXiv:2501.00 -07888, 2025.
14

---

Published as a conference paper at ICLR 2026
Feng Yue, Zhaoxing Zhang, Junming Jiao, Zhengyu Liang. Shiwen Cao, Feifei Zhang, and Rong Shen. Tempo-r0: A video-mllm for temporal video grounding through efficient temporal sensing reinforcement learning. arXiv preprint arXiv:2507.04700 2025.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 2025a.
Xinjie Zhang. Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Jiakui Hu, Yong Xien Chng, Guo-Hua Wang. Qing-Guo Chen, Zhao Xu, et al. Unified multimodal understanding and genera- tion models: Advances, challenges, and opportunities. arXiv preprint arXiv:2505.02567, 2025b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research.
Chunlin Zhong. Qiuxia Hou, Zhangjun Zhou, Shuang Hao, Haonan Lu, Yanhao Zhang, He Tang, and Xiang Bai. Owlcap: Harmonizing motion-detail for video captioning via hmd-270k and caption set equivalence reward. arXiv preprint arXiv:2508. /8634, 2025.
Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: Towards visual reasoning with temporal alignment across modalities. arXiv preprint arXiv:2505. 17862, 2025.
15

---

Published as a conference paper at ICLR 2026
A DETAILS OF THE TRAINING DATA
The vidcos used for our training dataset construction come from multiple sources to ensure diverse audiovisual content. Below provide detailed statistics for each dataset:
TikTok-10M is large-scale dataset containing 10 million short-form posts from TikTok. The dataset reflects authentic patterns of modern short-form videos, including diverse visual styles, short durations, rich background music and voiceovers, and a wide variety of themes such as entertainment, dance, humor, beauty, and pets. From the full dataset, we select 24K videos for our model training, ensuring a representative coverage of content, audio-visual styles, and user- generated characteristics.
Shot2Story is dataset comprises 43K multi-shot videos. The length of each video is ranging from 10s to 40s. 20K videos are chosen from the dataset. Each video in the dataset contains multiple shots. This rich multi-shot structure allows audiovisual caption model to learn to capture key events in each shot and associate them together.
ShortVideo is also large-scale video dataset from sideo platform including 153,561 videos. These videos have varying durations, ranging from under 30 seconds to over 5 min- utes, with most being less than one minute. We randomly choose 18k videos from the dataset for training our model.
Video a dataset with 43K videos that span 3.4K hours. The videos in the dataset are carefully filtered retain dynamic content with both visual actions and mid-fast pace spoken language by word density filtering and visual dynamism filtering methods. We select 29K videos from this dataset.
YouTube- Sabe-Commons is a collection of audio transcripts of 2,063,066 videos shared on YouTube under a CC-By license. The corpus is multilingual, with English as the majority language, and provides automatic translations into several languages such as French, Spanish, German, Russian, Italian, and Dutch. Each video is accompanied by detailed provenance information, including title, link, channel name, and upload date, ensuring transparency and reusability. We sample 1 11K videos from this dataset.
CinePile is a long-form video understanding dataset. The training set has 9,248 videos, from which we choose 5K videos. The videos are sourced from English-language films on the YouTube channel MovieClips, which provides self-contained clips.
B DETAILS OF BENCHMARKS
In this section, we will provide detailed description of the benchmark evaluated.
SLLo-S2 testset comprises 483 videos spanning 14 distinct domains. Each video has a duration ranging from 30 60 seconds, with an average length of 51 seconds. To evaluate caption quality, judge model is employed to process the generated caption along with the ground- truth event, which then identifies three types of errors: Missing Events, Incorrect Events, and Hallucination Events. The latter are categorized manifestations of model hallucination. The total error rate is then obtained by summing the missing rate and the hallucination rate.
UGC VideoCap consists of 1,000 short TikTok videos, each under seconds in duration and containing at least one meaningful audio segment lasting no less than 5 seconds. Each video's caption is evaluated by a judge model that assigns scores on a 1-to-5 scale across three dimensions: visual, audio, and details. These dimension scores are then normalized and aggregated to produce a final caption quality score.
Daily-Oue Ommi an audio-visual question answering benchmark comprising 684 videos depicting diverse everyday life scenarios, sourced from multiple platforms. These videos are densely mul- timodal, offering rich visual and auditory cues. The benchmark includes 1.197 multiple-choice question- pairs, distributed across six core tasks. In our experimental setting, we assess the quality of generated captions by feeding them into judge model and measuring their capacity to support accurate question answering.
WorldSense exhibits a tightly integrated coupling between audio and visual modalities, demand- ing that models effectively harness the synergistic perceptual power of omni-modal data. The
16

---

Published as a conference paper at ICLR 2026
dataset comprises 1,662 temporally synchronized audio-visual clips, systematically categorized into eight distinct semantic domains. To facilitate comprehensive evaluation, it further includes 3,172 multiple-choice
crestion-answer pairs spanning 26 diverse downstream tasks. In our exper-
imental framework, we evaluate the quality of generated captions by feeding them into a dedicated judge model and measuring their efficacy in enabling accurate question answering.
VDC comprises 1,027 diverse vidcos. The captioning model is required to generate captions for each video along five distinct dimensions using five specific prompts; these five categorics of cap- tions then fed into evaluation model to answer questions, thereby assessing the captioning capability. In our experiments, we evaluate our model on the "detailed" subset.
DREAM-IK is a challenging benchmark for detailed video description, featuring 1,000 clips from diverse sources such as films, stock footage, and short-form videos. Each video is paired with fine pee-grained -anaantdd descriptions, and evaluated using AutoDQ, a metric better suited for assessing rich, multi-even narratives than traditional captioning scores.
C IMPLEMENTATION DETAILS
In the AVoCaDO SFT stage, the model is trained for epochs with batch size of 128 and learning rate of 2 x 10-5. During the AVoCaDO GRPO stage, training is performed for epoch with a hatch size 64 and learning rate of 1 10-5, For each query, we sample 8 responses using temperature of 1.0. The KL-divergence regularization coefficient / is set to 0.04, which is commonly used in previous works (Feng et al., 2025a). Both the video and audio encoders remain frozen throughout training, and only the adapters and the LLM backbone are updated.
During both training and evaluation, video inputs are sampled at 2 fps, and the resolution of each frame is limited to a maximum of 512 x 28 x 28 pixels. Due to the base model's 's context window limitation of 32K tokens, the total video tokens restricted to 25600 28 x 28. All training conducted on 16 NVIDIA H200 GPUs, while evaluation is performed on NVIDIA H20 GPUs.
D ADDITIONAL ANALYSIS
D.1 ANALYSIS OF THE AUDIOVISUAL VIDEO CAPTION GENERATION BY GEMINI
In Fig. 6, we compare the audiovisual captions generated directly by Sommini-255-Pa with those produced by the two-stage audiovisual captioning approach used in constructing SFT dataset (Sec. 3.1). The results indicate that direct caption generation tends to omit information from either the audio or visual modality, unlike the two-stage strategy, which provides more comprehensive coverage. To ensure high data quality, we therefore adopted the two-stage captioning method for building our SFT dataset.
D.2 ANALYSIS OF THE THRESHOLDS LENGTH-REGULANEE REGULARIZED REWARD
In this section, detail the rationale for selecting the length thresholds Tı = 2048 and 72 = 4096 in the length- regularized reward R2 (Eq. 8). As preliminary, it im- portant to note that Qwen2.5-Omni supports a maximum context window of 32K tokens and encodes audio at rate of 25 tokens per second. In our training and evalua- tion, effectively capture video dynamics and preserve the visual detail of each frame, we sample videos at 2 fps, with each frame allocated a maximum of 512 tokens for encoding, Due to the context window constraint, the total number video tokens is capped at 25,600.
4K
3,982
Number of Caption Tokens
Mainaas
aK
2K
IK-
0-10% 1010
30.0 Po. So 10 So
Video Duration
Figure 5: Distribution of caption token lengths across video durations.
The upper threshold, 72 4096, is determined by the maximum feasible video duration that the model can pro- Fig. 5 shows analysis of the caption lengths gen- erated by Gemini-2.5 2.5-Pro for videos of varying durations, which reveals that for videos up to 100 seconds, the maximum caption length rarely exceeds 3,982 tokens. A 100-second high-resolution
17

---

Published as a conference paper at ICLR 2026
Cases on Two Caption Methods
Gg
GO
GC
GGgE go
GG
GO 9g
Gg
Direct Caption The video begins with a still image of a friendly-looking, purple cartoon gorilla with a pink bow her head. She is sitting with her hands clasped, smiling and looking upwards. To her right a white card displaying the uppercase and lowercase letter "Gg" in black. The entire image is set against a vibrant orange background with radiating lines. A female voice, presumably the gorilla's, speaks. "Hi, I'm a good gorilla. This is the /g/ sound. This is the letter G. The image then transitions. The gorilla and the letter card disappear, replaced by blank white card with an orange border. An animation demonstrates how to write the uppercase letter "G". A voice counts "One" as a large, curved C-shape is drawn in blue. The voice then says "Two" a short, horizontal pink line is drawn inward from the middle-right of the curve, completing the "G". Next, the animation demonstrates how to write the lowercase letter "g". The voice counts "One" as a small blue circle is drawn. The voice then says "Two" as pink line is drawn downwards from the right side of the circle, ending in a hook to the left, completing the The scene then returns to the original image of the purple gorilla next to the "Gg" card. The female voice says the sound, "/g', /g/, good gorilla." A child's voice repeats, "/g/, /g/, good gorilla. "The female voice then asks, "What's the beginning sound?" The child's voice answers, "/g/." The vidco ends on the still image of the gorilla and the letter card.
Two-Stage Caption:
The scene opens with a static shot featuring a cartoon gorilla and a letter card. On the left, a frienddy-looking, purple cartoon gorilla sits with its hands clasped together. It has large, expressive eyes looking upward and gentle smile. A small pink bow is perched its head. The gorilla and the letter card are set against vibrant orange and yellow sunburst background, all enclosed within a thick orange border. To the right of the gorilla is white rectangular card. The uppercase letter "G" and the lowercase letter "g" are printed in large, black, sans-serif font on the card. A female character speaks with high-pitched, sweet, and friendly voice. She begins by introducing herself enthusiastically, "Hi. I'm a good gorilla." Her tone then shifts to be more instructional, yet remains gentle and encouraging as she explains, "This is the 'guh' sound. This is the letter G." The scene transitions to instructional graphic demonstrating how to write the letter "G." There are no characters in this scene. The background remains a white rectangle framed by the orange and yellow sunburst border, On the left, completed uppercase "G"" is shown in black. To its right, an animated graphic demonstrates the strokes needed to form the letter. A male narrator counts "One" as large, curved C-shape is drawn in blue. The voice then says "Two" as a short, horizontal pink line is drawn inward from the middle- right of the curve, completing the "G". The setting is unchanged from the previous scene. The graphic for the uppercase "G" remains on the left. To the right, a completed lowercase "g" appears in black. Next to it, an animated graphic shows how to write the letter. A blue, circular shape is drawn first, followed by curved, downward pink line to form the tail. Dottee lines and numbers again illustrate the two-step process. The narrator's voice counts again, "One. Two." The scene returns to the original shot of the gorilla and the letter card. The purple gorilla with the pink bow is shown again, sitting in the sume pose with pleasant expression. The background is the same orange and yellow sunburst pattern. The white card displaying the uppercase "G" and lowercase "g" is positioned the right of the gorilla. The entire frame briefly fades to a lighter, washed- out color before returning to its full vibrancy. The female character's voice returns, clearly and slowly enunciating the phonetic sound, "Guh. Guh." " She then says the example phrase with cheerful lilt, "Good gorilla." A young boy's voice repeats after her, mimicking the sounds with a clear, youthful tone, "Guh. Guh. Good gorilla." Finally, the female character's voice poses a question in friendly, engaging manner, "What's the beginning sound?" The young buy's voice confidently unswers, "Guh."
Figure 6: Comparison between direct captioning and our proposed two-stage approach. Colored text highlights information present in the two-stage captions but absent in the direct captions, with audio-related and visual-related content distinguished accordingly.
video consumes 2,500 audio tokens (100s x 25 tokens/s) and the maximum 25,600 video tokens, totaling 28, 100 tokens for multimodal input. When combined with the input text prompt and the generated caption, the total token count approaches the 32K context limit. To prevent context over- flow and ensure the generation of complete and untruncated captions, we constrain our training dataset to videos of 100 seconds or less. Consequently, the maximum target output length, T2, is set to 4096, providing safe margin.
18

---

Published as a conference paper at ICLR 2026
Wine Chockktse botte Rowardd
Vilte of Didlouu lased Kewand
Eu422
W
Malur LLngB-Rcellaiii Rowaad
M
M
0.50
8.50
6.023
10
15
20
15
15
35
Training Step
Tisining Step
Training Slegy
Figure 7: Reward curves during the AVoCaDO-GRPO stage.
The lower threshold, 71 2048, is designed to strike balance between comprehensiveness and conciseness for practical applications, Fig. 5 shows that the mean caption lengths for vidcos under 100 seconds are below 1,437 tokens. Based this observation, we set the first threshold 7i at 2048, a value comfortably above the average, to grant the maximum length reward to outputs of typical length. For captions with lengths between Ti and T2. the length reward decreases linearly. This reward structure incentivizes the model to autonomously learn a trade-of off between generating more detailed caption and optimizing other reward metrics related to factual accuracy and completeness.
D.3 REWARD CURVES DURING TRAINING
In Fig. 7, present the evolution of the three reward functions used during the AVoCaDO-GRIE stage. As shown, the checklist-bart reward Re and the dialogue- based reward Ro steadily in- and approach convergence throughout training. The length-regularized reward R,, occa- sionally exhibits sharp dips during training, which occur when the model encounters particularly challenging samples that induce repetition collapse in the generated caption. Notably, the minimum values of these dips gradually rise over time, indicating that the model's generation stability is im- proving. By jointly optimized by these three complementary reward functions, AVoCaDO enabled to further enhance temporal coherence and dialogue accuracy while mitigating repetition collapse and effectively regulating caption length, ultimately demonstrating strong capabilities in generating high-quality audiovisual captions.
D.4 PERFORMANCE IN MUSIC AND GENERAL SOUND SCENARIOS
We evaluate AVoCaDO in music and gen- eral sound scenarios on AVQA (Yang al., 2022), MUSIC-AVQA (Li et al., 2022), and MUSIC-AVQA-v2.0 (Liu al., 2024), using Gemini-2.5-Pro as the judge model to answer QA queries based on generated textual cap- tions. The results are summarized in Tab.5. As shown, AVoCaDO not only demonstrates strong performance in video-speech related scenarios, but also exhibits significantly superior capa- bility in describing music and general sound, substantially outperforming the baseline model Owen2.5-Omin and approaching the perfor- mance of the commercial Gemini-2.5-Pro.
MUSIC- AVQA
MUSIC-
Model
AVQA
AVQA-v2.0
Gemini-2.5-Prr Pro Qwen-2.5-Omni
72.4
72.8
50.5
66.6
55.8
29.2
AVoCaDO (Ours)
71.8
62.0
45.8
Table QA performance by Gemini-2.5-PP based on textual captions in music and general sound scenarios. To mitigate answer guessing when the caption lacks necessary information, the model is instructed refrain from answering such questions, which are then marked as incorrect.
E FUTURE WORKS
Although AVoCaDO demonstrates substantial gains in audiovisual captioning, several promising directions remain for future enhancement: (1) detecting and mitigating hallucinations (Liu et al., 2025a; Chen et al., 2025b; Li et al., 2025a) in generated captions to improve their faithfulness and reliability; and (2) balancing the trade-off between latency and accuracy in real-time settings, for example by integrating token compression (Yao et al., 2025b; Ding et al., 2026) or related efficiency- oriented strategies to accelerate inference while preserving caption quality.
19

---

Published as a conference paper at ICLR 2026
Caption Comparison across Models
Andy
The video begins with serene seene of small-town street cloudy day... A dog leisurely follows Andy as he greets the viewers, saying. "Good morning, Andy here with Ruby and we' re off to ask expert." As the scene transitions, the logo for "Andy the Estate Agent' appears... Andy and Ruby continue their walk, passing by the familiar storefronts, as Andy shares, "As you can see, the weather's against us but we're off inside to go and see Andy Bright at Ayles Jones and Bright Solicitors." " The narrative then moves indoors, showing Andy entering the office of Ayles Jones and Bright Solicitors through glass doors. Inside, the office is modern, with wooden desk and potted plant adding touch of greenery. Another man, presumably Andy Bright, stands near the desk, engaged in a phone call. Andy greets Andy Bright warmly, "Come on, let's go in the dry," followed by, "Then we go out of here," as they proceed further into the office. Inside, the two men sit across from each other the desk, enjoying cups of coffee. Andy expresses gratitude towards Andy Bright, saying, "Alright, hello Andy. Thank you very much for inviting me round this morning to answer few questions to help people." Throughout the video, the audio complements the visuals effectively, capturing the casual yet professional atmosphere of the day while highlighting the community and business interactions taking place in this charming town.
Midco-SALMONN--
UGGC-VddeCaaptiones The video opens with a man in a black suit and purple shirt walking down a quaint, slightly overcast street... and small dog is seen walking alongside him. The scene then transitions to white screen with a logo that reads "Andy the estate agent"... ... The next scene shows man and his dog walking through a wet rain, and then entering a modern office with a large window showing the same street outside. Inside, he sits a wooden desk with young man in glasses, and they engage in conversation, with the man in the suit speaking and the young man listening attentively. The video concludes with the man in the suit continuing to speak, presumably answering questions, while the young man listens. The overall tone is professional and informative, focusing on estate related topics.
AVoCaDO: A man in a dark suit and purple shirt walks along a brick sidewalk in a quaint town. A golden- colored dog trots beside him... The man speaks in a friendly and upbeat tone, introducing himself and his companion, "Good morning. I'm Andy Sanderson. This is Ruby, and we're to ask an expert." white screen uppears, followed by wlack-and-white animated logo. The logo features the name "Andy" in stylized font, with the words "the estate agent" underneath. The audio transitions to a brief, modern, and slightly dramatic musical sting with whooshing sound effect, creating a professional and contemporary feel. The scene shifts to a view from inside a building, looking out through a rain-streaked window. The same man in the suit and his golden dog are the wet sidewalk outside. The man gestures with his arms he walks, saying with a slightly more energetic and cheerful delivery, "As you can see, the weather's against us, but we're 're off inside to go and see Andre Bright at Owain and Jones and Bright Solicitors." He then addresses his dog in a playful, encouraging tone, "Come on, Ruby, let's go in in the dry." A man in a maroon sweater stands inside at desk, holding the telephone receiver to his ear. The first man follows up with calm, direct instruction, "Andy, go out of here." The man and his dog walk through a glass door into the building. Inside an office, the man in the suit sits across a wooden table from a younger man with glasses, who is wearing durk suit jacket. The older man holds a white mug and speaks with polite tone, "Hello, Andre. Thanks very much for inviting round this morning to answer few questions to help people", while the younger man listens attentively. The sound of rain can heard faintly in background.
Figure 8: Qualitative comparison of AVoCaDO against two contemporary captioning models: video- SALMONN-2 and UGC- VideoCaiioaa Errors in baseline outputs are highlighted in red; the supe- rior coverage and precision of AVoCaDO are highlighted in blue. Correct / incorrect audiovisual temporal alignment is bolded, while sound effect descriptions are underlined.
F ADDITIONAL QUALITATIVE RESULTS
In Figs. 8 and 9, we present qualitative comparisons of AVoCaDO against two contemporary cap- tioning models, Video-SALMONN-2 and CiC-ViideCaaptioner..
20

---

Published as a conference paper at ICLR 2026
Caption Comparison across Models
video-SALMONN-2
The video begins with a static image displaying the logo and contact information for 'Suranga Akash Salon Suranga." The logo features a stylized letter with a decorative curve, accompanied by the salon's name in elegant gold text... As this introductory image fades, the scene transitions to more intimate setting where stylist... As the video progresses, the focus remains on the same stylist continuing his work. This time, is adding floral decorations to the woman's updo... Throughout these clips, the audio complements the visual narrative. A female voice begins by saying, "A story to begin with ultimate glamour and colour. Dressing embedded with sheer joy and affection. Making the moment remember and cherish." " The voice continues to speak softly, enhancing the sense of anticipation and the importance of looking onc's best on such special day. The narration throughout the video maintains this tone, providing context and depth to the visual storytelling of the stylistic transformation taking place in this elegant setting. The combination of clegant visuals and subtle, meaningful audio creates cohesive and engaging narrative that highlights the artistry and care behind every aspect of the styling process.
The video opens with a white sereen displaying gold logo and text for "Suranga Akash Salon Suranga" along with contact information. The scene then transitions to well lit, elegant room with white blinds and mirror, where man in white shirt... The video concludes with a close- up of the finished look, highlighting the elaborate headpiece... The audio throughout the video is a soft, melodic background music, which complements the visual presentation of the detailed bridal preparation.
UGC- VideoCaptioner:
AVoCaDO: A static shot displays a business card on plain white background. The card features a gold, stylized logo of the letters "SAA with flourish. Below the logo, the text reads "SURANGA AKASH" and "SALON SURANGA," followed by "MOBILE 071101650, 0112504770* and an address, "75/7, BAGORU, SRI SUBUTHI ROAD, WELLAWATTE, The audio opens with a gentle, modern electronic track featuring a soft, rhythmic beat and ambient synth pads, creating sophisticated and relaxed atmosphere. The scene shifts a view through a decorative, star-shaped mirror. A man white shirt... male narrator with deep, smooth, and professional voice begins to speak in a calm and measured tone, "A story to begin with ultimate glamour and color." The camera then moves narrator continues, "A dressing embedded with sheer love and affection, making the moment remember and cherish." The man, now more in focus, continues to work on the woman's hair... The narrator says, "Dedicated to ensure that you look the best on your most precious day, the wedding day." His delivery is warm and reassuring, conveying sense of and expertise that complements the elegant background music.
close-up of the woman's profile... The
Figure 9: Qualitative comparison of AVoCaDO against two contemporary captioning models: video- SALMONN-2 2 and UGC- VideoCaptioner Errors in baseline outputs are highlighted in red; the supe- rior coverage and precision of AVoCaDO are highlighted in blue. Correct / incorrect audiovisual temporal alignment is bolded, while sound effect descriptions are underlined.
As shown in Fig. 8, video-SALMONN-2 contains multiple inaccuracies in dialogue recognition, misaligns the temporal order between the man's speech and scene transitions, and concludes with an unfitting summary. UGC- CGC-VideoCaptionne on the other hand, omits dialogue content entirely and introduces redundant descriptions toward the end of the caption.
Similarly, in Fig. 9, SAMcc---LLONN-22 again fails align auditory and visual events chronologi- cally, only mentioning the audio content at the very end of the caption. Additionally, it misidentifics the speaker's gender and overlooks the final narration segment. UGC- UGC-VideoCCnm VideoCaptioner still neglects all spoken content, merely making a generic reference to background music at the end of the caption.
In contrast, leveraging an effective two-stage training pipeline, AVoCaDO generates high-qualit diovisual video captions that accurately synchronize audiovisual events temporally, faithfully tran- scribe dialogue content, and maintain strong semantic coverage in both cases.
21

---

Published as a conference paper at ICLR 2026
Prompts to generate video frame caption
You are professional video caption writer. Your task is create detailed, scene-by-scene narrative description of video. For each scene, your description must include the following elements:
Main Subjects: and gestures.
Describe the people present, including their appearance, clothing, actions,
Setting & Background: Detail the environment, background, and any notable objects. On-Screen Graphics: Mention the specific content of any text, titles, emojis that appear on the screen.
Camera Work: Note any significant camera movements like zooms, pans, or angle changes.
Figure 10: Prompts to generate video frame caption.
G DETAILS OF PROMPTS
G.1 PROMPTS TO GENERATE CAPTIONS FOR SFT
Figs. 10 12 present the prompts used generate video frame captions, audio captions, and to synthesize both, respectively, during the creation of the SFT caption data detailed in Sec. 3.1.
G.2 PROMPTS TO DECOMPOSE CAPTIONS INTO KEYPOINTS
In Fig. 13, we present the prompt used to decompose a caption into keypoints, which is the founda- tion of the checklist-based reward detailed in Sec. 3.2.2.
G.3 PROMPTS TO JUDGE KEYPOINT ACCURACY IN CAPTIONS
As illustrated in Fig. 14, we present the prompt designed to assess whether keypoints are accurately described in caption, which is used to compute the checklist-bart reward Re.
G.4 PROMPTS TO EXTRACT DIALOGUES CAPTIONS
In Fig. 15, we present the prompt used to extract dialogues in the caption, which is the foundation of the dialogue- based reward detailed in Sec. 3.2.3.
G.5 PROMPTS TO IDENTIFY SPEAKER SUBJECT CONSISTENCY
Fig. 16 shows the prompt to determine whether the speakers in each aligned pair refer the same subject based on the video content, which is used to calculate the number of correctly matched speaker pairs Sppeaker.
G.6 PROMPTS TO ANSWER QUESTIONS TEXTUAL CAPTIONS
In Fig. 17, provide the prompt used to assess the quality of caption by leveraging it to answer questions, as described in Sec. 4.2.2.
H THE USE OF LLMS
Throughout the coding and debugging stages, we leveraged LLMs for technical guidance. Follow- ing the collaborative drafting of the manuscript, again engaged LLMs to polish and refine its language and overall expression.
22

---

Published as a conference paper at ICLR 2026
Prompts to generate audio caption
You are a professional audio caption writer. Your task is to create detailed narrative description of audio in the video. Your description must include the following elements:
Narration / Dialogue: Please accurately transcribe the spoken words (narration or dialogue) from the audio. In addition to the transcription, describe the speaker's tone and emotional delivery during the speech- such as whether the tone calm, excited, hesitant, enthusiastic, serious, sarcastic, etc.- vocal cues like pitch, pace, volume, and emotion.
Music Sound: Describe the background music's s mood and any important sound effects.
The audio caption should be coherent and well-structured. Do not simply give the transcrip- tions without the speaker's tone and emotions.
Figure II: Prompts to generate audio caption.
Prompts fuse the video frame caption and audio caption
You tasked with fusing the visual caption and audio caption into single, coherent narrative based on the video content. Follow these strict rules:
1. Preserve every single sentence from both the visual caption and audio caption exactly as they appear.
2. Do NOT omit or delete any sentence in any way.
You may reorder the sentences (from both captions) to create a logical and temporally accurate sequence that reflects the video's 's events.
Ensure the integrated narrative flows naturally time with the video, aligning visual actions with corresponding sounds spoken content.
Verify before responding: Did I include every sentence from both captions?
Visual caption: (visual caption)
Audio caption: (audio caption)
Now generate the integrated audio-visual caption:
Figure 12: Prompts fuse the video frame caption and audio caption.
23

---

Published as a conference paper at ICLR 2026
Prompts decompose captions into keypoints
You expert assistant designed for fine-grained audiovisual content analysis. Your task is to decompose given video caption into a structured, comprehensive, and non-redundant inventory of distinct keypoints. Extract and categorize fine-grained keypoints from the given video caption according to the following five audiovisual-specifc dimensions. Ensure the keypoints are atomic, precise, and non-overlapping.
1. Static Entity Description: Attributes and spatial configurations of relatively stationary entities. This includes people, objects, animals, and environmental elements.
Dynamic Action & Interaction: Motions, events, and pairwise or group interactions among entities that describe the evolving narrative.
3. Auditory Elements: All sound-related content, including speech, music, and ambient or diegetic sound effects, which is essential for holistic multimodal comprehension.
Spatio-temporal & Cinematography: Structural, stylistic, and temporal features of the video, including scene settings. transitions. temporal progression, and camera techniques. 5. Cross-modal Narrative Logic: High-level coherence where auditory and visual elements explicitly explain, complement, or guide each other to reveal the storyline or intent. This must involve an explicit temporal alignment between a sound and a visual event.
Output Format: You should output the keypoints in Python List Format: ["xxx", "xxx",
Video Caption: {video caption]
Given the video caption, please list all the keypoints:
Figure 13: Prompts to decompose captions into keypoints.
Prompts to judge keypoint accuracy in captions
A good video caption is one that describes the various details in the video. Your task is to judge whether video caption good or not. You will be provided all the keypoints in the video, and also video caption be evaluated. You need to determine which keypoints described correctly in the given video caption.
There are totally (# keypoints) keypoints in the video. All the keypoints will be provided List format, i.c. ["xxx". "xxx". The video caption be evaluated will be provided as well.
Output Format:
Your output should be strict the following Python dictionary format without anything else: ("Count of correctly mentioned keypoints": x, "Correctly mentioned keypoints": [...]]
Keypoints in the video: {keypoints) Video caption to be evaluated: {video caption)
Given keypointss in the video and the video caption, please count the correctly mentioned keypoints and list them
Figure 14: Prompts to judge keypoint accuracy in captions.
24

---

Published as a conference paper at ICLR 2026
Prompts to extract dialogues in captions
You are highly skilled assistant specializing in extracting conversational dialogue from text. Your task is to carefully analyze the given description of video and accurately identify and extract all dialogue content within it.
Please directly output the dialogue in the following format without adding any other content. no dialogue present, state: "None."
Dialogue format:
Speaker A Description: "Dialogue from speaker A." Speaker B Description: "Dialogue from speaker B."
Speaker A Description: "Further dialogue..."
The description for each speaker (e.g., "Person in red dress") must align with the given description and should be simplified for brevity. The key is to be concise and clearly distinguish between speakers (c.g., "Man red shirt" sufficient).
Video description: (video description}
Figure 15: Prompts to extract dialogues in captions.
Prompts identify identify speaker subject consistency
Given video and several pairs of descriptive phrases about certain subject, please help me determine whether the subjects in each pair refer to the same entity in the video.
For each pair of phrases, respond with 'Yes' or 'No', separated by a single space, without any extra characters. For example, if three pairs of phrases are provided, a valid response format would be: 'Yes No Yes'.
Descriptive phrases (each line contains a single pair): {dialogue pairs}
Figure 16: Prompts identify speaker subject consistency.
Prompts to answer questions based on textual captions
You are a precise QA assistant. Your task is to answer multiple-choice questions based ONLY the video caption provided.
Do not any outside knowledge assumptions-yout answer must strictly reflect information from the caption. Always output only the capital letter corresponding to your choice (e.g., A, B, C, D). If the caption does not provide enough information answer the question, output "N/A" instead.
Here is the video caption: {video caption}
Question: {question] Choices: (choices)
Figure 17: Prompts to answer questions based on textual captions.
25


Reviews:
[{"review_id": "IP0yf6G8Qu", "reviewer": "Reviewer_yLZR", "summary": "This paper proposes an audiovisual video captioning system that generates temporally-aligned descriptions. The approach employs a two-stage training pipeline: (1) AVoCaDO SFT fine-tunes Qwen2.5-Omni on 107K synthetic audiovisual captions, and (2) AVoCaDO GRPO applies reinforcement learning with three complementary rewards targeting temporal coherence, dialogue accuracy, and caption quality. The paper is well-motivated by a compelling pilot study and demonstrates strong empirical results across multiple benchmarks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The pilot experiment (Figure 1) provides compelling evidence that joint audiovisual captioning with temporal alignment significantly improves understanding.\n2. The topic itself is of high value. Temporally-aligned audiovisual captions are important for training next-generation video understanding and generation models. \n3. Well-designed reward functions for AV captioning.\n4. The proposed method shows strong improvement over the base model", "weaknesses": "Comparing transcription accuracy against specialized ASR models (e.g., Whisper) would be useful to understand the quality of the model caption. Metrics like Word Error Rate (WER) on the dialogue portions could quantify transcription quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VeKMwpFT6l", "reviewer": "Reviewer_5P4K", "summary": "The paper proposes an audiovisual captioner built on Qwen2.5-Omni with a two-stage post-training pipeline: SFT on 107K curated, temporally aligned AV captions created via a two-stage prompting scheme, and RLHF using checklist rewards combined with a dialogue reward and a length regularizer to curb repetition and excessive verbosity. The paper shows strong results on video-SALMONN-2 testset, UGC-VideoCap, and caption-driven QA transforms of Daily-Omni and WorldSense, plus competitive performance on VDC-Detailed (visual-only). Ablations studies show the effectiveness of proposed rewards and the value SFT data to the model.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main novelty comes from the usage of checklist rewards and dialogue-based reward proposed in RLHF stage, and it proves very effective according to the ablation results (shown in Table 7).\n\n2. The value of the constructed dataset is proven to be very effective across both video captioning and joint audio video QA benchmarks.\n\n3. The paper is very well-written with clear motivation and clear analysis of the results both quantitatively and qualitatively.", "weaknesses": "1. The paper only showcases its strong capability in video-speech related scenarios, but audio could be from also other categories, such as music / general sound. It would be great to incorporate some experiments covering these aspects as well, which will make the contribution even stronger. Some related works in this area for references such as [1]-[3].\n\n2. How does the method perform when the sound / speech is very noisy scenario, does the visual help answering these questions? How is the captioning like in audio-only mode in these benchmarks? It would be more comprehensive if the authors incorprate these results to see the value of vision part only.\n\n3. How does the proposed method generalize to long context audio-visual video captioning and how does different sampling strategies in audio / video affect the final performance.\n\n4. Can such framework work in real-time mode and what are current limitations / trade-offs between latency and accuracy? Maybe some dicussions or future works could be mentioned to showcase the challenges / limitations of existing framework.\n\n[1] Learning to Answer Questions in Dynamic Audio-Visual Scenarios, CVPR 2023.\n\n[2] Tackling data bias in MUSIC-AVQA: Crafting a balanced dataset for unbiased question-answering, WACV 2024.\n\n[3] AVQA: A Dataset for Audio-Visual Question Answering on Videos, ACM MM 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KhDQNVx6ED", "reviewer": "Reviewer_vB4b", "summary": "This paper presents **AVoCaDO**, a model designed to enhance audiovisual video understanding by jointly learning from audio and visual modalities. The authors argue that most existing video captioning methods are overly vision-centric and fail to capture temporal alignment between auditory and visual events. To address this, AVoCaDO introduces a two-stage post-training framework (SFT, GRPO).\nExperimental results demonstrate that AVoCaDO achieves state-of-the-art performance among open-source models on audiovisual captioning, QA, and visual-only benchmarks.", "questions": "Could the authors clarify whether the curated 107K audiovisual caption dataset (SFT and GRPO versions) will be released publicly or partially shared to support reproducibility?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to follow. The core idea is clearly explained, with strong logical flow from the motivation to the experimental results. Each component of the proposed approach is described with sufficient clarity, and the reasoning behind design choices is well supported by empirical evidence. The experimental results are clearly presented and convincingly demonstrate the effectiveness of the proposed method across multiple benchmarks.", "weaknesses": "- **Limited transparency of data construction.**\nAlthough the paper claims to curate a 107K high-quality audiovisual dataset, the process relies heavily on closed-source models (Gemini-2.5-Pro, GPT-4.1). This limits the reproducibility and transparency of the dataset and may raise concerns about data bias or accessibility.\n- **Dependence on proprietary judge models.**\nThe evaluation heavily depends on Gemini-2.5-Pro as a judging model, which may introduce bias or instability in QA-based evaluation. A human or open-source baseline judge would make the comparison more robust.\n- **Lack of technical contribution in model design.**\nWhile AVoCaDO is built upon Qwen2.5-Omni, the paper does not clearly describe how the audio and video inputs are encoded, temporally synchronized, and fused within the model architecture. As a result, the contribution leans more toward fine-tuning and reward engineering rather than introducing a technically novel modeling component. A clearer explanation of the underlying multimodal representation mechanism would strengthen the technical contribution of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13iSx5OJfc", "reviewer": "Reviewer_Zz2V", "summary": "This paper introduces AVoCaDo, an audio-visual video captioner driven by the temporal orchestration between audio and visual modalities. AVoCaDo is built upon the Qwen2.5-Omni, which is enhanced by a two-stage post-training: 107K temporally-aligned audio-visual captions are used for SFT, followed by a reinforcement learning using three reward-based GRPO. Experiments verify the effectiveness of the proposed method.", "questions": "- What is the total training time for the SFT and RL? The calculation of the rewards also requires some time.\n- How to define the five key points in the checklist for computing Reward_c? Will there be other caption cases beyond the predefined five points?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- This paper involves a well-structured two-stage post-training pipeline including both SFT and RL.\n- The proposed 107K high-quality audiovisual caption dataset used in the SFT can be a good resource for relevant topics.\n- The paper is well-written, easy to follow. The proposed method is clearly described.", "weaknesses": "- The ablations lack evaluation on the dataset composition. Tab. 4 reports ablations only for reward terms, not for dataset subsets (TikTok-10M vs. FineVideo etc.). It is unclear whether gains stem from data diversity or model tuning.\n\n- During the RL stage, the paper briefly mentions that 2K caption data is randomly sampled and  8 rollouts are generated for GRPO calculation. It would be better to provide ablation studies on the number of video samples and rollouts used in the RL stage.  Moreover, the visualization of the different reward types during training would also be helpful for readers. \n- The literature review on audiovisual captioning could be more comprehensive, such as Fine-grained Audible Video Description (CVPR 2023), LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025), etc.\n- I appreciate the simple, elegant, and clearly written paper. However, since there are many research papers using SFT and GRPO, I am afraid that the paper does not provide sufficiently new technical insights to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)