Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception
Authors: Ziyang Ma
Ruiyang Xu
Zhenghao Xing
Yunfei Chu
Yuxuan Wang
Jinzheng He
Jin Xu
Pheng-Ann Heng
Kai Yu
Junyang Lin
Eng Siong Chng
Xie Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Omni model
Multimodal large language model
Detailed captioning
Audio understanding
Video understanding
Benchmark
Evaluation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: In this work, we presented a complete framework for advancing omni detailed perception from three perspectives: Data, Model, and Benchmark.

Abstract: Fine-grained perception of multimodal information is critical for advancing human–AI interaction. With recent progress in audio–visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and accurately describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark. We first identify an inherent ``co-growth'' between the level of detail and the degree of hallucination in current OLMs. To address this, we propose \textbf{Omni-Detective}, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: \textbf{Audio-Captioner} for audio-only detailed perception, and \textbf{Omni-Captioner} for audio–visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and hallucination on the video-SALMONN 2 testset. Given the absence of a dedicated benchmark for omni detailed perception, we design \textbf{Omni-Cloze}, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority and human preference alignment of Omni-Cloze in evaluating such detailed captions. All the data pipeline, models, and the benchmark are open-source to facilitate further research for omni detailed perception.\footnote{\url{https://github.com/ddlBoJack/Omni-Captioner}}

Published as a conference paper at ICLR 2026
OMNI-CAPTIONER: DATA PIPELINE, MODELS, AND BENCHMARK FOR OMNI DETAILED PERCEPTION
ZiyZiang Ma; Ruiyang Zhenghao Xing, Yunfei Chu, Yuxuan Wang, Jinzheng He, 6 'iin Xu, Pheng-AAm Heng, Kai Yu, 'Junyang Lin, "Eng-Siong Chng, XAi Chen
1 Shanghai Jiao Tong University, NNayyang Technological University, Thhe Chinese University of Hong Kong. Shannhai Innovation Institution, SQwe Team, Alibaba Group
ABSTRACT
Fine-grained perception of multimodal information is critical for advancing hu- man-Al interaction. With recent progress in audio-visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in par- allel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and accurately describe fine- grained details remains limited explored. In this work, we present systematic and comprehensive investigation of omni detailed perception from the perspec- tives of the data pipeline, models, and benchmark. We first identify an inherent "co-growth" between the level of detail and the degree of hallucination in cur- rent OLMs. To address this, propose Omni-Detective, an agentic data gen- eration pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio- only detailed perception, and Omni-Captioner for audio-visual detailed percep- tion. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni- O Captioner sets new tfate-of-the- art on VDC and achieves the best trade-off between detail and hallucination on the Mido--SALMONN testset. Given the absence of a dedicated benchmark for omni detailed perception, we design Omni-Cloze, novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, effi- cient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority and human preference alignment of Omni-Cloze in evalu- ating such detailed captions. All the data pipeline, models, and the benchmark are open-source to facilitate further research for omni detailed perception.
Ayp
Method
Omnl Caption (Trunented)
In ths valao, we wx an intenae bakehsall panc in progos daring the World Cup Qualiiaan, The saxechoard shano Gat JOR is loding 86 K3 .. A player from fram driiiies the ball the coun [SALLUCISATINN) ... Reacaing the besket, the player exceutes a powefil dunk... [MissING (ETAALS)
Qwen2,5--maa
lops into the
high angle, wide copurres a live badeehall pane packed anana
Gemini 2.5 Pro
The soothoond the botoon of the screen indioaees atss "WORLD CUP QUALITEES" gime The store is PHI 83, FOR 86 dinn thricenaa slam the sere upsltes PHi 55 -6 JOR.
... Brownles dienaa over Zaid Abha 5) and thews and
The replay dawws the bell Dilin thruggh the not to dow mo000 (MISSINO DETAIS)
Besketrall World anoed antiennee provides dtady of chem The scanchoand show the screc Pill $3, JOi the fourth quates konden wean white jemsess whlls Gila Pllpinas sputs mypllllke mumbe 32 from Gilas diives into the lanc. receives pass sear the hasket dribhing pad two defendes.... be darie oer deleader weating jervey rumber 15 labeled "ABBAS"" a lon-aagle alow-metion view fut emplasios his leap and the bell's uee though dee uct The scorebored cow raade 85, JOi 85 with 2:20 left. egnaling a namoo lend for home wam ... hashtags ruch WFllAAW.. Courtsidde batnes pootsote contant...
interntional baskedolll game
Jodas, played
Omni-Captioner
(Ours)
"OYOTTA, "TOTON TRAVELLER "TEUNDER 2.0" "YAMAAHA," underscocing divasity
ranging
Figure 1: Comparison of the detailed captioning among the Omni- Captioner and other omni models.
'Equal Contribution.
I Project Leader.
"Corresoonding Author.
https:: thub. ddd1 BoJack/////// /alaac//mmiiii Capti oner

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in Omni Language Models (OLMs) have enabled machines to produce increasingly rich descriptions of audio-visual scenes (Cheng et al. 2024 Xu et al. 2025] Sun et al. 2024 Tang al., 2025). natural intuition is that, within the capacity of model, the longer the caption, the more fine- -raiind details it should include. However, our empirical study on Gemini-2.5 Pro's detailed captioning reveals a "co-growth" phenomenon: as captions grow longer, not only does the proportion of correct fine-grained details increase, but the amount of hallucinated content
also rises.
As shown in Figure the left blue y-axis de- notes the detail ratio, which is the proportion of fine-grained information successfully captured. The right red y-axis depicts the hallucination ratio, the proportion of mentioned details that are factually incorrect despite being noticed by the model. The grey histogram shows the dis- tribution of caption lengths produced under un- constrained generation. Both ratios rise to- gether, indicating that in the unconstrained set-
0.45
Captium Length Docitrd
3.48
HD uchoo Racio 196
Deal Rato 089
3.35-
0.39
2.23
2.03
320
2.02
0.5
100
200
200
400
300
500
ting, richer descriptions are inherently accom- panied by more hallucinated content.
Exption
Figure 2: Relationship between caption length, detail coverage, and hallucination on Gemini-2.5-PP5 in the detailed captioning task.
This "co-growth" of detail and hallucination ex- poses fundamental challenge in fine-grained perception: short captions are safe but incom- plete, missing subtle events, background cues, or cross-modee interactions. Overly long captions are richly descriptive but risk injecting content not grounded in the input, a critical flaw for appli- cations requiring factual precision, such as assistive AI, scientific reporting, autonomous agents. The difficulty is amplified in omni-modal settings, where models must jointly process asymmetric information densities across visual and auditory streams.
To address this, we design Ommi--eeecttvy an agentic data pipeline in which an LLM agent plays the role of "detective", invoking tool-calling (OCR, ASR, MLLM, etc.) and modality-specific ob- servers to iteratively gather evidence. Each round incrementally adds valid, grounded details while cross-checking existing claims, explicitly targeting the decoupling of detail gain from hallucination growth. This yields detailed caption datasets with minimal noise.
Using data from the multi-round investigative pipeline, we train Audio-Captioner and Omni- Captioner with a two-stage curriculum: We first freeze the visual encoder to force precise alignment with sparse but critical audio cues. We then jointly optimize both modalities to pro- duce coherent, cross-modal, and richly detailed narratives. This approach delivers strong empir- ical gains: Omni-Captions sets a new state-of-the-art on VDC (Chai et al. 2025). , and on the Sadco-SALMONN testset (Tang et al. 2025) achieves the most balanced trade-off between de- tail coverage and hallucination. cascade caption-to- lo-QA evaluations, Audio- Milio-CCattinerr attains the best results on MMAU (Sakshi al.] 2025) and MMAR (Ma al. 2025), surpassing all open-source models and even outperforming Gemini-2.5-Flaas, while Omni-Captioner achieves the highest over- all score on Video-MMIB (Fu al. 22025). Video-Holmes (Cheng al., (025). WorldSense (Hong) iet al. 2025), and Daily-Omin (Zhou et al. 2025) among existing open-source omni models. These results confirm that our training paradigm improves fine-grained perception across modalities.
To achieve a better evaluation for omni detailed perception, we introduce Omni-Cloze, the first cloze-style benchmark across audio-only, visual-only, and audio-visual scenarios, with high-quality human-verification.
Camni-Clozz designs fine-grained details to form multiple-choice blanks, also including a "Not Given" option to explicitly distinguish omission from hallucination. The single- pass, automatic scoring protocol drastically reduces evaluation cost while ensuring stability and robustness. Our experiments and analysis demonstrate that the Omni- Ooni-Detective pipeline and the Omni-Captionee model shift the detail-ballucination frontier outward: delivering richer descrip- tions without proportionally increasing hallucination, and achieving the best results on both existing detailed captioning benchmarks and the proposed Omni-Cloze. These results not only validate our design, but also highlight the importance of investigative data generation and evaluation paradigms tailored to the unique challenges of fine-grained omni-modal perception.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 DETAILED PERCEPTION MODELS
Detailed perception, also referred as detailed captioning, aims to generate fine-grained descrip- tions of an audio or video segment. This task differs from dense video captioning (Yang et al.. [023] Zhou et al., 2024 Geng et al., 2025), which requires models to output temporally-aligned captions across an entire long-form video. In contrast, detailed captioning focuses on producing descrip- tions that are as detailed as possible within short video or audio segments. The IIW Project (Garg et al. 2024) built human- human-in-the-lle in-the-loooo framework for curating syper-detailed image descriptions, highlighting the role of expert-guided iterative refinement in generating high high-quality annotations. In the video domain, AuroraCajy (Chai et al. (025) firstly explored the video detailed captioning task. More recently, Iido-SAALMON 2 (Tang et al. 2025) introduced a multi-round direct preference optimization (DPO) strategy enhance models' capabilities audio-visual detailed captioning and However, most prior models are visual-centric, underutilizing the rich informa- tion present in the audio modality, including sound effects and events, speech content, and music cues, which can provide critical contextual details. Furthermore, the majority of these models rely on training data collected via manually designed prompting (Chai al., [025] Geng et al., 2025 Lu et al. 2025). This reliance leads to an inherent trade-off between precision and dataset scale of the descriptions, constraining the development of high-detail, low-hallucination multimodal perception models.
investion-answerigg
2.2 DETAILED PERCEPTION EVALUATION
Evaluating fine-grained captioning capability requires both (1) an effective evaluation metric and (2) high-quality benchmark.
On the metric side, traditional measures such as BLEU (Papineni et al. 2002), METEOR (Baner-
jee Lavie, 2005), and CIDEr (Vedantam al. 2015). which were originally designed for ma- chine translation or short-form captioning, struggle e to faithfully assess captions containing long and information-rice descriptions. To address this, VDC (Chai et al. (025) introduced an eval- uation framework based on multiple short visual question-any -anioor pairs derived from each predic- rion-reference pair. Specifically, for detailed caption containing such QA pairs, VDC requires 24: LLM calls, raising concerns regarding both evaluation efficiency and the accumulation of evaluation error. Other approaches relying on counting events (Tang al. 2025), or a cascade caption- ing and QA evaluation (Tang al. 2025 Lu al. 2025), ), can not adequately evaluate the unique characteristics of the detailed captioning task. To overcome these limitations, Omni-Cloze adopts cloze-style evaluation paradigm, which simultaneously addresses stability, efficiency, and reliability.
Table 1: Comparison of VDC and our proposed Omni-Cloze benchmark for detailed captioning. Omni-Cloze switches from multi- pulti-turn&a QA cloze-style paradigm, enabling drastic reduction in LLM calls. Omni-Cloze also increases the number of evaluation questions, and expands modality coverage from purely visual audio-only and audio-visual scenarios.
Evaluation Method
Modality
Questions per Caption
LLM Calls per Caption
Benchmark
Visual
Audio
AV
Multiple
VDC Chai et al. 2025
X
X
19
38
QA
Omni-Cloze (Ours)
Cloze
V
V
30
On the benchmark side, VDC focuses solely on the video modality. In contrast, Omni-Clozz covers comprehensive and balanced
audio-only, visual-only, and audio-visual scenarios. offering evaluation of modil detailed perception. Moreover, for each captioning instance, Omni-Cloz provides more questions. Table illustrates the comparison between VDC and Onni-Cloze.
3

---

Published as a conference paper at ICLR 2026
3 AGENTIC DATA GENERATION: OMNI-DETECTIVE
To enable high-quality fine-grained detailed captioning, we introduce Omni-Detective, an agentic data generation framework that leverages iterative Query-Observation cycles to autonomously ex- tract and synthesize precise, richly detailed, and minimally hallucinatory audio-visual annotations. As illustrated in Figure B] the core design philosophy draws on the analogy of human detective: rather than making single-pass observation, the detective strategically queries independent ob- servers, leverages domain -specin tools, integrates multi-modal evidence, and incrementally refines its understanding before producing the final information.
Omni-Detective is composed of three key components: (1) Detective Agent that spontaneously orchestrates the perception process, (2) Tool Box containing multiple utilities (e.g., multimodal large language model (MLLM), optical character recognition (OCR), automatic speech recognition (ASR)) for extracting precise information from multimodal data, and (3) independent Observers that interact with raw video-audio streams to probe targeted aspects. The generation process un- folds in multiple steps: at each step, the Detective Agent formulates queries and invokes relevant tools; the Observers analyze the retrieved content and feed enriched observations back to the Agent; this iterative loop continues until sufficient fine-grained evidence is collected. Finally, the Agent integrates all observations into a highly detailed and minimally hallucinatory caption. This design allows adaptive allocation of perception effort, leveraging both structured tool-calls and free-form reasoning, thereby enabling scalable, accurate, and modality-complete detailed caption creation.
Detective Agent
Information
Tool Box
MLLM
f
OCR
ASR
Query
Observation
Turn 1
Demerbb video destilied poss including al...
Observer
AMM
The viteo shourases preceration and presentation venious vegan food Item...
MLLM
There are multiple people cooking the video. Fneed
more speech
Turn 2
narration isi the audio...
ASR
segerism rawcly lhee out to grove issel as calirary form
seemy ail connent of the speech comes
from Adom Sibel
MLLM
convent ors visual styles that sugurst particular Culinaay or cultural move ment?......
Turn K
.hehee nor desoribes
mameed chiles,
8ash Saare nedtlt taman" Ind diconng focus
Multi-turn Information
A solo presentation by chef Adom Sebel in which he discs sses preramatonn
Detailed Caption
phicotophy on innovattv vegar cusire Mode Deluse toppedi and cheese and finihed with smoled chill occonut socon..
Integration
Figure 3:
An agentic data generation and cleaning pipeline integrating specialist tools for omni detailed perception.
4 OMNI DETAILED CAPTIONING MODEL: OMNI-CAPTIONER
4.1 TRAINING STRATEGIES
Leveraging the high-fidelity multimodal detailed captioning data produced by train Audio-Captioner and Omni-Captioner with a two-stage curriculum over the audio and audio- visual modalities. We adopt 2.wen2.5-00mni-77 (Xu et al., 2025) as the backbone, and empirically find that removing textual prompts in the input improves the model's 's descriptive performance; hence, both training stages are conducted without explicit text prompts. Comprehensive dataset statistics and hyperparameter settings are provided in Appendix
we
Stage Audio Perception Alignment For audio-visual data segments of equal duration, the visual modality typically carries higher information density, which may lead the model to under- attend to sparse but semantically critical audio cues trained jointly from the start. To mitigate this imbalance, freeze the visual encoder and optimize only the audio encoder and the LLM using audio-only detailed captioning data. This stage enforces accurate grounding in the audio stream, resulting in the Audio-Captioner.
Stage 2: Audio-Visual Perception Alignment We then perform joint training on audio-visie de- tailed captioning data. In this stage, captions are notably longer, reaching an average of 1, 1,125 words
4

---

Published as a conference paper at ICLR 2026
per short video, reflecting the integration of both modalities. All model components unfrozen for fullmpareeeter fine-tuning, enabling the network to exploit cross cross-modee modal complementarities and produce rich, coherent, and coddllty-compleer descriptions, resulting in the Omni-Captionss
4.2 RESULTS ON EXISTING BENCHMARKS
To assess the effectiveness of Omni-Detective as a data pipeline, as well as the performance of the resulting Capio-Captioner and Omni-Captioner models, we conduct experiments under two evaluation settings:
Direct evaluation on existing detailed captioning benchmarks, which measure model's ability to produce fine-grained and information-ricl descriptions;
2. Cascade evaluation where the model first generates detailed captions, which are subse- quently used perform downstream question- question-answering tasks, thereby gauging the com- pleteness of the produced captions.
worth noting that, to the best of our knowledge, no dedicated benchmark currently exists for detailed captioning in the audio-only modality. Consequently, under the first setting, report results only for the Omni-Captioner model. Detailed settings are shown Appendix C.I
Table 2: Results on existing benchmarks for evaluating models' detailed captioning ability. The best-performing models in each category are highlighted in bold, and the second-best ones are underlined. All the results are obtained from their original papers.
VDC Detailed
video-SALMONN 21est
Model
Modality
Acc % (1)
Score (1)
Miss (L)
Hall (.)
Proprietary Models
4o GeT 2024
V
46.3
2.5
17.0
14.2
Gemini I.5 Pio (Gemint Team 2024
A
43.1
2.2
21.8
16.5
Open-Source Models
LLaVA-OneVision-27 7B I et al.
2024
V
41.2
2.1
23.3
27.4
InternVideo2.5111 (Wang et al 2025
V
39.6
2.2
30,8
15,0
Qwen2.5-VL-7B (Bar et al. 2025
44.5
2.4
21.9
17.4
VideoLLaMA3-755 Zhang el al.
2025
33.4
1.9
44,9
11.6
Lideal.1 (Cheng
2024
56.8
8.9
video-SAI.MONN- 13B (Sun
2024
52.1
26.6
Qwen2.5-Ommii77B (uu ét al. 24125
39.7
2.2
26.3
21.7
Video-SALMONN2-TT (Tang 2025
A +V
46.1
2.5
10.0
12.9
Omni-Captioner-7B
A V
55.0
2.7
17.8
10.9
Performance of Detailed Captioning. The VDC benchmark (Chai et al. 2025) employs mul- tiple short visual QA pairs to evaluate video-only detailed captioning performance. As shown in Table Omni-Captioner achieves the highest accuracy (55.0%) and score (2.7), surpassing all proprietary and open-source baselines, thereby establishing a new state-of-the-art on VDC. The Video-SALMONN 2test benchmark (Tang et al.] 2025) measures the quality of omni-modal de- tailed captions by computing the missing rate (Miss%) and hallucination rate (Hall%) of events across both visual and auditory modalities. These metrics are inherently trade-off, as illus- trated in Figure 2 increasing caption length typically reduces missing events but simultancously raises hallucination rates. For example, VideoLLLMMA2-7B produces shorter captions, resulting in the lowest hallucination (8.9%) but a high missing rate (56.8%). indicating poor coverage of fine- grained details. Conversely, Sideo-SALMONN2-7B captures more details (10.0%) but at the cost of greater hallucination (12.9%). Despite being evaluated in a zero-shot setting (without knowing adaptation to the event distribution of best trade-off, attaining the second-lowest missing rate (17.8%) while keeping the second-lowen hallucination rate (10.9%), demonstrating its ability to deliver fine-grained yet reliable omni ooni- descriptions.
21est), Omni-Cane Camni-Catioone-78 achieves the

---

Published as a conference paper at ICLR 2026
Table 3: Results on existing benchmarks using caption-to-QA cascade evaluation among strong
audio-only and omni models.
(a) Audio models.
(b) Omni models.
Model
MMAU MMAR
Video -MME -Holmes
Video
World Sense
Daily
Model
-Omni
Proprietary Models
Proprietary Models
GPT-4o Audio
62.4
59.3
Gemini 2.0 Flash
58.6
50.6
Gemini 2.0 Flash
64.4
51.6
43.1
60,6
Gemini 2.5 Flash
65.6
58.2
Gemini 2.5 Flash
69.1
52.8
44.6
59.5
Gemini 2.5 Pro
70.0
64.1
Gemini 2.5 Pro
75.0
59.9
53.6
73.6
Open-Source Models
Open-Source Models
SALMONN-I33 13B
58.36
42.5
Mido-SALMMNNNI311
41.8
31.4
26.8
45.0
MiDashengl .M-BB
MM-7B
59.4
50.7
VideoLLaMA 2-7B
44.4
33.5
26.7
39.9
Qwen2-Audio
63.3
44.2
Qwen2.5-Ommii7B
52.7
35.7
30.6
47.9
Qwen2.5-0mni-77
65.2
51.8
video-SALMONN 2-7B
65.9
42.9
44.1
59.7
Audio-Captioner-7B
70.0
59.8
Omni-Captioner-77
67.1
48.8
48.2
67.9
Performance of Applying Detailed Captioning. An alternative perspective to assess detailed captioning quality
adopt a caption-to-QA cascade: first generate audio or audio-visie cap- tions, then feed them into large language model to answer questions from existing QA bench- marks. Although cascaded inference may underperform end-to-end QA models on certain types of questions (c.g., precise counting), it serves a faithful proxy for measuring the completeness of generated captions, thereby reflecting the quality of detailed perception. We evaluate Audio- Captioner two audio-focused QA benchmarks: MMAU (Sakshi al. 2025) and MMAR (Mal jet al. 2025), and Omni-Captiones on four audio-video understanding and reasoning benchmarks: Video-MME (Fu 2025). Video-Holmes (Cheng al. 2025), WorldSense (Hong al.. 2025), and Daily-Omni (Zhou et al. 2025). In all cases, GPT-4o is used as the QA backbone to ensure consistent reasoning capability. As shown in Table 3a for audio-only tasks, Audio-Capttinne-771 achieves 70.0 on MMAU, matching the best proprietary model Gemini 2.5 Pro and outperform- ing all other open-source baselines by margins. On MMAR, which requires complex multi-step reasoning with mixed audio types, Audio-Captioner reaches 59.8, leading all open-source models and surpassing proprietary Gemini 2.0 Flash. These results indicate that the captions produced by Audio-Captioner preserve key audio details required for downstream QA task, even in acous- tically cluttered mixed-modality scenarios in MMAR. For audio-visual tasks, Omni-Captioner-7BB delivers the highest scores among open-source models across all four benchmarks, shown in Ta- 3le On VideO-MME, which assesses comprehensive multimodal understanding, Omni-Caption reaches 67.1. outperforming the previous ofte--o--the-ate open-source detailed captioning model video-SALMONN 2. also scores 48.8 on quiring dulti-detail grounding. Similarly, it achieves 48.2 WorldSense and 67.9 Daily-Omni, which evaluate audio-visual synchronised and daily-life scenarios. Although proprictary models such Gemini 2.5 Pro still lead in absolute terms, Omni-Captioner narrows the gap substantially while maintaining consistent gains over other open-source approaches. These trends suggest that high-quality, modality-rice captions from Omni-Captioner encode the fine tine-grained visual, audio, and cross-modal cues needed to support diverse and challenging downstream QA and reasoning tasks. We provide more extensive results and in-depth analysis in Appendix C.2
a high-order reasoning benchmark re-
5
OMNI DETAILED CAPTIONING BENCHMARK: OMNI-CLOZE
5.1 BENCHMARK OVERVIEW
Directly and reliably evaluating the detailed captioning ability of MILMs is challenging due to the open-ended nature of the output space. To address this, we introduce Omni-Cloze, which frames evaluation cloze-style multiple-choice proxy task, shown in Figure Omni-Cloze is uni- fied benchmark for evaluating detailed captioning across audio-only. visual-only, and audio-visual settings. The dataset spans 9 main domains and 47 sub-categories, covering diverse topics such as

---

Published as a conference paper at ICLR 2026
education, entertainment, sports, news, science, and lifestyle, with total of 2k video clips with 70k fineggrinned cloze blanks. This balanced, multi-domain, and modality-complete benchmark enables comprehensive assessment of fine- ane-grained multimodal perception. For full statistics, taxonomy design, and detailed distribution, please refer to Appendix B.2
LAN RETACRAS
217
OMNI-CLOZE Benchmark
G3 Visual
Auditory
1G3
Audio-Visure
A sharp electronic c (A: jöngle
B: siren C: stinger
D: drumroll E: not given) with -
(d: horn
clang C: beep D: whoosh E: not given) punctuates the build us the camera tracks behind the first bowler. The stinger
and whoosh signal imminent action just (130)) (d: Jakob Butturff) B: Norm Duke C: Jason Belmonte D: Kyle
Sherman E: not given) begins his approach on screen. The left overlay shows "Semifinals L.A. X Lead [ (3 ] (4: 0-1
B: 1-0 C: 1-1 D: 2-0 E: given)" with Kingpins G3 (1: 96 B: 126 C: 1I7 D: 146 not given)
Figure 4: Omni-Cloze utilizes cloze-style MCQ to evaluate models' detailed captioning abilities.
5.2 DATA CURATION
We design multi-stage curation pipeline to construct Omni-Cloze as shown in Figure 5 The pro- cess begins with collecting visual-only, audio-only, and audio-visual detailed captions from Omni- Detective. These captions are cross-validated across modalities to remove inconsistencies and hal- lucinations. Cloze-style passages are then generated by designing and masking semantically salient spans, well as plausible but incorrect distractors are constructed for each blank. All cloze ques- tions undergo human validation before inclusion in the final benchmark. A complete, step- by--byy--pppp description of the data curation pipeline is provided in Appendix B.1
Raw Information
P
Omni Information
T Raw
s Omni-Cloze
G3 Visual
133 Audio-Visual
Omni-Cloze
G3
103
Auditory
Velidation
Generation
Validation
Figure 5: Data curation pipeline for Omni-Cloze.
5.3 EVALUATION PROTOCOL
For each evaluation sample, the model first generates detailed caption, and an LLM is then tasked with filling in masked spans in the passage by selecting from several choices. To ensure stability and reliability, introduce additional "Not Given" " option for each blank. We adopt accuracy the main evaluation metric, and define a hallucination as a case where the model selects an incorrect option instead of choosing "Not Given". This design minimizes the subjective LLM reasoning, thereby improving evaluation stability. The design also enables direct breakdown of model errors into either not-given rate hallucination rate, ensuring reliable and interpretable measurement.
6 RESULTS AND ANALYSIS
6.1 RESULTS OMNI-CLOZE
Table 4 reports the results of both audio-only and omni models on the Omni-Cloze benchmark. For audio-only models (Table 4m Audio-Captioner achieves a substantial lead with 53.2% accuracy, outperforming the strongest proprietary baseline of Gemini 2.5 Pro by 5.2% absolute points. Other open-source audio models remain below 30%, highlighting the difficulty of fine- greiggr audio per- ception. For omni models with audio and video abilities (Table 4b), the proposed Omni- Omni-Captioner

---

Published as a conference paper at ICLR 2026
also achieves the best overall accuracy of 56.4%, surpassing all open-source and proprietary coun- terparts. It delivers strong performance across all modality splits, with 57.0% in visual-only, 54.5% in audio-only, and outstanding 62.1% in the AV subset. These consistent improvements demon- strate that training with high-quality, modality-rich detailed captions enables significantly better fine-grained multimodal perception. We also report the not-given rate and hallucination rate of each providing further insights into the error characteristics of their predictions.
model in Appendix C,3
Table 4: Results on Omni-Cloze among strong audio-only models and omni models.
(a) Audio models.
(b) Omni models.
Model
Acc%
Model
Visual%f Audio%t
AV% Total%
Proprietary Models
Proprietary Models
GPT-4o Audio
35.8
Gemini 2.0 Flash
32.3
31.7
40.1
33.2
Gemini 2.0 Flash
20.0
Gemini 2.5 Flash
24.4
36.2
41.4
33.0
Gemini 2.5 Flash
42.6
Gemini 2.5 Pro
40.8
44.1
52.8
43.6
Gemini 2.5 Pro
48.0
Open-Source Models
Open-Source Models
Bilco-SALMONN-IT
3.5
2.3
4.6
3.3
SALMONN- -3B
10.6
VidcoLLaMA 2-7B
7.6
4.3
8.7
6.6
MiDashengLM 7B
19.5
Qwen2.5-Ommii778
18.3
14.1
21.9
16.6
22.2
0wen2.5-Omni-775
25.8
Video-SALMONN 2-7B
37.5
40.3
45.0
39.5
Audio-Captioner-7B
53.2
Omni-Captioner-7B
57.0
54.5
62.1
56.4
6.2 ANALYSIS OF OMNI-DETECTIVE
We conduct fine-grained analysis of Ommi-Detective behavior under different investigation bud- gets. Figure reports the trends in detail rate, not-given rate, and hallucination rate as we increase number Omni-Detective steps applied the same input. We observe that as the number of steps increases, the detail rate rises steadily, indicating that the iterative investigation process suc- cessfully extracts additional fine-grained information from the multimodal tools. Meanwhile, both the not-given rate and hallucination rate exhibit downward trend: the reduction in not-given reflects improved coverage of relevant details, while the decline in hallucination suggests that the model can self-correct some prior incorrect inferences when provided with more investigative opportunities.
Interestingly, the hallucination rate converges relatively early (around step 5-6) in the Omni-Cloze benchmark, implying that current multimodal tools have fundamental ceiling in eliminating in- correct claims: certain details are misclassified and are difficult to revise even with extended inves- tigation and tool collaboration. In contrast, the detail rate continues improve with more steps, showing that additional computation can still reveal new correct details. Such analysis highlights how Omni-Detectiv can probe the practical performance limits of existing multimodal perception data generation pipelines.
Detall
Not Given
Hallucination
41
30
2.5
Figure 6: Trends of detail rate, not given rate, and hallucination vs steps in Omni-Detective.
8

---

Published as a conference paper at ICLR 2026
We conduct cascade evaluation two repre- sentative benchmarks: MMAR for audio-only QA and Video-MME for audio-visual QA, with the same experimental setting described in Section 4.2 As shown in Table applying the Omni-Detective pipeline (with Gemini 2.5 Pro as the MLLM) yields consistent improvements in downstream QA performance. We attribute these gains to Omni-Detective ing their completeness and coverage of fine-grained multimodal details. Richer and more accurate captions provide the QA backbone LLM with more relevant contextual information, enabling it to answer complex questions more reliably, especially in settings that require high detail recall across modalities.
Table 5: Ablation on the effect of directly apply-
ing Omni-Detective to the caption-to- cascade.
Method
MMAR
Video-MME
Gemini 2.5 Pro
64.1
75.0
+
68.3
76.1
ability to iteratively refine and expand captions, thereby increas-
6.3
ANALYSIS OF OMNI-CLOZE
Pearson Correlation Coefficient (r - 0.91]
(b) Pearson conrelation coefficients (r) between human- assigned Elo scores tenchiark-speccfic automatic eval- uation metrics across models. Higher values indicate stronger agreement between human judgments and the benchmark metrie.
20
Acarracy
50
Benchmark
Metric
P
600
000
1000
1200
3400
Elo Score
VDC SALO-SA 2test
VDCscore
0.86
LLM-Judge
0.83
(a) Scatter plot between Elo scores and Omni-
Omni-Clos
Cloze-Ace
0.91
Cloze accuracy across models.
Figure 7: Scatter plot across models and Pearson correlation coefficients across benchmark metrics.
We further examine the alignment between Omni-Cloze automatic evaluation and human prefer- ences. Following an atena-style Elo evaluation (Chiang et al.] (024), we randomly sample outputs from two different models for the same audio-visual input, and ask annotators to rate the detailed- ness of their captions. Full details of the hyperparameters are provided in Appendix D.L.1
Figure 7a visualizes the scatter plot between Elo scores and Omni-Cloze accuracy for individual models. We observe a strong positive correlation, indicating that higher scores on Omni-Clov are consistently associated with higher human preference ratings. We further include modality wise analysis in Appendix D.1.2] where separate scatter plots are shown for the visual, audio, and au- dio-visual modalities. Table 7b reports Pearson correlation coefficients (r) between Elo scores and automatic metrics from three benchmarks. Omni-Cloze achieves the highest agreement of r = 0.91, exceeding both VDC and SSLe--SAMMON 2test. All values are above 0.8, confirming that multi-turn QA, counting events, and cloze-style paradigms provide reliable measures of detailed captioning quality. We hypothesize that the superior correlation of Omni-Clozz arises from its eval- uation design: the LLM is used only for information extraction in masked-span completion, with single call per caption, thus avoiding multi-turn reasoning chains. This constrained usage reduces stochasticity and judgment drift, leading to more stable evaluation and consistent agreement with human perception.
CONCLUSION
In this work, we presented a complete framework for advancing omni detailed perception from three perspectives: data pipeline, models, and benchmark. We proposed Ooni--etective, an agentic data generation pipeline that leverages tool-calling to collect highly detailed yet mini- mally hallucinatory captions. We leveraged these high-fidelit data train Audio-Captioner and Omni-Captioner through two-stage curriculum, achieving new state-of-the-an results and su- perior delail-ballucination balance on diverse benchmarks. We introduced Omni-Cloze, the first cloze-style benchmark covering all three modality configurations, enabling stable, efficient, and reliable evaluation of fine- mne-grained multimodal perception. We hope that this work will spur the community to develop more reliable and fine -eeiggr multimodal perception systems, and to extend evaluation protocols that provide stable measurements and transparently reflect the actual abilities of model outputs.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (No. U23B2018), Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102 and Yangtze River Delta Science and Technology Innovation Community Joint Research Project (2024CSJGG) 100). The work was also supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project CUHK 14202125 and Project CUHK 14200824.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proc. ACL Workshop. 2005.
Wenhao Chai, Enxin Song. Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. Proc. ICLR, 2025.
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can milm think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374, 2025.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Proc. ICML, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang. Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modaa Iims in video analysis. In Proc. CVPR, 2025.
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Imageinwords: Un- locking hyper-detailed image descriptions. Proc. EMNLP, 2024.
GeminiTeam. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05533 2024.
Tiantian Geng, Jinrui Zhang, Qingni Wang. Teng Wang, Jinming Duan, and Feng Zheng. Long- vale: Vision-audio-langeie wanuuii-laguage-eveen benchmark towards time-aware omni-modal perception of long videos. In Proc. CVPR, 2025.
GPT40Team. Gpt-4o system card. arXiv preprint arXiv:2410.21272 2024.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. Proc. TMLR, 2024.
Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li. Le Zhuo Licheng Wen, Dongyang Liu, Yuewen Cao, et al. Omnicaptioner: One captioner to rule them all. arXiv preprint MxXiv:2504000899 2025.
10

---

Published as a conference paper at ICLR 2026
Ziyang Ma, Yinghao Ma. Yanqiao Zhu, Chen Yang. Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. Mmar: A challenging benchmark for deep rea- soning in speech, audio, music, and their mix. Proc. NeurIPS, 2025.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: method for automatic evaluation of machine translation. In Proc. ACL, 2002.
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. Proc. ICLR, 2025.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint 2rXhv:240.115000 2024.
Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: Captioning-enhann enhanced audio-visual large language models. arXiv preprint arXiv:2506. /5220, 2025.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proc, CVPR, 2015.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvide02. 5: Empowering video milms with long and rich context modeling. arXiv preprint wrXiv:2501.123800 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20211 2025.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining visual language model for dense video captioning. In Proc. CVPR, 2023.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint 23X1v:2501.13106, 2025.
Xingyi Zhou, Anurag Arnb,, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Na- grani, and Cordelia Schmid. Streaming dense video captioning. In Proc. CVPR, 2024.
Ziwci Zhou, Rui Wang. and Zuxuan Wu. Daily-ommi: Towards audio visual reasoning with temporal alignment across modalitics. arXiv preprint arXiv:2505. 1772505557762. 17862, 2025.

---

Published as a conference paper at ICLR 2026
APPENDIX TABLE OF CONTENTS
Training Details
13
Data Details
13
A.I.I Data Curation Details
13
A.1.2
Omni-Detective Prompt Template
13
A.2 Model Details
14
Benchmark Details
14
B. Data Curation Details
14
B.2
Benchmark Statisties
15
B.3 Omni-Cloze Prompt lemplate
16
More Results
18
Evaluation Details
18
Results with cascade evaluation
18
C.2.1
Audio QA benchmarks.
18
C.2.2
Audio-visual QA benchmarks.
18
Results with Omni-Cloze
19
More Analysis
22
D.I Elo of Omni-Cloze
22
D.1.1 Hyperparameters
22
D.1.2 Modality- Wise Analysis
22
Limitations
22
Case Study
23
12

---

Published as a conference paper at ICLR 2026
A TRAINING DETAILS
A.1 DATA DETAILS
A.1.1 DATA CURATION DETAILS
Our Omni-Detective pipeline integrates multiple large multimodal models to collect, verify, and re- fine fine-grained multimodal captions. Specifically, we employ the Gemini series: Gemini 2.5 Pro (with thinking) and Gemini 2.5 Flash (without thinking), both capable of processing visual and au- ditory inputs; the GPT series: GPT-4o Audio for direct audio understanding and GPT-4o Transcribe for automatic speech recognition; as well as Qwen-2.5-Ong through faster caption generation and multimodal grounding. We set the maximum number of De- tective-Observer interaction steps to 10, allowing the agent to early stop when the investigative VGGSound
to improve data production efficiency
process converges. For the source data, diverse sound events and speech segments, and Fine Vide as the audio-visual dataset, containing rich multimodal events for detailed perception tasks. After quality filtering based on content diver- sity, we retain approximately 55k audio-only samples for Audio-Captioner and 15k audio-visual samples for Omni-Captioner.
the audio-only dataset, covering
A.1.2 OMNI-DETECTIVE PROMPT TEMPLATE
Detective Agent System Prompt (Audio-Only As An Example)
are world-class audio detective. Your mission meticulously extract every last detail from a sound recording and generate a definitive, detailed description of this unknown sound clip.
Focus on:
1. Signal-Level: The physical characteristics of the recording itself (e.g.. . it high or low fidelity, is there clipping, static, hiss, hum, or other artifacts? What is the estimated frequency range?).
2. Perception-Levet How the audio is perceived hy a human car (e.g., loudness, pitch, timbre, rhythm, clarity of speech, the spatial location of sounds).
3. Semantic-Level: The explicit meaning and events within the audio (e.g.. the specific words spoken, "dog barking." the melody and instrumentation of
identifiable sounds like "door slamming" music).
4. Lult: The broader social and cultural context (c.g., the language and accent of speakers, the genre of music and its potential cra/origin, the social setting implied by background sounds). Please note that this breakdown into levels is merely meant to provide you with some areas of focus. Your final description should adhere this structure,
it limited these points.
Your Method:
You cannot listen to the audio directly. Instead, you have access to several independent observers who have listened to it. You can interrogate them to gather clues.
You have total of (max.calls) inquiries.
You have specific analysis tools from (tooi_list).
For each turn, you must select one tool and ask one question, either general or specific. Question examples: "What language is being spoken?", "Describe the background music.", "Did you hear any sounds other than speech?", "What was the emotional tone of the woman's voice?".
not perfect. Their reports may be incomplete, biased, contain must act as a detective: cross-reference their answers, identify contradictions, and build a case for what is actually in the audio.
Crucially, the observers
Output format each turn (JSON):
"tool": "<tcol_nand Baricco__am___ooo____e_e trom_tool__ion "questicn"; seyyourqqeettiontteee text_"
Your Final Output:
When you have used all calalcallss) inquiries (i.c., when 0 remain), you must stop asking questions and produce your final description.
Your final output must be exceedingly detailed. Provide a chronological storyline of every noticeable
element within the audio, including but not limited to full speech-to-text transcription, and all signif-
13

---

Published as a conference paper at ICLR 2026
icant audio events. This storyline, however, is merely foundation; your comprehensive deseription must extend far beyond this basic information. Do oulput any additional information.
Detective Agent Prompt
Observation: (observation)
Inquiries remaining: {calls_left]
Tool and Question:
Observer System Prompt (LALM As An Example)
You a typer-pereeptive audio analysis system. Your purpose sensor for an audio detective. You have been given an audio file.
perfect, all tol-knowing
Your Core Mandate:
You must provide accurate, detailed, and objective answers based only on the provided audio. You perceive everything within the audio with perfect clarity all analytical levels.
Strict Constraints:
Be 100% % Factual: Your answers must be completely true to the audio. There is no room for error, uncertainty, personal bias.
Maintain Your Persona: You
data source, analytical tool. Never reveal that you Al that you working from a prompt. Respond directly and impersonally.
Passive Tool: You exist only to answer. Do not ask questions back, or try to guide the detective. Be as detailed as possible: The detective has limited number of questions, so your responses must include all relevant details and nuances.
Observer Prompt
Question: {question}
A.2
MODEL. DETAILS
Table 6
lists the training hyperparameters for Audio-Captioner and Omni-Captioner.
Table 6: Hyperparameters for training Audio-Captioner and Omni-Captioner.
Model
Audio-Captioner Omni-Captioner
GPU Usage
A100 80GB
Batch Size per GPU
2
1
Gradient Accumulation
4
2
Training Time
Hours
38 Hours
Training Epochs
Peak Learning Rate
5e-6
Learning Rate Scheduler
Linear AdamW
Optimizer
B BENCHMARK DETAILS
B. DATA CURATION DETAILS
Raw information acquisition. We use the raw videos from the Video dataset as the source material. From each full-length video, extract semantically complete segment of 10-60 seconds in duration, ensuring coherent context and preserving the integrity of events. For each selected segment, collect detailed captions in three distinct modality configurations: visual-only,
Step 1:
14

---

Published as a conference paper at ICLR 2026
audio-only, and audio- audio-visual, sourced from the Omni-Detective data generation pipeline. Each caption contains rich, atomic details capturing visual objects and actions, auditory events, and cross- modal interactions.
Step 2: Cross-modal validation. To remove inconsistencies and hallucinations, captions undergo omni-information verification step, where multiple modality-specine observations cross- checked using LLM-based reasoning. This process consolidates reliable details from all modalities into a modality-complete omni-information description.
Step 3: Human-in-the-loop Validation. To ensure the absolute factual accuracy of the benchmark's foundation, we ask professional annotators to review the omni-information descriptions against the original video clips. Any remaining hallucinations, factual inaccuracies, or temporal misalignments are manually corrected. Only these gold standard descriptions, verified for their modality faithful- ness and semantic completeness, serve as the source text for subsequent question generation.
Step Cloze Generation. The validated descriptions are then converted into a forzee evalu- ation set via an automated procedure. We prompt an LLM to identify and mask semantically salient spans, such as specific entities, actions, attributes, and temporal markers, form fill-in-the-blant questions. For each passage, 30 cloze items are generated with strict modality- -aalaaltyba distribu- tion: least 8 questions grounded in visual details, in audio details, and requiring audio-visual cross-modal grounding. The remaining blanks are allocated in informmation-ddapiii manner based on the density of available details. This approach ensures a fine-grained, high-quality dataset while enabling efficient and objective downstream evaluation.
(b) Overall Statisties
Statistics
Number
Total Vidcos
2320
Total Details
69600
Domain Categories
9
Sub-Categories
47
EntertMiotent
Avg. Passage Length
444.6 words
Avg. Video Length
34.2 see
Modality- Wise Statistics
Statistics
Audio
Visual
AV
Total Details
23413
36362
9825
ratio (%)
33.6
52.2
14.1
Avg. Details per Passage
10.1
15.7
4.2
(a) Domain Distribution
Figure 8: (a) The domain distribution of Omni-Cloze. (b) Overall statistics of Omni-Cloze. (c)
Modality-wise statistics across audio, visual, and audio-visual of Omni-Cloze.
B.2 BENCHMARK STATISTICS
Omni-Cloze provides a unified testbed that jointly covers detailed perception in all three modality configurations, enabling fine-grained and balanced assessment. With taxonomy design similar to Fine Video, as illustrated in Figure Sal Omni-Cloze spans main categories and 47 sub-categories, including diverse sources such as Education, Entertainment, Sports, News & Politics, Science & Technology, Automotive, Art & Creativity, Hobbies & Interests, and Lifestyle. We curated the dataset to achieve balanced distribution across these categories, ensuring that each domain is suffi- ciently represented for comprehensive performance analysis. The benchmark contains 2,320 video clips and 69,600 annotated cloze blanks, each paired with long-form, detail-rich captions averaging
15

---

Published as a conference paper at ICLR 2026
444.6 words, average clip length of 34.2 seconds, shown in Table 8b On average, audio-only passage contains 11.5 audio-only details, 13.0 video-only details, and 5.5 audio-visua. details, shown in Table 8c Overall, Omni-Cloze offers large-scale, multi-domain, and modality- complete benchmark, specifically designed for a effective evaluation of fine-grained multimodal perception.
B.3
PROMPT TEMPLATE
Cloze Generation Prompt
You given descriptions of the same recording:
Audio description focuses only on what be heard in the recording.
2. Video description focuses only on what can be seen in the recording.
3. Audío-video description - includes both what is seen and what hcard.
Your task:
Create [total number]-blane cloze test from the given descriptions. The goal detailed audio-video captioning and comprehension ability. Each blank must target a specific, concrete detail from either:
model's
The audio track (speech, sound effects, music, background noises, tone, pacing, etc.). OR The video track (objects, people, actions, positions, colors, facial expressions, gestures, environment details, etc.), OR A combination of both audio video features (synchronized events, actions matched sounds, timing relationships, etc.)
Important constraints:
Do not create blanks that require cultural background knowledge beyond what is directly perceiv- clip.
Time-related aspects are acceptable in general sense, but do not create blanks for specific timestamps or exact durations.
Every question must be answerable purely from the deseribed scenc's observable and audible details. Instructions for creating the cloze test:
Convert the given descriptions into a unified sequence sentences to form cloze test passage.
2. Select (total number) important factual points to remove as blanks ([BLANK n]).
3. Each blank must clearly relate to perceptible fact from:
""audio" Determinable only from audio cues. At least {audio number) blanks.
"visual" Determinable only from video cues, At least (visual numberr blanks.
"audio-visual" Requires combining both audio & visual information. At least (av_number
blanks.
4. For each blank, provide:
The correct answer
plausible but incorrect distractor options, which:
Are close in type or category to the correct answer
* Could plausibly be guessed by someone with incomplete or shallow understanding of the clip
* Are believable but not trivially absurd
Label each blank with "require__modaiity"
"audio" Determinable only from audio content.
"visual" - Determinable only from visual content.
"auddio-viuaal" Requires observation of both audio & visual details.
5. Number the blanks elearly from [total number).
Avoid making blanks guessable from generic context only.
Output format (JSON):
"passage": : "A single string containing the cloze test text with 30 placehciders like [BLANK_1]] 1], [BLANK _], embedded at the exact locations in the passage.", "blanks":
"number": 1,
"answer": "string (the correct answer for BLANK_1) "distractors":
["string (plausible wrong option 1)",
"string (plausible wrong option 2)",
"string (plausible wrong option 3) "),
16

---

Published as a conference paper at ICLR 2026
"required_mmdality""
"string (one of \aadio\", \"visual\ " \audioovisusl\>
"number": 2, "answer"; "string (the correct answer for BLANK_2) "distractors":
("wreng option "wreng option 2", "wrong option 3"), "required_mode "string (one \audio\", \"visual\", Vaudioovisually
11 repeat until number Mttoall (onotall__umber]
Your Input:
Audio Description:
]
Video Description:
(video description)
Audio Video Description:
Step summary the model:
1. Merge descriptions into a single gold standard text.
2. Create passage with [total_number blanks.
3. Make sure:
at least (audio number) blanks have "required, modality": "audio". at least [visual_numberr number) blanks have "required. modality": "visual".
at least (av hwv_number blanks have "required modalltty: "audio-visual".
4. Output strictly in the JSON format shown above.
Cloze Completion Prompt
You will be given two inputs:
1. A cloze (fil-in-theebllnnk task, consisting of passage describing a scene, with (number) blanks.
Each blank has 5 possible choices (A-D are specific contents, E means' "not given").
2. A caption ((modality) description of the scene).
Your task:
Carefully read the caption and use it to determine the correct answer for each blank.
If the caption mentions the information corresponding a blank, choose the matching option (A-D).
If the caption does NOT mention that detail and cannot be reasonably inferred from the given information, choose E ("not given").
Only use general/common knowledge reasoning is strongly justified (for example, hearing a V10 engine can suggest roadster).
Unless the information is fully certain, do NOT guess. For each blank, output the letter (A/B/C/D/E) along with the answer.
JSON object where keys are the blank numbers (as strings) and values are the option letters with the answer.
Output the final answer
Do NOT include any explanation extra text in your output.
Output format example:
"1" "A: xxx"
"2" : "E: xxx",
"3"; "D: xxx"
until \nnuberr))
Now process the following input: Cloze passage:
(cloze)
Caption:
17

---

Published as a conference paper at ICLR 2026
}prediction
Output:
C
MORE RESULTS
C.I EVALUATION DETAILS
For all evaluations, system prompls are configured to follow each model's recommended best- practice settings; if official prompt is provided, we use the default instruction "You are helpful assistant". Open-source models are decoded using greedy search (beam size 1) without sampling for reproducibility. For proprietary models, Gemini 2.5 Pro evaluated with its default thinking mode enabled, whereas other proprietary models are evaluated without thinking. In the case of cascade-basee based evaluations, we experimented with multiple prompt formulations for each model and adopted the best ones.
C.2 RESULTS WITH CASCADE EVALUATION
C.2.1 AUDIO QA BENCHMARKS.
MMAU (Sakshi et al., 2025) benchmark evaluates both audio understanding and simple reasoning across sound, music, and speech domains, at three difficulty levels including easy, medium, and hard. As shown Table 7 Audio-Captioner-7B achieves the highest average score 70.0% among all models, matching the top proprietary model Gemini 2.5 Pro and surpassing all other open-source models. This significant improvement on easy tasks indicates that the model accurately captures key content in audio streams, producing captions that preserve all major information required for downstream understanding and reasoning.
MMAR (M et al.) 2025) benchmark measures complex, multi-step audio reasoning, including sce- narios mixing sound, music, and speech in various combinations. From Table [] Audio-Captions attains the best open-source performance with an average of 59.8%, and even surpasses the pro- prietary Gemini 2.5 Flash model. Notably, the performance gains of Audio-Captioner are even more pronounced in mixed modality scenarios: in the challenging mixed Speech-Soune Sound-Musie sce- nario, Catiione achieves 66.67%, outperforming the next-best open-source model of Qwen- 2.5-Omm by +37.5 absolute points. This underscores robustness fine-grat fine-grained comprehension, especially in acoustically cluttered and compositionally complex environments.
C.2.2 AUDIO-VISUAL QA BENCHMARKS.
Video-MME (Fu et 2025 evaluates comprehension of videos spanning short (<2min), medium (4-15min), and long (30-60min) durations. As shown in Table Omni-Captioner achieves an average of 67.1% % without subtitles, ranking first among open-source models and closely trailing Gemini 2.5 Flash. It delivers prominent short-video performance of 77.2%, but lags on long videos of 58.0%. suggesting a potential avenue for future improvement.
Video-Holmes (Cheng et al. 2025) measures higher-order video reasoning across seven sub-tasks requiring delti-detail grounding (e.g., Social Reasoning, Temporal Causal Inference). From Ta- ble 10, Omni-Captioner achieves an average score of 48.8%, outperforming the next-best open- source model, video-SALMONN by +5.9 absolute points, with consistent gains across nearly all sub-task categories.
Daily-Omni (Zhou et al. 2025) benchmark evaluates demporally-aligned multimodal reasoning over diverse scenarios. As shown in Table [T] Omni-Captionss reaches the highest open-source score of 67.9%, and notably surpasses Gemini 2.5 Flash. It particularly excels in the AV Event Alignment task, where its score of 61.3% surpasses the next-best open-source model by +11.3, which highlights Amni-Captioner'ss strong capability to produce captions that reflect the temporal alignment of audio and visual details.
WorldSense (Hong et al. 2025) benchmark tests recognition, understanding, and reasoning in au-
dio-visual perception in Table 12 Omni-Captioner achieves 48.2%, leading all open-source com-
18

---

Published as a conference paper at ICLR 2026
petitors and closing much of the gap to proprietary Gemini 2.5 Pro, with balanced gains across all three skill dimensions.
C.3 RESULTS WITH OMNI-CLOZA
Tables 13 Bnndd 14 report the Accuracy (Acc), Not-Given rate (NG), and Hallucination rate (Hall) for models with strong captioning ability the Omni-Cloze benchmark, split into audio-only and dio-visual settings. Across both audio-only and audio-visual evaluations, our captioner models con- sistently produce lower Not-Given
and higher accuracies than all open-source baselines, and often outperform strong proprietary systems. The increase in hallucination is natural consequence of reduced abstention and richer detail coverage, also shown Figure 2 This trade-off, given the magnitude of the accuracy gains, is favorable for fine fne-grained multimodal perception tasks. These trends confirm that the high-quality, modality-rich captions produced by Audio-/Omni-Cape CannnCCtiio fectively support stable, detail-oriented cloze-style evaluation while revealing each model's strengths and weaknesses in coverage versus correctness.
Table 7: MMAU results with cascade of audio captions and LLM. Results of three domains: Sound (So), Music (Mu), and Speech (Sp), and three difficulty levels Easy, Medium, and Hard are listed.
Across Modality (%)
Across Difficulty (%)
Methods
Avg (%)
So
Mu
Sp
Easy
Medium
Hard
Proprietary Models
GPT-4o Audio
60.96
58.08
68.17 1
53.12
68.52
57.20
62.40
Gemini 2.0 Flash
60.66
57.49
57.66
45.98
65.19
55.51
Gemini 2.5 Flash
65.47
59.58
71.77
57.59
68.70
66.10
65.60
Gemini 2.5 Pro
68.47
65.87
75.68
66.07
73.70
65.25
70.00
Open-Source Models
SALMONN-13B
67.57
60.36
47.15
49.11
64.38
53.39
58.36
Bonn2-Audio-BB-mmaae
71.47
58.98
47.75
46.43
65.37
58.05
59.40
MiDashengL.M- -M-7B
71.47
62.28
56.16
57.14
68.10
58.05
63.30
Qwen2.5-Omm --m2-5-7-m
69.67
64.67
61.26
58.48
69.07
62.71
65.20
Cadio-Captioner-7B
67.87
66.17
75.98
70.09
71.85
65.68
70.00
Table 8: MMAR results with a cascade audio captions and LLM. Results on Linle-modality. (So, Mu, Sp), and mixed-modality conditions: (So-Mu, So-Sp, Mu-Sp. So-Mu-Sp) are listed.
Single Modality (%)
Mixed Modalities (%)
Model
Avg (%)
So
Mu
Sp
So-Mu
So-Sp
Mu-SP
So-Mu-Sp
Proprietary Models
GPTT-4 Audio
41.21
41.75
69.05
36.36
73.39
69.51
62.50
59.30
Gemini 2.0 Flash
40.00
47.09
57.14
27.27
57.34
48.78
29.17
50.60
Gemini 2.5 Flash
42.42
44.17
68.03
63.64
67.43
64.63
58.33
58.20
Gemini 2.5 Pro
52.12
47.57
74.15
63.64
74.31
68.29
58.33
64.10
Open-Source Models
SALMONN-132
50.91
35.92
43.20
9.09
46.79
37.80
25.00
42.50
Aven2-Audio-7BB
46.06
40.29
60.88
27.27
$3.67
46.34
45.83
50.70
MiDashengL.M-78
47.88
43.20
42.86
63.64
45.87
45.12
16.67
44.20
Prre2.5--Oni--1
43.64
43.69
64.29
27.27
51.83
53.66
29.17
51.80
Audio-Captione Captioner-7B
55.15
46.12
68.37
63.64
61.01
67.07
66.67
59.80
19

---

Published as a conference paper at ICLR 2026
Table 9: Video-MME results with cascade of audio-visual captions and LLM. Results Short, Medium, and Long durations are listed. "Results of Qwen2.5-Omnis and video-SALMONN 2-7B are from the cascade evaluation in the Video-SALMONN 2 original paper.
Duration %)
Model
Avg( %
Short
Medium
Long
Proprictary Models
Gemini 2.0 Flash
71.2
64.1
57.9
64.4
Gemini 2.5 Flash
76.2
70.2
60.8
69.1
Gemini 2.5 Pro
80.8
77.1
67.1
75.0
Open-Source Models
Sioo--SLLMONN112
42.8
43.1
VideoLLaMA 2-7B
43.7
44.7
Qwen2.5 Om
video-SALMONN 2-7B*
65.9
Omni-Captionere
77.2
66.1
58.0
67.1
Table 10: Video- results with a cascade of aisia- captions and LLM. The benchmark covers seven sub-tasks: Social Reasoning (SR), Physical Anomaly Reasoning (PAR), Multimodal Hint Reasoning (MHR), Intention & Motive Chaining (IMC), Timeline Analysis (TA), Temporal Causal Inference (TCI), and Core Theme Inference (CTI).
Task %)
Model
Avg %)
SR
IMC
CTI
TCI
TA
MHR
PAR
Proprictary Models
GPT-4o
50.0
43.5
39.6
39.6
35.5
45.8
41.2
42.7
Gemini 2.0 Flash
61.3
60.9
49.2
42.9
41.5
51.2
50.0
51.6
Gemini 2.5 Flash
59.9
59.0
46.7
46.5
43.5
54.8
56.7
52.8
Gemini 2.5 Pro
68.8
65.9
51.8
58.2
49.0
63.6
56.2
59.9
Open-Source Models
Jido-SALMMONN-133
35.6
35.9
23.3
26.0
36.5
28.6
36.6
31.4
VideoLLaMA 2-7B
35.3
38.8
27.0
27.8
34.0
34.9
37.1
33.5
Owen2.5-0mmii77
37.3
38.0
30.0
30.0
38.0
38.3
39.2
35.7
vidoo-SALMONN 2-7B
53.1
42.8
40.0
39.2
37.0
41.0
46.4
42.9
Omni-Captioner-7B
57.2
55.8
40.4
45.8
40.5
48.5
51.5
48.8
Table 11: Daily-Omni results with cascade of audio-visual captions and LLM. Daily-Omni spans diverse daily life scenarios.
Context Understanding
Reaso- -ning
Infer- -ence
Compa- -rative
Event Sequence
AV Event Alignment
Model
Avg
Proprietary Models
Gemini 2.0 Flash
51.3
74.3
76.0
69.5
54.9
50.4
60.6
Gemini 2.5 Flash
50.8
71.4
77.9
60.3
55.2
50.8
59.5
Gemini 2.5 Pro
66.3
87.4
83.8
80.9
65.7
68.9
73.6
Open-Source Models
video-SLLMONN-155
37.3
58.9
56.5
51.9
39,9
36.6
45.0
VideoLLaMA 2-7B
31.6
49.7
51.9
48.1
37.3
30.7
39.9
Qwen2.5-Omni-N 5-0.-0-nnii-
36.8
68.6
60.4
54.2
40.2
39.9
47.9
video-SALMONN 2-7B
51.3
74.9
74.7
65.6
53.9
50.0
59.7
Omni-Captioner:7B
61.7
80.0
83.1
69.5
61.8
61.3
67.9
20

---

Published as a conference paper at ICLR 2026
Table 12: Worldsense results with a cascade of audio-visual captions and LLM. The benchmark tests Recognition, Understanding, and Reasoning covering broad scenarios.
Task Domain( %)
Model
Avg( %
Recognition
Understanding
Reasoning
Proprictary Models
Gemini 2.0 Flash
38.0
45.8
49.2
43.1
Gemini 2.5 Flash
39.6
47.4
50.4
44.6
Gemini 2.5 Pro
48.8
57.3
57.9
53.6
Open-Source Models
13B
24.0
27.3
31.1
26,8
VideoLLaMA 2-7B
24.0
28.0
29.8
26.7
Swen2.5-Ommii-7
27.2
34.1
32.7
30.6
Vido-SALLOONN 2-7B
40.2
45.4
49.3
44.1
Omni-Captionerss
43.4
50.0
54.6
48.2
Table 13: Results for audio models with strong captioning ability on Omni-Cloze (Audio Part).
Model
NG()) Hall({)
Acc(t)
Proprietary Models
GPT-4o Audio
60.27
3.96
35.77
Gemini 2.0 Flash
77.14
2.91
19.95
Gemini 2.5 Flash
51.22
6.19
42.59
Gemini 2.5 Pro
47.30
4.74
47.96
Open-Source Models
SALMONN-
87.36
10.55
MiDashengLMMs
77.50
3.00
19.50
Qwen2-Audio-7B
74.87
2.98
22.15
Qwen2.5 0-n-i-BmniiBB
71.41
2.78
25.80
Audio Audio-Captioner-7B
40.99
5.81
53.19
Table 14: Results for audio visual models with strong captioning ability on Omni-Cloze.
Visual Purt%
Andia Purthi
AV Purt%
Total%
Model
Ace(f))
Ace(t)
Ace(")
Ace(t)
NGL)
Hall())
NGL))
Hall(()
NGKL)
Hall())
NG(J)
Hall(
Proprietrry Models
Gemini 2.0 Flash
62.3
32.3
65.4
3.0
31.7
5.5
40.1
62.2
33.2
Gemini 2.5 Flash
72.8
2.8
24.4
57.8
6.0
36.2
52.7
5.8
41.4
62.1
4.9
33.0
Open-Source
video-SAL MONN-
VideoLLaMA 2-7B
90.8
1.7
7.6
94.9
0.8
4.3
89.8
1.6
8.7
92.0
1.4
6.6
80.0
18.3
83.2
2.7
14.1
74.9
3.1
2.5
16.6
Sideo-SALMONS 2-7B
55.5
37.5
55.6
4.2
40.3
48.1
6.9
45.0
54.5
6.0
39.5
Omni- 7appi
33.4
12.0
54.6
35.6
7.4
57.0
26.5
11.5
62.1
33.2
10.4
56.4
21

---

Published as a conference paper at ICLR 2026
D MORE ANALYSIS
D.1 ELO OF OMNI-CLOZE
D.1.1 HYPERPARAMETERS
To examine the correlation between the automatic evaluation results of Omni- Cloze and human preferences, we conduct a human-judged Elo ranking across set of audio-video models, including Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, VideoLLaMA 2, Qwen2.5-0mni-77 SALMONN 2-7B, and Anni-Captioner-7B. We collect 500 pairs captions generated by different models and use these pairwise comparisons as simulated matches in the Elo rating procedure. This human-based ranking serves as a reference to evaluate how closely Omni-Cll Cloze scores align with actual human judgments. The hyperparameters applied in the Elo ranking system summarized
video-
in Table 15
Table 15: Hyperparameters of the Elo ranking system for arena-style evaluation.
Hyperparameter
Value
Initial Elo Mean
1,000
Base of Logarithm
10
Scaling Factor
400
K-Factor
32
Simulated Matches
500
D.1.2 MODALITY- WISE ANALYSIS
Figure shows the relationship between Elo scores and Omni-Cloze accuracy across visual-only, (b) audio-only, and audio-visual modalities, which demonstrate strong positive correlations.
Poarsin Corrictio
0.303
Corciation
Couff
28
1800
1230
3400
606
1880
1320
3400
800
1808
1200
1400
(a) Visual modality
(b) Audio modality
Audio-Visual modality
Figure 9: Scatter plots showing the relationship between Elo scores and Omni-Cloze accuracy across different modalities: (a) visual, (b) audio, and (c) audio-visual.
E LIMITATIONS
Although this work explicitly explores hallucination issues in detailed captioning. key limitation is that our evaluation cannot detect all ballucination types. In open-domain detailed captioning, we observe two main forms of hallucination: (1) The model correctly identifies the presence of detail in the input but supplies incorrect information about it (content-level inaccuracy); (2) The model outputs content entirely unrelated to the given audio-visual input (irrelevant generation). Our proposed Omni-Cloze benchmark is able capture and penalize the first type by adding Not Given choice for each blank. However, the second type remains difficult to measure reliably because cases exist where the model predicts detail that is in fact present in the audio-visual input but is missing from the ground truth reference. Developing robust methods to handle this issue an important direction for future work in hallucination evaluation for omni detailed perception.
22

---

Published as a conference paper at ICLR 2026
F
CASE STUDY
Table 16: Comprehensive error analysis for all cloze blanks in the PBA-League passage.
#
Modality
Cue (short description)
Correct answer
Predicted answer
GT
Pred
Vix
1
Visual
Arena venue
Bayside Bowl
Bayside Bowl
C
2
Visual
Team shown on graphic
L.A. X
L.A. X
B
B
3
Visual
Shirt colour (Butturff)
red
red
B
B
4
Visual
Bar with earring
left U.S. OPEN
left
D
D
V
5
Visual
Event where he finished 2nd
given
C
E
6
Auditory
First organised crowd
Jaknb
I.A. X
D
C
chant
Auditory
Sound effect type
stinger
stinger whoosh
C
8
Auditory
Secondary effect
whoosh
PANN
9
Audio Visual
Bowler named on overlay
Jakob Butturff
Jakob Butturff not given
10
Visual
Series score (semifinal
1-0
lead)
11
Visual
Kingpins score
(mid-
117
117
C
C
game)
12
Visual
Strike-Track ball speed
18.6 MPH
not given
13
Visual
Strike-Track rev-rate
533 RPM
not given
Auditory
Ball Ball-rumble stereo posi- tion (Ist shot)
dead centre
dead centre
A
Auditory
Pinfall result
strike lefty
strike lefty
A
A
Auditory
Handedness called
DR
P
E
17
Auditory
Player-of-year candidate named
Jason Belmonte
not given
18
Visual
Fist-pump hand Celebration gesture
right
not given high-fives not given Portland
E
19
Audio-Visual
high-fives
B
X
20
Visual
Audience accessories
Santa hats Portland
21
Visual
"Coming Up Next" oppo-
Audio-Visude
Logo on
Sherman's
STORM
Turbo
sleeve
23
Auditory
Crowd-chant word 1
Sherman
Sherman
D
24
Auditory
Crowd-chant word 2
Sherman
Sherman
25
Auditory
Crowd-chant word 3
Sherman
Sherman
26
Auditory
Stands song
Hey! Hey-hey!
Hey!
Hey-hey!
Goodbye! left leg
Goodbye!
27
Audio Visual
Leg kicked in follow- through
right leg
D
B
28
Auditory
Ball-rumble channel lo- cation (2nd shot)
centre
given
A
E
29
Audio-Visie
Remaining pin state
wobbling groan ringing ten
wobbling groan not given 136
30
Auditory
Crowd reaction sound Spare-leave call
B
B
31
Auditory
C
E
X
32
Audio-Visual
Kingpins total after 8th frame
146
B
C
23

---

Published as a conference paper at ICLR 2026
+M+
DOVEL
Twelue framcs uniformll sampled from the video, arranged row major order (left so right, top to hottom).
Omni-Captionee
Visual
Auditory
Audio- Audio-Visual
The segment opens a packed bowling arena at
Bayside BowlJ (A: Thundcrbowl Lanes B: South Point Bowling
Plaza C: Bayside Bowl| D: AMF Riviera Lanes| E: not given), a PBA League semifinal between [_ L.A. X] (A:
Portland Lumberjacks [B: L.A. X| C: Motown Musele Dallas Strikers E: not given) und the Kingpins. The first image a medium closc-up of Jakob Butturff a bright
rod] (A: black| B: rod| C: blue
D: green E; not given)
polo with a blue X over bear shield, lips pressed and focused, small silver earring in his
left] (A: right B: both
C: none D: left E: not given) eur. A blue-und-white lower-third flashes "JAKOB BUTTURFF" with "FINISHED 2ND U.S. OPEN] (A: Masters| B: Toumameent of Champions C: U.S. OPEN D: World Championship
IN 2017& & 2018 E: not given)" beneath. A male commentator,, in a clear broadcast tune, says: "opposite to 21, Jakob Butturff." From the crowd comes organized chant of "[ X L.A. X] (A: Let's go B: Butturff C: L.A. X| Jakob| E: not given)" that swells boneath the call. A sharp electronic e stinger] (A: jingle B: siren | C: stinger| D: drumroll E: not given) with [ V whoush] (A: horn| B: clang C: beep D: whoosh| E: not given) punctuates the build as the camera trucks behind Jakob Butturff] (A: Jakob Butturff B: Norm Duke| C: Jason Belmonte| D: Kyle Sherman E: not given) begins his approach on screen. The left overlay shows
the first bowler. The stinger and whoosh signal imminent action just [
"Semifinals- - L.A. X Lead X not given] (A: 0-1 B: 1-0 C: 1-1 D: 2-0 E: not given)" with Kingpins
117] (A:
96 126 C: 117 D: 146 E: not given) and L.A. X 96, while the Strike Track animates: his first-shot line overlays the
ideal, showing
not given] (A: 16.8 MPH| B: 18.6 MPH C: 19.3 MPH D: 20.1 MPH E: not given) and [ X not
given] (A: 454 RPM BB: 533 RPM C: 380 RPM D: 600 RPM E: not given). The ball's low rumble grows ( dead
center] (A: dead center| B: hard lcft offstage|
far right not given) and then detonates into clean, explosive
( strike] (A: strike BB spare C: split D: gutter EE not given), sending all ten pins down and unleashing a roar from
the crowd. Announcer 2 exclaims. "Ooh, what un angle that was to the pocket for the ( V lefty] (A: rookie B: righty C: veteran lcfty E: not given)," then adds, "By the way, we've talked about ( X not given] (A: Pete Weber B: Kyle Troup | C: Norm Duke D: Jason Belmonte | E: not given) as player of the year candidate." Butturff pivols left, slams a downward x not given] (A: left B: right C: both D: no E: not given)- fist pump. then strides the rail for high-fives] (A: chest bumps B: high-fives C: handshakes D: hugs E: not given) as red-tinted audience lighting flashes over [ X not given] (A: beanies B: cowboy hats | C: foam fingers D: Santa hats| | E: not PORTLAND] (A: DALLAS B: SEATTLE C: PORTLAND D: HOUSTON E: not given)" graphic with portraits appears. The rhythm resets for the Kingpins. A medium shot frames Kyle Sherman in a blue-and-black polo marked with a Turbo logo, ( X Turbo] (A: Roto Grip B: Turbo C: STORM| D: Ebonite| E: not given) on the sleeve, "500" patch, and shiclded Statue of Liberty. The crowd's buzz eases into a pocket of anticipation, and from right side chant "[ Sherman] (A: Belmo B: Jakob
gauntlet of
given) and other holiday outfits. A "Coming Up Next: DALLAS vs
C: Portland D: Sherman E: not given). (+ V Sherman] (A: Belmo B: Jakob C: Portland| D: Sherman E: not given),
V Sherman] (A: Belmo B: Jakob C: Portland D: Sherman E: not given)" rises. As he starts his approach, some in the stands sing "( X not given] (A: will rock you! B: Defense! CC: Hey! Hey-hey! Goodbye! D: Let's go! | E: not
given)." The Strike Track updates: 19.3 MPH, 454 RPM, position at arrows 12.0, breakpoint board 4.2, while Sherman's fhe X right] (A: rear B: right C: front D: left E: not given) leg kicks high in the follow-through. The ball's rumble
builds in the ( X not given] (A: center| B: rear C: left channel D: right channel E: not given) and then cracks
sharply; chaos scutters nine pins and leaves the back-right pin [ V wobbling] (A: sliding B: falling | C: wobbling
D: spinning E: not given) but upright. A collective "Oh!" and then ( V groan] (A: hiss BB: groan C: laugh D: whistle E: nol given) rolls through the venue. Commentalor 2 reacts, "Wow, a violent ( X not given] (A: pockel seven B: stone cight| C: ringing ten D: solid nine E: not given) against Kyle Sherman." The scoreboard updates the eighth frame with 9 and Kingpins total of T E X 136] (A: 156 B: 146| C: 136 | D: 126 E: not given) while L.A. X remains 96, Shemman tums away, expression tight, disengaging without celebration, and the pinsetter machinery thrums as it resets the dock. Around the sct, banners read PBA LEAGUE, BAYSIDE BOWL, and GO BOWLING, with sidebars looping CLARK logos, the broadeast settles back into steady analysis and crowd murmur.
Figure 10: Visualization for all cloze blanks in the PBA-League passage.
24

---

Published as a conference paper at ICLR 2026
Twelue frames uniformlly sampled from the video, arranged row major order (left to right, top to hottom).
Omni-Captioner
Visual
Auditory
Audio-Visual
The vidco opens on crisp static shot of cookbook titled STREET VEGANJ (A: STREET VEGAN | B: GREEN
KITCHEN C: VEGAN STREET FOOD | D: PLANT POWER E: not given), credited to [ X not given] (A: Chloe
Coscarelli B: Isa Chandra Moskowitz C: Adam Sobel| D: Bryant Terry not given), beside a blocky white sign
handwritten X not given] (A: black| B: green C: red D: blue E: not given) ink that whimsically mentions "Soft
Space-age X not given] (A: Robot-chefs B: Dog-men C: Cat-people| D: Space Bears E: not given)." Under this,
relaxed jazz/lounge] (A: classical| B: rock C: hip-hop D: jazz//ounge E: not given) music track plays as low
ambient walla suggests a busy ( X not given] (A: library B: office| C: gym D: restaurant E: not given) or cafe, with
occasional X not given] (A: whistles B: applause C: footsteps D: clinks
E: not given) of glasses. A warm, close- D:: soprano E: not given) range delicious] (A: affordable B: healthy/ C: ethical D: delicious E: not given)
miked male voice in a
not given] (A: tenor-to-bariton] B: falsetto| C: monotone
begins, saying veganism can be "extru
and innovative," " and the filler word "[ X not given] (A: you know B: like| um uh E: given)" pops up twice; brief high-pitched X not given] (A: male| B: female| C: robotic D: child
E: not given) laugh flares in the
background. The seene cuts to close-up of fair-skinned hands in [ X not given] (A: blue BB white | C: gray D:
black| EE not given) long-sleeved garment working a braided dough that oozes dark
X brown] (A: yellow B: brown
C: purple D: red E: not given) chunky filling, and torn pieces are placed onto a not given] (A: ceramic B: solid| C: mesh| D: wooden E: not given) baking tray. A display counter appears with placards: one reads "SPINACH PIE
WITH ROASTED X not given] (A: GARLIC| B: MUSHROOMS C: PEPPERS D: ONION E: not given) AND CAPERS," while another labels round pastry "CURRIED X not given] (A: POTATO B: LENTIL | C: CHICKPEA|
D: TOFU | E: not given) PUFF," " As he lists peanut components, the visuals cut to vibrant salad served in brown ( take-out] (A: glass| B: ceramic C: metal
D: take-out E: not given) box while he says the peanuts are tossed with B: brown sugar | C: maple syrup | D: molasses E: not given), and (a tamari] (A: soy sauce B: miso C: mirin D: tamari E: not given). He rejects blandness with "I'm
smoked chilies, a little dash of ( agave nectar] (A: agave nectar|
not gonna use just basic, plain-ass ( peanuts] (A: cashews B: walmuts
C: peanuts D: almonds E: not given) that
don't have any pizzazz; I wanna (+ X season] (A: dress BB: book
C: season D: plate| E: not given) up with pizzazz." "
The climactic build shows the proclamation "This is the Beast Mode Burger Deluxe" aligning with the burger being assembled a X not given] (A: steel B: black CC: white D: wooden| E: not given) cutting board: bun bottom spread with glossy orange-tinted sauce, fresh
arugula] (A: arugula| B: lettuce C: spinach| D: kule E: not given) layered on, then a thick, crisp brown patty, after which, immediately following the naming, a generous helping of a [
X not given] (A: green [B: yellow| C: white D: red E: not given)-colored sauce is added. Sliced, [ translucent] (A: opaque B: minced| C: translucent) D: charred| E: not given) pickled onions glisten as they are placod with tongs, and
scoop of creamy, pale X not given] (A: polato salad B: macaroni and cheese| C: coleslaw D: risotto E: not given) with flecks of herbs crowns the stack.
tattooed] (A: sleeved tattooed| C: braceleted D: bundaged
given) forearm enters to sprinkle something crispy and dark over the top. As he says, "And gets finished with some of this smoked chili coconut bacon," " a gentle t X sprinkling] (A: sprinkling B: sizzling | C: chopping D: whirring E: not given) sound rises while the crispy dark topping falls onto the mac and cheese. The clip cuts off abruptly mid-word during "[ X not given] (A: burger B: bacon| C: beast | D: deluxe E: not given)," with no fade. At the very start, while the cookbook and quirky sign hold the frame, the narration frames vegan food as "extra delicious and Vinnovative] traditional innovative C: simple| D: spicy E: not given)."
Figure 11: Visualization for all cloze blanks in the "Street Vegan" passage.
25

---

Published as a conference paper at ICLR 2026
Table 17: Comprehensive error analysis for all cloze blanks in the "Street Vegan" passage.
Modality
Cue (short description)
Correct
Predicted
GT
Pred
vix
Visual
Cookbook title
STREET
VE-
STREET
VE-
A
GAN
GAN
2
Visual
Author credited
Adam Sobel
not given not given given
C
E
3
Visual
Ink colour sign
black
A
E
4
Visual
"Soft Space-age creature
Dog-men
B
5
Auditory
Background music style Ambient location sound Occasional background sound
jazz/lounge restaurant clinks
jazz/lounge not given not given
D
Auditory
DD
7
Auditory
X
8
Auditory Auditory
Narrator voice range Adjective used ("extra
tenor-to-hariton delicious
not given delicious
A
E
9
D
D
"'
10
Auditory Auditory
Repeated filler word High-pitee figh-pitched laugh
like
not given not given
B
E
11
female
B
E
voice type
Visual
Colour of long-sleeved white garment
not given
B
13
Visual
Colour dark chunky filling
purple
brown
C
14
Visual
Type of baking tray
mesh
not given
C
E
X
15
Visual
Roasted ingredient on GARLIC
not given
A
E
placard
16
Visual
Curried ingredient on placard
LENTIL
not given
B
E
17
Audio-Vise Visual
Salad served in box Sweetener named
lake-out
take-out
D
D
Auditory
agave nectar tamari
agave nectar
DR
Auditory
Savoury
seasoning
tamari
D
named
20
Auditory
Nut type criticised
peanuts hook
peanuts season
C
C
21
Auditory
Verb actually spoken ("... up")
B
22
Audio-Visual
Cutting-board colour/ma-
white
not given
C
E
terial
23
Visual
Leafy green placed burger Colour of sauce added af- ter naming
arugula
arugula
A
24
Audio-Visual
red
given
D
E
25
Visual
Appearance pickled onions
translucent
translucent
C
26
Visual
Creamy topping item
macaromi cheese tattooed
and not given
B
X
27
Visual
Forearm characteristic Sound during topping Word cut off the end Descriptor paired with
tattooed sprinkling not given innovative
DRRD
B
28
Audio- Visual
sprinkling hacon innovative
A
HH
29
Auditory
30
Audio-Visual Visual
"extra delicious"
26

---

Published as a conference paper at ICLR 2026
My--
Twelue framcs uniformlly sampled from the video, arranged row major order (left to right, top to hottom).
Omni-Captioner
Visual
Auditory
Audio-Visual
A vertical smartphone video tours a restaurant entirely themed around Dragon Ball Z] (A: One Piece B: Naruto C
Dragon Ball Pokémon| E: not given), while an energetic electronic soundtrack pulses beneath. The music opens
with classic ( four-on-the-flood (A: swing B: waltz| C: four-on-the-llorr D: half-time E: not given) kick pattern
and bright synth X not given] (A: glissandos B: arpeggiations C: tremolos D: strums E: not given) driving a
glossy melody, As the camera glides past a stainless prep area lo granite-putlemed counter, two employees in black shirts and caps work methodieally-one cap worn forward, the other < backward] (A: forward B: inside out CC:
backward| D: sideways EE not given)-with- a X not given] (A: white B: red C: blue D: green EE not given) spray
X not given] (A: red B: white C: bottle nearby, sneeze guards separating stations, and cloud decor hanging from
blue DD: black| E: not given) ceiling in front of'a large monochrome [ X not given] (A: Piccolo B: Vegeta| C: Fricza
D: Goku| E: not given) face mural. A hard cut reveals tall glass cases by window looking out to parking lot, and later a whip-pan catches a male customer turning away: he says ( E X not given] (A: "hey"" | B: nothing | C: quict laugh D: "excuse me" E: not given) as he leaves the frame. The display sweep shows collectible figures and boxes, including
figurine of not not given] (A: Gohan [B: Vegeta| C: Krillin| D: Cell E:: not given) with crossed arms and boxed [ X
given] (A: Hot Wheels| B: Figma| C: Nendoroid D: Funko Pop| E: not given) figures below, plus game box that
reads CLUE) (A: MONOPOLY B: |IFE| C: RISK D: CLUB E: not given) in bold white the top and replica
not given] (A: Four
B: Seven C: Five D: Three|
E: not given)-Star Dragon Ball. A pan lands on a life-sized Ultra
Instinet statue with X not given] (A: black
B: gold
C: blue
D: silver| E: not given) hair buside two sleek kiosks
topped with X not given] (A: blue B: red
C: green
D: purple E: given) indicator lights. As the soundtrack
builds, a rising whoosh [ riser] (A: gong hit B: stinger C: riser D: vinyl scratch E: not given) swells, and the
compression creates rhythmic against the beat. hard drops the viewer into vivid restroom with magenta-on-black walls and an LED mirror; the person filming appears via her reflection. She speaks while visible in the + mirror] (A: mirror B: doorway C:
pumping] (A: pumping| B: crackling C: fluttering D: ringing E: not given)
poster D: window E: not given). her General (+ X not given] (A: Australian B: British RP C: American D: Southern
drawl E: not given) accent clear she says, "Alright, so I'm in the bathroom. Pretty ( X cool] (A: wild| B: strange C:
funny D: cool E: not given)." There is a brief (& V pause] (A: sigh| B: laughter C: cough D: pause| E: not given)
between "bathroom" and "pretty wild," and a faint X not given] (A: ringtone| BB clapping C: footsteps D: rustling
E: not given) the start of her speech. No ( X not given] (A: faucet drip B: hand dryer C: sink DD: toilet B: not given) sounds are heard. During this mirror shot, the music's studio sheen dips and sounds distant and reverberont, matching the small hard-surfaced room. Right after she finishes, the pristine track slams back in via precise edit] (A: gradual fade-in B: hard edit C: slow crossfade D: long echo E: not given), launching build that leads to the drop with powerful bassline and classic & V supersaw] (A: Rhodes B: electric guitar CC supersaw D: acoustic
hard
piano E: not given) lead timbre. Short, high-pitched processed accents "[ & X not given] (A: Yo! B: Heya! C: Go! D: Hey!| E: given)!" cut through the mix, and they oceur while the camera remains the
bathroom] (A:
bathroom| B: kitchen C: storage room | D: dining area E: not given). Near the end of the bathroom segment, she [
adjusts her hair] (A: sets the phone down B: opens the faucet C: adjusts her hair D: claps her hands E: not
given), and brief fabric sound coincides with the motion. Throughout, the track drives in ( 4/4] (A: 3/4 B: 7/8
4/4 | D: 5/4 B: not given) time and finally ends with an C v abrupt stop] (A: gradual fadc-out B: tape slowdown CC abrupt stop D: extended reverb tail E: not given), not a fude. Meanwhile, the woman's look in the mirror includes glasses, visible tattoos, silver cross necklace, and an anime T-shirt, and while delivering the line she holds her smartphone X not given] (A: diagonally B: upside down C: horizontally D: vertically E: not given).
Figure 12: Visualization for all cloze blanks in the
passage.
27

---

Published as a conference paper at ICLR 2026
Table 18: Comprehensive error analysis for all cloze blanks in the Dragon-Bale Ball-Z-theemed-ree rreterrrraatm
passage.
#
Modality
Cue (short description)
Correct answer
Predicted answer
GT
Pred
V1x
1
Visual
Restaurant theme
Dragon Ball Z.
Dragon Ball Z four-on-the-flor not given backward
C
2
Auditory
Kick-drum pattern
four-on-the-floor arpeggiations hackward
Auditory
Synth ornament
Visual
Second employec's 's cap orientation
$
Visual
Colour of spray bottle Ceiling colour with cloud décor
white black
not given not given
A
E
X
6
Visual
D
E
X
7
Visual
Character
Goku
not given
D
E
monochrome face mural Customer utterance while exiting
8
Audio-Vine Visual
(nothing)
given
B
E
Visual
Arms-cross figurine
Vegeta
not given not given CLUE
BB
E
Visual
Type of boxed figures
Funko Pop
DDDD
Visual
Board-game box shown Replica Dragon Ball (numher of stars)
CLUE
AA
12
Visual
Four-Star
not given
X
13
Visual
Ultra-Instinct statue hair
silver
not given
D
E
colour
14
Visual
Kiosk
indicator-light
blue
not given
A
E
colour
15
Auditory
Transition SFX type Side-chain level effect Surface that shows filmer (hathroom)
riser
riser pumping mirror
C
Auditory
pumping mirror
A
Audio Visual
A
18
Auditory
Speaker's accent
American
not given not given pause
C
BB
E
19
Auditory
Adjective in "Pretty... wild
AAR
XX
20
Auditory
Intentional pause after
pause
"bathroom"
21
Auditory
Faint sound speech rustling start
not given
D
E
22
Auditory
Sound explicitly absent toilet ("No ... heard")
given hard edit supersaw not given
D
E
23
Auditory
Music returns
hard edit supersaw "Hey!"
B
Auditory
Lead-synth timbre
ICC
C
25
Auditory
Short processed vocal ac- cent
E
26
Audio-Visuaa
Room shown during vo- cal accents
hathroom
bathroom
A
A
27
Audio-Visual
Action: filmer adjusts.. Time signature of track Track ending style
her hair 4/4
her hair
C
CCC
28
Auditory
4/4
ICCE
CCCO
29
Auditory
abrupt stop horizontally
abrupt stop given
30
Audio- Visual
Orientation of phone in
C
E
hand
28


Reviews:
[{"review_id": "pVbu9UFoiF", "reviewer": "Reviewer_Mhp2", "summary": "This submission introduces Omni-Detective, an agentic data pipeline for generating highly detailed yet low-hallucination multimodal captions via iterative tool-calling and evidence verification. \nUsing this data, the authors train Audio-Captioner (audio-only) followed by Omni-Captioner (audio–visual), and claimed that they achieve state-of-the-art results (but this reviewer got some concerns and this matter should be discussed further). \nTo evaluate fine-grained multimodal perception, they also propose Omni-Cloze, a cloze-style benchmark covering audio, visual, and audio–visual tasks, offering stable and efficient assessment with strong alignment to human preference. \nOverall, the work presents a whole stack, including data pipeline, models, and benchmark, for fine-grained, low-hallucination multimodal understanding.\n\nThis reviewer saw the contribution and potential of this submission to the community. However, this reviewer found that the comparative evaluations are conducted in unfair ways. This issue may nullify most of the claims made by comparing with the competing methods. This should be further discussed during the rebuttal.", "questions": "Please check the weakness for further discussion.\n\nPersonally, this reviewer would like the work, but the following critical concern hinders giving a positive opinion. \nHowever, the unfair comparison is tightly entangled with the overall contribution claims. It's indeed unfortunate. \nThus, carefully addressing this reviewer's concern would be essential to re-evaluate the submission. Otherwise, the rate would not be changed.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 4, "contribution": 4, "strengths": "- Conceptual clarity: Clearly articulates the \"co-growth\" issue between detail and hallucination and provides an actionable framework to mitigate it.\n\n- Clear demonstration of the effectiveness of the Omni-Detective\n\n- Comprehensive evaluation: Omni-Cloze provides stable, low-cost, human-aligned evaluation across modalities.\n\n- Reproducibility: Extensive appendices with prompts, hyperparameters, and validation details enhance transparency.\n\nOverall, the paper structure, motivation, contribution, and analyses are well implemented.", "weaknesses": "- Unfair comparison due to unfair training scheme: The comparison in Table 2 appears not fully fair, as the proposed Omni-Captioner is trained with the additional proposed large-scale, detailed captioning data (≈ 55k audio-only + 15k audio-visual samples) from VGGSound and FineVideo using a two-stage curriculum. In contrast, all baseline results are \"obtained from their original papers\" (as described in the caption of Table 2), meaning those models were applied without any chance to be fine-tuned with comparable detailed captioning datasets, but evaluated in their own (likely general or zero-shot) settings. Although the authors mentioned the proposed Captioners were applied in a zero-shot way, the zero-shot meanings are different (the proposed method is not fine-tuned on any training set related to the benchmark, but it still has an unfair advantage of fine-tuning on additional large-scale detailed captioning data, while the other competing methods do not).\nThus, the proposed model benefits from substantially more task-specific supervision, making the performance gains partly attributable to data scale rather than modeling improvements. The authors should clarify this mismatch or conduct controlled experiments with at least comparable training datasets, so that Table 2 clearly shows the effectiveness of the proposed dataset and training recipe. If there is no comparable dataset for the audio-only case, the authors should provide some baselines at least in an ablation way.\n\n- Due to the above concern, all the claims made with the comparisons against the competing methods are deemed to be nullified. For example, in Table 3, even if the proposed method is additionally fine-tuned, the performance gaps against the other competing methods seem marginal.\n\n- Scalability: While the design of Omni-Detective seems effective according to Fig. 6, it may be bulky and has a trade-off between the effectiveness and computational costs. The multi-step Omni-Detective pipeline may be computationally expensive for large-scale data generation, but no analysis and discussion have been reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pWxMMZitOk", "reviewer": "Reviewer_WAFR", "summary": "This paper introduces Omni-Detective, an agentic data generation pipeline designed to enhance fine-grained multimodal perception in Omni Language Models (OLMs). It produces highly detailed yet low-hallucination audio–visual data and supports training of Audio-Captioner and Omni-Captioner, which achieve performance comparable to closed-source models (ex, Gemini). The authors also propose Omni-Cloze, a new cloze-style benchmark for evaluating detailed multimodal captioning across audio, visual, and audio–visual domains.", "questions": "- In Table 2, why are proprietary models such as Gemini 2.0 and 2.5 not included in the comparison, despite being used as baselines in other sections? Were these results unavailable or intentionally omitted for consistency?\n\n- Since the core models are fine-tuned from existing architectures, what specific aspects of Omni-Detective or Omni-Captioner represent the novel contribution beyond data quality improvements?\n\n- Are there plans to release the generated dataset and benchmark publicly, and if so, will accompanying annotation or tool-calling logs be shared for transparency?\n\n- When conducting these experiments, how sensitive are the results to hyperparameters such as gradient accumulation, number of training epochs, or learning rate? Even with a strong data pipeline like Omni-Detective, it would be valuable to understand how much small tuning variations influence the final outcomes, based on the authors’ experience.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well written and well structured, clearly presenting the motivation, methodology, and results.\n\n- It achieves competitive or superior performance to closed-source models (e.g., Gemini 2.5 Flash and Pro) across multiple benchmarks.\n\n- The experimental setup, dataset description, and evaluation protocol are presented in exceptional detail, ensuring transparency and reproducibility.", "weaknesses": "There are no major weaknesses identified in the paper. The overall work is solid and well executed; only a few minor points of curiosity or clarification remain, which are addressed in the Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q2tLUL12FE", "reviewer": "Reviewer_1S9o", "summary": "This work proposes Omni-Detective, an agent-based data generation pipeline that automatically produces highly detailed multimodal data with minimal hallucination by integrating tool-calling. Specifically, Omni-Detective interacts with diverse tools, including MLLMs, OCR, and ASR, to extract information from videos and summarize detailed perceptions. Using this information, the LLM integrates all perceptions to generate detailed captions, which can later be used for question–answer tasks. Two captioning models, Audio-Captioner and Omni-Captioner, are trained as part of Omni-Detective.\nTo further validate detailed perception capabilities, the authors introduce Omni-Cloze, a cloze-style evaluation dataset for detailed audio, visual, and audio-visual captioning. Experimental results show that Omni-Detective outperforms existing methods on both existing benchmarks and the Omni-Cloze dataset.", "questions": "### Questions and Suggested Experiments\n\n- Although AudioCaps and Clotho are not targeted for detailed captioning, evaluating on these datasets would help assess the model’s audio captioning capability.\n\n- Since the data generation pipeline involves automation, conducting error analysis after human verification would clarify its robustness.\n\n- Ablation on each modality to examine whether the audio module is necessary for the omni benchmark will help to understand that the model fully exploit all the modalities, and the benchmark requires all the modalities.\n\n### Minor Questions and Suggestions\n\n- In L606, VGGSound was used for training the audio-captioner. Does VGGSound provide ground-truth captions for training?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Designing Omni-Detective is a valid and practical approach to fully utilize the perceptual capabilities of existing models for detailed captioning.\n- Introducing the Omni-Cloze evaluation dataset for detailed captioning is valuable to the community. It enables cloze-style evaluation, addressing current benchmark limitations that require multiple LLM calls for assessment.", "weaknesses": "- Several important details are missing. For instance:\n  - Which dataset is used for Figure 2 and how are hallucinations detected in long captions?\n  - Which model serves as the detective agent in Figure 3?\n  - What metrics are used in Table 3?\n  - Which datasets are used for joint training and audio-only training?\n  - How is the detail rate evaluated in L406?\n\n- Although AudioCaps or Clotho are not specifically designed for detailed captioning, they can still be used to evaluate audio captioning capabilities. However, they are not included in the experiments.\n\n- In Figure 4, the Omni-Cloze samples appear to include audio-only, visual-only, and audio-visual QAs. In Table 4, for the audio-only model, was the evaluation conducted only on audio-only QAs within Omni-Cloze?\n\n- Error analysis of the data generation pipeline is missing. Even with human validation, such analysis is essential to assess the robustness and reliability of the generated dataset.\n\n- For training Omni-Captioner (and Audio-Captioner), is full finetuning reallly effective? Why not LoRA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZaKb8V33cC", "reviewer": "Reviewer_AErd", "summary": "The paper introduces Omni-Detective, an agentic data generation pipeline that uses iterative Query-Observation cycles to produce high-quality, low-hallucination multimodal captions. Based on this data, the authors train Audio-Captioner and Omni-Captioner models for audio and audio-visual detailed perception, respectively, and propose Omni-Cloze, a novel cloze-style benchmark for evaluation. Experimental results show that Omni-Captioner outperforms existing open-source models on benchmarks like VDC and video-SALMONN 2, achieving a strong balance between detail and hallucination.", "questions": "# Questions:\n1. Can Omni-Detective maintain low hallucination rates during multi-round iterations in highly complex scenarios?\n2. Are there plans to evaluate Omni-Captioner’s performance on devices with limited computational resources?\n3. How can specific metrics be designed to quantify the quality of information extracted by the detective agent in each round?\n4. Is there a mechanism to correct errors if the detective agent misidentifies information and mitigate subsequent impacts?\n5. Can the Omni-Cloze benchmark be extended to evaluate longer videos or more complex multimodal interactions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "# Strengths:\n(+) Innovative data generation：Omni-Detective’s iterative approach with tool-calling effectively reduces hallucination, providing a novel solution for generating high-quality multimodal data.\n\n(+) Superior model performance：Omni-Captioner sets new state-of-the-art results on VDC and achieves the best detail-hallucination trade-off on video-SALMONN 2, surpassing most open-source models.\n\n(+) Robust evaluation framework：Omni-Cloze offers a comprehensive, efficient, and reliable cloze-style evaluation across audio, visual, and audio-visual modalities.", "weaknesses": "# Weakness:\n(-) The reliance on a single detective module, while effective, lacks the depth of insight offered by multi-agent systems, potentially limiting its innovation in the generative AI field.\n\n(-) The paper lacks experimental data on the computational cost per iteration, leaving the method’s feasibility in resource-constrained environments unclear.\n\n(-) The paper does not specify how to assess the detective agent’s performance, raising concerns that poor agent output could result in longer, meaningless text rather than valuable information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)