MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
Authors: Sara Papi
Maike Züfle
Marco Gaido
Beatrice Savoldi
Danni Liu
Ioannis Douros
Luisa Bentivogli
Jan Niehues
Venue: ICLR 2026 Poster (2026)
Keywords: benchmark
crosslingual
multimodal
instruction-following
speech
video
Primary area: datasets and benchmarks

Abstract: Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework. As these models are rapidly evolving toward general-purpose instruction following across diverse and complex tasks, a key frontier is evaluating their crosslingual and multimodal capabilities over both short- and long-form inputs.
However, existing benchmarks fall short in evaluating these dimensions jointly: they are often limited to English, mostly focus on a single modality at a time, rely on short-form inputs, or lack human annotations--hindering comprehensive assessment of model performance across languages, modalities, and task complexity. To address these gaps, we introduce MCIF (Multimodal Crosslingual Instruction Following), the first crosslingual human-annotated benchmark based on scientific talks on NLP and beyond. MCIF evaluates instruction following in crosslingual, multimodal settings over different input lengths and spans four macro-tasks: recognition, translation, question answering, and summarization. It covers three core modalities (speech, vision, and text) and four diverse languages (English, German, Italian, and Chinese), fully aligned across all dimensions. This parallel design enables a systematic evaluation of MLLMs' abilities to interpret instructions across languages and effectively integrate multimodal contextual information. Our benchmarking and analysis of 23 models highlight universal challenges across modalities and tasks, indicating substantial room for improvement in future MLLMs development. MCIF is released under CC-BY 4.0 license to promote open research.

Published as a conference paper at ICLR 2026
MCIF MULTIMODAL CROSSLINGUAL INSTRUCTION- FOLLOWING BENCHMARK FROM SCIENTIFIC TALKS
Sara Papi 9 Maike Züffe Marco Gaido
Beatrice Savoldi , Danni Liu
Toannis Douros Luisa Bentivogli Jan Niehues
Fondazione Bruno Kessler (Italy)
Karlsruhe Institute of Technology (Germany)
Translated (Italy)
ispapi,maaine bsavoldi, beativoveee... eu. [maike. zuefle, danni conni.11se 1iu, jan. nie.ueesssssss.ccd
ABSTRACT
Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework. As these models are rapidly evolving toward general-purposs instruction follow- ing across diverse and complex tasks, a key frontier evaluating their crosslin- gual and multimodal capabilities over both short- and long-form inputs. However, cxisting benchmarks fall short in evaluating these dimensions jointly: they are often limited English, mostly focus short-form inputs, or lack human annotations- comprehensive assess- ment of model performance across languages, modalities, and task complexity. To address these gaps, we introduce MCIF (Multimodal Crosslingual Instruction Following), the first crosslingual human-annotated benchmark based scien- tific talks on NLP and beyond. MCIF evaluates instruction following in crosslin- gual, multimodal settings over different input lengths and spans four macro-tase tasks: recognition, translation, question answering, and summarization. It covers three core modalities (speech, vision, and text) and four diverse languages (English, German, Italian, and Chinese), fully aligned across all dimensions. This parallel design enables a systematic evaluation of MILMss abilities to interpret instruc- tions across languages and effectively integrate multimodal contextual informa- tion. Our benchmarking and analysis of 23 models highlight universal challenges across modalities and tasks, indicating substantial room for improvement in fu- ture MLLMs development. MCIF is released on HuggingFace under CC-BY 4.0 license to promote open research.
single modality at time, rely on
INTRODUCTION
In recent years, large language models (LLMs) have achieved remarkable progress across wide range of tasks (Brown et al. 2020 Grattafiori et al. 2024), leading to a growing interest in extend- ing their capabilities beyond text to embrace multiple modalities such as speech (Rubenstein al.
|023] CChu et al. 2023 Gaido et al. 2024) and vision (Alayrac et al. 2022; Achiam et al.) [2023] Huang et a 2023). Early efforts typically extended L.MMs with a single additional modality and for task-specific applications (Li et al. 2023b Tang et al. 2023). Building on these foundations, multi- modal LLMs (MLLMs) have emerged to unity language, audio, and visual understanding within a single framework (Liang et al. 2024) and are now rapidly evolving toward more flexible and gener- alized usage, where they are expected to follow natural language instructions and perform diverse, complex tasks (Hendrycks al. 2020). This paradigm, widely known instruction following (IF), requires models to interpret a user instruction within the provided context and generate an appropri- ate response across more input modalities (Su et al. 2023 Caffagni al. 2024

---

Published as a conference paper at ICLR 2026
Multilingual and Crosslingual English German Italian and Chinese
A Up to 10 Hours of Numan-Annotated Content (Transcripts, Translations, Summaries) from the Scientific
220 Human- Creetted Q&A Pairs based on @Moodality Source (generic, transcript, summary). and X Unanswerable
23 State-of-the-Ar LLMs, SpeechLLMs, 5 VideoLLMs, and MLLMs
LANGUAGES
HUMAN LABELS
+200 QUESTIONS
23 B BASELINES
3 MODALITIES
CONTEXT TYPES
MULTIFA CEETED EVALUATION
13 TASKS
FIX AND MIX PROMPTS
Speech
Short-form and Lorg Text, Speech
and Video Zo Modalities
13 Diverse Tasks divided into macro-tasks: transcription, translation, question answering. and summarization
Sets of Prompts: Fix (consistent across samples) and Mix (natural, diverse formulations)
Parallle across modalities, languages, and context
contents
parallel and equally important challenge lies in extending these capabilities to multilingual and crosslingual settings (Hu et al.) 2020; Xu et al., (025b). General-purposs MLLMs must not only handle inputs and outputs in the same language by supporting the highest possible number of diverse languages, but also process crosslingual multimodal inputs, such as speech in language paired with an instruction in another language (Zeng et al. (2025). Despite rapid advances in both instruction-
following and multilingual modeling (Wei et al.) 2021 [anh al. 2021: Barrault al., 2023),
current benchmarks fall short of comprehensively analy these aspects. Recent work focuses exclusively on two modalities, such as vision-text (Fu al. 2023 Zhang al. 2023a Qian al.) 2024 Das et al. (024) or speech-text Yan et al.) 2025; Pandey et al. (025)), or restricts its scope to English (Li et al. 2023a Chen et al.] 2024a), thereby overlooking the complexities
multilingual
Gao and crosslingual interactions (Duan et al. 2021 Wang al. 2022, Gao et al. 2023 Pernes et al., 2024). Adding to these limitations, current multimodal benchmarks predominantly focus on short- form inputs, neglecting the evaluation of models' capabilities with long dependencies (Fu et al.) 2024), and only a few are numan-generater (Li et al. 2023a). raising concerns about data quality, potential biases, and the overall reliability of model evaluations (Zhang et al.) 2022a)..
To fill this gap, we introduce MCIF the first manually- curaated benchmark explicitly designed to evaluate crosslingual, multimodal IF abilities over both short- and long-form inputs. MCIF covers three core modalities-speech, video, and text-and spans four typologically diverse languages: En- glish, German, Italian, and Chinese. Supporting 4 macro-tasks (recognition, translation, question answering, and summarization), MCIF is fully parallel across modalities and languages, enabling systematic evaluation and ablation studies of MLLMs' abilities follow instructions across these different dimensions. Our extensive benchmarking and analysis of 23 different systems highlight that, despite recent progress, current models still face significant challenges: they struggle to handle long-form contexts-especiall. when tasked to summarize their content, to jointly integrate speech and video effectively, and to answer fine-grained, content-specific questions. These findings high- light main directions for improving crosslingual and multimodal processing in IF systems.
2 RELATED WORKS
In this section, we survey existing IF benchmarks for speech and vision, highlighting critical gaps crosslingual, multimodal, and long-form evaluation that MCIF is designed address.
Speech-Text Text IF Benchmarks. Speech-ifeval (Lu et al. 2025), SAKURA YYang et al. 2025b), and MMSU (Wang et al. 2025b)
Most existing efforts in IF speech-text evaluation datasets, such as
'The benchmark is released under CC-BY 4.0 license at wt. /tt.aattssseeeeeee /F /MCIF F to facil- itate research and broad adoption. The inference and evaluation code are available under Apache 2.0 license t-mt/meif /mci which also contains the systems' outputs under CC-BY 4.0 license.
O thub. com/h t-
2

---

Published as a conference paper at ICLR 2026
task name
acronym in mod out mod sre lang
fgt lang context type cross
Textual Question Answering
TQA
LONG
Text Summarization
TSUM
LONG
Machine Translation
MT
LONG
Automatic Speech Recognition
ASR
X
Spoken Question Answering
SQA
Speech Summarization
SSUM
LONG
Speech Translation
SHOSE
LONG
Vidco Question Answering
VQA
SHORT
1OME
Video Summarization
VSUM
LONG
Audio-Video Recognition
AVR
X
Audio-Videe Question Answering
AVQA
SHORT
LOME
Audio Video Summarization
AVSUM
LONG
Audio Video Translation
AVT
SECRT
LONE
Tasks in MCIF with their input/output modalities (in mod, out mod), input type (context type) that can be long-form Lows or short-form aacaт ), source/target languages (sre lang, tgt lang) among English = German Italian and Chinese Since all IF tasks involve text prompts, report f when task uses only the text modality. cross indicates whether the task can be crosslingual, i.e., if it involves a target language different from the source language. The detailed description of each task is provided in Appendix
Table I:
monolingual tasks, predominantly covering the English language. AIR-Bench (Yang et al. 2024). VoiceBench (Chen et al., 2024a), ADU-Bench (Gao al. 2024. URO (Yan al. 2025) and SpeechInstructBencch (Wang al. (025c) more dialogue-oriented tasks that are limited to English and Chinese, with the latter three benchmarks relying entirely on synthetic speech. SD-Eval (Ao al.) [2024), Dynamic-SUPERB (Huang al., AudioBench (Wang et al., 2025a). MSTEB (Beyene et al., 2025). and SIFT-50M (Pandey al. 2025) offer multilingual speech-text evaluation but rely on preexisting benchmarks, such as Common Voice (Ardila et al.] 2020), and FLEURS (Conneau et al. 2023). making them prone to data contamination (Sainz et al. [2023] Balloccu et al., 2024) Kocyigit et al. 2025) and limited to short-form, speech-only assessment. Overall, while interest in speech-text evaluation is growing, existing benchmarks do not support multimodal, crosslingual, and long-form instruction follouiiinnn in unified setting MCIF.
restrict their scope instruction-follee
2025
Vixion-Text IF Benchmarks. Similar to the speech-text domain, the vision-text domain has seen huge increase in the number of benchmarks designed to assess MILMS across diverse capabilities. MMMU (Yue et al.) 2024) and MIA-Bench (Qian et al.) 2024) evaluate MLLMs with image-textual
inputs across several domains, but cover English only. MME (Fu al. 2023) extends the evaluation to Chinese-to-English translation, and M3Exam (Zhang et al. 2023a) to 9 diverse languages, while languages. Despite their extensive language coverage, these vision-text benchmarks are all constrained to benchmark models' abilities when dealing with single images rather than videos- sequences of images. Video-based benchmarks such as Video-Bench (Ning et al. 2023). InfiniBench (Ataallah et al. 2024). VITATECS (Li et al.) 2024b), TempCompass (Liu et al.] 2024), LVBench (Wang et al. 2024b). MVBench (Li et al., 2024a), and MMBench- Video (Fang al.) (0224) focus on bimodal interactions (video and text), cover English only, and rarely incorporate human-authored multilingual instructions.
EXAMS-V (Das et al. 2024) further widens the coverage
VideoMME (Fu et al. 2024) and MF² (Zaranis et al. 2025) are the first benchmarks comprising the three modalities (speech, text, and video); however, VideoMME is not crosslingual and restricts its scope solely video-centric centricntic tasks, while MF² includes speech but does not evaluate this modality. As a result, no benchmark currently enables systematic evaluation across speech, video, and text modalities in crosslingual instruuutttttttttttti instruction-followinn framework.
3

---

Published as a conference paper at ICLR 2026
3 MULTIMODAL CROSSLINGUAL INSTRUCTION-FOLLOWWE BENCHMARK
We create MCIF from English videos of scientific presentations (from NLP topics and beyond), including their related audio, by manually creating transcripts and translations of their content, sum- maries (abstracts), and set of questions and open-ended answers. It results in a highly multi- task, natural, human-labeled, and expert-vetted benchmark characterized by: i) modalities: text, speech, and video; ii) 4 languages: English, German, Italian, and Chinese; iii) 2 context types: short-form and long-form text, speech, and video contents; and iv) 13 tasks: crosslingual and mul- timodal tasks, which divided into 4 macro-tasks (recognition, translation, question answering. and summarization), and reported in Table []] Each sample is composed of the input content (ei- ther short- long-form text, speech, or video), which is paired with a textual prompt containing the instruction to be followed (in English, German, Italian, or Chinese), and its corresponding tex- tual reference (transcription, translation, summary, or answer in the same language as the prompt), MCIF is designed to be parallel across languages and modalities, as each sample contains the input in all three modalities, and prompts and outputs in all four languages. We describe the data selection in Section 3.T, the human-annotation process in Section 3.2 and the cnstuctioonfollowiin prompt composition in Section 3.3
3.1 DATA SELECTION AND COLLECTION
We collected scientific talks from the ACL Anthology the main reference repository for research in the language technologies community. This source is well-suited our objective since i) it is openly available under a CC-BY 4.0 License, allowing unrestricted use and redistribution; ii) offers naturally multimodal and challenging material, i.e., video presentations self-recorded by speakers from various linguistic backgrounds and accents, accompanied by slides, spoken audio, and corresponding research papers. To avoid data contamination issues of testing models on material that has been used for training (Balloccu et al.) (2224) Kocyigit et al.) (2025). we selected the most recent available material at the time of collection namely, the ACI. 2023 paper presentations
We randomly picked videos from the ACI. 2023 main conference papers, covering different topics in the context of NLP and beyond, spanning from multimodality to explainability and ethics. The resulting selection is both diverse in terms of recording conditions (each presenter recorded their talk independently, using their own equipment and environment, with significant variation in au- dio and video quality, background, and presentation style), and speaker demographic, as ACL is an international conference that features researchers from around the world (representing a wide range of nationalities from Europe, the Americas, Asia, and beyond) [ The collection was man- ually inspected and validated to discard presentations with i) repeated speakers (i.e., each sample represents a unique speaker), ii) inaudible or low-quality speech (e.g., presentations with excessive background noise or featuring a speaker distant from the microphone). and iii) automatically gener- ated speech (e.g., text-to-speech synthesis is used to produce the audio). The resulting benchmark includes 21 presentations, with a total of 2 hours of video content and approximately 15.5k words. The videos are released in their original mp4 format, and they are converted into mono-channel, 16 kHz wav audios. To support both the exploration of how models handle long versus short context and to maximize usability for models with limited context capacity, we provide both the full-length video/speech context and an automatically segmented version generated with SHAS (Tsiamas et al.) 2022), with segments of 16 seconds. Together with the vidco, we collect the abstracts, which serve as summaries for the presentations. To improve test set representativeness for summarization, further collect 79 additional videos, yielding total of 100 samples-abel about 10 hours of content with summaries totaling 17k words. For these additional samples, audio and textual transcripts also available, ensuring alignment across all three modalities. A detailed breakdown of MCIF statistics provided in Fig.
3.2 DATASET MANUAL ANNOTATIONS
We describe the MCIF curation process, including the human annotation. Key steps are summarized below; details on costs, annotation, and design guidelines are in Appendix B
As of April h8 2025, the most recent conference with available videos is chosen for the henchmark. https:://act / anthol ogy. wwtg/orgg/33333 a long. report.po"
4

---

Published as a conference paper at ICLR 2026
samples
tot. length
avg. length
Video Drlyy 22.3%
Questions & Answers
th 58m
Transcripts
talks
16,341
778
9h 36m
49
Total Duration
Translations
tallss
16,025
34,225
17,066
763 1 612 B
1650
96
7m O5s
Question and
220une pairs
2,051
BL 2:104
10 10
2.7%
17
129
4m
Answers
2,311
84,915
121
62
Audro-Video 58.6%
25
Average Duration
5m 40s
Summaries
100 acstracts acts
To 16.951
18,350 8 831888
Wth 16,469
170 165
194 B319
Not Antweraale
General
Transoript
11.%
Figure I: Breakdown of MCIF statistics. Total length is measured in space-separated words for English, German, and Italian, and in characters for Chinese. Question-answed statistics in the inner circle refer to the question type, while the outer circle refers to the input modality (see Section 3.2
Recognition and Summarization. For each talk, we tasked professional linguists with producing high-quality gold transcripts in US English, following detailed guidelines (see Appendix B). This enabled the creation of aligned English video, speech, and text data. For summarization, instead, used the abstract from the associated paper as the textual summary, as prior work shows abstracts provide reasonably accurate representations of the scientific talks (Zifle et al.) 2025).
Question-Answering, To evaluate model understanding, we design a QA task intended to probe different aspects of contextual dependency and task realism. First, each talk was paired with least 10 QA pairs, which followed a structured distribution: i) General common questions, which are generic and applicable to any talk (e.g.. "What are the affiliations of the authors?"); ti) Transcript questions, created after watching the full talk and targeting narrow, context-dependent information retrieval; and ii) Abstract questions, generated after reading only the abstract, simulating a scenario where user queries about talk without having watched it in its full length. All QA pairs were created and verified by 16 expert annotators with high English proficiency and a background in machine learning and NLP. Each QA pair was also annotated with the input modality required to answer the question, explicitly including cases where no answer is available. Labels were assigned as follows: NA if the information was not present in either the video or audio, AV if the information was explicitly available in both the audio and video modalities, with either modality alone being sufficient to answer the question, A if the answer was explicit in the audio only, and V if it was explicit in the video only. A breakdown of the QA distribution among categories is illustrated in Fig. Overall, this setup enables a systematic evaluation of model performance across modality conditions and unanswerable cases, as detailed in Appendix B All QA pairs are created in English and, as described the following, are then translated into three additional languages.
Translation and Crosslinguality. To make MCIF crosslingual, all English textual data transcripts, summaries, and QA pairs-were translated into three additional languages: Italian, German, and Chi- nese (Mandarin). These languages were selected as they are well-resourced, allow for comparable analyses, and represent a diversity of language (sub) sambiiiilssss and writing systems. All translations were carried out by professional translators with expertise in scientific content. As translators viewed the original QA pairs, summaries, and transcripts during this process, translation also served as secondary verification step, further ensuring the quality and consistency of the source material.
3.3
COORUCCTION-FOLLOWIII PROMPTS
For each instance in the benchmark, information such as the specific task (e.g., ASR), the input modality (e.g., audio, video, or text), or the target language (e.g.. German) not provided as explicit metadata; rather, the model must infer these aspects from the prompt itself, simulating real human interaction, and fulfill diverse instructions across the supported language pairs (e-g.. "Rispondi in modo conciso alla seguente domanda dato il contenuto inglese: [QUESTION]""it) "Answer the following question concisely given the English content: [QUESTION]][cn)] Following previous work, always specify the source language in the prompt, which is written in the target language (Xu et al. 2024| Lu et al. 2024a).
We create two variants of the MCIF benchmark, MCIF and MCIF mix: based on the set of prompts. MCIF employs a fixed prompt for each macro-task (recognition, translation, question answering, and summarization). Instead, MCIF mix selects prompt at random from pool of ten alternatives
5

---

Published as a conference paper at ICLR 2026
for each macro-task, where the pool includes the fixed prompt from MCIF By contrasting the two settings-alee always using the same prompt versus sampling from diverse ones-we can directly measure the generalization and robustness of models to different prompt wordings. All prompts were manually crafted for each of the four languages and are reported in Appendix
EXPERIMENTAL SETTINGS
Models. We evaluate a range of models across modalities: LLMs on textual tasks, SLsechLLMs on speech tasks, VideoLLMs on video-only tasks (without speech), and MLLMs on all tasks (text, speech, video, and speech+video). To ensure compatibility within a unified evaluation framework, weinh models hosted on HuggingFace that can be
select publicly available state-of-the-an run using the HuggingFace Transformers library. Due to computational constraints, we restrict our selection to models with fewer than 20 billion parameters. Additionally, we evaluate commercial MLLM, Gemini 2.5 Flash (Comanici et al. (025), whose outputs are obtained through API calls) This results in 23 models: LLMs, SLsschLLMs, VideoLLMs, and 6 MLLMS. The full models list, and generation settings are detailed in Appendix D
The evaluation is carried out by computing separate scores for each of the tasks addressed, using commonly adopted metrics in the community. Namely, for recognition tasks (ASR, AVR), we compute WER using the jiWER library after normalizing the test using the Whisper normalizer (Radford et al. 2023), version 0 0 O. For translation tasks (MT, ST, AVT), we use COMET (Rei et al. 2022). with the standard model Unbabel/ / wmt 22 da, after concatenating all (speech or video) segments belonging the same talk in the case of the short context and reseg- menting the text with mwerSegmenter (Matusov et al. 2005) to pair them with the reference sentences. Lastly, for question answering (TQA, SQA, VQA. AVQA) and summarization (TSUM, SSUM, VSUM, AVSUM), we compute BERTScore (Zhagg et al., (2200) rescaled with the baseline to make scores more interpretable, with 0 corresponding to random outputs in the target language.
Metrics.
5 RESULTS
This section reports results on MCIF from several perspectives: Section 5.1 analyzes overall perfor- mance across all 23 models and macro-tasks, Section 5.2 focuses on MLLMs to study how different modalities impact task performance, and Section 5.3 studies how the best model in each category (LLM, SpeechL.LM, VideoLLM, MLLM) performs on question answering across question types.
5.1 OVERALL RESULTS
Table reports the model results on MCIF and MCIF mix for each context (long or short), and macro- macro-task, Extended results per language are reported in Appendix E
RECOGNITION Some SpeechLLMs (Phi4-Multimodal, GraniteSpeech) and MLLMs (Ola, Gemini 2.5 Flash) show strong performance (WER< < 10), demonstrating the feasibility of this macro- task. However, despite being the simplest one-an and on which all models are trained on-several sys- tems fail in one or both context types. UltraVox v0.5#, Ming-Lite- Lin-Liit-Omiiizzz and MiniCPM-o-22 achieve scores superior to 100 WER on both short- and long-form, DeSTA2 exceeds 100 WER on long-form, and, surprisingly, Ola drops sharply from 6.6/14.0 (long- form) to 98.8/104.1 (short- form). A manual inspection of the model's outputs revealed that Ola often misinterprets transcription prompts in short-form, opting instead to perform image captioning of accompanying slides, while this is not the case for long-form inputs (see Example in Appendix F). The long-form results of Ola also suggest that its architecture partuuulall the strategy of chunking and concatenating long speech segments using a Whisper-based encoder-enables the model to obtain high transcription quality over extended inputs.
TRANSLATION As expected, LLMs dominate due to the maturity of text-based translation, a trend consistent across target languages (see Appendix E}. Beyond LLMs, some SpeechLLMs and MLLMs achieve competitive results in short-form, with COMET >5 for DeSTA2 Ola and awen2.5-Omnit.ee and even >80 for Phit-muutmmodale. However, several models fail to perform this task, either across all conditions (UltraVox v0.5#. MiniCPM-o-200z)
long-

---

Published as a conference paper at ICLR 2026
Context
MCIF
MCIF
Input Modality
Model
REC
TRANS COMETT
QA BERTS.
SUM BERTS
REC
TRANS COMET
QA BERTS.
SUM
WER
WER.
BERTS.
DeSTA2
54.0
75.3
17.2
83.0
75.2
18.6
GraniicS petch
9.4
52.1
0.5
9.5
46.6
0.4
114 114-MMu Multimoda
6.8
80.2
37.1
x
6.7
80.1
37.4
Jwen2-Audma Aucho
31.7
74.9
32.6
31.9
74.6
32.8
Vliravox v0.5
127.7
43.3
19.6
172.6
43.2
19.1
InternVL3
31.7
31.3
SHORT
Lex
13.7
12.1
JwCn2..5VV
X
39.1
x
37.8
24.1
23.8
Video-XL22
13.6
13.6
Gemma
73.0
26.2
58.9
71.5
MineLite- Omii
117.5
53.0
15.8
128.2
53.3
13.3
MiniPPM-o-2
144.8
39.7
21.4
207.1
38.8
23.1
Ol
104.1
76.6
37.3
X
98.8 8
76.3
37.0
QwenZ. 3 5-mn
43.5
77.3
34.3
48.0
76.5
35.1
Gemint 2.5 Flash
14,9
67.0
40.6
12.8
69.2
39.5
Expanse
26.7
25.1
68.7
85.5
22.9
9.5
83.4
21.8
GF 1-oss
75.0
24.6
17.8
70.1
20.9
15.9
lama
81.4
30.3
24.8
79.5
31.0
24.5
Ph14
84.5
30.8
13.0
84.7
29.6
14.5
Jwen3
84.8
37.9
19.9
84.5
35.6
20.1
Towert
85.6
29.5
19.8
83.7
23.4
17.6
DeSTA2
112.9
41.3
12.5
3.0
132.5
40.8
12.6
2.8
LONG
GraniieSpeck necch
99.9
36.0
-23.7
2.8
34.6
-22.8
10.55
7.4
29.8
59.5
37.3
Owen2- Audio
92.9
41.0
28.8
7.3
93.1
41.1
28.9
Ulira v0.3
89.1
38.0
12.7
-3.0
92.5
38.0
12.5
-3.1
LnterVL33
27.6
20.4
27.9
20.4
LaVA-NcXT
7.2
-7.0
5.2
-6.7
Jwenz.3-V7 VL
X
33.7
23.3
34.9
20.2
Vident .1 .MMA3
26.8
-19.9
26.5
-33.0
Video-XL.
17.2
3.7
17.4
4.1
MiniCPM-o-2
170.6
24.9
-38.0
-39.1
179.2
25.9
-38.4
-39.7
Ul
14.0
63.2
36.2
12.3
6.6
58.7
36.2
13.8
QwenZ.5- Siven2.5-Omin Omni
98.5
47.5
32.5
8.9
94.9
40.2
34.8
Gemun
11.9
46.1
24.1
79.9
45.9
21.8
Table Overall results, averaged across languages, on MCIF and MCIF mix. divided into the four macro-tasks: recognition (REC), translation (TRANS), question answering (QA), and summariza- tion (SUM). Best result by context is marked in bold, and best overall result is underlined. marks unfeasible tasks, i.e., summarization with short-form, out of models' scope for given modality: recognition for LLMs and VideoLLMs (requires speech), and translation for VideoLLMs (requires speech text). Gemma 3n and Ling-Lite-Omin removed from process long-form inputs.
they not able to
form (DeSTA2 GraniteSpeech Qwen2-Audioe, Qwwn2.55Omniees where scores drop be- low 50 COMET. The degradation often due under-translation, with models skipping parts of the context-espenn context-especially in long-form (see Example 2 in Appendix [F). An exception is Gemini 2.5 Flash which performs better on long-form: manual inspection revealed that, on shorter seg- ments, it hallucinates over-elaborates on audio/video content (see Example 3 in Appendix F),
known issue current L.LMs (Briakou al. 2024).
QUESTION ANSWERING Surprisingly, not all LLMs excel in question answering even if pro- vided with human transcripts, as the best performance consistently comes from Gemini 2.5 Flash Results are inconsistent, particularly in short-form, where the top models vary by language (see Ap- pendix SpeechLLMMs (Phi4-Mo Multimodllddd)) VideoLLMs (Qwen2.5-VL) and MLLMs (Gemini 2.5 Flash). In contrast, Gemini 2.5 Flash clearly dominates in long long-form, consistently ranking first across languages with average BERTScores above 45. SpeechLLMs and VideoLLMs, instead, suf- fer significant performance drops, echoing trends observed in other tasks. Only few models fail

---

Published as a conference paper at ICLR 2026
entirely, such LLaVA-NeXT= (BERTScore <10, corresponding to outputs in the wrong lan- guage; see Example 4 in Appendix F and GraniteSpeech# (BERTScore around O, corresponding to random text in the correct language or the transcript of the content; see Example 5 in Appendix F.
SUMMARIZATION This is by far the most challenging macro-task, with some systems even producing negative BERTScoreS-worse than random outputs in the target language. Failures span SpeechLLMs (GraniteSpeech, UltraVo, vO.5), VideoLLMs (LLaVA-NeXT, VideoLLaMAA), and MLLMs (MiniCPM-o-2)) Manual inspection points to two recurring issues: models either default to the wrong language (often English across tasks; see Example 4 in Appendix(F) or ignore the instruc- tion altogether (e.g.. LLaVA-NeXTI transcribes slides instead of summarizing them; see Example 1 in Appendix FF). I.LMs achieve the strongest results, confirming that text-only inputs remain eas- ier to handle, followed by MLLMs, whose performance fluctuates widely (from the negative scores of MiniCPM-o-2 to the good scores of Gemini 2.5 Flash). VideoLLMs exhibit similar instability, although generally at lower BERTScores. SpeechLLMs remain weak, with the sole exception of Thi4-Multimodall
SHORT X LONG Across tasks, models generally perform better on short-form inputs, with long- for SpeechLLMs and VideoLLMs. An exception is Gemini 2.5 F-ath significant improvements all tasks-and Ola in recognition. Despite this, long-form recognition remains a major challenge for most systems, regardless of their input modality. Manual inspection revealed that the main source of degradation is models producing only partial outputs (see Example 2 in Appendix FF. This is particularly common in recognition but also in translation: for instance, DeSTA2# and Qwen2-Audion drop by about 34 COMET, while Qwen2.5--mmi = falls by roughly 30 COMET. In contrast, most MI.LMs improve or maintain performance on long-form question answering, notably Ola and Qwen2.5-O0me -00mii. Additional failure cases that especially pronounced in long-form settings include persistent use of the wrong language (GraniteSpeecht defaulting to English; see Example 4 in Appendix (F). common in all macro-tasks, or refusal to answer the user's requests (UltraVox v0.5#; see Example 6 in Appendix F. Lastly, some MLLMs (Gemma 3n and Ming-Lite-Omni) cannot handle long-form inputs at all, constrained by limited context windows.
form leading to notable
with
MCIF fix X MCIF mix. Comparing the two MCIF variants reveals that most models exhibit limited robustness to prompt reformulation, with sensitivity varying across tasks. Recognition is the most affected: some SpeechLLMs (DeSTA2, and UltraVox v0.5) and MLLM (MiniCPM-o-2) fluctuate up to more than 60 WER on short-form, and nearly all systems vary on long-form, with shifts up to 20 WER (e.g., GraniteSpeech
Translation remains relatively stable for LLMs, even if drops of up 4.9 COMET occur (e.g., GPT--ss), with SLs showing similar patterns. MILMs prove less reliable, particularly in long-form, with Soven2.5-Omniiee losing up to 7 COMET. Ques- tion answering generally stable, but most LLMs show notable variations. with changes up to 6.1 BERTScore (Tower+). Summarization follows an unclear trend: some models show little robustness to prompt reformulation (GraniteSpeecht. Phi4- Multimodale, Lade33.L with variations up to 13.1 BERTScore, while others remain consistent.
To sum up, results reveal consistent trends in current models' performance. Summarization the most difficult task, with no gains from adding speech or video to text-underscoring limitations in multimodal integration. QA shows the opposite pattern, benefiting from speech or video and high- lighting the value of non-textual modalities. LLMs continue to lead in translation, while recognition proves highly sensitive to prompt variability. Long-form proves challenging, with nearly all mod- els suffering significant drops across tasks compared to short-form, especially SpeechLLMs and MLLMs. Together, these findings expose the wide gap between current systems and the goal of robust, multimodal, crosslingual instruction following, pointing to clear avenues for future progress.
5.2 EFFECT OF MODALITIES INTEGRATION
Since MCIF is completely parallel across languages and modalities, it enables an ablation study on how different modalities contribute to MLLM performance. Specifically, we evaluate each model under four input conditions: text only, speech only, video only, and speech+video (as already re- ported in Table 2). To isolate the contribution of each modality, biases from the single fixed prompt in MCIF that could favor some models over others, and run the evaluation on both short and long contexts. The results shown in Fig.
the MCIF set to avoid
8

---

Published as a conference paper at ICLR 2026
SHORT
Video
Speach
Speach-Vision
LONG
Text
Video
Speach
RECOGNITION
TRANSLATION
QUESTION ANSWERING
SUMMARIZATION
200
DC
70
70
60
50
100
100
SC
20
20
10
50
10
10
Geeeent Sn
Ola
Creent Omme
Gercea
Mong Mascrraaeg
Crmant O-mms
Cla
Masicrat
Crvant Brbni
Oia
Gamma
bing Drend
Pren Grent
MHICHN
Guned D-Cimo
Figure 2: MLLM results on MCIF mix by inference modality, averaged across languages.
In recognition, comparing Speech and Speech+ Video in both short- and long-form, observe that the video modality provides no benefit and often degrades performance when combined with speech-except in case (Ola long-form). In translation, speech leads the performance in short-form, with Ola being the only model to benefit from the addition of video (improving by 8.6 COMET), while in long-form the text modality-availaie by speech. For short-form question answering, speech again proves fundamental, while video con- sistently underperforms. The only case where video outperforms speech is Owen2.5-Ommp but the margin is minimal (1.9 BERTScore): notably, it is also the only model where combining speech and video brings clear gain over speech alone. In long-form question answering, video alone yields negative scores (MiniCPM-o-2), confirming its limited exploitation in current MLLMs, but joint speech-video processing shows benefits two out of three models (MiniCPM-o-2 and Qwen2.55 Omni). Summarization trends are less consistent across systems: in two out of three models, text enables the best or comparable results, while video-only produces negative scores (MiniCPM-o-2 and Ola), and joint preech-video processing even harms performance in one case (MiniCPM-o-2), Overall, these findings indicate that current MILLMs struggle to effectively integrate speech and video, with joint multimodal processing often providing benefit even being counterproductive. Moreover, the video modality- showing the good results in short-form question answering (Table 2 remains underutilized in MLLMs, systematically yielding the weakest results.
only in this setring-doming
followed
5.3 BREAKDOWN ON QUESTION ANSWERING
To better understand model behavior beyond overall QA scores, we analyze performance across dif- ferent question types (Section 3.2)
Breaking results down by question modality (Audio- Visual,, Audio, and Video) and by source (General, Abstract, Transcript) helps reveal how well mod- els exploit specific input signals and how they handle varying levels of specificity and difficulty. For this analysis. we use long-form contexts,
Audio-Visual
General
which enable evaluation SpeechLLM VideoLLM MLLM
of all models, including LLMs, and report results on MCIF mis to avoid bi- ases from the single fixed prompt in MCIFix, which could favor some models over others. Scores are averaged across languages, and the best model from each family-Qwen3, Phi4- Multimodal, Qwen2.5-VL. and Ola-is shown in Fig. 3 We find that audio-only (A) questions are best handled by the SpeechLLM, while the VideoLLM performs strongest video-only (V) questions. Surprisingly, MLLMs underperform even if accessing both modalities. The LLM. despite working only on transcripts, achieves the highest score on audiovisual (AV) questions (44), slightly surpassing both the SpeechLLMM and MILM
Audio
Video Abstract
Transcript
Figure 3: Performance breakdown on MCIF mix
LONG
QA of the best
models by question modality and source.

---

Published as a conference paper at ICLR 2026
(42.6), confirming that text remains easier to process than multimodal inputs. As expected, modality mismatches cause substantial drops: SpeechLLMs underperform on V questions and VideoLLMs on A questions (losses of 7-13 points), though both remain above 35, likely thanks to contextual cues. These results reveal that, despite being explicitly designed for multimodality. MILMS still fail to effectively integrate speech and visual signals together, leaving substantial room for improvement. Breaking results down by question source reveals a consistent trend: generic questions (General) yield the highest scores (47.6-49.0), while talk-specific ones prove more challenging as performance drops to 33.7-36.6 on Transcript questions and further to 23.2-27.0 on Abstracts. This suggests that current models excel retrieving common information (e.g., paper title or affiliations) but remain weak at retrieving fine-grained content, regardless of modality.
6 CONCLUSIONS
In this work, we introduced MCIF, the first human-annotated multimodal and crosslingual instruction-following benchmark from the scientific domain. MCIF spans three modalities (text, speech, and video) and four typologically diverse languages (English, German, Italian, and Chi- nese), parallel across all dimensions. It incorporates both short- and long-form contexts and covers 13 tasks, organized into four macro-tasks: recognition, translation, question answering, and sum- marization. MCIF comprises two variants-MCIFi, with fixed prompts and MCIF with diverse ones-whose comparison assesses models' robustness and generalization to controlled instruction reformulation under preserved semantic equivalence.
Through extensive benchmarking of 23 state-of-the-art LLMs, SpeechLLMs, VideoLLMs, and MLLMs, we identified both their strengths and significant limitations, particularly regarding joint speech and video modality integration, long-form processing, and summarization. These findings point to key directions for future system design: improving multimodal fusion to process multi- ple modalities jointly and enhancing robustness to long-form content via sequence compression or extended context modeling. In addition, our results MCIF MCIF mix further show that robustness to prompt variation remains an open challenge. While our evaluation focuses on single- turn, simple instructions-reflecting common practice and training paradigms of current multimodal mrodels-futur work can leverage MCIF as a foundation to explore more complex prompting strate- gies, including prompt engineering (Liu al.) 2023 Chen et al.) 2025). prompt embedding align- ment (Kim & Angelova 2025). contrastive training between equivalent instructions (Yan et al.) 2024], instruction tuning based on carefully selected data (Qin et 2025), preference optimization (Ouyang et al. 2022), and chain-of-thougee reasoning (Wei et al. 2022/ZZangg et al.) (023b)).
Overall, MCIF provides a comprehensive evaluation framework and establishes foundation for advancing general-purpose. multimodal, and crosslingual instruction-followin systems.
ACKNOWLEDGMENTS
The work presented in this paper is funded by the European Union's Horizon research and innova- tion programme under grant agreement No 101135798, project Meetween (My Personal Al Medi- ator for Virtual MEETings BetWEEN People), and the PNRR project FAIR Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU.
ETHIC STATEMENT
Licensing and Attribution. The MCIF benchmark is derived from publicly available data released under a CC-BY 4.0 license, and we release under the same terms to support transparent and open science. As described in Section 3.2] all manual annotations were performed by the authors, colleagues, or compensated professionals, with informed consent; details of compensation are given in Appendix B Contributors will be acknowledged in the non-anonymous version of this work.
Use of Flags for Languages. In the paper, we use flags to denote the languages represented in MCIF. We acknowledge that this practice raises ethical and sociolinguistic concerns, since flags symbolize countries or national entities rather than languages, and may be misleading in multilingual contexts. In dataset, however, each flag corresponds to specific language variant used in MCIF
10

---

Published as a conference paper at ICLR 2026
transcripts and translations (e.g., US English for en, German for de, Italian for it, Mandarin for zh), which makes this representation informative for our use case.
Use of Large Language Models. For the writing process, ChatGPT employed exclusively to correct grammar in content authored by humans.
REPRODUCIBILITY STATEMENT
We provide a detailed description of the collection process for our newly introduced dataset MCIF in Section 3.1 and report comprehensive information on the manual annotations in Section 3.2
The annotation process is detailed in Appendix B including information on annotator recruitment through a professional agency, compensation, the number of annotators involved, and the tools em- ployed during the annotation. For transparency, we release the complete annotation guidelines: the instructions for the transcription and translation tasks, and the guidelines for the question answer- ing task are available at https:://tt / thub. com/h / E-mt. /cc i f /ttttre ree/ ma n /dd/aaaa set bui Id/annotat wewiiiinngnuilleeeeee ines also referenced in Appendix B For the MCIF baselines, we provide the full list of models used in Appendix [ including model references, links to the Hug- gingFace model weights, generation settings, and corresponding transformer versions. The prompts are part of our dataset and are listed in Appendix [ We release all code for reproducing the base- lines, along with the evaluation scripts, at h/tttts:///giatiae we release the outputs generated by each model with the same license as the benchmark, namely CC- BY 4.0, http://ggg..aoooaooo////ataa////
com h -t mcif In addition,
nes htt.aa/eenneeeenooouoouooo
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08900 2024a.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee. Yuanzhi Li, Weishung Liu, Caio C. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Zhang. Phi-4 technical report, 2024b. URL
1412.08903
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Inclusion AL Ming-omni: A unified multimodal model for perception and generation, 2025, URL https://srxiv..rrg http://xxxiv.../aaea.. abs/ 250.209366603
Bean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words. arXiv preprint arXiv:2406.11 13340, 2024.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A masssielyy-ultllinguue speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 20..4218-4222 4222, Marseille, France, May 2020. ISBN 979-10-95546-3444
| /ttttts:://acaaeeeee
ogy. .rgg//rg////.. 20wp.2020... 1rec- 1.000
Kirolos Ataallah, Chenhui Gou, Eslam AbdeIrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding. arXiv preprint arXiv:2406. 19875, 2024.

---

Published as a conference paper at ICLR 2026
Simone Balloceu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguis-
lies (Volume I: Long Pupers), pp. 67-93, St. Julian's, Malta, March 2024. URL https:
//aclanthol ogy www.ooollyoooooo.ooo...o wwwp:/oggg222222.. wttp://..4............... 1ong.
Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. Seamlessm4t: Massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308. 1559, 2023.
Luel Hagos Beyene, Vivek Verma, Min Ma, Jesujoba O. Alabi, Fabian David Schmidt, Joyce aakatumba-Nabende and David Ifeoluwa Adelani. msteb: Massively multilingual evaluation of Iims on speech and text tasks, 2025. URL http::tps:/// /ttp:::///axxvvv//// aa//aaxvvvvooo wwt./oggg///....o... 2506 08400
Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. On the implications of verbose Ilm outputs: A case study in translation evaluation. arXiv preprint arXiv: 20.0:44/0000865 2024.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020. URL 145 4ttp/////44777000010006666611110
wtwp::ddiiiiinunndndeu www.ipuuus....... Cc////s.ccc/// tttccc//ppppp/e papeer -a1-644---Ppper..e Paper. pdf
Files/ wales/paperz wwwppppppppp.aaaa 2020
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multi- modal large language models: A survey. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13590- 13618, Bangkok, Thailand, August 2024. doi: 10. 1/224 120.5/////222...nnnggns.sas aadinng--all.077 URL Landdings-acl..000
Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu, Unleashing the potential of prompt engineering for large language models. Patterns, 6(6):101260, 2025. ISSN 2666-3899.
doi: http:://oo..oo111000111../.ttt.t com/ http://ocooooeeeeeeeeee////////2
URL /ttp:://wwww... http:::/ww.cceeedd..eeee..
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T'n, and Haizhou Li. Voicebench: Benchmarking Ilm-based voice assistants. arXiv preprint XXiv:2410..1196, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv.2412.05271, 2024b.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07911 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv 2024.
preprint arXiv:2407.1077
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
12

---

Published as a conference paper at ICLR 2026
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal repre- sentations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 798-805. IEEE, 2023.
John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Made- line Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kub- lik, Meor Amer. Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang. Tim Chung. Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammic Bae, Alcksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Ed- die Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy. Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. Aya ex- panse: Combining research breakthroughs for new multilingual frontier, 2024. URL https: 7/arxiv. orq/ abs/ / 2412 04261
Rocktim Das, Simeon Hristov, Haonan Li, Dimitar Dimitrov, Ivan Koychev, and Preslav Nakov. EXAMS-V: multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7798-7791, Bangkok, Thailand, August 2024.
doi: 10. 1/tp3//11/222aaclllolo 420. URL /ttpss:/acclantho ogy. /trgg22244422 aacl- acw.-lng...............m.oo 4207
Zhichao Duan, Xiuxing Li, Zhengyan Zhang, Zhenyu Li, Ning Liu, and Jianyong Wang. Bridging the language gap: Knowledge injected multilingual question answering. In 2021 IEEE Inter- national Conference on Big Knowledge (ICBK), pp. 339-346, 2021. doi: 10. 10.1109/ 2021.00052.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098-899 891244. 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, 2ts/2306.13344 2023.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075 2024.
Marco Gaido, Sara Papi. Mattco Negri, and Luisa Bentivogli. Speech translation with speech foundation models and large language models: What is there and what is missing? In Lun-
Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers). pp. 14760- https:
14778, Bangkok, Thailand, August 2024. doi: 10. (...1866//112224.cl/ //aclanthology wtwphhollloooogg/////// o204 acl -acl-loong.. 7897
Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr, and Jindong Gu. Benchmarking open-ended audio
dialogue understanding for large audio-language models. arXiv preprint arXiv:2412.05167, 2024.
Mingqi Gao, Wenqing Wang, Xiaojun Wan, and Yuemei Xu. Evaluating factuality in cross- lingual summarization. In Findings of the Association for Computational Linguistics; ACL 2023, pp. 12415-12431, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023 findings-acl.786. httptt /eettpss//aclaathclooy.. org/220033 Findings-acl. 7867
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret,
13

---

Published as a conference paper at ICLR 2026
Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- s00 Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang. Olivier Duchenne. Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Hainng Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin. Dana Beaty. Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Flaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers. Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel. Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri. Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Gehoski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang,
14

---

Published as a conference paper at ICLR 2026
Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya I Lathi, Meghan Keneally, Miao Liu, Michael L.. Seltzer, Michal Valko, Michelle Restrepo. Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usumier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Cherngguz Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao. Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Detito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: / rxiv. otg/ abs
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 03000, 2020.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International conference on machine learning, pp. 4411-4421. PMLR, 2020.
Chien-yu Huang, Wei-Chih Chen, Shu-Wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei- Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Chih-Kai Yang, Xuanjun Chen, Chi- Yuan Hsiao, Puyuan Peng, Shih-Heng Wang. Chun-Yi Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Fabian Alejandro Ritter Gutierrez, and et al. Dynamic- expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: opwnoeeeiiiiee.eemmm net. / forum?id s71z 371ZZPAN7T
phase-2: A collaboratively
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Align- ing perception with language models. Advances in Neural Information Processing Systems. 36: 22096-72109, 2023.
Dahun Kim and Anelia Angelova. Context-adaptive multi-prompt embedding with large language models for vision- language alignment. In Second Conference on Language Modeling, 2025. URL.
https:://ppnree/e
net
Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, and Markus Freitag. Overestimation in Ilm evaluation: A controlled large-scale study on data con- tamination's 's impact machine translation. arXiv preprint arXiv:2501. 18771, 2025.
15

---

Published as a conference paper at ICLR 2026
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:3307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B 2: bootstrapping language-image pre- training with frozen image encoders and large language models. ICML'23. JMLR.org, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195 22206, 2024a.
Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. In European Conference on Computer Vision, pp. 331-348. Springer. 2024b.
Zijing Liang, Yanjic Xu, Yifan Hong, Penghui Shang. Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference Computer, Artificial Intelligence and Control Engineering, pp. 405-409 2024.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Comput. Surv, 55(9), January 2023, ISSN 0360-0300. doi: 10.1 145/335666.. URL httpt:ttttoooooooo org 10. 1et..1115566888
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Findings of the Association for Computational Linguistics: ACI 2024, pp. 8731-8772, Bangkok, Thailand, Au- gust 2024. doi: 10. 18.18/53/v 1/202// 1tw.://.iin...nnnnsndddalll acl.517 URL https: http::://aaaaaattooooo 2024. Fal.iiindigss acl. 517/
org/
Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint K5tv:2502.04228 2025.
Hongyuan Lu, Haoran Yang. Haoyang Huang, Dongdong Zhang, Wai Lam, and Furu Wei. Chain- of-dictionary prompting elicits translation in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pp. 958-976, Miami, Florida, USA, Novem- ber 2024a. doi: 10. 11186////11/22222mmmmpmmmm/aaaa 2024. www.emmain.mm..e.mmmmm 557
URL
Ke-Han Lu, Zhehuai Chen, Szu- Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lec. Developing instruction-following speech language model without speech instruction-tunin -utnin data, arXiv preprint w-Xiv:2409.20007, 2024b.
Ke-Han Lu, Chun Yi Kuan, and Hung-yi Lee. Speech-ifeval: Evaluating citruction-followinn and quantifying catastrophic forgetting in speech-aware language models. arXiv preprini arXiv:2505. 19037, 2025.
Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. Evaluating machine trans- lation output with automatic sentence segmentation. In Proceedings of the Second International Workshop on Spoken Language Translation, Pittsburgh, Pennsylvania. USA, October 24-25 2005.
URL /ttt:ss:/aallaathhollt wttpoeooooooooooo0000000. iwslt-1.15 1.9
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311. 2023.
OpenAI, :.
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin
16

---

Published as a conference paper at ICLR 2026
Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCal- lum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ash- ley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang. Yu Yang, Michihiro Yasunaga, Kristen Ying. Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b & gpt-oss-20b model card, 2025. URL pestps://arxxe org/ abs 2508 10925
Long Ouyang, Jeffiey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Prabhat Pandey, Rupak Vignesh Swaminathan, KV Girish, Arunasish Sen, Jian Xie, Grant P Strimel, and Andreas Schwarz. Sift-50m: A large-scale multilingual dataset for speech instruction fine- tuning. arXiv preprint arXiv:2504.0000 2025.
Diogo Pernes, Gonçalo M. Correia, and Afonso Mendes. Multi-target cross-lingual summariza- tion: novel task and a nentua approach. In Findings of the Association for Compu- tational Linguistics: EMNLP 2024, pp. 12908-12924, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/1120244 2an https://ach
URL
fiddi..-eeniignsmmmiii 1 755/
Yusu Qian, Hanrong Ye, Jean- Phillppp Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia- bench: Towards better instruction following evaluation of multimodal llms. arXiv preprint arXiv:2407.01509, 2024.
Yulci Qin, Yuncheng Yang. Pengeheng Guo, Gang Li, Hang Shao, Yuchen Shi. Zihan Xu, Yun Gu, Ke Li, and Xing Sun. Unleashing the power of data tsunami: A comprehensive survey data assessment and selection for instruction tuning of language models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL htttp::/ net forum? id=RJTibaPhay Survey Certification.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. URL
org 2angg/2222.a2 wmt 1a0cc-1.. 52/
Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, and André F. T. Martins. Towert: Bridging generality and translation specialization in multilingual 11ms, 2025. URL URL htt.h:ttt///xiiii/i
abs abtp:///...//..... 2206. 17080
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arxiv preprint arX6v:2306.1244 2023.
17

---

Published as a conference paper at ICLR 2026
Oscar Sainz, Jon Campos, Iker
Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp- 10776-10787, Singapore, December 2023. doi: 10.18653/ / 1/2023 22g.6//////222...nnnnngeem///// findings--ennlp7222 URL.
ht.tps::/acllamtholoog.. wow.ooo...o..g///. waw.-iiinnnnnionnnnnnnnnnnnne 1 722
ndi
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Armaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXi::2110.08207, 2021.
George Saon, Avihu Dekel, Alexander Brooks, Tohru Nagano, Abraham Daniels, Aharon Satt, Ashish Mittal, Brian Kingsbury, David Haws, Edmilson Morais, et al. Granite-speech: open- source speech-aware ilms with strong english asr capabilities. arXiv preprint arXiv:2505.065. 08699, 2025.
Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for soalescale video understanding. arXiv preprint arXiv:2409. 14485, 2024.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. In Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants!. pp. 11-23, 2023.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.133 13289, 2023.
Team Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, and Marta R. Costa-jussà. SHAS: Ap- proaching optimal Segmentation for End-to-End Speech Translation. In Proc. Interspeech 2022, pp. 106-110 110, 2022. doi: 11444//nterspecch2222222
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. AudioBench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies ( (Volume I: Long Papers). pp. 4297- 4316, Albuquerque, New Mexico, April 2025a. ISBN 8-199-8-8977---1 2025 2a..5.nacl--oon long.221 URL/https: / ac ae//acaann wwwllnnnntolyy..oooo org/ 2025. hat.ssaaaalllllo 2187]
doi: 10. 18633v1vl
Dingdong Wang, Jincenzi Wu, Junan Li, Donghaao Yang, Xueyuan Chen, Tianhua Zhang, and Helen Meng. Mmsu: A massive multi-task spoken language understanding and reasoning bench- mark. arXiv preprint arXiv:2506.044 04779, 2025b.
Dingdong Wang, Jin Xu, Ruihang Chu, Zhifang Guo, Xiong Wang, Jincenzi Wu, Dongchao Yang, Shengpeng Ji, and Junyang Lin. Inserter: Speech instruction following with unsupervised inter- leaved pre-training, 2025c. URL htttps:///aaii......a/ orwp:////ggbbbb///////o// 02769
Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. A survey on cross-lingual summarization. Transactions of the Association for Computational Linguistics, 10:1304-1320 11 2022. ISSN 2307-387X. doi: 10.1162/tacl a 00520. URL
https://doi. .rg /00.11101000 22116/taaa tacllattll_. a0520
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-languee model's perception of the world at any resolution. arXiv preprint arXiv:2409.122 1219,, 2024a.
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406 08035, 2024b.
18

---

Published as a conference paper at ICLR 2026
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting clicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in ma- chine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025a. URL https:: http://aaxvvvv.. www.:///g//aaa/aao.oo..oo.oess aaa///bs///555 20215
Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. survey on multilingual large language models: Corpora, alignment, and bias. Frontiers of Computer Science, 19(11):191 1362, 2025b.
Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, and Xie Chen. Uro-bench: A comprehensive benchmark for end-to-end spoken dialogue models. arXiv preprint vrXiv:2502.178/0 2025.
Tianyi Yan, Fei Wang, James Y. Huang, Wenxuan Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, and Muhao Chen. Contrastive instruction tuning. In Lun-We Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 10302-10302, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 18.13/ win.24..iiinns-aee
URL
orgp/024///2200/0 http:///........n.eoeeeoe..eo Findings-aci
613/
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0rX5v:2505.00388, 2025a.
Chih-Kai Yang, Neo Ho, Yen-Ting Piao, and Hung-yi Lee. Sakura: On the multi-hop reason- ing of large ludio-language models based on speech and audio information. arXiv preprint 2rXiv:2505.13237 2025b.
Qian Yang. Jin Xu, Wenrui Liu, Yunfci Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AIR-bench: Benchmarking large audio-language models via generative comprehension. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Pupers). pp. 1979-1998, Bangkok, Thailand, August 2024. doi: 10. 10.18633/v12000 acl-long. 109. URL wtttts::/aclaattolllleo /
org acl www- long 109/
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Emmanouil Zaranis, António Farinhas, Saul Santos, Beatriz Canaverde, Miguel Moura Ramos, Aditya K Surikuchi, André Viveiros, Baohao Liao, Elena Bueno-Beniee Nithin Sivakumaran, et al. Movie facts and fibs (mf-): benchmark for long movie understanding. arXiv preprint arXiv:2506.0000 06275 2025.
19

---

Published as a conference paper at ICLR 2026
Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-benchee
On multilingual instructionn-ollloinng capability of
large language models, 2025. URL
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2020.
URL /ttppeev//epeeeviee net/
Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exa:: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484-5505, 2023a.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal IIm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408. 13257, 2024a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. nexta-next: A strong zero-shot video understanding model, April 2024b. URL
https: //1las//11llalll github.thubb.... io /tt.///l/////ooooooooooooooooooo (alllo//22220-22 020//22222---22000 llavllllvaaas nexta-nextvvee video/
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00922, 202.b..
Maike 7üfle, Sara Papi, Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, and Jan Niehues. Nut- shell: dataset for abstract generation from scientific talks, 2025. URL https: /tttps://ariv.. org /a///////a/ 2502.0.00 16942
20

---

Published as a conference paper at ICLR 2026
A
TASK DESCRIPTIONS
The description for each task is provided in Table
Task Name
Acronym
Description
Textual Question
TQA
Given a textual context in the source language and question in the target language, the task involves generating an accurate open- ended answer in the target language based on the provided context.
Answering
Text Summarization
TSUM
Given a textual context in the source language, the task involves generating a shorter version in the target language that retains the most important information.
Machine Translation
MT
Given a textual context, the task involves tramslating the text from the source language into different target language, preserving the meaning while adapting to linguistic and grammatical norms.
Automatic Speech Recognition
ASR
The task involves converting the spoken language into written text in the same language, focusing accurate transcription of the speech signal.
Spoken Question Answering
SQA
Given speech context in the source language and a textual ques- tion in the target language, the task involves generating accurate open-ended answer in the target language based on the provided context.
Speech Summarization
SSUM
Given speech context in the source language, the task involves generating shorter version of the spoken content the target lan- guage that retains the most important information.
Speech Translation
ST
The task involves translating speech in the source language into text the target language, combining speech recognition and ma- chine translation in a single task.
Video Question
VQA
Given video context in the source language and textual ques- tion the target language, task involves generating an accurate open-ended answer in the target language based on the provided context.
Answering
Video Summarization
VSUM
Given a video context in the source language, the task involves generating a summary in the target language based on the provided context.
Audio-Video
AVR
Given both video and speech contexts, the task involves generating an accurate content transcript.
Recognition
Audio-Video Question Answering
AVQA
Given both video and speech contexts source language and textual question in the target language, the task involves generating accurate open ended answer in the target language based on the provided audio vistal context.
Aadio-Video Summarization
AVSUM
Given both video and speech contexts in the source language, the task involves generating shorter version of the content in the target language that retains the most important information.
Audio-Vicie
AVT
Given both video and speech contexts, the task involves generating accurate content translation.
Translation
Table 3: Extended description of the tasks supported by the MCIF benchmark.
B DATA CREATION PROCESS AND GUIDELINES
The MCIF gold data comprises English transcripts. QA pairs, and summaries well as their trans- lations into German, Italian, and Mandarin Chinese. In the following, we present all details about the data creation process.
Transcripts. To produce the English talks' transcripts we relied on a language service agency to hire 2 professional English linguists, who were paid per audio runtime (E3/min, in line
21

---

Published as a conference paper at ICLR 2026
with market rates). Professionals were provided with detailed guidelines on how to perform www.dee....ee/raaaaaiipi.......e
the task (available at: bttps::/ buiid/annotat
httpaaasscpppiioooouuuueeeeeeeee Guwdeeiieeenseeee pdf). The process began from the ASR outputs (model details are internal to the agency and thus confidential), with professionals revising and correcting the automatic transcripts using MateDDub CAT-assisted tool that integrates video playback for context-aware transcription. Professionals were instructed to pro- duce clean, fluent transcripts that closely align with the original audio, while respecting technical jargon, US spelling, and proper punctuation. Disfluencies and background noises were omitted. All transcripts were then reviewed by domain specialists who ensured their quality, as well as strict adherence to the task guidelines and correct use of terminology.
Questions and Answers. The creation of the original English QA pairs was carried out by MS/PhD students and researchers, who are all experts in NLP and machine learning. All contributors followed a set of detailed instructions, fully available at wetain/ddttttttttaeeee
/stps://githine
hewellinnnuuuestiioon www.eeinnnnnwenenn_naeaa Geweerriinniiiidiieeeeee..ee dewineseeeee....ee pdff, which outlined the process and quality criteria for creating and annotating the QA pairs. For each talk, annotators were asked to produce at least 10 QA pairs, divided by both the type of question and the type of answer. For question types, each talk required: i) 3 general questions (pre-assigned, the same for all papers), ii) 3 realistic, abstract-base transcript- questions. We enforced this distribution to ensure balanced representation of different question types. In all cases, contributors had to annotate each QA pair with timestamp indicating where the answer appeared in the video, and assign a label reflecting its source of information: A (answerable from audio only), V (from video only), AV (from both), or NA (not answerable). A target distribution of answer labels was also required for each talk: a minimum of 7 A/AV pairs (with at least 5 AV), 2 V pairs, and NA pair. The guidelines, linked above, provided detailed recommendations on how to formulate clear, specific, and concise questions, avoiding vague multi-part formulations, and ensuring answers directly addressed the question in more than two sentences. They also included conventions for US spelling and number formatting.
questions, created after reading the abstract, and iii 4 paper-specific,
Translations. Transcripts. QA pairs and summaries were all translated by professional translators hired via the same language service agency who provided linguists to create the talks' transcripts. Two professionals were assigned for each target language (German, Italian, and Mandarin Chinese), for a total of 6 professionals. Translations were paid per weighted word count (accounting for repetitions, translation memory matches, and MT suggestions) at an average rate of 0.04E/source word. The process started from internal MT system (internal to the agency and thus confidential), with translators working in the CAT tool MateCat They were free to entirely modify or disregard automatic suggestions ensure adequacy and fluency. Translators adhered the original English formatting and respected language variants (Italian for Italy, German for Germany, Mandarin for Chinese). Specifically for transcripts' translation, they were instructed not to translate i) original English paper titles, if any, ii) non-English words or expressions that were provided as multilingual examples during the presentation, if any, All produced translations were further revised by domain experts who are native speakers of each target language to ensure data quality, consistency with the guidelines, and terminological correctness.
Data quality and consistency.x To summarize, our data creation process ensures data quality by relying on hired professionals, domain experts, and by entailing multiple rounds of conformity revi- sions. Notably, we also maximize consistency to allow for comparability across several dimensions:
Cross-lingual consistency was ensured by design, as all original content (talks' transcripts, QA pairs, and summaries) was first created in English and subsequently translated into Italian, German, and Mandarine Chinese;
Cross-modality consistency is ensured by definition, as the same gold data are used for a given task across all modalities;
Consistency with task guidelines was maintained through expert verification, during which domain specialists not only assessed the linguistic quality of the professional transcripts
hntps::/mateddbbcoom
https::/www.matecat.ccm
22

---

Published as a conference paper at ICLR 2026
and translations but also confirmed strict adherence to task guidelines and the correct and translation of domain-specific terminology.
C LIST OF PROMPTS
The fixed prompts used for MCIFix are reported in Table sampled to create MCIF mix is presented in Tables to
The list of prompts from which we
Lang.
Prompt
Recognilion
Transcribe the English content.
Translation
Übersetze den englischem Inhalt nach Deutsch.
Traduci il contenuto inglese in italiano.
Question Answering
Answer the following question concisely given the English content: [QUESTION]
Beantworte die folgende Frage kurz und bündig basierend auf dem englischen Inhalt:
[QUESTION]
Rispondi modo conciso alla
seguente domanda dato contenuto inglese:
[QUESTION]
A " [QUESTION]
Summarization
Summarize the English content an abstract of approximately 200 words.
Fasse den englischen Inhalt in einem Abstract mit maximal 200 Wörtern zusammen. Riassumi il contenuto inglese in un abstract di circa 200 parole.
Table 4: Fixed prompts used for MCIF
23

---

Published as a conference paper at ICLR 2026
Lang.
Prompt
Recognition
Transcribe the English content.
2. Please write down what is said in the English content.
Generate a transcription of the English content.
4. Convert the English content into text.
Produce a written version of the English content.
6. Provide text transcript of the English content.
Accurately transcribe the English content.
8. Turn the English content into written text.
Create a verbatim transcript of the English content.
10. Write out the English content it is stated.
Translation
1. Übersetze den englischem Inhalt nach Deutsch.
Übersetze den englischen Inhalt ins Deutsche.
3. Gib den englischen Inhalt auf Deutsch wieder.
4. Übertrage den englischen Inhalt ins Deutsche.
Führe eine Übersetzung des englischen Inhalts ins Deutsche durch.
Übersetze den Inhalt aus dem Englischen ins Deutsche.
7. Formuliere den englischen Inhalt auf Deutsch.
Erstelle eine deutsche Übersetzung des englischen Inhalts.
Übertrage den englischen Inhalt in die deutsche Sprache.
10. Gib den englischen Inhalt sinngemäß auf Deutsch wieder.
1. Traduci il contenuto inglese in italiano.
2. Dammi
traduzione in italiano del contenuto in inglese.
3. Converti il contenuto inglese in italiano.
Scrivi una traduzione italiana del contenuto in inglese.
5. Traduci in italiano ciò che viene detto in inglese.
Riporta il contenuto inglese lingua italiana.
7. Fornisci uma versione italiana del contenuto inglese.
Effettua la traduzione
contenuto inglese in italiano.
9. Trasforma il contenuto in inglese in una versione italiana.
10. Rendi in italiano contenuto in inglese.
X.
3. B
4. # H
8.
##
9.
10.
Table List of prompts sampled to create MCIF mix (part 1).
24

---

Published as a conference paper at ICLR 2026
Lang.
Prompt
Question Answering
1. Answer the following question concisely given the English content: [QUESTION]
2. Based on the English content, respond to this question with a brief answer: (QUESTION]
3. Use the English content to provide concise answer to the question below: [QUESTION]
Consider the English content and provide a brief reply the question: {QUESTION}
5. Given the English content, what a concise answer to the question: {QUESTION]
6. Relying on the English content, provide a concise answer: [QUESTION]
7. Interpret the English content and concisely respond to the following: {QUESTION]
8. Consider the English content and briefly answer this: (QUESTION]
Use the content English to formulate concise response: {QUESTION)
10. Refer to the English content to answer the question. Be concise: [QUESTION]
1. Beantworte die folgende Frage kurz und bündig basierend auf dem englischen Inhalt:
[QUESTION]
Beantworte folgende Frage kurz und bündig unter Bezugnahme auf den englischen Inhalt: [QUESTION]
3. Verwende den englischen Inhalt, um diese Frage kurz und bündig beantworten:
{QUESTION)
Beziche dich auf den englischen Inhalt an und gib eine kurze Antwort auf die Frage:
[QUESTION]
5. Basierend auf dem englischen Inhalt, beantworte die nachfolgende Frage kurz und bündig:
[QUESTION]
6. Nutze den englischen Inhalt knappen Beantwortung der Frage: [QUESTION]
Analysiere den englischen Inhalt und beantworte die Frage kurz bundig: {QUESTION}
8. Beantworte diese Frage kurz und bündig mithilfe des englischen Inhalts: {QUESTION)
9. Analysiere den englischen Inhalt und beantworte dann diese Frage kurz und bündig:
{QUESTION] Orientiere dich englischen Inhalt und gib eine kurze Antwort:
10. {QUESTION]
1. Rispondi in modo conciso alla seguente domanda dato contenuto inglese: [QUESTION]
2. Rispondi brevemente alla seguente domanda utilizzando il contenuto inglese: {QUESTION]
3. Esamina il contenuto inglese rispondi alla domanda in modo conciso: {QUESTION)
Fornisci uma breve risposta alla domanda basandoti sul contenuto inglese: {QUESTION)
Considera il contenuto inglese e rispondi sinteticamente a questa domanda: (QUESTION)
6. Rispondi alla domanda in modo conciso servendoti del contenuto inglese: (QUESTION)
7. Sulla base del contenuto inglese, dai una risposta concisa alla domanda: [QUESTION]
8. Rispondi sinteticamente alla domanda usando le informazioni del contenuto inglese:
[QUESTION]
9. Considera il contenuto inglese per rispondere alla seguente domanda maniera concisa:
{QUESTION}
10. Utilizza contenuto inglese come hase per rispondere. Fornisci una risposta concisa:
[QUESTION]
1. HE ROBE XTI
H9. [QUESTION] [QUESTION] (QUESTION)
2. He JEA ME
TENNA HI
3. # KU
4. & L
{QUESTION)
14
(QUESTION)
6.
(3)))
(QUESTION]
(QUESTION)
8.
[QUESTION]
M {QUESTION) (QUESTION)
10.
3
Table List of prompts sampled to create MCIF mix (part 2).
25

---

Published as a conference paper at ICLR 2026
Lang.
Prompt
Summarization
1. Summarize the English content in an abstract of approximately 200 words.
Provide summary of the English content using roughly 200 words.
3. Condense the English content into summary of about 200 words.
Write brief summary (about 200 words) of the English content.
5. Summarize the English content, keeping it around 200 words.
Create a concise summary of the English content in about 200 words.
7. Using approximately 200 words, summarize the English audio content.
Capture the main points of the English content in about 200 words.
9. Give summary of approximately 200 words of the English content.
10. Write a short summary (about 200 words) of what's in the English content.
Fasse den englischen Inhalt in einem Abstract mit ungefähr 200 Wörtern zusam- men.
Fasse den englischen Inhalt in ungefähr 200 Wörtern zusammen.
3. Erstelle eine Zusammenfassuun (um die 200 Wörter) des englischen Inhalts. Schreibe eine kurze Zusammenfassung des englischen Inhalts mit ungefähr 200 Wörtern.
Gib den englischen Inhalt in ca. 200 Wörtern wieder.
6. Fasse den Inhalt auf Englisch in ungefähr 200 Wörtern zusammen.
Verfasse eine ungefähr 200 Wörter lange Zusammenfassung des englischen Inhalts.
8. Erstelle eine kompakte Zusammenfassung des englischen Inhalts in ungefähr 200 Wörtern.
Gib eine Kurzfassung des englischen Inhalts in ca. 200 Wörtern.
10. Formuliere eine Zusammenfassung des englischen Inhalts mit ungefääh 200 Wörtern.
1. Riassumi il contenuto inglese in abstract di circa 200 parole. Riassumi contenuto inglese in circa 200 parole.
3. Fai riassunto del contenuto in inglese con circa 200 parole.
4. Scrivi un breve riassunto del contenuto inglese (circa 200 parole).
5. Sintetizza il contenuto inglese in circa 200 parole.
6. Riassumi quanto detto nel contenuto inglese usando circa 200 parole.
Rendi in sintesi contenuto inglese (circa 200 parole).
8. Scrivi un riassunto in circa 200 parole dell' audio inglese.
Esprimi in forma sintetica il contenuto inglese (circa 200 parole).
10. Fornisci una sintesi del contenuto audio inglese in circa 200 parole.
1. 44400 1 17
#1
N
4. # 2
5. 7
6. M1
(400
: 35400
R H40 #400
H
10.
Table 7: List of prompts sampled to create MCIF mix (part 3).
26

---

Published as a conference paper at ICLR 2026
D MODELS
The models used for the analyses are listed in Table g (next page) and run using the HuggingFace Transformer version indicated for each model, some of them require specific version.
For all models, we use the default generation parameters, following the usage instructions reported in the model cards. When available, adopt the suggested system prompts for each model, with the additional instruction: "Only return the answer requested. Do not include any explanation or introductions." The maximum number of new tokens set 4096 for all models. The code used for inference is released upon paper acceptance. The inference is performed using a single NVIDIA GH200 120GB GPU.
27

---

Published as a conference paper at ICLR 2026
Model
Param.
In.Mod.
Weights
HFY
Aya Expanse Dang et al.
8B
bttps://uggiaggaace
4.51.3
2024
Cohere Lolerre.abs//yy==xxppp5s
Gemmaa 3
Team Gemma
12B
bttps:/htt.co/goon
4.51.3
et al.
2025
Llama Grattafioni
8B
Settss//ht..oommeta-1
4.513
et al. 2024)
8.1-a-33.--B--n--- Instruuctt
GPT-oss (OpenAl et al.
20B
https: https://huggintaaae
4.55.0
2025
apenai/gpt-oss-
Phi4 (Abdin et al.
2
14.7B
https://hf.co/mierrrs phi-4
4,51.3
Qwen3 Yang al. 2025a
14B
https://huggingfen
4.51.3
Owen/Qwen3-1e
Tower-Plus (Rei et al. 2025
9B
https: htggs://hhuginggfae.
4,51.3
Unbabe /aabbe//oooo /Toeer
DeSTA2
(u ct al 2024b
8B
https://ff.coDDS//1-1HG
4,51.3
DeSTA2-
GraniteSpeech 3.3 (San ct al. 2025)
8B
bttos://ht.co/imm-qraanii gran1te-8ppch--333-88
4.52.4
Phi4-Multimodal (Abdin el al. 2024b)
5.6B
https://hf.co/miccrose
4.48.2
Qwen2-Audio (Chu et al.
7B
https://nf.co/Dwas
4,51.3
2024)
UltraVos 0.51
8.07B
https://hf.cc/Fiiieee
4,51.3
502ravox-v0--1lammm 2-
Inter (Chen al. 2024b)
14B
httpss//huggnggfaee.oo
4.51.3
1pen@Viab/LntervV3333
LLaVA-NeXT Video Zhang et al. 2024b
7B
https://huggnng...
4,51.3
co co/llava-hf/
Qwen2.5-VL (Wang al. 2024a)
7B
EI
https://huggLnggacen
4.51.3
2wco:/pen22.5-11-7---111t
Vad33LLaMA3 (Zhang ct al. 2025
7B
Ittpss//nugginngacee.o
4.51.3
Video-XL XL-2 (Shu et al. 2024
8B
https:/huggnngaccc.oo
4.51.3
BAAI/Video-XI- XL-2
Gemma 3n
4B
https:
Bttpss://hqggingfacecco
4.53.0
google 000g1ee/ggmmm-33--E--3
Ming-Lite-Omne Omni AI
2.8B
M
https://huggingg/
4.45.0
2025
InelussionA.MMing-liii.
MiniCPM-o-2 Yao et al.
8B
https::/huuqinggfan
4.44.2
2024
Chenbmb/MiniCC71
Ola Liu et al.
2025
7B
4,43.4
THUdyh /0
Qwen2.5-0mni Xu et al.
7B
attps://ht.ccmmmccrooao
4.51.3
2025a
Phi-4-multinoda
Table 8: Details of the models, including the number of parameters (Param.), input modalities analyzed in this paper (In.Mod.), their public weights release (Weights), and the HuggingFace Transformer version (HFy) used for the experiments on the MCIF benchmark. |/ttps;; / www.
ultravox. ai/
https: betess:/ddeemiiii //deppiinnn groeemmmidd.gooollemmmmeee..s.s gttpl///gg//n/nn/aaeaa -aaa///emmmmnnnnnm
28

---

Published as a conference paper at ICLR 2026
E
EXTENDED RESULTS
The scores MC and MCIFmix per language are presented in Tables and 10
Context
REC. WER.
T'RANSLATION COMET
QUESTION ANSW. BERTSCORET
SUMMARIZATION
Input Modality
Model
BERTSCORE
DeSTA2
54.0
72.5
76.7
76.7
23.7
24.7
17.9
2.4
GraniteSpeech
9.4
42.0
52.2
62.1
11.8
1.5
0.4
-11.6
Ph14 Multmmoaal
6.8
77.7
81.2
81.6
42.3
33.0
32.9
40.0
Owdio
31.7
71.6
73.9
79.3
33.3
30.5
30.6
36.1
UltraVox v0.5
127.7
45.9
50.6
33.5
22.8 16.1
22.9
InterVLL3
30.9
28.6
30.0
37.4
ShORT
Lax--Nex
15.3
10.3
8.8
20.3
Owen2.3-V VL
34.3
38.9
38.8
44.5
Videol LaMA.
16.7
31.2
20.0
28.7
Vıdeo-X1.
16.7
12.1
Gemnma 3n
35.1
70.6
74.2
74.3
25.5
27.5
26.6
25.0
Ming-L Lite- Omni
117.5
55.8
55.9
47.4
21.0
14.6
14.6
13.1
MiniC PM-0-2
144.8
35.0
41.3
42.9
23.0
18.8
18.6
25.0
104.1
72.5
76.9
33.3
37.2
39.3
QwenZ.55 --mi
43.5
74.2
76.4
81.2
35.8
35.5
34.0
32.0
Gemin 2.3 Flash
14.9
62.7
65.7
72.6
45.5
41.6
37.6
37.7
Aya Expanse
62.5 68.6
74.9
28.1
28.9
25.2
24.4
16.7
22.7
24.6
36.4
Gemma
82.3 87.9
86.3
29.1
26.7
25.2
10.4
21.6
11.6
12.6
77.
GPT-OSS
72.0 78.8
74.4
21.0
24.6
22.2
30.6
11.0
15.8
12.0
32.4
Lama
x
84.3
29.7
31.3
29.4
30.7
16.7
23.9
Ph14
83.0 85.7
84.8
32.5
32.5
33.0
25.1
20.1
23.8
16.0
-7.7
3wen
82.5 86.4
85.4
37.9
40.7
36.3
36.8
22.4
12.5
22.6
22.3
Tower+
83.6 87.3
85.9
30.2
31.8
29.7
26.4
18.9
11.4
19.6
29.2
DeSTA2
112.9
43.7
40.4
18.3
18.3
-7.7
LONG
GranitcSpecch
99.9
35.4 40.3
32.3
-22.3
-26.1
-25.7
-20.5
-7.0
-10.0
-10.0
-14.7
Ph14 Multimodal
39.2
56.3 66.4
56.5
39.1
36.0
33.8
41.6
18.0
19.7
18.4
14.8
Auudoo
92.9
39,3
43.2
40.5
28.9
27.7
26.9
31.5
3.0
5.8
10.1
9.2
VotraVox v0.5
89.1
36.8
40.8
36.4
21.4
12.1
4.2
13.2
6.4
-4.9
-3.7
-9.7
L13
26.0
27.0
26.1
31.4
15.6
14.8
19.1
32.2
LLaVA Nex
9.7
-1.5
1.7
18.7
-6.9
-5.3
-6.3
.9.5
Owen2.5-VV VT
x
X
25.7
36.3
36.0
37.0
19.4
13.6
24.2
35.9
VideoLLaMA
22.9
31.4
20.7
32.2
-6..5
17.8
-21.7
-23.7
Video-
20.1
15.7
14.9
18.1
9.6
6.5
4.1
-5.4
MiniCPM-o-2
170.6
24.6
24.9
25.2
-48.2
-35.4
-38.2
-30.3
-5..7
-39.1
-37.1
-20.6
Ola
14.0
60.3
73.4
55.8
32.4
38.0
38.7
35.6
18.2
11.2
12.4
7.3
QwenZ.5 Omni
98.5
37.2
42.3
40.8
34.4
26.3
26.3
43.0
17.3
10.4
11.9
-3,9
Gemini 2.3 Flash
11.9
78.1
79.1
71.9
47.8
43.7
46.3
46.6
16.7
20.8
23.1
35.6
Table Results MCIF of the 23 models for each input modality, context, task (divided into the four macro-tasks: RECOGNITION, TRANSLATION, QUESTION-ANSWERING RIZATION), and target language for English, for German, for Italian, and for Chinese, while source language, being always English, is omitted). The best result by context is marked in bold, and the best overall result is underlined. Gemma and Ming-Lite-Omne Ommi are removed from LoNG scores as they are not able to process ffom speech.
and SUMMA-
29

---

Published as a conference paper at ICLR 2026
Context
REC.
TRANSLATION COMET
QUESTION ANSW.
SUMMARIZATION
Input Modality
Model
WERL
BERTSCORET
BERTSCORET
DeSTA2
83.0
72.4
76.5
76.6
0.8
peech
9.5
42.9
49.4
47.5
11.4
0.7
0.7
-11.4
h144 Mulmmul
6.7
77.7
81.3
81.3
43.4
37.0
35.5
33.7
Owen2 Audio
31.9
71.3
73.8
78.8
34.1
32.3
31.0
33.6
Ultra Vox v0.5
172.6
43.4
43.2
43.1
18.9
21.0
17.9
18.4
VL3
28.8
38.9
SHORT
LLaVA-NeX
12.2
8.8
8.2
19.4
Owen2.3- VL
x
X
33.9
39.2
37.6
40.5
VideoLLaMAA
22.3
25.8
21.6
25.5
Video-XL2
17.6
15.6
13.4
7.8
Gemma
58.9
70.7
72.3
30.4
26.0
17.7
Ming-L.ite Omni
128.2
57.3
55.1
47.5
15.4
16.2
10.7
11.0
MMmnCPM---22
207.1
34.5
40.0
41.8
19.6
24.7
20.4
27.7
X
Ola
98.8
72.5
76.4
80.1
34,3
36.1
37.8
39.9
QwenZ.S- Omni
48.0
74.0
74.4
81.0
36.9
35.9
35.4
32.0
Gemin 2.3 Flash
12.8
67.4
68.2
72.0
40.9
39.3
39.1
38.6
Ava Expanse
63.3 70.1
72.9
20.9
23.0
24.5
15.2
21.8
24.0
35.5
Gemma
86.0
82.7
30.6
25.3
25.4
19.1
11.4
12.6
-5.3
oss
66.7 73.5
70.2
18.3
22.6
21.8
21.0
7.7
13.7
12.9
29.3
Jama 3.
x
79.5 82.1
76.8
31.0
30.6
30.2
32.2
19.9
19.6
23.3
35.3
Ph14
82.0 86.9
85.1
31.8
31.7
32.5
22.5
18.7
21.4
17.4
0.6
Owen3
82.8 85.8
84.9
35.4
39.2
35.1
32.6
20.8
19.4
23.7
16.3
lower+
81.4 83.5
86.0
24.9
26.8
29.8
12.2
18.2
15.0
21.9
15.5
DeSTA2
132.5
39.5
43.2
39.7
17.8
18.3
16.8
-2.
4.7
5.6
7.9
-6.8
LONG
GramleSpeech
80.4
35.1
39.4
29.4
-21.9
-25.3
-24.2
-1999
-6.8
-11.0
-9.4
-14.6
Ph14-MMllin Mullimoda
29.8
60.7
65.4
52.4
39.5
35.8
34.4
39.6
17.6
18.0
18.1
17.8
Qwen Audio
93.1
39.5
43.4
40.5
29.5
28.5
28.3
29.5
2.1
6.4
10.0
6.5
Ultra Vox vU.5
92.5
36.7
40.6
36.7
21.9
8.2
8.9
11.1
6.1
-4.6
-5.2
-8.7
VL33
25.9
26.9
24.6
34.1
13.7
17.9
18.1
31.9
LLaVA-NeX
3.2
-1.8
3.4
16.2
-9.6
-7.2
-6,2
-3.7
QwenZ.3 VL
X
X
31.9
36.8
37.6
33.3
17.2
18.7
21.0
23.8
VideoLLaMAA
24.1
30.7
24.0
27.2
-3.7
-31.4
-25.2
-71.9
Video-X1.2
21.7
18.4
16.5
13.0
7.5
1.4
0.1
7.6
MiniCPM-o-
179.2
25.0
26.6
26.0
-47.5
33.2
-37.2
-30.7
59.0
-39.5
-39.4
-20.8
Ola
36.8
47.1
59.1
48.5
36.1
33.5
34.8
32.4
15.3
10.9
10.9
18.1
QwenZ.5 Ommi
94.9
37.5
41.6
41.6
41.3
31.4
31.0
35.7
8.8
10.8
10.5
7.5
Gemmn 2.3 Plash
7.9
75.2
81.2
83.3
45.3
44.9
46.5
47.0
16.0
19.7
20.7
30.6
Table 10: Results MCIF mix of the 23 models for each input modality, context, task (divided into the four macro-tasks: RECOGNITION, TRANSLATION, QUESTION ANSWERING, and SUMMARIZATION), and target language ( for English, for German, | for Italian, and for Chinese, while source language, being always English, is omitted). The best result by context marked bold, and the best overall result is underlined. Gemma 3n and Ming-Lite-Ome are removed from soNs scores as they are not able to process long-form speech.
30

---

Published as a conference paper at ICLR 2026
F FAILURE EXAMPLES
Examples of the failure cases of the benchmarked systems mentioned Section 5 Table
are presented in
II
Table 11: Examples of models failures on the MCIF benchmark (non-exhaustive).
#
Model
Task
Output
Ola, LLxV
REC SUM
Transcription output contains only table entry:
Main findings 85 FTw COSINE L2R MLC 80 75 70
65 60 55 50 All weak labels 40 30 20 10 0 5 10 20 30 40 50 All Validation Performance
Delta
FTw COSINE L2R MLC Adapterc FTc
LoRAC BiFiTe Adapterc
Output contains the description of single slide instead of sum- marizing the video:
The Lmage displays graphical representation of model's performance a dataset, showl ng the model's accuracy across different layers and the overall accuracy. It highlights that the model's performance improves it goes deeper into the layers, with the highest accuracy at the base layer and decreasing as it moves towards the top layer. The model's s performance is also compared to baseline model, which has lower accuracy.
Qwen2-Audio
REC TRANS
The transcription of whole talk contains only the first part: The original content of this audio is: Hi, my nanc is Matthias Lendermann, and today I am going to give you brief introduction our paper compositional generalisation without trees using multisets tagging and latent pezmutations. This is joint work with my advisors Alexander Colla and Ivan Tovstilov. Compositional generalisation be understood the ability of a learner to handle deeper recursion and unseen composi tions."
The translation of whole talk contains only the first part: Hallo und willkommen von DeepL, einem neuer: Modul für die deutsche Texterkennung auf Dokumentebene und auf Satzebene. Mein Name ist Regina Stodden und ich werde Ihnen in der ersten Teil des Vortrags führen. Lassen definiert was I'extsimppl i inplifiiierunn ist ein Prozess, bei dem ein Text angepasst wi wird, um seine Verständlichkeit für einen bestimmten Zieigruppe zu verbessern. Wie Menschen lesen.
Präsentation
uns zunächst ist.
Continued on next page
31

---

Published as a conference paper at ICLR 2026
Table 11: Examples of models failures on the MCIF benchmark noon-exhaustive). (Continued)
Gemini 2.5
TRANS
Output contains an over-claboration on audio-visual content:
Flash
SPECIAL INSTRUCTION: I will first think step by step how to translate the audio and then apply the thinking to the tzanslation.
Internal monologue: transiate the English audio into Gorman, I need to listen carefully to the speaker's s words and translate then accurately while maintaining the original meaning and [-.]]
user wants me to
"Jedoch kann dieser Effekt gemi Idert. werden, äh, wenn, ähm, äb, wenn [..]
Output contains an hallucination audio-visual content:
1. Semp wifficiiiiinsss del Testo
Cosa, perché e come?
** Testo parlato:** Ciao [...]
4
GraniteSpeech, LLaVA-NeXT,
TRANS QA
Output in English while prompted for translation in Chinese: hi welcome to
gresentation of deplain new corpus for german text semplificon cemplification [...] Output in English while prompted for QA in Italian: The Transformes model is known for its efficiency and effectivenes sequential data, which [...]
Tower+, Phi4- Multimodal
SUM
in handling
Output in English while prompted for summarization in German: We present CREST (A Joint Framework for Rationalization and Counterfactual Text Generation) a novel approach that [...] Output in English while prompted for summarization in Chinese: Shanshan Chen, master student from Shenzhen Un Universi presented her work on efficient open domain question arswering at ACL 2023. The work focuses on a two-stage model for question answering, [..]]
GraniteSpeech
QA
Answer to the question "Which model did they obtain the 82%-87% accuracy?" is random text: the 1dt corpus has 60CC alternative questions across three domains and it has 42000 indirect referring expressions results with t5x-large model are summarized below Full transcript as response to "What are the main data sources for language models?"
hi 1/m jianbin phd student at university of washi ngtto today i'm presenting [..] i think that's s pretty much all have for today thank you for your time
6
UltraVox v0.5
QA
False detection of harmful request or denial of responding. E.g., for 400 Ai B1 "in different samples the responses are:
1)
cannot provide information on illegal activities. Is there anything else I can help you with?
I cannot fulfill your request.
32


Reviews:
[{"review_id": "YHIHmTOYiV", "reviewer": "Reviewer_MiXV", "summary": "This paper presents MCIF, the first human-annotated, multimodal, crosslingual instruction-following benchmark built from scientific presentations. It evaluates large multimodal language models (MLLMs) across three modalities (text, speech, and video), four languages (English, German, Italian, and Chinese), and 13 tasks grouped into four macro categories: recognition, translation, question answering (QA), and summarization. The dataset includes short-form and long-form contexts, enabling the study of model performance across input complexity.", "questions": "1. How you control the quality to ensure annotation consistency across languages and modalities, especially for translations and Q&A? \n\n2. Can you elaborate on whether degradation in long-form tasks is primarily due to model architecture (context window limits) or instruction misalignment? or any other potential reasons?\n\n3. For the variation shown between MCIF_fix and MCIF_mix, do you suggest any strategies to improve real-world prompt robustness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a critical gap in evaluating instruction-following across multiple languages and modalities - an under-explored but essential part of MLLMs.\n\n2. MCIF is carefully constructed with parallel multimodal and multilingual alignment, allowing systematic and fair cross-dimensional comparisons—something absent in prior benchmarks. Using human-curated data from scientific talks ensures higher linguistic and contextual richness than synthetic or crowdsourced datasets. The academic domain provides realistic, challenging material.\n\n3. It conducts comprehensive evaluation over 23 diverse models (open and closed-source, text-only and multimodal) under both short- and long-form settings provides broad insight into model generalization and limitations. And it also introduces MCIF_fix vs. MCIF_mix is a thoughtful addition—it measures how sensitive models are to prompt variations, reflecting real-world user interaction diversity.", "weaknesses": "1. Scale issue: despite being well-curated, MCIF covers only ~10 hours of content and 21 talks, which may limit statistical robustness and task diversity.\n\n2. Human baseline: it is better to establish a human baseline, making it clear how far current systems are from “human-level” multimodal understanding.\n\n3. Evaluation metrics: Automatic evaluation on WER, COMET, and BERTScore may not well capture multimodal grounding or factual consistency - especially for tasks like question answering or summarization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jNoRRgE2Cx", "reviewer": "Reviewer_Qo2z", "summary": "This work introduces a new benchmark called MCIF, comprised of four classes of tasks (transcription, translation, question answering, and summarization) with aligned multimodal data (text, speech, and audio) from ACL lectures across four languages (English, German, Italian, and Chinese).\n\nThe benchmark uses human experts to provide annotations such as audio transcriptions and translations. Domain experts contribute the Q&A pairs, including unanswerable pairs.\n\nThe paper reports performance of 23 different models supporting various subsets of modalities (or all of them) and provides some analysis on prompt adherence and the contribution of individual modalities for those models that support multiple.", "questions": "* In table 1, the context type for the text-inputs is “-“. Does that mean that for the short context setting we still provide the full transcript and only non-text modalities are shortened to the relevant region of interest? If not, why is the performance of text input for short context not reported for translation and QA?\n* In the main results in table 2, are all question pairs used for QA regardless of whether the questions can be answered with the provided input modalities? If so, what is the “correct” way to answer questions that are not answerable with the provided input modalities (including questions that are generally unanswerable)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The construction of the benchmark heavily relies on human experts, including domain experts of the material used, for ground truth annotation. It is really great to see such a rigorous collection protocol.\n* The proposed task metrics (WER, BertScore, and COMET) are widely used and can be computed without relying on external APIs that may change over time.\n* The paper presents results on their benchmark across 23 different models from various model families.", "weaknesses": "* The proposed benchmark is cross-lingual only from English into one of the three alternative languages (German, Italian, Chinese). Input audio and video (e.g. slides) are always English. This may limit the utility of the work for evaluating multi-lingual models’ capabilities in practice.\n* While the work is clearly describing their contribution to be a benchmark about scientific talks, the data used seems quite narrow even within that domain; The benchmark exclusively uses recordings from presentations of main conference papers at ACL 2023. The limitation to this split was described as aiming to select videos newer than the knowledge cutoff of models to be evaluated, but this seems perhaps not well motivated as many of the evaluated models in the paper have a cutoff later than that (e.g. Phi 4, Gemma 3, GPT-oss, Gemini 2.5 Flash, for example). Selecting recordings from a single conference from a single year, and only main conference papers seems to significantly and perhaps arguably unnecessarily constrain the diversity of the produced benchmark (topics presented, recording setup and environment, speaker demographic,  etc. are presumably quite homogenous in the resulting work).\n* The limited diversity of the benchmark, in my opinion, makes it a bit difficult to draw strong conclusions from the observed results. For example, section 5.1 argues that since no improvements were observed on the summarization task when adding speech or video in addition to the text transcript, this “underscores the limitations in multimodal integration”. However, an alternative interpretation may be that since the target summary is the abstract of the presented work, the content expected there may be particularly well captured in a transcript of the talk, since by nature such abstract would not include anything that may only occur on stage or any specifics of the visual presentation.\n* The work focuses on models smaller than 20 billion parameters for its main study. While technical limitations are understandable, evaluating some stronger frontier models besides Gemini 2.5 Flash would be a welcome addition (GPT-5, Gemini 2.5 Pro, for example). Section 5.2 also does not seem to discuss Gemini 2.5 Flash, which is perhaps particularly limiting as the conclusion suggested, that “current MLLMs struggle to effectively integrate speech and video”, which may be particularly true for smaller model sizes, and notably Gemini 2.5 Flash was the best performing audio-video model reported in Table 2.\n* The paper discusses instruction following through the lens of prompt adherence by comparing the results of $MCIF_{fix}$ and $MCIF_{mix}$. This seems a somewhat limited view of instruction following since the main signal will be whether the model will treat one (specific) prompt similar to 10 other semantically equivalent paraphrases. For example, assume a model that is very sensitive to the prompt - as long as the particular selected prompt for $M_{fix}$ achieves performance similar to the average of performances of $MCIF_{mix}$, this issue would not be visible in the chosen metrics. Additionally, the prompts themselves are relatively basic one-sentence instructions which may not exercise the instruction following capabilities of modern LLMs to their fullest extend.\n* (Note that the first prompt for summarization in German as shown in Table 7 seems semantically slightly different to the other 9; it asks for a summary with at most 200 words instead of approximately 200 words; I am unable to verify all prompts, but minor semantic differences may also explain much of the observed fluctuation in scores?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpjysSVPsz", "reviewer": "Reviewer_sQwM", "summary": "The paper presents MCIF, a new benchmark to test how well large multimodal models can follow instructions across languages and input types. The dataset comes from real scientific talks with aligned text, audio, and video in four languages: English, German, Italian, and Chinese. It covers 13 tasks grouped into recognition, translation, question answering, and summarization. The authors also test 23 different models including text-only, speech, video, and multimodal ones. The results show that models still struggle a lot when dealing with long inputs, with combining speech and video, and when asked to summarize complex content.", "questions": "How consistent were the human annotations across different annotators and languages?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The idea is very timely. Multimodal and multilingual instruction following is clearly the next big step for LLMs.\n\n2) The dataset design is thoughtful. Everything is aligned across languages and modalities, which makes comparisons fair and controlled.\n\n3) It is fully human-annotated. The effort to manually create transcripts, translations, and QA pairs really increases reliability.\n\n4) The analysis is broad and systematic. The paper looks at different model types, context lengths, and prompt variations. The findings are detailed and easy to follow.\n\n5) The authors share code, prompts, and data guidelines openly, which helps with reproducibility and future research.", "weaknesses": "1) The dataset size feels small compared to other multimodal benchmarks. About ten hours of content may not capture much variation in topics or speakers.\n\n2) The evaluation relies mostly on automatic metrics. Some human checks or qualitative examples would make the results more convincing.\n\n3) The discussion could go a bit deeper on why models fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4uuo00WkbC", "reviewer": "Reviewer_2tAT", "summary": "This paper proposes a new benchmark for evaluation of instruction-following capabilities under a comprehensive context of multi-modal, cross-liingual, and varied input length. The benchmark is validated on 23 models, which is good for comparison, and provides guidance on future development.", "questions": "* Perhaps modify the descriptions about the scientific talks to be clear, explicit in the introduction.\n* The authors are encouraged to extend more metrics as a complement to existing rule-based computing metrics.\n* The user-simulation by LLMs is important to an evergreen benchmark which prevents from hacking. The authors are encouraged to provide such prompt augmentation.\n* The discussions should contain explicit take-home messages which help the development of future MLLMs. In addition, the case-by-case (especially typical cases in each task) analysis should be provided to be intuitive.\n* The authors might consider citation of recent instruction-following studies from the perspective of data recipe:\ne.g., Qin, Y., Yang, Y., Guo, P., Li, G., Shao, H., Shi, Y., ... & Sun, X. (2024). Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models. arXiv preprint arXiv:2408.02085.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* This benchmark is indeed covering a wide arange of topics, tasks, and languages. It could be used as a supplement to the existing benchmarks that challenge the IF capabilities of MLLMs.\n* The paper is generally well-written, easy to follow with pretty illustrations.\n* The detailed descriptions on manual annotations are provided for guaranteeing the quality of the benchmark.", "weaknesses": "* The definition of \"scientific talks\" might be limited in the ACL context, which might not be that broad enough to include nature science/medicine domains.\n* The evaluation metrics might not be comprehensive enough where recent LLM-as-a-Judge-based metrics are under-explored.\n* The prompt templates (see Table 4,5,6) might not be varied enough to be practical. The dynamic prompting by user-simulation (LLM-driven) is not considered and there might exists a risk of over-fitting where future MLLMs might overfit the patterns inherent in such fixed, less-varied templates.\n* Discussions and analyses are not enlightening enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)