video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
Authors: Guangzhi Sun
Yudong Yang
Jimin Zhuang
Changli Tang
Yixuan Li
Wei Li
Zejun MA
Chao Zhang
Venue: ICML 2025 poster (2025)
Keywords: audio-visual
chain-of-thought reasoning
large language models
process reward model
direct preference optimization
Primary area: deep_learning->large_language_models
TL;DR: video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM

Abstract: While recent advancements in reasoning optimization have significantly enhanced the capabilities of large language models (LLMs), existing efforts to improve reasoning have been limited to solving mathematical problems and focusing on visual graphical inputs, neglecting broader applications in general video understanding. This paper proposes video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM designed for general video understanding tasks. To enhance its reasoning abilities, we develop a reasoning-intensive dataset featuring challenging audio-visual questions with step-by-step solutions. We also propose process direct preference optimization (pDPO), which leverages contrastive step selection to achieve efficient step-level reward modelling tailored for multimodal inputs. Additionally, we introduce RivaBench, the first reasoning-intensive video understanding benchmark, featuring over 4,000 high-quality, expert-curated question-answer pairs across scenarios such as standup comedy, academic presentations, and synthetic video detection. video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision baseline across different video reasoning benchmarks. Besides, pDPO achieves 6-8% improvements compared to the supervised fine-tuning model on RivaBench. Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection capabilities.

video-SALMONN-ol:
Audio-visual Large Language Model
Guangzhi Sun 123
Yudong Yang'
Jimin Zhuang' Changli Tang' Chao Zhang'
Yixuan Li' Wei Zejun Ma
Abstract
els (LLMs) (OpenAI, 2024; DeepSeek Team, 2024; Qwen Team, 2024b; Zhao et al., 2024; Yuan et al., 2024) per- formance in answering complex logical questions, such as math problems (Yang et al., 2024; Wang et al., 2024b; Sun et al., 2024d; Ying et al., 2024) and coding tasks (Zhang et al., 2024f). These methods usually first split the solution into multiple simpler steps to form a reasoning path ending with the final solution, demonstrated in chain-of-thour (CoT) (Wei et al., 2022). Advanced training approaches have been developed such as the outcome reward model (ORM) (Cobbe et al., 2021; Yu et al., 2024a; Zhang et al., 2024b) that optimizes the entire reasoning path based on the final solution, and the process reward model (PRM) (Uesato et al., 2022; Lightman al., 2023; Luo al., 2024; Zhang et al., 2024a) that optimizes each reasoning step based on how likely each step would lead to a correct answer.
While recent advancements in reasoning optimiza- tion have significantly enhanced the capabilities of large language models (LLMs), existing ef- forts to improve reasoning have been limited to solving mathematical problems and focusing on visual graphical inputs, neglecting broader appli- cations in general video understanding. This pa- per proposes
the first open- source massoning-enhance audio-visual LLM de- signed for general video understanding tasks. To enhance its reasoning abilities, we develop a reasoning-intensive dataset featuring challenging audio-visual questions with step-by-step solutions. We also propose process direct preference op- timization (pDPO), which leverages contrastive step selection to achieve efficient step-level re- ward modelling tailored for multimodal inputs. Additionally, we introduce RivaBench, the first reasoning- intensive iensoning-intensvve video understanding bench- mark, featuring over 4,000 high-quality, expert- curated question-answer pairs across scenarios such as standup comedy, academic presentations, and synthetic video detection. Aideo-SALMONN. ol achieves 3-8% accuracy improvements over baseline across different video reasoning benchmarks. Besides, pDPO achieves 6-8% improvements compared to the supervised fine-tuning model on RivaBench. En- hanced reasoning enables video-SALMONNN-ol zero-shot synthetic video detection capabilities!
In addition to text-based questions, reasoning also plays an indispensable role understanding the physical world, such as comprehending concepts in an academic presen- tation, interpreting complex interactions among people or even detecting artificial anomalies. Thus, improving rea- soning ability is also critical for multimodal LLMs (Tang et al., 2024c;b; Sun et al., 2024b; Cheng et al., 2024; Zhang et al., 2024e; Lin et al., 2024; Team et al., 2024; Wang et al., 2024a; Tang et al., 2024a) that process audio and visual inputs in addition to text, as the interactions among mul- tiple modalities can largely increase the difficulty of the task. To this end, investigations have been performed on optimizing the reasoning process with multimodal inputs (Du et al., 2024), and particularly visual LLMs (Qwen Team, 2024a; Xu et al., 2024; Du et al., 2025). However, current research on enhancing reasoning capabilities for multimodal L.MSs has predominantly focused on solving mathematical problems and image inputs. This overlooks the importance of reasoning general video understanding and the interactions among audio, visual and text modalities, largely limiting their scopes of applications.
the
1. Introduction
The recent advancements in optimizing the reasoning pro- cess have further boosted text-based large language mod-
'Tsinghua University 'BttDDance University of Cambridge. Correspondence to: Chao Zhang <cz277
This paper proposes
the first open- source reasoning-enhance audio-visual L.M with im- proved reasoning abilities in general video understand- ing tasks. The audio-visual reasoning capability of video- SALMONN-ol is first enhanced by creating a new dataset with challenging questions and step-by-ster solutions for su-
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Coode available: attps://githin vide SALMONN-o1.

---

videe-SALMONN-ol: Reasoning- Ennannned Audio-Visual Large Language Model
pervised fine-tuning (SFT), and then further boosted by the proposed variant of direct preference optimization (DPO), process DPO (pDPO) (Rafailov et al., 2024; Zhang et al., 2024c). pDPO achieves step-level pairwise reward mod- elling via an efficient contrastive step selection approach tailored for multimodal inputs. While being more effective than the standard PRMs in general video understanding. pDPO and the step selection make audio-visual reasoning more efficient without the need for an external reward model or two-pass re-ranking pipeline.
as one of the most prevalent approaches in reasoning opti- mization tasks (Uesato et al., 2022; Lightman al., 2023; Luo et al., 2024; Zhang et al., 2024a; Li et al., 2023b).
However, constructing step-level annotations for PRM train- ing can be expensive and difficult to scale up. As mitigation, Wang et al. (2024b) and Luo et al. (2024) proposed auto- matic step annotation using rollout, which approximated the expected correctness of each step by sampling multiple paths till the end with the same prefix solution. In particular, Luo et al. (2024) treats the first wrong step as the critical step to perform rollout which was found by binary search.
To evaluate the performance on multimodal reasoning for general video understanding, we propose the first reasoning- intensive video with audio understanding benchmark (Riv- aBench). RivaBench primarily focuses on three represen- tative scenarios, including standup comedy, academic pre- sentation and synthetic video detection. In particular, Riv- aBench contains over 4k high-quality question-answer pairs that are carefully crafted by human experts (e.g. medical doctors). Our key contributions are summarized as follows:
2.2. Reasoning in Multimodal LLMs
Researchers have been investigating optimizing CoT rea- soning for multimodal LLMs to tackle increasingly chal- lenging tasks. Most of them focus on extracting graphical or text information from an image and solving mathemati- cal tasks based on the extracted information. Specifically, LLaVA-CO (Xu et al., 2024) investigated better sampling and search algorithms find better reasoning path for math questions with image inputs. Virgo, on the other hand, explores the fine-tuning data organization and trans- ferability of text-based reasoning tasks to image-based rea- soning tasks (Du et al., 2025). Recently, MAmmoTT-V VL (Guo et al., 2024) built a large-scale multimodal instruction- tuning dataset that can improve the question-answeriigg per- formance on diverse modalities including video. Differ- ent from these works, Aido-SAALMONN-ol particularly focuses on general video understanding scenarios, where different parts of the audio-visual information are constantly referred to during the reasoning process.
We propose Tide--SALMONNE the first open-source reasoning-enhancer audio-visue LLM for general video understanding tasks.
video-SALMONN-ol is the first to explore RL-based reasoning optimization for general video understanding. The proposed pDPO method with efficient contrastive step selection further enhances reasoning abilities.
We propose RivaBench, the first general video under- standing benchmark focusing on challenging audio-visual reasoning scenarios with human expert annotations.
-ideo-SALMONN-o consistently outperforms the strong LLaVA-One Vision visual baseline on VideOMME, NExT- QA and RivaBench, with 3-8% absolute accuracy im- provements. The pDPO training achieved 6-8% improve- ments on RivaBench over the SFT model. Moreover, Video-SALMOONNo11 is also the first open-source model that showed zero-shot synthetic video detection ability.
2.3. Benchmarks for Audio-visual LLMs
The fast-paced development of multimodal LLMs has boosted the creation of more challenging video understand- ing benchmarks. Benchmark focus evolves from video de- scription and perception abilities (Li et al., 2021; Alamri et al., 2019; Chen et al., 2023a; Li et al., 2022; Chen et al., 2023b; Ning et al., 2023; Mangalam et al., 2023; Yun et al., 2021; Sun et al., 2024a), to video reasoning abilities such as inference about temporal and causal relations (Xiao et al.,
2. Related Work
2.1. CoT Reasoning
CoT reasoning is one of the remarkable abilities of LLMs when solving difficult and complex problems. Earlier in- vestigations employed prompt tuning and various search algorithms such as the Monte-Carlo tree search during infer- ence time (Hao et al., 2023; Snell et al., 2024; Feng et al., 2024; Yao et al., 2023; Goyal et al., 2024). Later on, training stage approaches using reinforcement learning (RL) were developed to further and more radically boost the reasoning capabilities of LLMs. RL has also been used in multimodal LLMs but not from the perspective of improving reasoning (Yu et al., 2024b: Zhang et al., 2024d). PRMs which esti- mate the value function of each reasoning step have emerged
2021; Li et al., 2024b; 2023a; Fu et al., 2024; Liu et al., 2024a; Fang et al., 2024). In particular, NEXT-QA (Xiao et al., 2021) focuses on causal relation reasoning such as why a certain action is performed, and Video-MME (Fu et al., 2024) contains questions that require the combination of both audio and visual information to perform reasoning. Our proposed RivaBench has more challenging questions that require longer thinking steps, broader world knowledge and a tighter combination of audio-visual information.
2

---

video-SALMONN-ol: Reasoning- Eeahninne Visual Large Language Model
Response
3.2. Reasoning-intensive SFT Data
LoRA
LLM
We empirically discovered that video understanding models (Cheng al., 2024; Zhang al., 2024e; Lin et al., 2024) generally lose the ability perform step-by-step reasoning when a video is given, and always directly generate the final answer. To re-obtain the reasoning ability during the SFT stage, we create a set of more challenging question- answering pairs based on the same training set videos using proprietary LLMs, and the pipeline is shown in Fig. 2.
Interleaved Synchronization
Audio Encoder
Visual Encoder
Embedding
Question: Ii the item is trequently handled, which one should be choose? Answer: The glaas ocle
Reasoning Generation
Step Watch the video and understand the pros... Step 2: Analyse the situation the item is frequentyy
11
Text Prompt
Question Generation
Quality Check
model structure. The input video is processed by the visual and audio branches, generating encod- ings from the visual and audio frame sequences respectively. Two encoding streams combined in an interleaved fashion syn- chronize across time before sending to L.M.
Figure 1.
Not Qualified
Qualified
GPT4o
SFT Data
Figure 2. Acquisition pipeline of reasoning-intentn inttiini-gineeivveeeee SFT data. The question, answer and reasoning paths are generated by Gemini- 1.5-pro taking the video with paired audio as inputs. GPT4o is employed for quality checks to ensure the QA-pair and the reasoning steps are valid and require logical thinking.
3. video-SALMONN-ol
3.1. Model Structure
We adopt the same model structure as video-SALMONN
2, as shown in Fig. 1. As 2024a), the model is built based on a pre-trained visual LLM by adding the audio encoder branch. The input video and audio streams are processed separately by the audio encoder and visual encoder and are then separately mapped the di- mension of the LLM input via individual modality aligners. To combine the audio and visual encodings, the interleaved synchronization module is employed as illustrated in Fig. 1. The groups of encodings per visual frame are equally spaced across time, and the audio encodings corresponding to the time between two visual frames i and la are inserted between the two groups of visual encodings. The process is summarized in Eqn. (1):
2 (Tang et al.,
For each video with paired audio, we use Geminii-155ppp to generate a question- annseer pair with the reasoning steps. Then, to avoid bias in Gemini models and ensure the quality of the questions and reasoning steps, a quality check stage is employed using GPT-4o.. Questions with poor quality will be discarded and a new question-answer pair will be generated again. In addition to the newly created question, we augment the original training set by generating reason- ing paths with Gemini- 1.5-pro and checking by GPT-4o following the pipeline to avoid network learning two dis- tinct mechanisms for reasoning and direct answer. This turned out to be important to yield competitive reasoning performance from SFT in our empirical study.
HaV Concat(.... ... HY. H. (1:t2) H......
HY
4. Training to Enhance Reasoning Abilities
where НA E Rmxx and HY E Rxd represent groups of audio and visual encodings, and m and n are the number of encodings in each group.
4.1. Preliminary
The reasoning process refers to the LLM generating the Q - (81, (8,,88,...,8 -> A sequence, where Q is the ques- tion, A is the answer and sj are reasoning steps that logically connect the question Q to the final answer A. By treating this as Markov decision process (MDP) and the LLM as the policy model, PRM is to provide feedback for each step Se that guides the LLM in making accurate reasoning by optimising the policy to maximise the reward.
multi-stage SFT pipeline with the cross-entropy loss on reference response is adopted to train video-SALMONN-.. before optimizing the reasoning process with RL. Start- ing from the pre-trained visual model, the audio aligner is trained from scratch keeping other parts of the model frozen. Then, using paired audio-video data, the modality aligners and the low-rank adaptation (LoRA) module (Hu et al., 2022) are trained with other parts frozen.
Following Wang et al. (2024b), the PRM is to estimate the
3

---

video-SALMONN-oll Reasoning-Ennaance Enhanced Audio-Visual Large Language Model
Perturbation
Choose Top
d
dg
d
d
dg
110-0 1++
S5
If the item is frequently handled, which should he choose? Answer The glass
Question:
$2
Aliernative Step
Contrastive Step Selection
Pairwise Rollouts
$3.1
$4,1
Answer: The glass
$2
$3,2
$4,2
Rollouts
Anawer: Glass
Ps P= 0.67
$3.3
$4.3
Answer: Plaatic
$/
Rollouts for the Alternative Step
Pg 0.33
Figure Illustration of the contrastive step selection (top) and pairwise rollout (bottom) to construct per-step expected correctness score for pDPO. Contrastive step selection: Top 2 steps, 82 and are selected in this example, and for 82, alternative step, s2, is sampled to form the preference pair. Pairwise rollout: Three rollouts are shown for each step and #2 and s/2 are step pairs with the same prefix solution. The answer correctness is checked using GPT-4o by comparing against the reference answer.
expected answer correctness, Pag. of prefix solution {$ The expected correctness score be approximated with Monte Carlo sampling of multiple paths from the prefix solution to an answer A, as shown in Eqn (2).
paths, pDPO models the preference for a specific reasoning step given the same prefix solution. Specifically, the reward function for each step of interest can be written
r(sx) = Blgg Ta(sk| HAV) Tee (sr HAV) + B log Z(8ck, HAV (4)
Psp Σ N
Σ. (An, Are))
(2)
where Ta. Teef 3 and Z(-) the L.M policy, reference policy, a parameter controlling the deviation from Tet and the partition function as in Rafailov et al. (2024) respectively. Z(<ck. H =., (se each step, an alternative step sy is generated, and pairwise rollout is performed for both steps as shown in Fig. 3. The probability of sy being better than sy. is then defined using the Bradley-Tere Terry model as
where A is the reference answer and An one sampled answer. The sampled path {sk+1,ns S Sk+2.n1 that leads A, is referred to as rollout. The PRM training loss is then shown as
(x))),
f (sx))) For
LPRM Exe PoL logrs. T. - Poa log(1 - rea) (3)
where E [0, is the PRM prediction which can be derived from the LLM output at the last token of each step with fully connected layer with sigmoid function o()).
p(sk > $' = (r((s)) r(()))
(5)
Then, the pDPO loss can be written as:
4.2. Process DPO
L = -E -B(au au log p(s2 > s(.) (1 az) st) (6)
As pointed out by Zhang et al. (2024a), predicting an abso- capabili-
lute score fails to exploit the ties of LLMs well as influenced by ambiguities in score standards. Both problems are more severe in audio-visual LLMs. Therefore, we propose PDPO for Mileo--SLLMON- o1, which is a pairwise preference modelling approach by training the model to select the better reasoning path rather than giving absolute scores to the paths. Different from the pairwise preference reward model (PPRM) in (Zhang et al., 2024a) that leverages the partial ordering of entire reasoning
where an 1(p=, P.;). Alternatively, os = ( P=:)/M) can be used as soft labels for DPO to accommodate the estimation noise introduced by the limited number of rollouts in Ps., where μ is the calibration hyper-pann parameter determining how much we believe the process annotations. As a result, pDPO retains the advantages of PPRM while offering finer modelling granularity at each step. In practice, pDPO is integrated with PPRM to construct complete rea- soning paths, enhancing overall performance. While PPRM

---

video-SALMONN-ol: Reasoning- Ennhhncedd Audio-Visual Large Language Model
enables full-solution-level preference training, ensuring the generation of entire solutions, pDPO complements it by providing fine-grained, step-level preference guidance.
Table I. RivaBench basic statistics. The duration is given by mean + standard deviation. The SynthDee split contains 100 synthetic videos and 100 real videos that human annotators search to have similar content as synthetic videos. MCQ stands for multiple- choice questions. Video sources are from YouTube.
4.3. Contrastive Step Selection
While rollouts allow automatic process annotation, the com- putational cost can be high when the numbers of rollouts and steps grow. However, in pDPO, certain steps are more error-prone and hence more valuable to be optimized than others. For general video understanding, by examining a held-out validation set for reasoning paths with wrong an- swers, we found that over 70% of the reasoning errors occur at steps where the model misinterprets or hallucinates the video content. Therefore, we choose to particularly focus the pDPO on optimizing those steps.
Attribute
Academic
StandUp
SynthDec
Num. of QA
1,912 47.2+t 66.1
2,128
200
Duration (8)
43.2 15.1
8.14:32
Format
5-way MCQ
5-way MCQ
Yes/No
et al., 2024), the StandUp partition of RivaBench explores from an audio-visual reasoning perspective. Specifically, instead of prompting the model to list all funny elements in the video, we particularly focus on understanding why certain punchline interesting and task the human an- notators to set questions that require reasoning about the comedian's 's gestures, facial expression and speech content. Human annotators provide questions, answers and explana- tions (with automatically generated confusing choices), as shown in Figs. 7 and 8 in Appendix B. Subtitles exist in around 50% of the videos in the StandUp partition but not in the Academic partition. However, due to the low frame sampling rate and hence subtitles are mostly incomplete, the model has to leverage both audio and visual information. This is reflected by the fact that GPT-4o (visual only) is worse than Gemini-1.5 (audio-visual) in Table 2.
To locate those steps, we quantify the susceptibility of each reasoning step to the input video by applying a tiny perturba- tion to the input video and measuring the length-normalized per-token KL divergence. Specifically, as shown in the top part of Fig. 3, for each step 8% we compute the length- normalized KL-divergence by
dss.
1
Σ ME
DKL (P(yilvsci
[sk
where yi are wordpiece tokens and (+) computes the KL-divergence between the output distributions with the original inputs HAV and perturbed inputs HAY. A higher de, indicates that the reasoning step 8y is more susceptible to small input change, and this high susceptibility is likely to yield more diverged subsequent steps. We select the top T steps with the highest d., to perform pairwise rollout. While this selection biases pDPO training towards video- dependent errors, the other text- bext-baee logic errors can be accommodated by PPRM with entire reasoning paths.
This paper proposes the SynthDee partition for synthetic video detection, which has great potential since video gener- ation models are becoming increasingly powerful. This task requires LLM to classify whether given video clip is real synthetic by finding clues in the video such as motions violating physics rules or objects being distorted. Videos are generated using the Hunyuan-large model (Sun et al., 2024c) (see examples in Figs. 11 and 12) This is a challenging task that requires both logical reasoning and accurate perception of video content. The SynthDec partition can also serve as the performance indicator for reward models used to train video generators in the future.
5. Audio-visual Reasoning Benchmark
The RivaBench is proposed to extend the scope of complex video understanding with three new maasoning-intensinn ap- plication scenarios, including academie presentation (Aca- demic), stand-up comedy (StandUp) and synthetic video detection (SynthDec). The statistics of videos for each sce- nario partition are shown in Table 1.
6. Experimental Setup
6.1. Model and Training Specifications
The Academic partition is based on the MBAV (Chen et al., 2024) test set containing recordings of conference or lec- ture presentations spanning five different domains. Human experts with mathematical, engineering and medical back- grounds are recruited to provide questions, answers and detailed explanations based on the video clips. Example annotations are shown in Figs. 9 and 10 in Appendix C.
Malo--SALMONN-O is built based on the SigLIP (Zhai et al., 2023) visual encoder and Qwen 2 with 7B parameters backbone LLM. Two linear layers with GELU activation function are used (Hendrycks & Gimpel, 2016) as the visual aligner. The model processes videos at 2-frame--er--seoond rate with a maximum of 60 frames.
The Thisser-Large-a v3 encoder (Radford et al., 2023) is used as the audio encoder, and the window-level Q-Former (Tang et al., 2024c) with a window length of 0.2 seconds
While humour in videos has been explored from a descrip- tive perspective (Hyun et al., 2024; Liu et al., 2024b; Xie
5

---

video-SALMONN-ol: Ennhanncn Audio-Visual Large Language Model
Overall SFT data
Rensoning SFT data
selected, and an extra 100k pairs of step-level partial solu- tion pairs from these complete solutions used for pDPO. Greedy decoding is used during inference and the effort in generating data using pDPO only affects training time and has no impact to the latency of the model during inference.
Number of Samples
8x10
1x10-
8x10'
6x10-
6x10
4x10-
4x10"
2x10
2x10
Besides RivaBench, SALe-NAL I is also evaluated on Video-MME (Fu et al., 2024) and NExT-QA (Xiao et al., 2021) benchmarks with challenging reasoning questions where the former is an audio visual task and the latter fo- cuses visual information only. For consistency, paired audios are also provided for NExT-QA videos if they exist. Note that the synthetic video detection task is never seen in model training. and hence is a zero-shot emergent ability.
15
15
number steps
number steps
Figure 4. Distributions of the numbers of reasoning steps in SFT data. Left: Distribution of the entire SFT data. Right: Distribution on the reasoning- intensive subset of SFT data. Due to the diffi- culty of the reasoning-intensiv subset, more reasoning steps are required in general for samples in this set.
7. Results
used as the audio aligner, producing 150 audio tokens for every 30 seconds. We set LoRA hyper-parameters r = 64 and a = 256 for the backbone LLM for both SFT and PDPO. training, the visual encoder and aligner, audio encoder, and LLM remain frozen. SFT is performed on 16x GPUs for 48 hours and pDPO is trained with 8xA100 GPUs for 24 hours. Prompts used for reasoning are shown in Appendix E. The code, SFT data, pDPO data and model checkpoints will be released.
7.1. Main Results
The main results on VideoMME, NExT-QA and the Riv- aBench are shown in Table 2. No subtitles are given to any of the models under test for VideOMME. As performance references, we include GPT-4o (checkpoint at 2024-08-06) and Gemini-1.5-pro, with their results on VideoMME as reported in Fu et al. (2024). When testing GPT-4o with videos, each video is split into images at frame rate of 2 fps with maximum of 30 frames due to token limita- tion, and the sequence of images is sent as the input. For open-source models, we compare video- Aado--AALMONN-O to LLaVA-OneVisioo (Li et al., 2024a) (same visual encoder and LLM backbone), together with Aado--SALMONN (Sun et al., 2024b) and Video-LLaMA (Cheng et al., 2024) as the two most recent audio-visual LLMs.
6.2. Data
Following Tang et al. (2024a), the audio modality alignment stage employs LibriSpeech-960h (Panayotov et al., 2015) ASR data and AudioCaps (Kim et al., 2019) audio caption data to train the audio aligner. During the audio-visual SFT stage, 13k videos with rich audio information are selected with high-quality audio-vic vissal captions. Around 150k nor- mal -anwwer (QA) pairs are directly generated using GPT-4o by providing detailed audio-visual captions, and an additional subset of 30k reasoning intensive SFT QA pairs are generated with the proposed data generation pipeline. Each QA, regardless of the difficulty, is associated with reasoning steps, and the distributions of the numbers of rea- soning steps for the QA pairs used for SFT are shown in Fig. 4. Both captions and QA pairs are used for SFT.
Proprietary LLM performance on RivaBench: For the two proprietary LLMs, GPT-4o underperforms Gemini- 1.5- pro on StandUp and Academic test sets due to the lack of audio information. This indicates that RivaBench pro- vides challenging questions that require more audio-visual joint understanding compared to VideoMME. On the Syn- thDec set, since only the visual part is synthesized, GPT-40 demonstrated stronger ability. Moreover, by performing reasoning with GPT-4o and Gemini-1 larger improve- ments are found on StandUp and Academic test sets than VideoMME and NEXT-QA, indicating the necessity of rea- soning on RivaBench.
-assoning-intensive
The reasoning intensive subset is used to collect the data for pDPO training by sampling 10 paths for each QA. The QA pairs where the SFT model generates incorrect solutions are retained to perform rollouts and others that only contain cor- rect solutions are discarded. For complete solutions, instead of directly comparing the paths (Zhang et al., 2024a), we compare each pair of solutions against the reference answer using GPT-4o and choose the one closer to the reference as the preferred solution. For intermediate steps, we choose the top3 steps based on contrastive step selection, and 6 rollouts are performed for each chosen step. As a result, ~ 100k pairs of complete solutions from 5k video clips are
Open-source LLM performance comparison: Audio- visual SFT olLo--ALLMNNN-ol already yields better performance than LLaVA-Onee Vinioonn VideoMME due to the ability to comprehend speech and audio information, whereas no obvious improvements are found on the other benchmarks. The main improvements on other benchmarks come from pDPO, which achieved 4. 1.1% %, 8.1% and 5.8% absolute accuracy improvements on NEXT-QA, StandUp and Academic test sets respectively compared to the SFT
6

---

video-SALMONN-ol: Reasoning- CeassnniggEEnhanced Audio-Visual Large Language Model
Table Main results of VideoSAALMOONN compared against other visual (V) and audio-visual (A+V) I.L.ss SFT refers the model after SFT with reasoning data and pDPO refers to the model obtained after training with pDPO based on the same SFT model. Fl-score (Precision/Recall) is reported SynthDec and accuracy is reported others. Results with are directly taken from corresponding papers. video-SALMONN-o) performs reasoning during inference and other open-source models give answers directly.
Model
Modality
VideoMME NEXT-QA
RivaBench
StandUp
Academic
SynthDec (P/R)
Proprietary models
Gemini-1.5-pro (Team et al., 2024)
A+V
75.0%
79.2%
75.8%
67.1% 1%
23.6% (55%/15%)
Gemini-1.55 L.Spprotreasoning
A+V
75.1%
79.5%
81.8%
69.5%
40.0% (49%/34%)
GPT-4o (OpenAl Team, 2024)
V
71.9%
81.7%
63.3%
60.0%
34. (901%((0%))))
GPT-to+reasoning
V
72.1%
81.9%
69.6%
61.0%
(5.8%(33%/117%)
Open-source baselines
I.aVA-OneVision (Li et al., 2024a)
V
58.2%
79.4%
67.2%
45.8%
(97%/%%)))
video-SALMONN (Sun et al., 2024b)
A+V
43.3%
49.2%
47.8%
33.6%
(.0%(100%%0%%)
Video-LLaMA 2.1 (Cheng et al., 2024)
A+V
54.9%f
75.6%
53.7%
34.3%
0.9%(99%%)%%)
Video-SALMONNN-a (ours, SFT) video-SALMONNNol (ours, pDPO)
A+V
62.9%
78.2%
68.6%
42.5%
5.8%(97%%/5%)
A+V
65.6%
82.3%
76.7%
48.3%
(7789()77))139)
Table Effect of different parts of the audio-visual SFT data on VideoMME. Academic and StandUp test sets. Underscore for second-bos results. "w/o reasoning-intee intensive part" means removing the bessoning-intensiv SFT data, and "w/o any reasoning" always directly outputting answers during SFT. "Reasoning-intensive part only" always performs reasoning for QA.
Training Data
Inference Reasoning
VideoMME
NEXT-QA
Academic
StandUp
Full SFT data
X
63.7%
80.7%
45.2%
72.3%
Full SFT data
62.9%
78.2%
42.5%
68.6%
w/o any reasoning
X
63.2%
81.0%
44.1%
71.1%
w/o reasoning-intensivee intensive part
X
62.7%
78.9%
44.7%
71.5%
w/o reasoning-intensii part Reasoning-intensive part only
61.6%
76.6%
42.3%
67.5%
58.8%
75.2%
40.1%
63.5%
Full SFT data + pDPO
65.6%
82.3%
48.3%
76.7%
model. Larger improvements are found on the RivaBench with 6-8% absolute accuracy improvements obtained com- pared to LLaVA- VaaaVA-OOeViiiion and video-SAle SALMONN-OL even performs better on the StandUp test set than Gemini-1.5-pp proppro without reasoning. Besides, compared to other audio-visual I.Mss, video-SAL.MONN-0 exhibits better interpretability of the model output, and the cause of mistakes can be lo- cated by analyzing the reasoning process. We also provide the comparison between Mideo-SALMOON-ol and Gemini- 1.5-pro on VideoHallucer benchmark (Wang et al., 2024c)
to detect most of the time.
In addition, two qualitative examples are shown in Figs. and 17 Appendix G, where LlaVA-One Vison (and also other audio-visual models) are unable to provide the reasoning steps and the final answer is completely biased "Real". On the other hand, vidco-SALMONN-.. can look for distortions in the video part of its reasoning process, leading to the correct identification of synthesized videos.
7.2. Effect of SFT Data
Appendix for the hallucination aspect.
The audio-visual SFT data is crucial for Video-SALMONN. ol to gain the initial audio-visual reasoning ability, and the effect of different data partitions is shown in Table 3.
Zero-shot synthetic video detection: -ALLO-- ol achieves zero-shot synthetic video detection ability while other open-source models output "real" all the time, which also benefit from better explanation with examples of anomalies in synthesized videos in the prompt. However, even for the videos where the motions obviously violate physics rules, current tfat--o-thhe-a video L.LMs still fail
Direct answer outperforms reasoning after SFT: Directly outputting a short answer or an option has been the domi- nating output mode for audio-visual L.MSs general video understanding, major difference to math questions. Com-

---

videe-SALMONN-ol: Reasoning- Reasoning-EEnhaneed Visual Large Language Model
Table Effect of different reward modelling methods on VidcOMME, NEXT-QA, the StandUp and Academic split of RivaBench. Major@ @ 20 and RM@20 are evaluated following Zhang et al. (2024a), where Major@2) @0 refers to the accuracy under majority voting with 20 sampled paths, and RM @ 20 is the best-of-n with 20 samples. Samples are all generated from the model after SFT. pDPO with full paths only uses preference pairs of complete reasoning paths.
Training Configuration
Inference
VideoMME
NEXT-QA
StandUp
Academic
SFT SFT
1-best
62.9%
78.2%
68.6%
42.5%
Major@ 20
63.5%
81.5%
73.5%
45.3%
SET ORM
RM @ 20
62.7%
78.5%
69.0%
42.6%
SFT PRM
RM@1 20
63.5%
79.3%
72.1%
43.9%
SFT + pDPO
1-best
65.6%
82.3%
76.7%
48.3%
paring row 2 to row in Table 3, when using all the SFT data including the reasoning- inntenive part, the model after SFT is still better at directly generating the answer than performing reasoning. This is due to the exposure bias in teacher forcing which has a much higher impact on the reasoning paths as they are much longer sequences. By learning its own samples, pDPO mitigates this exposure bias and achieves consistently better performance than the SFT model. Next, comparing row to row 3 and row 4 in Table 3, when directly outputting the answer during infer- ence, incorporating reasoning steps in SFT does not always yield an improvement on videoMME and NEXT-QA, despite being slightly helpful on RivaBench.
Full Paths
Full Paths TTop Steps
Full Paths All Steps
67-
77-
76.7
50-
66-
65.6
65.7
76-
49-
Accurccy
75.1
40.3
48.3
64-
74-
47-
46.5
73
46
VideoMME
StandLip
Academic
Figure 5. Comparison between different top T steps selected for pDPO. Pairs of full solution paths are always used in addition to pairs of intermediate steps.
The reasoning-intensivv part is important: When exclud- ing the ceasoning-intensive part, there is a clear degrada- tion in model performance with reasoning during inference, showing the importance of this part of data to enable a bet- ter reasoning performance. However, when only using the reasoning-intensiin part for SFT, the model struggles to acquire the fundamental audio-visual perception abilities, yielding sub-optimal performance.
Moreover, majority voting among the 20 samples is used as baseline which is consistently marginally better than the 1-best solution across all test sets.
While ORM showed mixed results compared to the 1-best solution from the SFT model, PRM showed consistent but marginal improvements and is on par with majority voting. The training loss of PRM and ORM only dropped about 5%, which reflects the difficulty of learning general video understanding tasks. Last, comparing the models above against the pDPO model, the use of pairwise preference models is much more effective compared to pre- dicting the raw score, showing the difficulty of direct raw score modelling in general video QAs Qualitative examples comparing answers between SFT and pDPO are provided in Figs. 13 to 15 in Appendix F.
scores for
7.3. Effect of pDPO Training
We then analyse different reward modelling techniques for the model performance in Table 4. In addition to pairwise preference models, we include ORM and PRM proposed in Lightman et al. (2023) as follows:
ORM: A projection layer is added to LLM output states and projects the last output state to a scalar, which then passed through a sigmoid activation function to predict if the final answer is correct, and otherwise.
Effect of Contrastive Step Selection. To analyze the effect of the number of steps selected for pairwise training, we conducted experiments without intermediate pairs of steps and with all intermediate pairs, in addition to using the top three steps from the contrastive step selection. The com- parisons are given in Fig. 5. Using intermediate steps in pDPO achieved further consistently improves model perfor- mance compared to only using the full solutions, especially on questions that require frequent reference to the video or audio information at intermediate reasoning steps. A case study qualitatively showing the effect of contrastive step selection is included in Appendix H.
PRM: A projection layer is added to LLM output states and projects the state at the end of each step to scalar with sigmoid to predict [P,,, > 0). The score of each solution is the lowest score among all steps (Wang et al., 2024b).
Both ORM and PRM are initialized with Midco-SALMONE ol after the SFT stage. Best-of-n used for ORM and PRM where 20 sampled solutions are generated from the SFT model and the top one with the highest score is selected.
8

---

video-SALMONN-ol:
Reasoning- Ennainnegg- Vistio-isuall Large Language Model
8. Conclusions
arXiv:2304.083453 2023a.
We propose Video-SALMONN-ol, the first open-source audio-visual LLM with enhanced reasoning abilities. video- SALMONN-ol is the first explore reasoning process optimization for general video understanding and proposes the pDPO method with efficient contrastive step selec- tion algorithm. To further evaluate the reasoning abilities of audio-visual LLMs, the RivaBench is introduced with innovative and challenging tasks and over 4000 high-quality human expert annotations. video-SAI SALNNNN-O consis- tently outperforms the strong LLaVA- Vne Vision baseline with 3-8% absolute accuracy improvements, pDPO training consistently outperformed the SFT model. Moreover, video- SALMONN-ol showed zero-shot synthetic video detection abilities as a result of the enhanced reasoning abilities.
Chen, S., Li, H., Wang, Q., Zhao, Z., Sun, M., Zhu, X., and Liu, J. Vast: A aision-adio-ssubiiileeexx omni-modality foundation model and dataset. 2023b.
Chen, Z., Liu, H.,, Yu, W., Sun, G., Liu, H., Wu, J., Zhang, C., Wang, Y., and Wang, Y. M A multimodal, multi- genre, and multipurpose oudio-visual academic lecture dataset. In Proc. ACL., 2024.
Cheng, Z., Leng, S., Zhang, H., Xin. Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., and Bing, L. VideoL 2: Advancing spatial-temporal modeling and audio understanding in Video-LLMs. arXiv preprint arXi::2406.07476 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.
Impact Statement
By enhancing reasoning abilities in general video under- standing, video-SALMONN-o. provides more transpar- ent and interpretable interface that is compatible with gen- eral videos to access and explain model responses and be- haviours. This is indispensable to ensure the reliability of LLMs when applied to different video understanding scenar- ios and will be largely beneficial for pinpointing the specific causes or errors when the model generates dubious or toxic contents, thus enhancing AI safety.
DeepSeek Team. Deepseek-rl lite-errview is now live: unleashing supercharged reasoning power, 2024.
Du, Y., Ma, Z., Yang, Y., Deng, K., Chen, X., Yang, B., Xiang. Y., Liu, M., and Qin, B. CoT-ST: Enhancing LLM-based speech translation with multimodal chain-of- thought. arXiv:2409. 19510, 2024.
The approaches in this paper do not give rise to any addi- tional potential biases beyond the ones directly inherited from the pre-trained model checkpoints used. The audio en- coder and visual encoder might work worse for people from particular demographics. The framework also inherits biases from all the LLMS used in this paper. To mitigate potential biases, we clearly describe the nature of each dataset and provide clear and adequate references to all the resources we used for -o.
Du, Y., Liu, Z., Li, Y., Zhao, W. X., Huo, Y., Wang, B., Chen, W., Liu, Z., Wang, Z., and Wen, J.-R. Virgo: A preliminary exploration on reproducing ol-like milm. arXiv:2501.11900 2025.
Fang. X., Mao, K., Duan, H., Zhao, X., Li, Y., Lin, D., and Chen, K. MMBench- Video: A forg-form multi- shot benchmark for holistic video understanding. arXiv preprint arXiv:2406. 14515, 2024.
Feng, X., Wan, Z.,, Wen, M., Wen, Y., Zhang, W., and Wang, J. Alphazero-like tree-search can guide large language model decoding and training. In Proc. ICML, 2024.
The ability of -AMMON-- to understand speech in videos could lead to potential technology abuses like surveillance and eavesdropping. To counter this, we've con- sulted with legal experts establish clear usage guidelines, reducing risks and addressing concerns, highlighting our
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang.
Z., Zhou, C., Shen, Y.. Zhang, M., et al. Video-MME:
dedication to responsible research sharing.
The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. arXiv preprint arXiv: 200.22005. 2024.
References
Alamri, H., Cartillier, V., Das, A., Wang. J., Cherian, A., Essa, I., Batra, D., Marks, T. K., Hori, C., and Anderson, P. Audio visual scene-aware dialog. In Proc. CVPR, 2019.
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens. In Proc. ICLR, 2024.
Guo, J., Zheng, T.. Bai, Y., Li, B., Wang. Y., Zhu, K., Li, Y., Neubig, G., Chen, W., and Yue, X. MAmmoTH-VL. Eliciting multimodal reasoning with instruction tuning at scale. axXiv:24/2.05231 2024.
Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., and Liu, J. VALOR: iision-audio-languaa omni- perception pretraining model and dataset. arXiv preprint

---

video-SALMONN-ol: Reasoning- Ceassnnig-EEnhaneed Audio-Visual Large Language Model
Hao, S., Gu, Y., Ma, H., Hong. J., Wang, Z., Wang, D. Z.,, and Hu, Z. Reasoning with language model is planning with world model. In Proc. EMNLP, 2023.
Liu, Y., Li, S., Liu, Y., Wang, Y., Ren, S., Li, L.., Chen, S., Sun, X., and Hou, L.. Tempcompass: Do video LLMs re- ally understand videos? arXiv preprint arX1v:2403.00476. 2024a.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08444 08 2016.
Liu, Z.-s., Courant, R., and Kalogeiton, V. FunnyNet-W Multimodal learning of funny moments in videos in the wild. arXiv:2401. 24t.:2400004200,, 2024b.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, 2022.
Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision. wrXiv:2406.06592 2024.
Hyun, L., Sung-Bin, K., Han, S., Yu, Y., and Oh, T.-H. SMILE: Multimodal dataset for understanding laughter in video with language models. In Duh, K. Gomez, H., and Bethard, S. (cds.), Proc. NAACL Findings, 2024.
Mangalam, K., Akshulakov,,, R., and Malik, J. Egoschema: diagnostic benchmark for very long-form video language understanding. Proc. NeurIPS, 2023.
Kim, C. D., Kim, B., Lee, H., and Kim, G. AudioCaps: Gen- erating captions for audios in the wild. In Proc. NAACL- HLT, 2019.
Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L., Chen, D., and Yuan, L.. Video-bench: A comprehensive bench- mark and toolkit for evaluating video-based large lan- guage models. arXiv preprint arXiv:2311.1111 2023.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang. K., Zhang, P., Li, Y., Liu, Z., and Li, C. LLaVA- OneVision: Easy visual task transfer. arXiv:2408.0332 2024a.
OpenAl. Learning to reason with large language models, 2024.
Li, G.. Wei, Y., Tian, Y., Xu, C., Wen, J.-R., and Hu, D. Learning to answer questions in dynamic audio-visual scenarios, In Proc. CVPR, 2022.
OpenAl Team. Gpt-4o system card. arXiv:2410.21276. 2024.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: An ASR corpus based on public domain audio books. In Proc. ICASSP, 2015.
Li, K., Wang. Y. He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. Mvbench: A compre- hensive multi-mode modal video understanding benchmark. In Proc. CVPR, 2024b.
Qwen Team. To see the world with wisdom, 2024a.
Li, L., Lei, J., Gan, Z., Yu, L., Chen, Y.-C., Pillai, R., Cheng, Y., Zhou, L., Wang. X. E., Wang, W. Y., et al. Value: A multi-task benchmark for video-and-language under- standing evaluation. arXiv preprint 2021.
Qwen Team. QwQ: Reflect deeply on the boundaries the unknown, 2024b.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust Speech Recognition via Large-scale Weak Supervision. In Proc. ICMI., 2023.
Li, S., Li, L.,, Ren, S., Liu, Y., Liu, Y., Gao, R., Sun, X., and Hou, L. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. arXiv preprint arXiv:2311.17-00 2023a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct Preference Optimization: Your language model is secretly reward model. In Proc. NeurIPS, 2024.
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., and Chen, W. Making language models better reasoners with step-aware verifier. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proc. ACL, 2023b.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv:2408.033- 2024.
Sun, G., Manakul, P., Liusie, A., Pipatanakul, K., Zhang, C., Woodland, P., and Gales, M. CrossCheckGPT: Universal hallucination ranking for multimodal foundation models. arXiv preprint 24Xi 2024a.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker,
B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv:2305.20050 2023.
Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L., MA, Z., Wang. Y., and Zhang, C. video-SALMONN: audio-visual large language models. In Proc. ICMI., 2024b.
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. Video-LLaVA: Learning united visual representation by alignment before projection. In Proc. CVPR, 2024.
10

---

video-SALMONN-ol: Reasoning- Seasoning-Ennhaneed Audio-Visual Large Language Model
Sun, X., Chen, Y., Huang, Y., et al. Hunguan-Large: An open-source model with 52 billion activated parame- ters by tencent. arXiv preprint arXiv:2411.02266 2024c.
Xie, B., Zhang, S., Zhou, Z., Li, B., Zhang, Y., Hessel, J., Yang. J., and Liu, Z. FunQA: Towards surprising video comprehension. arXiv:2306. 1et-2999...499, 2024.
Sun, Z., Yu, L., Shen, Y., Liu, W., Yang, Y., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. In Proc. NeurlPS, 2024d.
Xu, G., Jin, P., Li, H., Song, Y., Sun, L., and Yuan, L. ILaVA-CoT; Let vision language models reason step-by- 2024.
step. arXiv:2411
Tang, C., Li, Y., Yang, Y., Zhuang. J., Sun, G., Li, W., Ma, Z., and Zhang, C. Enhancing multimodal LLM for detailed and accurate video captioning using multi-round preference optimization. arXiv:2410.0660 2024a.
Yang, A., Zhang. B., Hui, B.. Gao, B., Yu,
Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, x., and Zhang, Z. Qwen2.5-Matt technical report:
Toward mathematical expert model via self-improvement, arXiv:2409. 12122, 2024.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L.., MA, Z., and Zhang, C. SALMONN: Towards generic hearing abilities for large language models. Proc. ICLR, 2024b.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L.,, Cao, Y., and Narasimhan, K. Tree of Thoughts: Deliberate problem solving with large language models. In Proc. NeurIPS, 2023.
Tang. C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. Extending large language models for speech and audio captioning. In Proc. ICASSP, 2024c.
Ying, H., Zhang, S., Li, L., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J., Liu, K., Wang, Z., Wang, Y., Wu, Z., Li, S., Zhou, F,, Liu, H., Zhang. S., Zhang, W., Yan, H., Qiu, X., Wang, J., Chen, K., and Lin, D. InternI-M-Math Open math large language models toward verifiable reasoning. arXiv:2402.0633 2024.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J.. et al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:231..//000 2024.
Uesato, J., Kushman, N., Kumar, R., Song, F,, Siegel, N., Wang, L.., Creswell, A., Irving, G., and Higgins, Solv- ing math word problems with process- and outcome- based feedback. arXiv:2211.142755 2022.
Yu, F., Gao, A., and Wang. B. OVM, outeome-supprrised value models for planning mathematical reasoning. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proc. NAACL Findings, 2024a.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, 7., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, Qwen2-W VL: Enhancing vision-language model's per- ception of the world at any resolution. 22Xv:2409.121911, 2024a.
Yu, T., Zhang, H., Li, Q., Xu, Q., Yao, Y., Chen, D., Lu, X., Cui, G., Dang, Y., He, T., Feng, X., Song, J., Zheng, B., Liu, Z., Chua, T.-S., and Sun, M. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 17220, 2024b.
Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., and Sun, M. Advancing LLM reason- ing generalists with preference trees. arXiv:2404.02075 2024.
Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-Shi aath-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proc.
ACL, 2024b.
Wang, Y. Wang, Y., Zhao, D., Xie, C., and Zheng, 7. Video- hallucer: Evaluating intrinsic and extrinsic hallucina- tions in large video-language models. arXiv:2406. 16338, 2024c.
Yun, H., Yu, Y., Yang, W., Lee, K., and Kim, G. Pano- AVQA: Grounded audio-visual question answering on 360deg videos. In Proc. ICCV, 2021.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, arXiv preprint 23X3v:2303.155 15343, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Proc. NeurIPS, 2022.
Zhang, D., Wu, J., Lei, J., Che, T., Li, J., Xie, T.
Huang, X., Zhang, S., Pavone, M., Li, Y., Ouyang, W., and Zhou, D. LLaMA-Berry Pairwise optimiza- tion for ol ol-like Olympiad-leve mathematical reasoning. arXiv:24 10.02884, 2024a.
Xiao, J., Shang, X., Yao, A., and Chua, T.-S. Next-QA: Next phase of droestionansseeree to explaining temporal actions. In Proc. CVPR, 2021.

---

video-SALMONN-ol: Reasoning- Enaainne Audio- Visual Large Language Model
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative verifiers: Reward modeling as next-token prediction. arXiv:2408. 15240, 2024b.
Zhang, R., Gui, L.., Sun, Z., Feng. Y., Xu, K., Zhang, Y.,
Fu, D., Li, C., Hauptmann, A., Bisk, Y., and Yang, Y. Di- rect Preference Optimization of Video Large Multimodal Models from Language Model Reward. arXiv preprint arXiv:2404.01258, 2024c.
Zhang, R., Gui, L.., Sun, Z., Feng, Y., Xu, K., Zhang, Y., Fu, D., Li, C., Hauptmann, A., Bisk, Y., and Yang. Y. Direct preference optimization of video large multimodal models from language model reward. arXiv:240.001258, 2024d.
Zhang, Y. Wu, J. Li, W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data. arXiv:2410.02713, 2024e.
Zhang, Y., Wu, S., Yang, Y., Shu, Xiao, J., Kong, C., and Sang, J. ol-Coder: An ol replication for coding. arXiv:2412.00155 2024f.
Zhao, Y., Yin, H., Zeng, B., Wang, H., Shi, T., Lyu, C., Wang, L., Luo, W., and Zhang, K. Marco-ol: To- wards open reasoning models for open-ended solutions. arXiv:2411.14405, 2024.
12

---

video-SALMONN-ol: Reasoning-Enhaance Enhanced Audio- Visual Large Language Model
A. Reasoning SFT Data Example
I try that same thing on plppaticpphonee
GLASS
Speech: 99% of smart phones either made of glass plastic, but which one's better...
Question: If the item frequently handled but almost dropping risk, which type of material should he choose?
Answer:
The glass one
Reasoning:
The question focuses on two aspects: (1) frequent handling, which implies higher risk scratches, and minimal dropping risks, suggesting...
Analyze the first test (scratch test):
3. Analyze the second test (dropping test):
Compare the two test under the given scenario:
Conclude that under the given scenario, the glass preferred
6. The final answer the glass one
Figure Example reasoning SFT
B. St Data Examples
Two examples of the StandUp part of RivaBench are shown in Fig. 7 and 8 respectively.
Speech and Thank you guys. Are guys trying to judging my attractive level? It's weird level. Frustrating, and I find that out from things people say to me like, my friend's mom, like, "Chelsea! I saw an ugly Chelsea on the street the other day!". Basically that translate to me like you ugly person on the street and you thought about me. [audience laughed]
Question: Why did the aucience laugh when comedian mentioned her friend mom saying she saw ugly person on the street?
Answer: is using self- deprecating humor to make fanny situation.
Explain: The audience laughs because the person uses self-deprecating humor make funny situation. By sharing the comment their friend's mom, saw ugly person on the street and thought of her, the person employing self-mockery Choices:
A. A self-deprecating way to make fanny situation
comedian is known having many ddentical-looking friends
C. The comedian's outfic is considered ugly
D. The comedian's 's friend's
mentioned
attracting strange people
B. The comedian's friend's a mam is known for having bad eye-sight
Figure 7. Example of StandUp part of the RivaBench.
13

---

video-SALMONN-ol: Reasoning-Enhancee Enhanced Audio- Visual Large Language Model
Speech and audio: had so many, um, culture shocks since moves here. For example foreigner really hate tipping. [silence] But as barfender I really hate foreigners, [audience laughed]
Question: Why did the audience laugh when she said bartender really hate forcigners?
Answer: She uses technique called "reversal" to make contrast
Explain: The audience laugha because comedian employs linguistic and comedic technique known as "reversall" or Vrole inversion. By stating Vas bartender really hate foreigners," the comedian subverts is a humorous and unexpected twist.
statement as foreigner really hate tipping, which
Choices:
A. The comedian was drinking alcohol on stage.
B. The comedian made funny face while saying
C. The comedian used a comedic technique called "reversal"
D. The audience agreed with her statement
B. The audience misundesttoed her accent
Figure 8. Example of StandUp part of the RivaBench.
C. Academic Data Examples
Two examples of the Academic part of RivaBench are shown in Fig. 9 and 10 respectively.
Speech and audio:... In which she's used a creative use of the twin puir kind of study design, and this sort of interesting because you can. Because you can, hecause of their twins, who are discordant for Thi and discordant for for for carly genetics, you control for carly life exposures. And what Brenda has found is that, indeed, Tbi is increased with with risk of dementia, about a 60 % increase exactly the. The same kind of effect size we've been finding, but it wasn't
of dementia. you control
Twin Study to Control for Genetics Early Life Risk Factors: Findings Also Consistent with Non-AD Dementia
Alzheimer's. s. seems like
Alzheimer's
Manry
disease dementia. And think need to understand more. What is that? Is it vascular is? Is it Cte that's now met with aging? We don't really know...
Tyees
Question: How does the twin study isolate TBI's impact on dementia risk?
Answer: By analyzing twins with inconsiatent timing of onaet of TBI and dementia.
Explain: The increased risk of Vall-cause dementia"" and Ahon--Aleeimer's s demential" mentioned in the slidea suggesta association between TBI an incressed riak dementia, particulatly non- Alzheimer's type dementia...By By analyzing inconsistent pairs, researchers can more ancurately assess the association between TBI and dementia, thereby providing scientific basis for prevention and intervention.
Choices:
with inconsistent timing B. By excluding early life expamures from the study.
By analyzing
of TBI dementia.
C. By abserving the progression of dementia in twins without TBI.
D. By ensuring differences in dementia risk due to TSI.
By focusing only on Alzheimer's dementia nutcomes.
Figure 9, Example of Academic part of the RivaBench.
14

---

video-SALMONN-ol: Reasoning-Enhancee Enhanced Audio-Visual Large Language Model
Speech audio:...we observed statistically significunt improvement
of the 7 key competencies we evaluated them So the of robot AR improved the following key competencies, undcrstanding of voltage and current, connections in scries and parallel, circuitry, use breadhoard, incasurements, and finally having working circuit. Also, Zoom condition, only students managed complete all the exercises available. Mcanwhile, 7 out of the 12 students from the robot AR condition managed to...
[Key Learning Competencies]
Col domm peocle
Working
of the 12
Question: For how many students achieve fall competency condition, improvement ratic of RohotAR over Zoom?
Answer: RobotAR: 7, Zoom: 3, Improvement: 4/3
nead compare the proportion students who reach full conditions. Specific caleulations folpoes/nnrrpporioon fully capable students under Zoom conditions...
Explain: To calculate the imprevement
competence
Choices:
A. RobotAR: 14. Zoom: Improvement: 7/6.
RobotAR: 12, Zoom: 4, Improvement: 3/2.
C. RobotAR: 16, Zecm: 9, Improvement: 2/1.
D. RobotAR 10, Zeem: 5, Improvement: 2/1.
RobsotA:: Zoem: 3, Improvement: 4/3
Figure 10. Example of Academic part of the RivaBench.
D. Synthetic Video Detection Data Examples
Two synthetic video examples in the SynthDec partition of RivaBench are shown in Fig. 11 and 12 respectively.
Figure Example video clip of the SynthDee part RivaBench.
Figure 12. Example video clip of the SynthDec part of RivaBench.
E. Prompt Templates
Prompt templates for ol are shown Table 5. When trained with the reasoning prompt template, the same template is used during inference. Changing templates may cause small perturbations in performance.
15

---

vidco-SALMONN-ol:
Reasoning-Enhannee Audio-Visual Large Language Model
Table 5. Prompt used for different types of tasks.
Type
Prompt content
Direct answer
< MIDEO>~Selec the best answer to the following question based on the video. Re- spond with only the letter of the correct option. {Question]
Choose from: A. {Option A), B, {Option B....
Reasoning
<VIDEO>> Question:
{Question)
Choose from: A. {Option A). B. {Option B)...
Answer the question step by step. Output each thinking step. Mark the end of each step with <fnd_of_step>> token.
SynthDec
An Al-generated video contains unnatural distorted things, such as distorted hands or faces. Is the given video Al generated? Answer YES or NO. Answer step by step and output each step clearly.
E. Case Studies: Solution with Reasoning Examples
PUTZA BUbBE
YOU THINK WE STARK
YOUR MCDONALE
NEW INSTEAD
SHOT!
ORDER
What does the speakeer imply by saying he didn't need know that at the end of the video?lashoose Itoar/aı, He regrets ordering so much (ood. AnB When you went to eal, just est and don't prioritize calocies IarC, He doem't like knowing calorio counts of foo..nnD He faels guiliy about his food choines krE. The caloria count was surprisingly low luAnswer the question step by step. Output each thinking step.
Question:
SALMONN-o1
Video-SALMONN-o Process DPO:
spesker making joke how fast food restaurants started putting calotie counts their
The The apeaker la making joke about how fast food restaurants use calorie counts to manipulate
customers into buying more food.
He argues that this bad thing because makes people feel guilty about esting large pertions of unbealthy food.
scys he
already knew he going eat
and delivery suggest fizud humorous that doean't need be told how unhealthy his
actually upact
He then 'I didn't need to know that." implying that he doesn't like being reminded of the calorie count and feels like it's an unnecessory piece of information.
count,
choices (Combining audio-visual information)
Therefore, the speaker implies that when you want and don't prionitize calories. 5. The ansveer is B
Therefore, the anserer *C.. He doeen': like knowing calorie counts of food"" (Misinterpretation)
Figure 13, Example video and solutions from the StandUp test set.
16

---

Video-SALMONN-oo: Reasoning-Enhameee Enhanced Audio- Visual Large Language Model
Question: legend by turret? Chooze laB. Ice InC. Taunder
Fize. inAnrwer questicn step by step.
5abb--AAMMNNN-o1
video- SAMM-N-N--N- DPO:
(be question about legend when he by
The question about legend when
2. We need to identify the legend that appaars
2. The legend character in the game, and
when the turret hits the character.
structures that fire characters.
3. At 0:19 in the video, the turret hits the character and
3. The legend is turret around 0:15 seconds into the video.
legend appeats. (Time wrong)
4. The legend that appoats in 'Q: (Erelevant))
4. When the lagend hit by the turner, there is a burst
5. Thereforo, the answer in *1C, Thunder" because the
of fire.
legend Q represents the character's ability cast thunderclsp.
5. Therefore, the ansorer is "DD, Fize"".
Figure 14. Example video and solutions from videoMME test set.
Question: following
appear video? Choose InA. Iceberg.InB Barth./dd River inAnnwer question step by step.
Aideo-SALMONN-AA SFT:
Sideo-SALMONN-e1 Process DPO:
"Step 1- Understand the question" "* We need
The question which
present
idlontify which of the givan elementa (leabarg, Moon, Earth, River) not shown the video.
video.
We need carefully watch the video and identify the elements shown.
"Step 2: Watch the video carefully:" Pay atlention each and note the elements present.
The video shows various naturel landsscapes like mountains, waterfalls, forasts, disserts, occans, and view from space.
3. "Step 3. Identify the elements."" ** The video shows various natural landscapes like mountains, waterfalls, forests, dlese is, ocesns, and even a view from space. vlso festures cveries of joeieergs, rivers, and moonlit siry.
4. However, the video does not show the moon.
5. Therefore, the answer "B, Mount"
"Step Compare with the options:" We can see Joebbergs, Rivers, and Moon video. However, the Earth not explicitly shown as separate Moonlit siry Maon)
"Step Final Bazed observaticn.
does appear the
Earth (Option
Figure 15. Example vidco and solutions from vidcoMME test set.
17

---

video-SLLMONN-ol: Reasoning-Enhanned Audio- Visual Large Language Model
G. Case Studies: Zero-shot Synthetic Video Detection
Question: An Al-generated video contains unnatural distorted things, such as distorted hands or faces. Is the given video Al generated? Answer YES NO. Answer step step and output each step clearly.
No. The video real.
Pido-SALMMONN-ol:
The video shows man running lowards a train, then cullapsing pain.
The background is blurred and the colors are distorted, suggeating motion blur.
Al-generated videos often techniques motion blur
speed ugeecy.
This can sometimes result in unnatural distortions in the image. In this case, the blurring and distortion shake or movement.
pronounced than what would be expected from natural camera
The hands of the man appear particularly distorted, which not typical for real-life foctage.
Therefore, based the visual of video, likely that the video generated. So the answer YES.
GPT-4o:
Let's analyze the vidio step step: "Clothing and Movement" The person the video is wearing coat is running. The movement appears natural, and the coat flows as expected with the mation. There are no visible distartions in the clothing or the movement of the person.
"Background and Environmen:"" The background shows a train station with people and train. The environment looks consistent and realistic, with no obvious distortions anomalies in the structure of the train, platform.
other people.
The hands and other visible body parts of the person appear normal and propartionate. There signs unnatural distortions, such indicators al-geeerteed content.
"Hands and Body Parts"":
fingers oddly shaped limbs, which
"Color and Lighting"" The color and lighting in the video appear consistent with real-world aetting. There unusual color shifts lighting inconsistencies that would suggest Al generation.
"Conclusion"" NO, the video is not Al-generated..
Gemini-1.5-pe
how to figure out if the video Al-generated:
"Examine the man's face while he's running:" Does it look unusually smooth, blurry, or distorted in any way?
""Pay close attention the fall."" Do the movements look natural, they jerky and unrealistio? these observations, the answer likely NO.
The video doesn't exchibit the classic signs of Al-generated content. The man't face, while bit obscured by the speed, doesn't show the telltale signs of AI diatortion. Additionally, fall, though quick, appears have natural movement.
The final answer NO
Figure 16. Example output from video-SALMONN-ol, GPT-4o and Gemini- 1ommini-1.5-pae 1.5-pro for synthetic vidco detection.
18

---

video-SALMONN-ol: Reasoning-Enhannee Enhanced Audio- Visual Large Language Model
distorted things, cistorted hands or given video Al generated? Answer YBS or NO. Answer step by step and output each step clearly.
Question: Al-generated video
LLaVA-OneVision: NO.
-ol
The video shows cyclist riding on road with a field of flowers the background.
The colors are vibrant and the scene is smooth, suggesting it might be Al generated.
that the eyelist'a shadow perfectly aligned with cyclist, which unusual for a moving object. This suggests lack of natural motion blur or distortion, common characteristic of Al- generated videos.
Upon closer inspection,
Therefore, based the likely that the video la Al generated.
scene
alignment eyclist's it highly
So
YES.
GPT-4o:
Here the steps follow:
""Subject Examination"" The cyclist appears to have a natural posture and movement. There are no visible distortions in the limbs or face that suggest Al generation
"Background Analysis"": are no obvious texture or perspective issues.
landscape, inchuding road,
sky, appears
There
"Motion Artifacts"": The motion the eyclist and the background blur due speed appear natural and consistent with real-life video capture.
*CColar Anomalies":: The colors in the video, including the yellow fields and blue sky, appear natural and consistent without any abrupt unnatural transitions
""Overall Realism":: The viden maintains high level realism throughout, with no elements that seem artificial or out of place.
Gemini-1 1.5-pro:
"Examine the eyclist's form:" Does the body shape look natural throughout the pedaling motion? Are there any points where the limbs look distorted bend add ways?
"Check bike:" Does the bike's movement seem realistic? Do the wheels apin amoothly?
Look for unusual artifacts: Al-generated content can sometimes have strange textures or patterns, particularly in of motion blur.
"Withoot the actual video, it's impossible to give definitive YES or NO answer."
"However", if you observed any of the issues mentioned above while watching the video. it would be atrong indicator generation.
You haven't provided me with the analysis of the video. so can't give you final answer of YES or NO.
Figure 17, Example output from Jide-SALLMONN-o.. GPT-4o and Gemini-1.55 1.5-pro for synthetic video detection.
19

---

video-SALMONN-ol: Reasoning--Enhance Audio- Visual Large Language Model
H. Examples of Contrastive Step Selection Process
What does the girl after man turned to walk away the end? A. went the kitchen'nß. check her arminn. grab girl hands to dance'nI). thumbss upinE. walk to man
Question:
Answer:
Walk
PATH
The question asks about the girl'a action after the away end the video.
4 0.0098
Observe the girl's actions as the man walks away.
4 0.0038
We see that the girl extends her right hand and moves it up in a 'thumbs up' gesture.
0.2450
Therefure, the correct answer is "D. thumbs up"
d 0.0007
PATH 2:
The question asks about the girl's action after the walka away.
0.0018
Observe the video from 0:27 owwarda, where the man turns and atarts walking away.
4 0.0047
At 0:28, the girl reaches out her hand towards the man.
4 0.0358
This action aligns with the option 'walk to man'.
d
Therefure, the is E. walk
0.0074
Figure 18. Example of the contrastive step selection process where two sampled paths are shown and the scores d., are given for each reasoning steps. The 3rd step first solution is wrong due visual hallucination, and as result, very high score is assigned step and that step will be used to perform rollout.
I. Additional Results on VideoHallucer
Model
Object relation
Temperal
Semantic detail
Factual
Non-factual
Overall
Gemini-1.5-pro
52.0
18.5
53.5
16.5
48.5
37.8
video-SALMONN-ol 63.4
56.4
16.0
43.0
55.6
46.2
Table 6. Vidoo-AALMONN-ol performance compared against Gemini-1.5 1.5-pro VideoHallucer (Wang al., 2024c) and report the overall accuracy (when the entire pair is correct) for each category.
20


Reviews:
[{"review_id": "QxKpRHllaP", "reviewer": "Reviewer_BoZJ", "summary": "In this work, an RL-based optimization and reasoning-aware framework is proposed for training a large audio-video multi-modal model called Video-SALMONN-o1. This work emphasizes that significant effort has been invested in improving the mathematical and visual graphical inputs from the RL perspective, leading to the introduction of RL-based approaches for video-understanding models. To achieve this, a new step-by-step reasoning-based video-LLM training dataset has been proposed. This dataset is used to perform reasoning-based Supervised Fine-Tuning (SFT) and RL fine-tuning stages. For the second stage, a combination of PPRM and process DPO objectives has been formulated to improve both solution-level and fine-grained step-level preference training. To enhance training efficiency in the pDPO method, contrastive step selection is utilized, which applies optimization to steps that are most sensitive to video-based changes. The proposed model is trained using multiple stages, including SFT and the RL stage. To further benchmark audio-video models, a new benchmark called RivaBench is introduced, which effectively evaluates the model's reasoning and audio-encoding capabilities. After complete training, Video-SALMONN-o1 is evaluated on several benchmarks, and ablation studies are performed to justify the various optimization design choices proposed in the framework.\n\n## update after rebuttal\n\nDear Authors, \n\nThank you for providing a rebuttal response. The majority of my concerns have been sufficiently addressed. I strongly recommend the authors incorporate all discussions and additional experiments presented in the rebuttal period into the main manuscript. \n\nI will keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the detailed and constructive reviews provided by Reviewer BoZJ. We would like to address the concerns and suggestions as follows:\n1. We follow the evaluation described in VideoHallucer [1] and report the overall accuracy (when the entire pair is correct) for each category as follows:\n| Model    | Object relation | Temperal | Semantic detail | Factual | Non-factual | Overall |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| Gemini-1.5-pro [1] | 52.0 | 18.5 | __53.5__ | 16.5 | 48.5 | 37.8 |\n| video-SALMONN-o1 | __63.4__ | __56.4__ | 16.0 | __43.0__ | __55.6__ | __46.2__ |\n\n[1] VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models\n\n2. We appreciate the suggestions and hence provide the following descriptions on how we will improve the presentation in the revised version.\n  - __Validating the design choices__:\n    - __The SFT set__: contains 20k videos with 30k QA pairs. Mean duration 97.3s and maximum duration 256.1s. The majority of videos are randomly downloaded general YouTube videos, with 5k videos focusing on talks and presentations. We provide metadata files in the github repo.\n    - __How to filter__: We take the video description, question and answer and feed that into GPT-4o and ask if this question requires complicated logical reasoning. Keep the QA pair if the answer is Yes, and discard if No.\n    - __The large standard deviation of Academic category__: It contains math lectures which may last over 20 minutes, and short conference presentations that only last 1 or 2 minutes.\n  - __pDPO dataset__:  The description is provided in Section 6.2 from line 307. The above reasoning-intensive SFT data is used for pDPO. No additional data is generated for pDPO. \n  - __Model Design Choices__: The visual and audio encoders are frozen. The backbone LLM is trained with LoRA (rank=64) and the modality aligner (24M parameters) is trained.\n  - __Subtitle shortcut in RivaBench__:\n    - This only appears in 50% of the StandUp partition and none in the Academic partition.\n    - Even with subtitles, the frame rate of the video is low enough, such that the subtitles are often incomplete as seen by the visual encoder.\n    - The model has to leverage both audio and visual information, and this is reflected by the fact that GPT-4o (visual only) is worse than Gemini-1.5-pro (audio-visual).\n  - __Regarding comparison to video-SALMONN-2__: video-SALMONN-2 is not open-source yet. The proposed model only resembles the model structure and some opensource data. We do not have or use any model parameters from video-SALMONN-2.\n  - __Computational Cost__: The two additional computations are (i) sample generation and (ii) training the model with pDPO. For (i), we used 16 A100 GPUs for 48 hours to generate the number of samples described in lines 317-319. This is a one-off, and we do not need to generate them again. (ii) We use eight A100 GPUs for 24 hours to train pDPO (see lines 270-271). __Note that these are all training time costs, and it does not incur any additional test time costs at all.__\n  - __Open Source__: We promise to provide model checkpoint, training and inference code, as well as benchmark data upon acceptance of the paper.\n\n3. We will also include the following reference [2, 3] in section 2 of our revised paper:\\\n[2] RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness \\\n[3] Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward", "claims_and_evidence": "Yes, the main claims made in the submission are supported by extensive empirical benchmarking results and qualitative results (mostly in the supplementary part). The effectiveness of proposed RL-based framework is effective compared to other video-based foundation models.", "methods_and_evaluation_criteria": "Overall, the evaluation criteria make sense for the problem at hand. However, explicitly checking the hallucination performance of the resultant model is not evaluated. There are test-only benchmarks such as [1] where the proposed method should be tested to validate the proposed framework in terms of its ability to reduce hallucinations in video-related tasks. \n\n[1] VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models", "theoretical_claims": "No explicit proofs are proposed in the manuscript.", "experimental_designs_or_analyses": "I believe there are several aspects of the proposed dataset which are not clearly discussed in the manuscript. \n\n**1) Validating the design choices in preparing SFT Dataset:**\ni) It is unclear what are the initial data source for preparing the reasoning-based version of it. One of the main contributions of this work is to propose a RL/reasoning-friendly dataset. Unfortunately, details on that front are majorly missing. Additionally, discussion and statistics must be provided for various aspects such as QA count, number of videos, video-category distribution, etc. \n\nii) As mentioned in lines 138-140 (right column), what concrete filtering steps are employed to reject lower-quality QA samples? \n\niii) In Tab. 1, the duration for the Academic category has a strange standard deviation of 66.1 seconds, which is larger than the mean itself. \n\n**2) Missing details regarding the pDPO dataset**: \ni) The statistics and curation process of the proposed pDPO dataset has been discussed very late in the manuscript. A dedicated discussion should be added in the manuscript upfront in the initial sections of the manuscript. Instead of too much text, a visual diagram is recommended which outlines the major steps in the pDPO dataset generation process.\n\n**3) Model Design Choices:** \ni) It is unclear exactly what part of the model is frozen and what is trainable in each stage of the training. For example, it seems like most model components are frozen as written in lines 268-270 (right column). \n\n**4) RivaBench:** The manuscript has provided a few video examples of the proposed evaluation benchmark. It can be noticed that in most examples, the speech lyrics are directly rendered on the video. This could cause models to overlook the audio tokens and directly read the subtitles from the visual inputs which is a shortcut. How do the authors aim to resolve this issue? \n\n**5) Evaluations and Experiments:** \n\ni) In my understanding, the proposed Video-LMM is based on video-SALMONN 2 model, but unfortunately, a performance comparison with this model is not performed.\n\n\n**6) Training compute and efficiency comparison**: As the proposed framework utilizes the RL-based pDPO technique, how much extra compute and training cost is incurred compared to the baseline model? A clear compute cost analysis should be provided for the proposed framework.", "relation_to_broader_scientific_literature": "This paper aims to advance the audio-video-language reasoning abilities of foundation models using reasoning and RL based training frameworks.", "essential_references_not_discussed": "RL-based literature for vision-based LLM models is not discussed in the manuscript. It is important that this work provides a related work section for RL-based methods (e.g., [2], [3]) for image domains and contrasts it with the proposed techniques. \n\n\n[2] RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness\n[3] Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward", "other_strengths_and_weaknesses": "Overall, the proposed training and evaluation datasets, alongside the RL-based optimization framework are motivating and useful for the research community. However, the overall presentation style of the manuscript is not good, too generic, and many important details are missing in the paper. It is strongly recommended to address all the comments mentioned in the earlier sections of the manuscript.", "other_comments_or_suggestions": "Minor:\nIn line 133, there is a typo error.", "questions_for_authors": "Please refer to the above parts for questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2fHbspFpzf", "reviewer": "Reviewer_cRid", "summary": "This paper introduces video-SALMONN-o1, an open-source reasoning-enhanced audio-visual LLM designed for general video understanding tasks. The authors claim that existing reasoning models are merely focusing on either math problems or visual graphical inputs, without sufficient attention on general audio-video understanding. To fill this gap, they proposed video-SALMONN-o1 to tackle complex reasoning problems in the field of audio-video understanding. The main contributions include 1) a reasoning-intensive dataset featuring challenging audio-visual questions with step-by-step solutions, 2) a novel process direct preference optimization (pDPO) algorithm, and 3) RivaBench, the first reasoning-intensive video understanding benchmark, with over high-quality, expert-curated question-answer pairs across diverse scenarios. Extensive experiments demonstrate the effectiveness of the proposed scheme.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank you for acknowledging our effort and contribution!", "claims_and_evidence": "The key claim of this paper is about the state of existing reasoning-enhanced MLLM. The authors claim that existing models merely focus on narrow scenarios such as math, coding, or simple visual graphic inputs, while the exploration in general video understanding is missing. The above claim is correct and reasonable, as the community is still exploring the proper reasoning paradigms for video understanding.", "methods_and_evaluation_criteria": "Yes. The proposed method for reasoning-enhanced audio-visual understanding is reasonable. The interleaved synchronization design can better handle the multimodal audio-visual data. The evaluation metric for RivaBench is accuracy (MCQ or Yes/No), which is a common standard in evaluating MLLMs.", "theoretical_claims": "The paper does not contain any proofs or theoretical claims.", "experimental_designs_or_analyses": "The experiments were conducted on both public benchmarks for general video understanding and the proposed RivaBench. Results on these benchmarks can well-demonstrate the effectiveness and the significance of the proposed scheme.", "relation_to_broader_scientific_literature": "The key contribution of this paper is to generalize the reasoning capabilities of LLMs/MLLMs to video understanding, especially in the case when both audio and visual information are used. It diverses this work from existing publications.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Generally, this is a good paper about LLM-based & reasoning-enhanced video understanding. Extending existing paradigms to audio/video understanding is a non-trivial setting. The proposed video-SALMONN-o1 model can well-handle such scenarios.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The authors are encouraged to open-source the code and data mentioned in the submission, which would be a good contribution to the community.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RkELGkezxq", "reviewer": "Reviewer_3hnn", "summary": "The paper introduces video-SALMONN-o1, an open-source audio-visual LLM enhanced for general video understanding. It proposes pDPO for reasoning optimization and RivaBench, a new benchmark. The model shows improved accuracy over baselines and zero-shot synthetic video detection capabilities. However, the benchmark's reliance on LLM-generated data raises concerns about accuracy and bias.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and would like to resolve concerns and misunderstandings as follows:\n1. __Regarding the reliability and bias of the benchmark__:\n  - As stated in section 5 paragraphs 2 and 3, we __always__ use human annotators to generate questions and answers, and __always__ use human annotators to check and validate the questions and answers.\n  - We employed over 50 __expert__ annotators to perform this task. This way, we ensure diversity and minimize bias and hallucination to a level that is __above average human beings__.\n  - Please refer to the \"__RivaBench Demo Samples__\" section of the demo page in the paper for concrete examples of human-created questions:\n    - One medical example requires professional medical knowledge. The explanation reflects the annotator's knowledge.\n    - One math example requires the ability to understand parabolic partial differential equations. See the detailed step-by-step derivation provided by the annotator.\n    - All samples are accompanied with such expert explanations and will be released upon acceptance. \n\n2. __Comparisons with other SOTA methods__:\n  - We have compared LLaVA-OneVision in Table 2 as a strong open-source video understanding model baseline.\n  - We included the performance of Qwen2-VL as follows:\n| Models | VideoMME | NeXT-QA | Academic | StandUp | SynthDec |\n| -------- | ------- | ------- | ------- | ------- | ------- |\n| Qwen2-VL | 62.9 | 80.2 | 48.2 | 71.6 | 0.0 |\n| video-SALMONN-o1| __65.6__ | __82.3__ | __48.3__ | __76.7__ | __17.8__ |\n  - video-SALMONN-o1 achieved superior performance on all the benchmarks. The improvement on the academic partition is smaller because this data needs __less__ audio-visual understanding but __more__ mathematical reasoning. However, pDPO focuses more on optimizing audio-visual understanding in the reasoning process.\n  - We have also included the results for Gemini-1.5-pro in Table 2. Note that Gemini-1.5-pro is a proprietary LLM with unknown (potentially much larger) model sizes and unknown inference mechanism, and hence no direct comparison should be drawn with it. Given that, our model __still outperforms__ Gemini-1.5-pro on NeXT-QA and StandUp.", "claims_and_evidence": "The claims in the submission lack clear and convincing evidence. The benchmark results may be unreliable due to potential hallucinations and biases in the large models used to create them. Additionally, the authors fail to compare their model with true SOTA models like Gemini 1.5 Pro[3], Qwen2-VL[1], and GLM-4V-Plus[2], undermining the validity of their claims about achieving SOTA performance.\n\n\n- [1] Wang, Peng, et al. \"Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.\" arXiv preprint arXiv:2409.12191 (2024).\n- [2] Hong, Wenyi, et al. \"Cogvlm2: Visual language models for image and video understanding.\" arXiv preprint arXiv:2408.16500 (2024).\n- [3] Team, Gemini, et al. \"Gemini: a family of highly capable multimodal models.\" arXiv preprint arXiv:2312.11805 (2023).", "methods_and_evaluation_criteria": "The proposed methods, including the reasoning-enhanced audio-visual LLM and pDPO, are relevant for general video understanding. However, the evaluation benchmark RivaBench is generated by LLMs, which may introduce biases and inaccuracies due to hallucinations. Overall, while the methods are sensible, the evaluation criteria have significant limitations.", "theoretical_claims": "The paper does not present any formal theoretical proof. It focuses on empirical results and methodological contributions, such as the introduction of video-SALMONN-o1 and pDPO. Therefore, there are no proofs to verify.", "experimental_designs_or_analyses": "I checked the experimental designs and found issues, particularly with the benchmark construction. The RivaBench was generated using large language models, which can introduce hallucinations and biases, potentially compromising its validity. Additionally, the authors did not compare their model against true SOTA models like Gemini 1.5 Pro, Qwen2-VL, or GLM-4V-Plus, which raises concerns about the robustness of their claims.", "relation_to_broader_scientific_literature": "No broader impact.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: The paper introduces video-SALMONN-o1, a novel open-source audio-visual LLM with enhanced reasoning abilities, and proposes pDPO for efficient step-level reward modeling. It also introduces RivaBench, a reasoning-intensive benchmark.\n\nWeaknesses: The benchmark relies on LLM-generated data, which may introduce biases and hallucinations. The paper lacks comparisons with true SOTA models like Gemini 1.5 Pro and Qwen2-VL, potentially overstating its achievements. Supplementary materials are absent, limiting reproducibility.", "other_comments_or_suggestions": "1. Include supplementary material to support claims and provide detailed methodology.\n2. Validate benchmark results with human annotations to mitigate model biases and hallucinations.\n3. Compare with true SOTA models (e.g., Gemini 1.5 Pro, Qwen2-VL, GLM-4V-Plus) for a fair evaluation.", "questions_for_authors": "How do you address potential biases and hallucinations in the benchmark data generated by LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTMFvXij0T", "reviewer": "Reviewer_dh9B", "summary": "This paper introduces video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM designed to address the underexplored challenge of general video understanding, which requires complex multimodal (audio-visual-text) reasoning. Current reasoning-optimized LLMs focus narrowly on mathematical/textual tasks or image inputs, limiting their applicability to real-world video scenarios like academic presentations or synthetic video detection. The authors propose a two-pronged approach: (1) a reasoning-intensive dataset with step-by-step solutions for supervised fine-tuning, and (2) process Direct Preference Optimization (pDPO), a novel training method that optimizes reasoning paths via contrastive step selection, eliminating the need for external reward models. They also introduce RivaBench, a benchmark with 4,000+ expert-curated QA pairs spanning diverse video contexts. Results show 3–8% accuracy gains over visual baselines (e.g., LLaVA-OneVision), 6–8% improvement from pDPO on RivaBench, and novel zero-shot synthetic video detection capabilities.\n\n## update after rebuttal\n\nThanks for response. My concerns have all been solved.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate Reviewer dh9B for the positive comments and acknowledgement of our contribution. We would like to address the following questions:\n1. __Inference mechanics__:\n  - We use greedy decoding during inference.\n  - We use contrastive step selection only to construct training preference pairs for pDPO. Hence no impact on latency.\n2. __Generalizability validation__:\n  - While focusing on audio-visual tasks, we also present results on visual only tasks including NeXT-QA and SynthDec partition throughout the paper (Table 2-4).", "claims_and_evidence": "Yes, they are well supported.", "methods_and_evaluation_criteria": "Yes, they do.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, I checked the experimental designs and analyses in Sec. 6.", "relation_to_broader_scientific_literature": "The reasoning-enhanced training strategy can be applied to broad video tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n\n- The technical contribution is solid, and the conducted experiments are convincing.\n- The provided experimental results clearly show the effectiveness of the proposed method.\n\n## Weaknesses\n\n- Unclear inference mechanics: The paper lacks details about the inference pipeline. Please elaborate more on this. For example: \n  - How many reasoning paths are generated per video-question pair during testing? \n  - Is the reward model (or pDPO’s contrastive step selection) applied at inference time, and if so, how does this impact latency?\n\n- Limited generalizability validation: While focused on audio-visual tasks, the proposed pDPO and reasoning dataset appear applicable to vision-only LLMs (e.g., LLaVA-OneVision). Testing these components on other architectures would strengthen claims about methodological universality.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: The manuscript received ratings of 4, 4, 3, and 1. Reviewers appreciated that the proposed training and evaluation datasets, alongside the RL-based optimization framework are motivating, useful for the research community, and the results demonstrate the effectiveness of the proposed scheme. Reviewers also raised several questions including, missing details regarding the pDPO dataset, model design choices, training compute and efficiency comparison, and the benchmark relies on LLM-generated data. The authors provided a detailed rebuttal to address reviewers concerns. Post-rebuttal, reviewer BoZJ mentioned that majority of the reviewer's concerns are sufficiently addressed. To address the concerns of reviewer 3hnn about the reliance on LLM-generated data, the authors also clarified that human annotators are always used to generate questions and answers and to check and validate the questions and answers. Further, the authors clarified that they employed over 50 expert annotators to perform this task. The AC believes this addresses the concerns of reviewer 3hnn, which were not acknowledged by the reviewer post-rebuttal. Given that the remaining three reviewers are generally positive and the authors addressed the main concern of the fourth reviewer regarding potential issues in the data (e.g., LLM-generated data), the AC agrees that the proposed approach has merits and will be interesting for the research community.
Decision: Accept (poster)