Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization

## Abstract

Abstract Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding. In this paper, we present video-SALMONN 2, an advanced audio-visual large language model (LLM) with low-rank adaptation (LoRA) designed for enhanced video (with paired audio) captioning through directed preference optimization (DPO). We propose new metrics to evaluate the completeness and accuracy of video descriptions, which are optimized using DPO. To further improve training, we introduce a novel multi-round DPO (mrDPO) approach, which involves periodically updating the DPO reference model, merging and re-initializing the LoRA module as a proxy for parameter updates after each training round (1,000 steps), and incorporating guidance from ground-truth video captions to stabilize the process. To address potential catastrophic forgetting of non-captioning abilities due to mrDPO, we propose rebirth tuning, which finetunes the pre-DPO LLM by using the captions generated by the mrDPO-trained model as supervised labels. Experiments show that mrDPO significantly enhances video-SALMONN 2’s captioning accuracy, reducing global and local error rates by 40% and 20%, respectively, while decreasing the repetition rate by 35%. The final video-SALMONN 2 model, with just 7 billion parameters, surpasses leading models such as GPT-4o and Gemini-1.5-Pro in video captioning tasks, while maintaining competitive performance to the state-of-the-art on widely used video question-answering benchmark among models of similar size. Upon acceptance, we will release the code, model checkpoints, and training and test data. Demos are available at https://video-salmonn-2.github.io .

## 1 Introduction

Large language models (LLMs) have exhibited outstanding capabilities in a wide range of natural language processing (NLP) tasks, and in some instances, have even approached human-level performance (OpenAI et al., ; Dubey et al., ; Touvron et al., ; Du et al., ; Bai et al., ). LLMs’ remarkable ability to understand, generate, and reason with text has sparked widespread interest among researchers, attracting both academia and industry to extend them to multimodal understanding and generation.
To endow LLMs with multimodal understanding capability, recent studies adopted a paradigm of training modality adapters and aligners between multi-modal encoders and LLMs. This approach leverages world knowledge in the textual LLM to interpret diverse types of data perceived by multimodal encoders, enabling the generation of meaningful insights.
Over the past two years, many multimodal LLMs have emerged following this paradigm across different modalities. These include models for image and silent video understanding (Liu et al., ; ; Li et al., ; Bai et al., ; Lin et al., ; Chen et al., ; Lin et al., ; Chen et al., ), audio understanding (Wu et al., ; Tang et al., ; Chu et al., ; ; Gong et al., ; ; Tang et al., ; Zheng et al., ), and audio-visual understanding (Team et al., ; Cheng et al., ; Sun et al., ; Fu et al., ; Tang et al., ).

Text descriptions of multimodal data are critical for building multimodal LLMs. This is because most contemporary multimodal LLMs treat multimodal captions as a cornerstone task during pre-training or supervised fine-tuning (SFT), to align the representation spaces of multimodal encoders with that of textual large language models, helping LLMs recognise and understand events in multimodal data. Thus, collecting high-quality text descriptions paired with multimodal data is crucial for constructing high-performance multimodal LLMs, which implies training the model with more detailed and less hallucinated labels aligned with the multimodal data that could enhance the LLM’s ability to perform multimodal understanding and reasoning.
In video understanding, generating detailed and accurate captions is crucial but challenging, as videos contain rich content that encompasses not only spatial features within individual visual frames but also audio-visual events that unfold across multiple frames over time.
However, very few multimodal LLM-related works focus on improving the quality of video captions, due to the lack of quantitative metrics for evaluating video captions and the absence of training methods to enhance the completeness of these descriptions while reducing the risks of hallucination. Additionally, while audio is typically paired with video and provides crucial, complementary information to the visual content, most current visual LLMs lack audio-understanding abilities, leading to the omission of audio information in the generated captions.

In this paper, we introduce video-SALMONN 2, a multimodal LLM that supports both audio and visual inputs and primarily focuses on detailed and holistic audio-visual captioning. Building upon an already well-trained visual LLM, video-SALMONN 2 is further enhanced with auditory capabilities by training on audio-only data as well as videos with synchronized audio tracks. This enables the model to simultaneously “see” and “hear” the video, emulating the way humans perceive and interpret multimedia content.
To accurately assess the performance of the model, new metrics to evaluate captioning quality are proposed, which then serve as the objective to optimize during reinforcement learning (RL) based on direct preference optimization (DPO). A novel multi-round DPO (mrDPO) is proposed and performed based on the preferences guided by the metrics, followed by a novel rebirth tuning stage to avoid the degradation of the non-captioning abilities caused by the mrDPO. The rebirth tuning leverages the post-mrDPO model to revise the captions of the videos in the training set, and trains the model after audio modality alignment using supervised fine-tuning (SFT) with the revised training data.
Experiments demonstrate that video-SALMONN 2 with 7 billion (B) parameters can generate complete and accurate video descriptions and even outperforms much larger commercial multimodal LLMs such as GPT-4o and Gemini-1.5-Pro, and it also maintains competitive performance to the state-of-the-art (SOTA) multimodal LLM of similar model size on the commonly used Video-MME (Fu et al., ) video question-answering (QA) benchmark.

The main contributions of this work can be summarised as follows:

- •
We develop video-SALMONN 2, a powerful audio-visual LLM that generates high-quality video captions, outperforming larger commercial models such as GPT-4o and Gemini-1.5 in terms of completeness and accuracy.
- •
We introduce an evaluation pipeline that computes the missing and hallucination rates of audio-visual events in video captions using text-based LLMs, breaking down the process into sub-tasks suited for current LLMs. Additionally, we provide a new benchmark for video captioning with a human-annotated test set.
- •
We propose the mrDPO approach to optimize multimodal LLMs for video captioning, incorporating periodic updates to the DPO reference model, merging and reinitializing the low-rank adaptation (LoRA) (Hu et al., ) module, and smoothing the training loss using SFT based on ground-truth captions. To our knowledge, this is the first work applying RL to audio-visual LLMs.
- •
We introduce rebirth tuning to ensure the resulting model maintains high performance in both captioning and non-captioning tasks. The mrDPO process, followed by rebirth tuning, can be iteratively applied to further enhance performance.

## 2 Related Work

### 2.1 Multimodal LLMs

Following the paradigm of connecting multimodal encoders to LLMs using modality adapters, various models have been developed. For image-based LLMs, LLaVA (Liu et al., ; ) applies instruction tuning (Wei et al., ) to enhance performance on zero-shot tasks. BLIP-2 (Li et al., ) uses Q-Former to link a frozen encoder with an LLM, while VILA (Lin et al., ) explores pre-training strategies, achieving impressive results in video QA. InternVL (Chen et al., ) scales up the size of visual encoders for improved image representation. For silent video understanding, Video-LLaVA (Lin et al., ) aligns both image and video adapters to learn unified representations. ShareGPT4Video (Chen et al., ) uses GPT-4 to generate dense video captions, improving data quality, and LLaVA-Hound (Zhang et al., ) introduces DPO to enhance video LLMs’ understanding capabilities.

In the realm of audio perception, SALMONN (Tang et al., ) uses a dual-encoder structure and can perform zero-shot audio reasoning tasks. LTU (Gong et al., ) and LTU-AS (Gong et al., ) trained on a large audio question-answering dataset are able to answer open-ended questions about audio. Qwen-Audio (Chu et al., ) and Qwen2-Audio (Chu et al., ) are built on large amounts of audio data to achieve high performance on a wide range of carefully selected audio tasks. Zheng et al. () and Tang et al. () extend the LLM to perceive spatial audio information obtained from microphone array recordings.

As the visual frame sequence is often paired with audio in real-world video recordings, some studies investigate understanding non-silent video. video-SALMONN (Sun et al., ) uses a multi-resolution causal Q-Former to understand audio and video simultaneously. The Google Gemini model achieves video understanding as a native multimodal LLM built upon text, audio, and visual tokens (Team et al., ). AVicuna (Tang et al., ) achieves audio-visual temporal understanding by introducing pseudo-untrimmed video data. Video-LLaMA (Zhang et al., ) and Video-LLaMA 2 (Cheng et al., ) directly concatenating audio and visual tokens for joint audio and video understanding.

### 2.2 RL for LLMs

RL with human feedback (RLHF) (Ouyang et al., ) is commonly used to enhance text-based LLMs, with early efforts applying PPO (Schulman et al., ) alongside a reward model trained on human preference data. Building on this, DPO (Rafailov et al., ) proposes that the LLM itself can serve as a reward model, using paired preference data to optimize the model without the need for an external reward model. KTO (Ethayarajh et al., ) further eliminates the need for paired preference data. Expanding on this, RLAIF (Lee et al., ) takes a cost-efficient approach by utilizing feedback generated automatically by models, reducing reliance on human involvement.

## 3 Methods

Figure: Figure 1: The overall structure of video-SALMONN 2. The input video is processed separately by the visual and audio branches, generating visual and audio tokens from the visual and audio frame sequences. Next, the visual and audio tokens are interleaved synchronously, and combined with the tokens of the text prompt to form the input to the LLM backbone.
Refer to caption: x1.png

### 3.1 Model Architecture

The overall architecture of our model is illustrated in Fig. 1. The paired sequences of audio and visual frames from each video are fed into the audio and visual encoders separately. Users can provide textual prompts to guide the model in performing specific tasks based on the video content.
This structure is implemented by incorporating a separate audio encoder branch to a pre-trained visual LLM, which enables the model to process and understand paired audio-visual sequences without degrading its visual performance.

In this structure, audio and visual tokens are computed independently in their respective branches. For the visual branch, the input visual frame sequence is first downsampled at a fixed frame rate of $\phi$ frame/second, and the total number of frames to sample is $n=\phi\,T$, where $T$ is the duration of the input video in seconds. Let $m$ be the maximum number of frames to sample based on the resource constraint. If $n>m$, the frame rate is further reduced to $\phi^{\prime}=\lfloor m/T\rfloor$, resulting in $n=\phi^{\prime}T\leq m$. Let $\mathbf{I}_{i}$ be the $i$th sampled visual frame, and each visual frame in $\mathbf{I}_{1},\mathbf{I}_{2},\ldots,\mathbf{I}_{n}$ is transformed to visual tokens independently using a pre-trained visual encoder $\text{Encoder}_{\text{Visual}}$ followed by a visual modality aligner $\text{Aligner}_{\text{Visual}}$, as shown in Eqn. (1):

$$ $\mathbf{H}^{\text{Visual}}_{i}=\text{Aligner}_{\text{Visual}}(\text{Encoder}_{ \text{Visual}}(\mathbf{I}_{i})),~{}~{}1\leq i\leq n,$ (1) $$

where $\mathbf{H_{\text{V}}}_{i}$ represents the visual tokens corresponding to $\mathbf{I}_{i}$.

The audio frame sequence $\mathbf{S}$ is fed into a pre-trained audio encoder $\text{Encoder}_{\text{Audio}}$. Since $\text{Encoder}_{\text{Audio}}$ may have a maximum processing duration $t_{\text{max}}$, the audio will be sliced into $l=\lceil{T}/{t_{\text{max}}\rceil}$ segments of $t_{\text{max}}$-length and processed separately by $\text{Encoder}_{\text{Audio}}$, as shown in Eqn. (2):

$$ $\mathbf{Z}^{\text{Audio}}_{j}=\text{Encoder}_{\text{Audio}}(\mathbf{S}_{(j-1) \times t_{\text{max}}:j\times t_{\text{max}}}),~{}~{}1\leq j\leq l,$ (2) $$

where $\mathbf{Z}^{\text{Audio}}_{j}$ is the audio feature vector output by the audio encoder of the $j$th audio segment.

As suggested by Yu et al. (), a segment-level positional embedding is added before the modality aligner to improve the performance of long-form audio.
Denote $\mathbf{Z}^{\text{Pos}}_{j}$ as the segment-level position embedding matrix corresponding to the position $j$, $\text{Concat}(\cdot)$ as the concatenation operation along the time dimension, and $\text{Aligner}_{\text{Audio}}$ as the audio modality aligner. The audio token sequence $\mathbf{H}^{\text{Audio}}$ for the whole audio can be computed as Eqn. (3)–(5) shown:

$$ $\displaystyle\tilde{\mathbf{Z}}^{\text{Audio}}_{j}$ $\displaystyle=\mathbf{Z}^{\text{Audio}}_{j}+\mathbf{Z}^{\text{Pos}}_{j},~{}~{} 1\leq j\leq l$ (3) $\displaystyle\tilde{\mathbf{Z}}^{\text{Audio}}$ $\displaystyle=\text{Concat}(\tilde{\mathbf{Z}}^{\text{Audio}}_{1},\tilde{ \mathbf{Z}}^{\text{Audio}}_{2},\ldots,\tilde{\mathbf{Z}}^{\text{Audio}}_{l})$ (4) $\displaystyle\mathbf{H}^{\text{Audio}}$ $\displaystyle=\text{Aligner}_{\text{Audio}}(\tilde{\mathbf{Z}}^{\text{Audio}}).$ (5) $$

Next, the audio and visual tokens are interleaved chronologically to form the input audio-visual token sequence $\mathbf{H}$ fed into the LLM backbone, and $\mathbf{H}$ is obtained based on Eqn. (6)–(8) by

$$ $\displaystyle\alpha_{i}$ $\displaystyle=l\cdot{i}/{n},~{}~{}1\leq i\leq n$ (6) $\displaystyle\mathbf{H}_{i}$ $\displaystyle=\text{Concat}(\mathbf{H}^{\text{Visual}}_{i},\mathbf{H}^{\text{ Audio}}_{\alpha_{i-1}:\alpha_{i}}),~{}~{}1\leq i\leq n$ (7) $\displaystyle\mathbf{H}$ $\displaystyle=\text{Concat}(\mathbf{H}_{1},\mathbf{H}_{2},\ldots,\mathbf{H}_{n }).$ (8) $$

Finally, the text-based backbone LLM is required to generate a text response $\mathbf{\hat{Y}}$ given the user’s text prompt $\mathbf{P}$ and the audio-visual token sequence $\mathbf{H}$:

$$ $\mathbf{\hat{Y}}=\arg\max\nolimits_{\mathbf{Y}}P(\mathbf{Y}|\mathbf{P},\mathbf {H}).$ (9) $$

### 3.2 Training Strategies

To introduce audio perceptual capabilities to the visual LLM, we employ a multi-stage training approach that enables the model to fully utilize audio information for video understanding while maintaining its performance in processing visual data. Building on a well-trained visual LLM, the training proceeds through several stages: audio modality alignment, audio-visual SFT, RL based on the proposed mrDPO, and the newly introduced rebirth tuning.
For the pre-trained visual LLM that already understands video, with the LLM, visual aligner, and visual encoder well-initialized, both the LLM and video branch are kept frozen during all training stages. Similarly, the audio encoder parameters remain fixed, as they have already been trained on a large-scale audio dataset.

Audio modality alignment extends the visual LLM by adding a parallel audio branch, enabling auditory perception capabilities. During this stage, only the audio aligner is trained on a large audio dataset, while the rest of the model remains frozen to preserve its original visual understanding performance. Since the focus is exclusively on learning the audio branch, only audio data is needed for training.

After audio modality alignment, the backbone LLM can recognize both visual and audio tokens. However, due to the lack of training with paired audio and visual token sequences, the model is not yet capable of synchronizing and integrating audio-visual information for comprehensive video understanding. To address this, we conduct audio-visual SFT using supervised video data.
To improve the backbone LLM’s ability to process audio-visual token sequences, LoRA (Hu et al., ) is applied and trained during this stage. Additionally, the audio aligner is trained to align the output of the audio encoder with the input representation space of the LLM, making it easier for the backbone LLM to interpret audio tokens.

Although the model demonstrates the ability to describe synchronized audio-visual information in video after SFT, several issues persist, including missing information, hallucinations, and repetitive decoding. To address these shortcomings, we apply RL based on mrDPO to improve the model’s performance. Additionally, we introduce rebirth tuning after RL to further enhance the model’s performance in non-captioning tasks. Fig. 2 provides an overview of the entire training process involving mrDPO and rebirth tuning, with further details explained in Sections 3.3 and 3.4.

Figure: Figure 2: An overview of the complete training process includes audio modality alignment, SFT, mrDPO, and rebirth tuning. LoRA is introduced during the SFT stage, and a new LoRA proxy is added to the LLM before each round of DPO. After multiple rounds of DPO, the model generates new data, which is then utilized during rebirth tuning to further refine the model.
Refer to caption: x2.png

### 3.3 RL stage with mrDPO

We aim to leverage RL to improve the quality of video captions generated by the model. To establish an effective method for evaluating the completeness of video captions, we propose using atomic events as a bridge to automatically assess the preference of caption samples through artificial intelligence (AI) feedback, guiding the model to produce more accurate and detailed video descriptions.

Figure: Figure 3: The pipeline for mrDPO in video captioning uses GPT to process the video’s atomic event list to automatically identify preferred and dispreferred caption samples generated by the model. These sample pairs are then utilized in applying DPO-based RL algorithms. This process can be repeated over multiple rounds to iteratively enhance performance.
Refer to caption: x3.png

The pipeline for selecting preferred samples when applying RL to video-SALMONN 2 is illustrated in Fig. 3. First, distinct video captions are sampled from the model’s output distribution, given the input video. These captions may be either global captions describing the entire video or local captions focusing on a specific time interval.
To determine the preferred sample for global captions, the labelled caption of the input video is fed into a powerful text LLM, which is tasked with breaking down the caption into basic atomic events. This is relatively straightforward for commercial LLMs like GPT-3.5 and GPT-4o, and the resulting atomic events are generally reasonable.
Next, the text LLM is used to evaluate the caption by identifying missed or hallucinated events, calculating the information missing and hallucination rates. The total error rate is the sum of the missing and hallucination rates. For local captions, a similar evaluation process is followed, with atomic events being extracted using Gemini-1.5-Pro (as detailed in Appendix A).
In addition to metrics based on atomic events, we also consider the repetition rate of the video captions. The calculation procedure for this is provided in Appendix B.
DPO (Rafailov et al., ) is applied as the main RL method based on automatic AI feedback. We assume that only sample pairs with significant metric differences are suitable for RL. Therefore, sample pairs with minimal gaps in metrics are excluded from the RL training set.

Unlike previous approaches that applied only single-round DPO to multimodal LLMs, we introduce a multi-round strategy, as prolonged offline training with a single round fails to optimize the model effectively due to the reference model being biased against the most recent model update in the DPO algorithm. In the multi-round framework, at each $t$th round, the following steps are taken to perform DPO training for the current round.

- 1.
First, pre-trained LoRA module $\Delta_{t-1}$ is merged into the LLM backbone $\Lambda_{t-1}$ to derive a new LLM backbone $\Lambda_{t}$ that is equivalent to $\Lambda_{t-1}$ with $\Delta_{t-1}$, based on Eqn. (10):
$\mathbf{W}_{t}=\mathbf{W}_{t-1}+\alpha\mathbf{A}_{t-1}\mathbf{B}_{t-1},$
(10)
where $\mathbf{W}_{t}$ and $\mathbf{W}_{t-1}$ are the weight parameters to adapt in $\Lambda_{t}$ and $\Lambda_{t-1}$, $\alpha$ is the scaling factor of LoRA,
$r$ is the rank of LoRA, $d$ is the dimension of $\mathbf{W}_{t-1}$. $\mathbf{A}_{t-1}\in\mathcal{R}^{d\times r}$ and $\mathbf{B}_{t-1}\in\mathcal{R}^{r\times d}$ are the low-rank matrix parameters of LoRA in the previous round $t-1$, and $\mathbf{W}\in\mathcal{R}^{d\times d}$ is the parameter of LLM backbone.
- 2.
Next, the new LLM backbone $\Lambda_{t}$ is paired with a new randomly initialized LoRA module $\tilde{\Delta}_{t}$, forming the policy model for round $t$. To address the issue of the increasing difference between the reference and policy models caused by freezing the reference model in standard DPO, $\Lambda_{t}$ is used as the updated reference model used in round $t$.
- 3.
At last, $\tilde{\Delta}_{t}$ is trained to obtain a well-trained ${\Delta}_{t}$, which can be achieved using the standard DPO loss. However, after multiple training rounds, the model starts to produce unnatural language patterns such as unintelligible or meaningless sentences.
To alleviate this issue by stabilizing the training, a guided DPO (gDPO) loss is proposed as
$\displaystyle\mathcal{L}_{\text{gDPO}}(\mathbf{\pi}_{\theta};\pi_{\text{ref}})
=\mathcal{L}_{\text{DPO}}(\mathbf{\pi}_{\theta};\pi_{\text{ref}})+\lambda\,
\mathbb{E}_{(\mathbf{x},\mathbf{y}_{\text{gt}})\sim\mathcal{D}_{\text{gt}}}
\log\pi_{\theta}(\mathbf{y}_{\text{gt}}|\mathbf{x}),$
(11)
where $\mathcal{L}_{\text{DPO}}$ is the standard DPO loss, $\mathbf{\pi}_{\theta}=\{\Lambda_{t},\tilde{\Delta}_{t}\}$ and $\pi_{\text{ref}}=\Lambda_{t}$ represent the policy and reference models, respectively. $\mathcal{D}_{\text{gt}}$ denotes the SFT training dataset, where $(\mathbf{x},\mathbf{y}_{\text{gt}})$ corresponds to a video and its paired ground-truth text description, randomly selected from $\mathcal{D}_{\text{gt}}$. Finally, $\lambda$ is the weight of the second regularization term, which corresponds to cross-entropy learning towards the ground-truth text descriptions without unnatural patterns.

These steps complete the training for a single round. Our proposed mrDPO is implemented by repeating these steps across multiple rounds. Notably, by merging $\Delta_{t-1}$ into $\Lambda_{t-1}$ and equipping the resulting $\Lambda_{t}$ with a new LoRA module $\tilde{\Delta}{t}$, the new $\tilde{\Delta}{t}$ functions as a LoRA proxy for parameter updates. This proxy helps regularize the training by introducing a new random initialization at each round of mrDPO.

### 3.4 Rebirth Tuning

After multiple rounds of iteration with the LoRA proxy, the model demonstrates significant improvements in captioning, showcasing the strong potential for audio-visual understanding. However, despite efforts to preserve its language abilities, the model gradually begins to produce repetitive and unnatural text patterns in its responses. In some cases, its benchmark performance remains high even if unnatural patterns appear.
We believe this issue arises because RL methods primarily optimize the model’s output distribution using self-generated data. As a result, the model may overfit the feedback provided by AI models that mismatch with real human preferences, leading it to adopt these unnatural patterns. This tendency can cause a collapse in the model’s output distribution, resulting in frequent incoherent or repetitive outputs, and considerable declining performances on the non-captioning tasks.

Rebirth tuning is introduced to address the issue of declining non-captioning language abilities. This method applies teacher-forcing training on self-generated data, promoting a more stable learning process for video understanding. Teacher-forcing, which guides the model to predict the next token, helps prevent it from converging on limited and repetitive patterns. More specifically, before applying rebirth tuning, mrDPO is halted once we observe a significant decline in the model’s language capabilities. The final iteration of the model, which excels at generating complete and accurate video descriptions, is then used to label a large dataset of videos. Since the model’s language abilities remain relatively intact, natural and fluent descriptions can be easily filtered by detecting problematic patterns, with the remaining high-quality descriptions used as training data for rebirth tuning.

Rebirth tuning is applied to the model after audio modality alignment, allowing it to be ”reborn” from self-generated high-quality data to enhance video understanding. Following rebirth tuning, the model not only avoids catastrophic forgetting of non-captioning abilities but also supports the development of the next generation of models by applying mrDPO, followed by the subsequent stage of rebirth tuning.

## 4 Experimental Setup

### 4.1 Model Specifications

video-SALMONN 2 is built on an internally trained high-performance visual LLM. This visual LLM uses SigLIP (Zhai et al., ) as the visual encoder, Qwen 1.5 with 7B parameters as the backbone LLM, and two linear layers with GELU activation function (Hendrycks & Gimpel, ) as the visual aligner. The model processes video frames at a frame rate of 1 (i.e., $\phi=1$), and can handle up to 30 frames. For videos longer than 30 seconds, 30 frames are uniformly sampled from the video.

For the audio branch, we use the Whisper-Large-v3 encoder (Radford et al., ) as the audio encoder, and a window-level Q-Former (Tang et al., ) with a window length of 0.2 seconds as the audio aligner, producing a total of 150 audio tokens for a 30-second input. The Whisper encoder has a maximum processing duration of $t_{\text{max}}=30$ seconds.
The rank $r$ and scaling factor $\alpha$ of LoRA are set to 256 and 2.0, respectively. During training, the visual encoder, visual aligner, audio encoder, and LLM remain frozen.

### 4.2 Data and Training Specifications

During the audio modality alignment stage, LibriSpeech-960h (Panayotov et al., ) and AudioCaps (Kim et al., ) are used to train the audio aligner. LibriSpeech-960h is utilized for speech recognition training, while AudioCaps is employed for audio captioning training.

In the audio-visual SFT stage, experiments are conducted using an internal video dataset that will be released upon acceptance. A total of 13k videos with rich audio information are automatically selected, and high-quality audio-visual captions are regenerated with the assistance of GPT-4o (OpenAI et al., ), Whisper-Large-v3 (Radford et al., ), and SALMONN (Tang et al., ). The detailed pipeline is described in Appendix C. Additionally, to further enhance the quality of the SFT data, around 1.5k video captions were manually refined.

In the RL stage, two kinds of tasks are studied: global captioning for the whole video and local captioning for a given time interval. Before each round, a pair of captions for both global and local captioning are sampled from the model for each video in SFT data, respectively.
We consider the information missing rate, hallucination rate, and repetition rate to determine whether a sample pair is suitable for DPO and determine the chosen and rejected samples if yes. The selecting methods for each round are listed in Appendix F.

After mrDPO, the language abilities of the model are reduced. We stop RL training once significant degradation in language abilities is detected. The checkpoint of the last DPO round is used to label captions of a large number of videos. Unnatural captions are eliminated, and the remaining high-quality captions form the training data for the rebirth tuning stage.

For the test dataset, we curated a video captioning benchmark to evaluate the event missing rate (Miss), hallucination rate (Hall), and text repetition rate (Rep). Details of the test data and evaluation process can be found in Appendix D and Appendix E, respectively. The benchmark consists of 483 carefully selected videos, each labelled with complete audio-visual captions by human annotators. Atomic events for the test dataset were initially obtained using GPT-4o and then manually refined. For local captioning, we used Gemini-1.5-Pro (Team et al., ) to tag the start and end times of each event within specific time intervals. Since Gemini could not process some videos, only 457 videos were used for the local captioning evaluation.

Regarding training settings, we conducted audio modality alignment using 8$\times$A100 GPUs for 35k steps and audio-visual SFT using 16$\times$A100 GPUs for 4 epochs. Each RL round was trained with 64$\times$A100 GPUs for 1k steps. After six rounds of mrDPO training, rebirth tuning was performed. During the rebirth tuning stage, we used 64$\times$A100 GPUs and trained for 4 epochs. The batch size per device was set to 1, making the total batch size equal to the number of devices. The weight $\lambda$ in Eqn. (11) was set to 0.1 for all related experiments. The final video-SALMONN 2 model was obtained after one round of gDPO training following rebirth tuning.

## 5 Experimental Results

### 5.1 Overall Results

The results of our video captioning benchmarks are presented in Table 1. video-SALMONN 2 outperforms other models in both information missing and hallucination rates for global and local captioning. Among existing open-source multimodal LLMs, few can provide detailed and accurate video descriptions, whether purely visual models like Video-LLaVA (Lin et al., ) and VILA (Lin et al., ), or audio-visual models like Video-LLaMA 2 (Cheng et al., ) and video-SALMONN (Sun et al., ). Notably, many open-source models, such as Video-LLaVA and Video-LLaMA 2, tend to generate shorter captions, leading to high information missing rates but low hallucination rates.
GPT-4o and Gemini-1.5-Pro can generate more detailed captions and are of higher quality than current open-source models. However, the purely visual version of GPT-4o lacks audio comprehension, and Gemini’s understanding of visual content is somewhat limited, resulting in both models exhibiting some degree of information missing and hallucination.

Our visual base model, trained on a large dataset of images and silent videos, is capable of generating detailed text descriptions based solely on visual information, with a relatively low information missing rate. However, generating longer texts leads to a higher hallucination rate. After audio modality alignment and audio-visual SFT, the model can leverage audio content to reduce both information loss and hallucinations in its descriptions. However, the inclusion of audio tokens may confuse the visual LLM, resulting in frequent repetition during decoding. Building on the SFT model, we applied mrDPT and rebirth tuning, achieving approximately a 35% absolute reduction in the repetition rate and absolute reductions of around 40% and 20% in the total error rate for global and local captioning, respectively. The final video-SALMONN 2 model outperforms some commercial models like GPT-4o and Gemini-1.5-pro in video captioning. As an audio-visual LLM, video-SALMONN 2 retains strong visual understanding capabilities and performs well on various visual benchmarks, such as Video-MME (Fu et al., ). For more details, refer to Appendix G.

**Table 1: Results of our benchmark for detailed video captioning evaluation. “A” and “V” refer to the audio and visual modalities respectively. The repetition rate (Rep), event missing rate (Miss), hallucination rate (Hall), and total error rate (Total = Miss + Hall) are assessed for the captions. video-SALMONN 2, which undergoes an additional round of gDPO after rebirth tuning, achieves the best performance in both global and local captioning, with the lowest total error rates.**
| Model | Modality | Global | Local |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| %Rep$\downarrow$ | %Miss$\downarrow$ | %Hall$\downarrow$ | %Total$\downarrow$ | %Miss$\downarrow$ | %Hall$\downarrow$ | %Total$\downarrow$ |  |  |
| GPT-4o Visual | V | 3.6 | 16.6 | 17.2 | 33.8 | 35.3 | 30.7 | 66.0 |
| Gemini-1.5-Pro | A + V | 1.3 | 21.8 | 16.5 | 38.3 | 36.9 | 17.2 | 54.1 |
| 7B Video-LLaVA | V | 13.2 | 65.3 | 5.4 | 70.7 | 59.1 | 9.4 | 68.5 |
| 8B VILA | V | 4.5 | 39.3 | 18.6 | 57.9 | 47.9 | 23.4 | 71.2 |
| 7B Video-LLaMA 2 | A + V | 5.7 | 56.8 | 8.9 | 65.7 | 47.6 | 14.3 | 61.9 |
| 13B video-SALMONN | A + V | 1.2 | 52.1 | 26.6 | 78.7 | 47.8 | 40.7 | 88.4 |
| 7B Ours-Visual Base | V | 11.8 | 29.8 | 30.0 | 59.7 | 36.1 | 46.1 | 82.2 |
| 7B Ours-SFT | A + V | 36.0 | 26.7 | 26.9 | 53.6 | 30.8 | 33.3 | 64.0 |
| 7B video-SALMONN 2 | A + V | 1.4 | 6.9 | 6.8 | 13.7 | 22.2 | 21.4 | 43.6 |

### 5.2 Analysis of Multi-Round Reinforcement Learning

This section explores various approaches to training in mrDPO. In terms of loss functions, we compared the standard DPO loss, the proposed gDPO loss (which includes the regularisation term based on the ground-truth captions), and a similar loss referred to as “cDPO”, which is the sum of DPO loss and cross-entropy loss on chosen samples. Additionally, the LoRA proxy is evaluated against directly tuning the model’s original LoRA, referred to as “Direct”. Fig. 4(a) presents the total error rates for global video captioning. Training is halted when unnatural captions begin to appear frequently. Examples of such cases are provided in Appendix I.

Figure: (a) Comparison of different mrDPO settings.
Refer to caption: x4.png

Among the three loss functions, DPO shows the fastest improvement in captioning metrics, but unfortunately, it also quickly leads to outputs with frequent unnatural patterns. This is likely because the model only sees self-generated labels rather than ground-truth labels. cDPO faces the same issue but performs worse than DPO. By incorporating loss on ground-truth labels, gDPO makes the training process more stable, allowing the model to generate text responses without unnatural patterns for a longer period of training across multiple RL rounds. This stability also preserves the model’s potential for further improvement, with a significant drop in the error rate observed after three rounds of mrDPO.

The LoRA proxy that randomly initializes a new LoRA module in each RL round, is found to be more beneficial for mrDPO performance compared to directly training the same LoRA, especially after over 4 gDPO rounds. This reveals that the regularisation effect introduced by the LoRA proxies helps to alleviate over-fitting.
Since gDPO with LoRA proxy performs the best for mrDPO, we use the model after six gDPO rounds using LoRA proxy to generate captions for a large number of videos. After excluding unnatural patterns, a total of 180k video captions remain for rebirth tuning.

### 5.3 Analysis of Rebirth Tuning

**Table 2: The appearance rate of unnatural caption for global. We detect specific patterns that are viewed as unnatural and count the frequency of occurrence of specific patterns on the test set over mrDPO using gDPO with LoRA proxy. The results prove that mrDPO leads to a significant increase in the appearance of unnatural captions, especially in the final converging rounds.**
| Stage | SFT | #Rounds of mrDPO | Rebirth Tuning |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |  |  |  |
| %Unnatural Rate$\downarrow$ | 0.0 | 0.0 | 0.0 | 1.9 | 0.9 | 2.9 | 2.1 | 12.0 | 0.0 |

While multiple rounds of mrDPO with LoRA proxy significantly improve video captioning performance, they also lead to an increasing frequency of unnatural patterns in text responses. Table 2 shows the occurrence rate of these unnatural patterns in global captioning after each training stage. Through rebirth tuning, the backbone LLM discards the LoRA proxies and restores its ability to generate fluent captions. Additionally, the careful selection of rebirth-tuning data enhances data quality, ensuring the model is fine-tuned with superior data, further boosting its overall performance.

Another notable effect of rebirth tuning is to sustain continued training. As shown in Fig. 4(a), in the later rounds of mrDPO, the model starts to gain less with each round and eventually converges. The decline in the ability to generate fluent text responses is also more likely to occur in these later rounds, suggesting that the model has fallen into a local minimum after mrDPO.
However, after the rebirth tuning stage, where only teacher-forcing training is applied, the model escapes the local optimum from previous training and becomes receptive to further optimization with RL. Fig. 4(b) compares gDPO after rebirth tuning with six rounds of gDPO with LoRA proxy. It is observed that only minimal improvement can be achieved after sufficient RL rounds in terms of the total error rate for global captioning, while an extra RL stage following rebirth tuning yields significant performance gains once again.
This suggests the potential of iterating mrDPO and rebirth tuning.

## 6 Conclusions

This work introduces video-SALMONN 2, a powerful audio-visual LLM designed for detailed video captioning, and proposes the mrDPO method. To our knowledge, this is the first study of applying RL to audio-visual LLMs in literature. New metrics are designed to evaluate the information missing and hallucination rates in video captions, which are used to guide sample selection for DPO. To further stabilize training, the setting with novel gDPO and LoRA proxy is introduced. After mrDPO, we propose a novel rebirth tuning method to restore LLM’s performance on non-captioning tasks. As a result, video-SALMONN 2 demonstrates significant improvements in video captioning, outperforming notable models such as GPT-4o and Gemini-1.5-Pro, and setting a promising direction for achieving detailed and accurate video captioning for video understanding.

## Appendix A Pipeline for Getting Atomic Events in a Time Interval

Gemini-1.5-Pro is used to obtain the atomic events in some time intervals. We input the video and all its atomic events labelled by GPT to Gemini and ask it to tag the beginning and start time of each atomic event. Events are then selected if their time intervals overlap with the given time interval.
We have checked this process to confirm that the atomic events obtained for the given time interval are roughly accurate.

## Appendix B Calculation Procedure of The Text Repetition Rate

The procedure to calculate the repetition rate of a long and detailed text is shown as follows:

- 1.
Split the text into short phrases by punctuation;
- 2.
Counting the number of occurrences of each phrase;
- 3.
The number of recurring phrase words divided by the total number of words in the text is the repetition rate.

## Appendix C Pipeline for Labelling High-quality Audio-visual Captions

To curate training data for supervised fine-tuning, we employ GPT-4o to label the visual content in each frame, while SALMONN-13B and Whisper-Large-v3 are used to annotate the speech content and audio events in the audio track. This process is illustrated in Figure 5. Our initial aim is to automatically filter out videos that contain limited speech. We begin by slicing each video into 10-second segments, with the audio from each segment analyzed by SALMONN to generate automatic audio captions (AAC). These captions help us filter out videos that lack descriptive speech such as ”A man is speaking” or ”A woman says…”. This initial filtering step is somewhat coarse.

Next, the audio from each segment is processed by Whisper to produce automatic speech recognition (ASR) results. If the transcribed text is too brief or nonsensical, the corresponding video is deemed to lack rich audio content and is excluded from further consideration. For a video to be labelled, all of its segments should pass this exclusion criterion.

The segments from the remaining videos are then sampled at a rate of 1 fps and fed into GPT-4o to extract segment-level visual captions. Ultimately, the segment-level visual captions, AAC results, and ASR results form the input to GPT-4o concurrently to generate a detailed global audio-visual caption for each video.

Figure: Figure 5: The pipeline for labelling videos with high-quality audio-visual captions.
Refer to caption: x6.png

## Appendix D About The Test Dataset

Figure 6 shows the basic information of our caption benchmark. The benchmark covers 14 different fields. All the videos are between 30s to 60s, with an average duration of 51s.

Figure: (a) Video type distribution.
Refer to caption: x7.png

## Appendix E Process of Evaluating Detailed Captions

To evaluate the specific video caption generated by our model, we first use GPT-3.5 or GPT-4o to split the labelled caption of this video into several atomic events, where we use GPT-4o for the test set and GPT-3.5 for the RL training set. Then, the list of atomic events and the caption to be evaluated are simultaneously fed into GPT-3.5 to determine what events in the atomic event list are missed and what events in the caption are hallucinated.
Specifically, we ask GPT-3.5 to list all the missing events and hallucination events for better evaluation precision. Note that events that are described incorrectly are also regarded as hallucinations.
The quotient between the number of missing or hallucination events and the number of all events in the video is the final missing or hallucination rate.
For more robust testing, GPT-3.5 is used to evaluate 7 times for each caption and the medium number of the metric is reported. We have manually confirmed that the score GPT-3.5 gives is roughly plausible.

## Appendix F Samples Selecting Methods for RL of Each Round

To achieve better performance and training efficiency, we take a specially designed strategy to select proper preference pairs. A sample pair is selected if one sample is better than the other in all metrics with a threshold. For global captioning, we consider global error rate $\Delta e_{g}$ and global repetition rate $\Delta r_{g}$, while for local captioning we consider local error rate $\Delta e_{t}$ and local repetition rate $\Delta r_{t}$. Table 3 shows the threshold used in each round.

**Table 3: The data selecting threshold used in each RL round. A negative number means that the chosen sample can be worse than the rejected sample in this metric to some degree.**
| RL Round | Threshold Used |  |  |  |
| --- | --- | --- | --- | --- |
| $\Delta e_{g}$ | $\Delta r_{g}$ | $\Delta e_{t}$ | $\Delta r_{t}$ |  |
| 1 | $\geq 5\%$ | $\geq 1\%$ | $\geq 20\%$ | $\geq 1\%$ |
| 2 | $\geq 20\%$ | $\geq-1\%$ | $\geq 45\%$ | $\geq 0$ |
| 3 | $\geq 20\%$ | $\geq-1\%$ | $\geq 45\%$ | $\geq 0$ |
| 4 | $\geq 20\%$ | $\geq-1\%$ | $\geq 45\%$ | $\geq 0$ |
| 5 | $\geq 20\%$ | $\geq-1\%$ | $\geq 45\%$ | $\geq 0$ |
| 6 | $\geq 25\%$ | $\geq-1\%$ | $\geq 45\%$ | $\geq 0$ |
| 7 | $\geq 30\%$ | $\geq-1\%$ | $\geq 45\%$ | $\geq 0$ |

## Appendix G Results on Visual QA Benchmarks

Since QA data is not seen during the mrDPO process, the model’s QA abilities decrease a lot after mrDPO. Thanks to the rebirth tuning on captioning and QA, the non-captioning abilities are able to recover.
After one round of gDPO with a LoRA proxy, video-SALMONN 2 finally achieves detailed and accurate captioning while getting competitive results compared to SOTA models of similar size on QA benchmarks like Video-MME.
Since video-SALMONN 2 cannot support very long audio, we only consider the Video-MME Short set. Table G shows the results of our models, as well as those of SOTA models of 7B and 8B.

[ht]
Accuracy results on Video-MME Short set.
(#Params) Model
#Max Frames
Video-MME Short$\uparrow$
(8B) MiniCPM-V 2.6
64
71.3 ^1
(7B) Long-LLaVA
64
61.9 ^1
(7B) Ours-Visual Base
30
67.2
(7B) Ours-mrDPO
30
65.3
(7B) Ours-Rebirth
30
67.6
(7B) video-SALMONN 2
30
67.0

- 1
[https://video-mme.github.io/home_page.html#leaderboard](https://video-mme.github.io/home_page.html#leaderboard).

## Appendix H Video Captioning Cases of video-SALMONN 2

Some video captioning cases generated by video-SALMONN 2 are shown in Figure 7 and Figure 8. More demos can be found at [https://video-salmonn-2.github.io](https://video-salmonn-2.github.io).

Figure: Figure 7: Case I generated by video-SALMONN 2.
Refer to caption: x9.png

Figure: Figure 8: Case II generated by video-SALMONN 2.
Refer to caption: x10.png

## Appendix I Cases of Unnatural Responses

Unnatural responses might be generated after multiple RL rounds.
Figure 9 shows a caption generated by the model after 6 gDPO rounds with LoRA proxies, which contains some repeated patterns that make the caption unnatural. Figure 10 shows a caption generated by the model after 5 classical DPO rounds with LoRA proxies, which includes some strange characters and sentences.

Figure: Figure 9: A bad caption generated by the model after 6 gDPO rounds with LoRA proxies, which contains a lot of patterns like “#Video Description” and “#Audio:” that make the caption unnatural.
Refer to caption: x11.png

Figure: Figure 10: A bad caption generated by the model after 5 classical DPO rounds with LoRA proxies, which includes strange characters and sentences, like the first sentence in this generated text.
Refer to caption: x12.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: - Could the authors include more diverse evaluation tasks to assess the impact of rebirth tuning on non-captioning skills? For instance, evaluating on tasks like video question-answering at each stage of training could provide more insights into how rebirth tuning preserves or enhances the model’s broader language abilities.

- Considering the reliance on GPT-3.5 and GPT-4 for assessing metrics like hallucination rates, how have the authors addressed potential biases from these models in their evaluation pipeline?

- Could the authors clarify if there are plans to evaluate video-SALMONN 2 on open, widely-used datasets for multimodal captioning? Including results on open datasets or releasing parts of the internal test set would enhance the model's credibility. It would also allow for broader validation by the research community, helping others replicate and build upon the findings.

- Can the authors provide a detailed analysis of the impact of individual components of mrDPO, like LoRA updates, regularization terms, and the multi-round structure, on the overall performance? How does each element contribute uniquely to the training effectiveness?

- Given that mrDPO involves multiple LoRA proxies and additional training rounds, can the authors quantify the GPU memory, runtime, and computational load associated with each round? Is there an observed point where additional rounds yield diminishing returns? Common metrics are FLOPS, MACs, Latency and so on. 

- While the model is designed for audio-visual input, the paper lacks qualitative insight into its success rate in synchronizing audio and visual elements. Could the authors provide examples or failure cases?
soundness: 3
strengths: - The introduction of video-SALMONN 2 with its emphasis on synchronized audio-visual inputs for captioning represents a significant advancement. By integrating both visual and audio data, the model moves beyond purely visual or audio-focused LLMs, capturing richer and more contextually complete video descriptions. The novel use of multi-round Directed Preference Optimization (mrDPO) as a reinforcement learning technique to improve captioning quality is a noteworthy approach.

- The paper demonstrates high-quality experimental validation, showcasing the effectiveness of mrDPO and rebirth tuning. Through detailed metrics like missing information, hallucination, and repetition rates, the authors provide quantitative evidence of video-SALMONN 2’s superiority over leading commercial models like GPT-4o and Gemini-1.5-Pro in video captioning tasks. The thorough benchmarking against these models on multiple metrics adds robustness to the study’s findings.

- The paper is clear and methodically structured, making complex techniques like multi-round preference optimization accessible. Each step in the methodology is well-illustrated, from model architecture through training strategies, with detailed explanations of mrDPO and rebirth tuning. The clarity of the evaluation pipeline and the metrics for assessing captioning quality further aids in understanding the model's improvements.
summary: - The paper introduces video-SALMONN 2, an advanced audio-visual large language model (LLM) for generating detailed and accurate video captions, surpassing models like GPT-4o and Gemini-1.5-Pro on this task.
  
- A new evaluation pipeline is presented, with metrics specifically designed to quantify missing information, hallucination, and repetition rates in audio-visual captions, enhancing the precision of video content descriptions.

- Multi-round Directed Preference Optimization (mrDPO) is developed to improve caption accuracy through reinforcement learning, periodically updating a reference model and using low-rank adaptation (LoRA) for smoother and more effective model training.

- Rebirth tuning is introduced to prevent performance degradation in non-captioning tasks post-mrDPO, using supervised fine-tuning on self-labeled data to maintain both captioning quality and broader LLM capabilities.
weaknesses: - The paper introduces rebirth tuning to help the model retain non-captioning abilities, but the evaluation of this feature feels limited. Expanding the assessment to include the model’s performance on non-captioning tasks, like video question-answering, at each training stage could give a more comprehensive picture of its capabilities. This approach would better demonstrate how effectively rebirth tuning preserves broader language skills.

- The evaluation pipeline depends heavily on other large language models, such as GPT-3.5 and GPT-4, to assess metrics like missing information and hallucination rates. This reliance raises concerns about objectivity and reproducibility, as these models come with their own biases. Developing an alternative, self-contained evaluation method or comparing LLM-based assessments with human judgments could strengthen the evaluation’s reliability and transparency.

- The model’s performance evaluation relies mostly on internal datasets, which limits the generalizability and reproducibility of the results. Including open datasets or releasing the test set would allow the research community to more easily validate the findings and enhance the overall credibility of the work.

- The mrDPO technique is key to the model’s effectiveness, but the paper lacks a detailed analysis of its individual components, like LoRA updates, regularization terms, and the multi-round structure. Investigating how each component contributes to the model’s overall performance would provide insights into which aspects are most beneficial, potentially guiding future improvements and applications of mrDPO.

- The training strategy, particularly mrDPO and the use of several LoRA proxies, adds computational overhead without a clear analysis of the associated trade-offs. Quantifying the specific GPU memory, runtime, and computational load required for each training round would clarify the model’s resource demands. Additionally, identifying the point at which diminishing returns begin in mrDPO could shed light on the efficiency of each round, highlighting which may be resource-intensive with limited added benefit. Exploring alternative configurations for resource-limited environments and comparing high-resource to optimized setups would offer insights into how performance scales in different compute settings. This analysis would clarify the balance between computational cost and performance improvements, making the model more adaptable and accessible for varied research needs.

- Although the model is designed to process both audio and visual inputs, the paper provides minimal qualitative insight into how well it aligns these features in its captions. Including examples of successful synchronizations as well as cases where alignment fails would offer a clearer view of the model’s multi-modal understanding abilities.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to the weaknesses part.

Overall, as mentioned in the weaknesses part I think the paper needs large revision at this point. First, the methods part can be written in a better way to make readers understand the technical details and contributions of this paper more easily. Also, more experiments and analysis would make the proposed methods more convincing.
soundness: 2
strengths: 1. video-SALMONN shows competitive performance on multiple metrics on the custom dataset for evaluation.
2. The goal this paper is aiming on is a promising direction.
summary: In this paper, the authors propose a VideoLLM named video-SALMONN2, which is capable of understanding both the video and audio modality. To train the model, a training strategy with multi-round DPO (mrDPO) and rebirth tuning is applied. mrDPO iteratively performs DPO training for multiple steps, while rebirth tuning improves the original model with captions generated from the model updated with mrDPO. The performance is evaluated on the custom dataset, in terms of repetition and event missing rate.
weaknesses: 1. Lots of details about the method seem to be missing or not clearly stated, making it hard to understand the method.
    1. Training losses are not detailed. It can be inferred that for the audio alignment stage, speech recognition and audio training losses are applied, for the SFT stage (L231-238), video captioning loss is applied, and for the DPO stages, a DPO loss is applied. Still, it would be explicitly mentioned which loss is applied for each stage, as it was very confusing, especially in the case of the audio alignment stage, where the information about training loss is simply mentioned in L376-377. Also, since there exist multiple variants of DPO, it would be better to include the exact formulation of DPO in the equation.
    2. How are the captions for DPO sampled? It is only stated as “First, distinct video captions are sampled from the model’s output distribution, given the
    input video” in L268.
    3. What are the definitions of ‘basic atomic events’ (L289), ‘information missing and hallucination rates’ (L292), ‘global captions’ (L269), and ‘local captions’ (L293)?
    4. How is a problematic pattern detected when building training data for rebirth training, which is mentioned in L351?
    5. It seems that the final model (video-SALMONN) is a model trained with rebirth training with data generated with mrDPO, and is further trained with a single round of gDPO (L407), which is not mentioned in the method section. It would be better to state the final model in the methods sections for better understanding.
2. The advantage of mrDPO is confusing. 
    1. If training with mrDPO results in “the model gradually begins to produce repetitive and unnatural text patterns in its responses” (L337), why is it required? 
    2. Such claim in 2-(a) is also contradictory with the statement in L349, which says the final mrDPO model “excels at generating complete and accurate video descriptions”. 
    3. Also, are there experimental results that back up the claim that the mrDPO model “demonstrates significant improvements in captioning” (L335)?
    4. Results in Fig.4-(a) seem direct-DPO with single-round DPO shows the lowest total error rate, which makes the advantage of mrDPO more doubtful. Authors claim that the dashed line indicates a ‘high frequency of unnatural text generation’, but how are they determined as so?
3. Missing experiments and analysis
    1. Results on more conventional datasets are required. Current evaluations are only done on custom datasets and a Video-MME short set, where the method is compared only with a few baselines, extensive comparison on more conventional benchmarks with competitive baselines is required.
    2. The motivation for using GPT-based ‘missing’ and ‘hallucinatory’ events as metrics is not clear. Ablation results on metrics, comparing proposed metrics and conventional captioning metrics comparing generation results with ground-truth captions can make the design choice more convincing.
    3. Some qualitative examples showcasing outputs of an initial model, a model trained with mrDPO, and the final model would help understand the actual behavior of each model and the effect of each component.
4. Incorporating audio modality into VideoLLMs has already been proposed [1, 2, 3], and the proposed model seems to have no significant difference compared to them in terms of audio inclusion.

**References**

[1] Yang et al., Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, CVPR 2023

[2] Wang et al., InternVideo2: Scaling Foundation Models for Multimodal Video Understanding, ECCV 2024

[3] Zhang, Li, and Bing, Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding, EMNLP 2023 (demo track)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. for video QA evaluation，could authors provide result on the Video-MME Medium and Long Video track? and also ablation results on other video understanding benchmark, such as mvbench, videovista, MLVU, to make the comparison more transparent and persuasive?
2. for video captioning evaluation, could authors provide more comparison results on  public benchmark, such as MSR-VTT and Vatex？
3. will author plan to publicly release the internal benchmark?
soundness: 2
strengths: 1.the proposed novel pipeline including mrDPO, LoRa proxy and rebirth, is detailedly introduced, and experiment results on each iteration are well presented.  
2.good ablation experiments are provided on each mrDPO iteration, LoRa proxy, and rebirth. And the experiments provide good observation on the DPO' effect on the model performance.
3.the whole pipeline is interesting and inspirational.
summary: This paper introduces video-SALMONN 2, a powerful audio-visual large language model (LLM) designed for detailed and accurate video captioning. The model is built upon a pre-trained visual LLM and is further enhanced with auditory capabilities by training on audio-only data and videos with synchronized audio tracks. To accurately assess the model's performance, new metrics are proposed to evaluate the completeness and accuracy of video descriptions, which are then optimized using a novel multi-round direct preference optimization (mrDPO) approach. The mrDPO process involves periodically updating the DPO reference model, merging and re-initializing the low-rank adaptation (LoRA) module, and incorporating guidance from ground-truth video captions to stabilize the training. To address potential catastrophic forgetting of non-captioning abilities, a rebirth tuning process is introduced, which fine-tunes the pre-DPO LLM using the captions generated by the mrDPO-trained model as supervised labels.
weaknesses: 1.the model performance improvement is mainly evaluated on the internal benchmarks, and the quality of the internal benchmark is unknown.
2. video-mllms could not only do recgnition and captioning of a video, but also exploite the knowledge of LLM to do further recognition reseasoning. However, in paper, only one comparison with public benchmark "Video-MME short video" is provided. So it is not sure the proposed method could be beneficial for comprehensive improvements of a video-mllm.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please revise the Strengths and Weaknesses sections point by point. This is a paper with great potential. If the authors can provide additional responses to certain issues, discuss related work more thoroughly, and include more experiments and observations, I would be very happy to raise my score.

Additionally, although this is a concurrent work (and the authors can choose not to compare, as it's not a mandatory requirement), I would be very interested to see the model's performance on the VDC benchmark [1]. This would help me make a final decision. Again, this is completely optional.

[1] https://huggingface.co/datasets/wchai/Video-Detailed-Caption
soundness: 3
strengths: 1. Audio is very important in understanding video, and incorporating it is a great idea.
2. The approach of using DPO (Direct Preference Optimization) and LoRA (Low-Rank Adaptation) to enhance specific capabilities with limited data is intuitive and effective.
3. Detailed video captioning is an excellent task, although it is not novel. It's great to see that people are starting to pay attention to and evaluate this task.
4. The event missing rate (Miss), hallucination rate (Hall), and text repetition rate (Rep) are introduced as the new metric for evaluation the detailed video captioning tasks.
5. Dividing the tasks into global and local parts is very beneficial. A recent work, MovieChat [1] also made a similar categorization (for video QA). Perhaps the authors could discuss and compare this in future versions.

[1] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
summary: This paper presents a large multimodal model (audio-visual) for video captioning task. multi-round DPO and LoRA is used to train the model. Experiments look good, showing that the training recipe enhances video-SALMONN 2’s captioning accuracy. Besides, this paper introduce an evaluation pipeline that computes the missing and hallucination rates of audiovisual events in video captions using text-based LLMs, breaking down the process into sub-tasks suited for current LLMs and a new benchmark for video captioning with a human-annotated test set.
weaknesses: 1. The writing and illustrations in the paper need improvement, especially the figures, which severely hinder the proper understanding of the article.
2. Since DPO was used for training, based on my experience, DPO does not offer as stable training conditions as SFT. I hope the authors can present the specific parameters and training curves for DPO to further enhance reproducibility.
3. Although the authors present LoRA as one of the main contributions, there is no ablation study to demonstrate how LoRA prevents catastrophic forgetting or performance drop.
4. I believe that although the automated metrics are good (and perhaps better than metrics like CIDEr), the new metrics still need to align with human perception. I suggest that the authors refer to a concurrent work, AuroraCap [1], where they first use human Elo to score each model and then calculate the correlation between human Elo and the proposed metrics in the paper. This would make the benchmark more convincing.
5. There are too few details about the proposed benchmark. I only found statistics on video types and lengths in the supplementary material, but there is no clear information regarding caption length (word count, noun count) or the number of video sources.

[1] Chai, Wenhao, et al. "AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark." arXiv preprint arXiv:2410.03051 (2024).

Meta-review: # Summary and Recommendation for Rejection

---

## Strengths:
1. **Innovative Training Techniques**:
   - Introduces **multi-round Directed Preference Optimization (mrDPO)** and **rebirth tuning** for optimizing detailed and accurate video captioning.
   - mrDPO stabilizes training through iterative updates and LoRA proxies, while rebirth tuning mitigates catastrophic forgetting in non-captioning tasks.

2. **Advanced Model Capabilities**:
   - The proposed model, **video-SALMONN 2**, significantly improves video captioning tasks, surpassing GPT-4o and Gemini-1.5-Pro in custom benchmarks.
   - Synchronized audio-visual inputs enhance the model's multimodal understanding, producing richer and more contextually accurate descriptions.

3. **Novel Evaluation Metrics**:
   - Proposes new metrics for video captioning, including missing and hallucination rates.

4. **Comprehensive Experimental Validation**:
   - Demonstrates significant reductions in error and repetition rates, supported by thorough ablation studies on mrDPO and LoRA proxies.

5. **Potential Community Impact**:
   - The authors commit to releasing datasets, code, and model checkpoints, fostering transparency and future research opportunities.

---

## Weaknesses:
1. **Limited External Benchmarking**:
   - Evaluation is predominantly on internal datasets, with limited testing on public benchmarks (e.g., VideoMME Short, VDC). This raises concerns about the generalizability of the results.

2. **High Computational Costs**:
   - mrDPO’s iterative training process requires significant computational resources, limiting its accessibility to researchers with fewer resources.
   - Insufficient quantitative analysis of the trade-offs between performance gains and computational overhead.

3. **Dependence on Other Models for Evaluation**:
   - Metrics like hallucination and missing rates rely on GPT-3.5/4 for evaluation, introducing potential biases and reproducibility challenges.

4. **Unclear Advantage of mrDPO**:
   - Although mrDPO reduces error rates, it introduces issues like unnatural text patterns, which require additional fine-tuning. The advantage over single-round DPO remains unclear due to limited direct comparisons.

5. **Incomplete Presentation**:
   - Methodological details are not fully explained, such as specific loss functions, definitions of key terms, and the process of sampling captions.
   - Missing qualitative examples that showcase the incremental improvements of mrDPO and rebirth tuning.

---

## Authors’ Mitigation:
1. **Expanded Benchmarks**:
   - Added evaluations on new video QA datasets (e.g., NeXT-QA, MVBench, VideoVista) and public video captioning benchmarks (e.g., VDC).

2. **Improved Clarity**:
   - Revised the manuscript to clarify methodology, including detailed explanations of training losses, caption sampling, and mrDPO dynamics.

3. **Addressed Computational Concerns**:
   - Quantified computational costs for data generation and training, explaining methods to reduce overhead using merging techniques for LoRA proxies.

4. **Evaluated Metrics**:
   - Justified the reliability of new metrics through ELO tests and manual validations, addressing concerns about potential biases.

---

## Remaining Weaknesses:
1. **Generality and Scalability**:
   - Broader validation on open benchmarks like MSR-VTT and VATEX remains missing, hindering general applicability.
   - Computational demands of mrDPO make the approach less feasible for researchers with limited resources.

2. **Evaluation Pipeline Limitations**:
   - Heavy reliance on external LLMs (GPT-3.5/4) for evaluation raises concerns about reproducibility and objectivity.

3. **Incomplete Analysis**:
   - The paper lacks a detailed breakdown of individual components of mrDPO and their specific contributions to performance improvements.

4. **Presentation Issues**:
   - Despite revisions, some sections still lack sufficient detail, limiting full reproducibility.

---

## Recommendation:  **Reject**

### **Justification**:
1. **Evaluation Shortcomings**:
   - Over-reliance on internal datasets and limited external benchmarking weaken the paper’s generalizability and broader applicability.

2. **High Computational Overhead**:
   - The computational demands of mrDPO, combined with insufficient analysis of trade-offs, limit the practicality of the proposed approach.

3. **Insufficient Clarity and Reproducibility**:
   - Key methodological details remain unclear, and dependence on external LLMs for evaluation introduces significant reproducibility concerns.

4. **Potential but Unproven Impact**:
   - While promising, the paper does not yet demonstrate sufficient breadth or robustness to meet the standards of a top-tier conference like ICLR.

The paper would benefit from more comprehensive external benchmarking, improved computational efficiency analyses, and clearer methodological explanations to better establish its contributions.

Decision: Reject