Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization

Authors: Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun MA, Chao Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multi-modal large language models, video captioning, multi-round DPO, rebirth tuning

Abstract:
Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding. In this paper, we present video-SALMONN 2, an advanced audio-visual large language model (LLM) with low-rank adaptation (LoRA) designed for enhanced video (with paired audio) captioning through directed preference optimization (DPO). We propose new metrics to evaluate the completeness and accuracy of video descriptions, which are optimized using DPO. To further improve training, we introduce a novel multi-round DPO (mrDPO) approach, which involves periodically updating the DPO reference model, merging and re-initializing the LoRA module as a proxy for parameter updates after each training round (1,000 steps), and incorporating guidance from ground-truth video captions to stabilize the process. To address potential catastrophic forgetting of non-captioning abilities due to mrDPO, we propose rebirth tuning, which finetunes the pre-DPO LLM by using the captions generated by the mrDPO-trained model as supervised labels. Experiments show that mrDPO significantly enhances video-SALMONN 2's captioning accuracy, reducing global and local error rates by 40\% and 20\%, respectively, while decreasing the repetition rate by 35\%. The final video-SALMONN 2 model, with just 7 billion parameters, surpasses leading models such as GPT-4o and Gemini-1.5-Pro in video captioning tasks, while maintaining competitive performance to the state-of-the-art on widely used video question-answering benchmark among models of similar size. Upon acceptance, we will release the code, model checkpoints, and training and test data. Demos are available at https://video-salmonn-2.github.io.

Decision: reject
Reviewer scores: 3, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1729720696047

thread_type:
official_review

<thread_object>
creator: reviewer_aygd

<content>
summary:
This paper presents a large multimodal model (audio-visual) for video captioning task. multi-round DPO and LoRA is used to train the model. Experiments look good, showing that the training recipe enhances video-SALMONN 2’s captioning accuracy. Besides, this paper introduce an evaluation pipeline that computes the missing and hallucination rates of audiovisual events in video captions using text-based LLMs, breaking down the process into sub-tasks suited for current LLMs and a new benchmark for video captioning with a human-annotated test set.

strengths:
1. Audio is very important in understanding video, and incorporating it is a great idea.
2. The approach of using DPO (Direct Preference Optimization) and LoRA (Low-Rank Adaptation) to enhance specific capabilities with limited data is intuitive and effective.
3. Detailed video captioning is an excellent task, although it is not novel. It's great to see that people are starting to pay attention to and evaluate this task.
4. The event missing rate (Miss), hallucination rate (Hall), and text repetition rate (Rep) are introduced as the new metric for evaluation the detailed video captioning tasks.
5. Dividing the tasks into global and local parts is very beneficial. A recent work, MovieChat [1] also made a similar categorization (for video QA). Perhaps the authors could discuss and compare this in future versions.

[1] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

weaknesses:
1. The writing and illustrations in the paper need improvement, especially the figures, which severely hinder the proper understanding of the article.
2. Since DPO was used for training, based on my experience, DPO does not offer as stable training conditions as SFT. I hope the authors can present the specific parameters and training curves for DPO to further enhance reproducibility.
3. Although the authors present LoRA as one of the main contributions, there is no ablation study to demonstrate how LoRA prevents catastrophic forgetting or performance drop.
4. I believe that although the automated metrics are good (and perhaps better than metrics like CIDEr), the new metrics still need to align with human perception. I suggest that the authors refer to a concurrent work, AuroraCap [1], where they first use human Elo to score each model and then calculate the correlation between human Elo and the proposed metrics in the paper. This would make the benchmark more convincing.
5. There are too few details about the proposed benchmark. I only found statistics on video types and lengths in the supplementary material, but there is no clear information regarding caption length (word count, noun count) or the number of video sources.

[1] Chai, Wenhao, et al. "AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark." arXiv preprint arXiv:2410.03051 (2024).

questions:
Please revise the Strengths and Weaknesses sections point by point. This is a paper with great potential. If the authors can provide additional responses to certain issues, discuss related work more thoroughly, and include more experiments and observations, I would be very happy to raise my score.

Additionally, although this is a concurrent work (and the authors can choose not to compare, as it's not a mandatory requirement), I would be very interested to see the model's performance on the VDC benchmark [1]. This would help me make a final decision. Again, this is completely optional.

[1] https://huggingface.co/datasets/wchai/Video-Detailed-Caption
</content>
</thread_object>

</thread>
<thread>
creation date:
1730542753107

thread_type:
official_review

<thread_object>
creator: reviewer_7dpv

<content>
summary:
This paper introduces video-SALMONN 2, a powerful audio-visual large language model (LLM) designed for detailed and accurate video captioning. The model is built upon a pre-trained visual LLM and is further enhanced with auditory capabilities by training on audio-only data and videos with synchronized audio tracks. To accurately assess the model's performance, new metrics are proposed to evaluate the completeness and accuracy of video descriptions, which are then optimized using a novel multi-round direct preference optimization (mrDPO) approach. The mrDPO process involves periodically updating the DPO reference model, merging and re-initializing the low-rank adaptation (LoRA) module, and incorporating guidance from ground-truth video captions to stabilize the training. To address potential catastrophic forgetting of non-captioning abilities, a rebirth tuning process is introduced, which fine-tunes the pre-DPO LLM using the captions generated by the mrDPO-trained model as supervised labels.

strengths:
1.the proposed novel pipeline including mrDPO, LoRa proxy and rebirth, is detailedly introduced, and experiment results on each iteration are well presented.  
2.good ablation experiments are provided on each mrDPO iteration, LoRa proxy, and rebirth. And the experiments provide good observation on the DPO' effect on the model performance.
3.the whole pipeline is interesting and inspirational.

weaknesses:
1.the model performance improvement is mainly evaluated on the internal benchmarks, and the quality of the internal benchmark is unknown.
2. video-mllms could not only do recgnition and captioning of a video, but also exploite the knowledge of LLM to do further recognition reseasoning. However, in paper, only one comparison with public benchmark "Video-MME short video" is provided. So it is not sure the proposed method could be beneficial for comprehensive improvements of a video-mllm.

questions:
1. for video QA evaluation，could authors provide result on the Video-MME Medium and Long Video track? and also ablation results on other video understanding benchmark, such as mvbench, videovista, MLVU, to make the comparison more transparent and persuasive?
2. for video captioning evaluation, could authors provide more comparison results on  public benchmark, such as MSR-VTT and Vatex？
3. will author plan to publicly release the internal benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547267538

thread_type:
official_review

<thread_object>
creator: reviewer_meee

<content>
summary:
In this paper, the authors propose a VideoLLM named video-SALMONN2, which is capable of understanding both the video and audio modality. To train the model, a training strategy with multi-round DPO (mrDPO) and rebirth tuning is applied. mrDPO iteratively performs DPO training for multiple steps, while rebirth tuning improves the original model with captions generated from the model updated with mrDPO. The performance is evaluated on the custom dataset, in terms of repetition and event missing rate.

strengths:
1. video-SALMONN shows competitive performance on multiple metrics on the custom dataset for evaluation.
2. The goal this paper is aiming on is a promising direction.

weaknesses:
1. Lots of details about the method seem to be missing or not clearly stated, making it hard to understand the method.
    1. Training losses are not detailed. It can be inferred that for the audio alignment stage, speech recognition and audio training losses are applied, for the SFT stage (L231-238), video captioning loss is applied, and for the DPO stages, a DPO loss is applied. Still, it would be explicitly mentioned which loss is applied for each stage, as it was very confusing, especially in the case of the audio alignment stage, where the information about training loss is simply mentioned in L376-377. Also, since there exist multiple variants of DPO, it would be better to include the exact formulation of DPO in the equation.
    2. How are the captions for DPO sampled? It is only stated as “First, distinct video captions are sampled from the model’s output distribution, given the
    input video” in L268.
    3. What are the definitions of ‘basic atomic events’ (L289), ‘information missing and hallucination rates’ (L292), ‘global captions’ (L269), and ‘local captions’ (L293)?
    4. How is a problematic pattern detected when building training data for rebirth training, which is mentioned in L351?
    5. It seems that the final model (video-SALMONN) is a model trained with rebirth training with data generated with mrDPO, and is further trained with a single round of gDPO (L407), which is not mentioned in the method section. It would be better to state the final model in the methods sections for better understanding.
2. The advantage of mrDPO is confusing. 
    1. If training with mrDPO results in “the model gradually begins to produce repetitive and unnatural text patterns in its responses” (L337), why is it required? 
    2. Such claim in 2-(a) is also contradictory with the statement in L349, which says the final mrDPO model “excels at generating complete and accurate video descriptions”. 
    3. Also, are there experimental results that back up the claim that the mrDPO model “demonstrates significant improvements in captioning” (L335)?
    4. Results in Fig.4-(a) seem direct-DPO with single-round DPO shows the lowest total error rate, which makes the advantage of mrDPO more doubtful. Authors claim that the dashed line indicates a ‘high frequency of unnatural text generation’, but how are they determined as so?
3. Missing experiments and analysis
    1. Results on more conventional datasets are required. Current evaluations are only done on custom datasets and a Video-MME short set, where the method is compared only with a few baselines, extensive comparison on more conventional benchmarks with competitive baselines is required.
    2. The motivation for using GPT-based ‘missing’ and ‘hallucinatory’ events as metrics is not clear. Ablation results on metrics, comparing proposed metrics and conventional captioning metrics comparing generation results with ground-truth captions can make the design choice more convincing.
    3. Some qualitative examples showcasing outputs of an initial model, a model trained with mrDPO, and the final model would help understand the actual behavior of each model and the effect of each component.
4. Incorporating audio modality into VideoLLMs has already been proposed [1, 2, 3], and the proposed model seems to have no significant difference compared to them in terms of audio inclusion.

**References**

[1] Yang et al., Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, CVPR 2023

[2] Wang et al., InternVideo2: Scaling Foundation Models for Multimodal Video Understanding, ECCV 2024

[3] Zhang, Li, and Bing, Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding, EMNLP 2023 (demo track)

questions:
Please refer to the weaknesses part.

Overall, as mentioned in the weaknesses part I think the paper needs large revision at this point. First, the methods part can be written in a better way to make readers understand the technical details and contributions of this paper more easily. Also, more experiments and analysis would make the proposed methods more convincing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685822424

thread_type:
official_review

<thread_object>
creator: reviewer_zukj

<content>
summary:
- The paper introduces video-SALMONN 2, an advanced audio-visual large language model (LLM) for generating detailed and accurate video captions, surpassing models like GPT-4o and Gemini-1.5-Pro on this task.
  
- A new evaluation pipeline is presented, with metrics specifically designed to quantify missing information, hallucination, and repetition rates in audio-visual captions, enhancing the precision of video content descriptions.

- Multi-round Directed Preference Optimization (mrDPO) is developed to improve caption accuracy through reinforcement learning, periodically updating a reference model and using low-rank adaptation (LoRA) for smoother and more effective model training.

- Rebirth tuning is introduced to prevent performance degradation in non-captioning tasks post-mrDPO, using supervised fine-tuning on self-labeled data to maintain both captioning quality and broader LLM capabilities.

strengths:
- The introduction of video-SALMONN 2 with its emphasis on synchronized audio-visual inputs for captioning represents a significant advancement. By integrating both visual and audio data, the model moves beyond purely visual or audio-focused LLMs, capturing richer and more contextually complete video descriptions. The novel use of multi-round Directed Preference Optimization (mrDPO) as a reinforcement learning technique to improve captioning quality is a noteworthy approach.

- The paper demonstrates high-quality experimental validation, showcasing the effectiveness of mrDPO and rebirth tuning. Through detailed metrics like missing information, hallucination, and repetition rates, the authors provide quantitative evidence of video-SALMONN 2’s superiority over leading commercial models like GPT-4o and Gemini-1.5-Pro in video captioning tasks. The thorough benchmarking against these models on multiple metrics adds robustness to the study’s findings.

- The paper is clear and methodically structured, making complex techniques like multi-round preference optimization accessible. Each step in the methodology is well-illustrated, from model architecture through training strategies, with detailed explanations of mrDPO and rebirth tuning. The clarity of the evaluation pipeline and the metrics for assessing captioning quality further aids in understanding the model's improvements.

weaknesses:
- The paper introduces rebirth tuning to help the model retain non-captioning abilities, but the evaluation of this feature feels limited. Expanding the assessment to include the model’s performance on non-captioning tasks, like video question-answering, at each training stage could give a more comprehensive picture of its capabilities. This approach would better demonstrate how effectively rebirth tuning preserves broader language skills.

- The evaluation pipeline depends heavily on other large language models, such as GPT-3.5 and GPT-4, to assess metrics like missing information and hallucination rates. This reliance raises concerns about objectivity and reproducibility, as these models come with their own biases. Developing an alternative, self-contained evaluation method or comparing LLM-based assessments with human judgments could strengthen the evaluation’s reliability and transparency.

- The model’s performance evaluation relies mostly on internal datasets, which limits the generalizability and reproducibility of the results. Including open datasets or releasing the test set would allow the research community to more easily validate the findings and enhance the overall credibility of the work.

- The mrDPO technique is key to the model’s effectiveness, but the paper lacks a detailed analysis of its individual components, like LoRA updates, regularization terms, and the multi-round structure. Investigating how each component contributes to the model’s overall performance would provide insights into which aspects are most beneficial, potentially guiding future improvements and applications of mrDPO.

- The training strategy, particularly mrDPO and the use of several LoRA proxies, adds computational overhead without a clear analysis of the associated trade-offs. Quantifying the specific GPU memory, runtime, and computational load required for each training round would clarify the model’s resource demands. Additionally, identifying the point at which diminishing returns begin in mrDPO could shed light on the efficiency of each round, highlighting which may be resource-intensive with limited added benefit. Exploring alternative configurations for resource-limited environments and comparing high-resource to optimized setups would offer insights into how performance scales in different compute settings. This analysis would clarify the balance between computational cost and performance improvements, making the model more adaptable and accessible for varied research needs.

- Although the model is designed to process both audio and visual inputs, the paper provides minimal qualitative insight into how well it aligns these features in its captions. Including examples of successful synchronizations as well as cases where alignment fails would offer a clearer view of the model’s multi-modal understanding abilities.

questions:
- Could the authors include more diverse evaluation tasks to assess the impact of rebirth tuning on non-captioning skills? For instance, evaluating on tasks like video question-answering at each stage of training could provide more insights into how rebirth tuning preserves or enhances the model’s broader language abilities.

- Considering the reliance on GPT-3.5 and GPT-4 for assessing metrics like hallucination rates, how have the authors addressed potential biases from these models in their evaluation pipeline?

- Could the authors clarify if there are plans to evaluate video-SALMONN 2 on open, widely-used datasets for multimodal captioning? Including results on open datasets or releasing parts of the internal test set would enhance the model's credibility. It would also allow for broader validation by the research community, helping others replicate and build upon the findings.

- Can the authors provide a detailed analysis of the impact of individual components of mrDPO, like LoRA updates, regularization terms, and the multi-round structure, on the overall performance? How does each element contribute uniquely to the training effectiveness?

- Given that mrDPO involves multiple LoRA proxies and additional training rounds, can the authors quantify the GPU memory, runtime, and computational load associated with each round? Is there an observed point where additional rounds yield diminishing returns? Common metrics are FLOPS, MACs, Latency and so on. 

- While the model is designed for audio-visual input, the paper lacks qualitative insight into its success rate in synchronizing audio and visual elements. Could the authors provide examples or failure cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733207704027

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
**Dear Reviewers, Area Chairs and Senior Area Chairs,**

We thank all reviewers for their invaluable feedback on our work. We appreciate and highlight the positive feedback here:
- Our paper proposes a novel training pipeline (Reviewer zUKj, 7DpV and aYgd)
- Our paper focuses on detailed video captioning, which is a promising direction (Reviewer MeEe and aYgd)
- Our paper proposes novel metrics for detailed captioning evaluation (Reviewer zUKj and aYgd)
- Our approach achieved SOTA/competitive to SOTA performance on video captioning (Reviewer zUKj and MeEe)
- Our paper includes audio for video understanding (Reviewer zUKj and aYgd)
- The paper is clearly and well presented (Reviewer zUKj and 7DpV)

We have addressed all the questions and weaknesses the reviewers pointed out in our rebuttal. We have uploaded the revised version of the paper. Here is a summary:
- Evaluation of more video benchmarks (All reviewers)
  - We have evaluated our model on more video benchmarks in our rebuttal, including more video QA benchmarks (Video-MME, NeXT-QA, MVBench and VideoVista) and more video caption benchmarks (VDC). Reviewer aYgd and zUKj, the only reviewers who responded to us, both stated that this flaw in the paper has been perfectly addressed in the rebuttal.
- Improvement of the writing and illustrations (Reviewer MeEe and aYgd)
  - We have revised our paper to clarify more details, which has been acknowledged by Reviewer aYgd.
- Computational cost (Reviewer zUKj)
  - Although the training process of multi-round DPO takes up a lot of computational resources, the inference cost of the final model actually remains completely unchanged. This is because the models always have the same structure in all training stages, meaning no extra inference cost for deployment.
- Rationality of the new metrics (Reviewer zUKj and aYgd)
  - We have manually checked each step of computing the missing and hallucination rates, and the results are plausible. Besides, in our rebuttal, we conducted the ELO test following Reviewer aYgd's advice, and the results demonstrate that our proposed metrics are well aligned with human perception. This point is acknowledged by Reviewer aYgd.

We would like to summarize our contribution of this work:
- We develop video-SALMONN 2, a powerful audio-visual LLM that generates high-quality video captions, even surpassing GPT-4o and Gemini-1.5-Pro.
- We introduce a new evaluation pipeline that computes the missing and hallucination rates of audio-visual events in video captions.
- We propose novel training techniques to enhance detailed video captioning, including multi-round DPO with LoRA proxies and the guided loss function, and rebirth tuning.

While we have thoroughly addressed all the concerns raised by the reviewers, many of our responses have yet to receive further feedback. Our work represents a significant advancement in a challenging area, and we sincerely hope the reviewers can carefully review our rebuttal. We also kindly hope the Area Chair evaluates our work by taking into account its contribution and the detailed responses we have provided.
</content>
</thread_object>

</thread>


Metareview:
metareview:
# Summary and Recommendation for Rejection

---

## Strengths:
1. **Innovative Training Techniques**:
   - Introduces **multi-round Directed Preference Optimization (mrDPO)** and **rebirth tuning** for optimizing detailed and accurate video captioning.
   - mrDPO stabilizes training through iterative updates and LoRA proxies, while rebirth tuning mitigates catastrophic forgetting in non-captioning tasks.

2. **Advanced Model Capabilities**:
   - The proposed model, **video-SALMONN 2**, significantly improves video captioning tasks, surpassing GPT-4o and Gemini-1.5-Pro in custom benchmarks.
   - Synchronized audio-visual inputs enhance the model's multimodal understanding, producing richer and more contextually accurate descriptions.

3. **Novel Evaluation Metrics**:
   - Proposes new metrics for video captioning, including missing and hallucination rates.

4. **Comprehensive Experimental Validation**:
   - Demonstrates significant reductions in error and repetition rates, supported by thorough ablation studies on mrDPO and LoRA proxies.

5. **Potential Community Impact**:
   - The authors commit to releasing datasets, code, and model checkpoints, fostering transparency and future research opportunities.

---

## Weaknesses:
1. **Limited External Benchmarking**:
   - Evaluation is predominantly on internal datasets, with limited testing on public benchmarks (e.g., VideoMME Short, VDC). This raises concerns about the generalizability of the results.

2. **High Computational Costs**:
   - mrDPO’s iterative training process requires significant computational resources, limiting its accessibility to researchers with fewer resources.
   - Insufficient quantitative analysis of the trade-offs between performance gains and computational overhead.

3. **Dependence on Other Models for Evaluation**:
   - Metrics like hallucination and missing rates rely on GPT-3.5/4 for evaluation, introducing potential biases and reproducibility challenges.

4. **Unclear Advantage of mrDPO**:
   - Although mrDPO reduces error rates, it introduces issues like unnatural text patterns, which require additional fine-tuning. The advantage over single-round DPO remains unclear due to limited direct comparisons.

5. **Incomplete Presentation**:
   - Methodological details are not fully explained, such as specific loss functions, definitions of key terms, and the process of sampling captions.
   - Missing qualitative examples that showcase the incremental improvements of mrDPO and rebirth tuning.

---

## Authors’ Mitigation:
1. **Expanded Benchmarks**:
   - Added evaluations on new video QA datasets (e.g., NeXT-QA, MVBench, VideoVista) and public video captioning benchmarks (e.g., VDC).

2. **Improved Clarity**:
   - Revised the manuscript to clarify methodology, including detailed explanations of training losses, caption sampling, and mrDPO dynamics.

3. **Addressed Computational Concerns**:
   - Quantified computational costs for data generation and training, explaining methods to reduce overhead using merging techniques for LoRA proxies.

4. **Evaluated Metrics**:
   - Justified the reliability of new metrics through ELO tests and manual validations, addressing concerns about potential biases.

---

## Remaining Weaknesses:
1. **Generality and Scalability**:
   - Broader validation on open benchmarks like MSR-VTT and VATEX remains missing, hindering general applicability.
   - Computational demands of mrDPO make the approach less feasible for researchers with limited resources.

2. **Evaluation Pipeline Limitations**:
   - Heavy reliance on external LLMs (GPT-3.5/4) for evaluation raises concerns about reproducibility and objectivity.

3. **Incomplete Analysis**:
   - The paper lacks a detailed breakdown of individual components of mrDPO and their specific contributions to performance improvements.

4. **Presentation Issues**:
   - Despite revisions, some sections still lack sufficient detail, limiting full reproducibility.

---

## Recommendation:  **Reject**

### **Justification**:
1. **Evaluation Shortcomings**:
   - Over-reliance on internal datasets and limited external benchmarking weaken the paper’s generalizability and broader applicability.

2. **High Computational Overhead**:
   - The computational demands of mrDPO, combined with insufficient analysis of trade-offs, limit the practicality of the proposed approach.

3. **Insufficient Clarity and Reproducibility**:
   - Key methodological details remain unclear, and dependence on external LLMs for evaluation introduces significant reproducibility concerns.

4. **Potential but Unproven Impact**:
   - While promising, the paper does not yet demonstrate sufficient breadth or robustness to meet the standards of a top-tier conference like ICLR.

The paper would benefit from more comprehensive external benchmarking, improved computational efficiency analyses, and clearer methodological explanations to better establish its contributions.