Title: Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimoda large language models

large multimodal models

large video language models

large vision language models

video understanding

post-training

alignment
Abstract: Despite recent advances in Large Video Language Models (LVLMs), they still struggle with fine-grained temporal understanding, hallucinate, and often make simple mistakes on even simple video question-answering tasks, all of which pose significant challenges to their safe and reliable deployment in real-world applications. To address these limitations, we propose a self-alignment framework that enables LVLMs to learn from their own errors. Our proposed framework first obtains a training set of preferred and non-preferred response pairs, where non-preferred responses are generated by incorporating common error patterns that often occur due to inadequate spatio-temporal understanding, spurious correlations between co-occurring concepts, and over-reliance on linguistic cues while neglecting the vision modality, among others. To facilitate self-alignment of LVLMs with the constructed preferred and non-preferred response pairs, we introduce Refined Regularized Preference Optimization (RRPO), a novel preference optimization method that utilizes sub-sequence-level refined rewards and token-wise KL regularization to address the limitations of Direct Preference Optimization (DPO). We demonstrate that RRPO achieves more precise alignment and more stable training compared to DPO. Our experiments and analysis validate the effectiveness of our approach across diverse video tasks, including video hallucination, short- and long-video understanding, and fine-grained temporal reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets at improving Large Video Language Models (LVLMs) and propose a framework to have the model learn from the error they have made. In detail, the authors automatically generates pairs of preferred and non-preferred responses by introducing spatio-temporal perturbations into videos to simulate typical error patterns. Then, they designed Refined Regularized Preference Optimization, which is claimed to offer more precise alignment and more stable training, by utilizing reward at sub-sequence level. Both mathematical derivation and experimental validation has been provided to demonstrate the superiority of RRPO over DPO.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**:\n- The motivation is important: learning from the errors to further enhance the model itself is important for application usage in the real world, in particular for fine-grained temporal understanding and reducing hallucination. \n- The RRPO is novel and the idea of generating negative sequences by perturbing video is interesting.\n- Detailed and extensive ablation study and discussion\n\n**Weakness**:\n- I am curious whether any capability of LVLM before RRPO is compromised. For example, what is the performance on the LVLM after RRPO on the cases that the model answered correctly before. \n- Though I understand the claim regarding difference between coarse-grained and fine-grained level difference, but I am not sure whether this difference can be mitigated even with DPO. In other words, whether such difference really matters. Then, for the gradient scale (claim in Line 128), demonstration of the large gradient caused by long sentences is needed.\n- Several important implementation details are missed, e.g., what is the value alpha used in the experiments.\n- As part of future research direction, when the spatial & temporal perturbation is applied, the model prediction on some videos remains the same, which could be problematic. For example, when the key object for understanding is masked, if the model still makes the correct prediction, it can be treated as a hallucination, which is not preferred and should be mitigated. Then, how can your method be further improved to identifying such cases?\n\nQUESTIONS:\nPlease see my weakness\n\nLIMITATIONS:\nPlease see the weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduce Refined Regularized Preference Optimization (RRPO), a preference optimization method that utilizes sub-sequence-level refined rewards and token-wise KL regularization to address the limitations of DPO. The proposed method consistently improves performance over the base models and generalizes well across diverse training setups.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method shows promising improvements.\n2. The experimental evaluation is well-executed.\n3. The method consistently improves performance over the base models and generalizes well across diverse training setups.\n\nWeaknesses:\n1. The proposed method appears to be specifically designed for video understanding tasks. It would be helpful if the authors could clarify whether the approach is inherently limited to video inputs, or if it can be extended to image understanding tasks as well.\n\n2. Lacks sufficient qualitative comparison results. Visual comparisons with baseline methods are essential to assess generation quality beyond quantitative metrics.\n\n3. This work lacks an analysis of inference efficiency. It would be beneficial to include comparisons with baseline methods in terms of training time, GPU hours, and inference speed.\n\n4. The method presents interesting results on low-resolution video datasets, and it would be valuable to explore the method’s performance in higher-resolution scenarios. Real-world applications often involve high-fidelity video inputs, which may introduce different challenges such as increased feature complexity or computational demands. It could be beneficial for the authors to discuss potential extensions to high-resolution data in the discussion section, or consider supplementary experiments on datasets with varied visual fidelity.\n\nQUESTIONS:\nSee weakness.\n\nIf the authors can effectively resolve the issues I have highlighted, I would be willing to raise my rating accordingly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-alignment framework for Large Video Language Models (LVLMs), where models learn from their own errors via responses to perturbed videos. The key contribution is Refined Regularized Preference Optimization (RRPO), a novel training objective that combines sub-sequence-level rewards with token-wise KL regularization. RRPO improves fine-grained temporal reasoning, reduces hallucinations, and achieves consistent gains over prior alignment methods like DPO, as validated across multiple LVLM architectures and video understanding benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Technical Quality: The paper presents a rigorously designed training framework with clear theoretical and empirical justification.\n- Empirical Rigor: The authors evaluate their method across a broad spectrum of benchmarks covering temporal reasoning, hallucination mitigation, and long-video understanding. RRPO consistently improves over baseline and existing methods with performance gains up to 5.4%.\n- Innovation: The RRPO method combines fine-grained sub-sequence-level preference modeling and token-level KL regularization, addressing key weaknesses of DPO (e.g., coarse reward modeling, training instability).\n- Scalability & Automation: The self-alignment data generation pipeline is automatic and does not require human annotation, allowing scalable construction of (preferred, non-preferred) response pairs from perturbed video samples.\n\nWeaknesses:\n- Clarity and Writing Quality: While the technical content is solid, the presentation can be improved. Some notations are densely packed, and explanations (e.g., around Eq. 6–8) require careful reading to interpret. Moreover, acronyms like RS-Mask, LS-Mask, etc., appear in Table 5 without early clear definitions (though defined in text).\n- Formatting & Style: The manuscript contains minor formatting inconsistencies (e.g., spacing issues in equations and figure captions, long caption texts), which reduce readability.\n\nQUESTIONS:\n1. The paper uses a GPT-4o-mini model to assess correctness for open-ended tasks. Is there a risk of bias or alignment drift if this LLM acts as both the judgment oracle and learning target?\n2. The method relies on identifying key concept pairs (y+i, y-i) for sub-sequence reward calculation. How robust is this identification process when scaling to noisy or less structured video-language datasets?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a self-alignment framework for large vision-language models called RRPO, which enables them to learn from their own errors without any human-annotated preferences. Rather than relying on manually labeled preference pairs, RRPO generates spatial and temporal perturbations of video inputs, uses an external LLM to score and rewrite outputs, and applies sub-sequence-level token-wise KL-regularized rewards. RRPO demonstrates significant gains on hallucination reduction, temporal reasoning, and both short- and long-video understanding benchmarks across multiple LVLM architectures. It also provides tight gradient-norm bounds that justify its stable, efficient training and allow higher learning rates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and easy to understand.\n2. The fine-grained motivation is interesting and appears to complement DPO’s coarse-grained training approach.\n4. Comprehensive evaluation across three diverse LVLMs and six benchmarks (temporal reasoning, hallucination, short- and long-video understanding) demonstrates consistent performance gains.\n5. Theoretical analysis about RRPO’s gradient norm justifying its stability during model training.\n6. Code, data, and model weights are provided to ensure reproducibility. \n\nWeaknesses:\n1. Synthetic perturbations may not accurately capture real-world error distributions, risking overfitting to artificial failure modes. Since this strategy is not explicitly tied to actual model failures, it can lead to a distribution misalignment between the generated negatives and the model’s true hallucination behavior.\n2. The requirement for non-preferred responses to maintain structural similarity with the original responses may hinder the model’s ability to generalize to diverse response patterns.\n3. Experiments on failure cases or domain shifts are not thoroughly analyzed.\n4. The discussion of this work's limitations seems brief and insufficient.\n\nQUESTIONS:\nMy main concerns are in the method section, please see the weakness above.\n\nAdditionally:\n1. Does the fine-grained focus and reward-hacking token-wise KL regularizer result in a trade-off between hallucination suppression and general language-vision capabilities, where the model overfits to hallucination-related metrics, achieving high scores without genuinely improving visual grounding, since the gains are primarily on hallucination benchmarks?\n\nLIMITATIONS:\nThe discussion of this work's limitations seems brief and insufficient, as it does not properly address the failure modes of the approach or the trade-off between fine-grained and coarse-grained learning.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}