Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization
Authors: Pritam Sarkar
Ali Etemad
Venue: NeurIPS 2025 poster ()
Keywords: multimoda large language models
large multimodal models
large video language models
large vision language models
video understanding
post-training
alignment
Primary area: deep_learning
TL;DR: Introduced Refined Regularized Preference Optimization with a self-alignment framework to enable fine-grained alignment of large video language models by learning from their own errors.

Abstract: Despite recent advances in Large Video Language Models (LVLMs), they still struggle with fine-grained temporal understanding, hallucinate, and often make simple mistakes on even simple video question-answering tasks, all of which pose significant challenges to their safe and reliable deployment in real-world applications. To address these limitations, we propose a self-alignment framework that enables LVLMs to learn from their own errors. Our proposed framework first obtains a training set of preferred and non-preferred response pairs, where non-preferred responses are generated by incorporating common error patterns that often occur due to inadequate spatio-temporal understanding, spurious correlations between co-occurring concepts, and over-reliance on linguistic cues while neglecting the vision modality, among others. To facilitate self-alignment of LVLMs with the constructed preferred and non-preferred response pairs, we introduce Refined Regularized Preference Optimization (RRPO), a novel preference optimization method that utilizes sub-sequence-level refined rewards and token-wise KL regularization to address the limitations of Direct Preference Optimization (DPO). We demonstrate that RRPO achieves more precise alignment and more stable training compared to DPO. Our experiments and analysis validate the effectiveness of our approach across diverse video tasks, including video hallucination, short- and long-video understanding, and fine-grained temporal reasoning.

Self- of Large Video Language Models with Refined Regularized Preference Optimization
Pritam Sarkar
Ali Etemad
Queen's University, Canada and Vector Institute pritam www.arrquenssu.cccoommmmu ca
Queen's 's University, Canada
ali etemad6queensu.co
Website
Code
Models
Abstract
Despite recent advances in Large Video Language Models (LVLMs), they still struggle with fine-grained temporal understanding, hallucinate, and often make simple mistakes on even simple video question-answering tasks, all of which pose significant challenges their safe and reliable deployment in real-world appli- cations. To address these limitations, we propose self-alignment framework that enables LVLMs learn from their own errors. Our proposed framework first obtains a training set of preferred and non non-preferred response pairs, where non-preferred responses are generated by incorporating common error patterns that understanding, spurious correlations
often occur due to inadequate spatio-tempont between occurrinu concepts, and over-reliance on línguistic cues while neglect- ing the vision modality, among others. To facilitate self-alignment of LVLMs with the constructed preferred and non-preferred response pairs, we introduce Refined Regularized Preference Optimization (RRPO), novel preference optimization refined rewards and token-wise KI regu-
method that utilizes sub-sequence-les larization to address the limitations of Direct Preference Optimization (DPO). We demonstrate that RRPO achieves more precise alignment and more stable training compared to DPO. Our experiments and analysis validate the effectiveness of our approach across diverse video tasks, including video hallucination, short- and long-vide long-video understanding, and finegrained temporal reasoning.
Introduction
Despite recent progress in Large Video Language Models (LVL.Ms) [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15), these models continue to face limitations in fine- grained temporal understanding [16. 17, 18], demonstrate a propensity for halluci- nation [19, 20], struggle with long-vides video understanding [21, 22, 23, 24, 25], and frequently make naive mistakes in short- video question-answeriin 18]. Please see Figure for number of ex- amples. These shortcomings severely limit their safe and reliable deployment in real- world applications. The underlying causes
tasks [16, 17,
Figure 1: Examples of failure cases in simple video understanding tasks by state-of-the-ard art LVLMs, and im- provements observed after self-alignment with RRPO.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of these limitations are multifaceted, encompassing factors such as inadequate spatial and temporal
understanding [16, 17, 18], vision-language representational misalignments [26. 22], challenges in processing long visual sequences due to context length constraints [21, 22, 23], spurious comelaatios
between co-occurring concepts (27, 28], and an over-reliance on linguistic cues while neglecting the
visual information [16, 17, 18, 29].
To address these shortcomings and enhance video understanding in LVLMs, we design self- self-allinment [30] framework that en- ables LVLMs to learn from their own er- rors. Specifically, we begin by sampling video-question pairs from an open-soure, benchmark. We then apply spatio-temporal perturbations to the video content to mimic common errors that often arise from over-
Large Vodea Lunguage Nocul
Coower
No
selectione reeporees
Parurted
Alarge set video aato-quaeten--ee -oreoor
High-qua By pretered and mnnnntppnteeeedd meconnas
Prepessessing
Figure 2: An overview of our self-alignment framework.
reliance linguistic cues, spurious correlations between co-occurring concepts, and insufficient spatio-temporal understanding. Using the perturbed video and the corresponding question, we perform inference with the target LVLM. If the model's response is incorrect, we construct self- alignment pair by treating the incorrect response as a non non- preferreeredd sample and the correct response as the preferred one, which is kept for self-align aligmaligmeent training. Responses that are already correct discarded, as they offer limited self-improvement potential. Subsequently, we optimize a loss function to prioritize the prefemred response over the non-preferred one. Notably, our data generation pipeline is free from human annotation and can be easily scaled. A high-level overview of our self-alignment framework is depieted in Figure 2.
To effectively train LVLMs using the generated preferred and non non- preferred response pairs, we introduce Refined Regularized Preference Optimization (RRPO), an approach designed to address limitations of Direct Preference Optimization (DPO) [31]. RRPO is designed to address the key drawbacks of DPO. RRPO provides fine-grained sub-sequence-level reward to explicitly penalize the tokens containing the key concepts that are different between the preferred and non non-preferred response pairs. This is contrary to DPO's 's response-level reward which penalizes all tokens within non- preferred responses, thus lacking precision and often proving unsuitable for fine-graing reeeggrined alignment. It should be noted that our fine- grained reward further benefits from computing smaller gradient during optimization and is therefore less prone to diverge away from its initial state unlike DPO's s response-level feedback along with weak regularization which can cause significant divergence from the base model, leading to suboptimal performance. However, sub-sequence-leve rewards may incentivize the model exploit shorteuts, such as outputting correct concepts without proper context or complete responses. To mitigate this, we also minimize token-wise KL-divergence [27] using a reference model on the preferred responses. This ensures that the LVLM maintains a tight bound on its likelihood across the entirety of the preferred response while reducing the likelihood on the non-preferred concepts.
Through empirical and theoretical analysis, we demonstrate that RRPO exhibits greater stability and smoother convergence during optimization compared to DPO. We validate method on three popu- lar LVLMs specialized for video understanding. VideoChat2, LLaVA- and Von covering a wide range of architectures, LLMs, vision encoders, and training setups. Our in-depth evaluation demonstrates that our proposed self-alignment reduces hallucinations and improves performance in fine-grained temporal understanding, as well as in both short- and long-vided understanding tasks.
In summary, our contributions are as follows:
We design a self-aligmment framework to facilitate self-improvement of LVL.Ms based on their own errors. We introduce RRPO, a preference optimization method that addresses the limitations of DPO by utilizing sub-sqquence-lever refined rewards and token-wise strong KL regularizer, resulting in more precise alignment and stable training.
Our rigorous evaluation demonstrates the effectiveness of our proposed method across diverse video tasks, including video hallucination, short and long video understanding, and fine-grained temporal reasoning, among others. Moreover, our experimental and theoretical analyses highlight the superiority of RRPO over DPO in aligning LVLMs. We make our code, data, and model weights public enable fast and accurate reproducibility.

---

2 Preference Responses for Self-alignment
As the first step in our framework, we construct training dataset D comprising both preferred re- sponses and responses containing incorrect concepts in order to align the LVLM Th favor correct con- cepts over incorrect ones. We begin by utilizing a
large and diverse publicly available video instruc- tion tuning dataset, containing triplets of video F. question Ig, and their human-prec preferred answers y To generate non-preferred responses y we obtain perturbed videos E, = fp(xx), where fn is a perturba- tion function which masks large portion of frames and applies temporal shuffling, compromising spatio- temporal consistency. Our intention from this step is to provoke the LVLM to generate responses primarily based language cues their parametric knowl- edge. These perturbed videos S, are fed to Ty as inputs generate responses with potential erroneous concepts y = =T (T 2.)). An example is illustrated in Figure 3. We then verify the correctness of y retaining incorrect responses and discarding correct ones. Next, We employ an LLM to meticulously com- pare y against y" and identify key incorrect concepts within y. ensuring that for each incorrect concept y; ey there exists corresponding correct concept y," Ey+ In the context of our work. a concept can be actions, objects, attributes, relations, and other key elements in the response that contribute to the semantic understanding of the video. Furthermore, for lengthy responses, we maintain structural similarity between the incorrect and correct versions by rewriting the correct response while incorporating the incorrect concepts. Finally, we apply deduplication based on these correct-incorrect concept pairs, constructing a high-quality training dataset. Examples are provided in Figure 4.
Figure 3: An example of perturbed video.
meci saaing fode. lightst barks
saanting andeict ciaallyy clastng
pellie
mostl 2000
Frem bast coiee
Figure few training samples. We bold the correct and incorrect concepts.
RRPO
Preliminaries. Given an input x = (xgg Fe with a pair of responses {yt,y=}, where yt > y la. DPO [31] can be employed to align xg to favor y* over y This is achieved by maximizing the reward margin between Ty and a reference model Tref using the following training objective:
(Tro(To: Tra) -anart w- >~D log a (ro(z,v") -r((x,))))
(1)
==(())=)) and 3 controls the deviation from Ted Note that (a, y) is calculated at response-level by penalizing all the tokens in y. despite the fact that the difference between yt and y might only be limited few key tokens. Such response-level reward can be considered coarse-grained reward modeling and is not suitable for fine-grained alignment. Moreover, as the reward is calculated by penalizing all the tokens in the response, the gradient for LDPO tends to be very large for long responses, and accordingly ny can diverge to an undesired state thus losing its out-of-the-box capabilities [27, 32, 33].
where reward ro(x; y) = log (o(g)x))
Refined Regularized Preference Optimization. Our goal is to design a method that can provide fine- gre-grained reward to penalize individual sub-sequences consisting of the tokens belong to the key differing concepts between y+ and y: We refer to this as refined reward modeling. Let be expressed as a sequence of tokens T = {t1. f2. ... } Here, the i-th sub-sequence y: can be expressed T(s; : eg], where 8; and e; are the start and end indices of y with 1 I 8 S e, |y]- Therefore, the reward for y, can be computed as:
I Te(tylz, (z, tzj)
ro (o(,y)) v:) = 3 log
I कार (tyler Ir, (zح
3

---

Assuming there exists N such sub- sub-sequences, we can train To to maximize the total reward margin
N
Σ (ro(a, w(t) ro(2, ()))
U4
Subsequently, we replace the reward formulation in Equation 1 with our refined reward modeling as:
c(tan)) RRPO (To; Tzet) = (aay ar JwD (u))
(4)
However, the sparsity of the rewards used to calculate C(ank) may allow Ty to exploit shortcuts and effectively game the reward function by merely outputting key concepts without appropriate context or generating complete responses. This reward hacking may occur since To is incentivized maximize reward margin based the differing sub-sequences; the model can learn to produce short, incomplete responses that contain the correct key concepts, even those responses lack overall coherence, fluency, or completeness. To mitigate this reward hacking, we introduce regularizer between and Traf. based on token-wise KL (TKL) divergence [27, 34], as follows:
TTKL (Ir, Ui Jiet TT)
Σ DEL (Tral) t=1
[xr, we)))) Te(- [x, veel))
(5)
Accordingly, to ensure To retains its original likelihood across the entirety of yt while reducing the likelihood of non-preferres concepts, we optimize ΙIKL between Ty and ne over y*. We then modify Equation 4 and define the final RRPO loss as:
CRRPO(TH) Tret) (z,yy or )DD log (ug + DexD (x, (2,
(6)
where a controls the divergence between Te and Teef, and Tief is kept fixed.
How is RRPO update different from DPO? The gradient for CRRPO can be obtained as:
V VoLRRo = Vo - log (u) + a- D-DrL (2.v*) (.D = E Vo log ( + a VolDTKL (z. (7)
First, let's calculate the gradient for the ranking loss. Using chain rule, we can write Vo log o(u) ('())
Vau Using the identity d'(u) (u) - (u)), we have a'(w) c(a) = - ou)) = o--).
(
Therefore, Vg logo (u) = (a(-u) Vou
Recalling our sub-sequence-level reward modeling defined
in Equation 3, can obtain:
V Vo,, 8
V log To (e,
Σ V log no 1z, Tz
(8)
j=*7
as is fixed. Assuming the norm of the gradient of the log-probability for any single token is bounded, || Ve log No (z,z) & M for some M > O, and that each differentiating sub-sequence y or V, has an average length L, we can further bound the gradient norm of the ranking loss:
V C
< E
< Σ (ML + ML (M(NL)),
where L is the average length of the sub-sequences (L = L R L,). In contrast, the DPO gradient involves sums over the entire lengths of y and y A similar analysis for DPO yields bound proportional to the total lengths:
(10) Crucially, the total length of the differentiating sub-sequences is typically much smaller than the total length of the full responses, i.e., 2NL 1y I 1l Therefore, the upper bound the gradient magnitude stemming from the ranking objective is smaller for RRPO compared to DPO:
VoLoro 3M
V
RRPO
(tank)
V VeLppoll

---

Now, let's consider the term VoDIKL()) in Equation 7. Based on the formulation of (Trall
La Tre((a) log Tarla) where a represents a token in the vocabulary, and since aer fixed, we derive: ma(a]
1y
VeDDKL(Z, VODDK(
Tar(aa x. V22) Ve To(a y2).
(11)
Note that VKLT is always negative. Therefore, for in Equation the gradient magnitude of LRRPO is further reduced than RRRO
VeLnuroll < RRPO
VeClrll
This mathematical derivation confirms that the proposed RRPO loss function effectively reduces the gradient, as initially hypothesized. The reduced gradient of RRPO ensures more stable updates compared to DPO in gradient-based optimization while simultaneously enabling precise penalties on specific tokens without the risk of significant divergence. Furthermore, the lTкII term acts as a trust-region constraint [35], preventing the model from making large, destabilizing updates. As result, RRPO allows larger learning rates and yielding smoother convergence practice. We present the pseudocode of RRPO in Appendix A.
4 Experiment Setup
Base models. We use VideoChat2ns [3], LLLLV--Viiieos (also known as LLaVA--eet-Videeyyy [9], and LongVUns [1] as our base models. These models are carefully selected to evaluate our method across diverse LLM architectures, vision encoders, cross-modal adapters, and training setups. For instance, among these models, VideoChat2 employs video encoder, while the others rely on image encoders. Additionally, Lo incorporates two vision encoders, whereas the rest use a single vision encoder. VideoChat2 utilizes QFormer [36] as its cross-modal adapter, whereas ALLVA-Video employs MLP projection layers. These models further differ in their LLM architectures, context lengths, and training setups, among other aspects.
Training data. Based on the availability and diversity of video-language instructions, we use VideoChat-IT [3] as our primary source for training samples. Specifically, we select subset of VideoChat-ITT encompassing eight video datasets: Kinetics700 [37]. Something-Somethingg
[38]. VideoChat [39], VideoChatGPT [40]. CLEVRER [41], NEXTQA [42]. EgoQA [43], and TGIF [44]. These datasets span a range of tasks, including video description, question answering, reasoning, and conversation. For the perturbation step, we mask a significant portion (25%-50%) of each frame and shuffle the temporal order. We explore three types of temporal perturbations: (i) random shuffling, (ii) local shuffling, and (iii) global shuffling. In random shuffling, frames are shuffled arbitrarily across time. For local shuffling, frames are initially segmented into smaller chunks, and the frames within each chunk are then shuffled. In global shuffling, the order of these chunks is shuffled, rather than individual frames. During inference, based on the LVLMs' input capacity, we utilize maximum of 16, 64, and 100 frames for VideoChat2, LLaVA- Video, and LongVU, respectively.
Following the generation of the responses, we verify their correct- ness. For Multiple Choice Questions (MCQ) and Binary Question Answering (BinaryQA) tasks, verification is straightforward, us- ing regex-based checks. However, for open-ended questions, this method proves inadequate, semantically equivalent responses can be expressed in diverse phrasings. Consequently, for open-ended questions, we employ powerful LLM, GPT-4o-mini [45], a judge [46, 47, 48, 49], to ascertain correctness by comparing the generated response with the ground truth from the video instruction tuning dataset. Additionally, for long responses, we employ 4PT-4ommnni to rewrite the correct response while incorporating the incorrect concepts from the generated response. This ensures that correct and incorrect concepts aligned across both preferred and non non- preferred responses. The key statistics our training data are presented in Table 1. The prompts used during pre-processing with GPT-4o-mini are in Appendix D.
Table 1: Key statistics of the generated training samples.
Unique
Model
# Samples
VideoChat2:
25K
18K
Von LLaVA-Videom
22K
14K
21K
16K
Implementation details. For all base models, we utilize LoRA [50] for training, applying specifically to the LLMs while freezing all other parameters. Unless otherwise specified, we utilize
5

---

16 frames for self aelfmanignment training: the rest follows the default training setup of each respective base model. We use 4x A100 80GB GPUs for training, with the training time varying between 1 to 10 hours. Additional implementation details and hyperparameters are provided in Appendix E.
Evaluation benchmarks. To assess the impact of our self-alignment framework, we conduct evaluations across diverse range of video understanding tasks, Specifically, we choose TVBench [17] and TempCompass [20] for fine-grained temporal understanding, VideoHallucer [19] and VidHalluc [51] for video hallucination, MVBench [3] and VideoMME [52] for short video understanding, and MLVU [24] and Long VideeBeenn [25] for long video understanding. It should be noted that these benchmarks, while selected for specific tasks, often assess overlapping video understanding capabilities. For instance, while VideoHallucer and VidHallue are primarily used for hallucination detection, they also evaluate temporal grounding [19, 51]. Similarly, while TVBench mainly focuses on temporal understanding, it covers short video understanding as well [17]. Given its inclusion of short, medium, and long videos, VideoMME explores video understanding of varying lengths.
5 Results and Analysis
This section details our experimental evaluation, which encompasses a comprehensive and compara- tive analysis against other alignment methods and existing off-the-shelf aligned LVL.s. Furthermore, we present detailed analysis focusing on the trade-offs between post-alignment divergence and per- formance. To gain deeper insights into the impact of our proposed method, we conduct experiments exploring its effects on fine-grained temporal understanding, hallucination mitigation, and perfor- mance comprehensive video understanding of varying lengths. Finally, our evaluation includes an investigation into the influence of data, the scaling of input frames, and the presence of subtitles on the performance of our method. We conclude with discussion regarding the generalization capabilities of our approach.
Table 2: Comparison with existing preference optimization methods and RRPO ablation vari- ants. == - (acCaligned aCCbase) and %A = Σ (acCaligaed - aCCbase) / aCCpase where N is the number of evaluation benchmarks used for each ablation. RRPO consistently outperforms existing alignment methods.
59
RAPO
RRPO w/o R*
100 N
58
DDPO
X
TDPO
57
DPA
Accuracy
DPO
56
RRPO w/o 9Ta
+
VHall VMME MLVU A/KA
55
*
LongVUpg (base)
53.7
39.2
56.2
63.6
54
-DOO [3]]
51.3
40.9
56.6
63.6
0.7/1.5
IPA [27]
54.6
40.3
56.9
63.9
0.7/1.5
53
Baseline
TTDPO [34]
53.9
41.4
57.0
63.8
0.8/1.9
DDPO [53]
54.2
41.7
56.7
63.6
0.9/2.0
10
15
-RRPO R
54.3
43.0
57.8
64.5
1.7/3.8
KL Divergence
-RRPO w/o Dnal
54.9
39.1
57.4
63.9
0.6/1.1
Figure 5: Performance relative to model diver-
RRPO (ours)
56.5
44.0
57.7
64.5
2.5/5.4
gence. RRPO exhibits the best performance- divergence trade-off.
Abbbeviations, TVB: TVBenck; VHalt: Videoffollare;; VMME: VodenMMMF)
Q1. How well does RRPO perform compared to other alignment methods? We conduct an in-depth analysis of RRPO against other recent preference optimization methods designed to address DPO's limitations, namely TDPO [34], DDPO [53], and DPA [27] and present the results in Table 2. More specifically, DDPO was introduced to provide fine-grained rewards, TDPO to enhance DPO's regularization, and DPA to address both of these challenges. A detailed discussion of their objectives, highlighting similarities and differences, is provided in Appendix B. Furthermore, we include two ablation variants of RRPO: one without the refined reward (RRPO w/o R*) and the other without the coken-wise KL regularizer (RRPO w/o DIKL.). Our study covers various aspects of video understanding, including fine-grat fine-grained temporal understanding on TVBench, hallucination mitigation on VideoHallucer, comprehensive video understanding on VideoMMEE, and long video understanding on MLVU. The results presented in Table demonstrate that RRPO consistently outperforms DPO, DPA, TDPO, and DDPO across all benchmarks. Moreover, among the ablation variants of RRPO, the inclusion of DTKI, alone yields significant performance gains, which are further enhanced by incorporating the refined reward. We present qualitative comparisons in Appendix G.

---

Table 3: Comparison with off-the-shelf aligned LVLMs. Ours LLaVA- Video-RRPO outperforms
LaaV--iioo--- across all setups, both of which are based on LLaVA- LLaVA-Videom.
TV
TempCo-
Video Hallucer
Vid
MV
Video MME
MLVU.. (M-Avg)
LongVideo Benchya
Model
Bench
mpasS
Halluc
Bench
ILLVA- TLaVA-Viddeo-TRE Video-TPO [54]
51.1
66.1
50.6
76.3
60.6
65.6/7155
68.7
60.1
(ours)
52.2
67.4
55.8
76.6
62.0
65.5/71.8
69.4
61.3
Table Evaluating our self-aligned LVLMs on diverse video understanding benchmarks. We bold the best in each group. 16f and 32f indicate the number of frames utilized during training. #F indicates number of frames used during inference. Here, we presents the overall results averaged
across sub-categories where applicable, with detailed results a
available in Appendix C.
TV
TempCo- mpass
Video
Vid
MV
Video MME (M-Avg)
MLVU
LongVideo Benchya
Models
Bench
Hallucer
Hallue
Bench
Video-LLaaVAo [10]
33.8
49.9
17.8
40.3
42.5
39.9
29.3
39.1
VideoLLaMA2m
42.9
43.4
10.0
66.3
54.6
62.4
Long [21]
57.0
52.6
42.1
MiniCPM-V 2.6 [55]
66.3
60.9
$4.9
NVILA [56]
70.1
57.7
LongVIL.Ars [8]
67.1
60.1
57.1
Qwen2-Vl.sn [57]
43.8
67.9
67.0
63.3
65.5
$5.6
LLaVA-NeXT-Videe-DDO.. [58]
53.8
32.0
43.5
ShareGPT-Videoon [59]
61.5
15.8
30.9
39.9
34.2
39.7
PLLaVApe [60]
41.2
48.2
45,6
Ariaaa.3 [61]
$1.0
69.6
69.7
67.6
63.0
VlaWA-NeXI-Videe--DD [58]
50.5
Qwen2-VLam [57]
52.7
73.6
71.2
GPT-4o [62]
39.9
73.8
53.3
81.2
49.1
71.9
54.5
66.7
Gemini 1.5 Pro [11]
47.6
67.1
37.8
72.8
60.5
75.0
64.0
(The models presented for reference only may
for comparsonn.)
VhdeoChat2en [3]
7.8
60.4
39.3
VideoChar2<n
16
44.0
59.3
23.1
73.3
60.2
41.0
46.4
40.4
+ +DPO (16f)
16
45.7
60.0
22.1
72.4
59.6
43.0
47.4
41.0
+RRPO (16f) (Ours)
45.8
60.2
32.9
76.4
59.0
44.3
47.9
42.8
L.LaVA L.L.VA-Videog [9]
45.6
58.6
63.3
70.8
58.2
V--Viderm
51.0
50.0
76.6
60.1
DPO (1Gf)
64
51.9
66.4
53.3
76.5
60.6
63.1
G7.4
59.4
RRPO (16f) (Ours)
64
51.9
66.8
55.7
76.5
62.
64.5
69.1
60.4 -
RRPO (32f) (Ours)
52.2
67.4
55.8
76.6
62.1
64.5
69.4
60.1
Long [1]
66.9
65.4
Long Vmm
Lfps
53.7
63.9
39.2
67.3
65.5
56.2
63.6
48.6
DPO (16f)
1fps
54.3
64.3
40.9
68.5
65.9
56.6
63.6
49.4
RRPO (16f) (Ours)
Lips
56.5
64.5
44.0
71.7
66.8
57.7
64.5
49.7
For rransprency report base LVLMr vemits from the literature where available marked in grey. Differences reproduced remalts, both higher and lovert, new fiom in frome count, input prompt, GFT variant used evalnation, and occasional implomenation lssues. For faimess, self aligned vuriants are compared against our reproduced vesuits.
Q2. How much does the model diverge post alignment? Understanding the extent of model divergence after alignment is essential to ensure that the process refines behavior without under- mining core capabilities as excessive divergence can lead instability, reduced generalization, and loss of valuable pre-trained knowledge. Figure 5 presents a comparative analysis of performance improvements against model divergence following preference optimization. Despite using a 10x higher learning rate (which is facilitated through the smaller and more stable gradient updates), RRPO exhibits a KL divergence of compared to DPO's KL divergence of 20, While TDPO and DPA exhibit almost the same divergence RRPO, their performance across all evaluation benchmarks is substantially worse. RRPO, in contrast, demonstrates the optimal performance-divergeee
trade-off.
Q3. How does RRPO performance compared to off-the-shelf aligned LVLMs? We further evaluate our aligned LVLM against other off-the-shell aligned LVLMs. Specifically, we compare our ERPO-trained LLaVA-Vidhe Video with the concurrent work LLaVA-Video-TPE Video-TPO, both of which are based on the VALVA-VE VA-VV-ii Video--Qeenn [9]. LLaVA- Video-TPO is trained using a combination of DPO and SFT with a manually curated video-language dataset. The results in Table 3 demonstrate that RRPO significantly more effective than concurrent methods. Our V.-.
RRPO outperforms LLa VA- Video TPO across all setups, with performance gains of up to 5.2%.
7

---

Table 5: Impact of different spatio-tempora per- turbations in generating non-prefiin non-preferred responses. Default perturbations for each model are colored.
Table 6: Impact of data size.
TVB VHail VMME MLVU A/%A
Baseline
51.0
50.0
61.0
68.6
TVB VHall VMME MLVU
A A
5K
50.9
53.7
64.0
68.0
1.0/19
VideoChat2on (buse)
44.0
23.1
41.0
46.4
51.2
64.3
69.0
1.2/2.3
15K
51.8
54.4
64.2
68.9
1.4/2.8
40.7
16.0
39.9
43.8
-3.5/-11.6
20K
51.9
55.7
64.5
69.1
1.9/3.7
+ R8
43.0
23.3
41.8
46.3
0.0/0.1
Maas
44.0
26.2
43.4
48.8
2.0/6.1
Table 7: Impact of
Table 8: Impact of us- ing subtitles along with
+ -L8-Xask
41.6
28.2
11.6
19.4
3.1/9.7
varying the number of
-08-Nank
44.2
28.8
44.1
46.3
2.2/8.1
+ BB8 -aa-
45.8
32.9
41.3
17.9
4.1/14.4
frames at inference.
videos (VMME).
LLaVA LLaVA-Viderr Videors (base)
51.0
50.0
61.00
68.6
TVB MVB INB
without with
L.8-Naak
51.9
55.7
64.5
1.9/3.7
+ -G- -G8-Xaek
51.3
64.8
64.2
68.7
1.4/2.7
32 Baseline 49.6 61.2 58.4
VideoChan2nn -RPO
41.0
48.0
RB--Mask
51.5
51.6
64.6
69.6
0.9/1.6
RRPD 51.3 61.7 58.9
44.3
49.4
LongVUnn (base)
53.7
30.2
56.2
63.6
64 Baseline 51.0 61.1 60.1
LLaVA-VVddos
63.8
67.4
+ LS -aak
36.5
11.0
37.7
61.5
2.5/5.4
RRPO 52. 62. 1 60.1
+ RRPO
64.5
68.0
-0S-Nask
55.0
43.9
56.9
1.9/4.3
Baseline 49.4 60.5 00.3
Long
56.2
62.0
+ R8-Maale
55.1
42.4
57.0
64.3
1.5/3.3
128
RRPOO 51.3 61.2 61.3
+ RRPO
57.7
63.1
( Abbreviations, A.. Rundow Siufll; L8. Local Stuglle C8: Global Shufll; TVE: TVBenck, VHall: Videoffolluoer, VMME. VileoMME; LVR: Long Veden
Q4. Does our method improve fine-grained temporal understanding? To evaluate this, we utilize TVBench [17] and TempCompass [20], designed to test the temporal understanding abilities of LVLMs. TVBench tests capabilities across various temporal tasks, including action localization, directional movement, and scene transitions. among others. Similarly, TempCompass evaluates per- formance on video captioning, caption matching, MCQ, and BinaryQA, covering video understanding tasks such as event ordering, action identification, and state change. As shown in Table 4, our method, RRPO, consistently improves base model performance by up to 2.8%, demonstrating its effectiveness in enhancing fine-gree grained temporal understanding and outperforming DPO across all setups.
QS. Does our method effectively mitigate hallucinations? Hallucination occurs when LVLMs produce responses that are ungrounded, referred as intrinsic hallucination, or unverifiable, referred to as extrinsic hallucination. Hallucination presents a significant obstacle to the reliable use of LVLMs. To evaluate the impact of our method on video hallucination, we employ VideoHallucer [19] and VidHalluc [51]. VideoHallucer tests LVLMs for both extrinsic and intrinsic hallucinations while including both spatial and temporal hallucinations. Additionally, VidHalluc focuses specifically on temporal hallucinations, such as action hallucination. As shown in Table 4, RRPO significantly reduces hallucination across all base models. Specifically, RRPO improves performance by 4.8% to 8.8% on VideoHallucer and by up to 4.4% on VidHalluc. In most cases, RRPO demonstrates a substantial performance advantage over DPO, with gains reaching 10.8%.
Q6. Does our method improve comprehensive video understanding across varying video lengths? To evaluate the comprehensive video understanding capabilities of LVLMs across varying video lengths, we leverage four benchmarks: MVBench [3]. VidcoMME [52], MLVU [24]. and Bidn [25]. Together, these benchmarks span a wide variety of perception and reasoning tasks, focusing on objects, actions, their attributes, and holistic video understanding. Among these, MVBench specifically designed for a comprehensive evaluation of short videos, while MLVU and Long VideoBench offer thorough evaluations for long videos. VideoMME provides a comprehensive assessment across videos of varying lengths. As shown in Table 4, consistent improvements are observed across all benchmarks for LongVU and LLaVA-Next. For VideoChat2, self-alignment leads to substantial gains in three out of four setups, with only a minor regression MVBench. Furthermore, RRPO consistently outperforms DPO, further demonstrating the advantages of fine-grained alignment in enhancing comprehensive video understanding.
Q7. How do perturbations in our data generation pipeline impact the quality of coeepreferred responses? To assess the impact of the perturbations on the quality of non-preferred responses, we conduct in-depth analyses and present the results in Table 5. Our key observations are as follows: First, non- non-prefereed responses generated without video perturbations leads to diminished model performance, likely due to reduced generalizability. Second, temporal perturbations alone are ineffective, although their combination with masking significantly boosts performance. Among the spatio-temporal perturbations, random shuffling with masking (RS-Mask) improves performance for models processing fewer frames (e.g.. VideoChat2) whereas local shuffling with masking (LS-Mask) proves superior for models handling longer sequences (e.g., Long VU, LLaVA- LLaVA-Video).
8

---

Q8. How does our method scale with training data? To test the impact of scaling the amount of data, we perform an experiment by varying the number of training samples from 5K to 20K, incrementing by 5K. As shown in Table 6, performance improves with data size. This suggests that our data-generation pipeline is effective in producing high-quality training samples for self-alignment.
Q9. How does performance vary with the number of input frames? We investigate the effect of scaling the number of visual input frames during both inference and self-alignment training. For inference, we evaluate LLaVA-Video using 32, 64, and 128 frames across TVBench, MVBench, and LongVideoBenchh The results are presented in Table 7. Our key observations are as follows: First, RRPO consistently improves performance over the base model across all setups. Second, neither the base model nor RRPO exhibits performance gains beyond 64 frames on TVBench and MVBench. This is likely due to the short-video nature of these benchmarks, where higher frame counts result in redundant frame resampling. However, for long-video understanding, increasing frame counts yields performance improvements, particularly for RRPO with a 1.2% improvement compared to the base model's 0.2% gain. Subsequently, we explore the impact of increasing the number of frames during self-alignment training. Specifically, we raise the frame count from our default 16 to 32. The results presented in Table 4 demonstrate that this increase enhances performance. Notably, we observe a consistent performance improvement fine-grained temporal understanding tasks, evidenced by the gains on TVBench and TempCompass.
Q10. Does our method retain its performance advantage with subtitles? Subtitles generally enhance video understanding by providing additional language cues that LVLMs can leverage. Thus, we investigate whether our method maintains its benefits over base models when subtitles are included. As shown in Table 8, our method demonstrates consistent improvements across all setups.
QII. Does our method generalize across diverse LVLM architectures and training setups? Given the rapid evolution of LVLMs, we carefully select VideoChat2, LLa Video,, and Long VU as the base models to cover a wide variety of design choices and training methodologies (e.g.. Table 4). Specifically, VideoChat2 uses the UMT [63] video encoder, while LLa and Long VU use DINOv2 [64] and SigLIP [65] as their image encoders. On the other hand, LongVU employs dual vision encoders, unlike the others. Moreover, the cross-modal adapters range from query-based based for VideoChat2 to MLP projections for LLaVA-Videe and a combinations of both in the case of Von Furthermore, the overlap between self-alignment training samples and instruction tuning data differs across models, with VideoChat2 having the highest overlap and LLa Video having the least. Importantly, our self-alignment consistently improves performance across these diverse setups, even when reusing instruction tuning data.
Related work
Recent years have seen a surge in the development of LVLMs with improved video understanding
capabilities [1. 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]. This progress can be attributed to several key factors: (1) the development of diverse vidco benchmarks [3, 7, 66, 43, 67], which enable LVLMs to follow human instructions and reason across variety of video tasks; (2) architectural innovations that support the use of rich, dense visual features and enable efficient processing of long sequences [21, 8, 1, 68, 69, 70, 5, 71]; and (3) advancements in training algorithms for both the pre-training [72, 63, 65, 73] and post-training [74, 75, 31] stages. LVLM training generally involves multi-stage processes [76. 77], with pre-training typically focusing on representational alignment between video and language [3, 78, 36), and post-training refining the model's ability to follow instructions [3, 78, 79], reduce hallucination [53, 27], improve reasoning skills [80, 81 and align the model with human preferences [80, 81]. Our introduced RRPO is post-training alignment method designed to enhance the overall video understanding capabilities of LVLMs. While concurrent works [82, 83, 54] have explored post-training alignments of LVLMs, they directly adopt DPO [31], which proves ineffective in fine-graine -ne-grrieedd alignment, as discussed in Section 5.
7 Conclusion
To improve the video understanding abilities of LVLMs, we design self-alignment framework that enables LVL.M to learn from their own errors. These errors commonly occur due to their lack of spatio-temporal reasoning, over-reliance on linguistic cues, and spurious correlations between

---

co-occurring concepts, among others. To effectively align LVLMs against such errors, we introduce RRPO, a novel preference optimization method designed for fine-grained alignment through refined reward modeling with strong regularization. Our in-depth experiments and analyses show that RRPO training more stable and highly effective compared to prior and concurrent preference optimization methods across diverse tasks. Moreover, the fine-grained reward modeling in RRPO improves capabilities without causing significant divergence from the base models. Our proposed self-alignment with RRPO exhibits consistent improvements across all setups over the base models, effectively reducing hallucination and improving spatio-temporal reasoning, thus enabling safer and more reliable of LVLMs. Moreover, show that the approach scales well with more data and righ-resolution temporal inputs, and generalizes well across diverse LVLM architectures and training setups. Future work can further investigate iterative self-align ooff-aligmment methodologies with RRPO, moving beyond the static dataset used in this work.
Acknowledgment
We thank Ahmad Beirami for his valuable feedback and suggestions, which helped improve our paper. We also thank the Bank of Montreal and Mitacs for funding this research, and the Vector Institute for providing computational resources.
References
[1] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410. 17434, 2024. 7,9,
[2] Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.15844 2024. 1,9
[3] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark.
In CVPR, pages 22195-22206,, 2024. 1, 5, 6, 7, 8,
[4] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:231 1714443 2023. 1,
[5] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.074776 2024. 1, 7,9
[6] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang. Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. 1,
[7] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, pages 14313-14323,
2024. 1, 9
[8] Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang. Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024. 7,9
[9] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 1,5, 7,9
[10] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023,
1,7,9
[11] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser,, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv: 053.0553 2024. 1, 7.9
10

---

[12] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, al. Intervvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages
2024. 1,9
[13] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:2412.05271, 2024. 1,9
[14] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2, 2023. 1,9
[15] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13922
2025. 1,9
[16] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings multimodal llms. In CVPR, pages 9568-9578,
2024. 1,2
[17] Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tvbench: Redesigning video-language evaluation. arXiv preprint arXiv:2410.07752, 2024. 1,
2, 6, 8
[18] Pooyan Rahmanzadehgervi. Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. ACCV, pages 18-34, 2024. 1,2
[19] Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic ballucinations in large video-language models. arXiv preprint
arXiv:2406. 16338, 2024. 1, 6, 8
[20] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.0m 0044670 2024. 6,8
[21] Peiyuan Zhang, Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. /6552, 2024. 1, 7,9
[22] Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, and Xizhou Zhu. V2pe: Improving multimodal long-contex capability of vision-language models with variable visual position encoding. arXiv preprint arXiv:2412.09616 2024. 1,2
[23] Hongchen Wei and Zhenzhong Chen. Visual context window extension: A new perspective for long video understanding. arXiv preprint wrXiv:2409.20018, 2024. 1,2
[24] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 206.:224060 2024. 1, 6, 8
[25] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. NeurlPS, 37:28828-28857, 2025. 1, 6,
8
[26] Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Lu Cheng, and Mengnan Du. Large vision-language model alignment and misalignment: A survey through the lens of explainability. arXiv preprini (5ti::2000.01:444 2025. 2
[27] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Ank, and Tomas Pfister. Data-augmenter phrase-level alignment for mitigating object hallucination. arXiv preprint arXiv:2405.1 200.24055........ 1865,, 2024. 2, 4, 6, 9, 17
[28] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00
2023. 2
[29] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modali- ties in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11111
2024. 2
11

---

[30] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self alignmeitnnnt of language models from scratch with minimal human supervision. NeurlPS, 36:251 1-2565, 2023. 2
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 1.553328--3741, 2023. 2, 3, 9,
[32] Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijic Deng, and Dong Yan. 3d-properties: Identifying challenges in dpo and charting a path forward. arXiv preprint
wxXi:2246.003377 2024. 3
[33] Zhichao Wang. Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407. /62 16, 2024. 3
[34] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. arXiv preprint arXiv:2404.J 1999, 2024. 4, 6, 17
[35] John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In ICML, pages 1889-1897. PMLR, 2015.
(36) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023. 5,9
[37] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uni- formerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211.09552 2022. 5
[38] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pages 5842-5850. 2017. 5
[39] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0633. 2023. 5
[40] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0544 05424,, 2023. 5
[41] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901-2910, 2017. 5
[42] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, pages 9777-9786, 2021. 5
[43] Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995-19012. 2022. 5,9
[44] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question amswering. In CVPR, pages 2758-2766 2017. 5
[45] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5
[46] Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. 5
[47] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint 2rXXv:2304.03277, 2023. 5
[48] Lianmin Zheng, Wei-Lin in Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with me-bench and
chatbot arena. NewrIPS, 36:46595-46623, 2023. 5
12

---

[49] Cheng- Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023. 5
[50] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR,
1(2):3, 2022. 5, 21
[51] Chaoyu Li, Eun Woo Im, and Pooyan Fazli. Vidhalluc: Evaluating temporal hallucinattions in multimodal large language models for video understanding. arXiv preprint arXiv:2412.03735,
2024. 6, 8
[52] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprini arXiv:2405.21075,
2024. 6.8
[53] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu. Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. CVPR, pages 13807-13816, 2024.
6, 6,9,15 16
[54] Rui Li. Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung- Temporal preference optimization for long-form video understanding. arXiv preprint 0rXiv:2501.13919 2025. 7,
[55] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.00 01 800, 2024.7 7
[56] Zhijian Liu, Ligeng Zhu Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi. Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint PrXiv:2412.044666 2024. 7
[57] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024.
[58) Y Zhang, B Li, H Liu, Lee, I. Gui, D Fu, J Feng. 7. Liu, and C Li. Llava-next: A strong zero-shot video understanding model. 2024.7
[59] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zchui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. NewrlPS, 11111977--99995 2024.
[60] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.7 7
[61] Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen. Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yampeng Li, al. Aria: An open multimodal native model. arXiv preprini arXiv:2410.05992 2024. 7
[62] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088. 2307707773777 2023." 7
[63] Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, pages 19968-19960, 2023.9 9
[64] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin Bl-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. 2023.
[65] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, In ICCV, pages 11975-11986. 2023. 9
[66] Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, and Serena Yeung-Levy. Video- star: Self-training enables video instruction tuning with any supervision. arXiv preprint arXiv:2407.06118 2024. 9
13

---

[67] Max Bain. Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1728-1723 1738-173,, 2021. 9
[68] Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, pages 18221-18232, 2024. 9
[69] Xinhao Li, Yi Wang. Jiashuo Yu, Xiangyu Zeng. Yuhan Zhu. Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Sideochat-flaste Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00577 2024. 9
[70] De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localizaton assistant. In ECCV, pages 202-218. Springer, 2024. 9
[71] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint Nel:2406.094113 2024. 9
[72] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng. Jilan Xu, Zun Wang, et al. 1ntervideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403. 1437,7 2024. 9
[73] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, PmLR, 2021. 9
[74] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Neur/PS, 30, 2017. 9
[75] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707 06347707.6777.. 2017.
[76] Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Foundation models for video understanding: A survey. arXiv preprint arXiv:2405.03770, 2024.
9
[77] Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint arXiv:23/2.17433. 2023. 9
[78] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892-34916, 2023." 9
[79] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpoo vision-language models with instruction tuning. arXiv preprint arXiv:2305.00
2023. 9
[80] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025. 9
[81] Yi-Fan Zhang. Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv preprint arXiv:2502.
2025. 9
[82] Yogesh Kulkarni and Pooyan Fazli. Videosavi: Self-aligned video language models without human supervision. arXiv preprint arXiv:2412.0062. 2024. 9
[83] Daechul Ahn, Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. i-srt: Aligning large multimodal models for videos by iterative retrossctseeective judgment. arXiv preprint arXiv:2406. 12280, 2024. 9
[84] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Ii,, and Yuxiong He. 7ero-offload: Democratizing billion-scale model training. In USENIX ATC, pages 551-564, 2021. 21
[85] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning. In SC, pages 1-14, 2021. 21
14

---

Appendix
A RRPO Pseudocode (PyTorch Style)
import torch
import www.h.n.. mn. functionnal F
def (spo,, logits, ref ref_logits, phrase_ids, alpha, beta);
logits:
logits from pi_theta.
shape: (batch_size, Lengthh vocab_size)
ref ref_logits:
logits from pi_ref.
shape: (batch_size, sequence_ _eanghh vocab_siza phrase identifiers where tokens belonging to the phrase share the same value; additionally, they remain the same between correct and misaligned phrases to maintain correspondence.
phrase_ids:
shape: (batch_size, sequence_lengtin indices of the correct responses in batch indices
correct_idx:
wrong_idx:
wrong responses batch coefficient to control the token-wise KL divergence coefficient to control reward/penalty
alpha:
beta:
# compute log probabilities
logps logits Srcits.log.ae www.:looooofooooeoooo tmax soft (dim=-1)
ref ref_ps = ref _ef_loogts..o sof tax (dim=-1)
ref ref_logps ref ref_
# compute token-wise KL divergence token_wwit sisto__wiee_____ (ref_ps (ref_logps logps)) sum (dim=-1)
# compute the margin bogps_margine = logps ref _ogpps
# accumulate log probabilities of the phrases with key concepts
# here 0 indicates ignored tokens
umique_phrase_ion _dd torch. unique (phrase__ds ids, sorted=True) [1:] phrase_logps_marge
torch. zeros(
i, phrase_id enumerate (umique_phrase__dde
phrase_ids size (0), len
mask (phrase_ids == phrase_ise id) ff)oat()
phrase_logps [icgs__aargi[;, i] (logps_margil mask) sdm(din=-))
chosen, lastee__ogps__arg__ parase_logpe p [arraainloooreettttt
lerected__ogps_meeee lassen_oogps_maas
(wrongn[wwoo_dddd idx]
logits
logits_margin margin
rejected_logos_martee margsinmannnnnn_. sum (dim=-1)
tlosen_token.o wise kl
token_ kl. sum (dim=-1) [chosen_idx]
losses = -torch logsigmoiddbota * logits_margin)
alpha chosen token_wise_ke
return losses
15

---

59
RRPO
DDPO
58
TDPO
DPA
57
DPO
Accuracy
56
+
54
53
Baseline
10
15
KL Divergence
Figure S1: Comparison of characteristics among preference optimization methods. Both DDPO and DPO diverge significantly from their initial state after alignment. While TDPO and DPA are effective in restricting model divergence, they are less effective in improving performance. RRPO achieves excellent performance with minimal divergence from the base model.
B Comparing Loss Formulation of RRPO and Other Methods
This section presents comparative analysis between RRPO and other preference optimization methods studied in this work, i.e., DPO, DDPO, TDPO, and DPA. We begin by outlining the respective loss functions, followed by detailed discussion of their similarities and differences.
Comparison with DPO.
As previously discussed, the DPO [31] loss function defined as:
Coro(To: Feet) E log o (ro(x,y") -o---roa -ro(x.))))
(S1)
T (x) B log To (1c) Teer (x)
- log BBlog Ther (b++)x)))
The DPO loss function calculates reward over all tokens in y and y despite the fact that there might be a few sub-sequences that are conceptually different. This approach results in coarse-grained reward modeling, and because penalizes tokens in the response, the loss function accumulates large gradient and tends to diverge significantly from the base model, potentially resulting in weak alignment, as shown Figure $1. RRPO is introduced to address two key challenges of DPO: to provide fine-grained feedback and restrict the divergence of the model away from its initial state.
Comparison with DDPO.
DDPO [53] extends DPO by incorporating a weighted reward based on the sub-sequence level differences between + and y-, and is defined as:
LODPO(Te; Reat) -E log o ( [Blog To (x)
To (r) Teel (y (20)
9 log
(S2)
where log T ())())= = Σ pogp(eeaa Jr, Ves) is modified as:
yoEy
log TI (y/x)
Σ
logp(yslz, +
Σ
log p(y: (1, (r,)))
Here, Usame and jdifferent indicate unchanged and changed segments between y and y Moreover, 7 is weighting hyperparameter, and larger ry indicates more weight those changed segments.
16

---

While DDPO is designed to provide fine-gra tne-gnrined feedback, its loss formulation is not as effective as RRPO and is also prone to diverging far from its initial state, similar to DPO, due to weak regularization, see Figure $1.
Comparison with TDPO.
TDPO [34] is also derived from DPO, incorporating an additional regularization term (mxkL) between Te and Tref which is defined as:
(TD nee)) = E log a (re(z, - r
(S3)
a TllTL (r, Mwi TTee sg (3DTKL (x, Mes Tref Terllmm [I))
As shown Figure S1, TDPO is effective restricting the divergence of the base model, but its performance is almost the same DPO and falls short of our RRPO.
Comparison with DPA.
DPA [27] is a phrase-level alignment method unlike DPO and its variants. The DPA loss is composed of terms where the first computes the relative log-probability between phrases of y² and y and the second term works as a regularizer between To and Tres formulated as:
1 N
P(y
LDPA Tret) =
E
log P( P(y.) + a- Drra (I,y yt; Teee
(S4)
1=1
where P(y)) and P(y.) ) denote the probability i-th phrase in y* and y Assume, y is expressed as a sequence of tokens (41, ..,.....4.i) then the probability of i-th phrase can be computed as
II Na (1,la, where s; and e; represent the start and end token indices. =
The loss formulation of RRPO draws inspiration from DPA to achieve fine-grained alignment without the risk of divergence. However, identify fundamental limitation in DPA: it directly adjusts the probabilities Ty to modify the probability ratio between preferred and non-preferred phrases. This approach leads segments, as subsequent segment probabilities become dependent on their preceding elements. Therefore, DPA is accurate in providing fine-grained feedback for sequences composed of multiple sub-sequences of key concepts. As shown in Figure S1, DPA, while successful in controlling divergence, is ineffective improving performance.
inaccurate probability ratio after the initial pair of preferred and non-preferred
17

---

C
Additional Results
Statistical analysis between RRPO and DPO. We perform statistical analysis between RRPO and DPO aligned model. We consider the performance (denoted as Score) difference, whether an improvement or decline, of the Model1 relative to the Model2 on specific task be statistically significant if the Adjusted A exceeds zero, where A denotes the difference in performance between Model1 and Model2. Statistical significance assessed using the Standard Error (SE) at a 95% confidence level. The corresponding mathematical formulations are presented below.
Scorewadell (1 Scorewode12)
SE
number of samples
= Scoreyode11 - ScoreNode12:
Adjusted A A 1.96 SE.
The results presented in Table SI shows that RRPO almost always outperforms DPO and even in several cases improvements are statistically significant.
Table S1: Statistical analysis between RRPO and DPO variants. Green underline indicates that the performance improvements statistically significant. Also, there instances in which the DPO variants achieve statistically significant gains over RRPO.
TV
TempCo-
Video
Vid
MV Bench
Video MME
MLVU (M-Avg)
LongVideo Video Benchyal
Model
Bench
mpass
Hallucer
Hallue
+ide
45.7
60.0
22.1
72.4
59.7
43.0
47.4
41.0
VideoChat2:u+RRRPP
45.8
60.2
32.9
76.4
59.1
44.3
47.9
42.8
+La
51.9
66.4
53.3
76.5
60.2
63.1
67.4
59.4
LLaVA-Videom+RRRFO
51.9
66.8
55.7
76.5
62.0
64.5
69.1
60.4
LLaVA-Videosa Video==
51.1
66.1
50.6
76.3
60.6
65.6
68.9
60.1
L. LLaVA-Videom
52.2
67.4
55.8
76.6
62.0
63.5
69.4
61.3
Long DDngVUp+DDPO
54.3
64.3
40.9
68.5
65.8
56.6
63.6
49.4
LongVUnp
56.5
64.5
44.0
71.7
66.7
57.7
64.5
49.7
18

---

Detailed results. This section details the results for the subcategories of the evaluation benchmarks used in our study.
Table S2: Detailed results on TempCompass.
Models
Caption Matching
Captioning
Multi-choice
Yes-No
Avg.
VideoChal27
69.5
46.6
58.0
63.0
59.3
RRPO
73.2
48.5
56.6
62.6
60.2
Lidvavideo
75.1
50.2
67.6
71.0
66.0
RRPO
75.8
52.0
68.6
70.9
66.8
+ RRPO (32f)
76.6
53.0
71.3
67.4
Von
74.7
49.8
67.3
63.9
+ RRPO
75.2
50.6
64.7
67.4
64.5
Table S3: Detailed results VideoHallucer.
Model
Object relation
Temporal
Semantic
Factual
Non-facture
Avg.
VideoChat2.n + RRPO
47.5
8.0
38.5
1.0
20.5
23.1
53.5
24.0
55.0
5.0
27.0
32.9
LlavaVideo
66.0
56.5
65.5
50.0
+ RRPO
65.5
65.5
71.0
23.5
53.0
55.7
+ RRPO (32f)
65.5
65.5
71.5
23.5
53.0
55.8
LongVUn
50.5
46.0
43.0
17.0
39.5
39.2
+ RRPO
53.0
48.0
50.0
26.0
43.0
41.0
Table S4: Detailed results on VidHalluc.
Model
BinaryQA
MCQ
Scene Transition
Avg.
VideoChal2.
66.8
84.9
68.2
73.3
-RRO
72.7
85.5
70.9
76.4
Llava Vidcor
77.9
91.4
60.6
76.6
RRPO
78.4
91.6
59.5
76.5
RRPO (320)
78.6
91.7
59.5
76.6
VongVUns RRPO
71.4
87.0
43.4
67.3
74.2
88.2
52.7
71.7
Table S5: Detailed results on VideoMME.
Model
Short
Medium
Long
Avg.
VideoChat2.p
49.0
38.6
35.6
41.0
+RRPO
52.2
41.9
38.8
44.3
LlavaVideo>a
76.3
62.8
52.8
64.0
RRPO
76.6
63.1
53.8
RRPO (32f)
76.7
62.9
53.9
64.5
LongV V RRPO
66.1
54.7
47.9
56.2
67.7
55.0
50.3
57.7
19

---

D Additional Details of Training Data
Prompt templates.
The instructions used in processing open-ended generated responses employing GPT4o are presented in Figures S2 and S3.
X {Prompt used apen Thoroughly read the question and the given anavers.
open-anded anded reapanse processing stage 1}
Your task is daternine whether the "Pradicted anszer" is "Correct" "Wrong" based the "Question" and "Reference anaver".
To determine correctnese, focue the aspects in the objects, actions, and their attributes, anong others.
such
The "Predicted
partial information comparison "Reference anaver", in that case check whether at least the partial information fully verified
Please reepond with any the following and nothing elee:
"Correct" if the predicted ansver is correct based an the reference ansver. "Wrang" predicted anaver
sure fully based
"Undecided" if you
correctness.
Question: (question)
Reference (ground_truthh truth)
Predicted anszer: igenarated.reaponnse
Figure S2: Prompt used in open-ended response processing stage 1.
X {Prompt used ondndended response processing T*rTurn 14*.
2}
Identify the key differences betreen these two sentences. identify
on key aspects such as objacta, actions, and their attributes, among others. If there are key difference betveen these tuo sentences, with "None" " nothing else.
sentences,
please
Sentence (sentence, tron. Sentence 2: (sentence_iront sto.tee.tooonnnnntttteeeeeee soo.reoo....ooooossoeeeesss.ss
*exTurn 20x*
Please resrite the "Sentence incorporating the differences you nentioned carlier. Your final response should contain only the revised nothing alse.
(sentence, crttsenceerroo_grronoe www.eerodoooooorrrrrnorr.
Figure S3: Prompt used open-ended response processing stage
20

---

E Implementation Details
Training hyperparameters.
Table S6: Details of training hyperparameters.
VideoChat2 ALLVA-Vider
LongVU
LLM
Mistral
Qwen2
Qwen2
Vision encoder
UMT
SigLIP
SigLIP+DINOv2
Trainable module
LoRA in LLM and everything else is kept frozen
LoRA setup [50]
rank=128, alpha=256
Learning rate
2e-5
5e-6
5e-6
Learning scheduler
Cosine
Cosine
Cosine
Optimizer
Adam
AdamW
Adam
Weight decay
0.02
0.0
0.0
Warmup ratio
0,03
0.03
Epoch
1
Batch size per GPU
1
Batch size (total)
32
32
32
(loss coefficient)
0.01
0.01
0.05
(loss coefficient)
0.9
0.1
0.5
Memory optimization
Zero stage [84, 85)
FSDP
Licenses of existing assets used.
VideoChat2 (Apache License 2.0): http:///wngnnnnocc.oooooo//////a Mistral_
LLaVA-Video (Apache License 2.0): http://wuuninnneeeccccccommoooir
LongVU (Apache License 2.0): http::/wwneeiiiieeeeevvvvoooooct
2 VideoChat-IT (MIT): http:///wwweiiiooooooooo///////a
F Broader impact
As generative models are increasingly deployed in real-world applications, there is growing need for post-training methods that enable fine-grained alignment with human preferences and values. Our proposed method, RRPO, can be applied to align generative models in both language and multimodal settings. By facilitating more precise alignment, RRPO has the potential to improve the safety, reliability, and usability of these models for real-world usage.
G Qualitative Results
We present several examples in Figures $4 S7 highlighting the effectiveness of RRPO the base model and other preference optimization methods (e.g., DPO) diverse video understanding tasks. We also present some failure cases in Figures S8 S9.
21

---

Question The person usds ma iillee similrr cbjects play occiusioo on gome. game porsonts
Queation: Chorse citsely mateiee video.
Riddon dbject
VideoChar2 The video chows car loking paw and then two people liting secights.
VideoChat2 DPO. The vidao showss a cal lilking paer and than taoo passle (ilinng welghts
(A) thind
(8) Under the second object from the lont
licking paw people
(C) Under the linat stjact from the lell.
VideoChat2:
objectt
VideoChat2 ARPO:
Question: What de the person do fins?
Question: Choose the caption that closcly matches the vidco.
Optins::
LLaVA Video: The mon above ls juuu rope ot steady pooo.
Video The mon jumpiig ropo steady papo. jumpping
the dlethes.
speed.
LLaVik-Vider Video: (A) meddd phereetormmn..
LLaVVA-Videe DPP (A) mell the phoneetaamans.
LLaVAA-Vides
Trmas cisthea.
Question: What's the right opdon for how the socnes the vidco chango?
Question: Choose the sopiio that cloccly matches the video. LongVUt putting then grab DPO: RRPO:A takkng jocket jumping
Optamee: (A) alkorker sectratrr's
LongiV From
LongVUU DPO (A) From alevator aucrulary's offic..
LongVU RRPO: (51 From the secularr's office to the alevatoo..
Figure $4: Qualitative examples on fine-grahe fine-grained temporal understanding tasks.
Question: Bused on the video, should we sut up the sundhwiches a aet wih pltch
Question: Biawd on the vicie, should on rub back and forth with sunndapper or nemover
pat?
candlopee
cleaninng
VideoChat2 + DPO. You, & ed to set sandwidhos tant when pitciing Inet.
VideoChatz's DPO clean rusty pot is recommended to nandooper and rits
VideoChar2 vidwo down nol provide wny information wbrut wetting
VidenChatt2 RRPG: No, aczonding vidwo, we shouldd uow sureppeper nedy
Question: Basod the video, should wipe the front windshiedd when we replacn wpper head??
LLaVA- Wideo. Yoa
LLatts-Videe DPD Yoe LLaVA-Vidoo RRPO:NN
Via Video LLaVA-Vidko RRPO.
Question: ln thene wamen sditing all chair in the viden??
Question: Browd on the vicies, should ow sepp the wite aftert we mukke (LH45 cabile? LongVU extoptin sireet woeety erssimee persoe LongVVU ohiop wins ovenlyy proper
LongVU Yex thene wiman adiing
LongVU
woman shting
LongVU
Figure S5: Qualitative examples on video hallucination tasks.
22

---

Gusstin: appoar performmnoo?
Question: What Eunetionn
that uswd videe??
(A) Kreking the paper look more lin heart.
(C) Stibs addils
skili:
(b) Desthoying the papar.
Culling paper piecaes
(D) Human pyromid SkBls.
VideoChat2: (A) Pole
VideoChatt2
VideoChat2 RRPO: Stille ahills
VideoChat2 DPO, (C) Outting
VideoChat22 RRPOO: (A) Malkinn the pipprr look more like heart.
peecess.
Question: Wiett castoed the sartden bell the wotnn in The videc??
Quessionn: following options
poogle
Options: anide
(A1
[C] She was scared prank.
(B1
(DI Cannot determined.
V.AVViiee (D) Carnoil be distrmined.
(D)33
Video
ba
LLaVA-Video DPO- (D13
LLuVA-Video RRPO: (9)4
Question Whet diwccton motion abozh center?
sheap thing?
(A) zaunderdrrheeen
(5) dowmordd
slopping
(C) uoward
(C) extrig grees
(D)
Long eatinn grass
LongVU: deciciise
Long ealling gross
LongVU DPO (D) deccliise
VU RAPO: numning
Long/U RRPO: (A) countandocorrse
Figure S6: Qualitative examples on comprehensive video understanding tasks (short).
23

---

Daustoon PAB preaay Sherwo silind intte MAYS?
tach bollgia io snddt eiacd of
delect
VileChhdd PREPO Exeruda about Flome uedernadrr
Uy Duce te me comp unt coliiin and uniiee the jungin 24
CC The hourn fut aed Dhe game plameers
VideoChiaa and (DI Da soarch dia man the bluch (VC
CheeeCraaad of HERIO OO Fae anma cimriiall Bee mes and socviess the Jungla for Yours
Bentee What nessed Gennan of do wobe mncour effice abarcce to dooe of water 27?
The at bos dpppid the plact bust bulaanch shining tir in jump.
CC The atiire codiarl stoped abott apanaaci the proue for Poine laae litl time for and high ataagge
LLiRA-Andee: JAl The boke pod the about host be lance cuusing nor to jumpo
LL viA-iidee RRIOI 301 The atilla derlp alopped when vas aproacting the crosaber the leeking lide time othar high jumg adtancc.
xxxx
XXXX
Banntion: wincc celer Kolow HAngAs www.greeeeooooooooooooooooooo perfornnc in
veluo?
Dran 22.-ia dinmond Laliig meach.
Seenry Bargo throoght te insedee ougien
Outore:
Zh 1.
Sondeee Mhy man''
SC Bucauao volaad some wareemee mant and vis triedd
Longett ON Becaus bH goi Info al gigt sormoce
LongiU RIPOO 1S: Aeccher THA accida raa the wrinng perace
Cutore
Bi Comedy
LongeU DPCC (U S0i
Figure S7: Qualitative examples on comprehensive video understanding tasks (long).
24

---

odes
the iicuee codel
and und to
Figure S8: Building on the capabilities of the base models, we observe that RRPO-aligned models may still make mistakes or exhibit hallucinations in detailed video description tasks. For instance, VideoChat2 continues to display similar hallucinations after both DPO and RRPO training, as seen in the base model. In contrast, for LongVU, while both the base and DPO- DPO-aligned models hallucinate, the RRPO-aligned variant avoids such errors. Finally, in the case of F.L. Video, the RRPO-aligned model retains the base model's 's reliable behavior, as neither exhibits hallucinations.
25

---

The peo oaliiii Fraas couitiis la pectingg
SC1 thane eounties cne orp aiterinngg faneen
WitroCrvatt (B) The pous of eow cour thea growengg
Vileochuurd usl
as growing
warr
Site og tf
golden
looking Shacara
Langi DU Kocling presanta
looking
RASSPOET
Bhnndier Artta antted the silen what of of the Ran caried Vovcrts wenen nareel Tase
Figure $9: We also observe that RRPO-aligned models may still exhibit limitations in long video understanding tasks, primarily due to architectural constraints of the base model in processing extended frame sequences, as well computational constraints during RRPO training that limit the use of long video inputs.
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions of this paper are mentioned in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
The limitations are discussed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We present the mathematical derivation of our theoretical claims.
27

---

Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The experimental results are fully reproducible, and sufficient details are shared in the paper. Moreover, we release the code and data through an anonymized repository during the review process for reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: The data and code are shared through an anonymized repository during the review process for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ctipsscc/public//uu guwdee/iee///ee CCodeSubeesiaceecceee simiisssonn ssssnnoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The necessary details for the experimental settings are shared in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We test for statistical significance and consider performance improvements from the fine-tuned model over the base model statistically significant at the 95% confidence level, based on Standard Error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of the computation requirements are discussed in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: stctps://neurr http///eerrrppeeeeeeeeee cccppbblcc/.thhccssuueeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The Broader impact of this work is discussed in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: This work involves finetuning publicly available, off-the-shelf large video- language models (LVLMs) to enhance their performance across diverse video understanding tasks. The base LVLMs
originally trained carefully filtered public datasets and already widely adopted in the research community. Moreover, our finetuning process uses open-source, carefully curated training data. We do not anticipate this process introducing any new safety concerns.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Licenses for existing assets used in this work are mentioned the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi paperswithcode thcode. com datasets
31

---

has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, introduce code, model, and data in this paper, which available via an anonymous link during the review period and will be publicly released afterward. We also provide sufficient details of dataset/code/mooel as part of our submission throughout the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This work does involve the use of LLMs in any way that affects the core methodology, scientific rigor, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy bttpp::/neeuips... http://eeerrips.ccoooeeeeeeeeeee 220/ for what should or should not be described.
33


Reviews:
[{"review_id": "FkIA9L8BZ5", "reviewer": "Reviewer_xves", "summary": "This work proposes a self-alignment framework for large vision-language models called RRPO, which enables them to learn from their own errors without any human-annotated preferences. Rather than relying on manually labeled preference pairs, RRPO generates spatial and temporal perturbations of video inputs, uses an external LLM to score and rewrite outputs, and applies sub-sequence-level token-wise KL-regularized rewards. RRPO demonstrates significant gains on hallucination reduction, temporal reasoning, and both short- and long-video understanding benchmarks across multiple LVLM architectures. It also provides tight gradient-norm bounds that justify its stable, efficient training and allow higher learning rates.", "questions": "My main concerns are in the method section, please see the weakness above.\n\nAdditionally:\n1. Does the fine-grained focus and reward-hacking token-wise KL regularizer result in a trade-off between hallucination suppression and general language-vision capabilities, where the model overfits to hallucination-related metrics, achieving high scores without genuinely improving visual grounding, since the gains are primarily on hallucination benchmarks?", "limitations": "The discussion of this work's limitations seems brief and insufficient, as it does not properly address the failure modes of the approach or the trade-off between fine-grained and coarse-grained learning.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to understand.\n2. The fine-grained motivation is interesting and appears to complement DPO’s coarse-grained training approach.\n4. Comprehensive evaluation across three diverse LVLMs and six benchmarks (temporal reasoning, hallucination, short- and long-video understanding) demonstrates consistent performance gains.\n5. Theoretical analysis about RRPO’s gradient norm justifying its stability during model training.\n6. Code, data, and model weights are provided to ensure reproducibility. \n\nWeaknesses:\n1. Synthetic perturbations may not accurately capture real-world error distributions, risking overfitting to artificial failure modes. Since this strategy is not explicitly tied to actual model failures, it can lead to a distribution misalignment between the generated negatives and the model’s true hallucination behavior.\n2. The requirement for non-preferred responses to maintain structural similarity with the original responses may hinder the model’s ability to generalize to diverse response patterns.\n3. Experiments on failure cases or domain shifts are not thoroughly analyzed.\n4. The discussion of this work's limitations seems brief and insufficient.", "was_revised": true, "final_justification": "Strengths:\n1. The paper is well-written and easy to understand.\n2. The fine-grained motivation is interesting and appears to complement DPO’s coarse-grained training approach.\n4. Comprehensive evaluation across three diverse LVLMs and six benchmarks (temporal reasoning, hallucination, short- and long-video understanding) demonstrates consistent performance gains.\n5. Theoretical analysis about RRPO’s gradient norm justifying its stability during model training.\n6. Code, data, and model weights are provided to ensure reproducibility. \n\nAfter rebuttal:\nI was very impressed and satisfied with how authors addressed my concerns regarding experiments on failure cases or domain shifts. Others questions are also clearly resolved, thus I decided to raise my score.", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful feedback. We are pleased to see the overall positive remarks, especially regarding the clarity of writing, the motivation behind fine-grained reward modeling, the comprehensive evaluations, and the theoretical analysis. We have carefully addressed all points below, please let us know if any questions remain.\n\n>**W1. Can synthetic perturbations in generating negative responses capture the model’s true hallucination behavior?**\n\nOur experiments on the data generation pipeline  (see Sec. 5, Q7) show that mild perturbations are particularly effective in generating negative responses that capture true hallucination and error patterns without introducing unrealistic artifacts such as overfitting to artificial failure modes or misaligned negative responses. As a result, our method improves accuracy by 4.8% to 9.8% on hallucination tasks. Our study further reveals that the absence of any perturbations leads to diminished model performance due to poor generalizability (please see Table 5). \n\n>**W2. Could the requirement for non-preferred responses to maintain structural similarity with the ground-truth responses (for fine-grained alignment) limit the model’s generalizability to diverse response patterns?**\n\nMaintaining structural similarity is not restricted to only the non-preferred responses. In principle, we could rephrase the ground-truth responses to match the semantic structure of the non-preferred ones for fine-grained alignment. However, we adopted a more conservative approach, keeping the ground-truths fixed and modifying only the non-preferred responses, to avoid potentially noisy alignment targets. To further enrich error diversity multiple responses can also be obtained for one original sample with varying perturbations and sampling temperatures. We avoided this direction in order to stay within a fixed computational budget, where we opted for sample diversity across instances rather than generating multiple variants per instance. Our results in Table 3 show that our approach for generating negative responses is effective and generalizes well across a variety of video tasks including comprehensive short- and long-video understanding, hallucination tasks, and fine-grained temporal reasoning.\n\n>**W3. Experiments on failure cases or domain shifts.**\n\n**Failure cases**: Examples of failed cases are available in the root directory of our github repository (please see the link in the main paper) under a file named FailedCases.md. We notice that depending on the capabilities of the base models, RRPO-aligned models can still make mistakes or exhibit hallucinations in detailed video description tasks. We also observe that RRPO-aligned models may still exhibit limitations in long video understanding tasks, primarily due to architectural constraints of the base model in processing extended frame sequences, as well as computational constraints during RRPO training that limit the use of long video inputs. \n\n**Domain shift**: Given that the underlying general-purpose LVLMs are trained on massive web-scale corpora during their pre-training and post-training phase, including millions or billions of diverse videos and images, and trillions of language tokens, drawing a clear boundary around what constitutes a “shift” is non-trivial. Moreover, in many cases, the details of the pre-training datasets are not public. Nevertheless, we believe that the diverse set of benchmarks we evaluate on, spanning a wide range of video tasks including video hallucination, comprehensive short- and long-video understanding, and fine-grained temporal reasoning, provides a strong proxy for robustness under diverse conditions. \n\n>**W4. Discussion on limitations.**\n\nOur RRPO method introduces fine-grained reward modeling, which requires preference pairs to maintain structural symmetry, unlike the coarse-grained approach used in DPO, which imposes no such constraint. While fine-grained alignment requires an additional step in data preparation, modern LLMs (e.g., variants of GPT-4o, Gemini, LLaMA, Qwen) are increasingly adept at handling such language tasks, making this step relatively easy to automate. More importantly, given the strong performance gains achieved through our fine-grained alignment strategy, we believe this additional step is a worthwhile trade-off. We will include this discussion in the final version of our paper.\n\n>**Q1. Is there a trade-off between hallucination suppression and general language-vision capabilities?**\n\nWhile the gains on hallucination benchmarks are more pronounced than those on non-hallucination tasks, we do not observe a trade-off where reducing hallucinations comes at the cost of general capabilities. In fact, the fine-grained reward modelling in RRPO loss is explicitly designed to suppress hallucinations without compromising overall language-vision performance, unlike DPO. As shown in the table below, RRPO improves performance in both hallucination (VideoHallucer) and general benchmarks (MVBench, VideoMME, MLVU) while DPO improves performance  (while lower than RRPO) on hallucination benchmarks but leads to a drop in general capabilities. \n\n| Model | VideoHallucer $\\\\uparrow$ | MVBench $\\\\uparrow$ | VideoMME $\\\\uparrow$ | MLVU $\\\\uparrow$ |\n| :---- | :---- | :---- | :---- | :---- |\n| LLaVA-Video 7B | 50.0 | 61.1  | 64.0 | 68.6 |\n| LLaVA-Video 7B \\+ DPO | 53.3 | 60.6  | 63.1 | 67.4 |\n| LLaVA-Video 7B \\+ **RRPO** (**Ours**) | 55.7 | 62.2  | 64.5 | 69.1 |"}, {"review_id": "TnUyklY27f", "reviewer": "Reviewer_jPTL", "summary": "This paper proposes a self-alignment framework for Large Video Language Models (LVLMs), where models learn from their own errors via responses to perturbed videos. The key contribution is Refined Regularized Preference Optimization (RRPO), a novel training objective that combines sub-sequence-level rewards with token-wise KL regularization. RRPO improves fine-grained temporal reasoning, reduces hallucinations, and achieves consistent gains over prior alignment methods like DPO, as validated across multiple LVLM architectures and video understanding benchmarks.", "questions": "1. The paper uses a GPT-4o-mini model to assess correctness for open-ended tasks. Is there a risk of bias or alignment drift if this LLM acts as both the judgment oracle and learning target?\n2. The method relies on identifying key concept pairs (y+i, y-i) for sub-sequence reward calculation. How robust is this identification process when scaling to noisy or less structured video-language datasets?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Technical Quality: The paper presents a rigorously designed training framework with clear theoretical and empirical justification.\n- Empirical Rigor: The authors evaluate their method across a broad spectrum of benchmarks covering temporal reasoning, hallucination mitigation, and long-video understanding. RRPO consistently improves over baseline and existing methods with performance gains up to 5.4%.\n- Innovation: The RRPO method combines fine-grained sub-sequence-level preference modeling and token-level KL regularization, addressing key weaknesses of DPO (e.g., coarse reward modeling, training instability).\n- Scalability & Automation: The self-alignment data generation pipeline is automatic and does not require human annotation, allowing scalable construction of (preferred, non-preferred) response pairs from perturbed video samples.\n\nWeaknesses:\n- Clarity and Writing Quality: While the technical content is solid, the presentation can be improved. Some notations are densely packed, and explanations (e.g., around Eq. 6–8) require careful reading to interpret. Moreover, acronyms like RS-Mask, LS-Mask, etc., appear in Table 5 without early clear definitions (though defined in text).\n- Formatting & Style: The manuscript contains minor formatting inconsistencies (e.g., spacing issues in equations and figure captions, long caption texts), which reduce readability.", "was_revised": true, "final_justification": "The paper introduces a novel self-alignment framework for LVLMs with RRPO, which effectively improves temporal reasoning, reduces hallucinations, and outperforms previous methods. All my concerns have been addressed.", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful feedback. We are pleased to see the positive remarks, particularly on the rigorous empirical and theoretical analysis, the innovative fine-grained reward modeling, and the scalability of our proposed approach. We have carefully addressed all points below, please let us know if any questions remain.\n\n>**W1. Densely packed mathematical notations and acronyms.**\n\nDue to space constraints, some mathematical expressions are written in-line, which we acknowledge can reduce readability. We will use the additional page allowed in the camera-ready version to reformat these expressions into separate lines wherever possible to improve clarity.\n\nRegarding the acronyms RS (Random Shuffling), LS (Local Shuffling), and GS (Global Shuffling), their definitions are provided in Lines 190–193 under Section 4\\. Following the reviewer’s suggestion, we will revise the text to make such acronyms clearer and easier to read. \n\n>**W2. Formatting & Style**\n\nWe thank the reviewer for their attention to detail. We will ensure that these minor formatting inconsistencies are corrected in the camera-ready version.\n\n>**Q1. Risk of bias or alignment drift due to the usage of GPT-4o?**\n\nWe would like to clarify that GPT-4o is *not* used as the learning target, so concerns about bias or alignment drift do not apply in our setting. To illustrate the process, consider a video captioning example provided in Fig. 4 of the main paper. Suppose the ground-truth response of a video is: *“The video features a group of people playing soccer on a sandy field.”* Now assume the LVLM generates the response as: *“A group of people is seen playing volleyball on a beach covered in sand.”*. We use GPT-4o to extract the key *incorrect concepts* from the LVLM-generated response, in this case, “volleyball” vs. “soccer” and “beach” vs. “field”. We then rewrite the *ground-truth response* by incorporating these incorrect concepts to produce a closely *aligned* *non-preferred response* (which are used at fine-grained alignment training) such as: *“The video shows a group of people playing volleyball on a sandy beach.”*\n\nIn this way, the model is aligned with the ground-truth or preferred responses, which are obtained from the instruction-tuning dataset while explicitly contrasting its own mistakes. Thus, GPT-4o serves only as a tool for identifying and manipulating semantic differences, not as a learning target. Further details on the instructions used in processing the responses are provided in Appendix D.\n\n>**Q2. Robustness of the identification of incorrect concepts when scaling to noisy or less structured video-language datasets?**\n\nWe find that identifying key concept pairs $(y^+, y^-)$ using LLMs is fairly robust in practice, particularly as current LLMs such as GPT4o variants have become increasingly capable at handling such language understanding tasks. To further ensure reliability of this process, we extract incorrect concepts sentence by sentence, rather than treating them as monolithic texts. Our method successfully processes responses from a range of diverse video-language datasets, including VideoChat, VideoChatGPT, EGO-QA, TGIF-QA, and WebVid-QA, demonstrating its generalizability. While we acknowledge that LLMs can still make occasional errors in this identification process, this remains the most practical and scalable approach. Manual annotation or human-in-the-loop revision, though potentially more accurate, would be prohibitively expensive and infeasible at scale. Given these trade-offs, our approach offers a strong balance between scalability and perfection."}, {"review_id": "WyfrRNzBbW", "reviewer": "Reviewer_uoq4", "summary": "This work introduce Refined Regularized Preference Optimization (RRPO), a preference optimization method that utilizes sub-sequence-level refined rewards and token-wise KL regularization to address the limitations of DPO. The proposed method consistently improves performance over the base models and generalizes well across diverse training setups.", "questions": "See weakness.\n\nIf the authors can effectively resolve the issues I have highlighted, I would be willing to raise my rating accordingly.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method shows promising improvements.\n2. The experimental evaluation is well-executed.\n3. The method consistently improves performance over the base models and generalizes well across diverse training setups.\n\nWeaknesses:\n1. The proposed method appears to be specifically designed for video understanding tasks. It would be helpful if the authors could clarify whether the approach is inherently limited to video inputs, or if it can be extended to image understanding tasks as well.\n\n2. Lacks sufficient qualitative comparison results. Visual comparisons with baseline methods are essential to assess generation quality beyond quantitative metrics.\n\n3. This work lacks an analysis of inference efficiency. It would be beneficial to include comparisons with baseline methods in terms of training time, GPU hours, and inference speed.\n\n4. The method presents interesting results on low-resolution video datasets, and it would be valuable to explore the method’s performance in higher-resolution scenarios. Real-world applications often involve high-fidelity video inputs, which may introduce different challenges such as increased feature complexity or computational demands. It could be beneficial for the authors to discuss potential extensions to high-resolution data in the discussion section, or consider supplementary experiments on datasets with varied visual fidelity.", "was_revised": true, "final_justification": "The authors have addressed all my concerns and my final score is Accept.", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful feedback. We are glad to see the overall positive remarks, particularly regarding the well-executed experimental setup and the generalizability of our approach. We have carefully addressed all points below and would be happy to clarify further if any questions remain.\n\n\n>**W1. Can our approach be extended to image tasks?**\n\nWhile our approach currently focuses only on videos, it can be easily adapted to image tasks with minor modifications. For example, in that case, only spatial perturbations (e.g., random masks to occlude parts of the image) would be applied to generate negative responses, due to the absence of a temporal dimension. In fact, as shown in our ablation study in Table 5 (main paper), spatial perturbation alone is also quite effective even on videos. The rest of the pipeline, including data preparation, reward formulation, and training procedure, remains largely unchanged.\n\n>**W2. Qualitative comparisons.**\n\nQualitative examples are available in the root directory of our github repository (please see the link in the main paper) under a file named QualitativeComparisons.md. These examples further demonstrate the effectiveness of our method in comparison to the baselines on diverse video tasks including fine-grained temporal understanding, comprehensive video understanding, and hallucination. A few qualitative results are also provided in Fig. 1. \n\n>**W3. Training time comparisons and inference speed**\n\nWe present a training time comparison of our method RRPO against other preference optimization baselines in the table below. We use LongVU-7B as the base model and train on a single node of 4 A100 80GB GPUs. As shown, the training times across methods are close to each other, demonstrating that training time required for RRPO is in the same range as others. Minor variations in training time can be attributed to fluctuations in data loading, server load, and other similar factors at the time of training.\n\n| Method | Training time (A100 80GB GPU hours) |\n| :---- | :---- |\n| DPO | 32 hrs. | \n| DPA | 33 hrs. |  |\n| TDPO | 32 hrs. | \n| DDPO | 32 hrs. | \n| **RRPO (Ours)** | 30 hrs. | \n\nWe also evaluate the inference speed of all the methods. As expected, there is no difference across them, since none introduce architectural modifications that would impact runtime. We report inference speed under various generation lengths for completeness, in the table below.\n\nResponse Length|Inference Speed|\n| :---- | :---- |\n|Short| 0.41 seconds per 10 tokens |\n|Medium| 2.95 seconds per 128 tokens |\n|Long| 6.70 seconds per 300 tokens |\n\n>**W4. Evaluation on higher-resolution scenarios**\n\nWe would like to clarify that the evaluation benchmarks used in this work include videos with a wide range of spatial and temporal resolutions. Specifically, all benchmarks include videos with high pixel resolution (see table below). The temporal duration varies by task, for example, long-video understanding benchmarks feature videos of an hour long, while short-video tasks include videos of a few seconds to several minutes long. A summary of video characteristics is provided below. Table 3 in the main paper demonstrates strong performance across all these diverse benchmarks, indicating that our method is effective across various video resolutions (Res.) and durations (Dur.).  \n\n| Benchmark | Res. Max (pixel) | Res. Min (pixel)  | Res. Avg (pixel)  | Dur. Max (sec.) | Dur. Min (sec.) | Dur. Avg (sec.) |\n| ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| TVBench | 1920 | 360 | 1150.36 | 116.02 | 1.24 | 18.84 |\n| TempCompass | 912 | 596 | 633.71 | 35.21 | 1.53 | 11.38 |\n| VideoHallucer | 1920 | 192 | 1073.78 | 623.02 | 0.2 | 80.63 |\n| VidHalluc | 3414 | 160 | 929.53 | 228.49 | 0.6 | 25.47 |\n| MVBench | 1920 | 224 | 907.39 | 116.02 | 1.03 | 15.93 |\n| VideoMME | 1280 | 480 | 1195.26 | 3579.45 | 11.03 | 1021.27 |\n| MLVU | 2922 | 320 | 1183.84 | 32550.1 | 179.97 | 943.02 |\n| LongVideo | 1280 | 674 | 1241.52 | 3573.96 | 7.97 | 459.51 |"}, {"review_id": "PJHPuZ3RAx", "reviewer": "Reviewer_baXK", "summary": "This paper targets at improving Large Video Language Models (LVLMs) and propose a framework to have the model learn from the error they have made. In detail, the authors automatically generates pairs of preferred and non-preferred responses by introducing spatio-temporal perturbations into videos to simulate typical error patterns. Then, they designed Refined Regularized Preference Optimization, which is claimed to offer more precise alignment and more stable training, by utilizing reward at sub-sequence level. Both mathematical derivation and experimental validation has been provided to demonstrate the superiority of RRPO over DPO.", "questions": "Please see my weakness", "limitations": "Please see the weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strength**:\n- The motivation is important: learning from the errors to further enhance the model itself is important for application usage in the real world, in particular for fine-grained temporal understanding and reducing hallucination. \n- The RRPO is novel and the idea of generating negative sequences by perturbing video is interesting.\n- Detailed and extensive ablation study and discussion\n\n**Weakness**:\n- I am curious whether any capability of LVLM before RRPO is compromised. For example, what is the performance on the LVLM after RRPO on the cases that the model answered correctly before. \n- Though I understand the claim regarding difference between coarse-grained and fine-grained level difference, but I am not sure whether this difference can be mitigated even with DPO. In other words, whether such difference really matters. Then, for the gradient scale (claim in Line 128), demonstration of the large gradient caused by long sentences is needed.\n- Several important implementation details are missed, e.g., what is the value alpha used in the experiments.\n- As part of future research direction, when the spatial & temporal perturbation is applied, the model prediction on some videos remains the same, which could be problematic. For example, when the key object for understanding is masked, if the model still makes the correct prediction, it can be treated as a hallucination, which is not preferred and should be mitigated. Then, how can your method be further improved to identifying such cases?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful feedback. We are glad to see the overall positive remarks, particularly regarding the scope of this work for real-world applications and the novelty of our method. We have carefully addressed all points below and would be happy to clarify further if any questions remain.\n\n>**W1. What is the performance on the LVLM after RRPO on the cases that the model answered correctly before?**\n\nTo evaluate this, we measure how many samples previously answered correctly by the base model remain correct after RRPO training. Let $B$ be the set of indices correctly answered by the base model, $R$ be the set correctly answered by the RRPO-trained model, and $T$ is the set of all indices in the evaluation set. We define retention\\_score \\= $100-(\\frac{|B|}{|T|} - \\frac{|B \\\\cap R|}{|T|}) \\times 100$ . \n\nAs shown below, both RRPO and DPO retain a very high percentage (97.2-97.9) of these samples across all benchmarks. However, the average improvement in performance in case of RRPO is much higher compared to DPO (2.26 vs. 0.46). \n\n| Method | Avg. retention\\_score (%) | Avg. accuracy improved (%) |\n| :---- | :---- | :---- |\n| DPO | 97.9 | 0.46 |\n| **RRPO (Ours)** | 97.2 | 2.26 |\n\n>**W2. Does the difference between coarse-grained and fine-grained reward matter? Gradient comparison between DPO and RRPO.**\n\n**Difference between coarse-grained and fine-grained reward:** Fine-grained reward modeling is crucial for capturing subtle distinctions in response quality and correctness, especially when the model needs to understand why one response is preferred over another. This has far-reaching implications and is very crucial in domains where even minor errors can have significant consequences, for example, in medical, legal, and safety-critical applications. \n\nDPO provides only a coarse signal by treating the entire response as preferred or non-preferred. It does not indicate where the error occurred or which sub-sequence of the entire response contributed to the preference decision. As a result, the model receives limited guidance during alignment, which can lead to inefficiencies in learning. In contrast, our fine-grained reward modeling in RRPO explicitly identifies and localizes errors, providing more targeted supervision. As shown in Table 3 of the main paper, this leads to consistent and substantial improvements across all setups compared to DPO and other baselines, demonstrating the importance of fine-grained alignment.\n\n**Gradient comparison between DPO and RRPO; Claim in Line 128:** To support our claim in Line 128, we provide a theoretical analysis in Lines 149–174. Specifically, we show that the upper bound of the DPO gradient $\\\\nabla \\_\\\\text{DPO}$ (see Eq. 10\\) scales with the total length of the response pairs as $|y^+| \\+ |y^-|$. In contrast, the upper bound of the RRPO gradient $\\\\nabla \\_\\\\text{RRPO}$ (see Eq. 9\\) scales with the length of the differing concepts i.e., $2NL$ where $N$  is the number of differing concepts between $y^+$ and $y^-$, and $L$  is the average token length of each differing concept.  \nSince $|y^+| \\+ |y^-| \\>\\> 2NL$, the upper bound of $\\\\nabla\\_\\\\text{DPO}$ is significantly larger than that of   $\\\\nabla \\_\\\\text{RRPO}$. \n\nFurthermore, the term $\\\\nabla\\_{\\\\text{D}\\_\\\\text{TKL}}$  in RRPO gradiebt is negative (see Line 167), which makes:\n$\\\\nabla\\_{\\\\text{RRPO}} \\<\\< \\\\nabla\\_{\\\\text{DPO}}$.\n\nWe also provide empirical evidence in Figure 5, which shows that the large gradient in DPO leads to significant divergence from the model’s initial state, unlike RRPO. Importantly, **RRPO not only diverges less but also achieves better performance than DPO**, which is a highly desired trait at post-training.\n\n>**W3. Implementation details, e.g., value of $\\\\alpha$**\n\nAll key implementation details, including training configurations and hyperparameters, are provided in Appendix E. As reported in Table S6, the value of $\\\\alpha$ was empirically set to 0.01 for VideoChat2 and LLaVA-Video, and 0.05 for LongVU. Please note that we also share the full codebase with default configurations to support reproducibility.\n\n>**W4. As part of future research direction, when the key object for understanding is masked, if the model still makes the correct prediction, it can be treated as a hallucination, which is not preferred and should be mitigated? Then, how can your method be further improved to identify such cases?**\n\nAs a potential direction for future work, our method could be extended to identify and mitigate false positives arising from hallucinations. One possible approach is to adopt a guided masking strategy, where we explicitly track which objects or segments are masked. This would allow for more informed verification during training and help detect cases where the model relies on spurious cues. That said, implementing such a strategy may require richer annotations or additional visual grounding models during data generation, which introduces added complexity."}]

Author rebuttal:

Meta-review: This paper improves the DPO algorithm by introducing sub-sequence–level rewards and token-wise KL regularization. Experiments across multiple LVLM architectures and video understanding benchmarks demonstrate strong empirical gains. The authors have satisfactorily addressed reviewer concerns, and these clarifications should be incorporated into the final version. As a general enhancement to DPO, it would be interesting to explore the method’s applicability beyond video understanding in future work.
Decision: Accept (poster)