Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generation
Authors: Xiaomeng Yang
Mengping Yang
GONG JIA
Luozheng Qin
Zhiyu Tan
Hao Li
Venue: ICLR 2026 Poster (2026)
Keywords: video generation
diffusion models
DPO
Primary area: generative models
TL;DR: We propose Dual-IPO, a dual-iteraive preference optimization framework for aligning text-to-video generation models with human perferences.

Abstract: Recent advances in video generation have enabled thrilling experiences in producing realistic videos driven by scalable diffusion transformers. However, they usually fail to produce satisfactory outputs that are aligned to users' authentic demands and preferences. In this work, we introduce Dual-Iterative Optimization (Dual-IPO), an iterative paradigm that sequentially optimizes both the reward model and the video generation model for improved synthesis quality and human preference alignment. For the reward model, our framework ensures reliable and robust reward signals via CoT-guided reasoning, voting-based self-consistency, and preference certainty estimation. Given this, we optimize video foundation models with guidance of signals from reward model's feedback, thus improving the synthesis quality in subject consistency, motion smoothness and aesthetic quality, etc. The reward model and video generation model complement each other and are progressively improved in the multi-round iteration, without requiring tediously manual preference annotations. Comprehensive experiments demonstrate that the proposed Dual-IPO can effectively and consistently improve the video generation quality of base model with various architectures and sizes, even help a model with only 2B parameters surpass a 5B one. Moreover, our analysis experiments and ablation studies identify the rational of our systematic design and the efficacy of each component.

Published as a conference paper at ICLR 2026
DUAL-IPO: DUAL-ITERATIVE PREFERENCE OPTI- MIZATION FOR TEXT-TO- VIDEO GENERATION
Xiaomeng Yang
Mengping Yang!.2t Jia Gong" Luozheng Qin Zhiyu Tan Hao Li!
Fudan University,
SShangaai Academy of Al for Science
ABSTRACT
Recent advances in video generation have enabled thrilling experiences in pro- ducing realistic videos driven by scalable diffusion transformers. However, they usually fail to produce satisfactory outputs that are aligned to users' authentic demands and preferences. In this work, we introduce Iteaiiteeetive Optimiza- tion (Dual-IPO), an iterative paradigm that sequentially optimizes both the reward model and the video generation model for improved synthesis quality and human preference alignment. For the reward model, our framework ensures reliable and robust reward signals via CoT-guided reasoning, voting-based self-consistency, and preference certainty estimation. Given this, we optimize video foundation models with guidance of signals from reward model's 's feedback, thus improving the synthesis quality in subject consistency, motion smoothness and aesthetic qual- ity, etc. The reward model and video generation model complement each other and progressively improved in the multi-round iteration, without requiring te- diously manual preference amnotations. Comprehensive experiments demonstrate that the proposed Dual-IPO can effectively and consistently improve the video generation quality of base model with various architectures and sizes, even help a model with only 2B parameters surpass a 5B one. Moreover, our analysis exper- iments and ablation studies identify the rational of our systematic design and the efficacy of each component.
INTRODUCTION
Video generation Brooks et al. (2024); Kuaishou (2024); hailuo (2024); Kong et al. (2024); Yang al. (2024b); Tan al. (2025) has experienced tremendous advancement in recent years. Their technical breakthroughs, driven by scalable model architectures (e.g., diffusion transformer Peebles & Xie (2023)), web-scale data and massive computation, have revolutionized traditional concent creation in various domains like movie production Zhang et al. (2025), story Zhou et al. (2024) and scene generation Guo et al. (2025), etc. Despite their success, existing video generation models still fall short in producing satr-fatisfied outputs with consistent subject, smooth motion and compelling aesthetic. This identifies a considerable gap between the quality of current models and the authentic demands of users in practice.
To improve synthesis quality and alignment of human preference, post-training techniques like Direct Preference Optimization (DPO) Rafailov et al. (2023); Liu et al. (2025b) and Kahneman- Tversky Optimization (KTO) Ethayarajh et al. (2024) are employed. However, preference learning typically requires training on large annotated datasets of human preferences. which are laborious to construct. Alternatively, external reward models, such as HPS Wu et al. (2023). ImageReward Xu al. (2023), VideoAlign Liu et al. (2025a) and VisionReward Xu al. (2024), can be utilized to evaluate the quality of images and videos for preference annotations. Nonetheless, these re- ward models often exhibit distribution mismatch Kim al. (2024) across different video gener- ation models and evaluation criteria, making the reward signals unreliable for human preference alignment (see Tab. 1). Additionally, aligning video generation models on fixed offline preference datasets could easily cause overfitting and even collapsed to the preference signals Pal et al. (2024);
Equual Contribution, "Corresponding Authors. Our source code is available at https: //gi thub. com SS////SISSSFIISI IPC.

---

Published as a conference paper at ICLR 2026
(c) Human Preference Datsset
(b) Reward Model
(c) Duol-Iterative Optimization
legiress
Caprice
T2V Model
deranns
cretred pretetart Pers baldig toer
Pontame
Generofied Videas
Ganeratiad Video
hage
and
Iferstivee
coniitarcy
Optiniization
Wide mosu
VLM based Reward Model
a The fest vitee beuccatly mertore tect
eakartic
songle
Optimication
viieo Proferencc
Painiis Arautation @ commiriery
VLM-based Critic Model A
S Vise O 1ite The
T2V
Rihhe acter heiling wneed viles here firtt deaoo
apeep the rected ades, bat
Vidao
[Pantelte Annetatice
Teseeneeeeeee with
Leornahin
Frozen
Figure 1: Overview of our proposed dual-iterative preference optimization framework. (a) Human preference dataset used to initially train our reward model. (b) Reward model that learns auto- matically annotate generated videos with preference labels. (c) Dual-iterative optimization, which incorporates the reward model to iteratively optimize video foundation model, thus improving the synthesis quality and human alignment.
Feng et al. (2024), degrading the synthesis quality and model generalization Park et al. (2024); Chen et al. (2024b); Azar et al. (2024). Furthermore, the reward signal should evolve with the model since the artifacts are becoming subtler throughout the training, otherwise the reward might be biased and lead to instable optimization Dubey al. (2024).
To address the aforementioned challenges, this paper proposes Dual-iten haal-iterative preference optimiza- tion (Dual-IPO), post-training framework for improved synthesis quality and human alignment via iterative optimization on both the reward model and the video generation model. For the iterative refinement of our reward model, we propose self-refined preference optimization to unlock stronger reasoning capabilities from Vision-Language Models (VLMs) with a small set of Chain-of- Chain-of-Thought (CoT) guided annotated data, enabling high-quality and efficient reward modeling with minimal supervision. Then, we design voting-based reff-consistency mechanism across multiple inference paths to improve the accuracy and scalability of labeled rewards. Furthermore, a preference certainty estimator is developed to filter low-confidence results, ensuring reliable laeudo-labels for further training of the reward model. Such iterative self-labeling cycle strengthens the reward model's gen- eralization ability and robustness over time, ensuring reliable rewards for training video generation models. Regarding the iterative optimization of vidco generation models, the latest model is used to generate videos in each iteration. These generated videos are scored by the current reward model, serving as updated reward signals for the next optimization iteration. Notably, our reward model supports both pairwise and pointwise preference annotation, facilitating flexible deployment under different optimization settings, i.e., pairwise for Diffusion-DPO and pointwise for Diffusion- KTO. The two component complement each other and together enhance the overall quality and user preference alignment of generated outputs. Besides, only small amount of annotated preferences are required as cold start data for training our reward model. suggesting satisfactory data efficiency.
To testify the effectiveness of our proposed framework, we conduct extensive experiments on video generation models with different architectures (i.e., cross-attention DIT based Wanx Wan et al. (2025) and MMDIT based Cog Video Yang et al. (2024b)), different model scales (ie., 2B and 5B). Experimental results demonstrate that our method reflects substantial performance gains across dif- ferent baseline models in various aspects, including subject consistency, motion smoothness, visual aesthetic, etc. Moreover, the synthesis performance could be consistently improved with our pro- posed multi-round iterative optimization, indicating that both the reward model and the video gener- ation model are progressively refined through each iteration. To sum up, our primary contributions summarized blow:
2

---

Published as a conference paper at ICLR 2026
A dual-iterative preference optimization framework for improving video generation mod- els. By updating the reward model and the vidco generation model a multi- manner, we successfully improve the synthesis quality and human preference alignment.
A self-refined preference optimization strategy for reliable reward models. With CoT-guided reasoning. voting-based self-consistency, and preference certainty estimation, we ensure high- confidence and accurate reward signals iteratively, thus improving the generalization and robust- ness of the reward model. Notably, both pairwise and pointwise rewards are supported, enabling flexible deployment for different alignment strategies.
A multi-round scheme for consistently improvement on video quality. Leveraging these re- liable rewards, we refine T2V models iteratively across multiple aspects (semantic, motion, aes- thetic), consistently leading to better alignment and stronger generalization.
State-of-the-art performance under various settings. Extensive experiments show that our framework significantly advances several baseline models with different sizes. In particular, empower 2B Cog VideoX model to outperform a 2.5x x larger 5B variant, demonstrating superior alignment efficiency and effectiveness.
RELATED WORK
Text-to-video generation. Recent advances in video generation have been largely driven by diffu- sion models Ho et al. (2020); Sohl-Dickstein et al. (2015); Song et al. (2020); Zhang et al. (2022), achieving remarkable improvements in diversity, fidelity, and quality Chen al. (2023); Esser al. (2023); Ho et al. (2022b); Liu et al. (2023a); Ruan et al. (2023); Wang et al. (2025); Xing et al. (2023). Scaling model size and training data further enhance performance Ho al. (2022a); Singer et al. (2022); Wu et al. (2021). In text-to-video (T2V) generation, pretraining from scratch requires substantial computational resources and large-scale datasets Gen-3 (2024); Luma (2024); Kuaishou (2024); Brooks et al. (2024); Kong et al. (2024). Current methods Blattmann et al. (2023); Wang al. (2023); Guo et al. (2023) extend text-to-image models by incorporating spatial and temporal attention modules, often using joint image-video training Ma al. (2024), with Guo al. (2023) proposing a plug-and-pla temporal module for personalized image models. Remarkable visual quality and temporal consistency have been achieved by Zhang al. (2023); Chen al. (2024a), and diffusion transformers (DiT) Brooks et al. (2024); Yang et al. (2024b); Chen et al. (2024a) enhance spatiotemporal coherence and scalability, identifying the great potential of transformer ar- chitectures. Despite their success, they often fail to produce user-preferred videos in practice.
Post-training for human preference alignment. Post-training is widely adopted in Large Lan- guage Models (LLMs)Ouyang et al. (2022); Ramamurthy et al. (2022); Zheng et al. (2023). RLHFOuyang et al. (2022) aligns models via reward functions learned from comparisons. RRHF Yuan et al. (2023b) extends supervised fine-tuning, and SLiC Zhao et al. (2023) uses of- fline RL data. DPO Rafailov al. (2023) improves stability by directly optimizing preference data, while KTO Ethayarajh et al. (2024) adopts a prospect-theoretic view. RSO Liu et al. (2023b) leverages rejection sampling for better alignment. Post-training has also been applied to image foundation models Clark et al. (2023); Prabhudesai et al. (2024a); Xu et al. (2023). For diffusion (2023) extends DPO optimize likelihood. while Diffusion- KTO Li et al. (2025) maximizes expected utility without pairwise preferences. However, post- training for video generation remains underexplored. Existing efforts focus on reward design. In- Viruct Yuan et al. (2023a) proposes temporal decaying rewards (TAR) to prioritize central frames. T2V-Turbo Li et al. (2024) improves ODE solvers using motion priors. VADER Prabhude- sai et al. (2024b) fine-tunes with vision-based reward gradients, boosting generalization. Despite these, gains remain limited, and the full potential of post-training is yet to be realized. In this pa- per, we propose dual-optimization strategy to iteratively update the reward model and the video generation model. They complement each other and together deliver improved output quality and preference alignment.
models, Diffusion-DPO Wallace
3

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1 METHOD OVERVIEW
Fig. presents the overall framework of our method. First, we collect small amount of annotated user preferences for initially training our reward model. In this phase, both pairwise and point- wise annotations are constructed, enabling flexible alignment strategies like DPO with pairwise data and KTO with pointwise data. After training the human preference dataset, we incorporate the reward model and the T2V generation model in the iterative optimization loop, performing dual- optimization to fully unlock their potentials. We detail each component of our method below.
3.2 SELF-REFINED PREFERENCE OPTIMIZATION
We propose self-refined preference optimization (SRPO), self-iterative training loop that progres- sively enhances the reward model with minimal supervision.
CoT-guided annotation. We start by constructing small of Chain-of- annotations, unlocking the potential of a Vision-Language Model (VLM) for structured reasoning and reliable preference construction. Specifically, (2024b) to generate videos from a diverse prompts built by Qwen2.5 Yang et al. (2024a). These prompts span over semantic categories (c.g.. Human, Vehicle, Architecture) and multiple motion types (details are in the appendix). Each prompt yields four videos from different seeds, and the dataset iteratively expanded using an optimized regeneration strategy. We collect both pairwise ranked and pointwise scored preference data along three dimensions: text-video consistency, content faithfulness, and motion smoothness following Tan al. (2024b).
(CoTT) guided
employ CogVideoX-2B model Yang et al.
Self-consistency via multi-path inference. For preference queries with deterministic answers, voting across multiple reasoning paths has been shown to improve reliability Prasad et al. (2024); Wang et al. (2022). Following this, SRPO performs multi-path inference with CoT-enabled reward model to scale the preference data. Concretely, we adopt a self-consistency mechanism that aggre- gates responses by counting answer frequencies, automatically constructing stable pseudo prefer- ence labels and reducing noise from individual paths.
Preference Certainty (PCi While self-consistency improves pseudo-label quality, noise still arise due to randomness in VLM outputs and lack of reasoning trace inspection. To further filter unreliable labels, we introduce a preference certainty estimator (PCE) based on the reward formulation used in DPO Rafailov al. (2023). Given prompt æ, let yo and y denote the preferred and less-preferred videos in a pair. Formally, DPO models the preference probability Zhu al. (2025); Kim al. (2024)
No (aho x)) Po(u 2e ==)=== AAg
-Blog To( x²) Tee (2)
( xx)
where To and &eef the trained and reference models, respectively, and / is temperature param- eter. After DPO learning, the model implicitly learns a reward-like scoring to indicate the model's preference y over the baseline:
T r)
R(y x) = log Terr x))
(2)
We define the average reward across all samples in the current dataset as Q = (a [R æ)). To estimate the confidence that the truly better than the average, we define our certainty metric as:
PCE( xx) Po (Rgg ax)) Q)
Intuitively, this measures how likely the chosen sample is better than typical sample in the cur- rent distribution. We filter out weak or uncertain preferences by only retaining data points where PCE( æ) 0.5. This criterion ensures that the selected preference aligns not just with relative ranking, but also exceeds the global quality threshold, improving label robustness under distribution shift.
4

---

Published as a conference paper at ICLR 2026
Iterative Pseudo-Label Refinement. The refined pseudo-labels are used to train the critic model, which is then applied to newly generated video samples for the next iteration. This bootstrapping process allows the critic to progressively improve its alignment with human preferences and gener- alize to evolving data distributions.
We adopt the DPO objective Rafailov et al. (2023), which increases the preference margin between the preferred sample You and the less-preferred yi given the same imput (c.g., text prompt). The training loss is defined as:
po(
pa Peef (vr
(Dro - -Ra. 20 log log
log
Peee [x)
where a(.) is the sigmoid function, / is temperature parameter, and Pref is frozen reference model. To enhance robustness, we further introduce the preference certainty estimator (PCE), de- fined as:
yy x)) = Fo (R(y xx) Q)
(4)
where R(y x) = log Dolyl mv)
is the learned reward score, and Q = E [R(y x)] is the mean reward over the dataset. Consequently, the final loss incorporates PCE as confidence weight:
Pootlyla)
(o) Rar.y... [PCH()nn .)) (f)o
(5)
This formulation emphasizes reliable preferences during training and suppresses noisy supervision, leading to a more stable and generalizable reward model.
3.3
ITERATIVE ALIGNMENT FOR VIDEO GENERATION
After building reliable reward model, turn align text-to-video (T2V) models with its feed- back. Unlike one-time alignment methods that rely on offline fixed preference datasets, we iter- atively leverage dynamic feedback from the self-improving reward model to provide robust and adaptive preference supervision throughout training.
Optimization loop. Our core design is iterative training loop between the T2V model and the reward model. In each iteration, the generator produces a batch videos from diverse prompts. The reward model then evaluates these videos and produce pairwise data for DPO optimization and pointwise data for KTO optimization. To ensure training stability and efficient convergence, we monitor key indicators including the VBench metrics and training loss curves throughout the training process. When performance degradation (e.g., drop in semantic fidelity or motion quality) or abnormal loss dynamics are detected, the framework automatically triggers a critic update via the self-refined preference optimization procedure. This ensures more accurate and distribution- aware reward model, which in turn to re-score generated videos and reconstruct updated preference effectively mitigate reward misalignment and
datasets for the next iteration. In this way, overfitting, enabling progressive preference-driven refinement of the T2V model.
Different preference strategies. We support both pairwise and pointwise alignment paradigms. For pairwise supervision, we adopt Diffusion-DPe Wallace al. (2023), which extends Direct Preference Optimization to the diffusion setting. Given preference pairs (xl 2%)) scored by the critic, Diffuusion-DPO minimizes the following loss:
(0) = - (ay wf)) (-1(0.7) log of (Tw().)
(6)
||e" " <=(" " 1)1 ||e -ee"caa cml .0)1 ca(ax, 1)1 -Ile -mm((
where (>x)) is time-dependent weighting function, and e are noise vectors from the forward process. To improve generalization, add auxiliary negative log-likelihood (NLL) terms:
CDPO ! Lapo + Ep [- log (y"" [=)] Aa Eproa [-
log Po(y)=)),
where sammpl contains preference samples, and Dreal contains real videos for regularization.
For pointwise alignment, incorporate Diffusion-KTT KTO Li et al. (2025), which optimizes nonlin- car based objective without requiring preference pairs. The core loss is defined as:
[olo TT(20-1(x.)
Qiet) ))
Line (0) Brg DD D, Denga...tt (0.
(((a)
(8)
5

---

Published as a conference paper at ICLR 2026
where w(xo) {+1) denotes desirability, and the reference value Qret is:
Qut 3 Dxi. [To((]))) 9) mar(a]s))
To stabilize training, we further regularize the loss using worldworld data as:
L (0)
Tuw (0) + X pr | log po
(10)
Note that these two optimization paths are activated independently within our optimization pro- ces according to the available preference data format, allowing flexible deployment under varied alignment strategies. Our dual-iterative optimization framework offers several key advantages: 1) we dynamically align the reward model drift and generator distributional shifts through dual opti- mization, improving training robustness and stability; 2) we significantly reduce requirements on large-scale human annotations by leveraging critic self-refinement; 3) we provide unified yet flexi- ble framework for handling both pointwise and pairwise preference data; and 4) we achieve stronger generalization and stability by continuously aligning generation behavior with up-to-date human preferences.
4
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We fine-tune pre-trained VILA Lin et al. (2023) (13B/40B) models reward models, leveraging their strong video-text understanding and instruction-following abilities. A small set of Chain-of- Thought (CoT) annotations used for warm-start, followed by 5 epochs of training (LR 2e-5, batch size 4 on 32 GPUs). Videos are represented by 16 uniformly sampled frames with sadding-based preprocessing. Other settings follow VILA defaults. We further apply Self-Refined Preference Optimization (SRPO) with Co'T-guided inference, self-consistency voting, and confidence filtering, generating 20K pseudo-labels per iteration for iterative fine-tuning.
T2V alignment training. For generator alignment, the critic assesses semantic fidelity and tem- poral consistency under the Dual-IPO framework with Diffusion- and Diffusion-KTT Training batch size of 512 and LR 2c-6 (raised to 2e-5 after stabilization) across 128 GPUs. For each caption, the T2V model generates candidates, with the top-1 and bottom-1 (by critic ranking) forming preference pairs. Each iteration constructs about 100K such pairs, and single training round takes on average two weeks.
Datasets. To construct large-scale and diverse training data, we first build pool of structured el- ements (subjects, attributes, spatial relations, actions), which are randomly composed and expanded into natural language captions using an L.M., producing about one million captions. The target T2V model then generates corresponding videos for iterative optimization. In addition, subset of real video-text pairs from VidGen- 1M Tan al. (2024a) is incorporated for joint lne-tuning to enhance diversity and grounding in natural distributions.
Evaluation. For evaluation, we use both automatic metrics and human assessments. We report the overall and per-dimension scores from the VBench benchmark Huang al. (2024), including semantic consistency, motion smoothness, and visual quality. To complement these metrics, we conduct user studies verify synthesis quality.
4.2 EXPERIMENTAL RESULTS
Table 1: Comparison of reward models by human preference accuracy and downstream VBench score after DPO training.
A customized reward model is necessary for reliable alignment. We first evaluate the ef- fectiveness and generalizability of our reward model by measuring alignment with human preference labels. We randomly sample 1,000 videos from VogVideoX-2B and W.nX-1.3B, and manually annotate them set. Our critic is compared against three open- source models: VideoScore He et al. (2024),
Reward Models Accuracy (%)
VBench
VideoScore VideoAlign VideoReward
63.58
80.87
65.21
81.27
reference
68.44
81.31
Ours
81.33
81.54

---

Published as a conference paper at ICLR 2026
cetrossar with net stripe on thei eut sctridn powerful lot dan porroolcng a deaart oati The lonk eyen are flane as it turveye the invtecepe Th
setronsut has cestaa ottiched thair belt heptty water soo, The pain oces maary tearr,, tirn promiaa resef
non, banded in Thie waden dod and warn hel, wolks Herugg quset ellage decked in winter frost The ground is owerel prisfine loyer snew, and magiol villgg
Co0y
05 5,00
ployfu cog russ through vibeast of popptax under the wom sarmer ext, sets protts worbler penched on nearby flower, selictte feathore coteting light.
Figure 2: Qualitative Comparison of our proposed Dual-IPO model and the baseline Cog VideoXX22 model. We can see Dual-IPO improves the baseline in the aspects of prompt following (the 1st line), aesthetic quality (the 2nd line), subject consistency (the 3rd line) and motion smoothness (the 4th line). This demonstrates the efffectiveness of our Dual-IPO for enhancing the generation capability of video found models. Additional visualization results are available the supplementary materials.
VideoAlign Liu et al. (2025a), and VisionReward Xu al. (2024). As shown in Tab. reward model achieves 81.33% accuracy. significantly outperforming VisionReward (68.44%) VideoAlign (65.21% %),, and VideoScore (63.58%). The weaker generalization of these models is due to their training on different data and annotation schemes.
We further use each critic to construct preference datasets for fine-tuning CogVideoX-2B via Diffusion-Die DPO. As shown in Tab. 1, our critic boosts VBench from 80.91 to 81.54 (+0.67), while VideoScore slightly degrades performance, and VideoAlign and ResionReward yield only minor gains around +0.3. This shows critic models must be tailored to the specific T2V generator they supervise. Notably, our reward model trained on just 6,000 annotated pairs, fewer than other models, yet achieves strong alignment. We attribute this to Chain-of-Thouus reasoning and VLM- based inference, which offer richer supervision from limited data.
Comparison with current sota. We evaluate our method on VBench (Tab.3), where our optimized 2B model surpasses the baseline 5B across all dimensions, showing that Dual-IPO post-training effectively boosts generation quality in smaller models. shown in Tab.2, Dual-IPO consistently improves motion, aesthetics, and consistency during iterative training, demonstrating both stability and effectiveness. These gains also reflect better prompt alignment and fewer motion artifacts in generated videos.
To further validate our generalization, we apply our model to both Cog VideoX-SB and Wan-1.3BB 1.3.1.BB... As reported in Tab. 3, after three iterations, Cog VideoX-5B improves from 81.61 to 84.63 VBench, and Wan-1.3B increases from 84,26 to 86.28. These gains are evident in both visual quality and prompt consistency. We also evaluate the final models after five Dual-IPO in Tab. 5, it turns out that

---

Published as a conference paper at ICLR 2026
Table Ablation studies on the iteration number adopted in Dual-IPO, The evaluation is performed on VBench.
Total Score.
Quality Score.
Semantic Score.
Subject Consist.
Background Consist.
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Models
CogVideoX-2B
80.91
82.18
75.83
96.78
96.63
98.89
97.73
60.82
Cog VideooX-2B--PPO
81.69
82.87
77.01
96.77
97.01
99.01
97.84
61.55
Cog VideoX-2B-IPO,
82.42
83.53
77.97
96.81
97.23
99.21
98.03
62.35
Cog VidcoX-2B-IPOs
82.74
83.92
78.00
96.79
97.48
99.35
98.17
62.31
Table 3: Dual-IPO under various baselines.
Table 5: Comparison with sota T2V models on VBench.
Model
Total
Quality
Semantic
CogVidoo-228
80.91
82.18
75.83
Model
Total
Quality
Semantic
Cog VddeoX-2B-IPO-3
82.74
83,92
78.00
Vidu Q1
87.41
87.28
87.94
Cog CogVideoX-SB
81.61
82.75
77.04
Open
Cog Videoo--S--PPO3
84.63
85.40
81.54
Sora
84.28
85.51
79.35
Caus Vid
84.27
78.75
Wan-1.38
84,26
85. 30
80.09
Wan2.1
86.22
86.67
84.44
Wa--1.38-IPO-3
86.28
86.38
CausVid
83.88
85.21
78.57
HunyuanVidon
83.24
85.09
75.82
Table 4: Ablation studies: DPO vs KTO.
CogVideoX
79.17
+Ours
86.57
87.00
84.84
Method
Total
Quality
Semantic
Vog
80.91
82,18
75.83
+Ours
82.74
83,92
78.00
Cog VideoXX-22
82.18
CogVideoX-2B-KTTI
82.47
83.56
78.08
Wan-1.3B
84.26
85.30
80.09
CogVideoX-2B-DD
82.42
83.53
77.97
+Ours
88.32
87.83
90.25
our model yields consistent performance gains, demonstrating both the effectiveness and generaliz- ability of our proposed framework across architectures and scales.
Qualitative results. To further assess our approach, we present qualitative comparisons in Fig. 2. The optimized model consistently produces videos with better semantic fidelity, accurately reflect- ing prompt intent and context. also improves temporal consistency by reducing artifacts and abrupt transitions. Additionally, the outputs show enhanced realism with fewer structural errors, demonstrating the effectiveness of Dual-IPO.
Effectiveness of timization (SRPO). To evaluate SRPO's role con-
Preference Op-
Table Ablation study on SRPO during the third fourth Dual-IPO iteration the Cog VideoX- 2B baseline.
in improving critic generalization, duct controlled ablations during the training of CogVideoX-2B in Tab. 6. Specifically,, we ex- amine model performance between the third and fourth IPO iterations under conditions: (1) keeping the critic model fixed, and (2) up- dating the critic via SRPO. We can observe that without critic update, performance declines
Setting
Critie Update
SRPO loss
PCR
VRench Score
Baseline
x
82.74
update
SRPO (fall)
V
82,91
SRPO loss
/
82.83
from 82.74 82.33, indicating overfitting or misalignment. In contrast, applying SRPO increase the performance from 82.74 to 82.91, demonstrating the necessity and efficacy of our proposed reward model in-the -hehhe-loop optimization.
We ablate SRPO components. Removing PCE with SRPO loss causes minimal change (82.69), suggesting limited impact. Replacing SRPO loss with DPO, even with PCE, drops performance to 82.83, confirming SRPO loss is crucial for critic robustness and signal quality. These results underscore the importance of reward modeling, critic refinement, and self-refinement for stable alignment.
8

---

Published as a conference paper at ICLR 2026
82.5
CogvideoX-2B Ties Dual-IPO
VBench Total Score
80.0
54.6
43.5
40.5
75.0
33.8
35.2
72.5
25.7
29.4
21.3
70.0
Single-round
Iterative(IPO)
CogVideoX-2D
65.0
500
750
1000
1500 1750 2000 Training Steps
1250
Consistency
Faithfulocss
(a) VBench score progression
(b) Human evaluation
Figure 3: Comparison of IPO and baselines: (a) VBench shows DPO degrades, while Dual-IPO improves steadily. (b) Human evaluation compares Dual-IPO, CogVideoX-22 and Ties on consis- tency, motion, and faithfulness.
4.3 EFFECTIVENESS OF ITERATIVE ALIGNMENT
Why multi-round iteration matters. To evaluate iterative optimization, compare Dual-IPO with a static baseline using Diffusion-DPO without updating the preference data or the reward model. As shown in Fig.3a, the static baseline improves briefly but degrades due overfitting and misaligned rewards, while Dual-IPO maintains steady performance gains with progressive data improvement and reward optimization. Tab.3 and Tab. 2 confirm that Dual-IPO outperforms the static method in overall VBench scores and all quality dimensions, with consistent improvements in motion, aesthetics, and semantic consistency.
Benefits of incorporating real video data. We examine whether real-world video data improves post-training. Using VidGen- IM Tan et al. (2024a) with diffusion loss and preference optimization, sistent gains in total, quality, and semantic scores (Tab. 7). In contrast, real data alone slightly hurts performance, suggesting it is most effective when combined with prefer- ence guidance. We leave further analysis to future work.
Table 7: Ablation study on the effects of incorporat- ing real video data during Dual-IPO fine-tuning.
Model
Total
Quality
Semantic
observe con-
Cog VideoX-2B
80.91
82.18
75.83
only real data sft
80.63
81.79
76.02
data
81.17
82.33
76.53
w/ real data sft
81.69
82.87
77.01
Comparing DPO and KTO within Dual-IPO. To assess flexibility, we compare DPO and KTO within DuOl-IPO (Tab, 4). Both outperform the baseline. KTO slightly improves Total and Semantic scores, while DPO leads in Quality, showing Dual-IPO supports diverse alignment. KTO's more stable semantic gains suggest better high-level consistency, confirming the adaptability of Dual-IPO and effectiveness of KTO for fine-grained optimization.
Human evaluation. Fig. 3b shows the results of human visual evaluation that compare our Dual- IPO with the Cog V baseline. Obvisouly, Dual-IPO outperforms the bascline across all di- mensions. In "Consistency," both models perform equally in 40.5% of cases, suggesting the base- line handles text alignment reasonably well. However, Dual-IPO shows clear gains "Motion" and "Faithfulness," generating more natural movement and better text alignment. Full evaluation metrics in the appendix.
5 CONCLUSION
In this paper, we present a novel Iterative Preference Optimization framework for improvement video foundation models from the pespective of post training. In particular, Dual-IPO collects

---

Published as a conference paper at ICLR 2026
human preference dataset and trains a critic model for automatically labeling the generated videos from base model. Given this, Dual-IPO efficiently optimizes the base model in an iterative manner. By this way. Dual-IPO enhanced video results in motion smoothness, subject consistency and aesthetic quality, etc. In addition, adapt Dual-IPO with different optimization strategies: DPO and KTO, both achieve impressive improvements. This further verifies the flexibility of Dual-IPO. Thorough experiments on VBench demonstrates the efficacy of Dual-IPO.
effectively align the base model with human preference, leading to
ACKNOWLEDGMENTS
This work was supported by Al for Science Program, Shanghai Municipal Commission of Economy and Informatization (Grant No. 205-67Z--RG2N--BTXX00077
ETHICS STATEMENT
This work focuses on developing dual-iterative optimization framework for text-to-video gener- ation. All datasets used are either publicly available or synthetically generated through large-scale text-to-video models. Only small set of numan-labeles preference data is employed for initial training. after which the framework relies mainly on automated pscudo-labeling. No sensitive, pri- vate, personally identifiable information is used, and human subjects involved in ways that raise ethical concerns. The proposed method aims solely to improve video generation quality and alignment with human preferences. Therefore, believe that this research does not pose ethical risks.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of our framework, including algorithmic formulations, optimiza- tion objectives, and training procedures. Implementation details such as learning rates, batch sizes, GPU usage, model architectures, and dataset construction are explicitly reported. Extensive ex- periments are conducted with multiple baselines, ablation studies, and both automatic and human evaluations. All benchmarks employed (e.g.. VBench) are publicly available. These details ensure that the experiments and results presented in this paper can be reproduced by the research commu- nity.
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statisties, pp. 4457-4455. PMLR, 2024.
Andreas Blattmann, Robin Rombach, Huan Ling. Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023. URL https: http///axvvv/.///////ee//a.aa..a org/ abs/2303 08818.
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, Jan 2024a.
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive align- ment of language models with explicit rewards. Advances in Newral Information Processing Systems, 17:117744-118822 2024b.
Xinyuan Chen, Yaohui Wang. Lingjun Zhang, Shaobin Zhuang, Xin Ma. Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction, 2023. https: http::ss///xxiv.orr/.oo /
10

---

Published as a conference paper at ICLR 2026
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models, 2023. URL
03011.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URthttps: http//xaxxv.v../../...///// 2402.00 01306.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: theoretical perspective. arXiv preprint arXiv:2404.0463 046266, 2024.
Gen-3. Gen-3. https::////
2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint 2rX1v:230..04725, 2023.
Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma. Zhijie Lin, Zhenheng Yang. Dahua Lin, and Lu Jiang. Long context tuning for video generation. arXiv preprint arXiv:2503.10589 2025.
hailuo. hailuo. https: /eetps:/hhaiiee http/hhaalllaaaaa video/, 2024.
Xuan He, Dongfu Jiang, Ge Zhang. Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang. Aaran Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-graines human feedback for video generation. arXiv preprint arXiv:2406. 15252, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6855 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: high definition video generation with diffusion models (2022). arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Spread preference annotation: Direct preference judgment for efficient Ilm alignment. arXiv preprint arXiv:2406 24:.:2446.0441... 2024.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Kuaishou. Kling, attps://kkle /ttpps///aling..uuihhhuu
kuaishou. cwm/eo/////oooeooo 2024.
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v-turbo-v2 Enhancing video generation model post-training through data, reward, and conditional guidance design. arXiv preprint arXiv:2410.05611 2024.
11

---

Published as a conference paper at ICLR 2026
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. Advances Neural Information Processing Systems, 37:24897-249255 2025.
Ji Lin. Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
Binhui Liu, Xin Liu, Anbo Dai, Zhiyong Zeng, Dan Wang, Zhen Cui, and Jian Yang. Dual-stream diffusion net for text-to-video generation. arXiv preprint arXiv:2308. 08316, 2023a.
Jie Liu, Gongye Liu, Jiajun Liang. Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint ar 2rXvv2501..33/15 2025a.
Runtao Liu, Haoyu Wu, Ziqiang Zheng, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. Videodpo: ommi--reeeerece alignment for video diffusion generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8009-8019. 2025b.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06677 2023b.
Luma. Luma ai. https::/// lwwaalluaaaasaaa httpaaaaaaaaaaada.aaaaaeaa ai/dreem-machhin 2024.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
Richard Yuanzhe Pang. Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoming preference optimization. Advances Neural Information Processing Systems, 37:1 16617-116637 2024.
Ryan Park, Rafael Rafailov. Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 11t.29993..99,,,. 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp.
2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation, 2024a. URL https://arxiv.orrt abs/2310 03739.
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737, 2024b.
Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang. Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference opti- mization. arXiv preprint arXiv:2411.04109, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-523 -37441, 2023.
12

---

Published as a conference paper at ICLR 2026
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint 1rX00:2210.01241, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusiin: Learning multi-modal diffusion models for joint audio and video generation, 2023. URL https: / arxiv. org/ hatp//b//250 2212. 2912..9472.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video 2022.
data. arXiv preprint arXiv:2209.
Jascha Sohl-Dickstein, EricA. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv: Learning.arXiv: Learning, Mar 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. Vidgen-Im: A large-scale dataset for text-to-video generation. arXiv preprint arXiv:2408 02629, 2024a.
Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Supervised fine-tuning multimodal IIms with human-aligned data for evaluating text-to-image models. arXiv preprini arXiv:2406. 16562, 2024b.
Zhiyu Tan, Hao Yang, Luozheng Qin, Jia Gong, Mengping Yang, and Hao Li. Omni-video: Democ- ratizing unified video understanding and generation. arXiv preprint arXiv:2507.06119 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv: 1,97,,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 237800998788. 2023b.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023. URL https: / arxiv org/ aaa///////3// 22112111.21 12908.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.2033 203.14 2025.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.00 230.007700,,6,,,, 2023.
Wenjing Wang, Huan Yang. Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Swap attention in spatiotemporal diffusions for text-to-video generation. International Journal of Computer Vision, pp. 1-19, 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-connistency improves chain of thought reasoning in language models. arXiv preprint arXhv:2203.11112 2022.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint 2rXiv:2104.14806 2021.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint 23X.v:2306.09341 2023.
13

---

Published as a conference paper at ICLR 2026
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation, 2023. URL BtLhttps:// /ttp://aaavv../..a//aa/eeaaeawaa httpaaawwiv......aaaa///// wwwggas//////oooooooooooooooeeae 2200//333...0 09710.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 15903-15935, 2023.
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang. Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine- aine-graied mimti-dim human prefer- ence learning for image and video generation. arXiv preprint arXiv:2412.210055 2024.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511: 2024a.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint (XXx::2488.06072 2024b.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng. Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback, 2023a. URL https: / /ttpxixxvvvo/////// org/ /tt.a///.............aaaaatattta 12490.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprini arXiv:2304.05302. 2023b.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint SKKi::2206005564 2022.
Ruihan Zhang, Borou Yu, Jiajian Min, Yetong Xin, Zheng Wei, Juncheng Nemo Shi, Mingzhen Huang. Xianghao Kong, Nix Liu Xin, Shanshan Jiang. Praagya Bahuguna, Mark Chan, Khushi Hora, Lijian Yang. Yongqi Liang, Runhe Bian, Yunlei Liu, Isabela Campillo Valencia, Patri- cia Morales Tredinick, Ilia Kozlov, Sijia Jiang, Peiwen Huang, Na Chen, Xuanxuan Liu, and Anyi Rao. Generative ai for film creation: A survey of recent advances. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, pp. 6267-6279, June 2025.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-vide video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.0414 2023.
Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04966 2023.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Con- sistent self-attention for long-range image and video generation. Advances in Neural Information Processing Systems, 37:1 10315- - 2024.
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment, In The Thirteenth International Conference on Learning Representa- tions, 2025.
14

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIAL: DUAL-IPO: DUAL-ITERATIVE PREFERENCE OPTIMIZATION FOR TEXT-TO- VIDEO GENERATION
Human Annotation
16
A.I Annotator selection
16
A.2 Annotation Guideline
16
A.3 Dataset distribution
17
A.4 Annotation training and interface
18
B Additional Quantitative Analysis
18
B.1 All evaluation dimension results
18
B.2 Impact of Reward Model Scale on Accuracy
18
Human-Validated SRPO and Reliability of Synthetic Preferences
19
Number of Dual-IPO Rounds and Stopping Criteria
20
E
Theoretical Motivation and Convergence Analysis
21
Prompt Construction Pipeline
22
F.1
Design Goals
22
F2
Structured Prompt Space
22
F3
Sampling Strategy and Target Distributions
23
F.4
LLM-Based Expansion and Safety Filtering
24
F.5
Bias Mitigation and Limitations
24
F6
Reproducibility
24
Computational Cost and Practicality of Dual-IPO
24
G.I
Where the Compute Actually Goes
25
G.2
Comparison Standard RLHF Pipelines.
25
G.3 Generator and Critic Configurations
25
G.4 Cost-Reduction Strategies and Trade-offs
26
G.5 Summary
26
G.6 Impact of Reward Model Size on RL
27
H Additional Qualitative Comparison
27
I Use LLM
27
Societal Impacts
27
Limitations
28
15

---

Published as a conference paper at ICLR 2026
A HUMAN ANNOTATION
Prior the final manual annotation, undertook several steps to ensure the quantity, quality, and efficiency of the annotation process. In building the dataset, we focused on the diversity and complexity of prompts, creating annotated data across multiple dimensions and categories. We meticulously selected candidates for the annotation task and provided them with in-depth training sessions. We then designed a user-friendly interface and a comprehensive annotation program to streamline the process. Additionally, we developed detailed annotation guide that covers all aspects of the task and highlights essential precautions. To further ensure consistency and accuracy among annotators, we conducted multiple rounds of annotation and implemented random sampling quality checks.
A.1 ANNOTATOR SELECTION
The effectiveness of annotated data hinges significantly on the capabilities of the individuals re- sponsible for the annotation. As such, at the onset of the annotation initiative, placed strong emphasis on meticulously selecting and training a team of annotators to ensure their competence and objectivity. This process entailed administering a comprehensive evaluation to potential candidates, which was designed to gauge their proficiency in ten critical domains: domain expertise, tolerance for visually disturbing content, attention to detail, communication abilities, reliability, cultural and linguistic competence, technical skills, ethical awareness, aesthetic discernment, and motivation.
Given that the models under assessment could generate videos containing potentially distressing or inappropriate material, candidates were informed of this possibility in advance. Their participation in the evaluation was predicated on their acknowledgment of this condition, and they were assured of their right to withdraw at any time. Based on the results of the evaluation and the candidates' professional backgrounds, we sought
form a team that was diverse and well-rounded in terms of expertise and capabilities across the ten assessed areas. Ultimately, we selected a group of 10 annotators compriisng five men and five women, all of whom possessed bachelor's degrees. We conducted follow-up interviews with the selected annotators to reaffirm their suitability for the an- notation task.
A.2 ANNOTATION GUIDELINE
When it comes to assessing human preferences in videos, the evaluation process is structured around three fundamental aspects: semantic consistency, motion smoothness, and video faithfulness. Each of these dimensions is meticulously examined independently to provide a comprehensive un- derstanding of the video's 's overall quality. Semantic consistency ensures that the video's content aligns logically and contextually, while motion smoothness evaluates the smoothness and natural- ness of movements within the video. Video faithfulness, the other hand, assesses how well the video maintains realistic and high-quality appearance. By breaking down the evaluation into these distinct yet interconnected aspects, we can achieve a more nuanced and accurate assessment of human preferences in video content.
Semantic Consistency The consistency between the text and the video refers to whether elements such as the topic category (e.g.. people animals), quantity, color, scene descrip- tion, and style mentioned in the text align with what is displayed in the video.
Motion Smoothness The motion smoothness refers whether the movements depicted are smooth, coherent, and logically consistent, particularly in terms of their adherence to physical laws and real-world dynamies. This includes evaluating if the actions unfold in seamless and believable manner, without abrupt or unrealistic transitions, and whether they align with the expected behavior of objects or individuals in the given context.
Video Faithfulness Faithfulness refers to the accuracy and realism of the visual content in the video. It examines whether depictions of people, animals, or objects closely match their real-world counterparts, without unrealistic distortions like severed limbs, deformed features, or other anomalies. High fidelity ensures the visual elements are believable and consistent with reality, enhancing the video's authenticity..
16

---

Published as a conference paper at ICLR 2026
Cartee
Eperts
Surning
Gorakriancy
Candianny
AW Ties # Wroe
Rood
Danding
Fattfilneess
Aninal
5%
Object
Oney
Act
Homen
Percontaga(1)
Parsortageh)
(a) Prompt action Category
(b) Prompt type category
Pointwise scoring distribution
(d) Pairwise ranking distribution
Figure AI: The distribution statisties of Human Preference Dataset on prompts in (a) and (b) as well as scoring/ranking in (c) and (d).
Video A
fion
An tranaut stripe sur catrice somerful
overlssking
eyes
belt, haping to find water seen, The trees of the casss the brwente, brwete,afferng promite relia.
Q Condietency annotation: Scoring, storing the two videoe from the perspective of video and hext consilency
Video A
Good
Narma
Bed
Reason
Video B
Video
Good
Norma
Bad
Reason
Ql-Consatency Antotatior: Ranking, sideo bether the parpacties of video and hext contictency
Rank
@ Vidwo A
Vidwo
Thex
Figure A2: Our user interface presents one sample at time to annotators, featuring four distinct icons for different functions.
Pointwise Annotation For each dimension, annotators assign one of three ratings:
Good: No observable flaws; fully satisfies the criterion.
Normal: Minor imperfections that do not severely impact perception.
Bad: Significant violations that degrade quality or realism.
Annotations are performed independently across dimensions avoid bias.
Pairwise Annotation
Given two videos VA and VB, annotators compare them under each dimen-
sion:
For semantic consistency, determine which video better aligns with the intended context.
For motion smoothness, identify the video with more natural and continuous motion.
For video faithfulness, select the video exhibiting higher realism and fewer artifacts.
ranked preference (VA VB or VB > VA) recorded separately for each dimension. Ties are permitted only if differences are indistinguishable.
A.3 DATASET DISTRIBUTION
As shown in Fig. AI, the dataset includes over ten categories, such as Human, Architecture, Ve- hicles, and Animals, well more than ten different actions of human and animal. For each
17

---

Published as a conference paper at ICLR 2026
prompt, the video generation model produces four variants with different random seeds. The dataset is iteratively expanded through an optimized regeneration strategy to improve preference alignment.
A.4 ANNOTATION TRAINING AND INTERFACE
We convened comprehensive training session centered detailed user guidelines to ensure that the annotation team fully comprehends our goals and standards. During the training, we covered the purpose, precautions, standards, workload, and remuneration related the annotation process. Additionally, we formally communicated to the annotators that the human annotation is strictly for research purposes and that the annotated data might be made publicly available in the future. We reached a mutual agreement with the annotators regarding the standards, workload, remuncration, and intended use of the annotated data. The criteria for assessing video faithfulness, text-video align- ment, and motion smoothness are universal, which means that individual standards should not vary significantly. Consequently, we annotated several sample videos using our carefully designed anno- tation platform to ensure consistency among annotators. An overview of the developed annotation platform shown A2. Through this training, we also equipped the annotators with the necessary knowledge to conduct impartial and detailed human evaluations of video faithfulness, text video alignment, and motion smoothness.
Model
Bockgrood Consinency
Fltckering
Motion
Dynerti
Aestletio
Imeging
Object
Cophrinnk-on
96.63
5% BV 01
92.23
60:47
61 A3
81:13
OogVideoX KTO-
96 77 Oog KTO, S3gMidenX-22-K000
01.03
97.44
64.33
01.55
02
X5.41
Mutiple Ofjecte
Action
Chiue
Spatis Relatsmnep
Scere
Appoaranze Style
Terporal style
Ovrril Consitency
CogNideoX 2B Deviisoo--3---- KTTO,
63
98.00
$1.14
69 25
96.58 $4
K1.67
60
25:1
27.03
K8 64
K3.26
70 16
56.14
25:47
26 44
27:3
Oog Videoo 28 KTO,
63.51
95083
$2.53
70.23
$4.41
25:48
25.30
27.66
Table Al: Quantitative results video assessment metries. Display of comparison results between all indicators and baseline on vbench through multiple iterations.
B ADDITIONAL QUANTITATIVE ANALYSIS
B. ALL EVALUATION DIMENSION RESULTS
We present the evaluation metrics of VBench benchmark in 16 different dimensions in the Tab. Al. The results indicate that the performance shows continuous upward trend in multiple training iter- ations, indicating that the training is dynamically stable. Although small fluctuations are observed, they can be ignored and will not affect the overall positive progress of the indicator.
Model Scale
Pairwise Accuracy
Pointwise Accuracy
Dual-IPO-RRward-11 13B
78.41
82.95
Mhal-IPO-Reward-Bee
81.33
85.57
Table A2: Accuracy of Reward Models on Validation Sets Across Different Scales. The reward models used in this study finetuned from the ViLALin et al. (2023) multimodal large language model. This table evaluates their performance at different scales (13B and 40B) in terms of accuracy on pairwise and pointwise validation datasets. The results demonstrate that larger model scales yield higher accuracy for both pairwise ranking and pointwise scoring, indicating improved alignment with human preferences the model size increases.
B.2 IMPACT OF REWARD MODEL SCALE ON ACCURACY
To evaluate the impact of model scale reward performance, experimented with two ViLA- based critic models of 13B and 40B parameters. Fine-tuned on paired and pointwise annotated datasets, these models were assessed on validation tasks, shown in Tab. A2.
Results show that increasing the model size from 13B to 40B significantly improves accuracy, with the 40B model achieving 80.73% in paired ranking and 86.57% in pointwise scoring-2.99 -2.2%% and
18

---

Published as a conference paper at ICLR 2026
2.62% higher than the 13B model. This suggests that larger critic models better capture human preferences by representing complex multimodal relationships.
Our ablation study further highlights the importance of model scale. While smaller models perform reasonably well, larger models significantly boost accuracy, benefiting downstream reinforcement learning tasks. These findings suggest that scaling the critic model is simple yet effective way to enhance preference alignment and improve video generation quality.
Total Score
Quality Score
Semantic Score
Subject Consist.
Background Consist.
Models
Cog VideoXX-22
80.91
82.18
75.83
96.78
96.63
HIPOORRearr--I
82.42
83.53
77.97
96.81
97.23
w/IPO-Reward-40B
82.52
83.63
78.07
96.83
97.37
Temporal Flicker
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models
CogVideoX-221
98.89
97.73
60.82
59.86
61.68
e/IPO-Rewarde 13B
99.21
98.03
62.35
68.76
61.77
w/IPO-Reward-40F 99.19
98.09
62.38
68.92
62.01
Table A3: Evaluate the impact of the critic model on performance, the performance of the critic model results for different model sizes of 13B and 40B, more accurate critic model leads to improved results.
C HUMAN-VALIDARN SRPO AND RELIABILITY OF SYNTHETIC PREFERENCES
SRPO relies on pseudo-preference labels generated by the critic, which raises the question of how to ensure that these synthetic labels remain aligned with human judgments. To address this, our implementation couples every SRPO iteration with an explicit preference test set, and we only accept critic updates that pass this validation.
-aaaan-aaiiation step on a held-out
Human preference test set. We maintain a fixed buman-labeled preference test set constructed from the current T2V model. For each prompt. the model generates multiple candidate videos, and annotators rank these candidates. The resulting rankings are converted into pairwise preference labels and used solely for evaluation, not for training SRPO. This set allows measure how well the critic agrees with human judgments on realistic model outputs.
SRPO iteration and validation loop.
Each SRPO iteration proceeds as follows (see Figure A3):
Pseudo-label generation with PCE filtering, Given the current critic, we score candidate video pairs generated by the T2V model and retain only those with high confidence cording to a prediction-confiddnnc estimator (PCE). These high-confidence pairs form the synthetic preference set.
2. Critic update. The critic is fine-tuned on mixture of the original human-labeled seed data and the filtered pseudo-labeled pairs, yielding an updated reward model.
3. Hlmann-valddati test set and compute the agreement between the critic's predicted preferences and human rankings. On this same set, we also evaluate open-source reward models as baselines and report the resulting human agreement in Table 1. In Table 6, we track both this agreement and downstream VBench metrics before and after SRPO updates.
step. After the update, we evaluate the critic on the human preference
4. Acceptance criterion. In practice, we only accept an SRPO update the critic's agreement with human labels remains above fixed threshold (75% in our experiments). In addition, we monitor VBench metrics of the aligned generator; if these metrics degrade after an SRPO update, discard the update and retrain roll back the critic. This mechanism
19

---

Published as a conference paper at ICLR 2026
Human-labeled Preference Test Set
VBench Evaluation
agreement >= 758
improved stable
T2V Model
Retrain critic adjust
Video Preference Data
Generated Videos
Updata
Critic Model
Preference Certainty
Vote
8oe
Critic Model
Figure A3: Overview of the SRPO iteration and validation loop. High-confidence pseudo-labels are generated via PCE filtering, used to update the critic, and each update is accepted only if it passes human-agreement threshold and maintains or improves downstream VBench performance.
prevents error accumulation from low-quality pseudo-labels and ensures that SRPO steps are beneficial both for the reward model and for downstream video quality.
This tho-loop validation makes the SRPO refinement process both controllable and au- ditable: synthetic labels are used aggressively to scale supervision, but each refinement step gated by human-validate agreement and downstream metrics, which mitigates the risk of drift away from true human preferences.
D NUMBER OF DUAL-IPO ROUNDS AND STOPPING CRITERIA
Dual-IPO is designed as a multi-round optimization procedure in which the generator and critic are updated iteratively. An important practical question is how many rounds are beneficial, and when procedure should be stopped to avoid diminishing returns or overfitting. In this section, we summarize empirical observations and the resulting stopping strategy.
Empirical saturation of VBench performance. Beyond the rounds reported in the main exper- iments, we conducted an additional Dual-IPO iteration on Wan-1.3B. observed that the overall VBench score had essentially saturated: the score only changed from 88.32 to 88.33 in this extra round. Similar saturation phenomena appear on other configurations. In contrast, our internal re- ward continued to increase. This indicates that performance does not keep improving indefinitely with more Dual-IPO iterations, and that after a certain point further optimization mainly improves the internal reward signal rather than the external evaluation metrics.
20

---

Published as a conference paper at ICLR 2026
Diserepancy between internal reward and external metrics. On VBench's 16 dimensions, aligned models already achieve very high scores after a moderate number of rounds; the remaining headroom is concentrated in aesthetics and visual quality. In later iterations, the internal critic reward keeps increasing, but this no longer translates into measurable gains on VBench. One could potentially reweight redesign the reward dimensions to more aggressively target the remaining VBench dimensions and obtain higher benchmark scores, but this would amount to tuning the reward directly the benchmark, which risks drifting toward "benchmark hacking" rather than genuine quality improvements.
Practical early stopping rule. Given these observations, we do not aim to identify a globally optimal number of Dual-IPO rounds. Instead, adopt practical early stopping rule: stop the iterative procedure once VBench and human evaluation no longer show meaningful improvements, even if the internal reward continues rise. Concretely, we monitor:
the overall and per-dimension VBench scores,
human preference evaluations on held-out comparison sets,
the critic's agreement with human labels.
When these external signals saturate or begin to fluctuate within narrow range, additional rounds are deemed unproductive and the current model is selected.
Limitations and future exploration. Our experiments suggest that Dual-IPO provides practi- cally useful recipe for multi-round optimization with reasonable stopping criteria: it yields consis- tent gains over the baseline without inducing model collapse, severe reward hacking, overfitting to particular critic. Nonetheless, we do not claim to have identified a universally optimal config- uration. A truly complete "optimal recipe" will depend on multiple engincering factors, including model size, the number of rounds, the number of samples per round, and the choice of evaluation metries. more systematic study of these scaling behaviours, especially with additional metrics beyond VBench and on larger models, is an important direction for future work.
E THEORETICAL MOTIVATION AND CONVERGENCE ANALYSIS
DuO adopts dual- -ual-iterative optimization scheme in which the policy (generator) and the critic (reward model) are updated in alternating stages. This design supported by existing theoretical and empirical findings from iterative RL.HF-ttlle training, well as by standard KI.-regularized RI objectives used in DPO-like analyses.
Motivation from iterative RLHE Prior work iterative RLHF training Touvron et al. (2023a;b) shows that (i) updating the reward model during training is crucial for maintaining aligmment with the current policy, and (ii) using data labeled by the current policy to refine the reward model can significantly improve its accuracy and robustness. In these pipelines, the policy and reward model updated in alternating stages rather than in a single offline pass. Dual-IPO follows the same principle: our SRPO component refines the critic using self-labeled preference data, filtered by certainty estimator to reduce noise. As the generator improves, the critic is iteratively updated on higher-qualite
data, which gradually tightens the alignment between the learned reward and the target preference signal.
Connection to KL-regularized RI and DPO. The effectiveness of iterative policy updates in Dual-IPO is further motivated by the standard KL-regularized RL objective underlying DPO-like methods Zhang (2023); Pang et al. (2024). The optimal aligned policy for given reward function r(Zu, c) can be written as
(xo c) x no( () exp r(xx:.)) c)
(A1)
where To is reference policy and B 0 is temperature parameter. Online RL methods such as PPO and GRPO can be viewed as stochastic optimization procedures that (approximately) move mo toward under online data collection. Empirically, such iterative procedures often outperform
21

---

Published as a conference paper at ICLR 2026
single offline DPO step. This motivates iterative optimization scheme: by repeatedly updating the generator with DPO/KTO under an increasingly accurate critic, we expect the current policy to gradually approach x²" leading to more stable and higher-quality solutions than one-shot optimiza- tion.
Practical convergence and stopping criterion. In practice, we observe that performance metrics (e.g.. VBench scores) and training losses improve over successive Dual-IPO iterations and then satu- rate. We exploit this behaviour to define a practical stopping rule: we terminate the iterative process once additional Dual-IPO iterations bring only marginal improvements on VBench. We attribute this saturation partly to the current reward design and the coverage of existing video-generation metrics, well to potential capacity limits of smaller models. Although resource constraints prevent running arbitrarily long iteration sweeps, extended runs consistently show a pattern of improvement followed by saturation rather than divergence.
Stability in training. Mild oscillations in losses and metries are inevitable in reinforcement- learning-style training, and Dual-IPO is no exception. In our context, stability refers to controlled behaviour of the system rather than perfectly monotone curves. Concretely, in each Dual-IPO it- eration we monitor VBench scores and training losses; when signs of degradation or instability are detected (e.g., VBench scores drop or losses behave pathologically), we trigger new Dual-IPO iter- ation and refine the critic via SRPO. Across all reported experiments, the metrics exhibit consistent improvement followed by saturation, without wild oscillations or runaway divergence. We provide the corresponding curves, including later-iteration results, to empirically illustrate this convergence pattern.
Overall, Dual-IPO is theoretically motivated by KL-regularized RL and iterative RLHF, and empir- ically exhibits stable, saturating behaviour under our monitoring and stopping criteria.
F PROMPT CONSTRUCTION PIPELINE
We design a programmatic prompt construction pipeline that combines structured templates, large vocabularies, and LLM-based expansion and filtering. The goal is to obtain an effectively unbounded prompt pool that is (i) large and diverse, (ii) balanced across high-level categories, and (iii) explicitly designed to mitigate cultural and demographic biases. Figure the appendix summarizes the distributions of major subject and action sub-categories.
F.1 DESIGN GOALS
Our prompt pipeline is built around three main desiderata:
Size. Prompts generated from templates with random instantiation rather than from fixed list, so the space of possible prompts is effectively unbounded and exact repetitions are rare.
Diversity.
The templates factorize subjects, attributes, spatial relations, actions, and scene contexts, which are populated from rich vocabularies and combined into both simple and complex scenes.
Balance and bias awareness. We impose explicit sampling distributions over high-level categories (e.g., humans vs. animals vs. objects; single vs. multiple subjects; easy vs. dif- ficult actions) and design the subject vocabularies and filtering stages to reduce systematic underrepresentation of particular demographic or semantic groups.
F.2 STRUCTURED PROMPT SPACE
Subjects and attributes. We organize subjects into structured space with two primary groups, humans and animals, and additional object categories.
Human subjects. Humans are enumerated over age (child, adult, elderly), gender-neutral and gendered roles (e.g., teacher, student, firefighter, musician), and physical attributes
22

---

Published as a conference paper at ICLR 2026
such height, body shape, and hair style/color. Forms of address vary from generic ("a young person") to role-specific ("an elderly firefighter").
Animal subjects. Animals divided into land, flying, and marine species, with large set of concrete types and surface forms (e.g., "golden retriever", "tabby cat", "bald eagle"). Additional object categories. We include vehicles, food, and common objects (e.g.. fur- niture, tools, electronic devices, toys). These appear as primary subjects or contextual elements interacting with humans and animals.
Subjects appear single targets or multiple interacting entities. For multi-subject prompts, we additionally sample explicit spatial and relational patterns, such as left/right, in front of/behind, above/helow, role-based relations (e.g., "a child standing next dog". "two eyclists racing behind car"). This encourages the model to reason about structured multi-object configurations.
Action taxonomy. levels:
To better cover temporal behavior, we categorize actions into three difficulty
Simple (e.g., walking, running, sitting, waving),
Medium jumping, crawling, turning around, picking up objects),
Difficult (c.g., complex sports or task-oriented behaviors such as gymnastics or acrobatic moves).
Actions combined with subjects and then filtered. Non-realistic pairs (e.g., inanimate ob- ject performing complex motion) are either discarded or explicitly mapped to stylized domains ("cartoon", "dreamlike", "sci-fi"), so that even unrealistic combinations become coherent abstract scenes.
Scene contexts.
We define many scene types grouped into realistic and abstract contexts:
Realistic scenes include common natural and urhan environments such as forests, snow- fields, farmland, parks, kitchens, offices, and city streets.
Abstract and surreal scenes include deliberately unusual compositions, such as "a car driving inside coffee cup" or cat floating through neon-colored dreamscape". These are used to create a long tail of challenging prompts without dominating the dataset.
F.3 SAMPLING STRATEGY AND TARGET DISTRIBUTIONS
We do not fix finite list of prompts per category. Instead, each training run defines a target distri- bution over high-level prompt categories (e.g., humans animals objects, single vs. multiple subjects, easy vs. harder actions, realistic vs. abstract scenes). Prompts are then sampled on the fly from the global structured pool under this distribution.
Because subjects and attributes are drawn from broad, approximately uniform vocabularies, individ- ual prompts almost never repeat and single subject type is systematically underrepresented. At the same time, the category-wise sampling weights ensure that each epoch sees roughly balanced mixture of content types rather than being dominated by single class.
We also stratify actions by difficulty and choose their sampling probabilities according to the capa- bility of the base model (for example, placing more mass on simple actions for smaller backbones). Overall, we adopt a schedule in which the sampling probability gradually decreases from:
simple to difficult actions,
realistic abstract scenes,
single-subject to multi-subject compositions,
everyday to highly surreal situations.
This keeps the dataset dominated by realistic, common scenarios while still maintaining diverse long tail of challenging compositions involving multiple subjects, attributes, spatial relations, and abstract scenes.
23

---

Published as a conference paper at ICLR 2026
F.4 LLM-BASED EXPANSION AND SAFETY FILTERING
Textual prompts can themselves be a source of cultural and demographic bias, so we deliberately restrict and structure how we use large language models the pipeline.
Restricted of the LLM. The discrete choices-subjed subject category. attributes, actions, and scene type-are sampled directly from our structured, enumerated pools. The LLM is only used to:
1. expand these discrete tuples into fluent natural- naturallanguagge descriptions, and
2. detect and repair problematic prompts.
Crucially, the LLM does not decide which demographic or object type appears; those decisions are controlled by our explicit vocabularies and sampling weights.
Automatic filtering and auditing. We apply multiple LLM-based filtering stages to remove or rewrite prompts that are:
nonsensical or internally inconsistent,
unsafe (e.g., self-harm, explicit violence),
clearly discriminatory, offensive, or otherwise inappropriate.
Prompt-caption pairs that are flagged as problematic are discarded or rewritten into safe, abstract variants. addition, we perform light manual spot checks on random batches and monitor empirical distributions over subject and action categories during training to keep coverage roughly balanced. This design makes the prompt pool both controllable and auditable, and provides clearer levers for mitigating bias than relying on scraped, uncurated prompts.
F.5
BIAS MITIGATION AND LIMITATIONS
Our construction incorporates several mechanisms aimed at reducing cultural and demographic bias:
Enumerated human subject space with explicit variation age, roles, and physical attributes, sampled approximately uniformly.
Balanced high-level categories controlled by sampling weights for humans, animals, and objects, as well as single multiple subjects and action difficulty.
Restricted LLM usage so that demographic composition is determined by transparent, discrete pools rather than opaque generative behavior.
Automatic and manual filtering for unsafe or discriminatory content, coupled with mon- itoring of empirical distributions.
These measures do not eliminate bias entirely, but they make the prompt space more controllable and inspectable, and allow us balance demographic and semantic coverage more effectively than using unfiltered real-world text prompts. A more systematic faimess and bias analysis of downstream models trained under this prompt distribution remains an important direction for future work.
F.6 REPRODUCIBILITY
We will release the main prompt templates, vocabularies, and sampling configurations used in our pipeline to facilitate reproducibility and future research. This will allow others to regenerate com- parable prompt distributions or adapt our structured construction new domains and models.
G COMPUTATIONAL COST AND PRACTICALITY OF DUAL--IPO
We thank the reviewer for raising the issue of computational cost and scalability. Our current ex- periments indeed relatively large configuration in order to push performance and demonstrate
24

---

Published as a conference paper at ICLR 2026
the full potential of Dual-IPO, and explicitly list this limitation. At the same time, Dual- IPO designed to be small-lab friendly: it reduces the dependence on repeated human annotation, keeps the additional optimization overhead modest and largely reusable, and attains strong (often SOTA-level) performance starting from comparatively small generators. In this section, we clarify (i) where the main computational bottlenecks lie, (ii) in what sense Dual-IPO is resource-friendly, and (ii) what practical trade-offs and cost-reduction strategies are available.
G.1 WHERE THE COMPUTE ACTUALLY GOES
In our current system, the dominant practical bottleneck comes from video generation and evalu- ation, rather than from the Dual-IPO algorithm itself:
Video sampling. Generating large numbers of videos from a high-quality T2V model is intrinsically expensive and tends dominate wall-clock time in any RL- preference- based alignment pipeline for video (similar observations have also been made in prior work such as DanceGRPO).
Automatic evaluation. Running VBench and other automated evaluators on the gener- ated videos adds another substantial cost, which is again shared by essentially all strong alignment methods that rely on large-scale offline evaluation.
By comparison, the additional overhead introduced by Dual-IPO itself is relatively small. our implementation, a single SRPO-based critic update takes roughly day 8x A100 GPUs. Once a small seed set of human annotations is prepared, all later reward-model updates are fully automated and do not require additional human labeling. Thus, most of the overall cost is shared with any competitive T2V alignment pipeline, whereas the incremental cost of Dual-IPO is modest.
G.2 COMPARISON TO STANDARD RLHF PIPELINES
A key motivation for Dual-IPO is to avoid the continuous human annotation that typically constitutes the main bottleneck in RLHF-style pipelines. many RLHF setups for LLaMAA-tyle models [1,2], the reward model is repeatedly updated on fresh model outputs using large-scale human preference datasets, and each new RL round requires collecting additional human labels. This process is:
time- and labor-intensive,
difficult and expensive to reproduce for typical academic or small industrial labs,
tightly coupled to large-scale human preference collection infrastructure.
In contrast, Dual-IPO is designed keep human supervision small and front-loaded:
We start from a small amount of human-labeled data to seed the critic.
We then apply SRPO to generate pseudo-labels and perform self-refinement of the critic, without further human annotation.
One SRPO-based critic update is single, bounded procedure (about day 8xA100 in our configuration), after which the critic can be re-used across multiple generators or training runs.
Compared with previous approaches that rely on tens thousands hundreds of thousands of manually labeled preference pairs and repeated reward-model training, our main focus is to reduce human annotation cost while still allowing the reward model to be updated during training. From the perspective of annotation and maintenance cost, this makes Dual-IPO relatively lightweight and more practical for small research groups.
G.3 GENERATOR AND CRITIC CONFIGURATIONS
Dual-IPO is explicitly designed to reach strong performance starting from comparatively small gen- erators, with critic stage that is modest relative full T2V pretraining.
25

---

Published as a conference paper at ICLR 2026
Generators. Our main gains come from aligning 2B CogVideos model, which achieves SOTA- level VBench performance and surpasses a 5B baseline after only three Dual-IPO iterations. This is model scale much closer to what typical research labs afford, and demonstrates that substantial alignment benefits can be obtained without training or fine-tuning extremely large generators.
Crities. On the critic side, fine-tune pre- treitrained VILA (13B/40B) reward model for 5 epochs on 16 video frames with small batch size on 32 GPUs, and then apply SRPO to generate pseudo- labels for further refinement. This is modest post-training stage compared full T2V pretraining. and the resulting critic can be reused across multiple generators or experiments. Labs with more limited resources can:
smaller critic models,
reduce the number of SRPO iterations,
or train on fewer frames per video
to trade off performance against cost in controlled and transparent way.
G.4 COST-REDUCTION STRATEGIES AND TRADE-OFFS
The configuration we report in the main paper should be viewed as a best-practice setting aimed at pushing SOTA performance, not minimum requirement of the algorithm. In practical deploy- ments, there are several trade-offs that can substantially reduce computational cost:
Smaller T2V models. As shown by the Cog VogVideo-22B experiments, smaller base gener- ators can already benefit strongly from Dual-IPO and even surpass larger baselines after few iterations.
Smaller critics. A smaller reward model, adaptation, can still provide useful gradients for preference optimization.
a partially frozen backbone with lighter
Fewer samples and iterations. Reducing the number of generated videos per iteration and/or the total number of Dual-IPO iterations directly scales down both generation and evaluation cost, while still yielding noticeable performance gains.
Reusing crities across runs. Once critic has been trained with SRPO, it can be reused across different generators and experiments, amortizing its cost over multiple projects.
In all of these settings, the main wall-clock time remains dominated by video generation and evalu- ation, which any competitive prffrence-basee alignment method must pay. Dual-IPO adds a man- ageable, well-structured optimization layer on top of this shared cost, rather than introducing funda- mentally new bottlenecks.
G.5 SUMMARY
In summary, Dual-IPO designed with both computational efficiency and practical deployability in mind:
The dominant cost arises from video sampling and evaluation, which is common to strong RL--based video training pipelines.
Dual-IPO significantly reduces human annotation requirements compared to standard RLHF pipelines by using a small seed of human labels together with SRPO-based pseudo- labeling and critic self-refinement.
The algorithm achieves strong, often SOTA-level performance starting from a 2B generator and modest eritic post-training stage, which more accessible typical research labs.
Multiple knobs (model size, number of iterations, samples per iteration, critic capacity) allow practitioners to flexibly trade off performance against cost.
We therefore believe that, while reported configuration is computationally demanding, Dual-IPo as a framework remains relatively lightweight and scalable, and suitable for small and medium- sized labs that seek strong alignment performance without large-scale, ongoing human annotation.
26

---

Published as a conference paper at ICLR 2026
G.6 IMPACT OF REWARD MODEL SIZE ON RL
We conducted comparative analysis of reward models of different sizes in the iterative preference optimization training process. As shown in Tab. A3, larger reward models provide more precise reward estimations, leading to improved alignment with human preferences. This enhanced reward accuracy contributes to better optmizaaiinn during preference alignment training, ultimately resulting in superior model performance. Our findings highlight the importance of scaling the critic model to achieve more reliable preference optimization and reinforce the benefits of leveraging larger critic architectures for improved alignment outcomes.
H ADDITIONAL QUALITATIVE COMPARISON
We provide more detailed qualitative comparison in Fig. A4 and Fig. A5, which demonstrates the superiority of our model over the baseline. This visualization highlights key improvements in fidelity and overall quality.By comparing outputs side by side, we illustrate the advantages of our model in handling complex scenarios, reinforcing its effectiveness in real-world applications.
I USE OF LLM
Large Language Models (LLMs) were employed solely auxiliary tools to improve the efficiency of manuscript preparation. Their use included assisting formatting 15TiX mathematical expressions, polishing the language style, debugging minor code or visualization scripts, and formatting references.
in retrieving related literature. editing and
J SOCIETAL IMPACTS
The emergence of advanced text-to-video generation pipelines and optimized training strategies marks a major breakthrough in Al-driven content creation. These innovations significantly lower barriers to video production, making it more efficient, cost-effective, and widely accessible. In- dustries such education, entertainment, marketing, and accessibility technology stand to benefit immensely, as creators can generate high-quality videos with minimal effort and resources.
However, alongside these advancements come ethical concerns, particularly regarding potential mis- use. One of the most pressing risks is the creation of highly realistic yet mislcading content, such deepfakes, which could be exploited for misinformation, reputational harm, social manipula- tion. Additionally, automated video curation may unintentionally reinforce biases present in training datasets, leading to the perpetuation of stereotypes discriminatory narratives.
On a larger scale, the widespread adoption of Al-driven video generation could disrupt traditional media industries, impacting professions like video editing, animation, and scriptwriting. The prolif- eration of hyper-realistic Al-generated content also raises concerns about media authenticity, making increasingly difficult for audiences to distinguish between real and synthetic footage. Furthermore, if access to these technologies remains limited to well-funded organizations, existing digital divides could widen, leaving smaller creators disadvantage.
To mitigate these risks, proactive measures must be taken. Implementing transparency fea- tures-such as metadata labels distinguish Al-generated content- enhance accountability. Ensuring diverse and ethically curated datasets is crucial to reducing bias and promoting fairness in outputs. Policymakers, industry leaders, and researchers must work together to develop regulatory frameworks that balance innovation with ethical responsibility. Additionally, providing educational resources and affordable access to AI tools can help democratize their benefits, preventing techno- logical disparities.
The rapid evolution of generative Al necessitates ongoing ethical oversight. Continuous evalua- tion of its societal impact, coupled with adaptive safeguards, is essential to minimizing harm while maximizing positive outcomes. Addressing these challenges requires a multidisciplinary approach, fostering responsible Al ecosystem that empowers creators without compromising ethical integrity.
27

---

Published as a conference paper at ICLR 2026
K
LIMITATIONS
While our approach represents a significant advancement in text-to-video generation, several chal- lenges persist. Addressing these issues presents valuable opportunities for future research and de- velopment. Below, we outline key limitations and potential solutions:
Computational Constraints Despite improvements in training efficiency, large-scale text-to- video models still demand substantial computational power. This limitation may hinder accessi- bility, particularly for individuals and organizations with limited resources. Future research should focus on developing lightweight architectures, model compression techniques, and efficient infer- ence strategies to enhance accessibility in resource-constraingg environments.
Dataset Representativeness and Cultural Adaptability The effectiveness of our model is highly dependent on the quality and diversity of its training dataset. While significant efforts were made curate a representative dataset, certain cultural contexts, niche topics, and underrepresented com- munities may not be adequately covered. Additionally, biases present in pretraining data could lead unintended stereotypes or inappropriate outputs. Expanding dataset coverage, improving curation techniques, and integrating fairness-awen training strategies will be crucial to enhancing contextual accuracy and mitigating bias.
Multimodal Hallucination and Interpretability Since evaluation models are fine-tuned mul- timodal large language models (MLLMs), they are susceptible to hallucination gtlluiinationngggnaaainnnn content that appears plausible but is factually incorrect or cannot be inferred from the input text and images. Furthermore, MLLMS inherit biases from their pretraining data, which may lead to inappropriate responses. While careful curation of supervised fine-tuning (SFT) data helps alleviate these issues, they are not fully resolved. In addition to these concerns, MLLMs also suffer from opacity, in- terpretability challenges, and sensitivity to input formatting, making it difficult to understand and control their decision- decision-making processes. Enhancing model explainability and robustness remains crucial area for future research.
Human Annotation Limitations Human evaluation plays critical role in assessing model per- formance, but is inherently subjective and influenced by annotator biases, perspectives, and in- consistencies. Errors noisy labels may arise during annotation, potentially affecting evaluation reliability. To address this, we conducted multiple rounds of trial annotations and random sampling quality inspections improve inter-annotator consistency. Additionally, we designed user-friendly annotation guidelines and interfaces to streamline the process and enhance accuracy. However, variations annotation methodologies across different platforms and annotators can still lead to discrepancies, limiting the generalizability of our results. Moreover, human annotation remains time-consuming and resource-intensive, constraining scalability. Future efforts could explore semi- automated evaluation approaches or crowdsourced methods to improve efficiency while maintaining quality.
Ethical Considerations and Regulatory Compliance Ensuring responsible AI deployment mains an ongoing challenge, even with safeguards in place to prevent misuse. The rapid advance- ment of Al-generated media demands continuous revisions ethical policies and regulatory frame- works. Proactively engaging with policymakers, industry leaders, and researchers is crucial for developing guidelines that uphold accountability, transparency, and fair usage. Embedding meta- data or digital watermarks in Al-generated content can improve traceability and help differentiate synthetic media from authentic sources. Furthermore, raising public awareness and education on Al-generated content help users recognize potential risks. Developers should prioritize responsi- ble AI innovation by ensuring algorithmic fairness while minimizing bias. Additionally, fostering cross-industry collaboration and promoting global standards will contribute to a more secure and trustworthy AI ecosystem.
28

---

Published as a conference paper at ICLR 2026
Oogerdack -8
Ours-28 28
with beard playing the piano in concert hall. black tuxedo and bow tie. His fingers dance on the keys as he plays beautiful melody.
Cogg Cagpuboo
A director, dressed dark blazer and casual jeans, sits at modern desk cluttered with scripts and notes. He types intently on sleek toptop, brow furrowed In concentration. The room is dimly by single lamp, casting warm glow over his focused expression. Behind him, large window frames city skyllne at dusk, adding sense of urgency creativity to scene.
A waman sits cazy, well-lit room, her hands deftly working long, creamy white scarf. She alive-green sweater and gentle smile, her eyes focused on the intricate pattern of knitting. The soft, natural light streaming through nearby window highlights the texture of the yarn and calm, atmosphere the space. Her concentration and skill are evident as she effortiessly manipulates the needles, creating beautiful, handmade scarf.
An actor dressed in rugged, casual attire, including a dark jacket, Jeans, and hilking boots, walks alongside a dingo in natural, wooded setting. The dingo, with its distinctive reddish-brown fur and alert eyes, gracefully beside the actor, who looks calm and engaged. The background features lush greenery and dappled sunlight filtering through
harmonious atmosphere. evident in their synchronized steps and mutual awareness of each other.
creating
dingo sync, their bond
Figure A4: Enhanced qualitative comparison. We conduct qualitative comparison between Cog Vide--0B and Dual-IPO-2B to evaluate their performance
29

---

Published as a conference paper at ICLR 2026
Cogiideat -2
Cu
A silver flying motorcycle with sleek design and powerful engine zooms through a city. The matorcycle's rider wears jacket they navigate through traffic.
Cogytdloo
A dressed robot astronaut walks barren, rocky landscape distant planet. His sleek, metallic suit glimmers under the dim light of distant sun, and his helmet reflects the alien terrain. He strides purposefully, leaving faint footprints in the dust, with of determination and exploration. The desolate environment, dotted with boulders and sparse vegetation, enhances the otherworldly atmosphere of his journey.
Cogiddort 28
A rugged worker in plaid shirt and denim Jeans leads strong, chestnut-colored horse The horse, adorned a simple leather harness, walks steadily beside the worker, who holds the reins firm yet gentle grip. The scene set the late afternoon, with worm sunlight casting tasting long shadows and highlighting the tex turess the dirt road and surrounding countryside. The worker's wea weathered face and horse's 's powerful convey sense of steadfast determina tian and mutual respect.
dusty, rural path.
A young girl in futuristic spacesult runs acress vast, ley landscape on distant planet, with towering mommoth-llke
discovery massive, shaggy form long tusks contrast stork, allen environment. The sky above is deep, star-filled space, with distant planets and nebulse visible, The girl's mammoth's steady pace convey
beside
determined expression otherworldly setting.
Figure A5: Enhanced qualitative comparison. We conduct qualitative comparison between Cog Video-BB and Dual-IPO-2B to evaluate their performance
30


Reviews:
[{"review_id": "5PAdQjn2JK", "reviewer": "Reviewer_pm1R", "summary": "This paper proposes Dual-IPO (Dual-Iterative Preference Optimization), a novel post-training framework for text-to-video (T2V) generation that enhances both synthesis quality and alignment with human preferences. The method involves a dual iterative paradigm that sequentially optimizes a reward model and a video generation model. The reward sign is refined through Chain-of-Thought (CoT) guided reasoning, voting-based self-consistency, and a preference certainty estimator to ensure reliable feedback. Using these improved reward signals, the video generation model is iteratively updated to enhance semantic consistency, motion smoothness, and aesthetic quality. Through multiple optimization rounds, both models are progressively improved without the need for extensive manual preference annotations. Experimental results show that Dual-IPO substantially improves video generation quality and allows smaller models to outperform larger baselines.", "questions": "My main concern regarding this paper is the diversity of the user-specified training data construction and the potential bias led by this procedure. If the authors could address my concern effectively. I'd be happy to raise my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The authors introduce a comprehensive and well-motivated dual-iterative framework that jointly optimizes the reward and generation models, addressing limitations of static preference alignment methods.\n\n2) The proposed approach demonstrates strong data efficiency, requiring only a small amount of human-annotated preference data to initiate the self-refinement process.\n\n3) The paper effectively captures the evolving nature of human preferences, emphasizing that fixed offline datasets may lead to overfitting and reduced generalization.\n\n4) Extensive experiments across multiple model architectures and scales (e.g., CogVideoX and Wan) are provided, supported by quantitative metrics (VBench) and qualitative analysis.", "weaknesses": "1) The construction of textual prompts used for generating training data is under-specified. The authors mention the use of structured elements (subjects, attributes, spatial relations, and actions), but the paper would benefit from more details regarding the **size, diversity, and balance of the prompt pool**. For instance, how many combinations were used per category, and how does this affect diversity and representativeness?\n\n2) The construction of textual prompts may inadvertently introduce **bias** into the generated training data. Since prompts reflect the authors’ design choices, such as the selection of subjects, attributes, and actions. They could encode cultural, demographic, or semantic biases present in the prompt corpus. For example, certain object, action combinations might be overrepresented, or specific demographic groups may be underrepresented if not explicitly balanced. Such biases could propagate through the downstream model, affecting its generalization and fairness. The paper would benefit from a discussion of how prompt templates were curated, whether any bias detection or mitigation strategies (e.g., prompt balancing, debiasing filters, or controlled generation) were applied, and how residual biases were evaluated or monitored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u4RmoDRa3X", "reviewer": "Reviewer_bijg", "summary": "This paper proposes Dual-Iterative Preference Optimization (Dual-IPO), a post-training framework that iteratively refines both a reward model and a text-to-video (T2V) generator to better align video synthesis with human preferences.\nThe method combines:\n\n- Self-Refined Preference Optimization (SRPO) — leveraging Chain-of-Thought (CoT) guided reasoning, self-consistency voting, and a Preference Certainty Estimator (PCE) for robust pseudo-labeling.\n\n- Dual iterative optimization — alternately improving the critic (reward) model and the video generation model via both Diffusion-DPO and Diffusion-KTO objectives.\nExperiments on CogVideoX, Wan, and VBench benchmarks show consistent gains in motion smoothness, aesthetic quality, and prompt consistency, with a 2B model surpassing a 5B baseline.", "questions": "- How sensitive is Dual-IPO to the initial human preference dataset size and quality? Would small or noisy seeds destabilize the dual-loop training?\n\n- Do you observe any reward overfitting or “reward hacking” effects in later iterations? If so, how does SRPO mitigate them?\n\n- Could the authors provide a more quantitative analysis of the trade-off between computational cost and performance gains? Specifically, each iterative round seems to require large-scale training with multiple pseudo-labeling and re-optimization stages. How does performance improvement scale with training cost (e.g., GPU hours vs. VBench gain)? Is there an observed saturation point after which additional iterations bring diminishing returns?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Dual optimization design – The interplay between a self-refined reward model and iterative generator updates is conceptually elegant and empirically validated.\n\n- Methodological soundness – The CoT-guided pseudo-labeling and PCE-weighted DPO/KTO training are carefully formulated and ablated.\n\n- Strong experimental validation – Includes results across model sizes, architectures, and both automatic and human evaluations, showing consistent improvement.\n\n- Clarity and completeness – Writing, figures, and methodology are clear; the paper includes reproducibility and ethics statements, and provides rich supplementary material.", "weaknesses": "- High computational cost – Each optimization round involves dual training of large models (VILA-40B, CogVideo-5B), making it unclear how scalable or practical Dual-IPO is for typical research labs.\n\n- Dependence on synthetic preference labels – Despite PCE filtering, the pseudo-label quality may still drift; more human validation or robustness analysis would strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zcuOmvLifd", "reviewer": "Reviewer_Qzdo", "summary": "This paper presents Dual Iterative Preference Optimization (Dual-IPO), a novel post-training framework designed to enhance T2V generation models through iterative optimization of both the reward model and the video generation model. Unlike conventional alignment approaches such as DPO and KTO, which rely on fixed, human-annotated datasets, Dual-IPO introduces a bi-directional training loop. On the one hand, the reward model is iteratively refined via SRPO using CoT-guided reasoning, multi-path self-consistency voting, and a preference certainty estimator. The T2V model is updated based on dynamically evolving reward signals through Diffusion-DPO or Diffusion-KTO optimization. Experiments on CogVideoX and Wan show significant improvements in semantic alignment, motion smoothness, and aesthetic quality. The proposed framework achieves 81.33% human preference accuracy and strong VBench gains, establishing new SOTA results.", "questions": "1.\tWhat is the convergence criteria of the dual-iterative preference optimization? Will the excessive iterations cause overfitting? Is there a risk of reward hacking or mode collapse in later iterations?\n2.\tWhat is the bottleneck that restricts the computational efficiency? Have you explored any strategies to reduce computational requirements? How does the performance scale with reduced computational budgets?\n3.\tCould you provide the ablation study of CoT, self-consistency and PCE to prove their effectiveness individually? And what if replace SRPO with static DPO?\n4.\tCould Dual-IPO be generalized beyond T2V? The design seems extensible.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe paper provides a motivation for addressing static reward limitations and distribution mismatch in preference alignment for generative video models.\n2.\tThe proposed Dual-IPO is a novel optimization paradigm. The idea of jointly optimizing the reward model and generator in a feedback loop is interesting and novel. It addresses reward drift and distribution mismatch issues, which are common in previous methods like DPO or RLHF. It also reduce the requirements on large-scale human annotations.\n3.\tThis paper proposes SRPO to train the reward model, which integrates CoT-guided reasoning, multi-path voting, PCE and SRPO loss. It supports both pairwise and pointwise supervision, which is flexible and unified for various alignment tasks.\n4.\tDual-IPO achieves new state-of-of-art performance. Experiments on CogVideoX (2B/5B) and Wan (1.3B) show consistent gains (e.g., +3.0 on VBench), demonstrating robustness across architectures and parameter scales. The experiment validation is comprehensive and rigorous.", "weaknesses": "1.\tThe paper does not analyze the convergence of the dual-iterative process. There is no proof that the process will stably converge stably to an optimal point, rather than oscillating. The paper also does not clearly state when the iterative process should stop.\n2.\tLack of ablation study. For example, the paper does not contain the complete ablation study of the three key parts of SRPO (CoT, self-consistency and PCE).\n3.\tThe method is complicated and inefficient, which is pointed out by the authors. It requires 128 GPUs and approximately two weeks per iteration, making it extremely resource-intensive. The paper does not analyze the efficiency of each component, or explore the strategy to reduce computational cost.\n4.\tIt would be better and clearer to illustrate SRPO and its components with graphs.\n5.\tSome typos (eg. in Figure 1 (c)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0XwzVAd21", "reviewer": "Reviewer_gKUH", "summary": "The paper introduces Dual-Iterative Preference Optimization (Dual-IPO), a framework that jointly and repeatedly improves both the reward model and the video generation model to better align diffusion-based T2V models with human preferences. The reward model is strengthened in each round via Chain-of-Thought guided reasoning, voting-based self-consistency, and a preference-certainty estimator to produce reliable pseudo-labels without extensive manual annotation. Using these refined signals, the video generator is iteratively optimized via DPO/KTO-style preference learning, and both models co-evolve over multiple rounds. Experiments across architectures and model sizes show consistent gains in subject consistency, motion smoothness, and aesthetics, including cases where a 2B model surpasses a 5B baseline.", "questions": "Please see the weakness for the major questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Dual-IPO attains a level of preference alignment that was not achievable with prior approaches. It is validated on state-of-the-art backbones (CogVideoX, Wan 2.1), and even a small model is shown to surpass the performance of a larger one in Table 5.\n- The effectiveness of the constructed reward models is discussed, particularly centered around Table 1.\n- Human evaluation results (e.g., Fig. 3b) are also included for complementary aspects, demonstrating that the improvements are not merely due to over-optimization against the evaluation model.", "weaknesses": "- The discussion on the number of iterative rounds is insufficient. It is unclear whether increasing the number of rounds continues to improve performance or saturates, and therefore, it is difficult to claim that an optimal recipe has been established.\n\n- For the experiments on Wan, not all VBench metrics are reported. Since VBench is biased toward temporal consistency, it remains possible that the training procedure reduces temporal dynamics in videos. This should be explicitly addressed. The trade-off between temporal consistency and dynamics has been discussed in prior work (e.g., [1], [2]).\n\n- Human Preferences are not shown for Wam 2.1 experiments. Is the performance gain for Wan 2.1 and more powerful models not efficient enough? To address this concern, I recommend that you show the results of human preference experiments for Wan 2.1 1.3B/14B.\n\n[1] Liu, et al. VideoDPO: Omni-Preference Alignment for Video Diffusion Generation. CVPR2025.  \n[2] Oshima, et al. Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search. NeurIPS2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)