Title: Improving Video Generation with Human Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

human feedback

flow matching
Abstract: Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices\nimpact its rewarding efficacy. It brings three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Constructed a relatively comprehensive dataset (as listed in Table 1).\n- Proposed three effective algorithms specifically designed to enhance video generation quality for flow-matching methods.\n- Provided extensive experimental results, effectively demonstrating both the limitations and standout performance of the new reward model.\n\nWeaknesses:\n\nThe paper relies solely on model-generated data to construct the training set, which may be problematic. Based on our experience, even state-of-the-art models (e.g., Keling, Veo3, etc.) often fail to produce video clips of fully satisfactory quality, even after multiple sampling attempts. Consequently, we believe this approach makes it difficult for the trained reward model to surpass the performance ceiling of existing models.\n\nQUESTIONS:\nRegarding the reward model methodology, do you plan to open-source your dataset and pre-trained reward model? (This won’t affect my evaluation of the paper—it’s merely a suggestion.) For methods like DPO or GRPO, the quality of the constructed dataset often has a more significant impact than the algorithmic details themselves. Open-sourcing the code and data could greatly benefit future research and applications by enabling others to build upon your work.\n\nCurrently, training solely on synthetic data introduces the limitations I mentioned earlier for the reward model. However, using real data (e.g., actual video clips) directly often leads to failures in DPO/GRPO training. Have you explored solutions to this issue? Specifically, have you attempted training with real data, and if so, does your method still perform effectively?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces VideoReward, a three-dimensional reward model for video generation. To evaluate its effectiveness, the authors further propose three strategies: two training-time strategies (Flow-DPO and Flow-RWR) and one test-time strategy (Flow-NRG), all tailored for flow-based video generation models. Experimental results demonstrate the effectiveness of VideoReward and the proposed strategies on their internal VDM.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenth:**\n1. Proposes three novel optimization strategies for flow-based models, covering both training and test-time stages.\n2. Contributions in building a preference dataset, reward model, and reward benchmark, showcasing significant workload and effort.\n\n\n**Weaknesses:**\n1. **Unclear Motivation:** The motivation for introducing three strategies for flow-based models within a reward model is not clearly articulated. It would be helpful to clarify why these strategies are necessary.\n2. **Limited Preference Aspects:** The human annotations only consider three preference aspects (VQ, MQ, and TA), which seem insufficient for evaluating video generation. Additional aspects, such as temporal coherence, should be considered, or the authors could provide further justification for focusing solely on these three aspects.\n3. **Triplets Construction:** According to Line #99, preference video pairs are selected from distinct VDMs. However, due to the inherent quality differences between VDMs, selecting video pairs from the same VDM (with different random seeds) might result in more comparable positive and negative videos, enabling finer-grained preference modeling.\n4. **Video Alignment Evaluation:** The evaluation is conducted exclusively on the authors’ internal pretrained model, which is insufficient to validate the generalizability of both the reward model and the proposed optimization strategies. Broader evaluations on more base models are necessary.\n5. **Reference or Experimental Support:** Some claims lack adequate references or experimental support. For instance, Line #27 regarding previous reward models and Line #251 regarding the effects of full fine-tuning vs. LoRA fine-tuning require more evidence or citations.\n\nQUESTIONS:\nPlease refer to the weaknesses for detailed concerns.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an approach to enhance video generation by integrating human feedback, aiming to overcome issues such as unsmooth motion and misalignment between video content and prompts. The authors construct a comprehensive human preference dataset, focusing on advanced video generation models and incorporating pairwise annotations across multiple dimensions. The paper presents three alignment algorithms tailored for flow-based models: Flow-DPO, Flow-RWR, and an inference-time method, Flow-NRG.\nExperimental results demonstrate that VideoReward outperforms existing reward models and that Flow-DPO achieves superior performance over Flow-RWR and conventional supervised fine-tuning methods. Additionally, Flow-NRG offers users the flexibility to assign custom weights to multiple objectives during inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written with clear motivation, addressing two key challenges: the quality of reward data in previous methods and their focus on non-Flow Matching paradigms.\n2. The paper conducts extensive experiments, thoroughly verifying the performance of the proposed method, leading to reliable experimental conclusions.\n3. The design of Noisy Reward Guidance to replace classifier-free guidance is intriguing.\n\nWeaknesses:\n1. In the Discussion on βt section, the paper ultimately opts for a time-independent constant β. There seems to be no significant difference between Equations 6 and 4, as they merely change the prediction target from diffusion noise to the FM velocity field. Consequently, is there a substantial distinction between Flow-DPO and Diffusion-DPO? Is it necessary to differentiate them? Similarly, for Flow-RWR, Equations 9 and 8 present similar issues, raising questions about the necessity of distinguishing between Flow-RWR and diffusion-based RWR.\n2. The novelty of the paper is relatively limited. Although the introduced Noisy Reward Guidance is interesting, the core design concepts, such as Flow-DPO, do not seem to represent significant improvement or innovation over previous work.\n3. I am curious about the performance difference of the proposed method when using the original classifier-free guidance instead of Noisy Reward Guidance.\n4. According to the quantitative results on VBench, there does not appear to be a significant improvement in the total metric compared to the pre-trained model.\n\nMinor Weaknesses:\n1. The Efficient Reward on Noisy Latents section lacks detailed information, which can cause confusion. For instance, the statement \"We then adopt the Bradley–Terry loss to learn the reward function from these noised videos\" is unclear whether the loss training is based on noisy latents or pixel space noisy videos.\n\nQUESTIONS:\nI will consider whether to adjust my score based on how my concerns are addressed and the comments of other reviewers.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a complete solution for improving the visual quality of flow-based video generation models through human feedback. This includes:\n* A large-scale multi-dimensional human preference dataset,\n* A video reward model trained on the dataset,\n* A reward model benchmark dataset containing 26.5K video pairs,\n* 3 algorithms for improving video generation with the help of the reward model.\nEach individual component contain their own innovations and empirical justifications:\nThe preference dataset is collected from more recent video generators to raise the standard, and includes both point-wise and pair-wise annotations.\nNext, several design decisions for reward modeling are explored, with the Bradley-Terry model with ties achieving the best performance.\nFinally, the reward model is used to improve a flow-based video generator using newly-derived Flow-DPO, Flow-RWR, and Flow-NRG formulations. Flow-DPO improves the generation over the base model both quantitatively and visually.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* This paper provides a complete guideline for building a full RLHF workflow for flow-based video generation models and more. It serves as a valuable resource for future research.\n* The paper includes comprehensive ablations, such as the comparison between point-wise and pair-wise rewards, as well as the scaling factors for Flow-DPO. They are good references for future works.\n* The proposed reward model achieves SOTA performance, while the video generator show promising improvement after DPO.\n* The paper is well written and logically well-structured, including just the right amount of detail for such an involved pipeline without overwhelming the reader. For example, \n\n### Weaknesses\n* The quantitative evaluation for Flow-NRG seems to be missing.\n\nQUESTIONS:\n* How does Flow-NRG. compare with other methods?\n* The paper mentions reward hacking issues with Flow-DPO when using time-dependent $\\beta$. Could the authors elaborate how this reward hacking manifests visually in the generated videos?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-04"}