VADER: Video Diffusion Alignment via Reward Gradients

Authors: Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, Deepak Pathak
Primary area: generative models
Keywords: Diffusion Model, Text-to-Video Generation, Generative Models

Abstract:
We have made significant progress towards building foundational video diffusion models. As these models are trained using large-scale unsupervised data, it has become crucial to adapt these models to specific downstream tasks. Adapting these models via supervised fine tuning requires collecting target datasets of videos, which is challenging and tedious. In this work, we utilize pre-trained reward models that are learned via preferences on top of powerful vision discriminative models to adapt video diffusion models. These models contain dense gradient information with respect to generated RGB pixels, which is critical to efficient learning in complex search spaces, such as videos. We show that backpropagating gradients from these reward models to a video diffusion model can allow for compute and sample efficient alignment. We show results across a variety of reward models and video diffusion models, demonstrating that our approach can learn much more efficiently in terms of reward queries and computation than prior gradient-free approaches.

Decision: reject
Reviewer scores: 3, 6, 8, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730617964117

thread_type:
official_review

<thread_object>
creator: reviewer_tg9v

<content>
summary:
This work introduces VADER, a method for aligning video diffusion models using reward gradients. VADER repurposes off-the-shelf vision discriminative models as reward models to adapt video diffusion models more effectively. Additionally, the paper presents practical techniques to optimize memory usage, enabling efficient training of VADER on standard hardware.

strengths:
Originality:
VADER is a novel approach to aligning video diffusion models using reward gradients rather than policy gradient methods. It creatively repurposes various pre-trained vision models as reward models, expanding their utility in video generation.

Quality:
The paper is methodologically rigorous, with strong experimental results demonstrating clear performance gains. Additionally, memory-optimizing techniques make VADER more accessible, broadening its potential user base.

Clarity:
The work is well-organized, with clear explanations and visualizations that effectively showcase the benefits of reward gradients and diverse reward models for alignment.

Significance:
VADER significantly advances practical video generation, making it more accessible and adaptable. This positions VADER as a valuable contribution to generative AI in video synthesis.

weaknesses:
The paper lacks a quantitative evaluation of the temporal coherence achieved with the v-jepa reward model. While Figure 9 provides qualitative evidence of improvement, the analysis would be more robust with a quantitative assessment of temporal consistency. Adding such an experiment would offer a more comprehensive understanding of VADER's performance in maintaining coherence over time when trained on the v-jepa reward.

While VADER incorporates multiple reward models, the paper lacks a detailed examination of how each specific reward model influences alignment objectives like temporal coherence, aesthetic quality, or text-video alignment. Additionally, it would be valuable to understand how optimizing for one reward, such as aesthetics, might impact the performance on other metrics, like temporal coherence.

questions:
How are the visualizations selected?
Are the visual examples in the paper randomly chosen, or were they curated to highlight specific successes of VADER? Understanding the selection process would clarify whether the results are representative or potentially cherry-picked.

Why aren’t popular metrics, such as FID/FVD, used in the experiments?
Frechet Video Distance (FVD) is a commonly used metric for evaluating video quality in generative models, albeit with its own limitations and pitfalls. Including it would allow for a more standardized and comparable evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642467404

thread_type:
official_review

<thread_object>
creator: reviewer_msp2

<content>
summary:
This paper introduces VADER, a method for fine-tuning video diffusion models using reward gradients to improve task-specific alignment. By utilizing pre-trained reward models as discriminators, VADER enhances video quality, text alignment, and temporal consistency. The approach employs memory optimization techniques to enable efficient training, even with limited resources. Experimental results demonstrate that VADER achieves strong performance across various video generation tasks.

strengths:
1. The authors propose VADER, which uses reward gradients to fine-tune video diffusion models, achieving efficient task adaptation. 
2. This paper experiments with various reward models, making it suitable for multiple video generation tasks and achieving strong results in both subjective evaluations and quantitative metrics.
3. This paper employs several optimization techniques (such as truncated backprop) to enable operation in resource-limited environments.

weaknesses:
1. The proposed VADER approach, which fine-tunes video diffusion models using reward gradients, optimizes network parameters with various reward models serving as discriminators. This enables improved adaptation to specific tasks. However, the use of reward models in generative model training is a well-explored concept. This paper just extends that approach within the context of diffusion models.
2. Since using reward models to backpropagate gradients requires the diffusion model to produce fully denoised outputs, all denoising steps must be executed, which places high demands on training resources. This might also lead to very small batch sizes. Although the authors employ several tricks to reduce resource usage, it raises the question of whether these adjustments impact the training outcomes. For instance, how much does backpropagation through only one timestep in the diffusion model affect the network parameters?
3. Some visual results in the paper still show misalignment between text and image content. For example, in Figure 7, the prompt "A bear playing chess" leads VADER to generate two bears.

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679822651

thread_type:
official_review

<thread_object>
creator: reviewer_sdfv

<content>
summary:
The paper proposed a reward fine-tune method called VADER. By using dense gradient information tied to generated RGB pixels, VADER enables more efficient learning in complex spaces like video generation. Backpropagating gradients from reward models to the video diffusion model facilitates both compute and sample-efficient alignment. Results across various reward and video diffusion models demonstrate that this gradient-based approach learns more efficiently than previous gradient-free methods in terms of reward queries and computational resources.

strengths:
1. The paper proposes to use gradient information for critic model preference tuning.
2. The experiment results are good.

weaknesses:
1. Lack of novelty, the paper is more likely to be a tech report rather than a paper. Directly backward the gradient is not novel for RL.
2. The paper proposes a on-policy strategy, while all the comparisons are off-policy strategies which is not fair. There are so many on-policy strategies[1] that perform better than off-policy strategies. 
3. Missing experiment details. How do you use DPO/DDPO on your dataset since they need preference pairs for training. How do you create the preference pair?


[1] Yuan, Weizhe, et al. "Self-Rewarding Language Models." Forty-first International Conference on Machine Learning.

questions:
1. More methods should be considered for comparison.
2. More details of the experiments should be enclosed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701812176

thread_type:
official_review

<thread_object>
creator: reviewer_wlqi

<content>
summary:
The authors introduce an alignment tuning method for video generation models utilizing gradient backpropagation from reward models. This approach addresses a critical need for producing high-quality, aligned video content. By directly guiding the generation model through reward gradients, the method achieves notable sample and computational efficiency compared to gradient-free approaches.

strengths:
- The problem to be solved is clearly defined, and its importance is well-acknowledged.

- The use of reward model gradients is highly intuitive and well-explained. Additionally, as the proposed methodology is data-free, it is practically useful and less likely to inherit biases from datasets used in alignment fine-tuning.

- Experimental results show this method is computationally efficient compared to gradient-free approaches such as DDPO and DPO.

weaknesses:
**Clarity of Contribution:** It is unclear what the authors’ unique contributions are. Utilizing reward model gradients for alignment tuning has already been demonstrated in text-to-image generation by Prabhudesai et al. and Clark et al. The authors should clarify what specific advancements they are claiming in this area.

Additionally, the authors mention that there are significant memory overhead issues in video models when implementing these methods, yet a clear, step-by-step ablation study is needed to show how each component of VADER addresses this problem.

Prabhudesai et al., "Aligning text-to-image diffusion models with reward backpropagation." arXiv. 2023.  
Clark et al. "Directly fine-tuning diffusion models on differentiable rewards." ICLR. 2024.

**Objective of the Alignment Tuning:** The results shown in the DDPO and DPO papers use significantly larger reward query samples. In this paper, however, smaller-scale experiments were conducted to highlight the sample and computational efficiency of reward gradients. This discrepancy may have resulted in DDPO and DPO showing unusually poor results. Since the current alignment settings are data-free, with comparable test times, sample and computational efficiency are not as critical. Therefore, it would be beneficial to include comparisons with DPO and DDPO results over longer training times.

questions:
**Comparison with Existing Guidance Methods:** Reward gradients are commonly used in guidance-based research within diffusion models, as seen in studies like DOODL and Universal Guidance. It would be beneficial to analyze and compare the proposed method with these approaches. If direct implementation of these methods is challenging, an alternative comparison could involve modifying the reward guidance objective to apply guidance through $\nabla_{x_t} R$ during generation, similar to the approach taken in this paper.

**Clarification in Experiments (Table 1):** It is unclear which video generation model serves as the baseline in Table 1. The experimental setup mentions the use of VideoCrafter, Open-Sora 1.2, and ModelScope. Is the baseline an average of these models, or is it based on one specific model? Further clarification on this would be helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723177859

thread_type:
official_review

<thread_object>
creator: reviewer_hj2c

<content>
summary:
This paper addresses the problem of aligning pretrained video generation diffusion models to downstream tasks/domains using available reward functions without using fine-tuning datasets. Specifically, the authors propose updating the parameters of the diffusion model using the gradients of the target reward functions. This is motivated by the analysis provided in the paper (figure 3), showing that the feedback from reward gradients scale much more with video resolution compared to methods based on policy gradients. The authors apply their method to different reward functions, such as image aesthetics evaluation, image-text alignment, and video temporal consistency evaluation functions. The authors also incorporate some techniques to maintain efficiency when fine-tuning the retrained model. The proposed method is compared with multiple pretrained video generation models, as well as their aligned version using policy-gradient-based methods.

strengths:
- The paper is very well-written and well-structured, making it easy to follow and understand.

- The provided analysis in Fig. 3, which shows the gap between the feedback from reward gradients and policy gradients for higher resolution videos, is valuable.

- The method is evaluated on multiple base video generators, showing consistent results.

- The proposed method shows better results in comparison to the policy-gradient-based baselines in terms of the evaluated metrics. This is also noticeable in the visual results.

- The proposed method has better generalizability on unseen text prompts compared to the baseline alignment methods.

weaknesses:
- **Novelty**: To my understanding, the proposed method is in essence a standard fine-tuning method with the objective of maximizing task-specific discriminative/reward functions. The proposed techniques for efficiency, including LoRA, truncated back propagation, and frame subsampling are also all standard and commonly used in different areas. The amount of technical novelty is not a major concern as long as the method has significant findings. However, the behavior shown in the paper, i.e. better alignment of the diffusion model when directly optimized to maximize the target reward function, is not very surprising to me.

- **Experiments**: 
    - In addition to regression in generalizability, another potential down-side of fine-tuning methods is the reduced diversity of the aligned model. Therefore, it is important to properly evaluate the diversity of the generated videos using the aligned model. For example, it would be interesting to see how diverse the videos are the same text prompt compared to the base model. 
    - Additionally, I noticed no video visualizations are provided for aligned models using video reward functions. For example, in the results provided in Fig. 9 does not show much temporal motion in the generated videos. This could also relate to the previous point, where the model could sacrifice temporal variations for more consistent frames.

questions:
Please see the concerns in the Weaknesses section. I am open to increasing my score, if the authors could clarify their novelty and contribution more, and address my concerns about the experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730901801222

thread_type:
official_review

<thread_object>
creator: reviewer_bkeq

<content>
summary:
This paper proposes to utilize pre-trained reward models that are learned via preferences on top of powerful vision discriminative models to adapt video diffusion models. The results across a variety of reward models and video diffusion models showcase the effectiveness of the proposed approach.

strengths:
The authors propose the use of reward models aimed at enhancing the quality of generated videos. The experimental results demonstrate promising improvements in video quality, showcasing the effectiveness of the proposed approach.

weaknesses:
1. Algorithm 1 represents a standard approach that utilizes reward feedback within a diffusion model framework, lacking significant innovation.
2. A critical challenge in applying reward feedback to diffusion models is the precise definition and training of the reward model. The manuscript employs certain image-based methods to establish the reward function for video generation; however, these methods may not adequately encapsulate the unique characteristics of video data. It would be beneficial if the authors developed and trained a reward model specifically tailored for video content, similar to advancements made in the field of image rewards [1], thereby contributing more meaningfully to the domain.
3. There appears to be an error in Equation (231). Could you please provide a detailed derivation to clarify this point?
4. The ordering of equations has been overlooked starting from Equation (3). Please ensure all equations are correctly sequenced for clarity and coherence.
[1] Imagereward: Learning and Evaluating Human Preferences for Text-to-Image Generation

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732827124805

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their detailed feedback and thoughtful engagement with our work. We appreciate that several reviewers acknowledged the significance of aligning video diffusion models with reward gradients (HJ2c, wLqi, tG9V) . Reviewers also noted the clarity and quality of our presentation (HJ2c, tG9V), as well as the strong experimental results demonstrating the effectiveness of VADER across diverse reward models (HJ2c, wLqi, mSP2).

A common concern raised by reviewers is novelty. We therefore state the contributions of our work:

- We are amongst the first few works to successfully align video diffusion models using reinforcement learning. We are the first to do entirely online reinforcement learning, that is we do not use any external datasets during training. We show results using a wide range of video models, reward models and benchmarks.

- We show that using reward gradients for aligning video diffusion models is promising, and can be very resourceful (specifically in the video setting which is very compute bottlenecked). VADER with VideoCrafter as the base model requires less than 24 GPU hours  to train and can fit within 24GBs of GPU VRAM. 

- Finally we perform a very dense evaluation for video reward alignment. We compare against various popular methods in the image alignment community such as Diffusion-DPO, DDPO etc. Further we show results across various benchmarks (VBench, EvalCrafter), while using various base video diffusion models such as  text to video and image to video models. Lastly we show that existing image reward models and self-supervised trained video discriminative models such as V-JEPA, VideoMAE and ViCLIP can be successfully used as reward model.

Since the submission deadline, we have added new experiments which can be summarized as follows:
- We have included results with the **ViCLIP video reward model**, which helps improve the temporal dynamics of our generated videos. (bKeQ, HJ2c, mSP2, tG9V). Experiment link - https://vader-anonymous.github.io/#aesthetic-and-viclip-reward
- We have added strong benchmarks such as **VBench** and **EvalCrafter**, where we compare temporal dynamics, consistency, subject or background correctness and various other metrics of the generated videos. (bKeQ, HJ2c, wLqi, tG9V). Experiment link - (https://vader-anonymous.github.io/#vbench-evaluation | https://vader-anonymous.github.io/#evalcrafter-evaluation)
- We have added new baselines, which include **DOODL** [1] and an **on-policy versions of DDPO and DPO** (wLqi, sDFV). Experiment link - (https://vader-anonymous.github.io/#training-efficiency-comparison | https://vader-anonymous.github.io/#doodl-vader)
- We further study how finetuning using a specific reward model affects the results on the other reward functions, thus studying the **correlation between different reward functions** (tG9V) Experiment link - https://vader-anonymous.github.io/#reward-correlation
- We also study the **diversity of generated videos in VADER** when  finetune using various reward models. (HJ2C). Experiment link - https://vader-anonymous.github.io/#diversity-test
- We **ablate various memory reduction tricks** proposed in VADER along with the **number of truncation steps** used (wLqi, mSP2).  Experiment links - (https://vader-anonymous.github.io/#memory-usage-comparison | https://vader-anonymous.github.io/#truncated-backpropagation-ablation)
- Lastly we provide **hundreds of qualitative visualization** of the generated videos from VADER and the baseline (tG9V).  Experiment link - https://vader-anonymous.github.io/Video_Gallery.html

All these experiments can be found on our updated appendix in the paper  and at https://vader-anonymous.github.io/.  

We also apologize to the reviewers for a relatively late response. As we had six reviewers, it took us relatively more time to conduct all the requested experiments.

Below, we address each reviewer's concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313477864

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their consistent engagement and thoughtful critique of our work.  Below we summarize the experiments we ran since our first rebuttal response:

- We **compare VADER against T2V-Turbo and various other baselines**, on Vbench and EvalCrafter. We find VADER consistently outperforms the baselines on these benchmarks. Experiment Links - (https://vader-anonymous.github.io/#vbench-evaluation-standard-prompt-suite, https://vader-anonymous.github.io/#evalcrafter-evaluation) 
- We **train a custom video reward model using the automated rewards from VBench**, essentially creating a differentiable distilled VBench reward model. We then use this reward model to finetune VADER and find that this improves the dynamicness of the generated videos on unseen prompts. Experiment Links - https://vader-anonymous.github.io/#vbench-distilled-reward-model
-  We ran **qualitative visualization comparing the in-domain diversity between VADER and VideoCrafter** baselines. We do not find major reduction in diversity in VADER compared to VideoCrafter baseline. Experiment Link - https://vader-anonymous.github.io/Video_Diversity.html

We will update these results in the revised version of our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents a method to efficiently adapt large-scale video diffusion models to downstream tasks by leveraging pre-trained reward models. The results show the effectiveness of the method.

The paper is generally well-written. The results are convincing and demonstrate the effectiveness of the method. 

The paper has mixed review scores: 866385.
After the rebuttal, internal discussions among reviewers and AC were initiated. Two critical issues below were identified, which led to the rejection decision.

[limited novelty] reviewers sDFV, mSP2, and HJ2c raised their concerns about novelty, though some of them gave a score of 6. Using a discriminator to directly backward the gradient is widely used in many studies and lacks significant novelty. This work applies gradient backpropagation in video diffusion models. It is incremental work, despite good performance in several benchmark tests.

[text video alignment is not addressed well using reinforcement learning] Models fine-tuned with the Aesthetic Reward Model fail to align with text, because this reward model is trained to evaluate aesthetics. As pointed out by reviewer mSP2: in Figure 8, the output does not depict a starry sky as described. Similarly, in the newly presented experimental results titled “DOODL vs. VADER,” prompts like 2 dogs and a whale, ocean adventure and Teddy bear and 3 real bears continue to exhibit this problem. 

The authors are encouraged to incorporate this feedback into the revised version and resubmit the work to future conferences.