AVID: Adapting Video Diffusion Models to World Models

## Abstract

Abstract Large-scale generative models have achieved remarkable success in a number of domains. However, for sequential decision-making problems, such as robotics, action-labelled data is often scarce and therefore scaling-up foundation models for decision-making remains a challenge.
A potential solution lies in leveraging widely-available unlabelled videos to train world models that simulate the consequences of actions.
If the world model is accurate, it can be used to optimize decision-making in downstream tasks.
Image-to-video diffusion models are already capable of generating highly realistic synthetic videos.
However, these models are not action-conditioned, and the most powerful models are closed-source which means they cannot be finetuned.
In this work, we propose to adapt pretrained video diffusion models to action-conditioned world models, without access to the parameters of the pretrained model.
Our approach, AVID, trains an adapter on a small domain-specific dataset of action-labelled videos. AVID uses a learned mask to modify the intermediate outputs of the pretrained model and generate accurate action-conditioned videos. We evaluate AVID on video game and real-world robotics data, and show that it outperforms existing baselines for diffusion model adaptation. 1 1 1 Code is available on GitHub . Our results demonstrate that if utilized correctly, pretrained video models have the potential to be powerful tools for embodied AI.

## 1 Introduction

Large generative models trained on web-scale data have driven rapid improvement in natural language processing (Brown, ; Touvron et al., ; Achiam et al., ), image generation (Rombach et al., ), and video generation (OpenAI, ).
The potential for scaling to unlock progress in sequential-decision making domains, such as robotics, gaming, and virtual agents, has invoked a surge of interest in foundation models for decision-making agents (Reed et al., ), particularly for robotics (Brohan et al., ; ). However, the quantity of action-labelled data in these domains remains a significant bottleneck (Padalkar et al., ). This raises the question of to how to utilize widely-available unlabelled videos to bootstrap learning (Baker et al., ; Bruce et al., ).
One promising approach is to use video data to learn a world model (Ha & Schmidhuber, ), a model the predicts the consequences of actions and acts as a learned simulator.
Such a model can be used to optimize decision-making for downstream tasks (Hafner et al., ).

Current image and video diffusion models are highly adept at generating text-conditioned synthetic data (Podell et al., ; Zhang et al., ; Blattmann et al., ). If actions can be expressed as natural language, these models have the potential to be used out-of-the-box for decision-making (Kapelyukh et al., ; Zhu et al., ).
However, in many real-world domains the core challenge is optimizing low-level actions, such as joint angles in robotics, and therefore using natural language as the only interface is insufficient.
To overcome this limitation, one option is to finetune a pre-trained model to condition on low-level actions for a domain-specific dataset (Seo et al., ).
Another possibility is to apply existing adapter architectures such as ControlNet (Zhang et al., ), to add action-conditioning by modifying the activations inside the original model.
However, the parameters for state-of-the-art video diffusion models are usually not available publicly (LumaAI, ; OpenAI, ; RunwayML, ), which rules out these approaches.

In this work we address the problem of exploiting a pretrained video diffusion model to generate action-conditioned predictions without access to the parameters of the pretrained model.
Inspired by the recent work from Yang et al. (), we instead assume that we only have access to the noise predictions of the pretrained diffusion model.
We propose AVID, a domain-specific adapter that conditions on actions and modifies the noise predictions of the pretrained model to generate accurate action-conditioned predictions.
To train the adapter, we assume that we have access to a domain-specific dataset of action-labelled videos.
The core contributions of our work are:

- •
Proposing to adapt pretrained video diffusion models to action-conditioned world models, without access to the parameters of the pretrained model.
- •
Analyzing the limitations of the adaptation approach proposed by Yang et al. ().
- •
AVID, a novel approach to adding conditioning to pretrained diffusion models. AVID applies a learned mask to the outputs of of a pretrained model, and combines them with conditional outputs learned by a domain-specific adapter.

We evaluate AVID on video game data, as well as real-world robotics data where we use a 1.4B parameter model trained on internet-scale data as the pretrained model (Xing et al., ).
Our results show that our approach outperforms existing baselines, and demonstrates that AVID obtains a considerable benefit from using the pretrained model, even with limited access to the model.
We advocate for providers of closed-source video models to provide access to intermediate model outputs in their APIs to facilitate more flexible use of these models.

## 2 Preliminaries

#### Denoising Diffusion Probabilistic Models (DDPM)

Diffusion models (Ho et al., ; Sohl-Dickstein et al., ) are a class of generative models.
Consider a sequence of positive noise scales, $0<\beta_{1},\beta_{2},\ldots,\beta_{N}<1$.
In the forward process, for each training data point $\mathbf{x}_{0}\sim p_{\mathrm{data}}(\mathbf{x})$, a Markov chain $\mathbf{x}_{0},\mathbf{x}_{1},\ldots,\mathbf{x}_{N}$ is constructed such that $p(\mathbf{x}_{i}\ |\ \mathbf{x}_{i-1})=\mathcal{N}(\mathbf{x}_{i};\sqrt{1-
\beta_{i}}\mathbf{x}_{i-1},\beta_{i}\mathbf{I})$.
Therefore, $p_{\alpha_{i}}(\mathbf{x}_{i}\ |\ \mathbf{x}_{0})={\mathcal{N}(\mathbf{x}_{i};
\sqrt{\alpha_{i}}\mathbf{x}_{0},(1-\alpha_{i})\mathbf{I})}$, where $\alpha_{i}:=\Pi_{j=1}^{i}(1-\beta_{j})$.
We denote the perturbed data distribution as $p_{\alpha_{i}}(\mathbf{x}_{i}):=\int p_{\mathrm{data}}(\mathbf{x})p_{\alpha_{i
}}(\mathbf{x}_{i}\ |\ \mathbf{x})\mathrm{d}\mathbf{x}$.
The noise scales are chosen such that $\mathbf{x}_{N}$ is distributed according to $\mathcal{N}(\mathbf{0},\mathbf{I})$.
Define $s(\mathbf{x}_{i},i)$ to be the score function of the perturbed data distribution: $s(\mathbf{x}_{i},i):=\nabla_{\mathbf{x}_{i}}\log p_{\alpha_{i}}(\mathbf{x}_{i})$, for all $i$.
Samples can be generated from a diffusion model via Langevin dynamics by starting from $\mathbf{x}_{N}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and following the recursion:

$$ $\mathbf{x}_{i-1}=\frac{1}{\sqrt{1-\beta_{i}}}(\mathbf{x}_{i}+\beta_{i}s_{ \mathbf{\theta}}(\mathbf{x}_{i},i))+\sqrt{\beta}_{i}\mathbf{z},$ (1) $$

where $s_{\theta}$ is a learned approximation to the true score function $s$, and $\mathbf{z}$ is a sample from the standard normal distribution.
If we reparameterize the sampling of the noisy data points according to: $\mathbf{x}_{i}=\sqrt{\alpha_{i}}\mathbf{x}_{0}+\sqrt{1-\alpha_{i}}\bm{\epsilon}$, where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, we observe that
$\nabla_{\mathbf{x}_{i}}\log p_{\alpha_{i}}(\mathbf{x}_{i}|\ \mathbf{x}_{0})=-
\frac{\bm{\epsilon}}{\sqrt{1-\alpha_{i}}}.$
Therefore, we can define the estimated score function
in terms of a function $\mathbf{\epsilon_{\theta}}$ that predicts the noise $\bm{\epsilon}$ used to generate each sample

$$ $s_{\theta}(\mathbf{x}_{i},i):=-\frac{\mathbf{\epsilon}_{\theta}(\mathbf{x}_{i} ,i)}{\sqrt{1-\alpha_{i}}}.$ (2) $$

The noise prediction model $\mathbf{\epsilon}_{\theta}$ is trained to optimize the objective

$$ $\mathbf{\theta}^{*}=\operatorname*{arg\,min}_{\mathbf{\theta}}\mathbb{E}_{ \mathbf{x}_{0}\sim p_{\mathrm{data}}(\mathbf{x}),{\bm{\epsilon}\sim\mathcal{N} (\mathbf{0},\mathbf{I})},i\sim\mathcal{U}(\{1,2,\dots,N\})}\big{[}||\bm{ \epsilon}-\epsilon_{\theta}(\sqrt{\alpha_{i}}\mathbf{x}_{0}+\sqrt{1-\alpha_{i} }\bm{\epsilon},i)||^{2}\big{]}.$ (3) $$

To add a conditioning signal to the diffusion model, the denoising can be trained with an additional conditioning signal $\mathbf{\epsilon}_{\theta}(\mathbf{x}_{i},i,c)$, where $c$ is the desired conditioning signal.

#### Probabilistic Adaptation of Diffusion Models

Yang et al. () proposes to model the problem of diffusion model adaptation via a product of experts. Given a pretrained model $p_{\text{pre}}(\mathbf{x})$ and a small domain specific video model $p_{\text{adapt}}(\mathbf{x})$, the final adapted model is defined as the following product of expert (PoE) distribution:

$$ $p_{\textnormal{PoE}}:=\frac{p_{\text{pre}}(\mathbf{x})p_{\text{adapt}}(\mathbf {x})}{Z},$ $$

which generates samples that are likely under both of the original models, with the aim of maintaining strong video quality from $p_{\text{pre}}(\mathbf{x})$ and the desired domain-specific quality from $p_{\text{adapt}}(\mathbf{x})$.
In practice, $p_{\textnormal{PoE}}$ is intractable. To sample from the PoE, the authors propose to compose their scores:

$$ $\mathbf{\epsilon}_{\text{PoE}}(\mathbf{x}_{i},i,c):=\mathbf{\epsilon}_{\text{ pre}}(\mathbf{x}_{i},i,c)+\mathbf{\epsilon}_{\text{adapt}}(\mathbf{x}_{i},i,c),$ $$

and pass the combined score to a DDPM sampler.

## 3 Adapting Video Diffusion Models to World Models (AVID)

### 3.1 Problem Setting

In the context of video diffusion models each datapoint is a video, $\mathbf{x}=[x^{0},x^{1},\ldots,x^{T-1}]$, where $x^{\tau}$ indicates a video frame. Note that we use superscript indices, $x^{\tau}$, to indicate steps in time, and subscript indices, $\mathbf{x}_{i}$, to indicate steps in a diffusion process. We assume that we have access to a pretrained image-to-video diffusion model $\epsilon_{\textnormal{pre}}$, that is trained on web-scale data. Given an initial image, $x^{0}$, the image-to-video model $\epsilon_{\textnormal{pre}}$ generates a synthetic video $\hat{\mathbf{x}}=[x^{0},\hat{x}^{1},\ldots,\hat{x}^{T-1}]$.

We consider each video to be a sequence of observations generated by a partially observable Markov decision process (POMDP) (Kaelbling et al., ), with corresponding action sequence $\mathbf{a}=[a^{0},a^{1},\ldots,a^{T-1}]$. For each domain, we assume access to a dataset of action-labelled videos, $\mathcal{D}=\{(\mathbf{x},\mathbf{a}),\ldots\}$. Given a new initial image, $x^{0}$, and action sequence $\mathbf{a}$, the goal is to generate a synthetic video $\hat{\mathbf{x}}$ that accurately depicts the ground-truth realization of the actions.

To utilize the pretrained model $\epsilon_{\textnormal{pre}}$, we need to add action-conditioning to the model as accurate videos cannot be generated without the actions. Adding a new conditioning signal to a pretrained diffusion model is well-explored in previous works (Zhang et al., ; Mou et al., ; Mokady et al., ). However, most existing works assume access to the parameters of the pretrained model. In this work, we assume that we do not have access to the parameters of the pretrained model.

### 3.2 Limitations of naive adaptation of Yang et al. ( 2024b )

Yang et al. () propose to adapt a pretrained diffusion model to a specific use-case without access it its weights by composing (omitting the prior strength $\lambda$):

$$ $\displaystyle\mathbf{\epsilon}_{\text{PoE}}(\mathbf{x}_{i},i,c):=\mathbf{ \epsilon}_{\text{pre}}(\mathbf{x}_{i},i,c)+\mathbf{\epsilon}_{\text{adapt}}( \mathbf{x}_{i},i,c)$ (4) $$

However, this method has a fundamental limitation that it will produce biased samples. To see this, consider the following forward diffusion process

$$ $d\mathbf{x}=-\frac{1}{2}\beta(t)\mathbf{x}dt+\sqrt{\beta(t)}dW,$ $$

where $t$ indicates continuous time. Assume the intial target distribution at $t=0$ is $p_{0}(\mathbf{x}):=p_{\textnormal{PoE}}=\frac{p_{\text{pre}}(\mathbf{x})p_{
\text{adapt}}(\mathbf{x})}{Z}$.
Notice that the transition kernel can be written as (Särkkä & Solin, ):

$$ $p(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}(\mathbf{x}_{t};\mathbf{x}_{0}e^{- \frac{1}{2}\int_{0}^{t}\beta(s)ds},I-Ie^{-\int_{0}^{t}\beta(s)ds})$ $$

Therefore, for $\forall t>0$, the resulting distribution is the convolution $p_{t}(\mathbf{x}_{t})=p_{0}(\mathbf{x}_{0})*\mathcal{N}(\mathbf{x}_{t}-\mathbf
{x}_{0}e^{-\frac{1}{2}\int_{0}^{t}\beta(s)ds};0,I-Ie^{-\int_{0}^{t}\beta(s)ds})$.
Due to the fact that convolution does not distribute over multiplication, the resulting score $s(\mathbf{x}_{t},t)$ cannot be expressed as a sum of the two individual score functions:

$$ $\displaystyle s(\mathbf{x}_{t},t)=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log\left[p_{0}(\mathbf{x}_{0})*\mathcal{N }(\mathbf{x}_{t}-\mathbf{x}_{0}e^{-\frac{1}{2}\int_{0}^{t}\beta(s)ds};0,I-Ie^{ -\int_{0}^{t}\beta(s)ds})\right]$ $\displaystyle\neq$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log\left[p_{0,\text{pre}}(\mathbf{x}_{0}) *\mathcal{N}(\mathbf{x}_{t}-\mathbf{x}_{0}e^{-\frac{1}{2}\int_{0}^{t}\beta(s) ds};0,I-Ie^{-\int_{0}^{t}\beta(s)ds})\right]$ $\displaystyle+\nabla_{\mathbf{x}_{t}}\log\left[p_{0,\text{adapt}}(\mathbf{x}_{ 0})*\mathcal{N}(\mathbf{x}_{t}-\mathbf{x}_{0}e^{-\frac{1}{2}\int_{0}^{t}\beta( s)ds};0,I-Ie^{-\int_{0}^{t}\beta(s)ds})\right]$ $$

As a result, the true noise prediction of the target distribution also cannot be expressed as a sum: $\mathbf{\epsilon}_{\text{PoE}}(\mathbf{x}_{t},t,c)\neq\mathbf{\epsilon}_{\text
{pre}}(\mathbf{x}_{t},t,c)+\mathbf{\epsilon}_{\text{adapt}}(\mathbf{x}_{t},t,c)$. Therefore, the composition in Equation 4 does not hold and will result in biased samples.
In the following section, we propose AVID to overcome this limitation.
Rather than attempting to compose two independently trained models, AVID uses the outputs of the pretrained model to train an adapter that directly optimizes the denoising loss.

### 3.3 Adapting Video Diffusion Models to World Models (AVID)

AVID is a new approach for diffusion model adaptation that does not require access to the pretrained model. The motivation for AVID is that while pretrained image-to-video models can generate realistic videos, they cannot generate videos that are accurate with respect to a given sequence of actions. To achieve accuracy, the pretrained model must be guided towards the correct generation for the action sequence. However, as our experiments show, techniques such as classifier(-free) guidance do not perform well in this setting. AVID addresses this by training a lightweight adapter to adjust the output of the pretrained model to achieve accurate action-conditioned video predictions.

Figure: Figure 1: Overview of AVID world model adapter architecture.
Refer to caption: extracted/5892520/figures/video-adapter.png

AVID trains an adapter that takes the output of the pretrained model as an input. AVID learns to generate a mask, and uses this mask to combine the outputs of the pretrained model with those of the adapter. The final combined output is used to compute the standard denoising loss, and the adapter parameters are trained to optimize this loss.
We train the AVID adapter using samples $(\mathbf{x},\mathbf{a})$ from the action-labelled dataset.
For the pretrained image-to-video model $\epsilon_{\textnormal{pre}}$ we assume that we do not have to access to it’s parameters, $\theta_{\textnormal{pre}}$, but we can run inference using the model to obtain it’s noise predictions. To do this, we input a noisy video, $\mathbf{x}_{i}\in\mathbb{R}^{T\times h\times w\times c}$, initial image, $x^{0}\in\mathbb{R}^{h\times w\times c}$, and diffusion step, $i$, to the pretrained image-to-video model to obtain its noise prediction, $\epsilon_{\textnormal{pre}}(\mathbf{x}_{i},i,x_{0})\in\mathbb{R}^{T\times h
\times w\times c}$. The parameters of the pretrained model $\epsilon_{\textnormal{pre}}$ are not modified.

The adapter that we train is a 3D UNet (Ho et al., ; Ronneberger et al., ) consisting of a sequence of spatio-temporal blocks with residual connections. Each spatio-temporal block consists of a 3D convolution, spatial attention, and temporal attention (Ho et al., ). The UNet takes as input a tensor of shape $\mathbb{R}^{T\times h\times w\times 3c}$. The input consists of the noisy video, $\mathbf{x}_{i}$, the output of the pretrained model, $\epsilon_{\textnormal{pre}}(\mathbf{x}_{i},i,x_{0})$, and the initial image $x^{0}$ repeated $T$ times across the time dimension. These three inputs are concatenated channel-wise to create the input tensor.

The adapter is also conditioned on the diffusion step $i$ and the sequence of actions $\mathbf{a}$. For noisy video $\mathbf{x}_{i}$ we embed the diffusion timestep according to a learned embedding table to get the diffusion step embedding $e_{i}$. For each timestep $\tau$ of the noisy video $\mathbf{x}_{i}$ we embed the corresponding action $a^{\tau}$ to compute the action embedding $e_{a}^{\tau}$ using an embedding table for discrete actions or a linear layer for continuous actions. In each block, these two embeddings are concatenated and processed by an MLP to compute the scale and shift parameters, $\gamma^{\tau}$ and $\beta^{\tau}$, for the $\tau^{th}$ frame. These parameters scale and shift the feature maps of the $\tau^{th}$ frame after each 3D convolution (Perez et al., ).

To inject the correct action-conditioning into the predictions made by the pretrained model, the adapter needs to erase incorrect motions predicted by the pretrained model and add the correct motion. To facilitate this, the adapter outputs a tensor of shape $\mathbb{R}^{T\times h\times w\times(c+1)}$ consisting of a mask, $m\in\mathbb{R}^{T\times h\times w\times 1}$ which is bounded between 0 and 1 by a sigmoid layer, and the noise prediction from the adapter $\epsilon_{\textnormal{adapt}}$.
The mask is then used to combine the noise predictions from the pretrained model and the adapter according to:

$$ $\epsilon_{\textnormal{final}}(\mathbf{x}_{i},\mathbf{a},i,x_{0})=\epsilon_{ \textnormal{pre}}\odot m+\epsilon_{\textnormal{adapt}}\odot(1-m),$ (5) $$

where $\odot$ denotes the Hadamard product, and the mask is broadcast across all $c$ channels.
The parameters of the adapter model, $\theta_{\textnormal{adapt}}$, are optimized to minimise the standard unweighted denoising loss using $\epsilon_{\textnormal{final}}$:

$$ $\mathcal{L}(\theta_{\textnormal{adapt}})=\mathbb{E}_{(\mathbf{x},\mathbf{a}) \sim\mathcal{D},{\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})},i\sim \mathcal{U}(\{1,2,\dots,N\})}\|\epsilon_{\textnormal{final}}(\mathbf{x}_{i}, \mathbf{a},i,x_{0})-\bm{\epsilon}\|^{2}.$ (6) $$

In the case where $\epsilon_{\textnormal{pre}}$ is a latent diffusion model, we assume that we can also run inference on the corresponding encoder and decoder.
For each training example, we first encode the video: $\mathbf{z}=\mathtt{enc}(\mathbf{x})$ where $\mathbf{z}\in\mathbb{R}^{T\times h^{\prime}\times w^{\prime}\times c^{\prime}}$, and add noise to this latent representation of the video to produce noisy latent $\mathbf{z}_{i}$.
The rest of the pipeline proceeds in the same manner, except that the initial image $x^{0}$ is replaced by the latent corresponding to the first frame, $z^{0}$, and the noisy video $\mathbf{x}_{i}$ is replaced by the noisy latent $\mathbf{z}_{i}$. The adapter loss in Equation 6 predicts the noise added to the latent sample, and we decode the sampled latent to generate the final video.

## 4 Experiments

We evaluate AVID in two different domains with different pretrained base models. Details about the datasets are provided in Appendix B.1 and details about the pretrained models are in Appendix B.2. As the focus of our work is on training lightweight adapters with limited compute, we compare adapter models with limited parameters under a computational budget.
Code to reproduce our experiments is available at [https://github.com/microsoft/causica/tree/main/research_experiments/avid](https://github.com/microsoft/causica/tree/main/research_experiments/avid).

#### Procgen

The pretrained model is a pixel-space image-to-video diffusion model (Ho et al., ) trained on videos sampled from 15 out of the 16 procedurally generated games of Procgen (Cobbe et al., ), excluding the 16th game Coinrun.
The adaptation approaches are trained using an action-labelled dataset sampled from Coinrun.
At each timestep, the discrete action is one of 15 possible keypad inputs, and the models are trained on sequences of 10 frames.
Each adaptation approach is limited to 3 days of training on a single A100 GPU.
We test three dataset sizes, Coinrun100k/500k/2.5M, evaluate the models on a held-out test set of initial frames and actions.

#### RT1 + DynamiCrafter

In this benchmark, the pretrained model is DynamiCrafter (Xing et al., ) which is currently one of the best performing image-to-video models in the VBench image-to-video leaderboard (Huang et al., ). DynamiCrafter is a latent image-to-video diffusion model that uses the autoencoder from Stable Diffusion (Rombach et al., ) and a 1.4B parameter 3D UNet trained on web-scale video data. As DynamiCrafter is a latent diffusion model, we assume that we can run inference on the encoder and decoder as described in Section 3.3.
For the action-labelled dataset we use the RT1 dataset (Brohan et al., ) which consists of real-world robotics videos. The action at each step is a 7 dimensional continuous vector corresponding to the movement and rotation of the end effector and opening or closing of the gripper. The models are trained on trajectories of 16 frames. Each adaptation approach is limited to 7 days of training on $4\times$ A100 GPUs, and is evaluated using a held-out test set of ground truth trajectories.

### 4.1 Baselines

We compare AVID with several baselines, both with and without access to the parameters of the pretrained model. Further details, including hyperparameter tuning, are in Appendix B.5.

Full access to pretrained model parameters:

- •
Action-Conditioned Finetuning – tunes all of the parameters of the pretrained model on the action-conditioned dataset. To add action-conditioning, we first compute an action embedding according to Section 3.3. For Procgen and RT1, we concatenate and add the action embeddings with the time step embeddings respectively.
- •
Language-Conditioned Finetuning – finetunes the pretrained model using a language description of each video. The language is embedded using CLIP (Radford et al., ) and conditioned upon using cross-attention following Xing et al. ().
- •
ControlNet (Zhang et al., ) – freezes the parameters of the pretrained model and makes a trainable copy of its UNet encoder. The trainable part of the model is conditioned on a new signal, and connected to the decoder in the original model via convolutions which are initialised to zero. In our work, we use ControlNet to add action-conditioning.
- •
ControlNet Small – ControlNet still has a large number of trainable parameters. ControlNet Small freezes the pretrained model in the same way as ControlNet, but reduces the number of trainable parameters to a similar amount to AVID by decreasing the number of channels in the layers of the UNet encoder. A learned projection at each layer projects the activations of the smaller model to match the number of channels in the pretrained base model.

No access to pretrained model parameters   For a fair comparison to AVID, we evaluate the following approaches which either do not leverage the pretrained model or, like AVID, assume access to only noise predictions from the pretrained model:

- •
Action-Conditioned Diffusion – We train an action-conditioned diffusion model $\bm{\epsilon}_{\theta}(\mathbf{x}_{i},\mathbf{a},i,x_{0})$ from scratch, with the same number of parameters and same UNet architecture as AVID.
- •
Classifier Guidance (Dhariwal & Nichol, ) – We train a classifier $f_{\phi}(\mathbf{a}|\mathbf{x}_{i})$ on noisy images $\mathbf{x}_{i}$ to predict actions. With weighting $w$, the classifier is used to steer the diffusion sampling process towards samples consistent with the actions. The final noise prediction becomes:
$\displaystyle\bar{\bm{\epsilon}}_{\textnormal{final}}(\mathbf{x}_{i},\mathbf{a
},i,x_{0})=\bm{\epsilon}_{\textnormal{pre}}(\mathbf{x}_{i},i,x_{0})-\sqrt{1-
\bar{\alpha}_{t}}\;w\nabla_{\mathbf{x}_{i}}\log f_{\phi}(\mathbf{a}|\mathbf{x}
_{i}).$
- •
Product of Experts – Inspired by Yang et al. (), we add action conditioning to a pretrained video diffusion model by adding its score to an action-conditioned model.
We train a small video diffusion model on action conditioned data $\bm{\epsilon}_{\textnormal{adapt}}(\mathbf{x}_{i},\mathbf{a},i,x_{0})$, and compute the final denoising prediction as a weighted sum of predictions from the pretrained and action-conditioned models:
$\displaystyle\bar{\bm{\epsilon}}_{\textnormal{final}}(\mathbf{x}_{i},\mathbf{a
},i,x_{0})=\lambda_{p}\bm{\epsilon}_{\textnormal{adapt}}(\mathbf{x}_{i},
\mathbf{a},i,x_{0})+(1-\lambda_{p})\bm{\epsilon}_{\textnormal{pre}}(\mathbf{x}
_{i},i,x_{0}),$
where $\lambda_{p}$ controls the strength of the pretrained prior during video generation.
- •
Action Classifier-Free Guidance (Ho & Salimans, ) – We train a small action-conditioned diffusion model $\bm{\epsilon}_{\textnormal{adapt}}(\mathbf{x}_{i},\mathbf{a},i,x_{0})$ while randomly removing the action conditioning ($\mathbf{a}=\varnothing$) during training with probability $p=0.2$.
We then compute the final denoising prediction as:
$\displaystyle\bar{\bm{\epsilon}}_{\textnormal{final}}(\mathbf{x}_{i},\mathbf{a
},i,x_{0})=\bm{\epsilon}_{\textnormal{pre}}(\mathbf{x}_{i},i,x_{0})+\lambda_{a
}\left(\bm{\epsilon}_{\textnormal{adapt}}(\mathbf{x}_{i},\mathbf{a},i,x_{0})-
\bm{\epsilon}_{\textnormal{adapt}}(\mathbf{x}_{i},\mathbf{a}=\varnothing,i,x_{
0})\right).$
Note that unlike standard classifier-free guidance this approach combines predictions from two separate models, $\bm{\epsilon}_{\textnormal{pre}}$ and $\bm{\epsilon}_{\textnormal{adapt}}$, which are trained on different data.

### 4.2 Evaluation

For evaluation, we use a set of 1024 held-out test videos and their corresponding action sequences. We generate 1024 synthetic videos by conditioning the models on each initial frame and action sequence, and compare the generated videos against the ground truth using the following metrics:

- •
Action Error Ratio – To assess the consistency between the videos and the action sequences, we train a model to predict actions from real videos. The Action Error Ratio is the ratio of errors obtained by using this model on generated videos, divided by the error obtained on real videos. More details are in Appendix B.3.
- •
FVD (Cobbe et al., ) – measures the similarity between feature distributions of generated and real video sequences in the I3D network (Carreira & Zisserman, ), accounting for both spatial quality and temporal coherence.
- •
FID (Heusel et al., ) – compares the feature distributions of all real and generated images in the Inception-v3 network (Szegedy et al., ), measuring the quality and variety of the images.
- •
SSIM (Wang et al., ) – compares each ground truth and generated image, focusing on luminance, contrast, and structural information.
- •
PSNR (Hore & Ziou, ) – is computed using the mean squared error between each ground truth and generated video, and thus computes the distance between the videos in pixel-space.
- •
LPIPS (Zhang et al., ) – compares the similarity between the features representations of each ground truth and predicted image using the VGG network (Simonyan & Zisserman, ).

We bold results within 2% of the best performance in each model size category. We also compute normalized metrics by normalizing each value between 0 and 1. Details are in Appendix B.4.

Figure: Figure 2: Top three rows: Examples of videos generated for RT1 (extended in Figure 7, Appendix A.5). Bottom row: Mask generated in downsampled latent space by AVID. White indicates the mask is set to 1 and black indicates the mask is set to 0.
Refer to caption: extracted/5892520/images/rt1/real_videos/3-70-real_frames/frame_0.png

Figure: Figure 3: Top three rows: Examples of videos generated for Coinrun 500k (extended in Figure 8, Appendix A.5). Bottom row: Mask generated by AVID where white indicates the mask is set to 1 and black indicates the mask is set to 0.
Refer to caption: extracted/5892520/images/coinrun/real_videos/3-424_frames/frame_0.png

### 4.3 Results

Qualitative Results
Examples of generated videos are in Figures 2 and 3, with further examples in Appendix A.5. We observe for both RT1 and Coinrun that the action-conditioned diffusion model trained from scratch fails to maintain consistency with the original conditioning image: objects in the initial image disappear in later frames in the video. In contrast, the videos generated by AVID are consistent throughout. In both AVID and action-conditioned diffusion, we observe that the motion in the generated videos is accurate compared to the ground truth motion. The pretrained base models do not generate accurate videos for either domain (Appendix A.5). The masks generated by AVID are visualised in Figures 2 and 3. The lighter parts of the mask show that the pretrained models predictions are predominantly used by AVID for maintaining background textures. The mask is reduced nearer to 0 around the robot arm in RT1 and the character in Coinrun, showing that the adapter outputs are predominantly used for action-relevant parts of the video. In Appendix A.6 we show that AVID can be used to generate predictions for different actions given the same initial frame.

Quantitative Results   To evaluate overall performance, in Figures 4(a) and 4(b) we plot the normalized performance averaged across all evaluation metrics.
AVID obtains similar or slightly better overall performance compared to ControlNet/ControlNet Small on both Coinrun500k and RT1. Note that unlike ControlNet variants, AVID does not require access to the weights of the original pretrained model.
For RT1, we observe that training an action-conditioned diffusion model performs slightly worse than AVID at the largest model size. For Coinrun500k, AVID significantly outperforms action-conditioned diffusion at the larger model size. As the number of trainable parameters is reduced, the performance of action-conditioned diffusion declines much more quickly than AVID in both domains, and therefore AVID is considerably stronger at smaller model sizes.
In Figure 4(c) we plot the overall performance of these three approaches against the Coinrun dataset size. A similar trend is observed for all approaches as dataset size is modified.

Figure: (a)
Refer to caption: extracted/5892520/figures/rt1_performance.png

**Table 1: Results for Coinrun 500k dataset. Methods trained for 3 days on a single A100. Shading indicates method requires access to the model parameters. Brackets indicate trainable parameters.**
|  | Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Medium | AVID (Ours) (22M) | 1.257 | 28.5 | 8.07 | 0.666 | 0.192 | 22.4 |
| Action-Conditioned Diffusion (22M) | 1.500 | 41.6 | 8.95 | 0.562 | 0.284 | 18.7 |  |
| Product of Experts (22M) | 1.601 | 101.8 | 9.96 | 0.584 | 0.281 | 19.1 |  |
| Action Classifier-Free Guidance (22M) | 1.966 | 191.1 | 13.09 | 0.454 | 0.3495 | 17.5 |  |
| ControlNet-Small (22M) | 1.465 | 34.7 | 8.49 | 0.652 | 0.205 | 21.9 |  |
| Large | AVID (Ours) (71M) | 1.154 | 23.1 | 7.33 | 0.713 | 0.161 | 23.8 |
| Action-Conditioned Diffusion (71M) | 1.216 | 31.3 | 7.78 | 0.648 | 0.220 | 20.9 |  |
| Product of Experts (71M) | 1.247 | 84.8 | 9.07 | 0.651 | 0.229 | 21.0 |  |
| Action Classifier-Free Guidance (71M) | 2.079 | 188.7 | 13.05 | 0.463 | 0.341 | 17.7 |  |
| ControlNet (71M) | 1.418 | 18.5 | 7.16 | 0.758 | 0.128 | 25.5 |  |
| Full | Pretrained Base Model (97M) | 3.855 | 204.0 | 13.03 | 0.451 | 0.352 | 17.3 |
| Classifier Guidance | 4.070 | 192.9 | 12.68 | 0.450 | 0.351 | 17.4 |  |
| Action-Conditioned Finetuning (97M) | 1.227 | 14.1 | 6.52 | 0.761 | 0.120 | 25.8 |  |

**Table 2: Quantitative results for RT1 dataset. Methods trained for 7 days on $4\times$ A100. Shading indicates method requires access to the model parameters. Brackets indicate trainable parameters.**
|  | Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Small | AVID (Ours) (11M) | 2.572 | 54.0 | 4.344 | 0.811 | 0.166 | 24.5 |
| Action-Conditioned Diffusion (11M) | 2.238 | 80.4 | 5.329 | 0.767 | 0.226 | 22.9 |  |
| Product of Experts (11M) | 2.859 | 89.9 | 5.276 | 0.790 | 0.201 | 23.6 |  |
| Action Classifier-Free Guidance (11M) | 3.503 | 104.8 | 4.858 | 0.737 | 0.217 | 22.2 |  |
| ControlNet-Small (10M) | 2.640 | 38.6 | 3.730 | 0.811 | 0.169 | 23.4 |  |
| Medium | AVID (Ours) (34M) | 1.907 | 38.7 | 3.645 | 0.831 | 0.150 | 25.0 |
| Action-Conditioned Diffusion (34M) | 1.737 | 36.7 | 4.038 | 0.813 | 0.172 | 24.3 |  |
| Product of Experts (34M) | 2.421 | 61.7 | 4.533 | 0.813 | 0.175 | 24.4 |  |
| Action Classifier-Free Guidance (34M) | 3.129 | 71.4 | 4.690 | 0.748 | 0.205 | 22.6 |  |
| ControlNet-Small (38M) | 2.227 | 35.0 | 3.539 | 0.821 | 0.162 | 24.0 |  |
| Large | AVID (Ours) (145M) | 1.609 | 39.3 | 3.436 | 0.842 | 0.142 | 25.3 |
| Action-Conditioned Diffusion (145M) | 1.384 | 24.9 | 3.504 | 0.817 | 0.153 | 24.6 |  |
| Product of Experts (145M) | 1.947 | 47.0 | 4.026 | 0.819 | 0.160 | 24.8 |  |
| Action Classifier-Free Guidance (145M) | 3.188 | 79.3 | 4.775 | 0.748 | 0.205 | 22.8 |  |
| ControlNet-Small (170M) | 1.779 | 30.0 | 3.375 | 0.832 | 0.153 | 24.4 |  |
| Full | Pretrained Base Model (1.4B) | 4.183 | 237.6 | 5.432 | 0.712 | 0.228 | 20.6 |
| Classifier Guidance | 4.182 | 213.1 | 6.005 | 0.683 | 0.250 | 19.8 |  |
| ControlNet (654M) | 1.708 | 27.1 | 3.248 | 0.836 | 0.148 | 24.5 |  |
| Action-Conditioned Finetuning (1.4B) | 1.297 | 24.2 | 2.965 | 0.852 | 0.134 | 25.6 |  |
| Language-Conditioned Finetuning (1.4B) | 3.859 | 33.7 | 3.511 | 0.812 | 0.177 | 22.1 |  |

In Figure 4(d) we plot the average mask value at each step of the diffusion process. On RT1 AVID has a higher mask value, and therefore uses the pretrained model more heavily than on Coinrun. This is likely because the backgrounds in RT1 are mostly static and DynamiCrafter is a strong model. We see that the mask values are lower at diffusion steps where the noise level is high. This indicates that the adapter model is more responsible for generating low-frequency information, such as the positions of objects, which is defined early in the reverse process. Towards the end of the reverse process, where fine details are generated (Ho et al., ), the pretrained model is relied on more.

The values for every evaluation metric are reported in Tables 1 and 2. For Coinrun500k, AVID performs the best for every evaluation metric at the smaller 22M model size.
For the larger model size of 71M, AVID performs the second best for most metrics to ControlNet, but obtains the best performance for Action Error Ratio.
In RT1, AVID is the strongest in the metrics that make framewise comparisons (SSIM/LPIPS/PSNR) across all model sizes.
In our setting, where the goal is to generate accurate videos according to the input action sequence, these metrics are more suitable than comparing the overall distribution of images and videos (i.e. FID and FVD) (Zhu et al., ). ControlNet Small generally obtains the best performance for FVD and FID, while Action-Conditioned Diffusion is consistently the best for Action Error Ratio.
Poor performance was obtained for Classifier Guidance and Action Classifier-Free Guidance across both domains.
For standard implementations of classifier(-free) guidance, the unconditional model is trained on the trained on data from the target domain. In contrast, in our setting the pretrained models are not trained on data from Coinrun or RT1 which may explain the lackluster performance of these methods.
Product of Experts also performed poorly (PoE). However, for PSNR and SSIM, PoE did slightly outperform both of the models from which it is composed.

Across both domains, finetuning the pretrained model with action conditioning is the strongest baseline. However, like the ControlNet variants, this requires access to the weights of the pretrained model which we assume we do not have access to. In comparison, finetuning with language instead of action conditioning results in poor performance in all metrics except FID and FVD, demonstrating that fine-grained action conditioning is necessary to generate accurate synthetic videos.

AVID Ablations   We evaluate the following two ablations of AVID: No Mask (NM): The output adapter does not output a mask. Instead, the outputs of the pretrained model and adapter are directly added: $\epsilon_{\textnormal{final}}=\epsilon_{\textnormal{pre}}+\epsilon_{
\textnormal{adapt}}$. No Conditioning (NC): The adapter is no longer conditioned on the output of the pretrained model, $\epsilon_{\textnormal{pre}}$. Results for the ablations are in Table 3. For NM, performance across most metrics is worse for RT1, but similar for Coinrun500k. For NC, the performance on most metrics gets significantly worse for both Coinrun500k and RT1. Thus, conditioning on the outputs of the pretrained model is the most crucial component of our approach.

**Table 3: Results for AVID ablations (extended in Table 6, Appendix A.2).**
|  | Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Coinrun<br>500k<br>Large | AVID (Ours) (71M) | 1.154 | 23.1 | 7.33 | 0.713 | 0.161 | 23.8 |
| No Mask (71M) | 1.136 | 22.2 | 7.15 | 0.721 | 0.158 | 24.0 |  |
| No Conditioning (71M) | 1.141 | 27.8 | 8.02 | 0.682 | 0.189 | 22.3 |  |
| RT1<br>Large | AVID (Ours) (145M) | 1.609 | 39.3 | 3.436 | 0.842 | 0.142 | 25.3 |
| No Mask (145M) | 1.769 | 44.1 | 3.533 | 0.836 | 0.146 | 25.3 |  |
| No Conditioning (145M) | 1.775 | 36.2 | 3.550 | 0.827 | 0.149 | 25.0 |  |

## 5 Related Work

Diffusion for Decision-Making
Many works utilise diffusion models for generating actions (Pearce et al., ; Janner et al., ). However, the focus of our work is on generating action-conditioned synthetic data for world modelling (Ha & Schmidhuber, ), which can be used downstream for planning (Hafner et al., ).
SynTHER (Lu et al., ) employs an unconditional diffusion model to generate synthetic data for reinforcement learning (RL). Other works (Alonso et al., ; Yang et al., ; Rigter et al., ; Hu et al., ; Zhang et al., ) utilise diffusion models to train action-conditioned world models. All of these works train the diffusion model from scratch. In our work, the focus is on making effective use of a pretrained video diffusion model to leverage web-scale pretraining. Most related is Pandora (Xiang et al., ) which integrates an LLM and text-to-video diffusion model to generate videos conditioned on actions described as natural language. While natural language is suitable for describing high-level actions, it is inadequate for our goal of modelling low-level actions (McCarthy et al., ).

Video Pre-Training for World Models
To address the scarcity of action-labeled real-world data, several studies have investigated using unlabeled videos to enhance the efficiency of world model training. Seo et al. () and Wu et al. () pre-train an autoregressive video prediction model, which is later fine-tuned with action-labeled data. Mendonca et al. () develop a world model from videos of humans by defining a high-level action space that is shared between both embodiments. Another approach involves learning latent actions from videos (Bruce et al., ; Schmidt & Jiang, ). Our work integrates video pre-training by utilizing a pre-trained video diffusion model, with the distinction that we do not have access to the weights of the original model.

Adding Controllability to Diffusion Models   Classifier guidance (Dhariwal & Nichol, ) adds conditioning to a pretrained diffusion model using a separate classifier. Classifier-free guidance (Ho & Salimans, ) achieves stronger empirical performance but requires the model to be trained with conditioning signals, rendering it unsuitable for post-hoc application to a pretrained model.

Recent advancements include ControlNet (Zhang et al., ) and T2I-Adapter (Mou et al., ), which introduce conditional control to pretrained diffusion models by freezing the original UNet and injecting additive signals into the pretrained network. ControlNet-XS (Zavadski et al., ) extends this concept by increasing the interactions between the pretrained and adapter networks. Li et al. () incorporates additional trainable layers into the frozen pretrained UNet. These techniques have been applied to introduce controls such as language (Xing et al., ), optical flow (Hu & Xu, ), and depth maps (Chen et al., ) into video models. However, they are not applicable in our context as they require access to the original model’s weights, which we assume are inaccessible. Textual inversion (Gal et al., ) and null text inversion (Mokady et al., ) optimize text embeddings to add controllability to text-conditioned diffusion models. However, this requires backpropagation through the pretrained model which is not feasible in our setting. Cascaded diffusion models (Ho et al., ; ), train a sequence of diffusion models, which each condition on the outputs of the previous diffusion model. However, the focus of these works is improving the spatial and temporal resolution at each step, whereas our focus is on incorporating a new conditioning signal, actions, that was absent in the pretrained model.

Compositional Generative Models  Our work is related to compositional generative models, where generative models are combined probabilistically (Liu et al., ; Nie et al., ; Du et al., ). RoboDreamer applies this idea to world modeling by decomposing language conditioning into multiple components to generate several predictions that are then combined (Zhou et al., ). Yang et al. () modifies the style of a pretrained video diffusion model by combining it’s output with a domain-specific diffusion model that is trained independently. Unlike Yang et al. (), the focus of our work is on adding action-conditioning to a pretrained model, and our experiments demonstrate that the approach proposed by Yang et al. () works poorly in our setting.

## 6 Discussion

Limitations  AVID adapters are tailored to a specific pretrained model and therefore cannot be composed with different models. Developing a method that works across different pretrained models is an exciting direction for future work.
AVID does not require access to pretrained model weights, but it does require access to intermediate predictions during denoising, including the outputs of the encoder and decoder in the case of latent diffusion. Many closed-source APIs do not provide access to these quantities, and therefore we advocate for model providers to provide API access to the outputs of the denoising model and autoencoder to facilitate more flexible use of their models.

In the RT1 domain we found that training an action-conditioned diffusion model from scratch resulted in the best Action Error Ratio, despite the videos being less visually accurate. For some downstream applications, action consistency might be the most important performance metric. If this is the case, training from scratch may be the preferred approach for some domains.

Conclusion  We introduced the novel problem of adapting pretrained video diffusion models to action-conditioned world models without requiring access to the pretrained model’s parameters. Our proposed approach, AVID, generates accurate videos with similar performance to ControlNet variants without requiring access to the pretrained model parameters. AVID obtains superior overall performance to existing baselines that do not require access to the internals of the pretrained model. Our results demonstrate that AVID benefits from the pretrained model by maintaining better consistency with the initial image across both pixel-space and latent diffusion models.

As general image-to-video diffusion models continue to advance in capability, our findings highlight the considerable potential of adapting these models to world models that are suitable for planning and decision-making. This work represents an initial step in that direction. In future research, we aim to explore the use of synthetic data generated by AVID adapters for planning tasks and integrate AVID with more powerful pretrained models.

Contribution Statement

Marc Rigter:

- •
Conceived initial concept of adapting video diffusion models to world models
- •
Co-conceived the final approach
- •
Implemented training and model code for both latent and pixel-space diffusion
- •
Implemented all dataloading, preprocessing, and evaluation metrics
- •
Implemented AVID and all baselines except those mentioned below
- •
Trained all models and ran all evaluation except for IRASim
- •
Led paper writing
- •
Contributed to discussions throughout project

Tarun Gupta:

- •
Co-conceived the final approach
- •
Implemented Action Classifier-Free Guidance in pixel-space diffusion codebase and all ControlNet variants
- •
Ran evaluation of IRASim
- •
Contributed to paper writing
- •
Contributed to discussions throughout project

Agrin Hilmkil:

- •
Reviewed dataloading code
- •
Set up development environment

Chao Ma:

- •
Contributed to paper writing
- •
Contributed to early project discussions

## Appendix A Additional Results

### A.1 Coinrun100k and Coinrun 2.5M Results

Tables 4 and 5 contain results for Coinrun datasets of different sizes (100k and 2.5M). The results in the main paper use a dataset size of 500k.

**Table 4: Quantitative results for Coinrun 100k dataset. Shaded rows indicate that the method requires access to the model parameters.**
|  | Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Med. | AVID (Ours) (22M) | 1.355 | 57.4 | 10.77 | 0.592 | 0.270 | 19.9 |
| Action-Conditioned Diffusion (22M) | 1.501 | 65.6 | 11.82 | 0.487 | 0.365 | 17.2 |  |
| ControlNet-Small (22M) | 1.559 | 62.6 | 10.39 | 0.603 | 0.263 | 20.3 |  |
| Large | AVID (Ours) (71M) | 1.222 | 56.0 | 10.47 | 0.624 | 0.253 | 20.6 |
| Action-Conditioned Diffusion (71M) | 1.224 | 60.0 | 11.16 | 0.558 | 0.312 | 18.5 |  |
| ControlNet (71M) | 1.491 | 57.6 | 9.92 | 0.697 | 0.197 | 22.9 |  |
| Full | Pretrained Base Model (97M) | 3.855 | 204.0 | 13.03 | 0.451 | 0.352 | 17.3 |
| Action-Conditioned Finetuning (97M) | 1.311 | 34.7 | 8.38 | 0.716 | 0.167 | 23.9 |  |

**Table 5: Quantitative results for Coinrun 2.5M dataset. Shaded rows indicate that the method requires access to the model parameters.**
|  | Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Med. | AVID (Ours) (22M) | 1.277 | 24.7 | 8.01 | 0.679 | 0.177 | 23.0 |
| Action-Conditioned Diffusion (22M) | 1.574 | 34.8 | 8.88 | 0.580 | 0.254 | 19.4 |  |
| ControlNet-Small (22M) | 1.467 | 23.8 | 7.80 | 0.653 | 0.194 | 22.1 |  |
| Large | AVID (Ours) (71M) | 1.158 | 14.5 | 6.44 | 0.740 | 0.131 | 25.1 |
| Action-Conditioned Diffusion (71M) | 1.203 | 17.7 | 6.63 | 0.704 | 0.158 | 23.4 |  |
| ControlNet (71M) | 1.393 | 14.8 | 6.68 | 0.760 | 0.120 | 25.7 |  |
| Full | Pretrained Base Model (97M) | 3.855 | 204.0 | 13.03 | 0.451 | 0.352 | 17.3 |
| Action-Conditioned Finetuning (97M) | 1.216 | 12.0 | 6.28 | 0.782 | 0.107 | 26.6 |  |

### A.2 Full Results for AVID Ablations

Table 6 contains ablation results for AVID across the full range of model sizes.

**Table 6: Full results for AVID ablations.**
|  | Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Coinrun<br>500k<br>Med. | AVID (Ours) (22M) | 1.257 | 28.5 | 8.07 | 0.666 | 0.192 | 22.4 |
| No Mask (22M) | 1.241 | 28.6 | 8.06 | 0.675 | 0.186 | 22.7 |  |
| No Conditioning (22M) | 1.276 | 33.7 | 8.50 | 0.655 | 0.208 | 21.5 |  |
| Coinrun<br>500k<br>Large | AVID (Ours) (71M) | 1.154 | 23.1 | 7.33 | 0.713 | 0.161 | 23.8 |
| No Mask (71M) | 1.136 | 22.2 | 7.15 | 0.721 | 0.158 | 24.0 |  |
| No Conditioning (71M) | 1.141 | 27.8 | 8.02 | 0.682 | 0.189 | 22.3 |  |
| RT1<br>Small | AVID (Ours) (11M) | 2.572 | 54.0 | 4.344 | 0.811 | 0.166 | 24.5 |
| No Mask (11M) | 2.752 | 63.7 | 4.566 | 0.809 | 0.168 | 23.8 |  |
| No Conditioning (11M) | 2.938 | 60.5 | 4.620 | 0.806 | 0.172 | 24.5 |  |
| RT1<br>Med. | AVID (Ours) (34M) | 1.907 | 38.7 | 3.645 | 0.831 | 0.150 | 25.0 |
| No Mask (34M) | 2.155 | 49.7 | 3.940 | 0.825 | 0.156 | 24.5 |  |
| No Conditioning (34M) | 2.349 | 48.6 | 4.018 | 0.819 | 0.160 | 25.1 |  |
| RT1<br>Large | AVID (Ours) (145M) | 1.609 | 39.3 | 3.436 | 0.842 | 0.142 | 25.3 |
| No Mask (145M) | 1.769 | 44.1 | 3.533 | 0.836 | 0.146 | 25.3 |  |
| No Conditioning (145M) | 1.775 | 36.2 | 3.550 | 0.827 | 0.149 | 25.0 |  |

### A.3 RT1 Results with Larger Compute Limit

The results in the main paper train models for RT1 using a compute limit of 7 days of 4$\times$ A100 GPUs.
To provide reference values for performance we also evaluated IRASim (Zhu et al., ) using our evaluation setup. IRASim is a 679M parameter model trained using 100 GPU-days on A800 GPUs.
We also include results for action-conditioned finetuning of DynamiCrafter using a similar amount of compute (104 GPU-days on A100 GPUs).
The results for these two models are in Table 7.
By finetuning DynamiCrafter, we obtain slightly stronger performance than IRASim.

**Table 7: Quantitative results for RT1 dataset with greater computational budget.**
| Method | Action Err. Ratio $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| IRASim | - | 21.0 | 2.759 | 0.879 | 0.1296 | 28.2 |
| Action-Conditioned Finetuning (1.4B) | 1.151 | 19.5 | 2.655 | 0.871 | 0.1160 | 27.1 |

### A.4 Examples of AVID Masking

Figures 5 and 6 illustrate examples of the mask generated by AVID which is used to mix the pretrained model and adapter outputs.

Figure: Figure 5: Examples of the mask, $m$, produced by AVID averaged throughout the diffusion process for Coinrun500k. White indicates the mask is set to 1, and black indicates the mask is set to 0.
Refer to caption: extracted/5892520/images/coinrun/avid_71M/3-424_frames/frame_2.png

Figure: Figure 6: Examples of the mask, $m$, produced by AVID averaged throughout the diffusion process for RT1. Note that the mask is in the latent space, where the images have been downsampled by a factor of 8. White indicates the mask is set to 1, and black indicates the mask is set to 0.
Refer to caption: extracted/5892520/images/rt1/avid_145M/3-28-samples_frames/frame_0.png

### A.5 Extended Qualitative Examples

Figure: Figure 7: Extended qualitative comparison of videos generated for RT1.
Refer to caption: extracted/5892520/images/rt1/real_videos/3-70-real_frames/frame_0.png

Figure: Figure 8: Extended qualitative comparison of videos generated for Coinrun 500k.
Refer to caption: extracted/5892520/images/coinrun/real_videos/3-424_frames/frame_0.png

### A.6 Example Videos with Same Initial Frame

In Figure 9 we generate videos by providing the same initial conditioning frame but different actions. The provided action is fixed for all 16 steps of the video.

Figure: Figure 9: Examples of videos generated for RT1 with same initial frame but different actions.
Refer to caption: extracted/5892520/images/rt1_fixed_action/forward/1-14-samples_frames/frame_0.png

## Appendix B Experiment Details

### B.1 Dataset Details

#### Procgen Pretrained Model Dataset

The pretrained model is trained on videos sampled from 15 out of the 16 procedurally generated games of Procgen, excluding the 16th game Coinrun. Because the games are procedurally generated, each level in each Procgen game has different visual characteristics. To generate the dataset for the pretrained model, in each of the 15 games we sample an episode of 1000 steps long from each of the first 2000 levels by executing uniformly random actions. This results in 2M frames from each game for a total dataset of 30M steps, each with a resolution of $64\times 64$. For training, we sample windows of 10 steps from the episodes. The pretrained model is trained on videos from this dataset for 12 days in a single A100.

Figure: Figure 10: Examples from Procgen pretraining dataset on the 15 out of 16 Procgen games. Note that the pretraining dataset does not include any samples from Coinrun.
Refer to caption: extracted/5892520/images/procgen_base/1-12000_frames/frame_0.png

#### Coinrun Datasets

To train the Coinrun adapters, we generate an action-labelled dataset from the Coinrun game. At each timestep, the action is one of 15 discrete actions corresponding to keypad inputs. We sample episodes of 1000 steps from each of the first 100, 500, or 2500 levels by executing uniform random actions to create the Coinrun100k, Coinrun500k, and Coinrun2.5M datasets. For training, we sample action-labelled trajectories of 10 steps.

For evaluation, we create a held-out evaluation set of ground truth trajectories of videos and actions. We use 1024 ground truth trajectories of 10 steps each, sampled by executing random action sequences in levels sampled uniformly at random between level 10000 and 11000 of Coinrun. We use the same evaluation trajectories for all methods.

#### RT1 Dataset

The RT1 dataset contains 87212 action-labelled episodes of a robot performing tasks such as picking up and placing objects, with a total of 3.78M steps.
The action at each step is a 7 dimensional continuous vector corresponding to the movement and rotation of the end effector and opening or closing of the gripper.
The videos have a resolution of $320\times 512$.
For training, we use 95% of the episodes, equating to 82851 episodes in the training dataset. We sample windows of 16 steps from these episodes for training the models. For evaluation, we use 1024 trajectories of 16 steps each sampled uniformly at random from the held-out test set of 4361 test episodes. We use the same evaluation trajectories for all methods.

### B.2 Model and Training Details

#### Procgen and Coinrun

For both the pretrained model trained on 15 of the 16 Procgen games, and the adapters trained on the Coinrun datasets, we use the 3D UNet architecture from Video Diffusion Models (Ho et al., ) trained on videos of resolution $64\times 64$. We condition each model on two initial images to allow the initial direction the agent is moving in to be inferred and allow for more accurate video generation. Detailed hyperparameters for each of the models are in Table 8.

**Table 8: Hyperparameters for models trained on Procgen and Coinrun.**
| Hyperparameter | Value | Models |
| --- | --- | --- |
| Base Channels | 100 | 97M Pretrained Model, ControlNet |
| 90 | 71M |  |
| 50 | 22M, ControlNet-Small 22M |  |
| Attention Head Dims | 64 | 97M Pretrained Model, all ControlNet variants |
| 50 | 22M, 71M |  |
| Attention Heads | 8 | 97M Pretrained Model, all ControlNet variants |
| 6 | 71M |  |
| 4 | 22M |  |
| Learning Rate | 2e-5 | Finetuning |
| 1e-4 | All other models |  |
| Training Time | 12 days | 97M Pretrained Model |
| 3 days | All other models |  |
| Training Hardware | $1\times$ A100 GPU | All |
| EMA | 0.995 |  |
| Channel Multipliers | [1, 2, 4, 8] |  |
| Sequence Length | 10 steps |  |
| Batch Size | 64 |  |
| Noise Steps | 200 |  |
| Inference Steps | 200 |  |
| Sampling Method | DDPM |  |
| Prediction Target | $\mathbf{x}_{0}$ |  |
| Noise Schedule | Sigmoid |  |

**Table 9: Hyperparameters for models trained on RT1.**
| Hyperparameter | Value |  |
| --- | --- | --- |
| Base Channels | 320 | 1.4B, ControlNet |
| 160 | ControlNet-Small 170M |  |
| 96 | 145M |  |
| 64 | 34M, ControlNet-Small 38M |  |
| 32 | 11M, ControlNet-Small 10M |  |
| Attention Head Dims | 16 | 11M, ControlNet-Small 10M |
| 64 | All other models |  |
| Channel Multipliers | [1, 2, 4, 4] | 1.4B, 145M, all ControlNet variants |
| [1, 1, 2, 3] | 34M |  |
| [1, 2, 3] | 11M |  |
| Learning Rate | 2e-5 | Finetuning |
| 1e-4 | All other models |  |
| Attention Heads | Channels / Attention Head Dims | All |
| Training Time | 7 days |  |
| Training Hardware | $4\times$ A100 GPU |  |
| EMA | 0.9995 |  |
| Sequence Length | 16 steps |  |
| Batch Size | 64 |  |
| Noise Steps | 1000 |  |
| Inference Steps | 50 |  |
| Sampling Method | DDIM |  |
| Prediction Target | $\mathbf{v}$ |  |
| Noise Schedule | Linear |  |

#### RT1

The pretrained model is DynamiCrafter (Xing et al., ), a latent video diffusion model.
DynamiCrafter is trained to generate videos at a resolution of $320\times 512$. To accommodate this, we resize and pad the images from the RT1 dataset to this resolution.
DynamiCrafter is trained to optionally accept language conditoning. We use an empty language prompt, except in the case where the model is finetuned with language conditioning (see Section 4.1).
DynamiCrafter uses the 1.4B 3D UNet architecture from (Chen et al., ) and the autoencoder from Stable Diffusion (Rombach et al., ).

As discussed in Section 3.3, we
use the same autoencoder for our AVID adapter, as well as all baselines.
For all methods on RT1, we train a 3D UNet with the same architecture as Dynamicrafter from (Chen et al., ), but with a reduced number of parameters.
The hyperparameters for each of the models can be found in Table 9.

Parameterisation We parameterise all models on Procgen and Coinrun to predict the clean video, $\mathbf{x_{0}}$, meaning that in practice the output of the pretrained model and the adapter output are both predictions of $\mathbf{x_{0}}$ rather than the noise. DynamiCrafter is trained using the $\mathbf{v}$ prediction target parameterisation (Salimans & Ho, ), so it outputs a prediction of $\mathbf{v}$. We also train all models on the RT1 dataset to predict the $\mathbf{v}$-target.

### B.3 Action Error Ratio Evaluation Metric Details

#### Coinrun

The actions in Coinrun are discrete so to evaluate the Action Error Ratio we first train a classifier to predict the actions. The video is first processed using the encoder part of the 3D Unet architecture that we use for the diffusion model. This encoding is then flattened, and processed by an MLP which outputs a softmax over the 15 possible actions at each step of the video. The classifier is trained using the cross-entropy loss and has 60M total parameters. It is trained on a large dataset of 10M steps: 1000 steps sampled from each of 10,000 levels of Coinrun.

The Action Error Ratio is then computed as the ratio between the accuracy of the action classifier on real videos divided by the accuracy of the classifier on generated videos:

$$ $\mathtt{ActionErrorRatio(discrete)}=\frac{\mathtt{action_accuracy(real_ videos)}}{\mathtt{action_accuracy(generated_videos)}}$ $$

#### RT1

The actions in RT1 are continuous so we train a regression model to predict the actions. The actions are first normalised to mean zero and unit variance. The video is processed by the encoder part of the 3D UNet architecture and then an MLP which outputs a prediction for the action at each time step. The regression model is trained using the MSE loss on the same 82851 video dataset that we use for training the adapter models, and has 85M parameters.

The Action Error Ratio is then computed as the ratio between the mean absolute error of the action predictions of the regression model on the generated videos compared to the real videos:

$$ $\mathtt{ActionErrorRatio(continuous)}=\frac{\mathtt{action_MAE(generated_ videos)}}{\mathtt{action_MAE(real_video)}}$ $$

### B.4 Normalized Evaluation Metric Details

To compute normalized evaluation metrics plotted in Figures 4(a), 4(b) and 4(c), we first normalize each evaluation metric to between 0 and 1. In RT1, 0 represents the worst performance for each metric across the Small, Medium or Large model sizes for Action-Conditioned Diffusion,
ControlNet-Small, or AVID. 1 represents the best performance for each metric across these models.
In Coinrun, 0 and 1 represent the worst and best performance for each metric across the Medium or Large model sizes for Action-Conditioned Diffusion, ControlNet/ControlNet-Small, or AVID, across all of the three datasets: Coinrun100k, Coinrun500k, and Coinrun2.5M.

The minimum and maximum values used for normalization are summarised in Tables 10 and 11.

**Table 10: Minimum and maximum values for metric normalization in RT1.**
|  | Action Err. Ratio | FVD | FID | SSIM | LPIPS | PSNR |
| --- | --- | --- | --- | --- | --- | --- |
| Minimum | 1.384 | 24.9 | 3.375 | 0.767 | 0.142 | 22.9 |
| Maximum | 2.640 | 80.4 | 5.329 | 0.842 | 0.226 | 25.3 |

**Table 11: Minimum and maximum values for metric normalization in Coinrun.**
|  | Action Err. Ratio | FVD | FID | SSIM | LPIPS | PSNR |
| --- | --- | --- | --- | --- | --- | --- |
| Minimum | 1.154 | 14.5 | 6.44 | 0.487 | 0.120 | 17.2 |
| Maximum | 1.574 | 65.6 | 11.82 | 0.760 | 0.365 | 25.7 |

If the goal is to maximize the metric, the normalized value of the metric is computed using:

$$ $\mathtt{normalized_value}=\mathtt{(value-min_value)/(max_value-min_value)}$ (7) $$

If the goal is to minimize the metric, the normalized value of the metric is computed using:

$$ $\mathtt{normalized_value}=1-\mathtt{(value-min_value)/(max_value-min_value)}$ (8) $$

Once the metrics have been normalized, we compute the mean across all 6 normalized metrics. This final value is plotted in Figures 4(a), 4(b) and 4(c).

### B.5 Baseline Details

Here we provide additional details about the baselines that are not included in the main paper.

- •
Language-Conditioned Finetuning – the language description that we use is: “Robot arm performs the task {$\mathtt{task_description}$} ” where {$\mathtt{task_description}$} is the description given in the RT1 dataset.
- •
Classifier Guidance (Dhariwal & Nichol, ) – For the RT1 domain, since the actions are continuous, we discretise each action dimension into 256 bins uniformly following Padalkar et al. () to train the classifier. We tune the weight of the guidance within $w\in\{0.003,0.01,0.03,0.1,0.3\}$.
- •
Product of Experts – we tune the weighting of the two models within $\lambda_{p}\in\{0.2,0.4,0.6\}$.
- •
Action Classifier-Free Guidance – we tune the weighting within $\lambda_{a}\in\{0.3,1.0,3.0,10.0\}$.

For the baselines that require hyperparameter tuning, we sweep over each value of the hyperparameter and choose the value that obtains the best FVD in our evaluation.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: - The paper has a nice motivation: how does one adapt existing foundation models in order to add an action conditioning to them, so as to make it more relevant and useful for embodied robotics applications
- The paper writing is clear; first the limitations of prior work are built up and then a solution is proposed
summary: The paper proposes a mechanism for adapting current image-conditioned video diffusion models to action-conditioned video diffusion models. They do this by training an additional UNet after the standard video diffusion UNet which predicts an adjustment to the noise output by the standard UNet. Because of this setup, their "adapter" does not need access to the parameters of the pretrained video diffusion model. Experiments show that this kind of noise adaptation helps for some metrics and does not for some others.
weaknesses: - The way the paper starts with the motivation near L51-52 is a bit misleading. The paper actually cannot fix the issues in L51-52 because they still assume access to the internal inference pipeline of these closed-source model, because if I understand it correctly, this method needs access to a diffusion model's noise prediction at each of the N reverse diffusion steps that happens at inference. For closed source models, this information is not available.
- The performance gain in the quantitative metrics is not substantial. The metrics where the proposed method shines are mostly photometric quantities. It is not clear if the error margin between prior work and this work just results from the standard deviation or variance of the models. I think a better reflection of the proposed approach would have come from an application to a downstream robot task (maybe manipulation) that would evaluate a robot in action. PSNR and other photometric errors with the shown gain do not say much about the performance of the method.
- The paper heavily leans on the limitations of the POE approach and that is how a learnable adapter is motivated but qualitatively there is no comparison to that approach (even though POE is slightly better than the action conditioned diffusion across some metrics and settings).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Regarding W1, can authors elaborate more on how and why this analysis motivates design choices in methodology of AVID?
2. In the ablation study of “No mask” (Table 3), is the adapter trained with $\epsilon_{\text{final}}$ given in Equation 5 or the adapter not being able to output a mask?
3. Since the mask is an important component in design choices of AVID, could author visualize what the mask looks like in different timesteps of diffusion denoising process, which corresponds to Figure 4d?
4. Since AVID performs two diffusion denoising process, does this increase inference time and thus limit the scope of downstream applications of synthetic videos generated from this approach?
5. Regarding W3, is it technically possible to apply this approach to domains other than world modeling?
soundness: 3
strengths: 1. The paper is well written and easy to follow
2. The main idea of training a lightweight adapter for action-labeled domains is reasonable. It balances finetuning efficiency and task performance.
3. Baseline comparisons are comprehensive. Authors compared to many alternative baselines to demonstrate effectiveness of their approach. Authors provide qualitative visualizations for quality of generated videos and usefulness of learned masks.
summary: This paper focuses on the problem setting of action-conditioned video generation. It proposes to adapt pre-trained video generation model to action labeled dataset without access to parameters from pre-trained models. The goal is to add action information as conditioning to pre-trained models for more accurate video predictions. Authors also analyze limitations in previous related work, production of expert, under a specific case. The proposed approach, AVID, trains an adapter in smaller size on action labeled video datasets. It takes noise predictions outputs from pretrained models and action information as input, and learns to output a mask that is used to combine outputs from pre-trained model and adapter. The adapter is trained with reconstruction loss between final output from both models and ground truth on domain-specific datasets. Authors conducted experiments on two robotics datasets, Procgen and RT1, and compared proposed approach to several baselines that have full access and do not assume access  to pretrained model parameters. Experiments results demonstrate that AVID outperforms baselines in generating more realistic videos and better quality given action information on these domains.
weaknesses: 1. The presentation in Section 3.2 is a little unclear. It is hard to connect analysis about limitations of previous work [1] to motivations of the proposed approach
2. The novelty is somewhat limited. The main difference from previous work is to have domain-specific adapter output an element-wise mask that is used to combine noise predictions from pre-trained model and adapter.
3. The experimental domains are only two datasets within action-conditioned world modeling

[1] Yang, Mengjiao, et al. "Probabilistic adaptation of text-to-video models." arXiv preprint arXiv:2306.01872 (2023).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: see weaknesses.
soundness: 2
strengths: 1. The authors propose a novel method to condition pre-trained video diffusion models on action sequences without access to the pre-trained model's parameters.
2. The authors mathematically highlight the limitations of the adaptation method proposed in "Probabilistic Adaptation of Text-to-Video Models" and  this other approach.
3. The authors demonstrate that their adaptation method has better action consistency compared to the other approach, using a new metric that they introduce. 
4. The authors also propose multiple baselines to compare against their proposed method.
summary: 1. The authors propose a novel method to condition pre-trained video diffusion models on action sequences without access to the pre-trained model's parameters.
2. The authors demonstrate that their adaptation method is superior to the method proposed in "Probabilistic Adaptation of Text-to-Video Models" and mathematically highlight the limitations of this other approach.
3. The authors use different pre-trained base models and two different video domains, games and robotics to quantitatively evaluate their proposed method against the above adaptation approach and some other proposed baselines.
weaknesses: 1. In Table 2, Action conditioned diffusion has a better Action Error Ratio compared to the proposed approach for all three (small, medium, large) variants. While the authors do note this as a limitation, this needs to be explained/investigated more. If it is better to just train an action conditioned diffusion model from scratch why should there be a need to adapt pre-trained models ?

2. Instead of using the action embedding to just scale and shift the t-th frame feature, have the authors explored using cross-attention layers directly with the action embedding sequence similar to language conditioning ? Are there any specific challenges that prohibit such an approach ?

3. It would be interesting to see results for each task type in RT-1 . Are there tasks that are much harder to model than others and what does that tell us about the approach ?

4. Some video visualisations of the generated videos (especially for robotics) would also be very useful to judge the effectiveness of the approach. Are the videos temporally consistent visually ?

5. Why is IRAsim's Action error ratio empty in Table 7 ? is it not possible to evaluate the Action Error Ratio of IRAsim ?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Most questions and suggestions are detailed in the 'Weaknesses' section.

**Limitations of Naive Adaptation of Yang et al.**

Can the authors please highlight the exact source of discrepancy between the derivation in Yang et al. to the derivation presented in this section? Do you claim that there is an error in their derivation? Alternatively, are there different assumptions in your setting where their derivation does not hold?
soundness: 3
strengths: Summarized points:
- Tackle an interesting problem in the path for scaling robot learning
- Clear and well written paper
- Good positioning in related work
- Comparison to relevant baselines
- Thorough analysis of results
- Detailed appendix
summary: This work proposes a method for leveraging diffusion models pretrained on large-scale action-free video data for training action-conditioned diffusion world models on smaller action-labeled data from domains of interest. The motivation for training these world models is to solve downstream sequential decision-making tasks. The proposed method’s main novelty is that it requires access to some intermediate calculations of the pretrained diffusion model but not to its parameters. The proposed method, AVID, trains an adapter network which is conditioned on the pretrained model’s noise prediction and optimizes a denoising loss that incorporates both noise predictions from the pretrained model and the adapter’s output using a learned mask. The author’s evaluate world model performance on a real-robot and a video game domain based on multiple perceptual metrics as well as an action-prediction-based metric. Baselines include various diffusion-based methods some of which require full access to model parameters. The proposed method either outperforms or is comparable to baselines in most of the evaluated metrics while not requiring access to pretrained model parameters.
weaknesses: Summarized points:
- Intermediate model calculations are not necessarily more likely to be accessible than the model parameters
- Limitation analysis of previous work (Section 3.2) does not clearly motivate the author’s specific choice of solution
- Main motivation is sequential decision-making but evaluation metrics do not assess the world models’ efficacy in solving such tasks
- It is not clear from the experimental results that training from scratch is not preferable to the proposed method for downstream sequential decision-making

**Evaluation - Metrics**

The main motivation of your method is to accommodate sequential decision-making but evaluation metrics do not assess the world models’ efficacy in policy learning or planning.
All metrics excluding ‘Action Error Ratio’ are perceptual metrics that may be dominated by aspects of the videos that are not important for control. For this reason, I believe the most interesting and relevant metric out of the ones you display in your evaluation is the ‘Action Error Ratio’. Your evaluation could benefit from including additional metrics that are a better proxy for the world model’s usefulness in sequential decision-making. In the Procgen dataset for example, you may want to measure the ability to predict the reward from the generated frames as well as the actions.

I understand that evaluating the world models by actually using them to solve a sequential decision-making task may not be straightforward. Doing this for the RT1 dataset would be hard for multiple reasons, but it may be more feasible for the Procgen environments. One possible evaluation pipeline is training a separate model to predict the reward from a given frame and then use the cross-entropy method (CEM) or a similar sampling-based planning algorithm with model predictive control (MPC) on top of the world model to maximize the sum of rewards in the prediction horizon. Any decision-making algorithm you choose doesn’t have to be SOTA to demonstrate the point that a given world model is better than the other for this purpose.

What is the accuracy of the action predictor on each dataset? I believe this is important in order to validate the use of the ‘Action Error Ratio’ metric and that this information should at least be in the appendix.

**Evaluation - Baselines**

Why do you tune baseline hyperparameters based on FVD and not based on e.g. normalized evaluation metrics? I find this choice puzzling since you explicitly write in the results section that this metric is less suitable than others in the setting of action-conditioned video generation.

How do you choose which baselines out of the 8 you suggested appear in the result tables?

Can the authors please explain what is the purpose of the ‘Full’ row in the result tables?

**Evaluation - Results**

It is not clear from the experimental results that training from scratch is not preferable to the proposed method for downstream sequential decision-making, a point that is also suggested in the limitations section and is mostly based on the ‘Action Error Ratio’ metric. This is not to say that it clearly is not beneficial. I suggest adding a discussion about the differences in performance in the two domains which would incorporate further insights as to when and why training an adapter is preferable to training from scratch.

**Evaluation - Ablation Study**

*Mask ablation*: It is not clear from your results that the learned mask has performance benefits and can’t be ‘absorbed’ into the adapter noise prediction, especially since it hurts performance on one dataset and doesn’t in the other.
How do you explain the difference in the effects of the mask on performance in each dataset? I think a discussion with respect to factors like the relationship between pre-training and fine-tuning data in each dataset and with respect to the results presented in Figure 4d could shed more light on this matter.

*Conditioning ablation*: I think the method and/or ablation section can benefit from an explanation or intuition behind why conditioning on the pretrained model’s output is beneficial, given that the pretrained output is already accounted for in the objective.

*Request for ablation*: As I see it, the fundamental difference between the proposed method and the PoE baseline is that the parameters of the adapter network are trained on the denoising loss containing noise predictions from both the pretrained network and the adapter network. Therefore an interesting ablation would be combining both the NM and NC ablations.

Meta-review: This is a well-written paper about training an action-conditioning adapter for a pre-trained, frozen video diffusion model. 

The reviewer reception of this paper was mixed: while the writing and presentation are clear and of high quality, there were concerns about the generality of the method and the extent of the experimental evaluation to fully justify the claims made in the paper. For example, it would have been valuable to demonstrate the proposed adapter not just for action conditioning, but for other forms of controlling video diffusion (or even image diffusion) models.

The AC agrees with these concerns and recommends refining the evaluation of the method should the authors wish to submit a revised version of the paper to a future venue.

Decision: Reject