VideoAgent: Self-Improving Video Generation

## Abstract

Abstract Video generation has been used to generate visual plans for controlling robotic systems. Given an image observation and a language instruction, previous work has generated video plans which are then converted to robot controls to be executed. However, a major bottleneck in leveraging video generation for control lies in the quality of the generated videos, which often suffer from hallucinatory content and unrealistic physics, resulting in low task success when control actions are extracted from the generated videos. While scaling up dataset and model size provides a partial solution, integrating external feedback is both natural and essential for grounding video generation in the real world. With this observation, we propose VideoAgent for self-improving generated video plans based on external feedback. Instead of directly executing the generated video plan, VideoAgent first refines the generated video plans using a novel procedure which we call self-conditioning consistency , utilizing feedback from a pretrained vision-language model (VLM). As the refined video plan is being executed, VideoAgent collects additional data from the environment to further improve video plan generation. Experiments in simulated robotic manipulation from MetaWorld and iTHOR show that VideoAgent drastically reduces hallucination, thereby boosting success rate of downstream manipulation tasks. We further illustrate that VideoAgent can effectively refine real-robot videos, providing an early indicator that robotics can be an effective tool in grounding video generation in the physical world. Video demos can be found at https://video-as-agent.github.io . 1 1 1 Code available at https://github.com/Video-as-Agent/VideoAgent

## 1 Introduction

Large text-to-video models pretrained on internet-scale data have broad applications such as generating creative video content (Ho et al., ; Hong et al., ; Singer et al., ) and creating novel games (Bruce et al., ), animations (Wang et al., ), and movies (Zhu et al., ). Furthermore, recent work show that video generation can serve as simulators of the real-world (Yang et al., ; Brooks et al., ), as well as policies with unified observation and action space (Du et al., ; Ko et al., ; Du et al., ). These recent applications of text-to-video generation models hold the great promise of internet-scale knowledge transfer (e.g., from generating human videos to generating robot videos), as well as paving the way to generalist agent (e.g., a single policy that can control multiple robots with different morphologies in different environments to perform diverse tasks).

Nevertheless, text-to-video models have only had limited success in downstream applications in reality. For instance, in video generation as policy (Du et al., ; Ko et al., ), when an observation image and a language instruction are given to a video generation model, generated videos often hallucinate (e.g., objects randomly appear or disappear) or violate physical laws (e.g., a robot hand going through an object) (Yang et al., ; Brooks et al., ). Such hallucinations and unrealistic physics have led to low task success rate when generated videos are converted to control actions through inverse dynamics models, goal conditioned policies, or other action extraction mechanisms (Wen et al., ; Yang et al., ; Ajay et al., ).

While scaling up dataset and model size can be effective in reducing hallucination in large language models (LLMs) (Hoffmann et al., ), scaling is more difficult in video generation models. This is partially because language labels for videos are labor intensive to curate. Moreover, video generation has not converged to an architecture that is more favourable to scaling (Yang et al., ). Scaling aside, being able to incorporate external feedback to improve generation is one of the other most important breakthrough in LLMs (Ouyang et al., ). It is therefore natural to wonder what kind of feedback is available for text-to-video models, and how we can incorporate these feedback to further improve the quality of the generated videos.

To answer this question, we explore two types of feedback that are natural to acquire for video generation models, namely AI feedback from a vision-language model (VLM) and real-world execution feedback when generated videos are converted to motor controls. To utilize these feedback for self-improvement, we propose VideoAgent. Different from video generation as policy, which directly turns a generated video into control actions (Du et al., ; Ko et al., ), VideoAgent is trained to refine a generated video plan iteratively using feedback from a pretrained VLM. During inference, VideoAgent queries the VLM to select the best refined video plan, followed by execution of the plan in the environment. During online execution, VideoAgent observes whether the task was successfully completed
and further improves the video generation model based on the execution feedback from the environment and additional data collected from the environment. The improvement to the generated video plan comes in two folds: First, we propose *self-conditioning consistency* for video diffusion model inspired by consistency models (Song et al., ; Heek et al., ), which enables low-quality samples from a video diffusion model to be further refined into high-quality samples. Second, when online access to the environment is available, VideoAgent executes the current video policy and collect additional successful trajectories to further finetune the video generation model on the successful trajectories. A visual illustration of VideoAgent is shown in Figure 1.

We first evaluate the performance of VideoAgent in two simulated robotics manipulation environments, Meta-World (Yu et al., ) and iTHOR (Kolve et al., ), and show that VideoAgent improves task success across all environments and tasks evaluated. VideoAgent can even improve the success rate of difficult tasks by as much as 4X. Next, we provide a thorough study on the effect of different components in VideoAgent, including different ways to prompt the VLM and different types of feedback from the VLM, providing a recipe for utilizing VLM feedback for video generation. Lastly, we illustrate that VideoAgent can iteratively improve real-robot videos, providing early signal that robotics can be an important mean to ground video generation models in the real world.

Figure: Figure 1: The VideoAgent Framework. VideoAgent first generates a video plan conditioned on an image observation and task description similar to Du et al. (), and undergoes (1) iterative video refinement using feedback from a vision language model (VLM), (2) using the VLM to select the best refined video plan to convert to control actions through optical flow, and (3) executing the control actions in an environment and improving video generation using real-world feedback and additional data collected online.
Refer to caption: extracted/5927082/images/framework.png

## 2 Background

In this section, we provide the background on video generation as policy in a decision making process (Du et al., ). We also introduce consistent diffusion models (Song et al., ; Heek et al., ; Daras et al., ), which VideoAgent builds upon for self-refinement.

### 2.1 Video as policy in sequential decision making

We consider a predictive decision process similar to Du et al. (): $\mathcal{P}:=\langle\mathcal{X},\mathcal{G},\mathcal{A},H,\mathcal{E},\mathcal
{R}\rangle$, where $\mathcal{X}$ denotes an image-based observation space, $\mathcal{G}$ denotes textual task description space, $\mathcal{A}$ denotes a low-level motor control action space, and $H\in\mathbb{R}$ denotes the horizon length. We denote $\pi(\cdot|x_{0},g):\mathcal{X}\times\mathcal{G}\mapsto\Delta(\mathcal{X}^{H})$(^2^22We use $\Delta(\cdot)$ to denote a probability simplex function) as the language conditioned video generation policy, which models the probability distribution over $H$-step image sequences $\mathbf{x}=[x_{0},...,x_{H}]$ determined by the first frame $x_{0}$ and the task description $g$. Intuitively, $\mathbf{x}\sim\pi(\cdot|x_{0},g)$ correspond to possible visual paths for completing a task $g$. Given a sampled video plan $\mathbf{x}$, one can use a learned mapping $\rho(\cdot|\mathbf{x}):\mathcal{X}^{H}\mapsto\Delta\mathcal{(}\mathcal{A}^{H})$ to extract motor controls from generated videos through a goal-conditioned policy (Du et al., ), diffusion policy (Black et al., ), or dense correspondence (Ko et al., ). Once a sequence of motor controls $\mathbf{a}\in\mathcal{A}^{H}$ are extracted from the video, they are sequentially executed in the environment $\mathcal{E}$, after which a final reward $\mathcal{R}:\mathcal{A}^{H}\mapsto\{0,1\}$ is emitted representing whether the task was successfully completed. For simplicity, we only consider finite horizon, episodic tasks. Given a previously collected dataset of videos labeled with task descriptions $\mathcal{D}=\{(\mathbf{x},g)\}$, one can leverage behavioral cloning (BC) (Pomerleau, ) to learn $\pi$ by minimizing

$$ $\mathcal{L}_{\text{BC}}(\pi)=\mathbb{E}_{(\mathbf{x},g)\sim\mathcal{D}}[-\log \pi(\mathbf{x}|x_{0},g)].$ (1) $$

Equation 1 can be viewed as maximizing the likelihood of the videos in $\mathcal{D}$ conditioned on the initial frame and task description.

### 2.2 Consistency Models

Diffusion models (Ho et al., ; Song et al., ) have emerged as an important technique for data distribution modeling.
During training, the model learns to map noisy data (at various noise levels) back to clean data in a single step. Concretely, let $x^{(0)}$ denote a clean image and $x^{(t)}$ denote the noisy image at noise level $t$, where $t\in[0,T]$, the training objective for a diffusion model $f_{\theta}(x^{(t)},t)$ can be written as

$$ $\mathcal{L}_{\text{diffusion}}(\theta)=\mathbb{E}_{x^{(0)},\epsilon,t}\left[\| f_{\theta}(x^{(t)},t)-x^{(0)})\|^{2}\right],$ (2) $$

where $\epsilon\in\mathcal{N}(0,I)$ is the added noise, and $x^{(t)}=\sqrt{\alpha_{t}}x^{(0)}+\sqrt{1-\alpha_{t}}\epsilon$ where $\alpha_{t}$ are time-dependent noise levels. Although diffusion models have achieved high-quality image/video generation, they require hundreds or thousands of denoising steps during inference, which induces tremendous computational cost.
To overcome the slow sampling speed of diffusion models, *consistency models* (Song et al., ; Song & Dhariwal, ) were initially proposed
by enforcing a consistency loss across different noise levels, i.e.,

$$ $\mathcal{L}_{\text{consistency}}(\theta)=\mathbb{E}_{x^{(0)},\epsilon,t_{1},t_ {2}}\left[\|f_{\theta}(x^{(t_{1})},t_{1})-\texttt{stopgrad}\big{(}{f_{\theta}( x^{(t_{2})},t_{2})}\big{)}\|^{2}\right],$ (3) $$

which encourages the output of the single-step map between different noise levels to be similar. In fact, both the diffusion loss in Equation 2 and the consistency loss in Equation 3 can be understood as exploiting the structure of the denoising procedure which corresponds to an ordinary differential equation (ODE). Specifically, as introduced in (Song et al., ; ), the backward denoising procedure of a diffusion model can be characterized by an ODE, i.e.,

$$ $\frac{\mathrm{d}x^{(t)}}{\mathrm{d}t}=-t\cdot s(x^{(t)},t),$ (4) $$

with $s(x^{(t)},t)$ is some score function. During the entire path along $t\in(\epsilon,\infty]$, following this ODE should always maps $x_{t}$ to $x_{0}$. If we parametrize the model $f(x^{(t)},t)$ as the simulation following the ODE governed by $s(x^{(t)},t)$, we obtain the diffusion loss (2). Meanwhile, for all $t,t^{\prime}\in(\epsilon,\infty]$, we have $f(x^{(t)},t)=f(x^{(t^{\prime})},t^{\prime})$ along the simulation path, which induces the consistency loss (3).
Therefore, we can combine the diffusion loss and the consistency loss together for model training, i.e.,

$$ $\mathcal{L}(\theta)=\mathcal{L}_{\text{diffusion}}(\theta)+\lambda\cdot \mathcal{L}_{\text{consistency}}(\theta),$ (5) $$

where $\lambda$ denotes consistency regularization hyperparameter across different noise levels.

## 3 Video Generation as Agent

In this section, we introduce VideoAgent to improve video plan generation. In Section 3.1, we establish a new notion of consistency for video diffusion models, which we called *self-conditioning consistency*. In Section 3.2, we discuss how the video diffusion model trained with self-conditioning consistency can be used to *refine* generated video plans during inference.
In Section 3.3, we discuss how VideoAgent further closes the self-improvement loop by collecting additional online data to further train the video generation and refinement model.

### 3.1 Video refinement through self-conditioning consistency

We first consider first-frame-and-language conditioned video generation following (Du et al., ; Ko et al., ), which finds a sequence of image frames for completing the task described by the language starting from the initial image.
When a sample is drawn from a video generation model, it is commonly found that a part of the generated video (e.g., the beginning) is realistic while another part of the video (e.g., the end) hallucinates (Yang et al., ). In other words, while a generated video plan may not fully complete the task specified, it provides meaningful information that can be further improved to achieve the correct plan. To leverage such partial progress, we consider a video consistency model to condition on previous self-generated samples to diffuse for the ground truth video, so that the model can learn to preserve the realistic part of the video while refining the hallucinatory part.
Specifically, let $\mathbf{x}^{(0)}$ be a ground truth video, and $\hat{\mathbf{x}}$ be a generated video sample from the diffusion model. We define a *self-conditioning consistency* model as $\hat{f}_{\theta}(\hat{\mathbf{x}},\mathbf{x}^{(t)},t)$, which takes a generated video $\hat{\mathbf{x}}$ and a noisy version of the ground truth video $\mathbf{x}^{(t)}$ as input and predicts the clean video.

Figure: Figure 2: An illustration of Self-Conditioning Consistency model. Top row: denoising steps. Bottom row: refinement iterations. $\hat{\mathbf{x}}_{(i+1)}$ denotes the generated video plan at refinement iteration $(i+1)$. $\Delta$ denotes a fixed number of denoising steps. We condition the refinement on the generated video from the previous iteration $\hat{\mathbf{x}}_{(i)}$.
Refer to caption: extracted/5927082/images/self_conditioning_consistency-3.png

We make the observation that self-conditioning can be understood as a reparametrization of the implicit ODE solver for Equation 4  (Song et al., ; Lu et al., ; Zhang & Chen, ; Chen et al., ). For example, Song et al. () considered the $1$st order ODE solver for Equation 4 following

$$ $x^{(t-1)}=\sqrt{\alpha_{t-1}}\hat{x}+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\cdot s ({x^{(t)},t}),$ (6) $$

where $\alpha_{t}$ follows the noise scheduling in Equation 2. Higher-order ODE solvers have also been considered in (Lu et al., ), all of which depend on the previous sample $\hat{x}$. Based on this observation, we introduce a previous video sample $\hat{\mathbf{x}}$ into our parametrization of $\hat{f}_{\theta}(\hat{\mathbf{x}},\mathbf{x}^{(t)},t)$ to mimic these ODE solvers. We emphasize that (Song et al., ; Lu et al., ; Zhang & Chen, ) still follows the same parametrization for the score function $s({x^{(t)},t})$, which is learned by the vanilla diffusion loss. The self-conditioning ODE solver in Equation 6 is only exploited for accelerating generation. A visual illustration of self-conditioning is shown in Figure 2.

We can learn the ODE solver through self-conditioning consistency by directly predicting the clean video $\mathbf{x}^{(0)}$ through a self-conditioning consistency

$$ $\displaystyle\mathcal{L}_{\text{self-conditioning-consistency}}(\theta)$ $\displaystyle=\mathbb{E}_{\hat{\mathbf{x}},\mathbf{x}^{(0)},t}\left[\|\hat{f}_ {\theta}(\hat{\mathbf{x}},\mathbf{x}^{(t)},t)-\mathbf{x}^{(0)}\|^{2}\right]$ $\displaystyle+\lambda\mathbb{E}_{\hat{\mathbf{x}}_{1},\hat{\mathbf{x}}_{2},t} \left[\|\hat{f}_{\theta}(\hat{\mathbf{x}}_{1},\mathbf{x}^{(t)},t)-\hat{f}_{ \theta}(\hat{\mathbf{x}}_{2},\mathbf{x}^{(t)},t)\|^{2}\right],$ (7) $$

where $\hat{\mathbf{x}}_{1},\hat{\mathbf{x}}_{2}$ are two independent samples from the first-frame-and-language conditioned video generation model, and $\lambda$ is a hyperparameter for regularizing the similarity between different samples. To enable the “first guess” for $\hat{\mathbf{x}}$, we consider $f_{\theta}(\mathbf{x}^{(t)},t)$, which is still learned by the vanilla objective for video diffusion as

$$ $\mathcal{L}_{\text{video-diffusion}}(\theta)=\mathbb{E}_{\mathbf{x}^{(0)}, \epsilon,t}\left[\|f_{\theta}(\mathbf{x}^{(t)},t)-\mathbf{x}^{(0)})\|^{2} \right].$ (8) $$

The overall objective for training a self-conditioning-consistent video diffusion model boils down to

$$ $\mathcal{L}(\theta)=\mathcal{L}_{\text{video-diffusion}}(\theta)+\lambda \mathcal{L}_{\text{self-conditioning-consistency}}(\theta).$ (9) $$

Note that despite the video generation model $f_{\theta}$ and the
video refinement model $\hat{f}_{\theta}$ having different input arguments, we can nevertheless share the parameter between $f_{\theta}$ and $\hat{f}_{\theta}$ to train a single model for video generation and video refinement. We describe the training process for $f_{\theta}$ and $\hat{f}_{\theta}$ in Algorithm 1.

##### Feedback Guided Self-Conditioning Consistency.

While we can refine videos only from previously generated samples, it may be desirable to condition the refinement process on any additional feedback for the previously generated video that is available (e.g., feedback from humans or vision language models critiquing which part of the generated video is unrealistic).
When such feedback is available, we can have the refinement model $\hat{f}$ further take the additional feedback as input to guide the refinement process, i.e.,

$$ $\hat{f}_{\theta}(\mathbf{x},\mathbf{x}^{(t)},t,\text{feedback}),$ (10) $$

which can be plugged into our framework for learning using Equation 9.

### 3.2 Inference through VLM guided video generation.

After training the video generation model $f_{\theta}$ and the video refinement model $\hat{f}_{\theta}$ described in Equation 8 and Equation 7,
we can sample from $f_{\theta}$ and iteratively apply $\hat{f}_{\theta}$ for video refinement. Specifically, let $\eta$ be the step size for the noise schedule, $\sigma_{t}$ be a time dependent noise term,
VideoAgent first “guesses” the video plan using the first-frame-and-language conditioned video generation generation, i.e.,

$$ $\mathbf{x}^{(t-1)}=\mathbf{x}^{(t)}-\eta\cdot\nabla_{\theta}f_{\theta}(\mathbf {x}^{(t)},t)+\sigma_{t}\cdot\epsilon$ (11) $$

The sample $\hat{\mathbf{x}}$ after $T$ denoising steps corresponds to the generated video. Next, we can iteratively apply $\hat{f}_{\theta}$ to refine the generated video sample

$$ $\hat{\mathbf{x}}_{(i+1)}=\hat{f}_{\theta}(\hat{\mathbf{x}}_{(i)},\mathbf{x}^{( t)},t),$ (12) $$

where $i$ denotes the video refinement iteration, with $\hat{\mathbf{x}}_{(0)}=\hat{\mathbf{x}}=\mathbf{x}^{(T)}$. We denote the final video after refinement as $\hat{\mathbf{x}}_{\text{refined}}$. A natural question is when to stop the iterative video refinement process. One option is to always refine for a fixed number of iterations. However, over-refinement may lead to less diverse output. To overcome this, we leverage a VLM as a proxy for the environment’s reward to assess whether a refined video is likely to lead to successful execution in the environment. Specifically, we denote a VLM as $\hat{\mathcal{R}}$, which takes a refined video $\hat{\mathbf{x}}_{(i)}$ and returns a binary value $\{0,1\}$ to determine whether a video is acceptable based on overall coherence, adherence to physical laws, and task completion (See prompt for VLM in Appendix A). With $\hat{\mathcal{R}}$, the refinement stops when the VLM decides that the refined video is acceptable. Namely, we have

$$ $\hat{\mathbf{x}}_{\text{refined}}=\hat{\mathbf{x}}_{(i^{*})},\quad\text{where} \quad i^{*}=\min\left\{i:\hat{\mathcal{R}}(\hat{\mathbf{x}}_{(i)})=1\right\}$ (13) $$

Algorithm 2 shows how video plans are generated, refined, and selected at inference time.

Figure: Algorithm 1 Training of Video Generation and Refinement Models with VLM Feedback

### 3.3 Self-improvement through online finetuning

In addition to video refinement through self-conditioning consistency as described in Section 3.1, we can further characterize the combination of video generation and video refinement as a policy, which can be improved by training on additional real data collected from the environment during online interaction. Specifically, the goal is to maximize the expected returns of a policy through trial-and-error interaction with the environment:

$$ $\mathcal{J}_{\text{online}}(\theta)=\mathbb{E}\left[\mathcal{R}(\mathbf{a})\,| \,\pi_{\theta},\rho,\mathcal{E}\right],$ (14) $$

where $\mathcal{R}$ is the true reward function, $\mathcal{E}$ is the interactive environment, and $\pi_{\theta}$ corresponds to Algorithm 2, which contains both the video generation model $f_{\theta}$ and the video refinement model $\hat{f}_{\theta}$ as learnable components to be improved.

A broad array of reinforcement learning methods (Sutton & Barto, ) such as policy gradient (Schulman et al., ) can be employed to maximize the objective in Equation 14. For simplicity, we consider the setup of first executing the policy in the environment, then filtering for successful trajectories, continuing finetuning the video policy using additional online data, and executing the finetuned policy again to collect more data. Specifically, each online iteration constructs an additional dataset by rolling out the policy $\pi_{\theta}$ at the current online iteration

$$ $\mathcal{D}_{\text{new}}=\left\{\hat{\mathbf{x}}_{\text{refined}}\sim\pi_{ \theta}(x_{0},g)\mid\mathcal{R}(\rho(\hat{\mathbf{x}}_{\text{refined}}))=1 \right\},$ (15) $$

where $\rho$ is the optical flow model that maps the refined video to low-level control actions. See Algorithm 3 for details of online policy finetuning.

Figure: Algorithm 2 VLM Guided Replan

Figure: Algorithm 3 Online Finetuning of Video Generation and Refinement Models

## 4 Experiments

We now evaluate the performance of VideoAgent, introducing the experimental settings and variants of VideoAgent in Section 4.1, end-to-end success rate of VideoAgent against the baselines in Section 4.2 and the effect of different components of VideoAgent in Section 4.3. Finally, we show that VideoAgent is effective in improving the quality of real robotic videos in Section 4.4.

### 4.1 Datasets and Experimental Setups

##### Datasets and Environments.

We follow the same evaluation setting as Ko et al. (), which considers three datasets: Meta-World (Yu et al., ), iTHOR (Kolve et al., ), and BridgeData V2 (Walke et al., ). Meta-World consists of 11 robotic manipulation tasks performed by a simulated Sawyer arm, with video demonstrations captured from three distinct camera angles. iTHOR is a simulated 2D object navigation benchmark, where an agent searches for specified objects across four room types. BridgeData V2 is a real-world dataset of robotic manipulation. See more details of datasets and environments in Appendix C.

##### Baselines and VideoAgent Variants.

We consider the following methods for comparison:

- •
AVDC (baseline). This is the Actions from Video Dense Correspondences (AVDC) (Ko et al., ) baseline, which synthesizes a video and predicts optical flow to infer actions.
- •
AVDC-Replan (baseline). When the movement stalls,
AVDC-replan re-runs video generation and action extraction from the flow model to execute a new plan.
- •
VideoAgent. Our proposed video refinement model through self-conditioning consistency as introduced in Section 3.1. VideoAgent generates video and iteratively refines a video plan. We use GPT-4 Turbo for selecting the best video plan during inference (Section 3.2).
- •
VideoAgent-Online. As actions are executed in the online environment,
successful trajectories are collected and used to continue training the video generation and refinement model, as described in Section 3.3.
- •
VideoAgent-Online-Replan. This variant incorporates online filtering of successful trajectories with the replanning mechanism, where replanning is conducted first, and more successful trajectories after replanning are added back to the training data.

### 4.2 End-to-End Task Success

**Table 1: Meta-World Results. The mean success rates of baselines and VideoAgent on 11 simulated robot manipulation environments from Meta-World. VideoAgent consistently outperforms baselines across all tasks.**
|  | door-open | door-close | basketball | shelf-place | btn-press | btn-press-top |
| --- | --- | --- | --- | --- | --- | --- |
| AVDC | 30.7% | 28.0% | 21.3% | 8.0% | 34.7% | 17.3% |
| AVDC-Replan | 72.0% | 89.3% | 37.3% | 18.7% | 60.0% | 24.0% |
| VideoAgent | 40.0% | 29.3% | 13.3% | 9.3% | 38.7% | 18.7% |
| VideoAgent-Online (Iter1) | 41.3% | 32.0% | 17.3% | 12.0% | 45.3% | 14.7% |
| VideoAgent-Online (Iter2) | 44.0% | 29.3% | 18.7% | 18.7% | 46.7% | 16.0% |
| VideoAgent-Online-Replan | 80.0% | 97.3% | 40.0% | 22.7% | 72.0% | 40.0% |
|  | faucet-close | faucet-open | handle-press | hammer | assembly | Overall |
| AVDC | 12.0% | 17.3% | 41.3% | 0.0% | 5.3% | 19.6% |
| AVDC-Replan | 53.3% | 24.0% | 81.3% | 8.0% | 6.7% | 43.1% |
| VideoAgent | 46.7% | 12.0% | 36.0% | 0.0% | 1.3% | 22.3% |
| VideoAgent-Online (Iter1) | 38.7% | 13.3% | 36.0% | 0.0% | 4.0% | 23.2% |
| VideoAgent-Online (Iter2) | 49.3% | 21.3% | 44.0% | 1.3% | 1.3% | 26.4% |
| VideoAgent-Online-Replan | 58.7% | 36.0% | 85.3% | 8.0% | 10.7% | 50.0% |

##### Meta-World.

We report the task success of baselines and VideoAgent in Table 1. Following Ko et al. (), we measure the average success across 3 camera poses with 25 seeds for each camera pose. Without online environment access, VideoAgent already improves the overall success rate from self-conditioned consistency alone over the baseline (from 19.6% to 22.3%). Furthermore, some specific tasks such as faucet-close shows drastic improvement from 12% to 46.7%. With online data collection, VideoAgent-Online further improves the success rate, and each online iteration (rolling out the policy, collecting successful trajectories, and continuing finetuning the policy) further improves the performance. The benefit of VideoAgent persists when replanning is introduced, brining the overall task success to 50%, achieving state-of-the-art in this setup. Qualitative improvement in the refined videos can be found in Figure 8 in Appendix H,

##### iTHOR.

Figure: Table 2: iThor Success Rates comparing VideoAgent with the AVDC baseline.

Next, we evaluate VideoAgent on iThor. Due to the high computational cost of running the iThor simulator, we focus only on evaluating self-conditioning consistency (without online access). We follow the same setup as Ko et al. (), where we measure the average success rate across four rooms each with three objects using 20 seeds. As shown in Table 2, VideoAgent consistently outperforms the baseline, demonstrating the effectiveness of self-conditioning consistency in producing more plausible video plans.

### 4.3 Understanding the Effect of Different Components in VideoAgent

In this section, we aim to understand the effect of different components of VideoAgent. Specifically, we focus on the effect of (1) different types of feedback given to the refinement model, (2) the number of refinement and online iterations, and (3) the quality of the VLM feedback.

#### 4.3.1 Effect of Different VLM Feedback.

Figure: Table 3: Effect of Different Feedback used to train the refinement model. Descriptive feedback from the VLM leads to higher improvement in task success.

In the previous section, we only used VLM during inference to determine when to stop refining a generated video. However, it is natural to wonder if information-rich feedback from the VLM, such as language descriptions of which part of a generated video to improve, might lead to better refined videos. To answer this question, we propose a few variants of VideoAgent according to the feedback available when training the video refinement model as in Equation 10. Specifically, we use VideoAgent to denote training the video refinement model only conditioned on the original task description. VideoAgent-Binary denotes additionally conditioning on whether a generated video is determined to be successful by the VLM. VideoAgent-Suggestive denotes conditioning additionally on language feedback from the VLM on which part of the video needs improvement and how the video can be improved. We train these three versions of the video refinement model, and report the overall task success from Meta-World in Table 3. We see that VideoAgent-Binary improves upon the base VideoAgent, while training with descriptive feedback in VideoAgent-Suggestive leads to even better performance. This suggests that richer feedback from the VLM can facilitate better training of the video refinement model. Improvement for each individual task can be found in the Appendix F.

#### 4.3.2 Effect of Refinement and Online Iterations.

Next, we want to understand whether more refinement iterations and online finetuning iterations generally lead to higher task success. We found that while different tasks require a different number of iterations to achieve the best performance, VideoAgent does perform better as the number of refinement and online iterations increases, as shown in Figure 4 and Figure 4. During video refinement, specific tasks such as handle-press and faucet-close continue to see improvement even at the fifth refinement iteration. Faucet-close especially benefits from more refinement iterations, bringing success rate from 17.3% to 49.3% after five refinement iterations. The improved task success rates across refinement and online iterations suggests that self-conditioning consistency discussed in Section 3.1 and online interaction discussed in Section 3.3 can indeed effectively reduce hallucination and improve physical plausibility in the generated videos.

Figure: Figure 3: Effect of Refinement Iterations. The accuracy of downstream tasks generally increases as the number of refinement iteration increases.
Refer to caption: extracted/5927082/images/mw/cumulative_success_rate_two_tasks_styled_legend.png

#### 4.3.3 Accuracy of VLM feedback on Generated Videos.

Figure: Table 4: VLM Performance measured according to whether a VLM considers a generated video as acceptable using human label as the ground truth.

Since this work is among the first to leverage a VLM to give feedback for video generation, it is crucial to understand whether a VLM can in fact achieve a reasonable accuracy in providing feedback for video generation. To quantify the performance of a VLM, we use human labels on whether a generated video is acceptable as the ground truth, and measure precision, recall, F1-score, and accuracy based on whether GPT-4 Turbo thinks the generated video is acceptable according to trajectory smoothness (consistent across sequential frames), physical stablility, and achieving the goal (See full prompt in Appendix A). We report the average result across 36 generated videos from the Meta-World dataset in Table 4. We see that the original prompt we used (Unweighted) achieves 69% accuracy, suggesting that the VLM is capable of judging generated videos. Since VideoAgent uses multiple refinement iterations, we want to avoid false positives where a bad video is accidentally accepted. We can achieve this by penalizing false positives through reweighting its cost in the prompt, which leads to the VLM rejecting videos when the VLM is uncertain about the video’s acceptability. This adjustment results in a significant increase in precision as shown in Table 4. This weighted version of the prompt is used in the experiments in Section 4.2.

##### Partial Observability.

In the AVDC experimental setup, center cropping the third camera (what is used in the pipeline) often results in most of the robot arm being outside of the frame. We found that the accuracy of the VLM is affected by such partial observatbility. As shown in Table 4, removing the third camera from the prompt leads to much higher accuracy.

##### Descriptive Feedback.

While the VLM can provide binary feedback on whether a generated video is acceptable, we also measure the accuracy of the VLM in giving more descriptive feedback such as identifying the issue and providing suggestions on how to improve the video. We use three examples with human written language feedback as prompt for in-context learning. GPT-4 Turbo achieves 73.5% accuracy on identification and 86.1% accuracy on suggestion, as evaluated by humans. This result is highly encouraging and opens up future directions of leveraging descriptive feedback from VLMs to improve video generation.

### 4.4 Evaluating Self-Refinement on Real-World Videos

In this section, we evaluate VideoAgent’s ability to refining real-world videos, which often contain higher variability, intricate details, nuanced behaviors, and complex interactions. We study the effect of video refinement using both quantitative metrics and qualitatively for holistic evaluation.

**Table 5: BridgeData-V2 Results. Quantitative metrics comparing AVDC and VideoAgent on generated Bridge data. VideoAgent outperforms the baseline according to all except for one metric.**
| Metrics |  | AVDC | Video Agent |
| --- | --- | --- | --- |
| Clip Score |  | 22.39 | 22.90 |
| Flow Consistency |  | 2.48 ± 0.00 | 2.59 ± 0.01 |
| Video Score | Visual Quality | 1.97 ± 0.003 | 2.01 ± 0.003 |
| Temporal Consistency | 1.48 ± 0.01 | 1.55 ± 0.01 |  |
| Dynamic Degree | 3.08 ± 0.01 | 3.07 ± 0.02 |  |
| Text to Video Alignment | 2.26 ± 0.003 | 2.30 ± 0.03 |  |
| Factual Consistency | 2.02 ± 0.004 | 2.07 ± 0.01 |  |
|  | Average Video Score | 2.16 ± 0.01 | 2.20 ± 0.01 |
| Human Eval on Task Success |  | 42.0% | 64.0% |

##### Quantitative Evaluation.

Following previous literature on video generation, we consider two reference-free metrics, CLIP Score (Hessel et al., ) and Flow Consistency (Teed & Deng, ), as well as a set of Video-Scores (He et al., ). CLIP Score measures the cosine similarity between frame feature and text prompt, whereas Flow Consistency measure the smoothness and coherence of motion in the videos calculated from the RAFT model. Video-Scores use five sub-metrics with a focus on correlation with human evaluation and real-world videos.

We report the average across 2250
videos generated from the AVDC baseline and from VideoAgent in Table 5. VideoAgent performs better according to all metrics except for Dynamic Degree from Video-Score (which shows similar performance between the two methods).
Notably, the gain is significant in metrics critical for real-world videos, such as CLIP Score, Factual Consistency, and Text-to-Video Alignment. Improvement in Flow Consistency and Temporal Consistency suggests that VideoAgent produces smoother and more physically plausible videos that adhere better to the physical constraints of the real-world. This directly translates to better performance in real-world robotic tasks in Table 1.

Figure: Figure 5: Correcting Hallucinations in Video Generation: The AVDC model hallucinates after the second frame, removing the colander and placing the banana on the table. In contrast, VideoAgent accurately retains the colander’s position and correctly places the banana inside.
Refer to caption: extracted/5927082/images/bridge_example1.png

##### Qualitative Evaluation.

Next, we qualitatively evaluate generated videos from the AVDC baseline and from VideoAgent. We collect 50 generated videos from each model and conduct human evaluation on whether a generated video looks realistic. Videos with refinement from VideoAgent improves the acceptance rate by 22% as shown in Table 5. We further show an example video with and without refinement in Figure 5, where the baseline (middle row) hallucinates (the bowl disappears) whereas VideoAgent produces the video that completes the task (bottom row). We also present a more fine-grained analysis of Visual Quality, Temporal Consistency, Dynamic Degree, Text to Video Alignment, and Factual Consistency evaluated by humans in the Appendix G with the metrics in Table 8, which further echos the results of human evaluations presented in Table 5.

## 5 Related Work

##### Feedback and Self-improvement in LLMs.

Incorporating feedback and preference signals from feedback into the finetuning process of LLMs, has led to the enormous popularity and practical usability of the current versions of LLMs as chatbots  (Casper et al., ). Preference feedback from humans or other AI systems  (Ouyang et al., ; Lee et al., ; Kaufmann et al., ) are first collected to train a reward model to guide the LLM’s generation or do implicit policy optimization  (Schulman et al., ; Rafailov et al., ). Furthermore LLMs have shown the ability to further improve by iterative refinement during finetuning and inference  (Zelikman et al., ; Yuan et al., ; Tian et al., ). We incorporate this reward driven improvement mechanism in our work, but unlike the LLM setting where the feedback came from a reward model or some proxy of this prefernce model, in our VideoAgent we use natural feedback from real world when simulated videos are turned into actions that are executed in the real world.

##### Image and Video Generation and Editing.

With the advent of large scale foundation models pretrained on internet scale data  (Bommasani et al., ), generation of super realistic multimodal content has become easier. Text generation, image or video generation, and cross-modal generation  (OpenAI et al., ; Reid et al., ; Wu et al., ; Ho et al., ; Singer et al., ; Yang et al., ; Blattmann et al., ) has seen major advancements leveraging the autoregressive and diffusion based models architectures. And moving beyond simple generation, these models have been leveraged for guided text, image or video editing and enhancement  (Huang et al., ) to improve textual and visual aesthetics applied mostly to generative media  (Zhang et al., ). But none of these existing methods focus on grounding a generative simulator in the real world to perform more complex interactive multi-turn agentic and physical tasks needing both perception and control. To solve this bottleneck, we propose VideoAgent to self-improve or edit generated plan based on grounded feedback from real-world to execute robot manipulation tasks.

##### Video Generation for Robot Learning.

Video-based learning for robotics has been extensively studied  (Nair et al., ; Bahl et al., ; Shao et al., ; Chen et al., ; Pari et al., ; Sharma et al., ; Sun et al., ; Lee & Ryoo, ). Methods use video datasets for visual representation learning, goal extraction, and dynamic models for planning  (Finn & Levine, ; Kurutach et al., ), or imitation learning from expert actions  (Fang et al., ; Wang et al., ; Mani et al., ), state sequences  (Torabi et al., ; Lee et al., ; Karnan et al., ), and pretraining on videos followed by RL  (Baker et al., ; Escontrela et al., ). Recently, generative models have advanced video-based learning and planning, framing decision-making as text-conditioned video generation to predict trajectories  (Du et al., ; Ko et al., ; Wen et al., ). Vision-language and text-to-video models generate long-horizon plans for robotic tasks through abstract, visually grounded planning  (Du et al., ; Ajay et al., ). Generative models also simulate agent-environment interactions, enabling zero-shot deployment  (Yang et al., ), and test-time feedback for replanning  (Bu et al., ). Unlike these, our VideoAgent improves video generation during training with real-world feedback and refines actions through test-time self-iteration and replanning.

## 6 Conclusion and Future Work

We have presented VideoAgent, where a video generation model acts as an agent by generating and refining video plans, converting video plans into actions, executing the actions in an environment, and collecting additional data for further self improvement. Through interaction with an external environment, VideoAgent provides a promising direction for grounding video generation in the real world, thereby reducing hallucination and unrealistic physics in the generated videos according to real-world feedback. In order to fully achieve this overarching goal, VideoAgent needs to overcome a few limitations, which calls for future work:

- •
In the online setting, VideoAgent only considers filtering for successful trajectories for further finetuning. Exploring other algorithms such as online RL is interesting future work.
- •
VideoAgent utilizes optical flow for action extraction. It would be interesting to see how VideoAgent works with inverse dynamics model or image-goal conditioned diffusion policy.
- •
We only measured end-to-end task success in simulated robotic evaluation settings. It would be interesting to see how VideoAgent works with real robotic systems.
- •
As additional data is being collected in the online setting, in addition to finetuning the video prediction model, one can also finetune the action extraction module (flow model), and the VLM feedback model using the additionally collected data, which we defer to future work.

## 7 Acknowledgement

Thanks to Doina Precup and Douglas Eck for reviewing draft versions of this manuscript. Thanks to Wenhu Chen for his inputs on VLM feedback. We gratefully acknowledge Sebastian Fischmeister for providing funds for this work, and the Chippie cluster at the University of Waterloo for additional GPU resources.

## Appendix A Prompt Structure for VLM Feedback

### A.1 Binary Classification

We employ a structured prompting strategy to provide feedback on video sequences for the zero-shot classification. The process consists of one Query-Evaluation Phase, each with distinct sub-goals.

### A.2 Identification and Suggestion:

We employ a structured prompting strategy to provide descriptive feedback on video sequences via an in-context few-shot classification setup. The process consists of one Query-Evaluation Phase, each with distinct sub-goals.

## Appendix B Task Descriptions and In-Context Examples for VLM Feedback

Figure: Figure 6: Few-Shot Examples given to VLM: We provide some examples to the VLM and corresponding feedback to teach the VLM in-context how to critic the generated videos for task completion and success or failure.
Refer to caption: extracted/5927082/images/ICL_FewShot.png

## Appendix C Dataset Descriptions in Detail

Figure: Figure 7: Environments and Datasets that we work with: Meta-World, iThor, and BridgeData-V2
Refer to caption: extracted/5927082/images/envs.png

Meta-World  (Yu et al., ) is a simulation benchmark that uses a Swayer robotic arm to perform a number of manipulation tasks. In our experiments, we make use of 11 tasks as shown in Table 1. We capture videos from three distinct camera angles for each task and use the same camera angles for both the training and testing phases. We gather five demonstration videos per task for each camera angle. During the evaluation, we tested on each of the three camera angles with 25 seeds per camera angle. The position of the robot arm and the object is randomized at the beginning of each seed to ensure variability. A trajectory is considered successful if the Video Agent reaches within a really close threshold of the goal state.

iTHOR  (Kolve et al., ) is another popular 2D simulated benchmark that focuses on embodied common sense reasoning. We evaluate the Video as Agent framework on the object navigation tasks, where an agent is randomly initialized in a scene and tasked with finding an object of a specified type (e.g., toaster, television). At each time step, the agent can take one of the four possible actions (MoveForward, RotateLeft, RotateRight, or Done), and observes a 2D scene to operate in. We selected 12 objects ((e.g. toaster, television) to be placed in 4 different room types (e.g. kitchen, living room, bedroom, and bathroom). Again, the starting position of the agent is randomized at the start of each episode. During evaluation, we test the agent across 12 object navigation tasks spread across all 4 room types, 3 tasks per room. A trajectory is successful if the agent views and reaches within 1.5 meters of the target object before reaching the maximum environment step or predicting Done.

To test the usefulness of our framework across different videos types, we also use the BridgeData V2 dataset  (Walke et al., ), a large and diverse dataset of real world robotic manipulation behaviors designed to facilitate research in scalable robot learning. It contains 60,096 trajectories collected across 24 environments using a publicly available low-cost WidowX 250 6DOF robot arm. The dataset provides extensive task and environment variability, enabling skills learned from the data to generalize across environments and domains.

## Appendix D Extended Experiments

### D.1 Further Analysis of VideoAgent-Online

We train VideoAgent-Online for multiple iterations and observe that after 2 iterations, the results start to stabilize. The results for iteration 3 are shown in table 6.

**Table 6: Meta-World Result. The mean success rates of VideoAgent combined with Online and Replan modules as compared to the AVDC baseline**
|  | door-open | door-close | basketball | shelf-place | btn-press | btn-press-top |
| --- | --- | --- | --- | --- | --- | --- |
| VideoAgent | 40.0% | 29.3% | 13.3% | 9.3% | 38.7% | 18.7% |
| VideoAgent-Online(Iter1) | 41.3% | 32.0% | 17.3% | 12.0% | 45.3% | 14.7% |
| VideoAgent-Online(Iter2) | 44.0% | 29.3% | 18.7% | 18.7% | 46.7% | 16.0% |
| VideoAgent-Online(Iter3) | 46.7% | 28.0% | 18.7% | 18.7% | 45.3% | 20.0% |
|  | faucet-close | faucet-open | handle-press | hammer | assembly | Overall |
| VideoAgent | 46.7% | 12.0% | 36.0% | 0.00% | 1.3% | 22.3% |
| VideoAgent-Online(Iter1) | 38.7% | 13.3% | 36.0% | 0.00% | 4.0% | 23.15% |
| VideoAgent-Online(Iter2) | 49.3% | 21.3% | 44.0% | 1.33% | 1.33% | 26.4% |
| VideoAgent-Online(Iter3) | 48.0% | 21.3% | 42.0% | 1.33% | 1.33% | 26.5% |

## Appendix E Architectural Details of VideoAgent

### E.1 Video Diffusion training details

We use the same video diffusion architecture as the AVDC baseline. For all models, we use dropout=0, num head channels=32, train/inference timesteps=100, training objective=predict v, beta schedule=cosine, loss function=l2, min snr gamma=5, learning rate=1e-4, ema update steps=10, ema decay=0.999.

## Appendix F VLM Feedback for Correction

**Table 7: Meta-World: VideoAgent-Feedback Guided Results The mean success rates for various tasks, comparing different VideoAgent-Feedback Guided variants and the AVDC baseline.**
|  | door-open | door-close | basketball | shelf-place | btn-press | btn-press-top |
| --- | --- | --- | --- | --- | --- | --- |
| AVDC | 30.7% | 28.0% | 21.3% | 8.00% | 34.7% | 17.3% |
| VideoAgent | 46.7% | 29.3% | 13.3% | 9.3% | 38.7% | 18.7% |
| VideoAgent-Binary | 46.7% | 32.0% | 14.7% | 6.7% | 38.7% | 21.3% |
| VideoAgent-Suggestive | 46.7% | 33.3% | 18.7% | 12.0% | 41.3% | 22.7% |
| VideoAgent-Online-Suggestive | 52.0% | 28.0% | 21.3% | 16.0% | 46.7% | 22.7% |
|  | faucet-close | faucet-open | handle-press | hammer | assembly | Overall |
| AVDC | 12.0% | 17.3% | 41.3% | 0.00% | 5.30% | 19.6% |
| VideoAgent | 46.7% | 12.0% | 36.0% | 0.00% | 1.3% | 22.9% |
| VideoAgent-Binary | 46.7% | 17.3% | 32% | 0.00% | 5.3% | 23.8% |
| VideoAgent-Suggestive | 48.7% | 17.3% | 46.7% | 0.00% | 5.3% | 26.6% |
| VideoAgent-Online-Suggestive | 45.3% | 20.0% | 48.0% | 2.7% | 5.3% | 27.4% |

## Appendix G Details of Human Evaluation on BridgeData V2

##### Qualitative Evaluation.

Next, we qualitatively evaluate video generation quality using the five Video-Score dimensions: Visual Quality (VQ) for clarity and resolution, Temporal Consistency (TC) for smooth frame transitions, Dynamic Degree (DD) for capturing accurate object/environment changes, Text-to-Video Alignment (TVA) for matching the video to the prompt, and Factual Consistency (FC) for adherence to physical laws and real-world facts. Videos are rated on a 4-point scale based on the metric in He et al. (): 1 (Bad), 2 (Average), 3 (Good), and 4 (Perfect). Our evaluation is based on 50 generated videos from a held-out set.

**Table 8: Task Success and Other Fine-grained Human Evaluation Metrics on BridgeData-V2**
| Metrics |  | AVDC | Video Agent |
| --- | --- | --- | --- |
| Task Success via Human Eval |  | 42.0% | 64.0% |
| Holistic Assessment via Human Eval | Visual Quality | 1.74 | 1.84 |
| Temporal Consistency | 1.58 | 1.76 |  |
| Dynamic Degree | 3.14 | 2.98 |  |
| Text to Video Alignment | 2.66 | 3.04 |  |
| Factual Consistency | 3.22 | 3.30 |  |
|  | Human Eval Average | 2.47 | 2.98 |

In terms of VQ and TC, both the baseline AVDC and our VideoAgent generate average quality videos (graded 2), with AVDC hallucinating more and generating some choppy jumps in videos temporally (we grade such videos as 1) and Video Agent fixing some of these upon video conditioned iterative refinement.
The reason for AVDC baseline having higher DD is attributed to unruly movements that cause higher DD scores compared to VideoAgent, where movements are smoother. This also explains the result in fifth row of Table 5, and upon closer examination of the generated videos and their corresponding individual scores, we observed similar traits in videos having higher DD due to unnatural robot arm movements and object impermanence. TVA shows trends similar to ClipScore in Table 5 due to the better instruction following ability of VideoAgent leading to more controlled generation. FC is a very crucial metric for deployment of video generation agents as policy for task completion in robotics, scene navigation, and so on. Improved visual quality does not imply adherence to correct physical laws and real-world constraints, FC particularly checks for this aspect and due to video conditioned self-refinement, VideoAgent has better FC compared to AVDC.

## Appendix H Examples

### H.1 Improvements in Meta-World

Figure: Figure 8: Correcting Hallucinations in Video Generation: The goal prompt is “Assembly” as shown in the Target Video. The AVDC model has problem of object permanence and action incomplete in last frame. In contrast, our VideoAgent model accurately object permanence and correctly places the inside the peg properly.
Refer to caption: extracted/5927082/images/mw_example1.png

### H.2 Improvements in iThor

Figure: Figure 9: Correcting Hallucinations in Video Generation: The goal prompt is “Television” as shown in the Target Video, the goal is for the navigator to locate the object and reach near it. The AVDC model has difficulty reconstructing and navigating in the livingroom to find the television. In contrast, our VideoAgent model solves the initial frame hallucinations and accurately reaches near the television correctly.
Refer to caption: extracted/5927082/images/ithor_example1.png

### H.3 Identification and Suggestive Feedback Examples

Figure: Figure 10: Detailed VLM Feedback: We show the efficacy of VLMs to provide useful feedback even in the absence of access to a simulator or real-world execution environment. The VLM acts as a proxy reward model to condition VideoAgent on useful corrective signals, leading to improved performance as described in Table 3.
Refer to caption: extracted/5927082/images/long_feedback.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. There seems to be a discrepancy between the self-conditioning-consistency loss In Algorithm 1 and Equation 7, is the second term in Eq. 7 missing in Algorithm 1?
2. During evaluation, is every video plan executed in an open-loop manner? What is the planning horizon for each setup, and how does it compare to the task horizon?
3. In Figure 4, how many additional trajectories are collected per iteration to achieve the improvement?
4. In the last row of Table 5, how is the "task success" defined specifically in the Bridge human evaluation? Is it defined as whether the generated videos respect the prompt from human perception? or as the authors mentioned in line 467, it depends on whether a generated video looks realistic.
5. As the video models are language-conditioned, it would be clearer to include the conditioning variable in the equations or algorithms somewhere.
6. Typo in line 342: brining
soundness: 3
strengths: 1. This paper proposes several novel techniques, such as self-conditioning consistency and incorporating VLM feedback, that improve the quality of the generated video plans through iterative refinement
2. The proposed method adopts a self-improving loop by finetuning the video models with additional successful trajectories collected online
3. The authors provide extensive evaluation results as well as experiment details, which back up the efficacy of the proposed method
4. This paper is well-motivated and easy to follow, it thoroughly discusses the limitations of the proposed work
summary: This paper introduces VideoAgent, an approach that refines video plans generated by video diffusion models through self-conditioning consistency and feedback from pre-trained vision-language models. When online interaction is available, VideoAgent closes the self-improvement loop by alternating between collecting successful online data and finetuning video models. Evaluations are performed on three benchmarks, across both simulated and real-world domains, and from the perspective of both task success rate and video generation quality, showing the effectiveness of the proposed method.
weaknesses: 1. The loop of collecting successful data through environment interaction and finetuning video models might incur large computational overhead, and the improvement seems to become marginal after two online iterations in Figure 4.
2. [nitpick] I believe it is essential to reduce hallucinations and improve the quality of video plans for better decision-making performance and I appreciate the overall contribution. However, in the Metaworld example (Table 1) provided in this work, simply replanning during inference still seems to be a comparatively cost-effective approach that can achieve higher overall success rates with a less performant video plan generator. I understand refinement and replanning can be combined for better improvement, but I wonder if the authors have stronger examples to highlight the necessity of refinement and self-improvement for task success.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: When I read this paper, I encountered some areas of confusion, and I hope my feedback can help the authors improve the clarity of the writing. If there are any inaccuracies in my understanding, I welcome any corrections.

1. The motivation for introducing the consistency model should be clearly articulated. I didn’t see a compelling reason to use the consistency model, as it seems possible to achieve similar results by using the training objective of DDPM and employing DDIM for inference.

2. Lines 119-120 should describe the loss for the consistency model rather than for DDPM. In DDPM, the loss should be calculated from ($x^{t+1}$ to $x^t$), which also makes the "vanilla objective for video diffusion" mentioned in line 197 confusing.

3. In line 224, it seems that video generation should be a single step if using a consistency model. Why are multiple steps required, and if they are, what distinguishes this approach from DDIM?

4. The motivation in Section 3.1 could be more clearly defined. The overall objective of the proposed method is to (1) generate video and (2) refine the video based on feedback until it is accepted by the VLM, using the same parameters. However, this goal is not clearly explained at the beginning of Section 3.1, which requires readers to infer the motivation after understanding the method.

5. Line 164 states, "the model can learn to preserve the realistic part of the video while refining the hallucinatory part," which could be misleading. Feedback is applied to the entire video, not frame-level feedback. This statement may somewhat overstate the model’s capability.

6. It is unclear how the parameters are shared between the video generation model and the video refinement model.

7. It is not shown in the method how the video maps to action. Indicating where this is discussed in the paper would be helpful.
soundness: 2
strengths: 1. The idea of VideoAgent is novel; I haven’t seen an iterative approach to improving video generation quality before.
2. Obtaining feedback on videos from VLM is feasible.
summary: This paper proposes VideoAgent, a model designed to improve video generation quality based on feedback from a VLM. Specifically, VideoAgent trains both a video generation model and a video refinement model, which share parameters and are based on diffusion processes. The video generation model functions as a conventional text-to-video model, while the video refinement model takes as input the ground-truth video, the predicted video, and feedback from the VLM to produce a refined video. The final output video can be mapped to an action trajectory. When the environment is interactive and can confirm task success, VideoAgent performs rollouts within the environment to gather additional data for training (but very slow now). This method is evaluated at the policy level in the Meta-World and iTHOR benchmarks, and for video quality on the Bridge dataset.
weaknesses: 1. The writing section can be strengthened. The distinction between the consistency model and DDPM is vague in this paper, yet it is crucial. Given that I believe DDPM + DDIM can achieve the same results (referring to the implementation of video generation models and video refinement models), it’s even more necessary to explain the necessity and motivation for using the consistency model. Specific issues can be referenced in the questions section.

2. The experimental section of the paper is relatively weak. The baseline is only AVDC, a typical video generation model without a corresponding video refinement model. Possible baselines could include VLP[1] and a naive text-to-video diffusion model (under similar training and inference FLOP). Additionally, results from traditional baselines based on BC and RL should also be included for reference.

3. More realistic robotic manipulation benchmarks should be considered, such as Simper[2].

4. It would be better to conduct experiments on a real robot; currently, the focus is only on testing quality using real-robot videos. Success rate evaluations are missing, which may necessitate the use of a real robot. If a real robot is not available, providing results from Simper would be acceptable.

5. Even after refinement, the video generation quality of VideoAgent on the Bridge dataset is not high compared to modern classic diffusion models, and the resolution is very low.

6. Fundamentally, generating a video and then generating actions is not widely regarded as a correct approach; the improvements made in this paper are still limited by the fundamental flaws of video planning. The video generation is too slow to be suitable as a policy. If the paper could demonstrate the importance of video generation for policy, it would greatly strengthen its contributions.

7. The paper seems to avoid the discussion of video generation speed. In reality, how many actions can be generated per second with this method? Or what is the average time required to generate a single action? The resolution should also be reported.

[1] Video Language Planning.
[2] Evaluating Real-World Robot Manipulation Policies in Simulation

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. VLMs are prone to hallucinations in some cases. Were any hallucinations from VLM feedback observed in the experiments? If such hallucinations were observed, would they potentially mislead the refinement process?

2. How does the proposed method perform in generalization settings? Does introducing feedback from VLM help the proposed method on handling novel text instructions? It would be great to incorporate a generalization experiment on video plan generation on real-robot data (e.g. BridgeData V2).

3. In Algorithm 2, if I understood correctly, the $\pi_{\theta}$ in line 284 should be $\hat f_{\theta}$?
soundness: 3
strengths: The paper presents a novel method which uses feedback from VLMs to refine generated video plans.
It introduces a self-improving consistency model which predicts the clean video from a generated video and feedback from the VLM.
The proposed method can be continuously improved through online fine-tuning.
Experiment results indicate that the proposed method effectively enhances video generation and improves performance on downstream manipulation tasks.
The paper also includes ablation studies to examine the impact of different types of feedback from the VLM, and the effect of refinement and online iterations.
Additionally, it offers an in-depth analysis of the VLM's performance in providing feedback for video generation.
summary: The paper introduces VideoAgent which self-improves generated video plans based on external feedback.
Given an initial frame and a language instruction, VideoAgent first generates a video plan and then iteratively refines it based on feedback from a pre-trained VLM.
The refined video plan can be converted to low-level robot actions for execution.
Successful trajectories are added to the dataset for further fine-tuning the video generation and refinement models.
Experiments on video plan generation were performed in both simulation and real-robot videos.
Results show that the proposed method can reduce hallucination in video plan generation and thus boost performance in downstream manipulation tasks.
weaknesses: 1. The paper leverages a self-conditioning consistency loss (Eqn. 7) for video refinement. The first term in Eqn. 7 is a diffusion loss and the second term is for consistency. The reason why including the second term in Eqn. 7 is not very clear. It seems that it encourages generating consistent $x^{(0)}$ from different $\hat x_{i}$. Is it possible to provide more explanation on why the consistency loss is necessary to be included? It would be great to include an ablation study which compares the refinement performance with and without this consistency loss.

2. Figure 2 is not very clear. If I understood correctly, $x^{(t + \Delta)}, x^{(t)}, x^{(t - \Delta)}$ are noisy latents at different denoising timesteps in a diffusion. But what is the target of this diffusion process? Is it $\hat x_{(i+1)}$? It would be great to provide more descriptions in the caption for further clarification.

3. The paper lacks real-robot experiments to validate the effectiveness of the proposed method in real-world policy learning. The paper would benefit strongly from incorporating a real-robot policy learning experiment. And comparing the proposed method with baseline methods without self-improving consistency (e.g. AVDC) would help understand how the proposed video refinement process helps downstream policy learning in the real world.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 4
questions: See the weakness part.
soundness: 3
strengths: 1. Clear equation writing, which shows the strong technique capabilities of the authors.

2. Figure 1 is clear, which makes me understand the main idea of this paper quickly.

3. Experiments for the VLM feedback is interesting.
summary: This paper proposes VideoAgent for self-improving generated video plans based on external feedback. It first refines the generated video plans using self-conditioning consistency and utilizes feedback from a pretrained vision-language model (VLM), then it collects additional data from the environment to further improve video plan generation. From both simulation and real-world video generation experiments, it shows the effectiveness of the proposed method.
weaknesses: 1. I am unsure whether the *self-conditioning consistency* is an original contribution of this paper because of the poor writing clarity of Section 3.1. From the author's writing, it seems like some other papers also endow similar ideas. If the authors want to claim this as part of their contribution, the author should improve the writing of this part to clearly show which idea or equation is original.

2. I am confused about how to use the feedback from VLM for Equation 10. The authors did not provide any methodological description in Section 3.1 or Appendix, e.g., what is the *feedback*, or how to put the *feedback* into Equation 9.

3. Also, I am unsure whether using VLM feedback to improve video generation models is original since the author didn't discuss any related works for this part. If this is novel (or at least part of this is novel), the author should clearly point out which part is novel. Currently, people may assume that this section is totally based on some existing methods, because the author seems to assume that readers know how Equation 10 works, and therefore does not provide any method explanation.

4. From my point of view, the main contribution (if the above 3 questions can be solved) are: 1) self-consistency conditioning; 2) VLM-feedback; 3) real-world interaction feedback. However, from Table 1, we can see these three modules didn't bring much improvement (VideoAgent v.s. AVDC, only 2.7% increase). Instead, the main improvement comes from Online and Replan (23.5% increase for AVDC and 27.7% for VideoAgent). What's more, Online-and-Replan works across two different video generation methods (AVDC and VideoAgent). Thus, this greatly reduces the value of the three main contributions of this article.

Meta-review: This paper proposes VideoAgent, a method for self-improving video generation. It refines video plans generated by video diffusion models through self-conditioning consistency and feedback from pre-trained vision-language models (VLMs). The refined video plans can be converted to robot actions, and successful trajectories are used for further fine-tuning. Experiments are conducted in simulated robotic manipulation and on real-robot videos, showing reduced hallucination and improved task success rates.

The paper has several novel aspects and the authors have made efforts to address the reviewers' concerns. However, the overall improvement from the proposed key modules (self-conditioning consistency, VLM feedback, real-world interaction feedback) is relatively small compared to the previous approach, which works across different video generation methods. Additionally, the lack of quantitative results for video generation quality makes it difficult to fully assess the effectiveness of the proposed approach. Despite the authors' explanations and plans for future work, these limitations are significant enough to warrant a rejection at this stage.

Decision: Reject