A Control-Centric Benchmark for Video Prediction
Authors: Stephen Tian
Chelsea Finn
Jiajun Wu
Venue: ICLR 2023 poster (2023)
Keywords: benchmarking
video prediction
visual MPC
manipulation

Abstract: Video is a promising source of knowledge for embodied agents to learn models of the world's dynamics. Large deep networks have become increasingly effective at modeling complex video data in a self-supervised manner, as evaluated by metrics based on human perceptual similarity or pixel-wise comparison. However, it remains unclear whether current metrics are accurate indicators of performance on downstream tasks. We find empirically that for planning robotic manipulation, existing metrics can be unreliable at predicting execution success. To address this, we propose a benchmark for action-conditioned video prediction in the form of a control benchmark that evaluates a given model for simulated robotic manipulation through sampling-based planning. Our benchmark, Video Prediction for Visual Planning ($\text{VP}^2$), includes simulated environments with $11$ task categories and $310$ task instance definitions, a full planning implementation, and training datasets containing scripted interaction trajectories for each task category. A central design goal of our benchmark is to expose a simple interface -- a single forward prediction call -- so it is straightforward to evaluate almost any action-conditioned video prediction model. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling by analyzing five highly-performant video prediction models, finding that while scale can improve perceptual quality when modelling visually diverse settings, other attributes such as uncertainty awareness can also aid planning performance.

Published as a conference paper at ICLR 2023
A CONTROL-CENTRIC BENCHMARK FOR VIDEO PREDICTION
Stephen Tian, Chelsea Finn,& & .Jiajun Wu
Stanford University
ABSTRACT
Video a promising source of knowledge for embodied agents to learn models the world's dynamics. Large deep networks have become increasingly effective modeling complex video data in a self-supervised manner, as evaluated by met- ries based human perceptual similarity pixel-wise comparison. However, it remains unclear whether current metrics are accurate indicators of performance downstream tasks. We find empirically that for planning robotic manipulation, existing metrics can be unreliable at predicting execution success. To address this, we propose benchmark for eetion-conditioned video prediction in the form of a control benchmark that evaluates a given model for simulated robotic manip- ulation through compling-based planning. Our benchmark, Video Prediction for Visual Planning (VP*), includes simulated environments with 11 task categories and 310 task instance definitions, full planning implementation, and training datasets containing scripted interaction trajectories for each task category. A cen- tral design goal of our benchmark to expose a simple interface - a single forward prediction call so it is straightforward to evaluate almost any action-condittoned video prediction model. We then leverage our benchmark to study the effects scaling model size, quantity of training data, and model ensembling by analyzing five highly-performant video prediction models, finding that while scale can im- prove perceptual quality when modelling visually diverse settings, other attributes such as uncertainty awareness can also aid planning performance. Additional en- vironment and evaluation visualizations are at this link
INTRODUCTION
Dynamics models can empower embodied agents to solve range of tasks by enabling downstream policy learning or plamning. Such models can be learned from many types data, but video is one modality that is task-agnostic, widely available, and enables learning from raw agent observations in a self-supervised manner. Learning a dynamics model from video can be formulated as a video pre- diction problem, where the goal is to infer the distribution of future video frames given initial frames as well as the actions taken by an agent in the scene. This problem is challenging, but scaling up deep models has shown promise in domains including simulated games and driving (Oh) et al. 2015 Harvey et al. (022). as well as robotic manipulation and locomotion (Denton & Fergus, 2018 Villegas et 2019 Yan et 2021 Babaeizadeh et 2021 Voleti al. 2022).
more
As increasingly large and sophisticated video prediction models continue to be introduced, how can researchers and practitioners determine which ones to use in particular situations? This question remains largely unanswered. Currently, models are first trained on video datasets widely adopted by the community (Lonescu et al.) 2014, Geiger et al., 2013 Dasari et al., 2019) and then evaluated on held-out test sets using variety of perceptual metrics. Those include metrics developed for image and video comparisons (Wang et al.] (2004). as well as recently introduced deep perceptual metrics (Zhang et al., 2018 Unterthiner et al.) 2018). However, it is open question whether perceptual metrics are predictive of other qualities, such as planning abilities for an embodied agent.
In this work, we take step towards answering this question for one specific situation: how can compare action-conditioned video prediction models in downstream robotic control? We propose benchmark for video prediction that is centered around robotic manipulation performance. Our benchmark, which we call the Video Prediction for Visual Planning Benchmark (VP²), evaluates predictive models manipulation planning performance by standardizing all elements of control

---

Published as a conference paper at ICLR 2023
setup except the video predictor. It includes simulated environments, specific start/goal task instance specifications, training datasets of noisy expert video interaction data, and fully configured model- based control algorithm.
For control, our benchmark uses visual foresight (Finn & Levine 2017 Ebert et al. 2018). a model- predictive control method previously applied to robotic manipulation. Visual foresight performs planning towards a specified goal by leveraging a video prediction model to simulate candidate action sequences and then scoring them based on the similarity between their predicted futures and the goal. After optimizing with respect to the score (Rubinstein) 1999; de Boer et al. [2005] Williams) al., 2016), the best action sequence executed for a single step, and replanning performed at each step. This is natural choice for our benchmark for two reasons: first. it is goal-directed, enabling a single model to he evaluated on many tasks, and second, interfaces with models only by calling forward prediction, which avoids prescribing any particular model class or architecture.
The main contribution of this work a set of benchmark environments, training datasets, and control algorithms to isolate and evaluate the effects of prediction models on simulated robotic manipula- tion performance. Specifically, we include two simulated multi-task robotic manipulation settings with a total of 310 task instance definitions, datasets containing 5000 noisy expert demonstration trajectories for each of 11 tasks, and a modular and lightweight implementation of visual foresight.
Through our experiments, els that score well on frequently used metrics suffer when used in the context of control, as shown in Figure [] Then, to explore how can develop better models for control, leverage our benchmark to analyze other ques- tions such as the effects of model size, data quantity, and modeling uncertainty. We empir- ically test recent video prediction models, in- cluding recurrent variational models as well as a diffusion modeling approach. We will open source the code and environments in the bench- mark in an easy-to-use interface, in hopes that will help drive research in video prediction for downstream control applications.
find that mod-
Model
Melrics f
Control Failure
Starting state
Model B
Metrics J.
Control Success!
Figure 1: Models that score well on perceptual metrics may generate crisp but physically infeasible predictions that lead to planning failures. Here, Model A predicts that that the slide will move on its own.
2 RELATED WORK
Evaluating video prediction models. Numerous evaluation procedures have been proposed for video prediction models. One widely adopted approach to train models on standardized datasets (Geiger al. 2013 Jonescu et al. 2014)) Srivastava et al.) 2015 Cordts et al.) 2016 Finn jet al., 2016; Dasari et al. 2019) and then compare predictions to ground truth samples across sev- eral metrics on a held-out test set. These metries include several image metrics adapted to the video case, such as the widely used fe per-pixel Euclidean distance and Peak signal-to-nonn -o-noese ratio (PSNR). Image metrics developed correlate structural similarity (SSIM) (Wang et | 20004), as well as recent methods based on features learned by deep neural networks like LPIPS (Zhang al., 2018)) and FID (Heusel al. 2017). FVD (Un-) terthiner et al. (0188) extends FID to the video domain via pre-trained 3D convolutional network. While these metrics have been shown to correlate well with human perception, it not clear whether they are indicative of performance on control tasks. Geng et al. (2022) develop correspondence-wisg prediction losses, which use optical flow estimates make losses robust to positional errors. These losses may improve control performance and are an orthogonal direction to our benchmark.
specifically with human perceptual judgments include
Another class of evaluation methods judges a model's 's ability to make predictions about the outcomes of particular physical events, such whether objects will collide or fall over (Sanborn et al.) 2013) Battaglia et al. 2013 Bear et al.) 2021b. p.. This excludes potentially extraneous information from the rest of the frame. Our benchmark similarly measures only task-relevant components of predicted videos, but does so through the lens of overall control success rather than hand-specified questions. video prediction models Atari games by training Q-learning agent using predicted data. We evaluate learned models for planning rather than policy learning, and extend evaluation to robotic manipulation domains.
Oh et al. (2015) evaluate retion-coonitiinn
2

---

Published as a conference paper at ICLR 2023
Perceptual
Control
Perceptual
Control
Model Loss
FVD
LPIPS*
SSIM Success
Model Loss
FVD
LPIPS*
SSIM Success
FitVid MSE
30.7
3.4
87.8
65%
FilVid MSE
9.0
0.62
97.4
58%
+LPIPS=1
18.0
2.8
89.3
67%
+LPIPS=1
5.9
0.63
97.5
82%
+LPIPS=10
24.3
4.1
84.6
35%
+LPIPS=10
6.8
0.70
97.3
32%
SVG'
MSE
51.7
5.1
82.7
80 %
SVG'
MSE
10.6
0.97
95.3
70%
+LPIPS=1
40.7
4.4
83.2
80 %
+LPIPS=1
7.2
0.89
95.5
73%
+LPIPS=10
45.1
4.8
81.8
37%
+LPIPS=10
24.2
1.1
94.0
10%
(a) robosuite pushing tasks
(b) RoboDesk: push red button
Perceptual
Control
Perceptual
Control
Model Loss
FVD
LPIPS* SSIM Success
Model Loss
FVD
LPIPS*
SSIM Success
FitVid MSE
20.5
1.25
94.4
50%
FitVid MSE
15.1
1.08
95.8
38%
+LPIPS=1
9.8
1.26
93.3
75%
+LPIPS=1
10.2
1.08
94.9
36%
10
7.3
1.30
92.8
83%
+LPIPS=10
9.8
1.39
93.6
13%
SVG'
MSE
18.3
1.68
91.3
47%
SVG' MSE
22.5
1.88
90.6
58 %
+LPIPS=1
11.0
1.58
90.9
68%
+LPIPS=1
4.9
2.06
89.7
10%
+LPIPS=10
18.9
1.76
90.4
20%
+LPIPS=10
22.6
2.48
88.2
10%
(c) RoboDesk: upright block off table
(d) RoboDesk: open slide
Table I: Perceptual metrics and control performance for models trained using MSE objective, well with added perceptual losses. For each metric, the bolded number shows the best value for that task, *LPIPS scores are scaled by 100 for convenient display. Full results can be found in Appendix G
Benchmarks for model-based and offline RL. Many works in model-based reinforcement learning
evaluate on simulated RL benchmarks (Brockman et al., 2016 Tassa et al.. 2018 Ha & Schmidhu-
ber, 2018, Rajeswaran al., 2018 Yu al., [209]] Ahh et al. 2019: Zhu et al.] 2020] Kannan
et al. 2021D, while real-world evaluation setups are often unstandardized. Offline
RL and imita-
tion learning benchmarks (Zhu et al.) 2020; Fu et al. 2020 Gulcehre et al., 2020: Lu et al. 2022)
provide training datasets along with environments, Our benchmark includes environments based on the infrastructure of robosuite (Zhu et al. 2020) and RoboDesk (Kannan et al.) 202I). but further includes task specifications in the form of goal images, cost functions for planning, well as implementations of planning algorithms. Additionally, offline RL. benchmarks mostly analyze model-free algorithms, while in this paper we focus model-based methods. Because planning using video prediction models is sensitive to details such as control frequency, planning horizon, and cost function, our benchmark supplies all aspects other than the predictive model itself.
THE MISMATCH BETWEEN PERCEPTUAL METRICS AND CONTROL
In this section, we present case study that analyzes whether existing metrics for video predic- tion are indicative of performance downstream control tasks. We focus two variational video prediction models that have competitive prediction performance and are fast enough for planning: FitVid (Babaeizadeh et al., 2221) and the modified version of the SVG model (Denton & Fer- gus 2018) introduced by Villegas et al. (2019), which contains convolutional as opposed to fully- connected LSTM cells and uses the first four blocks of VGG19 the encoder/decoder architecture. We denote this model as SVG'. We perform experiments on two tabletop manipulation environ- ments, robosuit t and RoboDesk, which each admit multiple potential downstream task goals. Additional environment details are in Section 4
When selecting models analyze, our goal isting metrics. One strategy for learning models that align better with human perceptions of realism add auxiliary perceptual losses such as LPIPS (Zhang et al., 2018). Thus, for each environment, we train three variants of both the Vid and SVG' video prediction models. One variant is trained with a standard pixel-wise l2 reconstruction loss (MSE), while the other two are trained using an additional perceptual loss in the form of adding the LPIPS score with VGG features implemented by Kastryulin et al. (2022) between the predicted and ground truth images at weightings 1 and 10. We train each model for 150K gradient steps. We then evaluate each model in terms of FVD (Unterthiner ct al. 2018). LPIPS (Zhang et al. 2018), and SSIM (Wang et al.) 2004) on held-out validation sets, as well planning performance robotic manipulation via visual foresight (Finn & Levine 2017
train models that have varying performance on ex-
3

---

Published as a conference paper at ICLR 2023
Task Definitions
Planning Implementation
Enviromments (Robosuite & RoboDesk)
MPPI/CEN sampling- based optimizers
Pre-trained classifier
Task instance specifications
COSt functions
Video Prediction Interface
Stars
Goal
Only requtred implenent one function! def
Training Datasets
call__[seef contxt_franes, action_segls
Input: 2 context franes & actions
Expert scripted
Output: Predictions for T future frames nodel, podet__rredictinn:
interaction datascts
Overview of the Vp benchmark, which contains simulated environments, training data, task instance specifications, along with planning implementation with pre-trained cost functions. The interface for evaluating a new model on the benchmark is model forward pass.
Figure 2:
Ebert et al. 2018). As shown in Table 1 we find that models that show improved performance on these metrics do not always perform well when used for planning, and the degree to which they are correlated with control performance appears highly task-dependent.
For example, FVD tends to be low for models that are trained with an auxiliary LPIPS loss at weight 10 on robosuite pushing, despite weak control performance. At the same time, for the upright block off table task, FVD is much better correlated with task success compared to LPIPS and SSIM, which show little relation control performance. We also that models can perform almost identically on certain metrics while ranging widely in control performance.
This case study is conducted in relatively narrow setting, and because data from these simulated environments is less complex than "in-the-wild" datasets, the trained models tend to perform well absolute scale across all metrics. Existing metrics can certainly of general prediction quality, but we can observe even in this example that they can be misaligned with a model's performance when used for control and could lead erroneous model selection for downstream manipulation. Therefore, we believe that a control-centric benchmark represents another useful axis of comparison for video prediction models.
effective indicators
THE VP2 BENCHMARK
In this section, we introduce the vp² benchmark. The main goal of our benchmark is to evaluate the downstream control performances of video prediction models. To help isolate the effects of models as opposed to the environment or control algorithm, we include the entire control scheme for each task part of the benchmark. We design Vp2 with three intentions in mind: (1) it should be accessible, that is, evaluating models should not require any experience in control or RL, (2) it should be flexible, placing as few restrictions as possible on models, and (3) should emphasize settings where model-based methods may be advantageous, such as specifying goals on the fly after training on a multi-task dataset.
vp2 consists of three main components: environment and task definitions, a sampling-based planner, and training datasets. These components are shown in Figure
4.1 ENVIRONMENT AND TASK DEFINITIONS
One advantage of mrodel-based methods is that a single model can be leveraged to complete a va- riety of commanded goals at test time. Therefore, allow benchmark to evaluate models'
4

---

Published as a conference paper at ICLR 2023
abilities to perform multi-task planning, we select environments where there are many different tasks for the robot to complete. VP² consists of two envirouments: a tabletop setting created using robosuite (Zhu et al. 2020), and desk manipulation setting based on RoboDesk (Kannan et al. 2021]. Fach environment contains a rohot model as well as multiple objects to interact with.
For each environment, further define task categories, which represent semantic tasks in that environment. Each task category is defined by success criterion based on the simulator state.
Many reinforcement learning benchmarks focus single task and reset the environment to the same simulator state at the beginning of every episode. This causes the agent to visit a narrow dis- tribution of states, and does not accurately represent how robots operate in the real world. However, simply randomizing initial environment states can lead to higher variance in evaluation. To test models' generalization capabilities and robustness to environment variations, we additionally define task instances for each task category. A task instance is defined by an initial simulator state and an RGB goal image Ig E IR. 64x4xx3 observation of the desired final state. Goal images are a flexible way of specifying goals, but can often be unachievable based on the initial state and action space. We ensure that for each task instance, is possible to reach an observation matching the goal image within the time horizon by collecting them from actual trajectories executed in the environment.
Vp² currently supports evaluation on tasks across two simulated environments:
Tabletop robosuite environment. The tabletop environment is built using x robosuite (Zhu ct al. 2020) and contains objects: two cubes of varying sizes, a cylinder, and a sphere. To pro- vide more realistic visuals, we render the scene with the iGibson renderer (Li et al. 2021). This environment contains 4 task categories: push {large cube, small cube, cylinder, sphere]. We include 25 task instances per category. where the textures of all objects are ran- domized from a set of 13 for each instance. When selecting the action space, we prefer a lower- dimensional action space minimize the complexity of sampling-based planning, but would like one that can still enable a range of tasks. We choose to use an action space A E R. that fixes the end-effector orientation and represents a change in end-effector position command, as well as an action opening or closing the gripper, that is fed to an operational space controller (OSC).
RoboDesk environment. The RoboDesk environment (Kannan al., (02T) consists of a Franka Panda robot placed in front of a desk, and defines several tasks that the robot can com- plete. Because the planner needs to be able judge task success from several frames, we consider subset of these tasks that have obvious visual cues for success. Specifically, we use tasks: push {red, green, blue} button, open {slide, drawer), push fupright block, flat block) off table. We find that the action space in the original RoboDesk environment works well for scripting data collection and for planning. Therefore in this environment A E R represents a change in gripper position, wrist angle, and gripper command.
4.2 SAMPLING-BASED PLANNING
Our benchmark provides not only environment and task definitions, but also control setup that allows models to be directly scored based on control performance. Each benchmark run consists of a series of control trials, where each control trial executes sampling baseed planning using the given model on a particular task instance. At the end of T control steps, the success or failure on the task instance is judged based on the simulator state.
To perform planning using visual foresight, at each step the sampling- based planner attempts to solve the following optimization problem to plan an action sequence given a goal image 7 context frames Ic. cost function C, and a video prediction model fo: mina .02...er Σ Cff a1:T): Ig). The best action is then selected, and re-planning is performed at each step to reduce the effect of com- pounding model errors. We use context frames and predict T = 10 future frames across the benchmark. As in prior work in model-based RL, we implement sampling- based planner that uses MPPI (Wiliamss et al.] 2016 Nagabandi et al. 2019) to sample candidate action sequences, perform forward prediction, and then update the sampling distribution based on these scores. We provide de- fault values for planning hyperparameters that have been tuned to achieve strong performance with perfect dynamics model, which can be found along with additional details in Appendix B
T
vP² additionally specifies the cost function C for each task category. For the task categories in the robosuit te environment, we simply use pixel-wise mean squared error (MSE) as the cost. For
5

---

Published as a conference paper at ICLR 2023
the RoboDesk task categories, find that an additional task-specific pretrained classifier yields improved planming performance. We train deep convolutional networks to classify task success on each task, and weighted combination of MSE and classifier logits as the cost function. We provide these pre-trained model weights as part of the benchmark.
4.3 TRAINING DATASETS
Each environment in VP² comes with datasets for video prediction model training. Each training dataset consists of trajectories with timesteps, each containing 256 256 RGB image observa- tions and the action taken at each step. Specifics for each environment dataset are as follows, with additional details Appendix D
robosuite Tabletop environment: We include 50K trajectories of interactions collected with a hand-scripted policy to push a random object in the environment in a random direction. Object textures are randomized in each trajectory.
RoboDesk environment: For each task instance, we include 5K trajectories collected with a hand- scripted policy, for a total of 35K trajectories. To encourage the dataset to contain trajectories of varying success rates, we apply independent Gaussian noise to each dimension of every action from the scripted policy before executing it.
BENCHMARK INTERFACE
One of our goals is to make the benchmark easy as possible to use, without placing restrictions on the deep learning framework nor requiring expertise in RL or planning. We achieve this through two main design decisions. First, by our selection of sampling- based planning method, we remove as many assumptions as possible from the model definition, such as differentiability or an autoregres- sive predictive structure. Second, by implementing and abstracting away control components, we establish a code interface that requires minimal overhead.
To evaluate model Vp², models must first be trained on one dataset per environment. This is an identical procedure to typical evaluation on video benchmarking datasets. Then, with forward pass implementation, benchmark uses the model directly for plamning described in Section 4.2 Specifically, given context frames [41 .2.,,,,,,, Lt] and an action sequence [a1, a2 ..., 4++T-1]. the forward pass should predict the next N frames [T++1, 1, Ie+2,.., |++T]. We anticipate this will incur low overhead, similar functions often implemented track model validation performance.
While this interface is designed for for ease-of-use and comparison, Vp2 can also be used in an "open" evaluation format where controllers may be modified, benchmark entire planning systems.
EMPIRICAL ANALYSIS OF VIDEO PREDICTION AT SCALE FOR CONTROL
Next, we leverage our benchmark as starting point for investigating questions relating to model and data scale in the context of control. We first evaluate set of baseline models on our benchmark tasks. Then in order to better understand how we can build models with better downstream control performance on VP2, we empirically study the following questions:
How does control performance scale with model size? Do different models scale similarly? What are the computational costs at training and test time?
How does control performance scale with training data quantity?
Can planning performance be improved by models with better uncertainty awareness that can detect when they are queried on out-of-distribution action sequences?
6.1 PERFORMANCE OF EXISTING MODELS ON Vp2
To establish performance baselines, we consider five models that achieve either state-of-the-art or competitive performance on metrics such SSIM, LPIPS, and FVD. See Appendix A for details.
FitVid (Babaeizadeh et al. 2021 is a variational video prediction model that achieved of-te-of-the-ane results the Human 3.6M (lonescu et al. 2014) and RoboNet (Dasari et al. 2019) datasets. It has shown the ability to fit large diverse datasets, where previous models suffered from underfitting.

---

Published as a conference paper at ICLR 2023
Success rates for existing models on benchmark tasks
FRLVS
SVC'
MCVD
MsskVTT
Struct-VRNN
Simnalstor
1.0
Success rate
0.8
0.4
0.2
Robownito prshh Flnt binck Open disawer
Open slide
Bhe buttom Green button Red button Upright blck
Aggimgat
Task name
Figure 3:
Performance of existing models on VP We aggregate results over the 4 robosuite task cate- gories. Error bars show min/max performance over 5 control runs, except for MCVD where we use 3 runs due computational constraints. On the right, show the mean scores for each model averaged across all tasks, normalized by performance of simulator.
SVG' (Villegas et al. 2019) is also variational RNN-based model, but makes sev- eral unique architectural choices - opts for convolutional LSTMs and a shallower en- coder/decoder. We the f2 loss in the original SVG model rather than & prescribed by Villegas et al. (2019) to isolate the effects of model architecture and loss function.
Masked Conditional Video Diffusion (MCVD) (Voleti et al. 2022) diffusion model that can perform many video tasks by conditioning on different subsets of video frames. To our knowledge, diffusion-baa video prediction models have not previously been applied learn action-conditioned models. We adapt MCVD to make it cediiincondiiooned by following a sile-and-com ann-connatenetee procedure similar to Finn et al. (2016).
Struct-VRNN (Minderer et 2019) uses a keypoint- based representation for dynamics learning. We train Struet VRNN without keypoint sparsity or temporal correlation losses for simplicity, finding that they do not significantly impact performance on our datasets.
MaskViT (Gupta et al. 2022) uses a masked prediction objective and iterative decoding scheme to enable flexible and efficient inference using transformer-bat architecture.
We train each model except MCVD to predict 10 future frames given context frames and agent actions. For MCVD, we also use a planning horizon of 10 steps, but following the procedure from the paper, we predict 5 future frames time, and autoregressively predict longer sequences.
While we can compare relative planning performance between models based on task success rate, it difficult evaluate absolute performance when models are embedded into a planning framework. To provide an upper bound on how much the dynamics model can improve control, we include baseline that uses the simulator directly as the dynamics but retains the planning pipeline. This disentangles weaknesses of video prediction models from suboptimal planning or cost functions.
In Figure 3 we show the performance of the five models on our benchmark tasks. We see that for the simpler task push blue button. the performance of existing models approaches that of the true dynamics. However, for robosuite e and the other RoboDesk tasks, there are significant gaps in performance for learned models.
6.2 MODEL CAPACITY
Increasingly expressive models have pushed prediction quality on visually complex datasets. How- ever, unclear how model capacity impacts downstream manipulation results on Vp² tasks. In this section, we use our benchmark to study this question. Due to computational constraints, consider only tasks in the robosuite tabletop environment. We train variants the FitVid and SVG' models with varying parameter counts. For FitVid, create two smaller variants by halving the number encoder layers and decreasing layer sizes in one model, and then further decreasing the number of encoder filters and LSTM hidden units in another ("mini"). For SVG', we vary the
7

---

Published as a conference paper at ICLR 2023
Model Size vs. Control Performance on RoboSuite Tasks
Control success rate (threshold 00.05)
1.0
Model type
FitVid
Simplator model
MCVD
0.9
SVG'
SVG M=2,=11
SVG 0.5,K
SVG' M-1,K=1
SVG
0.8
$
SVG
M-
Vid
Vid Smaller encoder
M 1,K
Mini
MCVD
FitVid
0.7
Full
0.6
10a
10 Number of Parameters (log scale)
10a
10a
Figure 4: Control performance robosui tasks across models with capacities ranging from million 300 million parameters. Error bars represent standard error the mean (SEM) across control runs over set of 100 task instances, except for MCVD where we perform 2 control runs due to computational constraints. We hypothesize that larger versions of FitVid overfit the data, and see that in general, model capacity does not seem to yield signficantly improved performance these tasks.
model size by changing parameters M and K as described by Villegas et al. (2019), which represent expanding factors the size of the LSTM and encoder/decoder architectures respectively.
In Figure we plot the control performance on the robosuite tabletop environment versus the number of parameters. While SVG' sees slight performance im- provements with certain larger configurations, in general we do not see a strong trend that increased model capacity yields improved performance. We hypothesize that this is because larger models like the full FitVid architecture tend to overfit action sequences seen in the dataset.
Model
Variant
Params Pred. time (s)
FirVid
Full
302M
5.63
Small eneoder
6.5M
Mini
2.3M
0.29
SVG
M 2,K =
325M
3.58
2.K
M
1.K
96M
2.39
1.K
83M
1.21
1.K
21.5M
0.52
MCVD Basc
56M
220
We additionally note the wall clock forward prediction time for each model in Table 2 Notably, while the MCVD model achieves competitive control performance, its forward pass computation time more than 10x that of the full FitVid model. While diffusion models have shown comparable prediction quality compared to RNN-based video prediction models, the chal- lenge of using these models efficiently for planning remains.
Table 2: Comparison of median wall clock forward pass time for predicting 10 future frames. We use a batch size of 200 samples and one NVIDIA Titan RTX GPU.
6.3 DATA QUANTITY
Data-driven video prediction models require suffi- cient coverage of the state space to be able to per- form downstream tasks, and much effort has gone into developing models that are able to fit large datasets. Therefore, it is natural to ask: How does the number of training samples impact downstream control performance? To test this, we train the full FitVid and base SVG' models on subsets of the rebosuite tabletop environment dataset consist- ing of 1K, 5K, 15K, 30K, and 50K trajectories of 35 steps each. We then evaluate the control per- formance for each model for the aggregated 100 robosuite control tasks. Figure shows the con- trol results for models trained on consecutively in- creasing data quantities. We find that performance improves when increasing from smaller quantities of data on the order of hundreds to thousands of trajec-
Effect of data size on control success: RoboSuite pushing
Ccatrol mocess
Number training tonfectories
Evaluating the effects of increasing training dataset size all robosuite te tasks. Additional data boosts performance slightly, the benefit quickly plateaus. Shaded areas show 95% confidence intervals across 3 runs.
Figure 5:
8

---

Published as a conference paper at ICLR 2023
tories, but gains quickly plateau. We hypothesize that this is because the distribution of actions in the dataset is relatively constrained due to the scripted data collection policy.
6.4 UNCERTAINTY ESTIMATION VIA ENSEMBLES
Because video prediction models are trained on fixed datasets before being deployed for control in visual foresight, they may yield incorrect predictions when queried on out-of-distribution actions during planning. As result, we hypothesize that planning performance might be improved through improved uncertainty awareness that is, a model should detect when it is being asked to make pre- dictions it will likely make mistakes We test this hypothesis by estimating epistemic uncertainty through a simple ensemble disagreement method and integrating it into the planning procedure.
Concretely, apply score penalty during planning based on the instantiation ensem- ble disagreement from Yu al. (2020). Given an ensemble of N = video prediction mod- els. we use all models from the ensemble to perform prediction for each action sequence. Then, we compute the standard task score using randomly selected model, and then subtract a penalty based on the largest & model devi- ation from the mean prediction. Because sub- tracting the score penalty decreases the scale of rewards, experiment with the temperature = 0.01, 0.03, 0.05 for MPPI, and report the best control result for each task across values of y for single and ensembled models. Addi- tional details about the penalty computation can be found in Appendix
Control performance of ensemblee disagreement stralegy
Eusemble
Single model
1.-
Sucress rate
0.8
0.6
0-1
0.2
0.0
PiVVa SVG
FiVid SVG'
Pivaa svc Upright block table
Open slide
Push batton
Task
Figure 6: Control performance when using ble disagreement control on three tasks. can see that for upright block off table, ensem- ble disagreement improves performance, but on the other two tasks, performance is comparable slightly weaker than single model. Error bars show min/max performance across
The results of using this approach for control are shown in Figure 6 We find that the un- certainty penalty is able achieve slightly im- proved performance on the upright block off table task, comparable performance on push green button, and slightly weaker performance on open slide. This also causes expensive training and inference times to scale linearly with ensemble size when using naïve im- plementation. However, our results indicate that efforts in developing uncertainty-aware models may be one avenue for improved downstream planning performance.
control runs.
CONCLUSION
In this paper, proposed control-centric benchmark for video prediction. After finding em- pirically that existing perceptual and pixel-wise metrics can be poorly correlated with downstream performance for planning robotic manipulation, evaluation method. Our henchmark consists of 13 task categories in two simulated multi-task robotic manipulation environments, and has an easy-to-use interface that does not place any limits on model structure or implementation. We then leveraged our benchmark to investigate questions related to model and data scale for five existing models and find that while scale important to some de- gree for VP² tasks, improved performance on these tasks may also come from building models with improved uncertainty awareness. We hope that this spurs future efforts in developing action- conditioned video prediction models for downstream control applications, and also provides help- ful testbed for creating new planning algorithms and cost functions for visual MPC.
proposed Vp2 as additional control-cnntric
Limitations. vp2 consists of simulated, shorter horizon tasks. As models and planning methods improve, it may be extended to include more challenging tasks. Additionally, compared to widely adopted metrics, evaluation scores our benchmark are more computationally intensive to obtain and may be less practical to track over the course training. Finally, robotic manipulation is one of many downstream tasks for video prediction, and benchmark may not be representative of performance on other tasks. We anticipate evaluation scores on vp to be used conjunction with other metrics for holistic understanding of model performance.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide our open-sourced for the benchmark in Appendix We have also released the training datasets and pre-trained cost function weights at that link.
ACKNOWLEDGMENTS
We thank Tony Zhao for providing the initial FitVid model code, Josiah Wong for guidance with robosuite, Fei Xia for help customizing the iGibson renderer, as well as Yunzhi Zhang. Agrim Gupta, Roberto Martin-Martín, Kyle Hsu, and Alexander Khazatsky for helpful discussions. This work is in part supported by ONR MURI 1000014-22-1222 and the Stanford Institute for Human- Centered AI (HAI). Stephen Tian supported by NSF Graduate Research Fellowship under Grant No. DGE-1656511 18.
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: ROboties BEnchmarks for Learning with low-cost robots. In Conference on Robot Learning (CoRL), 2019.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv: Arxiv- 13195, 2021.
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327- 18332, 2013.
Daniel Bear, Elias Wang, Damian Mrowea, Felix J. Binder, Hsiao- Yu Tung. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan- Yun Sun, Fei-Fei Li, Nancy Kanwisher, Josh Tenen- baum, Dan Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In 3rd Annual Conference Robot Learning. CoRL 2019, Osaka, Japan, October 30 -ovember 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 885-897. PMLR, 2019.
Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19 67, February 2005.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1182-1191, PMLR, 2018.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: Arxiv-1812. 00568, 2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017. Singapore, Singapore, May 29 June 3, 2017, pp. 2786-2793. IEEF, 2017.
10

---

Published as a conference paper at ICLR 2023
Chelsea Finn, lan J. Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016. Barcelona, Spain, pp. 64-72, 2016.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
Daniel Geng, Max Hamilton, and Andrew Owens. Comparing correspondences: Video prediction with correspondence wise losses. In CVPR, 2022.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac- Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Ofir Nachum, George Tucker, Nicolas Heess, and Nando de Freitas. RI unplugged: A suite of benchmarks for offine reinforcement learning. arXiv preprint arXiv: Arxiv-2006. 13888, 2020.
Agrim Gupta, Stephen Tian, Yunzhi Zhang. Jiajun Wu, Roberto Martin- and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 0180000//1994, 1894, 2022.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: Arxiv-1803.00 /0/22, 2018.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. arXiv preprint arXiv: Arxiv-2205. 1495, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Newral Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626-66377 2017.
Kevin Huang, Sahin Lale, Ugo Rosolia, Yuanyuan Shi, and Anima Anandkumar. Cem-gd: Cross- entropy method with gradient descent planner for model-based reinforcement learning. arXiv preprint 1 2021.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.8 6.m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1325--3335-339 jul 2014.
Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task rein- forcement learning benchmark. /tttps:///i com/ researche rowodeskk 2021.
Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry V. Dylov. Pytorch image qual- ity: Metrics for image quality assessment, 2022. URL
14818
Chengshu Li, Fei Xia, Roberto Martin-Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simu- lation for robot learning of everyday household tasks. In Conference Robot Learning, 8-11 November 202 1, London, UK, volume 164 Proceedings of Machine Learning Research, pp- 455-465. PMLR, 2021.
Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Math. Programming, 1989.
Cong Lu, Philip J. Ball, Tim J. Rudner, Jack Parker-Holder Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv: Mrxi-22266.047799 2022.

---

Published as a conference paper at ICLR 2023
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In arXiv: 1906.07889. 2019.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30- November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 1101-1112. PMLR, 2019.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder Singh. Action-conditions video prediction using deep networks in atari games. In Advances in Neural Information Process- ing Systems 28: Anmual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp.
2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, deep
Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, learning library. In Advances in Neural Information Processing Systems 32. 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Reuven Rubinstein.
The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127-190, 1999.
Adam N. Sanborn, Vikash K. Mansinghka, and Thomas L. Griffiths. Reconciling intuitive physics 1-437, 11-433,
2013. and newtonian mechanics for colliding objects. Psychological review, 120(2):411
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms. In Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37, pp. 843-852, 2015.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap. and Martin Ried- miller. Deepmind control suite. arXiv preprint arXiv: Arxiv-1801 (801600 2018
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint a1210:111..01717, 2018
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 81-91, 2019.
Vikram Voleti, Alexia Jolicoeur-Martineauu and Christopher Pal. Masked conditional video diffu- sion for prediction, generation, and interpolation. (NeurIPS) Advances in Neural Information Processing Systems, 2022. URL https: wwwpaaiwwwwww..//aaa/aaaaaaaaa/a 22853
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004. doi: 10.1109/TIP.2003.888998
Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. Ag- gressive driving with model predictive path integral control. In 2016 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1433-1440, 2016. doi: 10.1109/CR...016
7487277.
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Videogpt: Video generation using va-vae and transformers, 2021.
12

---

Published as a conference paper at ICLR 2023
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcemmmttteeaaaee learning. In Conference Robot Learning (CoRL), 2019. URL attpe///azxLV.ong//bb//LLL1
10897
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Advances in Neural Infor- mation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simu- lation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
13

---

[1, 1, All scaled by 1/4 Published as a conference paper at ICLR 2023
A MODEL TRAINING DETAILS
In this section, we provide specific training details for video prediction model training here,
A.1 FITVID
We reimplement Vit according the official implementation by the authors. We train using the architecture defaults from the original paper. The training hyperparameters are shown in Table 3 For the smaller variants shown the model capacity experiments, we detail the modifications in architecture in Table and S
We train Fit Vid at 64 64 image resolution to predict 10 future frames given future frames.
When training, we use FP16 precision using PyTorch's autocast functionality and use either 2 or 4 NVIDIA TITAN RTX GPUs. We train all models for 153K gradient steps.
Hyperparameter
Value
Batch size
32
Optimizer
Adam
Learning rate
3e-4
Adam e
1e-8
Gradient clip
100
le-4
Table 3: Hyperparameters for FitVid training. Note that we use a learning rate of 3e 4 because we find that it allows for more stable training. We also tried 1e - 3 as in the original paper, but we found that this tended to cause numerical instability and did not yield significantly different results.
Hyperparameter
Value
Encoder (h) dimension
64
LSTM size
128
Num of encoder/decoder layers per stage
[1, 1, 1, 1]
Table 4: Hyperparameters for FitVid model: smaller encoder.
Hyperparameter
Value
Encoder (h) dimension
32
LSTM size
64
Num of encoder/decoder layers per stage
Encoder/Decoderr # filters
Table 5: Hyperparameters for FitVid model: mini.
A.2 SVG
For SVG', we start with the implementation of the SVG model by Denton Fergus (2018) Then, we make the modifications described by VVllegaa et al. (2019). Specifically, we use the first blocks of VGG as the encoder/decoder architecture. Then, rather than flattening encoder outputs before feeding them into the LSTM layers, we use Convolutional LSTMs to directly process the 2D feature maps. Unless otherwise stated, we train with model size = 1, K 1, which the base model size described in Villegas et al. (2019). Note that for fairer comparisons, unlike Villegas et al. 2019), we retain the & loss for the reconstruction portion the loss function.
For action conditioning with SVG', we tile the action for each timestep into 2D feature map with the same dimensions as the encoded image, where the number of channels is the action size. We then concatenate this along with the latent to the encoded image before passing it into the frame predictor RNN.
When testing higher and lower capacity variants, we adjust the values of M and K, which as defined by Villegas et al. (2019), are hyperparameters that multiplicatively scale the number of filters in the L.T layers and encoder/decoder, respectively.
14

---

Published as a conference paper at ICLR 2023
We train SVG' at 64 image resolution to predict 10 future frames given 2 context frames. We use 1-2 NVIDIA TITAN RTX GPUs to train SVG' for 153K gradient steps. Training hyperparameters shown in Table 6
Hyperparameter
Value
Batch size
32
Optimizer
Adam
Learning rate
3e-4
Adam 31
0.9
Adam B3
0.999
Adam
1e-8
Table 6: Hyperparameters for SVG' training.
A.3 MCVD
We the code implementation for MCVD provided by the original authors https: // github. cwm/co//////ooooo http://leet.//cvvdyyyeeeee
We use the SPATIN version of the model, following the setup for training the Denoising Diffusion Probabilistic Models (DDPM) version of the model on SMMNIST in the original paper.
We additionally modify the architecture be action-condittooed by concatenating first converting sequence of actions into flat vector containing actions from all timesteps. We then tile this action vector a E RT|A|| into a 2D feature map with the same spatial dimensions as the context images, and concatenate the feature map and context images channel-wise. Additional training hyperparameters are shown in Table 7
Hyperparameter
Value
Batch size
64
Optimizer
Adam
Learning rate
2e-4
Adam $1
0.9
Adam B2
0.999
Adam
le-8
Gradient clip
1.0
Weight decay
0.0
# diffusion steps (train)
100
# diffusion steps (test)
100
Table 7: Hyperparameters for MCVD training.
We train MCVD to generate 64 64 RGB images. MCVD can used for number of different video tasks based on which frames are provided for conditioning, but we use it for video prediction (only future frame prediction provided context). Although during planning we use MCVD predict to horizon of length 10 like other models, we train the model to predict future frames given 2 context frames, following the procedure by original authors. In order to make predictions of length 10, feed in the results of the first prediction autoregressively to the model to obtain the last predictions. Note that only the actions for the first frames are provided for the first forward pass, and only the actions for the second 5 frames are provided in the second pass.
We train the MCVD model for robosuite tasks for 360K gradient steps, and the one for Ro- boDesk tasks for 270K gradient steps. Each model is trained on 2 NVIDIA TITAN RTX GPUs.
A.4 MASKVIT
We use the hyperparameters used by the authors when training the BAIR dataset, but with slightly increased positional embedding size (1024 rather than 768). We did not tune these parame- ters further.
15

---

Published as a conference paper at ICLR 2023
A.5 STRUCT- VRNN
We reimplement the Struet-VRNN model from Minderer et al. (0119) We tune the weighting of the KL-divergence parameter in the values le 0, value of le 4, that achieves the best control performance the RoboSuite tasks, for the rest of the experiments. use 64 keypoints the representation, matching the largest number used for any dataset in the original paper. Full hyperparameters shown in Table g
2, 3, le le-e-de 4, and use the
Hyperparameter
Value
Batch size
Optimizer
Adam
Learning rate
3e-4
Reconstruction loss weight
10.0
KI. loss weight
0.001
Coordinate prediction loss
1.0
# keypoints
64
Keypoint o
0.1
Encoder initial number of filters
32
Appearance encoder initial number of filters
32
Decoder initial # of filters Dynamics model hidden size Prior/posterior # of layers
256
512
2
Prior/posterior hidden size
512
Table 8: Hyperparameters for Struct-VRNN training.
B PLANNING IMPLEMENTATION DETAILS
In this section we provide details for the planning implementation Vp2. For all tasks, we use 15 the rollout length.
B.1 MPPI OPTIMIZER
We perform sampling-bast planning using the model-predictive path integral (MPPI) (Williams et al. 2016), using the implementation from Nagabandi et al. (2019) as a guideline. Table details the hyperparameters that we use for planning for each task category.
Hyperparameter
Value
Number of samples
200 for all tasks except 800
for open fdrawer, s slide)
Scaling factor
0.05
Sampling distribution correlation coefficient 3 Sampling distribution stdev. (RoboDesk)
0.5
[0.5,0.5,.0..0O110011 [0.5, 0.5, 0.5, 0] 0
Sample distribution stdev. (robosuite)
Sampling distribution initial mean
Table Hyperparameters for MPPI optimizer.
B.2 CLASSIFIER COST FUNCTIONS
For each classifier cost function, we take 2500 trajectories from the RoboDesk dataset for the given task, and train binary convolutional classifier to predict whether or not given frame receives reward >= 30, where the reward is provided by the RoboDesk environment. We train for 3000 gradient steps, and the architecture is described in Table 10
B.3 PLANNING COST FUNCTIONS
Next detail the cost functions for each of the tasks. The cost for the RoboSuite tasks is the sum of the & wixel-wise enror between the 10 predicted images and the goal image, summed over time. For the RoboDesk tasks, it is 0.5 times the &z pixel loss plus 10 times the classifier logit
16

---

Published as a conference paper at ICLR 2023
Layer type
Out channelsticdenn units
Kernel
Conv2D
32
x
Conv2D
32
3
RoLU
Conv2D
ReI.U
Conv2D
x
ReLU
Conv2D
32
3
ReLU
Cony2D
32
3 3
Conv2D
32
3
ReLU
Conv2D
32
3
Flatten
Lincar
1024
ReLU
Table 10: Classifier cost function architecture, with layers named in PyTorch convention. The input is 64 x 64 x RGB image.
from deep convolutional classifier trained for predicting success on that given task. We tuned this weighting value after fixing the value of for MPPI. We tuned over the classifier weight as [1, 10, 100, 1000, 10000) using the simulator as the planner and found that 10 resulted in the best performance.
The score for the planner computed by negating the cost.
C ALTERNATIVE PLANNERS
While we implement and tune the sampling- based MPPI optimizer for accessibility and ease of use, our code framework also enables benchmark users to modify and swap out the controller and model independently. This allows the coupled development of models and controllers on the task and task instance definitions provided in the benchmark. We provide the following optimizerssconttoollers with the released codebase. A star (*) indicates that the method requires gradient information.
baased
Model-predictive path integral (MPPI) (Williams et al. 2016): This is our default plan- ner as described the main text. Our implementation is based off that of Nagabandi et (2019).
Cross-entropy method (CEM) (de Boer et al. 2005): The cross-entropy method itera- tively refines the sampling distribution of candidate distributions via importance sampling. In practice, this is implemented by computing the top percentile of elite samples and computing the mean and variance at each iteration. Our implementation is based off that of Nagabandi al. (2019).
Cross-entropy method with Gradient Descent (CEM-GD) (Huang et al., 2021): This is gradient-augmentee version CEM that refines individual CEM samples using gradi- ent steps. It also significantly reduces the number of CEM samples after the first planning step for computational speed. We keep most of the Py Torch implementation by the original authors.
Limited-memond BFGS* (L-BFGS) (Liu & Nocedal, 1989): A popular quasi-Newtor method for continuous nonlinear optimization. We use the implementation from Py- Torch (Paskke et al. 2019).
We conducted initial control experiments with these planners using the SVG' model on the robosuite task categories. We find that CEM-GD promisingly achieves an 87% success rate averaged across 3 seeds, and L-BFGS achieves a 48% success rate across seeds after tuning the learning rate for the optimizer across (1e-3, le-2, 5e-2, 7e-2).
D DATASET DETAILS
Here we provide details abut the datasets that come with VP2. Note that our datasets can be re- rendered at any resolution.
17

---

Published as a conference paper at ICLR 2023
robosuite Tabletop environment: We collect 50K trajectories of interactions collected with scripted policy. During each trajectory, random one of the four possible objects is selected as the target for the push. Then, a random direction in [0, π the plane is selected as the direction for the object to be pushed, and the target object position set to 0.3 meters in this direction from the initial starting position. Then, first navigate the arm in position to push the object in the desired direction, and then to push it. At every step, add independent Gaussian noise with a 0.05 each dimension of the action except the last one, which represents the gripper action. Even if the object is not successfully pushed the desired position using this policy, we still record the trajectory.
P-controller to
The robosuite Tabletop dataset is rendered using the iGibson (Li et al. (2221) renderer, with modifications the shadow computations to make the shadows softer and more alistic. We will supply the modified rendering code along with the environments in the released benchmark code.
RoboDesk environment: We script policies to complete each of the tasks. The structure each policy depends on the task, and they are included the provided code. We apply noise to each action by adding independent Gaussian noise every dimension. We create the dataset for the entire RoboDesk dataset by collecting 2500 noisy scripted trajectories using noise level of 0.1, and then 2500 additional trajectories with 0.2, for total of 35K trajectories.
The RoboDesk dataset is rendered using the MuJoCo viewer, provided by the original implementation.
E ENSEMBLING EXPERIMENTS
For the ensembling experiment, we apply the model disagreement penalty in the following way. Given action sequence, first all N models in the ensemble predict video sequences 12, We then compute the error of the prediction that most deviates from the mean of all predictions in & error, i.e. 8 = max;=(0,1,..N) 1..1...N | N /'112. We compute the standard task cost function e using one of the ensemble predictions, selected uniformly at random. We calculate the final cost as c 25, where À is hyperparameter that we set as 0.01 for all experiments.
F CODE
We provide the open source code used for running control experiments here: htttp:: com/ s- iinn/p22 Pretrained cost weights, datasets, and task definitions can be downloaded from that link.
G FULL RESULTS FOR METRIC COMPARISONS
In Table we present the results on the remaining two tasks from the case study presented in Section
we provide detailed per-task plots of LPIPS, FVD, and SSIM compared to control performance. We find that while none of these metrics is well-correlated with performance across all tasks, they appear be better correlated in the "push blue button", "push green button"", and "open drawer" tasks. However, we can see that even for those tasks, these metrics can conflict in ordering models. For specific tasks, we observe that the "upright block off table" task particularly appears well-correlated with FVD.
In Figures 14
18

---

Published as a conference paper at ICLR 2023
Perceptual
Control
Perceptual
Control
Model
FVD
LPIPS*
SSIM Success
Model Loss
FVD
LPIPS*
SSIM Success
FitVid MSE
9.6
0.65
98.1
95%
FitVid MSE
10.6
0.65
98.0
88%
+LPIPS=1
0.72
+LPIPS=1
0.69
97.9
88%
1L
9.2
0.88
97.8
88%
+LPIPS=10 10
8.3
0.87
97.4
67%
SVG' MSE
16.7
1.13
95.3
97%
SVG MSE
13.1
1.13
94.9
83%
+LPIPS=1
8.4
1.06
95.5
97%
+LPIPS=1
7.4
1.03
95.3
83%
+LPIPS=10
94.1
25%
+LPIPS=10
24.6
1.26
93.8
(a) RoboDesk: push blue button
(b) RoboDesk: push green button
Perceptual
Control
Model Loss
FVD
LPIPS* SSIM Success
FitVid MSE
9.1
0.77
96.3
10%
+LPIPS=1
5.8
0.72
96.0
10%
+LPIPS=10
7.2
0.91
94.8
10%
SVG'
MSE
13.6
1.28
92.8
10%
+LPIPS=1
9.7
1.19
92.7
13%
+1PIPS=10
20.0
1.54
91.0
10%
(c) RoboDesk: flat block off table
Table 11: Results for the remaining 3 tasks for the experiment described in Section
Centrd Performmnce RebeSuit
PVD Performane RebeSuita Thels
3oodd
FxVil
SVG
SNC
129P5
SVG'
PHNdd
LPIPS Sml
0
20Vid
EnVVd
LMPPS
2h LPWPS-38
LIPP'=ii
03
0.
0.050
3/86 L2ISS
0
0,060
20
56
50
(compated taok dasimms)
(computed taok dasiane))
SSTM we Cunrral Peefermara en RobveSate Tude
1.0
SVG:
0.8
80
84
11115:10
5VC7
0.80
0.58
6.34
0.57
0.59
0.30
S534
(cuspuatdd back datasot)
Figure 7: Detailed results comparing perceptual metric values and control performance: RoboSuite tasks.
19

---

Published as a conference paper at ICLR 2023
LPIPS va Cuntrol Paffetmanee on BoboDeak push nol Tosk
FVD w. Conicol Performance ca ReboCook push red Taska
10
1.0
SVG LPIPS-1
SVIT
0.6
LPtPS
SVW
0.66
FAVal
SVG
LITPS_
0.00
0.6
3.6
PlVVid
0.
PaVid
LEW05-10
0.2
30odd 1s
SVC2
3ooad type
SVC
LPIPS-18
F3VM
LPIPS-00
5V0
0.0
5Y0
0.000
0.000
8,08
0.00)
(BLL
7.5
100
13.5
16.0
27.5
20.0
22.5
25.
LPIPS (comptied on leck datame))
FVD (computied on tadi detaad))
SSDM va Contrd Performaacee RoboDok push zed Tasks
type
PPP5-1
SVG
LIPPtiid FiVid
0.
0.8
LPU
LPCPS-10
0.548
0.945
0.000
0.00
01993
0.955
0,99
0.566
S5IM
(ommutted on task dodaset)
Figure 8: Detailed results comparing perceptual metric values and control performance: RoboDesk push red button task.
LPIPS va. Costrol Prefocmaace on RoboDook push blae Tisks
PVD v. Cnstrol Peefocuuare on RoboDoli push ble Tolks
12
sVG: SVC
Frivst TPPSS.)
PitVic
FiVVal 12:5-
SVC
LP1PS-18
Mode type FRVM 5007
SYC LinESsiity 11y
Mooe type
SVC
PaVH
0.
2.007
0.008
0,02)
0.000
0.015
6.032
0.003
20
13
22
23
33
33
LPIS (oompoted on bank datawet)
PVD (oomputed on bock datioot)
853M va. Cintrol Performmnce on RobeDeek push Muc Tasks
Evo
SVC'
PVVial
ERAVIC
877
800
80
64
SVG LMtiiett
88
Mookt FaVM
5V00
0.900
0.965
0.060
0.955
8:963
D045
170
0,890
55TM (rompated on takk Aasuoee
Figure 9: Detailed results comparing perceptual metric values and control performance: RoboDesk push blue button task.
20

---

Published as a conference paper at ICLR 2023
LPIPS ya. Canteal Paffetuance on RobuDask push guccs Tindo
FVD vo. Coniesl Perliemaace on RoboDokk pub grea Tol
10
neViil
10
[PIPS
SVCC [PIPS-]
SVG'
SV12
LPRP
5312
LPIPS-10
Fa-V
LJSP5-10
30dd 1p
3ouad type
SVG
13VH
F3VM
-
5VC
5Y0
0.007
0,000
0.010
0.011
6012
180
12.8
15.
27.8
20.0
20.6
26.
LPIPS (compuated on tack datlaed)
FVD (computeed taak dotanc))
S5IM vol Control P Perfemminc RotoDeak pruhh
Tasl
Leang
108
SVC
LPTPS-)
06
F'n Vil
SVG'
Modil
LPMPS-00
Untile SVC
0:34
6:36
0.96
0.50
0.58
SSIM (omnuuted on task dosazet)
Figure 10: Detailed results comparing perceptual metric values and control performance: RoboDesk push green button task.
LPIPS vs. Gootcol Pes focmannce on BoboClel uplight bookk off tokie Tisks
FVD va. Contaol Perforasncee on RebeClesk spright block table Tada
1.0
FRtVa L
FHVV
LPMPS-1
1.PIPS
2x
8x
SVG:
Vic
Mods type
LPPIP--11
Mooe type
LMMPS-110
PRVM 5007
FaVi
0.033
0.015 LPPS (rompsted on taok distaee)
0LE44
8:118
0.007
18
22
18
FVD (compoted taok datwwet)
SSIM Control Pezfrnntace on RobaDesk upriglt back off table Torks
10
LPIPS-10
Model PaVaa
122
5V7?
6:96
0.008
0.855
6.000
0.085
0.99)
0.555
880M
(oomguted tack darawet)
Figure 11: Detailed results comparing perceptual metric values and control performance: RoboDesk upright block off table task.
21

---

Published as a conference paper at ICLR 2023
LPMD8
Conrnol Performanes on BoboDook block Tidks
PVD Centeol Perfommaccs RoboQwek faxt blnck off table Tasks
1.0
3Goaa 5Y0
1.0
Mtadal
Falal
590
Fatkid
Svigt
SV11
0.8 Fasked
Plaiic
SVC
SVG'
1411---1
0.0
Fave
BAUD
BM12
6006
15
20
LPIPS (comptted on teek dalane))
PVD (oospputed on task datanos)
SSIM w. Cuntral Preformane on RoiiaDek flxt lock off bable Thalo
1.0
type
RUB
00
FlVVll
S0VE2
LPPPS-30
LP1P9-1
Syc
1.PP5..
0.91
0.52
3.00
6:36
0,05
0.30
SSIM (sogguted teck dataeet)
Figure 12: Detailed results comparing perceptual metric values and control performance: RoboDesk flat block off table task.
LPIPE v.. Costrol Pedfemtaae on RoboDokk opea. deower Tasks
PVD vs. Uonied Poffocmance on RobeDelk opea drower Tosko
1.0
3oddd
Modd
FXVV
SVG
SYG
1P
LPIPS-14
PHAVSt
PAYS
WPG
SVG'
LPPPss
LPP3-1
SVG'
0.0
0.903
2.004
Iv003 LPIS (romputed on taok distaee)
0.066
8,007
15
23
30
FVD (compoted task datswet)
SBD3 Conteol Performance RoboDesk drewex Tesks
Lype
1P1PS-UU STAVI
SVC @
SVC'
FIEV
0.065
100
16G
6970
1,040
SMM (comprtei tack darswnt)
Figure 13: Detailed results comparing perceptual metric values and control performance: RoboDesk open drawer task.
22

---

Published as a conference paper at ICLR 2023
LIEPSS vs. Conteol Peeficmaacco on BotoiJesi open side Tasks
10
FVD vs. Cootrd Performaacce co Robochekk open elide Toels
Modal rype
Midal
SVD
SVC
PROV LPIPS-1
FHWS LPIPS-1
Chatrll
PXVid
FaVa
FiiV
11225110
SV1'
LPPIS--1
[
3.0
0.0
0.12
0.004
6/16
B0114
(000)
0,022
0.004
10
12
28
18
22
LPOSS
(competed dastoer)
(ampured on took datson))
SSIM Cont Prefoemance BoboDok open sldee Tasks
type
AUS
LP129-1
LPMP5-02
PnVVil
SYG
0.99
000
8.30
6:00
S50M (oumputied on takk dostovet)
Figure 14: Detailed results comparing perceptual metric values and control performance: RoboDesk open slide task.
23


Reviews:
[{"review_id": "wKepTZmLLI", "reviewer": "Reviewer_LW5N", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new benchmark for evaluating video prediction approaches. The authors remark that the metrics used for evaluating video prediction quality, based on pixel accuracy, can be poor guides to the quality of the prediction. In some cases, an apparently accurate prediction may lead to failure in, eg, robotic planning. The authors propose to address this with a new benchmark evaluating video prediction from the perspective of 11 simulated robotic tasks, across two simulators. \nState-of-the-art video predictions methods are compared using those tasks. ", "summary_of_the_review": "This is an interesting paper, that provides a useful and timely resource to the community to possibly help provide a broader understanding on the quality and the limitations of video prediction approaches. Despite my concerns that the chosen robotic tasks are too limited in variety, this is clearly a step in the right direction and would be very useful to the field. ", "strength_and_weaknesses": "### Strengths\n- The rationale for the paper is very good, and it is addressing a relevant question: How to find better metrics for video prediction (or image generation in general) is an important question for the field, and robotics tasks are a good choice. \n- The availability of such a benchmark is likely to stimulate discussion and research in the video prediction field. \n- The experiments are well-designed and thorough. \n- The paper is generally clear and pleasant to read. \n\n### Weaknesses\n- The choice of tasks for the benchmark is of very limited variety. Out of the 11 tasks: \n\t- 4 are pushing objects of different (but convex) shapes\n\t- 3 are pushing buttons of different colour\n\t- 2 are pushing off blocks\n\t- and two are opening - admittedly the most interesting ones. \n\tThis is a limitation as it is not clear that most of those tasks are really the best chosen to evaluate motion prediction in videos, and could very well fail to measure large errors in the prediction. More varied manipulation (stacking?) would be interesting. \n- The number of methods evaluated in the benchmark is limited, although if the benchmark is provided to the community this is not a very crucial issue. \n- The analysis of the result is limited, and provides only limited insights in the validity of the initial hypothesis. It would have been good, for example, to have a plot of pixel accuracy vs simulation accuracy of the different methods to assess whether the classical metrics are good proxy or not. Also, which tasks out of the 11 are correlated to which would also be an interesting consideration. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jndZrKznpm5", "reviewer": "Reviewer_5Vyr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a benchmark for evaluating vision-based dynamics models. The paper shows that common prediction metrics (typically revolving around predicted image quality and not physical plausibility or causal interaction effects) may not necessarily correlate with control performance, and thus proposes an evaluation approach structured around a number of tasks in the robodesk and robsuite environments. For this the controller is standardised (MPPI) and abstracted away to allow SOTA chasers to focus directly on the dynamics modelling aspects of the task.", "summary_of_the_review": "I like the analysis of video prediction vs control, but am unconvinced that the proposed new benchmark is of significant value to advancing dynamics modelling for control. I think the benchmark needs more controller options (beyond sampling) and to allow gradient propagation to be really useful.\n\n----- Post rebuttal updates ----\n\nThe authors have added additional controllers to the benchmark, addressing the primary concern I had with this work.", "strength_and_weaknesses": "Strengths:\n\nI think the empirical results showing that perceptual video predictive quality metrics do not correlate well with control success is an interesting finding that will likely be of value to others.\n\nI agree that abstracting away control will result in significant performance increases in dynamics modelling and short term progress in this area.\n\nWeaknesses:\n\nThe proposed benchmark abstracts controller details from the dynamics model in the name of simplicity and ease of use, but I am not convinced that this is a good decision when it comes to longer term progress in our field. I also think the benchmark itself is vulnerable to a number of the criticisms this paper makes about evaluating video prediction in isolation of the underlying control tasks. I have a strong suspicion that the same article could be written about this framework, with results showing that different dynamics models perform differently with different control strategies and planners (eg. CEM, MPPI, trajectory optimisation etc.) and arguing that we need a new benchmark to avoid this. It's likely that this framework will result in a set of good dynamics models for use with MPPI, but not with other approaches.\n\nI am not convinced this will advance control performance, which may well benefit from more tight coupling between dynamics modelling and control rather than less. For this reason, I would be more inclined to use robosuite and robodesk directly, and value the ability to explore multiple control, planning and dynamics modelling approaches. I think this benchmark needs more controller options (beyond sampling based approaches) and to ideally enable the use of gradient based approaches that better leverage the learned dynamics models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i47-j5z2ri8", "reviewer": "Reviewer_wekY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a benchmark that evaluates video prediction models using the performance of downstream control tasks to complement traditional perceptual similarity or pixel-wise metrics. Specifically, the benchmark exposes a simple forward dynamics interface for the video prediction model to use it in a model-based planning method, visual foresight. The benchmark uses the robosuite tabletop environment and robodesk environment, where 11 robot manipulation tasks are designed in total. The paper evaluated three video prediction methods including variational RNN- and diffusion-based models.", "summary_of_the_review": "This paper addresses an important problem, i.e., improving generative video prediction evaluation with downstream control tasks. It proposes an easy-to-use benchmark for this setting. While there are several concerns such as the number of baselines, I currently slightly lean toward acceptance.", "strength_and_weaknesses": "**Strength:**\n\n- The paper studies an important problem, i.e., the evaluation of generative models, which is hard since there are no predefined metrics. The paper provides an example of a better evaluation of generative video prediction models using downstream control tasks.\n- The paper proposes a good case study that shows popular metrics such as LPIPS and MSE may not reflect the true performance of downstream control tasks, which justifies the need for the proposed method.\n- The benchmark features a simple plug-and-play interface that makes it easy to use and general for almost all video prediction methods.\n\n**Weakness:**\n\n- As a benchmark paper, the number of baselines is too few. Only three methods are evaluated while there are many more video prediction methods that could be evaluated given the simple and general interface.\n- The performance is reported with 2 runs only (in Figure 3), which might not be enough to accurately capture the performance and variance.\n- Only a specific planning method with specific hyperparameters is used for evaluation, which might not be indicative, since different video prediction (dynamics) models may require different algorithms or hyperparameters to perform the best.\n- The ensemble experiment is weak since it was only showcased for 3 tasks where the improvement is only significant for one task while the ensemble actually performs worse for another task.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dvrXgMYTt2", "reviewer": "Reviewer_fMXK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The papers presents a benchmark to evaluate if video prediction\nquality metrics correlates with control/ robotics manipulation.\nAs expected good perceptual quality of prediction does not\nnecessarily indicates better control capabilities.\n\nThey compare diverse sota methods on the new benchmark\nusing different model sizes, data quantity etc.", "summary_of_the_review": "My recommendation is for weak acceptance.\nI think the benchmark is useful and give new insights about planning methods.\nI am however, not so familiar with the literature and I might be wrong about\nthe significance of this proposal.", "strength_and_weaknesses": "\n### Strengths \n\n1. The paper guides the planning literature that maybe they shouldn't be using \nreconstruction and perception losses when predicting future moves.\n2. The paper is well written and the results are clear. \n\n\n### Weaknesses\n\n1. I feel the main weakness is the potential significance of the results\nis limited as I feel people tend to know that reconstruction is not ideal.\nI think the paper would be better if the authors actually found any \nmetric that seems to correlate more ? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster