Forward Prediction for Physical Reasoning
Authors: Rohit Girdhar
Laura Gustafson
Aaron B. Adcock
Laurens van der Maaten
Venue:  (2021)
Keywords: Forward prediction
physical reasoning

Abstract: Physical reasoning requires forward prediction: the ability to forecast what will happen next given some initial world state. We study the performance of state-of-the-art forward-prediction models in the complex physical-reasoning tasks of the PHYRE benchmark (Bakhtin et al., 2019). We do so by incorporating models that operate on object or pixel-based representations of the world into simple physical-reasoning agents. We find that forward-prediction models can improve physical-reasoning performance, particularly on complex tasks that involve many objects. However, we also find that these improvements are contingent on the test tasks being small variations of train tasks, and that generalization to completely new task templates is challenging. Surprisingly, we observe that forward predictors with better pixel accuracy do not necessarily lead to better physical-reasoning performance. Nevertheless, our best models set a new state-of-the-art on the PHYRE benchmark.

Under review as a conference paper at ICLR 2021
FORWARD PREDICTION FOR PHYSICAL REASONING
Anonymous authors
Paper under double-blind review
ABSTRACT
Physical reasoning requires forward prediction: the ability to forecast what will happen next given some initial world state. We study the performance of state- of-the-art
models in the complex physical-reasoning tasks of the PHYRE benchmark (Bakhtin al., 2019). We do so by incorporating models that operate on object or pixel-based representations of the world into simple physical-reasoning agents. We find that forward forward-prediction models can improve thyyicall-resooiigg performance, particularly on complex tasks that involve many objects. However, we also find that these improvements are contingent on the test tasks being small variations of train tasks, and that generalization to completely new task templates is challenging. Surprisingly, we observe that forward predictors with better pixel accuracy do not necessarily lead to better physical phasical-reasoning performance. Nevertheless, our best models set a new state-of-the-a th-tte- on the PHYRE benchmark.
INTRODUCTION
When presented with a picture of a Rube Goldberg machine, can predict how the machine works. We do so by using our intuitive understanding of concepts such as force, mass, energy, collisions, etc., to imagine how the machine state would evolve once released. This ability allows us solve real world ahysical-reasonin pocket, or how to balance the weight of two children abilities of machine-learning models have largely been limited to closed domains such as predicting dynamics of multi-body gravitational systems (Battaglia et al., 2016), stability of block towers (Lerer et al., 2016). or physical plausibility of observed dynamics (Riochet et al., 2018). In this work, we approaches to solve complex physical-reasoning
tasks, such as how to hit a billiards cue such that the ball ends up in the
a see-saw. In contrast, physical-reasoning
explore the use of imaginative, corward-prediction puzzles. We study modern object- based (Battaglia et al., 2016; Sanchez-Gonzale
et al., 2020; Watters et al., 2017) and pixel- based (Finn al., 2016; Ye et al., 2019; Hafner et al., 2020) forward-pre forward-prediction prediction models in simple saach-based agents on the PHYRE benchmark (Bakhtin et al., 2019). PHYRE 2D world, such that the world reaches a state with a particular property (e.g., two balls touching) after being played forward. PHYRE tasks are very challenging because small changes in the action (or the world) can have a very large effect on the efficacy of action; see Figure 1 for an example. Moreover, PHYRE tests models' ability to generalize to completely new physical environments at test time, a significantly harder task than prior work that mostly varies number or properties of objects in the same environment. As result, physical-reasoning agents may struggle even when their forward-prediction model works well.
tasks involve placing one or two balls
Nevertheless, our best agents substantially outperform the prior state-of-the-art on PHYRE. Specifi- cally, we find that forward-prediction models can improve the performance of physical -reasnnngg agents when the models are trained on tasks that are very similar to the tasks that need be solved at test time. However, we find forward preedd-peeiicioon based agents struggle to generalize to truly unseen tasks, presumably, because small deviations in forward predictions tend to compound over time. We also observe that better forward prediction does not always lead to better poysical-reasoning performance on PHYRE (c.f. Buesing et al. (2018) for similar observations in RL). In particular, we find that object-based forward- prediction models make more accurate forward predictions but pixel-based models are more helpful in physical reasoning. This observation may be the result of two key advantages of models using pixel-based state representations, First, it is easier determine whether a task is solved in a pixel-basee representation than in an object- based one, in fully observ- able 2D environments like PHYRE. Second, pixel-based models facilitate end-to-end training of the
forward-prediction

---

Under review as a conference paper at ICLR 2021
forward-predd predictionnnnnn model and the task-solution model in way that object-based models do not in the absence of a differentiable renderer (Liu et al., 2019; Loper & Black, 2014).
2 RELATED WORK
Our study builds on a large body of prior research on forward prediction and physical reasoning. We discuss most closely related work in this section and report additional prior work in Appendix B,
Forward prediction models attempt to predict the future state of objects in the world based on ob- servations of past states. Such models operate either on object-based (proprioceptive) representations or on pixel-based state representations.
popular class of object- based models use graph neural networks to model interactions between objects (Kipf et al., 20 Battaglia et al., 2016), for example, to simulate environments with thousands of particles (Sanchez-Gonzalee et al., 2020; Li et al., 2019). Another class of object-based models explicitly represents the Hamiltonian Lagrangian of the physical system (Greydanus ct al., 2019; Cranmer ct al., 2020; Chen et al., 2019). While promising, such models are currently limited simple point objects and physical systems that conserve energy. Hence, they cannot currently be used on PHYRE, which contains dissipative forces and extended objects. Modern pixel-based forward-prediction models extract state representations by applying a convolutional network on the observed frame(s) (Watters et al., 2017; Kipf et al., 2020) or on object segments (Ye et al., 2019; Janner et al., 2019). The models perform forward prediction on the resulting state representation using graph neural networks (Kipf et al., 2020; Ye et al., 2019; Li et al., 2020), recurrent neural networks (Xingjaan et al., 2015; Hochreiter & Schmidhuber, 1997; Cho et al., 2014; Finn et al., 2016), physics engine (Wu et al., 2017). 'hee models can be trained to predict object state (Watters et al., 2017), perform pixel reconstruction (Villegas et al., 2017; Ye et al., 2019), transform the previous frames (Ye et al., 2018; 2019; Finn et al., 2016), or produce contrastive state representation (Kipf et al., 2020; Hafner et al., 2020).
Physical reasoning tasks gauge a system's 's ability to intuitively reason about physical phenom- ena (Battaglia et al., 2013; Kubricht et al., 2017). Prior work has developed models that predict stable (Lerer et al., 2016; Groth et al., 2018; Li et al., 2016), predict whether physical phenomena are plausible (Riochet et al., 2018), describe or answer questions about physical systems (Yi et al., 2020; Rajani et al., 2020), perform counterfactual prediction in physical worlds (Baradel et al., 2020), predict effect of forces (Mottaghi et al., 2016; Wang et al., 2018), or solve physical puzzles/games (Allen et al., 2020; Bakhtin et al., 2019; Du & Narasimhan, 2019). Unlike other physical reasoning tasks, physical-puzzle benchmarks such as PHYRE (Bakhtin et al., 2019) and Tools (Allen et al., 2020) incorporate full physics simulator, and contain large set of physical environments to study generalization. This makes them particularly suitable for studying the effectiveness of forward prediction for physical reasoning, and we adopt the PHYRE benchmark in our study for that reason.
whether physical structures
Inferring object representations involve techniques like generative models and attention mecha- nisms to decompose scenes into objects (Eslami et al., 2016; Greff et al., 2019; Burgess et al., 2019; Engelcke et al., 2019). Many techniques also leverage the motion information for better decomposi- tion or to implicitly learn object dynamics (Kipf et al., 2020; van Steenkiste et al., 2018; Crawford & Pineau, 2020; Kosiorek et al., 2018). While relevant to our exploration of bixel-based methods as well, we leverage the simplicity of PHYRE visual world to extract object-like representations simply using connected component algorithm in our approaches (c.f. STN in Section 4. I). However, more sophisticated approaches could help further improve the performance, and would be especially useful for more visually complex and 3D environments.
3 PHYRE BENCHMARK
In PHYRE, each task consists of an initial state that is 256 x 256 image. Colors indicate object properties; for instance, black objects are static while gray objects are dynamic and neither are involved in the goal state. PHYRE defines two task tiers (B and 2B) that differ in their action space. An action involves placing one ball (in the B tier) two balls (in the 2B tier) in the image. Balls parameterized by their position and radius, which determine the ball's mass. An action solves the task if the blue purple object touches the green object (the goal state) for minimum of three seconds
2

---

Under review as a conference paper at ICLR 2021
Action
Rollout
Action
Rollout
Action
Rollout
(a) Task not solved
(b) Task solved
(c) Task not solved
Figure 1: PHYRE tasks require placing an object (the red ball) in the scene, such that when the simulation is rolled out, the blue and green objects touch for least three seconds. (a), the ball is too small and does not knock the green ball off the platform. In (b), the ball is larger and solves the task. In (c), the ball is placed slightly farther left, which results the task not being solved. Small variations in the selected action (or the scene) can have large effect on the efficacy of the action.
Forcasd predesdeeeeetassee rode
input Deprossntttioon
State encociens
Dynamics model:
State Dlcoorer: d'y
Taskonpkition mocil:
Obpectr
Yowe
Pie
Ypu
Fax
Trpe Poa
Ipee Poe
Selwon
Plast Pecorstruciinnnnio
Figure 2: We study models that take as input a set of initial state via an object-based or pixel-based representation (blue box). We input the representation into range of forward- prredddpreeiition models, which generally comprise an encoder (yellow box), dynamics model (green box), and decoder (gray box). We feed that output to a task-solution model (red box) that predicts whether the goal state is reached. At inference time, search over actions that alter the initial state by adding additional objects to the state. For each action (and corresponding initial state) we predict a task-solution probability; we then select the action most likely to solve the task.
when the simulation is rolled out. Figure illustrates the challenging nature of PHYRE tasks: small variations can change incorrect actions (Figure I(a) and (c)) into a correct solution (Figure (b))..
Each tier in PHYRE contains 25 task templates. A task template contains 100 tasks that are structurally similar but that differ in the initial positions of the objects. Performance on PHYRE is measured in two settings. The within-template setting defines a train-test split over tasks, such that training and test tasks can contain different instantiations of the same template. The cross-template setting splits across templates, such that training and test tasks never correspond to the same template. A PHYRE agent make multiple attempts at solving a task. The performance of the agent is measured by the area under the success curve (AUCCESS; (Bakhtin et al., 2019)), which ranges from to 100 and is higher when the agent needs fewer attempts to solve task. Performance averaged over 10 random splits of tasks or templates, In addition to AUCCESS, we also measure a (FPA) that does not consider whether an action solves task. We define FPA the percentage of pixels that match the ground-truth in 10-second rollout at frame per second (fps); we only consider pixels that correspond dynamic objects when computing prrwero-nre accuracy. Please refer to Appendix E for exact implementation details.
accuracy
4 METHODS
We develop physical-reasoning agents for PHYRE that use learned forward-prediction models in a search strategy to find actions that solve a task. The search strategy maximizes the score of a task-solution model that, given world state, predicts whether that state will lead to task solution. Figure illustrates how our forward preddiction and task-solution models are combined. We describe both types of models, well as the search strategy we use, separately below.
3

---

Under review as a conference paper at ICLR 2021
4.1 FORWARD-PREDICTO MODELS
At time I, a forward-prediction model F aims to predict the next state, ē241. of a physical system based on series of r past states of that system. IF consists of state encoder e, forward-dynamies model f, and a state decoder d. The past states ( ,Fee are first encoded into latent repre- sentations {z--7..... 2 using learned encoder e with parameters te. i.e., 2 e( 0e). The latent representations are then passed into the forward forward-dynamies model f with parameters fy: If (fx.-r..... Zt ) (= 2e]; :x]:0y) - ÊL+1. Finally, the predicted future latent representation is decoded using the decoder d with parameters O2: d($++1; 0u) - ft+1 We learn the model parame- ters θ = (0.. o, 0a) on large training set of observations of the system's 's dynamics. We experiment with
models that use either object-based or pixel-based state representations.
Object-based models. We experiment with two object-based forward-prediction models that capture interactions between objects: interaction networks (Battaglia et al., 2016) and transformers (Vaswani ct al., 2017). Both object-basee tuples that contain object type (ball, stick, etc.), location, size, color, and orientation. The models are trained by minimizing the mean squared error between the predicted and observed state.
forward-prediction models represent the system's 's state as a set of
Interaction networks (IN; Battaglia et al. (2016)) maintain a vector representation for each object in the system at time 1. Each vector captures information about the object's 'ss type, position, and velocity. A relationship is computed for each ordered pair of objects, designating the first object as the sender and the second as the receiver of the relation. The relation is characterized by the concatenation of the two objects' feature vectors and a one-hot encoding representing the sender object's 'ss attribute of static or dynamic. The dynamics model embeds the relations into "effects" per object using a multilayer perceptron (MLP). The effects exerted on an object are summed into a single effect per object. This aggregated effect is concatenated with the object's previous state vector, from a particular temporal offset, along with a placeholder for external effects, e.g. gravity, The result is passed through another MLP to predict velocity of the object. We use two interaction networks with different temporal offsets (Watters et al., 2017), and aggregate the results in an MLP to generate the final velocity prediction. The decoder then sums the object's 's predicted velocity with the previous position to obtain the new position of the object.
Transformers (Tx; Vaswani et al. (2017)) also maintain a representation per object: they encode each object's state using 2-layer MLP. In contrast to IN, the dynamics model f in Tx is Transformer that uses self-attention layers over the latent representation to predict the future state. We add sinusoidal temporal position encoding (Vaswani et al., 2017) of time to the features of each object. The resulting representation is fed into a Transformer encoder with 6 layers and 8 heads. The output representation is decoded using MLP and added to the previous state to obtain the future state prediction.
Pixel-based models. In contrast to bbject-based models, pixel-based forward predicttnreiitioon models do not assume direct access to the attribute values of the objects. Instead, they operate on images depicting the object configuration, and maintain a single, global world state that is extracted by an image encoder. Our image encoder is ResNet-18 18 network (He et al., 2016) that is clipped the res4 block. Objects in PHYRE can have seven different colors; hence, the input of the network consists of seven channels with binary values that indicate object presence, consistent with prior work (Bakhtin et al., 2019). The representations extracted from the past T frames are concatenated before being input into the two models we study.
Spatial transformer networks (STN; (Jaderberg et al., 2015)) split the input frame into segments by detecting objects (Ye et al., 2019), and then encode each object using the encoder e. Specifically, we use a simple connected components algorithm (Weaver, 1985) to split each frame channel into object segments. The dynamics model concatenates the object channels for the input frames, and predicts a rotation and translation for each channel corresponding to the last frame using a small convolutional network. The decoder applies the predicted transformation to each channel. The resulting channels are combined into single frame prediction by summing them. Inspired by modern keypoint localizers (He et al., 2017), we train STNS by minimizing the spatial cross- entropy, which sums the cross-entropies of H x W softmax predictions over all seven channels. Deconvolutional networks (Dec) directly predict the pixels in the next frame using a deconvo- lutional network that does not rely on segmentation of the input frame(s). The representations for the last 7 frames are concatenated along the channel dimension, and passed through a small convolutional network to generate latent representation for the + 1th frame. Latent represen-
4

---

Under review as a conference paper at ICLR 2021
tation 2++1 is then decoded to pixels using deconvolutional network, implemented series of five cunsposed-coovolutiin and (bilinear) upsampling layers, with intermediate ReLU activation functions. We found Decs are best trained by minimizing the per-pixel cross-entropy, which sums the cross-entropy of seven-way softmax predictions at each pixel.
4.2 PASK-SOLUTION MODELS
We use our forvard-prediction models in combination with a task-solution model that predicts whether rollout solves physical-reasoning task. In the physical-reasoning tasks we consider, the task-solution model needs to recognize whether two particular target objects are touching (task solved) or not (task not solved). This recognition is harder than it seems, particularly when using an object-based state representation. For example, evaluating whether or not the centers of two balls "near" is insufficient because the radiuses of the balls need to be considered well. For more complex objects, the model needs to evaluate complex relations between the two objects, as well as recognize other objects in the scene that may block contact. We note that good task-solution models may also correct for errors made by the prwaaddprediction model.
Per Figure we implement the task solution model using a simple binary classifier C with parameters db. The classifier receives the 7 +- (initial and predicted) frames and/or latent representations as input model. We provide the input frames as well to account for potentially
from the forward-prediction poor performance of the dynamics model on certain tasks; in those cases the task-solution model can learn to ignore the rollout and only use input frames make a prediction. then produces binary prediction: C(ro..... II.: 20. 2r, #1....... - 2..1.... 2 15) + [- 1, +1]. Because both types of prrwarinnrediction models produce different outputs, we experiment with object-based classifiers and pixel-based classifiers that make predictions based on simulation state represented by object features or pixels respectively. We also experiment with pixel-based classifiers on object- forward- -reeardtroeiiiioo models by rendering the object- based state pixels first.
Object-basee classifier (Tx-Cls). We use a Transformer (Vaswani et al., 2017) model that encodes the object type and position into 128- 28-dimensional encoding using two-layer MLP. As before, a sinusoidal temporal position encoding is added to each object's s features. The resulting encodings for all objects over the 7 7 time steps are concatenated, and used in 16-head, 8-layer transformer encoder with LayerNorm. The resulting representations are input into another MLP that performs binary classification that predicts whether not the task is solved.
Pixel-based classifier (Conv3D- (Latent, Pixel)), Our pixel-based classifier poses the prob- video-classffication problem. Specifically, we adopt a 3D
lem of classifying task solutions convolutional network for video classification (Tran et al., 2015; 2018; Carreira & Zisserman, the latent state rep-
2017). We experiment with two variants of this model: (1) CD- resentations (z, 9) are concatenated along the temporal dimension, and passed through a stack of 3D convolutions with intermediate ReLUs followed by a linear classifier; and (2) Conv --3--Piiee xel: the pixel representations (æ, .)) encoded using ResNet- 18 (up to res4), ), and classifications are made by the -eetv model. Conv3D-Pin --iixel can also be used in combination with object-based preaardprediitton models, the predictions of those models can be rendered.
4.3 SEARCH STRATEGY
We compose forward-predictee model FF and a task-solution model C to form scoring function for action proposals. An action adds one or more additional objects to the initial world state. We sample K actions uniformly at random and evaluate the value of the scoring function for the sampled actions. To evaluate the scoring function, we alter the initial state with the action, use the resulting state input into the forward -preadd-reedicioo model, and evaluate the task-solution model on the output of the forward- forward-peediction prediction model. The search strategy selects the action that is most likely to solve the task according to the task-solution model, based on the output of the forward preeicctto model.
EXPERIMENTS
We evaluate the performance of the forward-predictte models the B-tier of the challenging PHYRE benchmark (Bakhtin et al. (2019); see Figure 1). We present our experimental setup and the results of our experiments below. We provide trained models and code reproducing our results online.
5

---

Under review as a conference paper at ICLR 2021
Within-template
Cross-template
Within-template
Cross-template
70
AUCCESS
15.0
70
AUCCESS
60
22.5
54
66
67
CrendD giue 50 -8- Tech
625
66
2.5
5.0
160
12.5
500
7.5
10.0
12.5
Seconds (T'
Seconds (7')
Seconds (7')
Seconds (T)
Figure 3: AUCCESS of object-based and pixel-based (A) task-solution model (y-axis) ap- plied on state obtained by rolling out an oracle forward-prediction model for seconds (æ-axis). AUCCESS of the OP OPTIMAL agent is shown for reference. Shaded regions indicate standard devi- ation across 10 folds. We note that object- based task-solution models struggle compared to pixel- based ones, especially in cross-template settings.
Figure 4: AUCCESS of task-solution model ap- plied on state obtained by rolling out five object- based and pixel-based (A) forward-prediction models (y-axis) for r' seconds (z-axis). Forward- prediction models were initialized with T = 3 ground-truth states. AUCCESS of agent without forward prediction (No-Ewd) is shown for refer- ence. Results are presented for the within-template (left) and cross-template (right) settings.
5.1 EXPERIMENTAL SETUP
Training. To generate training data for our models, we sample task-action pairs in a balanced way: half of the samples solve the task and the other half do not. We generate training examples for the forward-predictte T consecutive frames used to bootstrap the forward model from a random starting point in this obtained rollout. The model is trained to predict frames that succeed the selected T frames. For the cusk-solution model, we always sample r frames from the starting point of the rollout, or frame O. Along with these frames, the task-solution model also gets the r' autoregressively predicted frames from the forward-prediction model as input. We use t for most experiments, and eventually relax this constraint to use 7 = 1 frame when comparing to the state-of-the-ard in the next section.
models by obtaining frames from the simulator at I fps, and sampling
We train most prrdiio-rediction models using teacher forcing (Williams & Zipser, 1989): we only use ground-truth states as input into the forward model during training. The only exception is Dec, for which we observed better performance when predicted states are used as input when training. Furthermore, since Dec is trainable without teacher forcing, we able to train it jointly with the task-solution model, as it no longer requires picking a random point in the rollout to train the forward model. In this case, we train both models from frame 0 of each simulation with equal weights both losses, and refer to this model as Dec [Joint]. ]. For object-based models, we add a small amount of Gaussian noise to object states during training to make the model robust (Battaglia lia et al., 2016). We train all task-solution and pixel-basee forward-pre -reedd-peeiittion models using batch SGD, and train object-based forward-predde -predd-preiitioon models with Adam. We selected hyperparameters for each model based on the AUCCESS on the first fold in the within-template setting; see Appendix I.
Evaluation. At inference time, we bootstrap the forward preeicctpreiiiion models with T initial ground- truth states from the simulator for a given action, and autoregressively predict 7' future states. The T + r' states are then passed into the task-solution model to predict whether the task will be solved or not by this action. Following (Bakhtin et al., 2019), we use the task-solution model to score fixed set of K 1, 000 (unless otherwise specified) randomly selected actions for each task. We rank these actions based on the task-solution model score to measure AUCCESS. We also measure forward predicctt accuracy (FPA; see Section 3) on the validation tasks for 10 random actions each, half of which solve the task and the other half that do not. Following (Bakhtin et al., 2019), we repeat all experiments for 10 folds and report the mean and standard deviation of the AUCCESS and FPA.
5.2 RESULTS
We organize our experimental results based on series of research questions.

---

Under review as a conference paper at ICLR 2021
Table 1: AUCCESS and success percentage
71.0
of our Dec [Joint] agents using r' 0 (no roll-out, frame-level model) and r' = 10 (full roll-out) compared to current state-of-the art agents (Bakhtin et al., 2019) on the PHYRE benchmark. In contrast to prior experiments, all agents here are conditioned on r = 1 initial frame. Our agents outperform all prior work on both set- tings and metrics of the PHYRE benchmark.
AUCCESS
21.3
FPA
96
49.0
-5
15 -.
10
96
50
Seconds
FPA
Figure 5: Left: Within-template forward- prediction accuracy (FPA) after T' seconds roll out of five forward prediccion models, and an "iden- tity" baseline that repeats the last input frame (dashed gray line). Right: Maximum AUCCESS value across roll-out as a function of forward-
AUCCESS
Success % agg
Within
Cross
Within
Cross
RAND (Bachrie et al, 2009
13.7
13.0a50
7.7=08
6,8
MEM (Bultria et 20199
2.4
18.5
2.7=05
15.2#59
DQN (Baltria 2019)
77.6=1.1
36.8
81,419
14.5
prediction aceuracy averaged over r' = 10 sec-
Ours (r' =
76.7_n9
40.7.12
80.7.11
40.1
onds for the five models. Shaded regions and error
Ours cr'
80.0
40.3
84.1
39.2186
bars indicate standard deviation over 10 folds.
Can perfect forward prediction solve PHYRE physical reasoning? We first evaluate if perfect forward- wwward-pediccton can solve physical reasoning on PHYRE. We do so by using the PHYRE simulator as the forward-prediction model, and applying task-solution models on the predicted state. We exclude the SantvD--Laeee
task-solution model in this experiment because it requires the latent representation of learndd forward -reeuddcdreiiioonnnoo model, which the simulator shows the AUCCESS of these models as a function of the number of seconds the forward- forward-prediction prediction model is rolled out. We compare model performance with that of the OPT IMAL agent (Bakhtin et al., 2019), which is an agent that achieves the maximum attainable performance given that we rank only K actions. We observe that task-solution models work nearly perfectly in the within-template setting when the forward-prediction is rolled out for 7' 2 10 seconds. We also observe that pixel-based task-solution models outperform object-based models, especially in the cross-template setting. This suggests that it is more difficult for object- object-based models to determine whether or not two objects are touching than for pixel- pixel-based models, presumably. because the computations required are more com- plex. In preliminary experiments, we found that Conv3DD-Latem outperforms Conv3D-P --00piieee xee when combined with learned pixel-based forward-prediett models (see Appendix D.1). Therefore, we use 3onvvD-Latem as the task-solution model for pixel-based models and Conv3D-Pixen xee for object-based models (by rendering object-based predictions) in the remaining experiments.
not provide. Figure 3
How well do forward-prediction models solve PHYRE physical reasoning? We evaluate per- formance of our learned forward-prediction models on the PHYRE tasks. Akin to the previous experiment, roll out the prediccion model for r' seconds and evaluate the corresponding task-solution model on the 7' state predictions and the 7 = input states. Figure 4 presents the function of the number of seconds (7') that the forward-prediction
AUCCESS of this approach models were rolled out. The AUCCESS of an agent without forward prediction (No-fwd) is shown for reference. The results show that forward prediction can improve AUCCESS by up to 2% in the within-template setting. The pixel-based Dec model performs similarly to models that operate on object-based states, either extracted (STN) or ground truth (IN and Tx). Furthermore, Dec allows for end-to-end training of the forward-predictio resulting Dec [Joint] model performs the best in our experiments, which is why we focus on it in subsequent experiments. Similar joint training of object-based models (c.f. Appendix D.2) yields smaller improvements due to limitations of object-based task-solution models. Although the
and the pixel-based task-solltion models. The
results suggest that forward prediction can help physical reasoning, AUCCESS plateaus after r' 25 5 seconds. This suggests that -reeirtcrreiiiion models are only truly accurate on PHYRE for a short period of time. Also, forward- prediction models help little in the cross-template setting, suggesting limited generalization across templates.
Does better sewward-prediction imply better PHYRE physical reasoning? Figure 5 (left) mea- sures the forward predditirriiiionnn accuracy (FPA) of our forward prediccion models after 7' seconds of rolling out the models. We observe that FPA generally decreases with roll-out time although,
7

---

Under review as a conference paper at ICLR 2021
poeat arcende frestt artende
AUCCESS
AUCCESS
A AUCCESS
Templates
Objects
# Objects
Figure 6: Left: Per-template AUCCESS of Dec [Joint] 1f with r' = 0 (no forward prediction) and 7' 10 (forward prediction) of five task templates that benefit the least from forward prediction (left) and five templates that benefit the most (right). Right: Per-template AUCCESS of the T' = 0 model as a function of the number of objects in the task template (left) and improvement per- template AUCCESS, called A AUCCESS, obtained by the r' = 10 model (right).
interestingly, Dec recovers over time (c.f. Appendix C). While all models obtain a fairly high FPA, models that utilize object-centric representations (IN, Tx, and STN) clearly outperform their pixel- based counterparts. This is intriguing because, in prior experiments, Dec models performed best on PHYRE in terms of AUCCESS. To investigate this in more detail, Figure 5 (right) shows FPA averaged over 10 seconds as a function of the maximum AUCCESS over that time. The results confirm that more pixel-accurate forward predictions do not necessarily increase performance on PHYRE's
tasks.
How do borward-predicttee agents compare to the state-of-the-art on PHYRE? Hitherto, all our experiments assumed access to 3 input frames, which is not the setting considered by Bakhtin al. (2019). To facilitate comparisons with prior work, we develop an Dec agent that requires only input frame: we pad the first frame with 2 empty frames and train the model exclusively on roll-outs that start from the first frame and do not use teacher forcing. We refer to the resulting model as Dec [Joint] 1 1f. Table compares the performance of Dec [Joint] 1f to the state-of- the-art on PHYRE in terms AUCCESS and success percentage 10 (i.e., the percentage of tasks that were solved within 10 attempts). The results show that Dec [Joint] 1f outperforms the previous best reported agents in terms of metrics in both the within and the cross-template settings. In the within-template setting, the performance of Dec [Joint] 1 increases substantially for large modeling approach to PHYRE in that
T'. This demonstrates the benefits of using forward-predictin setting. Having said that, forward-prediction did not help in the cross-template setting, presumably, because rollouts on unseen templates, while realistic, were not accurate enough to solve the tasks.
Which PHYRE templates benefit from using forward-predictiin model? To investigate this, we compare Dec [Joint] 1f at r 0 (i.e., no crwwardpreediction and t = 10 seconds in terms of per-template AUCCESS. We define per-template AUCCESS as the average AUCCESS over all tasks in template in the within-template setting. Figure 6 (left) shows the per-template AUCCESS for the five templates in which forward- preediciieeiiioon models help the least (left five groups) and the five templates in which these models help the most (right five). Qualitatively, prediction does not help much in "simple" tasks that comprise a few objects, whereas it helps a lot in more "complex" tasks. This is corroborated by the results in Figure 6 (right), in which show AUCCESS and the improvement in AUCCESS due to forward modeling (A AUCCESS) as a function of the number of objects in the task. We observe that AUCCESS decreases (p= -0.4) with the number of objects in the task, but that the benefits of forward predictions increase (p=0.6).
observe that forward
6 DISCUSSION
While the results of our experiments demonstrate the potential of forward prediction for physical reasoning, they also highlight that much work is still needed for the full potential to materialize. The main challenge is that physical environments such as PHYRE are inherently chaotic: a small change in an action (or scene) may drastically affect the action's 's efficacy. Figure 7 shows example of this: our models produce realistic forward predictions (also see Appendix A), but they may still select incorrect actions. Furthermore, PHYRE expects single model to learn all templates and generalize across templates, exacerbating this challenge. Notably, recent work on particle- particle-based representations (Li et al., 2019; Sanchez-Gonzalez et al., 2020) has shown successful initial results
8

---

Under review as a conference paper at ICLR 2021
Iepor
Similtte
Ingur
lepert
Action (red ball) solving the task.
Using slightly smaller ball.
Using slightly larger ball.
Figure 7: Rollouts for three slightly different actions on the same task by: (1) the simulator and (2) a STN trained only on tasks from the corresponding task template. Although the STN produces realistic rollouts, its predictions do not perfectly match the simulator. The small variations in action change whether the action solves the task. The STN model is unable to capture those variations effectively.
in terms of generalization by limiting the set of object types to only one (viz. particles). However, such approaches yet to be studied in extreme generalization settings as expected in PHYRE, and are constrained in accuracy by the underlying particle representation of extended objects, i.e. pixel level decomposition would be most accurate though computationally infeasible. Nevertheless, we believe this is an interesting direction and we plan to study it in the context of PHYRE in future work. Furthermore, much of our analysis is specific 2D environments, used in the physical challenge benchmarks like TOOLS (Allen et al., 2020) and PHYRE. Extending this analysis to 3D or partially observable environments would also be interesting future work, where object-basee models may have an advantage over pixel-based. Finally, experimenting with more sophisticated scene-decomposition approaches (Kosiorek et al., 2018; Crawford Pineau, 2020; van Steenkiste al., 2018), can allow for better joint training of object-centric approaches and further improve performance on the PHYRE tasks.
REFERENCES
Kelsey R Allen, Kevin Smith, and Joshua B Tenenbaum. The Tools challenge: Rapid trial-and-error learning in physical problem solving. In CogSci, 2020.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. PHYRE: A new benchmark for physical reasoning. In NeurlPS, 2019.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counterfac- tual learning of physical dynamics. In ICLR, 2020.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. NeurlPS, 2016.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation an engine of physical scene understanding. PNAS, 2013.
Lars Buesing, Theophane Weber, Sébastien Racaniere, SM Eslami, Danilo Rezende, David P Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, and Daan Wierstra. Learning and querying fast generative models for reinforcement learning, arXiv preprint arXiv:1802.08 030000000000..0.. 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick. and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: /3900, 2019.
Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In ICRA, 2017.
Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In CVPR, 2017.
Michael Chang, Tomer Ullman, Antonio Torralba, and Joshua Tenenbaum. A compositional object- based approach to learning physical dynamics. In ICLR, 2017.
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. arXiv preprint arXiv: /909. 13334, 2019.

---

Under review as a conference paper at ICLR 2021
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Damitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder encoder-decoder statistical machine translation. In EMNLP, 2014.
decoder for
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS, 2018.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR Workshop, 2020.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. In AAAI, 2020.
Yilun Du and Karthik Narasimhan. Task-agnostic dynamics priors for deep reinforcement learning. In JCML, 2019.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2019.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. NeurIPS, 2016.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NeurlPS, 2016.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physies for playing billiards. ICLR, 2016.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In NeurlPS, 2019.
Oliver Groth, Fabian Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. In ECCV, 2018.
Radek Grzeszezuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural network emulation and control of physies- baseed models. In Computer Graphics and Interactive Techniques, 1998.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurlPS, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Georgia Gkioxari. Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NeurIPS, 2015.
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. In ICLR, 2019.
10

---

Under review as a conference paper at ICLR 2021
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In JCML, 2018.
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In ICLR, 2020.
Adam Roman Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. In NeurlPS, 2018.
James Kubricht, Keith Holyoak, and Hongjing Lu. Intuitive physies: Current research and controversies. Trends in cognitive sciences, 2017.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In ICML,, 2016.
Wenbin Li, Seyedmajid Azimi, Aleš Leonardis, and Mario Fritz. To fall or not to fall: A visual approach physical stability prediction. arXiv preprint vmXiv:160..00060 2016.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICI.R, 2019.
Yunzhu Li. Toru Lin, Kexin Yi, Daniel Bear, Daniel L. K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In ICML, 2020.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3D reasoning. In ICCV, 2019.
Matthew M. Loper and Michael J. Black. OpenDR: An approximate differentiable renderer. In ECCV, 2014.
Roozbeh Mottaghi, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi. "what happens if..." learning to predict the effect of forces in images. In ECCV, 2016.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. In NeurIPS, 2018.
Nazneen Fatema Rajani, Rui Zhang, Yi Chern Tan, Stephan Zheng, Jeremy Weiss, Aadit Vyas, Abhijit Gupta, Caiming Xiong, Richard Socher, and Dragomir Radev. ESPRIT: Explaining Solutions to Physical Reasoning Tasks. ACL, 2020.
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. IntPhys: A framework and benchmark for visual intuitive physics reasoning, arXiv preprint arXiv: 07616, 2018.
Reuven Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 1997.
Alvaro Sanzhez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning to simulate complex physics with graph networks. In ICML, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: physics engine for model-based control. In IROS, 2012.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look spatiotemporal convolutions for action recognition. In CVPR, 2018.
11

---

Under review as a conference paper at ICLR 2021
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. JCLR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-basee reinforcement learning. In CoRL, 2020.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In JCML, 2017.
Zhihua Wang, Stefano Rosa, Bo Yang, Sen Wang, Niki Trigoni, and Andrew Markham. 3d-physnet: Learning the intuitive physics of non-rigid object deformations. In IJCAI, 2018.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. In NeurlPS, 2017.
James R Weaver. Centrosymmetric (cross-symment matrices, their basic properties, eigenvalues, and eigenvectors. The American Mathematical Monthly, 1985.
Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1989.
Jiajun Wu, Erika Lu, Pushmeet Kohli, William T Freeman, and Joshua B Tenenbaum. Learning to see physics via visual de-animation. NeurIPS, 2017.
SHI Xingjian, Zhourong Chen, Hao Wang, Dan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NeurlPS, 2015.
Tian Ye. Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In ECCV, 2018
Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. In ICCV, 2019.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. CLEVRER: Collision events for video representation and reasoning. In ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2021
A ROLLOUT VISUALIZATIONS
We first show rollouts when model is trained train tasks from single template, and evaluated from the val tasks on that template. We chose one of the hardest templates in terms of FPA, Template 18. This evaluation comparable most prior work (Battaglia et al., 2016; Sanchez-Gonzales et al., 2020), where variations of single environment are used during training and testing. Note that our model's 's rollouts are quite high fidelity in this case, as expected.
Within- Template 18 (fold 0): wtw.pyy.....eeiiii..e.eee us us-east- 2. amazonaws -a--eaasttammmzaaasss..... com/ Within-templonnly
However, PHYRE requires training a single model for all the templates in the dataset. We now show comprehensive visualization of rollouts for all our forward -preddiciniiioonn models (as were used to compute forward-prediction accuracy), for both within and cross-template settings, Note that the rollout fidelity drops, understandably as the model needs to capture all the different templates.
Within-Template (fold 0):
www.wwbiteeee...eeeeeeebb.bbmee. -s--ust-eeaaee amazonaws www.w//./oo..oo. whto///tti///////mm/
Cross-Template (fold 0):
http://Ewd-pss prred phyre. s3 website. ust.iteeeuuueeuee. us east 2, amazonaws com/cross/
B ADDITIONAL RELATED WORK
Due to limited space in the main paper, we discuss some additional related work in this section.
Additional work in Forward Prediction. Learning of forward predictive models, including using neural networks, has a long history (Grzeszczuk et al., 1998; Byravan & Fox, 2017). As discussed in the main paper, many recent approaches tend factorize input into object particle level representations, and learns compositional models for forward prediction. Additional work in that direction includes Fragkiadaki al. (2016); Chang et al. (2017); Mrowca al. (2018); Veerapaneni et al. (2020).
Video Prediction. Also related is prior work in conditional pixel generation or video prediction, as that typically requires an implicit understanding of physics. Popular approaches model the past frames using variant of recurrent neural network (Xingjian et al., 2015; Hochreiter & Schmidhuber, 1997; Cho et al., 2014) and make predictions directly using a decoder (Villegas et al., 2017), or as transformation of the previous frames using optical flow (Ye et al., 2018) or spatial transformations (Ye et al., 2019; Finn et al., 2016). Our work complementary such approaches, building upon them to solve physical reasoning tasks.
Model-based RL. This class of RL approaches rely building models of the environment of the agent to plan in. Such approaches typically use recurrent stochastic state transition models supervised with reconstruction (Hafner et al., 2019; Ha & Schmidhuber, 2018; Hafner al., 2020; Janner et al., 2019) or contrastive (Hafner et al., 2020) objective. Given the learned forward model, the planning is typically performed using variant of cross entropy method (CEM) (Rubinstein, 1997; Chua et al., 2018). Our setup is similar to model-based RI., with the crucial difference being that we take only a single action, and the long-horizon dynamics we need model significantly more complex than typical RL control environments (Tassa et al., 2018; Todorov et al., 2012). Given the simplicity of our action space, we learn value function over actions using the predicted rollouts, and use it to search for the optimal action at test time. Future work involving more complex or even continuous action spaces can perhaps benefit from learning more sophisticated sampling approach using CEM.
C ROLLOUT ACCURACY IN CROSS- TEMSLATE SETTING
Similar to Figure in the main paper, we show the forward- preddictoeiiioon accuracy on the cross-template setting in Figure 8. As expected, the accuracy is generally lower in the cross-template setting, showing that the models struggle to generalize beyond training templates. Otherwise, we see similar trends as seen in Figure 5.
13

---

Under review as a conference paper at ICLR 2021
It is interesting to note that Dec accuracy goes down and then up. similar to observed in the within teatintt case. We find that it is likely because Dec better able to predict the final position of the objects than the actual path the objects would take. Since it tends smear out the object pixels when not confident of its position, the model ends up with lower accuracy during the middle part of the rollout.
98
STN
IN
50
Dec
Tx
Dec [Joint]
96
AUCCESS
45
FPA
94
40
92
35
30
10
12
92
94
96
Seconds
FPA
Figure 8: Left: Forward-prediction accuracy (FPA; y-axis) after r' seconds (æ-axis) rolling out five norward-predicte models. Right: Maximum AUCCESS value across roll-out (y-axis) as function of forward-pp prerrd-prediction accuracy averaged over T' 10 seconds (æ-axis) for five forward-pres preaad-predictton models. Shaded regions and error bars indicate standard deviation over 10 folds. Both shown for cross-template setting.
D OTHER MODELS
MODELS ON LEARNED FORWARD-PREDDICTO
D.I CONY3D-LATENT VS CONv3D-PIXEL. ON PIXEL-BASE BASSD MODELS
In Figure 9, we compare models. We find Conv3D Latent generally performs better, especially for Dec, since that model does not produce accurate future predictions in terms of pixel accuracy (FPA). However, the latent space for that model still contains useful information, which the Conv3D-Lat successfully. Hence for pixel-based forward- preerrd-reeiition models, given the option between latent or pixel space task-solution classifiers, we choose Conv3D-Latent for experiments in the paper.
and Conv3D Pixel on learned pixel-based forward
able exploit
D.2 Tx-CLS VS CONv3D-PIXEL. ON OBJECT- SREECT-BASED BHOWARD-PREDICTURRE MODELS
In Figure 10, compare the object-based task-solution model learned object-based forward- prediction models. Similar to the observations with GT simulator in Figure 3 (main paper), object-based task-solution model (Ix-Cls) performed worse than its pixel-based counterpart (Conv3D-P) ixel), even with learned forward-predictte models. Jointly training the object-based forward models with object-based task-solution model improves performance, as shown for the IN model, however it is still worse than using a pixel-based task-solution model on the object- based forward model. Hence, for experiments in the paper, we render the object-based models' predictions to pixels, and use pixel-based task-solution model (Conv3D (Conv3D-Pixel) Note that the other pixel-basee task-solution model, Conv3D-La ent, not applicable here, as object- object-based based forward- prediction forward-prediction models do not produce spatial latent representation which Conv3D-Lats ent operates on. Moreover, training object-based models jointly with the pixel-based classifier is not possible in the absence of differentiable renderer.
14

---

Under review as a conference paper at ICLR 2021
70
STN w/ Conv3D-L Lttent
STN Conv3D--Pixc Pixel
69
Dec w/ Conv3D-Latent
69
Dec w/ 3DDPixxl
AUCCESS
AUCCESS
688
67
66
w/ Conv3D-Pixel
65
Tx w/ Conv3D-Piss Pixieel
IN Tx-Cls
64
TX w/ Tx-Thh Cls
IN w/ Tx-Cis [Jinnt]
2
6
10
12
2
3
5
Scconds (T')
Seconds (r)
Figure 9: Comparison of Conv3D- [Latent, Pixel] classifiers on learned pixel-based forward- peward-prediction models: Dec and STN, in the within- temilat setting. Conv3D-Latent per- forms well or better than Conv3D-Pixel and hence we use it for the experiments in the paper.
Figure 10: Comparison of Tx-Cls and Conv3D-Pixel classifiers on learned object- based forward-prediction models: Tx, in the within-template setting. Since Conv3D-P Piie- generally performed better, we use it for the experiments shown in the paper.
and
E FORWARD PREDICTION ACCURACY (FPA) METRIC
To compute FPA, we first zero all pixels with colors corresponding non-moving objects, in both GT and prediction. This ensures that any motion of non moving objects over other non moving objects background will incur no reduction in the FPA score. Then, FPA for a frame of the rollout is defined as the percentage of pixels that match between GT and prediction. Hence, if any object of colors corresponding to moving objects (red, green, blue, gray) an incorrect position (either overlapping with static non static objects/background) it would incur reduction in FPA. A follows:
python-style code for the computation
def zero_ non moving_channne (img):
is_red np. isclose (img, RED)
is_green np. ipp.sslose (img, GREEN)
is_bluc np isclose (img, BLUE)
is_gray np. isclose (img, GRAY)
is_l_red = np. isclose (img, L_RED)
img[-(ia_red) | is_green is_blue is_gray | is_1_red) 0.
img
10
def Epa (prediction, gt): ***
12
13
prediction: predicted frame of dimensions (H, W, 3)
14
gt: Correspanding GT frame of dimensions (H, W, 3)
15
16
prediction zero_out_ lecoo_evinn__hhaaeee (prediction)
17
gt
18
is is_close_per_chang
np.
gt)
19
all Ail_channels_clos Srreicttion.hhpee111111 shape
callss_per_channee..or
20
axis=-1) = prediction. srame_size gt shape [0] shape [1]
np. (al1 (all_channel __mnnelsscccc)e frame_sis size
15

---

Under review as a conference paper at ICLR 2021
DECJ FWDO
70
DECJ
70
69
AUCCESS
AUCCESS
60
68
50
67
40
2
3
4
10a
Seconds (r')
#actions
Figure 11: Effect of setting forward predic- tion loss to in Dec [Joint]. The perfor- mance is stagnant with the rollout if loss the future prediction is set to O, as expected. The model performs comparably model without forward prediction.
Figure 12: Performance of Dec [Joint] 1f at different number of actions. The per- formance varies nearly linearly with number of actions ranked.
F JOINT MODEL WITH ONLY DASK-SOLUTION LOSS
For our best joint model, Dec [Joint], J, we evaluate the effect of setting the forward prediction loss O. As seen Figure 1, the model performs about the same would without forward model, obtaining similar performance at different number of rollout seconds (r'). This further strengthens the claim that forward prediction leads the improvements in performance (as also evident from the increase in AUCCESS on increasing r'), as opposed to any changes in parameters or training dynamics.
G PERFORMANCE WITH DIFFERENT NUMBER OF ACTIONS RANKED
Similar to Figure 4 in Bakhtin al. (2019), analyze the performance of our best model, Dec [Joint] 1f, at different number of actions being re-ranked at test time, in Figure 12. We find the performance varies nearly linearly with number of actions upto 10K actions, similar to the observations Bakhtin et al. (2019).
H TEMPLATES RANKED BY FPA
Figure 13 shows the easiest and hardest templates for each forward model. We find some of the hardest ones are indeed the ones that humans would also find hard, such as the template involving see-saw system or complex extended objects like cups.
I HYPERPARAMETERS
All experiments were performed using upto V100 32GB GPUs. Depending on the number of steps the model was rolled out for during training. the actual GPU requirements were adjusted. The training time for all netward-predictton models around 2 days, and the task solution models took upto days (depending on how far the forward preddcinno model was rolled out). Our code will be made available to reproduce our results.
16

---

Under review as a conference paper at ICLR 2021
(+'=10)
IN
75
70
100
3f (r')00)
85
80
14
70
100
STN (r==10)
95
90
85
STN
100
Dec (e'ml0)
95
Dec
100
Dec [Joint] 35 (r'=10)
Dec [Joint]
Figure 13: Easiest and hardest templates terms of FPA.
I.1 FORWARD-PREDICTION MODELS
train all object-based models with teacher forcing. We use a batch size per GPU over gpus. For each batch element, sample clips of length from the rollout, using the first three as context frames and the 4th as the ground truth prediction frame. We train for 200K iterations. We add Gaussian noise sampled from / 0.01 (4144)) distribution to the training data, similar to (Battaglia et al., 2016). We add noise to 20% of the data for the first 2.5% training, decreasing the percentage of data that noisy to 0% over the next 10% of training. The object based forward models only make
17

---

Under review as a conference paper at ICLR 2021
predictions for dynamic objects, and use hard tanh to clip the predicted state values between 0-1. The models don't 't use the state of static objects or the angle of ball objects when calculating the loss. For angles, we compute the mean squared enror between the cosine of the predicted and ground truth angle. We now describe the other hyperparameters specific to each object- and pixel-based model.
IN (object-based); We train these models using Adam and a learning rate of 0.001. We use two interaction nets, one that makes predictions based on the last two context frames, the other that makes predictions based on the first two context frames. Using the same architecture as (Battaglia et al., 2016), the relation encoder is a five layer MIP, with hidden size 100 and ReLU activation, that embeds the relation into a dimension 50 vector. The aggregated and external effects are passed through a three layer MLP with hidden size 150 and ReLU activation. Each interaction net makes a velocity prediction for the object, and the results are concatenated with the object's 's previous state and passed to a three layer MLP with hidden size 64 and ReLU activation to make the final velocity prediction per object. The predicted state is a sum of the velocity and the object's previous state. Tx (object-based); We train these models using Adam and a learning rate of 0.0001. We use two layer MI. with a hidden size of 128 and ReLUU activation to embed the objects into a 128 dimensional vector. sinusoidal temporal position encoding (Vaswani et al., 2017) of time is added to the features of each object. The result is passed to a Transformer encoder with 8 heads and 6 layers. The embeddings corresponding to the last time step are passed to the final three layer MLP with ReLU activations and hidden size of 100 to make the final prediction. The model predicts the velocity of the object, which is summed with the object's last state to get a new state prediction.
STN (pixel-based): We also train these models with teacher forcing. Since pixel-based models involve a ResNet-18 18 image encoder, we use a batch size of 2 per GPU, over 8 GPUs. For each batch element, we sample clips of length 16 from the rollout, and construct all possible sets with context frames and the 4th ground truth prediction frame. The models were trained using learning rate of 0.00005, adam optimizer, with cosine annealing over 100K iterations. The scene was split into objects using the connected components algorithm, and we split each color channel into upto 2 objects. The model then predicts rotation and transformation for each object channel, which are used to construct an affine transformation matrix. The last context frame is transformed using this affine matrix to generate the predicted frame, which is passed through the image encoder to get the latent representation for the predicted frame (i.e. for STN, the future frame is predicted before the future latent representation).
Dec (pixel-based): For these models, we do not use teacher forcing, and use the last predicted states to predict future states at training time. We use batch size of 2 per gpu over 8 gpus. For each batch element, we sample a 20-Iength clip from the simulator rollout, and train the model to predict upto 10 steps into the future (note with teacher forcing, models trained only predict 1 step into the future given 3 GT states). The model is trained for 50K iterations with a learning rate of 0.01 and SGD+Momentum optimizer.
For this model, we train both forward-prediction and task-solution models jointly, with equally weighted losses. For this, we sample 13-length rollout, always starting from frame O. Instead of considering all possible starting points from the 13 states (as in Dec and STN), we only use the first 3 states bootstrap, and roll out for upto 10 steps into the future. We only incur forward-prediction losses for upto the first 5 of those 10 steps, we observed instability in training on predicting for all steps. Here we use batch size of 8/gpu, over gpus. The model is trained with learning rate of 0.0125 with MGD+Mome for 150K iterations. The task-solution model used is - tt, which operates on the latent representation being learned by the forward- preeird-reeiction model.
Dec [Joint]
I.2 TASK-SOLUTION MODELS
For all these models, we always sample T = 3 frames from the start point of each simulation, and roll it out for different number of 7' states autoregressively, before passing through one of these task-solution models.
Tx-Cis (for bbject- We train Transformer encoder model the object states predicted by baject-based forward geward-prediction models. The object states are first encoded using a two layer MLP with ReI.U activation into an embedding size of 128. A sinusoidal temporal position
18

---

Under review as a conference paper at ICLR 2021
encoding (Vaswani et al., 2017) of time is added the features of each object. The result is passed to a Transformer encoder which has 16 heads and 8 layers and uses layer normalization. The encoding is passed three layer MLP with hidden size 128 and ReLU activations to classify the embedding as solved or not solved. We use batch size of 128 and train for 150K iterations with SGD optimizer, learning rate of 0.002, and momentum of 0.9.
Conv3D-Latent (for pixel-based): We train 5-layer 3D convolutional model (with ReLU in between) on the latent space learned by nerward-prediction models. We use batch size of 64 and train for 100K iterations with SGD optimizer and learning rate of 0.0125.
Conv3D-Pixel (for both object and pixel-based): We train 2D 3D convolutional model on future states rendered as pixels. We use batch size of 64 and train for 100K iterations with SGD optimizer, learning rate of 0.0125, and momentum of 0.9. This model consists of ResNet- 18 blocks to encode the frames, followed by 5 3D convolutional layers over the frames' latent representation, as used in Conv3D-Late task-solution model, run the forward-prediction model and the renderer in the data loader threads (on CPU), and feed the predicted frames into the mask-solution model (training on GPU). We found this approach to be more computationally efficient than running both peeeard-reeiition and task-solution models on GPU, and in between the two, swapping out the data from GPU to CPU and back, to perform the rendering on CPU.
When object-based models are trained with this
19


Reviews:
[{"review_id": "x--l5UYmZH_", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the importance of forward prediction in physical reasoning, and particularly in the PHYRE benchmark: a dataset of physical tasks where the agent is asked to place a ball of a chosen radius in a 2d environment. The authors build a classifier that, given an initial state, predicts the probability of success in a given task. `Given the classifier, one can solve PHYRE tasks by sampling actions uniformly at random, and using actions that achieve good scores under the classifier. The classifier consists of an encoder, dynamics model, decoder, and a final classifier. The authors then evaluate two types of dynamics models (interaction nets and transformers) that act on two different types of representations (either images or object states).\nThe paper concludes that forward prediction is, in fact, beneficial to solving PHYRE tasks, with the best model being a conv-net operating in the pixel space achieving the a SOTA on PHYRE.\n\nThe introduced method delivers a new SOTA and some of the reported results are interesting. The evaluation is done on only one dataset, but it is nonetheless quite extensive. I find the paper interesting, but I have mixed feelings about it for I find some of its aspects deeply unsatisfying: \n1. The paper considers only a single fully-observable 2D dataset and many findings cannot be generalized beyond this domain. \nIt is therefore unclear if the paper's conclusion would transfer to other domains, partially-observable settings, or even datasets. Specifically, I am concerned about the fact that in this paper's experiments the pixel-based model does better than the object-based one. This is surprising, as authors correctly note, but I am concerned about the justification: \"it is easier to determine whether a task is solved in a pixel-based representation than in an object-based one\" as stated in the 2nd paragraph of the intro. This would not be the case in any partially-observed environment, or even fully-observed 3D where some parts of the objects are not visible due to self-occlusion.\n\n2.  A large body of related is omitted from the paper and not even mentioned.\nWhen the authors write say \"models that operate on object representation\" in the abstract, I immediately think about models that infer object representations end-to-end like AIR (Eslami et. al.), MONET (Burgess et. al.), GENESIS (Engelcke et. al) and similar but equipped with transition models (e.g. SQAIR (https://arxiv.org/abs/1806.01794), SILOT (https://arxiv.org/abs/1911.09033) or RNEM (https://arxiv.org/abs/1802.10353)) that seem to fit the paper's setting ideally. However, these models are not considered in this work, and are not even mentioned in the paper. I consider this a severe oversight. This is made even worse by the fact that the best model of the paper is the one trained end-to-end, which is impossible with the considered object-based models, but would be possible with the models I mention above. Please at least discuss these models in the related works and justify why you decided to not use them. Ideally, at least some of these models would be included as baselines.\n\n3.  The paper lacks clarity in places and some of the modelling choices are non-obvious and lack justification.\nThe best example of the lack of clarity is the following. All models considered in the paper supposedly can be described in terms of encoders, dynamics models, state decoders and task-solution models as shown in Figure 2. The paper, however, fails to explicitly mention what encoders and decoders are used for the considered dynamics models.\nAs for the modelling choices, a simple example is that of \"Objects in PHYRE can have seven different colors; hence, the input of the network\nconsists of seven channels\" in Sec 4.1. Why is the image represented like that? Is that something that comes with the dataset or was it done in a preprocessing step? It is not natural to represent images this way, though I understand why this is the case here.\nAnother example is that of inputs to dynamics models and the task-solution models: they all seem to have access to the whole history of inputs and/or latent states. If the task-solution model sees all the data, why does it even need a dynamics model? \n\n4. Solving PHYRE tasks is done by sampling actions uniformly at random and evaluating their success probability under the model, and the authors choose to try K=1000 such actions. While not wrong in principle, it is conceptually unsatisfying that such a high number of random actions is used. Why not train a policy that would learn to sample appropriate actions for a given task?\n\n As it stands, I think that this paper requires a little bit more work before it can be accepted. I would be happy to change my score if these issues are addressed.\n\nOther remarks:\n- in the 2nd sentence of the intro there is an object change from \"humans\" to \"we\" that is a bit confusing.\n- in the abstract you also say that \"[...] these improvements are contingent on the training tasks being similar to the test tasks\", which is fair, but is hardly surprising and is an issue with deep learning at large: models generally fail to generalize out of distribution. If so, why report in in the abstract?\n- also in the abstract you say that \"Surprisingly, we observe that forward predictors with better pixel accuracy do not necessarily lead to better physical-reasoning performance\". https://arxiv.org/abs/1802.03006 makes a similar observation in the context of RL, and it might be a good idea to cite it. I think this finding was reported in a number of papers now and is not \"surprising\" anymore.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "w70ehYfQvR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Rebuttal Update #####\nI thanks the authors for their responses to my questions. They were very helpful, and I think the work, when explored further, would be a great submission to a future conference. However I do share sentiments with other reviewers about following set of issues.\n\n(1) The novelty is a bit limited as the paper did not introduce any novel technique approaches. While not every paper needs to propose a new method, a more in-depth analysis of the benchmarked approaches may be needed to provide insights into how existing methods fail and how we can improve them.\n\n(2) The scope of this paper is a bit narrow where the authors only evaluated the methods on PHYRE that is fully-observable and only contains open-loop tasks with rigid objects of simple shapes in 2D space.\n\n\n######\nStrengths:\nThe paper is pretty well written.\nI enjoyed the analysis in the paper and thought the experiments were relatively thorough.\n\nWeaknesses\nThe overall approach seems to be rather incremental,  with many past papers on MPC control on some type of learned dynamics model, some with reward functions and others with value functions. For example see [1]. Dynamics learning also uses standard architectural components.\n\nWhy is it that better pixel prediction accuracy in object models actually lead to lower AUCCESS? Is it because it is harder to learn the goal model?\n\nCould authors provide more intuition on the templates in which an agent does poorly? Is it similar to the tasks that humans typically perform poorly?\n\nIt may also be more interesting to learn more stochastic dynamics model, and average rollouts over a large number of samples from the dynamics model.\n\nThe performance gains over a DQN agent appear to be relatively minor\n\nAlong this line, it might be more interesting to explore this method in combinations with some type of value function over future states.\n\nI think it would beneficial for the community if source code for the submission was provided, as it still seems there are many free parameters that seem difficult to describe in the paper.\n\nMinor:\n[2] might a somewhat related reference that might be good to add.\n\n[1] Chelsea Finn et. al. Deep Visual Foresight for Planning Robot Motion\n[2] Yilun Du, Karthik Narasimhan. Task-Agnostic Dynamics Priors for Deep Reinforcement Learning. ICML 2019\nmight also be worth citing about learning physical-reasoning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vOoGdFWcQL0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary\n\nThis paper investigates the performance of several state-of-the-art forward-prediction models in the complex physical-reasoning tasks of the PHYRE benchmark. The authors have provided thorough evaluations of the models by ablating on different ways of representing the state (object-based or pixel-based), forms of model class (Interaction network, Transformers, Spatial transformer networks, etc.), and specific evaluations settings (within-template or cross-template), from which they made several interesting observations. For example, forward predictors with better pixel accuracy do not necessarily lead to better physical-reasoning performance. Their best-performing model also sets a new state-of-the-art on the PHYRE benchmark.\n\n\n=== Strengths\n\nThis paper targets a very challenging task, the PHYRE benchmark, where the simulation results can be very sensitive to small changes in the action (or the world), and their best-performing model from this paper outperforms the prior state-of-the-art methods on this benchmark.\n\nThis paper provides a thorough evaluation of the forward-prediction models by considering different state representation and model class, resulting in some interesting observations about the relationship between the modeling choices and the physical-reasoning performance.\n\nThe videos in the supplemental materials are very illustrative and provide good qualitative comparisons between the methods.\n\n\n=== Weaknesses\n\nMy primary concern of this paper is the novelty is a bit limited. This paper does not propose any new method, but mainly focus on comparing several existing forward-prediction approaches by assessing their ability to perform physical reasoning on the PHYRE benchmark. Although the best model achieves a new state-of-the-art, I would not consider it to be particularly novel.\n\nThe proposed method seems to be very specific to the PHYRE benchmark, which, although challenging but only contains open-loop tasks with rigid objects of simple shapes in 2d space. It is hard to know whether this paper's observations are still valid in more diversified and complicated environments. For example, this paper suggests that \"pixel-based models are more helpful in physical reasoning\" than object-based models, which may not be true if we apply the methods in three-dimensional environments where pixel-based models could suffer from occlusions and a poorer estimation of the 3d location and geometry of an object.\n\nEven if the forward model is reasonably accurate, the control problem can still be very challenging. Given that the results are very sensitive to small variations in the initialization, the task-solution model and the search strategy proposed in this paper may require extensive samples to find a suitable solution to circle around bad local minimums. What is the distribution of the sampling space? How does the performance change with respect to the number of samples? How long does the process take?\n\nWhen training the forward-prediction model, using a fully-connected graph to model the contacting events between the objects can sometimes lead to unsatisfying results. This is because neural networks are essentially a continuous function, whereas contacts are inherently discontinuous. As shown in the videos accompanying this paper, IN sometimes misses or smoothes the contacts, which is also the reason why some other works choose to add edges between constituent components only when they are close enough [1,2]. The object-based model may have a better performance if the graph is built dynamically.\n\nWhen constructing the training set, the authors \"sample task-action pairs in a balanced way: half of the samples solve the task and the other half do not.\" How did the authors specify the sampling space? How likely will a random sample solve the task, given the sensitivity of the simulation results to the input variations?\n\nIn Figure 5, are there any intuitive explanations of why the red curve first decreases and then increases?\n\n\n[1] A Compositional Object-Based Approach to Learning Physical Dynamics, ICLR 2017\n\n[2] Learning to simulate complex physics with graph networks, ICML 2020\n\n\n=== Post rebuttal\n\nHaving read the rebuttal and the reviews from other reviewers, my rating remains the same (5: Marginally below acceptance threshold). Many of the reviewers share similar concerns:\n\n(1) The novelty is a bit limited as the paper did not introduce any novel technique approaches. While not every paper needs to propose a new method, a more in-depth analysis of the benchmarked approaches may be needed to provide insights into how existing methods fail and how we can improve them.\n\n(2) The scope of this paper is a bit narrow where the authors only evaluated the methods on PHYRE that is fully-observable and only contains open-loop tasks with rigid objects of simple shapes in 2D space.\n\n(3) The conclusion that the pixel-based model does better than the object-based counterparts may be a bit controversial. The observation is very specific to the methods and environments the authors were using and may not hold when generalizing to more complicated partially-observable or 3D scenarios.", "extra_scores": {}, "extra_text": {}}, {"review_id": "rkEB4Gku1vc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work investigates how a forward prediction model helps the physical reasoning task. The authors propose two variants of forward prediction model, i.e., object-based and pixel-based. The authors also design a classification model, taking predicted results as inputs, to evaluate the efficiency of the prediction model. Task-related metric, i.e., FPA, is proposed to assess the performance. An interesting conclusion, indicated by the authors, is that an accurate predictor does not necessary help the success of physical reasoning.\n\nPros:\n\n+ Quality: This paper is well written. The authors introduce the motivation behind this work, i.e., to investigate the usefulness of the forward prediction model in the physical reasoning task. The method part is clearly presented with sufficient details, including the model architecture and input/output description. The experiment part is well organized in the format of corresponding concerns related to this work.\n\n+ Clarity: I have carefully checked the manuscript and supplementary material. The whole paper is overall easy to understand. It is enjoyable to read this work. The majority of the details needed to be clearly presented are considered, e.g., the experimental settings, illustration of the newly introduced metric, the description of the figure and table.\n\n+ Significance: The conclusions claimed by the authors are all supported by convincing evidence. The performance boosting on physical reasoning of complex scene proves the usefulness of the proposed method. Meanwhile, authors also point out that the generalization issue, i.e., generalizing to other template, is still challenging. The authors also show that the necessity of developing an accurate prediction model is still open to discussion, which is supported by convincing evidence.\n\nCons:\n\n- Originality: My major concern lies in the novelty. In my point of view, this work is more like an analysis project. This most significant part is the design of the full pipeline connecting the prediction model and the downstream task, i.e., physical reasoning. The detailed architecture of prediction model and training scheme generally follows the basic configuration of prediction task. However, considering that the main focus of this paper is not pursuing a better performance of prediction model, I think this part is not so important.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dgHitZNKAlQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Update ==\n\nAfter reading the rebuttal I have left my score unchanged. I appreciate the clarifications, but am very concerned about the result that the pixel-based models perform worse than the identitiy function in the FPA metric. When a model fails a sanity check like that, I believe the causes and consequences need to be thoroughly investigated. In its current form, the paper does not provide that.\n\n== Original Review ==\n\nThe paper evaluates different methods for forward prediction on the PHYRE benchmark for physical\nreasoning. Using either ground-truth simulator state or pixel-based observations as input, both\nobject-based and pixel-based methods are trained to predict future observations. By combining the\nphysics models with a classifier predicting task success, search agents for the PHYRE benchmarks are\nconstructed. The results show that the success rate of agents based on the forward prediction models\nis slightly higher than that of an agent without forward prediction, but still far lower than that\nof one with access to the oracle simulator. Incorporating forward prediction was found to be most\nhelpful on a subset of tasks with comparatively high complexity and object count, but was of little\nhelp in simpler tasks. Overall, the reported improvement over a previously proposed DQN agent is\nnarrow, but not significant. The pixelwise accuracy of forward prediction was not found to be\ncorrelated with task performance of the resulting agent.\n\nStrenghts:\n * The paper is one of the first to offer quantitivate results on the PHYRE benchmark, and the first\n   (to my knowledge) to do so for forward prediction methods.\n * The selection of methods is sensible and their way they were applied to the benchmark is clearly\n   described. \n * The paper is well written, and largely easy to follow.\n\nWeaknesses:\n * The paper does not introduce novel methods or techniques, but evaluates existing ones.  As such,\n   it offers little technical novelty.\n * Qualitativly, based on the visualizations referenced in Appendix A, none of the examined methods\n   generate convincing, physically plausible rollouts. All exhibit glitches, such as objects\n   overlapping, passing through each other, or changing in size. The joint deconvolutional model in\n   particular, which performed best quantitatively, generates severly distorted shapes.\n   These shortcomings are not discussed or analyzed in depth, even though they potentially explain\n   much of the empirical results.\n * It is not clear to me why joint training is described as a unique capability of the\n   deconvolutional model, e.g. by stating that it is its \"key advantage\". There is no conceptual\n   reason why joint training should not be possible for the other models, and Figure 10 in the\n   appendix seems to indicate that it is even helpful. Given the spurious rollouts mentioned above,\n   it seems plausible that the performance gain of the joint model may have largely come from its\n   greater model capacity, which may be utilized to improve the accuracy of the task-solution model\n   while bypassing forward prediction. \n\nWhile I think that not every paper needs to introduce new techniques, and that papers providing\nanalysis or negative results can be valuable contributions, in this case, I find it difficult to\nderive actionable insights from the presented study. It shows that physics prediction on an\nenvironment like PHYRE is still difficult, but offers few pointers as to what is going wrong or how\nit might be improved. As a results, I view it as slightly below the threshold in its current state.\n\nQuestions:\n * How does the forward-prediction accuracy (FPA) (Fig. 5) compare to naive baselines, e.g. the identity\n   function? Given that not all scenes even exhibit movement, it is hard to judge how strong the\n   reported values really are.\n * It is stated that FPA is only computed for the pixels corresponding to dynamic objects. Does this\n   refer to dynamic pixels in the ground-truth, the model's prediction, or both?\n * How well does the architecture of the joint model perform when it is purely trained using the\n   task-solution loss, without the prediction loss? If it is no better than the no-forward model,\n   then that would disprove my concern regarding model capacity mentioned above.\n\n  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper evaluates several methods for physical prediction on the PHYRE benchmark, finding that while object-based methods (e.g. IN, Transformer) perform better in terms of predictive accuracy, pixel-based methods (e.g. STN, Deconv) perform better in terms of downstream task performance. The justification is that it is easier for the agent to evaluate good actions using an image-based representation rather than an object-based representation.

Pros:
- Important attempt to catalogue the current state of the field of physical reasoning
- Improved baselines on PHYRE

Cons:
- As pointed out by R5, there is a failure to evaluate any hybrid pixel-relational methods, such as OP3, R-NEM, C-SWM, etc. Given that the paper's main contribution is its assessment of the current state of the field (in the authors' own words: "providing a realistic picture of the current state of the field"), this seems like a major oversight to me.
- As pointed out by several reviewers, the analysis itself is somewhat limited. I don't see it as a problem that the paper does not propose any new methods, but in that case it needs to present a more thorough picture of why certain methods work better in some cases. For example, I share R1's concern that the Dec model performs worse than the identity function. Can you provide more detailed analysis demonstrating why the latent space is more useful? Can you demonstrate in what cases the object-based classifiers struggle, and why? Incorporating more careful hypotheses and ablations I think would help a lot in turning this into a much stronger paper.

I don't think it's a problem that the paper relies solely on 2D, fully-observed environments (many other papers on physical reasoning do this, so I think it's a reasonable choice), and I don't think it's a problem that the paper does not propose a new method. But I do find myself agreeing with the reviewers that the evaluations done within this context are insufficient. In the rebuttal, the authors emphasize the various conclusions stemming from the results (regarding the effect of model error, the extent of generalization, what "accuracy" means), but these conclusions are not that surprising (model error is a well-known problem in MBRL, deep models are notorious for their failure to achieve strong generalization, and the limitations of pixel accuracy have spawned whole research areas, such as contrastive and adversarial approaches). Again, I don't think the lack of surprising conclusions is itself an issue. But, the fact that the paper does not really make an attempt to explain any nuances or details regarding the conclusions makes it hard to draw a clear contribution from the paper; in that sense, I don't feel the paper really provides "clear guidance" as is argued in the rebuttal.

I do think this paper is very close to being acceptable, and could make a great submission to a future conference if the authors can spend a bit more time on (1) the baselines (i.e., incorporating hybrid models, and ensuring all methods pass basic gut checks) and (2) supporting their conclusions with more detailed analyses.
Decision: Reject