Stochastic Neural Physics Predictor
Authors: Piotr Tatarczyk
Damian Mrowca
Li Fei-Fei
Daniel L. K. Yamins
Nils Thuerey
Venue:  (2020)
Keywords: physics prediction
forward dynamics
stochastic environments
dropout

Abstract: Recently,  neural-network  based  forward  dynamics  models  have  been  proposed that  attempt  to  learn  the  dynamics  of  physical  systems  in  a  deterministic  way. While near-term motion can be predicted accurately, long-term predictions suffer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth.  A system that predicts distributions of the future physical states for long time horizons based on its uncertainty is thus a promising solution.  In this work, we introduce a novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us to sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor.  By introducing a new shape preservation loss and training our dynamics model recurrently, we stabilize long-term predictions. We show that our model’s long-term forward dynamics prediction errors on complicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines. Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-free reinforcement learning agents faster and to better solutions on simple manipulation tasks.

Under review as a conference paper at ICLR 2020
STOCHASTIC NEURAL PHYSICS PREDICTOR
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, neural-network based forward dynamics models have been proposed that attempt learn the dynamics of physical systems a deterministic way. While near-term motion can be predicted accurately, long-term predictions suf- fer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth. A system that predicts distributions of the future physical states for long time horizons based on its un- certainty is thus promising solution. In this work, we introduce novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us sample multiple plausible trajectories for an initial state given weural based forward dynamics predictor. By introducing a new shape preservation loss and training our dynamics model recurrently, We show that our model's long-term forward dynamics prediction errors on com- plicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines. Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-fre reinforcement learning agents faster and to better solutions on simple manipulation tasks.
stabilize long-term predictions.
INTRODUCTION
Learning predict the physical motion of objects from data an open area of research. Yet, recent (hierarchical) relation network based forward dynamics predictors (Battaglia et al. 2016 Chang et al., (0166] Mrowca et al.) 2018 Li et al. 2019) seem to be promising alternative conventional physies en- gines that are key components of robot control, computer vi- sion and reinforcement learning (RL) systems. Physics sim- ulators, both traditional numerical solvers and learned predic- tion models, still suffer from insufficient accuracy challeng- ing scenarios. Small errors in the input and model can lead to dramatically different object trajectories. Take the orange ball that is falling the blue wedge in Figure [ Depending where the orange hall starts or what bias the model has, the ball could either end up on the left right side. Both valid outcomes. However, deterministic physics engines will either predict one trajectory or the other.
Figure 1: Uncertainty in physies. Small errors in the input and pre- diction can lead to significantly different object trajectories. The orange ball could either end up on the left or right side of the wedge.
While it is important to reduce errors in each prediction, it is also important to acknowledge that uncertain situations might not have one but multiple possible outcomes. In machine learning, uncertainty-awant neural networks avoid deterministic point estimates by predicting distri- butions or by randomly sampling in the prediction interval. In the context of dynamics predictions, we propose to use Monte Carlo sampling based dropout on the model weights learned forward dynamics predictor to model uncertainty and sample multiple plausible trajectories for an initial state. To stabilize each trajectory and reduce eтror accumulation over long-time horizons, we use state-invant inttriinviiiatt recurrent training mechanism. By feeding back predictions input over multiple time steps, the model becomes more robust to its own prediction errors without the need for a hidden state. Finally, we introduce new shape loss on the model predictions that constrains the pairwise

---

Under review as a conference paper at ICLR 2020
distances between objects and object parts and greatly improves shape preservation and the stability of trajectories over long-time horizons. Our final fully differentiable forward dynamics model is able to sample multiple, more accurate and more stable trajectories over long-time horizons compared to existing baselines.
An accurate forward dynamics predictor that is able to predict distribution of future states can be of great importance for robotic control. In model-free reinforcement learning, accomplishing tasks through random exploration is sample inefficient and hardly generalizable. Model-based methods promise greater generalization abilities, but suffer from deterministic world models that are hard to learn and fail in stochastic environments. With our stochastic forward dynamics predictor, we can move part of the sampling process into the environment, physically grounding the random explo- ration of model-free agents. As the agent is able to observe multiple trajectories without actually executing multiple actions, the sample efficiency is greatly improved while the stochasticity of each state and action implicitly learned. We show on several control experiments that model-free agent trained in our stochastic forward dynamics environment is not only able to better explore and learn faster but often also comes to better solutions than agents trained in deter- ministic environments.
given state
In summary, (1) we propose a stochastic differentiable forward dynamics model that is able to gen- erate multiple plausible trajectories via Monte Carlo (MC) based cruph-convolutional dropout. (2) We greatly improve the accuracy and stability of long-term predictions by proposing a new fully- connected shape loss term and training the model recurrently end-to-end in state-invariant way. (3) We demonstrate how our stochastic dynamics model can be used to improve the efficiency and performance of model-free reinforcement learning agents on several physical manipulation tasks.
RELATED WORK
Physical dynamics prediction has long been an open research questions (Fragkiadaki et al. 2015
Agrawal al. 2016 Li et al. 2016 Finn et al., 2016 Lerer et al., 2016 Mottaghi et al. 2016a b
Haber et al.] 2018 Tran et al. 2013 2016 Qi et al.
2017a bj B Byravan & Fox 2017 Recent advancements in deep learning allowed for emergence of successful systems that aim solving this problem by learning from data. [Bttaglia et al. (2016) and Chang et al. (2016) proposed a graph-based approach with object-centric and relation-
representations, and neural network architecture that predicts object dynamics and interaction between objects in complex 2D scenes. Similarly, MMrwwaa et al. (0018) and Li et al. (0119) implement relational network with particle representation for objects, but extend this approach to 3D scenes introducing hierarchical graph representations for computational tractability. These works rely however on single step prediction during training. We propose a recurrent training scheme on multiple step predictions and show lower long-term error in our experiments.
Simulating future plausible object states under physical and user constraints is commonly ad- dressed challenge in computer graphics Chenney & Forsyth (2000) analysed uncertainties in their simulation model for multi-body scenarios and used Markov chain Monte Carlo algorithm to predict multiple trajectories. Twigg & James (007)) based their work on psychological findings about human emrors in predicting object dynamics and simulated multiple future environment states by applying external random impulses to colliding bodies. Trajectories generated by both of the mentioned methods are visually plausible human, but can often diverge from the real physical behavior and require extensive expertise to choose simulation parameters that ensure convergence. Huberman & Struss (1992) support the psychological theory behind the latter work, further suggest- ing that human predictions of non- non-linear dynamic effects such as collisions are far from perfect, and thus allow for less advanced perturbation methods. Han et al. (2013) draw a line between physical and visual plausibility naming further factors improving the visual plausibility of a scenario such as the number of simultaneous collisions or homogeneity of colliding objects. In physical systems, situations that are non-intuitive to human can occur due to unobservable state of the environment, e.g an object colliding with a fast rotating wheel or an unexpected behavior of a compressed spring. This opposes the goal of computer graphics were visual plausibility becomes stronger requirement (Barzel et al. 1996) and motivates the search for more sophisticated methods for sampling probable states in physics engines.
2

---

Under review as a conference paper at ICLR 2020
Multiple techniques allow neural networks to incorporate uncertainty in model predictions. Mean- Muriance-Estimattie (Nix & Weigend, (994) method that circumvents point estimates in the output space by direetly predicting normal distribution. [ et al. (2018) used this method along with a particle-based representation for splash prediction. Assuming independent velocity distri- butions for each splash particle produced visually pleasing results. In our initial experiments for stochastic simulations, this method lead to unsatisfactory results. We observed that Mean-Van Variance- Estimation method is capable of indicating highly uncertain situations egg collisions or force appli- cations. Unfortunately, due to the lack of space-tim consistency between particles that are present real objects, this approach lead incorrect shape predictions during test time.
way of capturing model uncertainty is an active field of research with searce theoretical foundations. Nonetheless, we see growing number of practical applications of this group of algorithms in numerous research areas (Gal et al. 2017 Bhhttacharyya et al. 2017; Kendall et al. 2015 Kampffmeyer et al., 2016). Gal Ghahramani (2016) proposed applying Bayesian inference approximation with prediction vari-
Stochastic regularization
dropout during training and inference ance as the measure of the epistemic uncertainty. A clear advantage of this method is the ability to visualize the results of each predicted trajectory. On the other hand the computational cost grows linearly with the number of samples.
Prior work has shown that injecting noise into neural networks is successful not only regu- larization method (Noh et al. 2017; Kang et al. 2016) but also in the training of RI. agents Xia al. (2018): Chua al. (2018). In model-free RL, temporal credit assignment, sparse reward, and exploration-explottauie
trade-off's present significant challenges. Long episodes amplify both problems of credit assignment and reward sparsity, where naive exploration causes exponentially growing sample inefficiency (Osband et al., (0166). Reward shaping is one counter measure that im- proves credit assignment, and consequently sample efficiency, model-free RL (Grzes) 2017 Zou et al., 2019). Designing shaping functions usually requires expert knowledge and band-engineering while also imposing constraints on how the agent solves the task. Such constraints may prevent the agent from solving the task optimally. Predicting a set of trajectories can be framed here reward relaxation method with the clear advantage of depending on a single parameter dropout rate. Fortunato et al. (2017) introduced parametric noise learned with gradient descent into action prediction network, which led to significantly better exploration and higher rewards without creating large computational overhead. This method shows clear potential for stochastic methods to improve training efficiency in reinforcement learning.
3
APPROACH
Force Module
G₁
Collision Modulc
Hierachical Effect Propagation Module
Next State Prediction Module
G2
History Module
HRN
Figure 2: Hierarchical Relation Network (HRN) architecture. Force, collision and past effects on particles are computed and then propagated through each object hierarchy. The propagated effects used to predict the next particle positions. Gray blocks represent graph convolutional effect propagation modules.
Our stochastic forward dynamics model is based on the deterministic hierarchical relation network (HRN) proposed by Mrowca et al. (2018) and depicted in Figure
Hierarchical graph representation: Propagating effects through a fully-connected scene graph is computationally infeasible. HRN circumvents this problem by leveraging a tree-like graph form that defines a constrained subset of edges for more efficient effect propagation. Edges within each object graph comprise shape and material properties, describing how rigid, soft and "cloth-like" the material is as defined our simulator. Edges across object graphs describe physical relations between objects such contact forces. The hierarchy construction starts with the particles the
3

---

Under review as a conference paper at ICLR 2020
lowest level provided by our ground truth simulator. Each next level of the hierarchy is constructed by clustering particles based on their states. The state attributes are the position, the velocity and the mass. Finally, nodes at each hierarchical level represent the state of object, object part, an object subpart and so forth down to single object particles. HRN takes a sequence of the past two hierarchical physics graphs G1,2 which consist of hierarchical object graphs input and prediets the next state of the scene graph.
Force, Collision and History Modules: HRN assumes three effects that on particles at the lowest level of the hierarchy and that influence the next particle state: external forces, interactions with particles of other objects in close proximity and particle's state history. Inputs to these modules are current particle states, external forces (Force Module) and states of particles with which the particle interacts (Collision Module). Each module computes an embedding vector that represents the effects of the external forces, the collisions and the past states on the particle using pairwise graph convolutions. These effects are combined by summation and enter the Hierarchical Effect Propagation Module.
Hierarchical Effect Propagation Module: HRN predicts the next graph state by estimating the influence that particles within each object and across objects exert upon each other. The compounded effects from the Force, Collision and History Modules are propagated not only up and down the hierarchy, but also between particles at the same level by hierarchical graph convolutions that are implemented as fully-connected networks with weight sharing. Each forward pass takes the states of two connected particles and outputs embedding vector that describes the influence of the first particle on the latter. These effect embedding vectors are collected for each particle in the scene graph and are used to estimate the future particle velocity.
Next State Prediction Module This module is realized as a feed-forward fully-connected neural network and predicts per-particle future velocity. takes in the current particle state consisting of the position, velocity and mass, as well as the sum of effect embedding vectors belonging to the particle. The predicted particle velocity expressed in the local coordinate frame, i.c relative the corresponding particle at the higher level in the hierarchy. The state of the particle at the highest level additionally includes gravity and defined in the global coordinate frame.
To make the HRN stochastic and sample multiple plausible trajectories for given initial state, we introduce a Monte Carlo based dropout on the activations of the graph convolutional collision and force modules. We greatly improve model predictions by introducing a new fully-connected shape loss term and a state-invariant recurrent training procedure. Our final model creates realistic trajectories that are suited to train a model-free agent for manipulation tasks.
3.1 STABLE LONG PREDICTIONS VIA RECURRENT TRAINING AND SHAPE CONSTRAINTS
Stable realistic long-term predictions are crucial for planning tasks. While HRN predictions are ac- curate for large number of complex physical scenarios, identify that objects fall apart relatively quickly along boundaries of object parts for two reasons.
First, the HRN's loss function is designed to minimize the error between predicted and ground truth states while imposing a group shape constraint. As shown in the left panel of Figure 3 this group shape loss optimizes the pairwise distances between object nodes within a group to be the same as the ground truth pairwise distances. It does not impose that the pairwise distances across groups are the same as the ground truth distances, which leads to unrealistic deformations between groups as depicted in Figure 5 We therefore introduce a stronger fully-connected shape constraint (Figure Figure 3 right) that imposes pairwise distances to be the same as ground truth pairwise distances across all possible node combinations, which improves shape preservation significantly.
Second, the HRN's prediction errors accumulate exponentially as predictions are fed back in recur- rently during inference to generate multi-step trajectory predictions. However, during training the HRN is only supervised with the next ground truth state and thus never gets its own perturbed pre- dictions as input. To make the HRN robust against prediction errors, we therefore propose to train the model recurrently in a state- invariant way, iee. without using a hidden state as physical dynamics is free (Figure 44. The overall loss is the sum of losses from each time step. Learning recur- rently on long sequences, the network optimizes its weights taking into account its own prediction errors during training. This significantly reduces error accumulation during inference time.
4

---

Under review as a conference paper at ICLR 2020
Group Shape Loss
Fully-Connecied Shape Loss
G
G
G
HRN
HRNN
HRN
Shope constraints each same onloned group
Shape resgret to particle
Shape
respoct in red particle
Figure 3: Shape loss. The HRN shape loss only constraints particle distances within object particle groups (left). Our new fully-connected shape loss constraints distances between all particles pairs within each objects (right).
Figure 4: Recurrent training. Model predictions are fed back recurrently input stabilize long-term predictions.
3.2 SAMPLING REALISTIC TRAJECTORIES WITH MONTE CARLO DROPOUT
For sampling physically plausible object trajectories, we introduce a novel Monte Carlo sampling based graph-convolutional dropout method. Dropout as proposed by SSrivastava al. (2014)
moves a certain number randomly chosen nodes in a neural network to prevent overfitting with a probability commonly referred and only the edge weights attached to the active nodes are updated via backpropagation. In our novel approach, we use randomly sample dropout masks on graph-convolution kernels to sample physically plausible trajectories. To keep the kernel fixed independent of its position in the graph, only sample once per prediction step. To infer set of plausible trajectories, we randomly sam- ple a dropout mask for each generated trajectory during test time, similarly how Gal & Ghahramani (2016) uses dropout generate multiple predictions. The modular architecture of the HRN allows us apply dropout at different locations in a very interpretable way (Figure Dropout on the col- lision module makes sampled trajectories diverge collision points. Dropout on the force module leads to diverging trajectories during force applications. Dropout on other HRN modules lead to convergence problems and unrealistic predictions. In the following experiments, we thus only apply dropout the HRN's force and collision modules. Applying dropout based sampling method on our dynamics model results in physically plausible long-term predictions with consistent shapes.
dropout rate. In each iteration,
set of nodes is sampled
3.3 MODEL-FREE REINFORCEMENT LEARNING STOCHASTIC ENVIRONMENTS
The ability to sample a distribution of physically plausible trajectories can be used to improve the efficiency exploration of model-free reinforcement learning agents. thus train model-free policy our stochastic physics predictor to achieve physical manipulation tasks as follows. At each episode during training, we input the agent's action and current state into our stochastic forward dynamics predictor and sample set of 5 future states with our dropout method. For accelerated training. we introduce a reward relaxation method which consists of rewarding an agent as soon of the trajectories from the sampled rewards much quicker. If none of the trajectories hits the goal, one future state is chosen at random and the sampling process is repeated for the next future state. The level of reward relaxation is controlled by the dropout rate. The higher the dropout rate, the wider the set of trajectories and the easier for the agent to be rewarded. This directs the agent much quicker towards the reward in early training stages. In scenarios requiring high levels of accuracy and repeatability, we find that a gradual reduction of the dropout rate during policy training helps convergence and leads more efficient policies compared to a fixed dropout rate, which we show in the following experiments.
leads the goal, naturally exposing the agent to
EXPERIMENTS
In our experiments, we first show that our recurrent training and new fully- fully-connected shape loss significantly improve the prediction quality for single long-term trajectories complex physical scenarios over baselines. We then demonstrate how our proposed Monte Carlo sampling based dropout method generates multiple high-quality trajectories by visualizing stochastic model roll-
5

---

Under review as a conference paper at ICLR 2020
outs. Lastly, stochastic forward dynamics model's ability to generate multiple trajectories to train a model-free policy on two physical manipulation tasks more efficiently and to higher reward.
4.1 FORWARD DYNAMICS PREDICTION PERFORMANCE
4.1.1 EXPERIMENTAL SETUP
We evaluate our models forward dynamics prediction performance against the HRN baseline (Mrowca al. 2018) on two complex scenarios. The first scenario showcases the ability of our model to predict complex deformations: A deformable soft cube is first lifted off the ground by an upward impulse and then falls toward the ground while rotating and deforming
impact (Figure 5 left). In the second scenario, we evaluate our models performance on complex collisions. Col- lisions can greatly magnify object position and pose errors leading large discrepancies between predictions and ground truth. For our collision experiment, two rigid cubes are placed at a random distance from each other and then repeatedly accelerated by an impulse on each cube into another to generate collisions Fiiuur[S]right) We train our model on a multitude of examples of both scenarios and evaluate on held examples. We compare the mean squared error on positions, velocities and shape loss and show qualitative long-term predictions of our model and the HRN baseline.
4.1.2 RESULTS
In Table we present our quantitative results on the deformation and collision task. Our fully- connected shape loss and recurrent training procedure significantly lower long-term prediction rors in both scenarios. On the collision task, initial position and velocity error increase slightly compared the baseline but accumulate far lower errors in the long run. Empirically, that our recurrent training procedure works best with sequence lengths between 4 and 6 time steps. Longer sequence lengths prevent the model from converging during training. We found that gradu- ally increasing the sequence length during training is an effective countermeasure.
found
The improvements of model the HRN baseline obvious in visualizations of predicted trajectories (Figure(5). Whereas HRN predictions fall apart along object boundaries and sometimes penetrate objects, our method preserves shapes and resolves collisions much better and predicts positions much closer to the ground truth, leaving us with an adequate basis for generating multiple plausible trajectories with our sampling method.
Table 1: Quantitative dynamies prediction evaluation. We compare the mean squared error on positions, velocities and shape preservation our model with the baseline HRN model. Our model outperforms the baseline deformation predictions and long-term collision predictions.
Position
MSE
Velocity MSE
Shape
MSE
L+5
t+1
[+5
t+15
L+5
t+15
HRN
0.00064
0.017
0.00019
0.0013
0.037
0.092
Deformations
Ours
0.00039
0.011
0.00003
0.00043
0.0051
0.019
HRN
0.028
0.84
0.0045
0.032
0.018
0.054
Collisions
Ours
0.029
0.63
0.0061
0.019
0.0020
0.0039
Ours
Ground Truth
68
#+5
t+10
++22
1+35
1+:50
#+3
1+9
t+12
Figure 5: Dynamics prediction comparisons. Our method compared to the HRN baseline and ground truth. a) A soft cube bounces of the ground. b) Two rigid cubes collide. Our method preserves the geometry of objects better over long time horizons.

---

Under review as a conference paper at ICLR 2020
4.2 SAMPLING MULTIPLE PLAUSIBLE TRAJECTORIES
4.2.1
EXPERIMENTAL SETUP
In this section, demonstrate that our Monte Carlo sampling based dropout method can sample multiple physically plausible trajectories from our forward dynamics predictor under the same initial state. In two complex scenarios study model's uncertainty during force applications and collisions. In the first scenario, an external force lifts a soft body, which subsequently drops toward the floor rotating slightly. By applying dropout the force module (Figure Zp, generate multiple trajectories that arise due to our model's uncertainty during force applications. We use dropout rate 0.1 during training and 0.3 during testing. In the second scenario, we show that our proposed method produces realistic sets of trajectories in collision scenarios. Forces are applied to two rigid cubes pushing them towards each other causing collision. Dropout applied both force and collision module with a rate of 0.05 both during training and testing.
4.2.2 RESULTS
The visualizations of multi trajectory roll-outs in Figure that the sets of predicted trajectories are physically and visually plausible. Due to the modularity of the HRN model, targeted stochas- ticity can be applied within each submodule via dropout, introducing uncertainty in the output of force and collision predictions. Our proposed sampling method is able to capture trajectory dis- tributions ranging from single mode low variance complex, multi-modal distributions. Dropout rates between 0.05 and 0.3 allow for fast convergence during training and wide variety of visually plausible sample trajectories during inference. We notice that inference dropout rates that differ sig- nificantly from the training rates can cause biased predictions leading to, e.g. objects slowly drifting away in one direction. Additional results studying the effect of the dropout rate the width of the state distributions can be found in supplement Figure 10
Deformations
Vi
Collisions
[+]
E+2
113
C14
E16
t-7
L18
119
Falling wedge
t+]
112
1:3
F14
116 1-7
118
119
Multiple
++1
1+2
1+3
1+4
146
1-7
6+8
1+9
Figure 6: Example sampled multiple trajectories. We use dropout on the force and collision mod- ule to sample multiple trajectories given the same initial input. Dark colors depict the ground truth trajectory. Light colors depict imagined sampled trajectories. The performance of model in forward simulation is maintained at a high level despite the introduced graph-convolutional dropout. Deformations: A soft cube bounces off the ground. Collisions: Two rigid cubes collide. Our method is able to sample multiple physically plausible trajectory in each scenario. Falling on wedge: We simulate the situation from Fig T A rigid objects falls on wedge. Stochastic simu- lation allows for multi- modal prediction. Multiple materials: A rigid and soft object interact with cloth while falling on the ground.
4.3 MODEL-FREE REINFORCEMEN LEARNING
4.3.1
EXPERIMENTAL SETUP
We show that stochastic physical environments are useful for intelligent systems by training rein- forcement learning agents two different scenarios involving various physical interaction types and materials. We Proximal Policy Optimization (Schulman et al. 2017) model-free reinforce-

---

Under review as a conference paper at ICLR 2020
ment learning method in all scenarios. Similarly to Plappert et al. (2017), add further baseline in which we use the deterministic environment and add Gaussian noise in the action space.
Cube moving task: In this scenario, the agent learns to apply sequence of forces to rigid cubes such that at least one of the cubes is pushed towards the goal region. The maximum length of the episode is 10. Here, the cubes transparent each other and cannot collide. The stochasticity originates entirely from the uncertainty the force application. We use a constant dropout rate of 0.1 in the force module of our physies predictor throughout the whole training.
Ball hitting tower task: Tasks that require inducing collisions pose a significantly more difficult challenge to the RL agent. In this scenario, there is stack of three rigid cubes and ball which the agent can apply force. The agent gets a reward for pushing the middle cube out of the tower. To achieve the goal, the agent needs hit the tower with the ball to which it applies forces. The maximum episode length is 15 steps.
Stochastic em (scal 1)
Stuchastic esv. (fixed)
Stochastic (seed 1) Deterministic awy.
Stochastic cov. (sced 2)
Detemminitti
10
Stochastic (seed 2) Action space
Stochastic (seed
Action space
episode length
Avg. opisode lengsh
5
mam
3
1000
2000
3000
4000
3000
Training update
Training update
Figure 8: Average episode length in the "ball- hits-tower task". We compare determinis- tic environment against action space noise, a stochastic environment where the dropout rate is fixed and randomly seeded stochastic environ- ments where the dropout rate is annealed. The agent finds shorter policies earlier the train- ing indicating more efficient exploration in the stochastic environments.
Figure 7: Average episode length in the "cube moving task". We compare a de- terministic environment against action space noise and 2 randomly seeded stochastic envi- ronments. The agent learns faster in stochas- tic environments through better initial explo- ration and converges shorter policy.
Deserminatic anvireument
Stochustic invircament
Tinget achieved two frames earlier!
112
Figure 9: Policy comparisons. a) Cube moving task. Top: Policy learned deterministic environment (longer, 4 time steps). Bottom: Policy learned in a stochastic environment (shorter, 2 time steps). The first two frames are model inputs. The red cubes indicate the target position to which the green cubes have be moved. b) Ball hitting tower task. Agents in deterministic and stochastic environments converge to similar 4 step policies. The figure depicts one 4 step policy example.
4.3.2 RESULTS
Cube moving task: In this example, introducing the action space noise leads to faster learning compared to learning the deterministic environment. Training in stochastic physical environments outperforms both baselines, allows for better exploration and finding of shorter policies. In Figure 9 visualize the two learned policies, in stochastic and deterministic environments.
8

---

Under review as a conference paper at ICLR 2020
Ball hitting tower task: In this scenario, the agent finds more efficient policies through the applica- tion of stronger forces during training, which results in gradually shorter policies as shown in Figure The most effective learning method is learning in stochastic environment with the dropout rate annealing. We lower the dropout rate linearly from 0.1 at the start to 0 after 1200 training updates. This method allows for initial fast exploration, but does not introduce
g
high stochasticity when precision is needed as the agent begins applying strong forces later in the training. Without an- nealing, the randomness is too high and agents learn longer policies in a noisy training process, as indicated by the red curve in Figure 8 Furthermore, action space noise improves the pace at which the agent learns compared to the entirely deterministic environment. The policies learned by the presented methods do not significantly differ. An exemplary policy is visualized in Figure 9
5 CONCLUSION
Qualitatively our stochastic HRN predicts plausible future trajectories; an experiment in which hu- man subjects were asked to discriminate between ground-truth and predicted trajectories could be used to evaluate its performance quantitatively. Even though this method does not require exten- sive expert knowledge, few design decisions have be made e.g dropout rates for training and inference. During inference, too high of a dropout rate can lead to visually unrealistic dynamics and object interactions. Dropout rate scheduling during training should be investigated to improve con- vergence of the dynamics model during training, which may improve its performance as an environ- ment for the reinforcement learning tasks. Possible optimizations include more complex, potentially non- non-linear, annealing schedules during inference, delaying the dropout rate annealing, and finding appropriate starting values. Finding a universal schedule that can be applied to any environment and task has large potential for accelerating reinforcement learning. Further improvements for the physics predictor are key for its use as a physical environment. These can include improvements for: scenarios with multiple materials in one scene, penetrations during collisions that can lead to insufficient position prediction, and generalization to new scenes.
Our results show that the proposed sampling method produces physically plausible trajectories in single- and multi-object scenarios as well as across a range of materials. The quality of roll-outs, e.g. shape prediction is not compromised by the introduced noise. Furthermore, forcement learning experiments indicate that agents learning in physically stochastic environments able explore better and learn quicker, which confirms the quality of the sampled trajectories. In difficult reinforcement learning scenarios, where a high level of precision is needed to get a reward, we demonstrated that dropout rate annealing an effective method avoid high randomness at the same time not reducing the benefits of stochasticity for exploration in early stages of the training. In this regard, stochastic neural physics engines offer a clear advantage over conventional physics engines.
model-free rein-
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems, pp. 5074-5082, 2016.
poke by poking:
Ronen Barzel, John R. Hughes, and Daniel N. Wood. Plausible motion simulation for computer graphics animation. In Ronan Boulic and Gerard Hégron (eds.), Computer Animation and Simu- lation '96, pp. 183-197, Vienna, 1996. Springer Vienna.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physies. In D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4502-4510. Curran Associates, Inc., 2016.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Long-term on-board prediction of people in traffic scenes under uncertainty. CoRR, abts/1111.090000 2017. /tt.ppp////xi..../ orq/ abs/1711. 09026

---

Under review as a conference paper at ICLR 2020
Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp- 173-180. IEEE, 2017.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.0053 2016.
Stephen Chenney and David Forsyth. Sampling plausible solutions to multi-body constraint prob- lems. In Proceedings of the 27th annual conference on Computer graphics and interactive tech- niques, pp. 219-228, 01 2000.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754-4763 2018.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances neural information processing systems, pp. 64-72, 2016.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg- Noisy networks for exploration. CoRR, abs/1706. 10295, 2017. URL attp://arxiv..ot
abss/106..0 10295
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. arXiv preprint 1:Xiv:15110074:: 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML-16), 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. CoRR, 70s.01003009100 2017. URL
wwwp://////aaabbbsee..o.ss 02910
Marek Grzeš. Reward shaping in episodic reinforcement learning. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS '17, pp. 565-573, Rich- land, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems. URL http: acm. wwtpo/////iieo htt::/itaiiiiio cfm?id 3091125 3091208
Nick Haber, Damian Mrowca, Li Fei-Fei, and Daniel LK Yamins. Learning to play with intrinsically-motiive self-aware agents. arXiv preprint 1rXiv:1802.07442,20 2018.
Donghui Han, Shu-wei Hsu, Ann McNamara, and John Keyser. Believability in simplifications of large scale physically based simulation. In Proceedings of the ACM Symposium Applied Perception, pp. 99-106. ACM, 2013.
Bernardo Huberman and Peter Struss. Chaos, qualitative reasoning, and the predictability prob- lem. Recent advances in qualitative physics, pp. 119-135, 1992.
M. Kampffinyyer, A. Salberg, and R. Jenssen. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks, In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 680- 688, June 2016. doi: 10.1 10.1109/CVPRW.2016990
Guoliang Kang, Jun Li, and Dacheng Tao. Shakeout: A new regularized deep neural network training scheme. In Dale Schuurmans and Michael Wellman (eds.), AAAI, pp. 1751-1757. AAAI Press, 2016. ISBN 978- 978-1-57735-760-5
URL http://dblpp uni trierrrr..... de de/
html htt.:22001...ttmmmaagg..i.i.. #KangLT16
Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.
Bayesian segnet: Model uncer- tainty in deep convolutional encoder-decoder architectures for scene understanding. CoRR, abs/1511 1.06680, 2015. htthttt:: /tttt:://axxi.... wwg.aaawww....o..aaoooooooeooe.a absp////sl1111 wttp:///s..oa////1/1111111 02680
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. arXiv preprint 1003011603 2016.
10

---

Under review as a conference paper at ICLR 2020
Wenbin Li, Seyedmajid Azimi, Aleš Leonardis, and Mario Fritz. To fall not to fall: A visual approach to physical stability prediction. arXiv preprint 604.00066, 2016.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
Roozbeh Mottagii, Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. Newtonian scene understanding: Unfolding the dynamics of objects in static images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3521-3529, 2016a.
Roozbeh Mottaghi, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi. what happens if... learning to predict the effect of forces in images. In European Conference on Computer Vision, pp. 269-285. Springer, 2016b.
Damian Mrowca, Chengxu Zhuang, Elias Wang. Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physics prediction. In Advances in Neural Information Processing Systems, 2018.
D. A. Nix and A. S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume 1, pp. 55-60 60 vol.1, June 1994. doi: 10.1109/ICNN.119943.77 1944.37418..
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. CoRR, abs/1710.05179, 2017. URL
wwwpor///bb//oo/ooe 05179
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICML' 16, pp. 2377-2386. JMLR.org, 2016. URL http:
7/al. aal..acm.cm org/ wwwtatiittio... fm? 1e= 30010444599333345484 3045641
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. CoRR, abs/1706.01
2017. URL http::tttaxiiii// http:///axxxvv../aabaaaassss/a/s abs hltg///bb///1/.../../ 01905
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR),
IEEE, 1(2)4,, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. In Advances in Neural Information Processing Systems, pp. 5105-5114, 2017b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017. URL. httt:://erxxv..rr//bbt
1707.06347
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1919: 2014.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Computer Vision (ICCV), 2015 IEEE Internutional Conference on, pp. 4497. IEEE, 2015.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Deep end2end voxel2voxel prediction. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2016 IEEE Conference on, pp. 402-409. IEEE, 2016.
Christopher D Twigg and Doug L James. Many-worlds browsing for control of multibody dynamics. ACM Transactions on Graphics (TOG), 14, 2007.
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Liquid Splash Modeling with Neural Networks. Com- pufer Graphics Forum, 2018. ISSN 1467-8659. doi: 10.1111/cg 13522.

---

Under review as a conference paper at ICLR 2020
Victoria Xia, Zi Wang, and Leslie Pack Kaelbling. Learning sparse relational transition models. arXiv preprint arXiv:1810.11177 2018.
Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta-
learning. CoRR, abs/1901.09330, 2019. URL hhtt:://araae org/ abs http:/aab///11199/3/ 09330003300
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
6.1 EFFECT OF DROPOUT RATE ON THE WIDTH OF THE TRAJECTORY DISTRIBUTIONS
In Figure [0] we present the effect of the dropout rate on the width of the predicted distribution of trajectories. In this example, dropout is activated in the force module. We compare distributions generated with dropout rates 0.5, 0.3 and 0.1. In Figure [T] we present the stochastic predictions in "ball hitting tower" scenario. The dropout is active in the force and collision modules. The prediction variance corresponds to the dropout rate strength both scenarios. the "ball hitting tower" scenario, we observe highly complex behavior, where the time at which the cubes fall off the tower at different rates
Figure 10: Effect of the dropout rate on the distribution width of the sampled trajectories. 0.5 top.
0.3 middle, 0.1 bottom. Dropout applied in the force module.
Figure 11: Effect of the dropout rate on the distribution width of the sampled trajectories. 0.3 - top,
0.1 bottom. Dropout applied in the force and collision modules.
13


Reviews:
[{"review_id": "SJxWh3W0KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThere has been work on deep learning based forward dynamics model to learn the dynamics of physical systems. In particular, the hierarchical relation network (HRN) as proposed by Mrowca et al. (2018). However, HRN is deterministic. This is problematic for long-term predictions because of the uncertainty of physical world. This paper builds on top of HRN. It proposes a Monte Carlo sampling based graph-convolutional dropout method that can sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor. It also introduces a shape preservation loss and trains the dynamics model recurrently to better stabilize long-term predictions. It demonstrates the two techniques improve the efficiency and performance of model-free reinforcement learning agents on several physical manipulation tasks.\n\nStrengths\n\nLearning physics models which accounts for the multi-modal nature of the problem is very important. Graph-convolutional dropout is one method to deal with the multi-modal nature of the problem. It is a very good contribution. \n\n The tasks evaluated are not very sophisticated. \n\nWeaknesses\n\nThe paper's contribution is very incremental. \n\nThere should be a discussion on the different type of methods to account for uncertainties, e.g. bayesian neural networks and how they differ in terms of multi-modal predictions.\n\nBecause the tasks are not very complicated, it is not clear how good the whole neural physics predictor is.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gbnPT3tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThis paper introduces a method for physical dynamics prediction, which is a version of hierarchical relation network (Mrowca ‘18). HRNs work on top of hierarchical particle-based representations of objects and the corresponding physics (e.g. forces between different parts), and are essentially are graph-convolutional neural networks.\nUnlike the original work, the proposed method introduces several improvements: 1. an updated loss function, that adds distance constraints between *all* the particles of the object. 2. recurrent training, when the predictions are fed to the inputs. 3. adding dropout.  \n\nWriting:\nThe paper is relatively well-written and easy to follow.\n\nEvaluation:\nAuthors compare their model on a dynamics prediction task and seem to outperform the original HRN, especially on longer-term sequences. In addition they report results for trajectory sampling (qualitative) and model-free RL, where using their model as a stochastic simulator seems to have positive impact on agent training.\n\nDecision:\nAlthough the proposed improvements upon HRN generally make sense, it is not clear if those are very significant on their own: adding dropout and recurrent training do not seem particularly novel and, since there is no ablation study, it is hard to see what exactly contributes to the reported improvements. \nAs for the experimental evaluation, it seems like important baselines are missing, and the model seems to be very sensitive to hyperparameters (see questions). Thus, I am currently leaning more towards a rejection, hence the “weak reject” rating.\n\nVarious questions / concerns:\n\n* It is not clear why authors do not provide a comparison to DPI-Nets (Li ICLR‘19). This model seems to be outperforming HRN, and from what it looks like is publicly available: https://github.com/YunzhuLi/DPI-Net. I would encourage authors to provide comparison to this baseline, and potentially on similar sets of experiments, or explain why this comparison would not be possible (which seems unlikely).\n\n* Authors seem to acknowledge that the model is sensitive to the hyperparameter choice (dropout rates), however, there is no numerical evaluation that would help readers understand how critical this choice is for the final performance. Judging from very specific settings in different experiments, this could be a serious concern.\n\n* I find it a bit strange that results in Fig. 7-8 are for random seeds. Is it not possible to just plot an average for e.g. 10 runs?\n\nUpdate:\nI would like to thank the authors for a detailed response!\nIt seems like there is a common concern about the novelty among reviewers: improvements over HRN are quite incremental. Although authors provide a verbal justification for not comparing to another strong baselines, I do not see why would it not be possible to compare methods in the similar settings, even though that baseline might be more limited.\nGenerally, if the main contribution is actually only the \"stochastic\" part and not improved performance, then just adding dropout does not seem like a particularly novel approach to me: whether the convolutions are on graphs or on euclidean domains, this does not change the way dropout is done.\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1l8ZyHCtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe a neural architecture with dropout randomizer with the aim of producing an ensemble of physical trajectories that seem plausible for a human eye. The question is relevant and interesting.\n\nThe authors bring in two improvements to the HRN model by (Mrowca et al., 2018). They drop the hierarchy from the shape loss, which is decipeted in Figure 3, and provide a change in the recurrent training decipited in Figure 4.  Figure 4 is only showing the modified training, not the initial HRN training, although Figure 3 contains both. For the sake of consitency and good read the Figure 4 should contain the original training image.\n\nThe description of the system is very verbose, but a concrete description of the system is only in the references, which make the manuscript hard to read and it does not fit well in a conference where one cannot assume that the audience is not an expert of this particular subfield.\n\nAs a remedy, I would suggest, a simple one-hieararcy level architecture with hyperparameters to show what is really happening. One could use the system in Figure 1 as a more thorough example.\n\nThe results looks good and valuable, altough the images provided are quite small.\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a timely method for intuitive physics simulations that expand on the HTRN model, and tested in several physicals systems with rigid and deformable objects as well as other results later in the review. 

Reviewer 3 was positive about the paper, and suggested improving the exposition to make it more self-contained. Reviewer 1 raised questions about the complexity of tasks and a concerns of limited advancement provided by the paper. Reviewer 2, had a similar concerns about limited clarity as to how the changes contribute to the results, and missing baselines. The authors provided detailed responses in all cases, providing some additional results with various other videos. After discussion and reviewing the additional results, the role of the stochastic elements of the model and its contributions to performance remained and the reviewers chose not to adjust their ratings.

The paper is interesting, timely and addresses important questions, but questions remain. We hope the review has provided useful information for their ongoing research. 
Decision: Reject