Stochastic Neural Physics Predictor

1 INTRODUCTION Learning to predict the physical motion of objects from data is an open area of research . Yet , recent ( hierarchical ) relation network based forward dynamics predictors ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Mrowca et al. , 2018 ; Li et al. , 2019 ) seem to be a promising alternative to conventional physics engines that are key components of robot control , computer vision and reinforcement learning ( RL ) systems . Physics simulators , both traditional numerical solvers and learned prediction models , still suffer from insufficient accuracy in challenging scenarios . Small errors in the input and model can lead to dramatically different object trajectories . Take the orange ball that is falling on the blue wedge in Figure 1 . Depending on where the orange ball starts or what bias the model has , the ball could either end up on the left or right side . Both are valid outcomes . However , deterministic physics engines will either predict one trajectory or the other . While it is important to reduce errors in each prediction , it is also important to acknowledge that uncertain situations might not have one but multiple possible outcomes . In machine learning , uncertainty-aware neural networks avoid deterministic point estimates by predicting distributions or by randomly sampling in the prediction interval . In the context of dynamics predictions , we propose to use Monte Carlo sampling based dropout on the model weights of a learned forward dynamics predictor to model uncertainty and sample multiple plausible trajectories for an initial state . To stabilize each trajectory and reduce error accumulation over long-time horizons , we use a state-invariant recurrent training mechanism . By feeding back predictions as input over multiple time steps , the model becomes more robust to its own prediction errors without the need for a hidden state . Finally , we introduce a new shape loss on the model predictions that constrains the pairwise distances between objects and object parts and greatly improves shape preservation and the stability of trajectories over long-time horizons . Our final fully differentiable forward dynamics model is able to sample multiple , more accurate and more stable trajectories over long-time horizons compared to existing baselines . An accurate forward dynamics predictor that is able to predict a distribution of future states can be of great importance for robotic control . In model-free reinforcement learning , accomplishing tasks through random exploration is sample inefficient and hardly generalizable . Model-based methods promise greater generalization abilities , but suffer from deterministic world models that are hard to learn and fail in stochastic environments . With our stochastic forward dynamics predictor , we can move part of the sampling process into the environment , physically grounding the random exploration of model-free agents . As the agent is able to observe multiple trajectories at a given state without actually executing multiple actions , the sample efficiency is greatly improved while the stochasticity of each state and action is implicitly learned . We show on several control experiments that a model-free agent trained in our stochastic forward dynamics environment is not only able to better explore and learn faster but often also comes to better solutions than agents trained in deterministic environments . In summary , ( 1 ) we propose a stochastic differentiable forward dynamics model that is able to generate multiple plausible trajectories via Monte Carlo ( MC ) based graph-convolutional dropout . ( 2 ) We greatly improve the accuracy and stability of long-term predictions by proposing a new fullyconnected shape loss term and training the model recurrently end-to-end in a state-invariant way . ( 3 ) We demonstrate how our stochastic dynamics model can be used to improve the efficiency and performance of model-free reinforcement learning agents on several physical manipulation tasks . 2 RELATED WORK . Physical dynamics prediction has long been an open research questions ( Fragkiadaki et al. , 2015 ; Agrawal et al. , 2016 ; Li et al. , 2016 ; Finn et al. , 2016 ; Lerer et al. , 2016 ; Mottaghi et al. , 2016a ; b ; Haber et al. , 2018 ; Tran et al. , 2015 ; 2016 ; Qi et al. , 2017a ; b ; Byravan & Fox , 2017 ) . Recent advancements in deep learning allowed for emergence of successful systems that aim at solving this problem by learning from data . Battaglia et al . ( 2016 ) and Chang et al . ( 2016 ) proposed a graph-based approach with object-centric and relation-centric representations , and a neural network architecture that predicts object dynamics and interaction between objects in complex 2D scenes . Similarly , Mrowca et al . ( 2018 ) and Li et al . ( 2019 ) implement a relational network with particle representation for objects , but extend this approach to 3D scenes introducing hierarchical graph representations for computational tractability . These works rely however on a single step prediction during training . We propose a recurrent training scheme on multiple step predictions and show lower long-term error in our experiments . Simulating future plausible object states under physical and user constraints is a commonly addressed challenge in computer graphics . Chenney & Forsyth ( 2000 ) analysed uncertainties in their simulation model for multi-body scenarios and used a Markov chain Monte Carlo algorithm to predict multiple trajectories . Twigg & James ( 2007 ) based their work on psychological findings about human errors in predicting object dynamics and simulated multiple future environment states by applying external random impulses to colliding bodies . Trajectories generated by both of the mentioned methods are visually plausible to human , but can often diverge from the real physical behavior and require extensive expertise to choose simulation parameters that ensure convergence . Huberman & Struss ( 1992 ) support the psychological theory behind the latter work , further suggesting that human predictions of non-linear dynamic effects such as collisions are far from perfect , and thus allow for a less advanced perturbation methods . Han et al . ( 2013 ) draw a line between physical and visual plausibility naming further factors improving the visual plausibility of a scenario such as the number of simultaneous collisions or homogeneity of colliding objects . In physical systems , situations that are non-intuitive to human can occur due to an unobservable state of the environment , e.g an object colliding with a fast rotating wheel or an unexpected behavior of a compressed spring . This opposes the goal of computer graphics were visual plausibility becomes a stronger requirement ( Barzel et al. , 1996 ) and motivates the search for more sophisticated methods for sampling probable states in physics engines . Multiple techniques allow neural networks to incorporate uncertainty in model predictions . MeanVariance-Estimation ( Nix & Weigend , 1994 ) is a method that circumvents point estimates in the output space by directly predicting a normal distribution . Um et al . ( 2018 ) used this method along with a particle-based representation for splash prediction . Assuming independent velocity distributions for each splash particle produced visually pleasing results . In our initial experiments for stochastic simulations , this method lead to unsatisfactory results . We observed that Mean-VarianceEstimation method is capable of indicating highly uncertain situations e.g collisions or force applications . Unfortunately , due to the lack of space-time consistency between particles that are present in real objects , this approach lead to incorrect shape predictions during test time . Stochastic regularization as a way of capturing model uncertainty is an active field of research with scarce theoretical foundations . Nonetheless , we see a growing number of practical applications of this group of algorithms in numerous research areas ( Gal et al. , 2017 ; Bhattacharyya et al. , 2017 ; Kendall et al. , 2015 ; Kampffmeyer et al. , 2016 ) . Gal & Ghahramani ( 2016 ) proposed applying dropout during training and inference as a Bayesian inference approximation with prediction variance as the measure of the epistemic uncertainty . A clear advantage of this method is the ability to visualize the results of each predicted trajectory . On the other hand the computational cost grows linearly with the number of samples . Prior work has shown that injecting noise into neural networks is successful not only as a regularization method ( Noh et al. , 2017 ; Kang et al. , 2016 ) but also in the training of RL agents Xia et al . ( 2018 ) ; Chua et al . ( 2018 ) . In model-free RL , temporal credit assignment , sparse reward , and exploration-exploitation trade-offs present significant challenges . Long episodes amplify both problems of credit assignment and reward sparsity , where naive exploration causes exponentially growing sample inefficiency ( Osband et al. , 2016 ) . Reward shaping is one counter measure that improves credit assignment , and consequently sample efficiency , in model-free RL ( Grześ , 2017 ; Zou et al. , 2019 ) . Designing shaping functions usually requires expert knowledge and hand-engineering , while also imposing constraints on how the agent solves the task . Such constraints may prevent the agent from solving the task optimally . Predicting a set of trajectories can be framed here as a reward relaxation method with the clear advantage of depending on a single parameter - dropout rate . Fortunato et al . ( 2017 ) introduced parametric noise learned with gradient descent into action prediction network , which led to significantly better exploration and higher rewards without creating large computational overhead . This method shows clear potential for stochastic methods to improve training efficiency in reinforcement learning . 3 APPROACH . Our stochastic forward dynamics model is based on the deterministic hierarchical relation network ( HRN ) as proposed by Mrowca et al . ( 2018 ) and depicted in Figure 2 . Hierarchical graph representation : Propagating effects through a fully-connected scene graph is computationally infeasible . HRN circumvents this problem by leveraging a tree-like graph form that defines a constrained subset of edges for more efficient effect propagation . Edges within each object graph comprise shape and material properties , describing how rigid , soft and ” cloth-like ” the material is as defined in our simulator . Edges across object graphs describe physical relations between objects such as contact forces . The hierarchy construction starts with the particles at the lowest level as provided by our ground truth simulator . Each next level of the hierarchy is constructed by clustering particles based on their states . The state attributes are the position , the velocity and the mass . Finally , nodes at each hierarchical level represent the state of an object , an object part , an object subpart and so forth down to single object particles . HRN takes a sequence of the past two hierarchical physics graphs G1,2 which consist of hierarchical object graphs as input and predicts the next state of the scene graph . Force , Collision and History Modules : HRN assumes three effects that act on particles at the lowest level of the hierarchy and that influence the next particle state : external forces , interactions with particles of other objects in close proximity and particle ’ s state history . Inputs to these modules are current particle states , external forces ( Force Module ) and states of particles with which the particle interacts ( Collision Module ) . Each module computes an embedding vector that represents the effects of the external forces , the collisions and the past states on the particle using pairwise graph convolutions . These effects are combined by summation and enter the Hierarchical Effect Propagation Module . Hierarchical Effect Propagation Module : HRN predicts the next graph state by estimating the influence that particles within each object and across objects exert upon each other . The compounded effects from the Force , Collision and History Modules are propagated not only up and down the hierarchy , but also between particles at the same level by hierarchical graph convolutions that are implemented as fully-connected networks with weight sharing . Each forward pass takes the states of two connected particles and outputs an embedding vector that describes the influence of the first particle on the latter . These effect embedding vectors are collected for each particle in the scene graph and are used to estimate the future particle velocity . Next State Prediction Module This module is realized as a feed-forward fully-connected neural network and predicts per-particle future velocity . It takes in the current particle state consisting of the position , velocity and mass , as well as the sum of effect embedding vectors belonging to the particle . The predicted particle velocity is expressed in the local coordinate frame , i.e relative the corresponding particle at the higher level in the hierarchy . The state of the particle at the highest level additionally includes gravity and is defined in the global coordinate frame . To make the HRN stochastic and sample multiple plausible trajectories for a given initial state , we introduce a Monte Carlo based dropout on the activations of the graph convolutional collision and force modules . We greatly improve model predictions by introducing a new fully-connected shape loss term and a state-invariant recurrent training procedure . Our final model creates realistic trajectories that are suited to train a model-free agent for manipulation tasks .

Summary: There has been work on deep learning based forward dynamics model to learn the dynamics of physical systems. In particular, the hierarchical relation network (HRN) as proposed by Mrowca et al. (2018). However, HRN is deterministic. This is problematic for long-term predictions because of the uncertainty of physical world. This paper builds on top of HRN. It proposes a Monte Carlo sampling based graph-convolutional dropout method that can sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor. It also introduces a shape preservation loss and trains the dynamics model recurrently to better stabilize long-term predictions. It demonstrates the two techniques improve the efficiency and performance of model-free reinforcement learning agents on several physical manipulation tasks.