Title: A Compositional Object-Based Approach to Learning Physical Dynamics
Authors: Michael Chang
Tomer Ullman
Antonio Torralba
Joshua Tenenbaum
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.

TL;DR: We propose a factorization of a physical scene into composable object-based representations and also a model architecture whose compositional structure factorizes object dynamics into pairwise interactions.

Review discussion:
1. Battaglia et al. (2016) and this work make counter claims on the ability to simulate over many timesteps very effectively when only trained for next-timestep prediction. Why did this work not include a quantitative comparison?
2. Why does this article choose to use compare only velocities and not positions of the balls?
3. Why does this work choose only to vary the shape and not the size of the scene configurations?
4. How does the work compare to Fragkiadaki et al. (2015) ? Can you provide an experimental comparision?
1. We developed our work independently and in parallel to Battaglia et al (2016), and did not have access to any details of their approach until very recently.  We have not yet had time to conduct a rigorous quantitative comparison, but that is something we plan to do in the near future.
However, one quantitative comparison is already available in our graphs of prediction velocity accuracy, in ball worlds (where our experiments overlap with theirs). The bottom row of Figure 3a shows our training curves for our prediction experiment on 4 balls. This graph shows that our normalized predicted velocity loss is below 10^(-3.5), which averages to 0.000241. In Table 2 of their paper, they report a prediction MSE of 0.002, which when normalized by their max velocity is 0.0004 (We normalized our velocity by our max velocity, so we divide theirs by 5, their max velocity, to compare our numbers to theirs).
We are not sure what is meant by “counter claims” here.  Battaglia et al. (2016) write that their networks “trained on single-step predictions can be used to simulate trajectories over thousands of steps very effectively”, while we write that “because physics is Markovian, this prediction need only be for the immediate next timestep”, which we take the to be consistent with the same claim.  We have added a line in the introductory paragraph of Section 3 to clarify this, where we explicitly say that like Battaglia et al. our predictions can be effective for a large number of time steps even though we only train to predict the immediate next time step.
2. We choose to predict velocities of the objects to preserve spatial invariance. Predicting velocity helps the network avoid memorizing the environment, whereas training the network to predict position conditions the network on the worlds in the training domain, making it more difficult to transfer knowledge across environments. Our network essentially implements an operator that computes, for each object: “Figure out what are the effects of each context object on me to predict how I would move,” which is a principle of cause-effect that can be applied across different environments. If our network were to predict position, then the generality of this operator would be more constrained to the specific environment it was trained on. This is also the logic behind the training procedures of Battaglia et al (2016) and Fragkiadaki et al (2015). This is a good point that deserves further clarification and we inserted a line into the "Pairwise Factorization" paragraph of Section 2.2 to that effect.
3. This is a good point. The purpose of varying the size of the scene configurations would be to show that the network is invariant to different world boundaries, and different spatiotemporal extents of prediction.  Our goal in varying the wall geometries, with the “O”, “L’, “U’, and “I” shaped scenes, was to show that the network predicts ball movement in the context of different spatial constraints (both locally, in terms of corners and flat walls, and globally, in terms of different environment topologies).
However, this effectively also tests whether the network is invariant to different world boundaries, and different spatiotemporal extents of prediction, which vary to some extent across these scenes.
Hence we felt that varying these configurations (internal walls and obstacles) is sufficient to cover that test. We do agree that it would also be interesting to test differently sized scene configurations and will try that for the next version of our paper.
4. We agree that contrasting the NPE with a “bottom-up” approach, such as one that operates solely on pixels like Fragkiadaki’s, would provide a strong comparison of the difference approaches. There are several differences between ours and theirs outlined in the related work section of our paper.
For example, we presented a factorization of the state space that overcame the scenic combinatorial complexity that Fragkiadaki mentions. We have demonstrated generalization to more objects and more complex scene configurations. Our inferences about underlying physical parameters are different: our network infers mass, while their network infers force.
Therefore, differences in environment and modeling assumptions make a completely one-to-one comparison difficult.
However, we agree that it would be informative to make the quantitative comparison between their and our work clearer. We already report our basic prediction results using the same evaluation metrics (angle and magnitude of predicted velocity) as they do. An additional comparison we can do that can circumvent the differences in constants such as ball radius and velocity between our experiments and the experiments they report. They report MSE error up to t+20, which is where a ball can move to a maximum of 8 ball-radius lengths. 8 ball-radius lengths corresponds to 8 timesteps in our work, and we can make the comparison between our MSE at t+8 and their MSE at t+20. In this case, if we compare our results in the top two rows of Figure 3a with their results in Table 1, our error in relative magnitude in velocity is 0.0823, while their OC model has 0.09 overall error.
When we compare our results in angular error, our error in angle is 16.8 degrees while their error is 14.7 degrees.
However, as discussed in our response to question 3 of Reviewer 2, because different sources of error, the further into the future we simulate, the less informative the prediction error is to the intuitive notion of how well the model captures physical dynamics.
Another comparison to Fragkiadaki's work can be to qualitatively compare our GIFs to their videos. In this comparison, we show performance on 8 balls while they show performance on 3 balls. The ball movements in their work (https://sites.google.com/site/intuitivephysicsnips15/) appear to magnetically be attracted to the world boundaries, while ours do not suffer from this behavior. We added at note on this in the Related Work section.
* What happens if the yellow pairwise part of the NPE model is replaced with a difference between two object representations? The yellow pairwise encoder increases the capacity of NPE compared to NP, so how important is that added capacity versus the pairwise factorization?
* Is the LSTM model supposed to be both non-pairwise and non-compositional? It does not take advantage of the pairwise factorization, but it is compositional (each hidden state is a function of the previous hidden state). Is a more specific notion of compositionality important? Would a non-compositional but pairwise lesion be possible?
* There seems to be a disconnect between the evaluation metrics in figure 3 and my intuitions about performance from watching the provided simulation GIFs. The gap between NPE and NP/LSTM curves in figure 3 is significant, but only small.
On the other hand, it seems like NPE handles object collisions in a basically reasonable manner (balls bounce off each other and change directions) while NP/LSTM models fail utterly (balls are attracted to one another after collisions occur). Do the authors also observe this? If so, what are the evaluation metrics missing?
* Why is angular velocity predicted in addition to velocity if it is always 0?
1. In the architecture hyperparameters we described, it is indeed the case that the NPE has an additional layer more than the NP. The pairwise layer just embeds each object's state independently into an intermediate representation space before these representations are concatenated as input to the rest of the encoder (red). We have added a line in Section 2.4 to clarify this.
Therefore, we do not see the initial pairwise layer as necessarily computing the difference between object representations, so a way to normalize the comparison between the NPE and NP is to change the number of layers. Running a full analysis would take longer before the response period is over, but we plan to include such formal analysis in the camera ready version. We already have informal results comparing a 3-layer NPE (4 layers including the pairwise layer) with the 5-layer NP and show that the 3-layer NPE performs only slightly worse than the 5-layer NPE (6 layers including the pairwise layer), therefore still performing an order of magnitude better than the NP in prediction loss. This shows that the capacity issue is not primarily responsible the NPE's increase in performance, but rather it is the difference between inputting individual objects through the red part of the encoder vs inputting pairwise embeddings through the red part of the encoder.
2. We agree that the LSTM is compositional in the sense you describe.
However, we stress that the LSTM is not compositional in the sense we describe in this paper. Our notion of compositionality treats each object and pairwise interaction as indepdently encapsulated in a separate computational entity that can be reused and rearranged; the NPE encoder (red and yellow) is a function that is applied to each pair between the focus and its neighboring context object. This function encapsulates this computation and can be repeatedly applied for all neighboring context objects equally, such that the NPE composes this repeated encoding function with the decoder function to predict velocity. The LSTM does not exhibit this notion of compositionality because it is not designed to take advantage of the factorized structure of the scene.
However, we can also imagine a non-compositional but pairwise architecture in the sense defined in the comment. A feedforward network does not have a persistently updated hidden state as the LSTM does, so it is non-compositional in this sense; we can then imagine embedding each pairwise interaction first, and concatenating all the pairwise interactions as input to the feedforward neural network.
However, the topology of the feedforward network is not flexible to handle a variable number of pairwise interactions since the feedforwardn network's input dimension is fixed. THis is a good comment and we clarified our notion of compositionality in the LSTM section of Section 2.3.
3. This is a good observation and we also noticed this discrepancy. We specifically produced the GIFs because we felt that these quantitative metrics, while useful to compare with previous work such as Fragkiadaki et al (2015), do not fully capture the difference between the NPE and the baselines.
When comparing the NPE with the NP/LSTM, we see that the evaluation of the model predictions against the ground truth are being driven by two different processes. The NPE would differ because of accumulation of subtle errors while maintaining the general physical dynamics, while the NP/LSTM would differ because of failure to predict the plausible physical movement entirely. From Vul and Smith (2013), we see that like the NPE, the human perceptual system also accumulates error, both in forward straight-line prediction and at collisions. A better quantitiative metric to get the qualitative sense of the discrepancy between the NPE and the NP/LSTM in the predicted physics simulation is to look at the error in next timestep velocity error (bottom row of Figure 3, ab).
One way to try and formalize this intuitive sense that the GIFs differ would be to do this, but there are other ways of exploring this intuitive discrepancy, such as letting people rate the movies on how realistic they look. A full analysis of how humans respond to this intuitive discrepancy is beyond the scope of the paper, but this is a good comment that deserves further clarification, which we added at the end of the introductory paragraph to Section 3.
4. This is a good point. The angular velocity was included in the architecture but did not affect the evaluation results. We moved a clarification of this to a footnote becuase it is not importatnt the paper's experimental valuation.
Although they are all set to zero, we mentioned angular velocity, gravity, friction, and pairwise forces because that is how the code was implemented, and we wanted to be helpful for people looking at the code.
* Neighborhood Mask
Neighborhood mask implicitly assumes certain minimum velocity for contextual objects. I wonder if authors have experimented with getting rid of the mask and letting model implicitly figure out which objects should be used for context? Or alternatively, having dynamic mask which is the function of relative distance and velocity between context and focus objects?
* Comparison to Fragkiadaki et al. (2015)
Fragkiadaki et al. (2015) does appear to be the closest published state-of-the-art (Battaglia et al. (2016) will technically not be published until next week at NIPS). Is there a reason why a more direct comparison with Fragkiadaki et al. (2015) is not carried out? This would strengthen the paper.
* Error in Figure 2?
It appears that there maybe an arrow missing in Figure 2 (bottom, middle)?
* LSTM baseline
I understand the reasoning for perturbing the ordering of contextual objects.
However, shouldn't there be a more natural ordering? e.g., rank order based on time to collision -- relative distance between contextual and focus object divided by their relative velocity.
Thank you.
1. Yes, this is a good point, and we have tried exactly what you suggested. Informal experimental results show that having the mask shows considerable performance improvement over not having the mask. Running a full analysis would take longer before the response period is over, but we plan to include more formal analysis in camera ready version.
In addition to adding more spatial structure to the model, the neighborhood mask serves an important practical purpose of computational efficiency, and a similar masking mechanism is present commonly used physics engine exactly for this reason. It prevents the network from wasting time considering irrelevant context objects that cannot possibly interact with the focus object. Note, however, that the neighborhood mask only implements broad-phase: context objects inside the neighborhood could interact with the focus object, or they could not. The mask only constrains the search space context objects, and the network figures out how to detect and resolve collisions.
We note in the discussion that having a dynamic mask would be a useful extension in future iterations of the NPE; we leave that investigation for future work because it is useful to investigate the dynamic mask in worlds with contain forces that act from a distance with forces that don’t. This is a good point that deserves further clarification, and we have added this clarification in the "Neighborhood Mask" paragraph of Section 2. 2 in the paper.
2. This is a good point, and was also mentioned by reviewer 1. Our resopnse to this point is reproduced below.
We agree that contrasting the NPE with a “bottom-up” approach, such as one that operates solely on pixels like Fragkiadaki’s, would provide a strong comparison of the difference approaches. There are several differences between ours and theirs outlined in the related work section of our paper.
For example, we presented a factorization of the state space that overcame the scenic combinatorial complexity that Fragkiadaki mentions. We have demonstrated generalization to more objects and more complex scene configurations. Our inferences about underlying physical parameters are different: our network infers mass, while their network infers force.
Therefore, differences in environment and modeling assumptions make a completely one-to-one comparison difficult.
However, we agree that it would be informative to make the quantitative comparison between their and our work clearer. We already report our basic prediction results using the same evaluation metrics (angle and magnitude of predicted velocity) as they do. An additional comparison we can do that can circumvent the differences in constants such as ball radius and velocity between our experiments and the experiments they report. They report MSE error up to t+20, which is where a ball can move to a maximum of 8 ball-radius lengths. 8 ball-radius lengths corresponds to 8 timesteps in our work, and we can make the comparison between our MSE at t+8 and their MSE at t+20. In this case, if we compare our results in the top two rows of Figure 3a with their results in Table 1, our error in relative magnitude in velocity is 0.0823, while their OC model has 0.09 overall error.
When we compare our results in angular error, our error in angle is 16.8 degrees while their error is 14.7 degrees.
However, as discussed in our response to question 3 of Reviewer 2, because different sources of error, the further into the future we simulate, the less informative the prediction error is to the intuitive notion of how well the model captures physical dynamics.
Another comparison to Fragkiadaki's work can be to qualitatively compare our GIFs to their videos. In this comparison, we show performance on 8 balls while they show performance on 3 balls. The ball movements in their work (https://sites.google.com/site/intuitivephysicsnips15/) appear to magnetically be attracted to the world boundaries, while ours do not suffer from this behavior. We added at note on this in the Related Work section.
3. Thank you for pointing that out in Figure 2. To clarify, there is no missing arrow; the architecture is as described: the NP does not have an connection between the encoder of the focus object and the summation. The focus object’s encoding is concatenated with the sum of the neighboring context objects’ encoding before fed into the decoder. The reason behind this design is as follows.
First, we want to investigate benefit of a the NPE’s pairwise encoding.
Thus we want to clearly separate the representation of the context objects’ effects on the focus object from the focus object’s encoding. The summation of the context encodings are interpreted as summing the force fields produced by the context objects, which should be independent from the focus object. The decoder should be able to take the focus object’s encoding and the net force field and produce its prediction. Having the focus object in that sum would lose the distinction between context and focus object in the summation, thus making the meaning of that summation ambiguous.
Second, this mirrors the “social pooling” of the Social LSTM paper, such that we can make a more direct comparison between the Social LSTM’s approach to the NPE’s approach to trajectory prediction.
4.
That is a good suggestion for the ordering of the context objects.
However, we did not adopt that ordering because we do not make assumptions of whether the context object would collide with the focus object or not — the network figures out if the context object collides. The context objects are selected by the neighborhood mask, which only implements broad-phase (trims the list of context objects).
Thanks to all reviewers their insightful and useful comments, which has led to a strengthening of the paper along several dimensions. Please see more detailed responses below.
We uploaded a revised version of the paper on Dec 11, based on the helpful feedback from the reviewers.
The Related Work Section now contains a fuller qualitative comparison between our approach and that of Fragkiadaki et al (2015).
- summary
The paper proposes a differntiable Neural Physics Engine (NPE). The NPE consists of an encoder and a decoder function. The NPE takes as input the state of pairs of objects (within a neighbourhood of a focus object) at two previous time-steps in a scene. The encoder function summarizes the interaction of each pair of objects. The decoder then outputs the change in velocity of the focus object at the next time step. The NPE is evaluated on various environments containing bouncing balls.
- novelty
The differentiable NPE is a novel concept.
However, concurrently Battaglia et al. (NIPS 2016) proposes a very similar model. Just as this work, Battaglia et al. (NIPS 2016) consider a model which consists of a encoder function (relation-centric) which encodes the interaction among a focus object and other objects in the scene and a decoder (relation-centric) function which considers the cumulative (encoded) effect of object interactions on the focus object and predicts effect of the interactions.  Aspects like only considering objects interactions within a neighbourhood (versus the complete object interaction graph in Battaglia et al.) based on euclideian distance  are novel to this work.
However, the advantages (if any) of NPE versus the model of Battaglia et al. are not clear.
Moreover, it is not clear how this neighbourhood thresholding scene would preform in case of n-ball systems, where gravitational forces of massive objects can be felt over large distances.
- citations
This work includes all relevant citations.
- clarity
The article is well written and easy to understand.
- experiments
Battaglia et al. evaluates on wider variety senerios compared to this work (e.g. n-bodies under gravitation, falling strings). Such experiments demonstrate the ability of the models to generalize.
However, this work does include more in-depth experiments in case of bouncing balls compared to Battaglia et al. (e.g. mass estimation and varying world configurations with obstacles in the bouncing balls senerio).
Moreover, an extensive comparison to Fragkiadaki et al. (2015) (in the bouncing balls senerios) is missing. The authors (referring to answer to question 4) do point out to comaprable numbers in both works, but the experimental settings are different.  Comparison in a billiard table senerio like that Fragkiadaki et al. (2015) where a initial force is applied to a ball, would have been enlightening.
The authors only evaluate the error in velocity in the bouncing balls senerios. We understand that this model predicts only the velocity (refer to answer of question 2). Error analysis also with respect to ground truth ball position would be more enlightening. As small errors in velocity can quickly lead to entirely different scene configuration.
- conclusion / recommendation
The main issue with this work is the unclear novelty with respect to work of Battaglia et al. at NIPS'16. A quantitative and qualitative comparison with Battaglia et al. is lacking.
But the authors state that their work was developed independently.
Differentiable physics engines like NPE or that of Battaglia et al. (NIPS 2016) requires generation of an extensive amount of synthetic data to learn about the physics of a certain senerio.
Moreover, extensive retraining is required to adapt to new sceneries (e.g. bouncing balls to n-body systems). Any practical advantage versus generating new code for a physics engine is not clear. Other "bottom-up" approaches like that of  Fragkiadaki et al. (2015) couple vision along with learning dynamics.
However, they require very few input parameters (position, mass, current velocity, world configuration), as approximate parameter estimation can be done from the visual component.  Such approaches could be potentially more useful of a robot in "common-sense" everyday tasks (e.g. manipulation).
Thus, overall potential applications of a differentiable physics engine like NPE is unclear.
We thank Reviewer 3 for their helpful comments.
1. Reviewer 3 is concerned about the unclear novelty with respect to work of Battaglia et al.
It is true that we have developed our work independently and in parallel from Battaglia et al., and we think that formal comparisons with their work in future work would be very useful. Physics naturally decomposes into object-object interactions, so both our work and theirs take advantage of this observation. We see the similarities between our work and theirs as strengthening and converging evidence for the approach of object-based representations for modeling physics.
With regard to a comparison of experiments, our experiments focused on worlds with bouncing balls, which Battaglia et al. noted were especially challenging. We also analyzed generalization across variable obstacle configurations beyond simple bouncing balls, which goes beyond Battaglia et al. In this comment the reviewer raises another important point about generalization. We would like to distinguish between two types of generalization. The first type, to which the reviewer refers with regard to the variety in experiments in Battaglia et al., measures a model’s capacity to learn a new and different concept through training.
While the reviewer correctly points to the need to test the system on varying domains, our architecture makes few assumptions ((1) it assumes that there exist objects in the world and (2) that objects interact with objects in their context) so does not have to be restricted to only domains of bouncing balls. As expressed in the discussion section we plan to apply future iterations of the NPE on other worlds such as with block towers and liquids. The flexibility to learn new physics concepts is a core reason why we chose to learn a differentiable physics engine, where the various physics can be learned through training. The second type of generalization involves extrapolation of “common-sense” reasoning.
For example, a model that learns about physics with fewer balls should not have to be retrained to learn the physics of more balls. A model that learns about interactions between balls and obstacles should be invariant to the scene configuration because the scene configuration should not matter if the underlying physical laws remain the same. This type of generalization involves a difficult transfer of knowledge because it involves no retraining, and we studied this type of knowledge transfer in-depth in this paper with two generalization tasks (variable object count and various scene configurations). Performing this type of generalization is exactly our motivation for the state space representation and model architecture with regards to factorization and compositionality.
For example, expressing physics modeling as a local computation with a focus object and context objects allows the model to scale to arbitrary numbers of objects. Expressing a large structures in the scene as composed of smaller building blocks allows the model to be invariant to geometries of the large structures because they merely decompose into different spatial arrangements of the building blocks.
With regard to a comparison of modeling performance, following the reviewer’s pre-review comments about a comparison to Battaglia et al., we also added the following lines to the paper, reproduced here: “Like Battaglia et al. (2016) our predictions can be effective for a large number of time steps even though we only train to predict the immediate next time step…As can be seen by the videos, while the NP/LSTM fail to predict plausible physical movement entirely, the NPE's predictions initially adhere closely to the ground truth, then slowly diverge due to the accumulation of subtle errors, just as the human perceptual system also accumulates errors (Smith and Vul 2013).
However, the NPE preserves the general intuitive physical dynamics that may roughly be consistent with people's intuitive expectations.” We again thank Reviewer 3 pointing this out; this was an important addition to the paper.
With regard to a comparison of architecture design, a key advantage is that our architecture does not take object relations as explicit input.
Instead, it learns the nature of these relations through training. This makes our architecture more flexible in modeling physical phenomena where relationships may not be explicitly known.
For example, Battaglia et al.’s work require the spring constant to be specified, as well as gravitational attraction relations, whereas our architecture can infer such relations from observing the objects interact during training (for example, the NPE learns that objects with different masses interact differently from objects with similar masses).
Specifically with regards to modeling worlds where forces act at a distance (such as the n-body systems), the neighborhood mask described in the paper would not be as relevant. In our work, the neighborhood mask functions specifically for contact forces, and it is interesting that this broad-phase functionality is an explicit and separate function present in widely-used 2d and 3d physics engines for collision detection.
However, one can view our neighborhood mask as a specific case of a more general mechanism to select context objects, and we believe it would be interesting to investigate learning this more general mechanism in future work. This is an important point and we have added a line in the “Neighborhood Mask” paragraph of Section 2.2 to put the neighborhood mask into more context.
Another key advantage is in function reuse: we demonstrated that the same architecture can be reused for both prediction and inference; we demonstrated that a trained NPE model can automatically infer properties of its input such as mass without further retraining.
In contrast, Battaglia et al. required additional training to perform inference using an additional layer on top of the Interaction Network. These differences are outlined in the Related Work section. This is a good comment and we have made the advantages of our model more clear in the Related Work section.
It is important for us to emphasize that both our work and that of Battaglia et al.’s exhibit many similarities. Core to both our work are the assumptions that 1) there exist objects and 2) these objects interact with each other. The details we mentioned above are distinct to our approach, but we expect Battaglia et al. can also adapt into their model. Both our experiments complement each other in that theirs investigates a wider range of physics scenarios, and ours studies the challenging bouncing-ball environment more in-depth than they do, while focusing on a difficult “common-sense” extrapolation/generalization problem. We see all of these similarities as converging evidence for good representations and model architectures in modeling physics.
Overall, this is a good point from Reviewer3 and we have improved our comparison to Battaglia et al. in the Related Work section.
2. Reviewer 3 mentions a comparison with Fragkiadaki et al. We agree that the experimental settings are different, making it difficult to perform a direct comparison. Our response is similar to that for Reviewer 2, which adapt here. The reviewers also correctly point out that on the quantitative metrics of angular error and relative magnitude, both Fragkiadaki et al. and our work achieve comparable quantitative performance. Reviewer 1 had mentioned in a pre-review question that the above two quantitative metrics do not capture the full story on the model’s performance, observing that the the gap between the NPE and NP/LSTM curves is relatively small compared to the drastic difference in qualitative performance.
To assess qualitative performance, when we directly compare the NPE prediction videos with Fragkiadaki et al. ’s, several specific and qualitative advantages of our approach are evident, which we also summarize in the Related Work Section. Looking specifically at their video for 3 balls, we first observe the presence of random forces in their balls, causing the balls to sometimes magnetically attract each other.
Second, we observe “collisions” (reversals in movement) that occur not on crisp object contact as our balls do, but merely on close proximity (not touching).
Third, (most evident at around 00:27-00:29 in their video) their balls seem attracted to the walls and appear to bounce along the walls even when no attractive force should be present. The NPE does not exhibit these behaviors and preserves the intuitive physical dynamics of the colliding balls; their balls exhibit less realistic behavior along the specific aspects described above.
In addition to these differences, we crucially show strong predictive performance on generalizing to eight balls, five more than the balls in their videos. We also show this performance under stronger generalization conditions, variable mass, and more complex scene configurations. With regards to the billiard table scenario with initial force, we see the purpose of the initial force as providing the balls with initial velocity, since Fragkiadaki et al. do not examine forces applied during the course of the trajectory. Our work also represents initial velocity directly in the object state representation, and investigate balls bouncing in a billiard-table-like environment, where balls bounce off world boundaries and each other.
A fuller treatment of this qualitative evaluation may require a “physics Turing test,” where human subjects compare how realistic the model’s predictions are. We think this is worth investigating for future work, although we are confident that the differences between our prediction videos and Fragkiadaki et al. ’s videos are clear.
We are grateful for the reviewers’ comments on this point; it is because of their comments that we added this comparison into the Related Work section and outlined the specific and distinctive differences between ours and Fragkiadaki et al.’s predictive performance. The link to Fragkiadaki’s prediction videos is in their paper (https://sites.google.com/site/intuitivephysicsnips15/). If there is trouble viewing the videos on the website, one can try viewing it in a Firefox browser. Or alternatively, one can hover over the top-right corner of the video and click the “Pop-out” arrow-like button that appears. This redirects the viewer to Google Drive link, from which the video can be downloaded and viewed. The link to our videos is: https://drive.google.com/drive/folders/0BxCJLi4FnT_6QW4tcF94d1doLWs.
3. Reviewer 3 also suggested performing error analysis with respect to ground truth ball position. This is a good suggestion, and we have added that analysis in Figure 6 of the paper. Here, we analyze the NPE’s performance with respect to the predicted velocity as well as the Euclidean distance between the resulting predicted position and the actual ground truth position. The analysis on the position agree with the velocity MSE results in the bottom row of Figure 3ab: the NPE outperforms all baselines by 0.5 to 1 order of magnitude, and its prediction error is small enough such that its divergence from the ground truth trajectory is due to merely an accumulation of subtle approximation errors, rather than an inability to capture the intuitive physical dynamics.
4.
Lastly, Reviewer 3 is concerned with the potential applications of differentiable physics engines. This is a good comment that deserves attention. We explain the philosophy behind our approach in the Introduction Section, where we see applications for function reuse in “model-based planning and model-based reinforcement learning” by “disentangling the visual properties of an object from its physical dynamics.” For example, Hamrick et al (2017) in “Metacontrol for Adaptive Imagination-based Optimization” show various ways differentiable physics engines can be used for these applications.
We also believe that vision and dynamics can indeed be decoupled, where a vision model can map visual input to an intermediate state space, and a dynamics model can evolve objects in that state space through time, because object detection and segmentation can extract position and velocity, and work like Wu et al (2015) (“Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning”) can extract mass.
For example, by coupling vision with dynamics, a model may be trained to perform well only on bouncing balls that look a certain way. Modeling other balls may require retraining the entire model even when the dynamics are the same, since the parameters of the dynamics module would be coupled with the parameters of the vision model. This is not to say that vision and dynamics should not be combined; both are necessary, but we believe that keeping these modules separate is important for common-sense generalization. With the explicit decoupling between vision and dynamics of a differentiable physics engine like the NPE, the dynamics model is more robust to changes in visual input, because the parameters that govern dynamics are preserved.
Then perceptual models, including computer vision models that don’t require prior training, can be swapped in and out, leaving the NPE intact. Achieving such robustness and generalization is exactly the motivation for the state representation, compositionality, and modularity in our model design.
We agree that modeling physics, whether through generating new code of a symbolic physics engine or learning with a differentiable physics engine, is still a very open research problem. There are several advantages of the differentiable approach.
First, it requires fewer prior assumptions; the NPE architecture essentially makes two strong, but natural, assumptions, which are that 1) there exist objects in the world and 2) they interact with each other in a factorized manner. The NPE then leaves the dynamics, which may be very complex and hard for humans to currently express in code, to be learned from observation. To the best of our knowledge, it is still a challenge to generate source code for such complex interactions in physics, but we agree that it is an important area of research that should be pursued. A neural network is a differentiable program, and we showed that with the important assumptions stated above, it can be trained to model complex nonlinear dynamics of bouncing balls, as we have shown with the NPE.
Paper proposes a neural physics engine (NPE). NPE provides a factorization of physical scene into composable object-based representations. NPE predicts a future state of the given object as a function composition of the pairwise interactions between itself and near-by objects. This has a nice physical interpretation of forces being additive. In the paper NPE is investigated in the context of 2D worlds with balls and obstacles.
Overall the approach is interesting and has an interesting flavor of combining neural networks with basic properties of physics.
Overall, it seems like it may lead to interesting and significant follow up work in the field. The concerns with the paper is mainly with evaluation, which in places appears to be weak (see below).
> Significance & Originality:
The approach is interesting.
While other methods have tried to build models that can deal with physical predictions, the idea of summing over pair-wise terms, to the best of my knowledge, is novel and much more in-line with the underlying principles of mechanics.
As such, while relatively simple, it seems to be an important contribution.
> Clarity:
The paper is generally well written.
However, large portion of the early introduction is rather abstract and it is difficult to parse until one gets to 5th paragraph. I would suggest editing the early part of introduction to include more specifics about the approach or even examples ... to make text more tangible.
> Experiments
Generally there are two issues with experiments in my opinion: (1) the added indirect comparison with Fragkiadaki et al (2015) does not appears to be quantitatively flattering with respect to the proposed approach, and (2) quantitative experiments on the role the size of the mask has on performance should really be added. Authors mention that they observe that mask is helpful, but it is not clear how helpful or how sensitive the overall performance is to this parameter. This experiment should really be added.
I do feel that despite few mentioned shortcomings that would make the paper stronger, this is an interesting paper and should be published.
We thank Reviewer 2 for their helpful comments.
1. Reviewer 2 suggested editing the early part of the introduction to make the text more tangible. We thank the reviewer for this suggestion and have revised the introduction, by adding examples in the first paragraph, and rearranging the ordering of the subsequent paragraphs.
2. Reviewer 2 is concerned that the indirect comparison with Fragkiadaki et al. does not appear to be quantitatively flattering with respect to the proposed approach. Our response is similar to that for Reviewer 3, which adapt here.
While we agree that the similarity in performance on these quantitative metrics (angular error and relative magnitude) may not provide a clear advantage of ours over theirs, Reviewer 1 had mentioned in a pre-review question that the above two quantitative metrics do not capture the full story on the model’s performance, observing that the the gap between the NPE and NP/LSTM curves is relatively small compared to the drastic difference in qualitative performance. This is the reason that following the pre-review period, we added the qualitative comparison in the Related Work section that outlines several specific and evident advantages of our approach. Looking specifically at Fragkiadaki et al.’s video for 3 balls, we first observe the presence of random forces in their balls, causing the balls to sometimes magnetically attract each other.
Second, we observe “collisions” (reversals in movement) that occur not on crisp object contact as our balls do, but merely on close proximity (not touching).
Third, (most evident at around 00:27-00:29 in their video) their balls seem attracted to the walls and appear to bounce along the walls even when no attractive force should be present. The NPE does not exhibit these behaviors and preserves the intuitive physical dynamics of the colliding balls; their balls exhibit less realistic behavior along the specific aspects described above.
In addition to these differences, we crucially show strong predictive performance on generalizing to eight balls, five more than the balls in their videos. We also show this performance under stronger generalization conditions, variable mass, and more complex scene configurations. With regards to the billiard table scenario with initial force, we see the purpose of the initial force as providing the balls with initial velocity, since Fragkiadaki et al. do not examine forces applied during the course of the trajectory. Our work also represents initial velocity directly in the object state representation, and investigate balls bouncing in a billiard-table-like environment, where balls bounce off world boundaries and each other.
A fuller treatment of this qualitative evaluation may require a “physics Turing test,” where human subjects compare how realistic the model’s predictions are. We think this is worth investigating for future work, although we are confident that the differences between our prediction videos and Fragkiadaki et al. ’s videos are clear.
We are grateful for the reviewers’ comments on this point; it is because of their comments that we added this comparison into the Related Work section and outlined the specific and distinctive differences between ours and Fragkiadaki et al.’s predictive performance. The link to Fragkiadaki’s prediction videos is in their paper (https://sites.google.com/site/intuitivephysicsnips15/). If there is trouble viewing the videos on the website, one can try viewing it in a Firefox browser. Or alternatively, one can hover over the top-right corner of the video and click the “Pop-out” arrow-like button that appears. This redirects the viewer to Google Drive link, from which the video can be downloaded and viewed. The link to our videos is: https://drive.google.com/drive/folders/0BxCJLi4FnT_6QW4tcF94d1doLWs.
3. Reviewer 2 suggested experiments analyzing the size of the neighborhood mask. Following Reviewer 2’s pre-reviewe comments, we have now added experiments comparing the existence/non-existence of the mask, shown in Figure 3. We are currently working on a finer-resolution analysis on the neighborhood size, which we expect will be ready by early next week.
We have now included analysis on the neighborhood size in Figure 3d.
Previously we had performed a much coarser search and found 3.5 ball radii as an effective neighborhood radius, but from doing this search in higher resolution, we found that a neighborhood threshold of 3 ball radii empirically worked the best, although the threshold size is quite robust in the range [3, 5]. we added a paragraph in Section 3.3 to explain our analysis. Different neighborhood thresholds may be suited for different domains. We are grateful to Reviewer 2 for this suggestion, because the results from this more in-depth analysis have been enlightening and have shown that there were neighborhood thresholds that worked slightly better, although the performance gain is very small.
Summary
This paper proposes the Neural Physics Engine (NPE), a network architecture
which simulates object interactions.
While NPE decides to explicitly represent
objects (rather than video frames), it incorporates knowledge of physics
almost exclusively through training data. It is tested in a toy domain with
bouncing 2d balls.
The proposed architecture processes each object in a scene one at a time.
Pairs of objects are embedded in a common space where the effect of the
objects on each other can be represented. These embeddings are summed
and combined with the focus object's state to predict the focus object's
change in velocity. Alternative baselines are presented which either
forego the pairwise embedding for a single object embedding or
encode a focus object's neighbors in a sequence of LSTM states.
NPE outperforms the baselines dramatically, showing the importance of
architecture choices in learning to do this object based simulation.
The model is tested in multiple ways. Ability to predict object trajectory
over long time spans is measured. Generalization to different numbers of objects
is measured. Generalization to slightly altered environments (difference
shaped walls) is measured.
Finally, the NPE is also trained to predict
object mass using only interactions with other objects, where it also
outperforms baselines.
Comments
* I have one more clarifying question. Are the inputs to the blue box in
figure 3 (b)/(c) the concatenation of the summed embeddings and state vector
of object 3? Or is the input to the blue module some other combination of the
two vectors?
* Section 2.1 begins with "First, because physics does not
change across inertial frames, it suffices to separately predict the future state of each object conditioned
on the past states of itself and the other objects in its neighborhood, similar to Fragkiadaki
et al. (2015)."
I think this is an argument to forego the visual representation used by previous
work in favor of an object only representation. This would be more clear if there
were contrast with a visual representation.
* As addressed in the paper, this approach is novel, though less so after taking
into consideration the concurrent work of Battaglia et. al. in NIPS 2016 titled
"Interaction Networks for Learning about Objects, Relations and Physics."
This work offers a different network architecture and set of experiments, as
well as great presentation, but the use of an object based representation
for learning to predict physical behavior is shared.
Overall Evaluation
This paper was a pleasure to read and provided many experiments that offered
clear and interesting conclusions. It offers a novel approach (though
less so compared to the concurrent work of Battaglia et. al. 2016) which
represents a significant step forward in the current investigation of
intuitive physics.
We thank Reviewer 1 for their helpful comments.
1. Reviewer 1 asked about how the inputs to the decoder for the NPE and NP are combined. Yes, the inputs to both the NPE and NP decoders are concatenated. Thank you for pointing this out and we have clarified this in the paper. In the “Pairwise Factorization” paragraph of Section 2.2, we added “The sum of encodings of all pairs is then concatenated with the focus object's past state as input to the decoder function.” In the caption for Figure 2, we added that for the NPE, “The input to the decoder is the concatenation of the summed pairwise encodings and the state of object 3” and that for the NP, “The input to the decoder is the concatenation of the summed context encodings and the encoding of object 3.”
2. Reviewer 1 mentioned an argument to forego the visual representation in favor of an object only representation. This is a good point, and we clarify what we mean here. We do not mean that the vision component does not matter; we believe that that both should be incorporated, but decoupled, in an intelligent agent. In the Introduction Section, and reproduced from part of our a response to Reviewer 3, our argument is that though a vision component is necessary in an complete intelligent agent, we are confident that vision and dynamics can be decoupled, and there advantages to doing so.
First, we are that vision and dynamics can indeed be decoupled, where a vision model can map visual input to an intermediate state space, and a dynamics model can evolve objects in that state space through time, because object detection and segmentation can extract position and velocity, and work like Wu et al (2015) (“Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning”) can extract mass.
The advantage to decoupling vision and dynamics is that without this decoupling, a model may be trained to perform well only on bouncing balls that look a certain way. Modeling other balls may require retraining the entire model even when the dynamics are the same, since the parameters of the dynamics module would be coupled with the parameters of the vision model. This is not to say that vision and dynamics should not be combined; both are necessary, but we believe that keeping these modules separate is important for common-sense generalization. With the explicit decoupling between vision and dynamics of a differentiable physics engine like the NPE, the dynamics model is more robust to changes in visual input, because the parameters that govern dynamics are preserved.
Then perceptual models, including computer vision models that don’t require prior training, can be swapped in and out, leaving the NPE intact. Achieving such robustness and generalization is exactly the motivation for the state representation, compositionality, and modularity in our model design. This is a good point and we clarified this argument in the Introduction.
3. Reviewer 1 is also concerned with the novelty with respect to Battaglia et al. (2016). Our response is similar to that for Reviewer 3, which we adapt here.
It is true that we have developed our work independently and in parallel from Battaglia et al., and we think that formal comparisons with their work in future work would be very useful. Physics naturally decomposes into object-object interactions, so both our work and theirs take advantage of this observation. We see the similarities between our work and theirs as strengthening and converging evidence for the approach of object-based representations for modeling physics.
With regard to a comparison of experiments, our experiments focused on worlds with bouncing balls, which Battaglia et al. noted were especially challenging. We also analyzed generalization across variable obstacle configurations beyond simple bouncing balls, which goes beyond Battaglia et al. In this comment the reviewer raises another important point about generalization. We would like to distinguish between two types of generalization. The first type, to which the reviewer refers with regard to the variety in experiments in Battaglia et al., measures a model’s capacity to learn a new and different concept through training.
While the reviewer correctly points to the need to test the system on varying domains, our architecture makes few assumptions ((1) it assumes that there exist objects in the world and (2) that objects interact with objects in their context) so does not have to be restricted to only domains of bouncing balls. As expressed in the discussion section we plan to apply future iterations of the NPE on other worlds such as with block towers and liquids. The flexibility to learn new physics concepts is a core reason why we chose to learn a differentiable physics engine, where the various physics can be learned through training. The second type of generalization involves extrapolation of “common-sense” reasoning.
For example, a model that learns about physics with fewer balls should not have to be retrained to learn the physics of more balls. A model that learns about interactions between balls and obstacles should be invariant to the scene configuration because the scene configuration should not matter if the underlying physical laws remain the same. This type of generalization involves a difficult transfer of knowledge because it involves no retraining, and we studied this type of knowledge transfer in-depth in this paper with two generalization tasks (variable object count and various scene configurations). Performing this type of generalization is exactly our motivation for the state space representation and model architecture with regards to factorization and compositionality.
For example, expressing physics modeling as a local computation with a focus object and context objects allows the model to scale to arbitrary numbers of objects. Expressing a large structures in the scene as composed of smaller building blocks allows the model to be invariant to geometries of the large structures because they merely decompose into different spatial arrangements of the building blocks.
With regard to a comparison of modeling performance, following Reviewer 3’s pre-review comments about a comparison to Battaglia et al., we also added the following lines to the paper, reproduced here: “Like Battaglia et al. (2016) our predictions can be effective for a large number of time steps even though we only train to predict the immediate next time step…As can be seen by the videos, while the NP/LSTM fail to predict plausible physical movement entirely, the NPE's predictions initially adhere closely to the ground truth, then slowly diverge due to the accumulation of subtle errors, just as the human perceptual system also accumulates errors (Smith and Vul 2013).
However, the NPE preserves the general intuitive physical dynamics that may roughly be consistent with people's intuitive expectations.” We again thank the reviewer for pointing this out; this was an important addition to the paper.
With regard to a comparison of architecture design, a key advantage is that our architecture does not take object relations as explicit input.
Instead, it learns the nature of these relations through training. This makes our architecture more flexible in modeling physical phenomena where relationships may not be explicitly known.
For example, Battaglia et al.’s work require the spring constant to be specified, as well as gravitational attraction relations, whereas our architecture can infer such relations from observing the objects interact during training (for example, the NPE learns that objects with different masses interact differently from objects with similar masses).
Specifically with regards to modeling worlds where forces act at a distance (such as the n-body systems), the neighborhood mask described in the paper would not be as relevant. In our work, the neighborhood mask functions specifically for contact forces, and it is interesting that this broad-phase functionality is an explicit and separate function present in widely-used 2d and 3d physics engines for collision detection.
However, one can view our neighborhood mask as a specific case of a more general mechanism to select context objects, and we believe it would be interesting to investigate learning this more general mechanism in future work. This is an important point and we have added a line in the “Neighborhood Mask” paragraph of Section 2.2 to put the neighborhood mask into more context.
Another key advantage is in function reuse: we demonstrated that the same architecture can be reused for both prediction and inference; we demonstrated that a trained NPE model can automatically infer properties of its input such as mass without further retraining.
In contrast, Battaglia et al. required additional training to perform inference using an additional layer on top of the Interaction Network. These differences are outlined in the Related Work section. This is a good comment and we have made the advantages of our model more clear in the Related Work section.
It is important for us to emphasize that both our work and that of Battaglia et al.’s exhibit many similarities. Core to both our work are the assumptions that 1) there exist objects and 2) these objects interact with each other. The details we mentioned above are distinct to our approach, but we expect Battaglia et al. can also adapt into their model. Both our experiments complement each other in that theirs investigates a wider range of physics scenarios, and ours studies the challenging bouncing-ball environment more in-depth than they do, while focusing on a difficult “common-sense” extrapolation/generalization problem. We see all of these similarities as converging evidence for good representations and model architectures in modeling physics.
Overall, this is a good point from Reviewer1 and we have improved our comparison to Battaglia et al. in the Related Work section.
We thank all reviewers again for the thoughtful and in-depth reviews. The pre-review period helped us improve the paper by adding in comparisons with Battaglia et al. (2016) and Fragkiadaki et al. (2015), and clarify several arguments in the paper. The review period helped us add more specific and detailed comparisons with Battaglia et al. (2016) and Fragkiadaki et al. (2015). The review period also helped us add a numerical error analysis on the predicted position, add additional experiments analyzing the neighborhood mask, and clarify further arguments in the paper. We have uploaded our revised version.
Many reviewers mentioned a comparison with Fragkiadaki et al. (2015). In our review responses, we highlighted several specific advantages to our approach that are evident from comparing the videos. The link to Fragkiadaki’s prediction videos is in their paper (https://sites.google.com/site/intuitivephysicsnips15/). If there is trouble viewing the videos on the website, one can try viewing it in a Firefox browser. Or alternatively, one can hover over the top-right corner of the video and click the “Pop-out” arrow-like button that appears. This redirects the viewer to Google Drive link, from which the video can be downloaded and viewed. The link to our videos is: https://drive.google.com/drive/folders/0BxCJLi4FnT_6QW4tcF94d1doLWs.
While updating the introduction in the latest revision of the paper, we accidentally left a duplicate paragraph in the introduction. We now have uploaded a revision of the paper with that duplicate removed.

Meta-review:
role: PC
data: comment: The paper proposes a Neural Physics Engine for predicting intuitive physics. It is able to model the objects' dynamics and pairwise object interactions in order to build predictive models. This is a very nice direction, as also noted by the reviewers. One reviewer was particularly enthusiastic, while the other two less so. The main concerns were similarities to existing work, which however can be considered as done in parallel. The reviewers also had comments wrt evaluation, which the authors addressed. This is a good paper, and the AC recommends acceptance. The authors are also encouraged to look at "Learning Multiagent Communication with Backpropagation" by Sukhbaatar, whose method (albeit applied in a different context) seems relevant to the proposed approach.

Decision: Accept (Poster)