Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to add explicit/human-defined constraints to learning-based simulation frameworks, where a learned constraint function implicitly regularizes the dynamics, and future predictions are generated via a constraint solver. The authors built the framework on top of graph neural networks (GNNs) to capture the compositionality of the underlying system and enforce the constraint using an implicit constraint function optimized via gradient descent.

The authors tested the proposed method in four physical simulation environments, including rope, bouncing balls, bouncing rigids, and BoxBath. Experiment results show that the proposed C-GNS has a competitive or better performance compared to prior learned simulators. In addition, they have also demonstrated that C-GNS can generalize to unseen, hand-designed constraints by applying more solver iterations than experienced during training to improve the accuracy on larger systems.
main_review: [Strength]

This paper tackles an important question of how we can incorporate prior knowledge in the form of explicit physical constraints in the learning-based simulators to enable better generalization.

The experiments on four environments show that the proposed method can deliver better prediction results than unconstrained baselines.


[Weakness]

While I like the direction this paper is going, I have concerns regarding the expressiveness of the experiments and the missing details of the method.

Prior methods on learning-based physics simulators have shown results on a set of much larger-scale and more complex environments involving fluids on rough terrain, fabrics with novel geometries, etc. [1, 2]. The experiment environments used in this paper may be a bit too simple compared to what's out there in the literature, making it hard to know how the method works in larger and more complicated scenarios.

Continuing from my previous point, [1] showed generalization to environments with drastically different geometry than seen during training, and [2] showed that the model could scale up to significantly larger and more complex cloth than seen during training. Adding constraints based on our understanding of physics is supposed to improve the model's generalization ability. As a result, I don't think the current experiments in the paper are enough to demonstrate the benefit of the constrained optimization process. The authors should consider including concrete experimental evidence on how the incorporated constraints may lead to even better/larger-scale generalization than what's already shown in the literature.

The authors should also consider including more details on how they construct the constraint function, f_C, e.g., for the fluid, rigid object, boundary conditions, etc. Without further details, it is hard for me to imagine how they are defined and implemented.

Related to my previous point, do we need any assumptions on f_C other than being differentiable? For example, for discontinuous events like contacts, I imagine we can differentiate through the LCP constraints, but how useful are the gradients?

How much more computing resources and time are needed to apply gradient descent on the constraint function? Multi-step message passing and solver iterations do not come for free. It may significantly increase forward prediction time for each time step. Therefore, it is essential to provide the time spent on each forward pass for different design choices and discuss the trade-off between the performance gain and the decrease in computational efficiency.

According to Figure 3, it is a bit hard to know which of the following three methods works better: (i) Iterative GNN, (ii) C-GNS-GD, and (iii) C-GNS-FP. When giving a new scenario, is there a way to know which one we should use, or should we try all of them and choose the one that works the best?

How did you choose the weight on the constraint term when incorporating novel constraints at test time? From the video, the rope is jittering. What might be the reason, and is it possible to resolve it? [1] shows generalization results of fluid simulation on unseen terrains much different from what the model was trained on. [3] also showed examples of generalization to unseen obstacle configurations. How would you compare the way your paper and [1, 3] incorporate new boundary constraints during testing?


[1] Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, Vladlen Koltun, "Lagrangian Fluid Simulation with Continuous Convolutions"
[2] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, Peter W. Battaglia, "Learning Mesh-Based Simulation with Graph Networks"
[3] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, Peter W. Battaglia, "Learning to Simulate Complex Physics with Graph Networks"


===================

[Post Rebuttal]

I thank the authors for the detailed feedback, which addressed most of my concerns. I hope the authors can incorporate the response into the manuscript to improve its clarity. I'm happy to raise my score from 5 to 6.
summary_of_the_review: While I like the direction this paper is going, I have concerns regarding the expressiveness of the experiments compared to previous work on learning-based physics simulation, e.g., extreme extrapolation generalization on systems much larger than what the model was trained on.

There are also a lot of missing implementation details, making it hard to know (1) how the authors define and solve the constraints, e.g., for fluids, rigid objects, boundary conditions, etc., (2) how hard the constrained optimization problem really is, and (3) how does it affect the final performance.

As a result, I currently lean towards the rejection side.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents to simulate physics via a constraint-based approach instead of direct prediction. In particular, the authors first employ GNN taking as input the history positions and dynamics, to capture the interaction between different particles within the system, whose output is considered as the constraint satisfaction scalar. Then, the gradient with respect to the constraint function is applied as the update of the dynamics over a certain number of solver iterations. The experiments are conducted on a variety of challenging physical domains, including simulated ropes, bouncing balls, colliding irregular shapes and splashing fluids.
main_review: Strengths:

1.	This paper is overall well written. The authors have clearly demonstrated the pipeline of the proposed strategy.

2.	Although the novelty of combining GNN with constraint projection is weak (see the weaknesses below), it is valuable to check if this method can outperform those typical forward approaches (such as the work by Sanchez-Gonzalez et al., 2020). The experimental evaluations generally serve this purpose. 


Weaknesses:

1.	The biggest concern is that the novelty is weak. At its core, this paper applies the pipeline in (Sanchez-Gonzalez et al., 2020), including the strategy by first computing the predictor and then updater, and the usage of GNN for interaction modeling. The main difference is that for the predictor, it replaces the traditional forward prediction with the iterative gradient-based solver of the constraint approximator, which is interesting. Yet, this idea has been proposed by Yang et al. (2020) in terms of the iterative projections along the gradient direction of the constraint network. Although the authors have further augmented the input of the constraint function with Y to take the dynamics into account, this modification seems minor and a straightforward enhancement.  

2.	Regarding the constraint satisfaction. The authors first predict the changes in position (Y) and then update the next state X_{t+1} via an Euler integrator. Even the prediction of Y is derived via the constraint solver, the constraint will be broken after the following updater from X_{t} to X_{t+1}, which is problematic to maintain the hard constraints (such as the case in Figure 6 (c) where the movement within walls and floors is forbidden). How does the proposed method tackle this issue? In the work by Yang et al. (2020), the authors use an opposite order by first updating X_{t} and then projecting the positions onto the constraint manifold, which is able to meet any kind of constraints.

3.	For the comments above, there are several important baselines that are not tested in the experiments: 1) using the method by Yang et al. (2020) but with the GNN projector; 2) first updating X_{t} and then predicting Y with other setting unchanged in the current framework; 3) augmenting the explicit simulators (both the iterative and non-iterative versions) with a regulation loss to enforce certain hard-crafted constraints such as the cases in Figure 6 (c). 

Other comments:

1.	This paper is almost well organized, but there are still some confusions in the current version. 
1.1 In introduction, the authors mention that both families of simulators (explicit forward vs implicit constraint-based). Is this statement discussed in previous papers? Is there any citation?
1.2 In related work, the authors state that the neural projection by Yang et al. (2020) is the first work that uses learned constraints. However, they also introduce that “Recent methods have been proposed for learning constraint functions and solving them in a model’s forward pass” such as “Deep Implicit Layers” and “Deep Equilibrium Models”. These two statements seem self-contradictory.
1.3 In section 4.3, the authors claim that in Figure 3 the C-GNS-GD’s performance was generally better than the other model variants, which is not true. In terms of the rollout MSE, Iterative GNN outperforms C-GNS-GD in three out of four cases. The authors are suggested to provide more explanations here.

2.	It is good that the proposed method is translation-invariance. Yet, besides this symmetry, there are other cases, such as rotation invariance/equivariance. This is important for improving the generalization ability of the simulator, given the fact that if we rotate the input states under a certain angle, the output changes in the same way. Have the authors taken this symmetry into account? 

summary_of_the_review: Overall, given the limited novelty and insufficient experimental evaluations, I initially suggest weak rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes to learn the numerical solutions to Lagrangian physical simulation with a constraint-based inference method on graph neural networks. This method involves an iterative update during inference and thus enables test-time dynamical correction. The contributions are: (1) this manuscript builds a scalar predictor to indicate how well the constraint is agreed. (2) this manuscript proposes using the graph neural networks as the backbone to deal with a variable length of the physical domain. They also examine the effectiveness with a bunch of experiments including the state prediction experiment on four different environments and multiple ablation studies towards different hyper-parameters.
main_review: This manuscript extends an idea from [1] that the learning of physical simulation can be viewed as a constrained optimization problem. There are a few interesting points:

1. The authors extend it using graph neural networks. This modification generalizes this method to the indefinite number of states in the physical system.
2. The authors claim that this method trade-off between inference time and accuracy dynamically so that in the test time they are possible to increase the number of steps for better convergence.

Despite these respectful contributions, I still have a few questions and reservations:

1. I notice different activation functions are applied to different environments. Are they intentional? Is there a guideline for choosing?
2. LayerNorm will be affected by the magnitude of the static information. For example, if you simulate under the continuous mechanism, Young's Modulus can be so huge that kills other entries in the vector. I wonder how should the user deal with it.
3. How does this method generalize to the static information and time step?
4. Second-order optimization usually provides super-linear convergence when it is close to the optimum, which is a similar case referring to figure 5. I will find a comparison between the proposed first-order method and Newton's or quasi-Newton's method helpful.
5. The authors claim that adding an $\alpha$ improves the generalizability to unseen parameters including articulation length and optimization steps. Is there a reason why it is not a part of the standard model given it has an advantage?
6. 3 and 5 are separately used in different environments. Some discussion on the choice will be helpful (I understand 5 is from previous work). Is it because of the time integration method used in Mujoco?
7. My major reservation is the technical novelty compared to its preceding work [1]. I find the extend to graph neural networks exciting, but in the meantime, incremental. I wonder if the authors can give me more explanation on the delta between them, maybe supported with a more impressive application.

[1] Yang, Shuqi, Xingzhe He, and Bo Zhu. "Learning Physical Constraints with Neural Projections." Advances in Neural Information Processing Systems 33 (2020): 5178-5189.
summary_of_the_review: I appreciate the idea of using graph neural networks with a constraint-based learning paradigm. However, there are also a few technical questions remaining unsolved. I personally find this work is not satisfying me with solid novelty, but I am open to change if the authors address my concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a neural-network simulator that learns to solve constraints inspired by classic physics-based simulation. The proposed simulator uses a graph neural network to encode a constraint solver and shows results in a number of simulation environments.

The main contribution of the paper is its idea of using graph networks to encode constraint-based simulation.

main_review: 1. I find the notations X and Y with and without hats very confusing from the beginning. It would be great if the paper could clarify the semantic meaning of these notations.
* Fig. 1 (a): its caption says the UPDATER uses $\hat{Y}$ to update $X_t$, but the figure shows an UPDATER using $Y$ to update $\hat{X}\_t$. The notations in Fig. 1 (a) itself is extremely confusing: it starts with $X_1$ (without a hat), proceeds to $\hat{X}\_t$ (with a hat), but then uses $X\_{\leq t}$ (without a hat), and finally predicts $\hat{X}_{t+1}$ (with a hat). How did you determine when to use a hat and when not?
* When defining the rollout, $X$ with and without hats are mixed. From the rollout’s definition, it starts with $X_t$ but is followed by $\hat{X}\_{t+1}$, $\hat{X}\_{t+2}$, …. . Does this mean you manually choose t and separate the whole rollout into two parts: before $t$ every $X$ is without a hat and after $t$ every $X$ is with a hat (generated by the simulator)?

2. I do not see a strong reason for introducing the explicit iterative simulator for comparison. It looks like $f\_{DI}$ can be rewritten as $f_D$ plus a few linear operators: $f\_{DI}(X_{\leq t}) = (f_D(X\_{\leq t}) - X_t) / N$. Therefore, I would expect $f_D$ and $f\_{DI}$ to have very similar capabilities. What insight would we expect to get from comparing them?

3. It looks like the proposed constraint-based predictor runs a fixed number of gradient-based iterations and a fixed step size without guaranteeing the constraint is satisfied eventually. Therefore, I feel it is a bit too much to claim that the proposed approach is doing constraint-based simulation and to claim that the proposed network is a constraint solver. I think it would be necessary to either tone down this claim in the title, abstract, introduction, etc., or state explicitly at the beginning of the paper that the proposed approach does not guarantee the constraints are always satisfied.

4. The network design ensures translation-invariance by taking as inputs position offsets instead of absolute positions. Does the network also ensure rotation-invariance?

5. “These node-wise constraint values are averaged to compute a single scalar c constraint for the entire system”. This seems questionable to me if the goal is to solve constraints $f_C(X\_{\leq t},  \hat{Y}) = 0$, in which case I would expect that using an average of $|c|$ or $c^2$ makes more sense.

6. Looking at the experimental task domains, I feel there are two types of constraints involved in the simulator: equality constraints (e.g., end points of the consecutive rope segments must share the same location) and inequality constraints (e.g., bouncing balls must have nonnegative distances to the boundary). Does this paper deal with both equality and inequality constraints?

7. Both N and lambda seem to be crucial hyperparameters that need to be chosen for different environments individually. If this approach is applied to a new environment, how would you determine a proper N and lambda?

8. “In principle, iterative and constraint-based simulators should find more accurate solutions by increasing the number of solver iterations, N.” I am not sure I fully agree with this claim because the iterative solver is not equipped with a line search algorithm that adaptively changes the step size.

9. The large rollout MSE (orders of magnitude larger than #timesteps x One-step MSE) in Fig. 3 bottom seems to imply the learned simulator is not a good replacement of a numeric simulator because it accumulates errors from all time steps. I understand this may be a common issue that many other neural-network-based simulators also suffer from (all baselines in Fig. 3 have accumulated substantial errors and produced large rollout MSE), but I am still wondering whether the authors could give people a strong reason why it is useful to develop such a neural-network-based simulator if it is not accurate.

10. Similarly, I wonder if this paper could provide more discussions on the generalizability of the learned network on the static properties of the environment (the Z vector in the main paper), e.g., density, material types, time step size, etc. My understanding is that the network needs to be retrained if Z is updated, or the training set needs to be augmented to see various Z values. This does not seem to be very ideal for a simulator. Again, I understand this may be a common problem for many neural-network-based simulation papers, so I won’t hold it against this paper too much.

summary_of_the_review: I recommend rejection based on the current status of the paper. My concerns are listed above.
review_title: null

Meta-review: The paper uses graph-based neural networks to ensure constraint-based simulation.  Even though the approach is a good one, it is only incremental w.r.t. the work published by Yang et al at NeurIPS in 2020; then, the experimental section is not convincing enough.

While the authors indicate their dissatisfaction with one of the reviewers' assessment, the overall reviews of the paper are not very positive.
Decision: Reject