Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a learned model specialized on learning Lagrangian fluid dynamics for incompressible fluids. The model is a hybrid between a simulator with explicit advection, collision and pressure correction stages, and a learned model, trained by supervising each of those stages. The authors demonstrate improved stability/conservation of physical properties for a model, and some flexibility to the time-step being changed at test time.

The main general disadvantage I see is that the model seems oddly specific to the simulator used to generate the data, and specifically to incompressible fluids. The authors say “Although our model is entailed and customized for fluid simulation, it can be extended to simulation of other dynamics under Lagrangian framework as it takes universal feature“, but I am not sure how this work specifically supports extensions, since the main difference between the model and the baselines is precisely that while the baselines are general, this model specializes itself on one very specific type of simulation. Beyond this, there are some other important questions that I have listed below.

The advantages/claims are improved stability of divergence and density, and less learned parameters. The stability is not too surprising, since the model is built exactly to match the mechanisms that the simulator has to improve those, and in a sense it is using privileged information about the data generation. I would argue that it would have been easier to achieve the same in the baselines, just by adding additional loss components that explicitly minimize divergence and density error, using the same use of the privilege information. The smaller number of learned parameters is a nice result that potentially means the model can also run much faster than the baselines, so maybe this is the most important contribution: even assuming the model is meant to be specific for a domain, it can bring value by showing that it can be much faster than the baselines, and much faster than the ground truth simulator itself. Otherwise, I am not sure what the main selling point of a highly domain specific architecture is in this case.

All things considered, I am not sure the current state of the paper is sufficient to be accepted, both in terms of generality of the approach/main selling point, quality of the model description and comprehensiveness of the results. So my current rating is Reject (4) which I would be happy to immediately raise to Weak Reject (5) if the authors can confirm that they do not need access to intermediate simulator targets for training (Q1), and reconsider from there, given satisfactory explanations, additional evidence and discussion with other reviewers.

Main Comments/Questions:

1. The papers says:

> We train three networks, advection net, collision net and pressure net separately.

But also things like:

> the advection net predicts acceleration of particles (a^adv)
> The collision net takes … the relative position and velocity in intermediate state (let’s call this x*, v*)
> predicts correction to the velocity (Let’s call this \Delta v*, and the output v**)
> the updated intermediate position and velocity are take n as input by the pressure net (x*, v**0)
> predicts pressure (let’s call this p)

What are the inputs and targets of the advection net, the collision net and pressure net at train time, respectively? Does this means that apart from x^n and x^{n+1} your model needs access to all of the internal values of the simulator after each internal stage: (e.g., a^adv, x*, v*, v**, p), so you can supervise those when training each network separately. This would be a pretty strong requirement that the baselines do not require. How would you run this model on data obtained from a real experiment (assuming you had a way to do particle tracking, of course)? Or are in on the contrary the three models trained in order, always taking as inputs the outputs form the learned previous stage, and building the loss against the final position and velocity targets? The section around Equation (16) seems to imply that it is the former. If that is the case, improvements over baselines seem like a small gain considering the required access to all of that simulator-specific intermediate information.

2. In general the model description is quite hard to follow. For example, does the edge-focused and node-focused graph networks have a single step of message passing (later you talk about two layers of aggregation, but it is not clear how these two relate). Does the node focus graph network not embed the nodes features feature the aggregation? A lot of these decisions seem to deviate a bit from standard choices, and should probably be justified and explained more in detail. 

How is \Nabla p (gradient of the pressure) calculated, do you actually take the gradients of the neural network with respect to its own input positions? Or do you use equation (19) to get a finite different estimate? Assuming it is the latter, is then the model not very sensitive to the connectivity radius of this network? 

3. How is density error and velocity divergence defined? Specifically, in Figure 6, density deviation takes both positive and negative values, but in the table it takes only positive values. 
Assuming this is calculated with equations (18) and (20), are these metrics not highly dependent on the neighborhood radius chosen? Is the neighborhood radius the same for the model aggregations than for the metrics? In that case, why is that?

Would it make sense to also report error as a function of the mse of the position somewhere, which has been the main metric in the baselines work?

4. 
> we can observe oscillation on the free surface of fluids
> maintains a much more compact and smoother shape
> CConv fail to maintain smooth and compact fluid distributions
> GNS’ prediction is significantly slower than ground truth

It would also be very useful to be able to see some videos of the trajectories for the different models, to better understand the differences that the authors mention, rather than a single trajectory per dataset sampled at 5 points. Also, what does “slow” mean in this context?

5. In Fig 2(a) the first frame for GNS (t=100), seems very different from the rest, what is the reason for this? In similar DamCollapse domains in the GNS paper the accuracy seemed better somehow. Was the noise used to adjust the targets too, similar to the GNS model?

6. Any reason why the radius of connectivity is set differently for the advection/pressure net than for the collision net? Was the model sensitive to this?

7. Results section seems too short, only 16 lines worth of text, compared to two pages worth of tables and figures. I think more discussion is needed, and maybe more investigation of what specifically makes the model better. For example:

* Could the model be trained end to end, instead of training the three networks separately?

* The generalization section does not really elaborate enough about what the differences between training and generation datasets. The problem partly is that the dataset generation description itself is not very comprehensive, and seems hard to reproduce without access to the source code.

* The generalization to other time-steps should be studied in more detail, both how it was implemented, and what the results are for more both shorter and longer timesteps (rather than just the qualitative figure 3, with a single generalization time-step), and how specifically was the attempt to make it work for the baselines implemented.


Minor comments/typos (did not affect my decision):

“physic-informed” should be “physics-informed”?
“we aggregates” → (without s)
“are passed to processor” → “to the”
W in equation (13) is not defined in the text. Possibly add W next to: “using smooth
kernel as weight function”

The value for max density error, Dam collapse, ground truth, seems different in Table1 and Table2.

I would maybe put ground truth as the first row of the results tables for easier comparison to the first row, which has the strongest performance.

In Figure 6, could you make it so the color legend is consistent across plots, currently it is very confusing.

And in general the grammar needs a bit more work (especially in the 3.2 model section).

review_title: Very strong inductive bias for a very specific simulation type

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper presents a graph neural network approach specialized for Lagrangian fluid simulation.
Instead of an end-to-end approach a multi-step solution is proposed which divides the computations into 3 distinct networks.
The structure of the intermediate results computed by the networks follows the structure of solvers like PBF.
A goal of this approach is to more accurately model physical properties of fluids.
The experiments show comparisons with recent state-of-the-art methods and compare different information aggregation schemes.


Score:
Although I think the idea is interesting I tend towards reject. 
The reason for that is mainly the evaluation, which uses very limited data (number of scenes, sequence length, variety) and is not very convincing.
This can be fixed by increasing the test set size and even test on datasets from other works with more variety.
Another reason is the very specific network design with multiple small parts in combination with the small timesteps.
In comparison with a single end-to-end trained network this approach is less practicable.
Additional experiments can help to strengthen the design choices that were made.


Strengths:
* The paper examines important physical quantities such as the density error and velocity divergence.
* The method is parameter efficient and works with different time steps without retraining.
* The description of the implementation is easy to follow and Figure 1 gives a good overview.


Weaknesses:
* The proposed network structure is very specific to the problem and the solver used for generating the ground truth.
  Each of the 3 networks is trained separately imposing the need to compute the intermediate quantities for each of the networks.
  The sequential network design allows to factor out quantities like the time step which helps with generalization but the strong supervision and the sequential structure also significantly limits the power of the network. 
  This aspect needs to be investigated in detail as this distinguishes this work from other approaches which train end-to-end.
  It is possible to backpropagate through the update scheme (equations 7 to 12) during training and compare this to the proposed solution.
  Another comparison can be to remove the intermediate results and let the network output all desired physical quantities in the last layer.
* The evaluation (Table 1,2) seems to be based on only 2 sequences, which is not enough.
  The description of the dataset generation process is very short and important information is missing.
  Are there different types of obstacles or only cylinders? Does the number of particles differ for each scene? What is the number of fluid blocks per scene?
  Without knowing the variance of the training data and a small test set it is not clear if the evaluation is meaningful.
* The sequences are very short. The sequence length is only 2 seconds. 
  This is not enough time for the fluid to reach a steady-state, thus the behavior of the model for this particular but important state is not well studied.
* The generalization to different time steps is good but the generalization to different scenes seems very limited.
  All scenes are box-like environments.
* Learning physics is often motivated by reducing computation costs but there is no information about the computation time.
  Further, the time steps used in the paper are quite small, which increases the number of iterations needed for long simulations and significantly simplifies the learning task.


Questions:

Section 5.1 mentions that the particle density inputs are normalized for the pressure network.
This is counterintuitive since density is an important feature for computing the pressure.
Why is this necessary to stabilize the training?



Minor comments:

There are some minor problems with the writing. Here are some for the first page:

Abstract: 
- Our model uses _a_ graph ...

Introduction:
- fluids is an essential -> fluids _are_
- a large class of numerical models have -> a large class of numerical models _has_
- usually increase drastically when resolution -> usually _increases_ drastically when _the_ resolution
- data-driven model -> data-driven _models_
- dynamics under Lagrangian -> dynamics under _the_ Lagrangian
- universal feature (...) under Lagrangian framework as input -> universal _features_ (...) as input 

Related Works:
- build upon Lagrangian representation of fluid -> build upon a Lagrangian fluid representation
- when material interface -> when the material interface

Will the code be released? If not it would be important to have a section in the appendix with all the important parameters for reproducing the networks in tabular form (layer sizes, normalizations, activations, etc.).
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary
The paper presents a method for learning Lagrangian fluid dynamics from MPS data. By injecting domain knowledge, and separately training subcomponents of the solver, it achieves low error rates on e.g. divergence.

### Recommendation
There are a lot of different approaches for learning physical dynamics, which differ in the amount and type of inductive biases and domain knowledge injected. This is often a tradeoff between generality versus accuracy, and the question of what is a useful balance that has actual advantages over classical solvers is still open.

That said, I find it hard to make the case for the balance chosen in this paper; the architecture is limited to data from a specific solver (MPS) so unlike general end-to-end approaches (e.g. GNS [Sanchez et al 2020], CConv [Ummenhoffer et al 2019]) it's not a pathway to learning from general data. On the other hand, unlike methods that use a lot of domain knowledge to be more efficient than the ground truth solver (e.g. [Ladicky et al.]), this paper doesn't demonstrate any concrete advantages over MPS. It also doesn't study the impact of its individual choices in great detail, and does not back up some of its claims (generality & generalization, see detailed comments). In its current form, I'm not sure we will learn very much from this method, and hence I recommend rejecting this paper.

### Detailed comments
- Model: The architecture is very close to the SPH/MPS algorithm in structure. The main difference to a hard-coded solver seems to be that the interpolation kernel is learned, and there is a processor MLP after each block which can act as a corrector. In between sub-block, the output is projected back to position, velocity, so the NN can't learn a richer intermediate representation. The biggest limitation I see it caused by supervising the 3 subcomponent separately. If I understand this correctly, this means like you a) can only train on a solver which has exactly these subcomponents (i.e. MPS, even SPH works slightly differently), b) you need access to solver internals, and c) you will just learn to replicate the solver internals-- and if you can't learn e.g. a better kernel than the solver, why learn it at all?
- Claims: 
  1. Abstract: *"our model can adopt a range of time step sizes different from ones using in the training set"*. While different step sizes are *mentioned* in the results, there is no numbers, figures or other details to substantiate this claim. 
  2. Conclusion: *"Our model also has generalization capability, where it can remain stable when extrapolating to a wide range of different geometries"*. The only result related to this I could find is fig. 4. The dataset description was a bit vague, so it's a bit unclear to me that this really is an extrapolation test. Was the cylinder among the objects in the training set? In any case, I don't think a test on a single cylinder obstacle support the statement of "a wide range of different geometries".
  3. Intro: *"...it can be extended to simulation of other dynamics under Lagrangian framework"*. I'm not sure this is the case; all the specific details which make this method different from other graph-based approaches *are* specialized to fluids, specifically MPS. How would you e.g. extend this method for elastics or granular materials? For those, there is no pressure or density (depending on approach), but strain/stress etc., and it's unclear which sub-module you'd need or how to supervise them. I'm not even sure this approach would work for Lagrangian fluid data from a different solver (e.g. MPM).
  4. Although not a direct claim, the method is motivated by the computational cost and bad scaling behavior of classical solvers. However, is this method actually faster than the GT simulator, or can demonstrate better scaling? There's no results on this, and since the method is directly supervised on individual solver components it'd actually be surprised if that is the case.
- Results: The main result seems to be that divergence is better preserved than in end-to-end approaches like GNS and CConv. This finding is not very surprising, considering that this method directly supervises on the pressure correction module, while the baselines do not. I find it more surprising that the difference to GNS is rather small in fig. 6, even though it does not have a notion of pressure at all.

### Additional questions
- Just to make sure I have understood the method correctly, how exactly to you train the 3 modules? I.e. where do the labels for supervision come from, are those taken from the corresponding modules in the ground truth simulator?
- Where does the density come from? Is this computed with a hard-coded interpolation kernel like in SPH? And if yes, why-- as the other modules are specifically build around learning a kernel?
- Where do you see the advantages of your method compared to using the ground-truth solver?

### Further comments to the authors
While I recommended rejecting this paper in its current form, I think that this is an exciting area to perform research in, and there are many directions worth exploring. I think it's worth thinking about specifically what advantages a learned or semi-learned method can have over ground-truth solvers, and then finding ways of capitalizing on them. Some papers have explored performance, or generality, and the elephant in the room is how to learn from actual real-world sensor data while still exploiting domain knowledge. Also, there's a multitude of papers pushing a specific method, but very little work on thoroughly studying the concrete impact of all the little tricks, choices in representation, and the amount and concrete form of domain knowledge injected.

---
Update: The added generalization tests and performance numbers strengthen the paper, and make the aim more clear. The paper seems now mostly focused on accelerating MPS (or potentially other SPH variants). While the new result table shows speedups compared to MPS for model inference, in practice the model still is bottlenecked by NN search, and total runtime of inference+NN is not significantly different to MPS. And if speed is the aim, the appropriate baseline comparisons should be to methods which -- as this method-- use domain knowledge and solver internals to speed up the solver (e.g. Ladicky et al.).
On the method side, it does seems like the proposed approach is limited by supervising specific MPS subcomponent separately, which hinders learning richer intermediate representations, applying the model to other systems, or taking significantly larger timesteps. There could be potential benefits of the specific architecture chosen, but their effect is a bit unclear; as for the comparisons shown the most significant effect is supervision on subcomponents.
I will therefore keep my score.
review_title: Recommend to reject

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper deals with the prediction of 3D Lagrangian Fluid Simulations. Therefore the problem is divided into 3 subproblems, oriented on numerical simulations. An advection part, where the acceleration of the particles is calculated, a collision step, where the boundary effects are included, and a pressure prediction part, where the pressure for maintaining the volume is determined. A graph-based network is used for each part, which is either node or edge-based according to the requirements. 

I think it's a good idea to divide the problem into single subtasks and to orientate on the numerical method. I also find the evaluation of physical properties like density and velocity divergence very relevant and show in my eyes the superiority of the method compared to the state-of-the-art methods. However, there are still some open questions: 
* It is mentioned that tests with different time steps were made as proof of generalization, but I cannot see any corresponding results. 
* Also, I'm not quite sure what the advection network and the collision network calculate exactly. I think it would be more logical if the advection network also performs the integration step, so you could approximate higher-order integration schemes. With the collision network, I'm not quite sure what the GT should be. In most SPH solvers the boundaries are included directly in the pressure computation and not in a separate collision step. In general, I would like to have an ablation study where the relevance of the individual components is more obvious. 
* Another thing is that force-based SPH methods, as adapted in this paper, are usually less stable than position-based methods (PBF), from which state-of-the-art methods are based. I would therefore be interested in a comparison in an extreme scenario, e.g. a setup with a very high water column. I could imagine that the method would fail.
* Finally, I would recommend adding some videos to the supplementary material, which could prove the temporal coherence of the method.

There are a few things I noticed about the text itself:
* First, there are some linguistic errors in the text. 
* Furthermore I would recommend naming the intermediate results in the equations (e.g. equation 10) differently. This can be a bit confusing. 
* As of the last point I found the section about network architecture a bit short and not very informative. I would make it a bit longer and more detailed.

All in all, I find the method interesting, but in my eyes, there are still some missing points. Unfortunately, I think that the text needs a revision in general because there are quite a lot of linguistic errors. Therefore I would rather vote for a reject.

review_title: Potentially interesting results, needs some revision

Meta-review: The consensus recommendation is that the paper is not ready for publication at this time.
Decision: Reject