Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: - Summary

This paper presents a "physics-informed" deep learning model of fluid dynamics. The underlying deep learning architecture employed is a somewhat standard u-net, but one of the proposed method's distinguishing features is that it enforces its adherence to physical behavior at its loss terms, by penalizing predictions that are not incompressible or do not conserve momentum. Notably, this approach allows it to be trained unsupervisedly, without requiring the generation of ground-truth simulations.

- Pros

The enforcement of the physical behavior as a core feature of the architecture (e.g., through the "a" vector and the conservation losses) makes the network to generate stable simulations.

This also brings with it the interesting benefit of being able to learn without the need for ground truth data, lowering the cost of training the model.

The proposed model performs favorably, both in terms of speed and error, to phiflow, the current "reference" differentiable fluid simulator. Since the proposed method is still deep learning based, it is also differentiable.



- Cons

Most of the experiments serve only to validate the model's accuracy and its ability to learn a proper fluid simulation. 
The only practical application demonstrated is in a simple control task, which is not explored too deeply. 
Phiflow (Holl et al., 2020), for example, performs more extensive and diverse evaluation of control settings.


- Reasons for score

Overall, given the "pros" described above, notably the distinct loss formulation that allows the model to learn unsupervisedly to perform efficient, differentiable fluid simulations, I recommend this paper for acceptance. 
 

- Additional comments

Is training improved if actual simulations are used for data, instead of "cold starts"? 
If so how do these compare with training with cold starts (as presented in the paper), both in terms of training speed and in terms of final (test) results?
Does starting from more realistic starting points fix the issue of the initial error "spike" in Fig 4?


review_title: Review

final_score: rating: 7: Good paper, accept
review_content: The paper is generally a good contribution especially in the field of machine learning for physics. However, I do have some questions about the novelty of this work and what makes it different from other physics informed approaches. Below are the pros and cons and the I list out a set of comments and additional results that I believe would make the contribution of the paper much more clear. 

Pros:
-- A physics informed neural architecture that optimizes the governing PDEs to perform forward integration of the system. Generally a great approach since it does not require high-res numerical simulations
-- Generalizes to new geometries

Cons
-- I might be missing something, but I generally feel that there is a lack of novelty. Approach is very similar to certain papers I point out in the comments. That however, does not take anything away from the approach proposed and it is still a good contribution once certain concerns (in the comments) are resolved
-- The authors claim that their network generalizes but I do not see an explanation for it. Even an intuitive explanation would be good.

1) The introduction section of the paper does talk about different approaches to physics informed neural networks especially the Raissi et al., papers and mention that Raissi's approach (https://arxiv.org/pdf/1711.10566.pdf and https://arxiv.org/abs/1711.10561) does not generalize to new domains. Can the authors explain why they believe that their network would automatically generalize to new geometries? Does the introduction of $\Omega$ and $\partial \Omega$ in their input features make their framework generalizable?

2) The use of vector potentials to ensure divergence free by construction inside neural architectures using fixed convolution operations has been shown before By Mohan et al., (https://arxiv.org/pdf/2002.00021.pdf). The authors should cite this paper. 

3) In Figure 2, I'm concerned about how well the network satisfies BCs. If possible, I would like to see a plot showing velocity with time during inference near the edges of the obstacle.

4) There is a complete absence of baselines. Sure, the a-net outperforms v-net when it comes to the loss. However, the loss term is $10^{-3}$ for v-net and $10^{-5}$ for a-net. Both are pretty low. How much improvement in inference would one expect?  It is evident that constraining divergence by construction definitely helps in reducing the loss and this has been shown in https://arxiv.org/pdf/2002.00021.pdf. 

5) I would like to see a comparison between the a-net prediction and a numerical simulation and possibly a comparison with general PINNs (https://arxiv.org/pdf/1711.10566.pdf)  for the rectangular obstacle. It should clearly show that their approach is more accurate in terms of RMSE error between a-net and numerical simulation as compared to PINN and numerical simulation. What would be more interesting to see is how well the authors satisfy BCs as compared to PINN. Moreover according to the authors PINNs would not generalize to the airfoil shaped obstacle while their network would . This would clearly show an advantage of the proposed mechanism. Without a baseline however, there is not much to prove that this is a better approach.  

6)  The problem solved in this paper is a very simple problem. Even numerical solvers are not expensive when it comes to solving this problem. This physics informed approach is very promising because it can reduce the computational cost immensely (during inference). Would it be possible for the authors to show this for a *real* turbulent flow, e.g. the system considered in the Mohan et al., paper (https://arxiv.org/pdf/2002.00021.pdf) or the Kolmogorov system as shown in this paper (https://advances.sciencemag.org/content/advances/3/9/e1701533.full.pdf) or even the 2D Kraichnan system shown in https://arxiv.org/abs/1808.02983. Unless these approaches generalize to fully turbulent flow, the use for such architectures are limited for real applications

This is still an important contribution in the field of deep learning for computational physics, specifically in fluid dynamics. 
review_title: Good paper. But an absence of baselines for comparison. 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to learn the dynamics of an incompressible fluid via a physics informed loss formulation using an unsupervised training framework. It employs a custom solver that is executed at training time to learn a Navier-Stokes residual with a incompressible (curl of a stream function) formulation. This setup is demonstrated for two dimensional karman vortex streets, and a control example for the magnus effect. 

The paper definitely targets an interesting and relevant direction for machine learning, but it is a (fairly identical) resubmission of a paper that was rejected at NeurIPS. It is of course possible to resubmit papers, but unfortunately in this case it was a fairly clear rejection, and while I cannot compare versions side by side, I think the submitted content is largely the same. As both NeurIPS and ICLR are currently mostly on par in terms of aims, content, and expectations for accepted submissions, I cannot recommend accepting this paper in its current form to ICLR.

For the previous submission, I believe the following points were the most important issues, as raised in the reviews: Most importantly, the benefits of the proposed method over existing ones (ie supervised from regular solver) are not made clear enough. Then, there is the insufficient quantitative evaluation, that the stream function formulation does not properly extend to 3D, and that the direction was not fully clear (performance vs control applications).

I'm aware that it must be frustrating for authors to be rejected for very similar reasons, but it is likewise not nice for reviewers to repeatedly look at papers searching for differences, and discovering that authors have not taken comments from previous submissions into account. I would be curious to hear in the rebuttal how or whether the authors have updated their submission after the NeurIPS reviews. As already mentioned in the previous cycle: there is merit to the direction of the paper, but I think it is important to make a clear step forward for one or more of the issues raised with the submission (as outlined in the previous paragraph). With such extensions, there should be the potential for an improved evaluation. However, as it stands I don't think this submission is suitable for ICLR.

A minor note, but the title seems somewhat inapproprate to me - physics-informed typically implies that derivatives are computed via autodiff from a network representing the solution. Here, the authors instead discretize the solution on a grid, while a network yields the solution for the next step, and the physical model is evaluated with FDs on the grid. "Physics-constrained" (as in the section title) would be a better choice for the title.

review_title: Interesting direction with open issues from previous submission

final_score: rating: 7: Good paper, accept
review_content: ### Summary of my understanding

The authors propose an unsupervised (or rather, I would say self-supervised) method to build a simulator of incompressible fluid flows based on neural networks. Their neural network is given the pressure and velocity (or its potential) fields at time $t$ and is trained so that it outputs the fields at time $t+dt$ with the "physics-constrained loss functions." The proposed loss functions are designed to make the outputs of the neural net fulfill the Navier-Stokes equation and boundary conditions. They train the neural net on randomly-generated domains and boundary conditions. They show that the learned model outputs qualitatively plausible flows, even if the domain is not exactly handled in the training phase.

### Evaluation

The paper is easy to follow. The proposed method is technically reasonable. The related work section is comprehensive. While the technical novelty of each component of the proposed method looks moderate, the overall framework would be valuable as a fast differentiable fluid dynamics solver, which the authors claim. The experiment section is convincing to some extent, but it lacks comparison to classical solvers (i.e., somewhat inherently-faithful simulations), which limits our capability to assess the soundness of the results in Section 4.1 and Appendix D. Also, the paper lacks information on the range of hyperparameter search.

### Questions

[Q1]
How did you create the randomized boundary conditions $(v_d)_k^0$? I could not find it in the paper.

[Q2]
The training strategy described in Section 3.7, especially the pool's update and random renew, seems essential to the performance of the proposed method. Did you performed some ablation study in this regard, or do you have some notes on what happened if the presented strategy was not adopted partially? Such a description would help a reader's understanding much.

[Q3]
Can you provide the range of hyperparameter search and information on how intensively the search was done? Such information is important for assessing training computational cost.

review_title: A nice paper that might be improved with more information & simple baselines

Meta-review: The paper introduces a learning framework for solving incompressible Navier-Stokes fluid using a physics informed loss formulation. The PDE is solved on a grid, and the model, implemented via convolutions and a U-Net, is trained to minimize the NS residual. The model is trained on a variety of randomized contexts, in a way that allows training to explore a large number of configurations. The paper presents original contributions compared to previous Physics informed framework (discrete formulation, conditioning on the domain conditions, …). All the reviewers agree that the detailed rebuttal provides answers to their questions and that the contribution is significant, they all have a positive assessment of the paper.
Decision: Accept (Spotlight)