Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors adopt ideas from multigrid methods in solving PDEs to Physics Informed Neural Networks (PINNs). After giving a standard introduction to PINNs and Multigrid methods, they describe their approach.

They decompose the PDE solution as a sum of a coarse and a fine term, and train independent PINNs to learn each term. The coarse PINN is chosen to have smaller capacity than the fine PINN, and is also (potentially) trained on fewer datapoints on the domain. Training proceeds by alternating training between the two PINNs: $\nu_1$ epochs on the fine PINN, followed by $\nu_2$ epochs on the coarse PINN, and repeat.

The authors test their method on a linear non-homogeneous 1-D elliptic equation (trained with Adam or BFGS), a 2-D nonlinear equation, and 1-D Burger's equation. They compare a standard PINN vs a larger PINN vs their method (MPINN).
main_review: Multigrid training for PINNs could be potentially interesting. In particular, I agree with the motivation of the authors that the particular structure of MPINNs could make training easier by imposing a kind of inductive bias. That being said, the results in this paper are too preliminary to be accepted.

1-D Linear Elliptic equation:
- It seems like the two PINNs you use only differ by capacity of the network. I thought the main motivation for Multigrid was that the PINN with less capacity would be trained on fewer points?
- The equation is chosen so that you get two different modes in the solution. It would be interesting to vary $\alpha$ to see whether MPINN training helps as the difference in frequency between the modes increases. I feel like the main motivation for this toy problem could be the different modes. It would be interesting to see it for more difficult parameter settings rather than only $\alpha = 3$ or $5$.
- Why the distinction between Adam for the 1-layer networks and BFGS for the 4-layer networks? Since BFGS uses n^2 memory, it seems Adam would be more appropriate for larger networks.
- How were the learning rates for the different problems tuned? I am not entirely convinced that the difference in performance is not from using different learning rates / hyperparameters for the different problems.
- You mention that MPINNs are more robust, but I am not sure I agree with the definition of robustness. It seems you are defining a budget of training epochs, and if the method does not converge in time, it is considered not robust. This would just mean the method is "slower", but not necessarily "less robust".
- It seems you are reporting mean + std of the RMSE. Can you report median + IQR instead?

2-D Nonlinear equation:
- Could you give some intuition on what the equation represents and what the solution looks like?
- 400 points vs 484 points does not seem like much of a difference between "fine" and "coarse". How did you choose those numbers?
- Results do seem less convincing in this case.

Burger's equation:
- What optimizer did you use?
- It seems you are picking different networks for each PDE. Here you have 3 hidden layers. How did you choose these sizes?
summary_of_the_review: The framework is potentially interesting, but the experiments in the paper do not convince me. It would be helpful to see more difficult problems, as well as multilevel problems with n > 2. Furthermore, there are a lot of details on hyperparameter tuning that is missing, leaving the question open whether the stated performance improvements are due to the hyperparameters or the MPINN idea.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new physics informed neural network (PINN) that is motivated by multigrid methods. The idea of the proposed multilevel PINN is to write the solution for a given problem as sum of two terms, where the first term models fine-scale structures and the second term models coarse-scale structures. This approach yields models that converge faster and reduce the approximation error as compared to standard PINNs. The performance is demonstrated on both 1D and 2D problems.
main_review: Using multigrid methods as inspiration for training and designing PINNs is neat. Splitting the solution into two terms is interesting, because this scheme presents the model with an easy and a hard task (learning fine-scale structure is typically more challenging than learning coarse-scale structures). The proposed scheme can also be viewed as some form of data augmentation and it would be interesting to discuss this aspect in more detail. Further, it would be interesting to discuss whether there is some connection between the proposed multilevel learning scheme and Curriculum Learning.

Overall, the presentation of the materials is very clear and the paper reads nicely. However, a major shortcoming of the paper are experiments. 

I would like to see the following questions addressed:

* How do you construct $z_h$ and $z_H$? How do you determine whether $z_H$ is not expressive enough? Is this an automatic process or does this require manual tuning? How is the performance affected if $z_H$ is varied from less coarse to more coarse?

* How does the proposed multilevel PINN compare to other state-of-the art PINNs or to classical scientific computing methods for the problems under consideration? For papers that have mainly an empirical flavor it is typically good practice to show some benchmark studies. Burger's Equation is often considered as a standard problem, hence it should not be too difficult to provide some baselines for this problem. 

* In Table 2 and 3, the difference between the proposed model and the ablation models appear to be marginal in terms of the RMSE. For example, is there a practical difference between 1.0e-3 and 1.2e-3 for the problems under consideration?

* Why does PINN h and h+H start to fail if the number of parameters are increased? Do you regularize these models?

* How did you chose the tuning parameters for the different models? I played around with the provided notebook "MPINN_2D" for a few minutes and I was able to improve the performance of the standard PINN quite a bit by slightly increasing the learning rate (2e-3), introducing a small amount of weight decay (5e-3) and switching off the learning rate scheduler. The PINN_15 achieves now an RMSE of 0.092 as compared to 0.14 by the proposed multilevel PINN. I am pretty sure that the performance can be further improved with some more careful fine-tuning of the hyperparamters. Tuning models are crucial and this needs to be addressed.

* The term 'robustness' typically means something different in the ML community. I am not exactly sure what you mean by `increased robustness of MPINNs'.
summary_of_the_review: This paper introduces a new multilevel PINN for learning solutions of PDEs. The presented ideas are novel and of interest to the scientific machine learning community. One shortcoming of this paper is that the experiments appear to be too preliminary, i.e., no benchmarks are provided and the models are not carefully fine-tuned. This makes it difficult to judge the advantage of the proposed method as compared to other state-of-the-art PINNs. At this point I feel that the paper is slightly below the acceptance threshold, but I am willing to change my score if the authors can address my questions above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes multilevel physics informed neural networks (MPINNs). When compared to standard PINNs, MPINN uses additional networks for different levels with fine and coarse terms in the view of two-levels. By reclusive definition, this can be generalized to any multi-levels. The method is motivated via classical multigrid method, and is empirically studied with 3 simple ODEs/PDEs.    
main_review: The method of MPINNs makes sense and seems to be novel. Although it is motivated via the classical multigrid method with the operators P and R, the final method is a very simple alternate optimization with two sub-networks (for two levels) where one sub-networks minimizes the loss for ‘fine’ points, and another minimizes the loss for ‘coarse’ points. Because the solution of PDEs can contain fast/fine components as well as coarse/slow components, the proposed method makes sense and I would expect that this works well for such PDEs. In this regard, I really like the paper and recommend acceptance subject to very minor revisions for several concerns that I explain below. 

While the idea and method are great, the demonstration of the method can be improved a lot. The present experimental results are somewhat inconclusive because of the experimental settings and the definitions of the plotted values.  For the experimental setting, the authors mention that the learning rates are experimentally tuned, and the best-decreasing strategy is adopted for each network. This can be problematic without presenting more details, as the learning rates can be the reason for the reported superior performance of MPINNs. I would recommend reporting the results with various learning rates or fixing the learning rates a priori by directly using the learning rates of previous work of PINNs. In the current form, I cannot tell if we are overfitting the data and problems with the tuning of learning rates of MPINNs. At least, the authors should report the learning rates for each case.

The idea is very related to extended physics informed neural networks (XPINNs), which was theoretically studied by Hu et al [1]:

[1] Hu et al. When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?

The previous work [1] has theoretically and empirically shown that XPINNs can improve standard PINNs by using sub-networks, each for different subdomains with different complexities of the sub-solutions. This is very related to MPINNs. Instead of decomposing the domain space, MPINNs are decomposing the solution components for fine and coarse parts. Therefore, the authors should cite the previous work [1] to mention this relationship. Moreover, unlike the previous work on XPINNs [1], the present paper does not provide any theoretical results, which should be also mentioned as a limitation of MPINNs, when compared to XPINNs, at this moment. In sum, the relationship and the relative limitation should be made clear.

For the definitions of the plotted values, the authors should use a metric that can be comparable for all methods. Instead of these loss and objective used for training, the authors can plot the value of errors for the solution of PDEs, by using a set of new unseen data points. 

What are the precise definitions of RMSE and loss used in the tables and figures?

I am not entirely convinced by the results where the performance degrades as we increase h and H. Do you observe the same phenomena with BFGS instead of ADAM? It is possible that this can be caused by bad training practices by the authors. I would suggest plotting the training loss over epoch for different h and H as well to see what is going on there.

The paper makes several random claims without any evidence or support. I recommend deleting those claims from the paper. For example, the authors say that it shows improved performance compared to the state-of-the-art, but no state-of-the-art performance is cited or compared against in this paper. After Table 1, it reads “We observe that MPINNs clearly outperform classical PINNs.”, but it is not clear at all because of the uncertain learning rates and the stopping criteria. The following claim is also not supported by any evidence in this paper and it is an opinion of the authors instead of a finding in a scientific paper: “It is clear that structuring the parameters in a clever way is more beneficial than just augmenting their number, to gain both computational time and expressivity.”

In table 2, F is not used and the following sentence should be deleted in its caption: ”F” means a failure in all the runs. 

In table 4, is “0.04.6e-2” typo? On page 5, “the and of” should be replaced by “the end of”. 


summary_of_the_review: While there are several concerns and weak evidence to support the main claims, the idea is very novel and significant. I infer from the method and its idea that the additional evidence to support the main claims will come by sooner or later, either by this paper or other authors. There are so many incremental papers without any originality that have been published just because of large-scale experiments for demonstrations. In contrast, this paper has weak demonstrations but a good original idea. Whereas both types of papers have their own merits, I think that the latter type contributes to our community, science, and academia in a longer-term. Accordingly, I recommend this paper for acceptance, subject to minor revisions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript entitled “MULTILEVEL PHYSICS INFORMED NEURAL NETWORKS
(MPINNS)” introduces multilevel physics informed neural networks (MPINNs), which is inspired by classical multigrid methods for the solution of linear systems arising from the discretization of PDEs. The authors showed MPINNs provide a faster and smaller approximation error in the case both of elliptic and nonlinear equations and a more robust asymptotic behavior.

main_review: This paper is not well written in English. But since this MPINNs show interesting point that it can run faster and provide smaller approximation error for some PDE equations than using original PINNs, I recommend this paper to be published after polishing the language and also addressing all my below comments.

My comments are as follows:

1.	In the third line on page 5, the formula for R and P has extra ‘)’ at the end. On the same page after the Equation (6c) line, it should be ‘at the end of each cycle’ instead of ‘at the and of each cycle’. In this paper, there are too many ‘really’ and ‘then’.
2.	In this paper, all the experiments are done with h=2*H. Why do you choose 2 here? How does this number matter for the MPINNs training? Some explanations about how to choose this number should be added.
3.	On page 6, the authors state “Indeed, when the number of parameters increase the peculiar structure of the MPINN allows to make the training process easier and as a result the network reaches a better approximation.” The authors should explain why MPINN works better instead of just stating the observation.
4.	On page 7, on the last two lines the authors say “In 2D problems the effect of the size of the training set on the cost is far more important than in the 1D case, therefore we train the fine network on a grid of 484 points and the coarse one on a coarser grid of 400 points”. What is the rule of thumb to choose the grid size? 
5.	On page 8, the Figure 3 uses h=10 case while in Table 3 it starts with h=30. Since Figure 3 and Table 3 are talking about the same experiment “2D NONLINEAR EQUATION”, why do you choose different h for table and plot?

summary_of_the_review: This paper proposed the multilevel physics informed neural networks (MPINNs), which is inspired by classical multigrid methods for the solution of linear systems arising from the discretization of PDEs. It showed some interesting results, but those results are lacking of explanation in depth. The authors are mostly just stating the observations, which is not enough to meet the standards of a good paper. 

In summary, I would not recommend its publication in ICLR until all my comments are addressed.
review_title: null

Meta-review: The paper develops an instance of physics informed neural network inspired from multigrid methods for solving PDEs. The proposed framework describes the solution of a PDE problem as the sum of terms operating at different resolutions. Training is performed by an iterative optimization algorithm that alternates between the different resolution models. Experiments are performed on 1D and 2D problems.

All the reviewers agree on the originality and the potential of the proposed method. They however all consider that the current version of the work is too preliminary both in the form and in the content. The experimental contribution should be developed further with tests performed on more complex problems and complementary analyses. Some of the claims should be given more evidence or moderated. It also appeared during the discussion that the models are not well tuned, making the results inconclusive. The authors are encouraged to develop and strengthen their work.
Decision: Reject