Venue: ICLR 2022 Conference

Reviews:

final_score: rating: null
review_content: I am very curious about the author's response to review Tav6's novlety concern. I would adjust my score down to (3) reject if the authors did not respond in the given discussion time frame as I think novelty is an important consideration for ICLR.
review_title: Author's response regarding reviewer Tav6's novelty concern?

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the physics-informed neural operator (PINO). It combines the operating-learning and function-optimization
frameworks, which improves convergence rates and accuracy over traditional  methods. Experiments test the advantage of PINO.
main_review:  I have read the other reviews and responses from the authors. Two highly-related papers should be added. So the claims and experiments need major revisions. I lower my score to 5. 
===========================================

The paper is well-written and organized. The experiments are convincing. I have the following two main concerns, which prevents me from improving the score.
1. For a PDE solver, the data is specialized for the PDE. So the whole solving process should include the pre-training stage.
2. For vision tasks, using the pre-trained models to improve the accuracy and efficiency of the downstream tasks is a promising way. These vision tasks share the features from images. However, for a given PDE solving problem, when one can use the pre-trained model? Do we need to train a lot of pre-trained models?
summary_of_the_review: The paper is clear and the proposed method is reasonable. It has the weaknesses on the use of pre-trained models for PDE solving. However, I tend to accept the paper for its effectiveness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce physics-informed neural operator for learning partial differential equations (PDEs) from data with deep learning. This approach combines two recent and popular methods in the field of machine learning and PDEs, the physics-informed neural network (PINNs) for solving PDEs and Fourier neural operator (FNO) for learning solution operators. The paper takes advantages of both approaches to overcome convergence issues arising in PINNs by incorporating PDE constraints into FNO. Then, the authors apply their method to a number of challenging test cases and obtain better results than PINNs on PDE solving problems and FNO for learning solution operators.
main_review: The paper is well-written and highlights the current challenges in PDE learning before introducing the approach. The following paragraphs detail the strengths and weaknesses of the paper.

Strengths
---
- The paper is well-written and structured. The authors gives a good introduction of the field with its challenges and describe their contributions clearly. The next sections describe the PINN and FNO approaches before introducing the authors novelty method called PINO. The authors conclude by presenting a series of experiments.
- The authors introduce a method that combines the strengths of two popular approaches in the field of physics-informed machine learning: a stable and fast method for learning solution operators based on a large number of input-output training pairs (FNO) and a deep learning technique for solving PDEs using PDE constraints. The main contribution consists of applying the physics (PDE) constraints onto a trained neural operator. At the heart of this technique lies an efficient procedure for computing derivatives of Fourier neural operator described in section 3.3 of the paper.
- The authors show experiments on challenging (nonlinear, two-dimensional, chaotic, time-dependent) examples and achieve better results than standard deep learning methods on the three setups that they consider: learning solution operators, solving PDEs, solving inverse problems.

Weaknesses
---
- My main concern about the method described is that it requires the knowledge of the PDE expression and therefore the title is a bit misleading since the PDE is already known and hard-coded as a constraint in the optimization. Therefore, it also combines the two weaknesses of FNO and PINNS: the need for training input-output data and the knowledge of the PDE, this should be explicitly written in the conclusions of the paper as a limitation.
- I have some concerns about the choice of the training and testing data and optimization parameters. The training data used by the authors seem to be optimized for the problem considered, which questions the stability of the method. Is it able to tackle a new application without a lot of fine-tuning ? As an example, the Gaussian kernel used to sample the initial conditions vary across all the examples considered following a logic that I don't understand (see the experiments details described in the Appendix). This is a large limitation of the proposed method since it's heavily affects its flexibility for new applications. Similarly, the authors give little details about the training test used in the different applications. I am then wondering whether the good performances shown by the authors is due to the fact that the testing and training sets for the forcing terms (or initial conditions) are drawn from the same random distribution. In this case, the method could only be learning the solution operator on a specific vector space of functions and not in general (i.e. transfer learning would not work on a completely different distribution).
- I don't think that the authors report an example with noisy data for learning and solving PDEs, which is essential to check the robustness of the method.
- To my understanding, the Fourier neural operator method has severe limitations on the type of domains and boundary conditions, unlike PINNs (Raissi et al., Science, 2020), as it requires simple geometries and periodic boundary conditions to apply the Fourier transform layers. These limitations carries over in the present paper and the method can only be applied on simple domains with homogeneous/periodic boundary conditions. This questions the practical utility of the method for e.g. solving PDEs with respect to traditional numerical solvers such as FEM/spectral methods/finite differences. Can the authors comment on that, point out the limitations in the conclusions of the papers and discuss potential ways to overcome them ?
- It would be great if the authors could add a paragraph to give a hint on potential practical concrete applications of the present methods, where one would have access and take advantage of both training data (sampled from a carefully selected Gaussian distribution) and the expression of the PDE. Right now, I can hardly see any concrete applications beyond fluid dynamics problems.

Minor comments
---
Can the authors explain what they mean by "additional PDE instances" in Table 1 ? I couldn't find it in the Appendix and this should be described explicitly in the main text.

### Post-rebuttal update
I updated my score following the rebuttal.
summary_of_the_review: I find that the paper is well-written and introduces a method combining two known techniques for learning and solving PDEs from data. The authors present numerical results that outperform PDE benchmark used in the literature, taking advantage of the PDE constraint from the PINN approach or the neural operator learning from the FNO technique. Yet, there are some weaknesses in the study that I discuss in the main review. However, most weaknesses are inherent from the current challenges faced by the field of PDE discovery (such as choice of training data, practical applications, limitations on the domain geometry, boundary conditions).

Therefore, I believe that the paper could motivate further explorations and research between deep learning PDE solvers and PDE learning methods and recommend acceptance of the paper to ICLR provided that the authors address the points raised in the review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors presented a novel algorithm to solve PDEs by combining two mainstream PDE solving deep neural network genre, the PINN and the FNO method into PINO. Which is basically doing these two steps sequentially by first learning the neural operator and then do a optimization step using PINN method.
main_review: Given the claims of novelty is correct (that this paper is the first one proposing the combination of PINN and FNO) (I am not that familiar with the relevant latest literature there), I think this is a great paper with both technical contribution and strong emperical result.

Strength: The idea to combine these two mainstream pde solver is, although simple, very effective as shown from the emperical results. The only technical difficulty of combining the two algorithm (test time PINN-based optimization for FNO) that I see is the derivative calculation for neural operator space, where the authors used exact gradient calculation coded by chainrule. The loss landscape 

Weakness/Questions: Some of the questions and rooms for potential improvement:
1. Fig 2c the colors are very hard to recognize on the legend. 
2. Did the PINN benchmark in this work include the adaptive activation function? ("Adaptive activation functions accelerate convergence in deep and physics-informed neural networks" Jagtap & Karniadakis.) 
3. According to Table 2 for the Kolmogorov flow, the PINO have 0 solution error improvement with increase in both number of data samples and PDE instances. Although the test time optimization convergence is faster, didn't it take much longer time to pre-train in these two cases? 
4. The clarity for the section 4.3 inverse problem needs to be improved (maybe due to page limit?). For the forward model method, how many initializations of "a" were tried? What was the R(a) regularization term's underlying assumption there?
summary_of_the_review: I think the technical and algorithmic contribution is solid ,the emperical results are good, and (if the novelty is good) I think this is a publishable paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Physics-Informed Neural Operator (PINO) by combining two previous methods, Fourier neural operators (FNO) and physics-informed neural networks (PINNs). Similar to one use case of FNO, PINO learns the solution operator over multiple instances of a parametric PDE family in the first phase. In the second phase, given an instance from the parametric PDE family, PINO fine-tunes the learned solution operator as a pre-trained model to work well for the particular instance. The proposed method is evaluated by using numerical experiments.   
main_review: Update after author response: The author's responses do not address my concern on the novelty w.r.t. previous papers. Each of the 4 new aspects w.r.t. the previous method is too incremental given the standard knowledge in our communities. Each point is done in other papers without marking them as a contribution of the paper. Also, the comparison should be done with the comparable previous methods as well as better variants of PINNs, as I mentioned in my review. Accordingly, I cannot raise the score to be fair with other papers. I support this direction and I would personally like to recommend acceptance if there is no almost identical idea in the previous paper. However, there is an almost identical idea in the previous paper and I cannot recommend this for acceptance for that reason and additional reasons, including but not limited to the lack of fair comparisons with variants of PINNs. 

The proposed method makes sense as a transfer training method for PINNs, with many source tasks and a single target task, although it is limited for the transfer within each specific parametric PDE family. Here, note that this can be viewed as a type of transfer “training” (which is to accelerate training), instead of transfer “learning” (which is designed to accelerate training and generalization). 

In this regard, I like the paper and I think that the proposed method has a potential to be used as a transfer training method for PINNs.

However, I have several doubts. First of all, this seems to be too incremental given the following previous work: 

[1] Sifan Wang, Hanwen Wang, Paris Perdikaris. 2021. Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets. 

The authors should elaborate the difference more. If we replace DeepOnet by FNO in the previous method, does it result in something very similar to PINO?  

Moreover, if we know the PDE constraints, I don’t understand the point of learning the solution operator with the PDE constraints in the first phase. If we already know the PDE constraints, we can solve for the solution, without learning the solution operator, for each instance of the PDE family. Thus, we are introducing additional, potentially unnecessary step, in most cases. The authors should rewrite the paper much more carefully to organize the problem setting with what is given and what is not. 

From the current presentation, I infer that mathematically, this problem setting is nothing different from that of PINNs for the forward problems, if we don’t have the additional data for the operator learning phase. In this case, from the abstract view point, the proposed method is equivalent to (1) change the architecture from PINNs to FNO for the solution operator, and (2) add additional the pre-training step to train the solution operator for the PDE family, instead of one instance from the PDE family. Thus, the experiments are not convincing, because the performance improvements can be coming from the change of the architecture and the pre-training, which is expected and trivial. The authors mention that PINO is more expressive and thereby achieves better accuracy. However, then, we can also increase the expressive power of PINNs by changing architectures of PINNs. Thus, the authors must compare the proposed method, PINO, against various architectures of PINNs with various expressive powers. 

Similarly, to verify the claim of the paper, the paper must include variants of PINNs that have solved many drawbacks of PINNs, such as extended PINNs. At this point, I don’t see any fundamental reason why PINO against variants of PINNs with various architecture choices. 

Although the authors mention the limitation of optimization of PINNs, there are various methods proposed to mitigate such limitations, including self-adaptive PINNs (Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism), adaptive activation functions (Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks), and sf-PINN (Learning in Sinusoidal Spaces with Physics-Informed Neural Networks). Considering the purpose of the proposed method, the authors must compare PINO against those methods in the paper.

Moreover, for the pre-training, the papers should measure the time complexity of pre-training too to compare it fairly.  My worry is that under this problem setting, we can more efficiently solve each problem instance if we count the extra time of pre-training as well, if I use appropriate architectures and hyper-parameters. Thus, a comprehensive investigation on this is required before publication.

Another question is whether the model pre-trained with one PDE family can generalize well to other different PDE families. This is not conclusive from this paper as well. If we cannot generalize in this way, the applicability of PINO is crucially limited, as we can always solve each instance within a PDE family with variants of PINNs or other PDE solvers in the problem setting of this paper.  




summary_of_the_review: This seems to be too incremental given the previous work and the experimental setups are not sufficient to support the claims of the paper. There is no theory provided.
review_title: null

Meta-review: The paper proposes the physics-informed neural operator. It combines the operating-learning and function-optimization frameworks, which improves convergence rates and accuracy over traditional methods. While the paper was well written, several reviewers raised their concerns on the novelty of the paper, especially regarding the difference from PINN-DeepONet (Wang et. al.). Following this, there have been a long discussion between the authors and the reviewers, as well as among the reviewers. As a consequence, we think the authors somehow overclaimed their contributions on combining PINN and operator learning, and there are some important references missing and baselines not compared empirically. With this, the conclusion is that we cannot accept this paper in its current form, and we hope that authors can take all the review feedback into consideration and better position the novelty and impact of their work in the future submissions.
Decision: Reject