Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to use MAML-based meta learning to learn a neural operator that approximates complex PDE-based equations with different physical parameters. Instead of optimizing all parameters of the base model during inner optimization (like vanilla MAML) or only the last layer (like ANIL), this paper's main contribution was to show that the first layer of the neural operator should be optimized. Experiments were conducted both on synthetic data, benchmark data, and a real-world data that describes mechanical response of biological tissue specimens from DIC displacement tracking. Experiments were mainly concerned with baselines using the neural operator without the presented meta-learning formulati
strength_and_weaknesses: Strengths:

To be able to learn a neural operator for complex PDE-governed physics without varying parameters is important. The use of meta-learning is intuitive. 

The experiments attempted various types of dataset beyond benchmark data, which is appreciated.



Weakness:

- A major concern is regarding the baseline. The primary baselines includes a "single" model trained on the meta-training data with multiple tasks, and three variants of meta-formulation. The comparison with the "single" model however is not fair. All the meta-models are further optimized to the context data (referred to as test data in the paper) at meta-test time; this was not accessible to the "single model". Two stronger baselines need to be added. One is the "single" model fine-tuned to the same meta-test support data (used to optimize the model for all the other meta-models during meta-testing). The other is the "single" model trained to the "single" task used in the meta-training set (and fine-tuned to the meta-test task depending on how different the tasks are). 


- Another major concern is regarding the comparison to alternative models used to approximate PDE-based physics -- there is currently a large set of literature that does this; some of them are designed to generalize across different parameter settings (examples below). These works should be included for comparisons.

Yin et al. LEADS: Learning Dynamical Systems that Generalize Across Environments
Wang et al. Meta-Learning Dynamics Forecasting Using Task Inference


- The rationale for the choice of meta-learning formulations is not clear. There are many different frameworks for meta-learning, some of them do not require test-time optimization. Why this formulation versus alternatives?


- The difference between ANIL and MetaLast is not clear -- both does inner optimization to the projection layer? Please clarify.

- Intuitively, since MAML optimizes all parameters of the base model, it is not clear why it would be sub-optimal then optimizing only the first layer (except for computational overhead)? Comments on this would be appreciated.

- The wording of this paper is in general somewhat confusing, since it did not utilize the standard terminology of context/target samples for meta-learning. Assuming that N_test is referring to the context set size (i.e., size of meta-test samples used for inner loop adaptation at meta-test time), what is the corresponding context set size during meta-training? Is it the same as what is used at test time? Fig 2 showed how the test error changes as N_test changes. Is this N_test only referring to the context set size used during meta_test, or is the same parameter changing during meta_testing. In all experiments, N_test is given but not the size used during meta_training. This needs to be clarified. If N_test is different from the context set size used in meta-training, the effect of the latter would be more interesting to see in ablation studies.



- In synthetic experiments, it'd be good to more specifically describe exactly which parameters are used for meta-training, and which for meta-testing. i.e., it is good to have a sense of the distance between training and test task for in-distribution tasks.

- In real-world data experiments, it was mentioned that "we have 500 available data pairs" for each specimen, but it was then stated that MetaP was trained based on but N test = 500 samples and then evaluated on another 200 samples? If only 500 pairs are available, how can we use all of them as the context set? Please clarify

clarity,_quality,_novelty_and_reproducibility: As mentioned above, the paper describes a meta-learning methodology without using the standard terminology for meta-learning. For instance, what is exactly N_test (is it context set size) and what is it counterpart during training is not clear. Because of this, the clarity of the paper is affected.

Novelty of the paper as mentioned above is limited. It involves apply MAML to neural operators, and the main contribution seems to be at identifying that in neural operator the first layers should be adapted. It lacks discussion and comparison with related works that deal with approximating PDE-solvers, especially those with a meta-learning or generalization focus.
summary_of_the_review: This paper applies a MAML variant to a neural operator for learning to approximate PDE-based physics for various parameters. The motivation is intuitive, and the point on having to adapt the first layer of the neural operator is interesting. Overall, however, this paper can be improved both in its novelty in relation to existing works, and stronger baseline/comparison methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work developed a novel meta-learnt method to transfer learning knowledge between neural operators. Different from typical
final-layer transfer in existing meta-learning methods, this paper adapted the first layer of the neural operator model to capture hidden
parameter field. Evaluation on both synthetic and real world datasets demonstrate the efficacy of the proposed approach. 
strength_and_weaknesses: * Strength
1. The idea of transferring knowledge between neural operators is interesting and novel
2. It adapted the first layer of the neural operator model to capture hidden parameter field in contrast to the final-layer transfer in existing meta-learning methods
3. Conduct extensive experiments on benchmark and real-world datasets to verify the effectiveness of the proposed approach 

* Limitations
1. It only adopted Fourier Neural Operator to show the effectiveness of the method, and it would be better to study more neural operators to verify this idea.
2. Please repeat the experiments multiple times and then report the average error and standard deviation.
3. A couple of grammatical issues. For example, "As an motivating example," on page 1; and "ANIL In " on page 5, I am curious if a reference is missing?
4. Questions:
[1] In fig 2, why the accuracy of OOD task is better than ID task? Please provide possible explanation on it in more detail.
[2] I am curious where could we access the real-world dataset: biological tissue specimens from DIC displacement?
[3] I am wondering if the proposed method could learn multi-tasks from different PDEs rather than one common PDE with different physical parameter?
clarity,_quality,_novelty_and_reproducibility: The writing of this paper is clear and decent. The idea is novel and interesting. It would be better if the author could release the source code for readers to reproduce the experimental results.
summary_of_the_review: It is interesting and novel to transfer the knowledge between neural operators. It would be much better to explore more neural operators to further verify the effectiveness of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors propose a (somewhat) novel method that combines meta-learning with a popular operator learning model (based on the Fourier-Neural Operator learning paradigm) and develop a few-shot learning framework to learn PDE based domains. Additionally, authors also introduce a novel modification to the few-shot inner-loop model update wherein the updates are only localized to the “lifting” layers in the FNO based operator framework. 
strength_and_weaknesses: ## Strengths: 

**[Novel Few-Shot Operator Transfer Learning]**: Authors address an important problem of few-shot learning in the context of knowledge transfer across PDE domains. Specifically, the proposed MetaP, MetaP+ methods learn to effectively transfer knowledge with only a few instances of data from the target task of interest. 
 

**[Demonstration on Synthetic and Real-world Data]**: The proposed framework not only demonstrates good results on a set of synthetic PDE based domains but also on real-world task of learning to estimate mechanical response of biological tissue specimen from displacement tracking measurements. 
 

**[Effective Few-shot Performance]**: The MetaP, MetaP+ models demonstrate very effective few-shot generalization performance with as little as 2 instances required to achieve <5% prediction errors on synthetic data with similar results also on real-world datasets.  
 

**[Baseline Comparison]**: Authors compare with popular meta learning models MAML [2], ANIL [3] and a version of the base model [4] without meta-learning as ablation baselines. The proposed models outperform all baselines under few-shot learning settings.   

 

## Weaknesses: 

**[Some Baselines Missing]**: A related few-shot operator learning baseline model [5] based on one-shot-learning with deep operator networks is missing. 
 

**[Out-of-Distribution Results Require Further Clarity]**: Some of the out-of-distribution based experiments seem to produce results that are better than in-distribution test results. Without qualitative visualizations of training and testing regimes (out-of-distribution case), it is hard to comprehend the degree of challenge posed by out of distribution data regime. 
 

**[Limited Novelty]**: Although the problem is well motivated and results of the proposed pipeline are somewhat convincing, the proposed framework itself is not significantly novel. Specifically, the sole contribution of the authors may be considered to be the insight of transfer learning by fine-tuning the “lifting layer” in the neural operator (backed by theoretical and quantitative results) instead of the last layer as in traditional transfer-learning paradigms. Other parts of the model (I.e., based model , meta-learning framework) are directly employed from previously published work. 
 
**[Typos]**: Paper has a few typos (e.g., Eq. 7 is referenced in multiple places in reference to the solution operator `G` , but Eq. 7 contains no such symbol. 

## Questions for Authors: 

How has model architectures (number of layers, activations) selected? What is the validation set size and how was it selected?  
 

Please provide an explanation in terms of a performance breakdown for why the out-of-distribution test error of the proposed MetaP model is lower than in-distribution test error? This seems surprising considering the challenge of the task?  
 

Could you please provide justification about the difference in the domain characteristics in the training and testing regimes for out-of-distribution tasks? For example, please comment on what about the test regimes makes (e.g., more stiff) make them a true test of out-of-distribution performance?  
 

How were out-of-distribution ranges selected for synthetic datasets in section 4.1? 

 

## References: 

1. Li Z, Kovachki N, Azizzadenesheli K, Liu B, Bhattacharya K, Stuart A, Anandkumar A. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895. 2020 Oct 18. 
 

2. Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning 2017 Jul 17 (pp. 1126-1135). PMLR. 
 

3. Raghu A, Raghu M, Bengio S, Vinyals O. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157. 2019 Sep 19. 
 

4. Li Z, Kovachki N, Azizzadenesheli K, Liu B, Bhattacharya K, Stuart A, Anandkumar A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485. 2020 Mar 7. 
 

5. Lu L, He H, Kasimbeg P, Ranade R, Pathak J. One-shot learning for solution operators of partial differential equations. arXiv preprint arXiv:2104.05512. 2021 Apr 6. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the proposed model, experiments and results indicate a good quality paper. Unable to verify  / reproduce results due to non access to the source code. 
summary_of_the_review: The paper proposes a novel few-shot learning based neural network based operator learning framework. Although the results are convincing, the proposed model is significantly based on previously proposed research efforts and the main contribution of the paper is their insight about fine-tuning the "lifting layer" instead of the final layer as in traditional transfer learning paradigm. Overall, the novelty of the proposed method is somewhat lacking but the incrementally novel proposed model does produce effective few-shot transfer learning results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a MAML-like approach for neural operators aimed at transferring learning across slightly varying physical systems. It builds on recent work on implicit Fourier neural operators and proposes a meta-learning method that uses MAML for the (first) lifting layer, while the remaining layers (including the Fourier layers) are meta-learned but not adapted at meta-test time. The proposed method is experimentally evaluated on simulated PDE data as well as a real-world physical tissue response setting.

strength_and_weaknesses: Strengths:
* Relevant problem with clear application
* Real-world experiments
* The first meta-learning approach based on neural operators

Weaknesses:
* The presentation of the work could be improved a lot
* No standard errors are reported in the experiments. Were experiments even repeated with multiple seeds
* Not enough experiment details to reproduce experiments
* Assumptions in Section 3.3. are not motivated

Minor typos:
* 3rd paragraph on page 3: ‘neural operators requires’ -> ‘neural operators require’
* Page 4, ‘we denote the set trainable parameters’ ->  ‘we denote the set of trainable parameters’

clarity,_quality,_novelty_and_reproducibility: The clarity of the paper could be improved a lot. In particular:
* Section 2 is very verbose and could be made much more concise
* The formal problem setting which is introduced in the intro with formulas (i.e. Eq. 1) is not very clear since relevant concepts have not been introduced yet at that point. For instance, it is not clear at all how $\mathbf{b}$ interacts with the rest of the PDE model. While the intro already talks about the solution of Eq. 7, the formula is only introduced 4 pages later. I think it would be more pedagogical to first talk about the ‘normal’ neural operators and then introduce the multi-task structure later.
* The notation is heavy and not very well explained. Since mathematical symbols often re-appears 2 pages later again without any reminders of their meaning, I found myself often scrolling back to find out how each symbol was defined. For instance, it would help the reading flow a lot if you would write ‘a function from the space of property fields B to the loading fields A’  instead of ‘a function from B to A’
* A lot of the formalization and explanation in the paper is very specific to mechanical engineering & physics. This makes the paper not very accessible to the broader machine-learning audience at ICLR.
* Assumptions 1 & 2 are not motivated much. If you make assumptions to prove a theorem you should explain why they are necessary and when they hold in practice.

The paper is novel in the sense that it is the first work I know of that does meta-learning with neural operators. The proposed approach itself is a straightforward combination of gradient-based meta-learning and Fourier neural operators. I do not find the universal approximation result very useful or enlightening. From how it is presented, it seems like the authors just assumed whatever they needed so that theory of Lu et al. (2019, 2021) goes through. 

While the experiments include relevant baselines and appropriate evaluation problems, there are not enough details to reproduce the experiments. More critically, it seems like the experiments have not been repeated with multiple seeds, and no standard errors are reported. This casts serious doubts about the experimental methodology and the reliability of the presented experiment results.


summary_of_the_review: Overall, I think that the studied problem is relevant and the proposed method has its merits. Unfortunately, it is not very well presented which makes the paper hard to read. In addition, there are serious shortcomings in the experimental methodology and documentation. At the moment, I do not think the paper is ready for publication. Since I consider the proposed method relevant, I encourage the authors to seriously re-work and improve the writing of the paper based on my comments above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a meta-learning approach to transfer knowledge from neural operators for PDE approximations in order to transfer knowledge to unseen physical systems. A part of the contribution is to show that transferring from the first layer of the neural operator model is better. The method is evaluated on synthetic datasets, MNIST, and real datasets not governed by PDE.

Strengths:
-idea interesting for an important problem
-good intuitive explanations
-experimental evaluation both on synthetic and real-world data
  
Weaknesses:
-the writing of the paper lacks clarity in some aspects
-the novelty has to be better justified with respect to other baselines
-some aspects of the experimental setup were not convincing

In the rebuttal, authors have provided additional results and precisions on their work.  
During discussion, it has been recognized that the authors made an important effort to improve and precise their work, however there was a consensus among the reviewers to say the work still needs a global revision and that the elements provided are not sufficient for ICLR.  
I propose then rejection.  

justification_for_why_not_higher_score: There was a consensus among the reviewers that the paper needs a major revision, there are too many elements to modify.
Decision: Reject