Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: CAPE proposes a channel-attention-based method of generalizing existing neural PDE architectures to support training on equations with varying parameters. This is then combined with a curriculum learning strategy during training time. Evaluations are done on advection, Burgers, and 2d NS equations with Fourier Neural Operator and UNET being used as a baseline.
strength_and_weaknesses: Strength: A good number of PDEs are chosen to demonstrate the strengths (and weaknesses) of the CAPE method. A need for more parameter general neural PDE solvers is motivated and made clear.

Weakness: While other methods for incorporating PDE parameters are said to be insufficient, it would be useful to be comparisons against such naive methods. Additionally, ablations on the CAPE method would be useful to see which components of the architecture/curriculum learning strategy are the most critical. Also I would be curious to see a CAPE like method being used to augment pre-trained solvers, instead of the joint training required here.
clarity,_quality,_novelty_and_reproducibility: "An alternative approach attaches an external parameter embedding module to the network. However, there are too many
possible module structures and methods to provide the embedded parameter information to the base
network, and it is in general non-trivial to select the best one. Contrary to these ideas, we propose
a new and effective parameter embedding module by utilizing the channel-attention method" - It is unclear to me what is being said here. 
summary_of_the_review: Generalization of PDE parameters is motivated as an important problem and the CAPE method using channel attention + curriculum learning is proposed as a solution which can be combined with existing solvers. However additional experimentation demonstrating the insufficiency of existing methods for generalization + some ablations is absent.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  This paper proposes a CAPE module which allows any data-driven SciML models to incorporate PDE parameters. The key idea is to  transform the input variables $u_k,\lambda$  into an  intermediate field data, and make the final prediction based on the original input and the intermediate output.
strength_and_weaknesses: Strength: The problem considered is interesting and the proposed method seems novel.

Weaknesses:
The biggest limitation from the reviewer's point is that the proposed CAPE module is not well motivated. More discussion is needed on why this module is introduced. The author mentions that the straightforward  method of including the PDE parameters as additional input will have negative impact on the accuracy of the network. However, no reference  is provided and  no experiments include this method as a baseline. The reviewer believes that the author should provide more references on the similar topic and add  more baseline methods for comparison, for example, $u^{k+1} = Base([u^k,\lambda])$ and $u^{k+1} = Base([u^{k-1},u^k,\lambda]).$

Also, more explanations are needed on the ablation experiments. It is clear that on the Advection and Burgers model, the method without curriculum learning is at least as good as the full method, and what is the advantage of using curriculum learning?

Update on Dec. 5: I do not have further questions.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to understand and the proposed method seems novel.
summary_of_the_review: Please see Strength And Weaknesses.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed CAPE, which is a channel attention module that embeds PDE parameters. The module can be combined with off-the-shelf PDE solver. CAPE facilitates models to adapt to unseen PDE parameters, and is beneficial for inference efficiency. The authors also propose a curriculum learning strategy to improve model performance. Experiments are provided to demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strength:

* The proposed method is easy to understand, and the writing is clear. 

* The proposed method can reduce inference time since the method alleviates the need of running simulations. 


Weaknesses:

* My main concerns are about experiments.

  * The authors mentioned in the introduction that several approaches can be used to incorporate the PDE parameters into the model, such as adding the PDE parameters as additional inputs, and include the parameters in the embedding module. However, it is not clear why these methods are not good. And the authors should compare with these methods in the experiments.

  * The proposed method contains more parameters than existing methods because of the introduction of the channel attention module. Therefore, the current comparison is not fair. The authors should train all models with the same number of FLOPs (or equivalently same training time).

* I also have concerns about the proposed curriculum learning.

  * The design of the proposed curriculum learning seems rather arbitrary. What is the intuition here? Especially for Eq. 12, I cannot see the benefit of this particular deign. The authors should elaborate more on this part.
  * From the results in Table 3, it doesn’t seem like the curriculum learning method is effective. For the advection equation, w/o curriculum learning has the best performance. And for the Burgers equation, the curriculum learning method also does not increase model performance. 

Update:

I still believe the paper is below the acceptance bar. I've decided to keep my rating.
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: See above
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a channel-attention-based parameter embedding (CAPE) component for PDEs. The module takes as input the state time step t, and the PDE parameter labmda, predicts the state for several intermediate steps, which the main network takes as input (together with state at time t) to predict state at t+1. It details the architecture of the module, and also show in experiments that it results in improved accuracy.
strength_and_weaknesses: Strengths:

The paper addresses an important problem. The method seems to be novel.

Weaknesses:

It is not well justified or motivated why introduce this module. And why the simplest method of putting the parameter (or a transformation of the parameter) does not work. The improved accuracy of the module can be due to:
(a) The CAPE module contains the information of the PDE parameter
(b) It has more parameters than the model without this module, thus have higher expressivity
(c) The CAPE module needs to predict more future steps, thus having a regularizing effect.
(d) The inductive bias of the CAPE module is useful and can generalize better

I believe that the simple method of providing the PDE parameter will also benefit from the reason (a). To really demonstrate the usefulness of the CAPE module, the paper needs to rule out (a)(b)(c) (and other potential reasons) and then (d) can then be the likely reason.

To rule out (a), I think at least 2 more baseline are necessary: 
(I) a base network which takes as input the concatenation of the field data with the PDE parameter (expanded into channel features) concatenated on the channel dimension, i.e.
u^{k+1} = Base([u^k,\lambda])

(II) a base network which takes as input the concatenation of the field data with an embedding of PDE parameter:
u^{k+1} = Base([u^k,MLP(\lambda)])
where the MLP can be a 2- or 3-layer simple neural network. The layer number and the number of neurons needs to be hyperparameter searched.

In my previous experience, the above two simple methods works quite well, and can be a strong baseline.

To rule out (b), add another baseline, 
(III) a larger base network whose number of parameter is similar to the (4) CAPE + base network.

To rule out (c), the method needs to compare with a baseline where the base network needs to predict the same number of future steps as the CAPE.

The above baselines need to be hyperparameter searched with similar budget as in CAPE, to ensure a fair comparison.

There can also be other potential factors, but I think the above are necessary to show the benefit of CAPE.

Also, the CAPE architecture needs to be better justified

--
Update: 
Through the rebuttal, the reviewer has addressed my concerns and resolved the misunderstanding. Thus I have increased my score. 

clarity,_quality,_novelty_and_reproducibility: Quality: medium

Clarity: fair

Novelty: fair

Reproducibility: fair.
summary_of_the_review: In summary, the module the paper proposes is interesting. However, in its current form, there is no enough empirical evidence to justify the benefits of CAPE. Therefore, in its current form, I recommend for reject. On the other hand, if the concerns are addressed, I'm willing to increase my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces CAPE, a channel attention module that incorporates PDE parameters and can be integrated with any existing PDE solver. This module allows models to adapt to new PDE parameters and enhances inference efficiency. The authors also suggest using a curriculum learning approach to further improve model performance. The paper includes experimental results that demonstrate the effectiveness of the proposed method.

The reviewers raised several concerns on the experimental results: (1) The experiments only considered the training and testing sets and all hyper-parameters are directly tuned on the test set. There is a potential overfitting to the test sets. The authors need to use a validation set to tune the hyper-parameters, and the test set is only for calculating MSEs.  Therefore, the quality of the improvement cannot be assured, especially when the std is large (Table 2). (2) The curriculum leaning seems not effective either, especially considering it directly tuned over the test sets. From Table 8, the strategy is only effective for 3/6 of the experiments. (3) As can be seen from Appendix D.1, the proposed method is not very robust to the hyper-parameters. The results could be more convincing if the authors could improve the experiments in the next version.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject