Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The goal of this work is to automatically learn the activation functions via the combinations of different candidate activation functions for PINN. Specifically, it adopted gate function with a learnable parameter to identify the coefficients for different candidate activation functions. Extensive experiments are carried out to evaluate the proposed physics-informed activation functions on a variety of partial differential equations (PDEs). The results demonstrate the efficacy of the simple approach. 
strength_and_weaknesses: * Strength
1. It is interesting to automatically choose the activation functions for PINN
2. The proposed method is simple yet effective to solve PDE systems
3. Conduct extensive experiments on various PDEs to verify the effectiveness of the proposed approach

* Limitations
1. Technical innovation is somewhat limited. The proposed gate function to learn the coefficients of activation functions is the same as that in the literature [Qian et al. 2018]
2. Compare to more baselines. I am curious if you could compare the prior works that learn combinations of activation functions?
3. A few grammatical errors. For example, "by minimize the following objective function" in the bottom of page 2; and "we consider a extreme case of insufficient collocation points", a-> an

References:\
[Qian et al. 2018] Adaptive activation functions in convolutional neural networks, 2018.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and organized. However, the technical innovation is limited.
summary_of_the_review: It is interesting to automatically learn the activation functions for PINNs. The proposed method is simple yet effective for PDEs systems. However, this work adopted the same gate function method from prior work and apply it to PINNs. Hence, the technical innovation is somewhat limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors present a formulation to create a trainable activation function based on a convex combination of other activations.
This new mixture approach is intended to work on physics-informed neural networks (PINNd) where higher order derivatives play a role, thus causing problems for some of the piecewise linear based common activations.

The authors the provide an empirical evaluation as well as a link to the neural tangent kernel.
strength_and_weaknesses: The authors present a technically sound approach for PINNs. Evaluated on datasets that clearly present the difficulties in this domain. 

However, the main weakness of this paper is in the lack of comparison to other learnable activation functions. The authors present these family of activation functions in the section "Adaptive activation functions", yet they are not present in the evaluation.
Some of them could be relevant as they do provide higher oder derivatives. Maybe there is an issue with those activation functions and that is a good reason not to include them, but this is not clear from the paper.


clarity,_quality,_novelty_and_reproducibility: The paper is well written and technically sound. The gating function and components are presented clearly. The activation function can be reproduced from the information in the paper.
summary_of_the_review: In this paper, the authors present an activation function that is amenable to PINNs. The authors present the problem clearly and offer a solution for this particular context based on previous work. They however do not compare to other adaptable activation functions that might still behave well in this domain. 
The empirical evaluation would be much stronger, if you were to compare to that class of activations. In particular, it would be great to show the benefits of your activation function, or explain why the others are not suitable and demonstrate that empirically.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method for parameterizing activation functions to improve convergence in physics-informed-neural-networks. They demonstrate through empirical experimentation that the proposed method converges to better solutions than non-learned activation functions or adaptive slope methods. Finally, the authors provide an analysis based on Neural Tangent Kernel that sheds light on why the proposed method is effective.

strength_and_weaknesses: The method proposed by the authors addresses a significant issue in PINNs. The approach is demonstrated on 5 different PDEs. The method is compared to several baselines as well as some ablations. 

The NTK based analysis is quite hand-wavy. There is no treatment of the approximation error of the two-phase analysis and the notion that the NTK is learned to suite the underlying PDE is approached empirically with a single example.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. 
One thing that was not clear to me is: Do all activation functions in the network share the same parameters? or are they all learned independently? 
The approach is somewhat novel in the PINN literature.
The work appears to be reasonably reproducible. I have no concerns there.

The other reviewers have raised the criticism that the proposed method is very similar to existing work. I was not aware of that work when I wrote my review. I find that the existence of the prior work significantly reduces my view of the novelty of the authors' contribution. I have therefore reduced my score. I think the authors could improve their contribution by formally proving their NTK claims or perhaps by expanding the number of base activation functions to very large numbers (e.g. using a basis). 
summary_of_the_review: Overall this paper is a solid demonstration of an approach to address the training difficulty empirically observed in PINNs.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes trainable activation functions for physics-informed neural networks (PINNs). The basic idea is to consider a set of base activation functions, and build a trainable one as a convex combination of the base ones. They provide experiments on many problems, showing that PINNs are generally sensitive to the choice of AF, while the trainable one provides good performance across most cases. They also try to motivate the approach from the point of view of the neural tangent kernel (NTK).
strength_and_weaknesses: Strength: From the point of view of PINNs, the proposed neural networks have better accuracy and performance than the base ones. 

Drawbacks: 
- From the methodological point of view, the method is *equivalent* to the method proposed in (Manessi and Rozza, 2019), referenced in the paper, which was also proposed under the name of "adaptive blending unit" in (Sutfeld et al., 2020), which is not referenced here. The authors are renominating them as "physics-informed activation function" (PIAC), but as far as I can see there is no difference that warrants a change of name (apart from slightly modifying the number of base AFs). They mention that "previous methods experiment with limit choice of activation functions", but they use 5 base AFs which is similar to 6 base AFs in (Sutfeld et al., 2020). 
- The method is also compared only with fixed or parametric AFs, and not against similar expressive AFs (e.g., APLs).
- The NTK derivation is quite shallow and it does not provide any special insight into the AFs themselves.
clarity,_quality,_novelty_and_reproducibility: The paper is written well except for a small number of typos (e.g., "over-parameterizaiton"). The experimental part is well detailed but with an insufficient number of comparisons. Novelty is extremely small, as described above.
summary_of_the_review: The method is a rebranding of a known technique, applied on a different domain. While this is potentially interesting for PINNs, I feel this is not a good practice and it is also an artificial construct to increase novelty.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to learn an adaptive activation function (as a weighted sum of candidate functions) inside physics-informed neural networks (PINNs). The idea is simple and it is shown to improve the accuracy of the solution approximated by PINNs on multiple differential equations. This is the major strength of the paper acknowledged by all the reviewers.

During the discussion, the following points were raised by the reviewers:

- The idea of using learnable activation functions is not new and it has been explored in other domains (e.g. image classification). The authors admit that but they argue that the task of solving PDEs often benefits from using types of learnable functions which are different from the ones used in previous works (for example, using sin and exp in the trainable mixture may improve the performance). The authors argue that the idea of learnable activation functions has not been explored in the PINN community.

- Lack of comparison to other learnable activation functions. The authors presented experiments showing that the proposed adaptive activation function works well compared to previously proposed alternatives.

During the virtual meeting, the reviewers reached a general agreement that the results are definitely of interest to the PINN community but unfortunately the novelty is not strong enough for acceptance to ICLR. The reviewers encourage the authors to submit the paper to a venue which is more specialized on physics applications and PINNs (e.g. Journal of Computational Physics). The authors are also encouraged to improve writing by giving more credit to previous works on learnable activation functions and perhaps dampening the claim on the novelty of the proposed solution.
justification_for_why_not_higher_score: The idea of learning activation functions is not new and has been proposed in multiple papers.
justification_for_why_not_lower_score: The experimental results are good and the idea can be useful for training PINNs.
Decision: Reject