Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary:** This paper proposes to use evolutionary computation to search for optimal activation functions tailored to a given architecture. The proposed PANGEA method practically re-implements a partial Genetic Programming setup applied to searching for neural network activation functions, with interesting (though minor) results. Further investigation is necessary.

**Quality:** The paper is well written, with a good structure. The literature review covers most basis, though with some crucial misses (GP and Random Guessing). The investigation presented is correct and through, though insufficient in breadth. The results are significant and interesting. The core of the paper is well put together, but it requires some more work.

**Clarity:** The concepts are explained clearly and well exposed. The results are motivated and discussed. On the actual work presented, I am happy with the clarity of exposition and presentation.

**Originality:** The work explores a highly promising direction, building on previous results but with a new twist. The results are bare-bone proof-of-concept, the baseline misses some foundation work, and the actual improvements are marginal at best.

**Pros:**
- Very interesting and promising research direction
- Correct experimentation methods
- Very well written and easy to read
- I personally find this approach highly promising and innovative, and fully support its investigation

**Cons:**
- While the direction taken is most intriguing, the actual quantity of study is relatively small
- The claims are not supported by sufficient experimentation (insufficient experiments)
- Some claims are not supported at all (tailoring activation to architecture)
- Missing referencing and comparison with GP.

**Comments:**
- Working with 1 dataset and 3 variations of 1 architecture is too small a baseline to prove the point. While I personally believe in the presented results, further experimentation is necessary.
- The paper exposure is very pleasant, but it effectively hides the relatively small size of the presented work. As a results it sounds redundant at time, with over-sized pictures and over-complete tables, while the Conclusion paragraph is shorter than the Future Work.
- Figures 4 and 5 are unnecessarily over-sized and need to be shrunk.
- The lack of mention of Genetic Programming is unacceptable, especially given the paper available space. Reinventing part of it and rebranding it as PANGEA  for the specific application is interesting but insufficient.
- To establish the performance of a highly randomized algorithm such as PANGEA, a pure-random parameter guessing baseline needs to be established. Under the proposed rebranding, that would be a run where the operator Regenerate has a 1.0 chance while the others have 0.0.
- The claim that the evolved activation function is tailored to the architecture is central to the work, but it is not proven. It should be shown with cross-evaluation of activation functions and architectures or removed from the paper.
- Might be an oversight on my part, but it is not clear how many activation functions were evolved for each architecture (3 are shown in the results table) and how their performance is aggregated (in the results table, it is not). As mainly randomized algorithms, evolutionary methods can converge to very different results over subsequent runs, especially in the presence of highly multimodal, indirectly computed fitness landscapes such as this. The method needs then to be evaluated as a whole, not the single produced results, as the performance eventually depends on the number of reruns (tending to infinity).
- Paragraph 3.2: while Genetic Algorithms and related Genetic Programming are indeed discrete search algorithms, state-of-the-art Evolution Strategies are excellent continuous optimization algorithms. Please fix the claim accordingly.
- First paragraph in 3.3: while reading up on ES, please adopt the established $(\mu,\lambda)$ and $(\mu+\lambda)$ terminology in place of the introduced $P$ and $S$.
- The code is not published as open source, limiting distribution and reproducibility.

**Final remarks:**
The number of papers crossing and integrating SGD (or DL) and EC (or NE) is growing over the years. This is very good news for the field, but makes it important that papers exploring new foundation directions such as this one are incontestable and thoroughly proven. If the paper is accepted to ICLR21, the authors have my congratulations and best wishes; but if not, rather than just resubmitting the same content to another conference, I urge them to expand their work beyond reproach, as this direction (and the proven quality of your experimentation and writing) has the potential for an absolutely top-quality publication.

review_title: With great intuitions come extra effort expectations

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors focus on the task of discovering activation functions via evolutionary methods. The paper is well written, easy to follow and technically sound. 
My main concern with this paper is that the authors mention the importance of having activation functions learned for particular networks/tasks, yet failed to compare against any of the learnable activation functions that have the same objective.
Some of those activation functions are, e.g., APL [1], PAU [2], Mixtures of activations [3], and SReLU [4].

As a positive aspect, I see the benefit of your paper in that the given set of evolutionary operations might achieve some of the previous activation functions. However, your approach has an associated computational cost that is very high, and therefore it is very important to distill what makes it work and raise the following questions:

- Is it possible that the improvements come from the parametrization?.
- What is the behavior when you do exploration but not allowing parameters?.

One very interesting aspect of the functions that you learned, is that the form $log(\sigma(x))$ shows up repeatedly and in some cases including parameters. Mathematically, this is equivalent to $-SoftPlus(-x)$. The general form can be easily achieved by a parametric version $\alpha SoftPlus(\beta x)$, which appears often in table 2. Do you think this would be one big take-away message? The swish suggestion of your paper?
Since the SoftPlus is a soft version of the ReLU, one could also find interesting $\alpha ReLU (\beta x))$.

Indeed, your search algorithm is useful, but the results seem to point in the direction of parametric learnable activation functions, therefore a comparison to that family of activations is very important.


[1] F. Agostinelli, M. D. Hoffman, P. J. Sadowski, and P. Baldi. Learning activation functions to improve deep neural networks. In Workshop Track Proceedings of the International Conference on Learning Representations, 2015.

[2] A. Molina, P. Schramowski, K. Kersting. Pade Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks. In International Conference on Learning Representations, 2020.

[3] F. Manessi and A. Rozza. Learning combinations of activation functions. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 61–66. IEEE, 2018.

[4] X. Jin, C. Xu, J. Feng, Y. Wei, J. Xiong, and S. Yan. Deep learning with s-shaped rectified linear activation units. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.

[5] Ramachandran, P., Zoph, B. and Le, Q.V., 2017. Searching for activation functions. arXiv preprint arXiv:1710.05941.
review_title: Missing the whole family of learnable activation functions

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose to search for activation functions with regularized evolution, an evolutionary algorithm proposed by Real et al. Various mutations are proposed that allow to investigate a larger search space than prior work. In particular, a mutation is added which adds trainable parameters to the activation function. The discovered activation functions are compared on three different architectures to several state-of-the-art activation functions.

The authors clearly describe their method. The idea and method are well-motivated and leave no scope for criticism. Experimental results indicate that the discovered activation functions are significantly better than a wide range of alternatives. Many related works are discussed, the authors might consider adding [1] to their discussion.
Currently unclear are the contributions of this work. In my understanding the authors claim to use different mutations and a different search space. Given the title, the main contribution seems to be the addition of parameters to the activation function. However, this idea is not new. It is e.g. used for PReLU but also the related work by Ramachandran et al. considers the option to add parameters as a unary operator. The authors attempt to study the influence of adding these parameters to the search space by investigating how the effective learning rate changes during training and at various depths. It would be interesting to see how the method performs without the proposed addition in section 3.2. Do any preliminary results indicate that this is significantly improving the results?
Very related to this: in the appendix you define the swish activation as x*\sigma(x). It would be interesting to know whether the authors used this function or the parametric version proposed by the original authors defined as x*\sigma(\beta * x).

The authors investigate whether the activation function generalize across architectures and notice that many do not. It would be interesting to know whether the discovered activation functions generalize to different data sets. The authors claimed that CIFAR-100 is more difficult than CIFAR-10. How do the transferred activation functions compare in this case to the baselines?

The generalization is only considered on ResNets. How about entirely different architectures such as some without residual connections, DenseNets, NASNet, EfficientNet or MobileNet? Will it also transfer to recurrent neural networks and other sorts of non-image architectures?

Concluding, the paper is well-written and seems to provide a solid empirical evaluation. I think the authors could be more explicit about the novelties presented in this work and why they matter.


[1] Jin-Young Kim, Sung-Bae Cho: Evolutionary Optimization of Hyperparameters in Deep Learning Models. CEC 2019: 831-837

**After Authors' Response:**
In my understanding the main contribution of this work is the identification of a new search space for activation functions. This might be very similar to earlier ones but the authors convinced me that it is a useful contribution. All my comments were addressed and as far as I saw also the comments of all other reviewers were addressed. For this reason, I increased my score.
review_title: Solid paper with some unclarities in particular to contributions

Meta-review: The paper proposes the idea of searching parameterized activation functions, in contrast to the previous handcraft or learnable ones. It may be a counterpart of neural architecture search.

Pros:
1. The idea is very interesting.
2. The paper is well written.
3. The experiments show improvements over baseline activation functions.

Cons:
1. The AC fully agreed with Reviewer #4 that the whole literature of learnable activation function is neglected (Reviewer #2 also alluded to this issue). Although the authors added experiments with learnable baseline activation functionss, the literature review on learnable activation function was not included accordingly.
2. Although the idea of searching activation functions is interesting, the AC doubted the necessity. Since the rich literature of learnable activation functions is already there (note that it is more than introducing parameters to handcrafted ones), can we simply learn piecewise linear activation functions with more pieces so that it can approximate complex enough functions? This can be much more easily implemented (can go along with weight training on the standard deep learning platform) and the computation cost will be much lower. Such a comparison is absolutely necessary.
3. The AC was actually worried about the activation functions founded as they may be too complex, so the generalization issue (even numerical stability issue) may be a concern. More thorough testing is necessary (currently only tested on CIFAR-100 and three CNNs; and Reviewers #3 and #2 also concerned about this issue).

Although Reviewer #2 raised his/her score, the final average score is still below threshold. So the AC decided to reject the paper.
Decision: Reject