Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed combined form of flexible activation functions with carefully designed principle of choosing activation functions. It shows some gains on stock price and one standard image task over the baseline.

I'm leaning to reject or give borderline for this paper because (1) the paper don't have comparison with neural architecture search. For example, https://arxiv.org/abs/1710.05941 (Searching for Activation Functions). I don't know what the advantage of this approach compared to searched activation. I guess it's less computation heavy and maybe better motivated. But at least the author should give some pros/cons. (2) the paper has two benchmark, stock price prediction and CIFAR-10. I don't understand why as arch paper, it use such non standard benchmark (stock) and non standard arch (LeNet?). I don't think based these benchmark we can make solid conclusion. (3) These model seems introduce quite a bit more hyper-parameters. It's unclear if it is better than tuning other architecture e.g., batch norm/layer norm/dropout or even just optimizer. For example, "flexible is 0.032" does this parameter generalize to other dataset? Like if the gain is really significant, like resnet over AlexNet, hyperparam doesn't matter. But if it's marginal win over a weak baseline, the how to get the results is important.

Some comments:
In section 2, " back propagation of these activation parameters by stochastic gradient descent can be
done as follows"
Why we need to list the detail backprop formulation here? Are these special? Isn't just autograd?

Can the author explain more for principle 1? What is the "same domain" means here?
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The authors introduce a parameterized activation function to learn activation functions that have sigmoidal shapes that can be used in LSTMs. The authors apply their method to a dataset of forecasting stocks as well as to CIFAR-10. They also propose a method to regularize the activation function parameters.

The authors propose an activation function that helps very little in certain cases. I am not familar with this stock prediction dataset, but the differences shown are often less than 1%. I am familiar with CIFAR-10 and the architecture they use give bad baseline results and their method of regularizing a previous learned activation function (PReLU) gives marginally better results in half the cases.

There have been many learned activation functions proposed that give significantly better results than these. This paper is simply proposing another one and showing little to no improvement.

**After reading author feedback**
My score stays the same.

There were some issues I had with the response:
"For stock return forecasting, 1% improvement with statistical significance is sufficiently import...As we know, most newly proposed unbounded activation functions (flexible or not) cannot outperform ReLu by more than 0.5% in terms of test accuracy in a large range of image classification tasks."
The 1% improvement was on error, not accuracy, so the two are not related. If referring to error rate, there are many that can make improvements greater than 10% (APLs, SReLUs, Swish, etc.)
If the authors would like to claim superiority to other activation functions, they should compare to them directly.

In addition, my concern about the poor CIFAR-10 baseline was not addressed. The results from the 2014 dropout paper gives significantly better results (Dropout:  A simple way to prevent neural networks from overfitting) than the authors' baseline.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a family of parameterized composite activation functions, and a regularization technique for parameterized activation functions in general. While the three sets of experiments show potentially promising results, they aren't able to disambiguate clearly between the effect of the activation function you introduce and the effect of additional parameters in general, or between the effect of the regularization technique you introduce and the effect of regularization in general. I would really like to see the kind of carefully baselined, ablated, and hyperparameter-tuned results that would justify adding the techniques introduced to the toolbox of a typical deep learning practitioner.

Some feedback:

Typos:
- The formatting is off a bit (shifted horizontally?)
- In the abstract: RPeLu -> PReLU
- p. 5: "basement settings"->baseline, "logarithmic sale"->scale
- p. 6: basement->baseline again
- p. 7 etc.: trail->trial

Feedback:
- Intro:
  - Explain why maxout is similar to training piecewise activation functions?
  - It's unclear what "making the most of the non-linear properties by introducing adaptation policy on the input" means
  - An "autoencoder" is not an architecture as much as a broad family of architectures coupled with training approaches (I'm guessing you mean a fully-connected autoencoder)
- Methodology:
  - consider using "f" instead of "a" so it's easier to tell apart from alpha?
- Experiments:
  - I'm not sure I'm convinced by the statistical tests used on the LSTM results; they demonstrate that your approach, with a few specific sets of hyperparameter settings, does better than the baseline, but not that this represents a valid claim about your activation function's effect on this LSTM model in general.
  - The autoencoder experiments are even less convincing, in that they represent two seemingly arbitrary network architectures, with equally arbitrary placement of the activation function in one of them.
  - The regularization experiments use LeNet-5, which is not a compelling benchmark architecture with respect to contemporary practice. The effects of regularization techniques can be very different in different regimes of dataset and network size.
- Code:
  - I'm not sure why you performed backprop by hand for your activation functions and used torch.Function, rather than writing them directly as nn.Modules that make use of PyTorch autograd
  - There are lots of details in the code that aren't in the paper; in general, papers should aim to be relatively self-contained (although I'm very glad to see your code, and it's pretty simple to follow)
review_title: Official Blind Review #2

Meta-review: Main content: Proposes combining flexible activation functions 

Discussion:
reviewer 1: main issue is unfamiliar with stock dataset, and CIFAR dataset has a bad baseline.
reviewer 2: main issue is around baselines and writing. 
reviewer 3: main issue is paper does not compare with NAS.

Recommendation: All 3 reviewers vote reject. Paper can be improved with stronger baselines and experiments. I recommend Reject.
Decision: Reject