Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors introduce an activation function based on learnable Padé approximations. The numerator and denominator of the learnable activation function are polynomials of m and n, respectively. The authors name them Padé activation units (PAUs). The authors also propose a randomized a version of these functions that add noise to the coefficients of the polynomials in order to regularize the network. The authors show, at best, marginal improvements over a variety of baselines including MNIST, fashion MNIST, CIFAR10, and Imagenet. The authors also show that pruning neurons with PAU units results in slightly better accuracy that pruning neurons with ReLU units.

The improvements over baselines shown were marginal and I do not think they warrant publication at this conference. The accuracy improvements were no more impressive than other learned activation functions which the authors perhaps did not see, such as SReLUs (Deep Learning with S-Shaped Rectified Linear Activation Units) and APLs (Learning Activation Functions to Improve Deep Neural Networks).

** After author response **
Changing from reject to weak accept
The authors have included new experiments that compare to a wider range of learned activation functions. While not ground breaking, it shows that it is competitive with state-of-the-art learned activation functions and could have something to offer.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: 
This paper introduces a novel parametric activation function, called the Pade Activation Unit (PAU), for use in general deep neural networks. Pade is a rational function, which is a ratio of two polynomials, and which can very well approximate any of the usually used activation functions while having only a few parameters that can be learned from data. Moreover, the authors identify five properties that an activation function should have, and either prove or empirically show that PAUs satisfy all of them, unlike some of the baselines. Additionally, since Pade approximation can have poles and be unstable, this work introduces safe PAUs, where the polynomial in the denominator is constrained to attain values greater than or equal to one. Since one of the suggested properties is that a function using a given activation function be a universal function approximator, the authors provide a sketch of a proof that PAUs do allow that. This proof applies only to the unsafe version of the PAU, and it is unclear whether it extends to the safe PAU---an issue that is not mentioned by the authors.
Furthermore, the authors propose a stochastic version of PAU with noise injected into parameters, which allows regularization. The empirical evaluation is quite extensive, and the PAU is compared against nine baselines on five different architectures (LeNet, VGG, DenseNet, ResNet, MobileNet) on four different datasets (MNIST, Fashion MNIST, CIfar10, ImageNet) for the classification task. The evaluation confirms that PAUs can match the performance of or sometimes outperform even the best baselines while the attained learning curves show that PAUs also lead to faster convergence of trained models. Finally, the authors demonstrate that (and provide intuition why) using PAUs allow for high-performing pruned models.

I recommend ACCEPTing this paper as it is well written, extensively evaluated, and provides performance improvements or at least matches the performance of the best baseline across several datasets and model architectures.

My only two suggestions for improvement are a) make the universal approximation proof tighter by making sure that it extends to the safe PAU version, and b) evaluate the proposed activation function on tasks other than just classification.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This work proposes an activation function that contain parameters to be learned through training. The idea is to give the learning algorithm more "freedom" to choose a good activation function, and hopefully better performance can be achieved.

The paper is well written, and the experiment results look reasonable. However, there are several key issues.

1) as the authors stated, a "good" activation function should maintain the universal approximation property of the neural network. This seems not discussed for the PADE activation function.  Does (1) satisfy the conditions (i)-(v) listed in table I? Is there a rigorous proof? Table I seems to claim that the PADE based neural network satisfies (i), but there is no formal proof.

2)  In order to avoid poles, the activation function used in this work is (2). How well can (2) approximate (1)? What is the potential loss? Perhaps there should be more discussion on this - preferably some theoretical supports.

Overall, the reviewer feels that this paper starts with an interesting idea, but the developments on the theoretical side is a bit thin.
review_title: Official Blind Review #2

Meta-review: The paper proposed a new learnable activation function called Padé Activation Unit (PAU) based on parameterization of rational function. All the reviewers agree that the method is soundly motivated, the empirical results are strong to suggest that this would be a good addition to the literature. 
Decision: Accept (Poster)