Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks
Authors: Alejandro Molina
Patrick Schramowski
Kristian Kersting
Venue:  (2020)

Abstract: The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness.

Published as a conference paper at ICLR 2020
PADÉ ACTIVATION UniTS: END-TO-END LEARNING OF FLEXIBLE ACTIVATION FUNCTIONS IN DEEP NET- WORKS
Alejandro Molina', Patrick Schramowski', Kristian Kersting1.2 1.2
1 Al and Machine Learning Group, CS Department, TU Darmstadt, Germany
2 Centre for Cognitive Science, TU Darmstadt, Germany Imolina, schr kersting) @cs. tu-darmstadt..de
ABSTRACT
The performance of deep network learning strongly depends the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representa- tions. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way approximations with provable robustness.
https://gitiin thub...ttttuuttoo/gg com ml resea research/pau
INTRODUCTION
An important building block of deep learning is the non-linearities introduced by the activation functions fzr) They play major role in the success of training deep neural networks, both in terms of training time and predictive performance. Consider c.g. Rectified Linear Unit (ReLU) due to Nair
and Hinton 2010). The demonstrated benefits training deep networks, see e.g. (Glorot et al. 2011 brought renewed attention to the development of new activation functions. Since then, several ReLU variations with different properties have been introduced such as LeakyReLUs (Maas al. 2013). ELUs (Clevert et al. 2016), RReLUs (Xu et al. 2015), among others. Another line of research, such as (Ramachandran et al. (218)) automatically searches for activation functions. identified the Swish unit empirically
good candidate. However, for given dataset, there guarantees that Swish unit behaves well and the proposed search algorithm is computationally quite demanding.
The activation functions are traditionally fixed and, in turn, impose set of inductive biases on the network. One attempt to relax this bias are for instance PReLUs (He et al., 2015), where the negative slope is subject optimization allowing for more flexibility than other ReLU variants. Learnable activation functions generalize this idea. They exploit parameterizations of the activation functions, adapted in an end-to-end fashion different network architectures and datasets during training. For instance, Maxout (Goodfellow et al. 2013) and Mixout (Zhao et al. (017)) use a fixed set of piecewise linear components and optimized their (hyper-)parameee
Although they are theoretically universal function approximators, they heavily increase the number of parameters of the network and strongly depend on hyper-parameters such as the number of components to realize this potential. /ercellino and Wang (2017) used a meta-learning approach for learning task-specific activation functions (hyperactivations). However, as Vercellino and Wang admit, the implementation of hyperactivations, while easy to express notationally, can be frustrating to implement for generalizability over any given activation network. Recently, Goyal et al. (0099) proposed a learnable activation function based on Taylor approximation and suggested transformation strategy avoid exploding gradients. However, relying on polynomials suffers from well-known limitations such as exploding values and a tendency to oscillate (Trefethen) 2012).

---

Published as a conference paper at ICLR 2020
ReLU
Sigmoid
Tanh
2
Swish
f(z
Leaky ReLU
1
ReLU
Sigmoid"
Tanh
Swish*
Leaky ReLU*
3
2
Figure I: Approximations of common activation functions (ReLU, Sigmoid, Tanh, Swish and Leaky ReLU (a 0.20)) using PAUs (marked with *). As one can see, PAUs can encode common activation functions very well. (Best viewed in color)
As alternative, here introduce learnable activation function based the Padé approximation, i.c., rational functions. In contrast to approximations for high accuracy hardware implementation of the hyperbolic tangent and the sigmoid activation functions (Hajduk 2018), we do not assume fixed coefficients. The resulting Padé Activation Units (PAU) can be learned using standard stochastic gradient and, hence, seamlessly integrated into the deep learning stack. PAUs provide more flexibility and increase the predictive performance of deep neural networks, we demonstrate.
proceed as follows. start off by introducing PAUs. Then introduce Padé networks and show that they are universal approximators. Before concluding, we present our empirical evaluation.
2 PADÉ ACTIVATION UNITS (PAU)
Our starting point is the set of intuitive assumptions activation functions should ultimately fulfill shown Tab. [ The assumptions v concern the ability neural networks to approximate functions. Rational functions can fulfill assumptions ilvv and our experimental evaluation demonstrates that
assumptions ii also hold.
Assumptions
(i) They must allow the networks to be universal function approx- imators.
Activation
Learnable
ii
iii
IV
ReLU
N
Y
Y
Y
Y
ReLU6
N
Y
Y
Y
Y
RReLU
N
Y
Y
Y
Y
They should ameliorate gradi- ent vanishing.
LReLU
N
Y
Y
Y
Y
ELU
N
Y
Y
Y
Y
CELU
N
Y
Y
Y
Y
(iii)
They should stable.
Swish
N
Y
(iv) They should be parsimonious
PReLU
Y
Y
Y
Y
on the number of parameters.
Maxout
VYVEANA
Y
Y
YYY
Mixture
Y
(v) They should provide networks with high predictive perfor- mance.
APL
Y
Y
YYY
YY
Y
SReLU
Y
Y
Y
SLAF
Y
Y
N
Y
PAU
Y
Y
Table 1: (Left) The intuitive assumptions activation functions (AFs) should ultimately fulfill. (Right) Existing FLU, CELU and ReLU like AFS not fulfill them. Only learnable AFs allow one to tune their shape training time, ignoring syper-parameters such for LReLU. For Swish, our experimental results do not indicate problems with vanishing gradients. SLAF (Goyal et al. 2019) showed undefined values i and we could not judge their performance
2

---

Published as a conference paper at ICLR 2020
2.1 PADÉ APPROXIMATION OF ACTIVATION FUNCTIONS
Let us now formally introduce PAUs. Assume for the moment that we start with a fixed activation function (()) The Padé approximant (Brezinski and Van Iseghem| 1994) is the "best" approximation of (xx by rational function of given orders m and n.. Applied to typical actication functions, Fig. shows that they can be approximated well using rational functions.
More precisely, given fz)), the Padé approximant is the rational function F(x) over polynomials P(x), ((x) of order m, n of the form
P(x) F(x)= ((x) 1+2 baok
E E5z Gg + 01T + a22² + + am
(1)
b₁ + by2 + b,
which agrees with (()) the best. The Padé approximant often gives a better approximation of a function (1) than truncating its Taylor series, and it may still work where the Taylor series does converge. For these reasons, it has been used before in the context of graph convolutional networks (Chen et 2018). However, they have not been considered so far for general deep networks. Padé Activation Units (PAUs) go one step further, instead of fixing the coefficients aj. be to approximate a particular activation function, we allow them to be free parameters that can be optimized end-to-end with the rest of the neural network. This allows the optimization process to find the activation function needed at each layer automatically.
The flexibility of Padé is not only blessing but might also be a curse: it can model processes that contain poles. For a learnable activation function, however, a pole may produce undefined values depending on the input well as instabilities at learning and inference time. Therefore we consider restriction, called safe PAU, that guarantees that the polynomial Q(x) ) is not 0, i.e., we avoid poles. Rgo, but P(x:) + TR we can focus on (xx) + Ryo wlog. However, as limo(=)) F(z) + co learning and inference become unstable. To fix this, we impose a stronger constraint, namely Q(z) 2 q > 0. In this work, q 1, i.., Vz : (z)) 1, preventing poles and allowing for safe computation R:
In general, restricting (xx) implies that either (2x) + R=0 or ((x)
P(x) F(x) (x))
Em E E bprk
+ a1 + 022
+ am mm
|b + byx2
Other values for q E (0, 1) might still be interesting, as they could provide gradient amplification due to the partial derivatives having Q(X) in the denominator. However, we leave this for future work.
2.2 LEARNING SAFR PADÉ APPROXTMATIONS USING BACKPROPAGATION
In contrast to the standard way of fitting Padé approximants where the coefficients are found via derivatives and algebraic manipulation against given function, we optimize their polynomials via backpropagation and (stochastic) gradient descent. To do this, we have to compute the gradients with respect the parameters Day' 1 abg implement the forward pass as described
OF A simple alternative is to oF 8F as well as the gradient for the input du Eq. 2 and let automatic differentiation do the job. To be more efficient, however, we can also implement PAUs directly in CUDA (Nickolls et al. 2008). ) and for this we need to compute the gradients ourselves:
OF =
OP(x) 1 aQ(x) P(x) OF x²
A(X) P(X) ab = x (A()))((x)*
OF
(x) and
Oz
dr Q(x) Or Q(x)²' da,
where 8P(x) a + 2 +... + mam m-1 aQ(w) A(X)
(bL + 2byæ + ..... nbb,."
Bu
[A(]
A(X) = b + 222² + ... + b,,""' and Q(x) = 1 + (A(X). Here we reuse the expressions to
[ the sign of With the gradients at hand, PAUs can seamlessly be placed together with other modules onto the differentiable programming stack.
reduce computations. To avoid divisions by zero when computing the gradients, define &
3 PADÉ NETWORKS
Having PAUs at hand, one can define Padé networks as follows: Padé networks are feedforward networks with PAU activation functions that may include convolutional and residual architectures
3

---

Published as a conference paper at ICLR 2020
with pooling layers. To use Padé networks effeetively, one simply replaces the standard activation functions in a neural network by PAUs and then proceed to optimize all the parameters and use the network usual. However, even if every PAU contains low number of parameters (coefficients aj by). in the extreme case, learning one PAU per neuron may considerably increase the complexity of the networks and in turn the learning time. To ameliorate this and inspired by the idea of weight- sharing as introduced by Teh and Hinton (2001), we propose to learn one PAU per layer. Therefore we only add & many parameters, where @ L * (m + n) and L is the number of activation layers in the network. In our experiments we set 10L, rather small number of parameters (iv).
The last step missing before we can start the optimization process initialize the coefficients of the PAUs. Surely, one can do random initialization of the coefficients and allow the optimizer to train the network end-to-end. However, we obtained better results after initializing all PAUs with coefficients that approximate standard activation functions. For discussion on how to obtain different PAU coefficients, we refer Sec. A.I
Before evaluating Padé Networks empirically, let touch upon their expressivity and how to sparsify them.
3.1 PADÉ NETWORKS ARE UNIVERSAL FUNCTION APPROXIMATORS
standard multi-layer perceptron (MLP) with enough hidden units and non-polynomial activation
functions is a universal approximator, see e.g. (Hornik et al. 1989 Leshno et al. 1993). Padé Networks are also universal approximators.
Theorem 1.
Let p: R R be PAU activation function. Let No represent the class of neural networks with activation funcctioon Let K RD be compact. Then No is dense in C(K).
The proof holds for both PAUS and safe PAUS as it makes no assumptions on the form of the denominator Q(z), and is direct application of the following propositions:
Proposition 1. (From Theorem 1.1 in Kidger and Lyons 2019)) Let p: R- R be any continuous function. Let Nº represent the class of neural networks with activation function p. with n neurons in the input layer, neurons. Let K C R" be compact. Then Ng. is dense in C(K) ) if and only if p is non polyyolyl...ll.
neuron in the output layer, and one hidden layer with arbitrary number of
(From Theorem 3.2 in 'Kidger and Lyons 2019)) Let p: R- R be any continuous function which is continuously differentiable at at least one point, with nonzero derivative at that point. Let K C R be compact. Then NNP n.m.n is dense in C(K;RT)
Proposition
Proof. Let p(z) = P(x)/Q(x), we have to consider the following two cases:
Case 1: Q(x) * 1, by definition, p(r) is non-polynomial. Then by proposition 1, we get that N/p is dense in C(K).
that.N
Case 2: Q(x) 1, here p(z) Σド= a,x², is polynomial, continuous and continuously differen- tiable in R. Let any aj>0 O, then there exists a point E IR such that p'(a)) f 0 and then by proposition 2, we get that NN ANN,,, n=22 is dense in C(K:R")..
3.2 SPARSE PADÉ NETWORKS AND RANDOMIZED PAUS
Padé Networks can e-approximate neural networks with ReLU activations (Telgarsky) 2017). This implies that by using PAUs, we are embedding a virtual network into the networks we want to use. This, in turn, is the operating assumption of the lottery ticket hypothesis due to Frankle and Carbin (2019). Thus, smaller than their original counterparts while reducing inference time and potentially improving the predictive performance.
expect that lottery ticket pruning find well-performing Padé networks that are
Generally, overfitting is an important practical concern when training neural networks. Usually, can apply regularization techniques such Dropout (Srivastava al.] 2014).. Unfortunately, although each PAU approximates a small ReLU network section, we do not have access to the internal representation of this virtual network. Therefore,
regularize the activation function via
4

---

Published as a conference paper at ICLR 2020
RPAU
SReLU
ReLU
SLAF
ELU
CELU
Swish
PAU
PReLU
RReLU
ReLU6
APL
LReLU
90.50
0.00150
90.00
-0.00135
LeNet
Accuracy
89.00
00,0120
0.00105
88.50 5.50
00.00000
88,
87.50
0.000075
87.00
0.00060
0,00045
20
40
63 80 100 20 40 60
80
100
Figure 2: PAU compared to baseline activation function units over runs on Bashion-MNIST using the LeNet architecture: (left) mean test-accuracy (the higher, the better) and (right) mean train-loss (the lower, the better). As one can see, PAU outperforms all baseline activations and enable the networks to achieve lower loss during training compared to all baselines. (Best viewed in color)
standard Dropout. An alternative for regularizing activation functions introduced in Randomized Leaky ReLUs (RReLUs, Xu et al. (2013), where the negative slope parameter is sampled uniformly on range. This makes the activation function behave differently at training time for every input æ, forwarding and backpropagating according to x and the sampled noise.
We can employ a similar technique make PAUS resistant to overfitting. Consider PAU with coefficients C = {ao, , am. bo, J. We can introduce additive noise during training into each coefficient c; E C for every input 2y via Cr, C + Zz,j where 2s U14,, u,), 4 = (1 a%) C2
and reciprocally = (1 + a%) Cz. This results in Randomized PAU (RPAU):
C0.j + C1,5 + 22 4 + Cm, at
R(x,) = + 1cm-1 + Cm+222 + Cm-+22
(3)
We compute the gradients as before and simply replace the coefficients by their noisy counterparts.
4 EXPERIMENTAL EVALUATION
Our intention here is to investigate the behavior and performance of PAUs as well as to compare them to other activation functions using standard deep neural networks. All our experiments are implemented in PyTorch with PAU implemented in CUDA, and were executed on an NVIDIA DGX-2 system. all experiments, we initialized PAUS with coefficients that approximate Leaky for a rational function of order = 5, 4. In all experiments except for ImageNet, we report the mean of 5 runs initialized with different seeds for the accuracy on the test-set after training. And, we compared PAU to the following activation functions: ReLU, ReLU6 Leaky ReLU (LReLU), Random ReLU (RReLU), ELU, CELU, Swish, Parametric ReLU (PReLU), Maxout, Mixture of activations (Mixture), SLAF,, APL and SReLU. For details on all the activation functions, refer to Appendix A.2 As datasets we considered MNIST, Fahion-MNIST, CIFAR-10 and ImageNet.
4.1 EMPIRICAL RESULTS ON MNIST AND FASHION- BENCHMARKS
First we evaluated PAUS on MNIST (LeCun et al. 2010) and FaShion-MNIST (iao et al. 2017) using two different architectures: LeNet (LeCun et al. 1998) and VGG-8 (Simonyan and Zisserman 2015). For more details on the architectures, learning settings, and results, we refer Sec. A.3
As can be seen in Fig. 2 and Tab, 2. PAU outperformed on average the baseline activation functions on every network in terms of predictive performance. Moreover, the results are stable on different runs (c.f. mean + std). PAUs also enable the networks to achieve a lower loss during training compared to all baselines on all networks. Actually, PAU achieved the best results on both datasets and on Fashion-MNIST it provides the best results for both architectures. As expected, reducing the bias is beneficial in this experiment. Comparing the baseline activation functions on the MNIST dataset and the different architectures, there is clear choice of activation that achieves the best performance. However, PAU always matches or even outperforms the best performing bascline activation function. This shows that learnable activation function relieves the network designer of having commit to
5

---

Published as a conference paper at ICLR 2020
Table 2: Performance comparison of activation functions on MNIST and Fashion-MNIST (the higher, the better) on two common deep architectures. Shown are the results averaged over 5 reruns as well as the top result among these runs. The best ("*") and runner-up (**"") results per architecture are bold. As one can see, PAUs consistently outperform the other activation functions on average and yields the top performance on each dataset.
VGG-8 mean + std
LeNet mean std
VGG-8 mean std
LeNet mean + std
best
best
best
best
MNIST
ReLU
99.17 + 0.10
99.30
99.17 + 0.05
99.25
89.11 4: 0.43
89.69
89.86 + 0.32
90.48
ReLU6
99.28 + 0.04
99.31
99.09 + 0.09
99.22
89.87 + 0.62
90.38
89.74 + 0.27
89.96
LReLU
99.13 1
99.27
99.10 0.06
99.22
89.37 + 0.30
89.74
89.74 0.24
90.02
RReLU
09.16
99.28
99.20 0.13
099.38
88.46 + 0.85
89.32
89.74
89.88
ELU
99.15
99.28
99.15 + 0,06
99.22
89.65 + 0.33
90.06
90.25
CELU
99.15
99.28
99.15 0.06
99.22
89.65 + 0.33
90.06
89.84
90.25
Swish
99.10 0.06
99.20
99.19 0.09
99.29
88.54 0.59
89.36
19.44+0.22
89.89
PReLU
99.16 0.09
99.25
99.14 0.09
99.24
88.82 + 0.51
0.51
89.54
90.09 0.22
c90.29
SLAF
90.60 1 0.00
90.60
1.233
89.80
APL
99.35
1 0.11
99.50
99.18 1 0.10
99.33
91.41 1 0.48
92.25
89.72 0.30
90.01
SRtLU
99.15 1 0.03
99.20
99.13 1 0.14
99.27
89.65 1 0.42
90.31
89.83 1 0.30
90.28
PAU
99.30
0.05
c99.
40
999.21
0.04
99.26
99..25
0.18 o91.56
90.33
0.15 90.62
RPAU
99.35 + 0.04
99.38
99.26
1 0.11
99.42
91.23
1 0.15
91.41.90.0
20
1 0.11 c90.29
potentially suboptimal choice. Moreover, Fig. also shows that PAU is more stable than SLAF This not unexpected Taylor approximations tend to oscillate and overshoot (Trefethen] 2012). We also observed undefined values at training time for SLAF; therefore, we do not compare against it in the following experiments. Finally, when considering the number of parameters used by PAU, we can see that they are very efficient. The VGG-8 network uses 9.2 million parameters, PAU here uses 50 parameters, and for LeNet, the network uses 0.5 million parameters while PAU uses only 40.
In summary. this shows that PAUs are stable, parsimonious and can improve the predictive perfor- mance of deep neural networks
4.2 LEARNED ACTIVATION FUNCTIONS ON MNIST AND FASHION-MNIST
When looking at the activation functions learned from the data, we can that the PAU family is flexible yet presents similarities to standard functions. In particular, Fig. 3 iillustrates that some of the learned activations seem to be smoothed versions of Leaky ReLUs, since V-shaped activations are simply Leaky ReLUs with negative a values. This is not surprising as we initialize PAUs with coefficients that match Leaky ReLUs. Finding different initialization and optimization parameters is left as future work. In contrast, when learning piecewise approximations of the same activations using Maxout, we would require a high k. This significantly increases the number of parameters of the network. This again provides more evidence in favor of PAUs being flexible and parsimonious liv). SLAF produced undefined values during training on all networks except FaSSIon-MNIST where LeNet finished 4 runs and VGG only one run.
4.3 EMPIRICAL RESULTS ON CIFAR-10
After investigating PAU on MNIST and Rushion-MNIST" we considered a more challenging setting:
CIFAR-10 (Krizhevsky et al. (2009). We also considered other leurnable activation functions, namely Maxout (k=2) and Mixture of activations (Id and ReI.U) as well as another popular deep network architectures: MobileNetV2 (Sandler et al. 2018), ResNet101 (He et al.) 2016) and 1enseNet121 (Huang et al. 2017). For more details on the learning settings and results, we refer to Sec. Sec.
Let us start by considering the results for VGG-8 and MobileNetV2 on CIFAR-IO. These networks are the smallest of this round of experiments and, therefore, could benefit more from bias reduction. Indeed, we
in Tab. 3 that both networks take advantage of learnable activation functions, ie., Maxout, PAU, and RPAU. As expected, adding more capacity to VGG-8 helps and this is what Maxout is doing. Moreover, even if Mixtures do not seem to provide significant benefit on VGG-8,

---

Published as a conference paper at ICLR 2020
Table 3: Performance comparison of activation functions on CIFAR-IO (the higher, the better) on four state-of-the -rrt deep neural architectures. Shown are the results averaged over 5 reruns well as the top result among these runs. The best ("*") and runner-up (**"") results per architecture are bold. As one can see, PAUs are either in the lead or close to the best. ("***") are experiments that did not finish on time.
VGG-8
MobileNetV2 mean = std
ResNet101 std
DenseNet121
std
best
best
mean
best
mean std
best
ReLU
92.32 + 0.16
92.58
91.51
0.28
91.82
95.07 + 0.17
95.36
095.6 + 0.18 o95.63
ReLU6
92.36 + 0.06
92.47
91.30 + 0.23
91.57
95.11 + 0.24
95.29
95.33 = 0.14
95.46
LReLU
92.43 0.14
92.65
91.94 0.12
92.08
95.08 + 0.19
95.29
95.42
0.17
95.65
RReLU
92.32 0.07
92.42
94.
0.16 994.94
95.21
c95.51
95.00 =
95.14
ELU
91.24 0.09
91.33
90.43 0.14
90.61
+4.04:0.144
94.24
90.78 = 0.29
91.23
CELU
91.24 0.09
91.33
90.69 = 0.27
90.97
93.80 0.36
94.25
90.88 = 0.19
91.08
PReLU
02.22
92.51
93.54 + 0.45
93.95
+.35++039
94.50
94.98 0.16
95.15
Swish
91.58 0.18
91.86
92.04 = 0.13
92.21
91.83
92.84
93.04 = 0.16
93.32
Maxout
93.03
0.11
933.3
94.41 0.10
94.54
95.11 0.13
95.23
S**
*8*
Mixture
91.86 1 0.14
92.06
94.06 1.16
94.25
94.50
0.25
94.71
93.33 = 0.17
93.59
APL
91.63 1 0.13
91.82
1.66
94.50
94.12 1 0.36
94.50
94.45 = 0.23
94.78
SReLU
92.66 0.27
c93.13
94.03
94.25
95.
24 1 0.13
95.38
94.77 0.24 0.24
95.20
PAU
92.51 0.16
92.70
94.57
0.21
94.90
95.16
0.13
95.28
95.03 = 0.07
95.16
RPAU
92.50 1 0.09
92.62
94.2
1 0.21
95.13
95.34
1 0.13
95..4
95.27 = 0.10
95.41
they do help in MobileNetV2. Here, we see again that PAU and RPAU are either in the lead or close the best when it comes to predictive performance, without having to make choice apriori.
Now, let us have a look at the performance of PAU and RPAU on the larger networks ResNer101 and DenseNet121. As these networks are so expressive, we do not expect the flexibility of the learnable activation functions to have big impact on the performance. Tab. 3 confirms this. Nevertheless, they are still competitive and their performance stable as shown by the standard deviation. On ResNet101, PAUS actually provided the top performance.
4.4 FINDING SPARSE PADÉ NETWORKS
As discussed Sec. 3.2 using PAUS in a network is equivalent to introducing virtual networks of ReLUs in the middle of the network, effectively adding virtual depth to the networks. Therefore, also investigated whether pruming can help one to unmask smaller sub-networks whose performance similar to the original network. In sense, replaced by the virtual network. Here, we only do pruning on the convolutional layers. For details about the algorithm and byper-parameters,
removing blocks of the real network as they get
refer to Sec. A.4.3
Specifically, we compared PAU against the best activation functions for the different architectures. However, we discarded Maxout, as instead of pruning it introduces many more parameters into
Layer
Layer 2
Layer 3
Layer
Layer 5
4.0
3.6
28
56
3.0
24
18
2.4
2.4
20
40
(z)
1.6
1.8
32
0.8
1.2
12
24
0.0
0.6
8
16
0.8
0.0
1.6
0.6
2.4
1.2
Figure 3: Estimated activation functions after training the VGG-8 network with RPAU on Fashion- MNIST. The center line indicates the PAU while the surrounding area indicates the space of the additive noise in RPAUs. As one can see, the PAU family differs from common activation functions but capture characteristies of them. (Best viewed in color)

---

Published as a conference paper at ICLR 2020
VGG8
MobileNetV2
ResNet101
52
96
888
Accuracy
85
PAU p-
Accuracy
PAU y- 0%
94
PAU p IN
82)
PAU a01s
50
PAU 21%
80
PAU p 009
88
PAU 6.%
B3
PAU p 90%
87
8Rd
y
LRuLU 50%
80
RRaLU 30%
924
RRaLU y 20%
p 60%
81
p- 60%
y 30%
100
2000
300
100
100
2010
300
200
300
100
Epoch
Epoch
Epoch
Figure 4: Comparison of the predictive accuracy (higher is better) for the architectures VGG-8. MobileNetV2 and ResNet101 between PAU and the best activation functions according to Tab. 3 PAU is consistently better. On ResNet101 PAU is not affected by the increase pruning pressure. Furthermore, PAU enables the ResNet101 subnetwork, pruned by 30%, to achieve a higher accuracy compared to all pruned and not pruned networks. (Best viewed in color)
the network defeating the original purpose. As one can see in Fig. pruning on the already size- optimized networks VGG-8 and MobileNetV2 has an effect on the predictive performance. However, the performance PAU remains above the other activation functions despite the increase in pruning pressure. In contrast, when we look at ResNet101, we see that the performance of PAU is not influenced by pruning, showing that indeed we can find sparse Padé network without major loss in accuracy. And what is more, PAU enables the ResNet101 subnetwork, pruned by 30%, to achieve higher accuracy compared to all pruned and not pruned networks.
4.5 EMPIRICAL RESULTS ON IMAGENET
Finally, we investigated the performance on much larger dataset, namely ImageNet (Russakovsky et al. (2015)) used to train Mob,leNetV2. As can be seen in Fig. 5land Tab. 4 PAU and Swish clearly
dominate performance V PAU leads in top-1 accuracy and Swish top-5 accuracy. Moreover, both PAU and Swish show Taster learning compared to the other activation functions.
Furthermore, we argue that the rapid learning rate of PAU in all the experiments indicate that they do not exhibit vanishing gradient issues 11
PAU
ELU
PReLU
ReLU6
SReLU U Swish
CELU
LReLU
ReLU
RReLU
70.50
2.00
69.00
1.90
MobileNetV2
Accuracy 1
67.50
1.80
66.00
1.70
[o65
64.50
1.60
63.00
1.50
61.50
1.40
60.00
1.30
58.50
1.20
0
50 Epoch
100
0
50
100
Epoch
Figure 5: MobileNetV2 top-1 test accuracy on the left (higher is better) and training loss on the right (lower is better) for multiple activation functions in ImageNet. PAU achieves higher accuracy and lower loss values in fewer epochs. (Best viewed in color)
8

---

Published as a conference paper at ICLR 2020
MobileNetV2 ReLU
ReLU6
LReLU
RReLU
ELU
CELU
PReLU
Swish
SReLU
PAU
Acc@1 1 69.65
69.83
70.03
69.12 19.12
69.13
69.17
68.61
o71.24
70.62
71.355
5cc@5 89.09
89.34
89.26
88.80
88.46
88.59
88.51
89.95
89.59
c89.85
Table 4: MobileNeetV2 top-1 and top-5 accuracies in ImageNet (higher is better) for different activations. Best ("*") and runner-up ("o") are shown in bold. PAU is the best in top-1 accuracy and runner-up for top-5.
4.6 SUMMARIZED RESULTS
Finally, we compare the PAU family of activation functions to all the other activation functions and aggregate the number of occurrences where PAU performed better worse in comparison. The aggregate results can be found in Tab. [5] These results are the aggregates of all the experiments on all datasets and all architectures. As we can see, the PAU family is very competitive.
Baselines PAU/RPAU >= Baseline
ReLU RRL. Seeel.//Swss//axaaaa
Mixture A....PSS...A
34
35
34
33
40
40
39
41
9
20
32
33
PAU/RPAU < Baseline
8
7
8
9
2
2
3
6
0
7
8
Table 5: The number of models on which PAU and RPAU outperforms or underperforms each baseline activation function we compared against in our experiments.
To summarize, PAUs satisfy all the assumptions (i-v). They allow the network to be universal function approximators (i) as shown by theorem [ They present fast and stable learning behavior (ii, iii) as shown in Figs. 8 The number of parameters introduced by PAUs is minimal in comparison to the size of the networks. In our experiments, we add 10 parameters per layer, showing that PAUs parsimonious (iv). Finally, they allow deep neural networks to provide high predictive performance (v) as shown in Tab.
CONCLUSIONS
We have presented novel learnable activation function, called Padé Activation Unit (PAU). PAUS encode activation functions rational functions, trainable in an end-to-end fashion using backpropa- gation. This makes it easy for practitioners to replace standard activation functions with PAU units in any neural network. The results of our empirical evaluation for image classification demonstrate that PAUs can indeed learn new activation functions and in turn novel neural networks that are competitive state-of-the-er networks with fixed and learned activation functions. Actually, across all activation functions and architectures, Padé networks are among the top performing networks. This clearly shows that the reliance first picking fixed, hand-engineered activation functions can be eliminated and that learning activation functions is actually beneficial and simple. Moreover, our results provide the first empirically evidence that the open question "Can rational functions be used to design algorithms for training neural networks?" raised by Telgarsky 2017 can be answered affirmatively for common deep architectures.
Our work provides several interesting avenues for future work. One should explore more the space between safe and unsafe PAUs, in order to gain even more predictive power. Most interestingly, since Padé networks can be reduced to ReLU networks. one should explore globally optimal training (Arora et al., 2018) as well as provable robustness (Croce et al. 2019) of Padé approximations of general deep networks.
PS and KK were supported by funds of the German Federal Ministry of Food and Agriculture (BMEL) based on decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support program, project "DePhenS" (FKZ 28 288204775..
Acknowledgments.
9

---

Published as a conference paper at ICLR 2020
REFERENCES
Agostinelli, M. D. Hoffman, P. Sadowski, and P. Baldi. Learning activation functions to improve deep neural networks. In Workshop Track Proceedings of the International Conference on Learning Representations, 2015.
R. Arora, A. Basu, P. Mianjy, and A. Mukherjee. Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations, 2018.
Barron. Continuously differentiable exponential linear units. arXiv preprint arXiv: 2017.
C. Brezinski and J. Van Iseghem. Padé approximations. Handbook of numerical analysis, 3:47-222, 1994.
Z. Chen, F. Chen, R. Lai, X. Zhang, and C.-T. Lu. Rational neural networks for approximating graph convolution operator jump discontinuities. In 2018 IEEE International Conference Data Mining (ICDM). IEEE, 2018.
D. Clevert T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In 4th International Conference Learning Representations (ICLR), 2016.
Croce, M. Andriushchenko, and M. Hein. Provable robustness of relu networks via maximization of linear regions. In The 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 2057-2066, 2019.
Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pages 315-323, 2011.
J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio. Maxout networks. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 1319-1327, 2013.
M. Goyal, R. Goyal, and B. Lall. Learning activation functions: A new paradigm of understanding neural networks. arXiv preprint arXiv:1906.099 09529, 2019.
Z. Hajduk. Hardware implementation of hyperbolic tangent and sigmoid activation functions. Bulletin of the Polish Academy of Sciences. Technical Sciences, 66(5), 2018.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
K. Hornik, M. B. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-3666 1989.
G. Huang, Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
X. Jin, C. Xu, Feng, Y. Wei, Xiong, and S. Yan. Deep learning with s-shaped rectified linear activation units. In Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, 2016.
P. Kidger and T. Lyons. Universal approximation with deep narrow networks. arXiv preprint arXiv: /505:089000833 2019.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. In 3rd International Conference Learning Representations, (ICLR), 2015.
10

---

Published as a conference paper at ICLR 2020
A. Krizhevsky and G. Hinton. Convolutional deep belief networks cifar- 10. 2010.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Computer Science Department, University of Toronto, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recognition. Proceedings of IEEE, (6(11):2278-233 1998.
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit datahase. at&t labs, 2010.
M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-867, 1993.
A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
Manessi and A. Rozza. Learning combinations of activation functions. In 2018 24th International Conference Pultern Recognition (ICPR), pages 61-66. IEEE, 2018.
V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Pro- ceedings of the 27th international conference on machine learning (ICML-10), pages 807-814, 2010.
Nickolls, I. Buck, and M. Garland. Scalable parallel programming. 2008 IEEE Hot Chips 20 Symposium (HCS), pages 40-53. IEEE, 2008.
N. Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1): 145-151, 1999.
P. Ramachandran, B. Zoph, and Q Le. Searching for activation functions. In Proceedings of the Workshop Track of the 6th International Conference on Learning Representations (ICLR), 2018.
O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li. Imagenet large scale visual recognition challenge. Interna- tional Journal of Computer Vision, 115(3):2112255 2015.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4510-4520 2018
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Reprexentutions((CC.. 2015.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Y. W. Teh and G. E. Hinton. Rate-coded restricted boltzmann machines for face recognition. In Prcoeedings of Neural Information Processing Systems (NIPS), pages 908-914, 2001.
M. Telgarsky. Neural networks and rational functions. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 3387-3393 2017.
L. N. Trefethen. Approximation Theory and Approximation Practice. SIAM, 2012. ISBN 978-1-611- 97239-9.
C. Vercellino and W. Y. Wang. Hyperactivations for activation function exploration. 31st Conference on Neural Information Processing Systems (NIPS 2017), Workshop on Meta-learning. Long Beach, USA, 2017.
H. Xiao, K. Rasul, and R. Vollgraf. feshhon-mmiste a novel image dataset for benchmarking machine learning algorithms. CoRR, 2017.
11

---

Published as a conference paper at ICLR 2020
B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolutional network. CoRR, 2015.
H.-Z Zhao, F.-X. Liu, and L.-Y. Li. Improving deep convolutional neural networks with mixed maxout units. PloS one, 12(7), 2017.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 INITIALIZATION CORFFICIENTS
As show in Table 6 we compute initial coefficients for PAU approximations to different known activation functions. We predefined the orders to be [5,4] and for Sigmoid, Tanh and Swish, we have computed the Padé approximant using the standard techniques. For the different variants of PRelu, LeakyRelu and Relu we optimized the coefficients using least squares over the line range between [-3,3] in steps 0.000001.
Sigmoid
Tanh
Swish
ReL.U
(..0.U(0001)
(ReLUU0.20)
LReLUU0.30) (0.25
LReLU(-0.5)
40
1/2
0
0.029963448
0.02979246
0.02557776
D,02423485
0.02282166
0,02650444
61
1/4
1/2
0.61690165
0.61837738
0.66182815
0.67709718
0.69358438
0.807729 12
02
1/18
0
6/1
2.37539147
2.32335207
L581829755
1,38583633
1.30847332
13.56611639
1/144
1/9
36 /56
3.06608078
3,05202660
2,94478759
2.95497990
2,97681599 599
7.00217900
444
1/2016
6//168
1.52474449
148548002
0,95287794
0.85679722
0,77165297
4776187781
05
1/60-180
1/915
/3360
0,25281987
0,25103717
0,23319681
0,23229612
0,23252265
0.68720375
51
0
0
1.19160814
1.14201226
0,50962605
0.41014746
0,32849543
13.70648993
52
1/9
1/9
/28
11798
4.39522834
4.13376800
4.14691964
4.11557902
6.07781733
0
0.87154450 0.34885983
0.91 1 1034
0,37832090
0.30292546
0,24155603
12.32535229
b4
1/10008
1/63
6f/1680
0.34720652
0.32407314
0.32002850
0.31659365
0,54006880
Table 6: Initial coefficients to approximate different activation functions.
A.2 LIST OF ACTIVATION FUNCTIONS
For our experiments, we compare against the following activation functions with their respective parameters.
ReLU (Nair and Hinton 2010): y max (zx, 0)
ReLU6 KKrizhessky and Hinton 2010): y min (max (z, (z,, 0), 6), variation of ReLU with an upper bound.
Leaky ReLU (Maas et al.
2013): ): = (0, ) + a * min(0, x) with the negative slope,
which is defined by the parameter Leaky ReLU enables small amount of information to flow when x < O.
Random ReLU (Xu al. 2015): randomized variation Leaky ReLU.
ELU (Clevert et al.
2016 y www.maxx0.... (0, .r) + min(O, a (exp(x) - 1)).
CELU Barron
2017): y max (0, 2) + min (exp(z/a) 1)).
Swish Raaachandran et al. 2018
* sigmoid(z) which tends to work better than ReLU on deeper models across a number of challenging datasets.
Parametric ReLU (PReLU) (He al. 2015) max(0, mi)(0, where the
leaky parameter is a learn-able parameter of the network.
Maxout (Goodfellow et al. 2013); y = max(2z)), where 2ig = x"W + bi and W Rdxmxk and b E fem are learned parameters.
Mixture of activations (Manessi and Rozza 2018): a combination of weighted activation functions e.g. fid, ReLU), where the weight is learnable parameter of the network.
SLAF (Goyal et al.
2019): a learnable activation function based on a Taylor approximation.
APL (Agostinelli al.
2015): learnable piecewise linear activation function.
SReLU (Jin al.
2016); learnable S-shaped rectified linear activation function.
A.3 DETAILS OF THE MNIST AND FASHION-MNIST EXPERIMENT
A.3.1 NETWORK ARCHITECTURES
Here describe the architectures for the networks VGG and LeNet, along with the number of trainable parameters. The number of parameters of the activation function is reported for using PAU. Common not trainable activation functions do not have trainable parameters. PReLU has one
13

---

Published as a conference paper at ICLR 2020
trainable parameter. In total the VGG network 9224508 parameters with 50 for PAU, and the LeNet network has 61746 parameters with 40 for PAU.
No.
VGG
params
LeNet
# params
Convolutional 3x3x64
Convolutional
640
1.56
5x5x6
2
Activation
10
Activation
10
Max-Pooling
Max-Pooling
0
Convolutional
Convolutional
4
73856
2416
3x3x128
5x5x16
Activation
10
Activation
10
Max-Pooling
Max-Pooling
0
Convolutional
Convolutional
295168
48120
3x3x256
5x5x120
Convolutional
590080
Activation
10
3x3x256
Linear
Activation
10
10164
84
10
Max-Pooling
Activation
10
Convolutional 1180160
Linear 10
11
850
3x3x512
Convolutional 3x3x512 2359808
12
Softmax
0
13
Activation
10
14
Max-Pooling
Convolutional 3x3x512
15
2359808
Convolutional 3x3x512 2359808
16
17
Activation
10
18
Max-Pooling
Linear
19
5130
10
20
Softmax
Table 7: Architecture of Simple Convolutional Neural Networks
A.3.2 LEARNING PARAMETERS
The parameters of the networks, both the layer weights and the coefficients of the PAUs, were trained 100 epochs using Adam (Kingma and Ba 2015) with learning rate of 0.002 or SGD (Qian) 1999) with learning rate 0.01, momentum set to 0.5, and without weight decay. In all experiments used batch size of 256 samples. The weights of the networks were initialized randomly and the coefficients of the PAUs were initialized with the initialization constants of Leaky RELU, see Tab. We report the mean of different runs for both the accuracy on the test-set and the loss on the train-set after each training epoch.
14

---

Published as a conference paper at ICLR 2020
A.3.3
PREDICTIVE PERFORMANCE
MNIST
RPAU
SReLU
ReLU
ReLU6
APL
LRALU
Swish
PAU
PReLU
RRaLU
ELU
CELU
09.25
-0.00036
99.00
-0,00132
95.
-0.00028
VGG
Accuracy
96.50
0.00024
08. 25
-0.00020
-0.00016
97.75
-0.00012
97.
0.00008
97.25 25
0,00004
99.36
0.00018
99.28
0.00016
99.20
0.00014
LeNet
Accuracy
12
0.00012
99.04
-0,00010
08.96
-0.001088
08.88
5.00000
0.00004
08.80
0,00002
98.72
40
00
80
100 20
60
80
100
Figure 6: PAU compared to baseline activation function units 5 runs of MNIST using the VGG and LeNet: first column mean test-accuracy, second column mean train-loss. PAU consistently outperforms matches the best performances of the baseline activations. Moreover, PAUs enable the networks to achieve lower loss during training compared to all baselines.
Fashion-MNIS:
RPAU
SReLU
ReLU
SLAF
ELU
CELU
Swish
PAU
PReLU
RReLU
ReLU6
APL
LReLU
92.00
10.00225
91.00
H0.00200
90.00
H0.00175
VGG
Accuracy
89.00
40.00150
88. 00
H0.00125
87.00
H0.00100
86.00
20.00075
0.00050
$4.00
0.00125
90.50
0.001.0
00.00
0.00133
89.50
LeNet
Accuracy
89.00
0.00120
0.00105
88.50
0.00090
88.00
0.00075
87.50
87.00
0.00060
0.00045
20
40
60
80
100
20
60
100
Figure 7: PAU compared to baseline activation function units 5 runs of Mashion-MNIST using the VGG and LeNet architectures: first column mean test- acst-acuuray,, second column mean train-loss, PAU consistently outperforms the basclines activation functions in terms of performance and training time, especially on the VGG.
15

---

Published as a conference paper at ICLR 2020
A.4 DETAILS OF THE CIFARIO AND IMAGENET EXPERIMENT
A.4.1 LEARNING PARAMETERS
The parameters of the networks, both the layer weights and the coefficients of the PAUs, were trained over 400 epochs using SGD with momentum set 0.9. On the Cifar10 dataset we have different optimizer setups for PAU layers and the rest of the network. For the PAU layers we use constant learning rates per networks and no weight decay. For updating the rest of the network we use initial learning rate of 0.1, and learning rate decay of 0.985 per epoch and set weight decay to 5e 4. In all experiments we used a batch size of 64 samples. The weights of the networks were initialized randomly and the coefficients of the PAUs were initialized with the initialization constants of Leaky ReLUU, see Tab. A. The additive noise of the Randomized PAUs is set to a = 1% training the networks VGG8 and MobileNetV2, respectively
10% for ResNet101.
On the Imagenet dataset we use the same optimizer for PAU and the rest of the network. We follow the default setup provided by Pytorch and use an initial learning rate of 0.1, and decay the learning rate by 10% after 30, 60 and 90 epochs.
A.4.2 NETWORK ARCHITECTURES
The network architectures were taken from reference implementations in PyTorch and we modified them to PAUs. All architectures are the same among the different activation functions except for Maxout. The default amount of trainable parameters of VGG8 (Simonyan and Zisserman 2015) is 3,918,858. Using PAU 50 additional parameters are introduced. Maxout is extending the VGG8 network to a total number of 7,829,666 ,,622 parameters. MobileNetV2 (Sandler et al., (018)) contains by default 2,22 trainable parameters. PAU adds 360 additional parameters. The Maxout activation function is results in a total number of 3,524,506 parameters. With respect to the number of parameters ResNet101 (He al., 2016) the largest network we train. By default contains 42,512,970 trainable parameters, we introduce 100 PAUS and therefore add 1000 additional parameters to the network. one is replacing each activation function using Maxout the resulting ResNet101 network contains 75,454,090 trainable parameters. The default DenseNet12 121 (Huang) ct al. 2017) network has 6,956,298 298 parameters, Replacing the activation functions with PAÙ adds 1200 parameters to the network.
A.4.3 PRUNING EXPERIMENT
For the pruning experiment, we implement the "Lottery ticket hypothesis" (Frankle and Carbin (2019)) PyTorch. We compare PAUs against the best activation for the network architecture according to the average predictive accuracy from Tab. 3 More precisely, we compare the predictive performance under pruning for the networks N1 = TVGG-8pau MobilceNttc ResNet101, pau against the networks N₂ {VGG-& &LRLU, MobileNet ResNet101, p. Here we avoided Maxout as heavily increases the parameters in the model, defeating the purpose of pruning. Unlike the original paper,
compress the convolutions using fixed pruning parameter per iteration of p% 10, 20, 30, 40, 50, 60 and evaluated once per network. After each training iteration we remove p% of filters in every convolution and the filters remove are the ones where the sum of its weights is lowest. After pruning, we proceed to re- re-initialize the network and repeat the training and pruning proceedure with the next 5% parameter.
16

---

Published as a conference paper at ICLR 2020
A.4.4
PREDICTIVE PERFORMANCE CIFARI0
RRaLU
Swish
93.00
91.50
90
80
90.00
70
88.50
VGG8
Accuracy
60
$7.00
50
85.50
40
84.00
30
82.50
20
$1.00
10
95.00
180
92.50
160
MobileNetV2
90.00
140
Accuracy
87.50
120
85.00
100
90
82.50
90
40
77.50
20
94.50
72
03.00
64
56
ResNet101
91.50
Accuracy
48
90.00
40
88.50
32
87.00
24
85.50
16
$4.00
94.50
72
03.00
64
DenseNet121
01.50
56
Accuracy
90.00
48
40
88.50
32
87.00
24
85.50
16
84.00
82.50
400
100
200
300
Epoch
Epoch
Figure 8: PAU compared to baseline activation function units on 5 runs of CIFAR-10. Accuracy the left column and loss on the right one. (Best viewed in color)
17


Reviews:
[{"review_id": "SkxMQumTKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce an activation function based on learnable Padé approximations. The numerator and denominator of the learnable activation function are polynomials of m and n, respectively. The authors name them Padé activation units (PAUs). The authors also propose a randomized a version of these functions that add noise to the coefficients of the polynomials in order to regularize the network. The authors show, at best, marginal improvements over a variety of baselines including MNIST, fashion MNIST, CIFAR10, and Imagenet. The authors also show that pruning neurons with PAU units results in slightly better accuracy that pruning neurons with ReLU units.\n\nThe improvements over baselines shown were marginal and I do not think they warrant publication at this conference. The accuracy improvements were no more impressive than other learned activation functions which the authors perhaps did not see, such as SReLUs (Deep Learning with S-Shaped Rectified Linear Activation Units) and APLs (Learning Activation Functions to Improve Deep Neural Networks).\n\n** After author response **\nChanging from reject to weak accept\nThe authors have included new experiments that compare to a wider range of learned activation functions. While not ground breaking, it shows that it is competitive with state-of-the-art learned activation functions and could have something to offer.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgoV1AnFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper introduces a novel parametric activation function, called the Pade Activation Unit (PAU), for use in general deep neural networks. Pade is a rational function, which is a ratio of two polynomials, and which can very well approximate any of the usually used activation functions while having only a few parameters that can be learned from data. Moreover, the authors identify five properties that an activation function should have, and either prove or empirically show that PAUs satisfy all of them, unlike some of the baselines. Additionally, since Pade approximation can have poles and be unstable, this work introduces safe PAUs, where the polynomial in the denominator is constrained to attain values greater than or equal to one. Since one of the suggested properties is that a function using a given activation function be a universal function approximator, the authors provide a sketch of a proof that PAUs do allow that. This proof applies only to the unsafe version of the PAU, and it is unclear whether it extends to the safe PAU---an issue that is not mentioned by the authors.\nFurthermore, the authors propose a stochastic version of PAU with noise injected into parameters, which allows regularization. The empirical evaluation is quite extensive, and the PAU is compared against nine baselines on five different architectures (LeNet, VGG, DenseNet, ResNet, MobileNet) on four different datasets (MNIST, Fashion MNIST, CIfar10, ImageNet) for the classification task. The evaluation confirms that PAUs can match the performance of or sometimes outperform even the best baselines while the attained learning curves show that PAUs also lead to faster convergence of trained models. Finally, the authors demonstrate that (and provide intuition why) using PAUs allow for high-performing pruned models.\n\nI recommend ACCEPTing this paper as it is well written, extensively evaluated, and provides performance improvements or at least matches the performance of the best baseline across several datasets and model architectures.\n\nMy only two suggestions for improvement are a) make the universal approximation proof tighter by making sure that it extends to the safe PAU version, and b) evaluate the proposed activation function on tasks other than just classification.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgsBTOLKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an activation function that contain parameters to be learned through training. The idea is to give the learning algorithm more \"freedom\" to choose a good activation function, and hopefully better performance can be achieved.\n\nThe paper is well written, and the experiment results look reasonable. However, there are several key issues.\n\n1) as the authors stated, a \"good\" activation function should maintain the universal approximation property of the neural network. This seems not discussed for the PADE activation function.  Does (1) satisfy the conditions (i)-(v) listed in table I? Is there a rigorous proof? Table I seems to claim that the PADE based neural network satisfies (i), but there is no formal proof.\n\n2)  In order to avoid poles, the activation function used in this work is (2). How well can (2) approximate (1)? What is the potential loss? Perhaps there should be more discussion on this - preferably some theoretical supports.\n\nOverall, the reviewer feels that this paper starts with an interesting idea, but the developments on the theoretical side is a bit thin.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new learnable activation function called Padé Activation Unit (PAU) based on parameterization of rational function. All the reviewers agree that the method is soundly motivated, the empirical results are strong to suggest that this would be a good addition to the literature. 
Decision: Accept (Poster)