COMBINED FLEXIBLE ACTIVATION FUNCTIONS FOR DEEP NEURAL NETWORKS
Authors: Renlong Jie
Junbin Gao
Andrey Vasnev
Minh-Ngoc Tran
Venue:  (2020)
Keywords: Flexible
Activation Functions
Deep Learning
Regularization

Abstract: Activation in deep neural networks is fundamental to achieving non-linear mappings. Traditional studies mainly focus on finding fixed activations for a particular set of learning tasks or model architectures. The research on flexible activation is quite limited in both designing philosophy and application scenarios. In this study, we propose a general combined form of flexible activation functions as well as three principles of choosing flexible activation component. Based on this, we develop two novel flexible activation functions that can be implemented in LSTM cells and auto-encoder layers. Also two new regularisation terms based on assumptions as prior knowledge are proposed. We find that LSTM and auto-encoder models with proposed flexible activations provides significant improvements on time series forecasting and image compressing tasks, while layer-wise regularization can improve the performance of CNN (LeNet-5) models with RPeLu activation in image classification tasks.

Under review as a conference paper at ICLR 2020
COMBINED FLEXIBLE ACTIVATION FUNCTIONS FOR DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Activation in deep neural networks is fundamental achieving non-linear map- pings. Traditional studies mainly focus on finding fixed activations for a particular set of learning tasks or model architectures. The research on flexible activation is quite limited in both designing philosophy and application scenarios. In this study, we propose a general combined form of flexible activation functions as well as three principles of choosing flexible activation component. Based on this, we develop two novel flexible activation functions that can implemented LSTM cells and auto-encoder layers. Also two new regularisation terms based on assumptions as prior knowledge are proposed. We find that LSTM and fully connected auto-encoder models with proposed flexible activations provides sig- nificant improvements on time series forecasting and image compressing tasks, while layer-wise regularization improve the performance of CNN (LeNet-5) models with PReLu activation in image classification tasks.
INTRODUCTION
Deep learning is probably the most powerful technique in modern artificial intelligence (LeCun al. 201 One reason is its ability in approximating complex functions with large but limited number of parameters (Cybenko 1989; Hornik 1991), while the regular layer structures make it possible to be trained with efficient back propagation algorithms (Goodfellow al., 2016).
In deep neural network, the weights and bias take account of linear transformation of the data flow, while the activation functions bring in non-linearity. is remarked in (Hornik) 199T) that activation functions do not perform equally well if we take minimal redundancy or computational efficiency into account. Thus the selection of activation function for different tasks is an issue with importance. Traditionally, people train the weights of linear transformations between layers while keeping the activation functions fixed, and usually one identical activation function is used for all the neurons on each single layer. For example, rectifier linear units (Relu) are used the default choice for the activation in hidden units for feed forward units and a large proportion of convolutional neural networks (Nair and Hinton, 2010), while sigmoid and tanh functions used where output values are bounded, such as in output layers for classification problems and the gate activations in recurrent cells (Gers al. 1999 Chung al. 2014).
The drawback of Relu activation is the issue of dead unit when the input is negative, which makes people introduce functions with non-zero values in the negative range, including leaky-Relu and Elu (Maas et al. 2013 Clevert et al. 2015). On the other hand, explosion or vanishing gradients in back propagation are also issues that harm the performance of model largely due the shape of activations (Bengio et al. 1994 Hochreiter) 1998, Pascanu et al. 2012), while techniques such as clipping and batch normalization can be implemented to alleviate these issues to some extent (Toffe) and Szegedy 2015 Lin et al.; 2017).
With a large enough neural network and sufficient training time, the model can effectively learn the patterns from data with possible high accuracy. however it is not straightforward to confirm that learning process is the most efficient and the results are the most accurate. One possible solution for accelerating model training is introduce flexible trainable activation functions (Agostinellii jet al., 2014) He et al., 2015] Chung et al.) (0166). Even though this requires higher computing and storing cost that is proportional to the number of neurons, the performance of non-linear activation

---

Under review as a conference paper at ICLR 2020
function can be largely improved, which could be more efficient than increasing the number of basic model parameters or the number of neurons.
There are existing works trying to promote the predictive performance of deep neural networks based on trainable activation functions. As the Leaky Relu function has a hyper-parameter to be optimized, which is the slope of its negative part, parameterized Relu (PRelu) was proposed to make this slope adapt to the data within specific neurons and be learned during the training process (He) al. 2015). Meanwhile, another study proposes the parameterized version of Elu activation, which introduces two parameters to control the shape of exponential curve in the negative region (Li et al., 2018).
It can also be a blending of different commonly used activations, where the trainable parameters are the weights for the combination components (Sütfeld al. 2018 Manessi and Rozza 2018). Since different activation functions can have very similar behavior in some specific regions, more generative way is to consider their Taylor expansions at O point and use weighted combination of polynomial functions with different orders instead (Chung et al. 2016). For containing those func- tions that are not centered at 0, one choice is to train a piece-wise function adaptively (Agostinellii et al. 2014). The similar effect can be achieved by Maxout activation, which is quite helpful in promoting the efficiency of models with dropout (Goodfellow et al. 2013). Beyond that, there are also studies on making the most of the non- lineir properties by introducing adaptation mechanism on the Softmax layers (Flennerhag et al. 2018), which achieve the former state-of-the-ane results on several natural language processing (NLP) tasks.
The limitation of existing studies can be illustrated as follows. First, most of existing work focus on some specific forms of parameterized activation functions rather than a more general form. Second, there is a lack of study on flexible versions of bounded activations such as sigmoid and tanh. Third, the experiments of existing work are mainly on convolutional networks rather than other types of architectures such as recurrent networks and auto-encoder networks. In this study, we consider the activation function as combination of a set of functions following the constraints of several principles. Based on these principles, we develop two novel trainable activation functions that can be introduced to LSTT cells and auto-encoder architecture with significant performance improvement. In addition, layer-wise regularization on activation parameters is introduced to reduce the variance caused by activation functions. Correspondingly, we use three experiments to show the goodness of two novel activation functions and the effect of layer-wiss wise regularization on PRelu activation.
METHODOLOGY
2.1 PARAMETERIZED ACTIVATION FUNCTIONS
In this study, we introduce general form of parameterized activation functions linearly combined from different types of activation functions or a single activation function with multiple parame- ters. We assume that the parameters in the combined activation functions can be different for each neuron, which can be trained during the main training process of the model parameters with back propagation.
K where i indexes the neuron, and 2 Σ A=1
K Σ
0z 3))
aak (2, Bak),
o
(I)
WiXi-1 by is the input of the activation layer in- dexed by This means that, each neuron i, possible to have its own set of parameters a² = [aa1.... ..., )ax)² and g = [Gn,..., Bix] where Oa is the combination weights and Gi is the activation parameter vector for the k--th component activation fk, respectively. Thus Eq. equa- tion I defines form of activation function
linear combination of a set of basic parameterized non-linear activation functions fz, 3x) with the same input x to the neuron. Normally, we have 0 04,k < for all l and i. This setting will take advantage of the low computational costs of existing activation functions, while it will be much easier to implement weights normalization when need bounded activation function.
Since the specific activation function corresponding to each neuron only depends on its own activa- tion parameters, the back propagation of these activation parameters by stochastic gradient descent
2

---

Under review as a conference paper at ICLR 2020
can be done as follows:
aL
0L
do:
0L
aik 7 Olik
dack
Oik
do:
Joik
aik ~7
do;
f2
(2)
ÖT.
ØL.
do, Bik -
ØT.
afac2,
3 + Buk
Bsk
aak aß5²
8o:
do;
where i is the index of the hidden neuron with output o; and i is the index of combined flexible tivation functions. Here we use a simplified expression that does not include the indices of layer and training examples in each mini-batch. 7 is the learning rate of gradient descent for all the parameters in activation functions. With the gradients given by 0L/3agg- adaptive optimizers such as AdaGrad (Duchi al. 2011). Adam (Kingma and Ba 2014) and RMSProp (Tieleman and Hinton 2017) can also be applied. In general, gradient descent approach and its derivatives can push the activation parameters toward the direction that minimizes the empirical risk of the model on training data. In practice, considering the different nature of basic model parameters such weights and biases and activation function parameters, it could be more appropriate to implement different learning rates for each of them. However, this will increase the load of hhyer-ppaametee searching.
To build effective combinations with the general form given by Eq. equation following three principles for selecting the components:
we introduce the
Principle 1: Each component should have the same domain as the baseline activation function.
Principle 2: Each component should have equal range the baseline activation function.
Principle 3: Each component activation functions should be expressively independent of other component functions with the following definition.
Definition 1: If component activation function S& is expressively independent of a set of other component functions: f1, fi...,,n there does not exist set of combination coefficients a inner activation parameters $1....... parameters of the previous linear layers W', such that for any input X, activation parameters Bk, and parameters of the previous linear layer Wk. by. the following equation holds:
Sa (2k, Ba) S&(W (k)
3a)= Σ a.J. + +b
Σ as
(3)
For single-layer network with neurons, if component activation function fk. which is not expressively independent of other components, is excluded, we need at most 2m neurons to express the same mapping.
Proposition 1:
The first two principles are aiming at keeping the same ranges and domains of the information flow with the mapping in each layer. The third principle is aiming reducing the redundant parameters that do not contribute to the model expressiveness even with limited number of units. A short proof of Theorem provided in Appendix A.I For example, (2) 1/(1 (1+6 + - is expressively a X). Therefore, the combined activation a(2, B) (1) + (1 ag (33z) will not be good choice. Based on this, we can then design the combined trainable activation functions for both bounded or unbounded domains.
independent with (=) = 1/(1 + - since when W = 3W, we have (W'')
2.2 SIGMOID/TANH FUNCTION EXTENSION FOR RNNS
Sigmoid and Tanh activation functions are widely used in recurrent neural networks, including basic recurrent nets and recurrent nets with cell structure such as LSTMs and GRUs (Gers et al. 1999 Jozetowicz et al. 2015 Goodfellow et al. 2016). For example, an LSTM cell has the functional
mapping as follows:
ft W f. + Wi by)
4 = o W xt + With- + bi)
0, a (W a, Wor ho bo)
(4)
9: tanh W W 0; + W Wihht 1 bby)
C: f. C.-1 * gt
h = o, tanh(c:)
3

---

Under review as a conference paper at ICLR 2020
The cell structure includes multiple sigmoid and tanh activation functions, which can be replaced by weighted flexible combination between the original one and another activation function with the same domain. For the sigmoid function, the output should be in the domain of [0, 1/, while for tanh the output should be in [-1, 1). In the first case, one simple choice is:
olz; o (2) + a)- f(;
(5)
where 0 S a S and
if < 23
(=;
B2 +
if 1 S 2 23
2
if 2
2
In Eq. equation 5 [(; 9) can be considered as combination of two Ramp functions bounded between and with parameter b. The shapes of a sample of combined activation with Eq. build function with the same boundary tanh
shown in Appendix A.3 Similarly, function, and use the corresponding combination to replace tanh in the LSTT cell. Consequently. for each combined flexible activation function, there are two parameters be optimized during the training process. By combining the original activation function with another function f(z; 3) with the same boundary using normalized weights, the model can be trained with flexible gates and have the potential to achieve better generalization performance.
2.3 RELU FUNCTION EXTENSION FOR MLPs AND CNNS
The outputs of ReLu function is unbounded on positive side, while the derivative with respect to the inputs is Hcaviside step function. To build more flexible activation in the condition when ReLu function is used, we can make a weighted combination between Relu and other non-linear functions with unbounded ends. In the simplest case, ReLu function, 28 and 2//3, which can be written as:
make weighted linear combination between
o02; a1 a2) = a ReLu(z) + a22 (1 a1 - 22)
where and B are two parameters to be learned with back propagation. By merging basic Relu tivation and two functions that are expressively independent with other components, such as = and 21/3, the model could have the potential to learn non-convexity with much less hidden units. In addi- tion, for very deep networks, the combination of ReLu-like function and smooth non-Ro non-ReLu function could facilitate the information propagation in considering the Edge of Chaos (EOC) (Hayou et al., 2019)
2.4 LAYER- WAYER-WISE REGULARISATIN FOR ACTIVATION PARAMETERS
Similar to the weights decay regularisation for model weights in NN models, we introduce regular- isation terms for parameters in activation functions to avoid over-parameterizztion during learning process. When we set the summation of each component's 's weights in each flexible activation func- tion 1, is not suitable to implement a LI or L2 norm on the absolute value of activation weights. Instead, we use the L2 norm for the absolute difference between each specific activation parameter and the mean of corresponding parameters in the same layer. In addition, we introduce another L2 regularisation term controlling the difference between the trained parameters and the initial param- eters of benchmark activation function. This can make sure that the benchmark actually a specific case of flexible activation, while the variations be learned to adapt the training dataset and controlled by these regularisation effects. Thus, the cost function can be written as follows:
Cost function =Predictive Loss + 8, Σ 2m,
ΣΣ
lassa azHll
(8)
+ +2 ΣΣΣ ||oso - aijkl + other terms
where Aijk refers to the kth activation parameter 0g for ith element in jth layer, āzk is the average value of ag in jth layer, ago is the combination coefficient of kth component in basic standard ac- tivation functions (c.g. ReLu), m, is the number of neurons in jth layer, while XÚ is the layer-wist wise regularisation coefficients, and mutual coefficient. We consider these two regularization
4

---

Under review as a conference paper at ICLR 2020
terms as prioris. For the first one, since in the layer structure of deep neural networks, usually dif- ferent layer is learning different level of patterns, which could be in favor of using similar activation functions in each layer. Mcanwhile, the second regularization terms can be considered another priori in assuming that the initial activation functions are good enough and the learned activation parameters should not differ too much from the initial values.
3
EXPERIMENTS
All the experiments were conducted in the environment of Pytorch 1.3.1, we implemented the em- bedded functions of sigmoid, relu and prelu in the baseline models and manually created the pro- posed flexible functions with backward path in the flexible models. The first and second experiments are conducted with a cloud Intel Xeon 8-Core CPU, the third experiment conducted with a cloud Tesla K80 GPU.
3.1 EXPERIMENT WITH RECURRENT NEURAL NETWORKS
For testing the performance of the model with flexible activation in recurrent neural networks, we build multiple-layer LSTM model. We change the three sigmoid functions in Eq. equation 4 to the parameterized combined function as shown in Eq. equation then compare the model performances the cases with or without flexible activations.
The dataset being experimented on is a combination of daily stock returns of five G20 countries including Brazil, Canada, India, China and Japan from 02 Jan, 2009, which is multi-variate time series data. The five returns of each day can be considered as an input vector to the corresponding hidden unit, while the output one-step ahead forecast given a sequence of historical data. Instead of using random sampling, we directly split the set of sequences with 10 lagging vectors into training set (64%), validation set (16%) and lest set (20%), while the learning curve on validation set can be obtained during the training. The loss is selected as the average of mean squared errors of 5 forecasted values with respect to the true values for each example. For the hyper-parameter setting, the batch size was set to be 50, the window size is 10 time steps, the number of epochs is 30, while the optimizer implemented in training is Adam optimizer with the same learning rate on both weights, bias and activation parameters. The initialization of the flexible activation parameters in replacing sigmoid function is a = 1 and 3 = 0.1, which means that we train them from baseline settings. Four stacked LSTM models with different layer configurations are implemented, then we compare the validation and test performances of these models with fixed and flexible activations by 100 trials with different random initializations for each of them.
We introduce the regularizer proposed in Section 2nd tune the corresponding weight decay coef- ficient 8 for flexible activation parameters. First, for each configuration of layer size, we search for the optimal values of learning rates from 1000 random samples in the range of [0.001, 0.2) with log- arithm scale when the fixed activation functions are used. Based on these optimized learning rates for fixed models, we further search for the optimal regularization coefficients for flexible activation functions from the logarithm sale of range [0.001, 0.1] with 30 random samples. The optimal values of the learning rates and the regularization coefficients for activation parameters are listed in Table Here we do provide benefit to fixed models by using their optimal learning rates in the corresponding
Table Summary of haper--aramene parameter settings for LSTMs the experiment
Model
Layer size
Params
Learning rates
Regularization (act)
LSTM-1
[5. 16]
1408
6.71E-3
2.50E-1
LSTM-2
[5, 8,4,4)
800
3.47B-3
2.50E-1
LSTM-3
[5. 16, 8]
2208
1.93E-2
2.50E-1
LSTM-4
[5.
3520
4.32E-3
2.50E-1
flexible models. Meanwhile, to avoid too much extra computational cost paid flexible models, use the optimized activation regularization coefficients of LSTM-1 to all the models. Moreover, we use different configurations of layer size compare models with different capacity some level.
As is shown in Figure [ the learning curves of model with flexible activation functions (denoted "flexible models") validation set lie below the corresponding curves of models with fixed
5

---

Under review as a conference paper at ICLR 2020
fixed
0.0032025
2.000215
flexible_ reg
MSE
0,0031975-
5.003210
0.0001550-
5. 5.002005
0,0001925
2.00:300
0.0021875
5.003395
5
10
15
20
25
30
10
15
20
25
30
(a)
16), window
(b) Time
8, window
0.000256
5.002002
+ fixed
0.000224
+ flexible
flexible_rey reg
Valisation MSE
0.000202
0.30250
3.000198
0.000150
3.000195
0.000160
0.001154
3.000392
0.001152
0.001
10
30
15
25
(c) Time Step [5, 16, 81. window size 10
[d) Time Step (5, 16, 16), window size 10
Figure I: Comparison between the average learning curves (with error bars) of LSTM models with and without regularized flexible activation functions on Multi stock indices return data in forecasting multi-variate return. (a) Two-layer LSTM model with layer sizes: [5, 16, 8); (b) Three-layer LSTM model with layer sizes: [5, 8, 4]; (c) One-layer LSTM model with layer sizes: [5, 16); (d) Two- layer LSTM model with layer sizes: [5, 16, 16).
activation functions (denoted as "fixed models") during most of learning time. Still we consider the average of minimum validation loss during the learning process in each configuration. The descriptive statistical analysis of 100 trials in each setting is shown Table 2
Table 2: Summary table of stock indices forecasting with Stacked LSTM
Model
Data
Fixed
Flexible
Flexible (Regularized)
validation
1.747E-4 (1.7E-7)
1.747E-4 (I.SE-7)
1.746F-4 (1.4F-7)
LSTM-1
test
7,954E-5 (B-7))
7.872E-5 (E-1E-)
7.724E-5 (4.8E-7)
validation
1.823F:44 (1.9F-7)
1.804F.44
(2.8E-7)
1.809E-4 (3.1
LSTM-2
test
8.092E-5 (7.7B-7)
7.811E-5 (5.8E-7)
7.841B-5 (5.3E-7)
validation
(2.5E-)))
1.771E-4 (2.3E-7)
1.770E-4 (2.2E-7)
LSTM-3
test
8.046F-5 (9.3B-7)
7.939F-5 (6.2E-7)
7.794F-5 (5.1E-7)
validation
(2.1177)
1.756E-4 (2.51-7)
(7.77-4(1.9E.
LSTM-4
test
7.925F-5 (7.0R-7)
7.920F-5 (6.9F-7)
7.919F-5 (5.9E-7)
We can see that in all the configurations layer size, flexible models outperforms fixed models in terms of minimum validation error and the test error Especially in the best-performed case with layer size of [5. 16), which is neither the largest or the smallest in terms of the number parameters, the regularized flexible model achieves average minimum validation emror of 1.746E-04, signif- icantly outperforms all the other models evaluated in this experiment. Further pair-wise statistical tests with normal assumption give p-values between 10-2 10-6 Meanwhile, this model has only

---

dimension reduction (Baldi 2012 Goodfellow et al. 2016). In this experiment, connected auto-encoder networks, for which both the encoder and decoder have three hidden layers. Under review as a conference paper at ICLR 2020
6.82% more parameters compared with the corresponding fixed model as is calculated in A.3 which still much smaller than the poorly performed ones with larger sizes in this experiment. Moreover, this performance improvement also be observed in further experiments other combinations of stock indices. Another randomly drawn combination is investigated in A.4.2
3.2 EXPERIMENT WITH DEEP
The deep auto-encoder based on neural networks is widely implemented in data compression and use two fully
The difference between these two baseline models is the sizes of two layers in each of the encoder and decoder. The following are the flow graphs of these two models.
(nput(28*28) Linear(28*28, 128) ReLu
Linear( 128, di) ReLu Linear(d,, 12)
ReLu Linear( d2) Coding + Linear(d2, 12)
ReLu, Linear(12, 64) ReLu
Linear(64, 128) ReLu Linear( (128, 28*28) Output(28*28)
Here we use d1 and d2 to denote the layer sizes differ in two models. For the first model "AE1", dj = 36 and d2 = 6, while for the second model "AE2", d1 64 and d2 3. For models with flexible activation, we replace the ReLu activation function with the parameterized function shown in eq.equation [ as well as the existing PReLu activation (He et al., 2015). To avoid adding too many extra parameters, we only introduce flexible activation functions in the 3th and 4th ReLu layes in AEI, while for AE2, they only introduced in the 2th and 5th ReLu layers. In each trial,
average reconstruction error (MIS)
0.000
ReLu
ReLu
PRcLu
PReLu
0.655
Flexible
0.055
0.025
50
40
(a) Time Step (AE 1)
(b) Time Step (AE
Figure 2: Comparison between the average learning curves (with error bars) auto-encoder models with and without regularized flexible activation functions MNIST dataset.
randomly sampled 4,800 training examples, other 1,200 validation examples from 60,000 training examples the original MNIST dataset, and evaluate the model by the whole 10,000 examples. The batch size was set to 100 and the learning rates of Adam are optimized based on the validation performance of the baseline models, which is set be 0.00645. The training curves are averaged by 50 trials, and the results is demonstrated in Figure The super-parameter searching procedure the same that in Section 3.1 The optimal learning rates of fixed models 0.00645, and the regularisation coefficients of flexible activation functions is set to be 0.1. Based on the optimized hyper-paramest parameters of each model, 50 experiments are launched for comparing the model with fixed and flexible activations, and the results is demonstrated in Figure 2
As we can see, for both the two auto-encoder architectures, flexible models out-perform the fixed ones with stable performances. In AE2, the newly proposed activation function significantly out- perform PReLu in almost all the epochs, where the flexible activations are added in 2th and 5th ReLu layers. Table 3 gives the corresponding summary for comparing both the validation and test performances of these configurations. is shown that the test cost and minimum validation cost of flexible models generally better than that of fixed models with statistical significance in both the

---

Under review as a conference paper at ICLR 2020
Table 3: Comparison of auto-encoder models with and without flexible activation functions
Model
Params
Data
ReLu
PReLu
Flexible
validation
3,511E-2 (9.7E-4)
3.106E-2 (2.6k-4)
3.120E-2 (1.9E-)
AEI
212,070
test
3,519E-2 (9:9B-4)
3.108E-2 (2.7E-4)
3.122B-2 (1.9E-4)
validation
3.898E-2 (6.6E-4)
3,877E-2 (6.6E-5)
3,708E-2 (7.8E-5)
AE2
219,891
test
3.960F-2 (6.5B-3)
3.948E-2 (6.9E-5)
3,7701-2 (6.9F-5)
two stacked auto-encoder architectures. This advantage can be justified by the results on the test set. The performance of AE1 is generally better since the length of encoded vector is 6 rather than 3 as in AE2. Meanwhile, the flexible models has only less than 0.03% extra amount parame- ters compared with the corresponding fixed ones. Further experiments demonstrated in A.4.3 show that flexible auto-encoder models with 5 encoder layers and decoder layers also outperform fixed models.
3.3 EXPERIMENTS WITH CONVOLUTIONAL NEURAL NETWORK (LENET-5)
This experiment investigates the performance of layer-wisely regularized PRelu function in CNNs,
it is done with LeNet-5 on CIFAR-IO for image classification (LeCun et al. 2015b He et al. 2015)
The model architecture is shown follows:
Input(32*32*3) >> Conv2d(3, 6, 5) ReLa MP(2) - Conv2d(6, 16, 5)
ReLu MP(2)
(10)
Linear(16 120)
BN, ReLu Linear( 120, 84)
BN, ReLu Linear(84, 10) - Output
where "MP" refers to max- mox-lpooing layer, while "BN" refers to batch normalization. In flexible models, we only replace the ReLu activation in the last layer with equation 6 In each trial, we still randomly sample 20% of the training set in CIFAR-10 10 as validation set and use the whole remaining 80% as the training data. The whole test set in CIFAR-10 are used as the test set in our experiment as well. With naive random search, the optimized learning rate for fixed model is 0.0012, while the regularization coefficients for the parameter based on this learning rate in flexible 0.032. Meanwhile, the batch size is still set to be 100.
The average cross-entropy loss validation set during the training whole time of 10 epochs is given by Figure 3 We can see that the average loss curve of PReLu with layer-wise regularization almost always below the curve of fixed ReLu and PReLu without regularization, while all the three sets of models are over-fitted after 8th epoch. To check the significance, the corresponding summary statisties for 50 trials shown in Table 4 where the test results are given by accuracy in classification.
We can see that in Table the average minimal validation cross-entropy loss of regularized flex- ible models still achieves significant improvement compared with fixed ones (ReLu) and flexible model without regularization (PReLu). Even though the mean accuracy of fixed model seems to be slightly better with batch normalization, it is quite insignificant considering the standard errors. Since the stopping time (10 epochs) not optimized for both models, this does not make much sense considering the existence of over-fitting The significant validation results indicate that layer- wise regularization on activation parameters could provide an improvement on models with flexible activation functions even in finely designed benchmark architectures such LeNet-5.
Table 4: Comparison of ReL., PReLu and PReLu with Layer-wise Regularization in LeNet-5
Model
Params
Data
ReLu
PRcLu
PReLuureg
validation
1.084 (5.7E-3)
1.0796 (4.6E-3)
1.075 (4.7E-3)
LcNet-5
61,706
test (acc)
62.35 (2.0F-1)
62.39 (1.7R-1)
62.67 (1.6F-1)
validation
1.086 (5.9E-3)
1,083 (4.9E-3)
1.072 (4.9E-3)
LeNet-5 (BN)
62,522
test (acc)
62.53 12
62.36 (2.0F-1)
62.48 (1.7F-1)
8

---

Under review as a conference paper at ICLR 2020
ReLu PReLu
Relu
PReLu
aver aggagge cross entropy loss
2.30
PRetu_reg_laye
1.30
Ppeluureg_layce
1.25
1.20
1.2
10
(a) Time Step (LeNet-S)
(b) Time Step (LeNet-S with BN)
Figure 3: Comparison between the average learning curves (with error bars) of CNN models (LeNet- 5) with and without regularized flexible activation functions CIFAR- 10 dataset.
4 CONCLUSION
In this study, we proposed a set of principles for designing flexible activation functions in a weighted combination form. Especially, we developed a novel flexible activation function that can be imple- mented to replace sigmoid and tanh functions in the RNN cells with bounded domains, well as an alternative one to replace ReLu or PReLu activation in architectures with unbounded domains. In ad- dition, two regularization terms considering the nature of layer-wise feature extraction and goodness of original activation functions are proposed, which is essential in achieving stable improvement of the models. Experiments multiple time series forecasting show that, with replacing sigmoid ac- tivation by the flexible combination proposed this study, stacked LSTMs can achieve significant improvement. Meanwhile, another proposed flexible combination could significantly improve the performance of auto auto-encoder networks in image compression. Further experiments indicate that the models with moderately optimized regularized coefficients could also improve the performance of PReLu in CNN (LeNNtt-5) architectures for image classification. In future studies, it is worthwhile to investigate other flexible activations in combined form based on the proposed framework and princi- ples in this paper, while theoretical justification of the goodness or effectiveness of these activation functions is also a topic of interest.
REFERENCES
Agostinelli, M. Hoffman, Sadowski, and P. Baldi. Learning activation functions improve deep neural networks. arXiv preprint: 1412.6830, 2014.
A. A. Ariyo, A. O. Adewumi, and C. K. Ayo. Stock price prediction using the ARIMA model. In 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, pages 106-112. IEEE, 2014.
D. Asteriou and S. G. Hall. ARIMA models and the Box-Jenkins methodology. Applied Economet- rics, 2(2):265-226, 2011.
Baldi. Autoencoders, unsupervised learning, and deep architectures. Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pages 37-49, 2012.
Y. Bengio, P. Simard, P. Frasconi, al. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
H. Chung, S. Lee, and J. G. Park. Deep neural network using trainable activation functions. In International Joint Conference Neural Networks (IJCNN), pages 348-352. IEEE, 2016.
J. Chung, C. Gulcehre. K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.

---

Under review as a conference paper at ICLR 2020
D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by expo- nential linear units (elus). arXiv preprint:1511.07289, 2015.
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120ul):2121-2159) 2011.
S. Flennerhag, H. Yin, J. Keane, and M. Elliot. Breaking the activation function bottleneck through adaptive parameterization. Advances in Neural Information Processing Systems, pages 7739- 7750, 2018.
F. A. Gers, J. Schmidhuber, and F. Cummins. Learning forget: Continual prediction with LSTM. 1999.
I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In International Conference Machine Learning, pages 1319-1327, 2013.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT Press, 2016.
S. Hayou, A. Doucet, and J. Rousseau. On the impact of the activation function deep neural networks training. In International Conference on Machine Learning, pages 2672-2680, 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
S. Hochreiter.
The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Bel Based Systems, 6(02): 107-116, 1998.
K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2): 251-257, 1991.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pages 448-456, 2015.
R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network archi- tectures. In International Conference on Machine Learning, pages
2015.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv: 141::6412.6980 2014.
arXiv preprint
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436, 2015a.
Y. LeCun et al. Lenet-5, convolutional neural networks. URL: http://yann. lecun. com/exdb/lenet, 20:5, 2015b.
Y. Li, C. Fan, Y. Q. Wu, and Y. Ming. Improving deep neural network with multiple parametric exponential linear units. Neurocomputing, 301:1 301:11-24, 2018.
Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprin::1712.0188) 2017.
A. L. Maas, A. Y. Hannun, and Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
Manessi and A. Rozza. Learning combinations activation functions. In 2018 24th International Conference Pattern Recognition (ICPR), pages 61-66. IEEE, 2018.
V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Pro- ceedings of the 27th International Conference on Machine Learning (ICML), pages 807-814, 2010.
10

---

Under review as a conference paper at ICLR 2020
R. Pascanu, T. Mikolov, and Y. Bengio. Understanding the exploding gradient problem. CoRR, abs/1211.5063, 2, 2012.
L. R. Sütfeld, F. Brieger, H. Finger, S. Füllhase, and G. Pipa. Adaptive blending units: Trainable activation functions for deep neural networks. arXiv preprint: 10064, 2018.
Tieleman and G. Hinton. Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. Technical Report., 2017. 2017.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 EXPRESSIVELY INDEPENDENT
Definition 1: If component activation function fü is expressively independent of a of other component functions: f1,... fu. there does not exist set of combination coefficients o1...,,0. inner activation parameters fn...nn. parameters of the previous linear layers W', b', such that for any input X, activation parameters of function SR, and parameters of the previous linear layer: Wa, by, the following equation holds:
S&WW + bb, BR) Σ assi(W'XX X 3,)
(11)
For a single-layer network with m neurons, if a component activation function Sk. which is not expressively independent of other component functions, is excluded, we need at most 2m neurons to express the same mapping.
Proposition 1:
Ei= Assume that f& is not expressively independent. If we exclude fa from the combination and get F" = Ei= oi then for any input X, parameters of the previous linear layer: W, b, and activation parameters: {B)=1 of a specific neuron, we have:
Proof:
For any original combined activation function: F
F(WX +b,))
Σ
off + bb + (WWX +b
(12)
i= i
and there exist W b' and fi such that:
n
F(WX- + b,3)
Σ + i=1,isk
E d ( 1=1,i
(13)
which can be expressed by two neurons with function (/ with corresponding weights and bias in the previous linear layer. Therefore, for a single-layer network with m neurons, we need at most 2m neurons without fü express the any mappings by the original function.
A.2
THE SHAPES OF COMBINED FLEXIBLE ACTIVATION FUNCTIONS
A.3 CALCULATING THE NUMBER OF PARAMETERS
When batch-normalizztton is not implemented, the number of parameters each deep neu- ral network is equal the number of weighs plus the number of bias. For an FFNN, E where L is the number of layers including in-
N Nw + No st put layer, hidden layer and output layer, while n; is the number of units in ith layer. If flexible activations are introduced in all hidden and output units, with
independent extra parameters in each unit, the total number of model parameters will increase by N, > n,. The number of parameters in basic RNN models is depend on the type of hidden cells used, in general it is N = 0 L ( For basic RNN, g = 1, for GRU, g = 3, and for LSTM, = 4. Meanwhile, the extra number of parameters needed for RNNs with flexible activations is N₂ = 2 _ 8,74, where 8; is the number of activation functions replaced by flexible ones in ith layer.
n:) + 11441), where g is the number of weight matrices in each cell.
Following this derivation, in the LSTM models used in Section 3 we can make a summary table for the increasing ratio of model parameters in each case. As shown in Table[5] we can see that the models with flexible activations have about 5% to 10% increase in the number of parameters. However, it is shown in Table 6 that model I, model 3 and model 4 have very similar validation performances during training even though their number of parameters varies in much higher (even more than 100%) proportions, while the corresponding validation performances for models with flexible activations are better than all the models with fixed activations.
12

---

Under review as a conference paper at ICLR 2020
u=1.8=0
0-0.8-0.1
1.0
0.5. 8-0.1
0.8
0.8
0.8
0.6
0.6
0.6
0mp
0.2
0.2
0.2
0.0
0,0
0.0
-10
-5
10
-10
-5
5
10
-10
-5
10
0.5.5 0.05
0.0.5, F D.G3
0.8
0.0.5, 5-0.01
0.8
0.8
0.7
0.6
suggut
0.6
0.6
0.5
0.4
0.2
0.2
0.3
0.0
0.2
-10
-5
10
-10
-5
10
-10
-5
10
Irput
Figure 4: The shapes of combined activation function proposed in Section parameters.
2.2 with different set of
Table 5: Comparison of number of parameters in different settings of I LSTM models.
model
layer size
basic model
flexible activa-
of in-
tion
crease
[5.
2208
144
6.52%
[5. 8, 4, 4]
800
96
12.00%
[5. 16]
1408
96
6.82%
[5. 16, 16]
3520
192
5.45%
Similarly, in Table 2 the best performed model with layer size [5, 18] has a single hidden layer and the number of parameters is between the model with layer sizes [5, 16, 8] and [5, 8, 4]. Therefore, the number of parameters does not really matters in this range when we consider the minimum validation performance in long training time, and the improvement of model performance can be explained by the advantage of introducing flexible activation with trainable parameters.
For the two deep auto-encoder models studied in Section 3.3 the number of basic model pa- rameters can be calculated with the formula as same as feed-forward neural network, which is: N Nw N₁ -/-- 0,7441 + _ Based the layer structure shown in Eq. equation 9 the deep auto-encoder model AE2, for example, has 219,891 basic parameters. Meanwhile, the extra parameters flexible activation functions is 48, which is only 0.022% the basic parameters.
the basic model parameters include those in convolutional layers as well those in full connected layers. The number of parameters of each convolutional neural network is: N. = (n * / 1) k. where the filter size, while and è represent the input and output feature maps. The total number of two convolutional layers in Eq. equation 10 is 2,572, and the number of parameters fully connected layers is 59,134, giving a total number of 61,706 parameters in our Lenet-5 model. For model with batch normalization. the number batch normalization parameters is 4* 41200884) 816, and the total number of parameters 61, 7,60 816 62,522. In addition, for flexible models, the extra activation parameters introduced
For the CNN Lenet-5 model in Section 3.3
13

---

Under review as a conference paper at ICLR 2020
into the second last layer is 84*2=168, which is only about 0.13% of the total number of basic parameters.
To sum up, the results at least show that we can achieve a quite significant performance im- provement with only small proportion of extra parameters in the flexible activation functions. Although the same mapping may be learned without flexible activation functions, it may need much more number of parameters much larger effort of happer-parameter searching with bunch of different model architectures.
A.4 EXTENDED EXPERIMENTAL RESULTS
A.4.1 FORECASTING UNI-VARIATE TARGET WITHOUT REGULARISATION ON ACTIVATION PARAMETERS
In the first experiment for LSTM model on multi-variate financial time series data, we focus on the case when the target is uni-variate. Here choose the one-step-ahead daily return of Brazil index as the target to be predicted, while the indices of other four countries in the dataset are used as predictors along with the historical data of the target. To avoid the uncertainty caused by different hyper-parameters for different models, we use the recommended learning rate of 0.01 for Adam optimizer for all the models this experiment. In addition no regularizer for flexible activations is added in this stage, which will be considered later.
3.000750
flexible
3.0002400
8.0022755
3,0023300
0.900240
3.0023000
0.000235
3.0002275
0.000230
20
25
20
(a) Time Step [5, 16], window size
30
(b) Time Step [5, 8, 4, 4), window sizee 30
6.000245
0. 0.000245
flexible
0.000240
0.000260
Validation
0.000235
C.C0n230
0. 0.000000
0.000225
10
20
25
5
10
20 25
(c) Time Step [5, 16, 8), window size
30
(d) Time Step [5, 16, 16], window size 30
Figure 5: Comparison between the learning curve of LSTM model with and without flexible acti-
vation functions on Multi stock indices return data in forecasting uni-variate return. (a) Two-layer
L.STM model with layer sizes: (5, 16, 8): (b) Three-layer L.STM model with layer sizes: [5,8, 8, 4, 4]; (c) One-layer LSTM model with layer sizes: [5, 16]; (d) Two-layer LSTM model with layer sizes:
[5, 16, 16).
14

---

Under review as a conference paper at ICLR 2020
The result for learning curve of recurrent neural networks for multiple time series stock in- dices data is shown in Figure []] We find that for all stacked LSTM models with different layer configurations, the version with flexible activations outperforms the version with fixed activation in most of the learning time. To understand the significance of the improvement, we perform two statistical tests by assuming the validation performance of the model with different random initialization following normal distributions.
Table 6: Summary table of stock indices forecasting with Stacked LSTM
Fixed Activation
Flexible Activation of min
model
Layer size
mean of min
S.E.
S.E.
p-value
16,
2.194E-04
1.77B-07
2.185E-04
191E-07
8.173E-03
2
[5,8, 4]
2.272F-04
9.09F-07
2.228E-04
7.47F-07
8.31
3
[S,
16]
2.191E-04
1.77B-07
2.186E-04
1.91E-07
4.507E-02
4
[5. 16, 16]
2.193E-04
1.26E-07
2.187E-04
1.27E-07
2.632E-04
It is shown in Table 6 that the averages of minimum validation cost by using stacked LSTM with flexible activation are significantly smaller than that with fixed activation in terms of stock indices forecasting. On the other hand, we find that their costs after 30 epochs on test set both outperform traditional ARIMA models in on the same dataset (Asteriou and Hall [201]] Ariyo eet al.) 2014), which is 0.0002217. In terms of the standard errors, it does not show that one of these two set of models are more robust to the noise in initialization than the other.
Table 7: Comparison between different forecasting methods on test set
LSTM fixed
LSTM flexible
model
RW
ARIMA(S,1,0)
mean
S.F.
mean
S.F.
test mie
3.030B-04
2.217E-04
1.669E-04
3.173E-06
1.644E-04
2.18E-06
Table 7 shows the performances of different forecasting methods on test set, where the stacked LSTM models with flexible activations outperforms other methods including Naive random walk, ARIMA(5, 1, 0) and LSTM with fixed activations, in terms of the average test MSE for the whole 200 trials. However, since the test results LSTM model based on the model after 30 epochs rather than the optimal number of epoch during the training, the performance of LSTM with flexible activations is not significantly lower than that of fixed activations considering both the means and standard errors.
Table Summary table of stock indices forecasting with Stacked LSTM
Fixed Activation
Flexible Activation
model
Layer size
mean min
S.E.
mean min
S.E.
p-value
LSTM-1
[5, 16, 8]
1. 1.980E-04
1.5SE-06
1.972F-04
2.39F-06
2.47E-02
LSTM-2
8,
2.006E-04
6,91E-07
1.998E-04
2.07E-06
4.91E-03
LSTM-3
[6]
1.916E-04
6.39E-07
1.913E-04
4.89E-07
5.03E-03
LSTM-4
[5, 16]
1.933F-04
1.56F-06
1.918F-04
1.12F-06
3.88F-08
A.4.2 LSTM WITH A DIFFERENT COMBINATION OF STOCK INDICES
In the extended experiment on multiple stock indices forecasting with LSTM, we selected combi- nation of four different stock indices including indices of British, Australia, Korea and Russia. We do the experiment with two different window sizes (10 and 30), while the regularization coefficients for activation parameters are not optimized on the new dataset and window size. The corresponding results shown in Figure 7 and Table 9
15

---

Under review as a conference paper at ICLR 2020
fixed flexible
0.000202
0. 0200220
Validation
8.000198
0.000210
0. 000200
0.000194
10
15
20
25
30
5
10
20
25
(a) Time Step [5, 16],
window size
30
(b) Time Step [5, 4, 4), window size 30
0.000214
0.000206
fixed
flexible
0.000212
0.900204
0,000202
C.008208
6.000006
0.000204
9. 3.000198
C.000200
0.000196
0.000198
10
15
20
25
30
5
10
15
20
25
30
(c) Time Step [5, 16, 81, window size
(d) Time Step [5, 16, 16], window size 30
Figure 6: Comparison between the average learning curves (with error bars) of LSTM models with and without regularized flexible activation functions on Multi stock indices return data in forecasting multi-variate return.
Table Summary table of stock indices forecasting with Stacked LSTM
Window Size
Model
Data
Fixed
Flexible
Flexible (Regularized)
10
validation
1.552F-4 (1.6F-7)
1.549F-4 (1.4F-7)
1.549E-4 (1.2F-7)
LSTM-1
7.863E-5 (1.0E-6)
7.871E-5 (1.3E-6)
7.774E-5 (1.4E-6)
validation
1.682E-4 (3.3E-8)
1.692E-4 (3.9E-7)
1.611E-
LSTM-1
7.780F-5 (6.05-7)
7.661E-5 (3.1E-7)
7.761E-5 (4.9B-7)
A.4.3 5-LAYER AUTO-ENCODER
This is for checking the robust of flexible auto-encoder models with deeper architecture. The dia- gram of data flow is shown follows.
Input(28*28) Linear(28*28, 128) ReLu Linear(128 64) ReLa Linear(64, 36)
Relu, Linear(36, 12) Rela
Linear(12, 6) + Coding - Linear(6, 12) Relu Linear(12, 36) Rela
Linear(36, 64) Linear(64, 128) ReLa Linear( (128, 28*28) + Output(28+28)
ReL.
(14)
Here we still introduce two models: AE3 and AE4. In AE3, flexible activations are added to replace the 3th and 6th ReLu in Eq, 14, while in AB4, flexible activations are added replace the 4th and 5th ReLu. The experimental results are shown in Figure Baand Table TO
16

---

Under review as a conference paper at ICLR 2020
0.000195
fixed
flexible
6.000190
D. 000177
flexible_ reg
Validation MSE
2.000185
0.000174
0.000180
0.000175
0.000172
8.000170
0.000170
10
20
25
5 10
20
25
(a) Time Step [4, 16], window size 10
(b) Time Step (4, 16], window size 30
Figure 7: Comparison between the average learning curves (with error bars) of LSTM models with and without regularized flexible activation functions on Multi stock indices return data in forecasting multi-variate return. (a) layer size [4, 16], window size 10; (b) layer size [4, 16], window size 30
0.065
reconstruction (MSE)
+ ReLu
ReLu
0.060
PReLu
PReLu
0.060
Flexible
Flexible
0.055
0.050
0.050
0.045
0.045
0.025
40
(a) Time Step (AE
(b) Time Step (AE
Figure 8: Comparison between the average learning curves (with error bars) of auto-encoder models with and without regularized flexible activation functions on MNIST dataset (5-Layer).
Table 10: Comparison of auto-encoder models (5-layer) with and without flexible activation func- tions
Model
Params
Data
ReLu
PReLu
Flexible
validation
3.48 (3.14-4)
3.427R-2 (4.0F-4)
3.351E-2 (3.2F-4)
AE3
223,974
test
3.471E-2 (3.3B-4)
3,408E-2 (4.2E-4)
3.337E-2 (3.3E-4)
validation
3.799F-2 (7.0E-4)
3.496E-2 (3.9E-5)
3.521 (2.6E-4)
AE4
223,974
test
3.744F-2 (7.0E-4)
3.436--2 (3.9E-5)
3.460F-2 (2.5E-4)
17


Reviews:
[{"review_id": "r1xVKGHAYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a parameterized activation function to learn activation functions that have sigmoidal shapes that can be used in LSTMs. The authors apply their method to a dataset of forecasting stocks as well as to CIFAR-10. They also propose a method to regularize the activation function parameters.\n\nThe authors propose an activation function that helps very little in certain cases. I am not familar with this stock prediction dataset, but the differences shown are often less than 1%. I am familiar with CIFAR-10 and the architecture they use give bad baseline results and their method of regularizing a previous learned activation function (PReLU) gives marginally better results in half the cases.\n\nThere have been many learned activation functions proposed that give significantly better results than these. This paper is simply proposing another one and showing little to no improvement.\n\n**After reading author feedback**\nMy score stays the same.\n\nThere were some issues I had with the response:\n\"For stock return forecasting, 1% improvement with statistical significance is sufficiently import...As we know, most newly proposed unbounded activation functions (flexible or not) cannot outperform ReLu by more than 0.5% in terms of test accuracy in a large range of image classification tasks.\"\nThe 1% improvement was on error, not accuracy, so the two are not related. If referring to error rate, there are many that can make improvements greater than 10% (APLs, SReLUs, Swish, etc.)\nIf the authors would like to claim superiority to other activation functions, they should compare to them directly.\n\nIn addition, my concern about the poor CIFAR-10 baseline was not addressed. The results from the 2014 dropout paper gives significantly better results (Dropout:  A simple way to prevent neural networks from overfitting) than the authors' baseline.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJgnV5C_sB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed combined form of flexible activation functions with carefully designed principle of choosing activation functions. It shows some gains on stock price and one standard image task over the baseline.\n\nI'm leaning to reject or give borderline for this paper because (1) the paper don't have comparison with neural architecture search. For example, https://arxiv.org/abs/1710.05941 (Searching for Activation Functions). I don't know what the advantage of this approach compared to searched activation. I guess it's less computation heavy and maybe better motivated. But at least the author should give some pros/cons. (2) the paper has two benchmark, stock price prediction and CIFAR-10. I don't understand why as arch paper, it use such non standard benchmark (stock) and non standard arch (LeNet?). I don't think based these benchmark we can make solid conclusion. (3) These model seems introduce quite a bit more hyper-parameters. It's unclear if it is better than tuning other architecture e.g., batch norm/layer norm/dropout or even just optimizer. For example, \"flexible is 0.032\" does this parameter generalize to other dataset? Like if the gain is really significant, like resnet over AlexNet, hyperparam doesn't matter. But if it's marginal win over a weak baseline, the how to get the results is important.\n\nSome comments:\nIn section 2, \" back propagation of these activation parameters by stochastic gradient descent can be\ndone as follows\"\nWhy we need to list the detail backprop formulation here? Are these special? Isn't just autograd?\n\nCan the author explain more for principle 1? What is the \"same domain\" means here?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg-1E3aFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a family of parameterized composite activation functions, and a regularization technique for parameterized activation functions in general. While the three sets of experiments show potentially promising results, they aren't able to disambiguate clearly between the effect of the activation function you introduce and the effect of additional parameters in general, or between the effect of the regularization technique you introduce and the effect of regularization in general. I would really like to see the kind of carefully baselined, ablated, and hyperparameter-tuned results that would justify adding the techniques introduced to the toolbox of a typical deep learning practitioner.\n\nSome feedback:\n\nTypos:\n- The formatting is off a bit (shifted horizontally?)\n- In the abstract: RPeLu -> PReLU\n- p. 5: \"basement settings\"->baseline, \"logarithmic sale\"->scale\n- p. 6: basement->baseline again\n- p. 7 etc.: trail->trial\n\nFeedback:\n- Intro:\n  - Explain why maxout is similar to training piecewise activation functions?\n  - It's unclear what \"making the most of the non-linear properties by introducing adaptation policy on the input\" means\n  - An \"autoencoder\" is not an architecture as much as a broad family of architectures coupled with training approaches (I'm guessing you mean a fully-connected autoencoder)\n- Methodology:\n  - consider using \"f\" instead of \"a\" so it's easier to tell apart from alpha?\n- Experiments:\n  - I'm not sure I'm convinced by the statistical tests used on the LSTM results; they demonstrate that your approach, with a few specific sets of hyperparameter settings, does better than the baseline, but not that this represents a valid claim about your activation function's effect on this LSTM model in general.\n  - The autoencoder experiments are even less convincing, in that they represent two seemingly arbitrary network architectures, with equally arbitrary placement of the activation function in one of them.\n  - The regularization experiments use LeNet-5, which is not a compelling benchmark architecture with respect to contemporary practice. The effects of regularization techniques can be very different in different regimes of dataset and network size.\n- Code:\n  - I'm not sure why you performed backprop by hand for your activation functions and used torch.Function, rather than writing them directly as nn.Modules that make use of PyTorch autograd\n  - There are lots of details in the code that aren't in the paper; in general, papers should aim to be relatively self-contained (although I'm very glad to see your code, and it's pretty simple to follow)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content: Proposes combining flexible activation functions 

Discussion:
reviewer 1: main issue is unfamiliar with stock dataset, and CIFAR dataset has a bad baseline.
reviewer 2: main issue is around baselines and writing. 
reviewer 3: main issue is paper does not compare with NAS.

Recommendation: All 3 reviewers vote reject. Paper can be improved with stronger baselines and experiments. I recommend Reject.
Decision: Reject