Title: SYMMETRIC-APL ACTIVATIONS: TRAINING IN-SIGHTS AND ROBUSTNESS TO ADVERSARIAL AT-TACKS

Abstract: Deep neural networks with learnable activation functions have shown superior performance over deep neural networks with fixed activation functions for many different problems. The adaptability of learnable activation functions adds expressive power to the model which results in better performance. Here, we propose a new learnable activation function based on Adaptive Piecewise Linear units (APL), which 1) gives equal expressive power to both the positive and negative halves on the input space and 2) is able to approximate any zero-centered continuous non-linearity in a closed interval. We investigate how the shape of the Symmetric-APL function changes during training and perform ablation studies to gain insight into the reason behind these changes. We hypothesize that these activation functions go through two distinct stages: 1) adding gradient information and 2) adding expressive power. Finally, we show that the use of Symmetric-APL activations can significantly increase the robustness of deep neural networks to adversarial attacks. Our experiments on both black-box and open-box adversarial attacks show that commonly-used architectures, namely Lenet, Network-in-Network, and ResNet-18 can be up to 51% more resistant to adversarial fooling by only using the proposed activation functions instead of ReLUs.

1. INTRODUCTION:

Deep neural networks (DNNs) are a series of linear transformations followed by point-wise nonlinear functions. These non-linear functions are also referred to as activation functions. Activation functions directly influence the training dynamics and, therefore, affect the final performance of the DNN. For instance, activations such as Sigmoid and Tanh suffer from what is known as the "vanishing gradient" problem, when the inputs are in the saturation regions of these functions. This can adversely affect network performance and cause slow convergence. Rectified linear units (ReLU) (Nair & Hinton, 2010) have been shown to perform better than Sigmoid and Tanh in many scenarios. Instead of using a fixed activation function, one can learn a parameterized activation function to add expressive power to the model. Towards this goal, significant efforts have been made to introduce different learnable activation functions (Agostinelli et al., 2014; He et al., 2015; Ramachandran et al., 2017; Jin et al., 2016; Li et al., 2016) . Although most of these adaptive activation functions outperform fixed activation functions in multiple tasks, there is very little insight into why.

In this paper, we propose the Symmetric Adaptive Piecewise Linear unit (Symmetric-APL; pronounced as Symmetric apple: "aep.@l) as a new adaptive activation function which builds upon the adaptive piecewise linear unit (APL) (Agostinelli et al., 2014) . Similar to APL units, S-APL has multiple hinges which are connected by linear pieces. The S-APL activation is designed to give the same expressive power to both the positive and negative sides of the x-axis. We show that S-APL can approximate any continuous and zero-centered function, which encompasses many types of learned and fixed activations functions. Using this flexible learned activation function, we can then analyze how the shape changes during training and use ablation studies to investigate the role that these changes play in optimization. This helps us better understand learned activation functions and explain why they often provide increases in performance.

Finally, we evaluate the robustness of S-APL activated networks to adversarial attacks. Experiments are done using both black-box and open-box adversarial attacks where we observe S-APL equipped networks are more robust to attacks than their ReLU counterparts. Experiments show that the robustness improves against differential evolution attacks (Su et al., 2019b) by (up to) 51% on Lenet , 47% on ResNet-18 (He et al., 2016) , 43% on pure CNN, and 38% on Net in Net (Lin et al., 2013) . For open-box adversarial attacks, the robustness also improves against FGSM and CW-L2 (Goodfellow et al., 2014) attack by (up to) 36%, 28%, and 25% for Lenet, pure CNN, and ResNet-18 architectures, respectively using CIFAR-10 dataset.

2. RELATED WORKS:

ReLU and its variants such as Leaky-ReLU (Maas et al., 2013) , P-ReLU (He et al., 2015) , exponential linear units (ELU) (Clevert et al., 2015) , and scaled exponential linear units (SELU) (Klambauer et al., 2017) are believed to solve the vanishing gradient problem and some are shown to better normalize the network. By changing the negative side of x-axis, these variants push the mean activations of each unit closer to zero and accelerate learning. Some of the pioneering attempts to learn activations in a neural network can be found in Poli (1996) , Weingaertner et al. (2002), and Khan et al. (2013) where the authors proposed novel approaches to learning the best activation function per neuron among a pool of candidate activations by employing genetic and evolutionary algorithms. Maxout (Goodfellow et al., 2013) has been introduced as an activation function aimed at enhancing the model averaging properties of dropout (Srivastava et al., 2014) . However, not only is it limited to approximating convex functions, but it also requires a significant increase in parameters. Neural architecture search (Ramachandran et al., 2017) uses a combination of exhaustive and reinforcement learning based search. Although the authors discovered multiple novel activation functions, of which f (x) = x * sigmoid(βx) performed slightly better than ReLU, the search space of activation functions is limited. S-ReLU (Jin et al., 2016) was proposed as an adaptive activation function which can mimic both convex and non-convex functions.

It is worth mentioning that, in Lin et al. (2013) , the authors proposed the network in network approach where they replace activation functions in convolutional layers with small multi-layer perceptrons. Theoretically, due to Universal Approximation Theorem, this is the most expressive activation; however, it requires many more parameters.

3. ADAPTIVE PIECEWISE ACTIVATION FUNCTION:

3.1 ADAPTIVE PIECEWISE LINEAR UNITS Agostinelli et al. (2014) proposed an adaptive activation function which consists of multiple linear pieces. The formulation of this adaptive piecewise linear (APL) activation function is shown in Equation 1.

h i (x) = max(0, x) + S s=1 a s i max(0, -x + b s i ) (1)

Where h i is the non-linear transformation for the hidden unit i and S is a hyperparameter that determines the number of hinges. a i is the slope of the linear piece and b i is the location of the hinge.

However, APL units are not zero-centered as they can have a non-zero output for a input of zero. This behavior provides no apparent beneficial purpose. Furthermore, APL units can only represent piecewise linear functions whose output h i (x) = x for x > u, for some scalar u. This significantly restricts the class of piecewise linear functions that APLs can express.

3.2. SYMMETRIC ADAPTIVE PIECEWISE LINEAR UNITS (APL: "aep.@l):

To overcome the shortcomings of APL activation function, we define a new adaptive piecewise linear activation function that is zero-centered and gives the same expressive power to the positive and negative half of the input space. Since S-APL shares the variables a s + , b s + , a s -, and b s -among all the neurons of a layer (e.i., h i (x, S) does not depend on i), it requires fewer parameters in compare to the plain APL. We formulate the activation h i of the hidden unit i as the summation of symmetric hinge-shaped pieces.

EQUATION

Similar to APL units, S is a hyperparameter which determines the number of hinges (on each side of the x-axis). a s + and a s -control the slope of linear pieces in both the positive and negative sides of the x-axis, while b s + , b s -∈ [0, +∞) determine the location of hinge s. The total number of additional parameters is 4S for a single layer. Because our networks are trained with batch normalization (Ioffe & Szegedy, 2015) before each S-APL layer, the distribution of input to the S-APL function will follow that of a normal distribution. Therefore, we fix the hinges to be a pre-determined number of standard deviations away from the mean, decreasing the number of parameters to 2S.

Based on Equation 2, S-APL activation is zero-centered which leads to more stable statistical behavior during training. In case of using fixed hinges' locations for APL, due to increasing the number of parameters, S-APL offers more flexibility than the normal APL. However, this increase is of O(constant) which is negligible. With the assumption of a s + = a s -and b s + = b s -, since h s i (x) = h s i (-x), S-APL allows neurons to activate symmetrically. As it is mentioned in Zhao & Griffin (2016) , networks with symmetric activation functions are more robust to adversarial fooling. We will delve into this property in later sections. S-APL can better handle the limits. It could be inferred from the definition that, in contrast to normal APL, increasing the absolute value of the input x will also increase the number of terms affecting the h i (x). However, normal APL requires the rightmost linear section in all the component functions to have a unit slope and bias 0, which is not an appropriate constraint and undermines its representation ability.

The following theorem shows that S-APL can approximate any non-linear, zero-centered, and Lipschitz continuous function in a closed interval of real numbers. It is worth mentioning that APL was not flexible enough to do so. • A and B are finite real numbers.

• f is M-Lipschitz continuous.

Proof: We start by dividing the interval:

[A, B] into S equal sub-intervals [A i , B i ].

In other words, we put S equally distanced hinges in the interval [A, B]. So the length of each sub-interval will be B-A S . We start approximating f from the left most sub-interval which is

[A, A + B-A S ]

. The approximation of f in that sub-interval will be a line which connects

f (A) to f (A + B-A S ). Due to M-Lipschitz continuity of f , for any x ∈ [A, A + B-A S ]: |f (x) -S-APL(x)| ≤ |f (A) -f (A + B-A S ) + M (B-A) S | f (A) ≤ f (A + B-A S ), |f (A + B-A S ) -f (A) + M (B-A) S | f (A) ≥ f (A + B-A S ).

(3)

Again, due to M-Lipschitz continuity of f , we know that:

EQUATION

Using Equation 4, we can re-write the inequality 3 as follows:

EQUATION

So, by choosing S ≥ 2M (B-A) , we satisfy that |f (x) -S-APL(x)| ≤ .

in order to complete the proof, we need to show that this approximation can be applied to other sub-intervals of [A, B] as well. We do so by induction as follows: Suppose that the approximation is done for sub-intervals

[A 1 , B 1 ], ..., [A i , B i ], now we prove that for sub-interval [A i+1 , B i+1 ] we can do the approximation which is connecting f (A i+1 ) to f (B i+1 ). The slope of S-APL in the sub- interval [A i , B i ] is i 1 a i + or i 1 a i -(

depending on the sign of the sub-interval). However, the slope of S-APL in the sub-interval [A i+1 , B i+1 ] will be either

i+1 1 a i + or i+1 1 a i -.

In both cases, the extra term a i+1 + or a i+1 -is not fixed and can change the slop to any arbitrary value. This fact plus the assumption of continuity of S-APL guarantee that f (A i+1 ) can be connected to f (B i+1 ) which was our proposed approximation. As we showed the approximation for the left most sub-interval, the induction is complete.

4. COMPARISON TO COMMONLY USED ACTIVATION FUNCTIONS:

In order to show that S-APL unit is beneficial for deep neural networks, we compare it with wellknown activation functions in different architectures. We employed five different neural networks trained on three different datasets, MNIST, CIFAR-10, and CIFAR-100. We used S = 4 and set the hinges at positions x = -2.5, -2, -1, 0, +1, +2, +2.5 (0 considered as two hinges). Slope of a s + 1 is initialized to 1 and the remaining slopes are initiazlied to 0. With this initialization, the initialized S-APL mimics the shape of ReLU.

The results of the experiments are shown in Table 1 . The table shows that S-APL leads to a better validation accuracy in almost all architectures.

Table 1: ReLU, leaky-ReLU, PReLU, tanh, sigmoid, ELU, maxout (nine features), swish:

x.sigmoid(βx) and S-APL activations are compared in three different tasks. The performance of S-APL in comparison to the best of the other activations. For the sake of brevity, D-A refers to Data Augmentation. The values in the tables are error-rates and are reported in percentage. 

5. HOW S-APL CHANGES DURING TRAINING AND POSSIBLE REASONS WHY:

Figures 1 and 2 show how the shape of S-APL changes for each layer during training. From these figures, we can see that during the early stages of training, the shape of S-APL looks similar to that of leaky ReLU: having a negative output for x < 0 and a positive output for x ≥ 0. During the later stages of training, the activation function has a positive output for both x < 0 and x ≥ 0. In addition, the slope of the output decreases as the magnitude of x increases.

We hypothesize that there are two distinct stages during the optimization of these learned activation functions. In the first stage, the function returns negative values for x < 0 and in the second stage, the function returns positive values for x < 0. To investigate this hypothesis, we train S-APL under two different conditions where we: 1) force the output to only be negative for x < 0 (S-APLnegative) and 2) force the output to only be positive for x < 0 (S-APL-positive). We accomplish this by taking the absolute value of all a s - and multiplying by +1 or -1.

Figure 2 shows that, although S-APL-positive has the ability to mimic the final learned shape of the S-APL function, it barely deviates from the ReLU initialization. This shows that the ability to give a negative output for negative inputs is crucial for S-APLs. Figure 2 shows that S-APL-negative does initially go negative in the 4th layer and then returns to the ReLU shape. To gain further insight, we compare the loss of these activation functions in Figure 3 . We see that S-APL has a lower loss compared to S-APL-positive and S-APL-negative. Furthermore, S-APL-negative has a lower loss compared to S-APL-positive which has a lower loss than ReLU.

These results lead us to believe that it is possible that the first stage, S-APL-negative, adds gradient information by giving negative output for negative inputs. It is possible that S-APL-positive is not capable of doing so because giving positive outputs for negative inputs creates an identifiability issue. In other words, outputs for x < 0 and x ≥ 0 look the same. However, we can see that this property is useful for the S-APL function in the later stages of training. It is possible that this is because the DNN has adapted its parameters such that outputs on both the negative and positive sides capture a similar concept, adding expressive power to the model. Furthermore, one cannot achieve the final S-APL shape without the first stage of having negative outputs for negative inputs. To conclude, it appears that the S-APL units first go through two stages: 1) adding gradient information and 2) adding expressive power and that the second stage depends on the first stage. This information can be used to inform the design of learnable activation functions and even DNN optimization techniques. 

6. ROBUSTNESS TO ADVERSARIAL ATTACKS:

DNNs have been shown to be vulnerable to many types of adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014) . Research suggests that activation functions are a major cause of this vulnerability (Zantedeschi et al., 2017) . Zhang et al. (2018) bounds a given activation function with a few linear and quadratic functions and allows it to tackle general activation functions. This adds up with applying a different activation for each neuron so the resulting network shows efficiency and robustness to adversarial foolings. Wang et al. (2018) proposed a data-dependent activation function and empirically shows its robustness to both black-box and gradient-based adversarial attacks. Other In this section, relying on the properties of S-APL such as data dependency, piecewise linearity, noninjective behavior, better handling of extreme values, and most importantly, symmetric shape, we show that adding S-APL to a DNNs greatly improves the robustness against adversarial attacks. This claim is verified through a wide range of experiments with the CIFAR-10 dataset under both blackbox and open-box methods, including one-pixel-attack and Fast Gradient Sign Method.

Figure 4 provides an intuition for the robustness of S-APL activated network in comparison to ReLU activated ones. For each of the two networks, we take 100 random samples of frog and ship images and visualize the pre-softmax representations using tSNE visualization (Maaten & Hinton, 2008) in Figure 4 . As it is depicted, for S-APL activated networks, the two classes are better separated. We suspect that red samples such as those that are lying on the outer circle in the left plot of Figure 4 might be somewhere on the manifold of the data where the network is more susceptible. As one can see, there is no red point on the outer circle of the right plot which might be showing the robustness of S-APL networks. For other samples on the inner circle and the cloud next to circles, we can see better separation by the S-APL network. 

6.1. BLACK-BOX ADVERSARIAL ATTACK:

For black-box attacks against a DNN, we assume the adversary has no information about the structure or parameters of the DNN and does not have access to any large training dataset. The adversary's only capability is to observe labels assigned by the DNN for chosen inputs in a manner analog to a cryptographic oracle. A successful attacking technique of this type is Su et al. (2019a) which is based on differential evolution. We can iteratively generate adversarial images to try to minimize the confidence (probability) of the network's classification.

The process starts with random modifications of a few pixels to generate adversarial examples. At each step, one feeds several adversarial images to the DNN and only observes the output as the probability of different classes. Examples that lowered the confidence of the true class will be kept to generate the next generation of adversaries. The next generation modifications are then evolved through a certain mutation scheme to generate new adversarial images. By repeating these steps for a few iterations, the adversarial modifications generate more and more misleading images. The last step returns an adversarial modification that reduces the confidence of the true class the most. This means that the confidence of the true class would be reduced so much that a new incorrect category now has the highest classification confidence. This technique is called untargeted attack since we don't specify the labels of the modified images.

In the following experiment, we modify one, three, and five pixels of images to generate adversarial examples. The mutation scheme we used for this experiment is as follows:

EQUATION

Where r 1 , r 2 , and r 3 are three non-equal random indices of the modifications at level l. x l+1 i will be an element of a new candidate modification.

To evaluate the effect of S-APL activation on the robustness of DNNs, we employ commonlyused architectures, namely, Lenet5, Network-in-Network (Lin et al., 2013) , pure CNN (ours), and ResNet-18 (He et al., 2016) . Each architecture is trained with ReLU, APL, and Swish activation functions and once with S-APL activation functions. The results in Table 2 show that Lenet5 and ResNet-18 architectures are 51% and 47.8% more resistant to back-box adversarial attacks. As authors in Zhao & Griffin (2016) theoretically proved, that one can be robust to adversarial examples with symmetric behavior.

After observing adversarial samples which are deceiving to both networks, we found that S-APL activated network still assigns high confidence to the true labels of the perturbed images. In other words, the difference between the confidence of the true label and the confidence of adversarially wrong label is much smaller in S-APL network than networks that are activated with ReLU, APL, or Swish. More precisely, we measure the average of |Z(x ) true label -Z(x ) adversarial label | over all adversarial samples where each network is fooled. Z(.) is the post-softmax output and x is the adversarial sample. The avg(|Z(x ) true label -Z(x ) adversarial label |) for each model is also included in Table 2 . Due to the reported averages, we can conclude that S-APL is much more robust against being adversarially deceived.

6.2. OPEN-BOX ADVERSARIAL ATTACK:

To further explore the robustness of S-APL activated networks, in this section, we consider two of the popular benchmarks of open-box adversarial attacks: Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) and CW attack (Carlini & Wagner, 2017) . For both attacking strategy, we consider three different architectures and compare the rate of successful attacks for each of the networks with ReLU, APL, or Swich activations versus S-APL activation functions. The dataset and architectures are same as those were used for black-box adversarial attacks.

6.2.1. FGSM:

Fast Gradient Sign Method generates an adversarial image x from the original image x by maximizing the loss L(x , y), where y is the true label of the image x. This maximization problem is subjected to ||x -x || ∞ ≤ where is considered as the attack strength. Approximating with the first term of Taylor expansion, we have:

EQUATION

So the adversarial image x would be:

EQUATION

Where the is considered as the power of the attack. This form of FGSM is considered to be an untargeted attack where there is no pre-specified label t that x should be classified as. This method only yields an adversary x which will not be classified as y.

To see how S-APL increases the robustness against FGSM attack, we employ three architectures all trained to classify CIFAR-10 dataset. For different range of the results are summarized in Table 3 . In the best case, for Lenet architecture and = 0.02, S-APL activated network is 36% more robust than the ReLU activated one. The robustness also improves by 28% and 25% for pure CNN and ResNet-18 respectively. We can see the overall increase in the robustness over all networks and values. However, one can conclude that for smaller values of the robustness of S-APL activated network is higher in comparison to ReLU activated networks. As we increase the attack strength ( ), the robustness of S-APL networks gets closer to the robustness of ReLU activated networks.

6.2.2. CW-L2:

We showed that replacing ReLU with S-APL activation units can greatly improve the robustness of DNNs against FGSM attack. To further investigate the robustness that S-APL is adding to the network, we are using a more powerful open-box adversarial attack.

Another open-box adversarial attack which is generally more powerful than FGSM was introduced in Carlini & Wagner (2017) . For a given pair of image x and label y, this technique is trying to find the minimum perturbation δ, so that the perturbed image x is classified as t = y. Considering L 2 norm as the goal of minimization, this perturbation minimization problem can be formulated as follows:

∀t = y, min||δ|| 2 2 subject to f (x + δ) = t, x + δ ∈ [0, 1] n (9)

Or equally, to ease the satisfaction of equality, Equation 9 can be rephrased as min||δ|| 2 2 +c.g(x+δ) where g(x) = max(max t =y (logit(x) t -logit(x) y )), c is Lagrange multipliyer, and logit(x) is the pre-softmax vector for the input x.

Here, we examine the robustness of S-APL activated networks against CW-L2 attack. As it is shown in the table 4, CW-L2 attack is more powerful than FGSM. Although the S-APL activated network is less robust to CW-L2 attack than to FGSM attack, it is still showing more robustness in comparison to ReLU, APL, and Swish activated networks. Similar to black-box attack, the metric avg|Z true -Z adv | is calculated for successfully attacked images. The smaller values of avg|Z true -Z adv | for S-APL activated networks in both open-box attacks shows that S-APL activated networks are more resistant to being fooled.

7. CONCLUSION AND FUTURE WORKS:

We extend the idea of piecewise linear activation functions by introducing S-APL as a learnable and potentially symmetric activation function. We investigate the properties of S-APL and their evolution during training to understand how these functions accelerate learning and increase expressive power in DNNs. Finally, we show that S-APL networks show more robustness to adversarial attacks than ReLU networks. One can potentially design a hybrid defense method that combines other defense techniques and S-APL units.

8. APPENDIX:



8.1. NUMBER OF HINGES AND THE SYMMETRY OF S-APL:

In this section, we perform a variety of experiments to find the best setting for S-APL activation in terms of complexity and expressibility.

Before going to experiments, it is necessary to mention that we used batch normalization layer before every S-APL layer in our experiments. This means that the input of the S-APL activation is unit Gaussian. Relying on this fact and the experiments done in Agostinelli et al. (2014) we chose a fixed and symmetric set of hinges on ±σ, ±2σ, ±2.5σ, ±3σ, and ... . This will reduce the number of S-APL's parameters by a factor of 2 which makes it much less computationally expensive. First, we assess the effect of S on the performance of S-APL. Due to Theorem 3.1, greater S values increase the expressive power of the S-APL which results in better fitting the training data. We tried S ∈ [2, 3, 4, 5, 6], (which means 4, 6, 8, 10, and 12 hinges) with fixed hinges' locations on MNIST (LeCun & Cortes, 2010) , and binary classification of IMDB review dataset (Maas et al., 2011) to cover MLP, convolution, and recurrent architectures respectively. Each network is trained with two types of S-APL activation; 1) A shared S-APL: a shared unit among all neurons of a layer and 2) An independent S-APL unit for each neuron of a layer. As it is summarized in Table 5 , in all cases for S ≥ 4 the performance is not decreasing significantly.

On the other hand, due to the increase in the number of parameters of S-APL, the unit becomes more computationally expensive. In Table 6 we compare the per-epoch training run-time for different number of hinges of a shared S-APL. For small values of S, we can see that S-APL is comparable to an exponential activation unit such as Tanh, and much faster than heavier activation such as Maxout.

As one can conclude from both Table 5 and 6 , there is a trade-off between the complexity and expressibility. We believe that S = 4, or equally 8 hinges, is the best choice for the number of hinges where the hinges' locations are symmetrically fixed.

8.2. EXPERIMENTS' DETAILS AND STATISTICAL SIGNIFICANCE:

In this section, we explain the experimental conditions and all the parameters used for each experiment. Also, in order to make the results of Table 1 more interpretable, we perform a t-test (Kim, 2015) on all the error rates achieved in that experiment.

In section 4, the experiment corresponding to Table 1 are performed using six different architectures. Lenet 5 is used as it was introduced in LeCun et al. (1998) . It has two convolution layers followed by two MLPs that are connected to a softmax layer.

The MLP architecture which is only used for classifying MNIST dataset has three fully connected feed-forward layers with 258, 128, and 10 hidden units. A dropout of rate 0.3 is added in between each layer. We use the batch size of 32 and we train the MLP network for 50 epochs.

Pure CNN architecture which is only taking advantage of convolutional layers, was introduced in Springenberg et al. (2014) . All the specifications, including kernel size, stride, learning rate, batch size, and the number of epochs are taken from the main article.

ResNet18 architecture as a variant of residual networks (He et al., 2016) is taken from Napoletano et al. (2018) which has five residual convolution blocks, one MLP, and a softmax layer. The size of the convolution kernels, strides, padding, pooling size, pooling operation, learning rate, batch size, and epochs are taken as the defaults mentioned in Napoletano et al. (2018) .

:



Figures and Tables:

For any function f : [A, B] → R and ∈ R, ∃S ∈ N, where |f (x) -S-APL(x)| ≤ , assuming:

Figure 1: Shape of S-APL activation during training a simple network of MLPs on MNIST dataset.

Figure 2: Shape of S-APL during training a Lenet5 architecture on CIFAR-10 dataset.

Figure 3: Training loss trajectory for different S-APL initializations compared to fixed ReLU and leaky ReLU.

Figure 4: tSNE visualization of the pre-softmax layer's outputs. Left: a ReLU activated Lenet5 architecture trained on CIFAR-10. Right: same network trained with S-APL activation functions.

100 images are randomly chosen from CIFAR-10 test set to generate adversarial examples. We attacked each architecture five times and report the average number of successful attacks. The maximum number of iterations for all attacks is set to 40.

Table content: <table><tr><td>Model</td><td colspan="5">Activation one-pixel three-pixels five-pixels avg|Ztrue -Z adv |</td></tr><tr><td/><td>ReLU</td><td>61</td><td>88</td><td>92</td><td>0.721</td></tr><tr><td>Lenet5</td><td>APL Swish</td><td>50 63</td><td>71 85</td><td>84 90</td><td>0.570 0.542</td></tr><tr><td/><td>S-APL</td><td>29</td><td>58</td><td>67</td><td>0.401</td></tr><tr><td/><td>ReLU</td><td>29</td><td>69</td><td>80</td><td>0.411</td></tr><tr><td>Net in Net</td><td>APL Swish</td><td>24 29</td><td>50 71</td><td>69 77</td><td>0.400 0.446</td></tr><tr><td/><td>S-APL</td><td>18</td><td>39</td><td>58</td><td>0.281</td></tr><tr><td/><td>ReLU</td><td>16</td><td>53</td><td>64</td><td>0.483</td></tr><tr><td>Pure CNN</td><td>APL Swish</td><td>12 14</td><td>44 51</td><td>58 67</td><td>0.301 0.312</td></tr><tr><td/><td>S-APL</td><td>9</td><td>34</td><td>41</td><td>0.243</td></tr><tr><td/><td>ReLU</td><td>23</td><td>63</td><td>71</td><td>0.501</td></tr><tr><td>ResNet18</td><td>APL Swish</td><td>17 22</td><td>45 65</td><td>60 70</td><td>0.413 0.541</td></tr><tr><td/><td>S-APL</td><td>12</td><td>37</td><td>49</td><td>0.334</td></tr></table>

Similar to the black-box attack experiment, 100 images are randomly chosen from CIFAR-10 test set to generate adversarial examples. For each architecture, four different is used to FGSM attack. For each model, four different activation namely, ReLU, APL, Swish, and S-APL are tested. The settings of the networks and activations are thoroughly explained in the Appendix.

Table content: <table><tr><td>Model</td><td>Activation</td><td>= 0.02</td><td>= 0.04</td><td>= 0.06</td><td colspan="2">= 0.08 avg|Ztrue -Z adv |( = 0.04)</td></tr><tr><td/><td>ReLU</td><td>57</td><td>64</td><td>69</td><td>73</td><td>0.761</td></tr><tr><td>Lenet5</td><td>APL Swish</td><td>43 54</td><td>51 59</td><td>66 71</td><td>75 74</td><td>0.350 0.619</td></tr><tr><td/><td>S-APL</td><td>36</td><td>44</td><td>62</td><td>68</td><td>0.298</td></tr><tr><td/><td>ReLU</td><td>35</td><td>36</td><td>38</td><td>42</td><td>0.721</td></tr><tr><td>Pure CNN</td><td>APL Swish</td><td>30 36</td><td>34 40</td><td>38 42</td><td>42 44</td><td>0.509 0.613</td></tr><tr><td/><td>S-APL</td><td>25</td><td>28</td><td>33</td><td>38</td><td>0.388</td></tr><tr><td/><td>ReLU</td><td>36</td><td>38</td><td>39</td><td>43</td><td>0.691</td></tr><tr><td>ResNet18</td><td>APL Swish</td><td>33 36</td><td>36 39</td><td>39 40</td><td>42 42</td><td>0.445 0.511</td></tr><tr><td/><td>S-APL</td><td>27</td><td>29</td><td>35</td><td>40</td><td>0.391</td></tr></table>

Similar to the FGSM attack experiment, 100 images are randomly chosen from CIFAR-10 test set to generate adversarial examples. For each model, four different activation namely, ReLU, APL, Swish, and S-APL are tested. The settings of the networks and activations are thoroughly explained in the Appendix.

Table content: <table><tr><td>Model</td><td>Activation</td><td/><td>avg|Ztrue -Z adv |</td></tr><tr><td/><td>ReLU</td><td>83</td><td>0.871</td></tr><tr><td>Lenet5</td><td>APL Swish</td><td>79 81</td><td>0.551 0.570</td></tr><tr><td/><td>S-APL</td><td>72</td><td>0.392</td></tr><tr><td/><td>ReLU</td><td>58</td><td>0.611</td></tr><tr><td>Pure CNN</td><td>APL Swish</td><td>54 56</td><td>0.509 0.6.31</td></tr><tr><td/><td>S-APL</td><td>47</td><td>0.365</td></tr><tr><td/><td>ReLU</td><td>49</td><td>0.689</td></tr><tr><td>ResNet18</td><td>APL Swish</td><td>44 50</td><td>0.461 0.534</td></tr><tr><td/><td>S-APL</td><td>40</td><td>0.393</td></tr></table>

Three classifications tasks are performed with five different numbers of hinges for S-APL activation. The number of additional parameters due to the use of S-APL and training loss are compared below. The MLP architecture consists of three layers each with 256, 64, and 32 units. For CIFAR-10, Lenet5 is employed and for the IMDB comment dataset, a network with one bi-LSTM layer followed by two MLPs with 32 and 16 units is used.

Table content: <table><tr><td># of hinges</td><td>4</td><td>6</td><td>8</td><td>10</td><td>12</td></tr><tr><td>Error rate</td><td/><td/><td/><td/><td/></tr><tr><td>MNIST</td><td>1.73-1.61</td><td>1.70-1.57</td><td>1.61-1.57</td><td>1.53-1.53</td><td>1.55-1.59</td></tr><tr><td>CIFAR-10</td><td colspan="5">30.1-29.4 29.61-28.81 29.01-28.44 29.01-28.33 28.92-28.03</td></tr><tr><td>IMDB</td><td>6.11-6.35</td><td>5.87-5.91</td><td>5.41-6.12</td><td>5.47-6.32</td><td>5.41-7.24</td></tr><tr><td># of additional params</td><td/><td/><td/><td/><td/></tr><tr><td>MNIST</td><td>12-1408</td><td>18-2112</td><td>24-2816</td><td>30-3520</td><td>36-4224</td></tr><tr><td>CIFAR-10</td><td>16-&gt;75k</td><td>24-&gt;120k</td><td>32-&gt;150k</td><td>40-&gt;180k</td><td>48-&gt;225k</td></tr><tr><td>IMDB</td><td>12-384</td><td>18-576</td><td>24-768</td><td>30-960</td><td>36-1152</td></tr></table>

Per-epoch training time is reported in seconds. The same number is calculated for Tanh and Maxout units (with six features). S-APL is faster than Maxout and is slightly slower than Tanh unit. All models are trained using NVIDIA TITAN V GPU with 12036MiB memory and 850MHz.

Table content: <table><tr><td>Activation</td><td/><td/><td>S-APL</td><td/><td colspan="2">Tanh Maxout</td></tr><tr><td/><td>4</td><td>6</td><td>8</td><td>10 12</td><td/></tr><tr><td>MNIST</td><td>5</td><td>7</td><td colspan="2">10 12 13</td><td>6</td><td>12</td></tr><tr><td colspan="5">CIFAR-10 18 21 24 27 30</td><td>21</td><td>31</td></tr><tr><td>IMDB</td><td colspan="4">11 14 14 15 16</td><td>12</td><td>18</td></tr></table>

Peer review (iclr2020):
This paper proposes a learnable piece-wise linear activation unit whose hinges are placed symmetrically. It gives a proof on the universality of the proposed unit on a certain condition. The superiority of the method is empirically shown. The change of the activation during training is analyzed and insight on the behavior is provided. The robustness to adversarial attacks is also empirically examined. This paper discusses a very basic component of neural network models: activation function. Thus, it should be of interest to many researchers. The proposed method is simple and seems easy to use in real settings. A number of experiments are conducted to validate the method and the results look promising. The experiments in Section 5 is particularly interesting. It might give some hints for the following studies. However, there are several things to be addressed for acceptance. 1) What is actually proposed is not very clear. S-APL is formulated in Equation 2. However, there are some discussion after that which changes or restricts the equation. For example, it seems that b_i^s^+ = b_i^s^- is assumed throughout the paper. In that case, it should be just reflected in Equation 2. In the third paragraph of Section 3.2, it is mentioned that h_i^s(x) = h_i^s(-x) with b_i^s^+ = b_i^s^-. However, it should also assume that a^s^+ = a^s^-. From the experiments. apparently, a^s^+ = a^s^- is not assumed. It seems that the method has symmetry only for the hinge locations. In the first paragraph of Section 3.2, it is implied that parameters are shared across layers. It is not very clear what is shared and what is not. Please make that part clear. It will make it easier to understand the experimental settings, too. 2) Theorem 3.1 does not seem to prove the approximation ability of S-APL. It is clear that g(x, S) can represent arbitrary h(x, S), but I am not sure if it is clear that h(x, S) can represent arbitrary g(x, S). It should also depend on the conditions on a^s^+, a^s^-, b_i^s^+, b_i^s^-. I think it needs to prove that h(x, S) can approximate arbitrary piecewise linear function (i.e., g(x, S)) if you want to prove the approximation ability of h(x, S). Equation 4 seems to assume that all intervals are the same (i.e., ?i, B_i - A_i = (B-A) / S). It should be stated explicitly. This relates to the problem 1). I may not understand some important aspect. I am happy to be corrected. 3) Experimental conditions are not clear. Please cite the papers which describe the architecture of the models used in the experiments. The effectiveness of the proposed method should depend on the network architecture and it is importable to be able to see the details of the models. 4) On the sensitivity of optimization on the initial value. It is interesting to see that *fixed trained S-APL* is not comparable with *S-APL positive*. If the hypothesis in the paper is correct, it is natural to assume that *fixed trained S-APL* also has some issue on training. It would be interesting to see experimental results with *initialized with trained-S-APL* and *S-APL positive with non-zero initial value*. It is a bit weird to observe that *S-APL positive* never becomes non-zero for x < 0. 5) Comparison results with other activation units in Section 6. The proposed method is compared only with ReLU. It is important to see comparisons with other activations such as the plain APL. Some other minor comments: It is quite interesting that objects are actually modified for adversarial attack for the proposed method in Figure 5. It would be interesting to have some consideration on it.