Symmetric-APL Activations: Training Insights and Robustness to Adversarial Attacks
Authors: Mohammadamin Tavakoli
Forest Agostinelli
Pierre Baldi
Venue:  (2020)
Keywords: Activation function
Adaptive
Training
Robustness
Adversarial attack

Abstract: Deep neural networks with learnable activation functions have shown superior performance over deep neural networks with fixed activation functions for many different problems. The adaptability of learnable activation functions adds expressive power to the model which results in better performance. Here, we propose a new learnable activation function based on Adaptive Piecewise Linear units (APL), which 1) gives equal expressive power to both the positive and negative halves on the input space and 2) is able to approximate any zero-centered continuous non-linearity in a closed interval. We investigate how the shape of the Symmetric-APL function changes during training and perform ablation studies to gain insight into the reason behind these changes. We hypothesize that these activation functions go through two distinct stages: 1) adding gradient information and 2) adding expressive power. Finally, we show that the use of Symmetric-APL activations can significantly increase the robustness of deep neural networks to adversarial attacks. Our experiments on both black-box and open-box adversarial attacks show that commonly-used architectures, namely Lenet, Network-in-Network, and ResNet-18 can be up to 51% more resistant to adversarial fooling by only using the proposed activation functions instead of ReLUs.

Under review as a conference paper at ICLR 2020
SYMMETRIC-APL ACTIVATIONS: TRAINING IN- SIGHTS AND ROBUSTNESS TO ADVERSARIAL AT- TACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks with learnable activation functions have shown superior per- formance over deep neural networks with fixed activation functions for many dif- ferent problems. The adaptability of learnable activation functions adds expressive power to the model which results in better performance. Here, we propose a new Icarnable activation function based on Adaptive Piecewise Linear units (APL), which gives equal expressive power both the positive and negative halves on the input space and 2) is able to approximate any zero-centered continuous non-linearity a closed interval. We investigate how the shape of the Symmetric- APL function changes during training and perform ablation studies to gain insight into the reason behind these changes. We hypothesize that these activation func- tions go through two distinct stages: 1) adding gradient information and 2) adding activations
expressive power. Finally, we show that the use of Symmetric- can significantly increase the robustness of deep neural networks adversarial attacks. Our experiments on both black-box and open-hox adversarial attacks architectures, namely Lenet, Network-in-Netweer and
show that commonly-un ResNet-18 can be up to 51% more resistant to adversarial fooling by only using the proposed activation functions instead of ReLUs.
INTRODUCTION
Deep neural networks (DNNs) are series of linear transformations followed by point-wise non- linear functions. These non-linear functions also referred as activation functions. Activation functions directly influence the training dynamics and, therefore, affect the final performance of the DNN. For instance, activations such as Sigmoid and Tanh suffer from what is known as the "van- ishing gradient" problem, when the inputs are in the saturation regions of these functions. This can adversely affect network performance and cause slow convergence. Rectified linear units (ReLU) (Nair& & Hinton 2010) have been shown to perform better than Sigmoid and Tanh in many scenar- ios. Instead using a fixed activation function, one can learn a parameterized activation function to add expressive power to the model. Towards this goal, significant efforts have been made to intro- duce different learnable activation functions (Agostinelli et al.) 2014 He et al, 2015 Ramachandran ct al. 2017 Jin et al. 2016 Li et al. 2016). Although most of these adaptive activation functions outperform fixed activation functions in multiple tasks, there is very little insight into why.
In this paper, we propose the Symmetric Adaptive Piecewise Linear unit (Symmetric-APL: pro- nounced as Symmetric apple: aep.al as a new adaptive activation function which builds upon the adaptive piecewise linear unit (APL) (Agostinelli multiple hinges which are connected by linear pieces. The S-APL activation is designed to give the same expressive power both the positive and negative sides of the x-axis. We show that S- API. can approximate any continuous and zero-centered function, which encompasses many types of learned and fixed activations functions. Using this flexible learned activation function, we can then analyze how the shape changes during training and use ablation studies to investigate the role that these changes play in optimization. This helps us better understand learned activation functions and explain why they often provide increases in performance.
al. 2014). Similar APL units, S-APL has
Finally, we evaluate the robustness of S-APL activated networks to adversarial attacks. Experiments done using both black-box and open-box adversarial attacks where observe S-API. equipped

---

Under review as a conference paper at ICLR 2020
networks robust to attacks than their ReLU counterparts. Experiments show that the bustness improves against differential evolution attacks (Su et al. 2019b) by (up to) 51 % on Lenet, 47% on ResNet-18 18 (He et al., (2166) 43% pure CNN, and 38% on Net in Net (Lin et al. 2013). For open-box adversarial attacks, the robustness also improves against FGSM and CW-12 (Good- fellow et al.] (014) attack by (up to) 36%, 28%, and 25% for Lenet, pure CNN, and ResNet-18 architectures, respectively using CIFAR-10 dataset.
2 RELATED WORKS
ReLU and its variants such as Leaky- ReLU Maass et al. 2013), P-ReLU (He et al. 2015). exponen- tial linear units (ELU) (Clevert et al. 2013). and scaled exponential linear units (SELU) Klambbuurr et al. 2017) are believed to solve the vanishing gradient problem and some are shown to better nor- malize the network. By changing the negative side of x-axis, these variants push the activations of each unit closer to zero and accelerate learning.
Some of the pioncering attempts to learn activations neural network can be found in Poli 1996), Weingaertner et al. (2002), and Khan et al. (2013) where the authors proposed novel approaches to learning the best activation function per neuron among a pool of candidate activations by employing genetic and evolutionary algorithms. Maxout (Goodfellow et al., (203) has been introduced as an activation function aimed at enhancing the model averaging properties of dropout (Srivastava al.) 2014). However, not only is it limited to approximating convex functions, but it also requires significant increase parameters. Neural architecture search (Ramachandran et al.] (017) uses a combination of cxhaustive and reinforcement learning based search. Although the authors discov- ered multiple novel activation functions, which f(z) = x sigmoid( (3xz) performed slightly better than ReLU, the search space of activation functions is limited. S-ReLU (Jin al. 2016) was proposed as an adaptive activation function which can mimic both convex and non-convex func- tions.
is worth mentioning that, in Lin et al. 2013), the authors proposed the network in network ap- proach where they replace activation functions in convolutional layers with small multi-layer per- ceptrons. Theoretically, due to Universal Approximation Theorem, this is the most expressive acti- vation: however, requires many more parameters.
ADAPTIVE PIECEWISE ACTIVATION FUNCTION
3.1 ADAPTIVE PIECEWISE LINEAR UNITS
Agostinelli et al. (2014) proposed an adaptive activation function which consists of multiple linear pieces. The formulation of this adaptive piecewise linear (APL) activation function is shown in Equation]
s
h₂(2) = max(0, (0, x) +
Σ
a mar - b,)
(I)
Where h4 the non-linear transformation for the hidden unit i and S is a hyperparameter that determines the number of hinges. a, is the slope of the linear piece and 6; is the location of the hinge.
However, APL units are not cero-centered as they can have a non-zero output for a input of zero. This behavior provides no apparent beneficial purpose. Furthermore, APL units can only represent piecewise linear functions whose output (xx) for x: > for some scalar This significantly restricts the class of piecewise linear functions that API. can express.
3.2 SYMMETRIC ADAPTIVE PIECEWISE LINEAR UNITS (APL: 'æpp.l)
To overcome the shortcomings of APL activation function, define new adaptive piecewise linear activation function that is zero-centered and gives the same expressive power to the positive and negative half of the input space. Since S-APL shares the variables af bf a² and be among all
2

---

Under review as a conference paper at ICLR 2020
the neurons of layer (e.i., (x, S) does not depend on i), it requires fewer parameters in compare to the plain APL. We formulate the activation h2 of the hidden unit i as the summation of symmetric hinge-shaped pieces.
$ () = Σ af maz(0, -9,)
S
Σ a mar(0, (0, -x b-)
Similar to APL units, is hyperparameter which determines the number of hinges (on each side of the x-axis). a$_ and af control the slope of linear pieces both the positive and negative sides of the x-axis, while b%. 6 E [0, + determine the location of hinge s. The total number of additional parameters is 4S for single layer. Because our networks are trained with batch normalization (loffe & Szegedy 2015) before each S-APL layer, the distribution of input to the S-APL function will follow that of normal distribution. Therefore, we fix the hinges be a pre-determined number of standard deviations away from the mean, decreasing the number of parameters to 28.
Based on Equation 2] S-APL activation is zero-centered which leads to more stable statistical be- havior during training. In case of using fixed hinges' locations for APL, due increasing the number of parameters, S-APL offers more flexibility than the normal APL. However, this increase of O(constant) which is negligible. With the assumption of af a²_ and by be since hx(z) (-z). S-APL allows neurons to activate symmetrically. As it is mentioned in Zhao & Griffin (2016). networks with symmetric activation functions are more robust adversarial fooling.
We will delve into this property in later sections.
S-APL can better handle the limits. It could be inferred from the definition that, in contrast to normal APL, increasing the absolute value of the input will also increase the number of terms affecting the (z). However, normal APL requires the rightmost linear section in all the component functions to have unit slope and bias 0, which is not appropriate constraint and undermines its representation ability.
The following theorem shows that S-APL can approximate any non-linear, zero-centered, and Lips- chitz continuous function in a closed interval of real numbers. It is worth mentioning that APL was not flexible enough to do so.
Theorem 3.1 For any function [A, B] + and IR, 3S N, where 15(x) S-APL(x) S. assuming:
and B are finite real numbers.
f is M-Lipschitz continuous.
Proof: We start by dividing the interval [A, B) into S equal sub-intervals [Ai, Bz). In other words,
we put S equally distanced hinges in the interval [A, B). So the length of each sub-interval will
be Byd. We start approximating f from the left most sub-interval which is [A, A + g The approximation of f in that sub subiinterval will be a line which connects f(A) to f(A + B-A) Due to
M- M-Lipschitz continuity of for any x E [A, A + B-
- (((A)-((A+ B M(B-A [If(A+ B f(A) + M(B
f(A) < f(A + B-A
lf(z) - S-APL(x) <
(3)
f(A) 2 f(A B-A), )
Again, due to M-Lipschitz continuity Jf we know that:
B A
M(B A)
IS(A) S(A
(4)
S
S
Using Equation we can re-write the inequality 3 as follows:
15(x) S-APL(x) S 2M - A)
(5)
S
So, by choosing 8> 2M(B-A A)) we satisfy that (f(x) S-APL(x) S
3

---

Under review as a conference paper at ICLR 2020
in order complete the proof, need show that this approximation be applied to other sub-ini intervals of [A, B] as well. We do so by induction as follows: Suppose that the approximation is done for sub-intervals [Ai, Bil..... [A;, Bi), now prove that for sub-intervan [Ac+1, BB+1] we can do the approximation which is connecting I [A++1) to f(B++)). The slope of S-API. in the sub- interval [A,, B,) is Σi d'. or Σi a'_ (depending on the sign of the sub sub-interval). However, the slope of S-APL. the sub-interval [As+1, B1+1] will be either Ei+1 a' or Ei+1 a²_. In both cases, the extra term 5+1 or a'_1 is not fixed and can change the slop to any arbitrary value. This fact plus the assumption of continuity of S-APL, guarantee that f(A;-1) can be connected f(B++1) which was our proposed approximation. As we showed the approximation for the left most sub-interval, the induction is complete.
4 COMPARISON TO COMMONLY USED ACTIVATION FUNCTIONS
In order to show that S-APL unit is beneficial for deep neural networks, compare it with well- known activation functions in different architectures. We employed five different neural networks trained three different datasets, MNIST, CIFAR-10, and CIFAR-100. We used S = 4 and set the hinges at positions
2.5, 2, 1, 0, +1, +2, +2.5 (0 considered two hinges). Slope of af initialized to 1 and the remaining slopes are initiazlied to O. With this initialization, the initialized S-APL mimics the shape of ReLU.
The results of the experiments are shown in Table validation accuracy almost architectures.
1 The table shows that S-API. leads to better
Table 1: ReLU, teaky-ReLUs PReLU, tanh, sigmoid, ELU, maxout (nine features), swish: u.sigmoid(BB) and S-APL activations are compared in three different tasks. The performance of S-APL in comparison to the best of the other activations. For the sake of brevity, D-A refers to Data Augmentation. The values the tables are error-rates and are reported percentage.
MNIST
CIFAR-10
CIFAR-100
Activation
D-A
D-A
Lenet5 (best: PReLU)
1.11
31.93
27.42
46.31
45.39
Lenet5 (S-APL)
1.03
30.83
27.02
45.74
44.81
MLP (best: swish)
1.55
MLP (S-APL)
1.71
pure CNN (best: maxout)
12.01
10.92
35.02
35.33
pure CNN (S-APL)
11.81
10.32
34.42
34.10
ResNet-18 (best: PReLU)
9.41
9.02
20.14
19.60
ResNet-18 (S-APL)
8.93
8.07
19.24
18.71
EffectiveNet BO (best: ReLU)
7.03
6.43
15.91
15.65
EffectiveNet (S-APL)
6.54
5.93
15.01
15.31
Net-in-Net(ReLU)
10.41
8,81
33.72
31.63
Net-in-Net(APL)
9.59
7.51
32.40
30.83
Nct-in-Ncc(S-APL)
9.41
7.20
32.12
30.33
5 How S-APL CHANGES DURING TRAINING AND POSSIBLE REASONS WHY
Figures I [nd [andw]show how the shape of S-APL changes for each layer during training. From these figures, we can see that during the early stages of training, the shape of S-APL looks similar that
of leaky ReLU: having negative output for x < 0 and positive output for x 2 0. During the later stages of training, the activation function has positive output for both < 0 and x 0. In addition, the slope of the output decreases the magnitude of increases.
We hypothesize that there are two distinct stages during the optimization of these learned activation functions. In the first stage, the function returns negative values for 0 and in the second stage,
4

---

Under review as a conference paper at ICLR 2020
the function returns positive values for < O. To investigate this hypothesis, we train S-APL under two different conditions where we: 1) force the output to only be negative for æ < (S-APL- negative) and 2) force the output only be positive for < 0 (S-APL-positive), We accomplish this by taking the absolute value of all a* and multiplying by +1 1.
Figure shows that, although S-APL-oositive has the ability to mimic the final learned shape of the S-API. function, it barely deviates from the ReLU initialization. This shows that the ability to give a negative output for negative inputs is crucial for S-APL.. Figure[]shooee initially go negative in the 4th layer and then returns to the ReLU shape. To gain further insight, compare the loss of these activation functions Figure 3 We see that S-APL has lower loss compared to S-APL-positive and S-APL-negative. Furthermore, S-APL-negative has lower loss compared S-APL-positive which has lower loss than ReLU.
that S-APL--negativ does
These results lead us believe that it is possible that the first stage, S-APL--ngative, adds gradient information by giving negative output for negative inputs. It is possible that S-APL-ppoittive is not capable of doing so because giving positive outputs for negative inputs creates an identifiability issue. In other words, outputs for ı 0 and 2 0 look the same. However, we can see that this property useful for the S-APL function the later stages of training. It possible that this is because the DNN has adapted its parameters such that outputs on both the negative and positive sides capture similar concept, adding expressive power to the model. Furthermore, one cannot achieve the final S-APL shape without the first stage of having negative outputs for negative inputs. To conclude, appears that the S-API. units first go through two stages: 1) adding gradient information and 2) adding expressive power and that the second stage depends on the first stage. This information can be used inform the design of learnable activation functions and even DNN optimization techniques.
5000
2000
ST 10000
10000 0C 100000
1st S APL layer
2nd APL layer
3rd S APL layer
Figure 1: Shape of S-APL activation during training a simple network of MLPs on MNIST dataset.
ROBUSTNESS TO ADVERSARIAL ATTACKS
DNNs have been shown to be vulnerable to many types of adversarial attacks (Szegedy et al.) 2013) Goodfellow et al. 2014). Research suggests that activation functions are major cause of this vulnerability (Zantedeschi et al. 2017). Zhang al. (2018) bounds given activation function with a few linear and quadratic functions and allows it to tackle general activation functions. This adds up with applying different activation for each neuron the resulting network shows efficiency and robustness to adversarial foolings. Wang et al. (2018) proposed a data-dependent activation function and empirically shows its robustness to both black-box and gradient-based haaeed adversarial attacks. Other
5

---

Under review as a conference paper at ICLR 2020
Sditt SArL-nggaiiee SoPtipoiiivee
opoch:
EpoCH 25
6900
epoch 75
apoch: 100
1st S-APL layer
2nd S-APL layer
3rd S-APL layer
4th S APL layer
Figure Shape of S-APL during training Lenet5 architecture on CIFAR- 10 dataset.
20
S-APL negative
S-APL relu init
fixed trained S-APL
15
S-APL positive
fixed leaky ReLU
fixed ReLU
12-
trained S-APL init
Figure 3: Training loss trajectory for different S-APL initializations compared to fixed ReLU and leaky ReLU.
studies such Rakin al. (2018). Dhillon al.) (2018), and [Son al. (2018) focused on other additional properties of activation such as quantization and pruning and showed they can improve the robustness of DNNs against adversarial examples.
Recently, Zhao & Griffin (2016) theoretically showed that DNNs with symmetric activations less likely to get fooled. The authors proved that "symmetric units suppress unusual signals of exceptional magnitude which result in robustness adversarial fooling and higher expressibility." Due to the symmetric convergence of S-APL units and also their hinges' locations (b. b which result in hx=)= h(-2), S-APL units are capable of increasing the robustness of DNNs against adversarial attacks.
In this section. relying on the properties of S-APL such data dependency, piecewise linearity, non- injective behavior. better handling of extreme values, and most importantly, symmetric shape, we show that adding S-APL DNNs greatly improves the robustness against adversarial attacks. This claim is verified through wide range of experiments with the CIFAR-10 10 dataset under both black- box and open-box methods, including one-pixel-attack and Fast Gradient Sign Method.
Figure 4 provides an intuition for the robustness of S-APL activated network in comparison ReLU activated ones. For each of the two networks, we take 100 random samples of frog and ship images and visualize the pre-softmax representations using 1SNE visualization (Maaten & Hinton 2008 in

---

Under review as a conference paper at ICLR 2020
Figure As it is depicted, for S-APL activated networks, the two classes are better separated. We suspect that red samples such as those that are lying on the outer circle in the left plot of Figure 4 the manifold of the data where the network is more susceptible. As one can
might be somewhere see, there no red point on the outer circle of the right plot which might be showing the robustness of S-APL networks. For other samples the inner circle and the cloud next to circles, can better separation by the S-APL network.
Figure 4: iSNE visualization of the pre-softmax layer's outputs. Left: a ReLU activated Lenet5 architecture trained on CIFAR-10, Right: same network trained with S-APL activation functions.
6.1 BLACK-BOX ADVERSARIAL ATTACK
For black-box attacks against a DNN, we assume the adversary has no information about the struc- ture parameters of the DNN and does not have access to any large training dataset. The adversary's s only capability is to observe labels assigned by the DNN for chosen inputs in a manner analog to cryptographic oracle. A successful attacking technique of this type is Su et al. (2019a) which is based on differential evolution. We can iteratively generate adversarial images to try to minimize the confidence (probability) of the network's 's classification.
The process starts with random modifications of a few pixels to generate adversarial examples. At each step, one feeds several adversarial images the DNN and only observes the output as the probability of different classes. Examples that lowered the confidence of the true class will be kept generate the next generation of adversaries. The next generation modifications are then evolved through a certain mutation scheme to generate new adversarial images. By repeating these steps for a few iterations, the adversarial modifications generate more and more misleading images. The last step returns an adversarial modification that reduces the confidence of the true class the most. This means that the confidence of the true class would be reduced so much that new incorrect category now has the highest classification confidence. This technique is called untargeted attack since don't specify the labels the modified images.
In the following experiment, modify one, three, and five pixels of images to generate adversarial examples. The mutation scheme we used for this experiment as follows:
ot 24, +0.((1), + 2ty)
Where 71, 72, and are three non-equal random indices of the modifications at level I. af+1 will be an element of a new candidate modification.
To evaluate the effect of S-APL activation on the robustness of DNNs, employ commonly- used architectures, namely, Lenet5, Network-in-Networkk (Lin et al. 2013), pure CNN (ours), and ResNet-18 18 (He al. 2016). Each architecture trained with ReLU, APL, and Swish activation functions and once with S-APL activation functions. The results in Table 2 show that LenetS and ResNet-18 architectures are 51% and 47.8% more resistant to back- box adversarial attacks. As authors in Zhao & Griffin (206)) theoretically proved, that one can be robust to adversarial examples with symmetric behavior.
After observing adversarial samples which deceiving both networks, found that S-APL activated network still assigns high confidence to the true labels of the perturbed images. other words, the difference between the confidence of the true label and the confidence of adversarially
7

---

Under review as a conference paper at ICLR 2020
Table 100 images are randomly chosen from CIFAR-IO set to generate adversarial examples. We attacked each architecture five times and report the average number of successful attacks. The maximum number of iterations for all attacks set to 40.
Model
Activation
one-pixel
three-pixels
five-pixels
evgl Zerue Zarte
ReLU
61
88
92
0.721
APL
50
71
84
0.570
Lenet5
Swish
63
85
90
0.542
S-APL
29
58
0.401
ReLU
29
69
80
0.411
24
50
0.400
Net in Net
Swish
29
71
77
0,446
S-APL
18
39
58
0.281
ReL.U
16
53
64
0,483
APL
12
44
58
0.301
Pure CNN
Swish
14
51
67
0.312
S-APL
9
34
0,243
ReLU
23
63
71
0.501
17
45
60
0.413
ResNet18
Swish
22
65
70
0,541
S-APL
12
37
49
0.334
wrong label is much smaller S-APL network than networks that are activated with ReLU, APL, Jadversarial label over all or Swish. More precisely, we measure the average of (Z(x') tru label - Z((''aatvrariilll
adversarial samples where each network is fooled. (.) is the post-softmax output and x' is the adversarial sample. The avg( Z (z')) Jtue label ZZz') adversarial labei)) ) for each model is also included in Table Due the reported averages, we can conclude that S-APL is much more robust against being adversarially deceived.
6.2 OPEN-BOX ADVERSARIAL ATTACK
To further explore the robustness of S-APL activated networks, in this section, we consider two of the popular benchmarks of open-box adversarial attacks: Fast Gradient Sign Method (FGSM) Goodffllow et al.) 2014) and CW attack (Carlini & Wagner 2017). For both attacking strategy, consider three different architectures and compare the rate of successful attacks for each of the networks with ReI.U, API,, or Swich activations versus S-APL. activation functions. The dataset and architectures are same as those were used for black-box adversarial attacks.
6.2.1 FGSM
Fast Gradient Sign Method generates an adversarial image x' from the original image x by maxi- mizing the loss L(z', y). where is the true label of the image z. This maximization problem is subjected to ||. z' |loo S 6 where is considered as the attack strength. Approximating with the first term of Taylor expansion, we have:
L( v) L(x, v) (T) x¹)
So the adversarial image x' would be:
s' = = x e.sign VxL(x,))
(8)
Where the e considered as the power the attack. This form FGSM considered to be an untargeted attack where there is pre-specified label that x' should be classified as. This method only yields an adversary x' which will not be classified as y.
To how S-APL increases the robustness against FGSM attack, employ three architectures all trained to classify CIFAR- 10 dataset. For different range of the results are summarized in Table
3
8

---

Under review as a conference paper at ICLR 2020
Table Similar the black-box attack experiment, 100 images randomly chosen from CIFAR- 10 test set to generate adversarial examples. For each architecture, four different is used to FGSM attack. For each model, four different activation namely, ReLU, APL, Swish, and S-APL are tested. The settings of the networks and activations are thoroughly explained in the Appendix.
Model
Activation
c 0.02
0.04
0.06
0.08
wvg Zerue Zadu 0.04)
ReLU
64
69
0.761
APL
66
0.350
Lenet5
Swish
59
71
0.619
S-APL
36
44
62
68
0.298
ReLU
35
36
38
42
0.721
APL
30
34
38
42
0.509
Pure CNN
Swish
36
40
42
44
0.613
S-APL
28
33
0,388
ReL.U
38
39
43
0.691
API.
36
39
0.445
ResNet18
Swish
36
39
40
42
0.511
S-APL
27
29
35
40
0.391
Table 4: Similar to the FGSM attack experiment, 100 images are randomly chosen from CIFAR-I0 10 test set generate adversarial examples. For each model, four different activation namely, ReLU, APL, Swish, and S-APL are tested. The settings of the networks and activations are thoroughly explained in the Appendix.
Model
Activation
augl Zerue Zudtol
ReLU
83
0.871
APL
79
0.551
Lenet5
Swish
81
0.570
S-APL
0.392
ReLU
58
0.611
APL.
0.509
Pure CNN
Swish
56
0.6.31
S-APL
47
0.365
ReLU
49
0.689
APL
44
0.461
ResNet18
Swish
50
0.534
S-APL
0.393
In the best case, for Lenet architecture and 0.02, S-APL activated network is 36% robust than the ReLU activated one. The robustness also improves by 28% and 25% for pure CNN and ResNet-18 respectively. We can see the overall increase in the robustness over all networks and values. However, one can conclude that for smaller values of the robustness of S-APL activated network is higher in comparison ReLU activated networks. As we increase the attack strength (c), the robustness of S-APL networks gets closer to the robustness of ReLU activated networks.
6.2.2 CW-L2
We showed that replacing ReLU with S-API. activation units can greatly improve the robustness of DNNs against FGSM attack. To further investigate the robustness that S-APL is adding to the network, we are using a more powerful open-box box adversarial attack.
Another open-box adversarial attack which is generally more powerful than FGSM was introduced in Carlini & Wagner (201)). For given pair of image and label y, this technique is trying to find the minimum perturbation 6, so that the perturbed image x' is classified as f * y. Considering Ta norm as the goal of minimization, this perturbation minimization problem be formulated as

---

Under review as a conference paper at ICLR 2020
follows:
vt * min|811 subject to f(x+0) =t, x x+
[0,
(9)
Or equally, ease the satisfaction of equality, Equation 9 can be rephrased as min| | c.g(x +6)
where g(x) = max(mar, (logit(x); ()) c Lagrange multipliyer, and (o) the pre-softmax vector for the input
Here, we examine the robustness S-APL activated networks against CW-L2 attack. As is shown in the table[4] CW-L2 attack is more powerful than FGSM. Although the S-APL activated network is less robust to CW-122 attack than to FGSM attack, is still showing more robustness in comparison to ReL.U, API., and Swish activated networks. Similar to black-box attack, the metric avg) trrue Zodul is calculated for successfully attacked images. The smaller values of aegl Zerue Zadel for S-APL activated networks in both open-box attacks shows that S-APL activated networks are more resistant to being fooled.
CONCLUSION AND FUTURE WORKS
We extend the idea of piecewise linear activation functions by introducing S-APL as learnable and potentially symmetric activation function. We investigate the properties of S-APL. and their evolu- tion during training to understand how these functions accelerate learning and increase expressive power in DNNs. Finally, show that S-APL networks show more robustness adversarial at- tacks than ReLU networks. One can potentially design a hybrid defense method that combines other defense techniques and S-APL units.
REFERENCES
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation func- tions to improve deep neural networks, 2014.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Djork-Ami Amn Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus), 2015.
Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial de- fense, 2018.
Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint
2013.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi: 10.11099icc.22155.222 URL http://ddd doi.
org/10.1109/I00a 10110//ICV... www.c........... 123
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by 2015.
reducing internal covariate shift. arXiv preprint
Xiaojie Jin, Chunyan Xu. Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s- s-shaped rectified linear activation units. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
10

---

Under review as a conference paper at ICLR 2020
Maryam Mahsal Khan, Arbab Masood Ahmad, Gul Muhammad Khan, and Julian F Miller. Fast learning neural networks using cartesian genetic programming. Neurocomputinng, 121:274-289, 2013.
Tae Kyun Kim. T test as a parametric statistic. Korean journal of anesthesiology, 68(6):540, 2015.
Günttrr Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL htttp://wwk CS www.storoooooooooooo
wtw.ottttoo.oooooooooooooooo serrontt.eduu/kkii//ifaar html
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URLhtt:////ann.
lecun. comm/exbbmniise st/
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEF, 86(1 ((111)2227-22222 1998.
Hongyang Li, Wanli Ouyang, and Xiaogang Wang. Multi-bias non-linear activation in deep neural networks. In International conference on machine learning, pp. 221-229, 2016.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
Andrew I.. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/ /wwwwaacwwbb. org/ wttg//nthhlogyyPP11 1015
Andrew Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. icml, volume 30, pp. 3, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 97Nv))2577--2655 2008.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Paolo Napoletano, Flavio Piccoli, and Raimondo Schettini. Anomaly detection in nanofibrous materials by cnn-based self-similaariy. Sensors (Basel, Switzerland), 18, 2018. doi: 10.3390/s1801022
Riccardo Poli. Parallel distributed genetic programming. University of Birmingham, Cognitive Science Research Centre, 1996.
Adnan Siraj Rakin, Jinfeng Yi, Boqing Gong, and Deliang Fan. Defend deep neural networks against adversarial examples fixed anddynamic quantized activation functions, 2018.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv: 1710.0591,, 2017.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. arXiv preprint urXiv: 2017. URL http:
Sibo Song, Yueru Chen, Ngai-Man Cheung, and C. C. Jay Kuo. Defense against adversarial attacks with saak transform, 2018.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 441266806 2014.

---

Under review as a conference paper at ICLR 2020
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions Evolutionary Computattinn, pp. 1-1, 2019a. ISSN 1941- 0026. doi: 10. 1e.1099eecc221192899000
URL http:/ /ictp://dx.oo htctp::/dx..o..orggg/.oo www.:oo..oooooooo
2019 2890858
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 2019b.
Christian Szegedy, Wojciech Zaremha, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arxiv preprint 190. 2019.
Bao Wang, Alex T. Lin. Zuoqiang Shi, Wei Zhu, Penghang Yin, Andrea L. Bertozzi, and Stanley J. Osher. Adversarial defense via data dependent activation function and total variation minimiza- tion, 2018.
Daniel Weingaertner, Victor K Tatai, Ricardo R Gudwin, and Fernando Von Zuben. Hierarchical evolution of heterogeneous neural networks. In Proceedings of the 2002 Congress on Evolution- ary Computation. CEC'02 (Cat. No. 02TH8600), volume 2, pp. 1775-1780. IEEE, 2002.
Valentina Zantedeschi, Maria-Irina Nicolae, and Ambrish Rawat. Efficient defenses against adver- sarial attacks, In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 39-49. ACM, 2017.
Huan Zhang, Tsui-Wei Weng, Pin- Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions, 2018.
Qiyang Zhao and Lewis D Griffin. Suppressing the unusual: towards robust cnns using symmetric activation functions, 2016.
APPENDIX
8.1 NUMBER OF HINGES AND THE SYMMETRY OF S-APL.
In this section, we perform a variety of experiments to find the best setting for S-APL activation in terms of complexity and expressibility.
Before going to experiments, it is necessary to mention that we used batch normalization layer before every S-APL layer in our experiments. This means that the input of the S-APL activation is unit Gaussian. Relying this fact and the experiments done in Agostinelll et al. (2014) we chose fixed and symmetric set of hinges on toa +2, +22.5, +:3a, and This will reduce the number of S-APL's parameters by factor of 2 which makes it much less computationally expensive.
S-APL. Due to Theorem 3.1 greater S values increase the expressive power of the S-APL which results in better fitting the training data. We tried S 2,, 3, 4, 6). (which means 4, 6, 8, 10, and 12 hinges) with fixed hinges' locations on MNIST (LeCun Cortes 2010), CIFAR-10 (Krizhevsky al.) and binary classification of IMDB review dataset (Maas et al. 2 to cover MLP, convolution, and recurrent architectures respectively. Each network trained with two types of S-APL activation;
First, we assess the effect of on the performance
shared S-APL: shared unit among all neurons of layer and 2) An independent S-APL unit for each neuron of layer. As it is summarized Table 5 in all cases for 2 the performance is not decreasing significantly.
On the other hand, due the increase in the number of parameters of S-APL, the unit becomes more computationally expensive. Table we compare the per-epoch training run-time for different number of hinges of shared S-APL.
12

---

Under review as a conference paper at ICLR 2020
Table 5: Three classifications tasks are performed with five different numbers of hinges for S-APL activation. The number of additional parameters due to the use of S-APL and training loss are compared below. The MLP architecture consists of three layers each with 256, 64, and 32 units. For CIFAR-10, 10, Lenet5 is employed and for the IMDB comment dataset, a network with one bi-LSTM layer followed by two MLPs with 32 and 16 units used.
of hinges
4
6
10
12
Error rate
MNIST
1.73-1.61
1.70-1.57
1.61-1.57
1,53-1.53
1.55-1.59
CIFAR-10
30.1-29.4
29.61-28.81
29.01-28.44
29.01-28.33
28.92-28.03
IMDB
6.11-6.35
5.87-5.91
5.41-6.12
5,47-6.32
5.41-7.24
# of additional params
MNIST
12-1408
18-2112
24-2816
30-3520
36-4224
CIFAR-10
16- >75k
24-> 120k
32->150k
40->1800
48->225k
IMDB
12-384
18-576
24-768
30-960
36-1152
Table 6: Per-epoch training time is reported in seconds. The same number is calculated for Tanh and Maxout units (with six features). S-APL is faster than Maxout and is slightly slower than Tanh unit All models are trained using NVIDIA TITAN V GPU with 12036MiB memory and 850MHz.
S-APL
Tanh
Maxout
Activation
4
6
10
12
MNIST
5
7
10
12
13
12
CIFAR-I0
18
21
24
27
30
21
31
IMDB
11
14
16
For small values of S, we can that S-APL is comparable an exponential activation unit such as Tanh, and much faster than heavier activation such as Maxout.
As one can conclude from both Table 5 and 6 there is trade-off between the complexity and expressibility. We believe that S hinges where the binges' locations are symmetrically fixed.
equally hinges, is the best choice for the number of
8.2 EXPERIMENTS' DETAILS AND STATISTICAL SIGNIFICANCE
In this section, we explain the experimental conditions and the parameters used for each exper- iment. Also, in order to make the results of Table more interpretable, we perform t-test (Kim) 2013) on all the eтror rates achieved in that experiment.
In section 4, the experiment corresponding to Table performed using six different architectures. Lenet 5 is used as was introduced in LeCun et al. (198).. It has two convolution layers followed by two MLPs that are connected to a softmax layer.
The MLP architecture which is only used for classifying MNIST dataset has three fully connected feed-forward layers with 258, 128, and 10 hidden units. A dropout of rate 0.3 is added in between each layer. We use the batch size of 32 and we train the MLP network for 50 epochs.
Pure CNN architecture which only taking advantage of convolutional layers, was introduced in Springenberg et al.) (2014). All the specifications, including kernel size, stride, learning rate, batch size, and the number of epochs are taken from the main article.
ResNet18 architecture as variant of residual networks (He et al. 2016 taken from Napoletano al. 2018) which has five residual convolution blocks, one MLP. and softmax layer. The size of the convolution kernels, strides, padding, pooling size, pooling operation, learning rate, batch size, and epochs taken the defaults mentioned in Napoletano al. 2018
13

---

Under review as a conference paper at ICLR 2020
EffectiveNeets introduced in Tan & Le 2019)) and is used in experiments exactly the way in was originally introduced. It is necessary to mention that we use BO variant EffectiveNets.
Lastly, Net in Net architecture which is using an MLP instead of fixed nonlinear transformation is taken from Agostinelli al. (2014).
It is worth mentioning that in section 4, all the experiments were performed five times. To reduced the effect noise and randomness, the average of the five times is reported. Tabel we show the statistical significance of the experiments performed in section 4. Since each number is the average of five experiments, we are able to perform a t-test and provide p-values and statistical significance in Table 7 most of the numbers of Table
for each individual experiment. As one statistically significant.
Table 7: The best activation among ReLU, lcaky-ReLU, PReLU, tanh, sigmoid, ELU, maxout (nine features), Swish: z.sigmoid(3r) is chosen by the minimum average of the error rate. Then the significance of the comparison between the best network and S-APL activated network is calculated through a I-test. The p-vales each comparison is provided below.
MNIST
CIFAR-10
CIFAR-100
Activation
D-A
D-A
Lenet5 (PReLU vs S-APL)
0.061
0.041
0.049
0.054
0.062
MLP (Swish vs S-APL)
0.073
pure CNN (maxout vs S-APL)
0.043
0.057
0.072
0.068
ResNet-18 (PReLU S-APL)
0.029
0.031
0.049
0.040
EffectiveNet (ReLU vs S-APL.)
0.052
0.042
0.055
0.059
Net-in-Net(ReLU vs S-APL)
0.012
0.019
0,041
0.050
In section 5, we use two architectures to visualize S-APL shapes at different stages of the training process. For Figure T and Figure 2 we use the same MLP and Lenet 5 architectures as described above.
In section 6, we start by a iSNE visualization of 100 random samples of frogs and ships from the cifar10 test The tSNE mapping is performed using learning rate of 30 and perplexity of 40.
For the black-box adversarial attack experiments, each network is attacked five times and the ported number is the average of successful modifications in five different attacks. Attacks are done using the maximum number of iteration be 40 and the pop size be 400.
As for the open-box attacks, for both FGSM and CW-L2 attack, we employ the implementation and default parameess in Rauber et al. (201)).
Lastly, for activation function of networks in section 6, ReLU (y = for x > 0, otherwise ), APL (S 5, with fixed hinges 0, +1, and 2). Swish (y x.sigmoid(Bz)) with 0.2), and S-APL (with the configurations mentioned in the previous section) are used.
14


Reviews:
[{"review_id": "SyxadDtqFS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the S-APL (Symmetric Adaptive Piecewise Linear) activation function, based on the APL activation function proposed by (Agostinelli et al, 2014). This activation function is constructed as a piecewise linear function that is learned concurrently with training, and, in the case of S-APL, the activation function is forced to be symmetric. S-APL is claimed to help both with trainability and robustness of neural networks to adversarial examples.\n\nOverall, the idea is clearly presented, but appears to have many critical flaws, enumerated below:\n1. It is unclear what the motivation for the symmetry is upon first reading of the paper, i.e., Section 3.1 starts by saying \"To overcome the shortcomings of APL,\" but up to this point \u0010I cannot find any exposition that explains what these shortcomings are---the beginning of section 3 just presents the formulation of APL and does not discuss its advantages/disadvantages.\n\n2. The experimental results concerning network training are very hard to interpret, as they lack error bars, confidence intervals, and many critical experimental details (e.g. how many networks were trained, the hyper parameters for training, etc.) For these results to be interpretable, the authors should include a detailed description of the environment under which the experiments were performed, and present confidence intervals which demonstrate the significance of the improvement attained by S-APL. (As it stands, it is hard to tell whether, e.g., a 0.1% improvement on CIFAR-10 should be considered significant.)\n\n3. The adversarial evaluation section should be substantially revised to address several important flaws:\n(a) The evaluation for black-box attacks is done in very non-standard threat models (e.g. label-only 3-pixel black-box attacks) that do not seem to be relevant to the black-box robustness of a system. Even under these threat models, the authors should also use more powerful label-based black-box attacks, such as the BoundaryAttack [1] or the label-only attacks in [2] or [3].\n(b) The avg(|Z_{true} - Z_{adv}|) has an absolute value around Z_{true} - Z_{adv}, which means that failed adversarial attacks actually contribute positively to the score, which severely limits its usefulness for judging adversarial robustness. \n(c) The open-box (white-box) setting only uses FGSM to evaluate the robustness, which is known to be a weak attack [4] and is recommended against for evaluating adversarial robustness [5]. The methods should be evaluated with PGD or CW-attacks to better judge robustness.\n(d) Overall, the results presented are not sufficient to evaluate the adversarial robustness of the S-APL activation. A first step towards remedying this would be to follow the evaluation protocol suggestions outlined in [5].\n\nThe idea does seem promising, however, and the paper could be substantially improved by including the necessary introduction, evaluation protocols, and experimental details. However, this would require a substantial change to the paper, and thus my recommendation for now is to reject.\n\n[1] https://arxiv.org/abs/1712.04248\n[2] https://arxiv.org/abs/1804.08598\n[3] https://arxiv.org/abs/1807.04457\n[4] https://arxiv.org/abs/1802.00420\n[5] https://arxiv.org/abs/1902.06705", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hye1Aka8qB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a learnable piece-wise linear activation unit whose hinges are placed symmetrically. It gives a proof on the universality of the proposed unit on a certain condition. The superiority of the method is empirically shown. The change of the activation during training is analyzed and insight on the behavior is provided. The robustness to adversarial attacks is also empirically examined.\n\nThis paper discusses a very basic component of neural network models: activation function. Thus, it should be of interest to many researchers. The proposed method is simple and seems easy to use in real settings. A number of experiments are conducted to validate the method and the results look promising. The experiments in Section 5 is particularly interesting. It might give some hints for the following studies.\n\nHowever, there are several things to be addressed for acceptance.\n\n1) What is actually proposed is not very clear. \n\nS-APL is formulated in Equation 2. However, there are some discussion after that which changes or restricts the equation. For example, it seems that b_i^s^+ = b_i^s^- is assumed throughout the paper. In that case, it should be just reflected in Equation 2. In the third paragraph of Section 3.2, it is mentioned that h_i^s(x) = h_i^s(-x) with b_i^s^+ = b_i^s^-. However, it should also assume that a^s^+ = a^s^-. From the experiments. apparently, a^s^+ = a^s^- is not assumed. It seems that the method has symmetry only for the hinge locations. \n\nIn the first paragraph of Section 3.2, it is implied that parameters are shared across layers. It is not very clear what is shared and what is not. Please make that part clear. It will make it easier to understand the experimental settings, too.\n\n2) Theorem 3.1 does not seem to prove the approximation ability of S-APL.\n\nIt is clear that g(x, S) can represent arbitrary h(x, S), but I am not sure if it is clear that h(x, S) can represent arbitrary g(x, S). It should also depend on the conditions on a^s^+, a^s^-, b_i^s^+, b_i^s^-. I think it needs to prove that h(x, S) can approximate arbitrary piecewise linear function (i.e., g(x, S)) if you want to prove the approximation ability of h(x, S).\n\nEquation 4 seems to assume that all intervals are the same (i.e., ∀i, B_i - A_i = (B-A) / S). It should be stated explicitly. This relates to the problem 1).\n\nI may not understand some important aspect. I am happy to be corrected.\n\n3) Experimental conditions are not clear.\n\nPlease cite the papers which describe the architecture of the models used in the experiments. The effectiveness of the proposed method should depend on the network architecture and it is importable to be able to see the details of the models.\n\n4) On the sensitivity of optimization on the initial value.\n\nIt is interesting to see that \"fixed trained S-APL\" is not comparable with \"S-APL positive\". If the hypothesis in the paper is correct, it is natural to assume that \"fixed trained S-APL\" also has some issue on training. It would be interesting to see experimental results with \"initialized with trained-S-APL\" and \"S-APL positive with non-zero initial value\".  It is a bit weird to observe that \"S-APL positive\" never becomes non-zero for x < 0.\n\n5) Comparison results with other activation units in Section 6.\n\nThe proposed method is compared only with ReLU. It is important to see comparisons with other activations such as the plain APL.\n\n\nSome other minor comments:\n\nIt is quite interesting that objects are actually modified for adversarial attack for the proposed method in Figure 5. It would be interesting to have some consideration on it.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xhh7JecS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a new activation function, i.e. S-APL is proposed for deep neural networks. It is an extension of the APL activation, but is symmetric w.r.t. x-axis. It also has more linear pieces (actually S pieces, where S can be arbitrarily large) than the existing activation functions like ReLU. Experimental results show that S-APL can be on par with or slightly better than the existing activation functions on MNIST/CIFAR-10/CIFAR-100 datasets with various networks. The authors also show that neural networks with the proposed activation can be more robust to adversarial attacks.\n\nFirst of all, the activation function is much more complicated than the existing ones, as it has to determine the parameter S and the hinge positions. However, the gain is marginal as shown in Table 1. Besides, the authors never tell how to choose S and the hinge positions.\n\nSecondly, the neural networks used in the experiments are quite outdated. And the error rates shown in Table 1 are far away from state-of-the-art. Why don't you choose a latest network such as ResNet/DenseNet/EfficientNet and replace the activation with S-APL? The results could be more convincing.\n\nI am not an expert in adversarial attack. But is there any intuition why a complicated activation function is more robust to adversarial attack? Again, most of the models used in Table 2 are quite old (Lenet5, Net in Net, CNN).\n\nIn a word, the proposed activation function is unnecessarily complicated and the gain is not justified with the latest models and not significant enough to convince people to adopt it.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a learnable activation function based on adaptive piecewise linear (APL) units. Specifically, it extends APL to the symmetric form. The authors argue that S-APL activations can lead networks that are more robust to adversarial attacks. They present an empirical evaluation to prove the latter claim. However, the significance of these empirical results were not clear due to non-standard threat models used in black-box setting and the weak attacks used in open-box setting. The authors revised the submission and addressed some of the concerns the reviewers had. This effort was greatly appreciated by the reviewers. However, the issues related to the significance of robustness results remained unclear even after the revision. In particular, as pointed by R4, some of the revisions seem to be incomplete (Table 4). Also, the concern R4 had initially raised about non-standard black-box attacks was not addressed. Finally, some experimental details are still missing. While the revision indeed a great step, the adversarial experiments more clear and use more standard setup be convincing.

Decision: Reject