Revisiting Activation Function Design for Improving Adversarial Robustness at Scale
Authors: Cihang Xie
Mingxing Tan
Boqing Gong
Alan Yuille
Quoc V Le
Venue: Submitted to ICLR 2023 (2023)
Keywords: adversarial training
activation function
neural network architecture

Abstract: Modern ConvNets typically use ReLU activation function. Recently smooth activation functions have been used to improve their accuracy. Here we study the role of smooth activation function from the perspective of adversarial robustness. We find that ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Replacing ReLU with its smooth alternatives allows adversarial training to find harder adversarial training examples and to compute better gradient updates for network optimization.

We focus our study on the large-scale ImageNet dataset. On ResNet-50, switching from ReLU to the smooth activation function SILU improves adversarial robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. Smooth activation functions also scale well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness, largely outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://rb.gy/qt8jya.

Under review as a conference paper at ICLR 2023
REVISITING ACTIVATION FUNCTION DESIGN FOR IMPROVING ADVERSARIAL ROBUSTNESS AT SCALE
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern ConvNets typically use ReLUU activation function. Recently smooth ac- tivation functions have been used to improve their accuracy. Here we study the role of smooth activation function from the perspective of adversarial robustness. We find that ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Replacing ReLU with its smooth alternatives allows adversarial training to find harder adversarial training examples and to compute better gradient updates for network optimization.
We focus our study on the large-scale ImageNet dataset. On ResNet-50, switch- ing from ReLU the smooth activation function SILU improves adversarial ro- bustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on Ima- geNet. Smooth activation functions also scale well with larger networks: helps EfficientNet-LI to achieve 82.2% accuracy and 58. robustness, largely outper- forming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https::///// //tb./rr//////// ggtggt/j/ttttee
INTRODUCTION
It is known that convolutional neural networks can be easily fooled by adversarial examples (Szegedy et al. 2014). To improve robustness, many efforts have been made (Papernot et al.) 2016]]
Guo et al. 2018, Xie et al.] 2018 Liu et al. 2018 Pang et al.) 2019 |Schott et al. 2019); while adversarial training (Goodfellow et al. 2015] Kurakin et al. 2011 7 Madry et al. 2018), which trains of the most effective methods. Later studies further improve adversarial training by feeding networks with harder adversarial examples (Wang et al., 2019b), maximizing the margin of networks (Ding al., 2020), optimizing a regu- larized surrogate loss (Zhang et al., 22019) etc. While these methods achieve stronger adversarial robustness, they sacrifice accuracy on clean inputs. It is generally believed such trade-off between accuracy and robustness might be inevitable (Tsipras et al.] 2019). except for enlarging network capacities, e.g., making wider or deeper networks (Madry al. 2018 Xie & Yuille 2020
networks with adversarial examples on-the-fin fly. stands
Another popular direction for increasing robustness against adversarial attacks is gradient masking (Papernot et al., 2017: Xie al. 2018 Samangouei al.) 2018 (oong et al. 2018 Ma et al 2018 Guo et 2018). With the degenerated gradient quality, attackers cannot successtully optimize the targeted loss and therefore fail to circumvent such defenses. Nonetheless, the gradient mask- ing operation will be ineffective to offer robustness if its differentiable approximation is used for generating adversarial examples (Athalye et al. 2018).
To effectively build robust models, we hereby rethink the relationship between gradient quality and adversarial robustness, especially in the context of adversarial training where gradients are applied more frequently than standard training. In addition to computing gradients to update network pa- rameters, adversarial training also requires gradient computation for generating training samples. Guided by this principle, we identify ReLU, widely-used activation function in modern ConvNets, significantly weakens adversarial training due to its non-smooth nature, e.g., ReLU's gradient gets an abrupt change (from 0 to 1) when its input is close to zero (see Figure
In this paper. we revisit the activation function design for improving adversarial robustness, with special focus on the large-scale ImageNet dataset (Russakovsky et al. 2015). To fix the issue

---

Under review as a conference paper at ICLR 2023
Forward
Backward
0.8
12
0.5
1.0
0.2
0.4
0.1
0.2
0.0
0.0
0.6
-34
-02
0.0
0.2
0.4
06
-66
-04
-22
0.0 0.2
0.4 0.0
Figure 1: The visualization of ReLU and Parametric Softplus. Left panel: the forward pass for ReLU (blue curve) and Parametric Softplus (red curve). Right panel: the first derivatives for ReLU (blue curve) and Para- metric Softplus (red curve). Different from ReLU, Parametric Softplus is smooth with continuous derivatives.
induced by ReLU as aforementioned,, the gradient quality in adversarial training (Figure | shows Parametric Softplus, an example of smooth approximations for ReLUU). Our experiment results show that switching from ReLU to its smooth approximations in ad adversarial training can substantially improves ad adversarial robustness. For instance, by training with the computationally cheap single-step PGD attackes on ImageNet, the smooth activation function SILU significantly improves the robustness of the ResNet-50 baseline (which uses ReLU) by 9.3%, from 33.0% to 42.3%, meanwhile increasing standard accuracy by 0.9%. In addition, we note this performance improvement in both robustness and accuracy comes for "free", as the change of activation function does not incur additional computational cost.
propose to apply its smooth approximations for improving
We next explore the limits of adversarial training with smooth activation function using larger net- works. We obtain the best result by using EfficiienNNet-LI (Tan & Le 2019 Xie et al. (2020), which achieves 82.2% accuracy and 58.6% robustness ImageNet, significantly outperforming the pre- vious state-of-the- defense (Qin ct al. 2019) by 9.5% for accuracy and 11.6% for robustness.
RELATED WORKS
Adversarial training. Adversarial training improves robustness by training models on adversarial
examples (Szegedy et al. 2014 Goodfellow et al. 2015 Kurakin et al.
2017 Madry et al.) 2018).
Existing works suggest that, further adversarial robustness, we need to either sacrífice accuracy
on clean inputs (Wang et 20199] 2020 Zhang et al. 2019: Ding et al.)
2020). or incur additional
computational cost (Madry et | 20188 Xie & Yuille 2020; Xie et al.
2019). This phenomenon
is referred to as no free lunch in adversarial robustness (iipra et al.)
2019; Nakkiran 2019 Su
ct al., 2018). In this paper, we show that, using smooth activation function in adversarial training, adversarial robustness can be improved for "free"-no accuracy degradation on clean images and no additional computational cost incurred.
Our work is also related to the theoretical study (Sinha et al. 2018), which shows replacing ReLU with smooth alternatives can help networks get a tractable bound when certifying distributional robustness. In this paper, we empirically corroborate the benefits of utilizing smooth activations is also observable in the practical adversarial training on the real-world dataset using large networks.
Gradient masking. Besides training models adversarial data, other ways for improving adver- sarial robustness include defensive distillation (Papernot et al.) (016). gradient discretization (Buck- man et al.) 2018 Rozsa & Bouli 2019 Xiao 2019)) dynamic network architectures (Dhillon ct al. 2018 Wang et al. [2018]
(0018/200 220 9a Da]]
Lij Liu et [2018]
[eee Lee et al. 2020 Luo et al. (2020), randomized
transtormations Xie et al. [0188] Bhagoji
al. 2018 Xiao & Zheng 2020 Raff al.
2019 Ket-
tunen et al. 2019 AprilPyone & Kiyal 2020), adversarial input
(Guo et al.
2018 Prakash et al. 2018 Meng & Chen 2017 Song et al., 2018, Samangouei et al.
2018
Liao
et al. 2018 Bhagoji et al. 2018 Pang et al. 2020 Xu et al. 2017 Dziugaite et al.
2016 etc.
Nonetheless, most these detense methods degenerate the gradient quality the protected mod- els, therefore could induce the gradient masking issue (Papernot et al.) 2017). As argued in (Athalye) ctt al. 20188, defense methods with gradient masking may offer a false sense of adversarial robust- ness. In contrast to these works, we aim to improve adversarial robustness by providing networks with better gradients, but in the context of adversarial training.
'The term "smooth" hereby refer to this function satisfics the property of being C¹ smooth, iee., its first derivative is continuous everywhere. In practice, we note that single-step PGD adversarial training is only ~1.5x slower than standard training.
2

---

Under review as a conference paper at ICLR 2023
Table 1: ReLU significantly weakens adversarial training. By improving gradient quality for either the adversarial attacker or the network optimizer, resulted models obtains better robustness than the ReLU baseline. The best robustuess is achieved by adopting better gradients for both the attacker and the network optimizer. Note that, for all these model, ReLU is always used at their forward pass, albeit their backward pass may get probed in this ablation.
Improving Gradient Quality for Adversarial Attacker
Improving Gradient Quality for Network Optimizer
Network
Accuracy (%)
Robustness (8)
X
68.8
33,0
ResNet-50
68.3 (-0.5)
34.5 (+1.5)
(+0.6)
35.8 (+2.8)
68.9 (+0.1)
36,9 (+3.9)
RELU WEAKENS ADVERSARIAL TRAINING
In this section, we perform series of controlled experiments, specifically in the backward pass of gradient computations, to investigate how ReLU weakens, and how its smooth approximation strengthens adversarial training.
3.1 ADVERSARIAL TRAINING
Adversarial training (Szegedy al., 2014, Goodfellow et al. 2015;; Madry et al. 2018), which trains networks with adversarial examples on-the-fly, aims to optimize the following framework:
arg min (x.,)>= max L(0, + +,, cEB
(1)
where DD is the underlying data distribution, L(.,..) is the loss function, 0 is the network parameter, x a training sample with the ground-truth label y, e is the added adversarial perturbation, and S is the allowed perturbation range. To ensure the generated adversarial perturbation is human-
imperceptible, we usually restrict the perturbation range to be small (Szegedy et al. 2014 Good-
fellow al. 2015 Lou al. 2015).
As shown in equation T adversarial training consists of two computation steps: an inner maximiza- tion step, which computes adversarial examples, and an outer minimization step, which computes parameter updates.
Adversarial training setup. We choose ResNet-50 (He et al.) 2016) the backbone network, where ReLU is used by default. We apply PGD attacker (Madry et al. 2018) to generate adversar- ial perturbations Specifically, we select the cheapest version of PGD attacker, single-step PGD (PGD-1), to lower the training cost. Following (Shafahi et al.] 2019; Wong et al.) 2020). we set the maximum per-pixel change 4 and the attack step size ResNet training recipes to train models on ImageNet: models are trained for a total of 100 epochs using momentum SGD optimizer, with the learning rate decreased by 10x at the 30-th, 60-th and 90-th epoch; no regularization except a weight decay of le-4 is applied.
in PGD-I. We follow the standard
When evaluating adversarial robustness, we measure the model's top-1 accuracy against the 200-step PGD attacker (PGD--000) on the ImageNet validation set, with the maximum perturbation size 4 and the step size 3 = 1. We note 200 attack iterations are enough to let PGD attacker converge. Meanwhile, we report the model's t accuracy on the original ImageNet validation set.
3.2 How DOES GRADIENT QUALITY AFFECT ADVERSARIAT. TRAINING?
Figure [ shows ReLU is non-smooth. ReLU's gradient takes an abrupt change, when its input is close O, therefore significantly degrades the gradient quality. We conjecture this non-smooth nature hurts the training process, especially when we train models adversarially. This is because, compared to standard training which only computes gradients for updating network parameter 0, adversarial training requires additional computations for the inner maximization step to craft the perturbation c.
To fix this problem, we first introduce smooth approximation of ReLU,, named Parametric Sofiplus Nair & Hinton 2010), as f(a, x) = _ log(1 + exp(axr)), where the hyperparameter a is used to control the curve shape. Its derivative w.r.t. the input x is:
d dr f(a,2) 1 exx(-aax
(2)
3

---

Under review as a conference paper at ICLR 2023
To better approximate the curve of ReLU, we empirically set 10. As shown in Figure compared to ReLU, Parametric Softplus (a=10) is smooth because it has a continuous derivative.
I
With Parametric Softplus, next diagnose how gradient quality in the inner maximization step and the outer minimization step affects accuracy and robustness of ResNet-50 in adversarial training. To clearly benchmark the effects, only substitute ReLU with equation 2 in the backward pass, while leaving the forward pass unchanged, i.e., ReLU is always used for model inference.
Improving gradient quality for the adversarial attacker. We first take a look at the effects of gradient quality on computing adversarial examples (i.e., the inner maximization step) during train- ing. More precisely, in the inner step of ad adversarial training, we use ReLU in the forward pass, but Parametric Softplus in the backward pass; and in the outer step of adversarial training, we use ReLU in both the forward and the backward pass.
As shown in the second row of Table when the attacker uses Parametric Softplus's gradient (equa- tion 2 to craft adversrial training samples, the resulted model exhibits a performance trade-off. Compared to the ReLU baseline, it improves adversarial robustness by 1.5% but degrades standard accuracy by 0.5%.
Interestingly, we note such performance trade-off can also observed harder adversarial exam- ples are used in adversarial training (Wang et al. (019b). This observation motivates us to hypoth- esize that better gradients for the inner maximization step may boost the attacker's strength during training. To verify this hypothesis, we evaluate the robustness of two ResNet-50 models via PGD-1 (vs. PGD-200 in Table []. one with standard training and one with adversarial training. Specifically, during the evaluation, the PGD-1 attacker uses ReLU in the forward pass, but Parametric Softplus in the backward pass. With better gradients, PGD-I attacker is strengthened and hurts models more: it can further decrease the top-1 accuracy by 4.0% (from 16.9% to 12.9%) on the ResNet-50 with standard training, and by 0.7% (from 48.7% to 48.0%) on the ResNet-50 with adversarial training (both results are not shown in Table
Improving gradient quality for network parameter updates. We then study the role of gradient quality on updating network parameters (i.e., the outer minimization step) during training. More precisely, in the inner step of adversarial training. we use ReLU; but in the outer step of adversarial training, we use ReLU in the forward pass, and Parametric Softplus in the backward pass.
Surprisingly, by setting the network optimizer to use Parametric Softplus's gradient (i.e., equation 2 to update network parameters, this strategy improves adversarial robustness for "free". As shown in the third row of Table [ without incurring additional computations, adversarial robustness boosted by 2.8%, and meanwhile accuracy is improved by 0.6%, compared to the ReLU baseline. We note the corresponding training loss also gets lower: the cross-entropy loss on the training set is reduced from 2.71 to 2.59. These results of better robustness and accuracy, and lower training loss together suggest that, by using ReLU's smooth approximation in the backward pass of the outer minimization step, networks are able to compute better gradient updates in adversarial training.
Interestingly, we observe that better gradient updates can also improve standard training. For exam- ple, with ResNet-50, training with better gradients can improve accuracy from 76.8% to 77.0%, and reduces the corresponding training loss from 1.22 to 1.18. These results on both adversarial training and standard training suggest updating network parameters using better gradients could serve a principle for improving performance in general, while keeping the inference process of the model unchanged (i.e., ReLU always used for inference).
Improving gradient quality for both the adversarial attacker and network parameter updates. Given the observation that improving ReLU's gradient for either the adversarial attacker or the network optimizer benefits robustness, we further enhance adversarial training by replacing ReLU with Parametric Softplus in all backward passes, but keeping ReLU in all forward passes.
As expected, such a trained model reports the best robustness so far. As shown in the last row of Table D it substantially outperforms the ReLU baseline by 3.9% for robustness. Interestingly, this improvement still comes for "free"" it reports 0.1% higher accuracy than the ReLU baseline. We conjecture this is due to the positive effect accuracy brought by computing better gradient updates (increase accuracy) slightly overrides the negative effects on accuracy brought by creating harder training samples (hurt accuracy) in this experiment.
4

---

Under review as a conference paper at ICLR 2023
Forward
Backward
3.9
Piraeetic
Parsenti
8.2
1.0
MLU
2.5
Dreauteteey
1.8
0.8
1.1
08
-.0
.00
0.0
2.0
3.0
2.0
20
Figure 2: Visualizations of different smooth activation functions and their derivatives.
3.3 CAN RELU'S GRADIENT ISSUE BE REMEDIED?
More attack iterations. is known that increasing the number of attack iterations can create harder adversarial examples (Madry et al. 2018 Dong et al. 2018). We confirm in our own experiments that by training with PGD attacker with more iterations, the resulted model exhibits similar be- havior to the case where we apply better gradients for the attacker. For example, by increasing the attacker's s cost by 2x, PGD-2 improves the ReLU baseline by 0.6% for robustness while losing 0. % for accuracy. This result suggests we can remedy ReLU's gradient issue in the inner step of adversarial training if more computations are given.
Training longer. It is also known that longer training can lower the training loss (Hfffer et al, 2017), which we explore next. Interestingly, by extending the default setup 2x training sched- ule (i.e., 200 training epochs), though the final model indeed achieves a lower training loss (from 2.71 to 2.62), there still exhibits performance trade-off between accuracy and robustness. Longer training gains 2.6% for accuracy but loses 1.8% for robustness. On the contrary, our previous ex- periment shows that applying better gradients to optimize networks improves both robustness and accuracy. This discouraging result suggests that training longer cannot fix the issues in the outer step of adversarial training caused by ReLU's ' poor gradient.
Conclusion. Given these results, conclude that ReLU significantly weakens adversarial training. Moreover, seems that the degenerated performance cannot be simply remedied even with training enhancements (i.e., increasing the number of attack iterations & training longer). We identify that the key is ReLU's poor gradient- replacing ReLU with its smooth approximation only in the backward pass substantially improves robustness, even without sacrificing accuracy and incurring additional computational cost. the next section, we show that making activation functions smooth a good design principle for enhancing adversarial training in general.
ADVERSARIAL TRAINING WITH SMOOTH ACTIVATION FUNCTIONS
As demonstrated in Section improving ReLU's gradient can both strengthen the attacker and pro- vide better gradient updates in adversarial training. Nonetheless, this strategy may be suboptimal as there still exists a discrepancy between the forward pass (for which we use ReLU) and the backward pass (for which we use Parametric Softplus) when training the networks.
To fully exploit the potential of training with better gradients, we hereby propose exclusively apply smooth actvaition functions (in both the forward pass and the backward pass) in adversarial training. Noe that we keep all other network components exactly the same, as most of them are smooth and will not result in the issue of poor gradient
4.1 SMOOTH ACTIVATION FUNCTIONS
We consider the following activation functions as ReLU's smooth approximations in adversarial training (Figure 2 plots these functions and their derivatives):
Softplus(z) log exp(x)). We also consider its parametric version where a is set to 10 as in Section 3
Softplus (Nair & Hinton 2010):
We hereby ignore the gradient issue caused by max pooling, which is also non-smooth. This is because modern architectures rarely adopt it, e.g. only one max pooling layer is adopted in ResNet (He et al. 2016).
and none is adopted in EfficientNet. (Tan & Le! 2019
5

---

Under review as a conference paper at ICLR 2023
SILU (Elfwing et al., 2018 SILU(z) = szgmoid(z). Compared to other activation functions, SILU has non- monotonic "bump" when < O.
Hendrycks & Gimpel 2016 Ramachandran et al. 201
Exponential Linear Unit (ELU) (Clevert et al. 2016): if 2 0, ELU(x, a) = other- wise ELU(x, ox) = ((exp()) - 1), ELU(x, (x) = x if x 2 O, a(exp(x) - Jotherwis,, where we set o = 1 as default. Note that when 전 1, the gradient of ELU is not contin- uously differentiable anymore. We will discuss the effects of these non-smooth variants of ELU (a # 1) adversarial training in Section 4.3
Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel, 2016); GELU(z) = x (x)), where (xx cumulative distribution function of the standard normal distribution.
Main results. We follow the settings in Section 3 adversarially train ResNet-50 equipped with smooth activation functions. The results are shown in Figure Compared to the ReLU baseline, all smooth activation functions substantially boost robustness, while keeping standard accuracy almost the same. For example, smooth activation functions at least boost robustness by 5.7% (using Para- metric Softplus, from 33% to 38).%). Our strongest robustness achieved by SILU, which enables ResNet-50 to achieve 42.3% robustness and 69.7% standard accuracy. We believe these results can be furthered if more advanced smooth alternatives (e.g-. (Misra 2019 Lokhande et al. 2020 Biswas ct al. ) are used.
We then compare to the setting in Section where Parametric Softplus is only applied at the back- ward pass during training. Interestingly. by additionally replacing ReLU with Parametric Softplus at the forward pass, the resulted model further improves robustness by 1.8% (from 36.9% to 38. 38.7%) while without hurting accuracy. This result demonstrates the importance of applying smooth activa- tion functions in both forward and backward passes in adversarial training.
4.2 RULING OUT THE EFFECT FROM x <
Compared to ReLU, in addition being smooth, the functions above have non-zero responses to negative inputs (x < 0) which may also affect adversarial training. To rule out this factor, inspired by the design of Rectified Smooth Continuous Unit in (Shamir et al.) 2020), we hereby propose SmoothRel.U, which flattens the activation function by only modifying ReL.U after 2 0,
SmoothReLU(z,,)) = x log(ar + 1) if 2 0, otherwise,
(3)
where a is learnable variable shared by all channels of one layer, and is constrained to be positive. We note SmoothReLU is always continuously differentiable regardless the value of a,
az
if 1r20 O,
SmoothReLU(z,o)) dz
T+aa
(4)
otherwise.
Note SmoothReLU converges to ReLU when DO. Additionally, in practice, the learnable parameter a needs to be initialized at a large enough value (e.g.. 400 in our experiments) to avoid the gradient vanishing problem at the beginning of training. We plot SmoothRel.U and its first derivative in Figure 2
We observe SmoothReLU substantially outperforms ReLU by 7.3% for robustness (from 33.0% to 40.3%), and by 0.6% for accuracy (from 68.8% to 69.4%), therefore clearly demonstrates the importance of a function be smooth, and rules out the effect from having responses when æ <
4.3 STABILIZING ADVERSARIAL TRAINING WITH ELU USING CELU
In the analysis above, we show that adversarial training can be greatly improved by replacing ReLU with its smooth approximations. To further demonstrate the importance of being smooth, we provide another case study by using ELU (Clevert et al.) 2016). The first derivative of ELU is shown below:
if 2 0, ELU(x, a)= exx(()) otherwise. dx
(5)

---

Under review as a conference paper at ICLR 2023
Robustness (%)
ELU
CELU
GELU
41.1
Accuracy (%)
70
SILU
SmoothReLL
1.2
-0.3
+0.1
1.4
-20
-0.3
69
ReLU
Parametric Softplus
*ELU
1.6
-3.7
-0.3
Softplus
1.8
-6.2
-0.2
68
2.0
-7.9
-0.5
33
34
35
36
37
38
39
40
41
42
43
Table 2:
Robustness
Robustness (%)
comparison between
Figure 3: Smooth activation functions improve adversarial training- Com-
ELU (non-smooth when
pared to ReLU, smooth activation functions significantly boost robustness. while keeping accuracy almost the same.
a 1) and CELU (always smooth Va).
Here mainly discuss the scenario when ELU is non-smooth, i.e,, # 1. As be seen from equation ELU's gradient is not continuously differentiable anymore, ie., exp((z) * when = O, therefore resulting in an abrupt gradient change like ReLU, Specifically, we consider the range 1.0 < s 2.0, where the gradient abruption becomes more drastic with a larger value of a.
We show adversarial training results in Table Interestingly, we observe that adversarial robustness is highly dependent on the value of a a-the strongest robustness is achieved when the function is smooth (i.e., a 1.0, 41.4% robustness), and all other choices of monotonically decrease the robustness when a gradually approaches 2.0. For instance, with = 2.0, the robustness drops to only 33.2%, which 7.9% lower than that of using a 1.0. The observed phenomenon here is consistent with our previous conclusion ReLU ReLU-non-smooth activation functions significantly weaken adversarial training.
To stabilize adversarial training with ELU, we apply its smooth version, CELU BBarron 2017 which re- peraamettrize ELU to the following format:
x CELU(x, = a (exp (5) 1) otherwise.
(6)
Note that CELU equivalent to ELU when a 1.0. The first derivatives of CELU can be written as follows:
d CELU(Z, dæ
if 2 0, exp otherwise.
With this parameterization, CELU is now continuously differentiable regardless of the choice of a. As shown in Table we can observe that CELU greatly stabilizes adversarial training. Compared the best result reported by ELU/CELU (a 1.0), the worst case in CELU (a 2.0) merely 0.5% lower. Recall that this gap for ELU is 7.9%. This case study further support that it is important to apply smooth activation functions in adversarial training.
EXPLORING THE LIMITS OF ADVERSARIAL. TRAINING WITH SMOOTH ACTIVATION FUNCTIONS
Recent works (Xie & Yuille 2020 Gao al. 2019) show that, compared to standard training, adversarial training exhibits much stronger requirement for larger networks to obtain better per- formance. Nonetheless, previous explorations in this direction only consider either deeper networks (Xie & Yuille, 2020) or wider networks (Madry et al., 2018), which might be insufficient. To this end, we hereby present a systematic and comprehensive study on showing how network scaling up behaves in adversarial training, but using smooth activation functions. We set SILU as the default activation function, as achieves the best robustness among all other candidates (see Figure
5.1 SCALING-UP UP RESNET
We first perform the network scaling-up experiments with the ResNet family. In standard train- ing, Tan et al. (Tan & Le 2019) already show that, all three scaling-up factors, i.e., depth, width and image resolutions,
important to further improve ResNet performance. We hereby examine the effects of these scaling-up factors in adversarial training with smooth activation functions. We choose ResNet-50 (with the default image resolution 224) the baseline network.

---

Under review as a conference paper at ICLR 2023
Table 3:
Scaling-up ResNet with the smooth activation function SILU in adversarial training. We observe
larger networks consistently get better performance.
Accuracy (%)
Robustness (%)
ResNet-50
69.7
42.3
2x deeper (ResNet-101)
72.9
(43.2)
3x deeper (ResNet-152)
73,9 (+4.2)
46.0 (+3.7)
2x wider (ResNeXt:50-32xal)
11.2 (+1.5)
42.5 (+0.2)
4x wider (RssNcX--00:32x8)
(43.9)
45.1 (+2.8)
image size 299
(+1.2)
(+1.5)
image size 380
(+1.9)
44.1 (+1.8)
3x deeper & 4x wider (ResNeXi- 152-32x8d) & image size 380
78.2
(12.99
Depth & width. Previous works have shown that making networks deeper or wider can yield better model performance in standard adversarial training. We re-verify this conclusion using ResNet with the smooth activation function SILU. As shown in the second to fifth rows of Table we confirm that both deeper wider networks consistently outperform the baseline network. For instance, by training a 3x deeper ResNet-152, it improves ResNet-50's performance by 4.2% for accuracy and 3.7% for robustness. Similarly, by training 4x wider ResNeXt-50-32x8d (Xie et al. 2017), it improves accuracy by 3.9% and robustness by 2.8%.
Image resolution. Though larger image resolution benefits standard training. it is generally be- lieved that scaling up this factor will induce weaker adversarial robustness, as the attacker will have larger room for crafting adversarial perturbations (Galloway et al., (019)). However, surprisingly, this belief could be invalid when taking adversarial training into consideration. As shown in the sixth and seventh rows of Table 3 ResNet-50 with the smooth activation function SILU consistently achieves better performance when training with larger image resolutions.
We conjecture this improvement is possibly due to larger image resolution (1) enables attackers to create stronger adversarial examples (Galloway et al. 2019) (which will hurt accuracy but improve robustness); and (2) increases network capacity for better representation learning (Tan & Le 2019) (which will improve both accuracy and robustness); and the mixture of these two effects empirically yields a positive signal here.
Compound scaling, So far, we have confirmed that the basic scaling of depth, width and image resolution are all important scaling-up factors. As argued in (Tan & Le 2019) for standard training, scaling up all these factors simultaneously will produce much stronger model than just focusing on scaling up a single dimension. Hence we make an attempt to create a simple compound scaling for ResNet. As shown in the last row of Table we can observe that the resulted model, ResNeXt-152- 32x8d with input resolution at 380, achieves significantly better performance than the ResNet-50 baseline, Le., +8.5% for accuracy and +8.9% for robustness.
Adversarial training with ReLU. We first verify that the basic scaling of depth, width and image resolution also matter when ReLU is used in adversarial training e.g., by scaling up ResNet-50 (33.0% robustness), the deeper ResNet- 152 achieves 39.4% robustness (+6.4%), the wider ResNeXt- 50-32x8d achieves 36.7% robustness (+3.7%), and the ResNet-50 with larger image resolution at 380 achieves 36.9% robustness (+3.9%). Nonetheless, all these robustness performances are still lower than the robustness achieved by the basic ResNet-50 with the smooth activation function SILU (42.3% robustness, first row of Table 3).
We also find compound scaling is more effective than basic scaling for adversarial training with ReLUU, e.g., ResNNeXt-152-32x86 with input resolution at 380 reports 46.3% robustness. Although this result much better than the basic scaling above, it is still ~5% lower than ResNet + SILU with compound scaling, i.e., 46.3% v.s. 51.2%. In other words, even with larger networks, applying smooth activation functions remains essential for improving performance.
5.2 EFFICIENTNET RESULTS
The results on ResNet show scaling up networks with smooth activation functions in adversarial training effectively improves performance. Nonetheless, the applied scaling policies could be sub- optimal, as they are hand-designed without any optimizations. EfficientNet (Tan & Le 2019). which uses neural architecture search (Zoph & Le 2016) to automatically discover the optimal factors for network compound scaling, provides strong family of models for image recognition. Therefore we next use EfficientNet to replace ResNet. Note that all other training setups (e.g., using SILU) are the same as described in our ResNet experiments.
8

---

Under review as a conference paper at ICLR 2023
Similar to ResNet, Figure shows stronger EfficientNet consistently achieves better performance. For instance, by scaling the network from EfficientNet-Bi to EfficientNet-BR B7, the robustness is im- proved from 37.6% to 57.0%, and the accuracy is improved from 65.1% 1% to 79.8%. Surprisiggly the improvement is still observable for larger networks: EffficienNNet-L11 (Xie et al. (020) further im- proves robustness by 1.0% and accuracy by 0.7% over EfficientNNetB77. All these results corroborate the finding in (Cubuk et al. 2018)) that stronger backbone tend to yield higher robustness.
Training enhancements. So far all of our experiments follow the training recipes from ResNet, which may not be optimal for Effi cientNet training. Therefore, as suggested in the original Efficient- Net paper (Tan & Le 2019), we adopt the following training setups in our experiments: we change weight decay from le-4 to le-5,
86-
75
Rocustnese (54
Accurtyy (8)
80-
and add Dropout (Srivastava et al. 2014), Stochastic Depth (Huang) cet al.) 2016) and AutoAugment (Cubuk al., 2019) regularize the training process. Besides, we train models with longer sched- ule (i.e., 200 training epochs) to better cope with these training en- hancements, adopt the early stopping strategy to prevent the catas- trophic overfitting issue in robustness (Wong et al. 2020), and save checkpoints using model weight averaging (Izmailov et al. 2018) to approximate the model ensembling for stronger robustness (Pang et al. 2019 Strauss et al.] 2017). Additionally, as shown in the con- current work (Rebuffi et al. 2020), applying weight averaging is the key for enabling data augmentation to improve robustness. With these training enhancements, our EfficientNet-LI gets further im- proved, i.e., +1.7% for accuracy (from 80.5% to 82.2%) and +0.6% for robustness (from 58.0% to 58.6%).
Number Faramesers (Milions, log scate) Figure 4: Scaling-up Efficient- Net with the smooth activa- tion function SILU in adversarial training. Note EfficientNet-LI is not connected to the rest of the graph because it was not part of the compound scaling suggested by Tan et al. (Tan & L.e. 2019).
Comparing to the prior art (Qin et al. 2019). Table 4 compares our best results with the prior art. By applying the smooth activation function in adversarial training, we are able to train a model with strong performance on both adversarial robustness and standard accuracy-our best model (i.e.. EfficientNet-LI) achieves 82.2% standard accuracy and 58.6% robustness, which largely out- performs the prior art (Qin et al. 2019) by 9.5% for standard accuracy and 11.6% for adversarial robustness. Note this improvement mainly stems from the facts that we hereby exploit a better acti- vation function (SILU vs. Softplus) and a stronger architecture (EfficientNet-LI vs.. ResNet- 152).
Accuracy drop vs. model scale. Finally, we emphasize large reduction in the accuracy gap between adversarially trained models and stan- dard trained models for large networks. With the training setup above, EfficientNet-LI is able to
Table 4: Comparison the previous state-of-the-art.
Accuracy (5)
Robustness (5)
Prior art (Qin et al. 2019 EffiiienNdd
72.7
47.0
82.2 (+9.5)
58.6 (+11.6)
attain 84.1% accuracy in standard training, while this accuracy only slightly decreases to 82.2% (-1.9% in adversarial training. Note this gap is substantially smaller than the gap in ResNet-50 of 7.1% (76.8% in standard training v.s. 69.7% in adversarial training with the smooth activation function SILU). Moreover, it is also worth mentioning the high accuracy of 82.2% provides strong support to [lyas al. 2019) on arguing robust features indeed can generalize well to clean inputs.
6 CONCLUSION
In this paper, we revisit the activation function design for improving adversarial robustness at scale. Specifically, we propose replace the non-smooth activation function ReLU with its smooth ap- proximations (like Softplus or SILU) for ensuring architectural smoothness in adversarial training. Applying smooth activation functions in adversarial training improves adversarial robustness with- out sacrificing accuracy or incurring additional computation cost. Extensive experiments on Ima- geNet demonstrate the general effectiveness of adversarial training with smooth activation functions. By pushing the network scale to the very large EfficientNet-LI,, adversarial training with smooth activation functions reports the state-of-ther -tte--rrtt adversarial robustness on ImageNet, which substan- tially outperforms the prior art (Qin et al. 2019) by 9.5% for accuracy and 11.6% for robustness. Our results also corroborate the recent findings that there exist certain network architectures which have better adversarial robustness (Cubuk et al. 2018 Chen al., 2020: Guo et al.) 2020 Xie & jet al. 2019; [Hang et al.) 202I). We hope these works together can encourage more researchers to investigate this direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
MaungMaung yprilPyone and Hitoshi Kiya. Block-wise image transformation with secret key for adversarially robust defense. arXiv preprint 0rXiv:2010.00801, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICMI, 2018.
Jonathan T Barron. Continuously differentiable exponential linear units. arXiv preprint
(rXiv:/704.0748: 2017.
Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal. Enhancing robustness of machine learning systems via data transformations. In CISS, 2018.
Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Tanhsoft-a family of activation functions combining tanh and softplus. arXiv preprint arXiv:2009.0388 2020.
Jacob Buckman, Aurko Roy. Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In ICLR, 2018
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. 2019.
Hanlin Chen, Baochang Zhang, Song Xue, Xuan Gong, Hong Liu, Rongrong Ji, and David Doermann. Anti-bandit neural architecture search for model defense. arXiv preprint N.Xi::2008.006948 2020.
Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). ICLR, 2016.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fres free attacks. In ICML, 2020.
Ekin D Cubuk, Barret Zoph, Samuel S Schoenholz, and Quoc V Le. Intriguing properties of adver- sarial examples. In ICLR Workshop, 2018.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019.
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial de- fense. In ICLR, 2018.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin adver- sarial (mma) training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Xiaolin Hu, Jianguo Li, and Jun Zhu. Boost- ing adversarial attacks with momentum. In CVPR, 2018.
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A study of the effect of jpg compression on adversarial images. arXiv preprint mrXiiv1608.. 00853, 2016.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 2018
Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization is a cause of adversarial vulnerability. arXiv preprint arXiv:
2019.
Ruiqi Gao, Tianle Cai, Haochuan Li, Liwei Wang, Cho-Jui Hsich, and Jason D Lee. Convergence of adversarial training in overparametrized networks. NeurIPS, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
10

---

Under review as a conference paper at ICLR 2023
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using imput transformations. In ICLR, 2018.
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In CVPR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian arXiv: 08415, 2016.
error
linear
units (gelus). arXiv preprint
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Hanxun Huang, Yisen Wang, Sarah Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Exploring architectural ingredients of adversarially robust deep neural networks. 2021.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Adversarial examples are not bugs, they are features. NeurIPS, 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-lpips: Robust perceptual image similarity via random transformation ensembles. arXiv preprint
2019.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. In ICLR, 2017.
Hakmin Lee, Hong Joo Lee, Seong Tae Kim, and Yong Man Ro. Robust ensemble model training via random layer sampling against adversarial attack. arXiv preprint 1077,
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In NeurlPS, 2018.
Fangzhou Liao, Ming Liang, Yinpeng Dong, and Tianyu Pang. Defense against adversarial attacks using high-level representation guided denoiser. In CVPR, 2018.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In ECCV, 2018
Vishnu Suresh Lokhande, Songwong Tasnecyapant, Abhay Venkatesh, Sathya N. Ravi, and Vikas Singh. Generating accurate pseudo-labels in semi-supervised learning and avoiding overconfident predictions via hermite polynomial activations. In CVPR, 2020.
Yan Lou, Xavier Boix, Gemma Roig. Tomaso Poggio, and Qi Zhao. Roveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv: 151 1.0299, 2015.
Tiange Luo, Tianle Cai, Mengxiao Zhang, Siyu Chen, and Liwei Wang. Random mask: Towards robust convolutional neural networks. arXiv preprint 47Xiv:2007.14249.2000 2020.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.

---

Under review as a conference paper at ICLR 2023
Dongyu Meng and Hao Chen. Magnet: swo-pronged defense against adversarial examples. In CCS, 2017.
Diganta Misra. Mish: A self regularized non monotonoc neural activation function. arXiv preprint arXiv: 08681, 2019.
Vinod Nair and Geoffryy E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint arXiv:/90J.00532, 2019.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In ICMI., 2019.
Tianyu Pang, Kun Xu, and Jun Zhu. Mixup inference: Better exploiting mixup to defend adversarial attacks. In ICIR, 2020.
Tianyu Pang, Xiao Yang. Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In ICLR, 2021.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In SP, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In AsiaCCS, 2017.
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In CVPR, 2018.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In NeurlPS, 2019.
Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. CVPR, 2019.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:/710.0594 1710.05941, 2017.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation improve robustness. In NeurlPS, 2020.
Andras Rozsa and Terrance E Boult. Improved adversarial robustness by reducing open space risk via tent activations. arXiv preprint arXiv:1908. 1XXv:1908.02235, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. In ICLR, 2018.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on mnist. In ICLR, 2019.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NewrlPS, 2019.
Gil Shamir, Dong Lin, and Lorenzo Coviello. Smooth activations and reproducibility in deep net- works. arXiv preprint arXivv2010000931,, 2020..
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In ICIRR 2018.
12

---

Under review as a conference paper at ICLR 2023
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In ICLR, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting JMLR, 2014.
Thilo Strauss, Markus Hanselmann, Andrej Junginger, and Holger Ulmer. Ensemble meth- ods as defense to adversarial perturbations against deep neural networks. arXiv preprint arXiv: 100:: 03343, 2017.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In ECCV, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. There is no free lunch in adversarial robustness (but there are unexpected benefits), In ICLR, 2019.
Siyue Wang, Xiao Wang, Pu Zhao, Wujie Wen, David Kaeli, Peter Chin, and Xue Lin. Defensive dropout for hardeming deep neural networks under adversarial attacks. In ICCAD, 2018.
Xiao Wang, Siyue Wang, Pin-Yu Yu Chen, Yanzhi Wang, Brian Kulis, Xue Lin, and Peter Chin. Pro- tecting neural networks with hierarchical random switching: Towards better robustness-accuracy trade-off for stochastic defenses. In IJCAI, 2019a.
Yisen Wang. Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML,, 2019b.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
Eric Wong, Leslie Rice, and Zico Kolter. Fast better than free: Revisiting adversarial training. In ICLR, 2020.
Chang Xiao and Changxi Zheng. One man's 's trash is another man's treasure: Resisting adversarial examples by adversarial examples. In CVPR, 2020.
Chang Xiao, Peilin Zhong, and Changxi Zheng. Enhancing adversarial defense by k-winners-take- all. In ICLR, 2019.
Cihang Xie and al. Feature denoising for improving adversarial robustness. In CVPR, 2019.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In ICLR, 2020.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. ICLR, 2018.
Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoe Le. Self-training with noisy student im- proves imagenet classification. In CVPR, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In NDSS, 2017.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing. Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. ICLR, 2016.
13

---

Under review as a conference paper at ICLR 2023
HeLU
SILU
45-
Figure Comparison of loss landscapes between the network using ReLU and the network using SILU, on randomly selected ImageNet sample.
A SANITY TESTS FOR ROBUSTNESS EVALUATION
Correctly performing robustness evaluation is non-trivial, there exist many factors (e.g., gradient mask) which may derail the adversarial attacker from accurately accessing the model performance. To avoid these evaluation pitfalls,
set of sanity tests, following the recommendations in (Carlini et al. 2019). Specifically, we take the ResNet-50 with SILU as the evaluation target.
Robustness vs. attack iterations. By increasing the attack iterations from 5 to 200, the resulted PGD attacker consistently hurts the model more. For example, by evaluating against PGD-(5, 10, 50), the model reports the robustness 48.7%, 43.7% and 42.7%, respectively. This evaluation finally gets converged at 42.3% when using PGD-200.
Robustness perturbation sizes 6. We also confirm a larger perturbation budget strengthens the attacker. By increasing e from 4 to 8, the robustness drops more than 25%; the model will be completely circumvented if we set e 16.
Landscape visualization. We compare the loss landscapes between ours and the ReLU baseline, on a randomly selected samples from ImageNet val set. Specifically, following (Li et al. 2018), the x/y- axis refer to the directions of adversarial perturbation, and z-axis refers to the corresponding cross- entropy loss. As shown in Figure 5 compared to the ReLU baseline, we observe that adversarial training with smooth activation functions produces much smoother loss landscape. Note this smooth loss landscape can also be observed when using other smooth activation functions (besides SILU) and other randomly selected images.
In summary, these observations confirm our robustness evaluation with PGD attacker is properly done in this paper.
B CIFAR-10 10 RESULTS
Though the main focus of this paper to study adversarial robustness on the large-scale ImageNet dataset, in this section, we briefly check how adversarial training with smooth activation functions performs on CIFAR-10. 10. adversarially train R with following settings: during train- ing, the attacker is PGD-1 with maximum perturbation = 8 and step size 3 = 8; we use AutoAttack CCroee & Hein 2020 to holistically evaluate these models.
Firstly, expected, note Softplus, GELU and SmoothReLU all lead to better results than the ReLU basebaae--eooooe comparee- to ReLU, all these three activation functions maintain similar accuracy, but much stronger robustness. For examples, the improvements on robustness ranging from 0.7% (by GELU) to 2.3% (by Softplus).
However, unlike ImageNet experiments where smooth activation functions demonstrate sistent improvements over ReLU, we note there are two exceptions on CIFAR- 10 c0-compared to ReLU, ELU and SILU even significantly hurt adversarial training. More interestingly, this inferior
14

---

Under review as a conference paper at ICLR 2023
performance also be observed on the training set. For example, compared to ReLU, SILU yields 4.1% higher training error and ELU yields 10.6% higher training error. As suggested in (Pang et al, 2021 Wong et al. 2020), adversarial training CIFAR-10 is highly sensitive to different parameter settings, therefore we leave the exploring of the "optimal" adversarial training setting with ELU and SILU on CIFAR-10 10 as future work.
15


Reviews:
[{"review_id": "ZGF-NIBQXYY", "reviewer": "Reviewer_qQzz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an experimental evaluation on the use of smooth activation functions, approximating ReLU, in the context of adversarial robustness on ImageNet. The results would appear to show that smooth activations increase empirical adversarial robustness without negatively affecting standard accuracy. Furthermore, the experiments confirm the belief that capacity is crucial to enhance trade-offs between standard and robust accuracy.", "summary_of_the_review": "While the subject of this experimental paper is interesting and novel, I believe the authors' claims are not adequately supported due to a series of weaknesses of the experimental evaluation.\nI am more than willing to increase my score if the authors address the aforementioned limitations.", "strength_and_weaknesses": "**Strengths**.\nAnalyzing the interplay between activation function smoothness and adversarial robustness is definitely a topic of interest to the community. If confirmed, the provided experimental results could push practitioners towards the use of smooth activation functions, as these would appear to yields better empirical robustness at no cost in standard accuracy or training cost.\n\n**Weaknesses**.\nAs the authors themselves acknowledge in appendix A, \"*correctly performing robustness evaluation is non-trivial*\". In this sense, while the appendix itself provides some sanity checks, I believe these are insufficient to ensure the validity of the claims.\n- the vast majority of the training experiments relies on a single-step PGD attack (except for a small experiment in section 3.3): would the claims still hold when running the attacks for longer?\n- it would appear to me that the PGD attack step size is never tuned, neither at training nor at evaluation. Could the authors provide more information regarding this? The attack step size might be fairly sensitive to the activation function.\n- the evaluation of empirical robustness is inherently tied to the employed attack. The authors are focussing on PGD, but would the claims still hold under different attack methods? For instance, black-box attack schemes (e.g., Square Attack), or AutoAttack.\n- the evaluation mostly focuses on a single dataset (ImageNet). CIFAR-10 results (in the appendix) are only drafted, and seem to radically differ from those presented in the main paper. It would be nice to have evidence that the claims hold beyond a single dataset.\n- no guarantee about the actual robustness of a network (empirical robustness might always be an artifact of the attack itself) can be given without formal verification techniques. It would be very interesting to check whether the verified adversarial robustness of the trained networks (a bound can be computed by running [IBP](https://arxiv.org/abs/1810.12715) or adapting [CROWN](https://arxiv.org/abs/1811.00866) for the employed activation functions) is affected by the use of smooth activations.\n\nI would also like to point out that formal verification techniques usually rely on branch and bound (see [VNN-COMP-21](https://arxiv.org/abs/2109.00498)), which is guaranteed to terminate in finite time only for piece-wise linear activations. Therefore, verifiability of adversarial robustness is likely to be negatively affected by the use of smooth activation functions.\nIt would be nice if the authors could mention this in the introduction or related work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WrlCDfCmGUd", "reviewer": "Reviewer_yesB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a lot of ImageNet experiments justifying that smooth activation function helps adversarial training. ", "summary_of_the_review": "The authors provide a lot of experiments in ImageNet data set to show that adversarial training benefits from smoothed activation functions. I would encourage the authors to (1) provide detailed experiments in other data sets, e.g., CIFAR-10, and show that the proposed methods are uniformly good for all (or most) data sets; (2) update the literature review with recent studies and emphasize the novelties.\n\nI still rate the submission as 5, because the experimental results for ImageNet data set are thorough, and no previous empirical study has done as good as this paper based on my knowledge.\n", "strength_and_weaknesses": "\nStrength:\n\nThe authors have a thorough experimental analysis in the ImageNet data set.\n\nWeakness:\n\nI have several concerns of this paper:\n\n[1] My main concern is that the experiments are mainly for ImageNet, but no sufficient studies for other data sets, e.g., CIFAR-10. In Section B, it is mentioned that the performance of some of the smoothed activation functions are not as good as ReLU. The authors provides some explanation on the sensitivity of CIFAR-10 data set, but we can not claim that ImageNet is free from the same issue. As a result, some more in-depth analysis are wanted.\n\nIn addition, in the CIFAR-10 results, the authors only mention \"the improvements on robustness are ranging from 0.7% (by GELU) to 2.3% (by Softplus)\". In Table 2 of Croce & Hein (2022), there is a long list of CIFAR-10 results from different papers. The authors of this submission may provide some more details about the performance. \n\nThe paper title has the word \"at scale\", but it is not clear why the smoothing techniques cannot benefit small scale data sets like CIFAR-10. I would suggest the authors to try different training size to clarify whether the word \"at scale\" is really important or not. If using 50000 ImageNet samples from 10 classes, will it have the same problem as CIFAR-10? On the other hand, in literature such as\n\nGowal, Sven, et al. \"Improving robustness using generated data.\" Advances in Neural Information Processing Systems 34 (2021): 4218-4233.\n\nand\n\nCarmon, Yair, et al. \"Unlabeled data improves adversarial robustness.\" Advances in Neural Information Processing Systems 32 (2019).\n\nit is observed that more data improves adversarial training for CIFAR-10. If we use the 1M data provided by Gowal et. al. (2021, https://github.com/deepmind/deepmind-research/tree/master/adversarial_robustness), will we observe similar observation as the ImageNet data set?\n\n\n[2] All papers in the reference list are published before 2021. Could the authors supplement the introduction with some more updated literature in the recent two years?\n\n\nMinor issue:\n\n[1] Some theory literature already consider some smoothness problem in adversarial training. For example, in \n\nAllen-Zhu, Zeyuan, and Yuanzhi Li. \"Feature purification: How adversarial training performs robust deep learning.\" 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 2022.\n\nTo avoid the non-smooth issue, they directly drop the attack if the gradient is small (the equation above Definition 4.3). It mentions that \"because otherwise A(f, x, y) is not Lipscthiz continuous at those points.\" in footnote 9 (https://arxiv.org/pdf/2005.10190.pdf).\n\nCould you help explain what are the difference in the intuitions between the submission and this paper? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8jgmdNKnNRZ", "reviewer": "Reviewer_zKRu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effects of smooth activation functions on model robustness. It claims smooth activation functions lead to better performance compared with ReLU. The authors conduct extensive experiments on ImageNet to illustrate how smooth activation functions affect the adversarial example generation and model parameter update. In addition, this work also studies how the model capacity and input resolution affect the performance of models with smooth activation function. ", "summary_of_the_review": "Based on the concerns and weakness pointed above, I cannot recommend this paper to appear in ICLR. I welcome the authors to address my concerns during the rebuttal, after which I will do another evaluation.", "strength_and_weaknesses": "Strength:\n\nThis paper comprehensively study the effect of smooth activation function from different perspectives by extensive experiments.\n\nWeakness:\n\n1. To comprehensively evaluate the robustness of a model, it is not enough to just increase the iteration number of PGD. The authors should also try other methods, such as the ones using loss functions other than cross-entropy and some margin based attack methods (like FAB: https://arxiv.org/abs/1907.02044). Currently, the most reliable method to evaluate robustness is AutoAttack (http://proceedings.mlr.press/v119/croce20b/croce20b.pdf), the authors should report the robust accuracy on AutoAttack, which is more accurate than PGD.\n\n2. Using stronger adversarial attack does not necessarily lead to more robust models. For example, 1-step fast adversarial training can still achieve competitive performance, while fitting some out-of-distribution training instances with strong adversarial perturbations may hurt the performance. (https://arxiv.org/abs/1910.08051) Therefore, some related explanations in Section 3.2 are not convincing.\n\n3. This is a pure empirical work, as the difference between some methods is not very big, the authors should run the experiments multiple times and report the performance variance.\n\n4. In section 3.2, smooth activation functions are used only for backward, the authors should explain why.\n\n5. Just one setting (such as model architecture, the dataset) is studied in Section 3.3, so the conclusions in this section are not convincing.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-HjMPLhfMnj", "reviewer": "Reviewer_U7PC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper finds that non-smooth ReLU activation function weaken the adversarial robustness and smooth activation functions such as SiLU can improve the adversarial robustness. The experiments are based on large-scale ImageNet dataset and show the state-of-the-art performance.", "summary_of_the_review": "The authors empirically show that the smooth activation function improves the adversarial robustness on ImageNet dataset. Extensive experiments on ImageNet dataset are interesting. But, I have concerns about the checking for gradient masking and consistent results for other benchmark dataset and theoretical results ( or explanation ) of why smooth activation function improves adversarial robustness.", "strength_and_weaknesses": "\nStrength\n- This paper focuses on ImageNet dataset. To my best knoweldge, it is the first time to do extensive experiments for the large scale ImageNet dataset.\n- This paper proposes a new algorithm for adversarial training with ReLU in forward pass and with smooth activation in backward pass without additional computational cost. It improves the adversarial robustness without hampering standard accuracy.\n- This paper empirically shows SiLU is the most effective in adversarial robustness among various smooth activation functions on ImageNet.\n\nWeakness\n- This paper lacks intuitive explanation (and theoretical studies) about why smooth activation functions improve the adversarial robustness.\n- The experiments are conducted on only ImageNet. It is necessary to conduct experiments with other benchmark datasets to support the main claim. If the results of other benchmark datasets are not consistent, the novelty would be small.\n- The evaluation is not properly executed. Checking for gradient masking is very crucial. I admit that autoattack is difficult to use on ImageNet dataset. At least, one black-box attack such as Square Attack [1] is necessary for checking the gradient masking.\n\n[1] Andriushchenko, Maksym and Croce, Francesco and Flammarion, Nicolas and Hein, Matthias, Square Attack: a query-efficient black-box adversarial attack via random search, In ECCV, 2020.\n\nQuestion :\n- Authors insist that the gradient from a smooth activation function is better than ReLU. In what sense is it a better gradient?\n\nMinor\n[At second paragraph in 4th section] Noe that we keep => Note that we keep", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject