Neural Architecture Dilation for Adversarial Robustness
Authors: Yanxi Li
Zhaohui Yang
Yunhe Wang
Chang Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Learning
Neural Architecture Search
Adversarial Robustness

Abstract: With the tremendous advances in the architecture and scale of convolutional neural networks (CNNs) over the past few decades, they can easily reach or even exceed the performance of humans in certain tasks. However, a recently discovered shortcoming of CNNs is that they are vulnerable to adversarial attacks. Although the adversarial robustness of CNNs can be improved by adversarial training, there is a trade-off between standard accuracy and adversarial robustness. From the neural architecture perspective, this paper aims to improve the adversarial robustness of the backbone CNNs that have a satisfactory accuracy. Under a minimal computational overhead, the introduction of a dilation architecture is expected to be friendly with the standard performance of the backbone CNN while pursuing adversarial robustness. Theoretical analyses on the standard and adversarial error bounds naturally motivate the proposed neural architecture dilation algorithm. Experimental results on real-world datasets and benchmark neural networks demonstrate the effectiveness of the proposed algorithm to balance the accuracy and adversarial robustness.

Neural Architecture Dilation for Adversarial Robustness
Yanxi Li Zhaohui Yang 2,3 Yunhe Wang Chang Xu
School of Computer Science, University of Sydney, Australia
2 Huawei Noah's Ark Lab
Key Lab of Machine Perception (MOE), Department of Machine Intelligence, Peking University, China
yali07220uni. edu. au, zhaohuiyang@pku. edu. edu. en, yunhe. wwwg@hawwi...coommmm com, c xu@sydnney edu.au au
Abstract
With the tremendous advances in the architecture and scale of convolutional neural networks (CNNs) over the past few decades, they easily reach even exceed the performance of humans in certain tasks. However, a recently discovered short- coming of CNNs is that they vulnerable adversarial attacks. Although the adversarial robustness of CNNs can be improved by adversarial training, there a trade-off between standard accuracy and adversarial robustness. From the neural architecture perspective, this paper aims to improve the adversarial robustness of the backbone CNNs that have a satisfactory accuracy. Under a minimal computa- tional overhead, the introduction of a dilation architecture is expected to be friendly with the standard performance of the backbone CNN while pursuing adversarial robustness. Theoretical analyses on the standard and adversarial error bounds natu- rally motivate the proposed neural architecture dilation algorithm. Experimental results on real-world datasets and benchmark neural networks demonstrate the effectiveness of the proposed algorithm balance the accuracy and adversarial robustness.
Introduction
In the past few decades, novel architecture design and network scale expansion have achieved significant success in the development of convolutional neural networks (CNN) [12, 13, 11, 23, 9, 31, 35, 17, 15]. These advanced neural networks can already reach or even exceed the performance of humans in certain tasks [10, 21]. Despite the success of CNNs, a recently discovered shortcoming of them is that they are vulnerable adversarial attacks. The ingeniously designed small perturbations when applied to images could mislead the networks to predict incorrect labels of the input [7]. This vulnerability notably reduces the reliability of CNNs in practical applications. Hence developing solutions to increase the adversarial robustness of CNNs against adversarial attacks has attracted particular attention from the researchers.
Adversarial training can be the most standard defense approach, which augments the training data with adversarial examples. These adversarial examples are often generated by fast gradient sign method (FGSM) [7] or projected gradient descent (PGD) [18]. Tramèr et al. [24] investigates the adversarial examples produced by number of pre-trained models and developed an ensemble adversarial training. Focusing on the worst-case loss over convex outer region, Wong and Kolter [27] introduces a provable robust model. There are more improvements of PGD adversarial training techniques, including Lipschitz regularization [6] and curriculum adversarial training [2]. In a recent study by Tsipras et al. [25], there exists trade-off between standard accuracy and adversarial robustness. After the networks have been trained to defend against adversarial attacks, their performance over natural image classification could be negatively influenced. TRADES [32] theoretically studies this trade-off by introducing boundary error between the natural (i.e. standard) error and the robust error. Instead directly adjusting the trade-off, the friendly adversarial training (FAT) [33] proposes to exploit weak adversarial examples for a slight standard accuracy drop.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Numerous efforts have been made to defend the adversarial attacks by carefully designing various training objective functions of the networks. But less noticed is that the neural architecture actually bounds the performance of the network. Recently there are few attempts to analyze the adversarial robustness of the neural network from the architecture perspective. For example, RACI. [5] applies Lipschitz constraint on architecture parameters in one-shot NAS to reduce the Lipschitz constant and improve the robustness. RobNet [8] search for adversarially robust network architectures directly with adversarial training. Despite these studies, deeper understanding of the accuracy and robustness trade-off from the architecture perspective is still largely missing.
In this paper, we focus on designing neural networks sufficient for both standard and adversarial clas- sification from the architecture perspective. We propose neural architecture dilation for adversarial robustness (NADAR). Beginning with the backbone network of satisfactory accuracy over the natu- ral data, we search for a dilation architecture to pursue a maximal robustness gain while preserving a minimal accuracy drop. Besides, we also apply FLOPs-awes approach to optimize the architecture, which can prevent the architecture from increasing the computation cost of the network too much. We theoretically analyze our dilation framework and prove that our constrained optimization objectives can effectively achieve our motivations. Experimental results on benchmark datasets demonstrate the significance of studying the adversarial robustness from the architecture perspective and the effectiveness of the proposed algorithm.
2 Related Works
2.1 Adversarial Training
FGSM [7] claims that the adversarial vulnerability of neural networks is related to their linear nature instead of the nonlinearity and overfitting previously thought. A method to generate adversarial examples for adversarial training is proposed based on such a perspective to reduce the adversarial error. PGD [18] studies the adversarial robustness from the view of robust optimization. A first-order gradient-basee based method for iterative adversarial is proposed. FreeAT [22] reduces the computational overhead of generating adversarial examples. The gradient information in network training is recycled to generate adversarial training. With this gradient reusing, it achieves 7 to 30 times of speedup.
However, the adversarial robustness comes at price. Tsipras et al. [25] reveals that there is a trade-off' between the standard accuracy and adversarial robustness because of the difference between features learned by the optimal standard and optimal robust classifiers. TRADES [32] theoretically analyzes this trade-off. A boundary error is identified between the standard and adversarial error to guide the design of defense against adversarial attacks. As solution, tuning parameter À is introduced into their framework to adjust the trade-off. The friendly adversarial training (FAT) [33] generates weak adversarial examples that satisfy minimal margin of loss. The miss- classiiid adversarial examples with the lowest classification loss are selected for adversarial training.
2.2 Neural Architecture Search
NAS aims to automatically design neural architectures for networks. Early NAS methods [1, 35, 16, 19] computationally intensive, requiring hundreds thousands of GPU hours because of the demand of training and evaluation of a large number of architectures. Recently, the differentiable and one-shot NAS approaches Liu et al. [17] and Xu al. [29] propose to construct one-shot supernetwork and optimize the architecture parameter with gradient descent, which reduces the computational overhead dramatically. Differentiable neural architecture search allows joint and differentiable optimization of model weights and the architecture parameter using gradient descent. Due to the parallel training of multiple architectures, DARTS is memory consuming. Several follow- up works aim to reduce the memory cost and improve the efficiency of NAS. One remarkable approach among them is PC-DARTS [29]. It utilizes a partial channel connections technique, where sub-channels of the intermediate features are sampled to be processed. Therefore, memory usage and computational cost are reduced. Besides direct reducing the training cost, CARS [30] proposes novel efficient continuous evolutionary approach based on the historical evaluation. Similarly, PVLL-NAS [14] performs evaluation with a performance estimator, who samples neural architectures for both architecture searching and iterative training of the estimator itself.
Considering adversarial attacks in the optimization of neural architectures can help designing networks that are inherently resistant to adversarial attacks. RACL [5] applied constraint on the architecture parameter in differentiable she-shot NAS to reduce the Lipschitz constant. Previous works [3, 26] have shown that smaller Lipschitz constant always corresponds to more robust network. It is. therefore, effective to improve the robustness of neural architectures by constraining their Lipschitz constant, RobNet [8] directly optimizes the architecture by adversarial training with PGD.

---

3 Methodology
The adversarial training can be considered as minimax problem, where the adversarial perturbations generated to attack the network by maximizing the classification loss, and the network is optimized to defend against such attacks:
min (aaa)>>
max e(y, f(')()))))
x' (x,,))
where Di is the distribution of the natural examples x and the labels (x fa' 1 - E) defines the set of allowed adversarial examples x'' within the scale of small perturbations under lp normalization, and f is the network under attack.
3.1 Robust Architecture Dilation
The capacity of deep neural network has been demon- strated be critical its adversarial robustness [18, 25, 34). Madry et al. [18] finds capacity plays an important role in adversarial robustness, and networks require larger capacity for adversarial than standard tasks. Tsipras et al. [25] suggests simple classifier for standard tasks cannot reach good performance on adversarial tasks. However, it remains an open question to use the minimal increase of network capacity in exchange for the adversarial robust- ness.
Network
Suppose that we have backbone network fo that can achieve a satisfactory accuracy on the natural data. To strengthen its adversarial robustness without hurting the standard accuracy, we propose to increase the capacity of this backbone network fú by dilating it with network fa- whose architecture and parameter will be optimized within the adversarial training.
Backbone
Dilation Network
Network
Figure 1: The overall structure of NADAR hybrid network.
The backbone network fo is split into blocks. A block f( defined as a set of successive layers in the backbone with the same resolution. For a backbone with L blocks, i.e. fo ( 1, L),
we attach a cell f((() of the dilation network to each block f!"). Therefore, the dilation network also has L cells, i.e. fa = (L") , 1,... ... L}. For the dilation architecture, we search for cells within NASNet-like [35] search space. In a NASNet-like search space, each cell takes two previous outputs its inputs. The backbone and the dilation network are further aggregated by element-wise sum. The overall structure of a NADAR hybrid network is as shown in Figure 1. Formally, the hybrid network for the adversarial training is defined as:
fnyo(x)= h (OL=I....L (1 (zbyb ((-11))++ fa (2-11) <nys =
where =nyo = (10)) (=()=)) + 110) (2 =nyb ((-2)) is the latent feature extracted by the backbone block and the dilation block, and @ represents functional composition. We also define classification
(L) hypothesis h =nyb -> ÿ. where zayb is the latent representation extracted by the last convolutional layer L, and ú is the predicted label.
During search, the backbone network fo has fixed architecture and is parameterized by network weights 0,- The dilation network fa is parameterized by not only network weights 0₂ but also the architecture parameter ag. The objective of robust architecture dilation is to optimize Oa for the minimal ad versarial loss
min ax
c(edv) (fuyr: (aa)))
(3)
s.t. 0 (aaa))) = argmin ((avv) (frys), O.,
(4)
where c(adr) (fyy)) and c/adu) (aye: (a.)) are the adversarial losses of fays (with the form Eq. 1) on the training set Dirain and the validation set Dvaria. respectively, and 0 (xxa) is the optimal network weights of fa depending on the current dilation architecture Ord-
3.2 Standard Performance Constraint
Existing works on adversarial robustness often fix the network capacity, and the increase of adversarial robustness is accompanied by the standard accuracy drop [25, 32]. However, in this work, we increase
3

---

the capacity with dilation, which allows us to increase the robustness while maintaining a competitive standard accuracy. We reach that with standard performance constraint on the dilation architecture. The constraint is achieved by comparing the standard performance of the hybrid network Snyb to the standard performance the backbone. We denote the network using the backbone only as fucks which can be formally defined as:
focx(x)= = (OV (1) (=box"))
where back f/ (==bx")) is the latent feature extracted by the backbone block. The standard model is optimized with natural examples by:
min ((ara) (Sbex) = B(a,y)> [e(Jbek (1,, a)]]
(6)
0,
where ((std) is the standard loss. Similarly, we can define the standard loss C(std) (fiye) for the hybrid network fagb. In this way, we can compare the two networks by the difference of their losses and constrain the standard loss of the hybrid network be equal to or lower than the standard loss of
the standard network:
clstd) (fuye) - ((std) (foxx) S.
We do not directly optimize the dilation architecture the standard task, because it is introduced to capture the difference between the standard and adversarial tasks to improve the robustness of the standard trained backbone. It is unnecessary let both the backbone network and the dilation network to learn the standard task.
3.3
FLOPs-Aware Architecture Optimization
By enlarging the capacity of networks, we can improve the robustness, but a drawback is that the model size and computation cost raises. We want to obtain the largest robustness improvement with the lowest computation overhead. Therefore, a computation budget constraint on architecture search applied. As we are not targeting at any specific platform, the number of floating point operations (FLOPs) in the architecture instead of the inference latency is considered. The FLOPs is calculated by counting the number of multi-add operations in the network.
We use a differentiable manner to optimize the dilation architecture. In differentiable NAS, a directed acyclic graph (DAG) is constructed as the supernetwork, whose nodes are latent representations and edges are operations. Given that the adversarial training is computationally intensive, to reduce the search cost, a partial channel connections technique proposed by Xu al. [29] is utilized.
During search, operation candidates for each edge are weighted summed with a softmax distribution of the architecture parameter a:
exp(a (a
(0) 1.
0(h-) (2:)) = (1 S.,) x + Σ
o(Si. * a.)
oEO ENEO exp(a (T))
where O is a set of operation candidates, æ; is the output of the i-th node, and S is binary mask on edge (i, for partial channel connections. The binary mask Si,j is set to 1 or 0 to let the channel be selected or bypassed, respectively. Besides the architecture parameter a, the partial channel connections technique also introduces edge normalization weight G:
exp(B Ever ex)(Brr))
r
= i<j
(9)
where [6] the j-th node. The edge normalization can stabilize differentiable NAS by reducing fluctuation in edge selection after search.
Considering Eqs. 8 and 9, the expected FLOPs of the finally obtained discrete architectures from the one-shot supernetwork can be estimated according to and B.. We calculate the weighted sum of FLOPs of the operation candidates with the identical softmax distributions in Eqs. and 9, which can naturally lead to an expectation. Therefore, the expected FL.PPs of node (() can be calculated by: exp(a!))
exp(Br.)
(0)
FLOPs(16)) = Σ
i-J Ever Exerex((ee)) exp(Br OEO EveO exp(a FLOPs(o).
(10)
After that, the FLOPs of the dilation network FLOPs(fa) can be estimated by taking the sum of the FLOPSs of all the nodes and cells. The objective function in Eq. 3 can be rewritten with the FL.OPs constraint as:
min
7 >log(FLOss(fa))
c(adv) (fagb)
(11)

---

where y and are two coefficient terms. controls the sensitivity of the objective function to the FLOPs constraint, and y scales the constraint to reasonable range (e.g. around 1.0).
3.4 Optimization
We reformulate the bi-level form optimization problem defined in Eqs 3 and 4 into constrained optimization form. Combining with the standard performance constraint in Eq. 7 and the FLOPs- aware objectives in Eq. 11, we have
min y >oog(FL.Ossffa)> <laard (Jasb: (2(a))),
(12)
aa
s.t. clased (fayb) Clated (Fock) S O,
(13)
(aa) argmin Claan (fay)), s.t. (fays) -Crssh (fock) S 0.
(14)
0₂
To solve the constrained architecture optimization problem, we apply common method for con- strained optimization, namely alternating direction method of multipliers (ADMM). To apply ADMM, the objective function needs to be reformulate with the upper-level optimization of the architecture parameter atd-
augmented Lagrangian function. We first deal
L(fau), (A+)) = y log (o((FL(ss(faa)) claar) (fabt) + A h1.a a + ?!l max(0,
(15)
s.t. cl = claad (hys) clared ( (frok),
(16)
where d1 is the Lagrangian multiplier, and p E R. is a positive number predefined in ADMM. We update an and a alternately with:
((+1)
(t) - 11 (
dd + ad
(17)
x((+1) 110 +p.
(18)
where 7 is a learning rate for architecture. Similarly, the lower-level optimization problem of network weights 0n as an augmented Lagrangian function can be defined as:
L((Oa), (22)) = c(
(Snyt) + xa 02 + 2 max 02)
(19)
s.t. c2 ( (fayz) ((asa) (fock), where A2 is the Lagrangian multiplier. Similarly, we can update Oa and Aa with the same alternate manner:
(20)
0u+1)) + o² 72 (
(21)
(((11) 22) p2.
(22)
where 7/2 is the learning rate for network weights.
Theoretical Analysis
In this section, we provide theoretical analysis of our proposed NADAR. As there are two major goals in our optimization problem, i.e., the standard performance constraint and the adversarial robustness, this analysis is also twofold. Firstly, a standard error bound of NADAR is analyzed. We demonstrate that the standard error of the dilated adversarial network can be bounded by the standard error of the backbone network and our standard performance constraint. Secondly, we compare the adversarial error of the dilated adversarial network and the standard error of the backbone standard network. We demonstrate that the adversarial performance can be improved by adding a dilation architecture to the backbone, even if the backbone is fixed. These two error bounds can naturally motivate the optimization problem in Eqs. 12 and 13. Detailed proofs are provided in our supplementary material. Besides, through this analysis, we want to reveal two remarks: (1) enlarging the backbone network with dilation can improve its performance, which proves the validity of our neural architecture dilation; (2) the dilation architecture should be consistent with the backbone on clean samples and samples that are insensitive to attacks, which directly inspires our standard performance constraint.
We discuss the binary classification case for simplification, where the label space is y = {1),+1}. The obtained theoretical results can also be generalized to the multi-class classification case. A binary classification hypothesis h H. is defined as mapping h X IR where H is hypothesis space, and X' is an input space of natural examples. The output of the hypothesis is a real value score. The predicted label can be obtained from the score by applying the sign function sign(-) on it. Denote the backbone hypothesis as hgg By further investigating the influence of the dilation architecture, the hypothesis of the resulting hybrid network can be defined as hays (a: = hs(ar) + ha(a:), where hag stands for the change resulting from the dilation architecture. The standard model corresponds to a
5

---

hypothesis (x) hz(x). We further define the standard ervor of a hypothesis h as
(hta(h) == I2 (sign(((()))) # y}}:
(23)
and the adversarial error of it as
Radr := E [1 E (2(,,)), s.t. sign(h(<²)) # yyll
(24)
where 1.. denotes the indicator function.
4.1 Standard Error Bound
To compare the error of two different hypotheses, we first slightly modify the error function. Eq. 23 checks the condition that h is real value, can remove the sign function by replacing the condition with yh(ax) I 0. Then, (x)
+ y. Because the label space is binary and the output space of
by applying simple inequality 1 (y)((z) S 0} S e -yh(a), we have a very useful inequality about the standard error:
(aa S wh(a))
(25)
Eq. 25 can lead to our standard error bound Theorem 1.
Theorem 1. Let (x)x(())) = ho(x)) be a standard hypothesis, ha)b(x)) = hs(x)) + ha(x)) be hybrid hypothesis, and Rata (ficx) and Rara (Ingb) be the standard error of hacx and lnyb, respectively, For any mapping la, ha X + R. we huve
Rata(rnng)) S Rata (Mtbe)) + E (s()))))(()))
(26)
where a E X is the imput.
Theorem illustrates that the standard performance of the hybrid network bounded by the standard
performance of the backbone network and the sign disagreement between hy and ha()). This reflects our remark (2). If the backbone accurately predicts the label of the natural data æ, (x) shall make the same category prediction, which implies that the prediction by the hybrid hypothesis hyyb (z=) = has (æ) ) + ha())) can be strengthened and would not lead to a worse result than that of the stand hypothesis. To reach such objective, it naturally links with the standard performance constraint proposed in Eq. and applied in Eq. 13.
4.2 Adversarial Error Bound
Similar to Eq. 25, we can have an inequality about the adversarial error:
Radv S E max e eyy((a') (x,,))
(27)
based on which we can derive the following Lemma 2. Lemma 2. For any mapping h : X + R. we have
E
max e -vh((aa) < E
max e - (BEB.l.,e)
(28)
(B,(,..
where x E X is the input, y f-1, +1} is the corresponding label, and is the bound of allowed adversarial perturbation.
Lemma 2 is an inherent feature of a single hypothesis. We generalize it the case of dilating lock to baye with a dilation hypothesis ha.
Theorem 3. Let (mxck(x)= = ha( be a standard hypothesis, Imay = h)((x) + ha((x) be a dilated hypothesis, Reta (hocx) be the standard error of Docks and Ruav (beyb) be the adversarial error of hayb. For any mapping ha, ha X IR, we have
max e vho(()) (e-hulen)helt') (aatia(t""""ap((())))) uha(a") - xe Bp(..s)
Red (hagt)= S Rara(hbez) + E
(29)
where x E X is the input, E adversarial perturbation.
f-1, is the corresponding label, and E is the bound of allowed
By minimizing e (hha(w)) in Theorem 3, we expect the backbone network to have satisfactory accuracy the natural data, which is a prerequisite of the proposed algorithm. As the backbone network has been fixed in this paper, the term (=(())uh((a)) will not be influenced by the algorithm.
The remaining term e- -yhu(a") implies that even if the backbone network makes wrong prediction on the adversarial example æ', there is still chance for the dilation network hy to correct the cisssisssiiiiatoon and improve the overall adversarial accuracy of the hybrid network hayb. This capability of dilation reflects our remark (1). In another case, if hs makes a correct prediction, ha should agree with it, which reflects our remark (2) is also applied the adversarial error.

---

Table 1: The standard validation accuracy on natural images and adversarial validation accuracy
under various attacks of NADAR comparing to different SOTA methods CIFAR-10.
Category
Method
Params (M)
+ (G)
Valid Acc. Against (8)
Back.
Arch.
Back.
Arch.
Natural
RGSM
PGD-20
PGD-100
MI-FGSM
Standard
Standard
46.2
6.7
95.01
0.00
0.00
0.00
0.00
PGD-7 [18]
46.2
6.7
87.25
56.10
45.84
45.29
Adversarial
FAT [33]
46.2
6.7
89.34
65.52
46.13
46.82
Training
FreeAT-8 [22]
46.2
6.7
85.96
46.82
46,19
TRADES-1 [32]
88.64
51.26
TRADES-6 [32]
46.2
6.7
84,92
56.43
57.95
AmoebaNet (20)
3.2
0.5
83.41
56.40
39.47
47.60
Standard
NASNet [35]
3.8
0.6
83.66
55.67
48.02
53.05
NAS
DARTS [17]
3.3
0.5
83.75
55.75
44.91
$1.63
PC-DARTS
0.6
83.94
52.67
41.92
49.09
RobNet-small [8]
4.4
N/A
78.05
53,93
48.32
48.08
48.98
Robust NAS
RohNet- medium [8]
N/A
78.33
54.55
49.13
48.96
49.34
RobNet-large
N/A
78.57
54,98
49.44
49.24
49.92
RACL [5]
3.6
0.5
83.89
57.44
49.34
54.73
Dilation
NADAR-A (ours)
46.2
0.6
86.61
59,98
52.84
52.54
57.72
NADAR-B (ours)
46.2
4.4
6.7
0.7
86.23
60.46
53.43
53.06
58.43
5 Experiments
We perform extensive experiments to demonstrate that NADAR can improve the adversarial robustness of neural networks by dilating the neural architecture. In this section, we first compare both the standard and adversarial accuracy of our hybrid network to various state-of-the-ar (SOTA) methods. Then, we perform experiments to analyze the impact of each component in the NADAR framework, including the dilation -ased training approach and the standard performance constraint. Finally, we explore the sufficient scale of dilation and the effect FLOPs constraint. More results on other datasets under various attacking manners with different backbones are also available in the supplementary material.
5.1 Experiment Setting
We use similar pipeline to previous NAS works [17, 29, 5, 8]. Firstly, we optimize the dilating architecture in one-shot model. Then, discrete architecture is derived according to the architecture parameters a and B.. Finally, discrete network constructed and retrained for validation. During the dilation phase, the training set is split into two equal parts. One is used as the training set for network weights optimization, and the other one is used as the validation set for architecture parameter optimization. During the retraining and validation phases, the entire training set is used for training, and the trained network is validated on the original validation set.
We perform dilation under white- box attacks on CIAFR-10/101 [10] and ImageNet [21] and under black-box attacks on CIFATR-10. The NADAR framework requires a backbone to dilated. Following previous works [18, 22, 33, 32), we use the 10 times wider variant of ResNet, i.e. the Wide ResNet 30-10 (WRN34-10) [31], on both CIFAR dataset, use ResNer-50 [9] on ImageNet. The search space of dilated architecture and the dilated architectures are as illustrated in Section A of the supplementary material.
Considering both the optimization of neural architecture and the generation of adversarial examples is computational intensive, we apply methods reduce the computational overhead during the dilation phase. As aforementioned, we utilize partial channel connections to reduce the cost of architecture optimization. As for the adversarial training during search, we use FreeAT [22], which recycles gradients during training for the generation of adversarial examples and reduces the training cost.
5.2 Defense Against White-box Attacks
CIFAR-10. We compare the hybrid network with 4 categories of SOTA methods, including standard training, adversarial training, standard NAS, and robust NAS. The standard training method and all the adversarial training methods use the WRN34-10. For the standard NAS methods, the architecture the best architecture searched with standard training as reported their papers and is retrained with PGD-7 [18]. For the adversarial NAS methods, follow their original setting. We include two best dilation architectures obtained with our method, NADAR-A A and NADAR- B dilated without and with the FLOPs constraint, respectively. The architectures are visualized in Section A of the supplementary material. Our architectures are also retrained with PGD-7. In Table 1, the standard accuracy natural images and the adversarial accuracy under PGD-20 attack reported.
Comparing NADAR-A and NADAR-B, the FL.OPs constraint can obviously reduce the FL.OPs number (reducing by 14.28%) well as the parameters number (reducing by 18, 19%) of the dilation
7

---

Table 2: The standard and adver- sarial validation CIFAR- 100.
Table 3: The standard and adversarial validation accuracy on Imay-ImageNet with ResNet-50 as backbone.
Valid Against (%)
Training Method
GPU Days
Valid Acc.
Against (1)
Method
Architecture
Natural
PGD-20
Natural
FGSM
PGD-10
PGD-20
Standard
78.84
0.00
Backbone Dilation (ours)
PGD-4
0.19
43.23
24.13
22.25
22.16
PGD-7 [18]
23.20
PGD4
0.45
44.37
24.33
22.69
22.70
FreeAT-8 [22]
62.13
25.88
Backbone
FreeAT-4
0.05
42.73
24.10
22.67
22.58
lobNet-large [8]
23.19
Dilation (ours)
FreeAT-4
0.12
44.68
24.66
22.88
22.78
RACL [S]
27.80
NADAR-A (ours)
61.73
27.77
Backbone
FastAT
0.12
45.92
23.53
20.66
20.54
NADAR-B (ours)
62.56
28.40
Dilation (ours)
FastAT
0.22
46.22
23.90
21.21
21.14
architecture. The negative impact of it on the adversarial accuracy is marginal (only 0.59% under PGD-20 attack), and the standard accuracy can even be slightly improved.
As for adversarial training methods, we improve the adversarial performance by 7.59% with only 1.02% standard performance drop comparing to PGD-7 while we use the same training method but the hybrid network. This result illustrates that our dilation architecture can indeed improve the robustness of a network without modifying the training method. At the meantime, the standard accuracy is constrained to competitive level. Comparing to FreeAT-88 which is their best adversarial setting, our method reaches both lower standard accuracy drop and higher adversarial accuracy gain than it.
very different method PGD and FreeAT, namely friendly adversarial training (FAT), aims to improve the standard accuracy of adversarially trained models by generating weaker adversarial examples than regular adversarial training. Although FAT can make significant improvement on standard accuracy with weak attack, its adversarial accuracy gain against PGD-7 is marginal (only 0.29% It even has lower adversarial accuracy than FreeAT-8 despite its higher standard accuracy. Unlike FAT, our method is dedicated to another direction of improvement, which improves the adversarial robustness without significantly affecting the standard performance. Even though there is still a trade-off between the standard and the adversarial accuracy, we can increase the ratio of the standard drop to the adversarial gain to 1 7.44.
A previous work focuses on the trade-off is TRADES, which introduces a tuning parameter (A) to adjust the balance of the trade-off. Nevertheless, comparing the TRADES-1 (1/A = 1) and TRADES-6 (1/X= = 6), their trade-off ratio is only 1 : 2.02 (i.e. 3.72% standard accuracy drop for 7.53% adversarial accuracy gain). We can provide better ratio of trade-off than them. Besides, our standard performance is naturally constrained by Eq. 7. There are no hyperparameters in it that needs to be adjusted, which leads to our better trade-off ratio of the standard drop to the adversarial gain and more reasonable balance than TRADES.
Finally, comparing to NAS methods, our hybrid network can outperform both the standard and robust NAS architectures. The standard NAS architectures are not optimized for adversarial robustness. Except the NASNet, their adversarial accuracies are generally poor. Although they are optimized for standard tasks, their standard accuracy after adversarial training is significantly lower than the WRN34-10 10 trained with PGD-7. As for robust NAS methods, RobNet significantly sacrifices their standard accuracy for robustness, which has the lowest standard accuracy among all the works listed in Table 1. RACL has a better trade-off, but it can only reach the standard accuracy of standard NAS architecture, which is still lower than adversarilly trained WRN34-10. This demonstrates that dilating a standard backbone for both standard constraint and adversarial gain is more effectiveness than design new architecture from scratch.
CIFAR-100. We adapt architectures dilated on CIFAR-10 to CIFAR- 100, and report the results in Table 2. We consider two kinds of baselines, including traditional adversarial training methods (PGD-7 and FreeAT-8) and two robust NAS methods (RobNet and RACL). The results show that even with more categories, NADAR can still reach superior robustness under PGD-20 attack. As for the standard validation accuracy, we can reach competitive performance comparing to FreeAT-8. The other works do not report standard accuracy in their papers. As for the adversarial validation accuracy, our NADAR-B can outperform all the baselines, while NADAR-A is slightly lower than RACL but significantly better than the others.
Tmay-Ima We also adapt our architectures to a larger dataset, namely Tiny- ImageNet.. For efficient training on ImageNe,,,,,t, we compare our dilated architecture with PGD and two efficient adversarial training method, i.e. FreeAT [22] and FastAT [28]. We follow the ImageNet setting of
Shafahi et al. [22] and Wong et al. [28], which uses ResNet-50 as the backbone and set the clip size
e <= 4. For PGD and FreeAT, we set the number of steps K = 4 and the step size eş = The results
reported in Table 3. We also report the GPU days cost to train the networks with NVIDIA V100
8

---

GPU, Although NADAR consumes approximately 1.8~2.4X GPU days, our method can consistently outperform the baselines in terms of both natural and adversarial accuracy.
5.3 Defense Against AutoAttack
Beside the traditional attack methods, we also consider a novel and promising parameter-free evaluation method, namely AutoAttack [4]. We use the standard setting of AutoAttack, including four individual attacks: AFGD APGD FAB" and Square. The column AA is a combination of the four attacks. The valida- tion accuracy are reported in Table 4. As can be seen, our method reaches superior performance than the baselines. For simplification, PGD-7, FastATT and FreeAT-8 as follow: under APGDDE attack, we can outperform FastAT by 6.74%; outperform
Table 4: The adversarial validation accuracy of NADAR comparing to different SOTA methods under AutoAttack on CIFAR-10. 10.
Crogrey
Matuoll
Valid Acc. Agaisst
APCDDxx
FAAR"
Square
Advamorii Truning
FalAT [25]
PCa7-7 [1:g
45.90
46:75
43.22
64.28
43.74
64.75
53.32
53.00
43.21
46:04
PeoeAES [77)
43.66
41.64
43.44
51.05
41:47
Dilation
NADAR-B (ours)
NADAR-A tosmm)
52.27
52.64
50:45
30.00
50.88
59.33
49.83
50.44
only list the comparison to the best performance among
under APGD bLR PGD-7 by 6. 13%; under Square attack, we can outperform FastAT by 6.01% %.
DLR. we can outperform PGD-7 by 6.17% under FAB attacks
5.4 Defense Against Black-box Attacks
We perform black-box attacks on CIFAR-10. We use different source networks to generate adversarial ex- amples. For the source networks, we use the WRN34-10 backbone trained with natural images and adversarial images generated with FGSM and
Table 5: The adversarial validation accuracy under black-box attacks on CIFAR-10.
Datense Nerwork
Sonrce Network
Valid Ace.
FGSM
PGL-20
PGD-100
MI-FGSM
WRN36-10 10 PGD-7
WRNM4-10
Natural
83,99
84.56
84.76
$1.05
NADAR-B PGD-7
WRN34-10 Natural
85.94
86.51
85.95
WRN34-10 PGD-7
WRN34-10 FGSM
68.30
69:73
NADAR-B PGD-7
WRN34-10 PGSM
77.25
77.66
77.09
77.19
PGD-7. For the defense networks, we
WRN34-10 PGD-7
NADAR-B + PGD-7
69.33
67.08
67.11
68. 68.26
compare our best NADAR-B B architec-
NADAR-B B PGD-7
WRN24-10 10 PGD-7
70.78
68:25
68.30
09:73
ture with the plain WRN34-10 backbone. Both of them are trained with PGD-7. The results are reported in Table 5 grouped according to source networks. With the WRN34-10 source network trained with natural images and FGSM, NADAR-B can consistently outperform the backbone. We also use NADAR-B trained with PGD-7 and WRN34-10 10 trained with PGD-7 to attack each other. Our hybrid network can consistently reach superior performances.
5.5 NADAR Trained with Different Adversarial Training Methods
In previous experiments, our NADAR trained with PGD-7, and other competitors are trained according to their corresponding settings. To further investigate whether NADAR can work along with other stronger adversarial training methods than the PGD-7, we train the ob- tained architecture with various adversarial train- ing methods, including FAT, TRADES-1, 1, and TRADES-6. The results are shown in Table 6. As can be seen, our method can consistently out- perform the backbone in terms of adversarial ac- curacy. Regarding natural accuracy, NADAR is competitive to the backbone (only 0.52% lower in average, which is a marginal drop, given the magnitude of robustness improvement). Especially, under the TRADES-I setting, which focuses on natural accuracy, NADAR can outperform its back- bone on both natural accuracy (89.77% vs. 88.64%) and adversarial accuracy (55.13% vs. 48.90% under PGD-20 and 50.90% vs. 43.01% under Auto/ttack). In the meantime, its adversarial accuracy (55.13%) much closer to the one of TRADES-6 (56.45 43)),, which focuses on robustness and has low (4.85% lower) natural accuracy, than to the one of TRADES-1 (48.90%), which has a similar natural accuracy.
Table 6: Comparison of test accuracy of NADAR and WRN34-10 10 backbone when using various AT methods for training.
Model
AT Mothod
Valid Acc. Against (%)
Natural
PGD-20
AA
Backhone
PGD-7
87.25
45.84
44
NADAR (ours)
PGD-7
86.23
53.43147391
50.44
Backbone
FAT
89,34
46.13
N/A
NADAR (ours)
FAT
88.12
54.53/1840)
51.37NN
Backhone
TRADES-1
88.64
48.90
43.01
NADAR (ours)
TRADES-1
89.77
55.1314670
50.90
Backhone
TRADES-6 6
84.92
56.43
$3.08
NADAR (ours)
6
83.94
57.43
55,25
5.6 Ablation Study of Dilation Method
We perform ablation study of the dilation method. There are two crucial components in our method. Firstly, the separate optimization objectives of standard and adversarial tasks can ensure the that the backbone focus on clear images, and the dilation network learns to improve the robustness of the backbone. Secondly, the standard performance constraint prevents the dilation network from

---

harming the standard performance of the backbone network. This experiment demonstrates that both of them make crucial contributions the final results. Note that without the separate objectives, the hybrid network is trained as a whole. Therefore, there is also no standard constraint. We use the same settings to Section 5.2.
The standard and adversarial accuracy of the obtained networks by retraining is reported in Table 7. there is no standard performance constraint, dilating together or separately has the similar standard performance. Although the backbone of the latter is trained with standard objective, won't 't influence the retraining results too much (only 0.6% higher in average). The complete framework with standard constraint reaches the best standard accuracy after retraining. As for the adversarial accuracy, dilating with the separate objectives can consistently outperforms dilating with a single adversarial objectives.
Table 7: The standard and adversarial ac- curacy by retraining of various networks dilated with ablated manners.
Separate Objectives
Standard Constraint
Valid Acc. Against
PGD-20
No Yes
NIA
84.1920.32
45.9720.18
No
84.7910.55
48.53.1..33
Yes
85.97-0.26
53.18+0.255
6 Conclusion
The trade-off between accuracy and robustness is considered as an inherent property of neural networks. which cannot be easily bypassed with adversarial training or robust NAS. In this paper, we propose to dilate the architecture of neural networks to increase the adversarial robustness while maintaining a competitive standard accuracy with a straightforward constraint. The framework is called neural architecture dilation for adversarial robustness (NADAR). Extensive experiments demonstrate that NADAR can effectively improve the robustness of neural networks and can reach a better trade-off ratio than existing methods.
Acknowledgments
The authors would like thank the area chairs and the reviewers for their constructive comments. This work was supported in part by the Australian Research Council under projects DE180101438 and DP210101859.
References
[1] B. Baker, O. Gupta. N. Naik, and Raskar. Designing neural network architectures using reinforcement learning. arXiv preprint
2016.
[2] Q.-z Cai, M. Du, C. Liu, and D. Song. Curriculum adversarial training. arXiv preprint 1805:088,,, 2018.
[3] M. Cisse, Bojanowski, Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. International Conference on Machine Learning, pages 854-863. PMIRR, 2017.
[4] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International Conference on Muchine Learning, pages 2206-2216. PMLR, 2020.
[5] M. Dong, Y. Li, Y. Wang, and C. Xu. Adversarially robust neural architectures. arXiv:2009.0m 00902, 2020.
arXiv preprint
[6] F. Farnia, M. Zhang, and D. Tse. Generalizable adversarial training via spectral normalization. arXiv preprint arXiv: /8/ 1.07457, 2018.
[7] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint wwXiv:1412.657777 2014.
[8] Guo, Yang, R. Xu, Z. Liu, and D. Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 631 640, 2020.
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] A. Krizhevsky, G. Hinton, et al. Learning multiple layers features from tiny images. Technical report, Citeseer, 2009.
[11] A. Krizhevsky, 1. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
10

---

[12] Y. LeCun, B. Boser. J. S. Denker, D. Henderson, R. E. Howard. W. Hubbard. and L. D. Jackel. Backpropa- gation applied to handwritten zip code recognition. Neural computation, 144):541-551, 1989.
[13] Y. LeCun, I.. Jackel, I.. Bottou, C. Cortes, J. S. Denker, H. Drucker, I. Guyon, U. A. Muller, E. Sackinger, P. Simard, et al. Learning algorithms for classification: A comparison handwritten digit recognition. Neural networks: the statistical mechanics perspective, 261:276, 1995.
[14] Y. Li, M. Dong. Y. Wang, and C. Xu. Neural architecture search in proxy validation loss landscape. In Proceedings of the International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5853-5862. PMLR, 2020.
[15] Y. Li, Yang, Y. Wang, and C. Xu. Adapting neural architectures between domains. In Advances in Neural Information Processing Systems, 2020.
[16] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436. 2017.
[17] H. Liu, K. Simonyan, and Y. Yang. Darts: vrXiv:1806.09055 2018.
Differentiable architecture search. arXiv preprint
[18] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to 2017.
adversarial attacks. arXiv preprint
[19] E. Real, S. Moore, Selle, S. Saxena, Y. Suematsu, J. Tam, Q. V. Le, and A. Kurakin. Large- scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2902-2911. I. JMLR. org, 2017.
[20] E. Real, A. Aggarwal, Y. Huang, and Q. V. Lc. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4780-4789, 2019.
[21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)21112552 2015.
[22] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pages 3358-3369, 2019.
[23] C. Szegedy, Liu, Y. Jia, Sermanet, Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[24] F. Tramèr. A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint 170.:0705500704.22 2017.
[25] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv: vexxiv1805... 12/52, 2018.
[26] T.W.. Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su, Y. Gao, C.-J. Hsich, and L. Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. arXiv preprint 10XX::8801.1077 2018.
[27] E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5286-5295. PMIR, 2018.
[28] E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 2020.
[29] Y. Xu, I.. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong. Pc-darts: Partial channel connections for memory-efficient differentiable architecture search. arXiv preprint 1007.0:977005737 2019.
[30] Z. Yang, Y. Wang. X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu. Cars: Continuous evolution for efficient neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1826-1835, 2020.
[31] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 161.::6605.00114 2016.
[32] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.08572 2019.
11

---

[33] J. Zhang. X. Xu, B. Han, G. Niu, L. Cui, M. Sugiyama, and M. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In International Conference Machine Learning, pages 11278-11287 PMLR, 2020.
[34] J. Zhang. J. Zhu, G. Niu, B. Han, M. Sugiyama, and M. S. Kankanhalli. Geometry-aware instance- reweighted adversarial training, Proceeding of International Conference on Learning Representations. 2021.
[35] B. Zoph, Vasudevan, Shlens, and Q. Learning transferable architectures scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697 8710, 2018.
12


Reviews:
[{"review_id": "rgwcVoa8F2F", "reviewer": "Reviewer_DKEh", "summary": "The paper proposes a neural architecture search that learns a residual (called dilation in the paper) to an existing network in order to improve the robustness to adversarial attacks. The loss of the search is based on improving the robustness to the attack and retaining the original network accuracy. They show improvement for wideresnet for various datasets. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using NAS for improving robustness is very interesting. Yet, as the authors mention, it was already done before. \nWhile this does not eliminate the contribution of the paper, there are several problems in the current form of the paper that should be addressed before it fits for publication:\n\n1. Theory: The authors presents some results that are supposed to support the proposed approach. Yet, these results are very straight forward to achieve so they are not really novel. But even if they are novel, they don't really analyze the proposed approach. First, They give only an upper bound to an upper bound of the loss. Second, they analyze a different architecture than the one used, which is basically a summation of two networks, which is not the case in this work. So, therefore, the theory part is not really relevant or explains the success. So right now it does not really support the paper results but just present some inequalities that I am not sure if they are helpful to something else or not. This should be shown.\n\n2. The empirical results are limited due to the following two reasons:\na. In any architecture search there should be comparison to the best random architecture selected from the search space. This comparison is missing. \nb. There is no comparison to adversarial training of other searched architectures. The table in the paper just show their results without adversarial training. This is not a fair comparison.\nc. Is the improvement just because of  the search or because of the adversarial training? Did the authors tried to use TRADES also for the other NAS based approaches for adversarial training?\nd. Why not all results are reported for DARTS?\n\n3. Ablation should be done for searching for the whole architecture with the current loss with the same number of parameters?\n\n4. What if you use a larger resnet that has the same number of params like the resnet+search and train it with DARTS?\n\n5. Finally, in the paper Intriguing properties of adversarial training at scale (ICLR 2020) it is shown that using larger networks improve adversarial training. So basically, is not this just explain the reason why the adversarial training works better here as the network is bigger? All the experiments I suggested above should be done to prove that the success achieved is not merely due to using a larger network but that the specific used search is really the source of improvement \n\n\n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "not related", "review_text": "", "extra_scores": {"ethical_concerns": "nothing", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "lkBi1nfS02p", "reviewer": "Reviewer_babp", "summary": "This paper proposes a method termed Neural Architecture Dilation for Adversarial Robustness  (NADAR) which adversarially searches for a dilation architecture for the backbone network. Beside the dilation, a standard performance constraint is also introduced to protect the standard accuracy on clear samples form dropping too much, and a FLOPs-awareness mechanism is utilized to lower computational overhead. Theoretical analyses and emperical results validate that NADAR can improve adversarial robustness at almost no cost of generalization.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThe paper is well organized. Studying adversarial robustness from the perspective of neural architecture dilation is novel to solve the well-known accuracy-robustness trade-off, and is orthogonal to the adversarial training methods studied from the view of objective and optimizer.\n2.\tComprehensive experimental results show that the dilation can effectively improve adversarial robustness, and the generalization performance constraint can guarantee a competitive standard accuracy. Those experimental results can support their main claims. \n\nCons:\n1.\tIn Table 1, please explain the meaning of “Arch.” and “Back.”. It is better to clearly show the percentage of capacity has been increased by NADAR compared to the backbone network. \n2.\tIt is better to report robust accuracy against stronger attacks, e.g., CW attacks[1] and AutoAttack, in Table 1-3. \n3.\tThe paper only discusses how to apply NADAR on ResNet-like backbones. It seems to be a limitation of this paper. Please provide some discussions about utilizing the proposed method based on the backbones those do not fit the ResNet-like architecture.\n\n[1] Towards Evaluating the Robustness of Neural Networks. SP 2017", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: Please carefully refer to Main Review.\n\nPotential negative societal impact: This paper does not discuss the potential negative societal impact. However, I think the proposed method raises computational cost and requires larger storage sources, thus posing potential risks to environmental protection.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QLYEP4N8Ag-", "reviewer": "Reviewer_kvfV", "summary": "This paper studied the adversarial robustness from the neural architecture perspective. In particular, the authors introduced a dilation network to increase the overall capacity of the neural network, so that it can simultaneously handle the clean examples and adversarial examples very well. A performance constraint and a FLOPs constraint are developed to guarantee the introduced dilation network can indeed benefit the accuracy and robustness with the minimal FLOPs involved. The authors have done extensive experiments to evaluate the proposed algorithm. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In Line 121, the authors explained that the dilation network has been searched within a NASNet-like search space. In fact, there are quite a few different search spaces in the literature. Why the authors adopt such a search space? Further, is that necessary to customize a certain search space for the problem studied in this paper?\n\nThe authors suggested that the backbone network has a fixed architecture. But why not also do some modifications to the backbone network?\n\nI agree it is necessary to investigate the FLOPs constraint when designing the dilation network. But why this FLOPs constraint is implemented in the formate like Eq. (11). How about directly include a separated item about FLOPs in the objective function?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing particular", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "MitEEYbORX9", "reviewer": "Reviewer_WsAL", "summary": "This paper focuses on improving the adversarial robustness of the backbone CNNs with good natural accuracy. Different from those\nadversarial training methods, this paper employs the dilation architecture to increase the capacity of the backbone network with minimal computational overhead. This paper provides both theoretical analysis and empirical results.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The authors leverage neural architecture search approach to dilate the backbone network, in order to improve adversarial robustness while maintaining natural accuracy. The dilation has interesting motivation to provide adversarial robustness with minimal computational overhead.\n\nQuality: \n1) My main concern is that the robustness of the proposed method achieved is not significantly better than prior works. I list the detailed suggestions in the Limitations setction.\n     a) I suggest the authors include results under AutoAttack for all tables, since AA is stronger and can provide a more general prospective view. Besides, AA would play the biggest role after combining the four attackers, so the authors are suggested to report the standard version of AA instead of splitting it into four attacks and reporting the performance under each respectively. \n     b) It seems that no improvement is achieved on top of TRADES-6. For example, in Table 1, the average between nat and pgd-20 is 70.675 while the average of NADAR-B is 69.83; in Table 2&3, it is lack of TRADES result on CIFAR-100 and Tiny-ImageNet. \n     c) The authors are suggested to compare the same group of competitors on all datasets and under different scenarios to demonstrate the effectiveness of their proposed method.\n\n2) Another major concern is the computational cost during training.  After reading the experiment part, an intuitive question comes into my mind: what is the benefit of dilating an existing graph? Should the training process be faster than standard adversarial training? The authors are suggested to provide more evidence, either on saving training cost or improving performance, to demonstrate the motivation of their proposed method.\n\nClarity: This paper is well organized and clearly written, which allows an easy read. The abstract and introduction directly point\nout the focused research direction, the main contributions, and the overview of the paper structure.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors addressed the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a neural architecture search method to improve the adversarial robustness of the backbone networks while maintaining good natural accuracy. The key idea is to introduce a dilation architecture to increase the capacity of the backbone network with minimal computational overhead. The paper is well motivated and organized. The new solution is novel and technically sound.

However, there are several concerns on the technical details. 1) The theoretical analysis is not specifically related to the proposed method on dilation of neural networks. While analysis of a general function has some implications, neural networks have their special properties which are studied extensively in the subarea of deep learning theory. The analysis would be strengthened if the analysis could be more specific to the neural network architectures used in the proposed method. 2) The benefits of using bigger networks have been pointed out in the literature, it is not clear whether the performance improvement is due to larger networks or the search procedure or both. More analysis is needed to really show that search architecture is very important even with larger spaces, otherwise, the main contribution would not be strong enough because the idea of using larger networks is not originally proposed in this paper. 3) The proposed method is based on PGD, which is not a very recent adversarial training method. It is unclear whether the proposed method can still maintain the advantage when using more advanced adversarial training methods, especially those that have already considered the accuracy on natural data.

Overall, the paper is novel and interesting. However, given the above concerns, the novelty and significance of the paper will degenerate. Addressing the concerns needs a significant amount of work. Although we think the paper is not ready for NeurIPS in this round, we believe that the paper would be a strong one if the concerns can be well addressed.

Decision: Accept (Poster)