Distributed Adversarial Training to Robustify Deep Neural Networks at Scale
Authors: Gaoyuan Zhang
Songtao Lu
Sijia Liu
Xiangyi Chen
Pin-Yu Chen
Lee Martie
Lior Horesh
Mingyi Hong
Venue:  (2021)
Keywords: Adversarial robustness
min-max
distributed learning

Abstract: Current deep neural networks are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective and popular approach, known as adversarial training, has been shown to mitigate the negative impact of adversarial attacks by virtue of a min-max robust training method. While effective, this approach is difficult to scale well to large models on large datasets (e.g., ImageNet) in general. To address this challenge, we propose distributed adversarial training (DAT), a large-batch adversarial training framework implemented over multiple machines. DAT supports one-shot and iterative attack generation methods, gradient quantization, and training over labeled and unlabeled data. Theoretically, we provide, under standard conditions in the optimization theory, the convergence rate of DAT to the first-order stationary points in general non-convex settings.  Empirically, on ResNet-18 and -50 under CIFAR-10 and ImageNet, we demonstrate that DAT either matches or outperforms state-of-the-art robust accuracies and achieves a graceful training speedup.

Under review as a conference paper at ICLR 2021
DISTRIBUTED ADVERSARIAL TRAINING TO ROBUS- TIFY DEEP NEURAL NETWORKS AT SCALE
Anonymous authors Paper under double-blind review
ABSTRACT
Current deep neural networks are vulnerable to adversarial attacks, where adversar- ial perturbations the inputs can change manipulate classification. To defend against such attacks, an effective and popular approach, known as adversarial training, has been shown to mitigate the negative impact of adversarial attacks by virtue of a min-max robust training method. While effective, this approach is difficult to scale well to large models on large datasets (e.g., ImageNet) in general. To address this challenge, we propose distributed adversarial training (DAT), a large-batch adversarial training framework implemented over multiple machines. DAT supports one-shot and iterative attack generation methods, gradient quanti- zation, and training over labeled and unlabeled data. Theoretically, we provide, under standard conditions in the optimization theory, the convergence rate of DAT to the first-order stationary points in general non-convex settings. Empirically, ResNet-18 and -50 under CIFAR-10 10 and ImageNet, demonstrate that DAT either matches or outperforms sate-of-thheart robust accuracies and achieves a graceful training speedup.
INTRODUCTION
The rapid increase of research in deep neural networks (DNNs) and their adoption in practice is. in part, owed to the significant breakthroughs made with DNNs in computer vision (Alom et al., 2018)) Yet, with the apparent power of DNNs, there remains serious weakness of robustness. That is, DNNs can easily be manipulated (by an adversary) to output drastically different classifications and can be done so in controlled and directed way. This process is known as an adversarial attack and considered as one of the major hurdles in using DNNs in security critical and real-world applications (Goodiellow et al. 2013 Szegedy et al. 2013 Carlini & Wagner 2017 Papernot et al. 2016 Kurakin
ct al. 2016 Eykholt et al. 2018 Xu et al. 2019b).
Methods to train DNNs being robust against adversarial attacks now major focus in research
(Xu et al. 2019a). But most are far from satisfactory (Athalye et al. 2018)) with the exception of the adversarial training (AT) approach (Madry al. 2017b). AT is min-max robust training method that
minimizes the worst-case training loss at adversarially perturbed examples. AT has inspired a wide
range of state-of-the-arr defenses (Kannan al. 2018 Ross & Doshi- Velez, [2018] Moosavi-Deztooe
ct al. 2019;
Zhang et al. 2019b Wang et al.] 201 Sinha et al. 88] Chen et 2019 Boopathy
al.
2020 Wong Kolter 20
al.] 20188] Stanforth al. 2019) Carmon et al.) (019)]
Shafahi et al. 2019] Zhang et al. 2019a| Wong et al. (2020), which ultimately resort to min max
optimization. However, these methods, together with AT, are generally difficult scale well to large networks on large datasets.
While scaling AT is important, doing so effectively is non-trivial. We find that scaling AT with the direct solution of distributing the data batch across multiple machines may not work and leaves many unanswered questions. First, if the direct solution does not allow for scaling batch size with machines, then it does not speed the process up and leads to significant amount of communication costs (considering that the number of training iterations is not reduced over a fixed number of epochs). Second, without proper design, the direct application of a large batch size distributed adversarial training introduces a significant loss in both normal accuracy and adversarial robustness (c.g., more than 10% performance drops for ResNet-18 CIFAR-10 shown by our experiments). Third, the

---

Under review as a conference paper at ICLR 2021
direct approach does not confer a general algorithmic framework, which is needed in order to support different variants of AT, large-batch batch optimization, and efficient communication.
Taking all factors into consideration, a question that naturally arises is: Can we speed up AT by leveraging distributed learning with full utility of multiple computing nodes (machines), even when each only has access to limited GPU resources? Although a few works made empirical efforts to
scale AT up by simply using multiple computing nodes (Xie et al.) 2019; Kang et al.] 2019 Qin al. 2019). they were limited specific use cases and lacked thorough study on when and how distributed learning helps, either in theory or in practice. By contrast, we propose a principled and theoretically ggguunedd distributed (arge-batch) adversarial training (DAT) framework by making full use of the computing capability of multiple data-locality (distributed) machines, and show that DAT expands the capacity of data storage and the computational scalability. We summarize our main contributions as below.
Contributions (i) We provide a general problem formulation of DAT, which supports multiple distributed variants of AT, e.g., supervised AT and cemi-supervised AT. (ii) We propose a principled algorithmic framework for DAT, which, different from conventional AT, supports large-batch DNN training (without losing performance over a fixed number of epochs) and allows the transmission of compressed gradients for efficient communication. (iii) We theoretically quantify the convergence
speed of DAT to the first-order stationary points in general non-convex settings at a rate of O(1/vT)) where T is the total number of iterations. This result matches the standard convergence rate of classic training algorithms, c.g., stochastic gradient descent (SGD). for only the minimization problems. (iv) We make a comprehensive empirical study on DAT, showing that it not only speeds up in training large models on large datasets but also matches (and even exceeds) state-of-the-ar robust accuracies in different attacking and learning scenarios. For example, DAT on ImageNet with 6 x (machines x GPUs per machine) yields 38. 38.45% robust accuracy (comparable to 40.38% from AT) but only requires 16.3 hours training time (6 times larger batch size allowed in DAT), exhibiting 3.1 times faster than AT on a single machine of 6 GPUs. We also make significant effort to evaluate the empirical performance of DAT across different computing configurations at different learning regimes, e.g., semi-supervised learning and transfer learning.
2 RELATED WORK
Training robust classifiers The lack of robustness of DNNs has promoted a rapid expansion of defenses against adversarial attacks, ranging from heuristic defenses to robust (min-max) optimization based approaches. However, many heuristic strategies are easily bypassed by stronger adversaries due to the presence of obfuscated gradients (Athalye et al.) 2018). By contrast, the min-max optimization- based training methods are generally able to offer significant gains in robustness. AT (Madry et al.) 20176), the first known min-mas max optimization-baas based defense, has inspired a wide range of other effective defenses. Examples include adversarial logit pairing (Kannan et al.. 2018). input gradient or curvature regularization (Ross& & Doshi Velez 20188 Moosavi-
et al.) 2019). trade-off between robustness and accuracy (TRADES) (Zhang et al. 2019b), distributionally robust training (Sinha et al.) 2018), dynamic adversarial training (Wang et al. 2019b), robust input attribution regularization (Chen et al. 2019 Boopathy et al. 2020). certifiably robust training (Wong & Kolter 2017 Dvijotham et al. 2018), and semi-supervised robust training (Stanforth et al. Carmon
ct al. 2019).
In particular, some recent works proposed fast but approximate AT algorithms, such as 'free' AT (Shafahi et al. 2019), you only propagate once (YOPO) (Zhang et al.] (019a),, and fast gradient sign method (FGSM) based AT (Wong et al.) 2020). These algorithms achieve speedup in training by simplifying the inner maximization step of AT. Although there is vast literature on min-max optimization based robust training, it is designed for centralized model training and without care about the scalability issue in AT. In (Xie et al. 2019). although distributed version of vanilla AT was implemented via the large- batch SGD al algorithm (Goyal et al.] 2017), it is significantly different from our proposal. Compared to (Xie et al. 2019), we adopt a different distributed training recipe (layer-wise adaptive learning rate method vs. SGD), and make in-depth theoretical analysis of quantifying the convergence rate of DAT.
Distributed model training Distributed optimization has been found to be effective for the standard training of machine learning models (Dean et al.) [2122] Goyal et al. 2017: You et al.] 2019 Chen ctt al.; 2020). In contrast to centralized optimization, distributed learning enables increasing the batch size proportional to the number of computing nodes/machines. However, it is challenging to train
2

---

Under review as a conference paper at ICLR 2021
model via large-batch optimization without incurring accuracy loss compared to the standard training with same number of epochs (Krizhevsky [0144] Keskar et al., 2016). To tackle this challenge, was shown in (You et al., 2017| 2019) that adaptation of learning rates to the increase of the batch size is an essential mean to boost the performance of large- optimization. A layer-wise adaptive learning rate strategy was then proposed to speed up the training as well as preserve the accuracy. Although these works have witnessed several successful applications of distributed learning in training standard image classifiers, they leave the question of how to build robust DNNs with DAT open. In this paper, we show that the power of layer-wise adaptive learning rate also applies to DAT. Since distributed learning introduces
communication overhead, another line of work (Alistarh et al. 2017 Yu et al. 2019: Bernstein et al. 2018 Wangni et al. 201 8; Stich et al. 2018: Wang et al. 2019a) focused on the design of communication-efficie distributed optimization algorithms.
The study on distributed learning is extensive, but the problem of distributed min-mas max optimization is less explored, with some exceptions (Srivastava et al. 2011 Notarnicola et al. 2018: Hanada ct al. 2017; Tsaknakis et al,, 2020; Liu et 2011
key difference to our work is that none of the aforementioned literature studied the large-batch min- mix--mxx optimization with its applications to training robust DNNs, neither theoretically nor empirically. While there are recent proposed algorithms for training Generative Adversarial Nets (GANs) (Liu et al.) 2019a b). training robust DNNs against adversarial examples is intrinsically different from GAN training. In particular, training robust DNNs requires inner maximization with respect to each training data rather than empirical maximization with respect to model parameters. Such an essential difference leads to different optimization goals, algorithms, convergence analyses and implementations.
PROBLEM FORMULATION
In this section, we first review the standard setup of adversarial training (AT) Madry et al., 2017b). We then motivate the need of distributed AT (DAT) and propose a general min-max setup for DAT.
Adversarial training AT Madry et al. 2017b) min-max optimization method for training robust ML/DI models against adversarial examples (Goodfellow et al. 2015). Formally, AT solves the problem
minimize (x
maximize E(0, 8:1) 11
(1)
where 0 E IRd denotes the vector of model parameters, 8 E th is the vector of input perturbations within an lo ball of the given radius e, namely, ||0]|so S e, (x, y) E DD corresponds to the training example with label y in the dataset D, and represents pre-defined training loss, e.g., the cross- entropy loss. The rationale behind problem (T) is that the model is robustly trained against the worst-case loss induced by the adversarially perturbed samples. It is worth noting that the AT problem (T) is different from conventional stochastic min-max optimization problems (e.g., GANs training (Goodfellow al. 2014)). Note that in (I). the stochastic sampling corresponding to the expectation over yx E D is conducted prior to the inner maximization operation. Such a difference leads to the sample-specife adversarial perturbation 8(x) == maximize |8 11x f(0, x + 8;
Distributed AT (DAT) The need of AT in distributed setting arises from at least the following two aspects: 1) Training data are distributed, provided by multiple parties, which expands the individual capability of data storage or data privacy. 2) Computing units are often distributed, provided by distributed machines, which enables large-batch optimization and thus improves the AT's scalability.
Let us consider a popular parameter-server model of distributed learning (Dean et al. 2012 Formally, there exist M workers each of which has access to a local dataset D(1,, and thus D لM D())
There also exists a server/master node (e.g.. one of workers could perform as server), which collects local information (e.g., individual gradients) from the other workers to update the model parameters 0. Spurred by (I) DAT solves problems of the following generic form,
M
minimize 1
Σ
XB (x [e(0;x, v)] + E E
maximize o(0, 8; v)
M
18 oe
(2)
(: (())
3

---

Under review as a conference paper at ICLR 2021
where fi denotes the local cost function at the ith worker, & is robustness regularizer against the input perturbation 6, and X 2 is a regularization parameter that strikes a balance between the training loss and the worst-case robustness regularization. In 2 M = 1, D(1) DD À = 0 and
f, then the DAT problem reduces to the AT problem ① We cover two categories of (2):
DAT with labeled data: In 22. we consider Φ(0, 6; x, y) = {0, x 6; with labeled training data (x, y) E DD69 for E [M. Here [M] denotes the integer set {1, 2, ,,,.,
DAT with unlabeled data: In ( different from DAT with labeled data, we have D( with an unlabeled dataset ()(() (namely, 4 C D()), and define the robust regularizer & as (Stanforth et al.
2019 Zhang et al. 2019b):
0, 5: CF(2(x 0; (x; 0)).
Here zx;; 0) represents the probability distribution over class labels predicted by the model 0, and CE denotes the cross-entropy function.
A UNIFIED ALGORITHMEE FRAMEWORK FOR DAT
In this section, we will introduce the algorithmic framework of DAT, where we address three main challenges of algorithm design in DAT: a) inner maximization; b) gradient quantization/compresiion and c) outer large-batch training by leveraging state-of-the-an adversarial defense and distributed optimization techniques. We also achieve the first. theoretical convergence rate result of min-max optimization based robust training distributed large- setting.
We first discuss the key components of DAT and their respective roles in performing scalable training; see the meta-form of DAT Algorithm (or detailed Algorithm AT). The proposed DAT contains three algorithmic blocks. In first block (Steps 3-8 of Algorithm Al]. every distributed worker calls for a maximization oracle to obtain the adversarial perturbation for each sample within a data batch, then computes the gradient of the local cost function f. in (2) with respect to (w..t.) model parameters 0. And every worker is allowed to quantize/comprees the local gradient prior to transmission to a server. In the second block (Steps 9- of Algorithm/AI), the server aggregates the local gradients, and transmits the aggregated gradient (or the quantized gradient) to the other workers. In the third block (Steps 11-13 of Algorithm[AI] the model parameters are eventually updated by a minimization oracle at each worker based on the received gradient information from the server.
In contrast to standard AT, DAT allows for using a M times larger batch size to update the model parameters O. Thus, given the same number of epochs, DAT takes M fewer gradient updates than AT. addition, distributed learning intro- duces communication overhead. To address this issue, it is optional to perform gradient quantiza- tion at both worker and server sides when very large model is possibly trained. We elaborate on DAT in what follows.
Algorithm 1 Meta-form of DAT Algorithm A1
1: for Iteration f = ..,
2: for Worker = 1,2,...... M
Block 1
3: Sample-wit attack generation (AI)
4: Local gradient computation (22)
5: Worker-server communication (optional)
6: end for
7: Gradient aggregation server (A3) > Block
8: Server-oorker communication (optional)
9: for Worker 1, 2, .M
Block
10: Model parameter update A4
Inner maximization: Iterative and one-shot solutions In DAT, each worker calls for an inner maximization oracle to generate adversarial perturbations (Step of Algorithm 1). We consider two solvers of perturbation generation: iterative projected gradient descent (PGD) method used in standard AT (Madry et al. 2017a) and one-shot fast gradient sign method (FGSM) (Goodfellow et al. 2015). We specify attack generation in the unified form
11: end for
(1) (x) zK, 2 [[ [Me-1 + sign( Va (0, & E [K],
(4)
where K is the total number of iterations in the inner loop, the cases of = 1 and K > correspond to iterative PGD attack and FGSM attack respectively, 2₃ denotes the PGD update of 5 at the kth iteration, 2o is a given intial point, II (-)) denotes the projection onto the box constraint [-c, cjd, a > 0 is a given step size, and sign( (-) denotes the element- wise sign operation. The recent work (Wong et al. 2020) showed that if FGSM is conducted with random initialization 2g and proper step
4

---

Under review as a conference paper at ICLR 2021
size, e.g., 1.25c, then FGSM can be effective as iterative PGD in robust training. Indeed, will show in Sec. that the effectiveness of our proposed DAT-FGSM algorithm echoes the finding in Wong et al. 2020). We remark that other techniques (Shafahi et al. 2019 Zhang et al. 2019a) can also be used to simplify inner maximization, however, we focus on FGSM since it is the most
computationally-light
Gradient quantization In contrast to standard AT, DAT requires worker-server communications (Steps and of Algorithm 1). That is, a Langle-precssinn moating-poinn data type is used, then DAT needs to transmit 32d bits per worker-server communication at each iteration. Here recall that d is the dimension of 0. order to reduce the communication cost, DAT has the option to quantize the transmitted gradients using a fixed number of bits fewer than 32. We specify the gradient quantization operation as the randomized quantizer (Alistarh et al., 2017; Yu et al.) 2019). In Sec. 5 we will show that DAT, combined with gradient quantization, still leads to competitive performance. For example, the robust accuracy of ResNet-50 trained by a 8-bit DAT (performing quantization at Step of Algorithm 1) for ImageNet is just 0.55% lower than the robust accuracy achieved by the 32-bit DAT. It is also worth mentioning that the All-reduce communication protocol can be regarded as a special case of the parameter serveeeer-server setting considered in Algorithm when every worker performs as server. In this case, the communication network becomes fully connected and the server- quantization (Step of Algorithm 1) can be mitigated.
Outer minimization by layerwise adaptive learning rate (LALR) In DAT, the aggregated gra- dient (Step 7 in Algorithm 1) used for updating model parameters (Step 10 in Algorithm 1) is built on the data batch that is M times larger than standard AT. The recent works (You et al. 2019 2017) showed that the use of LALR is the key to succeed in training standard DNNs with large data batch. Spurred by that, we incorporate LALR in DAT. Specifically, the parameter updating operation A in Eq. (A4) is given by
0 = O..; T(10,.cll2) Dt
|| U.s Wi [M].
where 0,, denotes the ith-layer parameters, is the number of layers, u, is a descent direction computed based the first-order information (g.)) =(100,,illa)
min [|00,,||,000 et) ,.) is a layerwise scaling factor of the adaptive learning rate T Q = 0 and Cu = 10 are set in our mi
experiments (see Appendix 4.2 for results on tuning c.). and O, = [0....... OTA.T. In ( the specific form of the descent direction u, is determined by the optimizer employed. For example, if the adaptive momentum (Adam) method is used, then u, is given by the exponential moving average of past gradients scaled by square root of exponential moving averages of squared past gradients (Reddi et al.) 2018 CChen et al. 2018), Such variant of (5) that uses Adam as the base algorithm is also known as LAMB (You et al.) 2019) in standard training. However, was elusive if the advantage of LALR is preserved in latge-batch min-max max optimization. We show in both theory and practice that the use of LALR can significantly boost the performance of DAT with large data batch.
CONVERGENCE ANALYSIS OF DAT
To the best of our knowledge, none of existing work tackled the convergence of DAT and took into account LALR and gradient quantization even in standard AT, although AT has been proved with convergence guarantees (Wang et al., 2019b Gao et al., 2019). DAT needs to quantify the descent errors from multiple sources (namely, gradient estimation, quantization, adaptive learning rate, and inner maximization oracle). In particular, the incorporation of LALR makes our analysis of DAT highly non-trivial. The fundamental challenge lies in the nonlinear coupling between the biased gradient estimate resulted from LALR and the additional error generated from alternating update in AT. In our theoretical results, we show that even in the case where the gradient estimate is function of the AT variables, the estimate bias resulted from the layer-wise wise normalization can still be compensated by increasing the batch-size so that the convergence rate of DAT achieves linear speedup of reducing gradient estimate error w.r.t. the increasing number of computing nodes.
Upon defining (0))= := 1 EM D()))))) in (2). we measure the convergence of DAT by the M
first-order stationarity of W. Prior to convergence analysis, we impose the following assumptions: (A1) (0) is with layer- wise Lipschitz continuous gradients; (A2) o(0, 8;x) in Eq. (AI) is strongly concave with respect to 5 and with Lipschitz continuous gradients; (A3) Stochastic gradient estimate
5

---

Under review as a conference paper at ICLR 2021
in Eq. (22 is unbiased and has bounded variance for each worker denoted by o². Note that the validity of (A2) could be justified from distributional robust optimization (Sinha et al. [2018] Wang et al. 2019b). It is also needed for tractability of analysis. We refer readers Appendix 2.1 for more justifications on our assumptions (A1)-A3). In Theorem[] we present the sub-linear rate of DAT.
Theorem 1. Suppose that assumptions Al-A3 hold. the inner maximizer Eq. (AT) provides approximate solution (i.e., the fz-norm of inner gradient is upper bounded by E), and the learning rate is set by T) OL1/VT), then (0:) generated by DAT yields the following convergence rate
1 T
(
d vd
Σ 1=1
BIVoN(0))]3
o
+mn
)t)).
(6)
T
VMB
46
2
where b denotes the number of quantizzation bits, and = min | vt,i} stands for the smallest batch size per worker.
Proof: Please see Appendix
The eтrоr rate given by 6 involves four terms. The term OLL/VMB) characterizes the benefit of using the large per-worker batch size B and M computing nodes in DAT. It is introduced since the variance of adaptive gradients (i.e., o²) is reduced by a factor 1 1/MB, where 1/M corresponds to the linear speedup by M machines. In (6), the term min 3 2= compressed gradients, and the other two terms imply the dependence on the number of iterations as well as the E-accuracy of the inner maximization oracle. We highlight that our convergence analysis (Theorem I is not barely a combination of -enabled standard training analysis (You
vd arises due to the variance of
et al. 2019 2017) and adversarial training convergence analysis (Wang et al. 2019b Gao et al. 2019 Different from the previous work, we address the fundamental challenges in (a) quantifying
the descent property of the objective value at the presence of multi mouti-source errors during alternating min-max optimization, and (b) deriving the theoretical relationship between large data batch (across distributed machines) and the eventual convergence error of DAT.
5 EXPERIMENTS: DAT FOR ROBUST IMAGE CLASSIFICATION
In this section, we empirically evaluate DAT and show its success in training robust deep neural networks (DNNs) over CIFAR and ImageNet datasets. We measure the performance of DAT in the following four aspects: a) accuracies against clean and adversarial test inputs, b) scalability to multiple computing nodes, c) incorporation of unlabeled data, and d) transferability of pre-trained model by DAT.
DNN models and Datasets We use the DNN models Pre-act ResNet-18 (He al., 2016b) and ResNet-50 (He et al. 2016a) for image classification, where the former is shortened as ResNet-1 88 We train these models under datasets CIFAR-10 and ImageNet, but preserve ResNet-18 for CIFAR- 10 only. We also acquire unlabeled data from 80 Million Tiny Images following (Carmon al.) 2019). When studying pre-trained model's transferability, CIFAR-100 used as a target dataset for down-stream classification.
Computing resources We train a DNN using p computing nodes, each of which contains g GGUs (Nvidia V100 or P100). Nodes are connected with 1Gbps ethernet. A configuration of computing resources is noted by x If p > 1, then the training is conducted in distributed manner. And we split training data into p subsets, each of which is stored at local node. Based on our resources, in the CIFAR experiments, we consider p E {1, 6, 18, 24) machines, each of which has 1 GPU. In the ImageNet experiments, we consider p E {1, machines, each of which has GPUs. Based on our computing resource budget, we are not able to use as many GPUs as (Xie et al.) 2019 Kannan al. 2018). However, as will be evident later, our results clearly demonstrate the advantages of our proposal in applicability and scalability across various computing configurations and adversarial scenarios. In particular, our used batch size 6 x 512 3072 on ImageNet has only access to 36 GPUs. By contrast, Xie et al. (2019) used the batch size 4096 across 128 GPUs.
Training setting
We consider 2 variants of DAT: 1) DAT-PGD, namely, Algorithm with apply- ing (iterative) PGD the inner maximization oracle; and 2) DAT-FGSM, namely, Algorithm AT with

---

Under review as a conference paper at ICLR 2021
of FGSM the inner maximization oracle. Additionally, consider 4 training baselines: 1) AT (Madry et al. 2017b); 2) Fast AT (Wong et al. (020);; 3) DAT wo LALR, namely, direct distributed implementation of AT, which is in the form of DAT-PGD or DAT-FGSM but without considering LALR; and 4) DAT-LSGD (Xie et al. 2019), namely, a distributed implementation of large-batch SGD (LSGGD) for standard AT. We remark that both AT and Fast AT centralized training methods. In our setup, the number of GPUs is limited to 6 at a single machine, and thus the largest batch size that the centralized method can use is around 2048 for CIFAR-IO and 85 for ImageNet in our case. We also find that the direct implementation of Fast-AT AT in a distributed way leads to a quite poor scalability versus the growth of batch size, and thus worse distributed baseline than DAT-FGSM w/o LALR. Lastly, we remark that the work (Xie et al. 2019) proposed modifying model architecture by incorporating feature denoising. In contrast, DAT does not call for architecture modification. Thus, to enable a fair comparison, we use the same training recipe LSGD as (Xie et al.) 2019) in the DAT setting, leading to the considered distributed training baseline DAT-LSGD.
Unless specified otherwise, we choose the training perturbation size 8/255 and 2/ 2/5 for CIFAR and ImageNet respectively, where recall that was defined in T.. We also choose 10 steps and steps for PGD attack generation in DAT (and its variants) under CIFAR and ImageNet. respectively. Such training settings are consistent with previous state-of- oh-the-art (Zhang et al. 2019a Wong et al. 2020). The number of training epochs is given by 100 for CIFAR-TO and 30 for ImageNet. Note that the adversarially robust deep learning could be sensitive to the step size (learning rate) choice (Rice et al.) 2020). For example, the use of cyclic learning rate trick can further accelerate the Fast AT algorithm (Wong et al. 2020). However, such a trick becomes less effective when the batch size becomes larger (namely, the number of iterations gets smaller); see Appendix 4.T Meanwhile, the sensitivity of adversarially model training to step size can be mitigated by using early-stop remedy due to the existence of robust overfitting (Rice et al.) 2020). Spurred by that, we use the standard piecewise decay step size and an early-stop strategy during robust training. We refer readers to Appendix 4.2 for more implementation details.
Evaluation setting For ad- versarial evaluation, we report robust test accuracy (RA) of
Table 1:
Overall performance of DAT (in gray color), compared with baselines, in TA (%), RA (%), communication time per epoch (scconds), and total training time (including communication time) per epoch (scconds). For brevity, 'p q represents nodes # GPUs per node", 'Comm.' represents communication cost, and 'Tr. Time' represents training time.
a learned model against PGD
attacks (Madry et al.] 2017b) and C& attack (Carlini & Wagner 2017 Unless speci- fied otherwise, we choose the
CIFAR- 10, ResNet-18
Method
Pxq
Batch size
TA (%)
RA (%)
Comm.
Tr. time
1x1
2044
82.94
38.54
Fast AT
1x1
2018
81.58
38.34
NA
52
perturbation size same as the
PAT w/o LALR
2048
55.59
26.83
FGSM w/o LALR
18 1
18 x 2048
52.35
28.90
3.1
8
training& in evaluation, and
DAT-LSGD
18x 1
x 2048
64.15
34.12
3.2
22
the number of PGD steps is selected as 20 and 10 for CI- FAR and ImageNet, respec- tively. In addition to RA, we also measure the standard test accuracy (TA) of model against normal examples. All experiments run 3 times with different random seeds, and the mean metrics are reported]] In our experiments, we consider three different communication protocols, Ring-AllReduce (with one-sided quantization), parameter-server (with double quantization), and high performance computing (HPC) setting (without quantization). To measure the communication time, We use TORCH. DISTRIBUTED package with gloo and neel as communication backend We then measure the time of required worker- server communications per epoch. We use time module to measure communication time with TORCH. DISTRIBUTED. BARRIER to synchronize all processes on each node.
DAT-PGD
18x1
X 2018
80,28
38.44
3.4
22
DAT-FOSM
18
X 2048
73.42
38.55
3.1
DAT
24
24 2048
72.76
39.82
2.0
Image Net, ResNet-50
Fast AT
1
512
$8.99
40,78
NA
1544
DAT-PGD w/o LALR
6x6
6 512
57.09
34.02
865
1932
DAT-FGSM wio LALR
6X6
6 x 512
55.04
35.03
863
1080
PAT PGD DAT-FGSM
6
6 x 512
63.75
38.45
898
1960
6
6 x 512
58.32
41.48
859
1109
'Code will he released
htt.aees/pporooo..rrrr/////ttttt htm1
7

---

Under review as a conference paper at ICLR 2021
Overall performance of DAT In T we evaluate our proposals and baselines in TA, RA, communication and computation efficiency. Note that AT and Fast AT are centralized training methods in single node under the same number of epochs distributed training.
We observe that the direct extension from AT to DAT (namely, DAT-PGD w/o LALR) leads to significantly poor TA and RA. As the 18 times larger batch size is used, DAT-PGD w/o LALR yields more than 25% drop in TA and 10% drop in RA compared to the best AT case. We find that the performance of DAT-PGID w/o LALR rapidly degrades as the num- ber of computing nodes increases. The similar conclusion holds for DAT-FGSM w/o LALR vs. Fast AT. Furthermore, we observe that DAT-PGD outperforms DAT-LSGD (ie et al., 2019) with 16.13% and 4.32% im- provement in TA and RA, respectively. In Figure [ we further compare our proposed DAT with the DAT-LSGD baseline in terms of TA/RA versus the number of computing nodes. Clearly, our approach scales more gracefully than the baseline (without losing much performance as the batch sizes increases along the number of computing nodes).
60-
Acarray
DAI LLGGD RA: DATA,SCO
50
TA. DAT FCSH
DAT-FESH
Rd: DAT-41GMM
RA: DAT FSSH
6x2
12x3
15c1
24x1
12.6
Jc5
Figure 1: TA/RA comparison between DAT-FGSM and DAT- LSGD vs. GPU configurations. Left: (CIFAR-10 10, ResNet- 18). Right: (ImageNet, ResNet-50).
Moreover, we consistently observe that DAT-PGD (or DAT-FGSM) is able to achieve competitive performance to AT (or Fast AT) and enables graceful training speedup, e.g., by 3 times using 6 machines for ImageNet. In practice, DAT is not able to achieve linear speed-up mainly because of the communication cost. For example, when comparing the computation time of DAT-PGD (batch size 6 x 512) with that of AT (batch size 512) under ImageNet, the computation speed-up up (by excluding the communication cost) is given by (6022)/ / (1960 898) = 5.67, consistent with the ideal computation gain using 6x larger batch size in DAT-PGD. Furthermore, we observe that when the largest batch size (24 x 2048) is used, DAT-FGSM takes only (500 seconds) to obtain satisfactory robustness.
When comparing DAT-FGSM with DAT- PGD, we consistently observe that the former is capable of offering satisfactory (and even better) RA, but inevitably intro- duces TA loss. This phenomenon also holds for Fast AT versus AT, e.g., 0.4% RA improvement vs. 3.71% TA degrada- tion for ImageNet. We also note that the per-epoch communication time decreases when the more GPU machines (24) are used, since a larger batch size allows a smaller number of iterations per epoch, leading to less frequent communications among machines. Appendix 4.3
60.
AT
AT
DAT PGD DAT-FOSM
5
DAT-POD
50
Rothurt Arcumney
DATAPSOM
40
20
Ex
40
20
Figure 2: RA against PGD attacks for model trained by DAT- PGD, DAT-FGSM, and AT following (ImageNet, ResNet-50) in Table[] (Leftt) RA versus different perturbation sizes (over the divisor 255). (Right) RA versus different steps.
we present additional
results on CIFAR- 10 using ResNet-50.
Robustness against different PGD attacks In Figure 2 we evaluate the adversarial robustness of Re at ImageNet learned by DAT-PGD and DAT-FGSM against PGD attacks of different steps and perturbation sizes (i.e., values of c). We observe that DAT matches robust accuracies of standard AT even against PGD attacks at different values of and steps. We also note that although DAT-FGSM has the worst TA (e = 0), it yields slightly better robustness as attack steps increase. The similar results can be found in Appendix 4.4 for (CIFAR-10. ResNet-18) against PGD (and C&W) attacks.
Table 2: DAT in semi-supervised learning with unlabeled data, where the computing resource configuration and batch size are set the same as 8th and 10th rows (CIFAR- 10. ResNet-18) in Table The relative improvement over RA or TA obtained in supervised learning (CIFAR- 10 only) is marked by red color.
DAT under unlabeled data In Ta- 2 we report TA and RA of DAT in the ceni-supervised setting (Carmon et al.) 2019) with the use of 500K unlabeled im- ages mined from Tiny Images (Carmon ct al. 2019). Compared to the DAT su- pervised learning results (CIFAR-10,
CIEAR-10 + 500K unlabeled Tiny Images, Res-Net-1
Method
(%)
(%)
Comm.
Tr. time
DAI PGD
87.00 7.62)
47.34 (7 8.40)
86
451
DAT-FGSM
88,00 (1 12.42)
45.84 ( 4.92)
86
124
8

---

Under review as a conference paper at ICLR 2021
ResNet- 18) in T we observe that although the communication and computation costs increase due to the use of additional unlabeled images, both TA and RA are significantly improved. In particular, the performance of DAT-FGSM matches that of DAT-PGD. This suggests that unlabeled data might provide a solution to compensate the TA loss induced by FGSM-based robust training algorithms.
DAT from pre-training to fine-tuning In Figure 3 we investigate if DAT pre-trained model (ResNet-50) over a source dataset (ImageNet) can offer a fast fine-tuning to down-stream target dataset (CIFAR-100) 100). Here we up-sample a CIFAR image to the same dimension of an ImageNet image before feeding it into the pre-trained model (Shafahi et al. 2020). Compared with the direct application of DAT to the target dataset (without pre- training), the pre-training enables a fast adaption to the down-stream CIFAR task both TA and RA within just 3 epochs. Thus, the scalability of DAT to large datasets and multiple nodes offers a great potential in the pre-training + fine-tuning paradigm. The similar results can be found in Appendix 4.5 on CIFAR-10.
TA: Pre-trai ning on ImageNet
Pro-training imageNet
Accuracy (TA or RA)
TA: No Pre-training
100-
RA: No Pre training
Fine suuninnnn opochs
Figure 3: Fine-tuning ResNet-50 (pre-trainee trained on ImageNet) under CIFAR-100. Here DAT- PGD is used for hoth pre-training and fine- tuning 6 nodes with batch size 6 x 128.
Quantization effect In Appendix 4.6 we also study how the performance of DAT is affected by gradient quantization. We find that when the number of bits is reduced from 32 to 8, the resulting TA and RA becomes worse than the best 32-bit case. For example, in the worst case (8-bit 2-sided quantization) of CIFAR-IO, TA drops 1.52% and 6.32% for DAT-PGD and DAT-FGSM, respectively. And RA drops 4.74% and 5.58%, respectively. Note that our main communication configuration is given by Ging-AllReduce that calls for 1-sided (rather than 2-sided) quantization. We also observe that DAT-FGSM is more sensitive to effect of gradient quantization than DAT-PGD. Even in the centralized setting, the use of 8-bit quantization can lead to a non-trivial drop in TA (see Table(A5). However, the use of quantization reduces the amount of data transmission per iteration. We also show that if a high performance computing cluster of nodes (with NVLLink high-speed GPU interconnect (oleyy & Danskin 2017)) used, the communication cost can be further reduced.
LALR vs. centralized and distributed solu- tions We examine the effect LALR on both centralized and distributed robust training meth- ods given a batch size that is affordable to a single machine. We consider variant of AT by incorporating LALR, termed as AT w/ LALR. As presented in Table when the batch size is not large, both centralized and distributed meth- ods lead very similar performance although the former is slightly better as it is free of machine synchronization and communication. And the performance is not sensitive to LALR. By contrast, if the batch size is large (inapplicable centralized cases Table then DAT + LALR outperforms DAT (namely, LALR matters).
Table 3: Effect of LALR on centralized and distributed training under CIAR-10 with same batch size (2048).
# machines x butch size per machine
Method
TA (%)
RA (%)
DAT-PGD (w/ LALR)
x 341
83.48
39.76
AT LALR
1 2048
83.56
39.84
DAT w/o LALR
x 341
82.42
38.41
(w/o LALR)
1 2048
82.94
38.54
6 CONCLUSIONS
We proposed distributed adversarial training (DAT) to scale up the training of adversarially robust DNNs over multiple machines. We showed that DAT is general in that it enables large-batch min-max optimization and supports gradient compression and different learning regimes. We proved that under mild conditions, DAT is guaranteed converge to first-order stationary point with sub-linear rate. Empirically, we provided comprehensive experiment results to demonstrate the effectiveness and the usefulness of DAT in training robust DNNs with large datasets and multiple machines. In the future, it will be worthwhile to examine the speedup achieved by DAT in the extreme training cases, e.g.. using a significantly large number of PGD attack steps and/or computing nodes during training.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovie. QSGD: Communication- efficient SGD via gradient quantization and encoding. In Advances Neural Information Process- ing Systems, pp. 1709-1720, 2017.
Md Zahangir Alom, Tarek M Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S Awwal, and Vijayan K Asari. The his- tory began from alexnet: A comprehensive survey on deep learning approaches. arXiv preprint orXiv:1803.01164 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint
2018.
Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, signSGD: compressed optimisa- tion for non-convex problems. arXiv preprint arXiv:/802.0445: 2018.
Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Pin-Yu Chen, Shiyu Chang, and Luca Daniel. Visual interpretability alone helps adversarial robustness, 2020. URU/httsss//peeereeeee...
Eooummid--yyer 70EYDB
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled data improves adversarial robustness, In Advances in Neural Information Processing Systems, pp. 11190-1 1201, 2019.
Chia-Yo Yu Chen, Jiamin Ni, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, Xiao Sun, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Wei Zhang, and Kailash Gopalakrishnan. Scalecom: Scalable sparsified gradient compression for communication-efficcee distributed train- ing. In Advances in Neural Information Processing Systems, 2020.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization. In Advances in Neural Information Processing Systems, pp. 14300-14310, 2019.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization, In International Conference on Learning Representations, 2018.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Mare' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, 1223-1231, 2012.
Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned verifiers. arXiv preprint Xiv: 1805 100610005........ 2018
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Physical adversarial examples for object detectors. In Prof. of the 12th USENIX Workshop Offensive Technologies (WOOT 18), 2018
Denis Foley and John Danskin. Ultra-performance pascal GPU and NVLink interconnect. IEEE Micro, 37 2017.
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsieh, Liwei Wang, and Jason Lee. Convergence of adversarial training in overparametrized neural networks. In Advances in Neural Information Processing Systems, pp. 13029-13004 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamesssing adversarial examples. 2015 ICLR, arXiv preprint arXiv:
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: arXiv:/706.000 7706002677 2017.
Kenta Hanada, Ryosuke Morita, Takayuki Wada, and Yasumasa Fujisaki. Simple synchronous and asynchronous algorithms for distributed minimax optimization. SICE Journal of Control, Measurement, and System Integration, 10(6):557-52,,22 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630-645. Springer, 2016b.
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing robustness against unforeseen adversaries. arXiv preprint 1908.00900.00000 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint 160.:066, 2016.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02553 2016.
Mingrui Liu, Youssef Mrouch, Wei Zhang, Xiaodong Cui, Tianbao Yang, and Payel Das. Decentral- ized parallel algorithm for training generative adversarial nets. arXiv preprint arXiv: 1910. 12999, 2019a.
Weijie Liu, Aryan Mokhtari, Asuman Ozdaglar, Sarath Pattathil, Zebang Shen, and Nenggan Zheng. decentralized proximal point-type method for saddle point problems. arXiv preprint 1XXv:19100.13380. 2019b.
Songtao Lu, Meisam Razaviyayn, Bo Yang, Kejun Huang, and Mingyi Hong. Finding second-order stationary points efficiently in smooth nonconvex linearly constrained optimization problems. In Advances in Neural Information Processing Systems, 2020.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706 06083, 2017a.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1000: 100..1000600680 06083, 2017b.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 0078-9086, 2019.
Ivano Notarnicola, Mauro Franceschelli, and Giuseppe Notarstefano. A duality-based approach for distributed min min-max optimization. IEEE Transactions on Automatic Control, 64(6):2559-255 2566, 2018.
Nicolas Papernot, Ian Goodfellow, Ryan Sheatsley, Reuben Feinman, and Patrick McDaniel. clever- hans v1.0.0: adversarial machine learning library. arXiv preprint arXiv: 1610 00768, 2016.
11

---

Under review as a conference paper at ICLR 2021
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Advances in Neural Information Processing Systems, pp. 13847-13856, 2019.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. arXiv preprint arXiv:2002. 1,69, 2020.
Andrew Slavin Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Phirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pp. 3353-3364, 2019.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. In International Conference on Learning Representations, 2020. URL https: http////ppeeeeieeeeeeeeeeeeeeeee net. for.m/orum?22/ http:/ooommn.dreee000000000
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018. URL
net. forest/ooummme id Hk 6xP
Kunal Srivastava, Angelia Nedić, and Dušan Stipanović. Distributed min-max optimization in networks. In Proc. of the 17th International Conference Digital Signal Processing (DSP), pp- 1-8, 2011.
Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli, et al. Are labels required for improving adversarial robustness? arXiv preprint 13X15 2019.
Sebastian U Stich, Jean-Baptist Cordonnier, and Martin Jaggi. Sparsified sgd with memory. In Advances in Neural Information Processing Systems. pp. 4447-4458, 2018
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 131.6699, 2013.
Ioannis Tsaknakis, Mingyi Hong, and Sijia Liu. Decentralized min-max optimization: Formulations. algorithms and applications in network poisoning attack. Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5755-5759, 2020.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving
conmunication-effiieee distributed sgd with slow momentum. arXiv preprint 1rXiv:1910.000 2019a.
Yisen Wang, Xingjun Ma. James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In International Conference on Machine Learning, pp.
2019b.
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang, Gradient sparsification for communication- efficient distributed optimization. In Advances in Neural Information Processing Systems, pp. 1299-1309, 2018
Eric Wong and J Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint arXiv:1711.0083 2017.
Eric Wong, Lestie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. URL htlp/ttttp:::/perrevvi
met./oruu wtt.ooommmmmm000000000.
Cihang Xie, Yuxin Wu, Laurens der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 501-509, 2019.
12

---

Under review as a conference paper at ICLR 2021
Han Xu, Yao Ma, Haochen Liu, Debayan Deb, Hui Liu, Jiliang Tang, and Anil Jain. Adversarial attacks and defenses in images, graphs and text: review. arXiv preprint arXiv:1909.000 2019a.
Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen, Pin-Yu Chen, Yanzhi Wang, and Xue Lin. Evading real-time person detectors by adversarial t-shirt. arXiv preprint arXiv:1910.1109, 2019b.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.00 770..03888.8..8 6, 2017.
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proc. of the 47th International Conference on Parallel Processing, pp. 1. ACM, 2018.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song. James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2019.
Yue Yu, Jiaxiang Wu, and Longho Huang. Double quantization for communication-effiienn dis- tributed optimization. In Advances in Neural Information Processing Systems, pp. 4451,, 2019.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. arXiv preprint arXiv: 1XXivv905.00077 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning, 2019b.
13

---

Under review as a conference paper at ICLR 2021
APPENDIX
DAT ALGORITHM FRAMEWORK
Algorithm A1 Distributed adversarial training (DAT) for solving problem
1: Input: Initial 0,, dataset D(A) for each of M workers, and 7 iterations
2:
for Iteration t 1.2....,
3:
for Worker = 1, .......... M
Worker
4: Draw finite-size data batch Bi C D(i)
5:
For each data sample x E Bi, call for an inner maximization oracle:
arg max Φ(0,, 6)x), Se
(A1)
where omit the label or possible pseudo-label of for brevity Computing local gradient of in (2) with respect to 0 given perturbed samples:
6:
g
AE (
[V x)] + E Xxes(") [V (x))
(A2)
7:
(Optional) Call for gradient quantizer Q(-) and transmit Q(g to server
8:
end for
9:
Gradient aggregation server:
Server
8: = 37 EM Q(g!))
(A3)
10:
(Optional) Call for gradient quantizer g: - (gg:), and transmit g. to workers:
11:
for Worker i = 1, 1,2,.... .22...,.. M
Worker
12:
Call for an outer minimization oracle A(-) to update 0:
0c+1 A(0,ge. Th) T) is learning rate
(A4)
13:
end for end for
14:
Additional details on gradient quantization Let b denote the number of bits (b < 32), and thus there exists = 2h quantization levels. We specify the gradient quantization operation Q(-) in Algorithm A1 as the randomized quantizer (Alistarh et al. 2017: Yu et al. 2019 Formally, the quantization operation at the ith coordinate of a vector g iS given by AAlstar et al. 2017
Q(g.) ||g||2 sign(g.) (l s), Vi (1,2,......
(AS)
In (A5), (gi, s) is a random number drawn as follows. Given lo:1/1 gll2 E [1/s, (1+ 1)/s] (1///s for some I N and 0 < < s, we then have
((g),)) s) = { 1/s
with probability 1 (s|gxl/ ||e|| with probability (s|ga || l).
(1 11/s
(A6)
where la| denotes the absolute value of scalar and ||al|2 denotes the lg norm of vector a. The rationale behind using (A5) is that ((g.) is an unbiased estimate of gs, namely, () with bounded variance. Moreover, we at most need (32 + bd) bits to transmit the quantized Q(g), where 32 bits for ||g|2.. 1 bit for sign of 9; and bits for (. s), whereas it needs 32d bits for a worker performs server in DAT. then the quantization operation Step 10 of Algorithm is no longer needed. In this case, the communication network becomes fully connected. With synchronized communication, this is favored for training DNNs under the All-reduce operation.
g.
g. Clearly, a small saves the communication cost. We note that every
14

---

Under review as a conference paper at ICLR 2021
2 THEORETICAL RESULTS
In this section, we will quantify the convergence behaviour of the proposed DAT algorithm. First, we define the following notations:
@ @ x)
max
6 8 (x),
and P;(0) Ex (0:xx..
(A7)
||80 Sr
We also define
(0) = E x (0;x),
(A8)
where the label y is omitted for labeled data. Then, the objective function of problem expressed in the compact way
can be
M
1
V (
\a(0) + (0)
(A9)
M
and the optimization problem is then given by ming (()).
Therefore, is clear that if point 0* satisfies
V ||| s.
(A10)
then we say 0 is a a approximate first-order stationary point (FOSP) of problem ( Prior to delving into the convergence analysis of DAT, we make the following assumptions.
2.
ASSUMPTIONS
A1. Assume objective function has layer-wise Lipschitz continuous gradients with constant L; for each layer
VV.((e.,)) - (0,,) S L. 0', [h].
(A11)
where (O.,;) denotes the gradient w.r.t. the variables at the ith layer. Also, assume that (0) is lower bounded, i.c., 1 == ming (0) > -c and bounded gradient estimate, i.c., |/Vg ( < G.
A2. Assume that c(0, x) is strongly concave with respect to ô with parameter μ and has the following gradient Lipschitz continuity with constant Lø:
|o, d; x) Veo(O, (, 8'; x':x)]| Lg | 8'|1
(A12)
A3. Assume that the gradient estimate unbiased and has bounded variance, i.e.,
E E [o((0xx) (0), Vi, Ex V (( = 2 Vi,
(A13)
(A14)
where recall that B(O) denotes data batch used worker i, Vol(O) M EM Vol(()) and
(() = M EM (o); and
Vol(O;x) Vol(0))1 50 E
E x
(A15)
E
(0;(0: - vab S 0², Vi.
(A16)
Further, we define component-wise bounded variance of the gradient estimate
Ex [Vl((:x)) [v( of Vi, E [o ( | < df2.vi,
(A17)
(A18)
where j denotes the index of the layer, and & denotes the index of entry at each layer. Under A3, we have Eh Edy max(o);, o - a²
A4. Assume that the component wise compression error has bounded variance [[[Q([ (+) (0),,) Ig ( (0)1a) S 5% Vi.
(A19)
The assumption A4 satisfied as the randomized quantization is used (Alistarh et al. 2017 Lemma.3.1.
15

---

Under review as a conference paper at ICLR 2021
2.2 ORACLE MAXIMIZATION
In practice, P; x), Vi may not be obtained, since the inner loop needs to iterate by the infinite number of iterations achieve the exact maximum point. Therefore, we allow some numerical error term resulted the maximization step at (A)). This consideration makes the convergence analysis more realistic.
First, we have the following criterion to measure the closeness of the approximate maximizer to the optimal one.
Definition 1. Under A2, if point 8(x) satisfies
max اs|l
(5 8t (x), Vad(0, S
(A20)
then, it is a approximate solution to 8* (x), where
(x) arg max 20,,000,,:000 6)x).
(A21)
and x denotes the sampled data.
Condition (A20) is standard for defining approximate solutions of an optimization problem over a compact feasible set and has been widely studied in (Wang et al. 2019h Lu et al. 2020).
In the following, we can show that when the inner maximization problem is solved accurately enough,
the gradients of function 600, 0 (x) ; x) at 6(x) and 8' (x) are also close. A similar claim of this fact has been shown in [Wang et al. 20196 Lemma 2). For completeness of the analysis, we provide the specific statement for our problem here and give the detailed proof as well.
Lemma 1. Let (hh)) be the (με) IT? approximate solution of the inner maximization problem for worker iee, gax s00, g(k); x1), x1)), where x, denotes the sampled data the Lth iteration of DAT. Under A2, we have
Ved (x1): Vo (0c. (8*) (x1): X || SE.
(A22)
Throughout the convergence analysis, assume that 5(t)((x (x+), Vk, are all the (ue)/ L3 approximate solutions of the inner maximization problem. Let us define
v Vo(0,, 6. (xx) x;)ly - (o(0,, (8* (xx) (xls: = Esj-
(A23)
From Lemma
1 we know that when 6¹ (x.)
(x.) is ( approximate solution, then
d,
d:
Eij ΣΣ of (Vo(0, (6*) (x):: /| SE. (A24)
2.3 FORMAL STATEMENTS OF CONVERGENCE RATE GUARANTEES
In what follows, we provide the formal statement convergence rate of DAT. In our analysis, we focus on the I-sided quantization, namely. Step 10 of Algorithm|AT A1 is omitted, and specify the outer minimization oracle by LAMB (You al., 2019), see Algorithm The addition and multiplication operations in LAMB are component-wisc.
Theorem 2. Under AI-A4, suppose that (0,) is generated by DAT for a total number T iterations, and let the problem dimension at each layer be d, d/h. Then the convergence rate of DAT is given by
Au + E (1 (1+ 1)o?
KV3 |xxl
EE
(O.)l
482 +
C.TT
MB
20
(A25)
where A = E[ (e)) Vh is the learning rate, K = cu/a, at and cu are constants used
in LALR 5). X term with the (ih + j)th entry being v (1+X)o7
MB Eij + 8g and
16

---

Under review as a conference paper at ICLR 2021
Eij were given in A24), L [L...... L.]", C 1V 11---a
V 0 < 32 is given in LAMB,
B = min(1B() Vi), and G is given in Al.
Remark 1. When the batch size is large, i.e., B VT, then the gradient estimate error will be O /T). Further, it is worth noting that different from the convergence results of LAMB, there is a linear speedup of deceasing the gradient estimate error DAT with respect to M, i... Oaa² (Moo/((MM)))),))) which is the advantage of using multiple computing nodes.
Remark 2. Note that A4 implies E[(Q([g
()) (0)ls) - Ig
lg 19) (0)ls 1 S E プド= == = From
(Alistarh et al., 2017 Lemma 3.1), we know that < min(d/s² Vd/s)GG Recall that = 2 where b is the number of quantization bits.
Therefore, with proper choice of the parameters, we can have the following convergence result that has been shown in Theorem
Corollary 1. Under the same conditions of Theorem if we choose
7 ~ O(1/VT, E~0
(A26)
we then have
T
Av QCVT
(1 + A)o²
+ o (s. o
(S. min
Vd
VoV
T
MB
2CVT
VMT
4b 2a
(A27)
In summary, when the batch size is large enough, DAT converges to first-order stationary point of problem 2 and there is a linear speed-up in terms of M with respect to a². Next, we provide the details of the proof.
DETAILED PROOFS OF CONVERGENCE ANALYSIS
3.1 PRELIMINARIES
In the proof, we use the following inequality and notations.
1. Young's inequality with parameter is
1 2c
(x,
|x| |yyl?
(A28)
where x, y are two vectors.
2. Define the historical trajectory of the iterates F. (0,..................... 1........... 1........,88
0.). .
3. We denote vector [x]; as the parameters at the ith layer the neural net and [x]ıj represents the jth entry of the parameter at the ith layer.
We define
M
M
g+
E
X.EB
(AVI(0(;x)) + Voo 6.00 (x)): = M Σ
g
(A29)
M
17

---

Under review as a conference paper at ICLR 2021
3.2 DETAILS OF LAMB ALGORITHM
Algorithm A2 LAMB You al., 2019)
Input: learning rate Tt, 0 < $1, 82 < 1, scaling function T(-) 5
0
for t 1,... do
= -,,m,- + (1 31 }))g. where g. is given by (A3)
V, 182 (1
82)
m, m./ -39
V. = ve/(1 -55)
Compute ratio u vi++
end for Update
0 Oc+1. = 01,
||ur,il| U1
(A30)
3.3 PROOF OF LEMMA [
Proof. From A2, we have
(x:) Ve (O. 6f (xx):xx) Vo (0: (x)): Lo (x)) - -80)) (x+)l
(A31)
Also, we know that function @00,8, x) strongly concave with respect to d, so we have
!1 (0) (xx) (8*)(² (5) (x
< Voo(Or. (8*)( (x)):x,)= Vo0(0,, o, (xx): x1), 6,' (xx) - (68)) (xx)) (A32)
Next, we have two conditions about the qualities of solutions 68¹) (xx) and (8+) (xx). First, know that 812) (x:) is a-e approximate solution to (8*) ){{{ (xx), so we have (18)) ( (x.) 86 (x+), V V To (O., 5 (x:); x1) Se.
(A33)
Second, since (0))() (xt) is the optimal solution, it satisfies
(xx) - (x)),, ( (x)); S (x))
(8*)
(A34)
Adding them together, we can obtain
Vo0(O, (8*) (xx):xx) Voo 8. (xx): (x)xxxx) Se.
(x,)
(x:),
(A35)
Substituting A35 into
A32
we can get
p m|66 ( (xx) - (8*) (xx) Se.
(A36)
Combining A31
we have
Vo(Or, (80(0,)))) (xx))))
$ 13 E
(A37)
1
3.4 DESCENT OF QUANTIZED LAMB
First, we provide the following lemma as a stepping stone for the subsequent analysis.
Lemma 2. Under AI-A3, suppose that sequence {0,} } is generated by DAT. Then, we have
(e)) E E (..) 1
+ +Et (1 + X)o²
(A38)
MB
18

---

Under review as a conference paper at ICLR 2021
Proof. From A21
A7
and A2, we know that
(0,;(0,x) Voc(O, (x)(0
(A39)
so we can get
M
VOV(O)
AVo((() + (6);
(A40)
M 1m1
1 M
(V) +
XED (8))
(A41)
M
(0).
(A42)
Then, we have
( (TT -g.)
(A43)
= =B, E -
3:)
(A44)
A42
V 0, (0 -()
(A45)
E
Bi) +.(VV((.)),E. - -g.)
(A46)
where
g:
E Ex)
(
(0') AV (..))), (A47)
M
and
M
(AVI(O,,x) x.) + Voo(O,, (8*) (x.):.)).
E
(A48)
M
Next, we can quantify the different between and g, by gradient Lipschitz continuity of function 7(-) as the following
(0)
M Σ
A24
g+ gi
Ex
Ex,
(00) - Vor(O,, 5(0) 50000(00)))) (xx)): xx))
M
(A49)
where in (a) we use Jensen's 's inequality.
And the difference between ge and gž can be upper bounded by
1
Elg gil =FF
Ex, IJ, (8*)
Vod(O.))
M
+ AEFF
Ex,17 vi(0(;x;)) -v(()
(A50)
13 2)o²
(A51)
MB
Applying Young's inequality with parameter 2, we have
+ (
E
+ Ellg
+ Elg gg 8: 12 (A52)
2
E V 12) +
(1 + 2)o²
(A53)
MB
19

---

Under review as a conference paper at ICLR 2021
3.5 PROOF OF THEOREM 2
Proof. We set $1 0 in LAMB for simplicity. From gradient Lipschitz continuity, we have
A1 v(0++1) S. (V((0))
h
(oV((:))ss
O++1.1
02.))
0 1+1,
01,112
(A54)
(a)
h d.
(1 (ve(o)))s))
L
I ((())
(A55)
Cml
where in (a) use A30 and the upper bound of T r(1)0,,11).
Next, we split term I as two parts by leveraging (onn(V/((.)))) and sign([ur)e) as follows.
[u]ki (s)))( sign((u.)())
R 1/
(110 VV)
|u.eel
[ucJa (sign((VV(())))s) + sign((u.)s)) (A56)
1/8
0110,.:
VV)
|u,ee
(=1
(0)
32
(.))) (8:],, (s = sign(():)s))
1/
d
[urly (s)))( * sign((u,)())
7)
T
(VV
(A57)
1=1
(
32
11c11
V (1)))
2
[u.Ja ())))( * sign((u,);)))
0 t.
(
(A58)
where in (a) we use the fact that ||u S V d: and vv. G, and in (b) we add
32
C1
(sign(VV (gm(()))))))) 2 0.
(A59)
G2d,
Taking expectation on both sides of (58). we have the following:
h
h(1 - 32)
E R < <-1
E
(VV().))
G2d
i=
n.C, Σ Σ
E (V (sign( ((.))))) * sign((u,liy))
(A60)
20

---

Under review as a conference paper at ICLR 2021
Next, we will get the upper bounds of U and V separably as follows. First, we write the inner product between [ [V (0))ss and [Boks more compactly,
111 82)
h E ((VV (g.)))
u S - may
(A61)
Gdd
1=1
h(11 h(1- - (22)
-- V
Σ
E (VV [B:]: -[::): + (8:)))
(A62)
G2d
1=1
h(1 - (2)
(O(VV()),8))
Σ E E(VV(()))s [Bll -Bil)))
(A63)
G
Applying Lemma
we can get
h(1
82)
h(1 (1)
(1 + A)o²
4
B.|
+ +nay
& +
G2d
2
G2d
MB
11(1 (2)
E
((VVV 18:11 - (Bt|))
(A64)
G2d
h(1 B2)
h(1 B2)
(1 + A)o?
21 +
++
G2d
G2d
MB
11ct
h11 (82
E E +
h(1 B2)
Elg: gi l|
(A65)
4
G2d
G2d
(6)
n1c
h(1 (2)
E [V) + man
h(1 32)
(1 2)0²
(1 +
G²d
G²d
MB
h(1 -12)
+ M6
82
(A66)
G²d
where we use the in (a) use Young's inequality (with parameter 2), and in (6) we have
M
gel
M M (g!
)
(
$
82.
(A67)
Second, give the upper of V:
d:
V STC. ΣΣ (V)())) P (ssign(VV + sign(()))e))
(A68)
where the upper bound of W can be quantified by using Markov's 's inequality followed by Jensen's inequality as the following:
W -P (sign( (VV()))))) + (igl(()))s))
PP[V V (0.)l Edl (V)().))
(A69)
V (0 0.) [g
(A70)
V 0 )1
1
(A71)
I ( ml
A12
Jaj
[g7 + [ li
1 kj
8:
ig j2]
(A72)
(1+x)0² MBBI
+ + + 53
(A73)
V (O))ell
21

---

Under review as a conference paper at ICLR 2021
where (a) is true due to the following relations: from (A51
have
Bdles [g (B (1 + N)o
(A74)
MB
ii) from ) we can get
E[[ by Bilk)) S Exji
(A75)
and iii) from A67 we know
[[ (g));) - 82
(A76)
Therefore, combining (A55) with the upper bound of 24 shown in (A66) and V shown in A68
A73
we have
M( (3))
(1 (1+ 1j0²
(0++)) (e)] - may
+ maa
c+
G2d
G²d
MB
h(1 - 32)
h
de
1/1 Cr A
Zim L
+ nay
6² + 1.C.
v3
+ + 02 +
G2d
MB
2
(A77)
Note that the error vector X is defined as the following
(1+A)01
E11 +
MIB
(1+)) M|B]
=
Esy + +02
E Rd
(A78)
(1+ M
Ehdi + Ohdr.
and we have
L1
L
:
TR
(A79)
Li
Recall
k Cu
(A80)
C1
Rearranging the terms, we can arrive at
h(1 B2) 1
< ELV(O.))
((++)
(1 + AYO2
(IVV())))))
4082 +20
(+
G2d 4
n.at
MB
=
(A81)
2
Applying the telescoping sum over t = 1..... T, we have
E[D(O:)] - E (1)+)))))
(
(1 + 2)o²)
Σ
E
V
+2 E+
+ 482
T
CT
MB
+ N
|x||1 + Mccux|1L111
(A82)
20
22

---

Under review as a conference paper at ICLR 2021
4 ADDITIONAL EXPERIMENTS
4.1 DISCUSSION ON CYCLIC LEARNING RATE
It was shown in (Wong ct al. 2020 that the use of cyclic learning rate (CLR) trick can further accelerate the Fast AT algorithm in the small-batch setting (Wong et al. 2020). In Fi we present the performance of Fast AT with CLR versus batch sizes. We observe that when CLR meets the large-batcl setting, it becomes significantly worse than its performance in the small-batch setting. The reason is that CLR requires certain number of iterations proceed with the cyclic schedule. However, the use of large data batch only results in a small amount of iterations by fixing the number of epochs.
80
RA
70
Accuracy
60
50
40
30
128
1024
6144
12288
Batch Size for Cyclic LR
Figure A1: TA/RA Fast AT with CLR versus batch sizes.
4.2 TRAINING DETAILS
CIFAR-10 AT and Fast AT experiments are conducted at single computing node with 16-core CPU, 128GB RAM and Nvidia P100 GPU. The training epoch is 100 by calling for the momentum SGD optimizer. The weight decay and momentum parameters are set to 0.0005 and 0.9. The initial learning rate is set with 0.05 (tuned over (0.005, 0.01, 0.05, 0.13)), which is decayed by x x1/10 at the training epoch 70, 85 and 95, respectively.
CIFAR- 10 DAT experiments are conducted at {1, 6, 12, 18) computing nodes with 16-core CPU, 128GB RAM and Nvidia P100 GPU. The training epoch is 100 by calling for the LAMB optimizer. The weight decay is set to 0.0005. 31 and 32 are set to 0.9 and 0.999. The initial learning rate is tuned over {0.01, 0.05, 0.1,0.2), which is decayed by x1/10 10 at the training epoch 70, 85 and 95, respectively. To execute algorithms with the initial learning rate greater than 0.1, we choose the model weights after 10-epoch warm-up as its initialization for DAT, where each warm-up epoch uses the linearly increased learning rate (k/ 100mm
ImageNet AT and Fast AT experiments are conducted a single computing node with dual 22-core CPU, 512GB RAM and 6 Nvidia V100 GPUs. The training epoch is 30 by calling for the momentum SGD optimizer. The weight decay and momentum parameters are set to 0.0001 and 0.9. The initial learning rate is set to 0.1 (tuned over {0.01, 0.05, 0.1, 0.2}), which is decayed by x1/10 at the training epoch 20, 25, 28, respectively.
ImageNet DAT experiments are conducted at 31,3, 6} computing nodes with dual 22-core CPU, 512GB RAM and Nvidia V100 GPUs. The training epoch is 30 by calling for the LAMB optimizer. The weight decay is set to 0.0001. 31 and 32 are set to 0.9 and 0.999. The initial learning rate is tuned over (0.01. 0.05, 0.1, 0.2, 0.4), which is decayed by x x1/10 1/10 at the training epoch 20, 25, 28, respectively. To execute algorithms with the initial learning rate 77 greater than 0.2, we choose the model weights after 5-epoch warm-up as its initialization for DAT, where each warm-up epoch k uses the linearly increased learning rate (k/5)m.
23

---

Under review as a conference paper at ICLR 2021
Empirical model convergence. In Figure A2 of DAT-PGD. As we can see, our proposal converges well within 100 and 30 epochs in the setting of (CIFAR-10, 10, ResNet- 18) and (ImageNet, ResNet-50), respectively
present the training accuracy and the loss value
Loss
4.5
TA
Loss
60-
RA
Training Accurac)
70
Training Accuracy
50
Training Loss
60
3.5
40
3.0
40
30
2.5
2.0
20
60
80
100
5
10
15
20
25
Training Epochs
Training Epochs
(a) CIFAR-IO, ResNet-18
(b) ImageNet, ResNet-50
Figure A2: Training accuracy and objective value (loss) of DAT-PGD against training epochs. (a) DAT-PGD for (CIFAR-10, ResNet- 18) using x computing configuration and 2048 batch size. (b) DAT-PGD for (ImageNet, ResNet-50) using 6 computing configuration and x 512 batch size.
Table Al: TA/RA of DAT-FGSM under (CIFAR-I0, ResNet-18) using 18x2048 batch size versus different choices of LALR hyperparameter Ca.
Tuning LALR hyperparameter Cu. We also evaluate the sensitivity of the performance of DAT to the choice of the hyperparameter Cu in LALR. In Table A1 we fix = 0 (this natural choice) but varies G, E {8, 9, 10, 12} when DAT-FGSM is executed under CIFAR-10 10 using 18x2048 batch size, where Ca = 10 is our default choice. As we can see, both RA and TA not quite sensitive Cu and the default choice yields the RA-best model (in spite of minor improvement..
Value of co
TA (%)
RA (%)
C, =
73.57
38.19
73.72
38.00
Ca
73.42
38.55
C
73.75
38.18
73,63
37.87
24

---

Under review as a conference paper at ICLR 2021
4.3 OVERALL PERFORMANCE OF (CIFAR- 10, RESNET-50)
In Table we observe that in the large-batch setting, the proposed DAT-PGD and DAT-FGSM algorithms outperform the baseline algorithm DAT-PGD w/o LALR, and result in competitive performance to AT and Fast AT, which call for more iterations by using a smaller batch size.
Table A2: Overall performance of DAT (in gray color), compared with baselines, in TA (%), RA (%), communication time per epoch (scconds), and total training time (including communication time) per epoch (in seconds). For brevity, q' represents '# nodes x # GPUs per node', 'Comm." represents communication cost, and 'Tr Time' represents training time.
CIFAR- 10, ResNet-50
Method
Comm
Tr. Time
X
Batch size
TA (%)
RA (%)
per epoch (s)
per epoch (s)
AT Fast AT DAT-PGD w/o LALR PGT FASM
XXI
256
85.94
43.06
NA
894
X
256
75.28
40.48
NA
288
X
6 256
74.45
33.35
68
236
X
6 256
84.79
42.16
68
236
6 X 1
6 256
75.72
40.09
68
116
25

---

Under review as a conference paper at ICLR 2021
4.4 ROBUSTNESS AGAINST PGD AND C&W ATTACKS
A3 we evaluate the adversarial robustness of ResNet-18 at CIFAR-10 learned by DAT-PGD and DAT-FGSM against PGD attacks of different steps and perturbation sizes (namely, values of e). We consistently observe that DAT matches robust accuracies of standard AT even against PGD attacks different values of and steps. Specifically, DAT has slightly smaller RA than AT when facing weak PGD attacks with less than (5/255) and steps less than 5. Moreover, although DAT-FGSM has the worst RA against weak PGD attacks (which reduces TA at 0), it outperforms other methods when the attacks become stronger in CIFAR-10 experiments. In Figure/A4, we present the additional robust accuracies against C& attacks (Carlini & Wagner 2017) of different perturbation sizes, As we can see, the results are consistent with the aforementioned ones against PGD attacks.
In Figure A3
DAT-PGD
80
DAT-PGD DAT-FGSM
Robust Accuracy
DAT-FGSM
Accuracy
60
70
40
60
20
50
0
0
10
15
20
25
30
10
20
30
40
50
Perturbation Size
Attack Steps
Figure A3: RA against different PGD attacks for the model trained by DAT-PGD, DAT-FGSM, and AT under (CIFAR-10, 10, ResNet-18) (Left) RA against PGD attacks with different perturbation sizes (over the divisor 255). (Right) RA against PGD attacks with different steps.
AT
DAT-PGD
DAT-PGD
Robust Accuracy
DAT-FGSM
Robust Accuracy
50
DAT-FGSM
60
40
30
20
0
10
15
20
25
30
2 4
6
8
10
CW Attack Perturbation Size
CW Attack Perturbation Size (b) ImageNet, ResNet-50
(a) CIFAR- 10, ResNet-18
Figure A4: against different C&W attacks model trained by DAT-PGD, DAT- DGSM. and under the setting (CIFAR-10, 10, ResNet-18) and (ImageNet, ResNet-50), respectively. Here for ease C&W attack generation at ImageNet, we randomly select 1000 test ImageNet images (1 image per class) to generate C&W attacks.
26

---

Under review as a conference paper at ICLR 2021
4.5 DAT FROM PRE-TRAINING TO FINE- TUNINGG
In Figure A5, we investigate if a DAT pre-trained model (ResNet-50) over a source dataset (ImageNet) can offer fast fine-tuning to a down-stream target dataset (CIFAR-10), Compared with the direct application of DAT to the target dataset (without pre-training), the pre-training enables a fast adaption the
CIFAR-I0 10 task in both TA and RA within just 5 epochs.
TA: Pre-training on ImageNet
RA: Pre-training on ImageNet
Accuracy (TA or RA)
TA: No Pre-training
100
RA: No Pre-training
80
60
40
20
10 Fine-tuning epochs
Figure A5: Fine-tuning ResNet-50 (pre-trained on ImageNet) under CIFAR-IO. Here DAT-PGD is used for both pre- pre-training and fine-tuning at 6 nodes with batch size 6 x 128.
4.6 QUANTIZATION
present the performance of DAT by making use of gradient quantization. Two quanti- zation scenarios are covered: 1) quantization is conducted at each worker (Step 7 of Algorithm|AT] and 2) quantization is conducted both worker and server sides (Step 7 and 10 of Algorithm|A1]. As we can see, when the number of bits is reduced from 32 to 8, the communication cost and the amount of transmitted data is saved by 2 and times, respectively. Although the use of gradient quantization introduces performance loss to some extent, the resulting TA and RA are still comparable to the best 32-bit case. In the worst case of CIFAR-10 (8-bit 2-sided quantization), TA drops 0.91% and 6.33% for DAT-PGD and DAT-FGSM, respectively. And RA drops 4.73% and 5.22%, respectively. However, 8-bit 2-sided quantization transmitted the least amount of data per iteration.
In Table A3
To further reduce communication cost, we also conduct DAT at HPC cluster. The computing nodes of the cluster are connected with InfiniBand (IB) and PCIe Gen4 switch. To compare with results in Table [ we use 6 of 57 nodes of the cluster. Each node has 6 Nvidia V100s which are interconnected with NVLink. We use Nvida NCCL as communication backend. In Table[A4] we present the performance of DAT for ImageNet, ResNet-50 with use of HPC compared to standard (non-HPC) distributed system. As we can see, the communication cost is largely alleviated, and thus the total training time is further reduced.
In Table A5 we conduct an additional experiment by integrating a centralized method with gradient quantization operation CIFAR-10 under the batch size 2048 and 6 2048, respectively. We specify the centralized method as Fast AT with LALR, where LALR is introduced to improve the scalability of Fast AT under the larger batch size 6 x 2048. Due the centralized implementation, we only need 1-sided gradient quantization (namely, no server-worker communication is involved). As we can see, when the batch size 2048 is used, Fast AT w/ LALR performs as well as Fast AT even at the presence of 8-bit gradient quantization. On the other hand, as the larger batch size 6 x 2048 is used, Fast AT w/ LALR can still preserve the performance at the absence of gradient quantization. By contrast, Fast AT w/ LALR at the presence of quantization encounters 6.05% TA drop. This suggests that even in the non-DAT setting, 8-bit gradient quantization hurts the performance as the batch size becomes large. Thus, in DAT it is not surprising that 8-bit quantized gradients could cause a non-trivial accuracy drop, particularly for using 2-sided gradient quantization and a much larger data batch size (2 18x2048 on CIFAR-10) 10). One possible reason is that the quantization error cannot easily be mitigated as the number of iterations decreases (due to increased batch size under a fixed number of epochs).
27

---

Under review as a conference paper at ICLR 2021
Table A3: Effect of gradient quantization on the performance of DAT for various numbers of bits. The training
settings of (CIFAR-10, ResNet-18) and (ImageNet, ResNet-50) consistent with those in Table
CIFAR-IO, ResNet-18
Method
Comm.
transmitted per iteration (MB)
# bits
TA (%)
RA (%)
per epoch (S)
DAT-PGD
32
80.38
38.94
8.5
1278
DAT-PGD
16
79.38
38.32
8.3
639
DAT-PGD
78.18
37.34
4.3
320
DAT-PGD
8 (2-sided)
78.86
34.2
5.0
107
DAT-FGSM
32
75.58
40.92
1278
DAT-FGSM
75.74
40.86
8.3
639
DAT-FGSM
72.48
38.98
4,3
320
DAT-FGSM
8 (2-sided)
69.26
35.34
5.0
107
ImageNet, ResNet-50
Method
Comm.
transmilted
# bits
TA (%)
RA (%)
per epoch (s)
per iteration (MB)
DAT-PGD
32
63.75
38.45
898
2924
DAT-PGD
16
61.77
38.40
850
1462
DAT-PGD
56.53
37.90
592
731
DAT-PGD
(2-sided)
53.09
34.59
244
DAT-FGSM
58.32
41.48
859
2924
DAT-FGSM
54.71
39.29
849
1462
DAT-FGSM
50.11
36.38
594
731
DAT-FGSM
8 (2-sided)
48.27
33,20
1013
244
Table A4: Comparison to training high performance computing (HPC) cluster of nodes
ImageNet, ResNet-50
Method
Comm,
Tr. time
bits
TA (%)
RA (%)
per epoch (s)
per epoch (s)
DAT-PGD
32
63.75
38.45
898
1960
DAT-FGSM
32
58.32
41.48
859
1109
DAT-PGD (HPC)
32
63.43
38.55
15
1074
DAT-FGSM (HPC)
32
57.60
41.70
15
310
Table A5: Effect of 8-bit quantization centralized robust training Fast AT w/ LALR.
CIFAR-I0, 10, ResNet-18
Method
8-bit
Batch size
TA (%)
RA (%)
quantization (8)
Fast AT
No
2048
81.58
38.34
Fast AT w/ LALR
Yes
2048
80.66
38.60
Fast AT w/ LALR
No
2048
80.08
38.51
Fast AT w/ LALR
Yes
6 2048
75.53
38.45
28


Reviews:
[{"review_id": "kqHMI0j7Aoi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed distributed adversarial training (DAT) for robust models. The method is a combination of PGD-like adversarial training, LARS-like large batch training, and quantizing gradients for communication efficiency in distributed training. The authors show convergence of adversarial training with LARS-like learning rate under layer-wise assumptions, and empirical results of DAT can scale to 6x6=36 GPUs and batch size 6*512 for ImageNet.\n\nPros\n\n+ The paper is well written and easy to follow. \n+ The convergence rate looks reasonable. \n+ It is good to show that adversarial training can be accelerated through distributed settings.\n+ Extensive experiments on CIFAR-10 and ImageNet. \n\nCons\n\n- The general contribution of the paper is a bit incremental. The main message seems to be that LARS can help large-batch adversarial training?\n- The convergence rate contribution seems incremental given the known adversarial training convergence proof and LARS-like convergence proof. \n- As far as I know, scaling up learning rate (maybe with warmup) performs good for large batch training, we only need LARS for a certain regime, the ImageNet setting in table 1 where the batch size is 6*512 seems to fall in the regime where scaling LR works. Did the authors scale up LR for baselines without LARS?\n- The Fast FGSM method (Wong et al. 2020) uses cyclic LR etc. to make convergence fast. Are those tricks applied here? Wong et al. reported CIFAR training in about 10 min, which is faster than the baseline in table 1 (52 sec/epoch * 100 epochs). I am worried the authors may target a less significant problem than we expected.\n- [Xie et al. 2019 Feature Denoising for Improving Adversarial Robustness] and Kannan et al. 2018 use more GPUs than reported in the paper. And Xie et al. achieves better robustness on ImageNet. What is the practical advantage of this paper compared to Xie et al? What are the insights from the authors’ reimplementation of Xie’s method on CIFAR? ( DAT-LSGD in table 1)\n- Why does communication time decrease when more GPU machines (24) are used?\n\n\n\n==================== post rebuttal ==============================\n\nI do not think my concerns are addressed by the discussion. However, I also think this is a well-written paper in general and I will not be upset if it is accepted. \n\nMy main concerns,\n(1) The authors fail to show that the proposed method is non-trivial. I think this concern is raised by multiple reviewers. The authors keep clarifying the technical difficulty (especially the theory) of applying LARS etc, but my main concern is the necessity of these knobs added by authors. After a few rounds of discussion, we reach to a conclusion that adversarial training is different from standard training, but I do not think that could be considered insights from this paper. I would strongly suggest authors consider explaining why LARS is necessary by either theory or intuitive insights, and make it clear what exactly the difference is. \n(2) The authors claim contributions for large scale setting (ImageNet with large number of available GPUs), but the experiments are somewhat worse than previous results. Lacking computation resources is a good excuse, but since the authors claimed they can use larger batch size with smaller number of GPUs, I do not see a technical reason why they cannot use their method to re-run the large scale experiments to directly compare with previous results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_sYdb2pOVww", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Hello authors,\n\nThank you for the submission.  I found it to be well-written and I enjoyed reading it.  The generic description of DAT given in Equation 2 is nice.  However, there are some concerns on my end that I hope you can help me alleviate that are keeping me from giving the paper a positive review.  I have an open mind and can be convinced.\n\n1. Why is the solution of DAT not obvious?\n\nThe authors claim that \"the direct solution [for scaling AT] of distributing the data batch across multiple machines may not work\", but I cannot understand why when Algorithm 1 looks like exactly that.  The adversarial samples are generated on each machine at the beginning of each iteration so as to avoid computing them on the server at the beginning of each iteration.  To me, this is an unsurprising and obvious optimization.  Other than that minor variation, this looks exactly like the standard parameter server approach.\n\n2. Are the experiments really fair here?\n\nThe most clear claim appears to be from Table 1 and is mentioned in the introduction, where using DAT-PGD with 6 nodes gives a 3x speedup over AT while preserving standard test accuracy and almost preserving robust accuracy.  However, compare with when LALR is *not* used: then, the accuracy is significantly degraded.  So a natural question that I would like to understand is what LALR does when used with the original AT algorithm.  In fact, to get a clear apples-to-apples comparison, I would think that it would be best to use AT with a 6*512 batch size, plus LALR.  Basically, what I mean is that I am not sure if the preservation of accuracy is entirely due to LALR.  If it is, this implies that LALR can be used off-the-shelf with AT in a *non*-distributed setting with larger batch size, and that could result in a situation where DAT-PGD + LALR significantly underperforms AT + LALR---which would contradict the result that DAT is able to preserve accuracy, instead giving the result that LALR improves accuracy.\n\nAlternately, I might expect that DAT-PGD without LALR but with a batch size of 512/6 on each of the 6 nodes (giving the same as the original batch size) would give comparable performance.\n\nI hope the gist of my comment is clear here: it seems like the comparison is not really a fair comparison.  I think it muddles improvements from DAT and improvements from LALR, and makes it hard for the reader to understand what is really going on.\n\nSome additional small things:\n\n - In (1) boldface y is used, but in (2), non-boldface y is used.  Perhaps these should be made the same?\n - \"Server\" instead of \"Sever\" in Algorithm 1\n - In Section 4 there seems to be redundancy: \"see the meta-form of DAT in Algorithm 1 and details in Algorithm A1. \" but I believe it is meant that both references should point at the same thing.\n - In Section 4, it is written that \"DAT takes a M times fewer iteration than AT\".  (Also there are some grammar issues in that sentence.)  It may be clearer to say \"M fewer gradient updates on the server\".\n - In Section 4, it is claimed that the computation time of DAT with iterative PGD is K times more than that of DAT with FGSM.  I know that I am being pedantic but this is not correct---to make it correct, the computation time of *the adversarial perturbation generation step* is K times more, not the overall computational time.\n - Use \\citet not \\citep at the end of the sentence starting with \"Indeed, we will show ...\" at the top of page 5.\n - What is the unit for training time in Table 1?  Does that include communication time?\n\nOverall, I am open to discuss my thoughts, but in my current understanding, I am not able to make a strong enough case for DAT to be published at ICLR.\n\n## Post-rebuttal comments\n\nI am impressed at the amount of time the authors have spent trying to clarify the different concerns.  But unfortunately my concerns are not fully addressed, and also a new concern arises: if this much space had to be spent clarifying different confusions of the reviewers, I think the paper could do with a complete overhaul.  I would suggest that the authors take into account the general confusions that arose in this review process and rewrite the paper such that those particular confusions are alleviated.  For instance, I struggled for clarity on what makes this contribution non-trivial, whether the contributions are primarily theoretical or primarily empirical (and how I should understand the balance between the two), and the fairness of the empirical comparison.  Rewriting the paper such that those three concerns---and the others pointed out by the other reviewers---are discussed clearly would be a significant improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HbyhbtXxabY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a framework, called DAT, to scale out adversarial training to distributed settings. DAT combines three techniques: one-shot fast gradient sign method for more efficient inner maximization, gradient quantization for reduced communication volume, and layer-wise adaptive learning rate optimizer for large-batch training.  Equipped with the proposed techniques, the authors obtain promising speedups to perform adversarial training against ResNet18 and ResNet50 on CIFAR-10 and ImageNet with multi-node and multi-GPU, while achieving comparable (roughly under 2% difference) accuracy.\n \n \nPros:\n- By combing all the techniques, the proposed framework yields promising results for distributed adversarial training.\n- The paper shows that sparse gradients and large-batch training scheme also apply to adversarial training and provide some theoretical analysis to show the convergence rate of the composite of these schemes. \n \nCons:\n- The technical contribution of the work seems to be limited. All techniques used are from existing works. Therefore, the main contribution is on applying these techniques in the context of adversarial training. \n- Several optimizations employed by the framework have a subtle impact on model convergence. Therefore, although the training time for a specific run might be reduced, the overall development time and complexity might increase. \n \nComments: \n\nAdversarial training is slow, and the paper looks into how to accelerate AT through distributed training. The paper correctly identifies several techniques, such as large-batch, gradient compression, and FGSM, to improve the computation efficiency on single GPU as well as reducing the communication volume across GPUs and machines. The evaluation is thorough and convincing. The main concern is on the novelty side. All techniques are from existing work and have been studied heavily: FGSM reduces the amount of computation for inner maximization; gradient quantization reduces the communication volume; large-batch training improves GPU utilization. Therefore, the contribution is mainly on creating a framework that combines all the techniques and demonstrates empiricdally that they help scale out AT.\n \nThe reviewer appreciates the theoretical analysis of the convergence rate of DAT. However, similar convergence proofs have been given in each of the individual techniques in the past, as in [1-3]. Why isn't the convergence of DAT a simple composition of the three, or is there a more specialized aspect in the convergence proof for DAT?\n \nAnother main concern is the actual convergence impact of the composed framework. Each of the proposed techniques would have an impact on model convergence and is known to make training more difficult, e.g., each may lead to model divergence, slow convergence, or convergence to suboptimal local minima. How does DAT help avoid these challenges or does DAT actually impose more challenges because it involves the interaction of all three methods?\n  \nQuestion:\n\nThe evaluation does not include a comparison of Fast-AT under the 6x6 setting. Is it because there is no performance gain when training Fast-AT across multiple servers, or is it because Fast-AT does not support multi-node training? If it is the latter case, then it seems to be better to add Fast-AT (6x6) as a baseline since it does not seem to be too difficult to extend AT with DDP in PyTorch or multi-node training in TensorFlow, and it would show a better gap between the current multi-GPU multi-node training and DAT.  \n \n[1] Wang et. al. \"On the Convergence and Robustness of Adversarial Training \", http://proceedings.mlr.press/v97/wang19i/wang19i.pdf\n\n[2] You et. al., \"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes\", https://arxiv.org/abs/1904.00962\n\n[3] Alistarh et.al., \"The Convergence of Sparsified Gradient Methods\", https://papers.nips.cc/paper/7837-the-convergence-of-sparsified-gradient-methods.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "HCVa55pxVp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Adversarial training is a principled approach towards robust neural networks against adversarial attacks, but it is extremely computing intensive. This work tackles the problem by leveraging the general distributed training method, and addressing the problems of direct application by several effective innovations. \n\nStrength:\n\n+ The proposed method is practical.\n\n+ The proposed DAT can deal with both labeled data (supervised learning) and partial unlabeled data (semi-supervised).\n\n+ I like the idea to use gradient quantization/compression.\n\n+ Make theorectical conribution by convergence analysis for DAT with LALR and gradient quantization.\n\nAdditional comments and questions:\n\n1. One conclusion is the paper can speed up by 3 times with 6 times resource. Please comment on what's the typical speedup of distributed training with n times of resources. Do you think you can further speedup?\n\n2. In formula (2), additional regularization term by lamba is used, making it different from the direct generation of (1) into multiple workers. Why we must introduce the regularization term in DAT?\n\n3. Please provide more details about how to measure communication time please? any profiler used?\n\n4. In table 2, additional unlabeled data can improve accuray. Why?\n\n5. Which deep learning framework is used to support gradient quantization?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors claim to propose a distributed large-batch adversarial training framework to robustify DNN.
Although the authors made efforts to clarify reviewers' concerns,  it is clear that the authors still cannot convince some reviewers in several points after several rounds of discussion between reviewers and authors.

The reviewers were not in consensus on acceptance and some concerns were still not clearly addressed in the rebuttal phase.
Hence, I recommend acceptance only if there is a room. 

Decision: Reject