Phase Transition from Clean Training to Adversarial Training
Authors: Yue Xing
Qifan Song
Guang Cheng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial robustness
Adversarial training

Abstract: Adversarial training is one important algorithm to achieve robust machine learning models. However, numerous empirical results show a great performance degradation from clean training to adversarial training (e.g., 90+\% vs 67\% testing accuracy on CIFAR-10 dataset), which does not match the theoretical guarantee delivered by the existing studies. Such a gap inspires us to explore the existence of an (asymptotic) phase transition phenomenon with respect to the attack strength: adversarial training is as well behaved as clean training in the small-attack regime, but there is a sharp transition from clean training to adversarial training in the large-attack regime. We validate this conjecture in linear regression models, and conduct comprehensive experiments in deep neural networks.

Phase Transition from Clean Training to Adversarial Training
Yue Xing Department of Statistics
Qifan Song
Guang Cheng Department of Statistics University of California, Los Angeles guangcheng@ucla. edu
Department of Statistics Purdue University
Purdue University xing49@purdue.. edu
afsengruurdue. edu
Abstract
Adversarial training is one important algorithm to achieve robust machine learning models. However, numerous empirical results show a great performance degrada- tion from clean training to adversarial training (e.g., 90+% vs 67% testing accuracy on CIFAR- 10 dataset), which does not match the theoretical guarantee delivered by the existing studies. Such gap inspires us to explore the existence of an (asymptotic) phase transition phenomenon with respect to the attack strength: adversarial training is well behaved as clean training in the small-attack regime, but there is a sharp transition from clean training to adversarial training the large-attack regime. We validate this conjecture in linear regression models, and conduct comprehensive experiments in deep neural networks.
1 Introduction
Among various algorithms towards adversarialyy robust models, adversarial training is a popular and simple way improve the adversarial robustness of the model. Many studies improve adversarial training from either thcoretical or empirical aspects.
To justify the theoretical properties, abundant literature studies the performance of adversarially robust models under common statistical models, e.g., linear regression, Gaussian mixture model and
etc. in Javanmard et al.] 22200;; Mchrabi al. (2021): Javanmard and Soltanolkotabi] (2020); Dan et al. 2020): Taheri et al. (2020); Xing et al. 2021caa a). Besides, some other studies work on deep learning models. For example, Gao et al. (2019); Zhang al. (2020b); Allen-Zhu and L.i 2020) show the convergence of adversarial training in neural networks when the attack strength is sufficiently
small. Other literature, e.g., Zhang al. (2019) Wang et al. (2019b))) a), propose improvements in the
training loss of adversarial training.
However, there are some gaps between the promising theoretical results and the actual performance of adversarial training in real practice. First, the aforementioned theoretical works consider either simple models sufficiently small attack strength. For simple models, adversarial training is well- behaved because the adversarial loss is still convex. For complicated models with small attacks, they conduct similar analysis for adversarial training for clean training, in the sense that adversarial training only introduces some additional diminishing error. Such settings are different from the other studies also reveal several potential concerns of adversarial training in deep learning, e.g., the smoothness problem Lee and Chandrakasan (220)); Xie et al. 2020); Xing al. (202 a), overfitting issue Rice al. (2020): Wu al. (2020). Based on RobustBench (https: haap::/://robbbbecch gi thub. io/). the best robust accuracy for CIFAR-I0
setups used in real practice. Second,
using state-of-the-an methods 66.6% Rebufli et (202T) much worse than the clean accuracy of above 90% in clean training.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

As our attempt to fill these gaps, this work focuses on the role of adversarial attack strength. For instance, we are wondering whether taking 8/255 Cs attack is too strong for CIFAR-10 adversarial training using the current state-of-the-ar neural networks architectures and optimizer configurations. Although clean training generalizes well this setup and adversarial training can be viewed as an adaptation of clean training, they may act differently.
This paper investigates how adversarial training deviates from clean training as the attack strength grows. An illustration of this can be found in Figure | Under small attack strength e, the trajectory of adversar- ial training is close to that of clean training so that they converge to the same flat min- ima. As increases, the adversarial train- ing process deviates from the clean training dramatically. Our aim is to determine crit- ical attack level, denoted as e", such that (i) when < E", (S)GD trajectories of clean training and adversarial training are similar in some sense so that they can be handled similarly: but (ii) when e > ex, the ad- versarial training will significantly deviate from clean training.
Adv Training (med um attack)
Adv Training (small attack)
Adv Training (large attack)
Clean Training
Figure 1:
The trajectory of clean training, adversarial training with small/large attack.
Some other literature also supports our conjecture that adversarial training differs from clean training under large attack strength. [ong et al. (201)); Frederickson et al. (2018) show that, as the attack more detectable against clean samples. Such a difference in the training samples leads to the discrepancy between clean and adversarial training. In addition, the trained model from adversarial training is different from the clean model in terms of its Hessian Yao et al. (2018). Furthermore, the neural network capacity required by adversarial training is larger than clean training Xie and Yuille (019); Bubeck and Sellke (202I).
strength increases, the attacked samples
Note that the "phase transition" phenomenon in this paper refers to an asymptotic behavior rather than transition at some specific point. In thermodynamics, the phase transition means the state of a substance changes an exact threshold temperature, e.g., the melting point. In our case, the threshold e" refers to asymptotic order w.r.t. and model dimension. The behavior of the training trajectory different under < e* and >> E" (e/e* + and e/e* - oo in sample size and possibly in data dimension as well).
Contributions
The main contributions of this paper can be summarized follows:
Directly defining the critical value e* requires comparing the whole optimization trajectories between clean training and adversarial training. The whole path comparison can be rather difficult. Thus instead, from theoretical intuitions obtained in simple models, we identify the critical e* via surrogate measure that only relies on the training losses at trajectory destination. In particular, we propose to clean testing loss in clean training (denote R (0)) as benchmark for adversarial training. If the attack strength is small enough so that the (proper) adversarial training loss is smaller than R" (0), then the adversarial training is similar to the clean training (and vice versa).
To compute e", a naive way is to repeatedly perform the adversarial training from scratch (i.e., initializing from random start) for each possible until the corresponding adversarial training loss hits 12* (0). This could be time-consuming. To speed up the computation, we also propose a method based on adversarial fine-tuning and extrapolation to approximate e".
Due to the page limit, we postpone many details of approximating e* to Appendix
Last but not least, we emphasize that c" severs as a diagnosis of the adversarial training. but not a rule of how to choose the attack level. That is, when c < c", one can safely assume adversarial training works as well as clean training: when e > e*, it indicates that the attack strength is too large for the current data set using the current training configuration, and certain remedy is necessary such that the adversarial training performs as good as its clean counterpart. It is beyond our scope to provide complete solution of how to adjust adversarial training, but some discussions are given in Section

---

2 Other Related Works
Clean Training in Deep Learning Many existing literature studies deep neural networks in clean training. For the training performance, studies such as Du et al. 2019 2018): Du and a (018) work on the optimization convergence of the empirical training loss. In terms of the testing performance, there are many works showing the good performance of clean training in different aspects, e.g., double-descent phenomenon and benign over-fitting [Blkin et al. (2018): Hastie et al. (2019) [a et al.
(2200; Bartlett et al. (2020), implicit regularization Neyshabur et al. (2017b); Neyshabur (2017);
Baratin et (020), generalization bounds using PAC-Bayes method or other methods Neyshabur
et al. 201 7
Kohler and Langer 2019); Hu et al. (2020): Taheri et al. (2021).
Adversarial Training In the literature, there are several strands of theoretical studies related to adversarial training, e.g., the statistical properties or generalization performance of the global optimizer of adversarial training loss without any consideration of solving the optimization problem
Javanmard et al. (2020); Yin et al. (2018); Raghunathan et al. (2019); Min et al. (2020); Zhai et al.
(2019); Hendrycks et 2019); Chen al. (2020a); Xing et al. 2021e: Dan et al. (2020), the (local)
convergence of adversarial training algorithms under convexity assumptions Sinha et al. 2018):
Wang et al. (2019a). the adversarial training loss in deep neural networks (Gao et al., 2019 Zhang et al. 2020b Wu et al., 2020), and the generalization properties of deep neural networks, e.g. upper
bound Allen-Zhu and Li (2020) and lower bound in Bubeck and Sellke (2021).
3 Adversarial Training
To introduce advensarial training, let I denote the loss function and fo(x) be the model with parameter
0. The (population) adversarial loss is defined R(0, <) E. (1 (fo (x: Ac(So, w)], v)], where
A. is an attack of strength >> 0 and intends to deteriorate the loss in the following way
Ac(Jo, v. y) == argmax (1(fo(z + z),
(1)
=E,,(0.e)
where B2( r) is a Lp ball centering at x with radius r. Given n i.i.d. samples S = {(Ts, yi) the adversarial training minimizes the sample version of R(0,e) w.r.t. O:
R((0e) 1 Σ (SolF, + Ac W.), v:),
and the estimator ((e) aims to minimize R(O,e), e), and the minimized adversarial training loss is (e). We rewrite R(O,c) c) R(O) for simplicity when there no confusion.
The minimization in is often implemented through an iterative two-step (min-max) update. In the t-th iteration, we calculate the adversarial sample x:(()
+ Ae Ii, ys) based on the current g(t), and then update g(++1) based on the gradient of the adversarial training loss while fixing (())s with learning rate 7)- The algorithm runs for T iterations. Note that for some loss function I model fo (e.g. deep neural networks), there may not be an analytic form for A,, thus numerical methods, e.g. FGM and PGD, are utilized to approximate Ac. Denote (()(s) as the adversarially trained model using attack strength e.
4 Intuitions from Simple Models
To obtain insights into the eritical strength e", we consider the linear regression models. Briefly speaking, we justify that is in ( under L2 attack by comparing the training trajectories of clean and adversarial training, as well as establishing an explicit connection between the critical bound and the generalization error of clean training.
Simple Linear Regression For simple linear regression problem, to measure the difference between the trajectories of adversarial training and clean training, if E < v then asymptotically, there is
With slightly modification of usual notations, we denote vd/n log" n/ for any fixed k 0 accommodate with some tail probability bounds. The definition of c > vd/n is modified similarly.
3

---

no difference between adversarial training and clean training during the training process. Throughout the training (with proper early stopping), we always have that the updating gradient of adversarial training is dominated by the one in clean training. To be specific, we have the following result:
Theorem 1. Assume Y fn X + where X E R follows multivariate normal distribution with zero mean and covariance La- and is a Gaussian noise with constant variance. The true coefficient tn satisfies ||0ol| (1)) so that Var(Y) = (11) as well. Consider L2-adversarial training.
When < Vdin, with zero initialization proper learning rate and number of steps T, the optimization converges the global risk minimizer, i.e., (0) - 0(0) and (ET Ō(e). Besides, with probability tending to 1, for all S T, the updates in clean and adversarial training have insignificant difference (small attack in Figure D:
6 - (1)1/s/ss)))
0.
When >> v d/n, with probability lending 1,
lim inf d/n > both |[(())l and (I) |/()) converge to (implying a training trajectory wandering around O, ie.. large attack in Figure That the empirical adversarial risk minimizer is asymptotically zero, giving all training and testing adversarial predictions as zero.
If lim d/nn (ō(e) [
co, i.e., the adversarial training trajectory will be statistically different from the clean training process (medium attack in Figure 4.
Such a difference between O(e) and ((0) implies the difference the training trajectories.
The proof (and details of 17 and 2) of Theorem I
postponed to the appendix.
Theorem implies that e" = O @(vd/n), which gives an appropriate asymptotic order for the critical strength level under linear regression model. However, reproducing such an analysis to find e" for complex statistical models is more mathematically involved or even intractable. This motivates us to seek simpler surrogate measure identify 6".
We find that the difference between the clean training loss and clean testing loss can be a proper choice of such a surrogate measure, as illustrated in Theorem 2 Ibelow. Using the loss to do comparison helps avoid directly comparing model parameters (i.e., g(*) (c) vs (0)), which eases the comparison for complex models such neural networks in practice. Denote A the difference between the clean training loss and clean testing loss of clean trained model, i.c., practice, one can use (()())))) - (()( 0) as an estimate of A.
R(O(0), 0) - R(((0), 0). In
Theorem 2. Under the model setup as in Theorem ! regardless of the growth of dimension d and sample size n, when < /d/n, with high probubiliy,
( 0) R(((()), e)) +
When >> V d/nn the above amount does not go to O.
The proof of Theorem postponed the appendix.
Both Theorems 1 and describe thase-transition under linear models with the same phase-transition boundary. This synchronicity justifies our idea of defining critical attack strength implicitly via the empirical loss and generalization gap A. Furthermore, since in real practice mostly we have R R(((0). ((((()),)) 0) + A, e* can be defined as
sup(e e((((e),)) S R(O())
is worth mentioning that is not coincident that both the gap of training losses and the difference of training trajectories occur at the same point. Under linear models, the loss function is always convex for both clean and adversarial training; hence the similarity of gradient descent trajectories highly depends on the similarity of the loss functions.
Note adversarial loss is not differentiable at 0. since sufficiently small, does not affect convergence.
We exclude the case that d 1, 1].

---

coefficients
decn, small cps
deen. large
d>>n, small
mean/reative city
0.3
011
-15
-1.0
0.0
0.5
Ingide
Figure 2: Left: the difference between clean training and adversarial training throughout the training process. The relative difference calculated as ([)( g (e)l/|(oo) The difference gets larger when increasing e. Right: The p-value for hypothesis testing test whether ((T follows the same asymptotic distribution (12)) (0). For all setups, the p-value starts to dramatically decrease when log(e/e" R O.
Remark 1. Theorems andd] reveal negative relationship between E" and n. i.e., larger n leads to larger discrepancy between adversarial training and cleaning training. However, it does not imply that a larger n hurts the adversarial training. With larger n, the adversarial training may act different to clean training, but by the Law of Large Numbers, a large can force the adversarial training to converge to the correct place.
The concept of c is for adversarial training, rather than commonly used data augmen- tation, e.g., via adding Gaussian noise (Reed and Marksll 1999.. The noisy sample is randomly allocated around the original sample. As a result, it does not hurt the training too much when considering the average effect of Gaussian noise, which is not the case for the adversarial attack.
Remark 2.
Simulation Evidences We use a simulation study demonstrate the above observations in the theorems. particular, we would like validate numerically (1) the adversarial training parameter is similar to the one in the clean training when < t" and vice versa, and (2) the adversarial training loss is similar to the clean training loss when e < E" and vice versa.
To verify (1), calculate (((()()) g
for each step t 1,..., 100, and repeat this experiment 100 times to obtain the average. Due to page limit, we postponed the detailed configurations the appendix. As shown in Figure 2 when the attack strength is large, there is great difference between the clean trained model parameters and adversarial trained model parameters. Note that in left panel in Figure we only compare the curves for the same nd. The comparison among different (d, 12) is not meaningful because the attack strengths are different.
Besides, we conduct hypothesis testing to check whether (e) is statistically significantly different from (0). We repeat clean training 300 times to obtain the mean and variance of (0). and then calculate the p-value of (e)) The p-value represents "the probability of obtaining test results at least as extreme the result actually observed, under the assumption that the null hypothesis is correct" (Wikipedia). A close-to-zen p-value means that ((T) is significantly different from ((T) (0). We take n 100 and d E (2, 5, 10) in this experiment. The results are shown in the right panel Figure 2 and one can see that there dramatic change in the p-value when 6 around e*, i.e. log(c/c") R O.
To verify (2), we also repeat 300 times of clean training to get the distribution of R((O(T), 0) and use this distribution to test whether the adversarial training result is from this distribution. As shown in Figure 3 similar to Figure 2 when the attack strength exceeds e*, there is significant difference between the distributions of the losses, and the p-value is close to zero.
Two-Layer Neural Networks While the linear regression problem enjoys the above properties, fol- lowing [a et al. 2020); Xing et al. (0211),, the two-layer neural network with vanishing initialization can be proved share similar properties as follows:
5

---

04
0.3
0.1
0.5
logiste
Figure The p- p-value for hypothesis testing to test whether R(o(T) (c), follows the same asymptotic distribution as R((((T (0), For all setups, the p-value is close to when log(c/c") = 0.
Proposition (Two-layer neural networks). Consider a two-layer neural network with smooth activation function 6
((x)
Vh 6lx bxxT0))))
(3)
with as the number of hidden nodes. When - oo, with vanishing initialization of and aj if we fix the second layer (az) and only train the hidden layer (0z). the network purameter satisfies similar (but not the same)) property as Theorem 1 and the loss results are the same as Theorem
The details of Proposition and the main idea of the proof are postponed to the appendix.
Experiments in Deep Neural Networks
In this section, we study the performance of the proposed in terms of (1) whether our choice of e* is reasonable in deep learning, and (2) what implications the e* can bring towards commonly used configurations in deep learning.
5.1 General Configurations
Here we describe some general setups in the implementation for both this section and the next section.
For datasets CIFAR-10, CIFAR-100, SVHN, modified the code of Rice et al. 2020) for implementation. We keep all the existing configurations (optimizer, neural network architecture, transformer) from the original code. In particular, if there is no specification, with batch size 128 to train on the full training set for 200 epochs. The learning rate is initialized as 0.1 for CIFAR and 0.01 for SVHN, and is divided by 10 at the 100th and 150th epochs. We consider PreActResNet18 and WideRessNet34 as those in Rice et al. (2020). After training 200 epochs, we find the epoch with the smallest adversarial testing loss the final model for early stopping. For MNIST, we implement CNN with two convolution layers and two fully connected layers.
use SGD optimizer
Besides, we also the loss TRADES and MART in Zhang al. (0099) and Wang al. (019b))
5.2 Verifying the effectiveness of
In this section, we present some metrics of adversarial training to argue that our choice of e" is reasonable. We use CIFAR- 10 with PreActResNet18 and L. in this section.
To obtain c", we first run clean training to get the clean testing loss, then run adversarial training for a wide range of c's and perform linear interpolation to obtain c*. For both clean training and adversarial training, we train from scratch and summarize the results in Table I From Table [ one
WW cannot directly compare the parameters of the neural networks trained from clean training and adversarial training. but the output predictive models indeed similar when small enough.

---

70
10
0.36
0.52
70
28
3.067
0.000
9.643
0,007
0.023
D70
30
0.32
0.44
0.007
0:64
-10
0.023
0.007
Figure 4: Connectivity between a clean trained model and adversarial robust models in terms of their clean training loss. Upper left: e 0. Upper right: e 0.5. Lower left: 2. Lower right: When increasing the right point (the checkpoint for adversarial training) is away from the basin.
can see that the clean testing loss for clean training is 0.1837. The threshold e* is the attack strength such that the adversarial training loss O. 1837. From the table, when c = 0.5, the adversarial training loss is 0.1802, implying that c* is approximately 0.5.
Table I: Adversarial training in PreActResNer18 using CIFAR-10.
Epoch
Ady Training
Adv Testing
loss
acc
loss
acc
0
104
0.0761
0.9743
0.1837
0.9405
0,5/255
101
0.1802
0.9376
0.2937
0.8977
1/255
102
0,2148
0.9227
0.371
0.8723
2/255
102
0.3411
0.8722
0.5218
0.8101
We conduct comparisons in some aspects, e.g., connectivity, overfitting, and FGM catastrophic overfitting, to show that the adversarial training with < c" is more similar to the clean training compared to stronger attacks.
The connectivity of deep neural networks aims to answer whether there is a path in the parameter space between two deep neural networks (of the same architecture), such that all the neural networks along this path have good prediction performance. It a useful tool to study the loss landscape of deep neural networks, e.g., Chao et al. 2020). Good connectivity implies that the two neural networks are in the same "basin" " of the loss.
Connectivity
To numerically figure out path, we use the method in Garipov et al. (2018). In order to have graphical presentation, we take num bend as 3, i.e., besides start and end neural networks, there is only one neural network to be trained in the path parameters. Figure 4 shows the connectivity between the clean trained model and the robust model. the clean model the 101st epoch, the right point is the adversarially trained model at the 101st epoch, and the upper point is the extra checkpoint to be trained. We take the 101st epoch because the adversarial training generally achieves the best adversarial testing loss around the 101st epoch. These three points together determine the black curve in the figure, and they are the same 2D plane. We calculate the clean training loss for other points in this plane to obtain the contour.
each subfigure, the left point represents
From Figure E one can observe that when to the lower two points. The middle part is also well connected, i.e., most of the connecting path belongs to the low-loss region (red area). When > 0.5, the low-loss region gradually shifts towards the clean model. This indicates that the adversarial training gradually moves out of the "basin".
and e = 0.5, low-loss region is almost symmetric

---

02255
0.5/255
12255
4255
50 100
150
125 200
1ter
Figure 5: Overfitiing in clean training and adversarial training. We the ratio of generalization error over the one at the st epoch to examine the overfitting level, where generalization error is the difference between adversarial training and testing loss. When e < 1, this ratio is small. Results from other attack strengths, c.g. 2/255 and 16/255, are not included in this figure to keep the plot clear, and they do not alter the main observations.
Generalization and Overfitting In the literature, it is observed that in deep neural networks with clean training, overfitting training data does not affect the generalization performance. Some studies, e.g., [elkin et al. (2018 2019a); Li et al. (2021): Chatterji et al. (021)) provide theoretical
justifications towards this phenomenon.
Our numerical results show no harmful overfitting in clean training, and the overfitting in adversarial training with e* is also not severe. shown in Figure 5 when e gets larger, the ratio (generalization gap)/(generalization gap at the 10 epoch) increases after the 100th epoch.
FGSM and Catastrophic Overfitting Based on Andriushchenko and Flammarion 2020), FGSM (i.e., the FGM approximation of -PGD attack) leads to catastrophic overfitting when attack strength is large, yielding an almost-zero adversarial testing accuracy using -PGD attack. Since FGM is only approximation of PGD, the FGM attack adversarial testing performance under PGD attack is expected to be worse. However, for Coo attack, the performance drop testing accuracy far more severe, and Andriushchenko and Flammarion (2020) reveals some threats in FGSM which cause this phenomenon. This catastrophic overlitting is second type of overfitting different from the one discussed in the previous paragraph.
strong the PGD attack, and its
Based on our intuition of e", the above concern does not hurt the training process when e < e First, since is small, the difference between FGM and PGD is small. Second, since adversarial training similar to clean training and clean training does not suffer from severe over-fitting problems, those problems can be avoided. To verify this, train CIFAR-10 using 10000/50000 samples with PGD/FGM adversarial training in different levels of L.e attacks. We take as 1/255, 2/255, 4/255, 8/255, 12/255, 16/255 adversarial testing accuracy.
whether obtains large adversarial testing loss with almost-zero
The results are summarized in Table 2 The column "c"" shows the critical e", and the column "Unstable e" is the minimal attack strength when FGSM starts the catastrophic over-fitting. One can that e" is always smaller than the unstable threshold. This is intuitive because adversarial training still similar to clean training with e z e". On the other hand, this also implies that one can use FSGM search c*.
Table Catastrophic overfitting in FGSM adversarial training CIFAR-10 using PreActResNetI1.
norm 6*
Unstable
10K
Loo
3.3/255 0.5/255
12/255
50K
L.
8/255
5.3 Observations in common settings
The previous section conducts numerical experiments to justify our choice of e". In this section, use this method in various datasets and neural network architectures to provide more insights and study how e" is affected by these factors.

---

Table 3: The value of in different datasets, neural network architectures, and training sample size. "PAResNet" and "WResNet" refer "PreActResNett and "WideResNet" respectively to save the margin.
Dataset
Architecture
Dataset
Architecture
7
(C
CIFAR-I0
PAResNer18
50K
0.5/255
MNIST
MLP(128)
50K
48.0/255
CIFAR-10
PAResNet18
10K
3.3/255
MNIST
MLP(16)
50K
26.
CIFAR-10
WResNee3441
50K
1.1/255
SVHN
PAResNer18
50K
4.9/255
CIFAR-10
WResNet34-10
50K
1.1/255
CIFAR-100
PAResNet18
25K
4.0/255
CIFAR-100
PAResNet18
50K
1.0/255
5.3.1
General Observations
An important observation in Table 3 is that the commonly used attack strengths are greater than e*. In the literature, we usually use La attack with strength 8/ /25 for 0.3 for MNIST. From our experiments, we observe that using all the 50000 training samples, the corresponding c" for CIFAR-10/100, SVHN, and MNIST are 0.5/255, 1.0/255,
and SVHN, and
48.0/255.
5.3.2 More Detailed Observations
Wide neural networks Different neural network architectures lead different e*. For MNIST, using a simple neural network of two convolution layers and two fully connected layers, when there 16 hidden nodes for the first FC layer, e* is 26. 1/255. When there are 128 hidden nodes, becomes 48/255. This observation verifies the argument in the existing literature that wider neural network structures are essential in adversarial training to enlarge the model capacity Xie and Yuille (2019): Authhrr (202I). For CIFAR-10. c* changes little when expanding the width of the neural network from to 10, implying that WideResNer33-10 may not be sufficient for adversarial training. Note that one need to be cautious when enlarging the network size, as wider neural networks tend to overfit the data even clean training.
Improved adversarial loss Some literature tries to improve the adversarial training process via improving the loss. We would like to examine how e* changes along these loss functions as well. We run TRADES and MART in CIFAR-10 with PreActResNet18 for this experiment, and the e" are 0.7/255 and 0.8/255, respectively. Although 0.7/255 and 0,8/255 are larger than 0.5/255 (i.e., 6" for vanilla adversarial training; See Table B they are still less than 8/255.
Training size From Table E for both CIFAR-10 and CIFAR-100, larger training size indicates a smaller c". There are two reasons for this phenomenon. First, when enlarging the sample size, the generalization gap gets smaller; thus, e* smaller. Second, as observed in Author (202I). when increasing the training size, the neural network has larger norm, implying that larger model capacity is needed to fit the attacked samples. As a result, the neural network architecture cannot handle the adversarial training properly, so the corresponding e* is smaller.
Faster Approximation of
To compute e*, a naive way to repeatedly perform the adversarial training for each possible e until it hits the threshold. This could be very time-consuming when solving the adversarial training from scratch (i.c., initializing from a near-zero random start). For instance, takes 20 minutes to train the CIFAR-10 10 dataset for clean training but takes 10 hours to complete adversarial training for given Hence we propose to speed up this process via a linear extrapolation approximation.
Due to the page limit, the intuition, potential difficulties, and final algorithm for approximating c* are postponed to Section in the appendix. Briefly speaking, due to good connectivity between the clean and adversarial training when & e" as in Figure E we consider using adversarial fine-tuning (Chen et al. 2020b) to replace the whole adversarial training (from scratch) process the grid search for e To overcome the potential over-fitting and learning rate tuning problems, we conduct adversarial fine- mue-tuning under similar e's and use extrapolation to approximate e". Since we are using extrapolation rather than interpolation, we also provide theoretical support to justify the correctness

---

Table 4: Performance of fine tuning. 2(FGM.a) represents that the adversarial fine-tuning uses FGM attack with runs of fine tuning.
norm
(PGD.1)
aPGD,3)
2(FGM,1)
2(GM,3)
10000
Loo
3.3/255
4.7/255
2.5/255
1.7/255
2.1/255
50000
L2
24.8/255
20.1/255
17.9/255
18.4/255
17.4/255
50000
Eo
0.5/255
0.58/255
0.52/255
0.53/255
0.59/255
of the algorithm. The numerical results are shown in Table 4 One can see that the proposed algorithm gives good estimate of e
7 What Can We Do If >> E-?
Our paper focuses on how determine e* and the consequence of >> 6", and it beyond our scope to study how to adjust adversarial training when > c*. We provide some potential solutions.
In general, when e > E", one needs overcome series of potential problems in adversarial training. In the literature, methods such as MART Wang et al. (2019)), Dynamic Wang et al. (2019a). FAT
Zhang et al. (2020a), HAT Rade and Moosavi-Deziooli (222T), smoothing Xie al. (020);; Xing
et al. (2021b) can overcome some of the problems. However, they may not be sufficient to resolve every problem caused by the fundamental gap between adversarial training and cleaning training revealed by this work. Alternatively. we suggest two other ways: (1) adjusting the neural network architecture to enlarge e", and (2) utilizing more information/data force adversarial training to converge to the correct place.
For (1), in numerical experiment, enlarging the size of the neural network can make e" larger. If controlling the over-fitting issue properly, this can be solution to improve adversarial robustness. Similarly, empirical experiments in various literature (e.g.. Xie and Yuille 2019 Rice al.) 2020)
Gowal et al. 2021) show that wider neural networks lead to better performance. Some other studies (e.g.. Huang et al. 2021 also study how the neural network architecture affects robustness.
For (2), there are several ways to utilize more information from the data. For example, Gowal et al. (202T) trains a clean classifier and an unlabeled data generator to generate extra synthetic data, which facilitate the adversarial training. This framework does not introduce any new data source but improves adversarial robustness, implying that vanilla adversarial training overlooks some information from the data. Although ex may be smaller than the actual e, using more information/data can force the training to converge the correct place. Similar studies can be found Carmon et al.
(2019): Xing et al. (021c).
8 Conclusion
Observing the great gap between empirical results in adversarially robust models and the theoret- ical studies this area, we conjecture that adversarial training acts differently from our common understanding of clean training in deep learning. Through intuitions in simple statistical models, we design a metric of similarity to determine whether adversarial training is "similar" to clean training not. Our results reveal that the commonly used adversarial training setups in literature take large attack strength so that is different from clean training given the current neural network architectures and data. We reveal some potential factors which affect e*. Besides, since adversarial training from scratch is time-consee consuminn we propose to use adversarial fine-tuning and extrapolation to do approximate c*. Such method can reduce 80% running time compared adversarial training from scratch while leading to reasonable estimates.
A future direction of this work is to theoretically understand how the neural network architecture af- fects c and based on which, to provide proper guidance on architecture selection that accommodates stronger attacks (c.g., the commonly used 8/255 attack strength in practice).
10

---

Acknowledgements
This project partially supported by NSF-SCALE MoDL (2134209) and ONR N00014-22-1-2680.
References
Allen-Zhu, Z.. and Li, Y. (2020), "Feature Purification: How Adversarial Training Performs Robust Deep Learning," arXiv preprint arXiv:2005. 10190.
Andriushchenko, M. and Flammarion, N. (2020), "Understanding and improving fast adversarial training." arXiv preprint arXiv:2007.02617.
Author, A. (2021), "Adversarial Rademacher Complexity of Deep Neural Networks.".
Ba, J., Erdogdu, M., Suzuki, T., Wu, D., and Zhang, T. (2020), "Generalization of two-layer neural networks: asymptotic viewpoint," in 8th International Conference on Learning Representations.
Baratin, A., George, T., Laurent, C., Devon Hjelm, R., Lajoie, G., Vincent, P., and Lacoste-Julien, S. (2020), "Implicit Regularization in Deep Learning: A View from Function Space," arXiv e-prints, arXiv-2008.
Bartlett, L.,, Long, M.M, Lugosi, G., and Tsigler, A. (2020). "Benign overfitting in linear regression," Proceedings of the National Academy of Sciences, 117, 30063-30070.
Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019a), "Reconciling modern machine learning and the bias-variance trade-off, Proceedings of the National Academy of Sciences, 116, 15849-15854,
Belkin, M., Hsu, D., and Mitra, P. (2018), "Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate," arXiv preprint arXiv: 180.
Belkin, M., Hsu, D., and Xu, J. (2019h), "Two models of double descent for weak features," arXiv preprint arXiv: 0903.0033..7 075771.
Bubeck, S. and Sellke, M. (2021), "A Universal Law of Robustness via Isoperimetry," arXiv preprint arXiv:2105.12806
Carmon, Y., Raghumathan, A., Schmidt, L., Duchi, J. C., and Liang, P. S. (2019), "Unlabeled data improves adversarial robustness," Advances in Neural Information Processing Systems, pp. 1119--1203.
Chao, S.-K., Wang, Z., Xing, Y., and Cheng, G. (2020), "Directional Pruning of Deep Neural Networks," arXiv preprint arXiv:2006.0933: 09358..
Chatterji, N. S., Long, M., and Bartlett, I. (2021), "The Interplay Between Implicit Bias and Benign Overfitting in Twy-rayer Linear Networks," arXiv preprint arXiv:2108.11489
Chen, L., Min, Y., Zhang, M., and Karbasi, A. (2020a), "More data expand the generalization gap between adversarially robust and standard models," arXiv preprint arXiv:2002.04725.
Chen, T., Liu, S., Chang, S., Cheng, Y., Amini, L.,, and Wang, Z. (2020b), "Adversarial robustness: From self-supervised pre-training fine-tuning," in Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 699-708.
Dan, C., Wei, Y., and Ravikumar, P. (2020), "Sharp Statistical Guaratees for Adversarially Robust Gaussian Classification," " in International Conference Machine Learning, PMLR, pp. 2345- 2355.
Du, S. S. and Lee, J. D. (2018), "On the power of over- parametrization in neural networks with quadratic activation in Proceedings of the 35th International Conference on Machine Learning, PMLR, vol. 80 of Proceedings of Machine Learning Research, pp. -1388.
Du, S. S., Lee, J. D., Li, H., Wang, I.., and Zhai, X. (2019), "Gradient descent finds global minima of deep neural networks," Proceedings of the 36th International Conference on Machine Learning, PMLR, vol. 97 of Proceedings of Machine Learning Research, pp. 1675-1685.
11

---

Du, S. S., Zhai, X., Poczos, B., and Singh. A. (2018). "Gradient descent provably optimizes over- parameterized neural networks," arXiv preprint 11X.0::810.02054.
Frederickson, C., Moore, M., Dawson, G., and Polikar, R. (2018), "Attack strength vs. detectability dilemma in adversarial machine learning," in 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 1-8.
Gao, R., Cai, T., Li, H., Wang, L., Hsich, C.-J,, and Lee, J. D. (2019), "Convergence of adversarial training in overparametrized networks," arXiv preprint PrxXiv:/06.0793
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D., and Wilson, A. G. (2018), "Loss surfaces, mode connectivity, and fast ensembling of dnns," in Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8803-8812, https//gahub..com/mmgggrooov// mode-connecttiityy
Gong, Z., Wang, W., and Ku, W.-S. (2017), "Adversarial and clean data are not twins," arXiv preprint 1eXiv:/704.04900
Gowal, S., Rebuffi, S.-A., Wiles, O., Stimberg, F., Calian, D. A., and Mann, T. A. (2021), 'Improving Robustness using Generated Data," Advances in Neural Information Processing Systems, 34.
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. (2019), "Surprises in high-dimensional ridgeless least squares interpolation," " arXiv preprint UrXiv:1903.08500
Hendrycks, D., Lee, K., and Mazeika, M. (2019). "Using pre-training can improve model robustness and uncertainty." arXiv preprint arXiv: 1.XXvv:1011.0996:
Hu, T., Shang, Z., and Cheng, G. (2020), "Sharp Rate of Convergence for Deep Neural Network Classifiers under the Teacher-Student Setting," " arXiv preprint arXiv:2001.06892
Huang, H., Wang, Y., Erfani, S., Gu, Q., Bailey, J., and Ma, X. (2021), "Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks," Advances in Neural Information Processing Systems, 34.
Javanmard, A. and Soltanolkotabi, M. (2020), "Precise statistical analysis of classification accuracies for adversarial training," arXiv preprini arXiv:2010. 1/213.
Javanmard, A., Soltanolkotabi, M., and Hassani, H. (2020), "Precise tradeoffs in adversarial training for linear regression," arXiv preprint arXiv:2002. 10477.
Kohler, M. and Langer, S. (2019), "On the rate of convergence of fully connected very deep neural network regression estimates." arXiv preprint arXiv: 11133.
Lee, K. and Chandrakasan, A. P. (2020), "Rethinking Empirical Evaluation of Adversarial Robustness Using First-Order Attack Methods," arXiv preprint 2rXiiv2006.01304
Li, Z., Zhou, Z.-H., and Gretton, A. (2021), "Towards an Understanding of Benign Overfitting in Neural Networks," arXiv preprint arXiv:2 02.00
Mehrabi, M., Javanmard, A., Rossi, R. A., Rao, A.. and Mai, T. (2021), "Fundamental Tradeoffs in Distributionally Adversarial Training," arXiv preprint arXiv:2101.06300
Min, Y., Chen, L., and Karbasi, A. (2020), "The curious case of adversarially robust models: More data can help, double descend, or hurt generalization," arXiv preprint 10X.
Neyshabur, B. (2017), "Implicit regularization in deep learning." arXiv preprint arXiv: 170901193..
Neyshabur, B., Bhojanapalli, S., and Srebro, N. (2017a), "A pac-bayesian approach to spectrally- normalized margin bounds for neural networks," arXiv preprint arXiv
Neyshabur, B., Tomioka, R., Salakhutdinov, R., and Srebro, N. (2017b), "Geometry of optimization and implicit regularization in deep learning." arXiv preprint arXiv: 100.0077....071.
Rade, R. and -oosavi-Dezfooli, S.-M. (2021), "Helper-based adversarial training: Reducing exces- sive margin to achieve a better accuracy vs. robustness trade-off." in ICMI. 202/ Workshop on Adversarial Machine Learning.
12

---

Raghunathan, A., Xie, S. M., Yang, F., Duchi, J. C., and Liang, P. (2019), "Adversarial training can hurt generalization." arXiv preprint 1906. 06032..
Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg. F., Wiles, O., and Mann, T. (2021), "Fixing data augmentation to improve adversarial robustness," arXiv preprint arXiv:2 103v:2103.0/946.
Reed, R. and MarkslI, R. J. (1999), Neural smithing: supervised learning in feedforward artificial neural networks, Mit Press.
Rice, L.,, Wong, E.,, and Kolter, 7. (2020), "Overfitting in adversarially robust deep learning," arXiv preprint arXiv:2002. 1569, https///gitub.cco/loouaaaaa////u wowee///ooooooo/oooooooot.eennnn Loverfitti...
Sinha, A., Namkoong, H., and Duchi, J. (2018), "Certifying some distributional robustness with principled adversarial training,"
Taheri, H., Pedarsani, R., and Thrampoulidis, C. (2020), "Asymptotic Behavior of Adversarial Training in Binary Classification," " arXiv preprint arXiv:2010.13275.
Taheri, M., Xie, F., and Lederer, J. (2021), "Statistical guarantees for regularized neural networks," Neural Networks, 142, 148-161.
Wang. Y., Ma, X., Bailey, J., Yi, Zhou, B., and Gu, Q. (2019a), "On the convergence and robustness of adversarial training," in International Conference on Machine Learning, pp. 6586-6599
Wang. Y., Zou, D., J., Bailey, J., Ma, X., and Gu, Q. (2019b), "Improving adversarial robustness re- quires revisiting misclassified examples," in International Conference on Learning Representations, https://gituub.co//iieeee Wang//AA...
Wu, B., Chen, J., Cai, D., He, X., and Gu, Q. (2020), "Does Network Width Really Help Adversarial Robustness?" arXiv preprint arXiv:2010.01279.
Xie, C., Tan, M., Gong, B., Yuille, A., and Le, Q V. (2020), "Smooth adversarial training," arXiv preprint arXiv:2006.145: 14536.
Xie, C. and Yuille, A. (2019), "Inrrguingg properties of adversarial training at scale," arXiv preprint AXXiv:906.00770
Xing, Y., Song, Q., and Cheng, G. (2021a), "On the algorithmic stability of adversarial training," " Advances in Neural Information Processing Systems, 34, 26523-26535.
(2021b), "On the generalization properties of adversarial training," in International Conference on Artificial Intelligence and Statistics, PMLR, pp. 505-513.
Xing, Y., Zhang, R., and Cheng, G. (2021c), "Adversarially Robust Estimate and Risk Analysis in Linear Regression," in International Conference on Artificial Intelligence and Statistics, PMLR, pp. 514-522.
Yao. Z., Gholami, A., Lei. Q.. Keutzer, K., and Mahoney, M. W. (2018). "Hessian- analysis of large batch training and robustness to adversaries." arXiv preprint /802.08880.
Yin, D., Ramchandran, K., and Bartlett, P. (2018), "Rademacher complexity for adversarially robust generalization," arXiv preprint
Zhai, R., Cai, T., He, D., Dan, C., He, K., Hopcroft, J., and Wang, L. (2019), "Adversarially robust generalization just requires more unlabeled data," arXiv preprint arXiv: 1906.00555.
Zhang, H., Yu, Y., Jiao, J., Xing, P., Ghaoui, I.. E., and Jordan, M. I. (2019), "Theoretically Principled Trade-off between Robustness and Accuracy," in Proceedings of the 36th International Conference on Machine Learning, PMLR, vol. 97 of Proceedings of Machine Learning Research, pp. 7472-7482, http://gihbbb.ooo///ooonggg////
Zhang. J.. Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., and Kankanhalli, M. (2020a), "Attacks which do not kill training make adversarial learning stronger," in International Conference Machine Learning, PMLR, pp. 11278-11287
Zhang, Y., Plevrakis, O.,, Du, S. S., Li. X., Song, Z., and Arora, S. (2020b), "Over-parameteriin Adversarial Training: An Analysis Overcoming the Curse of Dimensionality." arXiv preprint 2rXiv:2002.06668.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] This is theoretical study on existing methodologies. It does not introduce new methods in the real-world applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the github links for the original code modified from.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We mention the time when introducing our speed-up algorithm.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
14


Reviews:
[{"review_id": "lsa5yYLtmk", "reviewer": "Reviewer_T2si", "summary": "The authors study the behavior of a simple linear model (and a shallow NN) under adversarial training and theoretically discover qualitative differences between the small-$\\epsilon$ regime and the large-$\\epsilon$ regime. They claim that these regimes are separated by a critical perturbation strength $\\epsilon^*$ and propose a theoretically motivated method of determining it. They run experiments on both their simple models as well as standard ResNets in order to validate their theoretical findings.", "questions": "In Theorem 1, it appears that the $\\epsilon$ refers not only to the attack strength but actually to the true Gaussian noise present in the data. Why does it make sense to jointly vary the true noise on the data and the budget of the adversarial training? Is the $\\epsilon$ even referring to the adversarial attack strength at all here?\n\nIn line 208, the authors talk about running adversarial training for a wide range of $\\epsilon$'s. Are they referring to the results in Table 1? Because, if so, that is not a wide range of $\\epsilon$'s. If not, then more results should be shown, ideally as plots.", "limitations": "The authors do not discuss societal impact of their work, but I agree with their assessment in the checklist that this does not apply to their work, as it is fairly theoretical.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "he paper is well-structured and given that the paper is technically complicated in some parts, the authors do a good job of clearly conveying their message.\n\nA central claim of the paper is the existence of a phase transition when increasing $\\epsilon$ through some critical value. Usually, phase transitions are characterized by a discontinuity in one or more properties of the system. To me it seems that (at fixed dimension $d$ and samples $n$) the paper simply shows that $\\epsilon=0$ and $\\epsilon=\\infty$ differ in loss and in optimal solution. This is completely obvious. What would be interesting is if some property of the system actually varies non-smoothly as we pass through a critical value of $\\epsilon$. The most direct way of studying this is by varying $\\epsilon$ in a fine grained way and demonstrating a discontinuity in one or several properties of the network. The authors justify why they do not do this by the high cost of running adversarial training (if this is a problem, the authors could consider using the method of [1] for their experiments).\n\nAs it is, I do not think that the authors provide enough evidence of an interesting discontinuity. For example, on the right-hand side of Figure 2, it seems fairly clear that the loss distribution is varying smoothly as a function of $\\epsilon$. This is not a surprising result. A similar observation is true for Figure 3 and an even more striking counter-example to the authors claim of a phase transition seems to be their own Figure A.1 in the appendix. The only weak evidence that I think the paper provides for the phase transition claim is Figure 4: here, looking at the generalization gap ratio after overfitting at 200 epochs, it seems that there might be a non-smooth increase as we vary $\\epsilon$. \n\n[1] \"Fast is better than free: Revisiting adversarial training\" Wong, Rice, Kolter at ICLR'20", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lFFZlA4bGdv", "reviewer": "Reviewer_Xy6A", "summary": "Adversarial training does not achieve performance comparable to the standard training on clean data. This paper investigates the critical point of the magnitude of the adversarial perturbations with which training trajectories of adversarial training become significantly different from those of standard training. By using simple linear regression models, this paper claims that the order of the critical point is in $\\Theta(\\sqrt{d/n})$ where $d$ is the dimension of data, and $n$ is data size.\nIn addition, it claims that the critical point can be estimated by using the difference between training loss of adversarial training and test loss of standard training. To support the claims, this paper provides results of various experiments, e.g., evaluation of the difference of parameters in standard training and adversarial training in the cases (d<<n and n<<d), connectivity of the loss, and empirical evaluation of the critical point.\n", "questions": "What do lim inf d/n > 0 and lim d/n=0 in Theorem 1 mean? I think d and n are fixed in training. ", "limitations": "Theoretical results are restricted to simple linear models and specific two-layer networks.  Though I admit that deriving theoretical results for deep neural networks are difficult due to nonlinearity, experiments might be made more sophisticated to convince that theoretical results are valid for deep models. For example, $\\epsilon$ in Table 1 could be evaluated at finer intervals to show the phase transition experimentally.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- This paper provides interesting analyses of training trajectories of adversarial training.\nThis study will help readers better understand adversarial training and may inspire the creation of new and stronger methods for adversarial training.\n\n- This paper claims the relationship between the critical point and catastrophic overfitting of FGSM.\nIf we know the magnitude of attacks against which FGSM can make models sufficiently robust, it is worth in practice because FGSM is an efficient method. However, the results of the experiments (Table 2) are less informative because they do not list the robust accuracy of FGSM against PGD.\n\n## Weaknesses\n- The theoretical analysis is based on simple models. Though I admit the analysis for deep models is difficult, experimental evidence of deep models can be obtained instead of theoretical results. In the current paper, the experimental results are not sufficient as described below.\n\n- The experimental conditions are not complete, and the results do not fully support the claim. The simulation on the left of Figure 2 provides little information because $\\varepsilon$ varies across conditions of n<<d and d<<n. I think that if $d$ is fixed across the conditions, small and large $\\varepsilon$ can be fixed across conditions to compare the difference between n<<d and d<<n conditions on the same scale. It would be useful if these results support Theorem 1. In addition, the experimental conditions on the right side of Figure 2 are not shown. It seems to be important to know what the value of $\\sqrt{n/d}$ is in this setting.\n\n- If this paper claims a phase transition, it is not convincing unless experiments show the discontinuous change in the trajectory against $\\varepsilon$. In Table 1, the evaluation interval of $\\varepsilon$ is too coarse and the losses seem to increase almost linearly, and there is no discontinuity like a phase transition. In addition, Experimental conditions are chosen artificially. For example, Figure 4 shows the results of $\\varepsilon$=1/255, 4/255, but the results of $\\varepsilon$=1/255, 4/255 are not in Table 1 and Figure 3. Conversely, the results of $\\varepsilon$=2/255 are shown in Table 1 and Figure 3 but not in Figure 4. These discrepancies in experimental conditions could be seen as cherry-picking in order to claim that the critical point is $\\varepsilon=0.5/255$. In my opinion, many experiments should be conducted on the interval [0,0.5/255] and on the interval [0.5/255,2/255] to obtain more convincing results.\n\n- The proof of theoretical results is unclear, and its validity is difficult to evaluate.  Since the claims in theorems are different from the claims in the proof, it is difficult to follow the proof. For example, Theorem 1 uses lim inf d/n > 0 and lim d/n=0 in the claims, but its proof uses $d/n<\\infty$ and $d/n\\rightarrow \\infty$, and the relationship is not explained. In line 635, A, B, C, D should be written by using $\\hat{\\theta}(\\varepsilon)$. To improve clarity, theorems and proofs should be written in a consistent and easy-to-understand notation.\n\n## Minor issues and Comments.\n- In Table 1, $\\varepsilon$ is divided by 255, but in the text, $\\varepsilon$ is 0.5 without being divided by 255. It is necessary to rewrite the text to determine which is correct.\n\n- In my opinion, Proposition 1 is incomplete. This theoretical claim contains unclearness though it is supplemented by the footnote. I think the term \"the similar property but not the same\" should be explained exactly in this proposition.\n\n- [a] reports that adversarial training can improve the accuracy on clean data under certain conditions. If this paper can connect such studies to the study fo the difficulty of adversarial training, this paper might be more valuable. For example, if the paper can reveal how weak attacks contribute to clean accuracy, this paper would be worth publishing.\n\n[a] Xie, Cihang, et al. \"Adversarial examples improve image recognition.\" CVPR2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "edWJc3Xl6II", "reviewer": "Reviewer_rFja", "summary": "This paper propose a way to study the effects of the strength of adversarial training on the trajectories of optimization of machine learning models. The specific examples used in this paper is deep neural networks. The proven theorems are on a much simpler linear model.", "questions": "no questions", "limitations": "The authors have not discuss potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weakness:\n1. The phrase \"phase transition\" must be use with care. In the theory of critical phenomena, phase transitions means existence of singularities in the thermodynamics limit. Most common types of phase transitions are first order and second order phase transitions. I recommend the authors to use another descriptor other than 'phase transition'.\n2. The paper prove theorems for simple linear model, I am skeptical that these results can be generalised to arbitrary models.\n3. The key equation in the paper is the definition of \\epsilon^* in line 151. I cannot make the connection of the theorems with this statement in line 150: \"This equivalency justifies our idea of . . . . \". I can hardly understand what was being justified. I feel that definition of \\epsilon^* may be intuitive.\n4. Is there a typo in line 132? \"lim inf\" means what? Also, for taking limits, it is better to state what quantity tends to this limit.\n5. Line 216: about connectivity. this analysis may not bring too much value in the understanding of the content of this paper.\n6. Table 2 is not being explained well.\n\nStrengths:\n1. While many past papers focus on prediction accuracies, this paper focus on understanding of the training process. Indeed the focus on tuning parameters and network to make predictions just that little bit better may not help this research field move forward. Papers focusing on understanding brings more value to this field of research. Any paper that push the boundaries of understanding should be encouraged.\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "aGgh0fOA1vi", "reviewer": "Reviewer_ZEbX", "summary": "The paper studies how to mitigate the gap between adversarial training and clean training by finding the optimal $\\epsilon$. They first validate the phase transition boundary in simple linear regression model then extend to large-scale neural network. Based on their observation, the commonly used attack strengths are greater than \u000foptimal $\\epsilon$. They proposed an efficient way to approximate the $\\epsilon^\\star$ which can make the adversarial training more reliable and evaluate on three dataset.", "questions": "The author claims the proposed method can find an optimal $\\epsilon$ for adversarial training. I'm wondering if there has a tradeoff between robust acc. and clean acc.. Table 1. shows the loss and acc. value between training and testing, but the author only focus on the explanation of loss value.\n\nCan it be used to evaluate on the ensemble attack method such as  AutoAttack?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: the paper is well organized and the proof of theorem is solid. The analysis of connectivity is good.\n\nWeaknesses: Some analysis can be added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept