Title: Wide Two-Layer Networks can Learn from Adversarial Perturbations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Adversarial Perturbations

Adversarial Examples

Adversarial Attacks

Non-Robust Features

Perturbation Learning

Learning from Adversarial Perturbations
Abstract: Adversarial examples have raised several open questions, such as why they can deceive classifiers and transfer between different models. A prevailing hypothesis to explain these phenomena suggests that adversarial perturbations appear as random noise but contain class-specific features. This hypothesis is supported by the success of perturbation learning, where classifiers trained solely on adversarial examples and the corresponding incorrect labels generalize well to correctly labeled test data. Although this hypothesis and perturbation learning are effective in explaining intriguing properties of adversarial examples, their solid theoretical foundation is limited. In this study, we theoretically explain the counterintuitive success of perturbation learning. We assume wide two-layer networks and the results hold for any data distribution. We prove that adversarial perturbations contain sufficient class-specific features for networks to generalize from them. Moreover, the predictions of classifiers trained on mislabeled adversarial examples coincide with those of classifiers trained on correctly labeled clean samples. The code is available at https://github.com/s-kumano/perturbation-learning.

Wide Two-Layer Networks can Learn from Adversarial Perturbations

Abstract

Adversarial examples have raised several open questions, such as why they can deceive classifiers and transfer between different models. A prevailing hypothesis to explain these phenomena suggests that adversarial perturbations appear as random noise but contain class-specific features. This hypothesis is supported by the success of perturbation learning, where classifiers trained solely on adversarial examples and the corresponding incorrect labels generalize well to correctly labeled test data. Although this hypothesis and perturbation learning are effective in explaining intriguing properties of adversarial examples, their solid theoretical foundation is limited. In this study, we theoretically explain the counterintuitive success of perturbation learning. We assume wide two-layer networks and the results hold for any data distribution. We prove that adversarial perturbations contain sufficient class-specific features for networks to generalize from them. Moreover, the predictions of classifiers trained on mislabeled adversarial examples coincide with those of classifiers trained on correctly labeled clean samples. The code is available at https://github.com/s-kumano/perturbation-learning .

1 Introduction

Adversarial examples [41], which are imperceptibly perturbed inputs designed to deceive machine learning models, have raised significant concerns about the robustness and reliability of these models. Despite their importance, the underlying mechanisms of adversarial examples are not yet fully understood. A prevailing hypothesis to explain the intriguing properties of adversarial examples is the 'feature hypothesis' [22]. This hypothesis posits that adversarial perturbations, while appearing as imperceptible noise to humans, contain class-specific features. The feature hypothesis provides a unified explanation for several puzzling phenomena associated with adversarial examples, such as their ability to deceive classifiers, transferability across models, and so on (cf. Section 2.1).

Perturbation learning [22] provides empirical evidence supporting the feature hypothesis. In this learning, classifiers are trained solely on adversarial examples that are mislabeled in human perception, 1 yet they demonstrate remarkable generalization to clean test data (Fig. 1). For example, classifiers achieved 77% accuracy on the correctly labeled clean test dataset of CIFAR-10 [27], even though they

bird

horse

cat

(Entirely mislabeled)

Test on clean samples in test dataset

cat

bird / horse / cat

cat

(Correctly labeled)

Training on adv. examples

from

trained on clean samples

̸

were trained on entirely mislabeled adversarial examples (e.g., a cat adversarial image labeled as a bird) [28]. This surprising result suggests that adversarial perturbations encode class-relevant features that enable classifiers to learn meaningful representations. However, despite the empirical support, the theoretical foundations of the feature hypothesis and perturbation learning remain limited. While a recent study [28] provided theoretical justifications, their results rely on stringent assumptions about data distribution, perturbation design, training procedure, and model architectures.

In this study, we theoretically address the understanding and justification of the feature hypothesis and perturbation learning. First, to support the feature hypothesis, we show that adversarial perturbations, while appearing as random noise, are parallel to the weighted sum of all training samples. This result suggests that a single perturbation derived from a classifier and input can potentially contain information about the entire training dataset. In particular, for some specific cases (e.g., when training samples are mutually orthogonal), perturbations include all training data and labels without loss of information. We then reveal that class features within perturbations enable classifiers to generalize from them. Specifically, under three mild conditions, the predictions of a classifier trained on adversarial perturbations are consistent with those of a classifier trained on correctly labeled clean samples. These three conditions can be interpreted from geometric and quantitative perspectives. Finally, we demonstrate that under similar conditions, the prediction agreement is observed between a classifier trained on mislabeled adversarial examples and one trained on correctly labeled clean samples, justifying the empirical success of perturbation learning.

Our analysis assumes two-layer neural networks with sufficient width but does not impose any assumptions on data distribution, which is a substantial progress from prior work [28] that considered mutually orthogonal training samples. In addition, our perturbation design, training procedure, activation functions, and bias availability are milder. In short, as shown in Tab. 1, except for the wide width assumption, our analysis requires milder conditions than prior work. Our contributions can be summarized as follows:

We provide a theoretical justification for the feature hypothesis and perturbation learning using wide two-layer neural networks, considering any data distribution and realistic problem settings. Except for the wide width, our assumptions are substantially milder than [28].

We demonstrate that adversarial perturbations are parallel to the weighted sum of training samples, suggesting that a single perturbation can potentially contain information about the entire training dataset. This result supports the feature hypothesis.

We prove that under three mild conditions, the predictions of a classifier trained on perturbations are consistent with those of a classifier trained on correctly labeled clean samples. Moreover, under similar conditions, the prediction agreement between a classifier trained on mislabeled adversarial samples and one trained on clean samples is observed, providing a theoretical justification for the empirical success of perturbation learning.

2 Background and Related Work

2.1 Feature Hypothesis and Perturbation Learning

It has been hypothesized that adversarial perturbations contain class-specific features, although appearing as random noise [22]. This hypothesis, or feature hypothesis, offers a unified explanation for several open questions related to adversarial examples. For example, misclassification by classifiers and transferability across models [21, 41] can be attributed to the response to features within perturbations. Furthermore, according to this hypothesis, adversarially robust models achieve robustness by discarding brittle yet predictive features and focusing on more stable and semantically meaningful features. This interpretation explains the phenomena observed with robust models, such as the trade-off between accuracy and robustness [13, 32, 34, 35, 40, 42, 46, 47], perceptually-aligned gradients [1, 4, 7, 17, 18, 26, 38, 39, 42, 48], and enhanced transfer learning capabilities [1, 12, 37, 43].

Perturbation learning 1 [22] provides empirical support for the feature hypothesis. In perturbation learning, the dataset appears entirely mislabeled to human perception. However, the hypothesis suggests that adversarial perturbations in the dataset include label-aligned class features. Indeed, it has been observed that classifiers trained through perturbation learning can extract generalizable features from these perturbations and achieve high test accuracy (e.g., 92% for MNIST [11], 54% for FashionMNIST [45], and 77% for CIFAR-10 [27]), empirically justifying the feature hypothesis [22, 28].

While the feature hypothesis and perturbation learning are empirically effective in understanding adversarial examples, their theoretical foundations are very limited. Only one recent study [28] theoretically demonstrated that perturbations contain class features and that classifiers can generalize from them. However, their results relied on stringent conditions (e.g., mutually orthogonal training samples), which might not fully explain the success of perturbation learning in diverse settings.

In this study, for wide two-layer networks, we obtain results equivalent to those in [28] under more relaxed conditions (cf. Section 3.4). We provide the first theoretical justification for the feature hypothesis and perturbation learning under any data distribution and in a mild training setting.

2.2 Theoretical Framework: Lazy Training

Theoretical analysis of neural networks is generally challenging due to the non-convex nature of the loss surface. To address this, recent studies have focused on the lazy training regime, where the parameters of neural networks hardly change during training [5, 6, 9, 20, 25, 30, 33, 44, 49]. In this regime, neural networks behave almost linearly around their initialization, simplifying the learning dynamics. One of the key observation in lazy training is that, in wide two-layer neural networks, most derivatives of hidden outputs through (Leaky-) ReLU activation remain constant during training [30], which forms the basis of our theoretical framework (cf. Section 3.3). This observation has been extended to show that the neural tangent kernel remains invariant during training [2, 3, 14, 15, 23, 29].

In contrast, the feature learning regime, where parameters move significantly away from their initialization, has been explored in various studies [9, 20, 44]. Prior work on justifying perturbation learning [28] employs the feature learning regime, building on related findings in this area [19, 24, 31]. In our study, we adopt the lazy training regime and relax several conditions assumed in previous work [28] by introducing a wide width assumption (cf. Tab. 1). This adjustment is enabled by differences in the theoretical tools used.

3 Theoretical Results

Notation. For n ∈ N , let [ n ] := { 1 , . . . , n } . For z 1 , z 2 ∈ R d , we denote the Euclidean norm by ∥ z 1 ∥ and the inner product by ⟨ z 1 , z 2 ⟩ . Vectors z 1 and z 2 are called parallel and are denoted by z 1 / / z 2 if there exists C ∈ R such that z 1 = C z 2 . Let N ( µ, σ 2 ) be the Gaussian distribution with mean µ ∈ R and variance σ 2 ≥ 0 and U ( S ) be the uniform distribution on a set S ⊂ R . We use Ω( · ) , Θ( · ) , and O ( · ) only to hide constant factors, and ˜ Ω( · ) , ˜ Θ( · ) , and ˜ O ( · ) to hide polylogarithmic factors.

3.1 Problem Setup

In this study, we consider the dynamics of perturbation learning in binary classification problem with a two-layer neural network trained by gradient flow. First, we formally define the perturbation learning framework. The outline of perturbation learning is as follows: (i) train a classifier on correctly labeled clean samples, (ii) create adversarial samples based on the trained classifier, and (iii) train another classifier on the mislabeled adversarial samples.

Network trained on correctly labeled clean samples. We consider a two-layer neural network f : R d → R . Let V := ( v 1 , . . . , v m ) ⊤ ∈ R m × d and a := ( a 1 , . . . , a m ) ⊤ ∈ R m be the hidden weight and bias, respectively. We also describe V := ( V ij ) 1 ≤ i ≤ m, 1 ≤ j ≤ d . Let α := ( α 1 , . . . , α m ) ⊤ ∈ R m be the readout weight. While V and a are trainable, α is fixed during training. Denote the trainable parameters by θ V , a := { V , a } . We initialize V ij ∼ N (0 , 1 /d ) , a i ∼ N (0 , 1) , and α i ∼ N (0 , 1 /m ) for each i ∈ [ m ] and j ∈ [ d ] . The activation function is either ReLU or Leaky-ReLU ϕ ( x ) := max( γx, x ) for γ ∈ [0 , 1) . Finally, the network is given by f ( x ; θ V , a ) := ∑ m i =1 α i ϕ ( ⟨ v i , x ⟩ + a i ) .

Network trained on mislabeled adversarial samples. Similarly to f , we define a network trained on mislabeled adversarial samples as g ( x ; θ W , b ) := ∑ m i =1 β i ϕ ( ⟨ w i , x ⟩ + b i ) . Note that the initializations of f and g are independent.

Loss function. We consider a differentiable, non-decreasing loss function ℓ : R → R , satisfying ℓ ′ ( z ) ≥ 0 for any z ∈ R . Examples of such loss functions include the identity loss ℓ ( z ) := z , exponential loss ℓ ( z ) := exp( z ) , and logistic loss ℓ ( z ) := ln(1 + exp( z )) .

Training. We here describe the training process of the network f on correctly labeled clean samples. The training of g is similarly defined. Let D := { ( x n , y n ) } N n =1 ⊂ R d ×{± 1 } be a correctly labeled training dataset. The loss over D is defined as L ( θ V , a ; D ) := (1 /N ) ∑ N n =1 ℓ ( -y n f ( x n ; θ V , a )) . The network parameters are updated by gradient flow d θ V , a ( t )/d t := -∂ L ( θ V , a ( t ); D )/ ∂ θ V , a , where t ≥ 0 is the training time. We consider T f > 0 training steps, producing f ( · ; θ V , a ( T f )) . For notational simplicity, we write f ( · ; t ) := f ( · ; θ V , a ( t )) .

Adversarial perturbations. We consider a single-step gradient-based perturbation, which is a common perturbation design [21]. An adversarial example x adv n ∈ R d and its corresponding adversarial perturbation r n ∈ R d are defined as follows:

Note that we do not consider whether f ( · ; T f ) perfectly classify D . We discuss whether the classifier g , trained on adversarial examples crafted via f ( · ; T f ) , can mimic the predictions of f ( · ; T f ) .

where ϵ > 0 is the perturbation constraint and y adv n ∈ {± 1 } is the target label. The adversarial perturbation r n on x n is designed to increase y adv n f ( x adv n ; T f ) under the constraint ∥ r n ∥ ≤ ϵ .

Mislabeled dataset. We consider two configurations of a dataset D adv for training g . First, we follow the original perturbation learning approach, where classifiers are trained on adversarial perturbations superposed on natural images, i.e., D adv := { ( x adv n , y adv n ) } N n =1 . This setting helps to understand the prior perturbation learning process. Second, we directly consider learning from perturbations rather than adversarial examples, i.e., D adv := { ( r n , y adv n ) } N n =1 . This setting directly addresses the question of whether classifiers can generalize from class features in perturbations.

Summary. The problem setting is summarized as follows:

Setting 3.1 (Perturbation learning) . Independently initialize V ij ∼ N (0 , 1 /d ) , W ij ∼ N (0 , 1 /d ) , a i ∼ N (0 , 1) , b i ∼ N (0 , 1) , α i ∼ N (0 , 1 /m ) , and β i ∼ N (0 , 1 /m ) for each i ∈ [ m ] and j ∈ [ d ] . Train a two-layer neural network f parameterized by θ V , a with ( γ -scaled Leaky-) ReLU on a dataset D := { ( x n , y n ) } N n =1 using gradient flow with a loss L ( θ V , a ; D ) for training time T f > 0 . Create a dataset D adv by one of the following procedures with { y adv n } N n =1 ∈ {± 1 } N :

Train a two-layer neural network g parameterized by θ W , b on the dataset D adv using gradient flow with a loss L ( θ W , b ; D adv ) for training time T g > 0 .

Our interests are (i) the relationship between perceptually-noise-like adversarial perturbations { r n } N n =1 and clean training samples { ( x n , y n ) } N n =1 (cf. Theorem 3.3), and (ii) whether the classifier g ( · ; T g ) trained on the adversarial perturbations or samples D adv can mimic the predictions of the classifier f ( · ; T f ) trained on the clean samples D (cf. Theorems 3.4 and 3.5).

3.2 Main Results

For z 1 , z 2 ∈ R d , we use Φ( z 1 , z 2 ) ∈ ( γ (1 + γ ) / 2 , (1 + γ ) / 2] defined as (cf. Lemma C.4):

where ϕ ′ ( x ) := d ϕ ( x )/d x . First, we introduce an assumption on network width.

Assumption 3.2 (Wide network) . Network width m satisfies

where x ′ n := r n for Scenario (a) and x ′ n := x adv n for Scenario (b) in Setting 3.1. In particular, m> ˜ O ( d 2 ( T f + T g ) 2 ) for ℓ ( s ) = s .

This assumption requires sufficiently large width m that regularizes the variations in parameters and forms the basis of lazy training (cf. Section 3.3). The width is always required to grow with the speed of the squared input dimension d 2 . The relationship between the width and two training times, T f and T g , depends on the training set { ( x n , y n ) } N n =1 and loss function ℓ . For example, if the training set is easily separable and the loss has an exponential tail, the derivative of the loss function might decrease rapidly with training time t and small m is enough to satisfy the assumption. For the identity loss, m is consistently required to satisfy ˜ Ω( d 2 ( T f + T g ) 2 ) . Note that the required values of T f and T g (and the corresponding m ) for a desirable loss value remain an open question in the community. Our experimental results show that m ≈ 100 is sufficient to verify our theorems for high-dimensional Gaussian distributions. Under this assumption, we consider the direction of the adversarial perturbation.

Theorem 3.3 (Direction of adversarial perturbation) . Let δ = Θ(1) be a small positive number. Under Assumption 3.2, for any n ∈ [ N ] , with probability at least 1 -δ , the adversarial perturbation

(a) Func. margin cond. 1

(b) Func. margin cond. 2

(c) Agreement cond.

(d) Intersection

r n is parallel to the weighted sum of training samples as follows:

where ξ n satisfies ∥ ξ n ∥ = ˜ O (1) . In particular, for ℓ ( s ) = s ,

Note that the confidence level δ only logarithmically affects the norm of the remainder term ξ n . This theorem indicates that the direction of a single perturbation can be represented as the weighted sum of y k x k and remainder term ξ n . Interestingly, this result suggests that a single perturbation derived from a classifier and sample can potentially contain information about the entire training dataset { ( x n , y n ) } N n =1 . Particularly, in some cases (e.g., training samples are mutually orthogonal), y k x k are not cancelled out by each other, and thus the single perturbation r n contains all training data and labels without loss of information. 2 These results theoretically support the feature hypothesis. Consider the case with the identity loss. While the norm of the first term is O ( T f √ d ) , the norm of the remainder is constrained to ˜ O (1) , suggesting that larger training time T f and input dimension d strengthen the alignment between the perturbation and weighted sum.

Then, we consider the learning solely from these perturbations. The following theorem is a special case of Theorem D.17, which addresses a broader loss class and any sampling of y adv n ∈ {± 1 } .

Theorem 3.4 (Perturbation learning, Scenario (a), special case of Theorem D.17) . Consider Scenario (a) in Setting 3.1. Assume ℓ ( s ) = s and y adv n ∼ U ( {± 1 } ) for every n ∈ [ N ] . Let δ = Θ(1) be a small positive number and

Under Assumption 3.2, for any z ∈ R d , if

then, with probability at least 1 -δ , sgn( f ( z ; T f )) = sgn( g ( z ; T g )) holds.

Note that the confidence level δ only logarithmically affects the right terms of Ineqs. (9) and (10), which is why these terms appear independent of δ . This theorem states that the predictions of a classifier g trained solely on adversarial perturbations { ( r n , y adv n ) } N n =1 coincide with those of a classifier f trained on standard training samples { ( x n , y n ) } N n =1 if the three conditions hold. The two

functions, ˆ f and ˆ g , which govern these conditions, can be viewed as key components that significantly influence the predictions of f and g (cf. Section 3.3). The conditions can be interpreted as follows.

Geometrical perspective. The functional margin conditions, Ineqs. (9) and (10), require the functional margins of ˆ f and ˆ g to exceed certain thresholds. In the input space z ∈ R d , these conditions correspond to regions far from the decision boundaries ˆ f ( z ) = 0 and ˆ g ( z ) = 0 (Fig. 2(a) and (b)). In a d -dimensional space, L 2 -distance scales with √ d , making the right terms of Ineqs. (9) and (10) relatively small when the perturbation size ϵ = Θ( √ d ) ; hence, a larger d facilitates the satisfaction of these conditions. Furthermore, Eq. (11) necessitates the agreement of the signs of these two decision boundaries (Fig. 2(c)). Consequently, the region where all conditions hold, i.e., where the prediction match occurs, can be characterized by the two piecewise linear functions (Fig. 2(d)).

Quantitative perspective (functional margin conditions). A large perturbation size ϵ facilitates the satisfaction of Ineq. (10). In high-dimensional spaces, achieving the required margin conditions demands perturbations of at least Ω( √ d ) in magnitude, aligning with empirical scaling laws for L 2 perturbations. However, increasing ϵ alone is insufficient for the satisfaction because the right term of Ineq. (10) has an ϵ -irrelevant term ˜ O (1 /T f ) . Assume ϵ = Θ( √ d ) . The absolute values of ˆ f and ˆ g grow with Θ( d ) due to ⟨ x , z ⟩ , while the right terms of Ineqs. (9) and (10) are independent of d . Thus, a larger d consistently facilitates the satisfaction. The training times T f and T g can reduce the right terms, but these terms contain time-independent terms ˜ O (1) and ˜ O (1 / √ N ) , indicating that longer training times do not necessarily guarantee the satisfaction. A large sample size N also helps to satisfy Ineq. (10), but similarly, it is not sufficient. In summary, while a larger input dimension d consistently support the success of perturbation learning, a larger perturbation size ϵ , sample size N , and training times T f , T g provide partial, but not definitive, benefits.

Quantitative perspective (agreement condition). 3 It is difficult to interpret the agreement condition, Eq. (11), in its current form. We consider the following sufficient condition (cf. Lemma D.19):

Note that the left term can exceed one as λ ( z 1 , z 2 ) lies in (0 . 34 , 1] . It is clear that a large negative slope of Leaky-ReLU γ facilitates the satisfaction. The magnitude of the left term depends on the consistency of the correlation (inner product) between z and y n x n for every n . For example, when z consistently exhibits a positive or negative correlation with y n x n , the left term exceeds one, and the condition is satisfied. In contrast, if z positively correlates with half of the samples and negatively with the other half, the left term may output a small value, and the condition is not satisfied. In summary, the agreement condition depends on the consistency of the correlation between z and y n x n .

Finally, we justify the success of perturbation learning in Scenario (b).

Theorem 3.5 (Perturbation learning, Scenario (b), special case of Theorem D.18) . Consider Scenario (b) in Setting 3.1. Assume ℓ ( s ) = s and y adv n ∼ U ( {± 1 } ) for every n ∈ [ N ] . Let δ = Θ(1) be a small positive number and

Under Assumption 3.2, for any z ∈ R d , if the functional margin condition 1 (Ineq. (9) ),

then, with probability at least 1 -δ , sgn( f ( z ; T f )) = sgn( g ( z ; T g )) holds.

This is a special case of Theorem D.18, which addresses differentiable, non-decreasing losses and any sampling of y adv n ∈ {± 1 } . Similarly to Theorem 3.4, this theorem states that the predictions of a classifier g trained on randomly labeled adversarial examples { ( x adv n , y adv n ) } N n =1 coincide with those of a classifier f trained on standard training samples { ( x n , y n ) } N n =1 if the three conditions hold. Functional margin condition 1 is consistent with that in Theorem 3.4, i.e., Ineq. (9). The definition of ˆ g b ( z ) is slightly different from ˆ g a ( z ) in Theorem 3.4, with r n replaced by x adv n . Due to this change in the definition of ˆ g b ( z ) , functional margin condition 2, Ineq. (15), and the agreement condition, Eq. (16), slightly differ from those in Theorem 3.4.

Assume ϵ = Θ( √ d ) . Similarly to Ineq. (10), the left term of Ineq. (15) grows with O ( d ) due to the inner product. In contrast to Ineq. (10), the right term of Ineq. (15) includes a term that grows with √ ∑ N n ( ⟨ x n , z ⟩ +1) 2 /N = O ( d/ √ N ) . This suggests that Scenario (b) necessitates a larger sample size N than Scenario (a) to mitigate the effect of d .

Furthermore, Eqs. (11) and (16) hold simultaneously (i.e., sgn( ˆ f ( z )) = sgn(ˆ g a ( z )) = sgn(ˆ g b ( z )) ) if ∑ N k =1 y k Φ( x n , x k ) ⟨ x k , z ⟩ outputs the same sign for any n . This indicates that there might exist regions where the prediction match is observed regardless of the scenarios, and these regions are partially determined by the n linear boundaries. Note that Ineq. (12) also serves as a sufficient condition for Eq. (16), and the quantitative analysis for Eq. (11) can be applied to Eq. (16) as well.

3.3 Sketch of Proof

In this section, for simplicity, we provide a sketch of the proof for Theorems 3.3 and 3.4 with infinite network width m →∞ , networks without biases, and identity loss. A proof for the general case can be found in Appendix D.

Lazy training. First, we introduce the concept of lazy training, where network parameters and outputs of hidden neurons change negligibly during training when the network width is sufficiently large [9, 30]. Since a readout weight α i is sampled from N (0 , 1 /m ) , from gradient flow, for any z ∈ R d ,

Therefore, as m →∞ , the inner product between the time variation of a hidden parameter and an input approaches zero. This suggests that the sign of the output of a hidden neuron do not change:

Therefore, ϕ ′ ( ⟨ v i ( T f ) , z ⟩ ) = ϕ ′ ( ⟨ v i (0) , z ⟩ ) . Recall ϕ ( z ) := max( γz, z ) .

Theorem 3.3. From the perturbation definition Eq. (1), the perturbation r n is parallel to ∇ x n f ( x n ; V ( T f )) . Using ϕ ′ ( ⟨ v i ( T f ) , z ⟩ ) = ϕ ′ ( ⟨ v i (0) , z ⟩ ) ,

The first term is constrained to ˜ O (1) . Let Φ( x n , x k ) := E v ∼N (0 , 1 /d ) [ ϕ ′ ( ⟨ v , x n ⟩ ) ϕ ′ ( ⟨ v , x k ⟩ )] . Using ∑ m i =1 α 2 i ϕ ′ ( ⟨ v i (0) , x n ⟩ ) ϕ ′ ( ⟨ v i (0) , x k ⟩ ) → Φ( x n , x k ) as m →∞ , the second term becomes

Theorem 3.4. Similarly to the above, we can represent the adversarial perturbation r n as follows:

Assuming ∑ m i =1 α i ϕ ′ ( ⟨ v i (0) , z ⟩ ) ⟨ v i (0) , z ⟩ = ˜ O (1) ≈ 0 for simplicity, the network prediction f ( z ; V ( T f )) trained on { ( x n , y n ) } N n =1 can be represented as follows:

and sgn( f ( z ; V ( T f ))) = sgn( ∑ N n =1 y n Φ( x n , z ) ⟨ x n , z ⟩ ) . In addition, g ( z ; W ( T f )) trained on { ( r n , y adv n ) } N n =1 can be represented as follows:

and sgn( g ( z ; W ( T g ))) = sgn( ∑ N n =1 Φ( r n , z ) ∑ N k =1 y k Φ( x n , x k ) ⟨ x k , z ⟩ ) . Thus, if the agreement condition Eq. (11) holds, then sgn( f ( z ; V ( T f ))) = sgn( g ( z ; W ( T g ))) .

Formal proof. In the above sketch of proof, we have introduced several approximations. Rigorous evaluations are provided in Appendix D. For example, in the sketch, we assumed m →∞ , ensuring that the signs of all hidden layer outputs remain unchanged. In contrast, the formal proof derives a bound on the width m that ensures that the number of hidden neurons with flipped signs is at most O ( √ m ) , which makes the discussion (e.g., about Eqs. (18) and (19)) more complicated. Moreover, in Eq. (21), we neglected the first term of Eq. (19), but the formal proof carefully considers the impact on the subsequent steps. The functional margin conditions arise from the evaluation of these remainder terms.

3.4 Comparison with Prior Work and Limitations

In this section, we compare our results with [28] and discuss the limitations of our work. In summary, our results justify the feature hypothesis and perturbation learning under substantially milder conditions than [28], except for network width (cf. Tab. 1). The assumption of wide two-layer networks is our main limitation.

Goals, results, and tools. The goals of our work and [28] are the same: justifying the feature hypothesis and perturbation learning. The conclusions drawn are also equivalent. However, our assumptions are much milder than theirs. This is due to the differences in the analytical approaches. While they leverage research on feature learning [19, 24, 31], we utilize the concept of lazy training [9, 30], which enables us to substantially relax the conditions.

̸

Data distribution. Prior work imposes a strong assumption that training samples with/without adversarial perturbations are mutually orthogonal, i.e., ⟨ x n , x k ⟩ ≈ 0 and ⟨ x adv n , x adv k ⟩ ≈ 0 for any n = k . This condition is stringent and is hard to hold for real-world datasets. Moreover, it may not even hold for data sampled from a zero-mean Gaussian distribution in some common situations (e.g., the sample size is sufficiently larger than the dimension). We do not impose any assumptions on the data distribution. This is the first result that theoretically supports the feature hypothesis and the success of perturbation learning on realistic data distribution.

Perturbation design. Prior work defined the perturbation form using the decision boundary of a classifier. However, this is not only uncommon but also theoretically computable only in limited problem settings. Additionally, they constrained perturbation size to ϵ = Θ( √ d/N ) , which becomes unrealistically small for a large sample size and is far from the practical constraint ϵ = Θ( √ d ) . We employ a single-step gradient-based method [21], which is commonly used in practice, and the perturbation constraint can be set arbitrarily.

Training time, loss function, network bias, and activation. First, it should be noted that these constraints are critical for deriving the results in [28]. This is because their theoretical framework [19, 24, 31] substantially requires the above conditions. We consider arbitrary training time, a wide class of loss functions, and (Leaky-) ReLU networks with bias availability. In contrast, they considered infinite training time, loss functions with exponential tails, homogeneous neural networks (thus requiring no bias), and Leaky-ReLU networks (the theorem becomes harder to hold as the negative slope of Leaky-ReLU approaches zero), which are essential for deriving their results (cf. the proofs of Theorem 4.4 in [31] and the proof of Theorem 3.2 in [19]).

10

20

30

40

50

60

70

80

90

100

Input dimension

d

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Limitations. Compared to [28], our main limitation is the requirement for sufficient, though finite, network width. Moreover, our analysis is confined to two-layer networks, a common constraint in previous work. In practice, perturbation learning often employs deeper, and not necessarily wider, networks, which limits the direct applicability of our theoretical insights to more complex architectures. This assumption of a shallow network introduces another limitation. While deep neural networks typically capture high-level features from images and adversarial attacks are considered to exploit them, our framework focuses solely the low-level features (i.e., x n itself) in adversarial perturbations and their extraction through perturbation learning, as shown in Theorems 3.3 to 3.5. Relaxing the shallow network constraint may allow us to capture a broader set of features present in adversarial perturbations. Despite these limitations, our work is the first to rigorously support the feature hypothesis and validate perturbation learning under realistic data distributions, perturbation designs, and training settings, marking a substantial advancement in the theoretical understanding of adversarial examples.

4 Experiments

A comprehensive set of experiments conducted to validate our theorems can be found in Appendix B. In this section, we briefly present two results that confirm Theorem 3.4. As a training dataset D := { ( x n , y n ) } N n =1 , we employed a synthetic training dataset to easily change the input dimension, which effectively helps perturbation learning in both scenarios, as predicted by our theorems. Note that the perturbation learning on real-world datasets can be found in the literature [22, 28]. We generated synthetic data and labels from the mean-shifted Gaussian distribution as follows: { x n } N n =1 are independently sampled from N (0 . 3 × y n × 1 , I ) , and y n is set to one if n ∈ [ N/ 2] and minus one otherwise. The experimental settings are as follows: d = 100 , N = 1 , 000 , m = 100 , γ = 0 , ℓ ( s ) := s , ϵ = 0 . 01 , and the number of training steps is set to 1,000 for both f and g . The experimental results for perturbation learning under Scenario (a) are shown in Fig. 3. A high input dimension facilitates the alignment between f and g . Our theoretical results assume a wide network width, and Fig. 3 indicates that a sufficiently large width consistently stabilize the alignment.

5 Conclusion

We provided a theoretical justification for perturbation learning and the feature hypothesis. We demonstrated that adversarial perturbations contain class-specific features sufficient for networks to generalize from. Moreover, we revealed that the predictions of a classifier trained solely on these perturbations or mislabeled adversarial examples coincide with those of a classifier trained on correctly labeled training samples under three mild conditions. Except for wide two-layer networks, our assumption is substantially milder than prior work [28].

Acknowledgments and Disclosure of Funding

S. Kumano was supported by JSPS KAKENHI Grant Number JP23KJ0789 and by JST, ACT-X Grant Number JPMJAX23C7, JAPAN. H. Kera was supported by JSPS KAKENHI Grant Number JP22K17962. T. Yamasaki was supported by JSPS KAKENHI Grant Number JP22H03640, JST

ASPIRE Program Grant Number JPMJAP2303, and Institute for AI and Beyond of The University of Tokyo.

References

G. Aggarwal, A. Sinha, N. Kumari, and M. Singh. On the benefits of models with perceptuallyaligned gradients. In ICLR WS , 2020.

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via overparameterization. In ICML , pages 242-252, 2019.

S. Arora, S. Du, W. Hu, Z. Li, and R. Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In ICML , pages 322-332, 2019.

M. Augustin, A. Meinke, and M. Hein. Adversarial robustness on in-and out-distribution improves explainability. In ECCV , pages 228-245, 2020.

Y. Cao and Q. Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In NeurIPS , volume 32, 2019.

Y. Cao and Q. Gu. Generalization error bounds of gradient descent for learning overparameterized deep relu networks. In AAAI , volume 34, pages 3349-3356, 2020.

P. Chalasani, J. Chen, A. R. Chowdhury, X. Wu, and S. Jha. Concise explanations of neural networks using adversarial training. In ICML , pages 1383-1391, 2020.

S.-H. Chang, P. C. Cosman, and L. B. Milstein. Chernoff-type bounds for the gaussian error function. IEEE Transactions on Communications , 59(11):2939-2944, 2011.

L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming. In NeurIPS , volume 32, 2019.

Y. Cho and L. Saul. Kernel methods for deep learning. In NeurIPS , volume 22, 2009.

L. Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Z. Deng, L. Zhang, K. Vodrahalli, K. Kawaguchi, and J. Y. Zou. Adversarial training helps transfer learning via better representations. In NeurIPS , volume 34, pages 25179-25191, 2021.

E. Dobriban, H. Hassani, D. Hong, and A. Robey. Provable tradeoffs in adversarially robust classification. IEEE Transactions on Information Theory , 2023.

S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In ICML , pages 1675-1685, 2019.

S. S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes overparameterized neural networks. In ICLR , 2019.

J. Duchi. Lecture notes on statistics and information theory, 2023.

L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry. Adversarial robustness as a prior for learned representations. arXiv:1906.00945 , 2019.

C. Etmann, S. Lunz, P. Maass, and C.-B. Schönlieb. On the connection between adversarial robustness and saliency map interpretability. In ICML , 2019.

S. Frei, G. Vardi, P. L. Bartlett, N. Srebro, and W. Hu. Implicit bias in leaky relu networks trained on high-dimensional data. In ICLR , 2023.

M. Geiger, S. Spigler, A. Jacot, and M. Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment , 2020(11):113301, 2020.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. In NeurIPS , volume 32, pages 125-136, 2019.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS , volume 31, 2018.

Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. In NeurIPS , volume 33, pages 17176-17186, 2020.

Z. Ji and M. Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. In ICLR , 2020.

S. Kaur, J. Cohen, and Z. C. Lipton. Are perceptually-aligned gradients a general property of robust classifiers? In NeurIPS WS , 2019.

A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

S. Kumano, H. Kera, and T. Yamasaki. Theoretical understanding of learning from adversarial perturbations. In ICLR , 2024.

J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS , volume 32, 2019.

Y. Li and Y. Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In NeurIPS , volume 31, 2018.

K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. In ICLR , 2020.

M. Mehrabi, A. Javanmard, R. A. Rossi, A. Rao, and T. Mai. Fundamental tradeoffs in distributionally adversarial training. In ICML , pages 7544-7554, 2021.

A. Montanari and Y. Zhong. The interpolation phase transition in neural networks: Memorization and generalization under lazy training. The Annals of Statistics , 50(5):2816-2847, 2022.

A. Raghunathan, S. M. Xie, F. Yang, J. Duchi, and P. Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In ICML , 2020.

A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang. Adversarial training can hurt generalization. In ICML WS , 2019.

P. Rigollet and J.-C. Hütter. High-dimensional statistics. arXiv:2310.19244 , 2023.

H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? In NeurIPS , volume 33, pages 3533-3545, 2020.

S. Santurkar, A. Ilyas, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Image synthesis with a single (robust) classifier. In NeurIPS , volume 32, 2019.

S. Srinivas, S. Bordt, and H. Lakkaraju. Which models have perceptually-aligned gradients? an explanation via off-manifold robustness. In NeurIPS , volume 36, 2023.

D. Su, H. Zhang, H. Chen, J. Yi, P.-Y. Chen, and Y. Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In ECCV , pages 631-648, 2018.

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR , 2014.

D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In ICLR , 2019.

F. Utrera, E. Kravitz, N. B. Erichson, R. Khanna, and M. W. Mahoney. Adversarially-trained deep nets transfer better: Illustration on image classification. In ICLR , 2021.

B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro. Kernel and rich regimes in overparametrized models. In COLT , pages 3635-3673, 2020.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 , 2017.

Y.-Y. Yang, C. Rashtchian, H. Zhang, R. R. Salakhutdinov, and K. Chaudhuri. A closer look at accuracy vs. robustness. In NeurIPS , volume 33, pages 8588-8601, 2020.

H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML , pages 7472-7482, 2019.

T. Zhang and Z. Zhu. Interpreting adversarially trained convolutional neural networks. In ICML , pages 7502-7511, 2019.

D. Zou, Y. Cao, D. Zhou, and Q. Gu. Gradient descent optimizes over-parameterized deep relu networks. Machine Learning , 109:467-492, 2020.

A Comparison with Prior Work

In this section, we compare our findings with those of prior work [28] and highlight new insights beyond technical contributions.

A.1 Feature Hypothesis (Theorem 3.3)

Our result offers a new insight into the alignment between perturbations and training samples through the residual term ξ n . The direction of a perturbation vector comprises two components: a weighted sum of the training samples (the main term) and a residual term. Our result suggests that as the input dimension increases, the residual term becomes smaller than the main term, thereby enhancing the alignment. In other words, perturbations more robustly encode class-specific features. This insight is unattainable in prior research due to the absence of a residual term in their limited problem setting.

Additionally, our finding suggests that extended training time reinforces the directional alignment between perturbations and training samples. This concept is supported by practical intuition and experience but not addressed in prior work.

Our result further introduces a coefficient, Φ( x n , x k ) , for each adversarial perturbation. The coefficient Φ( x n , x k ) depends on the slope of the activation function and the similarity between x n and x k (cf. Eq. (4)). This suggests that training samples with higher similarity to each other exhibit a stronger influence within a perturbation. Although prior work includes similar coefficients, they cannot be explicitly computed.

A.2 Perturbation Learning (Theorems 3.4 and 3.5)

Our result establishes an explicit connection between the success of perturbation learning and training factors, such as training time T , perturbation size ϵ , input dimension d , sample size N , and confidence level δ . This result enhances our understanding of how these factors influence perturbation learning. For example, perturbation learning is more likely to succeed with a larger input dimension d and sample size N . Notably, our findings indicate that while a larger d consistently facilitates successful perturbation learning, a larger N alone is insufficient. In contrast, existing research does not elucidate the roles of these variables, merely showing that success is achievable when both d and N approach infinity.

B Experimental Settings and Other Results

B.1 Datasets

We utilized two synthetic datasets and two widely used datasets, MNIST [11] and FashionMNIST [45]. The first synthetic dataset is derived from a zero-mean Gaussian distribution: { x n } N n =1 are independently sampled from N (0 , I ) and { y n } N n =1 are independently sampled from U ( {± 1 } ) . The second synthetic dataset is based on a mean-shifted Gaussian distribution: { x n } N n =1 are independently sampled from N (0 . 3 × y n × 1 , I ) and y n is set to one for n ∈ [ N/ 2] and minus one otherwise. We used data only from classes 1 and 2 in MNIST (i.e., digits 1 and 2) and those from classes 0 and 9 in Fashion-MNIST (i.e., T-shirt and ankle boot). To measure the agreement ratio between network predictions from standard training and perturbation learning, for the real-world dataset cases, we used standard test datasets, and for the synthetic dataset cases, we used 1,000 samples independently and identically sampled according to the training data distribution.

B.2 Settings

In this section, for notational simplicity, we denote the number of training epochs in standard training by T f and in perturbation learning by T g . Note that the original definitions of T f and T g are continuous training steps in gradient flow, i.e., gradient descent with an infinitely small learning rate (cf. Section 3.1), which is conceptually distinct from the discrete steps in gradient descent with finitely small learning rates in practice. In addition, we denote the learning rates in standard training and perturbation learning as η f and η g , respectively.

We used non-stochastic gradient descent (i.e., each gradient calculation uses the entire dataset) with 0.9 momentum and the learning rate scheduler that multiplies a learning rate by 0.1 when a training loss has stopped improving during 10 epochs. For Figs. 3 to A12, we selected the best accuracy, agreement ratio, and cosine similarity from training with multiple initial learning rates.

In Figs. A4 to A11, the blue lines represent the accuracy of the classifier from standard training on the training dataset { ( x n , y n ) } N n =1 . Namely, these mean the training accuracy of f ( · ; T f ) . The orange lines represent the accuracy of the classifier from perturbation learning on the standard (clean) training dataset { ( x n , y n ) } N n =1 rather than the perturbed dataset { ( x adv n , y adv n ) } N n =1 . Namely, these mean the ratio that counterintuitive generalization occurs. Note that the orange lines stay around fifty percent (chance accuracy) if adversarial perturbations are not included in { x adv n } N n =1 (cf. ϵ = 0 in Figs. A4 to A11). The green lines represent the agreement ratio between predictions f ( · ; T f ) and g ( · ; T g ) on a test dataset.

The cosine similarity in Fig. A12 is the average one between the experimentally calculated adversarial perturbation and the theoretically predicted one (cf. Eq. (1)) across all n . The blue lines are the same as those in Figs. A4 to A11.

The two axes in Fig. A13 are the normalized average vectors of samples from the positive and negative classes, respectively. The blue circles and orange crosses correspond to the projections of positive and negative samples onto these axes. The gray and green areas indicate regions where two predictions are consistent and inconsistent, respectively. The red solid lines represent ˆ f ( z ) = 0 . The black dashed lines represent ˆ g a ( z ) = 0 in Scenario (a) and ˆ g b ( z ) = 0 in Scenario (b).

Mean-Shifted Gaussian and Scenario (a). A common experimental setting for the mean-shifted Gaussian dataset and Scenario (a) in Figs. A4 and A12 is as follows: input dimension d = 100 , hidden dimension m = 100 , activation function slope γ = 0 , number of training samples N = 1 , 000 , loss function ℓ ( s ) = s , training epochs in standard training T f = 1 , 000 and in perturbation learning T g = 1 , 000 , perturbation size ϵ = √ d × 0 . 001 2 = 0 . 01 , and learning rates in standard training η f = 1 or 0 . 1 and in perturbation learning η g = 1 or 0 . 1 . However, for the comparison of T f (i.e., the graph in the fourth row and the first column in Fig. A4), we set η f only to 0.1. For the comparison of d (i.e., the graph in the first row in Fig. A4), we employed ϵ = √ d × 0 . 001 2 . In Fig. A13, we used d = 100 , m = 100 , γ = 0 , N = 1 , 000 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ d × 0 . 001 2 = 0 . 01 , η f = 1 , and η g = 1 .

Mean-Shifted Gaussian and Scenario (b). A common experimental setting for the mean-shifted Gaussian dataset and Scenario (b) in Figs. A5 and A12 is as follows: d = 5 , 000 , m = 100 , γ = 0 , N = 10 , 000 , ℓ ( s ) = s , T f = 1 , 000 , T g = 1 , 000 , ϵ = √ d × 0 . 01 2 = 0 . 7 , η f = 1 or 0 . 1 , and η g = 1 or 0 . 1 . However, for the comparison of T f (i.e., the graph in the fourth row and the first column in Fig. A5), we set η f only to 0.01. In addition, for the comparison of T g (i.e., the graph in the fourth row and the second column in Fig. A5), we set η g only to 0.01. Furthermore, we set η f = 10 , 5 , 1 , 0 . 1 , 0 . 01 and η g = 10 , 5 , 1 , 0 . 1 , 0 . 01 for the evaluation with the logistic loss (i.e., the graph in the first row and the second column in Fig. A5). For the comparison of d (i.e., the graph in the first row in Fig. A5), we employed ϵ = √ d × 0 . 01 2 . In Fig. A13, we used d = 100 , m = 100 , γ = 0 , N = 5 , 000 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ d × 0 . 01 2 = 0 . 1 , η f = 1 , and η g = 1 .

Zero-Mean Gaussian and Scenario (b). Acommon experimental setting for the zero-mean Gaussian dataset and Scenario (b) in Figs. A7 and A12 is as follows: d = 10 , 000 , m = 100 , γ = 0 , N = 10 , 000 , ℓ ( s ) = s , T f = 1 , 000 , T g = 1 , 000 , ϵ = √ d × 0 . 1 2 = 10 , η f = 1 or 0 . 1 , and η g = 1 or 0 . 1 . However, for the comparison of T f (i.e., the graph in the fourth row and the first column in Fig. A7), we set η f only to 0.1. In addition, for the comparison of T g (i.e., the graph in the fourth row and the second column in Fig. A7), we set η g only to 0.1. For the comparison of d (i.e., the graph in the first row in Fig. A7), we employed ϵ = √ d × 0 . 1 2 . In Fig. A13, we used d = 1 , 000 , m = 1 , 000 ,

Zero-Mean Gaussian and Scenario (a). Acommon experimental setting for the zero-mean Gaussian dataset and Scenario (a) in Figs. A6 and A12 is as follows: d = 10 , 000 , m = 100 , γ = 0 , N = 10 , 000 , ℓ ( s ) = s , T f = 1 , 000 , T g = 1 , 000 , ϵ = √ d × 0 . 001 2 = 0 . 1 , η f = 1 or 0 . 1 , and η g = 1 or 0 . 1 . However, for the comparison of T f (i.e., the graph in the fourth row and the first column in Fig. A6), we set η f only to 0.1. For the comparison of d (i.e., the graph in the first row in Fig. A6), we employed ϵ = √ d × 0 . 001 2 . In Fig. A13, we used d = 1 , 000 , m = 1 , 000 , γ = 0 , N = 2 , 000 , ℓ ( s ) = s , T f = 1 , 000 , T g = 1 , 000 , ϵ = √ d × 0 . 001 2 = 0 . 031 , η f = 1 , and η g = 1 .

γ = 0 , N = 10 , 000 , ℓ ( s ) = s , T f = 1 , 000 , T g = 1 , 000 , ϵ = √ d × 0 . 01 2 = 0 . 31 , η f = 0 . 1 , and η g = 0 . 1 .

MNIST and Scenario (a). A common experimental setting for MNIST and Scenario (a) in Figs. A8 and A12 is as follows: m = 100 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 or 0 . 001 , and η g = 0 . 01 or 0 . 001 . In Fig. A13, we used m = 1 , 000 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 , and η g = 0 . 01 .

Fashion-MNIST and Scenario (a). A common experimental setting for Fashion-MNIST and Scenario (a) in Figs. A10 and A12 is as follows: m = 100 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 or 0 . 001 , and η g = 0 . 01 or 0 . 001 . In Fig. A13, we used m = 1 , 000 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 , and η g = 0 . 01 .

MNIST and Scenario (b). A common experimental setting for MNIST and Scenario (b) in Figs. A9 and A12 is as follows: m = 100 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 or 0 . 001 , and η g = 0 . 01 or 0 . 001 . In Fig. A13, we used m = 1 , 000 , γ = 0 , ℓ ( s ) = s , T f = 1 , 000 , T g = 1 , 000 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 , and η g = 0 . 01 .

Fashion-MNIST and Scenario (b). A common experimental setting for Fashion-MNIST and Scenario (b) in Figs. A11 and A12 is as follows: m = 100 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 01 2 / 2 = 0 . 14 , η f = 0 . 01 or 0 . 001 , and η g = 0 . 01 or 0 . 001 . In Fig. A13, we used m = 1 , 000 , γ = 0 , ℓ ( s ) = s , T f = 100 , T g = 100 , ϵ = √ 784 × 0 . 1 2 / 2 = 1 . 4 , η f = 0 . 001 , and η g = 0 . 001 .

C Lemmas

In this section, we derive fundamental properties of random variables.

Lemma C.1 (Properties of Gaussian random variables) . Let σ 2 > 0 be a positive constant. Let X 1 , . . . , X m ∈ R be m ∈ N i.i.d. Gaussian random variables that follow N (0 , σ 2 ) .

(a) For 0 < δ < 1 , with probability at least 1 -δ ,

(b) Let Y 1 , . . . , Y m ∈ [ γ 2 , 1] be m independent random variables with 0 ≤ γ 2 < 1 . Suppose that Y 1 , . . . , Y m are independent of X 1 , . . . , X m . For 0 < δ < 1 , with probability at least 1 -δ ,

(c) For exp ( -2( η √ m +1) 2 /m ) < δ < 1 , with probability at least 1 -δ , there are at most η √ m ∈ [ m -1] instances such that

Proof. Let C > 0 be a positive constant.

(a) From [36],

Thus,

10

20

30

40

50

60

70

80

90

100

Input dimension

d

0

.

50

0

.

75

1

.

00

10

15

20

25

30

35

40

45

50

Input dimension

d

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

1

10

2

10

3

10

4

Sample size

N

0

.

50

0

.

75

1

.

00

1

2

3

4

5

6

7

8

9

10

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

(b) For any i ∈ [ m ] ,

By Jensen's inequality,

10

1

10

2

10

3

Input dimension

d

0

.

50

0

.

75

1

.

00

10

1

10

2

10

3

Input dimension

d

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

1

10

2

10

3

10

4

Sample size

N

0

.

50

0

.

75

1

.

00

1

2

3

4

5

6

7

8

9

10

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

Since E [ X 2 n i ] = σ 2 n (2 n -1)!! ≤ 2 n σ 2 n n ! and E [ Y n i ] ≤ 1 ,

For | t | ≤ 1 / (8 σ 2 ) ,

Thus, X 2 i Y i follows SE(64 σ 4 , 8 σ 2 ) , where SE ( a, b ) is a sub-exponential random variable with parameters a, b > 0 . Note that a random variable Z is called sub-exponential with parameters a, b > 0 , SE( a, b ) , if its moment generating function satisfies

10

2

10

3

10

4

Input dimension

d

0

.

50

0

.

75

1

.

00

10

2

10

3

10

4

Input dimension

d

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

1

10

2

10

3

10

4

Sample size

N

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

By [16], ∑ m i X 2 i Y i follows SE(64 mσ 4 , 8 σ 2 ) . In addition, by [16], Z ∼ SE( a, b ) satisfies

Therefore, with probability at least 1 -δ ,

(c) Let k ∈ [ m -1] be a positive integer. Let Bi( m,p ) be the Binomial distribution and Be( p ) be the Bernoulli distribution with p ∈ (0 , ( k +1) /m ) . By Hoeffding's inequality,

10

2

10

3

10

4

Input dimension

d

0

.

50

0

.

75

1

.

00

10

2

10

3

10

4

Input dimension

d

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

0

1

2

3

4

5

6

7

8

9

10

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

1

10

2

10

3

10

4

Sample size

N

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

P [ there are at least k +1 instances such that | X i | < C ]

Now,

0

.

028

0

.

056

0

.

084

0

.

112

0

.

140

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

0

.

028

0

.

056

0

.

084

0

.

112

0

.

140

Perturbation size

/epsilon1

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

20

30

40

50

60

70

80

90

100

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

Since the areas of a square with sides of length x and a quarter-circle with a radius of 2 x/ √ π are the same, and because s 2 + t 2 is always larger in the square than in the quarter-circle outside the common area, an upper bound of erf( x ) can be computed as

Thus,

Therefore,

Lemma C.2 (Hoeffding's inequality) . Let s 1 , . . . , s N ∈ R be real numbers and X 1 , . . . , X N ∈ {± 1 } be i.i.d. random variables. Suppose that E [ X n ] = 0 for every n ∈ [ N ] . Then, for 0 < δ < 1 ,

0

.

000

0

.

028

0

.

056

0

.

084

0

.

112

0

.

140

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

0

.

00

0

.

28

0

.

56

0

.

84

1

.

12

1

.

40

Perturbation size

/epsilon1

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

with probability at least 1 -δ ,

Proof. By Hoeffding's inequality, the claim is established.

Lemma C.3 (Expectation of product of derivatives of activation functions, part 1) . Denote a symmetric positive definite matrix by

where a, c > 0 and ac -b 2 > 0 . Then,

Proof. The inverse of Σ is

0

.

000

0

.

028

0

.

056

0

.

084

0

.

112

0

.

140

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

0

.

000

0

.

028

0

.

056

0

.

084

0

.

112

0

.

140

Perturbation size

/epsilon1

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

20

30

40

50

60

70

80

90

100

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

Using y := x / √ ac -b 2 , the quadratic form can be represented as

Because | ∂ x /∂ y | = ac -b 2 ,

With z := y 1 -by 2 /c ,

Now,

The integral along z can be computed as

0

.

000

0

.

028

0

.

056

0

.

084

0

.

112

0

.

140

Perturbation size

/epsilon1

0

.

50

0

.

75

1

.

00

0

.

00

0

.

28

0

.

56

0

.

84

1

.

12

1

.

40

Perturbation size

/epsilon1

(logistic)

0

.

50

0

.

75

1

.

00

Standard

Perturbation

Agreement

10

1

10

2

10

3

Hidden dimension

m

0

.

50

0

.

75

1

.

00

0

.

00

0

.

25

0

.

50

0

.

75

0

.

99

Slope

γ

0

.

50

0

.

75

1

.

00

10

0

10

1

10

2

Standard training time

T

f

0

.

50

0

.

75

1

.

00

10

20

30

40

50

60

70

80

90

100

Perturbation training time

T

g

0

.

50

0

.

75

1

.

00

Accuracy (blue and orange) / Agreement (green)

Using the above equation,

10

20

30

40

50

60

70

80

90

100

Input dimension

d

(shifted Gauss)

0

.

0

0

.

5

1

.

0

1

2

3

4

5

6

7

8

9

10

training time

T

f

(shifted Gauss)

0

.

0

0

.

5

1

.

0

Standard training accuracy

Cosine similarity

10

2

10

3

10

4

Input dimension

d

(Gauss)

0

.

0

0

.

5

1

.

0

10

0

10

1

10

2

training time

T

f

(Gauss)

0

.

0

0

.

5

1

.

0

10

0

10

1

10

2

training time

T

f

(MNIST)

0

.

0

0

.

5

1

.

0

10

0

10

1

10

2

training time

T

f

(FMNIST)

0

.

0

0

.

5

1

.

0

Standard training accuracy (blue) / Cosine similarity (orange)

Thus,

Using t := sgn( b ) z √ c/ 2 ,

From [8], with α = √ e/ (2 π ) ,

-

5

0

5

-

5

0

5

Shifted Gauss (a)

-

5

0

5

-

5

0

5

Shifted Gauss (b)

0

5

-

2

.

5

0

.

0

2

.

5

5

.

0

Gauss (a)

0

5

-

2

.

5

0

.

0

2

.

5

5

.

0

Gauss (b)

0

5

10

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

MNIST (a)

0

5

10

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

MNIST (b)

0

5

10

0

5

10

FMNIST (a)

0

5

10

0

5

10

FMNIST (b)

sgn(

f

(

z

)) = sgn(

g

(

z

))

sgn(

f

(

z

))

= sgn(

g

(

z

))

ˆ

f

(

z

) = 0

ˆ

g

(

z

) = 0

Positive samples

Negative samples

/negationslash

Thus,

Finally,

̸

Lemma C.4 (Expectation of product of derivatives of activation functions, part 2) . For any z 1 = z 2 ∈ R d ,

Proof. By the reproductive property of Gaussian distributions, ⟨ v , z 1 ⟩ + a follows N (0 , ∥ z 1 ∥ 2 /d +1) . Since any linear combination of ⟨ v , z 1 ⟩ + a and ⟨ v , z 2 ⟩ + a has a univariate Gaussian distribution, ⟨ v , z 1 ⟩ + a and ⟨ v , z 2 ⟩ + a follow a multivariate Gaussian distribution. The covariance matrix Σ can be computed as

Thus, by Lemma C.3, the claim is established.

D Main Proof

For notational simplicity, we use the following abbreviation for i ∈ [ m ] and n ∈ [ N ] :

Moreover, denote the subset of [ m ] consisting of the smallest η √ m ∈ [ m -1] elements in terms of | h f,i, 0 ( z ) | by S f ( z ) . Similarly, we define S g ( z ) based on | h g,i, 0 ( z ) | . The function κ f,i ( z ) returns one if i ∈ S f ( z ) and zero otherwise. Similarly, we define κ g,i ( z ) . For exp ( -2( η √ m +1) 2 /m ) < δ < 1 , let

D.1 Assumptions on Properties of Neural Networks and Their Justifications

For reference, we state the following assumption on the network f :

Assumption D.1 (Properties of neural network f ) . Let z ∈ R d be a real-valued vector and exp ( -2( η √ m +1) 2 /m ) < δ < 1 be a positive value.

(b) For any i ∈ [ m ] , | α i | < √ (2 /m ) ln(2 m/δ ) .

(a) For any i ∈ [ m ] and j ∈ [ d ] , | v i,j (0) | < √ (2 /d ) ln(2 dm/δ ) .

(c) For any i ∈ [ m ] , |⟨ v i (0) , z ⟩| < √ (2 /d ) ln(2 m/δ ) ∥ z ∥ .

(e) | f ( z ; 0) | < √ 2( ∥ z ∥ 2 /d +1)ln(2 /δ ) .

(d) For any i ∈ [ m ] , | h f,i, 0 ( z ) | < √ 2( ∥ z ∥ 2 /d +1)ln(2 m/δ ) .

(f) For any j ∈ [ d ] ,

(g)

(i) There are at most η √ m instances such that | h f,i, 0 ( z ) | < C thr ( z , δ ) / √ 2 .

Assumption D.1 is justified as follows:

Lemma D.2 (Justification of Assumption D.1) .

(a) With probability at least 1 -δ , Assumption D.1(a) holds.

(c) With probability at least 1 -δ , Assumption D.1(c) holds.

(b) With probability at least 1 -δ , Assumption D.1(b) holds.

(d) With probability at least 1 -δ , Assumption D.1(d) holds.

(f) With probability at least 1 -δ , Assumption D.1(f) holds.

(e) With probability at least 1 -δ , Assumption D.1(e) holds.

(g) With probability at least 1 -δ , Assumption D.1(g) holds.

(i) With probability at least 1 -δ , Assumption D.1(i) holds.

(h) With probability at least 1 -δ , Assumption D.1(h) holds.

(j) With probability at least 1 -9 δ , Assumption D.1 holds.

Proof.

(a), (b) and (h) See Lemma C.1.

(c) to (g) and (i) By the reproductive property of Gaussian random variables, ⟨ v i (0) , z ⟩ , h f,i, 0 ( z ) , f ( z ; 0) , ∑ m i α i ψ ′ f,i, 0 ( x n ) v i,j (0) , and ∑ m i α i ψ ′ f,i, 0 ( x n ) ⟨ v i (0) , z ⟩ follow the Gaussian N (0 , ∥ z ∥ 2 /d ) , N (0 , ∥ z ∥ 2 /d +1) , N (0 , ∥ z ∥ 2 /d +1) , N (0 , (1 /m ) ∑ m i ψ ′ f,i, 0 ( x n ) 2 v i,j (0) 2 ) , and N (0 , (1 /m ) ∑ m i ψ ′ f,i, 0 ( x n ) 2 ⟨ v i (0) , z ⟩ 2 ) , respectively. By Lemma C.1, the claim is established.

(j) By Bonferroni's inequality, the claim is established.

Similarly, we consider the following assumption on the network g and its justification:

Assumption D.3 (Properties of neural network g ) . Let z ∈ R d be a real-valued vector and exp ( -2( η √ m +1) 2 /m ) < δ < 1 be a positive value.

(a) For any i ∈ [ m ] and j ∈ [ d ] , | w i,j (0) | < √ (2 /d ) ln(2 dm/δ ) . (b) For any i ∈ [ m ] , | β i | < √ (2 /m ) ln(2 m/δ ) . (c) For any i ∈ [ m ] , |⟨ w i (0) , z ⟩| < √ (2 /d ) ln(2 m/δ ) ∥ z ∥ . (d) For any i ∈ [ m ] , | h g,i, 0 ( z ) | < √ 2( ∥ z ∥ 2 /d +1)ln(2 m/δ ) . (e) | g ( z ; 0) | < √ 2( ∥ z ∥ 2 /d +1)ln(2 /δ ) . (f) For any j ∈ [ d ] , ∣ ∣ ∣ ∣ ∣

(i) There are at most η m instances such that h g,i, 0 ( z ) < C thr ( z , δ ) / 2

Lemma D.4 (Justification of Assumption D.3) . With probability at least 1 -9 δ , Assumption D.3 holds.

D.2 Wide Width Assumptions

Then, we consider the condition of network width for lazy training.

Assumption D.5 (Wide width for neural network f ) . Let z ∈ R d be a real-valued vector and exp ( -2( η √ m +1) 2 /m ) < δ < 1 be a positive value. Network width m satisfies the following inequalities:

Note that only Ineq. (A90) is required to satisfy lazy training. We impose Ineq. (A91) to simplify some rearrangements of equations. This assumption restricts the transitions of the derivatives of hidden outputs during training.

Lemma D.6 (Lazy training in network f ) . If Assumptions D.1 and D.5 hold, then ψ ′ f,i,t ( z ) = ψ ′ f,i, 0 ( z ) for any i ∈ [ m ] \ S f ( z ) and 0 ≤ t ≤ T f .

Proof. By Assumption D.1, the time evolution of h f,i,t ( z ) from t = 0 to t = T f can be computed as

By Assumption D.1, if the right term of Ineq. (A95) is smaller than C thr ( z , δ ) / √ 2 , then the largest m -η √ m instances in terms of | h f,i, 0 ( z ) | satisfy sgn( h f,i,t ( z )) = sgn( h f,i, 0 ( z )) for 0 ≤ t ≤ T f . This condition can be represented as Ineq. (A90).

The same discussion can be applied to the network g .

Assumption D.7 (Wide width for neural network g ) . Let z ∈ R d be a real-valued vector and exp ( -2( η √ m +1) 2 /m ) < δ < 1 be a positive value. Network width m satisfies the following inequalities:

Lemma D.8 (Lazy training in network g ) . If Assumptions D.3 and D.7 hold, then ψ ′ g,i,t ( z ) = ψ ′ g,i, 0 ( z ) for any i ∈ [ m ] \ S g ( z ) and 0 ≤ t ≤ T g .

We can integrate Assumptions D.1 and D.3 into Assumption 3.2.

D.3 Main Part

Lemma D.9 (Representation of network f ) . If Assumptions D.1 and D.5 hold, then the network output at the training time T f can be represented as

Proof. First, see Lemma D.6. The time evolution of v i ( t ) from t = 0 to t = T f can be computed as

Similarly, the time evolution of a i ( t ) from t = 0 to t = T f can be computed as

Thus,

The original network at the training time T f can be computed as

The first term of Eq. (A110) can be rearranged as

The second term of Eq. (A110) can be rearranged as

The first term of Eq. (A114) can be rearranged as

The first term of Eq. (A116) can be rearranged as

The second term of Eq. (A114) can be rearranged as

The sum of the second term of Eq. (A116) and Eq. (A121) can be rearranged as

Lemma D.10 (Upper bounds of terms in Lemma D.9) . Assume Assumptions D.1 and D.5. (a)

Proof.

(a) By Assumption D.1,

(b) By Assumption D.1,

By Assumption D.5,

(c) By Assumption D.1,

By Assumption D.5,

Lemma D.11 (Network prediction of f ) . Assume Assumptions D.1 and D.5. If

then

then

Lemma D.12 (Adversarial perturbation) . If Assumptions D.1 and D.5 hold, then the adversarial perturbation defined as Eq. (1) can be represented as

Proof. The main term of the adversarial perturbation can be computed as

The leading term can be rearranged as

The first term of Eq. (A144) can be rearranged as

The second term of Eq. (A144) can be rearranged as

The first term of Eq. (A146) can be rearranged as

The second term of Eq. (A146) can be rearranged as

The sum of the second term of Eq. (A148) and Eq. (A150) can be rearranged as

Proof.

(a) The given left term can be rearranged as

By Assumption D.1,

(b) The given left term can be rearranged as

By Assumption D.1,

(c) By Assumption D.1,

By Assumption D.5,

(d) The given left term can be rearranged as

By Assumption D.1,

Thus,

By Assumption D.5,

(e) As ∥ s 1 + s 2 + s 3 + s 4 + s 5 ∥ 2 ≤ 25 max( ∥ s 1 ∥ 2 , ∥ s 2 ∥ 2 , ∥ s 3 ∥ 2 , ∥ s 4 ∥ 2 , ∥ s 5 ∥ 2 ) for any s 1 , s 2 , s 3 , s 4 , s 5 ∈ R d ,

(f) Similarly to (e),

Lemma D.14 (Upper bounds of inner products with adversarial perturbation) . Assume Assumptions D.1 and D.5.

(a)

Proof.

(a) By Assumption D.1,

(b) By Assumption D.1,

(c) and (d) Similarly to Lemma D.10.

Proof. Similarly to Lemma D.9,

The numerator can be also rearranged as

Lemma D.16 (Network prediction of g ) . Suppose that Assumptions D.1, D.3, D.5 and D.7.

(a) In Scenario (a), if

then

(b) In Scenario (b), if

then

Proof. We prove (a). Similarly, (b) can be established. By Lemmas D.10 and D.13 to D.15, if

then

Theorem D.17 (Perturbation learning, Scenario (a), general case) . Consider Scenario (a) in Setting 3.1. Let δ = Θ(1) be a small positive number and

Under Assumption 3.2, for any z ∈ R d , if

> ˜ O ( √ d ∫ T f 0 ¯ ℓ ′ f,k,t d t ϵ ( 1 + ∣ ∣ ∣ ∣ ∣ 1 N N ∑ n y adv n Φ( r n , z ) ∫ T g 0 ℓ ′ g,n,t d t ∣ ∣ ∣ ∣ ∣ ) + ∫ T g 0 ¯ ℓ ′ g,n,t d t ) , (A197) sgn( ˆ f gen ( z )) = sgn(ˆ g gen a ( z )) , (A198) then, with probability at least 1 -δ , sgn( f ( z ; T f )) = sgn( g ( z ; T g )) holds.

Proof. By Bonferroni's inequality and Assumptions D.5 and D.7 and Lemmas D.2, D.4, D.11 and D.16, the claim is established.

Theorem D.18 (Perturbation learning, Scenario (b), general case) . Consider Scenario (b) in Setting 3.1. Let δ = Θ(1) be a small positive number and

Under Assumption 3.2, for any z ∈ R d , if

then, with probability at least 1 -δ , sgn( f ( z ; T f )) = sgn( g ( z ; T g )) holds.

Proof. By Bonferroni's inequality and Assumptions D.5 and D.7 and Lemmas D.2, D.4, D.11 and D.16, the claim is established.

Theorem 3.3 (Direction of adversarial perturbation) . Let δ = Θ(1) be a small positive number. Under Assumption 3.2, for any n ∈ [ N ] , with probability at least 1 -δ , the adversarial perturbation r n is parallel to the weighted sum of training samples as follows:

where ξ n satisfies ∥ ξ n ∥ = ˜ O (1) . In particular, for ℓ ( s ) = s ,

Proof. By Bonferroni's inequality and Assumptions D.5 and D.7 and Lemmas D.2, D.4, D.12 and D.13, the claim is established.

Theorem 3.4 (Perturbation learning, Scenario (a), special case of Theorem D.17) . Consider Scenario (a) in Setting 3.1. Assume ℓ ( s ) = s and y adv n ∼ U ( {± 1 } ) for every n ∈ [ N ] . Let δ = Θ(1) be a small positive number and

Under Assumption 3.2, for any z ∈ R d , if

(Agreement condition) f ( z )) = sgn(ˆ g a ( z )) , (11) then, with probability at least 1 -δ , sgn( f ( z ; T f )) = sgn( g ( z ; T g )) holds.

Proof. By Bonferroni's inequality and Lemma C.2 and Theorem D.17, the claim is established.

Theorem 3.5 (Perturbation learning, Scenario (b), special case of Theorem D.18) . Consider Scenario (b) in Setting 3.1. Assume ℓ ( s ) = s and y adv n ∼ U ( {± 1 } ) for every n ∈ [ N ] . Let δ = Θ(1) be a small positive number and

Under Assumption 3.2, for any z ∈ R d , if the functional margin condition 1 (Ineq. (9) ),

(Agreement condition)

sgn( ˆ

f

(

z

)) = sgn(ˆ

g

b

(

z

))

,

then, with probability at least 1 -δ , sgn( f ( z ; T f )) = sgn( g ( z ; T g )) holds.

Proof. By Bonferroni's inequality and Lemma C.2 and Theorem D.18, the claim is established.

Lemma D.19 (Sufficient condition of agreement condition) . If

|⟨ ⟩| (A204) then sgn( ˆ f gen ( z )) = sgn(ˆ g gen a ( z )) = sgn(ˆ g gen b ( z )) holds.

(16)

In addition,

Thus, if Ineq. (A207) holds, then

Similarly, for any k ∈ [ N ] , if

then

When Ineq. (A209) holds for every k ∈ [ N ] ,

By integrating Ineqs. (A207) and (A209), the claim is established.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have accurately described the contributions and limitations in Abstract and Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are described in Section 3.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumption is described in Assumption 3.2 and a brief proof is provided in Section 3.3. The complete proof can be found in Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setup is described in detail in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets we used are either openly accessible or can be artificially generated. The code is provided as supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setup is described in detail in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars were not measured due to computational costs. However, we provide extensive experimental results to support our theoretical findings in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our experiments were conducted on an NVIDIA A100.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper strictly adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This does not apply as it is a theoretical study.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our theoretical research does not involve such releases.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have accurately cited credits.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not provide such assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not conduct such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We did not conduct experiments that require this.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors theoretically investigate the perturbation learning phenomenon from a future hypothesis perspective. Perturbation learning means that a classifier can be learned from adversarial examples with incorrect labels (the labels used to generate adversarial examples, but seems to be incorrect in human eyes). Their theory includes two parts. First, they show that the adversarial perturbations are the linear combination of training samples, thus containing the information of the clean training dataset. This theorem provides an intuition for the feature hypothesis. Then, they show that, under some conditions, the predictions of a classifier trained on adversarial perturbations (or adversarial examples) are consistent with those of a classifier trained on a clean dataset.\n\nSome experiments on a synthetic Gaussian dataset are provided to show the effect of the input dimension and hidden dimension, validating their theorem.\n\nSTRENGTHS:\n* Understanding perturbation learning and the feature hypothesis is quite important in the domain of adversarial learning. This paper provides a deeper understanding of this problem.\n* In this paper, the authors provide a more general theory about perturbation learning compared to prior work. The theory is based on fewer constraints in training data distribution, training time, etc.\n\nWEAKNESSES:\nMy main concern is whether the kernel regime is a suitable and extensible tool to study the feature hypothesis in adversarial training and explain other interesting phenomena such as the transferability of adversarial examples and the trade-off between robustness and accuracy. \n\nAccording to the feature hypothesis of adversarial training, the data contains a set of features that could used for prediction. Humans and all kinds of neural networks use different feature subsets to classify the image (the subsets have overlaps). The feature hypothesis may provide a **unified** explanation for several open problems in adversarial training. However, it seems that the theory in this paper only focuses on perturbation learning and is difficult to extend to explain other phenomena. In my opinion, the framework should have a deeper exploration of the feature subsets that are used by the trained models to make predictions.\n\nQUESTIONS:\nExperimental Questions:\n* What is the effect of the sampling of adversarial labels on the accuracy of the model $g$? If we set the adversarial labels of all negative data to be positive and vice versa, will the accuracy of $g$ be very low?\n* What is the effect of the structures of $f$ and $g$? When these two networks have more divergent structures, I think the accuracy of $g$ must be lower. Is it correct in experiments? If it is, this is similar to the transferability of adversarial examples, i.e., the adversarial examples are easier to transfer to similar networks.\n\nTheoretical Questions:\n* What do the functions $\\hat{f}$ and $\\hat{g}$ mean? What are the relationships between $f$ and $\\hat{f}$, $g$ and $\\hat{g}$?\n\nLIMITATIONS:\nThe authors state the limitation on the assumption of the width of the network in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

comments: SUMMARY:
This work aims to provide an alternative theoretical analysis to justify feature hypothesis and perturbation learning. The analysis is based on approximation theory in the kernel regime (i.e., infinite width). They show that the adversarial perturbation contains sufficient data information, which can be retrieved by the perturbation learning when y labels are uniformly sampled.

STRENGTHS:
The paper is theoretically sound and relaxed certain conditions required by prior works.

WEAKNESSES:
What is the contribution beyond technical novelty? Are any new insights obtained via the new analysis compared to existing results?
The conditions in the main theorems lack interpretability. I can understand the technical reason behind them, but how reasonable are these conditions, especially the agreement condition? The author argues that the agreement condition depends on the consistency of the correlation between z and y_nx_n. This statement is not rigorous, since z is a function argument, but not a r.v. and there is no correlation involved. I suppose a more meaningful question will be, if z follows the data distribution of x (e.g., a mixture of two Gaussian), what is the chance these conditions hold?

QUESTIONS:
See weakness.

LIMITATIONS:
Yes. The limitation is clearly statement in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPerturbation learning, where classifiers are trained on adversarial examples with their associated incorrect labels, results in non-trivial generalization. This work theoretically tackles the perplexing former phenomenon for wide two-layer networks in the kernel regime. The authors first prove that adversarial perturbations are parallel to a meaningful feature vector (that can contain whole dataset information under some additional assumptions) up to an error term. Second, the authors provide some conditions on when the predictions with perturbation learning match those of standard learning.\n\nSTRENGTHS:\n1. The problem is a very interesting one and lacks more analysis, so this paper addresses a significant problem in an original way.\n\n2. The setting and assumptions are very well described and clear.\n\nWEAKNESSES:\n1. I am overall confused about the choice of kernel regime to explain perturbation learning. As the authors acknowledge, there is no feature learning for the choice of width in this paper (the output of hidden units remains the same). It's not clear how such a framework can explain perturbation learning and, in particular, the \"feature hypothesis,\" which authors claim that they do. Since there is no feature learning in this regime, there should not be any \"feature hypothesis.\" This is not to say the analysis is uninteresting; it applies to perturbation learning with kernels. But this requires a completely different contextualization than the authors provide, one that focuses on the adversarial robustness of kernels rather than neural networks. And for the reasons above, I find the comparison with prior work a bit misleading as they tackle perturbation learning for different scenarios.\n\n2. It is difficult to judge the validity of the assumptions in the paper.\n\n* Assumption 3.2. is very convoluted, and I don't see how one can justify this assumption in finite width settings. I equate this assumption to an infinite-width assumption, which is perfectly reasonable, but as I discussed above, I believe it changes the object of study.\n\n* The assumptions of Theorem 3.3. and Theorem 3.4. are discussed, and some intuitions are provided. But it is not discussed why they would hold for a small $\\delta$.\n\n3. The $\\delta$ dependencies in Theorems 3.3, 3.4, and 3.5 are confusing. Let's focus only on Theorem 3.3. When we take $\\delta \\to 0$ while everything else remains fixed, the statement is trivial. So, the interesting part of the statement is when $T_f$ or $d$ scales with $\\delta$. I believe it is more reasonable to consider $T_f \\to \\infty$. So, this verifies that Assumption 3.2. is an infinite width assumption.\n\nQUESTIONS:\n1. Could you discuss again the choice of kernel regime vs. feature learning for your analysis after my comments in the Weaknesses section?\n2. Could you provide references or evidence towards why $\\epsilon$ should scale with $1/\\sqrt{d}$?\n\nLIMITATIONS:\nI think the authors overclaim by saying, \"We provided a theoretical justification for perturbation learning and the feature hypothesis\" (L293). Their analysis is limited to the kernel regime, which is acknowledged, but its implications for the feature hypothesis are not well explained. In addition, scaling with $\\delta$ in main theorems is not discussed, and as I pointed out, it points to other limitations towards infinite width.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}