Title: Provable Benefit of Cutout and CutMix for Feature Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Cutout

CutMix

feature learning

theory
Abstract: Patch-level data augmentation techniques such as Cutout and CutMix have demonstrated significant efficacy in enhancing the performance of vision tasks. However, a comprehensive theoretical understanding of these methods remains elusive. In this paper, we study two-layer neural networks trained using three distinct methods: vanilla training without augmentation, Cutout training, and CutMix training. Our analysis focuses on a feature-noise data model, which consists of several label-dependent features of varying rarity and label-independent noises of differing strengths. Our theorems demonstrate that Cutout training can learn low-frequency features that vanilla training cannot, while CutMix training can learn even rarer features that Cutout cannot capture. From this, we establish that CutMix yields the highest test accuracy among the three. Our novel analysis reveals that CutMix training makes the network learn all features and noise vectors "evenly" regardless of the rarity and strength, which provides an interesting insight into understanding patch-level augmentation.

Provable Benefit of Cutout and CutMix for Feature Learning

Abstract

Patch-level data augmentation techniques such as Cutout and CutMix have demonstrated significant efficacy in enhancing the performance of vision tasks. However, a comprehensive theoretical understanding of these methods remains elusive. In this paper, we study two-layer neural networks trained using three distinct methods: vanilla training without augmentation, Cutout training, and CutMix training. Our analysis focuses on a feature-noise data model, which consists of several label-dependent features of varying rarity and label-independent noises of differing strengths. Our theorems demonstrate that Cutout training can learn low-frequency features that vanilla training cannot, while CutMix training can learn even rarer features that Cutout cannot capture. From this, we establish that CutMix yields the highest test accuracy among the three. Our novel analysis reveals that CutMix training makes the network learn all features and noise vectors 'evenly' regardless of the rarity and strength, which provides an interesting insight into understanding patch-level augmentation.

1 Introduction

Data augmentation is a crucial technique in deep learning, particularly in the image domain. It involves creating additional training examples by applying various transformations to the original data, thereby enhancing the generalization performance and robustness of deep learning models. Traditional data augmentation techniques typically focus on geometric transformations such as random rotations, horizontal and vertical flips, and cropping (Krizhevsky et al., 2012), or color-based adjustments such as color jittering (Simonyan and Zisserman, 2014).

In recent years, several new data augmentation techniques have appeared. Among them, patch-level data augmentation techniques like Cutout (DeVries and Taylor, 2017) and CutMix (Yun et al., 2019) have received considerable attention for their effectiveness in improving generalization. Cutout is a straightforward method where random rectangular regions of an image are removed during training. In comparison, CutMix adopts a more complex strategy by cutting and pasting sections from different images and using mixed labels, encouraging the model to learn from blended contexts. The success of Cutout and CutMix has triggered the development of numerous variants including Random Erasing (Zhong et al., 2020), GridMask (Chen et al., 2020a), CutBlur (Yoo et al., 2020), Puzzle Mix (Kim et al., 2020), and Co-Mixup (Kim et al., 2021). However, despite the empirical success of these patch-level data augmentation techniques in various image-related tasks, a lack of comprehensive theoretical understanding persists: why and how do they work?

In this paper, we aim to address this gap by offering a theoretical analysis of two important patch-level data augmentation techniques: Cutout and CutMix. Our theoretical framework draws inspiration from a study by Shen et al. (2022), which explores a data model comprising multiple label-dependent feature vectors and label-independent noises of varying frequencies and intensities. The key idea of this work is that learning features with low frequency can be challenging due to strong noises (i.e., low signal-to-noise ratio). We focus on how Cutout and CutMix can aid in learning such rare features.

1.1 Our Contributions

In this paper, we consider a patch-wise data model consisting of features and noises, and use two-layer convolutional neural networks as learner networks. We focus on three different training methods: vanilla training without any augmentation, Cutout training, and CutMix training. We refer to these training methods in our problem setting as ERM , Cutout , and CutMix . We investigate how these methods affect the network's ability to learn features. We summarize our contributions below:

We analyze ERM , Cutout , and CutMix , revealing that Cutout outperforms ERM since it enables the learning of rarer features compared to ERM (Theorem 3.1 and Theorem 3.2). Furthermore, CutMix demonstrates almost perfect performance (Theorem 3.3) by learning all features.

Our main intuition behind the negative result for ERM is that ERM learns to classify training samples by memorizing noise vectors instead of learning meaningful features if the features do not appear frequently enough. Hence, ERM suffers low test accuracy because it cannot learn rare features. However, Cutout alleviates this challenge by removing some of the strong noise patches, allowing it to learn rare features to some extent.

We prove the near-perfect performance of CutMix based on a novel technique that views the non-convex loss as a composition of a convex function and reparameterization. This enables us to characterize the global minimum of the loss and show that CutMix forces the model to activate almost uniformly across every patch of inputs, allowing it to learn all features.

1.2 Related Works

Feature Learning Theory. Our work aligns with a recent line of studies investigating how training methods and neural network architectures influence feature learning. These studies focus on a specific data distribution composed of two components: label-dependent features and label-independent noise. The key contribution of this body of work is the exploration of which training methods or neural networks are most effective at learning meaningful features and achieving good generalization performance. Allen-Zhu and Li (2020) demonstrate that an ensemble model can achieve near-perfect performance by learning diverse features, while a single model tends to learn only certain parts of the feature space, leading to lower test accuracy. In other works, Cao et al. (2022); Kou et al. (2023a) explore the phenomenon of benign overfitting when training a two-layer convolutional neural network. The authors identify the specific conditions under which benign overfitting occurs, providing valuable insights into how these networks behave during training. Several other studies seek to understand various aspects of deep learning through the lens of feature learning (Zou et al., 2021; Jelassi and Li, 2022; Chen et al., 2022, 2023; Li and Li, 2023; Huang et al., 2023a,b).

Theoretical Analysis of Data Augmentation. Several works aim to analyze traditional data augmentation from different perspectives, including kernel theory (Dao et al., 2019), margin-based approach (Rajput et al., 2019), regularization effects (Wu et al., 2020), group invariance (Chen et al., 2020b), and impact on optimization (Hanin and Sun, 2021). Moreover, many papers have explored various aspects of a recent technique called Mixup (Zhang et al., 2017). For example, studies have explored its regularization effects (Carratino et al., 2020; Zhang et al., 2020), its role in improving calibration (Zhang et al., 2022), its ability to find optimal decision boundaries (Oh and Yun, 2023) and its potential negative effects (Chidambaram et al., 2021; Chidambaram and Ge, 2024). Some works investigate the broader framework of Mixup, including CutMix, which aligns with the scope of our work. Park et al. (2022) study the regularization effect of mixed-sample data augmentation within a unified framework that contains both Mixup and CutMix. In Oh and Yun (2023), the authors analyze masking-based Mixup, which is a class of Mixup variants that also includes CutMix. In their context, they show that masking-based Mixup can deviate from the Bayes optimal classifier but require less training sample complexity. However, neither work provides a rigorous explanation for why CutMix has been successful. The studies most closely related to our work include Shen et al. (2022); Chidambaram et al. (2023); Zou et al. (2023). Shen et al. (2022) regard traditional data augmentation as a form of feature manipulation and investigate its advantages from a feature learning perspective. Both Chidambaram et al. (2023) and Zou et al. (2023) analyze Mixup within a feature learning framework. However, patch-level data augmentation such as Cutout and CutMix, which are the focus of our work, have not yet been explored within this context.

2 Problem Setting

In this section, we introduce the data distribution and neural network architecture, and formally describe the three training methods considered in this paper.

2.1 Data Distribution

We consider a binary classification problem on structured data, consisting of patches of labeldependent vectors (referred to as features ) and label-independent vectors (referred to as noise ).

Definition 2.1 (Feature Noise Patch Data) . We define a data distribution D on R d × P ×{-1 , 1 } such that ( X , y ) ∼ D where X = ( x (1) , . . . , x ( P ) ) ∈ R d × P and y ∈ {± 1 } is constructed as follows.

Choose the label y ∈ {± 1 } uniformly at random.

Let { v s,k } s ∈{± 1 } ,k ∈ [ K ] ⊂ R d be a set of orthonormal feature vectors . Choose the feature vector v ∈ R d for data point X as v = v y,k with probability ρ k from { v y,k } k ∈ [ K ] ⊂ R d , where ρ 1 + · · · + ρ K = 1 and ρ 1 ≥ ·· · ≥ ρ K . In our setting, there are three types of features with significantly different frequencies: common features , rare features , and extremely rare features , ordered from most to least frequent. The indices of these features partition [ K ] into ( K C , K R , K E ) .

We construct P patches of X as follows.

Feature Patch : Choose p ∗ uniformly from [ P ] and we set x ( p ∗ ) = v .

Dominant Noise Patch : Choose ˜ p uniformly from [ P ] \{ p ∗ } . We construct x (˜ p ) = α u + ξ (˜ p ) where α u is feature noise drawn uniformly from { α v 1 , 1 , α v -1 , 1 } with 0 < α < 1 and ξ (˜ p ) is Gaussian dominant noise drawn from N ( 0 , σ 2 d Λ ) .

Background Noise Patch : The remaining patches p ∈ [ P ] \ { p ∗ , ˜ p } consist of Gaussian background noise , i.e., we set x ( p ) = ξ ( p ) where ξ ( p ) ∼ N ( 0 , σ 2 b Λ ) .

Here, the noise covariance matrix is defined as Λ := I -∑ s,k v s,k v ⊤ s,k which ensures that Gaussian noises are orthogonal to all features. We assume that the dominant noise is stronger than the background noise, i.e., σ b < σ d .

Our data distribution captures characteristics of image data, where the input consists of several patches. Some patches contain information relevant to the image labels, such as cat faces for the label 'cat,' while other patches contain information irrelevant to the labels, such as the background. Intuitively, there are two ways to fit the given data: learning features or memorizing noise. If a model fits the data by learning features, it can correctly classify test data having the same features. However, if a model fits the data by memorizing noise, it cannot generalize to unseen data because noise patches are not relevant to labels. Thus, learning more features is crucial for achieving better generalization.

In real-world scenarios, different features may appear with varying frequencies. For instance, the occurrences of cat's faces and cat's tails in a dataset might differ significantly, although both are relevant to the 'cat' label. Our data distribution reflects these characteristics by considering features with varying frequencies. To emphasize the distinctions between the three training methods we analyze, we categorize features into three groups: common, rare, and extremely rare. We refer to data points containing these features as common data , rare data , and extremely rare data , respectively. We emphasize that these terminologies are chosen merely to distinguish the three different levels of rarity, and even 'extremely rare' features appear in a nontrivial fraction of the training data with high probability (see our assumptions in Section 2.4).

Comparison to Previous Work. Our data distribution is similar to those considered in Shen et al. (2022) and Zou et al. (2023), which investigate the benefits of standard data augmentation methods and Mixup by comparing them to vanilla training without any augmentation. These results consider two types of features-common and rare-with different levels of rarity, along with two types of noise: feature noise and Gaussian noise. However, we consider three types of features: common, rare, and extremely rare, and three types of noise: feature noise, dominant noise, and background noise. This distinction allows us to compare three distinct methods and demonstrate the differences between them, whereas Shen et al. (2022) and Zou et al. (2023) compared only two methods.

2.2 Neural Network Architecture

For the prediction model, we focus on the following two-layer convolutional neural network where the weights in the second layer are fixed at 1 and -1 , with only the first layer being trainable. Several works including Shen et al. (2022) and Zou et al. (2023) also focus on similar two-layer convolutional neural networks.

Definition 2.2 (2-Layer CNN) . We define 2-layer CNN f W : R d × P → R parameterized by W = { w 1 , w -1 } ∈ R d × 2 . For each input X = ( x (1) , . . . , x ( P ) ) ∈ R d × P , we define

where ϕ ( · ) is a smoothed version of leaky ReLU activation, defined as follows.

where 0 < β ≤ 1 and r > 0 .

Previous works on the theory of feature learning often consider neural networks with (smoothed) ReLU or polynomial activation functions. However, we adopt a smoothed leaky ReLU activation, which always has a positive slope, to exclude the possibility of neurons 'dying' during the complex optimization trajectory. Using smoothed leaky ReLU to analyze the learning dynamics of neural networks is not entirely new; there is a body of work that studies phenomena such as benign overfitting (Frei et al., 2022a) and implicit bias (Frei et al., 2022b; Kou et al., 2023b) by analyzing neural networks with (smoothed) leaky ReLU activation.

A key difference between ReLU and leaky ReLU lies in the possibility of ReLU neurons 'dying' in the negative region, where some negatively initialized neurons remain unchanged throughout training. As a result, using ReLU activation requires multiple neurons to ensure the survival of neurons at initialization, which becomes increasingly probable as the number of neurons increases. In contrast, the derivative of leaky ReLU is always positive, ensuring that a single neuron is often sufficient. Therefore, for mathematical simplicity, we consider the case where the network has a single neuron for each positive and negative output. We believe that our analysis can be extended to the multi-neuron case as we validate numerically in Appendix A.2.

2.3 Training Methods

Using a training set sampled from the distribution D , we would like to train our network f W to learn to correctly classify unseen data points from D . We consider three learning methods: vanilla training without any augmentation, Cutout, and CutMix. We first introduce necessary notation for our data and parameters, and then formalize training methods within our framework.

Training Data. We consider a training set Z = { ( X i , y i ) } i ∈ [ n ] comprising n data points, each independently drawn from D . For each i ∈ [ n ] , we denote X i = ( x (1) i , . . . , x ( P ) i ) .

Initialization. We initialize the model parameters in our neural network using random initialization. Specifically, we initialize the model parameter W (0) = { w (0) 1 , w (0) -1 } , where w (0) 1 , w (0) -1 i.i.d. ∼ N ( 0 , σ 2 0 I d ) . Let us denote updated model parameters at iteration t as W ( t ) = { w ( t ) 1 , w ( t ) -1 } .

2.3.1 Vanilla Training

The vanilla approach to training a model f W is solving the empirical risk minimization problem using gradient descent. We refer to this method as ERM . Then, ERM updates parameters W ( t ) of a model using the following rule.

where η is a learning rate and L ERM ( · ) is the ERM training loss defined as

where ℓ ( · ) is the logistic loss ℓ ( z ) = log(1 + e -z ) .

2.3.2 Cutout Training.

Cutout (DeVries and Taylor, 2017) is a data augmentation technique that randomly cuts out rectangular regions of image inputs. In our patch-wise data, we regard Cutout training as using inputs with masked patches from the original data. For each subset C of [ P ] and i ∈ [ n ] , we define augmented data X i, C ∈ R d × P as a data point generated by cutting the patches with indices in C out of X i . We can represent X i, C as

Note that the output of the model f W ( · ) on this augmented data point X i, C is

Then, the objective function for Cutout training can be defined as

where D C is a uniform distribution on the collection of subsets of [ P ] with cardinality C , where C is a hyperparameter satisfying 1 ≤ C < P 2 . 1 We refer to the process of training our model using gradient descent on Cutout loss L Cutout ( W ) as Cutout , and its update rule is

where η is a learning rate.

2.3.3 CutMix Training.

CutMix (Yun et al., 2019) involves not only cutting parts of images, but also pasting them into different images as well as assigning them mixed labels. For each subset S of [ P ] and i, j ∈ [ n ] , we define the augmented data point X i,j,S ∈ R d × P as the data obtained by cutting patches with indices in S from data X i and pasting them into X j at the same indices S . We can write X i,j, S as

The one-hot encoding of the labels y i and y j are also mixed with proportions |S| P and 1 -|S| P , respectively. This mixed label results in the loss of the form

From this, the CutMix training loss L CutMix ( W ) can be defined as

where D S is a probability distribution on the set of subsets of [ P ] which samples S ∼ D S as follows. 2

Choose the cardinality s of S uniformly at random from { 0 , 1 , . . . , P } , and

Choose S uniformly at random from the collection of subsets of [ P ] with cardinality s .

We refer to the process of training our network using gradient descent on CutMix loss L CutMix ( W ) as CutMix , and its update rule is

where η is a learning rate.

2.4 Assumptions on the Choice of Problem Parameters

To control the quantities that appear in the analysis of training dynamics, we make assumptions on several quantities in our problem setting. For simplicity, we use choices of problem parameters as a function of the dimension of patches d and consider sufficiently large d .

We use the standard asymptotic notation O ( · ) , Ω( · ) , Θ( · ) , o ( · ) , ω ( · ) to express the dependency on d . We also use ˜ O ( · ) , ˜ Ω( · ) , ˜ Θ( · ) to hide logarithmic factors of d . Additionally, poly( d ) (or polylog( d ) ) represents quantities that increase faster than d c 1 (or (log d ) c 1 ) and slower than d c 2 (or (log d ) c 2 ) for some constant 0 < c 1 < c 2 . Similarly, o (1 / poly( d )) (or o (1 / polylog( d )) ) denotes some quantities that decrease faster than 1 /d c (or 1 / (log d ) c ) for any constant c . Finally, we use f ( d ) = o ( g ( d ) / polylog( d )) when f ( d ) /g ( d ) = o (1 / polylog( d )) for some function f and g of d .

Assumptions. We assume that P = Θ(1) and P ≥ 8 for simplicity. Additionally, we consider a high-dimensional regime where the number of data points is much smaller than the dimension d , which is expressed as n = o ( αβσ -1 d σ b d 1 2 / polylog( d ) ) . We also assume that ρ k n = ω ( n 1 2 log d ) for all k ∈ [ K ] , which ensures the sufficiency of data points with each feature.

In addition, as we will describe in Section 4, the relative scales between the frequencies of features and the strengths of noises play crucial roles in our analysis, as they serve as a proxy for the 'learning speed' in the initial phase. For common features k ∈ K C , we assume ρ k = Θ(1) and the learning speed of common features is much faster than that of dominant noise, which translates into the assumption σ 2 d d = o ( βn ) . For rare features k ∈ K R , we assume ρ k = Θ( ρ R ) for some ρ R , and we consider the case where the learning speed of rare features is much slower than that of dominant noise but faster than background noise, which is expressed as ρ R n = o ( α 2 σ 2 d d/ polylog( d ) ) and σ 2 b d = o ( βρ R n ) . Finally, for extremely rare features k ∈ K E , we say ρ k = Θ( ρ E ) for some ρ E and their learning is even slower than that of background noises, which can be expressed as ρ E n = o ( α 2 σ 2 b d/ polylog( d ) ) .

Lastly, we assume the strength of feature noise satisfies α = o ( n -1 βσ 2 d d/ polylog( d ) ) , and r, σ 0 , η > 0 are sufficiently small so that σ 0 , r = o ( α/ polylog( d )) , η = o ( rσ -2 d d -1 / polylog( d ) ) .

We list our assumptions in Assumption B.1 and there are many choices of parameters satisfying the set of assumptions, including:

3 Main Results

In this section, we provide a characterization of the high probability guarantees for the behavior of models trained using three distinct methods we have introduced. We denote by T ∗ the maximum admissible training iterates and we assume T ∗ = poly( d ) η with a sufficiently large polynomial in d . In all of our theorem statements, the randomness is over the sampling of training data and the initialization of models and all results hold under the condition that d is sufficiently large.

The following theorem characterizes training accuracy and test accuracy achieved by ERM .

Theorem 3.1. Let W ( t ) be iterates of ERM . Then with probability at least 1 -o ( 1 poly( d ) ) , there exists T ERM such that any T ∈ [ T ERM , T ∗ ] satisfies the following:

(Perfectly fits training set): For all i ∈ [ n ] , y i f W ( T ) ( X i ) > 0 .

(Random on (extremely) rare data): P ( X ,y ) ∼D [ yf W ( T ) ( X ) > 0]=1 -1 2 ∑ k ∈K R ∪K E ρ k ± o ( 1 poly( d ) ) .

The proof is provided in Appendix C.2. Theorem 3.1 demonstrates that ERM achieves perfect training accuracy; however, it performs almost like random guessing on unseen data points with rare and extremely rare features. This is because ERM can only learn common features and overfit rare or extremely rare data in the training set by memorizing noises to achieve perfect training accuracy.

In comparison, we show that Cutout can perfectly fit both augmented training data and original training data, and it can also learn rare features that ERM cannot. However, Cutout still makes random guesses on test data with extremely rare features. We state these in the following theorem with the proof provided in Appendix D.2:

Theorem 3.2. Let W ( t ) be iterates of Cutout training. Then with probability at least 1 -o ( 1 poly( d ) ) , there exists T Cutout such that any T ∈ [ T Cutout , T ∗ ] satisfies the following:

(Perfectly fits augmented data): For all i ∈ [ n ] and C ⊂ [ P ] with |C| = C , y i f W ( T ) ( X i, C ) > 0 .

(Perfectly fits original training data): For all i ∈ [ n ] , y i f W ( T ) ( X i ) > 0 .

(Random on extremely rare data): P ( X ,y ) ∼D [ yf W ( T ) ( X ) > 0] = 1 -1 2 ∑ k ∈K E ρ k ± o ( 1 poly( d ) ) .

In the case of CutMix , it is challenging to discuss training accuracy directly because the augmented data have soft labels generated by mixing pairs of labels. Instead, we prove that CutMix achieves a sufficiently small gradient of the loss, and the training accuracy on the original training data is perfect. We also demonstrate that CutMix achieves almost perfect test accuracy, as it learns all types of features regardless of rarity.

Theorem 3.3. Let W ( t ) be iterates of CutMix training. Then with probability at least 1 -o ( 1 poly( d ) ) , there exists some T CutMix ∈ [0 , T ∗ ] that satisfies the following:

(Finds a near stationary point): ∥ ∥ ∇ W L CutMix ( W ( T CutMix ) )∥ ∥ = 1 poly( d ) .

(Perfectly fits original training data): For all i ∈ [ n ] , y i f W ( T CutMix ) ( X i ) > 0 .

(Almost perfectly classifies test data): P ( X ,y ) ∼D [ yf W ( T CutMix ) ( X ) > 0 ] = 1 -o ( 1 poly( d ) ) .

To prove Theorem 3.3, we characterize the global minimum of objective loss of CutMix . Surprisingly, at the global minimum, the model has the same outputs for all patches of the input data. In other words, the contributions of all feature vectors and noise vectors to the final outcome of the network are identical, regardless of their frequency and strength (see Section 4.2 for more details). Moreover, this uniform 'contribution' is large enough, which allows the model to learn all types of features by reaching the global minimum. We provide the detailed proof in Appendix E.2.

Our three main theorems elucidate the benefits of Cutout and CutMix . Cutout enables a model to learn rarer features than ERM , while CutMix can even outperform Cutout . These advantages in learning rarer features lead to improvements in generalization performance.

4 Overview of Analysis

In this section, we discuss key proof ideas and the main challenges in our analysis. For ease of presentation, we consider the case α = 0 . Although our assumptions do not allow the choice α = 0 , the choice of nonzero α is to show guarantees on the test accuracy and does not significantly affect the feature learning aspect.

To provide the proof overview, let us introduce some additional notation. For each i ∈ [ n ] , recall that the corresponding input point can be written as X i = ( x (1) i , . . . , x ( P ) i ) . We use p ∗ i and ˜ p i to denote the indices of its feature patch and dominant noise patch, respectively. For each feature vector v s,k , where s ∈ {± 1 } and k ∈ [ K ] , let V s,k ⊂ [ n ] represent the set of indices of data points having the feature vector v s,k , and V s = ⋃ K k =1 V s,k denotes the set of indices of data with label s . For each data point i ∈ [ n ] and dominant or background noise patch p ∈ [ P ] \ { p ∗ i } , we refer to the Gaussian noise inside x ( p ) i as ξ ( p ) i .

4.1 Vanilla Training and Cutout Training

Wenowexplain why ERM fails to learn (extremely) rare features, while Cutout can learn rare features but not extremely rare features. Let us consider ERM . From (1), for s, s ′ ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } , the component of w s in the feature vector v s ′ ,k 's direction is updated as

and similarly, the 'update' of inner product of w s with a noise patch ξ ( p ) i can be written as

where the approximation is due to the near-orthogonality of Gaussian random vectors in the highdimensional regime. This approximation shows that ⟨ w ( t +1) s , v s ′ ,k ⟩ 's and ⟨ w ( t ) s , ξ ( p ) i ⟩ 's are almost monotonically increasing or decreasing. We address the approximation errors using a variant of the technique introduced by Cao et al. (2022), as detailed in Appendix B.3.

From (4) and (5), we can observe that in the early phase of training satisfying -ℓ ′ ( y i f W ( t ) ( X i )) = Θ(1) , the main factor for the speed of learning features and noises are the number of feature occurrence |V s ′ ,k | and the strength of noise ∥ ξ ( p ) i ∥ 2 . From our assumptions introduced in Section 2.4, if we compare the learning speed of different components, we have common features ≫ dominant noises ≫ rare features ≫ background noises ≫ extremely rare features ,

in terms of 'learning speed.' Based on this observation, we conduct a three-phase analysis for ERM.

Phase 1 : Learning common features quickly.

Phase 2 : Fitting (extremely) rare data by memorizing dominant noises instead of learning features.

Phase 3 : A model cannot learn (extremely) rare features since gradients of all data are small.

The main intuition behind why ERM cannot learn (extremely) rare features is that the gradients of all data containing these features become small after quickly memorizing dominant noise patches. In contrast, since Cutout randomly cuts some patches out, there exist augmented data points that do not contain dominant noises and have only features and background noises. This allows Cutout to learn rare features, thanks to these augmented data. However, extremely rare features cannot be learned since the learning speed of background noise is much faster and there are too many background noise patches to cut them all out.

Remark 4.1 . Shen et al. (2022) conduct analysis on vanilla training and training using standard data augmentation, sharing the same intuition in similar but different data models and neural networks. Also, we emphasize that we proved the model cannot learn (extremely) rare features even if we run poly( d ) η iterations of GD, whereas Shen et al. (2022) only consider the first iteration that achieves perfect training accuracy.

Practical Insights. In practice, images contain features and noise across several patches. A larger cutting size can be more effective in removing noise but may also remove important features that the model needs to learn. Thus, there is a trade-off in choosing the optimal cutting size, a trend also observed in DeVries and Taylor (2017). One limitation of Cutout is that it may not effectively remove dominant noise. Thus, dominant noise can persist in the augmented data, leading to potential noise memorization. We believe that developing strategies that can more precisely detect and remove these noise components from the image input could enhance the effectiveness of these methods.

4.2 CutMix Training

In learning dynamics of ERM and Cutout , inner products between weight and data patches evolve (approximately) monotonically, which makes the analysis much more feasible. However, analyzing the learning dynamics of CutMix involves non-monotone change of inner products, which is inevitable since CutMix uses mixed labels; this is also demonstrated in our experimental results (Section 5,especially the leftmost plot in Figure 1). Non-monotonicity and non-convexity of the problem necessitates novel proof strategies.

Then, Z represents the contribution of each noise patch and feature vector to the neural network output, and the nonconvex function L CutMix ( W ) can be viewed as the composition of Z ( W ) and a convex function h ( Z ) . By using the convexity of h ( Z ) , we can characterize the global minimum of L CutMix ( W ) . Surprisingly, we show that any global minimizer W ∗ = { w ∗ 1 , w ∗ -1 } satisfies

for all s ∈ {± 1 } , i ∈ V s , and p ∈ [ P ] , with some constants C 1 , C -1 = Θ(1) . In other words, at the global minimum, the output of model on each patch of the training data is uniform across the set of data with the same labels. We also prove that CutMix can achieve a point close to the global minimum within poly( d ) η iterations. As a result, the model trained by CutMix can learn all features including extremely rare features. The complete proof of Theorem 3.3 appears in Appendix E.2.

Remark 4.2 . Zou et al. (2023) investigate Mixup in a similar feature-noise model and show that Mixup can learn rarer features than vanilla training, with its benefits emerging from the early dynamics of training. However, our characterization of the global minimum of L CutMix ( W ) and experimental results in our setting (Section 5, Figure 1) suggest that the benefits of CutMix , especially for learning extremely rare features, arise from the later stages of training. This suggests that Mixup and CutMix have different underlying mechanisms for promoting feature learning.

Practical Insights. The main underlying mechanism of CutMix is that it learns information almost uniformly from all patches in the training data. However, this approach also involves memorizing noise, which can potentially degrade performance in real-world scenarios. We believe that a more sophisticated strategy such as considering the positional information of patches as used in Puzzle Mix (Kim et al., 2020) or Co-Mixup (Kim et al., 2021) could improve the ability to learn more from patches containing features and reduce the impact of noise.

5 Experiments

We conduct experiments both in our setting and real-world data CIFAR-10 to support our theoretical findings and intuition. We defer CIFAR-10 experiment results to Appendix A.1.

For the numerical experiments on our setting, we set the number of patches P = 3 , dimension d = 2000 , number of data points n = 300 , dominant noise strength σ d = 0 . 25 , background noise strength σ b = 0 . 15 , and feature noise strength α = 0 . 005 . The feature vectors are given as the standard basis e 1 , e 2 , e 3 , e 4 , e 5 , e 6 ∈ R d , where e 1 , e 2 , e 3 are features for the positive label y = 1 and e 4 , e 5 , e 6 are features for the negative label y = -1 . We categorize e 1 and e 4 as common features with a frequency of 0 . 8 , e 2 and e 5 as rare features with a frequency of 0 . 15 , and lastly, e 3 and e 6 as extremely rare features with a frequency of 0 . 05 . For the learner network, we set the slope of negative regime β = 0 . 1 and the length of the smoothed interval r = 1 . We train models using three methods: ERM , Cutout , and CutMix with a learning rate η = 1 . For Cutout , we cut a single patch of data ( C = 1 ). We apply full-batch gradient descent for all methods; for Cutout and CutMix , we utilize all possible augmented data points. 3 We note that this choice of problem parameters does not exactly match the technical assumptions in Section 2.4. However, we empirically observe the same conclusions, which suggests that our analysis could be extended beyond our assumptions.

For each feature vector v of the positive label, we plot the output of the learned filters for the feature vector ϕ ( ⟨ w ( t ) 1 , v ⟩ ) -ϕ ( ⟨ w ( t ) -1 , v ⟩ ) throughout training in Figure 1. Our numerical findings confirm that ERM can only learn common features, Cutout can learn common and rare features but cannot learn extremely rare features, and CutMix can learn all types of features. Especially, CutMix learn common features, rare features, and extremely rare features almost evenly. Also, we observed nonmonotone behavior of the output in the case of CutMix , which motivated our novel proof technique. The same trends are observed with different architectures, such as a smoothed (leaky) ReLU network with multiple neurons, as detailed in Appendix A.2.

0

2000

4000

6000

8000

10000

Iterations

0

2

4

6

8

10

Common Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0.0

0.2

0.4

0.6

0.8

Rare Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0.0

0.2

0.4

0.6

0.8

Extremely Rare Feature Output

ERM

Cutout

CutMix

6 Conclusion

We studied how Cutout and CutMix influence the ability to learn features in a patch-wise featurenoise data model learning with two-layer convolutional neural networks by comparing them with vanilla training. We showed that Cutout enables the learning of rare features that cannot be learned through vanilla training by mitigating the problem of memorizing label-independent noises instead of learning label-dependent features. Surprisingly, we further proved that CutMix can learn extremely rare features that Cutout cannot learn. We also present our theoretical insights on the underlying mechanism of these methods and provide experimental support.

Limitation and Future Work. Our work has some limitations related to the neural network architecture, specifically, the use of a 2-layer two-neuron smoothed leaky ReLU network. Extending our results to neural networks with deeper, wider, and more general activation functions is a direction for future work. Another future direction is to develop patch-level data augmentation based on our theoretical findings. Also, it would be interesting to perform theoretical analysis on state-of-the-art patch-level data augmentation such as Puzzle Mix (Kim et al., 2020) or Co-Mixup (Kim et al., 2021). These methods utilize patch location information, thus it may require the development of a theoretical framework capturing more complex characteristics of image data.

References

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 , 2020.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250, 2022.

Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regularization. arXiv preprint arXiv:2006.06049 , 2020.

Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint arXiv:2001.04086 , 2020a.

Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 21321-21333, 2020b.

Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning. Advances in neural information processing systems , 35: 23049-23062, 2022.

Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsieh, and Quanquan Gu. Why does sharpness-aware minimization generalize better than sgd? In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Muthu Chidambaram and Rong Ge. For better or for worse? learning minimum variance features with label augmentation. arXiv preprint arXiv:2402.06855 , 2024.

Muthu Chidambaram, Xiang Wang, Yuzheng Hu, Chenwei Wu, and Rong Ge. Towards understanding the data dependency of mixup-style training. arXiv preprint arXiv:2110.07647 , 2021.

Muthu Chidambaram, Xiang Wang, Chenwei Wu, and Rong Ge. Provably learning diverse features in multi-view data with midpoint mixup. In International Conference on Machine Learning , pages 5563-5599. PMLR, 2023.

Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. A kernel theory of modern data augmentation. In International conference on machine learning , pages 1528-1537. PMLR, 2019.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory , pages 2668-2703. PMLR, 2022a.

Spencer Frei, Gal Vardi, Peter L Bartlett, Nathan Srebro, and Wei Hu. Implicit bias in leaky relu networks trained on high-dimensional data. arXiv preprint arXiv:2210.07082 , 2022b.

Boris Hanin and Yi Sun. How data augmentation affects optimization for linear regression. Advances in Neural Information Processing Systems , 34:8095-8105, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Wei Huang, Yuan Cao, Haonan Wang, Xin Cao, and Taiji Suzuki. Graph neural networks provably benefit from structural information: A feature learning perspective. arXiv preprint arXiv:2306.13926 , 2023a.

Wei Huang, Ye Shi, Zhongyi Cai, and Taiji Suzuki. Understanding convergence and generalization in federated learning through feature learning theory. In The Twelfth International Conference on Learning Representations , 2023b.

Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning , pages 9965-10040. PMLR, 2022.

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. In International Conference on Machine Learning , pages 5034-5044. PMLR, 2021.

Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning , pages 5275-5285. PMLR, 2020.

Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2102.03065 , 2021.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International Conference on Machine Learning , pages 17615-17659. PMLR, 2023a.

Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks on nearly-orthogonal data. In Thirty-seventh Conference on Neural Information Processing Systems , 2023b.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Binghui Li and Yuanzhi Li. Why clean generalization and robust overfitting both happen in adversarial training. arXiv preprint arXiv:2306.01271 , 2023.

Junsoo Oh and Chulhee Yun. Provable benefit of mixup for finding optimal decision boundaries. In International Conference on Machine Learning , pages 26403-26450. PMLR, 2023.

Chanwoo Park, Sangdoo Yun, and Sanghyuk Chun. A unified analysis of mixed sample data augmentation: A loss function perspective. Advances in Neural Information Processing Systems , 35:35504-35518, 2022.

Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead to positive margin? In International Conference on Machine Learning , pages 5321-5330. PMLR, 2019.

Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. Data augmentation as feature manipulation. In International conference on machine learning , pages 19773-19808. PMLR, 2022.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. In International Conference on Machine Learning , pages 10410-10420. PMLR, 2020.

Jaejun Yoo, Namhyuk Ahn, and Kyung-Ah Sohn. Rethinking data augmentation for image superresolution: A comprehensive analysis and a new strategy. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8375-8384, 2020.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.04819 , 2020.

Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning , pages 26135-26160. PMLR, 2022.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 13001-13008, 2020.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. arXiv preprint arXiv:2108.11371 , 2021.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning. In International Conference on Machine Learning , pages 43423-43479. PMLR, 2023.

Contents

F

Technical Lemmas

A Additional Experimental Results

For all experiments described in this section and in Section 5, we use NVIDIA RTX A6000 GPUs.

A.1 Experiments on CIFAR-10 Dataset

A.1.1 Experimental Support for Our Intuition

We compare three methods, ERM training, Cutout training, and CutMix training on CIFAR-10 classification. For ERM training, we apply only random cropping and random horizontal flipping on train dataset. In comparison, for Cutout training and CutMix training, we additionally apply Cutout and CutMix, respectively, on training data. For Cutout training, we randomly cut 16 × 16 pixels of input images, and for CutMix training, we sample the mixing ratio from a beta distribution Beta(0 . 5 , 0 . 5) . We train ResNet-18 (He et al., 2016) for 200 epochs with a batch size of 128 using SGD with a learning rate 0 . 1 , momentum 0 . 9 , and weight decay 5 × 10 -4 . Trained models using ERM, Cutout, and CutMix achieve test accuracy 95 . 16% , 96 . 05% , and 96 . 29% , respectively.

We randomly generate augmented data using CutMix from pairs of cat images and dog images in CIFAR-10 with varying mixing ratios λ = 1 , 0 . 8 , 0 . 6 (Dog:Cat = λ : 1 -λ ). We randomly make 5 , 000 (cat, dot)-pairs in CIFAR-10 training set and apply CutMix randomly 10 times. By repeating this procedure 10 times, we generate total 5 , 000 × 10 × 10 = 500 , 000 augmented samples for each mixing ratio λ . We plot a histogram of dog prediction output subtracted by cat prediction output (before applying the softmax function), evaluated on 500 , 000 augmented data in Figure 2.

15

10

5

0

5

10

15

20

Dog Output - Cat Output

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Frequency

ERM

Cutout

CutMix

15

10

5

0

5

10

15

20

Dog Output - Cat Output

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Frequency

ERM

Cutout

CutMix

15

10

5

0

5

10

15

20

Dog Output - Cat Output

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Frequency

ERM

Cutout

CutMix

The leftmost plot represents the evaluation results for original dog images, as it uses a mixing ratio of λ = 1 . We can observe that the output of the model trained using Cutout is skewed toward higher values compared to the output of the model trained using other methods. We believe this aligns with the theoretical intuition that Cutout learns more information from the original image using augmented data.

The remaining two plots show the output for randomly augmented data using CutMix. We observe that the models trained with CutMix exhibit a shorter tail, supporting our intuition from the CutMix analysis that the models learn uniformly across all patches.

A.1.2 Experimental Support for Our Findings

We train ResNet-18 using ERM training, Cutout training, and CutMix training following the same experimental details described in Appendix A.1.1, except using only 10% of the training set. This data-hungry setting is intended to highlight the benefits of Cutout and CutMix. We then evaluated the trained models on the remaining 90% of the CIFAR-10 training dataset. The reason for evaluating the remaining training dataset is to analyze the misclassified data using C-score (Jiang et al., 2021), which is publicly available only for the training dataset.

C-score measures the structural regularity of data, with lower values indicating examples that are more difficult to classify correctly. In our framework, data with harder-to-learn features (corresponding to rarer features) would likely have lower C-scores. Since directly extracting and quantitatively evaluating features learned by the models is challenging, we use the C-score as a proxy to evaluate the misclassified data across models trained by ERM, Cutout, and CutMix.

Table 1 illustrates that Cutout tends to misclassify data with lower C-scores compared to ERM, indicating that Cutout learns more hard-to-learn features than vanilla training. Furthermore, the data misclassified by CutMix has even lower C-scores than those misclassified by Cutout, suggesting that CutMix is effective at learning features that are the most challenging to classify. This observation aligns with our theoretical findings, demonstrating that CutMix captures even more difficult features compared to both ERM and Cutout.

Since directly visualizing features learned by a model is challenging, we present data that were misclassified by the model trained with ERM but correctly classified by the model trained with Cutout instead. In Figure 3, we show 7 samples per class with the lowest C-scores, which are considered to have rare features. Similarly, we also visualize data misclassified by the model trained with Cutout but correctly classified by the model trained with CutMix to represent extremely rare data in Figure 4. This approach allows us to interpret some (extremely) rare features in CIFAR-10, such as frogs with unusual colors.

C-score: 0.024

C-score: 0.028

C-score: 0.029

C-score: 0.032

C-score: 0.032

C-score: 0.033

C-score: 0.037

C-score: 0.037

C-score: 0.038

C-score: 0.043

C-score: 0.045

C-score: 0.047

C-score: 0.049

C-score: 0.053

C-score: 0.056

C-score: 0.056

C-score: 0.057

C-score: 0.059

C-score: 0.060

C-score: 0.060

C-score: 0.065

C-score: 0.068

C-score: 0.072

C-score: 0.072

C-score: 0.077

C-score: 0.080

C-score: 0.081

C-score: 0.081

C-score: 0.083

C-score: 0.085

C-score: 0.085

C-score: 0.087

C-score: 0.088

C-score: 0.090

C-score: 0.091

C-score: 0.091

C-score: 0.094

C-score: 0.099

C-score: 0.102

C-score: 0.103

C-score: 0.105

C-score: 0.105

C-score: 0.108

C-score: 0.109

C-score: 0.111

C-score: 0.122

C-score: 0.124

C-score: 0.135

C-score: 0.143

C-score: 0.146

C-score: 0.150

C-score: 0.168

C-score: 0.180

C-score: 0.187

C-score: 0.187

C-score: 0.188

C-score: 0.204

C-score: 0.212

C-score: 0.213

C-score: 0.215

C-score: 0.222

C-score: 0.235

C-score: 0.255

C-score: 0.264

C-score: 0.308

C-score: 0.327

C-score: 0.344

C-score: 0.360

C-score: 0.432

C-score: 0.440

C-score: 0.017

C-score: 0.017

C-score: 0.019

C-score: 0.021

C-score: 0.022

C-score: 0.029

C-score: 0.033

C-score: 0.033

C-score: 0.035

C-score: 0.037

C-score: 0.037

C-score: 0.039

C-score: 0.041

C-score: 0.042

C-score: 0.045

C-score: 0.048

C-score: 0.049

C-score: 0.051

C-score: 0.052

C-score: 0.052

C-score: 0.054

C-score: 0.054

C-score: 0.055

C-score: 0.056

C-score: 0.059

C-score: 0.059

C-score: 0.062

C-score: 0.064

C-score: 0.065

C-score: 0.069

C-score: 0.069

C-score: 0.071

C-score: 0.074

C-score: 0.074

C-score: 0.082

C-score: 0.091

C-score: 0.091

C-score: 0.096

C-score: 0.101

C-score: 0.103

C-score: 0.104

C-score: 0.105

C-score: 0.110

C-score: 0.111

C-score: 0.113

C-score: 0.114

C-score: 0.118

C-score: 0.124

C-score: 0.126

C-score: 0.128

C-score: 0.132

C-score: 0.134

C-score: 0.136

C-score: 0.139

C-score: 0.148

C-score: 0.148

C-score: 0.154

C-score: 0.168

C-score: 0.171

C-score: 0.174

C-score: 0.178

C-score: 0.196

C-score: 0.206

C-score: 0.212

C-score: 0.216

C-score: 0.247

C-score: 0.247

C-score: 0.250

C-score: 0.276

C-score: 0.283

A.2 Additional Experimental Results on Our Data Distribution

In addition to the results described in Section 5, we further conducted numerical experiments on our data distribution by applying two variations to our architecture: increasing the number of neurons, and increasing the number of neurons with a smoothed ReLU activation (instead of smoothed leaky ReLU). We observed the same trends as predicted by our theoretical findings and shown in Figure 1.

Let us describe the setting of our experiments in detail. In both cases, We set the number of patches P = 3 , dimension d = 2000 , and the number of data n = 300 . The feature vectors are given by the standard basis e 1 , e 2 , e 3 , e 4 , e 5 , e 6 ∈ R d , where e 1 , e 2 , e 3 are features for the positive label y = 1 and e 4 , e 5 , e 6 are features for the negative label y = -1 . We categorize e 1 and e 4 as common features, e 2 and e 5 as rare features, and lastly, e 3 and e 6 as extremely rare features. We apply full-batch gradient descent with learning rate η = 1 and for Cutout and CutMix , we utilize all possible augmented data.

For the multi-neuron with smoothed Leaky ReLU case (Figure 5), we use 10 neurons for each positive/negative output with the slope of negative regime β = 0 . 1 and the length of polynomial regime r = 1 . We set the strength of dominant noise σ d = 0 . 25 , the strength of background noise σ b = 0 . 12 , and the strength of feature noise α = 0 . 05 . In addition, frequencies of common features, rare features, and extremely rare features are set to 0 . 72 , 0 . 15 , and 0 . 03 , respectively.

For the multi-neuron with smoothed ReLU case i.e., β = 0 (Figure 6), we set the length of the polynomial regime as r = 1 , and we use 10 neurons for each positive/negative output. We set the remaining problem parameters as follows: the strength of dominant noise σ d = 0 . 25 , the strength of background noise σ b = 0 . 12 , and the strength of feature noise α = 0 . 05 . In addition, frequencies of common features, rare features, and extremely rare features are set to 0 . 75 , 0 . 2 , and 0 . 05 , respectively.

0

2000

4000

6000

8000

10000

Iterations

0

2

4

6

8

Common Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0

1

2

3

4

Rare Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0.0

0.2

0.4

0.6

0.8

Extremely Rare Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0

2

4

6

8

10

Common Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0.0

0.5

1.0

1.5

2.0

Rare Feature Output

ERM

Cutout

CutMix

0

2000

4000

6000

8000

10000

Iterations

0.0

0.2

0.4

0.6

0.8

Extremely Rare Feature Output

ERM

Cutout

CutMix

B Proof Preliminaries

B.1 Properties of the Choice of Problem Parameters

In our analysis, we consider the choice of problem parameters as a function of the dimension of patches d and consider sufficiently large d . Let us summarize the assumptions on the parameters for the problem setting and assume they hold.

Assumption B.1. The following conditions hold.

A1. (The number of patches) P = Θ(1) and P ≥ 8 .

A2. (Overparameterized regime): n = o ( αβσ -1 d σ b d 1 2 / polylog( d ) ) .

A3. (Sufficient feature data): For all k ∈ [ K ] , ρ k n = ω ( n 1 2 log d ) .

A4. (Common feature vs dominant noise): For all k ∈ K C , ρ k = Θ(1) and σ 2 d d = o ( βn ) .

A5. (Rare feature vs noise): For all k ∈ K R , ρ k = Θ( ρ R ) with ρ R n = o ( α 2 σ 2 d d/ polylog( d ) ) and σ 2 b d = o ( βρ R n ) .

A6. (Extremely rare feature vs background noise) For all k ∈ K E , ρ k = Θ( ρ E ) with ρ E n = o ( α 2 σ 2 b d/ polylog( d ) ) .

A7. (Strength of feature noise) α = o ( n -1 βσ 2 d d/ polylog( d ) ) .

We now present some properties derived from Assumption B.1, which are frequently used throughout our proof.

From (A3), for all k ∈ [ K ] , we have the following inequality:

From (A1) and (A2), and given that β < 1 , σ b < σ d , we have

From (A2), (A3), and (A6), and given that α, β < 1 , we have

From (A1), (A2) and the fact that 0 < α < 1 , we have

From (A7) and (A4), we have

From (8) and (A8), η = o (1) and then we have

From (A2), (A3), (A4), and (A5) we have

B.2 Quantities at the Beginning

We characterize some quantities at the beginning of training.

Lemma B.2. Let E init the event such that all the following holds:

25 52 n ≤ |V 1 | , |V -1 | ≤ 27 52 n

For each s ∈ {± 1 } and k ∈ [ K ] , ρ k n 4 ≤ |V s,k | ≤ 3 ρ k n 4

∪ i ∈V 1 , 1 { p ∗ i } = [ P ]

For any s ∈ {± 1 } and i ∈ [ n ] , ∣ ∣ ∣ 〈 w (0) s , ξ (˜ p i ) i 〉∣ ∣ ∣ ≤ σ 0 σ d d 1 2 log d .

For any s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i , ˜ p i } , ∣ ∣ ∣ 〈 w (0) s , ξ ( p ) i 〉∣ ∣ ∣ ≤ σ 0 σ b d 1 2 log d .

̸

̸

For any i, j ∈ [ n ] and p ∈ [ P ] \ { p ∗ i , ˜ p i } , q ∈ [ P ] \ { p ∗ j , ˜ p j } with ( i, p ) = ( j, q ) , 1 2 σ 2 b d ≤ ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 b d and ∣ ∣ ∣ 〈 ξ ( p ) i , ξ ( q ) j 〉∣ ∣ ∣ ≤ σ 2 b d 1 2 log d .

{ v s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { x ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } is linearly independent.

Then, the event E init occurs with probability at least 1 -o ( 1 poly( d ) ) . Also, if ξ ∼ N ( 0 , σ 2 Λ) is independent of w (0) 1 , w (0) -1 and { ( X i , y i ) } i ∈ [ n ] , we have

for all i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } , with probability at least 1 -o ( 1 poly( d ) ) .

Proof of Lemma B.2. Let us prove the first three points hold with probability at least 1 -o ( 1 poly( d ) ) . By Höeffding's inequality,

where the last equality is due to (6). In addition, for each s ∈ {± 1 } , k ∈ [ K ] , by Höeffding's inequality

where the last equality is due to (6). Also, for each i ∈ [ n ] and p ∈ [ P ] ,

Hence,

Next, we will prove the remaining. Let us refer to the standard deviation of the Gaussian noise vector in p -th patch of i -th data as σ i,p . In other words, for each i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } ,

For each s, s ′ ∈ {± 1 } and k ∈ [ K ] , 〈 w (0) s , v s ′ ,k 〉 ∼ N (0 , σ 0 ) . Hence, by Höeffding's inequality, we have

Let { u l } l ∈ [ d -2 K ] be an orthonormal basis of the orthogonal complement of Span( { v s,k } s ∈{± 1 } ,k ∈ [ K ] ) . Note that for each s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } , we can write ξ ( p ) i and ξ as

where z s,l , z ( p ) i,l , z l i.i.d. ∼ N (0 , 1) . The sub-gaussian norm of standard normal distribution N (0 , 1) is √ 8 3 . Then ( z ( p ) i,l ) 2 -1 's are mean zero sub-exponential with sub-exponential norm 8 3 (Lemma 2.7.6 in Vershynin (2018)). In addition, z s,l z ( p ) i,l 's, z ( p ) i,l z ( q ) j,l 's and z ( p ) i,l z l 's are mean zero sub-exponential with sub-exponential norm less than or equal to 8 3 (Lemma 2.7.7 in Vershynin (2018)). We use Bernstein's inequality (Theorem 2.8.1 in Vershynin (2018)), with c being the absolute constant stated therein. We then have the following:

in addition,

Similarly, we have

Lastly, the last result holds almost surely due to (7). Applying the union bound to all events, each of which is at most poly( d ) due to (7), leads us to our first conclusion.

In addition, for each s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } ,

and

Applying the union bound to all events, each of which is at most poly( d ) due to (7), leads us to our second conclusion.

B.3 Feature Noise Decomposition

In our analysis, we use a technique that analyzes the coefficients of linear combinations of feature and noise vectors. A similar technique in a different data and network setting is introduced by Cao et al. (2022).

Lemma B.3. If we run one of ERM , Cutout , and CutMix training to update parameters W ( t ) of a model f W ( t ) , then there exist coefficients (corresponding to each method) γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's so that we can write W ( t ) = { w ( t ) 1 , w ( t ) -1 } as

where F s denotes the set of indices of data with feature noise v s, 1 . Furthermore, if we run one of ERM and Cutout , the coefficients γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's are monotone increasing.

We provide proof of Lemma B.3 for ERM in Appendix C.1, for Cutout in Appendix D.1 and for CutMix in Appendix E.1.

Since Gaussian vectors in a high-dimensional regime are nearly orthogonal, we can use the coefficients to approximate the inner products or outputs of neurons. The following lemma quantifies the approximation error.

Lemma B.4. Suppose the event E init occurs and 0 ≤ γ ( t ) s ( s ′ , k ) , ρ ( t ) s ( i, p ) ≤ ˜ O ( β -1 ) for all s, s ′ ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } at iteration t . Then, for each s ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , the following holds:

Proof of Lemma B.4. For each s ∈ {± 1 } , k ∈ [ K ] \ { 1 } , by (A8) and (8), we have

Similarly, by (A8) and (8),

Next, we will consider the case of v 1 , 1 and v -1 , 1 . For each s ∈ {± 1 } , we have

where the last equality is due to (8) and (A7). Similarly, we have

Hence, from (A8) and the fact that | ϕ ( z ) -z | ≤ (1 -β ) r 2 for any z ≥ 0 , we have

and

For each i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , we have

̸

where the last equality is due to (A8) and (9). By triangular inequality, (A8), and the fact that ϕ ′ ≤ 1 and | ϕ ( z ) -z | ≤ (1 -β ) r 2 for any z ≥ 0 , we have

Also, if i ∈ F s for some s ∈ {± 1 } ,

where we apply the triangular inequality, the fact that ϕ ′ ≤ 1 , the triangular inequality again, ρ ( t ) y i ( s, 1) = ˜ O ( β -1 ) and (10) sequentially. Similarly,

̸

and

Also, if i ∈ F s for some s ∈ {± 1 } ,

We define the set W as the collection of W = { w 1 , w -1 } , where w 1 -w (0) 1 , w -1 -w (0) -1 are elements of the subspace spanned by { v s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { x ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } . The following lemma guarantees the unique expression of any W ∈ W in the form of the feature noise decomposition.

Lemma B.5. Suppose the event E init occurs. For each element W = { w 1 , w -1 } ∈ W , there exist unique coefficients γ s ( s ′ , k ) 's and ρ s ( i, p ) 's such that

for each s ∈ {± 1 } . Using this fact, for each s ∗ ∈ {± 1 } and k ∗ ∈ [ K ] , we can introduce a function Q ( s ∗ ,k ∗ ) : W → R d × 2 such that for each W = { w 1 , w -1 } ∈ W ,

is given by:

The function Q ( s ∗ ,k ∗ ) plays a crucial role in Section C.2.4 and Section D.2.4. The key intuition behind our definition of Q ( s ∗ ,k ∗ ) is that Q ( s ∗ ,k ∗ ) ( W ( t ) ) represents the term updated by the data having the feature vector v s ∗ ,k ∗ , where W ( t ) are the iterates of either ERM or Cutout . As expected from this intuition, if we sum all Q ( s ∗ ,k ∗ ) 1 ( w 1 ) and Q ( s ∗ ,k ∗ ) -1 ( w -1 ) over all s ∗ ∈ {± 1 } and k ∗ ∈ [ K ] , the result will be equal to w 1 -w (0) 1 and w -1 -w (0) -1 , respectively.

Proof. From linear independency of { v s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { x ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } , we can express any element W = { w 1 , w -1 } ∈ W as

̸

with unique { ˜ γ s ( s, k ) , ˜ γ s ( -s, k ) } s ∈{± 1 } ,k ∈ [ K ] and { ρ s ( i, p ) } s ∈{± 1 } ,i ∈ [ n ] ,p ∈ [ P ] \{ i ∗ } . If we define γ s ( s, k ) and γ s ( -s, k ) as γ s ( s, k ) = ˜ γ s ( s, k ) , γ s ( -s, k ) = ˜ γ s ( -s, k ) for k = 1 , and

then we have

Next, we want to show the uniqueness part. Suppose { ˆ γ s ( s, k ) , ˆ γ s ( -s, k ) } s ∈{± 1 } ,k ∈ [ K ] and { ˆ ρ s ( i, p ) } s ∈{± 1 } ,i ∈ [ n ] ,p ∈ [ P ] \{ i ∗ } satisfies

We have

From the uniqueness of (13), we have

ˆ γ s ( s, k ) = ˜ γ s ( s, k ) = γ s ( s, k ) , ˆ γ s ( -s, k ) = ˜ γ s ( -s, k ) = γ s ( -s, k ) , for each s ∈ {± 1 } , k ∈ [ K ] \ { 1 } , and ˆ ρ s ( i, p ) = ρ s ( i, p ) for each i ∈ [ n ] , p ∈ [ P ] \ { p ∗ i } . Furthermore,

and

Hence, we obtain the uniqueness of the expression and Q ( s ∗ ,k ∗ ) is well defined for each s ∗ ∈ {± 1 } and k ∗ ∈ [ K ] .

C Proof for ERM

In this section, we use g ( t ) i := 1 1+exp ( y i f W ( t ) ( X i ) ) for each data i and iteration t , for simplicity.

C.1 Proof of Lemma B.3 for ERM

For s ∈ {± 1 } and iterate t ,

and we have

and

Hence, if we define γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's recursively by using the rule

starting from γ (0) s ( s ′ , k ) = ρ (0) s ( i, p ) = 0 for each s, s ′ ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] and p ∈ [ P ] \{ p ∗ i } , then we have

for each s ∈ {± 1 } . Furthermore, γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's are monotone increasing. □

C.2 Proof of Theorem 3.1

To show Theorem 3.1, we present a structured proof comprising the following five steps:

Establish upper bounds on γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's to apply Lemma B.4 (Section C.2.1).

Demonstrate that the model learns common features quickly (Section C.2.2).

Show that the model overfits dominant noise in (extremely) rare data instead of learning its feature (Section C.2.3).

Confirm the persistence of this tendency until T ∗ iterates (Section C.2.4).

Characterize train accuracy and test accuracy (Section C.2.5).

C.2.1 Bounds on the Coefficients in Feature Noise Decomposition

The following lemma provides upper bounds on Lemma B.3 during T ∗ iterations.

Lemma C.1. Suppose the event E init occurs. For any t ∈ [0 , T ∗ ] , we have

Proof of Lemma C.1. The first argument implies the second argument since log( ηT ∗ ) = polylog( d ) and

We will prove this by using induction on t . The initial case t = 0 is trivial. Suppose the given statement holds at t = T and consider the case t = T +1 .

Let ˜ T s,k ≤ T denote the smallest iteration where γ ( ˜ T s,k +1) s ( s, k ) + βγ ( ˜ T s,k +1) -s ( s, k ) > 2 log( ηT ∗ ) . We assume the existence of ˜ T s,k , as its absence would directly lead to our desired conclusion; to see why, note that the following holds, due to (14) and (11):

Now suppose there exists such ˜ T s,k ≤ T . By (14), we have

The inequality is due to γ ( ˜ T s,k ) s ( s, k ) + βγ ( ˜ T s,k ) -s ( s, k ) ≤ 2 log( ηT ∗ ) from our choice of ˜ T s,k and

from (11).

For each t = ˜ T s,k +1 , . . . T , and i ∈ V s,k , we have

The first inequality is due to Lemma B.4 and the second inequality holds due to (A7), (8), and our choice of t , γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) ≥ 2 log( ηT ∗ ) .

Hence, we obtain

where the last inequality holds for any reasonably large T ∗ . Merging all inequalities together, we have γ ( T +1) s ( s, k ) + βγ ( T +1) -s ( s, k ) ≤ 4 log( ηT ∗ ) .

Next, we will follow similar arguments to show that

for each i ∈ [ n ] and p ∈ [ P ] \ { p i } .

Let ˜ T ( p ) i ≤ T be the smallest iteration such that ρ ( ˜ T ( p ) i +1) y i ( i, p ) + βρ ( ˜ T ( p ) i +1) -y i ( i, p ) > 2 log( ηT ∗ ) . We assume the existence of ˜ T ( p ) i , as its absence would directly lead to our desired conclusion; to see why, note that the following holds, due to (15) and (11):

where the first inequality is due to ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ ≤ 3 2 σ 2 d d and (A4), and the last inequality is due to (11).

Now suppose there exists such ˜ T ( p ) i ≤ T . By (15), we have

and

The inequality is due to ρ ( ˜ T ( p ) i ) y i ( i, p ) + βρ ( ˜ T ( p ) i ) -y i ( i, p ) ≤ 2 log( ηT ∗ ) from our choice of ˜ T ( p ) i η n g ( ˜ T ( p ) i ) i [ ϕ ′ (〈 w ( ˜ T ( p ) i ) s , x ( p ) i 〉) + βϕ ′ (〈 w ( ˜ T ( p ) i ) -s , x ( p ) i 〉)] ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 2 η ≤ log( ηT ∗ ) , from ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 d d , (A4), and (11).

The first inequality is due to Lemma B.4 and the second inequality holds because from our choice of t , ρ ( t ) y i ( i, p ) + βρ ( t ) -y i ( i, p ) ≥ 2 log( ηT ∗ ) .

Therefore, we have

where the first inequality is due to ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 d , (A4) and the last inequality holds for any reasonably large T ∗ . Merging all inequalities together, we conclude ρ ( T +1) y i ( i, p ) + βρ ( T +1) -y i ( i, p ) ≤ 4 log( ηT ∗ ) .

C.2.2 Learning Common Features

In the initial stages of training, the model quickly learns common features while exhibiting minimal overfitting to Gaussian noise.

First, we establish lower bounds on the number of iterations ensuring that noise coefficients ρ ( t ) s ( i, p ) remain small, up to the order of 1 P .

Lemma C.2. Suppose the event E init occurs. There exists ˜ T > n 6 ηPσ 2 d d such that ρ ( t ) s ( i, p ) ≤ 1 4 P for all 0 ≤ t < ˜ T, s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } .

Proof of Lemma C.2. Let ˜ T be the smallest iteration such that ρ ( ˜ T ) s ( i, p ) ≥ 1 4 P for some s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } . We assume the existence of ˜ T , as its absence would directly lead to our conclusion. Then, for any 0 ≤ t < ˜ T , we have

where the inequality is due to g ( t ) i < 1 , ϕ ′ ≤ 1 , and ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 d d . Hence, we have

and we conclude ˜ T > n 6 ηPσ 2 d d which is the desired result.

Next, we will show that the model learns common features in at least constant order within ˜ T iterates. Lemma C.3. Suppose the event E init occurs and ρ k = ω ( σ 2 d d βn ) for some k ∈ [ K ] . Then, for each s ∈ {± 1 } , there exists T s,k ≤ 9 n ηβ |V s,k | such that γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) ≥ 1 for any t > T s,k .

Proof of Lemma C.3. Suppose γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) < 1 for all 0 ≤ t ≤ n 6 ηPσ 2 d d . For each i ∈ V s,k , we have

The first inequality is due to Lemma B.4, the second inequality holds since we can apply Lemma C.2, and the last inequality is due to (A1). Thus, g ( t ) i = 1 1+exp ( y i f W ( t ) ( X i ) ) > 1 9 and we have

Notice that |V s,k | = ρ k n . From the condition in the lemma statement, we have 9 n ηβ |V s,k | = o ( n 6 ηPσ 2 d d ) . If we choose t 0 ∈ [ 9 n ηβ |V s,k | , n 6 ηPσ 2 d d ] , then

and this is contradictory; therefore, it cannot hold that γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) < 1 for all 0 ≤ t ≤ n 6 ηPσ 2 d d . Hence, there exists 0 ≤ T s,k < n 6 ηPσ 2 d d such that γ ( T s,k +1) s ( s, k ) + βγ ( T s,k +1) -s ( s, k ) ≥ 1 and choose the smallest one. Then we obtain

Therefore, T s,k < 9 n ηβ |V s,k | and this is what we desired.

What We Have So Far. For any common feature v s,k with s ∈ {± 1 } and k ∈ K C , it satisfies ρ k = w ( σ 2 d d βn ) due to (A4). By Lemma C.3, at any iterate t ∈ [ ¯ T 1 , T ∗ ] with ¯ T 1 := max s ∈{± 1 } ,k ∈K C T s,k , the following properties hold if the event E init occurs:

(Learn common features): For any s ∈ {± 1 } and k ∈ K C ,

For any s ∈ {± 1 } , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , ρ ( t ) s ( i, p ) = ˜ O ( β -1 ) .

C.2.3 Overfitting (extremely) Rare Data

In the previous step, we have shown that common data can be well-classified by learning common features. In this step, we will show that the model correctly classifies (extremely) rare data by overfitting dominant noise instead of learning its features.

We first introduce lower bounds on the number of iterates such that feature coefficients γ ( t ) s ( s ′ , k ) remain small, up to the order of α 2 β -1 . This lemma holds for any kind of features, but we will focus on (extremely) rare features. This does not contradict the results from Section C.2.2 for common features since the upper bound on the number of iterations in Lemma C.3 is larger than the lower bound on the number of iterations in this lemma.

Lemma C.4. Suppose the event E init occurs. For each s ∈ {± 1 } and k ∈ [ K ] , there exists ˜ T s,k > nα 2 ηβ |V s,k | such that γ ( t ) s ′ ( s, k ) ≤ α 2 β -1 for any 0 ≤ t < ˜ T s,k and s ′ ∈ {± 1 } .

Proof of Lemma C.4. Let ˜ T s,k be the smallest iterate such that γ ( ˜ T s,k ) s ′ ( s, k ) > α 2 β -1 for some s ′ ∈ {± 1 } . We assume the existence of ˜ T s,k , as its absence would directly lead to our conclusion. For any 0 ≤ t < ˜ T s,k ,

and we have

We conclude ˜ T s,k > nα 2 ηβ |V s,k |

which is the desired result.

Next, we will show that the model overfits (extremely) rare data by memorizing dominant noise patches in at least constant order within ˜ T s,k iterates.

Lemma C.5. Suppose the event E init occurs and ρ k = o ( α 2 σ 2 d d n ) . Then, for each i ∈ V s,k , there exists T i ∈ [ ¯ T 1 , 18 n ηβσ 2 d d ] such that

for any t > T i .

Proof of Lemma C.5. Suppose ∑ p ∈ [ P ] \{ p ∗ i } ( ρ ( t ) s ( i, p ) + βρ ( t ) -s ( i, p ) ) < 1 for 0 ≤ t ≤ nα 2 ηβ |V s,k | . From Lemma B.4 and Lemma C.4, we have

where the last inequality is due to (10). Thus, we have g ( t ) i = 1 1+exp ( y i f W ( t ) ( X i ) ) ≥ 1 9 . Also,

where the last inequality is due to ∥ ∥ ∥ ξ (˜ p i ) i ∥ ∥ ∥ 2 ≥ 1 2 σ 2 d d and ϕ ′ ≥ β .

Notice that |V s,k | = ρ k n . From the given condition in the lemma statement, we have 18 n ηβσ 2 d d = o ( nα 2 ηβ |V s,k | ) . If we choose t 0 ∈ [ 18 n ηβσ 2 d d , nα 2 ηβ |V s,k | ] , then we have

This is a contradiction; therefore it cannot hold that ∑ p ∈ [ P ] \{ p ∗ i } ( ρ ( t ) s ( i, p ) + βρ ( t ) -s ( i, p ) ) < 1 for all 0 ≤ t ≤ nα 2 ηβ |V s,k | . Hence, we can choose the smallest 0 ≤ T i < nα 2 ηβ |V s,k | such that ∑ p ∈ [ P ] \{ p ∗ i } ( ρ ( T i +1) s ( i, p ) + βρ ( T i +1) -s ( i, p ) ) ≥ 1 . For any 0 ≤ t < T i ,

and we conclude that T i ≤ 18 n ηβσ 2 d d .

Lastly, we move on to prove T i > ¯ T 1 . Combining Lemma C.2 and Lemma C.3 leads to

Thus, we have T i > ¯ T 1 and this is what we desired.

What We Have So Far. For any k ∈ K R ∪ K E , it satisfies ρ k = o ( α 2 σ 2 d d n ) due to (A5). By Lemma C.5 at iterate t ∈ [ T ERM , T ∗ ] with

the following properties hold if the event E init occurs:

(Learn common features): For s ∈ {± 1 } and k ∈ K C ,

(Overfit (extremely) rare data): For any s ∈ {± 1 } , k ∈ K R ∪ K E , and i ∈ V s,k ,

(Do not learn (extremely) rare features at T ERM ): For any s, s ′ ∈ {± 1 } and k ∈ K R ∪ K E , γ ( T ERM ) s ′ ( s, k ) ≤ α 2 β -1 .

For any s ∈ {± 1 } , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , ρ ( t ) s ( i, p ) = ˜ O ( β -1 ) .

C.2.4 ERM cannot Learn (extremely) Rare Features Within Polynomial Times

In this step, we will show that ERM cannot learn (extremely) rare features within the maximum admissible iterations T ∗ = poly( d ) η .

From now on, we fix any s ∗ ∈ {± 1 } and k ∗ ∈ K R ∪ K E . Recall that we defined the set W and the function Q ( s ∗ ,k ∗ ) : W → R d × 2 in Lemma B.5. Let us omit superscripts for simplicity. For each iteration t , Q ( W ( t ) ) represents the cumulative updates contributed by data points with feature vector v s ∗ ,k ∗ until t -th iteration. We will sequentially introduce several technical lemmas and by combining these lemmas, quantify update by data with feature vector v s ∗ ,k ∗ after T ERM and derive our conclusion.

Let us define W ∗ = { w ∗ 1 , w ∗ -1 } , where

for each s ∈ {± 1 } with M = 4 β -1 log ( 2 ηβ 2 T ∗ α 2 ) . Note that (12), β < 1 , and T ∗ = poly( d ) η together imply M = ˜ O ( β -1 ) . Note that W ( t ) , W ∗ ∈ W for any t ≥ 0 .

Lemma C.6. Suppose the event E init occurs. Then,

where ∥·∥ denotes the Frobenius norm.

Proof of Lemma C.6. For each s ∈ {± 1 } ,

and we have

̸

From the event E init defined in Lemma B.2 and (A2), we have

̸

In addition, we have

where the first inequality is due to α < 1 and the second inequality is due to ∑ i ∈V s ∗ ,k ∗ ∥ ∥ ∥ ξ (˜ p i ) i ∥ ∥ ∥ -2 ≤ 2 |V s ∗ ,k ∗ | σ -2 d d -1 < 1 from (A5). Hence, from E init , we obtain

Lemma C.7. Suppose the E init occurs. For any t ≥ T ERM and i ∈ V s ∗ ,k ∗ , it holds that

Proof of Lemma C.7. We have

For any s ∈ {± 1 } and p ∈ [ P ] \ { p ∗ i , ˜ p i } ,

∗

ss

Q

(

p

s

(

w

∗

s

)

, ξ

i

)

〉

̸

( -1 )

̸

where the first inequality is due to the definition of Q and the second-to-last line is due to (9) and (A7).

Hence, applying (16) and (17) for s = s ∗ , -s ∗ and combining with ϕ ′ ≥ β , we have

By combining Lemma C.6 and Lemma C.7, we can obtain the following result.

Lemma C.8. Suppose the event E init occurs.

where ∥·∥ denotes the Frobenius norm.

Proof of Lemma C.8. Note that for any T ERM ≤ t < T ∗ ,

and thus

where the last inequality is due to Lemma C.7. By the chain rule, we have

For each s ∈ {± 1 } , i ∈ V s ∗ ,k ∗ , and p ∈ [ P ] ,

where the last inequality is due to Lemma C.1 and the event E init . By Lemma F.1,

where the last equality is due to r = o ( 1 polylog( d ) ) . Therefore, we have

From the convexity of ℓ ( · ) ,

where the last inequality is due to (A8), and we have

From telescoping summation, we have

Finally, we can prove that the model cannot learn (extremely) rare features within T ∗ iterations.

Lemma C.9. Suppose the event E init occurs. For any T ∈ [ T ERM , T ∗ ] , we have γ ( T ) s ( s ∗ , k ∗ ) = ˜ O ( α 2 β -2 ) for each s ∈ {± 1 } .

Proof of Lemma C.9. For any T ∈ [ T ERM , T ∗ ] , we have

where the first inequality is due to ϕ ′ ≤ 1 and the second inequality is due to -ℓ ′ ≤ ℓ . From the result of Section C.2.3 we know γ ( T ERM ) s ( s ∗ , k ∗ ) ≤ α 2 β -1 . Additionally, by Lemma C.8 and Lemma C.6, we have

The last line is due to (A5) and our choice M = 4 β -1 log ( 2 ηβ 2 T ∗ α 2 ) . Thus, we have our conclusion.

What We Have So Far. Suppose the event E init occurs. For any t ∈ [ T ERM , T ∗ ] , we have

(Learn common features): For each s ∈ {± 1 } and k ∈ K C ,

(Overfit (extremely) rare data): For each s ∈ {± 1 } , k ∈ K R ∪ K E and i ∈ V s,k ,

(Cannot learn (extremely) rare features): For each s ∈ {± 1 } and k ∈ K R ∪ K E ,

For any s ∈ {± 1 } , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , ρ ( t ) s ( i, p ) = ˜ O ( β -1 ) ,

C.2.5 Train and Test Accuracy

In this step, we will prove that the model trained by ERM has perfect training accuracy but has near-random guesses on (extremely) rare data.

For any i ∈ V s,k with s ∈ {± 1 } and k ∈ K C , by Lemma B.4, we have

for any t ∈ [ T ERM , T ∗ ] . In addition, for any i ∈ V s,k with s ∈ {± 1 } and k ∈ K R ∪ K E , we have y i f W ( t ) ( X i )

for any t ∈ [ T ERM , T ∗ ] . We can conclude that ERM with t ∈ [ T ERM , T ∗ ] iterates achieve perfect training accuracy.

Next, let us move on to the test accuracy part. Let ( X , y ) ∼ D be a test data with X = ( x (1) , . . . , x ( P ) ) ∈ R d × P having feature patch index p ∗ , dominant noise patch index ˜ p , and feature vector v y,k . We have x ( p ) ∼ N ( 0 , σ 2 b Λ ) for each p ∈ [ P ] \ { p ∗ , ˜ p } and x (˜ p ) -α v s, 1 ∼ N ( 0 , σ 2 d Λ ) for some s ∈ {± 1 } . Therefore, for all t ∈ [ T ERM , T ∗ ] and p ∈ [ P ] \ { p ∗ , ˜ p } ,

with probability at least 1 -o ( 1 poly( d ) ) due to Lemma B.2, (A8), (8), and (9). In addition, for any s ′ ∈ {± 1 } , we have

with probability at least 1 -o ( 1 poly( d ) ) due to Lemma B.2, (A8), (8), and (9).

Case 1: k ∈ K C

with probability at least 1 -o ( 1 poly( d ) ) . Suppose (18) and (20) holds. By Lemma B.4, we have

Therefore, we have

In addition,

where the second equality is due to Lemma B.4 and (A8). If (19) holds, we have

Note that

and

where the first inequality is due to (18), second-to-last line is due to (A8), (8) and (10), and the last inequality is due to (22). Therefore, we have yf W ( t ) ( X ) > 0 if y = s . Otherwise, yf W ( t ) ( X ) < 0 . Therefore, we have

Hence, combining (21) and (23) implies

D Proof for Cutout

In this section, we use g ( t ) i, C := 1 1+exp ( y i f W ( t ) ( X i, C ) ) for each data i , C ⊂ [ P ] with |C| = C and iteration t , for simplicity.

D.1 Proof of Lemma B.3 for Cutout

For s ∈ {± 1 } and iterate t ,

and we have and

starting from γ (0) s ( s ′ , k ) = ρ (0) s ( i, p ) = 0 for each s, s ′ ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] and p ∈ [ P ] \{ p ∗ i } , then we have

for each s ∈ {± 1 } . Furthermore, γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's are monotone increasing. □

D.2 Proof of Theorem 3.2

To show Theorem 3.2, we present a structured proof comprising the following five steps:

Establish upper bounds on γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's to apply Lemma B.4 (Section D.2.1).

Demonstrate that the model quickly learns common and rare features (Section D.2.2).

Show that the model overfits augmented data if it does not contain common or rare features (Section D.2.3).

Confirm the persistence of this tendency until T ∗ iterates (Section D.2.4).

Characterize train accuracy and test accuracy (Section D.2.5).

D.2.1 Bounds on the Coefficients in Feature Noise Decomposition

The following lemma provides upper bounds on Lemma B.3 during T ∗ iterations.

Lemma D.1. Suppose the event E init occurs. For any 0 ≤ t ≤ T ∗ , we have

Proof of Lemma D.1. The first argument implies the second argument since log( ηT ∗ ) = polylog( d ) and

We will prove the first argument by using induction on t . The initial case t = 0 is trivial. Suppose the statement holds at t = T and consider the case t = T +1 .

Let ˜ T s,k ≤ T denote the smallest iteration where γ ( ˜ T s,k +1) s ( s, k ) + βγ ( ˜ T s,k +1) -s ( s, k ) > 2 log( ηT ∗ ) . We assume the existence of ˜ T s,k , as its absence would directly lead to our desired conclusion; to see why, note that the following holds, due to (24) and (11):

By (24), we have

The inequality is due to γ ( ˜ T s,k ) s ( s, k ) + βγ ( ˜ T s,k ) -s ( s, k ) ≤ 2 log( ηT ∗ ) and

from our choice of ˜ T s,k and η .

The first inequality is due to Lemma B.4 and the second inequality holds due to (A7), (8), and our choice of t , γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) ≥ 2 log( ηT ∗ ) .

Hence, we obtain

where the last inequality holds for any reasonably large T ∗ . Merging all inequalities together, we have γ ( T +1) s ( s, k ) + βγ ( T +1) -s ( s, k ) ≤ 4 log( ηT ∗ ) .

Next, we will follow similar arguments to show that

for each i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } .

Let ˜ T ( p ) i ≤ T be the smallest iteration such that ρ ( ˜ T ( p ) i +1) y i ( i, p ) + βρ ( ˜ T ( p ) i +1) -y i ( i, p ) > 2 log( ηT ∗ ) . We assume the existence of ˜ T ( p ) i , , as its absence would directly lead to our desired conclusion; to see why, note that the following holds, due to (25) and (11):

where the first inequality is due to ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ ≤ 3 2 σ 2 d d and (A4), and the last inequality is due to (11). Now we suppose there exists such ˜ T i ≤ T . By (25), we have

The inequality is due to ρ ( t ) y i ( i, p ) + βρ ( t ) -y i ( i, p ) ≤ 2 log( ηT ∗ ) our choice of ˜ T ( p ) i and η n E C∼D C [ g ( ˜ T ( p ) i ) i, C · ✶ p/ ∈C ]( ϕ ′ (〈 w ( ˜ T ( p ) i ) s , x ( p ) i 〉) + βϕ ′ (〈 w ( ˜ T ( p ) i ) -s , x ( p ) i 〉)) ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 2 η ≤ log( ηT ∗ ) , from ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 d d , (A4), and (11).

The first inequality is due to Lemma B.4 and the second inequality holds since from our choice of t , ρ ( t ) y i ( i, p ) + βρ ( t ) -y i ( i, p ) ≥ 2 log( ηT ∗ ) .

Therefore, we have

where the first inequality is due to ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 d d and (A4). Hence, we conclude ρ ( T +1) y i ( i, p ) + βρ ( T +1) -y i ( i, p ) ≤ 4 log( ηT ∗ ) .

D.2.2 Learning Common Features and Rare Features

In the initial stages of training, the model quickly learns common features while exhibiting minimal overfitting to Gaussian noise.

̸

First, we establish lower bounds on the number of iterations, ensuring that background noise coefficients ρ ( t ) s ( i, p ) for p = p ∗ i , ˜ p i remain small, up to the order of 1 P .

Lemma D.2. Suppose the event E init occurs. There exists ˜ T > n 6 ηPσ 2 b d such that ρ ( t ) s ( i, p ) ≤ 1 4 P for all 0 ≤ t < ˜ T, s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i , ˜ p i } .

Proof of Lemma D.2. Let ˜ T be the smallest iteration such that ρ ( ˜ T ) s ( i, p ) ≥ 1 4 P for some s ∈ {± 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } . We assume the existence of ˜ T , as its absence would directly lead to our conclusion. Then, for any 0 ≤ t < ˜ T , we have

where the inequality is due to g ( t ) i, C < 1 , ϕ ′ ≤ 1 , and ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≤ 3 2 σ 2 b d . Hence, we have

and we conclude ˜ T > n 6 ηPσ 2 d b which is the desired result.

Next, we will show that the model learns common features in at least constant order within ˜ T iterates. Lemma D.3. Suppose the event E init occurs and ρ k = ω ( σ 2 b d βn ) for some k ∈ [ K ] . Then, for each s ∈ {± 1 } . there exists T s,k ≤ 9 nP ηβ |V s,k | such that γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) ≥ 1 for any t > T s,k .

Proof of Lemma D.3. Suppose γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) < 1 for all 0 ≤ t ≤ n 6 ηPσ 2 b d . For each i ∈ V s,k and C ⊂ [ P ] with |C| = C such that p ∗ i / ∈ C and ˜ p i ∈ C , we have

The first inequality is due to Lemma B.4, the second inequality holds since we can apply Lemma D.2, and the last inequality is due to (A1). Thus, g ( t ) i, C = 1 1+exp ( y i f W ( t ) ( X i, C ) ) > 1 9 and we have

From the given condition in the lemma statement, we have 9 nP ηβ |V s,k | = o ( n 6 ηPσ 2 b d ) . If we choose t 0 ∈ [ 9 nP ηβ |V s,k | , n 6 ηPσ 2 b d ] , then

and this is contradictory; therefore, it cannot hold that γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) < 1 for all 0 ≤ t ≤ n 6 ηPσ 2 b d . Hence, there exists 0 ≤ T s,k < n 6 ηPσ 2 b d such that γ ( T s,k +1) s ( s, k ) + βγ ( T s,k +1) -s ( s, k ) ≥ 1 and choose the smallest one. Then we obtain

Therefore, T s,k ≤ 9 nP ηβ |V s,k | and this is what we desired.

What We Have So Far. For any common feature or rare feature v s,k with s ∈ {± 1 } and k ∈ K C ∪ K R , it satisfies ρ k = ω ( σ 2 b d βn ) due to (A5). By Lemma D.3, at any iterate t ∈ [ ¯ T 1 , T ∗ ] with ¯ T 1 := max s ∈{± 1 } ,k ∈C T s,k , the following properties hold if the event E init occurs:

(Learn common/rare features): For s ∈ {± 1 } and k ∈ K C ∪K R , γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) = Ω(1) ,

For any s ∈ {± 1 } , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , ρ ( t ) s ( i, p ) = ˜ O ( β -1 ) .

D.2.3 Overfitting Augmented Data

In the previous step, we have shown that data containing common or rare features can be wellclassified by learning common and rare features. In this step, we will show that the model correctly classifies the remaining training data by overfitting background noise instead of learning its features.

We first introduce lower bounds on the number of iterates such that feature coefficients γ ( t ) s ( s ′ , k ) remain small, up to the order of α 2 β -1 . This lemma holds to any kind of features, but we will focus on extremely rare features. This does not contradict the results from Section D.2.2 for common features and rare features since the upper bound on the number of iterations in Lemma D.3 is larger than the lower bound on the number of iterations in this lemma.

Lemma D.4. Suppose the event E init occurs. For each s ∈ {± 1 } and k ∈ [ K ] , there exists ˜ T s,k ≥ nα 2 ηβ |V s,k | such that γ ( t ) s ′ ( s, k ) ≤ α 2 β -1 for any 0 ≤ t < ˜ T s,k and s ′ ∈ {± 1 } .

Proo of Lemma D.4. Let ˜ T s,k be the smallest iterate such that γ ( t ) s ′ ( s, k ) > α 2 β -1 for some s ′ ∈ {± 1 } . We assume the existence of ˜ T s,k , as its absence would directly lead to our conclusion.

For any 0 ≤ t < ˜ T s,k ,

and we have

α 2 β -1 ≤ γ ( ˜ T s,k ) s ′ ( s, k ) = ˜ T s,k -1 ∑ t =0 ( γ ( t +1) s ′ ( s, k ) -γ ( t ) s ′ ( s, k ) ) ≤ η |V s,k | n ˜ T s,k . We conclude ˜ T s,k ≥ nα 2 ηβ |V s,k | which is the desired result.

Next, we will show that the model overfits data augmented not containing common or rare features in at least constant order within ˜ T s,k iterates.

LemmaD.5. Suppose the event E init occurs and ρ k = o ( α 2 σ 2 b d n ) . For each i ∈ [ n ] and C ⊂ [ P ] with |C| = C , if (1) i ∈ V y i ,k and p ∗ i / ∈ C or (2) i ∈ [ n ] and p ∗ i ∈ C , then there exists T i, C ∈ [ ¯ T 1 , 18 n ( P C ) ηβσ 2 b d ] such that for any t > T i, C .

Proof of Lemma D.5. We can address both cases in the statement simultaneously. Suppose ∑ p/ ∈C∪{ p ∗ i } ( ρ ( t ) y i ( i, p ) + βρ ( t ) -y i ( i, p ) ) < 1 for all 0 ≤ t ≤ nα 2 ηβ |V y i ,k | .

From Lemma B.4 and Lemma D.4, we have

where the last inequality is due to ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 ≥ 1 2 σ 2 b d and ϕ ′ ≥ β . We also have

From the given condition in the lemma statement, we have 18 n ( P C ) ηβσ 2 b d = o ( nα 2 ηβ |V s,k | ) . If we choose t 0 ∈ [ 18 n ( P C ) ηβσ 2 b d , nα 2 ηβ |V s,k | ] , then we have

and this is a contradiction; therefore, it cannot hold that ∑ p/ ∈C∪{ p ∗ i } ( ρ ( t ) y i ( i, p ) + βρ ( t ) -y i ( i, p ) ) < 1 for all 0 ≤ t ≤ nα 2 ηβ |V y i ,k | . Thus, there exists 0 ≤ T i, C < nα 2 ηβ |V s,k | satisfying ∑ p/ ∈C∪{ p ∗ i } ( ρ ( T i, C +1) s ( i, p ) + βρ ( T i, C +1) -s ( i, p ) ) ≥ 1 and let us choose the smallest one.

For any 0 ≤ t < T i, C , we have

and we conclude that T i, C ≤ 18 n ( P C ) ηβσ 2 b d .

Lastly, we move on to prove T i, C > ¯ T 1 . Combining Lemma D.2 and Lemma D.3 leads to

Thus, we have T i, C > ¯ T 1 and this is what we desired.

What We Have So Far. For any k ∈ K E , it satisfies ρ k = o ( α 2 n σ 2 b d ) due to (A6). By Lemma D.5 at iterate t ∈ [ T Cutout , T ∗ ] with

the following properties hold if the event E init occurs:

(Learn common/rare features): For any s ∈ {± 1 } and k ∈ K C ∪ K R ,

(Overfit augmented data with extremely rare features or no feature): For each i ∈ [ n ] , k ∈ K E , C ⊂ [ P ] with |C| = C such that (1) i ∈ V y i ,k and p ∗ i / ∈ C or (2) i ∈ [ n ] and p ∗ i ∈ C

(Do not learn extremely rare features at T Cutout ): For any s, s ′ ∈ {± 1 } and k ∈ K E ,

For any s ∈ {± 1 } , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , ρ ( t ) s ( i, p ) = ˜ O ( β -1 ) .

D.2.4 Cutout cannot Learn Extremely Rare Features Within Polynomial Times

In this step, We will show that Cutout cannot learn extremely rare features within the maximum admissible iterate T ∗ = poly( d ) η .

we fix any s ∗ ∈ {± 1 } and k ∗ ∈ K E . Recall the function Q ( s ∗ ,k ∗ ) : W → R d × 2 , defined in Lemma B.5 and omit superscripts for simplicity. For each iteration t , Q ( W ( t ) ) represents quantities updates by data with feature vector v s ∗ ,k ∗ until t -th iteration. We will sequentially introduce several technical lemmas and by combining these lemmas, quantify update by data with feature vector v s ∗ ,k ∗ after T Cutout and derive our conclusion.

Let us define W ∗ = { w ∗ 1 , w ∗ -1 } , where

where M = 4 β -1 log ( 2 ηβ 2 T ∗ α 2 ) . Note that (12), β < 1 , and T ∗ = poly( d ) η together imply M = ˜ O ( β -1 ) . Note that W ( t ) , W ∗ ∈ W for any t ≥ 0 .

Lemma D.6. Suppose the event E init occurs. Then,

where ∥·∥ denotes the Frobenius norm.

Proof of Lemma D.6. For each s ∈ {± 1 } ,

̸

and we have

From E init and (A2), we have

̸

From the event E init defined in Lemma B.2, we have

and we obtain

Lemma D.7. Suppose the E init occurs. For any t ≥ T Cutout , i ∈ V s ∗ ,k ∗ and any C ⊂ [ P ] with |C| = C , it holds that

Proof of Lemma D.7. We have

For any

̸

where the last equality is due to (9). Also, for any s ∈ {± 1 } , ss ∗ ⟨ Q ( w ∗ ) , v ⟩ =

̸

where the last equality is due to (9) and (A7).

For any C ⊂ [ P ] with |C| = C , there exists p ∈ [ P ] \ { p ∗ i , ˜ p i } such that p = C since C < P 2 . By applying (26) and (27) for s = s ∗ , -s ∗ and combining with ϕ ′ ≥ β , we have

̸

̸

By combining Lemma D.6 and Lemma D.7, we can obtain the following result.

Lemma D.8. Suppose the event E init occurs.

where ∥·∥ denotes the Frobenius norm.

Proof of Lemma D.8. Note that for any T Cutout ≤ t < T ∗ ,

Therefore, we have

where the last inequality is due to Lemma D.7. By the chain rule, for each C ⊂ [ P ] with |C| = C , we have

〈 ∇ W ∑ i ∈V s ∗ ,k ∗ ℓ ( y i f W ( t ) ( X i, C )) , Q ( W ( t ) ) 〉 = ∑ i ∈V s ∗ ,k ∗ [ ℓ ′ ( y i f W ( t ) ( X i, C )) × ∑ p/ ∈C ( ϕ ′ (〈 w ( t ) s ∗ , x ( p ) i 〉)〈 Q s ∗ ( w ( t ) s ∗ ) , x ( p ) i 〉 -ϕ ′ (〈 w ( t ) -s ∗ , x ( p ) i 〉)〈 Q -s ∗ ( w ( t ) -s ∗ ) , x ( p ) i 〉 ) ] . For each s ∈ {± 1 } , i ∈ V s ∗ ,k ∗ , and p ∈ [ P ] , ∣ ∣ ∣ 〈 w ( t ) s , x ( p ) i 〉 -〈 Q s ( w ( t ) s ) , x ( p ) i 〉∣ ∣ ∣ = ∣ ∣ ∣ 〈 w ( t ) s -Q s ( w ( t ) s ) , x ( p ) i 〉∣ ∣ ∣

where the last inequality is due to Lemma D.1 and the event E init . By Lemma F.1,

where the last equality is due to r = o ( 1 polylog( d ) ) . Therefore, we have

From the convexity of ℓ ( · ) ,

where the last inequality is due to (A8), and we have

From telescoping summation, we have

Finally, we can prove that the model cannot learn extremely rare features within T ∗ iterations. Lemma D.9. Suppose the event E init occurs. For any T ∈ [ T Cutout , T ∗ ] , we have γ ( T ) s ( s ∗ , k ∗ ˜ O ( α 2 β -2 ) for each s ∈ {± 1 } .

) =

where the first inequality is due to ϕ ′ ≤ 1 and the second inequality is due to -ℓ ′ ≤ ℓ . From the result of Section D.2.3, γ ( T Cutout ) s ( s ∗ , k ∗ ) ≤ α 2 β -1 and by Lemma D.8 and Lemma D.6, we have

The last line is due to (A6) and M = 4 β -1 log ( 2 ηβ 2 T ∗ α 2 )

. This finishes the proof.

What We Have So Far. Suppose the event E init occurs. For any t ∈ [ T Cutout , T ∗ ] , we have

(Learn common/rare features): γ ( t ) s ( s, k ) + βγ ( t ) -s ( s, k ) = Ω(1) for each s ∈ {± 1 } and k ∈ K C ∪ K R

(Overfit augmented data with extremely rare features or no feature): For each i ∈ [ n ] , k ∈ K E , C ⊂ [ P ] with |C| = C such that (1) i ∈ V y i ,k and p ∗ i / ∈ C or (2) i ∈ [ n ] and p ∗ i ∈ C

(Cannot learn extreme features): γ ( t ) s ( s, k ) , γ ( t ) -s ( s, k ) = O ( α 2 β -2 ) for each s ∈ {± 1 } and k ∈ K E .

For any s ∈ {± 1 } , i ∈ [ n ] , and p ∈ [ P ] \ { p ∗ i } , ρ ( t ) s ( i, p ) = ˜ O ( β -1 ) ,

D.2.5 Train and Test Accuracy

In this step, we will prove that the model trained by Cutout has perfect training accuracy on both augmented data and original data but has near-random guesses on test data with extremely rare data.

for any t ∈ [ T Cutout , T ∗ ] . In addition, for any i ∈ [ n ] and C ⊂ [ P ] with |C| = C that does not correspond to the case above, by Lemma D.5 and Lemma B.4, we have

for any t ∈ [ T Cutout , T ∗ ] . We can conclude that Cutout with t ∈ [ T Cutout , T ∗ ] iterates achieve perfect training accuracy on augmented data.

Next, we will show that Cutout achieves perfect training accuracy on the original data. For any i ∈ [ n ] , let us choose C ⊂ [ P ] with |C| = C such that p ∗ i ∈ C . Then, from the result above, we have

for any t ∈ [ T Cutout , T ∗ ] and we conclude that Cutout with t ∈ [ T Cutout , T ∗ ] iterates achieve perfect training accuracy on original data.

Lastly, let us move on to the test accuracy part. Let ( X , y ) ∼ D be a test data with X = ( x (1) , . . . , x ( P ) ) ∈ R d × P having feature patch p ∗ , dominant noise patch ˜ p , and feature vector v y,k . We have x ( p ) ∼ N ( 0 , σ 2 b Λ ) for each p ∈ [ P ] \ { p ∗ , ˜ p } and x (˜ p ) -α v s, 1 ∼ N ( 0 , σ 2 d Λ ) for some s ∈ {± 1 } . Therefore, for all t ∈ [ T Cutout , T ∗ ] and p ∈ [ P ] \ { p ∗ , ˜ p } ,

with probability at least 1 -o ( 1 poly( d ) ) due to Lemma B.2, (A8), (8), and (9).. In addition, for any s ′ ∈ {± 1 } , we have

with probability at least 1 -o ( 1 poly( d ) ) due to Lemma B.2, (A8), (8), and (9).

Case 1: k ∈ K C ∪ K R

with probability at least 1 -o 1 . Suppose (28) and (30) holds. By Lemma B.4, we have

( poly( d ) )

Therefore, we have

Case 2: k ∈ K E

By triangular inequality and ϕ ′ ≤ 1 , we have

In addition,

where the second equality is due to Lemma B.4 and (A8). If (29) holds, we have

Note that

and

> 0 .

where the first inequality is due to (28), the second-to-last line is due to (A8), (8), and (10) , and the last inequality is due to (32). Therefore, we have yf W ( t ) ( X ) > 0 if y = s . Otherwise, yf W ( t ) ( X ) < 0 .

Hence, combining (31) and (33) implies

E Proof for CutMix

E.1 Proof of Lemma B.3 for CutMix

For each i, j ∈ [ n ] and S ⊂ [ P ]

For

Hence, if we define γ ( t ) s ( s ′ , k ) 's and ρ ( t ) s ( i, p ) 's recursively by using the rule

starting from γ (0) s ( s ′ k ) = ρ (0) s ( i, p ) = 0 for each s, s ′ ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } , then we have

for each s ∈ {± 1 .

E.2 Proof of Theorem 3.3

We will prove that the conclusion of Theorem 3.3 holds when the event E init occurs. The proof of Theorem 3.3 is structured into the following six steps:

Introduce a reparametrization of the CutMix loss L CutMix ( W ) to a convex function h ( Z ) for ease of analysis (Section E.2.1).

Characterize a global minimum of h ( Z ) (Section E.2.2).

Evaluate strong convexity constant in the region near the global minimum of h ( Z ) (Section E.2.3).

Show that near stationary point of h ( Z ) is close to a global minimum (Section E.2.4).

Prove that gradient descent on the CutMix loss L CutMix ( W ) achieves a near-stationary point of the reparametrized function h ( Z ) and perfect accuracy on original training data (Section E.2.5).

Evaluate the test accuracy of a model in near-stationary point (Section E.2.6).

E.2.1 Reparametrization of CutMix Loss

It is complicated to characterize the stationary points of CutMix loss L CutMix ( W ) due to its nonconvexity. We will overcome this problem by introducing reparameterization of the objective function. Let us define for i ∈ [ n ] , p ∈ [ P ] and

for each s ∈ {± 1 } , k ∈ [ K ] . We can rewrite CutMix loss L CutMix ( W ) as a function h ( Z ) of the defined variables Z := { z s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { z ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } as follows.

where we write z ( p ∗ i ) i = z s,k if i ∈ V s,k . For notational simplicity, let us consider Z as vectors in R 2 K + n ( P -1) with the standard orthonormal basis { e s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { e ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } which means

If there is no confusion, we will use e ( p ∗ i ) i to represent e s,k , for i ∈ V s,k . By the chain rule,

where each column of Jacobian matrix J ( W ) ∈ R 2 d × ( n ( P -1)+2 K ) is

Let us characterize the smallest singular value σ min ( J ( W )) of the Jacobian matrix J ( W ) . For any unit vector c = { c s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { c ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } ∈ R 2 K + n ( P -1) , we have

̸

̸

̸

For each s 1 , s 2 ∈ {± 1 } , k 1 , k 2 ∈ [ K ] such that ( s 1 , k 1 ) = ( s 2 , k 2 ) , and i ∈ [ n ] , p ∈ [ P ] \ { p ∗ i , ˜ p i } ,

and if k 1 > 1

since ⟨ v s 1 ,k 1 , v s 2 ,k 2 ⟩ = 〈 v s 1 ,k 1 , ξ ( p ) i 〉 = 〈 v s 1 ,k 1 , ξ (˜ p i ) i 〉 = 0 . Also, for each s ∈ {± 1 } and i ∈ F s , then

where the last inequality holds since

where we apply the fact from the event E init defined in Lemma B.2 and (A7). Also,

Furthermore, for each i, j ∈ [ n ] , p ∈ [ P ] \ { p ∗ i } , q ∈ [ P ] \ { p ∗ j } with ( i, p ) = ( j, q ) satisfies

̸

where the last inequality is due to AM-GM inequality and

which we show through a case analysis. For the case p = ˜ p i and q = ˜ p j , this inequality holds since

where the second inequality is due to

which is implied by the fact from the event E init defined in Lemma B.2, (A1), (A2), and (A7). In the remaining case,

where the second inequality is due to the fact from event E init defined in Lemma B.2, (A1), and (A2). For s ∈ {± 1 } , k ∈ [ K ] and i ∈ [ n ] , p ∈ [ P ] \ { p ∗ i } ,

and

where the last inequality is due to (8). By merging all inequalities together, we have

̸

̸

and we conclude σ min ( J ( W )) ≥ β 2 for any W .

E.2.2 Characterization of a Global Minimum of CutMix Loss

In this section, we will check that h ( Z ) is strictly convex and it has a global minimum.

For each i, j ∈ [ n ] and S ⊂ [ P ] let us define a i,j, S ∈ R 2 K + n ( P -1) as

and then

Since ℓ ( · ) is convex, h ( Z ) is also convex. Note that

and

where the last equality holds since ℓ ′′ ( z ) = ℓ ′′ ( -z ) for any z ∈ R . From the equation above, it suffices to show that { a i,j, S } i,j ∈ [ n ] , S⊂ [ P ] spans R 2 K + n ( P -1) to show strict convexity of h ( Z ) .

We define a function I : [ P ] → [ n ] such that for each p ∈ [ P ] , p ∗ I ( p ) = p with x ( p ) I ( p ) = v 1 , 1 , where the existence is guaranteed by Lemma B.2 (but not necessarily unique). Then for any i ∈ [ n ] and p ∈ [ p ] , we have

Hence, { a i,j, S } i,j ∈ [ n ] , S⊂ [ P ] spans R 2 K + n ( P -1) and h ( Z ) is strictly convex. Thus, it can have at most one global minimum. We want to show the existence of the global minimum and characterize it.

n

2

∇

h

(

Z

)

We can simplify terms as

Hence, if

for all i ∈ [ n ] and p ∈ [ P ] , then we have ∇ h ( Z ) = 0 . Let us consider a specific Z parameterized by z 1 , z -1 , of the form z ( p ) i = y i z y i for all i ∈ [ n ] and p ∈ [ P ] . We will find a stationary point with this specific form and then it should be the unique global minimum in the entire domain. Then for each i ∈ [ n ] and p ∈ [ P ] , we have

From Lemma F.4, there exists a unique minimizer ˆ Z = { ˆ z s,k } s ∈{± 1 } ,k ∈ [ K ] ∪ { ˆ z ( p ) i } i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } of h ( Z ) and it satisfies s ˆ z s,k = z ∗ s = Θ(1) for all k ∈ [ K ] and y i ˆ z ( p ) i = z ∗ y i = Θ(1) for all i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } due to (A1).

E.2.3 Strong Convexity Near Global Minimum

We will show that h ( Z ) is strongly convex in a set G containing a global minimum ˆ Z where G is defined as follows.

here ∥·∥ ∞ is ℓ ∞ norm. For any Z ∈ G and a unit vector c ∈ R 2 K + n ( P -1) with c = ∑ s ∈{± 1 } ,k ∈ [ K ] c s,k e s,k + ∑ i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } c ( p ) i e ( p ) i , we have

Note that for each i ∈ [ n ] , p ∈ [ P ] , from (34), we have

where we use the notational convention c ( p ∗ i ) i = c s,k for s ∈ {± 1 } , k ∈ [ K ] and i ∈ V s,k . By Cauchy-Schwartz inequality and the fact that P S∼D S [ S = ∅ ] , P S∼D S [ S = { q } ] ≥ 1 P ( P +1) for all q ∈ [ P ] ,

Hence, we have

and we conclude h ( Z ) is µ -strongly convex in G where µ := ℓ ′′ (2 P ∥ ˆ Z ∥ ∞ ) (4 K +2 n ( P -1)) P 2 n 2 . Due to (A1), (A2), and the fact that ∥ ˆ Z ∥ ∞ = Θ(1) , we have µ ≥ 1 poly( d ) .

E.2.4 Near Stationary Points are Close to Global Minimum

In this step, we want to show that near stationary points of h ( Z ) are close to a global minimum ˆ Z .

Lemma E.1. Suppose Z ∈ R 2 K + n ( P -1) satisfies ∥∇ h ( Z ) ∥ < µϵ with some 0 < ϵ < ∥ ˆ Z ∥ ∞ 2 . Then, we have ∥ ∥ ∥ Z -ˆ Z ∥ ∥ ∥ < ϵ .

̸

Proof of Lemma E.1. If Z = ˆ Z , we immediately have our conclusion. We may assume Z = ˆ Z .

Let us define a function g : R → R as g ( t ) = h ( ˆ Z + t ( Z -ˆ Z ) ) . Then g is convex and

Furthermore, for 0 ≤ t ≤ t 0 where t 0 := ∥ ˆ Z ∥ ∞ 2 ∥ Z -ˆ Z ∥ ∞ ,

We can conclude g is µ ∥ ∥ ∥ Z -ˆ Z ∥ ∥ ∥ 2 -strongly convex in [0 , t 0 ] . From strong convexity in [0 , t 0 ] and convexity in R , we have

If t 0 < 1 , we have

and

this is contradictory. Thus, we have t 0 ≥ 1 and Z ∈ G . From the strong convexity of h ( Z ) in G , we have

and we have our conclusion ∥ ∥ ∥ Z -ˆ Z ∥ ∥ ∥ < ϵ .

E.2.5 Gradient Descent Achieves a Near Stationary Point

We will show that L CutMix ( W ) is a smooth function.

Lemma E.2. Suppose the event E init occurs. CutMix Loss L CutMix ( W ) is L -smooth with L = 9 r -1 Pσ 2 d d .

Let ˜ W = { ˜ w 1 , ˜ w -1 } and W = { w 1 , w -1 } be any parameters of the neural network f W . For any i, j ∈ [ n ] and S ⊂ [ P ] ,

Since | ℓ ′ | ≤ 1 ,

and since | ϕ ′ | ≤ 1 ,

In addition, since ϕ is r -1 -smooth,

and since ℓ ′ and ϕ are 1 -Lipschitz, we have

Therefore,

where the last equality is due to (A1) and (A8). In the same way, we can obtain

and

where the last inequality holds since ∥ ∥ ∥ ξ ( p ) i ∥ ∥ ∥ 2 < 3 2 σ 2 d d and α 2 ≤ 3 4 σ 2 d d due to (A7). Hence, L CutMix ( W ) is L -smooth with L := 9 r -1 Pσ 2 d d .

Since our objective function L CutMix ( W ) is L -smooth and η ≤ 1 L due to (A8), descent lemma (see Lemma 3.4 in Bubeck et al. (2015)) implies

and by telescoping sum, we have

for any T > 0 .

Choose ϵ = µβ ∥ Z ∥ ∞ polylog( d ) . Then from (35), there exists T CutMix ≤ poly( d ) η such that

ˆ ∥ ∥ ∥ ∇L CutMix ( W ( T CutMix ) )∥ ∥ ∥ ≤ ϵ.

From characterization of σ min ( J ( W )) in Section E.2.1,

and thus

For sufficiently large d , the RHS becomes smaller than µ · ∥ ˆ Z ∥ ∞ 4 . Then, by Lemma E.1 we have seen in Section E.2.4,

and thus for all i ∈ [ n ] and p ∈ [ P ] , and therefore it reaches perfect training accuracy.

E.2.6 Test Accuracy of Solution Found by Gradient Descent

The final step is showing that W ( T CutMix ) reaches almost perfect test accuracy.

From the results of Section E.2.5, we have

for each s ∈ {± 1 } , k ∈ [ K ] , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } .

For any u > v , by the mean value theorem, we have

Hence, we have

and

for each s ∈ {± 1 } and k ∈ [ K ] . Similarly, for all i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } ,

and

where for each s ∈ {± 1 } ,

and

for each s ∈ {± 1 } , k ∈ [ K ] \ { 1 } , i ∈ [ n ] and p ∈ [ P ] \ { p ∗ i } . If we choose j ∈ [ n ] , q ∈ [ P ] \ { p ∗ j } such that ρ ( j, q ) = max i ∈ [ n ] ,p ∈ [ P ] \{ p ∗ i } ρ ( i, p ) , then we have

̸

From the event E init defined in Lemma B.2, (A8), and (8),

where the inequality holds since 〈 w ( T CutMix ) y j -w ( T CutMix ) -y j , ξ ( q ) j 〉 = Ω(1) . In addition, by triangular inequality, we have

̸

where the last inequality is due to (9). Hence,

Let ( X , y ) ∼ D be a test data with X = ( x (1) , . . . , x ( P ) ) ∈ R d × P having feature patch p ∗ , dominant noise patch ˜ p , and feature vector v y,k . We have x ( p ) ∼ N ( 0 , σ 2 b Λ ) for each p ∈ [ P ] \ { p ∗ , ˜ p } and x (˜ p ) -α v s, 1 ∼ N ( 0 , σ 2 d Λ ) for some s ∈ {± 1 } . Therefore, for all p ∈ [ P ] \ { p ∗ , ˜ p }

with probability at least 1 -o ( 1 poly( d ) ) due to Lemma B.2. In addition,

with probability at least 1 -o ( 1 poly( d ) ) , where the last equality is due to (8), (9), (10), and (A8).

̸

Suppose (36) and (37) holds. Then,

Hence, we have our conclusion.

□

F Technical Lemmas

In this section, we introduce technical lemmas that are used for proving the main theorems. We present their proofs here for better readability.

The following lemma is used in Section C.2.4 and Section D.2.4:

Lemma F.1. For any z, δ ∈ R ,

Proof of Lemma F.1.

and we obtain

The following lemma is used in Section C.2.4.

Lemma F.2. Suppose E init occurs. Then, for any model parameter W = { w 1 , w -1 } , we have

for each s ∈ {± 1 } and k ∈ [ K ] .

Proof of Lemma F.2. For each s ∈ {± 1 } and i ∈ [ n ] , we have

where the inequality is due to the condition from the event E init defined in Lemma B.2 and (A7).. Therefore, for each s ∈ {± 1 } , we have

The first inequality is due to triangular inequality, the third inequality is due to Cauchy-Schwartz inequality and the last inequality is due to 0 ≤ -ℓ ′ ≤ 1 , which can be used to show ( ℓ ′ ) 2 ≤ -ℓ ′ ≤ ℓ . As a result, we have our conclusion:

The following lemma is used in Section D.2.4.

Lemma F.3. Suppose E init occurs. Then, for any model parameter W = { w 1 , w -1 } , we have

for each s ∈ {± 1 } and k ∈ [ K ] .

Proof of Lemma F.3. For each s ∈ {± 1 } , i ∈ [ n ] and C ⊂ [ P ] with |C| = C , we have

where the inequality is due to the condition from the event E init defined in Lemma B.2 and (A7). Therefore, for any s ∈ {± 1 } , we have

The first inequality is due to triangular inequality, the third inequality is due to Cauchy-Schwartz inequality and the last inequality is due to 0 ≤ -ℓ ′ ≤ 1 , which can be used to show ( ℓ ′ ) 2 ≤ -ℓ ′ ≤ ℓ . As a result, we have our conclusion:

The following lemma guarantees the existence and characterizes the minimum of the CutMix loss in Section E.2.2.

Lemma F.4. Suppose the event E init occurs. Let g 1 , g -1 : R × R → R be defined as

for each s ∈ {± 1 } . There exist unique z ∗ 1 , z ∗ -1 > 0 such that g 1 ( z ∗ 1 , z ∗ -1 ) = g -1 ( z ∗ 1 , z ∗ -1 ) = 0 . Furthermore, we have z ∗ 1 , z ∗ -1 = Θ(1) .

Proof of Lemma F.4. For each z 1 > 0 ,

since ℓ ′ ( z ) ≤ -1 2 for any z ≤ 0 and we use 25 52 n ≤ |V 1 | , |V -1 | ≤ 27 52 n from the event E init defined in Lemma B.2. In addition,

where we use 25 52 n ≤ |V 1 | , |V -1 | ≤ 27 52 n from the event E init defined in Lemma B.2 and (A1) for the last inequality.

Since z ↦→ g -1 ( z 1 , z ) is strictly increasing and by intermediate value theorem, there exists S : (0 , ∞ ) → (0 , ∞ ) such that z = S ( z 1 ) is a unique solution of g -1 ( z 1 , z ) = 0 and S ( z 1 ) < Pz 1 +log9 . Note that S is strictly increasing since g -1 ( z 1 , z -1 ) is strictly decreasing with respect to z 1 and strictly increasing with respect to z -1 . Also, if S ( z ) is bounded above, i.e., there exists some U > 0 such that S ( z ) ≤ U for any z > 0 ,

and it is contradictory. Hence, we have lim z →∞ S ( z ) = ∞ .

Let us choose z > 0 such that and thus

We have

Next, we will prove the existence of z ∗ > 0 such that g 1 ( z ∗ , S ( z ∗ )) > 0 . Let us choose ϵ > 0 such that

and note that ϵ = Θ(1) . Since lim z →∞ S ( z ) = ∞ , we can choose z ∗ such that ℓ ′ ( 1 2 min { z ∗ , S ( z ∗ ) } ) = -ϵ 2 . Then, for any t ≥ z ∗ 2 , we have

From the definition of S and (39) with t = PS ( z ∗ ) > 1 2 min { z ∗ , S ( z ∗ ) } , we have

and we have

where the last inequality is due to (39). This is contradictory to (38), especially the first term inside the maximum, and we have S ( z ∗ ) -( P -1) z ∗ < 0 . In addition, if ( P -1) S ( z ∗ ) -z ∗ ≤ 0 , then

and we have

where the last inequality is due to (39). This is contradictory to (38), especially the second term inside the maximum, and we have ( P -1) S ( z ∗ ) -z ∗ > 0 . Note that we have

and since ϵ = Θ(1) in (38), we have z ∗ ≤ 2 P log ( 2 ϵ -1 ) = O (1) .

Thus, we have S ( z ∗ ) -( P -1) z ∗ < 0 < ( P -1) S ( z ∗ ) -z ∗ < PS ( z ∗ ) . One can consider dividing the interval [ S ( z ∗ ) -( P -1) z ∗ , P S ( z ∗ )] into a grid of length z ∗ + S ( z ∗ ) . Then, the interval is equally divided into P -1 sub-intervals and 0 belongs to one of them. In other words, there exists k ∈ [ P -2] such that

and note that if P = 3 , then k = 1 . The rest of the proof is divided into two cases: ( k +1) S ( z ∗ ) -( P -k -1) z ∗ ≥ 1 2 ( z ∗ + S ( z ∗ )) or ( k +1) S ( z ∗ ) -( P -k -1) z ∗ < 1 2 ( z ∗ + S ( z ∗ )) . In both cases, we show that g 1 ( z ∗ , S ( z ∗ )) > 0 .

From (39), we have and

Thus, we have

and we obtain k > P -1 2 since

where the second inequality is due to (40) and the fact that ℓ ′ ≥ -1 , and the last inequality is due to (38), especially the third term inside the maximum. Note that since k ∈ N , k ≥ P 2 .

Note that from (39), we have and

Hence, we obtain

where we use k ≥ P 2 for the last inequality. Therefore, we have

where the last inequality is due to (38), especially the fourth term inside the maximum.

In this case, we have kS ( z ∗ ) -( P -k ) z ∗ ≤ -1 2 ( z ∗ + S ( z ∗ )) . From (39), we have

and

Thus, we have

and we obtain k > P -1 2 since

where the second inequality is due to (40) and the fact that ℓ ′ ( z ) ≥ -1 2 ∀ z ≥ 0 , and the last inequality is due to our (38), especially the third term inside the maximum. Note that since k ∈ N , we have k ≥ P 2 .

Note that from (39), we have and

Hence, we obtain

where we use k ≥ P 2 for the last inequality. Therefore, we have

where the last inequality is due to (38), especially the fourth term inside the maximum.

In both cases, we have g 1 ( z ∗ , S ( z ∗ )) > 0 . By intermediate value theorem, there exist unique z ∗ 1 , z ∗ -1 > 0 such that g 1 ( z ∗ 1 , z ∗ -1 ) = g -1 ( z ∗ 1 , z ∗ -1 ) = 0 . In addition, z ≤ z ∗ 1 ≤ z ∗ and we have z 1 = Θ(1) since z = Ω(1) and z ∗ = O (1) . By using a similar argument, we can show that z ∗ -1 = Θ(1) , and we have our conclusion.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation on our problem setting and theoretical framework in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and a complete proof in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed descriptions of the experimental setting in Section 5 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not provide open access to the data and code since our main focus is theory.

Guidelines: The main focus of this paper is theory.

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed descriptions of the experimental setting in Section 5 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The main focus of this paper is theory.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the type of compute workers used (NVIDIA RTX A6000) in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research address to the NeurIPS Code of Ethics, ensuring ethical conduct throughout the study.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper mainly focuses on establishing a theoretical understanding of existing data augmentation techniques. Thus, there are no direct societal implications arising from the research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve the release of data or models that pose a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper offers a theoretical explanation for the effectiveness of two practically useful algorithms—Cutout and Cutmix—by applying typical feature learning analysis to multi-patched feature and noise data. The authors present negative results for ERM as a comparison and demonstrate positive results for Cutout and Cutmix, including positive margins and near-optimal test error.

STRENGTHS:
This paper provides the first theoretical analysis of two practically useful data augmentation algorithms — Cutout and Cutmix.

WEAKNESSES:
This paper applies a smoothed leaky ReLU activation function in the neural network for technical reason, which differs from the model used in practice.

QUESTIONS:
1. Can the analysis be generalized to non-smooth neural networks? What technical difficulties will you encounter when generalizing the analysis to non-smooth activation functions like ReLU and leaky ReLU?

2. Why is the near convergence result only presented for the Cutmix setting but not for the Cutout setting?

3. Can you provide stronger results about the margin besides its positiveness? What is the order/magnitude of the margin?

4. Why does the result for Cutout hold for any iteration T in an interval, while the result for Cutmix only guarantees the existence of an iteration that satisfies the properties?

LIMITATIONS:
Same as weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work theoretically investigates why patch-based data augmentation methods for image recognition, namely, CutOut and CutMix, improve performance based on the framework by [Zou+ICML23].\nSpecifically, they showed that when a CNN is trained with CutOut and CutMix, it can focus on rare and extremely rare features while ignoring noise. \nContrarily, the model trained with the standard ERM memorizes noise, which results in poorer performance.\nThese findings align with their numerical results.\n\nSTRENGTHS:\n- This work theoretically reveals why patch-based data augmentation, specifically CutOut and CutMix, improves the performance of CNNs.\nThese methods are so far powerful but ad-hoc heuristic, so research towards their theoretical understanding is important.\n- By comparing ERM, Cutout, and CutMix in a unified framework, the authors highlight the superior ability of CutMix that can exploit extremely rare features in data while effectively ignoring noises.\n- The theory well aligns with the numerical experiments on synthetic data.\n\nWEAKNESSES:\n- The important notions on features, i.e., common features, rare features, and extremely rare features, are not properly defined, making the manuscript difficult to read. I think improving the description in Section 2.1 would resolve this issue.\n- The technical contributions against [Zou et al. 23] are not clearly stated in the manuscript.\n\nQUESTIONS:\n- If some features are extremely rare, I think they are less likely to appear in test data. Why learning it improves the performance? In practice, such features are likely to be attributed to systematic mislabeling.\nClearly defining common features, rare features, and extremely rare features may resolve this question.\n\nLIMITATIONS:\n- Although the checklist says the limitations are discussed in Section 2, I could not find the discussion of limitations in the section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper investigates the effectiveness of patch-level data augmentation techniques, specifically Cutout and CutMix, in training two-layer neural networks. \n- The study compares vanilla training, Cutout training, and CutMix training using a feature-noise data model. \n- The findings show that Cutout can learn low-frequency features missed by vanilla training, while CutMix can capture even rarer features, resulting in the highest test accuracy.\n- The analysis reveals that CutMix enables the network to learn features and noise vectors evenly, offering new insights into patch-level augmentation.\n\nSTRENGTHS:\n- This paper provides a theoretical analysis of how Cutout and CutMix (i.e., patch-level data augmentation) \bimprove feature learning. \n- It specifically explains what features each method learns during the training process, offering deep insights beyond empirical results.\n\nWEAKNESSES:\n- While this paper provides a theoretical analysis of patch-level data augmentation methods, it only analyzes Cutout and CutMix, leaving other related methods for future work. \n- The theoretical analysis of this paper is expected to offer insights into practical applications and performance improvement, but it lacks detailed discussion in this regard. \n- It would be beneficial if the paper could provide more meaningful insights to the readers beyond providing a theoretical analysis, which is the most significant shortcoming of this work.\n- It is suggested to simplify the content of the “assumptions” summarized in Section 2.4 (with detailed explanations moved to the appendix) and add more experimental content in Section 5, particularly experiments related to CIFAR10.\n\nQUESTIONS:\n- What insights can be gained from the theoretical analysis presented in the paper? Can this help in developing better patch-level data augmentation techniques compared to existing ones?\n- The paper specifies that the NN architecture used is a 2-layer CNN, but shouldn’t it be considered a 1-layer CNN?\n- Additionally, the weights W exist only for each class individually. Is it not possible to perform a more general analysis with C channels, as done by Shen et al.?\n\nLIMITATIONS:\nThere is no specified section for limitations in this paper. The checklist includes a brief explanation, but the corresponding details are not found in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to provide a novel theoretical insight into the training dynamics of data augmentation methods such as Cutout and Cutmix. It also supports theoretically why Cutout and Cutmix perform better than ERM by showing that the ERM training is unable to learn rare and extremely rare features and that ERM can fit perfectly on the training data while being random on the extremely rare data. Thus the key contributions can be summarized as:\n\n**Comparative Analysis**: The paper reveals that Cutout outperforms ERM by facilitating the learning of rarer features, which ERM fails to do (Theorem 3.1 and Theorem 3.2). CutMix is shown to achieve nearly perfect performance by learning all features (Theorem 3.3).\n\n**ERM Limitations**: The authors propose that the negative results for ERM stem from its tendency to classify training samples by memorizing noise vectors instead of learning meaningful features, particularly when features do not appear frequently enough.\n\n**Cutout Benefits**: Cutout mitigates the challenge faced by ERM by removing some of the strong noise patches, thus enabling the learning of rare features to some extent.\n\n**CutMix Mechanism**: A novel technique to analyse the training dynamics of CutMix. This technique views the non-convex loss as a composition of a convex function and reparameterization, allowing characterization of the global minimum of the loss. It shows that CutMix forces the model to activate almost uniformly across every patch of inputs, facilitating the learning of all features.\n\nSTRENGTHS:\nWhat I like about the paper:\n+ It addresses an important aspect of better understanding data-augmentation methods like Cutout and Cutmix. \n+ It is theoretically well-motivated.\n+ Provides a simple explanation of common features ≫dominant noises≫ rare features ≫background noise≫ extremely rare features for ERM training dynamics.\n\nWEAKNESSES:\n- I believe the writing of the current draft version can be further improved. (see questions)\n- The modelling assumptions about 2-layer CNN are questionable. (see questions for further discussion)\n- The theory needs further experimental evaluation (real-world experiment on CIFAR-10 is uninformative to the story, see questions)\n\nQUESTIONS:\n1. **Definitions**: The definitions of common, rare and extremely rare features are central to the story and contributions of the paper. However, they are not explicitly defined anywhere in the main draft.  I assume there must be some condition on probability $\\rho_k$ for a feature to be rare and extremely rare. Defining these in problem formulation will help improve the readability of the paper significantly. \n\n2. **Model Definition**: The current model definition is not 2-layered CNN in my opinion since it does not contain a hidden layer. I can summarize the model definition as $f(X)=\\[1, -1\\]^T\\phi(\\mathbf{W}\\mathbf{X})1_{P}$. Where $\\mathbf{W}=[w_{1},w_{-1}]$ and $\\mathbf{X}\\in\\mathbb{R}^{d\\times P}$, this is akin to a single-layer network. \n\n3. **Simulation Evaluation**: About the numerical simulations, the common features are defined as frequency 0.8, rare as 0.15 and extremely rare as 0.05. Is there any reasoning or theoretical justification to do so? What happens when we change the frequencies of these features to skew them further? Is the assumption that extremely rare feature frequency is rare only in the training and at test time there exists another distribution? Otherwise, why is the low performance of ERM on extremely rare features negative? Also, for Fig 1 (leftmost plot) it is not obvious why Cutmix is non-monotonic here. It rises and then plateaus, perhaps authors can change the plot scale to appropriately demonstrate this point.  \n\n4. **Real World Evaluation**: I find the real-world evaluation for the theory extremely lacking. It need not be extensive with many datasets but in the current CIFAR-10 experiments, there is no intuition from the perspective of common, rare and extremely rare features. A similar experiment to simulations on CIFAR-10 would support the point of the paper much better. \n\n5. **Visualisation of extremely rare features**: What would the visualisation of extremely rare and rare features in the case of CIFAR-10 look like? If possible can authors visualise the features learnt by ERM trained and CutOut and CutMix? This would show that extremely rare features learned by cutmix are indeed interpretable, thus important for generalisation as compared to random noise in the dataset.\n\nLIMITATIONS:\nThe major limitations of the work are in its choice of definitions, namely model definition and definition of extremely rare features.\n1. Limitation of model definition: I am sceptical of the current conclusions generalising to networks with more layers since the current model definition is not really even 2 layered. \n2. Limitations of extremely rare features definition: Based alone on the frequency of a feature in the dataset, one can not distinguish noise from extremely rare features unless we have access to the data-generating process, which is often not the case in the real world.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically analyzed the benefits of two data augmentation techniques, Cutout and CutMix, for feature learning of two-layer and two-neuron convolutional neural networks. The authors considered an ideal data distribution in high dimension with one feature patch, one dominant noise patch, and some background noise patches, where the feature patch is generated by three different features in terms of the population: common, rare, and extremely rare features. This paper showed that classical training dynamic without data augmentation achieves perfect training accuracy but performs poorly on rare and extremely rare features, leading to low test accuracy; Cutout training achieves perfect training accuracy and learns rare features better than ERM, but still struggles with extremely rare features; and CutMix training achieves near-perfect performance by learning all features and noise vectors evenly, resulting in the highest test accuracy among the three methods. The paper validated the theoretical findings through extensive numerical experiments and real-world data experiments on CIFAR-10, showing that CutMix achieves the best performance in terms of test accuracy.\n\nSTRENGTHS:\nThe paper provided a novel theoretical framework for understanding the benefits of Cutout and CutMix, filling a gap in the literature where empirical success lacked a theoretical explanation. The presentation of the main theorems (Theorem 3.1, 3.2, and 3.3) is rigorous and clearly shows the differences in learning dynamics and performance between ERM, Cutout, and CutMix..\n\nWEAKNESSES:\n1. The use of a feature-noise data model to analyze the effectiveness of different training methods is restrictive. This feature noise patch data ideally separates the noise and feature orthogonally and enables us to consider the dynamics of the weights in feature and noise directions separately. However, in general data itself may contain some useful nonlinear features, which may not be characterized by this ideal model. It would be better to provide more motivations for this data assumption and present some real-world datasets exhibiting common, rare, and extremely rare features simultaneously. Besides this analysis only considered binary classification setting. \n\n2. The asymptotic choice of the hyperparameters in Section 2.4 is not natural. While analyzing high-dimensional limits is necessary for the theoretical analysis, the practical implications and sensitivity to these hyperparameters should be better addressed. Including a more detailed discussion or empirical validation of these assumptions would be better. Additionally, the paper could provide more guidance on how to choose the hyperparameters (or the scaling) in training and data augmentation for practitioners. \n\n3. There is limited discussion on the computational costs associated with implementing Cutout and CutMix compared with ERM. Including a theoretical analysis of memory usage, time complexity, or other computational costs would help in understanding the trade-offs involved.\n\nQUESTIONS:\n1. In Section 1.2, it would be better to include more recent work on feature learning theory. For instance, some references are below.\n\n2. In Cutout training, based on your analysis of training, is there an optimal choice of hyperparameter $C$? Can we set $C=0$ in Theorem 3.2 to get Theorem 3.1?\n\n3. In Section 2.4, why do you assume the number of data points $n$ is much smaller than the feature dimension? Is it because of the over-parameterized model? And how about the large learning rate $\\eta$ case, for training dynamics? Many previous papers [Yang and Hu. 2020, Abbe, et al. 2022, Ba, et al. 2022] have shown the benefits of large learning rates to obtain feature learning. Moreover, how do you assume $K$, the number of orthogonal feature directions?\n\n4. In (4), a typo in $w^{(t+1)}_s$\n\n5. The proof strategy of CutMix training is to find the global minimizer directly. Does this strategy also work for ERM and Cutout? Can this method generalize to wider neural networks and more general datasets?\n\n6. In Lemma B.3, the statement is not complete. You define the $\\gamma_s^{(t)}$ and $\\rho_s^{(t)}$ later for different cases, separately.\n\n-------------------------------------------------------------------------------------\nYang and Hu. 2020. Feature learning in infinite-width neural networks\n\nAbbe, et al. 2022. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks\n\nBa, et al. 2022. High-dimensional asymptotics of feature learning: How one gradient step improves the representation.\n\nBen Arous, et al. 2021. Online stochastic gradient descent on non-convex losses from high-dimensional inference.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}