A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective
Authors: Chanwoo Park
Sangdoo Yun
Sanghyuk Chun
Venue: NeurIPS 2022 Accept (2022)
Keywords: Augmentation
MSDA
CutMix
Mixup
Regularization

Abstract: We propose the first unified theoretical analysis of mixed sample data augmentation (MSDA), such as Mixup and CutMix. Our theoretical results show that regardless of the choice of the mixing strategy, MSDA behaves as a pixel-level regularization of the underlying training loss and a regularization of the first layer parameters. Similarly, our theoretical results support that the MSDA training strategy can improve adversarial robustness and generalization compared to the vanilla training strategy. Using the theoretical results, we provide a high-level understanding of how different design choices of MSDA work differently. For example, we show that the most popular MSDA methods, Mixup and CutMix, behave differently, e.g., CutMix regularizes the input gradients by pixel distances, while Mixup regularizes the input gradients regardless of pixel distances. Our theoretical results also show that the optimal MSDA strategy depends on tasks, datasets, or model parameters. From these observations, we propose generalized MSDAs, a Hybrid version of Mixup and CutMix  (HMix) and Gaussian Mixup (GMix), simple extensions of Mixup and CutMix. Our implementation can leverage the advantages of Mixup and CutMix, while our implementation is very efficient, and the computation cost is almost neglectable as Mixup and CutMix. Our empirical study shows that our HMix and GMix outperform the previous state-of-the-art MSDA methods in CIFAR-100 and ImageNet classification tasks.

A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective
Chanwoo Park + MIT
Sangdoo Yun NAVER AI Lab
Sanghyuk Chun NAVER Lab
cpark97@mit.edd
kungdoo.yunnmmavrroon
Siighyukk.ennveecorr.com
Abstract
We propose the first unified theoretical analysis mixed sample data augmentation (MSDA), such as Mixup and CutMix. Our theoretical results show that regardless of the choice of the mixing strategy. MSDA behaves as pixel-level regularization of the underlying training loss and regularization of the first layer parameters. Similarly, our theoretical results support that the MSDA training strategy can improve adversarial robustness and generalization compared to the vanilla training strategy. Using the theoretical results, we provide a high-level understanding how different design choices of MSDA work differently. For example, we show that the most popular MSDA methods, Mixup and CutMix, behave differently, e.g., CutMix regularizes the input gradients by pixel distances, while Mixup regularizes the input gradients regardless of pixel distances. Our theoretical results also show that the optimal MSDA strategy depends tasks, datasets, model parameters. From these observations, we propose generalized MSDAs, a Hybrid version Mixup and CutMix (HMix) and Gaussian Mixup (GMix), simple extensions of Mixup and CutMix. Our implementation can leverage the advantages of Mixup and CutMix, while our implementation is very efficient, and the computation cost is almost neglectable as Mixup and CutMix. Our empirical study shows that HMix and GMix outperform the previous state-of-the-art MSDA methods in CIFAR-100 and ImageNet classification tasks. Source code is available at bttps://githane com/ naver-aaaeeeee hemihmmimnn xggxx.
Introduction
As deep neural networks (DNNs) are data-hungry, the scale datasets has become a foundation of modern DNN training: recent ground-breakien deep models are built upon gigantic datasets, such as 410B language tokens [6], 3.5B images [47], and 1.8B image-text pairs [33]. While such tremendously large-scale datasets not always collectible, amplifying the dataset scale by synthesizing more data points by data augmentation techniques is common. Especially, mixed sample data augmentation (MSDA) [76, 63, 62, 30, 72, 66, 60, 3, 68, 74, 11, 65, 39, 25, 18, 54, 32, 16, 23, 38, 45, 14, 32, 58, 44] has become a standard technique to train a strong deep model by synthesizing a mixed sample from multiple (usually two) samples by combining both of their sample values and labels in a linear combination [76] or a cut-and-paste manner [72]. This simple idea, however, shows surprising performance enhancements various applications, including image object recognition [72, 39, 25, 64, 16], semi-supervised learning [4, 59] self-super learning [35, 40, 42], noisy label training [43], meta-learning [71], semantic segmentation [10, 20], natural language understanding [24, 34], and audio processing [48, 37, 49]. Another advantage of MSDA beyond the performance improvements that MSDA usually does not need Nomain--peciinn knowledge, such as strong image-specific [15] or audio-specific [53] transformations; hence MSDA can be universally employed
'Equal contribution
Works done while doing internship NAVER Al Lab
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

by various applications. However, although MSDA shows excellent benefits in practice, there is still yet not enough understanding how MSDA works well universally; can MSDA always show better generalization and robustness than the standard training with theoretical guarantee? Furthermore, the design choice of MSDA can be significantly varying and the optimal design choice is still ambiguous. For example, Lee et al. [42] showed that in self-supervised learning, Mixup is more effective than CutMix, while other studies [56, 57] observed opposite results. The ambiguity is originated from the fact that we do not have unified theoretical lens of understanding how different design choices affect the actual learning process; in short, how are Mixup and CutMix different?
There have been several attempts to theoretically understand Mixup, a special case of MSDA [77, 12, 8, 78]. They delve into the effect of Mixup in a loss function perspective, e.g., Mixup behaves as a regularization of the standard training [77]. or in a learning theory perspective, e.g.. Mixup training can provide an upper bound for the true loss [12, 71.. However, their analyses are limited to Mixup, while other MSDAs, such as CutMix, are poorly understandable through the lens of their analyses. In this paper, we extend the theoretical results of Zhang et al. [77] and Chidambaram er al. [12] to a general MSDA to provide first unified theoretical lens for understanding how general MSDAs work by different choices of mixing strategies. We show that MSDA behaves as an input gradient and Hessian regularization (Theorem I) as well as regularizer for the first layer parameters; MSDA improves adversarial robustness (Theorem 3) and generalization (Theorem 4). Our theoretical results show that popular MSDA methods, such Mixup and CutMix, behave differently in terms of regularization effects. Briefly, CutMix gives a strong regularization in the product of nearby distance pixel-level partial gradient and nearby distance Hessian of the estimated function f, while CutMix gives a weak regularization in the product of long-distance long-distance Hessian of the estimated function f- In contrast, Mixup gives regularization in gradient or Hessian of the estimated function ff regardless of the pixel-level distance.
pixel-level partial gradient and
From our unified theoretical lens for MSDA, we can conclude that there is no one-fit-all optimal MSDA fit to every data or model parameter. In other words, the optimal mixing strategy depends on applications, datasets, and model architectures. It supports previous empirical observations that combining different MSDA methods (e.g., alternatively using Mixup and CutMix during training) can outperform using only one MSDA [50, 5, 64, 73]. From these observations, simple MSDA methods that naturally generalize Mixup and CutMix, so that it can take advantage of both methods. Our first proposed method, Hybrid version of Mixup and CutMix (HMix), mixes two samples in both Mixup and CutMix manners; it first cut-and-paste two samples as CutMix, and then it linearly interpolates the out-of-box values of two samples Mixup. We let HMix be able to behave as both Mixup and CutMix by introducing a stochastic control parameter. Our second proposed method, Gaussian Mixup (GMix), also mixes two samples in both Mixup and Cutmix manners; firstly we select point, and then we mix two samples gradually using the Gaussian function. Our empirical results on CIFAR- 100 and ImageNet show that HMix and GMix outperform the state-of-the-as MSDA methods, including Mixup, CutMix, and Stochastic Mixup & CutMix.
propose two
2 A General Framework for Mixed Sample Data Augmentation (MSDA)
In this section, we define the formal definition of MSDA and notations. We define training dataset as = {z+ = (I,, M1) )m randomly sampled from a distribution P.. Here, x = (a, y) is the input (e.g., an image) and output (e.g., the target class label) pair. Then, for randomly selected two data samples, 2; and æj. an augmented sample by MSDA, (MSDA) is synthesized as follows
2i,j
(MSDA)
(MSDD)) (A, 1 - A)) (A,1 A) = (5/ (MSDA) (A, A), Fig
(MSDA) (A, A)= M(A) 3 x, (1- (1-M(()) 3 xj and Fij
where,
(MSDA) 1 -11-11 = yi + (1 (N()) o yj:
Vi.
where ) the ratio parameter between samples, drawn from DA (usually Beta distribution). & means a component-wise multiplication in vector (or matrix). M() is a random variable conditioned on that indicates how we mix the input (e.g., by linear interpolation [76] or by a pixel mask [72]). (A) denotes a random variable conditioned on À that demonstrates how we combine the output. We assume that the output y can be one-dimensional data or a matrix; the former means regression or classification task, and the latter means semantic segmentation task. For the sake of simplicity, we let y be one- mee-imensiional data: (A) À and E (A)] = AI.

---

Remark 1. If the meaning is not ambiguous, then we sometimes omit À (i.e., (A) to M). For
(MSSA)) (A, 1 111-2 A)). but we can the sake simplicity, we consider mixing only two samples (i.e., žijj similarly extend these analyses to mixing n-samples data augmentation [60, 32, 23]. If we combine n-samples, the ratio parameter will be a vector in general (See Appendix B).
As recent studies [68, 39, 38] have shown, M (A) or N (A) can depend on (zi, zy), e.g., by using a saliency map [8] or the class activation map [68]. Since the proof techniques for our theoretical analysis are invariant to the choice of M (A) and N [A), our proof techniques also can be applied to the dynamic MSDA methods. For simplicity, we assume that M is a random variable only depending on A. In other words, we assume W as a random sample space, and M : W x A + R" is a measurable function. We left the theoretical analysis of dynamic methods the future.
Remark 2.
Now, we re-write the two most popular MSDA methods, Mixup [76] and CutMix [72], for i-th and j-th samples with A, drawn from >x, by using the proposed framework (Equation (1)) as follows:
Mixup:
z(mikur) (A, A) A)
(2 (Z/(0ii0)) (A, 1 A),
M), w/ (A, A))
where x((mixu))
(A, 1 - 2) = Ax, + (1 A)æy and
Vi. (2, 2) Ay + (1 A)8s-
(mixup))
CutMix:
(A, 1 A) = (=!(gamin) (M, 1 M), Wi.s ((Cuimix))
(cutmix)
(), 1 A))
where
# [:feutmix] (N NT(cumii)) = MM (cutmix)
3. + (1 (1- 201-h0 (cutmii)) O.y (3)
and
(A, -- Vi.3
(cunmix)
Aza A)us-
Note that Equation (2) is equivalent to Equation (1) by putting M(A) AI. In Equation (3), (cutmix) is binary mask that indicates the location of the cropped box region with relative area X. Similarly, other MSDA variants can be easily formed as Equation (1) by introducing new M (A) and (A).
Notations. We define the loss function as =), where 0 C θ C TRd We define L(0)
Eg-p, 1(0, 2) as the non-augmented population loss and Lm (0) loss for the non-augmented population. For general MSDA, can define MSDA loss as
1 _
I(0, 2,) as the empirical
m
MSDA (0)
T (m))(((mm BAND, BM Pi.j
(MSDA)) (A, A))-
(4)
Therefore, the Mixup and CutMix losses can be written as
Lim
mixup
(0)
EAND,I(A
1(0, 2..5
(x,1-1))
m
1,==1
(5)
Lim
catmis (0)
EAND Exs((0 21.
s(cutmix) (A, -));
where [m] = 21, m} and Dx is a distribution supported on [0, 1] with a conjugate prior. Throughout this paper, consider Dx as Beta(a, B8), common selection for À in practice. We define Dx as the empirical distribution of the training dataset.
3 A Unified Theoretical Understanding of MSDA
In this section, provide unified theoretical lens of how MSDA works. Specifically, follow the theoretical results for Mixup provided by Zhang el al. [77], where Zhang et al. have shown that Mixup is equivalent
the summation of the original loss function and Mixup-originated regularization term. We will give a general approximation form for MSDA using À ~ Beta(o, 18). We also show that our analysis can be extended re-ample mixed augmentation (See Appendix B)
From an MSDA loss to an input gradient and Hessian regularization. We first consider the following class of loss functions for a twice differentiable prediction function fo(x) (e.g., softmax output of a neural network), a twice differentiable function ht and target y::
L (1(0,)) 2) [(0, 2) h(fo(z)) (zf( for twice differentiable function h}.
3

---

This function class L includes the loss function induced by Generalized Linear Models (GLMs) and cross-entropy. Now, we introduce our first theoretical result that induces an MSDA loss (i.e.. Equation (4)) can be re-written as the summation of the original loss (the empirical loss for the non-augmented population loss, T. (0) and input gradient-related regularization terms as follows.
Theorem 1. Consider a loss function I E L. We define Dx as a+g Beta 5)++
e a+B (eta(
Assume that B, [rx] 0. Then, we can re-write the general MSDA loss (4)
3
I. MSDA (0) Li (8) ))+ + Σ
R (MDDA) (0) 1D+-EADUN E (1 M) M)
M)). (6)
where lim +0 (p(a)= = 0,
R (MSD4) (0) m
(31 - ((f66x))) Bx,D, (1
R2 (MSDA) (0) \'( f..
(7)
m
R. (MSD4) (0) 2m
(N'(f(2zs) BA Hi, M.M,
and
G(Dx = EM - My (Vf(x:) o (r= 2,) (V) (r2 (1)))) MM
=E (o
(B [TagTa]] + 2zy2iz).
j.kEooond
(8)
HDx Uii S, M) = Er,~Dy Ea M) (V fa(x:) @ ((r. - 2:)( - x.) (1 10-MM M)
Σ
Ojk (Br,NDx I2 o(
J.kEcoord
where
0yk := E M1 - M+)].
(9)
Proof outline of Theorem 1. Using the definition of žyy and using the fact that the Binomial distri- bution and Beta distribution are in the conjugate, we can reformulate Lin (MSDA) In the process of reformulating Im (M)))A) we should define Dx. Then, we can make a quadratic Taylor approximation of the loss term. Here, E,,, [r=] = 0 is used for not only the simplicity of the results, but also for the fact that using normalization in the dataset. Details can be found in Appendix A. We also show that Theorem can be extended n-sample MSDA methods (Appendix B). In this case, the combinatorial terms in quadratic multivariate Taylor approximation also come out.
How is our approximation accurate? We call L (MMDDA() (m) + Li=1 R (MSDA) as the approximate MSDA loss. Here, we empirically demonstrate that our quadratic approximation is almost accurate by following numerical validations in [67, 8, 77]. Specifically, we train logistic regression models on two-moons dataset [7] in two ways: (1) by using the original MSDA loss function (2) by using our approximated loss function. We employ two MSDA examples as below.
The original Mixup Le., ~ Beta(1, and M = AI
Variants of CutMix i.e., X ~ Beta(1, 1) and A = (m1, m2) such that m, ~ Bernoulli (A).
Figure displays the approximate loss function and the original loss function. According empirical findings, we can conclude that the original MSDA loss is fairly close to the approximate MSDA loss.
What makes the difference between various MSDA methods? In the theorem, we define EM (1
M) = X, R. (MSDA) is the same for every MSDA method. Namely, the difference between MSDA methods originated from R. (MSDA) and R (MSDA) Note that if we set M = Xï, Theorem 1 indicates a Mixup loss (5), and the result is consistent with Zhang el al. [77]. In Equation (7) and Equation (8), we observe that R₂ is related to the input gradient V fe and R3 is related to input

---

0.75
MSDA
MSDA
Approximate
0.65
Approximate MSDA
loss
0.65
0.55
0.55
0.45
0.45
200
400
600
800
1000
0
200
400
600
800
1000
epoch
epoch
(a) Mixup
(b) CutMix
Figure 1: Comparison of the original MSDA loss with the approximate MSDA loss function.
Figure 2: The negative M value results. the image is A dog + (1 - A) cat where -0.75 < À S 0.75
Hessian 2 fo(x)) with mask-dependent coefficients 0zk (9). other words, different design choice of MSDA (e.g., how to design M) will lead to different magnitudes of regularization on the input gradients and Hessians. Because the values input gradients and Hessians are varying by datasets, tasks and model architecture choices, can conclude that the optimal choice of M is dependent on the applications. We also describe how the other MSDA methods (e.g., dynamic MSDAs [65, 39, 38]) can be interpreted through the lens of unified analysis in Appendix I.
In addition, to show that MSDA behaves as regularization on input gradients and Hessians for any desired Ajk- we also show that there always exists MSDA design choice M for any desired regularization coefficient matrix A(A) == (ays (A)) with the regular conditions.
Theorem For the given X we assume A(X) (1 T)T a nonnegative definite matrix. Then can construct real-valued musk M that B[ M2) - MR) = azk. for all j. k.
Proof. Setting M = = =1=1++++AA)) (1--(())-1111111 (1-A)(i))) where Z is normal distribution, the theorem holds.
Note that, in the proof, M values not bounded where typically choose 0 S Mf < 1. In other words, the theorem holds if we allow mask values out [o, 1]. To investigate the potentiality of unbounded mask, explore Mixup with unbounded masks in Figure 2. Although, allowing negative values to M can be beneficial, we leave a new mask design with unbounded values as future work.
Unfortunately, as the target loss function (6) is mingled with the choice of mask M, data sample z4. and pixel-level function gradient, the optimal choice of mixing strategy M not achievable in the closed-form solution. Instead, Theorem I implics that there is absolute superiority between the design choice of MSDA, but depends on datasets and the target tasks, as our empirical observation consistent with the theoretical interpretation. In Section 4, will provide more examples of how different M affects the actual coefficients Ojk and the input gradients for better understanding.
Using the regularization term R. R.MSDA) (MSDA) (7), can also provide theoretical connection between MSDA methods and the notion of flatness where a more flat solution leads to better generalization in applications [36, 21, 31, 19, 9]. Inspired by Ma et al. [46], we split the parameters by 0 (01, 02). and then the neural network can be represented by the form fo(x) (0)x). Therefore, we have
0f
(0 00 0f
(0,
ST.
0(0'z)
a(0')))
5

---

8f is the partial derivative of the first layer. Now, we have where 8(61))
of 005 2 (1- --)))))
((- -1
= o
tr ((((())))
Of T
0, diag(1 M) (( diag(1 M))
00
Note that the terms in G (8) can be re-written as follows
Ea((1
(MSDA), [[O Ve, Ve fa.)², i.e. the regularization (o
[[[[[1 [[((11 M) Vf(=))/x)+)
Eooodd
In other words, by minimizing the regularization term R2 effect of flatness at the interpolation solution can be minimized in sample-wist weighted manner. Therefore, the regularization term R. (MSDA) also can be interpreted as a regularization of the first layer parameters and their partial derivative of f-
Robustness and generalization properties of MSDA. As a number of studies [51, 52, 46] have shown that regularizing input gradient and Hessian will give better robustness and generalization to the target network 0, can be shown that MSDA also has adversarial robustness properties and generalization properties based on Theorem 1. The full statement of Theorem 3 and Theorem 4 can be found in Appendix and Appendix D, respectively.
Theorem 3 (Informal). With the logistic loss function under the ReLU network, the approximate loss function of MSDA greater than the adversarial loss with the lg attack of size evd.
Proof outline of Theorem 3. Defining adversarial loss function and using second order taylor expan- sion, we can prove that adversarial loss less than MSDA loss.
Theorem 4 (Informal). Under the GLM model and the regular conditions, and if we MSDA in training, we have
O (log(1/8))
L(#) < I (MSDA) (B)
with probability at least 1 5. This also holds for the MSE loss and a feature-level MSDA.
Proof outline of Theorem 4. MSDA regularization can be altered to the original empirical risk mini- mization problem with a constrained function set, and calculating Radamacher complexity of this function set gives the theorem.
In addition Theorem and Theorem 4, we can prove that the optimal solution of (4) can achieve perfect classifier (i.e., classifies every augmented sample x correctly) in the logistic classification setting by following Chidambaram al. [12]. The full statement are in Appendix E.
Summary. Our unified theoretical lens for MSDA shows that for any MSDA method formed as Equation (4), the method satisfies that (1) it behaves as regularizer of input gradients, Hessian, and the first layer parameters (Theorem 1); (2) there exists a mask M for any desired regularization coefficients ask (Theorem 2) (3) it achieves better adversarial robustness (Theorem 3) and generaliza- tion (Theorem 4) than the vanilla training. Interestingly, Theorem 1 shows the difference between different MSDA design choices (e.g., different M, such as linear interpolation [76], cropped box [72)) will lead to different magnitudes of the input gradient regularization (7).
4 Comparison of Different MSDA Design Choices: The Role of Masks
As we observed in the previous section, different design choices for MSDA (i.e., the choice of M) affect to the degree of the regularization in Theorem 1 (i.e., R. (MSDA) and R (MSDA)) by the relationships of pixels. this section, we show how different MSDA methods lead to different regularization effects by empirical studies; first show the values of the regularization coefficients

---

Mixed Images
Mixing Masks
Original
Original B
Mixup
CutMix
Hmix (ours)
Gmix (ours) 0.0
Figure 3: Examples generated by different MSDAs. From left to right, two original images to be mixed, Mixup, CutMix sample. HMix, and GMix. The first and the second rows show generated samples and their mixing masks M. respectively. We set À 0.65 for all images and 0.5 for HMix.
ajk by varying masks; then we show the input gradient values that are regularized by azk (9) after the MSDA training; finally, we show that the best choice of the mask design can be varying by the target task settings. In addition, we propose two generalized versions of Mixup and CutMix, called HMix and GMix, that empirically show the intermediate property of Mixup and CutMix.
Introduction to HMix and GMix. Recall that the regularization coefficients aga is determined by
M (Equation (9). For example, by choosing M >Î (i.e., Mixup), ajı is always (1 - A)2. On the other hand, the result slightly changes for CutMix: azж depends on how j and k are close. Informally, due to dependency between M, and M& (as M's component is always 0 in the cropped box regions and in others), close j and ls give large ajk, but distant j and k give small ajk- Ozk is calculated as
a&k max - 1(k1), ((k)) [(in D).0) max (axx(i)(((())) 1(k2), h(k2)) 1(j2)).0)
(10)
(n - [v] An])2
where j - (2, (k2, k2), M(1) = min(t, n MV Am)), max(1 - [VT Xn]. 0).
We visualize O,k of different MSDA methods in Figure 4. We compare Mixup. CutMix, Stochastic Mixup & CutMix. We also propose two generalized MSDA methods, named HMix and GMix. Before comparing the methods, we first formally define Stochastic Mixup & CutMix, HMix and GMix. These methods can be formed as (1) where the definition of M is varying by the methods.
Stochastic Mixup& & CutMix is practical variant of MSDA by considering Mixup and CutMix at the same time. By simple alternation of two augmentations, the state-of-the-art performances on large- scale datasets are shown [64, 70J. Stochastic Mixup & CutMix is the same as Equation (1) by setting M(A) (1 A) with probability g and ((A) Meutmix (A) with probability
q. We choose = 0.5 as [64, 70J. In our loss function perspective, the regularizing coefficient terms (i.e., R2, R3) become the average of Mixup and CutMix's 's regularization coefficient. Namely, let d:j
mixxp = (1 - X)2
be regularization coefficient of Mixup and d:y agyumix be regularization coefficient of CutMix (10), then the regularization coefficients of Stochastic Mixup & CutMix is qai qa + (1 (1- mixup
Here, we additionally propose two MSDA variants, HMix and GMix, that leverage the ad advantages of Mixup and CutMix, resulting in showing the intermediate property between Mixup and CutMix.
Hybrid version of Mixup and CutMix (HMix) combines Mixup and CutMix by shrinking the CutMix cropped box region and linearly interpolating two images in the areas out of the box as Mixup. The shrinking ratio of the cropped box region is determined by the ratio HMix can be written (1) by setting M by (I) randomly cropped box region with side length /I X/FN where N is the side length of the original image, and make M's s component in the box region as 0 (2) in the areas other than the box, we set M, as 1-r We can easily check that E[M] AI. As r - 0, this method goes to Mixup, and as r + 1, this method goes to CutMix. Note that the ratio r can be a random variable, such as Beta(r, 7). In this case, if we set + 0, as Beta 7) goes to Bernoulli distribution, this is equivalent to Stochastic Mixup & CutMix.
We propose Gaussian Mixpp (GMix) to relax the CutMix box condition to a continuous version as the rectangle cropping of CutMix causes implausible augmented data, e.g., the boundary between two mixed samples. Therefore, combine two ideas of Mixup and CutMix. Firstly, we select
7

---

Stochastic Mixup & CutMix
Mixup
OutMix
HMix (r=0.5)
HMix (r=0.7)
GMiX
0.5
30 45 60
0.0
Figure 4: Visualization of regularization coefficients for different MSDA methods. dy values of Mixup. CutMix, Stochastic Mixup & CutMix (the alternation of Mixup and CutMix), HMix, GMix (described in Section 4 and Appendix I1) are shown. Each (I, 3) value is computed by (aa,)+(s.)) where is a pixel vector.
point p from the given input. Then, make M, as the related function with || vi-ppi pl| Specifically, we use the Gaussian function for making M: (1) randomly select a point p in image (2) in the areas
||i-pl1* The proposed GMix has the following d.j
other than the box, we set Mj exp
2(1-))
2(1 A)NV (- -1i--pl" 11i - -pll²)
Gij
exp
N2
pEpixel
R2
exp
2(1
A).N2
(- ||i pll
115 - p13))
(1li-p1()) de
= (1 exp
(11)
(1 X)N2
2
As seen Equation (11), 0zj smoothly goes down when the pixel distance becomes larger.
Figure shows the examples generated Mixup, CutMix, HMix, and GMix. The proposed methods (HMix and GMix) generate images in hybrid form with the properties of both Mixup and CutMix.
Comparison in terms of regularization coefficients azs- We illustrate the regularization coeffi- cients azk of the different MSDA methods in Figure 4. In particular, fix the mask parameter À to 0.5 and the input resolution to 64 x 64. Figure 4 shows the difference between the MSDA methods in terms of how they regularize the imput gradients and input Hessians: Mixup has equal weights to every gradient component or Hessian component, while CutMix gives high regularization close coordinate gradient products Hessian. We also observe that the hybrid methods (e.g., Stochastic Mixup & CutMix, HMix, and GMix) show the intermediate coefficient values of Mixup and CutMix.
Comparison in terms of the regularized input gradients after MSDA training. Equation (8) shows that the regularization term azy directly affects to the pixel gradients [0 ( (xxx) in approximated loss function. The purpose of Figure 5 is to show how the pixel gradients actually regularized after training. We investigate the amount the regularized input gradients by [O. (2))))+++pff())) with respect the pixel distance vector p for trained models by different MSDA methods. Here, if our approximated loss function actually behaves as a regularization, then we can expect that the pixel gradients (z)) (z)) is small when a.j is large for the given p.
We first define the partial gradient product as follows:
(12) Now, we visualize the pixel-wise maximum values of PartialGradProd(z;; p) in Figure 5, We train different models fo on resized ImageNet (64 64) and measure the values on the validation dataset. The z-axis and y-axis of Figure 5 denote the pixel distance p along each x and y axis, and the scale of the colorbar denotes the value of the maximum partial gradient product. In the figure, we can observe that CutMix reasonably regularizes effectively in the input gradients products when pixel distance is small; these results aligned with our previous interpretation, CutMix behaves pixel-level regularizer where it gives stronger regularization (larger a,y) to the closer pixels. Note that not discussing the relationship between regularizing effects and accuracy but discussing regularizing coefficients and the optimized function's 's pixel gradients.
ParialGradProd((r, p) = max (0.
Understanding application cases when a specific MSDA design choice works better than others. From our theoretical results and empirical studies, have shown that the design choice of MSDAs

---

60
60
60
45
45
45
30
30
30
15
15
30
45
60
30
45
60
30
45
60
Vanilla (no MSDA)
(b) Mixup
CutMix
Figure 5: Regularized input gradients by MSDA. The normalized pixel-wise partial gradient norm product comparison among the models trained with vanilla setting (a), Mixup (b) and CutMix (c). We plotted (12), and x and y axis denote the pixel distance p along each axis.
Table 1:
Different tasks need different MSDA strategies. Validation accuracies of Mixup and CutMix trained networks on two different scenarios on ImageNet-100. Each scenario assumes different pixel importances.
Mixup
CutMix
A (CutMix Mixup)
Scenario Large crop
58.3
64.4
+6.1
Scenario Small crop
67.7
67.0
-0.7
(L..,, M) leads to different regularization effects by regularization coefficient azk- Furthermore, as we have shown in Theorem 2, there always exists mask that can form any desired azt- We hypothesize that for the given dataset, if a short distance relation is relatively more important than longer distance relations, then CutMix will be better than Mixup. On the contrary, in the opposite case, if short distance relation relatively less important, then Mixup will be better than CutMix.
Here, we study different task scenarios when different 4ju8 are required by controlling the pixel-Ievel importance of ImageNet- 100 [61] training images. particular, we design two different scenarios where each of them needs different regularization strategies due to the different pixel-level importance of each task. The results are shown in Table 1. We also report the performance of our proposed methods in both scenarios and 2 in Appendix G.
Scenario 1: Smaller objects by large crop size. We randomly crop large region (80% to 100%) of an image and resize to 64 x 64 to train a model. As the objects in the image become small, a close-distance relationship might be more important than large-distance relationship. Here, expect CutMix performs better than Mixup as shown in Table 1.
Larger objects by small crop size We randomly crop small region (25% to 40%) of an image and resize to 64 64 train model. Contrary to Scenario I, the objects in the image would become large in the cropping region and the harge-distacee relationship might be important, therefore, we expect that Mixup performs better than CutMix. This hypothesis is aligned Table 1.
Scenario 2:
5 Comparison of Different MSDA Design Choices: An Empirical Validation
In this section, we compare various MSDA methods two popular large-scale image classification benchmarks: CIFAR- [41] and ImageNet- IK [17]. We will confirm that our proposed design choices, HMix and GMix, not only theoretically interpolating Mixup and CutMix in the toy settings, but also taking benefits of each method by showing great performances in real-world applications. The implementation details and the haree-parrmeter study can be found in Appendix F,
Results on CIFAR-100 classification. We evaluate HMix and GMix against baseline MSDA methods including Mixup [76], CutMix [72]. Stochastic Mixup & CutMix, [64] and PuzzleMix [39] on CIFAR- 100 dataset. Here, we include PuzzleMix, to see the effectiveness method against the data-aware mask strategy. Note that although our theoretical results (Section 3) are based on the data-agnostic mask selection methods, our theoretical results can easily extended to the data-dependent mask selection methods. We leave the extension future research direction.
our data-agnostic

---

Table CIEAR-100 classification, Comparison of various MSDA methods on various network architectures.
Note that PuzzleMix needs additional computations (twice than others) for computing the input saliency.
Augmentation Method
RN56
WRN28-2
PreActRN18
PreActRN34
PreActRN50
Vanilla (no MDSA)
73.23
73.50
76.73
77.68
79.07
Mixup
73.12
74.05
77.21
79.02
79.34
CutMix
74.83
74.79
78.66
80.05
81.23
PuzzleMix
76.51
79.38
80.89
82.46
Stochastic Mixup CutMix
74.88
75.49
79.25
81.05
81.21
HMix (ours)
74.99
75.68
79.25
81.07
81.38
GMix (ours)
75.75
76.15
79.17
80.52
81.45
To see the generalizability of our methods, we train various network architectures including ResNet-56 (RN56) [26]. WideResNe128-2 (WRN28-2) [75], PreActResNet-18 (PreActRN18) [27], Pre4 (PreActRN34) [27] and PreActResNet-50 (PreActRN50) [27] with various MSDA methods. We train networks for 300 epochs using SGD optimizer with a learning rate 0.2. Table shows the summarized results. We set the hyper-parameter for Mixup, CutMix, and Stochastic Mixup & CutMix to 1. a for HMix and GMix were set to and 0.5, respectively. We use r = 0.5 for HMix. In the table, HMix and GMix outperform Mixup only and CutMix only counterparts and Stochastic Mixup & CutMix often show comparable performances to HMix and GMix. Our methods show comparable performance with the otate-of-the-arr data-depender strategy PuzzleMix.
Results on ImageNet-1K 1K classification. Ta- ble 3 shows the comparison of various MSDA methods on ImageNet-13 K. We train ResNet- 50 [26] with various MSDA methods for 300 epochs using SGD optimizer with learning rate 0.1. We set the hyper-parameter a for methods except Mixup to 1, while Mixup has a = 0.8. We use r = 0.75 for HMix. Here, do not include PuzzleMix because it needs heavy additional computations to compute the input saliencies. In the table, HMix shows the best performance, while GMix and Stochastic Mixup & CutMix show the second-best performances. We also include evaluations various robustness benchmarks in Appendix G.
Table 3: ImageNet- 1K classification. Comparison of various MSDA methods ResNet-50 architecture.
Augmentation Method
Top-1 accuracy
Vanilla (no MDSA)
75.68 (+0.00)
Mixup
77.78 (+2.10)
CutMix
78.04 (+2.36)
Stochastic Mixup & CutMix
78.13 (+2.45)
HMix (ours) GMix (ours)
78.38 (+2.70)
78.13 (+2.45)
Conclusion
We analyze MSDA by unified theoretical framework. Our unified theoretical results show that any MSDA method behaves as a regularization on the input gradients and Hessians, where the degree of the regularization controlled by the design choice of MSDA. We compare various MSDA methods in (1) regularization coefficient (2) regularized gradients (3) model performances in various scenarios with different pixel-Ievel importance. We propose two simple MSDA methods, HMix and GMix. which leverage the benefits of Mixup and CutMix by their design. Our experimental results show that HMiX and GMix outperform popular MSDA methods Mixup and CutMix. Furthermore, our methods show comparable or outperformed performances than the state-of-the-as MSDA method, Stochastic Mixup & CutMix, in CIFAR- -IFAR-100-R and ImageNet classification tasks.
Author Contributions
This work is done as an internship project by Park under the supervision S Yun. C Park contributed to the theoretical analysis, including theory ideas and proofs. Theoretical results were verified and interpreted by Park and Chun. HMix and GMix were designed by C Park and Yun. S Yun implemented and conducted CIFAR and ImageNet experiments. Empirical findings and analyses were contributed by all authors. Paper presentation, such as paper writing and storyline, mainly led by S Chun. All authors significantly contributed to the final manuscript.
10

---

References
[1] R. Arora, Bartlett P. Mianjy, and N. Srebro. Dropout: Explicit forms and capacity control. ICML, 2021. 23
[2] L.. Bartlett and S. Mendelson. Rademacher and gaussian complexitie:: Risk bounds and
structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002. 24, 25
[3] C. Beckham, S. Honari, V. Verma, A. M. Lamb, F. Ghadiri, R. D. Hjelm, Y. Bengio, and C. Pal.
On adversarial mixup resynthesis. NeurIPS, 2019.
[4] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, Oliver, and C. A. Raffel. Mixmatch: holistic approach to semi-supervised learning, NeurlPS, 2019. 1
[5] A. Bochkovskiy, C.-Y. Wang, and H.-Y M. Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint wrXiv:2004.10934 2020.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, et al. Language models few-shot learners. NeurlPS, 2020.
[7] L. Buitinck, G. Louppe. M. Blondel, F. Pedregosa, A. Mueller, O. Grisel, V. Niculae, P. Pretten- hofer, A. Gramfort, J. Grobler, et al. Api design for machine learning software: experiences from the scikit-learn project. arXiv preprint arXiv: 1309.02300 0238,. 2013.
[8] L. Carratino, M. Cissé, R. Jenatton, and J.-P. Vert. On mixup regularization. arXiv preprint arXiv:2006.00049, 2020. 2,
[9] J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park. Swad: Domain generalization by seeking flat minima. Neur!PS, 2021.
[10] Y.T. Chang, Q. Wang, W.-C Hung. R. Piramuthu, Y.-H. Tsai, and M.-H. Yang. Mixup-
semantic segmentation via uncertainty regularization. arXiv preprint arXiv:2008.01201, 2020. 1
[11] P. Chen, S. Liu, H. Zhao, and J. Jia. Gridmask data augmentation. arXiv preprint arXiv:2001.04086, 2020.
[12] M. Chidambaram, X. Wang, Hu, C. Wu, and R. Ge. Towards understanding the data dependency of mixup-style training. ICLR, 2022. 2, 6, 26
[13] S. Chun, S. J. Oh, S. Yun, D. Han, J. Choe, and Y. Yoo. An empirical evaluation robustness and uncertainty regularization methods. ICML Workshop on Uncertainty and Robustness in Deep Learning, 2019. 28
[14] S. Chun and S. Park. Styleaugment: Learning texture de-biased representations by style augmentation without pre-defined textures. arXiv preprint arXiv:2108, 10549, 2021.
[15] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. CVPR, 2020.
[16] A. Dabouei, Soleymani, F. Taherkhani, and N. M. Nasrabadi. Supermix: Supervising the mixing data augmentation. CVPR, 2021.
[17] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009. 9
[18] M. Faramarzi, M. Amini, A. Badrinaaraayanan, V. Verma, and S. Chandar. Patchup: A regularization technique for convolutional neural networks. arXiv preprint arXiv:2006.07794, 2020.
[19] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-awarr minimization for efficiently improving generalization. ICLR, 2021.
[20] G. French, S. Laine, Aila, M. Mackiewicz, and G. Finlayson. Semi-supervised semantic segmentation needs strong, varied perturbations. BMVC, 2020.

---

[21] T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. NeurIPS, 2018.
[22] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015. 28
[23] K. Greenewald, A. Gu, M. Yurochkin, J. Solomon, and E. Chien. k-mixup regularization for deep learning via optimal transport. arXiv preprint arXiv:2/06.02933, 2021. 1,3
[24] H. Guo, Y. Mao, and R. Zhang. Augmenting data with mixup for sentence classification: An empirical study, arXiv preprint arXiv:1905. 08941, 2019.
[25] E. Harris, A. Marcu, M. Painter, Niranjan, A. Prügel-Bennett, and Hare, Fmix: Enhancing mixed sample data augmentation. arXiv preprint arXiv:2002. /2047, 2020. 1, 29
[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016. 10,
[27] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. ECCV, 2016. 10
[28] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. ICLR, 2018. 28
[29] M. Hong. J. Choi, and G. Kim. Stylemix: Separating content and style for enhanced data augmentation. CVPR, 2021. 29
[30] H. Inoue. Data augmentation by pairing samples for images classification. arXiv preprint arXiv:1801.00292, 2018.
[31] Izmailov, Podoprikhin, Garipov, D. Vetrov, and A. G. Wilson, Averaging weights leads to wider optima and better generalization. UAI, 2018. 5
[32] J. Jeong, S. Cha, Y. Yoo, S. Yun, T. Moon, and J. Choi. Observations on k-image expansion of image-mixing augmentation for classification. arXiv preprint arXiv:2110.042488 2021. 1,
[33] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung. 7. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. ICMI., 2021.
[34] A. Jindal, D. Gnaneshwar, R. Sawhney, and R. R. Shah. Leveraging bert with mixup for sentence classification (student abstract). AAAI, 2020. 1
[35] Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, and D. Larlus. Hard negative mixing for contrastive learning. NeurlPS, 2020. 1
[36] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017. 5
[37] G. Kim, D. K. Han, and H. Ko. Specmix: A mixed sample data augmentation method for training withtime- frequeny- domain features. INTERSPEECH, 2021.
[38] H. Kim, W. Choo, H. Jeong, and H. O. Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. ICL., 2021. 5,
[39] J. H. Kim, W. Choo, and H. O. Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. ICML, 2020. 1, 1,3, 5, 9, 27, 29
[40] S. Kim, G. Lee, S. Bae, and S.-Y. Yun. Mixco: Mix-up contrastive learning for visual represen- tation. arXiv preprint arXiv:2010.06300 2020.
[41] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[42] K. Lee, Y. Zhu, K. Sohn, C.-I. Li, J. Shin, and Lee. $iS-mix: domain-agnostic strategy for contrastive representation learning. ICLR, 2021. 1,2
12

---

[43] J. Li, R. Socher, and S. C. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. ICLR, 2020.
[44] Z. Liu, S. Li, G. Wang. C. Tan, L. Wu, and S. Z. Li. Decoupled mixup for data-efficient learning. arXiv preprini urXiv:2203.10761, 2022. I
[45] Z. Liu, Li, D. Wu, Z. Chen, L. Wu, J. Guo, and S. Z. Li. Automix: Unveiling the power of mixup. ECCV, 2022. 1, 29
[46] C. Ma and L. Ying. On linear stability of sgd and input-smoothness of neural networks. NeurIPS, 2021. 5, 6
[47] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. ECCV, 2018.
[48] I. Medennikov, Y. Y. Khokhlov, A. Romanenko, D. Popov, N. A. Tomashenko, I. Sorokin, and A. Zatvornitskiy, An investigation of mixup training strategies for acoustic models in asr. Interspeech, 2018.
[49] L. Meng, J. Xu, X. Tan, J. Wang, T. Qin, and B. Xu. Mixspeech: Data augmentation for low-resource automatic speech recognition. ICASSP, 2021. 1
[50] D. Misra. Mish: A self regularized oon-oonotoiic activation function. BMVC, 2020. 2
[51] S.-M. Moosavi-Dezfooli, A. Fawzi, J. Uesato, and P. Frossard. Robustness via curvature regularization, and vice versa. CVPR, 2019. 6
[52] W. Mustafa, R. A. Vandermeulen, and M. Kloft. Input hessian regularization of neural networks. arXiv preprint arXiv:2009.06571, 2020. 6
[53] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. Specaugment:
A simple data augmentation method for automatic speech recognition. Interspeech, 2019.
[54] Qin, Fang, Q. Zhang, W. Liu, X. Wang, and X. Wang. Resizemix: Mixing data with preserved object information and true labels. arXiv preprint arXiv:2012.1110/ 2020. 1, 29
[55] S.-A. Rebuffi, S. Gowal, D. Calian, F. Stimberg, O. Wiles, and T. A. Mann. Data augmentation can improve robustness. NeurlPS, 2021. 22
[56] S. Ren, H. Wang, Z. Gao, S. He, A. Yuille, Y. Zhou, and C. Xie. A simple data mixing prior for improving self- suff-supe learning. CVPR, 2022.
[57] Z. Shen, Z. Liu, Z. Liu, M. Savvides, T. Darrell, and E. Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. AAAI, pages 2216-2224, 2022. 2
[58] Y. Sohn, L. Shang, H. Chen, J. Moon, D. Papailiopoulos, and K. Lee. Genlabel: Mixup relabeling using generative models. arXiv preprint arXiv:2201.02354 2022. 1
[59] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurlPS, 2020.
[60] R. Takahashi, T. Matsubara, and K. Uehara. Data augmentation using random image cropping and patching for deep cnns. IEEE Transactions Circuits and Systems for Video Technology, 30(9):2917-293) 2019. 3,3
[61] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. arXiv preprint /906. 05549, 2019.
[62] Y. Tokozume, Y. Ushiku, and T. Harada. Between-class learning for image classification. CVPR, 2018.1
[63] Y. Tokozume, Y. Ushiku, and Harada. Learning from between-class examples for deep sound recognition. ICLR, 2018. 1
13

---

[64] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers distillation through attention. ICML, 2021. 1, 2,
[65] A. Uddin, M. Monira, W. Shin, T. Chung, S. H. Bae, et al. Saliencymix: A saliency guided data augmentation strategy for better regularization. ICLR, 2021. 5,
[66] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio. Manifold mixup: Better representations by interpolating hidden states. ICML, 2019. 1,29
[67] S. Wager, S. Wang, and P. S. Liang. Dropout training as adaptive regularization. NeurIPS, 2013.
4
[68] D. Walawalkar, Z. Shen, Z. Liu, and M. Savvides. Attentive cutmix: An enhanced data augmen- tation approach for deep learning based image classification. arXiv preprint 3rX1v:2003..3048, 2020. 1,3
[69] R. Wightman.
Pytorch image models.
pytorch-image-mode
2019. 27
[70] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. NeurIPS, 2021.
[71] H. Yao, L.-K. Huang, L. Zhang, Y. Wei, L. Tian, Zou, J. Huang, et al. Improving generalization meta-learning via task augmentation. ICML, 2021. 1,2
[72] S. Yun, D. Han, S. Oh. S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. ICCV, 2019. 2, 6, 9, 22, 28
[73] S. Yun, S. J. Oh, B. Heo, D. Han, J. Choe, and S. Chun. Re-labeling imagenet: from single to multi-labels, from global localized labels. CVPR, 2021.
[74] S. Yun, S. J. Oh, B. Heo, D. Han, and J. Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020.
[75] Zagoruyko and N. Komodakis. Wide residual networks. BMVC, 2016. 10
[76] H. Zhang, M. Cisse, N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. ICLR, 2018. 1, 6, 9,
[77] L. Zhang, Z. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou. How does mixup help with robustness and generalization? ICLR, 2021. 2, 3, 4, 19, 21, 22, 23
[78] L. Zhang, Z. Deng, K. Kawaguchi, and J. Zou. When and how mixup improves calibration. arXiv preprint arXiv:2102.06289 2021. 2
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We
discuss potential negative societal impacts in the appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide code-level implementation details in the appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide the hyperparameter settings the appendix.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We provide them in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "lo9Dh4O6f03", "reviewer": "Reviewer_h7Ks", "summary": "This paper proposes a unified theoretical analysis of mixed sample data augmentation (MSDA). The theory shows that the MSDA training strategy can improve adversarial robustness and generalization of training. Based on these results, this paper proposes generalized MSDAs, a Hybrid version of Mixup and CutMix (HMix) and Gaussian Mixup (GMix), which are simple extensions of Mixup and CutMix. Where HMix unifies Mixup and Cutmix and uses the Beta distribution to determine the hybrid strategy hyperparameters, GMix is a smoother hybrid strategy.", "questions": "- The advantage of the beta distribution in HMix is not explained, simply because the Beta distribution is more commonly used and the Beta distribution has a closed-form of conjugate representation for easy derivation?\n- The proposed strategies do not consider the label mismatch problem [4]; they both construct hybrid strategies by randomly selecting a box or a point. If the mixed samples and labels do not match, does it affect the theoretical derivation of conclusions?\n- Both proposed strategies do not take full advantage of the previous conclusions, for example, MSDA is data-dependent, but both hybrid strategies are data-independent.\n- The effect of hyperparameters such as \\alpha, r, \\lambda, etc. in the hybrid strategy is not fully explored.\n\n[4] Liu, Zicheng, et al. \"Automix: Unveiling the power of mixup.\" European Conference on Computer Vision, 2022.", "limitations": "Based on previous theoretical works on mixup methods, this paper draws some interesting conclusions. However, the proposed method is very similar to the already published SmoothMix and the methods compared in the experimental section are not adequate. Therefore, I think this article is not up to the acceptance criteria.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper is well written and organized.\n-  Detailed theoretical derivations support the ideas of the paper.\n\n### Weaknesses\n- Although the paper draws some conclusions through theoretical derivation, the proposed method is too similar to SmoothMix [1].\n- Some classical baselines, such as ManifoldMix [2], are missing from the experimental comparison methods. in addition, some new mixup methods of comparison are missing, such as SaliencyMix [3] (comparable in computational complexity to vanilla mixup), etc.\n- Some problems of theoretical derivation (see the questions for details).\n\n[1] Lee, Jin-Ha, et al. \"Smoothmix: a simple yet effective data augmentation to train robust classifiers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020.\n\n[2] Verma, Vikas, et al. \"Manifold mixup: Better representations by interpolating hidden states.\" International Conference on Machine Learning. PMLR, 2019.\n\n[3] Uddin, A. F. M., et al. \"Saliencymix: A saliency guided data augmentation strategy for better regularization.\" International Conference on Learning Representations, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ei5ImxsJN0p", "reviewer": "Reviewer_4bQe", "summary": "The authors develop and unified theory to explain the various data augmentation (DA) schemes used in Deep network training literature. Particularly, taking motivation from mixed sample strategies (MSDA) and cropping based approaches such as CutMix, the authors propose a loss formulation that considers any non-DA loss, and can be used to generalize various masking and mixing approaches to DA. They show that the CutMix and MSDA losses can then be interpreted through first- and second-order gradient regularization, thereby providing a framework to incorporate/develop other DA schemes. They propose two alternatives DA schemes, HMix and Mix, that result in improved performance of certain classification tasks. \n\nWhile I did not explore these results completely, the inclusion of results on n-sample DA in the Supplementary helps broaden the scope of the future work/applications of the proposed results in the paper. ", "questions": "While I understand the space constraints present, some points in the Theorem’s proof could still be discussed in the Main paper, to help better understand the results — I’m not sure if this a standard assumption in the field, but it seemed unclear to me why one would need $E_{r_x}[r_x] = 0$, or if there are any specific practical implications to having this assumption. Could we guarantee this on the datasets? \n\n", "limitations": "I found the paper to be self contained and the authors seems to have sufficiently discussed most topics that would appear to be open ends for future work, such as explore scenarios with negative M in MSDA, etc. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " - While my expertise in DA is not specific to loss formulation and analysis topics covered in the paper, the results presented in the paper help improve the overall understanding of how various DA schemes affect the learning objective, which is clearly presented. \n - To the best of my knowledge, the proposed general loss function proposed is a significant contribution and should help the community analyze other DA schemes as well, those not considered in the paper. \n - The empirical results on HMix and GMix help validate the formulation developed.\n - The paper is generally well written, with a clear flow of thoughts and presentation, and is clear to read even for those with minimal expertise in the particular sub-field that the papers is about. \n - Minor Issue: The paper could use a bit of proofreading to fix typos here and there, eg. L147 can be benefit -> can be beneficial. \n\n==== x ==== x ==== x ====\n\nPost-Rebuttal comments: I have gone though the author's responses to my and the other reviewers' comments and as there is an overall consensus that the paper has good merit and is deserving of an acceptance, I will keep my score to accept the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YwHPm7xRk0v", "reviewer": "Reviewer_ynng", "summary": "This paper extends the previous theoretical analysis on Mixup to all Mixup-based methods with data-agnostic mask selection methods. Inspired by the previous analysis on Mixup, this work also shows that Mixup-based methods help improve generalization and robustness performances. And this paper shows that Mixup-based methods can be viewed as the regularizer of input gradient and Hessian as well as the first layer parameters. From such a view, this work further investigates that different Mixup-based methods have different effects as the regularizer on input gradient and Hessian and hence there is no optimal choice for all the cases. Stand on that, this paper proposed two combinations of Mixup and CutMix that combine the advantages of both of them.", "questions": "1. I didn't find Theorem 4 in the Appendix. Do Theorem 3 (a) and (b) in the Appendix correspond to Theorem 4?\n\n2. For understanding Fig. 4, could you add the meanings of the x & y-axes and the color bar? Are the x & y-axes mean the distances between pixels along a specific dimension instead of the location of a pixel?\n", "limitations": "1. This paper claims that Mixup & CutMix are good for different cases. But it only provides the results of their proposed methods when CutMix is the better choice. It would be better to provide the case when Mixup is alternatively the better choice to further support that the proposed methods combine the advantages of both these two methods.\n\n2. The captions of figures could be more detailed.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Extensive analysis to extend previous theoretical understanding of Mixup to most of the Mixup-based methods.\n2. Empirical results to support their theoretical findings.\n3. The overall flow is good, every time I have a question I see the answer.\n\nWeakness:\n\n1. The proposed new methods appear to have marginal improvements over current results, although the current best one - Stochastic Mixup & CutMix also supports their claims.\n2. Some of the parts could be written in a better manner,  e.g., the explanation in Fig. 4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IjVRAOK1Pbk", "reviewer": "Reviewer_phJK", "summary": "This paper proposes unified theoretical analysis of mixed sample data augmentation (MSDA) by extending Zhang et al. (2021) that analyzed theoretical model of Mixup. The authors show that MSDSA can improve performance by regularization effects in gradient and Hessian. Furthermore, they propose new augmentation methods, HMix and GMix, based on their theoretical analysis. Experimental results show that HMix and GMix outperform the previous MSDA methods in CIFAR-100 and ImageNet.", "questions": "1. Is second-order Taylor approximation of loss function valid? Zhang et al. (2021) suggest that the approximation is valid for Mixup. Can their result be generalized to other MSDA?\n\n2. Can results of Theorem 1 are applied to other MSDA methods such as ResizeMix (Qin et al., 2020), Fmix (Harris et al., 2021), or PuzzleMix?\n\n- Qin, J., Fang, J., Zhang, Q., Liu, W., Wang, X., and Wang, X. (2020). Resizemix: Mixing data with preserved object information and true labels. arXiv preprint arXiv:2012.11101. \n- Harris, E., Marcu, A., Painter, M., Niranjan, M., Prügel-Bennett, A., and Hare, J. Fmix: Enhancing mixed sample data augmentation. ICLR, 2021.\n", "limitations": "This paper does not consider online optimization of mask design $M$ or $N$. It may be an interested topic for future works.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- This is a first theoretical study that proposes unified theoretical analysis of MSDA. The authors show theoretical reason why MSDA works. Furthermore, they show that Mixup and CutMix have different regularization effects in gradient and Hessian.\n\n- The proposed HMix and GMix are consistent with their theoretical analysis. HMix and GMix outperform the previous MSDA methods in CIFAR-100 and ImageNet.\n\n- The paper is clearly written.\n\nWeaknesses: \n\n- Theorem 1 requires second-order Taylor approximation. This paper does not consider the validity of this approximation.\n\n- The authors do not provide $\\alpha_{ij}$ of HMix, while they provide the one of Mixup, CutMix, and GMix.\n\n- The authors argue that potential negative societal impacts are discussed in the appendix. However, I cannot find the discussion in the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept