A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training
Authors: Randall Balestriero
Ishan Misra
Yann LeCun
Venue: NeurIPS 2022 Accept (2022)
Keywords: data augmentation
regularization
understanding
implicit regularization

Abstract: Data-Augmentation (DA) is known to improve performance across tasks and datasets. We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA? How does the augmentation policy impact the final parameters of a model? We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model's output under a given DA distribution. Up to our knowledge, we obtain the first explicit regularizer that corresponds to using DA during training for non-trivial transformations such as affine transformations, color jittering, or Gaussian blur. Those derivations open new avenues to quantify the benefits and limitations of DA. For example, given a loss at hand, we find that common DAs require tens of thousands of samples for the loss to be correctly estimated and for the model training to converge. We then show that for a training loss to have reduced variance under DA sampling, the model's saliency map (gradient of the loss with respect to the model's input) must align with the smallest eigenvector of the sample's covariance matrix under the considered DA augmentation; this is exactly the quantity estimated and regularized by TangentProp. Those findings also hint at a possible explanation on why models tend to shift their focus from edges to textures when specific DAs are employed.

A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training
Randall Balestriero Meta AI Research, FAIR
Ishan Misra Meta AI Research, FAIR NYC, USA imisra@meta.com com
Yann LeCun Meta Al Research, FAIR, NYU NYC, USA
NYC, USA
rbalestrieroomett..
ylecundmeta.c com
Abstract
Data-Augmentation (DA) is known to improve performance across tasks and datasets. We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA? How does the augmentation policy impact the final parameters of model? We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model's output under given DA distribution. Up to our knowledge, we obtain the first explicit regularizer that corresponds to using DA during training for non-trivial transformations such as affine transformations, color jittering, or Gaussian blur. Those derivations open new avenues to quantify the benefits and limitations of DA. For example, given loss at hand, we find that common DAs require tens of thousands of samples for the loss to be correctly estimated and for the model training to converge. We then show that for training loss to have reduced variance under DA sampling, the model's saliency map (gradient of the loss with respect to the model's input) must align with the smallest eigenvector of the sample's covariance matrix under the considered DA augmentation; this is exactly the quantity estimated and regularized by TangentProp. Those findings also hint at a possible explanation on why models tend to shift their focus from edges to textures when specific DAs are employed.
1 Introduction
Data Augmentation (DA) is a prevalent technique in training deep learning models ||| 3 These Deep Networks (DNs) models f-. governed by some parameters Y E T, are trained on the train set and expected to generalize to unseen samples (test set). To combat the tendency of DNs to overfit [4], producing a large train-test performance gap, regularization and in particular DA is heavily employed, amongst other mechanisms such as weight-decay 5 The benefit of DA over those alternatives is that defining input transformations that preserve the semanties of their inputs a relatively simple task, at least in computer vision or acoustic processing 7 g 9. Furthermore, if the DA is well designed and rich enough, can effectively bring the number of non-trivial training samples close to the theoretical limits ensuring that any performance gap between the train and test set vanishes [0 as empirically observed various scenarios [ [2]. fact, DA has proven so useful that novel training methods such as self-supervised learning (SSL) entirely rely on DA 13 to learn meaningful data representations.
Despite its empirical effectiveness, our understanding of DA has many open questions, three of which we propose to study: (a) how do different DAS impact the model's parameters during training?: (b) how sample-efficient is the DA sampling, i.e., how many DA samples a model must observe to converge?; and (c) how sensitive evolves during training as a function of the model's ability to minimize the loss at hand, and as
loss/model to the DA sampling and how this variance
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

zoom = 0.6
transformed
model transformation an age through a matrix vector multipli- of lltttened image (red) and
sparse matrix (bluc) representing the transform.
original image!
0.0.8
expected image 2.(
Figure 1: We propose novel way express Data Augmentation (DA) analytically that allows understand the impact of DA the learned parameters of model and quantify DA's sample efficiency. This allows us compute the analytical expectation and variance the transformed data (or any function computed with it) in closc-form with respect to the transformation parameters.
function of the model's parameters? Our goal is analytically derive some preliminary answers and insights into those three areas enabled by a novel image transformation operator that we introduce in Section 3.2 coined Data-Space Transform (DST). DST employs mathematical formulation of input transformations akin to the one of [15] although slightly more general as we do not require the DA to form group. We study the hww.//rranaatiionnhaaairrrrrrrio
DAs in the main text and Gaussian blur, grayscale, color jitter and random crop in Appendix C code is available"] We summarize our contributions below:
(a) we derive the analytical first order moments of augmented samples, and of the losses employing augmented samples (Section 3.3). effectively providing us with the explicit regularizer induced
by each DA (Section (34]
(b) we quantify the number of DA samples that are required for a model/loss to obtain a correct estimate of the information conveyed by that DA (Section
(c) we derive the sensitivity, i.e. variance, of a given loss and model under DA policy (Section 4.2
leading us rediscover from first principles popular deep network regularizer: TangentProp, as being the natural regularization to employ to minimize the loss variance (Section
Upshot of gained insights: (a) will show us that the explicit DA regularizer corresponds to gener- alized Tikhonov regularizer that depends on each sample' covariance matrix largest eigenvectors; the kernel space of the model's Jacobian matrix aligns with the data manifold tangent space as modeled by the DA; (b) will show us that the number of augmented samples required for a loss/model to correctly estimate the information provided by a DA of single sample is on the order of 10*. Even when combining the information of thousands of samples, we find that the entire train set must be augmented at least x0x for DA policy to be correctly learned by model; lastly (c) will quantify the loss variance as simple function of the model's Jacobian matrix, and the eigenvectors of the augmented sample variance matrix. Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down as the kernel a model's 's Jacobian matrix aligns with principal directions of the data manifold tangent space; a by- by-product, recover TangentProp as the natural regularizer minimizing the loss sensitivity to DA sampling.

---

Limitations: all of our image-space results e.g. expectation and variance of images under DA are exact, no approximation is employed. Results on a model's output/loss are provided both in the case of linear model and of nonlinear model. For the former, results are exact, for the latter third-order Taylor approximations employed which is common and has been shown to provide accurate enough approximations to safely rely on the insights/quantitess obtained e.g. see Sec. A.2 of [6]. All the proof' and implementation details are provided in the appendix. We also emphasize that our primary goal is to provide theoretical understanding of DA, to quantify its efficiency and impact onto learned models and to derive and visualize its explicit regularizer; we leave the search for tractable alternatives to DA based on those insights for future work.
2 Background
Existing Explicit Regularizers From Data-Augmentation. It is widely accepted that data- augmentation (DA) regularizes model towards the transformations that are modeled [7], and that this regularization impacts performances significantly and positively, possibly as much as the regularization offered by the choice of DN architecture [1]] and optimizer [19].
To gain precious insights into the impact of DA onto the learned functional fn, the most com- mon strategy is derive the explicit regularizer that directly acts upon fn in the same manner as if one were to use DA during training. This explicit derivation is however challenging and DA strategies such additive white noise or multiplicative binary
so far has been limited noise applied identically and independently throughout the image and/or feature maps, as with dropout 120 21). In those settings, various works have studied in the linear regime the relation between such DA and its equivalence to using Tikhonov regularization [22] or weight decay [23] as
in minw En= VE-AV(.)) Ilyn. (..,. +
<)]l²] = minw > |/ Wa 12 + X(o) [W]?
[24] More recently, (25] 26] extended the case of additive white noise to nonlinear models and con-
cluded that additive white noise DA corresponds to adding an explicit Frobenius norm regularization
onto the Jacobian of fe evaluated at each data sample.
Going to more involved DA strategies e.g. translations zooms of the input images is challenging
and has so far only been studied from an empirical perspective. For example, 27 28 29 performed
thorough ablation studies the interplay between DAs and collection of known explicit regularizers to find correlations between them. It was concluded that weight-decat decay (the explicit regularizer of additive white noise) does not relate to those more advanced DAs. This also led other studies to suggest that norm-based regularization might be insufficient to describe the implicit regularization of DAs involving advanced image transformations
(30 We debunk this last claim in Sec. 3.3
Coordinate Space Transformation. Throughout this paper. we will consider a two-dimensional image [xx v) to be at least qquare-integrabee I E (RR2)) [31). Multi-channel images are dealt with by treating each channel as its own single-channel image, as commonly done in practice M- As we are interested in practical cases, will often assume that / has compact support e.g. has nonzero values only within a bounded domain such as [0, 1]2. Visualizing this image thus corresponds to displaying the sampled values of I on regular grid (pixel positions) of [o. 1 (32. The most common formulation to apply a transformation on the image / to obtain the transformed image T is to transform the image coordinates 33 34 35 That is, a mapping L: R2 - R² describes what coordinate ((u, v) of the original image maps to the coordinate u, v of the transformed image T as
in
T(u, v) [t((u,
(I)
This mapping often comes with some parameters 0 governing the underlying transformation as in coos(2 sin (0)
sin(0)
[x]]]
to (z v) [x 01. y 0z|T, to(x, v)
to 3) [01z, Ozyi"" (2)
cos(
y
for translation, rotation and zoom respectively. We provide visual depiction of the zoom transfor- mation applied in coordinate space in Fig. 6 in the appendix.
The formulation of Eq. has two key benefits. First, it allows simple and intuitive design of t to obtain novel transformations. Second, of images are 2/3 //3-dimmesi.onl. Those benefits have led to e.g. the Spatial Transformer Network 36). On the other hand, Eq. Taas one major drawback for our purpose: the exact moments of the transformed image under random 0 parameters are not tractable due to the composition of / with the nonlinear mapping T. And as it will become clear in Section 3.1 those quantities are needed to derive DAs' explict regularizers.
computationally efficient as the coordinate- space
3

---

+
+
+
original
hor+ver. transl. horizontal shear
vertical shear
rotation
700m
Figure 2: Top row: sparse matrix Eg[M producing the expected image via Eg [t(]] = Eg [M)]
(recall Eq. 7 and the discussion below Theorem 3.3 here with DA distributions of N'0 0.04) @ N (0, 0.01).
(o,,, 0.2). N (0,, 0.1) and (V1, for each column respectively. This sparse matrix independent of the image and derived in forse-form (Theorem 3.3). Middle & Bottom row: are two images that depict each column the "expected image" under each DA. Estimating those images (and any loss employing them) only through sampling the DA augmentations would require tens of thousands of samples (see Fig. 4). Gaussian blur. grayscale, color jätter and random crop DA are provided in Appendix along with discussions how to extend results CutOut and MixUp.
Analytical Moments of Transformed Images Enable Infinite Dataset Training
We first motivate this study by formulating the training process under DA sampling as doing Monte-Carlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1). Going beyond this sampling/estimation procedures requires knowledge of the average and variance of transformed sample, i.e. the first two centered-moments. This is our motivation to construct novel Data-Space Transformation (DST) (Section[32) that will enable close-form formula for those moments (Section (.3). From those, we will be able remove the need to sample transformed images to train a model, by obtaining the close-form expected loss Section 3.4 under which training can be performed sampling-free.
3.1 Motivation: Current Data-Augmentatiin Training Performs Monte-Carlo Estimation
Training model with DA consists in (i) sampling transformed images each training iteration for each sample Vn as in To. (xx) ) with 0, ~ 0 randomly sampled DA parameter c.g. the amount of translation to apply, (ūi) evaluating the loss L on the transformed using some flavor of gradient descent to update the parameters of the model fn-
and (iii)
This training procedure corresponds to one-sample Monte-Carlo (MC) estimate expected loss
37 38 of the
N Ee (f ~
N
((F,(Te
(3)
n=1
1=1
with f2 i.i.d samples of repeatedly sampled at each epoch. In supervised setting, the loss would also receive a per-sample target yn as input. Although a one-sample estimate might be insufficient to apply the central limit theorem [39] and guarantee training convergence, the combination of multiple samples in mini-batch training does provide convergence in most cases. Although, methods such Self-Supervised Learning that heavily rely DA, tend to be more sensitive and unstable [40].

---

To avoid such instabilities. and to incorporate all the DA transformations of In into the model's s parameter update, one would be templed to compute the model parameters' gradient on the expected loss (left-hand side of Eq. (see Section 3.4). Knowledge of the expected loss would also prove useful to measure the quality of the MC estimate (see Section 4.1) J. Note that even in the least-square setting, such approximation is employed as the expected loss under DAs is not know. Hence, one commonly obtain many DA samples, and then apply the least-square formula on this augmented training set, which is highly inefficient. Alternatively, this study will obtain the optimum least-square parameters under DA sampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parameters that would correspond to sampling infinitely many DA samples and applying the usual least-square formula on this infinite training set.
The close-form expected loss requires knowledge of the average and variance of the transformed sample (T(,,, taken with respect to the random variable 0. Hence, we first propose formulate novel and tractable augmentation model (Section 3.2 that will allow us to obtain those moments analytically Section 3.3
3.2 Proposed Data-Space Transformation (DST)
Instead of altering the coordinate positions of an image, done in the coordinate-space transformation of Eq. (T) we propose to alter the image basis functions.
Going back to the construction of functions, one easily recalls that any (image) function can be expanded into basis as in I (u, 2) = I(x, y) xz wー y)dzdy with 5 the usual Dirac distribution. Suppose for now that consider horizontal translation by constant A. Then, can obtain that translated image via
To (T) = f [(x, v),()) -, -2 y)dddy,
(4)
hence, and crucial to our study, we apply the transformation onto the basis functions onto which the image is evaluated, rather than onto the original image itself. As the image is now constant with respect to the transformation parameter 0, and as the basis functions have some convenient analytical forms, the derivation of the transformed images moments, under 0, will become straightforward. Because the transformed image T is now obtained by combining its pixel values (recall Eq. (4) we coin this transform as Data-Space Transform (DST), formally defined below.
Definition 3.1 "Data-Space Transform). We define the DST of image I E (RR) producing the transformed image [I) E L2( as
Ta())u,, = / [(x,))) (u, u, ydddyy, /
(5)
with hu, U, .,). E C7 (R²) encoding the transformation.
In Definition 3.1 we only impose for ho .).,.) ) to be with compact support. In fact, one should interpret ho (u, U, -) as a distribution whose purpose is to evaluate / a desired (coordinate) position on its domain. This evaluation -depending on the form of ha(u, -)- can extract a single pixel- value of the image I at a desired location (as in Eq. (4). ). or can combine multiple values e.g. with he u, .,.,, .) being a bump function. Additionally, we do not impose any restriction on the invertibility of this mapping i.c. (o( (u, v) can disregard parts of the original image.
Coordinate-space transformations as DSTs. The coordinate posed DST (Definition (.1) act in different spaces: the image coordinates and the image pixel values, respectively. Nevertheless, this does not limit the range of transformations that can be applied to an im- age. The following statement provides simple recipe turn any already employed coordinate-space transformation into a DST.
transformation and the pro-
Proposition 3.2. Any coordinate-space transformation 7 using La 22 IR2 can be expressed as data-space transformation 5 by setting ho(u, v, x. y) 5(t (1, y) [u, r)
We will focus in the main text on zomm/otaiio/traaaaiiooosaaaiiiii the case of Gaussian blur, color jittering, random crop, along with discussion on extending our results to CutMix 41] and MixUp [2]. Using Proposition 3.2 obtain the DST operators ho(u, v, v)
and we propose in Appendix C
5

---

to be
8(u + 01, 02); 8(u x - 0iv, 02ar), 8(u 0x, 0y), 8(u (0) sin(0) yy.y sin( cos(F)y)
(6)
for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively (compare with Eq. 2 Before focusing the analytical moments of the DST samples, we describe how those operators are applied a discrete setting.
Discretized version. We now describe how any DST of discrete image, flattened as vector, can be expressed as a matrix-vector product with the matrix entries depending on the employed DA and its parameter 0, as we display in Figs. and 2 The functional form of the DST from Eq. (5) producing the target image at a specific position To (T) v) is linear the original image T. Hence, in the discrete setting, the continuous integral over the image domain is replaced with summation with indices based on the desired sampling/resolution Expressing linear operators as matrix-vector products will greatly ease our development, we will denote by æ E hw the flattened (h w) discrete images I. Hence, our data-space transformation, given some parameters 0, takes the form of
i.e. the position of the pixel spatial positions.
t(0) M (f)x,
with t(0) E TRe the flattened transformed image (which can be reshaped as desired) and M()) E Re x hw the matrix whose rows encode the discrete and flattened ho v,.,.. For example, and employing uniform grid sampling for illustration, M (f) = holi//w: i%... j//w.//wwwww j%w) with // representing the floor division and % the modulo operation. For the case of multi-channel images, we consider without loss of generality the application of the matrix-vector operation on each channel separately. We depict this operation along with the exact form of (0) for the case of the zoom transformation in Fig. D A crucial property of the matrix (0) lies in its sparsity, in fact, for most DAs the transformations rely on displacing pixels rather than combining them. Hence, although M (0)'s 's total number of entries grow quadratically with the number of pixels in I, the number of nonzero entries only grows linearly with
3.3 Analytical Expectation and Variance of Transformed Images
The above construction (Definition 3.I turns out to make the analytical form of the first two moments of an augmented sample much simpler to derive. As this derivation at the core of our main contribution, we propose step-by-step derivation of Egg [T)( for the case of horizontal translation (recall Eq.
Let's consider again the continuous model (the discrete version is provided after Theorem 3.3 Fubini's 's theorem switch the order of integration and recalling Eq. (5). we have that
Using
Es (T) v)] =/ [(x, E) (uo v, x. v) drdy.
(8)
Using the definition for he from Eq. (6) for horizontal translation, Eg [he (u, v. v)] becomes
Eg 1ō(u 0,v v 5) 6(u 0,v v))(O)d0 p(u x)0(f= v):
with p the density function of 0 prescribing how the translation parameter is distributed. As result, in this univariate translation case, the expected augmented image at coordinate (u, v) is given by
Eo [To (T), v)] = ((c.v)p(u 2)5(U - y)drdy = / T v)p(u dr)dr,
which can be further simplified into Ea (T) w)] = I[., u) * p. Hence, the expected horizontally- translated image is the convolution (on the z-axis only) between the original image and the univariate density function We formalize this for the transformations of Eq. (6) below.
Theorem 3.3. The analytical form of Eg [ u, v)}, used to obtain the expected transformed image (recall Eq. (8) is given by
p(u Z. y) (translation), and p(u/z)6(u/z- -v// (rotation),
(10)
other cases and second-order moment (oT are deferred to the proof. (Proof in Appendix F

---

intation
diilion
316
132
rank of V Le
-84
1.6
1.2
transformation ampl.
Figure 3: First column: top eigenvectors (Q matrix in Theorem 3.4) of the sample variance matrix (=]( for rotation/shear augmentations with 44(155 15") and (115",,,111))) @ u 15") respectively. As per Theorem 4.1 training aligns the model's Jacobian matrix kernel to the largest eigenvectors of V[T(=)) reduce the loss variance under respective DA sampling. Second column: number nonzero eigenvalues of (a) (nonzero elements in A in Theorem 3.4p for increasing transformation amplitude (the datum dimension is 114 114 3). We observe that the dimension of the subspace spanned by augmented images constraint increases linearly with the amplitude of the transformation. Third column: Pixel variance i.e. diagonal of (=)) reshapes an image. Fourth column: pixel covariance between one background pixel and all other pixels, i.e. th row of V[T (x) reshaped an image for the cat of Fig. under translations (O 0.1) @ (0, 0.1)). As per Theorem 3.4 the variance of the pixel values,
the left, being much interior body/texture pushes the model focus on the cat texture,
higher edges of cat than phenomenon empirically observed deep networks 43
An interesting observation obtained from the above ties the expected image under random 2- dimensional translations with (2-dimensional) density to the convolution / p: this provides new portal to study and interpret convolutions with nonnegative, sum-to-one filters p. Again, and as per Eq. 77). the discretized version the expected image takes the form Eg [t)] = Eg (0)aa with the entries of E&g M (0) given by discretizing Eq. (10). We depict this expected matrix for various transformations as well as their application onto two different discrete images in Fig. 2 and
now proceed deriving the left-hand side of Eq. i.e. the explicit DA regularizer.
The Explicit Regularizer of Data-Augmentations
To keep notations as light as possible, we first consider linear regression model with Mean Squared that setting, the expected loss under
Error (MSE), the nonlinear case will studied in Section[4] DA sampling (recall Eq. (3)) becomes
N Σ n=1
Eo
In WTo(x.)) b12
(11)
with Dn IR 1, ,,....N the input (flattened) nth image Ins yn R the nh target vector, and W E R b E R the model's parameters. Recall from Section 3.1 that to learn W under DA, the current strategy consists in performing a Monte- Monte-Carlo (MC) estimation. Instead, let's derive (detailed derivation in Appendix D the exact loss of Eq. (II) as function of the sample mean and variance under the consider DA. We will drop the 0 subscript for clarity to obtain
L- Σ Un WE WE - bl [[Q
(12)
with the spectral decomposition (()))()) = (T())). The right term in Eq. 12 is the
explicit DA regularizer. It pushes the kernel space of W to align with the largest principal directions of the data manifold tangent space, as modeled by the DA. In fact, the largest eigenvectors in ((x) represent the principal directions of the data manifold tangent space at æ, encoded via V TT (x)).
We propose in Fig. visualization of Q and for different DAs, illustrating how each DA policy impacts the model's parameter W through the regularization of Eq. (12). The knowledge of E [Tx))
and V (x()) from Theorem 3.3 finally enables to train (linear) model on the true expected loss
(Eq. 12 as we formalize below.

---

Theorem 3,4. Training a linear model with MSE and infinite DA sampling is equivalent to minimizing Eq. [2) and produces the optimal W model's parameter
N w' Σ yn - - (x
N
N Σ
Σ
E BIT ((T(EN))
V V[T(xn))
(13)
Not surprisingly, whenever the DA is identity, [xT( zn. V[T (ax))) 0 and thus Eq. 13 recovers exactly the standard least-square solution. We visualize V[7 (a,)) for the translation DA in Fig. The same line of result can be derived in the nonlinear setting by assuming that the DA is
restricted to small transformations. In that case, one leverages a truncated Taylor approximation of the nonlinear mode and recovers that (local) DA applies the same regularization in Eq. 12 but with the model's s Jacobian matrix J (zx()) in-place of W (more details in Section 4.3
Given the above derivation of the exact expected loss, we can now turn to precisely measure how accurate is the MC estimate commonly used to train models under DA sampling.
Data-Augmentation Sampling Efficiency and Loss Sensitivity
In this section we study the convergence of the MC estimate (Section [4.)). and provide variance analysis of that estimate a function of the model's Jacobian matrix and the sample variance eigenvectors (Section (.2), concluding by the (re- dessovvrryvyy of TTngeetPrrp from first principles (Section Those results hold regardless of the type of DA employed. Additionally, the analysis is done on a single sample to case notation, in the i.i.d. setting, simply sum over all the training samples to analyze the training set.
4.1 Empirical Monte-Carlo Convergence of Transformed Images
Given the close-form average image and average loss, we empirically measure how efficient is the MC estimation from Eq. (3)
We first propose in Fig. 4 constructed (64 x 64) image for which we compute the expected loss (Eq. (12) and the MC estimate (right-hand side of Eq. 33). Surprisingly, we obtain that even for such a simple image and translation DA, between 1000 and 10000 samples are required to correctly estimate the MSE loss from the augmented samples. argue that the combination of the DA samples from different images allows to obtain better estimate with a smaller amount of augmentations per sample. Hence, we provide in Fig. [5] that experiment using linear model on MNIST 45 with varying train set size. We observe that as the number of samples grows as the required number of augmentation per sample reduces. Nevertheless, even with thousands of samples, at least 50 augmentations per sample are required to provide an accurate estimate (reproduction with nonlinear models given in Fig. 7
more practical scenario, one could rightfully
We now propose to specifically quantify the sensitivity of the MC estimate to DA sampling.
4.2
Loss Sensitivity Under Pata-Augmentation Sampling in the Linear and Nonlinear Regime
Recalling Section 3.1 current DA training performed on a MC estimate of the loss. The estimator's variance [46] is proportional to the variance of the quantity being estimated: V[(C o fT(T(())) We now characterize when, and why, would an MC estimator converge for a given model.
By leveraging the delta method [47 48 function mapping L f. we have
i.e. a truncated Taylor expansion of the model and loss
V(C () =
- (C ()
(14)
with ||u||
uT Au. Noticing that V[T(x)] E -
and using the
close-form moments of DST samples from Theorem 3.3 it is possible to write out Eq. 14 explicitly
for model and DA specific analysis. We visualize 7 in Fig
Linear regression case. To gain some insight into Eq. (44, let's first consider the linear regression
case leading to V[[ o (T((())) ~ | WE[T (x)) b bl (y)(("))) Hence, the estimated loss variance depends (i) on the model predicting the correct output when observing the expected sample
ET]())) and (ii) on the smallest right singular vector of W to align with the largest eigenvectors of
(T)())) echoing our observation below Eq. 12
as commonly done, see e.g. Sec. A.2 from [6 for justification and approximation error analysis
8

---

original image
expected Es To [w)]
estimared (N = 100)
(N = 10000)
estimated
Figure 4: Analytical expected synthetic image un- der translation (O(0,, 0.1) @ (0, 0.1)) against its N- sample Monte-Carlo estimate (top row), l2 distance be- tween the true and estimated images (bottom left) and the true and estimated MSE loss with a random Gaus- sian y. W (bottom right). In red is depicted the stan- dard deviation of the independent Monte- Carlo runs. Monte-Carlo
2g herweon extimate of the expected image (leff) and expected loss (right)
Clearly observe that even simple (64 64) im- age and using the translation transformation, thousands of sampled images are necessary to provide an accurate estimate of the loss at hand. That is, sampling based data-augmentations are rather inefficient medium to employ for injecting prior information into model this prior information will only emerge after tens thousands of images have been sampled.
13
04
N (log-scale)
N (log-scale)
Figure 5: MNIST with linear model and trans- lation DA with varying number of samples (col- ored lines) against the use of the expected loss (black line). We depict the MSE loss (left) and the accuracy (right) on the test set. At first, even 50x DA-based increase in dataset size fails to provide the performances reached by employing the analytical expected loss. As the training size increases. as the DA samples get redundant with the added samples closing the performance gap (the case of nonlinear model provided Fig. 7 See Tables to 5 for additional table results with standard deviation.
25
00
2.0
Text MNC
20
2.5
1.0
DA samples
semplex
semplles
Un samples
GA semples
De samppes
0.5
10 sampas
50 De samples
30
160
350
1000
3000
100
300
1000
Training 9 Got size
Training set size
General case. Since the result from Eq. (14) holds in the general setting, we can generalize the above observation and formalize it into the following statement.
Theorem 4.1. The variance of the loss's 's MC estimate (recall Eq. 33) for an input on goes to O if the loss gradient ( (x)))) goes to O, or if the kernel of the model's 's Jacobian matrix J ([EE((x))) aligns with the largest eigenvectors of the sample variance V[[(x)).
The proof of the above statement simply starts from Eq. (14 and applies the Cauchy- inequality two times to obtain
V((C (T)()))) < [VC((( X (TT (1T
(15)
with decomposition =
From Eq. [15.. it is quite realistic to assume that (this would not minimize the training loss in general). Hence, the most realistic case of Theorem 4.1 concerns the alignment between the model's Jacobian matrix kernel space and the top eigenvectors of V[T(2)). However, for models like ResNet DNs (4). the loss variance can only reduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is always full-rank. We recall that the above relied on the delta method to approximate the intractable LHS of Eq. (4) (see footnote on Page 7
= Q(=))(()(((x))
the model's Jacobian does not collapse
4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp
The expected loss (Eq. (3) has been derived in the linear regression setting (Eq. (22). But in a more general scenario, and as discussed below Theorem 3.4 this expectation might not be tractable and thus needs to be approximated e.g. based on Taylor expansion of the loss and model. Alternatvely to using the approximated expectation, one could employ the usual MC estimate of the expectation (right-hand side of Eq. (3)). and leverage the MC estimator variance obtained in Eq. (15) as a regularizer. We show here that both approaches are equivalent, and recover a popular regularizer known as TangentProp [50]. Although various extensions of TangentProp have been introduced
[5] 52 no principled derivation of it has yet been proposed. Using the same argument as in the expectation of nonlincar transformation of random variable (T (Txx,) can be
Section 4.2

---

expectation
Train. size (N)
#DA=1
#DA=2
#DA=5
#DA=10
(#DA=oc)
rotatione [-20,20] degrees
trunslations <[-0. E.-0.1.... 15%%
MNIST
N=100
10.39=1.82
11.70+2.02
10.60 1.33
24.30=1192
33.44
15,00
200m E|0.9. 1.1]%
N=1000
21.71=1.61
32.57+1.95
40.61+22.20
45.16+4..38
48.55
N=10000
47.82=2.20
51.98:1133
54.97
56.29:10.66
57.16
EMNIST
N=100
4.48=0.85
4.3610.57
3,85:10440
8.3810.58
13.34
N=1000
8.45=0.53
14.79+0.60
22.51+0.63
27,49+0.60
32.18
N=10000
30.49-0.75
35.63.10.50
39.98.1.449
42.13.10.40
43.30
FMNIST
N=100
11.18-3.31
9,49111.40
10.65-11.66
23.10.12.17
35.31
N=1000
24.43=2.75
38.71
47.49+2.07
12.1
54.92
N=10000
52.60=2.57
$4,9
1..84
1..29:1.44
57.63
5.0005%%
MNIST
N=100
12.72=1.95
12.42++2.02
12.45+1.47
45.82:t1.53
55.83
rotatione [-5.5] degrees
trunslations c[-0.05 e[-0.05.00057)
zoom E[0.98, I. 1.02]%
N=1000
50.44=1.07
64.96+0.95
72.64+0.67
75.06
76.15
N=10000
78.54-0.22
79.7610.24
80.4910.20
80.8510.17
80.86
FMNIST EMNIST
N=100
4.69-0.73
4.77.10.61
4.43
12.79110.51
19.53
N=1000
16.74-0.47
31.1010,29
40.5910.39
44.45:10.34
46.53
N= 10000
49.13=0.30
51.98:00.13
53.62+0.14
54,30+0.116
54.61
N=100
11,51=2.12
12.43+1.99
11,59+
37. 1.203
55.22
N=1000
46.97=1.17
66.22:10.51
72.75+0042
74.56:L0.35
75.03
N=10000
75.2=0.33
75.89:1.227
76.49
76.63+0.18
76.71
Table 1: Reprise of Fig. reporting test set classification performances averaged over 10 runs. with standard deviation corresponding to different DA realizations. For each specific run and configuration, all scenarios have access the exact same training set, #DA represents the number of new samples introduced for each training sample, total training set size thus N (x(1+ (DA).. We observe that the large dataset size regime (N = 10000) the gap between low number of DA samples and the expected regularizer is marginal which is expected as the introduced DA variations become redundant with the training set samples. This marginal gap be slightly increased by employing stronger DA (compare top half and bottom half of the table). However, the small dataset size regime, there is an important gain provided by employing the regularizer even when compared to observing many DA samples (#DA=10). This indicates potentially crucial regime in which introducing the analytical regularizer helps. More surprisingly, this is also the case if part of DA policy is misaligned with the task at hand (e.g. rotation for FashionMNiST), Additional empirical results with training sizes (N E [100, 200, 500, 1000, 3000, 10000, 20000)) and DA policies (one per table) available Tables 5 for MNIST, Tables for EMNIST, and in Tables 10 13 for FashionMNIST
approximated from the expectation of the Taylor expansion (detailed derivations in Appendix from which upper-bound is obtained by applying the Cauchy-Schwarz inequality
BI (T())) < (Co ((([(( + x(x) (E(T o A())
TangentProp regularization
with <<x: 2 O and the notations from Eq. 15 appears when using a Taylor approximation the expected loss, and corresponds adding an
As a result, the TangentProp regularizer naturally
explicit loss variance regularization term (compare the TangentProp with Eq. (15). We thus obtained from first principles that TangentProp emerges naturally when considering the second order Taylor approximation of the expected loss given a DA.
5 Conclusions and Limitations
In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which is possible to study DA and to provably answer some of the open questions around the efficiency and impact of DA to train a model. We first obtained the explicit regularizer produced by different DAs in Section 3.4 This led to the following observation: the kernel space of the W matrix is pushed to align with the largest eigenvectors of the sample covariance matrix. This was then studied in more general setting in Section 4.2 for nonlinear models when characterizing the loss variance under DA sampling. We also observed that Monte-Carle sampling of transformed images is highly inefficient (Section I). even if similar training samples combine their underlying information within a dataset. Lastly, those derivations led us to provably derive a known regularizer -TangentProp- as being the natural minimizer of a model's 's loss variance. The main limitation the current analysis concerns nonlinear models for which we rely on third-order Taylor expansion.
10

---

References
[T] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press, 2016.
[2] J. Kukačka, V. Golkov, and D. Cremers, "Regularization for deep learning: A taxonomy," arXiv preprint 1710. 2017.
[3] R. Balestriero and Y. LeCun, "Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods," arXiv preprint arXiv:2205.11508 2022.
[4] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, "Understanding deep learning (still) requires rethinking generalization," Communications of the ACM, vol. 64, no. 3, pp. 107-115, 2021.
[5] B. Neyshabur, R. Tomioka, and Srebro, "In search of the real inductive bias: On the role of implicit regularization in deep learning," arXiv preprint arXiv:1412.6614 2014.
[6] Loshchilov and F. Hutter, "Decoupled weight decay regularization," arXiv preprint arXiv: 1711v:1711.051000 2017.
[7] Ko, Peddinti, Povey, and S. Khudanpur, "Audio augmentation for speech recognition." " in Sixteenth annual conference of the international speech communication association, 2015.
[8] X. Cui, V. Goel, and B. Kingsbury, "Data augmentation for deep neural network acoustic modeling," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 9, pp. 1469-1477, 2015.
[9] C. Shorten and T. M. Khoshgoftaar, "A survey on image data augmentation for deep learning," Journal of Big Data, vol. no. pp. 1-48, 2019.
[10] M. Unser, "A representer theorem for deep neural networks.." J. Mach. Learn, Res., vol. 20, no. 110, pp. 1-30, 2019.
[11] L. Perez and J. Wang, "The effectiveness of data augmentation in image classification using deep learning," arXiv preprint arXiv: arXiv:1712.04621. 2017.
[12] A. M. Taqi, A. Awad, F. Al-Azzo, and M. Milanova, "The impact of multi-optimizers and data augmentation on tensorflow convolutional neural network performance," in 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 140-145, IEEE, 2018.
[13] I. Misra and L. v. d. Maaten, "Self-supervised learning of pretext-invariant representations," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
[14] J. Zbontar, I. Jing, I. Misra, Y. LeCun, and S. Deny, "Barlow twins: Self-supervised learning via redundancy reduction," arXiv preprint arXiv:2103.032200, 2021.
[15] S. Chen, E. Dobriban, and J. Lee, "A group-theoretic framework for data augmentation," Advances in Neural Information Processing Systems, vol. 33, pp. 21321-21333, 2020.
[16] C. Wei, S. Kakade, and T. Ma, "The implicit and explicit regularization effects of dropout," 2020.
[17] B. Neyshabur, "Implicit regularization in deep learning," arXiv preprint arXiv:1709.0195: 2017.
[18] S. Gunasekar, J. Lee, D. Soudry, and N. Srebro, "Implicit bias of gradient descent on linear convolutional networks," arXiv preprint arXiv:1806.00463. 2018.
[19] D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro, "The implicit bias of gradient descent on separable data," The Journal of Machine Learning Research, vol. 19, no. 1, pp. 2822-2878, 2018.
11

---

[20] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: simple way to prevent neural networks from overfitting," The journal of machine learning research, vol. 15, no. 1, pp. 1929-1958, 2014.
[21] X. Bouthillier, K. Konda, P. Vincent, and R. Memisevic, "Dropout as data augmentation," arXiv preprint arXiv:
2015.
[22] A. N. Tikhonov, "On the stability of inverse problems." in Dokl. Akad. Nauk SSSR, vol. 39, pp. 195 1955-18,11 1943.
[23] G. Zhang, C. Wang. B. Xu, and R. Grosse, "Three mechanisms of weight decay regularization." arXiv preprint arXiv: 1810. 12081, 2018.
[24] Baldi and P. Sadowski, "Understanding dropout," Advances in neural information process- ing systems, vol. 26, pp. 281 2814-282 2822, 2013.
[25] Z. Wang, R. Balestricro, and R. Baraniuk, "A max-affine spline perspective of recurrent neural networks," in International Conference on Learning Representations, 2018.
[26] D. LeJeune, R. Balestriero, H. Javadi, and R. G. Baraniuk, "Implicit rugosity regularization via data augmentation." arXiv preprint 19Xv::900.116333 2019.
[27] A. Harrnneez-Greía and König, "Do deep nets really need weight decay and dropout?," arXiv preprint arXiv:1802.00042, 2018.
[28] A. Hernández-Garíía and P. König, "Data augmentation instead of explicit regularization," 2018.
arXiv preprini
[29] A. Hernández-Garchh and P. König, "Further advantages of data augmentation convolutional neural networks," in International Conference on Artificial Neural Networks, pp. 95-103, Springer, 2018
[30] N. Razin and N. Cohen, "Implicit regularization in deep learning may not be explainable by norms," arXiv preprint 201i::200000639 2020.
[31] S. G. Mallat, "Multifrequency channel decompositions of images and wavelet models," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 12, pp. 2091-21100 1989.
[32] Heckbert, "Color image quantization for frame buffer display," ACM Siggraph Computer Graphics, vol. 16, no. 3, pp. 297-307, 1982.
[33] A. A. Sawchuk, "Space-variant image restoration by coordinate transformations." JOSA, vol. 64, no. 2. pp. 138-144, 1974.
[34] G. Wolberg and S. Zokai, "Robust image registration using log-polar transform." " in Proceedings 2000 International Conference on Image Processing (Cat. No. 00CH37101), vol. 1. pp. 493-496, IEEE, 2000.
[35] R. Mukundan, S. Ong, and P. A. Lee, "Image analysis by tchebichef moments," IEEE Transac- tions on image Processing, vol. 10, no. 9, pp. 1357-1364, 2001.
[36] M. Jaderberg, K. Simonyan, A. Zisserman, et al., "Spatial transformer networks," " Advances in neural information processing systems, vol. 28, pp. 2017-2025, 2015.
[37] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and Teller, "Equation of state calculations by fast computing machines," The journal of chemical physics, vol. 21, no. 6, pp. 1087-1092, 1953.
[38] W. K. Hastings, "Monte carlo sampling methods using markov chains and their applications," 1970.
[39] M. Rosenblatt, "A central limit theorem and strong mixing condition," " Proceedings of the National Academy of Sciences of the United States of America, vol. 42, no. 1, p. 43, 1956.
12

---

[40] Y. Tian, X. Chen, and S. Ganguli, "Understanding self self-super learning dynamics without contrastive pairs," in International Conference on Machine Learning, pp. 10268-10276 PMLR, 2021.
[41] S. Yun, D. Han, S. Oh, S. Chun, J. Choe, and Y. Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," " in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6023-60200 6032, 2019.
[42] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," arXiv preprini 1710:09710 2017.
[43] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, "Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and robust- ness.." in International Conference on Learning Representations, 2019.
[44] R. Balestriero, L. Bottou, and Y. LeCun, "The effects of regularization and data augmentation are class dependent," arXiv preprint arXiv:2204.0366 03632, 2022.
[45] Y. LeCun, "The mnist database of handwritten digits," http://yann. lecun. com/exdb/nmisst/ 1998.
[46] M. Newman and G. Barkema, Monte Carlo methods statistical physics. Clarendon Press, 1999.
[47] J. L. Doob, "The limiting distributions of certain statistics," " The Annals of Mathematical Statistics, vol. 6, no. 3, pp. 160-169, 1935.
[48] G. W. Ochlert, "A note on the delta method," The American Statistician, vol. 46, no. 1, pp. 27-29, 1992.
[49] K. He, X. Zhang, S. Ren, and Sun, "Identity mappings in deep residual networks," in European conference computer vision, pp. 630-645, Springer, 2016.
[50] Simard, B. Victorii, Y. LeCun, and J. S. Denker, "Tangent prop-a formalism for specifying selected invariances in an adaptive network," in NIPS, vol. 91, pp. 895-903, Citeseer, 1991.
[51] O. Chapelle, J. Weston, L. Bottou, and V. Vapnik. "Vicinal risk minimization," Advances in neural information processing systems, pp. 416 422, 2001.
[52] Rifai, Y. N. Dauphin, Vincent, Y. Bengio, and X. Muller, "The manifold tangent classifier," Advances in neural information processing systems, vol. 24, pp. 2294-2302, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contri- butions and scope? [Yes] We have precisely given the scope of our results in the end of the introduction. Additionally, recall the limitations and assumptions of our results within each section carefully.
(b) Did you describe the limitations of your work? [Yes] The main limitations comes from the approx- imation done for nonlinear models which we have repeatedly highlight in both the introductions and the corresponding sections. We also highlight those in the conclusion along with future work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not see any potential negative societal impact for this work
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] we have carefully given the assumptions each result relies on. Especially, at the end of the introduction, we have highlighted reference paper where was shown that our assumptions are common in the literature, and that they are good enough to gain insights into nonlinear models.
13

---

(b) Did you include complete proofs of all theoretical results? [Yes] Each formal statement and derivation linked a proof in the appendix. We also provide details derivations in the appendix of all the equations we have obtained in the main paper, even if they are not part of formal result.
3. If you ran experiments...
(a) Did you include the code. data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] We provide samples of the code to reproduce figures. The entire codebase will be released publicly upon completion of the review-process
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] We only explore MNIST and provided the training details in the caption of the corresponding figure
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [No] We did not provide error bars but multiple runs with various settings
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] The employed experiment is small scale and can be performed without GPU
4. If you using existing assets (c.g., code, data, models) curating/releasngg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] MNIST only
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "s0nNEaLn8ii", "reviewer": "Reviewer_97Wc", "summary": "The paper aims to provide a theoretical framework for analyzing the role of data augmentations (DA) for learning tasks involving both linear and linear models. To this end, it first proposes a framework by which image transformations are modeled as matrix-vector multiplications using data-space transforms. \n\nThis framework is subsequently used to derive the analytical form of the first two moments of an augmented sample. Next, the MSE loss under a linear model is considered in which an exact form of an explicit DA regularizer is derived. An exact expression of the optimal solution under infinite DA sampling is derived, which generalizes the standard Least-Squares solution which is known when no DA is applied. The expression depends exactly on the expectation and variance of transformed samples. For the non-linear case, it is mentioned that the Taylor approximation can generalize from the linear setting.  \n\nNext, the paper analyses DA's sampling efficiency and loss sensitivity. An empirical evaluation shows the DA's sampling strategy is inefficient and requires many samples to converge to a good solution. The sensitivity of a given loss and model under a DA is analyzed leading to a derivation of a popular regularization strategy of TangentProp. ", "questions": "1. How do the derivation provided in Sec.3.4 and 4.2 change assuming a non-linear model? In particular for which range of transformations is the approximation sufficiently close to the linear case?\n2. How does the analysis provided in Sec.4.1 generalizes for different transformations and images?", "limitations": "The authors mention the limitation regarding the third-order approximation of non-linear models. However, this is not probably characterized. In which cases (range of transformations, images) is it ok to make such assumptions?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a novel set of mathematical tools for analyzing DA in a theoretical setting, which has thus been mainly analyzed through an experimental lens. These tools could be important in analyzing proposed DA schemes and so are an important and original contribution. \n2. The authors use these tools to derive an explicit regularizer induced by each DA as well as derive the sensitivity, of a given loss and model under a DA policy. These are useful properties for understanding the role of DA as a regularization. \n3. The analysis of the variance (sensitivity) leads to a first-principles derivation of a TangentProp as a regularizer which minimizes the loss variance. This is significant as it demonstrates the applicability of the framework in deriving and understanding useful regularizers. \n\nThe paper is mostly clear, however many of the proofs are left for supplementary, and at least an intuition could be useful to include in the main text.  \n\nWeaknesses:\n1. The analysis provided in Sec.4.1 is given for a single image and for a particular transformation. It is unclear if this holds for a more general case and in particular for a wide range of natural images and transformations. For a more general setting, can an expression in terms of the number of images n be given in order to closely approximate estimate the MSE loss from the augmented samples?\n2. The analysis provided in the paper in Sec.3.4 and Sec.4.2 is exact for linear models. It is unclear how the derivation is affected under non-linear models even for small transformations. Further, in practice, one often performs large transformations (e.g. in SSL models) which are crucial to their success. Hence it is unclear if the framework provided in this paper can explain the non-linear or large transformation setting well, limiting the significance of the work. \n\n****\nTypo, line 176, Gaussian blue\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "imIGdvu7j65", "reviewer": "Reviewer_nBxs", "summary": "The paper provides a theoretical analysis of data augmentation (DA) methods.\n- It introduces a novel integral transform formulation (called Data-Space Transform (DST)) for describing a class of DA, e.g. translation or rotation.\n- It provides an explicite formula for expectation and variance of transformed images\n- It provides an explicite formula for regularisation that has the same effect on the loss as training with infinite DA samples\n- It provides a theoretical connection with TangentProp\n", "questions": "\nDoes the method help reduce the number of steps to train a model in a non-linear case?\n\nDoes it help generalisation in the regime with small data-set in the non-linear case? Fig 7 did not show results with explicit regularisation.\n\nPlease explain the experiments on fig 5 and 7: what is the exact loss, training set, number of epochs and relevant hyperparameters for each scenario.", "limitations": "\nThe authors addressed some limitations (see weaknesses) except for the efficiency of the method.\n\nPlease discuss how the explicate regularisation can be computed efficiently, i.e the spectral decomposition. Without this the method cannot work on large images.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides some insights of the effect of DA:\n- How it has the same effect as an explicite regulariser\n- It can explicitly recover the expectation and variance of the transformed samples\n- It provides interesting connections between the variance of the MC estimate and the model’s Jacobian matrix, which leads to the TangentProp regulariser (that minimises the variance).\n\nThe theoretical analysis looks original to me.\n\nThe paper discusses a very important topic, as understanding the effects of DA and regularisation could help design systems that generalise better.\n\nWeaknesses:\n\nThe presentation could be clearer:\n- fig 1 does not help understanding. One has to zoom a lot to see the details. I had to actually read the text to understand the image. Perhaps a much lover resolution image (6x6 max) depicting something very simple like a circle would be better. Then the reader could understand the transformation matrix. Also there should be an indication in the figure how the image is flattened.\n- figures are referenced out of place, eg. fig 1 in page 5 and some in captions in other figures.\n- It is not clear how the linear model for MNIST in fig 5 was trained. I am assuming. The training set is created with N DA samples. For all N, the number of epoch are the same. For the black line the explicite regularisation is used.\n\nThe proposed method has many limitations:\n- The class of DA is limited to the cases where they can be expressed as a linear function on images. Many relevant DA falls outside of this (e.g. jpeg compression).\n- It is hard to use in practice, as deriving the expectation and variance of a bit more complicated DA (e.g. some non-linear warp) is a very had manual task.\n- For non-linear models the method relies on 3rd order Taylor expansion.\n- Even in the linear regression case, for a bit complicated DA the method requires the spectral decomposition of a very large system (a matrix of size n^2 x n^2). It is not discussed how this computation can be done efficiently (I assume it can be done for simple DA). If it cannot be done, then what are the trade-offs?\n\nIt is not clear to me how practical the explicite regulariser is. See Questions.\n\nOverall I rate the significance of the method not very high because of the limitations, that I think are severe.\n\nEDIT:\nThe rebuttal changed my mind, I think the paper is worth reading despite the limitations of the method in its current form. The significance is in the theory and that is valuable. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WGzFbhLq84", "reviewer": "Reviewer_yaYp", "summary": "The authors study both theoritically and empirically the sample efficiency of training with data augmented samples on a set of augmentations such as translation, shearing, rotation and zoom. They derive mathematical tools for analytically studying data augmentations and its effects on loss. As a result they derive an explicit regularizer that produces a model that is the equivalent of using DA on non-trivial transformations (the first of its kind). \n\nThe authors also study the number of DA samples needed for a model to properly absorb the information given by a data augmentation strategy on a particular image, as well as the number of DA samples needed in a setting where we have thousands of unique images for a model to fully absorb such information. They find that for a single image that number is close to 10 ^ 4, and for datasets with thousands of images, it can be around 50 data augmented versions of each image. \n\nThe authors also study the variance of a given loss and architecture applied on a particular data augmentation strategy and rediscover a popular reqularizer called TangentProp.", "questions": "I recommend the following:\n- Repeat your experiments on more datasets and models to see whether any of the conclusions related to them hold.\n- Better motivate/communicate what your findings mean for future research and/or current machine learning practices. \n- Rewrite parts of your paper with an aim towards more clarity and more conciseness. ", "limitations": "Technical limitations were discussed above. I don't see any potential negative societal impacts. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThe investigation is both interesting and insightful, and follows a novel direction in how it conducts an analytical study on data augmentations and attempts its best to come up with a theoretical framework for studying the effects of data augmentation on models and losses.\n\nQuality:\n\nThe technical rigour of the work is of high quality, but the writing can vary in quality, sometimes making it harder to follow what the authors are trying to communicate. \n\nExperiments are strictly done on MNIST, which is a majour concern as to whether any of the empirically supported conclusions hold.\n\nClarity:\n\nThe writing quality of the paper is mediocre, but the authors have definitely excelled in their mathematical communication in the paper. Furthermore, the figures presented would benefit from additional textual descriptions of what they portray. \n\nSignificance:\n\nThe conclusions regarding the sample efficiency of data augmentation strategies can be useful for people in the field if applied on more datasets and models. Furthermore, I am not sure how TangentProp can help improve machine learning models in any additional ways as a result of this study. \n\nThe significance of the study is quite uncertain. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3idf9RJ68vp", "reviewer": "Reviewer_JxCU", "summary": "The authors propose a novel operator called Data-Space Transform (DST) to express data augmentation (DA) and use it to investigate three topics: 1) equivalent regularizer induced by each DA; 2) the number of DA samples for a Monte-Carlo (MC) approach to correctly estimate the information of the DA; 3) sensitivity analysis of loss/model pairs under a DA policy. The third topic led to a rediscovery of TangentProp from first principles.", "questions": "Section 4 (as well as the title)  suggests that the simple translation augmentation needs around 1000 to 10000 samples to correctly estimate the MSE loss. In the 2012 Alexnet paper, by counting the number of unique (fixed-sized) crops of sampling a 224x224 patch from a 256x256 image, a 1024x gain in the number of training samples is estimated. I understand that this work investigates the problem in a different framework, but can the authors comment whether these two estimates are related in some way?", "limitations": "The authors discuss various limitations of this work in several sections which include important mathematical assumptions and approximations. Overall I believe the limitations are well discussed. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper is generally well written and motivated. \n\n2. The new transform operator proposed is intuitive.\n\n3. One of the key discoveries enabled by DST is a closed-form expected loss for DAs (for linear models and MSE loss), which allows for measuring the number of samples needed to correctly estimate the information conveyed by each DA.\n\n4. The rediscovery of TangentProp from principled derivation provides a nice link to practical use cases beyond the linear model and MSE loss case.\n\n5. Limitations of the work including various assumptions and approximations are discussed in the paper.\n\nWeaknesses:\n\nMajor:\n\n1. As shown in figure 5, it seems that even for the linear model and MSE loss combination where an analytical solution is available, the classic MC sampling DA method can outperform the expected loss method with large enough training sets (e.g. N=3000 for MNIST), which suggests that there are aspects not captured by the closed-form loss. Although it’s hard to evaluate whether some of the effects as error bars are not included in this figure, the trend shows that increasing the size of the training set could lead to larger performance lead for the multi-sample DA method over the expected loss approach. Can the authors 1) provide error bars, 2) comment on why MC DA may outperform the expected loss method for the linear model, and 3) conduct experiments with larger training sets?\n\nMinor:\n\n1. Results for non-linear models seem to be limited to small augmentations. Many state-of-the-art augmentations are rather aggressive, how well does the results generalize to these aggressive augmentations?\n\n2. Figure 3 caption, missing space before “Fourth column”\n\n3. Line 243: “than” => “as”\n\n4. Line 263: remove “as”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept