Automatic Data Augmentation via Invariance-Constrained Learning
Authors: Ignacio Hounie
Luiz F. O. Chamon
Alejandro Ribeiro
Venue: Submitted to ICLR 2023 (2023)
Keywords: Automatic data augmentation
Invariance
Constrained Learning
Image classification

Abstract: Underlying data structures, such as symmetries or invariances to transformations, are often exploited to improve the solution of learning tasks. However, embedding these properties in models or learning algorithms can be challenging and computationally intensive. Data augmentation, on the other hand, induces these symmetries during training by applying multiple transformations to the input data. Despite its ubiquity, its effectiveness depends on the choices of which transformations to apply, when to do so, and how often. In fact, there is both empirical and theoretical evidence that the indiscriminate use of data augmentation can introduce biases that outweigh its benefits. This work tackles these issues by automatically adapting the data augmentation  while solving the learning task. To do so, it formulates data augmentation as an invariance-constrained learning problem and leverages Monte Carlo Markov Chain (MCMC) sampling to solve it. The result is a practical algorithm that not only does away with a priori searches for augmentation distributions, but also dynamically controls if and when data augmentation is applied. Our experiments illustrate the performance of this method, which achieves state-of-the-art results in automatic data augmentation benchmarks for CIFAR datasets. Furthermore, this approach can be used to gather insights on the actual symmetries underlying a learning task.

Under review as a conference paper at ICLR 2022
AUTOMATIC DATA AUGMENTATION VIA INVARIANCE- CONSTRAINED LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Underlying data structures, such as symmetries or invariances to transformations, are often exploited to improve the solution of learning tasks. However, embedding these properties in models or learning algorithms can be challenging and computa- tionally intensive, Data augmentation, on the other hand, induces these symmetries during training by applying multiple transformations to the input data. Despite its ubiquity, its effectiveness depends on the choices of which transformations to apply, when to do so, and how often. In fact, there is both empirical and theoretical evidence that the indiscriminate use of data augmentation can introduce biases that outweigh its benefits. This work tackles these issues by automatically adapting the data augmentation while solving the learning task. To do so, it formulates data augmentation
invariance-constriine learning problem and leverages Monte Carlo Markov Chain (MCMC) sampling to solve it. The result is practical algorithm that not only does away with a priori searches for augmentation distri- butions, but also dynamically controls if and when data augmentation is applied. Our experiments illustrate the performance this method, which achieves state- oh-the-art art results in automatic data augmentation benchmarks for CIFAR datasets. Furthermore, this approach can be used to gather insights on the actual symmetries underlying learning task.
INTRODUCTION
Exploiting the underlying structure of data has always been a key principle in data analysis. Its use has been fundamental to the success of machine learning solutions, from the translational equivariance of convolutional neural networks (Fukushima and Miyake| 1982) to the invariant attention mechanism in Alphafold (Jumper et al. 2021). However, embedding invariances and symmetries in model architectures is hard in general and when possible, often incurs high computational cost. This is the case, of rotation invariant neural network architectures that rely on group convolutions, which are feasible only for small, discrete transformation spaces require coarse undersampling due to their high computational complexity (Cohen and Welling 2016 Finzi et al. 2020).
A widely used alternative consists of modifying the data rather than the model. That is, to augment the dataset by applying transformations to samples in order to induce the desired symmetries or invariances during training. Data augmentation, as it commonly known, is used train virtually all state-of-the-ar models in a variety of domains (Shorten and Khoshgoftaar 2019). This empirical success is supported by theoretical results showing that, when the underlying data distribution is invariant to the applied transformations, data augmentation provides a better estimation of the statistical risk (Chen et al., 2019; (Sannai al.) 2019; Lyle al. 2020 Shao al., [0222p On the other hand, applying the wrong transformations can introduce biases that may outweigh these
benefits (Chen et al. 2019 Shao et al. 2022).
Choosing which transformations to apply, when to do so, and how often, is thus paramount to achieving good results. However, it requires knowledge about the underlying distribution of the data that is typically unavailable. Several approaches to learning over a fixed set of transformations exist, such as reinforcement learning (Cubuk et al. 2018), genetic algorithms (Ho al. 2019), density matching (Lim et al.] 2019 Cubuk et al. [2200]] Hataya et al. 2020), gradient matching (Zheng et al. 2022), bi-level optimization (Li et 20200) Liu et al. 2021), jointly optimizing transformations using regularised objectives (Benton al. 2020).
augmentation policy distribution

---

Under review as a conference paper at ICLR 2022
variational bayesian inference (Chatzipantazis al. 2021 bayesian model selection (Immer et al. 2022) and alignment regularization (Wang et al. (022). Optimization based methods often require computing gradients with respect to transformations (Chatzipantazis et al. 202 Li et al. 2020b). Moreover, several methods resort to computationally intensive search phases, the optimization of auxiliary models, additional data, while failing to outperform fixed user-defined augmentation
distributions (iüleer and Hutter 2021).
In this work, we formulate data augmentation as an invariance-constrainee learning problem. That is, we specify set of transformations and a desired level of invariance, and recover an augmen- tation distribution that enables imposing this requirement on the learned model, without explicitly parametrising the distribution over transformations. In addition, the constrained learning formulation mitigates the potential biases introduced by data augmentation without doing away with its potential benefits. More specifically, we rely on approximate notion of invariance that is weighted by the probability of each data point. Hence, we require the output of our model to be stable only on the support of the underlying data distribution, and more quirement as a constraint on the learning task and leveraging recent duality results, the amount of data augmentation can be automatically adjusted during training. We propose algorithm that combines stochastic primal-dure methods and MCMC sampling to do away with the need for transformations to be differentiable. Our experiments show that it leads state-of-the-ar results in automatic data augmentation benchmarks in CIFAR datasets.
common samples. By imposing this re-
2 DATA AUGMENTATION IN SUPERVISED LEARNING
As the standard supervised learning setting, let x E S. Red denote a feature vector and y D C R its associated label or measurement. For classification tasks, take y C N. Let D denote a probability distribution over the data pairs (x, y) and &: y x y be non-negative, convex loss function, e.g., the cross entropy loss. Our goal is to learn predictor fo X' + in some hypothesis class Ho = (so 0 E E S R²} that minimizes the expected loss, namely
minimize R(fo) = Exx,)~ (((fo(x) AEE
(SRM)
We consider the distribution D to be unknown, except for the dataset {(x, yi), i = 1,...,n) n) of n i.i.d. samples from D. Therefore, we rely on the empirical approximation of the objective of (SRM), explicitly
R(()) = N 1=1
((o(x:)
One of the aims of data augmentation is to improve the approximation it of the statistical risk R when dealing with dataset that not sufficiently representative of the data distribution. To do so, we consider transformations of the feature vector X' - X, taken from the (possibly infinite) transformation set G. Common examples include rotations and translations in images. Data augmentation leverages these transformations to generate new data pairs (gx, y) by sampling transformations according to a probability distribution & over G, leading to the learning problem
1 N
minimize o20
Rang (fe) = Eg e [((fo(ox)...)]
(2)
Note that the empirical risk approximation R in I can be interpreted data distribution D by a discrete distribution that places atoms each data point. In that sense, Ruwy in (2) can be thought of as the Vicinal Risk Minimization (Chapelle et al. 2000) counterpart of ([). in which the atoms on x; are replaced by local distribution over the transformed samples gx;, i.e.,
approximation of the
Rong (fo)
1 N Σ
E
((folox.). vs) dP(gx))
where the distribution P over X is induced by the distribution G over G. As it can be seen from if @ is not chosen adequately, Rang can be a poor estimate of R, introducing biases that outweigh
2

---

Under review as a conference paper at ICLR 2022
the benefits of data augmentation (Chen al. 2019; Shao et al. 2022). On the other hand, if the data distribution D is statistically invariant under the action of G. invariant solutions have provable
advantages in terms of sample complexity (Chen et al. 2019: Bietti and Mairal 2019 Sannai et al.
2019
Lyle et al. 2020).
In this work, given set of transformations 9, we tackle the choice of 6, Le.. how sample transformations so that the solution of the learning problem is sufficiently invariant with respect to the learning task, as defined the next section. Note that invariance to transformations in 9 may hold for the true distribution or may also be desirable property of the solution, for example, to achieve robustness (Kanbak al. 2017 Volpi et al., 2018: Joshi et al. 2019). Unlike invariance
learning (Jebara [003] Zhou et al., 2021a Benton et al. 2020 Immer et al. 2022
we do not seek to
learn the transformations from the data.
3 DATA AUGMENTATION VIA INVARIANCE CONSTRAINTS
Finding a transformation distribution @ that leads to the desired properties in the solution can be challenging. What is more, using a fixed @ as in ( prevents us from controlling when and how much augmentation is used during training, running the risk of biasing the final solution. On the other hand, it is straightforward to specify a set of transformations G to which the solution should be approximately invariant (e.g., image rotations and translations).
In the next sections, we explain how data augmentation distribution can be obtained from such an invariance requirement. We first show how invariance can be interpreted as an augmentation distribution (Section 3.T). We then incorporate this invariance in a constrained learning problem (Sec-
tion 3.2
3.1
FROM INVARIANCE TO AN AUGMENTATION DISTRIBUTION
The goal of this section is to replace the task of choosing an augmentation distribution by the task of choosing set of transformations we wish to be (approximately) invariant to. To do so, we we will show how invariance can be used to implicitly determine the augmentation distribution.
However, rather than requiring the output of the model to be invariant, i.e., fo(x) = fg(xx for all g E G. we will consider invariance terms of quality of its predictions as evaluated by the loss function. Namely,
{(fo (x).)) E(fo (gx), v), Vg g.
This notion of invariance explicitly incorporates the structure of the learning task by using the loss identify which changes in the output of the model would lead to significant change in prediction performance. More precisely, we want to limit the difference in our model's performance on a sample and its transformed versions, i.e., we wish to have (e(fo (x),9) E(fo (gx), u))l small for all in g. Equivalently, we wish to control the magnitude of
max [e(J (x) v) E(So (gx), .)) 9Eg
However, rather controlling (4) for all X, which may be overly conservative, we want restrict our attention to the support of the data distribution. Furthermore, we wish to weight different inputs depending on their probability, in order to reduce the importance of unlikely or pathological cases. We therefore average (4p over the data distribution to obtain
Rinv (fo) = (xx))
max [e(fo (x) E(fe (gx). ا)1l 9E9
(5)
To connect the invariant risk Rinv (5) to the data augmentation formulation in (2) notice that it can be bounded, using triangle inequality and the monotonicity of the expectation, by
Rine (fo) (xxx))= [f(o (x), v)1 + (xxx)> max {(fo (gx) vV)
(6)
geg
3

---

Under review as a conference paper at ICLR 2022
The first term is simply the statistical risk R that is usually minimized in order to tackle the learning task. The second term resembles the objective typically found in adversarial learning problems, e.g., (Madry et al. 2017). This term, it turns out, can be interpreted as augmentation distribution.
As shown by Robey et al. 2a, the maximisation of the loss over transformations can be written the semi- remi-infinite constrained optimization problem
max E (fo (gx), v) = sup gEg
(g))((f((ox))))
xec 9
s. to
Xgg)dg =1
Notice that the solution of this optimization problem (g) is non-negative, normalized function and can therefore be interpreted as a distribution over transformations that depends on the sample point (x,y) well as the model fo. This allows re-interpret the maximization over g as expectation, i.e.,
E (xxy)~> max {(So (gx) (1)) = (x,))/ [Ba=x (esssee((ooaa ([(o
(8)
geg
Observe that the right-hand side of 8 resembles the statistical form of the data augmentation objective in 2
3.2 A CONSTRAINED LEARNING APPROACH
Returning to the invariant risk bound in (6). notice that it is composed of two parts, namely, the statistical risk from (SRM) and what we have shown (8) to be a mtancc/modelldeeenddee data augmentation. However, order to address the potential biases introduced by transformations, rather than modifying the objective as in standard data augmentation, we propose to combine these two terms in constrained learning problem. Explicitly.
p min OEe
((x,)) [(([o(x),)) v)]
(CSRM)
S. to E(xx)~= Egax [((fo(gx), y)ll c
Notice that this formulation tackles the two terms forming the invariant risk bound in (6). but instead of combining them directly, it incorporates the data augmentation term as a constraint in the typical statistical risk minimization problem (SRM). This formulation has the advantage that if solution to the unconstrained problem is feasible, I.c., satisfies the invariance constraint in (CSRM). the presence of that constraint has no effect on the statistical problem. Yet, it can be beneficial when approximating the solution of (CSRM) empirically. We will explore this fact in the next section, where we tackle the practical challenges involved in solving (CSRM)
For conciseness in CSRM) we have included only one invariance constraint associated with single set of transformations G. However, our approach can extended to an arbitrary number of constraints defined by transformation sets Gi, i= l.... (each inducing augmentation distribution AT), and constraint levels eq. All of the following derivations still hold, regardless of the number of constraints.
4 ALGGORITHM DEVELOPMENT
Solving (CSRM) presents two challenges. First, it is a constrained statistical learning problem, which involves the unknown data distribution D. We address this by resorting to an empirical dual problem as explained on Section 4.1 Second, it can be hard to sample from A*. We address this by introducing a smooth approximation that leverages MCMC methods on Section 4.2)
4.1 EMPIRICAL DUAL CONSTRAINED LEARNING
To tackle the invariance- invariance-constrained statistical risk minimization problem we leverage recent duality results in constrained learning theory (Chamon al. 2021), that approximate the problem by its
4

---

Under review as a conference paper at ICLR 2022
empirical dual
- [ Eg (((Fo(x)) (D-CERM)
D max min
cmp
Σ ((fo(x)), ve) +7 220 OC
The advantage of D-CERM is that it is an unconstrained problem that, provided we have enough samples and the parametrization is rich enough, can approximate the constrained statistical problem ( (CSRM). Namely, the difference between the optimal value of the empirical dual Demp and the statistical primal P*, i.e., the empirical duality gap is bounded (Chamon et al., 2021).
As in regular data augmentation, we will also approximate the expectation over X by sampling transformations as discussed in Section 4.2 Then, the problem [DCERM becomes an unconstrained deterministic problem, which can be solved using the algorithm described in Section 4.3
Note that finding a Lagrangian minimizer for a fixed value of the dual variable (?) is equivalent to minimising the risk under fixed mixture augmentation distributior{ where the value of controls the probability sampling the identity. We can also interpret this as optimising penalised or regularised learning objective. However, solving the constrained problem, namely maximising over y, has fundamental differences.
First, constraints explicit the requirement they represent. While the degree of invariance imposed should depend only on the statistical problem at hand, the value of y needed to achieve it will depend on the sample size, the parametrization and the learning algorithm. In contrast, constrained learning dynamically adjusts the amount of augmentation - dictated by 7 to a particular learning setup.
Second, the optimal dual variable can give information about the trade-off minimising the loss over training samples and satisfying the invariance constraint. In penalised approaches, on the contrary, this trade-off is fixed.
Lastly, the aforementioned informativeness and interpretability can facilitate hyper-parameter tuning. The insights gathered from optimal dual variables can be leveraged a posteriori, for instance, to manually choose appropriate transformations, relax the invariance constraint levels, or change the learning setup (e.g. increase the capacity of the model class).
4.2 SAMPLING TOWARDS INVARIANCE
Sampling the augmentation distribution X" can be difficult when G is not finite and fo is deep neural network. Even when the transformation space is low dimensional, as the case of translations and rotations, the highly non-convex loss landscape of these models makes the maximization over G challenging (Engstrom et al. 20177.. If the optimal distribution x is not smooth, it is challenging to sample from it with sufficient accuracy (Homem de Mello and Bayraksan 2014]. Consequently,
obtaining an unbiased estimator of Eg-xx (([fe(gg) y)] may not be possible. Therefore, we add an Ta norm penalisation, which promotes smoothness, to leverage MCMC methods.
We then define the c-smoothed distribution X as a solution to the regularised problem
/
X argmax (g))((f((ox)) v)dg + c Xgg)2gg
/ A(g)dg
The regularization term introduces an optimality gap with respect to worst perturbations, ie., Ey-x; [((fo(gx), y)] & maxgeg (fo (gx). y). However, for particular values of the regular- ized problem has closed form solution (Robey al. 2021a) that allows us sample from it easily. Namely, there exists a constant e 2 such that X [x, v. {(fa(gx).v)
Since X is a smooth probability distribution, we do not need to estimate the multiplicative factor to sample from by leveraging Monte Carlo Markov Chain methods (MCMC).
MCMC methods (Hastings) 1970) are based on constructing a Markov chain that has the target distribution as an equilibrium distribution. Independent Metropolis Hastings uses a state independent
Explicitly, with probability , the identity is sampled, and with prohability _ 9 X".
5

---

Under review as a conference paper at ICLR 2022
usually fixed proposal for each step. In case, it only requires applying transformation and computing a forward pass of the neural network to evaluate the loss. This enables the use of non-differentiable transformations, and has the advantage that the density at consecutive proposals can be evaluated in parallel allowing speedups in the sampling step. Although MH methods thus allow to sample the proposal distribution with low computational cost, they exhibit random walk behaviour, which leads to slow convergence in high dimensional settings (Dellaportas and Roberts 2003 Holden et al., 2009).
We can then approximate Eg-A; [4[oo (gx:), 36)l by sampling transformations according to the loss. Namely, we can obtain set of m samples drawn from X and approximate the expectation over the group by the sample mean
m
Rggus (e(So(xx:),
RM m
Σ
(oo(gxx))
Lid. ((fe(gx:), w/)/ are m transformations sampled from the smoothed distribu- where g1...., 9m
tion (Se, 8i)
In the section, the implementation of independent-Min with uniform proposal is described in Algorithm 2
together with the primal-dual augmentation algorithm.
4.3
PRIMAL-DUAL ALGORITHM
Since the cost of the inner minimization, i.e. training the model, can be high, we adopt an alternating update scheme (K. J. Arrow and Uzawa. 1958) for the primal and dual variables, in (Chamon ctt al.
202I
Fioretto et al. 2020).
A bounded empirical duality gap does not guarantee that the primal variables obtained after running the alternating primal-dual algorithm| and solving the saddle point problem approximately are near optimal approximately feasible. Although stronger primal recovery guarantees can be obtained by randomizing the learning algorithm (Chamon et al. [202]), it requires storing model parameters 0 at each iteration, and there is empirical evidence (Chamon al. 2021 Robey et al. 2021a Elenter
et al. 2022 Shen et al.) 2022 Cervino et al. 2022 Zhagg et al. 2022) that good solutions can still be obtained without randomization.
Algorithm describes transformation sampling. By keeping only one sample (m 1) we recover the usual augmentation setting, that yields one augmentation per sample in the training batch. In our experiments we address this setting, since it has lower computational costs. However, simply keeping more samples from the chain (m > 1) allows to extend the method to the batch augmentation setting (Hoffer et al. 2020), which creates several augmented samples from the same instance in each batch.
Although several steps of the chain may be required to deviate enough from the proposal distribution, we show in the experimental section that sampling the constraint approximately by using few sampling steps suffices.
Algorithm Puimal-Dual Augmentation
1: X 0. 0 = Pp-
2: for Batch in {(x4, 9/ 11 1 do
3: for (x,, 31) E Batch do
4: Iel....: : Jim rid (( ((fe(gx)).))))
P Sample transformations
5: Baach Σ E(x.,o
m L (fo
Evaluate constraint slack
6:
Σ E(x.,x)) E Ratch (fo((;)) 9:)
Evaluate Loss
Baich
7:
L l 78
Compute Lagrangian
8:
0 0 7p VoL
Primal update
9:
7 J 1as8
> Dual Update

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Independent MH sampler
1: g( ~ ~U(g)
Sample initial State
2: (10) = (ff (x))))),1 .9)
b Evaluate loss
3: for = 1..... Mateps do
4:
9prop ~ (g)
P Sample next proposal Evaluate Loss
5: Eprop {So (9propX) :)))
6:
p - min 11 Aprop
Acceptance Prob b. Accept/Reeect
7:
w.p. p:
8:
g = Sprops g(() L prop
9:
else:
10:
g = (11 (10) = g(t-1)
5 AUTOMATIC DATA AUGMENTATION EXPERIMENTS
This section showcases Algorithm| in common image classification benchmarks. We compare it to state-of-the of-t--------tttt data augmentation methods in terms classification accuracy. Furthermore, we discuss other advantageous properties of our method through ablations. Namely, we focus on the ability to control the effect of data augmentation by modifying the constraint level, the informa- tiveness of dual variables, and the benefits of adapting the augmentation distribution throughout training. We follow the setup (including the transformation sets) used in recent automatic data augmentation literature (Müller and Hutter 2021). A complete list of transformations together with other hyperparameters and training settings can be found on Appendix B Note that four out of the sixteen transformations used are non-di con-differentiable Whereas other works have introduced gradient approximations for transformation operations with discrete parameters et al. 20206 2020), our approach does not require such approximations.
Hataya et al.
Throughout these experiments, we fixed the number of steps of the MH sampler (Algorithm) 2 to two, which has the added advantage of reducing the computational cost of evaluating proposals. The constraint level was determined by grid search targeting cross-validation accuracy. As shown in Table D in both transformation sets considered, we find that our approach improves or closely matches existing approaches. The failure to achieve large improvements in accuracy over baselines, which has been attributed to stagnation data augmentation research (Müller and Hutter 2021). can also reflect the limits of the benchmarking setup.
Standard DeepAA
Wide DeepAA
TA
OURS
TA
OURS
CIFARIO
202
96.55 0.11
96.43 + 0.09
96.76 0.14
96.24 0.11
96.27 + 0.19
97.05 0.18
Wide-ResNet-28-10
97.46 + 0.10
97.57 + 0.15
97.74 + 0.10
97.51 + 0.20
97.27 + 0.10
97.85 0.17
CIFARI00
Wide-ResNec-40-2
79.92 0.13
79.45 1: 0.42
80.83
0.31
79.96
0.45
= 0.27
81.19
0.34
Wide-ResNet-28-10
83.40 + 0.16
83.77 0.29
83.53 + 0.16
84.11 = 0.24
83.09 + 0.30
84.89 0.12
SVHNcore
Wide-ResNett28-10
98.05 0.03
98.04
0.08
98.15
0.09
98.07
0.03
97.93
0.03
98.01 0.04
Table 1: Image Classification accuracy for WideResnet architectures (Zagoruyko and Komodakis 2016) trained using different augmentation policies, defined on standard (Cubuk et al. 2018) and wider (Müller and Hutter 2021) augmentation search spaces. We include state-of-the-an methods TA (uüller and Hutter 2021) and DeepAA (Zheng et al.) 2022), and 95% confidence intervals computed over five independent runs. In CIFAR datasets our approach leads to an improvement in accuracy, whereas in SVHN (Netzer et al. 2011) it leads to a degradation in performance in the wide setup.
Regardless, approach yields improvements in test accuracy over baseline model without augmentation for a wide range constraint levels (Figure TD. This illustrates the robustness of the solution to this hyperparameter. Observe also that the constraint is relaxed (by increasing e), the

---

Under review as a conference paper at ICLR 2022
training error decreases while the generalization gap, i... the difference between train and test errors, increases. In other words, by loosening the invariance requirement the model can fit better to training samples at the cost of worse generalization. Eventually, only the generalization gap increases while the training error stagnates (c > 2.1 for CIFAR 100 and e > 0.8 for CIFAR 10). This transition occurs at the same point at which the final value of the dual variable r from (D-CERM) essentially vanishes (darker color). This showcases the infromativeness of the dual variable. In the case of CIFARIO, even the training error begins increase at that point, suggesting that the invariance requirement need not be at odds with accuracy.
Dual variable
CIFARIO0
CIFARIO
at epoch 200
X
6045
6/40
0.28
Error
0.15
16"
6A23
10
0000
Train
Test
Constraint level (E)
Baseline
Figure Constraint level ablation for WideResnnet-4-22 in CIFAR datasets. We plot error rates computed over the train and test set and averaged over five runs, for different constraint levels. We include the test error the unconstrained baseline (without augmentation) for comparison. The color of markers denotes the final value of the dual variable.
Not only does Algorithm I tune the effect of data augmentation on the solution (by adapting y in step 9), but also modifies the distribution over transformations during training (step 4). To showcase the benefits of this over the of fixed distribution, Figure compares the results obtained using our approach (sampling according to X²) and one where step 4 is replaced by uniform sampling over transformations. For the same constraint levels, lower test errors are obtained by sampling transformations according to A i.c., promoting invariance. Note also that for e = 2.1 in CIFAR 100 and 0.8 for CIFARIO, the performance gap is quite large. Once again, this occurs at the point in which y vanishes (darker color) for the uniform distribution, i.e., no data augmentation occurs by the end of training. At this point, however, there is still value in promoting invariance by sampling from X as evidenced by the positive value of the dual variable (lighter color) in this approach.
Dual variable
CIFARI00
CIFAR10
epoch 200
0.25
0.342
Tost Error
X
6.30
X
X
6:38
0.21
0:33
4.20
X
Constraint Distribution
6:333
1a
Uniform
Constraint level (E)
A
Figure 2: We compare our approach to a constraint on the uniform distribution, for WideResnet-40-2 in CIFAR datasets, at different constraint levels. We plot error rates computed over test set and averaged over five runs. Markers denote the augmentation distribution. The color of markers denotes the final value of the dual variable.
8

---

Under review as a conference paper at ICLR 2022
Furthermore, to assess the impact of the sampling approximation the performance of the solution we conduct an ablation on the number of steps the MH sampler (Algorithm (2) keeping the constraint level (c) fixed. Using more steps of the chain allows samples to deviate further from the uniform distribution, which as shown Figure 3 is reflected on the lower entropy of sampled transformations (first row). As shown in the second row, this results in higher classification error rates on the training set (represented by blue circular markers). However, we find that increasing the number of steps does affect significantly error (denoted by orange crosses). In Appendix C.1.3 we show how the number of sampling steps affects the evolution of dual variables.
CIFAR1O0
CIFAR1O
Entropy
2.5
2.4
2.4
2
4
8
16
2
4
8
16
0.2
x
0.03
X
train
Error
0.1
0.02
test
0.01
2
16
4
8
Number of MH Steps
Figure 3: Number of Metropolis Hastings steps ablation for WideResnet-40-21 in CIFAR datasets. The constraint level fixed for each dataset (0.8 CIFARIO and 2.1 for CIFARI00). The first row shows the entropy of the augmentations sampled at the last epoch of training. The second rows classification emor rates. Each point represents an independent run, experiments are repreated four times.
CONCLUSION
In this paper, have proposed constrained learning approach for automatic data augmentation, which instead using augmented samples as a modified learning objective, imposes an invariance- constraint. We have shown that this yields augmentation distribution that adapts during training, and found that coarse sampling approximations based on MCMC methods can improve generalization in automatic data augmentation benchmarks. Furthermore, experiments showed that the dual variable can give insights about the resulting augmentation distribution. We also found that strictly feasible solutions were obtained for wide range of constraint levels, with notably different generalization gaps, and that in some cases tightening the constraint even led to lower training error. Thus, analysing the interplay between the learning problem and the optimization algorithm promising future work direction.
REFERENCES
Benton, G., Finzi, M., Izmailov, P., and Wilson, A. G. (2020). Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems, 18:17605-76166
Bietti, A. and Mairal, (2019). Group invariance. stability to deformations, and complexity of deep convolutional representations. The Journal of Machine Learning Research, 20(1):876 924.
Blaas, A.. Suau, X., Ramapuram, J., Apostoloff, N., and Zappella, L. (2021). Challenges of adversarial image augmentations. In (Still) Can't Believe It's Not Better! NewrIPS 2021 Workshop.

---

Under review as a conference paper at ICLR 2022
Caruana, R., Lawrence, S., and Giles, C. (2000). Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Leen, T., Dietterich, T., and Tresp. V., editors, Advances in Neural Information Processing Systems, volume 13. MIT Press.
Cataltepe, Z., Abu-Mostafa, Y. S., and Magdon-Ismail, M. (1999). No free lunch for early stopping. Neural Computation, 11 11(4):995-109%
Cervino, J., Ruiz, L.., and Ribeiro, A. (2022). Training stable graph neural networks through constrained learning. ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4223-4227.
Chamon, F. O., Paternain, S., Calvo-Fullana, M., and Ribeiro, (2021). Constrained learning with non-convex losses. CoRR, abs/2103.05134.
Chapelle, O., Weston, J., Bottou, I.., and Vapnik, (2000). Vicinal risk minimization. Advances in neural information processing systems, 13.
Chatzipantaziss, E., Pertigkiozoglou. S., Dobriban, E., and Daniliidis, K. (2021). Learning augmenta- tion distributions using transformed risk minimization. arXiv preprint 1xXiv2211.088990
Chen, S., Dobriban, E., and Lec, J. H. (2019). A group-theoretic framework for data augmentation.
Cheung. T.-H. and Young, D.-Y. (2022). Adaaug:
Learning class- and instance-adaptive data augmentation policies. In International Conference on Learning Representations.
Cohen, T. and Welling, M. (2016). Group equivariant convolutional networks. In Balcan, M. F. and Weinberger, K. Q., editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990-2999, New York, New York, USA. PMLR.
Cubuk, E. D., Zoph, B., Mané, D., Vasudevan, V., and Le, Q. (2018). Autoaugment: Learning augmentation policies from data. CoRR, Abs/1805.09501.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops.
Dellaportas, P. and Roberts, G. O. (2003). An introduction to meme. In Spatial statistics and computational methods, pages 1-41. Springer.
DeVries, T. and Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout, arXiv preprint /rXiv:1708.044: 170.00452..
Duvenaud, D., Maclaurin, D., and Adams, R. (2016). Early stopping nonparametric variational inference. In Gretton, A. and Robert, C. C., editors, Proceedings of the 19th International Conference Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 1070-1077, Cadiz, Spain. PMLR.
Elenter, J., Naderializadeh, N., and Ribeiro, (2022). A lagrangian duality approach active learning. ArXiv, abs/2202.04108.
Engstrom, L., Tran, B., Tsipras, D., Schmidt, L.,, and Madry, A. (2017). Exploring the landscape of spatial robustness.
Esteves, C., Allec-Blanchette, C., Makadia, A., and Daniilidis, K. (2017). 3d object elassification and retrieval with spherical enns. CoRR, abs/17 1.06621..
Finzi, M., Stanton, S., Izmailov. P., and Wilson, A. G. (2020). Generalizing convolutional neural networks for equivariance to lie groups arbitrary continuous data.
Fioretto, F., Hentenryck, P. V., Mak, T. W. Tran, C. D.. Baldo, F., and Lombardi, M. (2020). Lagrangian duality for constrained deep learning. In ECMI/PKDD.
10

---

Under review as a conference paper at ICLR 2022
Fukushima, K. and Miyake, S. (1982). Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation neural nets, pages 267-285. Springer.
Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1)997-109.
Hataya, R., Zdenek, J., Yoshizoe, K., and Nakayama, H. (2020). Faster autoaugment: Learning augmentation strategies using backpropagation. In European Conference on Computer Vision, pages 1-16. Springer.
Ho, D., Liang, E., Chen, X., Stoica, I., and Abbeel, (2019). Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pages 7731-2741, PMLR.
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., and Soudry, D. (2020). Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8129-8133 8138.
Holden, L.., Hauge, R., and Holden, M. (2009). Adaptive independent Metropolis Hastiiggs The Annals of Applied Probability, 19(1):395 413.
Homem-de Mello, T. and Bayraksan, G. (2014). Monte carlo sampling-based methods for stochastic optimization. Surveys in Operations Research and Management Science, 19(1)566-85.
Immer, A., van der Ouderaa, T. F. A., Fortuin, V., Rätsch, G., and van der Wilk, M. (2022). In- variance learning in deep neural networks with differentiable laplace approximations. CoRR, abs/2202.106334
Jebara, T. (2003). Convex invariance learning. In Bishop, C. M. and Frey, B. J., editors, Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, volume R4 of Pro- ceedings of Machine Learning Research, pages 149-156. PMLR. Reissued by PMLR on 01 April 2021.
Ji, Z., J. D., and Telgarsky, M. (2021). Etrly-stopped neural networks are consistent. In Beygelz- imer, A., Dauphin, Y., Liang, P., and Vaughan, W., editors, Advances Neural Information Processing Systems.
Joshi, A., Mukherjee, A., Sarkar, S., and Hegde, C. (2019). Semantic adversarial attacks: Parametric transformations that fool deep classifiers.
Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunlasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D. A., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. (2021). Highly accurate protein structure prediction with alphafold. Nature, 596:583 - 589.
K. J. Arrow, L. H. and Uzawa., H. (1958). Studies in linear and non-linear programming. Stanford University Press.
Kanbak, C., Moosavi-Dezfooli, S.-M., and Frossard, P. (2017). Geometric robustness of deep networks: analysis and improvement.
Kondor, R. and Trivedi, S. (2018). On the generalization of equivariance and convolution in neural networks to the action of compact groups.
LeCun, Y., Cortes, C., and Burges. C. (2010). Mnist handwritten digit database. ATT Labs [Online]. Available: wtw.://aam...w...oooo../
2.
Li, M.. Soltanolkotabi, M., and Oymak, S. (2020a). Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pages 4313 2213-4324 PMLR.
11

---

Under review as a conference paper at ICLR 2022
Li, Y., Hu, G., Wang, Y., Hospedales, T., Robertson, N. M., and Yang, Y. (2020b). Dada: differentiable automatic data augmentation. arXiv preprint arXiv:2003.03780.
Lim, S., Kim, I., Kim, T., Kim, C., and Kim, S. (2019). Fast autoaugment. Advances in Neural Information Processing Systems, 32.
LingChen, T. C., Khonsari, A., Lashkari, A., Nazari, M. R., Sambee, J. S., and Nascimento, M. A. (2020). Uniformaugment: A search-free probabilistic data augmentation approach. ArXiv, Jbs/2003.143343
Liu, A., Huang, Z., Huang, Z., and Wang, N. (2021). Direct differentiable augmentation search. CoRR, abs/2104.04282.
Lyle, C., van der Wilk, M., Kwiatkowska, M., Gal, Y., and Bloem-Reddy, B. (2020). On the benefits of invariance in neural networks.
Madry, A., Makelov, A., Schmidt, L.., Tsipras, D., and Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
Mallat, S. (2016). Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):0150220
Müller, S. G. and Hutter, F. (2021). Trivialaugment: Tuning-free yet state-of-the-art data augmentation. CoRR, abs/2103.10155
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NIPS).
Olah, C., Cammarata, N., Voss, C., Schubert, L., and Goh, G. (2020). Naturally occurring equivariance in neural networks. Distill. http://diillluuoo//00eeeeeieii/r
Petrini, I.,, Favero, A., Geiger, M., and Wyart, M. (2021). Relative stability toward diffeomorphisms indicates performance in deep nets. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems.
Püsscell, M. and Moura, J. M. F. (2006). Algebraic signal processing theory. CoRR, abs/cs/061 2077.
Rice, L., Wong, E., and Kolter, Z. (2020). Overfitting in adversarially robust deep learning. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8093-8104. PMLR.
Robey, A., Chamon, L. F. O., Pappas, G. J., Hassani, H., and Ribeiro, A. (2021a). Adversarial robustness with semi-infinite constrained learning. ArXiv, abs/2110.15707
Robey, A., Pappas, G. J., and Hassani, H. (2021b). Model-based domain generalization. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan. J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 20210-20229. Curran Associates, Inc.
Sannai, A., Imaizumi, M., and Kawano, M. (2019). Improved generalization bounds of group invariant / equivariant deep networks via quotient feature spaces.
Shao, H., Montasser, O., and Blum, A. (2022). A theory of pac learnability under transformation invariances.
Shen, Z., Cervino, J., Hassani, H., and Ribeiro, A. (2022). An agnostic approach to federated learning with class imbalance. In International Conference Learning Representations.
Shorten, C. and Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48.
Volpi, R., Namkoong, H., Sener, O., Duchi, J. C., Murino, V., and Savarese, S. (2018). Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31.
12

---

Under review as a conference paper at ICLR 2022
Wang, H., Huang, Z., Wu, X., and Xing, E. P. (2022). Toward learning robust and invariant representations with alignment regularization and data augmentation.
Weiler, M. and Cesa, G. (2019). General e(2)-cquivariant steerable cnns. CoRR, abs/1911.0822)
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: novel image dataset for benchmarking machine learning algorithms.
Xu, Y., Noy, A., Lin, M., Qian, Q., Hao, L.., and Jin, R. (2021). Wemix: How better utilize data augmentation.
Zagonuyko, S. and Komodakis, N. (2016). Wide residual networks. CoRR, Abs/1605.07140
Zhang, H., Zhang, Y--F, Liu, w., Weller, A., Schölkopf, B., and Xing, E. P. (2022). Towards principled disentanglement for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), pages 8024-80000 4004.
Zhang, X., Wang, Q. Zhang, J., and Zhong, Z. (2020). Adversarial autoaugment. In International Conference Learning Representations.
Zheng, Y., Zhang, Z., Yan, S., and Zhang, M. (2022). Deep autoaugment. In International Conference on Learning Representations.
Zhou, A., Knowles, T., and Finn, C. (2021a). Meta-learning symmetries by reparameterization. In International Conference on Learning Representations.
Zhou, F., Li, Xie, C., Chen, F., Hong, L., Sun, R., and Li, Z. (2021b). Metaaugment: Sample-aware data augmentation policy learning. In AAAI.
13

---

Under review as a conference paper at ICLR 2022
A
ADDITIONAL RELATED WORK
A.1 CONSTRAINED LEARNING AND DATA AUGMENTATION
Xu al. 22021) also formulate data augmentation constrained learning problem. They impose a constraint on the excess risk, i.e. the difference between the statistical risk and its optimal value, on augmented data. Thus the constraint level the augmented risk also determined by the data distribution, augmentations considered, and model class, and the existence of a strictly feasible point guaranteed.
min R(fo) AEe
Rang (fo) min Rang (fa) c, 0, og
where R and Rang the statistical risk under the original and augmented distribution, exlipicity
R(fo) = Exx.))> v)l. Raug (fo) = ((x=,)),,1((fo((x)),
g-
Unlike our formulation, this formulation assumes fixed distribution of augmentations O is given.
By formulating it as a constrained problem, they aim to avoid introducing a bias when the data distribution is not invariant augmentations. Two types of biases induced by augmentation explicitly addressed, covariate shift (i.e. label-preseriing augmentations) and concept shift (i.e. label mixing augmentations).
Interestingly, they show that under some conditions on the risk, augmented risk and constraint level, by utilizing the augmented data to constrain the solution small region SGD can achieve lower error (Xu et al. 2021 Proposition 1).
Instead of resorting constrained optimization algorithms, they propose two stage algorithm that consists of first finding an approximate minimizer of the augmented risk and then using that solution as an initialisation the (unconstrained) statistical risk minimization problem. The first stage obtains a feasible point, and then under some conditions the SGD iterates obtained when solving the second Theorem 1).
problem remain feasible (Xu al. 2021
A.2 ADVERSARIAL DATA AUGMENTATION
Zhang al. 2020) have shown that using adversarial transformations which already mentioned is related to promoting invariance can give competitive results with respect to other automatic augmentation methods in image classification. However, [Blaas et al. (202T) have since evidenced the importance of two factors: the implicit learning curricula and the suboptimality of the adversarial used by Zhang et al. (2020). which mitigates the biases introduced by worst-case transformations. Furthermore, Blaas et al. 2021) report that an explicit cyclic curricula in which augmentations are mild first, then get harder as training progresses, and finally revert to milder augmentations the end of training, performs better empirically, We note some interesting commonalities with our approach and experimental results. First, the dynamics of our primal-dual algorithm resemble the aforementioned heuristically defined curricula. Second, the suboptimality with respect to worst case perturbations can be related to the smoothed approximation used in our approach.
A.3 CONSTRAINED LEARNING AND DOMAIN GENERALIZATION
Domain Generalization (DG) involves training the model in different but related data distributions, and evaluating in an unseen domain. For example, common benchmark consists of domains created by rotating images in the MNIST dataset by different angles. Constrained formulations have been proposed in this context (Robey al. 2222bb] Zhang et al., 2022), that enforce invariance under learnt domain translation transformations. In contrast, approach addresses pre-defined transformations that are commonly used data augmentation pipelines in order to improve generalization in the same domain used train the model.
A.4 GROUP INVARIANCE
The relationship between convolutional structure and equivariance has been long known in algebraic signal processing theory (Püschel and Moura 2006). Recently, necessity results have re-gained
14

---

Under review as a conference paper at ICLR 2022
attention in the context of neural network architecture design (Kondor and Trivedi 2018). Several Group Convolutional neural network architectures that generalize CNNs to different groups by leveraging group convolutions have been proposed (Cohen and Welling 2016 Esteves al. 2017] Finzi et al. 2020). In the case images, has been shown that that SE2 equivariant layers can be implemented efficienlly using regular 2D convolutions (Weiler and Cesa 20T9). This approach allows to derive a parametrization for CNN filters under finite subgroups of SE2. Among other works (Bietti and Mairal 2019; Sannai al. 2019) give theoretical analyses of the benefits of group invariance in
learning settings.
As already mentioned, achieving invariance through architecture design is both challenging and limited in the sense that it relies on transformations having a specific structure (e.g: group). The goal of exact invariance over the whole input space more strict than the approximate invariance notion that our work addresses. As argued by Mallat (016)) CNNs can learn locally invariant features with respect to arbitrary transformation groups, which could explain their generalization properties. Furthermore, empirical studies evidence modern CNN architectures learn approximately equivariant features to transformations such as scaling and rotations (Olah et al. 2020). or diffeomor- phisms (Petrini et al., 2021), even when trained without direct augmentation.
However, commonly used augmentations do not form a group. Our approach does not require this structure.
15

---

Under review as a conference paper at ICLR 2022
B EXPERIMENTAL SETUP DETAILS
B. TRANSFORMATIONS
As in recent automatic augmentation literature (Ho et al. 2019 Lim et al.
2019 Hataya al.
2020, Zhang et al. 2020; Cubuk et al., 2020; LingChen et al. 2020; Zhang et al. 2020; Müller and Hutter 2021 | Zhou al. 202 l Zheng et al. 2022 Cheung and Yeung 2022) we focus on image classification datasets and employ transformation search space comprising 14 operations, introduced by CCuuk et al. (2018). For those that have parameters, their magnitudes are discretized in thirty levels, which does not compromise performance and greatly reduces the search space (Cubuk ct al., 2020). Most approaches compose transformations, i.c. applying more than one transformation the image. However, recently Müller and Hutter (2021) have shown that applying only one transformation at time, defined over a wider magnitude space (noted Wide in Table ( can outperform other approaches. We thus use the same transformation space (Müller and Hutter 2021
Table from Müller and Hutter 202T) lists the operations and their magnitude ranges. In our experiments we used the wide search space, the standard ranges from Cubuk et al. (2020) are included for comparison. We extend the codebase provided by Müller and Hutter (202I), which uses the Pillow implementation of all transformations except for cutout, and refer to its documentation for further details about image operations.
Operation
Magnitude
Standard
Wide
Identity
ShearX
[-0.3, 0.3]
0.999, 0.99]
ShearY
0-3, 0.3]
[-0.99. 0.99)
Translate
[-10, 10]
[-32, 32]
Translate
- 0.45, 0.45]
[- 32, 32]
Rotate
[- 30, 30]
| --3,, 135)
AutoContrast
Invert
Equalize
Solarize
[0, 256]
Posterize
[4,8]
[2
Contrast
[0.1, 1.9
[0.01.2]
Color
1.9]
[0.01,2]
Brightness
[.91
[0.01,2]
Sharpness
1.91
[0.01.2]
Flips
Cutout
16(60)
16(60)
Crop
Table 2: Pillow image operations in the data augmentation search space and the range of magnitudes corresponding to the standard (Cubuk et al. (2020) and wide (Müller and Hutter 2021 search spaces. Some operations do not use magnitude parameters.
B.2 TRAINING SETUP
In order to enable comparisons and reproducibility we use the same training pipeline as in previous works Müller and Hutter (202T).. We apply the vertical flip and the pad-and-crop augmentations
16

---

Under review as a conference paper at ICLR 2022
and 16 pixel cutout (De and Taylor 2017) after any augmentation method. We trained Wide-ResNeet (Zagoruyko and Komodakis 2016) models in the 40-2 and 28-10 configurations.
SGD with Nesterov Momentum and learning rate of 0.1, a batch size of 128, a 5e-4 weight decay, and cosine learning rate decay. In ablation experiments also trained for 600 epochs and used custom learning rate schedule. For the first 185 epochs we followed the same cosine learning rate decay schedule, and then switching to a custom step learning rate scheduler detailed on Table This schedule was implemented after observing that just scaling the cosine learning rate schedule to 600 epochs resulted in slow convergence, thus yielding solutions similar training for 600 epochs, and failing reflect the effects of early stopping which this experiment addressed.
Except for epoch ablation experiments,
We used MH steps for the smoothed adversarial unless stated otherwise, and learning rate of 10-3 for the dual ascent step.
No other hyperparameters were tuned or modified with respect to standard settings. The constraint levels were set for Wide-Rest ResNet--4-22 in the wide augmentation space by maximising three fold cross validation over the grid specified on Table Then the constraint levels were adjusted for other architectures and search spaces so that the dual variables at the end of training were small but not zero (of the order of 10-1), which empirically showed good results. The resulting constraint levels corresponding to the results in are detailed in table 5
We used the provided code and reported hyperparameters for running TrivialAugment (Müller and Hutter T and DeepAA (Zheng al.) 2022). For the latter, unlike the results reported in (Zheng et al. 2022), we kept all evaluation hyperparameters (including weight decay) consistent with that of other methods. Results for the wide augmentation space were not included in (Zheng al. 2022). We thus performed the augmentation policy search for the wide search space using the same nyperparameters (except for the augmentation space) reported in (Zheng al.) (202) for CIFAR datasets. We also run the search for both augmentation spaces in SVHN datasets, and evaluated the policy with the same setup as other methods.
Epochs 180-230
LR Scheduler Step
10
230-430
20
430-600
40
Table 3: Learning Rate Custom schedule used when training for 600 epochs. We use the standard scheduler for the first 180 epochs, and then update the LR only every n epochs, where n is the number indicated the second column. This hand designed schedule outperforms using cosine learning rate schedule for 600 epochs, but could improvements convergence speed and performance by exploring other LR-schedulers or tuning it.
Dataset
Constraint Level Grid Range
CIFARTO
[0.2, 2.3]
CIFAR100
[0.3, 2.7]
SVHNcore
[0.1, 1.5]
Table 4: Constraint level grid search space. For each dataset, we evaluated three fold cross validation accuracy for 8 evenly spaced constraint levels in the ranges given the second column. The one with the highest cross validation score was then selected and used to train the model with the full dataset.
17

---

Under review as a conference paper at ICLR 2022
Constraint Level
Standard
Wide
CIFARIO
Wide-ResNet-40-2
0.6
0.8
Wide-ResNet-28-10
0.4
0.8
CIFAR100
Wide-ResNett400-2
0.9
0.8
Wide-ResNeet28--0
0.9
1.2
SVHNcore
Wide-ResNet-28-10
0.1
0.2
Table Constraint levels for different datasets and architectures, for the results presented in Table
18

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL EXPERIMENTS
C.I AUTOMATIC DATA AUGMENTATION
The following section contains additional ablations and discussions about our algorithm. As in previous sections, we use the wide Müller and Hutter (202I) augmentation space and CIFAR image classification benchmarks. Our main motivation is to analyse how the different hyperparameter choices and the learning algorithm affect the generalization and invariance of the obtained solutions. First, we show how the dual variables adapts to different constraint levels during training, and link its dynamics heuristically defined learning curricula. We then evaluate the effect of training for more epochs and link our observations to the known properties of early stopping in unconstrained learning, section C.T.2 section C.1.3 we analyse how the sampling approximation affects regularisation, by performing ablation the number of MH steps. Lastly, include the observed frequencies of sampled transformations for different setups, so as to obtain further insights in how the distribution adapts throughout training (Section
I...tionC..II4) C.1.4).
C.I.1 DUAL VARIABLE DYNAMICS
As already mentioned, the dual variables control the weight of augmented samples during training, thus balancing the trade-off between fitting the primal objective (i.e. loss over training samples) and satisfying the constraints (i.e. loss over transformed samples). the contrary, this trade-off is fixed. In Figure 4 we show how the dual variable adapts to different constraint levels, for Resd-RReNNet-4-22 in CIFAR datasets using the standard setup. Note that for stricter constraint levels, the algorithm has not converged when it reaches 200 epochs. We also observe that the dynamics of our primal-dual algorithm resembles the augmentation learning curricula proposed by Blaas et al. (202T).
penalised approaches, on
CIFARIO
Dual Variable (V)
Constraint
4-
Level (E)
0.4
2
1.2
1.6
0
0
5000
10000
15000
2.0
Training Step
2.4
CIFAR100
Dual Variable (V)
Constraint Level (E)
20
0.5
1.0
10
1.5
2.0
01
5000
10000
15000
2.5
Training Step
3.0
Figure 4: Evolution of dual variables during training for -0et--es in CIFAR datasets. For most levels, augmentation increases until the constraint becomes feasible and then decreases towards the end of training. For stricter levels, the algorithm does not converge in 200 epochs using the standard learning settings. However, the solutions obtained still show good properties.
C.1.2 EARLY STOPPING
Our approach has slower convergence than unconstrained approaches and stopping training arbitrarily after fixed number of epochs result in solutions that unfeasible sub-optimal. However,
19

---

Under review as a conference paper at ICLR 2022
early stopping is popular regulariser, particularly for neural networks trained through gradient descent. Several empirical (Caruana et al., 2000; Rice et al. 2020) and theoretical (Ji et al. 2021 al. 2016) results show its advantages in terms of generalisation and robustness to noisy training data. general, the advantages of early stopping do not lie in the sub-optimalitt of the solution in terms of training error, but nature of the regularization prior imposed, which leads to non-uniform model selection among models with a given training error (Cataltepe al. 1999).
Li et al. 2020a Duvenaud
To the best of our knowledge, there is no literature that explicitly addresses early stopping in empirical primal-dual learning. Whether the generalisation gap in constraint satisfaction can be reduced by early stopping regularisation, in the same manner early stopping regularisation can reduce the generalisation gap in unconstrained learning, thus remains unclear.
Figure shows ablation on the number of epochs. Non-zero dual variables and strict feasiblility show the algorithm has not yet converged 200 epochs. At 600 epochs whereas constraint satisfaction shows little change, training loss decreases and the generalization gap increases. Thus, observe early stopping has a larger impact on the primal objective than on the constraint. that although for stricter levels of the constraint the algorithm has not converged when training is stopped at 200 epochs, it can yield solutions that are still feasible and have a smaller generalization gap.
0.75
Epochs
Split
CR Loss (2)
200
Train
0.50
600
Test
0.25
1.0
1.5
2.0
0.0
Slack
-0.5
-1.0
1.0
1.5
2.0
Dual Variable (V)
10-
5-
1.0
1.5
2.0
Constraint level (E)
Figure 5: Training 400e for 200 (standard) and 600 epochs the CIFAR100 dataset, with different constraint levels.
C.1.3 SAMPLING STEPS ABLATION
As already mentioned in Section we used the uniform proposals. As shown in Figure using more steps of the chain allows samples to deviate further from the uniform distribution, as measured by the decrease in entropy. As already mentioned, dual variables give information of the trade-off between fitting clean and augmented data. We observe the final value of the dual variable is highly correlated with the entropy of sampled transformations. That is, transformations that deviate further from uniformity result in larger dual variables at the end of training. Furthermore, in we show the evolution of dual variables for different sampling
algorithm with independent
20

---

Under review as a conference paper at ICLR 2022
steps. Sampling distributions that are closer to worst perturbations results in stringent requirements, and thus dual variables grow more rapidly.
CIFAR 100
CIFAR 10
Entropy
2.5
2.4
2.4
2.2
8
16
2
4
16
Dual Variable
x
2.5
x
X
0.0-LM
4
8
16
2
4
16
Number of MH Steps
Figure 6: Number of Metropolis Hastings steps ablation for WideResnet-40-2 in CIFAR datasets. The constraint level is fixed for each dataset (0.8 in CIFARIO and 2.1 for CIFARI00), We compute the entropy of the augmentations sampled the last epoch of training. The value of the dual variable at epoch 200 increases with the number of steps, whereas the augmentation distribution entropy decreases. Each point represents an independent run. Experiments were repeated four times.
Dual Variable (V)
MH steps
2
0
5000
10000
15000
16
Training Step
4
Dual Variable (V)
2-
01
5000
10000
15000
Training Step
Figure 7: Evolution of dual variables during training for Wide-ResNet-40-- in CIFAR datasets. The constraint level is fixed for each dataset (0.8 in CIFAR10 and 2.1 for CIFAR 100). As the number of MH sampler steps increases so does the growth of dual variables, which reflects harder to satisfy constraints.
C.1.4 SAMPLED TRANSFORMATIONS
We now include plots of the empirical transformation distributions. As shown in Figure [ the fre- quency with which transformations are sampled varies depending on the dataset, which is desirable property. Similar ZZhang al. (2020), observe prevalence of geometric transformations,
21

---

Under review as a conference paper at ICLR 2022
unlike Cubuk al. (2018). In SVHN color transformations are less frequent than in CIFAR datasets, and the frequency of geometric transformations increases, which is interesting due to the perceptual importance of shape in the digit recognition task.
As already noted, using more steps allows the chain to deviate further from the uniform proposal distribution. We thus plot the frequency of sampled transformations across training epochs Figure 9 We observe that as training progresses, the observed frequencies also deviate further from uniformity. This suggests that, due to the dataset and architecture, some transformations may be harder to fit than others.
Similarly, include histograms for the sampled transformation levels in Figure TO Extreme levels are sampled more frequently, but the empirical distributions vary depending on the transformation. The deviation from uniformity and the differences between transformations are accentuated as sampling steps increase (Figure 11).
Proportion Transformations
1.0
TranslateX
Translatey
0.8
ShearX
0.6
ShearY
Rotate
0.4
AutoContrast
Brightness
0.2
Color
0.0
Contrast
CIFAR10
CIFAR100 Dataset
SVHNCORE
Equalize
Posterize
Sharpness
Solarize
Figure 8: Frequency of sampled transformations for CIFAR and SVHN datasets using MH
sampler steps, for Wide-Resnet-28-10.
22

---

Under review as a conference paper at ICLR 2022
First Epoch
Last Epoch
1.0
Translatex
0.8
Translatey
ShearX
0.6
0.6
ShearY
Steps:
AutaContrast
Brightness
0.2
Color
0.0
0.0
Contrest
CIFAR10
Equalize
Posterize
1.0
1.0
Sharpness
Solarize
0.6
Steps: 4
Proportion of Transformations
0.4
0.4
0.0
0.0
CIFAR100
CIFAR10
CIFAR100
CIFAR10
1.0
1.0
0.8
0.8
Steps: 8
0.4
0.4
0.2
0.2
0.0
CIFAR100
CIAR11O
CIFAR100
CIFAR10
1.0
0.8
0.8
0.6
0.6
Steps: 16
0.2
0.2
CIFAR100 CIFAR10
CIFAR100
CIFAR1O
Figure 9: Frequency of sampled transformations for CIFAR datasets in the first and last epochs of training, for Wide-Resme R0-2ee--0--. As the number of steps increases, the the entropy of sampled transformations decreases, i.e., observed frequencies get further from uniformity.
23

---

Under review as a conference paper at ICLR 2022
2 MH steps
CIFAR100
CIFAR10
250
250
Translatey
Epoch
First
20
30
0
10
20
30
250
250
Last
TranslateX
20
200
ShearY
0
10
20
30
0
10
20
30
200
200
ShearX
20
0
30
200
200
Rotate
20
10
20
30
Frequency
Brightness
100
100
0
10
20
30
0
10
20
30
100
100
Color
20
0
30
Contrast
100
100
0
10
20
30
0
10
20
30
Posterize
0
10
20
30
0
10
20
30
100
Sharpness
20
30
200
Solarize
20
10
20
30
Transformation Levels
Figure 10: Frequency of sampled transformation levels across epochs, for different transformations, using two MH steps. Extreme levels are sampled more frequently, but some transformations deviate further from uniformity.
24

---

Under review as a conference paper at ICLR 2022
16 MH steps
CIFAR100
CIFARIO
1000
TranslateY
500
Epoch
First
30
10
30
Last
1000
Translatex
500
0
10
20
30
10
20
30
250
250
ShearY
0
10
20
30
10
20
30
250
ShearX
10
30
10
30
250
250
Rotate
10
20
30
10
20
30
Frequency
200
250
Brightness
0
10
20
30
10
20
30
100
Color
10
30
10
30
200
Contrast
100
0
10
20
30
10
20
30
200
200
Posterize
0
10
20
30
10
20
30
100
100
Sharpness
10
30
10
30
200
200
Solarize
0
30
10
20
30
Transformation Levels
Figure 11: Frequency of sampled transformation levels for the first and last epochs, for different transformations, using sixteen MH steps. The frequencies concentrate in extreme values for some transformations.
25

---

Under review as a conference paper at ICLR 2022
C.1.5 RUNTIME ANALYSIS
As mentioned in section 4.3 the added computational cost of our algorithm is that of computing a forward pass for each MCMC step. As a result, trade-off between sampling X accurately and computation arises. Table O we provide an empirical runtime analysis for our method for different numbers of MH steps, and compare it with the training time of baseline methods. These times corre- spond solely to training, and it difficult to account for the time taken to tune the hyperparameters of each method, which hinders direct comparisons. In the case of DeepA (Zheng et al. (022), it requires running a data augmentation policy search that takes 11 hours (more than 5x training time) using the same hardware.
Method
Ours
TA
DeepAA
2 steps
4 steps
8 steps
16 steps
Epoch time (s)
12.6
13.3
32.5
51.2
89.4
165.2
Table 6: Time per epoch for WideResnet 40-2 in CIFAR 10 dataset, on workstation with one NVIDIA RTX 3090 GPU and AMD Threadripper 3960X (24 cores, 3.80 GHz) CPU.
C.2 DATA AUGMENTATION CAN HINDER PERFORMANCE
As already mentioned in section A.2 there is empirical evidence that certain distributions over commonly used transformations introduce biases that are detrimental to model performance and generalization (Blaas et al. 2021. We provide a simple experiment to show that there exist transformation distributions that make data augmentation deteriorate model performance, and that in such cases balancing the amount of data augmentation (e.g. by including the original data) is important to mitigate and overcome this issue.
We restrict the transformations in the wide augmentation space (Müller and Hutter 2021) to their maximum magnitude. We sample transformations according X using two MH steps as previously described. We compare against training without augmentation, and training using both augmentation and the original data (i.e. adding the identity) equally weighted. While sampling transformations according to X deteriorates performance with respect to the model without augmentation, including both the identity and the augmented samples achieves a superior performance, as shown in Table
No augmentation
X
X + Identity
78.42 0.31
75.19 19 + 0.54
80.01 + 0.26
Table 7: Image Classification test accuracy for WideResnet 40-2 in CIFARI00. trained using different augmentation policies defined over the wide (Müller and Hutter 2021) augmentation space. The first column corresponds to using the training data without applying any transformations, and the second column to sampling transformations according A which results in lower accuracy. The third column corresponds to using both the original and augmented data equally weighted, which leads an improvement in accuracy. We report the mean and confidence intervals computed over five independent runs.
C.3 SYNTHETIC INVARIANCES
Although our approach does not aim to learn the set of symmetries invariant transformations from the data, but rather to impose on the predictor, dual variables can be used to gather insights on the actual invariances underlying learning task.
We showcase this on datasets with artificial invariances, following the setup of [mmer et al. (022)) Namely, apply transformations independently drawn from the distributions specified in Table to each sample in the datasets MNIST (LeCun et al.) 2010) and FashionMNIST (Xiao et al.) 2017) We
the same MLP and CNN architectures and hyperparameters (Immer et al. 2022)
We run our algorithm constraining the loss on the transformation spaces detailed in Table 9 which (except for the fully rotated dataset) are larger than the true transformation range used construct
26

---

Under review as a conference paper at ICLR 2022
the synthetic dataset. Note that the same transformation sets and constraint levels (c) for all synthetic datasets. As shown in table 10 except for scalings in FashionMNIST, the dual variables (y) associated with transformations corresponding to the true synthetic invariances in the dataset are considerably smaller. This indicates that when the transformations in the constraint correspond to true invariance of the dataset, the constraint is easier satisfy. Since the transformation ranges in the constraint and those used to construct the dataset do not match exactly (except for the fully rotated dataset), the degree of invariance to the transformation sets in the constraints is only approximate.
Synthetic invariance
Parameter
Distribution
Full Rotation
Angle in radians.
U - 3.
Partial Rotation
Angle radians.
U n. m
Translation
Translation in pixels.
u[-8,8]2 (1) log(2)
Scale
Exponential Scaling factor.
Table 8: Sampling parameters for transformations used to obtain synthetically invariant datasets, from (Immer al. 2022).
Constraint Set
Parameter
Range
Rotations
Angle in radians.
[- m]
Translation
Translation in pixels.
[-16, 16]2
Scale
Exponential Scaling factor.
1.5,, 1.5]
Table 9: Transformation sets used as invariance constraints. All sets are used simultanously, with the same constraint level (c) for all datasets.
Synthetic Invariance
Dataset
Architecture
Rot.
Partial Rot.
Translation
Scale
MNIST
MLP
Rotation
0.000
0.004
3.224
0.035
Translation
1.344
0.038
0.289
0.032
Scale
1.800
0.045
4.206
0.004
CNN
Rotation
0.000
0.002
2.724
0.012
Translation
1.218
0.009
0.439
0.006
Scale
2.026
0.049
4.029
0.003
F-MNIST
MLP
Rotation
0,000
0.037
4.470
1.599
Translation
3.572
1.934
0.939
0.717
Scale
4.144
2.653
3.472
0.754
CNN
Rotation
0.000
0.107
3,301
1.352
Translation
3.572
1.426
0.515
0.441
Scale
4.144
2.332
2.725
0.904
Table 10: Value of dual variables (after 400 epochs) for different transformation constraints and synthetic invariant datasets. Columns correspond different transformations of the dataset, and rows to dual variables associated with different transformations. Except for scaling in FashionMNIST. for all architectures and dataset the dual variable associated with the constraint corresponding to the transformations applied to the dataset is considerably lower. The smallest dual variable for each dataset and architecture bolded.
27

---

Under review as a conference paper at ICLR 2022
D RELATED NOTIONS OF INVARIANCE
There are several definitions of invariance that capture different properties of the solution or data distribution under the action of transformations in G.
In the context of supervised learning, the data distribution is said to be exactly invariant iff it does not change when transformations are applied the covariates, i.e.,
(x, v) - (gx, y), for all 9,
where =y denotes equality in distribution. Note that this can be relaxed to approximate equality in distribution, i.e. that the distribution shift induced small in an appropriate metric, e.g. Wassertain distance (Chen et al. (019). This definition implies that the probability of a given label is exactly or approximately the same for a feature vector x and its transformed versions gx:
P()x)) P(ylgx)
On the other hand, a predictor can satisfy hard invariance, explicitly,
fa xx) fa(gx) for all E G, x X
which needs to hold point-wise over the transformation set and input space, This can relaxed by taking the mean over the data distribution to obtain definition of statistical invariance, i.c.
(xxa)~ [ffe (x)] = (x.))~> [fe (gx) for all g.
which still needs to hold for every transformation in G.
This notions of invariance do explicitly contemplate the task hand, that is, not all changes in equally affect performance. Thus, we use the loss to encode meaningful differences in labels with respect the task, described in section 3.1 Throughout this work, we thus refer to the invariant risk Ry defined on equation 5 as the degree of invariance, unless otherwise noted.
28


Reviews:
[{"review_id": "ffo_XD8cr7v", "reviewer": "Reviewer_wrTV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes to automatically learn which transformations to augment during training without introducing biases, for example, when the augmentations are not appropriate for the task. Authors rewrite the problem as a semi-infinite constrained problem similar to a recent adversarial robustness framework. Proposed approach learns to sample from a distribution over the transformations using a Metropolis-Hastings approach. ", "summary_of_the_review": "Paper can be improved with a more formal treatment of the problem and method (e.g., in defining the problem, discussing sample complexity of the proposed method). Empirical evaluation does not show case the claims of advantages of the proposed method vs the baselines (tasks where some augmentations are not appropriate, non-differentiable transformations). ", "strength_and_weaknesses": "Strengths:\n\n1. Automatically learning data augmentations for a particular task is a significant and important challenge which helps not just in-distribution but also for out-of-distribution robustness of models. \n2. Not requiring differentiable transformations (unlike baselines) can be helpful. \n3. Typically, the tradeoff between the original task loss and the augmentation loss is done via hyperparameter tuning, the proposed approach instead maximizes the relevant dual variable (whose advantages are discussed in the paper). \n\nWeaknesses:\n\n1. I think the paper can be improved with a more formal treatment of the problem and method. \n   1. The problem is not formally defined, for example, the paper never discusses formally the true invariances of the data distribution. Essentially, the task of an automatic data augmentation method will be to learn transformations that hold for the true (unknown) data generation process.\n   2. Degree of invariance and approximate invariance are used loosely and not well-defined. \n   3. The paper is motivated solely on improving the sample complexity but no such bounds are provided; while data augmentation itself can lower the sample complexity, it is unclear if that continues to hold when learning augmentations automatically especially with an approximation from constrained optimization (CSRM) to unconstrained optimization (D-CERM). \n\n2. Empirical evaluation does not show case the claims of advantages of the proposed method vs the baselines.\n\n   1. Experiments do not explicitly show cases when data augmentation results in biasing the model and is harmful, which is one of the main reasons for learning the appropriate distributions over the transformations. \n\n   2. Another advantage of the proposed method compared to the baselines is to allow non-differentiable transformations, but no such experiments are performed. \n   3. Overall, the empirical gain in performance is not significant compared to the baselines.\n\n\n\nOther Questions/Comments:\n\n1.  Introduction: “The use of invariance overcomes the need to search for a specific distribution over transformations”. I do not understand this statement. \n2. As opposed to the existing methods, $\\gamma$ is treated as a dual variable (and optimized) which allows the tradeoff to be learnt directly. However, $\\epsilon$ is a hyperparameter which is cross-validated which also trades-off the approximate invariance, thus leading to the same problem. Further, it is not clear from the paper, what cross-validation methodology is used (i.e., what is the training and test datasets for this cross-validation)?\n3. There seems to be a single slack variable \\epsilon that controls all different types of transformations (for example, rotations, translations, shear, etc) with no additional regularization to force toward invariance unlike [1].\n\n3. I do not agree with the statement “sampling $\\lambda^*_c$ accurately is not a concern given that we are interested in promoting approximate invariance” as we still need to characterize the kind of approximation. For instance, using a constraint level ($\\epsilon$) is one way of approximation but it is different from not able to accurately estimating $\\lambda_c^*$.\n\nMinor/Typos:\n\n1. Typos: “whithout”, “infromativeness”\n\n2. $\\lambda^*_c$ takes different inputs at different places in Section 4.2. \n\n   \n**References**\n[1] Benton, Gregory, et al. \"Learning invariances in neural networks from training data.\" Advances in neural information processing systems 33 (2020): 17605-17616.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_SC61x5D7t", "reviewer": "Reviewer_159N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies an interesting problem on how to automatically generate augmented data for learning invairance representations. The paper proposes a MCMC based methods, and then tested the results in some datasets. The paper seems to be on a novel track of goals, but needs further development.  ", "summary_of_the_review": "The paper seems still in its preliminary stage, a little bit limited in technical novelties, and more importantly empirical validation. I'm happy to give it a more thorough read if the authors can provide more convincing and more comprehensive empirical evaluations. ", "strength_and_weaknesses": "- strength\n    - the study of generating augmented samples for invariance seems very novel to me. \n    - the writing of the paper is reasonably good\n- weakness\n    - the algorithm still needs further development, it's hard for me to believe a MCMC based method is the best solution in this scenario, given the rich techniques developed in deep learning regime. \n    - the empirical scope is quite limited: the deepAA method is barely compared. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "YXStEoMbe5", "reviewer": "Reviewer_E3e9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel approach to learn a data augmentation method by formulating the task as invariance-constrained learning problem. They leverage Monte Carlo Markov Chain (MCMC) sampling to solve it, which achieves state-of-the-art results on CIFAR10 and CIFAR100 for specific wide neural network architectures.\n", "summary_of_the_review": "While the paper is well motivated and clearly written, some relevant analysis (like the runtime analysis, potentially additional experiments that highlight the specific advantages of the proposed approach, etc.) are missing. ", "strength_and_weaknesses": "Strengths:\n+ The overall presentation is clear, the background is explained in detail, and the respective terms and their high level interpretation are discussed.\n+ The proposed data augmentation learning method seems to outperform the baselines on CIFAR10 and CIFAR100.\n+ The presented problem formulation has the advantage that if a solution to the unconstrained problem is feasible, the presence of that constraint has no effect on the statistical problem.\n-> Could this fact also be helpful to assess whether the learned transformations are consistent with the data distribution or impose an unwanted bias?\n\nWeaknesses and open questions:\n- Competing papers (e.g., Müller and Hutter, 2021) compare run times of different augmentation algorithms. Such an analysis is completely missing.\n- The authors make multiple claims regarding the advantages of their formulation as constraint optimisation problem but these are not demonstrated experimentally.\n(Can invariances be learned?)\n- In contrast to competing methods, no experiments on ImageNet are provided. This is relevant for the question whether the proposed data augmentation method scales to large, complex datasets.\n- The authors claim themselves that they fail to achieve large improvements in accuracy over baselines but this could be attributed to a stagnation in data augmentation research (Müller and Hutter, 2021) and \ncould reflect limits of the benchmarking setup. In this case, the authors could also look for other datasets and demonstrate improvements of their method over benchmarks there.\n- A general question for this line of research: Why are no synthetic experiments conducted to test the ability of automated data augmentation algorithms to recover symmetries in the data? It would be easy to generate data that is invariant to some transformations and test how many samples are required to learn the correct transformations. The claim that the relevant transformations are learned is otherwise unsupported.\n- The data augmentation proposal seems to rely on a couple of hyper parameters that need to be tuned in practice.\n- A couple of hand-crated baselines are discussed in the introduction but not compared with in the experiment section.\n\nPoints of minor critique and open questions:\n- No guarantees with respect to the convergence of the sampler to the equilibrium distribution are provided. (This would, however, be a hard problem and might be too much to ask for.)\n- The related work section is quite short and adds little value in addition the introduction.\n- Figures 1 and 2 are a bit difficult to read for yellow markers.\n- Considering that a lot space is left in the main paper, the conclusions and insights into the proposed method could be extended. In particular, moving the analysis of the sampled transformations could highlight additional features of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QLv2fWchEV", "reviewer": "Reviewer_8185", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to treat data augmentation in the context of a constrained optimization problem, where the constraints are defined using a set of desired invariances. The data augmentations are then chosen using an MCMC scheme based on the loss of the model (which is supposed to be invariant), while the strength of their influence is optimized using a Langrage multiplier.", "summary_of_the_review": "Overall, I think this is a really nice idea with a strong theoretical motivation. If the experiments could be improved, I would be happy to recommend acceptance; for now, I'm keeping my score slightly lower due to the inconclusive experiments.\n\nUPDATE: Based on the changes made by the authors during the rebuttal, I have updated my score.", "strength_and_weaknesses": "Strengths:\n- The problem of tuning data augmentation to different tasks is important.\n- The paper is clearly written and the theory seems correct.\n\nWeaknesses:\n- The experiments are a bit limited and inconclusive.\n\nMajor comments:\n- How does the choice of just two MC steps in the experiments affect the performance? I would imagine that the samples stay quite close to the uniform proposal then instead of actually sampling from the right distribution.\n- DeepAA seems like a quite strong baseline, it outperforms all other methods in one out of two experiments where it is included. Why is it not included in the other ones? I feel like it should be.\n- Given that TA is designed to be a ridiculously simple approach, it performs quite well compared to the proposed method. How does it compare in terms of runtime? I would imagine that TA would be much faster, is that true?\n- I appreciate the image classification experiment, but I think it would be illustrative to have an experiment with artificial invariances (similar to the ones in [1]), so one could discern to what extent the learned augmentations would correctly capture those.\n\nMinor comments:\n- The related work could mention [1]\n- In Eq. 3, what is $h$? Should that be $f_\\theta$?\n- Sec. 3.2: a intance -> an instance\n\n[1] https://arxiv.org/abs/2202.10638", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject