Instance-Specific Augmentation: Capturing Local Invariances
Authors: Ning Miao
Tom Rainforth
Emile Mathieu
Yann Dubois
Yee Whye Teh
Adam Foster
Hyunjik Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: inductive bias
data augmentation
invariance

Abstract: We introduce InstaAug, a method for automatically learning input-specific augmentations from data. Previous data augmentation methods have generally assumed independence between the original input and the transformation applied to that input. This can be highly restrictive, as the invariances that the augmentations are based on are themselves often highly input dependent; e.g., we can change a leaf from green to yellow while maintaining its label, but not a lime. InstaAug instead allows for input dependency by introducing an invariance module that maps inputs to tailored transformation distributions. It can be simultaneously trained alongside the downstream model in a fully end-to-end manner, or separately learned for a pre-trained model. We empirically demonstrate that InstaAug learns meaningful input-dependent augmentations for a wide range of transformation classes, which in turn provides better performance on both supervised and self-supervised tasks.

Under review as a conference paper at ICLR 2023
INSTANCE-SPECIFIC AUGMENTATION: CAPTURING LOCAL INVARIANCES
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce InstaAug, a method for automatically learning input-specific augmen- tations from data. Previous data augmentation methods have generally assumed independence between the original input and the transformation applied to that input. This can be highly restrictive, as the invariances that the augmentations are based on are themselves often highly input dependent; e.g., we can change leaf from green to yellow while maintaining its label, but not a lime. InstaAug instead allows for input dependency by introducing an invariance module that maps inputs to tailored transformation distributions. It can be simultaneously trained alongside the downstream model in a fully end-to-end manner, or separately learned for a pre-trained model. We empirically demonstrate that InstaAug learns meaningful input-dependent augmentations for wide range of transformation classes, which in turn provides better performance on both supervised and self-supervised tasks.
INTRODUCTION
Data augmentation is an important tool in deep learning (Shorten & Khoshgoftuar, 2019). It allows one incorporate inductive biases and invariances into models (Chen et al., 2019; Lyle al., 2020), providing highly effective regularization technique that aids generalization (Goodfellow et al., 2016). It has proved particularly successful for computer vision tasks, forming an essential component of many modern supervised (Perez & Wang, 2017; Krizhevsky et al., 2012; Cubuk et al., 2020; Mikolajezyk
Grochowski, 2018) and self-supervised (Bachman et al., 2019; Chen et al., 2020; Tian et al., 2020; Foster et al., 2021) approaches.
Algorithmically, data augmentations apply a random transformation T X - X, ~ p(T), ), to each input data point transformations are resampled each time the data point is used (c.g. at each training epoch), effectively populating the training set with additional samples. Augmentation is also sometimes used at test time by ensembling predictions from multiple transformations of the input. A particular augmentation is defined by the choice of the transformation distribution p(T). whose construction thus forms the key design choice. Good transformation distributions induce substantial and wide-ranging changes to the input, while preserving the information relevant to the task at hand.
X. before feeding this augmented data into the downstream model. These
Data augmentation necessarily relies on exploiting problem-specific expertise: though aspects of P(7) can be learned from data (Benton al., 2020), trying to learn p(r) from the set of all possible transformation distributions is not only unrealistic, but actively at odds with the core motivations of introducing inductive biases and capturing invariances. One, therefore, restricts to transformations that reflect how we desire our model to generalize, such as cropping and color jitter for image data.
Current approaches (Cubuk et al., 2018; Lim al., 2019; Benton al., 2020) are generally limited to learning augmentations where the transformation is sampled independently from the input it is applied to, such that p(7) has no dependence on x. This means that they are only able to learn global invariances, , severely limiting their flexibility and potential impact. For example, when using color jittering, changing the color of a leaf from yellow to green would likely preserve its label, but the same transformation would change lemon to a lime (see Figure 1b). This transformation cannot be usefully applied as a global augmentation, even though it a useful invariance for the specific input instance of a leaf. Similar examples regularly occur for other transformations, as shown in Figure 1.
To address this shortfall, we introduce InstaAug, a new approach that allows one to learn instance- specific augmentations that encapsulate local invariances of the underlying data generating process,

---

Under review as a conference paper at ICLR 2023
0
6 6 o 9
(a) Rotation
(b) Color jittering
(c) Cropping
Figure I: Different inputs require different augmentations. In (a), the digit '0" is invariant to any rotation, but rotating the digit '6' by more 90º makes it a '9'. In (b), a similar phenomenon is observed for color jittering applied to leaf and lemon/lime. The red dashed lines in (a) and (b) are boundaries between different classes. In (c), the same effect is shown for cropping. Solid rectangles reepeee dashed rectangles represent patches with different labels to the original images.
represent the patches that preserve the labels of the original images (leeft] grass, [right] cattle), while
that is invariances specific to a particular region of the input space. InstaAug is based on using a transformation distribution of the form p(T; øl (x)), where & is a deep neural network that maps inputs to transformation distribution parameters. We refer to & as an invariance module. It can be trained simultaneously with the downstream model in fully end-to-end manner, using a fixed pre-trained model. Both cases only require access to training data and a single objective function that minimizes the training error while maintaining augmentation diversity. As such, InstaAug allows one to directly learn powerful and general augmentations, without requiring access to additional annotations.
We evaluate InstaAug in both supervised and self- supervised settings, focusing on image classification and contrastive learning respectively. Our experimental results show that InstaAug is able to uncover meaningful invariances that are consistent with human cognition, and improve model performance for various tasks compared with global augmentations. While we primarily focus on the case where the invariance module is trained alongside the downstream model (to allow data augmentation during training), we find that InstaAug can also provide substantial performance gains when used as a mechanism for learning test-time augmentations for large pre-trained models.
BACKGROUND
Data augmentation methods operate wrapper algorithm around some downstream model. f. randomly transforming the inputs x E X' before they are passed to the model. The outputs of the augmented model are given by fx((x)), where 7 X' + X represents the transformation, sampled from some transformation distribution P(T). The aim of this augmentation is to instil inductive biases transformanl
into the learned model, leading improved generalization by capturing invariances of the problem. It can be used both during training to provide additional synthetic training data, and/or at test-time, where ensembling the predictions from multiple transformations can provide useful regularization i
that often improves performance (Shanmugam et al.. 2021).
Some approaches look learn aspects of the augmentation (Cubuk et al., 2018; 2020: Lim et al., 2019; Ho et al., 2019; Hataya et al., 2020; Li et al., 2020; Zheng et al., 2022). These approaches can be viewed as learning parameters of p(7). helping to automate its construction and tuning. Of particular relevance, Augerino (Benton et al., 2020) provides a mechanism for learning augmentations using a simple end-to-end training scheme, where the parameters of the downstream model and transformation distribution are learned simultaneously using the (empirical) risk minimization
miny. Ex.yy-am [E--pp((r) [C(((r(x), w)]] + XIR (0),
XIR(O),
(1)
term that encourages large transformations. .
where L is a loss function and AR(() is regularization a
All of these approaches can be thought of global augmentation schemes, in that transformations are sampled independently to the input. For an unrestricted, universal, class of transformations, this assumption can be justified through the noise outsourcing lemma (Kallenberg Kallenberg, 1997): any conditional distribution Y|X = x can be expressed as a deterministic function g X' x R > y of the input and some independent noise s ~ (O I). Thus, using reparameterization, the dependency x can, in principle, be entirely dealt with by the transformation itself. However, in practice, the transformation class must be restricted to provide the desired inductive biases, meaning this result no longer holds and so the independence assumption can cause severe restrictions. For example, sampling rotations independently to the input is equivalent to the unrealistic assumption that the labels of all images invariant to the same range of angles (cf. Figure la).
conditionn
2

---

Under review as a conference paper at ICLR 2023
3 INSTANCE-SPECIHE AUGMENTATION: CAPTURING LOCAL INVARIANCES
In order to remedy the problems of global augmentations, propose InstaAug. InstaAug learns an imput dependent distribution p(T: makes use of the input x via the invariance module Φ as opposed learning a global transforma- tion distribution (T). This generalizes the hypothesis class of transformation distributions, and significandy increases the flexibility and expressivity of the augmentations we can lear,, without un- dermining our ability to carefully control the inductive biases that are imparted. It can also informally be viewed as mechanism for learning invariances which are local the specific input.
p(T:((x)) o(x)) of information prreserringg transformations that actively
We argue that good augmentation strategy needs to fulfill two properties. First, the transformations should preserve the information in x that is necessary for the task hand. For example, in classifica- tion, transformations must preserve sufficient information to correctly classify T Txx). Second, the set of transformations needs to have sufficient 'diversity' to effectively augment the data; we quantify this as the entropy of the transformation distribution p(T; ((x)). In addition to their intuitive nature, in Appendix A we provide theoretical analysis that shows these requirements naturally originate from decomposition of the generalization error that results from using é when training f. For simplicity, we describe InstaAug on the task of classification in the remainder of this section.
InstaAug
3.1 MODEL STRUCTURE
Inv module: &
InstaAug is based around using a simple plug-in invariance module, 6, between the input x and the classifier f, as shown in Figure 2. We assume a parametric family of distributions p(r:-) over some transformation space, then use é, which is a trainable neural network, to predicts its parameters for a given input. During training, we sample a transformation p(T; (xx)). which is applied to x to generate an augmented sample T(x), before feeding this into the classifier f.
((x)
o(x))
X
Classifier:
Figure 2: Summary of InstaAug.
3.2 TRAINING
Good augmentations should induce sub- stantial changes to the input x while pro- viding all necessary information of the task at hand, thereby capturing the maxi- mum possible invariance. Figure 3a illus- trates the tension between these two objec- tives experienced by global augmentation schemes. Wider Wider-ranging transformations are generally beneficial for generalization, but "excessive' transformations will gen- erate samples that will be incorrectly clas- sified. In Figure 3a we see this in the red area, where the augmentations for a pair of data points have started overlap, creating ambiguity and inevitably misclas- sifications. Using instance-specife aug- mentations (Figure 3b) allows for a bet- ter trade-off of these needs. However, to achieve this we need our objective to en- courage diversity in augmentations, not just low training error. It should also let the level of diversity vary between inputs, some points will be able to support larger transformations than others.
(a) Global augmentation
(b) InstaAug
Figure 3: InstaAug learns more diverse augmentations that also preserve labels compared to global augmenta- tions. * and
samples from two different classes. Blue and green shades represent label-preserving aug- mentations for each class. In (a), the upper * should be further augmented, but some of the augmented samples for the lower * are already over-augmented and indis- tinguishable from another class (see the red intersection). InstaAug solves this problem by learning a different aug- mentation for each instance, as shown in (b).
Based on these needs, training is done by simultancously minimizing a conventional expected loss with respect to both & and f (or just @ if ff is a fixed pre-trained classifier as per Section 5.3), while placing a hard constraint on the average entropy of the transformations, Ex ([][(T=((x)))
3

---

Under review as a conference paper at ICLR 2023
The core motivation for this setup is that minimizing the expected loss will naturally encourage the information needed for prediction to be preserved, but the constraints on the entropy are needed to enforce diversity. Further motivation is provided by the theoretical analysis of Appendix A.
By appropriately parameterizing p(T; d(x)) (see Section 3.3), we can write down its entropy in closed form. We can then formulate the problem as the following constrained optimization problem:
min f:P
Expxmran
(2a)
s.t.
Ex.gr-an ((x))]((())]]]] E [Hmin Hmux];
(2b)
where L is the loss, for which we will generally use the cross-entropy. Here the lower bound ((x)) enforces the desired diversity. We typically expect this constraint to be active at the true
optimal solution, so Himn can be thought of as a hyperparameter that controls the desired level of
diversity. The upper bound prevents p(;; o(x)) exploding at the start of training when the classifier is weak: without this we empirically find that the augmented samples from different classes tend to overlap in the initial phase of training, hindering the training of f.
The Lagrangian function, Ex [Ex=((r:(()) [X)(((x))) can be used to solve this constrained optimization, where À is the Lagrangian multiplier. In practice, we initialize À with small positive value, and increase (decrease) À when the average entropy drops below Hmin (exceeds Hmax). The invariance module and downstream model can thus be trained simultaneously using end-to-end gradient descent, utilizing the reparameterization trick to deal with the stochasticity of when possible (Kingma & Welling, 2014), and the REINFORCE estimator (Williams, 1992) otherwise. The approach can also be extended to regression or self- supervised learning by substituting the loss function L (ef. Appendix C).
X [T[[[;; d(x))]],
3.3 PARAMETERIZATION OF AUGMENTATIONS
We focus on parameterizing transformations that are frequently used in computer vision, though our framework can easily be extended to other domains. Due to the varied characteristies of different image transformations, we design two different parameterization methods for p(r; ((x)).
Uniform parameterization. For rotation and color jittering, we find that uniform distribution suitable for parameterizing p(r; Φ(x)), such that Φ(x) returns a pair (Omins mmax) representing extrema of the possible transformations. For example, for rotations these represent the maximum and minimum rotation angles, such that r(x) = R(0)= x, where ~ (d(mmin mmaxx). To compose multiple transformations (such hue, saturation and brightness in color jittering), we simply sample them independently, such that p(TI, p(Ti....; TK; (xx) = II P(Tx; du (x)). This provides similar parameterization to (Benton et al., 2020), but where (0minn mmax) now critically varies with the input x and there is no symmetry assumption on the transformation ranges. Output logits across patches
Location-relates parameterization. Using this uniform parameterization is unfortunately not appropriate for crop- ping. Firstly, the distribution crop centers may be multi-modal, since im- portant information may exist in dif- ferent parts of an image. Secondly, the desired crop size and center are of- ten highly correlated so cannot be sam- pled independently. Finally, we encoun- tered significant practical training issues when using the uniform parameteriza- tion for cropping, with ø often becom- ing trapped in local optima with little transformation diversity.
Flatter and concatanate
Input: image
CNN
Figure 4: Location-related parameterization of crops by CNN. The shaded area (bottom right) shows a simplified 3-layer CNN, and squares represent units at different con- volutional layers. Each units defines a patch in the input image (shown in the same color) through its receptive field. The value of the activation then gives the corresponding unnormalized log probability for that patch.
We therefore propose an alternative location-related parameterization (LRP) for cropping, which is based on defining a large fixed set of allowable crops then constructing ø to map from inputs a vector of probabilities over this set. As shown in Figure 4, this is achieved using
4

---

Under review as a conference paper at ICLR 2023
CNN where each hidden unit corresponds to possible crop. The units from all layers utilized, with those of earlier layers representing smaller crops. This parametrization proved more effective than simply outputting the probabilities from conventional network, due to the greater parameter sharing between related crops. We note that it can also be directly extended to other transformations, such masking, local blurring, pixel-wise perturbation, and local color jittering.
3.4 TEST-TIME AUGMENTATION
Besides augmenting data during training, the learned invariance can also be applied to test-time augmentation. Given test image x, sample n different transformations 7; from o(x)) and apply them to x to generate n different views (x). After feeding these views to the classifier, f. we use the mean logit I augmentation, InstaAug can be trained with fixed pre-trained classifier at a lower computation cost.
n Σi= (x)( to predict x's label. When only learning invariance for test-time
4 RELATED WORK
Hard-coded invariance. Much recent work has been devoted to hard-coding global invariance in neu- ral networks. For example, various models have been designed to be invariant translation (Chaman & Dokmanic, 2021; Zhang, 2019), rotation (Worrall et al., 2017: Zhou et al., 2017; Marcos et al., 2017), scaling (Worrall & Welling, 2019; Sosnovik et al., 2019) or other group actions (Cohen Welling, 2016; Xu et al., 2021). Unfortunately, they require the set of invariant transformations to be closed under composition, leaving out many practical transformations that do not form a group.
Learning augmentations. There have been numerous prior works that automatically learn global augmentations and invariance from data, As discussed in Section 2, Augerino (Benton et al., 2020) is perhaps the mostly closely linked such approach to InstaAug end training (see Appendix for further discussion on its similarities and differences to InstaAug). AutoAugment (Cubuk et al., 2018) instead uses reinforcement learning to find augmentation strategies that increase accuracy on a separate validation set. Various follow-up works have improved its efficiency and/or performance (Lim al., 2019; Ho et al., 2019; Hataya et al., 2020; Li et al., 2020; Cubuk et al., 2020; Tang et al., 2019; Zheng et al., 2022). A small number of works have further looked to learn augmentation policies that have some dependency label (Zhou et al., 2021; Cheung & Yeung. 2022). These approaches focus on choosing which type(s) of transformation to apply from fixed list-c... choosing from crop. blur, or color jitter-which may include small number of discrete options for transformation strength. By comparison, InstaAug keeps the type of transformation fixed and learns instance-specific parameter for the transform distribution, such as the positions and sizes of patches for cropping. In principle, it should be possible to combine these complementary approaches with InstaAug, though we note they require a separate validation dataset and cannot be used in unsupervised settings, unlike InstaAug.
it also relies end-to-
the input or just the class
Other related work. The spatial transformer (Jaderberg et al., 2015) aims to learn instance-specific transformations, but only applies single transformation each input rather than distribution of transformations, making it distinct from data augmentation. Luo et al. (2020) and Kim et al. (2020) both also learn instance-specific augmentations. However, the latter consider only test-time augmentation, while the former introduces an approach that is highly specialized to test recognition and cannot be applied in more general settings we consider. Tamkin al. (2020) and (Chen et al., 2021) both utilize adversarial augmentations to increase robustness. Zhou et al. (2020) learn symmetries shared across several datasets through meta-learning scheme.
SUPERVISED LEARNING EXPERIMENTS
5.1 ROTATED 2D IMAGES
We first consider a toy synthetic dataset proposed in Benton et al. (2020). The dataset contains four categories, (1) upright Mario; (2) upside-down Mario; (3) upright Iggy; and (4) upside-down Iggy. Each of the four base images is randomly rotated in the interval of [-T/4, म 7/4] to form the training dataset. The task is to predict the correct character (Mario Iggy) and the orientation (up vs down). We assess whether InstaAug is able to learn the 'best' rotation range for each sample- i... the maximum range that avoids 'up' and 'down' classes from overlapping.
5

---

Under review as a conference paper at ICLR 2023
Figure shows that InstaAug effectively recov- ers the broadest range of rotations for each im- age while preserving labels, while Augerino only learns a subset of these ranges. This can be most easily seen by the fact that the trans- formation distributions (shown in green) always extend very close to the true class boundary for Insta/ug, but not for Augerino. These gains are because Augerino learns single global augmentation distribution shared across all im- ages (note the shared transformation distribution ares), which are inevitably limited for any given input.
O
(a) Augerino
(b) InstaAug (Ours)
Figure 5: Learned invariances for the Mario and Iggy dataset. The blue arcs show the training data range, while the green ares show the learned trans- formation distributions for some examples.
5.2 CROPPING
We now move to more realistic images and to the most common and effective form of image augmentation: cropping. We first evaluate the performance of jointly training InstaAug and the classifier on Tiny-Images being within our computational budget. TI is a standard testbed for data augmentations that contains 100k images divided into 200 classes. Full experiment details are given in Appendix D.
(TinyIN, 64 64), it inherits the image complexity of ImageNet whilst
We benchmark InstaAug alongside several augmentation baselines, including Augerino, no augmen- tation, and random crops (random augmentation). The latter uniformly samples patch sizes and then randomly selects a patch inside the image. Since the effect of cropping crucially relies on scales of patches, we carefully tune this baseline by sweeping over all possible scale intervals between [0, 1] with stride of 0.1. We further compare to other prior works that have obtained competitive results on TinyIN (Ramé et al., 2021; Yun et al., 2019; Zhang et al., 2018).
In order to ablate the effects of input- dependency and location-related pa- rameterization on InstaAug, we ad- ditionally assess the performance of InstaAug (without LRP) which re- lies on the same uniform parameter- ization as Augerino rather than location-related parametrization (LRP, described in Figure 4); InstaAug (with- out imput) that uses the I.PP and gen- eral InstaAug setup. but shares the transformation distribution across all inputs rather than learning an input- specific augmentation; and InstaAug (class specific), which takes train- ing labels instead of images in- puts. Test-time augmentation using 50 transformation samples is deployed for all variants of InstaAug, along with the Augerino and random aug- mentation baselines. For InstaAug (class specific), this test-time augmen- tation is based on random cropping, due to the lack of class information being available at test-time and this approach performing better than simply omitting test- time augmentation. Following prior works, we choose the PreActResNet-118 architecture (He et al., 2b) with width = as the classifier for all methods.
Table I: InstaAug improves generalization on Tiny-
ImageNet by learning instance-specific cropping. 'Instance' and 'LRP' refer respectively to restance-specifie and 'location-related parameterization' Statistics are computed over 10 runs, except for MixMo, CutMix and Mixup, whose results from Ramé al. (2021). Other learnable augmen- tation methods are actually learning the size ranges of crop- ping. We Ieave their results out because forming this learning in the random cropping results through our hyperparameter tuning.
already per-
Method
Instance
LRP
Accuracy (%)
MixMo (Ramé et al., 2021)
64.80
CutMix (Yun al., 2019)
65.09
Mixup (Zhang et al., 2018)
63.74
No augmentation Random crop
X
X
55.06:1.110
X
X
64.49::0.12
Augerino (Benton et al., 2020)
X
X
55. 65.0220..9
InstaAug (without LRP)
/
X
55.39 0..19
InstaAug (without input)
X
V
63 0.
InstaAug (class specific) InstaAug
XIV
V
60.55+10.50
66.02+0.18
Table 1 shows the 1 accuracy for each method. In agreement with prior works, we find that random cropping increases top-1 accuracy by 9.4% over no augmentation, which is achieved where cropping scale = [0.1, Insta/ug outperforms random cropping and its own global version without input by 1.5% and 2.8% respectively, highlighting the effect of learning instance-specific augmentation.

---

Under review as a conference paper at ICLR 2023
Allowing only for class dependence actually pro- duces even worse performance than just ignor- ing the imput completely, presumably because of the inevitable resulting mismatch in the augmen- tations used in training and testing. Methods with mean-field field uniform parameterization (in- cluding Augerino and InstaAug without LRP) performed extremely poorly, noticeably worse than just random cropping. This is because they were found to become easily stuck at local min- ima with low cropping diversity, leading to sim- ilar performance as augmentation. Note that the potentially unexpectedly good performance of the random cropping baseline compared to the other global baselines stems from the careful hyperparameter sweep used tune its crop size, which proved more effective than these more direct training mechanisms. See Appendix E.3 for more discussion.
(A) InstaAug (wo input)
(B) InstaAug
(a)
(b)
(c)
(d)
(e)
Figure 6: InstaAug (B) learns more sensible crops compared to random and learned global (A) aug- mentations. Columns (a, d) show examples of sampled crops, with red edges indicating higher probability. Columns (b, e) show density maps for the crop centres, with brighter color meaning higher probability. Columns (c, f) give the propor- tion of crops (red) above particular size threshold, showing that InstaAug produces fewer large crops. a cropping scheme that focuses on the key aspect of the input image, while the baselines cannot.
Figure shows example crops and learned transformation distributions for InstaAug and global augmentation scheme (InstaAug with- out input). We see that InstaAug is able to learn
5.3 APPLYING INSTAAUG TO A FIXED CLASSIFIER
InstaAug can also be used to learn suitable augmentations for fixed pre-trained classifier. This can most notably be useful as a means to learn test-time augmentations. As the invariance module is itself only small network, can be done relatively cheaply, even when the dataset and downstream model are very large. We exploit this on the larger Imagenet dataset (224 x 224) (Deng et al., 2009), again focusing on cropping augmentations and utilizing the LRP parameterization from Section 3.3.
Training the invariance module in this setting is done in exactly the same way as elsewhere. using the training procedure of Section 3.2 with the normal training data. The only thing that changed that f is now fixed to a pre-trained classifer-sspecffcalll specifically, the ResNet-50 (He et al., 60) from Wightman (2019) (which did not use an invariance module during training)- -rather than being simultaneously learned. We are thus simply learning invariances, without affecting the training of
In Table we show the effect of using the learned invariance module for test- time augmentation, finding that it is able to noticeably improve accuracy, unlike the baseline test-time augmen- tations of random cropping, AutoAug- ment (Cubuk et al., 201 8018), and Fast Autoent (Lim al.. 2019). In order to evaluate the generalization performance of our learned augmen- tation module, we further apply the augmentation trained on ResNet-50 to two different models with zero fine- tuning: ResNet-18 18 (He et al., 2016a) and XCTT (Ali et al., 2021). We find that the learned augmentation trans-
Table 2: InstaAug boosts the test accuracy (%) with test-time augmentation on Imagenet. Invariance modules learned on ResNet-50 can also be directly applied to other models such as ResNet-18 and XCIT to improve generalization without fine-tunne fine-tuning By contrast,
that global augmentation schemes are actually detrimental to test-time augmentation.
Method
#Sample
RecNe150
ResNer18
XCTT
aug
80.43
69.73
86.34
Random trop
78.45:20.0
66.13-0.04
$2.05:10.01
AutoAug FastAutoAug InstaAug
4
77.84:0.00
59.50=0.01
81.40=0.00
4
77.87:10.00 0.06
61.43:20002
81.42-10.00
80.9210004
70.59=0.00
86.4312.00
Random trop
79. 79.60-0..0
67.87-0.01
$2.84000000
AutoAug FastAuro
10
79.20=20.00
63.96=0.00
82.43
79.28:10.0 0.001
65.6520.02
82. 22.45.0.00
InstaAug
81.1810.02
70.96-0.03
86.4720.02
fers very effectively to these different models, which implies that the local invariances InstaAug learns reflect the natural invariances of the underlying classification problem, rather than being specific to the model that was used to train the augmentation module.
7

---

Under review as a conference paper at ICLR 2023
5.4 COLOR JITTERING ON TEXTURES
Color jittering is another important type of data augmentation, which can help models generalize to different lighting conditions. We benchmark on the texture classification dataset RawFooT (Bianco et al., 2017). RawFooT includes 68 different samples of raw food and each sample has an image taken under each of 46 different lighting conditions (see Figure D.1 for some examples). We crop the original images to create the train set and test set. For each original image with a resolution of 800 x 800, we randomly sample 200 different 200 x 200 patches in the upper half as training images. The same procedure is taken on the lower half to produce test images, giving a train set and test set for each different lighting condition. To evaluate the generalization ability of each method to broader range of lighting conditions, we evenly mix test images from all lighting conditions to form general test set, while controlling the lighting conditions present during training.
We first train D45 (4500K, daylight) resembling natural light. Table 3 shows that InstaAug outperforms all baselines with and without test-time augmenta- tion. In this task, find that Augerino (with re- laxed symmetry restrictions on learned intervals) underperforms random augmentation because its parameters o are often stuck in neighbor- hood around their initial values. We believe this is due to the conservative nature of using global augmentations (cf. Figure 3), where even a small change in the parameters may largely increase the training loss, which prohibits wide-ranging augmentations.
single lighting condition
Table 3:
InstaAug achieves higher general accuracy than baseline methods when trained on D45 (Day- light, 4500K).
Method
Test aug?
Accuracy (%)
No aug
72.87
Random aug Augerino InstaAug
X
79.9920.13
X
78.97
X
81.11
Random aug Augerino InstaAug
80.55
79.34
81.35 10.19
We also compare in-distribution and out-of- distribution generalization by splitting the 46 test sets into two groups, according to the sim-
0.68
0.66
0.64
ilarity of their lighting conditions to D45-sea see Appendix D.2 for the details on the splitting method. In Figure we can see that above
Out-of-distriba didiiittibution
0.62
0.60
0.58
certain in-distribution performance, there exists trade-off for random augmentation between in in-distribution accuracy and out-of out-of-distribution generalization, controlled through the hyperpa- rameter settings. InstaAug, meanwhile, delivers higher -uttt--distribution performance than any of the hyperparameter configurations, while also simultancously giving better in-distribution ac- curacy to the vast majority of them well.
0.56
InstAug test aug
InstAug w/o test aug
0.54
Random aug
No aug
0.82
0.88 0.92 0.96 In-distribution accuracy
0.86
Figure 7: In-distribution and -fitt--disttiiutiin test accuracy for models trained on RawFooT D45, The round dots are random augmentation with different hyperparameter settings. The colors of dots change from yellow to red hue jittering increases; more saturated dots indicate higher satu- ration jittering; larger dots mean higher brightness jittering. Each thick line connects dots with the same hue and brightness jitter and thin lines link dots with the same hue and saturation jitter.
We can further vary the difficulty of the classi- fication task by using different numbers of light- ing conditions in the training data. In Table 4, randomly select ditions to use as the training set for each base- line. As expected, the accuracy increases with the number of lighting conditions for all meth- ods. However, the effect of random augmentation saturates: it performs similarly to no augmentation with lighting conditions. By contrast, InstaAug always provides improvements. In Appendix D, we show that these gains come at very little computational overhead at both train and test time.
number of lighting con-
INSTAAUG FOR CONTRASTIVE LEARNING
Contrastive learning aims to learn features that are approximately invariant to certain augmentations. Typical contrastive learning methods, such as SimCLR (Chen et al., 2020; Ermolov et al., 2021), first sample two independent transformations, TI, T2 p(T), and apply them an input image x,
8

---

Under review as a conference paper at ICLR 2023
Table 4: InstaAug significantly outperforms baseline methods in general test accuracy (%) on different difficulty levels. For each difficulty level, we randomly sample lighting conditions used for training and repeat each experiment 10 times. Test-time augmentation included for random and Insta/ug.
Method / #l.ighting conditions for training
2
No aug
68.5
78.1
84.8
87.8
Random augmentation
72.7
80.8+1.3
85.9
87.3
InstaAug
76.0
83.6
88.2
89.6
generating two views and X2. They then feed the transformed images to neural encoder f. which is trained to maximize the similarity between f(x1) and f(x2), measured with a contrastive loss.
As the choice of augmentations directly influences the learned invariance of the encoder, it is crucial ingredient of contrastive learning (Bachman et al., 2019; Chen et al., 2020; Tian et al., 2020). However, existing schemes use global augmentations which often introduce unrealistic assumptions. For example, if there are multiple entities an image, such as grass and cattle in Figure Ic, random cropping will pull features for different entities closer to each other. Consequently, we propose InstaAug as a more flexible instance-specific augmentation method for contrastive learning.
Applying InstaAug contrastive learning is similar to the supervised case shown in Sec- tion 3. The main difference is, given an in- put x, we sample two T independently from the input-specific distribution p(T; (x)), before they are applied to x. The training objective correspondingly changed to minimizing the contrastive loss while keeping the diversity in reasonable range.
(a)
(b)
Figure 8: Some examples (bird houses) of learned
cropping in contrastive learning.
We again consider TinyIN and evaluate three methods: InstaAug, InstaAug (without input), and random crop. We exclude methods with uniform parameterization, because of their poor performance in the simpler supervised setting. All experiments are based on the SimCLR framework and use the Pre 18 network as the encoder. We train each model with a batch size 512 for 500 epochs. We then train a linear classifier to evaluate feature quality. We use test-time inggeenttioon-wiee 10 sampled crops as has been shown to improve performance (Foster et al., 2021).
From Table 5, we that InstaAug outperforms the random and global augmentation schemes as well as Un-Mix (Shen al., 2022), which is recent variant of Mp methods on contrastive learning. We observe from the examples shown in Figure 8 that InstaAug focuses on the salient features containing important information. We also notice that the sizes of learned patches are correlated to the sizes of the main objects in images. Thus, InstaAug is able to learn sensi- ble instance-specife augmentations in a fully unsupervised setting.
Table 5: Representations learned by InstaAug per- form better in the downstream linear classification task than baselines. "Results of Un-Mix are di- rectly taken from Shen al. (2022), which has the same network structure (ResNet-18), training algorithms (SimCLR) and linear classifier as ours.
Method
Accuracy (%)
Un-Mix (Shen et al., 2022)
49.58*
Random crop InstaAug (without input) InstaAug
51.63 6..30
54.20+
55.0540021
DISCUSSION
In this paper we introduced InstaAug, method for learning instance-speciffc data augmentations that capture local invariances of the underlying data generating process. This is achieved by training an augmentation module that parametrizes whose samples are used to augment the training data on the fly and/or for test-time augmentation. The main benefits of InstaAug stem from its applicability to wide range of settings, its ease of use, and crucially its capacity to learn meaningful augmentations that in turn improve performance. Empirically, we demonstrated these benefits for both classification and contrastive learning problems, considering several classes of transformations-tune rotatiinn,, color jittering, and cropping.
input-dependent distribution
transformations,

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. Advances in Neural Information Processing Systems, 2020.
Simone Bianco, Claudio Cusano, Paolo Napoletano, and Raimondo Schettini. Improving cnn-based texture classification by color balancing. Journal of Imaging, 3(3):33, 2017.
Anadi Chaman and Ivan Dokmanic. Truly shift-invariant convolutional neural networks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3773-3783.
2021.
Shuxiao Chen, Edgar Dobriban, and Jane H. Lee. Invariance reduces variance: Understanding data augmentation in deep learning and beyond. CoRR, abs/1907.10005, 2019. URL http:
/ wxx...vv.om www.vv.org/bbs...ooo /1907 190.. 10905.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xiangning Chen, Cihang Xie, Mingxing Tan, Li Zhang, Cho-Jui Hsieh, and Boqing Gong. Robust and accurate object detection via adversarial learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16622-16631, 2021.
Tsz-Him Cheung and Dit- Yeung. Adaaug: Learning class- and instance-adaptive data aug- mentation policies. In International Conference on Learning Representations, 2022. URL
net. forum? (emummid==rie
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning. PMLR, 2016.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. Conference on Computer Vision and Pattern Recogni- fion, 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning, pp. 3015- 3024. PMLR, 2021.
Adam Foster, Rattana Pukdee, and Tom Rainforth. Improving transformation invariance in contrastive representation learning. In International Conference Learning Representations, 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. In European Conference on Computer Vision, pp. 1-16. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 20.
Daniel Ho, Eric Liang, Xi Chen, lon Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731-2741. PMLR, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Olav Kallenberg and Olav Kallenberg. Foundations of modern probability, volume 2. Springer, 1997.
Ildoo Kim, Younghoon Kim, and Sungwoong Kim. Learning loss for test-time augmentation. Advances in Neural Information Processing Systems, 35:4163-41744 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Yonggang Li, Guosheng Hu, Yongtao Wang. Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Dada: differentiable automatic data augmentation. arXiv preprint arXiv:2003.0370 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment.
Advances in Neural Information Processing Systems, 2019.
Canjie Luo, Yuanzhi Zhu, Lianwen Jin, and Yongpan Wang. Learn to augment: Joint data augmenta- tion and network optimization for text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33746-13755, 2020.
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint vrXiv:2005.00177 2020.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Agnieszka Mikotajczyk and Michat Grochowski. Data augmentation for improving deep learning in image classification problem. In 2018 international interdisciplimary PhD workshop (IIPhDW). IEEE, 2018.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.0462) 2017.
Alexandre Ramé, Rémy Sun, and Matthieu Cord. Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 823-833, 2021.
Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1214-1223, 2021.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2216-2224, 2022.
Connor Shorten and Taghi M Khoshgoftuar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
11

---

Under review as a conference paper at ICLR 2023
Ivan Sosnovik, Michat Szmaja, and Amold Smeulders. Scale-equivariant steerable networks. In International Conference on Learning Representations, 2019.
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks:
Learning views for unsupervised representation learning. In International Conference Learning Representations, 2020.
Zhiqiang Tang, Xi Peng, Tingfeng Li, Yizhe Zhu, and Dimitris Metaxas. Adatransform: Adaptive data transformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-90880 3006, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-68399 2020.
Ross Wightman.
Pytorch image models. pytorch- wwwiiimaaaeeeeeeeee moseeemmoels.... 2019.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256,, 1992.
Daniel Worrall and Max Welling. Deep scale-spaces, Equivariance over scale. Advances in Neural Information Processing Systems, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Jin Xu, Hyunjik Kim, Thomas Rainforth, and Yee Teh. Group equivariant subsampling. Advances in Neural Information Processing Systems, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning. PMLR, 2019.
Yu Zheng, Zhi Zhang, Shen Yan, and Mi Zhang. Deep autoaugment. In International Confer- ence on Learning Representations, 2022. URL httpt:htt/////ttc /openreview/ net wtt.mmrrrmmmmmm id= St-53J92ARE
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933, 2020.
Fengwei Zhou, Jiawei Li, Chuanlong Xie, Fei Chen, Langing Hong, Rui Sun, and Zhenguo Li. Metaaugment: Sample-aware data augmentation policy learning. Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021.
Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX A THEORETICAL ANALYSIS OF GENERALIZATION ERROR
We now provide a decomposition of the generalization error-ie. the difference between the true risk and the training when using & during training of the downstream classifier f. Here can view the objective of augmentation as adjusting the training objective to encourage the learned model have low true risk. As such, the generalization error provides a measure of the effectiveness of the augmentation for the training of f: by analysing the behaviour of the generalization error as a function of the augmentation module, we can derive a characterization of the desirable properties of the latter.
To start our analysis, we first define the true risk of the downstream model, f. as
R(f) == (XC(f(X),
(A.1)
where (X ~ Pirue (X, Y) are drawn from the true data generating distribution. In practice, one might also perform test-time augmentation, implying different predictive function and thus different true risk, but for the purposes of our analysis, we will assume that this is not done, as this allows us to focus on the impact the invariance module has on f during training.
One the other hand, the implied training risk (i.e. our objective for training ƒ) when using an invariance module is the augmented empirical risk
i( o) =- E(C(S(((Z:)) 9:)]
(A.2)
where i ~ Uniform (1... ...... N) is uniformly sampled index for point in the original training dataset (In, Un 1 and 7/i ~ p(T; ((x)) is the sampled transformation. Note that the expectation in Equation (A.2) is only over i and with the datapoints themselves not considered random variables for our purposes, because we are only provided with single fixed training dataset.
The generalization error can now be defined as R( Φ)- R(J).. At a high level, we are interested in finding a e that ensures this has a low magnitude. More precisely, we want 6 to ensure that the minimizer of the training risk, f* == arg min, y R( Φ). gives as low a true risk, R(f*), as possible. Therefore, we want to keep the generalization error magnitude small across different (relative to the corresponding variations in R(,, Φ) itself), so that the optima of the training and true risks are as similar as possible. In other words, we want é that ensures f(( R(f) is small for all f, especially those close to f*. If we do hypothetically drive the generalization error to zero for all f. we will have a mechanism for directly training to the true risk using finite original training dataset.
To aid with decomposing the generalization error, is convenient to further define the following random variables through their conditional distributions:
Y i ~ Ptrue (Y = VIx = x;)) with Y $
(A.3)
Y V Ptrue (Y = 81X = ((x)))
(A.4)
We now write down our decomposition follows:
R(f R(f) =
(A)
(A.5)
+ EIC(F(((2)), Y) ((f(X),)) E(C(f(T(x))) v.) ((((T(2))).)) P). (C)
(B)
From this, that if the magnitude of (A), (B), and (C) all small, then our generalization error magnitude will be small as well. Morcover, we can construct a é such that these terms are small for all f. then we can ensure effective generalization performance. We will now look at each term individually.
(A) provides a precise characterisation of how well our transformation preserves the label distribution; it is the difference between the expected loss under the true label distribution of the untransformed inputs and the expected loss under the true label distribution of the transformed inputs, making predictions using the transformed inputs in both cases. In particular, by noting that we have
(A) E E [(f(TT(Z))) - C((
(A.6)
13

---

Under review as a conference paper at ICLR 2023
where (x(() is deterministic given 7 and i, we have that Y V,, yi, Vi, T is a sufficient (but not necessary) condition to ensure (A) 0 for all f. That is, it is zero for all ff if the conditional distribution on the labels is the same for both the original and transformed inputs for all possible pairs (i, r), L.e. all possible original inputs and sampled transformations. One simple way ensure this is to have 7 always be equal to the identity mapping, so this term prefers limited transformations.
By contrast, if the transformation destroys information about the label, 8 and Viē, will now differ, such that, in general. (A) * O and, moreover, will vary with f. Here we typically expect that (A) 2 0, as we are making predictions using the transformed inputs, so the expected loss under the true label distribution for the transformed inputs will tend be less than that when labels are generated using the untransformed input. To keep the magnitude of (A) low, we need to ensure that transformations maintain the conditional label distribution as well as possible, i.e. that transformations preserve all input information that is salient for predicting labels.
Conveniently, minimizing R(f, Φ) with respect to o, done by the InstaAug training setup of Sec- tion 3.2, will naturally try to reduce (A). Given we expect the term to typically be positive, this provides an explanation for why InstaAug can be effective without any separate consideration in the objective for the need for transformations to maintain the class label distribution.
(B) represents how well our transformation captures the true input distribution. Here can utilize the fact that, by the definition of Y,
E [U(((())) (())) =
B (E(((X)) =x] =)=:(()))
(A.7)
to write it as
B B|r(((a;))) E[r(X))
(A.8)
where r: X'' - I maps inputs to their true expected loss. We thus see that F(x;) (but not necessary) condition to ensure that (B) = 0 for all f. That is (B) is always 0 if the process of choosing one of the training inputs random followed by applying sampled transformation to that input produces samples distributed exactly according to the true input distribution. Unlike for (A), there is no simple scenario in which we can ensure this is true, with the use of the identity transformation now likely to give significant discrepancies by failing to provide sufficient coverage of the input space: though the ny may originally have been sampled from Ptrue (X), there only a finite set of them, such that repeated sampling from this finite set represents a substantially different distribution to Ptrue (X). In fact, (B) nicely encapsulates the desire to perform augmentation in the first place, by showing how it can be used to increase the coverage of the input space.
d X is sufficient
How to best manage Term (B) will vary depending on the type of model used and the form of our transformations. In some situations, it may be that no matter how diverse our transformations are within the class of those allowable, r(x;) will still only cover a subset of the support of X. Here the most important factor for keeping (B) small will be to maximize the diversity of the transformations, e.g. by maximizing their entropy, to ensure the best possible coverage of the true input space. In other cases, it might also be possible "over-diversify" the imputs, such that 7(x;) can become more diffuse than X for some choices of Φ. potentially causing training to lack focus on the particular test-time input distribution we care about. Here we may need to ensure that the entropy of the transformation does not become so large as to cause such over-diversificatio, creating a more complex trade-off with the need to ensure sufficient coverage. These two scenarios respectively motivate the lower and upper bounds on the transformation distribution entropy used when training the augmentation module.
For augmentation of high digensiiieni data, the former, coverage- limited, scenario is expected to be significantly more likely, as our original training data will generally provide quite poor coverage
'Note that y _ y alone is not generally sufficient, matching in marginal distribution does not ensure that the joint distributions with and r also match, in turn yielding different expectations.
'Note, though, that this is not formally guaranteed, even for the cross entropy loss and an f that exactly captures the true distribution. This is because, while Gibbs' inequality ensures the optimal g given p for cross-validation expected loss E - log ((Y) ) q = p. in general, the optimal p given g is not p = q.
'Note here that the bounds in Equation (2b) are on are on the entropy on the parameters of r. rather than T(a;) itself. This is because it is difficult to directly control the latter during the training, with the former providing more practical proxy that is expected to generally he representative.
14

---

Under review as a conference paper at ICLR 2023
of the true input distribution, while our transformations will not generally be sufficiently powerful to produce unrepresentative inputs. Moreover, when working with large deep learning models, prediction in one region of the input space is rarely harmed by the addition of data in another input region. Thus, for the typical scenarios, we expect InstaAug to be deployed in, increasing the entropy of the transformations will directly relate reducing the magnitude of (B). Note here that it will typically be the case that (B) < 0 provided that the transformations maintain the label distribution, as the accuracy of the downstream model will typically be higher for the transformations of the original training data that for the test data.
Term (C) is the error from the fact that we only have one sample of the label for each original training input, rather than the full label distribution. As y IL T, we have limited ability to reduce through controlling Φ; it essentially represents the irreducible noise in f( 6) from only having a finite number of true labels. Note that it is not related to the model's ability to generalize unseen inputs, as it is based on variability in other possible labels we might have seen for our training inputs themselves; if Y X is actually deterministic, it is exactly zero. As such, it is of limited interest for our analysis, while it will thankfully generally be much smaller than the other terms for practical problems unless we have both very small dataset and very noisy true label distribution.
Putting everything together, we see that (A) and (B) respectively encapsulate the competing needs of the invariance module to maintain the conditional label distribution (i.e. preserve the label information) and maximize coverage of the input space. We have also seen that the former is typically naturally taken care of by minimizing R(f, Φ) with respect to é. motivating the objective used by InstaAug in Equation (2a), but the latter requires separate consideration, which we deal with through our constraints on the entropy in Equation (2b).
APPENDIX B DETAILS OF AUGERINO
As method to learn invariance, Augerino (Benton et al., 2020) is quite different from the previous approaches, which usually require an extra validation set. The basic idea behind Augerino is to use a few parameters (0) to control the transformation distribution on input images and learn these parameters with the training loss of the classifier. Specifically, it minimizes the loss
LA (x; E E(C(x; v)] + A (0),))
(B.1a)
where L(x;g) y) is the cross-entropy loss and R(0) is regularization function on the volume of the support of the distribution weighted by the hyper-parameter X.
Comparison with InstaAug. InstaAug shares with Augerino the ideas of tuning augmentation parameters by the classifier loss and using test time augmentation to boost performance, but they are different in the following aspects. The most significant difference is that InstaAug is instance-specific, while Augerino learns global augmentations. Besides, Augerino uses a single scalar to parameterize a symmetric uniform distribution (4[-6,6) over each type of transformations, which lacks the flexibility to model more complex augmentations, such as cropping.
In addition, Augerino uses fixed weight À to balance the training loss and augmentation diversity. However, we find that, in more complicated settings, this is quite impractical. Specifically, we need different À in different stages of training. If we use large À from the start of training, the diversity will quickly diverge to maximum, because the classifier is very weak and the loss is consequently dominated by the diversity term. This will block the training of the classifier because transformed samples from different classes are quite mixed with each other. Otherwise, if we choose a small X, the diversity will converge zero after few epochs, yielding similar results the vanilla model without augmentation. In neither of the case can we learn a useful augmentation. Consequently, InstaAug directly constrains the diversity keep stable during training.
APPENDIX C METHOD DETAILS
C.1 REGRESSION AND SULE-IUPERVISED LEARNING
In Section 3, we use classification as an example to introduce InstaAug. However, InstaAug can be easily applied to other tasks including regression and self-supervised learning. For regression,
15

---

Under review as a conference paper at ICLR 2023
the classifier (see Figure 2) is replaced by regressor and the loss function C in Equation (2a) is changed accordingly to absolute or square error. For supe-suude contrastive learning, we replace the classifier and cross-entropy loss with the feature extractor and contrastive loss (such as SimCLR loss (Chen et al., 2020)), respectively. In addition, the sampler samples rather than 1 transformations generate multiple views for input
C.2 IMPLEMENTATION OF LOCATION RELATTED PARAMETERIZATION
As an example, we show how to implement location-related parameterization with a basic CNN structure in the following algorithm..
Algorithm 1: Location related parameterization
Input: Image channel numbers M,, and layer number layer Output: Probability of patches p Fo x;
for ( i; n_layer; i i 1)
F Conv2d(F J- kernl=2,, stride=1, output chppptt_hhnn)))M):
// CNN Operation
// CNN Operation F; = Conv2d(F,, kernel=1, stride=1, (rtput_channel=l) To single channel // Logit vector at each level Logit vector at all levels // Probability after normalization
Pooling(F,, kemel=2);
logit: Flattee(F;))
logits Concct([logiti));
p (ormalize((xpp(looiss)))
C.3 OTHER PARAMETRIZATION METHODS
Besides the uniform and location-related parameterization, also tried VAE-like methods pa- rameterize augmentations, such as cropping. The main idea is to have a Gaussian latent variable and neural decoder map the latent Gaussian distributions formation parameters (in this case, the centers and sizes of crops). However, similar to the uniform parameterization, we find the VAE-like parameterization unstable and easily stuck at local minima.
continuous distribution on trans-
APPENDIX D EXPERIMENTAL DETAILS
D.1 CROPPING
Supervised training Based on the Mixmo codebase" (Ramé et al., 2021), we use stochastic gradient descent (SGD) optimizer train baselines and InstaAug. For the classifier, the initial learning rate is set to 0.2 (with momentum 0.9 and weight decay 1 4). A scheduler is used to decrease the learning rate by a factor of 0.9 once validation accuracy doesn't increase for 10 epochs. The learning rate of the augmentation module ò is fixed at le - 5. Batch size is set to 100 and we pre-train InstaAug for 10 epochs without augmentation. We train the model until convergence and the maximum epoch is set to 150.
Contrastive training We directly apply InstaAug on the codebase from Ermolov al. (2021). Because of the characteristics of contrastive learning, we set the batch size to 512. Same as the supervised case, use SGD optimizer train the augmentation module σ. Differently,
Adam optimizer (Kingma & Ba, 2015) (with learning rate le 3 and weight decay le - 6) to train the base model. We train each model for 500 epochs and decrease the learning rate by factor of 0.8 at step 450 and 475.
D.2 COLOR JITTERING ON TEXTURES
Training. We use PreActResNet-I1 18 (width = 1) on texture recognition task on RawFooT and train it with SGD optimizer. The learning rate is 0.02 (with momentum 0.9 and weight decay le - 4)
hhttps: httpp:://github... com/ awwm///eeaaa////em/mmmmmmmmm mi xmo- pytorchhhhhhh. .igh.iis under Apache License v2.0. 'https: //g thub. com httmm/ttt///eeeeeeeeeeeee ssel supervi sed. gi t, under Apache License v2.0.
16

---

Under review as a conference paper at ICLR 2023
DA5
D50
D55
D60
D65 (1=1)
(1-0.75) D27 (A=90) D65+D95
D65-D227
DS5+D27
Red
Green
Figure D.I: Examples of RawFooT data. Each row contains images in the same class (corn, candies, floor, red cabbage) under different lighting conditions. The left and right half of lighting conditions are in the easy and hard group, respectively.
for the classifier and le 5 for the augmentation module ø. We train each model for 50 epochs and learning rate schedulers are not necessary in this task.
Random augmentation baseline. We sweep over the variation range on each channel to find the best hyperparameters for the random augmentation baseline. For hue (h-jittering), sweep between [0, 0.5] with stride 0.1, and for saturation (s-jittering) as well as brightness value (v-jittering), we sweep between (0, 1.0) with stride 0.2, which yields 216 different settings in total. The best accuracy shown in Table is achieved where h,s,v= 0.0, 0.2, 0.8.
In-distribution outtof-distriuution generaliza- tion. To further investigate the effect of each aug- mentation method, we additionally split the 46 test
Table D.1: Splitting of Lighting conditions.
Group
Lighting id
sets into two equally-sized groups. The first group contains lighting conditions similar to D45, such daylight with different temperatures, for which the vanilla model without augmentation trained on D45 has high test accuracy. The second group contains lighting conditions that are dramatically different from D45, for example, pure red light, which are more difficult for the vanilla method. Then the average accuracy on the first group can be regarded as measure of in-distribution generalization, while the accuracy on the second group reflects out-of-distribution generalization.
Easy (1) 1-4,10,14-31
Hard (2) 5-9, 11-13,131120 32-46
D.3 TIME COMPLEXITY
We notice that InstaAug color jittering has similar training speed (0.37s/iter) as random aug- mentation (0.40s/iter) on single Nvidia 1080Ti GPU, though it takes more epochs (about 40) compared with random augmentation, which usually converges after 25 epochs. We also find the speed for evaluation is very fast even with test time augmentation (sample number =10), which is about 0.004s/sample. However, the training speed of InstaAug on cropping (0.25s/iter) is slower than random augmentation (0. (5.15s/) due to optimization issues on the more complex parameterization method. Training InstaAug alone takes a similar amount of time for each epoch compared with joint training. but it requires fewer epochs (less than 30) to converge and we can cache the outputs of the classifier for faster training. The evaluation speed is 0.01 Is/sample when sample number is set to 50 for test-time augmentation.
APPENDIX E ADDITIONAL RESULTS AND DISCUSSION
E. RAWFOOT
Figure E. shows some examples of learned color jittering. Though it's not easy to fully understand them, we can still find some patterns. For example, InstaAug tends to increase the brightness of
17

---

Under review as a conference paper at ICLR 2023
(I) Input
(II) InstaAug
(II) Augerino
1
1
h
s v
S v
1
0 S 1
s
v
1
0
h
s
h
v
1
h
SV
SV
(a)
(b)
(c)
(d)
(c)
(f)
gg)
Figure E.I: Examples of learned color jittering. (a) Original image; (b. f) Average hue (H) of
original image (blue dot) and learned hue jittering (red are) for InstaAug and Augerino; (c.g) learned saturation (S) and brightness value (V) of original image (blue dot) and learned hue jittering (red line
segment) for InstaAug and Augerino; (d,e) examples of images transformed by InstaAug.
darker images (row I and 3) and decrease the brightness of brighter images (row 4). Also InstaAug more likely to change saturation compared with hue and brightness, which is consistent with the common belief that saturation contains less information than hue and brightness.
InstaAug's behavior is quite different on different samples. It even decides not to augment the H and V channels of the image the second row, In comparison, Augerino adds or multiplies noise each channel with the same distribution across all samples, which is harmful many cases. For example, the input image in the last row is already very bright. but Augerino allows further increasing its brightness. Then brightness values of many pixels will capped 1.0, which leads to loss of information.
Table E. Model performance with different choice of Hmin and Hmax on supervised cropping.
H
H
Accuracy (%)
0.0
0.5
52.12
0.5
1.0
61.28
1.0
1.5
62.91
1.5
2.0
64.39
2.0
2.5
65.04
E.2 HYPERPARAMETER ABLATION
2.5
3.0
65.05
3.0
3.5
66.03
The two hyperparameters of InstaAug are Hain and Hmax
which reflect human preference on augmentation diversity. To
3.5
4.5
65.60
investigate how Hmin and Hmax influence model performance and provide guide on how choose them, we perform an ablation study for the experiment of Section 5.2, wherein we
4.0
4.5
64.35
4.5
5.0
64.17
0.0
1.0
51.78
sweep over possible intervals of length 0.5 and 1.0. From Table E. we find that the best accuracy is achieved when
1.0
2.0
63.96
2.0
3.0
65.25
[mmin Haaax] is set to [3, 3.5], while any sub-interval of [2, 4] produces significantly better results compared with random augmentation.
3.0
4.0
65.78
4.0
5.0
64.23
E.3 WHY IS THE RANDOM AUGMENTATION BASELINE SO STRONG?
is perhaps initially surprising that the Random Augmentation baseline in 5.2 is so strong compared the other global augmentation schemes. In short, this occurs because the extensive hyperparameter
18

---

Under review as a conference paper at ICLR 2023
sweep used for it turms out to be more effective tuning mechanism than directly training global parameters simultaneously to the model. To be more precise, for any global cropping scheme (which includes random crop, Augerino, and InstaAug without input), there is little to be gained from using a non-uniform distribution on the position of the crops. As such, the only thing that can be usefully learned is the distribution the size of the crops themselves. For the random crop baseline, do an exhaustive sweep to establish the best distribution on crop sizes, meaning that this baseline represents a near-optimal global cropping augmentation. By comparison, InstaAug (without input) must still learn the optimal cropping size distribution during training, and the results suggest that it does not always manage do this perfectly, tending to prefer under-diverse transformations. This is perhaps not surprising. as it does not have access to a validation set, unlike the hyperparameter sweep implicitly being deployed for the random crop baseline. The problem seen even more starkly for Augerino, where the lack of LRP causes training to become stuck in highly sub-optimal local optima that yield very little transformation diversity.
19


Reviews:
[{"review_id": "vDX0RQ0Q0O", "reviewer": "Reviewer_uSWY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an algorithm to learn instance-specific augmentation for images, which substantially improves computer vision models over a wide range of scenarios.", "summary_of_the_review": "The paper proposed a simple and effective instance-specific algorithm that generally improves the performance of computer vision models. However, there are still some limitations, as suggested in Strength And Weaknesses.", "strength_and_weaknesses": "Strength:\n1. The algorithm is simple and effective.\n2. It applies to a wide range of scenarios, including supervised learning and contrastive learning.\n\nWeakness:\n1. The two principles suggested at the beginning of Section 2, while intuitively correct, are not theoretically justified in the paper. In addition to empirical successes, is there a reason why these two principles lead to better representation? The authors may need to explain them from an information-theoretical perspective (since the loss function includes entropy) or from a learning-theoretical perspective.\n2. The proposed method introduces too many hyperparameters. For example, the algorithm introduces lower/upper limits of the entropy and the regularizer in the loss function. Furthermore, the algorithm requires a set of hyperparameters for each type of augmentation. Therefore, it is not easy to adopt the algorithm to a new scenario (without a tedious hyperparameters search).\n3. The current method only works for vision models since the augmentation types are mostly vision-specific. However, the proposed method should also work in other domains, particularly natural language processing. It will make the technique more general/impactful by demonstrating its capabilities other than computer vision. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "t3-U83kxMk", "reviewer": "Reviewer_9vFe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Data augmentations with common input transformations is a commonplace in ML training, and is known to improve performance. This work argues that example-agnostic common transformations are too restrictive and that example specific augmentations can allow much wider transformation. The proposed methods of learning instance specific augmentations is shown to improve performance when training on Tiny-Imagenet, test-time augmentations and for contrastive learning. ", "summary_of_the_review": "I am not yet fully convinced that the paper is foolproof. The motivation and problem setup is clear, but their method and technical details are not. I request the authors to address my concerns listed above and I will make a more decisive evaluation after looking at authors’ responses and based on other reviews. \n\n--------\n**Edit after response.**    \nI thank the authors for patiently addressing all the question. Most of my question/confusion have now been resolved. I thank the authors again for adding the comparison with per-label augmentations. After reading other reviews and author comments, I have the following outstanding concerns. \n\n* Reviewer eZc6 pointed to several related work. I agree with eZc6 that the paper should have elaborated more on their existence and comparison. I find the reasons the authors provide for not comparing with MetaAug/AdaAug unconvincing for the following reasons: (a) \"MetaAug/AdaAug do not search over parameterisation of augmentations\", which could perhaps be fixed simply by extending the search space to include augmentations with different parameterisations. (b) \"they require validation dataset\", given that train loss is a good proxy for InstaAug, it could also be a good proxy for MetaAug/AdaAug. Why not simply replace their need for validation dataset with train dataset? I agree that it is not trivial to use them for comparisons on test-time augmentations though. I did not go over the related work before and got the impression (after reading the draft) that instance specific augmentations is novel when it is not. The paper thus may need to be restructured to carefully communicate their contributions. \n* Reviewer 4op9 raised an important point about evaluation on the running examples of digit rotations. The authors' response to this question is concerning. If instance-specific augmentations are not useful in this simple example, I do not understand how it is helping for other (complex/real-world) datasets. In that regard, the paper should present a more faithful explanation/intuition of their method. \n\nExisting work on learning effective augmentation stratgies (including instance-specific augmentations) all use policy search based methods and therefore InstaAug is more desirable since it learns the augmentation policy end-to-end. LRP and parameterising of augmentations are all interesting contributions, which make InstaAug readily available for examples beyond training data such as in test-time augmentations. Although I am excited and recognise these merits of InstaAug, I am discourcaged by problems in exposition and comparison. The paper is an interesting contribution (especially given their performance gains) at some point, but the current draft needs further work. To make the acceptance decision more easier, I am now lowering my earlier score. ", "strength_and_weaknesses": "Strengths.\n1. Need for instance specific augmentation is well-motivated. I particularly like Section 5.1 and Figure 5. \n2. Significant empirical gains in Table 1 and 2. \n\n\nWeakness/questions.\n1. The proposed method is simple, which is nice. However, in (2b), I do not see what pushes the per-instance transformation entropy to be more than H_min. In other words, what part of the loss encourages diversity in the distribution of transformations?\n2. Comparison with per-label (instead of per-instance) augmentations should also be presented. \n3. In Table 1, InstaAug (without input) but with LRP is worse than global random crop, which is unexpected. I expected them to be at least comparable, why do we see a difference? \n4. Section 5.3 does not describe in detail how \\phi is re-learned and on what data.  Technical details on how \\phi is parameterized is also missing. Section 3.3 is also not very clear. \n5. With regard to test-time augmentations of Section 5.3, the improvement in performance with InstaAug is impressive. However, InstaAug can only learn augmentations that the trained model is already robust toward, I do not see the source of improvement in Table 2. Could you explain/comment? \n6. (Minor) Figure 7 contains some legends but not their plots making it look incomplete.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nm6lRSgbgHn", "reviewer": "Reviewer_4op9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an additional module that performs data-specific augmentation. Specifically, location-related parametrization taking account of spatial location has been proposed. Experimental results show the effectiveness of the proposed method on image datasets in supervised and self-supervised representation learning tasks.", "summary_of_the_review": "I am mostly happy with this paper, except for the unclear comparison with prior works. Please answer my concerns above.", "strength_and_weaknesses": "Strengths\n\n+ The task of capturing local invariances by learning data-specific augmentation is interesting.\n\n+ The proposed method is simple but looks effective; by looking at the description, simply parameterizing data augmentations and optimizing its entropy is enough.\n\nWeaknesses\n\n- It seems prior works are not properly discussed. Based on the description in Section 5.2, the uniform parameterization is the same as Augerino, which is proposed by [Benton et al.]. Also, I believe test-time augmentation is also used in [Benton et al.]. Indeed, the performance of Augerino and the proposed method without LRP looks almost the same by looking at Table 1. Please do side-by-side comparison with closely related works like [Benton et al.].\n\n- The first intuitive example in Figure 1(a) is digit recognition, but it is not appeared in the experimental results. I wonder if the range of rotation for 6 and 9 is indeed trained to be narrow, such that they are distinguishable.\n\n- Similar to supervised learning, authors could also compare the proposed method with mixup-variants, which also have shown to be effective in many prior works, such as [Lee et al.] and [Shen et al.]. Also, I believe mixup-variants become a default choice to boost the performance of both supervised and self-supervised learning, so I wonder if the proposed method can be combined with them.\n\n[Lee et al.] i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning. In ICLR, 2021.\n\n[Shen et al.] Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning. In AAAI, 2022.\n\n**post-rebuttal**\n\nThe authors addressed my concerns mostly well, but an exception is the MNIST example in the motivative figure: I think their demonstration is indeed not proper and should be either completely removed or thoroughly discussed with experimental results to avoid any confusion.\n\nOther reviewers found some critical points I was not initially aware of. For example, regarding AdaAug/MetaAug, if they indeed share the same spirit with the proposed method, then I think it is worth making some comparison (maybe by adapting the proposed method to the prior works' setting).\n\nAll in all, I do not change my initial rating, as I think other reviewers' concerns seem to be valid.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jft7zIuYlX", "reviewer": "Reviewer_eZc6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper elucidates the importance of instance-dependent transformation in augmenting data and proposes InstaAug that learns the input-dependent transformation which could preserve the information of original input while maintaining sufficient transformation diversity. The invariance module $\\phi$ is jointly trained with the classifier $f$, which can predict the parameters of transformation distribution based on a given input. The authors also propose a location-related parameterization technique to parameterize cropping augmentations.", "summary_of_the_review": "Overall, I think this work needs further refinement for acceptance due to the concerns mentioned above (please see Strength And Weakness section). I understand that it is no longer possible for the authors to revise their draft, but I would like to see the authors' response within the remaining period.", "strength_and_weaknesses": "(+) The paper is well-written and the proposed method is simple and straightforward.\n\n(+) The location-related parameterization (LRP) for cropping seems quite novel to me.\n\n(-) It seems the flow of the draft needs to be substantially modified. Most parts of the draft describe the necessity of instance-wise augmentation in comparison to global augmentations. As the idea of instance-wise (or class-wise) augmentation is not new [1,2], the differences and empirical comparisons with these works should be more intensively addressed in the paper. Only brief and limited descriptions can be \nfound in the related works section. The authors claim that these prior works focus on choosing ‘which type of transformation to apply’ while InstaAug keeps the type of transformation fixed and learns instance-specific parameters. However, [1] searchs not only the type of transformation but also the magnitude, and in this regard, the advantage of ‘keeping the transformation fixed’ is unclear given that \na set of hyperparameters have to be found for each transformation.\n\n(-) Some of the motivations for the proposed approach are not convincing. The authors explain the limitations of global augmentation approaches using Figure 1. In Figure 1 (b), it seems the global augmentation could rather boost the generalization performance since the model has to exploit other features (e.g. texture, shape) for correct prediction without simply relying on color. I believe this is why \nCutmix and Cutout have been successful as it prevents the model from predicting only highly discriminative features. Meanwhile, the proposed method might fortify the inductive bias of the classifier as the $\\phi$ is trained to simply reduce the classification error of the classifier. It would be great if the authors can provide evidence/analysis to further strengthen their claim.\n\n(-) Section 3.2 and Figure 3 are weak to support the authors’ claim. Of course, extreme global transformations will generate incorrect samples (‘red’ region in Figure 3). But it seems the proposed method also suffers from a similar issue and probably this is the reason why the authors introduce the upper bound $H_{max}$ in their constraints. Wouldn't this problem be solved by relaxing the type and \nintensity of transformation in global augmentation as well? (why should Figure 3 be an illustration rather than the visualizations of the actual decision boundary of Global Augmentation and InstaAug?)\n\n(-) This work aims to propose an augmentation that can conduct information-preserving transformations while keeping a sufficient diversity of transformations. Inspired by goals similar to this work, saliency-based augmentations [3, 4, 5, 6] are recently proposed for information-preserving augmentations. [5, 6] also consider the diversity of augmented data. Although previous related works did not compare with these works, for this work, empirical comparisons with these saliency augmentations are required given the highly similar goals (at least combining InstaAug with these saliency augmentations). Comparing the method against baselines such as Cutmix and Mixup is \ninsufficient.\n\n---\n### Questions\n- Does this method work well for data-deficient conditions? Since data augmentation is most needed when data is scarce, I wonder if the invariance module can learn the proper local invariance even when there is insufficient data.\n\n- Gradient-based saliency [5,7] (or other saliency methods) can detect instance-specific regions without relying on predefined sets. What are the advantages of LRP over these?\n\n- Although this fact doesn't affect my score, is there a reason to experiment with only one dataset per augmentation type? (e.g. TinyImageNet can be used also for testing color jitter). It would be good to experiment with commonly used datasets such as CIFAR and SVHN.\n\n- (minor) The tuning for $H_{min}$ and $H_{max}$ is necessary when practitioners deploy this method to their own data modality or augmentations. It would be nice if the insights of determining $H$ are explained in the main paper\n\n[1] Cheung, Tsz-Him, and Dit-Yan Yeung. \"AdaAug: Learning Class-and Instance-adaptive Data \nAugmentation Policies.\" International Conference on Learning Representations. 2021.\n\n[2] Zhou, Fengwei, et al. \"Metaaugment: Sample-aware data augmentation policy \nlearning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\n[3] Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. \"Puzzle mix: Exploiting saliency and local \nstatistics for optimal mixup.\" International Conference on Machine Learning. PMLR, 2020.\n\n[4] Uddin, A. F. M., et al. \"Saliencymix: A saliency guided data augmentation strategy for better \nregularization.\" arXiv preprint arXiv:2006.01791 (2020).\n\n[5] Park, Joonhyung, et al. \"Saliency grafting: Innocuous attribution-guided mixup with calibrated label \nmixing.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 7. 2022.\n\n[6] Kim, JangHyun, et al. \"Co-Mixup: Saliency Guided Joint Mixup with Supermodular\nDiversity.\" International Conference on Learning Representations. 2020.\n\n[7] Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. \"Deep inside convolutional networks: \nVisualising image classification models and saliency maps.\" arXiv preprint arXiv:1312.6034 (2013)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject