Adversarial Auto-Augment with Label Preservation: A Representation Learning Principle Guided Approach
Authors: Kaiwen Yang
Yanchao Sun
Jiahao Su
Fengxiang He
Xinmei Tian
Furong Huang
Tianyi Zhou
Dacheng Tao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Data Augmentation
Representation Learning
Adversarial Auto-Augment

Abstract: Data augmentation is a critical contributing factor to the success of deep learning but heavily relies on prior domain knowledge which is not always available. Recent works on automatic data augmentation learn a policy to form a sequence of augmentation operations, which are still pre-defined and restricted to limited options. In this paper, we show that a prior-free autonomous data augmentation's objective can be derived from a representation learning principle that aims to preserve the minimum sufficient information of the labels. Given an example, the objective aims at creating a distant ``hard positive example'' as the augmentation, while still preserving the original label. We then propose a practical surrogate to the objective that can be optimized efficiently and integrated seamlessly into existing methods for a broad class of machine learning tasks, e.g., supervised, semi-supervised, and noisy-label learning. Unlike previous works, our method does not require training an extra generative model but instead leverages the intermediate layer representations of the end-task model for generating data augmentations. In experiments, we show that our method consistently brings non-trivial improvements to the three aforementioned learning tasks from both efficiency and final performance, either or not combined with pre-defined augmentations, e.g., on medical images when domain knowledge is unavailable and the existing augmentation techniques perform poorly. Code will be released publicly.

Adversarial Auto-Augment with Label Preservation: A Representation Learning Principle Guided Approach
Kaiwen Yangl Yanchao Sun? Jiahao Su²
Fengxiang He*
Xinmei Tian Furong Huang² Tianyi Zhou? Dacheng Tao
University of Science and Technology of China'; University of Maryland, College Park2
Institute Artificial Intelligence, Hefei Comprehensive National Science Center
JD Explore Academy"; The University of Sydney² kuyang@mail ustc. edu. cn, xinmei@usta .ectttc.dd..... cn, {ycs, jiahaosu, furongh)@umd edu {fengxiang he, tianyi. tianyi.david david.zoon zhou, dacheng tao hagaalmmmlll..ommm
Abstract
Data augmentation is a critical contributing factor to the success of deep learning but heavily relies on prior domain knowledge which is not always available. Recent works automatic data augmentation learn policy to form sequence of aug- mentation operations, which are still pre-defined and restricted to limited options. In this paper, we show that prior-free autonomous data augmentation's 's objective can be derived from a representation learning principle that aims to preserve the minimum sufficient information of the labels. Given example, the objective aims at creating a distant "hard positive example" as the augmentation, while still preserving the original label. We then propose practical surrogate the objective that can be optimized efficiently and integrated seamlessly into existing methods for broad class of machine learning tasks, e.g., supervised, supervieee and noisy-label learning. Unlike previous works, our method does not require training extra generative model but instead leverages the intermediate layer represen- tations of the end-task model for generating data augmentations. In experiments, we show that method consistently brings non-trivial improvements the three aforementioned learning tasks from both efficiency and final performance, either or not combined with strong pre-defined augmentations, e.g., on medical images when domain knowledge is unavailable and the existing augmentation techniques perform poorly. Code is available at: https://github. wwwpggithhu....heeee wew-wwwweeeeeeee wen-yang/L.PA..
Introduction
Data augmentation has emerged as an effective data pre-processing or data transformation step to mitigate overfitting [31], encourage local smoothness [57], and to improve generalization [6] in machine learning pipelines such as deep neural networks. Notably, effective data augmentation, which incorporates class-related data invariance and enriches the in-class sample, is one of the key contributing factors for representation learning with weak or self supervision [9, 23).
Given task, we aim generate "good" augmentations efficiently. As part of the machine learning model pipeline, an autonomous domain-agnosic but task-informed data augmentation mechanism is desirable. However, number of challenges exist. (1) Existing augmentation operators usually hand-crafted based on domain expert knowledge, which is not always available in some domain [49]. For example, widely used augmentations for natural images are not effective on medical images. Moreover, the performance of those machine learning pipelines drastically varies with different choices of data augmentations. (2) Existing few autonomous augmentation approaches developed
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

lately are neither fully autonomous nor universally applicable to varying domains. Although few autonomous data augmentation approaches have been developed in recent years [14, 12], they train policies to produce a sequence of pre-defined augmentation operations and thus are not fully automated and are limited to a few domains. (3) Existing augmentations usually do not fully utilize the task feedback (i.e., task-agnostic) and may be sub-optimal for the targeted task. A class of automated data augmentation methods train an extra data generative model to generate new augmentations from scratch given real-world example [2]. However, they require training a generative model, which is non-trivial task in practice that may either relies on strong prior knowledge or a substantial increased number of training examples.
In this paper, we first investigate the conditions required to generate -agnoii but task- informed data augmentations. Consider a representation learning pipeline, we started from a proba- bilistic graphical model that describes the relations among the label Y, the nuisance N. the example X, its augmentation X', and the latent representations Z. We argue that minimum-suffiiien repre- sentation for the task preserves the label information but excludes other distractive information from the nuisance. We then investigate the conditions for augmentation X' that results in learning such preferred representations. These conditions motivate an optimization objective that can be used to produce automated domain-agnostie but task-informed data augmentations for each example, without replying on pre-defined augmentation operators or specific domain knowledge. Consequently, our proposed optimization objective addresses all aforementioned challenges.
For practicality, we further propose surrogate of the derived objective that can be efficiently computed from the intermediate-layer representations of the model-in-training The surrogate is built upon the data likelihood estimation through perceptual distance [24] defined on the intermediate layers' representations. Specifically. our proposed surrogate objective maximizes the perceptual distance between X and X', under a label preserving constraint on the model prediction of X'. This problem can be efficiently solved by optimizing its Lagaragian relaxation. Thereby, given X and its label Y. the solution to our surrogate objective generates "hard positive examples" for X without loosing its label information. Once generated, X' is used to train the model towards producing the minimum-sufffcicen representation Z for the targeted task. Our proposed method, named Label- Preserving Adversarial Auto Auto-Augment (LP-A3), does not require any extra generative models such as Generative Adversarial Networks, unlike previous automated augmentation methods [40]. We further propose a sharpness-awan criterion selecting only the most informative examples to apply our auto-augmenaation on so it does not cause expensive extra computation.
Our proposed LP-A3 is general and autonomous data augmentation technique applicable to variety of machine learning tasks, such as supervised, semi-supervised and noisy-label learning. Moreover, we demonstrate that it can be seamlessly integrated with existing algorithms for these tasks and consistently improve their performance, In experiments on the three learning tasks, we equip LP-A3 with existing methods and obtain significant improvement on both the learning efficiency and the final performance. The generated augmentations are optimized for the model-in moodel-n-training in a target-task- aware manner and thus notably accelerate the slow convergence in computationally intensive tasks such as semi-supervised learning. It is worth noting that our augmentation can consistently bring improvement to tasks without domain knowledge or strong pre-defined augmentations such medical image classification, on which previous image augmentations lead to performance degeneration.
2 Related work
Hand-crafted vs. Autonomous Data Augmentations. Most of the existing widely used data aug- mentations are hand-crafted based on domain expert knowledge [30, 39, 29, 9, 13]. For example, MoCo [16] and InstDis [47] create augmentations by applying a stochastic but pre-defined data aug- mentation function to the input. CMC [39] splits images across color channels. PIRL [29] generates data augmentations through random JigSaw Shuffling. CPC [30] renders strong data augmentations by utilizing RandAugment [12], which learns a policy producing a sequence of pre- pre-defined augmen- tation operations selected from a pool [14]. AdvAA [60] designs a adversarial objective to learn the augmentation policy. [12, 14, 60] are all based on pre- operations which is not available in certain domains, and their objective cannot guarantee the -abee-preserving of the generated data which may lead to suboptimal performance. "InfoMin" principle of data augmentation is proposed [40] to minimize the mutual information between different views (equivalent to min T(X, X')). However, their theory depends access to minimal sufficient encoder which may be difficult to obtain. In

---

contrast, we not only consider how to generate optimal views or augmentations, but also consider generating the minimal sufficient representation. The algorithm [40] deploys a generator to render augmentation (which may be costly to train especially on non-natural image domains), while we directly learn the augmentation through gradient descent w.r.t. the input.
Information Theory for Representation Learning, Information theory is introduced in deep learn- ing to measure the quality of representations [42. 1]. The key idea is use information bottleneck methods [41, 42] to encourage the learned representation being minimal sufficient. Mutual informa- tion objectives are commonly used in self-supervised learning. For example, InfoMax principle [27] used by many works aims to maximize the mutual information between the representation and the input [39, 4, 46]. But simply maximizing the mutual information does not always lead better representation in practice [43]. In contrast, InfoMin principle [40] minimizes the mutual information between different views. Both InfoMax and the InfoMin principles can be associated with our proposed representation learning criteria in Section 4, as they lead to sufficiency and minimality of the learned representation, respectively.
Augmentation in Self-supervised Contrastive Learning, Self-supervised Contrastive Representa- tion Learning [30, 18, 47, 39, 35, 9] learn representation through optimization of a contrastive loss which pulls similar pairs of examples closer while pushing dissimilar example pairs apart. Creating multiple views of each example is crucial for the success of self--suer contrastive learning. How- ever, most of the data augmentation methods used in generating views, although sophisticated, are hand-crafted or not learning-basee based. Some use luminance and chrominance decomposition [39], while others use random augmentation from pool of augmentation operators [47, 9, 4, 16, 53, 37, 62, 65]. Recently, adversarial perturbation based augmentation has been proposed to generate more challeng- ing positives/negatives for contrastive learning [51, 19].
Data augmentation plays an important role in semi- supervised learning, e.g., (1) consistency regularization [33, 34] enforces the model to produce similar outputs for a sample and its augmentations; (2) pseudo labeling [25] trains a model using confident predictions produced by itself [38] for unlabeled data. Data augmentations are critical [23, 7] because they determine both the output targets and input signals: (1) accurate pseudo labels are achieved by averaging the predictions over multiple augmentations: (2) weak augmentations (e.g.. flip-and-shin
Augmentation in Semi-supervised Learning.
are important to produce confident pseudo labels, while strong augmentations [14, 12]) are used to train the model and expand the confidence regions (so more confident pseudo labels can be collected later). Data selection [55, 63] for high-quality pseudo labels is also critical and its criterion is estimated on augmentations, e.g., the confidence [8] or time-consisteney [63] of each sample.
Augmentation in Noisy-label Learning Two primary challenges in noisy-label learning is clean label detection [28, 15, 20] and noisy label correction by pseudo labels [32, 3. 26]. Both significantly depend on the choices of data augmentations since the former usually relies on confidence threshold- ing and augmentations help rule out the overconfident samples, while the latter relies on the quality of superviee learning. Moreover, as shown previous works [26, 64], removing strong aug- mentations such as RandAugment can considerably degenerate the noisy label learning performance.
3 Preliminaries
Basics of Information Theory Our analyses make frequent use of information theoretical quan-
tities [1]]. Given a joint distribution P and its marginal distributions Px, Py, we define their entropy as HX, Y) = Ex,y[- log P(x,, v)). H(X) = Fx - log P(x)), and H(Y) Fy log P-y)] Furthermore, we define the conditional entropy of X given Y as H( = Ey log H (Xly)] H (X. ---H--H (Y). Finally, we define the mutual information between X and Y as [ A Y)= = (X) - (X = (X) + (Y) (X, Y).
Notations and Problem Setup. In this paper, we use bold capital letters (e.g., X, Y) to denote random variables, lowercase letters (e.g., x. 3/ to denote their realizations, and curly capital letters (e.g., )2) to denote the corresponding sample spaces.
Since we mainly consider supervised and semi-supervised problems, we define Let Px, be the joint distribution of data observation X and label Y, where X is random vector taking values on finite observation space X' (e.g., images) and Y a discrete random variable taking values on the label space y' (c.g., classes), Our goal is to learn a classifier to predict y y from observation e X.
3

---

Y
X
B
Y
Z'
N
A
Represstition
Argmentation
(a) Without Augmentation
(b) With Augmentation
Figure 1: Probabilistic graphical models of representation learning.
Task-nuisanc Decomposition. To advance the analysis, we decouple the randomness in X into two parts, pertaining to the label and another independent to the label. Concretely, variable nuisance N such that 1) the nuisance NN independent to the label Y. i.c., N Y; and 2) the observation X is deterministic function of the nuisance N and the label Y, i.e., X gY,, for some g. Lemma 3.1 demonstrates that such a random variable always exists.
define random
Lemma 3.1 (Task- nuisannus Decomposition [1]). Given a joint distribution Px,,, where Y is discrete random variable, we can always find a random variable N independent of Y such that X d for some deterministic function funciiooo d.
Remarks. We can rewrite the conditions of task-nuisance decomposition in terms of information theory. 1) Since the nuisance the nuisance N and the label Y determines the observation X, we have H N) 0.
independent to the label Y, we have T(Y A N) = O; and 2) Since
4 Principles of Representation Learning: Theoretical Interpretation
4.1 What Good Representation?
In real-world applications, the observation X is usually complex in high-dimensional space X, making it hard to directly learn a good classifier for Y. To remedy this curse of dimensionality, it is important to learn good representation of X, i.e., learn encoder E(-) that maps the high- dimensional observation X into a low-dimensional representation Z. We illustrate the process of data generation and representation learning by probabilistic graphical model as shown in Figure la.
An ideal encoder should keep the important information from (e.g. label-relevant information) and maximally discard the noise nuisance of X, such that it is much casier learn classifier from Z than from X. Based on the above intuition, we define an e-optimal representation of X, which has sufficient information for classifying w.r.t. Y. while remaining little information of the nuisance.
Definition 4.0.1 (c-Minimal Sufficient Representation (c-Optimal Representation)). For a Markov chain Y + X - Z, we say thata representation Z of X is sufficient for Y if T(Z A Y) = I(X A Y), and Z is e-minimal sufficient for Y Zf is sufficient and ((ZAX) < [(2 A X) for all Z satisfying 1 Y) IXX A Y).
Remark. Due the property of mutual information, we have 0 < H(X). The lower c is, the more "minimal" the representation is. When = O, the representation is minimal sufficient, which is desirable property as characterized by many prior works [42, 1].
Definition 4.0.1 characterizes how good a sufficient representation is, based on how much redundant information is remained. Recall that X comes from deterministic function of label Y and nuisance N. The redundancy of Z can also be measured by the mutual information between Z and N. Achille et al. [1] prove that if representation Z is sufficient and is invariant nuisance N, i.c. [ A N) = 0, then Z is also minimal. However, since is not known, is hard to directly encourage the representation to be invariant to N.
Can we learn an c-minimal sufficient representation in a principled way? Inspired by the recent success of data augmentation techniques in self- euperuuee learning and semi- semi-supervised learning, we find that data augmentation can implicitly encourage the representation to invariant to the nuisance N. However, most augmentation methods are driven by pre-defined transformations, which do not necessarily render a minimal sufficient representation. In the next section, we will analyze the effects of data augmentation in representation learning in details.

---

4.2 Proper Data Augmentation Leads to (Near-)Optimal Representation
In this section, we investigate the role of data augmentation for learning good representations. We first make the following mild assumption on the underlying relationship between X and Y.
Assumption 4.1. There exists a deterministic function क + V, i.e., H(Y 0.
Assumption 4.1 requires that there exists "perfect classifier" that identifies the label of the observation a with no error, which is common in practice. Note that for data with ambiguity, a tie breaker can be used to map each observation to unique label. Therefore, Assumption 4.1 is realistic.
Let 9 be deterministic augmentation function such that X' g(X, A) is the augmented data, where A is a random variable denoting the augmentation selection. For example, if X = z is an image sample, A rotated image sample. We learn an encoder E(() that maps the augmented data X' to a representation Z'. With this augmentation processes, the graphical model in Figure la is updated to Figure 1b.
the augmentation "rotate by 90 degree", then X' x' the corresponding
We show in the theorem below that if the augmentation process preserves the information of Y, Z' can be sufficient for Y. Furthermore, if the augmented data X' contains no information of the original nuisance N, Z' will be invariant to N and thus will become a minimal sufficient representation.
Theorem 4.2. Consider label variable Y, observation variable X and muisance variable N satisfying Assumption 4.1. Let A be the augmentation variable, X' be the augmented data, and Z" be the solution to
argmaXz
I(Z' x²) or I(Z' NY)
(1)
subject to I(Z' A) 0.
Then, Z" is e-minimal sufficient representation of X. for label Y if the following conditions hold: Condition (a): 1(X' Y) = I(X Y) (X' is an in-class augmentation) and
Condition (b): 1(X' A IN S (X' does not remain much information about N).
Remarks. (1) The objective learning Z can be either aak-inndeenndent (maximizing / (Z' A X')), or task-dependent (maximizing I (Z' A Y)). The former matches the "InfoMax" principle commonly used in self-supervised learning works [27, 18], while the latter can be achieved by supervised training (e.g., learning classifier of Z for Y with cross-entropy loss).
(2) When Condition (b) holds for = 0, representation Z' is optimal (minimal sufficient).
shows Theorem 4.2, proved in Appendix B. shows that if we have good augmentation that maximally
perturbs the label-irrelevant information while keeps the label-relevant information, then the repre- sentation learned on the augmented data can be minimal sufficient. Theorem 4.2 serves as principle of constructing augmentation. Based on this principle, we propose an auto-augment algorithm in Section 5, and verify the algorithm in wide range tasks in Section 6.
5 Proposed Methods
In this section, we introduce our data augmentation and how to obtain the augmentation using the representation learning network F(-). Then we show how to plug our augmentation into the representation learning procedure of F(.).
X
E
Z
M
-
Y
F ((::0 O)
5.1 Lrbee-Preserving Adversarial Auto-Augment (LP-A3)
Figure Network architecture.
As illustrated in the previous section, an ideal data augmentation X' for representation learning should contain as little information about nuisance N as possible while still keeping all the information about class Y. Since N not observed, we transfer the objective minx. I [(X' ^ N) into minx T(X' A X) since T(X' A X) T(X' A N) + T(X' A Y) and T(X' A Y) is a constant under the constraint I(X' Y) = IXX Y). Thus the optimization problem is:
min [(X' A X) X
S.L. 1(X' A Y) I(XAY).
Implementation of Mutual Information. To solve Equation (2), computing the mutual information terms T(X' A X), T(X' Y) and T(X Y) is required. Next, we will show how to compute these terms using neural net classifier F(:0), parameterized by 0, that consists of two components:
5

---

representation encoder E() and a predictor M(-). Specifically, F(=0)) = M(E))), where the representation encoder E(-) maps input X into representation Z. and the predictor M()) predicts the class of Z. This is demonstrated in Figure 2.
H(Y: H(Y(X') and T(X Y) H(Y) - HY X), we can remove the H() term in both sides and turn the constraint into [ (Y X Thus we only need to compute the conditional entropy of Y given X or X', which can be approximated through the neural net classifier: X Ex.y [-logP(y|=]] P Ex.y [-log (z(: [v]]), where we use softmax class probability F(x; (yl approximate the likelihood P(y|z). And H(|]X') can be computed similarly.
Constraint implementation. Since T(X' Y)
Objective implementation Then we show how to compute the objective I(X' A X). Since [(X' A X) = = - HXX where H(X) is not related to X' and thus can be neglected, we only need to compute H (7(XX')) Fx,xx --rla//)))/'']] We use the Learned Perceptual Image Patch Similarity (LPIPS) [59] between x and x' to compute the data likelihood P(z|zz)) since LPIPS distance widely used metric to measure the data similarity in data generative model field [21, 58] and many previous work has shown that LPIPS distance is the best surrogate for human comparisons of similarity [59, 24], compared with any other distance including & and loo distance. Although such surrogate may have error, it worth noting that Theorem 4.2 allows the surrogate to have e error. The LPIPS distance defined by the & distance of stacked feature maps from a neural network. Here we use F-; 0) to compute the LPIPS distance. Let F(; 0) has L layers and Fi denotes these thannel-normalizer activations at the (-th layer of the network. Next, the activations are normalized Fi((:))
Fi
/orghi : .... Verghe ), where and fy are the width and height of the activations in layer I, respectively. The LPIPS distance between input x and the augmentation x' is then defined as:
again by layer size and flattened into single vector Φ(x)
LPIPS(x,,') & [0(()) - (xx')l2.
Constraint Relaxation for Efficiency. computed by solving the following optimization problem using the neural network F(; 0) in practice:
Now, given an input x, its data augmentation x' can be
min -(((z) - x(z')1 )l s.t. logF(<' (yvl = lo:F(x;))lv))
(4)
The equality constraint in Equation (4) is too strict to solve since it may be inefficient to search for an x' that exactly satisfies (a'; 0) a and change the constaint into: log F(a; 0yl 2 F(;; [y] - 0. It's worth noting that a is sufficiently small, the label is still well preserved. There is trade-off to the value of a, we search to find a sweet spot where the problem is practical to solve and meanwhile the label is well preserved.
logF [)]. Thus we relax the constraint with small
There are many off-the-shelf methods that solve Equation (4), and here we apply the Fast Lagrangian Attack Method [24] as a demonstration. We initialize x' by æ plus uniform noise. And we find the optimal x' by solving the following the Lagrangian multiplier function and gradually scheduling the value of the multiplier X:
min (((x)) d(x')l2 + log
- log F (a::gF((20000 0)(v) - o)
(5)
The detailed procedure of the algorithm can be found in Appendix 2. The algorithm has a similar form as adversarial attack [61, 52] in that they both find an optimal augmentation x' by adding perturbations to the original image x. However, the difference is that we aim to generate hard augmentation that preserves the label, while adversarial attack aims change the class label.
5.2 Plugging LP-A3 into Representation Learning Task
One primary advantage of LP-A3 is that it only requires a neural net F(::0) to produce the aug- mentation and F(:; 0) can be the current representation learning model, so we can plug LP-A3 into any representation learning procedure requiring no additional parameters, which is plug-and-play and parameter- free. At each step, we first fix F:; 0) to generate the augmentation x' by solving Equation (5) using Algorithm 2. And then we train F(:: () by running the original representation learning algorithm using our augmentation x'.
Data selection. It not necessary to find hard positives for every sample. To save more compu- tation, we can apply sharpness-awart criterion, i.c,, time-consistency (TCS) [63], to select the most

---

informative data (7% data with the lowest TCS in Algorithm 1) that have sharp loss landscapes, which indicate the existence of nearby hard positives, and we only apply LP-A3 them. reduces the computational cost without degrading the performance because (1) the improvement brought by aug- mentations is limited for examples whose loss already reaches a flat minimum, while the model does not generalize well near examples with sharp loss landscapes; and (2) the hard positives for examples with flat loss landscape are distant from the original ones and might introduce extra bias to the training.
LP-A3 is compatible with any representation learning task minimizing loss L X' x y x W + R++, which takes data batch and a model output loss value. here denotes the groundtruth label for labeled data and pseudo label for unlabeled data. The pseudo-code of plugging LP-A3 into the representation learning procedure with TCS-based data selection provided in Algorithm
Algorithm Plug LP-A3 into any representation learning procedure
Input: Loss for the targeted task L X x W R+; training data (X;3"); neural network
0); class preserving margin e; data selection ratio learning rate
Output: Model parameter trained with LP-A3
1: while not converged do
2: Sample batch
((r,,))),.. (It, y6)) (X,
3: Data selection: S <- 7% data with the lowest TCS in B
4: LP-A3: Freeze f and solve Equation (5) using Algorithm for every sample in S, resulting in
augmented set A = {xx1, (z),.1)). v1),,... (I'm fm } of size |S];
5: Learning with LP-A3 augmented data and original data: t- V0 VoL(A
6: end while
6 Experiments
In this section, we apply LP-A3 as a data augmentation method to several popular methods for three different learning tasks, i.e., (I) semi-supervised classification; (2) noisy-label learning and (3) medical image classification. In all the experiments, LP-A3 can (1) consistently improve the convergence and test accuracy of existing methods and (2) autonomously produce augmentations that bring non-trivial improvement even without any domain knowledge available. A walk-clock time comparison is given in the Appendix, showing LP-A3 effectively reduces the computational cost. In addition, we conduct thorough sensitivity study of LP-A3 by changing (1) label-preserving margin and (2) data selection ratio the three tasks. More experimental details be found in the Appendix.
Original Image
ScoreCAM
LP-A3
ScoreCAM
RandAugment
ScoreCAM
Sample
0.27
0.28
0.11
Sample
0.18
Sample
Figure 3: Visualization of medical image augmentations on the test set of DermaMnist. Blue (red) bounding box marks the correct (wrong) prediction of a ResNet18 classifier and its confidence on the groundtruth class reported beneath the box. ScoreCAM [44] produces a heatmap highlighting important areas (by yellow color) of image that neural mainly relies make the prediction.

---

Accurary
Accuracy
Accuracy
Accuracy
FixMatch
203
DivideMix DivideMix+LP-A3
PES
RecNerS0 ResNetS0+
FixMatch+LP-A3
PES+LP-A3
2009 1M000 100000 20000 250000
Iteration
280
1009
5660
0000
280
Iteratioon
180
240
Iteration
1900
(a) FixMatch
(b) DivideMix
(c) PES
(d) MedMnist
Figure Convergence Curve when applying LP-A3 different tasks baselines.
6.1 Medical Image Augmentations produced by LP-A3 vs. RandAugment
We visualize data augmentations generated by LP-A3 and RandAugment [12] on the testset of DermaMnist [50] with ResNet-18 classifier and its confidence on the groundtruth class in Fig. We also use ScoreCAM [44] as an interpretation method to highlight the area in each image that the classifier relies on to make the prediction. we find that LP-A3 preserves relevant derma areas highlighted by ScoreCAM and they are consistent with those in the original image. On the con- trary, RandAugment changes the color or occludes those derma areas, resulting in highly different ScoreCAM heatmaps and hence wrong predictions (red bounding box in Fig. 3). Instead, LP-A3 can preserve the class information and mainly perturb the class-unrelated
class-unrelated area the original image.
6.2 Applying LP-A3 Three Different Representation Learning Tasks
Here we apply L.PA3 to three different tasks by pluging LP-A3 to existing baselines of each task. Fig. 4 shows that LP-A3 greatly speeds up the convergence of each baseline.
Semi-supervised learning To evaluate how LP-A3 improves the learning without sufficient labeled data, we conduct experiments semi- semi-supervised classification standard benchmarks including CI- FAR [22] and STL-10 [10] where only very small amount of labels are revealed. We apply LP-A3 in FixMatch [36] and compare it with the original FixMatch and InfoMin [40]. learnable augmentation method for semi-supervised learning. Their results are reported in Table 1, where LP-A3 consistently improves FixMatch and the improvement becomes It's worth noting that the original FixMatch already employs a carefully designed set of pre-defined augmentations [13] that have been tuned to achieve the best performance, indicating that LP-A3 is complementary to existing data augmentations, Moreover, LP-A3 also outperforms InfoMin by large margin (> 5%), which indicates that LP-A3 is also superior to existing learnable augmentations.
significant if reducing the labeled data.
Table Semi-supervised Learning performance on CIFAR with different amounts labeled data. 8 denotes results reproduced using the official code. FixMatch and LP-A3 are trained for 2 SGD steps. InfoMin's 's results on CIFAR are missing since their paper only reports the result on STL-10. Error bars (mean and std) are computed three random trails.
CIFARIO
CIFARIOD
STL-10
# Label
400
2500
10000
1000
InfoMin (RGB) [40]
86.0
InfoMin (YDbDr) [40]
87.0
FixMatch [36]
89.51=3.144
93.21
94.66=0.133
49.30:22.45
67.21=10.94
74.31
91.59:::0.16
FixMatch [36] LP-A3
92.39::1..1
94.03-10.31
95.111:0.11
16.16::1.82
72.23:10.57
0.11
92.63-10.14
Noisy-label Learning Data augmentation is critical to noisy-label learning by providing different views of data to prevent neural nets from overfitting to noisy labels. We apply LP-A3 to two state-of- the-art methods DivideMix [26] and PES [5] CIFAR with different ratios of noise labels. LP-A3 can consistently improve the performance of these two SoTA methods and the improvement is more significant in more challenging cases with higher noise ratios, e.g., CIFAR with 90% of labels to be noisy, LP-A3 improves PES by 2 15% (Table 2).
Medical Image Classification To evaluate the performance in specific areas without domain knowledge, we compare LP-A3 with existing data augmentations on medical image classification tasks from MedMNIST [50], which is composed of several sub-dataset with various styles of medical images. We compare our LP-A3 with RandAugment [13] on training ResNet- 18 and ResNet-50 [17].
We report the results in Table 3, where RandAugment designed for natural images fails improve

---

Table 2: Noisy-label learning performance on CIFAR with different ratios of symmetric label noises. 8 denotes the results reproduced by the official code. Error bars (mean and std) are computed over three random trails.
Dataset
CIFARIO
CIFARI00
Noise Ratio
50%
80%
90%
50%
80%
90%
Mixup [56]
87.1
71.6
52.2
57.3
30.8
14.6
P-correction [54]
88.7
76.5
58.2
56.4
20.7
8.8
M-correlation [3]
88.8
76.1
58.3
58.0
40.1
14.3
DivideMix [26]
94.4
92.9
75,4
74.2
59.6
31.0
DivideMix+LP-A3
94.89=0.05
93.70:=0.19
19.35++1.33
74.12:40.23
61.00++0.34
32.55=0.25
PES [5]
94.89=0.12
92.15++0.23
84.98-10.36
74.19440.23
61.47-00.88
21.15=3.15
PES+LP-A3
95.10 0.14
93.26++0.21
87.71 +-.36
74.57++025
62.98 0.49
40.61 1.10
Table 3: Medical Image Classification MedMNIST [50]. All the models trained for 100 epochs. Error bars (mean and std) computed over three random trails.
Method
PathMNIST
DermaMNIST
TissueMNIST
BloodMNIST
ResNet-18
94.34=0.18
76.14+t.009
68,2840.17
96,81
ResNet- BosNet-I8+Rande 18+RandAugment 93.52=00.99
73.3
62.03=0.14
95.00=0.21
ResNet-18+LP-A3
94.42=0.24
76.22:10.27
68.63:0.14
96.97=0.06
ResNet-50
94.47=00.88
75.24-10.27
69.69-10.23
96.91
ResVet--SORRnnAAgmmenn
94.02=0..37
71.65::0.30
65.13:10.33
95.14=10.06
ResNet-50+LP-A3
94.57=0.07
75.71:1.22
69.89:10.08
97.01:10322
OctMNIST
OrganAMNIST
OrganCMNIST
OrganSMNIST
ResNet-18
78.67-00226
94.21+00.09
91.81++012
81.57--0.07
ResNec-18+RRada
Augmentt
7..00=
94.18++0.20
91.38
80.52+0.32
ResNet-18+LP-A3
80.27
94.73.10.21
92.41-10.22
82.28
ResNct-50
78.37=0.52
94.31
91.80
81.11
ResNet-50+RandAuggmen
76.63=0.58
94.59+0.11
91.10++.12
80.47=0.37
RcsNoc-50+LP-A3
79.40=0.36
94.95+00.19
92.16+00.23
82.15=00008
the performance in this scenario. In contrast, LP-A3 does not rely on any domain knowledge brings improvement to all the datasets, especially for OctMNIST where the improvement is over 1%. The results indicate that hand-crafted strong data augmentations do not generalize all domains but LP-A3 can autonomously produce augmentations guided by our representation learning principle without relying on any domain knowledge.
Preserving Varginio
Selectio
weneewinggg Pargin
Dato Selection
Preservigg Margin"
Semi- supervised classification
Noisg-label Classification
MedMnist
Figure 5: Sensitivity Analysis of label preserving margin and data selection ratio.
6.3 Sensitivity Analysis of Hyperparameters
Label preserving margin a: We evaluate how LP-A3 performs with different label preserving margin a on the three tasks. The results are presented in Fig. 5, where a reverse U-shape observed. And LP-A3 using all the evaluated outperforms baselines, which indicates LP-A3 is robust to

---

Data selection ratio: We evaluate the performance of LP-A3 with different amount of data selected on the three tasks. As shown in Fig. 5, selecting all the data does not perform the best since some data' augmentations are useless to apply data augmentation. Moreover, selecting only 30% data to apply LP-A3 can outperform all baselines by a large margin, especially on MedMNIST where the improvement is 2 2%, which verifies the effectiveness of LP-A3 and our data selection method.
Conclusion
In this paper, we study how to automatically generate domain-agnoste but task-informed data aug- mentations. We first investigate the conditions required for augmentations leading to representations that preserves the task (label) information and then derive an optimization objective for the augmen- tations. For practicality, we further propose a surrogate of the derived objective that can be efficiently computed from the intermediate-laye representations of the model-in-training The surrogate is built upon the data likelihood estimation through perceptual distance. This leads to LP-A3, a general and autonomous data augmentation technique applicable to a variety of machine learning tasks, such as supervised, semi-super and noisy-label learning. In experiments, we demonstrate that LP-A3 can consistently bring improvement to SoTA methods for different tasks even without domain knowledge. In future work, we will extend LP-A3 to more learning tasks and further improve its efficiency.
Acknowledgements
This work was supported by, the Major Science and Technology Innovation 2030 "New Generation Ar- tificial Intelligence" key project under Grant 2021ZD011 1700, NSFC No. 61872329, No. 62222117, 17, and the Fundamental Research Funds for the Central Universities under contract WK3490000005. Huang, Sun and Su are supported by NSF-IIS-FAI program, DOD-ONR- Office of Naval Research, NOD-DARPA-Defienss Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD). Huang is also supported by Adobe, Capital One and JP Morgan faculty fellow- ships. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
References
[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):144-11980, 2018.
[2] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks, 2017.
[3] Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference machine learning, pages 312-321. PMLR, 2019.
[4] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[5] Yingbin Bai, Erkun Yang. Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
[6] Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmentation are class dependent, 2022.
[7] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: suni-super learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations (ICLR), 2020.
[8] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 5050-5060. 2019.
10

---

[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[10] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[11] Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of information theory, 1(1):279-333) 1991.
[12] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space, 2019.
[13] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[14] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaug- ment: Learning augmentation policies from data. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[15] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels.
In Advances in neural information processing systems, pages 8527-8537, 2018.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
[19] Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. Advances in Neural Information Processing Systems, 33: 11111001-111 11799,, 2020.
[20] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentorne:: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304-2313, 2018.
[21] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Alex Kurakin, Chun-Liang Li, Colin Raffel, David Berthelot, Ekin Dogus Cubuk, Han Zhang, Kihyuk Sohn, Nicholas Carlini, and Zizhao Zhang. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
[24] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. arXiv preprint arXiv:2006. 12655, 2020.
[25] Dong-Hyun Lee. Pseudo-label: : The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, 2013.
[26] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.0733 0770099 2020.
[27] Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
[28] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:447-461, 2016.
[29] Ishan Mísra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
11

---

[30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807003745 2018.
[31] Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. ArXiv,
2017.
[32] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6556, 2014.
[33] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems 29 (NeurlPS), pages 1173-117.. 2016.
[34] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi -uneiiiseee learning. In Advances in Neural Information Processing Systems 29 (NeurlPS), pages 1163-1171, 2016.
[35] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
[36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems, 33:596-608, 2020.
[37] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning, arXiv preprint arXiv:2004.044 04136 2020.
[38] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30 (NeurlPS), pages 1195-1204. 2017.
[39] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer, 2020.
[40] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020.
[41] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[42] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
[43] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2019.
[44] Haofan Wang. Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pages 24-25, 2020.
[45] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages 9929-9939. PMLR, 2020.
[46] Mike Wu, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. On the importance of views in unsupervised representation learning. preprint, 3, 2020.
[47] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018
[48] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoe V Le. Adver- sarial examples improve image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 119-828,7 2020. 2020.
12

---

[49] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: large-scale lightweight benchmark for 2d and 3d biomedical image classification, 2021.
[50] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv preprint arXiv:2110.1423
2021.
[51] Kaiwen Yang, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Identity-disentangled adversarial augmentation for self- supervisdd learning. In International Conference on Machine Learning, pages 25384. PMLR, 2022.
[52] Kaiwen Yang. Tianyi Zhou, Yonggang Zhang, Xinmei Tian, and Dacheng Tao. Class- disentanglement and applications in adversarial detection and defense. Advances in Neural Information Processing Systems, 14416051--6066 2021.
[53] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6210-6219, 2019.
[54] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7017-7025, 2019.
[55] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting eemi-supervised learning with curriculum pseudo labeling. In NeurIPS, 2021.
[56] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 191.0917. 2017.
[57] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
[58] Pan Zhang, Bo Zhang. Dong Chen, Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-basee image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5143-5153, 2020.
[59] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[60] Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong, Adversarial autoaugment. arXiv preprint arXiv:
2019.
[61] Yonggang Zhang, Xinmei Tian, Ya Li, Xinchao Wang, and Dacheng Tao. Principal component adversarial example. JEEE Transactions on Image Processing, 29:4804-488 2020.
[62] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. Distilling localization for self-supervised representation learning. In 35th AAAI Conference Artificial Intelligence (AAAI-21), pages 10990-10998. AAAI Press, 2021.
[63] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for semi- supervised learning. In International Conference on Machine Learning, pages 11523-11533. PMLR, 2020.
[64] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label de- tection to noisy label celf-correction. In International Conference on Learning Representations, 2021.
[65] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 6002-6012, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
13

---

(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See Sec.4
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL.? [No] Will open source later
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Aappendix C
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "l1k3eaxRU_p", "reviewer": "Reviewer_vSqH", "summary": "The paper proposes a new prior free data augmentation strategy while usual approaches rely on prior knowledge of elementary transformations that preserve the labeling. The method is cast as an optimization problem that aims at producing a hard positive example for every training sample. Augmentation data are refined epoch after epoch as the learning of the model progresses. \n\nThe method does not require learning an additional generative model and may be used in various tasks such as supervised learning, semi supervised learning and noisy label learning.\nThe method is well funded. It is derived from a formalization of the process of data augmentation as a probabilistic graphical model and of the formalization of what should be a good representation and what should be a good augmentation strategy. Details are provided on how to implement the strategy and experimental results are reported on few datasets and for different tasks, n supervised learning semi supervised learning, noisy label learning. ", "questions": "Section 3 line 132 it is said « We decouple the randomness into two parts… ». But then there is only one random variable N? and X is a deterministic function of N and of Y. What is the second part of randomness ? \n\nSection 4 build on the notion of sufficient representation but i don’t see a clear definition of this.\n\nSection 5.2. Maybe you could add some details about TCS, which you only mention, but which is part of your method (cf. Algorithm 1).    \n\nDatasets. What are the sizes of the datasets used in the experiments ?", "limitations": "Not applicable", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The approach looks innovative with a refinement of the augmentation data from which the model is learned, all along the learning process. I am not a specialist of data augmentation but it does not seem to be the mainstream approach. On the opposite badine methods usually rely on prior knowledge of label preserving transformations, and most often are not learned using a task feedback.   \n\nThe paper is well written and the approach is well detailed put to implementation details. \n\nExperimental  results show significant and consistent improvement of the methods over the baselines that look well chosen for the different tasks that are investigated.\n\nSome parts could be more detailed. In particular section 4 builds on the notion of sufficient representation but i don’t see a clear definition of this and i did not fully get the definition 4.0.1.  In the algortihm 1, i am not sure if selecting samples for which augmentation will be considered is a way of accelerating the learning or is necessary ti get good results. In the experimental section main ideas underlying baselines could have been explained (e.g. FixMatch [35])  at least the main common features and main differences with the proposed method could be discussed. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AIQEQ4-hcgS", "reviewer": "Reviewer_JwsS", "summary": "The paper proposes a data augmentation objective, LP-A3,  to create label-preserving adversarial examples that improve the model generalization of deep learning models. More precisely, the method perturbs an input image to maximize its perceptual difference while constraining its predicted class probability to be similar to the original image. Experimental results show the effectiveness of the proposed objective in supervised, semi-supervised and noisy-label learning tasks.", "questions": "__O1.__ It seems that LP-A3 relies on the representation learning network $F$ to compute the LPIPS distance $\\phi(x)$ and softmax class probability $F(x; \\theta)[y]$ in Eq. (4). During the early stages of training, when $F$ is not well trained, is it possible that it may generate incorrect training signals for computing the LPIPS distance and softmax class probability for the label-preserving constrain? How does the method deal with this problem especially for small and more complicated datasets?", "limitations": "Please see the above concerns.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\n\n__S1. [reasonable and well-motivated]__ Data augmentation is crucial for modern deep learning methods and becomes a bottleneck when deploying them to domains with less prior knowledge. From this perspective, the paper is tackling an important problem.\n\n__S2. [practical solution]__ The method is easy to be integrated into existing learning frameworks.\n\n__S3. [clear writing]__ Overall, the paper is easy to follow.\n\n__Weakness:__\n\n__W1. [Missing related work]__ There are some related works this paper omitted. For example, Adversarial AutoAugment [1] also uses an adversarial objective to learn an augmentation strategy formed by a set of label-preserving transformations for the target dataset. The paper can compare LP-A3 with other adversarial data augmentation strategies in terms of their idea and performance, so we will know which approach is better in which scenarios.\n\n__W2. [Insufficient experiment]__\n\n1. The experiments are not extensive enough. Only the results on several tiny datasets are provided. The results on the large-scale ImageNet dataset will make this paper stronger. In supervised image classification tasks, the paper compares LP-A3 with RandAugment on MedMNIST. It would be better to compare LP-A3 with other Automated Data Augmentation methods, like AutoAugment, Adversarial AutoAugment, and sample-mixing augmentations, like Mixup, Cutmix, on the full dataset of popular classification benchmarks, e.g. CIFAR10 and CIFAR100. This helps to verify the performance gains compared with the state-of-the-art data augmentation schemes.\n\n2. Prior works such as RandAugment run multiple times for the model and present the mean and standard deviation. Did the authors run multiple times for each model and what about presenting the mean and standard deviation?\n\n__W3. [Adversarial training]__ According to lines 252-255 and Appendix B, it seems that LP-A3 uses an adversarial-attack-like procedure to perturb the input images.  Adversarial attack perturbs an image to alter the class label; oppositely, LP-A3 aims to preserve the class label. With the opposing objectives, can LP-A3 be integrated with adversarial training? Does LP-A3 affect the use of perturbation-based adversarial attacks to create adversarial samples in adversarial defense?\n\n[1] Zhang et al., Adversarial AutoAugment. ICLR, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9riZwP5yRLk", "reviewer": "Reviewer_VALt", "summary": "This paper proposes a new data augmentation method that does not require training an extra generative model but instead leverages the intermediate layer representations of the end-task model for generating data augmentations. The augmentation is built based on a representation learning principle that aims to preserve the minimum sufficient information of the labels. Experiments across different datasets are conducted to demonstrate the effectiveness of the proposed method.\n\nAfter rebuttal:\nThanks for the authors' responses. The authors address some of my concerns so I raise my rate. ", "questions": "Please refer to the weaknesses.", "limitations": "-", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ A novel, simple, and intuitive idea for data augmentation\n+ The simple idea is very effective when implemented correctly\n+ The paper is easy to follow, well-written, and inspiring. Moreover, the theoretical interpretation of the method is sound.\n+ Results with various different tasks, including semi-supervised learning, noisy-label learning, and medical image classification, and datasets are provided in this manuscript to show the effectiveness of the proposed LP-A3.\n\nWeakness\n- How does the TCS select the hard positive data in Alg.1? I think it is necessary to provide the selection detail to make the manuscript self-contained.\n- The experimental results in the manuscript are insufficient to show the effectiveness of LP-A3. In semi-supervised learning tasks, only one augmentation method, InfoMin, is tested on STL-10 dataset as a comparison. What are the backbone networks in the experiments in Table 2? Moreover, How’s the performance of LP-A3 on CIFRA10/100 in a supervised classification manner? Evaluation with other backbone networks rather than ResNet-18 and ResNet-50 is also important.\n- How’s the performance of LP-A3 on the large-scale dataset, such as ImageNet\n- I’m curious about the results in Fig.5. As the proposed LP-A3 is designed to generate the augmented X’ to preserve the minimum sufficient information of the label, why the performance of LP-A3 can drop when all data is selected for augmentation? The author stated that some data augmentations are useless to apply data augmentation, which did not convince me. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5nRdRO9MdhY", "reviewer": "Reviewer_sdEW", "summary": "The paper proposes a data augmentation strategy without any pre-defined augmentation operations. The goal of the data augmentation is to create a distant hard positive example while preserving the label. The paper uses a surrogate, based on LPIPS distance and model predictions, to find the optimal augmentation for an image. The experiments are conducted on supervised, semi-supervised, and noisy-label learning.", "questions": "The main questions are presented in Strengths And Weaknesses, but there are some more questions that I am curious about.\n1. For Medical Image Classification, AutoAugment might be too strong. Some common data augmentations might have good results, such as random crop+colorjitter+random horizontal flipping, used in Mean Teacher.", "limitations": "Not discussed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n1. The auto-augmentation idea is clearly presented and easy to follow.\n2. The paper provides a strong theoretical derive to the formula of LP-A3 (Eq.2), where the augmentation should contain less information about the original image and preserve the label.\n3. The LP-A3 method achieves significant improvement on three different learning tasks.\n\nweakness:\n1. The implementation of the optimization problem has some obvious drawbacks. The paper uses the neural net classifier to approximate the conditional entropy of Y given X. However, the prediction of the classifier can be incorrect. Therefore, there might be some augmentation X' has the same classifier prediction as X, while the oracle label of X' is different from X, then the augmentation X' is undesired. On the other hand, there exists some augmentation X' has the same oracle label as X, while classifier prediction of X' is very different from X, but these augmentations are not considered by the introduced implementation. Virtual Adversarial Training considers the second situation and shows that augmentation should change the classifier prediction. So making the augmentation preserving the classifier prediction might not be the optimal choice.\n2. The experimental result of semi-supervised learning is not very convincing. The reported performance of FixMatch is lower than the results in the original paper of FixMatch. The result can have some variance, especially in semi-supervised learning with few labels. It is possible the reported result is selected from multiple random seeds. The author should explain the unsatisfactory reproduction and report the variance of results for multiple random seeds.\n3. It is not clear that the proposed LP-A3 itself can improve Semi-supervised Learning and Noisy-label learning. FixMatch uses RandAugment and CTAugment for data augmentation. The paper should test to replace all data augmentation in FixMatch with the proposed LP-A3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept