Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new prior free data augmentation strategy while usual approaches rely on prior knowledge of elementary transformations that preserve the labeling. The method is cast as an optimization problem that aims at producing a hard positive example for every training sample. Augmentation data are refined epoch after epoch as the learning of the model progresses. 

The method does not require learning an additional generative model and may be used in various tasks such as supervised learning, semi supervised learning and noisy label learning.
The method is well funded. It is derived from a formalization of the process of data augmentation as a probabilistic graphical model and of the formalization of what should be a good representation and what should be a good augmentation strategy. Details are provided on how to implement the strategy and experimental results are reported on few datasets and for different tasks, n supervised learning semi supervised learning, noisy label learning. 
strengths_and_weaknesses: The approach looks innovative with a refinement of the augmentation data from which the model is learned, all along the learning process. I am not a specialist of data augmentation but it does not seem to be the mainstream approach. On the opposite badine methods usually rely on prior knowledge of label preserving transformations, and most often are not learned using a task feedback.   

The paper is well written and the approach is well detailed put to implementation details. 

Experimental  results show significant and consistent improvement of the methods over the baselines that look well chosen for the different tasks that are investigated.

Some parts could be more detailed. In particular section 4 builds on the notion of sufficient representation but i don’t see a clear definition of this and i did not fully get the definition 4.0.1.  In the algortihm 1, i am not sure if selecting samples for which augmentation will be considered is a way of accelerating the learning or is necessary ti get good results. In the experimental section main ideas underlying baselines could have been explained (e.g. FixMatch [35])  at least the main common features and main differences with the proposed method could be discussed. 
questions: Section 3 line 132 it is said « We decouple the randomness into two parts… ». But then there is only one random variable N? and X is a deterministic function of N and of Y. What is the second part of randomness ? 

Section 4 build on the notion of sufficient representation but i don’t see a clear definition of this.

Section 5.2. Maybe you could add some details about TCS, which you only mention, but which is part of your method (cf. Algorithm 1).    

Datasets. What are the sizes of the datasets used in the experiments ?
limitations: Not applicable
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a data augmentation strategy without any pre-defined augmentation operations. The goal of the data augmentation is to create a distant hard positive example while preserving the label. The paper uses a surrogate, based on LPIPS distance and model predictions, to find the optimal augmentation for an image. The experiments are conducted on supervised, semi-supervised, and noisy-label learning.
strengths_and_weaknesses: strengths:
1. The auto-augmentation idea is clearly presented and easy to follow.
2. The paper provides a strong theoretical derive to the formula of LP-A3 (Eq.2), where the augmentation should contain less information about the original image and preserve the label.
3. The LP-A3 method achieves significant improvement on three different learning tasks.

weakness:
1. The implementation of the optimization problem has some obvious drawbacks. The paper uses the neural net classifier to approximate the conditional entropy of Y given X. However, the prediction of the classifier can be incorrect. Therefore, there might be some augmentation X' has the same classifier prediction as X, while the oracle label of X' is different from X, then the augmentation X' is undesired. On the other hand, there exists some augmentation X' has the same oracle label as X, while classifier prediction of X' is very different from X, but these augmentations are not considered by the introduced implementation. Virtual Adversarial Training considers the second situation and shows that augmentation should change the classifier prediction. So making the augmentation preserving the classifier prediction might not be the optimal choice.
2. The experimental result of semi-supervised learning is not very convincing. The reported performance of FixMatch is lower than the results in the original paper of FixMatch. The result can have some variance, especially in semi-supervised learning with few labels. It is possible the reported result is selected from multiple random seeds. The author should explain the unsatisfactory reproduction and report the variance of results for multiple random seeds.
3. It is not clear that the proposed LP-A3 itself can improve Semi-supervised Learning and Noisy-label learning. FixMatch uses RandAugment and CTAugment for data augmentation. The paper should test to replace all data augmentation in FixMatch with the proposed LP-A3.
questions: The main questions are presented in Strengths And Weaknesses, but there are some more questions that I am curious about.
1. For Medical Image Classification, AutoAugment might be too strong. Some common data augmentations might have good results, such as random crop+colorjitter+random horizontal flipping, used in Mean Teacher.
limitations: Not discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a data augmentation objective, LP-A3,  to create label-preserving adversarial examples that improve the model generalization of deep learning models. More precisely, the method perturbs an input image to maximize its perceptual difference while constraining its predicted class probability to be similar to the original image. Experimental results show the effectiveness of the proposed objective in supervised, semi-supervised and noisy-label learning tasks.
strengths_and_weaknesses: __Strengths:__

__S1. [reasonable and well-motivated]__ Data augmentation is crucial for modern deep learning methods and becomes a bottleneck when deploying them to domains with less prior knowledge. From this perspective, the paper is tackling an important problem.

__S2. [practical solution]__ The method is easy to be integrated into existing learning frameworks.

__S3. [clear writing]__ Overall, the paper is easy to follow.

__Weakness:__

__W1. [Missing related work]__ There are some related works this paper omitted. For example, Adversarial AutoAugment [1] also uses an adversarial objective to learn an augmentation strategy formed by a set of label-preserving transformations for the target dataset. The paper can compare LP-A3 with other adversarial data augmentation strategies in terms of their idea and performance, so we will know which approach is better in which scenarios.

__W2. [Insufficient experiment]__

1. The experiments are not extensive enough. Only the results on several tiny datasets are provided. The results on the large-scale ImageNet dataset will make this paper stronger. In supervised image classification tasks, the paper compares LP-A3 with RandAugment on MedMNIST. It would be better to compare LP-A3 with other Automated Data Augmentation methods, like AutoAugment, Adversarial AutoAugment, and sample-mixing augmentations, like Mixup, Cutmix, on the full dataset of popular classification benchmarks, e.g. CIFAR10 and CIFAR100. This helps to verify the performance gains compared with the state-of-the-art data augmentation schemes.

2. Prior works such as RandAugment run multiple times for the model and present the mean and standard deviation. Did the authors run multiple times for each model and what about presenting the mean and standard deviation?

__W3. [Adversarial training]__ According to lines 252-255 and Appendix B, it seems that LP-A3 uses an adversarial-attack-like procedure to perturb the input images.  Adversarial attack perturbs an image to alter the class label; oppositely, LP-A3 aims to preserve the class label. With the opposing objectives, can LP-A3 be integrated with adversarial training? Does LP-A3 affect the use of perturbation-based adversarial attacks to create adversarial samples in adversarial defense?

[1] Zhang et al., Adversarial AutoAugment. ICLR, 2020.

questions: __O1.__ It seems that LP-A3 relies on the representation learning network $F$ to compute the LPIPS distance $\phi(x)$ and softmax class probability $F(x; \theta)[y]$ in Eq. (4). During the early stages of training, when $F$ is not well trained, is it possible that it may generate incorrect training signals for computing the LPIPS distance and softmax class probability for the label-preserving constrain? How does the method deal with this problem especially for small and more complicated datasets?
limitations: Please see the above concerns.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new data augmentation method that does not require training an extra generative model but instead leverages the intermediate layer representations of the end-task model for generating data augmentations. The augmentation is built based on a representation learning principle that aims to preserve the minimum sufficient information of the labels. Experiments across different datasets are conducted to demonstrate the effectiveness of the proposed method.

After rebuttal:
Thanks for the authors' responses. The authors address some of my concerns so I raise my rate. 
strengths_and_weaknesses: Strengths

+ A novel, simple, and intuitive idea for data augmentation
+ The simple idea is very effective when implemented correctly
+ The paper is easy to follow, well-written, and inspiring. Moreover, the theoretical interpretation of the method is sound.
+ Results with various different tasks, including semi-supervised learning, noisy-label learning, and medical image classification, and datasets are provided in this manuscript to show the effectiveness of the proposed LP-A3.

Weakness
- How does the TCS select the hard positive data in Alg.1? I think it is necessary to provide the selection detail to make the manuscript self-contained.
- The experimental results in the manuscript are insufficient to show the effectiveness of LP-A3. In semi-supervised learning tasks, only one augmentation method, InfoMin, is tested on STL-10 dataset as a comparison. What are the backbone networks in the experiments in Table 2? Moreover, How’s the performance of LP-A3 on CIFRA10/100 in a supervised classification manner? Evaluation with other backbone networks rather than ResNet-18 and ResNet-50 is also important.
- How’s the performance of LP-A3 on the large-scale dataset, such as ImageNet
- I’m curious about the results in Fig.5. As the proposed LP-A3 is designed to generate the augmented X’ to preserve the minimum sufficient information of the label, why the performance of LP-A3 can drop when all data is selected for augmentation? The author stated that some data augmentations are useless to apply data augmentation, which did not convince me. 


questions: Please refer to the weaknesses.
limitations: -
review_title: null

Meta-review: metareview: This paper presents a new augmentation method (though it's similar to adversarial training) that significantly improves various benchmarks, methods, and tasks. Besides, the authors provide an information-theoretic ground for the proposed method. AC appreciate the technical contribution to the community. After rebuttal, the authors addressed most of the concerns. AC recommends accept.

AC also would like to suggest the authors comprehensively compare your method with adversarial training in revision.
confidence: Certain
Decision: Accept