Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new algorithm for data augmentation based on the same transformations used for autoaugment and follows-up papers. The algorithm considers policies that are composed of two sequential transformations selected from 16. Thus in total there are 256 different policies that can be selected. The probabilities of selecting those policies are latent variables that are estimated during training with a expectation maximization approach. In addition the algorithm can estimate also the probabilities of a policy conditioned to the given sample by normalizing the corresponding losses. Results show that the proposed algorithm outperforms previous approaches in most of the cases.
strength_and_weaknesses: \+ The proposed approach seems to generate more meaningful augmentations than previous approaches as shown in the experimental results

\- Different parts of the paper are disconnected and difficult to follow. See clarity for more details.

\- The differences between this work and adversarial autoaugment (advAA) seem minimal. As shown by theorem 2.1, with uniform unconditional probabilities and $\sigma \rightarrow \infty$ the proposed approach is advAA. As shown in table 3, the contribution of considering variable unconditional probabilities is quite low and in the order of the std. Not sure about the difference between $\sigma = 1$ and a high value. It would be interesting to see table 3.

\- Differences in classification accuracy are relatively small. How can we verify that those differences are not due to a larger batch sizes or other implementation details
clarity,_quality,_novelty_and_reproducibility: \- Clarity: the presentation of the paper should be improved in several aspects.
In the introduction I consider that the authors should not limit the presentation of previous methods to AutoAugment, even if the other approaches are mentioned in related work. I do not see many similarities between the proposed work and the Bayesian augmentations of Tran et al. (2017). Fig. 1 has a disconnect with the proposed formulation. For instance in Fig. 1 the E-step is applied on the multiplication of the conditional probabilities of the policies and the losses, while in Equ. 4 the conditional probabilities are multiplied by $P(y|o_z(x),\theta)$. In method, in the third line the authors mention the use of random augmentations, but then $\pi_z$ is used. In 2.1 first equation is presented as Bayes' rule, but in my understanding is just probability normalization. Algorithm 1 contains too much text and some definitions of variables are inaccurate. Equ. 3 seems a bit strange as it normalizes $h_z$ which is based on already normalized probabilities, thus twice exponentiation which might not be ideal for gradient propagation. 

\-Quality: the method seems to outperform previous approaches, however the differences are relatively small. It would be important to provide the code in order to make sure that the improvements are due to the algorithm and not different and better hyperparameters.

\- Novelty: as mentioned previously, the paper seems very similar to advAA, thus the novelty is limited. Authors should explicitly present the improvements with respect advAA in related work as well with experiments.

\- Reproducibility: authors provide the hyperparameters for the trained model. However, it would be important also to provide the code and make sure that the obtained results are due to only the algorithm and not different hyper-parameters than previous work.
summary_of_the_review: I consider this paper a valuable contribution for ICLR. However there are several points that the authors should improve for acceptance.

\- the writing the paper seems rushed, with some parts that are disconnected and other that should be improved and clarified (see clarity).
\- the differences with advAA seems minor. Authors should present and evaluate the differences with advAA.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a simple yet effective data augmentation method, which is called LatentAugment.
strength_and_weaknesses: Strength:

The proposed LatentAugment is straightforward and reasonable;

Compared to some previous methods, the proposed method is efficient in terms of computation cost;

Weakness:

The performance gain is relatively small. According to Table 1, UBS is also an efficient method, while the proposed method is only slightly better than UBS with more computation cost. Given the results, I'm curious about result of K=1 for Table 1, from which we may better compare the training cost and performance of proposed method and UBS.

Given that the results are slightly better than previous method, more experiments are needed. For example, instead of vanilla fully-supervised learning, how will the proposed method benefit some settings where data augmentations might be really important, including few-shot learning, transfer learning, meta learning, etc. 

Some presentation needs to be improved. For instance, Figure 2 looks incomplete because some bars seems to be missing without clear explanation. I would suggest the authors either compared all the related methods under all the same settings, or explain clearly about the figure/results.



clarity,_quality,_novelty_and_reproducibility: I believe the proposed method has good reproducibility and some novelty, while the quality of the paper needs to be improved. Specifically, I believe more experiments are needed, and the presentation needs to be polished.
summary_of_the_review: Although the paper propose a simple and effective method, I believe the paper needs improvement before getting accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the learn optimal policy for data augmentation on image classification task.  Learning the optimal augmentation policy, which is the latent variable, interesting research problem. The paper proposes a simple via EM-based method (call LatentAugment) to estimate the probability of optimal policy via conditional probability of the policy customized for given input and network model. It also provides the theorical analysis to show some exisitng methods (AdvAA and UBS) as the special cases. Experiments are on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets using different network architectures showing the encouraging results that it outperforms most of existing methods.  
strength_and_weaknesses: [Strength]

The paper is well-written and easy to follows. 

Novelty and with theoretical analysis provided.  

The model looks simple and computational efficiency, but would be good if with evidence supported.  

The results are encouraging with ablation study provided.  

[Weakness]

The claim on computational efficiency needs to be supported with theoretical or empirical results. For example, it would improve a paper if the authors can provide the computational comparison to other methods.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the idea is novel to me.  Code is provided in the supplementary (but I do not have chance to test the code). 
summary_of_the_review: Overall, I think the method is novel to me and most of claims are well-supported except one in the weakness. The paper is in the form can be published.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework for data augmentation that generalizes other existing data augmentation methods by incorporating Bayesian inference methods. Empirical performances on common benchmark datasets suggest that the LatentAugment methods outperforms other Adversarial benchmarks not only as measured by test accuracy in classification tasks, but also by training cost.
strength_and_weaknesses: Strengths:

1. The paper is well-organized, with each section clearly stating what it does. The contribution of the paper has been clearly outlined.

2. Experiments look sound and justify the main claim of the paper.

Weaknesses:

1. The level of novelty is relatively low. On a high-level, LatentAugment appears to be a straightforward application of the EM algorithm over the logarithmic losses defined with respect to conditional probability. The intuition behind the choice of loss function is also a bit vague. It would be helpful to explain why softmax is used and how the conditional probability relates to the goal of the LatentAugment algorithm.

2. Except for the classification task, no other task has been used to test the performance of LatentAugment. It would be helpful to incorporate other relevant tasks where data augmentation is helpful (e.g. inpainting, image generation) to see how LatentAugment is useful in general.

3. CIFAR, SVHN, and ImageNet are public datesets very commonly used across different image tasks. Has there been any tests on other less commonly used datasets where LatentAugment might be helpful? 
clarity,_quality,_novelty_and_reproducibility: The paper is written very clearly. However, the originality of the work should be more elaborated. 
summary_of_the_review: My personal evaluation would be a 5. I'd be happy to adjust my score, if the authors respond to the concerns accordingly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a data augmentation formulation which treats the optimal augmentation policy as a latent variable to be inferred. The empirical results for this approach improve results on a number of classification baselines. It also has a strong advantage that it is computationally cheap, and simple and straightforward to implement.

That said, reviews were borderline in part because the improvements were rather modest; suggestions from reviewers include exploring the use of data augmentation in other tasks beyond classification. There were also concerns from some reviewers regarding novelty relative to existing approaches such as AdvAA, as well as clarity of the paper and how the method connects to related work. This extends to understanding the overall formulation; the paper claims to be finding an optimal policy through EM, but it is not expanded on what notion of optimality is meant here, and in practice the objective function which is used is quite different from EM due to the use of the "softmin" in equation (3) rather than the posterior in equation (2). This is expanded upon below, in the summary of the AC-reviewer meeting.
justification_for_why_not_higher_score: Positive but modest empirical improvements; issues with clarity that lead to some confusion around the formulation.
Decision: Reject