LatentAugment: Dynamically Optimized Latent Probabilities of Data Augmentation
Authors: Koichi Kuriyama
Venue: Submitted to ICLR 2023 (2023)
Keywords: data augmentation
image classification
EM algorithm

Abstract: Although data augmentation is a powerful technique for improving the performance of image classification tasks, it is difficult to
identify the best augmentation policy. The optimal augmentation policy, which is the latent variable, cannot be directly observed. To address this problem, this study proposes \textit{LatentAugment}, which estimates the latent probability of optimal augmentation. The proposed method is appealing in that it can dynamically optimize the augmentation strategies for each input and model parameter in
learning iterations. Theoretical analysis shows that LatentAugment is a general model that includes other augmentation methods as special cases, and it is simple and computationally efficient in comparison with existing augmentation methods. Experimental results show that the proposed LatentAugment has higher test accuracy than previous augmentation methods on the CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. 


Under review as a conference paper at ICLR 2023
LATENTAUGMENT: DYNAMICALLY OPTIMIZED LA- TENT PROBABILITIES OF DATA AUGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Although data augmentation is a powerful technique for improving the perfor- mance of image classification tasks, it is difficult to identify the best augmentation policy. The optimal augmentation policy, which is the latent variable, cannot be directly observed. To address this problem, this study proposes LatentAugment which estimates the latent probability of optimal augmentation. The proposed method is appealing in that it can dynamically optimize the augmentation strate- gies for each input and model parameter in learning iterations. Theoretical anal- ysis shows that LatentAugment is general model that includes other augmen- tation methods as special cases, and it is simple and computationally efficient in comparison with existing augmentation methods. Experimental results show that the proposed LatentAugment has higher test accuracy than previous augmentation methods the CIFAR-10, CIFAR- 100, SVHN, and ImageNet datasets.
Data augmentation is widely used technique for generating additional data to improve the per- formance of computer vision tasks (Shorten & Khoshgoftaari 2019). Although data augmentation performs well in experimental studies, designing data augmentations requires human expertise with prior knowledge of the dataset, and it is often difficult to transfer the augmentation strategies across different datasets (Krizhevsky et al., 2012). Recent studies on data augmentation consider an au- tomated design process of searching for augmentation strategies from dataset. For example, Au- toAuument, proposed by [Cubuk et al. (2018), uses reinforcement learning to automatically explore data augmentation policies using smaller network models and reduced datasets. Although AutoAug- ment shows great improvement on image classification tasks of different datasets, requires thou- sands of GPU hours to search for augmentation strategies. Furthermore, the data augmentation operations optimized for reduced datasets using smaller network models may not be optimal for full datasets using larger network models.
To address this problem, this study proposes LatentAugment, which estimates the latent probability of the optimal augmentation customized to each input image and network model. There is doubt that an optimal augmentation policy exists for each input image using a specific network model. However, the optimal augmentation policy, which is a latent variable, cannot be directly observed. Although a latent variable itself cannot be observed we can estimate the probability of the latent variable being the optimal augmentation policy. LatentAugment applies Bayes' rule, to estimate the conditional probability of the augmentation policy, given the input data and network parameters.
Figure shows the concept of the proposed latent augmentation method. Following the Bayesian data augmentation proposed by Tran et al. (2017). Latent/ugmment uses the expectation- maximization (EM) algorithm update the model parameters. In the expectation (E)-step, the expectation of the weighted loss function is calculated using the conditional probability of the latent augmentation policies. In the maximization (M)-step, the expected loss function is minimized using the standard stochastic gradient descent. The conditional probabilities of the highest loss function with the augmentation policy were calculated using the loss function with the updated parameters and input data. The unconditional probabilities of the augmentation policies were generated using the moving average of the conditional probabilities. Note that the conditional probabilities of the latent augmentation policies dynamically optimized for the input and updated model parameters in the iterations of the EM algorithm.
The contribution of this study can be summarized follows:

---

Under review as a conference paper at ICLR 2023
Unconditional Probability of Augmentation Policies
Conditional Probubility of Highest Loss
1
hg
72
Moving Average
Pz
hg
:
hg
E-step
M-step
Loss with Augmentation
Policies
Weighted Loss
Minimizing Expected Loss
L
4
Dataset
L2
hz42
Σ hzCa
Lg
hgCs
:
Lx
hxCx
Figure I: An overview of the proposed LatentAugment. The loss functions with augmentation poli- cies are calculated using the imput data and the unconditional probability of augmentation policies. The model parameters are updated by the EM algorithm. In E-step. the expectation of the weighted loss function is calculated using the conditional probability of the highest loss. In M-step, the ex- pected loss function is minimized using the standard stochastic gradient descent. The conditional probabilities of the highest loss are calculated using the loss function with the updated parameters and input data. The unconditional probabilities of the augmentation policies are generated by the moving average of the conditional probability.
It provides theoretical model for LatentAugment. This study shows that LatentAugment can dynamically optimize the augmentation methods for each input and model parameter in the learning iterations by calculating the conditional probabilities of the latent augmentation policies. Furthermore. it shows that LatentAugment is a general augmentation model that includes other augmentation methods, such as Adversarial AutoAugment (Zhang et al., 2019) and incertainty-base sampling (Wu et al. 2020), as special cases.
LatentAugment is simple and computationally efficient. It does not require the augmen- tation policies be searched before training. Adversarial Auto/Augeett proposes the ap- plication a generative adversarial network (GAN) (Goodfellow et al.) 2014) to solve the maximization of the minimum loss function, which requires an additional training cost for the adversarial network. In contrast, the proposed Lagent can solve this problem using the simple stochastic gradient descent algorithm without an adversarial network.
Experimental results show that the proposed LatentAugment improve the test accu- racy for the CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. For example, a test accuracy of 08.72% was achieved with the Imada et al.) 2018) on CIFAR-10, 100 which significantly better performance compared to previous augmentation methods.
(Han et al. 2017 Ya-
2

---

Under review as a conference paper at ICLR 2023
1
RELATED WORKS
Several studies have been conducted on data augmentation methods in the literature on machine learning. [Shorten & Khoshgoftaar (2019) provided comprehensive review of image data augmen- tation. Recent studies have attempled to automatically identify data augmentation methods. Smart augmentation (Lemley et al. 2017) merges two more samples from the same class to improve the generalization of a target network. Auto/Augment (AA) (Cubuk et al.) 2018) applies a recurrent neural network (RNN) as sample controller to search for the best data augmentation policy using small proxy tasks of randomly drawn images from the training dataset. After identifying the best policy, fixed policies are applied to the training dataset. Population-based augmentation (PBA) (Ho) let al. 2019) generates dynamic augmentation policy schedules instead of a fixed augmentation pol- icy. RandAugment (RA) (Cubuk al., 2019) has a significantly reduced search space and allows training on the target task without a separate proxy task. Fast AutoAugment (Fast AA) {Lim et al.) 2019) determines the best augmentation policy using a more efficient search strategy based on den- sity matching. Faster AA (Hataya et al. (019) uses a differentiable policy search pipeline for data augmentation, which is much faster than previous methods. DADA (Li et al. (0200) also reduces the cost of policy search using differentiable optimization problem via Gunbel-Softmax, while DeepAA (Zeng et al. 2022) uses a multi-layer data augmentation pipeline.
Adversarial AutoAugment (AdvAA) (Zhang al. 2019) applies adversarial network to gen- crate data augmentation. While the training network minimizes the loss, the adversarial network maximizes training loss. Uncertainty- -asses Sampling (UBS) (Wu et al. 2020) generates data aug- mentation of the highest loss without an adversarial network. As shown in the next section, the loss functions of AdvAA and UBS can be regarded special cases of LatentAugment proposed in this study. MetaAugment (Zhou et al.] 2020) uses an additional augmentation policy network to mini- mize the weighted losses of augmented training images. DHA (Zhou et al., (021) uses super and child networks to achieve joint optimization of the data augmentation policy, hyyer-paramette and architecture. In contrast, the proposed LatentAugment does not require any additional network for searching the augmentation policy.
The best augmentation policies are the latent variables that cannot be observed. The expectation- maximization (EM) algorithm was proposed to analyze latent variables (Dempster et al. 1977, McLachlan & Krishnan 2007: Ng et al., 2012). The EM algorithm estimates parameters using an iterative process of expectation and minimization of the loss function. However, when the dataset is large, might be difficult to calculate the expectation or minimization of the full dataset. To ad- dress the difficulty of working with large datasets, some approaches have been proposed, including the generalized EM algorithm (Dempster et al. 1977). Monte Carlo EM algorithm (Wei & Tanner, 1990 Tanner 199T). stochastic EM algorithm (Nielsen) 2000), and generalized Monte Carlo EM Tran et al. 2017). For application to data augmentation, Bayesian data augmentation (Tran et al. 2017) estimates the parameters using the EM algorithm generate data augmentation using the Bayesian approach. Bayesian data augmentation requires an adversarial network, whereas Laten- tAugment does not use an adversarial network. Nevertheless, Bayesian data augmentation is mostly related to this study in the application of the EM algorithm to data augmentation.
METHOD
Consider a classification task with C categories for the N training data points X (21, ,NN) } and labels Y = {v1, V/, ... yN Let P (yy x. 0) denote the predicted probabil- ity of the output y. given the input x and the parameter O. Consider that each input transformed us- ing random data augmentation. Let S = (1. S} be the set of augmentation policies, and = (xx 0) be the optimal augmentation policy for the input given the parameter. We cannot directly observe the optimal policy; therefore, 2² (z, 0) is the latent variable. Let π₃ be the unconditional probability that the augmentation policy 2 is applied to the input data. The loss function using the augmented
data can be written L (@)- -x,
Σ T&P (y) og (z). where (x) denotes A
the augmented data using the augmentation policy 2, O (0, and ( ..
3

---

Under review as a conference paper at ICLR 2023
2.1 GENERALIZED EM ALGORITHM
The loss function with the latent variable can be minimized using the (EM) algorithm. The EM algorithm an iterative procedure used to compute the maximum like- lihood estimate in the presence of latent variables (Ng et al. 2012). In the E-step, the expected loss function calculated. In the M-step, the parameter is updated by minimizing the expected loss function. Let ( {0(e) ( be the parameter and the unconditional probability iteration and ( (ax ( 8) be the conditional probability of the augmentation probability of policy 2 for the individual data point æ given the label y at iteration Applying Bayes' rule, we can calculate the conditional probability (MeLachlan & Krishnan, 2007):
(1) P O2 (x) 0(0))
11 (z, 0
x ( O= (z) 8(0)
Using (0) as shown in Ng et al. 2012), the expected loss function & (0)@()) can be written as:
& (()01)) (E(,)))((X,))
Σ h!o log (TTz + >
h log (P 0g (z) (z).
(1)
2E8
In the M-step, the parameter 0 and the unconditional probability were estimated by minimiz- ing the expected loss function given the conditional probability h If solving the minimization problem of & (o10()) proves difficult, the generalized EM algorithm proposed by Dempster et al. 1977 can be used to estimate ((+1), where & (0(++)0019) & (0l@(1)..
Calculation of & (0))()) requires the expectation of possible augmentation policies. When the number of augmentation policies S is large, the computational burden of & (0j@(!) cannot be ne- glected. Alternatively, the subset K which randomly drawn from the full set'S, of the augmentation policies, can be used. Then, the conditional probability can be written as (1
n2 Pyyl 0z (I), 0(1))
K
Σ ZEK
19 P( (x), 0(0)
As shown in the Appendix AA.I the subset IK generated using simple random draws from the full set S, the expected loss function using the subset is equal to that obtained using the full set.
2.2 LATENT AUGMENTATION POLICY
The generalized EM algorithm can estimate the parameter by minimizing the expected loss function using latent variables. However, this may cause an overfitting problem. Following AdvAA, an augmentation policy is applied to maximize the loss function using harder augmentation policy.
Let c. log (t) P P(y oz (z))) be the the contribution to the loss function for input (x, y). using the augmentation policy z at iteration Consider the conditional probability of the latent augmentation policy Ti such that the augmentation policy has the highest loss in the set of K, using the softmin function:
exp(-11!)) /0)
7 = Pr(C() 2 (()) Vk = s (1) Vk EK]
(3)
ΣLek exp(-hi /0)
where a is the inverse scale parameter. Note that, from the definition of L,, the probability of minimum h2 is equal the one of maximum Cg. Thus, the softmin function is related to the goal of the LatentAugment, maximization of minimum loss. The proposed LatentAugment can be implemented by the EM algorithm with (())) In the E-step, the LatentAugment calculates the
4

---

Under review as a conference paper at ICLR 2023
Algorithm LatentAugment
Input: (); dataset
Require: B: the number of mini-batch, S: the full set of augmentation policies, S: the size of augmentation policies, and a: the inverse scale.
Initialize: nT = 1/S, for 2 = (1.....S). Initialize the network parameter ((0))
for / - 1, 1,..., B do
Randomly draw the subset K from
Calculate R. using equation 3
E-step: Calculate & using equation
M-step: Update the parameter ((() and (1) using equation 5 end for Return: () and x:
expected loss function weighted by the probability of the minimum conditional probability n!a) instead of (
E (01010)) - (xx,)))(XX,))RR... Exas
Σ T log (T=) +
Σ 7,0 log (P
(x),
(4)
LEE
SEK
In the M-step, the parameter is updated by minimizing the expected loss function with fixed h
((++1)= = 0 - -VVo (010(1)) ((+1) = Moving A Avergge of ((x.))((Xnn h
(5)
((a.)) Σ
(0)
zEK
where (XB, Yn) is the mini-batch of the input data. This process is iterated until convergence is achieved. The estimation procedure with LatentAugment is summarized as Algorithm
2.3 ADVANTAGES OF THE LATENTAUGMENT
The proposed Latent-ugment has following advantages over the existing augmentation methods:
1.
The weighted augmentation policies are optimized for the individual input. Most cent studies, such as AA, use randomly drawn policies; however, they do not apply policies that are appropriate for each input data. On the other hand, LagentAugment utilizes domly drawn policies customized for each input by calculating conditional probabilities for the given input.
2. It provides closed-form solution for the probability of optimal augment polices. La- tentAugment can estimate the unconditional probability (T₂z) of optimal augment policies using closed-form solution 5 of the loss minimization. Thus, LatentAugment does not involve the additional cost of searching for these policies.
3. It simple and computationally efficient. AdvAA proposed the use of GAN to solve for the maximization of the minimum loss function, which requires additional training costs for the adversarial network. In contrast, the proposed LatentAugment can solve the max max-min problem using the conditional probability (t) of the highest loss without an ad- versarial network. Lagent can be solved using simple stochastic gradient descent
Table 1: Comparing the training cost and test accuracy between proposed LatentAugment (LA) with RandAugment (RA (Cubuk et al.) (209)), Adversarial AutoAugment (AdvAA (Zhang al. 2019). and Uncertainty-Ban -Baee Sampling (UBS (Wu et al. 2020)) using the Wide-ResNet 28-10 10 model on CIFAR-10. Training cost of required GPU hours is reported relative to RA.
RA
AdvAA
UBS
LA (K 2)
LA (K = 1)
LA (K = 6)
Training cost
1
8
1.5
1.9
3.3
4.7
Test accuracy
95.8%
98.1%
97.9%
98.0%
98.2%
98.3%
5

---

Under review as a conference paper at ICLR 2023
algorithm. Table I compares the training cost of required GPU hours between proposed LatentAugment and other methods.
4. It a general model which includes other augmentation methods. The proposed La- tugtAugment is general augmentation method that includes other methods such as UBS and AdvAA. Following theorem shows that the UBS with a single data point and AdvAA could be considered to be special cases of LatentAugment:
Theorem 2.1. (Special Case of LatentAugment), Assume that the unconditional probabilities for all augmentation policies are the same (T, 1z), If the inverse scale parameter a - 0, the gradient of expected loss function of the LatentAugment can be equal the one of UBS. If oc, the gradient of expected loss function of the adversarial network with LatentAugment is equal to the one of AdvAA.
The proof can be found in Appendix A.2 Note that the the gradient of expected loss function of Lugmett with a + oo can be equivalent to the one of AdvAA. However, means that maximize minimum loss without additional network, thus the advantages in efficiency of LatentAugment will be lost.
LatentAugment
3 EXPERIMENTS
3.1 EXPERIMENT SETTING
This section describes the experiments investigating the performance of the proposed LatentAug- ment using the CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton 2009), SVHN (Netzer et al., 2011]. and ImageNet (Russakovsky et al. 2015) datasets. In these experiments, the network mod- els are Resnet-50 (He al.]62016) Wide-ResNNe 40-2 and Wide-ResNet 28-10 (Zagoruyko & Ko- Shake- 26 2x32d, 26 2x96d, and 26 2x1 12d (Gastaldi 2017). ), and PyramidNet with ShokeDrop with depth of 272 and alpha of 200 (Han et al.) [017]] Yamada et al.) (018). Tinle in the Appendix | provides the hyperparameters. All the hyperparameters of the network models the same those used in AA (Cubuk al. 2018), Fast AA (Lim et al.) 2019), and PBA (Ho et al. 2019). A cosine learning decay with one annealing cycle was applied to all models.
As proposed by Wu Contrast, Cutout, Equalize, Invert, Mixup, Posterize, Rotate, Sharpness, ShearX, ShearY, Solarize, TranslateX, and Translate [Shorten & Khoshgoftaar (2019) described these transformations, As in AA, augmentation policies are generated by the combination of the two transformations. The size of the policy set was S 16 16 256. Following AA, we set the magnitude of all the transform operations in moderate range. All values of the magnitude range are same as AA.
(2020), we use 16 transformations: AutoContrast, Brightness, Color,
The unconditional probability (π₂) was initialized as 1/5. The range of the magnitude of each transformation was discretized into 10, which were randomly drawn from the uniform distribution. The unconditional probability (Tg) was calculated using the moving average. The length of the moving average was fixed at 10 iterations in this experiment. It was difficult to estimate the expected loss function using the full set S of augmentation policies because of the computational burden of the large size S 256. Alternatively, subset K. could be used which is randomly drawn from the full set S. In this experiment, the subset size of the augmentation policies was set to six (K = 6). The inverse scale parameter size, and inverse scale are discussed in later sections of the paper.
one. The effects of the unconditional probability, subset
3.2 CIFAR-10 RESULTS
The CIFAR-10 dataset has a total of 60,000 images, including 50,000 for the training set and 10,000 for the test set. Each image with size of 32 x32 belongs to one of the 10 classes. The baseline is trained with standard data augmentation using horizontal flips with 50% probability, acro-padding, and random crops. The proposed LatentAugment first applies the baseline preprocessing, then ap- plies LatentAugment using six policies randomly drawn from 256 policies, and finally applies the Cutout (De & Taylor 2017) the Cutmix (Yun et al. 2019).
Table 2 shows the results of the test accuracy for different network models using the CIFAR- 10 dataset. For all models, the proposed Lugment method achieved better performance com-

---

Under review as a conference paper at ICLR 2023
Table 2: Test accuracy (%) on CIFAR-IO, 10, CIFAR-100, SVHN, and ImageNet. All experiments in this study replicate the results of Baseline and AutoAugment (AA) (Cubuk et al. 2018). Adversarial AutoAAgment (AdvAA) (Zhang et al. 2019).
Uncertainty-Bact Based Sampling (UBS) (Wu al. 2020).
and MetaAuument (MA) (Zhou et al. 2020). On the proposed Lagentt (LA), averages of five runs are reported. Network models are Wide-ResNet 40-2 and ReseeResNet 28-10 10 (Zagoruyko &
Komodaki 2016), Shake-Shake 26 2x32d, 26 2x96d, and 26 2x112d (Gastaldi, (017). PyramidNet with Drop (Han et al. 2017 for more details.
Yamada et al. 2018) and Resnet-50 (He et al. 2016). See text
Dataset
Model
Baseline
AA
AdvAA
UBS
MA
LA
CIFAR-10
WRN40-2
94.70
96.30
96.79
97.27=0.09
WRN28-10
96.13
97.32
98.10
97.89
97.76
98.25=0.08
S-S (26 2x32d)
96.45
97.53
97.64
97. 97.68+0.03
S-S (26 2x96d)
97.14
98.01
98.15
98.27
98.29
98,42=0.02
S-S (26 2x 2x11
97.18
98.11
98.22
98.28
98.44+0.00
PyramidNet
97.33
98.52
98.64
98.66
98.57
98.72=0.02
CIFAR-100
WRN40-2
74.00
79.30
80.60
80.90=0.15
WRN28- 10
81.20
82.91
84.51
84.54
83.79
84.98=0.12
S-S (26 2x96d)
82.95
85.72
85,90
85.97
85.88=0.
SVHN
WRN28-10
98.50
98.93
98.9620.01
ImageNet
ResNet-50 (Top 1)
75.30
77.63
79.40
79.74
80.02+0.10
ResNet-50 (Top
92.20
93.82
94.47
94.64
94.88=0.05
pared to existing augmentation methods. For example, LagentAugment achieved an improvement of 0. 0.15% and 0.36% compared to AdvAA and UBS on Wide-ResNet 28-10 model, respectively. The test accuracy of the proposed LatentAugment using PyramidNet+ was 0.08% and 0.06% better than that of AdvAA and UBS, respectively. To compare with the AA, we tested the proposed model using the same transformations AA, which uses the policy set with SamplePairing (Inoue 2018), instead of Mixup (Zhang et al. 2017). and finally applied the Cutout, instead of Cutmix. The test accuracies for the of the Augment using the same transformation as AA were 96.91#0.055 and +0.0%% respectively (Table 5). Thus, the proposed method outperforms AA, even when neither Mixup Cutmix were used. Adversarial AutoAAgment (AdvAA) also applies the same transformations as AA, although the subset size of AdvAA 8. To compare with AdvAA, we tested the model us- ing the same subset size. The test accuracy of Wide-ResNet 28-10 of the LatentAugment using the same transformations as AA with subset size K 8 98. 16*0.07% Thus, the proposed method is marginally better than AdvAA even when the same transformations and subset sizes are used. See table 5 and in Appendix.
was 98.72 %, which
40-2 model and Wide-ResNet 28-10 10
3.3 C1FAR-100 RESULTS
The CIFAR-100 dataset also has a total of 60,000 images, including 50,000 for the training set and 10,000 for the test set. The number of categories is 100. The procedure of the baseline and LatentAugment the same as that of CIFAR-10.
As for CIFAR-10, the proposed LagentAugment indicated better accuracy than existing augmenta- tion methods except Shake-Shake (26 2x96d), in which the test accuracy of LatentAugment was slightly lower than that of AdvAA and MetaAugment (MA).
3.4 SVHN RESULTS
The SVHN dataset has 73,257 digit images for the core training set, 531,131 for the additional training set, and 26,032 for the test set. In this experiment, both core and additional training sets used. The number of categories is 10. The baseline The proposed method first applies LatentAugment using six policies, randomly drawn from 256 policies, then normalizes the data, and finally applies the cutout with region size of 20 x20 pixels,
trained using the normalizing data.
7

---

Under review as a conference paper at ICLR 2023
LA
AdvAA
UBS
98.4
98.2
98.0
97.8
97.6
97.4
8
The size the subset (K)
Figure 2: The accuracies with the different size of the subset (K). It shows the test accuracies of LatentAugmmnt (LA), Uncertainty-Basee Based Sampling (UBS), Adversarial AutoAAgment (AdvAA) with the different size of the subset (K) using the Wide-ResNet 28-10 model on CIFAR-10. This figure replicates the results of UBS from Wu et al. (020)) and AdvAA from Zhang et al. (2019).
following the method proposed by DeVeis Taylor (007)) LatentAugment using Wide-ResNet 28-10 10 achieves 0.03% improvement compared to AA.
3.5 IMAGENET RESULTS
The ImageNet dataset has more than 1.2 million training images, 50,000 validation images, and 100,000 test images. The number of categories is 1,000. Following the AA, baseline augmentation uses the standard Inception-style pre-processing. including horizontal flips with 50% probability and random distortions of colors. The proposed LatentAugment first applies the baseline preprocessing, then applies LatentAugment using six policies randomly drawn from 256 policies, and finally applies the Cutmix. The proposed method outperformed previous augmentation studies.
3.6 CHOICE OF THE SUBSET SIZE
This experiment used subset size of K 6. To determine the optimal size of the subset, this study used the Wide-ResNee 28-10 to evaluate the performance of the proposed LatentAugment with different K, where K E {2, 6, Figure 2 suggests that the test accuracy of the model rapidly increases up to K 6. However, no significant improvement was observed when K was 8. In contrast, the computational cost increases with K. Therefore, after comparing the computational cost and performance, all the experiments in this study used K 6 for LatentAugment.
Figure also shows the results of AdvAA and UBS. AdvAA uses instances of K E {2, 4,8, 16, 32) for each input example, augmented by adversarial policies. The study of UBS reports the experimen- tal results using K = 4 with a single data point and K with four data points for training. This figure suggests that the proposed LatentAugment is more efficient than AdvAA and UBS, because LatentAugment with K = outperforms AdvAA and UBS with K 8.
3.7 THE EFFECTS OF THE INVERSE SCALE
LatentAugment requires determining the inverse scale parameter a which is assumed to have a value of 1 in the previous section. This section considers the effects of the inverse scale using different values. Figure 3 shows the test accuracy of LagentAugment with different inverse scale values using the Wide-ResNet 40-2 model CIFAR-10. This suggests that the test accuracy is maximum 1, although the effect of the inverse scale is weak except for - 0.
3.8 THE EFFECTS OF THE UNCONDITIONAL PROBABILITY
LatentAugment estimates the unconditional probabilities (T₂)) as well as the network parameters (0). As shown in Theorem 2.1 if the unconditional probabilities are fixed at the same values, the
8

---

Under review as a conference paper at ICLR 2023
97.4
97.3
97.2
97.1
97.0
96.9
96.7
96.6
96.5
0
0.01
0.1
10
100
The inverse scale parameter
Figure 3: The test accuracies with the different inverse scale parameter (a). It shows the test accu- racies of LatentAugment with the different inverse scale values using the Wide-ResNet 40-2 model on CIFAR-10.
Table 3:
The test accuracies using fixed unfixed unconditional probability with the different inverse scale parameters, Averages of five runs are reported.
Inverse scale parameter (r)
Diff.
0
Fixed Wa
(a) 96.71=0.10
(b) 97.1940.00
0.48
Unfixed m.
(c) 96.8120.10
(d) 97.2740.09
0.47
Diff.
0.10
0.08
derivative of LatentAugment can be reduced to that of adversarial AA or UBS. Table shows the effects of the unconditional probability in LatentAugment using the Wide-ResNet 40-2 model on C0.AR-10. The cell of (a), where the unconditional probabilities (Tz) are fixed and the inverse scale parameter (σ) is contrast, the cell of (d), where T. can be estimated and = 1, is the test accuracy of the proposed LatentAugment, which allows variable π. and multiple data points for the expectation of the loss function. This table suggests that an unfixed πy slightly improve the test accuracy over fixed Ts. However, the effect on test accuracy with an fixed T. is weaker than the effect of set to zero. Thus, for the better performance of the proposed LatentAugment, using multiple data points for the expected value of the loss function weighted by the conditional probability of the highest loss, has a more significant effect the performance than using the unfixed unconditional probability.
0, is equivalent UBS with a single data point of the highest loss. In
CONCLUSIONS
This study introduces LatentAugment, which estimates the probability of the latent augmentation customized to each input image and network model. The proposed method is appealing in that it can dynamically optimize the augmentation methods for each input and model parameter in learning iterations. As shown in the theoretical analysis, LatentAugment a general model that includes AdvAA and UBS as special cases. Furthermore, the proposed method is simple and computationally efficient in comparison with the existing methods, which requires generative adversarial network. Experimental results show that the proposed LatentAuummet has better performance than previous augmentation methods on the CIFAR-10, CIFAR-I00, SVHN, and ImageNet datasets. Finally, an open question remains in the robustness of LatentAugment using the EM algorithm, which typically converges to local optimum. While we checked the stability of the test accuracy with five runs using different random seeds, the issue of convergence of LatentAugment is important theme for future research. An application to the object detection, image generation, and text recognition using LatentAugment also an interesting topic. We leave such directions to future work.

---

Under review as a conference paper at ICLR 2023
5 REPRODUCIBILITY STATEMENT
Pytorch code of the experiments in this paper can be downloaded from GitHub (https:// github. com/ xxxx xxxxxx).
REFERENCES
Murray Aitkin and Init Aitkin. A hybrid em/gausss-newton algorithm for maximum likelihood in mixture distributions. Statistics and Computing, 6(2):127-130, 1996.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning augmentation policies from data. In CVPR, 2018.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical data augmentation with no separate search. arXiv preprint, arXiv:I909.1377 2019.
Arthur Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):
1-22, 1977.
Terrance Vries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint, arXiv:1708.0455: 2017.
Xavier Gastaldi. Shake-shake regularization. In CoRR, abs/l 705.07785, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 27, 2014.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In CVPR, pp- 5927-5935, 2017.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster AutoAugment: Learning augmentation strategies using backpropagation. arXiv preprint, arXiv: 1.06987, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Daniel Ho, Eric Liang, Ion Stoica, Picter Abbecl, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. In ICML, 2019.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv preprint, arXiv:1801.02929, 2018
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. NIPS, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient-base learning applied to document recognition. In Proceedings of the IEEE, volume 86, pp. 2278-2324. Icee, 1998.
Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran. Smart augmentation learning an optimal data augmentation strategy. IEEE Access, 1188885869, 2017.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In European Conference on Computer Vision, pp. 580-595. Springer, 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast AutoAugment. In NeurIPS, 2019.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey J McLachlan and Thriyambakam Krishnan. The EM algorithm and extensions, volume 382. John Wiley & Sons, 2007.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Shu Kay Ng. Thriyambakam Krishnan, and Geoffrey J McLachlan. The EM algorithm. In Handbook of computational statisties, pp. 139-172. Springer, 2012.
Søren Feodor Nielsen. The stochastic EM algorithm: estimation and asymptotic results. Bernoulli, pp. 457-489, 2000.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. In IJCV, 2015.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1): 2019.
Martin Tanner. Tools for statistical inference: observed data and data augmentation methods, volume 67 of Lecture Notes in Statistics. Springer Science & Business Media, 1991.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. bayesian data augmenta- tion approach for learning deep models. arXiv preprint arXiv: 10564, 2017.
Greg CG Wei and Martin A Tanner. A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms. Journal of the American statistical Association, 85 (41 (411):699-704, 1990.
Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. In ICML, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 1 2017.
Yoshihiro Yamada, Masakazu Iwamura, and Koichi Kisei. Shakedrop regularization. In CoRR, abs/1802.02375. 2018
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6033 2019.
Sergey Zagoruyko and Nikos Komodaki. Wide residual networks. British Machine Vision Confer- ence, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empiri- cal risk minimization. arXiv preprint arXiv: 2017.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial AutoAugment. In ICLR, 2019.
Yu Zheng, Zhi Zhang, Shen Yan, and Mi Zhang. Deep AutoAugment. In ICLR, 2022.
Fengwei Zhou, Jiawei Li, Chuanlong Xie, Fei Chen, Lanqing Hong, Rui Sun, and Zhen- guo Li. Metaaugment: Sample-aware data augmentation policy learning. arXiv preprint arXiv: 12076, 2020. doi: 10.48500ARXW.301222077 URL htttps://axxiv.ottt abs/ fos/2013 2012.022100000 12076
Kaichen Zhou, Lanqing Hong, Shoukang Hu, Fengwei Zhou, Binxin Ru, Jiashi Feng, and Zhen- guo Li. Dha: End-to-end joint optimization of data augmentation policy, hapereprr and architecture. arXiv preprint
2021. doi: 10.4850/APXXV2210005766 URL
https://arra www.xvv.ogg/eae... http:/bb//2119......

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1
RANDOMLY DRAWN SUBSET OF THE AUGMENTATION POLICIES
Let 0₂ be the probability that policy z can be drawn. The conditional probability with d₂ can be written as:
(4) P (x), O(0)
E O 02 (4)) 0(0)
ZE3
The expected loss function using the randomly drawn subset given 6g is
& (0)10(") .8)) ((x,)))(x,)) Σ
Syh log (T!! P ( oz (z).
=ES
T (t) P (y) (x), 0(1)
-(((
log P (x) (1).
Σ SEB
smm=
(1) P (y 02 (x), 00e))
Assume that the policies of the subset are drawn using simple random draws: 52 6, v> S. Under this assumption, the expected loss function using a randomly drawn subset is equal to the expected loss function using the full set:
ôT P ( Oz (x), 019)
& (0l0 8) = -B(x.))((X)))
log x P (x). (0)00))
Σ
(8) Pyylo oz (I),
ZE3
SEB
A.2 PROOF OF THEOREM 2.1 (SPECIAL CASE OF LATENTAUGMENT)
A.2.1
UNCERTAINTY-BASED SAMPLING (UBS)
The loss function of UBS is
LUBS = (xx.))((x,)) max [-log log (plyyo (x)
ZEk
If the inverse scale + 0, the conditional probability h can be approximated by the indicator
function:
hg [he S hy, Vr E K],
lim h
lim
exp
o
Σ
exp
foc
rEK
where I [:] is the indicator function such that I if h2 hr. Vr E K, and I = 00, otherwise.
If m= 1/S for all and [ (y) oz (z), LatentAugment
in the LatentAugment, h. - T [hg h,, Vr KK] (x), (x), Vr K]. Therefore, the expected loss function of the P (ylor
(010
Σ [ (v)o (x), 000) < SP (p(y)or(s),00))) Vr E K log (P (v(llo oz (z) + log (1/8)
LEEK
& {(010()) e=gin)
(x,,)) max zEK
llog log (p (ylo= (x)) ((0)) log (1/S).
Note that the first term the same as the loss function of the uncertainty- based sampling evaluated at
0 g(t), while the second term is constant. Therefore, Vo & (0)0!))
(=000) Va Lynslo=00>
if æ 1/5 for all and - O.
12

---

Under review as a conference paper at ICLR 2023
A.2.2 ADVERSARIAL AUTOAUGMENT (ADVAA)
The loss function of AdvAA is
AAduAA (
log (P (P)
K
FER
where (S. ju) is the adversarial network for the set of augmentation policies, with parameter μ.- a - oc in the LatentAugment, h - 1/K. Assume मN = 1/S for all 2 in LatentAugment. Then, the loss function of the adversarial network with Latent-ugment is equal to the loss function of AdvAA plus constant:
ER A(S.a) [ (0101:)] (X,)
log (P (z)
log (1/S).
Therefore, VoBK~A(3,))
(01@(1)) + VoLAdrAs, 1/S for all 2 and o - co.
A.3 COMPUTER RESOURCES
We train the models with the LatnetAugment using computers with NVIDIA RTX 2080Ti GPUs and 64 GB memory.
A.4 HYPERPARAMETERS
Table 4: Hyperparameters for the experiment. LR represents the learning rate, whereas WD repre- sents the weight decay.
Dataset
Model
LR
WD
Batch Size
Epoch
CIFAR-10
400t--22
0.1
0.0002
128
200
CIFAR-10
Wide-ResNet-28-10
0.1
0.0005
128
200
CIFAR-10
Shake-Shake (26 2x32d)
0.01
0.001
128
1800
CIFAR-10
Shake-Shake (26 2x96d)
0.01
0.001
128
1800
CIFAR-10
Shake-Shale (26 2x 12d)
0.01
0.001
128
1800
CIFAR-10
PyramidNet+Shakcl
0.05
5E-05
64
1800
CIFAR- 100
Wide-ResNett40-22
0.1
0.0002
128
200
CIFAR- 100
- 10
0.1
0.0005
128
200
CIFAR-100 100
Shake-Shake (26 2x96d)
0.01
0.0025
128
1800
SVHN
Wide-ResNet-28-
0.005
0.001
128
160
ImageNet
Resnet-50
0.1
0.0001
256
270
A.5 TEST ACCURACIES USING THE SAME TRANSFORMATIONS AS AA AND ADVAA.
This section provides comparison between proposed LatentAugmentt (LA) and AutoAugment (AA) or Adversarial AutoAugment (AdvAA) using same subset size and transformations. To compare with the AA, we tested the proposed model using the same transformations as AA, which uses the policy set with SampleParring (Inoue] 2018), instead of Mixup (Zhang et al. (017), and finally applied the Cutout, instead of Cutmix. Table 5 shows the test accuracies for the Wide-ResNet 40- 2 model and Wide-ResNes 28-10 of the LA using the same transformation
AA. This table also provides the result of UBS using same transformations as AA, reported by Wu et al. 2020).
AdvAA applies the same transformations AA, although the subset size of AdvAA is 8. To com- pare with AdvAA, we tested the model using the same subset size. T the test accuracy of Nede-ResNet 28- 18-00 of the LA using the same transformations as AA with subset size K = 8.
13

---

Under review as a conference paper at ICLR 2023
Test accuracies (%) CIFAR-10 using the transformations AutoAugment (AA). On the proposed LatentAugment (LA), averages of five runs are reported.
Table 5:
Model
AA
UBS
LA (K = 6)
WRN40-2
96.30
96.91 97.75
WRN28-10
97.32
98.01+00.05
Table 6: Test accuracies (%) on CIFAR-10 using the same subset size and transformations as Ad- versarial AutoAugment (AdvAA). On the proposed LatentAugment (LA), an average of five runs is reported.
Model
AdvAA
LA = 8)
WRN28-10
98.10
98.16:10.07
A.6 TRANSFERABILITY ACROSS DATASETS AND ARCHITECTURES
This section evaluates transferability with LatentAugment across different datasets and model ar- chitectures. We first take snapshot of the unconditional probability л. of ResNet-50 ImageNet using LA, and then apply the fixed n. to train the models of Wide-ResNet 40-2 on CIFAR-10 or CIFAR-100 100 using LA. Table 7 provides the experimental result of transferability. It suggests that LA with policy transfer has still good performance.
Table 7: The test accuracies of the transfer the unconditional probability of augmentation policies.
Dataset
Baseline
LA (direct)
LA (policy transfer)
CIFAR-10
94.7
97.27=00.09
97.21=00.09 80.77::0.14
CIFAR-100
74.0
10
A.7
ADDITIONAL EXPERIMENTS USING OTHER DATASETS
This section provides the results of additional experiments using datasets of MNIST (LeCun et al.
1998), Fashion MNIST (Xiao et al. 2017), and Oxford flowers102 (Nilsback Zisserman, 2008).
A.7.1 MNIST
The MNIST is large database of handwritten digits. has total of 70,000 images, including 60,000 for the training set and 10,000 for the test set. Each example is 28x28 grayscale image, associated with a label from classes. The baseline trained with standard data augmentation using zero-padding, and random crops. The proposed LatentAugment first applies the baseline preprocessing, then applies LatentAugment using six policies randomly drawn from 256 policies, and finally applies the Cutout. We use hyperparameters of WRN40-2 on CIFAR-10 in Table
A.7.2 FASHION MNIST
The Fashion MNIST is dataset of Zalando's article images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. The baseline is trained with standard data augmentation using hori- zontal flips with 50% probability, sero-padding, and random crops. The proposed LatentAugment first applies the baseline preprocessing, then applies LatentAugment using six policies randomly drawn from 256 policies, and finally applies the Cutmix. We use hyperparameters of WRN40-2 on CIFAR-18 10 in Table
14

---

Under review as a conference paper at ICLR 2023
A.7.3 OXFORD FLOWERS102
Oxford 102 Flower is image classification dataset consisting of 102 flower categories. The flow- ers chosen to be flower commonly occurring in the United Kingdom. Each class consists of be- tween 40 and 258 images. Baseline augmentation uses the standard Inception-style pre-processing, including horizontal flips with 50% probability and random distortions of colors. The proposed LatentAugment first applies the baseline preprocessing. then applies LatentAugment using six poli- cies randomly drawn from 256 policies, and finally applies the Cutmix. We use hyperparameters of ResNet-50 on ImageNet in Table
Table The test accuracies of the additional experiments using MNIST (LeCun al. 1998). Fash- ion MNIST (Xiao et al., 2017), and Oxford flowers 102 (Nilsback & Zisserman 2008
Dataset MNIST
Model
Baseline
LA
WRN40-2
99.66
99.77=0.02
Fashion MNIST
WRN40-2
94.54
96.04=0.06
Flowers
ResNet-50
95.70
98.19
A.8 CONVERGENCE OF LOss FUNCTIONS
Convergence of the EM algorithm usually defined as a sufficiently small change in the loss func- tion (Attinn & Aitkin 1996). To confirm the convergence, Figure shows the loss functions us- ing the network models Wide-ResNet 40-2, Wide-ResNet 28- 28-10, Pyramid, and Shake-Shake on CIFAR-10 10 and 100. This figure indicates the convergence of an estimation using the EM algorithm.
Training Loss
1.5
Training Loss
1.5
Training Loss
Validation Loss
Validation Loss
Validation Loss
10086
1.0
1.0
0.5
0.
0.0
00
100
450
Epoch Number
Epoch Number
Epoch Number
WRN40K2 CIFAR-10
(b) WRNZ8K1O CIFAR-10
(c) Pyramid CIFAR-10
4.0
3.0
Training Loss
Training Loss
3.0
Training Loss
2.5
Validation Loss
2.5
Validation Loss
2.5
Lass
1.0
6.6
0.5
0.5
0.0
09
00
100
450
Epoch Number
Epoch Number
Epoch
Number
(i) Shake-Sbake (26 2x96d)
(d) WRN40x2 CIFAR-100
WRN28x10 CIFAR-100
CIFAR-100
Figure 4: The loss functions of the different network models on CIFAR- 10 and CIFAR-100.
15


Reviews:
[{"review_id": "xvRNCU0neix", "reviewer": "Reviewer_LaAK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new algorithm for data augmentation based on the same transformations used for autoaugment and follows-up papers. The algorithm considers policies that are composed of two sequential transformations selected from 16. Thus in total there are 256 different policies that can be selected. The probabilities of selecting those policies are latent variables that are estimated during training with a expectation maximization approach. In addition the algorithm can estimate also the probabilities of a policy conditioned to the given sample by normalizing the corresponding losses. Results show that the proposed algorithm outperforms previous approaches in most of the cases.", "summary_of_the_review": "I consider this paper a valuable contribution for ICLR. However there are several points that the authors should improve for acceptance.\n\n\\- the writing the paper seems rushed, with some parts that are disconnected and other that should be improved and clarified (see clarity).\n\\- the differences with advAA seems minor. Authors should present and evaluate the differences with advAA.\n\n", "strength_and_weaknesses": "\\+ The proposed approach seems to generate more meaningful augmentations than previous approaches as shown in the experimental results\n\n\\- Different parts of the paper are disconnected and difficult to follow. See clarity for more details.\n\n\\- The differences between this work and adversarial autoaugment (advAA) seem minimal. As shown by theorem 2.1, with uniform unconditional probabilities and $\\sigma \\rightarrow \\infty$ the proposed approach is advAA. As shown in table 3, the contribution of considering variable unconditional probabilities is quite low and in the order of the std. Not sure about the difference between $\\sigma = 1$ and a high value. It would be interesting to see table 3.\n\n\\- Differences in classification accuracy are relatively small. How can we verify that those differences are not due to a larger batch sizes or other implementation details", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kr9gC3rPxu", "reviewer": "Reviewer_AC1v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a simple yet effective data augmentation method, which is called LatentAugment.", "summary_of_the_review": "Although the paper propose a simple and effective method, I believe the paper needs improvement before getting accepted.", "strength_and_weaknesses": "Strength:\n\nThe proposed LatentAugment is straightforward and reasonable;\n\nCompared to some previous methods, the proposed method is efficient in terms of computation cost;\n\nWeakness:\n\nThe performance gain is relatively small. According to Table 1, UBS is also an efficient method, while the proposed method is only slightly better than UBS with more computation cost. Given the results, I'm curious about result of K=1 for Table 1, from which we may better compare the training cost and performance of proposed method and UBS.\n\nGiven that the results are slightly better than previous method, more experiments are needed. For example, instead of vanilla fully-supervised learning, how will the proposed method benefit some settings where data augmentations might be really important, including few-shot learning, transfer learning, meta learning, etc. \n\nSome presentation needs to be improved. For instance, Figure 2 looks incomplete because some bars seems to be missing without clear explanation. I would suggest the authors either compared all the related methods under all the same settings, or explain clearly about the figure/results.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bkfpbw1x_39", "reviewer": "Reviewer_AEKr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework for data augmentation that generalizes other existing data augmentation methods by incorporating Bayesian inference methods. Empirical performances on common benchmark datasets suggest that the LatentAugment methods outperforms other Adversarial benchmarks not only as measured by test accuracy in classification tasks, but also by training cost.", "summary_of_the_review": "My personal evaluation would be a 5. I'd be happy to adjust my score, if the authors respond to the concerns accordingly.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is well-organized, with each section clearly stating what it does. The contribution of the paper has been clearly outlined.\n\n2. Experiments look sound and justify the main claim of the paper.\n\nWeaknesses:\n\n1. The level of novelty is relatively low. On a high-level, LatentAugment appears to be a straightforward application of the EM algorithm over the logarithmic losses defined with respect to conditional probability. The intuition behind the choice of loss function is also a bit vague. It would be helpful to explain why softmax is used and how the conditional probability relates to the goal of the LatentAugment algorithm.\n\n2. Except for the classification task, no other task has been used to test the performance of LatentAugment. It would be helpful to incorporate other relevant tasks where data augmentation is helpful (e.g. inpainting, image generation) to see how LatentAugment is useful in general.\n\n3. CIFAR, SVHN, and ImageNet are public datesets very commonly used across different image tasks. Has there been any tests on other less commonly used datasets where LatentAugment might be helpful? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LaRDYLsEtQ1", "reviewer": "Reviewer_M9Bp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the learn optimal policy for data augmentation on image classification task.  Learning the optimal augmentation policy, which is the latent variable, interesting research problem. The paper proposes a simple via EM-based method (call LatentAugment) to estimate the probability of optimal policy via conditional probability of the policy customized for given input and network model. It also provides the theorical analysis to show some exisitng methods (AdvAA and UBS) as the special cases. Experiments are on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets using different network architectures showing the encouraging results that it outperforms most of existing methods.  ", "summary_of_the_review": "Overall, I think the method is novel to me and most of claims are well-supported except one in the weakness. The paper is in the form can be published.  ", "strength_and_weaknesses": "[Strength]\n\nThe paper is well-written and easy to follows. \n\nNovelty and with theoretical analysis provided.  \n\nThe model looks simple and computational efficiency, but would be good if with evidence supported.  \n\nThe results are encouraging with ablation study provided.  \n\n[Weakness]\n\nThe claim on computational efficiency needs to be supported with theoretical or empirical results. For example, it would improve a paper if the authors can provide the computational comparison to other methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject