A Continuous Mapping For Augmentation Design
Authors: Keyu Tian
Chen Lin
Ser-Nam Lim
Wanli Ouyang
Puneet K. Dokania
Philip Torr
Venue: NeurIPS 2021 Poster (2021)
Keywords: Data Augmentation
Auto Augmentation

Abstract: Automated data augmentation (ADA) techniques have played an important role in boosting the performance of deep models. Such techniques mostly aim to optimize a parameterized distribution over a discrete augmentation space. Thus, are restricted by the discretization of the search space which normally is handcrafted. To overcome the limitations, we take the first step to constructing a continuous mapping from $\mathbb{R}^d$ to image transformations (an augmentation space). Using this mapping, we take a novel approach where 1) we pose the ADA as a continuous optimization problem over the parameters of the augmentation distribution; and 2) use Stochastic Gradient Langevin Dynamics to learn and sample augmentations. This allows us to potentially explore the space of infinitely many possible augmentations, which otherwise was not possible due to the discretization of the space. This view of ADA is radically different from the standard discretization based view of ADA, and it opens avenues for utilizing the vast efficient gradient-based algorithms available for continuous optimization problems. Results over multiple benchmarks demonstrate the efficiency improvement of this work compared with previous methods.

A Continuous Mapping For Augmentation Design
Keyu Tian" Software College Beihang University
Chen Lin" University Oxford chen. lin@eng. ox ac. .uk
tiankeyu. siankeyu.000gmail. com
Ser-Nam Lim Facebook AI
Wanli Ouyang University of Sydney
Puneet K. Dokania University of Oxford & Five Al Ltd.
Philip H.S. Torr University of Oxford philip. torrdeng. ox. ac
Abstract
Automated data augmentation (ADA) techniques have played an important role in boosting the performance of deep models. Such techniques mostly aim to opti- mize a parameterized distribution over a discrete augmentation space. Thus, are restricted by the discretization of the search space which normally is handcrafted. To overcome the limitations, we take the first step to constructing a continuous mapping from Rd to image transformations (an augmentation space). Using this mapping, we take a novel approach where 1) we pose the ADA as a continuous optimization problem over the parameters of the augmentation distribution; and 2) use Stochastic Gradient Langevin Dynamics to learn and sample augmentations. This allows us to potentially explore the space of infinitely many possible augmen- tations, which otherwise was not possible due to the discretization of the space. This view of ADA is radically different from the standard discretization based view of ADA and it opens avenues for utilizing the vast efficient gradient-bas based algorithms available for continuous optimization problems. Results multiple benchmarks demonstrate the efficiency improvement this work compared with previous methods.
Introduction
Data augmentation is one of the advanced techniques or recipes that proliferate the success of deep learning. The crux behind data augmentation is to make model invariant towards various input transformations that we expect to observe during inference, thus, improves the generalization of the model. Since is not clear which transformations would benefit most, manual design difficult. Thus, there has been growing interest in automatically learning such augmentations, a new thriving subarea
of data augmentation known as the Automated Data Augmentation (ADA) [5] 16 21 20
19 28
ADA is generally formulated as a bilevel optimization problem over discrete search space, and aims find an optimal augmentation policy. For example, the search space used in AutoAugment (5), the very first promising work in this topic, contains several augmentation candidates, e.g., rota- tion/bluring/briiheenngg with different discrete magnitudes, and the augmentation policy is defined as parameterized discrete distribution over this space. Such approach has been shown to provide
Authors contributed equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

far more superior performances over several datasets and tasks compared against the traditional handcrafted augmentation counterparts. This, however, comes at the expense of high computational budget, taking far more time than model training, resulting in an untenable training procedure in the presence of real-world large-scale datasets. While several approaches have been proposed to address the efficiency issue (20] 16 21 19], the compromise in performance due to such efficiency measures was nontrivial; some of the current state-of-art remains computationally expensive [28]. The contradiction between performance and efficiency has raised a key challenge in this field.
Another major limitation of such approaches is due the discretized search space which limits the diversity of the learnable policy. In fact, the discretization results in augmentations that do not always represent the variations in the real world sufficiently. In the real world, the data is collected with a great range of freedom such as varying camera viewpoint and lighting conditions. However, the discrete augmentation would only be able to produce transformed data with specific types and magnitudes. Furthermore, quantization of augmentation search space introduces extra manual design, which is contrary to the aim of ADA.
In this paper, we resolve the aforementioned issues of discretization by constructing a continuous and differentiable mapping e which maps an augmentation vector a R to an image transformation function la E T that transforms an image / to la (f). Specifically, we group basic augmentations into three categories: Color Adjustment that adjusts the color values, Image Filtering like blurring or sharpening, and Image Warping which represents the geometric transformation. We then split the augmentation vector a to three sub-vectors (OCCA. Qtp and arw), each of which would map to a specific transformation (lacs: lay or (. We finish the construction by regard la the composite transformation, i.e., la law o lay o lacs as visualized in Fig. Based on this construction, we pose ADA as learning an augmentation distribution in the continuous augmentation space. This gives us the freedom to search over the entire continuous space in order to obtain the optimal augmentations. For optimization and inference, we use the well-known Stochastic Gradient Langevin Dynamics (SGLD) and propose a pseudo one-step approximation that allows us to efficiently learn the underlying augmentation distribution and sample augmentations from it. On wide range of experiments involving ttate-ofthhe-a architectures on multiple image classification benchmarks, we show that approach not only provides results at par with existing ADA methods, it also is orders of magnitude faster than them.
Our main contributions can be summarized as:
1. We propose a continuous augmentation mapping and first pose ADA as a continuous optimization problem, thus, avoiding the pitfalls of the discrete augmentation space.
2. We develop gradient-baser based approach to learn the augmentation distribution in the continu- ous space that allows for highly efficient sampling of augmentations, offering a favorable pereonpeeformance trade-off.
2
Related Work
Recent automated augmentation methods have been proposed that optimize data augmentation policies on different datasets and tasks 55 16 augmentation search space and an RNN controller to parameterize the augmentation distribution. Reinforcement learning is then employed to search for the augmentation strategy. Subsequent work focuses more searching efficiently in the same space. To this end, OHL [21 proposed an online reinforcement learning pipeline to reduce the computation complexity. PBA [16) proposed a population based search strategy to obtain the augmentation distribution with fitness selection also in an online manner. Fast Automug (20) proposed a different objective function that applies data augmentation at test time. AWS (28 investigates the effectiveness of weight-sharing paradigm, yielding superior performance on various datasets. The most related work, DADA [19] take a continuous relaxation to the discrete augmentation distribution by Gumbel- to obtain better approximation. Our work eliminated such approximation tricks when computing the gradient thanks to the continuous nature of the proposed search space.
19 28 AutoAugment (5] provides a discrete
Markoy Chain Monte Carlo (MCMC) is a popular sampling method for approximating the pos- terior distribution in large-scale Bayesian modeling. MCMC draws samples from high- -igeeiiinnnio

---

CA
dF
dyw
G1
Cig
d11
Augmentation Vector
Differentiable
T
Augmentation Transformation
(!)
Color Adjustment
Image
Image
Filtering
Warping
LacA
Cargr
Cayw
LacA
Caye Lacs (1)
Layw
Car Caacs (1)
Figure 1: Schematic representation of the continuous image augmentation mapping Φ H+ ta the composite transformation of color adjustment faca (red the figure), image filtering for (blue), image warping tow (green). Notably, the augmentation process is fully differentiable w.r.t. a (explained Sec. 3.2
probability distributions the construction Markov chain. A broad category recent MCMC methods, namely Stochastic Gradient MCMC (SG-MCMC) B 9 30), comes from the traditional stochastic gradient optimization and thus inherits the efficiency Stochastic Gradient Descent (SGD). To cast a stochastic optimization process to sampling procedure, artificial noise needs to be added to the gradients. typical example gradient MCMC methods is Stochastic Gradient Langevine Dynamies (SGLD) (3), which injects Gaussian noise into the parameter gradients, pushing the optimization trajectory to asymptotically converge to the actual posterior distribution. In practice, some of these algorithms are also used in combination with several heuristic approximations, such as constant step sizes or diagonal approximations to the Hessian for the sake of simplicity or efficiency 25 In Sec. 4.2 of this paper, also the simple SGLD with constant noise optimize our augmentation distribution.
3 A Continuous and Differentiable Mapping
3.1 Preliminaries
Let the mapping @ a + to map a d-dimensional vector a E Rd to an image augmentation transformation fa E T, where the augmentation (-) maps image to its transformed one ta (f). For convenience, we introduce the notation ta to denote the transformation function T for which ta ((a:). In other words, could be regarded as the parameter of the transformation function fa. and @ is the realization function.
3.2 Constructing The Mapping
We start by developing taxonomy of image augmentation transformations. Popular basic augmen- tations can be grouped into three categories: Color Adjustment that adjusts the color values pixel by pixel, Image Filtering like blurring or sharpening the whole picture, and Image Warping which applies geometric transformations to the image. Now we discuss how to construct continuous and differential mapping for each augmentation categories respectively:
Color Adjustment is defined as transformation in the spatial domain that is equally applied to each pixel. Let n be the image size. For every location (z,y) with pixel vector Lzy = [hay Szy Vzyl
would be adjusted I'y [bey s'ep Vayll T. The augmentation can be described as:
Th'oy" S'y
+
(1 + (hayy) )M
I' t'=taca (T), where Iyy =
ass
+
(1 + (s
(V x. that S 2x,y $ n),
Vay
+
(1 + Be) )4
(1)
3

---

where the nine-dimensional vector &CA [on P Th 0, 3g % av B, N]" maps to a Color Adjustment augmentation tacc The first three values control the hue, the next three would change the saturation, and the last three control the brightness. This representation is motivated by the Power law (Gamma) transformation function 24 As shown in Equ. ( this augmentation family is differential w.r.t. all o:, A, y since it can be expressed as power functions of them and both RGB-to-HSV and HSV-to-RGB conversions are known to be continuous and differentiable [26]
Image Filtering is a typical image enhancement in the frequency domain. Here we consider using filters to blur sharpen images. Let a single value denote the degree of sharpening smoothing. Let Ga be a sharpness filter obtained by zero-mean normalizing k x k discrete Gaussian filter, and C2 be k x k identity filter which has an unique non-zero value 1.0 at its center. The sharpening or smoothing process that transforms an image / to 1' is:
I'= Lam (T) I Gp + Ck),
where the asterisk means convolution and for (.) is the Image Filtering function. If s is greater than zero, represents sharpening. Otherwise, it is equivalent to blurring. Thus, a one-dimensional vector [Cni]] could map to Image Filtering transformation.
Image filtering differentiable w.r. by nature. However, the computation grows fast with respect to the filter size k. When l is close to the image size n, a better implementation for computing the convolution would be applying discrete Fast Fourier Transform (FFT) and multiplying the frequencies. which achieve complexity of O( l).... In this work, we take direct convolution since we pick < n for simplicity.
Image Warping refers to a geometric distortion on an image where pixels are moved to other locations without changing their pixel values. Perspective transformation (also known projective transformation) is a popular warping method, which can be uniquely represented by a x 3 matrix H called homography (2). Each point in the original image (x. y) would move (æ', v') with the perspective transformation. This Image Warping can be described as:
I' = too (I)
HI + H12 + H13
x'
Hai + Hazy + H33
(3)
Tw Tays
where
(V y that S xx, n),
V' Harr + H22y + H2a
HaT H32y Hg3
where few (-) denotes the Image Warping augmentation via perspective transformation, and αiw
H H12 HIs H21 H22 H2s Hai H32 Haa] could represent an Image Warping augmentation. Rotations, translations, shearing, scaling, as well as more single geometric transformations and any combinations of them are all special cases of perspective transformation. Perspective transformation is originally non-dificrentiable to each element the homography H since index x, take discrete leverage interpolation between pixels and inverse warping to make the
values. However, transformation differentiable 17 27
In summary, an augmentation vector a = [atA OFF ] lies in continuous real space with 19 dimensions could map to a specific augmentation transformation ta
d = 1 faw o far a facA- Although composite order matters, for simplicity, leave the matter to future work.
One can easily verify that many commonly used augmentation transforms 6 20 16 21 19 28). such as brightening, blurring, and rotation, are all discrete samples in the augmentation function space T our mapping. Last but not least, practical constraint functions (tanh, exp) and clamp function are used before and after applying augmentations respectively, to ensure all augmentations are valid and all augmented pixels are in [0. 1]. The overview of this space is illustrated in Fig. More details on the augmentation process of this figure are provided in Supplementary A.

---

3.3 An Example Random Augmentation Policy
An example random augmentation policy in our continuous augmentation space is provided here to evaluate the space by comparing this policy with other policies in discrete space. This random policy is based on multivariate Gaussian distribution g(x)) over a with zero means and diagonal covariance. The according augmentation distribution is denoted Prand(fa)) where to Φ(x)) and a ~ g(a). This policy is one of the simplest continuous random policy one could define.
Table I:
Comparison between example random policies in continuous augmentation space and others based on discrete space. Top-1 errors of Fide-ResNet-40-2 CIFAR-10 are reported. Lower better.
Example Policy
Example Policy
AutoAugment (random, our impl.)
Augmmnt (scarched, from 55)
DADA
Method
(random, std=0.5)
(random, std=1.0)
(searched,
from
Error
3.25
3.34
4.08
3.7
3.5
In Tab. D we compare the example random policy with both random and optimized policies in discrete augmentation space to show the advance continuity. Note that DADA [[9] basically follows the same discrete space of Autonngmnnt's (5). Two example policies with different standard deviations (0.5 and 1.0) are evaluated for better comparison.
4 Automatic Augmentation Design using Φ
4.1 ADA Problem formulation
Let t(+) denotes the augmentation function. ADA aims to find the optimal distribution of aug- mentations q(t), which improves the generalization of the model. Let D;
and D. {xx; H:n be the training and validation datasets, respectively. Training a deep network fo parameterized by the process of minimizing a loss -(.-), which is cross-entropy in this work. We use po (yl (x;) to refer to the probability distribution over labels by fe for x,, i.e., the output of fo followed by a sof tmax function. In general, ADA aims at solving the following bilevel optimization:
arg min ((t)
Σ
C(fo- (x:),
(4)
(R.
s.t. 8" arg min Σ
((fo(((s))))))
(5)
().,
tt)e(t)
where g()) is the distribution of augmentations, ((x;) denotes the augmented data of x,. The goal is to minimize the outer-level validation loss of a network fo- (4) that is trained by minimizing the inner-level training loss using augmented training data in
Notably, by writing the cross-entropy explicitly, we will have:
[(fo(z). v) log(po(ylz)),
(6)
for any model parameter 0, and any labed data pair (x, v).
4.2 Gradient-Basdd ADA in Continuous Augmentation Space via SGLD
ADA in continuous space. Previous work mostly defines q(t) as parameterized categorical distribution or a discrete uniform distribution. In this work, given the proposed continuous mapping
: d - T where T the set of augmentations, we are now able to relate augmentation distribution q(!) for / T to continuous distribution p(a:) for E IRd by ( lo = Φ(aa. Thus
we can reform the bilevel optimization 5 by substituting the target distribution g(l) with p(a:)
and apply the realized transformation fa to training data x, then get (8):
5

---

arg min p(a)
Σ
[(So- (xs))))));:))
s.t. 0" = arg min (:))
Σ
(8)
am>((a)
One-step simplified inner-level. Notably, directly solving the bilevel optimization in would be prohibitive since it requires hundreds or thousands of full model training and validation. Thus, we refer the successful augmentation-ww techniques to make this problem more tractable.
weight-sharing (28] and one-step meta learning 10 23
Specifically, we begin with an augmentation -igeentatioo-wiie shared parameter 8, on D, using the random augmentation policy (c) introduced in Sec. 3.3 We replace the inner optimization (8) with pseudo one-step update: a single gradient step for batch of training samples B, {(x. y:) 1= augmented by a given la taken to get 02 that approximates the 0", where M the batch size. The inner optimization in 8 now becomes:
0" RJ 0.
0g
Σ (x,Y))EB
Vo, (la(xx)))
Bayesian formulation for ADA. In this setup, ADA boils down continuous distribution
optimization problem 7 In order to apply gradient-based methods, we drive new Bayesian formulation for distribution optimization ADA Let Pptor(()) be the prior distribution and p(a (Da) be the posterior distribution. The optimization in (7) for obtaining the target distribution p( o())) is modeled our Bayesian formulation by inferring the posterior distribution p(a (D.) follows:
No (1):).))))... 1=1
(Do) x Pprior (a))
(10)
where I p( (i)la) is the data likelihood of the validation set given fixed augmentation parameter a. After obtaining 0g using Equ. (9) for given a, this likelihood now can be computed efficiently as i Pa, (&,) Note that the inner-level optimization is efficient as we always start with the same 0. and then take single gradient step over the given batch under the augmentation la = ((a)) to obtain 0,. Assuming uniform prior Prrio the posterior in (10) can be computed as:
N D,)) & II PS. (v.12)))
(11)
Use gradients to search for augmentation policy. With the Bayesian formulation and the pro- posed differentiable mapping o, Gradient-MCMC methods like Stochastic Gradient Langevin Dy- namies (SGLD) is now applicable to find the target distribution by sampling. Recall Equ. (6). the j-th iteration of SGLD sampling on the validation batch B, {(x,, Y:)) can described as:
C M
(++1) a ( +
Σ
Vaci log P7. (5.((,))
(12)
2 N (R..,.)EB
= =0 +n+
cMA
Σ
Σ
aLSs, (2))..)) O PLioo (xx),
(13)
2N,
09,
00 30,0a()
(B
(E).
where is the step size of SGLD and ~000 E) is the Gaussian noise.

---

During the sampling, how to use a to augment images
How to sample at)
12
Brightness
a
Maps
SOLD Trajectory
T
(a)
4
Brightening
Rotation
Angle (Normalized)
Figure 2: Augmentation search phase of simplified MCMC-Aug. Suppose E two augmentations are brightening and rotation for simplicity. Left: we collect the samples of a (+) (where j = 1,2,-.. along the SGLD trajectory. Right: how to perform the augmentation.
(d 2) and the
In practice, one can use finite difference 13 2 to compute the second-order derivative in LB3 more
efficiently. By iterating Equ. [3), samples of be collected and SGLD would usymptotically converge to the posterior distribution p(a{ D.,) 32, which is the resulted augmentation policy.
The whole search pipeline of method (MCMC-Aug) is summarized in Alg. The augmentation search phase of a simplified (suppose d = 2) MCMC-Aug is also visualized in
The overview.
Fig. The samples (a( ( ... (N,) collected by SGLD form the resulted augmentation policy. When using the policy to train
model, would pick randomly from these samples to augment each training image. In Sec. [ and Supplementary we will evaluate and visualize the searched policy in detail.
Algorithm A Reference Method for ADA MCMC-Aug
Inputs: Training set De, validation set D.., pretrained weights 0., samples of the augmentation vector S; Hyperparameters: Number of updates Nj. mini batch size M, model learning rate A, SGLD step size Initiate: Ø; (0):
for j 0, 2, .,, do
Sample bastches B, and B.:
Perform pscudo one-step update in Equ. obtain 0.;
Perform SGLD update in Equ. 13 to get (+1);
S = (a (1=1));
end for return S;
Experiments
5.1
Implementation Details
We evaluate our method on CLFAR-10/100 18] and ImageNet (7]. For each datasets, a validation set split from the training set and the testing set only used for evaluating the final performance (not involved in the search phase). For ImageNet, a reduced ImageNet-122 120 is used for fair comparison [5, 20 16. 19). All batch sizes for augmentation search mentioned in this work are 256 for CIFARs and 1024 for ImageNet as standard. More details are included in Supplementary. B.
Search. Following prior work, we search policy for two base models, which are
31 and ResNet-50 [15] on CIFARs and ImageNet respectively. Before we start, an instance of using the random policy introduced in Sec. S
each base model pretrained (CIFARs/ImageNe) obtain the shared weights 0,, in Equ. 19. This procedure lasts 200 epochs for CIFARs and 120 epochs for ImageNet. Then we run our MCMC-Aug with 200 epochs, the same [16] over the training set. Step size and noise scale of SGLD is set according (25). The experiments reported have fixed 0.4

---

step size, and noise rate of x 10 10"5. Further analysis on the choice of hyperparameters could be found in Supplementary D., where we provide sensitivity analysis as well guideline for tuning SGLD. On CIFAR-100 and ImageNet, the same SGLD hyperparameters are adopted without any modification.
Evaluation. We evaluate MCMC-Aug by mea- suring the performance of a reinitialized model trained with augmentation from the SGLD samples (aam focto, , ( J. Theoretically, SGLD will be in its Langevin dynamics phase and sampling ap- proximately from the posterior distribution when its sample threshold statistic is much smaller than 1.0 [3 However, in practice, we found that collecting samples of from the later half of the SGLDD jectory is sufficient to obtain good performance. We run SGLD only for base models, and the collected samples could be used to retrain other models to as- sess the transferability. Thus RelleRReNNet-(40-- 28-10) [31] Shake-Shake (26 2 x 32d) [1). Pyramid- Deto are retrained on CIFARs, and ResNet-(50. 200 on ImageNet. Each of experiments repeated four times and the mean value reported, see Supplementary for the error bar. Other hyperparameters not mentioned here are directly imported from (28), provided in Supplementary C.
Table 2: Test error rates on ImageNet. Top-1 Top-5 errors are reported. Lower is better.
Method
ResNet-50
ResNet-220
Baseline
23.7 6.99
2..
AA
2..4/6.2
20.0
FAA
2.34/6.3
19.4/4.7
RA
2.24/6.2
OHL.
21.07 5.68
AWS
20.61 5.49
18.64 /466
DADA
2
MCMC
20.98 5.60 18.87 /462
5.2 Comparison with State-of-the-Arts
Summary. We conducted comprehensive comparisons with state-of-the-art augmentation methods that includes Cutout [8), AutoAugment (AA) (5), Fast AutoAugment (FAA) (20). Rand Augment
(RA) [6. Population Based Augmentation (PBA) [6]. and OHL-AutoAug (OHL) [2I]. Furthermore, we compare MCMC-Aug with two of the latest ADA methods: AWS-AutoAl (AWS) [28).
which is currently the best-performing ADA method reported; 2) Differentiable Automatic Data Augmentation (DADA) [19], which currently the fastest method that applies continual relaxation to the discrete search space using Gumbel-softman
Performance comparison. The performance MCMC-Aug and other methods are compared in Tab. and and Tab. The performance of the other methods is directly obtained from the corresponding papers. We dashes wherever the results are not reported. Following prior work [9] "Baseline" in the tables refers to model trained using only default pre-processing like normalization and basic augmentations such as random cropping and random horizontal flipping. The same pre- pre-processing and basic augmentations are also uniformly applied to all the methods. Cutout is used in all the ADA methods on CIFAR datasets.
Table Top-1 test errors on two CIFAR datasets. Lower better. "MCMC" represents MCMC-Aug.
Model
Baseline
Cutout
AA
FAA
RA
DADA
OHL
PBA
AWS
MCMC
CIFAR-10
WRN-40-2
5.3
3.7
3.7
3.5
2.96
WRN-28-10
3.08
2.7
2.61
2.58
1.97
Shake-Shake
2,56
2.0
2.0
2.03
1.65
1.53
PyramidNet
2.67
2.31
1.5
1.7
1.46
1.31
1.29
CIFAR-100
WRN-40-2
26.0
25.2
20.7
20.6
20.9
19.07
WRN-28-10 10
18.8
18.4
17.1
17.3
16.7
17.5
16.73
15.28
15.64
Shake-Shake
17.1
16.0
14.3
14.6
15.3
15.31
14.07
13.98
PyramidNet
13.99
12.19
10.7
11.7
11.2
10.94
10.40
10.48

---

2.8
2.6
Top-1 1 Emor
method
2.4
AA
AWS
2.2-
OHL
PBA
FAA
X
DADA
MCMC
10
10
10
10
10
10
10
GPU Hours
Figure 3:
Efficiency vs. performance among ADA methods. Some points are for full CIFAR-10 and the others are for reduced CIFAR-10.
Tab. and Tab. 3 show that MCMC-Aug significantly reduces the error rates over the baseline and Cutout on variety of network architectures, demonstrating the effectiveness and transferability of the resulted policies. Compared with other ADA methods, MCMC-Aug performs on par with AWS while outperforming the others, even though we did not conduct any comprehensive parameter tuning. Moreover, in Sec. 5.3] we show that increasing the number of searching epochs (proportional to N₁ in Alg. brings extra performance gains. Since we had only trained for 200 epochs for efficiency, we believe there is still room for improvements.
Efficiency comparison. We compare the computational cost of MCMC-Aug with others to check MCMC-Aug inherits the efficiency gradient- based optimization. Results on both the reduced and the full CIFAR-10 dataset reported to ensure fairness, since previous work had reported efficiency of their methods on one or both of the datasets. We used the same reduced CIFAR-I0 subset in [5] 20 16 [9], which consists of 4,000 images from the training set. We kept the same hyperparameters except for a smaller number of epochs in searching (five) and linearly scaled batch size and learning rate (four times). The results summarized in Tab, and visualized in Fig. 3
As shown in Tab. 4 OHL, FAA and DADA are much faster than AA, but at the expense of accuracy. AWS achieves the best performance, but is much more computationally expensive. Like PBA, MCMC-Aug strikes a good balance between accuracy and efficiency, but yields much better accuracy. We also note that there is noticeable gap between MCMC-Aug' performance on the full and reduced dataset, showing the potential for further performance gains when more computational budget becomes available. Overall, our results show that MCMC-Aug is the best in class when considering the need to balance performance and efficiency.
Table 4: Comparison of the efficiency (GPU hours). Top-1 test error changes of Wide-ResNet-28. relative to AA are listed. -: not reported. 1: approximated, from [19]. t: emor increased. 4: error decreased.
Method
AA
AWS
PBA
OHL
FAA
DADA
MCMC
GPU Device
P100
V100
TitanXP
V100
V100
TitanXP
V100
Reduced CIFAR-1O
GPU Hours
5000
5
3.5
0.1
0.05
Error Changes (normalized)
0.0%
0.8%
13.9%
5.00
16.15%
Full CIFAR-10
GPU Hours
125
83.3t
45
7.5t
5 24.2%
Error Changes (normalized)
J 25.0%
0.4%
1.99%
1 1.5%
5.3
Ablation Study
Space decomposition. show the necessity of three base catogories augmentation mapping, we remove two categories of transformation in turns, after which we look at the change in test accuracy.

---

As reported in Tab. 5G the performance of Wide-ResNNet-0-2 on CIFAR-10 10 would dramatically decrease, which suggests that each of them indispensable.
SGLD early-stopping. To investigate the optimization behavior of the SGLD, we stop collecting the SGLD samples 25%, 50%, and 75% of the total epochs and evaluate the corresponding performance respectively. As listed in Tab. 6 we see that the errors of the distributions arising from early-stopping gradually increase. shows the performance gain along with the SGLD procedue. This also suggests that increasing the number of searching epochs of our approach might bring extra performance gains.
Table 5: Ablation study on the importance of different augmentation categories.
Table 6: Ablation study on the effectiveness of SGLD.
Remaining
SGLD Early- Stopping at
Error
Error
Category
2.96
100% (original)
2.96
Color Adjustment
(.32
75%
3.06
Image Warping
3,45 (10.99
50%
3.15 (10.9)
Image Filtering
(.61
25%
3.20
For the sensitivity of hyperparameters and more details on the searched distribution, please see Supplementary D. and E.
6 Limitation
Non-differentiable augmentations that do not belong to any categories of the three are a clear limitation. However, an interpolation between the original image and its augmented counterpart could be made to bypass this issue. Suppose we have an image / and a non-differentiable transformation t, we can introduce continuous parameter À and create new augmentation t' via (1) X t(1) (1 - X) I. t' is surrogate augmentation which ensures the differentiability.
The presence of clean validation is vital our MCMC-aug as well other automated data augmentation algorithms. This is because the loss or accuracy on validation set is a common criteria for searching the augmentation policy. We expect to see better approach to reduce the dependency on validation set in the future. It is valuable direction and ought to be further explored.
7 Conclusion
continuous mapping to model a continuous space of augmentation is proposed. The continuous mapping be applied to many augmentation design tasks (e.g., instance discrimination contrastive learning, image alignment, and domain adaptation). We apply this mapping to pose ADA as continuous augmentation search problem. Furthermore, setup Bayesian formulation for ADA to enable the application of efficient gradient-based MCMC, presenting a new way to conduct ADA search via posterior sampling. Comprehensive comparison and studies verify its efficiency and effectiveness. Both source codes and checkpoints will be released to the public, and future work may concern more image modalities like Srraaee/X--ays//lrrasouuu imaging, more diverse augmentations, or wider applicants, e.g., in other computer vision areas or the times-series processing.
8 Funding Disclosure and Acknowledgement
This work is supported by the EPSRC grant: Turing Fellowship: EP/W002981/L EPSRC/MURI grant EP/N019474/1. Chen Lin is supported by Facebook AI. We would also like to thank the Royal Academy of Engineering and FiveAI.
10

---

References
[1] S. Ahn, A. Korattikara, and M. Welling. Bayesian posterior sampling via stochastic gradient fisher scoring. arXiv preprint arXiv: (2Xivv:206.6638) 2012.
[2] A. M. Andrew. Multiple view geometry in computer vision. Kybernetes, 2001.
[3] T. Chen, E. Fox, and C. Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683-1691. PMLR, 2014.
[4] G. Contardo, L. Denoyer, and T. Artières. A metaa-eearnng approach to one-step active learning.
arXiv preprint arXiv: 700670060000 770.008344, 08334,
2017.
[5] E. D. Cubuk, B. Zoph. D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019.
[6] E. D. Cubuk, B. Zoph, Shlens, and Q. V. Le. Randaugment: Practical automated data
augmentation with a reduced search space. arXiv preprint arXiv: 1ttp:9999.3779,, 2019.
[7] J. Deng, W. Dong, R. Socher, L..J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
248-255, Icee, 2009.
[8] T. Deerries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 708.. 045522, 2017.
[9] N. Ding, Y. Fang, R. Babbush, C. Chen, R. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. 2014.
[10] C. Finn, A. Rajeswaran, S. Kakade, and S. Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
[11] I. Finnveden, Y. Jansson, and T. Lindeberg. Understanding when spatial transformer networks do not support invariance, and what to do about it. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3427-3434, IEEE, 2021.
[12] X. Gastaldi. Shake-shake regularization. arXiv preprini arXiv:1705.0744 2017.
[13] W. Gong, Y. Li, and J. M. Hemúnndez-Lobato Meta-learning for stochastic gradient meme. arXiv preprint arXiv: 1806. 04522, 2018
[14] D. Han, Kim, and Kim. Deep pyramidal residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5927-5935, 2017.
[15) K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[16] D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In ICMI., 2019.
[17] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. arXiv preprint arXiv: 150h::55000000 02025, 2015.
[18] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[19] Y. Li, G. Hu, Y. Wang, T. Hospedales, N. M. Robertson, and Y. Yang. Dada: Differentiable
automatic data augmentation. arXiv preprint arXiv:2003.03788 2020.
[20] S. Lim, Kim, T. Kim, C. Kim, and S. Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pages 1662-6672, 2019.
[21] C. Lin, M. Guo, C. Li, X. Yuan, W. Wu, J. Yan, D. Lin, and W. Ouyang. Online hyper-paramete learning for auto-augmentation strategy. In Proceedings of the IEEE International Conference on Computer Vision, pages 6579-6588, 2019.
[22] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.44000 2013.
[23] H. Liu, K. Simonyan, and Yang. Darts: Differentiable architecture search. arXiv preprint
arXiv:1806. 09055, 2018
[24] R. Maini and H. Aggarwal. A comprehensive review of image enhancement techniques. arXiv preprint arXiv: 1003. 4053, 2010.
[25] Mandt, M. D. Hoffman, and M. Blei. Stochastic gradient descent as approximate bayesian
inference. arXiv preprint arXiv: 704.044299 2017.
[26] S. Moran, S. McDonagh, and G. Slabaugh. Curl: Neural curve layers for global image enhancement. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9796-9803. IEEE, 2021.

---

[27] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Kormia: open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3674-3683, 2020.
[28] K. Tian, C. Lin, M. Sun, L. Zhou, J. Yan, and W. Ouyang. Improving auto-augment via augmentation-wise weight sharing. Advances in Neural Information Processing Systems, 33, 2020.
[29] Y. Wang, J. Guo, H. Gao, and H. Yue. Uieč2-net: Con-based underwater image enhancement using color space. Signal Processing: Image Communication, page 116250, 2021. [30] M. Welling and Y. W. Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11, pages
681-688. Citeseer, 2011.
[31] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 1oXiv:6605007144
2016.
[32] Y. Zhang, C. Chen, Z. Gan, Henao, and L. Carin. Stochastic gradient monomial gamma sampler. In International Conference on Machine Learning, pages 3996-4005, PMLR, 2017.
12


Reviews:
[{"review_id": "lJwlj9enwIx", "reviewer": "Reviewer_YW9j", "summary": "    Authors present a continuous formulation of data augmentation for\n    images. The formulation is used to define a Bayesian model for augmentation\n    parameters. The Bayesian model does not rely on a bi-level optimization\n    problem as commonly used for optimizing data augmentation. Instead, only a\n    single gradient step approximates a full optimization of a network with a\n    given set of augmentation parameters. Gradient step is taken starting from\n    pretrained weights determined through initial training using random\n    augmentation. Samples from the posterior distribution are gathered using\n    Langevin dynamics MCMC with random batches. Samples are used to train\n    the ultimate networks. Experiments show that the proposed model outperforms\n    existing techniques in terms of accuracy of the final models at a fraction\n    of computation time. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "    The proposed technique is novel to the best of my knowledge. The continuous\n    formulation, which is emphasized a bit too much perhaps in the article, is\n    necessary for using Langevin dynamics MCMC during inference. The article\n    deviates quite a bit from common literature on this topic and comes up with\n    a novel method that yields great results at the fraction of the compute\n    time. \n\n    Experimental results are quite convincing. Authors only evaluate using\n    rather small datasets but I believe this is the common strategy taken by\n    most of the works in this domain, therefore, it is not a major drawback. \n\n    The article is mostly well written. The method and experiments are clearly\n    explained. The presentation can be improved but I do not think this is a\n    major drawback.\n\n    Data augmentation is an essential part for training neural networks. A\n    computationally light method for determining best augmentation parameters\n    would be very valuable in practice. \n\n    My questions are mostly about the experimental setup: \n    1. Authors show in Table 1 that the random sampling of the augmentation\n       parameters already yields better results than some of the methods that\n       optimize for the parameters. This is rather unexpected and\n       not intuitive. Can they provide an explanation for this? \n    2. It is unclear to me whether authors start all the augmentation methods\n       from some pretrained model. Using the pretrained model is essential, as I\n       understand it, for the approximation given in Equation 9. Without a\n       pretrained model, the proposed approach would not work most\n       probably. Such pretraining may also help the other methods for\n       determining optimal augmentation parameters, both in terms of final\n       accuracy and computation time. Have authors started the other methods\n       from a pretrained model?  \n    3. It is somewhat surprising that a single gradient step approximates a full\n       optimization so well in this work. Can authors elaborate on this? \n\n\n      --- After the update ---\n    I thank the authors for the responses. I keep my opinion that this is good work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "    I do not see a negative societal impact of this work. On the contrary, this\n    work may lead to reduction in training times and save energy. \n\n    Authors claim they discuss the limitations of the work in Section 6 but I\n    fail to see that in the text. The main limitation is the scope of the\n    experiments. Authors could have demonstrated the model on larger set of\n    problems, e.g., segmentation. They could have also used larger\n    datasets. However, I do not see them as crucial limitations. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5; 3.5 with the rebuttal. "}, "extra_text": {}}, {"review_id": "fcLRFbn9w89", "reviewer": "Reviewer_wJZx", "summary": "The authors pose the problem of learning data augmentation as a continuous optimization problem and introduces a Bayesian approach for learning and sampling augmentations. The method allows for optimizing over a space of infinitely many possible transformations in contrast to the majority of automated data-augmentation approaches (ADA) which are limited by discretization of the solution space. The values of the proposed approach over prior works in terms of prediction accuracy and speed have been demonstrated on a range of benchmarks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Clarity: The manuscript is very well-written, and its contributions are clearly communicated with respect to prior works. \n- Convincing empirical results:  the method has been compared against a range of relevant prior methods on multiple image classification benchmarks and demonstrates convincingly that it can attain a better accuracy-speed trade-off.\n- Solid method contribution:  the proposed method is carefully constructed by mixing different ingredients, all of which are important in different ways. Firstly, differentiable parameterization of transformations is a necessary first step to pose the ADD as a continuous optimization problem. Secondly, the single-step approximation used in eq. (9) offers a computationally efficient approximation to the bi-level optimization problem of ADD. Lastly, the use of Stochastic Gradient Langevin Dynamics allows ones to approximately sampling multiple data-augmentation strategies from the posterior distribution instead of resorting to a point-estimate of the optimal data-augmentation strategy, a key to ensuring diversity of relevant transformations. \n\n\nWeaknesses\n- There is little insight into when the proposed method may break or may not work as well as other competing methods. How does the efficacy of the proposed method vary with the size of the validation dataset? When would the single-step approximation be assumed in eq. (9). Is there any relevant form of data augmentation that may be hard to parametrize in a differentiable way? If so and if such transformations were to be critical for some applications, then wouldn't the competing approaches based on discrete optimization be more preferable?  \n- It is unclear what the benefits of the Bayesian approach are. For example, one could set the noise term $\\eta=0$ in eq.(13) and I suspect that the method may still work reasonably well (not shown in the manuscript). The noise injection may potentially improve the diversity of sampled augmentations, but I feel this should be shown to substantiate the benefits of the Bayesian approach.   \n\n\nMinor comments\n- line 103: \"differential\" => \"differentiable\"\n- line 124: \"we pick k<< n for simplicity\" -> what do we lose from this assumption? \n- line 182: \"pseudo one-step update\" -> why pseudo? \n- line 204: \"second-order derivative … computed more efficiently\" => efficient in time or space? \n- line 232: no space after '.' \n- line 255: \"Tab. 3 and Tab. 2\" => Tab. 2 and Tab. 3", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitations in my view are the lack of insights into the limitations of the proposed approach as elaborated above. \nNA for societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "QdEhmbfQd8", "reviewer": "Reviewer_Dpxh", "summary": "The paper presents an automated data augmentation technique that optimizes a distribution over a continuous augmentation space. Augmentation is formulated as learning a continuous mapping from an augmentation vector in R^d to an image transformation that composes three general families/modes of image transformations (color adjustment, image filtering, and image warping).   The proposed method uses a bilevel optimization approach where the primary learning task is trained on an augmented training set and the augmentation learning task is trained on the validation set. To make the optimization tractable, the inner level training is replaced by a one-step gradient update on an augmented mini-batch with model parameters being shared across the bilevel optimization iterations, i.e., incrementally updated during the bilevel optimization. A comprehensive set of experiments, including ablation experiments and comparisons to SOTA methods for automated data augmentation, demonstrate the efficacy and efficiency of the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents an efficient method for automated data augmentation that is learned end-to-end with the primary learning task without compromising performance. The paper moves away from restrictive assumptions about the discrete augmentation space, including predefined/handcrafted augmentations, and poses automated augmentation as a continuous optimization problem. This enables diversity in the augmented samples and avoids ad-hoc approximations when computing gradients in the discrete space cases. Experiments focus on natural image classification tasks, but the method is general to be adapted to other learning tasks and image modalities.\n\nThe work can be improved by addressing the below concerns:\n\n- The method relies on the predefined semantic groupings of augmentation modes (color adjustment, image filtering, and image warping) that assume natural images, but the method is generic enough to be applied to other image modalities by considering modality-specific parameterized transformations. This should be made clear in the discussion section.\n\n-  It would be insightful if a discussion is included on the choice of MCMC versus variational inference given the Bayesian formulation of the problem.\n\n- The learned augmentation relies heavily on the generalization of the model on the validation set. There is no machinery in the proposed method that guarantees the generation of challenging augmented samples for the classification task (e.g., samples that can be easily misclassified, near the decision boundary).\n\n- How sensitive/robust the proposed method under data shift and outliers?\n\n- How sensitive/robust the proposed method under a severely limited training budget (e.g., medical imaging)? For instance, in Algorithm 1, a single augmented sample is added to the training data at each bilevel iteration, this won't be applicable in high-dimension-low-sample-size scenarios.\n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not acknowledge any potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "3vnLm-EBUaK", "reviewer": "Reviewer_7Dyr", "summary": "This contribution is aimed at automated data augmentation. A continuous search space is defined using simple image operators, and thus enables learning of augmentation parameters. A randomised algorithm is proposed for parameter learning.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Data augmentation is an important topic and automated learning of augmentation methods and parameters can improve deep learning.  \nIn this paper the authors engineer a randomized algorithm for learning to augment.\n\nThe  originality could be better; there a numerous examples of continuous parameterizations of image augmentation starting with the work of Hauberg et al.  2016: \"Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation.\" In Artificial Intelligence and Statistics (pp. 342-350). PMLR, and later papers citing this one. Also the literature on generative modeling (GAN etc) is missing, see e.g. Antoniou et al 2017. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04340.\n\nThe quality of the work is good, has a fine motivation and good experimental evaluation. Promising results. However, I recommend in a revised version to do a much more careful literature search and place the work more specifically in the literature on learned augmentations.\n\nThe clarity of the report is excellent, it is well written.\n\nOverall the significance is limited by the novelty and also, if I may recommend to focus a bit more on analysis, to understand to the limitations of learning data augmentation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above - no immediate adverse societal impact ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is concerned with augmentation design, i.e. finding a set of parameters for generating random image variants via color adjustment / filtering / warping to aid generalization in models based on images. The basic idea is to reduce augmentations to a finite-dimensional continuous space, then learn a distribution over the parameters. State of the art methods [5] use bi-level optimization to learn this distribution. Instead, this paper proposes a Bayesian posterior where the likelihood of any set of hyperparameters can be computed in terms of scores on a held-out validation set. That construction, if defined exactly, would still involve a double-loop. However, essentially by "warm-starting" the inner-optimization, this is avoided and a single-loop sampling algorithm is obtained.

Reviewers were generally positive about this paper, and found it clear. The major concerns were that the experimental setup could be improved, discussion of related work could be more thorough, there could be more discussion of when the method might fail, and that it isn't entirely clear what the benefits of the Bayesian approach is. (The paper argues that it has the benefit of avoiding a double-loop optimization, but it appears that the idea of warm-starting the inner optimization could be applied to other methods as well, so it isn't entirely clear what is providing the benefit here.)

I found the central construction difficult to understand. (Reviews simply referred to it as a Bayesian model, but I wanted to understand what this model was, so I read the paper in detail.) After extensive discussion with the authors I am satisfied that there is a coherent model being defined. However, I strongly suggest that the notation of the paper be reconsidered to first explicitly state the model in the form of the equations (i), (ii), and (iii) that were given in the discussion below.
Decision: Accept (Poster)