A Continuous Mapping For Augmentation Design
Year: 2021

Abstract:
Automated data augmentation (ADA) techniques have played an important role in boosting the performance of deep models. Such techniques mostly aim to optimize a parameterized distribution over a discrete augmentation space. Thus, are restricted by the discretization of the search space which normally is handcrafted. To overcome the limitations, we take the first step to constructing a continuous mapping from R to image transformations (an augmentation space). Using this mapping, we take a novel approach where 1) we pose the ADA as a continuous optimization problem over the parameters of the augmentation distribution; and 2) use Stochastic Gradient Langevin Dynamics to learn and sample augmentations. This allows us to potentially explore the space of infinitely many possible augmentations, which otherwise was not possible due to the discretization of the space. This view of ADA is radically different from the standard discretization based view of ADA , and it opens avenues for utilizing the vast efficient gradient-based algorithms available for continuous optimization problems. Results over multiple benchmarks demonstrate the efficiency improvement of this work compared with previous methods.

heading: 1 Introduction
text: Data augmentation [22] is one of the advanced techniques or recipes that proliferate the success of deep learning. The crux behind data augmentation is to make a model invariant towards various input transformations that we expect to observe during inference, thus, improves the generalization of the model. Since it is not clear which transformations would benefit most, manual design is difficult. Thus, there has been a growing interest in automatically learning such augmentations, a new thriving subarea of data augmentation known as the Automated Data Augmentation (ADA) [5, 16, 21, 20, 19, 28].
ADA is generally formulated as a bilevel optimization problem over a discrete search space, and aims to find an optimal augmentation policy. For example, the search space used in AutoAugment [5], the very first promising work in this topic, contains several augmentation candidates, e.g., rotation/blurring/brightening with different discrete magnitudes, and the augmentation policy is defined as a parameterized discrete distribution over this space. Such approach has been shown to provide ∗Authors contributed equally.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
far more superior performances over several datasets and tasks compared against the traditional handcrafted augmentation counterparts. This, however, comes at the expense of a high computational budget, taking far more time than model training, resulting in an untenable training procedure in the presence of real-world large-scale datasets. While several approaches have been proposed to address the efficiency issue [20, 16, 21, 19], the compromise in performance due to such efficiency measures was nontrivial; some of the current state-of-art remains computationally expensive [28]. The contradiction between performance and efficiency has raised a key challenge in this field.
Another major limitation of such approaches is due to the discretized search space which limits the diversity of the learnable policy. In fact, the discretization results in augmentations that do not always represent the variations in the real world sufficiently. In the real world, the data is collected with a great range of freedom such as varying camera viewpoint and lighting conditions. However, the discrete augmentation would only be able to produce transformed data with specific types and magnitudes. Furthermore, quantization of augmentation search space introduces extra manual design, which is contrary to the aim of ADA.
In this paper, we resolve the aforementioned issues of discretization by constructing a continuous and differentiable mapping φ which maps an augmentation vector α ∈ Rd to an image transformation function tα ∈ T that transforms an image I to tα(I). Specifically, we group basic augmentations into three categories: Color Adjustment that adjusts the color values, Image Filtering like blurring or sharpening, and Image Warping which represents the geometric transformation. We then split the augmentation vector α to three sub-vectors (αCA, αIF and αIW), each of which would map to a specific transformation (tαCA , tαIF or tαIW ). We finish the construction by regard tα as the composite transformation, i.e., tα = tαIW ◦ tαIF ◦ tαCA , as visualized in Fig. 1. Based on this construction, we pose ADA as learning an augmentation distribution in the continuous augmentation space. This gives us the freedom to search over the entire continuous space in order to obtain the optimal augmentations. For optimization and inference, we use the well-known Stochastic Gradient Langevin Dynamics (SGLD) and propose a pseudo one-step approximation that allows us to efficiently learn the underlying augmentation distribution and sample augmentations from it. On a wide range of experiments involving state-of-the-art architectures on multiple image classification benchmarks, we show that our approach not only provides results at par with existing ADA methods, it also is orders of magnitude faster than them.
Our main contributions can be summarized as:
1. We propose a continuous augmentation mapping and first pose ADA as a continuous optimization problem, thus, avoiding the pitfalls of the discrete augmentation space.
2. We develop a gradient-based approach to learn the augmentation distribution in the continuous space that allows for highly efficient sampling of augmentations, offering a favorable speed-performance trade-off.

heading: 2 Related Work
text: Recent automated augmentation methods have been proposed that optimize data augmentation policies on different datasets and tasks [5, 16, 21, 20, 19, 28]. AutoAugment [5] provides a discrete augmentation search space and an RNN controller to parameterize the augmentation distribution. Reinforcement learning is then employed to search for the augmentation strategy. Subsequent work focuses more on searching efficiently in the same space. To this end, OHL [21] proposed an online reinforcement learning pipeline to reduce the computation complexity. PBA [16] proposed a population based search strategy to obtain the augmentation distribution with fitness selection also in an online manner. Fast AutoAugment [20] proposed a different objective function that applies data augmentation at test time. AWS [28] investigates the effectiveness of a weight-sharing paradigm, yielding superior performance on various datasets. The most related work, DADA [19] take a continuous relaxation to the discrete augmentation distribution by Gumbel-softmax to obtain better approximation. Our work eliminated such approximation tricks when computing the gradient thanks to the continuous nature of the proposed search space.
Markov Chain Monte Carlo (MCMC) is a popular sampling method for approximating the posterior distribution in large-scale Bayesian modeling. MCMC draws samples from high-dimensional
probability distributions in the construction of a Markov chain. A broad category of recent MCMC methods, namely Stochastic Gradient MCMC (SG-MCMC) [3, 1, 9, 30], comes from the traditional stochastic gradient optimization and thus inherits the efficiency of Stochastic Gradient Descent (SGD). To cast a stochastic optimization process to a sampling procedure, artificial noise needs to be added to the gradients. A typical example of gradient MCMC methods is Stochastic Gradient Langevine Dynamics (SGLD) [30], which injects Gaussian noise into the parameter gradients, pushing the optimization trajectory to asymptotically converge to the actual posterior distribution. In practice, some of these algorithms are also used in combination with several heuristic approximations, such as constant step sizes or diagonal approximations to the Hessian for the sake of simplicity or efficiency [25]. In Sec. 4.2 of this paper, we also use the simple SGLD with constant noise to optimize our augmentation distribution.

heading: 3 A Continuous and Differentiable Mapping

heading: 3.1 Preliminaries
text: Let the mapping φ : α 7→ tα map a d-dimensional vector α ∈ Rd to an image augmentation transformation tα ∈ T , where the augmentation tα(·) maps an image I to its transformed one tα(I). For convenience, we introduce the notation tα to denote the transformation function in T for which tα = φ(α). In other words, α could be regarded as the parameter of the transformation function tα, and φ is the realization function.

heading: 3.2 Constructing The Mapping
text: We start by developing a taxonomy of image augmentation transformations. Popular basic augmentations can be grouped into three categories: Color Adjustment that adjusts the color values pixel by pixel, Image Filtering like blurring or sharpening the whole picture, and Image Warping which applies geometric transformations to the image. Now we discuss how to construct a continuous and differential mapping for each augmentation categories respectively:
Color Adjustment is defined as a transformation in the spatial domain that is equally applied to each pixel. Let n be the image size. For every location (x, y) with pixel vector Ixy = [hxy sxy vxy]
T, it would be adjusted to I ′xy = [ h′xy s ′ xy v ′ xy ]T . The augmentation can be described as:
I ′ = tαCA(I), where I ′ xy = h′xys′xy v′xy  = αh + (1 + βh)(hxy)γhαs + (1 + βs)(sxy)γs αv + (1 + βv)(vxy)γv  (∀ x, y that 1 ≤ x, y ≤ n), (1)
where the nine-dimensional vector αCA = [αh βh γh αs βs γs αv βv γv]T maps to a Color Adjustment augmentation tαCA(·). The first three values control the hue, the next three would change the saturation, and the last three control the brightness. This representation is motivated by the Power law (Gamma) transformation function [24]. As shown in Equ. (1), this augmentation family is differential w.r.t. all α, β, γ since it can be expressed as power functions of them and both RGB-to-HSV and HSV-to-RGB conversions are known to be continuous and differentiable [26, 29].
Image Filtering is a typical image enhancement in the frequency domain. Here we consider using filters to blur or sharpen images. Let a single value s denote the degree of sharpening or smoothing. Let Gk be a sharpness filter obtained by zero-mean normalizing a k × k discrete Gaussian filter, and Ck be a k × k identity filter which has an unique non-zero value 1.0 at its center. The sharpening or smoothing process that transforms an image I to I ′ is:
I ′ = tαIF(I) = I ∗ (s ·Gk + Ck), (2)
where the asterisk ∗ means convolution and tαIF(·) is the Image Filtering function. If s is greater than zero, it represents sharpening. Otherwise, it is equivalent to blurring. Thus, a one-dimensional vector [αIF] T could map to an Image Filtering transformation.
Image filtering is differentiable w.r.t s by nature. However, the computation grows fast with respect to the filter size k. When k is close to the image size n, a better implementation for computing the convolution would be applying discrete Fast Fourier Transform (FFT) and multiplying the frequencies, which achieve a complexity of O(n2 log n). In this work, we take direct convolution since we pick k n for simplicity.
Image Warping refers to a geometric distortion on an image where pixels are moved to other locations without changing their pixel values. Perspective transformation (also known as projective transformation) is a popular warping method, which can be uniquely represented by a 3× 3 matrix H called homography [2]. Each point in the original image (x, y) would move to (x′, y′) with the perspective transformation. This Image Warping can be described as:
I ′ = tαIW(I)
⇐⇒ I ′x′y′ = Ixy, where  x′ = H11x+ H12y + H13 H31x+ H32y + H33 ,
y′ = H21x+ H22y + H23 H31x+ H32y + H33 ,
(∀ x, y that 1 ≤ x, y ≤ n), (3)
where tαIW(·) denotes the Image Warping augmentation via perspective transformation, and αIW = [H11 H12 H13 H21 H22 H23 H31 H32 H33]T could represent an Image Warping augmentation. Rotations, translations, shearing, scaling, as well as more single geometric transformations and any combinations of them are all special cases of perspective transformation. Perspective transformation is originally non-differentiable to each element in the homography H since index x, y take discrete values. However, we can leverage interpolation between pixels and inverse warping to make the transformation differentiable [17, 11, 27].
In summary, an augmentation vector α = [αTCA αTIF αTIW]T lies in continuous real space with d = 9 + 1 + 9 = 19 dimensions could map to a specific augmentation transformation tα = tαIW ◦ tαIF ◦ tαCA . Although composite order matters, for simplicity, leave the matter to future work. One can easily verify that many commonly used augmentation transforms [5, 6, 20, 16, 21, 19, 28], such as brightening, blurring, and rotation, are all discrete samples in the augmentation function space T of our mapping. Last but not least, practical constraint functions (tanh, exp) and clamp function are used before and after applying augmentations respectively, to ensure all augmentations are valid and all augmented pixels are in [0, 1]. The overview of this space is illustrated in Fig. 1. More details on the augmentation process of this figure are provided in Supplementary A.

heading: 3.3 An Example Random Augmentation Policy
text: An example random augmentation policy in our continuous augmentation space is provided here to evaluate the space by comparing this policy with other policies in discrete space. This random policy is based on a multivariate Gaussian distribution g(α) over α with zero means and diagonal covariance. The according augmentation distribution is denoted as prand(tα), where tα = φ(α) and α ∼ g(α). This policy is one of the simplest continuous random policy one could define.
In Tab. 1 we compare the example random policy with both random and optimized policies in discrete augmentation space to show the advance of continuity. Note that DADA [19] basically follows the same discrete space of AutoAugment’s [5]. Two example policies with different standard deviations (0.5 and 1.0) are evaluated for better comparison.

heading: 4 Automatic Augmentation Design using φ

heading: 4.1 ADA Problem formulation
text: Let t(·) denotes the augmentation function. ADA aims to find the optimal distribution of augmentations q(t), which improves the generalization of the model. Let Dt = {(xi,yi)}Nti=1 and Dv = {(x̃i, ỹi)}Nvi=1 be the training and validation datasets, respectively. Training a deep network fθ parameterized by θ is the process of minimizing a loss L(·, ·), which is cross-entropy in this work. We use pθ(y|xi) to refer to the probability distribution over labels by fθ for xi, i.e., the output of fθ followed by a softmax function. In general, ADA aims at solving the following bilevel optimization:
arg min q(t) ∑ (x̃i,ỹi)∈Dv L(fθ∗(x̃i), ỹi), (4)
s.t. θ∗ = arg min θ ∑ (xi,yi)∈Dt, t∼q(t) L(fθ(t(xi),yi), (5)
where q(t) is the distribution of augmentations, t(xi) denotes the augmented data of xi. The goal is to minimize the outer-level validation loss of a network fθ∗ in (4) that is trained by minimizing the inner-level training loss using augmented training data in (5).
Notably, by writing the cross-entropy explicitly, we will have:
L(fθ(x), y) = − log(pθ(y|x)), (6)
for any model parameter θ, and any labed data pair (x, y).

heading: 4.2 Gradient-Based ADA in Continuous Augmentation Space via SGLD
text: ADA in continuous space. Previous work mostly defines q(t) as a parameterized categorical distribution or a discrete uniform distribution. In this work, given the proposed continuous mapping φ : Rd → T where T is the set of augmentations, we are now able to relate the augmentation distribution q(t) for t ∈ T to a continuous distribution p(α) for α ∈ Rd by t = tα = φ(α). Thus we can reform the bilevel optimization (4)(5) by substituting the target distribution q(t) with p(α) and apply the realized transformation tα to training data xi then get (7)(8):
arg min p(α) ∑ (x̃i,ỹi)∈Dv L(fθ∗(x̃i), ỹi), (7)
s.t. θ∗ = arg min θ ∑ (xi,yi)∈Dt, α∼p(α) L(fθ(tα(xi),yi). (8)
One-step simplified inner-level. Notably, directly solving the bilevel optimization in (7)(8) would be prohibitive since it requires hundreds or thousands of full model training and validation. Thus, we refer to the successful augmentation-wise weight-sharing [28] and one-step meta learning [4, 10, 23] techniques to make this problem more tractable.
Specifically, we begin with an augmentation-wise shared parameter θs on Dt using the random augmentation policy prand(α) introduced in Sec. 3.3. We replace the inner optimization (8) with a pseudo one-step update: a single gradient step for a batch of training samples Bt = {(xi,yi)}Mi=1 augmented by a given tα is taken to get θ̄s that approximates the θ∗, where M is the batch size. The inner optimization in (8) now becomes:
θ∗ ≈ θ̄s = θs − λ ∑
(xi,yi)∈Bt
∇θsL(fθs(tα(xi),yi). (9)
Bayesian formulation for ADA. In this setup, ADA boils down to a continuous distribution optimization problem (7)(9). In order to apply gradient-based methods, we drive a new Bayesian formulation for distribution optimization in ADA . Let pprior(α) be the prior distribution and p(α|Dv) be the posterior distribution. The optimization in (7) for obtaining the target distribution p(α) is modeled in our Bayesian formulation by inferring the posterior distribution p(α|Dv) as follows:
p(α|Dv) ∝ pprior(α) Nv∏ i=1 p ( (x̃i, ỹi)|α ) , (10)
where ∏Nv i=1 p ( (x̃i, ỹi)|α ) is the data likelihood of the validation set given a fixed augmentation
parameter α. After obtaining θ̄s using Equ. (9) for a given α, this likelihood now can be computed efficiently as ∏Nv i=1 pθ̄s(ỹi|x̃i). Note that the inner-level optimization is efficient as we always start with the same θs and then take a single gradient step over the given batch under the augmentation tα = φ(α) to obtain θ̄s. Assuming uniform prior pprior(α), the posterior in (10) can be computed as:
p(α|Dv) ∝ Nv∏ i=1 pθ̄s(ỹi|x̃i). (11)
Use gradients to search for augmentation policy. With the Bayesian formulation and the proposed differentiable mapping φ, Gradient-MCMC methods like Stochastic Gradient Langevin Dynamics (SGLD) is now applicable to find the target distribution by sampling. Recall Equ. (6), the j-th iteration of SGLD sampling on the validation batch Bv = {(x̃i, ỹi)}Mi=1 can be described as:
α(j+1) = α(j) + η +
2
M
Nv ∑ (x̃i,ỹi)∈Bv ∇α(j) log pθ̄s(ỹi|x̃i) (12)
= α(j) + η + Mλ
2Nv ∑ (x̃i,ỹi)∈Bv ∑ (xk,yk)∈Bt ∂L(fθ̄s(x̃i), ỹi) ∂θ̄s ∂2L(fθs(tα(j)(xk),yk)) ∂θs∂α(j) , (13)
where is the step size of SGLD and η ∼ N (0, ) is the Gaussian noise.
In practice, one can use finite difference [13, 23] to compute the second-order derivative in (13) more efficiently. By iterating Equ. (13), samples of α can be collected and SGLD would asymptotically converge to the posterior distribution p(α|Dv) [32], which is the resulted augmentation policy.
The overview. The whole search pipeline of our method (MCMC-Aug) is summarized in Alg. 1. The augmentation search phase of a simplified (suppose d = 2) MCMC-Aug is also visualized in Fig. 2. The samples {α(1),α(2), · · · ,α(Nj)} collected by SGLD form the resulted augmentation policy. When using the policy to train a new model, one would pick randomly from these samples to augment each training image. In Sec. 5 and Supplementary E. we will evaluate and visualize the searched policy in detail.
Algorithm 1 A Reference Method for ADA : MCMC-Aug Inputs: Training set Dt, validation set Dv , pretrained weights θs, samples of the augmentation vector S; Hyperparameters: Number of updates Nj , mini batch size M , model learning rate λ, SGLD step size ; Initiate: S := ∅; α(0); for j = 0, 1, 2, · · · , Nj − 1 do
Sample bastches Bt and Bv; Perform pseudo one-step update in Equ. (9) to obtain θ̄s; Perform SGLD update in Equ. (13) to get α(j+1); S := S ∪ {α(j+1)};
end for return S;

heading: 5 Experiments

heading: 5.1 Implementation Details
text: We evaluate our method on CIFAR-10/100 [18] and ImageNet [7]. For each datasets, a validation set is split from the training set and the testing set is only used for evaluating the final performance (not involved in the search phase). For ImageNet, a reduced ImageNet-120 is used for fair comparison [5, 20, 16, 19]. All batch sizes for augmentation search mentioned in this work are set to 256 for CIFARs and 1024 for ImageNet as standard. More details are included in Supplementary. B.
Search. Following prior work, we search policy for two base models, which are Wide-ResNet-40-2 [31] and ResNet-50 [15] on CIFARs and ImageNet respectively. Before we start, an instance of each base model is pretrained (CIFARs/ImageNet) using the random policy introduced in Sec. 3.3 to obtain the shared weights θs in Equ. (9). This procedure lasts 200 epochs for CIFARs and 120 epochs for ImageNet. Then we run our MCMC-Aug with 200 epochs, the same as [16] over the training set. Step size and noise scale of SGLD is set according to [25]. The experiments reported have a fixed 0.4
step size, and a noise rate of 2× 10−5. Further analysis on the choice of hyperparameters could be found in Supplementary D., where we provide a sensitivity analysis as well as a guideline for tuning SGLD. On CIFAR-100 and ImageNet, the same SGLD hyperparameters are adopted without any modification.
Evaluation. We evaluate MCMC-Aug by measuring the performance of a reinitialized model trained with augmentation from the SGLD samples {tα(1) , tα(2) , · · · , tα(Nj)}. Theoretically, SGLD will be in its Langevin dynamics phase and sampling approximately from the posterior distribution when its sample threshold statistic is much smaller than 1.0 [30]. However, in practice, we found that collecting samples of α from the later half of the SGLD trajectory is sufficient to obtain good performance. We run SGLD only for base models, and the collected samples could be used to retrain other models to assess the transferability. Thus Wide-ResNet-{40-2, 28-10} [31], Shake-Shake (26 2×32d) [12], PyramidNet+ShakeDrop [14] are retrained on CIFARs, and ResNet-{50, 200} on ImageNet. Each of experiments
is repeated four times and the mean value is reported, see Supplementary B. for the error bar. Other hyperparameters not mentioned here are directly imported from [28], provided in Supplementary C.

heading: 5.2 Comparison with State-of-the-Arts
text: Summary. We conducted comprehensive comparisons with state-of-the-art augmentation methods that includes Cutout [8], AutoAugment (AA) [5], Fast AutoAugment (FAA) [20], Rand Augment (RA) [6], Population Based Augmentation (PBA) [16], and OHL-AutoAug (OHL) [21]. Furthermore, we compare MCMC-Aug with two of the latest ADA methods: 1) AWS-AutoAug (AWS) [28], which is currently the best-performing ADA method reported; 2) Differentiable Automatic Data Augmentation (DADA) [19], which is currently the fastest method that applies a continual relaxation to the discrete search space using Gumbel-softmax.
Performance comparison. The performance of MCMC-Aug and other methods are compared in Tab. 2 and Tab. 3. The performance of the other methods is directly obtained from the corresponding papers. We use dashes wherever the results are not reported. Following prior work [5, 19], “Baseline” in the tables refers to a model trained using only default pre-processing like normalization and basic augmentations such as random cropping and random horizontal flipping. The same pre-processing and basic augmentations are also uniformly applied to all the methods. Cutout is used in all the ADA methods on CIFAR datasets.
Tab. 2 and Tab. 3 show that MCMC-Aug significantly reduces the error rates over the baseline and Cutout on a variety of network architectures, demonstrating the effectiveness and transferability of the resulted policies. Compared with other ADA methods, MCMC-Aug performs on par with AWS while outperforming the others, even though we did not conduct any comprehensive parameter tuning. Moreover, in Sec. 5.3, we show that increasing the number of searching epochs (proportional to Nj in Alg. 1) brings extra performance gains. Since we had only trained for 200 epochs for efficiency, we believe there is still room for improvements.
Efficiency comparison. We compare the computational cost of MCMC-Aug with others to check if MCMC-Aug inherits the efficiency of gradient-based optimization. Results on both the reduced and the full CIFAR-10 dataset are reported to ensure fairness, since previous work had reported efficiency of their methods on one or both of the datasets. We used the same reduced CIFAR-10 subset in [5, 20, 16, 19], which consists of 4,000 images from the training set. We kept the same hyperparameters except for a smaller number of epochs in searching (five) and linearly scaled batch size and learning rate (four times). The results are summarized in Tab. 4 and visualized in Fig. 3.
As shown in Tab. 4, OHL, FAA and DADA are much faster than AA, but at the expense of accuracy. AWS achieves the best performance, but is much more computationally expensive. Like PBA, MCMC-Aug strikes a good balance between accuracy and efficiency, but yields much better accuracy. We also note that there is a noticeable gap between MCMC-Aug’s performance on the full and reduced dataset, showing the potential for further performance gains when more computational budget becomes available. Overall, our results show that MCMC-Aug is the best in class when considering the need to balance performance and efficiency.

heading: 5.3 Ablation Study
text: Space decomposition. To show the necessity of three base catogories of augmentation mapping, we remove two categories of transformation in turns, after which we look at the change in test accuracy.
As reported in Tab. 5, the performance of Wide-ResNet-40-2 on CIFAR-10 would dramatically decrease, which suggests that each of them is indispensable.
SGLD early-stopping. To investigate the optimization behavior of the SGLD, we stop collecting the SGLD samples at 25%, 50%, and 75% of the total epochs and evaluate the corresponding performance respectively. As listed in Tab. 6, we see that the errors of the distributions arising from early-stopping gradually increase. It shows the performance gain along with the SGLD procedue. This also suggests that increasing the number of searching epochs of our approach might bring extra performance gains.
For the sensitivity of hyperparameters and more details on the searched distribution, please see Supplementary D. and E.

heading: 6 Limitation
text: Non-differentiable augmentations that do not belong to any categories of the three are a clear limitation. However, an interpolation between the original image and its augmented counterpart could be made to bypass this issue. Suppose we have an image I and a non-differentiable transformation t, we can introduce a continuous parameter λ and create a new augmentation t′ via t′(I) = λ t(I) + (1− λ) I . t′ is a surrogate augmentation which ensures the differentiability.
The presence of a clean validation set is vital to our MCMC-aug as well as other automated data augmentation algorithms. This is because the loss or accuracy on validation set is a common criteria for searching the augmentation policy. We expect to see a better approach to reduce the dependency on validation set in the future. It is a valuable direction and ought to be further explored.

heading: 7 Conclusion
text: A continuous mapping to model a continuous space of augmentation is proposed. The continuous mapping can be applied to many augmentation design tasks (e.g., instance discrimination contrastive learning, image alignment, and domain adaptation). We apply this mapping to pose ADA as a continuous augmentation search problem. Furthermore, we setup a Bayesian formulation for ADA to enable the application of efficient gradient-based MCMC, presenting a new way to conduct ADA search via posterior sampling. Comprehensive comparison and studies verify its efficiency and effectiveness. Both source codes and checkpoints will be released to the public, and future work may concern more image modalities like infrared/X-rays/ultrasound imaging, more diverse augmentations, or wider applicants, e.g., in other computer vision areas or the times-series processing.

heading: 8 Funding Disclosure and Acknowledgement
text: This work is supported by the EPSRC grant: Turing AI Fellowship: EP/W002981/1, EPSRC/MURI grant EP/N019474/1. Chen Lin is supported by Facebook AI. We would also like to thank the Royal Academy of Engineering and FiveAI.

Review summaries:
    Authors present a continuous formulation of data augmentation for     images. The formulation is used to define a Bayesian model for augmentation     parameters. The Bayesian model does not rely on a bi-level optimization     problem as commonly used for optimizing data augmentation. Instead, only a     single gradient step approximates a full optimization of a network with a     given set of augmentation parameters. Gradient step is taken starting from     pretrained weights determined through initial training using random     augmentation. Samples from the posterior distribution are gathered using     Langevin dynamics MCMC with random batches. Samples are used to train     the ultimate networks. Experiments show that the proposed model outperforms     existing techniques in terms of accuracy of the final models at a fraction     of computation time. 

The paper presents an automated data augmentation technique that optimizes a distribution over a continuous augmentation space. Augmentation is formulated as learning a continuous mapping from an augmentation vector in R^d to an image transformation that composes three general families/modes of image transformations (color adjustment, image filtering, and image warping).   The proposed method uses a bilevel optimization approach where the primary learning task is trained on an augmented training set and the augmentation learning task is trained on the validation set. To make the optimization tractable, the inner level training is replaced by a one-step gradient update on an augmented mini-batch with model parameters being shared across the bilevel optimization iterations, i.e., incrementally updated during the bilevel optimization. A comprehensive set of experiments, including ablation experiments and comparisons to SOTA methods for automated data augmentation, demonstrate the efficacy and efficiency of the proposed method. 

This contribution is aimed at automated data augmentation. A continuous search space is defined using simple image operators, and thus enables learning of augmentation parameters. A randomised algorithm is proposed for parameter learning.

The authors pose the problem of learning data augmentation as a continuous optimization problem and introduces a Bayesian approach for learning and sampling augmentations. The method allows for optimizing over a space of infinitely many possible transformations in contrast to the majority of automated data-augmentation approaches (ADA) which are limited by discretization of the solution space. The values of the proposed approach over prior works in terms of prediction accuracy and speed have been demonstrated on a range of benchmarks.