RangeAugment:  Efficient Online Augmentation with Range Learning
Authors: Sachin Mehta
Saeid Naderiparizi
Fartash Faghri
Maxwell Horton
Lailin Chen
Ali Farhadi
Oncel Tuzel
Mohammad Rastegari
Venue: Submitted to ICLR 2023 (2023)
Keywords: Automatic Augmentation

Abstract: State-of-the-art automatic augmentation methods (e.g., AutoAugment and RandAugment) for visual recognition tasks diversify training data using a large set of augmentation operations. The range of magnitudes of many augmentation operations (e.g., brightness and contrast) is continuous. Therefore, to make search computationally tractable, these methods use fixed and manually-defined magnitude ranges for each operation, which may lead to sub-optimal policies. To answer the open question on the importance of magnitude ranges for each augmentation operation,  we introduce RangeAugment that allows us to efficiently learn the range of magnitudes for individual as well as composite augmentation operations. RangeAugment uses an auxiliary loss based on image similarity as a measure to control the range of magnitudes of augmentation operations. As a result, RangeAugment has a single scalar parameter for search, image similarity, which we simply optimize via linear search. RangeAugment integrates seamlessly with any model and learns model- and task-specific augmentation policies. With extensive experiments on the ImageNet dataset across different networks, we show that RangeAugment achieves competitive performance to state-of-the-art automatic augmentation methods with 4-5 times fewer augmentation operations. Experimental results on semantic segmentation and contrastive learning further shows RangeAugment's effectiveness.

Under review as a conference paper at ICLR 2023
RANGEAUGMENT: EFFICIENT ONLINE AUGMENTA- TION WITH RANGE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-art automatic augmentation methods (e.g., AutoAugment and Ran- dAugment) for visual recognition tasks diversify training data using a large set of augmentation operations. The range of magnitudes of many augmentation op- erations (e.g.. brightness and contrast) is continuous. Therefore, to make search computationally tractable, these methods use fixed and manualyy-defind mag- nitude ranges for each operation, which may lead to sub-optimal policies. To answer the open question on the importance of magnitude ranges for each aug- mentation operation, learn the range of magnitudes for individual as well as composite augmentation operations. Ra Rggeent uses auxiliary loss based on image similarity a measure to control the range magnitudes augmentation operations. As a re- sult, RangeAugment has single scalar parameter for search, image similarity, which we simply optimize via linear search. RangeAugment integrates seam- lessly with any model and learns model- and task-specific augmentation policies. With extensive experiments on the ImageNet dataset across different networks, we show that RangeAugment achieves competitive performance state-of-the- automatic augmentation methods with 4-5 times fewer augmentation operations. Experimental results on semantic segmentation and contrastive learning further shows RangeAugment 's efffcctiveness.
introduce RangeAugment that allows us to efficiently
INTRODUCTION
Data augmentation a widely used regularization method for training deep neural networks (LeCun et al., 1998; Krizhevsky et al., 2012; Szegedy et al., 2015; Perez & Wang, 2017; Steiner et al., 2021). These methods apply carefully designed augmentation (or image transformation) operations (e.g., color transforms) to increase the quantity and diversity of training data, which in turn helps improve the generalization ability of models. However, these methods rely heavily on expert knowledge and extensive aial--nn-erroo experiments.
Recently, automatic augmentation methods have gained attention because their ability to search for augmentation policy (e.g.. combinations of different augmentation operations) that maximizes validation performance (Cubuk et al., 2019; 2020; Lim et al., 2019; Hataya et al., 2020; Zheng et al., 2021). In general, most augmentation operations (e.g., brightness and contrast) have parame- ters: (1) the probability of applying them and (2) their range of magnitudes. These methods take set of augmentation operations with fixed (often discretized) range of magnitudes and produce a policy of applying some or all augmentation operations along with their parameters (Fig. 1). As an example, AutoAugment (Cubuk et al., 2019) discretizes the range of magnitudes and probabilities of 16 augmentation operations, and searches for sub-policies (i.e., composition of two augmentation operations along with their probability and magnitude) in a space of about 1032 possible combinations. These methods empirically show that automatic augmentation policies help improve performance of downstream networks. For example, AutoAugment improves the valida- tion top-1 accuracy of ResNet-50 (He et al., 2016) by about 1.3% on the ImageNet dataset (Deng et al., 2009). other words, these methods underline the importance of automatic composition of augmentation operations in improving validation performance. However, policies generated using these networks may be sub-optimal because they use hand-crafted magnitude ranges. The impor- tance of magnitude ranges for each augmentation operation is still open question. An obstacle in
input,

---

Under review as a conference paper at ICLR 2023
Modk opectt polley
space
Search stratigy
Performance estimation
Maria rage mape toles
opcindic
Keymie e Oradee suctiag
Asprooationn rapitall,
Standard automatic augmentation
lagores peranitaaio ability
Search
Search strategy
Performance estimation
Tadloppctii polcy
Tagel laage aialanty
Vakaa acorao
Fnttae 5
(b) RangeAugment
Figure I: Comparison between RangeAugment and standard automatic augmentation methods. RangeAugment's 's search space is independent of augmentation parameters, allowing us to learn model- and task-specific policies in a constant time.
Brightness
Contrast
Noise
Brightnes
Cuntrast
Noise
Model: SwinTransfomer
Classifiatiin
Dataset ImageNNe
Model: MohileNard
Classifiaation
ImagoNet
Model: ElficienNNet
Tesk: Classification
Dulaset: ImageNet
Model: MobileNesel
Task:
Segmeniation
Datoset ADE20K
(a) Model-specific policy
(b) Task-specific policy
Figure 2: RangeAugment learns model- and task-specific policies. (a) shows the range of magni-
tudes for two different models the same task and dataset while (b) shows the range of magnitudes for the same model on two different tasks. All models are trained end-to-end with RangeAugment. The target image similarity A (PSNR) is annealed from 40 to 5 in (a) and from 40 to 20 in (b).
answering this question is the range of magnitudes for most augmentation operations is continuous, which makes the search computationally intractable.
This paper introduces RangeAugment, simple and efficient method to learn the range of magni- tudes for each augmentation operation. Inspired by image similarity metries (Hore & Ziou, 2010), RangeAugment introduces an auxiliary augmentation loss that allows us to learn the range of magnitudes for each augmentation operation. We realize this by controlling the similarity between the input and the augmented image for given model and task. Rather than directly specifying the parameters for each augmentation operation, Range takes a target image similarity value input. The loss function is then formulated combination of the empirical loss and an augmentation loss. The objective of the augmentation loss is to match the target image sim- ilarity value. Therefore, the search objective in RangeAugment is to find the target similarity value that provides a good trade-off between minimizing the augmentation loss (i.e., matching the target similarity value) and the empirical loss. As result, the augmentation policy learning in RangeAugment reduces to searching for a single scalar parameter, target image similarity, that maximizes downstream model's validation performance. We search for this target image similarity value via linear search. Empirically, we observe that this trade-off between the augmentation and empirical loss leads to better generalization ability of downstream model. Compared to existing automatic augmentation methods that require large set of augmentation operations (usually 14-16 operations), RangeAugment is able to achieve competitive performance with only three simple operations (brightness, contrast, and additive Gaussian noise). Because RangeAugment's search space is independent of augmentation parameters and is fully differentiable (Fig. 1), it can be trained end-to-end with any downstream model to learn model- and task task-specific policies (Fig. 2).
We empirically demonstrate in Section 4 that RangeAugment. allows us to learn model-specific policies when trained end-to-end with downstream models the ImageNet dataset (Fig. 2a). Im-
2

---

Under review as a conference paper at ICLR 2023
portantly, RangeAugment achieves competitive performance to existing automatic augmentation methods (e.g., AutoAugment) with 4 to 5 times fewer augmentation operations. In Section 5, we apply RangeAugment to semantic segmentation and contrastive learning to demonstrate its sim- plicity and seamless integration ability to different tasks. We further show that RangeAugment learn task-specifie policies (Fig. 2b). To the best of our knowledge, RangeAugment. is the first au- tomatic augmentation method that learns the range of magnitudes for each augmentation operation.
2 RELATED WORK
Data augmentation combines different augmentation operations (e.g., random brightness, random contrast, random Gaussian noise, and data mixing) to synthesize additional training data. Traditional augmentation methods rely heavily expert knowledge and extensive trial-and-enee and-error experiments. In practice, these manual augmentation methods have been used to train different models on a variety of datasets and tasks (e.g., Szegedy et al., 2015; He al., 2016; Zhao al., 2017; Howard al., 2019). However, these policies may not be optimal for all models.
Motivated by neural architecture search (Zoph & Le, 2017), recent methods focus on finding optimal augmentation policies automatically from data. AutoAugment formulates automatic augmentation as a reinforcement learning problem, and uses model's validation performance as reward to find an augmentation policy leading to optimal validation performance. Because AutoAugment searches for several augmentation policy parameters, the search space enormous and computationally in- tractable on large datasets and models. Therefore, in practice, policies found for smaller datasets are transferred to larger datasets. Since then, many follow-up works have focused on reducing the search space while delivering similar performance to AutoAugment (Ratner et al., 2017; Lemley et al., 2017; LingChen et al., 2020; Li et al., 2020; Zheng et al., 2021; Liu et al., 2021a). The first line of research reduces the search time by introducing different hyper-paree parameerrrrter optimization methods, including population- based training (Ho et al., 2019), density matching (Lim et al., 2019; Hataya al., 2020), and gradient matching (Zheng al., 2021). The second line of research reduces the search space by making practical assumptions (Cubuk et al., 2020; Müller & Hutter, 2021). For instance, RandAugment (Cubuk et al., 2020) applies two transforms randomly with uniform prob- ability. With these assumptions, RandAugment reduces AutoAugment' 's search space from 10 to 102 while maintaining downstream networks performance.
One common characteristic among these different automatic augmentation methods is that they use fixed and canuulll--eefnnee range of magnitudes for different augmentation operations, and focus on diversifying training data by using a large set of augmentation operations (e.g., 14 transforms). This is because the range of magnitudes for most augmentation operations is continuous and large, and searching over this large range is practically infeasible. Unlike these works, RangeAugment focuses on learning the magnitude range of cach augmentation operation (Figs. I and 2). We show in Section 4 that RangeAugment is able to learn model- and task-specific policies while delivering competitive performance to previous automatic augmentation methods across different models.
3 RANGFAUGMENT
Existing automatic augmentation methods search for composite augmentations over a large set of augmentation operations, but each augmentation operation has a manually-defined range of magni- tudes. This paper introduces Ra RangeAugmnt,, a method for learning a range of magnitude for each augmentation operation (Fig. 3). RangeAugment uses image similarity between the input and augmented image learn the range of magnitudes for each augmentation operation. In the rest of this section, we first formulate the problem (Section 3.1) and then elaborate on RangeAugment 's policy learning method (Section 3.2), followed by implementation details (Section 3.3).
3.1 PROBLEM FORMULATION
Let T (Ti,... TN} be set of N differentiable augmentation operations. Each augmen- tation operation T E T is parameterized by scalar magnitude parameter m E R such that T): X + X' is defined on the image space X. Let T. be an augmentation policy that de- fines distribution over sub-policies S ~ Tÿ in RangeAugment such that S = (Ti(: m)) A
3

---

Under review as a conference paper at ICLR 2023
Sapur image
Augreeenel image
angroenations
Siniboity
Taggr Imuge sinitarit
Asgenoriation Boa
Yack loss Coall
Tesk label
Figure 3: RangeAugment: End-to-end learning of augmentation policy with downstream model.
sub-policy S applies augmentation operations to an input image with uniform probability as
S(x) = x(N) (()) (x (
For any given model and task, the goal of automatic augmentation is to find the augmentation pol- icy Ty that diversifies training data, and helps improve model's generalization ability the most. RangeAugment learns the range of magnitudes for each augmentation operation in T. Formally, the policy parameters RangeAugment are e = {(a; bi) and the magnitude parameter my ~ Ua; b,) for the i-th augmentation operation in a sub-policy S = formly sampled, where E I and b; E R are learned parameters.
is uni-
3.2 POLICY LEARNING
Diverse training data can be produced by using wider range of magnitudes, (a;, b.), for each aug- mentation operation. However, directly searching for the optimal values of (a, b,) for each model and dataset challenging because of its continuous nature. To address this, RangeAugment intro- duces auxiliary loss which, in conjunction with the task-specific empirical loss, enables learning model-specific range of magnitudes for each augmentation operation in an end-to-end fashion.
Let X'x TR. be differentiable image similarity function that measures the similarity between the input and the augmented image. To control the range of magnitudes for each augmentation operation, RangeAugment minimizes the distance between the expected value of d(x, S(x)) and a target image similarity value A E R using an augmentation loss function L, (e.g., smooth LI loss or L2 loss). An example of and are PSNR and target PSNR value respectively. When target PSNR value is small, the difference between the input and augmented image obtained after applying an augmentation operation (say brightness) will be large. In other words, for smaller target PSNR value, the range of magnitudes for brightness operation will be wider and vice-versa.
For given value of A and parameterized model fo with parameters the overall loss function to learn model- and task-specific augmentation policy is weighted sum of the augmentation loss Lm and the task-specific empirical loss Luski
8" arg min E(x.y)~DDmn ESS (foss + Ax,(x,, S(x);A]]]
0.6
where À and Demin represent weight term and training set respectively. Note that, in Eq. (2), re-parameterizatioo trick uniform distributions (Kingma & Welling, 2013) is applied to back- propagate through the expectation over Tt.
The in Eq. (2) allows RangeAugment control diversity of augmented samples. Therefore, the augmentation policy learning in RangeAugment reduces to searching a single scalar parameter, A, that maximizes downstream model's validation performance. RangeAugment finds the optimal value of A using linear search.
3.3 IMPLEMENTATION DETAILS
We use PSNR as the image similarity function in our experiments because it is (1) standard image quality metric, (2) differentiable, and (3) fast to compute. Across different downstream networks, we observe 0.5% --5%%%%% training overhead over the empirical risk minimization baseline.
4

---

Under review as a conference paper at ICLR 2023
To find an optimal value of A in Eq. (2), study two approaches: (1) fixed target PSNR (A E {5, 10, 20, 30}) and (2) target PSNR with curricu- lum, where the value of A annealed from 40 to 5 and 8 (5, 20, 30). The learned ranges of magnitudes, (a, b). can scale beyond the image space (e.g., negative values) and result in training instabil- ity. To prevent this, we clip the range of magnitudes they are beyond extreme bounds of augmentation operations. We choose these extreme bounds such that objects an image are hardly identifiable at or beyond the extreme points of the bounds (see Fig. 4). Also, because the focus of RangeAugment is to learn the range of magnitudes for each augmen- tation operation, we apply all augmentation operations in T with uniform probability.
Figure 4: Example outputs of brightness op- eration, T(x; m) mx, at different values of magnitude parameter m. At extremes (i.e., m 0.1 or 10), the bus is hardly identifiable.
To demonstrate the importance of magnitude ranges, we study RangeAugment with three basic operations (brightness, contrast, and additive Gaussian noise), and show empirically in Section that RangeAugment can achieve competitive performance to existing methods with to times fewer augmentation operations.
4 EVALUATING RANGEAUGMENT ON IMAGE CLASSIFICATION
RangeAugment learn model-specific augmentation policies. To evaluate this, first study the importance of single and composite augmentation operations using ResNet-50 on the ImageNet dataset (Section 4.2). We then study model-level generalization of RangeAugment. (Section 4.3).
4.1 EXPERIMENTAL. SET-UP
Dataset For image classification. we use the ImageNet dataset that has 1.28M training and 50k validation images spanning across 1000 categories. We use top-1 accuracy to measure performance.
Baseline models To evaluate the effectiveness of RangeAugment, we study different CNN- and transformer-based models. We group these models into two categories based on their complexity: (1) mobile: MobileNetvl (Howard et al., 2017), MobileNetv2 (Sandler et al., 2018), MobileNetv3 (Howard et al., 2019), and Mobile (Mehta & Rastegari, 2021) and (2) non-mobile: ResNet- 50 (He et al., 2016), ResNet-101, EfficientNet (Tan & Le, 2019), and SwinTransforma 2021b). We implement RangeAugment using the CVNets library (Mehta et al., 2022) and use their baseline model implementations and training recipes. Additional experiments, including the effect of individual and joint learning on RangeAugment's performance, along with training details are given Appendices C and F. respectively.
(Liu et al.,
Baseline augmentation methods For an apples to apples comparison, each baseline model is trained with three different random seeds and the following baseline augmentation strategies: (1) Baseline standard Inception-style pre-processing (random resized cropping and random horizontal flipping) (Szegedy et al., 2015), (2) RandAugment Baseline pre-processing followed by Ran- dAugment policy of Cubuk et al. (2020), (3) AutoAugment Baseline pre-processing followed by AutoAAgment policy of Cubuk al. (2019), and (4) RangeAugment Baseline pre-processing following by the proposed augmentation policy in Section 3.'
4.2 AUGMENTATION OPERATION CHARACTERIZATION USING RANGEAUGMENT
The quantity and diversity of training data can be increased by (1) increasing the magnitude range of each augmentation operation and (2) composite augmentation operations. Fig. characterizes the
"s noted Section 2, most previous works have focused on reducing search time of AutoAugment while achieving similar performance (e.g.. ResNet-50 on ImageNet: 77.6% (AutoAugment), 77.6% (RandAug- ment), and 77.6% (Fast AutoAugment). Also, many state-of ohe-hhe-rt models (e.g.. EfficientNet and Swin Trans- former) have used either RandAugment or Autment for data regularization. Therefore, to demonstrate the effectiveness of Ra negeAggmen,, choose Autmnugment and Randengment baseline methods.
5

---

Under review as a conference paper at ICLR 2023
Dag 151
1ap.1 (15)
(a) N
(b) N =
(c) N =
Figure 5: The performance of ResNet-50 on the ImageNet dataset when data diversity is increased by learning the range of magnitudes for single (N = 1) and composite augmentation operations
(N > 1) using RangeAugment. For curriculum learning, target PSNR value is annealed from 40
to the value mentioned in the legend.
Brightness
Contrast
Noise
Brightness
Contrast
Noise
5
15
(a) N = (without curriculum)
(b) N 3 (without curriculum)
Brightinss
Contrad
42 5
Fym.
40 1:
(c) N 3 (with curriculum)
Learned range of magnitudes for different augmentation operations using RangeAugment. In (a) and (b), the target image similarity value (PSNR; A) is fixed while in (c), A is annealed using cosine curriculum. The magnitude range in y-axis is in log scale.
Figure 6:
effect of these variables on ResNet-50's performance on the ImageNet dataset. We can make the following observations:
Fig. 6a shows that single augmentation operation (N 1) with wider magnitude ranges (e.g.. the range of magnitudes at target PSNR of 5 are wider than the ones at target PSNR of 15) helped in improving ResNet-50's validation accuracy and reducing training accuracy, thereby improving its generalization capability (Fig. 5a). Particularly, increasing the magnitude range of contrast (or brightness) operation increased ResNet-50's validation performance over baseline by 1.0% (or 0. 0.7%) while decreasing the training performance by 2% (or 3%). This is likely because wider magnitude ranges of an augmentation operation increases diversity of training data. On the other hand, the additive Gaussian noise operation with wider magnitude range slightly dropped the validation accuracy, but still reduces the training accuracy. In other words, it improves ResNet- 50's generalization ability.
2. Composite augmentation operations (N 1; Figs. 5b and 5c) reduces the training accuracy significantly while having a validation performance similar to single augmentation operation. This is expected composite operations further increases training data diversity.

---

Under review as a conference paper at ICLR 2023
3. Fig. 6C shows that progressively learning to increase the diversity of a samples (i.e., narrower to wider magnitude ranges²) using a cosine curriculum further improves the perfor- mance (Fig. 5c). A plausible explanation is that the learned magnitude ranges of different aug- mentation operations using RangeAugment with fixed target PSNR may get stuck near poor solutions. Because the range of magnitudes are wider for these solutions (e.g., the range of mag- nitudes at target PSNR value of 5 in Fig. 6a & Fig. 6b), RangeAugment samples more diverse data from the beginning of the training, making training difficult. Gradually annealing target PSNR from high to low (e.g., 40 to 5 in Fig. 6c) allows RangeAugment to increase the data diversity slowly, thereby helping model to learn better representations. Importantly. it also al- lows RangeAugment to identify useful ranges for each augmentation operation. For example, the range of magnitudes for the additive Gaussian noise operation relatively narrower when optimizing with curriculum (e.g., annealing target PSNR from 40 to 5; Fig. 6c) compared to op- timizing with a fixed target PSNR of 5 (Fig. 6b). This indicates that noise operation with narrow magnitude range is favorable for training ResNet-50 on the ImageNet classification task. This concurs with results in Figs. 5a and 5b where we observed that noise operation does not improve ResNet-50's validation performance. Moreover, our findings with progressively increasing data diversity are consistent with previous works (e.g., Bengio et al., 2009; Tan & Le, 2021) that shows scheduling training samples from easy to hard helps improve model's performance.
Interestingly. ResNet-50 with RangeAugment (N = 3) achieves comparable validation accu- racy and smaller generalization gap as compared to ofate-of-the-ar methods which use more augmentation operations (N > 14) to increase data diversity during training. We conjecture that the difference in the generalization gap between existing methods and RangeAugment is probably caused by insufficient policy search in existing methods as they use manually-defined magnitude ranges for each augmentation operation during search.
Observation I:
Composite augmentation operations with wider magnitude ranges is important for improving downstream model's generalization ability.
In the rest of experiments, we will use all three augmentations (NV = 3) with cosine curriculum.
4.3 MODEL-LEVEL GENERALIZATION OF RANGEAUGMENT
Fig. 5 shows RangeAugment is effective for ResNet-50. Natural questions that arise are:
1. Can RangeAugment be applied to other vision models? RangeAugment's 's seamless integra- tion ability with little training overhead (0.5% to 3%) over the baseline model allows us to study the generalization capability of different vision models easily. Fig. 7 shows the performance of different models with RangeAugment. When data regularization is increased for mobile models by decreasing the target PSNR value from 40 to 5, the training well validation ac- curacy of different mobile models is decreased significantly as compared to the baseline. This is likely because of the limited capacity of these models. On the other hand, data regularization im- proved the performance of non-mobile models significantly. Consistent with our observations for ResNet-50 in Fig. 5, we found that non-mobile models trained with RangeAugment are able to achieve competitive performance to sttteoffthe--a automatic augmentation methods, such as AutoAugment (N- = 16), but with fewer augmentations (N- 3).
Does RangeAugment learn architecture-speciin augmentations? Fig. 2a shows the learned magnitude ranges for different augmentation operations for transformer- and a CNN-based (EfficientNet) model. Though both of these architectures use the same cur- riculum in RangeAugment (i.c., target PSNR is annealed from 40 to 5), they learn different magnitude ranges for each augmentation operation. This shows that RangeAugment is capable of learning model-specific magnitude ranges for each augmentation operation.
3. Does RangeAugment increase variance on model performance? Because of the stochas- tic training and presence of randomness during different stages of training including RangeAugment, there may be some variability in model's performance. To measure the vari- ability in model's performance, we run each experiment with three different random seeds. For different models, the standard deviation of model's validation accuracy is between 0.01 and 0.2,
Widdr magnitude ranges produce diverse augmented samples and vice versa (Appendix D).
7

---

Under review as a conference paper at ICLR 2023
250
81.3
Valilation Top-1 (86)
Validation Top-3 (36)
26.0
80.3
24.
MidisNNerd Mobusdeencd ModiiaNevd
20.3
220
Hoshet-201 EileicaNNes
Mokitakir
Sorta Trankraaeeeeee
200
78.0
78.0
82.0
$6.0
90.0
60.0
96.8
07.0
96.0
95.9
Training (56)
Triiing Top (55)
(a) Mobile models
(b) Non-mobile models
Figure 7: Performance of different models on the ImageNet dataset using RangeAugment. Here, the target PSNR value is annealed from 40 to the value mentioned in the legend.
and is in accordance with previous works on the ImageNet dataset (Radosavovic et al., 2020; Wightman et al., 2021). These results show that training models with RangeAugment leads to a stable model performance.
Observation 2: Non-mo mobile models benefit from data regularization. On the ImageNet dataset, recommend to train non-mobile models using a curriculum that anneals A from high (e.g., target PSNR=40) to low target PSNR=5 10) similarity between input and augmented images.
TASK-LEVEL GENERALIZATION OF RANGEAUGMENT
Section shows the effectiveness of RangeAugment on different downstream models on the Im- ageNet dataset. However, might ask whether RangeAugment can be used for tasks other than image classification. To evaluate this, we study RangeAugment with two tasks, semantic segmentation (Section 5.1) and contrastive saaag-language pre-training (Section 5.2).
5.1 SEMANTIC SEGMENTATION THE ADE20K DATASET
Dataset and baseline models We use ADE20k dataset (Zhou et al., 2017) that has 20k training and 2k validation images across 150 semantic classes. We report the segmentation performance in terms of mean intersection over union (mloU) on the validation set.
We integrate mobile and non-mobile classification models with the Deeplabv3 segmentation head (Chen et al., 2018a) and finetune each model for 50 epochs. See Appendix F.1 for training details. We do not study SwinTrransformer for semantic segmentation because it not compatible with Deeplabv3's segmentation head design as it adjusts the atrous rate of convolutions to control the output stride of backbone network.
Baseline augmentation methods For semantic segmentation, experts have hand-crafted augmen- tation policies, and these manual policies are used to train state-of-the-un semantic segmentation methods (e.g., Chen et al., 2017; Zhao al., 2017; Xie et al., 2021; Liu al., 2021b). For ap- ples to apples comparison, we train each segmentation model with three different random seeds and compare with the following baselines: (1) Basetine standard pre-processing (randomly resize short image dimension, random horizontal flip, and random crop), (2) Manual baseline pre-processing with hand-crafted augmentation operations (color jittering using photometric distortion, random ro- tation, and random Gaussian noise), and (3) RangeAugmentt baseline pre-processing with learn- able range of magnitudes for brightness, contrast, and noise (i.e., = 3). For reference, we include Deepl abv3 results (if available) from popular segmentation library (MMSSegmentation, 2020).
Results Table shows that RangeAugment improves the performance of different models in comparison other augmentation methods. Interestingly. for semantic segmentation, the magnitude range of additive Gaussian noise is wider compared to image classification (Fig. 2b). This concurs with previous manual augmentation methods which also found that Gaussian noise is important for
8

---

Under review as a conference paper at ICLR 2023
Backbone
Baseline
Manual
Angmeutatiin method
RangmAugment (Ours)
MMScqmentation
(40,5)
10)
(40)
(40, 30)
(reference)
MobileNetei
38.77
0.20
12
0.16
36.46
0.1.9
MobileNNav2
37.74 4: 0.29
37.10
0.27
35.58
0:3.0
37.43
2 0.73
38.06
0.17
38.23 1: 0.36
34,88
MobileNaars
37,58 0.67
0.34 5668
14.50+0011
16.54
0.12
57.77
-024
38.10
I.13
MossleVit
37.00 0.50
$7.19
37.9-0044
0.4
35.04
+08
36,70
0.39
37.94
044
38.41 + 0.22
ReoNes-90
42.27+00.5
43.29
0.27
41.56.+00
42.95
0 0.22
43.31
0.16
43.00 + +-228
42.42
ResNer 101
0.17
44.04
0.52
43.22
0.522
43,89
1 0.43
44.77
0.28
43.95
1 0.31
45.8
EttcctenNeet
40.86 0.55
41.15.000
39442 1.029
40.39 0.48
41.43
0.36
41.08 1 0.36
Table
Semantic segmentation on the ADE20K dataset.
Model
Dataset
Test resolution
Top-1 Accuracy (%)
et (2021)
Progrictury LAION-400M
x
68.355
OpeaCTIP Ilare (2021)
x
x 192
67.4%
CLIP of RangeRugnent (Curs)
LAION-G0M
224 x 224
08.3%
256 x 200
68.8%
320 x 320
1%
Zero-shot performance on ImageNet. Each entry of CLIP with RangeAugment is the same model, but evaluated at different resolutions. Resultt of CLIP with the same language prompts as OpenCLIP.
Table 2:
semantic segmentation (Zhao et al., 2017: Asiedu al., 2022). Overall, these results suggest that RangeAugment learns task-specific augmentation policies.
5.2 CONTRASTIVE IMAGE-LANGUAL LANGUAGE PRE-TRAINING ON THE LAION-400M DATASET
Dataset and baseline models We crawl the LAION-400M dataset (Schuhmann al., 2021) and download about 304M image-language pairs, which then used for pre-training, We report zero- shot top-1 accuracy on ImageNet's validation set and use the same language prompts as OpenCLIP (Ilharco al., 2021).
We train CLIP (Radford et al., 2021) with Range RangeAugment from scratch (Appendix F.1). The model uses ViT-B/16 (Dosovitskiy al., 2020) as its image encoder and transformer its text encoder, and minimizes contrastive loss during training. We use multi-scale sampler of Mehta et al. (2022) to make CLIP more robust input scale changes. Because less data regularization is required at scale of 100M+ samples (Radford et al., 2021; Zhai et al., 2022), we anneal the target PSNR in RangeAugment from 40 to 20. We compare the performance with CLIP and OpenCLIP.
Results Table 2 compares the zero-shot performance of different models. For the same training dataset and zero-shot language prompts, RangeAugment delivers 1.2% better performance than OpenCLIP at an inference resolution of 224 x 224.
Observation RangeAugment learns model- and task-specific augmentation policies.
CONCLUSION
This paper introduces end-to-end method for learning model- and task-specific automatic aug- mentation policies with a constant search time. We demonstrated that RangeAugment delivers competitive performance
existing methods across different downstream models on the image classification task. This is despite the fact that RangeAugment uses only three basic augmenta- tion operations as opposed to large set of complex augmentation operations in existing methods. These results underline the importance of magnitude range of augmentation operations in automatic augmentation. We also showed that RangeAugment can seamlessly integrated with other tasks and achieve similar or better performance than existing methods. In the future, we plan to apply RangeAugment to learn the range of magnitudes for complex augmentation operations (e.g., geo- metric transformations) using different image similarity functions (e.g.. SSIM). In addition to learn ing the range magnitudes of each augmentation operation, we plan to apply RangeAugment to learn how to compose different augmentation operations with constant search time.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, and Mohammad Norouzi. Decoder denoising pretraining for semantic segmentation. arXiv preprint 1.Xtv:2205.11423, 2022.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp- 41 41-48, 2009.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. Xrviv preprint arXiv:1906.0m 07155, 2019.
Liang-Chieh Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint T0fk::/700.0556600 2017.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV). pp. 801-818, 2018a.
Liang-Chies CCigg- Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam, Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018b.
2018b.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
M. Everingham, L. Van Gool, K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- 2012.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation, In European Conference on Computer Vision, pp. 1-16. Springer. 2020.
Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference Machine Learning, pp. 2731-2741. PMLR, 2019.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369. IEEE, 2010.
10

---

Under review as a conference paper at ICLR 2023
Andrew Howard, Mark Sandler, Grace Chu, Liang- Ching Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukum Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference computer vision, pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
2017.
Zilong Huang. Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https: htt//dd..orgg 5281/ zenodo... 5143773. If you use this software, please cite it as below.
Diederik P Kingma and Max Welling. 1rXiv:1312.6114,220 2013.
Auto-encoding variational bayes. arXiv preprint
Alex Krizhevsky. Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (66(1))2277-2222 1998.
Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran. Smart augmentation learning an optimal data augmentation strategy. leee Access, 5:5858-5869, 2017.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In European Conference on Computer Vision, pp. 580-595. Springer, 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tom Ching LingChen, Ava Khonsari, Amirreza Lashkari, Mina Raf Nazari, Jaspreet Singh Sam- bee, and Mario Nascimento. Uniformaugment: search-free probabilistic data augmentation approach. arXiv preprint 2xXX::0003314345 2020.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct differentiable augmenta- tion search. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 12219-12228, 2021a
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Sachin Mehta and Mohammad Rastegari. Mobilevit: leigh, general-purpose and mobile- friendly vision transformer. arXiv preprint 10X.2:21 2021.
Sachin Mehta, Farzad Abdolhosseini, and Mohammad Rastegari. Cvnets: High performance library for computer vision. arXiv preprint arXiv:2206.02000 2022.
Contributors MMSegmentation. MMSegmentation: Openmmlab semantic segmentation toolbox
and benchmark. https: ttttp:://githiii com httm//pennmmmt ab/ abt/mm/gmeettatteeeeeeeeeeeeeeaa on, 2020.

---

Under review as a conference paper at ICLR 2023
Samuel G Müller and Frank Hutter. Trivialaugment: Tuning- free yet state-of-the-art data augmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 774- 782, 2021.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv: 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 1748-8763. PMLR, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10428- 10436, 2020.
Alexander J Ratner, Henry Ehrenberg. Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain-specitee transformations for data augmentation. Advances in neural in- formation processing systems, 30, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 1.02/ 14, 2021.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.102
2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net-
works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International Conference Machine Learning, pp. 10096-10106. PMLR, 2021.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NewrlPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021. URL httpt:tpp///c http:://ppeevvvieeeee wwtp:riwwwweeeeeeeeeeemeeeeeeeee teee/foormmt wtlffforrrmdiieee forummid=-G6M.mV 6M5.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
/ com/ http:///ccooooeeeerccceeeeeeeeee http:p/ss/.gggnncccccccccceeeeee
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings the European conference on computer vision (ECCV), pp. 418 434, 2018
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 14:12077-12090, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF international conference computer vision, pp. 6023-6032 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 12104-1211 12113, 2022.
12

---

Under review as a conference paper at ICLR 2023
Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2881-2890, 2017.
Yu Zheng, Zhi Zhang, Shen Yan, and Mi Zhang. Deep autoaugment. In International Conference on Learning Representations, 2021.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmen- tation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattem recogmition, pp. 633-641, 2017.
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 593-602, 2019.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017. URL https: peattps://oot http://///ppeeee////////e////eee net forum? id=r1Ue8Hcxg.
13

---

Under review as a conference paper at ICLR 2023
A COMPARISON WITH EXISTING METHODS
ImageNet classification State-of-the-art methods incorporate random erasing (Zhong et al., 2020), mixup transforms (Zhang al., 2017; Yun et al., 2019) in addition to automatic augmen- tation methods (e.g,, RandAugment and AutoAugment). Table shows that models trained with RangeAugment are able to achieve similar or better performance than existing automatic aug- mentation methods with 4 5x more augmentation operations.
Model
Source
Data augmentation methods
Top-1
Auto ang. method
Rundom crase
Mixup
accuracy
Mobile models
Orig. (Howant et al., 2017) Our repro. repro.
X
X
70.65
MobileNetVL-10
RandAugment (N 14)
X
73.7%
AutoAngment (N 16)
73.1%
RangeRugment
Our repro. repro. Orig. (Sandler et al, 2018)
X
X
72.0%
MobileNetV2-100
RandAngment (N 14)
X
72.7%
AutoAugment
RangeAugment
Our repro. repro. Orig. (Howant et al., 2019)
X
K
74 66
MobilkNetV3-Larg
RaudAugment (N =
75.2%
AutoAugment
RangeAugment
Orig. (Mehta & Rostegari, 2021) Our repro. repro.
X
78.4%
Mobile ViT--mall
RandAugment (N
4.5%
AutoAugment
Ours
Ranggehugment (N
78.2%
Nou-mobile models
Orig. (Hl et 2016)
76.2%
ResNet-50
(Wightmam 2021)
RandAugmeat
Ours
Rangehugment (N
80.2%
Orig. (He et al., 2016)
77.4%
ResNett-111
TIM (Wighiman
2021)
RandAugment (V 14)
$1.5%
RangeRugmant
Orig. (Tan & Le, 2019)
Augmmng (N 16)
77.1%
EfficientNet-B0
TIM (Wightman al., 2021)
RangeAugment 14)
RandAugment (N
77.0%
77.3%
Orig. 2019)
AutoAugment 16)
79.
EfficientNet-Il
TIMM (Wightman al., 2021)
RandAugment (N 14)
99.2%
Ours
RangeAugment (N 3)
79,5
AutoAugment
NetelentNet-122
(Wightman al., 2021)
RandAugment
Ours
RangeAogmant (N 3)
$1.3%
Orig. (Tan 2019)
AulaAugment (N 16)
6%%
Nelt
(Wightman ol., 2021)
RandAugment
RangeAugment
Swin-Tiny
Orig. (Liu al., 2021b) 2011))
RandAugment (. 14)
/
$1.3%
Ours
KangeAugmentt (N =)
81.1% %
Orig. al., 2021b) Ours
RandAugment
KangeAugment
82.8%
Table Accuracy comparison of different models trained with different methods on the Ima- geNet validation set. RangeAugment with simple and 4 5x fewer transforms is able to deliver similar or better performance state-of-the-ar methods with complex automatic augmentation poli- cies. For mobile models, we decay A from 40 to 30 while for non-mobile models, we decay A from 40 to (as per observations in Section 4.1). Methods whose performance is within the standard de- viation range of +0.2 of the best model are highlighted in bold. Note that RandAugment in TIMM a custom implementation that delivers better performance than the RandAugment of Cubuk et al. (2020), and is widely used for training recent classification networks the ImageNet, including SwinTransformers. Here, N denotes the number of augmentation operations.
Semantic segmentation ADE20k Table 4 compares the performance of different segmentation architectures for the same backbone. Compared to highly-tuned augmentation recipes MMSeg (MMSegmentation, 2020) and CSAIL (Zhou et al., 2017) segmentation libraries, RangeAugment able to achieve better performance consistently across different backbones.
Furthermore, Table 5 shows that DeepLabv3 with ResNet-101 backbone, when trained with Aageet, delivers the same performance as UPerNet (Xiao et al., 2018) with SwinTrans- former (Liu et al., 2021b) and ConvNext (Liu et al., 2022) backbones while being 3x FL.OP efficient.
14

---

Under review as a conference paper at ICLR 2023
Seg.
MedileNa
(Ziaoo al.,
CSAIL
repro.
Decplabo3 al., 2017 Bungstunnent
UPeNet
CSAIL
40.4
UPeNet (Xia et 2015)
MbbSeg
42.1
RenNre-50
PSPNa (2han ct all,20111 2017)
Dasplabe3 (Clas zi sl. 2017)
MMScc
MMSaeg
42.5
42-7
DecpLabb3 (Cles et th, 2017)
Our regm.
43.0
Deoplatedd (Chen et tl. 201/1 (3 w/ Rangefugnent (Oun)
UPeNNa eiaiecaal, 2013)
MiNSse
RoNNes101
DecpLabv3 2017) DecpLabo3 2017)
repro.
(Ours)
Table 4: Comparison between different state-of-the-ard segmentation method for the same back- bone. Models trained with RangeAugment is able to deliver better performance than highly-tuner augmentation pipelines in popular segmentation libraries (CSAIL (Zhou et al., 2017) and MMSeg (MMSeggentation,, 2020)).
ConuNear UPeNNe
ReaNa-10) 101 DacpLaby
Rangekoggment (Oun)
363 G
46.5
Table 5: DeepLabv3 with RangeAugment delivers similar performance to UPerNet while be- ing 3x more FLOP efficient. RangeAugment improved the segmentation accuracy of ResNet- 101 with DeepLabv3 significantly; delivering competitive performance to state-of-the-ee segmen- tation model, UPerNet, with recent backbones (SwinTransformer- and ConyNext).
Overall, these segmentation results underline the effectiveness of RangeAugment.
B TRANSFERRING AUGMENTATION POLICY
Searching model- and task-specific policy may be expensive. Therefore, common practice is to transfer the policy found on one dataset to another. This section evaluates the augmentation curriculum of RangeAugment accuracy of RangeAugment with publicly reproduced models as there performance is often better than those reported in the paper. For experiments this section, follow our observations in Section 5 and anneal the PSNR value A from 40 to 20.
be used across different tasks and datasets. We compare the
Object detection on COCO Following previous works, we use ResNet-50 as backbone and train Mask R-CNN on the COCO dataset (Lin et al., 2014). Table 6 shows that RangeAugment improves the detection accuracy of Mask-RCNN significantly.
Model
Optim. updates
BBOX mAP
Detectron2
270k
41.0
MMDetection
270k
40.9
RangeRugment (Ours)
70k
41.0
RangeAugment (Ours)
230k
42.6
Table 6: Enhanced object detection results of Mask R-CNN with RangeAugment COCO. Following standard convention for reporting object detection performance of Mask R-CNN, we also report the number of optimization updates (or schedule). We use similar hyper-parameters, including learning rate, as Detectron2 (Wu et al., 2019) and MMDetection (Chen et al., 2019).
Semantic segmentation on PASCAL VOC 2012 Following previous segmentation methods, we ResNet- 101 backbone and train DeepLabv3 on the PASCAL VOC 2012 dataset (Evering- ham et al., 2012). Table shows that DeepLabv3 with RangeAugment: attains the best perfor- mance.
15

---

Under review as a conference paper at ICLR 2023
Seg. model
mloU
ANN (Zhu ol., 2019)
76.7
APCNet (He al., 2019)
78.5
CCNet (Huang
2019)
77.9
DeepLaby3 (Chen et al, 2017)
77.9
DeepLabv3+ (Chen al. 2018b)
78.6
PSPNet (Zhao et al., 2017)
78.5
UPerNet (Xiso al., 2018)
77.4
DeepLabvv3 RangeAugment (Ours)
84.0
Table 7: Comparison with state-of-the-art semantic segmentation methods with ResNet--01 backbone on the PASCAL VOC validation set. We do not use multi-scale evaluation. The results of different segmentation models are from MMSegmentation (2020). Also, our training recipes, including batch size and learning rate, are similar to MMSegmentation (2020).
C ABLATIONS ON THE IMAGENET DATASET
In this section, we study different components of RangeAugment using ResNet-50., For learning augmentation policy, we anneal the target image similarity (PSNR) value A from 40 to 5.
Effect of different curriculum We trained RangeAugment with two curriculum's: (1) linear and (2) cosine. We found that cosine curriculum delivers 0.1-0.2% better performance than linear. Therefore, we use cosine curriculum.
Effeet of The weight term, X, in Eq. (2) allows RangeAugment to balance the trade-off be- tween augmentation loss L. and empirical loss Lusk To study its impact, vary the value of from 0.0 to 0.15. Empirical results in Fig. 8 shows that the good range for À between 0.0006 and 0.002. In our experiments, use X 0.0015.
7022
70.0
78.4
18.2
0.0
0.000
0.006
5.000
010015 Weight torm A
01007
0.003
0.0311
0.9101
0.135
Figure 8: Effect of weight term, X, on ResNet-50's performance on the ImageNet dataset.
Effect of joint vs. independent optimization An expected behavior for learning model-specific augmentation policy using RangeAugment that task-specific loss Cusk Eq. 2 should con- tribute towards policy learning. To validate it, ResNet-50 is trained independently" as well as jointly on the ImageNet dataset. We found that the top-1 accuracy of ResNet-50 dropped by about 1% when it is trained independently. This is likely because independent training allowed RangeAugmert to produce augmented images with more additive Gaussian noise (Fig. 9), resulting in performance drop. This concurs with observations in Section especially Figs. 5 and 6, where we found that sampling augmented images from wider magnitude range for additive Gaussian noise operation dropped ResNet-50's performance on the ImageNet dataset. A plausible explanation that PSNR is more sensitive to noise operation (Hore & Ziou, 2010), allowing RangeAugment to learn wider magnitude ranges for noise operation when trained independently as compared to joint training. Overall, these results suggest that joint training helps in learning model-specific policy.
'The augmented image is detached hefore feeding the model.
16

---

Under review as a conference paper at ICLR 2023
A
Brightness
Contrast
Noise
Megittue mage
Stginud rouge
Magiitule
40 ->
33m
480
300
400
3.0
630
Epoch
Dpoch
Epoch
Joint loss
Megsttuk range
Stgghtuie sange
23n1
Megnttude
10'
237
23mh
Fpoch
Independent loss
Figure 9: The effect of learning magnitude ranges by jointly optimizing the loss terms Lm and Lusk (top row) compared only optimizing L., (bottom row). Training ResNet-50 with the joint loss leads to smaller magnitudes of noise, and improves validation accuracy by approximately 1% on the ImageNet dataset.
D VISUALIZATION OF AUGMENTED SAMPLES
Fig. 10 visualizes augmented samples produced by RangeAugment different stages of training ResNet-50 with curriculum. We can see that the range of magnitudes (Fig. 10a) and diversity of augmented samples (Fig. 10b) increases as training progresses.
Brightness
Contrast
Noïse
Map da caue
2.0
200
20
Tp-di
Spack
Epuch
Inpot Images
Egoch 1
Epoch 100
Epoch 200
Epech 600
(b)
Figure 10: Visualization of augmented samples. (a) Learned magnitude ranges of different aug- mentations when ResNet-50 trained jointly with RangeAugment using cosine currieulum. (b) Four image samples visualized at different epoch intervals. For illustration purposes, we visualize four random augmented samples produced by RangeAugment for each image.
LEARNED MAGNITUDE RANGES FOR CLIP WITH RANGFAUGMENT
Fig. 1I shows the learned magnitude ranges of different augmentation operations for training the CLIP model with RangeAugment. Unlike image classification (Section 4) and semantic segmen- tation (Section 5.1) results, the CLIP model uses little augmentation. This is expected because the LAION dataset is orders of magnitude larger than classification and segmentation datasets.
17

---

Under review as a conference paper at ICLR 2023
Brightness
Contrast
Noise
Rpod
Rpoch
Speck
Figure 11: Learned magnitude ranges of different augmentations when the CLIP model is trained with RangeAugment using cosine curriculum. Target PSNR A is annealed from 40 to 20.
F TRAINING DETAILS
F.1
TRAINING HYPEE-PARRAETERS
Table 8 summarizes training recipe used for training different models across different tasks.
Non-mohile models
Mohile models
ReaNet-50
ResNet-101
SwinTrannsormer
MobileViT
MobileNedv1
MobileNett2
MobileNetv3
Epochs
600
600
300
300
300
300
Batch size
1024
1024
2048
1024
1024
512
1024 MSc-VBS
2048
Data sumpler
MS-VVBS
MSc-VBS
MSc-VBS
SSc-FBS
MSc-VBS
MS:VVS
MS--VBS
0.8
Min. LR
2 0.05
2 10-4
x 10-4
10
2 10
x 10
2 x 10
4
Warmup ini.. LR
0.05
0.1
10-4
2 10
0.05
0.05
0.1
Warmup
Annculing
cosinec
cosine
cusine
costnic
cosine
costne
cosing
cusine
Weight decay
4 10
4 10
x 10-5
10-2
0.01
x 10
x 10-5
10
Optimizer
SGD
SGD
SGD
Adam/V
Adam
SGD
SGD
SGD
Label smoothing
0.1
0.1
0.1
0.1
0.1
0.1
Stoch. Depth
X
X
0,3
X
X
X
Grad. clipping
5.0
parameters
25.6 M
44.5 M
12.3 1.9
49.6 M
5
4.2
3.5
5:4 220 M
FLOPS
4.0
7.7
8.8
2.0
579
314 M
(a) Image classification on ImageNet
Hyperparameter
Value
Hyperparameter
Value
Epochs
Basch size
Barch size
32.768
Data sampler
SSc-FBS
Data sampler
MSC-VBB
Warm-up
Warm-up ieeraionss
2000
Warm-up LR
Warm-up up init LR
10-4
Max. LR
0.02
Max. LR
5-d
10-4
10-4
Annealing
cosine
LR Annealing
cosine
Weight decay
10
Weight decay
0.2
Optimizer
Optimizer
AdamW
(b) Semantic segmentation on ADE20k
(c) CLIP (150M parameters) training on LAION-400M
Table 8: Hyper- parametes used for training models different tasks. Here, SSc-FBS and MSc- VBS refers to single-scale fixed batch size and multi-scale variable batch size data samplers respec- tively (Mehta et al., 2022).
F.2 BOUNDS FOR AUGMENTATION OPERATIONS
Table 9 shows the clipping bounds that RangeAugment uses to prevent training instability.
Operation
Clipping Rangekugment
Magnitude runge AutuAugment
(veference)
Brightness
0.1
10.0
[0.1
Continst
0.1
10.0
1o.1-1.91
Noise isid. dev)
1.0
Table Clipping bounds used in RangeAugment for preventing training instability. The range of magnitudes for augmentation operations in AutoAugment is also given as reference.
18


Reviews:
[{"review_id": "xpvUa_Igeq", "reviewer": "Reviewer_Q5Gq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new data augmentation technique for visual recognition tasks. The core idea is to replace the fixed, manually defined ranges for data augmentation operations (such as brightness and contrast) with a learned, adaptive policy for specifying the magnitude of the operation range. The learned policy is model-specific and task-specific. The paper shows an extensive evaluation of ImageNet across different networks. The results show that the proposed RangeAugment achieves similar performance as the state-of-the-art AutoAugment and RandAugment but requires significantly fewer data augmentation operations. The paper also demonstrates results on other tasks such as semantic segmentation and contrastive learning. ", "summary_of_the_review": "The paper presents a novel data augmentation technique. Learning the range of magnitude for each data augmentation supports learning model-specific and task-specific policies. The method is simple and effective (as shown in the experiments). While only simple augmentation operations are shown (brightness, contrast, noise) in this paper, I think the proposed data augmentation method could benefit the community.", "strength_and_weaknesses": "Strength:\n+ The paper's idea of replacing a fixed, manually defined range of data augmentation operations is novel and interesting. \n+ It's surprising that by only using three operations (brightness, contrast, noise), the proposed augmentation achieves state-of-the-art performance like other methods that use many more operations (e.g., AutoAugment, RandAugment).\n+ The evaluation is thorough. The experiments answer many questions regarding generalization across different model architectures and tasks.\n\nWeakness:\n- The font sizes of many of the figures are too small. It's hard to read (without zooming in on a computer).\n- I am curious whether the range of magnitude idea can be applied to other types of data augmentation beyond changing brightness/contrast. For example, it would be interesting to see how RangeAugment can be applied to geometric augmentation (zoom, cropping, rotation). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oW8KAwo67x", "reviewer": "Reviewer_D2H4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of data augmentation. Previous search-based data augmentation methods like AutoAugment adopt a large set of operations that is not flexible and leads to sub-optimal solutions. The author proposes a method named RangeAugment to learn the range of magnitudes for each individual augmentation. The experimental evaluations are conducted on ImageNet for image classification and ADE20K for semantic segmentation, and the results seem ok. ", "summary_of_the_review": "The claims in the paper are not fully supported by the experimental evaluations. The paper seems not well prepared, and further investigation is needed to make it stronger.", "strength_and_weaknesses": "**Strength**\n\nThe paper writing is clear, and the content is easy to follow. The observations and their related figure illustrations and analysis seem nice.\n\n**Weaknesses**\n\nThe paper lacks lots of experimental evaluations to support its claim. For the shown image classification and semantic segmentation tasks, only ImageNet with several backbones and ADE20K with limited frameworks are tested. This is far from satisfactory, and readers can not be fully convinced by these results. Besides, the limited number of experimental results are also far from the current best solutions and SOTA performance. Further, experimental evaluations with more tasks like object detection are also recommended.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RSaqe8YrddZ", "reviewer": "Reviewer_aoS6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Motivation**. Current automatic augmentation methods search over pre-determined range values for the strength of each augmentation. Ideally, the search should happen over continuous values for the range.\n\n**Approach**. This paper proposes to differentiate with respect to the range of the augmentations. This is done by:\n1. Using differentiable augmentation functions (brightness, contrast, additive gaussian noise)\n2. Using the random variable re-parameterization trick to have differentiability with respect to the uniform distribution\n3. For the loss, first the similarity between augmented and original image is computed (i.e., PSNR). Then, this value is compared to a desired value $\\Delta$.\n", "summary_of_the_review": "While this paper has an interesting idea in its core, I think the presentation and experiments need more work so that they clearly demonstrate the advantages of RangeAugment. The benefits of RangeAugment are not so clear to me in the current form of the paper.", "strength_and_weaknesses": "**Strengths**\n\n1. I think there is merit to the idea of having differentiable augmentations and optimizing them.\n\n**Weaknesses**\n\n1. The experiments don't convey a strong sense of RangeAugment outperforming previous methods. E.g. in figure 5, RangeAugment seems to be underperforming compared to AutoAugment/RandAugment. Also, RangeAugment seems to be somewhat sensitive to the PSNR value choice.\n2. I thought the authors would want to optimize the augmentation parameters based on the augmentation loss, but the paper suggests adding the augmentation loss term to the task loss. This did not make sense to me: Why would one want to optimize the model parameters with respect to the augmentation similarity loss?\n3. Having an augmentation loss that computes how close the image is to the original image did not make sense to me. Why would this be a good loss function for determining the best augmentations?\n4. I found the writing a bit difficult to follow - at times it felt like there was too much verbose information (e.g. section 4.2). I also found figures 5 & 7 hard to read - seemed crowded to me.\n\n**Suggestions**\n\nI could see one potentially exciting use-case for differentiable augmentations in sim2real scenarios: A common scenario is having labelled simulated training data (which often have perfect cameras), and some unlabelled real world images. People often augment the simulated training images to mimic real-world imperfections of the camera (which sometimes includes modelling the noise profile of the camera). Using differentiable augmentations, one can potentially look into optimizing augmentation parameters s.t. the features from the simulated training images are closer in distribution to real world images - basically omitting the need for manually modelling the camera noise profile.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EFPOaYmcMZa", "reviewer": "Reviewer_39wu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an automatic learning approach for constructing training samples of data augmentation. For this purpose, a framework with three candidate operations is built and a grid search algorithm is used for finding the optimal Delta (for the intensity of DA). Experiments show limited accuracy gain on image classification and semantic segmentation.", "summary_of_the_review": "The method is too simple and many aspects are not covered. In addition, the experiments are weak. I cannot suggest acceptance.", "strength_and_weaknesses": "Strengths\n1. The paper is easy to follow.\n2. The studied problem (effective data augmentation) is of broad interest to the community.\n\nWeaknesses\n1. The proposed method is too simple. Of course, being simple is not a weakness, but the method does not show sufficient accuracy gain in experiments. Technically, Eq (2) tries to formulate in a generalized way, but the final solution is not \"end-to-end\" as claimed in the paper, but to grid-search the only parameter, \\Delta. Besides, only studying the intensity is insufficient. There are many more aspects that need to be considered, such as the order of augmentation, the presence or absence of each augmentation operation, etc. In addition, the single function d (measured by PSNR) is only side evidence and I really doubt whether it can help to generate samples that help visual recognition (especially when I see the weak experimental results).\n\n2. I went through the paper several times, but I cannot find the exact definition of L_ra. This does not prevent me from understanding the main idea, but the details are not clear.\n\n3. The empirical study is trivial. I expect many technical tricks that can impact the effect of using DA, such as the use of knowledge distillation, e.g. [Wei et al., Circumventing outliers of autoaugment with knowledge distillation, ECCV20] suggested that the augmented data shall be checked by a teacher model to avoid less meaningful examples as shown in Fig 4. This technique was later used by DeiT [Touvron et al., Training data-efficient image transformers & distillation through attention, ICML21]. Of course, there are other tricks that can improve DA. Without considering these tricks, the experimental results are below satisfaction.\n\n4. Regarding the three observations. Ob 1 (as used in AutoAugment and RandAugment) is mostly known. Ob 2 seems superficial -- the reason behind the observation is that small models often cooperate well with moderate DA (or similarly, moderate-scale datasets). In addition, the limited experiments in this paper are insufficient to validate the statement. Ob 3 is confusing to me -- RangeAugment only learns a single parameter Delta and a few parameter-free strategies (e.g. composition of 3 DA operations). Since the learned model is simple, it is naturally generalized, yet the improvement is also marginal.\n\n5. Experimental results are weak. Please refer to the AA and RA papers for more thorough experiments. Most often, toy experiments are performed on CIFAR, and then transferred to ImageNet. Currently, the improvements on ImageNet-1K and ADE20K are mostly marginal, which cannot justify the effectiveness of RangeAugment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject