DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
Authors: kaichen zhou
Lanqing HONG
Shoukang Hu
Fengwei Zhou
Binxin Ru
Jiashi Feng
Zhenguo Li
Venue: Accepted by TMLR ()

Abstract: Automated machine learning (AutoML) usually involves several crucial components, such as Data Augmentation (DA) policy, Hyper-Parameter Optimization (HPO), and Neural Architecture Search (NAS). Although many strategies have been developed for automating these components in separation, joint optimization of these components remains challenging due to the largely increased search dimension and the variant input types of each component. In parallel to this, the common practice of searching for the optimal architecture first and then retraining it before deployment in NAS often suffers from the low-performance correlation between the searching and retraining stages. An end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search is desirable. In view of these, we propose DHA, which achieves joint optimization of Data augmentation policy, Hyper-parameter, and Architecture. Specifically, end-to-end NAS is achieved in a differentiable manner by optimizing a compressed lower-dimensional feature space, while DA policy and HPO are regarded as dynamic schedulers, which adapt themselves to the update of network parameters and network architecture at the same time. Experiments show that DHA achieves state-of-the-art (SOTA) results on various datasets and search spaces. To the best of our knowledge, we are the first to efficiently and jointly optimize DA policy, NAS, and HPO in an end-to-end manner without retraining.

Published in Transactions on Machine Learning Research (10/2022)
DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
Kaichen Zhou University of Oxford
Lanqing Hong' Huawei Noah's Ark Lab
honglanqing Siaauuuei.nniiiuuuucooooo.o
Shoukang Hu
The Chinese University of Hong Kong
Fengwei Zhou Huawei Noah's Ark Lab
Binxin Ru
SailYond Technology & Research Institute of Tsinghua University Shenzhen
Jiashi Feng
National University of Singapore
Zhenguo
Huawei Nooh's 's Ark Lab
Reviewed on OpenReview: Sttps://openreviees net/ formm? ? id= MHOAETTI
Abstract
Automated machine learning (AutoML) usually involves several crucial components, such as Data Augmentation (DA) policy, Hyper-Parameter Optimization (HPO), and Neural Architecture Search (NAS). Although many strategies have been developed for automating these components separation, joint optimization of these components remains challenging due the largely increased search dimension and the variant input types of each component. In parallel to this, the common practice of searching for the optimal architecture first and then retraining it before deployment in NAS often suffers from low performance correlation between the scarching and retraining stages. An end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search is desira ble. In view of these, we propose DHA, which achieves joint optimization of Data augmentation policy, Hyper-parameter and Architecture. Specifically, end-to-end NAS achieved in a differentiable manner by optimizing compressod lower-dimensional fcature space, while DA policy and HPO regarded as dynamie schedulers, which adapt themselves to the update of network parameters and network architecture at the same time. Experiments show that DHA achieves statc-of-the-art (SOTA) results on various datascts and scarch spaces. To the best of our knowledge, we are the first to efficiently and jointly optimize DA policy, NAS, and HPO in an end-to-end manner without retraining.
1 Introduction
While deep learning has achieved remarkable progress in various tasks such computer vision and natural language processing, the design and training of well-performing deep neural architecture for specific task usually requires tremendous human involvement. (He et. al. 2016 Sandler et al. 2018). To alleviate such

---

Published in Transactions on Machine Learning Research (10/2022)
burden on human users, AutoML algorithms have been proposed in recent years to automate the pipeline of designing and training a model, such as automated Data Augmentation (DA), ShyperParameter Optimization (HPO), and Neural Architecture Search (NAS) (Cubuk et al. 2018 Mittal et al. 2020 Chen et al. 2019
All of these AutoML components are normally processed independently and the naive solution of applying them sequentially separale stages, not. only suffers from low efficiency but also leads to sub-optimal results Dai et al. 2020; Dong et al. 2020). Indeed, how to achieve full-pipeline "from data to model" automation efficiently and effectively is still a challenging and open problem.
One of the main difficulties lies in understanding how to automatically combine the different AutoML components (e.g., NAS and HPÒ) appropriately without human expertisc. F3 (Dai et al. 2020) and AutoHAS (Dng et al. 2020 investigated the joint optimization of NAS and HPO, while (Kashima et al, 2020) focused on the joint optimization of neural architectures and data augmentation policies. The joint optimization of NAŞ and quantization policy were also investigated in APQ (Wang et al. 2020). Clear benefits can be scen in the above works when optimizing two AutoMI components together, which motivates the further investigation of "from data to model" automation. However, with the increasing number of AutoML components, the search space complexity is increased by several orders of magnitudes and it is challenging to operate in such a large scarch space. In addition, how these AutoMI. components affect cach other when optimized together is still unclear. Thus, further investigation is needed to open the black box of optimizing different AutoML, components jointly.
Another main challenge of achieving the automated pipeline "from data, to model' understanding how to perform end-to-end searching and training of models without the need of parameter retraining. Current approaches, even those considering only one AutoML component such NAS algorithms, usually require two stages, one for searching and one for retraining (Liu et al., 2019 Xie et al. 2019 Similarly, automatic DA methods such as FastAA (Lim et al., 2019) also need to retrain the model parameters once the DA policies have been searched. In these cases, whether the searched architectures or DA policies would perform well after retraining is questionable, due to the incvitable difference of training sctup between the scarching and retraining stages ( (Yang el al., 2019. To improve the performance correlation between searching and retraining stages, DSNAS (Hu al. 2020) developed a differentiable NAS method to provide direct NAS without parameter retraining. OnlineAugment (Tang ct al.) 2020) and OnlincHPO (Im et al. 2021 design direct DA or HPO policy, respectively, without model retraining.
Targeting the challenging task-specific end-to-end AutoML, we propose DHA, a differentiable joint optimization solution for efficient end-to-end AutoML components, including DA, HPO and NAS. Specifically, the DA and HPO are regarded as dynamic schedulers, which adapt themselves to the update of network parameters and network architecture. At the same time, the end-to-end NAS optimization realized in a differentiable manner with the help of sparse coding method, which means that instead of performing our search in a digh-dimensional network architecture space, we optimize compressed cover-dimensional feature space. With this differentiable manner, DHA can effectively deal with the huge search space and the high optimization complexity caused by the joint optimization problem. To summarize, our main contributions are as follows:
We propose an AutoML method, DHA, for the concurrent optimization of DA, HPO, and NAS. To the best of our knowledge, we are the first to efficiently and jointly realize DA, HPO, and NAS in end-to-end manner without retraining.
Experiments show that DHA achieves a statc-of-the-art (SOTA) accuracy on ImageNet with both cell-based and Mobilenet-like architecture search space. DHA also provides SOTA results on various datasets, including CIFAR10, CIFAR100, SPORT8, MTT67, FLOWERS102 and ImageNet with relatively low computational cost, showing the effectiveness and efficiency of joint optimization (see Fig. 2
Through extensive experiments, we demonstrate the advantages of doing joint-training over optimizing each AutoMI. component in sequence. Besides, higher model performance and smoother loss landscape are achieved by our proposed DHA method.
2

---

Published in Transactions on Machine Learning Research (10/2022)
Augment Probability Matrix
Update Augment Parameters
Update Architecture Parameters
Update Hyper- parameters "
Suger Network
Training Loss with 0 ++
Sample Transformation T-I
Sample Architecture A~ac a
Training Loss with 0
Optimizer
Update Network Parameters O
Mini-batch
Augmented Mini-batch
forward
backward
Child Network
End-to-end NAS with manually designed r
End-to-end NAS with manually designed
DHA
Figure 1: An overview of DHA. We first sample the DA operations for each sample based the data transformation parameters т. Then, a child network sampled based on the architecture parameters a, which used to process the transformed mini-batch. Training loss is calculated to update the data transformation parameters architecture parameters a, and neural network parameters 0. Then the training loss based on updated networks' weights Gt+1 is used to update pare-paaaeeters 17.
2 Related Works
Data augmentation. Learning data augmentation policies for a target dataset automatically has become a trend, considering the difficulty to elaborately design augmentation methods for various datasets (Cubuk et al. Specifically, AutoAugment (Cubuk et al. 2018) and Adversarial adopt reinforcement learning to train controller to generate policies,
2018 Zhang al. 2020 Lin al.
2019).
Augment (Zhang al. 2020
while OHL-Auto-Aug (Lin et al. 2019 formulates augmentation policy a probability distribution and
adopts REINFORCE (Williams) 1992 to optimize the distribution parameters along with network training.
PBA Ho et al., 2019) and FAA (Lim et al. 2019) use population-bace based training method and Bayesian optimization respectively to reduce the computing cost of learning policics. Cubuk et al. (2200) argues that the search space of policies used by these works can be reduced greatly and simple grid search can achieve competitive performance. They also point out that the optimal data augmentation policy depends on the model size, which indicates that fixing an augmentation policy when searching for neural architectures may lead to sub-optimal solutions.
Syper-parameter optimization. Hyper-parameters also play important role in the training paradigm of deep neural network models. Various black-box optimization approaches have been developed to address parammer tuning tasks involving multiple tasks (Mittal et al. 2020: Perrone et al. 2018) or mixed al, variable types (Ru al. 2020a). Meanwhile, techniques like multi-fidelity evaluations (Kandasamy
2017
Wu al. 2019), parallel computation (González al. 2016 Kathuria al.) 2016 Alvi et al.) 2019) and transfer learning (Swersky et al., 2013 Min et al. 2020)) also employed further enhance the query efficiency of the syper-parameter optimization. In addition, several works (Bengio 2000 Lorraine et al.
2020
MacKay et al. 2019, Shaban et al. 2019 Maclaurin et al. 2015) have proposed to use gradient-based bi-level optimization to tune large number of hyper-parameters during model training. Although many HPO strategies have been adopted in NAS strategies, methods that jointly optimize both architectures and Saper-parameters are rarely seen except the ones discussed below.
Neural architecture search. NAS has attracted growing attention over the recent years and provided architectures with better performance those designed by human experts (Pham et. al. 2018 Real et. al.
3

---

Published in Transactions on Machine Learning Research (10/2022)
2019 The rich collection of NAS literature can be divided into two categories: query-based methods and gradient-basse ones. The former includes powerful optimization strategies such as reinforcement
2018
Liu et al.
learning Zoph & Le 2017 Pham et al. 2018), Bayesian optimization (Kandasamy et al. 2018 Ru et al.
2020b) and evolutionary algorithms EEsken et al. 2019 Lu et al.) 2019). The latter enables the use of
gradients in updating both architecture parameters and work weights, which significantly reducing the
computation costs of NAS via weight sharing (Liu et al. 2019 Chen et al. 2019 Xie et al.) 2019 Hu et al. 2020). To reduce searching cost, most NAS methods search architectures in a low-fidelity set-up (e.g.fewer training epochs, smaller architectures) and retrain the optimal architecture using the full set-up before deployment. This separation of search and evaluation is sub-optimal (Hu et al. 2020), which motivates the development of end-to-end NAS strategies (Xie et al., 2019 Hu et al. 2020 that return read-to-deploy networks at the end of the search. Our work also proposes an end-to-end solution.
Joint optimization of AutoML components. Conventional neural architecture search methods perform a search over a fixed set of architecture candidates and then adopt a separate set of hyper-parametenn when retraining the best architecture derived from the architeclure search phase. Such search protocol may lead to sub-optimal results (Zela et al. 2018 Dong et al. 2020) as it neglects the influence of training hyper- paramcters on architecture performance and ignores superior architectures under alternative hyper-parameter values DDi et al. 2020). Given this, several works have been proposed to jointly optimize architecture
structure and training hyper-parameters (Dai et al. 2020 Wang et al. 2020 Dong et al. 2020) Zela et al. (2018) introduces the use of muuli-fidelitt, Bayesian optimization to search over both the architecture structure and training aaeer-parameters. Dai et al. (2020) trains an accuracy predictor estimate the network performance based both the architecture and training hyper-parameters and then uses evolutionary algorithm to perform the scarch. Both these methods are query-based and require a relatively large number of architecture and hyper-parameter evaluations to fine-tune predictors or obtain good recommendations.. To improve the joint optimization efficiency, AutoHAS (Dong et al. 20200 introduces a differentiable approach in conjunction with weight sharing for the joint optimization task, which empirically demonstrates that such a differentiable one-shot approach achieves superior efficiency over query-based methods. In addition to jointly optimizing neural architectures and hyper-parameters, the other line of research focuses on the joint. optimization of neural architectures and data augmentation typer-parameter (Kashima al. 2020). Our proposed method differs from the above works in mainly two aspects: first, our method is more efficient than AutoHAS since our method has need to update the whole network and only needs to update the sampled sub-network at each optimization step. Second, the joint optimization scope is further extended from NAS and training hyper-parameters to include DA, HPO and NAS.
3 Methodology
Consider datasct D= {Z,, 1:) where N is the size of this dataset, and y; is the label of the input sample 24. We aim to train neural network f(),, which can achieve the best accuracy on the test dataset Dieet Multiple AutoMI. components are considered, including DA, HPO, and NAS. Tet T, n, a, and O represent the data augmentation parameters, the paper--aramttes. the architecture parameters, and the objective neural network parameters, respectively. This problem can be formulated as
n, 0;: s.t. c(aa) S C, 1,...,7
(1)
where ((-) represents the loss function, D denotes the input data, c;(-) refers to the resource cost (o.g., storage or computational cost.) of the current architecture a, which is restricted by the i-th resource constraints C,, and y denotes the total number of resource constraints. Considering the huge search space, it is challenging to achieve the joint optimization of T, n, a, and 0 within onc-stage without parameter retraining. In this work, we propose to use the differentiable method to provide a computationally efficient solution. See Fig. for illustration.
4

---

Published in Transactions on Machine Learning Research (10/2022)
3.1 Data augmentation parameters
For every mini-batch of training data 8 = (( ya) k=) with batch size n²", we conduct data augmentation to increase the diversity of the training data. We consider K data augmentation operations, and each training sample is augmented by a transformation consisting of two successive operations (Cubuk et al. 2018: Lim et al. 2019). Each operation is associated with a magnitude that is uniformly sampled from 0,10 The data augmentation parameter T represents probability distribution over the augmentation transformations. For t-th iteration, we sample n² transformations according T² with Sumbel-Sofftass reparameterization (Maddison et al. 2016) and to generate the corresponding augmented samples in the batch. Given sampled architecture, the loss function for each augmented sample is denoted by L , 0'; (xx))))), where To represents the selected transformation. In order to relax T to be differentiable, we regard p& PT(),, the probability of sampling the transformation Tk, as an importance weight for the loss function of corresponding sample Ld (f((a,, 0¹; Tx(x))). The objective of data augmentation is to minimize the following loss function:
n" CDA(T)) = E ('' Txxa))
(2)
With this loss function, DHA intends to increase the sampling probability of those transformations that can generale samples with high training loss. By sampling such transformations, DHA can pay more attention to more aggressive DA strategies and increase model robustness against difficult samples (Zhang ct a... 20200. However, blindly increasing the difficulty of samples may cause the augment ambiguity phenomenon Wei et al. 2020]: augmented images may be far away from the majority of clean images, which could cause the under-fit.ting of model and deteriorate the learning process. Hence, besides optimizing the probability matrix of DA strategies, we randomly sample the magnitude of each chosen strategy from an uniform distribution, which can prevent lcarning heavy DA strategics: augmenting samples with large magnitude strategies. Moreover, instead of training controller to generate adversarial augmentation policies via reinforcement learning (Zhang et al. labels for augmented samples (Wei et al. 2020), we search for the probability distribution of augmentation transformations directly via gradient-based optimization. In this way, the optimization of data augmentation is very efficient and hardly increases the computing cost.
2020) or training an extra teacher model to generate additional
3.2 Hyper-parameters
As shown in Fig. given the batch of augmented training data ((T& (xk), y1 )r and the sampled child network, we need optimize the differentiable hyper-parameters n, such as learning rate and L2 regularization. At the training stage, we alternatively update @ and n. In t-th iteration, we can update O based the gradient of the unweighted training loss (f 0': (') = written as:
1 He (" Txxx)),) which can be
g++1 = OP(0', mn', Vol (f(a', 0¹, B'))).
(3)
where OP(-) is the optimizer. To update the hyper-parameters n we regard 0++1 as function of n and compute the training loss C" 00*+1 Brr) with network parameters (1*1(n")) mini-batch of training data Bir Then, n is updated with V, Vn(e ('+)("';; by gradient descent:
(4) where 3 is learning rate. Even 0r can also be deployed to 0--1 whose caleulation also involves n²' we take approximation method in Equ. 4 and regard Of here variable independent of 17' Instead of splitting an extra validation set for HPO, we directly sample subset from training set to update n, which could ensure that the whole training set is used in updating T, a, and O. This avoids the final learned weight decay coefficient to be zero as non-zero weight decay coefficient can help avoid the model to overfit to the training data.
n = n7 - ,,, (f(aa,, 0*+1 (+''n");
3.3 Architecture parameters
With the augmented data in previous section, we achieve the optimization of the architecture parameter through end-to-end NAS, motivated by SNAS (Xie et. al. 2019), DSNAS (Hu et al. 2020) and ISTA-
5

---

Published in Transactions on Machine Learning Research (10/2022)
Algorithm 1 DHA
Initialization: Data Transformation Parameters т, Hyper-parameters 7) Compressed Representation b,
Mcasurement Matrix A, and Network Paramcters 0 Input: Training Set D Parameters 7. b, A, 0, and the iteration number T
Return: 7, 7, a.
while < T do
Separately sample a mini-batch D from D
For each sample my in mini-batch B sample transformation Te(rs) according to r';
Recover at by solving Equ. 5 with b and A;
5: Extract a child network from the super network:
6: Compute the weighted training loss function as Equ. 2 and update r accordingly;
7: Calculate g with Eqn. 3
Use training loss function to update b through the gradient descent, then 4+1 updated with Eqn.
Compute the training loss function with +++ Dual and update n with Equ.
10: end while
NAS Yang et al. 2020 Following Liu et al. 2019 we denote the cach space as a single directed acyelic graph (DAG), where the probability matrix a consists of vector a.-- = j.., Lfj. ..., a4,1 and of represents the R",
probability of choosing nhhh operation associated with the edge (i, Instead of directly optimizing adopt ISTA-NAS to optimize its compressed representation b Rmm where m << n, which can be writlen as:
b b= Aa +
(5)
where e E I represents the noise and E Rxx the measurement matrix which randomly initialized. Eqn. is solved through using LASSO loss function (Tibshirani 1996) and the a is optimized by using iterative shrinkage thresholding algorithm (Daubechies al. 2004). which can be written as:
a 1+1 = ma/Laaa - - A (Aa - -AAa(AA))))))) =,,,,.
(6)
where L represents the LASSO formulation which can be written as min Aa -b1l + A||a|111 the A represents the regularization parameters and the aayL is the shrinkage operator as defined in Bock &
Teboulle 2009 Thus we have:
= - - (b, Ey)o,,
where o, refers to all possible operations connected to note j and E, = AJA, - I. With this relaxation, b can be optimized through calculating the gradient concerning training loss. The main reason for using this optimization algorithm that it can optimize the high-dimensional architecture parameters through optimizing low-dimensional embeddings, which can largely decrease the optimization difficulty and increase the optimization efficiency. Moreover, this algorithm also adopt the weights sharing in the optimization process which can be readily combined with our proposed data augmentation and syper-parameter optimization method.
3.4 Joint-optimizant
Basod the above analysis of each AutoML module, DHA realizes end-to-end joint optimization of automated data augmentation parameters T, hyper-parameters v,, and architecture parameters a. The DHA algorithm is summarized in Algorithm [ One-level optimization is applied to and in Line 6 and Line 8, while
bi-level optimization is applied to 77 as in Line 9.
One thing worth mentioning is that different optimizers are adopted for different parameters. There are two main reasons behind this choice. Firstly, 7,7, a and 0 work differently the optimization process, e.g., T controls the transformation strategy for the training set while a related to the architecture selection. Besides 0, other parameters could not directly be optimized through the gradient descent on training set. These differences cause the different. optimization methods for 7, az 7 and 0. Secondly, T, ax, n and 0 have

---

Published in Transactions on Machine Learning Research (10/2022)
CIFAR10
CIFAR100 00
SPORT8
99
96
Accuracy (%)
98
95
82
94
97
80
93
96
78
92
95
76
91
50
100
150
50
100
150
2.5
5.0
7.5
10.0
12.5
MIT67
FLOWERS102
IMAGENT
74
98
78
73
77
Accuracy
72
76
96
71
75
95
70
74
69
75 Time (Hours)
100
94
20
40 Time (Hours)
600 800 100012001400
Time (Hours)
ENAS NSGA-NET
DARTS PDARTS
MANAS
Sequential DHA
One-Stage ISTA
DHA
Figure 2: Top-1 accuracy and computational time of AutoML algorithuss for classification task on CIFARIO, CIFAR100, SPORT8, MIT67, FLOWERS110 and ImageNet. DHA is conducted with Cell-Based search space. Fllipse centres and ellipse edges represent the = {0, a/2),, respectively (mean μ4, standard deviation r). For ImageNet, we present test accuracy without error bars, as the error bars are not reported in existing works.
different dimensions and different scales, which makes the joint-optimization with a uniform optimizer extremely impracticable. This is why current works concerning DA (Ho al., 2019 NAS (Yao al. 2020 Nekrasov et al., 2019) and HPO (Falknr al. 2018) always adopt different optimizer for network's weights and prrper-parameers they aim to optimize. During the training process, DA and HPO adapt the online optimization strategy. The DA strategy and HPO settings are evolving with the weights of the parameters.
Moreover, the main reason that DHA could optimize large-scale search space in an effectively manner, is that DHA delicately adopt weight-sharing in the jointoptiimizatonn for different parameters. Instead of only optimizing a sub-network with a DA strategy and pepea-parameter setting check the performance of certain setting, we realize the jointooptimization with the help of super-net network, DA probability matrix and continuous hyper-parameter setting. In that way, DHA can make use of previous trained parameter weights to check the performance setting, which largely decreases the computational request.
4 Experiments
In this section, we empirically compare DHA against existing AutoML algorithms on various datasets. With extensive experiments, demonstrate the benefits of joint-optimization over sequential-opiimizziion in terms of generalization performance and computational efficiency.
4.1 Experiment setting
Datascts. Following Ru et al. (020b),, we conducted experiments on various datascts, including CIFAR1O and CIFAR100 for the object classification Lask (Krizhevsky al. 2009), SPORTS for the action classification
task (Li & Fei 2007), MIT67 for the classification task 'Quatooni & Torralba 2009), FLOWERS102
for the fine object classification task (Nilsback & Zisserman 2008). and
(uusskkkkuuskkkkk et al. 2015
for the large-scale classification task. The accuracy is calculated on the test set.
Search space. (1) Automated DA. Following Ho et al. 2019), we consider = 14 different operations for data augmentation, such as AutoContrast and Equalize. The magnitude of cach operation randomly sampled from the uniform distribution. Note that we follow widely used data augmentation setting (Zhang
et al. 2020 Zhou et al. 2021), which applies two successive data augmentation strategies each training
7

---

Published in Transactions on Machine Learning Research (10/2022)
Table Top-1 accuracy (%) and computational time (GPU hour) of different AutoML algorithms on CIFARIO, CIFAR100, SPORTS, MIT67 and FLOWERSI02 with Cell-Based Search Space. We report the sum of search time and tune time two-stage NAS and the whole running time for one-stage NAS. ourselves. Different NAS algorithms for one dataset are performed under the similar parameter weights constrain to ensure fair comparison.
methods are implemented by
CIFARIO
CIFARI00
SPORT8
MIT67
FLOWERSI02
Model
Acc
Ace
Acc
Acc
Acc
ENAS Pham et al.
2018
95.85-00.17
78.02+00.55
94.54+0.335
71.05+00.29
96.56-10.20
NSGA-NED LEGA-NET-PAA Lu
2019
96.18=0.37
77.31+0.14
92.5
00.20+0.441
95.30+0.266
DARTS Liu et al.
2019
97.2410.10
82.3710.34
93.8710..35
70.73.10.24
96.30.10.24
P-DARTS Chen
2019
97.13.0.007
82.4610.377
92.45.10.66
70.70.0..29
95.0910.226
MANAS Carlucci et al.
2019
97.18+0.07
82.07=0.14
94.46+0.22
71.36+0.19
96.57=0.18
One-Stag One-Stage ISTA Vang et al.
2020
97.64+0.20
83.10+0.11
94.33++0.12
72.12+0.03
96.59+-0.16
Sequential DHA
97.77=0.14
83.51=0.12
94.55+00.09
72.34:0.14
96.67-0.117
DHA
98.11-0.26
83.93:10.23
95.06:10.13
73.35+00.19
97.4110.09
Time
Time
Time
Time
Time
ENAS Pham al.
2018
67.2.12.1
67.214444
1.92
16.2412.6
21.2111.5
NSGA-NET
Lu ct a.. 2019
151.2.14.1
151.2.16.5
11.06.10.88
103.9144.1
54.412.11
DARTS
Liu et al. 2019
151.2+2.9
151.243.8
11.06+1.4
103.9+3
54.4+1.6
P-DARTS Chen a.. 201
76.8+1
76.8+2.6
5.62++088
52.8=2
27.6+1.6
MANAS
Carlucci et al.
2019
122.411.8
122.412.9
8.96
84.111.6
4411.
One-Stage ISTA (ang
2020
55.2115
55.2+12
4.110.3
37.910.4
19.811.3
Sequential DHA
101.9+1.6
101.9+2.3
7.3+0.5
67.9+1.6
35.4+1.9
DHA
76.6#3.2
76.613.1
5.640.5
52.5+1.4
27.411.2
sample. Our method can be easily extended to more general cases by modifying the probability matrix and the range of sampling distribution. For example, for the case of three successive data augmentation strategies, we could extend the size of probability matrix from K2 to K3, where is the number of data augmentation categories. (2) NAS. Following Liu et al. (2019) and Cai et al. (2099), we consider both the cell-based and the MobileNet scarch space, which regards the whole architecture
(3) HPO. We consider both the L2 regularization ion (i.e., weight decay) and the learning rate in the experiments. Detailed information is provided in Appendix
stack similar cells.
Basclincs. We compare DHA with various AutoML algorithms, such as ENAS Pham et al. 2018 NSGA- 2019), MANAS Carlucci el al., 2019 and One- Stage ISTA YYang et al. 2020) see Table D and Table 2). For datasets including CIFARIO, CIFAR100, SPORT8, MIT67, and FLOWERS102,
NET Lu et al. 2019 NSGA-NET (Lu el al. 2019) P-DARTS (Chen
re-implement the basclinc methods. Specifically, for papers with code, incluing ENAS, NSGA-NET, P-DARTS and One-Stage ISTA, directly use their official implementation code. For paper without code, including DARTS and MANAS, use the unofficial re-implementation code which achieves similar performance described in the original papers. As for ImageNet, we directly refer to the model performance reported in papers cited in Table 2 To further demonstrate the benefits of joint optimization ion of multiple AutoMT, components, we also include a baseline, Sequential DHA, which resembles the common practice by human optimize different components in sequence. Specifically, Sequential DHA consists of two stages. During the first stage, Sequential DHA performs NAS to find the optimal architecture under certain parameee settings. In the next stage, Sequential DHA performs the online DA and HPO strategy proposed in our paper and trains the architecture derived from the first. stage from scratch. Detailed algorithm of Sequential DHA could be found in Appendix
B
Implementation details. For paraemerssmeterss of DHA, we simply adopt the previous works without many modifications, including Meta-Aug (Zhou et al.] 2021). ISTA-NAS Yang et al. 2020), and DSNAS (Hu et al. (2220). Experiments are conducted on NVIDIA V100 under PyTorch-1.3.0 and Python 3.6. Detailed settings of baselines are provided in Appendix
used in
8

---

Published in Transactions on Machine Learning Research (10/2022)
Table 2: Comparison with SOTA image classifiers on ImageNet in the Cell-Based (C) setting or MobileNet/ShufileNe (M) setting. (* denotes the architecture is searched ImageNet, otherwise it is searched on CIFAR-10 CIFAR-100.1
Test Acc (%)
Params
Cost (GPU-day)
Search
Model
Search Attribute
Top-1
Top-5
(M)
Search
Eval
Space
DARTS (2nd) Liu et al.
2019
73.3
91.3
4.7
4.0
3.6 x
Arch
SNAS (mild) Xie
72.7
90.8
3.3
Arch
GDAS Dong Yang
74.0
Arch
Bayus AS Zhou el al.
2019
73.5
91.1
3.9
0.2
3.6 x
Arch
PARSEC Casale et a..
2019
74.0
91.6
5.6
1.0
3.6 x
Arch
P-DARTS (CIFATE-U) 10] Chen ct al.
2019
75.6
92.6
4.9
0.3
3.6 X
Arch
C
P-DARTS (CIFAR-100
CCen et al
2019
75.3
92.5
5.1
0.3
3.6 x
Arch
C
PC-DARTS (ImageNet)
2020
75.8
92.7
Arch
GAEA
PAEA-RC-DARTS
2021
76.0
92.7
Arch
DENAS Chen et al. 2021
76.3
92.9
5.7
4.6
3.9 X
Arch
Twa-Stage ISTA
Yang el al.
2020
75.0
91.9
5.3
2.3
3.4 x
Arch
One-Stage ISTA Yang
2020
76.0
92.9
5.7
4,2
Arch
LificientNee--D
Le
2019
77.1
93.3
Arch
SAs
al.
2019
74.7
Arch
ProxylessN AS (GPU) (Cai cl al.
2019
75.1
92.5
7.1
8.3
3.6 x
Arch
DSNAS Hu et al 2020
74.3
91.9
3.7 x 8
Arch
OFA (small) Caa a.. 2020
76.9
93.3
5.8
6.8 8
Arch + Resolution
APQ Wang
2020
75.1
12.5
Arch Pruming
AutoHAAS (Dong al
2020
74.2
Arch HPO
Sequential DHA
76.7
5.4
5.5
Arch DA + HPO
DHA!
77.4
94.6
5.8
Arch + DA + HPO
Sequential DHA
77.1
93.9
5.7
5.8
5.7 x
Arch + DA F HPO
DHA
77.6
94.8
5.3
5.4
x 8
Arch DA + HPO
M
Table 3: Top-1 accuracy (%) and computational time (GPU hour) different combination of AutoMI, components on CIFAR10, CIFAR100, SPORTS, MIT67 and FLOWERS102 Listed algorithms are described in Ablation study.
CIFAR10
CIFAR100
SPORTS
MIT67
PLOWERS100
Model
Acc
Time
Acc
Time
Acc
Time
Acc
Time
Acc
Time
Sequential NAS+DA
97.74
57.50
83.45
57.50
94.49
4.20
72.22
39.20
96.64
20.50
Sequential NAS+HPO
97.54
63.60
83.13
63.60
94.47
4.60
72.10
43.10
96.57
22.50
Sequential DHA
97.77
101.90
83.51
101.90
94.55
7.30
72.34
67.90
96.67
35.40
Joint-oplinuization NAS+DA
97.78
64.00
83.55
64.00
94.53
4.70
72.38
43.80
96.72
22.90
Joint-optimization NAS+HPO
97.75
70.80
83.23
70.50
94.50
5.20
72.23
47.80
96.69
25.40
DHA
98.11
76.60
83.93
76.60
95.06
5.60
73.35
52.50
97.41
27.40
4.2 Results
The test accuracy and computational time of various AutoML algorithms are summarized in Table D and Table 2
The timing results in these two tables measure the computational time taken to obtain ready-to- deploy network, which corresponds to the sum of search and retrain time for two-stage NAS methods and the search time for end-to-end methods like one-stage NAS method as well as our DHA.
Small-scale datasets. As shown Table D methods optimizing all DA, HPO and NAS automatically (i.e, Sequential DHA and DHA) consistently outperform those NAS algorithms with manual designed DA and HPO. Specifically, DHA achicves SOTA results on all datascts. This shows the clear performance gain of extending the search scope from architecture to including also data augmentation and hyper-parametent justifying the need for moltt-component optimization in AutoML. Moreover, despite optimising scarch space, DHA remains cost efficiency. For example, on CIFAR100, DHA enjoys 1.56% higher test accuracy than DARTS but requires 42% less time. Besides, the comparison between DHA and Sequential DHA reveals the evident. advantage of doing DA, HPO and NAS jointly over doing them separately in different. stages.
larger
Large-scale dataset. Results of the large-scale dataset ImageNet with cell- cell-based search space and MobileNet search space shown in Table 2 DHA consistently outperforms various NAS methods which only involves

---

Published in Transactions on Machine Learning Research (10/2022)
architecture optimization, demonstrating the benefits of joint-optimization. One-stage NAS methods like ISTA, DHA achieves up to 1.7% TOP-1 accuracy improvement. Moreover, in comparison with the joint-optimization algorithm APQ (Wang et al. 2020) and AutoHAS (Dong et al. 2020), DHA outperforms APQ by 2.5% and outperforms AutoHAS by 4.3%. These comparisons reveal that DHA proposes an efficient and high-performet joint-oplimization algorithm. The Top-1 accuracy and computation time of these AutoML algorithms are also summarized Fig. 2 As can be seen, DHA consistently gains highest test accuracy on all five datasets while being more cost efficient than NAS methods and Sequential DHA. The validation accuracy of Sequential-DHA and DHA during the training is shown in Fig. S We could notice that traditional sequential optimization algorithm would disrupt the learning process, it has to retrain the model after the searching phase. While benefiting from the end-to-end optimization process, DHA would have a smoother learning curve and also achieve belter performance.
Even when compared with
Learning Curve On Sequential DHA and DHA
Sequential DHA (Search) Sequential (Retrain)
DHA
80
Accuracy (Patcenagge
40
20
50
100
150
200
250
300
Epochs
Figure 3: The validation accuracy of Sequential-DHA and DHA over the training time ImageNet with Cell-Based Structure.
4.3
Analysis of loss landscape
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
-0.5
-0.5
-0.5
-1.0
-1.0
-1.0
(a) One-Stage ISTA
(b) Sequential DHA
(c) DHA
Figure 4: Lase-land of the trained models on SPORTS, (a) One-Stage ISTA (b) Sequential DHA and (c) DHA. The strong test performance of DHA across various datasets motivates to further check the geometry of the minimiscr achieved by our final trained model. We employ the filter-nrmaaliatioo method proposed in Li et al. (2017) lo visualise the local loss landscape around the minimiser achieved by One-Stage ISTA, Sequential DHA and DHA. We choose center point 0* in the graph, and two direction vector d and n. We the plot the 2D contour with the function f(a,5) 3) L(f(O" + ad + fn: Biest)), where grest represents the test set. The resultant contour plots are shown in Fig. g As can be seen, the minimum reached by optimising more AutoML components tend to be flatter than NAS with manual designed DA and HPO.
10

---

Published in Transactions on Machine Learning Research (10/2022)
The loss landscape of DHA is also flatter and smoother than that of Sequential DHA accounting for the better generalisation performance of DHA in previous experiments, explaining the superior test accuracy
achieved by DHA in Tables and 2 [eskar et al. 2016 Xu & Mannor 2012
4.4 Ablation study of AutoML components
To further verify the effectiveness of our proposed extended search scope and search strategy, we empirically investigate the performance considering all three components (i.e., DA, HPO and NAS) against combining any two of them. We cxamine both the sequential-optimizatonn and joint-optimization scttings with four designed optimization algorithms. Sequential NAS + DA first conducts NAS and during the tuning stage, the proposed DA optimization applied. Similar to the Sequential NAS+DAA Sequential NAS + HPO firstly conducts NAS. Then with the fixed architecture, Sequential NAS + DA optimizes the HPO and networks' weights simultaneoully. In contrast to them, Joint-optimization NAS + DA simultancously conducts DA strategy optimization, NAS and parameter weights optimization. Joint-optimization NAS + HPO simultancously conducts HPO, NAS and parameter weights optimization.
The comparison results are presented in Table We can notice that performing joint optimization for either NAS + DA NAS + HPO, achieves higher test accuracy than doing them in pipeline. This reconfirms previous conclusion that optimising different AutoML components jointly is better than doing them in sequence. Moreover, by comparing the results of the cint-optimization NAS+DA and the joint-oplimization NAS+HPO in Table 3 against the One-Stage ISTA in Table D1 it is clear that considering one more AutoML component on top of NAS can lead to clear performance gain of the final model. While such gain is higher for incorporating DA than HPO, is maximised when all three components are considered; our DHA obtains the best test accuracy among all joint-optimization baselines across all datasets.
We also provide (1) more ablation studies of DA and HPO (see Appendix C, (2) analysis of the searched architectures (see Appendix D and comparison in terms of FLOPs and Latency (see Appendix B Please refer to the Appendix for more details.
5 Conclusion
In this work, we present DHA, an end-to-end joint-optimization method for three important components of AutoML. including DA, HPO and NAS. This differentiable joint-optimization method can efficiently optimize larger search space than precious AutoML methods and achieve SOTA results on various datasets with a relatively low computational cost. Specifically, DHA achieves 77.4% Top-1 accuracy on ImageNet with cell based search space, which higher that current SOTA by 0.5%. With DHA, we show the advantage of doing joint-optimization of AutoML over doing co-optimization in sequence, and conclude that joint optimization of multiple AutoML components is necessary.
6 Acknowledgments
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
References
Ahsan S Alvi, Binxin Ru, Jan Calliess, Stephen J Roberts, and Michael A Osborne. Asynchronous batch bayesian optimisation with improved local penalisation. In ICML, 2019.
Amir Beck and Mare Teboulle, A fast iterative sarinkage-thresholddi algorithm for linear inverse problems. STAM Journal on Imaging Sciences, 2(1):183-202, 2009.
Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12( (188):1889-111 1900, 2000.
11

---

Published in Transactions on Machine Learning Research (10/2022)
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLRR 2020.
Fabio Maria Carlucci, Pedro M Esperança, Marco Singh, Victor Gabillon, Antoine Yang, Hang Xu, Zewei Chen, and Jun Wang. Manas: Multi-agent neural architecture search. arXiv preprint ar 1ix: 1909.01051, 2019.
01051,
Francesco Paolo Casale, Jonathan Gordon, and Nicolo Fusi. Probabilistic neural architecture search. arXiv preprinl trXiv:1902.00 05116, 2019.
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsich. DrNAS: Dirichlet neural architecture search. In ICLR, 2021.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In CVPR, 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXie preprint or 180::11155000000 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. F3NetV3: Joint architccture-reciee search using neural acquisition function. In CVPR, 2020.
Ingrid Daubcehics, Michel Defrise, and Christine De Mol. An itcrative thresholding algorithm for lincar inverse problems with sparsity constraint. Communications on Pure and Applied Mathematics; A Journal Issued by the Courant. Institute of Mathematical Sciences, 57(11):1413-14 1457, 2004.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In CVPR, 2019.
Xuanyi Dong, Mingxing Tan, Adams Wei Yu, Daiyi Peng, Bogdan Gabrys, and Quoe V Le. AutoHAS: Differentiable syper-paramett and architecture search. arXiv preprint arXie:2006.0300 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In ICLR, 2019.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In 1CML, 2018.
Javier González, Zhenwen Dai, Philipp Hennig, and Neil D Lawrence. Batch Bayesian optimization via local penalization. In International Conference Artificial Intelligence and Statistics, 2016.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXie preprint arxiv:1904. 00420, 2019.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Picter Abbeel. Population based augmentation: Efficient. learning of augmentatiin policy schedules. In ICML, 2019.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: Direct neural architecture search without parameter retraining. In CVPR, 2020.
Daniel Jiwoong Im, Cristina Savin, and Kyunghyun Cho. Online hyperparameter optimization by real-time recurrent learning. arXiv preprint arXiv:2102.078: 2021.
12

---

Published in Transactions on Machine Learning Research (10/2022)
Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. Multi-fidelity Bayesian optimisation with continuous approximations. In TCML, 2017.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In NeurIPS, 2018.
Taiga Kashima, Yoshihiro Yamada, and Shunta Saito. Joint search of data augmentation policies and network architectures. In CoRR, 2020.
Tarun Kathuria, Amit Deshpande, and Pushmeet Kohli, Batched Gaussian process bandit optimization via determinantal point processes. In NewrIPS, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint XrXiv:1609. 04090, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Tech Report, 2009.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint or orXi::1712.090) 2017.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. ar preprint arXiv:2002. 11770, 2020.
Li-Jia Li and Li Fei-Fei. What, where and who? classifying events by scene and object recognition. In ICCV, 2007.
Liam Li, Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Geometry-awar gradient algorithms for neural architecture search. In ICLR, 2021.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NearIPS, 2019.
Chen Lin, Minghao Guo, Chuming Li, Xin Yuan, Wei Wu, Junjie Yan, Dahua Lin, and Wanli Ouyang. Online hyper-parameter learning for auto-augmentation strategy. In CVPR, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Slatistics, 2020.
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Compulation Conference, 2019.
Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXie preprint orxiv:990s 000.033888 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparamoter optimization through reversible learning. In ICML, 2015.
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A contimuous relaxation of discrete random variables. arXiv preprinl 16100011, 2016.
Alan Tan Wei Min, Abhishek Gupta, and Yew-Soon Ong. Generalizing transfer bayesian optimization to source-target heterogeneity. IEEE Transactions on Aulomalion Science and Engineering, 2020.
Gaurav Mittal, Chang Liu, Nikolaos Karianakis, Victor Fragoso, Mei Chen, and Yun Fu. HyperSTAR: Task-aware hyperparameters for deep networks. In CVPR, 2020.
13

---

Published in Transactions on Machine Learning Research (10/2022)
Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In CVPR, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. In International Conference on Neural Information Processing Systems, 2018.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In ICML, 2018.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In CVPR, 2009.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv:1802.0 01 548, 2018.
Binxin Ru, Absan S Alvi, Vu Nguyen, Michael A Osborne, and Stephen J Roberts. Bayesian optimisation over multiple continnous and categorical inputs. In ICML, 2020a.
Binxin Ru, Pedro Esperanca, and Fabio Carlucci. Neural architecture generator optimization. In NewrIPS, 2020b.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenet V2: Inverted residuals and lincar bottlenecks. In CVPR, 2018.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots Truncated propapropngttion for bilevel optimization. In International Conference on Arlificial Intelligence and Statistics, 2019.
Kevin Swersky, Jasper Snock, and Ryan Prescott Adams. Multi-task bayesian optimization. In NewrIPS, 2013.
Mingxing Tan and Quoe Le. ElficientNet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Zhiqiang Tang, Yunhe Gao, Leonid Karlinsky, Prasanna Sattigeri, Rogerio Feris, and Dimitris Metaxas. Onlineaugment: Online data augmentation with less domain knowledge. In ECCV, 2020.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288) 1996.
Tianzhe Wang, Kuan Wang. Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. APQ: Joint search for network architecture, pruning and quantization policy. In CVPR, 2020.
Longhui Wei, An Xiao, Lingxi Xie, Xin Chen, Xiaopeng Zhang, and Qi Tian. Circumventing outliers of autoaugment with knowledge distillation. In CoRR, 2020.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229- 256, 1992.
Jian Wu, Saul Dascano-Palmerin Peter Frazier, and Andrew Gordon Wilson. Practical multi-fidelity Bayesian optimization for hyperparameter tuning. Tn UAT, 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic neural architecture search. In ICLR, 2019.
Huan Xu and Shie Mannor. Robustness and generalization. Machine Learning, 86(3):391 -423, 2012.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts:
Partial channel connections for memory-effiiiet architecture search. In TCL.R, 2020.
Antoine Yang, Pedro M Esperança, and Fabio M Carlucci. Nas evaluation is frustratingly hard. arXiv
preprint 191: 12522, 2019.
Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding. arXie preprint orXiv:2010.06177, 2020.
Lewei Yao, Hang Xu, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Sm-nas: structural-to-mmddhaar neural architecture search for object detection. In AAAI, 2020.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint 1XXt::807700906 2018.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong, Adversarial autoaugment. ICLR, 2020.
Fengwei Zhou, Jiawei Li, Chuanlong Xie, Fei Chen, Lanqing Hong, Rui Sun, and Zhenguo Li. Metaaugment: Sample-aware data augmentation policy learning. In AAAI, 2021.
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. architecture search. In ICML, 2019.
Bayesnas: A bayesian approach for neural
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
15


Reviews:
[{"review_id": "JUebrhhH0s", "reviewer": "Reviewer_HZRV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ Using the continuous parameters and a differential method to do joint DHA\n+ Experimental results are promising when compared to other NAS methods.\n\n\n## Weaknesses\n\n- In general, grammar needs to be checked throughout\n- Organization of the paper needs improvement \n- Most choices that make continuous optimization possible are simplifying ones. For example, the data augmentation case, defined as choosing 2 consecutive augmentations, with parameters of each augmentation ranging from [0,10]. In practice, this is could be flexible. A discussion on this may help - \"Can the formulation extend to multiple augmentations in any range?\"\n- Joint optimization is in fact $\\it{not}$ joint optimization. It is a sequential update of different dimensions in order. The reasons for this as stated are two-fold - 1. Each parameter works differently, and $\\theta$ or model parameters cannot be optimized directly, and 2. Intractable/ Impractical to optimize all parameters together. These are not strong reasons, and it is unclear if actual joint optimization may lead to even better results. Moreover, this seems like a justification to the final methodology chosen, and not like a motivation that lead to the method.   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores joint Data Augmentation, HPO and NAS. As the authors claim, it is rare to see joint DHA optimization due to its expensive nature. They key to implemnting joint optimization is to create a differential space parameterized individually for 'D', 'H' and 'A', followed by \"One-level optimization is applied to 'D' parameter and the 'H' parameter, while bi-level optimization is applied to 'A' parameter.\" ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- [Minor] Can Figure 1 be moved to an appropriate location (are these results?)\n- [Minor] Without any context in page 2, \"weight sharing is delicately applied.... super-network\" does not make sense. All these all terms in Xie et al? What is \"delicately adopting\"?\n-  Page 2 - with the help of \"a\" sparse coding method?\n- \"Specifically, the DA and HPO are regarded as dynamic schedulers, which adapt themselves to the update of network parameters and network architecture\" is in fact a great summary to add to the abstract\n- Organization of the paper (order of sections, figures etc needs to be verified again). For example, before reaching section 2, we have covered the motivation, some of the literature and results in Figure 1. We then start with another detailed literature review in section 2\n- [Minor] - \"the query based and the gradient based\" - remove the\n- [Minor] - \"and then apply or search for\" - rephrase sentence\n\n(Ignoring grammatical errors from here on as there are many to list)\n\n- For the experimental section, a discussion on choice of algorithms is required. IF AutoHAS is not similar to DHA in structure, joint optimization can be expected to get better results overall. Experimental results are also only marginally better, and the actual implementation of the competing algorithms is unclear apart from a mention of \"All the methods are implemented by\nourselves. Dierent NAS algorithms for one dataset are performed under the similar parameter weights constrain to\nensure fair comparison.\" It is unclear how this fair comparison is actually made; a discussion on these details will ensure readers understand this claim", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KN9gL1wgjd", "reviewer": "Reviewer_JcoF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---\n**Strengths:**\n\nThe problem studied in this paper is essential in real-world scenarios. Finding the proper data augmentation, hyper-parameter, and neural architecture is very challenging in practice. Therefore, it is vital to have a push-the-button solution that could achieve good performance. Though simple, the proposed DHA could be a valuable first step towards this goal.\n\nThe paper is well-written and easy to follow. The authors formulate the problem at the beginning of Section 3 and provide sufficient technical details in the following subsections. The authors also include some discussions at the end of Section 3 to help the readers better understand the relationship and differences between previous solutions and the proposed DHA.\n\n---\n**Weaknesses:**\n\nThe authors have only provided the accuracy and model size in Table 2. This is not very informative as most baselines are searched under either computational cost (#FLOPs) or latency constraints. It would be necessary to report these two efficiency metrics for an apples-to-apples comparison with the baselines. As the results on small-scale datasets are not representative, the authors could consider evaluating their proposed DHA on other large-scale benchmarks (such as COCO object detection). Even a direct policy transfer would be sufficient.\n\nFrom Table 2, the improvement of joint optimization is not very significant over sequential optimization (i.e., 0.5-0.7% top-1 accuracy). This makes it a bit questionable whether it is necessary to perform the joint optimization (especially when considering the extra engineering efforts needed). It would be great if the author could share some insights and thoughts on this.\n\nThe differentiable formulation is very nice since it can be effectively optimized using gradient-based methods. However, it is unclear whether it can cover all sorts of data augmentation policies and hyper-parameters. For instance, adding the quantization into the search space seems a bit challenging with the current differentiable formulation. The authors could comment on this.\n\nThe proposed pipeline requires a considerable amount of engineering effort to implement. Therefore, it would be super helpful (but no pressure!) if the authors could release their code upon the paper's acceptance to facilitate the reproduction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies end-to-end optimization of data augmentation policy, hyper-parameter, and neural architecture. Existing pipelined solutions are time-consuming and sub-optimal. Therefore, the authors introduce DHA to offer an end-to-end solution. Technically, the authors formulate data augmentation, hyper-parameter, and neural architecture in continuous spaces to be optimized differentiably. The authors have evaluated their proposed DHA on multiple small-scale benchmarks and on ImageNet, achieving encouraging results.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section for more details.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xr4XMd4BPH", "reviewer": "Reviewer_HSAw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- Firstly, the paper is generally well written and easy to follow.\n- Solid ablation studies in Table 3 on smaller datasets show the superior performance of joint optimization. \n- The loss landscape visualization results confirm that the proposed method leads to better optimization.\n- It targets a very ambitious target to perform end-to-end optimization on the whole training procedure. \n\nCons:\n- There is only parameter-constrained search, which may not provide a fair comparison to existing work since many of them are designed for computation efficiency (e.g., EfficientNet) or hardware efficiency/latency (e.g., ProxylessNAS).\n- For hyper-parameter optimization, the design space is limited to weight decay and learning rate. Is there any possibility of extending to more hyper-parameter settings like optimizers (SGD vs. Adam)?\n- The proposed method optimizes three components in the same loop. Can the author discuss the difficulty of hyper-parameter adjustment (in the DHA method itself but not training)? Is the adjustment easy and general? I am asking because some design knobs seem non-trivial (e.g., use different optimizers for different parameters).\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors proposed DHA, an end-to-end AutoML method to optimize data augmentation (DS), hyper-parameters (HPO), and model architectures (NAS) in the same loop. By using weight-sharing for super network, data augmentatio0n probability matrix, and continuous hyper-parameter setting, DHA can efficiently explore the large search space. Experiments on multiple datasets show that DHA outperforms existing work and sequential optimization methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "As discussed above, it would be better to:\n- Compare with existing AutoML work on their specific optimization setting (e.g., FLOPs, latency, etc.) instead of just comparing the model size. \n- Discuss the difficulty of hyper-parameter tuning of the proposed method. Do we need to carefully tune the experimental setting to get good results?", "broader_impact_concerns": "I don't see there is a prominent ethical concern in this paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for jointly optimizing architecture, hyperparameters, and the data augmentation policy of a neural network in a joint fashion using a differentiable mapping for each component. This is an ambitious and difficult problem, and the method appears to work quite well, finding settings with high accuracy, low latency/FLOPS (even if this isn't explicitly accounted for, from what I can see), in a reasonably efficient manner. There are limitations, such as for discrete data, that should be discussed in the paper.

The discussion with the reviewers highlighted some points of clarification. Among them:

- How to generalize the augmentation optimizer to incorporate more augmentation strategies.
- The fairness of the experiments.
- A discussion on setting the hyperparameters of DHA.

Amongst others. There are also additional results on FLOPS/Latency of the discovered models.

Please incorporate these points from the discussions raised with the reviewers. Otherwise, the reviewers unanimously lean towards acceptance and feel that this is a solid step towards an end-to-end AutoML solution.
Decision: Accept (poster)