DiffAutoML: Differentiable Joint Optimization for Efficient End-to-End Automated Machine Learning
Authors: Kaichen Zhou
Lanqing HONG
Fengwei Zhou
Binxin Ru
Zhenguo Li
Trigoni Niki
Jiashi Feng
Venue:  (2021)
Keywords: Differentiable
Automated machine learning
Neural Architecture Search
Data Augment
Hyperparameter Optimization

Abstract: The automated machine learning (AutoML) pipeline comprises several crucial components such as automated data augmentation (DA), neural architecture search (NAS) and hyper-parameter optimization (HPO). Although many strategies have been developed for automating each component in separation, joint optimization of these components remains challenging due to the largely increased search dimension and different input types required for each component. While conducting these components in sequence is usually adopted as a workaround, it often requires careful coordination by human experts and may lead to sub-optimal results. In parallel to this, the common practice of searching for the optimal architecture first and then retraining it before deployment in NAS often suffers from architecture performance difference in the search and retraining stages. An end-to-end solution that integrates the two stages and returns a ready-to-use model at the end of the search is desirable. In view of these, we propose a differentiable joint optimization solution for efficient end-to-end AutoML (DiffAutoML). Our method performs co-optimization of the neural architectures, training hyper-parameters and data augmentation policies in an end-to-end fashion without the need of model retraining.  Experiments show that DiffAutoML achieves state-of-the-art results on ImageNet compared with end-to-end AutoML algorithms, and achieves superior performance compared with multi-stage AutoML algorithms with higher computational efficiency.
To the best of our knowledge, we are the first to jointly optimize automated DA, NAS and HPO in an en-to-end manner without retraining.

Under review as a conference paper at ICLR 2021
AUTOML: DIFFERENTIABLE JOINT OPTIMIZA- TION FOR EFFICIENT END-TO-END AUTOMATED MA- CHINE LEARNING
Anonymous authors Paper under double-biee blind review
ABSTRACT
The automated machine learning (AutoML) pipeline comprises several crucial components such as automated data augmentation (DA), neural architecture search (NAS) and hyper-parameter optimization (HPO). Although many strategies have been developed for automating each component in separation, joint optimiza- tion of these components remains challenging due to the largely increased search dimension and different input types required for each component. While conduct- ing these components in sequence is usually adopted quires careful coordination by human experts and may lead to sub-optimal results. In parallel this, the common practice of searching for the optimal architecture first and then retraining it before deployment in NAS often suffers from architec- ture performance difference in the search and retraining stages. An end-to-end solution that integrates the two stages and returns a ready-to-use model at the end of the search is desirable. In view these, we propose differentiable joint op- timization solution for efficient end-to-end AutoML (DiffAutoMi Our method performs co-optimization of the neural architectures, training hyper-parameten and data augmentation policies in end-to-end fashion without the need of model retraining. Experiments show that DiffAutoM.. achieves ImageNet compared with end-to-end AutoML algorithms, and achieves superior performance compared with multi-stage AutoML algorithms with higher compu- tational efficiency. To the best of knowledge, are the first to jointly opti- mize automated DA, NAS and HPO in an en-to-end manner without retraining.
workaround, it often re-
results on
INTRODUCTION
While deep learning has achieved remarkable progress in various tasks such as computer vision and natural language processing, usually requires tremendous human involvement to design and train a satisfactory deep model for one task (He al., 2016; Sandler et al., 2018). To alleviate such burden on human users, a dozen of AutoML algorithms are proposed in recent years to enable training a model from data automatically without human experiences, including automated data augmentation (DA), neural architecture search (NAS), and hyper optimization (HPO) (e.g., Chen et al., 2019; Cubuk al., 2018: Mittal et al., 2020). These AutoML components are usually developed independently. However, implementing these AutoML components for a specific task in separate stages not only suffers from low efficiency but also leads to sub-optimal results (Dai et al., 2020; Dong et al., 2020). How to achieve full-pipeline "from data to model" automation efficiently and effectively is still a challenging problem.
One main difficulty for achieving automated "from data to model" is how to combine different Au- toML components (e.g., NAS and HPO) appropriately for specific task. Optimizing these compo- nents in a joint manner is an intuitive solution but usually suffers from the enormous and impractical search space. Dai et al. (2020) and Wang et al. (2020) introduced pre-trained predictors to achieve the joint optimization of NAS and HPO, and the joint optimization of NAS and automated model compression, respectively. For a new coming task, however, it is usually burdensome to pre- trait such a predictor. On the other hand, Dong et al. (2020) investigated the joint optimization between NAS and HPO via differentiable architecture and hyper- search spaces. Automated DA is seldom considered in the joint optimization of AutoMI. component, Nevertheless, our experimental

---

Under review as a conference paper at ICLR 2021
results showed that different data augmentation protocols may result in different optimal architec- tures (see Sec. 4.3 for more details). Based on these considerations, it is worthy to investigating the joint optimization among automated DA, NAS and HPO.
Another main challenge to achieve automated "from data to model" is the end-to-end searching and training of models without parameter retraining. Even considering only one AutoML component, many NAS algorithms require two stages including searching and retraining (e.g.. Liu et al., 2018; 88; Xie et al., 2019). Automated DA methods such as Lim al. (2019) also needed retrain the model parameters when the DA policies were searched. In these cases, whether the searched architectures or DA policies would perform well after retraining is questionable, due to the inevitable difference of training setup between the searching and retraining stages. Recently, Hu et al. (2020) developed a differentiable NAS methods to provide direct NAS without parameter retraining. Other AutoMI. components, including HPO and automated DA, are seldom considered in the task-specific end-to- end AutoM algorithms.
Considering the above challenges, we propose DiffAutoML, differentiable joint optimization so- lution for efficient end-to-end AutoML. In DiffAutoML, end-to-end NAS optimization is realized in differentiable one-level manner with the help of stochastic architecture search and the approx- imation for the gradient of architecture parameters. Meanwhile, the DA and HPO are regarded as dynamic schedulers, which adapt themselves to the update of network parameters and network chitecture. Specifically, Differentiable relaxation is used in DA optimization in an one-level way while the hyper-gradien is used in HPO in two-level way. With this differentiable method, Dif- fAutoML can effectively deal with the huge search space and the low optimization efficiency caused by this joint optimization problem. To summarize, our main contributions are as follows:
We propose well-defined AutoML problem, i.e., task-specific end-to-end AutoML frame- work, which aims to achieve automated "from data to model" without human involvement.
We first jointly optimize three different AutoML components, including automated DA, NAS and HPO, in a differentiable search space with high efficiency.
Experiment results show that, compared with optimizing modules in sequence, one-stage Di can effectively realize the co-optimization of different modules.
Extensive experiments also reveal mutual influence among different AutoML components, i.e., the change of settings for one module may greatly influence the optimal results of another module, justifying the necessity of end-to-end joint optimization.
2 RELATED WORKS
In this section, we briefly introduce some related AutoML algorithms, including automated DA, NAS and HPO. A more detailed introduction is provided in Appendix D.
Automated Data Augmentation Data augmentation is commonly used to increase the data diversity and thus to improve the generalization performance of deep learning models (DeVries & Taylor, 2017: Zhang et al., 2017; Yun al., 2019). Several prior works have been proposed to automate the search of data augmentation policies, using methods ranging from reinforcement learning (Cubuk al., 2018; Zhang et al., 2020), evolutionary algorithm (Ho al., 2019), Bayesian optimization (Lim et al., 2019), gradient-based approaches (Lin et al., 2019) to simple grid search (Cubuk et al., 2020). Cubuk et al. (2020) point out that the optimal data augmentation policy depends on the model size. This indicates that fixing an augmentation policy when searching for neural architectures may lead to sub-optimal solutions, thus motivating joint optimization.
Neural Architecture Search Recent advances in NAS have demonstrated state-of-the-art perfor- mance outperforming human experts' design on a variety of tasks (Zoph & Le, 2017; Cai et al., 2018; Liu et al., 2018; Kandasamy et al., 2018; Pham et al., 2018; Real et al., 2018; Zoph et al., 2018; Ru et al., 2020b). Especially the development of gradient- based one-shot methods (Liu et al., 2018; Chen et al., 2019; Guo et al., 2019; Xie et al., 2019; Hu et al., 2020) have significantly re- duced the computational costs of NAS. To further improve the search efficiency, most NAS methods search architectures in a low-fidelity set-up (e.g., fewer training epochs, smaller architectures) and retrain the optimal architecture using the full set-up before deployment. This separation of search and evaluation, however, usually suffer from sub-optimal results (Hu et al., 2020). End-to-end NAS
2

---

Under review as a conference paper at ICLR 2021
Update Augment Parameters
Augment Probability Matrix
Super Network
Update Architecture Parameters
Sample Transformation T-T
Sample Architecture A-ar
Hyper-parameter
Training
Optimizer
Loss
Update Network Parameters
Update Hyper- parameters
Validation Loss
Mini-batch
Child Network
An overview of DiffAAttML.. We first sample the DA operations for each sample based on the data transformation parameters т. Then, child network is sampled based on the architecture parameters a, which will be used to process transformed mini-batch. Training loss is calculated to update the neural network parameter by an optimizer with the hyper-parameter" 17. Parameters and or are updated based on the training loss, while is updated with the validation loss to achieve differentiable end-to-end joint optimization automated DA, NAS and HPO.
Figure I:
strategies (Xie et al., 2019; Hu et al., 2020) are thereby developed to return read-to-deploy networks the end of the search. Our work also proposes end-to-end solution.
for deep learning models has also at- tracted lots of research attention. Current HPO methods address various set-ups involving multi- fidelity evaluations, parallel computation, transfer-learning (Mittal et al., 2020) or a mixed variable types (Ru et al., 2020a). Although many HPO strategies have been adapted to form NAS strategies, methods that jointly optimize both architectures and hyper-parameters are rarely seen except the ones discussed below.
Hlyper-parameter Optimization Tuning
Joint Optimization of AutoML Components. Conventional neural architecture search methods perform search under fixed set of training hyper- paremmaaaeeters and apply or search for a separate set of hyper-parameters when retraining the best architecture found. Such search protocol may lead to sub-optimal results (Zela al., 2018; Dong al., 2020) as neglects the influence of training hyper-parameters architecture performance and ignores superior architectures under alternative hyper- hyper-parameter values (Dai et al., 2020). In view of this, several works have been proposed to jointly optimise architecture structure and training hyper-parameters (Dai et al., 2020; Wang al., 2020; Dong et al., 2020). Zela et al. (2018) introduces the use of multi-fidelity Bayesian optimiza- tion search over both the architecture structure and training aaper-parametery Dai al. (2020) trains an accuracy predictor to estimate the network performance based on both the architecture and training hyper-parameters, and then uses evolutionary algorithm perform the search. Both these methods are sample-based and requires relatively large number of architecture and hyper- parameter evaluations finetune predictor or obtain good recommendations. more related work is AutoHAS (Dong et al., 2020), which introduces differentiable approach in conjunction with weight sharing for the co-optimization task. Dong al. (2020) demonstrates empirically that such differ- entiable one-shot approach achieve superior efficiency over sampled-based methods. Our proposed method differs from AutoHAS mainly two aspects: first, AutoHAS updates the entire supernet- work at each optimization step while our method trains only subnetwork. Thus, method is much more memory efficient than AutoHAS. Second, we further extend the co-optimization scope from NAS and training hyper-parametenn the best of our knowledge, we are the first to jointly optimize all three aspects and we demonstrate the advantage of such practice in Sec.
also include data augmentation hyper- paremetrra.eters... To
3

---

Under review as a conference paper at ICLR 2021
3 METHODOLOGY
Consider a dataset D = {(2,, y:) where N is the size of this dataset, and yi is the label of the imput sample I2. We aim to train a neural network f(-), which can achieve the best accuracy on the test dataset Drest, Multiple AutoML components are considered, including automated DA, NAS, and HPO. Let a, n, and 0 represent the data augmentation parameters, the architecture parameters, the hyper-parameters, and the objective neural network parameters, respectively. This problem can be formulated
arg min T.0.7.0
mig L(T, a,,,0, 27,07,00) D)
(1)
S.L. (a(()) S -,,
where [(-) represents the loss function; D denotes the input data; cx(+) and y refer the formula and count of resource constraints C, such as storage cost and computational cost. We adopt different loss functions optimize the corresponding parameters, which will be introduced in the following subsections. Considering the huge search space, it is challenging to achieve the co-optimization of T, a, 77, and 0 within one-stage without parameter retraining. In this work, we propose to use the differentiable method to provide computationally efficient solution. See Fig. as an illustration.
3.1 OPTIMIZATION OF DATA AUGMENTATION PARAMETERS
For every mini-batch of training data D ((.)))<)) )() with batch size n²", we conduct data augmentation to increase the diversity of the training data. We consider K data augmentation op- crations, and each training sample is augmented by transformation consisting of two successive operations (Cubuk et al., 2018; Lim et al., 2019). Each operation is associated with magnitude that uniformly sampled from [o, 10). The data augmentation parameter 7 represents a probability dis- tribution over the augmentation transformations. For t-th iteration, sample n²" transformations according to T' with Humbel-Softmail reparameterization (Maddison et al., 2016) and to generate the corresponding augmented samples in the batch. Given a sampled architecture (see details in Sec. 3.2), the loss function for each augmented sample is denoted by Ld (j ( T'; (rx)))) where Tk represents the selected transformation. In order to relax be differentiable, we regard p& Px("), the probability of sampling the transformation TE. as an importance weight for the loss function hd (f(a", "a: (xx)))) The objective of data augmentation is to minimize the following loss func- tion:
nr
CDA(T))= =- pp - 0¹; Tx(x))),,
k=1
With this loss function, DiffAutoML. intends to increase the sampling probability of those trans- formations that can generate samples with high training losses. By sampling such transformations, DiffAutoML can pay more attention to relatively "hard" samples and increase model robustness against difficult samples. Instead of training a controller to generate adversarial augmentation poli- cies via reinforcement learning (Zhang et al., 2020), we search for the probability distribution of augmentation transformations directly via gradient- based optimization. In this way, the optimiza- tion of data augmentation is very efficient and hardly increases the computing cost.
3.2 OPTIMIZATION OF ARCHITECTURE PARAMETERS
With the augmented data in Sec. 3.1, we achieve optimization of the architecture parameter a through end-to-end NAS, motivated by SNAS (Xie et al., 2019) and DSNAS (Hu et al., 2020). Fol- lowing Liu et al. (2018), we denote the neural architecture search space as a single directed acyclic graph (DAG). Node m, refers to the feature map in DAG and edge (m,, refers to the informa- tion flow between m, and my. In addition, O,, and a, respectively refer to possible operations and the architecture parameters between m, and mj. and a consists of all possible ax, in the DAG. During the forward process, an ont-hot random variable Zi.j is sampled from a distribution defined by the architecture parameters Oi,j. Then, the sampled operation Öz, can be calculated by multiply Ziy to O..j- Thus, the intermediate node my can be expressed as:
Eo <<)
Σ 2T Or... i<1
(3)
4

---

Under review as a conference paper at ICLR 2021
Instead of using validation loss, the architecture parameters is optimized using the training loss. In that way, architecture parameter a and parameter can be trained with the same loss function. SNAS relaxes Z to a continuous random variable Z with the Gumble-Softman Thus gradient of a under training loss can be formulated as:
OC da
2C Jmj O (m)) - -k) Zi,) 7 Aa
(4)
where ot is the k-th element in oz.j Z is the k-th element in Zıj; À is the temperature in the Gumbel- Gumbel-Softmax. Softmax. As À is the denominator in the gradient, it cannot be zero, which prevents SNAS from realizing the architecture search and the network parameter tuning in an end-to-end manner. In order realize the one-stage optimization, we adopt the following Eqn. gradient:
approximate its
t
lim Eg A-
Zz=p(2)
Ezzp(2)
Vot, log log (eap(Zzn. Zo.s)
(5)
Dat
0
where Zi is the s-th element in the one-hot vector Zi where s-th element is equal to one.
3.3 OPTIMIZATION
HYPER-PARAMEIE
As shown in Fig. 1, given the batch of augmented training data {(Tb(T*)) ))) and the sam- pled child network, we need to optimize the differentiable hyper-parameters 77, such as learn- ing rate and L2 regularization. At the training stage, we alternatively update O and n. In t- th iteration, we can update O based on the gradient of the training loss Ld (f 0¹ (T))
Ek= (" 0 (";Te (Ix))) which can be written
g++1 OP(O', n/', VoL VoL'I(((a' 0r; B')))
(6)
where OP(.) is the optimizer. To update the hyper-parameters
we regard Or+1 as function of
and compute the validation loss Loal (f(a", 0i (n'); Bual)) with network parameters 0²+1 (n" on
mini-batch of validation data Bval Then, ny is updated with v,, (f(a", (n'); Bual) ( by
gradient descent:
17 -+ = vV (J(a", p0*""(n";; B"a)))
where / is learning rate. Even 0f can also be deployed to 0--1 whose calculation also involves 1), take an approximation method in this equation and regard 0f here variable independent of n..
3.4 DIFFAUTOML
Based above analysis of each AutoML module, DiffAutoML_ realizes end-to-end co-optimization of automated data augmentation parameters т, architecture parameters a. and hyper-parameters 7). The DiffAutoML algorithm is summarized in Algorithm 1. One-level optimization is applied to a and T as in Line 5 and Line 6, while bi-level optimization is applied to v as in Line 8.
4 EXPERIMENTS
Experiments are conducted to verify the performance of the proposed DiffAutoML. In Sec. 4.2, compare the performance of DiffAutoML with end-to-end NAS and multi-stage NAS in terms of final accuracy and the computational efficiency to show the advantages of end-to-end joint op- timization. In Sec. 4.3, we conduct ablation studies for DiffAutoML. First, given neural network architectures, we verify the effectiveness of the automated DA and HPO modules provided by Dif- fAutoMI.. Second, we investigate the mutual influence among different AutoMI. components.
4.1 EXPERIMENT SETTING
Dataset All experiments conducted on the ImageNet dataset (Russakovsky et al., 2015) for the classification task. This dataset consists of around 1.28 x 10 training images and 5 x 10 validation images.
5

---

Under review as a conference paper at ICLR 2021
Algorithm DiffAutoMM
Initialization: Architecture Parameters a, Data Transformation Parameters т, Hyper-parameters and Network Parameters 0 Input: Training Set Dtr, Validation Set D Optimizer, Parameters т, a, 77. 0, and the iteration
number T
Return: a.
1: while < T do
2: Separately sample mini-batch 8 and mini-batch Bual from Dir and Deal,
3: For each sample in B" sample a transformation according to r²;
4:
Based on a', sample child network from super network;
5:
Compute the weighted loss function as Eqn. (2) and update r++1 accordingly;
6:
Use normal loss function to update a (+1 with Equ. (5);
7:
Calculate 0 with Eqn. (6);
8:
Compute the loss function with 0 on D and update r7 with Eqn. (7);
120 68000n 100ton ACUuOS//N)
50-
10
searching
muning
00-
(a) Epoch
(b) Time
Figure 2: Comparison of validation accuracy of DSNAS and DiffAutoML. (a) the training epochs and (b) the run time. NAS optimization DiffAutoMI. can be divided into searching stage where is optimized with others, and tuning stage where is fixed while other modules still updated.
Search Space Automated DA. Following Ho et al. (2019), we consider 14 different operations for data augmentation, including AutoContrast, Equalize, Rotate, Posterize, Solarize, Color, Contrast, Brightness, Sharpness, Shear X/Y, Translate X/Y, and Identity. The magnitude of each operation randomly sampled from the uniform distribution. NAS. Following Hu al. (2020), we consider four candidates for each choice block in the parent network, i.c., choice 1, choice 2, choice .3, and choice.4. 4. These candidates differ in kernel size and the number of depthwise convolutions (see Appendix B for the detailed network architectures). The resulting search space consists of 4 single path models. HPO. For illustration purpose, we consider the L2 regularization (i.e., weight decay) in the experiments.
4.2 JOINT OPTIMIZATION OF AUTOML COMPONENTS
We first compare our DiffAutoML against several types of baseline methods: memory-efficient manual designed networks such as MobileNet V2 (1.0x) (Sandler al., 2018) and ShuffleNet V2 (1.5) (Ma et al., 2018), 8), conventional two-stage NAS methods like Proxyless-R (mobile) (Cai et al., 2019) and Single Path One- (Guo et al., 2019), which optimize the architectures during search stage and then retrain the best architecture before deployment, and finally the recently proposed end-to-end NAS approach, DSNAS (Hu et al., 2020). Detailed experimental settings are provided in Appendix A.
The results of this set of experiments are summarized in Table 1. First, is evident that all NAS methods including our DiffAutoML clearly outperform the manually designed networks both in terms of accuracy performance as well as FLOPS required; this verifies the effectiveness of per- forming architecture search. Second, if take into account the total time taken for obtaining

---

Under review as a conference paper at ICLR 2021
ready-to-deple network ', the end-to-end NAS approaches (DSNAS and DiffAutoMM) consumes less time than two-stage NAS approaches (Proxyless-R and Single Path One-Shot) to achieve better or comparable performance. Specifically, DiffAutoML exceeds Proxyless-R by 0.9% in terms of accuracy without retraining but takes 42% less time. Note for end-to-end NAS methods, the ac- euracy performance of the network found with and without training almost identical. Finally, the performance gain of our DiffAutoML over DSNAS shows the advantage of doing joint opti- mization of automated DA, NAS and HPO in comparison with doing NAS alone. In addition, we compare the validation accuracy of DiffAutoML. and DSNAS over epoch numbers (Fig. 2(a)) and training time (Fig. 2(b)). As can be seen, DiffAutoMML achieves higher validation accuracy with much fewer epochs. Although optimizing more AutoML. components requires slightly longer time for each epoch, DiffAutoML. results in less total searching time due to the fast convergence com- pared to DSNAS (see Fig. 2(b)). This indicates that appropriate automated DA and HPO could help to accelcrate and stabilize the NAS process, and also shows the efficiency of our differentiable joint optimization framework even with much larger search dimension.
Top-1 Accuracy (%) and Top-5 Accuracy (%) of architecture search algorithms in the ImageNet dataset. Here, indicates our application, $ represents the accuracy of the search set, and $ refers to a conversion from V100 GPU hours with ratio of 1 1.5.
Table 1:
Model
FLOPS
Search
Retrain
Time (GPU hour)
Top-1
Top-5 Acc(%)
Top-1 Acc(%)
Top-5 Acc(%)
Search
Retrain
Ac)(%)
MohileNet V2 (1.0x)
564M
Manual Manual
72.0
91.0
Manual Manual
ShuffleNet V2 (1.5x)
564M
72.6
90.6
Proxyless-R (mobile)
320M
62.5*
84.8*
74.6
92.2
3008
2 384
Single Path One-Shot
319M
68.71
74.3
250
384
DSNAS
324M
74.4 + 0.2
91.5
74.3 + 0.3
91.9
420
DiffAutoML.
318M
75.5
92.7
75.5
92.5
394
4.3
ABLATION STUDIES AND DISCUSSION
4.3.1 DIFFAUTOML GIVEN NETWORK ARCHITECTURES
We conduct this set of experiments to demonstrate the effectiveness of DA and HPO modules in DiffAutoMI., as well the advantages of conducting optimization of AutoML modules jointly over sequentially.
We first apply the joint optimization of automated DA and HPO (see Sec. 3.1 and 3.3) in the retrain- ing phases of the baseline NAS algorithms including DARTS and DSNAS. This is to simulate the common practice of optimizing different components of AutoML separately in sequence. The results in the top two rows of Table 2 show that performance gain can be obtained by doing such additional optimization on HP and DA policies, justifying the need of applying multiple AutoMI. components for a single task. It also shows that the DA and HPO optimization modules in our method remain performant even when being used in separation from our NAS approach. Finally, compared with the performance reported in Table 1, the superior performance of doing co-optimization of all the components (i.e., automated DA, NAS and HPO) over doing optimization in sequence shows the clear advantage of joint optimization.
We also apply joint optimization of automated DA and HPO provided by DiffAutoML to manual design networks such as ResNet-50. There is one thing worth mentioning that in this experiment, the HPO optimizes both learning rate and 1.2 regularization. The results are compared with recent automated DA algorithms including Fast Autoaugment (FAA) (Lim et al., 2019), Population-Base Augment (PBA) (Ho et al., 2019) and Adversarial Autoaugment (AdvAA), as summarized in the last seven rows of Table As can be seen, without delicate hyper-parameter tuning with burdensome human involvement, our DiffAutoML shows very competitive results, outperforming the current baselines in most cases on a variety of image classification tasks.
'The sum of search and retrain time for two-stage NSA methods and the search time along for the end-to-end NAS method.
7

---

Under review as a conference paper at ICLR 2021
Table Top-1 Top-5 Accuracy (%) of different DA algorithms and the NAS algorithms on Ima- geNet. Top-1 Accuracy (%) of different DA algorithms on CIFAR-10 and CIFAR-100.
Dataset
Model
Baseline
FAA
PBA
AdvAA
DifAautML
ImageNet
DARTS (DA+HPO)
73.1/91.0
73.8/91.5
DSNAS (DA+HPO)
74.3/91.9
75.0/92.5
ImageNet
ResNet-50
76.3/93.1
77.6/93.7
79.4/94.5
79.0/94.3
ResNet-2000 200
78.5/94.2
80.6/95. /9/3
81.3/95
81.5/95.5
CIFAR-I0
Wide-ResNet-28-11 10
96.1
97.3
97.4
98.1
98.0 0.1
Wede-Resseet-4-22
94.7
96.4
96.8 = 0.0
PyramidYet4ShukeDrpp
97.3
98.3
98.5
98.6
98.6 = 0.1
CIFAR-100
Wide RelleRReeNet22--- 10
81.2
82.9
83.3
84.5
84 0.2
Wide-ResNer Nele-ReeNN
74.0
79.3
80.6 + 0.0
Choice_1
Choice_2
Choice_3
Choice_4
DSNAS
(b) Di
(c) Di (NAS 14 DA w/o HPO)
(d) DiffAutoML. (NAS + DA w/o HPO)
Figure 3: Searched architectures with different AutoML components. The network architecture is divided into four components shown in the above figure with the help of the dotted line. The first block of each component has the stride number of 2 and the rest blocks have the stride number of 1. Detailed architectures of different choices are provided in Appendix B.
4.3.2 MUTUAL INFLUENCE AMONG AUTOML COMPONENTS
In this experiment, investigate the mutual inference among different AutoML components by comparing the searched architectures and the final accuracy. We first pay attention to the resulting architectures of DSNAS and Diff'AutML. in Sec. 4.2. For comparison purpose, different DA strategies in NAS without HPO when using DiffAutoML.. Specifically, we separately conduct DiffAutoML with all DA operations introduced in Sec. 4.1 and four DA operations only. The four DA operations are Shear X/Y and Translate X/Y.
also consider
The resulting architectures are provided in Fig. 3, and the corresponding final accuracy is provided in Appendix C. As can be seen, the resulting optimal architectures are quite different in terms of choices of basic building blocks. This indicates mutual influences among different AutoML components, thus justifying the need for co-optimization of AutoML components instead of optimizing them separately.
CONCLUSION
Previous AutoMI. algorithms usually focus on one or two components only, which ignore the cou- pling relationships among different AutoMI. components. DiffAutoMI. takes the co-optimization of the data augmentation, the architecture search, and the hyper-parameters into consideration, adapts the differentiable method to well treat this large search space, and realizes the end-to-end co-optimization. With less computation complexity, DiffAutoML achieves better results than alter- native one-stage NAS and comparable results with two-stage NAS.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI Conference on Artificial Intelligence 2018.
Han Cai, Ligeng Zhu, Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations (ICLR), 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging depth gap between search and evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane. Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 0800000050000000000 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quac V Le. Randaugment: practical automated data aug- mentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian Hc, Zhen Wei, Kan Chen, Yuandong Tian, Mathew Yu, Peter Vajda, al. FBNetV3: Joint architecture-rest recippe search using neural acquisition function. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 170iv:1708.n 04552, 2017.
Xuanyi Dong, Mingxing Tan, Adams Wei Yu, Daiyi Peng, Bogdan Gabrys, and Quoc V Le. AutoHAS: Differ- entiable hyper-parament and architecture search. arXiv preprint urXiv:2006.0366 2020.
Thomas Elsken, Hendrik Metzen, Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference on Learning Representations (ICLR), 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904 00420, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun. Deep residual learning for image recognition. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Daniel Ho, Eric Liang, Chen, Ion Stoica, and Pieter Abheel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning (ICML), 2019.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: Direct neural architecture search without parameter retraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural ar- chitecture search with bayesian optimisation and optimal transport. Advances in Neural Information Processing Systems (NeurIPS), 2018
Sungbin Lim, Ildoo Kim, Taesup Kim. Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Chen Lin, Minghao Guo. Chuming Li, Xin Yuan, Wei Wu, Junjie Yan, Dahua Lin, and Wanli Ouyang. Online hyper-parameter learning for aumennggmennition strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition (CVPR), 2019.
Hanxiao Liu. Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv preprint
arXiv:1806.000 100.005500 2018.
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. NSGA-Net: Neural architecture search using multi-ohjective genetic algorithm. In The Genetic and Evolutionary Computation Conference (GECCO), 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient architecture design. In Proceedings of the European Conference on Computer vision (ECCV), 2018.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 161 1,00 2016.

---

Under review as a conference paper at ICLR 2021
Gaurav Mittal, Chang Liu, Nikolaos Karianakis, Victor Fragoso, Mei Chen, and Yun Fu. HyperSTAR: Task- aware hyperparameters for deep networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Hieu Pham, Melody Guan. Barret Zoph. Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. International Conference on Machine Learning (ICML), 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution image classifier architecture search. arXiv:/802.01548. 2018.
Binxin Ru, Ahsan Alvi, Nguyen, Michael Osborne, Stephen Roberts. Bayesian optimisation over multiple continuous and categorical inputs. International Conference on Machine Learning (ICML), 2020a.
Binxin Ru, Pedro Esperanca, and Fabio Carlucci. Neural architecture generator optimization. arXiv preprint arXiv:2004.0139/ 2020b.
Olga Russakovsky, Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An- drej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large seale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mobilenet Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Tianzee Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. APQ: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement learning.
Machine Learning, 8(3-4):299 256, 1992.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. Interna- tional Conference Learning Representations (ICLR), 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint 1rX/v:1807.06606, 2018.
Hongyi Zhang. Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arxiv: 171000912, 2017.
Xinyu Zhang, Qiang Wang. Jian Zhang, and Zhao Zhong. Adversarial autoaugment. International Conference on Learning Representations (ICLR), 2020.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR), 2017.
Barret Zoph. Vijay Vasudevan, Jonathon Shlens, Quoe Le. Learning transferable architectures for scal- able image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018
10

---

Under review as a conference paper at ICLR 2021
Delon
Plaaent
Duadents
ERE Care
Tel
Do Dilon
Cenos
Stride Figure 4: Choice blocks in search space. From left to right are Choice_1, Choice_2, Choice_3, and Choice_4.
(a) Stride
A DETAILED EXPERIMENTAL SETTINGS
This experiments is optimized by the Adam optimizer with the learning rate x 10-3. The data transformation parameter T and the architecture parameter 7 optimized by the SGD optimizer separately with 10-6 and 1 x 10-6. Their L2 weight decay is set to be 5 x 10 and their
&*NVIDIA V100 under PyTorch-133.0 1.3.0 and python3.6. Hyper-parament
the learning rate momentum is set be 0.9. SGD optimizer is used optimize O. Cosine learning scheduler is used to gradually change the learning rate with the initial learning rate 0.5 and the moment of SGD optimizer set to 0.9.
As to the optimization of T and 7) with given architectures, we use the same optimizer and hyper- parameter as in Sec. 4.2. For the retraining of the architecture found by DSNAS, the SGD optimizer is used with the basic learning rate 0.5 and the L2 weight decay 4 x 10-5, The learning rate is also gradually modified by the cosine learning scheduler. The retraining of DARTS shares almost the same setting as that DSNAS, while its initial learning rate is set to 0.1 with the weight decay 3 x 10-5
B DETAILS ABOUT THE ARCHITECTURES
Following Hu et (2020), we consider four candidates for each choice block in the parent network, ie., Choice 1, Choice 2, Choice and Choice 4. These candidates differ in kernel size and the number of depthwise convolutions. Detailed network architectures are provided in Fig. 4.
C MUTUAL INFLUENCE AMONG AUTOML COMPONENTS
In this experiment, investigate the mutual inference among different AutoML components by comparing the searched architectures and the final accuracy. For comparison purpose, we consider different DA strategies in NAS without HPO when using DiffAutoML. Specifically, DiffAutoML with all DA operations introduced in Sec. 4.1 and four DA operations only, separately. The four DA operations are Shear X/Y and Translate X/Y. The resulting accuracy in ImageNet is reported in Table 3.
conduct
Table 3: The Top-1 Accuracy (%) and Top-5 Accuracy (%) the ImageNet dataset achieved by DiffAutoML trained under different DA settings.
Top-1 Acc(%)
Top-5 Acc(%)
Method
FLOPS
Co-optinization
Co-optimization
DiflAutoML (NAS + DA) DiffAutoML (NAS 14 DA)
74.8
92.2
323M
75.1
92.7
318M
11

---

Under review as a conference paper at ICLR 2021
D RELATED WORK IN DETAILS
D. NEURAL ARCHITECTURE SEARCH
NAS is important subfield of AutoML that automates the design of neural networks for various tasks. It has attracted growing attention over the recent years and discovered architectures with better performance over those designed by human experts (Zoph & Le, 2017; Cai et al., 2018; Liu et al., 2018; Kandasamy et al., 2018; Pham et al., 2018; Real et al., 2018; Zoph et al., 2018; Ru et al., 2020b). The rich collection of NAS literature can be divided into two categories: the query-based methods and the gradient-based ones. The former includes powerful optimization strategies such as reinforcement learning (Zoph & Le, 2017; Pham et al., 2018), Bayesian optimization (Kandasamy et al., 2018: Ru et al., 2020b) and evolutionary algorithms (Elsken et al., 2019; Lu et al., 2019). The latter enables the use of gradients in updating both architecture parameters and network weights, and significantly reduces the computation costs of NAS via weight sharing. Since the seminal work DARTS (Liu et al., 2018), many follow-up works (Chen et al., 2019; Xie et al., 2019; Hu et al., 2020) have been proposed to improve the performance and efficiency of gradient-based approaches. To achieve cost efficiency, most NAS methods search architectures in a cow-fidelity set-up (e. fe.g.rewer training epochs, smaller architectures) and retrain the optimal architecture using the full set-up be- fore deployment. This separation of search and evaluation is sub-optimal (I cannot find the exact drawbacks of this except something like poor performance ranking correlation between the architec- tures at the end of searched and after retraining) (Hu et al., 2020) and motivates the development of end-to-end NAS strategies (Xie et al., 2019; Hu et al., 2020) which return read-to-deploy networks at the end of the search. Our work also proposes an end-to-end solution.
D.2 DATA AUGMENTATION
Data augmentation is one of the commonly used techniques in deep learning increase the diver- sity of data and thus to improve the generalization ability of deep learning models. Elaborately de- signed augmentation methods have effectively improved the performance deep models (De Vries & Taylor, 2017; Zhang et al., 2017; Yun et al., 2019). However, the effectiveness of these manu- ally designed ones varies among datasets. As automated machine learning has achieved impressive success in quite a lot scenarios, learning data augmentation policies directly from target dataset has become a trend. AutoAugment (Cubuk et al., 2018) and Adversarial AutoAAgment (Zhang et al., 2020) adopt reinforcement learning train controller to generate policies, while OHL- Auto-Aug (Lin et al., 2019) formulates augmentation policy as a probability distribution and adopts REINFORCE (Williams, 1992) to optimize the distribution parameters along with network train- ing. PBA (Ho et al., 2019) and FAA (Lim et al., 2019) use population-based training method and Bayesian optimization respectively to reduce the computing cost of learning policies. Cubuk et al. (2020) argue that the search space of policies used by these works can be reduced greatly and simple grid search can achieve competitive performance. They also point out that the optimal data augmen- tation policy depends on the model size, which indicates that fixing an augmentation policy when searching for neural architectures may lead to sub-optimal solutions.
D.3 IYPER-PARAMETER OPTIMIZATION
The performance of neural networks heavily depends on the appropriate choices of hyper- parameters. Various black-box optimization approaches have been developed to address hyper- parameter tuning tasks involving multiple tasks (Mittal et al., 2020) or mixed variable types (Ru et al., 2020a). Meanwhile, techniques like multi-fidelity evaluations, parallel computation and transfer learning are also employed to further enhance the query efficiency of the hyper-paramets optimization. Although many hyper-pane parameter optimization strategies have been adapted to form query- basee NAS strategies, methods that jointly optimize both architectures and parameerr are rarely seen except the ones discussed below.
D.4 CO-OPTIMIZATION
Conventional neural architecture search methods perform search under fixed set of training hyper- parameters and apply or search for a separate set of chitecture found. Such search protocol leads to sub-optimal results (Zela et al., 2018; Dong et al.,
when retraining the best ar-
12

---

Under review as a conference paper at ICLR 2021
2020) it overlooks the influence of training syper-parametent architecture performance and ignores superior architectures under alternative byper-paramedt values (Dai et al., 2020). In view of this, several works have been proposed jointly optimize architecture structure and training hyper- parameters (Dai et al., 2020; Wang et al., 2020; Dong et al., 2020). Zela et al. (2018) introduces the use of multi-fidelity Bayesian optimization Dai et al. (2020) trains an accuracy predictor to estimate the network performance given its architecture and training hyper-parameters and then uses the predictor with evolutionary algorithm to perform the search. Both these methods are sample- cumple-based and requires relatively large number of aechiiteturr--yypppaaaeeeee good recommendations. A more related work is AutoHAS (Dong et al., 2020), which introduces a differentiable approach conjunction with weight sharing for the co-optimization task. Dong et al. (2020) demonstrates empirically that such differentiable one-shot approach achieve superior efficiency over sampled-bases methods. Our proposed method differs from AutoHAS in mainly two aspects: first, AutoHAS updates the entire supemetwork at each optimization step while method trains only a subnetwork. Thus, our method is much more memory efficient than AutoHAS. Second, further extend the co-optimization scope from NAS and training hyper-pann paraee-arrmeters to also include data augmentation jointly optimise all three aspects and demonstrate the advantage of such practice in Sec. 4.
search over both the architecture structure and
training
evaluations to finetune predictor or obtain
To the best of our knowledge, we are the first to
13


Reviews:
[{"review_id": "oSiH4b6T1ln", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Some of the choices that have to be made when training a neural net based image model are: type of data augmentation, architecture of the neural network, and other hyperparameters such as regularization and optimization hyperparameters (e.g. learning rate). Optimizing all of these is a challenging problem, NAS deals with architecture but ignores the others. More general hyperparameter optimization techniques such as Bayesian Optimization struggle with the dimensionality of the architecture parameters. And optimizing them independently might lead to local minima, and/or be slow.\n\nOne previous work (AutoHAS) already combined NAS with the ability to add differentiable hyperparameter. In this work data augmentation is incorporated as well and tuned jointly with the rest of the architecture and the other hyperparameters.\n\n**Their main contributions** of this paper are: 1. The authors describe a NAS+differentialble hyperparameter tuning technique that also include data augmentation, thus extending the setting somewhat compared to previous results 2. They provide experimental results on ImageNet\n\nThe paper is reasonably clear, and while some of the ideas are interesting (such as how to incorporate data augmentation as a differentiable NAS hyperparameter) I don't think this work should be accepted mainly because:\n* They describe only incremental improvements. Thus no major contribution from the theory/new tecniques part of the paper.\n* Limited experimental results, definitely not strong enough to compensate for the previous point. There is no comparison to AutoHAS, which appears to be the closest method, according to the paper itself. And most results are on ImageNet only. If the paper wants to provide a technique for automatically generating vision pipelines, stronger arguments and experimental results should be brought forward for why this will work on new problems and not just on ImageNet (for witch we can just take one of the existing pretrained models).\n\n\n##### Questions/comments: \n\nIn the results, it seems that the competing method use no data augmentation at all, is that correct? If so it would be more fair if the competing method use untuned but reasonable data augmentation. \n\nDoes baseline in table 2 contain any data augmentation? It's not really clear?\n\nThe results are the average over how many repetitions? It would be nice to know the number of repetitions and standard dev/error, or if a single one at least have the source code available for better reproducibility.\n\nThe validation set is used for selecting hyperparameters, thus should not be used for comparing methods, was a separate test set used for accuracy? It would be good to describe this more clearly in the paper.\n\nI would remove the epoch based plot, what we care about at the end is time.\n\n\n##### Typos:\n\nSection 1\n\"We first jointly optimize three different AutoML components, including\" -> you might want to remove \"including\" if you list all three\nSection 3.1\nFor t-th iteration -> for the t-th iteration", "extra_scores": {}, "extra_text": {}}, {"review_id": "m-aDBbZiIxU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper focuses on achieving automated \"from data to model\" including different components in modeling, namely data augmentation, Neural Architecture Search, Hyper Parameter Optimization. The proposed approach first use data augmentation to select the data argumentation transformation. It tries to select examples which incurs higher training loss for the model to address hard examples. Then use the DAG for neural architecture search. Given the data and architecture, it then alternatively update the model parameter and hyper parameter. The overall proposed framework is end-to-end. Experiment on ImageNet shows slight performance improvement over existing approaches. The authors also conduct ablation study to show the effectiveness of jointly modeling the three components (data augmentation, neural architecture search, hyper-parameter optimization).\n\nStrengths:\n1) This work considers three important components in modeling process including data argumentation, HPO, and NAS. The different components may interact with each other to impact the performance.\n2) This paper is clearly written and easy to follow.\n3) The experiment shows performance improvement with less time needed. \n\nWeakness\n1) The experiment is limited with only 1 task and data. Is it possible to show the results on more than 1 dataset? Otherwise, the result might still be dataset specific. However, the proposed approach should not be \n\nQuestions\nWill the proposed approach generalize beyond computer vision task?\nWhy are the data augmentation and neural architecture search grouped together? I was wondering what will happen if you group neural architecture search and hyper parameter optimization first?\nHow should I interpret figure 3? How could you demonstrate that the selected architecture by the proposed approach is better or make sense?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DJuwcDufCv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper shows how the complex autoML pipeline for neural networks can be trained in an end-to-end manner by combining existing methods. By using backpropagatable discrete sampling methods (Gumbel softmax), input transformed by data augmentation is seamlessly embedded in full backpropagation flow. And a differentiable architecture search algorithm is used, which also incorporates architecture search in full backpropagation flow. On top of this differentiable procedure, an alternating optimization is introduced to train network parameters and hyperparameters.\n\n\n**Strengths**\n1. The paper shows that end-to-end automl is possible in a fully differential way which allows better models with fewer resources.\n2. Even though, intuitively, this joint optimization should perform better than non-joint ones, making such training stable does not seem trivial. The authors appear to successfully have taken care of it.\n\n\n**Weaknesses**\n1. Even though 2 hyperparameters are considered in the ablation study, its capability and stability with a moderate number of hyperparameters are not well-addressed (in the main experiment only 1 hyperparameter is considered). And the use of a validation set only in hyperparameter update step make me question that this hyperparameter optimization component may cause some issues such as training instability. Can the authors share some thought, experience, and intuition on this?\n2. It is questionable whether the comparison in Figure 2 is fair since DiffAutoML utilizes validation data in its hyperparameter optimization step whereas DSNAS seems to not use validation data.\n3. Similar to above, the numbers for DSNAS in Table 1 seem to be taken from DSNAS paper (Table 3) which are numbers from the validation set.\n\n\n**Recommendation**\nThe paper tackles interesting and practical problems and shows the proposed method outperforms baselines. My main concern is whether DiffAutoML uses validation data that is not used by other baselines, which could be a reason for better performance. The concern on considering on a small number of hyperparameters is a minor one if some explanation can be added. As long as the concern on the validation data access is resolved then I will be willing to increase my score favoring acceptance. However, in its current form, I cannot stand for it.\n\n\n**Questions**\n- The numbers reported in Table 1 and Table 2 are test performances?\n\n\n**Additional feedback** (Irrelevant to the decision assessment)\n- Since the most of NAS component in the full pipeline is from DSNAS, it would be better for readers if this is detailed in Related Works or Appendix.\n- At 4th line in Algorithm 1, using the word 'parent' network is more consistent?\n- In the last line in Abstract, en-to-end -> end-to-end\n- At 3 lines above eq. 3. ont-hot -> one-hot\n- In the last paragraph in Introduction, in the 4th line from the last, Differentiable -> differentiable", "extra_scores": {}, "extra_text": {}}, {"review_id": "6sG_-GNBQK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary and contributions: \nThis paper proposes a joint differential search method to optimize data augmentation, discrete architecture choices, and hyperparameters. \n\n\nStrengths:\n1). This paper proposes an automated method to search for data augmentation, hyperparameters, and architectures using gradient descent, which is simple and easy to integrate. \n2). It achieves co-optimization on different components in one-stage.\n3). A modest improvement upon SOTA mobile models.\n\nWeaknesses:\nIn the optimization, the proposed method naively uses validation data to optimize the hyperparameters, and use training data to optimize architecture weights, data augmentation weights, and network weights. Wouldn't the architecture choices and data augmentation overfit to the training data in such case? Note that there exists a discrepancy in the optimization direction between NAS and  DAS, where NAS aims to minimize the training loss (e.g. cross entropy) while DAS tends to increase the training loss. Therefore, trying to optimize NAS and DAS under a single objective is intractable. The objectives for the jointly optimization needs to be carefully designed so that the optimization for NAS and DAS can contribute collaboratively to the final testing performance.\n\n\nCorrectness: Are the claims and methods correct? Is the empirical methodology correct? \nI am uncertain the method appears to be correct.\n\nClarity: Is the paper well written? \nYes, although a few aspects could be improved (see feedback).\n\nRelation to prior work: Is it clearly discussed how this work differs from previous contributions? \nYes.\n\nReproducibility: Are there enough details to reproduce the major results of this work? \nNo code is provided.\n\nAdditional feedback, comments, suggestions for improvement, and questions for the authors: \nClearly describe mathematically how the proposed method contribute collaboratively to the final testing performance including its generalizability. The mutual dependency between NAS and DAS arises from that NAS is based on the training data modified by the augmentation policy generated by DAS and the DAS is based on the network generated by NAS. Additionally, provide ablation studies on changing one single component while fixing others. \n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was discussed by the reviewers that acknowledged the rebuttal and the authors’ responses. In particular, they appreciated the fact that some of their concerns were alleviated (e.g., going beyond the single ImageNet evaluation).
 
More generally, while all the reviewers thought that the problem tackled by the paper was of clear interest (i.e., full end-to-end auto-ML encompassing DA, NAS and HPO), they still expressed concerns (even after the rebuttal), in particular:

* _Clarity of the methodology_: None of the reviewers could clearly and fully understand the mathematical formulation of the joint optimization, leading to a series of questions regarding the confusing usage of the training/validation set in the experimental setup. This unfortunately made the assessment of (some aspects of) the paper speculative for the reviewers.
* _Comparison with AutoHAS_: AutoHAS and DiffAutoML are obviously related methods. Even if AutoHAS has weaknesses compared to the proposed approach DiffAutoML, e.g., discretization of the continuous hyperparameters and no tuning of DA, it is still meaningful to carry out an actual comparison (possibly normalized by the different costs at play since the authors have highlighted the different memory overheads). Though the listed weaknesses of AutoHAS _should_ play in favor of DiffAutoML,  a proper experimental comparison would better support that claim.

Given those remaining concerns and the overall mixed scores, the paper is recommended for rejection. The detailed comments of the reviewers provide an actionable list of items to improve the paper for a future resubmission.

Decision: Reject