NASOA: Towards Faster Task-oriented Online Fine-tuning
Authors: Hang Xu
Ning Kang
Gengwei Zhang
Xiaodan Liang
Zhenguo Li
Venue:  (2021)
Keywords: Fine-tuning
AutoML
NAS

Abstract: Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks. The common practice of fine-tuning is to adopt a default hyperparameter setting with a fixed pre-trained model, while both of them are not optimized for specific tasks and time constraints. Moreover, in cloud computing or GPU clusters where the tasks arrive sequentially in a stream, faster online fine-tuning is a more desired and realistic strategy for saving money, energy consumption, and CO2 emission. In this paper, we propose a joint Neural Architecture Search and Online Adaption framework named NASOA towards a faster task-oriented fine-tuning upon the request of users. Specifically, NASOA first adopts an offline NAS to identify a group of training-efficient networks to form a pretrained model zoo. We propose a novel joint block and macro-level search space to enable a flexible and efficient search. Then, by estimating fine-tuning performance via an adaptive model by accumulating experience from the past tasks, an online schedule generator is proposed to pick up the most suitable model and generate a personalized training regime with respect to each desired task in a one-shot fashion. The resulting model zoo is more training efficient than SOTA NAS models, e.g. 6x faster than RegNetY-16GF, and 1.7x faster than EfficientNetB3. Experiments on multiple datasets also show that NASOA achieves much better fine-tuning results, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various time constraints and tasks; 40x faster compared to the BOHB method.

Under review as a conference paper at ICLR 2021
NASOA: TOWARDS FASTER TASK-ORIENTED ONLINE TUNNITUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Fine-tuning from pre-trained ImageNet models has been simple, effective, and popular approach for various computer vision tasks. The common practice of fine- tuning is to adopt a default hyperparameter setting with fixed pre-trained model, while both of them are not optimized for specific tasks and time constraints. More- over, cloud computing or GPU clusters where the tasks arrive sequentially a stream, faster online fine-tuning is more desired and realistic strategy for sav- ing money, energy consumption, and CO2 emission. In this paper, we propose a joint Neural Architecture Search and Online Adaption framework named NASOA towards a faster task-oriented fine-tuning upon the request of users. Specifically, NASOA first adopts offline NAS to identify group of training-efficient net- works to form a pretrained model zoo. We propose a novel joint block and macro level search space enable flexible and efficient search. Then, by estimat- ing fine-tuning performance via an adaptive model by accumulating experience from the past tasks, an online schedule generator is proposed to pick up the most suitable model and generate personalized training regime with respect to each desired task a one-shot fashion. The resulting model zoc is more training effi- cient than SOTA NAS models, e.g. 6x faster than RegNetY-16GHi and 1.7x faster than EfficientNetB3. Experiments on multiple datasets also show that NASOA achieves much better fine-tuning results, i.c. improving around 2.1% accuracy than the best performance in RegNet series under various time constraints and tasks; 40x faster compared to the BOHB method.
INTRODUCTION
Fine-tuning using pre-trained models becomes the de-facto standard in the field of computer vision because of its impressive results on various downstream tasks such as fine-grained image classifi- cation (Nilsback & Zisserman 2008 Welinder et al. 2010), object detection (He et al. 2019 Jiang al. 2018| Xu u et al.] 20199) and segmentation (Chen et al. 2017; Liu al.. (019). Kornblith et al. (2019 He et al. (2019) verified that fine-tuning pre-trained networks outperform training from scratch. It can further help to avoid over-fitting (Cui et al. 2018) as well as reduce training time significantly (He et al. 2019). Due those merits, many cloud computing and AutoML pipelines provide fine-tuning services for an online stream of upcoming users with new data, different tasks and time limits. In order save the user's time, money, energy consumption, or even CO2 emission, an efficient online automated fine-tuning framework is practically useful and in great demand. Thus, in this work, propose to explore the problem of faster online fine-tuning.
The conventional practice of fine-tuning is to adopt a set of predefined hyperparameters for training a predefined model (Li et al. (020). It has three drawbacks in the current online setting: 1) The design of the backbone model is not optimized for the upcoming fine-tuning task and the selection of the backbone model is not data-specific. 2) A default setting of hyperparameters may not be optimal across tasks and the training settings may not meet the time constraints provided by users.
3) With the incoming tasks, the regular diagram is not suitable for this online setting since it cannot memorize and accumulate experience from the past fine-tuning tasks. Thus, we propose to decouple our faster fine-tuning problem into two parts: finding efficient fine-tuning networks and generating optimal fine-tuning schedules pertinent to specific time constraints in online learning fashion.
'The efficient training model zno (ET-NAS) has been released at:
NASOA

---

Under review as a conference paper at ICLR 2021
Recently, Neural Architecture Search (NAS) algorithms demonstrate promising results on discover- ing top-accuracy architectures, which surpass the performance of
networks and saves
human's 's efforts (Zoph et al., 2018 Liu et al.] 20 bj Radosavovic et al. 2019 Tan et al. 2019b] Real et al. 2019a Tan & Le 2019 Yao et al. 2020). However, those NAS works usually focus on inference time/FL.PP optimization and their search space is not flexible enough which cannot guarantee the optimality for fast fine-tuning. In contrast, we resort to developing NAS scheme with a novel flexible search space for fast fine-tuning. On the other hand, hyperparameter optimiza- tion (HPO) methods such as grid search (Bergstra & Bengio 2012). Bayesian optimization (BO) (Strubell et al. 2019a; Mendoza et al.] 2016), and BOHB (Falkner et al.) 2018) are used in deep learning and achieve good performance. However, those search-basee methods are computationally expensive and require iterative "trial and error", which violate our goal for faster adaptation time.
In this work, we propose novel Neural Architecture Search and Online Adaption framework named NASOA. First, we conduct an offline NAS for generating an efficient fine-tuning model zoo. We de- sign novel block-level and macro-structure search space to allow flexible choice of the networks. Once the efficient training model zoo is created offline NAS by Pareto optimal models, the online user can enjoy the benefit of those efficient training networks without any marginal cost. We then propose an online learning algorithm with an adaptive predictor to modeling the relation between different hyperparameter, model, dataset meta-info and the final fine-tuning performance. The final training schedule is generated directly from selecting the fine-tuning regime with the hest predicted performance. Benefiting from the experience accumulation via online learning, the diversity of the data and the increasing results can further continuously improve our regime generator. Our method behaves in a one-shot fashion and doesn't 't involve additional searching cost as HPO, endowing the capability of providing various training regimes under different time constraints.
Extensive experiments are conducted on multiple widely used fine-tuning datasets. The searched model zoo ET-NAS is more training efficient than SOTA ImageNet models, e.g. 5x training faster than RegGer, and 1. 1.7x faster than EfficientNetB3. Moreover, by using the whole NASOA, our online algorithm achieves superior fine-tuning results in terms of both accuracy and fine-tuning speed, i.c. improving around 2.1% accuracy than the best performance in RegNet series under various tasks; saving 40x computational cost comparing to the BOHB method.
In summary, our contributions are summarized as follows:
To the best our knowledge, we make the first effort to propose a faster fine-tuning pipeline that seamlessly combines the training-efficient NAS and online adaption algorithm. Our NASOA can effectively generate a personalized fine- tuning schedule of each desired task via an adaptive model for accumulating experience from the past tasks.
The proposed novel joint block/macro level search space enables a flexible and efficient search. The resulting model zoo ET-NAS more training efficient than very strong ImageNet SOTA mod- els e.g. EfficientNet, RegNet. All the ET-NAS models have heen released to help the community skipping the computation-heavv NAS stage and directly enjoy the benefit of NASOA.
The whole NASOA pipeline achieves much better fine-tuning results in terms of both accuracy and fine-tuning efficiency than current fine-tuning best practice and HPO method,e.g. BOHB.
2 RELATED WORK
Neural Architecture Search (NAS). The goal of NAS is to automatically optimize network ar- chitecture and release human effort from this handcraft network architecture engineering. Most previous works (Liu et al. 2018b] Cai et al. 2019b Liu et al. 2018a Tan et al. 2019a Xie et al., 2019 Howard et al.) 2019 aim at searching for CNN architectures with better inference and fewer FL.OPS. Baker et al. (207)) Cai et al. (2018) 8); Zhong et al. (2018) 8) apply reinforcement learning to train an RNN controller to generate a cell architecture. Liu et al. (2018b); Xie et al. (2019); Cai et al. (2019) try to search cell structure by weight-sharing and differentiable optimization. Tan & Le 2019) use a grid search for an efficient network by altering the depth/width of the network with a NAS focuses creating an efficient training model zoo for
nxed block structure. On the contrary, fast fine-tuning. Moreover, the existing search space design cannot meet the purpose of our search.
Generating Hyperparameters for Fine-tuning. HPO methods such Bayesian optimization (BO) (Strubell et al. 2219a] Mendoza et al.) 2016). BOHB (Falkner et al., 2018) achieves very promising result but require lot of computational resources which is contradictory to our original
2

---

Under review as a conference paper at ICLR 2021
Offline Neural Architecture Search
Online Cross-task Adaption
Pareto Optimal Front
tunlng Regine Generator
Topa Acannay
Online
Online Performance predictor P
Updane
Model
Epochs.
Train ing Time of the Network
Objecthv: Rud o of offecent
effleit dratneg Aetworkk
Disten Training Model Zoe (ST-MAZ)
Overview of our NASOA. Our faster task-oriented online fine-tuning system has two parts: a) Offline NAS to generate an efficient training model zoo with good accuracy and training speed; b) An online fine- fine-tuning regime generator to perform task ais-speciffic fine-tuning with a suitable model under user's time constraint.
Figure 1:
objective of efficient fine-tuning, On the other hand, limited works discuss the model selection and HPO for fine-tuning. Kornblith et al. (2019) finds that ImageNet accuracy and fine-tuning accuracy of different models are highly correlated. Li et al. (2020); Achille al. (2019) suggest that the optimal hyperparameters and model for fine-tuning should be both dataset dependent and domain similarity dependent (Cui et al. 2018). HyperStar (Mittal et al., 2020) is a concurrent HPO work demonstrating that a performance predictor can effectively generate good hyper-parameters for a single model. However, those works don't give an explicit solution about how perform fine- tuning in a more practical online scenario. In this work, we take the advantage of online learning (Hoi et al., 2018 Sahoo et al., 2017) to build a schedule generator, which allows us to memorize the past training history and provide up-and-coming training regimes for new coming tasks on the fly. Besides, we introduce the NAS model zoo to further push up the speed and performance.
THE PROPOSED APPROACH
The goal of this paper is to develop an online fine-tuning pipeline to facilitate a fast continuous cross-task model adaption. By the preliminary experiments in Section 4.1 we confirm that the model architectures and hyperparameters such as the learning rate and frozen stages will greatly influence the accuracy and speed of the fine-tuning program. Thus, our NASOA includes two parts as shown in the Figure [ 1) Searching a group of neural architectures with good accuracy and fast training speed to create pretrained model zoo; 2) Designing an online task-oriented algorithm to generate an efficient fine-tuning regime with the most suitable model under user's ' time constraint.
3.1
CREATING AN EFFICIENT TRAINING MODEL. Zo0 (ET-NAS) BY NAS
The commonly used hand-craft backbones for fine-tuning including MobileNet (Sandler et al.) 2018). ResNet (He al.] 2016), and ResNeXt (Xie et al. 2017). Recently, some state-of-lie the-art backbone series such as RegNet (Radosavovic et al. 2020p. and EfficientNet (Tan et al. 2019b) are developed by automated algorithms for higher accuracy and faster inference speed. However, the objective of our NAS is to find a group of models with good model generalization ability and training speed. Suggested by Kornblith et al. (219)) the model fine-tuning accuracy (model generalization ability) has a strong correlation between ImageNet accuracy (r 0.96). Meanwhile, the training speed can be measured by the step time of each training iteration. Thus, our NAS can be formulated by a multi-objective optimization problem (MOOP) on the search space S given by:
max (A). (A)) subject (A) Ti
where A is the architecture, acc(.) is the Top-1 accuracy on ImageNet, T.()) is the average step time of one iteration, and T,, is the maximum step time allowed. The step time is defined to be the total time of one iteration, including forward/backward propagation, and parameter update.
Search Space Design is extremely important (Radosavovic et al. 2020). As shown in Figure 2 to enable an efficient model, we propose novel flexible joint block-level and macro-level search space to enable simple to complex block design and fine adjustment of the computation allocation on each stage. Unlike existing topological cell-level search space such as DARTS (Liu et al. 201 8b), AmoebaNet(Real et al. 2019a), and NASBench (Dong& & Yang 2019), ours is more compact and avoids redundant skip-connections search space is more flexible to adjust the width, depth (for each stage), when to lwwn-sample/raise the channels. In contrast, EfficientNet only scales up/down the total width and depth by a fixed allocation ratio, and RegNet cannot change the number/type of operations in each block.
which have great memory access cost (MAC). Our block-level
3

---

Under review as a conference paper at ICLR 2021
Block-level Search Space
Macro-level Search Space
Channel Residual Connection
OP
Dutpot
Soankee pro thapere
Search For:
Number, types and channels of Operations, Number, place, and types of additional skip connections
Stge But Blokk SEME P.40
Search For: Channd Siee, Depth, b blodk in coch stage, When double channets.
Our joint block/macro-level search space to find efficient training networks. Our block-level search space covers many popular designs such as ResNet, ResNext, MobileNet Block. Our macro-level search space allows small adjustment of the network in each stage thus the resulting models are more flexible and efficient.
Figure 2:
Block-level Search Space. We consider search space based on 1-3 successive nodes of 5 different operations. Three skip connections with one fixed residual connection are searched. Element- add or channel-wist concat is chosen to combine the features for the siip-connections. For each selected operation, we also search for the ratio of changing channel size: x0.25, x0.5, x1, x2, x4. Note that it can cover many popular block designs such as Bottleneck (He et al. 2016), ResNeXt Xie et al. 2017) and MB block (Sandler et al., 2018). consists 5.4 x 10 unique blocks.
Macro-level Search Space. Allocation computation over different stages is crucial for backbone (Liang et al. 2020). Early-stage feature maps in one backbone are larger which captures texture details, while late-stage feature maps are smaller which are more discriminative (Li ct al. 2018). 8). Therefore, for macro-level search space, we design flexible search space to find the optimal channel size (width), depth (total number of blocks), when to down-sample, and when raise the channels. Our macro-level structure consists of 4 flexible stages. The spatial size of the stages is gradually down-sampled with factor 2. In each stage, stack number of block architectures. The positions of the doubling channel block are also flexible. This search space consists of 1.5 x 10" unique architectures. Details of the search space and its encodings can be found in Appendix [B.
Multi-objective Searching Algorithm. For MOOP in Eq [ we define architecture Ai dominates Ag (i) A₁ no worse than A2 in all objectives: (ii) Ai strictly better than A in at least one objective. A' is Pareto optimal if there is no other A that dominate A'" The set of all Pareto optimal architectures constitutes the Pareto front. To solve this MOOP problem, we modify well-known method named Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al.) (0000) to optimize the Pareto front P, The main idea NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve group of elite architectures. Then a group of new architectures sampled and trained by mutation of the current elite architectures on the P,, The algorithm can be paralleled multiple computation nodes and lift the Py simultaneously. We modify the NSGA-II algorithm to become a NAS algorithm: a) To enable parallel searching on N computational nodes, we modify the non-dominated-son method to generate exactly N mutated models for each generation, instead of variable size as the original NSGA-II does. b) We define a group of mutation operations for our block/macro search space for NSGA-II to change the network structure dynamically. c) We add a parent computation node to measure the selected architecture's 's training speed and generate the Pareto optimal models. Details of our NSGA-II can be found in Appendix [B2.2
Efficient Training Model Zoo Zoo (ET-NAS). By the proposed NAS method, we then create an afficient-tranning model zoo Zoo named ET-NAS which consists of K Pareto optimal models A; on Py. Then At are pretrained by ImageNet. Details of our NAS and A& can be found in Appendix B
3.2 ONLINE TASK-ORIENTED FINE-TUNING SCHEDUL. GENERATION
With the help of efficient training Zoo. the marginal computational cost of each user is minimized while they can enjoy the benefit NAS. We then need to decide a suitable fine-tuning schedule upon the user's upcoming tasks. Given user's dataset D and fine-tuning time constraint Ti, an online regime generator G(.,.) is desired:
[Regimerr, Ai] G(D,Ti) such that Acc FineT FineeTi une Deat) is maximized,
where the Regimerr includes all the hyperparameters required, i.e., lr schedule, total training steps, and frozen stages. G(...) also needs to pick up the most suitable pretrained model At from Z Note that existing search-based HPO methods require huge computational resources and cannot fit.
4

---

Under review as a conference paper at ICLR 2021
in our online one-shot training scenario. Instead, first propose online learning predictor Accp to model the accuracy on the validation set Acc( "T Deal ) by the meta-data information. Then we can use the predictor to construct ((.,.) to generate optimal hyperparameter setting and model.
3.2.1 ONLINE LEARNING FOR MODELING Aco(A,T, (Dal))
Recently, Li et al. (2020) suggest that the optimal hyperparameters for fine-tuning are highly related some data statisties such domain similarity to the ImageNet. Thus, hypothesis that can model the final accuracy by group of predictors, e.g., model information, meta-data description, data statistics stat(D), domain similarity, and hyperparameters. We list the variables we considered to predict the accuracy result as follows:
Model A name tonc-hot dummy variable)
ImageNet Acc. of the A
Domain Similiarity ImageNet (EMD) (Cui et al. 2018
#Cllsssss
Number Iteration
Average it. images per class
images per class
Leurning
Stages
Those variables can be easily calculated ahead of the fine-tuning. One can prepare offline training data by fine-tuning different kinds of dataset and collect the accuracy correspondingly and apply Multi-layer Perceptron Regression (MLP) offline on it. However, online learning should be more realistic setting for our problem. In cloud computing service or GPU cluster, sequence of fine- tuning requests with different data will arrive from time time. The predictive model can be further improved by increasing the diversity of the data and the requests over time.
Using a fixed depth of MLP model in the online setting may be problematic. Shallow networks maybe more preferred for small number of instances, while deeper model can achieve better perfor- mance when the sample size becomes larger. Inspired by [Sahoo et al. (2017), ) we use an adaptive MLP regression to automatically adapt its model capacity from simple to complex over time. Given the input variables, the prediction of the accuracy is given by:
Acc Regimerr,stat(D))
Σ arfi, where I = O,..., 1=1
(3)
f = h,, h = RELU(Ō;hi-),, he = [A', Regimerr, sta(()))
The predicted accuracy is weighted sum of the output fę of each intermediate fully-connected layer h4 The Wi and d are the learnable weights of each fully-connected layer. The oy is weight vector assigning the importance adapt its model capacity from simple to complex along with incoming tasks. The learnable weight or controls the importance of each intermediate layer and the final predicted accuracy is a weighted sum of f of them. The network can be updated by a Hedge Backpropagation (Freund & Schapire 1999h in which oj updated based on the loss suffered by this layer follows:
each layer and ||a|]
Thus the predictor Accp can automatically
af + Aess)) w/ + Wt 17au D(f, Accge)
L. $ 2/ Σ aj Lw, al
of Σạ
where 3 E 10 1){e)))) is the discount rate, the weight af are re-normalized such that ||||| = 1, and is the learning rate. Thus, during the online update, the model can choose an appropriate depth by an based the performance of each output at that depth. By utilizing the online cumulative results, our generator gains experience that helps future prediction.
Generating Trik-oriented Fine-tuning Schedule. Our schedule generator G then can make use of the performance predictor to find the best training regime:
G(D,T) Ti) = arg max Ze.... Regimerree Accp (A, Regimerr,stal (D).
Once the time constraint Ti is provided, the max number of iterations for different Ai can be cal- culated by an offline step-time lookup table for Zoo- The corresponding meta-data variables can be then calculated for the incoming task. The optimal selection of model and hyperparameters is ob- tained by ranking the predicted accuracy of all possible grid combinations. The Detailed algorithm can be found in the Appendix B.6
5

---

Under review as a conference paper at ICLR 2021
Table 1:
Datasets and their statistics used in this paper. Datasets in bold are used to construct the online learning training set. The rest used to test our NASOA. It commonly believed that Airerafts, Flowers and Blood-cell deviate from the ImageNet domain.
DotaSets
Cliss
Thain
I.Tsst
DataSers
Task
A.Emin
#.eest
Plowers Nilbbckk & Zisserman
2008
102
Fine- Grined
2K
StanfordCCu Cur Krause et all
2013
196
Fine-Grained
8K
8K
CUB-Birds
Welinder al.
2010
200
Fine-Grinned
10K
2K
MIT67 QQurtoon MunerQQuztoon & Terraho
2009
67
Scese cls.
5K
1K
Caltech10 Feiet al
2006
101
General
8K
IK
FoodTor (Bssard et sl.
2014
101
Fing Grained
75K
25K
Callech251 Gnilin al.
2007
257
General
25K
6K
PGVC Aircrafis Majj
2013
100
Fine Grained
7K
3K
Stanford-Nogg Khosla
2011
120
Fino-Griined
RK
Blood-ooll Singh
Medical Img.
4 EXPERIMENTAL RESULTS
4.1 PRELIMINARY EXPERIMENTS
We conduct a complete preliminary experiment to justify our motivation and model settings. Details can be found in the Appendix A According to our experiments, we find that for efficient fine- tuning, the model matters most. The suitable model should be selected according to the task and time constraints. Thus constructing model zoo with various sizes of training-efffcient efficieet models and picking up suitable models should be a good solution for faster fine-tuning. We also verify some existing conclusions: Fine-tuning performs better than training from scratch (Kornblith et al.) 2019) so that our topic is very important for efficient GPU training: Learning rate and frozen stage are crucial for fine-tuning (Guo et 2019), which needs careful adjustment.
4.2 OFFLINE NAS AND MODEL Zoo RESULTS
During the NAS, we directly search on the ImageNet dussset(Russakovkky et al., 2015). We first search for group of efficient block structure, then use those block candidates to conduct the macro-level search. We use short training setting to evaluate each architecture. It takes about hour on average for evaluating one archi- tecture for the block-level search and 6 hours for the macro-level search. Paralleled on GPUs, it takes about one week on 64-GPU cluster to conduct the whole search (5K+ K arch). Implementation details and in- termediate results can be found in the Appendix B
Table 2: Comparsion our ET-NAS models and SOTA ImageNet models. Inference time and training step time are measured in ms on single Nvidia V100, with bs 64.
Top-1
Training Step
Model Name
(ms)
Time (ms)
RegNerY-200MF
70.40
14.25
62.30
ET-AS-C
71.29
8.94
26.28
RegNetY-HOIMF
74.10
20.57
90.61
ET-NA-D
74.46
14.54
36.30
RegNetY 600MF
75.50
22.15
90.11
Faster Fine-tuning Model Zoo (ET-NAS), After iden- tifying the A from our search, we fully train those models on ImageNet following common practice. Note that all the models including ET-NAS-L can be eas- ily pretrained on a regular 8-card GPU node since our model is training-efficient. We are confident to re- lease our models for the public to reproduce our re- A
MobileNes V3-Large
75.20
16.88
71,65
OFANet
76.30
17.81
73.10
Amoebonet
75.70
28.39
141.45
ET-NAS-E
76,87
25.34
61.95
ERicienNer-R0
77.70
24.30
120.20
RegNotY-800MF
78.00
45.59
170.96
ET-NAS-F
78,50
33,83
93,04
EmhsenNNet-2
277.60
sults from scratch") and let the public save their energy/CO2/cost. Due to the length of the paper, we put the detailed encoding and architectures of the final searched models in the Appendix B.4, Surprisingly, found that smaller models should use simpler structures of blocks while bigger models prefer complex blocks.
RegN
192.
677.68
ETNAS-G
53,08
133.97
ETNAS-H
193.40
ErhiiennNee-B3
81.50
57.33
455.86
ELNASA
81.38
94.60
265.13
ETNAS-J
131.92
370.28
ET-NAS-L
191.89
512.52
Comparing our searched backhone to the conventional
ResNet/ResNeXt we find that early stages in our models are very short which is more efficient since
feature maps in an early stage are very large and the computational cost is comparably large. This also verified our findings in Appendix B.4.1
Comparison with the state-of-the-art ImageNet models. We compare the training/inference ef- ficiency of our searched ET-NAS with the SOTA ImageNet models such as MobileNetV3 (Howard et al. 2019, RegNet series (Radosavovic et al. 2020). and EfficientNet series (Tan & Le 2019)
The efficient training model zoo (ET-NAS) has been released at: httpps://iithbb.co NASOA

---

Under review as a conference paper at ICLR 2021
the Hilicienry
Compariton of the Inference eridencc Our Starched Vodels
IT-M66-6
IT
ST-664
FT-AGE
MTA
ETAWSG
ETMASH
ET-
IT
ET-MAS
Teghart
Reghert-- 6,00
IT-N
Reghatt-ac2
Rnghttt- teplett-120r
Regheth
Rockeisk
Motiakeri3
85
SerreebbedET-Ne Models
Our Ser MAS Meeeh
60
Tralking Strp-tioe siope-tinooo on dol V1000 Giinn)
Suodires -RRggteer Sodas
-ffideettiin Sedie
Figure 3: Comparison of the training and inference efficiency of our scarched models (ET-NAS) with SOTA models on ImageNet. Our searched models are considerably faster, e.g., ET-NAS-G is 6x training faster than RegNetY- 16GG, and ET-NAS-I is 1.5x training faster than EfficientNeeB3. Although our models are optimized for fast training, the inference speed is comparable to EfficientNet and better than RegNet series.
as shown in Table and Figure 3 Overall, our searched models outperform other SOTA Ima- geNet models in terms of training accuracy and training speed from Figure X (left). Specifically, ET-NAS-G is about 6x training faster than RegNerY-16GF, and ET-NAS-I is about 1.5x training faster than EfficientNeeB3. Our models are also better than mobile setting models such as Mo- bileNetV2/V et al. 2019) and RegNerY-200ME, Although our model is optimized for fast training. we also compare the inference speed in Figure Iright). Our models still have very strong performance in terms of inference speed. outperforming RegNet series and achieving comparable performance with EfficientNet. Figure 3 (mid), our method more training efficient than other NAS results, e.g. some evolution- based NAS methods such as AmoebaNet, OFANet.
What makes our network training efficient? To answer this, we define an efficiency score and conduct a statistical analysis of different factors for efficient-training (Details can be found in Appendix B.4.I). We have the fol-
Compartion Training offeciency with Other NAS Methods
Our Searched Models
Top Accuracy on imagetee
EffecientNanlil
Reghety 12GF
lowing conclusions: a) By observing optimal A smaller models should use simpler blocks while bigger models prefer complex blocks. Using the same block structure for all sizes of models (Tan & Le 2019) Radosavovic et al.) 2020) may not be optimal. b) Adding redun- dant skip-connections which have great mem- ory access cost will decrease the training ef- ficiency of the model thus existing topologi- cal cell-level search space such as DARTS (Liu cct al. 20188), AmocbaNct (Real ct al. 2019a), and NASBench101 (Dong& & Yang 2019) is not efficient. c) The computation allocation on dif- ferent stages is crucially important. Simply in- creasing depth/width to expand the model as Tan & Le (009) may not be optimal will down- grade the performance. To conclude our novel joint search space contributes most the training efficiency.
OFAMet
Regheet 6.4GF
75
atsuleeeentv
lugherr-4oog -ocker
Darts
*MoblieNevvis
65
50
100
150
200
Training time on Neida V100 bseG4[ms]
Figure Comparison of the training efficiency of our searched models (ET-NAS) with 8 other NAS re- sults on ImageNet. It can be found that our method is more training efficient than some recent evolution- based NAS methods such as AmoehaNet, OFANet be- cause of our effective search space.
4.3 RESULTS FOR ONLINE ADAPTIVE PREDICTOR Accp
Experimental Settings. We evaluate our online algorithm based on ten widely used image classifi- cation datasets, that cover various tue-tt tasks as shown in Table [T] Five of them (in bold) are chosen be the online learning training set (meta-training dataset). 30K samples are collected by continually sampling a subset of each dataset and fine-tuning with a randomized hyperparameters on it. Each subset varies from #. classes and #. images. The variables in Section 3.2. are calculated
7

---

Under review as a conference paper at ICLR 2021
accordingly. The fine-tuning accuracy is then evaluated on the test set. Then 30K sample is split meta-validation validation samples.
into 24K meta-traimin samples and 6K meta-
Then an adaptive MLP regression in Eq 3 are used to fit the data and predict the Ace(AAT Dvat). We use == 10 with 64 units in each hidden layer. We use a learn- ing rate of 0.01 and B 0.99. As base- lines, we also compare the results of us- ing fixed MLP with plain backpropagation with different layers(L = 3, 6, 10, 14). MAE (mean absolute error) and MSE (mean square error) are performance met- rics to measure the cumulative error with different segments of the tasks stream.
Table 3: Online error rate of our method and fixed MLP. Our adaptive MLP with hedge backpropagation better in the online setting of predicting the fine-tuning accuracy.
Cumulazive lar.
Segment 20-40%
Segmen 50
100%
Models
MAE
MSE
MAE
MSE
MAR
MSL
Fixed MLP (L=33
10.07%
1.94%
8.999
1.50%
7.99%
1.21%
Fised MLP (1=+0)
9.12%
1.21%
9.039
1.62%
7.16%
1,44%
Fixed MLP (1=100)
8.45%
1.59%
R.46%
1.53%
0.96%
Fixed MLP (L=14)
11,2445
2.91%
8,345
1.545
0.46%
Adaptive MLP w Hedge BP
7.51%
1.36%
7.55%
1.11%
3.73%
0.28%
Comparison of online learning method. The cumulative error obtained by all the baselines and the proposed method to predict the fine-tuning accuracy is shown in Table It can be seen that our adaptive MLP with hedge backpropagation is better than fixed MLP in terms of the cumulative error of the predicted accuracy. Our method enjoys the benefit from the adaptive depth which allows faster convergence in the initial stage and strong predictive power in the later stage.
4.4
FINAL NASOA RESULTS
Stanford-Car Car
MI167
Blood-call
Food101
FEVC Aircntt
3.28
0.55
0.53
396
0.655
0.59
0.75
0.75
385
2.5
0.5
0.55
233
0.55
2.45
45
300
322
200
Tian ConTCoE
Tire 00
Time contraitt
-m-EPlo ieEPiceenheet upper bound Effiiennitt lover bound
Resnet upper bound
Reonet lover bound
Reggeet upper
Reggiett lower bound
Our NASOA
Figure 5: Comparison of the final fine-tuning results under four time constraints for the testing dataset. Red square lines the results of NASOA in one-shot. The dots on the other solid line are the best performance of all the models in that series can perform. The model and training regime generated by our NASOA can outperform the upper bound of other methods in most cases. Our methods can improve around 2.1 11% 7.44% accuracy than the upper bound of RegNeu/EfficienNNet series on average.
To evaluate the performance of our whole NASOA, we select four time constraints on the testing datasets and use (.cp(.) to test the fine-tuning accuracy. The testing datasets are MIT67, Food101, Aircrafts, Blood-cell, and Stanford-Car. The shortest/longest time constraint are the time of fine-tuning 10/50 epochs for ResNeel///esNNet00.. The rest are equally divided into the log-space. For our NASOA, we generate the fine-tuning schedules by maximizing the predicted accuracy in [q We also conduct fine- tuning on various candidates of baselines such as ResNet (R18 to R152), RegNet (200MF to 16GF), and EfficientNet (BO to B3) with the default hyperparameter setting in Li et al. (2020).
Table 4: Comparison of the final NASOA results with other HPO methods. "HPO only" means only optimizing the hy- perparameters with RegNetY- Other HPO methods opti- mize both selecting hyperparameters and model from RegNet series models. "OA only"' is our online schedule generator with RegNet series models. "Our Z..." " means using our ET models zoo to find suitable model. "Fixed MLP Predictor" is the offline baseline with fixed MI.P predictor. "Our NASOA" is the our whole pipeline with both training efficient model zoo and online adaptive scheduler. Without additional search cost (x40), NASOA can reach similar performance of BOHB.
Methods
Seanch Cost Aircrafis MIT67
Si-Car
St-Dog
Kandom Search (HPO only)
x40
63.07%
75.00%
67.47%
86.15%
BOHI (1IOO only)
x40
72.70%
77.61%
70.94%
87.41%
Random Search
x40
81.07%
79.93%
$8.99
59.00%
BOLIB
X40
82.34%
79.85%
89.09%
Our Zoo with Random Search
x40
85.719
80.97%
87.84%
92.75%
Comparison of the final fine-tuning re- sults with the SOTA networks. We plot the time versus accuracy comparison in Figure 5 As can be seen, the model and
Our Zon with BOUI
x40
84.87%
82.34%
89.00%
93.74%
only
81.22%
79,33%
84.56%
59.70%
Z.o with MLP Predictur (Oflline)
xl
81.31%
75.97%
88.81%
58.58%
Oue final NASOA
x1
82.54%
80.30%
88.20%
92.30%
8

---

Under review as a conference paper at ICLR 2021
training regime generated by our NASOA can outperform the upper bound of other methods in most cases. On average, our methods can improve around 2.1%/7.49% accuracy than the best model of RegNNe//ffiicentNeet series under various time constraints and tasks. It is noteworthy that our NASOA performs better especially in the case of short time constraint, which demonstrates that schedule generator is capable provide both efficient and effective regimes for fast fine-tuning.
Comparison of the final fine-tuning results with the HPO methods. In Table 4 we com- pare our method with the HPO methods which optimizing the hyperparameters and picking up models in ResNet, RegNetY, and EfficientNet series. "HPO only" means the method only op- timizes the hyperparameters with fixed model RegNetY-16GF: 16GF- "OA only" is our online sched- ule generator with RegNet series models. "Our Zoo " means using our ET models zoo to find suitable model. "Fixed MLP Predictor" is the offline baseline with fixed MLP predictor (L=10) with model zoo. "Our NASOA" is the our whole pipeline with both training effi- cient model zoo and online adaptive scheduler. Comparing to the offline baseline with our NASOA, our online adaption module can boost the average performance by 2.17%. It can also be found that our method can save up to 40x computational cost compared HPO methods while reaching similar performance. With more computational budget, our model zoo with BOHB search can reach even higher accuracy (+avg. 10.28%)
Table 5: This ablative study calculates the aver- age tin-ttnnig accuracy over 5 tasks.
Existing
Adaptive
Comp
Avg. tuning
Methods
Model
Models
Models Schedaler
Carl
Axcuracy
BOHBI 101
x:00
77.17%
Our 7oo
x40
87.45% 28%
Our OA
xl
83.70%
NASOA
85.84% 88.679
Ablative interpretation of performance superiority. Table 5 calculates the average fine-tuning accuracy over tasks. Our NAS model zoo can greatly increase the fine-tuning average accuracy from 77.17% to 87.45%, which is the main contribution of the performance superiority. Using our online adaptive scheduler instead of BOHB can significantly reduce the computational cost (-40x).
5 CONCLUSION
We propose the first efficient task-oriented the-tuunig framework aiming at saving the resources for GPU clusters and cloud computing. The joint NAS and online adaption strategy achieve much better fine-tuning results in terms of both accuracy and speed. The searched architectures are more training-efficient than very strong baselines such as RegNet and EfficcienNNet. Our experiments on multiple datasets show our NASOA achieves 40x speed-up comparing to BOHB. The proposed NASOA can be well adapted to more tasks such as detection and segmentation in the future.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6430-6439, 2019.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec-
tures using reinforcement learning. In ICLR, 2017. 2
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. JMLR, 2012.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 discriminative compo-
nents with random forests. In ECCV, 2014, 4.1
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018. 2
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: (90::099989999 2019a. TO

---

Under review as a conference paper at ICLR 2021
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. In ICLR, 2019b. 2
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In CVPR, 2018. 2 3.2.1
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and Tanaka Meyarivan. A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In International conference
on parallel problem solving from nature, pp. 849-858. Springer, 2000. 3.1 B.2.2
Xuanyi Dong and Yi Yang. Nas-bench-200 201: Extending the scope reproducible neural architecture search. In International Conference on Learning Representations, 2019. 3.1 4.2
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter opti- mization at scale. ICML, 2018.
2
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. TPAMI, 2006.
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-103, 1999. 3.2.T
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4805-4814 14, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In CVPR, 2016. 3.1 3.1
Kaiming He, Ross B. Girshick, and Piotr Dollár. Rethinking imagenet pre-training, In ICCV, 2019.
Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive survey. arXiv preprini arXiv: 1002:0800..0871, 2018. 2
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In CVPR, 2019. 2 4.2
Chenhan Jiang, Hang Xu, Xiangdan Liang, and Liang Lin. Hybrid knowledge routed modules for large-scale object detection. In NewrIPS, 2018. [
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In Workshop on Fine- Tine-Grained Visual Categorization, CVPR, 2011.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In CVPR, 2019.
2 3.1
A.2
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fne-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In ICLR, 2020.
3.2.1
A.I
Zeming Li, Chao Peng. Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: A backbone network for object detection. In ECCV, 2018. 3.1
10

---

Under review as a conference paper at ICLR 2021
Feng Liang, Chen Lin, Ronghao Guo, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Compu- tation reallocation for object detection. In ICLR. OpenReview.net, 2020. 3.1
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018a. 2
Chenxi Liu, Liang-Chies Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Fei- Fei Li. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2018b. 3.14..2 4.2
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 13X.vv3:30.... 3..3006.51,,, 2013.
Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. To- wards automatically-tume neural networks. In Workshop Automatic Machine Learning, pp- 58-65, 2016. 2
Gaurav Mittal, Chang Liu, Nikolaos Karianakis, Victor Fragoso, Mei Chen, and Yun Fu. Hyperstar: Task-aware hyperparameters for deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8736-8745, 2020. 2
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing. pp. 722-729. IEEE, 2008
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In CVPR, 2009.
4.1
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yer Lo, and Piotr Dollár. On network design spaces for visual recognition. In ICCV, pp. 1882-1890, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing
network design spaces. arXiv preprint arXiv:2003.13675 2020. 3.1 3.1 4.2 4.2
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019 9a.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedingy of the aaui conference artificial intelligence, volume 33, pp. 4789-4789, 2019b.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252 2015. 4.2
Doyen Sahoo, Quang Pham, Jing Lu, and Steven CH Hoi. Online deep learning: Learning deep neural networks on the fly. In IJCAI, 2017. 3.2.1
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chian Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520, 2018. 3.1 3.1
Ishpreet Singh, Narinder Pal Singh, Harnoor Singh, Saharsh Bawankar, and Alioune Ngom. Blood cell types classification using cnn. In International Work-Conference on Bioinformatics and Biomedical Engineering, pp. 727-738. Springer, 2020. 4.1
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. Linguistically. informed self-attention for semantic role labeling. In EMNLP, pp. 5027-5038. Association for Computational Linguistics, 2018, B.5
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep
learning in NLP. In ACI., 2019a. 1
11

---

Under review as a conference paper at ICLR 2021
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint M.X.v:1906.02220 2019b. 2019b.
B.5
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 2019. 12
B.4.1
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019a. 2
Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection.
arXiv preprint arXiv: 2019b. 3.1
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech- DCSC Birds 200. Technical Report
California Institute of Technology, 2010. I
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans-
formations for deep neural networks. In CVPR, 2017. 3.1 3.1
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search.
In ICLR, 2019. 2
Hang Xu, Lewei Yao, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Auto-fpn: Automatic network architecture adaptation for object detection beyond classification. In ICCV, 2019. !
Lewei Yao, Hang Xu, Wei Zhang, Xiaodan Liang, and Zhenguo Li. SM-NAS: Structural-to-modulae neural architecture search for object detection. 2020.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng- Lin Liu. Practical block-wise neural network architecture generation. In CVPR, 2018. 2
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. T
A PRELIMINARY EXPERIMENTS
A.1 EXPERIMENTS SETTINGS
The preliminary experiments aim at figuring out what kinds of factors impact the speed and accuracy of fine-tuning. We fine-tune several ImageNet pretrained backbones various datasets shown in Table 6 (right) and exam different settings of hyperparameters by a grid search such as: learn- ing rate (0.0001, 0.001, 0.01, 0.1), frozen stages (-1,0,1,2,3), and frozen BN (-1,0,1.2,3). Frozen stages/frozen BN= k means 1 to kth stage's 's parameters/BN statistics are not updated during train- ing. The training settings most follow Li al. (020) and report the Top-1 validation accuracy and training time. Its detailed experiment settings are hyperparameters are listed as follows:
Comparing fine-tuning and training from scratch. We use ResNet series (R-18 to R-50) evalu- ate the effect of fine-tuning and training from scratch. Following Li et al. 2220)) we train networks on Flowers 102, CUB-BI Birds,, MIT67 and Caltech 101 datasets for 600 epochs for training from scratch and 350 epochs for fine-tuning to ensure all models converge on all datasets. We use SGD optimizer with an initial learning rate 0.01, weight decay le-4, momentum 0.9. The learning rate is decreased by factor 10 at 400 and 550 epoch for training from scratch and 1550, 250 epoch for fine-tuning.
Optimal learning rate and frozen stage. We perform a simple grid search on Flowers 102, Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets with ResNet50 to find optimal learning rate and frozen stage on different datasets with the default tne-tune setting in Li et al. (2020). The hyperparameters ranges are: learning rate 10.1, 0.01, 0.001, 0.0001), frozen stage (-1, 0. 2,
Comparing different frozen stages and networks along time. We fix different stages of ResNet50 to analyze the influence of different frozen stages to the accuracy and along the training time on Flowers 102, Stanford-Car, CUB-Birds, MIT67, Stanford-Dog and 11 datasets. We pick
12

---

Under review as a conference paper at ICLR 2021
Table Comparison Top-1 accuracy and training time (min) on different datasets. Comparing to training from scratch, fine-tuning shows superior results in terms of both accuracy and training time.
ResNet18
ResNet34
ResNet50
Dataset
Method
Acc.
Time (min)
Acc.
Time (min)
Acc.
Time (min)
From Scratch 94.4%
11
93.8%
19
90.7%
38
Flowers102
Fine-tuning 98.3%
7
98.5 %
11
98.8%
22
From Scratch 80.6%
14
81.9%
24
81.5%
47
Stanford-Car
Fine-tuning %
8
89.6 %
14
91.1%
28
From Scratch 51.6%
11
53.6%
18
44.6%
35
CUB-Birds
Fine-tuning 69.2 %
6
71.9%
10
74.9%
21
From Scratch 67.8%
22
69.2%
37
66.0%
MIT67
Fine-tuning 76.4%
13
76.9%
21
78.1%
43
From Scratch 60.6%
21
62.6%
35
55.2%
70
Stanford- Dog
Fine-tuning 69.7%
12
73.3%
20
75.0%
41
From Scratch 82.5%
12
78.1%
20
75.7%
40
Caltech101
Finc-tuning
90.8%
7
91.8% %
12
91.8%
23
Table 7:
Fine-tuning on R50, the optimal learning rate and optimal frozen stage found by grid search are different and should optimized individually.
Optimal
Optimal
DataSet
H.Class
# #.Images
Best Acc.
Frozen Stage
Flowers
102
8K
0.01
0
99.3%
Stanford- Car
196
16K
0.1
1
91.8%
CUB-Birds
200
12K
0.01
2
81.3%
MTT67
67
16K
0.01
-1
80.8%
Stanford-Dog
120
21K
0.01
83.7%
Caltech 101
101
8K
0.001
-1
96.4%
Caltech256
257
31K
0.01
85.6%
the training curves CUB-Birds and Caltech101 to in the main text of this paper. We also com- pare the fine-tune results along time with various networks on these datasets as shown Figure 7 On Caltech101, ResNet50 dominates the training curve at the very beginning. However, on other datasets, ResNet18 8 and ResNet34 can perform better then ResNet50 when the training time is short.
A.2 FINDINGS OF THE PRELIMINARY EXPERIMENTS
With those preliminary experiments, we summarize our findings as follows. Some of the findings are also verified by some existing works.
Pine-tuning performs always better than training from scratch. As shown in Table 6 fine- tuning shows superior results than training from scratch in terms both accuracy and training time for all the datasets. This finding is also verified by Kornblith et al. (2019). Thus, fine-tuning is the most common way train a new dataset and our framework can be generalized to applications.
CUB-Brssacce
Caltecn200
Flower100
Caltedh00
Training LEEO
Trinking time
Trakingtiie
Teakiing TYre
Figure 6: (Left) Fine-tuning ResNet101 with different weight-frozen stages. "Freeze: k" means 0 to k stage's parameters are not updated during training. The number of frozen stage will effect training time and accu- racy. Its optimal frozen setting varies with datasets. (Right) Comparison of accuracy/time different fine-tuning models, Different models should selected upon the request of different datasets training constraints.
13

---

Under review as a conference paper at ICLR 2021
NITE7 an WiNCOr tewe
Stafford Ong arthileeterr acNS
Traning
Figure 7: Fine-tune results along time with various networks on these datasets. It can be seen that if the time constraints short, we should choose a smaller network.
We should the optimize learning rate and frozen stage for each dataset. From Table 7 it seems that the optimal learning rate and optimal frozen stage found by grid search are different for various datasets. Figure Galso shows that the number the frozen stages will affect both training time and final accuracy. Guo et al. (2019) also showed that frozen different stages are crucial for fine-tuning task. Those two hyperparameters should optimized for different datasets.
Model matters most. Suitable model should be selected according to the task and time con- straints. Figure 6 (right) suggests that always choosing the biggest model to fine-tune may be an optimal choice, smaller model can be better than the bigger model if the training time is limited. On the other hand, also important since better model can be converged faster by a limited GPU budget. For example, Figure / shows that if the time constraint is short, we should choose smaller network i.e. ResNet18 here. Thus, it is urgent to construct training-efficien model zoo.
consider the training efficiency of the model
BN running statistics should not be frozen during fine-tuning. We found that frozen BN has very limited effect on the training time (less than +5%), while not freezing BN will lead to better results almost all the datasets. Thus, BN is not frozen all experiments for our NASOA.
B DETAILS OF THE OFFLINE NAS
B. SEARCH SPACE ENCODINGS
The search space of our architectures is composed of block and macro levels, where the former decides what a block is composed of, such as operators, number of channels, and skip connections. while the latter concerned about how combine the block into a whole network, e.g., when to do down- wwwn-sampling, and where to change the number of channels.
B.1.1 BLOCK-LEVEL ARCHITECTURE
Block-level design. A block consists of most three operators, each of which is divided into species and has 5 different number of output channels. Each kind of operator is de- noted by an op number, and the output chan- nel of the operator is decided by the ratio be- tween it and that of the current block. Details shown in Table B.I.I By default, there a skip connection between the input and out- put of the block, which sums their values up. In addition to that, at most 3 other skip con- nections are contained in block, which ei- ther adds concatenates the values between them. Each operation is followed by a batch normalization layer, and after all the skip con- nections are calculated, a ReLU layer is trig- gered.
Input
BN
Retu
Figure 8: Block structure and two block samples. (a) shows three-node graph. (b) is an example with encoding "031-", and (c) is "02031-a02".
Block-level encoding. The encoding of each block-level architecture composed of two parts separated by 1-", which considers the operators and skip connections respectively.
14

---

Under review as a conference paper at ICLR 2021
For the first part (operators part), each operator is represented by two numbers: op number and ratio number (shown Table B.I.I.. As the output channel of the last operation always equals to that of the current block, the ratio number of this operator is removed. Therefore, the encoding of block with n operators always has length 2n for the first part of block-level encoding.
For the second part (skip connections part), every skip connection consists of one letter for addition ("a')/ concatenation ('c'), and two numbers for place indices. n operators separate the block to n parts, which after the first operator. Since the skip connection between the beginning and end of the block always exists, it is not shown in the encoding. Thus this part has length 3/- 3 (possibly when there is skip connections. Some of the encoding examples are shown in Figure
indexed with 0, 1, ....., Thus 'a01" means summing up the value before and
B.1.2 MACRO-LEVEL ARCHITECTURE
Macro-level design. We only consider networks with exactly 4 stages. The first block of each stage (except Stage I) reduces the resolution of both width and height by half, where the stride 2 is added to the first operator that is not conv Ixl. Other blocks do not change the resolution. One block's output channel is either the same,
integer multiple of the input channel.
Macro-level encoding. The stages are divided apart by 3 signs. For every stage, each block is represented by integer, which shows the ratio between output and input channel for this block.
B.1.3 ENCODING AS A WHOLE
Thus the whole backbone can be encoded by simply concatenating the block and macro encoding. The encoding of the whole network is formatted as:
Blookk ENCODING) [First. Boat..HHANT CHANNEL]
Some common architectures, including ResNet and Wide ResNet can be accurately represented by our encoding scheme, which is shown in Table B.I.3
B.2 NAS SEARCH ALGORITHM
B.2.1 NON-DOMINATED SORTING ALGORITHM
Non-dominated sorting is mainly used to sort the solutions in population according to the Pareto dominance principle, which plays very important role in the selection operation of many multi- objective evolutionary algorithms. In non-dominated sorting, an individual A is said to dominate another individual B, if and only if there is no objective of worse than that objective of B and there is at least one objective of A better than that objective of B. Without loss of generality, we assume that the solutions of a population S can be assigned K Pareto fronts F,, i 1.2..... K. Non-dominated sorting first selects all the non-dominated solutions from population $ and assigns them to F1 (the rank front); it then selects all the non-dominated solutions from the remaining solutions and assigns them to F2 (the rank front); it repeats the above process until all individuals have been assigned to a Pareto front.
B.2.2 NSGA-II: ELITIST NON-DOMINATED SORTING GENETIC ALGORITHM
To solve the problem in Eq. [ Elitist et al. 2000 is adopted to optimize the Pareto front Py as shown in Algorithm D In this paper,
sorting genetic algorithm (NSGA-II) (Deb)
Table The operations and channel changing ratios considered our paper. Encoding for operators and ratios. stands for the channels of the current block.
op number
operator(Input c)
ratio number
ratios
0
conv3x3
0
1/4
1
conv 1x11
x 1/22
2
conv3x3, group=2
2
x
3
conv3x3, group=4
3
x2
4
conv3x3, group=c
4
x4
15

---

Under review as a conference paper at ICLR 2021
Table 9: ResNets and Wide ResNets are represented by encoding scheme. Basic Block is repre- sented as '020-', as the two operators are both conv3x3 (denoted as '0"), and the output channel of the first operator equals that of the block output (represented as '2'), and other skip connection except the one connecting input and output; the macro-arch of ResNet 18 is encoded as '11-21-21- 21', as each stage contains two blocks, where the first block in Stage 3, doubles the number of channels.
model
encoding
ResNet18
020-664_11-21-20
ResNet34
ResNet50
1000--c4.411-21122111112
Wide ResNet50
11011:6644412211121111111
we choose this kind of sample-based NAS algorithm instead of many popular parameter-sharing NAS method. This is because we want to further analysis of the sampled architectures and achieve insights and conclusions of the efficient training. The main idea of NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve a group of elite architectures. Then a group of new architectures is sampled and trained by mutation of the current elite architectures on the Py The algorithm can be paralleled on multiple computation nodes and lift the Py simultaneously. The mutation in the block-level search space includes adding new skip-connection, modifying the current operations and ratios. Meanwhile, the mutation in the macro-level search space includes randomly adding or deleting one block in stage, exchanging the position of doubling channel block with its neighbor, and modifying the base channels. This well-known NSGA-II is easy to implement and can easily monitor the improvement of each iteration. The stop criterion depends on the time limit or the computation cost constraints.
Algorithm Our modified NSGA-II Searching Algorithm
Input Stop criterion, Search Space, number of computation nodes N.
1: Rondom A), generate group of initial architectures.
2: P. +-
3: Evaluate Pt
4: while not stop criterion do
5: Apply non-dominated sorting to P. to obtain non-dominated fronts A
6: Sort Aū by Crowding distance and left top-N A Parents
7:
Create new generation Qe by mutation on current A;
8:
Train the Qi on N computation nodes and Evaluate the accuracy of Q..
9:
P0+1 4 Q
10:
1=t+1
11:
end while
Output The Pareto Optimal Front AQ
B.3 NAS IMPLEMENTATION DETAILS
B.3.1 BLOCK-LEVEL. SEARCH
In the phase of block-level search, a proxy task of ImageNet is created, which is a subset sampled fromits training set. This subset constitutes 100 labels, each of which has 500 images the training set, and 100 as the validation set. We call this dataset ImageNet-100 -100 in the following parts of this paper.
To avoid interference with macro architecture, the macro-level architecture is fixed to be the same as that ResNet50. Each model is trained by ImageNet- 100 with a batch size 32 for 90 epochs and learning rate 0.1, which takes 3*10 hours single NVIDIA Tesla-V100 GPU. We do random search at first, which uniformly samples all the valid blocks in the search space. Evolutionary Algorithm (EA) is then performed with three kinds of mutations: 1) replace another; 2) change the output channel of one layer; 3) Add/remove/modify a skip connection. We keep updating the Pareto Front between step time and accuracy during the whole process. As
operator with
16

---

Under review as a conference paper at ICLR 2021
Table 10: The searched optimal efficient training models "ET-NAS" found by our NAS search. 'Acc' means the accuracy evaluated on the ImageNet; inference time and step time are measured in mss on single Nvidia V100, with batch size of 64. By observing the optimal model, smaller models should use simpler blocks while bigger models prefer complex blocks.
Top-1
inf time
Training step
Name
Facoding
MPhmmm
MAct
ETNAS-A
2.52.241-112-12111
2,6
0.255
1.3
62.06
5.30
14.74
081-322-4-221111221
ET-NAS-C
011-2222211-2-111122
7.1
0.58
2.0
71.29
8.94
26.28 28
ET-NAS-D
081-644---3311111211
1.55
2.6
74,46
36,30
011-8A22-211-22-111112
2.61
76.87
25.34
61.95
EF
10001 (0001-B8.44111-11222111
28.4
2.31
6.8
78,90
33.83
93.04
EA
211- 04411211-21-1111112
5.68
80.41
53,08
133.97
EE-AASH
10001 0000-66-4111111111211111
44.0
5,33
10.9
80.92
76.80
193.40
EENASA
02111-00944111--211121111
1111111-211
265.13
ETNAS-J
211 311-64.1112111-2111111
103.0
18.16
15.9
82.08
131.92
370,28
2001--00..641121--111
ET-NAS-K
87.3
27.51
31.3
82,42
185.75
505.00
-11111111111111
23312-11212244:21:2111
E
130,4
82.65
191,89
542.52
-111111111111
result, 10 blocks are selected the candidates for the following rounds. Practically, during our search, the performance of early stop models aligns well with the fully-train accuracy. We checked the Spearman Rank Correlation for 103 architectures: reduce the search cost by around 90% while keeping our NAS effective.
96.6%. Thus, using early stop greatly
B.3.2 MACRO-LEVEL SEARCH
We search the block-level architectures with the 10 blocks attained by block-level search. Random search is adopted at first, where the number of blocks is chosen ran- domly between and 50, and the first and last channel is drawn from {32, 64, 128] and {512, 1024, 2048) respectively. EA search is then applied, where the muta- tions allowed are: I) Add :1;;;" 2) Remove 3) Swap
Pytorch GPU time PF
70
different adjacent num- bers. Similar to block-level search, Pareto Front between step time and accuracy is also kept updated.
25
Figure 9: Results of the block-level search in ImageNet-100. The y-axis denotes the accuracy and x- axis denotes the latency. Blue dots are models searched in this step, while the red ones Basic Block with first channel 64, 128, 192; Inverted Bottleneck Block (expansion rate with first channel 64, 128; Bottle- Neck Block (expansion rate 4) with first channel 256, 320. It can be found that our algorithm find more efficient block in the block-level search.
Different from the previous phase, the whole ImageNet dataset is utilized for training. Each model is trained with a batch size of 1024 and learning rate of 0.2 for epochs.
B.4 ET-NAS: MODEL ZoO INFORMATION AND THEIR ENCODINGS
After the NAS process done in Section B.3 12 models are selected as model
Zoo ET-NAS of fine-tume tuning. Details of these models are shown Table B.4 The inference time and step time measured in the standard setting of ImageNet: 224x224. By observing the optimal model in the table, smaller models should use simpler blocks while bigger models prefer complex blocks.
single Nvidia V V100, with batch size of 64. The resolution follows
17

---

Under review as a conference paper at ICLR 2021
Comparison of Training effeciency with Other NáS Mechods
affeciency
85
80
Our Searched Models
Searched Models
Top Accurcy on imagetee.
ORAAAe
Effecienttaabb BegheetV-120
RegNittY-6
Top-1 Rocuracy on Imagetee
Tagl
Ne
RegNeff
Redebaaeet
75
MablleNetva A PCDarts
75
Mybile Nervs
PCDarts
Darts
20
-0
200
*MobileNeeV35
MobileNetvas
68
60
60
50
100
150
200
250
300
80
Training Step-time Nivilla b5:64imms
Inference Time on Notdla V100 bsn64/ms)
Figure 10: Comparison of the training and inference efficiency of our searched models (ET-NAS) with SOTA NAS models on ImageNet. further compared our models with 8 other NAS results. can be found that our method is more training efficient than some recent evolution based NAS methods such as AmoebaNet (Real 2019a) because of our effective search space.
et al. 2019b), OFANet (Cai et
Table 11: Regression Analysis: what makes a network efficient- training? We exam the effect of each component of network on the efficiency score. "Coef" and "SE Coef" are the estimated regression coefficient and standard error. "T--Vlle""/yg Value"" shows the significance of the variables.
level Regressicn Analysis
n-5500
Rosq-5616
Macro Regression Analysis
n=1200
R x-x--1%
Terms
Coel
SE Coef
V-luue
V-Value
Terms
Coef
SE Cuef
T-Value
Valuue
OP! Channel Change Ratio
-0.183
0.010
-28577
0.06
Channel Size
-0,210
0.023
-9.66
0,00
Channel Change
0.010
-27.55
Double Channel Pusition
-0.110
0.006
-4.17
0.00
Num of ship connection (add)
-0.272
0.018
-15.12
0.00
Double Chansel Position
0.035
0.000
1.15
0.25
Nuw of skip connection (concat)
0.018
-19.93
Double Chansel Position
0.000
-0.4
Ourput chansel
-0.539
0.010
-3.57
0.00
Double Channel Position 4
0.224
0.025
9.14
0.00
303 (ref)
0.000
Dauble Chansel Position
0.022
1.62
0.11
comexx]
-0.037
0.033
-0.72
0.08
Nam Mark be Siage-1
-0,562
0.023
-24.89
0.00
comvtc, sr group=2
0.190
0.035
5.69
0.00
Num block in Singe-2
-0.139
0.023
6.01
0.00
3033323, group=/
0.295
0.034
8.77
0.00
Num block in Singe-3
0,044
0.024
1.88
0.06
Separable comr3x3
-0.200
0.34
-5.91
0.00
Nam black in Stuge-4
.0.100
0.004
-0.42
0.67
Figure 10 shows the comparison of our ET-NAS models with other SOTA ImageNet models. Infer- ence time and training step time are measured in ms on single Nvidia V100, with bs = 64. Our ET-NAS series show superior performance comparing to RegNet, EfficientNet series. Comparing to some EA-based NAS methods such as OFANet and Amoebanet, our method is also efficient in terms of training. We found that there exists a performance ranking gap between inference time and training step time in [T] This is mainly due to the depth and the main type of operation of the models. We found that deeper networks with separable conv such as EfficientNe//MoiilNeee have a larger conv).
ratio comparing to our models (shlllower&more common
B.4.1
WHAT MAKES NETWORK EFFICIENT- TRAINING?
To answer this question, we first need to define a score for the efficiency of the searched models A. In MOOP, the goodness of a solution determined by dominance. Thus, we can use the non-dominated sorting algorithm to sort the A according to the Pareto dominance principle. Each architecture is assigned to one Pareto front and the rank Rp of that Pareto front can be regarded as the goodness of a solution, in our case, the efficiency. We then defined the efficiency score of A as: (x( = (e)))((((()))) Since Pareto optimal front front is the Rank Pareto front, larger efficiency score [t)( sg (A) means better efficiency.
Then we perform multivariate linear regression analysis the As. According to our search space, ordinal/nominal variables that describe the model are denoted as predictors to fit the (y() Table I1 shows the coefficients from the regression analysis both block-level and macro-level
18

---

Under review as a conference paper at ICLR 2021
Positive coefficients indicate positive relationship. "P-Value" shows the significance of the variables. We summarize and highlight several noteworthy conclusions uncovered by our analysis:
designs.
By observing optimal A,, smaller models should use simpler blocks while bigger models prefer complex blocks. Simply increasing depth/width to expand the model in Tan & Le (2019) may not be optimal.
Adding additional skip connections will decrease the training efficiency of the model (The Coef is significantly negative). Using "add" to combine the features is more efficient than "concat". Using "conv3x3, group=4" is the best operation among the searched operations (Coef 0.295).
Separable conv3x3 is not efficient for training (Coef is -0.2).
The first double-channel position should more close to the beginning of the network, while the
final double channel-position should be delayed to the end of the network.
Fewer blocks should be assigned the first two stages. More should be assigned to the 3rd stage.
B.5
CO2 CONSUMPTION ANALYSIS
Fine-tuning from the pretrained ImageNet/language model is a de-facto practice in the deep learning field (CV/NLP). Our NASOA improves the efficiency of fine-tuning which has the potentials to greatly reduce computational cost in GPU clusters/cloud computing. According to recent study (Stubell et al. 2019b), developing and tuning for one typical R&D project (Strubell et al.) 2018) in Google Cloud computing needs about $250k cost, 82k kWh electricity, and 123k lbs CO2 emission, which equals to the CO2 consumption of air traveling (NY+ 62 times. Among most of them, 123 hyperparameter grid searches were performed for new datasets, resulting 4789 jobs in total. is believed that the proposed faster fin-tuning pipeline can save up to 40x computational cost among them. Furthermore, we have released all the searched efficient models help the public skipping the computation-heat NAS stage and directly enjoy the benefit of our methods. In conclusion. our NASOA is meaningful for environment protection and energy saving.
B.6 DETAILED ALGORITTS OF NASOA
Detailed algorithms of our Model Zoo (ET-NAS) search can be found in Algorithm 2 The pseudo code of online fine-tuning schedule generator training, prediction, and update can be found in Algo-
rithm 3
Algorithm Efficient Training Model 7oo (ET-NAS) Creation
Input: Block/Macro Search Space S,, Sa. Stop Criterion r, #Computation Nodes K, Sensitive Factor e, # Blockk Architectures M, #Models Model Zoo N.
Output: Final Model Zoo Zoo
1: procedure BLOCKSBARCH( T, K)
2: P NSGA-II(T, S, K)
Our modified NSGA-II, Algorithm D Most common M cells from Pf
3: Cells 1alss-Moo MOSTCOMMONN( M)
4: end procedure
5: procedure MACROSEARCH((,,,, K,e,
6: Pj t NSGA-II(T, (Cells), K
7: Zno & N
Choose models based on nowding-distanee
8: end procedure
C IMPLEMENT DETAILS OF HPO METHODS
We use BOHB and random search in our experiments as the HPO baseline. As stated in Section 4.4, The shortest/longest time constraint (budget) is defined the time of fine-tuning 10/50 epochs for BesNNet88RRsNeeee as: tg = to * (ts/to) (t/s,) where x = [o. 1, 2,23,, 3), tyis the time to train ResNet18 for 10 epochs and ta is the time train ResNet101 for 50 epochs.
and the rest are equally divided into the log-space, which can be represented
We only compare the HPO setting under the same max computational budgets equal to tj in Table 5 (left). For random search, randomly sample candidates from predefined search space until
19

---

Under review as a conference paper at ICLR 2021
Algorithm 3 Online Fine-Tuning schedule Generator Training, Prediction, and Update
Input:
Model Zoo Zoo. Time Evaluator T. Acc Evaluator T,, Hyper-parameter Space SHP, Known Datasets Dold, New Dataset Dnew. #Meta-data HM. Time Constraint Ti, #Configurations H.
Output: Optimal Model A"" Hlyer-parameter Regime FT Predictor Accp
1: procedure OFFLINETRe RANNEETTTAINIINGGOO.. Tr, SHP, Dota, HM)
2: MelaDala +- 0, Acep <- ADAPTIVEMLP()
> Initialize default predictor
3: for D E Dald, 4- to M do
4: A, RegimerT +- RANDOM(Zoo SHP)
Randomly select from search space > Train with selected configuration Add this result meta-data
5: Ace - (-,, FT)
6: MetaData MetaData U ((A, Regimerr, Acc))
7: end for
8: Accp TRRIN(ACop,, MetaDataa
Train predictor with all meta-data
9:
end procedure
procedure ONLNEPREDICTIONNZoo Dnew Acep, T, H)
11: Meta + 0
for + H do
13:
A + RANDOM(2.o)
Epoch <- + T,(A, Dnew)
D Always choose the largest epoch within Ti Randomly select condition on Epoch
15:
RegimeFT +- RANDOM(SHP) (Lpocch) Epoch)
16:
MetaData + MetaData {(A, Regimert)}
17:
end for
18:
A", Regimeee + M
Choose the optimal from H configs
19:
Acc T,(D news A", Regimeery FT)
20: Accp &- TRAIN(AcCP, (A", Regimerrr, Acc)) b Improve predictor with this meta-data
21:
end procedure
reaching the max computational budget. And for BOHB, we use the opensource implementation of BOHB at l www.h/hwwwoooooaoooooooeeeeeeeee com/ aut oml /HppandSrerr ottpa//aaooaooooma/aaaaaaattterr percent of good 0bservations=115% min budget=25% and max budget=100% with respect to our max computational budget. And we use BOHB with 40x computational cost than our proposed methods with different model zoos. The results are presented in Section 4.4
We set random fraction=0.3,
20


Reviews:
[{"review_id": "_VLsnyIxhn9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a joint Neural Architecture Search and Online Adaption (NASOA) framework is proposed to achieve a faster task-oriented fine-tuning upon the request of users. In particular, two main contributions are made in this paper: (1) A fine-tuning pipeline that seamlessly combines the training-efficient NAS and online adaption algorithm is introduced, which can effectively generate a personalized fine-tuning schedule of each desired task via an adaptive model for accumulating experience from the past tasks. (2) A block-level and macro-level search space is introduced in the resulting framework, which enables a simple to complex block design and fine adjustment of the computation allocation on each stage.\n\nI have two  comments about this paper， which  go as follows:\n\n(1)\tIn this work, a joint NSAOA framework is introduced facilitate a fast continuous cross-task model adaption. Why not learn an effective fine-tuning regime based on a hand-crafted network? In my point of view, a good fine-tuning regime should be also independent of the choice of network.\n\n(2)\tThe block-level and macro-level search space can reduce the redundant skip-connections in the resulting structures. To the best of my knowledge, the skip-connection is effective to avoid the gradient vanishing problem in a very deep neural network, therefore the skip-connections at the lower layers are very important in the network design. However, the skip-connections at the lower layers will take more memory cost at both training and testing stages. From this point of view, authors should explain where the redundant skip-connections always occur in the rebuttal.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_NJCWzXMrRO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This works aims at task-oriented fine-tuning from pre-trained ImageNet models. It proposes a Neural Architecture Search and Online Adaption framework (NASOA) to perform fast task-oriented model fine-tuning. The NASOA first employ an offline NAS to select a group of models and then pick up the most suitable model from this group via an online schedule generator. \n\nWeaknesses:\nThe proposed method may be useful in some fine-tuning scenarios, however will have low overall impact. From the reviewer's view, lack of novel and interesting part. The proposed approach simply uses and combines existing methods, e.g., NAS, seems more like an engineering project.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YJjkmxoyHVv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\n\nThe paper presents a joint Neural Architecture Search (NAS) and Online Adaption (OA) framework named NASOA, aiming at providing faster task-oriented ﬁne-tuning system. The first step of the approach is an offline NAS to form a pretrained training-efficient model zoo, which is followed by an online scheduler which selects the most suitable model and generates a personalized training regime with respect to the target task.\n***************************************************************************\nSTRENGTHS\n- The paper is fairly well written and easy to follow. \n- The proposed approach is fairly novel: up to my knowledge, this is the first effort to combine NAS and Online Adaptation techniques for faster fine-tuning task. In addition, the proposed joint/block macro level search space is novel too.\n- The practical benefits of the proposed approach are obvious for AutoML systems, and the shared ET-NAS model zoo will help other researchers working on this topic.\n- The paper provides (in appendix B) a short but interesting analysis on CO2 consumption. This is a good practice which is rare enough to be worthy of note.\n***************************************************************************\nWEAKNESSES AND REMARKS\n- I have some concerns regarding the fact that the paper does not compare the proposed approach to a baseline offline setting. The idea would be to collect offline training data by finetuning on several datasets (of different nature) and collect the corresponding accuracies to train a predictor (e.g. an MLP). The paper mentions this possibility, but says that it is less realistic than the online setting. Which is true, but it would have been interesting to compare the two settings\n- I’m not sure to understand the relevance of the preliminary experiments presented in sub-section 4.1. The findings (i.e. 1. fine-tuning performs better than training from scratch, and 2. learning rate and frozen stage are crucial for ﬁne-tuning) seem obvious to me, and correspond to widely-adopted common practices.\n- It’s not clear to me how the models are organized into groups in Table 2. Are they grouped based on their respective complexity? If so, it should explicitly mentioned by adding a column with the number of parameters of each model. \n- When presenting the ImageNet results in Table 2, the paper does not mention that current sota on ImageNet is around 88% top-1 accuracy, i.e. about 6 points above the best result presented in the Table. The paper should add this information to slightly tone down the claim that ‘searched models outperform other SOTA ImageNet models in terms of training accuracy’.\n- It would have been interesting to give more details about the NSGA-II algorithm in the paper itself, rather than only referring to the appendix. I understand that it’s due to the lack of space, but it is a little bit frustrating to have no information in the paper about this point, especially about the modifications compared to the original algorithm.\n- Figures 3 and 4 are quite useless in their current form, since the text size is too small to be read. \n- Table 4 is hardly interpretable without reading its corresponding descriptive paragraph in the paper. Authors should give more information (e.g. in the caption) about each method.\n- Page 5, Line 5: MLP stands for Multi Layer Perceptron (not Perception).\n***************************************************************************\nJUSTIFICATION OF RATING\n\nDespite the concerns mentioned above, I think the proposed approach is an interesting addition to the task-oriented online fine-tuning literature, especially considering its practical benefits for AutoML systems. I also think the paper could be considerably improved by taking into account the comments made above. Overall, I’m leaning to reject (5: Marginally below acceptance threshold), but I would be considering to increase my rating if some modifications are made during rebuttal.\n\n***************************************************************************\nPOST REBUTTAL UPDATE\n\nThe authors provided a detailed rebuttal which addressed many of my concerns, and clarified some points. I will update my rating from 5 (marginally below acceptance threshold) to 6 (marginally above acceptance threshold).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PPgCI8vtabG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is the first to address the problem of searching better backbone architectures for downstream tasks and online hyper-parameter selection.  The whole system proposed in this algorithms offers an end-to-end solution for producing a well-trained architecture for a specific downstream task within a fixed training budget. I believe this system will have a large impact in industry model deployment. \nThis paper separate the overall searching process into (1) generating efficient training model zoo (pretrained on a source dataset) (2) task-oriented fine-tuning schedule (Mode selection, hyper-param selection for downstream task fine-tuning).  The search space of the efficient model zoo is decoupled to two stages: block-level search space and macro-level search space. This design could enrich the model zoo with diverse models architectures.   The major part of task-oriented fine-tuning schedule is a performance predictor which is fed with target dataset embedding, model identity and training hyper-params and output an assessment of the final performance. \n\nThrough extensive evaluation experiments, the effectiveness is of this system is demonstrated.  Through well defined ablation studies, the author offers our insightful conclusion of why their systems works: \n1. A better and more diverse model zoo, where  smaller models have simper block structures and larger models have more complicated block structure.\n2. The well-trained performance predictor in task-oriented fine-tuning  scheduler can successfully capture some sort of correlation between bunch of hyper-param and final performance. \n\nSome of my concerns: \n1. The searched architectures are not  directly optimized for down-stream task performance.  \n2. This system cannot address the problem with consistently changing down-stream task, and the cost of training the scheduler for a new downstream task is a little bit huge because of the large pretraining model zoo. \n\nOverall, I recommend this paper to be accepted due to its engineering efforts, several interesting conclusion drawn from empirical study and great impact in industries. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, a network architecture search (NAS) problem in a changing environment is studied and an online adaptation (OA) algorithm for the problem is proposed. Many reviewers found that the OA-NAS problem discussed in this paper is interesting and practically important. However, many reviewers (including those with high review scores) recognize that the weakness of this paper is the lack of sufficient theoretical verification. Furthermore, although extensive experiments are conducted, it is still not clear whether the experimental setups discussed in the paper are generally applicable to other practical problems. Overall, although this is a nice work in that a new practical problem is considered and a workable algorithm for the problem is demonstrated in an extensive simulation study, I could not recommend the acceptance in its current form because of the lack of theoretical validity and evidence of general applicability.
Decision: Reject