NASOA: TOWARDS FASTER TASK-ORIENTED ONLINE FINE-TUNING
Year: 2020

Abstract:
Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks. The common practice of finetuning is to adopt a default hyperparameter setting with a fixed pre-trained model, while both of them are not optimized for specific tasks and time constraints. Moreover, in cloud computing or GPU clusters where the tasks arrive sequentially in a stream, faster online fine-tuning is a more desired and realistic strategy for saving money, energy consumption, and CO2 emission. In this paper, we propose a joint Neural Architecture Search and Online Adaption framework named NASOA towards a faster task-oriented fine-tuning upon the request of users. Specifically, NASOA first adopts an offline NAS to identify a group of training-efficient networks to form a pretrained model zoo. We propose a novel joint block and macro level search space to enable a flexible and efficient search. Then, by estimating fine-tuning performance via an adaptive model by accumulating experience from the past tasks, an online schedule generator is proposed to pick up the most suitable model and generate a personalized training regime with respect to each desired task in a one-shot fashion. The resulting model zoo1 is more training efficient than SOTA NAS models, e.g. 6x faster than RegNetY-16GF, and 1.7x faster than EfficientNetB3. Experiments on multiple datasets also show that NASOA achieves much better fine-tuning results, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various time constraints and tasks; 40x faster compared to the BOHB method.

heading: 1 INTRODUCTION
text: Fine-tuning using pre-trained models becomes the de-facto standard in the field of computer vision because of its impressive results on various downstream tasks such as fine-grained image classification (Nilsback & Zisserman, 2008; Welinder et al., 2010), object detection (He et al., 2019; Jiang et al., 2018; Xu et al., 2019) and segmentation (Chen et al., 2017; Liu et al., 2019). Kornblith et al. (2019); He et al. (2019) verified that fine-tuning pre-trained networks outperform training from scratch. It can further help to avoid over-fitting (Cui et al., 2018) as well as reduce training time significantly (He et al., 2019). Due to those merits, many cloud computing and AutoML pipelines provide fine-tuning services for an online stream of upcoming users with new data, different tasks and time limits. In order to save the user’s time, money, energy consumption, or even CO2 emission, an efficient online automated fine-tuning framework is practically useful and in great demand. Thus, in this work, we propose to explore the problem of faster online fine-tuning.
The conventional practice of fine-tuning is to adopt a set of predefined hyperparameters for training a predefined model (Li et al., 2020). It has three drawbacks in the current online setting: 1) The design of the backbone model is not optimized for the upcoming fine-tuning task and the selection of the backbone model is not data-specific. 2) A default setting of hyperparameters may not be optimal across tasks and the training settings may not meet the time constraints provided by users. 3) With the incoming tasks, the regular diagram is not suitable for this online setting since it cannot memorize and accumulate experience from the past fine-tuning tasks. Thus, we propose to decouple our faster fine-tuning problem into two parts: finding efficient fine-tuning networks and generating optimal fine-tuning schedules pertinent to specific time constraints in an online learning fashion.
1The efficient training model zoo (ET-NAS) has been released at: https://github.com/NAS-OA/ NASOA
Recently, Neural Architecture Search (NAS) algorithms demonstrate promising results on discovering top-accuracy architectures, which surpass the performance of hand-crafted networks and saves human’s efforts (Zoph et al., 2018; Liu et al., 2018a;b; Radosavovic et al., 2019; Tan et al., 2019b; Real et al., 2019a; Tan & Le, 2019; Yao et al., 2020). However, those NAS works usually focus on inference time/FLOPS optimization and their search space is not flexible enough which cannot guarantee the optimality for fast fine-tuning. In contrast, we resort to developing a NAS scheme with a novel flexible search space for fast fine-tuning. On the other hand, hyperparameter optimization (HPO) methods such as grid search (Bergstra & Bengio, 2012), Bayesian optimization (BO) (Strubell et al., 2019a; Mendoza et al., 2016), and BOHB (Falkner et al., 2018) are used in deep learning and achieve good performance. However, those search-based methods are computationally expensive and require iterative “trial and error”, which violate our goal for faster adaptation time.
In this work, we propose a novel Neural Architecture Search and Online Adaption framework named NASOA. First, we conduct an offline NAS for generating an efficient fine-tuning model zoo. We design a novel block-level and macro-structure search space to allow a flexible choice of the networks. Once the efficient training model zoo is created offline NAS by Pareto optimal models, the online user can enjoy the benefit of those efficient training networks without any marginal cost. We then propose an online learning algorithm with an adaptive predictor to modeling the relation between different hyperparameter, model, dataset meta-info and the final fine-tuning performance. The final training schedule is generated directly from selecting the fine-tuning regime with the best predicted performance. Benefiting from the experience accumulation via online learning, the diversity of the data and the increasing results can further continuously improve our regime generator. Our method behaves in a one-shot fashion and doesn’t involve additional searching cost as HPO, endowing the capability of providing various training regimes under different time constraints.
Extensive experiments are conducted on multiple widely used fine-tuning datasets. The searched model zoo ET-NAS is more training efficient than SOTA ImageNet models, e.g. 5x training faster than RegNetY-16GF, and 1.7x faster than EfficientNetB3. Moreover, by using the whole NASOA, our online algorithm achieves superior fine-tuning results in terms of both accuracy and fine-tuning speed, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various tasks; saving 40x computational cost comparing to the BOHB method.
In summary, our contributions are summarized as follows:
• To the best of our knowledge, we make the first effort to propose a faster fine-tuning pipeline that seamlessly combines the training-efficient NAS and online adaption algorithm. Our NASOA can effectively generate a personalized fine-tuning schedule of each desired task via an adaptive model for accumulating experience from the past tasks.
• The proposed novel joint block/macro level search space enables a flexible and efficient search. The resulting model zoo ET-NAS is more training efficient than very strong ImageNet SOTA models e.g. EfficientNet, RegNet. All the ET-NAS models have been released to help the community skipping the computation-heavy NAS stage and directly enjoy the benefit of NASOA.
• The whole NASOA pipeline achieves much better fine-tuning results in terms of both accuracy and fine-tuning efficiency than current fine-tuning best practice and HPO method,e.g. BOHB.

heading: 2 RELATED WORK
text: Neural Architecture Search (NAS). The goal of NAS is to automatically optimize network architecture and release human effort from this handcraft network architecture engineering. Most previous works (Liu et al., 2018b; Cai et al., 2019b; Liu et al., 2018a; Tan et al., 2019a; Xie et al., 2019; Howard et al., 2019) aim at searching for CNN architectures with better inference and fewer FLOPS. Baker et al. (2017); Cai et al. (2018); Zhong et al. (2018) apply reinforcement learning to train an RNN controller to generate a cell architecture. Liu et al. (2018b); Xie et al. (2019); Cai et al. (2019b) try to search a cell structure by weight-sharing and differentiable optimization. Tan & Le (2019) use a grid search for an efficient network by altering the depth/width of the network with a fixed block structure. On the contrary, our NAS focuses creating an efficient training model zoo for fast fine-tuning. Moreover, the existing search space design cannot meet the purpose of our search.
Generating Hyperparameters for Fine-tuning. HPO methods such as Bayesian optimization (BO) (Strubell et al., 2019a; Mendoza et al., 2016), BOHB (Falkner et al., 2018) achieves very promising result but require a lot of computational resources which is contradictory to our original
objective of efficient fine-tuning. On the other hand, limited works discuss the model selection and HPO for fine-tuning. Kornblith et al. (2019) finds that ImageNet accuracy and fine-tuning accuracy of different models are highly correlated. Li et al. (2020); Achille et al. (2019) suggest that the optimal hyperparameters and model for fine-tuning should be both dataset dependent and domain similarity dependent (Cui et al., 2018). HyperStar (Mittal et al., 2020) is a concurrent HPO work demonstrating that a performance predictor can effectively generate good hyper-parameters for a single model. However, those works don’t give an explicit solution about how to perform finetuning in a more practical online scenario. In this work, we take the advantage of online learning (Hoi et al., 2018; Sahoo et al., 2017) to build a schedule generator, which allows us to memorize the past training history and provide up-and-coming training regimes for new coming tasks on the fly. Besides, we introduce the NAS model zoo to further push up the speed and performance.

heading: 3 THE PROPOSED APPROACH
text: The goal of this paper is to develop an online fine-tuning pipeline to facilitate a fast continuous cross-task model adaption. By the preliminary experiments in Section 4.1, we confirm that the model architectures and hyperparameters such as the learning rate and frozen stages will greatly influence the accuracy and speed of the fine-tuning program. Thus, our NASOA includes two parts as shown in the Figure 1: 1) Searching a group of neural architectures with good accuracy and fast training speed to create a pretrained model zoo; 2) Designing an online task-oriented algorithm to generate an efficient fine-tuning regime with the most suitable model under user’s time constraint.

heading: 3.1 CREATING AN EFFICIENT TRAINING MODEL ZOO (ET-NAS) BY NAS
text: The commonly used hand-craft backbones for fine-tuning including MobileNet (Sandler et al., 2018), ResNet (He et al., 2016), and ResNeXt (Xie et al., 2017). Recently, some state-of-the-art backbone series such as RegNet (Radosavovic et al., 2020), and EfficientNet (Tan et al., 2019b) are developed by automated algorithms for higher accuracy and faster inference speed. However, the objective of our NAS is to find a group of models with good model generalization ability and training speed. Suggested by Kornblith et al. (2019), the model fine-tuning accuracy (model generalization ability) has a strong correlation between ImageNet accuracy (r = 0.96). Meanwhile, the training speed can be measured by the step time of each training iteration. Thus, our NAS can be formulated by a multi-objective optimization problem (MOOP) on the search space S given by:
max A∈S
(acc(A),−Ts(A)) subject to Ts(A) ≤ Tm (1)
whereA is the architecture, acc(.) is the Top-1 accuracy on ImageNet, Ts(.) is the average step time of one iteration, and Tm is the maximum step time allowed. The step time is defined to be the total time of one iteration, including forward/backward propagation, and parameter update.
Search Space Design is extremely important (Radosavovic et al., 2020). As shown in Figure 2, to enable an efficient model, we propose a novel flexible joint block-level and macro-level search space to enable simple to complex block design and fine adjustment of the computation allocation on each stage. Unlike existing topological cell-level search space such as DARTS (Liu et al., 2018b), AmoebaNet(Real et al., 2019a), and NASBench101(Dong & Yang, 2019), ours is more compact and avoids redundant skip-connections which have great memory access cost (MAC). Our block-level search space is more flexible to adjust the width, depth (for each stage), when to down-sample/raise the channels. In contrast, EfficientNet only scales up/down the total width and depth by a fixed allocation ratio, and RegNet cannot change the number/type of operations in each block.

heading: Macro-level Search Space
text: Block-level Search Space. We consider a search space based on 1-3 successive nodes of 5 different operations. Three skip connections with one fixed residual connection are searched. Element-wise add or channel-wise concat is chosen to combine the features for the skip-connections. For each selected operation, we also search for the ratio of changing channel size: ×0.25, ×0.5, ×1, ×2, ×4. Note that it can cover many popular block designs such as Bottleneck (He et al., 2016), ResNeXt (Xie et al., 2017) and MB block (Sandler et al., 2018). It consists of 5.4× 106 unique blocks. Macro-level Search Space. Allocation computation over different stages is crucial for a backbone (Liang et al., 2020). Early-stage feature maps in one backbone are larger which captures texture details, while late-stage feature maps are smaller which are more discriminative (Li et al., 2018). Therefore, for macro-level search space, we design a flexible search space to find the optimal channel size (width), depth (total number of blocks), when to down-sample, and when to raise the channels. Our macro-level structure consists of 4 flexible stages. The spatial size of the stages is gradually down-sampled with factor 2. In each stage, we stack a number of block architectures. The positions of the doubling channel block are also flexible. This search space consists of 1.5 × 107 unique architectures. Details of the search space and its encodings can be found in Appendix B.1.
Multi-objective Searching Algorithm. For MOOP in Eq 1, we define architecture A1 dominates A2 if (i) A1 is no worse than A2 in all objectives; (ii) A1 is strictly better than A2 in at least one objective. A∗ is Pareto optimal if there is no otherA that dominateA∗. The set of all Pareto optimal architectures constitutes the Pareto front. To solve this MOOP problem, we modify a well-known method named Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al., 2000) to optimize the Pareto front Pf . The main idea of NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve a group of elite architectures. Then a group of new architectures is sampled and trained by mutation of the current elite architectures on thePf . The algorithm can be paralleled on multiple computation nodes and lift the Pf simultaneously. We modify the NSGA-II algorithm to become a NAS algorithm: a) To enable parallel searching on N computational nodes, we modify the non-dominated-sort method to generate exactly N mutated models for each generation, instead of a variable size as the original NSGA-II does. b) We define a group of mutation operations for our block/macro search space for NSGA-II to change the network structure dynamically. c) We add a parent computation node to measure the selected architecture’s training speed and generate the Pareto optimal models. Details of our NSGA-II can be found in Appendix B.2.2.
Efficient Training Model Zoo Zoo (ET-NAS). By the proposed NAS method, we then create an efficient-training model zoo Zoo named ET-NAS which consists of K Pareto optimal modelsA∗i on Pf . Then A∗i are pretrained by ImageNet. Details of our NAS and A∗i can be found in Appendix B.

heading: 3.2 ONLINE TASK-ORIENTED FINE-TUNING SCHEDULE GENERATION
text: With the help of efficient training Zoo, the marginal computational cost of each user is minimized while they can enjoy the benefit of NAS. We then need to decide a suitable fine-tuning schedule upon the user’s upcoming tasks. Given user’s datasetD and fine-tuning time constraint Tl, an online regime generator G(., .) is desired:
[RegimeFT,A∗i ] = G(D,Tl), such that Acc(AFineTunei , Dval) is maximized, (2)
where the RegimeFT includes all the hyperparameters required, i.e., lr schedule, total training steps, and frozen stages. G(., .) also needs to pick up the most suitable pretrained model A∗i from Zoo. Note that existing search-based HPO methods require huge computational resources and cannot fit
in our online one-shot training scenario. Instead, we first propose an online learning predictor AccP to model the accuracy on the validation set Acc(AFTi , Dval) by the meta-data information. Then we can use the predictor to construct G(., .) to generate an optimal hyperparameter setting and model.
3.2.1 ONLINE LEARNING FOR MODELING Acc(AFTi , Dval)
Recently, Li et al. (2020) suggest that the optimal hyperparameters for fine-tuning are highly related to some data statistics such as domain similarity to the ImageNet. Thus, we hypothesis that we can model the final accuracy by a group of predictors, e.g., model information, meta-data description, data statistics stat(D), domain similarity, and hyperparameters. We list the variables we considered to predict the accuracy result as follows:
ModelA∗i name (one-hot dummy variable) ImageNet Acc. of theA ∗ i
Domain Similiarity to ImageNet (EMD) (Cui et al., 2018) #.Classes Number of Iteration Average #. images per class Std #. images per class Learning Rate Frozen Stages
Those variables can be easily calculated ahead of the fine-tuning. One can prepare offline training data by fine-tuning different kinds of dataset and collect the accuracy correspondingly and apply a Multi-layer Perceptron Regression (MLP) offline on it. However, online learning should be a more realistic setting for our problem. In cloud computing service or a GPU cluster, a sequence of finetuning requests with different data will arrive from time to time. The predictive model can be further improved by increasing the diversity of the data and the requests over time.
Using a fixed depth of MLP model in the online setting may be problematic. Shallow networks maybe more preferred for small number of instances, while deeper model can achieve better performance when the sample size becomes larger. Inspired by Sahoo et al. (2017), we use an adaptive MLP regression to automatically adapt its model capacity from simple to complex over time. Given the input variables, the prediction of the accuracy is given by:
AccP (A∗i ,RegimeFT, stat(D)) = L∑
l=1
αlfl, where l = 0, ..., L (3)
fl = hlWl, hl = RELU(Φlhl−1), h0 = [A∗i ,RegimeFT, stat(D)].
The predicted accuracy is a weighted sum of the output fl of each intermediate fully-connected layer hl. The Wl and Φl are the learnable weights of each fully-connected layer. The αl is a weight vector assigning the importance to each layer and ‖α‖ = 1. Thus the predictor AccP can automatically adapt its model capacity from simple to complex along with incoming tasks. The learnable weight αl controls the importance of each intermediate layer and the final predicted accuracy is a weighted sum of fl of them. The network can be updated by a Hedge Backpropagation (Freund & Schapire, 1999) in which αl is updated based on the loss suffered by this layer l as follows:
α′l ← αlβL(fl,Accgt), W ′l ←Wl − ηαl∇WlL(fl, Accgt)
Φ′l ← Φ′l − η L∑
j=l
αj∇WlL(fj , Accgt), α ′′ l ← α′l∑ α′l
where β ∈ (0, 1) is the discount rate, the weight α′l are re-normalized such that ‖α‖ = 1, and η is the learning rate. Thus, during the online update, the model can choose an appropriate depth by αl based on the performance of each output at that depth. By utilizing the online cumulative results, our generator gains experience that helps future prediction.
Generating Task-oriented Fine-tuning Schedule. Our schedule generator G then can make use of the performance predictor to find the best training regime:
G(D,Tl) = arg maxA∈Zoo,RegimeFT∈SFTAccP (A, RegimeFT , stat(D)).
Once the time constraint Tl is provided, the max number of iterations for different A∗i can be calculated by an offline step-time lookup table for Zoo. The corresponding meta-data variables can be then calculated for the incoming task. The optimal selection of model and hyperparameters is obtained by ranking the predicted accuracy of all possible grid combinations. The Detailed algorithm can be found in the Appendix B.6.

heading: 4 EXPERIMENTAL RESULTS

heading: 4.1 PRELIMINARY EXPERIMENTS
text: We conduct a complete preliminary experiment to justify our motivation and model settings. Details can be found in the Appendix A. According to our experiments, we find that for an efficient finetuning, the model matters most. The suitable model should be selected according to the task and time constraints. Thus constructing a model zoo with various sizes of training-efficient models and picking up suitable models should be a good solution for faster fine-tuning. We also verify some existing conclusions: Fine-tuning performs better than training from scratch (Kornblith et al., 2019) so that our topic is very important for efficient GPU training; Learning rate and frozen stage are crucial for fine-tuning (Guo et al., 2019), which needs careful adjustment.

heading: 4.2 OFFLINE NAS AND MODEL ZOO RESULTS
text: Table 2: Comparsion of our ET-NAS models and SOTA ImageNet models. Inference time and training step time are measured in ms on single Nvidia V100, with bs = 64.
Model Name Top-1 Inf Training Step
Acc. Time (ms) Time (ms)
RegNetY-200MF 70.40 14.25 62.30
ET-NAS-C 71.29 8.94 26.28 RegNetY-400MF 74.10 20.57 90.61
ET-NAS-D 74.46 14.54 36.30 RegNetY-600MF 75.50 22.15 90.11 MobileNet-V3-Large 75.20 16.88 71.65 OFANet 76.10 17.81 73.10
Amoebanet 75.70 28.39 141.45
ET-NAS-E 76.87 25.34 61.95 EfficientNet-B0 77.70 24.30 120.29
RegNetY-800MF 78.00 45.59 170.96
ET-NAS-F 78.80 33.83 93.04 EfficientNet-B2 80.40 58.78 277.60
RegNetY-16GF 80.40 192.78 677.68
ET-NAS-G 80.41 53.08 133.97 ET-NAS-H 80.92 76.80 193.40
EfficientNet-B3 81.50 97.33 455.86
ET-NAS-I 81.38 94.60 265.13 ET-NAS-J 82.08 131.92 370.28 ET-NAS-L 82.65 191.89 542.52
During the NAS, we directly search on the ImageNet dataset(Russakovsky et al., 2015). We first search for a group of efficient block structure, then use those block candidates to conduct the macro-level search. We use a short training setting to evaluate each architecture. It takes about 1 hour on average for evaluating one architecture for the block-level search and 6 hours for the macro-level search. Paralleled on GPUs, it takes about one week on a 64-GPU cluster to conduct the whole search (5K+1K arch). Implementation details and intermediate results can be found in the Appendix B.
Faster Fine-tuning Model Zoo (ET-NAS). After identifying the A∗i from our search, we fully train those models on ImageNet following common practice. Note that all the models including ET-NAS-L can be easily pretrained on a regular 8-card GPU node since our model is training-efficient. We are confident to release our models for the public to reproduce our results from scratch2 and let the public to save their energy/CO2/cost. Due to the length of the paper, we put the detailed encoding and architectures of the final searched models in the Appendix B.4. Surprisingly, we found that smaller models should use simpler structures of blocks while bigger models prefer complex blocks. Comparing our searched backbone to the conventional ResNet/ResNeXt, we find that early stages in our models are very short which is more efficient since feature maps in an early stage are very large and the computational cost is comparably large. This also verified our findings in Appendix B.4.1.
Comparison with the state-of-the-art ImageNet models. We compare the training/inference efficiency of our searched ET-NAS with the SOTA ImageNet models such as MobileNetV3 (Howard et al., 2019), RegNet series (Radosavovic et al., 2020), and EfficientNet series (Tan & Le, 2019)
2The efficient training model zoo (ET-NAS) has been released at: https://github.com/NAS-OA/ NASOA
as shown in Table 2 and Figure 3. Overall, our searched models outperform other SOTA ImageNet models in terms of training accuracy and training speed from Figure 3 (left). Specifically, ET-NAS-G is about 6x training faster than RegNetY-16GF, and ET-NAS-I is about 1.5x training faster than EfficientNetB3. Our models are also better than mobile setting models such as MobileNetV2/V3(Howard et al., 2019) and RegNetY-200MF. Although our model is optimized for fast training, we also compare the inference speed in Figure 3(right). Our models still have a very strong performance in terms of inference speed, outperforming RegNet series and achieving comparable performance with EfficientNet. In Figure 3 (mid), our method is more training efficient than other NAS results, e.g.some evolution-based NAS methods such as AmoebaNet, OFANet.
What makes our network training efficient? To answer this, we define an efficiency score and conduct a statistical analysis of different factors for efficient-training (Details can be found in Appendix B.4.1). We have the following conclusions: a) By observing optimal A∗i , smaller models should use simpler blocks while bigger models prefer complex blocks. Using the same block structure for all sizes of models (Tan & Le, 2019; Radosavovic et al., 2020) may not be optimal. b) Adding redundant skip-connections which have great memory access cost will decrease the training efficiency of the model thus existing topological cell-level search space such as DARTS (Liu et al., 2018b), AmoebaNet (Real et al., 2019a), and NASBench101 (Dong & Yang, 2019) is not efficient. c) The computation allocation on different stages is crucially important. Simply increasing depth/width to expand the model as Tan & Le (2019) may not be optimal will downgrade the performance. To conclude our novel
joint search space contributes most to the training efficiency.
4.3 RESULTS FOR ONLINE ADAPTIVE PREDICTOR AccP
Experimental Settings. We evaluate our online algorithm based on ten widely used image classification datasets, that cover various fine-tuning tasks as shown in Table 1. Five of them (in bold) are chosen to be the online learning training set (meta-training dataset). 30K samples are collected by continually sampling a subset of each dataset and fine-tuning with a randomized hyperparameters on it. Each subset varies from #. classes and #. images. The variables in Section 3.2.1 are calculated
accordingly. The fine-tuning accuracy is then evaluated on the test set. Then 30K sample is split into 24K meta-training samples and 6K meta-validation samples.
Then an adaptive MLP regression in Eq 3 are used to fit the data and predict the Acc(AFTi , Dval). We use L = 10 with 64 units in each hidden layer. We use a learning rate of 0.01 and β = 0.99. As baselines, we also compare the results of using fixed MLP with plain backpropagation with different layers(L = 3, 6, 10, 14). MAE (mean absolute error) and MSE (mean square error) are performance metrics to measure the cumulative error with different segments of the tasks stream.
Comparison of online learning method. The cumulative error obtained by all the baselines and the proposed method to predict the fine-tuning accuracy is shown in Table 4. It can be seen that our adaptive MLP with hedge backpropagation is better than fixed MLP in terms of the cumulative error of the predicted accuracy. Our method enjoys the benefit from the adaptive depth which allows faster convergence in the initial stage and strong predictive power in the later stage.
4.4 FINAL NASOA RESULTS
To evaluate the performance of our whole NASOA, we select four time constraints on the testing datasets and use AccP (.) to test the fine-tuning accuracy. The testing datasets are MIT67, Food101, Aircrafts, Blood-cell, and Stanford-Car. The shortest/longest time constraint are the time of fine-tuning 10/50 epochs for ResNet18/ResNet101. The rest are equally divided into the log-space. For our NASOA, we generate the fine-tuning schedules by maximizing the predicted accuracy in Eq 2. We also conduct finetuning on various candidates of baselines such as ResNet (R18 to R152), RegNet (200MF to 16GF), and EfficientNet (B0 to B3) with the default hyperparameter setting in Li et al. (2020).
Comparison of the final fine-tuning results with the SOTA networks. We plot the time versus accuracy comparison in Figure 5. As can be seen, the model and
training regime generated by our NASOA can outperform the upper bound of other methods in most cases. On average, our methods can improve around 2.1%/7.4% accuracy than the best model of RegNet/EfficientNet series under various time constraints and tasks. It is noteworthy that our NASOA performs better especially in the case of short time constraint, which demonstrates that our schedule generator is capable provide both efficient and effective regimes for fast fine-tuning.
Comparison of the final fine-tuning results with the HPO methods. In Table 4, we compare our method with the HPO methods which optimizing the hyperparameters and picking up models in ResNet, RegNetY, and EfficientNet series. “HPO only” means the method only optimizes the hyperparameters with a fixed model RegNetY-16GF.“OA only” is our online schedule generator with RegNet series models. “Our Zoo” means using our ET models zoo to find suitable model. “Fixed MLP Predictor” is the offline baseline with fixed MLP predictor (L=10) with our model zoo. “Our NASOA” is the our whole pipeline with both training effi-
cient model zoo and online adaptive scheduler. Comparing to the offline baseline with our NASOA, our online adaption module can boost the average performance by 2.17%. It can also be found that our method can save up to 40x computational cost compared to HPO methods while reaching similar performance. With more computational budget, our model zoo with BOHB search can reach even higher accuracy (+avg. 10.28%).
Ablative interpretation of performance superiority. Table 5 calculates the average fine-tuning accuracy over tasks. Our NAS model zoo can greatly increase the fine-tuning average accuracy from 77.17% to 87.45%, which is the main contribution of the performance superiority. Using our online adaptive scheduler instead of BOHB can significantly reduce the computational cost (-40x).

heading: 5 CONCLUSION
text: We propose the first efficient task-oriented fine-tuning framework aiming at saving the resources for GPU clusters and cloud computing. The joint NAS and online adaption strategy achieve much better fine-tuning results in terms of both accuracy and speed. The searched architectures are more training-efficient than very strong baselines such as RegNet and EfficientNet. Our experiments on multiple datasets show our NASOA achieves 40x speed-up comparing to BOHB. The proposed NASOA can be well adapted to more tasks such as detection and segmentation in the future.

heading: A PRELIMINARY EXPERIMENTS

heading: A.1 EXPERIMENTS SETTINGS
text: The preliminary experiments aim at figuring out what kinds of factors impact the speed and accuracy of fine-tuning. We fine-tune several ImageNet pretrained backbones on various datasets as shown in Table 6 (right) and exam different settings of hyperparameters by a grid search such as: learning rate (0.0001, 0.001, 0.01, 0.1), frozen stages (-1,0,1,2,3), and frozen BN (-1,0,1,2,3). Frozen stages/frozen BN= k means 1 to kth stage’s parameters/BN statistics are not updated during training. The training settings most follow Li et al. (2020) and we report the Top-1 validation accuracy and training time. Its detailed experiment settings are hyperparameters are listed as follows:
Comparing fine-tuning and training from scratch. We use ResNet series (R-18 to R-50) to evaluate the effect of fine-tuning and training from scratch. Following Li et al. (2020), we train networks on Flowers102, CUB-Birds, MIT67 and Caltech101 datasets for 600 epochs for training from scratch and 350 epochs for fine-tuning to ensure all models converge on all datasets. We use SGD optimizer with an initial learning rate 0.01, weight decay 1e-4, momentum 0.9. The learning rate is decreased by factor 10 at 400 and 550 epoch for training from scratch and 150, 250 epoch for fine-tuning.
Optimal learning rate and frozen stage. We perform a simple grid search on Flowers102, Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets with ResNet50 to find optimal learning rate and frozen stage on different datasets with the default fine-tune setting in Li et al. (2020). The hyperparameters ranges are: learning rate (0.1, 0.01, 0.001, 0.0001), frozen stage (-1, 0, 1, 2, 3).
Comparing different frozen stages and networks along time. We fix different stages of ResNet50 to analyze the influence of different frozen stages to the accuracy and along the training time on Flowers102, Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets. We pick
the training curves on CUB-Birds and Caltech101 to in the main text of this paper. We also compare the fine-tune results along time with various networks on these datasets as shown in Figure 7. On Caltech101, ResNet50 dominates the training curve at the very beginning. However, on other datasets, ResNet18 and ResNet34 can perform better then ResNet50 when the training time is short.

heading: A.2 FINDINGS OF THE PRELIMINARY EXPERIMENTS
text: With those preliminary experiments, we summarize our findings as follows. Some of the findings are also verified by some existing works.
• Fine-tuning performs always better than training from scratch. As shown in Table 6, finetuning shows superior results than training from scratch in terms of both accuracy and training time for all the datasets. This finding is also verified by Kornblith et al. (2019). Thus, fine-tuning is the most common way to train a new dataset and our framework can be generalized to applications.
• We should the optimize learning rate and frozen stage for each dataset. From Table 7, it seems that the optimal learning rate and optimal frozen stage found by grid search are different for various datasets. Figure 6also shows that the number of the frozen stages will affect both training time and final accuracy. Guo et al. (2019) also showed that frozen different stages are crucial for fine-tuning task. Those two hyperparameters should be optimized for different datasets.
• Model matters most. Suitable model should be selected according to the task and time constraints. Figure 6 (right) suggests that always choosing the biggest model to fine-tune may not be an optimal choice, smaller model can be better than the bigger model if the training time is limited. On the other hand, it is also important to consider the training efficiency of the model since a better model can be converged faster by a limited GPU budget. For example, Figure 7 shows that if the time constraint is short, we should choose a smaller network i.e. ResNet18 here. Thus, it is urgent to construct a training-efficient model zoo.
• BN running statistics should not be frozen during fine-tuning. We found that frozen BN has a very limited effect on the training time (less than ±5%), while not freezing BN will lead to better results in almost all the datasets. Thus, BN is not frozen all experiments for our NASOA.

heading: B DETAILS OF THE OFFLINE NAS

heading: B.1 SEARCH SPACE ENCODINGS
text: The search space of our architectures is composed of block and macro levels, where the former decides what a block is composed of, such as operators, number of channels, and skip connections, while the latter is concerned about how to combine the block into a whole network, e.g., when to do down-sampling, and where to change the number of channels.

heading: B.1.1 BLOCK-LEVEL ARCHITECTURE
text: Block-level design. A block consists of at most three operators, each of which is divided into 5 species and has 5 different number of output channels. Each kind of operator is denoted by an op number, and the output channel of the operator is decided by the ratio between it and that of the current block. Details are shown in Table B.1.1. By default, there is a skip connection between the input and output of the block, which sums their values up. In addition to that, at most 3 other skip connections are contained in a block, which either adds or concatenates the values between them. Each operation is followed by a batch normalization layer, and after all the skip connections are calculated, a ReLU layer is triggered.
Block-level encoding. The encoding of each block-level architecture is composed of two parts separated by ‘-’, which considers the operators and skip connections respectively.
For the first part (operators part), each operator is represented by two numbers: op number and ratio number (shown in Table B.1.1). As the output channel of the last operation always equals to that of the current block, the ratio number of this operator is removed. Therefore, the encoding of a block with n operators always has length 2n− 1 for the first part of block-level encoding. For the second part (skip connections part), every skip connection consists of one letter for addition (‘a’) / concatenation (‘c’), and two numbers for place indices. n operators separate the block to n+1 parts, which are indexed with 0, 1, 2, . . . , n. Thus ‘a01’ means summing up the value before and after the first operator. Since the skip connection between the beginning and end of the block always exists, it is not shown in the encoding. Thus this part has length 3k − 3 (possibly 0) when there is k skip connections. Some of the encoding examples are shown in Figure 8.

heading: B.1.2 MACRO-LEVEL ARCHITECTURE
text: Macro-level design. We only consider networks with exactly 4 stages. The first block of each stage (except Stage 1) reduces the resolution of both width and height by half, where the stride 2 is added to the first operator that is not conv1x1. Other blocks do not change the resolution. One block’s output channel is either the same, or an integer multiple of the input channel.
Macro-level encoding. The 4 stages are divided apart by 3 ‘-’ signs. For every stage, each block is represented by an integer, which shows the ratio between output and input channel for this block.

heading: B.1.3 ENCODING AS A WHOLE
text: Thus the whole backbone can be encoded by simply concatenating the block and macro encoding. The encoding of the whole network is formatted as:
{Block ENCODING} {First CHANNEL} {Macro ENCODING} Some common architectures, including ResNet and Wide ResNet can be accurately represented by our encoding scheme, which is shown in Table B.1.3.

heading: B.2 NAS SEARCH ALGORITHM

heading: B.2.1 NON-DOMINATED SORTING ALGORITHM
text: Non-dominated sorting is mainly used to sort the solutions in population according to the Pareto dominance principle, which plays a very important role in the selection operation of many multiobjective evolutionary algorithms. In non-dominated sorting, an individual A is said to dominate another individual B, if and only if there is no objective of A worse than that objective of B and there is at least one objective of A better than that objective of B. Without loss of generality, we assume that the solutions of a population S can be assigned to K Pareto fronts Fi, i = 1, 2, . . . ,K. Non-dominated sorting first selects all the non-dominated solutions from population S and assigns them to F1 (the rank 1 front); it then selects all the non-dominated solutions from the remaining solutions and assigns them to F2 (the rank 2 front); it repeats the above process until all individuals have been assigned to a Pareto front.

heading: B.2.2 NSGA-II: ELITIST NON-DOMINATED SORTING GENETIC ALGORITHM
text: To solve the problem in Eq. 1, Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al., 2000) is adopted to optimize the Pareto front Pf as shown in Algorithm 1. In this paper,
we choose this kind of sample-based NAS algorithm instead of many popular parameter-sharing NAS method. This is because we want to further analysis of the sampled architectures and achieve insights and conclusions of the efficient training. The main idea of NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve a group of elite architectures. Then a group of new architectures is sampled and trained by mutation of the current elite architectures on the Pf . The algorithm can be paralleled on multiple computation nodes and lift the Pf simultaneously. The mutation in the block-level search space includes adding new skip-connection, modifying the current operations and ratios. Meanwhile, the mutation in the macro-level search space includes randomly adding or deleting one block in one stage, exchanging the position of doubling channel block with its neighbor, and modifying the base channels. This well-known NSGA-II is easy to implement and we can easily monitor the improvement of each iteration. The stop criterion depends on the time limit or the computation cost constraints.
Algorithm 1 Our modified NSGA-II Searching Algorithm Input Stop criterion, Search Space, number of computation nodes N.
1: t = 0 2: Pt ← Random(A), generate a group of initial architectures. 3: Evaluate Pt 4: while not stop criterion do 5: Apply non-dominated sorting to Pt to obtain non-dominated fronts A∗i 6: Sort A∗i by Crowding distance and left top-N A ∗ j as Parents 7: Create new generation Qt by mutation on current A∗j 8: Train the Qt on N computation nodes and Evaluate the accuracy of Qt. 9: Pt+1 ← Qt ∪ Pt
10: t = t+ 1 11: end while
Output The Pareto Optimal Front A∗i

heading: B.3 NAS IMPLEMENTATION DETAILS

heading: B.3.1 BLOCK-LEVEL SEARCH
text: In the phase of block-level search, a proxy task of ImageNet is created, which is a subset sampled fromits training set. This subset constitutes 100 labels, each of which has 500 images as the training set, and 100 as the validation set. We call this dataset ImageNet-100 in the following parts of this paper.
To avoid interference with macro architecture, the macro-level architecture is fixed to be the same as that of ResNet50. Each model is trained by ImageNet-100 with a batch size of 32 for 90 epochs and learning rate 0.1, which takes 3˜10 hours on a single NVIDIA Tesla-V100 GPU. We do a random search at first, which uniformly samples all the valid blocks in the search space. Evolutionary Algorithm (EA) is then performed with three kinds of mutations: 1) replace one operator with another; 2) change the output channel of one layer; 3) Add/remove/modify a skip connection. We keep updating the Pareto Front between step time and accuracy during the whole process. As a
result, 10 blocks are selected as the candidates for the following rounds. Practically, during our search, the performance of early stop models aligns well with the fully-train accuracy. We checked the Spearman Rank Correlation for 103 architectures: ρ = 96.6%. Thus, using early stop can greatly reduce the search cost by around 90% while keeping our NAS effective.

heading: B.3.2 MACRO-LEVEL SEARCH
text: We search the block-level architectures with the 10 blocks attained by block-level search. Random search is adopted at first, where the number of blocks is chosen randomly between 10 and 50, and the first and last channel is drawn from {32, 64, 128} and {512, 1024, 2048} respectively. EA search is then applied, where the mutations allowed are: 1) Add a ‘1’; 2) Remove a ‘1’; 3) Swap two different adjacent numbers. Similar to block-level search, Pareto Front between step time and accuracy is also kept updated.
Different from the previous phase, the whole ImageNet dataset is utilized for training. Each model is trained with a batch size of 1024 and a learning rate of 0.2 for 40 epochs.

heading: B.4 ET-NAS: MODEL ZOO

heading: INFORMATION AND THEIR ENCODINGS
text: After the NAS process done in Section B.3, 12 models are selected as our model zoo ET-NAS of fine-tuning. Details of these models are shown in Table B.4. The inference time and step time are measured in ms on a single Nvidia V100, with batch size of 64. The resolution follows the standard setting of ImageNet: 224x224. By observing the optimal model in the table, smaller models should use simpler blocks while bigger models prefer complex blocks.
Figure 10 shows the comparison of our ET-NAS models with other SOTA ImageNet models. Inference time and training step time are measured in ms on a single Nvidia V100, with bs = 64. Our ET-NAS series show superior performance comparing to RegNet, EfficientNet series. Comparing to some EA-based NAS methods such as OFANet and Amoebanet, our method is also efficient in terms of training. We found that there exists a performance ranking gap between inference time and training step time in Figure 10. This is mainly due to the depth and the main type of operation of the models. We found that deeper networks with separable conv such as EfficientNet/MobileNet have a larger training-step-time/inference-time ratio comparing to our models (shallower&more common conv).

heading: B.4.1 WHAT MAKES A NETWORK EFFICIENT-TRAINING?
text: To answer this question, we first need to define a score for the efficiency of the searched modelsA. In MOOP, the goodness of a solution is determined by dominance. Thus, we can use the non-dominated sorting algorithm to sort the A according to the Pareto dominance principle. Each architecture is assigned to one Pareto front and the rank RP of that Pareto front can be regarded as the goodness of a solution, in our case, the efficiency. We then defined the efficiency score of A as: sE(A) = −RP(A)−mean(RP(A))std(RP(A)) . Since Pareto optimal front is the Rank 1 Pareto front, larger efficiency score sE(A) means better efficiency. Then we perform a multivariate linear regression analysis on the AS . According to our search space, ordinal/nominal variables that describe the model are denoted as predictors to fit the sE(A). Table 11 shows the coefficients from the regression analysis on both block-level and macro-level
designs. Positive coefficients indicate a positive relationship. “P-Value” shows the significance of the variables. We summarize and highlight several noteworthy conclusions uncovered by our analysis:
• By observing optimal A∗i , smaller models should use simpler blocks while bigger models prefer complex blocks. Simply increasing depth/width to expand the model in Tan & Le (2019) may not be optimal. • Adding additional skip connections will decrease the training efficiency of the model (The Coef is significantly negative). Using “add” to combine the features is more efficient than “concat”. • Using “conv3x3, w group=4” is the best operation among the searched operations (Coef is 0.295). Separable conv3x3 is not efficient for training (Coef is -0.2). • The first double-channel position should be more close to the beginning of the network, while the final double channel-position should be delayed to the end of the network. • Fewer blocks should be assigned to the first two stages. More should be assigned to the 3rd stage.
B.5 CO2 CONSUMPTION ANALYSIS
Fine-tuning from the pretrained ImageNet/language model is a de-facto practice in the deep learning field (CV/NLP). Our NASOA improves the efficiency of fine-tuning which has the potentials to greatly reduce computational cost in GPU clusters/cloud computing. According to a recent study (Strubell et al., 2019b), developing and tuning for one typical R&D project (Strubell et al., 2018) in Google Cloud computing needs about $250k cost, 82k kWh electricity, and 123k lbs CO2 emission, which equals to the CO2 consumption of air traveling (NY↔SF) 62 times. Among most of them, 123 hyperparameter grid searches were performed for new datasets, resulting in 4789 jobs in total. It is believed that the proposed faster fine-tuning pipeline can save up to 40x computational cost among them. Furthermore, we have released all the searched efficient models to help the public skipping the computation-heavy NAS stage and directly enjoy the benefit of our methods. In conclusion, our NASOA is meaningful for environment protection and energy saving.

heading: B.6 DETAILED ALGORITHMS OF NASOA
text: Detailed algorithms of our Model Zoo (ET-NAS) search can be found in Algorithm 2. The pseudo code of online fine-tuning schedule generator training, prediction, and update can be found in Algorithm 3.
Algorithm 2 Efficient Training Model Zoo (ET-NAS) Creation Input: Block/Macro Search Space Si,Sa, Stop Criterion Γ, #Computation Nodes K, Sensitive
Factor , #Block Architectures M , #Models in Model Zoo N . Output: Final Model Zoo Zoo
1: procedure BLOCKSEARCH(Si,Γ,K, ,M ) 2: Pf ← NSGA-II(Γ, Si,K, ) . Our modified NSGA-II, see Algorithm 1 3: Cells← MOSTCOMMON(Pf ,M ) . Most common M cells from Pf 4: end procedure 5: procedure MACROSEARCH(Sa,Γ,K, ,N ) 6: Pf ← NSGA-II(Γ, Sa(Cells),K, ) 7: Zoo ← NSGASORT(Pf , N, ) . Choose models based on crowding-distance 8: end procedure
C IMPLEMENT DETAILS OF HPO METHODS
We use BOHB and random search in our experiments as the HPO baseline. As stated in Section 4.4, The shortest/longest time constraint (budget) is defined as the time of fine-tuning 10/50 epochs for ResNet18/ResNet101 and the rest are equally divided into the log-space, which can be represented as: tx = t0 ∗ (t3/t0)x/t3 , where x = [0, 1, 2, 3], t0is the time to train ResNet18 for 10 epochs and t3 is the time to train ResNet101 for 50 epochs.
We only compare the HPO setting under the same max computational budgets equal to t1 in Table 5 (left). For random search, we randomly sample candidates from predefined search space until
Algorithm 3 Online Fine-Tuning schedule Generator Training, Prediction, and Update Input: Model Zoo Zoo, Time Evaluator Ts, Acc Evaluator Tr, Hyper-parameter Space SHP , Known Datasets Dold, New Dataset Dnew, #Meta-data HM , Time Constraint Tl, #Configurations H . Output: Optimal Model A∗, Hyper-parameters Regime∗FT , Predictor AccP 1: procedure OFFLINETRAINING(Zoo, Tr,SHP , Dold, HM ) 2: MetaData← ∅, AccP ← ADAPTIVEMLP(.) . Initialize default predictor 3: for D ∈ Dold, i← 1 to HM do 4: A, RegimeFT ← RANDOM(Zoo, SHP ) . Randomly select from search space 5: Acc← Tr(D,A, RegimeFT ) . Train with selected configuration 6: MetaData←MetaData ∪ {(A, RegimeFT , Acc)} . Add this result to meta-data 7: end for 8: AccP ← TRAIN(AccP ,MetaData) . Train predictor with all meta-data 9: end procedure 10: procedure ONLINEPREDICTION(Zoo, Dnew, AccP , Tl, Ts, H) 11: MetaData← ∅ 12: for i← 1 to H do 13: A ← RANDOM(Zoo) 14: Epoch← Tl÷ Ts(A, Dnew) . Always choose the largest epoch within Tl 15: RegimeFT ← RANDOM(SHP |Epoch) . Randomly select condition on Epoch 16: MetaData←MetaData ∪ {(A, RegimeFT )} 17: end for 18: A∗, Regime∗FT ← PREDICT(G,MetaData) . Choose the optimal from H configs 19: Acc← Tr(Dnew,A∗, Regime∗FT ) 20: AccP ← TRAIN(AccP , {A∗, Regime∗FT , Acc}) . Improve predictor with this meta-data 21: end procedure
reaching the max computational budget. And for BOHB, we use the opensource implementation of BOHB at https://github.com/automl/HpBandSter. We set random fraction=0.3, percent of good observations=15%, min budget=25% and max budget=100% with respect to our max computational budget. And we use BOHB with 40x computational cost than our proposed methods with different model zoos. The results are presented in Section 4.4.

Review summaries:
In this paper, a joint Neural Architecture Search and Online Adaption (NASOA) framework is proposed to achieve a faster task-oriented fine-tuning upon the request of users. In particular, two main contributions are made in this paper: (1) A fine-tuning pipeline that seamlessly combines the training-efficient NAS and online adaption algorithm is introduced, which can effectively generate a personalized fine-tuning schedule of each desired task via an adaptive model for accumulating experience from the past tasks. (2) A block-level and macro-level search space is introduced in the resulting framework, which enables a simple to complex block design and fine adjustment of the computation allocation on each stage.

This works aims at task-oriented fine-tuning from pre-trained ImageNet models. It proposes a Neural Architecture Search and Online Adaption framework (NASOA) to perform fast task-oriented model fine-tuning. The NASOA first employ an offline NAS to select a group of models and then pick up the most suitable model from this group via an online schedule generator.