1

Two-Stage Evolutionary Neural Architecture Search
for Transfer Learning
Yu-Wei Wen, Sheng-Hsuan Peng, and Chuan-Kang Ting

Abstract—Convolutional Neural Networks (CNNs) haveachieved state-of-the-art performance in many image classification tasks. However, training a deep CNN requires a massive
amount of training data, which can be expensive or unobtainablein practical applications such as defect inspection and medical
diagnosis. Transfer learning has been developed to address thisissue by transferring knowledge learned from source domains totarget domains. A common approach is fine-tuning, which adapts
the parameters of a trained neural network for the new target
task. Nevertheless, the network architecture remains designed
for the source task rather than the target task. To optimize the
network architecture in transfer learning, we propose a two-stage
evolutionary neural architecture search for transfer learning(EvoNAS-TL), which searches for an efficient subnetwork ofsource model for the target task. EvoNAS-TL features two search
stages: structure search and local enhancement. The former
conducts a coarse-grained global search for suitable neuralarchitectures, while the latter acts as a fine-grained local search
to refine the models obtained. In this study, neural architecture
search (NAS) is formulated as a multiobjective optimization problem that concurrently minimizes the prediction error and modelsize. The knee-guided multiobjective evolutionary algorithm, amodern multiobjective optimization approach, is employed tosolve the NAS problem. In this study, several experiments are
conducted to examine the effectiveness of EvoNAS-TL. The
results show that applying EvoNAS-TL on VGG-16 can reducethe model size by 52%–85% and simultaneously improve thetesting accuracy by 0.7%–6.9% in transferring from ImageNet
to CIFAR-10 and NEU surface detection datasets. In addition,
EvoNAS-TL performs comparably to or better than state-of-theart methods on the CIFAR-10, NEU, and Office-31 datasets.
Index Terms—Transfer learning, neural architecture search,
multiobjective evolutionary algorithm, convolutional neural network.

I. I NTRODUCTION
Convolutional Neural Networks (CNNs) have been widely
applied to various image classification tasks. Deeper or more
complex CNNs can usually achieve higher accuracy [1].
However, training a deep CNN ordinarily requires a substantial
amount of training data, which could be unavailable in some
applications due to the expensive cost or other potential
difficulties in data collection. Transfer learning renders aneffective way to address this issue by applying the knowledge
learned from source tasks to target tasks [2], [3]. This machinelearning paradigm has improved the prediction accuracy in
low-resource image classification tasks [4]. Transfer learningtechniques involve fine-tuning, a loss function, and feature
Yu-Wei Wen is with the Department of Computer Science and Information Engineering, National Chung Cheng University, Taiwan. ShengHsuan Peng and Chuan-Kang Ting are with the Department of Power
Mechanical Engineering, National Tsing Hua University, Taiwan (e-mail:ckting@pme.nthu.edu.tw). (Corresponding author: Chuan-Kang Ting)

mapping [2]–[4]. The network architecture plays a crucialrole in the performance of neural networks; however, it receives less attention in transfer learning. Transfer learning
methods typically use the same network architecture for boththe source task and the target task, which may cause overparameterization when the target task has only a small amountof data. Moreover, the network architecture designed originallyfor the source task is not optimized and is likely unsuitable
for the target task [5]–[14]. The transferred networks, consequently, can suffer from over-parameterization and consume
much more computational cost and storage than necessary.
Neural architecture search (NAS) aims to optimize the
network architecture for machine learning tasks. Throughrepeatedly generating and testing various architectures, NAS
algorithms search for architectures with better performance
than those handcrafted by humans. Nevertheless, NAS can betime-consuming due to its repeated evaluation of generated
network architectures. In addition, it is commonly assumed
that the amount of data is sufficient to train the network, but
this amount is hardly attainable in many real-world applications. Some studies have attempted to combine NAS withtransfer learning. These methods transfer the architectures
discovered in the source task to the target task [15]–[18]or transfer the NAS controllers or architecture performance
predictors to facilitate NAS in new tasks [19]–[22]. Althoughreusing trained parameters has been shown to benefit learninga new task [2], [4], [5], [12], only a few studies transfer theparameters trained in the source model to the target task [23].
This study presents an evolutionary system, two-stage
evolutionary neural architecture search for transfer learning(EvoNAS-TL), to address the above issues of transfer learning.
The proposed EvoNAS-TL enables NAS in transfer learning toadapt network architecture and parameters jointly for the target
task. Restated, EvoNAS-TL transfers parameters from thesource model and searches for the optimal neural architecture
for the target task. The transfer of parameters, in addition to thearchitecture, exploits the parameter-related knowledge learned
from the source model while searching architectures for thetarget task. Based on this novel integration, EvoNAS-TL canenhance the effectiveness of the transfer and generate sizeefficient neural architectures for the target task. To reduce
the high computational cost of the extensive network architecture search, EvoNAS-TL confines the NAS search space tosubnetworks of the source model and focuses on seeking thesubnetwork that would achieve the best performance on thetarget task. EvoNAS-TL includes two stages of subnetwork
search: structure search and local enhancement. The former
aims to manipulate the source model layerwise, whereas the2latter proceeds to remove unnecessary operation units from thetransferred network. Both optimization stages are formulated
as a multiobjective optimization problem (MOP), of which theobjectives are minimization of the prediction error and modelsize. The knee-guided multiobjective evolutionary algorithm(KGEA) [24] is adopted to solve the MOP in both stages.
Moreover, we propose two representations for the structure
search: sequential representation and selective representation.
The two representations allow KGEA to reshape the source
model in different levels of detail. The proposed EvoNASTL is tested on three transfer learning scenarios, where thefirst two use ImageNet [25] as the source task and adopt
the CIFAR-10 dataset [26] and the NEU surface defectdatabase [27] as the target tasks, whereas the third scenariotransfers across domains on the Office-31 dataset [28].
The major contributions of this paper are summarized asfollows:
1) A two-stage multiobjective evolutionary NAS system isdeveloped to optimize the model structure for the target
task in transfer learning.
2) The first stage (structure search) serves as a coarsegrained global search, seeking suitable neural architectures for the target tasks and transferring knowledge from the source model to the candidate models.
Sequential representation and selective representation
are proposed to encode different levels of structural
information.
3) The second stage (local enhancement) acts as a finegrained local search, which refines the model obtained
from the structure search through evolutionary network
pruning.
4) An empirical study is conducted to examine the performance of EvoNAS-TL. The experimental results showthat EvoNAS-TL effectively improves the classification
error and reduces the model size in transfer learning.
The remainder of this paper is organized as follows. Section II
reviews the related studies on transfer learning and NAS.
Section III elaborates on the proposed EvoNAS-TL system.
Section IV presents the experimental settings and results.
Finally, concluding remarks are given in Section V.
II. R ELATED W ORK
The architecture of CNNs has become increasingly complex
to improve classification performance and deal with complex
image classification tasks [29]. For example, AlexNet [30] uses
an eight-layer architecture with over 60 million parameters,
VGG-16 [31] has a 16-layer architecture with 138 million
parameters, and ResNet [1] reaches more than 100 layers.
Transfer learning has shown to be capable of addressing thepreviously mentioned issues with low training resources byleveraging the prelearned knowledge from one task to anothertask [2]–[5], [32], as evidenced by practical applications,such as defect inspection [12] and medical diagnosis [8].
Some existing transfer learning methods focus on adapting
the source model to the target task through parameter tuning.
Although post-transfer pruning of neural architecture has been
proposed [33], [34], the transfer and pruning processes in thesestudies are performed separately without considering theirinteraction. Thus, the challenge of finding a suitable neuralarchitecture for transfer learning tasks remains.
NAS deals with the automatic design of network architectures that are suitable for a given machine learning task. Modern NAS systems commonly use evolutionary algorithms [35]–[37] and reinforcement learning [19], [38] to search network
structures. However, the generate-and-test NAS systems usually require enormous computational resources for the search
process, as it involves repeatedly training candidate models
from scratch [17], [38], [39]. For example, Zoph et al. [38]used 800 GPUs over 21–28 days for a single task.
Some approaches have been developed to reduce the computational cost of NAS systems by restricting the search
space [16], [17], [40], [41] and parameter sharing [16], [41],[42]. Zoph et al. [17] proposed searching only computational
cells rather than the entire network because the intra-cellstructure forms a smaller search space. After optimizing thecells, human experts need to arrange and stack the cells to
obtain a near-optimal network structure. Similar to this idea,
Xie and Yuille [43] attempted to apply the cell structurelearned from CIFAR-10 to ImageNet. The cells are stacked
in a more complicated way, but the resultant model achieves
no improvement in classification accuracy over the networks
designed by human experts. For parameter sharing, NAS
is formulated to search the optimal subgraph of a massivemodel represented by a directed acyclic graph. Bayesianlearning [44] and gradient-based learning [16], [17], [39],[41], [42] have been used in subgraph search. Sun et al. [45]proposed a fully automated evolutionary NAS system, where
NAS is formulated as a combinatorial problem based on
existing convolutional blocks, for example, residual blocks anddense blocks. This approach is effective in improving searchefficiency. Nonetheless, these types of methods still requiremassive amounts of data to train the large prototype model,
which can be unattainable in real-world applications.
Because of the respective issues of transfer learning andNAS, certain methods have been proposed to integrate them,seeking solutions to enable both the exploitation of architectural information in architectural optimization for the target
task and the reuse of learned knowledge from the source task
to the target task. Baker et al. [15] proposed transferring thearchitecture discovered on a source task along with pretrained
weights to the target task with parameter fine-tuning. Thisis one of the methods that perform NAS and transfer learning separately. Wong et al. [19] developed a reinforcement
learning-based NAS system in which the searching controller
is pretrained on a set of tasks and reused for new tasks.
Wistuba and Pedapati [22] presented inductive transfer, which
maintains an architecture pool and selects a promising architecture from the pool for a new task. Other studies [20],[21] suggested transferring architecture search experience byreusing the surrogate model trained as a performance predictor
for candidate architecture. However, these techniques center on
the transfer of search experience with no prelearned knowledge
involved.
Network pruning aims to remove redundant parametersfrom a trained model and further prevents overfitting caused

3

6RXUFHPRGHO6WUXFWXUHVHDUFK

6HOHFWLRQIRU
ORFDOHQKDQFHPHQW/RFDOHQKDQFHPHQW

6ROXWLRQVHOHFWLRQ

5HVXOWDQWPRGHOVRI(YR1$67/

Fig. 1. Flowchart of EvoNAS-TL.

by over-parameterization [46]. Pruning mechanisms can beroughly categorized into nonstructural pruning and structural
pruning [47]. Nonstructural pruning considers the removal ofparameters individually, whereas structural pruning also takes
the structural relationship of parameters into account. The
studies on nonstructural pruning face a significant challenge
in that the pruned networks are relatively slow on GPUs.
Although specialized hardware can alleviate the problem [48],implementing the hardware can be expensive and challenging
to general users. In contrast, structural pruning aims to remove
computational units such as filters and neurons. The models
obtained from structural pruning preserve the dimensional
coherence of tensor operations and thus are supported by
off-the-shelf deep learning frameworks. Identifying the filtersand neurons to be removed remains a major challenge inpruning CNNs. Greedy algorithms [49]–[51], reinforcement
learning [52], and group sparsity measurement [53] have been
employed to solve network pruning tasks. These approaches
prune the network step-by-step and tend not to recover pruned
units in later phases. Some studies use evolutionary algorithms
as a global optimizer for pruning [24], [54]. These structural
pruning methods downsize the width of the networks, but the
network depth remains unchanged. Considering that different
tasks may need different model depths, the present study seeks
to simultaneously optimize the network structure both widthwise and depthwise for transfer learning tasks by combining
NAS and structural network pruning.
III. E VOLUTIONARY N EURAL A RCHITECTURE S EARCHFOR T RANSFER L EARNING
The proposed EvoNAS-TL is a two-stage multiobjective
optimization system searching for the subnetworks of a given
source model that are also effective after being transferred tothe target domain. The first stage of EvoNAS-TL performs astructure search on the source model, and the second stage
conducts local enhancement to refine the models throughevolutionary network pruning. The MOP for both stages isformulated as follows:argmin f (M )M ∈Ωf (M ) = {f1 (M ), f2 (M )} .

(1)

The first objective is to minimize the classification error on
the target domain:
f1 (M ) = min E(M , Dtarget ),(2)

where E(·) measures the validation error of model M on thedataset of target domain Dtarget .
The second objective is to minimize the model size. In
this study, the model size is determined by the number ofparameters in the model. The second objective is formulatedbyf2 (M ) = min |M |,(3)
where the cardinality | · | stands for the total number ofparameters in model M .
EvoNAS-TL undertakes the above bi-objective optimizationin both stages. Figure 1 illustrates the workflow of EvoNASTL. First, the structure search stage explores the source model
subnetworks for the structures with the optimal accuracy andmodel size. Second, the local enhancement stage applies evolutionary network pruning to refine the subnetworks obtained
from structure search. More details about the two stages aredescribed in the following subsections.
A. Structure Search
Structure search is designed to seek the subnetworks ofthe source model while considering minimal prediction error

4*HQRW\SH)&RXWSXW

)&¬

)&¬

'LVFDUGHGFRQYROXWLRQDOOD\HUV

3RROLQJ

[&219

[&219

[&219

3RROLQJ

[&219

[&219

[&219

3RROLQJ

[&219

[&219

[&219

3RROLQJ

FRQYROXWLRQDOOD\HUV

[&219

[&219

3RROLQJ

[&219

3KHQRW\SH[&219

3UHVHUYHG

5HFRQVWUXFWHG

7UDQVIHUUHGIURPFig. 2. Mapping between the sequential representation x and its corresponding model with Msource VGG-16.

*HQRW\SH

'LVFDUGHG)L[HG)&RXWSXW

)&¬

)&¬

3RROLQJ¬'LVFDUGHG¬¬
[&219

[&219

[&219

3RROLQJ

[&219

[&219

[&219

3RROLQJ

[&219

[&219
¬'LVFDUGHG¬¬

[&219

3RROLQJ¬'LVFDUGHG¬¬
[&219

[&219

3RROLQJ

[&219

3KHQRW\SH[&219

7XQDEOH

5HFRQVWUXFWHG

7UDQVIHUUHGIURPFig. 3. Mapping between the selective representation x and its corresponding model with Msource VGG-16.

and minimal model size. More specifically, structure searchrepeatedly generates subnetworks originated from the source
model and then reshapes the subnetworks according to theirperformance on the target domain. This process transfers andadapts the source model to the target domain, but its computational cost is generally high because of the considerable
number of evaluations. To address this issue, we propose
two types of representation for candidate solutions of thebi-objective optimization problem: sequential representationand selective representation. Each representation encodes thelayer information that confines the search space to promote
efficiency.
Given a source model based on a CNN,
fc
Msource = {Lconv, . . . , Lconv
, Lfc
1
p
1 , . . . , Lq },Lconv
is the i-th convolutional layer, Lfcij is the j-th fully connected layer, and p and q indicate the numbers of convolutional
layers and fully connected layers, respectively.
1) Sequential Representation: Sequential representation encodes the depth of the convolutional layers and the numberof neurons of each fully connected layer. This representationconsiders the connection between adjacent layers of a trainedmodel and allows for controlling the complexity of the model
without breaking the inferencing coherence in convolutional
layers. According to the sequential representation, a candidate
solution is encoded asfc
x = (xconv , xfc
1 , . . . , xq ),(4)

where xconv ∈ {1, . . . , p} indicates the number of convolutional layers and xfc
i ∈ [0, 1] represents the proportion ofneurons to be used in the i-th fully connected layer. The
candidate model corresponding to a candidate solution x canbe written byfc

fc

Mcand = {Lconv, . . . , Lconv
1
xconv , L̂1 , . . . , L̂q }.

(5)

The transfer process directly copies Lconv, . . . , Lconv
1
xconv fromthe first xconv convolutional layers of source model Msourcefc
fc
and uses N(L̂1 ), ..., N(L̂q ) neurons for the q fully connected
layers, where N(·) gives the number of neurons in the given
fc
fully connected layer. The number N(L̂i ) is computed by
 fc  j mfc
N L̂i = xfc,(6)
i × N Li

5

where b·e rounds its argument to the nearest integer. The
fully connected layers in Mcand are reconstructed in acfc
fc
cordance with N(L̂1 ), ..., N(L̂q ), in which the parameters
are initialized randomly. Then, Mcand is fine-tuned withthe training data of the target domain Dtarget_train ; that is,the convolutional and fully connected layers are tuned. The
validation error and model size of Mcand then serve as theobjective values of x. Figure 2 illustrates the association of thesequential representation with a source model VGG-16 [31].
2) Selective Representation: Selective representation enables subtler manipulation of convolutional layers than sequential representation. Using selective representation, a candidate
solution is encoded asfc
x = (xconv, . . . , xconv
, xfc
1
p
1 , . . . , xq ).

(7)

The selective representation includes two parts: the first partxconv
∈ {0, 1, 2} indicates the operation for the i-th coni
volutional layer, and the second part xfc
j ∈ [0, 1] resembles
the corresponding term in sequential representation, which
accounts for the proportion of neurons used in the j-th fully
connected layer. The first part considers three operations:discarding, fixing, or fine-tuning a convolutional layer:


xconv
=0
discard,
i
fix,
xconv
=1
i


conv
fine-tune, xi=2Figure 3 illustrates the selective representation associated witha VGG-16 source model.
Algorithm 1 outlines the structure search procedure.
EvoNAS-TL uses either of the two representations and applies
KGEA [24] to solve the bi-objective optimization problem
in the structure search. Regarding the evolutionary process,the population is initialized at uniform random. Candidate
solutions, namely offspring, are generated by crossover andmutation. To evaluate a candidate solution x, a model Mcandis constructed with reference to source model Msource and therepresentation of x; then, Mcand is fine-tuned using training
data of target domain Dtarget_train . The performance of x is
assessed by the two objective functions (2) and (3) with thevalidation data of target domain Dtarget_valid . At the end of
the structure search, KGEA yields an approximate Pareto front(APF) formed by the resultant models. EvoNAS-TL selects
three models from the results: 1) the knee solution, 2) thesolution with minimal error, and 3) the solution with minimalmodel size, and it proceeds with local enhancement on eachmodel in the second stage.
B. Local Enhancement
Local enhancement uses evolutionary network pruning [24]on the models obtained from structure search to reduce themodel size and improve the classification performance. For
the latter, several studies (e.g., [55]–[57]) have shown that
network pruning can address the over-parameterization issueof deep neural networks. By removing less relevant units,
network pruning can enhance the network performance andgeneralization; that is, network pruning aims to refine models
by retaining useful units and removing trivial ones.

Algorithm 1 Structure search
Input: pretrained source model Msource ,training data of target domain Dtarget_train ,and validation data of target domain Dtarget_valid
Output: Mnondominated1: Q0 ← Population-initialization
2: M0 ← Model-construction(Q0 , Msource )3: M0 ← Fine-tuning(M0 , Dtarget_train )4: Evaluation(M0 , Dtarget_valid )5: t ← 0
6: while termination criterion is not satisfied do
7:
P ← Parent-selection(Qt )8:
Qt+1 ← Crossover-and-mutation(P)9:
Mt+1 ← Model-construction(Qt+1 , Msource )10:
Mt+1 ← Fine-tuning(Mt+1 , Dtarget_train )11:
Evaluation(Mt+1 , Dtarget_valid )12:
Qt+1 ← Survivor-selection(Qt ∪ Qt+1 )
13:
t←t+1
14: end while
15: Mnondominated ← Approximate-Pareto-front(Mt )Given a resultant model from the structure searchfc
MSS = {Lconv, . . . , Lconv
, Lfc
1
k
1 , . . . , Lm }with
∈ RHi ×Wi ×Ci ×Fi
Lconv
i

and

1×1×Ii ×Ni
,
Lfc
i ∈R

where Hi and Wi are the height and width of the filters, and,Ci and Fi are the numbers of channels and filters in Lconv
i
,
and
N
denotes
respectively. Ii is the number of inputs in Lfcii
the number of neurons in Lfc
i . A candidate solution z in local
enhancement is represented byz = (z conv , z fc ).

(8)

The first part z conv is a binary string:
conv
conv
z conv = (z1,1, . . . , zk,F
),
k
conv
where the binary variable zi,f∈ {0, 1} indicates whether thef -th filter in the i-th convolutional layer is activated (value 1)
or not (value 0). The second part z fc is a binary string:fc
fc
z fc = (z1,1, . . . , zm,N),mfc
where the binary variable zj,n∈ {0, 1} indicates whether then-th neuron in the j-th fully connected layer is activated or
not.
conv
The pruning mask associated with zi,fcan be written by(
conv
0, if zi,f
= 0,
conv
B i (:, :, :, f ) =(9)1, otherwise,fc
and the pruning mask associated with zi,nis denoted as(fc
0, if zi,n
= 0,fc
B i (1, 1, :, n) =(10)
1, otherwise.

6fc
Let B = (B conv
, . . . , B conv
, B fc1k
1 , . . . , B m ) denote the weightmask based on z. The resultant model of local enhancement
by applying model masking on MSS with weight mask B is

MLE = MSS ◦ B,
where operator ◦ refers to the elementwise product.
Like the structure search stage, the local enhancement
stage also uses KGEA to search for the optimal models interms of prediction error and model size. The evolutionary
optimization process of local enhancement is similar to thatof structure search, except local enhancement does not apply
the fine-tuning process. To evaluate a candidate solution z,
the model masking procedure is used to determine from z
whether a filter or neuron is activated or deactivated in model
MSS . More specifically, model masking performs elementwise
multiplication of the weights of MSS with the weight mask
B of candidate solution z. The masked model is then tested
on Dtarget_valid for the classification error. The size of apruned model is simply the number of activated weights inthe masked model. Algorithm 2 outlines the pseudocode of
local enhancement.
Algorithm 2 Local enhancement
Input: a model from structure search MSS andvalidation data of target domain Dtarget_valid
Output: Mnondominated1: Z0 ← Population-initialization
2: M0 ← Model-masking(Z0 , MSS )3: Evaluation(M0 , Dtarget_valid )4: t ← 0
5: while termination criterion is not satisfied do
6:
P ← Parent-selection(Zt )7:
Zt+1 ← Crossover-and-mutation(P)8:
Mt+1 ← Model-masking(Zt+1 , MSS )9:
Evaluation(Mt+1 , Dtarget_valid )10:
Zt+1 ← Survivor-selection(Zt ∪ Zt+1 )
11:
t←t+1
12: end while
13: Mnondominated ← Approximate-Pareto-front(Mt )IV. E XPERIMENTAL S TUDIESIn this study, several experiments were conducted to examine the effectiveness and efficiency of EvoNAS-TL comparedwith baseline and state-of-the-art methods. In this section,
we first introduce the experimental settings, including thesource task, source model, target tasks, data processing, andsettings for optimization and fine-tuning. Next, we investigate
the experimental results on each transfer task and discuss theadvantages and limitations of EvoNAS-TL.
A. Experimental Settings
The experiments consisted of three transfer learning scenarios that pose different challenges to transfer learning. The
first two scenarios shared the same source task, namely,
ImageNet [25], whereas their target tasks were the CIFAR10 dataset [26] and the NEU surface defect database [27],respectively. The third scenario used the Office-31 dataset [28]as a benchmark. Figure 4 shows image samples of the threedatasets. ImageNet is widely used as a source task in transfer
learning because of its comprehensive image contents andlabels. A model well trained on ImageNet is deemed to implicitly possess a variety of feature extractors that can be reusedin other image recognition tasks through transfer learning. The
major challenge of transferring from ImageNet to CIFAR10 lies in the significant difference in image size, whereas
the challenge of transferring from ImageNet to the NEU
dataset arises from the nontrivial intertask correlation and low
data volume. The Office-31 dataset comprises three domains:
Amazon (A), DSLR (D), and Webcam (W). The three domains
share the same 31-class image classification task. In lightof the considerable successes of VGG-16 [31] in transfer
learning [9], [11], [58], VGG-16 pretrained on ImageNet
was adopted as the source model in our experiments. Eachtarget dataset was partitioned into three disjoint parts: training,validation, and testing sets. For Office-31, we conducted
experiments on all domain combinations, which resulted in
six transfer tasks in total. The experimental settings for thebaseline methods and EvoNAS-TL followed the settings in [6],[14], [59]. Common data augmentation techniques, including
rotating, shifting, and flipping, were applied to extend thetarget datasets. For all training and fine-tuning processes in theexperiments, we employed stochastic gradient descent as the
optimizer with an initial learning rate 0.005 decaying 10−6 perbatch, momentum 0.9, and batch size 256. Training and finetuning were terminated if the validation loss ceased improving
for three consecutive epochs.
1) Baseline Models: Three baseline methods were used for
performance comparison: training from scratch, transfer learning [5], and network pruning [24] after transfer learning. Our
preliminary tests showed that directly using standard VGG16 resulted in low classification accuracy due to the overly
complex architecture. To improve the baseline results, we
modified VGG-16 as mVGG-16 with a global average pooling(GAP) layer [60] between the last convolutional layer and thefirst fully connected layer. Accordingly, the source model size
was reduced from 138 million parameters (standard VGG-16)
to 33.6 million parameters (mVGG-16). It is noteworthy that in
our experiments, this modification preserved pretrained filtersin the convolutional layers while the fully connected layers
were reconstructed.
The train-from-scratch model was obtained by training
mVGG-16 from scratch with the data of the target task.
Performance comparison included the common transfer learning approach based on the fine-tuning method [5]. In thisapproach, a model based on mVGG-16 for the target domain
was built by copying the convolutional weights of the source
model VGG-16 pretrained on ImageNet, randomly initializing
the weights of its fully connected layers, and then finetuning the whole network with the training data of the target
task. To demonstrate the difference between simultaneoustransfer learning and subnetwork search featured in EvoNASTL and the sequential procedure of transfer learning followedby network pruning, we included another baseline modelthat applied the state-of-the-art network pruning method [24]7regarding both objectives. Figure 1 illustrates that structure
search produces three models corresponding to the minimal
validation error, minimal number of parameters, and knee
model. The local enhancement was applied based on thesethree models and yielded an APF for each. Accordingly,
EvoNAS-TL outputs nine models for each transfer learning
task. Table I summarizes the parameter settings for the KGEA
used in EvoNAS-TL.

airplane
automobile
bird
cat
deer
dog
frog

B. Results and Discussions

horseFirst, we investigated the performance of EvoNAS-TL in
transferring from ImageNet to CIFAR-10. The visual components of the two datasets have observable interconnections.
The common features between the source and target tasks
were expected to facilitate the transfer of knowledge. Notethat the image size of the source task (224 × 224 pixels)
significantly differs from that of the target task (32×32 pixels)
in this test case, posing a challenge for the transfer. Second,
we assessed EvoNAS-TL in transferring from ImageNet to the
NEU surface defect database, which consists of 1,800 grayscale images containing six types of surface defect patterns.
The main challenge of this test case was its low visualcorrelation between the source task and the target task. Finally,
we evaluated EvoNAS-TL in comparison with state-of-the-art
methods on Office-31 [28], a popular benchmark for transfer
learning and domain adaptation [3].
Tables II–IV compare the performance of EvoNAS-TL withthe baseline and state-of-the-art methods on the learning tasksof transferring from ImageNet to CIFAR-10, from ImageNetto the NEU dataset, and from domain to domain in Office31, respectively. The tables specifically present the performance of EvoNAS-TL with minimal-error selection for the
final candidate solutions. For transferring from ImageNet to
CIFAR-10, performance comparison includes the state-of-theart automated machine learning methods that use NAS andhyperparameter tuning, i.e., Hyperband [61], Warmstart [62],and TrAutoML [23]. For transferring from ImageNet to the
NEU dataset, we compared EvoNAS-TL with the baseline
methods, i.e., training from scratch, transfer through finetuning [5], and applying state-of-the-art network pruning [24]after transfer through fine-tuning. For Office-31, the performance of EvoNAS-TL was evaluated by comparison withbaseline methods and state-of-the-art supervised domain adaptation methods using VGG-16, including FADA [63], CCSA
[13], d-SNE [14], and DAGE-LDA [59].
1) Results of Baseline Models: Table II shows that training
from scratch resulted in low prediction accuracy when using
the modified VGG-16 to deal with CIFAR-10. This poor
performance was caused by the attempt to train a deep model
for small-sized images. By transferring the filters pretrained on
ImageNet, the model obtained showed a substantial improvement in classification accuracy, confirming the effectiveness ofthe transfer. The results also indicate that evolutionary network
pruning downsized the transferred model by 13% in thenumber of parameters, which slightly compromised the testing
accuracy. The results of the experiments on the NEU dataset
show similar tendencies: training from scratch performed

ship
truck
(a) Example images of the CIFAR-10 dataset.

crazing

inclusion

patchespitted surface rolled-in scale

scratches

Amazon

...

DSLR

...

Webcam(b) Example images of the NEU dataset.

...
back_packcalculator

deskchair

file_cabinet

trash_can(c) Example images of the Office-31 dataset.
Fig. 4. Demonstration of the CIFAR-10, NEU surface defect, and Office-31
datasets.

to the fine-tuned model. Considering that network training,fine-tuning, evolutionary network pruning, and EvoNAS-TL
all involve stochasticity, our empirical study considered theaverage performance over 10 independent runs for the CIFAR10 and NEU datasets and five independent runs for each task
in the Office-31 dataset.
2) EvoNAS-TL: The proposed EvoNAS-TL performed twostages of bi-objective optimization for transfer learning. In thefirst stage, structure search of EvoNAS-TL used the modified
VGG-16 as the source model, searched for suitable subnetworks for the target domain, and yielded an APF of candidate
models. From the APF, the model with the minimal validation
error, the model with the minimal number of parameters, andthe knee model were used individually in the next stage. In
the second stage, EvoNAS-TL local enhancement received amodulated transferred model from structure search and applied
evolutionary network pruning to further improve the model8

TABLE IPARAMETER SETTINGS FOR E VO NAS-TL.
Structure search

Representation

Selective representationx ∈ {1, ..., 13} × [0, 1]2

x ∈ {0, 1, 2}13 × [0, 1]2

Uniform (xconv ); whole-arithmetic (xfc )Crossover
Mutation
Population size
Parent selection
Crossover rate
Mutation rate
Survivor selection#generations

Local enhancement

Sequential representation

Random resetting (xconv ); uniform (xfc )30
2-tournament
1.0
0.333

0.066
(µ + λ)30

poorly in training the huge and deep model. Reusing the filters
trained on ImageNet nevertheless showed the potential to solve
low-resource problems. In addition, the size of the transferredmodel was reduced by 35% through evolutionary network
pruning, while the testing accuracy was slightly affected.
The results of pruning after fine-tuning suggest the utilityof pruning the complex architecture of the source model in
CIFAR-10 and NEU, but they tend to overfit in Office-31.
The evolutionary network pruning can increase the validation
accuracy and reduce the model size, but this advantage is not
generalized to test data. This overfitting may have been causedby the scarcity of data in Office-31, which contains only 7–100
images for each category in a domain.
2) Results of EvoNAS-TL: The optimization process in
EvoNAS-TL comprises two stages: structure search (SS) andlocal enhancement (LE). As previously stated, this work proposes sequential representation and selective representation forstructure search, thereby considering the high-level structural
information of the source model in different ways. The experiments investigated the effects of these two representations
with and without local enhancement on the performance of
EvoNAS-TL.
As indicated in Tables II and III, EvoNAS-TL achieved
significantly better testing accuracy than the three baseline
methods in the CIFAR-10 test case and comparable accuracy
in the NEU test case. In addition, EvoNAS-TL effectively
reduced the number of parameters in the resultant models in areasonable search time. Both test cases showed that EvoNASTL was highly capable of tailoring neural architecture for thetarget tasks in transfer learning. Regarding the effect of representation, both representations yielded similar improvement in
the classification error. Selective representation, in particular,
was more effective in reducing the model size because it
enabled more precise manipulation of the neural architecture
than sequential representation. As for the improvement exerted
by the two stages, the experimental results indicate thatstructure search significantly contributed to both validation
accuracy and model size; to be precise, it reduced the numberof parameters by 52%–78% and improved the test accuracy
by up to 6.9%. Local enhancement could further downsize themodels by 4%–15%.
On Office-31 in Table IV, EvoNAS-TL using structure
search with sequential representation achieved comparable orz ∈ {0, 1}l , l ≤ 12416
Uniform
Bit-flip
200
2-tournament
1.0
1/l(µ + λ)500

better average classification accuracy and model size than thestate-of-the-art methods. Note that the Office-31 dataset was
originally developed for domain adaptation, instead of generaltransfer learning. In addition, the domain adaptation methods
used in the experiments assume that the target task is the sameas the source task, whereas EvoNAS-TL does not rest on suchan advantageous assumption. Even so, EvoNAS-TL performed
comparably to or better than the domain adaptation methods.
Table IV further reveals that network pruning was vulnerable
to overfitting, which may have been caused by overly pursuingclassification accuracy despite having only limited data. This
negative effect also occurred in local enhancement and caused
the deterioration in the average accuracy of EvoNAS-TL.
Figure 5 shows the APFs produced by structure searchand local enhancement. The solid triangles and diamond in
Figs. 5a and 5b respectively plot the minimal-error model, theminimal-size model, and the knee model in the APF obtained
from structure search. The hollow symbols in Fig. 5b depict
the APFs gained by applying local enhancement to the threemodels. The distribution of each APF shows a clear conflicting
relationship between the objectives of minimizing prediction
error and minimizing model size.
To further inspect the performance of models located indifferent regions of objective space, Tables V and VI compare the performance of three decision-making policies forselecting models from an APF. As the flowchart in Fig. 1
indicates, the knee solution and the best solutions of the two
objectives are selected from the resultant APF of KGEA. The
models that aim for minimal parameters are undesirable inpractice due to their poor classification accuracy. The knee
selection strategy presents a compromise between the two
conflicting objectives [24], [64]. Considering the synthetic
effects, EvoNAS-TL using the knee strategy for structure
search and the minimal-error solution for local enhancement
can strike a balance between the two objectives and generatemodels that have acceptable classification error with a substantial reduction in model size. In contrast, applying knee
selection to both stages causes KGEA to trade too muchaccuracy to reduce the model size. The preferable outcomes
above indicated that EvoNAS-TL showed strong capabilities
in advancing classification accuracy through minimal-error
selection in both stages and in finding satisfactory results
through the knee selection in structure search and minimal-

9

TABLE IIE XPERIMENTAL RESULTS OF TRANSFERRING FROM I MAGE N ET TO CIFAR-10. B OLDFACE DENOTES THE BETTER OF SS AND SS+LE RESULTS .
Accuracy
Val (%)
Test (%)

MethodTrain-from-scratch
Fine-tune [5]Fine-tune + prune [24]Hyperband [61]Warmstart + hyperband [62]
TrAutoML (MH) + hyperband [23]#Parameters(×106 )

Reduced size
(%)

FLOPs
(×107 )

Time of NAS(GPU days)

52.9
80.6
81.0

53.2
80.3
79.9

33.64
33.64
29.21

13%

33.2
33.2
30.0

0.67

-

78.5
73.5
77.5

-

-

-

0.67
0.67
0.67

EvoNAS-TL (sequential)

SS
SS + LE

87.8
88.2

87.2
87.2

16.20
14.69

52%
56%

30.0
28.1

2.62
3.10

EvoNAS-TL (selective)SS
SS + LE

87.6
88.1

87.1
86.8

12.60
11.21

63%
67%

26.3
24.7

2.32
2.78

TABLE IIIE XPERIMENTAL RESULTS OF TRANSFERRING FROM I MAGE N ET TO THE NEU DATASET. B OLDFACE DENOTES THE BETTER OF SS AND SS+LE RESULTS .
Accuracy
Val (%)
Test (%)

MethodTrain-from-scratch
Fine-tune [5]Fine-tune + prune [24]#Parameters(×106 )

Reduced size
(%)

FLOPs
(×109 )

Time of NAS(GPU days)

31.6
98.4
99.4

22.4
98.9
97.7

33.62
33.62
22.02

35%

15.37
15.37
10.58

1.01

EvoNAS-TL (sequential)

SS
SS + LE

98.6
99.8

99.6
98.9

12.40
7.36

63%
78%

14.38
8.90

1.15
1.84

EvoNAS-TL (selective)SS
SS + LE

99.2
99.9

98.6
98.7

7.55
5.13

78%
85%

11.33
8.01

1.09
1.66

TABLE IV
E XPERIMENTAL RESULTS OF TRANSFERRING BETWEEN DOMAINS IN THE O FFICE -31 DATASET, INCLUDING THE AVERAGE TEST ACCURACY AND
AVERAGE MODEL SIZE ( MILLIONS OF PARAMETERS ). B OLDFACE DENOTES THE BETTER OF SS AND SS+LE RESULTS .
MethodTest Accuracy (%)
D→A
D→W
W→A

W→D

Avg. acc.
(%)

Avg. size(×106 )

75.1
65.4

68.1
58.5

76.5
71.7

33.7
23.5

96.4
96.4
97.1
97.9

71.1
72.1
71.7
64.2

97.5
97.6
97.5
99.5

84.9
85.8
86.5
83.6

15.4
15.4
15.4
15.4

A→D

A→W

Fine-tune [5]Fine-tune + prune [24]

59.1
67.5

92.8
82.7

78.3
72.3

85.8
83.8

FADA (VGG-16) [63]CCSA (VGG-16) [13]d-SNE (VGG-16) [14]DAGE-LDA (VGG-16) [59]

88.2
89.0
91.4
85.9

88.1
88.2
90.1
88.7

68.1
71.8
71.1
66.2

EvoNAS-TL (sequential)

SS
SS + LE

89.6
66.3

93.8
82.8

79.1
74.9

94.1
81.6

77.7
74.5

83.8
61.9

86.4
73.7

15.2
12.8

EvoNAS-TL (selective)SS
SS + LE

77.6
64.1

95.3
80.3

77.3
75.2

90.9
81.2

77.0
71.6

83.8
68.8

83.7
73.5

18.4
14.5

error selection in local enhancement.
Figure 6 further depicts examples of resultant models from
EvoNAS-TL. The generated models using selective representation reflect that EvoNAS-TL tends to discard the last few
convolutional layers. Additionally, the models using sequential
representation tend to remove only a few layers. These tendencies are in line with the idea that the layers close to the input
can catch low-level features and the layers close to the output
act as high-level feature constructors as well as classifiers.
On the other hand, extended from sequential representation,
selective representation can remove arbitrary convolutional
layers and fix the parameters of arbitrary convolutional layers.
This extension allows EvoNAS-TL to flexibly manipulate thearchitecture of the source model and generate the models

in an efficient model size. Although selective representation
can fix the parameters of arbitrary convolutional layers, theexperimental results show that EvoNAS-TL rarely opted to doso.
In short, EvoNAS-TL can automatically search for suitable
subnetworks of the source model in transfer learning tasks.
The two representations showed no significant difference inimproving the prediction accuracy. However, selective representation was more effective in reducing the model size for
the CIFAR-10 and NEU datasets, whereas sequential representation is easier to implement. For applications seeking highclassification performance, applying minimal-error selection toboth stages gives satisfactory results. For applications pursuing
a compact model with acceptable classification performance,

10

applying the knee selection to structure search and minimalerror selection to local enhancement yields remarkable resultsin terms of both classification performance and model size.
V. C ONCLUSIONS
Transfer learning has been used extensively to enhance deeplearning models for learning from low-resource tasks. Despite
many successful applications, one notable concern about transfer learning is the potential inefficiency of neural architecture,as the transfer techniques focus on tuning parameters forthe target task but seldom modify the neural architectureof the source model. This issue impedes those real-world
applications with a limitation on model size or inference
latency, such as automatic optical inspection in manufacturing.
This study proposes EvoNAS-TL, a two-stage bi-objective
evolutionary system that performs structure search followed
by local enhancement for transfer learning tasks. The structure
search manipulates the source model over the structure bylayer as a coarse-grained architectural search, while the local
enhancement attempts to remove unfavored filters and neurons
as fine-grained architectural optimization. For structure search,
we propose two representations that encode different levels of
structural information.
The performance of EvoNAS-TL was examined and compared with three baseline methods and state-of-the-art methods
over three transfer learning scenarios. The experimental results
showed that, by automatically tailoring neural architecture forthe target tasks, the proposed EvoNAS-TL improved classification performance and significantly reduced the model size.
Using the minimal-error selection in both stages, EvoNAS-TL
improved the testing accuracy by 6.9% in transferring to the
CIFAR-10 dataset and 0.7% in transferring to the NEU dataset;meanwhile, it removed 52% to 85% of the parameters from thesource model. This strategy benefits applications that seek highclassification performance. Moreover, by selecting the knee
model in structure search and the minimal-error model in local
enhancement, EvoNAS-TL can strike a balance between test
accuracy and model size. On the Office-31 dataset, EvoNASTL achieved average accuracy and model size comparable to or
better than state-of-the-art domain adaptation methods. Thesesatisfactory outcomes validate that the proposed EvoNASTL, which aims to optimize neural architecture in transfer
learning tasks, shows promise in advancing the effectivenessand efficiency of the transferred models.
ACKNOWLEDGMENT
This work was supported by the Ministry of Science andTechnology of Taiwan, under contracts MOST 110-2634-F007-026 and MOST 110-2221-E-007-082-MY3. The authors
would like to thank National Center for High-performance
Computing (NCHC) in Taiwan for providing computationaland storage resources.
R EFERENCES[1] K.-M. He, X.-Y. Zhang, S.-Q. Ren, and J. Sun, “Deep residual learningfor image recognition,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.

[2] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–
1359, 2009.
[3] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and
Q. He, “A comprehensive survey on transfer learning,” Proceedings ofthe IEEE, vol. 109, no. 1, pp. 43–76, 2021.
[4] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on
deep transfer learning,” in Proceedings of International Conference on
Artificial Neural Networks (ICANN), 2018, pp. 270–279.
[5] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable arefeatures in deep neural networks?” in Proceedings of the Advances in
Neural Information Processing Systems (NIPS), 2014, pp. 3320–3328.
[6] E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko, “Simultaneous
deep transfer across domains and tasks,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV), 2015, pp. 4068–
4076.
[7] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable
features with deep adaptation networks,” in Proceedings of International
Conference on Machine Learning (ICML), 2015, pp. 97–105.
[8] H.-C. Shin, H. R. Roth, M.-C. Gao, L. Lu, Z.-Y. Xu, I. Nogues, J.H. Yao, D. Mollura, and R. M. Summers, “Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset
characteristics and transfer learning,” IEEE Transactions on Medical
Imaging, vol. 35, no. 5, pp. 1285–1298, 2016.
[9] K. Gopalakrishnan, S. K. Khaitan, A. Choudhary, and A. Agrawal,“Deep convolutional neural networks with transfer learning for computer
vision-based data-driven pavement distress detection,” Construction and
Building Materials, vol. 157, pp. 322–330, 2017.
[10] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learningwith joint adaptation networks,” in Proceedings of International Conference on Machine Learning (ICML), 2017, pp. 2208–2217.
[11] M. Hon and N. M. Khan, “Towards Alzheimer’s disease classification
through transfer learning,” in Proceedings of the IEEE International
conference on bioinformatics and biomedicine (BIBM). IEEE, 2017,
pp. 1166–1169.
[12] S.-H. Kim, W.-Y. Kim, Y.-K. Noh, and F. C. Park, “Transfer learning forautomated optical inspection,” in Proceedings of the International Joint
Conference on Neural Networks (IJCNN). IEEE, 2017, pp. 2517–2524.
[13] S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto, “Unified deepsupervised domain adaptation and generalization,” in Proceedings of theIEEE International Conference on Computer Vision (ICCV), 2017, pp.
5715–5725.
[14] X. Xu, X. Zhou, R. Venkatesan, G. Swaminathan, and O. Majumder,“d-SNE: Domain adaptation using stochastic neighborhood embedding,”
in Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), 2019, pp. 2497–2506.
[15] B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural networkarchitectures using reinforcement learning,” in Proceedings of International Conference on Learning Representations (ICRL), 2017.
[16] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in Proceedings of International
Conference on Machine Learning (ICML), 2018, pp. 4095–4104.
[17] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable
architectures for scalable image recognition,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2018,
pp. 8697–8710.
[18] Y. Li, Z. Yang, Y. Wang, and C. Xu, “Adapting neural architectures
between domains,” in Proceedings of Advances in Neural Information
Processing Systems (NeurIPS), 2020.
[19] C. Wong, N. Houlsby, Y. Lu, and A. Gesmundo, “Transfer learning withneural AutoML,” in Proceedings of the 32nd International Conference
on Neural Information Processing Systems (NIPS), 2018, pp. 8366–
8375.
[20] R. Istrate, F. Scheidegger, G. Mariani, D. Nikolopoulos, C. Bekas, and
A. C. I. Malossi, “Tapas: Train-less accuracy predictor for architecture
search,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 3927–3934.
[21] E. Kokiopoulou, A. Hauth, L. Sbaiz, A. Gesmundo, G. Bartok, andJ. Berent, “Fast task-aware architecture inference,” 2019.
[22] M. Wistuba and T. Pedapati, “Inductive transfer for neural architecture
optimization,” 2019.
[23] C. Xue, J. Yan, R. Yan, S. M. Chu, Y. Hu, and Y. Lin, “Transferable
automl by model sharing over grouped datasets,” in Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2019, pp. 9002–9011.

Fig. 6. Example models produced by EvoNAS-TL with minimal-error selection applied to both stages.

(d) A resultant model of transferring from ImageNet to the NEU dataset using selective representation in SS.
)&RXWSXW)&¬

)&

)&RXWSXW)&

)&¬

)&RXWSXW)&¬

)&¬

30
40
Validation error (%))&RXWSXW)&¬)&

20*OREDO$YHUDJH3RROLQJ

10*OREDO$YHUDJH3RROLQJ

APF of LE from

M
M
M

*OREDO$YHUDJH3RROLQJ(a) Resultant APF of structure search.
APF of LE from

*OREDO$YHUDJH3RROLQJ

100¬'LVFDUGHG¬

APF of LE from[&219

2¬'LVFDUGHG¬¬
[&219

10

[&219

3_¬'LVFDUGHG¬¬
[&219

10SS(min err)

[&219

3

[&219

10¬'LVFDUGHG¬¬
[&219

4¬'LVFDUGHG¬¬
[&219

10

[&219

4

[&219

10

[&219

5

[&219

10

3RROLQJ

5

3RROLQJ

10

3RROLQJ

6

3RROLQJ

10#Parameters

6

APF of SS

[&219

[&219

10_[&219

[&219

[&219

3RROLQJ

[&219

7

SS(min par)[&219
¬'LVFDUGHG¬¬

[&219

[&219

3RROLQJ

[&219

[&219

[&219

3RROLQJ

10

SS(knee)[&219
¬'LVFDUGHG¬¬

[&219

[&219

3RROLQJ

[&219

[&219

[&219

3RROLQJ

[&219

60

[&219

3RROLQJ

[&219

[&219

[&219

3RROLQJ

[&219

[&219

30
40
Validation error (%)

[&219

[&219

3RROLQJ

[&219

[&219

3RROLQJ

20

[&219

[&219

3RROLQJ

10[&219

3RROLQJ

[&219[&219

7

M
M
M

3RROLQJ

[&219[&219

102

[&219[&219

10

[&219[&219#Parameters11

SS(knee)SS(min err)

SS(min para)

_

_

60
100

(b) Resultant APF of local enhancement.

Fig. 5. Example of APFs generated by EvoNAS-TL using sequential representation in the case of transferring from ImageNet to the CIFAR-10 dataset.

(a) A resultant model of transferring from ImageNet to the CIFAR-10 dataset using sequential representation in SS.

(b) A resultant model of transferring from ImageNet to the NEU dataset using sequential representation in SS.

(c) A resultant model of transferring from ImageNet to the CIFAR-10 dataset using selective representation in SS.

12

TABLE VP ERFORMANCE OF E VO NAS-TL USING THREE DIFFERENT STRATEGIES IN STRUCTURE SEARCH AND LOCAL ENHANCEMENT IN TRANSFERRING FROMI MAGE N ET TO CIFAR-10. T HE PAIRWISE VALUES OF EACH CELL INDICATE TESTING ACCURACY IN PERCENTAGE AND REDUCTION PERCENTAGE OF THE
NUMBER OF PARAMETERS , RESPECTIVELY.
SS (sequential representation)LE–
min_errknee
min_parSS (selective representation)min_errknee

min_parmin_errknee

min_par87.2 / 51.8
87.2 / 56.3
75.4 / 81.5
12.2 / 90.6

82.3 / 97.3
82.4 / 97.5
65.1 / 99.2
15.0 / 99.7

22.2 / 99.9
23.2 / 99.9
21.5 / 99.9
11.7 / 99.9

87.1 / 62.6
86.8 / 66.7
75.5 / 84.3
11.3 / 92.7

84.3 / 93.4
84.2 / 94.0
66.1 / 97.9
14.9 / 99.0

62.1 / 97.9
62.7 / 98.3
51.3 / 99.3
19.0 / 99.8

TABLE VI
P ERFORMANCE OF E VO NAS-TL USING THREE DIFFERENT STRATEGIES IN STRUCTURE SEARCH AND LOCAL ENHANCEMENT IN TRANSFERRING FROMI MAGE N ET TO THE NEU DATASET. T HE PAIRWISE VALUES OF EACH CELL INDICATE TESTING ACCURACY IN PERCENTAGE AND REDUCTIONPERCENTAGE OF THE NUMBER OF PARAMETERS , RESPECTIVELY.
SS (sequential representation)LE–
min_errknee
min_parSS (selective representation)min_errknee

min_parmin_errknee

min_par99.6 / 63.1
98.9 / 78.1
87.4 / 90.3
26.6 / 94.5

80.0 / 97.5
85.8 / 98.6
61.6 / 99.7
20.7 / 99.9

22.5 / 99.9
34.9 / 99.9
32.3 / 99.9
18.2 / 99.9

98.6 / 77.6
98.7 / 84.8
87.9 / 93.7
26.2 / 97.3

85.4 / 94.5
90.4 / 97.1
63.4 / 99.3
21.3 / 99.7

54.6 / 97.8
67.2 / 98.9
50.5 / 99.9
26.0 / 99.9

[24] Y. Zhou, G. G. Yen, and Z. Yi, “A knee-guided evolutionary algorithm
for compressing deep neural networks,” IEEE Transactions on Cybernetics, 2019.
[25] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z.-H.
Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “ImageNet large
scale visual recognition challenge,” International Journal of Computer
Vision, vol. 115, no. 3, pp. 211–252, 2015.
[26] A. Krizhevsky, G. Hinton et al., “Learning multiple layers of featuresfrom tiny images,” Citeseer, Tech. Rep., 2009.
[27] K.-C. Song and Y.-H. Yan, “A noise robust method based on completed
local binary patterns for hot-rolled steel strip surface defects,” Applied
Surface Science, vol. 285, pp. 858–864, 2013.
[28] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual category models to new domains,” in Proceedings of the 11th European
Conference on Computer Vision (ECCV), 2010, pp. 213–226.
[29] W. Rawat and Z.-H. Wang, “Deep convolutional neural networks forimage classification: A comprehensive review,” Neural Computation,
vol. 29, no. 9, pp. 2352–2449, 2017.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proceedings of theAdvances in Neural Information Processing Systems (NIPS), 2012, pp.
1097–1105.
[31] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[32] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in Proceedings of the European Conference on
Computer Vision (ECCV). Springer, 2014, pp. 818–833.
[33] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” in Proceedingsof International Conference on Learning Representations (ICLR), 2017.
[34] C. Reinhold and M. Roisenberg, “Filter pruning for efficient transfer
learning in deep convolutional neural networks,” in Proceedings of theInternational Conference on Artificial Intelligence and Soft Computing,2019, pp. 191–202.
[35] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan,
Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,”
in Proceedings of the International Conference on Machine Learning(ICML), 2017, pp. 2902–2911.
[36] Y. Sun, B. Xue, M. Zhang, G. G. Yen, and J. Lv, “Automaticallydesigning CNN architectures using the genetic algorithm for image
classification,” IEEE Transactions on Cybernetics, pp. 1–15, 2020.
[37] M. Suganuma, S. Shirakawa, and T. Nagao, “A genetic programming
approach to designing convolutional neural network architectures,” inProceedings of the Genetic and Evolutionary Computation Conference(GECCO). ACM, 2017, pp. 497–504.

[38] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement
learning,” arXiv preprint arXiv:1611.01578, 2016.
[39] E. Real, A. Aggarwal, Y.-P. Huang, and Q. V. Le, “Regularized evolution
for image classifier architecture search,” in Proceedings of the AAAI
Conference on Artificial Intelligence, vol. 33, 2019, pp. 4780–4789.
[40] Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu, “Practical block-wiseneural network architecture generation,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2018,
pp. 2423–2432.
[41] H.-X. Liu, K. Simonyan, and Y.-M. Yang, “DARTS: Differentiablearchitecture search,” in Proceedings of International Conference on
Learning Representations (ICLR), 2019.
[42] B.-C. Wu, X.-L. Dai, P.-Z. Zhang, Y.-H. Wang, F. Sun, Y.-M. Wu, Y.D. Tian, P. Vajda, Y.-Q. Jia, and K. Keutzer, “FBNet: Hardware-awareefficient convnet design via differentiable neural architecture search,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2019, pp. 10 734–10 742.
[43] L.-X. Xie and A. Yuille, “Genetic CNN,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV), 2017, pp. 1379–
1388.
[44] H.-P. Zhou, M.-H. Yang, J. Wang, and W. Pan, “BayesNAS: A Bayesian
approach for neural architecture search,” in Proceedings of International
Conference on Machine Learning (ICML), 2019.
[45] Y.-N. Sun, B. Xue, M.-J. Zhang, and G. G. Yen, “Completely automated
CNN architecture design based on blocks,” IEEE Transactions on Neural
Networks and Learning Systems, vol. 31, no. 4, pp. 1242–1254, 2020.
[46] Y.-C. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep
convolutional networks using vector quantization,” arXiv preprint
arXiv:1412.6115, 2014.
[47] W. Wen, C.-P. Wu, Y.-D. Wang, Y.-R. Chen, and H. Li, “Learning
structured sparsity in deep neural networks,” in Proceedings of theAdvances in Neural Information Processing Systems (NIPS), 2016, pp.
2074–2082.
[48] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
deep neural networks with pruning, trained quantization and Huffman
coding,” in Proceedings of International Conference on Learning Representations (ICLR), 2015.
[49] J.-H. Luo, J.-X. Wu, and W.-Y. Lin, “ThiNet: A filter level pruning
method for deep neural network compression,” in Proceedings of theIEEE International Conference on Computer Vision (ICCV), 2017, pp.
5068–5076.
[50] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters
for efficient convnets,” in Proceedings of International Conference on
Learning Representations (ICLR), 2017.
[51] H.-Y. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang, “Network Trimming:

13

A data-driven neuron pruning approach towards efficient deep architectures,” arXiv preprint arXiv:1607.03250, 2016.
[52] Y.-H. He, J. Lin, Z.-J. Liu, H.-R. Wang, L.-J. Li, and S. Han, “AMC:
AutoML for model compression and acceleration on mobile devices,” inProceedings of the European Conference on Computer Vision (ECCV),
2018, pp. 784–800.
[53] Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating
very deep neural networks,” in Proceedings of the IEEE International
Conference on Computer Vision (ICCV), 2017, pp. 1398–1406.
[54] Y.-H. Wang, C. Xu, J.-Y. Qiu, C. Xu, and D.-C. Tao, “Towards evolutionary compression,” in Proceedings of the ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining (KDD). ACM,
2018, pp. 2476–2485.
[55] Y. Le Cun, J. S. Denker, and S. A. Solla, “Optimal brain damage,”in Proceedings of International Conference on Neural Information
Processing Systems (NIPS), 1989, pp. 598–605.
[56] B. Hassibi and D. G. Stork, “Second order derivatives for network pruning: Optimal brain surgeon,” in Proceedings of International Conference
on Neural Information Processing Systems (NIPS), 1992, pp. 164–171.
[57] B. Bartoldson, A. Morcos, A. Barbu, and G. Erlebacher, “Thegeneralization-stability tradeoff in neural network pruning,” in Proceedings of Conference on Neural Information Processing Systems, vol. 33,
2020.
[58] M. Shaha and M. Pawar, “Transfer learning for image classification,” inProceedings of the International Conference on Electronics, Communication and Aerospace Technology (ICECA). IEEE, 2018, pp. 656–660.
[59] L. H. Morsing, O. A. Sheikh-Omar, and A. Iosifidis, “Supervised domain
adaptation using graph embedding,” arXiv preprint arXiv:2003.04063,
2020.
[60] M. Lin, Q. Chen, and S.-C. Yan, “Network in network,” arXiv preprint
arXiv:1312.4400, 2013.
[61] L. Li, K. G. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar,“Hyperband: Bandit-based configuration evaluation for hyperparameter
optimization.” in Proceedings of International Conference on Learning
Representations (ICLR), 2017.
[62] M. Lindauer and F. Hutter, “Warmstarting of model-based algorithm
configuration,” in Proceedings of the AAAI Conference on Artificial
Intelligence, vol. 32, no. 1, 2018.
[63] S. Motiian, Q. Jones, S. M. Iranmanesh, and G. Doretto, “Few-shotadversarial domain adaptation,” in Proceedings of the 31st International
Conference on Neural Information Processing Systems (NIPS), 2017,
pp. 6673–6683.
[64] W.-Y. Chiu, G. G. Yen, and T.-K. Juan, “Minimum Manhattan distanceapproach to multiple criteria decision making in multiobjective optimization problems,” IEEE Transactions on Evolutionary Computation,
vol. 20, no. 6, pp. 972–985, 2016.

Yu-Wei Wen received the B.S. degree in computerscience and information engineering from National
Chung Cheng University, Taiwan, in 2015, where
he is currently pursuing the Ph.D. degree with the
Department of Computer Science and Information
Engineering.
His research interests include evolutionary computation, memetic algorithm, machine learning, andcomputer music composition.

Sheng-Hsuan Peng received the B.S. degree inpower mechanical engineering from National Tsing
Hua University, Taiwan, in 2018, where he is currently pursuing the M.S. degree with the Departmentof Power Mechanical Engineering.
His research interests include evolutionary computation, deep learning, and machine learning.

Chuan-Kang Ting (Senior Member, IEEE) received
the B.S. degree from National Chiao Tung University, Taiwan, in 1994, the M.S. degree from National
Tsing Hua University, Taiwan, in 1996, and the Dr.
rer. nat. degree in Computer Science from Paderborn
University, Germany, in 2005.
He is currently a Professor and the Chair ofDepartment of Power Mechanical Engineering, National Tsing Hua University, Taiwan. His researchinterests include evolutionary computation, computational intelligence, machine learning, and theirapplications in machinery, manufacturing, ethics, music and arts.
Dr. Ting is the Editor-in-Chief of IEEE Computational Intelligence Magazine, the Editor-in-Chief of Memetic Computing, an Associate Editor ofIEEE T RANSACTIONS ON E MERGING T OPICS IN C OMPUTATIONAL I NTEL LIGENCE , and an Editorial Board Member of Soft Computing. He served
as the IEEE Computational Intelligence Society (CIS) Newsletter Editor, theIEEE CIS Webmaster, the Chair of IEEE CIS Chapters Committee, and the
Chair of IEEE CIS Creative Intelligence Task Force. He is an Executive Board
Member of Taiwanese Association for Artificial Intelligence.