Neural Operator Search
Authors: Wei Li
Shaogang Gong
Xiatian Zhu
Venue:  (2020)
Keywords: deep learning
autoML
neural architecture search
image classification
attention learning
dynamic convolution

Abstract: Existing neural architecture search (NAS) methods explore a limited feature-transformation-only search space while ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions. This disables the NAS algorithms to discover more advanced network architectures. We address this limitation by additionally exploiting feature self-calibration operations, resulting in a heterogeneous search space. To solve the challenges of operation heterogeneity and significantly larger search space, we formulate a neural operator search (NOS) method. NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in a unified structure, and an attention guided search strategy for facilitating the search process over a vast space. Extensive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.


Under review as a conference paper at ICLR 2020
NEURAL OPERATOR SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Existing neural architecture search (NAS) methods explore a limited feature- transformation-only search space, ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions. This dis- ables the NAS algorithms to discover more advanced network architectures. We address this limitation by additionally exploiting feature self-calibration opera- tions, resulting in a heterogeneous search space. To solve the challenges of opera- tion heterogeneity and significantly larger search space, we formulate a neural op- erator search (NOS) method. NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in unified structure, and attention guided search strategy for facilitating the search process over a vast space, Exten- sive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.
INTRODUCTION
Recent advances of Neural Architecture Search (NAS) are remarkable in challenging tasks, e.g. im- age classification (Zoph & Le, 2017), object detection (Ghiasi et al., 2019), and semantic segmenta- tion (Liu al., 2019a; Nekrasov et al., 2019), greatly alleviating the demands for human knowledge and interventions by automating the laborious process of designing neural network architectures. One common scheme for the standard proxy-based neural architecture search methods (Pham al., 2018; Zoph et al., 2018; Liu et al., 2019b) is to factorise the search space via repeatedly stacking the same cell structure, within which a computing block generates an output tensor FR by combining the transformations of two input feature tensors F, and Fy:
= (F.) ② o' (F)) s.t. i &
(I)
where o and o² are the i-th and j-th primitive operations for feature transformation, selected from a candidate operation set O, and @ is the element-wise addition. Existing NAS methods use only the standard feature leaming/ransformattinn operations (convolution, pooling and identity mapping) as the building components.
Besides, extensive studies (Hu et al., 2018b; Bertinetto al., 2016; Wang et al., 2017; Jia et al., 2016; Wu et al., 2019b; Zhu et al., 2019) have proven that other advanced operations for feature self-calibration, such attention learning and dynamic convolutions, can bring great benefits for representation learning. For example, Hu et al. (2018b) proposes saueezeeand-Exciinee to explicitly model inter-dependencces between channels by learning channel-wise self-attention. Jia et al. (2016) presents Dynamic Filter Networks to generate context-aware filters for increasing the flexibility and adaptiveness of networks. However, these useful feature calibration elements have never been well exploited in NAS, significantly limiting the potentials of NAS which aims for automatically discovering more sophisticated and advanced network architectures without human engineering.
Networks
In this work, we aim to address this limitation by extending the search space of NAS with feature self-calibration operations for scaling up the search boundary. This makes heterogeneous search space. Consequently, the way of feature tensor interaction and combination is dramatically diversi- fied, from the conventional addition operator + only to the combination of addition @ multiplication O for attention modelling, and dynamic convolution @ In this regard, we call the proposed method Neural Operator Search (NOS).
Such a search space enhancement is critical since NAS is enabled to explore stronger and previously undiscovered network architectures, which opens door to potentially take the NAS research to the

---

Under review as a conference paper at ICLR 2020
next level. In the no free lunch saying, this also comes with two new challenges: (i) It non-trivial and more challenging to assemble such heterogeneous tensors and operations (i.e. features, atten- tions and dynamic weights) in unified computing block, compared to the conventional homoge- neous feature-tensortto-fell fer-t--r-etrreeeesooo transformation; (ii) The search space increases exponentially which leads to much harder NAS problem.
To address the first challenge, we formulate a heterogeneous operator cell characterised by a novel
heterogeneous residual block. This block, formulated in residual learning spirit (He et al., 2016), is designed specially for fusing all the different types of tensors and operations synergistically. To solve the second challenge, we propose leveraging the attention transfer (Zagoruyko & Komodakis, 2017) idea to facilitate the search behaviour across this significantly larger network space via following the attention guidance of a pretrained teacher model. As we will show, this guidance not only makes the search more efficient but also improves the search result.
Our contributions in this work are: (1) We present a novel heterogeneous search space for NAS characterised by richer primitive operations including both conventional feature transformations and newly introduced feature self-calibration. This breaks the conventional selection limit of candidate neural networks and enables the NAS process to find stronger architectures, many of which are impossible to be discovered in the conventional space. This opens new territories for supporting stronger NAS algorithms and new possibilities for most expressive architectures ever to be revealed.
(2) We formulate a novel Neural Operator Search (NOS) method dedicated for NAS in the proposed heterogeneous search space, with a couple of key designs heterogeneous residual block for fusing different types of tensor operations synergistically and attention guided search for facilitating the search process over a vast search space more efficiently and more effectively. (3) With extensive comparisons to the state-of-the-art NAS methods, the experiments show that our approach is highly competitive on both CIFAR and ImageNet- mobile image classification tests.
RELATED WORK
Neural Architecture Search. Since the seminal work by Zoph & Le (2017), neural architecture search has gained a surge of interest, effectively replacing laborious human designs by the compu- tational process. From the strategy point of view, NAS methods can be categorised into two types: (1) proxy-hased (Zoph & Le, 2017: Zoph et al., 2018; Pham et al., 2018; Liu et al., 2019h) and (2) proxy-less (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019a) NAS. Specifically, to alleviate the computational cost during search, the proxy-based NAS methods search for building cells on proxy tasks, with one or more of following compromised strategies: starting with fewer cells; using a smaller dataset (e.g. CIFAR-10); learning with fewer epochs. Then, to transfer to the large-scale target task, one can build a network by stacking searched cells without further exploration. How- ever, suffering from lacking of directness and specialisation, the searched cells by proxy-based NAS methods are not guaranteed to be optimal on the target task. contrast, proxy-less NAS methods directly learns architectures on target task by starting with an wer-parameterised network (super- net) that contains all possible paths, in which the redundant paths are pruned to derive the optimised architecture. Notwithstanding significant better results than proxy-based approaches, proxy-less NAS methods require massive computational cost and GPU memory assumption, due to learning with the vast-size supernet. From the optimisation point of view, existing NAS methods usually fall into three groups: reinforcement learning (RL) based methods, evolutionary algorithm (EA) based methods, and gradient differentiable (GD) methods. In particular, RL-based NAS methods (Zoph & Le. 2017; Pham et al., 2018; Tan et al., 2019) control the selection of architecture component in a sequential order with policy networks. EA-based NAS methods (Real et al., 2019; Liu et al., 2018b) employ the validation accuracies to guide the evolution of population of initialised architectures. RL- an EA-based NAS methods usually suffer from low efficiency and high computational resource demand, due to the fundamental searching challenge in a discrete space. In contrast, GD-based NAS methods (Liu et al., 2019b; Xie et al., 2019; Luo et al., 2018) conduct searching over a continuous space by relaxation or mapping, substantially reducing the search cost to a few GPU days. Whilst varying in the algorithmic aspects, all these works commonly explore the traffuratrassoommattonn only search spaces without more diverse and advanced operations as investigate here. To show the NAS potential of the proposed richer search space with self-calibration learning operations, we take the efficient proxy-based GD optimisation due the resource constraint.
2

---

Under review as a conference paper at ICLR 2020
Self-Calibration. Self-calibration is type of mechanism enabling network to dynamically per- form input-conditional adjjsmeuttt,n,, which has been studied extensively in both the computer vision (Hu et al., 2018b; Jia et al., 2016; Li et al., 2018; Park et al., 2018) and natural language pro- cessing (NLP) literature (Wu et al., 2019b; Vaswani et al., 2017). There are two typical paradigms of self-calibration: self-attention learning and dynamic convolutions, realised via element-wise multiplication operator @ and a dynamic convolution operator @ respectively. Despite showing significant efficacy, self-calibration only exploited independently after architecture hand-design (Hu et al., 201 2018b) or auto-search (Tan et al., 2019). We move step further by fully exploring the potential of self-calibration along with feature transformation in joint optimisation, bringing richer search space for neural architecture search.
Knowledge Distillation. There are recent works that use knowledge distillation help computer vision and NLP tasks. Three types of knowledge are usually considered in distillation: features (Yim et al., 2017), attention (Zagoruyko & Komodakis, 2017), and predictions (Hinton al., 2015). We leverage the attention distillation with a different objective alleviating the intrinsic training-test discrepancy issue of the proxy-based NAS strategy, particularly with a more expressive search space. This represents novel exploitation of attention distillation (Zagoruyko & Komodakis, 2017).
METHOD
In this section, start by formulating heterogeneous search space for NAS (Section 3.1), fol- lowed by dedicated heterogeneous operator cell to enable composing the heterogeneous opera- tions in unified computing block with synergistic interaction and cooperation (Section 3.2). To overcome the intrinsic architecture discovery challenges from more expressive search space, we further develop an attention guided search scheme (Section 3.3).
3.1 HETEROGENEOU'S SEARCH SPACE
To enrich the NAS search space so that more advanced network architectures be discovered, introduce heterogeneous search space A that considers three dif- ferent types of representation learning capabilities: (1) Feature transformations; (2) Attention learning; and (3) Dynamic convolutions. More concretely, we form three sets of primitive computing operations that produce fea- tures, attentions and dynamic weights, respectively. This novel search space generalises the conventional counter- part which limited to the first type of operations (Liu et al., 2019b; Pham al., 2018), and incorporates the self-calibration learning capabilities (i.e. the second and third types) in NAS. Importantly, while the search space changes, the generic search strategies still apply therefore being largely open for collaborating with existing NAS methods. For instance, in the proxy-based NAS strategy we may first search for a computing cell with heteroge- neous operations the building block and then form the final network architecture by sequentially stacking multi- ple such cells layer-by-layer.
F
CXHXW
(Ix1)
c/rxHxW
conv (1x1)
x X HXW
Softmax
c/rXHW
NWxK2
c/rXXx
conv (1x1)
CXK*K
Cxx?
Softmax
cXK*K Y
D:
Next, let us describe the heterogeneous primitive opera- tion set O which consists of the following three disjoint subsets: O, O2 and On. along with their aggregation or application operators.
Figure 1: Structure of the proposed dy- namic convolutions for image classifi- cation. @ denotes matrix multiplication.
Feature Transformation Operations Oy. We adopt the feature transformation//larming operation set O same as in Liu et al. (2019b; 2018a), including the following operations: 3 x 3 and 5 separable convolutions, 3 x 3 and 5 x 5 dilated separable convolutions, x average pooling, 3 x max pooling, and identity. Every operation Of O, takes input feature tensor and
3

---

Under review as a conference paper at ICLR 2020
outputs another feature tensor, ie. homogeneous For multiple feature tensor aggregation, the element-wise addition operator ① is typically used.
transformation.
Attention Learning Operations Oa. Inspired by recent exquisite designs of attention learning modules (Hu et al., 2018b; Li et al., 2018; Park et al., 2018), we form the O₂ by considering two types of attention learning prototypes: spatial-wise and channel-wise attentions. Specifically, spatial-wise attention operation learns a saliency map for an imput feature tensor in order to calibrate the importance of different spatial positions. In contrast, a channel-wise attention operation produces a vector of scaling factors from the aggregated global context of an input tensor for adaptively calibrating the channel dependency. To enforce attentive calibration on feature tensor, the element- wise multiplication operator ④ is typical choice for both spatial-wise and channel-wist attentions.
Dynamic Convolution Operations Os- Dynamic convolutions, designed for the sake of self- adaptation, generate dynamic kernel weights in accordance with the input feature tensor. It is often in form of depth-wise separable convolution as the feature transformation operation. Tailored for either NLP dense prediction tasks, existing dynamic convolution designs (Wu et al., 2019b; Jia et al., 2016) are not suitable for image classification (our focus) with different problem nature. hence needs to be reformulated in order to be effective for learning discriminative image representa- tions. We consider two design principles: (i) structurally lightweight whilst (ii) functionally strong with great modelling capability.
To that end, we present an exquisite dynamic con- volution structure specialised for cost-effective im- age classification, as shown in Fig. 1. Concretely, consists of three compact modules composed in an exquisite cooperation: (a) a bottleneck module, compress an input feature tensor by ratio of r: (b) a kernel transform module, to learn latent rep- resentations with kernel dimension of k a kernel decode module, to read out the dynamic kernel weights with the channel dimension same as the input feature tensor. This design is motivated, in part, by the long-range dependency modeling (Wang et al., 2018; Cao et al., 2019) and global context ag- gregation (Hu et al., 20;8);a). elegantly integrating their merits via unified formulation. For the output of dynamic convolutions, we consider two common kernel sizes: 3 and 5x 5. In a depth-wise manner, we apply a standard or dilated convolution operator to transform the input feature tensor. is notewor- thy to point out that, this type of convolutional kernel specific for each feature tensor of particular im- age sample (i.e. dynamic), rather than learned from a training dataset and fixed for all the input samples (i.e. static) as the conventional convolutional opera- tions in the feature transformation set.
of
L'kk
FR'
i-k
(c)
Ax
D
FR'
(a)
o
of
*
A
D3x
(b)
Fx
Figure/ 2: Heterogeneous Residual block for formulating the inner node computa- tion. (a) First-tier individual computation; (b) Second-tier collective computation.
Detailed implementations of self-calibration opera- tions are presented in Appendix A.I.
3.2 HETEROGENEOUS OPERATOR CELL
Due to different natures of heterogeneous computing capabilities, unification structure is needed for composing the primitive operations O = O, O. u Oa and aggregation/applicttoon operators C= 10, O @ in such way that their representation learning potentials can be well mined. To that end, we formulate a heterogeneous operator cell, a directed acyclic graph (DAG) G = =(V, E). join- ing conventional feature transformations and proposed self-calibration operations synergistically.
Formally, a heterogeneous operator cell consists of N ordered feature (tensor) nodes V> (Fx), <= k <= N). Following (Zoph et al., 2018), F, and Fz are the outputs from the previ- ous cells regarded as inpul nodes, NF+ denotes the inner nodes that perform computation,
4

---

Under review as a conference paper at ICLR 2020
and the N-th node FN is the cell output node formed the concatenation of all the inner nodes, i.c. FN concat( N-1). (=)))_)) The edge e (i, k.) E specifies the connection between the i-th and k-th nodes (the information flow i - k), associated with specific operation o² selected from the heterogeneous primitive operation set O. The key is to design a computing block for the inner nodes with heterogeneous computations.
Heterogeneous Residual Block. It is non-trivial to design a heterogencous computing block due to being nor straightforward centurr-tensortto-feee feature-tensor transformation as in the conventional homogencous operation. It involves self-calibrating the input feature tensor itself in addition to the homogeneous feature transformation. To facilitate adding the extra capacity, heterogeneous residual block (see Fig. 2) characterised by a surrogate node k' in the computing block associated with each inner node k, for enabling richer feature tensor manipulations. This is in a residual learning spirit (He et al., 2016), allowing to conduct self-calibration reliably.
formulate
Moreover, design two-fier computing hierarchy: the first tier for individual computation per input feature tensor to capture the specificity, and the second tier for collective computation on the all the input feature tensors feature tensors and the global input properties. The two tiers are connected by the surrogate node k.
whole capture the intrinsic structural relations between
Formally, we take input all the previous nodes (F < k), process them separately with hetero- geneous operations, and combine the processed results by summation (Fig. (a)):
FR OF (Fi), A
Σ
oa d (F,), Dg fo (F:) icck
where Fay, Ave and Dy are the three types of intermediate outputted tensors, i.c. features, atten- tions, and dynamic weights, respectively. These are subsequently aggregated into an intermediate calibrated tensor, i.e. the surrogate node FRs, using element-wise addition in-hetween on feature self-calibration and transformation as:
F FR
+
O +
Σ EDr
F @ D
(3)
feature
etfention
dynamic conv
Next, Fgr is used the input for the second-tier set-level collective computation (Fig. 2 (b)). Like- wisc, we consider the same three types operations:
FR =f - (Fx), A 0f k (Fwr), D2 = (F))
(4)
and form the inner node F5 via further feature self-calibration and transformation as:
F
F
F A
Σ
F
(5)
feature
attention
D.E.
dynamic
In doing so, heterogeneous residual block presents two-tier combinatorial operations structure for each inner node, resulting in a more expressive search space (see Section 4.2).
3.3 ATTENTION GUIDED SEARCH OPTIMISATION IN HETEROGENROUS SEARCH SPACE
To showcase the effectiveness the proposed heterogeneous search space and operator cell, adopt the proxy-based NAS strategy, due to the computing resource constraints and the enormous search space. This search is done by constructing small proxy network parametrised by O.
Attention Guided Search. Compared with proxyless search strategy, pbaxy-based NAS is more ef- ficient but relatively less optimal due not directly optimising the final network architecture. This training-test test discrepancy problem can be worsened when the search space provides more flexibil- ity and combinatorial capability, such the proposed space. To solve this obstacle, we propose attention guided search, which optimises the proxy network in knowledge distillation manner by injecting an external guidance from pre-trained teacher network into the NAS process.
5

---

Under review as a conference paper at ICLR 2020
Specifically, leverage the attention transfer idea (Zagoruyko & Komodakis, 2017) that en- courages student (the proxy network in our case) to hierarchically imitate a teacher's hid- den attention knowledge. Intuitively, this may benefit the search for self-calibration learning. Formally, let us denote feature tensor at the j-th stage of the teacher and student network as Fl.. and F.s. separately. Attention transfer is re- alised by imposing an alignment loss function across the two networks as:
To
T2
Air
AAT
Cir
Levant
Sg
Teacher
Proxy
O
Attention Transfer (AT)
Figure 3: Overview of attention guided search. T, and S (i E {0, 1, 2}) denote the i-th stage of the teacher and proxy (student) networks.
1 LAT
Σ JEJ
ll,, with 5/T = veel Σ F ((s)(():)0))))
(6)
2
|
where x S/STT is the spatial-wise accumulated feature vector. overview of attention guided search depicted in Fig.
Optimisation. For NAS optimisation, adopt the DARTS method (Liu et al., 2019b). In context, we conduct the continuous relaxation over all the possible heterogeneous operations for making continuous search space:
exp (a: a
o (z)=
Σ
o(x)
EVeO exp
where architecture vector a R|O| is used for each possible connection i j. We summarise the architecture vector of all the connections as a matrix A = [a'.... a e R @ With this relaxation, we can jointly optimise the architecture parameters A and the network weights @ in a fully gradient differentiable manner. Equipped with the proposed attention guidance search, the search objective function finally formulated as the following bilevel optimisation process:
(A) arg min Cirain ( A) ALAT ( A),
A = arg min Looi ( (*), A) AC* (A),
A
where À denotes the weighting syper-parameter For the first level Eq. (8). we learn the optimal pa- rameters @* for a given architecture A w.r.t a training objective Ltrain and the attention alignment loss LAT. The second level Eq. (9) then explores the optimal architecture A over the heteroge- neous search space w.r. a validation objective Lool and LAT. For image classification, Ltrain and Leat usually take the cross-entropy loss function.
Search Outcome. Once the above alternated optimisation is done, we derive an amenable cell architecture with heterogeneous operators. In practice, for each heterogeneous computing block retain the top-2 strongest incoming operations with at least one feature transformation operation for the first-tier (Fig. 2(a)), and the top-1 strongest operation for the second-tier (Fig. 2(b)).
4 EXPERIMENTS
We evaluated the proposed NOS method on image classification using three common datasets. CI- FAR10/100; Both CIFARIO and CIFAR 100 have 50K/10K train/test RGB images of size 32 32 x 3, categorised into 10 and 100 classes, respectively (Krizhevsky et al., 2009). ImageNet: We use the ILSVRC2012 version for large-scale image classification evaluation, containing 1. 1.28M training im- ages, 50K validation samples, and IK classes (Russakovsky et al., 2015).
We first conduct preliminary experiments erations O. To test the efficacy and transfcrability of NOS, we search the cell structures on CIFARIO only, and compare the performance with existing methods CIFAR10/100 and ImageNet.
DEFAR10//00 to select the heterogencous primitive op-

---

Under review as a conference paper at ICLR 2020
Table 1: Evaluating the feature self-calibration operations on CIFARIO and CIFARIO0.
CIFARIO
CIFARIO0
Model
Type
Kernels
FLOPS(M)
#Params(MB)
Top-I(%)
Top-3(%)
Top-I(%)
Top-5(%)
ResNet-18
4.95
0.22
23.61
7.16
555.42
11.17
3
4.63
0.13
22.63
6.44
+.885
+.003
Normal
5
4.78
0.14
23.45
6.82
+ 7.62
0.04
+ Dynamic
4.97
0.23
24.00
7.28
+ +.85
+.03
Dilated
5
4.92
0.17
23.75
7.20
+ 7.62
+.04
Spatial
4.79
0.16
23.51
7.04
+ 1.08
+.001
+ AAtenttio
Channel
4.83
0.19
23.20
6.89
+ 0.40
+.15
Better than the bascline.
Worse than the baseline.
4.1 PRELIMINARY STUDY OF FEATURE CALLIBAARIAAOO OPERATIONS
We conducted a controlled experiment to test the introduced self-calibration operations on CIFAR- 10 and CIFAR-I00. Specifically, for the proposed dynamic convolutions, we considered both normal and dilated convolutions and two kernel sizes (3 x and x 5). We adopted the channel-wise and spatial-wise attention learning. For the baseline model, we used ResNet-18 (He al., 2016) with 4 stages in the backbone. To build a model with self-calibration, we added each self-calibration fair comparison, we trained each
operation at the stages 1, 2, of ResNet-18, respectively. model in the same setting (see Appendix 1.2.1). In Table 1, we summarised the model parameters and FLOPs in addition to the test set performance (error rates). We observed that: (1) Both attention operations and normal dynamic convolutions outperform the baseline consistently; (2) Adding dilated dynamic convolutions causes performance drop in most cases. We hence exclude it from the candidate set; (3) Very marginal FLOPs and parameters increase from these self-calibration operations over the baseline, suggesting their high cost-effectiveness.
Normal Cell
Reduction Cell
fo
fo
ao
c_(k)
(k-1)
c_()
fz
f2
d
f1
(k-1)
3
fo
(k(k2)
d
fo
c (k-22
fo
f6
dp
2
Figure 4: Normal cell and reduction cell searched CIFAR-10, fo: sep_conv 3x33,,3, fa: sep_conv _onv f2: dil.conv.3323 fa: bax.pooling, fe: identity, spatial attention, channel attention, do: dynamic.ccom connvic.covv..oo 3x3, di: dynamic_cony 5x5.
4.2 CELL SEARCH
Search Space. found out above, the heterogeneous primitive operation set O contains 11 oper-
ations in total: |Oyl feature transformation operations, |Oa| 2 attention learning operations, OOa] = dynamic convolutions, respectively. We constructed the proposed heterogeneous operator cell (G (V,E)) with |V] 7 nodes (2 input nodes, 4 inner nodes and I ouiput node). So. all heterogeneous residual blocks contain ||| 18 edges in total (14 first-tier connections and 4 second- tier connections). To derive the final cell architecture, we kept first-tier connections and 1 second-
tier connection for each block. As result, there total number of I[ (x+1)n x 113 10
possible choices, orders of magnitude larger than the conventional size of (n+1)) 7 10 as in (Liu et al., 2019h; Dong & Yang, 2019; 9 Xie et al., 2019).
Training. Following the setup of existing methods (Real et al., 2019; Liu et al., 2019b; 2018a; Akimoto et al., 2019), searched the convolutional architectures on CIFAR10. We constructed

---

Under review as a conference paper at ICLR 2020
Table Comparisons with the state-of-the-arl architectures on CIFARIO and CIFARIO0.
Error (%)
Params Search Cost
Architecture
Type
CIFAR10
CIFAR100
(M)
GPUs Days
PyramidNet (Han et al., 2017)*
3.92
20.11
2.5
Manual
DenseNet-BC (Huang et al., 2017)
3,46
17.18
25.6
Manual
NASNet-A (Zoph et al., 2018)
2.63
3.3
450
1800
RI.
AmoebaNet-B (Real et al., 2019)
0.05
2.8
450
3150
EA
Hicrarchical--volttiin (Liu et al., 2018b)
0.15
15.7
200
300
EA
PNAS (Liu et al., 2018a)
0.40
3.2
100
1.5
SMBO
ENAS (Pham et al., 2018)
2.89
4.6
0.5
RL
ProxylesSNAS Cai et al. (2019)
2.08
5.7
4
GD
RENAS (Chen et al., 2019)
2.88+0.02
3.5
4
6
EA&RL
DARTS((st) (Liu ct al., 2019b)
3.00
3.3
1.5
GD
DARTS(2nd) (Liu et al., 2019b)
0.769
17.54
3.3
4.0
GD
SNAS (moderate) (Xie et al., 2019)
0.02
2.8
1.5
GD
GHN (Zhang al., 2019)
0.007
5.7
0.84
GD
GDAS (Dong & Yang, 2019)
2.93
18.38
3.4
0.84
GD
BayesNAS(000055 (Zhou al., 2019)
0.004
3.4
0.2
GD
ASNG (Akimoto et al., 2019)
0.13+
3.9
0.11
GD
Random Bascline
3,85
21.66
2.4
Random
NOS (best)
2.53
16.21
2.6
0.35
GD
NOS (average)
0.67
1..7
2.6
0.35
GD
The teacher model. Best architecture among 30 random samples.
small proxy network with heterogeneous operator cells, and two reduction cells at 1/3 and 2/3 of the total network depth for feature shape reduction. We used 25K images split from the training set for validation. We randomly initialised the architecture parameters A E R 18xx11 in the normal distri- bution. We used pre-trained PyramidNet- model. We set the weight
(bottleneck, 84) (Han et al., 2017) the teacher 10s for attention guidance loss A After epochs of training on the proxy network, we derived the final heterogeneous operator cells from the architecture matrix A. See Appendix 1.2.1 for more configurations for training the proxy and teacher networks.
The search CIFARIO took only 8.4 hours using single NVIDIA Tesla V100 GPU. The searched heterogeneous operator cells by NOS is shown in Fig. 4, in which the self-calibration operators O and @ appear in both first-tier and second-tier. For example, there two attention operations in first-tier and two dynamic convolutions in second-tier in the normal cell.
4.3 ARCHITECTURE EVALUATION
CIFAR. To measure the final image classification performance of the searched heterogencous op- crator cells on CIFAR1O and CIFAR100, we created an evaluation network with 20 cells, 36 initial channels, and auxiliary tower with loss weight 0.4. See Appendix A.2.1 for more configurations for training the evaluation network. Due to high variance of results on CIFAR, we conducted 10 independent runs and reported both the best and average results. We summarised the results of NOS and the state-of-the-art methods in Table 2. The comparisons show that: (1) NOS achieves a very competitive result (second best) on CIFARIO, whilst enjoying the smallest model parameters (only 2.6M). Comparing with the best performer ProxylessNAS at the size of 5.7M (searched with su- pernet) (Cai al., 2019). shows the significant cost-effectiveness and compactness advantages of our method. (2) Despite significantly larger search space (10 vs 109 in (Liu et al., 2019b; Akimoto et al., 2019; Xie et al., 2019; Dong & Yang, 2019; Akimoto et al., 2019)), NOS shows high cost-effectiveeess in computing cost (only 0.35 GPU day). (3) NOS achieves the best result on CIFAR 100 by directly transferring the CIFARIO 10 searched network, significantly outperforming DARTS (Liu al., 2019b) and GDAS (Dong & Yang, 2019). This challenging cross-dataset test indicates a superior transferability the network searched by NOS.
ImageNet. To evaluate the transferability of architecture discovered by NOS large scale Ima- geNet, we used the mobile setting same as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019), where the number of multiply-add operations is restricted to be less than 600M at the input
8

---

Under review as a conference paper at ICLR 2020
Table 3: Comparisons with the state-of-the-ar architectures on ImageNet-mobile.
Architecture
Test Err. (%)
Params
X+
Search Cost
Type
top-1
top-5
(M)
(M)
(GPU-days)
MobileNet-v1((1O) (Howard et al., 2017)
29.4
10.5
4.2
575
Manual
MobileNettvv((1O) (Sandler et al., 2018)
28.0
3.4
300
Manual
ShuffleNet (x (v1) (Zhang et al., 2018)
26.4
10.2
R:5
524
Manual
ShuffleNet 2x (Ma et al., 2018)
25.1
x:5
591
Manual
NASNet-A (Zoph et al., 2018)
26.0
8.4
5.3
564
1800
RL
NASNet-B (Zoph al., 2018)
27.2
8.7
5.3
488
1800
RL
NASNet-C (Zoph al., 2018)
27.5
9.0
4.9
558
1800
RL
PNAS (Liu et al., 2018a)
25.8
8.1
5.1
588
1.5
SMBO
AmocbaNet-A (Real et al., 2019)
25.5
8.0
5.1
555
3150
EA
AmoebaNet-B (Real et al., 2019)
26.0
8.5
5.3
555
3150
EA
AmocbaNet-CC (Real et al., 2019)
24.3
7.6
6.4
570
3150
EA
RENAS (Chen et al., 2019)
24.3
7.4
5.4
580
6
EA&RL
MnasNet-A3 (Tan et al., 2019)
23.3
6.7
5.2
403
_f
RL
ProxylessNAS (GPU) (Cai et al., 2019)
24.9
7.5
7.1
465
8.3
GD
FBNet-C (Wu et al., 2019a)
25.1
5.5
375
9.0
GD
GHN (Zhang
2019)
27.0
8.7
6.1
569
0.84
GD
DARTS (Liu et al., 2019b)
26.7
8.7
4.7
574
4.0
GD
SNAS (Xie al., 2019)
27.3
9.2
4.3
522
1.5
GD
GDAS (Dong & Yang, 2019)
26.0
8.5
5,3
581
0.84
GD
BayesNAS (0.005) (Zhou al., 2019)
26.5
8.9
3.9
0.2
GD
NOS (searched on CIFARIO)
25.8
8.1
4.0
440
0.35
GD
The architecture search takes 4.5 days 64 TPUv2 devices.
size of 224 x 224. Specifically, we constructed an evaluation network with 14 cells and 48 initial channels. An auxiliary tower with loss weight 0.4 was also applied. See Appendix A.2.2 for more training details. Table 3 shows the ImageNet results the mobile setting. Notably, the cell architec- tures found by NOS on CIFAR 10 can achieve highly competitive performance with significantly less computational cost (0.35 day on GPU vs 4.5 days using 64 TPUv2 devices required by MnasNet- A3 (Tan et al., 2019)). Unlike MnasNet-A3 (Tan et al., 2019) and ProxylessNAS (Cai et al., 2019) searching the network on ImageNet directly tresource-intensiin CIFAR10 can be successfully transferred. Also, compared other oftte- art gradient based proxy-based NAS (GHN, DARTS, SNAS, GDAS and BayesNAS), NOS discovers a cell structure that performs better with higher efficiency (only 440M FLOPs).
the network searched by NOS on
4.4
FURTHER ANALYSIS
100
train (w/ AT)
val (w/ AT)
train (w/o AT)
Top-1(%6)
90
val (w/o AT)
Table Testing attention guided search (AGS).
80
AGS w/o
Test Error (%)
CIFAR1O
CIFAR100
3,44
18.80
70
w/
2.53
16.21
10
15
20
Epoch
Figure 5: Train/Val set accuracy on CIFARIO.
We evaluated attention guided search (AGS) CIFAR10/100 by comparing NOS variant without attention transfer loss. The same training setting was used (Appendix A. We used a PyramidNet- 10 teacher. Table 4 shows that learning with attention guidance can significantly benefit the NOS search process. We further showed the training curves in Fig. 5 and observed that
'The absence of AGS gives slight difference the search GPU day 0.34 0.35 when using AGS.

---

Under review as a conference paper at ICLR 2020
AGS clearly improves the train/val accuracies. This suggests that AGS is effective alleviate the architecture training-ter test discrepancy issue involved in the proxy-based NAS.
CONCLUSION
We presented Neural Operator Search (NOS), featured by a heterogeneous search space for neural architecture search (NAS). This search space expansion enables NAS to discover more expressive and previously undiscovered architectures, significantly expanding the search horizon and enriching the possible search outcomes. We further formulated heterogeneous residual block and attention guided search to solve the intrinsic search challenges involved. Extensive experiments on image classification show that NOS can discover novel and high-quality cell architectures in cost-effective process. We hope that this work will shed light on the future directions for the NAS community.
REFERENCES
Youhei Akimoto, Shinichi Shirakawa, Nozomu Yoshinari, Kento Uchida, Shota Saito, and Kouhei Nishida. Adaptive stochastic natural gradient method for one-shot neural architecture search. In ICML, 2019.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In NeurlPS, pp. 523-531, 2016.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Yue Cao. Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squecze-excitation networks and beyond. arXiv, 2019.
Yukang Chen, Gaofeng Meng, Qian Zhang, Shiming Xiang, Chang Huang, Lisen Mu, and Xinggang Wang. Renas: Reinforced evolutionary neural architecture search. 2019.
CVPR, pp. 4787-4796,
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv, 2017.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR, pp. 1761-1770 2019.
Golnaz Ghiasi, Tsung-Yi Lin, and Quoe V. Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. In CVPR, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv, 2017.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In CVPR, pp- 5927-5935, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv, 2017.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In NewrIPS, pp. 9401-9411, 2018a.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, pp. 7132-7141, 2018b.
10

---

Under review as a conference paper at ICLR 2020
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In CVPR, pp. 4700-4708, 2017.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In NeurlPS, pp. 667-675, 2016.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich, Fractalnet: Ultra-deep neural net- works without residuals. 2017.
Wei Li, Xiatian Zhu, and Shaogang Gong. identification. In CVPR, pp. 2285-2294, 2018.
Harmonious attention network for person
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy, Progressive neural architecture search. ECCV, pp. 19- 19-34, 2018a.
Chenxi Liu. Liang-Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmenta- tion. In CVPR, 2019a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hi- erarchical representations for efficient architecture search. In ICLR, 2018b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019b.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In NeurlPS, pp. 7816-7827, 2018
Ningning Ma, Xiangyu Zhang. Hai-Tar Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, pp. 116-131, 2018.
Vladimir Nekrasov, Hao Chen, Chunhua Shen, and lan Reid. Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In CVPR, 2019.
Jongchan Park, Sanghyun Woo, Joon- Young Lee, and In So Kweon. Bam: Bottleneck attention module. In BMVC, 2018.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-22) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chie -hiih Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-,520, 2018.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, pp. 2820- 2828, 2019.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. CVPR, pp. 3156-3164, 2017.
11

---

Under review as a conference paper at ICLR 2020
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR. pp. 7794-7803, 2018.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-avee efficient convnet design via differentiable neural architecture search. In CVPR, pp. 10734-10742, 2019a.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019b.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In ICLR, 2019.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast
optimization, network minimization and transfer learning. In CVPR, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor-
mance of convolutional neural networks via attention transfer. ICLR, 2017.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In ICLR, 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, pp. 6848-6856, 2018.
Hongpeng Zhou, Minghao Yang. Jun Wang, and Wei Pan. neural architecture search. In ICML, 2019.
Bayesnas: A bayesian approach for
Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and Jifeng Dai. An empirical study of spatial attention mechanisms deep networks. arXiv, 2019.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In /CLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, pp. 8697-8710, 2018.
A APPENDIX
A.1 SELF-CALIRRATION OPERATIONS
A.1.1 ATTENTION LEARNING
import torch
import torch.nn an
import torch.nn nn. functional as F
channel-wt -wine attention class AttentionC kan.Modulee
def init _niii_int_l______ (self, C Ciin, cut,, reduction=16, affine=True):
10
super (AttentionC, self). init_()
self.conv_1 cn.conva C_in // reduction, 1, stride=1, padding -0, bias=False)
12
self.relu nn. Rn..ee
13
self.ccnv_2 = nn. ConCoon2d // reduction, __out, 1, stride=1,
padding=0, bias=False)
self.sigm nn.Sigmoid
15
16
def forward(sel:, x):
17
y F.avg_pool2d (x, kernel_sizen size() [2:4])
y self.relu (self www.vcoo.........oooooooooooo (y))
12

---

Under review as a conference paper at ICLR 2020
y self. seli.sigm(self.comm (y))
21
return y
22
23
spatial-wisc attention
class AttentionS (nn. (nn.Mueeee))
26
def _ _inniin__s___ (self, C_in, C_out, stride, reduction-16, affine-True):
27
super (Attentions, self) .ini)__.iii____
28
self.conv_1 Co..2d in, C_in // reduction, 1, stride=stride,
padding-0, bias=Walse)
30
self.bn_l nn, mm.Battshhoorm2dlC..in // reduction, affine=affine)
31
self.relu (m..ee.uiippllaeetaall
32
self.conv_2 2 (C. in // reduction, 1, 3, stride=1, padding=l,
bias=False)
33
self.sigm nn. Sigmooid
34
35
def fsrward x):
y relf. (sel..covv_ (x)
38
y self.
return [self.conv_]] (y))
39
A.1.2 DYNAMIC CONVOLUTIONS
dynamic convolution
class DynamicF (nn. Module)
def ini__intsssll C_in, C_out, F_size, reduction-8, affine-True)=
super super (Dyynaicc, self) initt(())
self.f_size F_size
self.reduction reduction
self.c_in C_in
self.c_cut C_out
self.conv_1 (C C_in // reduction, 1, stride=1, padding
=0, bias=Faise)
11
self.bn_l nn. (C // reduction, affine-affine)
12
self.relu (i.Rel.lingglaaeetaal
self.conv_2 (n. F_size=F_size, 1, stride=1, padding-0,
bias=Talse)
14
self.soft nn. Son.Softmax (dim=2)
15
self.ccnv_3 = nn. Conv2d // reduction, __t 1, stride=1,
padding=0, bias-Falsc)
17
def forward x):
18
19
input_x = relf. (self.bn.) (self. [onll. )
N, C, H, w innpnt_t.xsse sisuut_xxsiielll
input_x input_x.view (N, C, W)
IN,
W]
22
input_x xuput_xxunsqueeze (1)
IN, 1, C,
23
dynamic mask = self.conv
IN, -1, H
W
24
dynamic_mask dynamic_mask.vieww
-1, H W)
[N, -1,
H
W]
dynamic_mask
(dynaaic__aai]]
[N, -1,
B
26
dynamic_mask dynamic__aakk wws.ueeeeeeeeuueeeeeeeeee (3)
[N, -1,
H
11
27
dynamic_mask maak dynamic_mmak.peermute (0, 3, 2, 1)
[N, 1, + W1 -11
28
dynamic torch. matmml
dynamic (ypamic_mask))
[N, 1, C, -1]
dynamic = dynamic. squeeze (1)
[N, C, -1]
dynamic dynamic.view (N, C, self.f_size self self.f_size)
31
dynamic selffconv_3 (dynamic)
32
dynamic dynamic.vien (N, self. celf,c_out, -1)
33
dynamic = self, soft (dynamic)
dynamic dynamis view fiee
(N, self. c_ouut self f_siiii felf.f_s1
return dynamic
13

---

Under review as a conference paper at ICLR 2020
A.2
DETAILS OF TRAINING CONFIGURATIONS
A.2.1 CIFAR
ResNet-18 and PyramidNet-110. We trained these models for 300 epochs with batch size 32. The learning rate was initialised as 0.025, which was decayed by 10 every 30 epochs. The standard SGD optimiser with momentum of 0.9 was employed. We set weight decay value of 1 10- to avoid overfitting. Other additional enhancements were not involved except the standard data augmentations.
Cell Search. For network parameters θ of proxy network, used SGD with an initial learning rate 0.025 and set the momentum value as 0.9. This learning rate was decayed to 0 with cosine scheduler. A weight decay value of 3 x 10-4 was imposed to avoid over-fitting. For learning architecture matrix A. we used the Adam optimiser with fixed learning rate value 6 x 10 and the weight decay
10-3
Cell Evaluation. The evaluation network trained from scratch directly for 600 epochs with batch size 128. Note that, the attention transfer was not involved for training. We set the weight decay values for CIFAR-I0 and CIFAR- to 3 10-4 and 5 10 'individullly. The standard SGD optimiser with momentum of 0.9 was applied. The initial learning rate was 0.25, decayed with a cosine scheduler. Following existing works (Liu et al., 2019b; Pham al., 2018; Zoph et al., 2018; Real et al., 2019), we performed two additional enhancements: the cutout regularisation (DeVries & Taylor, 2017) with length and the drop-path (Larsson et al., 2017) of probability 0.3.
A.2.2 IMAGENET
We trained the evaluation model for ImageNet using SGD optimiser for 300 epochs with batch size 512. We initialised the learning rate 0.25 and reduced it to 0 by linear scheduler. Learning rate warmup (Goyal et al., 2017) was applied for the first 5 epochs to deal with the large batch size and learning rate.
A.3 VISUALISATION OF SEARCH SPACE
F
XM
of *
comiana 2tride=2
comdar3 2tridem2
Stride=2
ImageNet
comv2x3
Normal Cell
Reduction Cell
CIFAR
Fx
Figure 6: (Left) The overall model architecture for CIFAR-10 and ImageNet, consisting repeated Normal Cells and Reduction Cells. M is the stacking choice for the number of Normal Cells. Each cell contains blocks. (Right) An example of two-tier block construction in cell: Each block takes two input features (F,, F2) from previous nodes; The operator (?) in block is determined by the choices of two operations (o' -kk in the first-tier; An extra operation (o selected in the second-tier.
A.4 DISTILLATION EFFECT ATTENTION GUIDED SEARCH
We examined the effect of knowledge distillation in the proposed Attention Guided Search (AGS). We conducted this analysis on CIFARIO. In this evaluation, we compared three methods: (1) Vanilla Search: Using the original DARTS search method; (2) Distillation Only: Using the attention transfer loss for training the proxy network only; (3) AGS: The proposed method (full). We tracked the model performance both training (train) and validation (val) data sets. Figure 7 shows that (i)
14

---

Under review as a conference paper at ICLR 2020
100
Top-1(%)
80
train (AGS)
val (AGS)
70
train (Vanilla Search)
val (Vanilla Search)
train (Distillation Only)
val (Distillation Only)
60
5
10
15
20
25
Epoch
Figure The train and val set accuracies on CIFARIO different search processes.
knowledge distillation brings positive performance gain the vanilla DARTS and (ii) using attention guidance for the network search can further improve the searched architecture.
A.5 GENERALITY OF ATTENTION GUIDED SEARCH
We tested the general effect of the proposed Attention Guided Search (AGS) using the original DARTS search space (O₃ + zero operation) CIFARIO. During the search process, followed the same settings as DARTS with the first-order optimisation. We obtained the error rates: 3.00 + 0.14 (DARTS) 2.92 0.05 (DARTS AGS). This suggests general efficacy of AGS over different search spaces.
15


Reviews:
[{"review_id": "rylEwLw6Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a complicated NAS approach, with a lot more ops, a larger search space and an extra teacher network. However, the results on CIFAR10 and ImageNet are both not competitive to other SOTA results.\n\nMy major concern is whether the extra complexity introduced by this paper is necessary:\n\n1. The new search space includes a lot more ops and optimizations, which by themselves should improve other models. For example, according to Table1, dynamic conv and attention improve ResNet-18 without any architecture search. What if you simply apply the same optimizations to other SOTA models in Table 2 and 3?\n2. The extra ops already make the comparison in Table 2/3 unfair. Despite that, NOS is still not competitive to other SOTA results (prroxylessNAS on CIFAR-10 and MnasNet-A3 on ImageNet).\n3. The authors argue that “NOS shows significant compactness advantages than proxylessNAS”, but it is somewhat misleading. By reading this paper, it seems the search algorithm used in this paper aims to find the highest accuracy model WITHOUT resource constraints, so smaller model size should not be related to the search algorithm.\n\nHere are a few other comments and suggestions:\n\n4. Section 3.2 is difficult to follow. I recommend the authors providing some visualization for the search space (see NASNet paper for example).\n5. Ablation study is relatively weak: for example, Figure 5 compares w/ and w/o attention-guided search, but it is unclear whether the gain is by the search or by the teacher distillation. A more fair comparison is to w/o attention-guided search but also perform the distillation. It would be also helpful to verify whether the propose attention-guided search can work for existing  search space (such as the NASNet/DARTS search space).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklNz2NCYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for neural architecture search that searches architectures with not only feature transformational components but also self-calibration and dynamic convolution ones. Although the idea seems to be straightforward and innovating, it is difficult to assess the effectiveness of the proposed approach comparing to other methods just by looking at the experimental results. \n\nLooking at the CIFAR10 results, the authors claimed that their method achieved second best among all the method compared but with significantly less number of parameters. Although indeed the number of parameters differ by a lot, the error rates are also differed significantly. It might be worth comparing these two baselines using the same number of parameters just to conduct a fair comparison.\n\nIn the Imagenet results, the proposed method is behind many of the state-of-the-art methods, casting concerns on the effectiveness of the proposed approach considering its added search space. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJg7yneLtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nOften in (neural architecture search) NAS papers on vision datasets, only feature transform operations (e.g. convolution, pooling, etc) are considered while operations like attention and dynamic convolution are left out. These operations have shown increasing performance improvements across many domains and datasets. This paper attempts to incorporate these operations into the search space of NAS.\n\nSpecifically they design a new cell (residual block in a resnet backbone) which contains these operations (traditionally used feature transforms, attention, and dynamic convolutions). This new cell contains two-tiers. The first tier separately computes the three kinds of operations and combines the results via elementwise summation to form an 'intermediate calibrated tensor'. This is then fed to the second tier where again the three kinds of operations are performed on them and they net output is again formed via elementwise summation. \n\nFor the search algorithm the authors use the bilevel optimization of DARTS. The one difference to aid in the search is that an 'attention-guided' transfer mechanism is used where a teacher network trained on the larger dataset (like ImageNet) is used to align the intermediate activations of the proxy network found during search, by adding a alignment loss function. \n\nResults of search with this new search space on cifar10 and transfer to cifar100, imagenet are presented. It is found via manual ablation in resnet backbone that dilated dynamic convolutions dont help and are dropped from the search space. \n\nThe numerical results are near state-of-the-art although as is pervasive in the NAS field actual fair comparisons between methods are hard to get due to differences in search space, hardware space, stochasticity in training etc. But the authors do a best-attempt and have included random search and best and average performances so that is good. \n\nComments:\n\n- The paper is generally well-written and easy to understand (thanks!)\n\n- Experiments are generally thorough. \n\n- The main novelty is the design of the cell such that attention and dynamic convolution operations can be incorporated. I was hoping that the results in terms of error would be much better due to the more expressive search space but they are not at the moment.\n\n- I am curious how the attention guided alignment loss has been ablated. How do the numbers look without alignment loss keeping gpu training time constant? Basically I am trying to figure out how much is the contribution of the new search space vs. the addition of attention guided search. Do we have a negative result (an important one though) that attention and dynamic convolutions don't really help?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension of the search space of neural architecture search to include dynamic convolutions, teacher nets among others. The method is evaluated on CIFAR-10 and Imagenet with a similar setup as other architecture search methods. The reviewers found that the results did not convincingly show that the proposed improvements were better than other ways of improving neural architecture search such as rroxylessNAS.
Decision: Reject