Year: 2019

Abstract:
Existing neural architecture search (NAS) methods explore a limited featuretransformation-only search space, ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions. This disables the NAS algorithms to discover more advanced network architectures. We address this limitation by additionally exploiting feature self-calibration operations, resulting in a heterogeneous search space. To solve the challenges of operation heterogeneity and significantly larger search space, we formulate a neural operator search (NOS) method. NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in a unified structure, and an attention guided search strategy for facilitating the search process over a vast space. Extensive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.

heading: 1 INTRODUCTION
text: Recent advances of Neural Architecture Search (NAS) are remarkable in challenging tasks, e.g. image classification (Zoph & Le, 2017), object detection (Ghiasi et al., 2019), and semantic segmentation (Liu et al., 2019a; Nekrasov et al., 2019), greatly alleviating the demands for human knowledge and interventions by automating the laborious process of designing neural network architectures. One common scheme for the standard proxy-based neural architecture search methods (Pham et al., 2018; Zoph et al., 2018; Liu et al., 2019b) is to factorise the search space via repeatedly stacking the same cell structure, within which a computing block generates an output tensor Fk by combining the transformations of two input feature tensors Fi and Fj : Fk “ oiÑk pFiq ‘ ojÑk pFjq s.t. i ă k & j ă k, (1) where oiÑk and ojÑk are the i-th and j-th primitive operations for feature transformation, selected from a candidate operation set O, and ‘ is the element-wise addition. Existing NAS methods use only the standard feature learning/transformation operations (convolution, pooling and identity mapping) as the building components.
Besides, extensive studies (Hu et al., 2018b; Bertinetto et al., 2016; Wang et al., 2017; Jia et al., 2016; Wu et al., 2019b; Zhu et al., 2019) have proven that other advanced operations for feature self-calibration, such as attention learning and dynamic convolutions, can bring great benefits for representation learning. For example, Hu et al. (2018b) proposes Squeeze-and-Excitation Networks to explicitly model inter-dependencies between channels by learning channel-wise self-attention. Jia et al. (2016) presents Dynamic Filter Networks to generate context-aware filters for increasing the flexibility and adaptiveness of networks. However, these useful feature calibration elements have never been well exploited in NAS, significantly limiting the potentials of NAS which aims for automatically discovering more sophisticated and advanced network architectures without human engineering.
In this work, we aim to address this limitation by extending the search space of NAS with feature self-calibration operations for scaling up the search boundary. This makes a heterogeneous search space. Consequently, the way of feature tensor interaction and combination is dramatically diversified, from the conventional addition operator‘ only to the combination of addition‘, multiplication d for attention modelling, and dynamic convolution ~. In this regard,we call the proposed method Neural Operator Search (NOS).
Such a search space enhancement is critical since NAS is enabled to explore stronger and previously undiscovered network architectures, which opens a door to potentially take the NAS research to the
next level. In the no free lunch saying, this also comes with two new challenges: (i) It is non-trivial and more challenging to assemble such heterogeneous tensors and operations (i.e. features, attentions and dynamic weights) in a unified computing block, as compared to the conventional homogeneous feature-tensor-to-feature-tensor transformation; (ii) The search space increases exponentially which leads to a much harder NAS problem.
To address the first challenge, we formulate a heterogeneous operator cell characterised by a novel heterogeneous residual block. This block, formulated in a residual learning spirit (He et al., 2016), is designed specially for fusing all the different types of tensors and operations synergistically. To solve the second challenge, we propose leveraging the attention transfer (Zagoruyko & Komodakis, 2017) idea to facilitate the search behaviour across this significantly larger network space via following the attention guidance of a pretrained teacher model. As we will show, this guidance not only makes the search more efficient but also improves the search result.
Our contributions in this work are: (1) We present a novel heterogeneous search space for NAS characterised by richer primitive operations including both conventional feature transformations and newly introduced feature self-calibration. This breaks the conventional selection limit of candidate neural networks and enables the NAS process to find stronger architectures, many of which are impossible to be discovered in the conventional space. This opens new territories for supporting stronger NAS algorithms and new possibilities for most expressive architectures ever to be revealed. (2) We formulate a novel Neural Operator Search (NOS) method dedicated for NAS in the proposed heterogeneous search space, with a couple of key designs – heterogeneous residual block for fusing different types of tensor operations synergistically and attention guided search for facilitating the search process over a vast search space more efficiently and more effectively. (3) With extensive comparisons to the state-of-the-art NAS methods, the experiments show that our approach is highly competitive on both CIFAR and ImageNet-mobile image classification tests.

heading: 2 RELATED WORK
text: Neural Architecture Search. Since the seminal work by Zoph & Le (2017), neural architecture search has gained a surge of interest, effectively replacing laborious human designs by the computational process. From the strategy point of view, NAS methods can be categorised into two types: (1) proxy-based (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018; Liu et al., 2019b) and (2) proxy-less (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019a) NAS. Specifically, to alleviate the computational cost during search, the proxy-based NAS methods search for building cells on proxy tasks, with one or more of following compromised strategies: starting with fewer cells; using a smaller dataset (e.g. CIFAR-10); learning with fewer epochs. Then, to transfer to the large-scale target task, one can build a network by stacking searched cells without further exploration. However, suffering from lacking of directness and specialisation, the searched cells by proxy-based NAS methods are not guaranteed to be optimal on the target task. In contrast, proxy-less NAS methods directly learns architectures on a target task by starting with an over-parameterised network (supernet) that contains all possible paths, in which the redundant paths are pruned to derive the optimised architecture. Notwithstanding significant better results than proxy-based approaches, proxy-less NAS methods require massive computational cost and GPU memory assumption, due to learning with the vast-size supernet. From the optimisation point of view, existing NAS methods usually fall into three groups: reinforcement learning (RL) based methods, evolutionary algorithm (EA) based methods, and gradient differentiable (GD) methods. In particular, RL-based NAS methods (Zoph & Le, 2017; Pham et al., 2018; Tan et al., 2019) control the selection of architecture component in a sequential order with policy networks. EA-based NAS methods (Real et al., 2019; Liu et al., 2018b) employ the validation accuracies to guide the evolution of a population of initialised architectures. RL- an EA-based NAS methods usually suffer from low efficiency and high computational resource demand, due to the fundamental searching challenge in a discrete space. In contrast, GD-based NAS methods (Liu et al., 2019b; Xie et al., 2019; Luo et al., 2018) conduct searching over a continuous space by relaxation or mapping, substantially reducing the search cost to a few GPU days. Whilst varying in the algorithmic aspects, all these works commonly explore the feature-transformationonly search spaces without more diverse and advanced operations as we investigate here. To show the NAS potential of the proposed richer search space with self-calibration learning operations, we take the efficient proxy-based GD optimisation due to the resource constraint.
Self-Calibration. Self-calibration is a type of mechanism enabling a network to dynamically perform input-conditional self-adjustment, which has been studied extensively in both the computer vision (Hu et al., 2018b; Jia et al., 2016; Li et al., 2018; Park et al., 2018) and natural language processing (NLP) literature (Wu et al., 2019b; Vaswani et al., 2017). There are two typical paradigms of self-calibration: self-attention learning and dynamic convolutions, realised via an element-wise multiplication operator d and a dynamic convolution operator ~, respectively. Despite showing significant efficacy, self-calibration is only exploited independently after architecture hand-design (Hu et al., 2018b) or auto-search (Tan et al., 2019). We move a step further by fully exploring the potential of self-calibration along with feature transformation in joint optimisation, bringing a richer search space for neural architecture search.
Knowledge Distillation. There are recent works that use knowledge distillation to help computer vision and NLP tasks. Three types of knowledge are usually considered in distillation: features (Yim et al., 2017), attention (Zagoruyko & Komodakis, 2017), and predictions (Hinton et al., 2015). We leverage the attention distillation with a different objective – alleviating the intrinsic training-test discrepancy issue of the proxy-based NAS strategy, particularly with a more expressive search space. This represents a novel exploitation of attention distillation (Zagoruyko & Komodakis, 2017).

heading: 3 METHOD
text: In this section, we start by formulating a heterogeneous search space for NAS (Section 3.1), followed by a dedicated heterogeneous operator cell to enable composing the heterogeneous operations in a unified computing block with synergistic interaction and cooperation (Section 3.2). To overcome the intrinsic architecture discovery challenges from more expressive search space, we further develop an attention guided search scheme (Section 3.3).
3.1 HETEROGENEOUS SEARCH SPACE
To enrich the NAS search space so that more advanced network architectures can be discovered, we introduce a heterogeneous search space A that considers three different types of representation learning capabilities: (1) Feature transformations; (2) Attention learning; and (3) Dynamic convolutions. More concretely, we form three sets of primitive computing operations that produce features, attentions and dynamic weights, respectively. This novel search space generalises the conventional counterpart which is limited to the first type of operations (Liu et al., 2019b; Pham et al., 2018), and incorporates the self-calibration learning capabilities (i.e. the second and third types) in NAS. Importantly, while the search space changes, the generic search strategies still apply therefore being largely open for collaborating with existing NAS methods. For instance, in the proxy-based NAS strategy we may first search for a computing cell with heterogeneous operations as the building block and then form the final network architecture by sequentially stacking multiple such cells layer-by-layer.
Next, let us describe the heterogeneous primitive operation set O which consists of the following three disjoint subsets: Of , Oa and Od, along with their aggregation or application operators.
Feature Transformation Operations Of . We adopt the feature transformation/learning operation set Of same as in Liu et al. (2019b; 2018a), including the following 7 operations: 3 ˆ 3 and 5 ˆ 5 separable convolutions, 3 ˆ 3 and 5 ˆ 5 dilated separable convolutions, 3 ˆ 3 average pooling, 3 ˆ 3 max pooling, and identity. Every operation of P Of takes as input a feature tensor and
outputs another feature tensor, i.e. homogeneous feature-tensor-to-feature-tensor transformation. For multiple feature tensor aggregation, the element-wise addition operator ‘ is typically used. Attention Learning Operations Oa. Inspired by recent exquisite designs of attention learning modules (Hu et al., 2018b; Li et al., 2018; Park et al., 2018), we form the Oa by considering two types of attention learning prototypes: spatial-wise and channel-wise attentions. Specifically, a spatial-wise attention operation learns a saliency map for an input feature tensor in order to calibrate the importance of different spatial positions. In contrast, a channel-wise attention operation produces a vector of scaling factors from the aggregated global context of an input tensor for adaptively calibrating the channel dependency. To enforce attentive calibration on feature tensor, the elementwise multiplication operator d is a typical choice for both spatial-wise and channel-wise attentions. Dynamic Convolution Operations Od. Dynamic convolutions, designed for the sake of selfadaptation, generate dynamic kernel weights in accordance with the input feature tensor. It is often in form of depth-wise separable convolution as the feature transformation operation. Tailored for either NLP or dense prediction tasks, existing dynamic convolution designs (Wu et al., 2019b; Jia et al., 2016) are not suitable for image classification (our focus) with different problem nature. It hence needs to be reformulated in order to be effective for learning discriminative image representations. We consider two design principles: (i) structurally lightweight whilst (ii) functionally strong with great modelling capability.
To that end, we present an exquisite dynamic convolution structure specialised for cost-effective image classification, as shown in Fig. 1. Concretely, it consists of three compact modules composed in an exquisite cooperation: (a) a bottleneck module, to compress an input feature tensor by a ratio of r; (b) a kernel transform module, to learn latent representations with a kernel dimension of k ˆ k; (c) a kernel decode module, to read out the dynamic kernel weights with the channel dimension same as the input feature tensor. This design is motivated, in part, by the long-range dependency modeling (Wang et al., 2018; Cao et al., 2019) and global context aggregation (Hu et al., 2018b;a), elegantly integrating their merits via a unified formulation. For the output of dynamic convolutions, we consider two common kernel sizes: 3ˆ3 and 5ˆ5. In a depth-wise manner, we apply a standard or dilated convolution operator ~ to transform the input feature tensor. It is noteworthy to point out that, this type of convolutional kernel is specific for each feature tensor of a particular image sample (i.e. dynamic), rather than learned from a training dataset and fixed for all the input samples (i.e. static) as the conventional convolutional operations in the feature transformation set.
Detailed implementations of self-calibration operations are presented in Appendix A.1.

heading: 3.2 HETEROGENEOUS OPERATOR CELL
text: Due to different natures of heterogeneous computing capabilities, a unification structure is needed for composing the primitive operations O “ Of YOa YOd and aggregation/application operators C “ t‘, d, ~u in such a way that their representation learning potentials can be well mined. To that end, we formulate a heterogeneous operator cell, a directed acyclic graph (DAG) G “ pV, Eq, joining conventional feature transformations and proposed self-calibration operations synergistically.
Formally, a heterogeneous operator cell consists of N ordered feature (tensor) nodes V “ tFk|, 1 ă“ k ă“ Nu. Following (Zoph et al., 2018), F1 and F2 are the outputs from the previous cells regarded as two input nodes, tFkuN´1k“3 denotes the inner nodes that perform computation,
and the N -th node FN is the cell output node formed as the concatenation of all the inner nodes, i.e. FN “concatptFkuN´1k“3 q. The edge eiÑk“pi, kq PE specifies the connection between the i-th and k-th nodes (the information flow i Ñ k), associated with a specific operation oiÑk selected from the heterogeneous primitive operation set O. The key is to design a computing block for the inner nodes with heterogeneous computations.
Heterogeneous Residual Block. It is non-trivial to design a heterogeneous computing block due to being not straightforward feature-tensor-to-feature-tensor transformation as in the conventional homogeneous operation. It involves self-calibrating the input feature tensor itself in addition to the homogeneous feature transformation. To facilitate adding the extra capacity, we formulate a heterogeneous residual block (see Fig. 2) characterised by a surrogate node k1 in the computing block associated with each inner node k, for enabling richer feature tensor manipulations. This is in a residual learning spirit (He et al., 2016), allowing to conduct self-calibration reliably.
Moreover, we design a two-tier computing hierarchy: the first tier for individual computation per input feature tensor to capture the specificity, and the second tier for collective computation on the set of all the input feature tensors as a whole to capture the intrinsic structural relations between feature tensors and the global input properties. The two tiers are connected by the surrogate node k1.
Formally, we take as input all the previous nodes tFi|, i ă ku, process them separately with heterogeneous operations, and combine the processed results by summation (Fig. 2 (a)):
Fk1 “ ÿ iăk oiÑk
1
f pFiq, Ak1“ ÿ iăk oiÑk
1 a pFiq, Dk1 “ ! oiÑk 1 d pFiq )
iăk (2)
where Fk1 , Ak1 , and Dk1 are the three types of intermediate outputted tensors, i.e. features, attentions, and dynamic weights, respectively. These are subsequently aggregated into an intermediate calibrated tensor, i.e. the surrogate node Fk1 , using element-wise addition in-between on feature self-calibration and transformation as:
Fk1 “ Fk1 lomon
feature
‘ pFk1 d Ak1q looooomooooon
attention
‘ ÿ
Dk1PDk1 Fk1 ~ Dk1 looooooooomooooooooon
dynamic conv
(3)
Next, Fk1 is used as the input for the second-tier set-level collective computation (Fig. 2 (b)). Likewise, we consider the same three types of operations:
Fk “ ok 1Ñk f pFk1q, Ak “ ok 1Ñk a pFk1q, Dk “ ! ok 1Ñk d pFk1q ) , (4)
and form the inner node Fk via further feature self-calibration and transformation as:
Fk “ Fk lomon
feature
‘ pFk d Akq loooomoooon
attention
‘ ÿ
DkPDk Fk ~ Dk looooooomooooooon
dynamic conv
(5)
In doing so, our heterogeneous residual block presents a two-tier combinatorial operations structure for each inner node, resulting in a more expressive search space (see Section 4.2).

heading: 3.3 ATTENTION GUIDED SEARCH OPTIMISATION IN A HETEROGENEOUS SEARCH SPACE
text: To showcase the effectiveness of the proposed heterogeneous search space and operator cell, we adopt the proxy-based NAS strategy, due to the computing resource constraints and the enormous search space. This search is done by constructing a small proxy network parametrised by Θ.
Attention Guided Search. Compared with proxyless search strategy, proxy-based NAS is more efficient but relatively less optimal due to not directly optimising the final network architecture. This training-test discrepancy problem can be worsened when the search space provides more flexibility and combinatorial capability, such as the proposed space. To solve this obstacle, we propose attention guided search, which optimises the proxy network in a knowledge distillation manner by injecting an external guidance from a pre-trained teacher network into the NAS process.
Specifically, we leverage the attention transfer idea (Zagoruyko & Komodakis, 2017) that encourages a student (the proxy network in our case) to hierarchically imitate a teacher’s hidden attention knowledge. Intuitively, this may benefit the search for self-calibration learning. Formally, let us denote a feature tensor at the j-th stage of the teacher and student network as FjT and F j S , separately. Attention transfer is realised by imposing an alignment loss function across the two networks as:
LAT “ 1
2
ÿ jPJ } x
j S
}xjS}2 ´ x
j T
}xjT }2 }2, with xjS{T “ vecp
ÿ
i
|FjS{T p¨, ¨, iq| 2q (6)
where xjS{T is the spatial-wise accumulated feature vector. An overview of attention guided search is depicted in Fig. 3.
Optimisation. For NAS optimisation, we adopt the DARTS method (Liu et al., 2019b). In our context, we conduct the continuous relaxation over all the possible heterogeneous operations O for making a continuous search space:
oiÑjpxq “ ÿ
oPO
exp ` aiÑjo ˘
ř o1PO exp ´ aiÑjo1 ¯opxq, (7)
where an architecture vector aiÑjo P R|O| is used for each possible connection iÑ j. We summarise the architecture vector of all the connections as a matrix A “ “ a1, ¨ ¨ ¨ ,a|E| ‰
P R|E|ˆ|O|. With this relaxation, we can jointly optimise the architecture parameters A and the network weights Θ in a fully gradient differentiable manner. Equipped with the proposed attention guidance search, the search objective function is finally formulated as the following bilevel optimisation process:
Θ˚pAq “ arg min Θ LtrainpΘ,Aq ` λLAT pΘ,Aq, (8)
A˚ “ arg min A LvalpΘ˚pAq,Aq ` λLAT pΘ˚pAq,Aq, (9)
where λ denotes the weighting hyper-parameter. For the first level Eq. (8), we learn the optimal parameters Θ˚ for a given architecture A w.r.t a training objective Ltrain and the attention alignment loss LAT . The second level Eq. (9) then explores the optimal architecture A˚ over the heterogeneous search space A w.r.t a validation objective Lval and LAT . For image classification, Ltrain and Lval usually take the cross-entropy loss function. Search Outcome. Once the above alternated optimisation is done, we derive an amenable cell architecture with heterogeneous operators. In practice, for each heterogeneous computing block we retain the top-2 strongest incoming operations with at least one feature transformation operation for the first-tier (Fig. 2(a)), and the top-1 strongest operation for the second-tier (Fig. 2(b)).

heading: 4 EXPERIMENTS
text: We evaluated the proposed NOS method on image classification using three common datasets. CIFAR10/100: Both CIFAR10 and CIFAR100 have 50K/10K train/test RGB images of size 32ˆ32ˆ3, categorised into 10 and 100 classes, respectively (Krizhevsky et al., 2009). ImageNet: We use the ILSVRC2012 version for large-scale image classification evaluation, containing 1.28M training images, 50K validation samples, and 1K classes (Russakovsky et al., 2015).
We first conduct preliminary experiments on CIFAR10/100 to select the heterogeneous primitive operations O. To test the efficacy and transferability of NOS, we search the cell structures on CIFAR10 only, and compare the performance with existing methods on CIFAR10/100 and ImageNet.

heading: 4.1 PRELIMINARY STUDY OF FEATURE SELF-CALIBRATION OPERATIONS
text: We conducted a controlled experiment to test the introduced self-calibration operations on CIFAR10 and CIFAR-100. Specifically, for the proposed dynamic convolutions, we considered both normal and dilated convolutions and two kernel sizes (3 ˆ 3 and 5 ˆ 5). We adopted the channel-wise and spatial-wise attention learning. For the baseline model, we used ResNet-18 (He et al., 2016) with 4 stages in the backbone. To build a model with self-calibration, we added each self-calibration operation at the stages 1, 2, 3 of ResNet-18, respectively. For fair comparison, we trained each model in the same setting (see Appendix A.2.1). In Table 1, we summarised the model parameters and FLOPs in addition to the test set performance (error rates). We observed that: (1) Both attention operations and our normal dynamic convolutions outperform the baseline consistently; (2) Adding dilated dynamic convolutions causes performance drop in most cases. We hence exclude it from the candidate set; (3) Very marginal FLOPs and parameters increase from these self-calibration operations over the baseline, suggesting their high cost-effectiveness.

heading: 4.2 CELL SEARCH
text: Search Space. As found out above, the heterogeneous primitive operation set O contains 11 operations in total: |Of | “ 7 feature transformation operations, |Oa| “ 2 attention learning operations, |Od| “ 2 dynamic convolutions, respectively. We constructed the proposed heterogeneous operator cell (G “ pV, Eq) with |V| “ 7 nodes (2 input nodes, 4 inner nodes and 1 output node). So, all 4 heterogeneous residual blocks contain |E | “ 18 edges in total (14 first-tier connections and 4 secondtier connections). To derive the final cell architecture, we kept 2 first-tier connections and 1 secondtier connection for each block. As a result, there is a total number of
ś4 n“1 pn`1qn 2 ˆ 11 3 « 1014
possible choices, 5 orders of magnitude larger than the conventional size of ś4 n“1 pn`1qn 2 ˆ7 2 « 109 as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019).
Training. Following the setup of existing methods (Real et al., 2019; Liu et al., 2019b; 2018a; Akimoto et al., 2019), we searched the convolutional architectures on CIFAR10. We constructed a
small proxy network with 8 heterogeneous operator cells, and two reduction cells at 1/3 and 2/3 of the total network depth for feature shape reduction. We used 25K images split from the training set for validation. We randomly initialised the architecture parameters A P R18ˆ11 in the normal distribution. We used a pre-trained PyramidNet-110 (bottleneck, α “ 84) (Han et al., 2017) as the teacher model. We set the weight λ “ 103 for attention guidance loss LAT . After 25 epochs of training on the proxy network, we derived the final heterogeneous operator cells from the architecture matrix A. See Appendix A.2.1 for more configurations for training the proxy and teacher networks.
The search on CIFAR10 took only 8.4 hours using a single NVIDIA Tesla V100 GPU. The searched heterogeneous operator cells by NOS is shown in Fig. 4, in which the self-calibration operators d and ~ appear in both first-tier and second-tier. For example, there are two attention operations in first-tier and two dynamic convolutions in second-tier in the normal cell.

heading: 4.3 ARCHITECTURE EVALUATION
text: CIFAR. To measure the final image classification performance of the searched heterogeneous operator cells on CIFAR10 and CIFAR100, we created an evaluation network with 20 cells, 36 initial channels, and an auxiliary tower with loss weight 0.4. See Appendix A.2.1 for more configurations for training the evaluation network. Due to high variance of results on CIFAR, we conducted 10 independent runs and reported both the best and average results. We summarised the results of NOS and the state-of-the-art methods in Table 2. The comparisons show that: (1) NOS achieves a very competitive result (second best) on CIFAR10, whilst enjoying the smallest model parameters (only 2.6M). Comparing with the best performer ProxylessNAS at the size of 5.7M (searched with a supernet) (Cai et al., 2019), it shows the significant cost-effectiveness and compactness advantages of our method. (2) Despite a significantly larger search space (1014 vs 109 in (Liu et al., 2019b; Akimoto et al., 2019; Xie et al., 2019; Dong & Yang, 2019; Akimoto et al., 2019)), NOS shows high cost-effectiveness in computing cost (only 0.35 GPU day). (3) NOS achieves the best result on CIFAR100 by directly transferring the CIFAR10 searched network, significantly outperforming DARTS (Liu et al., 2019b) and GDAS (Dong & Yang, 2019). This challenging cross-dataset test indicates a superior transferability of the network searched by NOS.
ImageNet. To evaluate the transferability of architecture discovered by NOS on large scale ImageNet, we used the mobile setting same as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019), where the number of multiply-add operations is restricted to be less than 600M at the input

heading: Architecture Test Err. (%) Params ˆ` Search Cost Type
text: size of 224 ˆ 224. Specifically, we constructed an evaluation network with 14 cells and 48 initial channels. An auxiliary tower with loss weight 0.4 was also applied. See Appendix A.2.2 for more training details. Table 3 shows the ImageNet results in the mobile setting. Notably, the cell architectures found by NOS on CIFAR10 can achieve highly competitive performance with significantly less computational cost (0.35 day on 1 GPU vs 4.5 days using 64 TPUv2 devices required by MnasNetA3 (Tan et al., 2019)). Unlike MnasNet-A3 (Tan et al., 2019) and ProxylessNAS (Cai et al., 2019) searching the network on ImageNet directly (resource-intensive), the network searched by NOS on CIFAR10 can be successfully transferred. Also, compared to other state-of-the-art gradient based proxy-based NAS (GHN, DARTS, SNAS, GDAS and BayesNAS), NOS discovers a cell structure that performs better with higher efficiency (only 440M FLOPs).

heading: 4.4 FURTHER ANALYSIS
text: Figure 5: Train/Val set accuracy on CIFAR10.
Table 4: Testing attention guided search (AGS).
AGS Test Error (%)CIFAR10 CIFAR100 w/o 3.44 18.80 w/ 2.53 16.21
We evaluated attention guided search (AGS) on CIFAR10/100 by comparing a NOS variant without attention transfer loss. The same training setting was used (Appendix A.2.1)1. We used a pre-trained PyramidNet-110 as teacher. Table 4 shows that learning with attention guidance can significantly benefit the NOS search process. We further showed the training curves in Fig. 5 and observed that
1The absence of AGS gives a slight difference to the search cost: GPU day 0.34 vs. 0.35 when using AGS.
AGS clearly improves the train/val accuracies. This suggests that AGS is effective to alleviate the architecture training-test discrepancy issue involved in the proxy-based NAS.

heading: 5 CONCLUSION
text: We presented Neural Operator Search (NOS), featured by a heterogeneous search space for neural architecture search (NAS). This search space expansion enables NAS to discover more expressive and previously undiscovered architectures, significantly expanding the search horizon and enriching the possible search outcomes. We further formulated heterogeneous residual block and attention guided search to solve the intrinsic search challenges involved. Extensive experiments on image classification show that NOS can discover novel and high-quality cell architectures in a cost-effective process. We hope that this work will shed light on the future directions for the NAS community.

heading: A APPENDIX

heading: A.1 SELF-CALIBRATION OPERATIONS

heading: A.1.1 ATTENTION LEARNING
text: 1 import torch 2 import torch.nn as nn 3 import torch.nn.functional as F 4
5 6 # channel-wise attention 7 class AttentionC(nn.Module): 8
9 def __init__(self, C_in, C_out, reduction=16, affine=True): 10 super(AttentionC, self).__init__() 11 self.conv_1 = nn.Conv2d(C_in, C_in // reduction, 1, stride=1, padding =0, bias=False) 12 self.relu = nn.ReLU(inplace=False) 13 self.conv_2 = nn.Conv2d(C_in // reduction, C_out, 1, stride=1, padding=0, bias=False) 14 self.sigm = nn.Sigmoid() 15 16 def forward(self, x): 17 18 y = F.avg_pool2d(x, kernel_size=x.size()[2:4]) 19 y = self.relu(self.conv_1(y))
20 y = self.sigm(self.conv_2(y)) 21 return y 22 23 # spatial-wise attention 24 class AttentionS(nn.Module): 25 26 def __init__(self, C_in, C_out, stride, reduction=16, affine=True): 27 super(AttentionS, self).__init__() 28 29 self.conv_1 = nn.Conv2d(C_in, C_in // reduction, 1, stride=stride, padding=0, bias=False) 30 self.bn_1 = nn.BatchNorm2d(C_in // reduction, affine=affine) 31 self.relu = nn.ReLU(inplace=False) 32 self.conv_2 = nn.Conv2d(C_in // reduction, 1, 3, stride=1, padding=1, bias=False) 33 self.sigm = nn.Sigmoid() 34 35 def forward(self, x): 36 37 y = self.relu(self.bn_1(self.conv_1(x))) 38 y = self.sigm(self.conv_2(y)) 39 return y

heading: A.1.2 DYNAMIC CONVOLUTIONS
text: 1 # dynamic convolution 2 class DynamicF(nn.Module): 3 4 def __init__(self, C_in, C_out, F_size, reduction=8, affine=True): 5 super(DynamicF, self).__init__() 6 self.f_size = F_size 7 self.reduction = reduction 8 self.c_in = C_in 9 self.c_out = C_out
10 self.conv_1 = nn.Conv2d(C_in, C_in // reduction, 1, stride=1, padding =0, bias=False) 11 self.bn_1 = nn.BatchNorm2d(C_in // reduction, affine=affine) 12 self.relu = nn.ReLU(inplace=False) 13 self.conv_2 = nn.Conv2d(C_in, F_size*F_size, 1, stride=1, padding=0, bias=False) 14 self.soft = nn.Softmax(dim=2) 15 self.conv_3 = nn.Conv2d(C_in // reduction, C_out, 1, stride=1, padding=0, bias=False) 16 17 def forward(self, x): 18 19 input_x = self.relu(self.bn_1(self.conv_1(x))) 20 N, C, H, W = input_x.size() 21 input_x = input_x.view(N, C, H * W) # [N, C, H * W] 22 input_x = input_x.unsqueeze(1) # [N, 1, C, H * W] 23 dynamic_mask = self.conv_2(x) # [N, -1, H , W] 24 dynamic_mask = dynamic_mask.view(N, -1, H * W) # [N, -1, H * W] 25 dynamic_mask = self.soft(dynamic_mask) # [N, -1, H * W] 26 dynamic_mask = dynamic_mask.unsqueeze(3) # [N, -1, H * W, 1] 27 dynamic_mask = dynamic_mask.permute(0, 3, 2, 1) # [N, 1, H * W, -1] 28 dynamic = torch.matmul(input_x, dynamic_mask) # [N, 1, C, -1] 29 dynamic = dynamic.squeeze(1) # [N, C, -1] 30 dynamic = dynamic.view(N, C, self.f_size, self.f_size) 31 dynamic = self.conv_3(dynamic) 32 dynamic = dynamic.view(N, self.c_out, -1) 33 dynamic = self.soft(dynamic) 34 dynamic = dynamic.view(N, self.c_out, self.f_size, self.f_size) 35 return dynamic

heading: A.2 DETAILS OF TRAINING CONFIGURATIONS

heading: A.2.1 CIFAR
text: ResNet-18 and PyramidNet-110. We trained these models for 300 epochs with batch size 32. The learning rate was initialised as 0.025, which was decayed by 10 every 30 epochs. The standard SGD optimiser with momentum of 0.9 was employed. We set a weight decay value of 1 ˆ 10´4 to avoid overfitting. Other additional enhancements were not involved except the standard data augmentations.
Cell Search. For network parameters Θ of proxy network, we used SGD with an initial learning rate 0.025 and set the momentum value as 0.9. This learning rate was decayed to 0 with a cosine scheduler. A weight decay value of 3 ˆ 10´4 was imposed to avoid over-fitting. For learning architecture matrix A, we used the Adam optimiser with a fixed learning rate value 6 ˆ 10´4 and set the weight decay to 1ˆ 10´3.
Cell Evaluation. The evaluation network was trained from scratch directly for 600 epochs with batch size 128. Note that, the attention transfer was not involved for training. We set the weight decay values for CIFAR-10 and CIFAR-100 to 3 ˆ 10´4 and 5 ˆ 10´4 individually. The standard SGD optimiser with a momentum of 0.9 was applied. The initial learning rate was 0.25, decayed to 0 with a cosine scheduler. Following existing works (Liu et al., 2019b; Pham et al., 2018; Zoph et al., 2018; Real et al., 2019), we performed two additional enhancements: the cutout regularisation (DeVries & Taylor, 2017) with length 16 and the drop-path (Larsson et al., 2017) of probability 0.3.
A.2.2 IMAGENET
We trained the evaluation model for ImageNet using SGD optimiser for 300 epochs with batch size 512. We initialised the learning rate as 0.25 and reduced it to 0 by a linear scheduler. Learning rate warmup (Goyal et al., 2017) was applied for the first 5 epochs to deal with the large batch size and learning rate.
A.3 VISUALISATION OF SEARCH SPACE

heading: A.4 DISTILLATION EFFECT IN ATTENTION GUIDED SEARCH
text: We examined the effect of knowledge distillation in the proposed Attention Guided Search (AGS). We conducted this analysis on CIFAR10. In this evaluation, we compared three methods: (1) Vanilla Search: Using the original DARTS search method; (2) Distillation Only: Using the attention transfer loss for training the proxy network only; (3) AGS: The proposed method (full). We tracked the model performance on both training (train) and validation (val) data sets. Figure 7 shows that (i)
knowledge distillation brings a positive performance gain over the vanilla DARTS and (ii) using attention guidance for the network search can further improve the searched architecture.

heading: A.5 GENERALITY OF ATTENTION GUIDED SEARCH
text: We tested the general effect of the proposed Attention Guided Search (AGS) using the original DARTS search space (Of + zero operation) on CIFAR10. During the search process, we followed the same settings as DARTS with the first-order optimisation. We obtained the error rates: 3.00 ˘ 0.14 (DARTS) vs. 2.92 ˘ 0.05 (DARTS + AGS). This suggests a general efficacy of AGS over different search spaces.

Review summaries:
Often in (neural architecture search) NAS papers on vision datasets, only feature transform operations (e.g. convolution, pooling, etc) are considered while operations like attention and dynamic convolution are left out. These operations have shown increasing performance improvements across many domains and datasets. This paper attempts to incorporate these operations into the search space of NAS.