Title: No title found

Abstract: Existing neural architecture search (NAS) methods explore a limited featuretransformation-only search space, ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions. This disables the NAS algorithms to discover more advanced network architectures. We address this limitation by additionally exploiting feature self-calibration operations, resulting in a heterogeneous search space. To solve the challenges of operation heterogeneity and significantly larger search space, we formulate a neural operator search (NOS) method. NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in a unified structure, and an attention guided search strategy for facilitating the search process over a vast space. Extensive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.

1. INTRODUCTION:

Recent advances of Neural Architecture Search (NAS) are remarkable in challenging tasks, e.g. image classification (Zoph & Le, 2017) , object detection (Ghiasi et al., 2019) , and semantic segmentation (Liu et al., 2019a; Nekrasov et al., 2019) , greatly alleviating the demands for human knowledge and interventions by automating the laborious process of designing neural network architectures. One common scheme for the standard proxy-based neural architecture search methods (Pham et al., 2018; Zoph et al., 2018; Liu et al., 2019b) is to factorise the search space via repeatedly stacking the same cell structure, within which a computing block generates an output tensor F k by combining the transformations of two input feature tensors F i and F j :

EQUATION

where o iÑk and o jÑk are the i-th and j-th primitive operations for feature transformation, selected from a candidate operation set O, and ' is the element-wise addition. Existing NAS methods use only the standard feature learning/transformation operations (convolution, pooling and identity mapping) as the building components.

Besides, extensive studies (Hu et al., 2018b; Bertinetto et al., 2016; Wang et al., 2017; Jia et al., 2016; Wu et al., 2019b; Zhu et al., 2019) have proven that other advanced operations for feature self-calibration, such as attention learning and dynamic convolutions, can bring great benefits for representation learning. For example, Hu et al. (2018b) proposes Squeeze-and-Excitation Networks to explicitly model inter-dependencies between channels by learning channel-wise self-attention. Jia et al. (2016) presents Dynamic Filter Networks to generate context-aware filters for increasing the flexibility and adaptiveness of networks. However, these useful feature calibration elements have never been well exploited in NAS, significantly limiting the potentials of NAS which aims for automatically discovering more sophisticated and advanced network architectures without human engineering.

In this work, we aim to address this limitation by extending the search space of NAS with feature self-calibration operations for scaling up the search boundary. This makes a heterogeneous search space. Consequently, the way of feature tensor interaction and combination is dramatically diversified, from the conventional addition operator ' only to the combination of addition ', multiplication d for attention modelling, and dynamic convolution . In this regard,we call the proposed method Neural Operator Search (NOS).

Such a search space enhancement is critical since NAS is enabled to explore stronger and previously undiscovered network architectures, which opens a door to potentially take the NAS research to the next level. In the no free lunch saying, this also comes with two new challenges: (i) It is non-trivial and more challenging to assemble such heterogeneous tensors and operations (i.e. features, attentions and dynamic weights) in a unified computing block, as compared to the conventional homogeneous feature-tensor-to-feature-tensor transformation; (ii) The search space increases exponentially which leads to a much harder NAS problem.

To address the first challenge, we formulate a heterogeneous operator cell characterised by a novel heterogeneous residual block. This block, formulated in a residual learning spirit (He et al., 2016) , is designed specially for fusing all the different types of tensors and operations synergistically. To solve the second challenge, we propose leveraging the attention transfer (Zagoruyko & Komodakis, 2017) idea to facilitate the search behaviour across this significantly larger network space via following the attention guidance of a pretrained teacher model. As we will show, this guidance not only makes the search more efficient but also improves the search result.

Our contributions in this work are: (1) We present a novel heterogeneous search space for NAS characterised by richer primitive operations including both conventional feature transformations and newly introduced feature self-calibration. This breaks the conventional selection limit of candidate neural networks and enables the NAS process to find stronger architectures, many of which are impossible to be discovered in the conventional space. This opens new territories for supporting stronger NAS algorithms and new possibilities for most expressive architectures ever to be revealed.

(2) We formulate a novel Neural Operator Search (NOS) method dedicated for NAS in the proposed heterogeneous search space, with a couple of key designs -heterogeneous residual block for fusing different types of tensor operations synergistically and attention guided search for facilitating the search process over a vast search space more efficiently and more effectively.

(3) With extensive comparisons to the state-of-the-art NAS methods, the experiments show that our approach is highly competitive on both CIFAR and ImageNet-mobile image classification tests.

2. RELATED WORK:

Neural Architecture Search. Since the seminal work by Zoph & Le (2017) , neural architecture search has gained a surge of interest, effectively replacing laborious human designs by the computational process. From the strategy point of view, NAS methods can be categorised into two types:

(1) proxy-based (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018; Liu et al., 2019b) and

(2) proxy-less (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019a) NAS. Specifically, to alleviate the computational cost during search, the proxy-based NAS methods search for building cells on proxy tasks, with one or more of following compromised strategies: starting with fewer cells; using a smaller dataset (e.g. CIFAR-10); learning with fewer epochs. Then, to transfer to the large-scale target task, one can build a network by stacking searched cells without further exploration. However, suffering from lacking of directness and specialisation, the searched cells by proxy-based NAS methods are not guaranteed to be optimal on the target task. In contrast, proxy-less NAS methods directly learns architectures on a target task by starting with an over-parameterised network (supernet) that contains all possible paths, in which the redundant paths are pruned to derive the optimised architecture. Notwithstanding significant better results than proxy-based approaches, proxy-less NAS methods require massive computational cost and GPU memory assumption, due to learning with the vast-size supernet. From the optimisation point of view, existing NAS methods usually fall into three groups: reinforcement learning (RL) based methods, evolutionary algorithm (EA) based methods, and gradient differentiable (GD) methods. In particular, RL-based NAS methods (Zoph & Le, 2017; Pham et al., 2018; Tan et al., 2019) control the selection of architecture component in a sequential order with policy networks. EA-based NAS methods (Real et al., 2019; Liu et al., 2018b) employ the validation accuracies to guide the evolution of a population of initialised architectures. RL-an EA-based NAS methods usually suffer from low efficiency and high computational resource demand, due to the fundamental searching challenge in a discrete space. In contrast, GD-based NAS methods (Liu et al., 2019b; Xie et al., 2019; Luo et al., 2018) conduct searching over a continuous space by relaxation or mapping, substantially reducing the search cost to a few GPU days. Whilst varying in the algorithmic aspects, all these works commonly explore the feature-transformationonly search spaces without more diverse and advanced operations as we investigate here. To show the NAS potential of the proposed richer search space with self-calibration learning operations, we take the efficient proxy-based GD optimisation due to the resource constraint.

Self-Calibration. Self-calibration is a type of mechanism enabling a network to dynamically perform input-conditional self-adjustment, which has been studied extensively in both the computer vision (Hu et al., 2018b; Jia et al., 2016; Li et al., 2018; Park et al., 2018) and natural language processing (NLP) literature (Wu et al., 2019b; Vaswani et al., 2017) . There are two typical paradigms of self-calibration: self-attention learning and dynamic convolutions, realised via an element-wise multiplication operator d and a dynamic convolution operator , respectively. Despite showing significant efficacy, self-calibration is only exploited independently after architecture hand-design (Hu et al., 2018b) or auto-search (Tan et al., 2019) . We move a step further by fully exploring the potential of self-calibration along with feature transformation in joint optimisation, bringing a richer search space for neural architecture search.

Knowledge Distillation. There are recent works that use knowledge distillation to help computer vision and NLP tasks. Three types of knowledge are usually considered in distillation: features (Yim et al., 2017) , attention (Zagoruyko & Komodakis, 2017) , and predictions (Hinton et al., 2015) .

We leverage the attention distillation with a different objective -alleviating the intrinsic training-test discrepancy issue of the proxy-based NAS strategy, particularly with a more expressive search space. This represents a novel exploitation of attention distillation (Zagoruyko & Komodakis, 2017) .

3. METHOD:

In this section, we start by formulating a heterogeneous search space for NAS (Section 3.1), followed by a dedicated heterogeneous operator cell to enable composing the heterogeneous operations in a unified computing block with synergistic interaction and cooperation (Section 3.2). To overcome the intrinsic architecture discovery challenges from more expressive search space, we further develop an attention guided search scheme (Section 3.3). To enrich the NAS search space so that more advanced network architectures can be discovered, we introduce a heterogeneous search space A that considers three different types of representation learning capabilities: (1) Feature transformations;

3.1. HETEROGENEOUS SEARCH SPACE:

(2) Attention learning; and (3) Dynamic convolutions. More concretely, we form three sets of primitive computing operations that produce features, attentions and dynamic weights, respectively. This novel search space generalises the conventional counterpart which is limited to the first type of operations (Liu et al., 2019b; Pham et al., 2018) , and incorporates the self-calibration learning capabilities (i.e. the second and third types) in NAS. Importantly, while the search space changes, the generic search strategies still apply therefore being largely open for collaborating with existing NAS methods. For instance, in the proxy-based NAS strategy we may first search for a computing cell with heterogeneous operations as the building block and then form the final network architecture by sequentially stacking multiple such cells layer-by-layer.

Next, let us describe the heterogeneous primitive operation set O which consists of the following three disjoint subsets: O f , O a and O d , along with their aggregation or application operators.

Feature Transformation Operations O f . We adopt the feature transformation/learning operation set O f same as in Liu et al. (2019b; 2018a) , including the following 7 operations: 3 ˆ3 and 5 ˆ5 separable convolutions, 3 ˆ3 and 5 ˆ5 dilated separable convolutions, 3 ˆ3 average pooling, 3 ˆ3 max pooling, and identity. Every operation o f P O f takes as input a feature tensor and outputs another feature tensor, i.e. homogeneous feature-tensor-to-feature-tensor transformation. For multiple feature tensor aggregation, the element-wise addition operator ' is typically used.

Attention Learning Operations O a . Inspired by recent exquisite designs of attention learning modules (Hu et al., 2018b; Li et al., 2018; Park et al., 2018) , we form the O a by considering two types of attention learning prototypes: spatial-wise and channel-wise attentions. Specifically, a spatial-wise attention operation learns a saliency map for an input feature tensor in order to calibrate the importance of different spatial positions. In contrast, a channel-wise attention operation produces a vector of scaling factors from the aggregated global context of an input tensor for adaptively calibrating the channel dependency. To enforce attentive calibration on feature tensor, the elementwise multiplication operator d is a typical choice for both spatial-wise and channel-wise attentions.

Dynamic Convolution Operations O d . Dynamic convolutions, designed for the sake of selfadaptation, generate dynamic kernel weights in accordance with the input feature tensor. It is often in form of depth-wise separable convolution as the feature transformation operation. Tailored for either NLP or dense prediction tasks, existing dynamic convolution designs (Wu et al., 2019b; Jia et al., 2016) are not suitable for image classification (our focus) with different problem nature. It hence needs to be reformulated in order to be effective for learning discriminative image representations. We consider two design principles: (i) structurally lightweight whilst (ii) functionally strong with great modelling capability. To that end, we present an exquisite dynamic convolution structure specialised for cost-effective image classification, as shown in Fig. 1 . Concretely, it consists of three compact modules composed in an exquisite cooperation: (a) a bottleneck module, to compress an input feature tensor by a ratio of r; (b) a kernel transform module, to learn latent representations with a kernel dimension of k ˆk; (c) a kernel decode module, to read out the dynamic kernel weights with the channel dimension same as the input feature tensor. This design is motivated, in part, by the long-range dependency modeling (Wang et al., 2018; Cao et al., 2019) and global context aggregation (Hu et al., 2018b; a) , elegantly integrating their merits via a unified formulation. For the output of dynamic convolutions, we consider two common kernel sizes: 3ˆ3 and 5ˆ5. In a depth-wise manner, we apply a standard or dilated convolution operator to transform the input feature tensor. It is noteworthy to point out that, this type of convolutional kernel is specific for each feature tensor of a particular image sample (i.e. dynamic), rather than learned from a training dataset and fixed for all the input samples (i.e. static) as the conventional convolutional operations in the feature transformation set.

Detailed implementations of self-calibration operations are presented in Appendix A.1.

3.2. HETEROGENEOUS OPERATOR CELL:

Due to different natures of heterogeneous computing capabilities, a unification structure is needed for composing the primitive operations O " O f Y O a Y O d and aggregation/application operators C " t', d, u in such a way that their representation learning potentials can be well mined. To that end, we formulate a heterogeneous operator cell, a directed acyclic graph (DAG) G " pV, Eq, joining conventional feature transformations and proposed self-calibration operations synergistically.

Formally, a heterogeneous operator cell consists of N ordered feature (tensor) nodes V " tF k |, 1 ă" k ă" N u. Following (Zoph et al., 2018) , F 1 and F 2 are the outputs from the previous cells regarded as two input nodes, tF k u N ´1 k"3 denotes the inner nodes that perform computation, and the N -th node F N is the cell output node formed as the concatenation of all the inner nodes, i.e. F N " concatptF k u N ´1 k"3 q. The edge e iÑk " pi, kq P E specifies the connection between the i-th and k-th nodes (the information flow i Ñ k), associated with a specific operation o iÑk selected from the heterogeneous primitive operation set O. The key is to design a computing block for the inner nodes with heterogeneous computations.

Heterogeneous Residual Block. It is non-trivial to design a heterogeneous computing block due to being not straightforward feature-tensor-to-feature-tensor transformation as in the conventional homogeneous operation. It involves self-calibrating the input feature tensor itself in addition to the homogeneous feature transformation. To facilitate adding the extra capacity, we formulate a heterogeneous residual block (see Fig. 2 ) characterised by a surrogate node k 1 in the computing block associated with each inner node k, for enabling richer feature tensor manipulations. This is in a residual learning spirit (He et al., 2016) , allowing to conduct self-calibration reliably.

Moreover, we design a two-tier computing hierarchy: the first tier for individual computation per input feature tensor to capture the specificity, and the second tier for collective computation on the set of all the input feature tensors as a whole to capture the intrinsic structural relations between feature tensors and the global input properties. The two tiers are connected by the surrogate node k 1 . Formally, we take as input all the previous nodes tF i |, i ă ku, process them separately with heterogeneous operations, and combine the processed results by summation (Fig. 2 (a) ):

EQUATION

where F k 1 , A k 1 , and D k 1 are the three types of intermediate outputted tensors, i.e. features, attentions, and dynamic weights, respectively. These are subsequently aggregated into an intermediate calibrated tensor, i.e. the surrogate node F k 1 , using element-wise addition in-between on feature self-calibration and transformation as:

EQUATION

Next, F k 1 is used as the input for the second-tier set-level collective computation (Fig. 2 (b) ). Likewise, we consider the same three types of operations:

EQUATION

and form the inner node F k via further feature self-calibration and transformation as:

EQUATION

In doing so, our heterogeneous residual block presents a two-tier combinatorial operations structure for each inner node, resulting in a more expressive search space (see Section 4.2).

3.3. ATTENTION GUIDED SEARCH OPTIMISATION IN A HETEROGENEOUS SEARCH SPACE:

To showcase the effectiveness of the proposed heterogeneous search space and operator cell, we adopt the proxy-based NAS strategy, due to the computing resource constraints and the enormous search space. This search is done by constructing a small proxy network parametrised by Θ.

Attention Guided Search. Compared with proxyless search strategy, proxy-based NAS is more efficient but relatively less optimal due to not directly optimising the final network architecture. This training-test discrepancy problem can be worsened when the search space provides more flexibility and combinatorial capability, such as the proposed space. To solve this obstacle, we propose attention guided search, which optimises the proxy network in a knowledge distillation manner by injecting an external guidance from a pre-trained teacher network into the NAS process. Specifically, we leverage the attention transfer idea (Zagoruyko & Komodakis, 2017 ) that encourages a student (the proxy network in our case) to hierarchically imitate a teacher's hidden attention knowledge. Intuitively, this may benefit the search for self-calibration learning. Formally, let us denote a feature tensor at the j-th stage of the teacher and student network as F j T and F j S , separately. Attention transfer is realised by imposing an alignment loss function across the two networks as:

L AT " 1 2 ÿ jPJ } x j S }x j S } 2 ´xj T }x j T } 2 } 2 , with x j S{T " vecp ÿ i |F j S{T p¨, ¨, iq| 2 q (6)

where x j S{T is the spatial-wise accumulated feature vector. An overview of attention guided search is depicted in Fig. 3 .

Optimisation. For NAS optimisation, we adopt the DARTS method (Liu et al., 2019b) . In our context, we conduct the continuous relaxation over all the possible heterogeneous operations O for making a continuous search space:

EQUATION

where an architecture vector a iÑj o P R |O| is used for each possible connection i Ñ j. We summarise the architecture vector of all the connections as a matrix A " " a 1 , ¨¨¨, a |E| ‰ P R |E|ˆ|O| . With this relaxation, we can jointly optimise the architecture parameters A and the network weights Θ in a fully gradient differentiable manner. Equipped with the proposed attention guidance search, the search objective function is finally formulated as the following bilevel optimisation process:

EQUATION

A ˚" arg min A L val pΘ ˚pAq, Aq `λL AT pΘ ˚pAq, Aq,

where λ denotes the weighting hyper-parameter. For the first level Eq. ( 8), we learn the optimal parameters Θ ˚for a given architecture A w.r.t a training objective L train and the attention alignment loss L AT . The second level Eq. ( 9) then explores the optimal architecture A ˚over the heterogeneous search space A w.r.t a validation objective L val and L AT . For image classification, L train and L val usually take the cross-entropy loss function.

Search Outcome. Once the above alternated optimisation is done, we derive an amenable cell architecture with heterogeneous operators. In practice, for each heterogeneous computing block we retain the top-2 strongest incoming operations with at least one feature transformation operation for the first-tier (Fig. 2(a) ), and the top-1 strongest operation for the second-tier (Fig. 2(b) ).

4. EXPERIMENTS:

We evaluated the proposed NOS method on image classification using three common datasets. CI-FAR10/100: Both CIFAR10 and CIFAR100 have 50K/10K train/test RGB images of size 32ˆ32ˆ3, categorised into 10 and 100 classes, respectively (Krizhevsky et al., 2009) . ImageNet: We use the ILSVRC2012 version for large-scale image classification evaluation, containing 1.28M training images, 50K validation samples, and 1K classes (Russakovsky et al., 2015) .

We first conduct preliminary experiments on CIFAR10/100 to select the heterogeneous primitive operations O. To test the efficacy and transferability of NOS, we search the cell structures on CIFAR10 only, and compare the performance with existing methods on CIFAR10/100 and ImageNet. 

4.1. PRELIMINARY STUDY OF FEATURE SELF-CALIBRATION OPERATIONS:

We conducted a controlled experiment to test the introduced self-calibration operations on CIFAR-10 and CIFAR-100. Specifically, for the proposed dynamic convolutions, we considered both normal and dilated convolutions and two kernel sizes (3 ˆ3 and 5 ˆ5). We adopted the channel-wise and spatial-wise attention learning. For the baseline model, we used ResNet-18 (He et al., 2016) with 4 stages in the backbone. To build a model with self-calibration, we added each self-calibration operation at the stages 1, 2, 3 of ResNet-18, respectively. For fair comparison, we trained each model in the same setting (see Appendix A.2.1). In Table 1 , we summarised the model parameters and FLOPs in addition to the test set performance (error rates). We observed that: (1) Both attention operations and our normal dynamic convolutions outperform the baseline consistently;

(2) Adding dilated dynamic convolutions causes performance drop in most cases. We hence exclude it from the candidate set;

(3) Very marginal FLOPs and parameters increase from these self-calibration operations over the baseline, suggesting their high cost-effectiveness. 

4.2. CELL SEARCH:

Search Space. As found out above, the heterogeneous primitive operation set O contains 11 operations in total: |O f | " 7 feature transformation operations, |O a | " 2 attention learning operations, |O d | " 2 dynamic convolutions, respectively. We constructed the proposed heterogeneous operator cell (G " pV, Eq) with |V| " 7 nodes (2 input nodes, 4 inner nodes and 1 output node). So, all 4 heterogeneous residual blocks contain |E| " 18 edges in total (14 first-tier connections and 4 secondtier connections). To derive the final cell architecture, we kept 2 first-tier connections and 1 secondtier connection for each block. As a result, there is a total number of ś 4

n"1 pn`1qn 2 ˆ11 3 « 10 14 possible choices, 5 orders of magnitude larger than the conventional size of ś 4

n"1 pn`1qn 2 ˆ72 « 10 9 as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019) .

Training. Following the setup of existing methods (Real et al., 2019; Liu et al., 2019b; 2018a; Akimoto et al., 2019) , we searched the convolutional architectures on CIFAR10. We constructed a Table 2 : Comparisons with the state-of-the-art architectures on CIFAR10 and CIFAR100.

Architecture:

Error (%) Params Search Cost Type CIFAR10 CIFAR100 (M) GPUs Days PyramidNet (Han et al., 2017) * 3.92 20.11 2.5 --Manual DenseNet-BC (Huang et al., 2017) 3.46 17.18 25.6 --Manual NASNet-A (Zoph et al., 2018) 2.65 -3.3 450 1800 RL AmoebaNet-B (Real et al., 2019) 2.55˘0.05 -2.8 450 3150 EA Hierarchical-Evolution (Liu et al., 2018b) 3.75˘0.12 -15.7 200 300 EA PNAS (Liu et al., 2018a) 3.41˘0.09 -3.2 100 1.5 SMBO ENAS (Pham et al., 2018) 2.89 -4.6 1 0.5 RL ProxylessNAS Cai et al. (2019) 2.08 -5.7 -4 GD RENAS (Chen et al., 2019) 2.88˘0.02 -3.5 4 6 EA&RL DARTS(1st) (Liu et al., 2019b) 3.00˘0.14 -3.3 1 1.5 GD DARTS(2nd) (Liu et al., 2019b) 2.76˘0.09 17.54 3.3 1 4.0 GD SNAS (moderate) (Xie et al., 2019) 2.85˘0.02 -2.8 1 1.5 GD GHN (Zhang et al., 2019) 2.84˘0.07 -5.7 1 0.84 GD GDAS (Dong & Yang, 2019) 2.93 18.38 3.4 1 0.84 GD BayesNAS(0.005) (Zhou et al., 2019) 2.81˘0.04 -3.4 1 0.2 GD ASNG (Akimoto et al., 2019) 2.83˘0.14 -3.9 small proxy network with 8 heterogeneous operator cells, and two reduction cells at 1/3 and 2/3 of the total network depth for feature shape reduction. We used 25K images split from the training set for validation. We randomly initialised the architecture parameters A P R 18ˆ11 in the normal distribution. We used a pre-trained PyramidNet-110 (bottleneck, α " 84) (Han et al., 2017) as the teacher model. We set the weight λ " 10 3 for attention guidance loss L AT . After 25 epochs of training on the proxy network, we derived the final heterogeneous operator cells from the architecture matrix A. See Appendix A.2.1 for more configurations for training the proxy and teacher networks.

The search on CIFAR10 took only 8.4 hours using a single NVIDIA Tesla V100 GPU. The searched heterogeneous operator cells by NOS is shown in Fig. 4 , in which the self-calibration operators d and appear in both first-tier and second-tier. For example, there are two attention operations in first-tier and two dynamic convolutions in second-tier in the normal cell.

4.3. ARCHITECTURE EVALUATION:

CIFAR. To measure the final image classification performance of the searched heterogeneous operator cells on CIFAR10 and CIFAR100, we created an evaluation network with 20 cells, 36 initial channels, and an auxiliary tower with loss weight 0.4. See Appendix A.2.1 for more configurations for training the evaluation network. Due to high variance of results on CIFAR, we conducted 10 independent runs and reported both the best and average results. We summarised the results of NOS and the state-of-the-art methods in Table 2 . The comparisons show that: (1) NOS achieves a very competitive result (second best) on CIFAR10, whilst enjoying the smallest model parameters (only 2.6M). Comparing with the best performer ProxylessNAS at the size of 5.7M (searched with a supernet) (Cai et al., 2019) , it shows the significant cost-effectiveness and compactness advantages of our method.

(2) Despite a significantly larger search space (10 14 vs 10 9 in (Liu et al., 2019b; Akimoto et al., 2019; Xie et al., 2019; Dong & Yang, 2019; Akimoto et al., 2019) ), NOS shows high cost-effectiveness in computing cost (only 0.35 GPU day).

(3) NOS achieves the best result on CIFAR100 by directly transferring the CIFAR10 searched network, significantly outperforming DARTS (Liu et al., 2019b) and GDAS (Dong & Yang, 2019) . This challenging cross-dataset test indicates a superior transferability of the network searched by NOS.

ImageNet. To evaluate the transferability of architecture discovered by NOS on large scale Ima-geNet, we used the mobile setting same as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019) , where the number of multiply-add operations is restricted to be less than 600M at the input (Sandler et al., 2018) 28.0 -3.4 300 -Manual ShuffleNet 2ˆ(v1) (Zhang et al., 2018) 26.4 10.2 «5 524 -Manual ShuffleNet 2ˆ(v2) (Ma et al., 2018) 25.1 -«5 591 -Manual NASNet-A (Zoph et al., 2018) 26.0 8.4 5.3 564 1800 RL NASNet-B (Zoph et al., 2018) 27.2 8.7 5.3 488 1800 RL NASNet-C (Zoph et al., 2018) 27.5 9.0 4.9 558 1800 RL PNAS (Liu et al., 2018a) 25.8 8.1 5.1 588 1.5 SMBO AmoebaNet-A (Real et al., 2019) 25.5 8.0 5.1 555 3150 EA AmoebaNet-B (Real et al., 2019) 26.0 8.5 5.3 555 3150 EA AmoebaNet-C (Real et al., 2019) 24.3 7.6 6.4 570 3150 EA RENAS (Chen et al., 2019) 24.3 7.4 5.4 580 6 EA&RL MnasNet-A3 (Tan et al., 2019) 23.3 6.7 5.2 403 -: RL ProxylessNAS (GPU) (Cai et al., 2019) 24.9 7.5 7.1 465 8.3 GD FBNet-C (Wu et al., 2019a) 25.1 -5.5 375 9.0 GD GHN (Zhang et al., 2019) 27.0 8.7 6.1 569 0.84 GD DARTS (Liu et al., 2019b) 26 The architecture search takes 4.5 days on 64 TPUv2 devices.

size of 224 ˆ224. Specifically, we constructed an evaluation network with 14 cells and 48 initial channels. An auxiliary tower with loss weight 0.4 was also applied. See Appendix A.2.2 for more training details. Table 3 shows the ImageNet results in the mobile setting. Notably, the cell architectures found by NOS on CIFAR10 can achieve highly competitive performance with significantly less computational cost (0.35 day on 1 GPU vs 4.5 days using 64 TPUv2 devices required by MnasNet-A3 (Tan et al., 2019) ). Unlike MnasNet-A3 (Tan et al., 2019) and ProxylessNAS (Cai et al., 2019) searching the network on ImageNet directly (resource-intensive), the network searched by NOS on CIFAR10 can be successfully transferred. Also, compared to other state-of-the-art gradient based proxy-based NAS (GHN, DARTS, SNAS, GDAS and BayesNAS), NOS discovers a cell structure that performs better with higher efficiency (only 440M FLOPs). We evaluated attention guided search (AGS) on CIFAR10/100 by comparing a NOS variant without attention transfer loss. The same training setting was used (Appendix A.2.1)1 . We used a pre-trained PyramidNet-110 as teacher. Table 4 shows that learning with attention guidance can significantly benefit the NOS search process. We further showed the training curves in Fig. 5 and observed that AGS clearly improves the train/val accuracies. This suggests that AGS is effective to alleviate the architecture training-test discrepancy issue involved in the proxy-based NAS.

4.4. FURTHER ANALYSIS:



5. CONCLUSION:

We presented Neural Operator Search (NOS), featured by a heterogeneous search space for neural architecture search (NAS). This search space expansion enables NAS to discover more expressive and previously undiscovered architectures, significantly expanding the search horizon and enriching the possible search outcomes. We further formulated heterogeneous residual block and attention guided to solve the intrinsic search challenges involved. Extensive experiments on image classification show that NOS can discover novel and high-quality cell architectures in a cost-effective process. We hope that this work will shed light on the future directions for the NAS community.

A APPENDIX:

A 

:

The absence of AGS gives a slight difference to the search cost: GPU day 0.34 vs. 0.35 when using AGS.

Figures and Tables:

Figure 1: Structure of the proposed dynamic convolutions for image classification. b denotes matrix multiplication.

Figure 2: Heterogeneous Residual block for formulating the inner node computation. (a) First-tier individual computation; (b) Second-tier collective computation.

Figure 3: Overview of attention guided search. T i and S i (i P t0, 1, 2u) denote the i-th stage of the teacher and proxy (student) networks.

Figure 4: Normal cell and reduction cell searched on CIFAR-10. f0: sep conv 3x3, f1: sep conv 5x5, f2: dil conv 3x3, f4: max pooling, f6: identity, a0: spatial attention, a1: channel attention, d0: dynamic conv 3x3, d1: dynamic conv 5x5.

Figure 5: Train/Val set accuracy on CIFAR10.

= nn.Conv2d(C_in, C_in // reduction, 1, stride=1, padding =0, bias=False) 11 self.bn_1 = nn.BatchNorm2d(C_in // reduction, affine=affine) 12 self.relu = nn.ReLU(inplace=False) 13 self.conv_2 = nn.Conv2d(C_in, F_size * F_size, 1, stride=1, padding=0, bias=False) 14 self.soft = nn.Softmax(dim=2) self.conv_3 = nn.Conv2d(C_in // reduction, C_out, 1, stride=1, padding=0, bias=False) .squeeze(1) # [N, C, -1] 30 dynamic = dynamic.view(N, C, self.f_size, self.f_size) 31 dynamic = self.conv_3(dynamic) 32 dynamic = dynamic.view(N, self.c_out, -1) 33 dynamic = self.soft(dynamic) 34 dynamic = dynamic.view(N, self.c_out, self.f_size, self.f_size) 35 return dynamic

Evaluating the feature self-calibration operations on CIFAR10 and CIFAR100.

Table content: <table><tr><td>Model</td><td colspan="2">Type Kernels</td><td colspan="4">CIFAR10 Top-1(%) Top-5(%) Top-1(%) Top-5(%) CIFAR100</td><td colspan="2">FLOPS(M) #Params(MB)</td></tr><tr><td>ResNet-18</td><td>-</td><td>-</td><td>4.95</td><td>0.22</td><td>23.61</td><td>7.16</td><td>555.42</td><td>11.17</td></tr><tr><td>+ Dynamic</td><td>Normal Dilated</td><td>3 5 3 5</td><td>4.63 Ò 4.78 Ò 4.97 Ó 4.92 Ò</td><td colspan="3">0.13 Ò 22.63 Ò 6.44 Ò 0.14 Ò 23.45 Ò 6.82 Ò 0.23 Ó 24.00 Ó 7.28 Ó 0.17 Ò 23.75 Ó 7.20 Ó</td><td>+ 3.85 + 7.62 + 3.85 + 7.62</td><td>+ 0.03 + 0.04 + 0.03 + 0.04</td></tr><tr><td>+ Attention</td><td colspan="2">Spatial Channel</td><td>4.79 Ò 4.83 Ò</td><td colspan="3">0.16 Ò 23.51 Ò 7.04 Ò 0.19 Ò 23.20 Ò 6.89 Ò</td><td>+ 1.08 + 0.40</td><td>+ 0.01 + 0.15</td></tr></table>

Comparisons with the state-of-the-art architectures on ImageNet-mobile.

Table content: <table><tr><td>Architecture</td><td colspan="5">Test Err. (%) Params ˆ`Search Cost Type</td></tr><tr><td/><td>top-1 top-5</td><td>(M)</td><td colspan="2">(M) (GPU-days)</td></tr><tr><td colspan="2">MobileNet-v1(1.0) (Howard et al., 2017) 29.4 10.5</td><td>4.2</td><td>575</td><td>-</td><td>Manual</td></tr><tr><td>MobileNet-v2(1.0)</td><td/><td/><td/><td/></tr></table>

Testing attention guided search (AGS).

Table content: <table><tr><td>AGS</td><td colspan="2">Test Error (%) CIFAR10 CIFAR100</td></tr><tr><td>w/o</td><td>3.44</td><td>18.80</td></tr><tr><td>w/</td><td>2.53</td><td>16.21</td></tr></table>

Peer review (iclr2020):
Summary: Often in (neural architecture search) NAS papers on vision datasets, only feature transform operations (e.g. convolution, pooling, etc) are considered while operations like attention and dynamic convolution are left out. These operations have shown increasing performance improvements across many domains and datasets. This paper attempts to incorporate these operations into the search space of NAS. Specifically they design a new cell (residual block in a resnet backbone) which contains these operations (traditionally used feature transforms, attention, and dynamic convolutions). This new cell contains two-tiers. The first tier separately computes the three kinds of operations and combines the results via elementwise summation to form an *intermediate calibrated tensor*. This is then fed to the second tier where again the three kinds of operations are performed on them and they net output is again formed via elementwise summation. For the search algorithm the authors use the bilevel optimization of DARTS. The one difference to aid in the search is that an *attention-guided* transfer mechanism is used where a teacher network trained on the larger dataset (like ImageNet) is used to align the intermediate activations of the proxy network found during search, by adding a alignment loss function. Results of search with this new search space on cifar10 and transfer to cifar100, imagenet are presented. It is found via manual ablation in resnet backbone that dilated dynamic convolutions dont help and are dropped from the search space. The numerical results are near state-of-the-art although as is pervasive in the NAS field actual fair comparisons between methods are hard to get due to differences in search space, hardware space, stochasticity in training etc. But the authors do a best-attempt and have included random search and best and average performances so that is good. Comments: - The paper is generally well-written and easy to understand (thanks!) - Experiments are generally thorough. - The main novelty is the design of the cell such that attention and dynamic convolution operations can be incorporated. I was hoping that the results in terms of error would be much better due to the more expressive search space but they are not at the moment. - I am curious how the attention guided alignment loss has been ablated. How do the numbers look without alignment loss keeping gpu training time constant? Basically I am trying to figure out how much is the contribution of the new search space vs. the addition of attention guided search. Do we have a negative result (an important one though) that attention and dynamic convolutions don*t really help?