Neural Operator Search

1 INTRODUCTION . Recent advances of Neural Architecture Search ( NAS ) are remarkable in challenging tasks , e.g . image classification ( Zoph & Le , 2017 ) , object detection ( Ghiasi et al. , 2019 ) , and semantic segmentation ( Liu et al. , 2019a ; Nekrasov et al. , 2019 ) , greatly alleviating the demands for human knowledge and interventions by automating the laborious process of designing neural network architectures . One common scheme for the standard proxy-based neural architecture search methods ( Pham et al. , 2018 ; Zoph et al. , 2018 ; Liu et al. , 2019b ) is to factorise the search space via repeatedly stacking the same cell structure , within which a computing block generates an output tensor Fk by combining the transformations of two input feature tensors Fi and Fj : Fk “ oiÑk pFiq ‘ ojÑk pFjq s.t . i ă k & j ă k , ( 1 ) where oiÑk and ojÑk are the i-th and j-th primitive operations for feature transformation , selected from a candidate operation set O , and ‘ is the element-wise addition . Existing NAS methods use only the standard feature learning/transformation operations ( convolution , pooling and identity mapping ) as the building components . Besides , extensive studies ( Hu et al. , 2018b ; Bertinetto et al. , 2016 ; Wang et al. , 2017 ; Jia et al. , 2016 ; Wu et al. , 2019b ; Zhu et al. , 2019 ) have proven that other advanced operations for feature self-calibration , such as attention learning and dynamic convolutions , can bring great benefits for representation learning . For example , Hu et al . ( 2018b ) proposes Squeeze-and-Excitation Networks to explicitly model inter-dependencies between channels by learning channel-wise self-attention . Jia et al . ( 2016 ) presents Dynamic Filter Networks to generate context-aware filters for increasing the flexibility and adaptiveness of networks . However , these useful feature calibration elements have never been well exploited in NAS , significantly limiting the potentials of NAS which aims for automatically discovering more sophisticated and advanced network architectures without human engineering . In this work , we aim to address this limitation by extending the search space of NAS with feature self-calibration operations for scaling up the search boundary . This makes a heterogeneous search space . Consequently , the way of feature tensor interaction and combination is dramatically diversified , from the conventional addition operator ‘ only to the combination of addition ‘ , multiplication d for attention modelling , and dynamic convolution ~ . In this regard , we call the proposed method Neural Operator Search ( NOS ) . Such a search space enhancement is critical since NAS is enabled to explore stronger and previously undiscovered network architectures , which opens a door to potentially take the NAS research to the next level . In the no free lunch saying , this also comes with two new challenges : ( i ) It is non-trivial and more challenging to assemble such heterogeneous tensors and operations ( i.e . features , attentions and dynamic weights ) in a unified computing block , as compared to the conventional homogeneous feature-tensor-to-feature-tensor transformation ; ( ii ) The search space increases exponentially which leads to a much harder NAS problem . To address the first challenge , we formulate a heterogeneous operator cell characterised by a novel heterogeneous residual block . This block , formulated in a residual learning spirit ( He et al. , 2016 ) , is designed specially for fusing all the different types of tensors and operations synergistically . To solve the second challenge , we propose leveraging the attention transfer ( Zagoruyko & Komodakis , 2017 ) idea to facilitate the search behaviour across this significantly larger network space via following the attention guidance of a pretrained teacher model . As we will show , this guidance not only makes the search more efficient but also improves the search result . Our contributions in this work are : ( 1 ) We present a novel heterogeneous search space for NAS characterised by richer primitive operations including both conventional feature transformations and newly introduced feature self-calibration . This breaks the conventional selection limit of candidate neural networks and enables the NAS process to find stronger architectures , many of which are impossible to be discovered in the conventional space . This opens new territories for supporting stronger NAS algorithms and new possibilities for most expressive architectures ever to be revealed . ( 2 ) We formulate a novel Neural Operator Search ( NOS ) method dedicated for NAS in the proposed heterogeneous search space , with a couple of key designs – heterogeneous residual block for fusing different types of tensor operations synergistically and attention guided search for facilitating the search process over a vast search space more efficiently and more effectively . ( 3 ) With extensive comparisons to the state-of-the-art NAS methods , the experiments show that our approach is highly competitive on both CIFAR and ImageNet-mobile image classification tests . 2 RELATED WORK . Neural Architecture Search . Since the seminal work by Zoph & Le ( 2017 ) , neural architecture search has gained a surge of interest , effectively replacing laborious human designs by the computational process . From the strategy point of view , NAS methods can be categorised into two types : ( 1 ) proxy-based ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Pham et al. , 2018 ; Liu et al. , 2019b ) and ( 2 ) proxy-less ( Cai et al. , 2019 ; Tan et al. , 2019 ; Wu et al. , 2019a ) NAS . Specifically , to alleviate the computational cost during search , the proxy-based NAS methods search for building cells on proxy tasks , with one or more of following compromised strategies : starting with fewer cells ; using a smaller dataset ( e.g . CIFAR-10 ) ; learning with fewer epochs . Then , to transfer to the large-scale target task , one can build a network by stacking searched cells without further exploration . However , suffering from lacking of directness and specialisation , the searched cells by proxy-based NAS methods are not guaranteed to be optimal on the target task . In contrast , proxy-less NAS methods directly learns architectures on a target task by starting with an over-parameterised network ( supernet ) that contains all possible paths , in which the redundant paths are pruned to derive the optimised architecture . Notwithstanding significant better results than proxy-based approaches , proxy-less NAS methods require massive computational cost and GPU memory assumption , due to learning with the vast-size supernet . From the optimisation point of view , existing NAS methods usually fall into three groups : reinforcement learning ( RL ) based methods , evolutionary algorithm ( EA ) based methods , and gradient differentiable ( GD ) methods . In particular , RL-based NAS methods ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Tan et al. , 2019 ) control the selection of architecture component in a sequential order with policy networks . EA-based NAS methods ( Real et al. , 2019 ; Liu et al. , 2018b ) employ the validation accuracies to guide the evolution of a population of initialised architectures . RL- an EA-based NAS methods usually suffer from low efficiency and high computational resource demand , due to the fundamental searching challenge in a discrete space . In contrast , GD-based NAS methods ( Liu et al. , 2019b ; Xie et al. , 2019 ; Luo et al. , 2018 ) conduct searching over a continuous space by relaxation or mapping , substantially reducing the search cost to a few GPU days . Whilst varying in the algorithmic aspects , all these works commonly explore the feature-transformationonly search spaces without more diverse and advanced operations as we investigate here . To show the NAS potential of the proposed richer search space with self-calibration learning operations , we take the efficient proxy-based GD optimisation due to the resource constraint . Self-Calibration . Self-calibration is a type of mechanism enabling a network to dynamically perform input-conditional self-adjustment , which has been studied extensively in both the computer vision ( Hu et al. , 2018b ; Jia et al. , 2016 ; Li et al. , 2018 ; Park et al. , 2018 ) and natural language processing ( NLP ) literature ( Wu et al. , 2019b ; Vaswani et al. , 2017 ) . There are two typical paradigms of self-calibration : self-attention learning and dynamic convolutions , realised via an element-wise multiplication operator d and a dynamic convolution operator ~ , respectively . Despite showing significant efficacy , self-calibration is only exploited independently after architecture hand-design ( Hu et al. , 2018b ) or auto-search ( Tan et al. , 2019 ) . We move a step further by fully exploring the potential of self-calibration along with feature transformation in joint optimisation , bringing a richer search space for neural architecture search . Knowledge Distillation . There are recent works that use knowledge distillation to help computer vision and NLP tasks . Three types of knowledge are usually considered in distillation : features ( Yim et al. , 2017 ) , attention ( Zagoruyko & Komodakis , 2017 ) , and predictions ( Hinton et al. , 2015 ) . We leverage the attention distillation with a different objective – alleviating the intrinsic training-test discrepancy issue of the proxy-based NAS strategy , particularly with a more expressive search space . This represents a novel exploitation of attention distillation ( Zagoruyko & Komodakis , 2017 ) . 3 METHOD . In this section , we start by formulating a heterogeneous search space for NAS ( Section 3.1 ) , followed by a dedicated heterogeneous operator cell to enable composing the heterogeneous operations in a unified computing block with synergistic interaction and cooperation ( Section 3.2 ) . To overcome the intrinsic architecture discovery challenges from more expressive search space , we further develop an attention guided search scheme ( Section 3.3 ) . 3.1 HETEROGENEOUS SEARCH SPACE To enrich the NAS search space so that more advanced network architectures can be discovered , we introduce a heterogeneous search space A that considers three different types of representation learning capabilities : ( 1 ) Feature transformations ; ( 2 ) Attention learning ; and ( 3 ) Dynamic convolutions . More concretely , we form three sets of primitive computing operations that produce features , attentions and dynamic weights , respectively . This novel search space generalises the conventional counterpart which is limited to the first type of operations ( Liu et al. , 2019b ; Pham et al. , 2018 ) , and incorporates the self-calibration learning capabilities ( i.e . the second and third types ) in NAS . Importantly , while the search space changes , the generic search strategies still apply therefore being largely open for collaborating with existing NAS methods . For instance , in the proxy-based NAS strategy we may first search for a computing cell with heterogeneous operations as the building block and then form the final network architecture by sequentially stacking multiple such cells layer-by-layer . Next , let us describe the heterogeneous primitive operation set O which consists of the following three disjoint subsets : Of , Oa and Od , along with their aggregation or application operators . Feature Transformation Operations Of . We adopt the feature transformation/learning operation set Of same as in Liu et al . ( 2019b ; 2018a ) , including the following 7 operations : 3 ˆ 3 and 5 ˆ 5 separable convolutions , 3 ˆ 3 and 5 ˆ 5 dilated separable convolutions , 3 ˆ 3 average pooling , 3 ˆ 3 max pooling , and identity . Every operation of P Of takes as input a feature tensor and outputs another feature tensor , i.e . homogeneous feature-tensor-to-feature-tensor transformation . For multiple feature tensor aggregation , the element-wise addition operator ‘ is typically used . Attention Learning Operations Oa . Inspired by recent exquisite designs of attention learning modules ( Hu et al. , 2018b ; Li et al. , 2018 ; Park et al. , 2018 ) , we form the Oa by considering two types of attention learning prototypes : spatial-wise and channel-wise attentions . Specifically , a spatial-wise attention operation learns a saliency map for an input feature tensor in order to calibrate the importance of different spatial positions . In contrast , a channel-wise attention operation produces a vector of scaling factors from the aggregated global context of an input tensor for adaptively calibrating the channel dependency . To enforce attentive calibration on feature tensor , the elementwise multiplication operator d is a typical choice for both spatial-wise and channel-wise attentions . Dynamic Convolution Operations Od . Dynamic convolutions , designed for the sake of selfadaptation , generate dynamic kernel weights in accordance with the input feature tensor . It is often in form of depth-wise separable convolution as the feature transformation operation . Tailored for either NLP or dense prediction tasks , existing dynamic convolution designs ( Wu et al. , 2019b ; Jia et al. , 2016 ) are not suitable for image classification ( our focus ) with different problem nature . It hence needs to be reformulated in order to be effective for learning discriminative image representations . We consider two design principles : ( i ) structurally lightweight whilst ( ii ) functionally strong with great modelling capability . To that end , we present an exquisite dynamic convolution structure specialised for cost-effective image classification , as shown in Fig . 1 . Concretely , it consists of three compact modules composed in an exquisite cooperation : ( a ) a bottleneck module , to compress an input feature tensor by a ratio of r ; ( b ) a kernel transform module , to learn latent representations with a kernel dimension of k ˆ k ; ( c ) a kernel decode module , to read out the dynamic kernel weights with the channel dimension same as the input feature tensor . This design is motivated , in part , by the long-range dependency modeling ( Wang et al. , 2018 ; Cao et al. , 2019 ) and global context aggregation ( Hu et al. , 2018b ; a ) , elegantly integrating their merits via a unified formulation . For the output of dynamic convolutions , we consider two common kernel sizes : 3ˆ3 and 5ˆ5 . In a depth-wise manner , we apply a standard or dilated convolution operator ~ to transform the input feature tensor . It is noteworthy to point out that , this type of convolutional kernel is specific for each feature tensor of a particular image sample ( i.e . dynamic ) , rather than learned from a training dataset and fixed for all the input samples ( i.e . static ) as the conventional convolutional operations in the feature transformation set . Detailed implementations of self-calibration operations are presented in Appendix A.1 .

Summary: Often in (neural architecture search) NAS papers on vision datasets, only feature transform operations (e.g. convolution, pooling, etc) are considered while operations like attention and dynamic convolution are left out. These operations have shown increasing performance improvements across many domains and datasets. This paper attempts to incorporate these operations into the search space of NAS.