Title: PROXYLESSNAS: DIRECT NEURAL ARCHITECTURE SEARCH ON TARGET TASK AND HARDWARE

Abstract: Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. 10 4 GPU hours) makes it difficult to directly search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to utilize proxy tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task. In this paper, we present ProxylessNAS that can directly learn the architectures for large-scale target tasks and target hardware platforms. We address the high memory consumption issue of differentiable NAS and reduce the computational cost (GPU hours and GPU memory) to the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of directness and specialization. On CIFAR-10, our model achieves 2.08% test error with only 5.7M parameters, better than the previous state-of-the-art architecture AmoebaNet-B, while using 6× fewer parameters. On ImageNet, our model achieves 3.1% better top-1 accuracy than MobileNetV2, while being 1.2× faster with measured GPU latency. We also apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.

1. INTRODUCTION:

Neural architecture search (NAS) has demonstrated much success in automating neural network architecture design for various deep learning tasks, such as image recognition (Zoph et al., 2018; Cai et al., 2018a; Liu et al., 2018a; Zhong et al., 2018) and language modeling (Zoph & Le, 2017) . Despite the remarkable results, conventional NAS algorithms are prohibitively computation-intensive, requiring to train thousands of models on the target task in a single experiment. Therefore, directly applying NAS to a large-scale task (e.g. ImageNet) is computationally expensive or impossible, which makes it difficult for making practical industry impact. As a trade-off, Zoph et al. (2018) propose to search for building blocks on proxy tasks, such as training for fewer epochs, starting with a smaller dataset (e.g. CIFAR-10), or learning with fewer blocks. Then top-performing blocks are stacked and transferred to the large-scale target task. This paradigm has been widely adopted in subsequent NAS algorithms (Liu et al., 2018a; b; Real et al., 2018; Cai et al., 2018b; Liu et al., 2018c; Tan et al., 2018; Luo et al., 2018) .

However, these blocks optimized on proxy tasks are not guaranteed to be optimal on the target task, especially when taking hardware metrics such as latency into consideration. More importantly, to enable transferability, such methods need to search for only a few architectural motifs and then repeatedly stack the same pattern, which restricts the block diversity and thereby harms performance.

In this work, we propose a simple and effective solution to the aforementioned limitations, called ProxylessNAS, which directly learns the architectures on the target task and hardware instead of with Figure 1 : ProxylessNAS directly optimizes neural network architectures on target task and hardware. Benefiting from the directness and specialization, ProxylessNAS can achieve remarkably better results than previous proxy-based approaches. On ImageNet, with only 200 GPU hours (200 × fewer than MnasNet (Tan et al., 2018) ), our searched CNN model for mobile achieves the same level of top-1 accuracy as MobileNetV2 1.4 while being 1.8× faster.

proxy (Figure 1 ). We also remove the restriction of repeating blocks in previous NAS works (Zoph et al., 2018; Liu et al., 2018c) and allow all of the blocks to be learned and specified. To achieve this, we reduce the computational cost (GPU hours and GPU memory) of architecture search to the same level of regular training in the following ways.

GPU hour-wise, inspired by recent works (Liu et al., 2018c; Bender et al., 2018) , we formulate NAS as a path-level pruning process. Specifically, we directly train an over-parameterized network that contains all candidate paths (Figure 2 ). During training, we explicitly introduce architecture parameters to learn which paths are redundant, while these redundant paths are pruned at the end of training to get a compact optimized architecture. In this way, we only need to train a single network without any meta-controller (or hypernetwork) during architecture search.

However, naively including all the candidate paths leads to GPU memory explosion (Liu et al., 2018c; Bender et al., 2018) , as the memory consumption grows linearly w.r.t. the number of choices. Thus, GPU memory-wise, we binarize the architecture parameters (1 or 0) and force only one path to be active at run-time, which reduces the required memory to the same level of training a compact model. We propose a gradient-based approach to train these binarized parameters based on Bina-ryConnect (Courbariaux et al., 2015) . Furthermore, to handle non-differentiable hardware objectives (using latency as an example) for learning specialized network architectures on target hardware, we model network latency as a continuous function and optimize it as regularization loss. Additionally, we also present a REINFORCE-based (Williams, 1992) algorithm as an alternative strategy to handle hardware metrics.

In our experiments on CIFAR-10 and ImageNet, benefiting from the directness and specialization, our method can achieve strong empirical results. On CIFAR-10, our model reaches 2.08% test error with only 5.7M parameters. On ImageNet, our model achieves 75.1% top-1 accuracy which is 3.1% higher than MobileNetV2 (Sandler et al., 2018) while being 1.2× faster. Our contributions can be summarized as follows:

• ProxylessNAS is the first NAS algorithm that directly learns architectures on the largescale dataset (e.g. ImageNet) without any proxy while still allowing a large candidate set and removing the restriction of repeating blocks. It effectively enlarged the search space and achieved better performance. • We provide a new path-level pruning perspective for NAS, showing a close connection between NAS and model compression (Han et al., 2016) . We save memory consumption by one order of magnitude by using path-level binarization. • We propose a novel gradient-based approach (latency regularization loss) for handling hardware objectives (e.g. latency). Given different hardware platforms: CPU/GPU/Mobile, ProxylessNAS enables hardware-aware neural network specialization that's exactly optimized for the target hardware. To our best knowledge, it is the first work to study specialized neural network architectures for different hardware architectures.

• Extensive experiments showed the advantage of the directness property and the specialization property of ProxylessNAS. It achieved state-of-the-art accuracy performances on CIFAR-10 and ImageNet under latency constraints on different hardware platforms (GPU, CPU and mobile phone). We also analyze the insights of efficient CNN models specialized for different hardware platforms and raise the awareness that specialized neural network architecture is needed on different hardware architectures for efficient inference.

2. RELATED WORK:

The use of machine learning techniques, such as reinforcement learning or neuro-evolution, to replace human experts in designing neural network architectures, usually referred to as neural architecture search, has drawn an increasing interest (Zoph & Le, 2017; Liu et al., 2018a; b; c; Cai et al., 2018a; b; Pham et al., 2018; Brock et al., 2018; Bender et al., 2018; Elsken et al., 2017; 2018b; Kamath et al., 2018) . In NAS, architecture search is typically considered as a meta-learning process, and a meta-controller (e.g. a recurrent neural network (RNN)), is introduced to explore a given architecture space with training a network in the inner loop to get an evaluation for guiding exploration. Consequently, such methods are computationally expensive to run, especially on large-scale tasks, e.g. ImageNet.

Some recent works (Brock et al., 2018; Pham et al., 2018) try to improve the efficiency of this meta-learning process by reducing the cost of getting an evaluation. In Brock et al. (2018) , a hypernetwork is utilized to generate weights for each sampled network and hence can evaluate the architecture without training it. Similarly, Pham et al. (2018) propose to share weights among all sampled networks under the standard NAS framework (Zoph & Le, 2017) . These methods speed up architecture search by orders of magnitude, however, they require a hypernetwork or an RNN controller and mainly focus on small-scale tasks (e.g. CIFAR) rather than large-scale tasks (e.g. ImageNet).

Our work is most closely related to One-Shot (Bender et al., 2018) and DARTS (Liu et al., 2018c) , both of which get rid of the meta-controller (or hypernetwork) by modeling NAS as a single training process of an over-parameterized network that comprises all candidate paths. Specifically, One-Shot trains the over-parameterized network with DropPath (Zoph et al., 2018) that drops out each path with some fixed probability. Then they use the pre-trained over-parameterized network to evaluate architectures, which are sampled by randomly zeroing out paths. DARTS additionally introduces a real-valued architecture parameter for each path and jointly train weight parameters and architecture parameters via standard gradient descent. However, they suffer from the large GPU memory consumption issue and hence still need to utilize proxy tasks. In this work, we address the large memory issue in these two methods through path binarization.

Another relevant topic is network pruning (Han et al., 2016) that aim to improve the efficiency of neural networks by removing insignificant neurons (Han et al., 2015) or channels (Liu et al., 2017) . Similar to these works, we start with an over-parameterized network and then prune the redundant parts to derive the optimized architecture. The distinction is that they focus on layer-level pruning that only modifies the filter (or units) number of a layer but can not change the topology of the network, while we focus on learning effective network architectures through path-level pruning. We also allow both pruning and growing the number of layers.

3. METHOD:

We first describe the construction of the over-parameterized network with all candidate paths, then introduce how we leverage binarized architecture parameters to reduce the memory consumption of training the over-parameterized network to the same level as regular training. We propose a gradient-based algorithm to train these binarized architecture parameters. Finally, we present two techniques to handle non-differentiable objectives (e.g. latency) for specializing neural networks on target hardware.

3.1. CONSTRUCTION OF OVER-PARAMETERIZED NETWORK:

Denote a neural network as N (e, • • • , e n ) where e i represents a certain edge in the directed acyclic graph (DAG). Let O = {o i } be the set of N candidate primitive operations (e.g. convolution, pooling, identity, zero, etc). To construct the over-parameterized network that includes any architecture in the search space, instead of setting each edge to be a definite primitive operation, we set each edge to be a mixed operation that has N parallel paths (Figure 2 ), denoted as m O . As such, the over-parameterized network can be expressed as < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f W

N (e = m 1 O , • • • , e n = m n O ). Given input x,

f / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z V T 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 + g b J v

a 5 5 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f W

f / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z V T 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 + g b J v

a 5 5 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f W

f / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z V T 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 + g b J v

a 5 5 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f W

f / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z V T 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 + g b J v a 5 5 < / l a t e x i t > Loss = Loss CE + 1 ||w|| 2 2 + 2 E[

< l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8 

+ g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V

E D k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3

6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8

+ g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3

6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8

+ g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3

6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8

+ g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3 6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > E[Latency] = ↵ ⇥ F (conv 3x3) ⇥ F (conv 5x5) ⇥ F (identity)+ ...... ⇣ ⇥ F (pool 3x3) < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0 3 u P U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

x o h 5 7 7 w P 3 i f v s / / e / + h / 8 b 8 e t / p e z 7 m B / g n / x 0 9 Y V g H C < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0

3 u P U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

x o h 5 7 7 w P 3 i f v s / / e / + h / 8 b 8 e t / p e z 7 m B / g n / x 0 9 Y V g H C < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0

3 u P U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

x o h 5 7 7 w P 3 i f v s / / e / + h / 8 b 8 e t / p e z 7 m B / g n / x 0 9 Y V g H C < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0 the weights are calculated by applying softmax to N real-valued architecture parameters {α i }:

3 u P U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

m One-Shot O (x) = N i=1 o i (x), m DARTS O (x) = N i=1 p i o i (x) = N i=1 exp(α i ) j exp(α j ) o i (x).

(1)

As shown in Eq. ( 1), the output feature maps of all N paths are calculated and stored in the memory, while training a compact model only involves one path. Therefore, One-Shot and DARTS roughly need N times GPU memory and GPU hours compared to training a compact model. On largescale dataset, this can easily exceed the memory limits of hardware with large design space. In the following section, we solve this memory issue based on the idea of path binarization.

3.2. LEARNING BINARIZED PATH:

To reduce memory footprint, we keep only one path when training the over-parameterized network.

Unlike Courbariaux et al. (2015) which binarize individual weights, we binarize entire paths. We introduce N real-valued architecture parameters {α i } and then transforms the real-valued path weights to binary gates:

g = binarize(p 1 , • • • , p N ) =    [1, 0, • • • , 0] with probability p 1 , • • • [0, 0, • • • , 1] with probability p N .

(2)

Based on the binary gates g, the output of the mixed operation is given as:

EQUATION

As illustrated in Eq. ( 3) and Figure 2 , by using the binary gates rather than real-valued path weights (Liu et al., 2018c) , only one path of activation is active in memory at run-time and the memory requirement of training the over-parameterized network is thus reduced to the same level of training a compact model. That's more than an order of magnitude memory saving. Unlike weight parameters, the architecture parameters are not directly involved in the computation graph and thereby cannot be updated using the standard gradient descent. In this section, we introduce a gradient-based approach to learn the architecture parameters.

3.2.1. TRAINING BINARIZED ARCHITECTURE PARAMETERS:

Published as a conference paper at ICLR 2019

In BinaryConnect (Courbariaux et al., 2015) , the real-valued weight is updated using the gradient w.r.t. its corresponding binary gate. In our case, analogously, the gradient w.r.t. architecture parameters can be approximately estimated using ∂L/∂g i in replace of ∂L/∂p i :

EQUATION

where δ ij = 1 if i = j and δ ij = 0 if i = j. Since the binary gates g are involved in the computation graph, as shown in Eq. ( 3), ∂L/∂g j can be calculated through backpropagation. However, computing ∂L/∂g j requires to calculate and store o j (x). Therefore, directly using Eq. ( 4) to update the architecture parameters would also require roughly N times GPU memory compared to training a compact model.

To address this issue, we consider factorizing the task of choosing one path out of N candidates into multiple binary selection tasks. The intuition is that if a path is the best choice at a particular position, it should be the better choice when solely compared to any other path.2 

Following this idea, within an update step of the architecture parameters, we first sample two paths according to the multinomial distribution (p 1 , • • • , p N ) and mask all the other paths as if they do not exist. As such the number of candidates temporarily decrease from N to 2, while the path weights {p i } and binary gates {g i } are reset accordingly. Then we update the architecture parameters of these two sampled paths using the gradients calculated via Eq. ( 4). Finally, as path weights are computed by applying softmax to the architecture parameters, we need to rescale the value of these two updated architecture parameters by multiplying a ratio to keep the path weights of unsampled paths unchanged. As such, in each update step, one of the sampled paths is enhanced (path weight increases) and the other sampled path is attenuated (path weight decreases) while all other paths keep unchanged. In this way, regardless of the value of N , only two paths are involved in each update step of the architecture parameters, and thereby the memory requirement is reduced to the same level of training a compact model.

3.3. HANDLING NON-DIFFERENTIABLE HARDWARE METRICS:

Besides accuracy, latency (not FLOPs) is another very important objective when designing efficient neural network architectures for hardware. Unfortunately, unlike accuracy that can be optimized using the gradient of the loss function, latency is non-differentiable. In this section, we present two algorithms to handle the non-differentiable objectives.

3.3.1. MAKING LATENCY DIFFERENTIABLE:

To make latency differentiable, we model the latency of a network as a continuous function of the neural network dimensions3 . Consider a mixed operation with a candidate set {o j } and each o j is associated with a path weight p j which represents the probability of choosing o j . As such, we have the expected latency of a mixed operation (i.e. a learnable block) as:

EQUATION

where E[latency i ] is the expected latency of the i th learnable block, F (•) denotes the latency prediction model and F (o i j ) is the predicted latency of o i j . The gradient of E[latency i ] w.r.t. architecture parameters can thereby be given as: ∂E[latency i ] / ∂p i j = F (o i j ). For the whole network with a sequence of mixed operations (Figure 3 left), since these operations are executed sequentially during inference, the expected latency of the network can be expressed with the sum of these mixed operations' expected latencies: 

EQUATION

x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 +

x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 +

x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 +

x L 2 + Q q O D M z W g Y J 8 V x 0 4 + C h E E V w a J C O K S C Y M V S T R A W V O 8 K 8 R g J h J U u u q 5 L s G d P n i f d 4 6 Z t N e 3 b k 0 b r o q q j B v b B A T g C N j g F L X A N 2 q A D M H g A z + A N v B u P x q v x Y X x O o w t G N b M H f s H 4 +

k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o

k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o

k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o

k K t d H M R h U M s M B S V J 4 V G i E N F Z 4 G J 9 t 1 f 3 D c 9 R G 5 t k + T Q o M U z j J 5 F g K I E d F H f i R G / O N 1 3 9 k t 7 Y r v s F H y s k T i A I + n V 5 M p 1 H / 2 P Y b n v A n N R u t x q S y j 3 N 9 P k q B T u P Y b l f D B u r U K i D M x K Q K q 6 j T 9 X t + U / w l C G a g y 2 a 1 G 3 W u R k k u y h Q z E g q M G Q Z + Q a E F T V I o

U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C

U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

x o h 5 7 7 w P 3 i f v s / / e / + h / 8 b 8 e t / p e z 7 m B / g n / x 0 9 Y V g H C < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0

3 u P U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

x o h 5 7 7 w P 3 i f v s / / e / + h / 8 b 8 e t / p e z 7 m B / g n / x 0 9 Y V g H C < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0 We incorporate the expected latency of the network into the normal loss function by multiplying a scaling factor λ 2 (> 0) which controls the trade-off between accuracy and latency. The final loss function is given as (also shown in Figure 3 right)

3 u P U Y j M A P u 2 2 Q v 9 N r i X Y R M = " > A A A D I H i c f V J N b 9 Q w E H X C V w l f W z h y s V i B i p B W S U u B C 1 I F A n H g U C S 2 r b S O V o 4 z m 7 X q 2 J H t V B u i 8 E + 4 8 F e 4 c A A h u M G v w U n D V 7 v L S J a e Z u Z 5 3 j w 7 K Q Q 3 N g y / e / 6 Z s + f O X 1 i 7 G F y 6 f O X q t c H 6 9 T 2 j S s 1 g z J R Q + i C h B g S X M L b c C j g o N N A 8 E b C f H D 5 t 6 / t H o A 1 X 8 r W t C o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w j J 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9 B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S L

EQUATION

where Loss CE denotes the cross-entropy loss and λ 1 ||w|| 2 2 is the weight decay term.

3.3.2. REINFORCE-BASED APPROACH:

As an alternative to BinaryConnect, we can utilize REINFORCE to train binarized weights as well. Consider a network that has binarized parameters α, the goal of updating binarized parameters is to find the optimal binary gates g that maximizes a certain reward, denoted as R(•). Here we assume the network only has one mixed operation for ease of illustration. Therefore, according to REINFORCE (Williams, 1992), we have the following updates for binarized parameters:

EQUATION

where g i denotes the i th sampled binary gates, p(g i ) denotes the probability of sampling g i according to Eq. ( 2) and N g i is the compact network according to the binary gates g i . Since Eq. ( 8) does not require R(N g ) to be differentiable w.r.t. g, it can thus handle non-differentiable objectives. An interesting observation is that Eq. ( 8) has a similar form to the standard NAS (Zoph & Le, 2017) , while it is not a sequential decision-making process and no RNN meta-controller is used in our case. Furthermore, since both gradient-based updates and REINFORCE-based updates are essentially two different update rules to the same binarized architecture parameters, it is possible to combine them to form a new update rule for the architecture parameters.

4. EXPERIMENTS AND RESULTS:

We demonstrate the effectiveness of our proposed method on two benchmark datasets (CIFAR-10 and ImageNet) for the image classification task. Unlike previous NAS works (Zoph et al., 2018; Liu et al., 2018c) that first learn CNN blocks on CIFAR-10 under small-scale setting (e.g. fewer blocks), then transfer the learned block to ImageNet or CIFAR-10 under large-scale setting by repeatedly stacking it, we directly learn the architectures on the target task (either CIFAR-10 or ImageNet) and target hardware (GPU, CPU and mobile phone) while allowing each block to be specified.

4.1. EXPERIMENTS ON CIFAR-10:

Architecture Space. For CIFAR-10 experiments, we use the tree-structured architecture space that is introduced by Cai et al. (2018b) with PyramidNet (Han et al., 2017) as the backbone 4 . Specifically, Published as a conference paper at ICLR 2019 Model Params Test error (%) DenseNet-BC (Huang et al., 2017) 25.6M 3.46 PyramidNet (Han et al., 2017) 26.0M 3.31 Shake-Shake + c/o (DeVries & Taylor, 2017) 26.2M 2.56 PyramidNet + SD (Yamada et al., 2018) 26.0M 2.31 ENAS + c/o (Pham et al., 2018) 4.6M 2.89 DARTS + c/o (Liu et al., 2018c) 3.4M 2.83 NASNet-A + c/o (Zoph et al., 2018) 27.6M 2.40 PathLevel EAS + c/o (Cai et al., 2018b) 14.3M 2.30 AmoebaNet-B + c/o (Real et al., 2018) 34.9M 2.13 Proxyless-R + c/o (ours) 5.8M 2.30 Proxyless-G + c/o (ours) 5.7M

2.08:

Table 1 : ProxylessNAS achieves state-of-the-art performance on CIFAR-10.

we replace all 3 × 3 convolution layers in the residual blocks of a PyramidNet with tree-structured cells, each of which has a depth of 3 and the number of branches is set to be 2 at each node (except the leaf nodes). For further details about the tree-structured architecture space, we refer to the original paper (Cai et al., 2018b) . Additionally, we use two hyperparameters to control the depth and width of a network in this architecture space, i.e. B and F , which respectively represents the number of blocks at each stage (totally 3 stages) and the number of output channels of the final block.

Training Details. We randomly sample 5,000 images from the training set as a validation set for learning architecture parameters which are updated using the Adam optimizer with an initial learning rate of 0.006 for the gradient-based algorithm (Section 3.2.1) and 0.01 for the REINFORCEbased algorithm (Section 3.3.2). In the following discussions, we refer to these two algorithms as Proxyless-G (gradient) and Proxyless-R (REINFORCE) respectively.

After the training process of the over-parameterized network completes, a compact network is derived according to the architecture parameters, as discussed in Section 3.2.1. Next, we train the compact network using the same training settings except that the number of training epochs increases from 200 to 300. Additionally, when the DropPath regularization (Zoph et al., 2018; Huang et al., 2016 ) is adopted, we further increase the number of training epochs to 600 (Zoph et al., 2018) .

Results. We apply the proposed method to learn architectures in the tree-structured architecture space with B = 18 and F = 400. Since we do not repeat cells and each cell has 12 learnable edges, totally 12 × 18 × 3 = 648 decisions are required to fully determine the architecture.

The test error rate results of our proposed method and other state-of-the-art architectures on CIFAR-10 are summarized in Table 1 , where "c/o" indicates the use of Cutout (DeVries & Taylor, 2017) .

Compared to these state-of-the-art architectures, our proposed method can achieve not only lower test error rate but also better parameter efficiency. Specifically, Proxyless-G reaches a test error rate of 2.08% which is slightly better than AmoebaNet-B (Real et al., 2018) (the previous best architecture on CIFAR-10). Notably, AmoebaNet-B uses 34.9M parameters while our model only uses 5.7M parameters which is 6× fewer than AmoebaNet-B. Furthermore, compared with PathLevel EAS (Cai et al., 2018b) that also explores the tree-structured architecture space, both Proxyless-G and Proxyless-R achieves similar or lower test error rate results with half fewer parameters. The strong empirical results of our ProxylessNAS demonstrate the benefits of directly exploring a large architecture space instead of repeatedly stacking the same block.

4.2. EXPERIMENTS ON IMAGENET:

For ImageNet experiments, we focus on learning efficient CNN architectures (Iandola et al., 2016; Howard et al., 2017; Sandler et al., 2018; Zhu et al., 2018) that have not only high accuracy but also low latency on specific hardware platforms. Therefore, it is a multi-objective NAS task (Hsu et al., 2018; Dong et al., 2018; Elsken et al., 2018a; He et al., 2018; Wang et al., 2018; Tan et al., 2018) , where one of the objectives is non-differentiable (i.e. latency). We use three different hardware platforms, including mobile phone, GPU and CPU, in our experiments. The GPU latency is measured on V100 GPU with a batch size of 8 (single batch makes GPU severely under-utilized).

The CPU latency is measured under batch size 1 on a server with two 2.40GHz Intel(R) Xeon(R) Additionally, on mobile phone, we use the latency prediction model (Appendix B) during architecture search. As illustrated in Figure 5 , we observe a strong correlation between the predicted latency and real measured latency on the test set, suggesting that the latency prediction model can be used to replace the expensive mobile farm infrastructure (Tan et al., 2018) with little error introduced.

Architecture Space. We use MobileNetV2 (Sandler et al., 2018) as the backbone to build the architecture space. Specifically, rather than repeating the same mobile inverted bottleneck convolution (MBConv), we allow a set of MBConv layers with various kernel sizes {3, 5, 7} and expansion ratios {3, 6}. To enable a direct trade-off between width and depth, we initiate a deeper over-parameterized network and allow a block with the residual connection to be skipped by adding the zero operation to the candidate set of its mixed operation. In this way, with a limited latency budget, the network can either choose to be shallower and wider by skipping more blocks and using larger MBConv layers or choose to be deeper and thinner by keeping more blocks and using smaller MBConv layers.

Training Details. We randomly sample 50,000 images from the training set as a validation set during the architecture search. The settings for updating architecture parameters are the same as CIFAR-10 experiments except the initial learning rate is 0.001. 

Conv 3x3 MB1 3x3 MB6 3x3 MB3 3x3 MB3 3x3 MB3 3x3 MB6 3x3 MB3 3x3 MB3 3x3 40x112x112 24x112x112 3x224x224 32x56x56 32x56x56 32x56x56 32x56x56 48x28x28 48x28x28 MB6 3x3 MB3 5x5 48x28x28 48x28x28 MB6 5x5 MB3 3x3 MB3 3x3 MB3 3x3 MB6 5x5 MB3 3x3 MB6 5x5 Pooling FC 88x14x14 104x14x14 104x14x14 104x14x14 216x7x7 216x7x7 216x7x7 216x7x7 360x7x7 MB3 3x3 88x14x14 MB3 5x5 MB3 5x5 104x14x14

(2) Efficient CPU architecture found by ProxylessNAS.

(3) Efficient GPU architecture found by ProxylessNAS.

(a) Efficient GPU model found by ProxylessNAS. (1) Efficient mobile architecture found by ProxylessNAS.

(2) Efficient CPU architecture found by ProxylessNAS.

(3) Efficient GPU architecture found by ProxylessNAS.

(b) Efficient CPU model found by ProxylessNAS. (1) Efficient mobile architecture found by ProxylessNAS.

(2) Efficient CPU architecture found by ProxylessNAS.

(3) Efficient GPU architecture found by ProxylessNAS. we allow specializing network architectures for different platforms by directly incorporating the measured hardware latency into optimization objectives. We compared the optimized models on CPU/GPU/mobile and raised the awareness of the needs of specializing neural network architecture for different hardware architectures.

Published as a conference paper at ICLR 2019 A THE LIST OF CANDIDATE OPERATIONS USED ON CIFAR-10

We adopt the following 7 operations in our CIFAR-10 experiments:

• 3 × 3 dilated depthwise-separable convolution • Identity • 3 × 3 depthwise-separable convolution • 5 × 5 depthwise-separable convolution • 7 × 7 depthwise-separable convolution • 3 × 3 average pooling • 3 × 3 max pooling

B MOBILE LATENCY PREDICTION:

Measuring the latency on-device is accurate but not ideal for scalable neural architecture search.

There are two reasons: (i) Slow. As suggested in TensorFlow-Lite, we need to average hundreds of runs to produce a precise measurement, approximately 20 seconds. This is far more slower than a single forward / backward execution. (ii) Expensive. A lot of mobile devices and software engineering work are required to build an automatic pipeline to gather the latency from a mobile farm. Instead of direct measurement, we build a model to estimate the latency. We need only 1 phone rather than a farm of phones, which has only 0.75ms latency RMSE. We use the latency model to search, and we use the measured latency to report the final model's latency.

We sampled 5k architectures from our candidate space, where 4k architectures are used to build the latency model and the rest are used for test. We measured the latency on Google Pixel 1 phone using TensorFlow-Lite. The features include (i) type of the operator (ii) input and output feature map size (iii) other attributes like kernel size, stride for convolution and expansion ratio.

C DETAILS OF MNASNET'S SEARCH COST:

Mnas (Tan et al., 2018) trains 8,000 mobile-sized models on ImageNet, each of which is trained for 5 epochs for learning architectures. If these models are trained on V100 GPUs, as done in our experiments, the search cost is roughly 40,000 GPU hours.

D IMPLEMENTAION OF THE GRADIENT-BASED ALGORITHM:

A naive implementation of the gradient-based algorithm (see Eq. ( 4)) is calculating and storing o j (x) in the forward step to later compute ∂L/∂g j in the backward step:

EQUATION

where ∇ y L denotes the gradient w.r.t. the output of the mixed operation y, "•" denotes the elementwise product, and "reduce sum(•)" denotes the sum of all elements.

Notice that o j (x) is only used for calculating ∂L/∂g j when j th path is not active (i.e. not involved in calculating y). So we do not need to actually allocate GPU memory to store o j (x). Instead, we can calculate o j (x) after getting ∇ y L in the backward step, use o j (x) to compute ∂L/∂g j following Eq. ( 9), then release the occupied GPU memory. In this way, without the approximation discussed in Section 3.2.1, we can reduce the GPU memory cost to the same level of training a compact model.

:

Pretrained models and evaluation code are released at https://github.com/MIT-HAN-LAB/ProxylessNAS.

In Appendix D, we provide another solution to this issue that does not require the approximation.

Details of the latency prediction model are provided in Appendix B.

The list of operations in the candidate set is provided in the appendix.

Figures and Tables:

the output of a mixed operation m O is defined based on the outputs of its N paths. In One-Shot, m O (x) is the sum of {o i (x)}, while in DARTS, m O (x) is weighted sum of {o i (x)} where Published as a conference paper at ICLR 2019

h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4

Figure 2: Learning both weight parameters and binarized architecture parameters.

Figure 2 illustrates the training procedure of the weight parameters and binarized architecture parameters in the over-parameterized network. When training weight parameters, we first freeze the architecture parameters and stochastically sample binary gates according to Eq. (2) for each batch of input data. Then the weight parameters of active paths are updated via standard gradient descent on the training set (Figure 2 left). When training architecture parameters, the weight parameters are frozen, then we reset the binary gates and update the architecture parameters on the validation set (Figure 2 right). These two update steps are performed in an alternative manner. Once the training of architecture parameters is finished, we can then derive the compact architecture by pruning redundant paths. In this work, we simply choose the path with the highest path weight.

t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f Wf / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z VT 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r

g b J v a 5 5 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f Wf / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z VT 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r

g b J v a 5 5 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f Wf / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z VT 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r

g b J v a 5 5 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l n t r w t q g r P O 1 V S k 1 e a e y N u f Wf / Q = " > A A A C N X i c f V D L S s N A F J 3 4 r P U V d e l m s A i u S i K C b o S i C C 5 c V L A P a E K Y T C f t 0 J k k z E y E E P J T b v w P V 7 p w o Y h b f 8 F J m o W 2 4 o G B w z n n M v c e P 2 Z U K s t 6 M R Y W l 5 Z X V m t r 9 f W N z a 1 t c 2 e 3 K 6 N E Y N L B E Y t E 3 0 e S M B q S j q K K k X 4 s C O I + I z 1 / c l n 4 v X s i J I 3 C O 5 X G x O V o F N K A Y q S 0 5 J k 3 D k d q 7 P v Z VT 7 I S i 5 4 x p A i I U 7 z 3 I X n 0 J E J 9 z K a w 3 + S h a / T n t m w m l Y J O E / s i j R A h b Z n P j n D C C e c h A o z J O X A t m L l Z k g o i h n J 6 0 4 i S Y z w B I 3 I Q N M Q c S L d r L w 6 h 4 d a G c I g E v q F C p b q z 4 k M c S l T 7 u t k s a 2 c 9 Q r

g b J v a 5 5 < / l a t e x i t >Loss = Loss CE + 1 ||w|| 2 2 + 2 E[latency]< l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8+ g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F Xb H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D

r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3 6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8 + g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D

r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3 6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8 + g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D

r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3 6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v M 7 E 2 w 6 l X t W 5 0 L P l 1 8 + g D N 1 T 4 k = " > A A A C S n i c b V B N S x x B E O 3 Z a G J W T T b J 0 U v j I g j C M r M E k k t A I o K H H B R c F X b H o a a n V h t 7 P u i u 0 S y 9 8 / t y 8 e T N H + H F Q 0 L w Y s + 4 g t E U d P N 4 V Y 9 6 9 e J C S U O + f + 2 1 X s 3 N v 3 6 z 8 L a 9 u L T 8 7 n 3 n w 8 c D k 5 d a 4 E D

r N q j 0 m A B 4 g x O c O h g B i m a 0 D b G K r 7 m m I S P c + 1 e R r x h n y o s p M Z M 0 t h N 1 k 7 N 8 1 5 N / q 8 3 L G n 8 N b Q y K 8 r 6 s I d F 4 1 J x y n m d K 0 + k R k F q 4 g A I L Z 1 X L k 5 B g y C X f t u F E D w / + S U 4 6 P c C v x f s f e 5 u f p / F s c B W 2 C p b Z w H 7 w j b Z D t t l A y b Y L 3 b D f r M / 3 q V 3 6 / 3 1 7 h 5 G W 9 5 M 8 4 n 9 U 6 2 5 e 9 C 6 t B w = < / l a t e x i t > E[Latency] = ↵ ⇥ F (conv 3x3)+ ⇥ F (conv 5x5)+ ⇥ F (identity)+ ...... ⇣ ⇥ F (pool 3x3) < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0 3 u P

o h z m k k + 4 4 x a l 5 q u e w 9 I A h m X N R U 8 k 5 A 2 A c m p n S d J / a y Z d F D n 9 U t q Q b K q i f F j f O c t s b C w 3 e Q 6 E S U 0 N a G i m N M G E 8 t z M P j 5 x i 8 e U / K I T L c W W 8 1 d f A 8 T E i w jJ 2 A d d x V 5 e 7 H 9 P 7 L h W b 5 s M k 9B O j e q P 9 x R F 6 v u e b N C R K G U 6 D c I C M j 0 t 0 3 T w T A c h V 3 g 0 y D q w R D 1 s T s d f C O p Y m X u d D F B j Z l E Y W H j m m r L m Q B n e 2 m g o O y Q Z j B x U F I n J K 4 7 k Q 2 + 7 T I p n i n t j r S 4 y / 7 N q G l u T J U n r r M V b k 7 W 2 u S y 2 q S 0 s 0 d x z W V R t i 9 8 P G h W C m w V b n 8 L T r k G Z k X l A G W a O 6 2 Y z a m m z L o / F T g T o p M r n w Z 7 m 6 M o H E W v 7 g 9 3 n v R 2 r K G b 6 B b a Q B F 6 i H b Q C 7 S Lx o h 5 7 7 w P 3 i f v s / / e / + h / 8 b 8 e t / p e z 7 m B / g n / x 0 9 Y V g H C < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L m 2 s 0 3 u P

Figure 3: Making latency differentiable by introducing latency regularization loss.

Figure 4: ProxylessNAS consistently outperforms MobileNetV2 under various latency settings.

(c) Efficient mobile model found by ProxylessNAS.

Figure 6: Efficient models optimized for different hardware. "MBConv3" and "MBConv6" denote mobile inverted bottleneck convolution layer with an expansion ratio of 3 and 6 respectively. Insights: GPU prefers shallow and wide model with early pooling; CPU prefers deep and narrow model with late pooling. Pooling layers prefer large and wide kernel. Early layers prefer small kernel. Late layers prefer large kernel. Model Top-1 (%) GPU latency CPU latency Mobile latency Proxyless (GPU) 75.1 5.1ms 204.9ms 124ms Proxyless (CPU) 75.3 7.4ms 138.7ms 116ms Proxyless (mobile) 74.6 7.2ms 164.1ms 78msTable4: Hardware prefers specialized models. Models optimized for GPU does not run fast on CPU and mobile phone, vice versa. ProxylessNAS provides an efficient solution to search a specialized neural network architecture for a target hardware architecture, while cutting down the search cost by 200× compared with state-of-the-arts(Zoph & Le, 2017;Tan et al., 2018).

ProxylessNAS achieves state-of-the art accuracy (%) on ImageNet (under mobile latency constraint ≤ 80ms) with 200× less search cost in GPU hours. "LL" indicates latency regularization loss. Details of MnasNet's search cost are provided in appendix C.

Table content: <table><tr><td colspan="4">Published as a conference paper at ICLR 2019</td><td/><td/><td/></tr><tr><td>Model</td><td colspan="2">Top-1 Top-5</td><td colspan="4">Mobile Hardware No Latency -aware Proxy Repeat (GPU hours) No Search cost</td></tr><tr><td>MobileNetV1 [16]</td><td>70.6</td><td>89.5</td><td>113ms</td><td>-</td><td>-</td><td>Manual</td></tr><tr><td>MobileNetV2 [30]</td><td>72.0</td><td>91.0</td><td>75ms</td><td>-</td><td>-</td><td>Manual</td></tr><tr><td>NASNet-A [38]</td><td>74.0</td><td>91.3</td><td>183ms</td><td/><td/><td>48, 000</td></tr><tr><td>AmoebaNet-A [29]</td><td>74.5</td><td>92.0</td><td>190ms</td><td/><td/><td>75, 600</td></tr><tr><td>MnasNet [31]</td><td>74.0</td><td>91.8</td><td>76ms</td><td/><td/><td>40, 000</td></tr><tr><td>MnasNet (our impl.)</td><td>74.0</td><td>91.8</td><td>79ms</td><td/><td/><td>40, 000</td></tr><tr><td colspan="2">Proxyless-G (mobile) 71.8</td><td>90.3</td><td>83ms</td><td/><td/><td>200</td></tr><tr><td>Proxyless-G + LL</td><td>74.2</td><td>91.7</td><td>79ms</td><td/><td/><td>200</td></tr><tr><td colspan="2">Proxyless-R (mobile) 74.6</td><td>92.2</td><td>78ms</td><td/><td/><td>200</td></tr><tr><td/><td/><td/><td>76.7</td><td/><td/><td/></tr><tr><td/><td>74.6</td><td>1.83x faster</td><td>74.7</td><td/><td/><td/></tr><tr><td/><td>72.0</td><td/><td/><td/><td/><td/></tr><tr><td>68.2</td><td/><td/><td/><td/><td/><td/></tr><tr><td>65.4</td><td/><td/><td/><td/><td/><td/></tr></table>

The over-parameterized network is trained on the remaining training images with batch size 256.Published as a conference paper at ICLR 2019

Table content: <table><tr><td>3x224x224</td><td>Conv 3x3</td><td>40x112x112</td><td>MB1 3x3</td><td>24x112x112</td><td>MB3 5x5</td><td>32x56x56</td><td>MB3 7x7</td><td>56x28x28</td><td>MB3 3x3</td><td>56x28x28</td><td>MB6 7x7</td><td>112x14x14</td><td>MB3 5x5</td><td>112x14x14</td><td>MB6 5x5</td><td>128x14x14</td><td>MB3 3x3</td><td>128x14x14</td><td>MB3 5x5</td><td>128x14x14</td><td>MB6 7x7</td><td>256x7x7</td><td>MB6 7x7</td><td>256x7x7</td><td>MB6 7x7</td><td>256x7x7</td><td>MB6 5x5</td><td>256x7x7</td><td>MB6 7x7</td><td>432x7x7</td><td>Pooling FC</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Region 1</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>April</td><td>May</td></tr></table>

Peer review (iclr2019):
 This paper addresses the problem of architecture search, and specifically seeks to do this without having to train on "proxy" tasks where the problem is simplified through more limited optimization, architectural complexity, or dataset size. The paper puts together a set of existing complementary methods towards this end, specifically 1) Training "cumbersome" networks as in One Shot and DARTS, 2) Path binarization to address memory requirements (optimized using ideas in BinaryConnect), and 3) optimizing a non-differentiable architecture using REINFORCE. The end result is that this method is able to find efficient architectures that achieve state of art performance with fewer parameters, can be optimized for non-differentiable objectives such as latency, and can do so with smaller amounts of GPU memory and computation. Strengths + The paper is in general well-written and provides a clear description of the methods. + Different choices made are well-justified in terms of the challenge they seek to address (e.g. non-differentiable objectives, etc.) + The results achieve state of art while being able to trade off other objectives such as latency + There are some interesting findings such as the need for specialized blocks rather than repeating blocks, comparison of architectures for CPUs vs. GPUs, etc. Weaknesses - In the end, the method is really a combination of existing methods (One Shot/DART, BinaryConnect, use of RL/REINFORCE, etc.). One novel aspect seems to be factorizing the choice out of N candidates by making it a binary selection. In general, it would be good for the paper to make clear which aspects were already done by other approaches (or if it*s a modification what exactly was modified/added in comparison) and highlight the novel elements. - The comparison with One Shot and DARTS seems strange, as there are limitations place on those methods (e.g. cell structure settings) that the authors state they chose "to save time". While that consideration has some validity, the authors should explicitly state why they think these differences don*t unfairly bias the experiments towards the proposed approach. - It*s not clear that the REINFORCE aspect is adding much; it achieves slightly higher parameters when compared against Proxyless-G, and while I understand the motivation to optimize a non-differentiable function in this case the latency example (on ImageNet) is never compared to Proxyless-G. It could be that optimized the normal differentiable objective achieves similar latency with the smaller number of parameters. Please show results for Proxyless-G in Table 4. - There were several typos throughout the paper ("great impact BY automatically designing", "Fo example", "is build upon", etc.) In summary, the paper presents work on an interesting topic. The set of methods seem to be largely pulled from work that already exists, but is able to achieve good results in a manner that uses less GPU memory and compute, while supporting non-differentiable objectives. Some of the methodological issues mentioned above should be addressed though in order to strengthen the argument that all parts of the the method (especially REINFORCE) are necessary.