user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving One-Shot NAS By Suppressing The Posterior Fading

Paper content:
There is a growing interest in automated neural architecture search (NAS).
To improve the efficiency of NAS, previous approaches adopt  weight sharing method to force all models share the same set of weights.
However, it has been observed that a model performing better with shared weights does not necessarily perform  better when trained alone.
In this paper, we analyse existing weight sharing one-shot NAS approaches from a Bayesian point of view and identify the posterior fading problem, which compromises the effectiveness of shared weights.
To alleviate this problem, we present a practical approach to guide the parameter posterior towards its true distribution.
Moreover, a hard latency constraint is introduced during the search so that the desired latency can be achieved.
The resulted method, namely Posterior Convergent NAS (PC-NAS), achieves state-of-the-art performance under standard GPU latency constraint on ImageNet.
In our small search space, our model PC-NAS-S attains76.8% top-1 accuracy, 2.1% higher than MobileNetV2 (1.4x) with the same latency.
When adopted to our large search space, PC-NAS-L achieves 78.1% top-1 accuracy within 11ms.
The discovered architecture also transfers well to other computer vision applications such as object detection and person re-identification.
Neural network design requires extensive experiments by human experts.
In recent years, there has been a growing interest in developing algorithmic NAS solutions to automate the manual process of architecture design (Zoph & Le, 2016; Liu et al., 2018a; Zhong et al., 2018; .
Despite remarkable results, early works on NAS (Real et al., 2018; Elsken et al., 2017) are limited to searching only using proxy or subsampled dataset due to the exorbitant computational cost.
To overcome this difficulty, (Bender et al., 2018; Pham et al., 2018) attempted to improve search efficiency via sharing weights across models.
These approaches utilize an overparameterized network (supergraph) containing every single model, which can be further divided into two categories.
The first category is continuous relaxation method (Liu et al., 2018c; Cai et al., 2018) , which keeps a set of so called architecture parameters to represent the model, and updates these parameters alternatively with supergraph weights.
The resulting model is obtained using the architecture parameters at convergence.
The continuous relaxation method suffers from the rich-get-richer problem (Adam & Lorraine, 2019) , which means that a better-performed model at the early stage would be trained more frequently (or have larger learning rates).
This introduces bias and instability to the search process.
The other category is referred to as one-shot method (Brock et al., 2017b; Bender et al., 2018; Chu et al., 2019) , which divides the NAS procedure into a training stage and a searching stage.
In the training stage, the supergraph is optimized along with either dropping out each operator with certain probability or sampling uniformly among candidate architectures.
In the search stage, a search algorithm is applied to find the architecture with the highest validation accuracy with shared weights.
The one-shot approach ensures the fairness among all models by sampling architecture or dropping out operator uniformly.
However, as identified in (Adam & Lorraine, 2019; Chu et al., 2019; Bender et al., 2018) , the problem of one-shot method is that the validation accuracy of the model with shared weights is not predictive to its true performance.
In this paper, we formulate NAS as a Bayesian model selection problem (Chipman et al., 2001 ).
This formulation is especially helpful in understanding the one-shot approaches in a theoretical way, which in turn provides us a guidance to fundamentally addressing one of the major issues of one-shot approaches.
Specially, we show that shared weights are actually a maximum likelihood estimation of a proxy distribution to the true parameter distribution.
Most importantly, we identify the common issue of weight sharing, which we call Posterior Fading, i.e., as the number of models in the supergraph increases, the KL-divergence between true parameter posterior and proxy posterior also increases.
To alleviate the Posterior Fading problem, we proposed a practical approach to guide the convergence of the proxy distribution towards the true parameter posterior.
Specifically, we divide the training of supergraph into several intervals and maintain a pool of high potential partial models and progressively update this pool after each interval .
At each training step, a partial model is sampled from the pool and complemented to a full model.
To update the partial model pool, we first generate candidates by extending each partial model and evaluate their potentials, keeping the best performancing ones.
The search space is effectively shrunk in the upcoming training interval.
Consequently, the parameter posterior get close to the desired true posterior during this procedure.
Main contributions of our work is concluded as follows:
• We for the first time analyse one-shot approaches from a Bayesian point of view and identify the associated disadvantage which we call Posterior Fading.
• Guided by the theoretical result, we introduce a novel NAS algorithm fundamentally different from existing one-shot methods, which guides the proxy distribution to converge towards the true parameter posterior.
To examine the effectiveness of our newly proposed method, we benchmark its performance on ImageNet (Russakovsky et al., 2015) against the existing methods.
In one typical search space (Cai et al., 2018) , our PC-NAS-S attains 76.8% top-1 accuracy, 0.5% higher and 20% faster than EfficientNet-B0 (Tan & Le, 2019a) , which is the previous state-of-the-art model in mobile setting.
To further demonstrate the advantage of our method, we test it on a larger space and our PC-NAS-L boosts the accuracy to 78.1%.
In this paper, a new architecture search approach called PC-NAS is proposed.
We study the conventional weight sharing approach from Bayesian point of view and identify a key issue that compromises the effectiveness of shared weights.
With the theoretical motivation, a practical method The operators in our spaces have structures described by either Conv1x1-ConvNxM-Conv1x1 or Conv1x1-ConvNxM-ConvMxN-Conv1x1.
We define expand ratio as the ratio between the channel numbers of the ConvNxM in the middle and the input of the first Conv1x1.
Small search space Our small search space contains a set of MBConv operators (mobile inverted bottleneck convolution (Sandler et al., 2018) ) with different kernel sizes and expand ratios, plus Identity, adding up to 10 operators to form a mixoperator.
The 10 operators in our small search space are listed in the left column of Table 5 , where notation OP X Y represents the specific operator OP with expand ratio X and kernel size Y.
Large search space We add 3 more kinds of operators to the mixoperators of our large search space, namely NConv, DConv, and RConv.
We use these 3 operators with different kernel sizes and expand ratios to form 10 operators exclusively for large space, thus the large space contains 20 operators.
For large search space, the structure of NConv, DConv are Conv1x1-ConvKxK-Conv1x1 and Conv1x1-ConvKxK-ConvKxK-Conv1x1, and that of RConv is Conv1x1-Conv1xK-ConvKx1-Conv1x1.
The kernel sizes and expand ratios of operators exclusively for large space are lised in the right column of Table 5 , where notation OP X Y represents the specific operator OP with expand ratio X and K=Y.
There are altogether 21 mixoperators in both small and large search spaces.
Thus our small search space contains 10 21 models, while the large one contains 20 21 .
The specifications of PC-NAS-S and PC-NAS-L are shown in Fig. 3 .
We observe that PC-NAS-S adopts either high expansion rate or large kernel size at the tail end, which enables a full use of high level features.
However, it tends to select small kernels and low expansion rates to ensure the model remains lightweight.
PC-NAS-L chooses lots of powerful bottlenecks exclusively contained in the large space to achieve the accuracy boost.
The high expansion rate is not quite frequently seen which is to compensate the computation utilized by large kernel size.
Both PC-NAS-S and PC-NAS-L tend to use heavy operator when the resolution reduces, circumventing too much information loss in these positions.
224×224×3  112×112×16  112×112×16  56×56×32  56×56×32  56×56×32  56×56×32  28×28×64  28×28×64  28×28×64  28×28×64  14×14×136  14×14×136  14×14×136  14×14×136  14×14×136  14×14×136  14×14×136  14×14×136  7×7×264  7×7×264  7×7×264

assistant: text: Our paper identifies the issue of existing weight sharing approach in neural architecture search and propose a practical method, achieving strong results.