Title: PROBABILISTIC MODEL-BASED DYNAMIC ARCHITECTURE SEARCH

Abstract: The architecture search methods for convolutional neural networks (CNNs) have shown promising results. These methods require significant computational resources, as they repeat the neural network training many times to evaluate and search the architectures. Developing the computationally efficient architecture search method is an important research topic. In this paper, we assume that the structure parameters of CNNs are categorical variables, such as types and connectivities of layers, and they are regarded as the learnable parameters. Introducing the multivariate categorical distribution as the underlying distribution for the structure parameters, we formulate a differentiable loss for the training task, where the training of the weights and the optimization of the parameters of the distribution for the structure parameters are coupled. They are trained using the stochastic gradient descent, leading to the optimization of the structure parameters within a single training. We apply the proposed method to search the architecture for two computer vision tasks: image classification and inpainting. The experimental results show that the proposed architecture search method is fast and can achieve comparable performance to the existing methods.

1. INTRODUCTION:

Convolutional neural networks (CNNs) make remarkable progress in various computer vision tasks. As researchers have developed deeper architectures and new components to improve performance, the architecture of CNNs is becoming complicated. Since numerous modules exist to construct CNN architectures, designing the appropriate CNN architecture for a target problem is critical. However, finding a better combination of basic modules for CNNs, e.g., convolutional and pooling layers, requires tremendous trial and error.

In this context, the methods for designing deep neural network architectures are actively proposed. Most existing methods treat the structure parameters, such as the type of layer and the connectivity, as hyper-parameters and optimize them by reinforcement learning (Zoph & Le, 2017) or evolutionary algorithms (Real et al., 2017; Suganuma et al., 2017) . These methods search for a better architecture that maximizes the performance for validation data in the hyper-parameter optimization manner, i.e., they need neural network training for an architecture evaluation. This approach has succeeded in finding the state-of-the-art CNN architectures; however, it is computationally inefficient in general. Reducing the computational cost of the architecture search is critical for practical usage. Several pieces of research have focused on reducing the computational cost of architecture search (Liu et al., 2018b; Pham et al., 2018) by reusing the trained weights on different architectures and simultaneously optimizing the weights and the structure parameters. Another practical requirement is that the architecture search methods have fewer hyper-parameters to reduce the total effort of the architecture search process. This work aims to develop an efficient architecture search method with fewer hyper-parameters. As neural network architectures can be represented by a sequence of categorical variables (Pham et al., 2018; Suganuma et al., 2017; 2018) , we consider the neural networks consist of categorical variables of the structure parameters and continuous variables of the weights. Introducing multivariate categorical distribution as the underlying distribution for the structure parameters, we formulate an expected loss function under the distribution that is differentiable with respect to (w.r.t.) both the weights and the parameters of the distribution. We iteratively update the weights and the param-eters of the distribution to the gradient steps within a single training, realizing a computationally efficient architecture search. The idea of the proposed method is based on Shirakawa et al. (2018) , which instantiates the algorithm using the multivariate Bernoulli distribution and has proofed the concept on the simple neural network structure optimization. We extend their work and make it possible to search flexible architectures. We call the algorithm proposed in this paper the probabilistic model-based dynamic architecture search (PDAS). The straightforward probabilistic modeling of the architectures adopted in this work leads to the simple update rule of the parameters of the distribution, which relates to the stochastic natural gradient method (Ollivier et al., 2017) . Moreover, we establish the parameter-adaptive architecture search method by injecting the learning rate adaptation mechanism of the stochastic natural gradient proposed in Nishida et al. (2018) .

We apply the PDAS to search the architectures for two computer vision tasks: image classification and inpainting. The experimental results show that the PDAS is fast and can achieve comparable performance to the existing architecture search methods on both tasks.

The contribution of this paper is as follows: (1) we derive the algorithm for categorical distribution and make it possible to apply the framework proposed in Shirakawa et al. (2018) to the architecture search spaces represented by categorical variables. To the best of our knowledge, the natural gradient of categorical distribution has not been introduced in the context of stochastic natural gradient methods.

(2) we show that PDAS, which has fewer hyper-parameters than efficient neural architecture search (ENAS) (Pham et al., 2018) , is fast and can reach state-of-the-art performance. The intrinsic hyper-parameters of PDAS are the sample size and the learning rate, but the learning rate can be adaptive.

2. PROBABILISTIC MODEL-BASED DYNAMIC ARCHITECTURE SEARCH:

Formulation: Following Shirakawa et al. (2018) , we consider the neural network φ(W, M ) having two different sets of parameters: the connection weights W ∈ W and the structure parameters M ∈ M. We assume that the weights are real-valued and differentiable for loss function; however, the structure parameters can be discrete, i.e., the loss function is not differentiable w.r.t. these parameters. Our original objective is minimizing the loss L(W, M ) = D l(z, W, M )p(z)dz, where D and l(z, W, M ) indicate the dataset and the loss function value of a datum z, respectively. Introducing a family of probability distributions p θ (M ) of M parametrized by a real-valued vector θ ∈ Θ, we formulate a minimization of the expected loss under p θ (M ), namely

EQUATION

where dM is a reference measure on M. That is, we try to minimize the loss L(W, M ) indirectly by minimizing G(W, θ). This formulation is inspired from the recently-introduced black-box optimization framework called Information Geometric Optimization (Ollivier et al., 2017) . The point is, one can choose the family of probability distributions so that G is differentiable w.r.t. both W and θ. It allows us to employ a gradient descent to optimize W and θ simultaneously within a single training process.

We apply a stochastic gradient descent to minimize G on W × Θ equipped with the Fisher metric on Θ. That is, we take the vanilla gradient w.r.t. W and the natural gradient (Amari, 1998) w.r.t. θ.They can be approximated by Monte-Carlo using λ samples drawn from p θ (M ) and mini-batch loss

L(W, M ) ≈ L(W, M ) = N -1 N k l(z k , W, M

) with mini-batch size N . Then, we get

EQUATION

where ∇θ ln p θ (M ) = F -1 (θ)∇ θ ln p θ (M ) is the natural gradient of the log-likelihood ln p θ (M ), and F (θ) is the Fisher information matrix of p θ (M ). Shirakawa et al. (2018) proposed the framework that simultaneously optimizes both W and θ by using approximated gradients of (2) and (3), and they instantiated the algorithm using the multivariate Bernoulli distribution. In other words, they used the binary vector to select the network components, such as the unit, layer, and type of activation.

Categorical Parameters: In this paper, we consider categorical parameters as they can represent more flexible network architecture than binary parameters. We extend the previous work by introducing the multivariate categorical distribution as p θ (M ). We denote the D-dimensional categorical variables by h = (h 1 , . . . , h D ) T and the number of categories for the i-th variable by K i (> 1). Let us introduce the one-hot representation of h i , denoted as m i ∈ {0, 1} Ki , where the entries of m i are all zero but for the h i -th entry, which is one. Our structure parameter vector M is then written as M = (m 1 , . . . , m D ) T .

We introduce the multivariate categorical distribution p θ as the underlying distribution for M , whose probability mass is

p θ (M ) = D i=1 Ki j=1 (θ ij ) mij

, where θ ij ∈ [0, 1] represents the probability of m ij to be 1 and must satisfy Ki j=1 θ ij = 1. The natural gradient of the log-likelihood for the above distribution is given by ∇θ ln p θ (M ) = M -θ, since the natural gradient of the log-likelihood for an exponential family with sufficient statistics T (M ) with the expectation parameterization θ = E[T (M )] is known to be ∇θ ln p θ (M ) = T (M ) -θ and the above parametrized distribution is an exponential family with the expectation parameterization. See Appendix A for details.

With this natural gradient of log-likelihood, we get the update rule of θ as follows: Learning Rate Adaptation for Stochastic Natural Gradient Ascent: The update formula (4) requires two hyper-parameters, namely the learning rate η θ and the number λ of Monte-Carlo samples. For Bernoulli distribution, Shirakawa et al. (2018) set λ = 2 and η θ = D -1 , the former of which is the minimum requirement to use a ranking-based utility transformation. As Bernoulli distribution is a special case of the categorical distribution with K i = 2 for all i, the learning rate setting may be generalized as

EQUATION

η θ = D i (K i -1)

-1 . However, as is not difficult to imagine, an adequate value for η θ depends heavily on problem characteristic and stage of optimization. Nishida et al. (2018) proposed to adapt η θ so as to keep the signal-to-noise ratio of successive parameter updates. It has been shown that the adaptive learning rate typically speeds up the convergence of the parameter vectors in the case of Bernoulli distributions. The adaptation mechanism can be applied to an arbitrary exponential family with expectation parameterization, which includes our categorical distribution.

We employ this learning rate adaptation in our experiments. See Appendix C for details. From the preliminary experiment in the classification task, we confirmed that the learning rate adaptation improves the quality and stability of the architecture search. In the learning rate adaptation, the sample size λ is fixed to two. From the viewpoint of stochastic approximation theory, the small learning rate has a similar effect of the large sample size. Therefore, even if the sample size equals two, we can optimize the distribution parameters properly with the appropriate small learning rate. In fact, we observed that the learning rate decreases by the learning rate adaptation mechanism.

Overall Algorithm: The overall algorithm of PDAS is displayed in Algorithm 1. We calculate the gradients of W and θ using different mini-batches from a dataset. ENAS (Pham et al., 2018) and differentiable architecture search (DARTS) (Liu et al., 2018b) , the methods for optimizing the structure parameters within a single training, use different datasets for the weight and structure parameter optimizations as well. Following these studies, we split the training dataset into halves Update W (t) by a SGD method 6:

Update θ (t) by (4) using M 1 , M 2 , and a mini-batch from D θ with updated W 3) are calculated using mini-batches from D W and D θ , respectively, and the parameter updates are performed alternately. As each dataset is sampled from the original one, the losses of mini-batch samples from both datasets approximate the original loss of all of the data if the dataset size is sufficiently large. Therefore, even if we use split datasets, we can view that the losses in the equations of ( 2) and (3) approximate the original loss.1 Note that we do not need the back-propagation for calculating (3). Namely, the computational cost of the θ update is less than that of W .

After the optimization of W and θ, we can get the most likely structure parameters as M * = argmax M p θ (M ), which is obtained trivially for the multivariate categorical distribution. Given the optimized structure parameters M * , we re-train the neural network represented by M * with initialized weights. In the re-training stage, we can exclude the redundant weights (the weight parameters in the unused layer modules) and no longer need to update them. Re-training the obtained architecture is a commonly used technique (Brock et al., 2018; Liu et al., 2018b; Pham et al., 2018) to improve final performance. We have experimentally observed that the re-training of W can improve the predictive performance.

3. RELATED WORK:

The ordinary architecture search methods (Real et al., 2017; Suganuma et al., 2017; Zoph & Le, 2017) for deep neural networks repeat the following steps: the architecture generation, the weight training, and the architecture evaluation. Since the weight training of deep neural networks is timeconsuming, the overall process requires a tremendous computational cost. Several techniques are introduced to reduce the training cost, such as inheriting the trained weights to the next candidate architectures (Real et al., 2017) and stopping the weight training based on the performance prediction (Baker et al., 2018) . In contrast, our method is computationally more efficient than these approaches because it only needs the training twice (including the re-training).

The existing methods similar to our method are SMASH (Brock et al., 2018) and ENAS (Pham et al., 2018) . SMASH randomly samples an architecture in memory-bank representation and determines its weights by a meta-network called HyperNet. Instead of training the weights in the generated network, the HyperNet is trained through back-propagation. Differently from our method, the probability distribution of network architectures does not change in SMASH, and it still needs the meta-network design. ENAS is a method based on the neural architecture search (NAS) (Zoph & Le, 2017) . NAS defines a recurrent neural network, called the controller, that generates a sequence of categorical variables representing architecture for the main task and optimizes the controller using the policy gradient method in a hyper-parameter optimization manner. ENAS shares the weight parameters in all generated architectures and optimizes the weights and the controller parameters alternatively.

Our method is similar to ENAS from the viewpoint of optimizing both of the weights and distribution parameters in a single training. The main difference between PDAS and ENAS is the probabilistic model of architectures, i.e., PDAS uses the categorical distribution, and ENAS uses the recurrent neural network. Since our method uses the categorical distribution, the modeling of architectures is intuitive and simple, and all we need is to design the categorical variables for representing architectures. As a result, we do not need to design the architecture of the controller neural network that is required for ENAS. In addition, the simple modeling of architecture makes it possible to derive the natural gradient in PDAS. As ENAS uses the LSTM network as the controller, it cannot derive the analytical natural gradient. Also, our method does not require to design the special operator for architecture search, such as crossover and mutation used in the evolutionary algorithms (Real et al., 2017) . By injecting the learning adaptation mechanism (Nishida et al., 2018) , we can further reduce the effort of the hyper-parameter tuning for PDAS. This property is convenient in practice. In the experiments, we always use the same hyper-parameter setting for PDAS. Another attractive property of PDAS is that the promising architecture is determined easily after the training described before, whereas it is difficult in ENAS.

4. EXPERIMENT AND RESULT:

We apply the proposed method, the probabilistic model-based dynamic architecture search (PDAS), to the task of finding better architectures for image classification and inpainting. In the architecture search, two research directions exist: developing an efficient search method (Liu et al., 2018b; Pham et al., 2018) and designing a search space (Liu et al., 2018a; Zoph et al., 2018) . Since PDAS is a search framework for neural network architectures, we concentrate on evaluating PDAS in terms of search efficiency: the quality of found architecture and the computational cost. Therefore, the experiment adopts the search spaces provided in the previous works, Pham et al. (2018) for classification and Suganuma et al. (2018) for inpainting. All experiments are run using a single NVIDIA GTX 1080Ti GPU, and PDAS is implemented using PyTorch 0.4.1 (Paszke et al., 2017) .

4.1. IMAGE CLASSIFICATION:

Dataset: We use the CIFAR-10 dataset which consists of 50,000 and 10,000 RGB images of 32 × 32, for training and testing. All images are standardized in each channel by subtracting the mean and then dividing by the standard deviation. We adopt the standard data augmentation for each training mini-batch: padding 4 pixels on each side, followed by choosing randomly cropped 32 × 32 images and by performing random horizontal flips on the cropped images. We also apply the cutout (DeVries & Taylor, 2017) to the training data.

Search Space: The search space is based on the one in Pham et al. (2018) and the author's code2 , which consists of models obtained by connecting two motifs (called normal cell and reduction cell) repeatedly. An example of the overall model structure can be found in Appendix D. Each cell consisted of B (= 5) nodes and receives the outputs of the previous two cells as inputs. Each node receives two inputs from previous nodes, applies an operation to each of the inputs, and adds them. Our search space includes 5 operations: identity, 3 × 3 and 5 × 5 separable convolutions (Chollet, 2017) , and 3 × 3 average and max poolings. The separable convolutions are applied twice in the order of ReLU-Conv-BatchNorm. We select a node by 4 categorical variables representing 2 outputs of the previous nodes and 2 operations applied to them. Consequently, we treat 4B-dimensional categorical variables for each cell. After deciding B nodes, all unused outputs of the nodes are concatenated as the output of the cell. In the reduction cell, all operations applied to the inputs of the cell have a stride of 2. The number of the categorical variables is D = 40, and the dimension of θ becomes 180.

Training Detail: In the architecture search phase, we optimize W and θ for 200 epochs with mini-batch size of 64. We stack 2 normal cells (N = 2) and set the number of channels at the first cell to 16. For the purpose of absorbing effect of the dynamic change in architecture, we fix affine parameters of batch normalizations. We use SGD with a momentum of 0.9 to optimize W . The learning rate changes from 0.025 to 0 followed the cosine schedule (Loshchilov & Hutter, 2017) . We apply weight decay of 3 × 10 -4 and clip the norm of gradient at 5. In the re-training phase, we optimize W for 600 epochs with mini-batch size of 80. We stack 6 normal cells (N = 6) and increase the number of channels at the first cell so that the model of the obtained architecture M * has nearly three million weight parameters. In contrast to the architecture search phase, we make all batch normalizations have learnable affine parameters because the architecture no longer changes. We apply the ScheduledDropPath (Zoph et al., 2018) dropping out each path between nodes, and the drop path rate linearly increases from 0 to 0.3 during the training. We also add the auxiliary classifier (Szegedy et al., 2016) with the weight of 0.4 that is connected from the second reduction cell. The total loss is a weighted sum of the losses of the auxiliary classifier and output layer. Other settings are the same as the architecture search phase. We conducted the experiment five times and report the average values.

Result and Discussion: Table 1 shows the comparison with other architecture search methods.

Since the search space of the first three methods is different from the one used in PDAS, we should not directly compare the test errors to PDAS. At least, we can say that NAS and EAS require much more computational cost than PDAS because they repeat training the model many times to optimize network architecture. Although the search cost of SMASH is reasonable, PDAS is still faster than SMASH.

The other conventional methods adopt search spaces similar to PDAS. 3 Compared with these methods, our method is the fastest and shows a comparable error rate to ENAS, DARTS, and RENASNet. The architecture search of PDAS is realized by optimizing θ using (4), and its computational cost is small, whereas ENAS updates the controller recurrent network that generates the structure parameters. This might be one reason that PDAS is fast. DARTS models the architecture by the mixture of all possible operations and connections and optimizes the weights and the continuous structure parameters (mixture coefficients) by the gradient-based optimization. In the architecture search phase, DARTS requires to compute all possible operations and connections to calculate the gradient. In contrast, since PDAS computes the gradient using a few sampled structures, it is computationally more efficient than DARTS. The error rates of NASNet and NAONet outperform our method, but they have enormous search costs if they are implemented on a single GPU. We observe PDAS can take a good balance between the test error rate and search cost. The method denoted RANDOM uses the architecture randomly sampled from our search space. The result shows that PDAS can find the better architecture in a reasonable computational time by optimizing the structure parameters in the single training.

We observed that the value of θ converges to a certain category, and the average value of max j θ ij , implying the convergence of the parameters of the distribution approaches 0.9 at 50th epoch. The architecture of the best model obtained by PDAS appears in Appendix E. We note that PDAS has fewer hyper-parameters and can be used by only providing the categorical variables for representing the architecture, whereas, other methods still leave the controller network design or strategy parameter tuning. As our algorithm is based on the stochastic natural gradient method, we can easily add the improving techniques, such as the learning rate adaptation used in PDAS.

4.2. INPAINTING:

The inpainting task is one of the image restoration tasks, restoring a clean image from a damaged image with large missing regions (e.g., masks). Suganuma et al. (2018) have shown the potential of the architecture search by the evolutionary algorithm for image restoration tasks including inpainting. In this section, we apply the PDAS to the problem of architecture search for inpainting and evaluate its performance. We refer the experimental setting employed in Suganuma et al. (2018) .

Dataset and Evaluation Measure: We use three benchmark datasets: the CelebFaces Attributes Dataset (CelebA) (Liu et al., 2015) , the Stanford Cars Dataset (Cars) (Krause et al., 2013) , and the Street View House Numbers (SVHN) (Netzer et al., 2011) . The CelebA is a large-scale human face image dataset that contains 202,599 RGB images. We select 101,000 and 2,000 images for training and test, respectively, in the same way as Suganuma et al. (2018) . All images were cropped to properly contain the entire face by using the provided bounding boxes and resized to 64 × 64 pixels. We test three different masks based on Suganuma et al. (2018) ; Yeh et al. (2017) ; a central square block mask (Center); a random pixel mask, as 80% of all the pixels were randomly masked (Pixel); and a half image mask, as a randomly selected vertical or horizontal half of the image (Half). The mask was randomly generated for each training mini-batch and each test image.

Following Suganuma et al. (2018) , we use two standard evaluation measures: the peak-signal to noise ratio (PSNR) and the structural similarity index (SSIM) (Wang et al., 2004) to evaluate the restored images. Higher values of these measures indicate a better image restoration.

Search Space: The search space we use is based on Suganuma et al. (2018) . We employ the CAE, which is similar to the RED-Net (Mao et al., 2016) , as a base architecture. The RED-Net consists of a chain of convolution layers and symmetric deconvolution layers as the encoder and decoder parts, respectively. The encoder and decoder parts perform the same counts of downsampling and upsampling with a stride of 2, and a skip connection between the convolutional layer and the mirrored deconvolution layer can exist. For simplicity, each layer employs either a skip connection or a downsampling, and the decoder part is employed in the same manner. In the skip connected deconvolution layer, the input feature maps from the encoder part are added to the output of deconvolution operation followed by ReLU. In the other layers, the ReLU activation is performed after the convolution and deconvolution operations. We prepare six types of layers: the combination of the kernel sizes {1 × 1, 3 × 3, 5 × 5} and the existence of the skip connection. The layers with different settings do not share weight parameters.

Since we consider the symmetric CAE, it is enough to represent the encoder part; we only need to determine the encoder part of the CAE by the categorical variables, and then the decoder part is automatically decided according to the encoder part. We consider N c hidden layers and the output layer. We encode the type, channel size, and connections of each hidden layer. The kernel size and stride of the output deconvolution layer are fixed with 3 × 3 and 1, respectively; however, the connection is determined by a categorical variable. The numbers of categories for the hidden layer type and the output channel size are 6 and 3, respectively. We select the output channel size of each layer from {64, 128, 256}. To ensure the feed-forward architecture and to control the network depth,

5. CONCLUSION:

We proposed an efficient architecture search method called PDAS that optimizes the parameters of the categorical distribution based on the stochastic natural gradient method during weight training. Our probabilistic modeling of the architecture is straightforward, and the derived algorithm has fewer hyper-parameters and can incorporate the learning rate adaptation mechanism. The experimental results for the image classification and inpainting tasks have shown that PDAS is fast and achieves comparable performance to the existing methods. In the future, we will apply PDAS to other neural network architecture searches (e.g., the recurrent networks) and large-scale datasets (e.g., ImageNet). Regarding the extension of PDAS, the distribution of continuous variables can be introduced to PDAS; then we will be able to optimize the architecture represented by mixed variables (e.g., categorical and continuous variables).

A DERIVATION OF THE NATURAL GRADIENT:

We derive the natural gradient of the log-likelihood associated with our categorical distributions. We first reduce the number of parameters by D, number of variables, by forcing the parameter value for the last category of each variable, namely θ iKi = 1 -Ki-1 k=1 θ ik . It does not decrease the degrees of freedom of the family of distributions. We let the reduced parameter and corresponding random variable denoted by θ and M , respectively. The probability mass reads

pθ( M ) = D i=1   Ki-1 j=1 (θ ij ) mij     1 - Ki-1 j=1 θ ij   1- K i -1 j=1 mij .

(5)

We continue to represent the deleted variables and parameters as m iKi and θ iKi for the simple notation. The element of derivative of log-likelihood is given by

∇ θij ln pθ( M ) = mij θij - m iK i θ iK i

.

The Fisher information matrix, E[∇θ ln pθ(M )∇θ ln pθ(M ) T ], is a block-diagonal matrix whose i-th diagonal block F i ( θ) (i = 1, . . . , D) is of form

EQUATION

where diag( θi ) is the diagonal matrix whose diagonal elements are the corresponding elements of θi = (θ i1 , . . . , θ iKi-1 ) and 1 is the vector consisting of 1 for all entries. Then, it is easy to see that its inverse is again block-diagonal, and each block reads (thanks to the Sherman-Morrison formula)

EQUATION

The natural gradient, the product of the inverse Fisher information matrix and the vanilla gradient, can be written as ∇θ ln pθ(M ) = F -1 ( θ)∇θ ln pθ(M ) = Mθ .

With the learning rate η θ , the θ update rule is written as

θ(t+1) = θ(t) + η θ λ λ n=1

u n ( Mnθ(t) ) .

The update of θ 

As a consequence, the θ update rule is reduced to (4).

B RESTRICTION FOR THE RANGE OF θ:

To guarantee a small yet positive probability for all combinations of categorical variables, the parameters are projected into a subset of the domain of the parameters, namely, θ min for each i = 1, . . . , D, where D is the number of categorical variables and K i > 1 is the number of categories for i-th variable. To realize this, we apply the following steps after θ update by (4):

θ ij ← max{θ ij , θ min i

} for all i and j, then (11)

EQUATION

)

The first line guarantees θ ij ≥ θ min i . The second line ensures 

:

We can formulate the update rules with different datasets by starting from different original objectives for the weight and distribution parameters as done in ENAS.

https://github.com/melodyguan/enas

The setting, e.g., types of operations, channel sizes, and the number of normal cells N differs among the methods.

Figures and Tables:

where η θ is the learning rate. As done inOllivier et al. (2017);Shirakawa et al. (2018), we transform the loss value L(W, M n ) into the utility u n to make the update invariant for the scale of the objective values. We use the following ranking-based utility transformation: u n = 1 for best λ/4 samples, u n = -1 for worst λ/4 samples, and u n = 0 otherwise. The update rule (4) is the generalization of the case for Bernoulli distribution and ensures no prior knowledge. Moreover, we set the lower bound for θ ij as θ min i = (D(K i -1)) -1 to leave open the possibility of generating all values (see Appendix B for details).

PDAS with learning rate adaptation Require: training data D = {D W , D θ } 1: Initialize the weight and distribution parameters as W (0) and θ (0) and the parameters used in the learning rate adaptation 2: for t = 0 • • • T do 3: Sample M 1 and M 2 independently from p θ (t) 4:Compute (2) using M 1 , M 2 , and a mini-batch from D W 5:

by the learning rate adaptation mechanism (See Appendix C for details) 8: end for 9: Get the structure parameters as M * = argmax M p θ (T ) (M ) 10: Re-train the weights using the fixed architecture represented by M * as D = {D W , D θ }. The gradients (2) and (

Figure 2: A conceptual example of the decoded symmetric CAE architecture and the corresponding categorical variables. The decoder part is automatically decided from the encoder structure as a symmetric manner.

Comparison with other architecture search methods on CIFAR-10. The notation "+c/o" indicates the cutout(DeVries & Taylor, 2017). The search cost indicates the GPU days for the architecture search, i.e., without the re-training cost. The result of the architecture randomly sampled from our search space is also listed as RANDOM.

Table content: <table><tr><td>Method</td><td colspan="2">Search Cost Params (GPU days) (M)</td><td>Test Error (%)</td></tr><tr><td>NAS (Zoph &amp; Le, 2017)</td><td>16.8-22.4K</td><td>7.1</td><td>4.47</td></tr><tr><td>EAS (DenseNet) (Cai et al., 2018)</td><td>20</td><td>10.7</td><td>3.44</td></tr><tr><td>SMASHv2 (Brock et al., 2018)</td><td>1.5</td><td>16.0</td><td>4.03</td></tr><tr><td>NASNet-A + c/o (Zoph et al., 2018)</td><td>2000</td><td>3.3</td><td>2.65</td></tr><tr><td>NAONet + c/o (Luo et al., 2018)</td><td>200</td><td>128</td><td>2.07</td></tr><tr><td>NAONet-WS (Luo et al., 2018)</td><td>0.4</td><td>3.7</td><td>3.53</td></tr><tr><td>RENASNet + c/o (Chen et al., 2018)</td><td>6.0</td><td>3.5</td><td>2.98 (±0.08)</td></tr><tr><td>DARTS first order + c/o (Liu et al., 2018b)</td><td>1.5</td><td>2.9</td><td>2.94</td></tr><tr><td>DARTS second order + c/o (Liu et al., 2018b)</td><td>4</td><td>3.4</td><td>2.83 (±0.06)</td></tr><tr><td>ENAS + c/o (Pham et al., 2018)</td><td>0.45</td><td>4.6</td><td>2.89</td></tr><tr><td>RANDOM + c/o</td><td>-</td><td>3.21</td><td>4.12 (±0.44)</td></tr><tr><td>PDAS + c/o</td><td>0.24</td><td>3.20</td><td>2.98 (±0.12)</td></tr></table>

The Cars is a middle-scale cars image dataset that contains 16,185 RGB images, and it consists of 8,144 and 8,041 images for training and testing, respectively. Similar to the CelebA, all images were cropped by using the provided bounding boxes and resized to 64 × 64 pixels. The SVHN is a large-scale house-number image dataset that contains 99,289 RGB images without extra training data, and it consists of 73,257 and 26,032 images for training and testing, respectively. The images of SVHN were resized to 64 × 64 pixels. All images are normalized by dividing by 255, and we perform data augmentation of random horizontal flipping on the training images.

Table content: <table/>

Peer review (iclr2019):
 This paper presents a joint optimization approach for the continuous weights and categorical structures of neural networks. The idea is the standard stochastic relaxation of introducing a parametrised distribution over the categorical parameters and marginalising it. The method then follows by alternating gradient descent on the weights and the parameters of the categorical distribution. This exact approach was proposed in https://arxiv.org/abs/1801.07650 by Shirakawa et al. The only innovation in this work is that it uses categorical distributions with more than two values. This is a minor innovation. The experiments are however interesting as the paper compares to the latest hyper-parameters optimization strategies for neural nets on simple tasks (eg CIFAR10) and gets comparable results. However, given that this is the biggest contribution of the paper, it would have been nice to see results in more complex tasks, eg imagenet or translation. I very much enjoyed the simplicity of the approach, but the question of innovation is making wonder whether this paper makes the ICLR bar of acceptance. The paper is also hard to read because of many English typos.