Deep Bayesian Structure Networks
Authors: Zhijie Deng
Yucen Luo
Jun Zhu
Bo Zhang
Venue:  (2020)

Abstract: Bayesian neural networks (BNNs) introduce uncertainty estimation to deep networks by performing Bayesian inference on network weights. However, such models bring the challenges of inference, and further BNNs with weight uncertainty rarely achieve superior performance to standard models. In this paper, we investigate a new line of Bayesian deep learning by performing Bayesian reasoning on the structure of deep neural networks. Drawing inspiration from the neural architecture search, we define the network structure as random weights on the redundant operations between computational nodes, and apply stochastic variational inference techniques to learn the structure distributions of networks. Empirically, the proposed method substantially surpasses the advanced deep neural networks across a range of classification and segmentation tasks. More importantly, our approach also preserves benefits of Bayesian principles, producing improved uncertainty estimation than the strong baselines including MC dropout and variational BNNs algorithms (e.g. noisy EK-FAC). 

Under review as a conference paper at ICLR 2020
DEEP BAYESIAN STRUCTURE NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Bayesian neural networks (BNNs) introduce uncertainty estimation deep net- works by performing Bayesian inference on network weights. However, such models bring the challenges of inference, and further BNNs with weight uncer- tainty rarely achieve superior performance to standard models. In this paper, we investigate new line of Bayesian deep learning by performing Bayesian reason- ing on the structure of deep neural networks. Drawing inspiration from the neural architecture search, we define the network structure as gating weights on the re- dundant operations between computational nodes, and apply stochastic variational inference techniques to learn the structure distributions of networks. Empirically, the proposed method substantially surpasses the advanced deep neural networks across a range of classification and segmentation tasks. More importantly, our ap- proach also preserves benefits of Bayesian principles, producing improved uncer- tainty estimation than the strong baselines including MC dropout and variational BNNs algorithms (e.g. noisy EK-FAC).
INTRODUCTION
Bayesian deep learning aims at equipping the flexible and expressive deep neural networks with ap- propriate uncertainty quantification (MacKay] 1992 Neal 1995 Hinton & Van Camp 1993 Graves] 2011 Blundell et al., 20155) Gal & Ghahramani] (016). Traditionally, Bayesian neural networks (BNNs) introduce uncertainty in the network weights, addressing the over-fitting issue which stan- dard neural networks (NNs) are prone to. Besides, the predictive uncertainty derived from the weight uncertainty is also of central importance in practical applications, e.g., medical analysis. automatic driving, and financial tasks.
Modeling the uncertainty on network weights is plausible and well-evaluated (Blundell al. 2015 Ghosh et al., (2118). However, BNNs usually preserve benefits of Bayesian principles such as well- calibrated predictions at the expense of compromising performance and hence are impractical in real-world applications (Osawa et al. 2019), due to various reasons. On one hand, specifying sen- sible prior for networks weights is difficult (Sun et al. 2019 Pearce et al., 2019). On the other hand, the flexible variational posterior of BNNs comes with inference challenges (Louizos & Welling, 2017] [Zang et al.] [0188] Shi et al.) 2018 Recently, the efficient particle- based variational meth- ods (Liu & Wang. 2016) have been developed with promise, but they still suffer from the particle collapsing and degrading issues for BNNs due to the high dimension of the weights and the over- parameterization nature of such models (Zhuo et al. 2018 Wang et 2019).
In this work, we investigate a new direction of Bayesian deep learning that performs Bayesian rea- soning on the structure of neural networks while keeping the weights as point estimates. We propose an approach, named Deep Bayesian Structure Networks (DBSN). Specifically, in the spirit of differ- entiable neural architecture search (NAS) (Liu et al. 2019 Xie et al. 2019). DBSN builds a deep network by repeatedly stacking computational cell in which any two nodes (i.e. tensors) are con- nected by redundant transformations (see Figure Tp. The network structure defined as the gating weights on these transformations, whose distribution is much easier to capture than those of the high-dimensional network weights. To jointly optimize the network weights and the parameterized distribution of the network structure, we adopt stochastic variational inference paradigm (Blundell et al. 2015) and use the reparameterization trick (Kingma & Welling 2013). One technical chal- lenge is driving DBSN to achieve satisfying convergence, since the network weights can hardly fit all the structures sampled from the structure distribution. To overcome this challenge, we propose two techniques. First, we advocate reducing the variance of the sampled structures with simple mod-

---

Under review as a conference paper at ICLR 2020
aujan, on, Categerical
N
NP
BNNs with uncertainty the weights (left) vs. DBSN with uncertainty the network structure (right) (we only depict three operations between tensors N' and N² for simplicity). w and a represent network weights and network structure, respectively. In DBSN, w is also learnable.
Figure/ 1:
ification of the sampling procedure. Second, we suggest using more compact structure learning space than that of NAS, to make the training more feasible and more efficient.
There are at least two motivations that make DBSN appealing choice: 1) DBSN bypasses the frustrating difficulties of characterizing weight uncertainty and enables the performance-enhancee structure learning (Zoph & Le. 2016 Liu et al. 2019), so DBSN shall have better predictive per- formance than classic BNNs. 2) Previous analysis (Wang et al. 2019) shows that due to the over- parametrization nature of BNNs, the state-of-the-art inference algorithms for weight uncertainty can suffer from mode collapsing, as multiple configurations of weights with a fixed structure corre- spond to one single function. In contrast, DBSN compactly models the uncertainty of structure and performs inference in much lower-dimensional space, avoiding this issue and hence being able exhibit more calibrated predictive uncertainty. Moreover, in the perspective of NAS, DBSN is also promising as it provides another principled way to learn network structures by resorting to the Bayesian formalism instead of the widely used meta-learning formalism differentiable NAS.
To empirically validate these hypotheses, we evaluate DBSN with extensive experiments. We first testify the data fitting and structure learning ability DBSN on challenging classification and seg- mentation tasks. Then, compare the quality of predictive uncertainty estimates via calibration, which is a common concern in the community. We further evaluate the predictive uncertainty on adversarial examples and out-of-distributio samples, drawn from shifted distributions from the training data, to verify whether the model knows what it knows. At last, we perform an experiment validate promising application of DBSN in the one-shot NAS (Bender al. 2018 Guo al.) 2019). Surprisingly, across all the tasks, DBSN consistently achieves comparable or even better results than the strong baselines.
BACKGROUND
We first review the necessary background for DBSN and then elaborate DBSN in the next section.
2.1 STOCHASTIC VARIATIONAL. INFERENCE FOR BNNS
Let D = {(z,, y:) be a set N data points. BNNs are typically defined by placing a prior p(v) on some variables of interest (e.g., network weights network structure) and the likelihood is p(D(p). Directly inferring the posterior distribution p( ( 'D) is intractable because it is hard to integrate w.r.t. v exactly. Instead, variational BNNs (Hinton & Van Camp) 1993 Graves 2011 Blundell et al. 2015) suggest approximating p(v(D) (D) with a O-parameterized distribution q(v by minimizing the Hullback-Leible (KL) divergence between them:
min (gl(((v00)) (p(m()) = BB (oop(()))))) + (a(((((()))))( + log (1)
where (D) is a constant w.r.. and usually omitted in the minimization. To solve prob- lem (1). the most commonly used method is the low-variance reparameterization trick (Kingma & Welling 2013 Blundell et al. 2015), which replaces the sampling procedure v ~ q(v(0) with the corresponding deterministic transformation enable the direct gradient pack-propagation through 0.
1(0, e) with sample of parameter-free noise e, to
2.2 CELL-BASED DIFFERNNTABLEE NEURAL ARCHITECTURE SEARCH (NAS)
Cell-based NAS has shown promise Zooph et al. 2018 Pham et al.
2018
and been developed
to be differentiable for better scalability (Liu et al. 2019 Xie al.
2019 Weng al. 2019)
2

---

Under review as a conference paper at ICLR 2020
Generally, the network in cell-based differentiable NAS is composed of sequence of cells (e.g., modules) which have the same internal structure and are separated by upsampling or downsampling modules. Every cell contains B sequential nodes (i.e., tensors): N N'..... ..... N Each node N is connected to all of its predecessors N so long as < j by K possible redundant operations 0f (t.) (6.5)
KK e.g.. convolution, skip connection, pooling. The network structure is defined as (1 S < S B) where (x65)) A corresponds the gating weights the K' available operations from No to N Therefore, the information gathered from N to N is a weighted sum of the outputs from K different operations on N' (we denote the including the parameters of all the operations in the network as W):
K
(
(N::).
(2)
k=1
Then, the node N is calculated by summing all the information from its predecessors:
N
(3)
Meta-leaming-like gradient descent is adopted for optimization to reduce the prohibitive compu- tational cost needed by RL or evolution (Liu al. 2019 Xie al. 2019). However, the goal of the optimization is the network structure instead of the model performance. Thus, after training, this kind of NAS needs to prune the searched structure and re-train new network model with the compact structure for performance comparison, which is labor-intensive and is avoided in our work.
3 DEEP BAYESIAN STRUCTURE NETWORKS
In this work, we propose a novel Bayesian structure learning approach for the deep neural net- works. Concretely, we follow the network design of NAS but we view a as Bayesian variables and w as point estimates (see the graphical model in Figure D. To infer the posterior distribu- tion p(a = p(a)p(Pla..u) where p((x) is the prior (we omit its dependency on the hyper- p(D)
parameter 0g here), we adopt the techniques in Section 2. We assume both the prior and the in- troduced variational are fully factorizable categorical distributions, namely, p(a:) = II, (fa() where = (0((.)) e k S < j S B} denotes the train- able categorical logits. We rewrite Eq. [ and obtain the negative evidence lower bound (ELBO):
and q( (x Ih
L(O, w) = - Eytatt) [log ((oogggDDaa,,, w)] + (xI((((a(0)) (p(()))))
(4)
Notably, minimizing L w.r.t. O and corresponds to Bayesian inference posteriori (MAP) estimation of u respectively. Thus, the optimization of the network structure and network weights can be unified as ming, L(0, w)) To resolve this, we relax both ((a(t
and maximum
and q( (t.5) ((65)) to be the concrete distributions (Maddison et al. 2016). Then, samples a from q(((()) generated via the softmax transformation:
9= c) (softmax + (1.)))/))
(5)
where 6 = (E( E R (3.ه) Gumbel i.i.d.) are the Gumbel variables and T E R, is the temperature. Then we derive the following gradient estimators:
VoL(e, w) E.l V log c), w) + Vo log (og c)(o) V ()))
(6)
V V w) E. V w log pog (D)g(( w.)]
(7)
The first term in Eq. (6) corresponds to the gradient of the negative log likelihood and we leave how estimate the last terms (i.e. log densities) in the next section. In practice, we approximate the expectation in Eq. 66 and Eq. (7) with 7' Monte Carlo (MC) samples, and update the structure and the weights w simultancously.
After training, we gain the following predictive distribution:
(8) where 0* and w" denote the converged parameters. Eq. (8) implies that the model predicts by ensembling the predictions of the networks whose structures are randomly sampled.
ply|=nnww w²) Eg e.)) (re>)l((u (ryww Or, w
We will refer to the cell-based differentiable NAS as NAS for short if there is no mislcading. This because we use regularizor weights, e.g., weight decay, alleviate over-fitting.
3

---

Under review as a conference paper at ICLR 2020
samplrer sumplee sampleez samolees
PoLp 1.0
Payy 0.5
B.. 0.3
Pa.p=0.1 0.1
Bn.p 0
Figure 2: Each column includes samples a (1J) from adaptive concrete distribution with some ((13) at 1. Samples in every row share the same e(.)), The base class probabilities are softmax(0 ((())) [0.05, 0.05, 0.5, 0.4) in each sample.
3.1 ADAPTIVE CONCRETE DISTRIBUTION
The weight sharing mechanism in DBSN is non-trivial contribution for Bayesian structure learning, enabling computationally efficient optimization. But it also causes unignorable training challenges. Specifically, because of the limited capacity of the shared weights w, we have challenges to train it sufficiently well be suitable for all the structures. The under-fittigg learning of a's variational posterior and results in unsatisfying convergence of the whole model. We note that an analogous phenomenon was also observed by Mackay et al. (2019) in the gradient-based hyper-parameter optimization scenario.
w then brings bias in the
Therefore, to facilitate w to fit the structure distribution better and eventually benefit the Bayesian structure learning, we expect to reduce the variance of the structure distribution. Specifically, we analyze the reparameterization procedure of the concrete distribution, and propose to multiply a tunable scalar (3(,5) with (f--)) in the sampling:
a (i.) g (((5))) ((( (f-))) = softmax((0(0.)) + (( (f))
Accordingly, we derive the log probability density of the adaptive concrete distribution which is slightly different from that of the concrete distribution (see the detailed derivation in Appendix AA):
log p(a (i.5) (o(55) ((ti)) -log((K 1))) + (K 1)log (K
1) log (1,5)
log (15)
Σ
(i.)) log a (k.))
K
K LEE k=
0 ((o)
log α (1
(10)
B(.))
(
where LEE represents the log-sum-exp operation. With this, the last two terms of Eq. 66 can be estimated exactly.
Obviously, the adaptive concrete distribution degrades to the concrete distribution when (i.)) = 1. As shown in Figure - sliding (+d) from 1 to 0 decreases the diversity of the sampled structures gradually. Therefore, we should also keep ((iJ) from being too small to avoid the over-fitting issue which point-estimate structure (i.e., (b,))
0) may suffer from. In practice, we choose to gradually reduce the sample variance along with the convergence of the weights, by decaying (b.j) from to 0.5 with linear schedule in the training.
3.2 PRACTICAL IMPROVEMENTS OF THE STRUCTURE LEARNING SPACE
In order make the training more stable and more efficient, modify changes to the structure learning space (i.c., the support of the structure distribution) commonly adopted in NAS.
Overall modification. To facilitate more effective information flow in the cell, we let the input of cell (i.e., the output of the previous cell) be fixedly connected to all the internal nodes by 1x 1/3 x convolutions in the classifcattoon/semmentte B internal nodes, as shown in Appendix FF The resulted nodes are concatenated along with the input get the cell's output. In spirit of DenseNet (Huang et al. 2017) and FC-DenseNet (Jégou
tasks. We only learn the connections between the
4

---

Under review as a conference paper at ICLR 2020
et al. 2017 we constrain the downsampiin//upsamping modules be the typical BN-ReLU-Com Poolig/Cony Trannspooee operations, to ease the learning of the network structure.
Batch normalization. NAS usually adopts the order of ReLU-Conv- BN in operations. However, in the searching stage, the learnable affine transformations in batch normalizations are always disabled to avoid the output rescaling issue (Liu et al. 2019). NAS does not suffer from this since it trains other network with learnable batch normalizations in the extra re- re-training stage. Instead, DBSN has fix the issue because we do not re-train the model. Thus, we propose put complete batch nor- malization in the front of the next layer. Namely, we adopt the layers, where the first BN has learnable affine parameters while the second one does not.
convolutional
Candidate operations. In order to make the training more efficient, we remove the operations which are popular in NAS but unnecessary in DBSN, including all the 5x5 convolutions that can be replaced by stacked 3 3 convolutions, and all the pooling layers which are mainly used for the downsampling module. Then, the candidate operations in DBSN are: 3x3 separable convolutions, 3x3 dilated separable convolutions, identity and zero. We follow Liu et al. (2019) for the detailed settings of these operations.
To obtain the j node in cell, there are (j - K) operations from its predeces- sors to calculate, which can be organized into K groups according to the operation type. Note that the operations in a group independent, we advocate replacing them with a group operation (e.g., group convolution), which improves the efficiency significantly.
Group operation.
3.3 DISCUSSION
One may concern that the practical choice of weight sharing could push the structure distribution toward the most likely point for the weights and result in Dirac structure distribution. However, the prior keeps the variational posterior from collapsing via a KL regularization (last term of Eq. Besides, recall that w is set including the parameters of all the redundant operations. Then, in fact, different network structures adjust w.r.t. different subsets of w, further alleviating the structure collapsing issue. The widely used technique of MC Dropout (Gal & Ghahramani] 2016 Gal et al. 2017] can also be seen as using the same weights for different structures. Their empirical results also prove that this kind of model choice is reasonable. Nevertheless, capturing the dependency of w on a may indeed bring more accurate modeling and we leave this as future work.
We also emphasize that using point estimates for the weights benefits the whole model's learning significantly. On one hand, as stated in the introduction, there are still frustrating difficulties to achieve scalable Bayesian inference on the nigh-dimensional network weights, which is also proven by the results in Table [ Table 3 and Appendix On the other hand, DBSN deploys weight decay regularizor on weights, which impicitly imposes a Gaussian prior on w. Then, DBSN performs maximum a posteriori (MAP) estimation of w, namely, estimating the mode of w's posterior distri- bution p(w) ((w()), DD), which can be viewed as doing approximate Bayesian inference on w.
4 RELATED WORK
Learning flexible Bayesian models has long been the goal of the community (MacKay 1992 Neal
1993 Balan et al. 2013: Wang & Yeung] 2016). The stochastic variational inference methods for Bayesian neural networks are particularly appealing owing to their analogy to the ordinary back- propagation (Graves 2011 Blundell al. (0155). More expressive distributions, such matrix- variate Gaussians (Sun et al. 2017) or mu muliipicaativ normalizing flows (Louizos & Welling (017). have also been introduced to represent the posterior dependencies, but they are hard to train without heavy approximations. Recently, there is an increasing interest in developing Adam-like optimizers to perform natural- -raniiet variational inference for BNNs (Zhang et al.) 2018 |ae et al. 2018 Khan et al. 2018). Despite enabling the scalability, these methods seem to demonstrate compromis- ing performance compared to the ffae--ffthhear deep models. Interpreting the stochastic techniques of the deep models as Bayesian inference is also insightful (Gal & Ghahramani 2016 Kingma et al.) 2013 Teye et al. 2018 Mandt et al. 2017: Lakshminarayanan et al. 20.7), but these methods still have relatively restricted and inflexible posterior approximations. Dikov & Bayer (2019) propose a unified Bayesian framework to infer the posterior of both the network weights and the structure, which is most similar to DBSN, but the network structure considered by them, namely layer size
5

---

Under review as a conference paper at ICLR 2020
and network depth, is essentially impractical for complicated deep models. Instead, we inherit the design of the structure learning space for NAS, and provide insightful techniques to improve the convergence, thus enabling effective Bayesian structure learning for deep neural networks.
Neural architecture search (NAS) has drawn tremendous attention, where reinforcement learn- ing (Zoph & Le 2016, Zoph et al.; 2018 Pham et al. 2018). evolution (Real al. 2019) and Bayesian optimization (Knddasayy et al.] 2018) have all been introduced to solve it. More recently, differentiable NAS (Liu al. 2019: Xie et al. 2019; Cai al. 2019 Wu et al. 2019) attractive because it reduces the prohibitive computational cost immensely. However, existing differentiable NAS methods search the network structure in meta-learning way (Finn et al., 2017). and need to re-train another network with the pruned compact structure after the searching. In contrast, DBSN unifies the learning of weights and structure in one training stage, alleviating the mismatch of struc- tures during the search and re-training, as well as inefficiency issues suffered by differentiable NAS.
5 EXPERIMENTS
To validate the structure learning ability and the predictive performance of DBSN, we first evaluate it on image classification and segmentation tasks. For the estimation of the predictive uncertainty. we concern model calibration and generalization of the predictive uncertainty to adversarial examples as well as duitt--disti samples, following existing work. We show that DBSN outperforms strong baselines in these tasks, shedding light on practical Bayesian deep learning.
5.1 IMAGE CLASSIFICATION ON CIFAR-10 10 AND CIFAR-100
Sctup. We set B = 7, and K 4, thus, consists of 7 6/2 21 sub-variables. The whole network is composed of 12 cells and 2 downsampling modules which have channel compression factor of 0.4 and are located the 1/3 and 2/3 depth. We employ a 3x3 convolution before the first cell and put a global average pooling followed by fully connected (FC) layer after the last cell. The redundant operations all have 16 output channels. We initialize w and @ following He al. 2015) and Liu et 2019), respectively. The prior distributions of (i.)) are set to be concrete distributions with uniform class probabilities. A momentum SGD with initial learning rate 0.1 (divided by 10 at 50% and 75% of the training procedure following (Huang et al.) 2017)) momentum 0.9 and weight decay 10-4 is used to train the weights w. An Adam optimizer with learning rate 3 x 10-4, momentum (0.5, 0.999) is used to learn O. We deploy the standard data augmentation scheme (mirroring/shifting whole training set is used for optimization. We train DBSN for 100 epochs with batch size 64, which takes one day on 4 GTX 1080-Tis. The implementation depends PyTorch (Paszke et al., 2017) and the codes are available online at witttts::/// / com/ anonymousest/ /DBSN
and normalize the data with the channel statistics. The
Baselines. Besides comparison to the advanced deep models, we also design a series of baselines for fair comparisons. 1) DBSN#: we substitute the concrete distribution for the adaptive concrete distribution. 2) DBSN-1:
T 1 sample in the gradient estimation. 3) Fixed a: we fix the structure of the network by setting the weight of every operation to be 1/ 4) Dropout: based on Fixed a, we further add dropout on every computational node with a drop rate of 0.2. 5) Drop-path: based on Fixed a, we further apply drop-path (Larsson et al. 2016) regularisation on the convolutional redundant operations with path drop rate of 0.3. 6) Random a: we fix the distributions of (+-5)) concrete distributions with uniform class probabilities and only train w with randomly sampled a. 7) PE: we view the structure as point estimates and train it as well as w simultaneously. 8) DARTS: we view the structure as point estimates but we train on half of the training set while train w on the other half, resembling the first order DARTS (Liu et al. 2019). 9) NEK-FAC: we train a VGG16 network with weight uncertainty using the noisy EK-FAC (Bae et al. 2018)) and the corresponding default settings. 10) BNN-LS: we replace all the convolutional and fully connected layers in PE with their Bayesian counterparts to build a BNN with Learnable Structure. 11) Fully Bayesian DBSN: we replace all the convolutional and fully connected layers in DBSN with their Bayesian counterparts to build a Fully Bayesian neural network. In BNN-LS and Fully Bayesian DBSN, we employ fully factorized Gaussian distributions weights and adopt BBB (Blundell et al.) 2015) for inference. When testing DBSN, DBSN DBSN-1, Random a, NEK-FAC, Dropout, Drop-path, BNN-LS and Fully Bayesian DBSN, we ensemble the predictive

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison with competing baselines in terms of the number of parameters and test error rate. DBSN and its variants have 1.1 M parameters on CIFAR-100 due to larger FC layer.
Method
Params (M)
CIFAR-10 ( (%)
CIFAR-100 (%
ResNet (He et al.
2016a)
1.7
6.61
Stochastic Depth Huang et al. 2016
1.7
5.23
24.58
ResNet (pre-activation) (He et al. 2016b
1.7
5.46
24.33
NenseNet (Huang et al., 2017)
1.0
5.24
24.42
DenseNet-BC (Hunng et al. 2017
0.8
4.51
22.27
NEK-FAC (Bae al.) 2018
3.7
7.43
37.47
BNN-LS
2.0
9.85 + 0.42
30.98 + 0.36
Fully Bayesian DBSN
2.0
9.57 0.55
31.39 + 0.06
DBSN
1.0
4.98 0.24
22.50 0.26
DBSN
1.0
5.22 0.34
22.78 0.19
DBSN-1
1.0
5.60
23.44 + 0.28
Fixed a
1.0
5.66 #.224
24.27 0.15
Random
1.0
6.12
23.60 + 0.19
Dropout
1.0
5.83+0.. 0.19
23.67 + 0.28
Drop-path
1.0
5.77 4.77
23.12 0.13
PE
1.0
5.79 + 0.34
24.19 + 0.17
DARTS
1.0
8.91 0.16
31.87 0.12
probabilities from 100 random runs (we adopt this strategy in all the following experiments, unless stated otherwise).
We repeat every experiment 3 times and report the averaged error rate and standard deviation in Table 1 Notably, DBSN demonstrates comparable performance with state-of-the-art deep ral networks. DBSN outperforms the powerful ResNet (He et al., (016a) and DenseNet (Huang al. 2017) with statistical evidence, and only presents modestly higher error rates than those of De (Huang et al. 2017), which probably results from the usage of the expressive and efficient bottleneck layer in DenseNet-BC. This comparison highlights the practical value of DBSN.
Comparisons between DBSN and the baselines designed by ourselves are more insightful and con- vincing. 1) DBSN surpasses DBSN* revealing the effectiveness of the adaptive concrete distribu- tion. 2) DBSN-1 is remarkably worse than DBSN owing to the higher variance of the estimated gradients with only one sample. 3) Comparison of DBSN and Fixed a validates that adapting the network structure w.r.t. the data distribution benefits the fitting of the model, resulting in sub- stantially enhanced performance. 4) Random a, Dropout, and Drop-path train the networks with meanually-desinnen untunable randomness, and hence inferior to DBSN. 5) NEK-FAC gains rather compromising performance, with the powerful VGG16 architecture and one of the most ad- vanced variational BNNs algorithms, suggesting to prefer DBSN instead of the classic BNNs in the scenarios where the performance is a major concern. 6) BNN-LS and Fully Bayesian DBSN both have poor performance, due to the fundamental difficulties of modeling distributions over high dimensional weights. 7) PE and DARTS are two methods to learn the point-estimate network struc- ture, both of which fall behind in terms of the test error. In particular, DARTS is much worse as only trains the weights on half of the training set. This shows that DBSN is an appealing choice for effective neural structure learning with only one-stage training.
5.2 SEMANTIC SEGMENTATION ON CAMVID
To further verify that learning the network structure w.r.t. the data helps DBSN obtain better performance than the standard NNs and BNNs, we apply DBSN to the challenging segmentation benchmark Cam Vid (Brostow et al. 2008). Our implementation is based on the brief FC-DenseNet framework (Jégou al. 2017). Specifically, we only replace the original dense blocks with the struture-learnabll cells, without introducing further advanced techniques from the semantic seg- mentation community, figure out the performance gain only resulted from the learnable network structure. For the setup, we set B = 5 (same as the number of layers in every dense block of FC- DenseNet67) and T 1, and learn two cell structures for the downsampling path and upsampling

---

Under review as a conference paper at ICLR 2020
Table Comparison of semantic segmentation performance on CamVid dataset. * indicates results from our implementation.
Method
Pretrained
Params (M)
Mean IoU
Global accuracy
SegNet (Badrinarayanan et al. 2015
29.5
46.4
62.5
Bayesian SegNet (Kendall et al. 2015
29.5
63.1
86.9
FC-DenseNet67 (Jégou et al. 2017)
X
3.5
63.1*
90.4*
DBSN
x
3.3
65.4
91.4
Figure 3: Visualization of the segmentation and uncertainty results of DBSN on Cid From left to right: original image, ground-truth segmentation, the estimated segmentation, and pixel-wise predictive uncertainty. The black color in ground-truth labels represents the background (void) class.
path, respectively. We a momentum SGD with initial learning rate 0.01 (which decays linearly after 350 epochs), momentum 0.9 and weight decay 10-1 instead of the original RMSprop for better results. The other settings follow Jégou et al. (2017) and the classification experiments above. We also implement FC-DenseNeu67 as a baseline. We present the results in Table and Figure
is evident that DBSN surpasses the competing FC-DenseNet67 by large margin while ing fewer parameters. DBSN also demonstrates significantly better performance than the classic Bayesian SegNet which adopts MC dropout for uncertainty estimation. We emphasize this exper- iment shows that the proposed approach is generally applicable. It is also worth noting that the uncertainty produced by DBSN interpretable (see Figure 33) the edges of the objects and the regions which contain overlapping have substantially higher uncertainty than the other parts.
5.3 ESTIMATION OF PREDICTIVE UNCERTAINTY
To validate that DBSN can provide promising predictive uncertainty, we evaluate via calibra- tion. We further examine the predictive uncertainty on adversarial examples and the model knows what it knows. We also pay particular attention the com- parison between Drop-path and Dropout to double-check if more structured randomness (Larsson al. 2016) benefits predictive uncertainty more.
(OOD) samples
Calibration is orthogonal to the accuracy (Lakshminarayanan et al. 2017) and can be well estimated by the Expected Calibration Error (ECE) (Guo et al. 2017). Thus, we evaluate the trained models on the test set of CIFAR-10 and CIFAR- 100 and calculate their ECE, as shown in Table 3 We also plot some reliability diagrams (Guo et al. 2017) in Appendix D to provide a direct explanation of calibration. Unsurprisingly, DBSN achieves state-of-the-an calibration. DBSN outperforms the strong baselines, Dropout and NEK-FAC. NEK-FAC, BNN-LS and Fully Bayesian DBSN all have much worse ECE than DBSN, implying structure uncertainty's superiority over weight uncertainty. We also notice that Drop-path is better than Dropout in terms of ECE, supporting our hypothesis that more structured randomness more beneficial the predictive uncertainty.
To test the predictive uncertainty on the adversarial examples, we apply the fast gradient sign method (FGSM) (Goodlellow al. 2014) attack the trained models on CIFAR-10 and CIFAR-100 using the corresponding test samples Then we calculate the predictive entropy of the generated adversar- ial examples and depict the average entropy Figure As expected, the entropy of DBSN grows rapidly as the perturbation size increases, implying DBSN becomes pretty uncertain when encoun-
For DBSN, DBSN Random NEK-FAC, Dropout, and Drop-path, attack using the ensemble of predictions from 30 stochastic and then test the manipulated adversarial examples with 30 well.
8

---

Under review as a conference paper at ICLR 2020
Table Comparison of model calibration in terms of the Expected Calibration Error (ECE). Smaller is better.
Dataset
DBSN+
Fixed cx
Dropout
Drop-path
NEK-FAC
BNN-LS
Fully Bayesian DESN
CIFAR-10
0.0109 0.0599
0.0111
0.0327 0.1259
0.0150 0.0617
0.0133
0.0131
0.0745
0,0966
CIFAR-100
0.U677
0.0524
0.1665
0.0700
0.1091
NEC-FAC
Raed
Drep rath
DARTS- 065R
NECFAC
Fied Drsppath
DARTS-*- 0654
Random ov-e-
Dropout---
DOSN
Random o-e-
Dropout-
1.0
100
1.0
2.0
1.8
1.6
Accuracy
0.6-
Entrapy
Accuracy
0.6-
0.4-
0.4
1.0
0.8
0.2
6.0
0.0
Perturbation size
Perturbation
Figure 4: Accuracy (solid) and entropy (dashed) vary w.r.t the adversarial perturbation size on CIFAR-10 (left) and CIFAR-100 (right).
tering adversarial perturbations. By contrast, the change in entropy of Dropout and NEK-FAC is relatively moderate, which means that these methods are not as sensitive as DBSN to the adversarial examples. Besides, Drop-path is still better than Dropout, consistent with the conclusion above. We also note that Random a has the highest predictive entropy. We speculate that this is because Random adopts the most diverse network structures (which results from the uniform class proba- bilities), and the ensemble of predictions from the corresponding networks is casier to be uniform. We further attack with more powerful algorithms, e.g., the Basic Iterative Method (BIM) (Kurakin et al. 2016), and provide the results in Appendix
Moreover, we look into the entropy of the predictive distributions on OOD samples, to adequately evaluate the quality of uncertainty estimation. We use the trained models on CIFAR-10 and CIFAR- 100, and take samples from the test set of SVHN as OOD samples. We calculate their predictive entropy and draw the empirical CDF of the entropy in Figure following[Louizos & Welling (017)) The curve close to the bottom right corner expected as means most OOD samples have relatively large entropy (i.e., low prediction confidence). Obviously, DBSN demonstrates comparable better results than the competing methods like Dropout and NEK-FAC, In addition, Drop-path attains
substantially improved results than Dropout. Analogous to the experiments adversarial examples, Random a provides impressive predictive uncertainty on the OOD samples.
In conclusion, DBSN consistently delivers state-of-ther thet--rr predictive uncertainty in various scenar- ios, validating the effectiveness of structure uncertainty.
5.4 RETHINKING OF THE ONE-SHOP NAS
One-shot NAS (Bender al., 2018 Guo al. 2019) first trains the weights super network and then searches for a good structure given the weights. This avoids the bias induced by the gradient- based joint optimization of the differentiable NAS. However, we argue that the super network trained with the fixed (Bender et al. 2018) uniformly sampled (Guo et al.) 2019) network structures can- not flexibly focus its capacity on the most crucial operations, harming the subsequent searching. To this end, we have conducted a set of experiments to check whether dynamically adjusting the network structure at the stage of weight training helps to find better network structures eventually. Observing that DBSN trains super network with adaptive network structures and Random trains a super network with unadjustable structures (similar to the uniform sampling used by Guo et al. 2019)), we choose search for the optimal structure distributions based the trained weights

---

Under review as a conference paper at ICLR 2020
0.8
NEK-FAC
NEK-MA
Bandam
Random
0.4
Drepout
Drop path
Droop path
0.2
PE
DARTS
0.2
PE
DARTS
DRSV
DOOk
DRSN
0.0
1.4
0.0
Figure 5: Empirical CDF for the entropy of the predictive distributions on SVHN dataset of models trained on CIFAR-10 (left) and CIFAR-100 (right). The curves that are closer to the bottom right corner are better.
Table 4: Comparison of the searched structure distributions based on the trained network weights from DBSN and Random a.
DBSN
Random
Test error (%) 5.46
5.98
from DBSN and Random o After searching, we train new networks with the searched structure distributions (fixed in the training) from scratch, and then test their performance. The results shown in Table 4 The searched structure distribution based on the weights trained by DBSN out- performs the other ably adapt the structure in the weight-training stage of one-shot NAS, which drives the most useful operations to optimized thoroughly and eventually yields more powerful network structures.
significantly, supporting hypotheses. Therefore,
propose reason-
5.5 VISUALIZATION THE LEARNED STRUCTURE DISTRIBUTIONS
We visualize the learned structure distributions in Appendix F The structure distributions for dif- ferent tasks look quite different, which implies that the structures are learned in a way that accounts for the specific characteristics in the data.
6 CONCLUSION
In this work, we have introduced novel Bayesian structure learning approach for deep neural net- works. The proposed DBSN draws the inspiration from the network design NAS and models the network structure as Bayesian variables. Stochastic variational inference is employed to jointly learn the network weights and the distribution the network structure. We further develop the adaptive concrete distribution and improve the structure learning space facilitate the convergence of the whole model. Empirically, DBSN has revealed impressive performance on the discriminative learning tasks, surpassing the advanced deep models, and presented state-of-the-as predictive uncer- tainty in various scenarios. In conclusion, DBSN provides a more practical way for Bayesian deep learning, without compromise between the predictive performance and the Bayesian uncertainty.
There are two major directions for future work. On one hand, the current DBSN is not efficient enough, some strategies need be discovered make DBSN more efficient. On the other hand, DBSN still has relatively restricted structure learning space. Thus, more operations can be introduced and more global network structures can be learned in future work.
We initialize ((+-)) randomly and initialize ((-)) with 1. Given the fixed network weights, we optimize (lob and ((+-)) by gradient deseent. The searching lasts for 20 epochs.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for robust semantic pixel-wise labelling. arXiv preprint arXiv:/505.077 2015.
Juhan Bac, Guodong Zhang, and Roger Grosse. Eigenvalue corrected noisy natural gradient. arXiv preprint arXiv:1811.125505 2018.
Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowl- edge. In Advances in Neural Information Processing Systems, pp. 3438- 0488-344,,00 2015.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing, pp. 549-558, 2018.
Charles Blundell, Julien Corneiiee, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622, 2015.
Gabriel Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recog- nition using structure from motion point clouds. In European conference computer vision, pp- 44-57. Springer, 2008
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019. URL https:
//openzeview. http:nee//eeee.
Georgi Dikov and Justin Bayer. Bayesian learning of neural network architectures. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 730-738, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050~1059, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581-3590, 2017.
Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez. Structured variational learning of bayesian neural networks with horseshoe priors. In International Conference on Machine Learning, pp- 1748, 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pp. 2348-2356. 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1321-1330. JMLR. org, 2017.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint 1rXiv:1904.00420, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1034-1034, 2015.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 201
Geoffrey Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the descrip- tion length of the weights. In in Proc. of the 6th Ann. ACM Conf. on Computational Learning Theory. Citeseer, 1993.
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 11-19, 2017.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing- Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pp. 2016-2025, 2018.
Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv preprint arXiv: /51 1.02680, 2015.
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivas- tava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning, pp. 2616-2625, 2018.
Diederik P Kingma and Max Welling, arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, pp. 25832 2015.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.0253: 2016.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural net- works without residuals. arXiv preprint arXiv:1605.07648 2016.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In
International Conference on Learning Representations, 2019. URL URL.ntts:://ppenevviie net Fet//orru:1dd-leeee
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, pp. 2378-230 2386, 2016.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 2218-2227. JMLR. org, 2017.
David JC MacKay, A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):488 472, 1992.
12

---

Under review as a conference paper at ICLR 2020
Matthew Mackay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In International Conference Learning Representations, 2019. /RRp::t::////p/ee/////
feet/forum?idee 2
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous
relaxation of discrete random variables. arXiv preprint arXiv:1111.00712, 2016.
Stephan Mandt, Matthew Hoffman, and David Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873 4907, 2017.
Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. PhD thesis, Citeseer, 1995.
Kazuki Osawa, Siddharth Swaroop. Anirudh Jain, Runa Eschenhagen, Richard E Tumerr Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with bayesian principles. arXiv preprint arXiv: 1906.09000000 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Tim Pearce, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Expressive priors in bayesian neural networks: Kernel combinations and periodic functions. arXiv preprint arXiv: (tt::/905.000777 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference Machine Learning, pp. 4092 4092-4101, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4780-4789, 2019.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. In International Conference on Learning Representations, 2018. URLLtttpss//openeeviies net. /forum?
id=r114eQW0Z
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertainty in bayesian neural networks. In Artificial Intelligence and Statistics, pp. 1283-1292, 2017.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks. In International Conference on Learning Representations, 2019. URL https:
Mattias Teye, Hossein Azizpour, and Kevin Smith. Bayesian uncertainty estimation for batch nor- malized deep networks. In International Conference on Machine Learning, pp. 4914-4923, 2018.
Hao Wang and Dit- Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. IEEE Transactions on Knowledge and Data Engineering, 28(12):3395-3408 2016.
Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. In International Conference Learning Representations, 2019. URL
htttps::/cpenreeilew... net. forum? w
Yu Weng, Tianbao Zhou, Yujie Li, and Xiaoyu Qiu. Nas-unet: Neural architecture search for medical image segmentation. IEEE Access, 7:44247-44257, 2019.
Bichen Wu, Xiaoliang Dai. Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019. URL https:// lopenreview.. net www.tneuut.eeeieeee wtw.:///ooouuiilyyyooooooot
13

---

Under review as a conference paper at ICLR 2020
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. International Conference on Machine Learning, pp. 5847-5856, 2018.
Jingwei Zhuo, Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang. Message passing stein variational gradient descent. In International Conference on Machine Learning, pp. 6013-6022, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:161..0157,, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018.
A DERIVATION OF THE LOG PROBABILITY DENSITY OF THE ADAPTIVE CONCRETE DISTRIBUTION
For clear expression, We simply denote ((3)), ((..), ((-) and el-i)) a, 0, B and respectively. Let softmax (0). Consider
exp((02 + BEx)/T)
exp((logp: + BEx)/T)
0
Σ 1 (xp - BEt)/T) E exp((log + 36t)/T)
Let 2/ log Pk. Sc, log Ps= - :log(( (loo(u))) where un (((0,1)) i.i... has density
PPu 1/8 exp(
2k
)expl -
expl
1/9 exp(
=x)
B
3
We denote c= EK exp(zi/T), then a = exp(2x/ (if/zs/TT/e/ //cc. We consider this transformation:
F(z zk) (a),.... .,, QK-1,
which has the following inverse transformation:
(a) = (rol a + log c)...., T (log ak + log
whose Jacobian has the determinant (refer to the derivation of the concrete distribution Maddison al. 2016)):
K
O2
Multiply this with the density of z, we get the density
1/0 //0 exp(- 7(log +og P.P
exp( p. 1/0 (/ppexp(pplppp 9
7(log +loge,
8
i=
Let log c, then apply the change of variables formula, obtain the density:
1/9
Krr K exp(- (p.a, /a/p:xxxpppp]p 77 ))
P.
exp(
3K
(1+/8))
B
8
1i=
We use 7 substitute log >K k (pp:;TTJJ1
then get:
K I
p
1/8
Krr
exp - exp((exp(y> Tr )).
exp(y
exp(7)8
T (1+7/8)
B
14

---

Under review as a conference paper at ICLR 2020
503
00
414
comples
somples
Figure 6: Test loss (left), test error rate (middle). and test ECE (right) of DBSN vary w.I.t the number of MC samples used in estimating Eq. 8 (CIFAR-10)
13
105
676
698
323
Kumber camples
Bumber comples
Kumber camples
Figure 7: Test loss (left), test error rate (middle), and test ECE (right) of DBSN vary w.r.t the number of MC samples used in estimating Eq. S (CIFAR-100)
Naturally, we can integrate out T, and get:
7 1 K 1/3
P
B
expl7 --))(K)))))
exp(n)
3K 1 K
O
(1+1/3)
1/8
|=1 Pj'
exp( exp(-K>))(()
8K 1==1 ag
(1+r/9)
(KK-
1)1) K-1
(p:o,
BK-
(pia, 7)188(
1= O2
Then, the log density is:
K
log P,
-Tlog a
log P. - T log
log((K- 1}!) - (K - 1)log
log
K * LEE
B
B
B
i=1
0
loog a;
K 0
7 log α
= =log((K -K]K-]]]] 1)!) + (K-1) 1) log
log O4
K I.E.
B
9
which is equal to Eq. (T0).
B THE EFFECTS OF THE NUMBER OF MC SAMPLES IN TEST PHASE
We draw the change of test loss, test error rate and test ECE with respect to the number of MC samples used for testing DBSN in Figure 6 (CIFAR- 10) and Figure (CIFAR- is clear that ensembling the predictions from models with various sampled network structures enhances the final predictive performance and calibration significantly. This is in marked contrast to the situation of classic variational BNNs, where using more MC samples does not necessarily bring improvement over using the most likely sample. As shown in the plots, we would better utilize 20+ MC samples to predict the unseen data, for adequately exploiting the learned structure distribution. Indeed, use 100 MC samples in all the experiments, except the adversarial attack experiments where we use 30 MC samples for attacking and evaluation.
15

---

Under review as a conference paper at ICLR 2020
Table 5: Comparison with competing baselines which deploy uncertainty on weights and adopt
Adam-like VOGN (Khan et al. 2018) method for inference. (CIFAR-10)
Training time (hours)
Test error rate (%)
ECE
DBSN
1.2
9.90
0.0070
BNN-LS (with VOGN)
13.0
28.4
0.5391
Fully Bayesian DBSN (with VOGN)
13.0
30.5
0.5169
C MORE COMPARISONS BETWEEN DBSN AND COMPETING BASELINES WITH WEIGHT UNCERTAINTY
We realized the BBB method used for modeling weight uncertainty in BNN-LS and Fully Bayesian DBSN may be restrictive, resulting in such weakness. Therefore, we further implemented these two baselines with most-recently proposed mean-field natural-gradient variational inference method, called Variational Online Gauss-Newton (VOGN) (Khan et al.) [2088] Osawa et al. 2019). VOGN is known to work well with advanced techniques, c.g., momentum, batch normalisa- tion, data augmentation. As claimed by Osawa et al. (2019), VOGN demonstrates compara- ble results to Adam. Then, we replaced the used BBB Blundell et al.) (015) in BNN-LS and Fully Bayesian DBSN with VOGN, based on VOGN's official repository
/ttt:s://gihhbb With the original network size (B = 7, 12 cells). the baselines trained with VOGN needed more than one hour for one epoch. Thus we adopted smaller networks (B = 4, 3 cells), which have almost 41K parameters, for the two base- lines. We also trained DBSN in the same setting. The detailed parameters initialize VOGN are here wwttth:tt//ccccc bnn bon_torchsso.re py L220). The experiments were conducted on CIFAR-TO and the results are provided in Table 5 The predictive performance and uncertainty gaps between DBSN and the two baselines are very huge, which possibly results from the under-fitting of the high-dim weight distri- butions in BNN-LS and Fully Bayesian DBSN. We believe that our implementation is correct be- cause our results are consistent with the original results Table I of et al.) (2019) (VOGN has 75.48% and 84.27% validation accuracy even with even larger 2.5M AlexNet and 11.IM ResNet- 18 architectures). Further, DBSN is much more efficient than them. These comparisons strongly reveal the benefits of modeling structure uncertainty modeling weight uncertainty, highlighting the practical value of DBSN.
com/tean aporox bayes/ sal-wwwthaassss
thub. com/ anonymousest. /DBSN/D b ob/ master /tatteerrdbb///ee /wt////bs//aarerreeee
D MORE RESULTS FOR CALIBRATION
We plot the reliability diagrams of 4 typical methods, which represent the deep BNN with structure uncertainty, the classic BNN with weight uncertainty, the deterministic NN with MC dropout and the standard NN, respectively, in Figure g Obviously, DBSN has better reliability diagrams than NEK-FAC and Dropout, proving the effectiveness of the uncertainty on network structure.
E ATTACK WITH BIM
We perform an adversarial attack using BIM algorithm. Concretely, we set the number of iteration to be 3 and set the perturbation size in every step to be 1/3 of the whole perturbation size. The experiments mainly focus on the models trained on CIFAR-10. Figure shows the results. Random a, D and DBSN have increasing entropy when the perturbation size changes from 0 to 0.01, but all the other approaches are attacked successfully with entropy dropping. However, strictly, only the Random at perturbation size 0.01 provides useful predictive uncertainty, and we can use the entropy to reject the predictions. Therefore, we have to agree that BIM is powerful enough to break all the methods, including DBSN. So we advise adjusting DBSN accordingly (e.g.. employing adversarial training, using more robust loss) we want to use DBSN to defend the adversarial attacks.
16

---

Under review as a conference paper at ICLR 2020
DBSN
NEK-FAC
Dropout
Fixed
POP 0.0504
FOP 0/560
0/002
PGRE: 0,2022
PGB 0.207
Figure 8: Reliability diagrams for DBSN, NEK-FAC, Dropout and Fixed a on CIFAR-10 (top row) and CIFAR-100 (bottom row). The bars aligning more closely to the diagonal are preferred. Smaller ECE better.
NEK-FAC
Fixed Drop-path-<e
Random
Dropout-
DBSN
1.0
0.8
Accuracy
0.3
Entropy
0.2
0.2
0.0
0.0
0.00
0.02
0.04
0.06
0.08
0.10
Perturbation size
Figure 9: Accuracy (solid) vs entropy (dashed) function of the adversarial perturbation size on CIFAR-10. Attack with BIM.
F VISUALIZATION OF THE LEARNED STRUCTURES
We visualize the learned structure distributions on different tasks in Figure 10, Figure [1 Figure 12
and Figure (we do not draw the zero operation). The structure distributions learned on different tasks look different, validating that DBSN can adapt the network structure according to the data distribution flexibly.
17

---

Under review as a conference paper at ICLR 2020
0.64
15 2
Figure 10: Structure of the cell learned on CIFAR-10. The pen width of an edge implies the sampling probability of its corresponding operation.
6:61
4029 1x1
#g.ossed
S2 3x3
G/e 3x5
Figure 11: Structure of the cell learned CIFAR-100. The pen width of edge implies the sampling probability of its corresponding operation.
18

---

Under review as a conference paper at ICLR 2020
COBY
slip connect
sop com 3x3
cony Sec
Figure 12: Structure of the cell learned on Cam (in the downsampling path). The pen width of an edge implies the sampling probability of its corresponding operation.
skip mimrel
Scp com
Figure 13: Structure of the cell learned on Cam (in the upsampling path). The pen width of an edge implies the sampling probability of its corresponding operation.
19


Reviews:
[{"review_id": "rygXoHanYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to do approximate Bayesian inference in neural networks by treating the neural network structure as a random variable (RV), while inferring the parameters with point estimates. \nWhile performing Bayesian inference for the neural network structure is sensible, I am not convinced by the approach taken in this work. \n\nThe biggest problem is that the model uses a point estimate of the same weights for different, random network structures. \nMajor problems:\n- In the motivation the authors write “DBSN places distributions on the network structure, introducing more global randomness, thus is probable to yield more diverse predictions, and ensembling them brings more calibrated uncertainty”. What is “more global randomness”? This is used multiple times. Does it refer to the hierarchy in the graphical model? Please be precise here and point that out in your model by using an equation or graphical model. Or is it just an intuition? \n- Generally, I would agree that integrating out multiple network structures provides better calibrated uncertainty. However, given that the authors use point estimates for the weights, it is not clear if that is still true, especially since the number of different architectures used in practice is small. \n- What’s more, the approach uses *the same* point estimates for different structures. This leads to a graphical model, where the weights are not conditioned on the architecture/structure. This modeling choice could be a big limitation, because the weights now have to fit multiple different architectures; it may thus defeat the calibration completely. One can easily imagine that only a single random architecture works well with the learned point estimates, thus resulting in an (almost) deterministic model. I assume that this modeling choice was made for practical reasons, but could you expand on its implications / interpretation / limitations? Does the posterior of such a constrained model not quickly converge to an “almost” dirac, effectively just one network structure? \n- Sec. 3.1. presents the above problem resulting from a modeling decision as a “training challenge”. To counter this problem, the authors propose to reduce the variance of the structure distribution. By doing so, the approach becomes even less Bayesian and the predictive uncertainty becomes even less reliable. \n- “We only learn the connections between the B internal nodes, as shown in Appendix D”. All deterministic weights are learned, but the structure only for some parts of the model? If this is the case, then approach becomes again less probabilistic.\n\nRegarding the experiments, the stddevs are calculated from 3(!) independent runs and thus completely misleading (imagine the stddev of the stddev estimate). \n\nIn summary, the model choice of point estimates for the weights, which are not conditioned on the architecture, leads to various problems. The authors have to introduce tricks such as reducing the variance of the random network structures or learning only a part of the whole structure to make the approach converge. The resulting probabilistic model and its predictive uncertainty is questionable. For this reason, this paper should be rejected. \n\nMinor problems\n- Sec. 3.2. “Improvements of the structure learning space”. What is a “structure learning space”? \n- Section 3 introduces the ELBO in Eq. (4) before the complete model is specified. Please specify the whole model first. How do w and alpha depend on each other in your model? \n- The prior for the weights is omitted; at the same time it is mentioned in the experiments (Sec.5.1.) that weight decay is applied. Why not just be explicit about it and say that a Gaussian prior is used?\n- Background Sec. 2.2. is not clear.  what is a cell? some deterministic transformation in general? bunch of neural network layers? What are the operation (last term in Eq. (2)) doing? This is not detailed and abstract to me. Are the alphas probabilities? Is Eq. (2) consequently a mixture model of different architectures? Or is this here just a weighted sum, where the weights take arbitrary values? A small visualization (additionally) might help here, but can probably be rectified by better explanation.\n- Bayesian reasoning on the structure. Inference?\n- Writing that you propose a new “framework” is a bit grandiose for what is actually proposed. There has been previous work in which the architecture is inferred as well and these approaches would certainly be part of the same framework. Please just say model/algorithm/approach, whatever is applicable. \n- new paragraph starting at “To empirically validate” in the intro.\n- Before (4): “Then we rewrite the approximation error”. Eq. (4) is the ELBO, this is not an approximation error. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rygDVQg2FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed deep Bayesian structure networks (DBSN) to model weights, \\alpha, of the redundant operations in cell-based differentiable NAS. The authors claim that DBSN can achieve better performance (accuracy) than the state of the art. \n\nOne of my concerns is the Bayesian formulation introduced in Eq. (4) seems problematic. It is not clear what priors are placed on alpha. In the case of Bayes by BP (BBB), which is cited as Blundell et al. 2015 in the paper, a Gaussian prior (with zero mean) is used. Therefore there is a KL term between the variational distribution q(w) and the prior distribution p(w) to regularize q(w). In DBSN, q(\\alpha) is parameterized by \\theta and \\epsilon, and so is p(\\alpha), meaning that the KL term is effectively zero. This is very different from what is done in BBB.\n\nThe second major concern is on the experiments. (1) The authors use DARTS as a main baseline and show that DBSN significantly outperforms DARTS. However, looking at the DARTS paper, the test error on CIFAR-10 is around 3% for both the first-order and second-order versions. The test error in Table 1 is around 9%, which is a lot lower. I notice that the DARTS paper has a parameter number of 3.3M, while in the current paper it set to 1M. Given that DARTS is the main baseline method and the same dataset (CIFAR-10) is used, it would make much more sense to use exactly the same architecture for comparison. The current results is hardly convincing. (2) Besides, note that in the DARTS paper, DenseNet-BC has test error of 3.46%, much higher than DARTS (~3%). In Table 2 of this paper however, DARTS is significantly worse than DenseNet-BC (8.91% versus 4.51%). These results are highly inconsistent with previous work.\n\nAs mentioned in the paper, Dikov & Bayer 2019 has a very similar idea to perform NAS from a Bayesian perspective. It would be best (and would definitely make the paper stronger) to include some comparison. Even if Dikov & Bayer 2019 is not very scalable, it is at least possible to compare them in smaller network size. Otherwise it is hard to evaluate the contribution of DBSN given this highly similar work.\n\nThe authors mentioned in the introduction that DBSN ‘yields more diverse prediction’ and therefore brings more calibrated uncertainty comparing to ensembling different architectures. This is not verified in the experiment section. Table 3 only reports the ECE for one instance of trained networks. For example, it would be interesting to sample different architecture from the alpha learned in DARTS and DBSN, train several networks, ensemble them, and use the variance of the ensemble to compute ECE. This would verify the claim mentioned above.\n\nDo you retrain the network from scratch after the architecture search (which is done in DARTS) for DARTS and DBSN?\n\nI am not convinced by the claim that BNN usually achieve compromising performance. Essentially, BNN, if trained well, is a generalization of deterministic NN. If very flat priors and highly confident variational distributions are used, BNN essentially reduces to deterministic NN.\n\nMissing references on Bayesian deep learning and BNN:\n\nBayesian Dark Knowledge\nTowards Bayesian Deep Learning: A Survey\nNatural-Parameter Networks: A Class of Probabilistic Neural Networks", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1ebOaz0FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper combines ideas from neural architecture search (NAS) and Bayesian neural networks. Instead of maintaining uncertainty in network weights, the authors propose to retain uncertainty in the network structure. In particular, building on cell-based differentiable NAS, the authors infer a distribution over the gating weights of different cells incident onto a tensor while relying on point estimates for the weights inside each cell. \n\nOverall, I liked the paper and vote for accepting it. The notion of maintaining uncertainty about the network structure is a sensible one, and the paper explores an as yet under-explored area at the intersection of state-of-the-art network architecture search algorithms and Bayesian neural networks. Moreover, this is accompanied by compelling empirics — results demonstrate gains in both predictive performance and calibration across diverse tasks and careful comparisons to sensible baselines are presented to evaluate various aspects of the proposed approach (Table 1). \n\nDetailed Comments:\n+ One issue the experiments fail to adequately disentangle is the effect of weight uncertainty vs structure uncertainty. Are the observed gains in accuracy and calibration simply a product of better structure learning? In particular, I would love to see a baseline where point estimates of \\alpha are learned but posterior distribution over weights is inferred. I realize NEK-FAC was an attempt at providing such a comparison, but since it uses a different structure, it remains unclear whether it’s poor performance stems from the fundamental difficulty of learning posteriors over high dimensional weights or simply a sub-optimal network structure. \n\n+ In a similar spirit, one can imagine a fully Bayesian DBSN where one infers posterior distributions overbite \\alpha and w. Presumably, this would close the OOD entropy gap between random \\alpha and DBSN. \n\n+ How many Monte Carlo samples were used to evaluate Equation 8. In variational BNNs one often finds that using more MC samples doesn’t necessarily improve predictive accuracy over using the most likely sample (the mean if using a Gaussian variational family). It would be interesting to see predictive performance as a function of the number of MC samples for DBSN. \n\n+ Clarity: While I am mostly upbeat about this paper, the writing could be significantly improved. While the overall ideas come across, there are several instances where the text appears muddled and needs a few more polishing passes. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors develop stochastic variational approaches to learn Bayesian "structure distributions" for neural networks. While the reviewers appreciated the updates to the paper made by the authors, there will still a number of remaining concerns. There were particularly concerns about the clarity of the paper (remarking on informality of language and lack of changes in the revision with respect to comments in the original review), and the fairness of comparisons. Regarding comparisons, one reviewer comments: "I do not agree that the comparison with DARTS is fair because the authors remove the options for retraining in both DARTS and DBSN. The reason DARTS trains using one half of the data and validate on the other is that it includes a retraining phase where all data is used. Therefore fair comparison should use the same procedure as DARTS (including a retraining phrase). At the very least, to compare methods without retraining, results of DARTS with more data (e.g., 80%) for training should be reported." The authors are encouraged to continue with this work, carefully accounting for reviewer comments in future revisions.
Decision: Reject