Joint Inference for Neural Network Depth and Dropout Regularization
Authors: Kishan K C
Rui Li
Mahdi Gilany
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Learning
Nonparametric Bayesian Model Selection
Stochastic Variational Inference

Abstract: Dropout regularization methods prune a neural network's pre-determined backbone structure to avoid overfitting. However, a deep model still tends to be poorly calibrated with high confidence on incorrect predictions. We propose a unified Bayesian model selection method to jointly infer the most plausible network depth warranted by data, and perform dropout regularization simultaneously. In particular, to infer network depth we define a beta process over the number of hidden layers which allows it to go to infinity. Layer-wise activation probabilities induced by the beta process modulate neuron activation via binary vectors of a conjugate Bernoulli process. Experiments across domains show that by adapting network depth and dropout regularization to data, our method achieves superior performance comparing to state-of-the-art methods with well-calibrated uncertainty estimates. In continual learning, our method enables neural networks to dynamically evolve their depths to accommodate incrementally available data beyond their initial structures, and alleviate catastrophic forgetting.

Joint Inference for Neural Network Depth and Dropout Regularization
Kishan K C²
Rui Li
Mahdi Gilany² "Qeeens University
*Rochester Institute of Technology (kk3671, rxlics)@rit.. edu
mahdi. www..ggggguuuuesu....m
Abstract
Dropout regularization methods prune a neural network's pre-determined backhone structure to avoid overfitting. However, deep model still tends be poorly calibrated with high confidence on incorrect predictions. We propose a unified Bayesian model selection method jointly infer the most plausible network depth warranted by data, and perform dropout regularization simultaneously. In particular, infer network depth we define beta process over the number of hidden layers which allows it to go to infinity. Layer-wise activation probabilities induced by the beta process modulate neuron activation via binary vectors of conjugate Bernoulli process. Experiments across domains show that by adapting network depth and dropout regularization to data, our method achieves superior performance comparing to state-of-th of-the-art aht methods with well-calibrated uncertainty estimates. In continual learning, our method enables neural networks to dynamically evolve their depths to accommodate incrementally available data beyond their initial structures, and alleviate catastrophic forgetting.
Introduction
Both dropout regularization and network depth are critical to the success of deep neural networks Finely tuned or selected structures empower DNNs with proper model capacity that can not only efficiently capture statistical regularities in data but also avoid being deceived by random noise into "discovering" non-existent relationships.
(DNNs) N 3
Dropout as an effective regularization method sparsifies a neural network's 's structure by randomly deleting neurons along with their connections in training to prevent it from overfitting (5) 6) Recent studies extend the method from a Bayesian perspective to allow automatic tuning of the dropout probability in large models 4 g 10 Τл.. Although dropout and its variants can effectively govern model capacity, without uncertainty calibration deep models tend to be overly confident with their predictions 3 [2). Probabilistic inference approaches for network structure selection propose to bypass or down-weight certain hidden layers to reduce training cost and build smaller models that perform just as well larger ones B
15 These methods lead to more efficient or better-calle cetter-calibratd models only by reducing the depth of a pre-determined network structure. They cannot scale the network structure up accommodate incrementally available data beyond the upper limit of its capacity.
We thus propose novel Bayesian model selection framework to jointly infer network depth and perform dropout regularization simultaneously. In particular, we model the depth of a network structure stochastic process by defining the beta process [16 17] over the number of hidden layers to enable it to go to infinity in theory, as in Figure [(a) (left). The beta process induces layer-wise activation probabilities, which allows its conjugate Bernoulli process generate binary
"Coresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Beta process
goes lufinity
Fixed depth
Fixed deptá
Bernoulli pronoulllprooeee process
+0
Indian buf fet Process
Fired
(b)
Figure (a) Demonstrations of our proposed framework (left), dropout variant (middle), and typical structure selection methods (right). Our framework enables depth goes to infinity by modeling the number of hidden layers with a beta process. The dropout variant defines Indian buffet process per layer to infer width, but depth is fixed [I]
20 Structure selection methods can only reduce beta processes with > B on the left and /
depth of a pre-determined network structure 2 13 [5]. (b) On top, random draws from two
on the right over hidden layer function space H = (h; 1 - x.). atom location oh, indexing a hidden layer function h,, and the height denotes its activation probability n, For both cases, the conjugate Bernoulli processes at bottom are obtained by random filled dots Emd = 1 with probability n and empty dots 2ml with probability
1 So each column is binary vector zι corresponding to layer l.
vector per hidden layer to prune neurons for regularization, as in Figure [(b) The probabilistic inference framework automatically balances network depth and dropout regularization by computing a marginal likelihood over the hidden layers and their neuron activations, and provides well-calibrated uncertainty estimates for predictions. The exact computation of the marginal likelihood is intractable due to the nonlinear nature of neural networks with potentially infinite number of hidden layers. We thus employ the structured stochastic variational inference [18] 19] with continuous relaxation on the Bernoulli variables to efficiently approximate the integral with a continuum of lower bounds. The relaxation allows to maintain differentiability and mitigate overfitting via model averaging of the sampled hidden layers and neuron activations. Benefiting from theoretical analysis, we readily integrate the joint inference with the parameter learning process through alternating approach to efficiently update the parameters, and perform hidden layer sampling and dropout regularization.
We analyze the behavior of our joint inference framework over multilayer perceptrons (MLPs) and convolutional neural networks (CNNs), and evaluate their performance across domains. The experiments show that method leads to a compact neural network by balancing its depth and dropout regularization with uncertainty calibration, and achieves superior performance comparing to state-of-thee art dropout and structure selection methods. We also demonstrate our method on a continual learning task. By enabling both network depth and neuron activations to dynamically evolve to accommodate incrementally available data, can alleviate catastrophic forgetting.
2 Related work
Dropout is an effective neural network regularization technique [1 By randomly pruning neurons
and their connections from network structure with dropout rate during training, can prevent
DNNs from overfitting [22 23 24 7 interprets dropout from Bayesian perspective quantify uncertainty. Variational dropout proposes stochastic gradient variational inference approach to learn the dropout rate from data with constraint on large dropout rate values (8]. [9] extends the variational dropout to set unbounded individual dropout rate per weight. Concrete dropout proposes a continuous relaxation of the dropout's 's discrete masks automatically tune the dropout probability, and obtain its uncertainty estimates [0]] Bayesian nonparametrics are also applied to extend the
variational dropout approaches by defining Indian buffet process (IBP) over neurons per hidden layer [1] or over channels in CNNs (20 to infer network widths, as in Figure ((a) (middle). Another body of work leverage the non non-parametric Bayesian framework to infer the dimensionality of latent representation encoded by DNNs [5] (6].
Structure selection methods improve training efficiency by reducing the depth of a pre-arranged network structure, as Figure I(a) (right). Some lead well- perfoperforme smaller networks, and others achieve better uncertainty calibration. In order to build smaller models that perform just as

---

well, 12 proposes to reduce a large neural network by merging adjacent hidden layers with only linear relationships being activated between. A stochastic depth method randomly drops a subset of hidden layers from a large network by bypassing them with an identity function in each mini-batch to speed up the training session, but it still deploys the large network structure at test time [3]. [S]] shows that the multiplicative noise from dropout induces structured shrinkage priors over neural network's weights, and they further extend the shrinkage framework based on ResNet structures [27] to model the probabilities of hidden layers being used. In particular. the work also indicates that the framework is equivalent to the stochastic depth regularization [3.. 14] proposes to tune hidden layer's influence connections and skip connections with a variational Bayes method. The method also prunes neurons separately. [3] proposes a Bayesian model averaging method to down-weight deeper layers by directly connecting every hidden layer to the output layer, and achieve competitive performances with uncertainty calibration. (28) achieves model effectiveness and computational efficiency by inferring distribution of connections and units in the context of local winner-takes-all DNNs with IBP.
prediction by learning bypass variable between adjacent-
3
The joint inference framework
We propose to model network depth as a stochastic process. and jointly perform dropout regularization upon the inferred hidden layers. Theoretical analysis shows that by approximating its marginal likelihood the inference framework achieves an optimal balance between network depth and neuron activations as Bayesian model selection.
3.1 Network structure with infinite hidden layers
Let h denote the 1-th hidden layer composed of neurons (i.e., non-linear activation functions) (.)- The neural network has the form:
h a (, 1) X + h E (1,......
(1)
where W, E RMM is the layer I's weight matrix with Gaussian prior p( () (WW 0, s T), X denotes element-wise multiplication of two vectors, and M is the maximum number of neurons in layer. For simplicity, we set M to be the same for all hidden layers. In particular, layer's outputs by multiplying them elementwisely by a binary vector ži where its element variable 2md E 10. Each random variable 2md takes the value 1 with कं E [0. 1] indicating activation probability of the /-th layer, as in Figure [(b) The combination of the previous layer's outputs with the current layer's via skip connections not only avoids vanishing gradient but also propagates the output of the last hidden layer with activated neurons directly to the output layer f(-)
prune the I-th
Given a dataset D = {(x.. vn) J with N input-output pairs (X., Yn) as training examples, for regression task we express the likelihood of the neural network as:
N DDDZ W) = II WY (x,,, Z, W),
where Z a binary matrix whose I-th column 2,, and W = -[ denotes the set of the weight matrices. s² denotes likelihood noise with I as identity matrix with the same dimensionality as the training examples' outputs. For classification tasks, we replace f(-) with a softmax function and the normal distribution with multinoulli distribution.
3.2
Beta process over layer number
We treat the number of hidden layer functions h, as a stochastic process by defining a beta process over their space H (hr as shown in Figure ((b)
Conceptually, we generate beta process B from its base measure Bo B BP BB) by drawing a set of samples (h,, m)) E x [0 from a non-homogeneous Poisson process [16] Let B Ex mid where ôb, unit point mass at h, In our setting, pair (h,, ग) corresponds to a hidden layer function hy E H and its activation probability ग. E [0, 1]. We define a conjugate
A graphical demonstration of the network structure is in Appendix.
3

---

Bernoulli process Zm ~ (B as Zm = Emtób at the same locations ón, as B where 2ml are independent Bernoulli variables with probability that 2mi equals Ty. As I 2mm 1 activates the m 'th neuron in layer 7.
Computationally, we employ the stick-breaking construction of beta-Bernoulli processes 29 17
2md Ber(m)))
II Pj in ~ Beta(a,
where by are sequentially drawn from beta distribution. According to the formulation, is decreasing with l. a and are the hyperparameters governing preference over a balance between network depth and dropout regularization. In particular, if a > B > the hidden layers tend have lower activation probabilities with less number of activated neurons, as in Figure l(b)left. The setting thus prefers narrower but deeper network structure. On the other hand, 3 >> but wider network structure, where fewer hidden layers tend to have higher activation probabilities with more neurons being activated, in Figure I(b) right.
favors shallower
We thus define a prior over Z via the beta process as
M
= p(vla, ()p(Z))
Beta( 8) II
Ber( (er(z
3.3 Marginal likelihood over network structures
By combining the beta process prior over hidden layers in 4 with the neural network structure in
we can infer the number of hidden layers by integrating over {zilt and {pi)
p(D W, L, 3) = p(DIZ, W)p(Z, v/a, 3)dZdv
(5)
The exact computation of this marginal likelihood is intractable due to the non-linearity of the neural network and + 00.
3.4 Efficient inference with SSVI
We employ structured stochastic variational inference (SSVI) to approximate the marginal likelihood with variational lower bound (ELBO) retaining the dependence between network structures and hyperparameters [18 19 30]. In particular, we define the variational distribution as
K
M
(ZZ v/x K (bx (bx) = ((v)
II
Beta( lap, bx) II m=
ConBer(Zma//rA)
(6)
where Ty 1T vj and (ak, by h variational parameters of q(v), and K denotes truncation level, which can be relaxed as in [31. We relax the constraint of the discrete variables with continuous ones by reparameterizing the Bernoulli distribution into a concrete Bernoulli distribution [32 33];
ConBer(Zmal Zmnk (T+) TT --- xx) 2mk)
(TT() (1-mm1{ x1) Zmk) -=
where r a temperature controlling the distribution smoothness. We generate random samples from the distribution by first sampling from logistic distribution as the external source of randomness and then putting the samples through a logistic function as follow:
Zyn
+ exp(-7-1 गे log(1 nạ) +e))
~ Logistic(O,, 1)
This reparameterization of the Bernoulli distribution allows backpropagate the gradients of the lower bound with respect to the parameters while sampling from it.
We derive lower bound of the log marginal likelihood in (5) as:
L. 8) 2 ( log (DIZ, W)dZdv + (ZZ, (ZZ.v) (Z, v)
dZdd
(9)

---

Ntrain:
Ntrain: 400
Ntrain: 1600
Nimin
kinin:
RMSE
71k
17p
TTA
0
Diopoon (b:2)
+-hoco (349
8-0
Dropoes 13:4)
- (2:)
(b)
3.0
Uncertainty (std)
Eplstemic
MW
MV
0.0
Aleatoric
Predictive
0-
0.4
0.2
-2
101
107
10'
10*
X
Number points
(a)
(c)
Figure 2: (a) Top row shows the activation prohabilities Ty. (black bars) of the inferred hidden layers, and the neuron activations Z (filled dots denote activated neurons). The network becomes deeper with more activated neurons as the training dataset size increases. Bottom row shows the predictive distributions overlaying the data points, and the green bands are łone standard deviation over the predictions of 5 sampled network structures. (b) Predictive performance evaluation of our method and vanilla dropout with different dropout rates for the three cases. (c) Our method's 's estimates of different uncertainties as the number of data points increases.
The ELBO from the right-hand side of (9) becomes our objective function:
C(Z, v, W) Eg(,,M) log pl D Z, w) [L[g(Z)v) [[]] - KL(g(V)llP(v))
(10)
The first term of the ELBO in 10) is the averaged log- lige-iikeihood.. The second and the third terms are the KI. divergence between the variational distribution and the true prior, respectively. particular, for the second term we relax the prior p(Z(v) with concrete Bernoulli as in (7) with different temperatures. For the third term the KI. divergence between the variational distribution and the prior distribution over is:
(a, B(ax, bx)
(L.(g())]p((1) Σin
(ax ax) ()(a))
(11)
(bx B) (bbb)))
(a ax + B bu) vbaa + bR)
where (-) denotes di-gamma function.
Theorem Assume W is the optimized value of the variable, and the likelihood and the induced prior on for Z are specified in (2) and (4), in the large sample limit, the limiting of the ELBO in (10) becomes equivalent to the Bayesian information criterion (34), such that BIC(Z) = log W)-
Z
log 2 where Z| denotes the number of activated neurons in the network structure.
The proof is in Appendix. This theorem indicates that optimizing the limiting case our SSVI framework therefore be seen optimizing the popular model selection criteria. We adopt the implicit differentiation of beta distribution for backpropagation [35], which allows us to stochastically optimize L with respect the variational parameters every time sample network structure. We iteratively maximize ELBO with respect to the variational parameters and weights W, and approximate the ELBO using random samples from the variational distribution.
We approximate the posterior predictive distribution by sampling from the variational posterior distribution for predictions. Thus, the predictive distribution for new data x" is
p(y" |x', W, (āx), (b+)) )) = p(y"|x", W)) v(aa (b)))
(12)
where W is the MAP estimate of the neural network weights based mini- batch gradient descent.
5

---

rank
boston
concrete
energy
kin8nm
naval
power
protein
wine
yacht
-10
100
-
1.00
16
0.25
RMSE
0009
0.00
0065
0000
039
Ours
Drapout
SDapth
DUN
CD
IBPD
BAL
DeapEns
Figure 3: Mean values with standard deviation for test log-likelihood (LL) and RMSE UCI standard splits. Average ranks are computed across datasets. For LL, higher is better. For rank and RMSE, lower is better. The metries are defined as in [3] with the codes generating the plots. Details are in Appendix.
4 Experiments
We analyze the behavior of our framework by applying it to MLPs and CNNs on a variety tasks. We study how it enables network depth to adapt to dataset sizes, and capture different types of uncertainty on synthetic dataset. Our framework outperforms state-of-the-art methods on the UCI datasets with uncertainty estimation. To obtain performance comparison, Optimisation and Hyperband (BOHB) to determine the best configurations for each method [36]. We also investigate the effect of truncation level K and M, and show that their settings have influence on the performance as long as they are reasonably large. The experiments with image datasets show that our framework improves CNN performances with more compact network structures and well- calibrated predictive distributions. Lastly, we demonstrate our method in two continual learning tasks, and show that by allowing network depth dynamically augment to accommodate incrementally available information we can effectively alleviate catastrophic forgetting.
perform Bayesian
4.1 Synthetic experiments
We analyze method's behaviors given different dataset sizes, and its uncertainty estimation. We incrementally generate 20, 500 and 2000 data points from periodic function [14]
y sin (6x) + 0.4z 0. cos (9 (xx(x))) +
(13)
where ~ (O, 0.12) with 20% for validation. The data are shown in Figure 2(a) bottom row. We set the maximum number of neurons per layer M 20, and use leaky ReLU activation functions the input and hidden layers with batch normalization to retain stability. We simulate 5 samples per mini- batch to approximate the ELBO in (IO), and evaluate convergence by computing the cross-correlations of the sample values over 3000 epochs.
As shown in Figure 2(a) to accommodate more information from larger datasets, the neural network becomes deeper with more activated neurons. Although we set the truncation level K = 100 for all the training cases, only the first few hidden layers have activated neurons (i.e., layer, 3 layers, and 4 layers for the respective cases). Figure 2(a) bottom row shows how the predictive distributions in ( are capturing the function and estimating uncertainty the dataset size increases. We compare our method's performance with vanilla dropout [6] whose backbone structure sizes are the same as the respective ones inferred by our method. The predictive performances on 700 held-out data points in Figure 2(b) indicate that by obtaining the critical balance between network depth and neuron activations our method achieves more stable performance with less variance on the small dataset, and significantly better performance on the larger ones.
Implementation details of all the experiments in Appendix.

---

MNIST
FashionMNIST
SVHN
CIFAR10
Test Error rate 96
0.9
10
22
0.8
20
9-
18
0.7
16
0.6
0-
0.5
12
10
10.15
30
50
15 30 35
50
5
20 35
20
40
K/L
KIL
KiL
K/L
SDepth
DUN
Figure 4: Analysis of the influence of our method's truncation level K and other methods' backbone- structure depth L in CNNs with the four image datasets. As long as K is reasonably large (2 10), it no longer has an influence on our method's 's performances. On the contrary, the performance of the other methods depends on L.. As I. becomes large, they tend have an overfitting problem with increased test errors. By jointly inferring network depth and neuron activations, our method is robust to overfitting, and achieve the best performances.
The datasets generated with known variance allow us to assess the different types of uncertainty of our method's s predictions. particular, the epistemic uncertainty obtained by drawing multiple samples of the inferred depth and neuron activation, and evaluating them on the test set of 700 data points from the same data distribution. Figure showss that the epistemic uncertainty decreases as the amount of data points increases. The aleatoric uncertainty approaching the true uncertainty (0.1) shows an increasingly improved estimate as more data is given. Our method's predictive uncertainty obtained by combining epistemic and aleatoric uncertainties converges constant value.
4.2 Performance comparisons on UCI datasets
We evaluate all methods on UCI datasets 37 using standard splits (38 and report their performance in terms of log-likelihood (LL) and root mean square error (RMSE). In particular, we compare the performances with vanilla dropout (Dropout), concrete dropout (CD) [Q), Indian buffet process dropout (IBPD) [I]. Bayesian architecture learning (BAL) [14, stochastic depth (SDepth) [13), depth uncertainty networks (DUNs) [3], and Deep Ensembles (DeepEns) [39]. We perform hyper- parameter optimization with Bayesian Optimisation and Hyperband (BOHB) determine the best configurations including backbone-struccte depths and hyperparameter settings for each method 36), suggested in (.. The detailed implementation is in Appendix.
In Figure 3 we rank the methods from to 8 based on their mean performance across each dataset and metric, and report mean ranks with + one standard deviation. Our joint inference framework outperforms other methods terms of both test log-likellng log-likelihood (LL.) and RMSE by achieving the best rank across all datasets. In particular, LI measures both accuracy and uncertainty calibration [40]. The narrower standard deviations of our method indicate robustness. The vanilla dropout achieves the second-bee -eet rank followed closely by the Deep Ensembles in terms of LL and concrete dropout in terms of RMSE.
4.3 Effect of truncation level
We further investigate the effect of truncation level K on our method's performance, and compare it with the vackbone-structur depth L dropout variants and structure selection methods.
All the methods have the same maximum width M = 64 (i.c., the maximum number of feature maps convolutional layers). In Figure we apply all the methods on CNNs with the backbone-structure
depth L and the truncation level K over the range L K E (5. 10, 15, 20, 25, 30. 35, 40, 45, 50} for classification tasks of four image datasets: MNIST 41 FashionMNIST 42). SVHN (43, and CIFAR10 [4]. When L = K = 5, all the methods have an underfitting problem due to the limited model capacity, although our method still achieves competitive or better performances. As truncation level becomes reasonably large (2 10), it no longer affects our method's ''s performance. The results are consistent with Theorem 1, as our joint inference essentially works as Bayesian model selection network structure truncation. On the contrary, the depths of backbone structure L
7

---

Steepth
Dewptri
Dropott
- DUN
50gthh
CD
Docatrn
Dropcut
3.0
-05
0.6
-1.
ECE
0.15
0.0
-10
0.55
-12
60
120
150
180
0.0
60
90 Rotation
120
150
180
-25
0.00
2
Rotation
Corruption
Corruption
(a) Rotated MNIST
(b) Corrupted CIFARIO
Figure 5: Evaluation of uncertainty estimates for (a) MNIST for varying degrees rotation and (b) CIFARI0 at varying degrees of corruption severities with log- ligeliiei (LL) and expected calibration emror (ECE), as in
affect the other methods to great extent. The increase of the test errors shows that they all have an overfitting problem when the backbone-structure depths become large. Individual method reaches its own best performance, as L increases. Our method outperforms them for all these cases with orders of magnitude less computing resources across the four datasets. The respective evaluations in Figure T (i.e., .M = 64) show that our method only activates neurons in 6, 6, 10 and 6 hidden layers for predictions on MNIST, MNSHI SVHN, and CIFARIO, respectively. Only 10% of total neurons in the truncation level are activated for all the cases. This suggests by jointly inferring network depth and neuron activations our method is quite robust overfitting, and the balance between the two also leads to superior performances with compact network structures. In contrast, the backbone-struutuu sizes of the dropout variants and structure selection methods have to set carefully to determine the best configuration for given dataset.
We also evaluate the uncertainty calibration of our framework and other methods with their best settings in Figure 4 In Figure 5(a) we train all methods on MNIST and evaluate their predictive distributions on increasingly rotated digits 45). The uncertainty calibration of our method is more robust to the data with large angle rotations than other methods, and it achieves the performance with the lowest expected calibration error (ECE) [46]. In Figure 5(b) to make a fair and rigorous comparison as in 3, we also evaluate the corruption robustness of all methods on the CIFARI 10 dataset with 16 types of algorithmically generated corruptions [47. Each type of corruption has 5 levels of severity. The Deep Ensembles outperform other methods, and our model achieves
comparable performance all corruption levels.
MNIST
FashionMNIST
SVHN
CIFAR10
Activated (
14
24
100
14
12
Activated Neuron(%)
12
Depth
10
10
15
60
12
20
16
32
64
128
192 256 320
16 32 64 128 1922553322 320
16 32
64
128
192
256
320
16
32
64
128
192 320
M
M
M
M
Figure 6: Influence of the maximum number of feature maps (M) on our method with four image datasets. When M is small, we tend to have deeper network structures (blue). As M becomes reasonably large (e.g., 128), it tends not to have influence the inference of network structure sizes. Meanwhile, the percentages of activated neurons in the truncation level are stable (orange).
4.4 Effect of M
We next assess the influence the maximum number of neurons/feature maps (M) on our method with classification tasks on the four image datasets. Figure 6 shows the evolution of the number of inferred convolutional layers as M increases. When M 16, we tend to have deeper network structures to compensate for the relatively narrow layers. As M increases, the structures become

---

Rotated MNIST
Permuted MNIST
Rotated MNIST
Permuted MNIST
100
Activation prob. (mL)
1.0
90
Layer
Accuracy
80
0.6
-
Layer
Layer
70
0.4
60
SGD Decpout
0.2
50
Our
0.0
1
2
3 Tasks
Tasks
Tasks
Tasks
(a)
(b)
Figure 7: (a) The validation accuracy averaged over five runs for each task after training on all tasks in sequence. (b) Evolution of layer-activation probabilities (Ttz) with the five tasks.
shallower. When M reasonably large, it has no influence on the inferred structure depth. The percentage of activated neurons in the truncation level remains relatively stable across different values of M. This suggests that our method can adapt both the network depth and the neuron activations to maintain the best performances as M changes.
4.5 Continual learning
Continual learning is an important application since real-world tasks are dynamic and non-stann non-stationary. Machine learning models need to learn consecutive tasks without forgetting how perform previously trained ones. Assuming the pre-determined backbone structures are sufficient to accommodate all information from the continual tasks, network regularization approaches alleviate catastrophic forgetting by regularizing the updates of neural weights [48]. Dropout and its variants are applied to learn an implicit gating mechanism that activates different gates for different tasks 49 [50. However, the rigid backbone structure constrains the applicability of these methods in real-world settings.
We slightly modify method to enable network depth and neuron activations to dynamically evolve to accommodate incrementally available data. Given a set of sequentially arriving datasets {D,} where each may contain a single datum, update our ELBO in (10) as:
C. = E (2,,) log W)] (Z))) - (L(a())lla-10))
(14)
by replacing the priors p( vv) and p(v) with their variational approximation for the previous dataset 4:-1 (Z) 1z) and gt-1 (v). We initialize them as go (Zlv) p(Z and qo (v() = p(v). The variational
distribution in (6) after seeing the t-th dataset is recursively updated by taking the distribution after seeing the (t T) dataset, multiplying by the likelihood and re-normalizing.
We compare the performance of our method with baseline continual learning methods including elastic weight consolidation (EWC) (48), orthogonal gradient descent (OGD) [51]. and stochastic gradient descent with dropout (SGD dropout). We conduct extensive grid search on the hyperparameter setting of these methods, and evaluate them on two popular continual learning benchmarks: permuted MNIST 48) and rotated MNIST datasets 12 Permuted MNIST at each time step D, consists of labeled MNIST images whose pixels undergo a fixed random permutation. Rotated MNIST is generated by the rotation the original MNIST images. Figure ( shows our method outperforms the other methods on the first four tasks, and is only slightly worse than some (i.e., SGD for Permuted MNIST). Over the five tasks, our method achieves the highest average accuracies (95.16% on Rotated MNIST and 92.34% on permuted MNIST). Although some methods perform well on the fifth task after being trained on it, they fail to preserve the knowledge learned in previous tasks. contrast, our method can preserve the knowledge from old tasks and achieves comparable or better accuracy on new tasks. Figure ((b) shows the evolution network structures for the five sequential tasks, as our method activates more neurons in the inferred hidden layers. In particular, the first hidden layer tends be full from the first task with its activation probability close to one, the activation probabilities of the second and the third layers go up as the tasks are coming.

---

5
Conclusions
The proposed general joint inference framework can be applied to various neural networks. The experiments on MLPs and CNNs show that our method balances network depth and neuron activations achieve superior performance. By enabling neural network structures dynamically evolve to accommodate incrementally available data, we effectively alleviate catastrophic forgetting.
6 Funding
This material is based upon work supported by the National Science Foundation under Award No. 2045804 and Award No. 1850492.
References
[1] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), pages 5580 5590, 2017.
[2] Suraj Srinivas and Venkatesh Babu. Learning neural network architectures using backpropagation. In Proc. of the British Machine Vision Conference (BMVC), pages 1-10..1,,, 2016.
[3] Javier Antorán, James Allingham, and José Miguel Hernández- Lobato. Depth uncertainty in neural networks. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
[4] Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), page 2290-3300, 2017.
[5] Sida Wang and Christopher Manning. Fast dropout training. In Pruc. of the 30th International Conference on Machine Learning (ICML), pages 118-126, 2013.
[6] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, ((1)::199-1158 2014.
[7] Yarin Gal Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proc. of the 33rd International Conference on Machine Learning (ICMI.), pages 1050-1059, 2016.
[8] Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and local reparameterization trick. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), page 2015.
[9] Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proc. of the 34th International Conference on Machine Learning (ICML), pages 2498 2507, 2017.
[10] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. Proc. of the Advances in Neural Information Processing Systems (NeurlPS), pages 3581 3581-3590, 2017.
[11] Juho Lee, Sachoon Kim, Jachong Yoon, Hae Boom Lee, Eunho Yang, and Sung Ju Hwang. Adaptive network sparsification with dependent variational beta-bernoulli dropout. arxiv preprint arviv / 1805. 10896, 2019.
[12] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proc. of the IEFE Conference on Computer Vision and Pottern Recognition (CVPR), pages 427-436, 2015.
[13] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Proc. of the 14th European Conference on Computer Vision (ICCV), pages 646-66 661, 2016.
[14] Georgi Dikov and Justin Bayer. Bayesian learning of neural network architectures. In Proc. the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 730-738, 2019.
[15] Eric Nalisnick, Jose Miguel Heraanez--Lobale and Padhraic Smyth. Dropout structured shrinkage prior. In Proc. of the 36th International Conference on Machine Learning (ICML), pages 47 4712-4722, 2019.
[16] Romain Thibaux and Michael 1 Jordan. Hierarchical beta processes and the indian buffet process. In Proc. of the Artificial Intelligence and Statistics (AISTATS), pages 564-571, 2007.
10

---

[17] Tamara Broderick, Michael I. Jordan, and Jim Pitman. Beta processes, stick-breaking and power laws. Bayesian Analysis, 7(2):439-476, Jun 2012.
[18] Matthew Hoffiman and David Blei. Stochastic structured variational inference. In Proc. of the Artificial Intelligence and Statistics (AISTATS), pages 361-369, 2015.
[19] Rachit Singh, Jeftrey Ling, and Finale Doshi-Velez. Structured variational autoencoders for the beta- beroouli process, In N/PS Workshop on Advances in Approximate Bayesian Inference. 2017.
[20] Jiashi Feng and Trevor Darrell. Learning the structure deep convolutional networks. Proc. of the IEEE International Conference on Computer Visions (ICCV), page 2749-2757, 2015.
[21] Suraj Srinivas and R Venkatesh Babu. Generalized dropout. arXiv preprint arXiv: 1.t.:1111..0911, 2016.
[22] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proc. of the 30th International Conference on Machine Learning (ICML), pages 1058-10666 2013.
[23] Song Han, Jeff Pool, Jeff Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), page 1135-1143, 2015.
[24] John Ingraham and Debora Marks. Variational inference for sparse and undirected models. In Proc. of the 34th International Conference on Machine Learning (ICML), pages 1607-1616, 2017.
[25] Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Erie P. Xing. Nonparametric variational auto-encoders for hicrarchical representation learning. In Proc. of the International Conference on Computer Vision (ICCV), pages 5102,, 5102,2017.
[26] Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autnencoders. In Proc. of the International Conference on Learning Representations (ICLR), 2017.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[28] Konstantinos Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric bayesian deep networks with local competition. In Proc. of the 36th International Conference on Machine Learning (ICML), pages 4980 4988, 2019.
[29] John Paisley, Aimee Zaas, Christopher W. Woods, Geoffrey S. Ginsburg, and Lawrence Carin. A stick- breaking construction of the beta process. In Proc. of the 27th International Conference Machine Learning (ICML), pages 2902-29029 2911. JMLR. org, 2010.
[30] Matthew Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):333-1137, 2013.
[31] Kai Xu, Akash Srivastava, and Charles Sutton. Variational russian roulette for deep bayesian nonpara- metrics. In Proc. of the International Conference on Machine Learning (ICMI.,, pages 6963-6972, 2019.
[32] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proc. the International Conference on Learning Representations (ICLR), 2017.
[33] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. Proc. of the International Conference on Learning Representationns (ICI.), 2017.
[34] Hagai Attias. A variational bayesian framework for graphical models. In Proc. of the Advances Neural Information Processing Systems (NeurIPS), page 209-215, 1999.
[35] Martin Jankowiak and Fritz Obermeyer. Pathwise derivatives beyond the reparamcterization trick. In Proc. of the International Conference on Machine Learning (ICML), pages 2235 233--2444. 2018.
[36] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In Proc. the 35th International Conference on Machine Learning (ICML), page 4436-1445 2018.
[37] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. 2017.

---

[38] José Miguel Loraat and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In Proc. of the International Conference on Machine Learning, pages 1861-1869. PMLR, 2015.
[39] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Proc. of the Advances Neural Information Processing Systems, 30, 2017.
[40] Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359 378, 2007.
[41] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(1 ())2278 2324, 1998.
[42] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for henchmarking machine learning algorithms. arXiv preprint 77Xiv:/70..0777,, 2017.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[44] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[45] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Proc. of the Advances in Neural Information Processing Systems, volume 32, 2019.
[46] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht, Obtaining well calibrated probabilitics using bayesian binning. In Proc. of the AAAI Conference on Artificial Intelligence, 2015.
[47] Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proc. of the International Conference on Learning Representations (ICLR), 2019.
[48] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barver Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proc. of the national academy of sciences, 11((13)3521-3526 2017.
[49] Seyed Iman Mirzadeh, Mehrdad Farajtabar, and Hassan Ghasemzadeh. Dropout as an implicit gating mechanism for continual learning. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 232-233, 2020.
[50] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
[51] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Proc. of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 3762-3773. PMLR, 2020.
[52] Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical eval- uation of deep architectures on problems with many factors of variation. In Proc. of the 24th International Conference on Machine learning (ICML), pages 473-480, 2007.
Checklist
1. For all authors...
(a) Do main claims made the abstract and introduction accurately reflect the paper's 's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Line 135 and Line 162
(c) Did you discuss any potential negative societal impacts your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
12

---

3. If you experiments...
(a) Did you include code, data, and instructions needed to reproduce main experimental results (either in the supplemental material or as URL)? [Yes] See Appendix
(b) Did you specify the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
If you are using existing assets (e-g.. code, data, models) or curatingfreleasig new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See
Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you'le 'r us- ing/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? [N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "pkqAr4d7gP7", "reviewer": "Reviewer_tkqv", "summary": "The paper proposes a comprehensive but effective method to simultaneously perform inference for the NN depth and width. As the idea might not be new, the execution of the method is solid. The methodology is also linked to Bayesian information criterion with a theorem.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe structure selection has been studies by a few related works. This work models the network depth with Beta process while the Bernoulli variables indicate the if a node is kept or dropped. With this modeling, it is easy to control the network shape with the distribution parameter - alpha and beta, which is smart. It is interesting to see behaviors of network shape under different (alpha, beta) setups. The inference is done with standard stochastic variational inference.\n\nQuality, clarity, and significance:\n1. The paper is well-written and easy to read. The graphs are clear and intuitive.\n2. The major doubt comes from the experiment section. \n\n-As the network architecture (depth and width) is considered, why there's no results on efficiency reported? \n\n-The test error rate on Cifar10 is over 10%, which is not a competitive results. With standard VGG network, it could easily achieve less than 10%. What is the network used for Cifar10? \n\n-Only small scale datasets are used. Does the method scale to large datasets? For example, TinyImagenet might be a proper choice for Bayesian deep learning.\n\n-How is the performance of detecting out-of-domain data?\n\n-How is the performance compared with deep ensembles? Or other Bayesian method related to sparsity and model selection? For example: \n\nLouizos et al, Learning sparse neural networks through l 0 regularization, ICML 2018.\nDai et al., Compressing Neural Networks using the Variational Information Bottleneck, ICML 2018.\nCui et al, Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression, CVPR 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "_Szvh0TWDV6", "reviewer": "Reviewer_FqX9", "summary": "The paper proposes a new method for training deep neural networks, where the parameters controlling model depth and activation 'drop' probabilities are being trained and updated, as well as the model's parameters. After developing a theoretical framework where the model structure is treated as a stochastic process, they run a series of experiments to evaluate it over a series of tasks, and present promising results regarding their model's ability to estimate uncertainty.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \n\nThe method presented in the paper is new. Several similar methods are discussed under 'prior work', and the differences are highlighted.\n\n**Quality:**\n\nThe submission appears to be technically sound. \n\nIn section 3, some of the decisions made felt arbitrary and I would appreciate more care being given for their advantages and disadvantages. (See limitation: 1)\n\nSeveral things struck me as odd with the method:\n\n1. The dropout variant used in the paper doesn't include the 1/(1-p) factor which we know to be vital for standard dropout. Dropout is still used during inference so it may be less important, but it seems to me that the factor is still necessary for training since the value of the drop-rate is inconsistent.\n\n2. The paper used batch-norm liberally for experiments (4), but its effect wasn't addressed in section 3 and I worry it could be substantial. Regularization effect aside, it can also affect the training's dynamics, especially if the dropout scalar factor is missing.\n\n3. In Algorithm 1, $S$ samples of networks (Depth + Dropmask) are being drawn, and their results are averaged to estimate log-likelihood (for the ELBO). While it's not completely the same, it has similarities with $\\text{Dropout}_K$, an experimental regularization that was explored in this paper: [2]. Paper [2], however, showed that $\\text{Dropout}_K$ is suboptimal, since by averaging over different dropout masks you lost the *implicit* regularization of dropout. I worry that this may happen in this paper as well (this can explain why the chosen value for $S$ is so low). The good news is that even if this is a problem, this could possibly be fixed with the regularization term they propose.\n\nThe experimental section is well organized, and supported by complementary details in the supplementary. The authors claim superior performance when comparing their method to state-of-the-art methods albeit I was not convinced that the baselines were good enough for this assessment. (see: A Survey of Uncertainty in Deep Neural Networks[3], Table IV).\n\nOne more thing I felt was missing is an illustration of the dynamics of $a_k, b_k$ during training. As is, I am not confident enough that the motivations presented for designing the new method align with what it does in practice.\n\nThe submission included code, which allowed easy recreation of the synthetic experiment. I ran few sanity checks (Like detaching loss components and monitoring network depth)-- they all checked out, but they were somewhat limited by the easy benchmark.\n\n**Clarity:**\n\nThe paper is well written.\n\nBoth the introduction and related work sections detail the relevant work from the machine learning aspect. The relevant background regarding the mathematical aspects (like [1]), however, is only briefly mentioned. This makes the later understanding of the introduced method harder for the readers who are not familiar with this works.\n\nAlgorithm 1 (in the Appendix) (\"Training of our proposed network\") was very helpful for understanding section 3 from a birds' eye view, consider incorporating it into the main text.\n\n**Significance:**\n\nThe submission addresses difficult tasks (uncertainty estimation, continual learning), with a new approach. While I was not convinced of the practicality of the approach as is, others are likely to use/ build upon this work.\n\n[1] https://arxiv.org/abs/1106.0539\n[2] https://arxiv.org/abs/2002.12915\n[3] https://arxiv.org/abs/2107.03342", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. [Request] Section 3.2: Please provide more early intuition for the usage of Beta Process.\n\n2. [Request] The number of samples ($S=5$) used at Algorithm 1 seems arbitrary. Furthermore, as a key component in the method, I would be interested to see how it was chosen. Can you run an ablation study to show its effect? I would be interested to learn how the dynamics of $a_k, b_k$ change in accordance to $S$.\n\n3. [Question] Please address the points I have made about the dropout scalar factor, batch-norm, and implicit regularization.\n\n4. Please specify the limitations of the new method. What is the computational cost?\n\n5. [Question] Please comment on the baselines selected for the UCI test. Are the baselines sufficient for the claim of surpassing \nSOTA?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "MScopVhK5mo", "reviewer": "Reviewer_fgUw", "summary": "The authors propose a framework that performs dropout and infers the number of layers jointly for neural network. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that learning neural network architecture, uncertainty quantification, and dropout regularization in neural networks are all interesting and important topics of research. \n\nThe paper is written in a clear, accessible and well organized way. \n\nI think the main pro going for this paper is the experiments, which shows that this approach captures uncertainty in a reasonable and intuitive way that exceeds or is in general comparable to other methods. The experimental performance is the sole reason I am giving it an above acceptance threshold rating. \n\nSeveral comments below that I think if addressed, could make the paper stronger/clearer:\n1. Computation: there is little mention of computational time or scalability or big Oh running times in the main paper. I think this is a key factor. In my experience dealing with bayesian neural networks, even with variational inference computational and convergence is sometimes an issue, and I think the authors should analyze and compare the computational complexity/actual running time of their methods to other methods more explicitly. My experience working with anything involving indian buffet processes is that it often messes things up/slows things down, so I am curious to see what the author's experience with this is in their work. \n2. Motivation: I find the motivation of the setup, or the lack thereof, slightly unsatisfying.  It seems like the authors just decided to put the most convenient prior they could think of on the layer number, conveniently pick some truncation threshold, and then just applied variational inference and called it a day. I do not blame the authors for this, since choice of priors and computational issues are difficult open problems and this happens in many a Bayesian papers, and but if they have any rationale or deeper reasons other than convenience to pick these priors, I would like to hear it. \n\nIn particular, one motivation that the authors mentioned is that existing methods can often only reduce depth of a fixed architecture, but in the case of this work, I suspect that if one does the truncation, and set the hyperparameters in any reasonable kind of way, with very high probability the posterior is going to allocate most probability on networks with number of layers below some (potentially big) constant number, and then basically this method is also a priori putting a lot of mass onto networks with number of layers below a certain threshold, which kind of defeats a small part of its purpose.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not think potentially negative societal impacts is applicable to this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "-NJAWkcOE-D", "reviewer": "Reviewer_udxa", "summary": "The authors propose to perform joint inference over the depth and effective width of a deep neural network. The depth is modelled as a stochastic process (in particular a Beta process), which allows for the depth to be infinite in theory. Given a fixed maximum width, the effective width is determined using a conjugate Bernoulli distribution which models the dropout probability for each neuron in each layer.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n### Thorough experimental evaluation\n\nThe authors have provided a wide range of benchmarks comparing their proposed method against numerous baselines from the literature. It was great to see a mix of experiments including toy data, small scale regression, image classification, and continual learning. The experiments shown in figures 4 and 6 were both interesting and worthwhile inclusions. The additional experiments in the appendix were also appreciated.\n\nHowever, there is also room for improvement here. The paper was missing:\n\n* An ablation study to disentangle the effects of inference over depth and dropout. \n\n* Deep ensembles as a baseline. This is very important as a standard point of reference in the Bayesian deep learning literature and it is simple to implement. \n\n* Some out-of-distribution (rather than distribution shift) benchmarks. For example out- vs in-distribution classification using entropy or the max logit for CIFAR10 vs SVHN, CIFAR10 vs CIFAR100, and MNIST vs FashionMNIST. Similarly for OOD rejection classification. See Antoran et al. (2020) for descriptions of these.\n\n* A wider range of toy data experiments in order to build intuition for the model's behaviour. \n\n* Dropout as a baseline for all of the experiments (given its strong performance on the UCI regression benchmark).\n\n### Strong empirical performance\n\nNot only was the experimental evaluation thorough but it also showed that the authors proposed method works well! \n\n### Theoretical soundness\nAs far as I can tell the proposed method is sound. However, the caveat is that I am not an expert in Bayesian non-parametric models.\n \n### Novelty\n\nOther papers have done inference over depth (e.g. Antoran et al. (2020)) and over depth + width (e.g. Nalisnick et al. (2019), Dikov and Bayer (2019)). The main novelty of this paper is that there is no fixed upper limit on the depth in theory. However, as with any Bayesian non-parametric model in practice, we need to apply truncation. As a result, this method does have a maximum depth. However, as shown in figure 12, the method allows for a large truncation level without a large increase in train/test time, which is *not* the case in previous depth inference works.  \n\n## Cons\n\n### Clarity\n\nThe paper was not clearly written. In many places, there were typos and grammatical mistakes. I would urge the authors to use a tool like Grammarly to help fix these issues. \n\n### Plagiarism concerns\n\nThis is my biggest issue with this paper. Numerous of the sentences were taken from Antoran et al. (2020) and modified in small/trivial ways. Similarly, several of the figures and tables were adapted with only small changes. These issues are mostly in the experiment descriptions. I am not suggesting that the main idea or contributions of this paper are plagiarised, nor am I suggesting that there was any malice or ill-intent from the authors. However, I do not believe that the paper can be published in this state. \n\nA (non-exhaustive) list of such issues:\n\n* \"All methods’ accuracy degrades quickly for rotations larger than 30◦, but our method is the least overconfident\" (this paper)\nvs\n\"Although all methods perform well on the original test-set, their accuracy degrades quickly for rotations larger than 30°. Here, DUNs and S-ResNets differentiate themselves by being the least overconfident\" (Antoran et al.).\n\n* \"... we train the methods on CIFAR10 and evaluate them on data subject to 16 different corruptions with 5 levels of intensity each\" (this paper) \nvs\n\"we train models on CIFAR10 and evaluate them on data subject to 16 different corruptions with 5 levels of intensity each\"  (Antoran et al.).\n\n* \"We perform Bayesian Optimisation and Hyperband (BOHB) to determine the best configurations including backbone-structure depths and hyperparameter settings for each method\" (this paper)\nvs\n\"We select all hyperparameters, including NN depth, using Bayesian optimisation with HyperBand\" (Antoran et al.).\n\n* \"... employ BOHB that combines the strengths of BO (strong final performance) and Hyperband (scalability, flexibility, and robustness). In particular, we use the HpBandSter implementation of Bayesian Optimization and Hyperband (BOHB) (https://github.com/automl/HpBandSter).\" (this paper)\nvs\n\"... we use Bayesian Optimisation and Hyperband (BOHB). This method, as the name suggests, combines BO with Hyperband, a bandit based HPO method. BOHB has the strengths of both BO (strong final performance) and Hyperband (scalability, flexibility, and robustness). In particular, we use the HpBandSter implementation of BOHB: https://github.com/automl/HpBandSter. \" (Antoran et al.).\n\n* \"We select the best configurations for each method on the standard splits of UCI regression benchmark datasets using the settings, as shown in Table 1. We find these settings to be sufficiently large to ensure all methods’ convergence.\" (this paper)\nvs\n\"We run BOHB for each dataset and split for 20 iterations using the same settings, shown in Table 2. We find these values to be sufficiently large to ensure all methods’ convergence.\" (Antoran et al.).\n\n* Compare figure 3 of this paper to figure 5 of Antoran et al. – it looks to me like the authors of this paper used the code of Antoran et al without proper attribution.\n\n* Similarly, compare tables 1-4 in this paper with figures 2-5 in Antoran et al. \n\n* Compare Appendix D in this paper with Appendix D in Antoran et al. This is similarly filled with minor rewrites and most likely uses the same LaTeX code for the formulae.\n\n## Summary\n\nOverall I thought that the paper was interesting, theoretically sound and provided a strong performance on a range of benchmark tasks vs several baselines. However, due to clarity issues, some missing experimental results, and the liberal usage of content from Antoran et al., I can't recommend acceptance at this point. Should the authors \n1) add OOD experiments, the deep ensembles baseline, and ideally the requested ablation study, as well as\n2) satisfactorily address the offending sections of the paper by rewriting and attributing to Antoran et al. where appropriate,\nI would be happy to increase my score to a 7.\n\n## Other minor suggestions\n\nShow test LL rather than accuracy for fig 4 since this measure both uncertainty calibration as well as the goodness of fit. (At least add the same plot with LL to the appendix.)\n\nShow results for M={16,32} in fig 6. This will show the behaviour better since we have saturation after M=64.\n\n## Post-rebuttal update\n\nGiven that all of my concerns have been addressed by the authors, I have increased my score to a 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitations. One example would be that the proposed method doesn't easily apply to modern neural network architectures such as VGG or ResNet where the layers at different depths are not the same size and which include things like pooling, residual connections and a mix of convolutional and dense layers. \n\nThe authors also do not discuss the potential negative impact of their work. One thing that could be mentioned is the consequences of miscalibrated predictions, particularly for safety-critical applications. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a model and variational inference scheme to learn the depth and width of neural networks simultaneously. All reviewers were impressed with the strong empirical performance of the proposed method compared to multiple baselines in a suite of tasks. One reviewer raised a concern that some text in the experiments were copied directly from Antoran et al. (2020), which the authors promised to rewrite. Overall, a good paper and should be of interest to the community.
Decision: Accept (Poster)