Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a framework that performs dropout and infers the number of layers jointly for neural network. 
main_review: I believe that learning neural network architecture, uncertainty quantification, and dropout regularization in neural networks are all interesting and important topics of research. 

The paper is written in a clear, accessible and well organized way. 

I think the main pro going for this paper is the experiments, which shows that this approach captures uncertainty in a reasonable and intuitive way that exceeds or is in general comparable to other methods. The experimental performance is the sole reason I am giving it an above acceptance threshold rating. 

Several comments below that I think if addressed, could make the paper stronger/clearer:
1. Computation: there is little mention of computational time or scalability or big Oh running times in the main paper. I think this is a key factor. In my experience dealing with bayesian neural networks, even with variational inference computational and convergence is sometimes an issue, and I think the authors should analyze and compare the computational complexity/actual running time of their methods to other methods more explicitly. My experience working with anything involving indian buffet processes is that it often messes things up/slows things down, so I am curious to see what the author's experience with this is in their work. 
2. Motivation: I find the motivation of the setup, or the lack thereof, slightly unsatisfying.  It seems like the authors just decided to put the most convenient prior they could think of on the layer number, conveniently pick some truncation threshold, and then just applied variational inference and called it a day. I do not blame the authors for this, since choice of priors and computational issues are difficult open problems and this happens in many a Bayesian papers, and but if they have any rationale or deeper reasons other than convenience to pick these priors, I would like to hear it. 

In particular, one motivation that the authors mentioned is that existing methods can often only reduce depth of a fixed architecture, but in the case of this work, I suspect that if one does the truncation, and set the hyperparameters in any reasonable kind of way, with very high probability the posterior is going to allocate most probability on networks with number of layers below some (potentially big) constant number, and then basically this method is also a priori putting a lot of mass onto networks with number of layers below a certain threshold, which kind of defeats a small part of its purpose.  
limitations_and_societal_impact: I do not think potentially negative societal impacts is applicable to this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a comprehensive but effective method to simultaneously perform inference for the NN depth and width. As the idea might not be new, the execution of the method is solid. The methodology is also linked to Bayesian information criterion with a theorem.
main_review: Originality:
The structure selection has been studies by a few related works. This work models the network depth with Beta process while the Bernoulli variables indicate the if a node is kept or dropped. With this modeling, it is easy to control the network shape with the distribution parameter - alpha and beta, which is smart. It is interesting to see behaviors of network shape under different (alpha, beta) setups. The inference is done with standard stochastic variational inference.

Quality, clarity, and significance:
1. The paper is well-written and easy to read. The graphs are clear and intuitive.
2. The major doubt comes from the experiment section. 

-As the network architecture (depth and width) is considered, why there's no results on efficiency reported? 

-The test error rate on Cifar10 is over 10%, which is not a competitive results. With standard VGG network, it could easily achieve less than 10%. What is the network used for Cifar10? 

-Only small scale datasets are used. Does the method scale to large datasets? For example, TinyImagenet might be a proper choice for Bayesian deep learning.

-How is the performance of detecting out-of-domain data?

-How is the performance compared with deep ensembles? Or other Bayesian method related to sparsity and model selection? For example: 

Louizos et al, Learning sparse neural networks through l 0 regularization, ICML 2018.
Dai et al., Compressing Neural Networks using the Variational Information Bottleneck, ICML 2018.
Cui et al, Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression, CVPR 2021.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose to perform joint inference over the depth and effective width of a deep neural network. The depth is modelled as a stochastic process (in particular a Beta process), which allows for the depth to be infinite in theory. Given a fixed maximum width, the effective width is determined using a conjugate Bernoulli distribution which models the dropout probability for each neuron in each layer.  
main_review: ## Pros

### Thorough experimental evaluation

The authors have provided a wide range of benchmarks comparing their proposed method against numerous baselines from the literature. It was great to see a mix of experiments including toy data, small scale regression, image classification, and continual learning. The experiments shown in figures 4 and 6 were both interesting and worthwhile inclusions. The additional experiments in the appendix were also appreciated.

However, there is also room for improvement here. The paper was missing:

* An ablation study to disentangle the effects of inference over depth and dropout. 

* Deep ensembles as a baseline. This is very important as a standard point of reference in the Bayesian deep learning literature and it is simple to implement. 

* Some out-of-distribution (rather than distribution shift) benchmarks. For example out- vs in-distribution classification using entropy or the max logit for CIFAR10 vs SVHN, CIFAR10 vs CIFAR100, and MNIST vs FashionMNIST. Similarly for OOD rejection classification. See Antoran et al. (2020) for descriptions of these.

* A wider range of toy data experiments in order to build intuition for the model's behaviour. 

* Dropout as a baseline for all of the experiments (given its strong performance on the UCI regression benchmark).

### Strong empirical performance

Not only was the experimental evaluation thorough but it also showed that the authors proposed method works well! 

### Theoretical soundness
As far as I can tell the proposed method is sound. However, the caveat is that I am not an expert in Bayesian non-parametric models.
 
### Novelty

Other papers have done inference over depth (e.g. Antoran et al. (2020)) and over depth + width (e.g. Nalisnick et al. (2019), Dikov and Bayer (2019)). The main novelty of this paper is that there is no fixed upper limit on the depth in theory. However, as with any Bayesian non-parametric model in practice, we need to apply truncation. As a result, this method does have a maximum depth. However, as shown in figure 12, the method allows for a large truncation level without a large increase in train/test time, which is *not* the case in previous depth inference works.  

## Cons

### Clarity

The paper was not clearly written. In many places, there were typos and grammatical mistakes. I would urge the authors to use a tool like Grammarly to help fix these issues. 

### Plagiarism concerns

This is my biggest issue with this paper. Numerous of the sentences were taken from Antoran et al. (2020) and modified in small/trivial ways. Similarly, several of the figures and tables were adapted with only small changes. These issues are mostly in the experiment descriptions. I am not suggesting that the main idea or contributions of this paper are plagiarised, nor am I suggesting that there was any malice or ill-intent from the authors. However, I do not believe that the paper can be published in this state. 

A (non-exhaustive) list of such issues:

* "All methods’ accuracy degrades quickly for rotations larger than 30◦, but our method is the least overconfident" (this paper)
vs
"Although all methods perform well on the original test-set, their accuracy degrades quickly for rotations larger than 30°. Here, DUNs and S-ResNets differentiate themselves by being the least overconfident" (Antoran et al.).

* "... we train the methods on CIFAR10 and evaluate them on data subject to 16 different corruptions with 5 levels of intensity each" (this paper) 
vs
"we train models on CIFAR10 and evaluate them on data subject to 16 different corruptions with 5 levels of intensity each"  (Antoran et al.).

* "We perform Bayesian Optimisation and Hyperband (BOHB) to determine the best configurations including backbone-structure depths and hyperparameter settings for each method" (this paper)
vs
"We select all hyperparameters, including NN depth, using Bayesian optimisation with HyperBand" (Antoran et al.).

* "... employ BOHB that combines the strengths of BO (strong final performance) and Hyperband (scalability, flexibility, and robustness). In particular, we use the HpBandSter implementation of Bayesian Optimization and Hyperband (BOHB) (https://github.com/automl/HpBandSter)." (this paper)
vs
"... we use Bayesian Optimisation and Hyperband (BOHB). This method, as the name suggests, combines BO with Hyperband, a bandit based HPO method. BOHB has the strengths of both BO (strong final performance) and Hyperband (scalability, flexibility, and robustness). In particular, we use the HpBandSter implementation of BOHB: https://github.com/automl/HpBandSter. " (Antoran et al.).

* "We select the best configurations for each method on the standard splits of UCI regression benchmark datasets using the settings, as shown in Table 1. We find these settings to be sufficiently large to ensure all methods’ convergence." (this paper)
vs
"We run BOHB for each dataset and split for 20 iterations using the same settings, shown in Table 2. We find these values to be sufficiently large to ensure all methods’ convergence." (Antoran et al.).

* Compare figure 3 of this paper to figure 5 of Antoran et al. – it looks to me like the authors of this paper used the code of Antoran et al without proper attribution.

* Similarly, compare tables 1-4 in this paper with figures 2-5 in Antoran et al. 

* Compare Appendix D in this paper with Appendix D in Antoran et al. This is similarly filled with minor rewrites and most likely uses the same LaTeX code for the formulae.

## Summary

Overall I thought that the paper was interesting, theoretically sound and provided a strong performance on a range of benchmark tasks vs several baselines. However, due to clarity issues, some missing experimental results, and the liberal usage of content from Antoran et al., I can't recommend acceptance at this point. Should the authors 
1) add OOD experiments, the deep ensembles baseline, and ideally the requested ablation study, as well as
2) satisfactorily address the offending sections of the paper by rewriting and attributing to Antoran et al. where appropriate,
I would be happy to increase my score to a 7.

## Other minor suggestions

Show test LL rather than accuracy for fig 4 since this measure both uncertainty calibration as well as the goodness of fit. (At least add the same plot with LL to the appendix.)

Show results for M={16,32} in fig 6. This will show the behaviour better since we have saturation after M=64.

## Post-rebuttal update

Given that all of my concerns have been addressed by the authors, I have increased my score to a 7.
limitations_and_societal_impact: The authors do not discuss limitations. One example would be that the proposed method doesn't easily apply to modern neural network architectures such as VGG or ResNet where the layers at different depths are not the same size and which include things like pooling, residual connections and a mix of convolutional and dense layers. 

The authors also do not discuss the potential negative impact of their work. One thing that could be mentioned is the consequences of miscalibrated predictions, particularly for safety-critical applications.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new method for training deep neural networks, where the parameters controlling model depth and activation 'drop' probabilities are being trained and updated, as well as the model's parameters. After developing a theoretical framework where the model structure is treated as a stochastic process, they run a series of experiments to evaluate it over a series of tasks, and present promising results regarding their model's ability to estimate uncertainty.
main_review: **Originality:** 

The method presented in the paper is new. Several similar methods are discussed under 'prior work', and the differences are highlighted.

**Quality:**

The submission appears to be technically sound. 

In section 3, some of the decisions made felt arbitrary and I would appreciate more care being given for their advantages and disadvantages. (See limitation: 1)

Several things struck me as odd with the method:

1. The dropout variant used in the paper doesn't include the 1/(1-p) factor which we know to be vital for standard dropout. Dropout is still used during inference so it may be less important, but it seems to me that the factor is still necessary for training since the value of the drop-rate is inconsistent.

2. The paper used batch-norm liberally for experiments (4), but its effect wasn't addressed in section 3 and I worry it could be substantial. Regularization effect aside, it can also affect the training's dynamics, especially if the dropout scalar factor is missing.

3. In Algorithm 1, $S$ samples of networks (Depth + Dropmask) are being drawn, and their results are averaged to estimate log-likelihood (for the ELBO). While it's not completely the same, it has similarities with $\text{Dropout}_K$, an experimental regularization that was explored in this paper: [2]. Paper [2], however, showed that $\text{Dropout}_K$ is suboptimal, since by averaging over different dropout masks you lost the *implicit* regularization of dropout. I worry that this may happen in this paper as well (this can explain why the chosen value for $S$ is so low). The good news is that even if this is a problem, this could possibly be fixed with the regularization term they propose.

The experimental section is well organized, and supported by complementary details in the supplementary. The authors claim superior performance when comparing their method to state-of-the-art methods albeit I was not convinced that the baselines were good enough for this assessment. (see: A Survey of Uncertainty in Deep Neural Networks[3], Table IV).

One more thing I felt was missing is an illustration of the dynamics of $a_k, b_k$ during training. As is, I am not confident enough that the motivations presented for designing the new method align with what it does in practice.

The submission included code, which allowed easy recreation of the synthetic experiment. I ran few sanity checks (Like detaching loss components and monitoring network depth)-- they all checked out, but they were somewhat limited by the easy benchmark.

**Clarity:**

The paper is well written.

Both the introduction and related work sections detail the relevant work from the machine learning aspect. The relevant background regarding the mathematical aspects (like [1]), however, is only briefly mentioned. This makes the later understanding of the introduced method harder for the readers who are not familiar with this works.

Algorithm 1 (in the Appendix) ("Training of our proposed network") was very helpful for understanding section 3 from a birds' eye view, consider incorporating it into the main text.

**Significance:**

The submission addresses difficult tasks (uncertainty estimation, continual learning), with a new approach. While I was not convinced of the practicality of the approach as is, others are likely to use/ build upon this work.

[1] https://arxiv.org/abs/1106.0539
[2] https://arxiv.org/abs/2002.12915
[3] https://arxiv.org/abs/2107.03342
limitations_and_societal_impact: 1. [Request] Section 3.2: Please provide more early intuition for the usage of Beta Process.

2. [Request] The number of samples ($S=5$) used at Algorithm 1 seems arbitrary. Furthermore, as a key component in the method, I would be interested to see how it was chosen. Can you run an ablation study to show its effect? I would be interested to learn how the dynamics of $a_k, b_k$ change in accordance to $S$.

3. [Question] Please address the points I have made about the dropout scalar factor, batch-norm, and implicit regularization.

4. Please specify the limitations of the new method. What is the computational cost?

5. [Question] Please comment on the baselines selected for the UCI test. Are the baselines sufficient for the claim of surpassing 
SOTA?
review_title: null

Meta-review: The submission proposes a model and variational inference scheme to learn the depth and width of neural networks simultaneously. All reviewers were impressed with the strong empirical performance of the proposed method compared to multiple baselines in a suite of tasks. One reviewer raised a concern that some text in the experiments were copied directly from Antoran et al. (2020), which the authors promised to rewrite. Overall, a good paper and should be of interest to the community.
Decision: Accept (Poster)