Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work advances an approach for generating bootstrapped neural networks through a single training run, instead of training individual networks for each bootstrap subsets of the dataset. The method, dubbed Neural Bootstrapper (NeuBoots), can be seen as a last-layer approach for uncertainty quantification with an adaptive bootstrapping block at the end. This bootstrapping block borrows ideas from random weight bootstrapping, generative bootstrap samplers and block bootstrapping. In detail, groups of samples are given the same bootstrapping weight (sampled from a Dirichlet distribution randomly initialized at each epoch), with this weight used for perturbing the penultimate layer features and to ponderate the loss for each corresponding sample. Multiple predictions can be obtained from different samplings of the bootstrap weights applied again to the penultimate layer features leading to computationally efficient bagging.
NeuBoots is evaluated on an array of datasets against MC-Dropout and DeepEnsembles, displaying interesting results.

main_review: 
## Recommendation
I find that this paper advances a nice idea with lots of experiments and results. However a large body of literature is missing from this discussion and evaluation (NeuBoots is essentially compared against baseline and MC-Dropout and on image classification against DeepEnsemble), while some of the inner workings of NeuBoots are not clear. 
I'm currently leaning towards rejection. Please check more detailed notes below and at the end some questions for improving the paper or for the rebuttal.


## Paper strengths
- in essence, I find the idea of this paper really nice. Directly adapting bootstrapping to neural networks hasn't lead to exciting results so far [a] and this work advances an interesting approach

- the method is simple (in spite of the loaded formalism used for presenting the concept) and relatively accessible to deploy. The authors have readily shared the implemetation code.

- NeuBoots is evaluated over a rich range of settings and datasets. Few uncertainty estimation works venture to this amount of tasks.


## Paper weaknesses

### Related work
- This section ignores a large part of the literature on uncertainty quantification in deep learning from the past years, essentially stopping at MC-Dropout and DeepEnsemble. There has been significant activity in last-layer (Prior Networks [b], evidential models [c], multiple last-layer approaches [e], DUQ [f], etc.) and/or single training (SWAG [g], TRADI [h], PEP [i]), efficient ensembles (BatchEnsemble [j]) research that is not mentioned nor evaluated here. 

- I recommend including some of these methods for discussion and comparison

- please note that [e] also used bagging on the last layer of a previously trained network, though the method was not as advanced as NeuBoots.

### Quality and clarity 
- I appreciate that the paper and the appendix are rich in experiments and proofs towards better assessing the benefits of this methods. However, to my understanding, there are a few aspects of this method that are not clear from the provided description and intuition for NeuBoots. See below:

- **Diversity.** The bootstrap block weights $\mathbf{w}_{\alpha}$ are randomly initialized at each epoch from $\boldsymbol \alpha$ that is sampled Dirichlet distribution. If I understand correctly $\boldsymbol \alpha$ is never updated. The intuition is that at each epoch the network is trained on a different bootstrap subset. As networks are known to be prone for catastrophic forgetting, there is a risk that the network to "overfit" on the bootrap from the last epoch(s) and forget the previous ones. From what I can see there is no explicit regularization mechanism in NeuBoots to deal with that. One way to assess this is to analyze the diversity of the predictions (diversity is usually one of the main advantages of ensembles [k],[j]): a higher diversity would infirm this hypothesis. Usually more compelling diversity scores are obtained on distribution shift (e.g., ImageNet-C, Cifar-C [l]) or OOD. The confidence bands in OOD plots in the appendix do not hint a high diversity.

- **Grouping samples in blocks**. Block boostrapping has been mostly used for correlated or redundant samples. In this case, the samples are seemingly grouped from a random initialization before training. Did the authors try other strategies? Was there an impact? It would be useful to include some strategies or rules of thumb for selecting the number of segments S. In the active learning experiments, it is not clear how are the new samples added to the bootstrap segments.


### Experiments
- as mentioned above, there are several other efficient uncertainty approches that are missing from the discussion and evaluation. MCDrop and DeepEnsemble are certainly useful baselines, but there are many works in-between and it would be helpful to know where is NeuBoots placed.

- accuracy and calibration on in-domain data offers just a partial view on the effectiveness of the approach. CIFAR experiments can be extended to dataset shift [m],[l], where uncertainty estimation and calibration is even more important. 

- for the OOD experiments I believe there are some inconsistencies in the description: in L206 "we tune hyperparameters in the training phase using in-distribution samples to keep the fairness of our method", then in L213 "For our method, we tune whole hyperparameters using a separate validation set, which consists of 1,000 images from in-distribution and out-distribution, respectively". It's not clear whether NeuBoots uses the OOD data for training or not. Also for OOD detection, the classifier prediction values can be used directly without the requirement of training a logistic regression for this.  
Please note that most of the recent approaches refrain from using OOD during training as it's choice can largely impact the performance on the test OOD set [n]. Directly evaluating on unknown OOD data (as done in DeepEnsembles [18])


## Questions
Here are some questions that could be potentially addressed in the rebuttal:

1. How is NeuBoots doing in terms of diversity of predictions? Can the authors please explain if and how NeuBoots does not "overfit" on the bootstrap weights from the final epochs?

2. I suggest a simple baseline to validate that NeuBoots had learned richer information and can come up with diversity in predictions. This baseline consists in leveraging Gaussian Dropout (essentially perturbing the hidden features with noise sampled from a user-defined Guassian, section 10 in [o]) to perturb last layer features and generate and ensemble like NeuBoots does. How is NeuBoots expected to perform here?

3. How is NeuBoots expected to perform against other computationally efficient methods for uncertainty estimation?

4. Is there any recommended strategy for grouping samples in blocks?


**References:**


[a] J. Nixon et al., Why Are Bootstrapped Deep Ensembles Not Better?, NeurIPS 2020 workshop

[b] A. Malinin and M. Gales, Predictive uncertainty estimation via prior networks, NeurIPS 2020

[c] M. Sensoy et al., Evidential deep learning to quantify classification uncertainty, NeurIPS 2018

[d] T. Joo et al., Being bayesian about categorical probability, ICML 2020

[e] N. Brosse et al., On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation, arXiv 2020
 
[f] J. van Amersfoort et al., Uncertainty estimation using a single deep deterministic neural network, ICML 2020

[g] W. Maddox et al., A Simple Baseline for Bayesian Uncertainty in Deep Learning, NeurIPS 2019

[h] G. Franchi et al., TRADI: Tracking deep neural network weight distributions, ECCV 2020

[i] A. Mehrtash et al., PEP: Parameter Ensembling by Perturbation, NeurIPS 2020

[j] Y. Wen et al., BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning, ICLR 2020

[k] S. Fort et al., Deep Ensembles: A Loss Landscape Perspective, arXiv 2019

[l] D. Hendrycks and T. Dietterich,  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR 2019

[m] Y. Ovadia et al., Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, NeurIPS 2020

[n] A. Shafaei, A Less Biased Evaluation of Out-of-distribution Sample Detectors, BMVC 2019

[o] N. Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 2014


--------------------------------------------------
--------------------------------------------------

## Final recommendation

The authors have addressed in the rebuttal several concerns from reviewers and I thank them for the detailed responses, clarifications and new experiments. They are not wasted and I'm sure they'll contribute to making the paper stronger.

After the exchange of questions and responses with the authors, reading other reviews and responses and the paper again I would recommend this paper for acceptance.
The paper advances an interesting idea and the additional experiments and clarifications should enforce the findings in this work, i.e., that there is some form of bootstrapping happening. 
However, I find that the experimental evaluation is not complete yet. As mentioned in the discussions, I strongly encourage the authors to evaluate NeuBoots on data distribution shift (CIFAR-C) and to include other relevant and relatively recent methods, i.e., newer than MC-Dropout,  in their comparison.

limitations_and_societal_impact: Limitations and societal impact were not discussed.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The work proposes NeuBoots: a new ensembling method for neural networks inspired by bootstrapping estimation. The method is simple and involves training one neural network to simultaneously represent a whole family of models; each such model is scored during training with a loss where different training objects have different weights. These models are obtained from the base model with a single multiplication of weights of training objects (or, more precisely, weights of sets of training objects) and representations in the final part of the network. During testing, models are ensembled for multiple random weightings for training objects. The authors validate the method on computer vision tasks and synthetic data. It is concluded that the method is superior to other bagging methods on various metrics related to uncertainty estimation such as active learning performance or out-of-distribution detection metrics and is more computationally efficient.
main_review: The method and the experiments are explained with decent clarity. I like the overall composition of the paper, but at a finer level there are some issues. Some details in the narrative are unclear. For example, I didn’t understand why GBS performance degrades and there isn’t data on this in the paper to see for myself. Also, when looking at algorithm 1 on page 4 and the related description, I didn’t understand if sampling of alpha is occurring every optimization step or every epoch; the text states that it’s only happening once per epoch, but then this means that the final network only works for one alpha vector and isn’t trained to work for any other alpha for the whole last epoch? Shouldn’t that lead to deterioration of ensembling performance?

The method mostly builds on the GBS bootstrapping method. NeuBoots reduces computational complexity of bootstrapping very nicely, but also it isn’t clear why elementwise multiplication of block bootstrap weights with representations in a neural network was chosen, compared to any other operation, and this isn’t touched upon in the paper.

Finally, there are linguistic errors (>3 per page on average).

The experiments are sufficiently diverse and comparisons of the method are presented with other method on various tasks and with various architectures and datasets. The results on some tasks like out-of-distribution detection and semantic segmentation seem good. However, parts of the evaluation were confusing for me and I think there are some issues with it.

In Table 3.1 it is shown that NeuBoots outperforms the traditional bootstrap significantly with equal numbers of ensembling members. It isn’t immediately clear to me why this does occur, because as I understand it NeuBoots only presents an approximation to the standard bootstrap with various things relaxed (amortization, weights for blocks of training set examples instead of for individual training set examples, “soft” weights versus binary “hard” weights). Of course it’s completely possible that it just happens to work better for neural models in general but authors don’t attempt to provide an explanation for this in the text.

The paper doesn’t present a comparison of NeuBoots and various methods which, in my opinion, actually make state-of-the-art trade-offs between uncertainty estimation quality and computational complexity of training and inference. Among such methods I can name [MIMO](https://openreview.net/forum?id=OGg9XnKxFAH), [BatchEnsemble](https://arxiv.org/abs/2002.06715), [rank-1 BNNs](http://proceedings.mlr.press/v119/dusenberry20a.html) (see the comparison here: https://github.com/google/uncertainty-baselines/tree/main/baselines/cifar). Instead, it is most often compared with MC dropout in the paper, which is not a state-of-the-art method. Based on the values that I see in those papers it seems like they would provide better uncertainty estimation performance as measured at least by some metrics (e.g. NLL) than NeuBoots. Because of this, it’s difficult to establish the significance of NeuBoots.

The same concern about absence of comparison to state-of-the-art may apply to out-of-distribution detection as well, see, e.g. [Energy-based Out-of-distribution Detection](https://arxiv.org/abs/2010.03759).

The authors may answer that bagging is orthogonal to other ensembling methods and should mostly be evaluated separately. I think I would not agree with this as it isn't clear NeuBoots would yield benefits when combined and applied together with state-of-the-art ensembling methods. See for example [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://arxiv.org/abs/1511.06314) where they showed that combining bootstrap with Deep Ensembles can reduce classifier accuracy.

The comparison between methods in Table 4.1 is confusing. NeuBoots is compared with Deep Ensemble, Monte-Carlo dropout, and usual neural network training there.
- The major source of confusion for me here is that the performance of the Deep Ensemble method with 5 networks in the ensemble is worse than a single neural network training. In principle prior work established it to be much better, see, e.g. analogous benchmark comparison in Ashukha et al: [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://arxiv.org/pdf/2002.06470.pdf), Table 3, row 6.
- The text mentions that adversarial training and the Brier loss function is utilized for Deep Ensemble, but presumably not for other methods, which may be seen as unfair, because most often Deep Ensemble is evaluated without these additions (see e.g. in MIMO, BatchEnsemble, rank-1 BNNs papers with links above).
- ECE, NLL and Brier Score metrics are given in percents, but it isn’t clear what is taken as 100% on each metric.
- It seems from Table 4.1 that NeuBoots ensemble NLL is worse compared to a single network. However, it is shown that NeuBoots outperforms a single network and MC dropout on tasks like semantic segmentation and imbalanced classification. This discrepancy in performance on various methods is a bit surprising and it isn’t explained by the authors.

limitations_and_societal_impact: Limitations of the method don't seem to be discussed at length.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper leverages ideas from statistical bootstrapping and proposes a neural network architecture and training loss that can computationally efficiently generate uncertainty estimates. The method is evaluated on a wide range of tasks where uncertainty quantification is important and shows comparable or better performance compared to some popular methods. 
main_review: The paper proposes a practical approach for uncertainty quantification and conducts an extensive set of experiments that illustrate the effectiveness of the approach. I think this paper will add value to the community.

While the experiments are quite extensive, I think the paper will benefit a lot from providing more justification/explanation for the proposed approach.

For example, the authors might want to comment on the following sanity check: given infinite data, is the resulting distribution going to concentrate on the right function?

Also, the index function that assigns an integer index to each data point is not very well motivated. It feels strange to relate a datapoint to a particular node in the last layer. Why not make the mapping smoother? Instead of projecting alpha on one-hots, why not project on random unit vectors, etc.?

Other comments:
- Loss (2.3): When the weights are all positive, wouldn’t this loss lead to the same minimizer over all possible weights when the network is overparameterized, and then there is no diversity? (Although this doesn’t apply to the architecture you propose.)
- Line 66 Bootstrap Distribution Generator: The idea seems very similar to [1] (although [1] considers a particular form of additive bootstrap in the loss, it can be easily extended to other forms of bootstrapping). Could you comment on the difference.
- Section 4.2 Active Learning: Why not compare to ensembles?

[1] Hypermodels for Exploration, ICLR 2020.

limitations_and_societal_impact: Not discussed.
review_title: null

Meta-review: Reviewers recognized the importance of providing an efficient way to bootstrap neural nets for uncertainty quantification and praised the proposed approximate scheme.

During the review phase, reviewers highlighted how many experimental details and comparison w.r.t. modern neural uncertainty quantification methods were missing from the current version of the paper.
The authors provided an extensive rebuttal and managed to clarify the missing details and convince reviewers on certain points. However, some questions were left open: a predictive performance comparison is still missing as well as many references as highlighted by the reviewers.This work is accepted subject to the authors incorporating the promised changes and additional experimental results.

Decision: Accept (Poster)