Deep Combinatorial Aggregation
Authors: Yuesong Shen
Daniel Cremers
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep combinatorial aggregation
deep combinatorial weight averaging
consistency enforcing loss
uncertainty estimation

Abstract: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue. This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks. In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA). DCA creates multiple instances of network components and aggregates their combinations to produce diversified model proposals and predictions.  DCA components can be defined at different levels of granularity. And we discovered that coarse-grain DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation. For fine-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training. It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers. Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA. We experiment on in-domain, distributional shift, and out-of-distribution image classification tasks, and empirically confirm the effectiveness of DCWA and DCA approaches.

Deep Combinatorial Aggregation
Yuesong Shen 1,2
Daniel Cremers 1,2
Technical University of Munich, Germany Munich Center for Machine Learning, Germany (yuesong.shee shen, cremers) Stum.dee
Abstract
Neural networks are known to produce poor uncertainty estimations, and vari- ety of approaches have been proposed remedy this issue. This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks. In this work, we explore combinatorial gener- alization of deep ensemble called deep combinatorial aggregation (DCA). DCA creates multiple instances of network components and aggregates their combina- tions to produce diversified model proposals and predictions. DCA components can be defined at different levels of granularity. And we discovered that coarse-grain DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation. For fine- fine-grain DCAs, we dis- cover that average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training. It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule adaptation of BatchNorm layers. Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA. We experiment in-domain, distributional shift, and -fist--iiitiitutiinn image classification tasks, and empirically confirm the effectiveness of DCWA and DCA approaches,
1 Introduction
Deep learning has achieved groundbreaking progress and neural networks are now widely used in various domains
(24). However, they are known to produce poor uncertainty estimations 37 10,
which can be problematic for challenges like safety-critich applications [17,26) or active learning
[35]. Numerous approaches have been proposed tackle this issue, among which an effective yet simple method is deep ensemble [23] Deep ensemble yields state-of-the-art results for uncertainty aware learning 37 10. and it does not require elaborate architectural design and hyperparameter search. However, while it aggregates multiple separately trained models, it can neither generate new samples from the posterior to obtain more diverse predictions, nor produce a summarizing average model which improves on the individual models in the ensemble.
Motivated by the success of deep ensemble, this paper, we propose deep combinatorial aggregation (DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neural networks, we explore the idea of ensembling components of the network architecture and combining them to form an enriched collection of model proposals. DCA inherits the simplicity and effectiveness of deep ensemble, and additionally leads to several new possibilities: Apart from generating diversified set of model proposals, we discover that fine-grain DCA can lead to a new average proposal via deep combinatorial weight averaging (DCWA). Furthermore, DCA training can benefit from consistency enforcing loss, which can produce DCA models that surpass standard deep ensemble both in classification performance and uncertainty estimation.
Source code is available at htttpp///ggthab..cco//tua//iiiis
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.1 Related work
Improving not only the predictive performance also the uncertainty estimation of neural networks has been core objective of Bayesian deep learning, where an abundance of prior work exists. This includes methods based on variational inference and weight perturbation such as Bayes by backprop
8, [] and its variants 41 Bayesian interpretation of dropout including MC dropout [6] and variational dropout [T 30 expectation propagation which leads to probabilistic backpropagation
[14, Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics (SGLD) 40 and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [3. stochastic gradient descent (SGD) approximate MCMC which results in SWAG (88] well as Bayesian noisy optimizers such as variational online Gauss-Newton (VOGN) [8] and approaches using Laplace approximation 20 (3). Beyond the Bayesian formulation, methods like post-hoc calibration 9| 39
readjust trained networks to produce more calibrated predictions, while approaches like evidential 34 also make of ideas like subjective logic.
deep learning
Most relevant our work the deep ensemble method [23], which aggregates multiple independently trained network models with different initial parameters, Deep ensemble has been shown to produce state-of-the ofte--of-the-att results for uncertainty estimation [37 10). It can be combined with methods like MC dropout [5] and SWAG 44, extended to the hyperparemeter space [3] Several variants have also been proposed, which often aim at providing more computationally or memory-efficient alternatives. This includes snapshot ensemble [15], BatchEnsemble [42] fast geometric ensembling (7]. TreeNet [25] etc. In contrast, this work aims at exploring a combinatorial generalization of deep ensemble to obtain new features and improve the performance of uncertainty estimation.
Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable to stochastic weight averaging (SWA) [16]
1.2 Contributions
The main contributions of this paper are the following:
We propose deep combinatorial aggregation, combinatorial generalization of deep ensem- ble that can produce more diverse model proposals and predictions.
We explore DCA different levels of granularity and propose deep combinatorial weight averaging (DCWA) for fine-grain DCA models. It produces new average model that improves on standard training and is competitive w.r.t. alternatives like stochastic weight averaging (SWA) [[6]
We introduce consistency enforcing loss adapted for DCA training. strengthens the predictive consistency of DCA model proposals and consistently improves the performance
DCA and DCWA models.
We conduct experiments in-domain, distributional shift, and classification tasks, which validate our analysis and demonstrate the effectiveness DCA for uncertainty-awart learning.
image
Deep combinatorial aggregation (DCA)
In this section, introduce the deep combinatorial aggregation (DCA) method. To simplify discussion, we start with a layerwise setting and assume that our base model is neural network with layers. This setting can easily be generalized other DCA variants discussed Section 2.3
DCA
Figure 1: Illustration of layerwise deep combinatorial aggregation using a three-layer neural network: two sets of DCA parameter instances result in 23 model proposals. random proposal is chosen for each forward pass during both training and test time to generate diverse prediction samples.

---

2.1
Methodology
The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble [23] creates multiple copies of the entire model with different initializations, layerwise DCA instead creates multiple instances for each layer in the model. Randomly selected instances from network layers can be combined to form a variety of model proposals. As shown in Figure [ this results in an exponential number of proposals w.r.t. the network depth, since n sets of layer instances lead n total network proposals.
To ensure consistency among the instance combinations, all layer instances are jointly trained: during each feed-forward, a random instance choice from each layer is made to construct model proposal, and the parameters belonging to the selected layer instances then updated via backpropagation. This differs from deep ensemble [33] where model copies are independently trained. In Appendix A we provide a pseudo-code for layerwise DCA training.
During inference time, to obtain uncertainty-awe aware predictions, one can simply sample multiple DCA model proposals and aggregate their predictions.
2.2 Understanding DCA
While DCA is simple and intuitive procedure, it is helpful to conduct a more in in-depth theoretical analysis to understand the assumptions it implicitly makes and their implications.
Being able freely combine samples of layer parameters assumes that they follow mutually inde- pendent distributions. This translates to the assumption that the weight posterior p(0lys ly, x) should be layerwise decomposable:
L p(O x)) (o')
this a valid assumption? Not exactly. Admittedly, the posterior is proportional to the product of prior p(0) and likelihood p(y|2,, 0) (iee. p(Oly,x) x p(0)p(y))a 0)), and the weight prior p(0) often satisfies layerwise independence assumption, e.g., commonly used weight decay equivalent to Gaussian prior with constant diagonal covariance matrix. However, the likelihood term p(y| 0) is in general not decomposable. In fact, following the directed graphical model (21) interpretation of standard neural networks (36), the base network represents the overall distribution p(h1:2-1 y/a, 0) where ht' represents hidden neurons in layer l. Interestingly, p(h! 1-L-1 .ulz, (g 0) is actually layerwise decomposable itself
p(h 1 1/2,,, 0) = II
p( ,0), (z.y:=h",m))
(2)
1-1
Nevertheless, the likelihood term p(yl /x, 0) requires marginalization of all hidden neurons 1
ply|z, 0) f
Juzz-1 p(h 2 y/2 --
(3)
This entangles the layer parameters, and p(y|<, 0) is no longer decomposable in general.
Thus approximations made when perform layerwise DCA, and experiments show that this indeed results in some performance penalty. The above analysis also implies that weight aggregation coarser multilayer level could alleviate the issue. This is also confirmed by empirical findings (of'. Section 5.3).
2.3
Granularity aggregation
The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregation different levels of granularity: From finest to coarsest, DCA can be defined at neuronwise, layerwise, multilayer, modelwise levels. For convolutional neural networks, due to spatial dimensions, DCA can work with channel components instead of neurons. Note that modelwise DCA is similar to deep ensemble, except that in each epoch DCA model copies are trained with distinct subsets of training data which results from the random component selection, and it can benefit from consistency enforcing loss introduced in Section
along the
3

---

In general, DCA with finer granularity generates greater amount of model proposals and more varied predictions. However, the DCA components are more tightly coupled, deviating more significantly from the assumption of decomposable posterior, which could lead to worse performance. This results in tradeoff between performance and prediction variety. Alternatively, one can also enrich the set of model proposals by using more DCA instances, at the cost of a higher computational budget. This can also lead to improved performance (ef. Section 5.3).
Among the broad range of granularities, there exists a notable dichotomy between
DCA and multilayered DCA. This comes from the fact that component instances from multilayered DCA variants can have similar behaviors but dissimilar weights. Neural networks admit a large number of equivalent reparameterizations via the reordering of hidden layer neurons. While the joint training of DCA enforces consistency among different DCA components of the network model, it does not prevent equivalent reordering of hidden neurons inside multilayer components. This issue does not occur for (sub)layerwise DCA cases. To make the distinctions, we refer to (sub)layerwise cases as fine-grai graigrain DCA and multilayered cases as coarse-grain DCA.
Deep combinatorial weight averaging (DCWA) for fine-grain aggregation
For fine-grain DCA models, it turns out that averaging the learned weights of DCA components leads to an improved parameterization of the base model. We discuss this procedure here detail.
Deep combinatorial weight averaging Since component instances of a fine-grail mhe-grain DCA model have compatible weights after the joint training, it is sensible to consider their mean value. This produces a new average parameterization for the base network model. We refer to this process as deep combinatorial weight averaging (DCWA).
Consider as an example the layerwise DCA model for a base neural network with / layers. After the joint training of sets of DCA layer instances parameterized by O = (0) (oim...., Of.)). DCWA simply computes the average parameterization 0 (0) ,.... 0²) for the base network model, where for each layer 1, we have 0' = Σm=0 0%
Experiments show that DCWA achieves comparable test accuracy w.r.t. corresponding DCA predic- tions (ef. Section 5.3). It also consistently outperforms the standard training of the base network, and delivers comparable results w.r.t. to SWA [16] (cf. Section 5.1
Comparison to SWA It is interesting to compare DCWA with SWA [16] since they are both weight averaging schemes that improve the standard training. This said, DCWA and SWA based on different principles: SWA averages over the SGD trajectory while DCA relies on combining component aggregations. In practice, SWA requires custom learning rate scheduling and careful choice of end learning rate. Also, SWA requires an additional Batch Normalization update [16] to produce good predictions, which leads to an extra overhead. In contrast, DCWA does not have any of these issues and is simple to implement and deploy.
4 Consistency enforcing loss for DCA and DCWA
Through component combination, DCA is able to produce combinatorial amount of model proposals. However, during the joint training, each DCA component receives gradient updates from different model proposals. These updates can be inconsistent, which can lead to suboptimal training of the DCA model. To remedy this issue, we propose in this section consistency enforcing loss to encourage consistency among DCA model proposals.
Consistency enforcing loss To promote consistency among DCA model proposals, we encourage DCA predictions agree with both the ground-truth and the predictions from other model proposals. To achieve this, given an input with ground-truth y and a DCA model proposal parameterized by 0. instead of minimizing the negative log- log-likelihood f (æt y; 0) - log p(yla;; θ), the consistency enforcing loss includes an additional KI. divergence term between the predictive output probability
p(y|2; 0) and reference output probability p:
f(x y, p; 0) log p(yl DKL
(4)

---

Here the reference output probability ğ should be chosen to reflect the predictions from other model proposals, In practice, we perform multiple feed-forward steps on the same imput using randomly chosen model proposals (0(9) Isssa and the i-th consistency enforcing loss ( simply uses the output probability prediction p from the previous step as reference distribution: g03 (x, y, 1); ) Note that the output predictions are reused and one additional feed-forward pass is needed to compute first reference distribution p!o), The gradient updates from the multiple feed-forward steps can be accumulated together for single optimization step on the global DCA parameter set +
Also, observe that the consistency enforcing loss uses forward KI. divergence D to regularize the predictive output distribution p. The forward KI. divergence (plpp))) "zero- avoiding" and allows p to be flat and uncertain. This is helpful for the output predictive distribution to find a good compromise between the reference prediction p and the ground-truth y. especially in cases where they strongly disagree with each other.
Relation to existing approaches KL divergence as a regularization term in loss function appears in many contexts: for Bayesian neural networks [8] is used regularize weight posterior; in variational autoencoders (VAE) 19 it regularizes the latent representation. Applying KL divergence on output distributions has been proposed by Zhang al. [46] for knowledge distillation, and Song and Chai [8 for collaborative learning. However, to the best of our knowledge, this has not been extensively explored in the context of uncertainty aware learning. Also, existing approaches [46] 38
require output predictions from all other models, which is impractical for DCA with a combinatorial amount of model proposals. comparison, our proposed consistency enforcing loss efficient and synergizes well with DCA training.
Consistency vs. diversity While strengthening the predictive consistency can help the DCA pa- rameter training, has the risk of reducing the diversity of model predictions. Since model diversity has been found beneficial for ensemble approaches [43 45 29, the consistency enforcing loss has mixed effect on uncertainty estimation and could be advantageous in some cases. In practice (cf Section 5.3), we observe that the consistency enforcing loss boosts the predictive performance of DCWA models and improves the uncertainty estimation of modelwise DCA.
To further analyze the consistency enforcing loss, we empirically study its effect on the classification performance of individual DCA model proposals. The results collected in Appendix C We observe that the consistency enforcing loss improves the prediction results of individual models but reduces their diversity.
Experiments
To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conduct series of experiments: we benchmark the predictive performance of DCWA in Section [. and evaluate the uncertainty estimation of DCA models in Section Also, ablation studies on DCA variants are included in Section 5.3 to further analyze several design aspects. For all experiments we use a common preactivation ResNet-20 architecture [0] as the base network, and five independent runs are executed to produce the final results. Our implementation uses PyTorch [32 and can run on a single modern GPU with 10Gb VRAM.
5.1 Image classification using DCWA
Table 1: Image classification results CIFAR-IO and SVHN. DCWA has comparable performance w.r.t. SWA and simple to use. Both SWA and DCWA outperform the standard baseline training.
CIFAR-10
SVHN
Accuracy t
NLL
Accuracy
NLL +
Standard SWA
0.9264 + 0.0020
0.2507 = 0.0078
0.9590 + 0.0012
0.1786 + 0.0038
0.9327 = 0.0016
0.2167 = 0.0069
0.9610 + 0.0010
0.1809 + 0.0016
DCWA
0.9337 = 0.0022
0.2355 = 0.0101
0.9606 + 0.0024
0.1713 + 0.0060
5

---

Firstly, we evaluate the performance of the DCWA approach. For this we use CIFAR-10 22 and SVHN [iI] datasets and compare against the standard training and SWA [T6] as baselines. DCWA models are obtained from training layerwise DCA models with five copies and consistency enforcing loss used. SWA models are trained following the settings from Izmailov et al. [16]. Test accuracy
and negative log-likelihood (NLL) on both datasets are reported in Table [
We observe that DCWA and SWA have comparable results, and both of them outperform the standard training baseline. However, DCWA has the appealing practical advantages over SWA of not requiring custom learning rate scheduling nor readjustment of batch normalization layers.
5.2
DCA for uncertainty-awary learning
We now focus on comparing the uncertainty estimation of DCA models against a selection of practical uncertainty-awaat learning methods, including MC dropout (6) SWAG 288, and deep ensemble 223]
Also, results from the standard training of the base network are reported for reference. These baselines are compared with trunkwise" and modelwise DCA models. Following the ablation study results in Table 3
consistency enforcing loss is used with modelwise DCA models while standard negative ligllikel loss is used to train trunkwise DCA models for best performance.
To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain, distributional shift, and out-of-distribution (OOD) image classification problems. We consider common metrics for uncertainty quantification, including negative likelikeldoood (NLL), Brier score, and expected calibration error (ECE) ()] For OOD experiments, we follow the settings of Liang et al. [27] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error, AUPR- AUt,-out, and AUROC in Appendix [B.3
Overall, we observe that modelwise DCA produces the best results both in terms of predictive performance and uncertainty estimation. It is the method of choice if the performance of uncertainty- aware learning is the only concern. Trunkwise DCA is overall competitive with deep ensemble and morcover it produces more diverse model proposals. It offers a trade-off for more varied predictions.
In-domain comparison We start by training the baselines and DCA models for the standard in- domain image classification problems on the CIFAR-10 10 [22 and SVHN [3T datasets. For all training we use SGD with momentum 0.9. We use a drop rate of 0.1 for MC dropout, and for SWAG we follow the settings from Maddox et al. (28). Deep ensemble is performed on five separate base networks. And trunkwise and modelwise DCA models also use five copies. The results on CIFAR-10 are summarized in Table and SVHN results are Appendix B.T
Table 2: In-domain image classification results on CIFAR-10. In general, modelwise DCA has the best predictive performance and uncertainty estimation, while trunkwise DCA has slightly better results w.r.t. deep ensemble while producing a richer set of model proposals.
Accuracy +
NLL $
ECE
Brier
Standard
0.9264 = 0.0020
0.2507 0.0078
0.0301 = 0.0020
0.1132 = 0.0030
MC dropout
0.9281 + 0.0015
0.2186 + 0.0034
0.0121 + 0.0013
0.1057 + 0.0021
SWAG
0.9329 + 0.0018
0.2016 0.0045
0.0149 + 0.0023
0.0994 + 0.0020
Deep ensemble
0.9434 + 0.0017
0.1746 + 0.0028
0.0085 + 0.0011
0.0834 + 0.0013
Trunk. DCA
0.9454 = 0.0008
0.1676 + 0.0018
0.0080
= 0.0011
0.0809 = 0.0011
Model. DCA
0.9481 = 0.0008
0.1550 + 0.0008
0.0084 + 0.0009
0.0762 = 0.0007
In general, modelwise DCA obtains the best performance both in terms of classification accuracy and uncertainty estimation. Also, trunkwise DCA slightly outperforms deep ensemble in this case. SVHN results in Appendix B. also confirm these observations.
Distributional shiftt experiments ize the quality of uncertainty estimation, since they do not reflect the case when there is mismatch between the training data and the test set. This calls for additional distributional shift experiments,
In-domain experiments alone are not sufficient to fully character-
A trunk consists of (residual) block with a non- identity main branch together with subsequent residual blocks having identity main branch, following the graphical model interpretation of Shen and Cremers [36].

---

0.8
Accuracy
Standard
MC Dropout
SWAG
Deep ensemble
0.2
Trunkwise DCA
Severity
2.5
Standard
Negative log-lkkelinoo
MC Dropout
2.0
SWAG
ensemble
1.5
Trunkwise DCA
Modelwise DCA
1.0
0.5
0.0
Severity
Figure 2: Distributional shift image classification results on CIFAR-10-C with various degrees of corruption severities. Again, modelwsie DCA has the best overall performance. Severity corresponds to the in-domain case.
which gradually transform the in-domain test set to make increasingly different than the training data. In case, use the CIFAR-10- [12] dataset, which applies wide range of corruptions the original CIFAR-10 test set. CIFAR-10-C quantizes corruptions into five severity levels where higher levels generate stronger coruptiions and are more dissimilar to the in-domain test set. We use the models previously trained on CIFAR-10 in-domain training data, and evaluate their performance on CIFAR-IO-C separately for each severity level. Figure summarizes the results.
Similar to the in-domain scenario, modelwise DCA has the best results overall. Trunkwise DCA has comparable test accuracies w.r.t. deep ensemble. It tends outperform deep ensemble in terms of negative log- log-likelhood, however, it is worse in terms Brier score and ECE, as indicated by the additional plots in Appendix B.2
Out-of-diste distribution results Additionally, we complement the distributional shift experiments with out-of-distributtio image detection [13].
True positive rate (in-d omain data)
The goal is to distinguish in-domain test sam- ples from outliers that are drawn from a differ- ent dataset. Thus we have binary classifica- tion setting. For our case, we reuse again the models trained on in-domain CIFAR-10 10 training data. Samples from the CIFAR- 10 test set are re- garded as in-domain positive samples while test
0.8
0.6
Standard
MC Drop.
0.4
SWAG
Deep ens.
samples from SVHN are treated as negative out- liers. We show the ROC curves for DCA models and their baselines in Figure 3 Quantitative met- rics following Liang et al. (27). which include FPR95, detection error, AUPR-in, AUPR-out and AUROC, reported in Appendix [B.]
Trunk. DCA
0.2
Model. DCA
Ideal
0.0
Random
0.0
0.2
0.4
0.6
0.8
1.0
False positive rate (OOD data)
Again, we observe that modelwise DCA achieves the best OOD performance. Trunk- wise DCA the second-best performance and outperforms deep ensemble and SWAG, Inter-
Figure 3: ROC curves with CIFAR-10 as the in- domain dataset and SVHN as the OOD dataset. Modelwise DCA shows the best performance for OOD detection. followed by trunkwise DCA.

---

estingly, MC dropout turns out to perform poorly in the OOD case and is even worse than the standard training baseline. The quantitative results in Appendix B.3 agree with the ROC curve.
5.3 Ablation studies
We wrap up the experiment section with a series of ablation studies on variants DCA models. We consider three main aspects: the effect of granularity, the usage of consistency enforcing loss during training, and the influence of component instance count on DCA predictive results. We use the in-domain CIFAR-10 classification task to compare the performance of different DCA variants.
Effect of granularity To understand the effect of DCA component granularity on the performance of DCA models, we define five DCA model variants with different component granularities. This includes DCA models whose components are channels, layers, residual blocks, residual trunks, and entire models. All DCA model variants are constructed with five sets of component instances and trained with the consistency enforcing loss (NLL loss leads similar observations). We summarize the results of these DCA variants and their DCWA models Figure
10"
No negattive log looclhoood scale)
Expected callibaati ation ertor (log scale)
Standard
Deep
Deep ensemble
Deep ensemble
DCA
DCA
DCA
Error Accuracy, log
DCWA
10a
DCWA
DCWA
10-1
10-2
channel layer block trunk model
channel
layer
block
trunk
model
channel layer
block
trunk model
Figure 4: Classification errors (= 1-acccurcy, lower is better), NI.Ls, and ECEs of DCA (blue)
DCWA (orange) models with different component granularities on in-domain CIFAR-10 classification. As references, the results standard bascline training (dashed lines), as well as deep ensemble (dotted lines), are also provided. The metrics are all plotted in log scale.
Several interesting observations can be drawn from this experiment: For DCA models, coarser component granularity generally leads to better results. This agrees with the analysis Section 2.2 For DCWA models we observe a clear distinction between fine-grain (channel and layer component DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: fine-grain DCWA variants improve on the standard baseline, while coarse-grain grain DCWA variants result in considerably worse performance than the standard baseline. This also matches our analysis in Section 2.3 Interestingly, the DCWA model with block component is still able yield significantly better than this case, reparameterization mismatch only happens on the side branches of the residual blocks, and the trained weights on the main branch are still able to produce meaningful predictions despite the erroncous parameterization on the residual side branches.
random (i.e. 10%) accuracy.
Table 3: Comparison of DCA training using negative log-likelihood (NLL) loss and consistency enforcing loss (CEL) for (layerwise) DCWA, trunkwise DCA, and modelwise DCA models.
Accuracy 1
NLL 4
ECE +
DCWA (NLL)
0.9314 + 0.0017
0.2651 + 0.0066
0.0365 0.0014
DCWA (CEL)
0.9337 + 0.0022
0.2355 + 0.0101
0.0301 + 0.0014
Trunkwise DCA (NLL)
0.9454 = 0.0008
0.1676+0.000
0.0080
0.0011
Trunkwise DCA (CEL)
0.9441 + 0.0020
0.1728 +
0.0084 = 0.0017
Modelwise DCA (NLL)
0.9495 0.0008
0.1601 = 0.0006
0.0107 0.0004
Modelwise DCA (CEL)
0.9481 + 0.0008
0.1550 + 0.0008
0.0084 + 0.0009
analyze the influence of consistency enforcing loss for DCA train- ing, compare DCWA, trunkwise DCA, and modelwise DCA models which are trained with
Consistency enforcing loss

---

negative logllikel loss with those trained with consistency enforcing loss. We summarize the results on CIFAR-10 10 in T
We observe that the consistency enforcing loss improves the overall performance of the DCWA model. It is also beneficial for the uncertainty estimation of the modelwise DCA model, both in terms of NLL and ECE. On the other hand, negative log-likelihood loss is the better choice for the trunkwise DCA model. In Appendix B.4 we provide additional results for other DCA variants.
Number of component instances Finally, we investigate the influence of DCA instance count. To properly train all the DCA component in- stances jointly, we find out that it is beneficial to multiply the training epochs and per-minibatch gradient backpropagations accordingly. For ex- ample, in our experiments, we schedule 200 epochs for standard baseline training. Then, for training a DCA model with three sets of com- ponent instances use 600 epochs, and accu- mulate for each minibatch an average gradient randomly se- lected DCA proposals for parameter update. We train series of layerwise DCA models with two, three, four, and five sets of component instances, and collect their results in Figure
0.936
0.27
DCA
Negattive log likellhood (dached lime)
DCWA
Standard
Accuracy (solid line)
0.932
0.25
0.928
0.23
0.926
0.22
three backpropagations
0.924
0.21
DCA component instance
Figure 5: Accuracy (solid lines) and negative log- likelihood (dashed lines) of layerwise DCA models (blue) and their corresponding DCWA models (or- ange) with two, three, four and five sets of DCA components. The results standard baseline train- ing (green) are also provided for reference.
In general, we observe that more DCA instances lead to better results (higher accuracy and lower NLL loss in Figure 5). Therefore, DCA pro- vides a reasonable trade-off between the com- putational footprint and the predictive power. Moreover, see that minimal DCA model with two sets of components can already significantly improve the performance compared to the standard base network training case.
6 Discussion
In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that has also simple setup and, unlike the choice of dropout for MC dropout or the custom learning rate scheduling of SWAG, requires no elaborate hyperparameter search. We find that coarse-grain DCAs are well suited for uncertainty-awarn learning: modelwise DCA consistently outperforms all other uncertainty- trunkwise DCA produces more diverse predictions while still delivering competitive performance w.r.t. deep ensemble, which is the best among existing baselines. Additionally, we discover that fine-grain DCA can produce a DCWA parameterization for the base network. DCWA improves predictive performance compared to the standard baseline training. It has on-par performance with SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers. Thus DCWA can be attractive alternative to SWA that is easy to implement and deploy.
learning methods for in in-domain, distributional shift, and OOD scenarios;
Similar deep ensemble, DCA requires multiple times the computational budget to perform the joint training, compared to the standard baseline. While in this work we focused on the predictive power, we plan to improve the efficiency of DCA future work. Apart from this, can also be interesting to search for variant that combines the benefits of both the fine-grain and coarse-gras -rain DCAs, as currently DCWA is only possible for fine-grain DCAs, while good uncertainty estimations can only be attained via coarse-grain DCAs instead. Additionally, it is possible to combine DCA with other uncertainty-awar methods, which might further improve the performance. We plan further investigate these aspects in future work.

---

Acknowledgments and Disclosure of Funding
This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC Advanced Grant SIMULACRON. The authors would like to thank Nikita Araslanov for proofreading and helpful discussions, as well the anonymous reviewers and area chair for their constructive feedback.
References
[1] A. Blum, N. Haghtalab, and A. D. Procaccia. Variational dropout and the local reparameterizatioo trick. In NIPS, 2015.
[2] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In ICML, 2015.
[3] T. Chen, E. B. Fox, and C. Guestrin. Stochastic gradient hamiltonian monte carlo. In ICML, 2014.
[4] S. Farquhar. M. A. Osborne, and Y. Gal. Radial bayesian neural networks: Beyond discrete support in large-scale bayesian deep learning. In AISTATS, 2020.
[5] A. Filos, S. Farquhar, A. N. Gomez, T. G. Rudner, Z. Kenton, L. Smith, M. Alizadeh, A. De Kroon, and Y. Gal. systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks. arXiv preprint arXiv: 10488, 2019.
[6] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. JCML, 2016.
T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of duns. NeurlPS, 2018.
A. Graves. Practical variational inference for neural networks. In NIPS, 2011.
[9] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks, In ICML,, 2017.
[10] F. K. Gustafsson, M. Danelljan, and T. B. Schön. Evaluating scalable bayesian deep learning methods for robust computer vision. CVPR Workshop, 2020.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016.
[12] D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[13] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and dusto--distribution examples in neural networks. In ICLR, 2017.
[14] J. M. Hemandez-Lobato and R. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In ICML, 2015.
[15] G. Huang, Y. Li. G. Pleiss, Z. Liu, J. E. Hoperoft, and K. Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In ICLR, 2017.
[16] P. Izmailov, D. Podoprikhin, T. Garipow, D. P. Vetrow, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018.
[17] A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NIPS, 2017.
[18] M. F. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava. Fast and scalable bayesian deep learning peitht-nerturbaation adam. In JCML. 2018.
[19] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In JCLR, 2014.
[20] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, al. Overcoming catastrophic forgetting in neural networks. PNAS,
T. Ramalho, A.
2017.
[21] D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. MIT Press, 2009.
10

---

[22] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report. University of Toronto, 2009.
[23] B. Lakshminarayanan, A. Pritzel, and C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
[24] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521((553):436 444, 2015.
[25] S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra. Why heads better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:/51..000 1.06114 2015.
[26] C. Leibig, V. Allken. M. S. Ayhan, P. Berens, and S. Wahl. Leveraging uncertainty information from deep neural networks for disease detection. Nature Scientific reports, 7(1):1-14, 2017.
[27] S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distrihution image detection in neural networks. In ICIRR, 2018.
[28] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for bayesian uncertainty in deep learning. In NewrIPS, 2019.
[29] A. Masegosa. Learning under model misspecification: Applications variational and ensemble methods. In NeurlPS, 2020.
[30] D. Molchanov, A. Ashukha, and D. P. Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017.
[31] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
[32] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In NewrIPS, 2019.
[33] H. Ritter, A. Botev, and D. Barber. A scalable laplace approximation for neural networks. In ICLR, 2018.
[34] Sensoy, M. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurlPS, 2018.
[35] B. Settles. Active learning literature survey. Technical report, University of Wisconsin Madison, 2009.
[36] Y. Shen and D. Cremers. A chain graph interpretation of real-world neural networks. arXiv preprint arXiv:2006. 16856, 2020.
[37] J. Snock, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. V. Dillon, J. Ren, and Z. Nado, Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In NeurIPS, 2019.
[38] G. Song and W. Chai. Collahorative learning for deep neural networks. In NewrIPS, 2018.
[39] C. Tomani, S. Gruber,
Erdem, Cremers, and Buettner. Post-hoc uncertainty calibration for domain drift scenarios. In CVPR, 2021.
[40] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In JCML, 2011 1.
[41] Y. Wen, P. Vicol, J. Ba, D. Tran, and R. B. Grosse. Flipout: Efficient pscudo-independent weight perturbations mini- batche.. In ICLR, 2018
[42] Y. Wen, Tran, and Ba. Batchensemble; an alternative approach to efficient ensemble and lifelong learning. In ICLR, 2020.
[43] F. Wenzel, Snock, Tran, and Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In NewrIPS, 2020.
[44] A. G. Wilson and Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In NewrIPS, 2020.
[45] W. Zhang. J. Jiang, Y. Shao, and B. Cui. Efficient diversity-driver ensemble for deep neural networks. In ICDE, 2020.
[46] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu. Deep mutual learning. In CVPR, 2018.

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "XFqeQD2AkoR", "reviewer": "Reviewer_dyii", "summary": "This paper proposes an effective ensembling technique for deep neural networks. The key idea is sharing the intermediate representation spaces among ensemble components. If we interpret deep neural networks as a chain of non-linear transformations mapping one representation space into another, this paper attempts to ensemble such intermediate mappings (DCA; Deep Combinatorial Aggregation). Furthermore, the authors suggest a weight averaging strategy compressing the proposed combinatorial ensemble to reduce the inference cost (DCWA; Deep Combinatorial Weight Averaging).\n\n* * *\nPOST-REBUTTAL EDIT: Thank you for your time on the rebuttal. One remaining concern is the lack of large-scale experiments. Nevertheless, I believe the presented results (i.e., outperforming deep ensembles for in-domain, common corruptions, and out-of-distribution settings) are sufficient to verify the proposed method can benefit the community. Accordingly, I raised my score to 6.", "questions": "\n- How are the weights initialized for DCA and DCWA? In the case of DCWA, ensemble components should have nearly identical weight values to be averaged, and the same should probably go for the initial weights.\n- In Table 3, why does CEL improve Modelwise DCA? It seems counter-intuitive; CEL reduces the predictive diversity, which can worsen the ensemble performance (as stated in lines 188-194). Perhaps CEL improves the performance of each individual ensemble member while decreasing the diversity among ensemble members. To check this, why don't we measure the classification performance of each individual?\n- I also suggest clarifying that the improvement of Consistency Enforcing Loss (CEL) is distinct from regularization effects due to the smoothed label; the objective of CEL, defined in Equation 4, can be interpreted as Knowledge Distillation or Label Smoothing [Yuan et al., 2021]. I believe that further comparison with \"NLL + Label Smoothing\" will enhance the arguments in lines 160-178 and 295-302.\n- Experimental evidence on large-scale datasets (e.g., ResNet-50 on ImageNet-1k) will be strong support for the paper, but I know that there are computational constraints to do so. One compromise is to start DCA (or DCWA) training from the pre-trained checkpoint (e.g., provided by torchvision); SWA [Izamilov et al., 2018] and SWA-Gaussian [Maddox et al., 2019] also provided ImageNet results in this way.\n\n1. Yuan et al., Revisiting Knowledge Distillation via Label Smoothing Regularization, 2021.\n2. Izmailov et al., Averaging Weights Leads to Wider Optima and Better Generalization, 2018.\n3. Maddox et al., A Simple Baseline for Bayesian Uncertainty in Deep Learning, 2019. ", "limitations": "This paper described the limitations of the work in Section 6; “Similar to deep ensemble, DCA requires multiple times the computational butdge to perform the joint training, compared to the standard baseline”.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is noteworthy that the proposed method explicitly verifies the benefits of ensembling intermediate mappings of deep neural networks. Although some of the existing Bayesian Neural Networks (BNNs) also can be interpreted in this way, further analysis of granularity provided in the paper makes distinct contributions.\n\nI especially like the variety of experiments provided in the paper, including in-domain classification performance, robustness on common corruptions, and predictive uncertainty on out-of-distribution examples. Such experimental results clarify the benefits of the proposed ensembling method.\n\nNevertheless, there isn’t enough experimental evidence to fully accept the superiority of the proposed DCA and DCWA methods. Experiments for PreResNet-20 on CIFAR-10 and SVHN, provided in the paper, are limited in terms of scalability and generality. It is necessary to secure diversity in datasets and network structures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "58Ctg7ZzHZz", "reviewer": "Reviewer_GFCL", "summary": "The paper proposes a new method called Deep Combinatorial Aggregation (DCA) which is a family of approaches for ensembling deep neural network models, often with weight-sharing. DCA trains multiple different versions of each neural network \"component\" (with different initializations), then samples from all possible combinations of the components at test-time to generate an empirical distribution of predictions. Among the \"components\" considered by the paper are layers, trunks (i.e., residual blocks), and complete models. The paper shows that the uncertainty estimates generated by DCA are comparable, or better than, existing methods such as MC-Dropout across CIFAR-10 and SVHN datasets, for in-domain, distribution shift, and out-of-distribution settings. The authors measure uncertainty via negative log-likelihood (NLL), Brier score, and ECE. Furthermore, the paper also proposes a method called deep combinatorial weight averaging (DCWA) for merging the models from a DCA ensemble into a single model.", "questions": "**Q1. Would it be appropriate to consider DCA to be generalization of MC-Dropout?** \\\nSpecifically, it seems to me that neuronwise DCA is essentially MC-Dropout. Is my understanding correct?\n\n**Q2. How is modelwise DCA different from deep ensemble?** \\\nThe paper writes:\n> Note that modelwise DCA is similar to deep ensemble, except that its model copies are trained with distinct data within each training epoch, ...\n\nDoes \"its\" refer to modelwise DCA or deep ensemble? In either case, I don't fully understand what you mean by \"model copies are trained with distinct data within each training epoch.\"\n\n**Q3. Why do the in-domain results not include layerwise DCA models?**", "limitations": "My main concern about DCA's limitations is how it scales to larger datasets/models and other data modalities. The authors do not seem to discuss this limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**S1. DCA generalizes the notion of using stochastic networks for predictive uncertainty estimates** \\\nAssuming my understanding to be correct (see Q1), DCA generalizes MC-Dropout to other stochastic model components, including layers, trunks, and even entire models. Furthermore, like dropout, DCA proposes a method (DWCA) to merge the individual models into a single model. This perspective is helpful in understanding ways to leverage model stochasticity for empirical uncertainty quantification.\n\n**S2. Experiments show strength of DCA** \\\nThe reasonably comprehensive set of experiments show that DCA outperforms (or at least matches) the performance of existing methods such as MC-Dropout and deep ensemble.\n\n## Weaknesses\n\n**W1. Unclear justification for consistency enforcing loss (CEL)** \\\nWhile the experimental evidence suggests that CEL is helpful, I struggle to understand why such an approach should be beneficial. It seems to go against the intuition that to get good uncertainty estimates, one should train diverse and independent models, instead of explicitly encouraging their predictions to be similar (which also breaks independence).\n\n**W2. Lacking clarity in OOD experiments** \\\nHow are the ROC curves in Figure 3 generated? Namely, how are you using the models trained on multi-class classification to perform outlier detection (binary classification)? Some more description is warranted.\n\n**W3. Missing downstream tasks** \\\nWhile the results on uncertainty estimates are impressive, I would hope to see a discussion of how much the gain in uncertainty estimation performance affects downstream tasks (e.g., active learning or Bayesian optimization).\n\n**W4. Missing discussion of scalability** \\\nHow does the DCA approach scale to larger datasets or other data modalities? Would similar approaches work on, e.g., transformer models in NLP tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "31OYc9RRMf3", "reviewer": "Reviewer_tdmd", "summary": "* **(1)** The authors introduce the deep combinatorial aggregation method, which could produce exponential ensemble model proposals by combining different parts of two neural networks.\n* **(2)** A weight averaging strategy called DCWA is proposed to produce a new average model.\n* **(3)** A consistency enforcing loss function for DCA is proposed to promote the similarity of different ensemble proposals.\n\n", "questions": "**However, I still have the following concerns and confusions:**\n* **(Concerns 1)** Compared to other ensemble methods such as MCDropout [5] and SWA, the efficiency and complexity of this training method may increase significantly. Meanwhile, MCDropout[5] and other related works also could provide a variety of models without any additional computation cost. Compared with them, the method seems to have no advantages. \n* **(Concerns 2)** Since only one model proposal is trained per iteration, I am concerned that the model is hard to converge when the number of ensemble proposals is large. Meanwhile, when the number of model proposals is large, the training time may increase significantly.\n* **(Concerns 3)** As elaborated in Section 2.2, the model parameters are not decomposable in general. Therefore, a coarser multilayer trick is employed. However, this trick will reduce the number of model proposals. \n* **(Concerns 4)** To help the DCA parameter training, predictive consistency is employed. However, as illustrated in Section 4, it has the risk of reducing the diversity of the model predictions.", "limitations": "This paper studies ensemble learning, a classic machine learning algorithm with almost no potential negative social impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* **(Strengths 1)** The key motivation of this paper is similar and clear. \n* **(Strengths 2)** The paper is well-organized and easy to follow.\n* **(Strengths 3)** The potential limitations of the method have been well illustrated in the paper.\n\n**However, I still have the following concerns and confusions:**\n* **(Concerns 1)** Compared to other ensemble methods such as MCDropout [5] and SWA, the efficiency and complexity of this training method may increase significantly. Meanwhile, MCDropout[5] and other related works also could provide a variety of models without any additional computation cost. Compared with them, the method seems to have no advantages. \n* **(Concerns 2)** Since only one model proposal is trained per iteration, I am concerned that the model is hard to converge when the number of ensemble proposals is large. Meanwhile, when the number of model proposals is large, the training time may increase significantly.\n* **(Concerns 3)** As elaborated in Section 2.2, the model parameters are not decomposable in general. Therefore, a coarser multilayer trick is employed. However, this trick will reduce the number of model proposals. \n* **(Concerns 4)** To help the DCA parameter training, predictive consistency is employed. However, as illustrated in Section 4, it has the risk of reducing the diversity of the model predictions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept