Packed Ensembles for efficient uncertainty estimation
Authors: Olivier Laurent
Adrien Lafage
Enzo Tartaglione
Geoffrey Daniel
Jean-marc Martinez
Andrei Bursuc
Gianni Franchi
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Efficient Ensembling
Uncertainty Quantification
OOD Detection

Abstract: Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties. We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds. PE is designed to operate within the memory limits of a standard neural network. Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity, and performs equally well in terms of accuracy, calibration, out-of-distribution detection, and robustness to distribution shift. We make our code available at https://github.com/ENSTA-U2IS/torch-uncertainty.

Published as a conference paper at ICLR 2023
PACKED-ENSEMBLES FOR EFFICIENT UNCERTAINTY ESTIMATION
Olivier Laurent, 1.2." Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel,' Jean-Marc Martinez,' Andrei Bursuc* & Gianni Franchi
Université Paris-Saclay, CEA, SGLS, U2IS, ENSTA Paris, Institut Polytechnique de Paris, LTCI, Télécom Paris, Institut Polytechnique de Paris, valeo.ai"
ABSTRACT
Deep Ensembles (DE) are a prominent approach for achieving excellent perfor- mance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world sys- tems constrain smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties. We introduce Packed-Ensemne (PE), strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolu- tions to parallelize the ensemble into single shared backbone and forward pass improve training and inference speeds. PE is designed to operate within the mem- ory limits of standard neural network. Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity, and performs equally well in terms of accuracy, calibration, out-of-distributio detection, and robust- ness to distribution shift. We make our code available at github.com/ENSTA. U2IS/torch wnwprrcicnnnettttt....
Packed Eroembles (2. 4,2)
81
Packed Ensembles (2.4.1)
Accuracy (%)
Deep Ensembles (x+)
MIMO (4)
79-
10M
20M
90M
Single
Images/sec (x10*)
Figure 1: Evaluation of computation cost performance trade-offs for multiple uncertainty quantifica- tion techniques CIFAR-100. The y-axis and x-axis respectively show the accuracy and inference time images per second. The circle area proportional to the number of parameters. Optimal approaches are closer to the top-right corner. Packed-Ensembles, strikes a good balance between predictive performance and speed.
INTRODUCTION
Real-world safety-critical machine learning decision systems such as autonomous driving (Levin- al., 2011; McAllister et al.. 2017) impose exceptionally high reliability and performance requirements across broad range of metrics: accuracy, calibration, robustness to distribution shifts, uncertainty estimation, and computational efficiency under limited hardware resources. Despite sig- nificant improvements performance in recent years, vanilla Deep Neural Networks (DNNs) still
cqual contribution
corresponding author gianni.franchi franchi @ensta-paris.fr

---

Published as a conference paper at ICLR 2023
exhibit several shortcomings, notably overconfidence in both correct and wrong predictions (Nguyen et al., 2015; Guo et al., 2017; Hein et al., 2019). Deep Ensembles (Lakshminarayanaa et al., 2017) have emerged prominent approach to address these challenges by leveraging predictions from multiple high-capacity neural networks. By averaging predictions or by voting, DE achieves high ac- euracy and robustness since potentially unreliable predictions exposed via the disagreement be- tween individuals. Thanks to the simplicity and effectiveness of the ensembling strategy (Dietterich, 2000), DE have become widely used and dominate performance across various benchmarks (Ovadia et al., 2019; Gustafsson et al., 2020).
DE meet most of the real-world application requirements except computational efficiency. Specif- ically, DE are computationally demanding in terms of memory storage, number of operations, and inference time during both training and testing, as their costs grow linearly with the number of individuals. Their computational costs are, therefore, prohibitive under tight hardware constraints.
This limitation of DE has inspired numerous approaches proposing computationally efficient al- ternatives: multi-head networks (Lee et al., 2015; Chen & Shrivastava, 2020), ensemble-imitaiing layers (Wen et al.. 2019 Havasi et al., 2020; Ramé et al., 2021), multiple forwards on different weight subsets of the same network (Gal & Ghahramani, 2016; Durasov et al., 2021), ensembles of smaller networks (Kondratyuk et al., 2020; Lobacheva et al., 2020), computing ensembles from (Huang et al., 2017; Garipov et al., 2018), and efficient Bayesian Neural Net- works (Maddox et al., 2019; Franchi et al., 2020). These approaches typically improve storage usage, train cost, or inference time at the cost of lower accuracy and diversity in the predictions.
single training
An essential property of ensembles to improve predictive uncertainty estimation is related to the diversity of its predictions. Perrone & Cooper (1992) show that the independence of individuals is critical to the success of ensembling. Fort et al. (2019) argue that the diversity of DE, due to random- ness from weight initialization, data augmentation and batching, and stochastic gradient updates, is superior to other efficient ensembling alternatives, despite their predictive performance boosts. Few approaches manage to mirror this property of DE in computationally efficient manner close single DNN (in terms of memory usage, number of forward passes, and image throughput).
In this work, we aim design DNN architecture that closely mimies properties of ensembles, in particular, having a set of independent networks, in a computationally efficient manner. Previous works propose ensembles composed of small models (Kondratyuk et al., 2020; Lobacheva et al., 2020) and achieve performances comparable to single large model. We build upon this idea and devise a strategy based on small networks trying to match the performance of an ensemble of large networks. To this end, we leverage grouped convolutions (Krizhevsky et al., 2012) to delincate multiple subnetworks within the same network. The parameters of each subnetwork are not shared across subnetworks, leading to independent smaller models. This method enables fast training and inference times while predictive uncertainty quantification is close to DE (Figure 1).
In summary, contributions are the following:
We propose Pucked-Ensemblle (PE), efficient ensembling architecture relying on grouped convolutions, as a formalization of structured sparsity for Deep Ensembles;
We extensively evaluate PE regarding accuracy, calibration, OOD detection, and distribu- tion shift on classification and regression tasks. We show that PE achieves state-of-the-art predictive uncertainty quantification.
We thoroughly study and discuss the properties of PE (diversity, sparsity, stability, behavior of subnetworks) and release our PyTorch implementation.
2 BACKGROUND
In this section, we present the formalism for this work and offer brief background on grouped convolutions and ensembles of DNNs. Appendix summarizes the main notations in Table 3.
2.1 BACKGROUND ON CONVOLUTIONS
The convolutional layer (LeCun et al., 1989) consists of series of cross-correlations between fea- ture maps h E HC, wW, regrouped in batches of size B and weight tensor wi E RRC++*x,xx5;
2

---

Published as a conference paper at ICLR 2023
x1
x
x
M 3
Y
width
width
width
fo
fox
f0z
S0a
fez
f0z
foz.
8
a)
b)
C)
Figure Overview the considered architectures: (left) baseline vanilla network; (center) Deep M = 7 2).
Ensembles; (right)
with Cj. Hj, W three integers representing the number of channels, the height and the width of h respectively. Cj-1 and 8j are also two integers corresponding to the number of channels of hi+1 (the output of the layer) and the kernel size. Finally, the layer's index and will be fixed in the following formulae. The bias of convolution layers will be omitted in the following for simplicity. Hence the output value of the convolution layer, denoted 88 is:
z' (t,1).,;.)) (h3 w (c.;;))
Σ (c,kk,)))) he (k,
where e E [0, C++1 - 1] is the index of the considered channel of the output feature map, the classical 2D cross-correlation operator, and z is the pre-activation feature map such that h ((z)) with 6 an activation function.
To embed an ensemble of subnetworks, we leverage grouped convolutions, already used in ResNeXt (Xie et al., 2017) to train several DNN branches in parallel. The grouped convolution operation with y groups and weights w', E JR S xet is given in (2), y dividing Cj for all lay- ers. Any output channel e is produced by specific group (set of filters), identified by the integer
which only uses 1 of the input channels:
7
21+1 (c, = (h' (c,
Σ w k;)) + h³
7C
Cj γ ..:
C++1
The grouped convolution layer is mathematically equivalent to classical convolution where the weights are multiplied element-wise by the binary tensor mask,, E {0, 1] C2+1 XC,xx; xx such that mask (k: :,;;)
2
7/ Cs++
m for each group m [0, 1]. The complete
layer mask is finally defined as mask'
Σ
mask/ and the grouped convolution can therefore be
m==
rewritten as x1+1 h (wo mask)), ), where is the Hadamard product.
2.2 BACKGROUND ON DEEP ENSEMBLES
For an image classification problem, let define dataset D {x,, |P] containing [D] pairs of samples x, hē E IRCoxHoxMy and one-hot-encoded labels y; RRN modeled as the realization of a joint distribution P(x,Y) where Ne is the number of classes in the dataset. The input data x; is processed via neural network fe which is parametric probabilistic model such that ý: fo(x,) = P(Y y; = x; 0). This approach consists in considering the prediction ý, as parameters of Multinoulli distribution.
3

---

Published as a conference paper at ICLR 2023
X
X
X
X
X
X
X
3,3x33112
3,3x3,12
3,3x3,12
Rearrange
3,3x3,36
3,3x3,36
36,383,72
24,
12,3x3,22 24
12,3x3,24
groups
groups
36,4x4,72
Av. Pool
Av. Pool
Av.
Av.
groups 3
24.1x1.48
24 1x1,48
24 1x1.48
Ax Pool
72,1x1,144
groups
18
48 1x1,10
48 1x
72,1x1,144
144,1x1, x1,30
groups
=
144,1x1,30
equivalent
architectures
b)
Figure 3: Equivalent architectures for -nnkeeebs.ssms.es.... (a) corresponds to the first sequential
version, (b) to the version with the rearrange operation and grouped convolutions and (c) to the final version beginning with full convolution.
To improve the quality of both predictions and estimated uncertainties, as well as the detection of OOD samples, Lakshminarayanan et al. (2017) propose ensemble M randomly initialized DNNs as large predictor called Deep Ensembles. These ensembles can be seen as a discrete approximation of the intractable Bayesian marginalization on the weights, according Wilson & Izmailov (2020). If note {0m m=o the of trained weights for the M DNNs, Deep Ensembles consists in averaging the predictions of these M DNNs as in equation (3).
1 M--1
P(y:)x;, =
Σ m==
Plys 0m)
M
PACKEID-ENSEMBLES
This section describes how to train multiple subnetworks using grouped convolution efficiently. Then, we explain how our new architectures are equivalent to training several networks in parallel.
3.1 REVISITING DEEP ENSEMBLES
Although Deep Ensembles provide undisputed benefits, they also come with significant draw- back that the training time and the memory usage in inference increase linearly with the number of networks. To alleviate these problems, we propose assembling small subnetworks, which are essen- tially DNNs with fewer parameters. Moreover, while ensembles to this day have mostly been trained sequentially, we suggest leveraging grouped convolutions to massively accelerate their training and inference computations thanks to their smaller size. The propagation of grouped convolutions with M groups, MM being the number of subnetworks in the ensemble, ensures that the subnetworks are trained independently while dividing their encoding dimension by factor M. More details on the usefulness of grouped convolutions to train ensembles can be found in subsection 3.3.
To create Packed- Bnakeebes (illustrated in Figure 2), we build on small subnetworks but compen- sate for the dramatic decrease of the model capacity by multiplying the width by the hyperparameter a, which can be seen as an expansion factor. Hence, we propose Packed Ensembles-(cr M, as flexible formalization of ensembles of small subnetworks. For an ensemble of M subnetworks, Docked-Ensembled (a, M, 1) therefore modifies the encoding dimension by a factor 37 and the in- ference of our ensemble is computed with the following formula, omitting the index i of the sample:
a
M Σ
(0, with 0a.m {wo mask bi,
(4)
M
where who is the weight of the layer of dimension (aCj+1) (aC;)

---

Published as a conference paper at ICLR 2023
m=1
0
:
:
1
0
0
m
1
(b) M 2,7 Figure 4: Diagram representation of a subnetwork mask: mask?, with M = 2, an integer corre- sponding to a fully connected layer
(a) M 2,
In the following, we introduce another hyperparameter y corresponding the number of groups of each subnetwork of the Enseedde, creating another level of sparsity. These groups are also called "subgroups" and are applied to the different subnetworks. Formally, we denote our technique Packed- Ensembles-(G M. with the hyperparameters in the parentheses. In this work, we consider constant number of subgroups across the layers; therefore, 7 divides aC, for all j.
3.2 COMPUTATIONAL COST
For a convolutional layer, the number of parameters involving Cj input channels, Cj+1 output chan- nels, kernels of size 8j and subgroups is equal to M x
aC, aC 8 M
A8
The same formula applies dense layers as 1 convolutions. Two cases emerge when the architectures of the subnetworks are fully convolutional or dense. If a VM, the number of parameters in the ensemble equals the number of parameters in single model. With a = M, each subnetwork corresponds to single model (and their ensemble is therefore equivalent in size to DE).
3.3 IMPLEMENTATION DETAILS
We propose simple way of designing efficient ensemble convolutional layers using grouped con- volutions. To take advantage of the parallelization capabilities of GPUs in training and inference, we replace the sequential training architecture, (a) in Figure 3, with the parallel implementations (b) and (c). Figure summarizes different equivalent architectures for a simple ensemble of M DNNs with three convolutional layers and final dense layer (equivalent to a 1 x convolution)
with a = =
In (b), we propose to stack the feature maps on the channel dimension (denoted as the rearrange operation). 1 This yields a feature map h', of size M x C2 x H, x W, regrouped by batches of B with B the batch size of the ensemble. One solution to keep the same batch size
size only Tf to repeat the batch M times that its size equals B after the rearrangement. Using convolutions with M groups and 7 subgroups per subnetwork, each feature map is convoluted separately by each subnetwork and yields its own independent output. Grouped convolutions propagated until the end to ensure that gradients stay independent between subnetworks. Other operations, such as Batch Normalization (loffe & Szegedy, 2015), can be applied directly long as they can be grouped or have independent actions on each channel. Figure 4a illustrates the mask used to code Packed- Ensembles in the case where M 2. Similarly, Figure 4b shows the mask with M 2 and y
Finally, (b) and (c) are also equivalent. is indeed possible to replace the rearrange operation and the first grouped convolution with a standard convolution if the same images are to be provided simultaneously to all the subnetworks. We confirm in Appendix F that this procedure is not detri- mental the ensemble's 's performance, and we take advantage of this property to provide this final optimization and simplification.
See htaes
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
To validate the performance of our method, conduct experiments classification tasks and measure the influence of the parameters a and y. Regression tasks are detailed in Appendix N.
4.1 DATASETS AND ARCHITECTURES
First, we demonstrate the efficiency of Packed-Ensembles 100 (Krizhevsky, 2009), showing how the method adapts to tasks of different complexities. As we propose replace single model architecture with several subnetworks, we study the behavior of PE on various sizes architectures: ResNet-18, ResNet-50 (He et al., 2016), and Wide ResNet28- 10 (Zagoruyko & Komodakis, 2016). We compare against Deep Ensembles (Laaksmiinarayanan et al., 2017) and three other approximated ensembles from the literature: BatchEnsemble (Wen et al., 2019), MIMO (Havasi et al., 2020), and Masksembles (Durasov et al., 2021).
CIFAR-10 and CIFAR-
Second, we report our results for Packed-Ensemble on ImageNet (Deng et al., 2009), which we compare against all baselines. We run experiments with ResNet-50 and ResNet-50x4. All training runs are started from scratch.
4.1.1 METRICS, OOD DATASETS, AND IMPLEMENTATION
We evaluate the overall performance of the models in classification tasks using the accuracy (Acc) in % and the Negative Log- Likelihood (NLL). We choose the classical Expected Calibration Error (ECE) (Naeini et al., 2015) for the calibration of uncertainties and measure the quality of the OOD detection using the Areas Under the Precision/Recall curve (AUPR) and Under the operating Curve (AUC), as well as the False Positive Rate at 95% recall (FPR95), all expressed in %,, similarly to Hendrycks Gimpel (2017).
We use accuracy the validation criterion (i.e., the final trained model is the one with the highest accuracy). During inference, we average the softmax probabilities of all subnetworks and consider the index of the maximum of the output vector to be the predicted class of the ensemble. We define the prediction confidence as this maximum value (also called maximum softmax probability).
For OOD detection tasks CIFAR-10 and CIFAR-100, use the SVHN dataset (Netzer et al., 201 as an out-of-distribution dataset and transform the initial classification problem into a binary classification between in-distribution and OOD data using the maximum softmax probability as criterion. We discuss the different OOD criteria in appendix distribution datasets: ImageNet-O (Hendrycks et al., 2021b) and Texture (Wang al., 2022), and use the Mutual Information (MI) as a criterion for the ensembles techniques (see Appendix E for details on MI) and the maximum softmax probability for the single model and MIMO. To measure the robustness under distribution shift, we use ImageNet-R (Hendrycks et al., 202 la) and evaluate the Accuracy, ECE, and NLL, denoted rAcc, rECE, and rNLL on this dataset, respectively.
For ImageNet, we use two out-of-
We implement our models using the PyTorch-Lightning framework built on top of PyTorch. Both are open-source Python frameworks. Appendix and Table 4 detail the our experiments across architectures and datasets. Most training instances are completed on single Nvidia RTX 3090 except for ImageNet, for which we use 2 to Nvidia A A100-80GB.
used in
4.1.2 RESULTS
Table presents the average performance for the classification task over five runs using the hyper- parameters in Table 4. We demonstrate that Baecked-Ensembles. in the setting of a = and = 2, yields similar results to Deep Ensembles while having lower memory cost than single model. For CIFAR-10, the relative performance of PE compared to DE appears to increase as the original architecture becomes larger. When using ResNet-18. 18, nacked-Ensemble matches Deep Ensembles on OOD detection metrics but shows slightly worse performance on the others. However, using ResNet-50, both models seem to perform similarly, and PE slightly outperforms DE classification performance with WideResNet28-10
Note that the benchmark uncertainty- baselines only uses ECE measure calibration

---

Published as a conference paper at ICLR 2023
Table Performance comparison (averaged over five runs) on CIEAR-10/100 using ResNet-18 (R18), ResNet-50 (R50), and Wide ResNet28-10 (WR) architectures. All ensembles have M = subnetworks, we highlight the best performances bold. For our method, we consider = ? 2, except for WR C100, where y = 1. Mull-Adds corresponds to the inference cost, i.e., the number of Giga multiply-add operations for a forward pass which estimated with Torchinfo (2022).
Method
Data
Acc 1
NLL J
ECE
AUPR
AUC
FPR95
Params (M)
Mult-Adds
Single Model
C10
94.0
0.035
94.0
89.7
33.8
0.56
BaichEnneeble
C10
R18
92.9
0.257
0.031
92.4
87.8
32.1
11.21
2.22
MIMO (p
94.0
94.4
Maskssemlees
C10
R18
94.0
0.188
0.009
93.6
89.5
27.8
11.24
2.22
Packed-Ensemblee
C10
94.3
0.178
0,007
94.7
91.3
23.2
8.18
0.48
Deep Ensembles
C10
RI8
95.1
0.156
0.008
94.7
91.3
18,0
44.70
2.22
Single Model
C10
R50
95.1
0.211
0.031
95.2
91.9
23.6
23.52
1.30
BaitchEnsemle
MIMO (p
C10
R50
95.4
0.197
0.030
95.1
90.8
26.0
23.59
1.30
Masksembles
95.7
Fasked-Fneemlles
C10
R50
95.9
0.137
0.008
97.3
95.2
14.4
14.55
1,00
Deep Ensembles
C10
96.0
0.136
0.008
97.0
94.7
94.08
5.19
Single Model
C10
WR
95.4
0.200
0.029
96.1
93.2
20.4
36.49
5.95
BatchEnsemble
C10
WR
95.6
0.206
0.027
95.5
92.5
22.1
36.59
23.81
MIMO
C10
94.7
0.034
90.6
36.51
Masksembles
C10
WR
94.0
0.186
0.016
97.2
95.0
14.5
36.53
23.82
Packed-Ensemhles
96.2
96.5
4.06
Deep Ensembles
C10
WR
95.8
0.143
0.013
97.8
96.0
12.5
145.96
23.82
Single Model
C100
R18
75.1
1.016
0.093
88.6
79.5
55.0
11.22
0.56
BatchEnsemble
C100
71.2
1.236
0.116
86.0
75.4
11.25
2.22
MIMO (p
C100
R18
75.3
0.962
0.069
89.2
80.7
$2.9
11.36
0.56
Masksembles
C100
74.2
1.054
0.061
86.7
76.3
11.24
2.22
C100
R18
76.4
0.858
0.041
88.7
79.8
$7.1
8.27
0.48
Deep Ensembles
82.4
Single Model
C100
R50
78.3
0.905
0.089
87.4
77.9
57.6
23.70
1.30
RatchFnsermlee
C100
R50
66.6
1.788
0.182
85.2
74.6
60.6
23.81
5.19
MIMO 11
C100
R50
79.0
0.876
0.079
87.5
76.9
24.33
1.30
Masksembles
C100
R50
78.5
0.832
0.046
90.3
81.9
52.3
23.81
5.19
Packed-Ensemblle
81.2
1,00
Deep Ensembles
C100
R50
80.9
0.713
0.026
89.2
80.8
52.5
94.82
5.19
Single Model
C100
WR
80.3
0.963
0.156
81.0
64.2
$0.1
36.55
5.95
RatchEnseemle
C100
WR
82.3
0.835
0.130
88.1
78.2
69.8
36.65
23.81
MIMO =
C100
WR
80.2
0.822
0.028
84.9
72.0
72.8
36.74
5.96
Masksembles
C100
74.4
0.063
76.1
36.59
23.82
Ensembbessss
C100
WR
83.9
0.678
0.089
86.2
73.2
80.7
36.62
5.95
Deep Ensembles
C100
82.5
0.229
81.6
67.9
23.82
On CIFAR-100 100. Deep Ensembles outperforms Packed- Ensembles on ResNet-18. However, we gue that ResNet-18 18 architecture need more representation capacity to be divided into subnetworks CIFAR-100. Indeed, when we look the results ResNet-50, we can see that Packed- Ensembles has better results than Deep Ensembles. This analysis demonstrates that, given suf- ficiently large network, parameters. In Appendix D, we discuss the influence of the representation capacity.
is able match Deep Ensembles with only 16% of its
Based on the results in Table 2, we can conclude that Tacked-Ensemblee improves uncertainty quan- tification for OOD and distribution shift ImageNet compared to Deep Ensembles and Single model and that it improves the accuracy with a moderate training and inference cost.
4.1.3 STUDY ON THE PARAMETERS a AND ?
Table reports results for a = and 2. However, the optimal values these hyperparameters depend on the balance between computational cost and performance. To help users strike the best compromise, we propose Figures 6 and 7 in Appendix D, which illustrate the impact of changing a on the performance of Packed-Ensenn Ensembles.
DISCUSSIONS
We have shown that Benked-Enssmbles has attractive properties, mainly by providing similar quality of uncertainty quantification Deep Ensembles while using reduced architecture and

---

Published as a conference paper at ICLR 2023
Table 2: Performance comparison on ImageNet using ResNet--0 (R50) and ResNet-50x4 (R50x4). All ensembles have M = 4 subnetworks and y = 1. We highlight the best perfor- mances in bold. For OOD tasks, we use ImageNet-O (IO) and Texture (T), and for distribution shift we use ImageNet-R. The number of parameters and operations are available in Appendix M.
Method
Net
Acc
ECE
AUPR
AUC-
FPR95 T
AUPR 10
AUC-1O
FPK95
10
rAce
(NLL
:ECE
Single Model
R50
77.8
0.121
18.0
$0.9
68.5
3,6
50.5
90.8
23,5
5.187
0.082
Beachinsemble
K50
75.9
10035
20.2
81.6
66.5
4.0
55.2
82.3
21.0
6.348
0.166
MIMO (p=
R50
77.6
0.147
18.4
81.6
66.8
3.7
52.2
90.6
23.4
5.115
0,059
Masksembles
R50
73.6
0.209
13.5
79.7
3.3
47.7
81.7
21.2
5.139
0,011
Ensembles
23.8
Deep Ensembles
79.2
19.6
85.5
24.9
0.018
Single Model
R50x4
80.2
0,022
20.5
82.6
63.9
60.2
87.4
26.0
5,190
0.1721
Buschinsemble
X30x4
72.7
0,024
25.8
82.8
63.8
4.8
58.4
50.5
23.4
5,079
0.20
MIMO (p- 1)
R.50x4
80.1
0.015
19.3
82.5
66.1
4.9
60.7
86.4
25.8
5.278
0.189
Masksembles
R50x4
79.8
0.137
21.5
83.3
63.5
4.4
58.4
80.5
23.1
6.079
0.207
Packed Ensembles
R50xx4
81.3
0.103
34.6
88.1
50.3
9.66
69.9
79.2
26.6
4.548
0,075
Ensemblls
23.0
28.2
computing cost. Several questions can be raised, and we conducted some studies detailed in the Appendix sections to provide possible answers.
Discussion the sparsity As described in section 3, one could interpret PE leveraging group convolutions to approximate Deep Ensembles with a mask operation applied to some components. In Appendix C, by using simplified model, we propose bound of the approximation error based on the Kullback-Leibler divergence between the DE and its pruned version. This bound depends on the density of ones in the mask p. and, more specifically, depends on p(1 - p) and (1 - p)²/p p. By manipulating these terms, corresponding to modifying the number of subnetworks M, the number of groups y, and the dilation factor a, we could theoretically control the approximation error.
On the sources of stochasticity Diversity is essential in ensembles and is usually obtained by exploiting two primary sources of stochasticity: the random initialization of the model's parameters and the shuffling of the batches. last source of stochasticity is introduced during training by the non-deterministic behavior of the backpropagation algorithms. In Appendix F, we study the function space diversities which arise from every possible combination of these sources. It follows that only one of these sources is often sufficient to generate diversity, and no peculiar pattern seems to emerge to predict the best combination. Specifically, we highlight that even the only use of non non-deterministic algorithms introduces enough diversity between each subnetwork of the ensemble.
Ablation study We perform ablation studies to assess the impact of the parameters M, a, and Appendix D provides in-depth details of this study.
y on the performance of No explicit behavior appears from the results we obtained. A trend shows that higher number of subnetworks helps get better OOD detection, but the improvement in AUPR is not significant.
Training speed Depending on the chosen hyperparameters a, M, and 7, PE may have fewer parameters than the single model, as shown in Table 1. This translates into expected lower number of operations. A study of the training and inference speeds, developed in Appendix H, shows that using 1F-(2,4,1) does not significantly increase the training and testing times compared to the single model while improving accuracy and uncertainty quantification performances. However, this also hints that the croup-convolution speedup not optimal despite the significant acceleration offered by 16-bits floating points.
OOD criteria The maximum softmax probability is often used as criterion for discriminating OOD elements. However, this criterion is not unique, and others can be used, such as the Mutual Information, the maximum logit, or the Shannon entropy of the mean prediction. Although no re- lationship is expected between this criterion and PE, we obtained different performances in OOD detection according to the selected criterion. The results CIFAR-I00 are detailed in Appendix E and show that an approach based on the maximum logit seems to give the best results detecting OOD. It should be noted that the notion of OOD depends the training distribution. Such dis- cussion does not necessarily generalize to all datasets. Indeed, preliminary results have shown that Mutual information outperforms the other criteria for our method applied to the ImageNet dataset.
8

---

Published as a conference paper at ICLR 2023
6 RELATED WORK
Ensembles and uncertainty quantification. Bayesian Neural Networks (BNNs) (MacKay, 1992; Neal, 1995) are the cornerstone and primary source of inspiration for uncertainty quantification in deep learning. Despite the progress enabled by variational inference (Jordan et al., 1999; Blundell et al., 2015), BNNs remain challenging to scale and train for large DNN architectures (Dusenberry et al., 2020). DE (Lakshminarayanna et al., 2017) arise a practical and efficient instance of BNNs, coarsely but effectively approximating the posterior distribution of weights (Wilson Iz- mailov, 2020). DE currently the best- rest-performing approach for both predictive performance and uncertainty estimation (Ovadia et al., 2019; Gustafsson et al., 2020).
Efficient ensembles. The appealing properties in performance and diversity of DE (Fort et al., 2019), but also their major downside related to computational cost, have inspired large cohort of approaches aiming to mitigate it. BatchEnsemble (Wen et al., 2019) spawns an ensemble at each layer thanks
efficient parameterization of jubnetwork-spectine parameters trained in parallel. MIMO (Havasi et al., 2020) shows that a large network can encapsulate multiple subnetworks using multi-input multi-output configuration. A single network can be used in ensemble mode by dis- abling different sub-sets of weights at each forward pass (Gal & Ghahramani, 2016; Durasov et al., 2021). Liu al. (2022) leverage the sparse networks training algorithm of Mocanu al. (2018) to produce ensembles of sparse networks. Ensembles can be computed from single training run by collecting intermediate model checkpoints (Huang al., 2017: Garipov et al., 2018). by computing the posterior distribution of the weights by tracking their trajectory during training (Maddox et al., 2019: Franchi al., 2020), and by ensembling predictions over multiple augmentations of the input sample (Ashukha et al., 2020). However, most of these approaches require multiple forward passes.
Neural network compression. The most intuitive approach for reducing the size of a model is employ DNNs that are memory-efficient by design, relying on, e.g., channel shuffling (Zhang & Yang, 2021), point-wise convolutional filters (Liang et al., 2021), weight sharing (Bender et al., 2020), or a combination of them. Some of the most popular architectures that leverage such models are SqueezeNet (landola et al., 2016), ShuffleNet (Zhang et al., 201 8b), and MobileNet-v3 (Howard et al., 2019). Some approaches conduct automatic model size reduction, e.g., network sparsifi- cation (Molchanov al., 2017; Louizos et al., 2018; Frankle & Carbin, 2018; Tartaglione et al., 2022). These approaches aim at removing as many parameters as possible from the model to im- prove memory and computation efficiency, also at train time Bragagnolo et al. (2022). Similarly, quantization approaches (Han et al., 2016; Lin et al., 2017) avoid or minimize the computation cost of floating point operation and optimize the use of the much more efficient integer computation.
Grouped convolutions. To the best of our knowledge, grouped convolutions (group of convolu- tions) were introduced by Krizhevsky et al. (2012). Enabling the computation of several independent convolutions in parallel, they developed the idea of rumning single model on multiple GPU devices. Xie et al. (2017) demonstrate that using grouped convolutions leads to accuracy improvements and model complexity reduction. So far, grouped convolutions have been used primarily for compu- tational efficiency but also to compute multiple output branches in parallel (Chen Shrivastava, 2020). PE re-purpose them delineate multiple subnetworks within network and efficiently train an ensemble of such subnetworks.
7 CONCLUSIONS
We propose a new ensemble framework, Packed-Ensembles, that can approximate Deep Ensembles in terms of uncertainty quantification and accuracy. Our research provides several new findings. First, we show that small independent neural networks can be as effective as large, deep neural networks when used in ensembles. Secondly, we demonstrate that not all sources of diversity are essential for improving ensemble diversity. Thirdly, we show that Packed-Ensse Ensembles are more stable than single DNNs. Fourthly, we highlight that there a trade-off between accuracy and the number of parameters, and Packed-Ensemtn enables us create flexible and efficient ensembles.
In the future, we intend to explore
for more complex downstream tasks.

---

Published as a conference paper at ICLR 2023
8 REPRODUCIBILITY
Alongside this paper, we provide the source code Packed-Ensembles layers. Additionally, we have created two notebooks demonstrating how train ResNet-50-based Packed-Ensembles using public datasets such as CIFAR-10 and CIFAR-I00, To ensure reproducibility, we report the perfor- mance given specific random seed with deterministic training process. Furthermore, it should be noted that the source code contains two PyTorch Module classes to produce Packed-Ensembll Ensembles efficiently. A readme file at the root of the project details how install and run experiments. In addition, we showcase how to get
from LeNet (LeCun et al., 1998).
To further promote accessibility, we have created an open-source pip-installable Py PyTorch package, torch-uncertainty, that includes Ensembllsss layers. With these resources, we hope to encour- age the broader research community to engage with and build upon our work.
ETHICS
The purpose of this paper to propose a new method for better estimations of uncertainty for deep-leariing-based models. Nevertheless, we acknowledge their limitations, which could become particularly concerning when applied to safety-critical systems. While this work aims to improve the reliability of Deep Neural Networks, this approach is not ready for deployment in safety-critical systems. We show the limitations of our approach in several experiments. Many more validation and verification steps would be crucial before considering its real-world implementation ensure robustness to various unknown situations, including corner cases, adversarial attacks, and potential biases.
ACKNOWLEDGMENTS
This work supported by AID Project ACoCaTherm and Hi!Paris. This work was per- formed using HPC resources from GENCI-IDRIS (Grant 2021-AD01 101 1970R1) 1) and (Grant 2022- 107000)
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In ICLR, 2020. 9
William H Beluch, Tim Genewein, Andreas Nümbeeggr, and Jan M Köhler. The power of ensembles for active learning in image classification. In CVPR, 2018. 21
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoe V. Le. Can weight sharing outperform random architecture search? An investigation with TuNAS. In CVPR, 2020. 9
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In JCML, 2015. 9
Andrea Bragagnolo, Enzo Tartaglione, and Marco Grangetto. To update or not to update? neu- rons at equilibrium in deep models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /pwpreeeeeeeeeeeeeeeeeeeee.eeeee net forummid= LorrummLL-LGEEEEVUTEEE
Hao Chen and Abhinav Shrivastava. Group ensemble: Learning an ensemble of convnets single
convnet. arXiv preprint arXiv:2000.006649, 2020. 2, 9
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated
data augmentation with reduced search space. In CVPR, 2020. 17
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6, 25
Thomas G Dietterich. Ensemble methods in machine learning. /WMCS, 2000. 2
Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua. estimation. In CVPR, 2021. 2, 6, 9,
Masksembles for uncertainty
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In ICML, 2020. 9
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv:1912.02757, 2019. 2,
Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, and Isabelle Bloch. Tradi: Tracking deep neural network weight distributions. In ECCV, 2020. 2,
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018. 9, 22
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016. 2, 9, 25
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurIPS, 2018. 2,
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017. 2
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon. Evaluating scalable bayesian deep learning methods for robust computer vision. In CVPR Workshops, 2020.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In ICLR, 2016. 9
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Lakshmi- narayaman, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In ICLR, 2020. 2, 6,9
11

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog nition. CVPR, 2016. 6, 17
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019. 2
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019. 23, 24
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017. 6
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-o--distriuution generalization. In CVPR, 2021a. 6
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
José Miguel Hernández-Lobate and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In ICML, 2015. 25
Andrew G. Howard, Mark Sandler, Grace Chu, Liang-Chieh Chiih Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for MobileNetV3. In ICCV, 2019. 9
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get for free. In ICLR, 2017. 2.9
Forrest landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 mb model size. arXiv preprint wr.iv: 2016. 9
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 1999.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? NeurIPS, 2017. 25
Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models. arXiv preprint w:Xiv:2005.00570 2020. 2
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, MIT, 2009. 6, 23
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012. 2,9
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017. 2, 4, 6, 9, 20, 25
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub-
bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition.
Neural computation, 1989. 2
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. 10
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M heads are better than one: Training a diverse ensemble deep networks. arXiv preprint arXiv: I5/ 1.160 14, 2015. 2
12

---

Published as a conference paper at ICLR 2023
Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J. Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, Michael Sokolsky, Ganymed Stanek, David Stavens, Alex Teichman, Moritz Werling, and Sebastian Thrun. Towards fully autonomous driv- ing: Systems and algorithms. In IV, 2011.
Feng Liang. Zhichao Tian, M. Dong, Shuting Cheng, Li Sun, Hai Helen Li, Yiran Chen, and Guohe Zhang. Efficient neural network using pointwise convolution kemelss with linear phase constraint.
Neurocomputing. 2021. 9
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In NewrlPS, 2017.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no over- head for either training testing: The all-round blessings of dynamic sparsity. In ICLR, 2022.
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry Vetrov. On power laws in deep ensembles. In NeurlPS, 2020. 2
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. In ICLR, 2018. 9
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 1992. 9
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. NeurlPS, 2019. 2,9
Rowan McAllister, Yarin Gal, Alex Kendall, Mark Van Der Wilk, Amar Shah, Roberto Cipolla, and Adrian Weller. Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning. In IJCAI, 2017.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 2018. 9
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2015.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. In ICML Workshops, 2019. 24
Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPS Workshops, 2011. 6
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015. 2
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. In ICLR, 2020. 22
D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In ICNN, 1994. 25
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan. and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In NeurlPS, 2019. 2, 9, 24
13

---

Published as a conference paper at ICLR 2023
Michael P Perrone and Leon N Cooper. When networks disagree: Ensemble methods for hybrid neural networks. Technical report, Brown University, 1992.
Alexandre Ramé, Rémy Sun, and Matthieu Cord. outputs via deep subnetworks. In ICCV, 2021.
Mixmo: Mixing multiple inputs for multiple
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 17
Enzo Tartaglione, Andrea Bragagnolo, Attilio Fiandrotti, and Marco Grangetto. Loss-based sensi- tivity regularization: towards deep sparse neural networks. Neural Networks, 2022.
Torchinfo. Torchinfo. https: http///gtttttttett http://ww...cm//yy//e//////rrrre 1.7.1. 7, 25
2022. Version:
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. ViM: Out-of-distributtin with virtual-
logit matching. In CVPR, 2022. 6
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach efficient
ensemble and lifelong learning. ICLR, 2019. 2, 6,
Ross Wightman. Pytorch image models. pytorch- weacch--mmageeee Laaichh-mmmemmmeeel 2019. 17
https://githab.coom rw wttpw///wwwwtgttgno/ooooooc.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NeurlPS 2021 Workshop ImageNet PPF, 2021. 17
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In NeurIPS, 2020. 4,
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017. 3, 9, 24
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In CVPR, 2019. 17
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. 6, 17
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018a. 17
Qing-Long Zhang and Yubin Yang. SA-Net: Shuffle attention for deep convolutional neural net- works. In ICASSP, 2021. 9
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018b. 9
14

---

Published as a conference paper at ICLR 2023
TABLE OF CONTENTS SUPPLEMENTARY MATERIAL
Notations
16
Implementation details
17
Discussion the sparsity
17
D Ablation study
20
Discussion about OOD criteria
F Discussion about the sources of stochasticity
21
G Discussion about the subnetworks
22
Discussion about the training velocity
23
I Distribution shift
23
J Stabilization of the performance
24
On the equivalence between sequential training and
24
Using groups is not sufficient to equal
24
M Efficiency of the networks trained on ImageNet
25
Regression
25
15

---

Published as a conference paper at ICLR 2023
A NOTATIONS
We summarize the main notations used in the paper in Table 3.
Table 3: Summary of the main notations of the paper.
Notations
Meaning
D = (())) P)
The set of [D] data samples and the corresponding labels
3. m, L
The index of the current layer, the current subnetwork, and the number of layers
2²
The preactivation feature map and output of the layer (j 1)/input of layer
6
The activation function (considered constant throughout the network)
h
The feature map and output of layer j. h <(z²)
Hj. W
The height and width of the feature maps and output of layer j
C₃
The number of channels of the feature maps and output of layer j
The number of parameters of layer j
B
The batch size training procedure
mask/ m
The mask corresponding to the layer j of the subnetwork m
[.]
The floor function
*, @,
The 2D cross cross-correlation, the convolution, and the Hadamard product
8y
The size of the kernel the layer j
M
The number of subnetworks in ensemble
The prediction of the subnetwork number mn concerning the inpul x,
The prediction of the ensemble concerning the input x;
The widlh-augmeenationn factor of Packed-Ensembles
The number of subgroups of Packed-Ensemble
0 cm
The set of weights of the subnetwork m with width factor a
wi!
The weights of layer with groups and width factor
16

---

Published as a conference paper at ICLR 2023
Table 4: Hyperparameters for image classification experiments. HFllip denotes the classical horizontal flip.
Dataset
Networks
Epochs
Batch size
start Ir
Momentum
Weight decay
7-1r
Milestones
Data sugmentations
C10
R18
75
128
0.05
0.9
Se.4
0.1
25, 50
11Flip
C10
R50
200
128
0.9
Se-d
0.2
60, 120, 160
Hi'lip
C10
WR28-10
200
128
0.1
0.9
Se-4
0.2
60, 120, 160
HIFlip
C100
RI8
128
0.05
0.9
le-d
0.2
25,50
HF'lip
C100
R50
200
128
0.9
5o-4
0.2
60, 120, 160
HFlip
C100
WR28-10
200
128
0,9
5p-d
0.2
60, 120, 160
Medium
B IMPLEMENTATION DETAILS
General Considerations. Table 4 summarizes all the hyperparameters used in the paper for CIFAR-10 and CIFAR-100. In all cases, we use SGD combined with a multistep-leamming-uuu scheduler multiplying the rate by 7-Ir at each milestone. Note that BatchEnsemble based ResNet- 50 uses a lower learning rate of 0.08 instead of 0.1 for stability. The "Medium" data augmentation corresponds combination of mixup (Zhang et al., 2018a) and cutmix (Yun et al., 2019) with 0.5 switch probability and using timm's 's augmentation classes (Wightman, 2019), with coefficients respectively 0.5 and 0.2. this case, we also use RandAugment (Cubuk et al., 2020) with m 9, 2, and mald = and label-smoothing (Szegedy et al., 2016) of intensity 0.1.
To ensure that the layers convey sufficient information and are not weakened by groups, we have set constant minimum number of channels per group to 64 for all experiments presented in the paper. the number of channels per group is lower than this threshold, 7 is reduced. Moreover, we do not apply subgroups (parameterized by 7) the first layer of the network, ResNet's blocks. Experiments in which this minimum number of channels could play a significant role and bring confusion are not presented (see, for instance, PE-(1, 4, 4) in Table 5).
the first layer of
For ImageNet, we use the A3 procedure from Wightman et al. (2021) for all models. Training with the exact A3 procedure
not always possible. Refer to the specific subsection for more details.
Please note that the hyperparameters of the training procedures have not been optimized for our method and have been taken directly from the literature (He et al., 2016; Wightman et al., 2021). I). We strengthened the data augmentations for WideResNet on CIFAR-100 as we were not able to replicate the results from Zagoruyko & Komodakis (2016).
Masksembles. We use the code proposed by (Durasov et al., 2021) 3. We modified the mask gen- eration function using binary search, as proposed by the authors since was unable to build masks for ResNet50x4. We note that the code implies performing batch repeats at the start of the forward passes. All the results regarding this technique are therefore computed with this specification. The ResNet implementations are built using Masksemble2D layers with M 4 and scale factor of 2 after each convolution.
Ensemble For BatchEnsemble, we different values for weight decay: table 4 pro- vides the weight decay corresponding to the shared weights but we don't apply weight decay to the vectors and (which generate the rank-l matrices).
ImageNet. The batch size of Masksembles ResNet-50xx is reduced to 1120 because of memory constraints. Concerning the BatchEnsembles based on ResNet-50 and ResNet-50x4, we clip the norm of the gradients to 0.0005 to avoid divergence.
C DISCUSSION ON THE SPARSITY
In this section, we estimate the expected distance between dense, fully-connected layer and a sparse one. For simplicity, we are here assuming operate with fully-connected layer. First, let us write our first proposition:
Proposition C.1. Civen fully connected layer j + defined by:
C5-1
2+1(
Σ
(c,
(S)
k=0
available hithuh..oonhiiiddaassssnasssess
17

---

Published as a conference paper at ICLR 2023
-4
4
DRL(z.
3
2
0.0
6.0
0.2
0.2
0.4
0.5
0.8
0.8
1.0
Figure 5: KL divergence for different values of p and o with 2 (k) 0.1 Vj, and and w (c, k) 0.1 vj.c,
and its approximation defined by:
C5-1 Σ (c, k)cskmm (k, c)) (k)
(c)
(6)
Under the assumption that the j follows a Gaussian distribution he ~ NNw E),, where D the covariance matrix, and μ³ the mean vector, the Kullback Leibler divergence between the layer and its approximation is bounded by:
DRI (c) S p - 2 P. (1 - CG-1 (e + [11 x 1 (c)
2
(o
p 2(
(7)
where p [0; 1] is the fraction of the parameters of z +1 (c) included in the approximation ž (c).
A plot for (7) is provided in Figure 5.
Proof. To prove Prop. C.I, we state first that, since (k) follows a Gaussian distribution, and considering that w at inference time is constant and linearly-combined with gaussian random variable, z²+1 will be as well gaussian-distribuee.. From the property of linearity of expectations, know that the mean for z z (c) is:
4 (c)
Σ ((
(8)
K=0
and the variance is:
C,-1 (o(+))((c)= Σ w' (c) ((GH))( [W(GH))(KH
(9)
Wack
If we assume E(i,h) 00i k, (9) simplifies into:
Gy-1 ( = Σ
(cc
(10)
K=
18

---

Published as a conference paper at ICLR 2023
Let us now consider the case with the mask, similarly presented at the end of section 2.1:
C
(c) Σ (c) k) (k, (1)
(11)
We assume here that mask' ~ Ber(p) where is the probability of the Bernoulli (or 1-pruming rate). In the limit of large Cy. we know that z (c) follows a Gaussian distribution defined by mean and variance equal to:
C,-1 Σ k=
(c)
w (k)
(12)
C-11 Σ
(c)+
pa (c, k)2 (A() -2(,))
(13)
k=
Hence, we have:
i (c) =p 1xt
(14)
C--1
(2)+))(())= = ())))) (1 - p)
Σ
()
w
(15)
k=
In order to assess the dissimilarity between and ă, we can write the Kullback-Leiblrr divergence:
DKL. (z, (c) 1 log
(o)t))(())
+ (c)+))(() + (c) (c)]
} (16)
((c)
(G!+1)2(c
Straightforwardly we can write the inequality:
1 1 (62+2) + (o)+2) + [n_ (c) -c) (c)) 1} (17)
DKL (z, (2, ()) V (o)+))(c)
22
According to (15) write:
p (1 E 10 (k))
(1-
()),
1+
(o!+1)?(c)
(og+1) + [ng (g)
(18)
(o + (1 p) E wi (c. k) (k))
(a) (g (1(+)()
(a (1111) fag (c) (c)- (c))
Since we know that
p (())))))))1)p)))) Eie (cck)
plor 1)p(s))
we
can also write:
1
p - 1+ p. (1 pp) Σ Cke (c, (b)
(2,
2
(ot (ott1p(c)
(0)+1)(()) (N6+1 (c) pit+1())
-13
(19)
p )2(c))(())
Finally, according to: (14)
2 p. (1 p)
=k=0 w
C3-1
wi (c, k) (c, (k) + [11 [11 p) x ue)((c))²
(2 (c) < p p
(
plo
finding back (7).
19

---

Published as a conference paper at ICLR 2023
Accuracy
AUPR
Accui any
AUPR
0.0
M-4.y=1
M-4.y=1
8
N 8.y.
Figure 6: Accuracy and AUPR with ResNet-50 on CIFAR-100 100 depending on
Accuracy
AUPR
M 2
M 2
M=
0,
8, ==3
096
Accaracy
0.0
AUPR
0.5
0.00
038
Figure 7: Accuracy and AUPR of Packed-Ensemblle with ResNet-50 CIFAR-100 depending y.
D ABLATION STUDY
Our algorithm mainly depends on three hyperparameters. M represents the number of subnetworks in the ensemble, controls the power of representation of the DNN, and is extra parameter that controls the sparsity degree the DNN. To evaluate the sensitivity of Packed-Ensemblle Ensembles to these parameters, train 5 ResNet-50 Figures and 7 show that the more we add subnetworks increasing M, the better the performance, in terms of accuracy and AUPR. We also note that the results stable with 7. Moreover, the resulting accuracy tends to increase with a until reaches plateau. These statements are confirmed by the results in Table 5.
CIFAR-10 similarly the protocol explained in section 4.1.
E DISCUSSION ABOUT OOD CRITERIA
Deep Ensembles (Lakshminarayanaa et al., 2017) and Tacked-Ensembrr are ensembles of DNNs that can be used to quantify the uncertainty of the DNNs prediction. Similarly to Bayesian Neu- ral Network, one take the softmax outputs of posterior predictive distribution, which define the MSP max,, {P(y. /x, D) The MSP can also be used for classical DNN, yet we use the conditional likelihood instead of the posterior distribution in this case.
One can also use the Maximum Logit (ML) and the entropy of the posterior predictive distribution as uncertainty criteria, which is defined by Ent. H((P(y.x,, |x, D)) with H. being the entropy function. Another metric is the mutual information between two random variables, which is defined by: MI
M-1
H(P(y, x, D)) 1
Σ
H(PPylOa.ma x)). It represents a measure of the ensemble entropy, which
M
the entropy of the posterior minus the average entropy over predictions.
20

---

Published as a conference paper at ICLR 2023
Table 5: Performance (Acc / ECE/ AUPR) of Packed--nsembles for various and with ResNet-50 on CIFAR-100 and M
4.
3
4
7
0.7872 0.0015 0.8999
0.8 16 0.0203 / 0.8966
0.8187 0.0201 0.8825
0.8183 / 0.0230 / 0.8939
2
0.7857 0.0115 0.9024
0.8103 / 0.0025 / 0.9115
0,8186 / 0.0197 0.9127
0.8242 0.0099/ / 0,9088
0.8119/ 0.0180 0.9066
0.8182/ / 0.0236 0.9140
0.8225/ 0.0226 0.9229
Table 6: Comparison of the effect of the different uncertainty criteria for OOD on CIFAR-100 with different sets of parameters for Beeked-Enssmmles.
Criterion
eval
3=3,1=
a=
1=6,7=1
0=
MSP
AUPR (1)
0.8952
0.0132
0,9055
0.0034
0.9153 : 0.0012
0.9149
0.0071
0.9141 # 0.0057
ML
AUPR
0,9183
0.0098
0.9175
0.0044
0.9285 + 0.0012
0,9265
0.0070
0.9268
0,0068
Eat.
AUPR
0.9105
0.0138
0.9152
0.0035
0.9260 0.0016
0,9237
0.0066
0,9252
0.0060
MI
AUPR (T)
0.8649-
0.0061
0,9139 0.0077
0.9157
0.0072
0,91% 0.0109
0.9245 0.0091
AUPR (T)
0.8404
0.0071
0,8746 1 0.0056
0,8827 : 0.0033
0.0102 0.8842
0,8931 +: 0.0072
MSP
AUCCC)
0.8056
0.02KI
0.8214 0.0101
NAUX 0.0033
0.8432 0.01134
0.837 0.0154
ML
AUC (*)
0.8562
0.0194
0,8421 0.0115
0,8665
0.0027
0.8621 0,0144
0.8607 0.0114
Eat.
AUC (*)
0.836
0.0271
0,8427 : 0.6095
0.8662
0.0027
0.8617 : 0.0136
0.8614 = 0.009
MI
AUC
0,7711
0.0064
0,8312 0.0135
0,8402
0.0116
0,8468 0,0163
0,8513 0.0120
AUC
0.7305
0.0153
0.7799 0.0129
0.7943
0.0082
0.7999 0.0166
0.8092
The last metric, used in active learning, the variation ratio (Beluch al., 2018), which measures the dispersion of nominal variable and is calculated as the proportion of predicted class labels that are not the modal class prediction. It defined by: v = f// where ʃ; is the number of predictions falling into the modal class category.
In Table 6, the results for the different metrics are reported. We note that ML seems to be the best metric to detect OOD. This metric is followed by Ent. and then MI. Note that v. widely used in active learning, does not seem effective in detecting OOD samples. This shows us that it is essential to use good criterion in addition to good ensembling.
F DISCUSSION ABOUT THE SOURCES OF STOCHASTICITY
As written in the introduction of the paper, diversity is essential to the success of ensembling, be it for its accuracy but also for calibration and OOD detection. Three primary sources can induce weight diversity, and therefore diversity in the function space, during the training. These sources are the initialization of the weights, the composition of the batches, and the use of nonedeteriiiistic backpropagation algorithms On Table 7, we measure the performance and diversity of Packed- Ensembles trained on CIFAR-100. The quantity of diversity is measured by the mutual informa- tion and is twofold: we compute the in-distribution mutual information (IDMI) on the test set of CIFAR- 100 and the OOD mutual information (OODMI) on SVHN. Concerning the performance, compute the accuracy, ECE, and AUPR, which are proxies of the quality of this diversity. Results of Table 7 lead to several takeaways. First, they hint that there is no clear best set of trivial sources of stochasticity. Except for the first (and greyed) line, which corresponds ensembling completely identical networks (the training being totally deterministic, which the null MI confirms), the results seem equivalent in diversity (via mutual information) and ID/OOD performance. Secondly, it shows that the use of non-deterministic algorithms can be sufficient to generate diversity. It was noted that this effect does not always happen depending on the selected architecture and the precision used (float 1 6, or f2)at
Given that there is no emerging best set of stochasticity, we use the faster non-deterministic back- propagation algorithms and different initializations to ensure enough stochasticity and for program- ming convenience.
see httm:///dbsswiidaoeeeee/ee/aaamm
21

---

Published as a conference paper at ICLR 2023
Table 7: Comparison of the diversities and the performance wrt. the diffferent sources of stochasticity on C1FAR-100. ND corresponds to the use of Non-deterministic backpropagation algorithms, DI different initializations, and DB to different compositions of the batches. A stan- dard error (over five runs) is included in small font.
Stochasticity
ResNet-18
ND DI
DB
Acc (t)
ECE (1)
AUPR (T)
IDMI
OODMI
71.7040.06
0.0497=0.0013
87.3240.91
0+D
0z0
75. 7.....2
0.0365:100044
89.53:10.47
0.1945
0.4001
76.20+0.01
0.041920.000
89.54:20.9
0.2011
0.4391
v
76.06+0.02
0.0434+0.001
88.70+0.27
0.1987
0.4079
76.1010.05
0.042440.0001
88.65:10.22
0.1995
0,4360
v
76.19+0.11
0.0433+0.0010
88.87=00115
0.2032
0.4090
v
76.14+0.07
0.0137=000008
$9.21:10.88
0.19-13
0.4195
76.2940.07
0.0445#0.000
89.00:1.51
0.1954
0,4060
Stochasticity
ResNet-50
77. 77.63+0..2
0.0825=000018
89. 19.19.5665
0+00
0=:0
80.94+0.10
0.0179:0.0010
90.23:10.22
0.1513
0.4022
81.01+:0.06
0.0202:0.0011
91.10=10.39
0,1524
0.4088
80.87++0.10
0.0178+0.0010
90.80+0.30
0.1505
0.4115
81.1610.10
0.0210#0.000
91.69:10.56
0.1584
0.4135
v
81.14+00.07
0.0200=0.000 0007
90.41:t..0
0.1503
0.3897
81.10::0.05
0.0186:000016
90.85:10.299
0.1521
0.4034
81.0810.08
0.019820.0011
90.68410.25
0.1534
0.4031
G DISCUSSION ABOUT THE SUBNETWORKS
The width and depth of deep neural networks are crucial research topics, and researchers strive to determine the best approaches for increasing the depth of DNNs, which can lead to improved accuracy. According to Nguyen et al. (2020), the width and depth of DNN connected with its capacity to learn block structures, which can improve accuracy. Therefore, the model's capacity may decrease if the width is divided.
Deep neural networks are heavily over-parameterized, as stated by the lottery ticket hypothesis (Frankle & Carbin, 2018). It suggests that up to 80% of neurons icant loss of performance. The MIMO approach builds on this assumption by allowing multiple networks be trained simultaneously, and neurons may be used by several subnetworks. In our work, however, we assign each neuron to a specific DNN in the ensemble, guaranteeing their in- dependence. This way, the DNNs can learn independent representations. However, in MIMO, we rely on the fact that not all neurons are helpful, so we split the width of the initial DNNs into a set of DNNs. Although the decomposition may seem crude, it facilitates better parallelization of Backed-Ensembles. during training and inference.
be removed without signif-
To address the problem of not having sufficiently wide subnetworks, we added hyperparameter to increase the width of subnetworks. In Figure 8, we explore the impact of subnetwork width.
Our observations reveal that the accuracy of the DNN increases with the width while the AUPR remains relatively constant. This finding suggests that the a is paramount in maintaining a balance in the DNN's 's width. We also note that reducing the width of the DNN does not significantly impact its accuracy. Hence, decision to split the width of the DNN to create multiple subnetworks is justified since the uncertainty quantification remains unaltered, and the accuracy is not significantly compromised.
22

---

Published as a conference paper at ICLR 2023
Accuracy
AUPR
Accurany
65
Resnet18
Resnet18
Recnef50
Recnet50
04
044
45
13
2.5
30
65
23
3.5
2,0
2.5
10
3b
Xwidthh
X Xwidd
Figure 8: Accuracy and AUPR curves of ResNet-18 in red and ResNet-50 in blue on CIFAR-100 with different widths. When the width is equal 1, corresponds to the original ResNet; when the width equal to z, the width of every layer multiplied by x.
Table 8: Comparison of training and inference times of different ensemble techniques using 1orch1.12.1+cht an RTX 3090. All ensembles have four subnetworks.
132 Precision
f16 Precision
models ResNet-50 CIFAR-100
Training I
Inference
Training
Inference 1
slepoch
inds
s/epoch
im/s
Single Model
37.06
3709
22.42
5718
Pucked- Daaeee--Ennemblss(((2442.
179.50
1381
51.20
3406
Packed- Pucked-Ensembll Ensembles-(2.4.2)
175.10
1501
52.11
3440
Deep Ensembles
145.30
1001
84.86
1609
MIMO BatchEnsemble
37.90
3574
24.44
5649
58.78
1809
53.97
1916
In addition, a provides an additional degree of freedom to our ensemble, enabling to enhance its accuracy. This is significant advantage, it allows us to further balance the performance of the ensemble, which can lead to more accurate predictions and the number of parameters linked to its computational cost.
H DISCUSSION ABOUT THE TRAINING VELOCITY
Our experiments show that grouped convolutions are not as fast they could theoretically be, and confirm the statements made by many PyTorch and TensorFlow users 5, Following the idea that grouped convolutions are bandwidth-bounde we advise readers leverage Native Automatic Mixed Precision (AMP) and cuDNN benchmark flags when training bandwidth bottleneck compared to the bascline. AMP also divides the VRAM usage by two while yielding equally good results. Future improvements of PyTorch grouped convolutions should help Backed-Ensembles develop its full potential, increasing its current assets. We note Table that is only 1.6x slower than the single model during inference. is only 2.3x slower during training than the single model, making it efficient model capable of training four models in half the time of Deep Ensembles.
to reduce the
using float 1 6, Furthermore,
I DISTRIBUTION SHIFT
In this section, we evaluate the robustness of Packed-Ensemmen under dataset shift. We models trained on CIFAR-100 (Krizhevsky, 2009) and shift the data using corruptions and perturbations proposed by (Hendrycks Dietterich, 2019) to produce CIFAR-100-C. There are five levels of perturbations called "severity", from one, the weakest, to five, the strongest. In real-world scenarios,
For instance https::///// //g thub. http://///ggt.bb..ooooooooooo///
/75747
23

---

Published as a conference paper at ICLR 2023
Accuracy
ICE
2.7
0.30
510
8
2.2
SID
DC
DC
Ph: 12411
Pt 12-12111
(C-1
PC-10011
5.2
9.2
3.5
3300
Severha
Figure 9: Accuracy and Calibration under distributional shift. Comparison of the accuracy and ECE under all types of corruptions (a) CUFAR-100-C (Hendrycks & Dietterich, 2019) with different levels of severity.
distributional shift is crucial, explained by (Ovadia et al., 2019), and it is critical to study how much a model prediction shifts from the original training data distribution. Thanks to Figure we notice that Packed- -nsembble achieves the highest accuracy and lowest ECE under distributional shift, leading to a method robust against this uncertainty.
J STABILIZATION OF THE PERFORMANCE
We perform five times each training task CIFAR-10 and CIFAR-100 to estimate better value and be able to compute the variance. us first note that the standard deviation for the single DNN on LIFAR-100 with ResNet-50 architecture amounts to 0.68%. Ensemble strategies shrink the standard variation to 0.43% for Deep Ensembles and 0. 19% for Packed-Ensembles. Thus it seems stable in addition to improving accuracy and
that -nekembbes makes DNN predictions uncertainty quantification. This result is interesting as appears to contradict Neal et al. (2019), who claim that wider DNNs have smaller variance. This stability might come from the ensembling.
K ON THE EQUIVALENCE BETWEEN SEQUENTIAL TRAINING AND PACKED-ENSEMBLEES
The sequential training of Deep Ensembles differs significantly from the training procedure of Docked-Ensembler The main differences lie in the subnetworks' batch composition and the best models' selection.
the batches are strictly the same for all subnetworks, thus removing one source of stochasticity compared to sequential learning. Yet, in practice, we show empirically that random initialization and stochastic algorithms are sufficient to get diverse subnetworks (see Appendix F for more details).
Concerning
For the selection of models, Packed- -nakeebbesssbbes considers subnetworks as a whole (i.c., maximize the ensemble accuracy the validation set) and therefore selects the best ensemble given epoch. On the other hand, sequential training selects the best networks individually, possibly on different epochs, which does not guarantee that the best ensemble is selected but ensures the optimality of subnetworks over the epochs.
L USING GROUPS IS NOT SUFFICIENT TO EQUAL PACKED-ENSEMBLEE
To make that the of groups cannot simply explain results, compare Packed- Ensembles to a single ResNeXt-50 (32x4d) (Xie et al., 2017) in Table ResNeXt-50 is fairly equivalent to method but does not propagate groups, only used in the middle layer of each
24

---

Published as a conference paper at ICLR 2023
Table 9: Comparison between the results obtained with Packed-Ensemlle and similar
ResNeXt-50. The dataset is CIFAR-10.
Network
Acc
NLL
ECE
AUPR
AUC
FPR95
Params (M)
PE ResNet-50
96.0
0.1367
0.0087
97.1
94.9
14.5
23.6
ResNeXt-50
90.4
0.4604
0.0709
90.4
82.5
63.4
23.0
Comparison of the efficiency of the networks trained on ImageNet (Deng al., 2009). All ensembles have M = 4 subnetworks and y = 1. Mult-Adds corresponds to the inference cost, i.e., the number of Giga multiply-add operations for forward pass which is estimated with Torchinfo (2022).
Table 10:
Method
Net
Params (M)
Mult-Adds (G) 5
Single Model
R50
25.6
4.09
BatchEnsemble
R50
25.7
16.36
MIMO
R50
31.7
4.45
Masksembles
R50
25.7
16.36
Packed-Ensemblle (a 3)
R50
59.1
9.29
Deep Ensembles
R50
102.4
16.36
Single Model
R50x4
383.6
70.0
BatchEnsemble
R50x4
384.4
256.0
MIMO
R50x4
408.3
65.4
Masksembles
R50x4
384.0
256.0
Tacked-Ensemblee (a
R50x4
392.0
64.47
Deep Ensembles
R50x4
1534.4
280.0
block, which are therefore not independent. We keep the same training optimization procedures and autteeugmentttionn strategies detailed in Appendix B.
M EFFICIENCY OF THE NETWORKS TRAINED ON IMAGENET
Table 10 provides the efficiency of the networks trained on ImageNet- (see section 4.1.3), num- ber of parameters and multiply-addiitinn. PF-(3, 4, 1) was preferred to PE-(3, 4, 2) for ResNet50 to improve the representation capacity of the subnetworks.
N REGRESSION
To generalize our work, we propose to study regression tasks. We replicate the setting developed by Hernández-Lobato & Adams (2015), Gal & Ghahramani (2016), and Lakshminarayam et al. (2017).
For the training in the one-dimensional regression setting, we minimize the gaussian NLL (20) us- ing networks with two outputs neurons which estimate the parameters distribution (Nix & Weigend, 1994; Kendall & Gal, 2017). One output corresponds to the mean of the predicted gaussian distribution, and the softplus applied on the second is its variance. The ensemble's 's mean jág (x,) is computed using the empirical mean over the estimators and the variance using the formula of mixture ōg (x,)) M- Σm (0v... (x;)² + 1o... (x,)²) (x());) (Lakshmii- narayanan al., 2017).
heteroscedastic gaussian
(ys mm (x,))²
L (Hom.. (x:), Fem (xx)2,14 = 20 (x;)²
+ log Je (x,)² + log 2 2
(20)
We compare Packed-Ensembles--2 subnetworks of these methods are based on multi-layer perceptrons with a single hidden layer, con- taining 400 neurons for the extensive Protein dataset and 200 for the others, and ReLU non-linearity. The results show that Bieked-Enseebles. and Deep Ensembles provide equivalent results most datasets.
1) and Deep Ensembles the UCI datasets in Table 11. The
25

---

Published as a conference paper at ICLR 2023
Table 11: Comparison of the results obtained with eeeeed- and Deep Ensembles regression tasks.
RMSE
NLL
Datasets
Packeu-Enneebles
Deep Ensembles
Packed-Ensembles
Deep Ensembles
Boston housing
2.218 0.099
2.219 1 0.098
2.028 0.034
2,47 0.028
Concrete
5.092 1 0.225
5.167 1 0.234
2.854 0.028
2,885 0.032
Energy
1.675 + 0.085
1.712 + 0.067
1.543 1 0.072
1.553 0.060
Kin8nm
0.058 + 0.003
0.058 + 0.003
-1.442 = 0.010
-1.452
Naval Propulsion Plant
0.002 = 0.000
0.002 = 0.000
4.835 0.066
-4.833 +
Power Plant
3.127 = 0.018
3.097 +
2.607 0.007
2.600 + 0.007
Protein
3.476 0.030
3.412 1 0.017
2.472 0.033
2.422 0.015
Wine
0.482 0.006
0.483 1 0.006
0,622 - 0.014
0.611 1 0.013
Yacht
1.949 + 0.215
2.511 +
0.075
0.074
26


Reviews:
[{"review_id": "y9pqofNy8j", "reviewer": "Reviewer_x31y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple method for the neural network ensemble by integrating the ensemble into one neural network with the group convolution operation. Empirical evaluations show a strong performance and space efficiency of the proposed method.", "summary_of_the_review": "In general, this paper proposes a simple and useful ensemble technique. ", "strength_and_weaknesses": "Strength:\n1. The proposed group convolution based ensemble has comparable or better performance with a full ensemble, using less parameters and computation.\n2. The method is easy to implement with standard high-level neural network programming library.\n3. The results on ImageNet is persuasive. Although not shown in the comparisons, it outperforms the Rank-1 BNN which uses the BatchEnsemble as backbone, in terms of accuracy and other important metrics.\n\nWeaknesses:\n1. There are some missing results on ImageNet about efficiency. Why is that?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oIevNIYVqbf", "reviewer": "Reviewer_GYMm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an improved version of Deep Ensembled, i.e., Packed-Ensembles (PE), which cleverly leverages grouped convolutions to parallelize the ensemble into a single backbone to speed up the inference and training (eliminating the need to train multiple nns). The authors also extensively demonstrated the advantages of PE over the original DE. ", "summary_of_the_review": "In summary, I tend to accept this paper as it is a simple and effective approach to improve the performance of DE, but I would like to see a more thorough experiment section to examine the pro and con of PE. Thus, my recommendation is marginal accept. ", "strength_and_weaknesses": "Strength: \n\nSimple and effective implementation with extensive comparisons and discussions. Overall a well written paper. \n\nWeakness: \n\nThe introduction of group convolution into DE might be too incremental for ICLR. Also, it seems like this idea has been entertained before https://arxiv.org/abs/2007.00649.\n\nI am not sure that I totally get the conclusions in Table 7. All the performance discrepancies are very marginal (maybe some hypothesis testing is needed? But 5 is too small a sample size to draw conclusions), and I don't think better ECE, NLL etc. leads to better diversity. In fact, the fuction space diversity can be easily checked by the correlations among predictions of different ensemble members. Maybe the authors wanted to add experiment here, similar to what the MIMO work did.  \n\nOne downside of group conv is the scalability, i.e., with more ensemble members, the performance of group convolutions might drop. DE does not suffer from the same issue, and it would be interesting to test the performance of PE vs DE with an increasing number of ensemble members. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jq0MX9MPSk", "reviewer": "Reviewer_FMDg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce PackedEnsembles (PE) to improve the efficiency of deep ensembles. Although deep ensembles achieve SOTA results on a variety of benchmarks, this comes at significant inference-time and memory costs, since the same model architecture is repeated $M$ times for ensembles of size $M$. In this sense, this paper belongs to the same line of research as MIMO and BatchEnsembles (and extensions thereof), as well as mixtures of experts (MoEs) to a lesser degree.\n\nPackedEnsembles rely fundamentally upon the notion of grouped convolutions, first introduced in (Krizhevsky et al., 2012) for hardware constraint reasons, and used in more modern architectures such as ResNext. Grouped convolutions essentially partition the filters at a given convolution layer into $M$ groups, and each output channel is obtained by the filters in of such groups (as opposed to all filters).\n\nAs this reduces significantly the capacity of each individual \"ensemble member\", the authors introduce a parameter $\\alpha$, which multiplies the width of the original model (resulting in ensemble members of size rescaled roughly by $\\alpha / M$). Furthermore, each individual ensemble member may itself make use of grouped convolutions, parameterized by the number of groups $\\gamma$.", "summary_of_the_review": "This paper proposes a straightforward, well-motivated, and experimentally validated technique to approach the performance of deep ensembles while significantly improving upon the complexity of deep ensembles. The results are validated across several ResNet-based architectures and standard image benchmarks. This paper could be improved by including more baselines for the ImageNet experiments, as the Cifar datasets are significantly easier; however, achieving this can be costly.", "strength_and_weaknesses": "# Strengths\n- This paper proposes a simple technique to improve the inference and memory cost of deep ensembles. \n- Experimental results confirm the value of this method compared to commonly used \"efficient ensemble\" baselines such as MIMO and BatchEnsemble on Cifar datasets, across a variety of different metrics *and* architectures: the proposed PE model comes close, and sometime outperforms, the performance of deep ensembles on accuracy and uncertainty metrics, while drastically cutting down on the number of parameters and required operations.\n- The authors provide an extensive amount of ablation experiments to gauge the importance of the $\\alpha$ and $\\gamma$ parameters.\n\n# Weaknesses\n- It would be interesting to evaluate the other baselines on ImageNet tasks. I realize this can be an investment, but doing so would significantly improve the paper. Some results (for $M=4$ presumably, given reported parameter counts) are publicly available at the link referenced in footnote 2.\n- Although this is not necessary given the scope of this work, the authors may be interested in looking into the Mixtures of Experts literature, as decisions in data routing and partitioning may be also relevant.\n\n**Questions**\n- I'm curious about the position of MIMO in Figure 1: it looks like MIMO (4) has the same number of parameters as a single model, but much slower inference. Could you explain what causes this?\n\n**Nitpicks**\n- Equation (1) doesn't make use of the activation function $\\phi$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%