SuperWeight Ensembles: Automated Compositional Parameter Sharing Across Diverse Architechtures
Authors: Piotr Teterwak
Soren Nelson
Nikoli Dryden
Dina Bashkirova
Kate Saenko
Bryan A. Plummer
Venue: Submitted to ICLR 2023 (2023)
Keywords: efficent ensembles
anytime inference

Abstract: Neural net ensembles boost task performance, but have excessive storage requirements. Recent work in efficient ensembling has made the memory cost more tractable by sharing learned parameters between ensemble members. Existing efficient ensembles have high predictive accuracy, but they are overly restrictive in two ways: 1) They constrain ensemble members to have the same architecture, limiting their usefulness in applications such as anytime inference, and 2) They reduce the parameter count for a small predictive performance penalty, but do not provide an easy way to trade-off parameter count for predictive performance without increasing inference time. In this paper, we propose SuperWeight Ensembles, an approach for architecture-agnostic parameter sharing. SuperWeight Ensembles share parameters between layers which have sufficiently similar computation, even if they have different shapes. This allows anytime prediction of heterogeneous ensembles by selecting a subset of members during inference, which is a flexibility not supported by prior work. In addition, SuperWeight Ensembles provide control over the total number of parameters used, allowing us to increase or decrease the number of parameters without changing model architecture. On the anytime prediction task, our method shows a consistent boost over prior work while allowing for more flexibility in architectures and efficient parameter sharing. SuperWeight Ensembles preserve the performance of prior work in the low-parameter regime, and even outperform fully-parameterized ensembles with 17% fewer parameters on CIFAR-100 and 50% fewer parameters on ImageNet.

Under review as a conference paper at ICLR 2023
SUGERWEIGHT ENSEMBLES: AUTOMATED COMPOSI- TIONAL PARAMETER SHARING ACROSS DIVERSE AR- CHITECTURES
Anonymous authors Paper under double-blind review
ABSTRACT
Neural net ensembles boost task performance, but have excessive storage require- ments. Recent work in efficient ensembling has made the memory cost more tractable by sharing learned parameters between ensemble members. Existing ef- ficient ensembles have high predictive accuracy, but they are overly restrictive two ways: 1) They constrain ensemble members to have the same architecture, limiting their usefulness in applications such as anytime inference, and 2) They reduce the parameter count for small predictive performance penalty, but do not provide an easy way to trade-off parameter count for predictive performance without increasing inference time. In this paper, we propose SuperWeight En- sembles, an approach for architecture-agnosiic parameter sharing. SuperWeigth Ensembles share parameters between layers which have sufficiently similar com- putation, even if they have different shapes. This allows anytime prediction of het- erogeneous ensembles by selecting a subset of members during inference, which is flexibility not supported by prior work. In addition. SuperWeight Ensembles provide control over the total number of parameters used, allowing us to increase decrease the number of parameters without changing model architecture. On the anytime prediction task, our method shows a consistent boost over prior work while allowing for more flexibility in architectures and efficient parameter shar- ing. SuperWeight Ensembles preserve the performance of prior work in the low- parameter regime, and even outperform pallleaaaraeeeeeedd ensembles with 17% fewer parameters on CIFAR-100 and 50% fewer parameters on ImageNet.
INTRODUCTION
Ensembling aggregates the predictions of multiple models an effort to boost task performance (Atwood et al. 2020 Ostyakov & Nikolenko 2019) while also improving robustness, calibration, and accuracy Lakshminarayanan et al. 2017 Dietterich, 2000). However, it also require more memory and compute since multiple models need to be trained, stored, and used for inference. Ef- ficient ensembling reduces the total number of parameters by sharing parameters between members (e.g., (Lee et al.) 2015) Wen et al.) 2020 Wenzel et al. 20200). As illustrated in Figure Ia), these methods share parameters while introducing diversity by a) perturbing the shared parameters to cre- ate distinct layer weights or b) perturbing layer inputs for each ensemble member. significant drawback of these methods is they often make strong architectural assumptions, such as ensemble member homogeneity (i.e., each member is the same architecture), which limits their use. For ex- ample, homogeneous ensembles are ill-suited to tasks like anytime prediction because one only has n options for computational complexity, where n is the number of ensemble members. In contrast, heterogeneous ensembles select subset of its ensemble members to provide range of infer- ence times (e.g., 4 member heterogeneous ensemble can adjust to ()) (1) (t) + ( 15 levels of inference latency). Thus, we present SuperWeight Ensembles, which allow for parameter- efficient anytime inference by using heterogeneous ensemble members. Figure [cc shows that heterogeneous ensembles achieve state-of th-the--rr performance in anytime inference.
In this paper, propose SuperWeight Ensembles, method for efficient ensembling using auto- mated parameter sharing between diverse architectures. A key challenge we address learning where reuse parameters for heterogeneous ensembles, where ensemble members may have

---

Under review as a conference paper at ICLR 2023
Prior Work
c.) Flexible Inference Results
Manual Staring Patterns for Single Architecture Enseebble
CIFAR-100
Base Mode
Perturer
Emsenble
Member Architecture
Accuracy
b.) SuperWeight Ensembles (Our Work)
HNE(Multi-Depth;
Automated Sharing Pattern for Multi- Architecture Ensemble
77-
SiimableeMutttWWiitt
BetchEnsemmielldeniial Members)
Mi
M:
Mi
Mi
MIMOXSingle Model)
Member
75
SWE-HE Multi Depth(Withh(Ourss)
Architecture
SWE-HE Multt-WitthhOurs)
75
"Same colors indicate shared params Different shades indicate perturbed weights
0.0
0.6
0.8
144
Average Inference Time (ms)
Figure 1: Comparison to prior work. (a) Prior work in efficient ensembling (e.g., (Lee et al. 2015
Wen 2020 Wenzel al. (2200) uses hand-crafted strategies that required ensemble members to have identical architectures and adds diversity by perturbing weights and/or features. In contrast, (b) illustrates our SuperWeight Ensembles, which learn effective soft parameter sharing between members, even for diverse architectures. As shown in (c), this enables our approach to support a range of inference times while outperforming prior work in efficient ensembling and anytime
inference (Ruiz & Verbeek] 2021] Havasi et al. 2021; Wen et al.] 2020 Yu et al. 2019 Yu &
Huang 2019a) on CIFAR- 100 using WRN-28-5 ZZaoruyko Komodakis 2016).
architectures that vary both the number of layers and/or channels. Prior work has explored strategies where parameters are shared between members that have constant depths, but varying widths YYu & Huang 2019b Yu et al. 2019 Wang et al. 2020 Li et al. 2021), as well cases where the widths are held constant, but the depths vary (Ruiz & Verbeek [002
(002/KKal Kaya et al. 2019 Huang et al. 2018 Yang et al. 2020 Wu et al. 2018). In contrast, as illustrated in Figure (b, SuperWeight Ensem- bles members can vary in both the widths and depths of the network. Our approach is built on the intuition that, despite being independent, two models trained the same task likely have detect similar features. However, these features may occur in different location(s) (i.e., different layers) across ensemble members, especially when members represent distinct networks architectures. Our goal then is to detect where these recurring computations exist in our heterogeneous ensemble.
neural network can be seen as a composition of feature detectors (Savarese & Maire 2019 Motivated by this and the Hebbian principle that "neurons which wire together fire together", we propose a technique that groups detectors that frequently co-occur, which we call SuperWeights. Our SuperWeights are analogous to SuperPixels (Ren & Malik 2003), which represent semantically coherent region of pixels within an image, but where our SuperWeights represent feature detectors which frequently co-occur. An example of SuperWeight would be filters of CNN that capture unique pattern. Prior work has shown these patterns may repeat across many types classification networks (Raghu et al. 202T), which take advantage of in approach. A single layer may require a concatenation of several SuperWeights each of which may be shared with layers within the same ensemble member or across members. To learn these complex sharing patterns, we propose a gradient analysis approach for determining where Super Weights can be reused.
To further improve our model's parameter efficiency, also take advantage of template mixing (e.g., [Bagherinezhad et al.] (2017): PPlummer et al. (2022); [Savarese & Maire (011)). We con- struct Suighheightss using a weighted linear combination templates made up of trainable param- eters which we call Weight Templates. This creates a hierarchical representation for neural network weight generation, where we begin by combining Weight Templates to create SuperWeights, then concatenating together SuperWeights to create the layer weights used by each ensemble member. Two SuperWeights using the same Weight Templates may learn different linear coefficients for combining templates, allowing for per-layer SuperWeight sub-specialization. This template mix- ing can boost performance when reusing the same parameters many times, as each combination of templates use unique set of coefficients, and it also helps support wide range of parameter budgets. Adjusting the parameter count can accomplished by increasing or decreasing the num- ber of shared templates, allowing for parameter budget performance trade offs with change in
2

---

Under review as a conference paper at ICLR 2023
architecture. In contrast, prior work in efficient ensembling (e.g. Lee et al. (2013); Wen et al. (2020)
Wenzel et al. (2020)) can only support a single parameter budget per architecture. Our contributions are follows:
We propose SuperWeight Ensembles, a new technique for automated parameter sharing across layers and ensemble members, even when members have different architectures.
We demonstrate that using our approach with homogeneous architectures matches perfor-
mance of prior parameter efficient ensembles on CIFAR-10 (Krizhevsky et al. 22009) CIFAR-
100 (Krizhevsky et al. 2009). CIFAR-100-C (Hendrycks & Dietterich, 2019). and Ima-
geNet (Deng et al. 2 in the low parameter budget setting.
We also illustrate Super Weight Ensembles' ability to tradeoff between its parameter budget and ensembles while still using 17% fewer parameters on CIFAR-100 and 50% fewer parameters on ImageNet.
performance to surpass mully-parameteriin
We show that using our approach with heterogeneous architectures achieves state-of-the-art any- time inference results on CIFAR-I0 and CIFAR-I00 by evaluating a subset of ensemble members for different inference budgets, an application not supported by prior work in efficient ensembling.
2 RELATED WORK
Ensembles have been shown to improve predictive accuracy and uncertainty estimates of classifica- tion models (Dietterich) 2000 Lakshminarayanan et al. 2017). Prior work has shown this is due, in part, to the fact that different neural network initializations result in diverse final solutions (Fort eet al. 2019). which helps to decorrelate errors and boost performance. In fact, [Wasay & Idreos (2021) show that ensembling many smaller models sometimes is a more efficient way to use ad- ditional compute than making single model larger. Ensembles' major drawback is that they memory and compute intensive since multiple models must be trained and used for inference. To address this, many efficient ensembling methods share parameters between members (e.g.. (Havasi ett al. 2021 Lee et al.] 2015 Teterwak, 2014 Wen et al. 2020), either by introducing a branch- ing point between shared and unshared layers (Lee al. Teterwwak 2014) or, more recently, perturbing the input features and/or shared parameters to create ensemble members (Gal & Ghahra- mani] 2016: Havasi al., [021]] Wen et al.) 2020). However, as mentioned in the introduction, these methods use haad-crafted strategies that rely on each ensemble member having the same ar- chitecture. Additionally, they require architecture changes in order alter parameter budgets since the parameter sharing pattern is fixed. In contrast, our SuperWeight Ensembles automatically learn how to share parameters between multiple architectures given parameter budget, enabling us to generalize to more tasks and settings like anytime inference.
In anytime inference the goal is to make a high performing prediction within a given inference time budget. To address this task, most methods learn to make predictions using subnetworks that have varying computational requirements to support a range of inference times (e.g., (Bolukbasi et al, 2017 Tuzzolino et al. [222 Ruiz& & Verbeek 1021 Wu al. 2018 Yu & Huang, 2019b] Yu al.
2019 Huang et al. (018)). These subnetworks vary either the network width Yu & Huang 2019b Yu et al. 2019) or network depth (Bolukbasi al., 2017 Juzzolino et al. [021 Ruiz & Verbeek 2021 Wu et al. 2018 Huang et al. 2018). In contrast, SuperWeigh Ensembles learn heterogeneous ensembles that can vary both width and depth, or members could use a completely different network architecture altogether, increasing the diversity and inference times we can support. In addition, unlike in efficient ensembling, where individual models may perform poorly but still obtain good performance since they have high diversity across the set of models, in anytime inference the perfor- mance of individual models hold greater weight since they are meant to support specific inference time budget. SuperWeight Ensembles obtain state-of-the-art results in both.
Neural Architecture Search (NAS) is another approach for finding high performance networks. Neural Ensemble Search (Zaidi et al. 2020) searches for optimal ensemble member architectures by evolving ensemble members. Single-shot NAS methods search subnetworks over a pre-trained supernetwork (Yu & Huang 2019a; Yu et al. 2020a). However, strict parameter sharing over child sub-networks is suboptimal. Therefore, few-shot NAS (Zhou et al.] 2022 Hu et al. 2022) has been introduced, which train several supernetworks. GM-NAS (Hu et al. 2022
a gradient- based splitting criterion to find supernetwrrs,, and CLOSE (Zhou et al., (022) decouples parameter assignment from operations. Nevertheless, NAS is very costly due to search over architectures.
3

---

Under review as a conference paper at ICLR 2023
Super Weight Cluster
(Sectio 3.3)
SuperWeight Construction (Section 3.1)
Layer Weight Construction (Section
Model Constraction (Section 3.3)
Woggh Tomplaii
0
Laye:
Menter
Woight Templaies
6'
Laver
Waighi Templeass
Layer 3
Mentber
Super Weight Cluster 2 (Section 3.3)
6
Weight Templatee
Layor 4
DDiflom shasks reproure ahanol Wäght Tompbess with dilfern livear melliiia
+ *SpperWecigtt on bt comaiaatta in malliple dinemiiee, for staring baween layons of differee isputand sugu clamels
Figure 2: SuperWeigh Ensembles overview. Left:: Weight Templates are clustered into Super- Weight Clusters; layers
only share SuperWeights from Weight Templates in the same cluster (Section (.2.2) Center Left: SuperWeights are constructed from linear combinations of Weight
Templates (see Eq. (I) using the SuperWeight Clusters learned in Section 3.3 Center Right: The
SuperWeights assigned to a layer are then concatenated to generate layer weights. We automatically identify when we should use the same linear coefficients or different coefficients for generating layer's SuperWeight using the procedure in Section 3.2.2 Right: These layer weights are then as- sembled into ensemble members, which can have diverse architectures.
3 SUPERWEIGHT ENSEMBLES
Given a parameter budget B and a set of target model architectures M, one for each ensemble member, our goal to generate the weights of the layers of each member that maximize task per- formance. To address this task, we propose SuperWeight Ensembles, an automated approach for sharing parameters between ensemble members, even when they contain diverse architectures (i.e,, ensemble members may vary in both the widths and depths from each other, or can be different architecture families). Our approach handles this challenge by constructing layers from one or more SuperWeights, which are linear combination of templates and get reused by multiple layers and cap- ture a single operation on the input features (e.g., edge or texture detectors). Then, to generate the weights for single layer, we must first construet SuperWeights from the trainable parameters held by Weight Templates (discussed in Section B.I), and then concatenate together all SuperWeights used by the layer create its final weights (process illustrated in center-right column of Figure
This hierarchy provides a very parameter efficient representation, as the trainable parameters used by each Weight Template may be shared by multiple SuperWeights, while still being diverse as different coefficients are used to combine the Weight Templates into the SuperWeight, perWeights themselves can also be shared by one or more layers, either from the same ensemble member, or different ensemble members. However, some layers cannot be shared without substan- tial performance degradation. Therefore, we learn where parameter sharing most advantageous by starting with strict sharing where many layers share parameters, then removing sharing where it hurts performance. The overall algorithm consists of the following steps:
The Su-
1. We start by determining the sizes of the SuperWeights needed for the network (Section 3.2. and construct an initial sharing pattern based on relative depth.
2. After small number of training epochs, we refine this sharing pattern using gradient analy- sis to determine which layers should share Weight Templates, resulting in the creation of our SuperWeight Clusters (Section 3.3)
3. We reinitialize the entire network using our SuperWeigh Clusters and learn which Weight Tem- plate Coefficients (i.e.. which SuperWeights) should be shared across layers (Section 3.2.2).
4. We resume training until we achieve our stopping criteria e.g. number of training epochs.
3.1 GENERATING SUPERWEIGHTS FROM TEMPLATES
Our ensemble uses a set of Weight Templates as a fundamental building block for transforming shared parameters into layer-specific weights the SuperWeight. Traditional methods of (hard)
4

---

Under review as a conference paper at ICLR 2023
parameter sharing directly reuse weights (e.g.. (Collobert & Weston 2008 Havasi al. 2021
Gal & Ghahramani 2016 Zhang et al. 2014)). but this limits ensemble diversity as shared layers then encode the same learned function. Instead, we take inspiration from recent work in cross- layer parameter sharing (e.g., (Bagherinezhad et al. 2017] Plummer et al. 2022; Savarese & Maire 2019)) that performs template mixing. In these works, parameters and layer weights are decoupled; each layer's weights are a linear combination of parameter matrices called templates. We build each SuperWeight in our SuperWeight ensembles in the same way, More formally, let us consider generating SuperWeight 0g. where i refers to SuperWeight & generated using Weight Templates g- g contains set of templates Ti....N with the same dimensions as the SuperWeight 0y. which is generated via a linear combination using coefficients a², i.e.,
0
1
Multiple SuperWeights can share g while having different coefficients a² To create layer's weights, one or more SuperWeights are generated using Eq. ( and then concatenated together (details in Section 3.2.1). Templates and coefficients are learned jointly via gradient descent.
Plummer et al. (202)) also shared parameters via template mixing between layers of different sizes, but they would directly reshape the available parameters into templates of the target layer size. This means that it would be very difficult to accurately represent subnetworks within a larger network. For example, let us assume we are generating the weights for two networks that are identical except that network X is twice as wide as network Y. Using the approach from Plummer et al. (2022), we would either have to generate layers for X and Y independently, or we would generate the layers of X, and then slice out the weights that would fit network Y. We found that the first option was difficult to optimize that resulted in lower performance in our experiments. The second option would reduce diversity, as network X would contain exactly the same weights as network Y. Instead, our SuperWeight Ensembles can directly optimize any subnetworks, like in the second option in our example above, but while minimizing any diversity loss (details in the next section).
3.2
CONSTRUCTING NETWORKS FROM SUGHRWEIGHTS
3.2.1
DETERMINING SUPERWEIGHTT SIZES
The target sizes for our SuperWeights are based on layer shapes. Consider the case where we are generating weights for a set of layers f.....s, where each layer is a different size. We create an initial set of SuperWeights by sorting the layers in order from smallest to largest. The first SuperWeight we would create would be the same size as the smallest layer l,, and every layer within that SuperWeight Cluster that was the same size as & would initially share the same SuperWeight. Then, when we generate the weights for the next layer fi-1, we assume we have the SuperWeight from layer f. Thus, the SuperWeights for layer l++1 need only generate the additional weights required beyond those provided by the SuperWeight from 4. For example, if é, needed 100K parameters and l.+1 needed 400K, then the new SuperWeight would generate 400 100 = 300K weights. If l is larger than f, in only single dimension, then only single new SuperWeight is generated. If it's larger in both input and output channels, f4+1 generates two new SuperWeights The first is to extend the size in input channels dimension, and the second is concatenated in the output channel dimension. See Layer I in the center-right column of Figure 2 for a graphical illustration. Any subsequent layers would follow the same process, but would have all the weights from the previous layers (i.e., lit2 would have 400K weights from 4 100K of which originally came from E.
3.2.2 LEARNING WHEN SUGHRWEIGHTS SHOULD SHARE WEIGHT TEMPLATE COEFFICIENTS
We use the initial set of SuperWeights from Section 3.3 for the first E epochs of training (we found
E = 10 worked well). After that, we analyze which layers that share the same Super Weight benefit from decoupling linear coefficients combining Weight Templates. Similar to recent multi-task work Yu et al.) 2020b). we compute layers which have confficting gradients of the loss with respect to These layers playing gradient tfg-of-war, hurting optimization. In other
linear coefficients words, if the sum of gradients from two layers is close to zero, no learning occurs. To prevent this, we cluster layers by gradient similarity. We place the layers sharing the same SuperWeight
5

---

Under review as a conference paper at ICLR 2023
into a priority queue by the cosine similarity with respect to the gradients of the shared coefficients aggregated over an epoch. We pop the first two layers, Es, lg off the queue and checking whether their cosine similarity exceeds some threshold B, i.e.,
0c
ac t
Veoef. = cos
> B,
(2)
where a is the shared coefficient corresponding to layer f,, and C is the loss. If both layers belong to an existing group and they satisfy Eq. ( we merge their groups, meaning that the layers within a group share SuperWeights. If pair of layers satisfy Eq. 2), but only one layer is in an existing group, then we add the new layer into that group's s set. Once we have a pair of layers that does not satisfy Eq. (2. split any ungrouped layers into their own individual groups (e.g., 4 ungrouped layers would result in 4 additional single layer groups). We set 3 via grid search on a validation set. See Appendix B for pseudocode for our priority-queue assignment procedure. After we have identified the groups of similar layers, each obtains their own copy of the coefficients for that SuperWeight o* and we resume training. This creates new Super Weigh for each group of layers whose gradients point in a similar direction, allowing for layer specialization.
3.3 LEARNED PARAMETER SHARING WITH SUGHR CLUSTERS
In Section 3.2.2 we describe a procedure for decoupling linear coefficients, softening the sharing be- tween layers. However, some layers may not find any sharing beneficial. To account for such cases, we first separate layers into SuperWeight Clusters, sets of layers where sharing is most beneficial, and do not share parameters between layers in different clusters. Plummer et al. (2022) automati- cally learned where parameter sharing could be effective by forcing layers to share templates. but not coefficients, and then used K K-means clustering the coefficients to group layers that produced similar weights (due to having similar coefficients). This has two major drawbacks. First, layers clustered based on the final value of the coefficients, but two layers may be drifting apart, and waiting until convergence is also not optimal due to the resulting increase in training time. Second, sharing parameters across all layers requires ppsamplin/ddwwsaapliii different numbers of weights, which is accomplished using bilinear interpolation. This effectively warps the receptive field of the parameters, so they were not truly comparable across layers, while also being computationally expensive to do every layer in a forward pass.
templates for layers needing
Our SuperWeight Ensembles are designed to address these issues in prior work. Specifically, since we break layers into combinations of SuperWeights, they do not need to be upsampled/downsamppa since the larger layer would simply share a subset of its weights with the smaller layer. In addition, instead of each layer that shares SuperWeight getting their own coefficients and then clustering them (effectively adapting the approach of Plummer et al. (2022) to use our SuperWeights, which we compare to in Table 3). we force all layers to share coefficients so that we can tell if two layers should share the Super Weight coefficients by analyzing their gradients.
More formally, given set of layers 41....s in our ensemble, our goal is to determine which layers can share parameters. We begin by creating a set of initial SuperWeight Clusters that we will refine. First, we split layers fi....s into groups by operation (e.g., convolutional and fully connected layers get split into different clusters). When instantiating convolutional layers, we consider layers with different kernel sizes as performing different operations. so convolutional layers with x 1 and 3 kernels never belong to the same group. Since shallow and deep layers can be assumed to perform entirely different types of computation, we create N bins based on the depth of layer and ensure layers that are in different depth bins are in different groups Layers from different members are placed in the same bin; ensuring cross-member parameter sharing.
Once we obtain the initial layer groupings using the process above, we create a set of shared Super- Weights using the procedure from Section 3.2.1 independently per group. The set of SuperWeights belonging to a single layer grouping is called SuperWeight Cluster. At this point, every set of Weight Templates has exactly one set of coefficients. For example, if cluster has three layers that all need SuperWeight of 100K dimensions, then each layer would use shared Weight Template that produces the same 100K dimension SuighWeight. Thus, we can infer that if the gradients of the loss w.r.t. the layers sharing SuperWeights are misaligned, then the model will have optimiza- tion difficulties. To capture this, we modify the procedure in Section 3.2.2 While Section 3.2.2 discussed gradient conflicts between coefficients, in this case the concern is with overly restrictive

---

Under review as a conference paper at ICLR 2023
Table Homogeneous ensembling comparison on CIFAR-100 (clean) Krizhevsky al. 2009) CIFAR-100-C (corrupt) (Hendrycks & Dietterich 2019). and CIFAR-I0 using WideRes- Nets ZZgoruyko Komodakis 2016) averaged over three runs. (a) shows that our approach out-
performs prior work in efficient ensembling (Wen et al. 2020 Havasi et al.) (021). (c) compares the performance of increasing the number of parameters (without changing the architecture) using our approach compared to Deep Ensembles, which trains 4 independent networks members.
CIFAR- 100 (clean) Params Top-1 f NLL + ECE 4 Top-1 1 NLL | ECE 4 Top-1 NLL ECE
CIFAR-10
Method
(a) WRN-28-10
36.5M
79.8
0.875
8.6
51.4
2.70
23.9
96.0
0.159
2.3
BE
36.5M
81.5
0.740
5.6
54.1
2.49
19.1
96.2
0.143
2.1
BE+ + EnsBN
36.5M
81.9
2.8
54.1
n/a
19.1
96.2
n/a
1.8
MIMO
36.5M
82.0
0.690
2.2
53.7
2.28
12.9
96.4
0.123
1.0
Thin Deep Ensembles
36.5M
81.5
0.694
1.7
53.7
2.19
11.1
96.3
0.115
0.8
SWE-HO (Ours)
36.5M
82.2
0.702
2.7
52.9
2.17
10.3
96.3
0.120
0.8
SWE-HE (Ours)
36.5M
82.4
0.663
3.0
53.0
2.17
10.0
96.5
0.115
0.8
(b) Deep Ensembles SWE-HO (Ours)
146M
82.7
0.666
2.1
54.1
2.27
13.8
96.6
0.114
1.0
120M
82.9
0.666
2.2
54.7
2.00
10.3
96.6
0.119
0.8
sharing of Weight Templates. Thus, we compute gradient similarity over SuperWeights rather than coefficents. However, some layers may be composed of multiple SuperWeights, only some of which may share with other layers in the initial cluster. For example, one layer may be composed of two SuperWeights, while another layer j may use three (two of which are shared with layer i). Va, refers to the set of SuperWeights shared by layers and j. W; refers to the instantiated weights of layer Then the gradient similarity between W; and W, would be computed as:
OC oW, OL a aw, Ooi,"" oW, Oos,s
vssw COS
>
where is hyperparameter representing the minimum threshold for which two layers that share Vi should remain in the same cluster. We found - = 0.1 worked well in our experiments. We first train our model for 20 epochs (10% of the training time in our experiments) using our initial SuperWeight Clusters (step 1 of training algorithm at the end of Section 3 Then we create our refined SuperWeight Clusters by using Eq. (3)
replacement for Eq. (2) for the priority-queue based clustering from Section [32.2]] (step 2 of our algorithm). Following Plummer et al.] (2022), we reinitialize our ensemble and re-train from scratch with our new SuperWeight Clusters. Finally, after some more training, we decouple linear coefficients (step 3 of our algorithm, which is described in Section 3.2.2,, and continue training (step 4 of our algorithm).
4 EFFICIENT ENSEMBLING EXPERIMENTS
We first compare our method to other efficient ensembling methods in a common application sce- nario; where all ensemble members share parameters but have the same architecture. In this special case; each layer consists of only single SuperWeight. Implementation details are in the Appendix. Datasets. We evaluate our method on three standard benchmarks: CIFAR-10 (Krizhevsky et al. 2009). CIFAR- -00 (Krizhevsky et al., 2009), and ImageNet (Deng et al.) 2009). Additionally, as an advantage of ensembles is their robustness to out-of-distribun our method CIFAR-100-C (Hendrycks & Dietterich) 2019), which is version of the CIFAR-100 test set which is corrupted with distortions such as Gaussian blur and JPEG compression. Metrics. For given parameter budget. compute Top-1 accuracy in addition to calibration metrics (Guo ct al. 2017): Negative Log-Likelihood (NLL) and Expected Calibration Error (ECE).
data, we evaluate the robustness of
Results. Table Iaa a) compares our SuperWeight Ensembles (SWE-HO) the CIFAR-100 CIFAR- 100-C, and CIFAR-10 10 with prior work in efficient ensembling. Note that all the methods boost performance single model without requiring additional model parameters. However, our Su- per Weight Ensembles outperforms all other methods on CIFAR- 100 when using 36.5M parameters. Unlike methods like BatchEnsemble (BE) (Wen al. 2020 and MIMO (Havasi et al. 2021) which

---

Under review as a conference paper at ICLR 2023
Method
Params
Top-1
Table 2: Homogeneous ensembling com-
parison on ImageNet using ResNet-50. (a) shows that our approach outperforms or is
(a) ResNet-50 (He ct al.
2016)
25.6M
76.4
BE (Wen al. 2020
25.6M
76.7
on par with prior work in efficient ensem- bling. (b) increases the number of parame- ters (without changing the architecture) us-
MIMO (Havasi et al.
2021
25.6M
77.5
SWE-HO (Ours)
25.6M
77.2
(b) Deep Ensembles
102.4M
77.5
ing our approach compared to Deep Ensem-
SWE-HO (Ours)
51.2M
77.9
bles. See Section for discussion
cannot change the number of parameters without making architecture adjustments to the widths and/or number of layers, our SuperWeight Ensembles can support any parameter budget without requiring architecture changes by adjust the number of templates or the amount of sharing between layers. Thus, if the number of parameters are not a concern, one can simply provide our approach with more parameters to boost performance as illustrated in Table [[]], where our approach can outperform standard Deep Ensembles, which trains completely independent networks for ensemble members, while still retaining 17% fewer parameters. Table I[a) also shows that heterogeneous Su- perWeight Ensembles (SWE-HE). consisting of a WRN-34 -3-8,, 28-12. 28-10, and 28-8, outperforms SWE-HO in over half of the metrics while also supporting many inference times. A similar pattern is reported in on ImageNet. Specifically, Table ([a) shows we obtain on par or better perfor- mance than prior work, demonstrating that our approach can generalize to other settings including large scale recognition tasks. [b) also reports better performance by boosting parameter bud- gets, where we outperform standard Deep Ensembles with 50% fewer parameters. We do not include MIMO and BE in Table [b)) and Table b) as increasing number of parameters increases training and inference time, due to necessary increased width or depth of networks. This is not the case in SuperWeight Ensembles. In addition, methods like MIMO and BE cannot learn heterogeneous ensembles for anytime inference (as we do with SuperWeight Ensembles in Section 4.1) since their approaches rely on their ensemble members all being the same architecture.
4.1 ANYTIME INFERENCE EXPERIMENTS
In anytime inference the goal is to make high-performing predictions within a given time budget. As the time budget increases, good method will use the additional time improve performance. A homogeneous ensemble, like those used in the experiments in Section 4 would only provide lim- ited time budgets as each member has an identical computational complexity. Thus, their flexibility is limited because one has only M options for computational complexity, where M is the number of ensemble members. However, for a heterogeneous ensemble this limitation removed with an ensemble since each member provides a different inference time resulting in (M) (M) +...+ + (M) M possible inference times to choose from. This results in a highly effective anytime inference model. We note that although we evaluate our individual ensemble members in series, our method is trivial to parallelize increase inference speeds by using multiple GPUs This is unlike many early- carly-exit anytime inference methods (e.g.. (Bolukbasi et al., 2017 Huang et al. 2018 Kaya et al.) 2019 Li et al. 2019; "Teerapittayanon al. [016]] Yang et al. 2020p), which are intrinsically serial. We use two settings for our Heterogeneous Weight Ensembles (SWE-HE) in our experiments: SWE- HE -Multi-With, which trains three member WRN (Zagoruyko & Komodakis 2016) ensemble WRN-22-(7.4.33, and WWW--HH-Mulli-Depphh-We
a four member ensemble WRN-28-17,44 16- [7.4]. SSNs (Plummer al. 2022) and Slimmable (Yu et al., 2019) use the same ensemble con- figurations, but other approaches use method-specific strategies (e.g.. HNE (Ruiz& & Verbeek 202T) trains a set of tree-nested ensembles). See Appendix for additional details.
Results. Figure 3 reports top-1 accuracy vs. average inference time using single P100 GPU on CI and CTFAR-100 (Krizhevsky et al. 2009). When comparing to other dynamic width methods such as Slimmable Yuu et al. 2019). Universally Slimmable YYu & Huang. 2019b) and AutoSlim (Yu & Huang 2019a) models, SuperWeight Ensembles perform on par or better than them for inference times they support, but our approach can provide a wider range of inference times that improve performance. We reiterate that other efficient ensembling methods such as BatchEnsemble (Wen et al.) 2020) and MIMO (Havasi et al., (021) are not suitable for Anytime Inference, because each ensemble member has the exact same inference time. This results in a very limited set of possible inference times (see Figure []. We also significantly outperform the tree-based ensemble
8

---

Under review as a conference paper at ICLR 2023
CIFAR-100
CIFAR -1
96.5
96.0
Accuracy
79
95.
SEDRet (MUB) Destr)
NSDAM: Deptit
HIE (MUR) Depdl
HAE (HUIT OrgoTo
Sámimable MNLE WIHDO
95.
Siimmatle INU WATTH
77
94.5
SINE-EE wwa.taathh...o
SWC IE Withhoun
75 0.2
94.0
0.4
Average Inference Time (ms)
Average Inference Time (ms)
Figure 3: Anytime Inference comparison using 1op-1 accuracy vs. inference time (ms) averaged
over three runs using single P100 GPU on (IFAR-100 and CIFAR-10 (Krizhevsky et al. with WRN backbones (Zagoruyko & Komodakis
2009)
2016 for most methods (Huang et al. 2018
Plummer et al. 2022 Yu & Huang/ 2019a Yu et al.
2019 Yu & Huang 2019b), except HNE (Ruiz
& Verbeek 2021) which modifies ResNer-50 (He et al. 2016).
Table Multi-width Super Cluster creation comparison using top-1 accuracy on CIFAR- 100 averaged over three runs. See Section 4.1 for discussion
Method
WRN-28-3 WRN-28-4
WRN-28-7
Full Ensemble
Shared Coefficients
77.3 + 0.09 78.0 0.15 79.7 0.10
80.4 + 0.06
Single SuperWeight Cluster
76.3 0.37 77.6 0.30 78.8 0.20
81.1 + 0.17
Depth-binning
76.8 + 0.15 77.7 + 0.17 79.2 0.19
81.4 0.15
Coefficient Clustering (Plummer et al. 2022)
76.1 0.19 77.0 0.15 78.9 0.15
80.7 + 0.14
SWE-HE (Ours)
78.9 + 0.26 79.7 = 0.08 80.9 0.03
81.5 1 0.13
HNE Ruiz & Verbeek 2021) on both datasets, as well as the early-exit model MSDNet Huang et al. 2018 Lastly, we use SuperWeight Ensembles to construct a dynamic width and depth network. Our main comparison is to adaptation of Shapeshifter Networks Plummer et al. (022) to ensemble dynamic widths and depths. We show consistent improvement over Shapeshifter Networks across inference times. These results demonstrate that SuperWeight Ensembles can share parameters across members of diverse architectures more effectively than other approaches.
Comparison of SuperWeight Clustering methods. Table demonstrates the effectiveness of our SuperWeight Clustering approach described Section We provide four baselines: Shared Coef- ficients, which learns SuperWeight Clusters, but shares coefficients between all layers (i.e., removing Section 3.2.2): Single SuperWeight Cluster, which allows layers to have their own coefficients, but does not learn clusters (i.e., removing Section 3.3 Depth-binning, heuristic used for our initial SuperWeight Clusters Section 3.3 and Coeffcient Clustering (Plummer et al. 2022), which clus- ters coefficients in Eq. (T) to group layers. Our results show that approach outperforms all of our baselines. Notably, we find we that Coefficient Clustering performs in par or worse than other baselines. contrast, our gradient analysis approach (Section 3.3) takes into account the direction of change rather than just the current value of the coefficients. Thus, we obtain a 2% gain on indi- vidual models and small boost to ensembling performance with our approach (Table 3). We show a visualization of Super Weight cluster assignment in Appendix C.6
CONCLUSION
We introduce SuperWeight Ensembles, a method for parameter sharing in heterogeneous ensembles. SuperWeight Ensembles outperform existing anytime prediction work by leveraging gradient infor- mation for parameter sharing. Our automatic sharing improves single member performance by 2% compared to the baselines. SuperWeight Ensembles also match performance in the low-parameter regime, compared to prior work. When we add parameters, we outperform even deep ensembles on ImageNet with 50% the parameters. We believe that SuperWeight En- sembles a promising step forward in parameter-efficiene Future work will include more deeply exploring architecture diversity; Gontijo-Lope et al.] (202I) show that model architecture hetero- geneity can be key contributor ensemble diversity challenging tasks.
efficient ensembles

---

Under review as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Ensembles are more robust and better calibrated than single models, which can translate directly to fairer and safer deployments. Nevertheless, we caution readers that even more robust and accurate models should not be trusted blindly. It is prudent to thoroughly audit all models before and during deployment, and consider how they may affect the lives of others.
Similarly, effective anytime inference models allows to use less compute, potentially running networks in efficient modes and conserving energy. However, they can also be used to maximize the use of compute if it's available, using energy with corresponding drawback. We urge readers to be aware of the carbon and energy footprint of the models they train.
7 REPRODUCIBILITY STATEMENT
We provide all training details in the Appendix, and provide code in the Supplementary Materials. At acceptance, we will provide GitHub repository for readers.
REFERENCES
James Atwood, Yoni Halpern, Pallavi Baljekar, Eric Breck, Sculley, Pavel Ostyakov, Sergey I Nikolenko, Igor Ivanov, Roman Solovyev, Weimin Wang, et al. The inclusive images competition. In The NeurlPS Competition, pp. 155-186. Springer, 2020.
Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. Lenn: Lookup-based convolu- tional neural network. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, 2017.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learn- ing Research, pp. 527-536. PMLR, 06-11 Aug 2017. URL https: /tttittss:/pprceeeee http///poceedeng/e///// .ng.mlrss............ mlr.
html
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, 2008
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Terrance Vriii and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: /70..0078... 2017.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv: 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pp. 1050-1059.
PMLR, 2016.
Raphael Contijo-Lopes, Yann Dauphin, and Ekin D Cubuk. No one representation to rule them all: Overlapping features of training methods, arXiv preprint arXiv:21 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2023
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Lak- shminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference Learning Representations, 2021. URL
https: /pwnnevieee www.rreiiewe.eee. net forum? 3 9
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations, 2019.
Shoukang Hu, Ruochen Wang, Lanqing Hong, Zhenguo Li, Cho-Jui Hsieh, and Jiashi Feng. Gener- alizing few-shot nas with gradient matching. arXiv preprint arXiv:2203 2022.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. In International Confer- ence on Learning Representations, 2018. URL hattps::/operrevieww net/ forum?id
Hk2almxAb
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Michael Juzzolino, Michael C Mozer, and Samy Bengio. Improving anytime prediction with parallel cascaded networks and temporal-difference loss. Advances in Neural Information Processing Systems, 34, 2021.
Yigitean Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In International conference on machine learning, pp. 3301- 3310. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint orXiv:/511.033 2015.
Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8607-8617, 2021.
Hao Li, Hong Zhang, Xiaojuan Qi. Ruigang Yang, and Gao Huang. Improved techniques for training adaptive deep networks. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1891-1900, 2019.
Pavel Ostyakov and Sergey Nikolenko. Adapting convolutional neural networks for geographical domain shift. arXiv preprint arXiv:1901.06344 2019.
Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, and Kate Saenko. Neural parameter allocation search. In International Conference on Learning Representations (ICLR), 2022.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems,
2021.
Xiaofeng Ren and Jitendra Malik. Learning classification model for segmentation. In Computer Vision, IEEE International Conference on, volume 2, pp. 10-10. IEEE Computer Society, 2003.
11

---

Under review as a conference paper at ICLR 2023
Adria Ruiz and Jakob Verbeek. Anytime inference with distilled hierarchical neural ensembles. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter
sharing. In International Conference Learning Representations, 2019. URL https:
//openzeview. www.e//oomm.dddrgyie.....
Surat Teerapittayanon, Bradley McDanel, and Assiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464-2469, IEEE, 2016.
Piotr Teterwak. Shared roots: Regularizing neural networks through multitask learning. Dartmouth College Technical Report, 2014.
Xin Wang, Fisher Yu, Lisa Dunlap. Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in artificial intelligence, pp. 552-562. PMLR, 2020.
Abdul Wasay and Stratos Idreos. More or less: When and how to build convolutional neural network ensembles. In International Conference Learning Representations, 2021. URL https:
///ppnrevieww.. htte:t/ooommmmdmd222220022
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations, 2020.
net wtt.:/orrrmmm/ URL httphtttps//oppnneeette wttp://wwwweeeee/oroooddddd..e. wiw.rrummidsklllrrrrrr..
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for
robustness and uncertainty quantification. Advances in Neural Information Processing Systems, 33:6514-6527, 2020.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8817-8826, 2018.
Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2369-2378, 2020.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint 1903v 2019a.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques.
In Proceedings of the IEEÉ/CVF international conference computer vision, pp. 1803-1811, 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networss. In International Conference Learning Representations, 2019. URL /tRL/itttts://peerevieee
net pest/fose htte///oommmmmddddooee H1M
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In European Conference on Computer Vision, pp- 702-717. Springer, 2020a.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836. 2020b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. Neu- ral ensemble search for performant and calibrated predictions. arXiv preprint arXiv:2006.08577 2:3, 2020.
12

---

Under review as a conference paper at ICLR 2023
Zhampeng Zhang. Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision, pp. 94-108. Springer, 2014.
Zixuan Zhou, Xuefei Ning. Yi Cai, Jiashu Han, Yiping Deng, Yuhan Dong, Huazhong Yang, and Yu Wang. Close: Curriculum learning on the sharing extent towards better one-shot nas, arXiv preprint ar arXiv:220..07866 2022.
A IMPLEMENTATION DETAILS
For the readers convenience, we review the training details of both CIFAR Krizhevsky et al. (2009) and ImageNet (Deng et al., 2009) experiments. Note these details were used for both the efficient ensembling (Section and the anytime inference results (Section 4.1.
CIFAR: For CIFAR- 10 and CIFAR- 100 experiments, we train for 200 epochs with an initial learning rate of 0.1. We use SGD with Nestorov momentum value of 0.9. We use weight decay value of 5e-4 on all parameters except weight coefficients. We decay the learning rate by factor of 0.2 at 60, 120,and 180 epochs. We use a batch size of 128, and use asynchronous BatchNorm across two devices (so BatchNorm batch size is 64). We pad images by 4 pixels and crop to 32 x 32 pixels, and also randomly flip and normalize such that it is zero-mean for training. We use the WideResNet Zagoruyko & Komodakis 2016) architecture for all CIFAR experiments.
ImageNet: For ImageNet experiments we use the ResNet-50 architecture (He et al. 2016). We train for 90 epochs with a learning rate of 1.6. We use SGD with non-Nestorov momentum value of 0.9. We use a batch size of 1024. We decay the learning rate at epochs 30, 60, and 80 by a factor of 0.1. We use label smoothing value of 0.1, and weight decay of 0.0001. The weight decay is not applied to batch norm parameters. We do a linear warmup for the first 10 epochs. We use standard Inception style data augmentation. For ImageNet experiments we use 64 16GB V100 GPUs.
A.1 EFFICIENT ENSEMBLE EXPERIMENTS
SuperWeight Ensembles: We use WRN-28-10 models for all of our efficient ensembling experi- ments. For our Efficient Ensemble models, found that each layer having it's own SuperWeight works well. Each SuperWeight is constructed from 5 templates. Following Plummer et al.] (2022), templates allocated by iterating over the learnable parameters in a round-robin fashion over lay- ers of the same shape. This is done with a double loop. with the outer loop over ensemble members and the inner loop over layers, make it easy modulate parameter count. In order share pa- rameters between functionally similar layers in CIFAR experiments, corresponding layers across ensemble members explicitly share templates. Because all members share the same architecture, we find doing gradient- based SuperWeight Cluster learning is not needed.
A.2 ANYTIME INFERENCE EXPERIMENTS
SuperWeight Ensembles: We train all SuperWeight Ensembles on CIFAR-IO and CIFAR-100 us- ing the settings above except we train on a single GPU, the learning rate is decayed at epochs 60, 120, and 160, and we apply cutout (DeVries & Taylor 2017) during training (unlike efficient en- sembling, which does not use cutout for a fair comparison to prior work). SWE-Multt Width is
3 member WRN (8,[3,) ensemble. SWE-Multi WWTE-Muuti-DDpphhWiee DelttDDeethhittee is 4 member WRN 28-[7,4] 16- [7,4] ensemble. All models start from 4 initial depth-binned SuperWeight Clusters and are refined using the gradient similarity threshold from Eq. (3), T 0.1. When learning where to share coeff- cients, PTTE-MMlti-Widhe is trained with the gradient similarity threshold from Eq. (2), 3 0.9 for CIFAR- 100 and 3 = 0.95 for CIFAR-10. and B = 0.5 for CIFAR-10. 10. See Section C.4 for sensitivity study for hyperparameters / and т.
WWNE-Mulii-DephhWiidle uses 3 = 0.9 for CIFAR-100
Note that in Figure of the main paper, we compare against MIMO and BatchEnsemble. These are WRN-28-5 architectures to more fairly compare to our SuperWeight Ensembles for Anytime Infer- ence, which are narrower than WRN-28- 10. That said, we also report MIMO and BatchEnsemble results using a WRN-28-10 architecture in the efficient ensembling experiments.
13

---

Under review as a conference paper at ICLR 2023
A.2.1 BASELINES
ShapeShifter Networks (Plummer et al. 2022), Slimmable Networks (Yu et al. 2019). Universally Slimmable Networks (Yu & Huang 2019b) are each trained using the same training settings the tation).
SWE-Multi- Width and SWE-Multi WWE-MuliiDDppthWWe Width models (including using cutout for data augmen-
Slimmable Networks train a Multi- Width network by executing a subset of the channels for a predefined set of width configurations, called switches.
Universally Slimmable Networks extend Slimmable Networks to execute any width of a network within a max and minimum by training each iteration random subset of switches and calibrating the the batch normalization statistics of the final switches following training. We train Universally Slimmable using max width of 7, a minimum width of 3, and the number of widths trained each iteration, n = 4. We calculate batch normalization statistics of network widths [3,4,5,6,7] over one epoch following training.
ShapeShifter Networks (SSNs) automatically learn where to share parameters within a network through clustering coefficients to form groups and then within each group generating weights for layers from parameter banks via template mixing methods. For SSNs we give each ensemble member its own independent coefficients a and batch normalization parameters. For the Multi- Depth/Width network the layers are grouped manually by depth. Section 4.1 shows Multi-Width comparison of our gradient analysis vs. SSN cluster coefficient approach to forming layer groups.
Multi-Scale Dense Network (MSDN) (Huang et al.) 2018) is Multi-Depth network that provides dense connectivity between early exits and operates at multiple scales for an efficient Anytime In- ference model. We train MSDNet using the training scheme from the original paper. All model hyperparameters are kept the same as the original except the network is built with 16 blocks with 12, 24, and 48 output channels for each of the three scales, respectively, in order to match the infer- ence time with other methods.
Hierarchical Neural Ensemble (Ruiz & Verbeek 2021 is Multi-Depth network that trains tree based ensemble using a novel distillation loss. Results are pulled directly from the original paper with models evaluated for inference time using the author's code. The model uses modified ResNet-50 with N = 16 ensemble members.
Efficient Homogeneous Ensemble Baselines: BatchEnsemble (Wen al. 2020 and MIMO (Havasi et al. 2021) are two homogeneous ensembling methods (which we apply to the anytime inference task Figure I of the main paper). BatchEnsemble perturbs weights with rank-l matrix for each ensemble member. MIMO hard shares all parameters between ensemble members, except for the input convolutional layer and output layer. BatchEnsemble is ineffective as an anytime in- ference method because there are only N evaluation speeds, where N is the number of ensemble members. MIMO only provides a single inference speed because of its shared backbone.
B PRIORITY-QUEUE WEIGHT TEMPLATE ASSIGNMENT
In Section 3.2.2 of our main paper we introduce a method that uses gradient similarity between shared parameters in order to determine where sharing is effective. The intuition behind our ap- proach is that sharing between layers is likely less effective when those layers provide confficting gradients the shared parameters. We proposed greedy approach that would train model for N epochs and then measure similarity between gradients supplied to the shared parameters aggregated over an entire epoch. Gradient similarity between layers l, ly. which we denote as Va..s com- puted using Eq. (2) when learning where Weight Templates should share coefficients, and Eq. (3) when learning SuperWeight Clusters. This similarity would be used to construct a priority-queue Q. which we traverse merging any layers into group G that are above some threshold c, with any remaining layers being placed into the same group (see Algorithm T). Note that in our experiments e is a threshold on gradient similarity, but one could also use threshold on the number groups instead (or in combination), which we leave for future work. Layers in the same group would continue to share parameters, whereas layers in different groups would no longer completely share parameters. Details on how this is implemented is in Section 3.2.2 for sharing in Weight Templates and Section 3.3 for creating SuperWeight Clusters.
14

---

Under review as a conference paper at ICLR 2023
Algorithm
Priority-Queut Group Assignment
Input: threshold e, priority-queue Q (gradient similarity vh., (layers e, Eÿ)
Groups G list()
while [] > 0 do
Vi L, Q.pop()
if Maj then
& Eg & Eg egg, where g.g' E G then G.appendlgugh
Merge into new group
delete 9,9' from G
else if l, Eg then
b One layer already belongs group
g u
else if l E then o' - (4:
clse
b. Neither layer belongs group
(. end if
else
similarity is below threshold, add any remaining ungrouped layers
l & 9. Vg E G then G.append((4.))
end if
if
l & Vg' E G then G.append((e])
end if
end if end while
return
C EXPANDED RESULTS
C.1 SLIMMABLE NETWORKS AND UNIVERSALLY SLIMMABLE NETWORKS ENSEMBLE COMPARISON
Slimmable Networks Yuu et al. 2019) and Universally Slimmable Networks (US) Yu & Huang. 2019b) both train dynamic width network such that given budget at inference time, match the budget by running slim version of the network through executing a subset of the chan- nels each layer. These methods can be through multiple widths. In Figure 4 we compare our SWE-Multi- Width model to Slimmable and Universally Slimmable ensembles. SuperWeight Ensembles benefit from ensembling diverse chitectures, improving performance. In contrast, ensembling multiple widths of Slimmable and Universally Slimmable Networks only leads slight boost, even decrease, in accuracy.
ensemble similar
by running inference
C.2 CIFAR-IO-C
We present results CIFAR-10-C below. Compared efficient ensembling baselines, SWE- HE outperforms others on two out of three metrics, while also providing flexibility not present in prior work via heterogeneous ensembles and the ability adjust the number of parameters in our network without changing architecture.
Table 4: CIFAR-10-C
Method
Top-1
Acc
NLL
BCE
BatchEnsembles Havasi al. (2021)
77.5
1.02
12.9
MIMO Havasi et al. (202
76.6
0.927
11.2
SuperWerght ensemblestours)
76.01
0.885
10.2
SuperWeight Ensembles(ours)
76.6
0.872
8.8
15

---

Under review as a conference paper at ICLR 2023
CIFAR-100
CIFAR-10 10
03
97.0
96.5
81
Accuracy
Accuracy
79
06.5
95.0
Stammable (Multh Wisth)) [10)
94.5
Sammable (Mults Wiathi [10]
(Mult-Width) [9]
Us Muuti-Witth) [9]
SWE (utss))dtxxUurs
SWE (utrsWidthhOurs)
0.0
0.2
0.4
0.6
0.8
1.0
12
14
0.0
0.2
0.4
0.6
0.8
10 (ms)
12
1.4
Average Inference Time (ms)
Average Inference
Figure 4: Slimmable Networks and Universally Slimmable Networks ensemble comparison. Anytime inference results using Slimmable (Yu et al., (019) and Universally Slimmable (Yu & Huang 2019b) ensembles of multiple widths compared SWE-Multi-Win
C.3 ENSEMBLE DIVERSITY ANALYSIS
key attribute of ensembles which makes them effective is their diversity; if the errors of ensemble members are not decorrelated then there no additional benefit of doing inference through addi- tional ensemble members. In this section, we demonstrate that SuperWeight Ensembles accomplish this.
We use a diversity metric introduced in Fort et al. Fort et al. 2019) which measure the fraction of differing predictions by two ensemble members, normalized the by the error of one of them. We present these results in Table 5 we can see that SuperWeight Ensembles is more diverse than all other shared parameter methods. One interesting finding is that 120M parameter SuperWeight Ensembles outperform Standard Deep Ensembles, yet have lower diversity. Looking deeper, the individual model accuracies are improved for SuperWeight Ensembles (average deep 79.9% vs av- erage SuperWeight Ensembles 80.4%) Therefore, it seems like SuperWeight Ensembles helps the model generalize better. This could come from the fact that the parameter factorization limits the space of possible weights. This is especially interesting because WRN-28-10 is already highly optimized model from hyper-parameter perspective. Of note is that one of the ensemble mem- bers from the SuperWeight Ensembles gives higher performance (80.5% for the best model) than standard model (80.1%) 1)). Therefore one could use SuperWeight Ensembles to train a highly perfor- mant single model, which signifies another advantage our approach has over prior work Havasi al. (22TI: Wen et al. (020).
To provide another diversity metric, in Figure we interpolate between two WRN-28-10 10 ensemble members in parameter space to see if the models indeed are in different optimization basins, and report accuracy at each operating point. We accomplish this by interpolating parameters, and leaving Batch Normalization Toffe & Szegedy (2015) in train mode because accumulated statistics are not meaningful at interpolated points. If interpolates have high accuracy, this indicates the ensemble members landed in the same optimization basin and therefore not diverse they could be. We find that interpolates have much decreased accuracy compared to the end points, supporting the idea that model learns diverse ensemble members.
We can see that although all networks experience some degree accuracy drop between ensemble members, the accuracy drop for the parameter model is significantly lower. This seems to indicate that as the function space becomes constrained with a lower number parameters in the parameter bank, the ensemble diversity starts suffer.
16

---

Under review as a conference paper at ICLR 2023
Million Parameters
36.5 Million Parameters
120 Million Parameters
80
74
60
Accurary
72
Accuracy
Accuracy
65
60
40
68
88
20
0.8
1.0
Interpolation Value
Interpolation Valuue
Interpolation Value
Linear interpolations in parameter space on CIFAR- 100, with different numbers of param- eters an ensemble of size 4. We plot accuracy vs interpolation point. Because the accuracy dips, we know the ensemble members are diverse and find different local minima.
Figure 5:
Table Diversity on CIFAR-100. Diversity is measured as the proportion of samples two ensemble members disagree on, normalized by the еπrоr rate of one of the member as in Fort et al. (2019). SuperWeight Ensembles are more diverse than Batchens,mbles, the other shared parameter model. Although 120M model is slightly less diverse than Standard Deep Ensembles, the performance
higher, due to the average member accuracy being higher than Standard Deep Ensembles
Method
Params
Diversity
Standard Deep Ensemble Havasi et al. (2021)
146.0M
0.88
Batchensembles Havasi et al. (202T)
36.5M
0.40
MIMO Havası et al. (2021)
36.5M
0.91
SuperWeight Ensembles(ourg
36.5M
0.78
SuperWeight Ensembles(ours)
120.0M
0.85
C.4 GRADIENT ANALYSIS HYPERPARAMETER SENSITIVITY
SuperWeight Ensembles learn where to share parameters through two steps of gradient analysis; first to form SuperWeight Clusters (Section 3.3), and then separate SuperWeights by decoupling shared coefficients (Section 3.2.2h. Here we explore the sensitivity of the hyperparameters used in each step.
When separating SuperWeights using Eq. (2), we give unique coefficients to SuperWeights where the similarity between the gradients of shared coefficients is less than B. is used the threshold e in Algorithm [ for separating coefficients. In Figure ((a) we show how sensitive our method is the selection of 3 using the SWE-Multi-Width architecture on CIFAR-100 (Krizhevsky et al. 2009). In addition to reporting results for values of 3 E (0.1, 0.5. 0.9} (note that 7 = 0.1), we also report results when no coefficients are shared, and when all coefficients are shared between layers sharing Weight Templates. The results in Figure 6(a) show that no-coefficient and strict coefficient sharing (referred as "Not Shared" and "Shared," respectively) underperform compared to using our approach. Note that we found optimal values of /3 to come from dataset-specific tuning.
When generating SuperWeight Clusters using Eq. (3) we split layers into separate groups if the gradient similarity is less than т. The higher r, the more groups are formed, the less layers sharing Weight Templates, and the less parameters given to each Weight Template. Setting T 2 0 results in sharing between all layers which have gradients that are not conflicting average. As 7 is increased, layer gradients must point in closer directions to be shared. Note т is given as the input value to in Algorithm In Figureo(b Multi-Width architecture on CIFAR- 100 (Krizhevsky et al.) 2009). which we found to be consistent across settings and datasets. Thus, use this same value of across all experiments.
report best performance comes when 0.1 when using the SWE-
17

---

Under review as a conference paper at ICLR 2023
C.5 PERFORMANCE UNDER SEVERE PARAMETER CONSTRAINT
One key feature of SuperWeight Ensembles is that the parameter count decoupled from backbone. It is therefore interesting to how the network behaves under stronger parameter constraints. In Table 6 we present CIFAR-100 top-1 accuracies under various parameter constraints. SWE- single superweight refers the homogenous ensemble which has a single superweight per layer. This is what is presented in Table of our paper. SWE-Gradien Conflict is also a homogenous ensemble, but with learned Weight clusters using the gradient conflict criterion. Finally, SWE heterogenous is a WRN 34-8, 28-12, 28-10 10, and 28-8 ensemble. The gradient conflict criterion becomes more important at lower parameter counts, The improvement of the heterogenous ensemble over the homogenous ensemble also increases with decreased parameter counts.
Table 6: Changing parameter constraint: CIFAR-100 Top-1 accuracies. The gradient conflict criterion becomes more important at lower parameter counts. The improvement the heterogenous ensemble over the homogenous ensemble also increases with decreased parameter counts.
Method
7 million
15 million 36.5 million
SWE-Single Superweight
79.2%
81.3%
82.3%
SWE Gradient Conflict
80.1%
81.4%
82.2%
SWE- Heterogenous
80.8%
81.6%
82.4%
C.6 SUPERWEIGHT CLUSTER MEMBERSHIP
In Figure we present sharing patterns (SuperWeight Cluster membership) of two networks (WRN- 1-77/WNN----777 The initial depthwise sharing pattern would segment each network into four equal groups of layers. Our gradient-based learning of SuperWeight clusters allows indvidual layers to specialize. Interestingly, it seems like method learns separate adjacent layers into separate SuperWeight clusters. This could increase diversity between ensebmle members.
C.7 DIVERSE ARCHITECTURE FAMILIES
Although we show results primarily on ResNets our work, our sharing methodology does indeed function well for diverse ensembles. For example, consider ensemble consisting of Mobilenetv2 and WRN-28-5, shown in Table C.7 We can see that our sharing procedure even helps compared to standard Deep Ensemble performance, despite diverse architectures. Note that these results are over 2 ensemble members, whereas the results in Table T of our paper use ensemble members.
C.8 EFFICIENT ENSEMBLES ON CIFAR
In Table 8 we present CIFAR results from the main paper, with error bars (representing standard error) to provide additional context. Note that even with this additional information, it is clear our method outperforms all baselines on CIFAR in the low parameter regime, and even outperforms standard ensembles in the high parameter regime (with 17% fewer parameters). Results we report baselines are taken from prior work and not provide error bars. See Section the main paper for more discussion.
18

---

Under review as a conference paper at ICLR 2023
Table 7: Diverse Architechture Families: Here we show an ensemble consisting of Mobilenetv2 and WRN-28-5, shown in Table. We that sharing procedure even helps compared to standard DeepEnsemble performance, despite diverse architectures.
Method
Top-1
NLL
ECE
Deep Ensembles 80.1%
0.776
6.5%
SWE-HE
80.3%
0.762
5.33%
Table 8: Homogeneous ensembling comparison CIFAR-I00 (clean) (Krizhevsky et 2009)
CLFAR-100-C (corrupt) (Hendrycks & Dietterich 2019). and CIFAR-10 (Krizhevsky et al., 2009)
using the WideResNet architecture (Zagoruyko Komodakis 2016) averaged over three runs. We add error bars representing standard error for our experiments for context. Prior work did not report error bars. (a) shows that our approach outperforms prior work in efficient ensembling.
(b) com- pares the performance of increasing the number of parameters (without changing the architecture) using our approach compared to standard Deep Ensembles, which trains 4 independent networks as ensemble members. See Appendix C.8 and Section of the main paper for more discussion.
CIFAR (clean)
CIFAR-IOSC
CIFAR-10
Mesbod
Farams
SLL
BCE
Top
NIL
ECE
NLL
BCE
(9) WRN-28-10
36.5
0.875
3.6
0.59
BE
36.5M
81.5
0.740
566
54.1
2.49
19.1
96.2
0.143
BE EmsBN
36.M
81.9
28
54.1
u/a
19.1
962
inla
1.8
0.123
(Ours)
07::000
088
SWE-EE (Ours)
36.M
82.4
4:440000
0.02
0.663
+0.001
3.0
0.25
530
0.06
2.17
0.01
10.0
0.45
96.5
= 0.02
0.115 + 0,003
+ 0.83
[by Ensembbis
146M
1,666
21
1.27
96.6
0.114
SWE-IIO
a.) Coefficient Sharing
b.) Template Sharing
81
Accuracy
Accuracy
79
79
77
77
9-05
8-0.1
1-0.4
Shared
Tw01
75 0.0
0.4
0.6
0.2
10
Average Inference Time (ms)
Average Inference Time (ms)
Figure 6: Gradient analysis hyperparameter sensitivity. Anytime inference results on CIFAR- 100 (Krizhevsky et al.) 2009) using SWE-Multi-Widhe architecture when: a) varying the gradient similarity threshold 8 from Eq. (2) for coefficient sharing, no coefficient sharing, and strict coef- ficient sharing; and b) varying the gradient similarity threshold from Eq. (3) for SuperWeight Cluster forming. For both and 7 best performance is between 0.0 and 1.0, indicating that too much sharing and too little sharing are both harmful.
19

---

Under review as a conference paper at ICLR 2023
SuperWeight Clusters membership for a WRN-28-7/WRRN-1644 ensemble
WRN-16-7
Cony wx3x16x112
Conv 3x3x112x112
Conv 3x3x112x112
Conv 3x3x112x112
Conv 5x3x112x224
Conv 3x3x224x224
Conv 3x3x224x224
Conv 3x3x224x224
Conv 3x3x224x448
Conv 3x3x448x448
Conv 3x3x44Bxx488
Conv 3x3x444B448
SuperWeight Cluster ID
Conv 3x3x3x 16
5
7
WRN-28-7
Conv 3x3x 16x112
Conv 3x3x 112x112
Conv 3x3x112x112
Conv 3x3x112x112
Conv 3x3x112x112
Conv 3x3x112xx12
Conv 3x3x112x112
Conv 3x3x112x112
Conv 3x3x 113x212x24
Conv 3x3x224x224
Conv 3x3x224x224
3x3x224x224
Conv 3x3x224x224
Conv 3x3x224v224
Conv 3x3x224x224
Conv 3x3x224x224
Conv 3x3x224x44B
Conv 3x3x448x448
Conv 3x3x448y448
Conv 3x3x448x448
Conv 3x3y448x448
Conv 3x3x448x448
Conv 3x3x448x448
Conv 3x3x44Bx444B
Conv 3x3x3x16
Figure 7: SuperWeight Cluster Membership of a two member ensemble, WRN-28-7 and WRN- 16-7. The initial depthwise sharing pattern would segment each network into four equal groups of layers. Our gradient-based learning of SuperWeight clusters allows indvidual layers specialize. Interestingly, it seems like our method learns to separate adjacent layers into separate SuperWeight clusters. This could increase diversity between ensemble members.
20


Reviews:
[{"review_id": "wywLf7AHW-", "reviewer": "Reviewer_owxX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SuperWeight Ensembles which generate parameters of diverse architectures from single SuperWeight parameters. To efficiently find the parameters, authors suggest making SuperWeight from templates with the linear combination using coefficients \\alpha. Then, cluster the weights based on the gradient similarity. After constructing SuperWeight, layer weight construction determines the coefficients for generating a layer-wise SuperWeight and assembled into diverse members of architecture. SuperWeight Ensembles are able to generate parameters for both homogeneous architectures and heterogeneous architectures.", "summary_of_the_review": "Overall, I recommend marginally above the acceptance threshold. I hope the authors could resolve my concerns in the rebuttal.", "strength_and_weaknesses": "**Strength**\n\n- This paper propose parameter sharing from SuperWeight for both homogeneous architectures and heterogeneous architectures.\n- The proposed method is able to obtain better performance both in ensembling and in any time inference tasks while it shows a marginal improvement in ensembling compared to baselines.\n- The approach that the current paper is proposing seems reasonable and convincing.\n\n**Weakness**\n\n- Since the proposed methods contain several steps to construct diverse architectures with the sharing parameters, it was difficult to understand at first for me. I hope the authors provide a broad concept for each step which is more abstract than figure 2.\n- As mentioned in the related works, I think the author also should compare the performance with the zero-shot NAS which does not cost at all to find high-performance networks.\n- There are missing results of CIFAR-10-C and ImageNet-C. I hope the authors to provide the experimental results also in CIFAR-10-C and ImageNet-C.\n\n**Question**\n\nI might have missed the following information in the paper.\n- How to generate templates is unclear in section 3.1. Is \\alpha learnable variables? Or deterministic variables?\n- How to construct each member from the layer weight?\n- What is the stop criteria that is explained in the 4. of Section 3?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rr7lAItwrgv", "reviewer": "Reviewer_Qjhr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper builds from the work in [1] to make efficient ensembles. This paper proposes to build ensembles by using shared templates/layers across ensemble members. Unlike previous work in this domain, they mention that because they can use average weights of different dimensions [1], they can combine models with layers of different sizes which allows them to form ensembles of different sizes given fixed parameter setting, which can help tradeoff accuracy vs compute time. \n[1] Plummer et al. 2022\n", "summary_of_the_review": "Overall this paper proposes an interesting method motivated by an important problem (anytime prediction) not usually motivated in the literature. The paper would greatly benefit from a more detailed discussion of the methods and the results (see above) to improve its accessibility to a wider audience.", "strength_and_weaknesses": "- The paper proposes a new way to form deep ensembles which is not seen in the literature\n- The paper could benefit from an explanation of the results\n\n\nOpen questions and comments\n- From the paper, the reader is required to go over [1] to understand the template generating process, as this is not a standard approach I would suggest to devote a section of the paper to this part, which in turn would also make the paper accessible to a wider audience.\n- It woul be ideal if the authors could outline the contributions of the paper vs [1]. \n- In Fig 1 caption the authors claim their approach allows them to learn optimal soft parameter sharing. Why is it optimal? \n- Most results don’t show confidence intervals.\n- Table 2. What would happen if the authors added an ablation study for different # of parameters for imagenet? Does adding more parameters match the performance level for deep ensembles or perform even better?\n- Table 1(b) is the text is referred as table 1(c). \n- The paper could also benefit from a discussion of the computational implications/limitations of the proposed approach during training time.  This would be informative for readers considering to use the proposed approach over standard deep ensembles.\n- The authors mention that because they need to cluster layers across models by layer similarity they use a coefficient lambda to check the gradient cosine similarity. In this case, the layers would need to have the same form don’t they? Could be combine layers from a vit model with a can model using this approach? The authors cite Raghu which notes that there are several differences across the representations, so this part is confusing to me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gLD4EeTF9Z7", "reviewer": "Reviewer_d45p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose SuperWeight Ensemble, a new method for parameter sharing and network ensemble which is more flexible. Better results are obtained in experiments.", "summary_of_the_review": "The paper propose a useful method which leads to flexible and effective parameter sharing and ensemble, good experimental results are obtained.", "strength_and_weaknesses": "The proposed method is flexible, which allows ensembles of network with different architectures, and enables anytime inference, which is important.\n\nBetter performance with fewer parameters are obtained, which illustrates the effectiveness of the proposed method.\n\nWriting of section 3 needs to be polished, currently it is a little bit vague.\n\nSome ablation studies are suggested, how does each component/step of the algorithm contribute to the performance. For example, what will the result be if we only have one cluster, or only have one group in each cluster?\n\nHow about the training time needed compared to other related works.\n\nHow will the performance be on more complicated datasets other than CIFAR.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject