Traversing Between Modes in Function Space for Fast Ensembling
Authors: Eunggu Yun
Hyungi Lee
Giung Nam
Juho Lee
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep ensemble
mode connectivity

Abstract: Deep ensemble is a simple yet powerful way to improve the performance of deep neural networks. Under this motivation, recent works on mode connectivity have shown that parameters of ensembles are connected by low-loss subspaces, and one can efficiently collect ensemble parameters in those subspaces. While this provides a way to efficiently train ensembles, for inference, one should still execute multiple forward passes using all the ensemble parameters, which often becomes a serious bottleneck for real-world deployment. In this work, we propose a novel framework to reduce such costs. Given a low-loss subspace connecting two modes of a neural network, we build an additional neural network predicting outputs of the original neural network evaluated at a certain point in the low-loss subspace. The additional neural network, what we call a “ bridge”, is a lightweight network taking minimal features from the original network, and predicting outputs for the low-loss subspace without forward passes through the original network. We empirically demonstrate that we can indeed train such bridge networks and significantly reduce inference costs with the help of the bridge networks.

Under review as a conference paper at ICLR 2023
TRAVERSING BETWEEN MODES IN FUNCTION SPACE FOR FAST ENSEMBLING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep ensemble is simple yet powerful way improve the performance of deep neural networks. Under this motivation, recent works on mode connectivity have shown that parameters of ensembles are connected by low-loss subspaces, and one can efficiently collect ensemble parameters in those subspaces. While this provides a way to efficiently train ensembles, for inference, one should still exe- cute multiple forward passes using all the ensemble parameters, which often be- comes a serious bottleneck for real-world deployment. In this work, we propose novel framework to reduce such costs. Given low-loss subspace connecting two modes of a neural network, we build an additional neural network predicting outputs of the original neural network evaluated at a certain point in the low-loss subspace. The additional neural network, what we call a "bridge", is a lightweight network taking minimal features from the original network, and predicting outputs for the low-loss subspace without forward passes through the original network. We empirically demonstrate that we can indeed train such bridge networks and significantly reduce inference costs with the help of the bridge networks.
INTRODUCTION
Deep Ensemble (DE) (Lakshminarayanan et al., 2017) is a simple algorithm to improve both predic- tive accuracy and uncertainty calibration of deep neural networks, where a neural network trained multiple times using the same data but with different random seeds. Due to this randomness, the parameters obtained from the multiple training runs reach different local optima, called modes, on the loss surface (Fort et al., 2019). These parameters represent a set of diverse functions serving as an effective approximation for Bayesian Model Averaging (BMA) (Wilson and Izmailov, 2020).
An apparent drawback of DE is that it requires multiple training runs. This cost can be huge pecially for large-scale settings for which parallel training is not feasible. Garipov et al. (2018); Draxler et al. (2018) showed that modes in the loss surface of deep neural network connected by relatively simple low-dimensional subspaces where every parameter on those subspaces retains low training error, and the parameters along those subspaces are good candidates for ensembling. Based on this observation, Garipov et al. (2018): 8); Huang et al. (2017) proposed algorithms to quickly construct deep ensembles without having run multiple independent training runs.
While the fast ensembling methods based on mode connectivity reduce training costs, they do not address another important drawback of DE; the inference cost. One should still execute multiple forward passes using all the parameters collected for ensemble, and this cost often becomes critical for a real-world scenario, where the training done in absudact-auundant setting with plenty of computation time, but for the deployment, the inference should be done in resource-limited environment. For such settings, reducing the inference cost much more important than reducing the training cost.
In this paper, we propose a novel approach to scale up DE by reducing inference cost. We start from an assumption; if two modes in ensemble are connected by simple subspace, we can predict the outputs corresponding to the parameters on the subspace using only the outputs computed from the modes. In other words, can predict the outputs evaluated at the subspace without having to forward the actual parameters on the subspace through the network. this is indeed possible, for instance, given two modes, we approximate ensemble of three models consisting of

---

Under review as a conference paper at ICLR 2023
(0.5 v (1.2(0.5)
(0.53
1 #2)
(
e/ (21)
Jo (
(ch0 (x:)
(#t
(.s)) (2s)
(a)
wiyy (oe)
1 (z)
18 (2)
piny oit
(c))
Figure 1: Comparing ensembles with Bezier curve (left) and type II bridge network (right).
parameters collected from three different locations (one from a subspace connecting two modes, and two from each mode) with only two forward passes and a small auxiliary forward pass.
We show that we can actually implement this idea using an additional lightweight network whose inference cost is relatively low compared to that of the original neural network. This additional net- work, what we call "bridge network", takes some features from the original neural network, (e.g., features from the penultimate layer), and directly predict the outputs computed from the connecting subspace. In other words, the bridge network lets us travel between modes in the function space.
We present two types of bridge networks depending the number of modes involved in prediction, network architectures for bridge networks, and training procedures. Through empirical validation various image classification benchmarks, show that 1) bridge networks can predict outputs of connecting subspaces quite accurately with minimal computation cost, and 2) DES augmented with bridge networks can significantly reduce inference costs without big sacrifice in performance.
PRELIMINARIES
2.1 PROBLEM SETUP
In this paper, we discuss the K-way classification problem taking D-dimensional inputs. A classifier constructed with deep neural network fe IR.' - RK which is decomposed into feature extractor ((it) RDD - Dn and a classifier p(cb) RR + R i.e., fe( = f o p(i)) (a:).
Here, & E + and ab E A denote the parameters for the feature extractor and classifier, respectively, 0 (Φ, vb) O and Da is the dimension of the feature. An output from the classifier corresponds to class probability vector.
2.2 FINDING LOW-LOSS SUBSPACES
While there are few low-loss subspaces that are known to connect modes of deep neural networks, in this paper, we focus on Bezier curves suggested in (Garipov et al., 2018). Let O and 0j be two parameters (usually corresponding to modes) of a neural network. The quadratic Bezier curve between them is defined
{11 (1-r)²0, +r00 2r(1-r)0! +r20, 1r [0.11].
where 0 0!0c) is pin-point parameter characterizing the curve. Based this curve paramerization, low-loss subspace connecting (0;, 0,) found by minimizing the following loss w.r.t. (bb)).
[. (())))()))s
(2)
where (bee) (r) denotes the point at the position r of the curve,
he) 1.
2r (00---)0f0 r)o + +r20, = (1)
(3)
2

---

Under review as a conference paper at ICLR 2023
and L Q - R is the loss function evaluating parameters (e.g., cross entropy). Since the integration above is usually intractable, we instead minimize the stochastic approximation:
the) (r))
where (0, 1) is the uniform distribution on [0, 1]. For more detailed procedure for the Bezier curve training. please refer to Garipov et al. (2018).
2.3 ENSEMBLES WITH BEZIER CURVES
Let (0....., m j be of parameters independently trained as a deep ensemble. Then, for each pair (0,,0) 0,), we can construct low-loss Bezier curve. Since all the parameters along those Bezier curves achieve low loss, we can actually add them to the ensemble for improved performance. For instance, choosing r = 0.5, we can collect // predictor as
(0.5) for all (i,j) pairs, and construct an ensembled
TR Σ (x)) + (oo (
(5)
+
(22
While this strategy provide an effective way to increase the number of ensemble members, for inference, an additional Omm²) number of forward passes are required. Our primary goal in this paper is to reduce this additional cost by bypassing the direct forward passes with و!15) (r).
3 MAIN CONTRIBUTION
In this section, we present a novel method that directly predicts outputs of neural networks evaluated at parameters on Bezier curves without actual forward passes with them.
3.1 BRIDGE NETWORKS
Let us first recall our key assumption stated in the introduction; if two modes an ensemble are con- nected by simple low-loss subspace (Bezier curve), then we can predict the outputs corresponding to the parameters on the subspace using only the information obtained from the modes. The intuition behind this assumption is that, since the parameters are connected with simple curve, the corre- sponding outputs may also be connected via a relatively simple mapping which is far less complex than the original neural network. If such mapping exists, we may learn them via lightweight neural network.
More specifically, let 2, (n) (ar) and v, (z) p/o (z.) for i (1....,ml) m). Let (r) = (o)) (z)). In order to use (r)(r) with v. to get an ensemble, we should forward through fo starting from the bottom layer. Instead, we reuse z; to predict (r,((r) with lightweight neural network. We call such lightweight neural network a "bridge network", since lets us directly move from to (r) in the function space, not through the actual parameter space. bridge network is usually constructed with a Convolutional Neural Network (CNN) whose inference cost is much lower than that of fo.-
From the following, we introduce two types of bridge networks depending on the number of modes involved in the computation.
Type bridge networks A type bridge network h takes a feature z; from only one mode, and predicts v.,5(7
(r)(r) (r,(r) = M
(6)
A type I bridge network can be constructed between any pair of connected modes (0, and an ensembled prediction for specific mode 0 with its Bezier parameter 0 o!be) can be approximated as
v; n13 (2.)
3

---

Under review as a conference paper at ICLR 2023
whose inference cost is nearly identical to that of Us (nearly single forward pass). One can also connect 0₃ with multiple modes (0), J. learn bridge networks between (1,31, (8, and construct ensemble
Algorithm 1 Training bridge networks
Require: Training dataset D, a pair of parame-
ters (01, 02) and corresponding Bezier parame-
ter 0 bridge network 1/12 (type 1) or 11 (type II) with parameters w. learning rate rj, regularization scale X, mixup coefficient a.
1 I+k k U
hi (zi)
(8)
Initialize w.
Still, since the costs for h 1.jN are far lower than v, the inference cost does not significantly in- crease.
while not converged do Sample a mini-batch for i = I..... [ do
D.
Take the input x; from B. a; m)
Type II bridge networks A type II bridge network between (0,, O,) takes two features (z,, zj) to predict (r)
21 + 1.) (a:), 0i t s (21).
v1.2(r) + (0 (zi).
if type I then
vr)(r) R (r.s(r) = (21,
(9)
(),)(r) (1222) (2)
else
An ensembled prediction with the type II bridge network is then constructed as
22 . (a:), 02 .lchh)())))) (zz).
Ô (1))(r) = (0.5) (21 w):
1 ( v Uo u, + + (2)2))) 3
end if
(10)
&, -
D DKL (1 (0.5 (0.5)
- -Dxx (v)l 21.2
where we construct an ensemble of three mod- els with effectively two forward passes (for v, and B,). Similar to the type bridge networks, may construct multiple bridges between single curves and use them together for an en- semble. Fig. 1 presents schematic diagram comparing forward passes of ensembles with- /without type II bridge network.
end for w x-
V [8| E.
end while return w.
3.2 LEARNING BRIDGE NETWORKS
Fixing a position on Bezier curves In the definition of the bridge networks above, we fixed the value т. In principle, we may parameterize the bridge networks to take r as an additional input to predict viy(() for any [0, 1], but we found this to be ineffective due to the difficulty of learning all the outputs corresponding to arbitrary r values. Moreover, as we empirically observed in Fig. 2, the ensembling with Bezier parameters are most effective with 0.5, and adding additional parameters evaluated at different r values does not significantly improve the performance. To this end, we fix r 0.5 and aim learn bridge networks predicting (0.5) throughout the paper.
Training procedure Let {0,..... 0m) be a set of parameters in an ensemble. Given a set of Bezier parameters (o(be)) connecting them, we learn bridge networks (either type I II) for each Bezier curve. The training procedure is straightforward. We first minimize the Kullback-Leibler divergence between the actual output from the Bezier parameters and the prediction made from the bridge network. It makes the bridge network imitate the original function defined by the Bezier parameters in the same manner as a conventional knowledge distillation (Hinton al., 2015). In addition, we also maximize the Kullback-Leibler divergence between the base prediction and the bridge prediction to regularize the bridge to predict differently from the base model. Such regular- ization is quite important, when the training error of the base model is near zero; the base network and the target network (the one on the Bezier curve) will produce almost identical outputs. Further, apply the mixup (Zhang al., 2018) method to explore more diverse responses, preventing the bridge from learning to just copy the outputs of the base model. Refer to Algorithm for the detailed training procedure.
4

---

Under review as a conference paper at ICLR 2023
ACC (T)
NL)
ECE (41
BS 14)
0.04
0.35
75.0
0.92
0.90
0.02
Bezlary
DE Basier
77.0
0.98
76.5
0.04
0.35
76.0
0.03
0.34
0.90
75.0
0.86
Rezier
Figure 2: Performance of an ensemble of two modes and a parameter from the Bezier curve con- necting them, evaluated for ResNet-32x4 CIFAR-100 Here, E (0, 1) denotes a position the curve. Top row shows ensemble performances when one member from Bezier curve r is added DE-2. Bottom row shows ensemble performances when members are sequentially added to DE-2 from Bezier curve. For accuracy, higher is the better, and for NLL, ECE and BS, lower is the better.
4 RELATED WORKS
Mode connectivity The geometric properties of deep neural networks' loss surfaces have been studied, and one notable property is the mode connectivity (Garipov et al., 2018; Draxler et al., 2018); there exists a simple path between modes of neural network on which the network retains low training error along that path. From this, fast ensembling methods that collect ensemble mem- bers on the monde-connectin-paee Extending this idea, Izmailov al. (2020) approximated the posteriors of Bayesian neural nets via the low-loss subspace and used them for BMA. Wortsman et al. (2021) also presented method for further improving performance by ensembling over the subspaces.
have been proposed (Huang et al., 2017; Garipov et al., 2018). 8).
Efficient ensembling Despite the superior performance of DE (lakshminarayaaan al., 2017; Ovadia et al., 2019), suffers from additional computation costs for both the training and the infer- ence. There have been several works that reduce the computational burden in training by collecting ensemble members efficiently (Huang et al., 2017; Garipov et al., 2018; Benton et al., 2021), but they did not consider inference costs that arose from multiple forward passes. On the other hand, there also exist eeffeccceffi ensembling methods by sharing parameters (Wen et al., 2020; Dusenberry al., 2021) or sharing representations (Lee et al., 2015; Siqueira et al., 2018; Antoran et al., 2020; Havasi et al., 2021). In particular, Antoran et al. (2020) and Havasi et al. (2021) pre- sented the methods obtain an ensemble prediction by single forward pass. Nevertheless, these methods do not scale well for complex large-scabe scale datasets require large network capacity.
5 EXPERIMENTS
In this section, we are going to answer the following three big questions:
Do bridge networks really learn to predict the outputs of a function from the Bezier curves?
How much ensemble gain we obtain via bridge networks with lower computational complexity?
How many bridge networks do we have to make in order to achieve certain ensemble performance?
We sequentially answer them in Sections 5.2 to 5.4 with empirical validation.
5.1
SETUP
Datasets and networks We evaluate the proposed bridge networks various image classification benchmarks, including CIFAR- 10, CIFAR- 100, Tiny ImageNet, and ImageNet datasets. Throughout the experiments, we use the family of residual networks introduced in He al. (2016) base
5

---

Under review as a conference paper at ICLR 2023
Better Fridge
airpiane
Cat
deee
dag
trog
hore
ship
airptanee
bing
Get
hore
track
airplene
dveer
dog
ship
Figure Bar plots in the third column depict whether the bridge network (orange) outputs the same
logit values as the base model with the Bezier parameters (blue) for given test inputs displayed in the first column. We also depict the predicted logits from the base model with 0, and 02 the second and fourth columns, respectively. Additional results are available in Fig. 6.
Table 1: R2 scores quantify how similar the following models to the target function defined with Bezier parameters ((he)) (0.5) are in output probabilities; 'Type I/II Bridge', 'Other Type I/II Bridge', and 'Other Bezier'. Refer to the main text in Section 5.2 for detailed description for each model. All values are measured on the test split of each dataset.
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny ImageNet
Model
12
Model
A2 (1)
Model
11
Type Bridge
0.910 1:0003
Type Bridge
0.784 ::0.005
Type Bridge
0.746 1:0.006
Other Type Bridge
0.885 =0.003
Other Type I Bridge
0.741 =0.005
Other Type I Bridge
0.728 =0.007
Type II Bridge
0.924 =0.002
Type II Bridge
0.814 =0 =0.002
Type II Bridge
0.765 +0.003
Other Type II Bridge
0.895 ::0.002
Other Type II Bridge
0.752 #10000
Other Type II Bridge
0.732 +:0.00
Other Bezier
0.871 +0.005
Other Bezier
0.726 =:0003
Other Bezier
0.712 +00003
model: ResNet-32x2 for CIFAR-10, ResNet-32 x4 for CIFAR-100, ResNet-18 for Tiny ImageNet and ResNet-50 for ImageNet, where x2 and x4 respectively denotes doubling and quadrupling of number channels for convolutional layers. We construct bridge networks with CNNS with a residual path whose inference costs are relatively low compared to those of ResNet base models. For detailed training settings, including bridge network architectures or hyperparameter settings, please refer Appendix A.2.
By changing the channel size of the convolutional layers in the bridge network, we can balance the trade-off between performance gains with computational costs. We check this trade-off in Table 2. We refer to bridge network with less than 15% of foating-point operations (FLOPs) compared to the base model as Bridge (small version), and bridge with more than 15% (medium version).
Efficiency metrics We choose FLOPs and the number parameters (#Params) for efficiency evaluation as these metries commonly used consider the efficiency (Dehghani et al., 202 l). Because FLOPss and #Params of the hase model are different for each dataset, we report the relative FLOPs and the relative #Params with respect to the corresponding base model instead for better comparison.
Uncertainty metrics As suggested by Ashukha et al. (2020), along with the classification accu- racy (ACC), we report the calibrated versions of Negative Log-likelihood (NLL). Expected Calibra- tion Error (ECE), and Brier Score (BS) as metrics for uncertainty evaluation. We also measure the Deep Ensemble Equivalent (DEE) score proposed in Ashukha et al. (2020), which shows the relative performance for DE in terms NLL and roughly be interpreted as effective number of models for ensemble. See Appendix A.3 for more details.

---

Under review as a conference paper at ICLR 2023
Table 2: FLOPs, #Paaams, R scores, and ensemble performance metries of various type II bridge network sizes on CIFAR-100. We use ResNet-32 base model and 3 blocks of CNN with residual connection as bridge networks. The number after CNN indicates the number of channels. R² scores are measured with respect the target Bezier 0.5.
Bridge
FL.OPs ())
#Params (L)
(1)
ACC
NL)
BCF (1)
BS (J)
CNN 32
x 0.012
0.009
0.709 = 0.000
75.62 #011
0.914 0.005
0.03:=0.001
0.342 0.0002
0.029
0.758 + 0.004
75.78 0.30
0.901
0.004
0.016 0.002
0.001
128
0.079
0.060
0.793 0.003
75.98 0:220
0.894 0.003
0.021 0.002
0.335 0.001
256
0.246
0.188
0.814 0.002
76.13 0.114
0.890 0.004
0.023 0.000
0.334 0.002
Table Performance improvement of the ensemble by adding type I bridges to the single base
ResNet model on Tiny ImageNet dataset. FLOPs, #Params, and DEE metrics are measured with respect to the single base model. Bridgeam and Bridgema denote the small and the medium versions
of the bridge network based on their FLOPs.
Model
FLOPs (4)
#Params (4)
ACC (7)
NLL (U)
ECE (L)
BS (4)
DEE (1)
ResNet (DE-I)
x 1,000
x 1.000
63.42
0.22
1.618 00005
0.037 00000
0.485 = 0.008
1,000
Bridge
x 1.088
x 1.093
65.38
= 0.09
1.444
1.0.05
0.015 0.001
0.461 0.000
2.179 0.1110
+ Bridge.
x 1.176
x 1.186
65.55
L 1.115
1.405
1.0.05
0.013
0.000
0.456 1 0.001
2,750 1 0.086
+.3 Bridge.n
x 1,264
x 1,279
65.61
0.10
1,388
0.003
0.014
114400000
0.455 = 0.000
3.022 0.079
1 Bridgend
x 1,277
x 1,290
65
0.15
1.418
0.003
0.018
0.002
0.453 + 0.001
2,562 + 0.056
+2 Bridgema
x 1,554
x 1,580
66.59
L 0.09
1.372
1.0.00
0.016
1 0.000
0.445 1 0.000
3.437
1. 0.033
Bridgend
x 1.831
x 1,870
66.79
0.11
1.353
1: 0.000
0.015
0.001
0.443 0.000
3.967 1: 0.043
DE-2
x 2,000
x 2,000
66.21 = 0.10
1.456
0.004
0.022
0.002
0.450 + 0.002
2,000
5.2 CORRESPONDENCE BETWEEN BRIDGE NETWORK AND BEZIER CURVE
To assess the quality of the prediction of bridge networks, we use a set of ensemble parameters {0,, .z..... 0m) and Bezier curves between them. If the bridge network (1 predicts (0.5) well compared to the other baselines, we can confirm that there exists the correspondence between the bridge network and the Bezier curve. To this end, measure the R² score which quantifies how similar outputs of the following baselines to that of the target function fo (1) 'Type I/II
Bridge' denote the bridge network imitating the function of 0 0 (0.5), (2) 'Other Type I/II Bridge' denote the bridge network imitating the function of 8 ( (0.5) for some (1, * (1,2), and (3)
'Other Bezier' denotes the base model with the parameters (he)) (0.5) for some (t, # (1,2).
Table 1 summarizes the results. Compared to the baselines (i.e., 'Other Type I/II Bridge' and "Other Bezier'), ) the bridge networks produce more similar outputs to the target outputs. The R² values be- tween the predictions and targets significantly higher than those from the wrong targets, demon- strating that the bridge predictions indeed are approximating our target outputs of interest.
Fig. 3 visualizes whether the bridge network H (1.05) predicts the logits from 0 (///) (0.5). To be more
specific, we visualize the predicted logits from 0,, O2. 0fthe) (0.5), and the bridge network H for two test examples of CIFAR-10. Indeed, the bridge network predicts well the logits from the Bezier parameter. Appendix B.1 provides additional examples which further verify this.
Relation between model size and regression result We measure the relation between the size of bridge networks and the goodness of fits of prediction measured by R² scores. Table 2 shows that achieve decent R scores with small number of parameters, and the prediction gets better as we increase the flexibility of our bridge network. Also, the results show that a higher R² score leads to better ensemble results.
7

---

Under review as a conference paper at ICLR 2023
NCC (1)
NLL
ECE (4)
B5 (4)
0.035
0.48
66.0
1.55
0.47
0.025
1.45
0.020
0.46
64.0
0.45
1.35
1.25
1.5
1.75
2
1.25
1.5
1.75
1.25
1.5
1.75
1.25
1.5
1.75
2
Relative FLOPs
Relative FLOPs
Relative FLOPs
Relative FLOPs
BridgRar
Bridgenu
Figure 4: The cost- post-performanc plots of rype bridge(s) compared on Tiny ImageNet. The x-axis denotes the relative FLOPs quantifying the inference cost of the model compared to a single base model, and the y-axis shows the corresponding predictive performance. On the basis of DE (black dashed line), the upper left position is preferable in ACC, and the lower left position is preferable in NLL, ECE, and BS.
(4)
0.022
0.45
69.0
68.0
1.35
0.015
0.43
67.0
1.30
0.013
Relative FLOPS
Relative FLOPs
Relative FLOPs
Relative FLOPS
Bridgeen
Bridggea
Figure 5: The cost-perfone pess-performaec plots of rype II bridge(s) compared to DE on Tiny ImageNet. Others are identical Fig. 4 except that we extend the DE basis from DF-2 DE-7 (black dashed lines).
5.3 CLASSIFICATION WITH BRIDGE NETWORKS
5.3. TYPE BRIDGE NETWORKS
Single Model performances with type I bridge networks In situations where multiple forward passes are not allowed for inference, we can approximate an ensemble of single base model and the ones from Bezier curves with type bridge networks. The results are shown in Table 3. The results show that for Stochastic Gradient Descent (SGD) trained single base ResNet model, an ensemble with type I bridge networks improves the performance both in terms of accuracy and uncertainty es- timation. Only adding one small type bridge to the base model (ResNet + Bridgesm) dramatically improves the accuracy (+= x 1.701) and DEE (%h x2.179).
Using multiple type I bridge networks As type bridge network requires features from only one mode of each curve for inference, we can use multiple type I bridge networks for a single base model without significantly increasing inference cost, we mentioned Eq. 8. Table 3 ports the performance gain of single base model with increasing number of type I bridges. Each bridge approximates the models on different Bezier curves between a single mode and others (i.e., Bezier curves between modes A-B, A-C, and models on a single Bezier curve. Adding more bridge networks introduces more diverse out- puts to the ensembles. One can see that the performance continuously improves the number of bridges increases, with low additional inference cost. Fig, shows how much type bridge networks efficiently increase the performances proportional to FLOPs.
where A, B. and C are different modes.), not
5.3.2 TYPE II BRIDGE NETWORKS
Performance Table 4 summarizes the classification results comparing DE, DE with Bezier curves, and DE with type II bridge networks. For the more experimental results including other datasets,
8

---

Under review as a conference paper at ICLR 2023
Table 4: Performance improvement of the ensemble by adding type II bridges as members exist- ing DE ensembles on Tiny ImageNet dataset. FLOPs, #Params, and DEE metrics are measured with respect to corresponding DEs. Type II bridges consistently improve the accuracy and uncertainty metrics of the ensemble before saturation. Bridgesm and Bridgemd denote the small and the medium versions of the bridge network based on their FLOPs.
Model
FLOPS (1)
#Params (1)
(TC
NLL
BCE
BS (J)
DEE (1)
DE-4
x 4,000
4,000
68.16 10111
1.352
10.001
0.016 0.000
0.427 0.001
4.000
Bridge..
4.125
4.132
68.48
1.315
0.013 0.000
0.423 0.000
5.962
0.122
Bridge
4.250
4.264
68.67
0.10
1.297
00000
0.015 0.00
0,422 + 0.000
7.239
0.226
Bridge...
x 4.500
4.528
68.69
1018
1,281
1.0002
0.012 0.002
0.421
0.000
8.432
1. 0.383
Bridgesm
x 4.750
4.792
68.58
0.04
1.276
0.002
0.011
0.001
0.422 0.000
8.768
1 0.441
Bridgend
x 4.352
4.367
68.68
0.11
1,309
0.002
0.016
0.000
0,422 0.000
6.377 + 0.178
Bridgend
4.704
4.734
68.98
0.14
1.287
00002
0.016
0.000
0,419 0.000
7,986
0.347
Bridgend
x 5,056
5.101
69.01
0.17
1,275
1.000
0.015 0.001
0.417 0.000
8.892
1: 0.389
+44 Bridgend
x 5.408
5.466
69.14
0.13
1,266
+0000
0.001 0.015
0,416 0.000
9.539
0.481
DE-5
5,000
5.000
68.54 0.08
1.329 00000
0.018 0.001
0,422 0.001
5,000
please refer to Appendix B. From Table 4, one can see that with only sightly increase in the com- putational costs, the ensembles with bridge networks achieves almost DEE 1.962 ensemble gain for DE-4 case. This gain is not specific only for DE-4; the ensembles with type II bridge networks consistently improved predictive accuracy and uncertainty calibration with negligible increase the inference costs. Fig. 5 shows how much our type II bridge network achieve high performance in the perspective of relative FL.OPs.
Computational cost We report FLOPs for inference on Table 4 to indicate how much relative computational costs are required for the competing models. Fig. 5 summarizes the tradeoff between see from these results, bridge networks
FLOPs and performance in various metries. As could achieve remarkable gain in performance, so for some cases, adding bridge ensembles achieved performance gains larger than those might be achieved by adding entire ensemble members. For instance, in Tiny ImageNet experiments, DE-4 + 2 bridges was better than DE-5 (DEE x x7.239). Please refer Appendix B for the full results including various DE size and other datasets.
5.4 How MANY TYPE II BRIDGES ARE REQUIRED?
For an ensemble m parameters, the number of pairs can be connected by Bezier curves is (%) which grows quadratically with m. In the previous experiment, we constructed Bezier curves and bridges for all possible pairs (which explains the large inference costs for Bezier ensembles), but in practice, we found that it is not necessary to use bridge networks for all of those pairs. As an example, we compare the performance of DE-4 + bridge ensembles with increasing number of bridges Tiny ImageNet dataset. The results dramatically increases the performance, and the performance gain gradually saturates as we add more bridges. Notably, only one bridge would suffice to outperform DE-5 (DEE x5.962).
summarized in Table 4. Just bridge
6 CONCLUSION
In this paper, we proposed novel framework for efficient ensembling that reduces inference costs of ensembles with lightweight network called bridge networks. Bridge networks predict the neu- ral network outputs corresponding to the parameters obtained from the Bezier curves connecting two ensemble parameters without actual forward passes through the network. Instead, they reuse features and outputs computed from the ensemble members and predict the outputs corresponding to Bezier parameters directly in function spaces. Using various image classification benchmarks, demonstrate that
train such bridge networks with simple CNNS with minimal inference costs, and bridge augmented ensembles could achieve significant gain both in terms of accuracy and uncertainty calibration.

---

Under review as a conference paper at ICLR 2023
Reproducibility statement Please refer to Appendix A for full experimental detail including datasets, models, evaluation metrics and computing resources.
REFERENCES
A. Ashukha, A. Lyzhov, D. Molchanov, and D. P. Vetrov. Pitfalls of in-domain uncertainty estima- tion and ensembling in deep learning. In International Conference on Learning Representations (ICLR), 2020. 6, 13
G. W. Benton, W. J. Maddox, S. Lotfi, and A. G. Wilson. Loss surface simplexes for mode con- necting volumes and fast ensembling. In Proceedings of The 38th International Conference on Machine Learning (ICML. 2021), 2021.
M. Dehghani, Y. Tay, A. Armab, L. Beyer, and A. Vaswani. The efficiency misnomer. In International Conference on Learning Representations (ICLR), 2021. 13
F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of The 35th International Conference on Machine Learning (ICML 2018), 2018. 5,5
S. Fort, H. Hu, and B. Lakshminarayanan. Deep ensembles: loss landscape perspective. arXiv:1912.02757, 2019. 1
T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson. Loss surfaces, mode connec- tivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018), 2018 2,
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings The 34th International Conference Machine Learning (ICMI. 2017), 2017. 13
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In Deep Learming and Representation Learning Workshop, NIPS 2014, 2015. 4, 20
G. Huang, Y. Li, G. Pleiss, Liu, F. Hopcroft, and K. Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In International Conference Learning Representations (ICLR), 2017. 1,5
loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing in- ternal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning (ICMI. 2015), 2015.
P. Izmailov, W. J. Maddox, P. Kirichenko, T. Garipov. D. Vetrov, and A. G. Wilson. Subspace inference for bayesian deep learning. In Uncertainty Artificial Intelligence, 2020.
P. Izmailov, S. Vikram, M. D. Hoffman,, and A. G. Wilson. What are bayesian neural network posteriors really like? In Proceedings of The 38th International Conference on Machine Learning (ICMI. 202/), 2021. 12
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009. 12
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. 5.5
F-F. Li, A. Karpathy, and J. Johnson. Tiny ImageNet. https://wwww http://wwwwwklll.eoeoooc Siiyyimmaenete 2017. [Online; accessed 19-May-2022). 12
Y. Ovadia, E. Fertig, J. Ren, Z. Nado. D. Sculley, S. Nowozin, J. V. Dillon. B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019.
10

---

Under review as a conference paper at ICLR 2023
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32. 2019. 13
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,
M. Bemsttein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015. 12
S. Singh and S. Krishnan. Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 12
Wenzel, K. Roth, B. S. Veeling, Swikatkowski, L. Tran, S. Mandt, Snoek, T. Salimans, R. Jenatton, and S. Nowozin. How good is the bayes posterior in deep neural networks really? In Proceedings of The 37th International Conference on Machine Learning (ICMI. 2020), 2020. 12
A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generaliza- tion. In Advances Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
M. Wortsman, M. Horton, C. Guestrin, Farhadi, and M. Rastegari. Learning neural network subspaces. In Proceedings of The 38th International Conference on Machine Learning (ICML 202/), 2021. 5
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
11

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
A.1 FILTER RESPONSE NORMALIZATION
Throughout experiments using convolutional neural networks, we use the Filter Response Nor- malization (FRN; Singh and Krishnan, 2020) instead of the Batch Normalization (BN; loffe and Szegedy, 2015) to avoid recomputation of BN statistics along the subspaces. Besides, FRN is fully made up of learned parameters and does not utilize dependencies between training examples, thus, it gives us more clear interpretation of the parameter space (Wenzel et al., 2020: Izmailov et al., 2021).
A.2 DATASETS AND MODELS
Dataset We use CIFAR-IO, 10, CIFAR- 100 (Krizhevsky et al., 2009). Tiny ImageNet (Li et al., 2017) and ImageNet (Russakovsky et al., 2015) datasets. We apply the data augmentation consisting of random cropping of 32 pixels with padding of 4 pixels and random horizontal flipping. We subtract per-channel means from input images and divide them by per-channel standard deviations.
Network We CNN with residual path similar to the ResNet block structure. To use the features of base models, we embed one or more features from different layers of base models.
For CIFAR-10 dataset, we ResNet-32x2 layers with widen factor of 2, and we use CNN 3 blocks as type and type II bridge networks. The bridge networks use the features of the third last block.
base network which consists of 15 blocks and 32
For CIFAR-100 dataset, we use ResNet-32 x x4 as a base network which is almost same as ResNet-32 with widen factor of 2, and we use CNN blocks type and type II bridge networks. The bridge networks use the features of the third to last block.
For Tiny ImageNet dataset, we use ResNet- 18 a base network which consists of blocks and 18 layers, and we use CNN 2 blocks as type I and type II bridge network. The bridge networks use the features z of the third to last and the second last blocks.
For ImageNet dataset, we use ResNet-50 layers, and we use CNN 3 blocks as type I and type bridge network. The bridge networks use the features z of the third to last and the second to last blocks.
base network which consists of 17 blocks and 50
Optimization We train base ResNet networks for 200 epochs with learning rate 0.1. We use the SGD optimizer with momentum 0.9 and adjust learning rate with simple cosine scheduler. We give weight decay 0.001 for CIFAR-10 10 dataset, 0.0005 for CIFAR-100 and Tiny ImageNet dataset, and 0.0001 for ImageNet dataset.
Regularization We introduced two additional hyperparameters for training bridge models; 1) the regularization scale À and 2) the mixup coefficient a. Since the training eттоr of the base network is near for the family of residual networks on CIFAR-10/100, given training input without any modification, the base network and the target network (the one on the Bezier curve) will produce almost identical outputs, network. To prevent this, we perturb the inputs via mixup, and regularize the bridge to produce outputs different from the ones computed from the base models. On the other hand, for the datasets such as ImageNet where the models fail to achieve near zero training errors, the base network and the target networks are already distinct enough, so we found that the bridge can be trained easily without such tricks (i.e., we used X = 0.0 and regularization scale A. We use a = 0.4 for CIFAR- 10/100 and Tiny ImageNet datasets. We do not use mixup(a = 0.0) for ImageNet dataset.
the bridge trained with them will just copy the outputs of the base
0.0). We search 0.0 < X < 0.4 for the
A.3 EVALUATION
12

---

Under review as a conference paper at ICLR 2023
Efficiency metrics Dehghani al. (2021) pointed out that there can be contradictions between commonly used metrics (e.g.. FLOPs, the number of parameters, and speed) and suggested re- fraining from reporting results using just single one. So, present FLOPs and the number of parameters the results.
Uncertainty metrics Let p(z) [0. be a predicted probabilities for given input æ, where (b) denotes the kth element of the probability vector, i.e., (k)) is a predicted confidence kth class. We have the following common metrics on the dataset DD consists of inputs x and labels
Accuracy (ACC):
ACC(D) = () arg max (x)
(1I)
Negative log-lik bog-likelhhoo (NLL):
NLL(D)= E ((ny)E
log (e)p(()) (z))
(12)
Brier score (BS):
||(()) - 113
BS(D) = B(
(13)
where y denotes one-hot encoded version of the label y, i.e., g(v) 1 and (k) 0 for
k + y.
Expected calibration error (ECE):
Nis
Thy
16p
ECE(D, Nbin)
(14)
where Noin the number of bins, ny is the number of examples in the Oth bin, and 5n the calibration error of the 6th bin. Specifically, the bth bin consists of predictions having the maximum confidence values [(b- 1)/K, K. b/K), and the calibration error denotes the difference between accuracy and averaged confidences. We fix Nain 15 in this paper.
We evaluate the calibrated metrics that compute the aforementioned metrics with the temperature scaling (Guo et al., 2017), as Ashukha et al. (2020) suggested. Specifically, (1) we first find the op- timal temperature which minimizes the NLL over the validation examples, and (2) compute uncer- tainty metrics including NLL, BS, and ECE using temperature scaled predicted probabilities under the optimal temperature. Moreover, we evaluate the following Deep Ensemble Equivalent (DEE) score, which measure the relative performance for DE in terms of NLL,
DEE(D) = min {m 2 0 NLL(D) & NLLDEE
(15)
where NLL (D) denotes the NLL of DE-m on the dataset D. Here, linearly interpolate NLL (D) values for m E R and make the DEE score continuous.
A.4 COMPUTING RESOURCES
We conduct Tiny ImageNet experiments on 8 TPUv2 and 8 TPUv3 cores, supported by TPU Re- search Cloud' and the others on 8 RTX3090 cores. We attached code to the supplimentary material. We use PyTorch (Paszke et al., 2019) with BSD-style license. Visit PyTorch GitHub repository" for more details.
13

---

Under review as a conference paper at ICLR 2023
B ADDITIONAL EXPERIMENTS
B.1 ADDITIONAL EXAMPLES
We visually inspect the logit regression of type bridge network. Our bridge network very ac- curately predicts the logits of r = 0.5 from Bezier curve when the two base models (r = 0 and r 1) gives similar output logits (deer, ship, and frog). When the base models are confident on the samples (airplane, bird, cat and horse), the network recovers the scale of logits approximately. However fails to predict some very difficult samples (truck and dog) when even the base models very confused.
B.2 FULL TYPE AND TYPE II BRIDGE RESULTS
We report full experimental results for classification tasks; 1) Type I bridge network results in Ta- ble 5, Table 7, Table and Table 11, 2) Type II bridge network results in Table 6, Table 8, Table 10 and Table 12.
14

---

Under review as a conference paper at ICLR 2023
Brigge
looi
airpieme
Rriage
logi
Bridge
truck
loot
Briogg
brd
logit
Brioge
loot
hore
looi
ship
logt
frog
looit
Rriage
lodt
Birplane
2010
cat
deeer
dog
(mgg
horse
mip
truck
airponn
aidoo
Dird
car
denr
dmg
mgg
horse
ship
trock
ayplana
aldo Inr
cat
frog
horse
dhip
trock
Figure 6: Bar plots in the third column depict whether the bridge network (orange) outputs the same logit values the base model with the Bezier parameters ((b)) (0.5) (blue), for given test inputs displayed in the first column. We also depicts the predicted logits from 0, and 0 in the second and fourth columns, respectively.
15

---

Under review as a conference paper at ICLR 2023
Table 5: Full result of performance improvement of the ensemble by adding type I bridges CIFAR-10 10 dataset. We use same settings as described Table 3.
Model
FLOPs (1)
#Params (4)
ACC (1)
NLL
ECE (L)
BS (4)
DEE (1)
ResNNet (DE-I)
1.000
1.000
91.78 0.10
0.287
00.001
0.019 0.001
0.126 0.001
1,000
Bridge
x 1.062
1.048
92.07 = 0.04
0.001
0.009 0.00
0,119 0.000
1.606 + 0.035
+2 Br
x 1.124
1.096
92.13 0.06
0.250
0.0001
0.008
0.118
0.000
1.677
Bridgesm
1.186
L.144
92.13 0.01
0.250
0.000
0.009
0.000
0.118
0.000
1.692 0.033
Bridge..
1.248
L.192
92.12
0.250
0.000
0.009
0.001
0.119 0.000
1.695 0.041
Bridgena
x 1.205
1.159
92.09
0.05
0.253
0.000
0.009
0.000
0.119
0.000
1.623
Bridgemd
x
1,318
0.06
0.001
0.009
0.000
0.000
1,705 1: 0.047
Bridgend
1.615
L.477
92.15
0.01
0.248
0.000
0.008
0.000
0.118
0.000
1.729 0.048
Bridgend
1.820
L.636
92.14
0.247
0.000
0.009
0.001
0.118
0.000
1.736 0.049
DE-2
2.000
2,000
93.07 0.11
0.003
0.012
0.001
0.107
0.001
2,000
Table 6: Full result of performance improvement of the ensemble by adding type II bridges on CIFAR-10 dataset. We use same settings as described in Table 4.
Model
F FLOPS (1)
#Params (1)
ACC (7)
NLL-
BCE (L)
BS (1)
DEE (?)
DE-2
x 2.000
x 2,000
93.07 0.111
0,233 L. 1.000
0.012 a 0..001
0.107 0.00
2,000
Bridgea
2,081
x 2.063
93.09 = 0.09
0.220
+ 0.000
0.008
0.0011
0.104 + 0.001
2,774 + 0.084
Bridgend
2,247
2.192
93.07
0.07
0.219
1. 0.003
0.006
0.001
0.103
0.001
2,808 1. 0.105
Bezier
3.000
3.000
93.09
0.227
+ 0.004
0.009
0.000
0.104 + 0.001
2.357 + 0.080
DE-3
3.000
3.000
0.03
0.216
0.009
0.001
0.100
0.001
3.000
Bridge.
3.081
3.063
93.46
0.06
0.206 + 0.002
0.007 0.002
0.098
0.001
3.880 + 0.118
+ Bridge..
3.162
3.126
93.50 = 0.04
0.204 + 0.002
0.006 0.001
0.097
0.001
4.214 + 0.143
Bridge,
3.243
3.189
0.001 0.006
0.097 0.001
4.384
Bridgennd
3.247
x 3.192
93.45 = 0.044
0.206 1 0.002
0.007
0.001
0.097
0.001
3.918 1: 0.117
Bridgena
3,494
3,384
93.54 0.02
0.203 0.002
0.006
0.002
0.097
0.001
4.301 + 0.137
Bridgend
3.741
3.576
0.08
0.006
0.001
0.001
Bezier
6.000
x 6.000
93.44 = 0.10
0.207 1: 0.083
0.008
0.001
0.097
0.001
3.852 : 0.214
DE-4
4.000
x 4.000
93.59 = 0.
0.205 + 0.002
0.010 = 0.001
0.096
0.001
4,000
Bridge.n
x 4.081
x 4.063
93.58 = 0.06
0.199 1: 0.002
0.007 = 0.001
0.094 : 0.001
5.017 1: 0.088
2 Bridgem
4.162
x 4.126
93.64
1 0.03
0.197 0.002
0.006 1 0.001
0.094
0.001
5,333 1. 0.044
+ Bridge..
4.243
x 4.189
93.58
= 0.04
0.196
+ 0.002
0.006 = 0.002
0.094 = 0.001
5.484 + 0.090
Bridge
4.324
x 4.252
93.64
0.06
0.196 + 0.002
0.007
0.001
0.094
0.001
5.590 + 0.133
+ Bridges
4.405
x 4.315
93.58 = 0.04
0.196
0.002
0.005
0.001
0.094
0.001
5.647 + 0.15
Bridge.m
x 4.486
4.378
93.60
0.10
0.196
0.002
0.005
0.000
0.094
0.001
5.607 1. 0.120
+11 Bridgend
x 4.247
x 4.192
93.57 = 0.04
0.198
0.002
0.007 = 0.000
0.094 : 0.001
5.055 + 0.061
+ Bridgeod
4.494
x 4.384
93.61
= 0.05
0.197
+ 0.002
0.006
0.001
0.094
0.001
5.428 + 0.069
+ Bridgend
X 4.741
x 4.576
93.64 = 0.09
0.196 0.002
0.006 = 0.001
0.094
0.001
5,610 + 0.141
Bridgend
4.988
x 4.768
93.64
0.10
0.195
1 0.002
0.005
0.001
0.094
0.001
5.774 1. 1.1883
Bridgend
5.235
4,960
93.58
= 0.08
0.195
= 0.002
0.005
0.001
0.094
0.001
5,849
= 0.136
Bridge.nd
5.482
5.152
93.62
= 0.05
0,195
1 0.012
0.005
0.000
0.094
0.001
5,855
: 0.114
Bezier
x 10.000
x 0.000
93.63
0.10
0.197 1 0.002
0.005
0.001
0.094
0.001
5.322 1. 1.1511
DE-5
x 5.000
x 5.000
93.68 = 0.10
0.199 0.002
0.011 = 0.000
0.093 0.001
5,000
16

---

Under review as a conference paper at ICLR 2023
Table 7: Full result of performance improvement of the ensemble by adding type I bridges
CIFAR-100 dataset. We use same settings as described in Table 3.
Model
FLOPs (1)
#Params (4)
ACC (1)
NLL (0)
ECE (L)
BS (4)
DEE (1)
ResNeet (DE-I)
1.000
1.000
73.11 0.10
1.094 00008
0.047 0.003
0.379 0.002
1,000
Bridge
x 1.062
1.047
73.94 = 0.16
0.981
+.0004
0.022 0.003
0.363 0.00
1.802 + 0.036
+2 Br
x 1.124
1.094
74.02
0.05
0.957
0.000
0.018
0.0002
0.360
0.001
1.984
Bridge.m
1.186
L.141
74.00
0.09
0,948
0.002
0.017 0.001
0.360
0.001
2.078 0.070
Bridge.m
1.248
L.188
74.06
0.06
0,944
000001
0.018 0.001
0.360
2.143
Bridgena
x 1.211
1.161
74.18
0..11
0.965
0.000
0.024
0.0.00
0.358
0.001
1.914
+ Bridgemd
x
1.322
74.41
0,940
0.000
0.020
0,353
0.077
Bridgend
1,633
L.483
74.52 0.07
0,930
00.001
0.020
0.002
0.352
0.000
2.416 0.055
Bridgenn
1.844
L.644
74.54
0.02
0,925
0.000
0.021
0.001
0.352
2.517
DE-2
2.000
2,000
75.62 0.17
0.005
0.033
0.002
0.344
2,000
Table 8: Full result of performance improvement of the ensemble by adding type II bridges on CIFAR-100 100 dataset. We use same settings described in Table 4.
Model
F FLOPSs (1)
#Params (1)
ACC (7)
NLL-
ECE (L)
BS (1)
DEE (1)
DE-2
x 2.000
x 2,000
75.62 0.117
0,952 1 1.0.05
0.033
1.0002
0.344 1 0.002
2,000
Bridgen
x 2.079
x 2,060
75.98 = 0.20
0.894 + 0.003
0.021
0.002
0.335 0.001
3.158 + 0.061
Bridgend
2,246
x 2.188
76.13 1 0.144
0.890 1. 0.004
0.023
0.003
0.334
0.002
3,290 1. 0.084
Bezier
3.000
3.000
76.21 0. 10
0.909 +: 0.002
0.021
0.003
0,334
0.001
2,818 + 0.034
DE-3
3.000
3,000
76.58
0.899
0.003
0.025
0.003
0.329
0.001
3.000
Bridge.
3.079
3.060
76.74
0.09
0.859 + 0.003
0.019
0.002
0.325
0.001
4.357 + 0.107
+ Bridge..
3.158
x 3.120
76 82
0.10 16
+ 0.002
0.016
0.000
0.323
0.001
5.100 + 0.199
Bridge,
3.237
3.180
0.003
0.017
0.002
0.001
0.089
Bridgennd
3.246
x 3.188
76.82 = 0.13
0.858 1: 0.003
0.020
0.002
0.324 0.001
4.419 1 0.095
Bridgena
3,492
x 3,376
77.06
0.08
0.841 0.002
0.020
0.003
0.321
0.001
5.333 + 0.093
Bridge.d
3.738
3.564
0.04
0.018
0.002
0.001
6.123 0.098
Bezier
6.000
x 6.000
77.37 = 0.14
0.840 1: 0.083
0.017
0.001
0.317
0.001
5,407 1: 0.147
DE-4
4.000
x 4.000
77.14
0.16
0.867
0.001
0.023 = 0.001
0.321 + 0.001
4,000
Bridge.n
4.079
x 4.060
77.29
= 0.10
0.838
0.003
0.019 = 0.001
0.318 11 0.001
5.553 1: 0.116
+ Bridgem
4.158
x 4.120
77.35
1 0.06
0,826
1 0.002
0.016 1 0.001
0.317 1 0.001
6,825 1. 0.122
+ Bridge..
4.237
x 4.180
77.
0.06
0,820 0.001
0.015 0.001
0.317 = 0.001
7.675 + 0.280
Bridge
4.316
4.240
77.38
= 0.05
0.815 + 0.002
0.016 0.001
0.317 0.001
8.307 + 0.331
+ Bridge
4.395
x 4,300
77.33
= 0.06
0.812 + 0.002
0.014 = 0.002
0.317 + 0.001
8.695 + 0.363
Bridge.m
x 4.474
x 4,360
77.35
0.02
0.811 1 0.002
0.016 0.001
0.317 0.001
8.862 1. 0.381
+11 Bridgend
x. 4.246
x 4.188
77.40 = 0.14
0.837
0.002
0.020 = 0.001
0.317 = 0.001
5.578 + 0.079
+ Bridgeod
4.492
4.376
77.34
= 0.13
0.824 + 0.001
0.016 0.001
0,316
0.001
7.015 + 0.16
+ Bridgend
X 4.738
x 4.564
77
0.11
0.817
0.001
0.017 0.002
0.315
0.001
7,987 + 0.347
Bridgend
4.984
x 4.752
77.41
L 0.12
0.811
0.000
0.016
0.001
0.315
0.001
8.814 1. 0.398
Bridgend
5,230
x 4.940
77.45
0.12
0.809
0.002
0.018
0.002
0.315
0.001
9.228 : 0.529
Bridge.nd
5.476
x 5.128
77.41
= 0.14
0.806 : 0.012
0.018
0.002
0.001 0.315
9.563 :: 0.528
Bezier
x 10.000
10.000
77.82
0.06
0.808
0.002
0.016
0.002
0,310
0.001
9.270 1. 0.386
DE-5
x 5.000
x 5.000
77.49 = 0.11
0.845 0.001
0.021 = 0.000
0.316 0.000
5,000
17

---

Under review as a conference paper at ICLR 2023
Table 9: Full result of performance improvement the ensemble by adding type I bridges on Tiny ImageNet dataset. We use same settings as described in Table 3.
Model
FLOPs (4)
#Params (4)
ACC (7)
NLL (U)
ECE (L)
BS (4)
DEE (t)
ResNet (DE-I)
x 1.000
x 1.000
63.42 = 0.23
1.618 0.005
0.037 0.002
0.485
0.003
1,000
Bridge
x 1,088
x 1.093
65.38 = 0.09
1.444 0.000
0.015 0.00
0.461 + 0.001
2.179 0.101
+ Bridgea
x 1.176
x 1.186
65.55
a 0.15
1.405
0.000
0.013
0.001
0.456 0.001
2.750 1 0.086
Bridgesm
x 1.264
x 1.279
65.61
L 0..00
1.388
0.000
0.014
0.000
0.455
0.000
3.022 1. 0.079
Bridge.m
x 1.352
x 1.372
65.68
= 0.06
1.380
+ 0.002
0.012
0.002
0.454 0.000
3.233 = 0.081
1 Bridgeod
x 1.277
x 1,290
65.94
0.15
1.418
0.003
0.018
0.002
0.453
0.001
2.562 + 0.056
+ Bridgend
x 1.554
x 1,580
66.59
0.09
1.372
1.0.01
0.016
0.002
0.445 0.000
3.437 1. 0.036
Bridgend
x 1.831
x 1,870
66.79
=: 0.11
1.353
0.000
0.015
0.001
0.443
0.000
3.967
4: 0.043
Bridgendd
x 2.108
x 2.160
66.88
= 0.15
1.342
0.000
0.018
0.001
0.441
0.000
4.450 : 0.082
DE-2
x 2,000
x 2,000
66.21 = 0.11
1.456 0.00
0.022 = 0.002
0.450 + 0.002
2,000
Table 10: Full result of performance improvement of the ensemble by adding type bridges on Tiny ImageNet dataset. We use same settings as described in Table 4.
Model
FLOPs (1)
#Params (L)
ACC (1)
NLL (U)
ECE (J)
BS
DEE (t)
DE-2
x 2,000
x 2.000
66.21 i: 0.10
14466 0.004
0,022 + 0.002
0.450 0.002
2,000
Bridgeum
x 2.125
x 2.132
67.25
0.26
1,371
0.001
0.017 0.0002
0.438
0.001
3.478 :: 0.052
Bridgeat
x 2.352
x 2.367
67.57
+: 0.13
1.355 + 0.003
0.016 + 0.002
0.434
0.00
3.904 0.097
Bezier
x 3.000
x 3.000
67.43
0.04
1,385 +: 0.001
0.017 = 0.002
0.436 = 0.001
3,108
0.1121
DE-3
x 3,000
x 3,000
67.56
1: 0.08
1,388 1. 0.001
0.019 1. 0.001
0.435 1.0001
3.000
Bridge...
x 3.125
3,132
67.98
+ 0.09
1.336
0.002
0.017
0.002
0.429
0.001
4.702 = 0.079
+ Bridge,.
x 3.250
x 3,264
68.21
+: 0.21
1.314
+ 0.002
0,015 + 0.003
0.427
0.00
6.015 = 0319
+3 Bridgeun
x 3.375
x 3.396
68.29
1: 0.25
1,303 1. 0.002
0.014 1. 0.000
0.426
1: 0.000
6,822 1: 0.199
+1 Hridgens
x 3.352
x 3.367
68.29 + 0.14
1.327 + 0.002
0,016 + 0.000
0.426 + 0.000
5.159 + 0819
Bridgeme
x 3,704
x 3.734
68.43
1: 0.15
1,300 0.002
0.014 1 0.001
0.423 0.000
7.048
:: 0,250
Bridgeat
x 4.056
x 4.101
68.57
+ 0.17
1,285
+ 0.001
0.014 + 0.000
0.421 0.001
8.110 = 0.312
Bezier
x 6,000
x 6.000
68.72
0.21
1,307 + 0.002
0.016 = 0.002
0.420 0.001
6.464
0.100
DE-4
x 4.000
x 4,000
68.
+ 0.11
1.352 + 0.001
0.016 6 0.000
0.427 0.001
4,000
Bridge...
x 4.125
4.132
68.48 0.07
1.315 0.002
0.013 1 0.001
0.423 0.000
5.962 :: 0.127
4.2 Bridge...
x 4.250
x 4.264
68.67
+ 0.10
1,297
+ 0.002
0.015 0.001
0.422 0.000
7.239 = 0.226
+. Bridge.n
x 4.375
x 4,396
68.66
1 0.09
1,287 1: 0.001
0.012 1. 0.001
0..42 1: 0.000
7,970 1: 0.298
+-4 Bridge..
x 4.500
x 4.528
68.69 0.18
1,281 0.002
0.012 + 0.002
0.421 0.000
8.432 = 0.383
+ Bridge.
4.625
x 4.660
68.63
+ 0.11
1.279
+ 0.002
0.013 + 0.001
0.422 = 0.000
8,596 + 0.431
Bridge..
x 4.750
x 4,792
68,58 + 0.04
1.276 + 0.002
0.011 + 001
0.422 = 0.000
8.768 = 0.441
+1 Bridgem.
x 4.352
x 4.367
68.68 0.17
1.309
0.002
0.016 + 0.002
0.422 0.000
6,377 = 0.178
+22 Bridgeus
x 4.704
x 4,734
68.98 + 0.14
1,287
0.002
0.016
+ 0.001
0.419 0.000
7,986 : 0.347
+. Bridgems
x 5.056
x 5.101
69.01
1.17
1.275
0.001
0.015
0.001
0.417
1 0.000
8.892 1. 0.389
Bridgem.
x 5.408
x 5.468
69.14
0.13
1,266
0.001
0.015
1 0.001
0,416 0.000
9.539 = 0.481
Bridgeas
5.760
x 69.12
0.12
1.261
0.001
0.014 0.000
0.416 0.000
9.916 0.516
Bridgems
6.112
6.202
69.15 0.19
1.257
0.015
0.000
0.416 0.000
10.198
Bezier
10.000
10.000
69.26 0.07
1.271 0.002
0.016 - 0.001
0.414 = 0.000
9.118 = 0.383
DE-5
x 5.000
x 5.000
68.54 0.08
1.329 0.001
0.018 0.001
0.422 0.001
5,000
18

---

Under review as a conference paper at ICLR 2023
Table 11: Full result of performance improvement the ensemble by adding type I bridges on ImageNet dataset. We use same settings as described in Table 3.
Model
(LOPs
#Params (L)
ACC (T)
NLL (L)
BCE (U)
BS (U)
(1)
Res/et (DF-1)
x 1.000
x 1.000
75.85
0.06
1.618 0.005
0.037 0.002
0.485
0.003
1,000
1 Bridgend
x 1.194
x 1.222
76.57
0.02
1.418 +0000
0.018 0.002
0.453 = 0.001
2.562 + 0.056
Bridgead
x 1,388
x 1.444
76.74
0.05
1.372
0.001
0.016
0.000
0.445 0.000
3.437 = 0.036
Bridgend
x 1.582
1.666
76.85
0.05
1.353
0.000
0.015
0.001
0.443
0.000
3.967
4: 0.043
BridgSod
x 1.776
1.888
76.96 = 0.03
1.342 0.000
0.018 0.001
0.441 0.000
4.450 : 0.062
DE-2
x 2,000
x 2,000
77.20 0.07
1.456 0.00
0.022 0.002
0,450 0.002
2,000
Table 12: Full result of performance improvement of the ensemble by adding type bridges on ImageNet dataset. We use same settings described in Table 4.
Model
FLOPs (4)
#Params (4)
ACC (7)
NLL (U)
ECE (L)
BS (4)
DEE (1)
DE-2
2,000
x 2.000
77.20 0.07
0.880 0.002
0.013 0.000
0,317:0000
2,000
Bridgead
2.243
x 2.256
77.43 0.05
0,870 + 0.001
0.012 0.0000
0.314 0.000
2.564 + 0.046
+ Bezier
3,000
3,000
77.65 0.08
0.861 0.001
0.011
0.001
0.311 0.001
3,059 + 0.082
DE-3
3.000
3,000
77.64 0.04
0.862 0.001
0.013 0.001
0.311 0.000
3.000
Bridgend
3.243
3.256
77.76
0.07
0.856
0.001
0.012 0.000
0.310 0.000
3.559 0.038
Bridgend
3.486
3.512
77.82
0.853
0.012
0.309
3.850
3. Bridgend
3.729
3.768
0.06 77.92
0.851 0.000
0.012 0.001
0.000 0.308
4,010
0.063
Bezier
6.000
6.000
0.05
0.834
0.012 0.000
0.303
7.821
0.04
0.001
0.012 0.001
0.000
19

---

Under review as a conference paper at ICLR 2023
Table 13: FLOPs, R scores, and model performance metrics between type bridge network and CNN Xn CNN models with various sizes on CIFAR-100. Here, Xn denotes the number of convolution layers to use as frontal feature extractor CNN and CNN denotes the same architecture used in the type I bridge network. R scores are measured with respect to the target Bezier 0.5.
Model
FLOPs ())
Rº (1)
ACC (1)
NLL (U)
ECE (J)
BS (U)
Type Bridge
0.211
0.786
+0.003
72.18 +0.19
1.016 +0.004
0.031 +00002
0.379 +00001
CNNINO x2 + CNN
0.230
0.626 +0.007
63.45 +0.71
1.318 +0.019
0.015 +0.002
0.481 +0.007
CNN x4 + CNN
x 0.373
0.682
+0.005
67.39 40.42
1.166 #0.015
#0.001
0,436 #0,004
CNN x6 + CNN
x 0.515
0.685
+0.006
67.43 +0.55
1.156 #0.019
#0.001
0,433 #0.006
CNN x8 | CNN
x 0.648
0.701 10.001
68.36 10.35
1.121 10.009
0.012 10.002
0,422 10.004
B.3 COMPARISON WITH THE TYPICAL KNOWLEDGE DISTILLATION
We note that mimicking the original function defined by deep neural networks using relatively cheaper networks reminds of Knowledge Distillation (KD) (Hinton et al., 2015), and thus one can think of the proposed approach as a special instance of the knowledge distillation. However, the proposed bridge network differs fundamentally from KD in that; 1) it uses very small network that cannot be properly trained with a typical distillation procedure, and 2) while KD builds a student mapping input to the output, ours reuses outputs from the models related to the target function to be mimicked, and this actually plays a key role in the function matching.
Here, we empirically validate the claim. Specifically, Table 13 compares bridge networks mimicking output probabilities from (bbe) 1) when takes inputs as in the typical knowledge distillation framework, and 2) when it takes outputs from 0, and 02 we proposed. The former consistently the latter, even we introduce some frontal convolutional layers for dealing with image inputs. It indicates that the typical knowledge distillation procedure suffers from an insufficient capacity the bridge network, while our proposed method does not. Consequently, proposed method, which reuses informative outputs from 0 and 0g. is distinct from the typical knowledge distillation when the capacity of the bridge network limited.
underperforms compared
20


Reviews:
[{"review_id": "iIxJpheDIF", "reviewer": "Reviewer_bBzc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes two methods to reduce the inference cost of Deep Ensemble (a collection of ensemble models of the same model structure). Bézier Curve is fitted on the extracted features by a neural network (bridge neural network), so that \"interpolated\" inference results can be cheaply approximated by the bridge NN.", "summary_of_the_review": "The paper is well motivated and easy to follow, but still lacks solid grounding for its claims. ", "strength_and_weaknesses": "Strength\nThe work is well motivated and the two proposed bridge networks method are well crafted.\n\nWeakness\n1. there is a key alternative not compared, in Learning Neural Network Subspaces a method is proposed to train models that can be linearly interpolated. Though it is not easy to compare the approaches, one is altering the training process and the other is more about adding an ancillary equipment, the two approaches can be evaluated in the same coordinate system with inference cost and accuracy on two axes.\n1. the method has not been verified on large scale datasets, though it criticized literature with \"these methods do not scale well for complex large-scale datasets\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fsF66LiejrM", "reviewer": "Reviewer_Kw57", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way of using an ensemble of neural networks to construct a new classifier that achieves better accuracy at a lower number of FLOPS (at inference time). This is achieved by creating so-called \"bridge-classifiers\" that approximate the output of an interpolating classifier between pairs of ensemble-base classifiers. The interpolating classifiers are picked from the low-loss Bezier curve in parameter space between the pair of ensemble members. The approximation classifiers are faster than computing more ensemble members because they are smaller classifiers trained on intermediate features that have already been computed in the ensemble. The paper provides experiments on (small) image datasets (CIFAR and Tiny-ImageNet) and (small) classifiers of the ResNet family. ", "summary_of_the_review": "The paper contains interesting approaches and results. However, there are also some weaknesses and limitations. I therefore think that the paper in its current form is marginally below the acceptance threshold, but I would be happy to increase my score during the discussion period if some of the concerns can be addressed by the authors.", "strength_and_weaknesses": "**Strengths**\n* The paper proposes a novel idea for approximating gains of ensembles at lower computational cost (at inference time).\n* The presented results are interesting and show that the presented method works (for small data set and classifier sizes).\n* The paper is generally easy to follow. \n\n**Weaknesses**\n* The method is relatively complex in relation to the performance gains that can be achieved.\n* A direct comparison to other methods in terms of the accuracy/computation frontier is missing. It seems that the numbers are reasonable, but e.g. the range of 63% to 67% on Tiny-INet (Table 4) is in the area that can be also reached by a single ResNet-18 according to [Papers with Code](https://paperswithcode.com/sota/image-classification-on-tiny-imagenet-1). Also, there are no direct comparisons to other \"efficient ensemble\" approaches.\n* FLOPS alone are often not a sufficient measure of model efficiency, see e.g. [The Efficiency Misnomer](https://arxiv.org/pdf/2110.12894.pdf)\n\nFurther comments:\n\nIt was not clear to me after reading the paper at which point in the base networks the z_{i} are extracted. I think this should be clarified (or maybe I missed it?)\n\nI did not see an explanation for the term \"Bridge-S\" in the paper. (Maybe I missed it?) I guess it refers to a \"small\" Bridge model, but I did not see what exact configuration \"Bridge\" and \"Bridge-S\" refer to.\n\nI found the discussion of knowledge distillation a bit distracting from the main idea. Especially the experimental setting described in the last paragraph of Sec 5.2 has a form that is not super-convincing in my opinion.\n\nIt seems to me that the most interesting setting among those discussed is the DE-1 + n bridges. In that case it would actually be possible to try to approximate the r=1 responses of the ensemble (i.e. approximate the outputs of the other ensemble members from the intermediate feature). If this worked similarly well, the whole construction of the Bezier interpolation could be avoided. I think this might be an interesting ablation experiment, but maybe not feasible to perform during the rebuttal.\n\nOverall, I found Sec.4 rather short, given that this area of research is currently quite active. For example, the following papers could be considered relevant here. (I'm not requesting that all of these should be cited, just to give a few examples).\n* [Batch Ensembles](https://arxiv.org/pdf/2002.06715.pdf)\n* [Ensembles with Shared Representations](https://arxiv.org/pdf/2103.03934.pdf)\n* [Evaluating Scalable BDL](https://arxiv.org/pdf/1906.01620.pdf)\n\nMinor points or typos not affecting the evaluation:\n* It seems that Eq. (3) and (1) can be combined into a shorter version of the two equations.\n* p.3 line ~5: parameters ... achieves -> achieve\n* p.3 line ~5: add them to [the/an] ensemble\n* p.3 line ~8: \"these strategy provide\" -> this strategy provides\n* p.4 Eq. (9): last \"i\" should be a \"j\"\n* Algorithm 1 looks like a fairly standard training loop, unless I overlooked something. Maybe removing it would enable moving parts of the Appendix into the main paper. \n* p.4 bottom: \"when the bridge network does not step toward the Bezier predictions\" - I did not fully understand what this means.\n* Table 2: The FLOPS are given as a multiple. I assume this is as a multiple of a base network? \n* p.7 ~center: \"achieve decent R^2 score\" -> \"a score\" or \"scores\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NQIk5pN8arg", "reviewer": "Reviewer_tqBD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new ensembling framework to reduce inference cost and boost the performance. The proposed method build additional light-weight head as bridges to ensemble different runs. The experiments show the effectiveness of the proposed method on CIFAR-10, CIFAR-100, and tiny-ImageNet.", "summary_of_the_review": "(1)The idea of the proposed deep ensembling is novel and interesting.\n(2) The experiments on CIFAR-10, CIFAR-100, and tiny-ImageNet shows the improvement, but do not scale up to large-scale datasets and large-scale backbones (or SOTA backbones) .", "strength_and_weaknesses": "Strengths:\n(1)\tThe idea of building bridges between different models for ensembling is interesting. It re-uses the features and only focuses on classification heads. Therefore, it would reduce the inference cost compared with directly ensembling different models.\n(2)\tThe experiments on several datasets show the effectiveness of the proposed framework.\n\n\nWeaknesses:\n(1)\tI concern only considering classification would not lead to diverse solutions to ensembling. As described in the paper, the bridges are light-weight MLPs.\n(2)\tThe paper pointed out “these methods (the existing methods) do not scale well for complex large-scale datasets or require network capacity”. However, this paper also DO NOT scale up on complex large-scale datasets, such as ImageNet-1k. Since the experimental platform is on 8 TPUs, it is expected to conduct on large-scale datasets such as ImageNet-1K, large-scale backbones (or SOTA backbones) such as transformers. The experiments on CIFAR-10, CIFAR-100 or tiny ImageNet are not convincing to me.\n(3)\tMinor comments, Page 1: the modes, three models, two modes? Mode or models?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ig5isFeo4T", "reviewer": "Reviewer_iSwW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address an important drawback of Deep Ensemble, the inference cost of executing multiple models. The intuition behind the proposed method is that the outputs in the function subspace can be estimated from the modes without having to forward the actual parameters on the subspace. Based on this intuition, an additional lightweight network is trained as a bridge network to predict the outputs from the connecting subspace. ", "summary_of_the_review": "I have some concerns about \n1) fundamental difference between type I and II bridge models\n2) Why number of bridge models impact \n3) generalization of this method to the large-scale datasets\n\nI am willing to raise my score if the above concerns are addressed. ", "strength_and_weaknesses": "### Strength \n\n* Reducing the inference cost of DE is an important topic. This paper is well-motivated. \n* The presentation of this paper is clear and easy to follow. \n* Extensive experiments demonstrate the effectiveness of the proposed two types of bridge networks on TinyImageNet and Cifar. \n\n### Weaknesses\n* In Table 4, why do more bridge models lead to better results? \n* What's the functional difference between type I and II bridge models according to theoretical and empirical results? Is there any conclusion we can reach about how to choose types I and II?\n* Are there any important/sensitive hyper-parameters during the training of bridge models? How to determine the optimal number of bridge models in practice? \n* I am curious about how the proposed methods perform on large-scale datasets like ImageNet. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject