Fast Ensembling with Diffusion Schrödinger Bridge
Authors: Hyunsu Kim
Jongmin Yoon
Juho Lee
Venue: ICLR 2024 poster (2024)
Keywords: Deep Ensemble
Diffusion Schr\"odinger Bridge
Bayesian
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Deep Ensemble approach is a straightforward technique used to enhance the performance of deep neural networks by training them from different initial points, converging towards various local optima. However, a limitation of this methodology lies in its high computational overhead for inference, arising from the necessity to store numerous learned parameters and execute individual forward passes for each parameter during the inference stage.We propose a novel approach called Diffusion Bridge Network to address this challenge. Based on the theory of Schr\"odinger bridge, this method directly learns to simulate an Stochastic Differential Equation (SDE) that connects the output distribution of a single ensemble member to the output distribution of the ensembled model, allowing us to obtain ensemble prediction without having to invoke forward pass through all the ensemble models. By substituting the heavy ensembles with this lightweight neural network constructing DBN, we achieved inference with reduced computational cost while maintaining accuracy and uncertainty scores on benchmark datasets such as CIFAR-10, CIFAR-100, and TinyImageNet.

Published as a conference paper at ICLR 2024
FAST ENSEMBLING WITH DIFFUSION SCHRÖDINGER BRIDGE
Hyunsu Kim", Jim',Jongmin Yoon", Juho Lee
Kim Jaechul Graduate School of AI
KAIST Daejeon, South Korea
..m.husuu, jm. yoon, jubolee))kaisst.ac.k
2XXi::444.1581444 [cs.LG] 24 2024 2024
ABSTRACT
Deep Ensemble (DE) approach is a straightforward technique used to enhance the performance of deep neural networks by training them from different initial points, converging towards various local optima. However, a limitation of this methodology lies in its high computational overhead for inference, arising from the necessity to store numerous learned parameters and execute individual for- ward passes for each parameter during the inference stage. We propose a novel approach called Diffusion Bridge Network (DBN) to address this challenge. Based on the theory of the Schrödinger bridge. this method directly learns to simulate an Stochastic Differential Equation (SDF) that connects the output distribution of a single ensemble member to the output distribution of the ensembled model, al- lowing to obtain ensemble prediction without having to invoke forward pass through all the ensemble models. By substituting the heavy ensembles with this lightweight neural network constructing DBN, achieved inference with reduced computational cost while maintaining accuracy and uncertainty scores on bench- mark datasets such CIFAR-10, CIFAR- 100, and TinylmageNet. Our imple- mentation is available at https: // tig....b........... com/ com/kim -immmmmmmmm hy don.
INTRODUCTION
Deep Ensemble (DF) (Lakshminarayanan et al., 2017) is one of the straightforward yet powerful techniques for improving the performance of deep neural networks. This method involves averaging the outputs of multiple models trained independently with different random initializations and data scan orders. DE is an instance of bagging (Breiman, 1996) where the models are trained with fixed datasets, and can be interpreted as an approximatation Bayesian Model Averaging (BMA). DE can significantly improve the prediction accuracy of deep neural networks, and more importantly, the uncertainty quantification and out-of-distribution robustness for tasks across various domain.
However, the major drawback of DE. lies in the fact that the computational overhead and memory usage during inference calculations scale linearly with the number of ensemble members, which may be problematic for resource-limited environments or when model is prohibitly large. Various approaches have been proposed to mitigate this issue, ideally reducing the inference cost of DE down single forward pass while minimizing the degradation in the predictive performance. A popular method in this direction is ensemble distillation, which is based on knowledge distillation (Hinton al., 2015) over the ensemble outputs. ensemble distillation, the average output of multiple ensemble members is set as the output of the teacher network, and single model is set student whose output is learned minimize the discrepancy from the teacher outputs. Additionally, tech- niques involving the use of multi-head models (Tran et al., 2021), shared weights (Wen al., 2019), learning ensemble distributions (Malinin al., 2020), employing (Nam et al., 2021), training generators simulating the ensemble predictions (Penso et al., 2022) have been presented. However, these methods either still demonstrate performance inferior to DE or require an inference cost comparable to DE in order to achieve equivalent performance.
augmentation
Recently, Yun et al. (2023) proposed an orthogonal approach to reduce inference costs of DE. They achieve this by connecting ensemble members through low-loss subspaces, employing tech-

---

Published as a conference paper at ICLR 2024
niques such as Bezier curves between each pair of ensemble members based methods outlined by Garipov et al. (2018). For each ensemble member, a low-loss subspace is chosen, originating from that member, and a parameter within the subspace is sampled (usually from the center of the subspace). To streamline the inference process, they introduce Bridge Network (BN), a lightweight neural network that takes intermediate feature from the ensemble member and directly predicts the output originally computed from the model on the low-loss subspace. The final output is ap- proximated through an average of the ensemble member's output and the output predicted by BN. Crucially, BN is designed to have negligible number of parameters and inference cost compared to the full ensemble model, resulting in significantly reduced inference costs. Experimental results us- ing real-world image classification benchmarks and large-scale deep neural networks validate their approach, showcasing faster ensemble inference with sub-linear scaling of inference costs relation to the number of ensemble members.
However, BN presented in Yun al. (2023) comes with critical limitations. First, does not directly predicts the output from the other ensemble members, but only for the models on the low-loss sub- spaces. This incurs an extra training cost of learning low-loss subspaces between all pairs of ensem- ble members. Second, single BN can only be constructed between pair of ensemble members. This means that as the number of ensemble members grow, the number of BNS to be constructed grows quadratically. Due to this structure, even though they achieved sub-linear growth in inference cost, there still is a large room for improvement.
In this paper, improving upon BN, present novel method for reducing inference costs of en- semble models. Given a set of ensemble members, we designate one of the ensemble members as a starting point. Then we train a mapping that transports an output from the starting point to the out- put computed from the ensemble model (averaged output). Compared to BN, our approach does not require learning low-loss subspaces, and do not require learning as many mappings as the number of pairs among ensemble members, at the cost of increased complexity to learn the mapping. Instead it learns this mapping via Diffusion Schrödinger Bridge (DSB) (Bortoli et al., 2021; Liu et al., 2023), a powerful model that can build a stochastic path between two probability distributions. Leverag- ing DSB, we create a sequence of predictions that progressively transition from the predictions of the starting point model to those of the ensemble model, capturing the inherent correlations among these predictions. Importantly, recognizing our primary objective of lowering inference costs, we design the score network of DSB to be lightweighted and incorporate diffusion step distillation to further minimize computational overhead. We refer to our approach as Diffusion Bridge Network (DBN), and their experimental results demonstrate significant improvements in the efficiency of en- semble inference cost reduction, surpassing the performance of BN and other ensemble distillation techniques.
2 BACKGROUNDS
2.1 DEEP ENSEMBLE AND BRIDGE NETWORK
In the ensemble methods as DE (laashminarayannn et al., 2017), a single neural network trained M times with the same data but different random seeds (thus with different initializations and data processing orders), yielding M different models with parameters { located in different modes in the loss surface. For the prediction, the outputs from those M models are averaged the output level to construct final output. The M members participating in ensemble often disagree on their predictions, thus giving functional diversity facilitating more accurate, robust, and better calibrated decision makings. However, it requires M number of models loaded on memory and M number of forward computations.
The BN (Yun et al., 2023) is one of methods suggested for reducing computational costs of ensemble methods. BN hinges on the mode connectivity (Garipov et al., 2018) of ensemble members, meaning that it is possible to connect two different modes via a low-loss subspace, indicating the intrinsic connection between them. The main intuition behind BN is that, if we can learn such a low-loss subspace, then we may directly predict the outputs computed from the parameters in the output level. In detail, consider a neural network fo(.) with a parameter 0 trained with a loss function L(0). Let O and 0 be two modes. BN first search for new parameter (i.((a) that
the subspace
2

---

Published as a conference paper at ICLR 2024
satisfies on the parametric Bezier curve,
(I) where the anchor parameter oig is obtained by minimizing Eo- |c (02)(0))) so that the parameters J
(i)(a) = (1 - a) + aj(1-a)0;, + a²
the curve locate on the low-loss subspace. After building such curve, a light-weight neural network s (BN) is trained, where & gets a feature vector 2; of an input æ computed solely from the first model 0, and try to predict the output evaluated at the center of the Bezier curve (0.5) for the same input æ. That is, the BN is an estimator trying to approximate (0.5) (az) R s(2)). Then the ensemble of two models, O, and (.g can be approximated via the BN as follows:
{ Se. (x) - (0., (1) (z) = 2
1 ( So. (m) + s(2i) s((z))
2
Since the (0.5) encompasses the information of both 0; and 0,. is expected to approximate the outputs to decent quality. However, since BN does not directly predict the output of 0, but only approximates the models with (0.5), has limitation in mimicking the actual ensemble predic- tions. In addition, single bridge network is constructed between only pair of given ensemble members. As a result, the number of BNs to be constructed may grow quadratically the number of ensemble members, leading to extra training costs.
2.2 DIFFUSION SCHRÖDINGER BRIDGE
DSB (Bortoli et al., 2021; Chen et al., 2022) is a conditional diffusion model that solves Schrödinger Bridge (SB) problem (Schrödinger, 1932), an entropy-regularize optimal transport problem that finds the diffusion process between two distributions. Even though the SB problem provides the finite-time solution on finding the probability path, it requires iterative optimization and inference procedure which is time-consuming. and has rarely demonstrated its practicality in deep learning models albeit its theoretical soundness. Recently, Liu et al. (2023) proposed tractable special case of DSB called Image-to-Imee Image Schrödinger Bridge (I'SB) for an application of image manipulation such as image restoration or super-resolution. In their work, a tractable special case of DSB is proposed, and has demonstrated its efficiency for real-world image datasets. Due to its training stability incurred from training a single score network, we choose it out of several DSBS despite of its strict condition. In this section, we introduce brief overview of i'S method.
First of all, Schrödinger Bridge (SB) is an optimal transport problem that seeks to find the forward and backward processes
dz. = [f: + 3, V log V (Z,, + v B,WW, Zo ~ po Z1 ~M~0
(3)
dz. [ft - 3r v log V (Z,, d) + / B,dW t,
where (po. P1) the boundary distributions, (W, W. are the standard Wiener process and its time-reversal, and 3.} are the drift and diffusion coefficients. If the pair of functions {v, $ solves the following coupled PDE
aue (v1( t) Vy f. B.AV,
1
- ai _ (v,.) 2
1 3.AV
ØL
with (zo, 0)0 0) = po())),
(z1,,
then (3) provides the optimal solution to an cutropy-regulariziin optimization problem that finds the optimal path between Po and P1. Then (4) and its time-reversal directly follows the Fokker-Planck equation of the SDE in (5) as follows, respectively.
dz,- = fidt + /B.WW, Zo ~ ( 0) and dZ, = fidt + VB,dW, Z, ~ ~.,1l (5)
However, both and v are intractable drifts so 1' assumes a certain form of the boundary distributions Po and p1- Followed from Liu et al. (2023), we take the energy potential functions 1). Here 5a(-) is the Dirac delta distribution
(.,00) (-(-) centered at a E Rd which makes the diffusion process computationally tractable. Then, we can ap-
(.) and () Pı
proximate both forward and backward Stochastic Differential Equations (SDES) using single score network in the framework of DDPM (Ho et al., 2020) with the following Gaussian posterior:
of
o
Σg = ofo
Z, ~ q Zz Zo. Z1) = NZ Ex), 1t = 2
Zo
+
Z1,
(6)
0 + o
2
0 of
3

---

Published as a conference paper at ICLR 2024
((
Z
Zn
21
EsoLogis
Conditional Diffusion Bridge Eg
CA
X2
33
fo
foz
f0
90h
Figure 1: Overview of DBN. For given data, the conditional diffusion bridge learns transition between logit distribution of the ensembles (left; source) and that of the target ensemble models (right; target).
where of fo dy' and ož S: dy are cumulative forward and backward noise vari- ances, respectively. For algorithmic design for restoration problems, take p(Zo, Zi) po( (o) 2 (Zo and f 0, and construct tractable SBS between individual samples from Zo
and p1 Zo).
DIFFUSION BRIDGE NETWORKS
In the previous BN (Yun et al., 2023), there was limitation in learning the transport between in- dependent models, prompting them to adopt an alternative approach: utilizes a neural network to predict the output of local optima aligned between the source and target models in terms of a low loss subspace instead directly predicting the output of the target model. To overcome this limi- tation, instead of tackling the problem of predicting the ensemble outputs as one-step prediction (a single neural network evaluation), we cast the problem as a diffusion bridge construction between the output distributions of ensemble members.
3.1 SETTINGS AND NOTATIONS
We restrict our focus on the K-way classification problem, where the goal to train a classifier taking d-dimensional inputs and predict K K-dimensional classification logits. We assume that clas- sifier with parameter is decomposed into two parts, a feature extractor J6- Rd + R& that first encodes an input æ into a feature vector h, and then a classifier cx Re + R that transforms the feature vector h into logit of the class probability. That is, So (x: (cx I (ar) and (1b A). A collection of M ensemble members denoted θ (0, ==1 and 0, (+), A4)-
3.2 CONDITIONAL DIFFUSION BRIDGE IN LOGIT SPACE
Our goal is to construct a conditional diffusion bridge on the logit space, that is, stochastic path (Z.heejo.1) where Z, constructed from the logit of the source model and Zo from that of the target ensemble model. More specifically, given O, we choose one of the ensemble members 0 as a source model, and set 21 be the logit distribution computed from O,. Then the target 2n set to be the logit distribution of the ensembled prediction.
More specifically, let Rdd be an input, and let h; Iw, (x) be the corresponding feature vector computed from the source model 0,. Conditioned on we define the source logit distribution as an implicitly defined distribution as follows.
z cx (hi), T Premps
where T is annealing temperature drawn from distribution Piemp- The target logit, Zo, is then set to be the logit computed by the ensemble model as follows, for 2 = fo. (x),
K
= Soft (2), Zu = EnsLogit ((29)) = log P
1
Σ
log Pi
(8)
K
k=
4

---

Published as a conference paper at ICLR 2024
Algorithm Training DBNS
Require:
An (empirical) data distribution Piaas temperature distribution Prenp- ensemble parameters {0,) and the score network SΦ.
Fix source model fo,- while not converged do
Sample æ ~ Pilata-
for = 1 to M do
Compute the logits z, = fo, (z))
end for
Gct a target ensemble logit Zo EnsLogit((2
Draw temperarture Phemp and compute the annealed source logit Zi Z1/T.
Compute the loss according to (10), and update Φ t- Φ- - V (()).
end while return Øb.
where the ensemble output p = E P./M. Then, we get Softmax Zo) = p, allowing us to get ensemble output p through approximating Zo. Next we construct i'SB between 11 and Zo.
M
The intuition behind this construction is as follows. If we directly the original logit æ1, the learned DSB can easily be trapped in a trivial solution where it just produces the copy of the source logit 11 along the path [Ze (Ee) I), the discrepancy between the source logit and the target logit is not large compared to the typical situation for which DSB is constructed. That is, the source 2 be a strong hint that acts simplicity bias for DSB learning. Also, 12sB requires the source of the diffusion to be conditional probability distribution, but z1 is a deterministic value. Hence by randomly annealing the source logit via a temperature T, we can naturally construct the source as the distribution of the annealed logits and also dilute the information included in 21, and this encourages DSB to discover non-trivial paths between the source and the target.
Based on the formulation on constructing the conditional diffusion bridge between the source and target distributions, we build an approximate reverse SDE derived from (3) that simulates the path from Z to Zo by estimating the score function V log & ,, 1/ (1)) ) = E (h1, Ze, t)/B, as
dz: = (Be/ (t//o))s Ze, t)dt + v B,dW,, Zi PI 1 x);
where is the distribution of Z, implicitly defined as in (7) and S 1 x IR K x [0, 1] + g is the score function estimator built with a neural network parameterized by Φ. and trained estimate the score function V z log (z
x)) by minimizing the following objective function,
[ Z.,t) Z. Zo
[(Φ) = Ex Z.
(10)
with Pdnaa L ([0, 1]), and Z, pipeline is summarized in Algorithm 1.
q(Z, Zo, Zi) as defined in (6). The overall training
3.3 DISTILLATION OF DIFFUSION BRIDGE
Even though the family of diffusion models (Ho al., 2020; Bortoli et al., 2021; Liu et al., 2023) achieves superior performance in learning generative models or constructing paths between dis- tributions, they typically suffer from the slow sampling speed due evaluations required for simulation. The distillation techniques for diffusion models, which distill multiple steps of the reverse diffusion process to a single step, do not significantly harm the gener- ation performance while accelerating the sampling speed. In this paper, we adapt the progressive distillation proposed in (Salimans & Ho, 2022) to reduce the sampling cost of DBN.
large number of function
Let T = [4:), be the discretized time interval used for diffusion bridge, with 0 = =0 <4 < IN 1. Then let T (1,) be the distilled time interval with T TT Let Z P& Ze,, Z.,) be the sample from ancestral sampling following the score network &s. Then distilled score network with parameter Φf is then trained with the loss
Z 2
Caisan = Exs
Exr (h1 Zez E2)
(1I)
5

---

Published as a conference paper at ICLR 2024
where j ~ U([1...... NN) and z - (2 (6,1)) with tj-1 t and tjj t,-
This distillation then recursively repeated until there only remains a single time step (N' 1).
3.4 INFERENCE PROCEDURE
The inference with DBN consists of forwarding input through the source model and computing single diffusion step from the model distilled by (11). Given an input æ, we first compute its feature h1 using the feature extractor of the source model 9 with h1 94,, (æ). Then we initialize the diffusion by drawing T premp and put Z = T.. The corresponding ensembled prediction y is then approximated as,
Zo = 1 + (81/01) (h1, 21, 0) + E where E ~ ~N(00,
1
(12)
ply xr, {0:)
Sof Softmax(z(")) Softmax(Zan (o)).
M
3.5 COMBINING MULTIPLE DBNS
Note that the size of the score network EΦ/ should be limited, because otherwise the cost from the diffusion simulation can outnumber the cost of computing the full ensemble. Hence, there is an intrinsic limit in the capacity of Es' representing the path between the source model and an full ensembled model (we study this capacity empirically in Š 4.3). When single DBN reached out to the limit, we may introduce multiple DBNs, increasing the approximation quality at the cost of additional inference time. Given M ensemble models, we first build a DBN starting from a source model 01. Then we build another DBN sharing the same model 01 the source and
Let
) (0 be set of score networks built in that way, with L M. Then we can approximate the ensembled prediction y for x as,
(
1 - tmat (Z I. 2=1
ply αx (0:1M)) R2
(13)
where 2 is the sample drawn as in (12) with hth score network. In our implementation, we combine L DBNS where 1th DBN estimates the recovered predictions by a collection of N + models {fo. fou- fo
}. approximating the ensembled prediction of M = LN + ensemble models. For example, to simulate total M = 5 ensemble models 02. 33. 04, 0s), we first train a score net that estimate ensemble distribution of {0, 02. 03) starting from 0, and secondly train another score net approximating {0,, 04, 05) starting from 0,. Note that the source models for DBNs need not be different; in $ 4, we show that a single source model can be shared for all multiple DBNS, minimizing the additional inference cost while signifi- cantly improving the accuracy.
EXPERIMENTS
Settings. We evaluate our approach using three widely adopted image classification benchmark datasets: CIFAR-10, CIFAR-100, and Tiny (Li et al., 2017). In our experiment, the en- semble models that we construct to serve as bridges adopt the configuration outlined in the Bridge Network (Yun et al., 2023) and are trained based on the ResNet architecture. We use widely used Res,et-32x2, ResNet-32x4 and ResNet-34 networks (He al., 2016) for baseline ensemble clas- sifiers for CIFAR-10, CIFAR-100, and TinylmageNet datasets, respectively. In this context, the suffices 2 and "x4" denote that the channel widths of the convolutional layers are multiplied by 2 and 4 from the conventional ResNet-32, respectively. For the score network, inspired by Sandler et al. (2018), we utilize residual connections (He et al., 2016) and Depthwise Separable Convo- lution (DSC) (Chollet, 2017) to mitigate redundant computations and implement wiggh-weighted score network. Further details on datasets, model architectures, and hyperparameter settings used to evaluate our experiments are listed in Appendix D.
Training. We train a single score network with 3 ensembles. If we need to mimic more than ensembles, we can average 2 more diffusion bridges according to & 3.5. Since they share the

---

Published as a conference paper at ICLR 2024
stapo
Scaror
Dillaao Stege
Begel
10.3
3.00
D+0
$..00
50.0
Figure 2: Confidences from the source model (first column), from the ensemble model (third column), and from the diffusion bridge (middle column) in the CIFAR-10 dataset. The middle column illustrates a transition of the diffusion process.
50m
Figure 3: The number of teachers that single model can distill in terms of ACC (left) and DEE (right). DEEs less than are set to 0.5.
source model, we can easily enhance the performance with low extra costs. We train the diffusion bridge with steps before distillation for fast and efficient training the score networks and steps are enough to approximate the transport between the two conditional logit distributions.
Baseline methods. We validate how well our method accelerates the inference speed with lightweight networks by comparing DBN with the target DE (Lakshminarayanan al., 2017) as the oracle ensemble. We also compare our method with three existing methods that are widely used in fast and efficient ensemble distillation or bridge networks: Ensemble Distillation (ED) (Hin- ton et al., 2015), Ensemble Distribution Distillation (EnD²) (Ryabinin et al., 2021), and BN (Yun al., 2023). We use more refined version of END² (Ryabinin et al., 2021) instead of the original EnD² (Malinin et al., 2020) that improved convergence.
Metrics. We measure the computational cost of each model in terms of FLoating point OPerations (FLOPS) and the number of parameters (#Params). FLOPS count the number of additions and multi- plications operations, and it represents a cost during an inference of a model. #Params represents memory usage required for the inference. For the performance, we consider the classification accu- racy (ACC), Negative Log-Likelihood (NLL), Brier Score (BS) (Brier, 1950), Expected Calibration Error (ECE) (Guo et al., 2017). and Deep Ensemble Equivalent (DEE) (Ashukha et al., 2020). BS and ECE measure how much the classification output (confidence) aligned with the true probabil- ity and thereby it implies the reliability of the model output. DEE approximates the number of DE similar to a given model in terms of NLL. More profound formulations of the metrics are described in Appendix A.3.
4.1 CLASSIFICATION PERFORMANCE AND UNCERTAINTY METRICS
The ACC, NLL, BS, ECE, and DEE comparisons with the baselines with respect to FLOPS and #Params on CIFAR-10 10, CIFAR- and TinylmageNet are shown in Table 1. We assume the situ- ation where both BN and DBN can utilize only single source model to make the problem difficult. As we can see the results of CIFAR-10 and C00,R-100, with only a small increase in the compu- tational costs (FLOPS and #Params), DBN achieves almost DE-3 performance, whereas BN struggles to achieve even DE-2 performance with more computational costs than DBN. In TinylmageNet, DBN outperforms DE-3 with less than half of computation costs. On the other hand, the distil- lation methods, ED and EnD2, are competitive with BN but they shows poor uncertainty metrics such with high perfor-
NLL,, BS, and ECE, and interestingly DBN also shows poor ECE scores mance in the other uncertainty metrics. In addition, the actual output results through the diffusion processes illustrated in Figure 2 and the results for the other images are listed in Appendix C.
4.2 PERFORMANCE LOSS COMPUTATIONAL EFFICIENCY
Furthermore, depending on the number of ensemble models that DBN is trained on, the left hand side of Figure 4 demonstrates how much performance loss occurs compared to the DE when the number of target ensemble models increase. Conversely, the right hand side of Figure 4 illustrates how much cost savings DBN can achieve compared to DE, given the same computational cost (FLOPS). The comparison is made from the perspectives of ACC and DEE. In this experiment, maximum of ensemble models are used and we also compare with our competing model, BN. DBN trains one diffusion bridge with three ensembles, while BN learns low-loss curve between 2 ensembles for

---

Published as a conference paper at ICLR 2024
Table 1: Performance CIFAR-10/100, and TinyImageNet. BNNedium is the standard size of BN and BNsmall has reduced channels compared to BNmedium- The parentheses next to each model (e.g. (DE-3)) means the number of DES that each model learns target.
CIFAR-10
Model
FLOPs
#Params (
ACC (?)
NLL (3)
BS (J)
ECB (U)
DEE (1)
RosNet (DE-1)
1.000
11000
91.30
00.10
0.3362 0.0002
0.1409 1 1000011
0.0658 0.003
1,000
+2 BNarall (DE-3)
1.125
x 1.099
91.79 00.06
0,2579 0.0000
0.1198 + 0.000
0.0599
0.0057
1.899
44 RNasall (DR-S)
x 1.245
x 1.283
91.87 = 00.06
0,2580
+.00010
0.1195 +.0.004
0.0624
+.000
1.898
BNaattann (D1-3)
1.411
1.319
91.91 00.00000
0.00011 0,2544
0.1182 1 0.0000
0,0591
0.0006
1.938
(DE-3)
92.98
0,2403
0,1084
1.00013
0.0666
2.363
DaN (DF-S)
1,332
1.426
93.23
0.2247 0.0005
0.1033 +.0005
0.0662
0.000
3,031
ED (DB-3)
11.000
1,000
91.96
00.42
0,3505
0.0014
0.1366 1: 0.0074
0.0674
0.0057
<1
ED (DB-5)
X 1,000
X 1.000
91.86 00.21
0,3577
0.0003
0,1391 1.00001
0.0683
0.0017
<1
EaD (DE-3)
1.000
X 1,000
91.99 00.14
0,3405
0.0009
0,1358 1 1.00026
0.0690
0.0013
<11
EaD² (DE-5)
92.11
13.23
0.3313
+ 0.002)
0.0645
L.077
DE-2
2,000
2,000
92.72
0.2489
0.1125
0.0484
0.0000
2,000
ne-3
3,000
x 3,000
93.06
00.14
0.2252
0.0024
0.1038
0.000
0.0469
+000012
3,000
DE-5
5.000
5.000
93.61 00.11
0.2005 0.0015
0.0951
1: 0.0004
0.0466 0.0009
5,000
CIFAR-100
Model
FLOPS (U)
#Params (.)
ACC (?)
NLL (1)
BS (1)
ECB (L)
DEE (t)
ResNet (DR-1)
72.29
00.36
0.0000
+ 0.0044
0.1526 +.0000
1.000
BNnattm (DE-3)
1.320
74.97
100600
0.0022
0,3500
0.1228
1.642
+1 DBN (DE:3)
1.166
1.213
76.02
100.1
0,9434
0.0061
0.3438
0,0009
0.1352
0.0020
2.461
+2 DaN (DE:S)
1.332
x 1.426
76.82
00.22
0.8998
0.006G
0,3305 +: 0.0018
0.1269 0.0013
3.297
ED (DZ-3)
X 1.000
x 1.000
74.18
00.22
1,2375
0.0125
0.4095 1. 1.00019
0,1819
0.0013
<1
ED (DE-5)
1.000
x 1.000
74.00
00.29
1,2428
0.0174
0,4120 1. 0.0041
0.1840
0.0022
<11
(DE:3)
73.35
01.22
1.3572
0.0079
0,4350 # 0.0013
0.1973
EnD² (DE-5)
1.000
1.000
73.22
1,3597
0.0156
0.4370
0.1980
<1
ne-2
2,000
2.000
74.98
00.42
0,9721
0.0009
0.3505 + 0.0034
0.1259 0.0022
2,000
DR-3
x 3.000
x 3.000
76.04
00
0.9098
0.0019
0.3342
1: 0.0007
0.1233 0.0000
3,000
DE-5
x 5,000
x 5.000
77.03 = 00.08
0.8606
0.0006
0.3216 + 0.0013
0.1234 0.0001
5,000
TinyImageNet
Model
FLOPs
#Params
ACC (?)
NLL
BS (L)
ECE (L)
DEE
ReaNet (D6-1)
1.000
59.28
1,8356 0.0066
0.5426 1.00002
0.2024 0.0025
1.000
BNustimn (DE-3)
1.359
1.412
64.07
1,5066 0.0021
0.4745 1 1.0000
0.1267 0.0137
4.102
+1 DaN (DE:3)
1,209
x 1.149
64.60 = 00.06
1.5299
0.0022
0,4718 : 0.0013
0.1836 0.0021
3.444
+2 DaN (DE-S)
x 1.418
x 1.298
65.34 = 00.10
1,4890
0.0045
0.4616 + 0.001
0.1803 0.0013
4,701
ED (DE-3)
1.000
x 1.000
60.81 = 00.26
1.8312
0.0129
0,5443 1 0.0002
0.2126
0.0021
<1
(DE-3)
60.71
00.31
1,9991
0.0165
# 0.0026
0.2279
<1
DE-2
2.000
2.000
62.47
00.36
1.6264
0.0110
0.4942
0.1834 0.0013
2,000
ne-3
3,000
3,000
63.78 00.23
1,5538 0.0055
0.4788 0.0015
0.1833 0.0021
3.000
one bridge. For more than four ensembles, DBN conducts ensemble inference aggregating two or more diffusion bridges as BN does. We use the standard size of BN in "ResNet + 2 BNs" which has 1.411 relative FLOPs for a single network compared to "ResNet + DBNS" which has 1.166 relative FLOPs. As shown in Figure 4, DBN (left) achieves significant ensemble gains even when the number target ensemble increases, whereas BN (left) saturates at ACC 92.0% and DEE 2%. Moreover, DBN (right) shows rapid ensemble inference compared to the target ensemble DE (right) and even faster than BN (right) to get the same ensemble performance.
4.3 ENSEMBLE CAPACITY OF A SINGLE DBN
Figure demonstrates the ensemble capacity of how much a single DBN model learns and distills knowledge from multiple teacher (ensemble) models in terms of ACC and DEE in the CIFAR- 10 dataset. Existing ensemble distillation models have limited performance in learning likelihood and are saturated in terms of ACC at learning at most 3 number of target ensembles. On the other hand, BN assumes that a single model can cover at most two ensemble modes, hence the maximum gain on ACC and DEE is capped at two. Compared to these competing methods, our DBN method succeeds in learning slightly less than three ensembles only with a single lightweight model and the source model.
8

---

Published as a conference paper at ICLR 2024
94.0
94.0
83.5
8.00
93.5
6.00
6.00
ACO
02.5
DEE
DER
4.00
4.00
82.0
92.0
81.5
91.5
1.5
2
2.5
1.5
2
2.5
Ensembles
Ensembles
FLOPs
-e-
DE-n
+ ResNet 28 BNs
ReeNet + DINN
Figure 4: From left (19) right (4 Number of target ensembles accuracy (1)) DEE (2"d), and relative FLOPs vs. accuracy (3t) or DEE (4th in CIFAR-10.
5 RELATED WORK
Fast Ensembling Methods for Neural Network Prediction. Many works have suggested to reduce the computational cost of the typical ensemble methods that are multiplied proportional to the num- ber of the ensemble networks. To provide output diversity with lightweight computation, sharing either weights or latent features of neural networks enabled to compress redundancy throughout the ensemble members and add minor differences between them (Wen et al., 2019; Dusenberry et al., 2020; Lee et al., 2015; Siqueira et al., 2018; Antorán et al., 2020; Havasi et al., 2021). I). This can also be interpreted the knowledge distillation from the ensemble of networks to the single network; (Hinton et al., 2015), Malinin et al. (2020); Ryabinin et al. (2021); I); Penso et al. (2022) showed that distilling deep ensemble single network helps retaining information of the ensemble distribution without heavy computational burden.
Restoration with Conditional Diffassio Models. Our model is the line of the reconstruction prob- lem from some degraded measurement, if we consider our method as reconstructing the ensemble distribution from the degraded function output from a single model. This line of work starts with the conditional diffusion model (Saharia et al., 2022), that refines the image given some conditional features. Conditional diffusion models have achieved success in various problems such as time se- ries imputation (Tashiro et al., 2021), deblurring (Whang et al., 2022), and super-resolution (Saharia et al., 2023). As a generalized perspective, inverse problems dealt with diffiusion (Song et al., 2022) aims to restore the underlying clean signal from the noisy measurement. Wang et al. (2023) delves into the null space of the image and utilize the valid space that should recovered, and achieved zero-shot restoration using diffusion models.
CONCLUSION
We have proposed a novel approach to approximate the performance of Deep Ensembles with re- duced computational cost. To achieve this, constructed Conditional Diffusion Bridge that con- nects the logit distribution between one of the ensemble models and the target ensemble. We ap- proximated the output distribution of the target ensemble using the features and logits obtained from the source model. The computations involved in this process consist of a single forward pass of the source model and the diffusion process with lightweight score network. Additionally, during the training process, distilling the multiple diffusion steps into one step accelerates the inference speed while retaining the performance of three deep ensemble models. We evaluated this method on three widely used datasets and achieved superior performance compared to the baselines. Notably, we demonstrated significantly faster computations compared to competitive models such as Bridge Network while achieving similar performance. However, there are still some limitations to consider. First, a single Diffusion Bridge still has limitations in terms of the number of ensembles it can learn. If more ensembles required, additional diffusion bridges must be used. Secondly, the use of multiple diffusion bridges leads to a proportional training time because diffusion models demand a long training time due to their multiple diffusion steps.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work partly supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-0007), Artificial Intelligence Graduate School Program (KAIST), No. No.22022-0-0711: Meta-learning Applicable to Real-world Problems, No.2022-0-00184. Development and Study of Al Technologies to Inexpen- sively Conform to Evolving Policy on Ethics), and KAIST-NAVER Hypercreative Al Center. Our research is supported with Cloud TPUs from Google's TPU Research Cloud (TRC).
Ethics statement. This paper does not include any ethical issues. This paper presents a fast en- semble inference algorithm of mainly image classifications which does not cause ethical issues.
Reproducibility statement. We described experimental details in Appendix D and Ap- pendix A.3 which includes information about datasets, architectures, and hyperparameters used.
REFERENCES
Javier Antorán, James Urquhart Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural networks. Advances Neural Information Processing Systems (NeurIPS 2020),
2020. 9
Arsenii Ashukha. Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-din uncertainty estimation and ensembling in deep learning. In International Conference on Learning
Representations (ICLR), 2020. 7, 14
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 2. 3.5
Leo Breiman. Bagging predictors. Machine Learning, 1996.
Glenn W. Brier. Verification Forecasts Expressed in Terms of Probability. Monthly Weather Review, 1950.
Tianrong Chen, Guan-Horng Liu, and Evangelos A. Theodorou. Likelihood training of schrödinger bridge using forward-backwart sdes theory. In International Conference Learning Represen- tations (ICLR), 2022. 3
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In Proceedings of The 37th International Conference on Machine Learning (ICML 2020), 2020. 9
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson Loss surfaces, mode connectivity, and fast ensembling DNNs, In Advances in Neural Informa- tion Processing Systems 31 (NeurIPS 2018), 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of The 34th International Conference on Machine Learning (ICML 2017), 7
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations (ICLR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016. 6
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019. 17
Geoffrey E. Hinton, Oriol Vinyals, and Jeffiry Dean. Distilling the knowledge in neural network.
In Advances in Neural Information Processing Systems 27 (NIPS 2014), 2015. 7,9
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020. 3,5
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 15
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. 13
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. 2,
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruy Batra. Why m heads are better than one: Training diverse ensemble of deep networks. arXi::551.00611,, 2015. 9
Fei-Fei Li, Andrej Karpathy, and Justin Johnson. Tiny ImageNet. https:tt//////tt https://wwwwkaagt www.w.aggl..com com/ c/ c/tiny- imagenet, 2017. [Online; accessed 19-May-2022j. 6, 13
Guan-Homg Liu, Arash Vahdat, De-An Huang, Evangelos A. Theodorou, Weili Nie, and Anima Anandkumar. 12sb: Image-to-image schrödinger bridge. In Proceedings of The 40th International Conference Machine Learning (ICML 2023), 2023. 3,5
Andrey Malinin, Bruno Mlodozeniec, and Mark J. F. Gales. Ensemble distribution distillation. In International Conference on Learning Representations (ICLR), 2020. 1,7,9
Giung Nam, Jongmin Yoon, Yoonho Lee, and Juho Lee. Diversity matters when learning from ensembles. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 1
Coby Penso, Idan Achituve, and Ethan Fetaya. Functional ensemble distillation. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. 1,9
Max Ryabinin, Andrey Malinin, and Mark Gales. Scaling ensemble distribution distillation many classes with proxy targets. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 7,9
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: image-to-imagee diffusion models. In SIGGRAPH (Con- ference Paper Track), 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. Pattern Anal. Mach. Intell., 2023. 9
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations (ICLR), 2022. 5, 15
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 6, 13, 14
E. Schrödinger. Sur la théorie relativiste de l'électron et I'interprétation de la mécanique quantique. Annales de l'institut Henri Poincaré, 1932. 3
Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch de- pendence in the training of deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 13

---

Published as a conference paper at ICLR 2024
Henrique Siqueira, Pablo Barros, Sven Magg, and Stefan Wermter. An ensemble with shared rep- resentations based on convolutional networks for continually learning facial expressions. In 2018 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2018. 9
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations (ICLR), 2022.
Yusuke Tashrro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 9
Linh Tran, Bastiaan S. Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V. Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation, 2021.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In International Conference on Learning Representations (ICLR), 2023.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. In International Conference Learning Representations (ICLR), 2019. 1,9 9
Jay Whang, Mauricio Delbracco, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 9
EungGu Yun, Hyungi Lee, Giung Nam, and Juho Lee. Traversing between modes in function space for fast ensembling. In Proceedings The 40th International Conference on Machine Learning (ICMI. 2023), 2023. 2, 4, 6, 7, 13, 15
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL DETAILS
We describe the overall details of our experiment below.
A.1 DATASETS
We employ CIFAR-10/100 (Krizhevsky al., 2009), and TinyImageNet (Li al., 2017) datasets for our study. Our data augmentation strategy involves randomly cropping images by 32 pixels with additional 4-pixel padding, well applying random horizontal flipping. Furthermore, we normalize input images by subtracting per-channel means and dividing them by per-channel standard deviations.
A.2 TARGET MODEL ARCHITECTURES
In our investigation, we implement comparable ResNet block configurations. The overall network architectures are consistent with one of our baseline, BN (Yun et al., 2023), to compare reliable condition with minor difference in TinyImageNet and ImageNet.
Classifier Architectures.
CIFAR-10.
ResNet-32 2, characterized by 15 basic blocks distributed as 5, and total of
32 layers with the Filter Response normalization (FRN) (Singh Krishnan, 2020) and the Swish activation layer. This model incorporates widen factor of and operates with in-planes set at 16.
CIFAR-100. ResNet-32 x 4, which closely resembles the CIFAR-10 network with a widen factor of 4 with FRN and the Swish activation.
ResNet-34, which encompasses 16 basic blocks organized as (3, 4, 6, 3) and 34 layers in total with FRN and Swish. The in-planes parameter for this model established 64, The only difference with the target model of Yun et al. (2023) is that we use bias in the convolutional layers but they don't.
TinyImageNe
ImageNet64. ResNet-34, which encompasses 16 basic blocks organized as (3, 4. 6, 3) and 34 layers in total with FRN and Swish. The in-planes parameter for this model is established 64 as in TinylmageNet. However, we don't 't use bias in every convolutional layers in this task.
InvertedResiduals
Convix1
h
Conv3x3
Lincar
t
Z
MLP
MLPs
Figure 5: Score network architecture.
Score Network Architectures. The score networks consist of three embedding networks for h Z,, and t, some inverted residual blocks (Sandler et al., 2018), and the output layer. The embedding network for Z, consists of LayerNorm layer followed by the average pooling layer, and returns the embedding output by forward computation to a lightweight MLP (width 2 - 2 - 1 with the ReLU6 activation layer), and concatenate the embedding outputs to the embeddings of hT The embedding network for i firstly use the sinusoidal time-step embeddings with the maximum period 10,000 and then feed-forward to a light weight MLP (width d/4 + d/2 - with the Swish activation, where d is the dimension of the logits and is the input channels of each inverted residual blocks.) The specifications the inverted residuals and the size of the output layers in the score networks vary across the tasks in Table 2.
13

---

Published as a conference paper at ICLR 2024
Table 2: Specification of the inverted residual blocks and output layers in the score network across the tasks, is the expansion factor (Sandler et al., 2018). c is the output channels. n is the number of layers. s is the stride applied in the first layer of each inverted residual.
CIFARIO
CIFAR100
TinylngoNNe
InageNet6t
Operatoo
Openitor
Openator
Operator
InvertedResicual
InvertedResidual
InvertedResdiaal
ImeeteRReshuull
64
3
InvertedResicual
InvettedResddall
2
InverteReesdual
2
InvenedRecidall
96
3
InvertedResicual
3
LnvestedResidual
3
InvenedResdiual
128
3
InveneRReidual
160
3
InvertodResicual
InvetteReesidaal
InveteRResiiual
4
InveneResidull
320
InvertedResicual
3
128
2
InvestedResidull
3
2
InverteReesdual
256
3
ImenedResdual
640
1
Cony Ixl
Coev Ixl
Conv Ixl
Com 1xl
1280
AvgPool
AvgPool
AvgPool
AvgPoel
Lineur
Lineur
Linear
A.3 METRICS
We introduce the metries used in experiments.
Accuracy
(z))))
Tly arg max (6) (x:)]
(14)
h
where I is the indicator function.
Negative logeliiklhooo (NLL)
E log (2) (x))
(15)
Brier score (BS)
1 (m)))) p(x))
(16)
where is one-hot-encoded label
Expected calibration error (ECH)
Nus
ECE(D, Nbin)
(17)
n1
where Noin is the quantity of bins, ny is the number of instances within the 8th bin, and 6, is the calibration discrepancy associated with the 5th bin. To elaborate, the 5th bin encompasses predictions characterized by the highest confidence levels falling within the interval [[b 1)/ K, b/ K), and the calibration error is defined as the disparity between accuracy and the mean confidence values. We maintains Nbin 15 throughout the paper.
Deep ensemble equivalent score (DER)
First introduced in Ashukha et al. (2020), this metric assumes that the NLL of the DE-k model decreases monotonely by h; and obtain how equivalent the objective model to how many ensemble of the baseline model, as
NLL(f) - NLL(F))
DEE( = s + NLL(fo+1) - NLL(f.)
(18)
arg max e N: NLL(f)) 2 NLL(f))
where ft the DE-i model. In our paper, DEE is linearly extrapolated below 1 NLL(f) > NLL(f)).
B HYPERPARAMETER SETTINGS
We list common hyperparameters for training DBNS in every dataset as follows:
14

---

Published as a conference paper at ICLR 2024
Optimizer. For training DBN model, the ADAM (Kingma & Ba, 2015) optimizer with zero weight decay and apply consine-decay decay as learning-rate scheduling. For training the teacher ensemble model, we followed the BN paper by using the SGD optimizer with weight decay, as followed in Table
Regularization. We use Exponential Moving Average (EMA) with 0.99995 decay factor and the mixup augmentation with = 0.4 except for ImageNet, following (Yun et al., 2023).
Diffusion Model. Our DBNS follow the training policy of discrete-time conditional diffusion model with uniform timesteps. After training the baseline DBNs, we distill them to one step, follow- ing Salimans & Ho (2022).
Feature vector h1. We exploit the output of the first residual block of the teacher ResNet.
Temperature Distribution Ptemp- The probability density of the temperature distribution Ptemp follows the Beta distribution as follows: 2(1 0.2a), a Beta( 1,,5)
The other hyperparameters are altered across the datasets and they are shown in Table 3. In addi- tion, we also report the full list of hyperparameters used in training the baseline ensemble teacher networks in Table 4.
Table The hyperparameter settings used learn the DBN network.
Dataset
CIFARIO
CIFAR100
TinyImageNet
ImageNet64
#Params of Teacher
1,860,986
7,460,708
21,798,504
21,798,504
#Params of Score
395,543
1,547,617
3,186, 117
8,278,222
Batch Size
128
128
128
256
Epochs
800
800
250
250
Epochs (distill)
50
50
50
50
Learning Rate
0.00025
0.00025
0.0005
0.0005
Learning Rate (distill)
0.000025
0.000025
0.00005
0.00005
Mixup &
0.4
0.0
8.
0.0001, t E [0, 1]
0.0001, e [0,
0.001, t e [o, 1]
0.005, [o. 1]
Table 4: The hyperparameter settings used to learn the baseline ensemble models.
Dataset
CIFAR10
CIFAR100
TinylmageNet
ImageNet64
#Params
1,860,986
7,460,708
21,798,504
21,798,504
Batch Size
256
128
128
256
Epochs
200
200
200
250
Learning Rate
0.1
0.1
0.1
0.1
Cosine Decay Scheduling
Yes
Yes
Yes
Yes
Weight Decay
0.001
0.0005
0.0005
0.0001
Warmup Steps (Linear)
0
5
5
5
Initial Learning Rate
0.1
0.001
0.001
0.001
15

---

Published as a conference paper at ICLR 2024
C
GENERATION QUALITY OF DIFFUSION BRIDGE
Images
Diffasion Stepo
Target
Confidences
t-02
t:04
Confidencess
1-0.8
t=1.0
piane anto
aind
day
fryy
bors
etip
irick
plane
anto
sind
deer
day
fray
alip
trick
piane
anto
sind
deey
day
fryg
horrs alii trock
Images
Source
Diftusion
Target
Confitences
plase
t=02
t=
Confdences 8
ts0.8
bird
I-10
plane
auto
Hird
cut
deege
day
Eroy
home
stip
trock
plane
anto
Hird
cat
day
fray
ahip
treck
plane
anto
Jird
cue
day
fryg
alip
trick
Source
Diftusion
Target
Confodeeccs
truck
tm
Confidences
t-10
plase
woro
Her
day
froy
home
ahip
truck
plane
aoto Jard
desge
day
froy
home
ahip
trock
plane
ento
Jar
on
day
free
home alop
track
Images
Source
Diffusion Steps
Target
Confideecces
L-02
Confidences
ts1.0
3
nioss
anto
Hard
day
froy
horse abip
truck
wito hirt car
day
froy
horte
ahi
trock
plane
anto
lire
ragt
day
frog
horte all
track
Images
Source
Diffusion Steps
Target
Confidences 3
calt
L-0.2
Confoences
1-1 1.0
Oat
R
plone
anto
out
dowe
hove
ship
brank
plasss
autoo
hird
oud
dear
hovee
atip
planse
ando
hidd
cnd
dearr
lovee
abip
Figure 6: Confidences from the source model (first column), from the ensemble model (third col- umn), and from the diffusion bridge (middle column) for the given images the CIFAR- 10 dataset. The middle column illustrates a transition of the confidence from the source model to the target during the diffusion process.
16

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EXPERIMENTS
D.1 RESULTS ON IMAGENET
Because the limit of the training budget, conducted this on the downscaled datasets and ob- tained the following result. We also attached the results of BN in the baseline ImageNet and com- pared the DBN and BN with the baseline DEs independently. Then we observed that the perfor- mance of our DBN method surpasses DE-2 in terms of DEE and accuracy (ACC) and it is close to that DE-3 with only one diffusion bridge, while BN achieves the same level performance with two bridges.
Table 5: Performance on ImageNet dataset. cNLL stands for NLL calibrated with optimal tem- perature.
Model
FLOPs (J)
#Params (L)
ACC (1)
cNLL (4)
DEE (L)
ResNet (DE-1)
x 1.000
x 1.000
66.47
1.3960
1.000
+1 DBN (DE-3)
X 1.244
x 1.380
69.79
1.2140
2.702
DE-2
x 2,000
x 2,000
69.56
1.2542
2,000
DE-3
x 3.000
x 3.000
70.82
1.1969
3.000
D.2 OUT-OF-DISTRIBUTION PERFORMANCE
We evaluated our method on the widely used CIFAR-10-C (Hendrycks Dietterich, 2019), which the dataset of common corruptions the CIFAR-10 dataset. and compared it to the existing BN method as in Table 1. Achieving superior performance in the CIFAR- 10-C dataset implies that the model robust with respect the out-o--diitribution distribution, tilted by variety of distortions and corruptions the in-distribution dataset (in this case, CIFAR-10).
Table Test results on CIFAR-10-C. cNIL stands for NLL calibrated with an optimal temperature.
Model
ACC (t)
NLL (L)
cNLL (L)
ResNet (DE-1)
86.95
0,5158
0.4477
+2 BNmedium (DE-3)
87.80
0.3740
0.3749
+1 DBN (DE-3)
88.91
0.3759
0.3682
DE-2
88.85
0.3860
0.3693
DE-3
89.47
0.3459
0.3410
17

---

Published as a conference paper at ICLR 2024
E DEPTHWISE SEPARABLE CONVOLUTION
Depthwise separable convolution is method to modify conventional convolutional layer, which in- clude a convolution operation between every input channel and the convolution filters. Precisely, let the number of the input feature to have H x W spatial size with Ci channels, and the convolutional layer with w receptive field with Cou filters. For simplicity, we take full padding and do not allow strides,and and biases. Then the number of parameters and the FLOPS of the convolutional layer is Cn Con and H h X W
Cn X Cout. respectively.
Compared to standard convolution, the depthwise separable convolution consists of two parts: sep- arable convolution and point-wise convolution. First, separable convolution is the convolution with h x receptive fields and Cj filters, taking the number of groups same as the input feature (Cin)- Then, each filter convolved by the corresponding filters, followed by x features. The number of parameters and the FLOPS of the separable convolution is h x Cin and H h x W x w x Cin, respectively. Then the second part of the depthwise separable convolution the point-wise convolution, which is 1 sizes. The number of parameters and the FLOPS of the point-wise convolution Ci x Cou and H W x C. Cout, respectively. We do not consider biases for evaluating the complexity of the convolution layers.
Cin intermediate
convolution with Cin input and Cour output filter
Table 7: The complexity measures the standard convolution and depthwise separable convolution.
Model
# Parameters
FLOPs
Standard
w x Cj Cout
x h W x w On X Cout
Depthwise separable
(h x ur + Cout) x Cin 1 1 Cout hw
(h x w + Cour
x H x W x Ci
1
1
D-S/ / Standard
+
Cout
hw
18


Reviews:
[{"review_id": "fOxugeKbNP", "reviewer": "Reviewer_unod", "summary": "The paper proposes to use Diffusion Schrodinger Bridge (DSB) as a way of bridging two arbitrary distributions, namely that of the output activations space of deep models, thereby achieving more efficient ensemble. In contrast to the previous Bridge Network approaches, the DSB approach imposes no (direct) restriction on the low-loss subspace of the parameter manifold, but instead learn a diffusion that solves the SB problem to diffuse from any ensemble member to the \"target\" ensemble model. The authors claim that the empirical results demonstrate superior performance and FLOPs requirement of the Diffusion Bridge Network (DBN).", "questions": "1. The introduction of temperature does define a distribution, but essentially at a cost of compromising the performance of the original ensemble member (for example, if you look at the NLL loss of the logits after applying temperature, it will be worse). Is this really a reasonable design? If you look for bridging on logit space with another, wouldn't it make more sense to consider the logit space as is (e.g., get a distribution via input perturbation, etc.)?\n\n2. Why DSB rather than a conditional diffusion from Gaussian? In $\\textsf{I}^2\\textsf{SB}$ the motivation was clearer as they hoped to leverage the structural prior of the image. Empirically, how big would the difference be?\n\n3. The paper has a bunch of analysis on the ensemble approach, but little (almost none) on the diffusion portion. For example, how much training (data and speed) is needed for the DSB training? Why diffusion instead of some simpler approaches like normalizing flows, and bringing all ensemble members to a (same) tractable distribution?\n\n\n--------------\n\nPost-rebuttal edit: I'm raising my score slightly from 3 to 5, in light of the amendments the authors make during the rebuttal. However, I still believe this paper is not completely ready for publication at the venue. See comment below for details. \n\n4. About the baseline: the Bridge Network paper seems to have better numbers on TinyImageNet than what the authors reported in this paper (e.g., 65.82% accuracy for 3-BN-sm and the FLOPs seems to be much smaller (1.15x). The BN paper also evaluated on ImageNet, where in my experience the accuracy is **much less volatile**. If, as authors claim that the BN approach has more training cost (at the end of Sec. 2.1), perhaps the DSB approach should be compared with BN on that turf as well.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The paper generally tackles the important issue of accelerating/simplifying the costly deep ensemble process.\n- The overall idea is clearly presented, and it's interesting to motivate a generative solution on the ensemble problem.\n- Relatively comprehensive comparison to baseline approaches.", "weaknesses": "- The baseline performances shown in the experiment section does not completely agree with the prior works. The scale of the experiments were somewhat small.\n- Lack of analysis on the diffusion part of the design.\n- The methodology somewhat concerns me (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZoSfZtLEC", "reviewer": "Reviewer_bFwN", "summary": "This work proposes to lower the inference costs associated with running a full ensemble of neural networks. \nOne problem with other fast ensembling approaches is that they require mode connectivity between the ensemble members, but finding such a low-loss path or subspace through the weight space is difficult. other methods either assume the location of these parameters directly, or else learning-based approaches learn a subspace between just two modes (ensemble members). This method uses a diffusion model to transform logits from a single ensemble member, into a sample from the logit distribution of the ensemble itself.", "questions": "1. Given a path from model z_1 to the ensemble model z, can anything be said about the intermediate logit distributions in terms of their loss? \n2. I'm not clear how useful the source distribution construction is when its just annealing T. This preserves relative ordering over classes, which is not adding much additional information about the \"distribution\" of z_1. Are we in some sense integrating over paths from the set of temperature-augmented z_1 -> z?  Is that a non-trivial path? \n3. In algorithm 1: is a single temperature value drawn for each member, for all data points in the minibatch?\n4. Does performance increase with multiple diffusion steps?\n4a. What's the \"cost\" of each diffusion step in term of % of DE inference speed. Can DBN admit more than diffusion step at inference and still be faster than DE?  \n5. Can the authors comment on the calibration error increase with this method?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this is a strong approach, it alleviates inference costs in a straightforward way that makes use of novel techniques. \n\n* Interesting approach to speed up ensemble inference\n* Frames ensemble knowledge distillation as a diffusion problem\n* Linear scaling of bridges in the worst case, where bridges are smaller than ensemble members. \n* Good results on accuracy across benchmarks. DEE is very good.", "weaknesses": "* DBN models are notably less calibrated  than ensembles or bridge networks across tasks.\n* No evaluation on in / out of domain uncertainty / calibration. IMO this is a huge reason why ensembling is done in the first place. \n* Evaluation is with relatively small models\n\nComments but not weaknesses:\n* Section 4.1 \"?? of ??\" still in text. \n* I think parameter-count/memory savings is useful here and could be reported.\n\n\n---------------------------------------------------------------------------------------------\nPost Rebuttal Comments\n---------------------------------------------------------------------------------------------\nAfter reading the authors responses and the other reviews I will not be updating my score. \nDetailed comment below. \n\nI think the authors addressed most of my concerns in their comments to my and other reviews. \n\n* W2 and W3 have been mostly addressed with additional experiments\n* Q1 was answered empirically, though I would hope any accompanying text came with intuition about why/if loss should decrease monotonically with increased t\n* Q2 was brought up by most other reviewers, and the common response is fine, if not somewhat \"hand-wavey\". Nonetheless, its not a glaring concern and improvement could be a component of future work. \n* Q3/4/4a/5 were answered", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ik3QeKELJ", "reviewer": "Reviewer_Qbom", "summary": "This article presents a framework to approximate Deep Ensembles (DE) called Diffusion Bridge Network (DBN). Following the theory of the Schrödinger bridge, a conditional diffusion bridge is learned between single models' logits and the target ensemble's logits. The score network learned is distilled during the training process and multiple score networks can be ensembled when the number of models increases.\n\n[Edit: as indicated in my response, I am increasing my rating from 5 to 6 considering the strong improvements in writing for both the theory and the implementation, despite limited results on TinyImageNet after correction of the authors mistake]", "questions": "* Is the DSB not able to learn if there is no temperature annealing?\n\n* Why choose in particular ResNet-32x2 and x4 rather than a standard ResNet-32?\n\n* Do the authors have an idea why DBN shows poor EC scores compared to the other metrics?\n\n**Various mistakes or remarks:**\n* Abstract: \"theory of [the] Schrödinger bridge\"\n* Introduction: \"incurrs an extra\"\n* 2.2 \"has rarely been demonstrated its practicality\" \"demonstrated for real-world image dataset\" \"Gaussain\" \"by a certain PDEs\" \n* 3.2 \"Based on the formulation\" ??\n* 3.3 \"super performance\". No use for the (NFE) abbreviation if it's never used after.\n* 3.4 $\\Phi$ or $\\Phi'$? \n* 4. \"because Ryabinin et al. (2021) refined poor convergence of\" \"more profound formulations\"\n* 4.1 \"results on CIFAR-10 is shown in ?? of ??\"\n* Conclusion \"Additionally,  [...] of 3 DE models\", nominal sentence.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The main idea of the article is well-motivated, improving on Bridge Networks with the strong Schrödinger bridge to learn the ensemble distribution. \n\nThe results obtained show that the conditional diffusion bridge can be learned, leading to efficient deep ensemble estimation, with a strong Deep Ensemble Equivalent score. However with limited insight on the networks used and their training, it is hard to conclude and compare with Bridge Networks.", "weaknesses": "**Writing** This article's writing is very hard to follow. There are many spelling or grammar mistakes (see Questions). Ideas are presented poorly or too quickly, leaving the reader confused. The theory, which originates from Liu et al. 2023 is presented very quickly and unclearly, with some mistakes. ($f'$?, no definitions of $W_t$, $\\bar W_t$, $\\beta_t$, no dimensions, Schrödinger systems are not presented, it's never clarified that $f=0$, although $f$ was never even defined. It's not clear how equation (8) is obtained.\n\n**Training** Many important points are missing, in particular, the architecture of the score network is never given anywhere in the article. Contrary to what the authors claim, the appendix does not give any hyperparameter settings: the training details of both the ensemble networks and the DBN are left unclear (Optimizer, scheduler, number of epochs/steps, distillation schedule...) This is not a detail, in particular for the DBN. If its training is particularly expensive/slow, it is an important drawback despite the speedup it provides during inference. Are Bridge Networks also using distillation, and is their training comparable to DBN? Without this information, it is hard to compare the results obtained.\n\n**Number of ensembles** I find unclear the number of ensembles that a score network is trained on. Section 4 indicates that a score network is trained on 3 ensembles, with no clarification on why this number is chosen (presumably Section 4.3). Why also not simply increase the size of the DBN network? For more than 3 ensembles, 2 or more DBNs are trained. In this case, are the DBN trained on all of the ensembles at the same time? Are they limited to a given subset of 3 ensembles, as indicated previously? \nThis becomes even more confusing in Section 4.2, where the number of ensembles increases to 9, and the number of DBNs is left unclear (\"two or more\"), as well as for BNs. \n\nDespite promising results, this article needs a serious rewrite before I can accept it for now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xIcGPjz22Q", "reviewer": "Reviewer_jMzV", "summary": "In this manuscript, the authors endeavor to address the limitations of existing bridge networks by introducing a novel \"diffusion bridge networks\" framework, inspired by the celebrated Schr\\\"odinger Bridge problem. Specifically, the authors commence by elucidating the shortcomings inherent in constructing bridges within the feature space represented by bridge network, and subsequently propose their diffusion bridge as a solution to generalize from a single network output to an ensemble output. In order to empirically substantiate the efficacy of their method, a variety of experiments are executed across the CIFAR-10, CIFAR-100, and TinyImageNet datasets, utilizing ResNet-$32\\times2$, ResNet-$32\\times4$, and ResNet-$34$ as backbone architectures. Overall, the paper offers an intuitive approach to the problem at hand, and I will delineate its strengths and weaknesses in the subsequent sections.", "questions": "My questions are listed in weakness, please refer to weakness.\n\n----\nPost Rebuttal Comments:  \nThank you for the comprehensive response to my queries and for conducting additional experiments. Having thoroughly reviewed your revised manuscript and the expanded experimental evidence, I am inclined to revise my evaluation. Accordingly, I am considering raising my score from a 6 to an 8, reflecting the improvements and clarifications you have made.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Problem Reformulation** In the domain of problem formulation, the authors ingeniously recast the challenge of deep ensembles as an optimal transport problem, subsequently addressing it through a dynamical optimal transport methodology. This novel perspective not only refines the research question but also paves the way for utilizing advanced mathematical tools to provide robust solutions.\n\n3. **Ample Experiments** With regard to empirical validation, the authors carry out an extensive array of experiments on multiple datasets and furnish an in-depth analysis of the experimental outcomes. This comprehensive experimental setup serves to fortify the credibility of their proposed method and provides valuable insights into its performance characteristics.", "weaknesses": "1. **Other Transport Models** To the best of my understanding, related works such as \"Rectified Flow\" [1] and \"Flow Matching\" [2] are capable of accomplishing similar functionalities. Additionally, when compared to the Schrödinger bridge approach, these models operate through a deterministic process, which could potentially benefit from lower variance properties. The omission of such deterministic flow models from the discussion may limit the generalizability and applicability of the method proposed in the current manuscript.\n\n0. **Inference Algorithm** In the manuscript, it appears that the authors have not provided details regarding the inference algorithm for the proposed DBN during the model inference stage. This omission leaves a critical gap in the paper, as understanding the inference mechanism is essential for a comprehensive evaluation of the proposed method.\n\n2. **Annealing of Temperature** In Section 3.2, the authors discuss the concept of temperature distribution, yet the experimental section lacks elaboration on how the temperature distribution is selected and what principles govern temperature annealing. This absence of information creates a gap in the methodological clarity and poses questions regarding the thoroughness of the experimental design. Furthermore, from a scholarly perspective, treating the distribution as a Gumbel-Softmax distribution [3] could raise additional questions. Specifically, one might inquire whether the training variance of the Diffusion Bridge Networks (DBN) is influenced by the temperature parameter. Addressing such intricate relationships between the temperature and training variance would enhance the paper's academic rigor and contextual relevance.\n\n3. **LaTeX Compile** In Section 4.1, line 3, the manuscript contains two instances of \"??\", which clearly indicate placeholders or unresolved references. The authors should rectify this issue to enhance the document's professionalism and completeness prior to submitting the finalized manuscript. The presence of such markers detracts from the paper's overall quality and could create potential ambiguities for the reader.\n----\nReferences:  \n1. Liu X et al. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, ICLR 2023  \n2. Lipman Y et al. Flow Matching for Generative Modeling, ICLR 2023  \n3. Jang E,  et al. Categorical Reparameterization with Gumbel-Softmax, ICLR 2017", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oM50LfBHEl", "reviewer": "Reviewer_3R6G", "summary": "The paper introduces a diffusion-based approach for enhancing Deep Ensemble (DE) performance. In particular, it suggests the direct transfer of samples from a chosen source model to the ensemble, utilizing the Image-to-Image Schrödinger bridge (I2SB). This proposed method seems to perform well on the conventional benchmark for ensembling tasks.", "questions": "- I believe I2SB remains applicable without the use of temperature annealing ($T$). Can the authors provide further clarification on the role of $T$ and explain how it contributes to differences in performance?\n\n- It's unclear to me how multiple DBNs are combined in Sec 3.5. For instance, if there are 5 models (M=5) and each DBN is trained with 3 ensembles (as suggested in Sec 4), what is the appropriate value for $L$ and what're the ensembles for each DBN?\n\n- I2SB has demonstrated decent performance with NFE=1. Can the authors present an ablation study or provide justification for why additional distillation is necessary for achieving optimal performance?\n\n- The font size in Alg 1 seems to be smaller than the one in the main paper. Is this intended?\n\nI am open to reevaluating the score, with the condition that the authors thoroughly address my questions and make improvements to the presentation, particularly addressing the weaknesses I've highlighted.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents a compelling application of I2SB in the context of deep ensembles.\n\n- The proposed method delivers impressive performance across various benchmarks, by achieving minimal performance degradation when compared to the ground-truth oracle, all while using significantly fewer FLOPs, making it an exciting development.", "weaknesses": "- Equation (4) contains an error: the forward and backward drifts should not be the same, and I2SB specifically provides a tractable solution only for the backward drift. Additionally, the definition of $f_t^\\prime$ is missing. I suggest presenting the analytical backward drift directly, similar to what is provided for the target in (9), and ignore the forward diffusion.\n\n- The variables \"y\" in (11) and (12) lack proper definitions. While their meanings can be inferred from the context, it would be better to provide clear definitions.\n\n- The labels in Figure 2 are too small and may need to be enlarged for better readability.\n\n- There appears to be a typo in (11). I think it should be \"Softmax(Z_0)\" instead of the current notation.\n\n- Missing references in Sec 4.1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)