ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training
Authors: Yue Zhao
Yantao Shen
Yuanjun Xiong
Shuo Yang
Wei Xia
Zhuowen Tu
Bernt Schiele
Stefano Soatto
Venue: Submitted to ICLR 2023 (2023)
Keywords: positive-congruent training
negative flip
ensemble learning

Abstract: Negative flips are errors introduced in a classification system when a legacy model is updated. Existing methods to reduce the negative flip rate (NFR) either do so at the expense of overall accuracy by forcing a new model to imitate the old models, or use ensembles, which multiply inference cost prohibitively. We analyze the role of ensembles in reducing NFR and observe that they remove negative flips that are typically not close to the decision boundary, but often exhibit large deviations in the distance among their logits. Based on the observation, we present a method, called Ensemble Logit Difference Inhibition ELODI, to train a classification system that achieves paragon performance in both error rate and NFR, at the inference cost of a single model. The method distills a homogeneous ensemble to a single student model which is used to update the classification system. ELODI also introduces a generalized distillation objective, Logit Difference Inhibition (LDI), which penalizes changes in the logits between the reference ensemble and the student single model.
On multiple image classification benchmarks, model updates with ELODI demonstrate superior accuracy retention and NFR reduction. 

Under review as a conference paper at ICLR 2023
ELODI: ENSEMBLE LOGIT DIFFERENCE INHIBITION FOR POSITIVE-CONGRUENY TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Negative flips errors introduced in classification system when legacy model is updated. Existing methods to reduce the negative flip rate (NFR) either do so at the expense of overall accuracy by foreing a new model imitate the old models, or use ensembles, which multiply inference cost prohibitively. We analyze the role of ensembles in reducing NFR and observe that they remove negative flips that are typically not close to the decision boundary, but often exhibit large deviations in the distance among their logits. Based on the observation, we present method, called Ensemble Logit Difference Inhibition (ELODI), to train classification sys- tem that achieves paragon performance both error rate and NFR, at the inference cost of single model. The method distills homogeneous ensemble student model which is used to update the classification system. ELODI also in- troduces generalized distillation objective, Logit Difference Inhibition (LDI), which penalizes changes in the logits between the reference ensemble and the stu- dent single model. On multiple image classification benchmarks, model updates with ELODI demonstrate superior accuracy retention and NFR reduction.
single
INTRODUCTION
The rapid development of visual recognition in recent years has led to the need for frequently updat- ing existing models in production-scall scale systems. However, when replacing legacy classification model, one has to weigh the benefit of decreased error rate against the risk of introducing new er- rors that may disrupt post-processing pipelines (Yan et al., 202T) or cause friction with human users (Bansal al. 2019). Dositive-Congrucee Training (PC- (Pa-nnnni refers any training procedure that minimizes the negative flip rate (NFR) along with the error rate (ER).
Negative flips are instances that are misclassified by the new model, but correctly classified by the old one. They are manifest in both visual and natural language tasks (Yan et al.) 2021 Xie et al. 202T). They typically include not only samples close to the decision boundary, but also high- confidence mistakes that lead to perceived "regression" in performance compared to the old model. They are present even in identical architectures trained from different initial conditions, or with different data augmentations, or using different sampling of mini- batches. Yan et al.) (202I) have shown that in state-of-the-as image classification models, where 1% improvement considered significant, NFR can be in the order of 4-55% even across models that have identical ER. These intriguing properties motivate us to investigate causes of negative flips and mechanism of reducing negative flips to establish a model update method that achieves the cross moosl compatibility, thus lower NFR, and lower error rate, for better PC-training.
Two questions. A naive approach to cross-model compatibility is to bias one model to mimic the other, as done in model distillation (Hinton et al. 2015). In this case, however, compatibility comes at the expense of accuracy (Yan al. 2021 Bansal et al., 2019). On the other hand, averaging number of models in a deep ensemble (Lakshminarayamm
et al. 2017) can reduce NFR without negative accuracy impact (Yan al.) 2021), even if it does not explicitly optimize NFR nor its surrogates. The role of ensembles in improving accuracy is widely known, but our first question arises: what is the role of ensembles in reducing NFR?
Even though using deep ensembles achieves fate-of-the-an performance in terms of reducing NFR (Yan et al. 2021). it is not viable in real applications scale since it multiplies the cost of

---

Under review as a conference paper at ICLR 2023
ResNet-50 NFR (98)
Xm
Xm
NFR N
O O Ensembic legts
O 000
Focal Distil (FD)
Bietilation w/: 1.F
Ensemble wi. FD
Ensemble wi. KD
0
O
O
Ensemble. Paragon
Update w/. Ecoor
ELODI (Ours)
22
24
26
ResNet-50 ER (%)
Figure 1:
Left: In ELODI, one model is trained using the Logit Difference Inhibition (LDI) loss w.rl an ensemble of m models with its same architecture. The result is a single model which achieves significantly reduced negative flip rate (NFR) with the other. Right: Seatter plot of ResNet-50's ER vs its NFR w.rr ResNet-18, The more left and lower, the better. ELODI improves both ER and NFR than baseline methods. Particularly, ELODI is close the ensemble paragon, without the prohibitive computation cost of ensembles.
inference by an integer factor. Therefore, a second key question arises: Is it possible to achieve the PC-Training performance of ensembles the inference cost of a single model?
Key ideas. To address the first key question above, we analyze the pattern of negative flip reduction in deep ensembles. We observe that deep ensembles reduce NFR by remedying potential flip samples that have relatively large variation in the logits space of different single models. When deep ensemble composed of member models with the same architecture but trained with independent initialization on the same dataset, which we denote as homogeneous ensembles, this behavior can be theoretically predicted and empirically validated.
To address the second key question, we propose to train single model by penalizing the difference of sample logits from the mean of a deep homogeneous ensemble and to use this single model to perform model update. As illustrated in Figure Left), we independently train replicas of single model with different random seeds to form the deep ensemble. We introduce a generalized distillation objective, Logit Difference Inhibition (LDI), which only penalizes significant changes in the logits hetween the reference ensemble and the student single model, to realize the ensemble to single model distillation. The result is what we call Ensemble Logit Difference Inhibition (ELODI)..
Contributions. ELODI improves the state of the art in reducing perceived regression in model updates three ways: (I) Generality, by not targeting distillation reducing NFR; (2) Absence of collateral damage, by retaining the accuracy of a new model, or even improving it, while ensuring reduction of NFR: (3) Efficiency, as ELODI does not require evaluating ensembles of models at inference time These improvements are made possible by two main contri- butions: (1) an analysis on deep ensembles which sheds light on their role in reducing NFR and the direction to obtain the their performance for PC-training with single models; (2) ELODD, that inte- grates the NFR reduction of deep ensembles and running cost of single models by first training deep networks using the LDI loss with respect to an ensemble and then deploying the resulting single model at inference time. This results in a significant reduction of NFR (29% relative reduction on ImageNet for ResNet- 18 + ResNet-50) over previous methods. As a side benefit, ELODI increases top-1 accuracy in several cases, and is comparable in others.
specific legacy model, yet
2 RELATED WORK
Cross-model compatibility is becoming increasingly important as real world systems incorporate trained components that, if replaced, can wreak havoc with proc-psrg pipelines. [Toneva et al. (2019) empirically study prediction flip training samples between epochs, termed "forgetting events", while Yan et al. (2021) address perceived regression using held-out sets between different models. Both are particular instance of cross-model compatibility (Shen al. 2020 Bansal al.) 2019 Srivastava et al., 2020). Focal Distillation (Yan et al.) (202)) minimizes the distance between the old and new predictions, with increased weights on samples correctly classified by the old model.
Träuble et al. (2021) use a probabilistic approach to determine whether the prediction should update when a new model comes. While it improves cumulative NFR, it requires multiple models to be available at inference, which is prohibitive in practice.
Ensemble learning methods (Breiman 1996. Freund & Schapire 1997 Breiman 2001) are widely adopted in machine learning. The understanding for these methods is sometimes explained as en-
Note that ELODI able to deal with existing models trained without treatment, shown in Appendix B.2
2

---

Under review as a conference paper at ICLR 2023
larging the margins (Bartlett et al. 1998). Recently, the "multi-view" hypothesis (Allen-Zhu & Lij 2020) suggests that each independent model in an ensemble of deep networks learns a subset of fea- ture views and memorizes data not separable using this subset. In practice, one can always boost the performance of a classifier by averaging multiple models that are trained separately under a certain level of variation in training including model type, training data, initialization, etc. In this paper, we take a different aspect for ensembling to reduce NFR, not to improve accuracy. We apply ensemble as a teacher's model to guide the student model in reducing negative flips during model updates. In particular, we present an alternative explanation from the perspective of representations' dispersion in the logit space. ELODI can be thought of as variance reduction regularization in Bayesian NN ensemble, which is replaced by its mean at inference time. The literature on variance reduction is too vast to survey here, but relevant references include (Hoeting et al. 1999 Fragoso et al. 2018).
Knowledge distillation (KD) (Hinton et al. 2015) was proposed to transfer "dark" knowledge from a larger "teacher" network to a smaller "student" by minimizing the distance between the distribu- tion of predictions. In self distillation (Zhang et al. 2019), teacher and student are the same. Focal Distillation (Yan et al. 2021) is a special case of KD with a sample-specific filtering function, devel- oped for model updates where the legacy "teacher" model is actually weaker than the student (new) model, as in Reversed KD (Yuan et al., (2020). where it is used as regularization. Ensemble distilla- tion uses multiple teachers to improve accuracy in vision and other applications (Reich al.] [020 Fukuda et al., [017] Asif et al. 2020 Malinin et al., [020]] Lin et al. (020). Our method is related to ensemble distillation while having two distinctive differences: (1) Our method uses a different term for the loss to achieve reduction of NFR; (2) our method use a homogeneous ensemble whose
members have the same architecture and are trained on the same dataset with different initialization seeds, unlike the traditional case that uses diverse models in the ensemble (Kuncheva & Whitaker 2003). which we call a heterogeneous ensemble.
3 REPRESENTATION LANDSCAPE OF ENSEMBLE-BASED PC-TRAINING
To answer the first key questions in Section I we explore (1) how negative flips occur and (2) why ensembles yield fewer negative flips. To do so, we analyze the so-called logit space, where the representations computed by a deep network before the softmax operation live. The reason we analyze logits rather than feature or softmax probabilities is as follows. Compared to the feature space, the logits of an arbitrary sample produced by different models, trained on the same dataset, live in the same vector space which is defined by the label set of the training samples. Compared to the space of post-softmax output, the logit distribution is easier analyze because the softmax operation will skew the distribution. From a practical perspective, averaging in the logit space for ensembles is also common in recent works (Gontijo-Lopes et al. 2022 Wortsman et al. 2022)
3.1 NEGATIVE FLIPS AND LOGIT DISPLACEMENT MAGNITUDE
200
Given input image x and a learned model Φ. let (xx)) RC de- note its output logit vector before soft max, where C' is the number of classes. The logit vector varies across models due to architecture, initialization, optimization method, and training dataset to name a few. For any model pair & and v, we define the logit displacement to be the difference between two output logits, i.e x(x)= - (xx).
250
203
150
budidoan didance butmoon (No modes 14p-2 ligit
Lc 11 -Ga.11ll
Once its norm reaches a threshold and changes the order of the top predictions, a flip occurs.
Negative flips with large mross- logit displace- ment can be significantly re- duced by ensembles. Specifi- cally, 88 out of 97 (90.7%) neg- ative flipped samples are reverted in the red box.
Figure 2:
Our observation, as shown in Figure 2 that negative flips with large cross-model logit displacement can be significantly reduced particular, we examine the Fu- clidean distance of negative flipped sample's 's Top-2 highest out- put logits between two models. Replacing the single model's logits with an ensemble's, s, many of these negative flips can be reverted especially those with high logit distance (red box). Below we show that homogeneous ensembles indeed reduce negative flips through reducing the magnitude of logit displacement.
by homogeneous ensembles.
Let M, = f... (b(), .... and M, {.... 25(1) } denote the set of models of two ensem- bles. For simplicity, we consider [M,]]] || = || m, where - is the cardinality of each
3

---

Under review as a conference paper at ICLR 2023
set. Each ((+) or x6(1) has the same model architecture, respectively, and are trained on the same dataset, despite being independently initialized so that they are homogenous ensembles. Given an input image vi {<)())))) (x),.... (5(m)()) can be considered m i.i.d random variables drawn from second-order by an expectation μ1 and a co-variance matrix Σ,, i.e ((z) ~ DH1, Σ1). Likewise (b()) ~ D(pug Σg). The ensembles' logit vectors are computed as d (ens) (I) and z (ens)) (x) by averaging the individual models' logits. The multi-dimeensonal central limit theorem (CLT) (Rvačeva 1962 Ferguson| (017) states that this average converges in distribu-
a distribution approximated
tion to multivariate normal distribution with the increase of m. ie
Σ (c) N(u -x).
Therefore, the logit displacement between the two ensembles converges in distribution to another multivariate normal distribution, i.e
(oms)
(eme)
(ac)
1 JR
Σ (z)
Σ
(x) (x) 2N pI 142,
Σ, +
(2)
HEMI
JM
The norm of logit displacement will follow a generalized x² distribution (Mathai & Provost 1992
Das & Geisler 202I).
As special case, if M, and Mg have the same model architecture. then we have a normal distribu- tion with zero mean and co-variance inversely scaled by the ensemble size:
(ana)
(emns) (x) 2
Σ1 + Σ2
m
We connect the analysis to the observations shown in Figure 4 (1) Equation 3 implies that when ensembles become larger, the expectation of logit difference is zero and the covariance keeps de- creasing, resulting in consistently decreasing NFR. This is consistent with the observation in (Yan) ct al. 202T), which is redrawn in the red curve, that two very large ensembles with the same ar- chitecture have almost flips. (2) In the case of two homogeneous ensembles with different architectures, μ1 and μ2 could have a non-zero difference which results in NFR not converging to zero. But the decrease of covariance part still contributes to consistent non-trivial reduction in NFR. These explain the observation in the orange curve that NFR stagnates at a smaller non-zero value. Additionally, although we cannot conduct similar analysis for a heterogeneous ensemble, we can empirically verify that the NFR between two heterogeneous ensembles has significantly higher variance, as shown in the purple curve, which suggests they may be inferior for NFR reduction.
3.2 VALIDATING THE HYPOTHESIS OF REDUCING LOGIT DISPLACEMENT MAGNITUDE
We validate our hypothesis on the representation landscape through large-scale experiments. Specif- ically, we train 256 R8 models on ImageNet with different seeds and split them into two halves. For arbitrary image, randomly draw m models without replacement in each half and compute the averaged logits of this drawn ensemble. We repeat the process and present in Figure Ba the histogram of the logit displacement's s &z norm between two random ensembles. Note (Seging let al. (202T) argue that the logit distribution is highly affected by the number of training epochs, therefore we follow the standard training recipe in Appendix A for its assumption hold.
We examine our hypothesis in Section 3.1 by comparing the histogram with the probability mass function (PMF) of logit difference norm. We start by using all available single models estimate mean and co-variance (fa, Σ) for the logit vectors' distribution & and examine whether the norm of logit displacement will follow a generalized (central) x² distribution. Since the probability density function (PDF) of a generalized x² variable does not have simple closed -ooeed-form expression, we estimate it by Kernel Density Estimation (KDE) method (Parzen 1962). From Figure Bal we see that the simulated PMFs in solid lines fit the histogram of single models well, implying that logits of these models could indeed follow a normal distribution. We conduct the same experiments above on more images in Appendix T and the conclusion holds well, suggesting this property is not incidental. If we move to the ensemble case, the logit displacement follows another normal distribution with scaled co-variance matrix, which leads to another generalized x2 distribution with shifted mean and scaled covariance. Its estimated PMF, shown in dashed lines in Figure Ba is consistent with the corresponding histogram.
4

---

Under review as a conference paper at ICLR 2023
Homog, v2 Hamog.
pedel Man pa21
Homog R11 vs Homog:
Enoemblcs s Brorrstte 19
40
Hetero, weak Hetern strong
Hamog f118
Exa
Homog, R50
Hoceen strong
6 1977 ou optchp
(a)
Homog. R-18 Ensemble
(b) Homog. R-18 Ensemble
Homog.
R-18 Ensemble.
Homog.
R-50 Ensemble.
Ap = 0, E' 2
Ap 0, Σ Σg.
20
35
32
54
120
Figure 3: & norm histogram of logit displacement between two random ensembles. The bin size is 0.5. We also plot sim- ulated PMFs: solid lines for & norm of a simulated normal distri- bution N (Aje, [Σ1 + Σz)) whose parameters are estimated from all available single models; dashed lines for those of extrapolated distribution N' (Au,, 1 E¹) Consistency between the histograms
Ehsermb
Figure 4: ER/NER w.r.f ensemble size. The behaviors of model updates under dif- ferent ensemble settings can be explained by our hypothesis. Please refer to the last paragraph Section
PMFs supports our hypotheses in Section 3.T
Algorithm 1: Sequential model update with ELODI
Data: Dataset X; number of versions to update T
Result: A sequence of models of (Vee--)) of (Ve-T) t <--00 O;
/* initial version +/
repeat
for
i 1... m do
(1. +- arg min Ezex LCE (c((Ver());
/x Train m models */
end
for
x E X do
(ons, Vor-t) (z) f 1 Eie(1,... m² gli. ((r.Vt))))
/* Build old ensemble +/
end
* (Ver-t) + argming ExEx Laistill (Φ(x), (ens, Ver-t) single model and publish */ until T:
/+ Distill into
The sample-specific logit displacement norm between two homogeneous ensembles with different architectures can be analyzed likewise. Considering (xx) N( E1) for ResNet-18 and xb(x) (u,, Σg) for ResNer-50, the norm of logit displacement should follow a generalized non-central x2 distribution. From Figure 3b we can see that the estimated distribution of the logit displacement norm fits the empirical distribution well. It still condenses as the ensemble get larger but much slower than Figure 3a Also the mean converges to non-zero (the dotted vertical line Figure 3b).
4 METHOD: ENSEMBLE LOGIT DIFFERENCE INHIBITION (ELODI)
The above analysis suggests the effectiveness of large homogeneous ensembles in reducing NFR, but an ensemble is less practical compared with single model due to its multiplied inference cost to run every member model on a new input. In this work, we propose re-purpose the knowledge dis- tillation technique (Hinton et al. (2015), which was previously used for improving model accuracy, for the task of transterring NFR reduction capability from ensembles to single models.
MODELS Given an ensemble &i (eus) composed of a set models MI = (1) Jm Jm we learn a single model I such that for each sample I E X, the random variable of sample logits can have ensemble-like reduced variance. We then use single models learned in this way in every model update of system.
4.1 UPDATES WITH ENSEMBLE-
As illustrated in Figure I Left), when an old model (old) needs to updated a new model, we first train homogeneous ensemble on the same dataset and having the same architecture as the desired new model. Next distill this reference ensemble the actual new model (new)) If the distillation process able to convey the property of reducing NFR to the learned single model, updating from & (old), which was preferably produced in the same manner, to o (new) would result in significantly reduced NFR than that of model pair without this treatment.
5

---

Under review as a conference paper at ICLR 2023
We empirically verify this model update method, called ELODI, in Section 5.1 and find it effecttve in reducing NFR while retaining the accuracy gain introduced by the new architecture used in the new model. It avoids the prohibitive inference cost of deep ensembles due to only using & (B(nww in the model update, with reference ensemble discarded after the distillation. Another benefit of ELODI is that new model does not need to target any specific existing model in an update, enabling chain of models to yield relatively low NFR between any pair of them. This rather helpful when multiple updates are consecutively executed, which is common for a long-running machine learning system. The overall pipeline of sequential model update via ELODI summarized in Algorithm
4.2 Loss CHOICES OF ELODI
Generally we obtain the model to be deployed using the distillation technique (Hinton et al. 2015).
arg min
Σ ZEX
Edistill (b(zz), Φ (ene) (x))
Various types of distilling function have been proposed to improve single-model accuracy: (1) The
vanilla KD loss (Hinton et al. 20155 minimizes the KL-divergence between two models" output logits; (2) FitNet (Romero et al. 2013) and Attention
(Zagoruyko & Komodakis) (017)
mimics the intermediate hidden layer's activation attention; FSP (Yim et al. 2017) mimics the cross-layer Gram matrices. Our goal is to minimize the norm of covariance matrix in terms of sample logits between the learned model and the ensemble,
arg min
[|ar(((()) - Var(ø (ens) (x))
(5)
#EX
However, computing either term in Equation 5 is hard: (1) Var(Ø(a is ill ill-posed while training a single model; (2) So is Var ( (cnss) (x)) because the matrix dimension C, i.e the number of classes, is usually by several orders of magnitude larger than the ensemble size m (a typical value in the experiments is C 1, 000 and m = 8). As an alternative, we propose a new distillation loss named logit difference inhibition (LDI) that focuses on the first-order statistics but achieves similar goal of variance reduction. It can be viewed as a generalized version of KD loss but is more focused on penalizing large logit displacement towards the ensemble rather than exact logit matching.
Let ( (x) denote the k-th element of the logit vector. Motivated by the analysis in Section 3.1 instead of exact match for boosting accuracy, we only inhibit significant difference of logit vectors elements between the distilled model and the ensemble as follows:
LLDI(E) (x) = max (o. (0.1 - (cns) (z)] ())0))))) -))"
(6)
& is a non-negative truncating threshold such that difference below & is truncated. is set to be 0.1. K is a data-dependent subset of classes where the loss will be applied. K can include either all classes (f1...., C)) or the classes with the top- K highest logit elements. This top- design choice is motivated by our observation that flips are likely to happen in classes with high logit displacements. As shown in experiments, the Top-K variant leads to no loss in NFR reduction compared to the full form. It could instead help in reducing computation cost when number of classes are extremely large (An al. 2020). p is 2 in our experiments. When { = 0 and K = {1, C), LDI degrades to the logit matching loss (Hinton et al. 2015).
The overall objective of the distillation in ELODI is weighted sum of standard Cross Entropy and the LDI loss, i.e C= (1 - aCE + aLDDI, where the weight is set such that the magnitude of LCE and LLDI is similar.
EXPERIMENTS
We validate the proposed approaches on two standard image classification datasets: ImageNet (Deng et al., 2009) and iNaturalist (Van Horn et al. 20118). For a new model in an update experiment, we
measure top-1 error rate (ER) and its negative flip rate to the old model (NFR) (Yan et al. 2021).
computed as NFR = 1 E I(0) (new) * L,, l..0(mii (old) = f.). where I(-) is the indicator function, &
N
is the label, and g) [new)) (g) (ld)) is the new (old) model's prediction. The training recipe is described in Appendix A Unless otherwise specified, ELODII experiments are done with ensemble size = 8.

---

Under review as a conference paper at ICLR 2023
Table I: Comparing ELODI with other PC-Training Methods. in ensemble paragon means that the number is from a collection of models. The middle rows targeted model update baselines, where the old ResNet-18 referenced when training the ResNet-50. The bottom ELODI and its Top-K K variants.
NFR_(%)
Method
ResNet-18
ResNet-50
No treatment (single)
30.24
24.66
4.30
Ensemble Paragon (8x)
26.34"
22.44
1.95
(TARGETED MODEL UPDATE)
BCT Shen et al.
2020
30.24
25.00
4.34
KD Hinton al. 2015
28.38
3.20
FD-KL
Yan el al. 2021
26.32
2.90
BU-CR Traubble et al.
2021
26.51
4.56
ELODI
31.34 30.95
23.15
2.18
Top-K ELODI (K = 10)
23.10
2.11
Table ELODI in data-growth settings on ImageNet.
means that model is trained on half (full) data.
Increasing ficlasees
Increasing #samples/class
Method
Error Rate
Ertor Rate)(%)
NFR.
NFR,
R18
R50
R18
R50
No treatment
22,02
24.66
14.07
34.26
24.66
3.52
FD Yan et al. 2021
39.96
5.45
33.06
2.65
Ensemble Paragon (8X)
18.70"
22.44
4.12
29.16*
22.44
2.11
ELODI
21.80
23.15
4.19
34.08
23.15
2.25
5.1 MAIN RESULTS OF ELODI
Comparison with other PC-Training Methods. We first compare ELODI with other PC-Training methods under the setting of updating ResNet-18 to ResNet-50. "No treatment" means that both ResNet-18 and -50 are trained with standard cross entropy loss. We can see that ELODI outper- forms previous methods such Backward-Compatie Training (BCT) (Shen et al. 2020 Knowl- edge Distillation (KD) (Hinton et al. 2015). Focal Distillation (FD-KD/LM) ( Yann et al. 2021) and
Bayesian Update (BU) (Träuble et al. 2021 Note that these baselines all targeted model up- date, where the old ResNet-18 is used as target when training ResNet-50 while ELODI does not target any legacy model. Second, can apply the LDI loss the targeted model update setting and it achieves slightly worse NFR compared to FD while attaining ResNet-50's accuracy signifi- cantly better. See more discussion in Section 5.3 Third, we study a few variants ELODI. Using the Top-K highest-logit class subset in ELODI with K E =22, 5, 10} does not deteriorate the perfor- mance (+.0.1%) 1%). From an ER-NFR scatter plot in Figure Right), EL.ODI achieves a similar level of ER-NFR results as the ensemble paragon (Yan et al. 2021 at the inference cost of a single model.
5.2 MORE UPDATE SETTINGS OF ELODI
Data-growth settings. Model updates may also come with the growth of training data, egg (1) increasing number of classes, and (2) increasing number of per-class samples. We follow the same data/class split in (Yan et al. 2021). which uses 50% classes/samples for old model and full data for we find that conclusions from the full-data setting also hold for these settings.
new. From Table 2
Fine-tuning other datasets. We validate the effectiveness of ELODI when transferring iNat- uralist (Van Hor et al.] 2018) following the protocol in (Yan et al., (022]. Results of both full-data and Mata-growth setting summarized in Table where ELODI consistently outperforms FD.
ELODI on a chain of model updates. To illustrate Algorithm 1 we show the transitivity of NFR reduction induced by ELODDI in chain updates of three models, ie 18 ReeeieeeeeeeeteOO-teeeeee
101. As shown in Table 6 with ELODI, NFRs between the three models reduce to 2.04% ~ 2, 2.25% from 3.92%
4.41% (a relative reduction of 44.1% 52.3%). Note this is achieved without
In our re-implementation, use CostRatio(CR (ClikaatiooR)-baaana -baed Bayesian update rule get the target and train the model online instead of offline model update in the original paper.

---

Under review as a conference paper at ICLR 2023
Table 3:
ELODI on NNaturalist, ELODI is effective when fine-tuning on iNaturalist under multiple settings.
Increasing #classes
Increasing #samples/class
Full data
Method
ER_
ER,
ER, (%)
NFR,
NFR,
NFR,
R18
R50
R18
R500
R18
R50
No treatment
47.58
35,95
5.38
78.88
35.95
3.82
40.69
35.95
4.76
FD Yan et al. 2021
45.87
3.44
66.91
2,00
40.03
3.95
Ensemble Paragon (8x)
36.03'
29.47
1.68
47.04"
29.47*
1.23
36.25*
29.47*
2.10
ELODI
43.56
34.29
1.91
52.96
34.29
1.47
40.37
34.29
2.46
Table 4: ELODI with different guiding ensembles. We consider ResNet-18 ResNect-0 via ELODI with an 8x-model ensemble. All-diff-weak (-strong) ensemble is composed of 8 different weak (strong) models with Top-1 Acc. 69% (75%) ImageNet. The model list can be found in Appendix [ Mixed-weak (-strong)
ensemble is mixture ResNet-18 and 4x VGG-13 (4x ResNet-50 and 4x DenseNet-121).
Error Rate, (%)
NFR, (%)
Old Reference
New Reference
R18
R50
-R500
N/A
N/A
30.24
24.66
4.30
All-diff-weak
All-diff-weak
32.38
26.11
1.94
Mixed-weak
Mixed-weak
32.75
26.88
1.99
R-18 (x8)
R-18 (x8)
31.32
26.82
2.06
All-diff-weal All-diff-wcak (calibrated)
All-diff-strong All-diff-strong (calibrated)
32.73
23.14
2.25
33.38
23.33
2.35
Mixed-weak
Mixed-strong R-50 (x8)
32.75
23.68
2.24
R-18 (x8)
31.32
23.15
2.18
Table 5: ELODI with different architectures on ImageNet. ELODI effectively reduces NFR on wide range of architectures. is obtained reproduction with different augmentation and training schedule. Note that all new models' NFR is measured w.r. ResNet-18 listed the leftmost column.
ER, (%) R-18 (old)
ER, (%) R-101
NFR (%)
ER, (%)
NFR, (%)
ER, (%) Swin'T
NFR, (%)
-R-101
D-161
+D-161
-SwinT
None (single)
30.24
24.66
3.64
21.82
3.73
20.40
3.77
Ensemble (8x)
26.34*
20.05*
1.72
18.90
2.06
18.37
2.60
F1.0D1 31,34
21.09
2.19
21.74
2.57
19.84
2.95
crafting the complex reference schemes which are necessary for baselines since they require old models' references. The details of other baseline methods are discussed in Appendix [B.1]
Updates to dissimilar architectures. In Table 5 we study if ELODI is applicable to updates across dissimilar architectures We see that EL.DDT effectively reduces NFR in all cases, with retained or sometimes decreased ER.
Net MaeNNeet-161Tnnn 161/Tiny-Swin besides similar ones (esNet--10))
5.3 DESIGN CHOICES OF ELODI
we use homogeneous en- semble. However, in ensemble learning, members with strong diversity such as model architectures are usually favored for better generalization. In Table 4 we observe that using homogeneous en- semble for guidance achieves comparable or slightly better results in both NFR and ER than hetero- geneous ("all-differentt) ensembles. We also additionally calibrate "all-different" ensembles (Guo) al. 2017) but observe no gain. This suggests that strong diversity in guiding ensemble may not lead to lower NFR. Also, ELODI with homogeneous ensembles is easier to implement and extend.
Homogeneous vs heterogeneous ensembles. In both Section 3.1 and 4.1
Change of architecture for the guiding ensemble. In Table 4 we find that training new model guided by ensemble with the old model's architecture has to trade ER for reduction of NFR, which is not desired. This corroborates with the hypothesis in Section 3.1 that models with different ensemble with the
architectures has different representation landscape and thus it is better same architecture for guiding ELODI. When a system has gone through multiple updates, always guiding EL.ODI with the new model's architecture also provides clear guideline for practice.
8

---

Under review as a conference paper at ICLR 2023
Table 6: ELODI achieves lower Pairwise NFR on chain of models. M + M2 means that measure M2's NER wer Mr. M means
Table 7: Distilling ensembles with differ- loss functions. Considering model update of ResNet-- -888--Re--55500 ELODI achieves lower NFR and ER than KD/FD.
that M trained from ELODI.
Method
Pairwise NER
|
(%)
Method
R18
R50
3.92%
None
R18
R50
R101
ELODI
31.0
23.28
2.26
4.28%
4.41%
Ensemble w/. KD r=101
32,09
23.67
2.23
2194
Ensemble w/. FD,=100
32.19
23.97
2.16
ELODI
R18
2,04%
R50
2.25%
R101 9
Ensemble w/. FD,=1
31.62
24.06
2.43
Table 8: Comparison between offiine and on- line distill on ImageNet. Inferring teacher log- its during training (online) achieves both lower ER and NFR compared to pre-extracting (offline).
22
Error Rate,
NFR,(90
Rocket
Radk-11 PL-8ANS
Method
R18
R50
(a) Varying loss weights.
(b) Ensemble size.
Offline Online
32.49
24.26
2.38
Figure 5: Ablating loss weights and ensemble sizes for ELODI on ImageNet. ER ResNet-18 18 (-50) is shown in the light (dark) blue bar plot while NFR the red curve.
30.97
23.81
2.15
Choices of distillation functions. We compare different choices of distillation loss functions in Table 7 We can see that using LDI loss, i. ELODI, outperforms FD (Yan et al. 2021 (Ensemble w/. FD) or KD (Hinton al. 2015) (Ensemble w/. KD) in both ER and NFR.
5.4 EXPLORATORY STUDIES OF PARAMETERS IN ELODI
The effect of loss weight. We experiment with different loss weight a and summarize the results in 5a OEELODI 0 equivalent the no-treatment baseline. When OLDI increases from 0.5 to 1, the distilled model's s ER first decreases and then increases for both models. On the other hand, NFR consistently decreases and stays at ~2.2%. We find @ggOOD 0.8 achieves a good balance between the distilled model's ER and NFR. Therefore we use it by default for all ELODI experiments.
The size of reference ensemble. We study ELODI'S efficiency for reducing NFR by varying the 0 is the no-treatment baseline. The case of
ensemble size in Figure 5b. The case of m can be viewed as self distillation (Zhang et al., (009) except that the new model's weight is re-initialized with different random seed. NFR decreases from 4.30% to 2.15% by when the ensemble size increases from 8.
Online rs offline distillation. In ELODI, the ensemble's logits can be either inferred during training (online) or pre-extracted before training (offline). Table g we find that offline distillation less effective in reducing NFR and ER. Similar observation is also reported in (Beyer et al. 2022 Therefore we use the online approach in all experiments.
6 DISCUSSION
Our experiments show that ELODI performs positive congruent training by reducing negative flips with large logit displacement and reducing the variance of logits from the ensemble estimates. This behavior can be transferred to single models through the ELODI method can benefit updates with single models. As discussed in Section 3.1 and later observed
experiments, the difference in representation landscape could still lead to non-zero NER in the updates even with ELODI, which requires future works on in-depth characterizing of representation landscape change between model architectures. Another limitation of ELODI is that the training cost is still higher than the normal training process of a classification model update, due to the additional training of the ensemble and online inference of the ensemble logits, calling for further efficiency improvement.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09876, 2020.
Xiang An, Xuhan Zhu, Yang Xiao, Lan Wu, Ming Zhang, Yuan Gao, Bin Qin, Debing Zhang, and Fu Ying. Partial fe: Training 10 million identities on single machine. In Arxiv 2010.05222, 2020.
Umar Asif, Jianbin Tang, and Stefan Harrer. Ensemble knowledge distillation for learning improved and efficient networks. In ECAI, 2020.
Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel Weld, Walter S Lasecki, and Eric Horvitz. Up- dates in human-ai teams: Understanding and addressing the performanc/compatbiiyy tradeoff. In AAAI, 2019.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert Schapire. Boosting the margin: A new explanation for the effectiveness of voting methods. The annals of statisties, 2665):6651-1666 1998.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher is patient and consistent. In CVPR, 2022.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-14) 1996.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprimi 1rXiv:100.061774 2016.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In NeurlPS, 2017.
Abhranil Das and Wilson S Geisler. A method to integrate and classify normal distributions. Journal of Vision, 21(10):1-1, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. RepVGG: Making vtg-style convnets great again. In CVPR, 2021.
Thomas S Ferguson. A course in large sample theory. Routledge, 2017.
Tiago M Fragoso, Wesley Bertoli, and Francisco Louzada. Bayesian model averaging: A systematic review and conceptual classification. International Statistical Review, 86(1):1-28, 2018.
Yoav Freund and Robert E Schapire. A decision-theoreticc generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139, 1997.
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramab- hadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, 2017.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D Cubuk. No representation to rule them all: Overlapping features of training methods. In JCLR, 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint /7Xiv:
2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint OrXiv:1503.02533 2015.
Jennifer A. Hoeting, David Madigan, Adrian E. Raftery, and Chris T. Volinsky. Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. George, and rejoinder by the authors. Statistical Science, 14(4):382 417, 1999. doi:
URL
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Ludmila Kuncheva and Christopher J Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning, 51(2):181-207 2003.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NewrIPS, 2017.
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NeurlPS, 2020.
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. ICLR, 2020.
Arakaparampil M Mathai and Serge Provost. Quadratic forms in random variables: theory and applications. Dekker, 1992.
Emanuel Parzen, On estimation of probability density function and mode. The annals of mathe- matical statistics, 33(3):1065-1076, 1962.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
Steven Reich, David Mueller, and Nicholas Andrews. Ensemble distillation for structured predic- tion: Calibrated, accurate, fast-choose three. In EMNLP, 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
EL Rvačeva. On domains of attraction of Mathematical Statistics and Probability, 2:183-205, 1962.
distributions. Selected Translations in
Landan Seguin, Anthony Ndirango, Neeli Mishra, YueYeon Chung, and Tyler Lee. Under- standing the logit distributions of udversarially-trained deep neural networks. arXiv preprint arXiv:2108.1 12001, 2021.
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatible repre- sentation learning. In CVPR, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. An empirical analysis of backward compatibility in machine learning systems. In KDD, 2020.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.

---

Under review as a conference paper at ICLR 2023
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
Frederik Träuble, Julius von Kügelgen, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Peter Gehler. Backward-compatibee prediction updates: A probabilistic approach. In NeurlPS, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep high-resolution representation learning for visual recognition. TPAMI, 43(10):334-3334, 2020.
Ross Wightman. Pytorch image models. pytorch-image-mode 2019.
Dttps://github.cmmm/nw
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
Yuqing Xie, Yi-an Lai, Yuanjun Xiong, Yi Zhang, and Stefano Soatto. Regression bugs in your model! measuring, reducing and analyzing regressions in nlp model updates. In ACL, 2021.
Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-congruent training: Towards regression-free model updates. In CVPR, 2021.
Junho Yim, Donggyu Joo, Jihoon Bae. and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR, 2018.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019.
12

---

Under review as a conference paper at ICLR 2023
A TRAINING RECIPE
We follow standard training recipe of ImageNef Specifically, all classification models trained with SGD with momentum of 0.9 and base learning rate of 0.1, which is reduced by IO every 30 epochs until 90 epochs. The batchsize is 256 with 8 GPUs.
For ELODI with larger models and ensemble size (e.g 8x DenseNet-161) GPU memory becomes bottleneck. To handle the memory issue, reduce batch size while linearly scaling base learning rate (Goyal et al. 2017).
gradient checkpointing (Chen et al. (016) and
B
MORE EXPERIMENTS ON ELODI
B.1 ELODI ON A CHAIN OF MODEL UPDATES
As discussed in Section 4.1 ELODI does not involve the guiding ensembles at the inference stage. Only the single model trained with ELODI deployed to replaced the old model, which is also trained with ELODI. When the number of updates increase, this naturally forms a chain of models having low NFR between them.
We illustrate this Iransitivity of NFR reduction induced by ELODI in chain updates of three mod- els, iee ResNet- 1eeectt18-Reess -ReeNeet50--ResSee 101. As shown in Table 9a| with EL.DI,, the NFR between the three models tested reduced to 2.04% ~ 2.25% from 3.92% 4.41% (a relative reduction of 44.1% ~ 52.3%), outperforming all previous methods, including variants of FD (Yan et al. 2021) and LDI. Note this is achieved without crafting the complex reference schemes which are necessary for baselines since they require old models' references.
B.2 INTEGRATING ELODI WITH EXISTING MODELS
Dealing with old models without ELODI is necessary when updating an existing system. We consider the simple case of one old model not trained with ELODI. In this case, we augment ELODI with an additional LDI loss w.r. to the old model, i.e
Liotal X Meww
M new
(1 A)LLDI Mola),
where M denote the model to be guided from the ensemble.
To show some empirical results, here we consider three models, 1 101, where ResNet- 18 model is trained without ELODI. Therefore both ResNet-500 and -101 will have a higher NFR compared with ResNet-18, To handle this, we introduce an additional LDI loss targeted at ResNet-18 18 when training 5esNet-50 and (or) ResNet-101 using ELODI. The results are shown in Table 9b We can see that ELODI w/. LDI outperforms ELODI w/o. LDI on all pairwise NFRs, indicating that augmenting ELODI with LDI loss towards the existing model is effective in dealing with this legacy case.
C DETAILS OF HETEROGENEOUS ENSEMBLES IN FIGURE AND TABLE
Model list of heterogeneous ensembles. The all-diff-weal ensemble is composed of 8 different weak models with Top-1 Accuracy 69 70% on ImageNet, including: ResNet-18 18 (He et al.,
2016), GoogleNet (Szegedy et al. 2015), VGG-11, VGG-13, VGG- I-BN, VGG-16 (Simonyan & Zisserman 2014; [offe & Szegedy 2015), HRNet-WI18 (Wang et al.] 2020), DLA-34 (Yu et al.) 2018). The all-dif--strong ensemble is composed of 8 different strong models with Top-1 Accu- racy 75 76% on ImageNet, including ResNet-50 (He et al. 2016)). DenseNet- 121 (Huan
al. 2017). Inception- V3 (Szegedy et al.] 2016), VGG-19-BN (Simonyan & Zisserman
2014).
RegNetY (Radosavovic et al. (020)) RepVGG-AZ (Ding et al.) 2021), DPN-68 (Chen et al.
2017
DLA-X-60-C (Yu al. (2018). All of the model weights are adopted from PyTorch Image Models
(TIMM) (Wightman 2019).
13

---

Under review as a conference paper at ICLR 2023
Table 9: Pairwise NER on multiple models. Mr M2 means that we measure Ma's NER W Mi. M1 7 Ma means that M2 is trained with M1 being teacher using distillation loss, egg FD or LDI. M
means that M is trained from ELODI.
Method
Pairwise NER
Method
Pairwise NFR
3.92%
2.19%
None
R18
R50
R101
ELODI
R18
R50
R101
9
4.41%
2.04%
2.25%
3.46%
2.00%
FD (chain)
R18
R50
R101
LDI (chain)
R18
R50
R101
2.90%
2.13%
2.86%
2.33%
2.63%
2.67%
FD (radial)
R18
R50
R101
LDI (radial)
R18
R50
R101
2. 90%
2.38%
2.86%
3.09%
2.96%
68%
FD (fe)
R18
R50
R101
LDI (fc)
RI8
R50
R101
1.97%
(a) Sequential update, (1) chain: each model targets its elosest predecessor; (2) radial: each model targets at its farthest ancestor; (3) fully-connected (fe): each model targets at all its ancestors.
LDI usage
Pairwise NFR
2 2.56% R-50
None
R-18
2.98%
2.25%
R-101
2 56% R-50
Once
R-18
R-101
285%
2.14%
Both
R-18
R-50
R-101
211%%
(b) Integrating ELODI with existing models.
Calibration of heterogeneous ensembles. We use temperature scaling (Guo et al. 2017) for each individual model the ensemble set to make the confidence score match the true correctness like- lihood on validation set. The temperature for scaling is optimized based on the loss on validation set with LBFGS. Since each model in the ensemble set has similar accuracy, after such calibration, their confidence scores matched the same level.
D A TWO-DIMENSIONAL EXAMPLE
To illustrate the behavior of models in logit space, we create a toy example by selecting two classes") from ImageNet (Deng et al., 2009) and training several ResNet-16 18 models for binary classification. The models differ by their initialization, determined by distinct random seeds; we then collect output logits for each test datum and model in the ensemble. In Figure 6a] plot the two-dimensional logit vectors of multiple data points when updating from an individual model to another. We can roughly categorize the negative flipped samples, highlighted with the purple arrows, into types: (1) those close to the decision boundary in the old model; (2) those far from the decision boundary in the old model but still flipped in the one, due to significant displacement of the logit vector. Figure 6b shows the logit vectors of the same set of data points but in the update case two ensemble models each having 3 members (3x). Compared magnitude of displacement during the update. To validate that this observation is not incidental, we construct many cases of model updates and measure the distribution of the logit vector displacement on certain data sample. As shown in Figure 6d in updates between ensembles, the logit vectors are less likely to exhibit significant displacement" This suggests that the ensemble may be reducing the negative flip rate through the reduction of displacements of the logit vectors.
Figure 6a| we can observe a clear reduction in the
4-| "abrador retriever" (n020997712 12) and "French bulldog" (n02108915).
'Data sample is randomly selected. We present the same visualization more data points in Appendix F
14

---

Under review as a conference paper at ICLR 2023
abllegt
<boc
single
1.10
0.00
0.25
onsemble modals
0.20
new es egt
510
(a) Two single models: Test samples flip even if not close to the boundary (long arrows).
(b) Two ensembles of models each (3x): Fewer samples that are far from the boundary flip (shorter arrows).
Logits of ensembles their members: Individual models' logits (lighter circles) center around the mode.
435
(d) PMF of logit displacement.
Figure 6: Visualization of a 2-class example. (a-c): Two-class logits of two single models and/or ensembles. and refer to the ground-truth classes, while red and green data points refer old and new model's 's logits. Magenta arrow, blue arrow, and gray arrow link negative flip, positive flip, and consistent (either both correct both wrong) prediction pairs. All dots with black borders are depicting the same image. (d): Estimated probability mass function (PMF) of logit displacement between two single models or ensembles. The æ, y-axes denote the two classes' logit displacement. The heatmap value denotes the estimated probability density. The ensemble's co- carianeece is significantly smaller than the single model. The figure is best viewed in color.
(a) val 00009585.
(b) val 00015098.
(c) val val__a0034000000 00034619.
(d) val__001 0001 4560.
Figure 7: Four example images for visualization. (n02108915) and "Welsh Corgi" (n02113023).
The two classes we select here are "French bulldog"
E VISUALIZATION ON MORE DATA POINTS
As mentioned Appendix D and Section 3.2 we provide more examples to verify our hypothesis. We select four images of two classes from ImageNet (Deng et al., (209).. which are illustrated in Figure 7 as input data. With these input images, the estimated probability mass function (PMF) of logit displacement between two single model and two ensembles are shown in Figure 8 We can observe that the logit displacements are reduced with ensembles, which verifies our hypothesis of output logit vectors are actually independent and identically distributed (i.i.d) random variables and with multi dimensional central limit theorem (CLT), their sum is a normal distribution (Equation [].
To verify our hypothesis in higher dimension space, train a standard ResNet-18 on full ImageNet dataset with 256 random seeds. We take the images in Figure as inputs and illustrate the l norm histogram of logit displacement between
random ensembles with different ensemble sizes in Figure 9 For heterogeneous case, we train standard ResNet-50 on full ImageNet dataset and observe the f norm histogram of logit displacement between random ResNet- 18 ensembles and random ResNet-50 ensembles with different ensemble sizes. The results are shown in Figure 10
F REPRESENTATION LANDSCAPE OF PC-TRAINING AT THE FEATURE LEVEL
We have discussed the representation landscape of PC-Training in Section 3.1 at the logit space and provide more data points above. The analysis be done in the feature space well. The main challenge is that features from two arbitrary models are not directly comparable and we address this feature interoperabilitt issue by Backward-Compatible Training (Shen al.) 2020
15

---

Under review as a conference paper at ICLR 2023
single models
single models
single models
single models
0.42
0.4
0.33
033
0.30
0.33
0.33
0.5
0.23
0.2
0.21
023
ensemble
0.23
0.23
0.25
ensemble
023
0.15
0.15
015
4.10
0.1
01c
(a) PMF logit displacement (val 111_000095800
(b) PMF of logit displacement wval_00015098)
(c) PMF logit displacement [val_000346)
(d) PMF of logit displacement (va1_00014560)
Figure 8: Estimated probability mass function (PMF) of logit displacement between two single models or ensembles. The æ, y-axes denote the two classes' logit displacement. The heatmap value denotes the estimated probability density. The ensemble's co-variance is significantly smaller than the single model. The figure is best viewed in color.
We first introduce the BCT method and then derive that formulation to attain the feature at the penultimate layer of an ensemble. Based on these we can analyze two-dimensional examples and the higher-dimension validation experiments.
Preliminaries. Shen et al. (2020) propose an approach termed BCT to align two arbitrary deep interoperable with each other. Formally speaking, model
models that the embeddings M includes an embedding module (z = F(r), a.k.a backbone) and a classification layer (s = H((z), a.k.a head) on top, i. M(x) = o(x) = (H (x). Given a reference model M(tref), BCT imposes loss term so that two model heads are close, i. (bet) H(ref 6 As result, F(bet)()) and (r(ef) (a:) lie in a same vector space and are thus comparable, i.e F ((act)()) (x)) F((rf)) (x), gardless of the underlying architecture.
Ensemble of many feature-interoperahhe models. It is noteworthy that feature interoperability does not affect NER as reported by) Yan et al, (2021). We also re- validdaatt that two models, F((bt) (x)
and F (bct) (x), trained using BCT w.r. (ref) have similar NFR compared two without BCT. However, their features are comparable, ie (bct)) (x) combination in between.
(bbtt) (x: ~ F(ref) (x). So is any linear
The arguments hold when the number of feature-interoperable models increases. Therefore, if write down their averaged logits, we can factor out the head, i.e
S (ett ens) (x)
L
(bet) (w)
1 Σ
Hr(bect)
F!bcet) (x)
(8)
N
(ettot) F(bbts) (x) (rot))
Σ F(bet) x(bc))(2)))
(x))
(9)
N
N
implies that the averaged feature can be viewed as this ensemble's feature, i.e F(bctt 1
N Σ E=(bbt)) (bet) (z).
(wo-dimensional example. Similar Section D illustrate the behavior of negative flips the feature space, we create toy example by selecting three classe from ImageNet (Deng et al. and training a ResNet- 18-like models with slight modification such that the penultimate layer's
2009
dimension is changed to 2. The feature level visualization is presented in Figure 11a and 11d We
'In fact if we assume that (bet) and (rel)) have the same shape. we can also do follows: we train (ref) and then (ber) with parameters randomly initialized except the head copies weights from (ref) and is fixed. Nevertheless, we follow BCT'ss formulation since it is more generic.
Labrado retriever" (n02099712) 12), "Weimaraner" (n02092339), and "French bulldog" (n02108915).
16

---

Under review as a conference paper at ICLR 2023
0.6
pmf of I|60|1. de Ato. pmf [|á0]], áo- ,))
0.6
pmt of [| 69 MlA., 2)
[| 69 -1 2(2)))
|uul
single Ens. Ens.
4x
0.2
Ens.
0.2
8x Ens.
16x Ens. 32x Ens.
16x Ens.
32x Ens.
0.0
20
50
0.0
0
10
20
30
40
50
60
70
output logit displacemment
f; norm output logit displacement
(a) Input image: val 101000000585
(a) Input image: va _aal_00000 0.00000005000
0.6
pmf of [| Ao Ni. F) pmf of [[60]]. 2
0.6
pmt of [| 4/ ANAM 2*)
of ||00|| 60 Allu..j(7)
Auu
single Ens. Ens.
2x
4x Ens.
0.2
Ens. 16x Ens. 32X Ens.
0.2
8x Ens.
16x Ens.
32x Ens.
0.0
10
50 output logit displacement
20
0.0
0
10
20
30
40
50
60
70
f. norm output logit displacement
(b) Input image: val val_000010000 00015098.
(b) Input image: va 00015098.
0.6
pmf of |[ 4e .N00, F2 pmf of [[c0l]. 32))
0.6
pmf of ||40]]] 40 All. F)
pmf of ||00]] 60 2
single Ens.
2x
4x Ens.
4x Ens.
0.2
8x Ens.
0.2
8x Ens.
16x Ens. 32X Ens.
16x Ens.
32x Ens.
0.0
0.0
10
20
50
0
10
20
30
40
50
60
70
output logit displacement
fg of output logit displacement
(c) Input image: val_03034619
(c) Input image: va 00034609.
0.6
pmf of [|áo|] áo- .io. 2) pmf of [[á0]], de ,
0.6
pmf of || & Ald.. E)
pmf of [|40)] 40 ((E))
|لسا||
0.4
single Ens.
0.4
single
2x Ens.
4x Ens.
4x Ens.
0.2
8x Ens.
0.2
8x Ens.
16x Ens. 32x Ens.
16x Ens.
32x Ens.
10
20
50
0
10
20
30
40
50
60
70
norm output logit displacement
f norm of output logit displacement
(d) Input image: wa1_00014560.
(d) Input image: val_00014560
Figure 9: l norm histogram of logit displace- ment between two random ensembles. The bin size 0.5. Two random ensembles are the same type (ho- mogeneous ResNet-15 18 vs homogeneous ResNet-18) Aμ = 42 0, E' = 2E> = 2
Figure 10: & norm histogram of logit displace- ment between two random ensembles. The bin size is 0.5. Two random ensembles are different types (homogeneous ResNet- 18 vs homogeneous ResNet- 50). Aμ = 42 + 0, E' = Σi + Σz.
can observe the similar observations as the logit space after the penultimate layer features are aligned with BCT (Shen et al. 2020).
Validations on higher dimensions. We repeat the nigh-dimensional validation in text on the penul- timate layer features, the results are shown in Figure 12 We see that the PMF curve fits the histogram of single models well, implying that feature of these models could indeed follow a Normal distri- bution. We conduct the same experiments above on many more images and the conclusion holds well. If we move to ensembles of m models each, the feature difference follows another normal distribution whose co- vo-vaniance matrix is scaled by factor of m, ie (ens) N (0, 2 Σ). We demonstrate that the rest of histograms are indeed consistent with the estimated PMF of A (ens) I| (dashed lines in Figure 12
17

---

Under review as a conference paper at ICLR 2023
TEn urbee.
ran arbed.
0:2
5.8
ansembre modals
036
nan ena, untoed
0995
0994
nin Kolly emnei
034
(a) Two single models: Test samples flip even if not close to the boundary (long arrows).
(b) Two ensembles of models each (3x): Fewer samples that are far from the boundary flip (shorter arrows).
Embeddings of ensembles and their members: Individual models' embeddings (lighter circles) center around the mode.
432
(d) PMF of feature difference Az.
Visualization of a 3-class 2-dimensional example. (a-c): 2D feature embedding of two single and refer to the ground-truth classes for each sample. Red and green data points refer to old and new model's embeddings. Magenta arrow, blue arrow, gray arrow link negative flip, positive flip, and consistent (either both correct or both wrong) prediction pairs. All dots with black borders are depicting the same image. (d): Estimated probability mass function (PMF) of feature difference between two single models ensembles. The z-- and y-axes denote the 2D feature difference. The heatmap value denotes the estimated probability density. The ensemble's co-variance significantly smaller than the single model. The figure is best viewed in color.
Figure 11:
models ensembles.
pmf of [|AZ]], 4Z -to, x²) pmf of [|AZ]], 4Z -VT0,
pff of [|A2]], AZ - AR0, x))
pmf [|AZ]], AZ -A70.
single
2x Ens.
16x Ens.
Ens.
16x Ens.
4x Ens.
32x Ens.
4x Ens.
32x Ens.
10 f2 narm of output feature difference
1. norm of output feature difference
(a) Input image: val va1_00009585.
(b) Input image: va laa__00015099
pmf of [|AZ]], aZ -tt. 2²) pmf of [|AZ]], 4z -NT0, 2)
pff of ||42], AZ - AR0, X') pmf [|AZ]], AZ - -.
single
Ens. 32x Ens.
single
8x
Ens.
4x Ens.
4x Ens.
32x Ens.
10 1g narm output feature difference
1g norm of output feature difference
(c) Input image: val_00034619.
(d) Input image: val_00014560.
Figure 12: E2 norm histogram of feature difference between two random ensembles. Note that the bin size is 0.1. Two random ensembles are of the same type (ResNet-18 vs ResNet-18). Áμ = 0. E' = Σ, + Eg. We also plot the simulated probability mass function (PMF): the solid line for the norm of simulated normal distribution (Ap,, (Ei Σz)) whose parameters estimated from all available single models; dashed lines extrapolated distribution N (Aμ, 1 [(D: + Σz)). Consistency between the ensembles" histograms and PMFs supports our hypotheses in Section
f2 of
18


Reviews:
[{"review_id": "phCn3MpAiO", "reviewer": "Reviewer_asve", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the task of updating a model by learning a stronger model while ensuring that formerly correctly classified instances will get wrong predictions as seldom as possible, a quantity called a negative flip rate (NFR). The paper starts with a theoretical analysis of using ensembles in this context. It then proposes a new training objective to distill a single model based on an ensemble, with the goal of minimizing NFR. Experiments on image datasets ImageNet and iNaturalist show the effectiveness of ELODI compared to other NFR-reducing methods.", "summary_of_the_review": "The paper addresses an important topic and proposes new effective methods. It also studies the theoretical justification behind the proposed methods. In my opinion, an important aspect has not been covered as an ablation study, justifying the difference of the proposed method from the existing logit matching loss by Hinton et al 2015. There are some shortcomings in the theoretical part, as highlighted above in the list of weaknesses.", "strength_and_weaknesses": "Strengths:\n* Theoretical analysis of the differences of average logits in ensembles;\n* The paper is mostly clearly written, with plenty of insightful figures and tables;\n* The proposed method ELODI is effective in reducing NFR while mostly retaining accuracy.\n\nWeaknesses:\n* At the end of Section 4, it is stated that the logit matching loss (Hinton et al., 2015) is a special case of the proposed LDI term. Therefore, I find it absolutely necessary to do an ablation study to see whether the proposed method improves over using the plain logit matching loss. That is, the paper should have experimental comparisons of $\\xi=0.1$ against $\\xi=0$. Currently there is no such comparison in the paper (nor in the appendices).\n* The theoretical part first starts by making claims about any particular fixed $x$, while considering the training process (or training seed) as a source of randomness, hence the random variable $\\phi^{(i)}(x)$. This nicely gives results stated by Eqs. (1), (2) and (3). However, in Section 3.2 there seems to be a shift to consider $x$ also as a random quantity, because there is a single Gaussian distribution about the whole architecture, e.g. see the last paragraph of page 4 with $\\phi\\sim\\mathcal{N}(\\mu_1,\\Sigma_1)$. This is confusing and should be clarified. If there is now indeed a single Gaussian per architecture, what is the theoretical justification behind that? If there is still only a Gaussian per instance, then how are the covariance matrices estimated for Figure 3?\n* The first variance term $Var(\\hat{\\phi}(x))$ in Eq.(5) does not make sense to me. As minimization is over $\\hat{\\phi}(x)$ and thus this quantity is bound within the expression to be minimized, then what is the random variable that the variance operator is applied to?\n\nMinor weaknesses:\n* From the introduction it is not quite clear whether the term positive-congruent training is novel or not;\n* The text about Figure 2 suggests that particularly the negative flips with large cross-model logit displacement are reduced by ensembles, highlighting the percentage of 90.7% in the red box. However, it would then also be important to provide the respective percentage outside the red box. Is it significantly smaller?\n* The last rows of page 3 have '$\\phi^{(i)}$ and $\\phi^{(i)}$' instead of '$\\phi^{(i)}$ and $\\psi^{(i)}$';\n* Sometimes $\\phi$ is in bold and sometimes not (e.g. in Eqs (1) and (3)), not being consistent.\n* The number of current Figure 6 should be Figure 5 instead. The text already refers to it as Figure 5.\n* In Figure 6b (that is 5b), it has not been explained what ensemble size 0 means.\n* It has not explained clearly enough what the difference between online and offline distillation is.\n* The current theory and the discussion around it say nothing about decreasing the difference between $\\mu_1$ and $\\mu_2$ which could also be a separate goal. In principle, this might possibly be achieved by changing the training objective for the members of the ensemble. Maybe it is a bad idea, but I feel this should be at least mentioned.\n\nSuggestions:\n* The acronyms LDI and ELODI are both about logit difference inhibition: wouldn't it be better to use the same suffix, such as LDI and ELDI, or LODI and ELODI?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "VusKDt5mn4E", "reviewer": "Reviewer_uaPa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors repurpose the well-known knowledge distillation paradigm, which is typically used for improving model accuracy, for the task of transferring the capability of reducing negative flip rate (NFR) from an ensemble to a single model. To this end, the authors devise a method called Ensemble Logit Difference Inhibition (ELODI). ELODI leverages the advantages of deep ensembles for NFR reduction at the running cost of a single model by (1) training deep networks using a Logit Difference Inhibition (LDI) loss with respect to a reference ensemble and then (2) using the resulting single model to perform inference.", "summary_of_the_review": "This paper leveraged deep ensembles for NFR reduction and showed that they can be distilled into single models by means of a Logit Difference Inhibition (LDI) objective. Despite the methodologically incremental contribution, the proposed ensemble method is capable of reducing NFR while maintaining accuracy and can be distilled to a readily deployable single model which exhibits much lower inference time compared to the deep ensemble. The latter in particular makes this work applicable to large-scale real-world systems where online inference and low latency are paramount. Overall, although there are certain weak points that need to be addressed (listed in the “Weaknesses” part of this review), the strengths of this work certainly outweigh its weaknesses. Therefore, I consider this paper to be a fairly good fit for ICLR and I recommend that it is accepted.", "strength_and_weaknesses": "Strengths:\n\n* ELODI trains a deep homogeneous ensemble that can achieve Positive-Congruent Training (PC-Training) performance and be distilled to a single model (which does not necessarily need to be a specific legacy model). This allows the proposed ELODI method to reduce NFR while retaining the accuracy gain introduced by the new model.\n\n* The authors introduce Logit Difference Inhibition (LDI), a knowledge distillation objective that penalizes significant differences between the logits of the reference homogeneous ensemble and the single student model. Such an objective allows for effective distillation of the reference ensemble to a single student model.\n\n* ELODI exhibits efficient inference as it does not require the homogeneous reference ensemble to be evaluated at inference time but rather only the single student model.\n\n* The choice of homogeneous ensembles as reference ensembles is empirically well supported by the findings summarized in Figure 4. Moreover, the hypothesis of reducing logit displacement magnitude (on which the introduced LDI objective relies) is validated empirically in Section 3.2.\n\n* The authors have conducted extensive experiments on multiple image classification benchmarks to assess the effectiveness and efficiency of ELODI in comparison with several PC-training and ensemble methods. The experimental results demonstrate that ELODI performs positive congruent training by reducing NFR with large logit displacement and reducing the variance of the ensemble’s logits. It is also empirically demonstrated that this behavior can be transferred to a single model which considerably reduces the inference time. For instance, this allows ELODI to achieve comparable ER-NFR performance to that of the ensemble paragon [Yan et al., 2021] at the inference cost of a single model.\n\n* The authors have conducted a plethora of additional experimental studies to assess the effect of various aspects including:\n\n    (1) model update settings (including fine-tuning on other datasets, transitivity of NFR reduction induced in chain of model updates, and ELODI’s applicability to updates across dissimilar architectures);\n\n    (2) design choices (including the effects of using homogeneous instead of heterogeneous ensembles for guidance, changes in the guiding ensemble’s architecture, and different choices of distillation loss functions);\n\n    (3) exploratory parameter analysis assessing the effects of the loss weight $\\alpha$, the size of the reference ensemble as well as the manner in which the logits are inferred (offline compared to online).\n\n-------------------------------------------------------------------------------------------------\n\nWeaknesses:\n\n* At the end of the introductory paragraph of Section 3, the authors claim that “the averaging operation in deep ensembles is also often performed in the logit space (Lakshminarayanan et al., 2017), which makes it interesting for our study.” Nevertheless, I am not confident how often this is the case, namely because in classification settings voting schemes are typically used upon applying the activation function of choice (e.g., softmax) to the logits estimated by each member model rather than averaging the logits. I would encourage the authors to further clarify their motivation behind the choice of the averaging aggregation scheme in the logit space, or consider revising the aforementioned statement in the paper.\n\n* The assumption that the logits outputted by the base members of a homogeneous ensemble follow a normal distribution is a fair assumption to make. This is also empirically supported in Section 3.2. However, [1] suggests that, for this to hold, the base networks need to be trained in a sufficient number of epochs. Therefore, including a brief discussion in Section 3.2 on the relationship between the number of training epochs and the members’ logit distributions would be quite insightful. In addition, I believe that [1] is relevant to this paperr as it discusses logit distributions of deep neural networks and thus I would suggest that the authors consider including it as a part of their related work.\n\n    [1] Seguin, L., Ndirango, A., Mishra, N., Chung, S., & Lee, T. (2021). Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks. arXiv preprint arXiv:2108.12001.\n\n* In Section 3.2, while discussing Figure 3a, the authors state the following: “If we move to the ensemble case, the logit displacement follows another normal distribution with scaled co-variance matrix”. However, according to Figure 3a, apart from the scaled co-variance matrix, the mean is shifted. This should also be included in the same discussion.\n\n-------------------------------------------------------------------------------------------------\n\nMinor weaknesses:\nThere are also certain grammatical and typographical errors and remarks that require attention. Some of them are summarized as follows:\n- In the “Contributions” paragraph on page 2, “the” should be removed from the phrase “obtain the their performance for PC-training”. Later in the same paragraph, the hyphen should be removed from “with re- spect to”.\n- Section 3.1, at the end of page 3: In the phrase “Each $\\phi(x)$ or $\\phi(x)$”, I believe that one of the $\\phi(x)$’s should be corrected.\n- In the second paragraph of Section 4.1, replace “an model pair” with “a model pair”.\n- Section 4.2: In the sentence right after Eq. (6), it is mentioned that “difference below $\\xi$ is tolerated”; however, according to Eq. (6) it seems that a difference below $\\xi$ is truncated, i.e. not tolerated. Later in that same paragraph, “when number of classes are extremely largencated” should be corrected to “when the number of classes is extremely large”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "K7HS1jjN4K", "reviewer": "Reviewer_V2YQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle the problem of updating existing models in production-scale\nsystems while performing Positive-Congruent Training (minimize negative flip\nrate and error rate simultaneously). To do so, they make two key observations:\n\n- Previous studies have demonstrated that ensembles can reduce NFR without\n  negative accuracy impact.\n- Using ensembles may not be a viable strategy in real applications due to\n  computational costs.\n\nBased on these observations, the authors propose ELODI, a novel model\ndistillation method that seeks to achieve PC-Training performance of ensembles\nusing a single model. First, they analyze how the distribution of logit spaces\nof both homogeneous and heterogenous ensembles behave and conclude that, as\nensembles become larger, NFR consistently decreases. This is further validated\nwith experiments. Next, they propose a novel, margin-based distillation loss\nfunction, namely LDI, that attempts to mimic the variance reduction phenomenon\npresent in ensembles. The authors perform extensive experiments and conclude\nthat ELODI has similar performance (NFR and ER) to an ensemble method with\nreduced inference costs.", "summary_of_the_review": "The authors propose ELODI, a distillation technique that reaps both the\nbenefits of ensembles to reduce NFR and ER and distillation to mimic the\nbehavior of the said ensemble using a single model. Although mainly limited to\nhomogeneous ensembles, ELODI has the potential to positively affect deep\nlearning model deployment in production.", "strength_and_weaknesses": "*Strengths:\n\n- The paper is well-motivated, since the problem the authors present frequently\n  occurs in production systems.\n- The problem of negative flips in the context of ensembles is clearly\n  explained in Section 3 and further details the experiments in (Yan et al.\n  2021).\n- The experimental section compares the results of ELODI with several other\n  state-of-the-art techniques.\n\n*Weaknesses:\n\n- ELODI works mainly for homogeneous ensembles. This goes against one of the\n  biggest advantages of using ensembles, which is to combine models with\n  different architectures.\n- Training a model using ELODI requires one to first train an ensemble and, as\n  the authors themselves point out, the number of models in the ensemble\n  drastically impacts the quality of the distilled model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3ov0UrDZoJJ", "reviewer": "Reviewer_38Tg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method in the area of Positive-Congruent Training, by distilling a homogeneous ensemble to a single student model. The authors carefully analyze the properties of logic displacement magnitude in ensembles, and present a method called Ensemble Logit Difference Inhibition (ELODI). The proposed approach is validated on two standard image classification datasets.", "summary_of_the_review": "This paper introduces a new knowledge distillation method with the goal to reducing NFR. Although the analysis and the methods are well presented, I feel the novelty of the approach is a bit limited since the key part is on modifying the loss function used in knowledge distillation; and the part on the distribution of ensemble distributions is only empirically without any theoretical supports.\n\nThat being said, I think if the authors can provide a better explanation on the benefits of the proposed loss function, and provide more theoretical discussion on the prediction distributions of ensembles, the paper can be much improved. \n\nI mark the paper as marginally below the threshold, but also note I am not familiar with this specific application area. So I am willing for further discussions. ", "strength_and_weaknesses": "Strength\n\n- The analysis on the properties of the distribution of ensemble predictions is interesting, and the simulation studies are clear and supportive of the findings. \n- The algorithm is well presented with detailed follow up discussions. \n\nWeakness\n- I am not super clear on the motivation of the proposed method. The loss function of the distillation algorithm is a key part but it's basically just cross entropy plus a term to panelize l_2 difference in the logit. Despite being used for model updating, are there any other benefits can be discussed, such as smoothing the landscape of the student model? It seems to me using and additional LDI term could have some alternative motivations, and the use case seems a bit unnatural to me. \n- Although the discussion on the properties of ensemble prediction distributions are empirically correct, can the authors provide more discussions in this area? For example, where is the randomness coming from? Initialization, randomness in training algorithms, etc? For trees, there have been some statistical results (see for example https://arxiv.org/pdf/1510.04342.pdf, https://www.jmlr.org/papers/volume17/14-168/14-168.pdf?ref=https://githubhelp.com). I wonder whether some primitive results can be discussed here.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject