Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the task of updating a model by learning a stronger model while ensuring that formerly correctly classified instances will get wrong predictions as seldom as possible, a quantity called a negative flip rate (NFR). The paper starts with a theoretical analysis of using ensembles in this context. It then proposes a new training objective to distill a single model based on an ensemble, with the goal of minimizing NFR. Experiments on image datasets ImageNet and iNaturalist show the effectiveness of ELODI compared to other NFR-reducing methods.
strength_and_weaknesses: Strengths:
* Theoretical analysis of the differences of average logits in ensembles;
* The paper is mostly clearly written, with plenty of insightful figures and tables;
* The proposed method ELODI is effective in reducing NFR while mostly retaining accuracy.

Weaknesses:
* At the end of Section 4, it is stated that the logit matching loss (Hinton et al., 2015) is a special case of the proposed LDI term. Therefore, I find it absolutely necessary to do an ablation study to see whether the proposed method improves over using the plain logit matching loss. That is, the paper should have experimental comparisons of $\xi=0.1$ against $\xi=0$. Currently there is no such comparison in the paper (nor in the appendices).
* The theoretical part first starts by making claims about any particular fixed $x$, while considering the training process (or training seed) as a source of randomness, hence the random variable $\phi^{(i)}(x)$. This nicely gives results stated by Eqs. (1), (2) and (3). However, in Section 3.2 there seems to be a shift to consider $x$ also as a random quantity, because there is a single Gaussian distribution about the whole architecture, e.g. see the last paragraph of page 4 with $\phi\sim\mathcal{N}(\mu_1,\Sigma_1)$. This is confusing and should be clarified. If there is now indeed a single Gaussian per architecture, what is the theoretical justification behind that? If there is still only a Gaussian per instance, then how are the covariance matrices estimated for Figure 3?
* The first variance term $Var(\hat{\phi}(x))$ in Eq.(5) does not make sense to me. As minimization is over $\hat{\phi}(x)$ and thus this quantity is bound within the expression to be minimized, then what is the random variable that the variance operator is applied to?

Minor weaknesses:
* From the introduction it is not quite clear whether the term positive-congruent training is novel or not;
* The text about Figure 2 suggests that particularly the negative flips with large cross-model logit displacement are reduced by ensembles, highlighting the percentage of 90.7% in the red box. However, it would then also be important to provide the respective percentage outside the red box. Is it significantly smaller?
* The last rows of page 3 have '$\phi^{(i)}$ and $\phi^{(i)}$' instead of '$\phi^{(i)}$ and $\psi^{(i)}$';
* Sometimes $\phi$ is in bold and sometimes not (e.g. in Eqs (1) and (3)), not being consistent.
* The number of current Figure 6 should be Figure 5 instead. The text already refers to it as Figure 5.
* In Figure 6b (that is 5b), it has not been explained what ensemble size 0 means.
* It has not explained clearly enough what the difference between online and offline distillation is.
* The current theory and the discussion around it say nothing about decreasing the difference between $\mu_1$ and $\mu_2$ which could also be a separate goal. In principle, this might possibly be achieved by changing the training objective for the members of the ensemble. Maybe it is a bad idea, but I feel this should be at least mentioned.

Suggestions:
* The acronyms LDI and ELODI are both about logit difference inhibition: wouldn't it be better to use the same suffix, such as LDI and ELDI, or LODI and ELODI?
clarity,_quality,_novelty_and_reproducibility: The paper is sufficiently novel and reproducible, and also mostly clear, except for the shortcomings highlighted above.
summary_of_the_review: The paper addresses an important topic and proposes new effective methods. It also studies the theoretical justification behind the proposed methods. In my opinion, an important aspect has not been covered as an ablation study, justifying the difference of the proposed method from the existing logit matching loss by Hinton et al 2015. There are some shortcomings in the theoretical part, as highlighted above in the list of weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method in the area of Positive-Congruent Training, by distilling a homogeneous ensemble to a single student model. The authors carefully analyze the properties of logic displacement magnitude in ensembles, and present a method called Ensemble Logit Difference Inhibition (ELODI). The proposed approach is validated on two standard image classification datasets.
strength_and_weaknesses: Strength

- The analysis on the properties of the distribution of ensemble predictions is interesting, and the simulation studies are clear and supportive of the findings. 
- The algorithm is well presented with detailed follow up discussions. 

Weakness
- I am not super clear on the motivation of the proposed method. The loss function of the distillation algorithm is a key part but it's basically just cross entropy plus a term to panelize l_2 difference in the logit. Despite being used for model updating, are there any other benefits can be discussed, such as smoothing the landscape of the student model? It seems to me using and additional LDI term could have some alternative motivations, and the use case seems a bit unnatural to me. 
- Although the discussion on the properties of ensemble prediction distributions are empirically correct, can the authors provide more discussions in this area? For example, where is the randomness coming from? Initialization, randomness in training algorithms, etc? For trees, there have been some statistical results (see for example https://arxiv.org/pdf/1510.04342.pdf, https://www.jmlr.org/papers/volume17/14-168/14-168.pdf?ref=https://githubhelp.com). I wonder whether some primitive results can be discussed here.  
clarity,_quality,_novelty_and_reproducibility: Overall the paper is written clearly. I feel the novelty of the approach is a bit limited since the key part is on modifying the loss function used in knowledge distillation; the part on the distribution of ensemble distributions is only empirically without any theoretical supports.

Some minor things: 

- Figure 2 the blue color is not accurate when laid over by the orange. 
- Better to write down exact formulation of L_CE in page 6.
summary_of_the_review: This paper introduces a new knowledge distillation method with the goal to reducing NFR. Although the analysis and the methods are well presented, I feel the novelty of the approach is a bit limited since the key part is on modifying the loss function used in knowledge distillation; and the part on the distribution of ensemble distributions is only empirically without any theoretical supports.

That being said, I think if the authors can provide a better explanation on the benefits of the proposed loss function, and provide more theoretical discussion on the prediction distributions of ensembles, the paper can be much improved. 

I mark the paper as marginally below the threshold, but also note I am not familiar with this specific application area. So I am willing for further discussions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of updating existing models in production-scale
systems while performing Positive-Congruent Training (minimize negative flip
rate and error rate simultaneously). To do so, they make two key observations:

- Previous studies have demonstrated that ensembles can reduce NFR without
  negative accuracy impact.
- Using ensembles may not be a viable strategy in real applications due to
  computational costs.

Based on these observations, the authors propose ELODI, a novel model
distillation method that seeks to achieve PC-Training performance of ensembles
using a single model. First, they analyze how the distribution of logit spaces
of both homogeneous and heterogenous ensembles behave and conclude that, as
ensembles become larger, NFR consistently decreases. This is further validated
with experiments. Next, they propose a novel, margin-based distillation loss
function, namely LDI, that attempts to mimic the variance reduction phenomenon
present in ensembles. The authors perform extensive experiments and conclude
that ELODI has similar performance (NFR and ER) to an ensemble method with
reduced inference costs.
strength_and_weaknesses: *Strengths:

- The paper is well-motivated, since the problem the authors present frequently
  occurs in production systems.
- The problem of negative flips in the context of ensembles is clearly
  explained in Section 3 and further details the experiments in (Yan et al.
  2021).
- The experimental section compares the results of ELODI with several other
  state-of-the-art techniques.

*Weaknesses:

- ELODI works mainly for homogeneous ensembles. This goes against one of the
  biggest advantages of using ensembles, which is to combine models with
  different architectures.
- Training a model using ELODI requires one to first train an ensemble and, as
  the authors themselves point out, the number of models in the ensemble
  drastically impacts the quality of the distilled model.
clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and easy to follow.
- There are minor typos throughout the text (e.g., "res- pect").
- Figure 1 is in page 2, but is referenced for the first time in page 5.
- Figures 3 and 4 are somewhat hard to understand, since there is too much
  content packed into very little space.
- Algorithm 1 seems unnecessary, given that the procedure is explained
  throughout the text.
- The novelty of the paper is the distillation loss function targeted at
  approximating the logits between a new model and an ensemble. The authors
  compare the proposed loss with several baselines and conclude that it gives
  better results.
- The paper seems to be easy to reproduce (hopefully the authors will publish
  their source code).
summary_of_the_review: The authors propose ELODI, a distillation technique that reaps both the
benefits of ensembles to reduce NFR and ER and distillation to mimic the
behavior of the said ensemble using a single model. Although mainly limited to
homogeneous ensembles, ELODI has the potential to positively affect deep
learning model deployment in production.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors repurpose the well-known knowledge distillation paradigm, which is typically used for improving model accuracy, for the task of transferring the capability of reducing negative flip rate (NFR) from an ensemble to a single model. To this end, the authors devise a method called Ensemble Logit Difference Inhibition (ELODI). ELODI leverages the advantages of deep ensembles for NFR reduction at the running cost of a single model by (1) training deep networks using a Logit Difference Inhibition (LDI) loss with respect to a reference ensemble and then (2) using the resulting single model to perform inference.
strength_and_weaknesses: Strengths:

* ELODI trains a deep homogeneous ensemble that can achieve Positive-Congruent Training (PC-Training) performance and be distilled to a single model (which does not necessarily need to be a specific legacy model). This allows the proposed ELODI method to reduce NFR while retaining the accuracy gain introduced by the new model.

* The authors introduce Logit Difference Inhibition (LDI), a knowledge distillation objective that penalizes significant differences between the logits of the reference homogeneous ensemble and the single student model. Such an objective allows for effective distillation of the reference ensemble to a single student model.

* ELODI exhibits efficient inference as it does not require the homogeneous reference ensemble to be evaluated at inference time but rather only the single student model.

* The choice of homogeneous ensembles as reference ensembles is empirically well supported by the findings summarized in Figure 4. Moreover, the hypothesis of reducing logit displacement magnitude (on which the introduced LDI objective relies) is validated empirically in Section 3.2.

* The authors have conducted extensive experiments on multiple image classification benchmarks to assess the effectiveness and efficiency of ELODI in comparison with several PC-training and ensemble methods. The experimental results demonstrate that ELODI performs positive congruent training by reducing NFR with large logit displacement and reducing the variance of the ensemble’s logits. It is also empirically demonstrated that this behavior can be transferred to a single model which considerably reduces the inference time. For instance, this allows ELODI to achieve comparable ER-NFR performance to that of the ensemble paragon [Yan et al., 2021] at the inference cost of a single model.

* The authors have conducted a plethora of additional experimental studies to assess the effect of various aspects including:

    (1) model update settings (including fine-tuning on other datasets, transitivity of NFR reduction induced in chain of model updates, and ELODI’s applicability to updates across dissimilar architectures);

    (2) design choices (including the effects of using homogeneous instead of heterogeneous ensembles for guidance, changes in the guiding ensemble’s architecture, and different choices of distillation loss functions);

    (3) exploratory parameter analysis assessing the effects of the loss weight $\alpha$, the size of the reference ensemble as well as the manner in which the logits are inferred (offline compared to online).

-------------------------------------------------------------------------------------------------

Weaknesses:

* At the end of the introductory paragraph of Section 3, the authors claim that “the averaging operation in deep ensembles is also often performed in the logit space (Lakshminarayanan et al., 2017), which makes it interesting for our study.” Nevertheless, I am not confident how often this is the case, namely because in classification settings voting schemes are typically used upon applying the activation function of choice (e.g., softmax) to the logits estimated by each member model rather than averaging the logits. I would encourage the authors to further clarify their motivation behind the choice of the averaging aggregation scheme in the logit space, or consider revising the aforementioned statement in the paper.

* The assumption that the logits outputted by the base members of a homogeneous ensemble follow a normal distribution is a fair assumption to make. This is also empirically supported in Section 3.2. However, [1] suggests that, for this to hold, the base networks need to be trained in a sufficient number of epochs. Therefore, including a brief discussion in Section 3.2 on the relationship between the number of training epochs and the members’ logit distributions would be quite insightful. In addition, I believe that [1] is relevant to this paperr as it discusses logit distributions of deep neural networks and thus I would suggest that the authors consider including it as a part of their related work.

    [1] Seguin, L., Ndirango, A., Mishra, N., Chung, S., & Lee, T. (2021). Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks. arXiv preprint arXiv:2108.12001.

* In Section 3.2, while discussing Figure 3a, the authors state the following: “If we move to the ensemble case, the logit displacement follows another normal distribution with scaled co-variance matrix”. However, according to Figure 3a, apart from the scaled co-variance matrix, the mean is shifted. This should also be included in the same discussion.

-------------------------------------------------------------------------------------------------

Minor weaknesses:
There are also certain grammatical and typographical errors and remarks that require attention. Some of them are summarized as follows:
- In the “Contributions” paragraph on page 2, “the” should be removed from the phrase “obtain the their performance for PC-training”. Later in the same paragraph, the hyphen should be removed from “with re- spect to”.
- Section 3.1, at the end of page 3: In the phrase “Each $\phi(x)$ or $\phi(x)$”, I believe that one of the $\phi(x)$’s should be corrected.
- In the second paragraph of Section 4.1, replace “an model pair” with “a model pair”.
- Section 4.2: In the sentence right after Eq. (6), it is mentioned that “difference below $\xi$ is tolerated”; however, according to Eq. (6) it seems that a difference below $\xi$ is truncated, i.e. not tolerated. Later in that same paragraph, “when number of classes are extremely largencated” should be corrected to “when the number of classes is extremely large”.

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well written, organized, and technically detailed. The used notation is clear and consistent throughout the paper.

Quality:
The design and justifications for the proposed Ensemble Logit Difference Inhibition method are technically sound. The observations made regarding the reduction in NFR and inference cost introduced by the proposed method as well as its capability to maintain image classification accuracy gains are empirically well supported. Overall, the work seems to be well developed and mature in terms of quality.

Novelty:
From a methodological perspective, the contribution of this work can be considered incremental since it is built upon the well-established knowledge distillation framework that utilizes the proposed simple yet well-crafted Logit Difference Inhibition (LDI) objective. Nevertheless, the authors provide an extensive analysis on the role of deep ensembles in a knowledge distillation scheme which sheds light on their capability of reducing NFR and obtaining performance for PC-training with a single model, at the running cost of a single model. To the best of my knowledge, this is the first work to conduct such an extensive analysis regarding the aforementioned points.

Reproducibility:
The experiments were conducted on widely-used image classification benchmark datasets which are publicly available. On the other hand, the code for the proposed ELODI method is not made available by the authors in the present anonymized version, however, the architecture of the method is clearly explained in the paper, thus it is safe to say that one should be able to implement ELODI by following its detailed description in Section 4.
summary_of_the_review: This paper leveraged deep ensembles for NFR reduction and showed that they can be distilled into single models by means of a Logit Difference Inhibition (LDI) objective. Despite the methodologically incremental contribution, the proposed ensemble method is capable of reducing NFR while maintaining accuracy and can be distilled to a readily deployable single model which exhibits much lower inference time compared to the deep ensemble. The latter in particular makes this work applicable to large-scale real-world systems where online inference and low latency are paramount. Overall, although there are certain weak points that need to be addressed (listed in the “Weaknesses” part of this review), the strengths of this work certainly outweigh its weaknesses. Therefore, I consider this paper to be a fairly good fit for ICLR and I recommend that it is accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the task of updating a model in production with a better performing model. One practical aspect is in such situations is to minimize the Negative Flip Rate (NFR): the rate at which the new model makes mistakes on examples that were previously correctly classified.  

The paper presents a rather strati-froward combination of prior work: using ensembles to reduct the NFR, and using knowledge distillation to reduct the memory and computational footprint of ensembles. As such, the papers' contribution is of limited significance. A second contribution of the paper is to propose a novel loss for the knowledge distillation procedure that generalized the original LDI loss. However, the authors failed to show that he new generalization is useful. It introduces extra complexity (an extra hyperparameter that needs  to be set) but according to the results provided by the authors during the discussion phase there is no significant performance benefit to be gained. 
justification_for_why_not_higher_score: The contributions of the paper are not significant enough to warrant publication.
Decision: Reject