Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes a method for transferring knowledge from a complex, high-performing model to a small/lower-performing one. The approach is based on reweighting the train examples according to the ratio of confidences of the complex and simple models, and then retraining the simple model on the reweighed dataset. 

Pros:
* Some interesting ideas motivating the approach

Cons:
* An important error in the main result and various other issues (imprecise statements, lack of details) raise doubts on the theoretical contribution of the paper.
* Limited novelty. Contribution boils down to importance reweighting via a ratio of probabilities.
* Limited awareness of related work.
* Experimental framework lacks meaningful baselines and important details .

Based on the points above, I have to recommend rejection of this paper to ICLR. 

Detailed Comments:
* The claim that per-sample hardness reweighting is novel to this work is a bit of a stretch. Most reweighting-based methods for distillation use some form of this idea, though how each method estimates “hardness” varies, ranging from raw confidence scores of the teacher model (e.g., classic distillation), to scaled versions of it, to a ratio of this and the student’s score (this work).
* More generally, there’s quite a few very relevant related works which are not cited here (see list below for a non-exhaustive list).
* I believe the inequality used to prove Lemma 3.2 is not true in general. It’s easy to come up with counterexamples for which w>=1 and x>1 yet log(wx) <= w log(x) does not hold. For example, for w=2, any x in (1,2) will not satisfy this. In general, this inequality is only true if x >= w^{1/(w-1)}. I would be prepared to pass this off as a minor error, were it not for the fact that the w and x in the proof of Lemma 3.2 could quite conceivably be outside this range, i.e., rendering the inequality (and therefore the bound) invalid.
* There are various other technical aspects of the paper that are either non-rigorous or lack details. For example, Lemma 3.1 requires more information. For which y does w>= hold? Is it just for the true class y*? The proof is equally obscure, and seems to use the same problematic inequality of Lemma 3.2 (for the cross-entropy loss). 
* The statement of Lemma 3.2 is confusing. The last sentence should read loss smaller than *that* of theta*. Also, I don’t see the optimality of theta* being used anywhere. The statement could very well be phrased in terms of any two models, one of which has a lower loss than the other. 
* Assuming the complex “teacher” classifiers are very accurate (which is assumed several times throughout the paper), re-weighting training examples by max(1, p_c / p_theta) does not provide too much information beyond the ground truth labels. This is a core problem in knowledge distillation, already identified as early as Hinton et al (2015), and which is often alleviated by some form of temperature annealing or by operating on pre-softmax logits rather than final probabilities. That doesn’t seem to be the case here, so I’m quite puzzled by the “intuitive justification” provided in this work.
* The discussion in the first paragraph of pg 4 is a well known dichotomy (namely, up-weighting easy or hard examples) in the distillation/boosting/instance-reweighting literature. This discussion would really benefit from awareness of previous work discussing this duality (e.g., (Bengio et al 2009; Ren et al 2019)
* Where and how is the ordering of the “graded” classifiers used? In step (3) of Algo 1 all \zeta’s seem to be weighted equally, so I’m missing why the classifiers need to be delta-graded.
* I find it quite surprising that in almost half of the datasets in Table 1, distillation yields the exact same performance as just training the Simple Model, and sometimes even degrades the performance. Section 4.1 points out that this is actually “an equivalent model to Distillation”, but no further explanation is given, so it is hard to conjecture what might be the problem. Futhermore, as the authors point out in the introduction, Distillation alla Hinton 2015 usually assumes both teacher and student models are soft predictors (e.g., neural nets), but the models used in the UCI experiments trees and SVMs, so I wonder how exactly they did this. 
* For problems involving distillation into simple interpretable models (like trees), such as the UCI experiments, there are many reasonably strong baselines from previous work that could have been compared against (e.g. Frosst and Hinton 2017), which could have provided a more meaningful evaluation.


[1] Learning to Reweight Examples for Robust Deep Learning, Ren et al. ICML 2018
[2] Distilling a Neural Network Into a Soft Decision Tree, Frosst and Hinton, 2017
[3] Born-Again Neural Networks, Furlanello et al., ICML 2018
[4] Fidelity-Weighted Learning, Deghani et al., ICLR 2018
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper presents a simple yet effective trick to reweigh the samples fed into a simple model to improve the performance. The new weights are functions of both the simple model and another complex (and strong) model. Some theoretical and intuitive explanations are provided to support the main claim of the paper. 
The main weakness of the paper is its presentation and this is apparent from the abstract of the paper and afterwards. Referring to a prior work (Dhuranhar, 2018b) in the abstract which might be known to the reader is not a good idea. The connection of lemma 3.1 to the rest of the paper is not clear. Similarly, it's not readily clear how lemma 3.2 leads to algorithm 1. The introduction of the graded classifiers was very sudden. It's difficult to pinpoint how we replaced the complex model with the graded classifiers. To me, the graded classifiers were and incremental work and make the main point of paper (section 3.1 and 3.2) could have been emphasized better. 
What would be the effect of changing the number of graded classifiers? Some theoretical explanations or empirical results could be beneficial. 
The reweighting scheme looks very related to importance sampling ratios in off policy evaluation in reinforcement learning and counterfactual analysis. Connecting the dots could be interesting. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper is about re-weighting training sets in such a way that simple and interpretable models like trees or small networks can mimic the performance of potentially very complex learning architectures. The main difference to some existing approaches of this kind is the universal applicability to many complex-simple model combinations (and not only layered networks). Technically, this paper is an extension and formalization of some ideas proposed in (Dhurandhar 2018). The main methodological contribution is the formal justification of the proposed weighting scheme for samples which utilizes the ratio of conditional probabilities p_complex(y|x)/p_simple(y|x) in the Lemma 3.1 and 3.2. The practical algorithm described in the paper is based on the concept of delta-graded classifiers which basically defines a nested set of classifiers of increasing accuracy -- such as, for instance, simple classifiers trained on intermediate layers of a deep network. The second important input argument is the learning algorithm for the simple model, which is trained on the re-weighted samples. The motivation for this algorithmic procedure is somewhat "hand-waving", but rather intuitive. Experiments for different benchmark datasets and different complex-simple model combinations nicely demonstrate that this method is indeed quite useful in practice. In summary, I think that this work addresses a highly relevant problem and provides a relatively simple method that might be very useful in many applications. A potential weakness could be the gap between the formal treatment of the cross-entropy loss in Lemma 3.2. and the rather "ad hoc" practical algorithm that crucially depends on two tuning parameters. Nevertheless, in my opinion, this work provides some interesting ideas that have the potential to trigger future research in this area.        
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a means of improving the predictions of a "simple" (low-capacity) model. The idea is to take the predictions of a "complex" (high-capacity) model, and weight the loss in the simple model based on the ratio of complex to simple models' predictions. Intuitively, this seeks to focus on instances which the complex model can fit, but the unweighted simple model cannot. Experiments show the proposed method to have some benefits over existing approaches.

The application of importance-weighting to the "model distillation" problem is interesting, and the paper gives a reasonable intuition for why this approach might work. One general comment is that in contrasting their approach to a number of existing approaches, they note that several of them are typically employed with fairly complex simple models (e.g., neural networks). This may be true, but it was not clear that any of them require this to be the case. Surely they can also be used with simple models as ones you consider in the paper? In this case, I would've liked more elucidation as to why the proposed method can be expected to offer superior performance.

The theoretical justification of the approach is provided by means of Lemmas 3.1 and 3.2, for which I have some comments:
- Lemma 3.1: the notation here is a bit imprecise. In general, a loss for example (x, y) takes in a true label y and predicted score z(x). You refer to the loss of a probabilistic prediction p(y | x), but do not refer to the actual label y itself. From the proof, it is implicit that you are considering y to be binary, and the use of a margin loss. This is ok, but for the hinge loss one doesn't use a probability estimate p(y | x) as input to the loss, but rather, a real-valued score.

I think the result itself could be proven by noting that if φ is non-increasing and q < p, then φ(p)/φ(q) <= 1 < p/q.

- Lemma 3.2: the result is interesting, but it seems that for practical purposes you are only using the first term, since it is the only quantity that depends on θ. Since max(1, .) >= 1 and -log pθ(y | x) >= 0, it seems one could trivially bound the LHS by the first term since -log pθ(y | x) <= max(1, .) * -log pθ(y | x)? Would this not suffice for the purposes of justifying your method? It should also be noted here how the subsequent requirement that the weights be capped (so as to prevent outliers) fits into the analysis.

In describing the method itself, the authors introduce a notion of δ-graded subsets. I found this to be a bit difficult to parse, and it was not clear why this notion was needed. It does not seem to feature in the description of Algorithm 1, nor the subsequent discussion. On the other hand, I felt that the meaning and need for parameters β and γ ought to have been discussed more prominently upfront.

The experiments show favourable performance of the proposed method over baselines, including the distillation approach of Hinton. The datasets are mostly small-scale, but this is in keeping with the goal of the paper, viz. addressing scenarios where simple models may be desired. Per earlier comments, I did not have a deep sense of what additional information the proposed method exploits so as to improve performance. I gather that the weighting including the predictions of the simple model is one difference; it might have been nice to give a sense of what fraction of points this amplifies or suppresses, compared to just using the predictions of the complex model.

There is a nice illustration in Fig 1 (right) as to the class-labels of the training samples assigned various weights. This shows that points with low weight tend to have low agreement with their neighbours' labels. One question is how using a nearest neighbour probability estimate itself (i.e., using this as the "complex" model) would fare.

Minor comments:
- the title is a bit confusing. It is not clear what "its" refers to. You are leveraging the predictions of a complex model to enhance those of a simple model?
- the text has a number of long sentences that could benefit from rewriting or splitting.
- proof of Lemma 3.2, use \cdot not *.
- proof of Lemma 3.2, θ* should use superscript.
- Fig 1, the caption is overlong. Most of this should be in the text.
- Fig 1, use crisper fonts for the text.
review_title: Official Blind Review #1

Meta-review: The authors propose a sample reweighting scheme that helps to learn a simple model with similar performance as a more complex one. The authors contained critical errors in their original submission and the paper seems to lack in terms of originality and novelty of the proposed method.
Decision: Reject