Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper propose to leverage the frozen classifier from the teacher model for training better representations for the student model. By further combining with another feature matching loss, the proposed methods outperforms previous methods (such as KD, AT, CRD) on many benchmarks.

Pros:
- The proposed method is very simple, and effective.
- Authors demonstrate the effectiveness of this method, on CIFAR-10, CIFAR-100, ImageNet, as well as transferring to STL-10. They also apply real-to-binary distillation, which is interesting.
- The paper is well-written and very easy to follow.

Overall, I did not see much disadvantages of this proposed method. I am a bit surprised that such simple method has been ignored before. But some parts that can be further enhanced is:
- Limitation: despite its simplicity, this method, in its current format, assumes that the teacher classifier is available, which is not always true in may applications. Besides, it's not clear that whether this method is still effective, if the teacher model is not a standard classification model, e.g., if you want to compress a regression model, whether you can still use teacher's frozen regressor to guide student representation is still unknown. So I encourage the authors to add a section to discuss it.

Overall, I like this simple and surprisingly effective method.

=== update ===

The provided response is reasonable and helps address some of my concerns. I would keep my rating as acceptance.
review_title: Simple and effective methods

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a novel method for knowledge distillation. The idea is to utilize the teacher’s pre-trained classifier to train the student’s penultimate layer feature by adopting two losses: (a) the Feature Matching loss LFM and (b) the Softmax Regression loss LSR. The latter is designed to take into account the classification task at hand, which imposes that for the same input image, the teacher’s and student’s feature produce the same output when passed through the teacher’s pre-trained and frozen classifier.

The whole idea is easy and clear. The authors also provided necessary experiments to rationalize the approach, for example, why the penultimate layer? why student classifier not included? The final model is thoroughly compared with state-of-the-arts knowledge distillation methods. Despite in several tasks, the proposed model fails to succeed the CRD method, given the simplicity of the proposed method and its superiority over most of the tasks, I think the results are satisfactory.

Still, I would like to give a suggestion:
1.	After CRD, proposed in 2019, there appears many new methods with refreshing performance. For example, [1] Xu G , Liu Z , Li X , et al. Knowledge Distillation Meets Self-Supervision[C]. in ECCV 2020. It would be more persuasive if comparisons with more recent methods could be provided.
review_title: A simple yet effective form of loss for general knowledge distillation

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Paper summary**
This paper proposes a new knowledge distillation method by enhancing the student network's representation learning. The proposed framework is very simple: use the teacher's final fully-connected layer (or, 'projection matrix $W$' in the paper) to obtain both the teacher's and the student's logit output and optimize the student network by minimizing the gap between two logits. Experiments cover various network architectures on CIFAR and ImageNet datasets and show improvements over the other competitive knowledge distillation methods. 

**Strengths**
1. The main idea to share the teacher's projection matrix $W$ for both the student and teacher's penultimate features seems new. 
2. Proposed method is simple and easy to implement. 
3. Experiments include a wide range of network architectures and datasets. 

**Weaknesses**
1. Definition of representation learning is unclear. 
The paper claims that using the pre-trained and frozen projection matrix brings decoupling of "representation learning" and "classification". Also, the paper says "Because direct feature matching might be difficult due to the lower representation capacity of the student ..." on page 2. 
After reading this section, I got the following questions: 1) According to the paper's solution, the teacher's projection matrix is used to solve the lack of representation capacity of the student. Does the "representation capacity" of a network only exist in the final fully-connected layer (projection matrix)? If true, references or pieces of evidence for this should be provided. 2) Why the method of utilizing the teacher's projection matrix is called representation learning? 

2. On page 4, by comparing eq (5) and eq (6), the paper claims that the KD loss $L_{KD}$ might have worse generalization ability than $L_{SR}$ since the KD loss simultaneously updates the feature extractor and classifier. The paper only shows empirical results where $L_{SR}$ works better than $L_{KD}$. This argument is not easily accepted for me, so more theoretical backup is necessary. 

3. Study on the hyper-parameter $\alpha$ and $\beta$ on eq (8) is missing

4. The main idea ("softmax representation learning", or $L_{SR}$) is very simple and seems to be easily applied with other KD methods beyond FitNets method. Combining $L_{SR}$ with other KD methods (e.g, AT, OFD, etc) will verify the generalization and applicability of the proposed method. 


**Post-comments to the author's response**
- Replies for Q3 and Q4 are good.
- However, the responses for Q1 and Q2 did not address my concern well. 
- For Q1, the authors claim that the representation capacity is in the CNN features not in the fully-connected layer. Then what is the meaning of utilizing the teacher's projection matrix for better representation learning? 
- Overall, I still this paper has more strengths than weaknesses, so I will keep my rating (6)





review_title: Review of the paper "Knowledge distillation via softmax regression representation learning"

final_score: rating: 6: Marginally above acceptance threshold
review_content: #####################################################################################
Summary:

This paper proposes a new formulation of knowledge distillation (KD) for model compression. Different from the classic formulation that matches the logits between student and teacher models, this paper suggests to match the output features of the penultimate layers between student and teacher models, based on L2 distance. Two complementary variants are introduced, with one directly minimizing the distance of the original feature vectors and with the other minimizing the distance of the feature vectors projected on the teacher classifiers. The approach is evaluated in a variety of scenarios, such as different network architectures, teacher-student capacities, datasets, and domains, and compared with state-of-the-art results.

####################################################################################
Pros:

The proposed knowledge distillation formulation is simple. The paper is well written. Experimental evaluations clearly demonstrate the effect by directly matching the output features of the penultimate layers.

####################################################################################
Cons: 

- While the proposed approach is interesting, its novelty seems limited, with formulations being special cases of existing methods. In particular, as the authors also mentioned, the proposed L_FM loss is a simplified FitNet loss in Romero et al., which focuses only on matching the final representation without considering the intermediate representations. The proposed L_SR loss is similar to the standard KD formulation in Hinton et al., with the only difference that the pre-trained teacher’s classifier is used for both teacher and student models.

- From the result tables, the performance improvement of the proposed approach is marginal, which is on par with existing works.

- As the authors mentioned, the L_SR loss is inspired by that only using the L_FM loss ignores the inter-channel dependencies of the feature representations h^S and h^T. So L_SR-CE is introduced. However, empirically, L_SR-CE is worse than L_SR-L2 which directly minimizes the distance between projected features. It seems that the empirical formulation is inconsistent with the motivation.

- In Eq 8, I assume that a separate classifier is also trained for the student model using the L_CE loss. How is the performance if we completely remove the student classifier? That is, the student model only consists of a feature extractor, and during inference time, we directly insert the pre-trained teacher classifier on top of the student model.

- It would be interesting to show the results when h^T and h^S have different dimensionality.

- In the ablation studies, the authors investigated “where should be losses be applied”. How is the classifier generated for the intermediate layers?

- How does the performance change with respect to different settings of the hyper-parameters alpha and beta?

- The notation is inconsistent throughout the paper. For example, h^T and h^S are used in the first half of the method section, while h_T and h_S are used later.

#################################################################################### Updated:

The authors' rebuttal addressed my concerns and I lean toward acceptance.
review_title: Official Blind Review #2

Meta-review: This paper proposes a new idea for performing knowledge distillation by leveraging teacher’s classifier to train student’s penultimate layer feature via proposing suitable loss functions. Reviewers appreciate the simultaneous simplicity and effectiveness of the method. A comprehensive set of studies are performed to empirically show the effectiveness of the method. Specifically, the proposed distillation method is shown to outperform state-of-the-art across various network architectures, teacher-student capacities, datasets, and domains. The paper is well-written and is easy to follow. All reviewers rate the paper on the accept side (after the rebuttal) and believe the new perspective this work provides on distillation and its simplicity to implement can lead it to gain high impact. I concur with the reviewers and find this submission a convincing empirical work, and thus recommend for accept.

Decision: Accept (Poster)