Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Paper summary**
This paper proposes a knowledge distillation on the feature maps using sparse representation. The proposed method firstly constructs an over-complement dictionary to express the teacher's feature maps and learn sparse representation to express the teacher's feature map using the dictionary. Since directly utilizing the sparse representation is a too strong restriction for the student network, the loss function is designed to find the indices of sparse codes. The proposed distillation method is validated through several experiments. 

**Pros**
1. This paper proposes a way of utilizing sparse representation for knowledge distillation.
2. The algorithm is written in clear formulations. 

**Cons**
1. The main idea of sparse representation matching (SRM) is the combination of sparse representation and knowledge distillation. However, the actual implementation of SRM does not transfer the sparse representation of the teacher to the student. Only the indices or the entire image's sparse code were transferred via knowledge distillation, so the feature map information of the teacher is not transferred. This looks counter-intuitive and weakens the arguments of the paper. Therefore, it is necessary to describe the information transferred by SRM in detail and the rationale for using this kind of information transferring.
2. All experiments were conducted on All-CNN, but this network is not usually used by other knowledge distillation papers, so the experiments should be re-conducted on a more standard and efficient setting. For example, All-CNN got 74.7% accuracy using SRM with 2.2M parameters (in table 1), but a recent paper [1] (CRD) got 75.5% accuracy using WRN16-2 with 0.7M parameters, which uses only 1/3 number of parameters. In other words, All-CNN is not proper to compare with other distillation methods. Use popular network architectures (WRN, ResNet, VGG etc.) to get more reasonable performance. 
[1] Contrastive Representation Distillation (ICLR 2020)
3. In the case of transfer learning, it is common to tune the learning rates and weight decays for each dataset and each network. However, the experiments in the paper consistently use the same learning hyper-parameters. As a result, the performance gap between the baseline and the proposed method seems to have greatly inflated. A great gap against baseline (about 10%~20%) seems to be very different from the results of the other distillation papers.  In short, to be fair comparisons with other distillation methods, learning parameter tuning is necessary. 


review_title: Review of "Knowledge Distillation By Sparse Representation Matching"

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strength and weaknesses:
+
The method is reasonable and competitive. Especially for the task transfer scenario is seems to advance the state of the art

-
Comparison is lacking as a main relevant competitor was not tested
I believe the method can be much simplified by instead of using sparse decomposition using plain clustering of pixel columns. Experiments testing this option should be conducted
The task transfer experiments, whose results are the most impressive, are done without weight Initialization of the student networks. It is not clear why, and the results may be different if initialized networks are used. 
At the bottom line: cleaner experiments (adding comparison to Jain et al, using plain clustering instead of sparse decomposition, using student networks with ImageNet initialized weights) are required to take this paper beyond reasonable doubt. 


Detailed comments:

-	Page 2: “However, we argue that the intermediate feature maps by themselves are not a good representation of the knowledge encoded in the teacher to teach the students” – This sentence states a claim, which is a main claim of this paper. However, the claim is not supported by any argument or justification in the introduction
-	Page 3: the subscript \Tau in D_{\Tau}^{(l)} seems to be meaningless (why do we need it?)
	Later is becomes clearer as D_S is introduced, but a note should be given before to make \Tau meaningful
-	Page 4: the pixel-labels are based on the 1-nn dictionary item, and so the knowledge transfer proposed in actually based a simple clustering of the pixels (each pixel represented by a single cluster index), not on the sparse representation. First, this means that K=1 can be naturally used, as K>1 is not really adding information to be transferred. Second, the method is very similar to transfer by clustering, seemed to be proposed by (Jain et al.) (based on the description of this work in the relevant work – I am not familiar with it)
-	Page 4: the image labels are enforcing similarity of word histograms between teacher and student. It is not clear, tough, why the BCE loss is used and not some histogram distance loss (l_1, Xi square, or even l_2). BCE is appropriate when the entrees in the two input vectors are probabilities of binary classification for independent problems. This is not the case here.
-	Page 5: it seems that comparison to the most similar method of (Jain et al.,) is not conducted 
-	Page 6: the experiments showing that SRM is not sensitive to \mu and \lambda increase the likelihood that simple clustering, where each pixel is hard-assigned to one of the clusters can be used instead of sparse code encoding (we know that the identity of the single first dictionary item is the only one transferred in the pixel-labels, so it is likely that similarity to other dictionary items is not relevant)
-	The task transfer results (table 5) are the most impressive results of the paper, indicating clear advantage of SRM over competitors. However, two drawbacks:
o	Why are the student networks randomly initialized and not initialized with their trained ImageNet weights (at least for ResNet 18 weights are publically available, I believe). Using pre-trained weight is likely to improve accuracy, and it is not clear why they are avoided
o	Comaprison to Jain et al is missing
-	Section 4.3: the teacher accuracy is missing

review_title: The method presented looks promising but cleaner experiments are required to clarify its working elements and provide comparison to a close aompetitor

final_score: rating: 5: Marginally below acceptance threshold
review_content: [Summary] 
This paper proposes to do knowledge distillation via matching the coefficients of sparse representations. The sparse representation reconstructs deep neural networks’ intermediate representation via a set of over-complete dictionary and corresponding coefficients. The method matches the coefficients at two levels. One is at the pixel level, and the other is at the image level, which pools the coefficients from the pixel level. The experiments perform model compression for image classification tasks and show a performance gain over previous methods.

[Strengths]
1. The idea of using sparse representation is novel and interesting.
2. The clarity is good and easy to follow. The related works are sufficiently discussed.

[Weaknesses]
1. The method lacks some details, such as how D_T and D_S are learned and optimized. Please consider using an algorithm block to describe the full procedure of the proposed method.
2. The experiment part has a large room to be improved. It is known that KD methods are sensitive to the type of architectures used in the teacher or student model. Moreover, most of the methods are very sensitive to the choice of hyper-parameters. There are two concerns: (1) Does the proposed method generalize to a broader range of network architectures pairs? (ex: the analysis in Tian et al. 2020) (2) Do all methods in the table be given the same amount of tuning budget for the hyper-parameter? Based on the text, different methods are given a different tuning budget. For example, Tables 1 and 5 only have part of the methods that are tuned with a validation set while others use the default value from other papers, making the comparison unfair.
3. There are already more than a dozen distillation methods try to match the intermediate representation in a wide variety of ways, while most of them seem to work. Maybe matching a random projection of intermediate representation between teacher and student will work as well. Then why we want to use a more complicated method? The work will be more valuable if it provides a more in-depth insight into why using sparse representation helps. The paper has some arguments in the introduction, but they are not convincing since the coefficients can be the same even though the dictionaries between teacher and student are very different. A mathematical explanation is preferred. 


review_title: Review

final_score: rating: 3: Clear rejection
review_content: <Paper summary>
This paper focuses on the problem of knowledge transfer between deep learning models, with the goal of transferring some of the information contained in a (typically large) teacher model to a smaller student network, improving performance of the latter. In particular, this work proposes to require the sparse representations of the activations of the teacher model to be 'similar' to that of the student model, as way of knowledge transfer. The method is presented and empirically evaluated.

<Review summary>
This reviewer likes the general motivation of this work, proposing that in some cases it might be better to enforce similarity between activations (or features) in a different domain, and requiring these representations to be sparse under some transformation is natural. While the general idea is appealing, several of the motivating claims are vague and the way these ideas are implemented (e.g. via classification to enforce similarity between vectors) are questionable.

<Details comments>
Strengths:
- the authors study an interest problem.
- the proposed method obtains good empirical performance.

Weaknesses:
- The idea of matching the representations for two different data in order to enforce some similarity (or in this context, 'knowledge transfer') has been extensively used. However, this only makes sense if the dictionaries for one and other case are related (see [1,2,3]). Enforcing the representations to be similar (or, as it's done here, to use the same leading atom) is reasonable when the atoms from one dictionary share some properties (or 'code' for related things) in the other.  In this work, the authors learn dictionaries for the teacher and student networks (Ds and Dt) completely independently, as there's no connection between the atoms in one and other dictionaries. 
- The authors include an 'image-level labeling' which basically compares the mean value in the approximate features from the teacher network ($\tilde{t}$) to that of the vector of similarities of the student one ($k_s$). 1) I do not see how this is informative of relevant information between models, but more importantly 2) Comparing these two real numbers with a logistic loss makes little (if any) sense to me.
- The idea of employing sparse representations for data (in this case, the intermediate representations of networks) is natural. There is a large body of work that the authors seem to ignore. For example: on pg 2 they mention that "[sparse representations learning] were not proposed to be jointly optimized with other objectives". Please see refs [4-6] below for examples of this.
- Representations under redundant dictionaries are not unique, and the problem of finding sparse representations is NP-hard (see e.g [7]). Certainly, one can propose relaxations of this problem and even heuristic approximations (see e.g. [8]) but this is never discussed, and it is unclear how the obtained representations in this work fit in this context.

Smaller comments:
- The proposed method seems to be a pre-processing step: firs train teacher dictionaries, then student dictionaries and student weights, and then employ the KD method from Hinton et al. The authors should consider making this more explicit, perhaps detailing the full algorithm in a formal way.
- On page 4, the authors motivate the use of the sigmoid as activation function saying that the 'gradients in the backward pass are stable'. What does this mean? Would they be unstable if a ReLU was used instead (as used in most deep learning models and in the other models in this work)?
- At the end of Pixel-level labeling, stating the definition for $c_{n,i,j}$, the authors take the argmax over m, but there's no m in the expression.
- 'Relaxing' the problem of requiring the representations to be similar simply by turning a regression problem into a classification problem seems unfounded: why would the latter be easier than simply allowing the representations to be similar (say, with small L2 norm)?
- In defining the similarity kernel $\kappa$, I believe the $x$ and $y$ should be bold according to the authors notation.
- On a subjective note, the notation is not standard and thus a bit confusing: calligraphic capital letters usually denote sets or distributions, whereas here they denote vectors (as do bold non-calligraphic letters).

References:
1] Wang, Shenlong, et al. "Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis." 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012.

2] Qiu, Qiang, et al. "Domain adaptive dictionary learning." European Conference on Computer Vision. Springer, Berlin, Heidelberg, 2012.

3] Peleg, Tomer, and Michael Elad. "A statistical prediction model based on sparse representations for single image super-resolution." IEEE transactions on image processing 23.6 (2014): 2569-2582.

4] Mairal, Julien, Francis Bach, and Jean Ponce. "Task-driven dictionary learning." IEEE transactions on pattern analysis and machine intelligence 34.4 (2011): 791-804.

5] Sprechmann, Pablo, Alexander M. Bronstein, and Guillermo Sapiro. "Learning efficient sparse and low rank models." IEEE transactions on pattern analysis and machine intelligence 37.9 (2015): 1821-1833.

6] Monga, Vishal, Yuelong Li, and Yonina C. Eldar. "Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing." arXiv preprint arXiv:1912.10557 (2019).

7] Mairal, Julien, Francis Bach, and Jean Ponce. "Sparse modeling for image and vision processing." 

8] Makhzani, Alireza, and Brendan Frey. "K-sparse autoencoders." arXiv preprint arXiv:1312.5663 (2013).
review_title: Anonymous Review

Meta-review: The paper received four negative reviews. The overall idea was found to be interesting, but several concerns were raised. There is a general consensus that the experimental part and the results are not convincing. Several comments have also been made regarding the clarity and motivation, which needs to be strengthened. R4 also mentions references from the sparse estimation literature that would help for positioning the paper. The rebuttal did address some of these points, but it was not sufficient to change their opinion.

Overall, the area chair agrees with the reviewers and follows their recommendation.
Decision: Reject