Knowledge Distillation By Sparse Representation Matching
Authors: Dat Thanh Tran
Moncef Gabbouj
Alexandros Iosifidis
Venue:  (2021)
Keywords: Knowledge Distillation
Sparse Representation
Transfer Learning

Abstract: Knowledge Distillation refers to a class of methods that transfers the knowledge from a teacher network to a student network. In this paper, we propose Sparse Representation Matching (SRM), a method to transfer intermediate knowledge obtained from one Convolutional Neural Network (CNN) to another by utilizing sparse representation learning. SRM first extracts sparse representations of the hidden features of the teacher CNN, which are then used to generate both pixel-level and image-level labels for training intermediate feature maps of the student network. We formulate SRM as a neural processing block, which can be efficiently optimized using stochastic gradient descent and integrated into any CNN in a plug-and-play manner. Our experiments demonstrate that SRM is robust to architectural differences between the teacher and student networks, and outperforms other KD techniques across several datasets. 

Under review as a conference paper at ICLR 2021
KNOWLEDGE DISTILLATION BY SPARSE REPRESEN- TATION MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge Distillation refers to class of methods that transfers the knowledge from a teacher network to a student network. In this paper, we propose Sparse Representation Matching (SRM), a method to transfer intermediate knowledge obtained from one Convolutional Neural Network (CNN) to another by utilizing sparse representation learning. SRM first extracts sparse representations hidden features of the teacher CNN, which are then used to generate both pixel- level and image-level labels for training intermediate feature maps the student network. We formulate SRM as neural processing block, which can be efficiently optimized using stochastic gradient descent and integrated into any CNN in a plug- and-play manner. Our experiments demonstrate that SRM is robust to architectural differences between the teacher and student networks, and outperforms other KD techniques across several datasets.
the
INTRODUCTION
Over the past decade, deep neural networks have become the primary tools to tackle learning problems in several domains, ranging from machine vision (Ren et al. 2015
Redmon Farhadi 2018)) natural language processing (Devlin et al. 2018; Radford et al. 2019) to biomedical analysis (Kiranyaz) et al. 2015) or financial forecasting (Tran et al. 2018b Zhang et al. 2019). Of those important developments, Convolutional Neural Networks have evolved a de facto choice for nigh-dimensional signals, either as a feature extraction block or the main workhorse in a learning system. Initially developed in the 1990s for handwritten character recognition using only (LeCun et al. 1998), state-of-the-ar CNN topologies nowadays consist of hundreds of layers, having millions of parameters (Huang al.] 2017 Xie al.) (2017). In fact, not only in computer vision but also in other domains, state-of-the-art solutions are mainly driven by very large networks (Devlin et al.) 2018, Radford et al., 2019), which limits their deployment in practice due to the high computational complexity.
convolutional layers
The promising results obtained from maximally attainable computational power has encouraged lot of research developing smaller and light-weight models while achieving similar performances. This includes efforts on designing more efficient neural network families (both automatic and hand-
crafted) (Howard et al. (017 Tran et al. 201 8a 2019: Zoph & Le| (016)). compressing pretrained networks through weight pruning (Manessi et al. 2018 Tung & Mori 2018), quantization (Hubara
al. 2017 Zhou et al. (017) or approximation Denton al. (2014); Jaderberg al. (2014), as well as transferring knowledge from one network to another via knowledge distillation Hinton et al. (2015). Of these developments. Knowledge Distillation (KD) (Hinton et al.) 2015) is simple and widely used technique that has been shown to be effective in improving the performance of a network, given the access one many pretrained networks. KD and its variants work by utilizing the knowledge acquired in one or many models (the teacher(s)) as supervisory signals to train another model (the student) along with the labeled data. Thus, there are two central questions KD:
How to represent the knowledge encoded in a teacher network?
How to efficiently transfer such knowledge to other networks, especially when there are architectural differences between the teacher and the student networks?
In the original formulation (Hinton et al. 2015). soft probabilities produced by the teacher represent its knowledge and the student network is trained mimic this soft prediction. Besides the final

---

Under review as a conference paper at ICLR 2021
predictions, other works have proposed to utilize intermediate feature maps of the teacher as additional knowledge (Romero et al. 2014 Zagoruyko & Komodakis 2016, Gao et al. 2018 Heo et al. 2019;
Passalis & Tefas 2018). Intuitively, intermediate feature maps contain certain clues how the imput is progressively transformed through layers of a CNN, thus can act as a good source of knowledge. However, we argue that the intermediate feature maps by themselves are not good representation of the knowledge encoded in the teacher to teach the students. To address the question of representing the knowledge of the teacher CNN, instead of directly utilizing the intermediate feature maps of the teacher as supervisory signals, we propose to encode each pixel (of the feature maps) in sparse domain and use the sparse representation as the source of supervision.
Prior to the era of deep learning, sparse representations attracted great amount of interest in computer vision community and a basis of many important works (Zhang et al., 2015). Sparse representation learning aims at representing the input signal in a domain where the coefficients are sparsest. This achieved by using an overcomplete dictionary and decomposing the signal as a sparse linear combination of the atoms in the dictionary. While the dictionary can be prespecified, it is often desirable to optimize the dictionary together with the sparse decomposition using example signals. Since hidden feature maps in CNN are often smooth with high correlations between neighboring pixels, they are compressible, e.g., in Fourier domain. Thus, sparse representation serves as a good choice for representing information encoded in the hidden feature maps.
Sparse representation learning is a well-established topic in which several algorithms have been proposed (Zhang et al., 2015). However, to the best of our knowledge, existing formulations computationally intensive to fit a large amount of data. Although learning task-specific sparse representations have been proposed in prior works (Mairal et al. 2011, Sprechmann et al.) [015 Monga et al. 2019), we have not seen its utilization for know Knowledge transfer using deep neural networks and stochastic optimization. In this work, we formulate sparse representation learning as a computation block that can be incorporated into any CNN and be efficiently optimized using mini- update from stochastic gradient-deseen based algorithms. Our formulation allows us to take advantage of not only modern stochastic optimization techniques but also data augmentation to generate target sparsity on-the-fly.
Given the sparse representations obtained from the teacher network, we derive the target pixel-level and image-level sparse representation for the student network. Transferring knowledge from the teacher to the student is then conducted by optimizing the student with its dictionaries to induce the target sparsity. Thus, our knowledge distillation method is dubbed as Sparse Representation Matching (SRM). Extensive experiments presented in Section 4 show that SRM significantly outper- forms other recent KD methods, especially in transfer learning tasks by large margins. In addition, empirical results also indicate that SRM exhibits robustness to architectural mismatch between the teacher and the student.
2 RELATED WORK
The idea of transferring knowledge from one model another has existed for long time. This idea was first introduced in /Breiman & Shang (1996) in which the authors proposed to grow decision trees to mimic the output of complex predictor. Later, similar ideas were proposed for training neural networks (Ba & Caruana 2014 Bucilu et al. 2006 Hiinonn et al. 2015). mainly for the purpose of model compression. Variants of the knowledge transfer idea ditler the methods of representing
and transferring knowledge (Cao et al. 2018 Heo et al. 2019 Romero et al. 2014 Zagoruyko &
Komodakis 2016 Passalis & Tefas 2018 Park al. 2019 Tian et al.] 2020), as well as the types of
data being used (Lopes et al.] 2017
Yoo et al. 2019 Papernot et al. 2016 Kimura et al., 2018).
In Bucilu et al. (006)). the final predictions of an ensemble on unlabeled data are used to train a single neural network. In Ba & Caruana (2014), the authors proposed to use the logits produced by the source network as the representation of knowledge, which is transferred to a target network by minimizing the Mean Squared Error (MSE) between the logits. The term Knowledge Distillation was introduced in Hinton et al.] (2015) in which the student network is trained to simultaneously minimize the cross-entropy measured on the labeled data and the Kullback-Leinn
(KL) divergence between its predicted probabilities and the soft probabilities produced by the teacher network. Since its introduction, this formulation has been widely adopted.
2

---

Under review as a conference paper at ICLR 2021
In addition to the soft probabilities of the teacher, later works have been proposed to utilize intermedi- ate features of the teacher as additional sources of knowledge. For example, in FitNet (Romero et al. 2014), the authors referred to intermediate feature maps of the teacher hints and the student first pretrained by regressing its intermediate features to the teacher's 's hints, then later optimized with the standard KD approach. In other works, activation maps (Zagoruyko & Komodakis 2016) well as feature distributions (Passalis & Tefas 2018) computed from intermediate layers have been proposed. In recent works (Park al. 2019 Tian al. 2020), instead of transferring knowledge about each individual sample, the authors proposed to transfer relational knowledge between pairs of samples.
Our SRM method bears some resemblances previous works in the sense that SRM also uses intermediate feature maps as additional sources of knowledge. However, there are many differences between SRM and existing works. For example, in FitNet (Romero et al. 2014), the student network learns to regress from its intermediate features to the teacher's; however, the regressed features are not actually used in the student network. Thus, the hints in FitNet only indirectly influence the student's 's features. Since the sparse representation is another representation (equivalent) of the feature maps, SRM directly influences the student's 's features. In addition, by manipulating the sparse representation rather than the hidden features themselves, SRM is less prone feature value range mismatch between the teacher and the student. This is because by construction. the sparse coefficients generated by SRM only have values in the range [0. 1] as we will in Section 3 Attention- bbased KD method (Zagoruyko & Komodakis) 2016) overcomes this problem by normalizing (using l2 norm) the attention maps of the teacher and the student. This normalization step, however, might suffer from numerical instability (when l2 norm is very small) when attention maps are calculated from activation layers such as ReLU,
In (Liu et al. 2019), the authors employed the idea of sparse coding, however, to represent the network's 's parameters rather than the intermediate feature maps as in our work. (Jain et al. 2019). the authors used the idea of feature quantization and k-means clustering using intermediate features of the teacher to train the student with additional convolutional modules to predict the cluster labels. Our pixel-level label bears some resemblances to this method. However, we explicitly represent the intermediate features by sparse representation (by minimizing reconstruction error) and use the same process to transfer both local (pixel-level) and global (image-level) information.
KNOWLEDGE DISTILLATION BY SPARSE REPRESENTATION MATCHING
3.1 KNOWLEDGE REPRESENTATION
Given the n-th input image Xx, let us denote by T.! E &RHixWiXC, the output of the I-th layer of the teacher CNN, with Hi and Wi being the spatial dimensions and Cj is the number of channels. In the following, we used the subscript T and S to denote variable that is related to the teacher and student networks, respectively. In addition, we also denote by m,L, = Tnl (i, E RRC which the pixel
(1)
at position (i,j) of (h The first objective SRM is to represent each pixel m.iy in a sparse domain. To do so, SRM learns an overcomplete dictionary of M, atoms: DJ = _ d M, E RCxM
(Mr > Ct), which used to express each pixel n,ü.j as a linear combination d T as follows:
Mi
th ( d. k(thing a. df.
n.i.
m=1
where
n(th)). al)..) denotes a function that measures the similarity between n,i.j and atom a
We further denote by (1) kT,,,i = [n di) d...))...,, ((t (( (l) T.M,) the vector that contains n,i.j
similarities between (1)
n.i.j and all atoms in the dictionary D
the (t) (t d!!)) denotes the indicator function that returns value of 1 if ((t()) k(t n,ij" di belongs
to the set of top-k values in k() mmnii and value of 0 otherwise.
3

---

Under review as a conference paper at ICLR 2021
basically means that ( is expressed as the linear combination of
The decomposition in Eq.
k most similar atoms in D
with the coefficients being the corresponding similarity values. Let
An.ij.m th (t n. : T.
(1)
(1)
((t n.i.j. j. a
(1) is then defined T. then the sparse representation of n.j
as:
(0] P m.
A.j.jar E R
n.d.j
((0) By construction, there are only & non-zero elements in ",a.j" and l defines the degree of sparsity,
which is a hyper-parameter of SRM. In order to find E(0) error in Eq. [1] as follows:
",i.j" we simply minimize the reconstruction
M
arg min
Σ D n.A4
whe (t a Σ m=1
n(t n.a.
n.ay
m
m
There are many choices for the similarity function w such as linear kernel, RBF kemel, sigmoid kernel and so on. In our work, we used the sigmoid kernel k(x, y) sigmoid(x") y + c) since the dot- dot-product makes it computationall efficient and the gradients in the backward pass are stable. Although the RBF kernel is popular in many works, we empirically found that RBF kernel is sensitive to the learning rate, which easily leads to numerical issues.
3.2 TRANSFERRING KNOWLEDGE
Let us denote by S!P)) E HRH, the output of the p-th layer of the student network given input image is Kr. In addition, / E RC, denotes the pixel at position (i,j) of skP). We consider the task of transferring knowledge from the 1-th layer of the teacher to the p-th layer of the student. To do so, we require that the spatial dimensions of both networks match (Hp Ht and W,, = Wi) while the channel dimensions might differ.
Given the sparse representation the teacher in Eq. 2 a straightforward way is to train the student network to produce hidden features at spatial position (i, j)) having the same sparse coefficients as its teacher. However, trying to learn exact sparse representations as produced by the teacher is a too restrictive task since this enforces learning the absolute value of every point in a high-dimensional space. Instead of enforcing an absolute constraint on how each pixel of every sample should be represented, to transfer knowledge, we only enforce a relative constraints between them in the sparse domain. Specifically, we propose to train the student to only approximate sparse structures of the teacher network by solving a classification problem with two types of labels extracted from the sparse representation E(!) of the teacher: pixel-level and image-level labels.
n,i,
Pixel-level labeling: for each spatial position (i,j), we assign a class label, which is the index of the (!) This basically means
(!) largest element of ". i.e., the index of the closest (most similar) atom in D
that we partition all pixels into M, disjoint sets using dictionary D4,, and the student network is trained to learn the same partitioning using its own dictionary (()) = -
a - E RR M, n.agg a(MM)) denote the vector that contains similarities
Let k (p) S,n,i.j = [m(sl1...]
(n) mn,.j" s, ).... k(s
(1)
(p)
between pixel Sn.i. and M, atoms in D (p) The first knowledge transfer objective in our method
using pixel-level label is defined as follows:
(p) arg min LCE((GnRs (.sniisikssns)) kP)
(4)
Dg. n,Ky
where Og denotes parameters of the student network. Log denotes the cross-entropy loss function,
and Cn,i.j = arg max((1)) (( Here we should note that the idea of transferring the structure instead of the absolute representation is not new. For example, in (Park al. 2019) and (Tian et al. 2020), the authors proposed to transfer the relative distance between the embeddings of samples. In our case, the pixel-level labels provide supervisory information on how the pixels in the student network should be represented in the sparse domain so that their partition using the nearest atom is the same.
Image-level labeling: given the sparse representation E,, of the teacher, we generate image-
=(1) level label by averaging n..... over the spatial dimensions. While pixel-level labels provide local
4

---

Under review as a conference paper at ICLR 2021
Table 1: Performance in CIFAR 100
Model
Test Accuracy %
DenseNet121 (teacher)
75.09 00.29
AIICNN
67.64 + 01.87
AIICNN- KDD
73.27 = 00.20
MICNN--FiNG
72.03 = 00.27
AIICNN-AT
70.88 + 0.29
AICCNN-PTT
72.22 + 0.35
ALICNN-RKD
70.39 0.17
AIICNN-CRD
72.70 + 0.18
AIICNN-SRI (our)
74.73 00.26
supervisory information encoding the spatial information, image-level label provides global supervi- sory information, promoting the ihirt-invariance property. Image-level label bears some resemblances to the Bag-of- of---eeuu model (Passalis & Tefas 2017), which aggregates the histograms of image patches to generate an image-level feature. The second knowledge transfer objective our method using image-level labels is defined as follows:
Ens mii Ei, S.n.d.d Hi Wi H, W
arg min D 6
LBCE
(5)
where LECE denotes the binary cross cross-entropy loss. Here we should note that since most kernel functions output a similarity score in [0. 1], elements of z(!) making the two inputs to LBCE in Eq. (5) valid.
n,i.j and (p) S,n.i are also in this range,
To summarize, the procedures of our SRM method similar FitNet (Romero al. 2014). which consists of the following steps:
Step 1: Given the source layers (with indices 1) in the teacher network T, find the sparse representation by solving Eq.
Step 2: Given the target layers (with indices p), optimize the student network S predict pixel-Ievel and image-level labels by solving Eq. (4) (5).
Step 3: Given the student network obtained in Step 2, optimize it using the original KD algorithm.
All optimization objectives in algorithm are solved by stochastic gradient descent.
EXPERIMENTS
The first of experiments conducted CIFAR100 dataset (Krizhevsky et al. 2009) compare our SRM method with other KD methods: KD (Hinton et al. 2015). FitNer (Romero et al. 2014), AT
(Zagoruyko & Komodakis 2016), PKT (Passalis & Tefas 2018), RKD (Park al.] 2019) and CRD (Tian et al. (2020). In the second set of experiments, we tested the algorithms under transfer learning setting. In the final of experiments, evaluated SRM on the large-scale ImageNet dataset.
For every experiment configuration, we ran 3 times and reported the mean and standard deviation. Regarding the source and the target layers for transferring intermediate knowledge. we simply selected the outputs of the downsampling layers. Detailed information about our experimental setup provided in the Appendices.
4.1 EXPERIMENTS ON CIFARION 100
Since CIFARI00 has validation set, randomly selected 5K samples from the training set for validation purpose. reducing the training set size to 45K. Our setup is different from the conventional practice of validating and reporting the result the test set of CIAR100 100. In this set of experiments,

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison (test accuracy %) between FitNet and SRM in terms of quality of intermediate
knowledge transfer on CIFARI00
ALICNN-FitNel
AIICNN-SRM (our)
Linear Probing
69.95 + 00.20 68.06 = 00.10
69.10 + 00.31
Whole Network Update
71.99 00.08
used DenseNet121 (Huang et al., 2017) as the teacher network (7.1M parameters and 900M FLOPs) and non-residual architecture (a variant of AIICNN network (Springenberg et al. 2014)) as the student network (2.2M parameters and 161M FLOPs). Other details about training hyperparameters can be found in Appendix
Overall comparison (Table : It is clear that the student network significantly benefits from all knowledge transfer methods. The proposed SRM method clearly outperforms other competing methods, establishing more than 1% margin compared to the second best method (KD). In fact, the student network trained with SRM achieves very close performance with its teacher (74.73% versus 75.09%), despite having a non-residual architecture and 3x less parameters. addition, recent methods, such PKT and CRD, perform better than FitNet, however, inferior the original KD method.
Quality of intermediate knowledge transfer: Since FitNet and SRM have pretraining phase to transfer intermediate knowledge, we compared the quality of intermediate knowledge transferred by FitNet and SRM by conducting two types of experiments after transferring intermediate knowledge: (1) the student network is optimized for 60 epochs using only the training data (without teacher's soft probabilities), given all parameters are fixed, except the last linear layer for classification. This experiment is called Linear Probing; (2) the student network is optimized with all parameters for 200 epochs using only the training data (without teacher's soft probabilities). This experiment is named Whole Network Update. The results are shown in Table
Firstly, both experiments show that the student networks outperform their baseline, thus, benefit from intermediate knowledge transfer by both methods, first experiment when we only updated the output layer, the student pretrained by FitNet achieves slightly better performance than by SRM (69.95% versus 69. 69.10%) However, when optimized with respect to all parameters, the student pretrained by SRM performs significantly better than the one pretrained by FitNet (71.99% versus 68.06 68.06%). While full parameter update led the student pretrained by SRM to better local optima, it undermines the student pretrained by FitNet. This result suggests that the process of intermediate knowledge transfer in SRM initialize the student network at better positions in the parameter space compared to FitNet.
without the final KD phase. In the
Effects of sparsity (A) and dictionary size (μ): in Table we show the performance of SRM with different degrees of sparsity (parameterized by X = k/M, lower À indicates higher sparsity) and dictionary sizes (parameterized by μ = MI/ Ct, higher μ indicates higher overcompleness). As can be seen from Table 3 SRM is not sensitive to À and p. In fact, the worst configuration still performs slightly better than KD 73%.71% versus 73.27%), and much better than other AT, PKT, RKD and CRD.
Table SRM test accuracy (%) on CIFAR 100 with different dictionary sizes (parameterized by μ1) and degree of sparsity (parameterized by X)
# == 1.5
μl = 2.0
3.0
0.01
74.00+-00.15
74.12+000.35
74.09-000.08
0.02
74.34400.007
74.20=00.27 74.73
0.03
73.77400.05
73.83+00.12
73.71=00.51
Pixel-level label and image-level label: finally, show the importance of combining both pixel-
level and image-level label, we experimented with two other variants SRM on CIFAR100: using either pixel-level image-level label. The results shown in Table 4
The student network

---

Under review as a conference paper at ICLR 2021
performs poorly when only receiving intermediate knowledge via image-level labels, even though it was later optimized with the standard KD phase. Similar to the observations made from Table 2 this again suggests that the position in the parameter space, which is obtained after the intermediate knowledge transfer phase, and prior to the standard KD phase, heavily affects the final performance. Once the student network is badly initialized, even receiving soft probabilities from the teacher as additional signals does not help. Although using pixel-level label alone is better than image-level label, the best performance is obtained by combining both objectives.
Table 4: Effects of pixel-level label and image-level label in SRM on CIFAR
Pixel-level
Image-level label
Test accuracy %
label
73.16 + 00.39 53.50 + 09.96 74.73 + 00.26
4.2 TRANSFER LEARNING EXPERIMENTS
Since transfer learning is key in the success of many applications that utilize deep neural networks, we conducted experiments in transfer learning problems (Flowers (Nilsback & Zisserman, 2008). CUB (Wah et al. 201 1). Cars (Krause et al. 2013), Indoor-Scence (Quattoni & Torralbal (2009) and PubFig83 (Pinto et al. 2011)) to assess how well the proposed method works under transfer learning setting compared others.
In our experiments, we used a pretrained ResNext50 (Xie et al.) (017) on ILSVRC2012 database as the teacher network, which is finetuned using the training set of each transfer learning task. We then benchmarked how well each knowledge distillation method transfers both pretrained knowledge and domain specific knowledge Both residual (ResNet18 (He et al., 2016), 11.3M parameters, 1.82G FLOPs) and non-residual (a variant of AIICNN (Springenberg al. | 20144) 5.1M parameters, 1.35G FLOPs) architectures were used as the student.
randomly initialized student network using domain specific data.
Full transfer setting: this setting, we used all samples available in the training set perform knowledge transfer from the teacher to the student. The test accuracy achieved by different methods shown in the upper part of [ on many datasets, except Cars and Indoor-Scence datasets for AIICNN student. While KD, AT, PKT, CRD and SRM successfully led the students to better minima with both residual or non non-residual students, FitNet was only effecttvv with the residual one. The results suggest that the proposed intermediate knowledge transfer mechanism in SRM is robust to architectural differences between the teacher and the student networks.
clear that the proposed SRM outperforms other methods
Few-shot transfer setting: we further assessed how well the methods perform when there a limited amount of data for knowledge transfer. For each dataset, we randomly selected 5 samples (5-shot) and 10 samples (10-shot) from the training set for training purpose, and kept the validation and test set similar to the full transfer setting. Since the Flowers dataset has only 10 training samples in total (the original split provided by the database has 20 samples per class, however, we used 10 samples for validation purpose), the results for 10-shot are similar full transfer learning setting. The test performance (% in accuracy) is reported in the lower part of Table 5 Under this restrictive regime, the proposed SRM method performs far better than other tested methods for both types of students, largely improves the baseline results.
4.3 IMAGENET EXPERIMENTS
For ImageNet (Russakovsky et al. 2015) experiments, we followed similar experimental setup as in (Cho & Hariharan 2019: Zagoruyko & Komodakis 2016): ResNet34 and ResNetl 88 were used as the teacher and student networks, respectively. Table shows top-1 classification errors of SRM and related methods having the same experimental setup. The teacher network (Resnet34) achieves top-1 error of 26.760000 70%. Using SRM, we can successfully train ResNet18 to achieve 28.79% classification emror, which is lower than existing methods (except CRD), including the recently proposed ESKD+AT
7

---

Under review as a conference paper at ICLR 2021
Table 5: Transfer learning using AlICNN (ACNN) and ResNet18 (RN18) (test accuracy %). The standard deviation of the test accuracy few-shot settings reported in Table
Model
Flowers
CUB
Cars
Indoor-Scenes
PubFig83
ResNext50
89.35 + 00.62
69.53 + 00.45
87.45 + 00.27
63.51 + 00.43
91.41 = 00.14
Full Shot
ACNN
40.80 + 02.33
47.26
00.18
61.93 01.38
35.82 00.43
78.47 = 00.17
ACNN-KD
46.14 1: 00.39
51.80 1: 00.41
66.12 1 00.17
38.44 00.99
81.54 1: 00.09
ACNN-FitNet
30.10 1 02.41
44.30 1: 01.25
60.20 1: 03.83 03.83
30.87 00.35
77.61 1: 00.87
ACNN-AT
51.62 1 00.69
51.74 1: 00.39
7 00.06
43.56 1 00.52
81.11 1 01.51
ACNN-PKT PTN
47.12 00.52
47.60 00.85
00.16
37.71 00.64
82.03 1 00.26
ACNN-RKD
42.00 = 01.10
39.99 + 00.61
02.44
30.94 00.45
75.44 = 00.50
ACNN-CRD
46.99 = 01.13
51.12 - 00.34
08.89
42.82 = 00.21
83.02 00.11
ACNN-SRM
51.72 00.58
54.51 01.72
04.76
43.09 00.60
82.89 = 01.78
RNI8
44.25 = 00.42
44.79 + 00.68
51.97
36.72 = 00.29
79.08 = (0.36
RNIS-KD
48.26 00.33
54.91 = 00.33
75.29
43.84 00.67
84.49 = 00,28
RNI8-FilNet
+ 1 01.24
61.28 + 00.48
+
45.93 + 01.00
89.78 + 00.20
RNI8-AT
51.49 + 00.12
53.13 + 00.40
00.15
41.13 + 00.55
83.60 + 00.19
RNI8-PKT
45.32 + 00.51
45.24 + 00.39
7
37.27 00.79
82.00 + 00.10
RNI8-RKD
12.32 + 00.28
36.29 + 00.58
56.87 + 02.64
29.57 00.90
70.90 + 01.79
ANI8-CRD
47.67 + 00.05
53.25
+ 00.83
76.51 + 00.74
43.76 + 00.40
84.43 = 00.40
RNI8-SRM
67.46 01.06
63.11 00.45
84.40 + 00.67
54.59 + 00.38
89.79 00.53
5-Shot
10-shot
ACNN
32.91 + 40.80
13.19 + 25.53
05.03 + 09.50
09.21 + 16.23
05.15 + 10.09
ACNN-KD
35.98 + 46.14
21.60
+ 34.63
10.61 + 23.43
14.81 + 21.76
11.97 4 28.11
ACNN-FiNNet
28.73 30.10
14.78
+ 29.40
06.11 15.81
08.36 + 15.71
08.25 + 17.89
ACNN-AT
38.21 - 51.62
17.69
- 30.26
08.52 25.22
08.76 - 17.90
08.02 - 26.27
ACNN-PKT
33.25 - 47.12
11.30 - 24.93
06.16 + 13.82
10.75 + 16.78
06.13 - 10.67
ACNN-RKD
30.27 > 42.00
09.55 + 19.36
01.82 09.60
09.68 - 12.40
04.82 - 06.99
ACNN-CRD
35.01 46.99
18.09
29.72
06.77
16.96
09.73 + 17.60
06.33
17.21
ACNN- MANN-RRM
41.14 - 51.72
22.89 + 35.86
11.71 + 36.2 28
16.63 + 24.82
13.96 + 31.47
RNI8
32.95 + 44.25
+ 22.72
05.00 11.76
09.04 + 15.16
04.98 + 08.92
RNI8-KD
38.07 + 48.26
25.53 + 40.57
11.37 + 35.44
14.61 + 23.85
10.69 + 29.67
RNI8-FilNel
39.17 + 48,29
26.50 + 43.83
12.36 51.88
13.72 + 24.62
11.49 + 36.79
RNI8-AT
37.36 51.49
18.22 30.47
08.96
27.70
09.93 + 17.48
08.76 + 29.80
RNIS-PKT
33.21 45.32
10.63 + 20.62
05.88
11.00
11.03 + 15.76
05.31 + 08.80
RNI8-REC -NI8-RKD
29.97 - 42.32
08.83 + 17.73
01.98
08.73
08.41 + 11.82
04.19 + 06.49
ANI8-CRD
31.21 47.67
18.36 + +30.44
06.95 - 17.11 17.11
09.01
+ 17.82
06.36 + 16.08
RNI8 SRM
51.24 67.46
34.67 48.42
61.22 26.63
32.21 21.18
29.31 51.99
8

---

Under review as a conference paper at ICLR 2021
(Cho & Hariharan 2019) that combines carly-stopping trick and Attention Transfer Zagoruyko & Komodakis 2016).
Table 6: Top-1 1 Error ResNet18 on ImageNet. (*) indicates results obtained by 110 epochs. References of the methods in this table are mentioned in the Appendix C
Seq. KD
SRM (our)
KD
AT
KD+ONE
ESKD
ESKD+AT CRD*
30.79
29.30
29.60
29.45
29.16
28.84
28.62"
28.79
5
CONCLUSION
this work, we proposed Sparse Representation Matching (SRM), a method to transfer intermediate knowledge from one network to another using sparse representation learning. Experimental results on several datasets indicated that SRM outperforms related methods, successfully performing inter- mediate knowledge transfer even if there significant architectural mismatch between networks and/or a limited amount of data. SRM serves as a starting point for developing specific knowledge transfer use cases, e.g., data-free knowledge transfer, which is interesting future research direction.
REFERENCES
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pp. 2654-2662, 2014.
Leo Breiman and Nong Shang. Born again trees. University of California, Berkeley, Berkeley, CA, Technical Report, 1:2, 1996.
Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Chensi Cao, Feng Liu, Hai Tan, Deshou Song, Wenjie Shu, Weizhong Li, Yiming Zhou, Xiaochen Bo, and Zhi Xie. Deep learning and its applications in biomedicine. Genomics, proteomics & bioinformatics, 16(1):17-33 2018.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4794 4802, 2019.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances neural information processing systems, pp. 1269-1277, 2014
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: prrXiv:181..1 1810.04803 2018.
Mengya Gao, Yujun Shen, Quanquan Li, Junjie Yan, Liang Wan, Dahua Lin, Chen Change Loy, and Xiaoou Tang, An embarrassingly simple approach for knowledge distillation. arXiv preprint arXiv:1812.018894 2018
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3779-3787, 2019. 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint /503. 025311, 2015.

---

Under review as a conference paper at ICLR 2021
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
2017.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1700-4708, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18()):686968888 2017.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXivv 2014.
Himalaya Jain, Spyros Gidaris, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Quest: Quan- tized embedding space for transferring knowledge. arXiv preprint
2019.
Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few- shot learning of neural networks from scratch by pseudo example optimization. arXiv preprint arXiv:/802.088 00099 2018.
Serkan Kiranyaz, Turker Ince, and Moncef Gabbouj. Real-time patient-specific ecg classification by 1-d convolutional neural networks. IEEE Transactions on Biomedical Engineering, 63(3):664-675, 2015.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (1:22)222222222 1998.
Junjie Liu, Dongchao Wen, Hongxing Gao, Wei Tao, Tse-Wei Chen, Kinya Osa, and Masami Kato. Knowledge representing: Efficient, sparse representation of prior knowledge for knowledge distillation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535. 2017.
Julien Mairal, Francis Bach, and Jean Ponce. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence, 34(4):791-804, 2011.
Franco Manessi, Alessandro Rozza, Simone Bianco, Paolo Napoletano, and Raimondo Schettini. Automated pruning for deep neural network compression. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 657-664. IEEE, 2018.
Vishal Monga, Yuelong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. arXiv preprint arXiv: 2019.
Maria-Elene Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729. IEEE, 2008
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755 2016.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3967-3976. 2019.
10

---

Under review as a conference paper at ICLR 2021
Nikolaos Passalis and Anastasios Tefas. Neural bag-of-featuree learning. Pattern Recognition, 64: 277-294, 2017.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference Computer Vision (ECCV), pp. 268-284, 2018.
Nicolas Pinto, Zak Stone, Todd Zickler, and David Cox. Scaling up biologically-inppiiee computer vision: A case study in unconstrained face recognition facebook. In CVPR / WORKSHOPS, pp. 35 42. IEEE, 2011.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 413-420. IEFE, 2009.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Joseph Redmon and Ali Farhadi. Yolov3: arX4v:1804.000 02767, 2018.
An incremental improvement. arXiv preprint
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint wrXiv:1412.6550. 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211225 2015.
Pablo Sprechmann, Alexander M Bronstein, and Guillermo Sapiro. Learning efficient sparse and low rank models. IEEE transactions pattern analysis and machine intelligence, 3709)18211-1833, 2015.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: arXiv: 1e..1/4/266666 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna-
tional Conference on Learning Representations, 2020.
Dat Thanh Tran, Alexandros Iosifidis, and Moncef Gabbouj. Improving efficiency in convolutional neural networks with multilinear filters. Neural Networks, 105:328 055:22-339,, 2018a.
Dat Thanh Tran, Alexandros Iosifidis, Juho Kanniainen, and Moncef Gabbouj. Temporal attention- augmented bilinear network for financial time-series data analysis. IEEE transactions on neural networks and learning systems, 30(5):140)) 140--148811 2018b.
Dat Thanh Tran. Serkan Kiranyaz, Moncef Gabbouj. and Alexandros Iosifidis. Heterogencous multilayer generalized operational perceptron. IEEE transactions on neural networks and learning systems, 2019.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in -araaallll pruning- quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7873-7882, 2018.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-20 200-2011 Dataset. Technical Report 200-TRR-211-000 Californi Institute of Technology, 2011.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Chenglin Yang, Lingxi Xie. Siyuan Qiao, and Alan Yuille. Knowledge distillation in generations: More tolerant teachers educate better students. arXiv preprint arXiv: a 8888805.5555 2018
11

---

Under review as a conference paper at ICLR 2021
Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. Knowledge extraction with observable data. In Advances in Neural Information Processing Systems. pp. 2701-2710, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1612.0392, 2016.
Zheng Zhang, Yong Xu, Jian Yang. Xuelong Li, and David Zhang. A survey of sparse representation: algorithms and applications. IEEE access, 3:490-530, 2015.
Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deeplob: Deep convolutional neural networks for limit order books. IEEE Transactions on Signal Processing, (7(11):300133022 2019.
Shu-Chang Zhou, Yu-Zhi Wang, He Wen, Qin-Yao He, and Yu-Heng Zou. Balanced quantization: An effective and efficient approach to quantized neural networks. Journal of Computer Science and Technology, 32(4):667-682, 2017.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. In Advances in neural information processing systems, pp. 7517-7527, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.
A CIFAR100
In this experiment, we used ADAM optimizer and trained networks for 200 epochs with the initial learning rate of 0.001, which is reduced by 0.1 epochs 31 and 131. For those methods that have pre-training phase, the number of epochs for pre-training was set to 160. For regularization, we used weight decay with coefficient 0.0001. For data augmentation, protocol, which randomly performs horizontal flipping, random horizontal and/or vertical shifting by four pixels. For SRM, KD and FitNet, we used our own implmentation, for other methods (AT, PKT, RKD, CRD), we used the code provided by the authors of CRD method (Tian et al., 2020).
followed the conventional
For KD, FitNet and SRM, we validated the temperature (used soften teacher's probability) and the weight a (used to balance between classification and distillation loss) from the following set: {2.0, 4.0,0 8.0). and (0.25, 0.5. 0.75). For other methods, we used the provided values used in CRD paper (Tian et al. 2020). In addition, there are two hyperparameters of SRM: the degree of sparsity = M)) and overcompleteness of dictionaries (μ MCCC). Lower values of indicate sparser representations while higher values of indicate larger dictionaries. For CIFAR100, we performed validation with E {0.01, 0.02, 0.03) and μ = ( 2,0, 3.0}.
B TRANSFER LEARNING
For each transfer learning dataset, we randomly sampled a few samples from the training set to establish the validation All images were resized resolution 224 224 and we used standard ImageNet data augmentation (random crop and horizontal flipping) during stochastic optimization. Based the analysis of hyperparameters in CIFAR 100 experiments, we validated KD, FitNet and SRM using a E {0.5, 0. 0.75) and 5 E {4.0, 8.0}. Sparsity degree À = 0.02 and dictionary size μ == 2.0 were used for SRM. For other methods, we used the hyperparameter settings provided in CRD paper (Tia et al. 2020). All experiments were conducted using ADAM optimizer for 200 epochs with the initial learning rate of 0.001, which is reduced by 0.1 epochs 41 and 161. The
weight decay coefficient was set to 0.0001.
C IMAGENET
For hyperparameters of SRM, μ 4.0. 0.02, 4.0, 0.3. For other training protocols, we followed the standard setup for ImageNet, which trains the student network for 100 epochs using SGD optimizer with initial learning rate of 0.1, dropping by 0.1 at epochs 51, 81, 91.
12

---

Under review as a conference paper at ICLR 2021
Table Transfer learning: standard deviation (%) test accuracy
Model
Flowers
CUB
Cars
Indoor-Scenes
PuhHig83
5-Shot
ACNN
00.94
00.46
00.11
00.81
00.45
ACNN-KD
00.76
00.83
00.52
00.67
01.40
ACNN-FitNet
01.18
00.60
00.37
00.52
01.36
ACNN-AT
00.88
00.94
01.50
00.39
00.70
ACNN-PKT
00.31
00.20
00.29
01.04
00.18
ACNN-RKD
00.27
00.34
00.31
00.39
00.16
ACNN-CRD
00.57
00.91
00.55
00.49
00.28
ACNN-SRM
01.09
01.18
01.21
00.34
00.52
RNI8
00.37
00.10
00.27
00.44
00.22
RNI8-KD
01.17
00.35
00.58
00.88
00.99
RNI8-FiNNe
00.62
00.46
01.00
01.07
00.66
RNI8-AT
01.23
00.47
00.92
00.73
00.34
RNI8-PKT
00.47
00.18
00.82
00.11
RNI8-RKD
00.55
00.40
00.13
00.49
00.24
RNI8-CRD
00.32
02.09
00.26
00.23
00.21
RN18-SRM
00.48
00.63
01.82
01.29
00.51
10-Shot
ACNNI8
02.33
01.07
00.82
00.46
00.19
ACNNI8-KD
00.39
00.35
01.47
00.53
00.50
ACNNI8-FiNNG
02.41
00.63
02.16
01.32
00.67
ACNNI8-AY
00.69
00.35
02290
00.40
01.76
ACNNI8-PKT
00.53
01.92
01.18
00.59
00.26
ACNNI8-RKD
01.10
00.50
00.25
00.76
00.57
ACNNI8 CRD
01.13
00.25
00.77
00.85
02.98
ACNNI8-SRM
00.58
01.39
04.33
00.47
01.84
RNI8
00.42
00.91
01.35
00.54
00.73
RN18-KD
00.33
00.53
01.13
00.78
00.91
RNI8-FiNNe
01.24
00.47
01.72
00.47
01.27
RNI8-AT
00.42
00.55
02.72
00.38
01.41
RNI8-PKT
02.56
00.63
00.62
00.54
RNI8- RNIS-RKD
00.28
00.14
00.71
00.34
00.27
ANI8-CRI
00.05
00.13
00.66
00.49
01.67
RN18-SRM
01.06
01.86
06.84
01.21
02.95
13

---

Under review as a conference paper at ICLR 2021
Weight decay coefficient was set to 0.0001. In addition, the pre-training phase took 80 epochs with an initial learning rate of 0.1, dropping by 0.1 for every 20 epochs.
References of the methods reported in Table are:
KD Hinton al. (2015)
AT Zagoruyk & Komodakis 2016)
Seq. KD Yang et al. (2018)
KD+ONE Zhu et al. (0188
ESKD Cho Hariharan 2019
ESKD+AT Cho & Hariharan
2019
CRD Tian et al.
2020
14


Reviews:
[{"review_id": "in-swVyEztP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] \nThis paper proposes to do knowledge distillation via matching the coefficients of sparse representations. The sparse representation reconstructs deep neural networks’ intermediate representation via a set of over-complete dictionary and corresponding coefficients. The method matches the coefficients at two levels. One is at the pixel level, and the other is at the image level, which pools the coefficients from the pixel level. The experiments perform model compression for image classification tasks and show a performance gain over previous methods.\n\n[Strengths]\n1. The idea of using sparse representation is novel and interesting.\n2. The clarity is good and easy to follow. The related works are sufficiently discussed.\n\n[Weaknesses]\n1. The method lacks some details, such as how D_T and D_S are learned and optimized. Please consider using an algorithm block to describe the full procedure of the proposed method.\n2. The experiment part has a large room to be improved. It is known that KD methods are sensitive to the type of architectures used in the teacher or student model. Moreover, most of the methods are very sensitive to the choice of hyper-parameters. There are two concerns: (1) Does the proposed method generalize to a broader range of network architectures pairs? (ex: the analysis in Tian et al. 2020) (2) Do all methods in the table be given the same amount of tuning budget for the hyper-parameter? Based on the text, different methods are given a different tuning budget. For example, Tables 1 and 5 only have part of the methods that are tuned with a validation set while others use the default value from other papers, making the comparison unfair.\n3. There are already more than a dozen distillation methods try to match the intermediate representation in a wide variety of ways, while most of them seem to work. Maybe matching a random projection of intermediate representation between teacher and student will work as well. Then why we want to use a more complicated method? The work will be more valuable if it provides a more in-depth insight into why using sparse representation helps. The paper has some arguments in the introduction, but they are not convincing since the coefficients can be the same even though the dictionaries between teacher and student are very different. A mathematical explanation is preferred. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YDrIDnDu_N2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength and weaknesses:\n+\nThe method is reasonable and competitive. Especially for the task transfer scenario is seems to advance the state of the art\n\n-\nComparison is lacking as a main relevant competitor was not tested\nI believe the method can be much simplified by instead of using sparse decomposition using plain clustering of pixel columns. Experiments testing this option should be conducted\nThe task transfer experiments, whose results are the most impressive, are done without weight Initialization of the student networks. It is not clear why, and the results may be different if initialized networks are used. \nAt the bottom line: cleaner experiments (adding comparison to Jain et al, using plain clustering instead of sparse decomposition, using student networks with ImageNet initialized weights) are required to take this paper beyond reasonable doubt. \n\n\nDetailed comments:\n\n-\tPage 2: “However, we argue that the intermediate feature maps by themselves are not a good representation of the knowledge encoded in the teacher to teach the students” – This sentence states a claim, which is a main claim of this paper. However, the claim is not supported by any argument or justification in the introduction\n-\tPage 3: the subscript \\Tau in D_{\\Tau}^{(l)} seems to be meaningless (why do we need it?)\n\tLater is becomes clearer as D_S is introduced, but a note should be given before to make \\Tau meaningful\n-\tPage 4: the pixel-labels are based on the 1-nn dictionary item, and so the knowledge transfer proposed in actually based a simple clustering of the pixels (each pixel represented by a single cluster index), not on the sparse representation. First, this means that K=1 can be naturally used, as K>1 is not really adding information to be transferred. Second, the method is very similar to transfer by clustering, seemed to be proposed by (Jain et al.) (based on the description of this work in the relevant work – I am not familiar with it)\n-\tPage 4: the image labels are enforcing similarity of word histograms between teacher and student. It is not clear, tough, why the BCE loss is used and not some histogram distance loss (l_1, Xi square, or even l_2). BCE is appropriate when the entrees in the two input vectors are probabilities of binary classification for independent problems. This is not the case here.\n-\tPage 5: it seems that comparison to the most similar method of (Jain et al.,) is not conducted \n-\tPage 6: the experiments showing that SRM is not sensitive to \\mu and \\lambda increase the likelihood that simple clustering, where each pixel is hard-assigned to one of the clusters can be used instead of sparse code encoding (we know that the identity of the single first dictionary item is the only one transferred in the pixel-labels, so it is likely that similarity to other dictionary items is not relevant)\n-\tThe task transfer results (table 5) are the most impressive results of the paper, indicating clear advantage of SRM over competitors. However, two drawbacks:\no\tWhy are the student networks randomly initialized and not initialized with their trained ImageNet weights (at least for ResNet 18 weights are publically available, I believe). Using pre-trained weight is likely to improve accuracy, and it is not clear why they are avoided\no\tComaprison to Jain et al is missing\n-\tSection 4.3: the teacher accuracy is missing\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EkDilRQvn3r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper summary**\nThis paper proposes a knowledge distillation on the feature maps using sparse representation. The proposed method firstly constructs an over-complement dictionary to express the teacher's feature maps and learn sparse representation to express the teacher's feature map using the dictionary. Since directly utilizing the sparse representation is a too strong restriction for the student network, the loss function is designed to find the indices of sparse codes. The proposed distillation method is validated through several experiments. \n\n**Pros**\n1. This paper proposes a way of utilizing sparse representation for knowledge distillation.\n2. The algorithm is written in clear formulations. \n\n**Cons**\n1. The main idea of sparse representation matching (SRM) is the combination of sparse representation and knowledge distillation. However, the actual implementation of SRM does not transfer the sparse representation of the teacher to the student. Only the indices or the entire image's sparse code were transferred via knowledge distillation, so the feature map information of the teacher is not transferred. This looks counter-intuitive and weakens the arguments of the paper. Therefore, it is necessary to describe the information transferred by SRM in detail and the rationale for using this kind of information transferring.\n2. All experiments were conducted on All-CNN, but this network is not usually used by other knowledge distillation papers, so the experiments should be re-conducted on a more standard and efficient setting. For example, All-CNN got 74.7% accuracy using SRM with 2.2M parameters (in table 1), but a recent paper [1] (CRD) got 75.5% accuracy using WRN16-2 with 0.7M parameters, which uses only 1/3 number of parameters. In other words, All-CNN is not proper to compare with other distillation methods. Use popular network architectures (WRN, ResNet, VGG etc.) to get more reasonable performance. \n[1] Contrastive Representation Distillation (ICLR 2020)\n3. In the case of transfer learning, it is common to tune the learning rates and weight decays for each dataset and each network. However, the experiments in the paper consistently use the same learning hyper-parameters. As a result, the performance gap between the baseline and the proposed method seems to have greatly inflated. A great gap against baseline (about 10%~20%) seems to be very different from the results of the other distillation papers.  In short, to be fair comparisons with other distillation methods, learning parameter tuning is necessary. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "E1QfDE1aQAq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Paper summary>\nThis paper focuses on the problem of knowledge transfer between deep learning models, with the goal of transferring some of the information contained in a (typically large) teacher model to a smaller student network, improving performance of the latter. In particular, this work proposes to require the sparse representations of the activations of the teacher model to be 'similar' to that of the student model, as way of knowledge transfer. The method is presented and empirically evaluated.\n\n<Review summary>\nThis reviewer likes the general motivation of this work, proposing that in some cases it might be better to enforce similarity between activations (or features) in a different domain, and requiring these representations to be sparse under some transformation is natural. While the general idea is appealing, several of the motivating claims are vague and the way these ideas are implemented (e.g. via classification to enforce similarity between vectors) are questionable.\n\n<Details comments>\nStrengths:\n- the authors study an interest problem.\n- the proposed method obtains good empirical performance.\n\nWeaknesses:\n- The idea of matching the representations for two different data in order to enforce some similarity (or in this context, 'knowledge transfer') has been extensively used. However, this only makes sense if the dictionaries for one and other case are related (see [1,2,3]). Enforcing the representations to be similar (or, as it's done here, to use the same leading atom) is reasonable when the atoms from one dictionary share some properties (or 'code' for related things) in the other.  In this work, the authors learn dictionaries for the teacher and student networks (Ds and Dt) completely independently, as there's no connection between the atoms in one and other dictionaries. \n- The authors include an 'image-level labeling' which basically compares the mean value in the approximate features from the teacher network ($\\tilde{t}$) to that of the vector of similarities of the student one ($k_s$). 1) I do not see how this is informative of relevant information between models, but more importantly 2) Comparing these two real numbers with a logistic loss makes little (if any) sense to me.\n- The idea of employing sparse representations for data (in this case, the intermediate representations of networks) is natural. There is a large body of work that the authors seem to ignore. For example: on pg 2 they mention that \"[sparse representations learning] were not proposed to be jointly optimized with other objectives\". Please see refs [4-6] below for examples of this.\n- Representations under redundant dictionaries are not unique, and the problem of finding sparse representations is NP-hard (see e.g [7]). Certainly, one can propose relaxations of this problem and even heuristic approximations (see e.g. [8]) but this is never discussed, and it is unclear how the obtained representations in this work fit in this context.\n\nSmaller comments:\n- The proposed method seems to be a pre-processing step: firs train teacher dictionaries, then student dictionaries and student weights, and then employ the KD method from Hinton et al. The authors should consider making this more explicit, perhaps detailing the full algorithm in a formal way.\n- On page 4, the authors motivate the use of the sigmoid as activation function saying that the 'gradients in the backward pass are stable'. What does this mean? Would they be unstable if a ReLU was used instead (as used in most deep learning models and in the other models in this work)?\n- At the end of Pixel-level labeling, stating the definition for $c_{n,i,j}$, the authors take the argmax over m, but there's no m in the expression.\n- 'Relaxing' the problem of requiring the representations to be similar simply by turning a regression problem into a classification problem seems unfounded: why would the latter be easier than simply allowing the representations to be similar (say, with small L2 norm)?\n- In defining the similarity kernel $\\kappa$, I believe the $x$ and $y$ should be bold according to the authors notation.\n- On a subjective note, the notation is not standard and thus a bit confusing: calligraphic capital letters usually denote sets or distributions, whereas here they denote vectors (as do bold non-calligraphic letters).\n\nReferences:\n1] Wang, Shenlong, et al. \"Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis.\" 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012.\n\n2] Qiu, Qiang, et al. \"Domain adaptive dictionary learning.\" European Conference on Computer Vision. Springer, Berlin, Heidelberg, 2012.\n\n3] Peleg, Tomer, and Michael Elad. \"A statistical prediction model based on sparse representations for single image super-resolution.\" IEEE transactions on image processing 23.6 (2014): 2569-2582.\n\n4] Mairal, Julien, Francis Bach, and Jean Ponce. \"Task-driven dictionary learning.\" IEEE transactions on pattern analysis and machine intelligence 34.4 (2011): 791-804.\n\n5] Sprechmann, Pablo, Alexander M. Bronstein, and Guillermo Sapiro. \"Learning efficient sparse and low rank models.\" IEEE transactions on pattern analysis and machine intelligence 37.9 (2015): 1821-1833.\n\n6] Monga, Vishal, Yuelong Li, and Yonina C. Eldar. \"Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing.\" arXiv preprint arXiv:1912.10557 (2019).\n\n7] Mairal, Julien, Francis Bach, and Jean Ponce. \"Sparse modeling for image and vision processing.\" \n\n8] Makhzani, Alireza, and Brendan Frey. \"K-sparse autoencoders.\" arXiv preprint arXiv:1312.5663 (2013).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received four negative reviews. The overall idea was found to be interesting, but several concerns were raised. There is a general consensus that the experimental part and the results are not convincing. Several comments have also been made regarding the clarity and motivation, which needs to be strengthened. R4 also mentions references from the sparse estimation literature that would help for positioning the paper. The rebuttal did address some of these points, but it was not sufficient to change their opinion.

Overall, the area chair agrees with the reviewers and follows their recommendation.
Decision: Reject