NORM: Knowledge Distillation via N-to-One Representation Matching
Authors: Xiaolong Liu
LUKING LI
Chao Li
Anbang Yao
Venue: ICLR 2023 poster (2023)
Keywords: Knowledge distillation
model compression
image classification

Abstract: Existing feature distillation methods commonly adopt the One-to-one Representation Matching between any pre-selected teacher-student layer pair. In this paper, we present $N$-to-$O$ne $R$epresentation $M$atching (NORM), a new two-stage knowledge distillation method, which relies on a simpleFeature Transform (FT) module consisting of two linear layers. In view of preserving the intact information learnt by the teacher network, during training, our FT module is merely inserted after the last convolutional layer of the student network. The first linear layer projects the student representation to a feature space having $N$ times feature channels than the teacher representation from the last convolutional layer, and the second linear layer contracts the expanded output back to the original feature space. By sequentially splitting the expanded student representation into $N$ non-overlapping feature segments having the same number of feature channels as the teacher's, they can be readily forced to approximate the intact teacher representation simultaneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair. After training, such an FT module will be naturally merged into the subsequent fully connected layer thanks to its linear property, introducing no extra parameters or architectural modifications to the student network at inference. Extensive experiments on different visual recognition benchmarks demonstrate the leading performance of our method. For instance, the ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches 72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an absolute improvement of 2.01%|4.63%|3.03% against the individually trained counterpart. Code is available at https://github.com/OSVAI/NORM.

Published as a conference paper at ICLR 2023
NORM: KNOWLEDGE DISTILLATION VIA N-TO-ONE REPRESENTATION MATCHING
Xiaolong Liu*, Lujun Li Chao Li Anbang Yao* Intel Labs China (x {xiaolong 1iu, lu jun. 11, chao3. 11, anbangg yao)e
ABSTRACT
Existing feature distillation methods commonly adopt the One-to-one Representa- tion Matching between any pre-selected teacher-student layer pair. In this paper, we present N-to-One Representation Matching (NORM), new two-stage knowl- edge distillation method, which relies on simple Feature Transform (FT) module consisting of two linear layers. In view of preserving the intact information learnt by the teacher network, during training, our FT module is merely inserted after the last convolutional layer the student network. The first linear layer projects the student representation to a feature space having N times feature channels than the teacher representation from the last convolutional layer, and the second linear layer contracts the expanded output back to the original feature space. By sequentially splitting the expanded student representation into N non-ovette oon-overlapping feature segments having the same number of feature channels as the teacher's s, they can be readily forced to approximate the intact teacher representation simul- taneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair. After training, such an FT module will be naturally merged into the subsequent fully connected layer thanks to its linear property, introducing no extra parameters or architectural modifications to the student network at inference. Extensive experiments on different visual recognition benchmarks demonstrate the leading performance of our method. For instance, the ResNetl 8 Ne 72. 14.11%//4.. 14.22% 68.6688888 03% top-1 accuracy the ImageNet dataset when using a pre- trained ResNet34 ResNeee0NN ReeeseeesORRReeetS00 ResNe150 model as the teacher, achieving an absolute improvement of 2.01 %
1/4 model trained by NORM reaches
against the individually trained counterpart. /tptt:s::ggtiin hhw.bb...httttggggggg..g........ com/ OSVAI DOVVIINNOM
Code is available at ht tes:pps/// /
INTRODUCTION
Knowledge distillation (KD), an effective way to train compact yet accurate neural networks through knowledge transfer, has attracted increasing research attention recently. |Bciil al. (2006) and [Ba & Caruana (2014) made early attempts in this direction. Hinton et al. (2015) presented the well-known KD using teacher-student framework, which starts with pre-training a large network (teacher), and then trains a smaller target network (student) on the same dataset by forcing it to match the logits predicted by the teacher model. Many subsequent methods follow this two-stage KD scheme but use hidden layer features as extra knowledge, while others use a one-stage KD scheme in which teacher and student networks are trained from scratch jointly (Guo al. 2021). In this paper, we focus on two-stage feature distillation (FD) research, mainly for supervised image classification tasks.
Existing two-stage FD methods primarily use feature maps (Romero al. (015). or attention maps (Zagoruyko & Komodakis 2017), or other forms of features (Chen et al. (021a) at one or multiple hidden layers as knowledge representations. Generally, modern neural network architectures engineered the ImageNet classification dataset (Russakovsky et al. 2015) adopt a multi-stage design paradigm. At a pair of the same staged layers, teacher network typically has more output feature channels than student network while keeping the same spatial feature size. All feature
XI., L.I. and CL. contributed to the hasic method implementations. XI. conducted the main experiments. AY proposed the original idea, supervised the project and led the paper writing. Corresponding author.

---

Published as a conference paper at ICLR 2023
Takeegg
The thontee Bock
Ohe insd edffnd
Riy Pia Or
An architectural comparison of prevailing One-to-one Representation Matching (ORM) schemes (left figure) and our N-to-One Representation Matching (NORM, right figure). Based on ORM, existing feature distillation methods apply their feature transforms (FTs) to (1) the student network, or (2) the teacher network, or (3) both of them, at any pre-selected hidden layer pair. Unlike them, NORM leverages a simple linear FT module added after the last convolutional layer of the student network to formulate many-to-one representation matching scheme via feature expansion, splitting and mimicking. For inference, our FT module will be merged into its subsequent fully connected layer, introducing no extra parameters or architectural modifications to the student network. Best viewed with zoom in, and see the Method section for a detailed formulation of NORM.
Figure 1:
dimensions may be different a cross-stage layer pair. To align the feature dimensions, there have been many feature transform (FT) designs. However, prevailing teacher FT designs cause information loss due dimension reduction, as studied in (Heo et al. 2019a Tian et al. 2020). More importantly, we observe that existing FD methods adopt the One-to-ore one Representation Matching (ORM) between each peleselected teacher-student layer pair, indicating that only one knowledge transfer route is introduced, We argue that this leaves considerable room to promote two-stage FD research.
Driven by the above analysis, in this paper, we present a new two-stage feature distillation method dubbed N-to-One Representation Matching (NORM) that relies on a simple FT module consisting of two linear layers. An architectural comparison of popular ORM schemes and NORM is depicted in Figure 1 When formulating NORM, we leverage three basic principles: (1) using as few FTs as possible; (2) enabling many-to-one feature mimicking flow via student representation expansion and splitting: (3) making FT module absorbable. With the first principle, our FT module is merely inserted after the last convolutional layer of the student network. In this way, the intact information learnt by the teacher network is preserved, and knowledge transfer flow only needs to be considered between the last convolutional layer pair. With the second principle, our FT module starts with a linear layer that projects the student representation to feature space having N times feature channels than the teacher representation. This allows NORM to introduce many parallel knowledge transfer routes between a single teacher-student layer pair via simple student feature splitting and group-wise feature mimicking operations. With the third principle, our FT module ends with another linear layer that projects the expanded student representation back to the original feature space, and it does not contain any non-linear activation functions, making all of its operations linear. As result, after training the FT module can be directly merged into its subsequent fully connected layer, without introducing any extra parameters or architectural modifications to the student network at inference.
We evaluate the performance of NORM on different visual recognition benchmarks. On the CIFAR- 100 dataset, the student models trained by NORM show a mean accuracy improvement of 2.88% over teacher-student pairs of the same type network architectures. Over teacher-student pairs of ifferent type network architectures, the mean accuracy improvement reaches 5.81%. and the maximal gain is 6. 6.92%. Leading results are obtained on the large-scale ImageNet dataset. With NORM, the ResNet18 18 Ressiie00 1/4 model reaches 72.14% 74.26% (6..0%% top-1 accuracy when us- ResNet50
ing a pre-trained ResNet34 Resset00 model as the teacher, showing 2.01 % /4.63%//.... (3.0%% absolute gain to the baseline model. Thanks to its simplicity and compatibility, we show that improved performance could be further attained by combining NORM with other popular distillation strategies like logits based supervision (Hinton et al. 2013) and contrastive learning (Tian et al. 2020).
2 RELATED WORK
Two-stage KD methods. This category of KD methods first assumes that a pre-trained teacher network is available, and then uses its learnt representation extra supervision to guide the training
2

---

Published as a conference paper at ICLR 2023
of student network. The vanilla KD (Hinton et al. 2015 uses the logits output from the teacher network as soft supervision. FitNets (Romero et al. 2015) show that the feature maps from hidden layers can also be used as hints to improve distillation performance. AT (Zagoruyko & Komodakis 2017) uses spatial attention maps instead of source feature maps. Many follow-up methods intend to
improve the knowledge representation via different techniques, such as feature encoding (Yim et al.
2017: Chen et al.) 202 Srinivas & Fleuret 2018). feature selection (Heo et al.) 2019a b Chen
Chen
al.
202 distribution learning (Ahn et al. 2019 Huang & Wang 2017: Passalis & Tefas 2018
Malinin et al. 2020 Yang et al. 2021), attention rephrasing (Kim et al. 20188 Ji et al. 202 and
reuse of teacher classifier (Chen al. (022). Besides, some works (Peng et al. 2019] Yin et al.
2020)
Park et al. 2019; Tung & Mori 2019) explore the use of sample relations. It is also worth noting that
several recent works (uue et al. 2020, Liu et al. 2020b; Deng et al. 2022) propose reinforcement
learning based searching methods to improve feature distillation process.
One-stage KD methods. This category of KD methods adopts an online training framework that does not require to pre-train the teacher model. ONE (Lan et al. 2018) presents multi-branch training strategy using an on-the-fly branch ensemble to guide the training of individual branches. DML (Zhang al. 2018b) uses mutual learning strategy to jointly train set of peer models from scratch. During traming, every peer model acts as a teacher of the others. DCM (Yao & Sun (2020) shows that the logits from auxiliary branches of teacher and student networks can improve mutual distillation performance. Many one-stage KD variants have been presented recently, including but
not limited (Guo et al. 2020a, Chung et al., 2020 Malinin et al. 2020 Wu & Gong 202I)
Other KD variants. Extending knowledge distillation methodology from standard supervised learning other learning scenarios has gained broad attention. BANs (Furlanello al. 2018) improve
the training of a target network via progressive self-distillation formulation. CRD (Tian et al. 2020)
formulates an effective structural representation matching loss based on contrastive learning. Xu
et al. (2020) further presented way to combine self-supervision and contrastive learning. Besides, there also exist various KD variants for life-long learning (Li & Hoiem 2016), semi-supervised
learning (Pham et al. (02T), few-shot learning (Shen et al. (001), data-free learning (Lopes et al.
2017). adversarial learning (Chung et al. 2020) and distributed learning (Anil et al. 2018). Although these methods are still tailored to image classification, KD has also been applied to handle other tasks, such as semantic segmentation (Liu et al. 2019b), object detection (Wang et al. 2019). image super-resolution (Liu et al. 2020a), and neural machine translation (Kim & Rush (016)).
Our work focuses on two-stage KD research, mainly for supervised image classification tasks. Specifically, we attempt to shift the prevailing one-to-one feature representation matching paradigm to a many-to-one alternative conditioned on a single teacher-student layer pair. This makes our method differ with existing KD methods both in motivation and formulation.
METHOD
3.1 BACKGROUND: ONE-TO-ONE REPRESENTATION MATCHING
We first review the representation matching of existing two-stage feature distillation (FD) methods in general formulation. Suppose that have a pre-trained teacher network T, a target student network S, a given image classification dataset X and its ground truth label set Y. For a pre-selected teacher-student layer pair (two same staged layers of T' and S, as default), let F. IR HX xC; F,, R X xC, denote the output feature maps for the teacher sttuent network, where H, W and C, are the channel height, width and number, respectively. Let T. denote the feature transform (FT) for the teacher jeacher|student istudent network, which projects F, to the same feature space, respectively. Then, the representation matching loss of existing FD) methods, in general, can be defined as:
Laf = (F., (FF.).
Various distance metrics d(.). such as 12-morm distance (Romero et al. 2015 Zagoruyko & Ko- mooaais 2017 Heo et al., 2019b). / -norm distance (Kim et 2018) and maximum mean discrep- ancy (Huang& & Wang 2017) are popularly used in FD research. Furthermore, existing FDD methods usually apply the feature representation matching to one or multiple pre-selected teacher-studens layer pairs. Under this context, it is clear that the of FT and the design of FT are two key problems for the feature representation matching. To the first problem, there are three choices: applying one or multiple FT's to (1) the student network (Romero et al. 2015); (2) the teacher network (Yue et al.
3

---

Published as a conference paper at ICLR 2023
2020); (3) both teacher and student networks (Zagoruyko & Komodakis 2017 Heo et al. 2019b
Srinivas & Fleuret 2018: Heo et al. 2019a Chen et al., 2021a). They are illustrated in Figure (left figure). Comparatively, the last choice much more popular than the other two. To the second problem, there exist many FT designs (Guo et al.
2021) that generate aligned feature maps, or atten- tion maps, or other forms of features used as the knowledge representation for matching. However, according to [ we can see that existing FD methods use the One-to-one Representation Matching (ORM). More specifically, they perform global single-shot feature mimicking process, meaning that there exists only one knowledge transfer route between any reacher-studen layer pair. We conjecture that if we can formulate an effective mechanism to introduce multiple global knowledge transfer routes between any teacher-studen layer pair, it will offer more chances to inject the intact teacher's knowledge into the student network, and improved FD performance could be attained.
3.2 N-TO-ONE REPRESENTATION MATCHING
Motivated by the above analysis, we present N-to-One Representation Matching (NORM), a new two-stage feature distillation method that relies on simple FT module. In the formulation of NORM, we rethink the representation matching mechanism from the following perspectives: (1) regarding the use of FT, where to place it? (2) regarding the design of FT, how to adapt for introducing many parallel global knowledge transfer routes between any pre-selected teacher-student layer pair? tradeoff? To the first and third questions, our basic principle is "using as few FTs as possible", partially inspired by (Romero et al. 2015 Heo et al., 2019a] Tian et al. 2020). Accordingly, we merely insert an FT module after the last convolutional layer of the student network. In this way. NORM preserves the intact information learnt by the pre-trained teacher network, and knowledge transfer flow only needs to be considered between the last convolutional layer pair. Note that the design of directly plugging an FT module into the student network is in sharp contrast to existing FD methods which typically use their FT's auxiliary branches that will be discarded at inference. To the second question, our basic principle is "enabling many-to-one knowledge mimicking flow via student representation expansion and splitting" Accordingly, we construct student FT module which starts with linear layer that projects the student representation to a feature space having N times feature channels than the teacher representation. By sequentially splitting the expanded student representation into N non-overlapping segments having the same number of feature channels as the teacher's, s, we can force them to approximate the intact teacher representation simultaneously, formulating an effective many-to-one feature matching mechanism conditioned on a single teacher-student layer pair. To the third question, we additionally use a "making FT module absorbable" " principle. Accordingly, our student FT module ends with another linear layer that projects the expanded student representation back to the original feature space, and it does not contain any non-linear activation functions. As result, such an FT module will be naturally merged into its subsequent fully connected layer after training, without introducing any extra parameters or architectural modifications to the student network at inference. An overview of NORM is depicted in Figure (right figure).
(3) regarding the distillation performance, how to make the accuracy-efficiene
Now, we provide the formulation of NORM following the notations in Eq. [ Given a teacher-student layer pair (last convolutional layer pair, default) with the output feature maps F, E R xC. and F. E R xC, let T, (W se, W (se) denote our student FT module consisting of two linear layers. The first linear layer uses a X 1 convolutional kernel W.c E glx1xC,XNC XNC, along the channel dimension to project each pixel in F. to desired channel dimension NC,, producing an expanded student representation F.. E I HxW x NC. having N times feature channels than the teacher representation. The second linear layer uses another 1 x 1 convolutional kernel Wes sc E IR 1x NC, xC, to project each pixel in F.n back to the original channel dimension C., producing Fsc. E HxCx, Mathematically, the student FT module with the input F, can be written as:
F.c Wse * Ps, Fsc W sc F
where * denotes the convolution operation. Next, we sequentially split the expanded student rep- resentation F.e into N non-overlapping segments F E R number of feature channels as the teacher's. This allows us to force N student feature segments to approximate the intact teacher representation simultancously by minimizing an ly no-mor distance metric, formulating our many-to-one representation matching mechanism conditioned on a single
N having the same
4

---

Published as a conference paper at ICLR 2023
teacher-student layer pair. Specifically, our many-to-one representation matching loss is defined as:
N
Lnorm
Σ N
1
Fell..
(3)
and the total training loss of NORM to be minimized defined as:
Liotal = Lee + aLnorrs
(4)
where Lee denotes the standard cross-entropy loss of the student network supervised by the ground truth labels, and a is positive coefficient (a 10, as default) to weight the loss Lnorm-
Interprefation of NORM. The formulation of NORM can be interpreted as a novel way of learning a dynamically mixed feature ensemble over multiple augmented views of the same student represen- tation by forcing them to mimic the intact teacher representation simultaneously. More precisely, in NORM, the first linear layer W.c acts as set of independently initialized feature transforms to generate N channel- exxaadee views of the student feature F, whose representation abilities are then parallelly augmented by the distillation supervision from the intact teacher feature F. and the second linear layer Wsc performs a learnable ensemble of N Distilattion-uugmnnte student feature views via fully connected channel mixing operations (which make the feature ensemble F,e has the same size to F., guaranteeing the absorbable property of NORM at inference). Our mixed student feature ensemble learning further benefits from the standard cross-entropy loss supervised by the ground truth labels. In order to understand what enables the distillation effectiveness, we systematically study the major components of NORM in the Experiments section.
Augmented NORM, Thanks to its simplicity, NORM can be easily augmented by additionally introducing a vanilla logits based KD loss (at the network head) (Hinton al. 2015) into Eq. 4
Leotal = Lee + abborrm + BLadd
(5)
where Lid matches the logits distribution of the student network to target distribution produced by the teacher network, and | is a positive coefficient (3 = 4, as default). Besides, NORM can be also combined with contrastive KD loss (Tian et al. 2020) for improved results, tested in experiments.
Implementation and Inference. As our FT module does not have any non-linear activation functions, we empirically find that inserting it after the last convolutional layer of different student networks will mostly lead to obvious accuracy drop in the standard training regime (individually train the student model), but with NORM the performance of final student models will be improved significantly. To suppress the accuracy drop issue, we add a linear residual connection (identity mapping, as shown in Figure from the input to the output of our student FT module, while maintaining the absorbable property. In implementation, we use this student FT module as our default setting to NORM, and set N = 8 for all main experiments (the choice of N is studied in Figure 2 For inference, let denote the learnt parameters of the fully connected (FC) layer after the student FT module T. (Wse, se, Ws.), we can directly merge the student FT module into its subsequent FC layer by Wf WWe(W.e sc W.e + I), where I CC. is identity matrix. Note that modern neural networks typically have a global average pooling layer before the FC layer, and it does not affect the absorbable property of our student FT module inference. We put the proofs of them in the Appendix.
Wfe E
Differences with network re-parameterization methods. The absorbable property of our student FT module is based the network re- ce-parameterization. In recent years, there have been lots of network re-parameterization methods that are presented in the deep learning field. ACNet (Ding et al. 2019) uses an absorbable multi-branch block based on ID asymmetric convolutions to replace the square convolution. RepVGG (Ding et al. 202T) presents an absorbable multi-branch block to replace stack of 3 x 3 convolutions in VGG-like networks (Simonyan & Zisserman 2015). This category of methods focuses on strengthening the learning power of basic convolutions via equivalent multi- branch alternatives, particularly from the perspective of network structure engineering in the standard training regime. Clearly, our method differs with them both in focus, formulation and application. ExpandNets (Guo et al. 2020b) and WIN (Zhou et al. 2020) leverage paeeterraettiinntioonn designs based on linear expansion and contraction to improve the training of a thin network, which are more closely related to our work. ExpandNets expand all convolutional and FC layers during training, and convert the expanded network back to the original one inference. WIN first trains wider network generated by uniformly expanding the width of all building blocks in a given thin network, and uses it as the teacher. Then, inserts a pair of linear expansion and contraction layers between
5

---

Published as a conference paper at ICLR 2023
Table I: T I mean accuracy (%) comparison on CIFAR-100. The teacher and student have the same type network architectures. The results of the current mainstream KD methods are obtained from the papers of CRD, SemCKD, ReviewKD, SimKD and DistPro. The plain FT and the default FT denote our student feature transform module without and with a linear residual connection, respectively. NORM+KD and NORM+CRD denote combining NORM with the vanilla logits based KD and the contrastive KD, respectively. The best and second best results are bolded and underlined, respectively.
Texcher
WRN--00-2
WRN-40-2
56 ResNe:20
ReeNet 10 RecNec20
ResNenl 10 ResNee322
ResNet ResNetlad
VGGI3
Studett
WRN 16
WRN
VGG8
Teacher
75.61
75.61
74,31
74.31
79.42
74.64
Stddant (eppotedd
75.26
69.06
72.50
Studant cpreducca)
75.80
69:53
72.87
70.75
plain FTy
72.59
68.09
73.51
70.17
Student owr detuult FT)
73:72
72.09
09.55
73.72
70.64
KD
74-92
70.67
73.08
73.23
72.98
HoNer
73:59
72.24
68.99
71,06
73.50
71,02
74.08
70:23
72.91
73.44
71.43
SP
73.83
72.43
69.50
70.01
72.69
72.94
72.68
CC
73.56
72.21
69.63
69.48
71.48
72.97
70.71
VID
14:11
73, 30
70.36 38
70. 16
72.61
73,00
71.23
RKD
73.35
72.23
69.
69. 25
71. 82
71
A8
PKT
74.54
73.45
70.34
70.25
72
73.64
72.88
AB
72.50
72.38
69. 53
70.98
73.17
70.54
FT
73.25
71.59
66.
70.23
72.5 37
72.86
70.58
FSP
72.91
60.
70.11
71 89
72.62
70.23
NST
73.68
72.24
69.
69.55 553
71.96
73.20
71.53
CRD
75.48
46
75.51
SRRL
75.30
ScnCKD
76.23
74.43
ReviewKD
76.12
SDmKD
n/a
75.56
Bia
78,08
B/a
DiaPro
76.36
1978
72,03
1651
73.74
NORM (wl plasin FT)
75:57
74.78
21:01
73.27
76.76
73.64
75:65
74.82
71 55
73.67
76:49
73.05
NORM-KD
26:26
75.42
71.61
72.00
73.95
76.98
74,46
NORMACRD
76.02
75.37
71.51
21.90
73.81
76.49
73.58
any two neighboring building blocks of the given thin network, and considers the sequential stacks of expanded building blocks as set of subnetworks, which are trained progressively one by one using the one-to-one feature representation matching. ExpandNets and WIN all adopt the vanilla KD (Hinton al. 2015) further fine-tune their trained models. In sharp contrast them, method aims to advance two-stage feature distillation research by presenting a novel many-to-ons representation matching strategy conditioned on single teacher-student layer pair. Accordingly, our method inserts only one linear FT module to the last convolutional layer of a student network, without need of multi-layer feature mimicking and progressive training with multiple restarts. Furthermore, our method is applicable to various teacher-student pairs with both the same type and different type network architectures. Comparatively, our method is a more simple and easy to use, and it (without the vanilla KD) outperforms ExpandNets and WIN with large margins on ImageNet (see Table 6)
4 EXPERIMENTS
4.1 PERFORMANCE ON IMAGE CLASSIFICATION TASK
Datasets and experimental setups. We use CIFAR- 100 (Krizhessky & Hinton, 2009) and Ima- geNet (Russakovsky et al., 2015) datasets for basic experiments. CIFAR-100, which consists of 50,000 training images and 10,000 test images with 100 classes, is a popular classification dataset for KD research. Following the settings of CRD (Tian et al. 2020), we use 13 teacher-student pairs having either the same type or different type network architectures (see Table 1 for experiments. Each experiment is conducted for 5 separate runs, and we report top-1 mean recognition rate on the test set. ImageNet contains over 1.2 million images for training and 50,000 images for validation, including 1.000 image classes. Comparatively, ImageNet much more challenging than CIFAR- -00. Following the settings of Tian et al. 2020);; Yang et al. (202T): [Cen et al. (2021b) we use 2 popular teacher-student pairs (see Table for experiments. We report top- 1 recognition rate on the validation set. For comprehensive comparisons, we compare our method with the current mainstream two-stage and one-stage KD methods, including the vanilla KD (Hinton et al. 2015), FitNets (Romero et al. 2015), AT (Zagoruyko & Komodakis 2017). SP (Tung & Mori] 2019). CC (Peng et al.] 2019)
VID (hh et al. 2019)) RKD (Park et al., 2019). 1). PKT (Passalis & Tefas 2018). AB (Heo et al.
2019b), FT (Kim et al. 2018), FSP ( Yim et al. (017). NST (Huang & Wang 2017). CRD (Tian et al.
2020y, OFD (Heo et al. (019)). SSKD (Xu et al. 2020). ONE (Lan et al.] 2018), PCL (Wu & Gong
2021 SRRL (Yang et al. 2021), Reviewki DD (Chen al. 2021b), SemCKD (Chen et al.
2021a)

---

Published as a conference paper at ICLR 2023
Table 2: Top-1
I mean accuracy (%) comparison CIFAR-100 The teacher and student have different
type network architectures. Basic settings are the same to those described in the caption of Table
Teacher
VOG11
RosNet50 MohileNerV2
ResNec3U VIGG
ResNet3Zic4 ShafflNNeVV
ResNetS224 ShamBoNerV2
WRN -RRN-40-2
Suuden
MobiloNarV2
ShafflNNerV
Teacher
74. 64
79.34
79. 34
79.42
70-42
75.61
Suuken (oeported in CRD)
64.60
64.50
70:36
70.50
12
70.90
Studen reproduced)
64.81
64.81
70.75
7L.63
72.95
71.63
Sudden (wl plain PT)
63.95
63.95
70.17
71.82
72.55
71.82
Student detaulk FT)
64.13
64.33
70,64
72.71
KD
67.37
67.35
73.81
74.07
74.65
74.83
Huleet
63.36
70.00
73.54
AT
$8.58
72.75
74.56
71.29
67.57
70.30
04.52
04.43
71.50
72.21
67.13
66.52
73.01
AR
06.05
67.2
70.65
73.34
61.78
60.99
70.29
71.75
72.50
72.03
NST
58.10 16
64.56
71.28
74,12
74 88
24.89
CRD
69.73
69.11
74,30
75.11
73.65
76.05
SRRL
m/
75,18
n/a
SemCKD
1/2
77.82
ReviowKD KD
70.37
69
77.45
77.78
77.14
SinKD
ada
77.18
nda
DisPPr
R/a
I/as
m/a
77.18
77.54
27.24
NORM (wl plain FT)
69:37
70.94
74.57
75.93
77.8
76.61
NORM
68.94
70.56
75.17
77.42
78.07
77.06
NORM-KD
69.38
71.17
75.67
77.79
75.32
77.63
NORM+CRD
69.17
21.08
25.51
77.50
77.96
77,09
Table 3: Top-1 1 accuracy (%) comparison on ImageNet. The results in the bracket are for our reproduced student baselines, and the results of the current mainstream KD methods are obtained from the papers of CRD, SSKD, SRRL, SemCKD, ReviewKD, SimKD and DistPro.
Teadees RosheeT 37.31
Stalent
S5KD
AT
ORD
SaaCKD
ReriewDD
SinKD
DeaPo
RecNont (301T))
89.00 70.67
30-55
13.47
71 K2
TD:59
71.0
71.37
71:33
708T
11
11.66
71
73.14
RasNeat 76.14
MarroNor (6963)
30
76.31
70.30
71.08
77.49
79.56
09
21366
74.36
SimKD (Chen et al., 2022) and DistPro (Deng al.) 2022). All experiments are implemented with PyTorch (Paszke et al. 2019). Experimental details are put in the Appendix.
Results on CIFAR-100. Table shows the results comparison CIFAR- 100 with 7 teacher-student pairs having the same type network architectures. In average, NORM brings 2.88% top-1 gain to the baseline student models, with the maximal gain of 3.99%. Table 2 provides the results comparison with 6 teacher-student pairs having different type network architectures. In average, NORM brings 5.81% top-1 gain the baseline student models, with the maximal gain of 6. 6.92%. Generally, NORM shows very competitive results compared to the current mainstream KD methods which usually adopt multi-layer feature distillation schemes. Note that many top KD methods use the vanilla logits based distillation (Hinton et al.) 2015) as the extra supervision. Besides, CRD, SRRL and SSKD utilize the contrastive learning to augment knowledge distillation process. In Table I and Table [ we also explore the compatibility of NORM with these two types of distillation regularization (denoted as ND and NORM+CRD). We can see that: (1) the vanilla logits based distillation can further improve the performance of NORM, showing 0.46% extra gain in average, with the maximal extra gain of 0.61%, %; (2) the contrastive learning can also improve the performance of NORM most cases, showing 0. 0.18% extra gain in average, with the maximal extra gain of 0.55%. Finally, NORM+KD achieves the best and the second best results on 7 and 4 of 13 teacher-studes pairs, respectively.
Results on ImageNet. Table shows the results comparison on ImageNet. We can see that our method always shows the best performance compared to these competing KD methods. Specifically, for the teacher-student pair of ResNeer34>RReNeet
NORM improves top-1 accuracy of the student model from 70.13% to 72.10 14%, outperforming the current best method by a margin of 0.25%. Taking pre-trained ResNet50 as the teacher model and MobileNet (Howard et al., 2017) as the target student, the MobileNet model trained by NORM attains 74.26% I accuracy, showing 4.63% gain to the baseline model trained individually. Compared to ResNet34 Nette... the capacity gap of the teacher and student network becomes larger for ResNet50 MeeaNee5--MMobllen -obileNet. Under this context, the top-1 margin of NORM against the current best method is pronounced, reaching 1.00%.
4.2 ABLATION STUDY
To have deep analysis of NORM, we further provide a lot of ablative experiments mostly performed on CIFAR- 100, unless otherwise stated. For the experiments on CIFAR- 100, we run our method times for each setting with random initialized seeds, and report top-1 mean recognition rate.
7

---

Published as a conference paper at ICLR 2023
170
76.5
Tope Maan Anurar (k0
Moan Assuayy PS
76.0
360
25.5
75.0-
150
74.5
140
3/6
1/8
14
1/2
10
12
15
18
20
Figure The selection of N. On CIFAR-100 Figure The selection of a. On CIFAR-I00 with RessNet32x4-Reshh ResNetx8x, compare the with ResNet32x4-RRee performance of NORM with different N settings. performance of NORM (N 8) by changing a.
we compare the
Table 4: Top-1 accuracy (%) comparison of ap- plying NORM more teacher-student layer pairs. On ImageNet with ResNet34-RResNe
37.0
we in-
Monn Accomey
76.5
sert 2 FT modules (N 8) after conv5_x x (our default design) and conv4_x x of ResNet18, and compare the following two settings: (1) using NORM to the same staged teacher-student layer pairs; (2) forcing the student representations ex- panded from conv4_x and conv5_x to match the teacher representation learnt from conv5_x. Plain
760
75.5
2.9
3.3
4.3
518
6.5
78
Segrores Sor reprreeeeetttoo matching (Ni-k)
Figure 4: The role of many-to-one representation matching. On CIFAR- 100 with ResNet32x4 ResNet8x4, we compare the performance of NOR- M using (IşnSN) of N 8 student feature segments to match the teacher representation.
FT does not have linear residual connection.
Layer pait#
1 (dcfaul))
(same staged)
(eroes staged)
NORM plain NORM
72.14
72.03
71.49
The selection of N and a. our formulation, NORM has two hyper-parameters: N to control the desired number of representation matching routes between single teacher-student layer pair and to weight the many-to-one representation matching loss Lnorm. Accordingly, our first two sets of ablative experiments on CIFAR- are conducted for the selection of N and a. Specifically, use a teacher-stur studdntt pair of comparison of NORM with different settings of N and a, respectively. From Figure [ we can observe that: (1) increasing N from to 2 brings 2.28% extra accuracy gain; (2) this gain becomes larger when N gradually increases, and reaches the first peak value at 8 (3.02% to N 1 and 3. 3.62% to the baseline student model). To balance training accuracy and efficiency, we set N = 8 for the experiments both on CIFAR-I00 and ImageNet. According to Figure] we typically set a = 10 for the experiments on CIFAR-100. For the experiments on ImageNet, we empirically set 8.
Figure 2
and Figure 3 the performance
The role of the many-to-one representation matching. In NORM, the many-to-one representation matching is performed after sequentially splitting the expanded student representation into N non- overlapping segments having the same number of feature channels as the teacher's. Given N student segments, it necessary to force them simultaneously approximate the teacher representation? Again, we use the teacher-student pair of question. From Figure 4 we can see: the more the segments used to match the teacher representation, the larger the accuracy gain. Specifically, using all 8 segments outperforms using only of 8 segments by margin of 0.98%. Moreover, using of 8 segments for representation matching (75.51% vs. 73.47%) is much better than N = in Figure These results validate the advantage of our design.
with N = 8 to explore this
Applying NORM to more teacher-student layer pairs. Recall that NORM performs the proposed -oneeet--neen representation matching between single teacher-studen layer pair, as our FT module merely inserted after the last convolutional layer of the student network. A natural question how about the performance when applying NORM to multiple teacher-student layer pairs. We study this question on ImageNet with ResNet34 as teacher and ResNet18 as student. In the experiments, we add two FT modules (N 8) after conv5_x (our default design) and conv4_x of ResNetl 8, and consider the following two settings: (1) applying NORM to the same staged teacher-student layer pairs simultancously; (2) forcing two different staged student representations to simultaneously match the teacher representation from conv5 x. Architectural illustrations of these two NORM variants referred to the Appendix. From the results shown in Table 4 we can that these two variant
8

---

Published as a conference paper at ICLR 2023
Table The effect of linear FT modules to top-1 baseline accuracy (%) On ImageNet, we insert 1 or our linear FT modules into the ResNet- 18 net- work, and train it from scratch individually. Plain FT does not have a linear residual connection.
Table Top-1 accuracy (%) comparison of NOR- M with network re-parameterization methods us- ing KD. The results of ExpandNets and Win on
ImageNet are obtained from their original paper-
8 (Guo et al.
2020b; Zhou et al. 2020).
FT module number 8 (wl plain FT)
O (buseline)
Student model
ExpandNew>KI
WIN+KD
NORM
70.13
69.20
68.70
MobileNet ResNet50-144
70.47
N/A
74.26
ResNet18 (w/ default
70.13
70.40
70.39
N/A
67.50
68.03
designs lead to worse results than our default design no matter the FT module has a linear residual connection not, although both of them improve the student accuracy. At first blush, this might seem surprising. since multi-layer feature matching strategies are widely used in KD research (Zagoruyko)
& Komodakis 2017;] Yim et al. 2017 |Hoo et al., 2019b) Srinivas & Fleuret (0188) HHoo al. 2019a Chen et al. 2021a). As we explained next, this is due to our linear FT modules.
The effect of linear FT modules. Someone may concern that the model accuracy improvement may mainly from inserting linear FT module into each student network. Accordingly. we conduct ablative experiments on ImageNet to explore this concern. In the experiments, we add 1 or linear FT modules into ResNet18 (after conv4_x and conv5_x) x) first, then train each of them from scratch individually. We consider the linear FT module without or with a linear residual connection, separately. Table 3Jsowssssww the results. We can observe that adding one linear FT module without a linear residual connection into ResNer18 leads to obvious accuracy drop (0.93%), and the accuracy drop becomes more serious when adding two this type FT modules. The accuracy drop issue is suppressed by our default FT module having a linear residual connection which only brings marginal model accuracy improvement. Similarly, adding two default FT modules into ResNet18 8 shows slightly worse model accuracy compared to just adding one. Furthermore, in Table D and Table 2 we provide the individually trained model results when adding two types of FT modules separately to many different student networks on CIFAR- 100, and similar observations can be found. These experimental observations are mainly due to the structure of our FT modules. Note that our FT design does not contain popular non-linear activation functions which are critical to stabilize and improve the training process, in order to gain the absorbable property for maintaining efficient inference. Because of this, applying NORM to more reacher-student layer pairs does not achieve further improved performance compared to NORM conditioned on the last convolutional pair, as shown in Table
Performance comparison with network re-araamteerization methods. In Section 3.2 we discuss the connections and differences of NORM and network re- ee-paaamtterization methods. In Table 6 we further compare their performance on ImageNet. Clearly, our method obtains more accurate student models than top network re- rerarammetrrizatoon methods that also apply knowledge distillation during training. Specifically, for the MobileNet (student), NORM without KD reaches 74.26% top-1 accuracy with pre-trained ResNet50 as teacher, while ExpandNets (Guo et al. 2020b with KD gets 70.47% top-1 accuracy using pre-trained ResNet152 as teacher; for the ResNet50- 1/4 (student), NORM without KD reaches 68.03% top-1 accuracy with a pre-trained ResNet50 (teacher), while WIN (Zhou et al.) 2020) with KD gets 67.50% top-1 accuracy using a pre-trained wide teacher which is 4 times larger than the student.
More experiments and discussions. Please note that in the Appendix, we provide more ablative experiments, for a better understanding of NORM. The limitations of NORM are also discussed.
5 CONCLUSION
In this paper, we present NORM, a new two-stage feature distillation method. It relies on linear feature transform module inserted after the last convolutional layer of the student network, and enables a novel many-to-one representation matching mechanism conditioned on single teacher- student layer pair via feature expansion, splitting and group-wise mimicking. Thanks to its linear property, after training such feature transform module will be naturally merged into the subsequent FC layer, maintaining the same student network architecture at inference. Extensive experiments on popular image recognition benchmarks show that NORM can attain promising performance in both distillation accuracy and efficiency. We hope NORM would inspire the community to pay more attention to many-to-one representation matching research.

---

Published as a conference paper at ICLR 2023
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. In ICLR, 2018.
Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers, In ICLR, 2022.
Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Dafang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifer. CVPR, 2022.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Feng Yan, and Chun Chen. Cross-layer distillation with semantic calibration. In AAAI, 2021a.
Liang- Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint AxXiv1700.005580 2017.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
Inseop Chung, SeongUk Park, Jangho Kim, and Nojun Kwak. Feature-map-level online adversarial knowledge distillation. In ICMI, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. Microsoft coco: Common objects in context. In CVPR, 2016.
Xueqing Deng, Dawei Sun, Shawn Newsam, and Peng Wang. Distpro: Searching fast knowledge distillation process via meta optimization. arXiv preprint wwxXiv:204.00544/ 2022.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In ICCV, 2019.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In CVPR, 2021.
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In ICML, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learnin.. In NeurlPS, 2020.
Jianping Guo, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 2021.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo.
Online knowledge distillation via collaborative learning. In CVPR, 2020a.
10

---

Published as a conference paper at ICLR 2023
Shuxuan Guo, Yuchen Guo, Guiguang Ding, and Jungong Han. Expandnets: Linear parameterization to train compact convolutional networks. In NeurlPS, 2020b.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019a.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint DoXiv:1503.02533 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704:0/704. 2017.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer.
arXiv preprint arXiv: arXiv: 777700229, 2017.
Mingji Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via attention-based feature matching. In AAAI, 2021.
Jangho Kim, ScoungUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. In NeurlPS, 2018.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In EMNLP, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble, In NeurlPS, 2018.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung- Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017a
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, 2017b.
Jie Liu, Jie Tang, and Gangshan Wu. Residual feature distillation network for lightweight image super-resolution. In ECCV, 2020a.
Yifan Liu, Ke Chen, Chris Liu, Zengehang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In CVPR, 2019a.
Yifan Liu, Changyong Shun, Jingdong Wang, and Chunhua Shen. Structured knowledge distillation for dense prediction. In CVPR, 2019b.
Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. In CVPR, 2020b.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. In NIPS, 2017.
11

---

Published as a conference paper at ICLR 2023
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales, Ensemble distribution distillation. In ICLR, 2020.
Wonpyo Park, Yan Lu,, Minsu Cho, and Dongju Kim. Relational knowledge distillation. CVPR. 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabiliistc knowledge transfer. ECCV, 2018.
Adam Paszke, S. Gross, Francisco Massa, A. Lerer, J. Bradbury, G. Chanan, Killeen, Z. Lin, N. Gimelshein, L.. Antiga, Alban Desmaison, Andreas Köpf, E. Yang. Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, B. Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.
Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
Baoyun Peng. Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, bong-sheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. Meta pseudo labels. In CVPR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Chengchao Shen, Xinchao Wang, Youtan Yin, Jie Song, Sihui Luo, and Mingli Song. Progressive network grafting for few-shot knowledge distillation. In AAAI, 2021.
Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In ICCV, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Suraj Srinivas and Francois Fleuret. Knowledge transfer with jacobian matching. In ICML, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Shlens Jonathon, and Wojna Zbigniew. Rethink- ing the inception architecture for computer vision. In CVPR, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR. 2020.
Hugo Touvron, Matthieu Cord, Mathhijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ICML, 2021.
Frederick Tung and Greg Mori. Similarity-preseeviii knowledge distillation. In ICCV, 2019.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In CVPR, 2019.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. ECCV, 2020.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In AAAI, 2021.
12

---

Published as a conference paper at ICLR 2023
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In ECCV, 2020.
Chuanguang Yang. Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. In CVPR, 2022a.
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In ICLR, 2021.
Zhendong Yang. Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In CVPR, 2022b.
Zhendong Yang. Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In ECCV, 2022c.
Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209.02433, 2022d.
Anbang Yao and Dawei Sun. Knowledge transfer via dense cross-layer mutual-distillatonn. In ECCV, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
Sukmin Yin, Jonjing Park, Kimin Li, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In CVPR, 2020.
Kaiyu Yue, Jiangfan Deng, and Feng Zhou. Matching guided distillation. In ECCV, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
Hongyi Zhang, Cisse Moustapha, N Dauphin Yann, and Lopez-Paz David. mixup: Beyond empirical risk minimization. In ICLR, 2018a.
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In ICI.R, 2021.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018b.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Sun, Quoc Le, Qiang Liu, and Dale Schuurmans. Go wide, then narrow: Efficient training of deep thin networks. In ICML, 2020.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOFS OF THE ABSORBABLE PROPERTY OF OUR FEATURE TRANSFORM MODULES
Here provide the proofs show that our feature transform (FT) module without with linear residual connection (identity mapping) could be merged into the subsequent fully connect (FC) layer of the student network after training.
Given a student network S, let T.W W.,) sc) denote an FT module (without a linear residual connection) inserted after the last convolutional layer of S, let Wfe denote the weight matrix of the subsequent FC layer, let F E I #xxxx, denote the input feature maps of T., where H. W and C. are the channel height, width and number, respectively. For inference, the mathematical operations from the FT module the subsequent FC layer can be defined as:
F' = Weo * (Wae * F), O = W)o(GAP(('),
(6)
where * denotes the convolution operation, W.. RRIXIXC,XNC; W.c E RRIXIIGGXXC denote two point-wise convolutional kernels for channel expansion and contraction, respectively; Wfe RCx in the FC layer can be also seen point-wise convolutional kernel; GAP denotes global average pooling operation; F' BBxxxx XC, denotes the output feature maps of the FT module; O E RCr denotes the output vector of the FC layer.
We use Pi, RC, to denote the vector of channel pixels spatial location (i,j) of F. For the point-wise convolutional kernel W.e. we have:
(Wee se F) WoeFi.g
Therefore, the vector of channel pixels spatial location
(i, of F' can be calculated as:
Fl. = (Woc * (Wse * F)),s Wsc (W = (W W Pi.
(8)
then we can write:
O = Wj.GAP(F)
H
F
Wie fc
HW
H W
(W
= We
(9)
HW
H
W
F.j
= WWo Wse)
HW
1=1
GAP(F)
where w W fo Wse. Now, it is clear that can directly merge the FT module (without linear residual connection) into its subsequent FC layer of the student network at inference.
For the FT module with linear residual connection, the calculation of E' is defined as:
F = (Wse (Wsc F) + F)as
= (Wac(W (Wee F))is + Fij
(10)
(WooWW.)) P Fig
= WWacW + 1)Fis,
14

---

Published as a conference paper at ICLR 2023
where I RCXC is identity matrix. Then we have:
H
W
Fiy
0 = W/ fc
HW
H
W
(W sc ue + Fis
W
HW
(11)
j=l
Rij
Wcc(W.o se +
DE 1= HW
GAP(F)
where W/c Wee(W + I). Again, it is clear that we can directly merge the FT module (with a linear residual connection) into its subsequent FC layer of the student network at inference.
A.2 DATASETS AND IMPLEMENTATION DETAILS
Recall that our basic experiments are conducted on two image classification datasets: CIFAR- 100 (Krizhevsky & Hinton 2009) and ImageNet (Russakovsky al. 2015). This section provides the experimental details.
A.2.1 IMAGE CLASSIFCATION ON CIFAR-100
CIFAR-100, which consists of 50,000 training images and 10,000 test images with 100 classes, is popular classification dataset for KD research. Following the settings of CRD (Tian al. 2020). we use 13 teacher-student pairs having either the same type or different type network architectures (see Table 12 in the main paper) for experiments. Each experiment conducted for separate runs, and we report top-1 mean recognition rate on the test set. For fair comparisons, we use the same training settings to conduct the experiments with our method, following CRD. Specifically, for each reacher-student pair, the model is trained by the stochastic gradient descent (SGD) optimizer for 240 epochs, with batch size of 64, weight decay of 0,0005 and a momentum of 0.9. The initial learning rate is set to 0.1 and decreased by factor of 10 at epoch 150, 180 and 210.
All models are trained on an Intel Xeon Silver 4214R CPU server using one NVIDIA GeForce RTX 3090 GPU.
A.2.2 IMAGE CLASSIFICATION ON IMAGENET
ImageNet is much more challenging than 100, which contains over 1.2 million images for training and 50,000 images for validation, including 1,000 image classes. Following the settings of Tian et al. 2020); Yang et al. (202I): I: Chen et al. (022b). we use popular teacher-student pairs (see Table in the main paper), namely ResNet34 ResNet18 and
(Howard) et al. 2017), for experiments. For fair comparisons, we adopt the standard data augmentation to train and evaluate each network. For training, we first resize the input images to 256 x 256, then randomly sample 224 224 image crops their horizontal flips. We standardize the cropped images with mean and variance per channel. For evaluation, we use the center crops of the resized images, and report 1 recognition rate the ImageNet validation set.
Training setup for R The model is trained by SGD optimizer for 100 epochs, with batch size of 256. weight decay of 0.0001 and momentum of 0.9. The initial learning rate is set to 0.1 and decreased by a factor of 10 every 30 epochs.
Training setup for ResNet50 MobiileNe The model is trained by SGD optimizer for 100 epochs, with batch size of 256, a weight decay of 0.0001 and a momentum of 0.9. The initial learning rate is set to 0.1 and and scheduled to arrive at zero with cosine decaying strategy.
All models are trained on an Intel Xeon Silver 4214R CPU server with NVIDIA GeForce RTX 3090 GPUs.
15

---

Published as a conference paper at ICLR 2023
A.3 APPLYING NORM TO MORE TEACHER TEACHER-STUDENT LAYER PAIRS
Since our FT module is merely inserted after the last convolutional layer of a student network, NORM performs the proposed many-to-one representation matching between single teacher-student layer pair. A natural question is how about the performance when applying NORM to multiple teacher-student layer pairs. In the main paper, we provide set of ablative experiments explore this question on ImageNet with ResNet34 as teacher and ResNet18 as student (see Table 4 in the main paper). In the experiments, we add two linear student FT modules (N 8) after conv5 (our default design) and conv4_x of ResNet18 8, and consider the following two settings: (1) using the same staged teacher-student representations for many-to-one matching simultaneously; (2) forcing two different staged student representations to match the teacher representation from conv5_x _x.. For the macher-student layer pair of conv5_x wtw.S__x>toooxxx.. we apply 2 x average pooling with a stride of 2 to the expanded student representation in order match the spatial dimension of the teacher representation. Figure [3 shows an architectural overview these two NORM variants.
Fe
0001-1
caro55 X
Teachernetwors network trined
d
a(
W1
cocod
0unr5_
Studeentnewworkk
F
GAP
Teacher networkk (Pro tained and foxcl)
d(-)
d
W/c
com55
Studentnetwork
Figure 5: Architectural overview of applying NORM to two teacher-student layer pairs. For the ablative experiments on ImageNet with ResNet34 as teacher and ResNet18 as student (Table in the main paper), we add two linear FT modules (N 8) after conv5_x xxx (our default design) and conv4_x of ResNet18, and consider the following two settings: (1) applying NORM to two same staged teacher-student layer pairs (after conv4 x and conv5 x) simultancously (the figure in the first row); (2) forcing the student representations expanded from conv4 x and conv5 x to match the teacher representation from conv5_x x (the figure in the second row). For the teacher-student layer pair of conv5_x 11155_xxxooo44. x, we apply a 2 x 2 average pooling with stride of 2 to the expanded student representation in order match the spatial dimension of the teacher representation.
A.4 CONTRACTING TEACHER REPRESENTATION
In our formulation, NORM enables the many-to-one representation matching via expanding the student representation have N times feature channels than the teacher's. Naturally, reversed way for enabling the many-to-one representation matching is to contract the teacher representation to have 1 1/N times feature channels than the student's. In principle, such reversed NORM variant is
16

---

Published as a conference paper at ICLR 2023
Table 7: Comparison of expanding student representation vs. contracting teacher representation. On ImageNet with teacher-student pair ResNet34-ResNetle we compare NORM (N = 8 and 8) with reversed NORM variant which enables the many-to-one representation matching via contracting the teacher representation to have 1/N feature channels than the student's.
Student
Baseline
NORM (dotault)
Reversed NORM
ResNet18 (w/ plain FT)
70.13
71.42
70.82
ResNer18 (w/ default FT)
70.13
72.14
71.20
Table 8: Effect of different distance metrics. On CIFAR-100, compare the performance of using 3 different distance metrics to compute the many-to-one representation matching loss of NORM (N and 10).
Distance metric
4-normm ly-norm (default)
MMD
DesNett2x4-ReeNeetk.. 75.10
76.49
76.62
more efficient than NORM. However its performance is limited by the information missing caused by dimension reduction to the teacher representation. To validate this problem, we perform a set of ablative experiments on ImageNet with ResNet34 as teacher and ResNet18 as student. We compare NORM (using linear FT module either with or without linear residual connection) and the reversed NORM (using a popular FT consisting one 1 X convolutional layer). Table 7 shows detailed results, from which we find the reversed NORM can also improve the baseline student but performs obviously worse than NORM. Specifically, NORM outperforms the reversed NORM by a top-1 margin of 0.60% and 0.94% for the FT module without and with a linear residual connection, respectively.
A.5 PERFORMANCE COMPARISON WITH NETWORK RE-PAR PARAMETERIZATION METHODS
In Table 6 of the main paper, we compare the performance of NORM with two network re- parameterization methods on the ImageNet dataset, which also apply the vanilla logits based KD during model training. For the student MobileNet, a pre-trained ResNet50 is used as the teacher in NORM, while more powerful ResNet1 is used the teacher in the paper of ExpandNets (Guoo et al. 2020b) when performing the logits based KD. Even with a less powerful teacher model, the MobileNet model trained by our NORM without the logits based KD is obviously more accurate than that trained by ExpandNets+KKD showing 3. 797 top-1 gain. For the student ResNet50-1144 1/4, we follow basic training settings of WIN Zhou et al. 2020). Specifically, pre-trained ResNet50 is used as our teacher. The student model is trained by SGD optimizer for 100 epochs, with batch size of 256, a weight decay of 0.0001 and a momentum of 0.9. The initial learning rate is set to 0.1 and scheduled to arrive at zero with a cosine decaying strategy. Strong augmentations like label smoothing (Szegedy et al. 2016) and mixup (Zhang et al., 2018a) are not used in our method. After training. for the ResNet50- 1/4 (student), NORM without KD reaches 68.03% top-1 accuracy with pre-trat pre-trained ResNet50 (teacher), while WIN (Zhou et al. 2020) with KD gets 67.50% top-1 accuracy using a pre-trained wide teacher which is 4 times larger than the student.
A.6 EFFECT OF DIFFERENT DISTANCE METRICS
In NORM, we use the 4-- n-norm distance metric to compute the many-to-one representation matching loss Lnorm defined in Eq. of the main paper. In Table g we compare the performance of our method with three different distance metrics including 4 (y-norm (our choice), ( norm and mean maximum discrepancy (MMD). In the experiments, we use the teacher-student pair of ResNet32x4 Ness84 with N = 8 and 10. Comparatively, lo-morm is superior to l -orrm. With MMD, NORM reaches 76.62% accuracy for the student model, showing 031% improvement to normm. This indicates that, with better distance metric, our method might yield a student model with higher accuracy. We currently choose the Ig-norm distance metric owing to its simplicity and effectiveness.
A.7 TRAINING COST COMPARISON
Table shohwws a comparison of the total training cost of NORM and the individual training. We can that the total training cost of NORM for the student ResNet18, 18, MobileNet and ResNet50-1/4 1/4 is
17

---

Published as a conference paper at ICLR 2023
Table 9: Training cost comparison. All models are trained ImageNet with the same settings to those for Table and Table in the main paper, using an Intel Xeon Silver 4214R CPU server with two NVIDIA GeForce RTX 3090 GPUs.
student pair
Total training cost individual training (hour) truining cost (bour)
ReiNet34RResNnn
24. 14
67.11
ResNer50 DeNNerS0-ReeNat50-11 ResNet50-Mobille
39.92
97.28
30.33
72.52
2.78x, 2.48x, 2.39x than the individual training, with the teacher network ResNet34, ResNet50 and ResNet50- 1/2, respectively. These results indicate that NORM is efficient since it uses the standard teacher-student training framework in which the pre-trained teacher model runs in the forward phase of the whole training procedure. The forward time of the teacher can be saved by offline representation pre-computing. For easy implementation, we do not use it.
A.8 VISUALIZATION RESULTS
NORM achieves the feature distillation goal with the proposed many-to-one representation matching mechanism at a single teacher-student layer pair. This means that the feature distribution of the student network would be more similar that of the teacher network after NORM training, compared to the individual training. Under this context, it is necessary to study the learnt feature distributions with and without NORM. To this end, we use a teacher-student model pair (ResNetl 10 (0eRe)))))3))) well trained by NORM (N = 8 and = 10) on the CIFAR-10 10 dataset (which is popularly used to analyze the learnt feature distributions in KD research) and all images the validation set, and conduct a set of experiments to analyze the learnt last-layer feature distributions using t-SNE (Maaten Hinton (008). Comparative visualization results are shown in Fig. 66] from which we can observe that NORM shows relatively strong feature mimicking capability. Each color denotes one image category the learnt feature distribution.
43
Figure 6: Comparison of learnt feature distributions. We well-trained ResNetl Resseee2222 model pair (N and a = 10) on the CIFAR-10 dataset and all images in the validation set, and conduct set of experiments & Hinton| 2008). The left figure shows the fcature distribution from an individually trained student model; the middle figure shows the feature distribution from the student model trained by NORM; and the right figure shows the feature distribution from the pre-trained teacher model. We can observe that NORM shows relatively strong feature mimicking capability. Each color denotes one image category in the learnt feature distribution.
analyze the learnt last-layer feature distributions using t-SNE (Maaten
A.9 MORE EXPERIMENTS AND DISCUSSIONS FOR THE REBUTTAL
In this section, we provide a lot of extra experiments and discussions provided for the rebutta fT
A.9.1 A SYSTEMATIC STUDY OF NORM
So far, we have provided several sets of ablative experiments to show: (1) a large N value is much better than N 1, see Figure 2] (2) given the expanded student representation consisting of N feature segments, the more the feature segments used to match the teacher representation, the larger the accuracy gain, Figure (3) inserting linear FT module into different student networks usually does not bring accuracy improvement under the individual training, see Table [ Table 2 and Table 5
We would like to acknowledge the contributions by intern Jiawei Fan supervised by Anbang Yao. He conducted the experiments to test the effectiveness of our method on image segmentation task, object detection task, and image classification with the masked generative learning.
18

---

Published as a conference paper at ICLR 2023
Table 10: The role of the learnable ensemble layer (i.e., the second linear layer of student FT module). On ImageNet with a tcacher-student pair ResNet34-ResNeee we compare NORM (N and 8) with different student FT designs. Best result is bolded.
Methods
Top-1 accuracy (%)
Baseline
70.13
NORM + a default 2-layer FT module
72.14
NORM + 2-layer FT module w/ the fixed ensemble layer
71.91
NORM 3-layer FT module
72.21
Table 11: The role of the student FT module initialization. On ImageNet with teacher-student pair
we compare NORM (N and a = 8) with 3 different weights initialization
strategies. Best result is bolded.
Methods
Top-1 accuracy (5)
Baseline
70.13
NORM (N=1)
71.23
NORM (N=8) default weights initialization
72.14
NORM (N=8) initialization with higher weights variance
72.22
NORM (N=8) initialization with the same weights to N student feature segments
72.08
In order to better understand what enables the distillation effectiveness, here we systematically study the major components of NORM from the following four more aspects.
All following ablative experiments are performed ImageNet dataset with ResNet34 as teacher and ResNet18 as student, N and a = 8.
The role of the learnable ensemble layer. In light of the interpretation of NORM given in the Method section, the second linear layer Wsc of our student FT module performs a learnable ensemble of N ietiilaationaugmmente Accordingly, we conduct a set of ablative experiments to explore the role of this component. In the experiments, compare NORM with 3 different student FT designs: (a) default two-layer student FT module, (b) two-layer student FT module with the fixed ensemble layer (i.e., feature channels from different segments are sequentially averaged), and (c) three-layer student FT module (i.e., having two same-size linear layers as a learnable ensemble). Detailed results are summarized in Table TŌ, can be seen that all three student FT modules bring clear accuracy improvements. validating the importance of the second linear layer for ensembling N augmented student feature views. Comparatively, NORM with the learnable ensemble layer better than NORM with the fixed ensemble layer, and further improved result is attained when using more complex learnable ensemble (two-layer).
student feature views via fully connected channel mixing operations.
The role of the student FT module initialization. Note that N expanded student feature segments generated by the first linear layer of our student FT module enable the many-to-one representation matching. Next, perform set of ablative experiments to study the effect of different weights initialization strategies for the first linear layer. In the experiments, we compare NORM with different weights initialization strategies: (a) the default weights initialization we used, (b) initializa- tion with higher weights variance (8x of the default), and (c) initialization with the same weights to N segments. Detailed results are summarized in Table [T] It can be seen that all three weights initialization strategies get promising performance, showing the robustness of our method to different initialization strategies. Comparatively, initialization with higher weights variance tends bring bit more gain. What's 's more, initialization with the same weights to N = 8 segments is much better than This benefits from the second linear layer which uses fully connected operations for dynamically mixed feature ensembling. As result, the gradients are different to the weight groups initialized with the same values, producing different student feature segments (we also confirm this by recording and comparing them the fly) to match the teacher representation.
The role of the designs to construct the many-to-one representation matching. NORM enables the many-to-om representation matching via expanding the student representation to have N times feature channels than the teachers. We can also construct multiple student-to coudent-to-teachee to-teacher matching perform another set of ablative experiments to study the
routes in other ways. Accordingly, effect of different designs to construct the many-to-one representation matching. In the experiments, compare 5 different designs: (a) our default design, (b) reversed design (contracting the
19

---

Published as a conference paper at ICLR 2023
Table 12: The role of the designs construct the many-to-one representation matching. On ImageNet with a teacher-student pair ResNet344RReseee we compare NORM (N = 8 and a 8) with 5 different designs to construct the many-to-one representation matching. Best result is bolded.
Methods
Top-1 accuracy(5))
Baseline
70.13
NORM (default, student-to-teacher matching routes
72.14
Reversed NORM (1 FT contract teacher features, woo our linear student FT module, N:
71 11.20
Reversed NORM (N teacher FTs, w/o linear student FT module, 1:
71.18
Paired NORM (.V weaheer FTs, N student FTs, :N)
70.62
Paired NORM (N teacher FTs, our linear student FT module, N: :
71.78
Table 13: The role of the ways to split the expanded student representation. On ImageNet with a teacher-student pair -essNNe we compare NORM (N = 8 and a = 8) with 3 different feature splitting strategies. Best result is bolded.
Methods
Top-1 accuracy (%)
Baseline
70.13
NORM sequential feature splitting (defiult)
72.14
NORM- random feature splitting
72.13
NORM importanco-based feature splitting
72.15
teacher representation by single FT to have 1/N times feature channels than the students, without using our linear student FT module), (C) a reversed design (using N independent FT's to the teacher representation, without using our linear student FT module), (d) paired design (using N independent FTs to the teacher representation and another N independent FTs to the student representation), and (e) paired design (using N independent FTs to the teacher representation, and using our linear student FT module). Detailed results are summarized in Table 12, can be seen that all five designs improve the performance of the student network, validating the importance of the many-to- representation matching concept. Comparatively, our default design achieves the best performance, and the designs that apply single FT multiple FTs to the teacher representation show obviously worse performance due to the information loss (that is, it is important to preserve the intact pre-trained teacher representation).
The role of the ways to split the expanded student representation. In NORM, for simplicity, we sequentially split the expanded student representation into N feature segments. Our last set of ablative experiments is to study the effect of different feature splitting strategies. In the experiments, we test NORM with 3 different feature splitting strategies: (a) sequential feature splitting, (b) random feature splitting, and (c) imoortance-based feature splitting in which we first sort feature channels in descending order based on the learnt mean values of channel-wise batch normalization parameters at the fifth epoch, and then use sequential feature splitting to enable our many-to-one representation matching. Detailed results are summarized in Table 13 It can be seen that all three feature splitting strategies show almost the same performance. This is because that there is no semantic channel-wist aligmment between the teacher and the expanded student representations before the many-to-one representation matching.
Based on all ablations described above, we validate the roles of the major components of our method, and provide deep understanding of what enables the distillation effectiveness of NORM.
A.9.2 SEMANTIC SEGMENTATION ON CITYSCAPES DATASET
In order to test the effectiveness of our method on image segmentation task, we perform two sets of experiments with Cityscapes dataset (Cordts et al. 2016). Cityscapes dataset contains 5000 images with a split of 2975, 500 and 1525 images are for training validation and test, respectively. In the experiments, we test our method with both same type and different type teacher-student network pairs for semantic segmentation, and compare it with SKD (Liu et al. 2019a). IFVD (Wang et al. 2020), CWD (Shu et al.) 2021), CIRKD (Yang et al. 2022a) and MGD Yang et al. 2022c). Specifically, we test our method using ResNet18 as student network first, and then using Nest (Zhao et al., 2017) as teacher network and DeepLabV3. 3- used in very recent works of CIRKD and MGD. We add NORM (N = 2 for training efficiency) after the last feature layer of V All models are trained with 8 NVIDIA Tesla
(Chen et al., 2017) teacher network and DeepLabV3-
student network, following the training and test settings
20

---

Published as a conference paper at ICLR 2023
Table 14: Results comparison Cityscapes dataset with the same type teacher-student network pair. Best result is bolded.
Teacher
Student
Methods
Performance (mIOU, %.
DeeplaabV3 DeeplabbV3RReNeet0011 Re (78.07)
DeepLabb33 18 (74,21)
Despl.abV3.BB. R (78.07)
Deepl aee3.aahhRRseee (74.21)
IFVD
75.59
DeeplaabV3 RelNet1l1 (78.07)
DeegLabVV3RReNeet (74,21)
(eLLabbV3-ResNelDD (78.07)
DespLabV3-ReNee14 (74,21)
CIRKD
76.38
Deppabb33RRsNeet01 (78.07)
DeepLabV3-ReaNett8 (74.21)
MGD
DeeplabV3-ReeNert0n (78.07)
DeepLahhV-ResNettt (74.21)
NORM (ours)
77.03
Table 15: Results comparison on Cityscapes dataset with the different type teacher-student network pair. Best result is bolded.
Teacher
Student
Methods
Performance (mIOU, %%)
ReeNeel1 (78.34)
DeepLab 3 ResNeteN (73.20)
73.87
PSPNet ResNer101 (78.34)
DeeplaahV3 RecNet18 (73.20)
IFVD
n/a
PSPNet-ResNetl (78.34)
DeepLabVV-ResNert (73.20)
75.95
PSPNat-ReNNet101 (78.34)
DeepLaVV3RReNatt) (73.20)
CIRKD
n/a
USPNa-RReNet1011 (78.34)
DepplabV3.ReNettt (73.20)
MGD
76.02
PSPNet ResNet101 (78.34)
DeeplahV3 ResNet18 (73.20)
NORM (ours)
76.51
V100-SXM3 GPUs. Detailed results summarized in Tande Table 15
It can be seen that
method achieves new state-of-the-art results on these two teacher-student network pairs, validating its effectiveness in handling semantic segmentation task.
segmenta
A.9.3 OBJECT DETECTION ON MS COCO DATASET
In order to test the effectiveness of method object detection task, also perform two sets of experiments with MS COCO dataset (Lin et al. 2014). MS COCO dataset (2017 version) contains 1 118,000 training images and 5,000 validation images with 80 object classes. In the experiments, test our method with both same type and different type teacher-student network pairs, and compare with FKD (Zhang & Ma 2021), ) CWD (Shu al. 2021), FGD (Yang al. (202bb and MGD (Yagg (Lin et al. 2017b) as ResNet50 as student detector first, and then using Cascade Mask
et al. 2022c). Specifically, y, we test our method using RetinaNel-Res.cXIIU
teacher detector and RetinaNee-ResNet550 RCNN-ResNeettlO (He et al.) (017) as teacher detector and Faster
(Ren et al.)
2015) as student detector, following the training and test settings used in very recent papers of FGD and MGD. We add NORM (N = 4 for training efficiency) after each output of the FPN neck (Lin ct al. 2017a) of ResaNeer-RsNettOFFastttt
MeNN-RRssNe50. All models are trained with 8 NVIDIA Tesla VI00-SXM3 GPUs. Detailed results summarized in Table [6] and Table 17 It can be that our method achieves new state-of-the-a art mAP results on these two teacher-student detector pairs, validating its effectiveness in handling object detection task.
A.9.4 MORE DISCUSSIONS AND EXPERIMENTS
Here, we additionally discuss some potential extensions of NORM.
NORM vs. self-supervised learning methods. Recently, self-supervised learning research with self-distillation settings (a particular type of unsupervised learning) has attracted increasing attention. In this line of research (c.g., SimCL (Chen et al. 2020b), MoCo (He et al. (020), SWAV (Caron ct al. 2020), BYOL Grill et al., 2020), DenseCL (Wang et al.) 2021 DINO (Caron al. 2021 IGPT CChn et al. 3 iBOT (Zhou et al. 2022), BeiT (Bao et al., 2022) and MAE (He et al. 2022)) matching N features of the student to N features of the teacher by referring dense features
of different image patches has been explored. Indeed, they also have the concept of many parallel
Table 16: Results comparison MS COCO dataset with the same type teacher-student network pair. Best result is bolded.
Teacher
Student
Methods
mAP (%)
APs (%)
AP (%)
AP, (%)
RetinaNNe-ResNeXitue ResNeXt101 (41.0)
RetinaNet-Resta Re5tee (37.4)
FKD
39.6
22.7
43.3
$2.5
RetinaNet. ResNeXt101 (41.0)
RetinaNet ResNet50 (37.4)
CWD
40.8
22.7
44.5
55.3
SeinaNer-ResNeXt1011 (41.0)
BetinaNet-ResNee530 (37.4)
FGD
40.7
22.9
45.0
54.7
RctinaNec-RcaNeet1001
RetinaNet-ResNct550 (37.4)
MGD
41.0
23.4
45.3
55.7
RetinaNet- ResNe11110 (41.0)
RedinaNee-ResNee500 (37.4)
NORM (ours)
41.1
23.3
45.3
55.7
21

---

Published as a conference paper at ICLR 2023
Table 17: Results comparison on MS COCO dataset with the different type teacher-student network pair. Best result is bolded.
Teacher
Student
Methods
mAP (S6)
APs (%6
APN (%)
APr, (%)
Cascade Mask RCNN- ResNeX:101 (47.3)
Faster RCNN- ResNer50 (38.4)
FKD
41.5
23.5
45.0
55.3
Cascade Mask RCNN- ResNeX1101 (47.5
Faster RCN BCNN-RRO ResNNer0 (38.4
CWD
41.7
23.3
45.5
55.5
Cascade Mask RCNN- ResNeX:101
(47.3)
RCNN- ResNe (38.4)
FGD
42.0
23.8
46.4
55.5
Cascade Mask ICNN--R=NNXX101 (47.3)
Fister RCNN-RosNat00 (38.4)
MGD
42.1
23.7
46.4
56.1
Cascade Mask RCNN RONNN-ReNNNXIOL (17.3)
Faster RCNN-RosNa550 (38.4)
NORM (ours)
42.4
24.0
46.6
56.3
Table 18: Results comparison on ImageNet dataset with a teacher-student pair of vision transformer architectures. We combine our NORM with the masked generative learning proposed in VITKD. Best result is bolded.
Teacher
Student
Methods NKD
Top I accuracy (16
DedT/Taay (74.42)
DeiT-Smull
Daii-Tay (74.42)
75.48
DeiT Small:0 69)
Tiny (74,4)
VITKD
75,40
Sall-Smallsso000 (0)
DeiFTay (74,42)
VITKDINKI
76.18
DeTT-Suull(00699
DisT:Tiny (74.42)
VITKDANORM (Ours)
76.55
matching routes with their corresponding components, but our method differs with them in focus, formulation and application. Firstly, existing methods of self-supervised learning with self-distillation settings aim to learn proper visual representation of a single backbone from a large set of unlabeled images, while our work aims improve the visual representation of small student network by a pre-trained larger teacher network under condition that both of them are trained on a set of labelled images. Secondly, in formulation, many existing self self-supe learning methods (e.g., SimCL, MoCo. SWAV, B YOL, DenseCL and DINO) define a contrastive predication task that encourages the encoder (backbone) to attract similar (positive) sample views and dispel different (negative) sample views with their corresponding contrastive losses leveraging data augmentation, and some others (c.g.. iGPT, BeiT and MAE) define a masked reconstruction task that uses a decoder to predict the original image spixel-wise/patee wise) given the representation learnt by the encoder (backbone) with their corresponding reconstruction losses leveraging masked image modeling. iBOT further combines masked image modeling into seff-supervised contrastive learning for improved performance. To these methods such as DenseCL and iBOT, self-distillation is performed between the target model (student) and the online model (teacher, an exponential moving average of student parameters) via contrasting dense sample view pairs of the whole image or the patch (some patches may be masked), and both models share the same encoder (backbone). In contrast, our method leverages a linear FT module added after the last convolutional layer of small student network to formulate many-to-one representation matching scheme between a single teacher-student layer via student feature expansion and splitting, and dense mimicking. All segments of our expanded student features are simultaneously forced to be similar to the teacher features, and there is no contrastive matching and no paired data augmentation to the input image. Thirdly, in application, the trained encoder of self-supervised methods is typically used to downstream tasks, while the efficient student network trained by our method is for direct deployment. Besides, iBOT, iGPT, BeiT and MAE with masked image modeling are particularly used to train vision transformers in unsupervised learning regime, while our method is mainly for training efficient convolutional neural networks in supervised learning regime.
Note that recent work CRD extends contrastive learning into KD research by presenting a novel contrastive distillation loss, and in the main paper we already validated the effectiveness of combining our method with CRD (see Table and Table 2).
Combining NORM with masked generative learning. In the main paper, we evaluate our method with 16 teacher-student network pairs (13/3 pairs on CIFARR100//mageh 100//mageNet dataset). The main reason for this is that in the community, mainstream KD methods for image classification, e.g.. 19 recent KD methods compared in our work, typically use convolutional architectures for performance evaluation. We follow them for fair and easy comparisons. Benefited from the simplicity of our method, we can easily use it to other network architectures. We notice that a very recent work VITKD (Yang et al. 2022d) explores the use of the aforementioned masked generative learning for vision transformer based KD research. Benefited from the simplicity of our method, we can easily combine NORM with VITKD to test our methods effectiveness in training a vision transformer under masked generative learning. Accordingly, we perform set of new experiments on ImageNet dataset, using DeiT-Small (Touvron et al. [202I) as teacher network and DeiT-Tiny as student network, following the settings of VITKD. For the student network, we add NORM (N for training
22

---

Published as a conference paper at ICLR 2023
efficiency) to the same layers VITKD. Detailed results summarized in Table 18 It can be that our method also works well on transformer architectures, achieving promising results. Here, should be noted that the standard training of a popular transformer model is very time consuming due to significantly larger model size and longer training schedule, compared to that of mainstream convolutional networks (e.g., 300 epochs for DeiT's vs. 100 epochs for ResNets). One run of training the above teacher-student transformer pair by NORM needs about 2.5 days with 8 NVIDIA Tesla V100-SXM3 GPUs. We will continue explore the potential of our method more teacher-student transformer pairs in the future.
A.10 LIMITATIONS OF NORM
Despite of its simple formulation and effectiveness, NORM has two limitations. The major limitation is applying the many-to-one representation matching to multiple teacher-student layer pairs cannot achieve further improved distillation performance, unlike many existing KD methods. This is due to the linear property of our student FT design, Table and Table{5 in the main paper). Besides, applying the many-to-one representation matching multiple teacher-student layer pairs will lead to increased training cost because the many-to-one representation matching also needs to be computed multiple times.
explored in the ablative experiments (see
23


Reviews:
[{"review_id": "VUjTAvH69T", "reviewer": "Reviewer_Xp75", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for feature distillation between a teacher model and a student model. It aims to improve the student model's accuracy by utilizing knowledge from the more capable teacher model. \n\nThe method adds a linear residual module after the last convolution layer of the student model. The module first transforms the student features to N times the number of channels for the teacher model. Then the expanded student features are grouped into N equal-width groups. Each group is compared with the teacher's features for computing the distillation loss. The distillation loss is optimized in conjunction with the task(classification) loss to learn the student model parameters. \n\nThe method is examined on multiple classification benchmarks with different teacher/student pairs. It achieves competitive accuracy with other baseline methods. The authors provide an analysis of the effectiveness of the proposed module through a comparative study. ", "summary_of_the_review": "Overall I feel the submission is of good quality and has satisfying results. My initial recommendation is to accept this paper. A few points as listed in the weakness section describe potential improvements that could be made before publication. ", "strength_and_weaknesses": "Strength:\n\n+ The paper is well written with a clear description of the method and experiments. I also like that the method is simple and seems straightforward to implement based on the provided information. \n\n+ The experiment results seem strong compared with other baseline methods. These methods are up to date. \n\n+ The idea of having N representation matching objectives is interesting. The fact that different transform groups differ only in initialization but together improve accuracy is very intriguing. \n\nWeakness/suggestions:\n\n+ Though effective, there seems to be no obvious motivation to construct multiple transforms at first.  I would like to see some discussion or theoretical analysis on where the effectiveness comes from or what it suggests for the community of model distillation. Because a transform in each group can be considered as an independently initialized tiny model in an ensemble,  could it be possible that it has any connection with ensembling or mixture-of-expert methods? This is more of a question not explored than a weakness, but having this discussion may greatly amplify the impact of this work. \n\n+ Most experiments are conducted on classification tasks. It may be beneficial to show the results in other tasks, such as segmentation and object detection, where distillation is also widely used. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9uWsr9_HYwF", "reviewer": "Reviewer_d7Sn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work present NORM, a  two-stage feature distillation method. It relies on a linear feature transform module that is inserted after the last convolutional layer of the student network to enable a many-to-one representation matching mechanism conditioned on a single teacher-student layer pair via feature expansion, splitting and group-wise mimicking. Experiments are peformed on CIFAR-100 and ImageNet  to show the effectiveness of the proposed method.", "summary_of_the_review": "The paper is in general well written. The authors put forward a plug-and-use many-to-one presentation learning module to improve the single teacher-student distillation performance. Experiments under different variants on CIFAR100 and ImageNet are implemented to evaluate the effectiveness of the proposed method. However, some details are missing for reproduction. The method seems also limited to convolutional networks and Image classification tasks. ", "strength_and_weaknesses": "Pros:\n\n+ This work put forward a plug-and-use module that is easy to deploy.\n\n+ The method is experimentally compared with many State-of-the-Art methods on ImageNet and CIFAR-100.\n\n+ The paper is in general well-written.\n\n\nCons:\n\n- The method seems to be limited to the convolutional architecture.\n\n- The idea of creating multiple heads with the exact same shape and distill the teacher into each of them simultaneously seems intuitive and I wonder why it is benefitial for improving the distillation performance.\n\n- Experiments are only performed on image classification datasets, will it still work on other tasks such as image segmentation?\n\n- In Equation 3, $d(,)$ is undefined. I guess it is a distance function yet the authors still need to make it clarified.\n\n- It is not clear to me how to find the desired channel dimension $NC_t$ for different teacher-student pairs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7GX4lALU26q", "reviewer": "Reviewer_xShr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an architectural trick for two-stage knowledge-distillation from a large to a small network. It expands the student feature tensor of the last layer from C to NC channels, applies distillation from the teacher feature tensor to each of the N sub-tensors, then contracts back to the original feature dimensions, C. Both mappings are linear and at inference they are merged into a single CxC layer.", "summary_of_the_review": "This is an interesting idea. The distillation loss is unclear but I am sure this can be corrected. However, the experimental setting is too narrow and the method does not appear to be very effective.\n\n**Update**: The rebuttal addresses my concerns to an extent and although further improvement is possible, I raise my recommendation from 5 to 6.", "strength_and_weaknesses": "Strengths:\n- Well written\n- Mostly clear\n- Somewhat novel\n\nWeaknesses:\n- Too narrow experimental setting\n- Not particularly effective\n\nThe paper is well written and the method is mostly clear except for the distillation loss.\n\nIn particular, the default is Euclidean distance between features (3), referred to as NORM. In the experiments, the \"vanilla logits based distillation\" is additionally used, referred to as NORM+KD, citing the original distillation paper (Hinton et al. 2015).\n\nBut the original distillation loss in (Hinton et al. 2015) is cross entropy with soft targets given by the teacher, whereas the Euclidean distance between logits is only discussed as an approximation when the temperature is high. So the \"vanilla logits based distillation\" is not clear. It should be given formally as a second option below (3). This is very important given that NORM+KD is the best performing option. It cannot remain undefined.\n\nThe names NORM and NORM+KD are also not appropriate, because it is implied that NORM does not include distillation; which is not true as (3) is a form of distillation. It is more appropriate to use NORM+A and NORM+B for some A and B.\n\nThe novelty appears to be ok as I am not aware of such trick for knowledge distillation. However, a similar trick exists of introducing a number of classifiers by Littwin and Wolf 2015, \"The Multiverse Loss for Robust Transfer Learning.\" The differences are that we have N classifiers rather than N subtensors, C classes rather than C channels, cross-entropy rather than distillation loss and an additional constraint that the classifiers are orthogonal. But structurally, the ideas are very similar. Some discussion is in order.\n\nAnother very related idea is to match N features of the student to N features of the teacher by referring to dense features of different image patches. This has been explored in self-distillation settings for self-supervision, e.g.\n(Dense CL) Wang et al. 2021. Dense Contrastive Learning for Self-Supervised Visual Pre-Training.\n(iBOT) Zhou et al. 2022. iBOT: Image BERT Pre-Training with Online Tokenizer.\nSuch dense matching is an alternative way \"to introduce many parallel knowledge transfer routes.\" A discussion is in order. A comparison would be a great addition.\n\nThe paper includes extensive experiments on several teacher-student model pairs and a lot of ablation studies. However:\n- In Table 1, NORM+KD beats the competition only in two columns where many or all of the competitors are missing. In Table 2, it is best in more columns but again many of the competitors are missing. I do not find these results conclusive.\n- More importantly, two-stage knowledge-distillation from a larger to a smaller network for supervised classification is too narrow. For this architectural modification to be properly validated, I would suggest extending to online self-distillation, self-supervised and semi-supervised settings, continual learning, vision transformers and other tasks like object detection. It would be particularly interesting to combine with methods like BYOL, DINO, iBOT, Dense CL etc., which use self-distillation.\n\nAn ablation of a \"reverse\" idea in addition to \"Contracting teacher representation\" would be to match 1 representation of the student to N representations of the teacher. N-to-N could also be explored.\n\nAppendix A.1 appears to be an overkill. The composition of linear layers is obviously linear.\n\nMy final rating will depend on how many additional experiments can be done and how effective NORM will be shown to be.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6XJnrBxNzaf", "reviewer": "Reviewer_iHQs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new knowledge distillation method called NORM, based on the idea of many-to-one representation matching. The method follows a two-stage distillation process using a linear Feature Transform (FT) module. The FT module has two layers: the first expands the number of channels to N times those from the teacher, and the second shrinks the number of channels back to the original student feature space. The expanded representation is forced to approximate the teacher representation N times. Experiments are conducted on the CIFAR100 and ImageNet datasets with various teacher-student pairs.", "summary_of_the_review": "The paper tackles an important problem, proposing a modification to the standard two-stage distillation paradigm. Experiments are comprehensive and seem to show improved performance in common benchmarks, but the method lacks justification.", "strength_and_weaknesses": "Strengths:\n\nS1) Knowledge distillation is an important and active topic in the community, which is addressed in this work.\n\nS2) The paper is generally well written and easy to follow.\n\nS3) The method is very simple and easy to understand.\n\nS4) The experimental evaluation is significant, considering many different network architectures and several baselines. Ablations are performed to understand which parts of the system matter more.\n\nWeaknesses:\n\nW1) The main concern I have with this paper is that we don’t really understand why the proposed method works. Simply create multiple heads with the exact same shape and distill the teacher into each of them simultaneously. Why is this supposed to be better than simply having one head here? I do recognize that the authors provide many experiments indicating that this replication of heads apparently really helps, which to me feels surprising.\n\nIn general, I think there should be more discussion on this. For example, what if all N heads are initialized with the exact same random weights? If this is the case, I believe that would degenerate into the single-head (N=1) case? What types of initialization are needed here? Maybe we need higher initialization variance in order to capture different optimization routes?\n\nW2) A question: can you please clarify if the networks are using global average pooling? Or if the fully connected layer from the classifier is applied on top of the final feature map directly? Can you please clarify additionally if the baseline methods you are comparing to also use global average pooling, or if in their cases the FC layer is applied on top of the final feature map directly? I wanted to make sure everything is comparable in this respect, and the gains we see are not coming from the removal of the global pooling, for example.\n\nMinor details:\n\nM1) Section 3.1 does not concern new contributions from this paper. I suggest the authors revise its title to “Background: One-to-one representation matching”, to emphasize the fact that background is presented.\n\nM2) In several parts of the manuscript the word “typed” is used incorrectly, eg “same typed”, “different typed”. In most cases it should be “type”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster