ProxylessKD: Direct Knowledge Distillation with Inherited Classifier for Face Recognition
Authors: Weidong Shi
Guanghui Ren
Yunpeng Chen
Shuicheng YAN
Venue:  (2021)
Keywords: inherited classifier
embedding space alignment
face recognition
knowledge distillation

Abstract: Knowledge Distillation (KD) refers to transferring knowledge from a large model to a smaller one, which is widely used to enhance model performance in machine learning. It tries to align embedding spaces generated from the teacher and the student model (i.e. to make images corresponding to the same semantics share the same embedding across different models). In this work, we focus on its application in face recognition. We observe that existing knowledge distillation models optimize the proxy tasks that force the student to mimic the teacher’s behavior, instead of directly optimizing the face recognition accuracy. Consequently, the obtained student models are not guaranteed to be optimal on the target task or able to benefit from advanced constraints, such as the large margin constraint (e.g. margin-based softmax).  We then propose a novel method named ProxylessKD that directly optimizes face recognition accuracy by inheriting the teacher's classifier as the student's classifier to guide the student to learn discriminative embeddings in the teacher's embedding space. The proposed ProxylessKD is very easy to implement and sufficiently generic to be extended to other tasks beyond face recognition.  We conduct extensive experiments on standard face recognition benchmarks,  
and the results demonstrate that ProxylessKD achieves superior performance over existing knowledge distillation methods.

Under review as a conference paper at ICLR 2021
PROXYLESSKD: DIRECT KNOWLEDGE DISTILLATION WITH INHERITED CLASSIFIER FOR FACE RECOGNI- TION
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge Distillation (KD) refers to transferring knowledge from a large model smaller one, which is widely used enhance model performance in machine learning. It tries to align embedding spaces generated from the teacher and the student model (i.e. to make images corresponding to the same semantics share the same embedding across different models). In this work, we focus on its applica- tion in face recognition. We observe that existing knowledge distillation models optimize the proxy tasks that force the student to mimic the teacher's 's behavior, instead of directly optimizing the face recognition accuracy. Consequently, the obtained student models are not guaranteed to be optimal able to benefit from advanced constraints, such as large margin constraint (e.g. margin-based softmax). We then propose novel method named ProXylessKD that directly optimizes face recognition accuracy by inheriting the teacher's is clas- sifier as the student's classifier to guide the student learn discriminative em- beddings in the teacher's s embedding space. The proposed ProxylessKD is very easy to implement and sufficiently generic to be extended to other tasks beyond face recognition. We conduct extensive experiments on standard face recogni- tion benchmarks, and the results demonstrate that ProxylessKD achieves superior performance over existing knowledge distillation methods.
the target task
INTRODUCTION
Knowledge Distillation (KD) is process of transferring knowledge from a large model smaller one. This technique is widely used to enhance model performance in many machine learning tasks such image classification (Hinton al. 2015), object detection (Chen et al. 2017b) 7b) and speech translation (Liu et al. 2019c). When applied to face recognition, the embeddings of a gallery are usually extracted by a larger teacher model while the embeddings of the query images are extracted by smaller student model. The student model is encouraged to align its embedding space with that of the teacher, so as to improve its recognition capability.
Previous KD works promote the consistency in final predictions (Hinton et al., 2015). or in the activations of the hidden layer between student and teacher (Romero al.) [014]] Zagoruyko & Komodakis 2016). Such an idea of only optimizing the consistency in predictions activations brings limited performance boost since the student is often a small model with weaker capacity compared with the teacher. Later, Park et al.] (2019); |eng al. (2019) propose to exploit the correlation between instances to guide the student to mimic feature relationships of the teacher over a batch of input data, which achieves better performance. However, the above works all aim at guiding the student to mimic the behavior of the teacher, which is not suitable for practical face recognition. In reality, it is very important directly align embedding spaces between student and teacher, which can enable models across different devices to share the same embedding space for feasible similarity comparison. To solve this, a simple and direct method is to directly minimize the L2 distance of embeddings extracted by student and teacher. However, this method (we call it I.2KD) only considers minimizing the intra-class distance and ignores maximizing the inter-class distance, and is unable to benefit from some powerful loss functions with large margin (e.g. Cosface loss (Wang et al. 2018a), Areface loss (Deng et al. 2019a)) constraint to further improve the performance.

---

Under review as a conference paper at ICLR 2021
In this work, we propose an effective knowl- edge distillation method named ProxylessKD. According to Ranjan et al. (2017), the classifier neurons in a recognition model can be viewed the approximate embedding centers of each class, This can be used to guide the embed- ding learning as in this way, the classifier can encourage the embedding to align with the ap- proximate embedding centers corresponding to the label of the image. Inspired by this, we propose to initialize the weight of the student's classifier with the weight of the teacher's s clas-
10
06
04
02
0.0
L2KD
ProcylesKD
Figure 1: The embedding distributions extracted by (a) L2KD, and (b) ProxylessKD
sifier and fix it during the distillation process, which forces the student to produce an embedding space as consistent with that of the teacher as possible. Different from previous knowledge dis-
tillation works (Hinton et al. 2015 Zagoruyko & Komodakis, 2016, Romero et al.] 2014 Park et al. 2019| Peng et al., 2019) and L.KDD, the proposed ProxylessKD not only directly optimizes the target task but also considers minimizing the intra-class distance and maximizing the inter-class distance. Meanwhile it can benefit from large margin constraints (e.g. Cosface loss (Wang al.) 2018a) and Areface loss (Deng et al. 2019a)). As shown in Figure 1 the intra-class distance in ProxylessKD combined with Areface loss is much closer than L2KD, and the inter-class distance in ProxylessKD combined with Areface loss is much larger than L2KD. Thus it can be expected that our ProxylessKD able to improve the performance of face recognition, which will be experimen- tally validated.
The main contributions in this paper are summarized follows:
We analyze the shortcomings of existing knowledge distillation methods: they only opti- mize the proxy task rather than the target task; and they cannot conveniently integrate with advanced large margin constraints to further lift performance.
We propose a simple yet effective KD method named ProxylessKD, which directly boosts embedding space alignment and can be easily combined with existing loss functions to achieve better performance.
We conduct extensive experiments on standard face recognition benchmarks, and the results well demonstrate the effectiveness of the proposed ProxylessKD.
RELATED WORK
Knowledge distillation. Knowledge distillation aims to transfer the knowledge from the teacher model to small model. The pioneer work is BBucilu et al. (2006). and Hinton et al. (2015) popu- larizes this idea by defining the concept of knowledge distillation (KD) as training the small model (the student) by exploiting the soft targets provided by a cumbersome model (the teacher). Unlike the one-hot label, the soft targets from the teacher contain rich related information among classes, which can guide the student to better learn the fine-grained distribution of data and thus lift perfor- mance. Lots of variants of model distillation strategies have been proposed and widely adopted in the fields like image classification (Chen et al. 2018), object detection (Chen et al.] 2017a), seman- tic segmentation (Liu et al., 2019a Park & Heo 2020), etc. Concretely, Zagoruyko & Komodakis (0166) proposed response- based KD model, Attention Transfer (AT), which aims to teach the stu- dent to activate the same region as the teacher model. Some relation-based distillation methods have also been developed, which encourage the student mimic the relation of the output in different stages (Yim et al. (217)) and the samples in batch (Park et al., 2019). The previous works mostly optimize the proxy tasks rather than the target task. In this work, we directly optimize face recog- nition accuracy by inheriting the teacher's classifier as the student's classifier to guide the student to learn discriminative embeddings in the teacher's embedding space. In (Deng et al., 2019b), they also directly copy and fix the weights of the margin inner-product layer of the teacher model to the student model to train the student model and the motivation of (Deng et al., 2019b) is the student model can be trained with better pre-defined inter-class information from the teacher model. How- ever, different from (Deng et al. 2019b). we firstly analyze the shortcomings of existing knowledge distillation methods. Specifically, the existing methods target optimizing the proxy task rather than
2

---

Under review as a conference paper at ICLR 2021
the target task; and they cannot conveniently integrate with advanced large margin constraints to further lift performance. These valuable analyses and observations are not found in (Deng et al.) 2019b) and other existing works. Secondly strong motivation and the physical explanation of the proposed Proxyless is well explained in our work. Figure corresponding analysis explained why ProxylessKID can achieve better performance than the existing methods that optimize the proxy task, Such in-depth analysis and strong physical explanation are novel and cannot be found in (Deng al., 2019b) and other existing works. We believe these novel findings and the proposed solution are valuable to the face recognition community and will inspire researchers in related fields. Fi- nally, solid experiments are designed and conducted to justify the importance of directly optimize the final task rather than the proxy task when doing knowledge distillation. And the properties of ProxylessKD about using different margin-based bagedd loss function and syper-parameterr are well exam- ined. These detailed analyses about ProxylessKD cannot be found in (Deng et al. (2099b) and other existing works. We believe the above important differences and novel contributions make our work differs from (Deng et al. 2019b) and existing works.
Loss functions used in face recognition. Softmax loss is defined as the pipeline combination of the last fully connected layer, softmax function, and cross-entropy loss. Although it help the network separate categories in a high-dimensional space, for fine- fine-grained classification prob- lems like face recognition, it offers limited accuracy due to the considerable inter-class similarity. Liu et al. (2017) proposed Sphereface to achieve smaller maximal intra-class distance than minimal inter-class distance, which can directly enhance feature discrimination. Compared with SphereFace in which the margin m is multiplied on the angle, Wang et al. (20tBa):Whitelam Whitelam et al. (2017) pro- posed CosFace, where the margin directly subtracted from cosine, achieving better performance than SphereFace and relieving the need for joint supervision from the softmax loss. To further im- prove feature discrimination, Deng et al. (2018) proposed the ArcFace that utilizes the arc-cosine function to calculate the angle, i.e. adding an additive angular margin and back again by the cosine function. In this paper, we combine our ProxylessKD with the above loss functions to further lift performance, e.g. Arcface loss function.
3
METHODOLOGY
We first revisit popular loss functions in face recognition in Sec. 3.1 and elaborate on our Proxy- lessKD in Sec. 3.2) Then we introduce how to combine our method with existing loss functions in Sec.
3.1
REVISIT LOSS FUNCTION IN FACE RECOGNITION
The most classical loss function in classification is the Softmax loss, which is represented as follows:
e (-Cyy(
L
log e (cDelluug
+ Re (00co0(
Here, wh denotes the weight of the model classifier, where k E {1, K} and K denotes the number of classes. I; the embedding of i-th sample and usually normalized with magnitude replaced with scale parameter of s. Owg,a; denotes the angle between Wk and 24. y is the ground truth label for the input embedding Tt, N the batch size. In recent years, several margin-based softmax loss functions (Liu et al. 2017]] Waang et al., 2017 2018a: Deng et al. (019a) have been proposed to boost the embedding discrimination, which is represented as follows:
1 L = N Σ log s-f(m, Bmy..:)) + +E e-cos(pugg:s))
(mm, 0uy.=,)
In the above equation, [mm, ow,,.r)) is a margin function. Precisely, f'mm fw,,,,)) cos(m (w)) is A-Softmax loss proposed in (Liu et al., (207)), where m is an integer and greater than zero fm, 0., = cos ...) - m is the AM-Softmax loss proposed in Wang et al. (2018a) and the hyper-parameter is greater than zero. f(m, 0m,,.,)== coos0or +m)) with m introduced in Deng et al. (2019a), which achieves better performance than the former. Fortunately, the proposed ProxylessKD can be combined with the above loss function, conveniently. In this paper, we combine our proposed ProxylessKD method with above loss functions and investigate their performance.
is Arc-Softmax
3

---

Under review as a conference paper at ICLR 2021
Model
Model
Model
-Ep
weight share
weight
share
Model
Model
Eg
Model
Fix weight
Fix weight.
Input
(a) KD with single teacher
Input
(b) KD with multiple teachers
Figure 2: The diagram of proposed ProxylessKD. We firstly train teacher model (a) or ensemble multiply trained teacher models (b), and then initialize the weight of the student's classifier with the weight of the teacher's the ensembled teachers' classifier and fix the weight in the distillation stage.
3.2 INHERITED CLASSIFIER KNOWLEDGE DISTILLATION
The models trained for different devices are expected to share the same embedding space for similar- ity comparison. However, most existing knowledge distillation models only optimize proxy tasks, encouraging the student to mimic the teacher's behavior, instead of directly optimizing the target accuracy. In this paper, we propose to directly optimize the target task by inheriting the teacher's classifier to encourage better embedding space consistency between the student and the teacher.
Knowledge distillation with single teacher. In most of the existing distillation works (Chen al.) 2018 Liu al. 2019a; Park & Hco 2020), single large model is utilized as the teacher to guide the student. Hence, we firstly introduce our ProxylessKD method under the common single teacher model knowledge distillation, shown in Figure (a). where Ey and Es represent the embedding extracted by the teacher model and the student model S, respectively. Unlike previous works (Hinton al.] 2015 Zagoruyko & Komodakis 2016) that optimize the proxy task, we emphasize on optimizing the target task. To this end, we try to directly align the embedding space between the teacher model and the student model. Specifically, we firstly train teacher model with the Areface loss (Deng et al. (019a), and then initialize the weight of the student's classifier with the weight of the teacher's 's classifier and fix the weight in the distillation stage. Using Areface loss enables our method to benefit from large margin constraints in the distillation procedure. The distillation form of Figure (() (a) can be defined as follows:
N log log s smmm Ek, s-cos(0 (g
Σ L3 N
1
(3)
(m, Owp,a;) is margin function, æ; is the the embedding of i-th sample in batch, w and w are classifier's weights from the teacher model, y the class of the Ii, k E 2,, 3) and K is the number of classes in the dataset. Out is the angle between w' and I;. m denotes the preset hyper-parameter. When we adjust the value of m, the interval among different intre-class samples will be changed.
Knowledge distillation with multiple teachers: Using an ensemble of teacher models would fur- ther boost the performance of knowledge distillation, according to previous work (Asif et al.) 2019)) Therefore, we here introduce how to implement ProxylessKD with the ensemble of teacher models in Figure 2 which better aligns with a practical face recognition system. To do this, we firstly train n different teacher models to acquire n embeddings for each input and concatenate the n embed- dings to produce a high-dimensional embedding. Secondly, employee dimensionality reduc- tion layer or the PCA (Principal Component Analysis) method (Wold et al.] 1987) to reduce the nigh-dimensional embedding to adapt to student's 's embedding dimensional. Finally, embedding after dimensionality reduction into a new classifier and retrain it. We will do the same operate the knowledge distillation with single teacher, when obtain the new classifier. The
input the
4

---

Under review as a conference paper at ICLR 2021
ensemble of teachers' classifier can be optimized
FT = spcomcatenattin/ Ei Eagi... Frn)))
(4)
4
log s-f e-I(m,,urr.ee + E g.
Et., 1.,,.., n),, 2 2 and N+, is the embedding of the i-th sample extracted by teacher models, ET is the dimensionality reduction vector of the i-th sample, concatenation is the operation of concatenating embeddings, ş is dimensionality reduction function (i.e., PCA function or the dimensionality reduction layer function).
3.3 INCORPORATING WITH OTHER LOSS FUNCTIONS
In Sec. 3.1 we only introduce the classic loss functions (i.e., Equation and in face recognition. As long as the loss function uses the output of the classifier calculate the loss (e.g., ArcNegFace (Liu et al. (2019b) for optimizing the network, the proposed ProxylessKD method can combine with it. Therefore, more powerful loss functions can be incorporated into our ProxylessKD to further improve the performance. The unified form can be defined as follows:
L
Σ
((w", x3))
(5)
I; is the embedding of the i-th sample. w' is the weight of the classifier from the teacher model. batch. We use (-) to represent the loss calculated by various loss
N is the number of samples function types. Note, it is not restricted to the field of face recognition but also applicable to other general classification tasks where our ProxylessKD is used for model performance improvement. For example, the recent work (Sun et al.) (020) proposed circle loss with excellent results achieved for the fine-grained classification tasks. It can integrated into the proposed ProxylessKD to further boost the performance in the general classification tasks.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Datasets. We adopt the high-quality version namely MSIMV2 refined from MS-Celeb- IM dataset (uuo et al. 2016) Deng al. (2019a) for training. For testing. we utilize three face verification datasets, i.e. LFW (Huang et al.) 2008), CPLFW (Zheng & Deng 2018). CFP- FP (Sengupta al., (016). Besides, also test our proposed method on large-scale image Samnelmachhr-Shhizemmme datasets MegaFace (Whitelam et (017)) and et al. 2016). IJB-B
Datasets
#ldentity
#lmage
MSIMV2
85K
5.8M
LFW
5,749
13,233
CPLFW
5,749
11,652
CFP-FP
500
7,000
MegaFace
530(P)
IM(G)
IJB-B
1,845
76.8K
JJB-C
3,531
148.8K
IJB-C (aze et al. 2018)). Details about these datasets are shown in Table
Table Face recognition datasets for training and testing. (P), (G) mean the probe, gallery set, spectively.
Data processing. We follow WWan et al.
(0188); Deng et al. (2019a) to generate the normalized face crops (112 x 112) with five fa-
cial points in the data processing. All training faces horizontally flipped with probability 0.5 for data augmentation.
Network architecture. In this parper, we set the n=4, i.e., the four models are the 555 ResNet101, AttentionNet92 and DenseNet201 as the ensemble of teacher models, and choose ResNet18 the student model. After the last convolutional layer, leverage the FC-BN struc- ture to get the final 512-D embedding. In the ensemble procedure of four teacher models, we train again new dimensionality reduction layer and the classifier layer with the feature that is cascaded
5

---

Under review as a conference paper at ICLR 2021
from four teacher models input to acquire new embedding. This new embedding is applied to knowledge distillation in L2KD, and the new classifier is inherited by student model to do knowl- edge distillation.
1983) and 512 batch size for each teacher training and 1024 for the remaining training procedure. The momentum is set to 0.9 and the weight decay 4e-5. The dimension of all embedding is 512. The initial learning rate is set 0.1, 0.1, 0.9, 0.35 in the training of the teacher, student, L2KD, and ProxylessKD, respectively. The training process for the teacher model is finished with epochs, and 16 epochs is used for all remaining experiments. We use the cosine decay in all training and Arcface loss as the supervision, in which m = 0.5 and s=64 following Deng et al. (2019a). All experiments are done on 8x2080Ti GPUs in parallel and implemented by the Mxnet library based on Gluon--Face
Training, All models are trained from scratch with NAG Nesterov
Testing, In practical face recognition or image search, the embeddings of database usually are ex- tracted by larger model while the embeddings of query images are are extracted by a smaller model. Considering this, we should evaluate the consistency of embeddings extracted by the larger model and the smaller model, which represents the performance of different KD methods. Specifi- cally, in the identification task, large model is used to extract the embeddings of the database, and a small model is used to extract embeddings of the query images. In the verification task, we firstly calculate the verification accuracy using embeddings of image pair extracted by the large and the small model respectively. then calculate the verification accuracy using embeddings of image pair extracted by the small and the large model in turn, finally take the average of them. In particular, use the same verification method on the small datasets (i.e., LFW, CPLFW, and CFP-FP) following Wang et al. (2018b); Deng al. (2019)). We use kinds of measure methods (i.e., verification and identification) to test IJB-B, UB-C dataset and MegaFace dataset. Note the images in the IM interference set are extracted by the larger model, and the query images are extracted by the small model in the MegaFace dataset. Meanwhile, we introduce performance when we only use small model.
4.2 ABLATION STUDY
Results different loss functions. Our ProxylesSKD can be easily combined with the existing loss functions (e.g., L2softmax (Ranjan et al.) 2017), Cosface loss (Wang et al. (0188)). Areface loss (Deng et al. 2019a)) to supervise the learning of embedding and achieve better embedding space alignment. In Table 2 we show the performance difference our method under the supervision of different loss functions.
As shown in Table 2 compared with L2softmax and Cosface loss, our method achieves better results with the supervision of the Areface loss function. This proves that our ProxylessKD is able lift performance by incorporating powerful loss function. Hence, we can foresee that our method will achieve better results with the development of more powerful loss functions in classification.
Method
LFW
CPLFW
CFP-FP
ProxylessDD + L2softmax
99.50
90.45
94.55
ProxylessKD= Cosface
99.66
90.46
93.83
ProxylessKD + Arcface
99.66
90.55
93.95
Table 2: The evaluation results of different loss functions on LFW,CPLFW and CFP-FP
MegaFace Rank-
Method
ProxylessKD L.2 softmax
84.94
87.66
ProxylessKD Cosface
95.55
96. 15
ProxylessKD + Arcface
95.80
96.21
Table 3: The evaluation results of different loss functions on MegaFace dataset
Results different margins. As shown in Figure to further illustrate the impact of different margins on different scale datasets, we utilize Areface loss with different margins as the supervised loss of the proposed ProxylessKD to test its sensitivity margins. Red points mark the best results.

---

Under review as a conference paper at ICLR 2021
LIBB
LBC
Method
T-N
T-N
67 FAR==1c-
@FAR=1c-6
Top1
G FAR=
FFRR-I--6
Top1
ProxyleskKD + L2 soitmax
91.93
24.40
92.57
93.97
82.57
93.78
ProxylessKD Cosface
92.91
39.81
93.57
94.70
88.15
94.82
ProxylessKD Areface
93.05
41.07
93.75
94.71
87.90
94.88
Table 4: The evaluation results of different loss functions on IJB-B and LJB-C dataset.
Specifically, from Figure 3 (a), (b), and (c), we observe that margin=0.2 achieves better results than others. Though on the LFW dataset the best result is gained at margin=0.4, the gap is tiny and merely 0.02%, which demonstrates the small margin more appropriate the small scale dataset. However, the IJB-//C and MegaFace that are large scale datasets, we find larger margins bring better results. In particular, when the margin is set to 0.5, the same as the setting training ensemble of teacher's classifier, the performance is the best. This indicates the performance of ProxylessKD will be better if using larger margin at a large scale dataset, as shown in Figure
(d) ~ (k).
99
94.0
99.67
90.4
93.0
93.6
99.64
93.4
99
00.0
93.2
93.0
99.61
02.8
00.60
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
margin
margin
margin
margin
(a) Armmeyy(%) on
Acomar(99)
(c) Acmmeyy9% CFF-FF
Magafooce
96
93.7
32.9
92.0 0
38
93.5
93.4
93
92.5
03.2
93.1
92.3
30
0.3
margin
margin
(r) (0TTARCM00FFRR-0066 MazaRom (0 TARCWe on [[B B(1-1) (g) TARR% (JB-BBB(j)B)1)
(h) TOP1(%) on (JB-R(1-N)
94.7
34.6
04.5
87.0
94.4
86.0
0.3
0.4
margin
margin
TOP1(%)
Figure 3: The evaluation of different margins in Arcface loss function
4.3 COMPARING WITH OTHER METHIODS
Single and multiple model mode. We show two evaluation modes in Table 5 6
L2KD-s and ProxylessKD mean the embeddings are only extracted by the student model (single model mode), and the suffix of "-m" represents the embeddings of database are extracted with a teacher model while the embeddings of query images are extracted by the student model (multiple model mode). The more detailed information can be found in Sec.
As shown in Table 5 the accuracy on LFW is similar between L2KD and ProxylessKD, but our ProxylessKD achieves better performance CPLFW and CFP-FP under two evaluation modes. In particular, we achieve 0.91% % and 1.32% improvements on CPLFW and CFP-FP under single model mode, and 1.02% and 0.74% better than the L2KD under multiple model mode. Note that ProxylessKD is trained with margin=0.2 with Areface loss and the margin set to 0.5 in the next experiments.

---

Under review as a conference paper at ICLR 2021
Method
LFW
CPLFW
CHP-FP
Teacher
99.73
93.00
97.63
Student
98.68
85.20
89.50
L2KD-s
99.65
87.82
90.51
ProxylessKD-s
99.61
88.73
91.88
L2KD-m
99.72
89.98
94.3
ProxylessKD-m
99.70
91.00
95.04
Table 5: Verification performance (%) of single model mode(-s) and multiple model mode(-m) LFW, CPLFE, CFP-FP datasets.
Megatace
Method
Rank-
Ver( (%) (1))FFAReee
Teacher
97.86
97.98
Student
62.15
58.85
L2KD-s
91.42
93.50
ProxylessKD-s
92.45
93.32
1.2KD-m
95.67
96.22
ProxylessKD-m 95,80
96.21
Table 6: Face identification and verification evaluation results of single model mode(-s) and multiple model mode(-m) on MegaFace. Rank-1 is face identification accuracy with IM distractors, and "Ver" refers to the face verification TAR 10 FAR.
The MegaFace dataset contains 100K photos of 530 unique individuals from FaceScrub (Ng & Winkler 2014) as the probe set and 1M images of 690K different individuals as the gallery set. On Megalace, employ testing tasks (verification and identification) under two mode (i.e.. single model mode and multiply model mode). In the testing, except the features of gallery images extracted by the teacher model, the features the images input the gallery each time is also extracted by the teacher model. In Table 6 we show the performance of I.2KD) and ProxylessKD. Though MegaFace larger scale dataset, and for more complex recognition task, our proposed method still achieves better results in Rank-1 and boosts the performance by 1.03% and 0.12% under the single model mode and multiple model mode, respectively. And, multiple model mode evaluation, it performs better than single model mode.
LIB-
JB-C
Method
I-N
I-N
@ OFAR=1e-4
@FAR=1e-6
Topl
@ @FAR=1e44
@ @FAR=1e-6
Topl
Teacher
93.60
46.85
94.34
95.07
54.54
95.30
Student
77.05
24.50
86.12
80.94
43.99
85.20
1.2KD-s
88.87
41.74
91.42
91.01
70.70
92.73
ProxylessKD-s
90.43
39.39
92.30
92.50
76.83
93.36
L2KD-m 92.25
40.08
93.39
94.04
85.30
94.60
ProxylessKD-m
93.05
41.07
93.75
94.71
87.90
94.88
Table 7: 1:1 verification TAR(%) (CuuracyToop(%)..
(@FAR=1c-4 and @FAR=1c-6) and 1:N identification
The IJB-B dataset (Whitelam et al. (011)) has 1,845 subjects with 21.8K static images and 55 frames from 7,011 1 videos. There are 12,115 templates with 10,270 authentic matches and M impostor matches in all. The IJB-C dataset (Maze al. 2018) the extension IJB-B, containing 3,531 subjects with 31.3K static images and 117.5K frames from 11.779 videos. There are 23.124 templates with 19,557 genuine matches and 15,639 impostor matches in all. As shown in Table compared with the L2KD, ProxylessKD achieves the best results among all evaluation methods IJBB and IJBC and
consistent improvement than L2KD in embedding space alignment between teacher and stu- dent. We can see the performance of multiple model mode is better than single model mode, which explains the reason why the base database embeddings extracted by large model in the practical
8

---

Under review as a conference paper at ICLR 2021
face recognition. Especially @FAR 1e 6 on IJB-C, the accuracy of multiple model mode is improved by 10 ~ 15 than single model mode. This explains why we advocate directly aligning the embedding space is more essential due to the base database features also extracted by large model in the parctical face recoginition. The experiments prove ProxylessKD is a more effective strategy than the existing practical face recognition method (i.e., L.2KD).
CONCLUSIONS
In this work we propose a simple yet powerful knowledge distillation method named ProxylessKD, which inherits the teacher's classifier the student's classifier to directly optimize the remaining networks of the student while fixing the classifier's weights in the training procedure. Compared with L2KD, which only considers the intra-class distance and ignores the intra-class distance, our proposed ProxylessKD pays attention to them both. Meanwhile, it can benefit from the large margin of existing constraints, which new to exiting knowledge distillation research. Our method can achieve better performance than other distillation methods in most evaluations, proving its effective- ness.
REFERENCES
Umar Asif, Jianbin Tang, and Stefan Harrer. Ensemble knowledge distillation for learning improved and efficient networks. arXiv preprint WxXiv:/90.008/00 2019.
Cristian Buciluä, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro-
ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-535-1111 2006.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances Neural Information Process- ing Systems, pp. 742-751, 2017a.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances Neural Information Process- ing Systems, pp. 742-751, 2017b.
Wei-Chun Chen, Chia-Che Chang, and Che-Rung Lee. Knowledge distillation with feature maps for image classification. In Asian Conference on Computer Vision, pp. 200-215. Springer, 2018.
Jiankang Deng, Jia Guo, and Stefanos Zafeiriou Arcface. Additive angular margin loss for deep face recogmition. CORR, 2018.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4690-4699, 2019a.
Jiankang Deng, Jia Guo, Debing Zhang, Yafeng Deng, Xiangju Lu, and Song Shi. Lightweight face recognition challenge. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019b.
Gluon-Face. [EB/OL].
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- lm: A dataset and benchmark for large-scale face recognition. In European conference computer vision, pp- 87-102. Springer, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 10Xiv:1503.0255: 2015.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Leamed-Mille Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. 2008.

---

Under review as a conference paper at ICLR 2021
Ira Saneemmacher-Shizzmmma Steven M Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: 1 million faces for recognition at scale. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4873 4882, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li. Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212--20,,10 2017.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2604-2613, 2019a.
Yu Liu et al. Towards flops-constrained face recognition. In Proceedings of the IEFE International Conference on Computer Vision Workshops, pp. 0-0, 2019b.
Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. End-to-end speech translation with knowledge distillation. arXiv preprint 10Xiv:900400805, 2019c.
Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. larpa janus benchmark-c: Face dataset and protocol. In 2018 International Conference on Biometrics (ICB), pp. 158-165. IEEE, 2018.
Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of conver- genee o (1/k" 2). Doklady an ussr, volume 269, pp. 543-547, 1983.
Hong-Wey Wei Ng and Stefan Winkler. data-driven approach to cleaning large face datasets. In 2014 IEEE international conference on image processing (ICIP), pp. 343-347. IEEE, 2014.
Sangyong Park and Yong Seok Heo. Knowledge distillation for semantic segmentation using chan- nel and spatial correlations and adaptive cross entropy. Sensors, 20(16):4616, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE International Conference Computer Vision, pp. 5007-5016, 2019.
Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discrimi- 2017.
native face verification. arXiv preprint
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Soumyadip Sengupta, Jun Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-9. IEEE, 2016
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6398-6407, 2020.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embed- ding for face verification. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1041-1049, 2017.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25(7):926-99) 930, 2018a.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with genera- tive adversarial networks. In Advances in Neural Information Processing Systems, pp. 775-786, 201 1b...
10

---

Under review as a conference paper at ICLR 2021
Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K Jain, James A Duncan, Kristen Allen, et al. larpa janus benchmark- b face dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 90-98, 2017.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 2016.
Tianyue Zheng and Weihong Deng. Cross-pose lfw: A database for studying cross- pose face recog- nition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5, 2018.


Reviews:
[{"review_id": "gp9LY45dkw1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes ProxylessKD method from a novel perspective of knowledge distillation. Instead of minimizing the outputs of teacher and student models, ProxylessKD adopts a shared classifier for two models. The shared classifier yields better aligned embedding space, so the embeddings from teacher and student models are comparable. Since the optimization objective for student model is learning discriminative embeddings,  the face recognition performance is improved compared to the vanilla KL counterpart.\n\nHowever, I still have some concerns.\nFirst, ProxylessKD makes an assumption that the subjects of the dataset for training both models are somehow overlapping, while L2KD does not have such limitation. So more analysis and detailed discussions on the pros and cons of ProxylessKD and L2KD are needed.\n\nSecond, ProxylessKD can be interpreted as initializing classifier of student model by the classifier of teacher model. It would be interesting to see how performance changes with more layers of student model inherited from teacher model. For example, the last two layers, the last three layers.\n\nThird, experimental comparisons with more advanced KD methods are necessary, e.g.[1], [2], [3] etcs. Currently the only comparison with other method is a self-implemented L2KD, which couldn't comprehensively show the effectiveness of the proposed method.\n[1] Park, Wonpyo, et al. \"Relational knowledge distillation.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n[2] Peng, Baoyun, et al. \"Correlation congruence for knowledge distillation.\" Proceedings of the IEEE International Conference on Computer Vision. 2019.\n[3] Karlekar, Jayashree, et al. \"Deep face recognition model compression via knowledge transfer and distillation.\" arXiv preprint arXiv:1906.00619 (2019).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X4hUCAzpu9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a knowledge distillation method for face recognition, which inherits the teacher’s classifier as the student’s classifier and then optimizes the student model with advanced loss functions.  The paper demonstrates using an ensemble of teacher models can boost the performance of knowledge distillation.\n\nStrength:\n- The proposed method is simple and easy to implement. \n- The experimental results demonstrate the effectiveness of the technique.\n- The paper is well organized and well written.\n\nWeakness:\n- The novelty is limited. Directly inheriting the teacher’s classifier is a common strategy in the face recognition community, which can be found in Ref.1.\n- The experiment lacks comparison with the general knowledge distillation methods (Ref.2) in image classification and the specific used methods (Ref.3) in face recognition.\n\n【1】Deng, Jiankang, and Guo, Jia et al. Lightweight Face Recognition Challenge. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops\n【2】Hinton, G., Vinyals, O., Dean., J.: Distilling the knowledge in a neural network. In: arXiv preprint arXiv:1503.02531 (2015)\n【3】Xiaobo Wang and Tianyu Fu et al. Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition. ECCV2020\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MrSB61DPjh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new KD method to inherit classifier from teacher models and utilize it to train the student model feature representation, where previous KD methods are mostly focusing on the proxy task other than the target task itself.\n\nThe idea of using teacher model’s classifier to directly reshape the student model’s feature representation is somewhat novel. It considers the situation of single teacher model and multiple teacher models. The teacher ensemble is achieved by concatenating features from each of the teacher model and then conducting dimension reduction using PCA. The methodology illustration is simple yet clear. There are multiple experiments on major face recognition datasets and demonstrate superior performance against baselines such as L2KD-s.\n\nRegarding the concerns, I am listing them into bullets.\n\n1. why the experiments make the setting of templates using teacher model to extract feature, while the query using student model to extract feature?\n\nWould the comparison of using student model for extracting both template and query feature be possible? It can provide a direct comparison to other methods, i.e. ArcFace trained using ResNet18 compared to the student model with ResNet18. \n\nCurrent experiments lack the comparison to the state-of-the-art methods, i.e. ArcFace and CosFace.\n\n2. the ablation is emphasizing on the ProxylessKD combining with different losses. It does not consider the knowledge distillation itself. For example, when ProxylessKD is combined with the proxy task, i.e., feature distillation loss, how would it perform compared to only ProxylessKD? Meanwhile, in many KD papers, there are also intermediate layer feature distillation, would it harm the overall performance under this paper's setting? It needs sufficient analysis to justify the authors' choice of only applying the teacher model's classifier as distillation.\n\n3. In ablation, how would the number of teachers influence the student performance? Meanwhile, how would the network architecture influence the student performance? i.e., fixing the teachers to be the same, while varying student architecture with multiple hypothesis, i.e., ResNet, AttentionNet, DenseNet? It is good to know what specific architecture is favored under the authors' proposed framework.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a knowledge distillation method for face recognition, by inheriting the teacher’s classifier as the student’s classifier and optimizing the student model with advanced loss functions. It received comments from three reviewers: 1 rated “Ok but not good enough - rejection”, 1 rated “Marginally below” and 1 rated “Marginally above”. The reviewers appreciate the simple yet clear methodology illustration and the well written paper. However, a number of major concerns are raised by the reviewers, including limited novelty, lack of comparison with more advanced knowledge distillation methods and their special case in face recognition. During the rebuttal, the authors made efforts to response to all reviewers’ comments. However, the rating were not changed. The ACs concur these major concerns and more comprehensive comparisons with the state of the art KD methods are necessary to better illustrate the contribution of this work. Therefore, this paper can not be accepted at its current state.

Decision: Reject