ProxylessKD: Direct Knowledge Distillation with Inherited Classifier for Face Recognition

1 INTRODUCTION . Knowledge Distillation ( KD ) is a process of transferring knowledge from a large model to a smaller one . This technique is widely used to enhance model performance in many machine learning tasks such as image classification ( Hinton et al. , 2015 ) , object detection ( Chen et al. , 2017b ) and speech translation ( Liu et al. , 2019c ) . When applied to face recognition , the embeddings of a gallery are usually extracted by a larger teacher model while the embeddings of the query images are extracted by a smaller student model . The student model is encouraged to align its embedding space with that of the teacher , so as to improve its recognition capability . Previous KD works promote the consistency in final predictions ( Hinton et al. , 2015 ) , or in the activations of the hidden layer between student and teacher ( Romero et al. , 2014 ; Zagoruyko & Komodakis , 2016 ) . Such an idea of only optimizing the consistency in predictions or activations brings limited performance boost since the student is often a small model with weaker capacity compared with the teacher . Later , Park et al . ( 2019 ) ; Peng et al . ( 2019 ) propose to exploit the correlation between instances to guide the student to mimic feature relationships of the teacher over a batch of input data , which achieves better performance . However , the above works all aim at guiding the student to mimic the behavior of the teacher , which is not suitable for practical face recognition . In reality , it is very important to directly align embedding spaces between student and teacher , which can enable models across different devices to share the same embedding space for feasible similarity comparison . To solve this , a simple and direct method is to directly minimize the L2 distance of embeddings extracted by student and teacher . However , this method ( we call it L2KD ) only considers minimizing the intra-class distance and ignores maximizing the inter-class distance , and is unable to benefit from some powerful loss functions with large margin ( e.g . Cosface loss ( Wang et al. , 2018a ) , Arcface loss ( Deng et al. , 2019a ) ) constraint to further improve the performance . ( a ) L2KD ( b ) ProxylessKD 1.0 0.8 0.6 0.4 0.2 0.0 Figure 1 : The embedding distributions extracted by ( a ) L2KD , and ( b ) ProxylessKD In this work , we propose an effective knowledge distillation method named ProxylessKD . According to Ranjan et al . ( 2017 ) , the classifier neurons in a recognition model can be viewed as the approximate embedding centers of each class . This can be used to guide the embedding learning as in this way , the classifier can encourage the embedding to align with the approximate embedding centers corresponding to the label of the image . Inspired by this , we propose to initialize the weight of the student ’ s classifier with the weight of the teacher ’ s clas- sifier and fix it during the distillation process , which forces the student to produce an embedding space as consistent with that of the teacher as possible . Different from previous knowledge distillation works ( Hinton et al. , 2015 ; Zagoruyko & Komodakis , 2016 ; Romero et al. , 2014 ; Park et al. , 2019 ; Peng et al. , 2019 ) and L2KD , the proposed ProxylessKD not only directly optimizes the target task but also considers minimizing the intra-class distance and maximizing the inter-class distance . Meanwhile it can benefit from large margin constraints ( e.g . Cosface loss ( Wang et al. , 2018a ) and Arcface loss ( Deng et al. , 2019a ) ) . As shown in Figure 1 , the intra-class distance in ProxylessKD combined with Arcface loss is much closer than L2KD , and the inter-class distance in ProxylessKD combined with Arcface loss is much larger than L2KD . Thus it can be expected that our ProxylessKD is able to improve the performance of face recognition , which will be experimentally validated . The main contributions in this paper are summarized as follows : • We analyze the shortcomings of existing knowledge distillation methods : they only optimize the proxy task rather than the target task ; and they can not conveniently integrate with advanced large margin constraints to further lift performance . • We propose a simple yet effective KD method named ProxylessKD , which directly boosts embedding space alignment and can be easily combined with existing loss functions to achieve better performance . • We conduct extensive experiments on standard face recognition benchmarks , and the results well demonstrate the effectiveness of the proposed ProxylessKD . 2 RELATED WORK . Knowledge distillation . Knowledge distillation aims to transfer the knowledge from the teacher model to a small model . The pioneer work is Buciluǎ et al . ( 2006 ) , and Hinton et al . ( 2015 ) popularizes this idea by defining the concept of knowledge distillation ( KD ) as training the small model ( the student ) by exploiting the soft targets provided by a cumbersome model ( the teacher ) . Unlike the one-hot label , the soft targets from the teacher contain rich related information among classes , which can guide the student to better learn the fine-grained distribution of data and thus lift performance . Lots of variants of model distillation strategies have been proposed and widely adopted in the fields like image classification ( Chen et al. , 2018 ) , object detection ( Chen et al. , 2017a ) , semantic segmentation ( Liu et al. , 2019a ; Park & Heo , 2020 ) , etc . Concretely , Zagoruyko & Komodakis ( 2016 ) proposed a response-based KD model , Attention Transfer ( AT ) , which aims to teach the student to activate the same region as the teacher model . Some relation-based distillation methods have also been developed , which encourage the student to mimic the relation of the output in different stages ( Yim et al. , 2017 ) and the samples in a batch ( Park et al. , 2019 ) . The previous works mostly optimize the proxy tasks rather than the target task . In this work , we directly optimize face recognition accuracy by inheriting the teacher ’ s classifier as the student ’ s classifier to guide the student to learn discriminative embeddings in the teacher ’ s embedding space . In ( Deng et al. , 2019b ) , they also directly copy and fix the weights of the margin inner-product layer of the teacher model to the student model to train the student model and the motivation of ( Deng et al. , 2019b ) is the student model can be trained with better pre-defined inter-class information from the teacher model . However , different from ( Deng et al. , 2019b ) , we firstly analyze the shortcomings of existing knowledge distillation methods . Specifically , the existing methods target optimizing the proxy task rather than the target task ; and they can not conveniently integrate with advanced large margin constraints to further lift performance . These valuable analyses and observations are not found in ( Deng et al. , 2019b ) and other existing works . Secondly strong motivation and the physical explanation of the proposed ProxylessKD is well explained in our work . Figure 1 and corresponding analysis explained why ProxylessKD can achieve better performance than the existing methods that optimize the proxy task . Such in-depth analysis and strong physical explanation are novel and can not be found in ( Deng et al. , 2019b ) and other existing works . We believe these novel findings and the proposed solution are valuable to the face recognition community and will inspire researchers in related fields . Finally , solid experiments are designed and conducted to justify the importance of directly optimize the final task rather than the proxy task when doing knowledge distillation . And the properties of ProxylessKD about using different margin-based loss function and hyper-parameters are well examined . These detailed analyses about ProxylessKD can not be found in ( Deng et al. , 2019b ) and other existing works . We believe the above important differences and novel contributions make our work differs from ( Deng et al. , 2019b ) and existing works . Loss functions used in face recognition . Softmax loss is defined as the pipeline combination of the last fully connected layer , softmax function , and cross-entropy loss . Although it can help the network separate categories in a high-dimensional space , for fine-grained classification problems like face recognition , it offers limited accuracy due to the considerable inter-class similarity . Liu et al . ( 2017 ) proposed Sphereface to achieve smaller maximal intra-class distance than minimal inter-class distance , which can directly enhance feature discrimination . Compared with SphereFace in which the margin m is multiplied on the angle , Wang et al . ( 2018a ) ; Whitelam et al . ( 2017 ) proposed CosFace , where the margin is directly subtracted from cosine , achieving better performance than SphereFace and relieving the need for joint supervision from the softmax loss . To further improve feature discrimination , Deng et al . ( 2018 ) proposed the ArcFace that utilizes the arc-cosine function to calculate the angle , i.e . adding an additive angular margin and back again by the cosine function . In this paper , we combine our ProxylessKD with the above loss functions to further lift performance , e.g . Arcface loss function . 3 METHODOLOGY . We first revisit popular loss functions in face recognition in Sec . 3.1 , and elaborate on our ProxylessKD in Sec . 3.2 . Then we introduce how to combine our method with existing loss functions in Sec . 3.3 . 3.1 REVISIT LOSS FUNCTION IN FACE RECOGNITION . The most classical loss function in classification is the Softmax loss , which is represented as follows : L1 = − 1 N N∑ i=1 log es·cos ( θwy , xi ) es·cos ( θwy , xi ) + ∑K k 6=y e s·cos ( θwk , xi ) . ( 1 ) Here , wk denotes the weight of the model classifier , where k ∈ { 1 , 2 , ... , K } and K denotes the number of classes . xi is the embedding of i-th sample and usually normalized with magnitude replaced with a scale parameter of s. θwk , xi denotes the angle between wk and xi . y is the ground truth label for the input embedding xi . N is the batch size . In recent years , several margin-based softmax loss functions ( Liu et al. , 2017 ; Wang et al. , 2017 ; 2018a ; Deng et al. , 2019a ) have been proposed to boost the embedding discrimination , which is represented as follows : L2 = − 1 N N∑ i=1 log es·f ( m , θwy , xi ) es·f ( m , θwy , xi ) + ∑K k 6=y e s·cos ( θwk , xi ) . ( 2 ) In the above equation , f ( m , θwy , xi ) is a margin function . Precisely , f ( m , θwy , xi ) = cos ( m · θwy , xi ) is A-Softmax loss proposed in ( Liu et al. , 2017 ) , where m is an integer and greater than zero . f ( m , θwy , xi ) = cos ( θwy , xi ) −m is the AM-Softmax loss proposed in Wang et al . ( 2018a ) and the hyper-parameterm is greater than zero . f ( m , θwy , xi ) = cos ( θwy , xi+m ) withm > 0 is Arc-Softmax introduced in Deng et al . ( 2019a ) , which achieves better performance than the former . Fortunately , the proposed ProxylessKD can be combined with the above loss function , conveniently . In this paper , we combine our proposed ProxylessKD method with above loss functions and investigate their performance .

Summary: This paper proposes ProxylessKD method from a novel perspective of knowledge distillation. Instead of minimizing the outputs of teacher and student models, ProxylessKD adopts a shared classifier for two models. The shared classifier yields better aligned embedding space, so the embeddings from teacher and student models are comparable. Since the optimization objective for student model is learning discriminative embeddings,  the face recognition performance is improved compared to the vanilla KL counterpart.