Discriminability Distillation in Group Representation Learning

1 INTRODUCTION . With the rapid development of deep learning and the easy access to large-scale group data , recognition tasks using group information have drawn great attention in the computer vision community . The rich information provided by different elements can complement each other to boost the performance of tasks such as face recognition , action recognition , and person re-identification ( Wang et al. , 2017b ; Zhong et al. , 2018 ; Girdhar et al. , 2017 ; Simonyan & Zisserman , 2014 ; Yang et al. , 2017 ; Liu et al. , 2019a ; Rao et al. , 2017b ) . While traditional practice for group-based recognition is to either aggregate the whole set by average ( Li et al. , 2014 ; Taigman et al. , 2014 ) or max pooling ( Chowdhury et al. , 2016 ) , or just sampling randomly ( Wang et al. , 2016 ) , the fact that certain elements contribute negatively in recognition tasks has been ignored . Thus , an important issue is to select representatives from sets for efficient group understanding . To tackle such cases , previous methods aim at defining the “ quality ” or “ saliency ” for each element in a group ( Liu et al. , 2017c ; Yang et al. , 2017 ; Rao et al. , 2017b ; Nikitin et al. , 2017 ) . The weights for each element can be automatically learned by self-attention . For example , Liu et al . ( 2017c ) proposes the Quality Aware Network ( QAN ) to learn quality score for each image inside an image set during network training . Other works adopt the same idea and extend to specific tasks such as video-based person re-identification ( Li et al. , 2018 ; Wu et al. , 2018 ) and action recognition ( Wang et al. , 2018c ) by learning spatial-temporal attentions . However , the whole online quality or attention learning procedures are either manually designed or learned through a black box , which lacks explainability . In this work , we explore deeper into the underlying mechanism for defining effective elements instead of relying on self-learned attention . Assuming that a base network has already been trained for element-based recognition using class labels , we define the “ discriminability ” of one sample by how difficult it is for the network to discriminate its class . As pointed out by Liu et al . ( 2018 ) that the feature embedding of elements lies close to the centroid of their corresponding class are the representatives , while features far away or closer to other classes are the confusing ones which are not discriminative enough . Inspired by this observation , we identify a successful discriminability indicator by measuring one embedding ’ s distance with class centroids and compute the ratio of between positive and hardest-negative , where the positive is its distance with its class ’ s corresponding centroid and the hardest-negative is the closest counterpart . This indicator is defined as the discriminability distillation regulation ( DDR ) . Armed with recent theories on the homogeneity between class centroids and projection weights of classifiers ( Wang et al. , 2017a ; Liu et al. , 2017b ; 2018 ; Deng et al. , 2019a ) , the entire distancemeasuring procedure can be easily accomplished by simply encoding all elements in one group . Thus , the DDR scores can be assessed for each element after the training of the base network . This assessing procedure is highly flexible without human supervision nor re-training the base network , so it can be adapted to any existing base . With our explicitly designed discriminability indicator on the training set , the distillation of such discriminability can be successfully performed with a lightweight discriminability distillation network ( DDNet ) , which shows the superiority of our proposed indicator . We call the whole procedure uniformly as discriminability distillation learning ( DDL ) . The next step is towards finding a better aggregation policy . At the test phase , all elements are firstly sent to the light-weight DDNet . Then element features will be weighted aggregated by their DDR score into group representation . Moreover , in order to achieve the trade-off between accuracy and efficiency , we can filter elements by DDR score and only extract element features of high score . Since the base model tends to be heavy , the filter can save much computation consumption . We evaluate the effectiveness of our proposed DDL on several classical yet challenging tasks . Comprehensive experiments show the advantage of our method on both recognition accuracy and computation efficiency . We achieve state-of-the-art results without modifying the base networks . We highlight our contributions as follows : ( 1 ) We define the discriminability of one element within a group from a more essential and explicable view , and propose an efficient indicator . ( 2 ) We verify that a light-weight network has the capacity of distilling discriminability from the assessed elements . Combining the post-processing with the network , the great computation burden can be saved comparing with existing methods . ( 3 ) We validate the effectiveness of DDL for both efficiency and accuracy on set-to-set face recognition and action recognition through extensive studies . State-ofthe-art results can be achieved . 2 RELATED WORK . 2.1 SET-TO-SET RECOGNITION . Set-to-set recognition which utilizes a group of data of the same class , has been proved efficient on various tasks and drawn much attention these years since the more videos and group datasets are available . Compared with recognition with a single image , set-to-set recognition can further explore the complementary information among set elements and benefit from it . Particularly in this paper , we care for the basic task of face and action recognition . Face Recognition . To tackle set-to-set face recognition problem . ( Wolf et al. , 2011 ; Kalka et al. , 2018 ; Beveridge et al. , 2013 ; Klare et al. , 2015 ) , traditional methods directly estimate the feature similarity among sets of feature vectors ( Arandjelovic et al. , 2005 ; Harandi et al. , 2011 ; Cevikalp & Triggs , 2010 ) . Other works seek to aggregate element features by simply applying max pooling ( Chowdhury et al. , 2016 ) or average pooling ( Li et al. , 2014 ; Taigman et al. , 2014 ) among set features to form a compact representation . However , since most set images are under unconstrained scenes , huge variations on blur , resolution , and occlusion appear , which will degrade the set feature discrimination . How to design a proper aggregation method for set face representation has been the key problem for this approach . Recently , a few methods explore the quality or attention mechanism to form set representation . GhostVLAD ( Zhong et al. , 2018 ) improves traditional VLAD and down weight low quality element features . While Rao et al . ( 2017a ) combine LSTM and reinforcement learning to discard low quality element features . Liu et al . ( 2017c ) and Yang et al . ( 2017 ) introduce attention mechanisms to assign quality scores for different elements and aggregate feature vectors by quality weighted sum . To predict the quality score , an online attention network module is added and co-optimized by the target set-to-set recognition task . However , the definition of generated ’ quality ’ score remains unclear and the learning procedures are learned through a black box , which lacks explainability . In our work , we claim that the most significant indicator to show whether the group representation can be benefited from an element is not the quality or an inexplicable score , but the discriminability . And a novel discriminability distillation learning procedure is proposed . Action recognition . With the advance of multimedia era , millions of hours of videos are uploaded to video platforms every day , so video understanding task like action recognition has become a popular research topic . Real-world videos contain variable frames , so it is not practical to put the whole video to a memory limited GPU . The most usual approach for video understanding is to sample frames or clips and design late fusion strategy to form video-level prediction . Frame-based methods ( Yue-Hei Ng et al. , 2015 ; Simonyan & Zisserman , 2014 ; Girdhar et al. , 2017 ) firstly extract frame features and aggregate them . Simonyan & Zisserman ( 2014 ) propose the twostream network to simultaneously capture the appearance and motion information . Wang et al . ( 2017b ) add attention module and learn to discard unrelated frames . Frames-based methods are computational efficient , but only aggregate high-level frame feature tends to limit the model ability to handle complex motion and temporal relation . Clip-based method ( Tran et al. , 2015 ; 2018 ; Feichtenhofer et al. , 2018 ) use 3D convolutional neural network to jointly capture spatio-temporal features , which perform better on action recognition . However , clip-based methods highly rely on the dense sample strategy , which introduces huge computational consumption and makes it unpractical to application . In this article , we show that by combing our DDL , the clip-based methods can achieve both excellent performance and computation efficiency . 3 DISCRIMINABILITY DISTILLATION LEARNING . In this section , we first formulate the problem of group representation learning in section 3.1 and then define the discriminability distillation regulation ( DDR ) in section 3.2 . Next , we introduce the whole discriminability distillation learning ( DDL ) procedure in section 3.3 . In sections 3.4 and 3.5 , we discuss the aggregation method and the advantage of our DDL , respectively . 3.1 FORMULATION OF GROUP REPRESENTATION LEARNING . Group representation learning focuses on formulating a uniform representation for a whole set of elements . The core of either verification task or classification task is how to aggregate features of a given element group . Define fi as the embedded feature of element Ii in a group IS , then the uniform feature representation of the whole group is FIS = G ( f1 , f2 , · · · , fi ) , ( 1 ) where G indicates the feature aggregation module . While previous research has revealed that conducting G with quality scores ( Liu et al. , 2017c ) has priority over simple aggregation , this kind of methods is not explainable . In this article , we propose a discriminability distillation learning ( DDL ) process to generate the discriminability of feature representation . 3.2 DISCRIMINABILITY DISTILLATION REGULATION . Towards learning efficient and accurate G , we design the discriminability distillation regulation ( DDR ) to generate the discriminability to replace the traditional quality score . In DDR , we jointly consider the feature space distribution and explicitly distill the discriminability by encoding the intra-class distance and inter-class distance with class centroids . Let X denote the training set with the identities C and Wj , j ∈ [ 1 , C ] is the class centroid . For feature fi , i ∈ [ 1 , s ] with class c where Under review as a conference paper at ICLR 2020 Enhanced quality aware network s denotes the size of X , the intra-class distance and inter-class distance are formulated as Cic = fi ·Wc ‖fi‖2 ‖Wc‖2 , Cij = fi ·Wj ‖fi‖2 ‖Wj‖2 , j 6= c. ( 2 ) The intra-class distance Cic and inter-class distance Cij are shown in Figure 2 . After training the base model on classification task , features of the elements from the same class are projected to hyperspace tightly in order to form an explicit decision boundary . Furthermore , elements close to the centroid of their corresponding class are the representative ones , while elements far away from their corresponding class or closer to other classes are not discriminative enough . Based on this observation , we define the discriminability Qi of fi as : Qi = Cic max { Cij | j ∈ [ 1 , C ] , j 6= c } , ( 3 ) i.e. , the ratio of feature distance between the centroids of its class and the hardest-negative class . Considering the variant number of elements in different groups , we further normalize the discriminability by : Di = τ ( Qi − µ ( { Qj | j ∈ [ 1 , s ] } ) σ ( { Qj | j ∈ [ 1 , s ] } ) ) ( 4 ) where τ ( · ) , µ ( · ) and σ ( · ) denote the sigmoid function , the mean value and the standard deviation value of { Qj | j ∈ [ 1 , s ] } , respectively . We denote Di as the discriminability distillation regulation ( DDR ) score . Cooperated with the feature space distribution , the DDR score Di is more interpretable and reasonable than the quality score in traditional quality learning . It can discriminate features better by explicitly encoding the intra- and inter-class distances with class centroids .

Summary: In this paper, the authors proposed a discriminability distillation learning (DDL) method for the group representation learning, such as action recognition recognition and face recognition. The main insight of DDL is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t. the element space can be distilled by a light-weight auxiliary distillation network. The experimental results on the action recognition task and face recognition task show that the proposed method appears to be effective compared with some related methods. The detailed comments are listed as follows,