Discriminability Distillation in Group Representation Learning

1 INTRODUCTION . With the rapid development of deep learning and the easy access to large-scale group data , recognition tasks using group information have drawn great attention in the computer vision community . The rich information provided by different elements can complement each other to boost the performance of tasks such as face recognition , action recognition , and person re-identification ( Wang et al. , 2017b ; Zhong et al. , 2018 ; Girdhar et al. , 2017 ; Simonyan & Zisserman , 2014 ; Yang et al. , 2017 ; Liu et al. , 2019a ; Rao et al. , 2017b ) . While traditional practice for group-based recognition is to either aggregate the whole set by average ( Li et al. , 2014 ; Taigman et al. , 2014 ) or max pooling ( Chowdhury et al. , 2016 ) , or just sampling randomly ( Wang et al. , 2016 ) , the fact that certain elements contribute negatively in recognition tasks has been ignored . Thus , an important issue is to select representatives from sets for efficient group understanding . To tackle such cases , previous methods aim at defining the “ quality ” or “ saliency ” for each element in a group ( Liu et al. , 2017c ; Yang et al. , 2017 ; Rao et al. , 2017b ; Nikitin et al. , 2017 ) . The weights for each element can be automatically learned by self-attention . For example , Liu et al . ( 2017c ) proposes the Quality Aware Network ( QAN ) to learn quality score for each image inside an image set during network training . Other works adopt the same idea and extend to specific tasks such as video-based person re-identification ( Li et al. , 2018 ; Wu et al. , 2018 ) and action recognition ( Wang et al. , 2018c ) by learning spatial-temporal attentions . However , the whole online quality or attention learning procedures are either manually designed or learned through a black box , which lacks explainability . In this work , we explore deeper into the underlying mechanism for defining effective elements instead of relying on self-learned attention . Assuming that a base network has already been trained for element-based recognition using class labels , we define the “ discriminability ” of one sample by how difficult it is for the network to discriminate its class . As pointed out by Liu et al . ( 2018 ) that the feature embedding of elements lies close to the centroid of their corresponding class are the representatives , while features far away or closer to other classes are the confusing ones which are not discriminative enough . Inspired by this observation , we identify a successful discriminability indicator by measuring one embedding ’ s distance with class centroids and compute the ratio of between positive and hardest-negative , where the positive is its distance with its class ’ s corresponding centroid and the hardest-negative is the closest counterpart . This indicator is defined as the discriminability distillation regulation ( DDR ) . Armed with recent theories on the homogeneity between class centroids and projection weights of classifiers ( Wang et al. , 2017a ; Liu et al. , 2017b ; 2018 ; Deng et al. , 2019a ) , the entire distancemeasuring procedure can be easily accomplished by simply encoding all elements in one group . Thus , the DDR scores can be assessed for each element after the training of the base network . This assessing procedure is highly flexible without human supervision nor re-training the base network , so it can be adapted to any existing base . With our explicitly designed discriminability indicator on the training set , the distillation of such discriminability can be successfully performed with a lightweight discriminability distillation network ( DDNet ) , which shows the superiority of our proposed indicator . We call the whole procedure uniformly as discriminability distillation learning ( DDL ) . The next step is towards finding a better aggregation policy . At the test phase , all elements are firstly sent to the light-weight DDNet . Then element features will be weighted aggregated by their DDR score into group representation . Moreover , in order to achieve the trade-off between accuracy and efficiency , we can filter elements by DDR score and only extract element features of high score . Since the base model tends to be heavy , the filter can save much computation consumption . We evaluate the effectiveness of our proposed DDL on several classical yet challenging tasks . Comprehensive experiments show the advantage of our method on both recognition accuracy and computation efficiency . We achieve state-of-the-art results without modifying the base networks . We highlight our contributions as follows : ( 1 ) We define the discriminability of one element within a group from a more essential and explicable view , and propose an efficient indicator . ( 2 ) We verify that a light-weight network has the capacity of distilling discriminability from the assessed elements . Combining the post-processing with the network , the great computation burden can be saved comparing with existing methods . ( 3 ) We validate the effectiveness of DDL for both efficiency and accuracy on set-to-set face recognition and action recognition through extensive studies . State-ofthe-art results can be achieved . 2 RELATED WORK . 2.1 SET-TO-SET RECOGNITION . Set-to-set recognition which utilizes a group of data of the same class , has been proved efficient on various tasks and drawn much attention these years since the more videos and group datasets are available . Compared with recognition with a single image , set-to-set recognition can further explore the complementary information among set elements and benefit from it . Particularly in this paper , we care for the basic task of face and action recognition . Face Recognition . To tackle set-to-set face recognition problem . ( Wolf et al. , 2011 ; Kalka et al. , 2018 ; Beveridge et al. , 2013 ; Klare et al. , 2015 ) , traditional methods directly estimate the feature similarity among sets of feature vectors ( Arandjelovic et al. , 2005 ; Harandi et al. , 2011 ; Cevikalp & Triggs , 2010 ) . Other works seek to aggregate element features by simply applying max pooling ( Chowdhury et al. , 2016 ) or average pooling ( Li et al. , 2014 ; Taigman et al. , 2014 ) among set features to form a compact representation . However , since most set images are under unconstrained scenes , huge variations on blur , resolution , and occlusion appear , which will degrade the set feature discrimination . How to design a proper aggregation method for set face representation has been the key problem for this approach . Recently , a few methods explore the quality or attention mechanism to form set representation . GhostVLAD ( Zhong et al. , 2018 ) improves traditional VLAD and down weight low quality element features . While Rao et al . ( 2017a ) combine LSTM and reinforcement learning to discard low quality element features . Liu et al . ( 2017c ) and Yang et al . ( 2017 ) introduce attention mechanisms to assign quality scores for different elements and aggregate feature vectors by quality weighted sum . To predict the quality score , an online attention network module is added and co-optimized by the target set-to-set recognition task . However , the definition of generated ’ quality ’ score remains unclear and the learning procedures are learned through a black box , which lacks explainability . In our work , we claim that the most significant indicator to show whether the group representation can be benefited from an element is not the quality or an inexplicable score , but the discriminability . And a novel discriminability distillation learning procedure is proposed . Action recognition . With the advance of multimedia era , millions of hours of videos are uploaded to video platforms every day , so video understanding task like action recognition has become a popular research topic . Real-world videos contain variable frames , so it is not practical to put the whole video to a memory limited GPU . The most usual approach for video understanding is to sample frames or clips and design late fusion strategy to form video-level prediction . Frame-based methods ( Yue-Hei Ng et al. , 2015 ; Simonyan & Zisserman , 2014 ; Girdhar et al. , 2017 ) firstly extract frame features and aggregate them . Simonyan & Zisserman ( 2014 ) propose the twostream network to simultaneously capture the appearance and motion information . Wang et al . ( 2017b ) add attention module and learn to discard unrelated frames . Frames-based methods are computational efficient , but only aggregate high-level frame feature tends to limit the model ability to handle complex motion and temporal relation . Clip-based method ( Tran et al. , 2015 ; 2018 ; Feichtenhofer et al. , 2018 ) use 3D convolutional neural network to jointly capture spatio-temporal features , which perform better on action recognition . However , clip-based methods highly rely on the dense sample strategy , which introduces huge computational consumption and makes it unpractical to application . In this article , we show that by combing our DDL , the clip-based methods can achieve both excellent performance and computation efficiency . 3 DISCRIMINABILITY DISTILLATION LEARNING . In this section , we first formulate the problem of group representation learning in section 3.1 and then define the discriminability distillation regulation ( DDR ) in section 3.2 . Next , we introduce the whole discriminability distillation learning ( DDL ) procedure in section 3.3 . In sections 3.4 and 3.5 , we discuss the aggregation method and the advantage of our DDL , respectively . 3.1 FORMULATION OF GROUP REPRESENTATION LEARNING . Group representation learning focuses on formulating a uniform representation for a whole set of elements . The core of either verification task or classification task is how to aggregate features of a given element group . Define fi as the embedded feature of element Ii in a group IS , then the uniform feature representation of the whole group is FIS = G ( f1 , f2 , · · · , fi ) , ( 1 ) where G indicates the feature aggregation module . While previous research has revealed that conducting G with quality scores ( Liu et al. , 2017c ) has priority over simple aggregation , this kind of methods is not explainable . In this article , we propose a discriminability distillation learning ( DDL ) process to generate the discriminability of feature representation . 3.2 DISCRIMINABILITY DISTILLATION REGULATION . Towards learning efficient and accurate G , we design the discriminability distillation regulation ( DDR ) to generate the discriminability to replace the traditional quality score . In DDR , we jointly consider the feature space distribution and explicitly distill the discriminability by encoding the intra-class distance and inter-class distance with class centroids . Let X denote the training set with the identities C and Wj , j ∈ [ 1 , C ] is the class centroid . For feature fi , i ∈ [ 1 , s ] with class c where Under review as a conference paper at ICLR 2020 Enhanced quality aware network s denotes the size of X , the intra-class distance and inter-class distance are formulated as Cic = fi ·Wc ‖fi‖2 ‖Wc‖2 , Cij = fi ·Wj ‖fi‖2 ‖Wj‖2 , j 6= c. ( 2 ) The intra-class distance Cic and inter-class distance Cij are shown in Figure 2 . After training the base model on classification task , features of the elements from the same class are projected to hyperspace tightly in order to form an explicit decision boundary . Furthermore , elements close to the centroid of their corresponding class are the representative ones , while elements far away from their corresponding class or closer to other classes are not discriminative enough . Based on this observation , we define the discriminability Qi of fi as : Qi = Cic max { Cij | j ∈ [ 1 , C ] , j 6= c } , ( 3 ) i.e. , the ratio of feature distance between the centroids of its class and the hardest-negative class . Considering the variant number of elements in different groups , we further normalize the discriminability by : Di = τ ( Qi − µ ( { Qj | j ∈ [ 1 , s ] } ) σ ( { Qj | j ∈ [ 1 , s ] } ) ) ( 4 ) where τ ( · ) , µ ( · ) and σ ( · ) denote the sigmoid function , the mean value and the standard deviation value of { Qj | j ∈ [ 1 , s ] } , respectively . We denote Di as the discriminability distillation regulation ( DDR ) score . Cooperated with the feature space distribution , the DDR score Di is more interpretable and reasonable than the quality score in traditional quality learning . It can discriminate features better by explicitly encoding the intra- and inter-class distances with class centroids .

Summary: This paper studies how to aggregate features from group inputs. The paper proposes  Discriminability Distillation Learning (DDL) to compute the aggregation coefficients. The method assumes that each sample has a discriminability property that is directly related to the task. The authors define this property and propose to learn such property by an auxiliary network. Such a network can be used in many models without affecting their original training procedure and is able to improve the performances on many tasks, including set-to-set face recognition and action recognition. The experimental results are comprehensive and convincing.