Title: DISCRIMINABILITY DISTILLATION IN GROUP REPRESENTATION LEARNING

Abstract: Learning group representation is a commonly concerned issue in tasks where the basic unit is a group, set or sequence. Previously, the computer vision community tries to tackle it by aggregating the elements in a group based on an indicator either defined by human such as the quality or saliency of an element, or generated by a black box such as the attention score or output of a RNN. This article provides a more essential and explicable view. We claim the most significant indicator to show whether the group representation can be benefited from an element is not the quality, or an inexplicable score, but the discrimiability. Our key insight is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t. the element space can be distilled by a light-weight auxiliary distillation network (DDNet). This procedure is denoted as discriminability distillation learning (DDL). We show the proposed DDL can be flexibly plugged into many group based recognition tasks without influencing the training procedure of the original tasks. Comprehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efficiency, and it pushes forward the state-of-the-art results on these tasks by an impressive margin.

1. INTRODUCTION:

With the rapid development of deep learning and the easy access to large-scale group data, recognition tasks using group information have drawn great attention in the computer vision community. The rich information provided by different elements can complement each other to boost the performance of tasks such as face recognition, action recognition, and person re-identification (Wang et al., 2017b; Zhong et al., 2018; Girdhar et al., 2017; Simonyan & Zisserman, 2014; Yang et al., 2017; Liu et al., 2019a; Rao et al., 2017b) . While traditional practice for group-based recognition is to either aggregate the whole set by average (Li et al., 2014; Taigman et al., 2014) or max pooling (Chowdhury et al., 2016) , or just sampling randomly (Wang et al., 2016) , the fact that certain elements contribute negatively in recognition tasks has been ignored. Thus, an important issue is to select representatives from sets for efficient group understanding.

To tackle such cases, previous methods aim at defining the "quality" or "saliency" for each element in a group (Liu et al., 2017c; Yang et al., 2017; Rao et al., 2017b; Nikitin et al., 2017) . The weights for each element can be automatically learned by self-attention. For example, Liu et al. (2017c) proposes the Quality Aware Network (QAN) to learn quality score for each image inside an image set during network training. Other works adopt the same idea and extend to specific tasks such as video-based person re-identification (Li et al., 2018; Wu et al., 2018) and action recognition (Wang et al., 2018c) by learning spatial-temporal attentions. However, the whole online quality or attention learning procedures are either manually designed or learned through a black box, which lacks explainability.

In this work, we explore deeper into the underlying mechanism for defining effective elements instead of relying on self-learned attention. Assuming that a base network has already been trained for element-based recognition using class labels, we define the "discriminability" of one sample by how difficult it is for the network to discriminate its class. As pointed out by Liu et al. (2018) that the feature embedding of elements lies close to the centroid of their corresponding class are the representatives, while features far away or closer to other classes are the confusing ones which are not discriminative enough. Inspired by this observation, we identify a successful discriminability indicator by measuring one embedding's distance with class centroids and compute the ratio of between positive and hardest-negative, where the positive is its distance with its class's corresponding centroid and the hardest-negative is the closest counterpart. This indicator is defined as the discriminability distillation regulation (DDR).

Armed with recent theories on the homogeneity between class centroids and projection weights of classifiers (Wang et al., 2017a; Liu et al., 2017b; 2018; Deng et al., 2019a) , the entire distancemeasuring procedure can be easily accomplished by simply encoding all elements in one group. Thus, the DDR scores can be assessed for each element after the training of the base network. This assessing procedure is highly flexible without human supervision nor re-training the base network, so it can be adapted to any existing base. With our explicitly designed discriminability indicator on the training set, the distillation of such discriminability can be successfully performed with a lightweight discriminability distillation network (DDNet), which shows the superiority of our proposed indicator. We call the whole procedure uniformly as discriminability distillation learning (DDL). The next step is towards finding a better aggregation policy. At the test phase, all elements are firstly sent to the light-weight DDNet. Then element features will be weighted aggregated by their DDR score into group representation. Moreover, in order to achieve the trade-off between accuracy and efficiency, we can filter elements by DDR score and only extract element features of high score. Since the base model tends to be heavy, the filter can save much computation consumption. We evaluate the effectiveness of our proposed DDL on several classical yet challenging tasks. Comprehensive experiments show the advantage of our method on both recognition accuracy and computation efficiency. We achieve state-of-the-art results without modifying the base networks.

We highlight our contributions as follows: (1) We define the discriminability of one element within a group from a more essential and explicable view, and propose an efficient indicator. (2) We verify that a light-weight network has the capacity of distilling discriminability from the assessed elements. Combining the post-processing with the network, the great computation burden can be saved comparing with existing methods. (3) We validate the effectiveness of DDL for both efficiency and accuracy on set-to-set face recognition and action recognition through extensive studies. State-ofthe-art results can be achieved.

2. RELATED WORK:



2.1. SET-TO-SET RECOGNITION:

Set-to-set recognition which utilizes a group of data of the same class, has been proved efficient on various tasks and drawn much attention these years since the more videos and group datasets are available. Compared with recognition with a single image, set-to-set recognition can further explore the complementary information among set elements and benefit from it. Particularly in this paper, we care for the basic task of face and action recognition.

Face Recognition. To tackle set-to-set face recognition problem. (Wolf et al., 2011; Kalka et al., 2018; Beveridge et al., 2013; Klare et al., 2015) , traditional methods directly estimate the feature similarity among sets of feature vectors (Arandjelovic et al., 2005; Harandi et al., 2011; Cevikalp & Triggs, 2010) . Other works seek to aggregate element features by simply applying max pooling (Chowdhury et al., 2016) or average pooling (Li et al., 2014; Taigman et al., 2014) among set features to form a compact representation. However, since most set images are under unconstrained scenes, huge variations on blur, resolution, and occlusion appear, which will degrade the set feature discrimination. How to design a proper aggregation method for set face representation has been the key problem for this approach.

Recently, a few methods explore the quality or attention mechanism to form set representation. GhostVLAD (Zhong et al., 2018) improves traditional VLAD and down weight low quality element features. While Rao et al. (2017a) combine LSTM and reinforcement learning to discard low quality element features. Liu et al. (2017c) and Yang et al. (2017) introduce attention mechanisms to assign quality scores for different elements and aggregate feature vectors by quality weighted sum. To predict the quality score, an online attention network module is added and co-optimized by the target set-to-set recognition task. However, the definition of generated 'quality' score remains unclear and the learning procedures are learned through a black box, which lacks explainability. In our work, we claim that the most significant indicator to show whether the group representation can be benefited from an element is not the quality or an inexplicable score, but the discriminability. And a novel discriminability distillation learning procedure is proposed.

Action recognition. With the advance of multimedia era, millions of hours of videos are uploaded to video platforms every day, so video understanding task like action recognition has become a popular research topic. Real-world videos contain variable frames, so it is not practical to put the whole video to a memory limited GPU. The most usual approach for video understanding is to sample frames or clips and design late fusion strategy to form video-level prediction.

Frame-based methods (Yue-Hei Ng et al., 2015; Simonyan & Zisserman, 2014; Girdhar et al., 2017) firstly extract frame features and aggregate them. Simonyan & Zisserman (2014) propose the twostream network to simultaneously capture the appearance and motion information. Wang et al. (2017b) add attention module and learn to discard unrelated frames. Frames-based methods are computational efficient, but only aggregate high-level frame feature tends to limit the model ability to handle complex motion and temporal relation.

Clip-based method (Tran et al., 2015; 2018; Feichtenhofer et al., 2018) use 3D convolutional neural network to jointly capture spatio-temporal features, which perform better on action recognition. However, clip-based methods highly rely on the dense sample strategy, which introduces huge computational consumption and makes it unpractical to application. In this article, we show that by combing our DDL, the clip-based methods can achieve both excellent performance and computation efficiency.

3. DISCRIMINABILITY DISTILLATION LEARNING:

In this section, we first formulate the problem of group representation learning in section 3.1 and then define the discriminability distillation regulation (DDR) in section 3.2. Next, we introduce the whole discriminability distillation learning (DDL) procedure in section 3.3. In sections 3.4 and 3.5, we discuss the aggregation method and the advantage of our DDL, respectively.

3.1. FORMULATION OF GROUP REPRESENTATION LEARNING:

Group representation learning focuses on formulating a uniform representation for a whole set of elements. The core of either verification task or classification task is how to aggregate features of a given element group.

Define f i as the embedded feature of element I i in a group I S , then the uniform feature representation of the whole group is

EQUATION

where G indicates the feature aggregation module. While previous research has revealed that conducting G with quality scores (Liu et al., 2017c) has priority over simple aggregation, this kind of methods is not explainable. In this article, we propose a discriminability distillation learning (DDL) process to generate the discriminability of feature representation. s denotes the size of X , the intra-class distance and inter-class distance are formulated as

C ic = f i • W c f i 2 W c 2 , C ij = f i • W j f i 2 W j 2 , j = c.

(2)

The intra-class distance C ic and inter-class distance C ij are shown in Figure 2 . After training the base model on classification task, features of the elements from the same class are projected to hyperspace tightly in order to form an explicit decision boundary. Furthermore, elements close to the centroid of their corresponding class are the representative ones, while elements far away from their corresponding class or closer to other classes are not discriminative enough. Based on this observation, we define the discriminability Q i of f i as:

EQUATION

i.e., the ratio of feature distance between the centroids of its class and the hardest-negative class.

Considering the variant number of elements in different groups, we further normalize the discriminability by:

EQUATION

where τ (•), µ(•) and σ(•) denote the sigmoid function, the mean value and the standard deviation value of {Q j | j ∈ [1, s]}, respectively. We denote D i as the discriminability distillation regulation (DDR) score.

Cooperated with the feature space distribution, the DDR score D i is more interpretable and reasonable than the quality score in traditional quality learning. It can discriminate features better by explicitly encoding the intra-and inter-class distances with class centroids.

From section 3.2, given a training dataset θ and a corresponding model, the DDR score D i of f i can be naturally computed by Eq (2)-( 4). However, in group representation learning, D i is unavailable to test set T without W j . In order to better embed D i to the group representation learning procedure, we introduce the discriminability distillation learning (DDL) to generate an approximated DDR score Di for D i . Given an arbitrary CNN architecture N and an image I i , Di can be approximated by where θ is the parameter of the light-weight DDNet N . At the training stage, we apply mean squared error between Di and target D i as

EQUATION

L = 1 2N N i ( Di -D i ) 2 (6)

where N is the batch size.

3.4. FEATURE AGGREGATION G:

At the test stage, we can generate Di via Eq (5) for each test element I i in the given element set I S .

The feature aggregation G in Eq (1) can be formulated as

EQUATION

)

where n is the element number of I S , and Ri is the re-scaled DDR score of Di via

EQUATION

In Eq (8), we scale the DDR score of element set I S between 0 and 1 to ensure a same range for element sets with different lengths. K and B are formulated as

EQUATION

EQUATION

)

3.5. ADVANTAGE OF DISCRIMINABILITY DISTILLATION LEARNING:

Different from the subjective quality judgment of an image or quality learning via attention mechanism, we explicitly assign discriminability for each element via the feature space distribution. By jointly considering the inter-and intra-class distances with class centroids, DDL can effectively approximate how discriminative a feature is. According to aggregating more information from features with high discriminability in an element set, the recognition performance can be significantly boosted. With the assistant of DDL, both hard samples (the model is easy to fail on them) or lowquality images (the model is easy to give wrong predict results) can be weakened. Moreover, the well design discriminability distillation learning process needn't modify the base model, making it easy to be plugged into many popular recognition frameworks.

Enhanced quality aware network YTF: The YouTube Face dataset includes 3425 videos of 1595 identities with an average of 2.15 videos per identity. The videos vary from 48 frames to 6,070 frames. We report the 1:1 face verification rate of the given 5,000 video pairs in our experiments without fine-tuning. All set images will be sent to the DDN at the inference stage, so the DDN is sensitive to computational burden. To make the inference stage efficient, the DDN is designed to be light-weighted. Since the results on YouTube Face benchmark tend to be saturated, we test our DDL on the challenging video face verification dataset IQIYI-VID-FACE, which is of large scale and more close to industrial application. Compared with the average pooling, DDL improves performance by 3.21%. Even aggregating the top 1 DDR score features only can still achieve the performance of average aggregation for all frames. It shows that our DDL has selected the most discriminative element of the set. By combining PolyNet base model, our DDL achieves the state-of-the-arts performance on the IQIYI-VID-FACE dataset.

What's more, the good performance of only aggregating top 1 discriminability frame shows that we needn't extract all frames by a heavy base model, but only the most discriminative one during inference. Since the computational consumption for DDNet is efficient, usually only one-tenth of other heavy models. It can be seen that DDL makes a huge difference for industrial video face recognition application.

EVALUATION ON IJB-A AND IJB-C:

Tables 3 and 4 show the results on the IJB-A and IJB-C dataset for different methods. From the two tables, we can see that our DDL improves verification performance by a convincing margin with average pooling for both two benchmarks, especially under severe FAR at 0.001% by 1.94% and FAR at 0.0001% by 6.88% on IJB-C. Compared with IJB-A, IJB-C has more images and covers more variations among images, such as pose, blur, resolution, and conditions. So the performance gain with DDL is huger.

Compared with the state-of-the-art methods, our DDL improves IJB-A by 4.83% when FAR =0.001 and IJB-C by 4.89% when FAR = 0.0001%. 2018). For testing, they firstly split the whole videos into clips and inference them individually. Then clip-level results will be average pooled to get the whole video's prediction. All three methods hugely rely on the dense sampling strategy during testing, which is quite computational consumed.

For our DDL, we firstly filter clips by the light-weight DDNet. Then we only extract and aggregate features from the top-K discriminability clips. The K value selected for Activitiynet-1.2 and Kinetics-700 is 9 and 5 respectively, for that they achieve the best efficiency and accuracy trade-off in our experiments. For comparison, we randomly or uniformly sample K clips in videos. Dense sampling is also used to compare.

EVALUATION ON ACTIVITYNET-1.2

As shown in Table 5 , DDL improves recognition performance for all baseline models. For the state-of-the-art clips-based model SlowFast, combining it with DDL can achieve around 4% accuracy gain compared with randomly or uniformly sample. What's more, DDL can even outperform dense evaluation along with the whole video by a huge margin. It is reasonable as DDL can select and aggregate clips with the highest discriminability, which avoids video clips with visual blur or ambiguous content to pollute the video-level prediction.

What's more, combining with DDL, the inference process can be hugely accelerated. For testing a two minute videos with SlowFast-50 model as example, original dense sampling strategy introduces 17 Glops × 60 computation , but for DDL enhanced, only top discriminability videos need to be predicted by the heavy base model. The total consumption is 1.7 Glops× 60 for assessing the discriminability plus 9×17 Gflops for extracting the top discriminability clip feature, which achieves 4x speed up.

EVALUATION ON KINETICS700. Table 6 : Video action recognition results on Kinetics-700 dataset. We compare randomly, uniformly and filter by DDL to select 5 clips to aggregate. Accuracy is reported on the validation set and is the average of top1 and top 5 accuracy. Dense sample strategy sample 10 clips along the temporal axis and random crop 3 clips on the spatial axis.

As shown in:

CROSS DATASET AND MODEL EVALUATION.

In the last two sections, we train specialized DDNet for different datasets and base models, respectively. To demonstrate the generalization ability of our DDL, we conduct the cross dataset and cross model evolution experiment. Action recognition model R(2+1)D for ActivityNet-1.2 dataset is combined with DDL trained with Kinetics-700 and base model SlowFast, the results are shown in 

5. CONCLUSION:

In this paper, we have proposed a novel post-processing module called DDL for all group-based recognition tasks. We explicitly define the discriminability with observations on feature embedding, then apply a light-weight network for discriminability distillation and feature aggregation. We identify the advantage of our proposed methods in the following aspects: (1) The entire discriminability distillation is performed without modifying the pre-trained base network, which is highly flexible comparing with existing quality-aware or attention methods. (2) Our distillation network is extremely light-weighted with great computational cost salvage. In Eq(4), we normalize the discriminability by the mean and standard deviation to deal with a variant number of elements in different groups. In our experiments, if the normalization step is removed, the DDNet can't converge.

As for linear map the minimum and maximum DDR score of the element set to 0 and 1 in Eq(8)(9)(10), this step leading to 0.4% performance gain for YTF and 1.6% performance gain for IQIYI-VID-FACE. But the scale process doesn't contribute to template-based set-to-set recognition like IJB-A and IJB-C dataset. The reason is that in a video, frames are captured in the almost same condition, so the variation of frames tends to be limited. The mapping process will separate the score distribution, making it more focuses on frames with higher discriminability. While for template-based set-to-set recognition like IJB-A and IJB-C, the variation of elements is large and the discriminability score distribution has been separated well, so there is no obvious gain when applying score re-scale. More aggregation strategy scripts can found in our GitHub.

IMPLEMENTATION DETAILS FOR ACTION RECONGNTION:

To train on ActivityNet-1.2 dataset, we pre-trained our model on Kinetics-700 datasets. Video clips with adjacent 64 frames are randomly sampled from raw videos for training and testing. For the spatial domain, we randomly crop 112×112 pixels with a shorted side randomly sampled in [128, 160] 

:



Figures and Tables:

Figure 1: The pipeline of group representation learning with our DDL. Given a base feature extract model, we first compute the DDR score for each training element and then train a light-weight discriminability distillation network (DDNet) to distill it. During testing, the DDNet will predict discriminability for each element in the test set.

Figure 2: The formulation of discriminability distillation regulation. It is the ratio of element feature distance with its class centroids and the hardest-negative class's. After training the base model with classification task, element features from the same class are projected to hyperspace tightly in order to form a clear decision boundary. While the outliers who lie far away from its centroids or closer to other classes are of low discriminability.

Figure 3: The pipeline of the test stage of DDL. For a set of elements, we first predict DDR score by the trained-well DDNet for each element. Then the feature extracted by the base model will be weighted aggregation to form the group representation. To achieve accuracy and computation consumption trade-off, a top-K filter can also be applied to select the most discriminative elements and only extract features and aggregate with them.

Figure 4: The visualization results of discriminability for images of Template ID 17762 and 16800 from IJB-C dataset.

(3) With our DDL and feature aggregation, we achieve state-of-the-art results on set-to-set face recognition and action recognition tasks. A APPENDIX IMPLEMENTATION DETAILS FOR SET-TO-SET FACE RECOGNITION Pre-processing: RetinaFace Deng et al. (2019b) is used to detect faces and their corresponding landmarks for all datasets. Images are aligned to 112 × 112 by similarity transformation based on their landmarks. Training: We train our base recognition model and DDL on MS-Celeb-1M dataset Guo et al. (2016). Since the original dataset has been proved to be dirty, we use the cleaned version by Deng et al. (2019a), which consists of 5,179,510 pictures from 93,431 identities. For the base model training, we select the modified version of ResNet-101 He et al. (2016) introduced by Deng et al. (2019a) and PolyNet Zhang et al. (2017b) as the backbone network.The loss function is ArcFaceDeng et al. (2019a

Figure 5: Visualization discrimiability of frames from different videos.

Figure 6: Visualization discrimiability of frames from one video.

The IARPA Janus Benchmark C (IJB-C) is an extended version of IJB-A. It has 140, 740 faces images of 3, 531 subjects, so we can test under more strict False Accept Rates (FAR). Since the images in IJB-C dataset have large variations, it is regarded as a challenging set-to-set face recognition benchmark.

Table content: <table/>

The typical DDN is a channel reduced ResNet-18 network, which only introduces 81.9 Mflops computation for 112×112 input. Compared with base feature extraction models which often reach 24 Gflops (ResNet-101), it is super-efficient. More discussions on the DDN architecture can be found in the Appendix A. Video face verification performance on YouTube Face dataset, compared with state-of-the art methods and baseline methods.

Table content: <table><tr><td colspan="2">Method</td><td>Accuracy(%)</td><td>Method</td><td>Accuracy(%)</td></tr><tr><td colspan="2">Li et al. (2014)</td><td>84.8</td><td colspan="2">Taigman et al. (2014)</td><td>91.4</td></tr><tr><td colspan="2">Schroff et al. (2015)</td><td>95.52</td><td colspan="2">Yang et al. (2017)</td><td>95.72</td></tr><tr><td colspan="2">Sun et al. (2015)</td><td>93.20</td><td colspan="2">Liu et al. (2017c)</td><td>96.17</td></tr><tr><td colspan="2">Gong et al. (2019)</td><td>96.50</td><td colspan="2">Rao et al. (2017b)</td><td>96.52</td></tr><tr><td colspan="2">Liu et al. (2019b)</td><td>96.21</td><td colspan="2">Rao et al. (2017a)</td><td>94.28</td></tr><tr><td colspan="2">Wang et al. (2018b)</td><td>97.6</td><td colspan="2">Deng et al. (2019a)</td><td>98.02</td></tr><tr><td colspan="2">Average</td><td>98.27</td><td>Top 1</td><td>97.11</td></tr><tr><td/><td/><td/><td>DDL</td><td>99.19</td></tr><tr><td>Method</td><td colspan="2">TPR@FPR=1e-4(%)</td><td>Method</td><td>TPR@FPR=1e-4(%)</td></tr><tr><td>MSRA</td><td/><td>71.59</td><td>Alibaba-VAG</td><td>71.10</td></tr><tr><td>Insightface</td><td/><td>67.00</td><td>DDL (PolyNet)</td><td>72.98</td></tr><tr><td>Average</td><td/><td>65.84</td><td>Top 1</td><td>65.22</td></tr><tr><td/><td/><td/><td>DDL</td><td>69.05</td></tr></table>

Comparsion with different participants and aggreation stragey on the IQIYI-VID-FACE challenge. By combing with PolyNet, DDL achieves state-of-the-arts peformance.

Table content: <table/>

Peformance comparsions on IJB-A verificatiton benchmark. The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported.

Table content: <table><tr><td>Method</td><td colspan="2">FAR=0.001</td><td colspan="4">IJB-A (TAR@FAR) FAR=0.01</td><td>FAR=0.1</td></tr><tr><td>Crosswhite et al. (2018)</td><td colspan="2">83.6 ± 2.7</td><td/><td>93.9 ± 1.3</td><td/><td>97.9 ± 0.4</td></tr><tr><td colspan="3">Sankaranarayanan et al. (2016) 81.30 ± 2.0</td><td/><td>91.0 ± 1.0</td><td/><td>96.4 ± 0.5</td></tr><tr><td>Xie &amp; Zisserman (2018)</td><td colspan="2">92.0 ± 1.3</td><td/><td>96.2 ± 0.5</td><td/><td>98.9 ± 0.2</td></tr><tr><td>Liu et al. (2017c)</td><td colspan="6">89.31 ± 3.92 94.2 ± 1.53 98.02 ± 0.55</td></tr><tr><td>Cao et al. (2018)</td><td colspan="2">92.1 ± 1.4</td><td/><td>96.8 ± 0.6</td><td/><td>99 ± 0.2</td></tr><tr><td>Yang et al. (2017)</td><td colspan="2">88.1 ± 1.1</td><td/><td>94.1 ± 0.8</td><td/><td>97.8 ± 0.3</td></tr><tr><td>Liu et al. (2017c)</td><td colspan="6">89.31 ± 3.92 94.2 ± 1.53 98.02 ± 0.55</td></tr><tr><td>Zhong et al. (2018)</td><td colspan="2">93.5 ± 1.5</td><td/><td>97.2 ± 0.3</td><td/><td>99.0 ± 0.2</td></tr><tr><td>Liu et al. (2019b)</td><td colspan="6">93.61 ± 1.51 97.28 ± 0.28 98.94 ± 0.31</td></tr><tr><td>Deng et al. (2019a)</td><td colspan="2">97.89 ± 1.5</td><td/><td>98.51 ± 0.3</td><td/><td>99.05 ± 0.2</td></tr><tr><td>Average</td><td colspan="2">97.71 ± 0.6</td><td/><td>98.43± 0.4</td><td/><td>99.01± 0.2</td></tr><tr><td>DDL</td><td colspan="2">98.44 ± 0.3</td><td/><td>98.79 ± 0.2</td><td/><td>99.13 ± 0.1</td></tr><tr><td>Method</td><td colspan="5">IJB-C (TAR@FAR) 0.0001% 0.001% 0.01%</td><td>1%</td><td>1%</td></tr><tr><td>Yin et al. (2019)</td><td>-</td><td>-</td><td/><td>-</td><td/><td>0.1</td><td>83.8</td></tr><tr><td>Xie et al. (2018)</td><td>-</td><td>-</td><td/><td>88.5</td><td colspan="2">94.7</td><td>98.3</td></tr><tr><td>Zhao et al. (2019)</td><td>-</td><td colspan="2">82.6</td><td>89.5</td><td colspan="2">93.5</td><td>96.2</td></tr><tr><td>Xie &amp; Zisserman (2018)</td><td>-</td><td colspan="2">77.1</td><td>86.2</td><td colspan="2">92.7</td><td>96.8</td></tr><tr><td>Cao et al. (2018)</td><td>-</td><td colspan="2">74.7</td><td>84.0</td><td colspan="2">91.0</td><td>96.0</td></tr><tr><td>Shi &amp; Jain (2019)</td><td>-</td><td colspan="2">89.64</td><td colspan="3">93.25 95.49 97.17</td></tr><tr><td>Deng et al. (2019a)</td><td>86.25</td><td colspan="2">93.15</td><td colspan="3">95.65 97.20 98.18</td></tr><tr><td>Average</td><td>84.06</td><td colspan="2">93.81</td><td colspan="3">95.73 96.98 97.87</td></tr><tr><td>DDL</td><td>91.14</td><td colspan="2">95.75</td><td colspan="3">96.94 97.72 98.36</td></tr><tr><td colspan="7">Table 4: Peformance comparsions on IJB-C verificatiton benchmark. The True Accept Rates (TAR)</td></tr><tr><td colspan="2">vs. False Postive Rate (FAR) are reported.</td><td/><td/><td/><td/></tr><tr><td colspan="7">criminability generated with the base model on the same training set. So the fine-tune process is not</td></tr><tr><td>needed.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="7">To qualitatively evaluate the discriminability pattern learned by our DDL, we visualize the discrim-</td></tr><tr><td colspan="7">inability score distribution for two template images in IJB-C datasets. As shown in Figure 4, DDL</td></tr><tr><td colspan="7">can effectively identify image discriminability. Pictures with large poses, visual blur, occlusion, and</td></tr><tr><td colspan="7">incomplete content are regarded to be low discriminative. The efficient discriminability judgment</td></tr><tr><td colspan="7">ability for our DDL leads to an extraordinary performance on set-to-set face recognition problem.</td></tr><tr><td colspan="5">More visualization examples can be found in the Appendix A.1.</td><td/></tr><tr><td>4.2 VIDEO ACTION RECOGNITION</td><td/><td/><td/><td/><td/></tr><tr><td>DATASETS</td><td/><td/><td/><td/><td/></tr><tr><td colspan="7">ActivityNet-1.2. The ActivityNet-1.2 dataset Caba Heilbron et al. (2015) contains 4,819 training</td></tr></table>



Table content: <table><tr><td>Model</td><td>DDL (9 clips) Acc(%) GFLOPs×clips</td><td>Random ( 9 clips) Acc(%) GFLOPs×clips</td><td>Uniform (9 clips) Acc(%) GFLOPs×clips</td><td>Dense ( all clips) Acc(%) GFLOPs×clips</td></tr><tr><td>3D-RS-50</td><td>86.38 37×9+1.7×60</td><td>82.83 37×9</td><td>83.14 37×9</td><td>83.92 37×60</td></tr><tr><td>R(2+1)D-RS-50</td><td>89.08 39×9+1.7×60</td><td>84.51 39×9</td><td>84.89 39×9</td><td>85.46 39×60</td></tr><tr><td>SlowFast-RS-50</td><td>90.21 16×9+1.7×60</td><td>85.92 16×9</td><td>86.14 16×9</td><td>87.72 16×60</td></tr></table>

Video action recognition results on ActivityNet-1.2 dataset. We compare randomly, uniformly and filter by DDL to select 9 clips to aggregate. Accuracy is reported on the valiation set. And the compute consumption is estimated on a 2 minutes videos (above 60 clips), for that 2 minutes is the average video length of ActivityNet-1.2.

Table content: <table><tr><td>Model</td><td>DDL (5 clips) Acc(%) GFLOPs×clips</td><td>Random ( 5 clips) Acc(%) GFLOPs×clips</td><td>Uniform (5 clips) Acc(%) GFLOPs×clips</td><td>Dense ( 30 clips) Acc(%) GFLOPs×clips</td></tr><tr><td>3D-RS-50</td><td>71.01 37×5+1.7×30</td><td>68.26 37×5</td><td>67.43 37×5</td><td>68.83 37×30</td></tr><tr><td>R(2+1)D-RS-50</td><td>72.51 35×5+1.7×30</td><td>69.24 35×5</td><td>68.79 35×5</td><td>70.94 35×30</td></tr><tr><td>SlowFast-RS-50</td><td>74.23 16×5+1.7×30</td><td>72.39 16×5</td><td>72.05 16×5</td><td>73.37 16×30</td></tr></table>

It can be found that through training with the different dataset and base model causes a performance drop, DDL can still improve about 1% accuracy for dense sampling. These results show that the discriminability pattern learned by DDL is robust and well to generalize to other datasets. What's more, the generalization ability can avoid training the DDL model for each action model and each dataset, making the DDL more efficient and practical.

Table content: <table><tr><td>Base Model, Datatset</td><td>Accuracy(%)</td><td>Base Model, Datatset</td><td>Accuracy(%)</td></tr><tr><td>SlowFast-RS-50, ActivityNet-1.2</td><td>87.91</td><td>R(2+1)D, ActivityNet-1.2</td><td>89.08</td></tr><tr><td>SlowFast-RS-50, Kinetics-700</td><td>86.31</td><td>R(2+1)D, Kinetics-700</td><td>86.92</td></tr><tr><td>Uniform</td><td>84.51</td><td>Dense</td><td>85.46</td></tr><tr><td>Random</td><td>84.80</td><td/><td/></tr></table>

Results for cross dataset and cross model experiment. All results are reported with base action recognition R(2+1)D-ResNet-50 on ActivityNet-1.2 dataset. The training procedures for DDL vary from dataset and base model.

Table content: <table/>

). The learning rate starts from 0.1 and is divided by 10 after 60k, 80k iterations, and we finish our training at 11k. The base model is trained on 16 GPUs with a total batch size of 1024 and based on PytorchPaszke et al. (2017) framework. After training the base network, we extract features for each picture from the training set and compute its DDR score as discussed in section 3.2. Then the light-weight DDNet will regress the score by L2 loss with the same training image input. The learning rate for training our DDNet starts from 0.1 and is divided by 10 after 30k, 60k, and 80k iterations, and the total number of iterations is 100k. Linear warm-up with 10k iterations is applied towards stable training. The DDNet is trained on 8 GPUs with a total batch size of 512 are used.DISCUSSION OF DDNET ARCHITECTUREAs has discussed in section 4.1, the design principle of our DDNet is computation efficient, to introduce less computation burden. The typical architecture of DDNet is shown in Table8, which is a channel reduction Resnet-18 network. We also conduct experiments on deeper DDNet which has 34 layers and wider DDNet which is a standard Resnet-18 network. As shown in Table9, neither the deeper or the wider network leads to better performance, which illustrates that the discriminability of face images can be well learned with limited parameters. As for action recognition, we utilize the same DDNet architecture where the 2D convolution kernel is replaced by 3D. The architecture of our light-weight DDNet. Typical DDNet is a channel reduction Resnet-18 network and only introduces 81.9 Mflops computation, which is super efficient.

Table content: <table><tr><td>stage</td><td/><td>DDNet</td><td/><td>output sizes H × W</td></tr><tr><td>raw input</td><td/><td/><td/><td>112×112</td></tr><tr><td>conv 1</td><td/><td colspan="2">3 × 3, 8 stride 1, 1</td><td>112×112</td></tr><tr><td>res 2</td><td/><td colspan="2">1 × 1, 8, stride2, 2 1×1, 8, stride1, 1</td><td>× 2</td><td>56×56</td></tr><tr><td>res 3</td><td colspan="3">1 × 1, 16, stride2, 2 1 × 1, 16, stride1, 1</td><td>× 2</td><td>28×28</td></tr><tr><td>res 4</td><td colspan="3">1 × 1, 32, stride2, 2 1 × 1, 32, stride1, 1</td><td>× 2</td><td>14×14</td></tr><tr><td>res 5</td><td colspan="3">1 × 1, 48, stride2, 2 1 × 1, 48, stride1, 1</td><td>× 2</td><td>7×7</td></tr><tr><td colspan="4">global average pool, fc</td><td>DDR score</td></tr><tr><td>Architecture</td><td/><td colspan="3">IJB-C(TAR@FAR) 0.0001% 0.001% 0.01%</td><td>1%</td><td>1%</td></tr><tr><td>ResNet-18</td><td/><td>90.93</td><td>95.74</td><td>96.92 97.73 98.35</td></tr><tr><td colspan="2">ResNet-34-CD</td><td>91.13</td><td>95.74</td><td>96.90 97.72 98.33</td></tr><tr><td colspan="2">ResNet-18-CD</td><td>91.14</td><td>95.75</td><td>96.94 97.72 98.36</td></tr></table>

Peformance comparsions for different architecture on IJB-C verificatiton benchmark. The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported. CD represents channel reduction as shown in Table8.RESULT OF COMBINING MORE BASELINE WITH DDL ON YOUTUBEFACEThe baseline model we select in the main part is trained with ArcFace loss functionDeng et al.  (2019a). We also re-implement more baseline model with different loss function likeWang et al.  (2018b),Liu et al. (2017a). As shown in Table10, when enhanced by DDL, all baseline achieve performance above 1% performance gain, which indicates the good robustness and generalization ability of our DDL. Tail data: We use one-tenth of raw data with the minimum sample count in each class. It has insufficient data per class and contains 519,379 pics from 29,210 classes. • Head data: We use one-tenth of raw data with the maximum sample count in each class. It has enough data per class and contains 544,381 pics from 4,833 classes.

Table content: <table><tr><td>DISCUSSION ON UNBALANCED DATA CLASS WHEN TRAINING DDL</td></tr><tr><td>Since the data imbalance between classes and such a long-tail distribution has become concerns for</td></tr><tr><td>training recognition models Zhang et al. (2017a); Van Horn &amp; Perona (2017), we conduct experi-</td></tr><tr><td>ments with two settings for MS1M dataset Guo et al. (2016) :</td></tr><tr><td>•</td></tr></table>

Combine DDL with more baselines including SphereFace (SF) and CosFace (CF). All baseline models are trained with same backbone and datasets. Average pooling is used as the default aggregation strategy.The comparison results for the two settings on IJB-C datasets are illustrated in Table11. The performance for tail data and head data are almost the same because the formulation of DDR doesn't explicitly encodes identity information. For the classification task, insufficient data for one class may lead to bad representation for that class in subspace, but for our DDL training, the DDR definition is class-unspecified, which means only the feature's intra-class and inter-class distance matters, but not the identity information.What's more, the slight performance drop for training with one of the tens of amount indicates that by increasing the training sample, the performance of DDL can further improve. But around 500,000 data are enough for training a good DDL model.

Table content: <table><tr><td>Data</td><td colspan="3">IJB-C(TAR@FAR) 0.0001% 0.001% 0.01%</td><td>1%</td><td>1%</td></tr><tr><td>Tail</td><td>89.81</td><td>95.38</td><td colspan="2">96.62 97.48 98.14</td></tr><tr><td>Head</td><td>89.92</td><td>95.41</td><td colspan="2">96.59 97.51 98.13</td></tr><tr><td>Full</td><td>91.14</td><td>95.75</td><td colspan="2">96.94 97.72 98.36</td></tr></table>

Peformance comparsions for discussion of class data imbalance when training DDL. Head and tail data setting are of near same number samples, but varies from class numbers. RESULTS OF TRAINING DDL WITH OTHER DATASETS.In the main article, we train our base model and DDL on the MS1M datasetGuo et al. (2016), which is the largest open face datasets. In this section, we also train our base model and DDL on the IMDB datasetWang et al. (2018a), which contains 59k identities and 1.7M images. The results on IJB-C dataset are in Table12. Training with the smaller dataset, DDL also achieves competitive results and better than average pooling for a huge margin. At FPR=0.01%, compared with average pooling, DDL boosts the performance by 29.64%. This indicates that a weak model can benefit more from DDL.

Table content: <table><tr><td>Method</td><td colspan="3">IJB-C(TAR@FAR) 0.0001% 0.001% 0.01%</td><td>1%</td><td>1%</td></tr><tr><td>average</td><td>0.002</td><td>0.22</td><td colspan="2">39.39 91.55 96.34</td></tr><tr><td>DDL</td><td>0.002</td><td>0.40</td><td colspan="2">66.33 94.83 97.55</td></tr></table>

Peformance for training both base model and DDL on IMDB dataset on IJB-C benchmark. The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported.

Table content: <table><tr><td>Method</td><td colspan="3">IJB-C(TAR@FAR) 0.0001% 0.001% 0.01%</td><td>1%</td><td>1%</td></tr><tr><td>DDL</td><td>91.14</td><td>95.75</td><td colspan="2">96.94 97.72 98.36</td></tr><tr><td>cascade training</td><td>91.53</td><td>95.91</td><td colspan="2">97.01 97.82 98.40</td></tr></table>

Peformance comparsions for cascade training on IJB-C dataset. By refining the class centroid and cascade training, DDL achieves performance gain. DISCUSSION OF NORMALIZATION AND SCORE SCALE FOR DDL

Table content: <table/>

pixels. 32 NVIDIA V100 GPUS with synchronized SGD training and global BN are used to train and we find that result for typical training in one 8 GPU machine is the same. Similar to the 2D-DDL, we generate the pseudo discriminability label for each clip and use a channel reduction 3D resnet-18 network to regress the score after training the base model. The training setting for Kinetics is the same with ActivityNet-1.2, except we sample clips with adjacent 32 frames.A.1 QUALITATIVE ANALYSISIn the main part, we visualize two templates from IJB-C datasets. In this subsection, we visualize more examples from YouTube Face dataset. As we can see in these experiments, our DDL has learned meaningful discriminative patterns.

Table content: <table><tr><td>0.11</td><td>0.13</td><td>0.13</td><td>0.15</td><td>0.17</td><td>0.18</td><td>0.19</td><td>0.19</td></tr><tr><td>0.21</td><td>0.22</td><td>0.24</td><td>0.25</td><td>0.26</td><td>0.29</td><td>0.30</td><td>0.31</td></tr><tr><td>0.33</td><td>0.35</td><td>0.35</td><td>0.37</td><td>0.39</td><td>0.40</td><td>0.42</td><td>0.43</td></tr><tr><td>0.44</td><td>0.47</td><td>0.49</td><td>0.51</td><td>0.53</td><td>0.55</td><td>0.57</td><td>0.59</td></tr><tr><td/><td/><td/><td colspan="2">From confused to discriminative</td><td/><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes a discriminability indicator using embedded class centroids on a proxy set, and show the discriminability distribution w.r.t. the element space distilled by a light-weight auxiliary distillation network, which is called discriminability distillation learning (DDL). This methodology was tested on a selection of datasets addressing set-to-set face recognition and action recognition. I have the following concerns, which made me to suggest rejecting this paper. 1) The methodology explained in Section 3 should be improved. In the way it is presented, it is not clear and reproducible. In detail, 1.1) It is better to describe the pipeline together with Figure 1. 1.2) Contribution of the top part of Figure 1 in proposed pipeline is not clear. More descriptive caption should be added. 1.3) In section 1, where you ask reader to observe the feature embedding of elements which lie close to the centroid… Which section or figure are you referring to? 1.4) In subsection 2.3, clearly discuss how proposed approach is different than traditional distillation methods. Also, explain your modifications and discuss why new structure is better than default ResNet-18. 1.5) Subsections 3.2-3.4 do not contain enough explanation to support equation of proposed method. 1.6) It is not clear why authors use cosine distance, because it is simple? 1.7) The quality scores generated by DDL is very hard to interpret, which to me the one of the biggest problem of the proposed methodology. 2) The related work section should include studies more related to the motivation behind DDL (“….to select represent samples from the whole set efficiently for group understanding.”). For instance, attention models, saliency detection, key-frame detection methods, outlier detection methods, quality aware networks, etc. should be discussed and compared with DDL. Instead, authors more focused on face recognition and action recognition tasks themselves. Therefore, many related works are missing. Especially, Section 2.3., should not be a part of related work but should be integrated either to Section 1 (Introduction) or Section 3. 3) Experimental analysis: 3.1 ) It is not possible to figure out if the proposed method is really performing better than SOA. In other words, the experimental analysis should be extended. One thing authors should consider is including more datasets, especially more challenging datasets i.e., the ones not already saturated. Some examples can be: iLIDS-VID, PRID2011, YouTube Face, IJB-A benchmark. 3.2) Is there any reason to mainly focus on face recognition and action recognition? 3.3) “For face recognition, DDL can easily improve the performance by concentrating the discriminative information and for video action recognition, DDL can further accelerate the pipeline by eliminating the frames with insufficient information.” This sounds like DDL behaves differently depends on the task, but perhaps the conclusion driven by the authors cannot be fully correct given that the number of used datasets are limited to make such a general statement. I suggest authors to re-write this sentence. 3.4) The experimental analysis should include comparisons with methodologies such as quality learning via attention mechanism, etc. to better understand the necessity and effectiveness of proposed DDL. However, only for one dataset such a comparison was performed. It is strange why for other datasets the same comparison was not done. 3.5) Experimental analyses so much focuses on to justifying the performance improvements of DDL is independent to the base model selected (e.g. mainly Table 3, but also Table 4, 5 and 6 include). However, to me this is a supporting experiment and it is more important to show the real necessity of the proposed method by comparing it with SOTA methods. 3.6) I do not find using Youtube Face benchmark suitable as it is already saturated, i.e. many existing models perform around 97%, which does not allow to show whether DDL can significantly contribute to the task or not. From the corresponding results, it looks like DDL does not provide any significant improvement. 3.7) In Table 3, it is hard to observe the contribution of DDL in terms of the performance. The improvements are in the level of 1%. More discussion is needed about that table. 3.8) “The discriminability distillation learning is more practical to untrimmed video action recognition since there are more diversity videos chip with ambiguous content and visual blur problem” This means that DDL should not be used in some tasks, it is not generalizable, task-specific? Please elaborate it more clearly. 3.9) Table 2 is hard to interpret, improve the discussions regarding it. 3.10) Table 1 and table 2 were not referred in the text. What does DATA refer to in Table 1, why they are different?, if they are really different i.e. it is not me who misinterpret, then is not it unfair to compare the results? 4) Other suggestions: 4.1) Support your statements with references and appropriate experimental analysis. 4.2) In Section 4, only YTF dataset is mentioned but authors also use IQIYI-VID-FACE challenge dataset. 4.4) Section 4.2. is not clear at all, and should be re-written. 4.5) Some references to the tables are wrong or some are missing, pls. check them all. E.g., in section 4.2., paragraph two, table 3 should be table 2. 4.6) There is no reference to the any figures in the text. 4.7) Paper contains several typos and grammatical mistakes. Also, most of the points were not explained well.