Discriminability Distillation in Group Representation Learning
Authors: Manyuan Zhang，Guanglu Song，Yu Liu，Hang Zhou
Venue:  (2020)

Abstract: Learning group representation is a commonly concerned issue in tasks where the basic unit is a group, set or sequence.
The computer vision community tries to tackle it by aggregating the elements in a group based on an indicator either defined by human such as the quality or saliency of an element, or generated by a black box such as the attention score or output of a RNN. 

This article provides a more essential and explicable view. 
We claim the most significant indicator to show whether the group representation can be benefited from an element is not the quality, or an inexplicable score, but the \textit{discrimiability}. 
Our key insight is to explicitly design the \textit{discrimiability} using embedded class centroids on a proxy set, 
and show the discrimiability distribution \textit{w.r.t.} the element space can be distilled by a light-weight auxiliary distillation network. 
This processing is called \textit{discriminability distillation learning} (DDL).
We show the proposed DDL can be flexibly plugged into many group based recognition tasks without influencing the training procedure of the original tasks. Comprehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efficiency, and it pushes forward the state-of-the-art results on these tasks by an impressive margin.

Under review as a conference paper at ICLR 2020
DISCRIMINABILITY DISTILLATION IN GROUP REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Learning group representation is a commonly concerned issue in tasks where the basic unit is group, set sequence. Previously, the computer vision community tries to tackle it by aggregating the elements in group based on an indicator either defined by human such as the quality or saliency of element, or generated by a black box such as the attention score or output of a RNN.
This article provides more essential and explicable view. We claim the most sig- nificant indicator to show whether the group representation can be benefited from an element is not the quality, or inexplicable score, but the discrimiability. Our key insight is to explicitly design the discrimiability using embedded class cen- troids on proxy set, and show the discrimiability distribution w.E.. the element space can be distilled by a light-weight auxiliary distillation network (DDNet). This procedure is denoted show the proposed DDL can be flexibly plugged into many group based recog- nition tasks without influencing the training procedure of the original tasks. Com- prehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efficiency, and pushes forward the state-of oftee-of-iae sice-of-theeate art results on these tasks by an impressive margin.
discriminability distillation learning (DDL). We
INTRODUCTION
With the rapid development of deep learning and the easy access large-scale group data, recog- nition tasks using group information have drawn great attention in the computer vision community. The rich information provided by different elements can complement each other to boost the per- formance of tasks such as face recognition, action recognition, and person re-identification (Wang) 2017b; Zhong et al. 2018 Girdhar et al. 2017 Simonyan & Zisserman, 2014 Yang et al. Liu et al. [201 Rao al. (0017)). While traditional practice for group-baser recognition is to either aggregate the whole set by average (Li et al.) 2014; Taigman et al. 2014) or max pool- ing (Chowdhury et al. 2016), or just sampling randomly (Wang 2016). the fact that certain
ctt al.
2017
elements contribute negatively in recognition tasks has been ignored. Thus, an important issue is to select representatives from sets for efficient group understanding.
To tackle such cases, previous methods aim at defining the "quality" or "saliency" for each element group (Liu et al. 2017c]] Yang et al. 2017 Rao et al.) [2017;; Nikitin al. (017). The weights for each element can be automatically learned by self-attention. For example. Liu et al. (001cch proposes the Quality Aware Network (QAN) to learn quality score for each image inside an image set during network training. Other works adopt the same idea and extend to specific tasks such as
video-based person re-identification (Li et al. 2018 Ww et al. 2018) and action recognition (Wang et al. 2018c) by learning pptiplltemporal attentions. However, the whole online quality or atten- tion learning procedures are either manually designed or learned through a black box, which lacks explainability.
In this work, we explore deeper into the underlying mechanism for defining effective elements in- stead of relying self-learned attention. Assuming that base network has already been trained for element- based recognition using class labels, we define the "discriminability" of one sample by how difficult it is for the network discriminate its class. As pointed out by Liu al. (2018) that the feature embedding of elements lies close to the centroid of their corresponding class are the representatives, while features far away closer to other classes the confusing ones which

---

Under review as a conference paper at ICLR 2020
not discriminative enough. Inspired by this observation, identify successful discriminabil- ity indicator by measuring one embedding's 's distance with class centroids and compute the ratio of between positive and hardest-negative, where the positive is its distance with its class's corre- sponding centroid and the hardest-negative is the closest counterpart. This indicator is defined as the discriminability distillation regulation (DDR).
Armed with recent theories on the homogeneity between class centroids and projection weights of classifiers (Wang et al. 2017a Liu al.) 2017b 2018 Deng et al. (019a), the entire distance- measuring procedure can be easily accomplished by simply encoding all elements in one group. Thus, the DDR scores can be assessed for each element after the training of the base network. This assessing procedure is highly flexible without human supervision nor re-training the base network, so it can adapted to any existing base. With our explicitly designed discriminability indicator on the training set, the distillation of such discriminability can be successfully performed with a light- weight discriminability distillation network (DDNet), which shows the superiority of our proposed indicator. We call the whole procedure uniformly as discriminability distillation learning (DDL).
The next step is towards finding a better aggregation policy. At the test phase, all elements are firstly sent to the light-weight DDNet. Then element features will be weighted aggregated by their DDR score into group representation. Moreover, in order to achieve the trade-off between accu- racy and efficiency, score. Since the base model tends to be heavy, the filter can save much computation consump- tion. We evaluate the effectiveness of our proposed DDL several classical yet challenging tasks. Comprehensive experiments show the advantage of our method on both recognition accuracy and computation efficiency. We achieve state-of-the-art results without modifying the base networks.
can filter elements by DDR score and only extract element features of high
We highlight our contributions as follows: (1) We define the discriminability of one element within a group from a more essential and explicable view, and propose an efficient indicator. (2) We verify that a light- weight network has the capacity of distilling discriminability from the assessed elements. Combining the post-proot post-processing with the network, the great computation burden can be saved com- paring with existing methods. (3) We validate the effectiveness of DDL for both efficiency and accuracy on set-to-set face recognition and action recognition through extensive studies. State-of- the-art results can be achieved.
2 RELATED WORK
2.1 SET-TO-SET RECOGNITION
Set-to-set recognition which utilizes a group of data of the same class, has been proved efficient on various tasks and drawn much attention these years since the more videos and group datasets are available. Compared with recognition with a single image, set-to-set recognition can further explore the complementary information among set elements and benefit from it. Particularly in this paper, we care for the basic task of face and action recognition.
Face Recognition. To tackle set-to-set face recognition problem. Wolf et al., 2011; Kalka al., 2018 Beveridge et al.] 2013) Klare et al. 2015). traditional methods directly estimate the feature similarity among sets of feature vectors (Arandjelovic
al. 2005 Harandi al., 2011 Cevikalp & Triggs 2010). Other works seek to aggregate element features by simply applying max pooling (Chowdhury et al. 2016) average pooling (Li et al. 2014, Taigman et al., 2014) among set fea- tures to form a compact representation. However, since most set images are under unconstrained scenes, huge variations on blur, resolution, and occlusion appear, which will degrade the set feature discrimination. How to design a proper aggregation method for set face representation has been the key problem for this approach.
Recently, few methods explore the quality or attention mechanism to form set representation. GhostVLAD (Zhong et al. 2018) improves traditional VLAD and down weight low quality element features. While et al. (2017a) combine LSTM and reinforcement learning to discard low quality element features. Liu et al. (2017c) and Yang et al. (001)) introduce attention mechanisms to assign quality scores for different elements and aggregate feature vectors by quality weighted sum. To predict the quality score, an online attention network module added and co-optimized by the target set-to-set recognition task. However, the definition of generated "quality" score remains unclear and
2

---

Under review as a conference paper at ICLR 2020
the learning procedures are learned through black box, which lacks explainability. In work, claim that the most sigmificant indicator to show whether the group representation can be benefited from element is not the quality discriminabilityy distillation learning procedure is proposed.
inexplicable score, but the discriminability. And novel
Action recognition. With the advance of multimedia era, millions of hours of videos uploaded to video platforms every day, so video understanding task like action recognition has become a popular research topic. Real-world videos contain variable frames, it not practical to put the whole video to a memory limited GPU. The most usual approach for video understanding is to sample frames or clips and design late fusion strategy form video-level prediction.
Frame-based methods Yue-Hei Ng et al. 2015 Simonyan & Zisserman, 2014 Girdhar et al. 2017) firstly extract frame features and aggregate them. [Simonyan Zisserman (2014) propose the two- stream network to simultaneously capture the appearance and motion information. [Wang et al. (2017b) add attention module and learn discard unrelated frames. Frames-baat based methods are computational efficient, but only aggregate high-level frame feature tends to limit the model ability to handle complex motion and temporal relation.
Clip-based method (Tran et al., 2015 2018 Feichtenhofer et al. 2018) use 3D convolutional neu- ral network to jointly capture spatio-tempora features, which perform better on action recognition. However, clip-based methods highly rely the dense sample strategy, which introduces huge com- putational consumption and makes unpractical to application. In this article, we show that by combing our DDL, the clip-based methods can achieve both excellent performance and computa- tion efficiency.
3 DISCRIMINABILITY DISTILLATION LEARNING
In this section, we first formulate the problem of group representation learning in section 3. and then define the discriminability distillation regulation (DDR) in section 3.2) Next, we introduce the
whole discriminability distillation learning (DDL) procedure in section 3.3 In sections 3.4 and 3.5
we discuss the aggregation method and the advantage of our DDL, respectively.
3.1 FORMULATION OF GROUP REPRESENTATION LEARNING
Group representation learning focuses on formulating uniform representation for a whole set of elements. The of either verification task classification task how to aggregate features of given element group.
Define f; the embedded feature of element 4; in group Is, then the uniform feature representa- tion of the whole group is
Fis = S21, ,f),
(1)
where G indicates the feature aggregation module. While previous research has revealed that con- ducting G with quality scores (Liu al. 2017c) has priority over simple aggregation, this kind of methods is not explainable. In this article, we propose a discriminability distillation learning (DDL) process generate the discriminability of feature representation.
3.2 DISCRIMINABILITYY DISTILLATION REGULATION
Towards learning efficient and accurate G. design the discriminability distillation regulation (DDR) to generate the discriminability to replace the traditional quality score. DDR, we jointly consider the feature space distribution and explicitly distill the discrininabillty by encoding the intra-class distance and clter-class distance with class centroids. Let X denote the training set with the identities C' and W E [1,C] C] is the class centroid, For feature fi, i [1, with class where
3

---

Under review as a conference paper at ICLR 2020
Base Model
W
W2
DDNet
Regress
DDR score
Figure I: The pipeline of group representation learning with our DDL. Given a base feature extract model, first compute the DDR score for each training element and then train light-weight discriminability distillation network (DDNet) to distill it. During testing, the DDNet will predict discriminability for each element in the test set.
denotes the size of X. the intra-class distance and inter-class distance are formulated as
fc We
Cie lffella 1|Wll2
(2)
fe W3
C.j l1fil2 |IW
j #
The intra-class distance Cie and inter-class distance Cj are shown in Figure 2 base model on classification task, features of the elements from the same class are projected to hyperspace tightly in order to form explicit decision boundary. Furthermore, elements close to the centroid of their corresponding class are the representative ones, while elements far away from their corresponding class closer to other classes not discriminative enough. Based on this observation, we define the discriminability Q₃ of f, as:
After training the
Cin
Q: max (Cij E [1,C)++c
ie.. the ratio of feature distance between the centroids of its class and the hardest-negative class. Considering the variant number of elements in different groups, we further normalize the discrim- inability by:
D. - ( 12--1 M(Q [ 01Q,, lj (
[1,s]})
where 7 7-)),(.)) and a(.) denote the sigmoid function, the mean value and the standard deviation value of (Q₂ Ij [1, s]]. respectively. We denote D; as the discriminability distillation regulation
(DDR) score.
Cooperated with the feature space distribution, the DDR score D; is more interpretable and rea- sonable than the quality score traditional quality learning. can discriminate features better by explicitly encoding the intra- and inter-class distances with class centroids.
3.3 DISCRIMINABILIT DISTILLATION LEARNING
From section 3.2, given a training dataset 0 and a corresponding model, the DDR score DD of fi can be naturally computed by Eq (2)-4). However, in group representation learning, D; is unavailable test set T without W,. In order to better embed D; to the group representation learning procedure, we introduce the discriminability distillation learning (DDL) generate an approximated DDR score D, for D,. Given an arbitrary CNN architecture. N and an image I,, D. can be approximated by
(5)
4

---

Under review as a conference paper at ICLR 2020
Wo
Cie
fi
Figure The formulation of discriminability distillation regulation. is the ratio element feature distance with its class centroids and the hardest-negative class's. After training the base model with classification task, element features from the same class are projected to hyperspace tightly order to form clear decision boundary. While the outliers who lie far away from its centroids closer to other classes are of low discriminability.
where 0 the parameter of the light-weight DDNet At the training stage, we apply mean squared error between D. and target D; as
N
L
2N Σ (D.
D,)2
(6)
where is the batch size.
3.4 FEATURE AGGREGATION g
At the test stage, we can generate D. via Eq 5) for each test element I; in the given element set Is.
The feature aggregation 9 in Eq (T) can be formulated
Rifi
Fis G(fi,fe fnh
R
where n is the element number Is. and R2 re-sealed DDR score of D, via
R, KD, + B.
(8)
In Eq 8 we scale the DDR score element set Is between and to ensure a same range for element sets with different lengths. K and B are formulated as
K max i' [1, - min L [1,n|}'
(9)
B max (D. [1,
(10)
3.5 ADVANTAGE OF DISCRIMINABILITY DISTILLATION LEARNING
Different from the subjective quality judgment of an image or quality learning via attention mech- anism, we explicitly assign discriminability for each element via the feature space distribution. By jointly considering the inter- and intra-class distances with class centroids, DDL can effectively approximate how discriminative feature is. According to aggregating more information from fea- tures with high discriminability in an element set, the recognition performance can be significantly boosted. With the assistant of DDL, both hard samples (the model is easy to fail on them) or low- quality images (the model is easy to give wrong predict results) be weakened. Moreover, the well design discriminability distillation learning process needn't 't modify the base model, making easy to be plugged into many popular recognition frameworks.

---

Under review as a conference paper at ICLR 2020
DDNet
DDR score
eggregation
Group Representation
Base Model
Top-K Filter
Figure 3: The pipeline of the test stage of DDL. For set of elements, we first predict DDR score by the trained-well DDNet for each element. Then the feature extracted by the base model will be weighted aggregation to form the group representation. To achieve accuracy and computation consumption trade-off, top-K filter also be applied select the most discriminative elements and only extract features and aggregate with them.
EXPERIMENTS
4.1 SET-TO-SET FACE RECOGNITION
DATASETS AND PROTOCOLS
In this section, we evaluate DDL for set-to-set face recognition on four datasets including two video sets: the YouTube Face (YTF) Wolf et al. (2011), FDYY-VVID-AACEJOOIYI based sets: the IARPA Janus Benchmark (UBBA) Klare et (2015) and the TARPA Janus Bench- mark C (UB-C).
and two template-
YTF:
The YouTube Face dataset includes 3425 videos of 1595 identities with an average 2.15 videos per identity. The videos vary from 48 frames to 6,070 frames. We report the 1:1 face verifi- cation rate of the given 5,000 video pairs in our experiments without fine-tuning.
IQIYI-VID-FACE: The iQIYI-VID-FACE dataset iQIYT (2019) aims identify the person in en- the largest video face recognition test benchmark so far, containing 643,8 36 video clips of 10,034 identities. The test protocol 1:1 verification and the True Positive Rate (TAR) under False Positive Rate (FAR) at 0.01 % is reported.
tertainment video by face images.
IJB-A:
unconstrained face recog- nition benchmark, containing 25, 813 faces images of 500 identities. It is a template-base bassed test benchmark where both still images and video frames included in templates. We report 1:1 veri- fication results by following official protocol.
The IARPA Janus Benchmark A (UB-A) Klare et al. (2015)
IJB-C:
The IARPA Janus Benchmark C (IJBC) C) is extended version of UB-A. It has 140, 740 faces images of 3, 531 subjects, so we can test under more strict False Accept Rates (FAR). Since the images in IJBC dataset have large variations, regarded challenging set-to-set face recognition benchmark.
THE DETAILS OF DISCRIMINABILITY DISTILLATION NETWORK
All set images will be sent to the DDN at the inference stage, so the DDN is sensitive to computa- tional burden. To make the inference stage efficient, the DDN is designed to be light-weighted. The typical DDN is a channel reduced ResNet-15 network, which only introduces 81.9 Mflops computa- tion for 112x 112 input. Compared with base feature extraction models which often reach 24 Gflops (ResNet-1001, it is super-efficient. More discussions on the DDN architecture can be found in the Appendix A
EVALUATION ON YOUTUBE FACE AND IQIYI-VID-FACE
As shown in Table I DDL achieves state-of-the-an art performance the Youtube Face bench- mark. outperforms Deng et al. (2019)) by 1.17% and other set-to-set face recognition methods by impressive margins. Since train stronger base model, the baseline average method is higher

---

Under review as a conference paper at ICLR 2020
Method
Accuracy(%)
Method
Accuracy
et al. (2014)
84.8
Taigman et al. (2014)
91.4
Schroff et al.
(2015)
95.52
Yang et al.
(2017)
95.72
Sun et al.
(2015)
93.20
Liu ct al.
(017cc
96.17
Gong et al.
(2019
96.50
Rao et al.
(2017b)
96.52
Liu et al.
(2019b)
96.21
Rao et
(2017a)
94.28
Wang al.
(2018b)
97.6
Deng et al.
(2019a)
98.02
Average
98.27
Top
97.11
DDL
99.19
Table I: Video face verification performance on YouTube Face dataset, compared with state-of-the art methods and baseline methods.
Method
TPR @ FPR=1e-49%)
Method
TPR TPR@FPR=le-4(6)
MSRA
71.59
AlibabaVVAG
71.10
Insightface 67.00
DDL (PolyNet)
72.98
Average 65.84
Top 1
65.22
DDL
69.05
Table Comparsion with different participants and aggreation stragey on the IQIYI-VID- FACE challenge. By combing with PolyNet, DDL achieves state-of- -ttte-of-the-ats peformance. http://wwwwwseeaaaac-ccaalllle//
than Deng et al. (0119a), but for comparison with different aggregation strategies like average pool- ing, DDL also boost performance by 0.92%, which indicates DDL has learned a meaningful pattern for discrininability. As a post-training module, DDL can cooperate with any pre-trained model. To prove the robustness and good generalization ability of our DDL., we re-implement more baseline and combine them with DDL, the results are shown in the Appendix A
Since the results YouTube Face benchmark tend to be saturated, lenging video face verification dataset IQIYI-VID-FACE, which is of large scale and more close to industrial application. Compared with the average pooling, DDL improves performance by 3.21 %. Even aggregating the top 1 DDR score features only can still achieve the performance of average aggregation for all frames. It shows that our DDL has selected the most discriminative element of the set. By combining PolyNet base model, our DDL. achieves the state-of-the tf-the-arts performance on the IQIYI-VID-FACE dataset.
our DDL the chal-
What's 's more, the good performance of only aggregating top discriminability frame shows that we needn't extract all frames by heavy base model, but only the most discriminative one during inference. Since the computational consumption for DDNet is efficient, usually only one-tenth of other heavy models. can seen that DDL makes a huge difference for industrial vidco face recognition application.
EVALUATION ON IJB-A AND IJB-C
Tables 4howw the results on the IJB-A and IJBC dataset for different methods. From the two tables, we can see that our DDL improves verification performance by convincing margin with average pooling for both two benchmarks, especially under severe FAR at 0.001% % by 1.94% and FAR at 0.0001% & by 6.88% on IJB-C. Compared with IJB-A, IBBC has more images and covers more variations among images, such as pose, blur, resolution, and conditions. So the performance gain with DDL is huger.
Compared with the state-of of-the-artttttt methods, our DDL improves IJB-A by 4.83% when FAR =0.001 and IJB-C C by 4.89% when FAR 0.0001% %. These results indicate the effectiveness and robustness of our DDL. What's more, unlike many previous methods that need fine-tune with the base model set-to-set recognition training datasets, the only supervision for DDL training is the feature dis-

---

Under review as a conference paper at ICLR 2020
UJBA (TAR@FAR)
Method
FAR=0.001
FAR=0.01
FAR=0.1
Crosswhite et al.
2018
83.6 2.7
93,9
97.9 0.4
Sankaranarayanan al.
(2016)
81.30 + 2.0
91.00
96.4 + 0.5
Xie & Zisserman
(2013
92.0 1.3
9.5
98.9 0.2
Liu al. (201
89.31 + 3.92
94.2
98.02 + 0.55
Cao al.
(2018)
92.1 + 1.4
96.8 + 0.6
9.2
Yang et al. (2017
88.1
9..1
97.8
Liu et al.
(2017c)
89.31 +
94.2 1.53
98.02 + 0.55
Zhong ct al.
(2018)
9
97.2 0.3
99.0 = 0.2
Liu et al.
(2019b)
93.61 + 1.51
97.28 + 0.28
98.94 = 0.31
Deng et al.
(2019a)
97.89 + 1.5
98.51 0.3
99.05 = 0.2
Average
97.71 + 0.6
98.43+ 0.4
99.01 0.2
DDL
98.44 + 0.3
98.79 = 0.2
99.13 + 0.1
Table Peformance comparsions IJBA verificatiton benchmark. The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported.
UBBC @ARRFFAR)
Method
0.0001%
0.001%
0.01%
1%
1%
Yin et al.
(0099
0.1
83.8
Xie et al.
(2018)
88.5
94.7
98.3
Zhao et al.
(2019)
82.6
89.5
93.5
96.2
Xie & Zisserman (2018)
77.1
86.2
92.7
96.8
Cao et al.
(2018)
74.7
84.0
91.0
96.0
Shi Jain (0199)
89.64
93.25
95.49
97.17
Deng et (2019a)
86.25
93.15
95.65
97.20
98.18
Average
84.06
93.81
95.73
96.98
97.87
DDL
91.14
95.75
96.94
97.72
98.36
Table 4: Peformance comparsions IBBC verificatiton benchmark. The True Accept Rates (TAR)
vs. False Postive Rate (FAR) are reported.
criminability generated with the base model on the same training set. So the fine-tune process is not needed.
To qualitatively evaluate the discriminability pattern learned by our DDL, we visualize the discrim- inability score distribution for two template images in IJB-C datasets. As shown in Figure 4 DDL can effectively identify image discriminability. Pictures with large poses, visual blur, occlusion, and incomplete content are regarded be low discriminative. The efficient discriminability judgment ability for DDL leads to extraordinary performance set-to-set face recognition problem. More visualization examples can be found in the Appendix A.
4.2 VIDRO ACTION RECOGNITION
DATASETS
ActivityNet-1.2. The ActivityNet-112 dataset Caba Heilbron al. (2015) contains 4,819 training videos and 2,383 validation videos for 100 action class. The duration of those videos varies from 2 seconds 4 minutes and the average length of videos is 2 minutes. We carefully remove videos with more than one label. Frames are extracted from videos and their width is resized to 240 pixels. It is untrimmed dataset, so large visual and content variation exists.
Kinetics-700. Kinetics Kay et al. (011)) is a well-trimmed and popular action recognition bench- mark. Since around 2% videos of Kinetics-600 and 12% of videos of Kinetics-400 are not accessi-
8

---

Under review as a conference paper at ICLR 2020
Tomplate 17782
0.813
0:638
0:170
3.258
0.178
0.989
0.512
0.172
Template 1800
Figure 4: The visualization results of discriminability for images of Template ID 17762 and 16800 from IJB-C dataset.
ble from youtube, we conduct experiments on the Kinetics-700 dataset. Kinetics-700 contains 700 classes and 650k videos, and each action class has at least 600 video clips. Each clip is human annotated with a single action class and lasts around 10s.
BASE MODEL AND TEST PROTOCOL
To combine and compare with DDL, we choose three clip-based 3D CNN models 3D 3D-ResNet-50 Hara et al. (018), SlowFast-50fFeiictenll
et al. al.] (2018) and R(2+1)D-50 Tran al. (2018). For testing, they firstly split the whole videos into clips and inference them individually. Then clip-level results will be average pooled to get the whole video's prediction. All three methods hugely rely on the dense sampling strategy during testing, which is quite computational consumed.
For our DDL, we firstly filter clips by the light-weight DDNet. Then we only extract and aggre- gate features from the top-K discriminability clips. The K value selected for Activitiynet- 1.2 and Kinetics- 700 is and respectively, for that they achieve the best efficiency and accuracy trade-off in experiments. For comparison, randomly uniformly sample K clips in videos. Dense sampling is also used to compare.
EVALUATION ON ACTIVITYNET-122
As shown in Table 5 DDL improves recognition performance for all baseline models. For the state-of-the-art clips-based model SlowFast, combining it with DDL can achieve around 4% accu- racy gain compared with randomly or uniformly sample. What's more, DDL can even outperform dense evaluation along with the whole video by a huge margin. It is reasonable as DDL can select and aggregate clips with the highest discriminability. which avoids video clips with visual blur or ambiguous content to pollute the vidco-level prediction.
What's 's more, combining with DDL, the inference process can hugely accelerated. For testing two minute videos with SlowFast-50 model as example, original dense sampling strategy introduces 17 Glops 60 computation but for DDL enhanced, only top discriminability videos need to be predicted by the heavy base model. The total consumption is 1.7 Glops 60 for assessing the discriminability plus 9x Gflops for extracting top discriminability clip feature, which achieves 4x speed up.
EVALUATION ON KINETICS700.
As shown in Table 6, DDL outperforms random sampling by 1.84% and uniform sampling by 2. 2.18. For dense sampling, DDL can achieve 0.86% gain with 6x speed up. Since Kinetics-700 well trimmed, the improvement is not so significant compared with the untrimmed video dataset ActivityNet-Le 1.2. However,
post-training module, DDL be incorporated with any clip-based 3D action recognition model without base model re-trained. can achieve more efficient and accu- rate inference, which is very suitable for
video understanding applications.

---

Under review as a conference paper at ICLR 2020
DDL clips)
Random elips) Acc(Si) G
Uniform (9 clips)
Dense all clips)
Model
Ace(R)
GFLOPsxclips
Acc(S) GFLOPsxelips
Acc(S) GFLOPsxcllps
3D-RS-50
86.38
37x9+1.7x60
82.83
37x9
83,14
37x9
83,92
37x60
R(2+))D-RS-50
89.08
39x9+1.7x60
84.51
39
84.89
x9
85.46
39
SlowFast-RS-50
90.21
16x9+1.7x60
85.92
16x9
86.14
16x9
87.72 16x60
Table 5: Video action recognition results ActivityNet-1.2 dataset. We compare randomly, uni- formly and filter by DDL to select clips to aggregate. Accuracy is reported on the valiation set. And the compute consumption is estimated on 2 minutes videos (above 60 clips), for that 2 minutes the average video length of ActivityNet-1.2.
DDL (5 clips)
Random (5 clips)
Uniform (5 clips)
Dense ( (30 clips)
Model
Acc(%)
GFip
Acc(%) GFLOPsxclip
Acc(%) GFLOPxxclips
Acc(%) GFLOPsxelips
3D-R5-50
71.01
37x3+1.7x30
37
67.43
x7
R(++)D RS.
72.51
35x5+117xx0
35x30
SlowFastt
74.23
16x5+1.7x30
72.39 16x5
72.05
16x5
73.37 16
Table Video action recognition results on Kinetics- dataset. We compare randomly, uniformly and filter by DDL to select 5 clips to aggregate. Accuracy is reported on the validation set and is the average of topl and top accuracy. Dense sample strategy sample 10 clips along the temporal axis and random crop 3 clips on the spatial axis.
CROSS DATASET AND MODEL EVALUATION.
In the last two sections, we train specialized DDNet for different datasets and base models, respec- tively. To demonstrate the generalization ability of our DDL, we conduct the dataset and cross model evolution experiment. Action recognition model R(2+1)D for Activity 1.2 dataset is com- bined with DDL trained with Kinetics-700 and base model SlowFast, the results are shown in Table 7 It can be found that through training with the different dataset and base model causes a perfor- mance drop, DDL can still improve about 1% accuracy for dense sampling. These results show that the discriminability pattern learned by DDL is robust and well to generalize to other datasets. What's s more, the generalization ability can avoid training the DDL model for each action model and each dataset, making the DDL more efficient and practical.
Base Model, Dalatset
Accuracy(S)
Base Model, Dalatset
Accuracy(Ri
SlowFast-RS-50 ActivityNNt-1.2
87.91
R(2+1DD, ActivityNet-122
89.08
SlowFast- RS- 50,50, Kinetics-700
86.31
R(2+1)D, Kinetics-700
86.92
Uniform
84.51
Dense
85.46
Random
84,80
Table 7: Results for cross dataset and cross model experiment. All results are reported with base ac- tion recognition R(2+e)D-ResNNet550 on ActivityNet-1. dataset. The training procedures for DDI. vary from dataset and base model.
CONCLUSION
In this paper, we have proposed novel post-processing module called DDL for all group-based recognition tasks. We explicitly define the discriminability with observations on feature embed- ding, then apply light network for discriminability distillation and feature aggregation. We identify the advantage of our proposed methods in the following aspects: (1) The entire discrim- inability distillation is performed without modifying the pre-trained base network, which is highly flexible comparing with existing quality- or attention methods. (2) Our distillation network is extremely liggh-weighted with great computational cost salvage. (3) With our DDL and feature
aggregation, we achieve state-of-fe of the-ar art results on set-to-set face recognition and action recognition tasks.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ognjen Arandjelovic, Gregory Shakhnarovich, John Fisher, Roberto Cipolla, and Trevor Darrell. Face recognition with image sets using manifold density divergence. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (C), volume 1, pp. 581- 588. IEEE, 2005.
J Ross Beveridge, Jonathon Phillips, David S Bolme, Bruce A Draper, Geof H Givens, Yui Man Lui, Mohammad Nayeem Teli, Hao Zhang, W Todd Scruggs, Kevin W Bowyer, al. The chal- lenge of face recognition from digital point-and-shoot cameras. In 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), Pp. 1-8. IEEE, 2013.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, 2015.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67-74. IEEE, 2018.
Hakan Cevikalp and Bill Triggs. Face recognition based on image sets. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2567-2574 IEEE, 2010.
Aruni Roy Chowdhury, Tsung- Yu Lin, Subhransu Maji, and Erik Learned One-to-many face recognition with bilinear cnns. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-9. IEEE, 2016.
Nate Crosswhite, Jeffrey Byrne, Chris Stauffer, Omkar Parkhi, Qiong Cao, and Andrew Zisserman. Template adaptation for face verification and identification. Image and Vision Computing, 79: 35-48, 2018.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Paitern Recognition, pp.
2019a.
Jiankang Deng, Jia Guo, Yuxiang Zhou, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-stage dense face localisation in the wild. arXiv preprint arXiv:1905.00641 2019b.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. arXiv preprint arXiv: 02392220..2222... 2018.
Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell, Actionvlad: Learn- ing spapioltemporal aggregation for action classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 971-980, 2017.
Sixue Gong, Yichun Shi, and Anil K Jain. Video face recognition: Component-wiss feature aggre- gation network (c-fan). arXiv preprint arXiv: /92207337 2019.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-In: A dataset and benchmark for large-scale face recognition. In European Conference on Computer Vision, pp. 87-102. Springer, 2016.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018
Mehrtash T Harandi, Conrad Sanderson, Sareh Shirazi, and Brian C Lovell. Graph embedding discriminant analysis on grassmannian manifolds for improved image set matching. In CVPR 2011, pp. 2705-2711 2712. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Under review as a conference paper at ICLR 2020
iQIY. iqiyi-vid-face 2019. URL Utt.:tt///eaalleee.......././//.
Nathan D Kalka, Brianna Maze, James A Duncan, Kevin O'Connor, Stephen Elliott, Kaleb Hebert, Julia Bryan, and Anil K Jain. ljb-s: larpa janus surveillance video benchmark. In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1-9.
IEEE, 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17050060000005500, 2017.
Brendan F Klare, Ben Klein, Emma Taborsky, Austin Blanton, Jordan Cheney, Kristen Allen, Patrick Grother, Alan Mah, and Anil K Jain. Pushing the frontiers of unconstrained face detection and recognition: larpa janus benchmark a. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1931-1939, 2015.
Haoxiang Li, Gang Hua, Xiaohui Shen, Zhe Lin, and Jonathan Brandt. Eigen-pep for video face recognition. In Asian conference on computer vision, pp. 17-33. Springer, 2014.
Shuang Li, Slawomir Bak, Peter Carr, and Xiaogang Wang. Diversity regularized spatiotemporal attention for video-based person re-identification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017a.
Xiaofeng Liu, Zhenhua Guo, Site Li, Lingsheng Kong, Ping Jia, Jane You, and BVK Kumar. Permutation-invariiat feature restructuring for correlation-aware image set-based recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4986-4996, 2019a.
Yu Liu, Hongyang Li, and Xiaogang Wang. Rethinking feature discrimination and polymerization for large-scale recognition. arXiv preprint
2017b.
Yu Liu, Junjie Yan, and Wanli Ouyang. Quality aware network for set to set recognition. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5790-5799, 2017c.
Yu Liu, Guanglu Song, Jing Shao, Xiao Jin, and Xiaogang Wang. Transductive centroid projec- tion for semi-supervised large-scale recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 70-86, 2018
Zhaoxiang Liu, Huan Hu, Jinqiang Bai. Shaohua Li, and Shiguo Lian. Feature aggregation network for video face recognition. In Proceedingy of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019b.
Mikhail Yurievich Nikitin, Vadim Sergeevich Konouchine, and AS Konouchine. Neural network model for video-based face recognition with frames quality assessment. Computer Optics, 41)): 732-742, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS 2017 Workshop, 2017.
Yongming Rao, Ji Lin, Jiwen Lu, and Jie Zhou. Learning discriminative aggregation network for video-based face recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3781-3790, 2017a.
Yongming Rao, Jiwen Lu, and Jie Zhou. Attention-awanl deep reinforcement learning for video face recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3931-3940, 2017b.
12

---

Under review as a conference paper at ICLR 2020
Swami Sankaranarayanan, Azadeh Alavi, Carlos D Castillo, and Rama Chellappa. Triplet proba- bilistic embedding for face verification and clustering. In 2016 JEEE 8th international conference biometrics theory, applications and systems (BTAS), pp. 1-8. IEEE, 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering, In Proceedings the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015. 2015.
Yichun Shi and Anil K. Jain. Probabilistic face embeddings. 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 568-576, 2014.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deeply learned face representations are sparse, selective, and robust. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2892-2900, 2015.
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap human-level performance face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701-1708. 2014.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497, 2015.
Du Tran, Heng Wang, Lorenzo Torresani. Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild.
arXiv preprint w 2017.
Fei Wang, Liren Chen, Cheng Li, Shiyao Huang, Yanjie Chen, Chen Qian, and Chen Change Loy. The devil of face recognition is in the noise. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 765-780, 2018a.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: 12 hypersphere embed- ding for face verification. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1041-1049, ACM, 2017a.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5265-5274, 2018b.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.
2017b.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794- 7803, 2018c.
Lior Wolf. Tal Hassner, and Itay Maoz. Face recognition in unconstrained videos with matched background similarity. IEEE, 2011.
Lin Wu, Yang Wang, Junbin Gao, and Xue Li. Where-and-when to look: Deep siamese atten- tion networks for video-based person re-identification. IEEE Transactions on Multimedia, 21(6): 1412-1424, 2018.
13

---

Under review as a conference paper at ICLR 2020
Weidi Xie and Andrew Zisserman. Multicolumn networks for face recognition. arXiv preprint arXiv:1807.09192, 2018.
Weidi Xie, Li Shen, and Andrew Zisserman. Comparator networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 782-797, 2018.
Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4362-4371, 2017.
Bangjie Yin, Luan Tran, Haoxiang Li, Xiaohui Shen, and Xiaoming Liu. Towards interpretable face recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp- 9348 9357, 2019.
Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijaanaarasimhan. Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4702- 2015.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5409-5418, 2017a.
Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin. Polynet: A pursuit of structural diversity in very deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 718-726, 2017b.
Jian Zhao, Yu Cheng, Yi Cheng, Yang Yang. Fang Zhao, Jianshu Li, Hengzhu Liu, Shuicheng Yan, and Jiashi Feng. Look across elapse: Disentangled representation learning and photorealistic cross-age face synthesis for age-invariant face recognition. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 9251-9258, 2019.
Yujie Zhong, Relja Arandjelović, and Andrew Zisserman. Ghostvlad for set-based face recognition. In Asian Conference on Computer Vision, pp. 35-50. Springer, 2018.
A APPENDIX
IMPLEMENTATION DETAILS FOR SET-TO-SET FACE RECOGNITION
Pre-processing:
RetinaFace Deng et al. (2019b) is used to detect faces and their corresponding landmarks for all datasets. Images are aligned to 112 112 by similarity transformation based their landmarks.
Training: We train base recognition model and DDL on MSS-Celeb-11 1M dataset et al. (2016).
Since the original dataset has been proved to be dirty, we use the cleaned version by Deng et al. (2019a), which consists of 5,179,510 pictures from 93,431 identities. For the base model training. we select the modified version of ResNet-101 101 He et al. (2016) introduced by Deng et al. (019a) and PolyNet Zhang al. (0177)) as the backbone network. The loss function ArcFacc et al. (019a).. The learning rate starts from 0.1 and is divided by 10 after 60k, 80k iterations, and we finish our training at 11k. The base model is trained on 16 GPUs with total batch size of 1024 and based on Pytorch |Paszee et al. (2017) framework. After training the base network, we extract features for each picture from the training set and compute its DDR score as discussed in section 3.2 Then the light- DDNet will regress the score by L2 loss with the same training image input. The learning rate for training our DDNet starts from 0.1 and is divided by 10 after 30k, 60k, and 80k iterations, and the total number of iterations is 100k. Linear warm-up with 10k iterations is applied towards stable training. The DDNet is trained on 8 GPUs with a total batch size of 512 are used.
14

---

Under review as a conference paper at ICLR 2020
DISCUSSION OF DDNET ARCHITECTURE
As has discussed in section 4.1 the design principle of our DDNet computation efficient, to introduce less computation burden. The typical architecture of DDNet is shown in Table 8 which is channel reduction Resnet- network. We also conduct experiments on deeper DDNet which has 34 layers and wider DDNet which is standard Resnet- 1 network. As shown in Table neither the deeper or the wider network leads to better performance, which illustrates that the discriminability of face images can be well learned with limited parameters. As for action recognition, we utilize the same DDNet architecture where the 2D convolution kernel is replaced by 3D.
stage
DDNet
output sizes x W
raw input
112x
3 x 3, stride 1,
conv1
112x
1 stride2,
rC82
2
56x56
1x1,8,
1 1, 16, strr 2
TE83
x 2
28x28
1 x 1, 1tr,st
1 x 12, stride2, 2 2
T684
14x 14
1 x 12, stridel,
1 1, 48, stride2, 2
x
7x7
1 x 1, 48, stridel,
global average pool, fc
DDR score
Table The architecture of our light-weight DDNet. Typical DDNet is channel reduction Resnet- 18 network and only introduces 81.9 Milops computation, which is super efficient.
(B-C(TAR@FAR)
Architecture
0.0001% %
0.001%
0.01%
1%
1%
ResNet-18
90.93
95.74
96.92
97.73
98.35
ResNet-34-CD
91.13
95.74
96.90
97.72
98.33
ResNet-18- 91.14
95.75
96.94
97.72
98.36
Table 9: Peformance comparsions for different architecture IJB-C verificatiton benchmark. The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported. CD represents channel reduc- tion shown in Table
RESULT OF COMBINING MORE BASELINE WITH DDL ON YOUTUBEFACE
The baseline model select in the main part trained with ArcFace loss function Deng et al. 2019a). We also re-implement more baseline model with different loss function like Wang et al. (2018)) Liu al, (2017a). As shown Table I0 when enhanced by DDL, all baseline achieve performance above 1% performance gain, which indicates the good robustness and generalization ability of our DDL.
DISCUSSION ON UNBALANCED DATA CLASS WHEN TRAINING DDL
Since the data imbalance between classes and such long- tail distribution has become concerns for training recognition models Zhang et al. (2017a); Van Horn & Peronal (2017). we conduct experi- ments with two settings for MS1M dataset Guo al. (2016):
Tail data: We use one-tenth of raw data with the minimum sample count in each class. has insufficient data per class and contains 519,379 pics from 29,210 classes.
Head data: We use one-tenth raw data with the maximum sample count each class. has enough data per class and contains 544,381 pies from 4,833 classes.
15

---

96.34 Under review as a conference paper at ICLR 2020
SF
SF(DDL)
CF
CF(DDL)
AF
AF(DD))
95.8
97.1
97.8
98.7
98.3
99.2
Table 10: Combine DDL with more baselines including SphereFace (SF) and CosFace (CF). All baseline models are trained with same backbone and datasets. Average pooling is used as the default aggregation strategy.
The comparison results for the two settings IJB-C C datasets are illustrated in Table I1 The per- formance for tail data and head data are almost the same because the formulation of DDR doesn't t explicitly encodes identity information. For the classification task, insufficient data for class may lead to bad representation for that class in subspace, but for our DDL training, the DDR defini- tion is class-unspccified, which means only the feature's intra- class and inter-class distance matters, but not the identity information.
What's more, the slight performance drop for training with of the tens of amount indicates that by increasing the training sample, the performance of DDL can further improve. But around 500,000 data are enough for training good DDL model.
IJB-C(TAR FAR)
Data 0.0001 %
0.001%
0.01%
1%
1%
Tail
89.81
95.38
96.62
97.48
98.14
Head
89.92
95.41
96.59
97.51
98.13
Full
91.14
95.75
96.94
97.72
98.36
Peformance comparsions for discussion of class data imbalance when training DDL. Head and tail data setting of same number samples, but varies from class numbers.
Table 11:
RESULTS OF TRAINING DDL WITH OTHER DATASETS.
the main article, we train our base model and DDL on the MSIM dataset et al. (2016), which is the largest open face datasets. In this section, we also train our base model and DDL on the IMDB dataset Wang et al. 20) which contains 59k identities and 1.7M images. The results on IJB-C dataset in Table 12 Training with the smaller dataset, DDL also achieves competitive results and better than average pooling for huge margin. At FPR=0.01 %, compared with average pooling, DDL boosts the performance by 29.64%. This indicates that weak model benefit more from DDL.
TIB-CCTAR@FARR
Method 0.0001%
0.001%
0.01%
1%
1%
average
0.002
0.22
39.39
91.55
DDL 0.002
0.40
66.33
94.83
97.55
Table 12: Peformance for training both base model and DDL on IMDB dataset on IJB-C benchmark. The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported.
RESULTS CASCADE TRAINING WITH DDL
Applying cascade training refine better centroid could be reasonable idea. We conduct exper- iments. After training one DDL, we use the DDR scores to weighted refine each class's 's centroid and generate DDR scores for DDNet regress. The results in Table 13] By cascade train- ing and refined centroids, DDL boosts performance for 0.4% at FAR=0.0001 %2 which means good centroids leading better diseriminability representation.
16

---

Under review as a conference paper at ICLR 2020
LUB-C(TAR@FAR
Method
0.0001 %
0.001
0.01% %
1%
1%
DDL
91.14
95.75
96.94
97.72
98.36
cascade training
91.53
95.91
97.01
97.82
98.40
Table 13:
Peformance comparsions for cascade training on IJB-C dataset. By refining the class centroid and cascade training, DDL achieves performance gain.
DISCUSSION OF NORMALIZATION AND SCORE SCALE FOR DDL
In Eq(4), we normalize discriminability by the mean and standard deviation to deal with a variant number of elements in different groups. In our experiments, if the normalization step is removed, the DDNet can't converge.
As for linear map the minimum and maximum DDR score of the element set to 0 and in (1(8)))(10), this step leading to 0.4% performance gain for YTF and 1.6% performance gain for IQIYI- MHYI-VID-FACEE But the scale process doesn't contribute to template- bastte set-to-set recognition like IJB- and IJBCC dataset. The reason is that in a video, frames are captured in the almost condition, the variation of frames tends be limited. The mapping process will separate the score distribution, making it more focuses on frames with higher discriminability. While for template-based set-to-set recognition like IJB-A and IJB-C, the variation of elements is large and the discriminability score distribution has been separated well, so there is no obvious gain when applying score re-scale. More aggregation strategy scripts can found our GitHub.
IMPLEMENTATION DETAILS FOR ACTION RECONGNTION
To train ActivityNet- dataset, pre-trained model on Kinetics-700 datasets. Video clips with adjacent 64 frames are randomly sampled from raw videos for training and testing. For the spatial domain, 160] pixels. 32 NVIDIA V100 GPUS with synchronized SGD training and global BN are used to train and we find that result for typical training in one GPU machine is the same. Similar to the 2D-DDL, we generate the pseudo discriminability label for each clip and use channel reduction 3D resnet-18 18 network regress the score after training the base model. The training setting for Kinetics is the same with ActivityNet-1.2, except sample clips with adjacent 32 frames.
randomly crop 11 112x 112 pixels with shorted side randomly sampled in [128,
A.1 QUALITATIVE ANALYSIS
In the main part, we visualize two templates from IJB-C datasets. In this subsection, we visualize more examples from YouTube Face dataset. As in these experiments, our DDL has learned meaningful discriminative patterns.
17

---

Under review as a conference paper at ICLR 2020
0.18
0.30
0.93
0.35
0.35
0.57
0.39
0.40
0.42
0.43
0.44
0.47
0.48
0.51
0.53
0.55
0.57
0.59
From confused discriminative
Figure 5: Visualization discrimiability of frames from different videos.
18

---

Under review as a conference paper at ICLR 2020
0:5
0.18
0.16
0.17
0.37
0.18
0.18
0.18
0.18
0.18
0.19
0.20
0.20
0.20
0.21
0.11
0.22
0.22
0.25
0.4
0.26
0.30
0.47
0.47
0.51
0.11
0:12
0.12
0.12
0.18
0.13
0.14
0.14
0.11
0.12
0.12
0.19
0.14
0:15
0.15
0.29
0.39
0.40
0.40
0.41
0.1
0.42
0.42
0.42
0.48
0.43
0.44
0.44
0.45
0.4G
0.48
Figure Visualization discrimiability of frames from one video.
19


Reviews:
[{"review_id": "r1l-AFjntS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a discriminability indicator using embedded class centroids on a proxy set, and show the discriminability distribution w.r.t. the element space distilled by a light-weight auxiliary distillation network, which is called discriminability distillation learning (DDL).\nThis methodology was tested on a selection of datasets addressing set-to-set face recognition and action recognition. I have the following concerns, which made me to suggest rejecting this paper.\n\n1) The methodology explained in Section 3 should be improved. In the way it is presented, it is not clear and reproducible. In detail,\n1.1)  It is better to describe the pipeline together with Figure 1.\n1.2) Contribution of the top part of Figure 1 in proposed pipeline is not clear. More descriptive caption should be added.\n1.3) In section 1, where you ask reader to observe the feature embedding of elements which lie close to the centroid… Which section or figure are you referring to?\n1.4) In subsection 2.3, clearly discuss how proposed approach is different than traditional distillation methods. Also, explain your modifications and discuss why new structure is better than default ResNet-18. \n1.5) Subsections 3.2-3.4 do not contain enough explanation to support equation of proposed method. \n1.6) It is not clear why authors use cosine distance, because it is simple? \n1.7) The quality scores generated by DDL is very hard to interpret, which to me the one of the biggest problem of the proposed methodology.\n\n2) The related work section should include studies more related to the motivation behind DDL (“….to select represent samples from the whole set efficiently for group understanding.”). For instance, attention models, saliency detection, key-frame detection methods, outlier detection methods, quality aware networks, etc. should be discussed and compared with DDL. Instead, authors more focused on face recognition and action recognition tasks themselves. Therefore, many related works are missing. Especially, Section 2.3., should not be a part of related work but should be integrated either to Section 1 (Introduction) or Section 3.\n\n3) Experimental analysis: \n3.1 ) It is not possible to figure out if the proposed method is really performing better than SOA. In other words, the experimental analysis should be extended. One thing authors should consider is including more datasets, especially more challenging datasets i.e., the ones not already saturated. Some examples can be: iLIDS-VID, PRID2011, YouTube Face, IJB-A benchmark.\n3.2) Is there any reason to mainly focus on face recognition and action recognition?\n3.3) “For face recognition, DDL can easily improve the performance by concentrating the discriminative information and for video action recognition, DDL can further accelerate the pipeline by eliminating the frames with insufficient information.” This sounds like DDL behaves differently depends on the task, but perhaps the conclusion driven by the authors cannot be fully correct given that the number of used datasets are limited to make such a general statement. I suggest authors to re-write this sentence.\n3.4) The experimental analysis should include comparisons with methodologies such as quality learning via attention mechanism, etc. to better understand the necessity and effectiveness of proposed DDL. However, only for one dataset such a comparison was performed. It is strange why for other datasets the same comparison was not done.\n3.5) Experimental analyses so much focuses on to justifying the performance improvements of DDL is independent to the base model selected (e.g. mainly Table 3,  but also Table 4, 5 and 6 include). However, to me this is a supporting experiment and it is more important to show the real necessity of the proposed method by comparing it with SOTA methods.\n3.6) I do not find using Youtube Face benchmark suitable as it is already saturated, i.e. many existing models perform around 97%, which does not allow to show whether DDL can significantly contribute to the task or not. From the corresponding results, it looks like DDL does not provide any significant improvement.\n3.7) In Table 3, it is hard to observe the contribution of DDL in terms of the performance. The improvements are in the level of 1%. More discussion is needed about that table.\n3.8) “The discriminability distillation learning is more practical to untrimmed video action recognition since there are more diversity videos chip with ambiguous content and visual blur problem”  This means that DDL should not be used in some tasks, it is not generalizable, task-specific? Please elaborate it more clearly.\n3.9) Table 2 is hard to interpret, improve the discussions regarding it.\n3.10) Table 1 and table 2 were not referred in the text. What does DATA refer to in Table 1, why they are different?, if they are really different i.e. it is not me who misinterpret, then is not it unfair to compare the results?\n\n\n4) Other suggestions:\n4.1) Support your statements with references and appropriate experimental analysis. \n4.2) In Section 4, only YTF dataset is mentioned but authors also use IQIYI-VID-FACE challenge dataset.\n4.4) Section 4.2. is not clear at all, and should be re-written.\n4.5) Some references to the tables are wrong or some are missing, pls. check them all. E.g., in section 4.2., paragraph two, table 3 should be table 2.\n4.6) There is no reference to the any figures in the text.\n4.7) Paper contains several typos and grammatical mistakes. Also, most of the points were not explained well. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gD7AVC9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to aggregate features from group inputs. The paper proposes  Discriminability Distillation Learning (DDL) to compute the aggregation coefficients. The method assumes that each sample has a discriminability property that is directly related to the task. The authors define this property and propose to learn such property by an auxiliary network. Such a network can be used in many models without affecting their original training procedure and is able to improve the performances on many tasks, including set-to-set face recognition and action recognition. The experimental results are comprehensive and convincing.\n\nThe idea is simple yet interesting and the results are good, so I tend to give a positive rating.\n\nBut overall, the paper is not well-written, and there are some questions:\n\n1. The paper needs more proofreadings. For example,\n    a. only frames **will** high scores will be weighted and aggregated\n    b. As shown in table ??\n2. It seems that the proposed method is limited to tasks that have the concept of “centroids”. Many tasks may not have well-defined centroids.\n3. Discriminability is also a kind of quality in some ways, and it is used in a similar way to the previous methods. The authors are encouraged to discuss more the differences to highlight the novelty and contributions. Especially after viewing Fig. 3 and 4, their boundaries become less clear.\n4. I might be wrong but I didn’t see any figures referenced in the main body. Please add references and organize them better.\n5. The discriminability is used only once. I’m just wondering if the authors have considered cascade training, i.e., use the discriminability information to train better centroids, which then will define discriminability better.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1eS1vpA5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a discriminability distillation learning (DDL) method for the group representation learning, such as action recognition recognition and face recognition. The main insight of DDL is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t. the element space can be distilled by a light-weight auxiliary distillation network. The experimental results on the action recognition task and face recognition task show that the proposed method appears to be effective compared with some related methods. The detailed comments are listed as follows, \n\nThere are many grammar errors and typos in the current manuscript, such as \n-\tOur key insight is to explicitly design the discrimiability using embedded class centroids on a proxy set…\n\nThe authors proposed DDL based on the principle of the intra-class distance and inter-class distance. How to avoid the imbalance of the dataset, namely some classes have the insufficient data?\n\nIn Eq4, the authors adopt a normalization method. How about the influence of the model if we ignore this normalization? Some ablation study whether or not the normalization is missing.\n\nThe implementation details of the proposed framework are unclear. For example, how to fine tune the network?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1eyMbYEcr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a post-processing method for improving group-based recognition tasks. Several manually designed features based on the pretrained networks are supervised trained on a light-weighted network like the teacher-student module. \n\nThis paper should be rejected because (1) the novelty of the algorithm is limited: only using the well-known intra-class distance and inter-class distance as features. Besides, the superiority of such features should be better explained; (2) Similarly, the discriminability of testing image is too complicated, such as Eq. (9, 10). Why these params are designed in such ways? Explain them; (3) Why don’t you learn the discriminability by directly operating on the extracted pretrained features? The used two distances certainly are not the best choices compared with learning methods. You should compare the performance with your methods of training under the manually-designed features. (4) The written is poor, such as Figure 6: “table ??”; Table 2: “avgerage”. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes discriminability distillation learning (DDL) for learning group representations. The core idea is to learn a discriminability weight for each instance which are a member of a group, set or sequence. The discriminability score is learned by first training a standard supervised base model and using the features from this model, computing class-centroids on a proxy set, and computing the iter and intra-class distances. A function of these distance computations are then used as supervision for a distillation style small network (DDNet) which may predict the discriminability score (DDR score). A group representation is then created through a combination of known instances, weighted using their DDR score. The method is validated on face recognition and action recognition.

This work initially received mixed scores, with two reviewers recommending acceptance and two recommending rejection. After reading all the reviews, rebuttals, and discussions, it seems that a key point of concern is low clarity of presentation. During the rebuttal period, the authors have revised their manuscript and interacted with reviewers. One reviewer has chosen to update their recommendation to weak acceptance in response. The main unresolved issues are related to novelty and experimental evaluation. Namely, for novelty comparison and discussion against attention based approaches and other metric learning based approaches would benefit the work, though the proposed solution does present some novelty. For the experiments there was a suggestion to evaluate the model on more complex datasets where performance is not already maxed out. The authors have provided such experiments during the rebuttal period.

Despite the slight positive leanings post rebuttal, the ACs have discussed this case and determine the paper is not ready for publication.
Decision: Reject