Cluster and Aggregate: Face Recognition with Large Probe Set
Authors: Minchul Kim
Feng Liu
Anil Jain
Xiaoming Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: feature fusion
face recognition

Abstract: Feature fusion plays a crucial role in unconstrained face recognition where inputs (probes) comprise of a set of $N$ low quality images whose individual qualities vary. Advances in attention and recurrent modules have led to feature fusion that can model the relationship among the images in the input set. However, attention mechanisms cannot scale to large $N$ due to their quadratic complexity and recurrent modules suffer from input order sensitivity. We propose a two-stage feature fusion paradigm, Cluster and Aggregate, that can both scale to large $N$ and maintain the ability to perform sequential inference with order invariance. Specifically, Cluster stage is a linear assignment of $N$ inputs to $M$ global cluster centers, and Aggregation stage is a fusion over $M$ clustered features. The clustered features play an integral role when the inputs are sequential as they can serve as a summarization of past features. By leveraging the order-invariance of incremental averaging operation, we design an update rule that achieves batch-order invariance, which guarantees that the contributions of early image in the sequence do not diminish as time steps increase. Experiments on IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition.

Cluster and Aggregate: Face Recognition with Large Probe Set
Minchul Kim
Feng Liu
Department of Computer Science
Department of Computer Science Michigan State University East Lansing, MI 48824
Michigan State University
East Lansing, MI 48824
kimminc2@msu. edu
liufeng6emsu
Anil Jain
Xiaoming Liu
Department of Computer Science Michigan State University East Lansing, MI 48824 j jain@msu. edu
Department of Computer Science Michigan State University East Lansing, MI 48824 liuxm@msu. edu
Abstract
Feature fusion plays crucial role in unconstrained face recognition where inputs (probes) comprise of set of N low quality images whose individual qualities vary. Advances in attention and recurrent modules have led to feature fusion that can model the relationship among the images the input set. However, attention mechanisms cannot scale to large N due to their quadratic complexity and recurrent modules suffer from input order sensitivity. We propose two-stage feature fusion paradigm, Cluster and Aggregate, that can both scale to large N and maintain the ability to perform sequential inference with order invariance. Specifically, Cluster stage is linear assignment of N inputs to M global cluster centers, and Aggregation stage a fusion over M clustered features. The clustered features play integral role when the inputs are sequential as they can serve as summarization past features. By leveraging the inder-invariance of incremental averaging operation, we design update rule that achieves batch-order invariance, which guarantees that the contributions of early image in the sequence do not diminish as time steps increase. Experiments IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition. Code and pretrained models are available in Link.
Introduction
Face Recognition (FR) matches set of input query imagery, known probe, to enrolled identity database, known as gallery. Verification to confirm the claimed probe's identity and identification is to identify the unknown probe's s identity by searching known database [41]. In either case, probe can go beyond an image and include set of images, videos, or their combinations [4]. Thus FR involves fusing features of multiple images videos create discriminative feature for probe.
Due to the interest in unconstrained surveillance scenarios, e.g. IJB-S [21]. the role of fusion is becoming more important. Unconstrained FR is often based on probes from low quality images and videos. is challenging due to two issues: 1) individual video frames can be of poor quality, causing erroneous FR model prediction and 2) the number of images in a probe can be very large, e.g., probe video in IJB-S may have 500, 000 frames. Feature fusion across all frames the probe is especially crucial if frame-based predictions are unreliable. While prior works [23,36] address the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unconstrained Serting (low quality multippe frames)
Intra-Set Relation
Large X Probe
Order
-Naive -Maaie--PEE-CERR PFE--CFAN RSA CAAISA-CAATA CAFace
Invartance
8.0 % inc. over Naive
Single
X
6.0
Proposed
4,0
Intro Set Attention
2.0
Recorrent Modale
()): probe size
Contritation of bad quality images
Contribution of pood quality images
Proposed
UJBA (9)
IJB-B
(21)
IJB-C
(23)
(7114)
IJB-S
a) Fusion without intra-set relationship
b) Drawbucks of prior works
<) Benefit of feature fusion with large N
Figure 1: a) An illustration of the importance of the intra-set relationship in feature fusion. Without the intra-set relationship. large weight on good quality image can still be outweighed by many bad quality images in a probe sct. b) We need framework that can both account for the intra-sct relationship of large N probes and handle sequential inputs with order invariance. c) The role of fusion model increases with larger probe size. For our proposed method, CAFace, the relative performance gain over Naive (simple averaging) method, i.e. wew.rporrooooee
CAFax-Niiv * 1 100%, increases with the probe size acorss four datasets, PFE [43] and CFAN [14] are single-image Navve
based and lack intra-set relationship. RSA [28] computes intra-set relationship but unusable for large N.
first issue of prediction in low quality images, the large size of probe set was not addressed. Fig. 1 a) illustrates the problem caused by the absence of proper feature fusion. The contribution of good quality image can be made insignificant in the presence of many other poor quality images in the set.
This paper aims learn a fusion function that maps unordered set of N probe features {fa J of the same person to a single fused output f. Note that f: = Exx;) is the feature extracted from the i-th sample in the set, using a fixed feature extractor E. The task of fusing multiple features involves I) estimating the quality of individual features and 2) modeling the intra-set relationship of the features. Prior feature fusion works utilize either simple average pooling [10,39], reinforcement learning [29], recurrent models [15] or self- atf-attennion [14, 28, 31, 533.
Typically. to compute the intra-set relationship among inputs of an arbitrary size N, one would adopt set-to-set functions such as Multihead Self Attention (MSA) [28, 47,49), enabling inputs to propagate information among themselves. The downside of this approach is its computational cost of (N2) which becomes infeasible when N exceeds a few thousand. Also, when the inputs are sequential as in live video feed, it is nontrivial model the intra-set relationship except to compute attention all past frames at each time step. Recurrent methods [15, 18] are useful in the sequential inference but their drawback is set order inconsistency, i.e., as the number of sequential steps T increases, the contribution of early frames in a set decreases. Fig. 1 b) contrasts various fusion methods.
A feature fusion framework that consider both 1) intra-set relationship for large N and 2) efficient sequential inference is necessary in the real-world unconstrained FR scenarios. Fig. c) shows the average probe sizes of four datasets. IJB-$ [2s]'s probe size is too large for intra-set attention such as RSA [28] to perform inference with all frames concurrently.
We present feature fusion framework, Cluster and Aggregate (CAFace), that achieves two abovemen- tioned criteria. It consists of two modules: Cluster Network (CN) and Aggregation Network (AGN). CN makes soft assignments of N features into M fixed number of clusters, i.e., {f.] {ff) where M << N. While N varies from one set to other, M fixed. AGN combines M clustered features into single feature f. i.e., {ff) M f. Conceptually, M intermediate cluster features serve as a summarization of N inputs and AGN models the intra-set relationship among {ff) M.
The proposed framework depends on learning global cluster assignments (f.) - ( that are consistent across different probes. Thus, we propose learning shared cluster centers that are input independent. These centers govern the clustering assignments. But, is not obvious which clustering criterion is the best for feature fusion. Thus, we design CN to discover learned clusters with an end-to-end differentiable framework that allows AGN to back ppooapaae the gradients to CN. The cluster assignments are learned to maximize the FR performance. We also design an input pipeline, Style Input Maker (SIM) that can helps CN perform class (identity) agnostic clustering efficiently.
The purpose of introducing an intermediate stage {f}} M is to facilitate the sequential inference. The key design of CN is formulate {f M linear combination of {f:} N. This guarantees that even when the input sequence of set length N divided into T smaller batches of set length N", (ff) can be sequentially updated with batch-order invariance. This is due to our update rule, inspired by

---

Global Ceaters
Ciolar Newark
C R
Fit E
R KG
Cluser Newwork
Updnn:
Agprogriin Naveed
Talk
/ /E
Fizz 12
Charr
Updas
Aggeggriin Netaork
Natwon
Rall
fc R
Fin
Individual Paradigm
Joint Intra-set Paradigm
R
Cluster Aggregate Paradign
Figure 2: Comparison of feature fusion paradigms. a) In the individual paradigm, each probe sample's weight is determined independently. b) In the intra-set paradigm, the sample weight is determined based on all inputs. However, when N is large or sequential, intra-set calculations become infeasible. e) In the Cluster and Aggregate paradigm, the intermediate representation F" (green) can be updated across batches, allowing for large N intra-set modeling and sequential inference. Sharing universal cluster centers C' ensures consistency of F across batches. Unlike RNN, the update rule is batch-order invariant.
the order invariance property of the averaging operation, as in Eq. 8. When the inputs are sequential, we feed only the new features to CN and update the cached {f M It achieves the similar effect as having used all previous features simultaneously. Fig. 2 shows the contrast with previous approaches. For readability, we will interchange the set notation {ffe N with the matrix notation F R NxC.
In summary, the contributions of this paper include:
A novel feature fusion framework for both large N feature fusion and efficient sequen- tial inference. To our knowledge, this is the first approach to utilize linearly combined intermediate clusters achieve batch-order invariance with intra-set relationship modeling.
An task-driven clustering mechanism that can discover latent clustering centers that maxi- mize the task performance. our case, the task is FR. We achieve the task-driven clustering with an assignment algorithm using the global query and decoupled key and value structure.
We show the superiority of CAFace in unconstrained face recognition on multiple datasets.
2
Related Work
Feature Fusion (Unordered Set). The simplest way of feature fusion is to average over set of features (f:}* [10,39]. In this case, the features with larger norms play bigger role, and it generally works since easy samples tend to show larger norms [36,40] 40). To learn the weights, CFAN and QAN utilize the self-attention mechanism, learned weighted averaging mechanism [14,31]. The drawback of these approaches is the lack of an intra-set relationship during the weight calculation process.
Previous works that adopt the intra-set attention mechanism Non-local Neural network and RSA 49]. These works use intermediate feature maps U, of size RCu x H xW during aggregation because feature maps provide rich and complementary information that can be refined by taking the spatial relationship into account. However, the drawback is in the heavy computation in the attention calculation. For set of N features maps, an attention module involves making (N H x W)² sized affinity map. Our Cluster Network utilizes a compact style vector from SIM and makes N2 sized affinity map which greatly increases the computation efficiency in attention computation.
DAC [29] and MARN [15] propose RL-based and RNN-based quality estimators, respectively. Yet, they fail to be agnostic to input stream order and are unsuitable for modeling long-range dependencies. Our method can split the N inputs into smaller batches and still achieve batch-order invariance.
Video Recognition (Ordered Set). The feature fusion for recognition has a resemblance to video- based recognition [30], but set inputs cannot always expect the temporal dependencies to be available. Therefore, most video-based approaches for tasks such as action recognition or quality enhancement 33,,,2,,3,,8,,56 focus on exploiting the relationship between nearby frames, whereas feature fusion approaches do not define them. In video-based FR, the general trend is to focus more on assessing the quality of individual frames as opposed to exploring the relationship among nearby frames. Some examples of video-based FR utilize n-order statistics [34], affine hulls [8, 19, 54], SPD matrices [20] and manifolds (.. Recently, probabilistic representation such as PFE [43] gained popularity [1,9, 42,433 since the variance in distribution serves quality estimation for individual frames.
3

---

Cluster and Aggregate (CAFace)
Claster Network (CN)
Ap- previous butch
Aggregation Network LAGN)
M
Pc ga
Global Ceake
Query:
A8 gatta
styce Input (SIM)
woigle
Aviignment
Stylk
map A
F: cotput
Transforner
AF
miighted gvanape
Sadiffa
MxN
Vatur: (5.8)
Decouples
R
St Fc zest botch
(Opiiceallyy intermedistes with. prosious butch intermediatoss fise..
N imagee
sdentity
Identity
Assigns inputs to globally stared oemers
Figure An overview of CAFace with cluster aggregate paradigm. The task to fuse a sequence images to a single feature vector f for face recognition. SIM is responsible for decoupling facial identity features F from image style S that carry information for feature fusion (Sec. 3. Cluster Network (CN) calculates the affinity of 5 to the global centers C' and produces an assignment map A. It will be used to map F and S to create fixed size representations F' and S'. Note that F' and S' are linear combinations of raw inputs F, S respectively. This property ensures that the previous and current batch representations can combined using weighted average, which is order-invariant. Lastly, AGN computes the intra-sel relationship of S' to estimate the importance of F' for fusion. For interpretability, AGN produces the weights for averaging F' to obtain f.
Attention Mechanism. Multihead Self Attention (MSA) [47] is a widely adopted set-to-set function that models intra-set relationships via an affinity map. It also a key component in transformer architectures which outperform CNNs in various vision tasks [7, 11, 13,27,32,44 55]. The underlying mechanism of MSA which uses the affinity of query and key to update the value is versatile in its application beyond recognition and has led to its usage in memory retrieval and grouping [5, 15,44,52), The unique property of the proposed Cluster Network in the linear combination of value assignment which enables batch-order invariance using an incremental average update rule. Unlike MSA which requires concurrent inputs during inference for intra-set relationship, ours can split the inference and establish connection across batches without decreasing the contribution of early inputs.
Proposed Approach
The Cluster and Aggregate paradigm seeks to divide the large N inference into partitioned inferences while still obtaining the result as seeing all inputs at the same time. This can be achieved if 1) each partitioned inference can update the intermediate representation with necessary information and 2) the order of inference does not affect the final outcome, so the information in early batches is not forgotten. In essence, the intermediate representation serves as a communication channel across batches. We achieve this by designing a Cluster Network (CN) and Aggregation Network (AGN). Fig. 2 c) shows the proposed paradigm. this section, we will elaborate on how we obtain the global assignment that is consistent across batches and how the update rule can be batch-order invariant. We formally layout a few assumptions for the Cluster and Aggregate paradigm in the face recognition (FR) task shown in Fig. 3.
Let {a J be a set of N facial images from the same person. The task is to produce single feature vector if from {x;} that is discriminative for the recognition task. We assume that single image based pretrained face recognition model E x; + fi is available following the settings of previous works [14,28,43]. For readability, we will interchange the set notation {f.} with the matrix notation F E RNX where (f,) the input is a set of length N and F simplifies equations. For clarity, we denote N to be the probe size (the number of images in set) and N' to be the partitioned set size when N is large. During training, we fix the number of images for fusion as N'. Note that the shape of inputs during training would have one more dimension, training batch size B, i.e. F BB XC Training batch size refers to the number of persons sampled in mini-batch, different from the number of images per person, N'. We drop the training batch size dimension in equations for brevity.
3.1 Architecture
Cluster Network (CN), Cluster Network responsible for mapping inputs F NCC of variable size N' to F' E MC of a fixed size, M. A natural choice for the architecture would be Trans- former [13, 47] as a set to set function. However, there are two problems with it. 1) It cannot handle large inputs due to the quadratic complexity of MSA. 2) When the inputs are partitioned and

---

1.0
6.8
1000
0.00
0.00
0.00
01-
1:00
0.00
9.00
.00
Q1
0.50
0.00
0.00
0.50
5.6
Q2
0.00
0.00 0.00
100
Q2
0.00
0.00
1:00
8.00
02-
1:00
0.00
6.4
6.2
Q3
0.0
1.00
0.00
0.00
q3
03
Out
8)
Row-SooMMax
b) Columa-SoflMax
<) Column-SoflMax (noemalized)
d) Depicted Assignment
Figure 4: a) Row-SoftMax. The sum across the row should be 1.0. b) Column SoftMax. Each column sums to 1.0. <) Column SoftMax with row normalization (Eq. 3). d) Depiction of how values are assigned to centers when A is multiplied to V. The matrix is deliberatly made sparse for visualization but it can be soft- noft-assignments.
inferred sequentially, the intra-set information across the batch is lost, MSA computes the affinity within the given inputs. CN solves this problem by modifying Transformer with 1) shared queries and 2) linear value mapping. These changes result in a clustering mechanism.
We first consider the following generic attention equation [47] with query Q. key K and value V. W.V.
Attn( K, V) SoftMaxrow QWg (KW+)*
Vd
where W WR. W. are learnable weights and is the channel dimension of K. The row-wise Softmax ensures that the output is the weighted average of all projected values W. V for each query index. We modify this to
CW, (KW&) Vd
A (K, V) = SoftMax,col
-- AV.
First, unlike K and V which inputs, the query is now shared learnable parameter C' initialized at the beginning of training. Secondly, removing W, and the column-wiss Softmax ensure that for each query index of C, the output is the (soft) selection of values V. These two modifications result in a learned soft assignment mechanism where C' serves as the global shared center. We name the assignment map A. The difference between A from row and column SoftMax is shown in Fig. a) and b). We then divide A by the weight of samples assigned to each center (row-sum of A) as in
Clusterc K, V)
Σ, A.. V, CN(K, V) = Clusterc [K, CT), V)).
(3)
Note that Cluster (, V) is linear in V, while the prediction of A is nonlinear. To further add the nonlinearity of to the Cluster Network, we first embed the keys K with shallow Transformer before clustering. The combined result CN(-) is the learned soft assignment of values according to the affinity between keys and global queries. More details on the architecture provided in Supp.
Style Input Maker (SIM), So far, we have discussed the generic Cluster Network algorithm. For face recognition, we still need to decide keys K and values V for feature fusion. It is clear that V should be F, the facial identity features, as it is what we are interested in merging. It is possible to use F for K as well, but K should ideally contain useful information for fusion and be compatible with queries which are the global center C. However, fi is optimized to be invariant to any characteristics other than the identity. Thus lacks input image style which encompasses various image traits such as brightness, contrast, quality, pose, or domain differences from the training data.
In light of the success of using first and second-orde feature statistics we propose SIM for extracting style information using the intermediate representations of feature extractors. The benefit of modeling keys K in clustering with style over using identity is shown in Sec. 4.2. We also ablate the benefit of further including feature norm [| in K as it sometimes used approximate the confidence of the prediction [23,36].
image style [22,26.37]
Let UM xHxW be the intermediate feature. We capture image style by a style vector 7 E R644 O r))))
7/ Bachhoom((FCReelU[AAypollll
where I = [May uny, oyyl
May SpatialMeann(U)) suy SpatialStd(UU.)
A learnable matrix W. E XM controls the importance of Mua and day via element-wise multipli- cation @ Simply put, SIM is shallow network on spatial mean and standard deviation U, One can take U, from more than one intermediate locations and in such a case, concatenate them.
5

---

To verify whether the feature norm would further benefit the fusion process, we embed the feature norm l| 64-dim vector, following the convention of Sinusoidal conversion [47], which is analogous to the position embeddings in ViT [13]. The norm embedding n, is a 64-dim vector and the details can be found in Supp.B. Finally, the output SIM the concatenation, 8; = (y), n:] where
8, E Rd and d = 64 + 64 = 128. For readability, we denote the set {s E RNNN N x28 as S.
In summary, we decouple style S and identity F and use S as keys to map
F' = CKey = S, value = F). S' = CN(kky = S, value = 5),
(5)
which are the intermediates that will be used for subsequent fusion in AGN or stored for sequential inference. We also map S to S' using the same assignment. Fig. 3 shows the overall diagram.
Aggregation Network (AGN). The Aggregation Network is responsible for fusing fixed number of MM inputs, F' and 5' into a single fused output f with intra-set relationship. We adopt MLP. Mixer [45] as it can efficiently propagate information for the fixed-size input. For interpretability, we predict weights E IR Mx that combines F' to f E R Specifically, f AGN(S', F") is
f= S=EPOFI M
P= SoftMax S'' ,')))
(6)
where [S, C] denotes the concatenation along the channel dimension. The magnitude of P is an interpretable quantity showing the importance of each cluster during fusion. The final output f is weighted average of F' whose weight is P. The details of MLP-Mixer [45] can be found in Supp.
Sequential Inference. A key characteristic of CAFace its ability divide the inputs into T-step sequential inference of smaller set length N' when N is large, and still achieve similar results as the concurrent inference. It is possible the intermediates F' and S' are linear combinations of F, S respectively, although estimating the combination weights A is non-linear. This allows us to formulate the update rule as the incremental weighted average whose innate property is order-invariant.
Consider partitioned inputs F1, Fr, with corresponding predicted weights A Ar. Since by definition (Eq. 5), F = F, E (L.j))) we can write the cumulative intermediate, Fr as
A + + Fr Σ Σ A,
F1
This formulation requires storing all inputs of timestep 1, I,..,, T. We can easily convert this to
Fy aT F N' Fr OT Σ AT,S))
T-1 N
where ar-1 Σ
A
(8)
which requires storing only the cumulative row-summed assignment map ar and a cumulative intermediate _ of the previous time-step. The same logic applies to S' as well. Note that this operation, by design, is invariant inference order (batch-order) as the final result will always be the total weighted average. However, we do not obtain element-wise permutation invarinace as the prediction of A, will change with different inputs. We test the susceptibility to element- permutation in Sec. 4.3 and it has minimal impact on the overall performance.
3.2 Loss Function
Template Loss. The objective is to make the fused output f be close to the ground truth class center for. The task can be viewed correctly inferring the true class center in the presence of low quality features {f. Here for is dependent on the pretrained feature extractor E and can be either taken from the last FC layer of E or computed using per-subject average of the embeddings f. in the training data. Our loss function can be viewed as the cosine distance version of the Center loss [50].
In training, we randomly sample B number of subjects and N' images per subject. Let the superscript in (b)) denote the 6-th subject. The loss to increase the cosine similarity is,
B
C,
1
Σ
CosSim (AGN(S(()) Frb)), ((
(9)
B
b=1

---

Chuster
Mean 09.05
0.03
0.02
Choses 2
Moan Py: 0.25%
0.30 0
0.25
0.25
0.20
0.09
0.24
0.08
0.08
0
06
0.36 16 0.01 0 07 0.09 0
0.04 0.00
0.01
0.02
0.02
0.00
0.00
0.00 0.00
0.089
0.22 16
0.33
0.35
0.30
0.53
0.07
0.42
0.21
0 36 0.14 0 o 01
0.02 0.09 0.08
0.01
0.00
0.01
0.00 0.00 0.00
Chustrr 4
Mean P4: 9,000
0.04
18
0.20
0.68
0.97
Pcc PCRM00, My agt
Figure 5: plot of assignment map A E R 2x23 (right) and the mean of cluster weights P (left) for samples in
UBBB [51]. For each column in A, the values sum up to 1.0. A shows that 1) high quality images are assigned to clusters 1, and 3, with large mean cluster weights P; low quality images are assigned to cluster with near 0.0 weight. 2) There are variations among clusters 1, 2 and 3 as to which images have more influence, e.g. cluster 3 focuses on relatively blurred or occluded images.
Set Permutation Consistency Loss. The Cluster And Aggregate paradigm achieves batch-wise order invariance by formulation, but it does not achieve element-wise permutation invariance, as noted in Sec. 3.1. Therefore, we explore the element-wise permutation's added benefit using an additional loss function. Thus, the set permutation consistency loss Lp is
B (- b=1
- = B
M
(1-CosSim
CosSim
(GG((S"t) (F)()))) AGN( ST ((b) Fr FT)))
(10)
It lets the splitted inference outcome similar to the concurrent inference. Sec. 4.2 shows the benefit of Lp but it small, meaning the batch-order invariance from model design already powerful. The final loss is
C= Le+ + ApCp.
(11)
where Ap is the scaling terms for Lp respectively.
4 Experiments
4.1
Datasets and Implementation Details
We use WebFace4M [57] as our training dataset. It is a large-scale dataset with 4.2M facial images from 205,90 990 identities. The single image based pretrained face recognition model E has been trained with the whole training dataset. To train the aggregation module, we use its randomly sampled subset, consisting of 813, 482 images from 10, 000 identities. We do not use VGG-2 [6] or MSIMV2 [12, 16] as they were withheld by their distributors due to privacy and other issues.
For the pretrained face recognition model E, we use the IResNet-101, trained with ArcFace loss [12]. Since the performance of the aggregation depends on the quality of E, we set E' be the same for all experiments. E produces an embedding vector fi E 5 for each image. To offer variations in the training data features, we randomly augment the dataset with cropping, blurring, and photometric augmentations. The training hyper- parameters such as optimizers are detailed in Supp.A.
We test on IJB-B [51], IJB-C [35] and IJB-S [21] datasets. UB-B is a widely used FR test set containing both high-quality images and low-quality videos of celebrities (see Fig. 5 for examples). IJB-C is an updated version of IJB-B with more complex motions in the video. IJB-S is surveillance video dataset, benchmarking extremely low-quality image/video face recognition. The probe and gallery set size can exceed 500, 000. Within the set, there are many low-quality images, making IJB-S very challenging and suitable for measuring the feature fusion framework (see Fig. for examples).
For IJB-S, word in the protocol refers to the probe/gallery image source. Surv. is the surveillance video, 'Single' is the frontal high-quality enrollment image and 'Booking' refers to the 7 high-qualit enrollment images. For the ablation study, Sec. 4.2, we report the average of all 9 metrics listed in Tab. 4.
protocols, Surv-to-Singlle Surv.-to- Surv.-to-Booking and Surv.-to-urv. The first/second
4.2 Ablation and Analysis
Effect of Different Style Input. In Tab. 1, we ablate the efficacy of various components in SIM which prepares the input for CN. The table shows that using f; for clustering is harmful to performance and increases the no. of parameters for CN. It also shows that using the additional norm embedding n, along with 8; produces the best results for IJB-B and IJB-S datasets. However, the margin is small, so simply using s, would suffice for the setting that requires faster computation.
Effect of Cy. As noted in Sec. 3.2, we further propose to constrain the set permutation consistency with the additional loss Cp. The ablation between Ap 0 and Ap 1 is shown in Tab. 1.
7

---

0,0100
Sample Weight
000075
CAFace
0,0023
0.0
Gallery Image
10,0000
Samp Weights
Simple Averaging (magnitude f weight)
0,0000
0.3
0.6
Cosine Similant between Gallery and Indivdunt Probe
Figure 6: A plot of similarity of fused probe vs gallery. The circles represent individual probe images in UJBS. The colors represent the contribution of each image during fusion. (top: CAFace. bottom; averaging). The x-axis the cosine similarity with the gallery feature (closer to right: better the match). The black star represents the fused feature. For CAFace, we also plot 4 intermediate features F" that go into AGN. (1) Compared to averaging scheme, in CAFace, only select few are contributing to fusion (few red). Since most samples are low quality, sparse selection of samples lead to better result. (2) Note that, out of 4 intermediates. FýJ falls behind others. It is because CN tends to assign bad quality samples
cluster, e.g. C3. More examples can be found in Supp.
Table I: Ablation of varied inputs, loss functions and the number of centers.
fz
72
of Centers (M) E,
#f Params UJB- FAR 1o-3) BB-B (TAR& lc-)lc-4)
UB-S (AVG)
16.28M
96.11
94.38
53.83
X
x
4
0.25M
96 81
95.53
57. 42
0.79M
95.53
X
>
V
0.79M
96.91 91
95,53
57.55
96.
95.52
57.36
0.7860M
96. 10
94.31
53.87
0.7862M
95.52
57.11
0.7865M
96.91
95.53
57.55
0.78741M
96 66.90
95.61
57.33
Naivo Average Fusion
96 10
9-1.30
51.12
Effect of Number of Clusters. In Tab. 1, the effect of the number of clusters M is shown. The IJB-S performance peaks when M = 4. When M = 2, the summary representations F' and S' have only two assignment options where one cluster takes the poor quality images with low weights. The be- havior could be interpreted as performing an outlier detection, which is powerful enough to give high performance. When M > 2, F' and S' have the capacity to store richer history previous frames which would be beneficial in sequential inference. IJB-S has large N probes which require dividing the inference into batches. Higher IJB-S performance when M indicates that the freedom to assign samples to different clusters is important in the sequential setting. A similar phenomenon is observed in Fig. 7. The performance gap widens for A = 2 as we re- duce the batch size to make more sequential steps in inference.
M=2
M=4 - M=8
Stable across varied B
Weight Visualizations. An example of clustering assignments when M = 4 could be viewed in Figs. 5 and 6. Fig. 5 shows how samples are soft-assigned different clusters along with the weight estimation. The cluster weight is calculated by averaging along 512 dimensions of P E R Mx522 Note that each column sums to but F' and S' are calculated by aver- aging each row. Thus, the relative contribution of samples in each row is important. Fig. 6 shows the actual contribution of each sample during fusion. The contribution can be calculated by multiplying the magnitudes of A and P. Note that in the presence of many poor quality images, selecting a few good ones is very important and the sample weight of our method can effectively select a subset of samples during fusion.
56.9
96.5
56.7
96.8
96.4
N==8
N'=32
N'-64
NNNN'
12=
N5256
NEN
Sharper Decrease for M-2
Figure plot of IJB-B performance CAFace with varied temporal batch size N' for models with different num- bers of clusters M.
4.3 Comparison with SoTA methods
To compare with prior feature aggregation methods, we use the same feature extractor F. as in Sec. 4.1, for fair comparison. Average is the conventional embedding f. averaging scheme that is adopted
8

---

Table 2: A performance comparison of recent methods the IJB-B [51] dataset.
Method
Intra-sct Ant Seq. Inference
FPS f
FAR
TAR@FAR=IO-T
TAReFAR=IOS f
Average
0
X
96
94.30
89.53
PFF[43]
13.37M
X
3
96.37
94.82
91.02
CFAN
12.80M
X
5
96.43
94.83
91.10
RSA [28]
2.62M
V
x
3.1x
96.41
95.00
91.22
CAFace
0.79M
4
64.4X
96.91
95.53
92.29
Table 3: performance comparison of recent methods on the IJB-C [25] dataset. CAFace achieves the best result in IJB-C dataset. We also compare two different backbones ArcFace [12] and AdaFace [23]]
[3-1
Dataset
Backbone E
TAR&FAR=Ie-3 TAR&FAR=Ie-
FAR-Te--
Naive Average
WchFace44 [57]
[12]
97.30
96. 78
92.60
PFE [43]
WchFaco-M [7]
IResNet101 ArceFas [12]
97.63
96 33
94.16
CFAN [14]
WehFace4M [57]
IResNet ArcFace [1]
97.55
96. 45
94.40
RSA 128]
WebFacoIM [57
[ResNecl01
97.49
96, 49
94.58
CAFace
WebFacoIM [57]
(RosNett01-ArcHac
97.99
97.15
95.78
Naive Average
WebFaco4M [57]
IRoNNeclO-+AAaFacc [2]
97.63
96.42
941.47
CAPace
WebFaco-M
IRouNactO1+AdaFaac [23]
98.08
97.30
95.96
Table 4: A performance comparison of recent methods on the JJBS [21] dataset.
Survellancetoo-Snnll
SurvelliancettoBowe BBokk-igg
Surveillane-o-SSuvelllaas
Methed
Rank-1
Rank-5
1%
Rank-1
Rank-5
1%
Rank-1
Rank-5
1%
Naive Average
69.26
74.31
57.06
70.32
75.16
56.89
32.13
46.07
5.32
PFE [43]
69.50
74.39
57.51
70.53
75.29
57.98
32.27
46.70
5.41
CFAN [14]
70.00
74.58
57.93
70.90
75.58
58.09
31.66
45.59
5.79
RSA [28]
63.04
67.33
51.62
63.54
68.23
51.89
16.82
31.80
0.75
CAFace
71.61
76.43
62.21
72.72
77.41
62.68
36.51
49.59
8.78
CAFace (Random Order)
71.65
76.37
62.27
72.77
77.37
62.70
36.43
49.40
8.89
100055
20.04
1:0.11
1.004
10.03
1.0.06
1:0.08
1.0.05
4.0.3
in the absence learned aggregation model. It is equivalent to the stand-alone ArcFace model performance. The rest of the methods learn additional network for fusing the set of features.
In Tab. 2. we show the performance of various feature fusion methods on IJB-B. CAFace achieves a large performance gain in all TAR @FAR metrics. CFAN [14] and PFE [43] do not use any intra-set relationship, as they learn to predict the confidence a single image. RSA [28] calculates intra-set attention of feature maps, which is computationally costly and incapable of sequential inference. CAFace obtains the best results with the least number of parameters. In Tab. 3, the performance in UB-C dataset is also shown with the similar observation as in IJB-B. We also include an additional backbone, AdaFace [23] to highlight how CAFace can work across different backbones. More performance comparisons can be found in Supp.C.
In Tab. 4, we compare feature aggregation models in the IJB-S dataset that has large N low quality images/videos in probes. RSA [28] cannot load all images in the probes concurrentyy for large N. As an alternative, we divide the probes into manageable size of N' 256 and average the results. Since RSA does not have a sequential update mechanism, dividing large N probes reduces the performance, which shows why the sequential capacity is important. CAFace also divides the probes image into batch size of N' 256 images yet achieves large margin improvement in IJB-S. It shows that our two-stage mechanism very effective in the large N setting. Particularly, the performance gain in the hardest protocol, Surveillance to Surveillance, is the largest. We also randomly shuffle images within the probes 5 times and measure the mean and std. of the performance in the last row. The result shows that our model is robust to input ordering. We also include an experiment quality image dataset, IJB-A [25], in Supp.C, and note that the performance gain with feature fusion negligible. As noted in Fig. c), the improvement over the baseline (averaging) goes up with the increased number of images in the probe, which highlights the importance of large N scalability.
a high
4.4 Resource and Computation Efficiency
Since CAFace is build on top of a single image feature extractor E, we show the relative FPS of CAFace with respect to the FPS of E in Tab. 2. The relative FPS reported in Tab. 2 computed with the input sequence length N = 256. It shows that the single image based quality estimation methods, PFE and CFAN are the fastest. And RSA with intra-set attention the slowest. CAFace can achieve relatively good speed and obtain the best performance.

---

Another aspect of computation requirement is GPU memory usage. In the second column of Tab. 5, we show the maximum sequence length N that each method can take simultaneously perform the feature fusion. It shows that RSA with the inner-set attention cannot handle a sequence length N larger than 384. This is drawback that prevents the method from fusing large N features. On the other hand, CAFace can take a large N sequence upto 12, 000 simultaneously. Note that for the sequence length larger than this can still be handled because CAFace has a sequential inference scheme as described in Sec. 3.1. In other words, we can divide the input into smaller set size N", and the intermediate representation is updated to account for all elements in a set. Tab. 5 also shows the relative FPS of the fusion model compared to the backbone FPS under different sequence lengths N. The details on the experiment setting can be found in the Supp.D.
Table 5: A table of relative FPS of the fusion model with respect to the FPS of the backbone. We compare various fusion models with varied input size N. As N increases, it requires more GPU memory well. Max N in the second column refers to the maximum number of images that can be in without causing the out of memory error (OOM). The third to the seventh columns represent the relataive FPS under different set length N. The higher the relative FPS, the faster the fusion method.
Max N
N = 16
N = 32
= 64
N = 256
512
PFE
115, 200
21.8x
44.1x
86.3x
360.1x
2133.6x
CTAN
115, 200
82.6x
158.7x
268.8x
544.1x 1x
664.2x
CAFace
000
4.2x
8.2x
16.4x
64.4x
129.3x
RSA
384
0.9x
13.1x
9.2x
3.1x
0OM
Conclusions
We address the two problems arising from the feature fusion of large N inputs, common scenario in unconstrained FR. With large N features, modeling intra-set relationships with attention mechanisms are prohibitive due computational constraints while sequential inference suffers from the reduced contribution of early frames. In this work, we explore the possibility of dividing N inputs into T smaller unordered batches whose inference result is the same as concurrent N inference. To this end, we introduce a two-stage cluster and aggregate paradigm. The clustering stage, inspired by order-invariance of incremental mean operation, designed linearly combine N inputs M global cluster centers, whose assignment is invariant to the batch-order. The aggregation stage efficiently produces a fused output from M clustered features, while utilizing the intra-set relationship. We show our proposed CAFace outperforms baselines on unconstrained face datasets such as IJB-B and JJBS.
Cluster and Aggregate is feature fusion framework that learns the weights of individual inputs, given a fixed feature extractor E. Weight estimation, in other words, is an interpolation among the given set of features which is a double-edged sword, as gives the interpretability, but is not capable of extrapolation. Therefore, when the given feature extractor E is sub-optimal, it could be favorable relax the constraint and let the model extrapolate for better performance.
Limitations.
Potential Negative Societal Impacts. We believe that the machine learning community as a whole should strive to minimize the negative societal impacts. Large scale face recognition training datasets inevitably comprise web-crawled images which are without formal consent, or IRB review. We refrained from using any dataset withdrawn by its creators such as VGG-2 [6] MS MSIMV2 [16] to avoid any known copyright issues. We hope that the FR community can collectively move toward collecting datasets with informed consent, fostering R&D without societal concern.
Acknowledgments. This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2022-21 102000 The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
10

---

References
[1] Ognjen Arandjelovic, Gregory Shakhnarovich, John Fisher, Roherto Cipolla, and Trevor Darrell. Face recognition with image sets using manifold density divergence. In CVPR, 2005.
[2] Anurag Arna,, Mostafa Dehghani, Georg Hcigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit; vidco vision transformer. In ICCV, 2021.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding. arXiv preprint
(eti:2102.05000 2(3)4, 2021.
[4] Lacey Best-Rowden, Hu Han, Charles Otto, Brendan F Klare, and Anil K Jain. Unconstrained face recognition: Identifying a person of interest from media collection. TIFS, 2014.
[5] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arX6: 1527, 2020.
[6] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VGGFace2: A datasct for recognising faces across pose and age. In FG, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. RCCV, 2020.
[8] Hakan Cevikalp and Bill Triggs. Facc recognition based image sets. In CVPR, 2010.
[9] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition.
In CVPR, 2020.
[10] Jun un-CChen Chen, Rajeev Ranjan, Amit Kumar, Ching-Hur Chen. Vishal Patel, and Rama Chellappa. An end-to-end system for unconstrained face verification with deep convolutional neural networks. In ICCVW, 2015.
[11] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
[12] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. AreFace: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021.
[14] Sixue Gong, Yichu Shi, Nathan D Kalka, and Anil K Jain. Video face recognition: Component-wiss wisc feature aggregation network (C-Fan). In ICB, 2019.
[15] Sixue Gong, Yichun Shi, and Anil Jain. Low quality video face recognition: Multi-mode aggregation recurrent network (MARN). In ICCVW, 2019.
[16] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong Hc, and Jianfeng Gao. MS-Celeb- I:: datasct and benchmark for large-scale face recognition. ECCV, 2016.
[17] Mchrtash T Harandi, Conrad Sanderson, Sarch Shirazi, and Brian Lovell. Graph embedding discriminant analysis grassmannian manifolds for improved image set matching. In CVPR, 2011.
[18] Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Newral Computation, 1997.
[19] Yiqun Hu, Ajmal S Mian, and Robyn Owens. Sparse approximated nearest points for image set classifica- tion. In CVPR, 2011.
[20] Zhiwu Huang. Ruiping Wang. Shiguang Shan, Xianqiu Li, and Xilin Chen. log-cuclidcan metric learning on symmetric positive definite manifold with application image set classification. In ICML, 2015,
[21] Nathan D Kalka, Brianna Maze, James A Duncan, Kevin O'Connor, Stephen Elliott, Kaleb Hebert, Julia Bryan, and Anil K Jain. UB-S: TARPA Janus Surveillance Vidco Benchmark. In BTAS, 2018.
[22] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[23] Minchul Kim, Anil K Jain, and Xiaoming Liu. AdaFace: Quality adaptive margin for face recognition. In CVPR, 2022.
[24] Tae Hyun Kim, Mehdi SM Sajjadi. Michael Hirsch, and Bernhard Scholkopf. Spatio-temporal transformer network for vidco restoration. In ECCV, 2018.
[25] Brendan F Klare, Ben Klein, Emma Taborsky, Austin Blanton, Jordan Cheney, Kristen Allen, Patrick Grother, Alan Mah, and Anil K Jain. Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark-A. A. CVPR, 2015.
[26] HyunJae Lee, Hyo-Eun Kim, and Hyeonseob Nam. Srm: A style-based recalibration module for convolu- tional neural networks. In ICCV, 2019.

---

[27] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. ICCV, 2021.
[28] Xiaofeng Liu, Zhenhua Guo, Site Li, Lingsheng Kong, Ping Jia, Jane You, and BVK Kumar. Permutation- invariant feature restructuring for correlation-aware image set-based recognition. In ICCV, 2019.
[29] Xiaofeng Liu, BVK Kumar, Chao Yang, Qingming Tang, and Jane You. Dependency- attention control for unconstrained face recognition with image sets. In ECCV, 2018.
[30] Xiaoming Liu and Tsuhan Chen. Vidco-based face recognition using adaptive hidden markov models, In CVPR, 2003.
[31] Yu Liu, Junjie Yan, and Wanli Ouyang. Quality aware network for set to set recognition. CVPR, 2017.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[33] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106.1 S00i::2006..33300 2021.
[34] Jiwen Lu, Gang Wang, and Pierre Moulin. Image set classification using holistic multiple order statistics features and localized multi-kernel metric learning. In /CCV, 2013.
[35] Brianna Maze, Jocelyn Adams, James Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil Jain,
W Tyler Niggel, Janet Anderson, Jordan Cheney, and Patrick Grother. IARPA Janus Benchmark C: Face dataset and protocol. In JCB, 2018.
[36] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. MagFace: A universal representation for face recognition and quality assessment. In CVPR, 2021.
[37] Hyconscob Nam, HyunJac Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, 2021.
[38] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In ICCV, 2021.
[39] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC, 2015.
[40] Rajeev Ranjan. Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discriminative face verification. arXiv preprint 1xXx:1700300507 2017.
[41] Syed A Rizvi, P Jonathon Phillips, and Hyconjoon Moon. The feret verification testing protocol for face recognition algorithms. In Proceedings third IEEE international conference automatic face and gesture recognition, pages 48-53. IEEE, 1998.
[42] Gregory Shakhnarovich, John W Fisher, and Trevor Darrell. Face recognition from long-term observations. In ECCV, 2002.
[43] Yichun Shi and Anil K Jain. Probabilistic face embeddings. In ICCV, 2019.
[44] Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mchta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a diffcrentiable search index. arXiv preprint arXiv:2202.00 06999) 2022.
[45] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. In NeurlPS, 2021.
[46] Hugo Touvron, Mathhiuu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017.
[48] Ruiping Wang, Shiguang Shan, Xilin Chen, and Wen Gao. Manifold-manifold distance with application face recognition based on image set. In CVPR, 2008
[49] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[50] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference computer vision, pages 515- Springer. 2016.
[51] Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K Jain, James A Duncan, Kristen Allen, al. IARPA Janus Benchmark-B face dataset. In CVPRW, 2017.
[52] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprint arXiv: 2022.
12

---

[53] Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. In CVPR. 2017.
[54] Meng Yang, Pengfei Zhu, Luc Van Gool, and Lei Zhang. Face recognition based on regularized nearest points between image sets. In FG, 2013.
[55] Li Yuan, Yunpeng Chen, Wang, Weihao Yu, Yujun Shi. Ha-Hang Jiang, Francis Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-tokes to-toke vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
[56] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Vidco transformer without convolutions. ICCV, 2021.
[57] Zheng Zhu, Guan Huang. Jiankang Deng. Yun Ye, Junjie Huang. Xinze Chen, Jiagang Zhu, Tian Yang. Jiwen Lu, Dalong Du, et al. WchFace260M: henchmark unveiling the power of million-scale deep face recognition. In CVPR, 2021.
13


Reviews:
[{"review_id": "meE4u9yVic_", "reviewer": "Reviewer_pygM", "summary": "This paper aims to improve face recognition performance in case each probe has multiple images by proposing Cluster and Aggregate (CAFace), a two-stage technique to better aggregate extracted probe features. In the Cluster stage, the model learns to softly assign N inputs into M global clusters, with the number of clusters M predefined. It first passes the extracted features through a Style Input Maker (SIM) to extract \"style\" features, then passes the \"style\" features through a Cluster Network (CN) to softly assign them to clusters based on learnable cluster centers. In the Aggregation stage, the model computes M clustered features, then fuses them using an Aggregation Network (AGN) to get the final output. CAFace divides inputs into T-step sequential inference with incremental averaging and guarantees set order invariance by employing a Set Permutation Consistency loss. CAFace provides the best performance boost compared with other feature fusion techniques on IJB-A, IJB-B, IJB-C, and IJB-S benchmarks.", "questions": "- It is unclear to me when the cluster centers are trained. Are they trained along with the fusion network and be universal to all probes? Or are they optimized to fit each probe? In the case of universal cluster centers, some analyses of facial images near the cluster centers are recommended to add.\n- The proposed method seems over-complicated. The authors should add more ablation studies to show the need for the proposed network components. For example, what if we do not use the style embeddings but learn the cluster centers for identity features f_i themselves? What if we do not use the Aggregation Network and simply average the clustered features in F'?\n- In Section 3.2, the definition of f^{(p)}\\_{GT} is confusing. Is not all training data labeled? What is the difference between f^{(p)} and f^{(p)}_{GT}?\n- The authors should improve the description for Fig. 6:\n  - What are C0, C1, and C2? Are they cluster centers? C3 is not visualized?\n  - In CAFace plot, there are many blue points with high cosine similarity to the Gallery. The authors should visualize and analyze the differences between the images corresponding to these points and the red/yellow points that make them have different weights.\n- Which network layer was used to extract the intermediate feature maps M_i in the experiments, and what is the feature map size?", "limitations": "The paper mentioned Limitations and Potential Negative Societal Impacts.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- As shown in Fig. 1b, CAFace avoid the weaknesses of previous methods. It could handle a large number of images per probe (N) while considering Intra-set relation. It is batch order invariant, and guarantees set order invariance by employing a Set Permutation Consistency loss.\n- CAFace provides the best performance boost compared with other feature fusion techniques on IJB-A, IJB-B, IJB-C, and IJB-S benchmarks.\n- CAFace seems to group low-quality images into a cluster, which has the fusion weith near zero.\n\n### Weaknesses\n- L18-19: This definition is only valid for Face Identification, one of two subproblems in Face Recognition. In Face verification, we do not have Gallery or Probe.\n- It is unclear to me when the cluster centers are trained. Are they trained along with the fusion network and be universal to all probes? Or are they optimized to fit each probe? In the case of universal cluster centers, some analyses of facial images near the cluster centers are recommended to add.\n- The proposed method seems over-complicated. The authors should add more ablation studies to show the need for the proposed network components. For example, what if we do not use the style embeddings but learn the cluster centers for identity features f_i themselves? What if we do not use the Aggregation Network and simply average the clustered features in F'?\n- In Section 3.2, the definition of f^{(p)}\\_{GT} is confusing. Is not all training data labeled? What is the difference between f^{(p)} and f^{(p)}_{GT}?\n- The authors should improve the description for Fig. 6:\n  - What are C0, C1, and C2? Are they cluster centers? C3 is not visualized?\n  - In CAFace plot, there are many blue points with high cosine similarity to the Gallery. The authors should visualize and analyze the differences between the images corresponding to these points and the red/yellow points that make them have different weights.\n- Which network layer was used to extract the intermediate feature maps M_i in the experiments, and what is the feature map size?\n- In Table 1, an empty cell can be either in a merged cell or means \"No\". It may cause confusion. The authors should improve this, e.g., use 'x' to denote 'No' instead of using an empty cell.\n- The norm embedding definition should be moved to the main text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OQJSo7hs2V", "reviewer": "Reviewer_FHWH", "summary": "In this paper, the authors introduce a feature fusion framework, namely Cluster and Aggregate (CAFace), for face recognition with a large probe set.\nCAFace consists of two main modules: Cluster Network (CN) and Aggregation Network (AGN). \nWhile the first module aims at grouping images into M clusters (based on their quality) and fusing the features within a cluster (i.e. based on their correlation to the centers), the second modules aggregates features of all M clusters (i.e. different quality) into a single feature to represent the ID.\nThe proposed approach is validated on IJB-B, IJB-C, and IJB-S.", "questions": "1. For the cluster network (line 153-155), Multihead self-attention is adopted with the attention matrix of (N + M) x (N + M). As N could be very large, how efficiency is this design? \n\n2. Does the correlation of samples within a batch affect the assignment of a sample to one of M centers? If yes, the set element permutation may significantly affect the accuracy. Moreover, How is the contribution of the intra-set relationship in Aggregation Network?\nIn experiment of Table 3, CAFace seems to be robust to the order. How are the probe images shuffled in this experiment? In other words, does the set elements permutation differ significantly among runs?\n\n3. When M = 1, the accuracy partially indicates the contribution of Clustering Network and its attention as most of the fusion is taken place in this stage. However, as shown in Table 1, its accuracy is similar to Naive Average Fusion.\nHow effective is the attention learned by Clustering Network?\n\n4. For prior works such as PFE, CFAN, are they retrained with the same training set as CAFace? \n\n5. The experiment would be more complete with the analysis of the effect of N (i.e. from small to very large number of sample per ID to show the advantage of CAFace in comparison to prior works.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n- The paper is well-motivated.\n- The idea of two-step feature fusion (I.e. from N features to M intermediate cluster features and aggregate these intermediate features for a single fused output) is interesting. \nWhile the first step aims at fusing features within a cluster (i.e. similar image quality), the second step aggregate features of all clusters (i.e. different quality).\n- Experimental results show improvements against prior works.\n\nWeaknesses\nAlthough the paper is well-motivated and experiments seem to be completed with ablation study, there are some concerns regarding the design and efficiency of the proposed approach. Please see the Questions section.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LRafArfhzpN", "reviewer": "Reviewer_uvJX", "summary": "The paper proposes a method for better aggregation of embeddings used to describe faces in a face recognition problem. For aggregation a pair of networks are used, named Cluster Network and Aggregation Network. The method is trained on WebFace4M and tested on IJB-B, IJB-C and IJB-S. ", "questions": "What does it mean:\n- \"The magnitude of P is an interpretable quantity showing the importance of each cluster during fusion\" \n- \"sequential inference with incremental averaging\" l180\n- \"sample orders\" from \"invariant to sample orders\" l184\n- \"invariance to the order of batch\" l190\n- \"identity,\" l197 \"person identity\" or the mathematical function of identity?", "limitations": "The paper addresses the face recognition theme, which is one of the topics in ML/CV that raised most ethical issues. It's main usage refers to retrieving the identity of a person from a low quality (potential surveillance camera)\n \nThe authors included a short discussion on potential negative societal impacts. Also, the paper clearly states that it avoided datasets that were withheld \"by their distributors due to privacy and other issues\".\n\nYet the later aspect is debatable: the paper did not used VGGFace-2 because it was upheld by VGG authors as being under investigation by their university legal department, but used WebFace4M dataset collected in the same conditions as VGG, just larger, but still publicly available.\n\nHowever, again, I must emphasize that this paper does not introduce any new dataset, and therefore has not broke any confidentiality.  ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - the idea to use a set of networks that approach different task for aggregation of embeddings is interesting\n- results better than SOTA; ablation inclusion\n\nWeaknesses:\n- the method is dedicated to FR problem, which is important but rather limited in auditorium in NIPS context\n - while the idea to use two different nets is appealing, the approach for each one of them is know. Clustering may be tracked to Center-Loss. ref 31 and 52 brought the idea of aggregation.\n - the method is very hard to follow. Overall the figures complement rather well the description, but the later rather un-inspiring. For instance the second part of eq (1) says nothing that was not explainable in plain text. eq(2-3-4) are combination of mathematical equations and acronyms and so on. Overall too many non-intuitive notations are gathered in a very condensed text so to easy to read.    \n\nOverall I see the paper as being potentially acceptable, but it needs a through revision of the method presentation to improve its readability.\n\nPost rebuttal: My questions and other raised questions have been answered, therefore I still view this paper as acceptable ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8OsRqLX0ze", "reviewer": "Reviewer_zymB", "summary": "This paper proposed a two-stage feature fusion paradigm, Cluster and Aggregate, that can both scale to large N and maintain the ability to perform sequential inference with order invariance. \n\nThe clustering stage is a linear assignment of N inputs to M global cluster centres, and the aggregation stage is a fusion over M clustered features. The clustered features play an integral role when the inputs are sequential as they can serve as a summarization of past features. \n\nBy leveraging the order-invariance of incremental averaging operation, this paper designed an update rule that achieves batch-order invariance, which guarantees that the contributions of the early image in the sequence do not diminish as time steps increase.", "questions": "Please reply to my questions raised in the weaknesses:\n\n(1) Why IJB-C is not tested and reported?\n\n(2) Efficiency comparison is suggested to move to the main paper.\n\n(3) There are too many implementation details in Fig 3 and the motivations behind these designs are hard to follow.\n\n(4) Typo needs to be fixed and overlapped subscript needs to be changed.", "limitations": "The limitations and potential negative societal impact is discussed at the end of this paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) Feature fusion is a practical research topic for set-based face recognition.\n\n(2) Experiments on IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition.\n\n Weaknesses:\n\n(1) IJB-C is mentioned in this paper but only the performance on IJB-B is reported.\n(Line 227: IJB-C is an updated version of IJB-B with more complex motions in the video)\n\n(2)  Efficiency comparison is given in the supp, it is better to include it in the main paper as 'Intra-set calculation becomes infeasible when N is large or sequential for intra-set methods' is one of the main motivations of the proposed method.\n\n(3) The idea of clustering and aggregation is clear. However, the motivations to implement in this way (Fig 3) are hard to follow.\n\n(4) Typos in abstract \"diminish\"; M is the cluster number but also used in line 138 C_M.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept