Improving Federated Learning Face Recognition via Privacy-Agnostic Clusters
Authors: Qiang Meng
Feng Zhou
Hainan Ren
Tianshu Feng
Guochao Liu
Yuanqing Lin
Venue: ICLR 2022 Spotlight (2022)

Abstract: The growing public concerns on data privacy in face recognition can be partly relieved by the federated learning (FL) paradigm. However, conventional FL methods usually perform poorly  due to the particularity of the task, \textit{i.e.},  broadcasting class centers among clients is essential for recognition performances but leads to privacy leakage. To resolve the privacy-utility paradox, this work proposes PrivacyFace, a framework largely improves the federated learning face recognition via communicating auxiliary and privacy-agnostic information among clients. PrivacyFace mainly consists of two components: First, a practical Differentially Private Local Clustering (DPLC) mechanism is proposed to distill sanitized clusters from local class centers. Second, a consensus-aware recognition loss subsequently encourages global consensuses among clients, which ergo leads to more discriminative features. The proposed schemes are mathematically proved to be differential private, introduce a lightweight overhead as well as yield prominent performance boosts (\textit{e.g.}, +9.63\% and +10.26\% for TAR@FAR=1e-4 on IJB-B and IJB-C respectively). Extensive experiments and ablation studies on a large-scale dataset have demonstrated the efficacy and practicability of our method.  

Published as a conference paper at ICLR 2022
IMPROVING FEDERATED LEARNING FACE RECOGNI- CLUSTERS
TION VIA
Qiang Meng', Feng Zhou', Hainan Ren', Tianshu Feng", Guochao Liu', Yuanqing Lin Algorithm Research, Aibee Inc." Independent Researcher
ABSTRACT
The growing public concerns on data privacy in face recognition can be greatly addressed by the federated learning (FL.) paradigm. However, conventional FL methods perform poorly due to the uniqueness of the task: broadcasting class centers among clients is crucial for recognition performances but leads to privacy leakage. To resolve the privacy-utility paradox, this work proposes PrivacyFace, a framework largely improves the federated learning face recognition via commu- nicating auxiliary and privacy-agnostic information among clients. PrivacyFace mainly consists of two components: First, a practical Differentially Private Local Clustering (DPLC) mechanism is proposed to distill sanitized clusters from local class centers. Second, a consensus-awan recognition loss subsequently encour- ages global consensuses among clients, which ergo results in more discriminative features. The proposed framework is mathematically proved to be differentially private, introducing a lightweight overhead as well as yielding prominent per- formance boosts (e.g.. +9.63% and +10.26% for TAR FAR=1e-4 on IJB-B and IJBCC respectively). Extensive experiments and ablation studies on a large-scale dataset have demonstrated the efficacy and practicability of our method.
INTRODUCTION
Face recognition technique offers great benefits when used in right context, such as public safety, personal security and convenience. However, misuse of this technique is a concern as it involves unique and irrevocable biometric data. The rapid commercial applications based on face recognition and facial analysis techniques have stimulated global conversation on ethics, and have resulted in various actors from different countries issuing governance initiatives and guidelines. EU's Gen- eral Data Protection Regulation (GDPR) (Voigt Von dem Bussche (017)) California Consumer Privacy Act (CCP) and Illinois Personal Information Protection Act (IPI) enforces data protection "by design and by default" in the development of any new framework. On the nationwide "315 show" of year 2021, the China Central Television (CCTV) called out several well-known brands for illegal face collection without explicit user consent. As researchers, it is also our duty to prevent the leakage of sensitive information contained in public datasets widely used by the research com- munity. Therefore, faces in ImageNet (Deng et al., 2009) were recently all obfuscated (Yang et al., 2021) and a large face dataset called MS-Celeb- 1M (Guo et al. 2016) was pulled of the Internet.
In the wake of growing social consensus on data privacy, the field of face recognition calls for fundamental redesign about model training while preserving privacy. A potential solution is the paradigm called Federated Learning (FL) (McMahan et al. 2017a). Given C' clients with local datasets {D', D ... DC} as shown in Fig. Dp. FL decentralizes the training process by combining local models fine-tuned on each client's private data and thus hinders privacy breaches. Typical examples of these clients include personal devices containing photo collections of a few family members, worldworld scenarios such tourist attractions visited by tens of thousands of people. In most circumstances, we can safely assume very few classes would co-exist in two or more clients.
Despite the numerous FL-based applications in various domains (Kairouz et al. 2019 ranging from health to NLP, there are very little progress (Aggarwal et al. [2021] Bai et al.) 2021 Liu et al. 2021 in training face recognition models with FL schemes. Unlike other tasks, parameters of the last classifier for a face recognition model are crucial for recognition performance but strongly associated with privacy. These parameters can be regarded as mean embeddings of identities (Wang al. 2018

---

Published as a conference paper at ICLR 2022
local dataset
Seature estractor
train models
gether/ceploy cass Information
feature space
not shamable
differentall primncy
grther/oeploy feature extectors
overlppedd
server
server
DP
global feature distribation
client
client C
client
client C
(a)
(b)
Figure I: Under the federated setting, multiple clients communicate non-sensitive model parameters Φ² (ex- cluding the last fully connected layer W which are greatly tied to privacy) under the orchestration by central server. (a) Since W are kept locally in conventional FL updating. the embedding space could overlap for different classes during training. (b) In contrast, the proposed PrivacyFace framework learns an improved face embedding by aggregating discriminative embedding clusters that are proved to achieve diffcrential privacy.
Meng et al. 2021b [hhen et al., 2020) (also called as class centers), from where individual privacy could be spied out as studied by plenty of works (Kumar Jindal et al. 2018 Boddeti 2018 Mai eet al. 2020; Dusmanu et al.) 2021). That prevents the FL approach from broadcasting the whole model among clients and the central server, and consequently leads to conflicts in the aggregation of local updates. As depicted in the global feature distribution of Fig. [T both clients try to spread out their own classes in the same area (pointed by the arrow) of the normalized feature space. Thus, the training loss could oscillate achieve consensus given the sub-optimal solutions from multiple clients. On the other hand, a large hatch with sufficient negative classes is necessary to learn a discriminative embedding space for advanced face recognition algorithms. During the conventional FL updates, each class is only aware of local negative classes while those from other clients are untouchable. This further limits performances of FL approaches in face recognition.
The privacy-utility paradox motivates us to introduce PrivacyFace, framework improves feder- ated learning face recognition by broadcasting sanitized information of local class globally. In the framework, novel algorithm called Differentially Private Local Clustering (DPLC) first gener- ates privacy-agnostic clusters of class centers while any specific individual the cluster cannot be learned, irrespective of attacker's prior knowledge, information source and other holds. Recall that the privacy cost of a differential privacy scheme is propotional to the lz-sensitivity while inversely propotional to the query number. Our DPLC reaches low (z-sensitivity by restricting the cluster size as well as covering sufficient class centers. In addition, the number necessary centers to com- municate in DPLC is irrelevant to the number of classes in the training data. These characteristics jointly equip DPLC with much smaller privacy cost than the naive alternative, which sanitizes each class center individually by Gaussian noise. In experiments, DPLC's privacy cost is only 1.7e-7 of that of the naive approach. That persuasively reveals the high security level of our approach.
The second part of PrivacyFace is the consensus aware face recognition loss. Following principles of Federated Averaging (FedAvg) (McMahan et al. 2017a), a server iteratively gathers feature ex- tractors and privacy-agnostic clusters from clients, averages parameters of feature extractors and distributes them to clients. Accordingly, the consesus-aware loss notifies each client not to embed samples in the inappropriate zone (differential private clusters marked by DP) of the feature space during the local optimization, as shown in Fig. [b. This process aids each client to train more dis- criminative features as well as align all consensuses. Compared to the conventional approach, our PrivacyFace boosts performances by +9.63% and +10.26% for TAR @ MAR@FAR=Ic-4 respectively with only single-digit privacy cost. Moreover, the additional computational cost as well as communication cost negligible (e.g., the extra clusters to broadcast only occupy 16K stor- age while the backbone already takes 212M). In a word, PrivacyFace an efficient algorithm which improves conventional federated learning face recognition by large margin on performances, while requires little privacy cost as well as involves lightweight computational/commiicca overheads.
IJB-B and IJB-
2

---

Published as a conference paper at ICLR 2022
2 PRELIMINARIES
2.1 DEEP FACE RECOGNITION
Most of early works in deep face recognition rely on metric-learning based loss, including con- trastive loss (Chopra et al., 2005), triplet loss (Schroff' et al., 2015) and N-pair loss (Sohn) 2016) These methods are usually inefficient in training on large-scale datasets. One possible reason that their embedding spaces at each iteration are constructed only by a positive sample and limited negative ones. Therefore, the main body of research (Ranjan an et al. 2017, Liu al. 2017, Wang et al. 2018 Deng et al. 2019: Xu et al. 2021 Meng et al. 202 has focused on devising more effective classification--asses loss and achieved leading performances on a number of benchmarks.
Suppose that where each 24 is a face image and 34 E {1.... ,n} denotes its associated class label. Consider- ing feature extractor & generating the embedding f: ((x;) and classifier layer with weights W [1. ... Wn), a series of face recognition losses can be summarized as
given training dataset D with N face samples {I,, 3 of identities,
(dl W)
log eulwy, f:) Ej (m():)
The specific choices of the similarity functions u(w, f), v (w yield different variants, e.g.: v(w, f) = s cos 0 v(w, f) cos 0,
CosFace WWnn et al.
2018
u(w, f) = s (cos 0 - m), u(w, f) cos(O m),
(2)
ArcFace (Deng et al.
2019
(3)
where are hyper-parameters and 0 arccos w" s/||w² T fl]
is the angle between w and f.
2.2 DIFFERENTIAL PRIVACY
Differential Privacy (DP) (Dwork et al. 2006
well-established framework under which very
little about any specific individual can be learned in the process irrespective of the attacker's prior knowledge, information source and other holds (Feldman et al.) 2017] Nissim et al. (016)] Stemmer
& Kaplan 2018). We state the relevant definitions and theories (Dwork & Roth (014]) below.
Definition (Differential Privacy). A randomized algorithm M : X - y is (c. D)-DP if for every pair of neighboring datasets X. X' E X (i.e., X and X' differ in one row), and every possible oulput T E y the following equality holds: Pr(M(() E T S Pr(M(XX)) E T) + 5.
Here 6,6 2 are privacy loss parameters, which we consider a privacy guarantee meaningful 8 ( where n is the size of dataset.
Definition (l=-sensitivity). The le-siiiity of function f X + Rd is
22(f)= =
max
- ((x))-((')la>
(4)
X. X.X'Ex
X.X neighbors
Definition (Gaussian Mechanism). Suppose a function f X - Rd and La is the d-
dimensional identity matrix. The mechanism M(X) f(X) + oL) is (c.o)-DP if
Aull) / /21n(1 L2
Dwork & Lei (009)). Suppose M= (M, M2, M&) is a sequence of algorithms, where M, (eg, de)- differentially private, and the M, 's are potentially chosen sequentially and adaptively. Then M is C G, E/ 5x) de)-differettiallly private.
Definition 4 (Composition of Differentially Private Algorithms (Dwork & Roth 2014
PRIVACYFACE
This section details the PrivacyFace framework illustrated in Fig. 2 At its core is novel clus- tering algorithm that extracts non-sensitive yet informative knowledge about the local class distri- bution (Fig. 2a After the central server broadcasting local DP- ge-guaaatteed outputs (Fig, 2b). each client optimizes over a consensus-aware objective that takes into account both the local data and the privacy- clusters learn discriminative embedding space (Fig. 2c for face recognition.
3

---

Published as a conference paper at ICLR 2022
local cless centers
Busten from Jocher
local classes
(a) Local clustering.
(b) Gaussian mechanism.
(c) Optimization consensus.
Figure 2: Compared to conventional federated learning methods, PrivacyFace learns more discriminative fea- tures three additional steps in each client: (a) Find cluster with margin P and calculate the average of class centers covered in the cluster; (h) Perturb with Gaussian noise v, which makes oulputted be differentially private. (c) After the server gathering and distributing p, a consensus-aware face recognition loss enables each class to be separable with local negative classes as well as class clusters from other clients.
3.1 DIFFERENIALLY PRIVATE LOCAL CLUSTERING
Our target is to improve performances of federated learning face recognition by communicating auxiliary and privacy-agnostic information among clients. To distill useful information from each client and resolve the privacy-utility paradox, we propose a specialized algorithm called Differential Private Local Clustering (DPLC) with rigorous theoretical guarantees.
Problem Design. To facilitate the discussion, let us first introduce the margin parameter p E [0, which defines the boundary of cluster centered at p by saying that any w in the cluster satisfies arccos( w p) S We design the local clustering problem with two principles. First, the clustering results are expected to carry information only about population rather than individual from local dataset. Supported by Theorem 2 proved later, this principle ensures PrivacyFace to gain insight privacy of individual record. Second, the
about the underlying distribution, while preserving spaces confined by the clusters should be tight because they represent the inappropriate zones to escape from in other elients' perspectives. A cluster with large margin p would occupy a huge proportion of the sphere, leaving limited room for other embeddings (c.g., half of sphere would be occupied if 2/2 More specifically, we quantify the occupancy ratio by the following theorem:
Theorem 1. Assume there is unit and d-dimensional sphere Sd and an embedding point p E $d. Embeddings with angles less than p to p (i.e. {f arccos( p) S p}} occupy {=stm=(o) [d1
of the surface area of Sd. Here (a, b) is the regularized incomplete beta function.
Proved in Sec. A.1.3 of the appendix, Theo- rem inddiaaess that the occupancy ratio increases monotonically with respect the margin p given a feature dimension d. Fig. 3 plots series of functions between cluster margin p and occu- pancy ratio under different dimensions. Take the rightmost curve corresponding
Occupancyy Rabio
6-16
d-12
6-64
typical setting d = 512 in face recognition as an example. The occupancy ratios are 0.055, 10 10-5 5,-5,4-- 10-10 when = 1.5, 1.4, 1.3 respectively. In another word, the whole feature space would be fully occupied if we sample 20, 000 clusters over the original face classes when 1.4. It is obvious that the more space left for optimization, the higher likelihood achieve better performance. There- fore, we seek for margin-constrainn better privacy-utility trade-offs. A counter example to the latter principle is the classical k-means algorithm, which have been extended to differennially private versions in a number of works (Nissim al., 2016 Feldman et al. 2017 Stemmer Kaplan 2018). However, k-means naturally has no control about the scopes of the generated clusters, not to mention the difficult arper-parameter & to pick. That motivates us to propose specified clustering algorithm for our task.
e-258
8-512
0.0
02
0.4
0.6
0.8 3.0
1.2
14
1.6
Cluster Margin
Figure 3: Visualization the occupancy ratio curves.
clustering of local data with fixed < 1.4 when d = 512 for
We adopt cosine similarity by assuming all vectors lie unit sphere in face recognition.
4

---

Published as a conference paper at ICLR 2022
An Approximated Solution. Given the margin p. the problem of finding clusters to cover all class centers is generalization of the well-studied 2D disk partial covering problem which is unfortu- nately NP-complete (Xiao al. 2004). We take greedy approach by iteratively finding cluster centered at p" with margin to cover the most class centers wy and weeding out those covered before the next round. We cast the problem of identifying such cluster at each iteration as:
Definition 5 (Spherical Cap Majority Covering Problem). Denote 1(.) as an indicator function. Assuming unit d-dimensional sphere S and class centers {wi, W2, ... wn S", the target is to find a cluster center p" E &6 where p = arg max pESs Ei=1 1 (arccos( (w) p) S P)
Finding the optimal p' is also NP-complete. To address this, we first sort out the densest area S in terms of the neighbor count for each w. An efficient approximation p" then average of class centers in the area, p [ Eies w,. Lines 3-5 of Algorithm summarize the detailed procedure, by which the approximate cluster centers p; can be quickly enumerated depicted in Fig. 2a In line 8, the class centers would be further normalized in accordance with the DP mechanism.
Algorithm 1: Differentially Private Local Clustering (DPLC)
Data: Class center embeddings W [wi, W2, Wn encoded in the classifier layer. Parameters: Margin Minimum cluster size T; Maximum #queries Q; Privacy budget c, 5. Result: Privacy-agnostic cluster centers p's.
Let I {1, 21 .n) and calculate Oig arccos(w; wx) for E (1, .n];
2 for g = 1,2, .... do
For each {w,, 2 I}. find the indexes of its neighbors by S, {j 0, S s.j E I); Find a set with the most elements, i.e. S = arg max |
4
[S] Eies w;;
// approximate solution for problen
if S > T then
Sample from distribution (10, Ia) where
[Tp (1 cos In( 112
2
pio Pg p
I I/ (arccos(w," Tplla _ S p, IJ:
9
else
Break out of the loop;
Differential Privacy Endorsement. Although the centers reveal the population-level property of the local training set, they are still outcomes of a deterministic algorithm, thereby vulnerable to adversary attack during the FL updating. We prove below that p can be perturbed to achieve DP:
Theorem 2. Define function p - f(S) [s| Eies wi Then the Gaussian Mechanism p A
2 M(S) = f(S) + +O is (e, DP if s
cosccoc))))))) In 125
Proof. Let S be the set of indexes of class centers which have cosine similarities larger than cos p with respect to a center w., Assuming that S. are neighbors differed at and w', where vectors are normalized, i.e., ||||| = ||w01| [.r 1, then we have w w cos p and ur ww' 2 cos p.
Lemma 2 in appendix states that (w, w', ). Therefore, the lower bound of w w' is
w |
w=" w'
cos(arccos
T.w.ll w
w w
arccos
w w' Wal Te 1
for all
w" "' w Tw''
w w |
w w
|| 2 cos(arccos
arcoos
w
| w
2 cos(p = cos(2p)
Following this inequality, the 12 le-seniityv of f(S) is [S] /2 cos cos(2p)
[[{(S) - f(S')l2
[8] ||w w/ ||2 M(S)
| /2 2w T w/ I / 2 cos (2p). By Definition easily conclude that [S] Eies w. +0(0 o I1) is (c. () DD if
1
cos cos()p)) In 1.25
Additional proofs can be found in Sec. A.1.2 of appendix. Theorem outlines the setting of noise variance given the privacy budget e, and the cluster margin p. Basically, the lower bound of
5

---

Published as a conference paper at ICLR 2022
is proportional to [s]- That implies when the size of cluster |S] is large enough, adding small noise sufficient to achieve promising privacy level. We found setting minimum cluster size to T - min [S] 512 yields empirically stable performance. It is worth emphasizing that the number of original classes n does not directly influence the introduced noise a. This advantage ensures that DPLC maintains high utility when dealing with large-scale face recognition problems.
Full algorithm. Algorithm I presents the full DPLC and we highlight several key properties below. Properties
2 state that DPLC is efficient and differentially private in theory. Besides the efficency and privacy. another major concern of the DP algorithm is the utility. Specifically, as p is combi- nation of the source cluster center p and Gaussian noise u, we should prevent the noise a to have overwhelming effects to the output, which is further verified by our Property 3 and Property 4
Property 1. The complexity of the DPLC algorithm is O(Q - n²))
Proof. The computational complexity of calculating Oig is O(n²). In each loop, we also compare the value o Vi,j S with p. whose worst time complexity O Because remaining steps of linear complexity, the total complexity is On n²) O ((Q n²).
Property 2. DPLC is (Q 6, Q (2 -)diifferentially private algorithm.
Proof. The mechanism f(S) + o2 is (c. diifferrrrey private according to Lemma As DPLC queries at most results, it's easy conclude that DPLC algorithm is (Q e, Q differentially private based on Definition 4
Property 3. Denote & as the cumulative distribution function of a standard Gaussian distribution.
For vector magnitudes p, we have [pl| E (cos p. 1] and P( S r) ~ 4
d-1
02 /2/22(d-1)
Proof. For p. the upper bound is as |pl2 = l| उ Eies will2 < 3 Eies |10:122 = 1. As S
denotes the set of indexes of class centers which has cosine similarities larger than cos p with center wo, the lower bound of ||p||a can be calculated by
l|pll= = l|llll/woll2 2 p" w. = (5)
1 Σ
w., "
=
1 S
Σ FES
wi w. 2
1 [S] (1+ [1+(1S]- 1) cos p) > cos
VES
Therefore, we have ||p|| (cos p, 1).
For o ~ (0, let v = [01, 02. ... val. we have 5 following the standard normal distribution for all i. Thus, the sum of square of 2h follows di-distribution with - 1 degrees of freedom, i.e.,
~ xa-1- The mean and variance of || are E(] = o² - 1) and Var( [|]]]] 20 (d -dd-11 1), based on properties of the x²-distribution. By the central limit theorem,
1 || 2 = Ei= (%)
o2
1),
(d 11111,,, 1), (d - 1)) when d is large enough. In practice, for d > 50, ||01 is sufficiently close to No - 1),, 20" (d- 1)) (oxx et al. 1978). Therefore, we have
||lll converges
P([[u]aa Sr) = |1ll3 (d
,2 oz
+6 02
22 (d 1) (2-
Note that in our work, can be small value with little privacy cost. Thus, with high probability, the norm of is close zero.
Property 4. If ||l| 2 ||0||2 lz, the cosine similarity between p and is always greater than
/I
Proof. Denote magnitudes of p. as 1ps44 and cosine similarity between and v x (i.e.,
p-u
E 1), Let 1 1 and be the cosine similarity between p + v and p, then
iptul"
1taz The first order derivative of with respect V1+a2+2ar
1ptv 1p 1 122+2++2,,0
is ds dr
a² (a++) It's easy to conclude that s takes the smallest value when x aa, which can
2
be reached as a is always smaller than 1. In the end, we have s 2 VL
12/13. 1-

---

Published as a conference paper at ICLR 2022
In DPLC, reasonably require large local clusters S at line 6 while preserving privacy in terms of e, That constraint ensures the synthesized noise v to have a small magnitude (property 33. which thereby results in slight offsets of the sanitized vector P the source p (property 44. With these theoretical guarantees, DPLC can obtain good end-to-end utility with low privacy cost.
3.2 OPTIMIZATION WITH GLOBAL CONSENSUS
PrivacyFace employs Federated Learning (FL) paradigm for optimization. Suppose there are C clients, each of which c 1, ... C. owns training dataset De with N₂ images from Yh identi- ties. For client c, its backbone parameterized by of and the last classifier layer encodes class centers in W = [wf. normalized. Central to PrivacyFace is the DPLC algorithm, which generates for each client, Q clusters defined by the centers P {pp.
w 1. Without loss of generality. we assume that each class center w
PQQ } with margin As presented in Algorithm the training alternates the following updates in a sufficient number of rounds M.
Client-side. In line 3 of Algorithm 2 the server broadcasts current feature extractor Φ to each client. Then each client updates its local cluster centers pe that can be safely shared others in line 5. After receiving P = (PI.... PC), each client executes one-round optimization over the consensus- -wwrre aware face recognition loss on local dataset D in line 7:
( .15)
Σ log ul fi) + ーー f Ek= Ede eH(pf (f--))
where fi is the feature extracted by & on i-th instance in client e. p(p, J p. 0}) computes the similarity between f and the cluster centered P with margin p. As illustrated in Fig. 2c PrivacyFace aims to learn globally consistent embedding that can not only classify local classes, but also achieve consensuses with other clients on the incompatible clusters p.
cos cos(maxx(
Server-side. Similar to other FL approaches, a central server orchestrates the training process and receives the contributions of all clients to the new feature extractor at line 8. The well-known Fe- dAvg (McMahan et al.) 2017a) is then utilized to compute an average of all local models. Compared to the conventional FL method as described Fig. [T. PrivacyFace empirically achieves better con- vergence thanks to the consensus-aware loss that implicitly takes the data distribution of other clients into account. Built on the client-wise DPLC algorithm, the framework potentially compatible with other optimizer, e.g., FedSGD (Shokri & Shmatikov 2015) as shown in Sec. A.2. of appendix.
Algorithm 2: The PrivacyFace Training Scheme.
Data: Des exclusively owned by each of the C' clients; A pre-trained extractor do- Parameters: -PLLC-related p; T,Q; Privacy budget e, 6; Maximum #communications M. Result: A general recognition model d...
for t = 1..., .MM do
for cach client 11 C do
3
Synchronize the local extractor Φi-1 = Φ.-1 1 as the up-to-date one in server;
Generate privacy-agnostic pe = (1i. Pe, ... Dúg via Algorithm [
4
5
The server gathers p and distributes p = (p1 p2, po) to all clients; for each client 1.... do
7
Update local model { W } to {Øf, W by optimizing the loss (Eg. 5 Communicate the feature extractor of to the server while keeping W. locally;
8
The server updates the model by Φ: = E. &8
4 EXPERIMENTS
This section together with the appendix describes extensive experiments on challenging benchmarks to illustrate the superiority of PrivacyFace in training face recognition with privacy guarantee.
Datasets. CASIA--ebFaae eYi al. 2014 contains 0.5M images from 10K celebrities and serves the dataset for pre-training. BUPT Balanceeffeeee (Wang & Deng. 2020), which comprises four
7

---

Published as a conference paper at ICLR 2022
P- 5-
6-50 datty S4-273
p-1.5 2-222
30, -177
high utiltty
202-
+-3. oeer (5-04)
High zone
high ohee
011
Coatee
Dosine Siitarty
Doline Simlartt
(a) = 1,Q= =1
(b) p 1.3, Q
(c) p = 1.3, Q = 3
Figure 4: Distributions of cosine similarities of p and under different parameters with 1000 runs.
sub-datasets categorized by racial labels (including African, Asian, Caucasian and Indian) and each sub-dataset contains 7K classes and 0.3M images, is used to simulate the federated setting. We adopt RFW (Wang et al. 2019), IJB-B (Whitelam al. 2017) and IJB-C (Maze et al. 2018) for evaluation. RFW is proposed to study racial bias and shares the same racial taxonomy as BUPT-Balancedface JJBB and IJB-C are challenging ones, containing 1.8K and 3.5K subjects respectively from large- volume wiid images/videos. All images are aligned to 112 x 112 based on five landmarks.
Training. We assign one client for each of the four sub-datasets and use the perfect federated setting (i.e. no client goes offline during training). For the pre- pre-trained model Φb. we adopt an open- source one trained on MASIA-WebFace which builds on ResNet18 8 and extracts 512-d features. We finetune Φn by SGD for M 10 communication rounds 0.001, batch size 512 and weight decay 5e-4. For reproducibility and fair comparison, all models trained on 8 1080Ti GPUs with fixed seed. To alleviate the large domain gaps across sub- datasets, we build lightweight public dataset with the first 100 classes from EASIA-WebFaad finetune local models before gathered by the server. Unless stated otherwise, the parameters for PrivacyFace are default to T = 512, Q = 1, 1.3 and = 1. As the parameter 5 is required to be O [ Dwork & Roth 2014). we set drlia 5e-5 in BUPT- AUPT-Balancedface.
HUPT-Balancedfaal
with learning rate
to
Baselines. In the absence of related methods, we compare PrivacyFace with or without Gaussian noise added, and against a conventional FL method as indicated in Fig. [a. We denote these methods as + Φ + p and & respectively, based on communicated elements among the server and clients. Besides, we implement centralized training on the global version of BUPP-Balancedfacee and denote the trained model as "global training". The model is also finetuned from Φo for 10 epochs with learning rate of 0.001 and serves an upper bound in all experiments.
Privacy-Utility Trade-Offs of the DPLC Algorithm. The goal of DPLC is to broadcast discrim- inative yet
knowledge through a federated network among clients. This part investigates its effectiveness in terms of similarities between the noise-free center and the per- turbed one P by the Gaussian Mechanism with respect to different parameters. We first derive the class centers W from the Caucasian sub-dataset of BUPT- Balancedface by Φu. Taking W as input, DPLC generates 1000 p's for each p. yielding the distribution of cosine similarities shown in Fig. As common sense, two face features holding cosine similarity over 0.7 are recognized as from the same identity with high probability. Fig. 44 reveals that p > 1.2 can always lead to accurate ò with 1 and Q 1. If {p. Q) = {1.3, 1]. conclude that p. p similar with privacy cost over 0.3, as indicated by Fig. 4bb Although Q = 1 leads to good recognition performance, Fig. [4c studies the effect when more queries are required (Q 3) at different cost c. At p = 1.3, we can generate three clusters with descending sizes (1773, 743, 581) w.r.t the query index. By choosing a strict privacy cost 0.33, queries 2 and are too noisy to carry useful information. Setting e to reasonable privacy level, however, all queries will convey accurate descriptions of features.
Ablation Studies on Hyper-parameters, We conduct several ablation studies on PrivacyFace and present recognition performances on IJB-C in Fig. 5 Models are trained by ArcFace with default parameters not specifically stated. In Fig. 5a poor recognition performance and unstable training process can be observed for conventional FL method & mainly due to the insufficient number of local classes and inconsistent stationary points achieved by different clients. These drawbacks can be significantly relieved by PrivacyFace & + p. Fig. 5b reveals that performances of PrivacyFace improves as privacy cost e increases. The performance is nearly saturated when e > 0.3, closing to
8

---

Published as a conference paper at ICLR 2022
the method Φ without privacy protection. This end-to-end evidence indicates DPLC achieves high privacy-utility trade-offs as analyzed in Fig. 4b] The effect of cluster margin is explored in Fig. and the optimal p around 1.3. A small p leads to over weer-fined clusters with inadequate class centers, adding requirements to increase noise and hurt recognition. Alternatively, the performance would drop by adopting large p which generates trivial clusters with high occupancy ratio.
TAR@FAR-1e on UB-C (%)
TARSFAR-1e-- on aja-c (%)
72
70
70
21
68
%
60
S
55
%
+
0.2
0.4
0.5
0.8
1.2
1.4
1.15
1.20
1.25
1.30
1.35
1.40
Training epoch
Privacy
Cluster
(a) E 1,p 1.3
(b) = 1.3, varies
(c) = 1, varies
Figure 5: Effects of pypar-parameters PrivacyFace performances.
Performances on Benchmarks. Tab. presents verification performances on various benchmarks. By finetuning on BUPT- BulP--Balncedffcee Φ, Φ+D and o achieve performance boosts. Compared to the conventional FL method (Φ), performances of our PrivacyFace (Φ + p) are consistently higher regardless of which training loss used. Specifically, improvements on TAR FAR=lee4 on IJB-B and IJB-C are 3.54% and 3.56% with CosFace, and 9.63% and 10.26% with ArcFace, which are significant and demonstrate the superiority of our method. also observe that & + á and & + p achieve very close results in all benchmarks. That implies the robustness of the proposed method.
Additional experiments can be found in appendix. We implement FedSGD (Shokri Shmatikov) 2015) in A.2. show the scalability of the PrivacyFace. Sec. compares the DPLC with a naive approach while Sec. A.2.3 discusses the necessity of FL methods in federated setting. In Sec. A.2.4 analyze experimental attacks to further verify privacy guarantees of our framework.
Table I: Verification performances (%) on various benchmarks.
RFW
IJB-B
UB-C
Loss
Method
African Asian Caucasian Indian TAR@FAR= TAR@FAR=Ie4
d 81.08
82.13
89.13
86.55
5.94
8.79
Φ
83.40
83,38
89.62
87.23
68.09
70.16
Φ
83,50
83,38
89.93
87.28
71.63 (43,54)
73.72 (43,56)
CosFace
Φ p
83.50
83.47
89.95
87.27
71.62
73.73
Global Training
86.30
84.58
91.48
88.92
77.35
83.20
83.50
83.08
90.26
87.32
58.62
60.98
83.80
83.08
90.32
87.38
68.25 (49.63)
71.24 (+10.26)
ArcFace
Ø p
83.82
82.97
87.38
68.57
71.66
Globall Training
87.32
84.58
92.03
88.90
71.26
79.74
Cost Analysis. PrivacyFace introduces little computational cost thanks to the efficient DPLC algo- rithm as well as the consensus-awas loss. Apart from the backbone (over 200M) to distribute as in the conventional FI., the extra variables to communicate are p's, which only occupy about 16K storage. Thus, additional communication cost is negligible. Moreover, the total privacy cost still of low level with number Me 10 (i.e., communication rounds times the cost for each round).
CONCLUSIONS
With the carefully designed DPLC algorithm and novel consensus recognition loss, we im- prove federated learning performances on face recognition by communicating auxiliary embedding centers among clients, while achieving rigorous differential privacy. The framework runs efficiently with lightweight extended to other metric-learning tasks such re-identification and image retrieval. In the future, more efforts can be spent on designing a more accurate clustering algorithm in conjunction with the FL optimization, e.g., adaptive querying the confusion instead of brute-force sampling.
communication/compuaaiioo
overheads. Besides, PrivacyFace can be potentially

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Sec. 3.1 provides key proofs as well properties of the proposed method. Additional proofs are described in Sec. Sf the appendix.
The involved training/test datasets and training configurations are detailed in Sec. 4 when using FedAvg scheme. Similar settings are applied for FedSGD Sec. A.2.1 in appendix. Sec. |A.2.4oof the appendix shows visualizations for two potential attacks: K-nearest neighbor attack and inversion attack. We as well present involved datasets, network structures, training losses as well as training schedules for these attacks. Those are sufficient for reproducibility.
REFERENCES
California consumer privacy act.
Illinois personal information protection act. https://bit.ll/3cPJeRRl.
Divyansh Aggarwal, Jiayu Zhou, and Anil K Jain. FedFace: Collaborative learning of face recogni- tion model. arXiv preprint arXiv:2104.0300 2021.
Fan Bai, Jiaxiang Wu, Pengcheng Shen, Shaoxin Li, and Shuigeng Zhou. Federated face recognition. arXiv preprint :xXi:21055.02501 2021.
Vishnu Naresh Boddeti. Secure face matching using fully homomorphic encryption. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10. IEEE, 2018.
George EP Box, William H Hunter, and Stuart Hunter. Statistics for experimenters, volume 664. John Wiley and sons New York, 1978.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In IEEE Conference on Computer Vision and Pattern Recognition, volume pp. 539-546. IEEE, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 16990-499,,. 2019.
Mihai Dusmanu, Johannes L Schonberger, Sudipta N Sinha, and Marc Pollefeys. Privacy-preserving image features via adversarial affine subspace embeddings. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14267-14277, 2021.
Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pp. 371-380, 2009.
Cynthia Dwork and Aarons Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211 1--07, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265 284. Springer, 2006.
Dan Feldman, Chongyuan Xiang, Ruihao Zhu, and Daniela Rus. Coresets for differentially pri- vate k-means clustering and applications to privacy in mobile sensor networks. In ACM/IBEE International Conference on Information Processing Sensor Networks (IPSN), pp. 3-16. IEEE, 2017.
Robin Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint 1rxiv 2017.
10

---

Published as a conference paper at ICLR 2022
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. MS-Celeb- and benchmark for large-scale face recognition. In European Conference on Computer Vision, pp. 87-102. Springer, 2016.
C--.--eelM-M. 1M: A dataset
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, and Rachel Cummings. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.
Arun Kumar Jindal, Srinivas Chalamala, and Santosh Kumar Jami. Face template protection us- ing deep convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 462-470, 2018.
Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian Journal of Mathematics and Statistics, 4(1):66-70, 2011.
Chih-Ting Liu, Chien-Yi Wang. Shao-Yi Chien, and Shang-Hong Lai. FedFR: Joint opti- mization federated framework for generic and personalized face recognition. arXiv preprint arXiv:2112.12496, 2021.
Weiyang Liu. Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hy- persphere embedding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 212-220, 2017.
Guangcan Mai, Kai Cao, Xiangyuan Lan, and Pong C Yuen. SecureFace: Face template protection. IEEE Transactions on Information Forensics and Security, 16:262-277 2020.
Brianna Maze, Jocelyn Adams, James A Dunean, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, and Jordan Cheney. IARPA Janus benchmark-C: Face dataset and protocol. In International Conference Biometrics, pp. 158-165. IEEE, 2018.
Brendan MeMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efinn Intelligence and Statistics, pp. 1273-1282 2017a. 7a.
learning of deep networks from decentralized data. In Artificial
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv: 1710.06663,, 2017b.
Qiang Meng, Xiaqing Xu, Xiaobo Wang, Yang Qian, Yunxiao Qin, Zezheng Wang, Chenxu Zhao, Feng Zhou, and Zhen Lei. Poseface: Pose-invariant features and pose-adaptive loss for face recognition. arXiv preprint arXiv:2107.11721, 2021a.
Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. In International Conference on Computer Vision, 2021b.
Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. MagFace: A universal representation for face recognition and quality assessment. In IEEE Conference on Computer Vision and Pattern Recognition, 2021c
Kobbi Nissim, Uri Stemmer, and Salil Vadhan. Locating small cluster privately. In ACM SJGMOD-SIGACT-SN Symposium on Principles of Database Systems, pp. 413-427, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0666 1.00444,,, 2015.
Rajeev Ranjan, Carlos Castillo, and Rama Chellappa. L2-constrained softmax loss for discrimi- native face verification. arXiv preprint LtXiv:1703.099550 2017.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 815-823, 2015.
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward- collpable repre- sentation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6368- 2020.
11

---

Published as a conference paper at ICLR 2022
Reza Shokri and Vitaly Shmatikov. Srivacy-preserving deep learning. In Proceedings of the 22nd
ACM SIGSAC conference on computer and communications security, pp. 1310-1321, 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Annual Conference on Neural Information Processing Systems, pp. 1857-1865, 2016.
Uri Stemmer and Haim Kaplan. Differentially private k-means with constant multiplicative error. In Annual Conference on Neural Information Processing Systems, 2018.
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and federated learning. In Proceedings of the 12th ACM Workshop Artificial Intelligence and Security, pp- 2019.
Yi Zhou. A hybrid approach to
Paul Voigt and Axel Von dem Bussche. The EU general data protection regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10:3152676, 2017.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and
Wei Liu. CosFace: Large margin cosine loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5265-5274, 2018
Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness- aware reinforce- ment learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9322-9331, 2020.
Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang. Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In IEEE Conference on Computer Vision and Pattern Recogmition, pp. 692-702, 2019.
Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K Jain, James A Duncan, and Kristen Allen. IARPA Janus benchmark-B face dataset. In IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 90-98, 2017.
Bin Xiao, Jiannong Cao, Qingfeng Zhuge, Yi He, and Edwin H.-M Sha. Approximation algo- rithms design for disk partial covering problem. In 7th International Symposium on Parallel Architectures, Algorithms and Networks, pp. 104-109, 2004.
Xiaqing Xu, Qiang Meng, Yunxiao Qin, Jianzhu Guo, Chenxu Zhao Feng Zhou, and Zhen Lei. Searching for alignment in face recognition. In AAAI Conference on Artificial Intelligence, volume 35, pp. 3065-3073, 2021.
Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfusca- tion in ImageNet. arXiv preprint arXiv:2103.06191 2021.
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. Learning face representation from scratch, arXiv preprint arXiv: 141 1. ,7923, 2014.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 MATHEMATICAL PROOFS
In this section, we present extra proofs for PrivacyFace.
A.1.1 PROOF FOR THEOREM
Proof. A spherical cap is a portion of a d-dimensional sphere with radius cut off by a plane. As shown in (L. 2011), the surface area of a spherical cap is
(d-11) 2 2
Aar ((2hk-
= 2
Here h is the height of the cap, and A, Ad are the surface area of the spherical cap and the sphere, respectively. Because the set {f arccos( p) I p.f Sd) is the spherical cap with height h = r. (1 cos p) and the radius is in our problem, it can be derived that the occupancy ratio is
A
d d-
2 Aar
[[22) (1 p)2)/r2
A4
d-1 1
2 Aan
12-22005-1-oo 200ep)
2 2
(d-1
(p)
'2
A.1.2 A WEAK VERSION OF THEOREM
In this part, we present straightfward but weak version of the Theorem 2
Theorem 3. Define function p - f(S) [s| Eies w; Then the Gaussian Mechanism A
M(S)
1 [S] w.
W. AN La))) ( (c. S v(L - cos p) In( 1.25 (1-
8
Proof. S stores indexes of class centers which have cosine similarities larger than cos with a center (denoted as w, in this proof). Assuming that S,S' are neighbors differed at w, w', then
[If(S) - f(S')l2 w
w' w'll2
1 (I woll2 + l|w' -wolk)) S
2 S
/2 coos
(6)
S
According to Definition
we conclude that M(S)
1 [8] Eies w. + +N is (c,5)-DP if
A2(
2 21
1.25
[e| /
cos p) In 1.25
This lower bound of is S/c (1 cos p) In L25 ).
which is larger than our final bound because
4
(1 - cos p) In( 1:23 1)² -
1.25
- V (1- cos(2p)) (1oco((2))) In( 1.25 1)²
2
S -
4
1.25
In(
(4 - 4 cos p - 1 cos(2p)
1.25
In(
(3 4cos p 20o2 p 1)
|SF
2
4
1.25
In(
(2 2 200 p)3 0.
S
tighter bound can reduce the privacy cost with same level of noise added. For example, the privacy cost in our experiments be dropped by around 21 % if using p 1.3 (from to 0.793).
13

---

Published as a conference paper at ICLR 2022
A.1.3 PROOF FOR THEOREM
Lemma 1. For any a, y, 2 R,, the following inequality holds:
(x _ az 2 x² zæ z) (y" yz y" "/=yy² 2) 2 (x" 22" y - x² yz 2)?
Proof. Let x = (1), X2, ra) (11,,/2, ,,,),, 2 = (21 "2,""" ;d). Then
d
d
xx 2z=z"
x² 2²=x" T z
Σ
Σ
2j
(=1
3=1
1=1
2
j=1
4
(2)2) m13))2s)
|=
(1)2) + 27 2m(z,azzzz)
2
d
(122) 2,
2
Similarly, it's easy to prove y" yz x zy=y""= z = 2 Ei=1 Σ3= (3iz) - y,,,), In addition, we
have
d
T y - 1 T x² zz
yz x
Mi Σ 2زز1 - Σ
E,y4 > 23
jm1
j=1
[1 - -.3))22())
2
1
(m:2y (2:05 112;)) + 2zzz (zz - -3zz))
2
Σ
(maz - xy)))( 2129))
2
Then we can deduce that
d ΣΣ>
(x" xxx T x x=" xx² x)===)==0=0)))))
(y" y²"
"²²y²" 2n²²
1
Σ
(x:t) xzz1)²
(gr22 U)2;)²
2
2
i=1
1=1 1=1
(2i2) 2,22) y124))
=( zz
Here the second step is the Cauchy-Schwand inequality.
Lemma 2. Assume that w. w', w. E Rdd and
COS 02
1 w
w''
cos 3,0 S a, 0,
T. Then "Tw! [
2 cos(a (3) always holds.
14

---

Published as a conference paper at ICLR 2022
Proof. Let w nn' 1 wllla
cos We thereby need to prove cos? 2 cos(a + () cos cos B
sin a sin 3 008 cos
v(1 cos² (1) cos² 3), which is equivalent to
(1 cos a1 cos B) 2 cos cos , 0o6 7
(7)
If cos cos 3 - cos y, the inequality equation 7 holds.
Cos a cos B > cos 7/ then
(1 - cos a:) (1
cos² (3) 2 (cos a cos / - cos co27)2
w To u
- (1 w uu
w w w w'
wF un'
(1
)²) 2 (
|10l 1lll//ll2> || )2
||woll21100 / l| l/ (ww.lallww' w/ll
([llwollal1o 12)² 2 - (w: w) )
(11er, ww ||2|| 20 [2)² (w, w "²)²) 2 (ww"" w) (wo w') w2 "o' 1100011))
(w" www w. w" w.w" w.) (w" w " w. w" "ow"" wo) 2 (w" wows w' w w w wo)
The last step is correct according to Lemma
A.2
EXTRA EXPERIMENTS
A.2.1 PRIVACYFACE WITH FEDSGD
Algorithm is mainly built on FedAvg (McMahan et al. (017a) where model parameters are com- municated between clients and the server. In this part, we show that our PrivacyFace is also compat- ible with another popular FL. method called FedSGD (Shokri & Shmatikov 2015). FedSGD mainly differs FedAvg in two aspects: (1) During training locally, the model is frozen to ensure gradients across mini-batches are from same model. (2) Gradients rather than model parameters are aggre- gated and distributed among clients and the server. Apart from changing FedAvg to FedSGD, we let other training setting consistent with those described before and present the results in Tab. 2 Com- pared to the conventional FL method Φ. performances of our PrivacyFace Φ + P are close those on RFW and noteworthily higher in very close results. All these phenomenons are in accord with those observed when using FedAvg, which shows the scalability of the proposed PrivacyFace.
benchmarks. Besides, ø + P and Φ + p achieve
However, FedSGD does not achieve that significant improvements as FedAvg- possible reason is that FedSGD naturally requires more communication rounds than FedAvg to achieve comparable re- sults, which may be caused by low-frequent updates during training. Experiments in McMahan et al. (2077)) showed that FedSGD requires about 23 times more communication rounds to achieve sim- ilar performances with FedAvg. With such a fact, the improvements using FedSGD in PrivacyFace still remarkable.
Table 2: Verification performances (%) on various benchmarks with FedSGD and ArcFace.
RFW
DB-B
JBB-C
Method
Alrican Asian Caucasian Indian TAR 1e TAR@
Φ 81,08
82.13
89.13
86.55
5.94
8.79
&
81.40
82.38
89.08
86.68
63.24
68.35
Φ p
81.40
82.35
89.12
86.77
63.72(+0.49)
68.73(+0,38)
Φ +
81.40
82,35
89.12
86.77
63.72
68.73
Global Training
87.32
84.55
92.03
88.90
71.26
79.74
A.2.2 COMPARISON WITH NAIVE DP APPROACH
naive alternative to DPLC is to directly make each class center of W be differential private by adding noise. However, that can lead to a large privacy cost due to two reasons:
1. With the same noise added, the privacy cost is propotional to the lz-sensitivity based on definition 2 The lz-sensitivity of the naive approach is 2 as maxinum distances between two class centers distributed in unit sphere are 2. In contrast, the lz-sensitivity DPLC is
T V22 2 cos(2pp. In our experiment, we let |S| > 512 and p = 1.4, which makes the dz-eensitvvity to be smaller than 0.0024.
15

---

Published as a conference paper at ICLR 2022
2. The training dataset are of large number of identities. According to the composition rule (Definition 3) the privacy cost grows linearly with respect to the queries (i.e., number of identities). In DPLC, low privacy cost as the clustering mechanism highly reduce the number of queries.
During training, the number of query is 1. Therefore, the privacy cost in an local client (with 7000 classes) of DPL.C is 1-0.021 1.7e - 7 of that of the naive approach.
7000-2
We experimentally compare the performances of our method and the naive approach. For fair com- parisons, the privacy cost in each round is fixed to be and all training settings are consistent with those in the main text. Tab. are the results, where the & + W represents the naive approach. It can be seen that Φ + W can achieve similar results on RFW benchmark as learning local discriminative features with introduced fixed class centers are relatively easy (for example, if introduced class cen- ter distributed in the south of the embedding space, placing local features to the north is a solution). However, as the class centers from other clients are very noisy, the global embedding space will become a mess. & + W even achieves much worse results than the conventional FL methods Φ on the challenging benchmarks. The tar decreases 23.99% on IJB-B and 25.16% on IJBC when far is le-4. In contrast, PrivacyFace increases the performances by 9.63% and 10.26% respectively, which shows the superiority of our framework.
Table 3: Verification performances (%) on various benchmarks.
RFW
UB-B
UB-C
Method
African Asian Caucasian Indian & TAR@FAR=Ie-
83.50
83.08
90.26
87.32
58.62
60.98
Φ W
84.03
83,58
90.33
87.26
34.63 (-23.99)
35.82 (-25.16)
Φ
+
p
83,80
83.08
90.32
87.38
68.25 (49.63)
71.24 (+10.26)
A.2.3 NECESSITY OF FEDERATED LEARNING
Besides using federated learning, can also choose finetune client-expert models local datasets. We also implement this approach by training four independent models on sub-datasets of BUPT-Balancedface and test their performances tent with what we described before in Sec. 4 Specifically, we finetune each model from do by ArcFace for 10 epochs with learning rate 0.001, batch size 512 and weight decay 5e-4.
benchmarks. The training procedure is consis-
We present our results in Tab. 4 where the "African/Asian Caape/CCaaaaaanniiiiiiia means the model trained on the corresponding sub-dataset of BalT-Balaccddfcce. Models finetuned on African and Caucasian sub-dataset achieve satisfying results on local benchmarks while those on Asian and Indian have even worse performances than the initial model 6 Moreover, all these local models perform poorly on the IJB-B and IJB-C benchmarks, which indicates the lack of generality. To conclude, local training highly relies on the properties and qualities of local datasets, and can not lead to general models. In contrast, models trained by conventional FL method and PrivacyFace can achieve competitive performances on benchmarks, and therefore are more robust to various scenarios.
in the first column
Table 4: Verification performances (%) on various benchmarks.
RFW
UB-B
UB-C
Method African Asian Caucasian Indian TAR@FAR=Ic-
TAR@FAR=Ie-
81.08 82.13
89.13
86.55
5.94
8.79
African
87,53
8.56
11.26
Asian
64.08
0.30
0.40
Caucasian
91.87
53,51
$7.89
Indian
83.62
39.01
39.62
Conventional FL. 83,50
83.08
90.26
87.32
58.62
60.98
PrivacyFacc
83.80
83.08
90.32
87.38
68.25
71.24
A.2.4 VISUALLIZATION
In this section, we experimentally examine the privacy attacks to demonstrate that class centers lead to privacy leakage while our sanitized clusters are resistent to these attacks. We mainly consider two types of attacks: K-nearest neighbor attack and reversion attack.
16

---

Published as a conference paper at ICLR 2022
K-Nearest Neighbor Attack We assume that attacker identities and attacks the exposed class centers by finding their neighbors. To simulate the K-nearest neighbor attack, class centers sampled from each identity on a large dataset called MS1M-V2 (Deng et al. 2019). Fig. 6 presents the results. It be observed that the K-nearest neighbor attack successtully find faces of the corresponding identity from the class centers, which enables the attacker to know which identities contained the training dataset. The observation further verilies the fact that sharing class centers leads to privacy leakage in face recognition.
gallery of faces with enormous
extracted on the -UPT-Balancedface dataset and three faces
Figure 6: Real mean faces and the top-3 nearest faces of the corresponding mean class centers.
Reversion Attack We design reconstruction network (modified from DCGAN (Radford et al. 2015 as described Tab. 55 which projects a 512 dimensional embedding to face image. CASIA- WebFace dataset is used as the training dataset. Specifically, we extract features from images by the initial model Φp. With supervision of features and images, we train the network by Adam optimizer with L.1 loss, which penalizes the absolute value of pixel differences between reconstructed and real images. The learning rate initialized from 0.001 and divided by 10 every 10 epochs, and we stop the training at the 50th epoch. The weight decay is 5e and batch size is 512 during training.
Table 5: Details of the reconstruction network. Note that all the convtranspose2d layers use stride 2 and kernel
size X 4.
Layer type
input dim.
output dim..
[c BN
S12
8192
reshape
8192
[512, 4.
convtranspuse22 BN ReLU
[S12, 4,
[256. 8,
convtransposc2d + BN ReLUU
[256, 8,
[128, 16, 16]
convtransoose2d BN ReLU
[128, 16, 16)
[64, 32, 32]
conviranspuse22d + BN + ReLU
[64, 32, 32]
[32, 64, 64]
convtransposc2d + Sigmoid
64, 64]
[3. 128, 128]
We test our model on BUPT-Balancedde dataset and present reconstruction visualizations. In each image pair of Fig. 7 the left one the average of faces from one identity while the right one is the reconstructed image from the corresponding class center. It can be seen that reconstructed images reveal the real individual identity, which demonstrates that w carries human face privacy and therefore should be protected. Fig. 8 shows the reconstructions from and p under various privacy cost e. The left column is the images by p. which are much blurred than those from w and only the group properties can be identified. For example, the first image seems like African male while
17

---

Published as a conference paper at ICLR 2022
Figure Real mean faces (first image in each pair) and faces reconstructed from class centers w (second image in each pair).
p
0.1
0.3
0.5
0.7
1.0
1.3
1.5
Figure 8: Reconstructed faces from cluster centers and the differentially private ones P under different privacy cost e.
the second last very likely be Caucasian female. The output P from DPLC algorithm
18

---

Published as a conference paper at ICLR 2022
can further protect privacy one can hardly recognize face from the generated images, especially when e is small. The hardness of successful identification is negative related to e according to the visualization.
A.2.5 COMPARISONS WITH FEDFACE
FedFace (Aggarwal al., 2021) is recent work applying federated learning face recognition. In each communication round, the FedFace uploads all local class centers to the server and con- sequently utilizes the spreadout regularizer section, we adjust FedFace to fit our federated setting in the main text. Specifically, in each commu- nication round, each local client uploads its 7K class centers the server. The spreadout regularizer then computes distances of 588 million (7K X 7K 3 x 4) pairs of class centers and back propa- gates the model parameters once. The updated parameters are then distributed each the clients. However, directly computing all distances consumes 1,962M GPU memory, which is out of the capacity of our machine. Therefore, we sample 1K out of 7K class centers from each client for spreadout regularizer in our machine. For hyper-paramettess of the spreadout regularizer, the authors recommend loss weight A = 10 and margin v 0.9. For fair comparisons, we further run 18 experiments with À E {0.5, 1,5, 5, 10, 15) and v {0.3, 0.6, 0.9} and report the best results of FedFace.
separate class centers from different clients. In this
Table 6: Comparisons of FedFFace and PrivacyFace.
TAR@FAR= (%)
Privacy Cost
Additional cost per round Computation
Method
UB-B
UB-C
Communication
GPU memory
FedAvg
58.62
60.98
0
FedFace
59.40
63.25
00
14,000 vectors
High
1525M
PrivacyFace
68.25
71.24
vectors
Low
<
Tab.
summarizes the overall comparisons between FedFace and PrivacyFace. We observe that
Performance. FedFace only boosts the FedAvg by 0.82%, 2.27% on TAR@FAR=Ie-4 for IJB-B and IJB-C. In contrast, PrivacyFace achieves 9.63%, 10.26%. respectively. Besides the sampling strategy, the main reason can that FedFace only separates classes from different clients for one time (the spreadout regularizer) in communication round, while the model is locally updated 0.3 million times (number of local images) without considering the conflicts among clients. In contrast, our consensus-aware recogntion loss takes the conflicts into account for each local update.
Privacy cost. In each round, FedFace uploads 7, 000 original class centers to the server. Therefore, the privacy cost is 7000 x oo = oo. For PrivacyFace, the number is only from the outputted noisy cluster center in the client.
Additional communication cost. For each client in communication round, FedFace addi- tionally uploads 7, 000 512-d class centers to the server and sends back the updated class centers. In PrivacyFace, a local client uploads one 512-d cluster centers to the server and downloads 4 cluster centers from the server. The additional cost from FedFace is 7000+77000
2, 800 times of that from PrivacyFace.
1+4
Additional computational cost. The additional computations in FedFace is from the spreadout regularizer, which calculates 588 million pairs of 512-d features and updates the model parameters on the server side by one-time back propagation. That requires enor- mous computation in each round. In contrast, only extra items in the consensus-awan loss requires additional computation in PrivacyFace, which is of tiny volume.
Additional usage of GPU memory. Even though we sample 1, 000 out of 7, 000 clients to reduce GPU usage of the spreadout regularizer, FedFace still consumes 1, 525M extra GPU memories. One the other hand, the additional usage of PrivacyFace is negligible, as shown in Tab. 6
A.2.6 ABLATION STUDIES ON INITIAL MODELS
In this section, examine the effects of different initial models the proposed method. Besides the On we used before, we further adopt two intermediate checkpoints at epoch 20, when training on the CASIA-WebFace (Yi al. 2014) and denote them Φ1, , Φ2, respectively. One additional
19

---

Published as a conference paper at ICLR 2022
model Øy is trained MSIM-V2 (Deng et al. 2019) for 10 epochs with learning rate 0.1 and the ArcFace loss. Note that we do not follow the conventional training configuration where the learning rate is decreased stepwisely until 0.001. That's because such a configuration will lead to strong model that does not require finetuning on BUPT-BalancedFace (Wang & Deng 2020).
With other training settings same those in the main text, we finetune the models BUPT- BalancedFace (Wang & Deng 2020) with the ArcFace loss. Tab pressens the comparisons of plain FedAvg and PrivacyFace. On the LJB-B/UB-C benchmarks, PrivacyFace can consistently achieve better results than the plain Avd For TAR @ FAR=1e-4 on IJB-C, PrivacyFace can boost the per- formances by 10.26%. 12. 12.03% 3.43%, 22. 22. with the four initial models. It can be observed that improvements are affected by the initial model. For example, the poorest initial model Φ achieves the lowest improvements. The reason may be that the training loss are overwhelmed by separating local classes as initial features are not discriminative enough. Consequently, increasing inter-client distances is of the secondary importance to the training in this scenario. For the best initial model IJBB/C decreases when using federated learning. This is because the
23. the performances finetuning process also focuses on improving local performances as the global consensus is already reached. In this senario, FedAvg decreases the TAR @ FAR=1e-4 by more than 40% on the bench- marks. In contrast, PrivacyFace can highly alleviate the performance degradations by taking global consensus into account during training. Finetuing on BUPT-BalancedFacc mainly benefits the per- formances on the RFW benchmarks as these two datasets are both designed for racial bias. If the initial model is good enough, the finetuning procedure will focus more on fitting the racial issue instead of improving the general performances. This may be the reason why model Ø3 leads to the largest improvements.
Table 7: Verification performances (%) with different initial models.
Initial Model
RFW
DB--
DB-C
Method
African Asian Caucasian Indian FAR-FAR= TAR@FAR=Ic-4
db
81,08
82.13
89.13
86.55
5.94
8.79
FedAvg
83,50
83.08
90.26
87.32
58.62
60,98
PrivacyFacc
83,80
83.08
90.32
87.38
68.25 (+9.63)
71.24 (+10.26)
Φ1
81.37
82.13
89.22
86.75
2.04
2.56
Φ1
FodAvg
83.33
81.70
89.73
86.65
50.70
53.97
PrivacyFace
83.43
82.47
90.02
86,97
61.97
(+11.27)
66.00 (+12.03)
6b2
76.72
77.90
84.90
82.11
1.72
1.63
622
FedAAg
78.01
77.62
85.03
82,78
35.83
36.17
PrivacyFace
78.93
78.50
85.55
83.05
38.47 (+2.64)
39.6 (+3.43)
Φ FedAvg
dy
83.65
84.45
89.08
87.55
69.20
72.00
87.07
86.48
92.08
89.15
26.70
26.14
PrivacyFace
87.60
86.80
92.37
89.85
48.10
(+21.40)
48.14 (+22.00)
A.2.7 EFFECTS OF CLIENT DISCONNECTIONS
Previously we use perfect federated setting where all clients are available during training. In real- world applications, some clients may drop for certain rounds of communication due to network issues. We further simulate the case by randomly select one of the four clients and exclude it from the training for an offline probability in each communication round. For example, if we set the offline probability to 100%, there will be exactly one client offline per round. Note here we only block one client because the total number of clients is small. We keep other settings unchanged and compare performances of PrivacyFace and plain FedAvg on the IJB-C benchmark with different offline probability. We refer the offline policy as disconnecting patterns during training, and use the same offline policy for each pair of experiments on PrivacyFace and FedAvg for fair purpose. To reduce the randomness, we run each experiment with 3 different offline policies and report the mean TAR FAR=I.4.
Fig. 9 presents our results. When the offline probability is 25% and 50%, PrivacyFace improves the performances of federated learning by about 6%. The improvement decreases to 2.4% with 75% offline probability. When each round witnesses one random client offline, the performances of two methods are below 45% while the improvement brought by our approach is only 0.4%. That reveals the performance boosts of PrivacyFace can be influenced by the network conditions.
20

---

Published as a conference paper at ICLR 2022
70
model
65
Fed/vg
TAR@FAR- lB-88
PrivacyFace
60
55
50
45
25
50
100
offline probability
Figure Rffeett of client disconnections during training.
A.3 DIFFERENCES WITH DIFFERENTIAL PRIVATE FEDERATED LEARNING
In PrivacyFace, we utilize the differential privacy improve the performance of conventional feder- ated learning in face recognition. Our work basically distills differentially private information from clients to achieve that purpose. We notice that there are plenty of recent works on differential private federated learning (DP-FL) methods (Geyer et al. 2017 McMahan et al. 2017b and would like emphasize the differences:
Truex et al. 2019)
1. PrivacyFace and current DP-FL. methods aim solve different privacy leakages. As dif- ferent data points lead to different model updates, DP-FI. methods essentially add noise to model update prevent data from being inspected. Recall the model update is essentially the average of gradients, which has a small /2-sensitivity (definition 2 thanks to the tremen- dous number of data points. Then the protection relatively easy with the low privacy cost (propotional to the lz-sensitivity). In contrast, PrivacyFace aims to protects class centers in face recognition which are directly linked individual privacy. The problem is much harder these class centers are not naturally averaged like model updates and therefore leads to large privacy cost.
2. The targets to protect is different. Current DP-FL methods protect the data points in each local dataset. In contrast, PrivacyFace protects some of the model parameters (i.e., the classifier weights), which is the first work in federated learning to our best knowledge.
3. Current DP-FL methods theoretically decrease performances of federated learning as they use noisy updates. However, PrivacyFace distill differentially private information from each client. The extra information can broadcasted to improve the performances of feder- ated learning face recognition.
4. We can directly combine DP-FL method and PrivacyFace, which lead to a more secure framework protecting both data points well as class centers together.
21


Reviews:
[{"review_id": "gLk-Ud-ma2B", "reviewer": "Reviewer_ZPue", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) First and foremost, the motivation and threat model of the proposed scheme is not clear. For example, what is the exact privacy goal of a participant in federated face recognition? Is it to prevent other participants from learning about the faces/identities in the local dataset? If yes, how does differential privacy help in achieving this goal? Given the updated feature embedding model from the server AND the 'noisy' cluster centers of the weight vectors, isn't it fairly straight forward to reverse engineer the face images that correspond to each cluster?\n\n2) Secondly, the proposed scheme is likely to work if all the face images in the local dataset are somewhat similar and form a compact cluster. Moreover, the face images in the other local datasets must not have large overlap with the local face images in the feature space. In the absence of these constraints, it may be hard to reach global consensus when the participants have mixed datasets (a more realistic setting). This may be the reason the experiments are conducted based on each participant having samples only from a single race.\n\n3) The paper appears to analyze the differential privacy guarantees of revealing the cluster centers and the model parameter updates of the feature embedding separately. However, these two strands of information may be used in conjunction to develop more sophisticated attacks.\n\n4) The experimental validation is weak. The proposed approach has been compared only against a centrally trained model and the proposed model without differential privacy. It has not been benchmarked against recent works on federated face recognition such as:\n\nD. Aggarwal, J. Zhou and A. K. Jain, \"FedFace: Collaborative Learning of Face Recognition Model\", International Joint Conference on Biometrics (IJCB) 2021", "summary_of_the_paper": "The main contribution of the paper is a slight modification of the standard federated learning (FL) framework for the purpose of improving face recognition performance. Since face recognition involves learning a deep neural network for feature embedding and the weight vectors for mapping the feature embedding to an identity, the standard FL framework (e.g. FedAvg) that updates only the feature embedding network does not suffice. Hence, the paper proposes to cluster the weight vectors of each participant and transmit the cluster centers (albeit with Gaussian noise to ensure differential privacy) to the other participants. Once the cluster centers of other participants are known, the local weight vectors can be learned to avoid overlap with those received clusters.", "summary_of_the_review": "The motivation and threat model of the proposed work are not clear. The proposed approach appears to have some practical limitations, which have not been considered. The privacy analysis is not very robust and the experimental validation is also weak. \n\nDuring the rebuttal process, concerns 2 and 4 have been addressed to a great extent. Concern 1 still remains a theoretical possibility, but given that I do not have strong evidence to disprove the claims in the paper, it is fair to give the benefit of doubt to the authors. Concern 3 is still a very valid concern, but has been dismissed lightly in the rebuttal. Overall, I would like to upgrade my rating by one level based on all the discussion.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZKYad8x_I7v", "reviewer": "Reviewer_ytHU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes a novel framework to train a face recognition network for the federated learning setting.\nThe points of the strength for the work are listed as follows:\n\n1. the proposed differentially private local clustering mechanism with solid theoretical guarantees allows to securelly share the information of class centers of a local client to other clients. \n2. In addition, the consensus-aware recognition loss encourages the global consensuses for the learned face embeddings among clients and significantly improves the performance as compared with the baselines. \n\nThe points of the weakness:\n\n1. some parameters for the experiments are missing. I could not find the number of clients participating in the training. In addition, is the data distributed in iid or non-iid way. How many clients (the percentage) do participate in the training for each communication round? The details of these setting and other experimental results and discussion are totally missing from the paper.\n2. It seems that the training requires to start with a pretrained face recognition model using some large-scale dataset. However, for the current face recognition models, most of them can be trained from scratch even using the CASIA-WebFace instead of using the MS1M dataset. The bad initial class centers could still lead to the issue as shown in Fig. 1(a) where the embedding space could still overlap. The studies of different initial pretrained models \\phi_{0}'s are missing to show if the proposed method is senstivie to different initial models.", "summary_of_the_paper": "This work proposes a novel framework to train a face recognition network for the federated learning setting. The proposed method leverages differentially private local clustering mechanism to allows to securelly share the information of class centers of a local client to other clients. \n In addition, the consensus-aware recognition loss encourages the global consensuses for the learned face embeddings among clients and significantly improves the performance as compared with the plain federated learning baselines. ", "summary_of_the_review": "I think the idea is novel and there are many solid theoretical proofs for differentially private guarantee to secure the information exchange among clients for model training. However, some important parts for federated learning or face recognition are still not well explained in the current paper. Since the federated learning is to train the model distributedly, some clients may drop for certain round of communication due to network issues. However, from algorithm 2, the experimental results are mainly done in the perfect setting that each round, all clients participate in the training and share their clusters to others through the server. It can be imagined that if some clusters do not participate in the training for a certain round, the consensus-aware recognition may fail and still result in the same situation as shown in Fig. 1(a). Thus, It is worth a study how it  would influence the final performance.\n\nDoes the author try different initial models to start training? Does the proposed framework require a strong face model to begin with (trained with a larger dataset and finetuned onto a smaller or similiar-size dataset)? Is it possible to start from scratch or a pretrained model with smaller dataset (CASIA-WebFace) and then to finetune onto a larger dataset (MS1M)?\n\nCan the model scale to train with MS1M or deepGlinit or recent released WebFace260M which contains much more subjects (class centers) than the CASIA-WebFace (10575) and BUPTBalancedFace? \n\nI will adjust my rating according to the feedback from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ne_RmE23UfK", "reviewer": "Reviewer_7nj6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The proposed method can well address the issue of privacy leakage.\n+ The face recognition performance benefits from the supervision combination of local data and online cluster.\n- We doubt whether the gaussian perturbation is able to cover the privacy of facial ID. How to balance the privacy preserving and the training effect through tuning the extent of the gaussian perturbation? This is not clear in the paper.\n- The urgency of the problem itself is not solid to me. Even though the value of $W^c$ is leaked, the ID information cannot be easily recovered without the remaining part of the neural network. \n", "summary_of_the_paper": "The aim of this paper is to tackle the privacy leakage issue when using federated learning technique to train deep face recognition models. The naïve practice, broadcasting the last fully connected layer, results in the leakage of ID features. But if we cut off the broadcast of $W^c$, it will cause the issue of overlapping IDs among the clients, which harms the training of face recognition model. Therefore, the authors propose their method, DPLC, to tackle this problem. The idea is to conduct online clustering, gaussian perturbation, combined supervision of cluster and local data.", "summary_of_the_review": "Based on the primary opinion above, I will temporarily give the ranking of “marginally below the acceptance threshold”. The final rating will depend on the authors’ feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Kv9sCHsDZAr", "reviewer": "Reviewer_VU44", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I think this is an impactful work. It aims to solve a practical and challenging task: FL for face recognition (FR). To my knowledge, in industry, the face recognition companies have not (at least, not widely) deployed the edge/distributed learning for face recognition products. Thus, this is a poineering work. The authors seem understand both FL and FR in depth, the proposed solution can meet the requirements of differetiable priacy and improve the face recognition performance in the edge/distributed learning scenarios.\n2. I am from the filed of FR rather than FL. I did not check very carefully the proof and claims of the solutions meet the requirements of differential priacy. I just quickly went through those proof and claims, looked sound for me.\n3. I have very minor concerns for the experiments. (1) The authors do not compare with any existing methods. Then the proposed method looks like a black box in terms of performance. (2) The performance still drops greatly comopared with training using all the client data. (3) More analysis experiments are needed to help people to understand every component of the proposed method.\n4. I am not the expert on FL, then I have some questions on that.  (1) In introduction section, it is claimed 'FL decentralizes the training process by combining local models fine-tuned on each client’s private data and thus hinders privacy breaches.' Does it mean, traditinoally, one client cannot add other clients' face information at all for improving one client's training  due to the privacy problem? Traditionally, can the server collect the face information from all the clients and use them for training? If no for both questions, does it mean this work is the first to ask the client and server to use others' face information to improve the training? (2) 'That prevents the FL approach from broadcasting the whole model amongclients and the central server, and consequently leads to conflicts in the aggregation of local updatesin the overlapped regions. ' What do you mean the ovelapped regions?  (3) The authors mentioned privacy cost and the proposed method is efficient. Can you define what is the privacy cost? Which procedure does this cost happen?", "summary_of_the_paper": "This paper proposes a FL strategy for face recognition. Despite the wide investigation of FL, very littler research discusses the use of FL for face recognition.  The proposed of PrivacyFace can: (1) distill sanitized clusters from local class centers using Differentially Private Local Clustering (DPLC) . (2) use a consensus-aware recognition loss to  encourage global consensuses among clients, leading to a more discriminative feature learning. \n", "summary_of_the_review": "As aformentioned, I think this is an impactful work. The authors understand both FR and FL in depth and proposed a convincing solution. I think this work is inspiring for both industry and academia. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 quality reviews. The rebuttal and discussions were effective. All reviewers raised their ratings after the rebuttal. It finally received 3 ratings of 8, and 1 rating of 5. The AC concurs with the contributions made by this work and recommend acceptance.
Decision: Accept (Spotlight)