Improving Federated Learning Face Recognition via Privacy-Agnostic Clusters

1 INTRODUCTION . Face recognition technique offers great benefits when used in right context , such as public safety , personal security and convenience . However , misuse of this technique is a concern as it involves unique and irrevocable biometric data . The rapid commercial applications based on face recognition and facial analysis techniques have stimulated a global conversation on AI ethics , and have resulted in various actors from different countries issuing governance initiatives and guidelines . EU ’ s General Data Protection Regulation ( GDPR ) ( Voigt & Von dem Bussche , 2017 ) , California Consumer Privacy Act ( CCP ) and Illinois Personal Information Protection Act ( IPI ) enforces data protection “ by design and by default ” in the development of any new framework . On the nationwide “ 315 show ” of year 2021 , the China Central Television ( CCTV ) called out several well-known brands for illegal face collection without explicit user consent . As researchers , it is also our duty to prevent the leakage of sensitive information contained in public datasets widely used by the research community . Therefore , faces in ImageNet ( Deng et al. , 2009 ) were recently all obfuscated ( Yang et al. , 2021 ) and a large face dataset called MS-Celeb-1M ( Guo et al. , 2016 ) was pulled of the Internet . In the wake of growing social consensus on data privacy , the field of face recognition calls for a fundamental redesign about model training while preserving privacy . A potential solution is the paradigm called Federated Learning ( FL ) ( McMahan et al. , 2017a ) . Given C clients with local datasets { D1 , D2 , · · · , DC } as shown in Fig . 1a , FL decentralizes the training process by combining local models fine-tuned on each client ’ s private data and thus hinders privacy breaches . Typical examples of these clients include personal devices containing photo collections of a few family members , or open-world scenarios such as tourist attractions visited by tens of thousands of people . In most circumstances , we can safely assume very few classes would co-exist in two or more clients . Despite the numerous FL-based applications in various domains ( Kairouz et al. , 2019 ) ranging from health to NLP , there are very little progress ( Aggarwal et al. , 2021 ; Bai et al. , 2021 ; Liu et al. , 2021 ) in training face recognition models with FL schemes . Unlike other tasks , parameters of the last classifier for a face recognition model are crucial for recognition performance but strongly associated with privacy . These parameters can be regarded as mean embeddings of identities ( Wang et al. , 2018 ; Meng et al. , 2021b ; Shen et al. , 2020 ) ( also called as class centers ) , from where individual privacy could be spied out as studied by plenty of works ( Kumar Jindal et al. , 2018 ; Boddeti , 2018 ; Mai et al. , 2020 ; Dusmanu et al. , 2021 ) . That prevents the FL approach from broadcasting the whole model among clients and the central server , and consequently leads to conflicts in the aggregation of local updates . As depicted in the global feature distribution of Fig . 1a , both clients try to spread out their own classes in the same area ( pointed by the arrow ) of the normalized feature space . Thus , the training loss could oscillate to achieve consensus given the sub-optimal solutions from multiple clients . On the other hand , a large batch with sufficient negative classes is necessary to learn a discriminative embedding space for advanced face recognition algorithms . During the conventional FL updates , each class is only aware of local negative classes while those from other clients are untouchable . This further limits performances of FL approaches in face recognition . The privacy-utility paradox motivates us to introduce PrivacyFace , a framework improves federated learning face recognition by broadcasting sanitized information of local class globally . In the framework , a novel algorithm called Differentially Private Local Clustering ( DPLC ) first generates privacy-agnostic clusters of class centers while any specific individual in the cluster can not be learned , irrespective of attacker ’ s prior knowledge , information source and other holds . Recall that the privacy cost of a differential privacy scheme is propotional to the l2-sensitivity while inversely propotional to the query number . Our DPLC reaches a low l2-sensitivity by restricting the cluster size as well as covering sufficient class centers . In addition , the number of necessary centers to communicate in DPLC is irrelevant to the number of classes in the training data . These characteristics jointly equip DPLC with much smaller privacy cost than the naive alternative , which sanitizes each class center individually by Gaussian noise . In our experiments , DPLC ’ s privacy cost is only 1.7e-7 of that of the naive approach . That persuasively reveals the high security level of our approach . The second part of PrivacyFace is the consensus-aware face recognition loss . Following principles of Federated Averaging ( FedAvg ) ( McMahan et al. , 2017a ) , a server iteratively gathers feature extractors and privacy-agnostic clusters from clients , averages parameters of feature extractors and distributes them to clients . Accordingly , the consesus-aware loss notifies each client not to embed samples in the inappropriate zone ( differential private clusters marked by DP ) of the feature space during the local optimization , as shown in Fig . 1b . This process aids each client to train more discriminative features as well as align all consensuses . Compared to the conventional approach , our PrivacyFace boosts performances by +9.63 % and +10.26 % for TAR @ FAR=1e-4 on IJB-B and IJBC respectively with only single-digit privacy cost . Moreover , the additional computational cost as well as communication cost are negligible ( e.g. , the extra clusters to broadcast only occupy 16K storage while the backbone already takes 212M ) . In a word , PrivacyFace is an efficient algorithm which improves conventional federated learning face recognition by a large margin on performances , while requires little privacy cost as well as involves lightweight computational/communication overheads . 2 PRELIMINARIES . 2.1 DEEP FACE RECOGNITION . Most of early works in deep face recognition rely on metric-learning based loss , including contrastive loss ( Chopra et al. , 2005 ) , triplet loss ( Schroff et al. , 2015 ) and N-pair loss ( Sohn , 2016 ) . These methods are usually inefficient in training on large-scale datasets . One possible reason is that their embedding spaces at each iteration are constructed only by a positive sample and limited negative ones . Therefore , the main body of research ( Ranjan et al. , 2017 ; Liu et al. , 2017 ; Wang et al. , 2018 ; Deng et al. , 2019 ; Xu et al. , 2021 ; Meng et al. , 2021a ; c ) has focused on devising more effective classification-based loss and achieved leading performances on a number of benchmarks . Suppose that we are given a training dataset D with N face samples { xi , yi } Ni=1 of n identities , where each xi is a face image and yi ∈ { 1 , · · · , n } denotes its associated class label . Considering a feature extractor φ generating the embedding fi = φ ( xi ) and a classifier layer with weights W = [ w1 , · · · , wn ] , a series of face recognition losses can be summarized as Lcls ( φ , W ) = − N∑ i=1 log eu ( wyi , fi ) eu ( wyi , fi ) + ∑n j=1 , j 6=yi e v ( wj , fi ) . ( 1 ) The specific choices of the similarity functions u ( w , f ) , v ( w , f ) yield different variants , e.g . : CosFace ( Wang et al. , 2018 ) : u ( w , f ) = s · ( cos θ −m ) , v ( w , f ) = s · cos θ , ( 2 ) ArcFace ( Deng et al. , 2019 ) : u ( w , f ) = s · cos ( θ −m ) , v ( w , f ) = s · cos θ , ( 3 ) where s , m are hyper-parameters and θ = arccos ( wTf/‖wTf‖ ) is the angle between w and f . 2.2 DIFFERENTIAL PRIVACY . Differential Privacy ( DP ) ( Dwork et al. , 2006 ) is a well-established framework under which very little about any specific individual can be learned in the process irrespective of the attacker ’ s prior knowledge , information source and other holds ( Feldman et al. , 2017 ; Nissim et al. , 2016 ; Stemmer & Kaplan , 2018 ) . We state the relevant definitions and theories ( Dwork & Roth , 2014 ) below . Definition 1 ( Differential Privacy ) . A randomized algorithm M : X → Y is ( , δ ) -DP if for every pair of neighboring datasets X , X ′ ∈ X ( i.e. , X and X ′ differ in one row ) , and every possible output T ∈ Y the following equality holds : Pr [ M ( X ) ∈ T ] ≤ e Pr [ M ( X ′ ) ∈ T ] + δ . Here , δ ≥ 0 are privacy loss parameters , which we consider a privacy guarantee meaningful if δ = o ( 1n ) , where n is the size of dataset . Definition 2 ( l2-sensitivity ) . The l2-sensitivity of a function f : X → Rd is ∆2 ( f ) = max X , X′∈X X , X′are neighbors ‖f ( X ) − f ( X ′ ) ‖2 . ( 4 ) Definition 3 ( Gaussian Mechanism ) . Suppose a function f : X → Rd and Id is the ddimensional identity matrix . The mechanism M ( X ) = f ( X ) + N ( 0 , σ2Id ) is ( , δ ) -DP if σ ≥ ∆2 ( f ) √ 2 ln ( 1.25δ ) . Definition 4 ( Composition of Differentially Private Algorithms ( Dwork & Roth , 2014 ; Dwork & Lei , 2009 ) ) . Suppose M = ( M1 , M2 , · · · , Mk ) is a sequence of algorithms , where Mi is ( i , δi ) differentially private , and the Mi ’ s are potentially chosen sequentially and adaptively . Then M is ( ∑k i=1 , ∑k i=1 δk ) -differentially private . 3 PRIVACYFACE . This section details the PrivacyFace framework as illustrated in Fig . 2 . At its core is a novel clustering algorithm that extracts non-sensitive yet informative knowledge about the local class distribution ( Fig . 2a ) . After the central server broadcasting local DP-guaranteed outputs ( Fig . 2b ) , each client optimizes over a consensus-aware objective that takes into account both the local data and the privacy-agnostic clusters to learn a discriminative embedding space ( Fig . 2c ) for face recognition . p local class centers p local class centers p local class enters 3.1 DIFFERENTIALLY PRIVATE LOCAL CLUSTERING . Our target is to improve performances of federated learning face recognition by communicating auxiliary and privacy-agnostic information among clients . To distill useful information from each client and resolve the privacy-utility paradox , we propose a specialized algorithm called Differential Private Local Clustering ( DPLC ) with rigorous theoretical guarantees . Problem Design . To facilitate the discussion , let us first introduce the margin parameter ρ ∈ [ 0 , π ] , which defines the boundary of a cluster centered at p by saying that any w in the cluster satisfies1 arccos ( wTp ) ≤ ρ . We design the local clustering problem with two principles . First , the clustering results are expected to carry information only about population rather than individual from local dataset . Supported by Theorem 2 proved later , this principle ensures PrivacyFace to gain insight about the underlying distribution , while preserving the privacy of individual record . Second , the spaces confined by the clusters should be tight because they represent the inappropriate zones to escape from in other clients ’ perspectives . A cluster with large margin ρ would occupy a huge proportion of the sphere , leaving limited room for other embeddings ( e.g. , half of sphere would be occupied if ρ = π/2 ) . More specifically , we quantify the occupancy ratio by the following theorem : Theorem 1 . Assume there is a unit and d-dimensional sphere Sd and an embedding point p ∈ Sd . Embeddings with angles less than ρ to p ( i.e. , { f : arccos ( fTp ) ≤ ρ } ) occupy 12Isin2 ( ρ ) ( d−1 2 , 1 2 ) of the surface area of Sd . Here Ix ( a , b ) is the regularized incomplete beta function . Proved in Sec . A.1.3 of the appendix , Theorem 1 indicates that the occupancy ratio increases monotonically with respect to the margin ρ given a feature dimension d. Fig . 3 plots a series of functions between cluster margin ρ and occupancy ratio under different dimensions . Take the rightmost curve corresponding to a typical setting d = 512 in face recognition as an example . The occupancy ratios are 0.055 , 5 · 10−5 , 4 · 10−10 when ρ = 1.5 , 1.4 , 1.3 respectively . In another word , the whole feature space would be fully occupied if we sample 20 , 000 clusters over the original face classes when ρ = 1.4 . It is obvious that the more space left for optimization , the higher likelihood to achieve better performance . Therefore , we seek for a margin-constrained clustering of local data with fixed ρ ≤ 1.4 when d = 512 for better privacy-utility trade-offs . A counter example to the latter principle is the classical k-means algorithm , which have been extended to differentially private versions in a number of works ( Nissim et al. , 2016 ; Feldman et al. , 2017 ; Stemmer & Kaplan , 2018 ) . However , k-means naturally has no control about the scopes of the generated clusters , not to mention the difficult hyper-parameter k to pick . That motivates us to propose a specified clustering algorithm for our task . 1We adopt cosine similarity by assuming all vectors lie on a unit sphere in face recognition . An Approximated Solution . Given the margin ρ , the problem of finding clusters to cover all class centers is a generalization of the well-studied 2D disk partial covering problem which is unfortunately NP-complete ( Xiao et al. , 2004 ) . We take a greedy approach by iteratively finding a cluster centered at p∗ with margin ρ to cover the most class centers wi and weeding out those covered before the next round . We cast the problem of identifying such a cluster at each iteration as : Definition 5 ( Spherical Cap Majority Covering Problem ) . Denote 1 ( · ) as an indicator function . Assuming a unit d-dimensional sphere Sd and class centers { w1 , w2 , · · · , wn } ∈ Sd , the target is to find a cluster center p∗ ∈ Sd where p∗ = arg maxp∈Sd ∑n i=1 1 ( arccos ( w T i p ) ≤ ρ ) . Finding the optimal p∗ is also NP-complete . To address this , we first sort out the densest area S in terms of the neighbor count for eachw . An efficient approximation of p∗ is then the average of class centers in the area , p = 1|S| ∑ i∈S wi . Lines 3-5 of Algorithm 1 summarize the detailed procedure , by which the approximate cluster centers pi can be quickly enumerated as depicted in Fig . 2a . In line 8 , the class centers would be further normalized in accordance with the DP mechanism . Algorithm 1 : Differentially Private Local Clustering ( DPLC ) Data : Class center embeddings W = { w1 , w2 , · · · , wn } encoded in the classifier layer . Parameters : Margin ρ ; Minimum cluster size T ; Maximum # queries Q ; Privacy budget , δ . Result : Privacy-agnostic cluster centers p̂ ’ s . Let I = { 1 , 2 , · · · , n } and calculate θi , j = arccos ( wTi wj ) for i , j ∈ { 1 , 2 , · · · , n } ; 1 for q = 1 , 2 , · · ·Q do2 For each { wi , i ∈ I } , find the indexes of its neighbors by Si = { j : θi , j ≤ ρ , j ∈ I } ; 3 Find a set with the most elements , i.e. , S = arg max |Si| ; 4 p← 1|S| ∑ i∈S wi ; // approximate solution for problem 55 if |S| ≥ T then6 Sample a v from distribution N ( 0 , σ2Id ) where σ = 2|S| √ ( 1− cos ( 2ρ ) ) ln ( 1.25δ ) ; 7 p̂q ← p+v‖p+v‖ , I ← I/ { arccos ( w T i p ‖p‖2 ) ≤ ρ , i ∈ I } ; 8 else9 Break out of the loop ; 10 Differential Privacy Endorsement . Although the centers p reveal the population-level property of the local training set , they are still outcomes of a deterministic algorithm , thereby vulnerable to adversary attack during the FL updating . We prove below that p can be perturbed to achieve DP : Theorem 2 . Define a function p , f ( S ) = 1|S| ∑ i∈S wi . Then the Gaussian Mechanism p̂ , M ( S ) = f ( S ) +N ( 0 , σ2Id ) is ( , δ ) -DP if σ ≥ 2|S|· √ ( 1− cos ( 2ρ ) ) ln ( 1.25δ ) . Proof . Let S be the set of indexes of class centers which have cosine similarities larger than cos ρ with respect to a centerwo . Assuming that S , S ′ are neighbors differed atw andw′ , where vectors are normalized , i.e. , ‖w‖ = ‖w′‖ = ‖wo‖ = 1 , then we have wTo w ≥ cos ρ and wTo w′ ≥ cos ρ. Lemma 2 in appendix states that w Tw′ ‖w‖2‖w′‖2 ≥ cos ( arccos wTo w ‖wo‖2‖w‖2 + arccos wTo w ′ ‖wo‖2‖w′‖2 ) for all { w , w′ , wo } . Therefore , the lower bound of wTw′ is wTw′ = wTw′ ‖w‖2‖w′‖2 ≥ cos ( arccos w T o w ‖wo‖2‖w‖2 + arccos wTo w ′ ‖wo‖2‖w′‖2 ) ≥ cos ( ρ+ ρ ) = cos ( 2ρ ) Following this inequality , the l2-sensitivity of f ( S ) is 1|S| √ 2− 2 cos ( 2ρ ) as ‖f ( S ) − f ( S ′ ) ‖2 = 1 |S|‖w−w ′‖2 = 1|S| √ 2− 2wTw′ ≤ 1|S| √ 2− 2 cos ( 2ρ ) . By Definition 3 , we easily conclude that M ( S ) = 1|S| ∑ i∈S wi +N ( 0 , σ2Id ) is ( , δ ) -DP if σ ≥ 2 |S|· √ ( 1− cos ( 2ρ ) ) ln ( 1.25δ ) . Additional proofs can be found in Sec . A.1.2 of appendix . Theorem 2 outlines the setting of noise variance σ given the privacy budget , δ and the cluster margin ρ . Basically , the lower bound of σ is proportional to 1|S| . That implies when the size of cluster |S| is large enough , adding small noise is sufficient to achieve promising privacy level . We found setting minimum cluster size to T , min |S| = 512 yields empirically stable performance . It is worth emphasizing that the number of original classes n does not directly influence the introduced noise σ . This advantage ensures that DPLC maintains a high utility when dealing with large-scale face recognition problems . Full algorithm . Algorithm 1 presents the full DPLC and we highlight several key properties below . Properties 1 , 2 state that DPLC is efficient and differentially private in theory . Besides the efficency and privacy , another major concern of the DP algorithm is the utility . Specifically , as p̂ is a combination of the source cluster center p and Gaussian noise v , we should prevent the noise v to have overwhelming effects to the output , which is further verified by our Property 3 and Property 4 . Property 1 . The complexity of the DPLC algorithm is O ( Q · n2 ) . Proof . The computational complexity of calculating θi , j is O ( n2 ) . In each loop , we also compare the value θi , j , ∀i , j ∈ S with ρ , whose worst time complexity is O ( n2 ) . Because remaining steps are of linear complexity , the total complexity is O ( n2 +Q · n2 ) = O ( Q · n2 ) . Property 2 . DPLC is a ( Q · , Q · δ ) -differentially private algorithm . Proof . The mechanism f ( S ) + N ( 0 , σ2Id ) is ( , δ ) -differentially private according to Lemma 2 . As DPLC queries at most Q results , it ’ s easy to conclude that our DPLC algorithm is ( Q · , Q · δ ) differentially private based on Definition 4 . Property 3 . Denote Φ as the cumulative distribution function of a standard Gaussian distribution . For vector magnitudes of p , v , we have ‖p‖2 ∈ ( cos ρ , 1 ] and P ( ‖v‖2 ≤ r ) ' Φ ( r 2 σ2 √ 2 ( d−1 ) − d−12 ) . Proof . For p , the upper bound is 1 as ‖p‖2 = ‖ 1|S| ∑ i∈S wi‖2 ≤ 1 |S| ∑ i∈S ‖wi‖2 = 1 . As S denotes the set of indexes of class centers which has cosine similarities larger than cos ρ with a center wo , the lower bound of ‖p‖2 can be calculated by ‖p‖2 = ‖p‖2‖wo‖2 ≥ pTwo = ( 1 |S| ∑ i∈S wi ) Two = 1 |S| ∑ i∈S wTi wo ≥ 1 |S| ( 1 + ( |S| − 1 ) cos ρ ) > cos ρ . Therefore , we have ‖p‖2 ∈ ( cos ρ , 1 ] . For v ∼ N ( 0 , σ2Id ) , let v = [ v1 , v2 , · · · , vd ] , we have viσ following the standard normal distribution for all i . Thus , the sum of square of viσ follows a χ 2-distribution with d− 1 degrees of freedom , i.e. , 1 σ2 ‖v‖ 2 2 = ∑d i=1 ( vi σ ) 2 ∼ χ2d−1 . The mean and variance of ‖v‖22 are E ( ‖v‖22 ) = σ2 ( d − 1 ) and V ar ( ‖v‖22 ) = 2σ4 ( d− 1 ) , based on properties of the χ2-distribution . By the central limit theorem , ‖v‖22 converges toN ( σ2 ( d− 1 ) , 2σ4 ( d− 1 ) ) when d is large enough . In practice , for d ≥ 50 , ‖v‖22 is sufficiently close to N ( σ2 ( d− 1 ) , 2σ4 ( d− 1 ) ) ( Box et al. , 1978 ) . Therefore , we have P ( ‖v‖2 ≤ r ) = P ( ‖v‖22 − σ2 ( d− 1 ) √ 2σ4 ( d− 1 ) ≤ r 2 − σ2 ( d− 1 ) √ 2σ4 ( d− 1 ) ) ' Φ ( r 2 σ2 √ 2 ( d− 1 ) − √ d− 1 2 ) . Note that in our work , σ can be a small value with little privacy cost . Thus , with high probability , the norm of v is close to zero . Property 4 . If ‖p‖2 ≥ ‖v‖2 , the cosine similarity between p and p̂ is always greater than√ 1− ‖v‖22/|p‖22 . Proof . Denote magnitudes of p , v as lp , lv and cosine similarity between p and v as x ( i.e. , x = p·v ‖p‖‖v‖ ∈ [ −1 , 1 ] ) . Let a = lr lp < 1 and s be the cosine similarity between p + v and p , then s = ( p+v ) Tp ‖p+v‖2‖p‖2 = l2p+lplvx lp √ l2p+l 2 v+2lplvx = 1+ax√ 1+a2+2ax . The first order derivative of s with respect to x is dsdx = a2 ( a+x ) ( 1+a2+2ax ) 3 2 . It ’ s easy to conclude that s takes the smallest value when x = −a , which can be reached as a is always smaller than 1 . In the end , we have s ≥ √ 1− l2v/l2p . In DPLC , we reasonably require large local clusters S at line 6 while preserving privacy in terms of , δ . That constraint ensures the synthesized noise v to have a small magnitude ( property 3 ) , which thereby results in slight offsets of the sanitized vector p̂ to the source p ( property 4 ) . With these theoretical guarantees , DPLC can obtain good end-to-end utility with low privacy cost .

Summary: This work proposes a novel framework to train a face recognition network for the federated learning setting. The proposed method leverages differentially private local clustering mechanism to allows to securelly share the information of class centers of a local client to other clients.   In addition, the consensus-aware recognition loss encourages the global consensuses for the learned face embeddings among clients and significantly improves the performance as compared with the plain federated learning baselines.