Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a FL strategy for face recognition. Despite the wide investigation of FL, very littler research discusses the use of FL for face recognition.  The proposed of PrivacyFace can: (1) distill sanitized clusters from local class centers using Differentially Private Local Clustering (DPLC) . (2) use a consensus-aware recognition loss to  encourage global consensuses among clients, leading to a more discriminative feature learning. 

main_review: 1. I think this is an impactful work. It aims to solve a practical and challenging task: FL for face recognition (FR). To my knowledge, in industry, the face recognition companies have not (at least, not widely) deployed the edge/distributed learning for face recognition products. Thus, this is a poineering work. The authors seem understand both FL and FR in depth, the proposed solution can meet the requirements of differetiable priacy and improve the face recognition performance in the edge/distributed learning scenarios.
2. I am from the filed of FR rather than FL. I did not check very carefully the proof and claims of the solutions meet the requirements of differential priacy. I just quickly went through those proof and claims, looked sound for me.
3. I have very minor concerns for the experiments. (1) The authors do not compare with any existing methods. Then the proposed method looks like a black box in terms of performance. (2) The performance still drops greatly comopared with training using all the client data. (3) More analysis experiments are needed to help people to understand every component of the proposed method.
4. I am not the expert on FL, then I have some questions on that.  (1) In introduction section, it is claimed 'FL decentralizes the training process by combining local models fine-tuned on each client’s private data and thus hinders privacy breaches.' Does it mean, traditinoally, one client cannot add other clients' face information at all for improving one client's training  due to the privacy problem? Traditionally, can the server collect the face information from all the clients and use them for training? If no for both questions, does it mean this work is the first to ask the client and server to use others' face information to improve the training? (2) 'That prevents the FL approach from broadcasting the whole model amongclients and the central server, and consequently leads to conflicts in the aggregation of local updatesin the overlapped regions. ' What do you mean the ovelapped regions?  (3) The authors mentioned privacy cost and the proposed method is efficient. Can you define what is the privacy cost? Which procedure does this cost happen?
summary_of_the_review: As aformentioned, I think this is an impactful work. The authors understand both FR and FL in depth and proposed a convincing solution. I think this work is inspiring for both industry and academia.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The main contribution of the paper is a slight modification of the standard federated learning (FL) framework for the purpose of improving face recognition performance. Since face recognition involves learning a deep neural network for feature embedding and the weight vectors for mapping the feature embedding to an identity, the standard FL framework (e.g. FedAvg) that updates only the feature embedding network does not suffice. Hence, the paper proposes to cluster the weight vectors of each participant and transmit the cluster centers (albeit with Gaussian noise to ensure differential privacy) to the other participants. Once the cluster centers of other participants are known, the local weight vectors can be learned to avoid overlap with those received clusters.
main_review: 1) First and foremost, the motivation and threat model of the proposed scheme is not clear. For example, what is the exact privacy goal of a participant in federated face recognition? Is it to prevent other participants from learning about the faces/identities in the local dataset? If yes, how does differential privacy help in achieving this goal? Given the updated feature embedding model from the server AND the 'noisy' cluster centers of the weight vectors, isn't it fairly straight forward to reverse engineer the face images that correspond to each cluster?

2) Secondly, the proposed scheme is likely to work if all the face images in the local dataset are somewhat similar and form a compact cluster. Moreover, the face images in the other local datasets must not have large overlap with the local face images in the feature space. In the absence of these constraints, it may be hard to reach global consensus when the participants have mixed datasets (a more realistic setting). This may be the reason the experiments are conducted based on each participant having samples only from a single race.

3) The paper appears to analyze the differential privacy guarantees of revealing the cluster centers and the model parameter updates of the feature embedding separately. However, these two strands of information may be used in conjunction to develop more sophisticated attacks.

4) The experimental validation is weak. The proposed approach has been compared only against a centrally trained model and the proposed model without differential privacy. It has not been benchmarked against recent works on federated face recognition such as:

D. Aggarwal, J. Zhou and A. K. Jain, "FedFace: Collaborative Learning of Face Recognition Model", International Joint Conference on Biometrics (IJCB) 2021
summary_of_the_review: The motivation and threat model of the proposed work are not clear. The proposed approach appears to have some practical limitations, which have not been considered. The privacy analysis is not very robust and the experimental validation is also weak. 

During the rebuttal process, concerns 2 and 4 have been addressed to a great extent. Concern 1 still remains a theoretical possibility, but given that I do not have strong evidence to disprove the claims in the paper, it is fair to give the benefit of doubt to the authors. Concern 3 is still a very valid concern, but has been dismissed lightly in the rebuttal. Overall, I would like to upgrade my rating by one level based on all the discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a novel framework to train a face recognition network for the federated learning setting. The proposed method leverages differentially private local clustering mechanism to allows to securelly share the information of class centers of a local client to other clients. 
 In addition, the consensus-aware recognition loss encourages the global consensuses for the learned face embeddings among clients and significantly improves the performance as compared with the plain federated learning baselines. 
main_review: This work proposes a novel framework to train a face recognition network for the federated learning setting.
The points of the strength for the work are listed as follows:

1. the proposed differentially private local clustering mechanism with solid theoretical guarantees allows to securelly share the information of class centers of a local client to other clients. 
2. In addition, the consensus-aware recognition loss encourages the global consensuses for the learned face embeddings among clients and significantly improves the performance as compared with the baselines. 

The points of the weakness:

1. some parameters for the experiments are missing. I could not find the number of clients participating in the training. In addition, is the data distributed in iid or non-iid way. How many clients (the percentage) do participate in the training for each communication round? The details of these setting and other experimental results and discussion are totally missing from the paper.
2. It seems that the training requires to start with a pretrained face recognition model using some large-scale dataset. However, for the current face recognition models, most of them can be trained from scratch even using the CASIA-WebFace instead of using the MS1M dataset. The bad initial class centers could still lead to the issue as shown in Fig. 1(a) where the embedding space could still overlap. The studies of different initial pretrained models \phi_{0}'s are missing to show if the proposed method is senstivie to different initial models.
summary_of_the_review: I think the idea is novel and there are many solid theoretical proofs for differentially private guarantee to secure the information exchange among clients for model training. However, some important parts for federated learning or face recognition are still not well explained in the current paper. Since the federated learning is to train the model distributedly, some clients may drop for certain round of communication due to network issues. However, from algorithm 2, the experimental results are mainly done in the perfect setting that each round, all clients participate in the training and share their clusters to others through the server. It can be imagined that if some clusters do not participate in the training for a certain round, the consensus-aware recognition may fail and still result in the same situation as shown in Fig. 1(a). Thus, It is worth a study how it  would influence the final performance.

Does the author try different initial models to start training? Does the proposed framework require a strong face model to begin with (trained with a larger dataset and finetuned onto a smaller or similiar-size dataset)? Is it possible to start from scratch or a pretrained model with smaller dataset (CASIA-WebFace) and then to finetune onto a larger dataset (MS1M)?

Can the model scale to train with MS1M or deepGlinit or recent released WebFace260M which contains much more subjects (class centers) than the CASIA-WebFace (10575) and BUPTBalancedFace? 

I will adjust my rating according to the feedback from the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The aim of this paper is to tackle the privacy leakage issue when using federated learning technique to train deep face recognition models. The naïve practice, broadcasting the last fully connected layer, results in the leakage of ID features. But if we cut off the broadcast of $W^c$, it will cause the issue of overlapping IDs among the clients, which harms the training of face recognition model. Therefore, the authors propose their method, DPLC, to tackle this problem. The idea is to conduct online clustering, gaussian perturbation, combined supervision of cluster and local data.
main_review: + The proposed method can well address the issue of privacy leakage.
+ The face recognition performance benefits from the supervision combination of local data and online cluster.
- We doubt whether the gaussian perturbation is able to cover the privacy of facial ID. How to balance the privacy preserving and the training effect through tuning the extent of the gaussian perturbation? This is not clear in the paper.
- The urgency of the problem itself is not solid to me. Even though the value of $W^c$ is leaked, the ID information cannot be easily recovered without the remaining part of the neural network. 

summary_of_the_review: Based on the primary opinion above, I will temporarily give the ranking of “marginally below the acceptance threshold”. The final rating will depend on the authors’ feedback.
review_title: null

Meta-review: This paper received 4 quality reviews. The rebuttal and discussions were effective. All reviewers raised their ratings after the rebuttal. It finally received 3 ratings of 8, and 1 rating of 5. The AC concurs with the contributions made by this work and recommend acceptance.
Decision: Accept (Spotlight)