Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors study the impact of different deep learning networks on the fairness of face recognition algorithms. A Neural Architecture Search is performed, and a wide variety of network architectures (with multiple sets of hyperparameters) are employed for the analysis. The Rank Disparity metric is used for fairness evaluation in the face verification task. The authors also release the code for reproducibility.

strength_and_weaknesses: Strong points: 
1. The usage of NAS in the context of the fairness of algorithms is novel and can provide insight into the performance of models from an architectural and hyperparameter standpoint.
2. The authors conduct extensive experimentation by utilizing 29 different model architectures and a total of 355 training runs (explained in Section 3 of the paper).
3. The authors utilize a balanced subset of the large-scale CelebA dataset for training and evaluation. They further showcase transferability results on the RFW and VGGFace2 datasets. 

Weak points: 
1. There is a lack of a concrete conclusion based on the experiments that have been performed in the study.
2. The authors utilize only one fairness metric, while a wide variety of metrics exist in the literature, which limits the analysis. Another popular metric, such as standard deviation across the verification accuracy of subgroups, could have been added. 
3.  Several bias mitigation strategies exist in the literature. The authors have highlighted this in the Related Work section. Implementing some of those approaches and identifying the impact of hyperparameters and network architectures on face recognition performance will be an interesting addition to this study.
4. The results and analysis sections are limited.


Typos/Minor Comments: 
1. The authors can provide additional information about the Rank Disparity metric stating its range and behavior.
2. In the Error vs Fairness Value curves, it can be useful to also color-code the non-optimal configurations based on the model architecture. It might provide further insight into the behavior of different models.

clarity,_quality,_novelty_and_reproducibility: The details are there for reproducibility. The concept is important but I will not rate it very high in terms of quality and novelty.
summary_of_the_review: While there is always scope to train using more search strategies and databases, the authors perform extensive experimentation of the different architectures and hyperparameters for fairness in face recognition. However, a strong conclusion and key takeaways are missing from the study.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors ran a large-scale analysis of 29 architectures and trained them with various hyperparameters. They spent 88,493 GPU hours building this study to prove that the recognition accuracy of a network can only sometimes improve the fairness of recognition. Based on this, they conduct a neural architecture search (NAS) for fairness and accuracy. SOTA results achieved by searched networks are illustrated with Pareto pictures.
strength_and_weaknesses: Strengths:
1, Extensive experiments on existing face recognition algorithms reported in the paper may help researchers better understand those baselines and their hyperparameters.
2, The code and raw result files are released, which makes it easier for readers to follow the work. 
3, They outperform other structures in terms of accuracy and fairness on VGGFace2 and RFW datasets by conducting NAS and hyperparameter optimization.

Weakness:
1, The proposed method seems to need more novelty. There are no new concepts or search strategies presented.  
2, Although the paper is focused on recognition fairness. It might be worth reporting the accuracy comparison between the searched network and SOTA results in a table. 
3, This paper illustrates only results on VGGFace2 and RFW. Although some datasets may not have identity labels, it is necessary to show the verification/recognition performance on mainstream benchmarks, such as LFW, CFP-FP, CPLFW, AgeDB, CALFW and IJB-B/C. Experiments on two datasets are insufficient to prove the searched network achieved a good trade-off between accuracy and fairness.

clarity,_quality,_novelty_and_reproducibility: The paper has provided enough evidence to prove that recognition accuracy improvements cannot lead to fairness. They have clearly stated their search methods and searching strategies. Since they released their code, it would be easy to reproduce their results. However, they just applied the existing NAS on face recognition in terms of accuracy and fairness. It may not be novel enough.
summary_of_the_review: The motivation is well explained and proven. However, more results on popular mainstream benchmarks should be reported. Moreover, if the author can clarify the differences (the novelty) between the NAS they adopted and the existing SOTA NAS methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Unlike other previous works, the work first studies the fairness issues for face recognition in terms of the neural architecture and hyperparameter search. The authors run a large-scale study of 29 architectures from ViT to Xception for 355 models in total with different hyperparameters and spend 88,493 GPU hours. They culminate in a set of architectures which pareto-dominate all models in a large set of modern architectures. The works provide a foundation for the future works to further study the fairness in terms of network architecture and hyperparameters.

strength_and_weaknesses: Strength:
1. The work runs a very large scale architecture, hyperparameter search, and training  for face recognition  to study the relation between the fairness metric and accuracy under different perceived attributes (gender and ethnicity). 
2. Although this is an experimental paper without any novel methodologies, the thorough experimental results provide insight of how different architectures and hyperparameters could affect the performance of fairness.

Weakness:
1. The study is mainly conducted using the CelebA dataset which only contains around 200 thousands face images for around 10,000 people and most of the faces are in frontal pose. The dataset is relatively small with respect to commonly used MSCeleb-1M, VGGFace-2, Glint360K, or WebFace260M (unfortunately, the proposed approach does not take VGGFace2 for any training.) The performance study may not generalize well to other face datasets.
2. The fairness metrics proposed in other recent works are not considered together. For example, the bias/performance coefficient proposed in "Dhar, Prithviraj, Joshua Gleason, Aniket Roy, Carlos D. Castillo, and Rama Chellappa. "PASS: Protected Attribute Suppression System for Mitigating Bias in Face Recognition." In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15087-15096. 2021."
The metric, rank disparity, used in the paper only considers the performance of face retrieval and cannot fully cover the performance of face verification which is usually measured by true positive rate vs false negative rate. This limits the usefulness of the conducted studies.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and also provides the code for reproducibility purpose. However, since the paper is mainly an experimental paper of studying the correlation between the fairness and network architecture/hyperparameter, they employ the existing technologies for the job. Although it is of limited technology novelties, the finding of this work provide some references for the future research for the same topic.
summary_of_the_review: My main concern about the paper is that the dataset for study may not be representative enough to reflect the performance of current state-of-the-art face recognition models and the metric studies in the paper could not well represent the trade-off between face recognition accuracy and fairness since the face recognition contains two main tasks, face identification and face verification. The rank metric seems to reflect the performance of face identification but ignore face verification. If the authors could well-addressed my concern, I will change my rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper derives a conclusion that the hyper-parameters and architectural properties are relative to the accuracy, bias, and disparity. Thus they conduct NAS and HPO to obtain a powerful architecture and hyper-parameters, which outperforms other methods in terms of accuracy and fairness by a large margin.
strength_and_weaknesses: Strength:
1. This paper conducts a thorough experiment to analyze the relation among the hyper-parameters and architectures, and the accuracy, bias and disparity.
2. This paper concerns fairness, a long-stand problem in face recognition.
3. The final results outperform other algorithms by a large margin.

Weaknesses:
1. The conclusion of the relation among the hyper-parameters, architectures and fairness is too natural to me. Actually, if you conduct this analysis on other tasks, like face detection, face alignment, face reconstruction, face synthesis, etc., I believe this conclusion still holds as well. This is why the tasks like NAS and HPO exist. So even though the efforts were really worth respecting in this part, I did not find the conclusion exciting.
2. In the NAS and HPO, from my perspective, authors mostly borrow existing algorithms, where I still fail to find much novelty.

clarity,_quality,_novelty_and_reproducibility: Clarity: 8 out of 10. The writing is clear and easy to follow.
Quality: 6 out of 10. The engineering part of this paper is solid. The results are beautiful.
Novelty: 3 out of 10. Although this is the first paper to use NAS and HPO to enhance the fairness, from my perspective, it lacks technical novelty and does not provide new conclusion.
Reproducibility: 8 out of 10. Authors have included the code.
summary_of_the_review: In conclusion, I think the project is a solid work, and the paper is a nice report, but lacks technical novelty to be a ICLR paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the impact of network architecture and hyperparameter under accuracy-fairness metric for face recognition. The paper adopts existing approaches to evaluate different models under accuracy-fairness metric. Extensive experiments have been performed with empirical suggestions are made.

Strength
- The presentation is easy to follow.
- Experiments are extensive.

Weakness
- Novelty (main issue). It is obvious that network architectures and hyperparameters can have great impact on predicting performance, and many existing works have study similar problems (see [1] for an example). Compared with existing benchmark / understanding approaches, the new things introduced is "fairness" metric. However, no new metric is proposed. Besides, it is not clear what "special" insights can be generated for "fairness". 
- Insufficient comparison. Existing NAS methods should be compared. There are popular types of NAS methods that can handle the discrete "Rank disparity" metric. For example, the authors may have a check over [2]. 


[1]. Graph Structure of Neural Networks. ICML 2020.
[2] Single Path One-Shot Neural Architecture Search with Uniform Sampling. ECCV 2020.
justification_for_why_not_higher_score: Novelty is the main concern. This paper focuses on empirical study, it offers good guidance to pickup architecture for face recognition under accuracy-fairness trade-off. However, it does not exhibit new technologies or insights. In its current form, NeurIPS benchmark track or TMLR can be a better place to go.
Decision: Reject