FedEvalFair: A Privacy-Preserving and Statistically Grounded Federated Fairness Evaluation Framework

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:26:42

Abstract: Federated learning has rapidly gained attention in the industrial sector due to its significant advantages in protecting privacy. However, ensuring the fairness of federated learning models post-deployment presents a challenge in practical applications. Given that clients typically rely on limited private datasets to assess model fairness, this constrains their ability to make accurate judgments about the fairness of the model. To address this issue, we propose an innovative evaluation framework, FedEvalFair, which integrates private data from multiple clients to comprehensively assess the fairness of models in actual deployment without compromising data privacy. Firstly, FedEvalFair draws on the concept of federated learning to achieve a comprehensive assessment while protecting privacy. Secondly, based on the statistical concept of 'estimating the population from the sample', FedEvalFair is capable of estimating the fairness performance of the model in real-world settings from a limited data sample. Thirdly, we have designed a flexible two-stage evaluation strategy based on statistical hypothesis testing. We verified the theoretical performance and sensitivity to fairness variations of FedEvalFair using Monte Carlo simulations, demonstrating the superior performance of its two-stage evaluation strategy. Additionally, we validated the effectiveness of the FedEvalFair method on real-world datasets, including UCI Adult and eICU, and demonstrated its stability in dealing with real-world data distribution changes compared to traditional evaluation methods.

Main paper: 
 introduction: 
 Due to its significant advantages in protecting privacy, federated learning has quickly attracted the attention of the industrial sector. As the deployment of federated learning models in critical areas such as criminal sentencing, employee recruitment, and loan approval continues to expand, the issue of fairness in federated learning models has garnered widespread concern [1, 10] . Especially in recent years, many studies have introduced various concepts of fairness, such as group fairness. This concept aims to reduce biases against protected groups defined by sensitive attributes such as gender and race, and has become a focus of research [6, 23] . Group fairness is particularly concerned with mitigating the unfairness that may be exhibited by models during the training process, especially when models are trained on biased data. Although many studies have been devoted to measuring and mitigating this algorithmic bias, formal guarantees for fairness attributes are still lacking in practice [5, 8, 38, 40] .Ignoring the fairness evaluation of models during their actual deployment can lead to serious consequences [12, 19, 22, 36] . For instance, in models predicting patients' reactions to specific medications, if the differences in patients' genetic backgrounds, lifestyle habits, and environmental factors are not adequately considered, the model may produce misleading predictions for certain groups. Such biases, if not identified and corrected through reliable fairness evaluations, could lead to serious errors in treatment decisions for these patients, causing irreversible impacts on their health. However, there are problems with the methods of evaluating model fairness at the time of actual deployment. For example, existing methods for assessing model fairness are often based on existing datasets, but these datasets may not fully reflect the situation when the model is deployed in the real world, leading to these assessment methods failing to accurately reflect the model's fairness during actual deployment. Secondly, model developers often only have access to limited real-world data, and conducting more detailed fairness evaluations would require more data. However, considerations of data privacy limit the acquisition of additional data, further exacerbating the difficulty of accurately assessing model fairness.As awareness of data privacy protection strengthens, obtaining private data to assess the fairness of machine learning models has become increasingly difficult and costly [11] . Many data providers, especially organizations dealing with sensitive information such as medical institutions and financial companies, are becoming more reluctant to share their data, fearing that this may violate privacy protection regulations or compromise the privacy and security of their client [31] . This situation poses a significant challenge for developers who aim to enhance model fairness and ensure that their decision-making processes do not adversely affect specific groups. They are faced with the problem of how to effectively assess model fairness in the absence of sufficient data support. Therefore, there is an urgent need to develop a new type of model fairness assessment method that can utilize multi-source private data for evaluation without infringing on data privacy.Existing common methods for assessing model fairness primarily rely on evaluations conducted on test datasets, reflecting the model's fairness through metrics such as demographic parity or equality of opportunity [9, 26, 27] . However, a key limitation of this approach is its assumption that the test dataset can accurately represent the real-world data distribution when the model is deployed, an assumption that does not hold true in many practical scenarios. Therefore, model fairness assessments based solely on test datasets may not be reliable, especially when evaluating the fairness of models in real-world environments [5] . Given this, there is an urgent need to develop a new assessment method that can more accurately and reliably infer a model's fairness performance in actual deployment environments based on limited real-world datasets. Such a method would help more authentically capture and address potential fairness issues in models as they are applied in practice.Under the premise of data privacy protection, especially when data does not leave the local environment, how to utilize the private data of multiple participants to more reliably assess the fairness of models in actual deployment environments has become an important research question. To address this, we propose a model fairness federated evaluation framework named FedEvalFair, inspired by the concept of federated learning, aimed at enabling more reliable assessments of model fairness without direct access to the data. Specifically, the FedEvalFair framework first utilizes the bootstrap method at each participant's local site to generate multiple test sets through resampling. Then, based on the statistical principle of estimating population parameters from samples, it reliably estimates the model's fairness performance across different data providers. Moreover, we have developed an innovative two-stage fairness testing method, which can comprehensively assess the fairness of models on multi-source private data, thereby more accurately evaluating the model's fairness performance in actual deployment environments.The FedEvalFair framework holds significant practical value. Relying solely on limited private data from a single source for model evaluation can lead to uncertainty and bias. FedEvalFair addresses this by enabling participation of multi-clients' datasets through privacy-preserving measures, substantially expanding the scope and depth of evaluation. Grounded in statistical principles like estimating population parameters from samples and hypothesis testing, it ensures reliable and accurate fairness assessment in realworld scenarios. Applicable not only for pre-deployment global fairness analysis of federated models to identify biases against disadvantaged groups, FedEvalFair can also periodically monitor fairness changes in dynamic data settings, facilitating timely model adjustments or retraining.The main contributions of this study can be summarized as follows:• We developed the first multi-source evaluation system dedicated to assessing model fairness, the FedEvalFair framework. Its core objective is to accurately evaluate the fairness of models when deployed in real environments, ensuring that the decisionmaking process is fair to all user groups. • We proposed an innovative privacy-preserving framework that utilizes multi-party private datasets to comprehensively assess the fairness of models while fully protecting the data privacy of all participants. FedEvalFair ensures data security while allowing for in-depth analysis of model fairness. • Drawing on the principles of estimating population parameters from samples and statistical hypothesis testing methods in statistics, we constructed a novel method for evaluating model fairness. This method enhances the reliability of evaluation results and provides a solid theoretical foundation for the quantitative analysis of model fairness. • Compared to traditional methods confined to test datasets, FedE-valFair can more comprehensively assess model fairness in actual deployment environments. Experimental results demonstrate higher stability and reliability of our method when evaluating model fairness in real-world settings.This paper is structured as follows: Section 2 reviews related work, setting the theoretical background for the FedEvalFair framework. Section 3 explains the framework's background, including assumptions and definitions. Section 4 details FedEvalFair's core, the two-stage algorithm, covering theoretical underpinnings and implementation. Section 5 presents experimental setup and results, showcasing the framework's effectiveness. Finally, Section 6 summarizes the findings and conclusions of this study. 
related work 2.1 model fairness optimization: 
 In the field of machine learning, the core concept of fairness is to ensure that models do not engage in discriminatory behavior when processing individuals or groups with different protected attributes, such as race, gender, age, and religion. These protected attributes, as part of the input features, should not alone determine the decisionmaking process of the model. As the understanding of model fairness in the machine learning field deepens, numerous studies have been dedicated to exploring methods to enhance fairness. Zafar et al. developed an innovative mechanism by introducing a fairness metric for decision boundaries, designing algorithm classifiers that can maintain both high accuracy and fairness, successfully applied to logistic regression and support vector machines [36] . Zhang et al. introduced a framework that reduces bias by training a predictor and an adversary model, with the goal of maintaining accuracy from input to prediction while preventing the adversary model from accurately predicting protected variables [38] . Additionally, with the increasing demand for privacy protection, researchers have also explored how to improve model fairness while protecting privacy. Ezzeldin et al.'s FairFed, an innovative server-side algorithm for fairness-aware aggregation in federated learning, aims to enhance group fairness without needing centralized access to sensitive data, showing the potential to build fairer models [18] . Yue et al. 's GIFAIR-FL framework, by adding regularization terms in federated learning to achieve both group and individual fairness, has proven to maintain or enhance predictive accuracy while improving fairness in image classification and text prediction tasks [35] . Zeng et al. introduced the FEDFB algorithm, which significantly improves model fairness by modifying the FEDAVG protocol to simulate centralized fair learning, achieving the performance of centrally trained models in some cases [37] . Du et al. 's AgnosticFair framework, aimed at achieving model fairness under unknown testing data distributions, achieves high accuracy and fairness on unknown testing data by kernel reweighting of training samples in the loss function and fairness constraints [16] . Mohri et al. proposed a concept of weak fairness for machine learning models, reducing bias during the training process by minimizing the maximum training loss for protected categories, but this approach is limited to the training stage and cannot ensure comprehensive fairness of the model [28] .Researchers have developed a variety of model fairness assessment strategies to identify unfair phenomena in machine learning models [1, 20, 40] . Biswas et al. proposed a causal fairness approach aimed at assessing the impact of data preprocessing stages on fairness in the machine learning process, demonstrating through case studies how to mitigate bias by selecting appropriate data transformers [6] . Joymallya Chakraborty et al. introduced the Fairway method, which reduces bias in models by adjusting strategies during the preprocessing and training stages [10] . This method emphasizes the impact of real-world biases on the fairness of AI systems and demonstrates its effectiveness in discovering and reducing model biases while minimizing the impact on predictive performance, recommending the integration of bias testing and mitigation into standard practices of machine learning software development. Hort et al. introduced Fairea, a technique for evaluating the variability in model behavior as a result of machine learning bias mitigation methods, and tested the effectiveness of 12 bias mitigation methods through a large-scale empirical study [23] . Aggarwal et al. developed a novel method for automatically generating test inputs to detect individual discrimination in machine learning models, combining symbolic execution with local interpretability to efficiently create test cases, showing higher efficacy in detecting individual discrimination compared to existing benchmarks [1] . Zhang et al.introduced an innovative and scalable approach for identifying instances of individual discrimination in deep neural networks (DNNs) by employing lightweight processes like gradient computation and clustering, significantly enhancing scalability over current state-of-the-art techniques [40] .Despite these advancements, there remains a lack of empirical research to measure and compare the fairness of ML models in practical applications and to analyze the specific impact of mitigation algorithms on model performance [5] . As the protection of privacy increases, there is a particular focus in trustworthy federated learning on whether federated learning models can be deployed in real-world scenarios, and evaluating whether models comply with fairness principles during actual deployment is an important evaluation goal. 
preliminaries: 
 In this section, we will present the foundational setup related to the FedEvalFair framework and provide definitions pertinent to model fairness.In the domain of model fairness metrics, fairness is often divided into two core categories: individual fairness and group fairness. Individual fairness focuses on evaluating whether the model's predictions for similar pairs of samples are fair [17, 25] . On the other hand, group fairness concentrates on assessing the balance of the model's predictions for specific protected groups (e.g., "women" vs. "men") [7, 8, 15, 19, 22] . This study will focus on the measurement methods of group fairness.In existing research, to measure fairness, numerous metrics have been proposed [3, 4, 14] . These metrics typically involve calculating the classification ratios for specific groups, such as the true positive rate and false positive rate (e.g., true positives, false positives), and assess fairness by comparing the differences in these rates across different groups. Consider a dataset 𝐷 containing 𝑛 samples, with true classification labels 𝑌 , the model's predicted classification labels Ŷ , and a sensitive attribute 𝐴. In this setting, 𝐴 = 1 can represent a privileged group (such as males), while 𝐴 = 0 represents a non-privileged group (such as females). For binary model outputs Ŷ (and their corresponding labels 𝑌 ), we assume Ŷ = 1 indicates a positive outcome. We adopts typical group fairness metrics [5, 20] , with a particular focus on Demographic Parity (DP) and Equal Opportunity Disparity (EOD).EQUATIONEQUATIONWe denote the total disparity between Δ𝐷𝑃 (Demographic Parity disparity) or Δ𝐸𝑂 (Equal Opportunity disparity) as Δ𝐷𝑖𝑠. A model can be considered to have achieved fairness when it completes training and satisfies the following condition:Δ𝐷𝑖𝑠 -𝜀 ≤ 0,where 𝜀 represents a small perturbation value, introduced to ensure that the fairness assessment accounts for a certain degree of flexibility.For fairness across multiple data sources, a trained model ℎ is considered to have achieved fairness on multiple data sources if it satisfies the following condition for each data source:Δ𝐷𝑖𝑠 ℎ (𝑘) -𝜀 𝑘 ≤ 0, ∀𝑘 ∈ {1, . . . , 𝑁 }.When evaluating model fairness across multi-source private data, we measure the composite fairness performance using Δ𝐷𝑖𝑠 (𝑘) for fairness disparity on each of the 𝐾 clients (represent a data source). We derive multiple Bootstrap test datasets (𝑛 𝑖 ) from each client to compute Bootstrap disparity value, represented as 𝑌 𝑖1 , 𝑌 𝑖2 , . . . , 𝑌 𝑖𝑛 𝑖 . These datasets, created via non-parametric Bootstrap from limited original data, lead to varying statistical distributions. It is reasonable to assume that these Bootstrap disparity value follow a normal distribution 𝑁 (𝜇 𝑖 , 𝜎 2 𝑖 ), an assumption that becomes increasingly reliable after a sufficient number of Bootstrap iterations. Thus, we can collect a comprehensive set of fairness disparity data for each client. According to the law of large numbers, the validity of this assumption strengthens with the increase in dataset size, which is also supported by our empirical evidence. Remark 1. FedEvalFair is based on locally obtained bootstrap fairness disparity distributions from each client, rather than on the raw data, so there is no assumptions about the clients' raw data.For the fairness disparity data on the 𝑖-th client, we can calculate the sample mean and sample variance as follows:𝑌 𝑖 = 1 𝑛 𝑖 𝑛 𝑖 ∑︁ 𝑗=1 𝑌 𝑖 𝑗 , 𝑆 2 𝑖 = 1 𝑛 𝑖 -1 𝑛 𝑖 ∑︁ 𝑗=1 (𝑌 𝑖 𝑗 -𝑌 𝑖 ) 2 .From this, we can estimate the population mean and variance of fairness disparitys for each client as follows:μ𝑖 = 𝐸 1 𝑛 𝑖 𝑛 𝑖 ∑︁ 𝑗=1 𝑌 𝑖 𝑗 = 1 𝑛 𝑖 𝑛 𝑖 ∑︁ 𝑗=1 𝐸 (𝑌 𝑖 𝑗 ) = 𝑌 𝑖 ; 𝐷 ( μ𝑖 ) = 1 𝑛 𝑖 2 𝐷 𝑛 𝑖 ∑︁ 𝑗=1 𝑌 𝑖 𝑗 = 𝜎 2 𝑖 𝑛 𝑖 ,where 𝐸 (𝑋 ) denotes the mathematical expectation, and 𝐷 (𝑋 ) denotes the variance of variable 𝑋 . We set 𝜇 𝑖 = 𝜇 + Δ𝜇 𝑖 , where 𝜇 𝑖 represents the population mean of fairness disparity for the model on the 𝑖-th client, and Δ𝜇 𝑖 represents the difference from the common mean 𝜇 across all clients. The hypothesis testing problem of interest is as follows:EQUATIONWe can divide this hypothesis testing problem into two stages: The hypothesis testing problem for the first stage is:EQUATIONThe second stage of our analysis involves a critical hypothesis testing problem, formulated as: 𝐻 0 : 𝜇 𝑖 = 0 vs 𝐻 1 : 𝜇 𝑖 ≠ 0, for 𝑖 = 1, 2, . . . , 𝑘.(EQUATIONThis test is versatile; for instance, it can be independently applied when a company seeks to evaluate the fairness of its model on a specific dataset. More significantly, in scenarios aiming for a comprehensive fairness assessment through the integration of multi-source private data, the application of this two-stage hypothesis test proves to be especially crucial. Addressing these hypothesis testing challenges is central to our study.Bootstrap is a powerful statistical method, whose core idea involves simulating the process of sampling from the original sample by repeated resampling, used to estimate statistical characteristics of sample data [29, 32, 39] . The Bootstrap method is divided into two main forms: parametric Bootstrap and non-parametric Bootstrap methods. The lies in that the parametric Bootstrap generates new sample sets based on the known statistical distribution of the sample, whereas the non-parametric Bootstrap does not assume any specific data distribution and directly performs replacement resampling from the original dataset to construct new sample sets. The application of non-parametric Bootstrap is particularly important for those data types whose statistical distributions are difficult to precisely define, such as images or audio. [2, 33, 34] In practice, suppose the original dataset is 𝑋 = {𝑋 1 , 𝑋 In model evaluation, the non-parametric bootstrap provides an approach that can be used to more accurately evaluate models and quantify their uncertainty, especially in complex environments where the data distribution is unknown. 
methodology 4.1 overview of fedevalfair: 
 As illustrated in Figure 1 , the FedEvalFair framework integrates the core concept of federated learning and places a special emphasis on enhancing data privacy. Unlike traditional evaluation methods, FedEvalFair avoids the direct access requirement to real datasets. Within this framework, instead of sharing their original data and model parameters, clients only need to share intermediate parameters related to the central aggregation for evaluation purposes. Additionally, we apply homomorphic encryption algorithms to encrypt the shared parameters. These measures render it impossible for semi-honest yet curious servers to infer any relevant information about the private data of clients, thereby safeguarding client data privacy.Moreover, another innovative aspect of the FedEvalFair framework is its reliance on statistical theory-specifically, the ideas of estimating populations from samples and statistical hypothesis testing-to conduct more trustworthy assessments of model fairness. Data holders independently utilize a non-parametric bootstrap method to create multiple bootstrap test datasets, to as comprehensively as possible evaluate the model's fairness performance on their respective datasets. This method effectively uses limited sample data to infer the model's fairness performance in a broader population data space. Based on statistical hypothesis testing theory, model fairness undergoes a two-stage rigorous hypothesis testing. Another significant advantage of the FedEvalFair is that evaluating model fairness on multi-clients' private data sources requires at most three rounds of communication. This not only demonstrates the practicality of the FedEvalFair method but also proves its effectiveness.In the following sections, we will elaborate on the two-stage fairness evaluation process using the FedEvalFair framework, focusing on the theoretical underpinnings of Algorithm 1 and 2.In this subsection, based on the principles of statistical hypothesis testing, we provide a rigorous theoretical derivation for the twostage fairness evaluation method, along with a detailed description of their algorithmic implementation processes.Next, we begin with the theoretical underpinnings of Algorithm 1. When the null hypothesis 𝐻 0 in (4) is true and 𝜎 𝑖 is known, drawing upon the concept of Graybill-Deal estimation [21] , the estimator for the common mean 𝜇 is derived asEQUATIONwhere𝜔 𝑖 = 𝑛 𝑖 𝜎 2 𝑖 𝑘 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 , 𝑖 = 1, 2, • • • , 𝑘. Then, a natural statistic isgiven byEQUATIONUnder the null hypothesis 𝐻 0 in (3), the mathematical expectation and variance of 𝑍 𝑖 are given by By Central Limit Theorem, it readily follows thatEQUATIONEQUATIONAs𝑛 𝑖 → ∞, 𝑍 𝑖 -𝐸 (𝑍 𝑖 ) √︁ 𝐷 (𝑍 𝑖 )follows a standard normal distribution, denoted by 𝑁 (0, 1), 𝑖 = 1, 2, • • • , 𝑘. Then, a natural statistic for testing (3) is given by𝑇 = ∑︁ 𝑘 𝑖=1 Ȳ𝑖 -𝑘 𝑖=1 𝜔 𝑖 Ȳ𝑖 √︂ 𝜎 2 𝑖 𝑛 𝑖 (1 -𝜔 𝑖 ) 2 = ∑︁ 𝑘 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 Ȳ𝑖 -𝑘 𝑖=1 𝜔 𝑖 Ȳ𝑖 2 1 -𝜔 𝑖 .Before delving into the detailed discussion of the test statistic 𝑇 , we first introduce Lemma 1, which provides a theoretical underpinning for our analysis. Here, 𝑎𝑠𝑦 ∼ is used to denote approximate adherence to a certain statistical distribution. EQUATION)As a result, we can set up the test rule, as given by𝑝 = 𝑃 𝑇 1 > 𝜒 2 𝑘-1,𝛼 ,where 𝜒 2 𝑘-1,𝛼 denotes the upper 𝛼-th quantile of a chi-square distribution with degrees of freedom 𝑑 𝑓 = 𝑘 -1. The null hypothesis 𝐻 0 in (3) is rejected whenever the above 𝑝-value is less than the nominal significance level of 𝛼, which indicates that at least two means are unequal. In the context of this study, 𝐻 0 is rejected if the model exhibits differing fairness disparities across at least two data sources.We now present Algorithm 1 that operationalizes the concepts discussed above. Next, we present the theoretical foundation for the second phase of fairness evaluation, which entails devising a solution to the hypothesis testing problem (5) . Subsequently, we will design a test statistic specifically for this hypothesis testing. Based on the Central Limit Theorem, we can deduce the following:𝑊 = 𝑘 ∑︁ 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 ( μ -𝜇) = 𝑘 ∑︁ 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 𝑘 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 μ𝑖 𝑘 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 -𝜇 .In the context of hypothesis testing problem (5) , when the null hypothesis holds true, we obtain the test statistic as follows:𝑊 * = 𝑘 ∑︁ 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 𝑘 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 μ𝑖 𝑘 𝑖=1 𝑛 𝑖 𝜎 2 𝑖 -𝜇 0 . If 𝜎 2𝑖 is known, then 𝑊 * serves as the test statistic for hypothesis testing problem (10) . However, in practical scenarios, 𝜎 2 𝑖 is often unknown. Therefore, we can substitute the maximum likelihood estimate of 𝜎 2 𝑖 into 𝑊 * , yielding the following result:𝑊 * = 𝑘 ∑︁ 𝑖=1 𝑛 𝑖 𝑠 2 𝑖 𝑘 𝑖=1 𝑛 𝑖 𝑠 2 𝑖 Ȳ𝑖 𝑘 𝑖=1 𝑛 𝑖 𝑠 2 𝑖 -𝜇 0 .It is evident that as 𝑛 𝑖 → ∞ for 𝑖 = 1, 2, ..., 𝑘, it follows that 𝑊 * ∼ 𝑁 (0, 1). Subsequently, we can establish the testing rule as follows:𝑝 = 2 × min 𝑃 𝑈 > 𝑤 * , 𝑃 𝑈 < 𝑤 * . where 𝑤 * is the observed value of 𝑊 , and 𝑈 represents a random variable following the standard normal distribution 𝑁 (0, 1). The null hypothesis 𝐻 0 in (10) is rejected if the p-value is less than the nominal significance level of 𝛼. This rejection indicates that: the model evaluation results on these distributed private data sources are unfair, implying that the model, when deployed in practice, exhibits bias towards certain groups with sensitive attributes.Building upon the theoretical framework, Algorithm 2 translates these ideas into actionable steps for implementation.The Second Stage of FedEvalFair Clients encrypt 𝜔 𝑖 , 𝜔 𝑖 Ȳ𝑖 , and upload to the server 10: end for 11: Server computes 𝑤 * on the EncryptedData 12: Server computes the 𝑝-value 13: Output: 𝑝-value 
experiments: 
 In this section, we first analyze the theoretical performance of FedEvalFair and its sensitivity to fairness perception through Monte Carlo simulation experiments. Subsequently, we validate the effectiveness of FedEvalFair on real datasets. Finally, we design a comparative experiment to test the stability of FedEvalFair in real-world scenarios dealing with non-independent and identically distributed (non-IID) data, and compare its reliability with that of traditional evaluation methods.5.1.1 Datasets and Models. We utilized the Adult [28] and eICU datasets [30] for federated learning training with the Logistic Regression model. For the Adult dataset, which includes over 40,000 records of adults, we employed the settings within the [13] , segregated the data into two clients: one encompassing individuals with a doctoral degree, and the other consisting of those without. During this process, we identified race (with a focus on White individuals) as a sensitive attribute, with the objective of predicting whether an individual's annual income surpasses $50,000. The training spanned 1800 rounds, with learning rates adjusted between 0.02 and 0.10. For the eICU dataset, this dataset contains over 200,000 samples, covering 17 different features. our attention was directed towards the patient data table, which detailed ICU admissions across various hospitals. This dataset was divided into multiple groups, with each group representing a consolidation of data from six hospitals, amounting to a total of 11 clients [24] . In this context, African American was designated as the sensitive attribute, aiming to forecast whether a patient's hospital stay exceeded one week. This training comprised 2300 rounds, with a consistent learning rate of 0.50.In the Monte Carlo simulations, we set a nominal significance level of 5% and carried out 5000 iterations. Similarly, we configured a wide range of parameter combinations to simulate the performance of the fairness assessment methods in different real-world scenarios. For details on parameter settings, please see Appendix and our GitHub: https://github.com/anonymous5929/FedEvalFair. All experiments are conducted on Ubuntu 22.04.3 server with 48 cores of 2.20GHz CPU, 256GB RAM, and NVIDIA V100 GPUs with 32 GB memory.5.2.1 Monte Carlo Simulation. In the Monte Carlo simulation experiments, we focused on evaluating the effectiveness and sensitivity of the FedEvalFair method in detecting unfairness. Since FedEvalFair is built upon the theoretical foundation of statistical hypothesis testing, its effectiveness is typically assessed by the Type I error rate. If the Type I error rate is close to the set significance level, it indicates that the likelihood of FedEvalFair mistakenly categorizing a fair model as unfair is extremely low. Regarding sensitivity to detecting unfairness, we measure this through the power of the FedEvalFair method. The higher the power, the more effectively the FedEvalFair method can identify unfair situations. Therefore, the experiments aim to validate that FedEvalFair maintains a low Type I error rate while demonstrating high sensitivity and detection capability for unfairness. As shown in Figure 2 , in the vast majority of scenarios, both Algorithm 1 and 2 effectively maintain the Type I error probability under the 5% nominal significance level. Notably, when the sample size is small, the probability of a Type I error may sometimes be slightly higher than 0.05. However, as the sample size increases, this error probability rapidly decreases towards 5%. This phenomenon indicates that while the performance of these algorithms may fluctuate under small sample conditions, they can stably control the Type I error at an ideal level with larger sample sizes.Figure 3 demonstrates the effectiveness of the FedEvalFair method in identifying unfairness, where an increase in efficacy indicates a stronger ability to recognize unfairness. As the fairness deviation increases, both Algorithm 1 and 2 show a significant improvement in efficacy, proving that FedEvalFair can quickly and accurately detect fairness disparity among clients and within the model. Notably, this detection capability significantly enhances with an increase in sample size, highlighting FedEvalFair's potential in monitoring model fairness and preventing fairness drift. Real-Datasets. Next, we turn our attention to experiments conducted on real datasets. Our main goal is to verify the effectiveness of the FedEvalFair method in real datasets. Specifically, we will explore key questions: First, when a model actually maintains fairness across multiple distributed data sources, can the FedEvalFair method accurately recognize and confirm this? Second, when a model is actually unfair, can the FedEval-Fair method effectively give a rejection conclusion and accurately identify unfair situations? To carry out our experiments, we referenced the FCFL algorithm proposed by Cui et al. [13] . This algorithm employs a multi-objective optimization strategy to train a fair federated learning model and notes that models trained without fairness constraints tend to be unfair. Based on this theory, we used the FCFL algorithm to simulate fair and unfair models on the Adult and eICU datasets.Firstly, consider the simulated unfair model. On the Adult and eICU datasets, the evaluation results based on Algorithm 2 showed that the p-value is far less than 0.0001. This indicates that the disparity value of the model is significantly different from 0. Similarly, we performed non-parametric bootstrap on the test set to mimic the data distribution in real-world scenarios as closely as possible. As illustrated in Figure 4 , the bootstrap disparity value of the model is also significantly different from 0, which corroborates the conclusion of the FedEvalFair. This suggests that if such a model is deployed in a real production environment by the data provider, it may exhibit bias against groups with sensitive attributes. We designed a new comparative experiment aimed at evaluating the reliability of the FedEvalFair method compared to existing evaluation methods (EEM) when assessing model fairness in real deployment environments. By summarizing existing research on fairness in federated learning( [28] , [37] , [13] ,etc), existing evaluation methods typically calculate the model's fairness disparity values based solely on test datasets, assuming that the test dataset can represent the data distribution encountered in actual deployments. However, limited test datasets often fail to fully reflect the data distribution in real environments. While existing methods can compare the strengths and weaknesses of different algorithms, they may lack accuracy when evaluating model fairness in actual deployments. In contrast, the FedEvalFair method adopts the concept of "estimating the population from the sample," aiming to infer the model's performance on real data as accurately as possible from limited datasets, thus better reflecting the model's fairness performance in production environments. For experimental details and the experimental framework, please refer to the appendix.Specifically, we designed a comparative experimental framework mainly to evaluate the stability of the EEM and the FedEvalFair method on partitioned test datasets and simulated real datasets. We considered various comparison scenarios: 1. When the distribution of the test dataset is completely consistent with the real production dataset; 2. When there is a varying degree of shift between the test dataset and the real production dataset; 3. When the test dataset and the real production dataset are completely different distributions. We used Cohen's d value to calculate the difference in results between the two evaluation methods under the test dataset and the real production dataset, reflecting whether there is a significant difference between the two sets of values. The larger the Cohen's d value, the more significant the difference, and a Cohen's d value greater than 0.8 indicates a very significant difference. As shown in Figure 6 and Table 1 , EEM already exhibits a Cohen's d value exceeding 0.8 when the distribution shift between the test dataset and the real production dataset is greater than a certain threshold, indicating a very significant difference. In contrast, while the FedEvalFair method also shows some difference, its stability remains superior to that of EEM. This finding suggests that the FedEvalFair framework has excellent reliability in assessing the fairness of models when deployed in real production environments. 
conclusion: 
 In this paper, we introduce FedEvalFair, a new framework that leverages statistical sampling and hypothesis testing principles along with multi-source private data to assess fairness in real-world model deployments, ensuring data privacy. By employing bootstrapping, FedEvalFair accurately evaluates fairness, utilizing a flexible two-stage strategy. This framework not only provides theoretical backing for federated learning applications but also promotes the advancement of trustworthy federated learning.  
 
 Appendix:  


Decision: The paper presents FedEvalFair, a new evaluation framework that assesses the fairness of federated learning models in real-world deployments while maintaining privacy. It tackles the issue of limited and biased fairness assessments caused by relying on test datasets. I believe this paper will provide valuable insights into privacy-preserving techniques in federated learning, and the proposed evaluation framework can stimulate further research in this area. Therefore, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes FedEvalFair, a privacy-preserving and statistically grounded federated fairness evaluation framework for assessing the fairness of machine learning models post-deployment. The proposed framework enables the evaluation of model fairness using multi-source private data without compromising data privacy, through the use of federated learning and statistical methods.
strength_weakness: strength_weakness: FedEvalFair is stated to be the first multi-source evaluation system dedicated to assessing model fairness. It introduces a novel approach to evaluating the fairness of models in real-world deployment environments.
limitations: limitations: 1. The motivation behind the framework seems unclear. 
- If the framework is intended for use in federated learning (FL) settings, it would be more appropriate to integrate it directly into the FL process to assess model fairness across various training rounds during the analysis and evaluation phase.  
- Conversely, if the framework is designed as a general tool for evaluating fairness among multiple clients, it falls short by not considering the practical FL factors such as client participation rate, data heterogeneity across clients and etc.
2. The related work section lacks thoroughness. The paper provides an overview of previous studies without clearly comparing their respective strengths, weaknesses, and specific application scopes. It would be better to divide the previous works into several categories and separately introduce their application scope and advantages.
3. The writing quality needs enhancement.
- In Section 3.1, the paper only presens the mathematical formulations of DP and EOD without offering insights into their meanings, derivation processes or distinctions. 
- The paper fails to articulate its improvement made to the Bootstrap method, especially in the FL scenario.
4. The evaluation of the framework is insufficiently. The paper does not benchmark its performance against existing fairness evaluation methods in FL.
5. The use of hypothesis testing to assess fairness is not adequately supported by existing references. The authors are expected to clairy its superiority to existing works on fairness evaluation.
6. The generalizability of the proposed method to other metrics is uncertain, which could constrain the overall impact of this work.
7. The evaluation is based on simplistic models and datasets, which may not adequately demonstrate the practical applicability of the proposed method.
paper_summary: paper_summary: This paper explores the fairness of federal learning, and the authors propose a model fair federal assessment framework named FedEvalFair. By adopting the concept of federated learning, the proposed framework is able to assess model fairness more reliably without direct access to data. In order to improve the operability, the authors developed a two-stage fairness test method to comprehensively evaluate the fairness performance of the model in a real-world deployment environment. In addition, abundant experiments and visual analysis are carried out to verify the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1. This paper proposes a novel research question and develop the first multi-source evaluation system specifically designed to evaluate model fairness. Ensure that the decision process is fair for all user groups by evaluating the fairness of the model at deployment time.
2. This paper propose a new principle of sample estimation of population parameters and the method of statistical hypothesis testing, a method of evaluating model fairness is designed. The quantitative analysis strategy constructed by this method is helpful to the experimental analysis of model fairness.
limitations: limitations: 1. The study did not analyze links to the multimedia community or the work's contribution to research in the multimedia field. The author should clearly state in the paper the significance, value, or potential impact of the research on the multimedia field.
2. Authors should specify which algorithms the baselinse in Table 1 are refer to, including reference information or links to sources.
3. There are some spelling mistakes in the paper (e.g., page 5, line 521).
4. In the writing of the paper, some formulas are missing labels (e.g., page 3, lines 301 and 309, etc.)
paper_summary: paper_summary: The paper introduces FedEvalFair, a novel evaluation framework designed to assess the fairness of federated learning models in real-world deployments while preserving privacy. It addresses the challenge of limited and biased fairness assessments resulting from reliance on test datasets. By integrating multi-client data and leveraging federated learning principles, FedEvalFair offers a comprehensive fairness evaluation without compromising data privacy. The framework incorporates statistical sampling methods and a two-stage evaluation strategy based on hypothesis testing, demonstrated through Monte Carlo simulations to provide reliable fairness analysis. Validation on datasets like Adult and eICU confirms its effectiveness. This work constitutes the first multi-source evaluation system dedicated to assessing model fairness, enhancing fairness measurement accuracy.
strength_weakness: strength_weakness: The paper proposes an innovative framework, FedEvalFair, aimed at solving the problem of fairness evaluation after the deployment of federated learning models, ensuring that model decisions are fair to all user groups while strictly protecting data privacy. This framework integrates private data from multiple parties, and combines statistical sampling theory and hypothesis testing methods to achieve a comprehensive evaluation of model fairness without compromising data privacy. Its core advantage lies in providing a two-stage evaluation strategy, which has been validated for effectiveness and stability through Monte Carlo simulation and on real-world datasets such as Adult and eICU.

For the ACM MM community, although the experiments in this paper focus on fairness evaluation of single-mode data, the framework proposed is expected to be applied in the future to federated learning and fairness assurance of multimodal data in the multimedia field.
limitations: limitations: The FedEvalFair proposed in the paper is novel, but the explanation of the paper is not clear.

1)	The paper does not explicitly mention which homomorphic encryption algorithm it adopts

2)	In the face of imbalanced classification, such as when a client lacks data about a certain classification (such as women), How FedEvalFair calculates DP

3)	Why divide the algorithm into two stages. These two stages do not seem to have a clear sequential relationship and can be merged into one stage.

4)	What does the arrow from Algorithm 1 to Algorithm 2 mean in Figure 1? It seems that Algorithm 2 does not rely on the results generated by Algorithm 1.

5)	The description of Figure 3 is unclear, especially regarding the missing explanation of the two coordinates at the bottom of the chart, which makes understanding difficult. This critical information should be presented in the paper rather than in supplementary materials.

6)	The description of Figure 6 is unclear. What does the horizontal axis (50,50) mean? What do different colors represent?

7)	This paper summarizes the EEM based on three relatively earlier papers (with the most recent one being only in 2021), without comparing FedEvalFair with more and recent methods.