SPACE: Single-round Participant Amalgamation for Contribution Evaluation in Federated Learning
Authors: Yi-Chung Chen
Hsi-Wen Chen
Shun-Guei Wang
Ming-Syan Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Contribution Evaluation
Shapley Value
Knowledge Amalgamation

Abstract: The evaluation of participant contribution in federated learning (FL) has recently gained significant attention due to its applicability in various domains, such as incentive mechanisms, robustness enhancement, and client selection. Previous approaches have predominantly relied on the widely adopted Shapley value for participant evaluation. However, the computation of the Shapley value is expensive, despite using techniques like gradient-based model reconstruction and truncating unnecessary evaluations. Therefore, we present an efficient approach called Single-round Participants Amalgamation for Contribution Evaluation (SPACE). SPACE incorporates two novel components, namely Federated Knowledge Amalgamation and Prototype-based Model Evaluation to reduce the evaluation effort by eliminating the dependence on the size of the validation set and enabling participant evaluation within a single communication round. Experimental results demonstrate that SPACE outperforms state-of-the-art methods in terms of both running time and Pearson’s Correlation Coefficient (PCC). Furthermore, extensive experiments conducted on applications, client reweighting, and client selection highlight the effectiveness of SPACE. The code is available at https://github.com/culiver/SPACE.

SPACE: Single-round Participant Amalgamation for Contribution Evaluation in Federated Learning
Yi-Chung Chen National Taiwan University (0942081@ntu, edu.tw
Hsi-Wen Chen National Taiwan University hwchen@arbor.c ee ntt. edu. tw
Shun-Gui Wang National Taiwan University r11921099@ntu, edu. tw
Ming-Syan Chen
National Taiwan University mschenEntu. edu. tw
Abstract
The evaluation of participant contribution in federated learning (FL) has re- cently gained significant attention due to its applicability various domains, such as incentive mechanisms, robustness enhancement, and client selection. Previous approaches have predominantly relied on the widely adopted Shapley value for participant evaluation. However, the computation of the Shapley value is expensive, despite using techniques like gradient-based model reconstruction and truncating unnecessary evaluations. Therefore, we present an efficient ap- proach called Single-round Participants Amalgamation for Contribution Evalu- ation (SPACE). SPACE incorporates two novel components, namely Federated Knowledge Amalgamation and Prototype-bused Model Evaluation to reduce the evaluation effort by climinating the dependence on the size of the validation set and enabling participant evaluation within a single communication round. Ex- perimental results demonstrate that SPACE outperforms state-of-the-arr methods in terms of both running time and Pearson's 's Correlation Coefficient (PCC). Fur- thermore, extensive experiments conducted applications, client reweighting, and client selection highlight the effectiveness SPACE. The code available at https://gihuubccom//ll/e///////
Introduction
With the rapid growth of mobile services, the combination of rich user interactions and robust sen- sors means they are able access an unprecedented amount of data. While conventional machine learning approaches require centralizing the training data in a single machine or a data center, which may limit the scalability of machine learning applications, Federated Learning (FI.) (36) has been proposed to leave the training data distributed on the client side and then aggregate lo- cally computed updates, e.g., the gradient of the neural network, to a central coordinating server, without accessing the raw data. A line of subsequent efforts has significantly improved the effi- 63) FL. Moreover, such distributed training ciency [5] 30 62 70 733, and robustness [17 38
paradigm has been adapted to client personalization 8 27
40 and heterogeneous model archi-
tecture 10 75 making it even more practical.
Recently, there has been increasing interest in evaluating the participant's contribution to FL for fair incentive mechanism 112 29 33
[9]
45
46 65] 69
72] based on the participant contribution
in attracting data owners. Additionally. ly, deciding the participant weights [54 (7 during the training process based on their contributions can accelerate the convergence of the training process but also enhance the model's robustness by alleviating the negative effect from the confused or even incorrect
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

data. Moreover, if the communication budget is limited. participant contributions can be utilized as a reference to select important participants 7 while maintaining the model's performance.
53 56 73 reducing communication costs
One straightforward method to evaluate participant contribution is assess their individual per- formance independently. For instance, the accuracy of the local model has been employed as a measure [39] as well as the consistency between the local and global models, indicating their sim- ilarity to the consensus of the coalition [45] 72). However, these approaches do not fully consider the impact of client cooperation. To consider cooperation, some studies have adopted the influence function [21) from centralized data evaluation to assess participant contribution [57 64). However, calculating the influence function involves clients sending their locally computed Hessian matrices to the server, leading to unacceptable client computation and communication costs. On the other hand, researchers have explored using the Shapley value [22 in cooperative game theory to measure participant contributions. While the exact computation of the Shapley value requires enumerating all possible client combinations, Song et al. [48] approximate the target model with gradient updates to avoid model retraining. To further speed up the computation, Wei et al. [59] and Liu et al. [32] propose truncation techniques, eliminating unnecessary model reconstruction and evaluation.
Despite the aforementioned efforts, the approximation cost of the Shapley value can still be pro- hibitively high for the following two reasons. i) Multi-Round Training: FL algorithms, such as FedAvg, involve multiple communication rounds wherein models are transmitted between the server and clients, and local training takes place on the client side before achieving model convergence. Previous studies (32 48 57-59 compute participant contributions for each communication round and aggregate them to obtain the overall participant contribution. Consequently, the communica- tion and computation efforts escalate with an increase in the number of communication rounds. ii) Pataset-Dependeee Evaluation: As demonstrated in [32), the evaluation of models constitutes the primary computational time component required for calculating the Shapley value. Even with the adoption of permutation sampling approximation, the complexity of evaluating per-round contribu- tions depends on the size of the validation dataset on the server. As the validation set continually grows, the evaluation process becomes computationally unacceptable.
In this paper, we propose Single-round round Participants Amalgamation for Contribution Evaluation (SPACE) to efficiently calculate the participant contribution in FI.. Specifically, we analyze the data distribution between the server and the client dataset. If the server and client distributions are similar, the client dataset proves helpful in correctly classifying the data in the validation set of the server. While previous works 51 52, 68) have shown that prototypes, i.e., the embedding vectors repre- senting each class, can be employed as a crivacy-preservvv we propose Prototype-bused Model Evaluation, which evaluates the model performance by mea- suring the similarities between the local and server prototypes. Therefore, the distribution of the validation set can be constructed and thus eliminates the dependency of evaluation complexity on the size of the validation set.
approach to compare data distribution,
To derive meaningful prototype embeddings, we propose a novel approach called Federated Knowl- edge Amalgamation, wherein client models can be viewed as multiple distributed teacher models, and their knowledge can be distilled to educate the server model simultaneously. Thus, federated knowledge amalgamation enables us to distill knowledge from all local models in just one communi- cation round to reduce the time for prototype construction. The educated model is distributed to each participant, who then constructs prototypes based on their local data. We then evaluate the model performance by measuring the similarities between server and client prototypes. Besides, unlike previous works that adopt the model performance as the utility function, we modify the utility func- tion by incorporating logistic function. This adjustment aims to reflect users' satisfaction better, thereby enhancing the rationality of the utility function in real-world scenarios. Extensive results demonstrate the advantage of SPACE in multiple applications, including contribution evaluation, client re- re-weighting, and client selection.
The contributions of this paper are summarized as follows:
We propose SPACE, an efficient and effective approach for client contribution evaluation in an FL setting, which significantly reduces the required communication and computational cost and exhibits outstanding performance in terms of Pearson's 's correlation coefficient (PCC).
We introduce Prototype- Model Evaluation, which effectively removes the computation de- pendency on the size of the validation set. To efficiently establish robust embedding space for

---

building prototypes, we introduce Federated Knowledge Amalgamation that enables the aggrega- tion of embedding space information from local models in an FL setting, all within a single round of communication.
Extensive experiments have been conducted to show that SPACE outperforms previous state-of- the-art methods in terms of both running time and PCC. SPACE consistently exhibits exceptional performance applications including client reweighting and selection.
2 Related Works
Contribution evaluation in FL. Client contribution evaluation in federated learning (FL) has gained significant attention for its potential upplications, including attack detection [1] 31 de-
bugging 25 26]. incentive mechanisms 33
45
46
48
55
65
[2], and convergence accelera-
tion (54 This concept draws inspiration from centralized data evaluation tasks [66 19 Recent
works have proposed methods tailored for FI.. Pandey et al. 33 consider the relative accuracy of the local model individual contribution. Zhang et al. [72] calculate the cosine similarity between local gradients and unit vector pointing towards the optimal model as contributions. Shi et al. [4S] measure contribution based on the cosine similarity between local and global gradients. However, these methods do not fully capture the impact of cooperation. To address this limitation, the Shap- ley value is widely adopted for evaluating client contribution in FL due to its desirable properties, including efficiency, symmetry. linearity, and null player property. However, directly computing the Shapley value incurs exponential computational costs involving model retraining and inference. To alleviate this burden, Song al. (48) reconstruct the target model using gradient updates to avoid model retraining. Wang et al. (58) enhance the evaluation using permutation sampling and a group testing approach. Additionally, Wei et al. [59) and Liu et al. (32 incorporate truncation techniques to reduce computation costs. Despite these efforts, approximating the Shapley value still requires
significant computation.
Knowledge Amalgamation. Knowledge amalgamation (KA), a variant knowledge distilla- tion [18,, involves setup with one student model and multiple teacher models. KA has been widely 61 66 67 due to its desirable properties. Shen et al. [43] introduced KA, where compact feature representations are learned from teachers and used to update the student model. Shen et al.
adopted in prior works [5] 20 34
43
44
44 proposed a strategy that extracts task-specific knowledge from diverse teachers create component networks, which then combined to construct the student network. Luo et al. (34] introduced a common feature learning scheme in heterogeneous network settings, transforming the features of all teachers into a shared space. Xie et al. [6]] proposed related approach applying KA to transfer prior knowledge from multiple pre-trained models. How- ever, their focus is knowledge transfer from given pre-trained models, while our approach the first to employ KA within a conventional FL framework. In our setting, client models can be viewed as distributed teacher models, and their knowledge can be distilled to educate the server model.
3 Preliminary
Since federated learning aims to construct a unified, global machine-lcarning model to collect data information located on numerous remote devices (client), the objective is to develop this model while ensuring that the data generated by each device is processed and stored locally, with only intermediate updates being sent central server regular intervals. The primary objective is usually to minimize the objective function as follows. Let W denote set consisting of n clients,
0- arg min
Σ pL),
where p; 2 0 and Σ, may utilize the same objective function, such as the empirical risk, which can be defined as follows.
P: 1. L; is the objective function for the i-th client. Generally, each client
L
Σ
1(0, 23 Vi.)),
(2)
where D, is the dataset i-th client. Ps specifies the relative impact of each device, such as the uniform weighting 1 or depending on the number of samples in each dataset [P]]
[D,
3

---

Federated Knowledge
Server
Contribution Evaluation On Server
Validation set (Py)
Amalgamation
sPMs
Common Emtedding Space
f
Pepertanti
fyrer
fyeat
Ayeat
Soles conlion
Conatric 20000o
URIR Colourdae
PPM
FPM
IPM
FMM
000
6"
Client
Client
Client
Client (I))
Client (D:)
Client (2.:)
Figure I: Overall architecture of SPACE. Initially, the clients train their respective local models using their local datasets. Subsequently, the locally optimized models are frozen and serve as teacher models to educate the student model the server by Federated Knowledge Amalgamation. Then, the educated model 0* is distributed to all clients, who then utilize it to construct their corresponding prototypes P,. These prototypes are then uploaded to the server. The server employs the Prototype- based Model Evaluation approach to assess the model's performance and determine the contribution of each participant.
Despite early efforts to enhance the efficacy of federated learning, many data owners hesitate when it comes to partaking in data federation owing to apprehensions surrounding the possibility of in- equitable remuneration. Furthermore, in federated learning, privacy concerns preclude the from directly accessing unprocessed data stored on client devices. Consequently, there is a press- ing need for a privacy-preservin federated contribution evaluation method that ensures equitable distribution of rewards.
Specifically, the evaluation of participant contributions can be abstracted into cooperative game scenarios. Given cooperative game (N, V), where N denotes the set of n participants and (.)) is utility function defined V 2 - R. which assigns a value coalition S E A,, the objective of payoff allocation to determine a payoff allocation vector, denoted by = (Ci,..., Cn), where c; represents the payoff for the i-th participant in N.
The idea of cooperative games has been extensively researched 2 3 with the Shapley value being the most commonly used metric due to its existence and uniqueness in cooperative game. Addi- tionally, the Shapley value is characterized by a set of desirable axiomatic properties, including (i) efficiency, (ii) symmetry, (iii) null player, and (iv) lincarity, which make it desirable approach for evaluating contributions. The Shapley value is defined as follows,
t. (N,
Σ SENVER
[S
[S| 1)! (V(SU V(S)).
(3)
S denotes the subset of participants from N. The Shapley value can be seen the average marginal gain that a participant joins the coalitions. To accelerate the approximation, the Monte Carlo esti- mation method for approximating Shapley values from [16] is adopted,
E (V(S (i)) V(S)),
(4)
where St is the set of data from FL participants joining before i in the T-th permutation of the sequence FL participants. The calculated Shapley values are clipped be non-negative to follow the non-negative assumption as in 3
4 Method
Shapley value a widely used approach for evaluating participant contributions in federated learn- ing due to its desirable properties. However, despite the proposal of various acceleration techniques

---

such as Monte Carlo sampling, gradient-basee model reconstruction, and truncation, the compu- tation of the Shapley value remains prohibitively slow due to the following two key challenges Multi-Round Training: multiple communication rounds are required for model convergence, and Dataset-dependent Evaluation: the evaluation of model performance depends on the size of the validation set on the server. Thus, we propose the Single-round Participant Amalgamation for Con- tribution Evaluation (SPACE) approach, which analyzes participant contributions using only one communication round. The key idea behind SPACE is to compare the distribution between the local dataset and the validation dataset directly via prototype embeddings of the local dataset and the val- 68]. To establish robust prototype embeddings, we introduce Federated Knowledge Amalgamation to distill feature information from all locally optimized models to the server model simultancously, which enables the server model to lear information in single com- munication round. Once the prototypes are obtained, we evaluate the performance of coalition S by comparing the similarities between the prototypes formed by clients in coalition Ps and prototypes formed by validation set Py. Using prototypes eliminates the need to iterate over all samples in the validation set, resulting in faster performance evaluation. Figure shows the architecture of SPACE.
idation dataset [37 [51 52
4.1 Federated Knowledge Amalgamation
As previously mentioned, the SPACE framework utilizes prototypes to assess participant contri- butions. Therefore, employing semantically meaningful deep model is essential to ensure the extraction of expressive features from data samples. One straightforward approach is to utilize the converged model achieved through federated training, which encompasses training with all the local data. However, the convergence of federated learning algorithms like FedAvg necessitates numerous rounds of local-global communication and local training. Consequently, the evaluation of contribu- tions can only be conducted after extensive training, resulting in excessively high costs and impeding the utilization of participant contributions in applications like client selection.
In contrast to conventional knowledge distillation tasks, which typically involve a single student model and single teacher model, knowledge amalgamation techniques are employed to entail one student model and multiple teacher models. Furthermore, in federated learning, the client models can be considered teacher models that acquire knowledge from their local datasets. Consequently, the application of knowledge amalgamation facilitates the construction of an expressive embedding space for federated learning.
SPACE first distributes the global model to each client and each client implements empirical risk minimization to obtain locally optimized model 0, The knowledge amalgamation is then applied to transfer the knowledge to the initial global model 9 with the locally optimized models serving as teacher models and the server validation dataset for distillation. To simultancously distill knowledge from all the teacher models, the features of both the student and teacher models must be projected to common feature space for the feature alignment.
Therefore, we use feature projector modules (FPM) to project the intermediate feature of models to the common feature space. Since the data distributions of clients are different, the projections from the original feature space to the common feature space are also different. Therefore, each model, including the student model and teacher models, would be assigned an individual learnable FPM for projection. After projection, the student model then starts to learn information from teacher models by minimizing the feature distillation loss Leett which is the L1 distance between the projected student features f, and the projected teacher features f... However, due to the differences in local data distribution, the teacher models are not equally important, and the loss should be weighted differently depending on the sample used for knowledge amalgamation. We define the weighting of each teacher model w, by its confidence in predicting the class of data sample. Specifically, given a data sample with its class as j for distillation, the weighting of each teacher model is the softmax of the j-th logit of its prediction F(0;, d), which can be written as follows:
F(9g,d)s
Lfeat= = Σ w. ff - -f,,, wi = E,
(5)
To prevent the FPM from projecting all the features to a constant vector, which is a trivial opti- mal solution for the feature distillation loss, reconstruction loss is proposed to ensure the projected feature contains the knowledge in the original feature space. For each teacher model, we apply an inverse feature projector module (iFPM) to reconstruct features in the original feature space, which
5

---

projects f. to the reconstructed feature f...
The reconstruction loss is defined as follows:
Lvee Σ
fr. f.,l
(6)
where f;; the original feature of i-th client before projection. Aside from feature distillation, we also use prediction distillation to enhance the overall amalgamation performance. For prediction distillation, the student model only learns from the teacher that predicts the highest confidence in the correct prediction. The prediction distillation loss LKL is defined as follow:
LKL (L(F((e, f), F(Oucst; d)), Obrat arg max (F),..))))
where (-) denotes the KL divergence loss. The overall amalgamation loss Lamig becomes, Lamig Af + AreeLvee + AKLLKL, where À are used for balancing the loss terms. Af- ter knowledge amalgamation, the server model 0" is then distributed to each client for extracting features and building prototypes of each category.
While the theoretical complexity of knowledge amalgamation is (n)), practical considerations arise as the number of clients increases, leading training. This poses a challenge due to the limited memory capacity of GPUs, as training all FPMs simultaneously may not be feasible. To address this issue, suggest hierarchical amalgamation framework that alleviates the high GPU memory requirement. By allocating a GPU memory budget, amalgamate groups of G teacher models instead of all teacher models at once. Although this modification increases the complexity of knowledge amalgamation to O(n log n), it enables the implementation of knowledge amalgamation on any GPU device capable of simultaneously amal- gamating at least two teacher models.
increase in the number of FPMs that require
4.2 Prototype-based Model Evaluation
The concept of utilizing prototypes in classification models has its roots in the Prototypical Networks introduced by Snell et al. [47). which were initially proposed for addressing few-shot classification tasks. In this framework, the classification of an embedded query point is achieved by identifying the nearest class prototype. The underlying assumption is that an embedding space exists where data points tend to cluster around single prototype representation for each class. By leveraging prototypes, it becomes unnecessary to compute the distances between all samples in the query set and the support set, as required in Sung et al. (50). The prototypes capture the distribution of samples in the support set, simplifying the classification process.
In study, adopt similar perspective Snell et al. 47 by considering the clients' local datasets as the support set. From the embedding space derived through the Federated Knowledge Amalgamation described earlier, we construct prototypes for each client, denoted as p(") represent- ing the prototype for class j of the i-th client. To address the challenge of evaluation complexity dependence the size of the validation set, we introduce a modification to the evaluation step. Rather than iterating through all validation samples, we devise a strategy where prototypes are built for the server's validation set. These prototypes, denoted as Pl.) repreeeen the prototypes for class j of the validation set. Model evaluation is then performed by comparing the similarities of the proto- types between the clients and the server. This modification effectively eliminates the computational burden associated with iterating through all validation samples. The prototypes are built as follows:
p!u)
1
Σ (E,))E,,,
f(0'.x)
(8)
D
where Di.j implies the data that is class j in the client i-th dataset and function f(-) is used to extract feature from data æ with model weight 8* We set the prototype of clients as a zero vector the
client does not possess any data of the corresponding class, such that p()) = 0 if (Disl = 0. When clients cooperate in coalitions, the prototypes for class j of coalition is denoted as p!!) which can simply be achieved by the weighted sum of prototypes in the coalition.
Σ Eies
[Da,y
[ p())
(
We then define the performance of the coalition by the confidence of correctly classifying the given prototypes of the validation set. The confidence is calculated by the relative value between the

---

similarity of the same class and the sum of the similarities of all classes, while softmax function applied to ensure the value lies within the range [0, 1]. To be more specific, the value is computed as follows:
Sim(pi)) pg
V'()
Σ
(10)
C
exm(pp)) .p(
where [C] represents the number of classes in the classification task. We adopt cosine similarity as our similarity function.
4.3 Contribution Evaluation
Shapley value is applied evaluate the participant contribution. However, adopting the model performance V' as a utility function may usually lead to a violation of rationality. This violation occurs when the marginal performance gain from additional clients decreases as the coalition size increases, as observed in [32). Consequently, the expected value of the marginal gain becomes smaller than the individual value, thereby violating individual rationality [3] which states that c, 2 u((t), where u(-) denotes a utility function. To illustrate this issue, we provide an example below:
Example 4.1. Given two clients, each client's local dataset allows them achieve model accuracy of 0.8 and 0.6, respectively. However, when collaborating in a federated learning setting, they can attain model performance of 0.9. The Shapley value of each participant can be determined by using the model performance as the utilityy function, resulting in Shapley values of them becoming 0.55 and 0.35. Thus, their payoft's are lower with cooperation.
Despite the seemingly irrational nature of the decision, data owners in real-world scenarios demon- strate a willingness participate in the federation in the above example due the higher level of user satisfaction achieved by model with an accuracy of 0.9. Relying solely on model performance as the utility function fails adequately capture the underlying user satisfaction. Based on this ob- servation, we rectify the conventional utility function with a logistic function [9] representing the percentage of users satisfied with the model performance.
V(S) e (VV
(11)
where i denotes the logistic growth rate, which determines the steepness of the curve and T repre- sents the threshold, where half of the population is satisfied. The choice of T should be greater than max, V'({+) to encourage the cooperation of participants because the convexity of the left-sided logistic function would up-weight the utility value of stronger coalitions. When i is set to infinity, the individual rationality would be satisfied since the V((i)) is compressed to zero and participant contribution is lower-bounded by zero.
Finally, the contribution of i-th participant can be calculated by Shapley value defined in Equation The proposed rectification of the utility function with the logistic function makes the utility value more sparse because the utility of the coalition whose performance does not surpass the threshold T would be highly compressed. Inspired by (32), we propose pruning technique that leverages the sparsity to further accelerate the computation of the Shapley value. We set a pruning threshold т, such that given coalition S with V(S) < r, prune the computation of subsets of S.
4.4 Complexity Analysis
Table 1 presents a comparative analysis of SPACE's computational complexity in contrast to estab- lished methods. The sampling-based approximation of the Shapley value, applied in TMC-Shapley, GTG-Shapley, and SPACE, necessitates sampling complexity of n log ni, as studied in [35]. It is worth noting that SPACE achieves notably reduced computational complexity in calculating the Shapley value at merely (n logn [C]), which represents a substantial reduction compared to the On logn Ry [Dyl) required by other Shapley value-based approaches. This efficiency gain is attributed to the use of the single-round amalgamation and the prototype-based model evaluation. The prototype- based model evaluation eliminates the necessity for repeated iteration over the valida- tion set. Consequently, the computation of the Shapley value is facilitated by direct comparisons of
Further discussion is provided in the appendices.
"The overall algorithm of SPACE is shown in appendices.
7

---

Table 1: Comparison of complexity between previous works. R. Rg and Rg denote the number of iterations for local training, global communication rounds, and knowledge amalgamation. |0] rep-
resents the number of model parameters, [P] denotes the dimension of prototypes, and |C] indicates the number of classes. [D]] and [Dy] represent the sizes of the local dataset and the validation set.
Method
Communication Cost
Client Computation Cost
Server Computation Cost
Real Shapley
2" Rg |0
2" Rg R4 [D.]
2" [Dvl
GT-Shapley
n(logn) m)² Rg |0]
nloogn) Rgg RA [D,]
<llog n)* [Dyl
TMC-Shapley
n log n Rg [0]
n bloog n Rg Rí [D:]
n log n [Dyl
GTG-Shapley
Rg (0)
Rg R [Da]
n nlog n Rg Dvv
DIG-FL
Rg 10)
Rg Ri [Da]
Rg Dv
SPACE(OUrs)
Pl
R₁ [Da]
nog Ra- Dv
distances between the weighted sum of prototypes. Moreover, by employing the knowledge amalga- mation technique, SPACE is capable of evaluating participant contributions with single communi- cation round, thereby significantly alleviating the communication and computational load on clients with limited resources by factor of Rg. This acceleration does introduce an additional computa- tion load for knowledge amalgamation on the server, with complexity of O log n R. [Dvl). However, this trade-off is generally more suitable for generic federated learning frameworks, given that the server typically possesses superior computational resources compared to the clients.
5 Experiment
In this section, we demonstrate the effectiveness of the proposed SPACE method for evaluating participant contributions. We compare it with previous approaches and show that SPACE achieves superior computational efficiency without compromising performance. We also evaluate the SPACE on two leaarinn tasks, client reweighting, and client selection.
5.1 Experimental Setup
Datasets and Non-IID Setting. Following 37 we conduct experiments on the widely adopted image dataset MNIST (24) and CIFARIO [22 For fair comparison, we adopted the experimental settings used in the HFL framework [57] with two different scenarios: mislabeled data and non- IID data. the mislabeled data scenario, we deliberately introduced label corruption
variable proportion of the total clients. Specifically, we corrupted different percentages of clients, ranging from 0% to 80%. Within each client, we intentionally corrupted 50% of the local training labels to simulate the presence of mislabeled data. For the non-IID scenario, selected subset of clients, ranging from 0% to 80%, and assigned them local datasets containing incomplete categories. We set the total number of clients to 10 for the MNIST dataset and 5 for the CIFARII 10 dataset.
Baselines and Implementation. We compare SPACE with four state-of-tt of-th are methods, including i) GT-Shapley [19)ii TMC-Shapley [16]. iii) GTG-Shapley [32) and iv) DIG-FL [57]. We also compare a variant of SPACE, v) SPACE(Avg). which applies FedAvg instead of knowledge amalga- mation for aggregation. For adjusting the utility function, we empirically set & as 100 while T as 0.95 and 0.5 for evaluation on MNIST and CIFAR10.
Evaluation Metric. The performance of all approaches assessed from two perspectives: accuracy and efficiency. To evaluate the accuracy, we employ Pearson's Correlation Coefficient to measure the correlation between the estimated and actual Shapley values. The actual Shapley values are obtained by performing retraining for a total of 2 times. In terms of efficiency, we consider two vital metrics. Execution time (in seconds) is measured on a single V100 GPU without parallel training to assess the time efficiency of the approaches. Communication cost (in megabytes) is calculated according to the formula Comm = 2 n Rg 10), where R, denotes the number of communication rounds, [0] represents the number of model parameters(MiB) and constant of stands for the upload and download of the models.
"Since Cha-lly and ThC-Shapley are centralized learning schemes, we set the retraining rounds for TMC-Shapley to log and for GT-Shapley to nılog n2)², where is the number of participants for the federated learning scenario.
8

---

Table Results of participant contribution evaluation.
Dataset
Scenario
GT
TMC
GTG
DIG-FL
SPACE(Avg)
SPACE
Non-III
0.6877
0.9824
0.9287
0.87IS
0,9713
0,9448
Mislabel
0.4230
0.9507
0,9608
0.9580
0,9529
0.9612
MNIST
Time(s)
97137
84796
62473
315
294
160
Comm(MB)
11813.12
10292.48
35.2
35.2
35.2
1.76
Non-IID
0.6089
0.8877
0.8208
0.7546
0.9540
0,9290
Mislabel
0.5192
0,9595
0.4148
0.9598
0,9565
0.9641
CIFAR
Time(s)
7468
4950
835
536
315
295
CoMm(MB) 307800
202920
11400
11400
11400
570
MIAST 1D
VIIST Mistbbl
OTARCO Non
CERATO Mebsel
500
37.
64003 800
05-
96.3
17:2
17.0
12-
953
of clert
siabbel
Pertertge dierti Pn
midabe clents
Figure 2: The effect of client reweighting on model performance.
5.2
Quantitative Results
Participant contribution. Tabee[psrseetts the outcomes of participant contribution evaluation. The proposed SPACE approach demonstrates outstanding efficiency and achieves the highest Pearson correlation coefficient (PCC) in scenarios involving mislabeled data. In terms of PCC, SPACE per- forms similarly to SPACE(Avg), but with superior efficiency resulting from the single-round knowl- edge amalgamation. The communication and computational costs associated with GT-Shapley and TMC-Shapley are excessively high due to the requirement for model retraining. However, TMC- Shapley shows improved efficiency by attaining high PCC results with fewer sampled permutations. GTG-Shapley adopts a reconstruction technique to expedite the computation process, which ef- fectively reduces the required communication cost, yet the testing time in GTG-Shapley remains considerably long. The resource-saving version of DIG-FL achieves comparable computational ef- ficiency to SPACE. Nevertheless, it has tendency to overestimate the contribution of clients with incomplete categories and exhibits inferior PCC results non-lII scenarios.
Client Reweighting. In non-IID scenarios or when some clients have erroneous data, aggregating local models based on data ratio may lead to suboptimal performance. Leveraging participant con- tribution as a weighting factor can enhance the robustness of federated learning [57]. We propose two reweighting mechanisms, namely static and dynamic, with the former employing participant contribution calculated using the knowledge amalgamated model and the latter recalculating par- ticipant contribution for each communication round using the current model. Figure shows the reweighting results. Despite the dynamic approach introducing additional computational overhead, it yields improved performance. The static and dynamic approaches achieve accuracy improvements of 2.53% and 2.69%, respectively, under the non-llD scenario, and 3.21 % and 3. 3.52%, respectively, under the mislabel scenario, for the CIFAR1O 10 dataset. The DIG-FL approach measures client con- tribution in federated learning by the similarities between the gradients of the local datasets and the gradients of the validation set. However, this approach disregards clients in communication rounds where the two gradients diverge, leading to significant variances in the federated training process.
Client Selection. Client selection is a critical issue when dealing with large number of clients. Li et al. (8 proposed client sampling approach based on the multinomial distribution (MD) that ensures zero bias but has significant variances. To tackle this issue, Fraboni al. [3] suggested clustered sampling (CS), which has been theoretically shown to reduce variance. They proposed two CS algorithms: Algorithm I clusters clients based on data ratio, and Algorithm incorporates the similarities between local and global models. Algorithm 2 outperforms both MD and Algorithm
1. However, Algorithm 2's time complexity is O RRy (n²10 [ + X)), where ( is the number of
model parameters and O(X) is the complexity of the clustering algorithm. This computational cost

---

=0
a= 0.001
67.5
45
65.0
40
Accuracy (%)
35
30
Sampling
57.5
55.0
CS(Alg.1]
20
CS(Alg.2)
52.5
CS(SPACE)
15
200
400
600
200
600
800
1000
rounds
rounds
Figure 3: The effect of client selection on model performance.
results from updating a similarity matrix and re-clustering at each communication round. We pro- pose a method that combines SPACE with CS. We utilize the prototypes obtained from SPACE to construct the similarity matrix, eliminating the need for repeated updates. Additionally, we enhance the weighting by incorporating the estimated contribution, resulting in a new weighting scheme for client P; given by (1 B)p: + Beg, where
hyper-parameter that balances the two terms. As in [13], the CIFAR 10 dataset is distributed among 100 clients using Dirichlet distribution with 0 and 0.5 when
{10, 0.001}, where smaller a implies larger heterogeneity. We set = 10 and a 0.001, respectively. Figure prrusettssssntts the results of client selection. Our approach, which combines prototypes and participant contribution, outperforms prior methods. indicating the superiority of our proposed method. This improvement can be attributed to the fact that prototypes enhance the accuracy of the clustering process. Furthermore, as discussed in Section 5.2 the corporation of participant contribution has proven to be useful in enhancing the performance of federated learning, especially in non-IID settings.
Conclusion
We propose Single-round Participant Amalgamation for Contribution Evaluation (SPACE), an effi- cient approach for evaluating participant contribution in federated learning. The novelty of SPACE lies in two key components: Federated Knowledge Amalgamation and Prototype-based Model Eval- uation. By leveraging single communication round, Federated Knowledge Amalgamation con- structs robust embedding space, while Prototype-based Model Evaluation reduces the complexity associated with the validation set size. Our experimental results demonstrate that SPACE outper- forms existing methods in contribution evaluation and shows versatility in client reweighting and selection. However, efficient client selection for amalgamation and the impact of the validation set quality are areas that require further exploration in future research.
Broader Impact
In study, highlight the importance of participant contribution through three applications: in- centive mechanisms, client reweighting, and client selection. Furthermore, the potential applications extend to client debugging and attack detection. Since the computational complexity associated with participant contribution hampers its widespread implementation, we propose a novel FL framework, SPACE, to alleviate this issue and pave the way for broader adoption of participant contribution. An efficient algorithm for evaluating participants provides decision-maken with valuable insights before initiating extensive federated learning processes, which can be both computationally and temporally intensive. Furthermore, rapid evaluation offers enhanced flexibility when dealing with dynamic cooperative entities, facilitating quick adaptation to new collaborations-a capability that previous methods lack. Our proposed approach, SPACE, significantly contributes to these applica- tions. However, our framework relies on the assumption having proper and reliable validation set with accurately labeled data. This prerequisite can be particularly challenging to fulfill in real-world settings and thus limits the application to specific scenarios characterized by significant commercial incentives 33 65 Exploring methods to decrease dependence the validation set while maintain- ing functionality, even when erroneous data impact many clients, presents an intriguing direction for future exploration. We defer this critical aspect to be addressed in future research efforts.
10

---

Acknowledgement
This work was supported in part by the National Science and Technology Council, Taiwan, under grant NSTC-111--223---000-000 and by the Ministry of Education, Taiwan, under grant MOE 11219009.
References
[1] S. Bibikar, H. Vikalo, Z. Wang, and X. Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In AAAI Conference Artificial Intelligence, 2022.
[2] S. Brânzei. State the art: Solution concepts for coalitional games. 2010.
[3] G. Chalkiadakis, E. Elkind, and M. Wooldridge. Computational aspects of cooperative game theory. In Computational Aspects of Cooperative Game Theory, 2011.
[4] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsich. Zoo: Zeroth order optimization based black- -ox attacks to deep neural networks without training substitute models. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017.
[5] W.-T. Chen, Z.-K. Huang, C.-C. Tsai, H.-H. Yang, J. Ding, and S. Kuo. Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regulariza- tion: Toward a unified model. 2022 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 17632-17641, 2022.
[6] Y. Chen, Gui, H. Lin, W. Gan, and Y. Wu. Federated learning attacks and defenses: A survey. 2022 IEEE International Conference on Big Data (Big Data), pages 4256-42663 2022.
[7] Y. J. Cho, J. Wang, and G. Joshi. Towards understanding biased client selection in federated learning. In International Conference Artificial Intelligence and Statistics, 2022.
[8] L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai. Exploiting shared representations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[9] S. Cramer. The origins of logistic regression. Econometrics eJournal, 2002.
[10] E. Diao, Ding, and V. Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[11] G. Ding, Z. Li, Y. Wu, X. Yang, M. Aliasgari, and H. Xu. Towards efficient client selection system for federated learning. In JEEE International Conference on Cloud Computing, 2022.
[12] N. Ding, Fang, and J. Huang. Incentive mechanism design for federated learning with multi- dimensional private information. In International Symposium on Modeling and Optimization Mobile, Ad-Hoe and Wireless Networks, 2020.
[13] Y. Fraboni, R. Vidal. L. Kameni, and M. Lorenzi. Clustered sampling: Low-variance and im- proved representativity for clients selection in federated learning. In International Conference on Machine Learning, 2021.
[14] Y. Fraboni, R. Vidal, and M. Lorenzi. Free-rider attacks on model aggregation in federated learning.
Banerjee and K. Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Reseurch, pages 1846-1854, PMLR, 2021.
[15] H. Gao, A. Xu, and H. Huang. On the convergence of communication-effiiiee local sgd for federated learning. In AAAI Conference on Artificial Intelligence, 2021.
[16] A. Ghorbani and Y. Zou. Data shapley: Equitable valuation of data for machine learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2242-2222222222222222 22511 PMLR, 2019.
11

---

[17] Y. Han and X. Zhang. Robust federated learning via collaborative machine teaching. In AAAI Conference on Artificial Intelligence, 2020.
[18] G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. ArXiv, Dass/553.005331 2015.
[19] R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gürel, B. Li, C. Zhang, D. X. Song, and C. J. Spanos. Towards efficient data valuation based on the shapley value. In International Conference on Artificial Intelligence and Statistics, 2019.
[20] Y. Jing, Y. Yang, X. Wang, M. Song, and D. Tao. Amalgamating knowledge from heteroge- neous graph neural networks. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15704-15713, 2021.
[21] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning- Volume 70, ICML'17, 2017.
[22] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[23] Y. Le and X. S. Yang. Tiny imagenet visual recognition challenge. 2015.
[24] Y. LeCun and Cortes. MNIST handwritten digit database. 2010.
[25] A. Li, Zhang. J. Wang, Han, and X. Li. srvacc-preserring efficient sederated-learning model debugging. IEEE Transactions on Parallel and Distributed Systems, 33:2291-2303, 2022.
[26] A. Li, L. Zhang, Wang, Tan, F. Han, Y. Qin, N. M. Freris, and X. Li. Efficient federated- learning model debugging. 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 372-383, 2021.
[27] T. Li. S. Hu, A. Beirami, and V. Smith. Ditto: Fair and robust federated learning through personalization. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6357-6368. PMLR, 2021.
[28] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In S. Dhillon, D. Papailiopoulos, and V. Sze, editors, Pro- ceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org 2020.
[29] Y. Li, X. Wang, R. Zeng, M. Yang, K.-X. Li, M. Huang, and S. Dustdar. Varf: An incentive mechanism of cross-silo federated learning in mec. IEEE Internet of Things Journal, 2023.
[30] Z. Li, D. Kovalev, X. Qian, and P. Richtárik. Acceleration for compressed gradient descent in distributed and federated optimization. In Proceedings of the 37th International Conference Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5895-5904, PMLR, 2020.
[31] J. Lin, M. Du, and J. Liu. Free-riders in federated learning: Attacks and defenses. ArXiv, abs/191 1///1911.. 1.1556600, 2019.
[32] Z. Liu, Y. Chen, H. Yu, Y. Liu, and L. zhen Cui. GTG-shapley: Efficient and accurate partic- ipant contribution evaluation in federated learning. ACM Transactions Intelligent Systems and Technology (TIST), 13:1 - 21, 2021.
[33] Z. Liu, Chen, Zhao, H. Yu, Y. Liu, R. Bao, J. Jiang, 7. Nie, Q. Xu, and Q. Yang- Contribution-awan federated learning for smart healthcare. In AAAI Conference Artificial Intelligence, 2022.
[34] Luo, X. Wang, G. Fang, Hu, D. Tao, and M. Song. Knowledge amalgamation from heterogeneous networks by common feature learning. In S. Kraus, editor, Proceedings of the Eveenty-Fightl International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3087-3093. ijcai.org, 2019.
12

---

[35] S. Maleki, L. Than,, G. Hines, T. Rahwan, and A. Rogers. Bounding the estima- tion error of sampling- based shapley value approximation with/without stratifying. ArXiv, abs/1306.4265, 2013.
[36] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas. Communication-effiee learning of deep networks from decentralized data. In A. Singh and X. J. Zhu, editors, Pro- ceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FI, USA, volume 54 of Proceedings of Machine Learning Research, pages 1273-1282. PMLR, 2017.
[37] U. Michieli and M. Ozay. Prototype guided federated learning of visual feature representations. ArXiv, 0tbs/2105.089883 2021.
[38] L. Nagalapatti, R. S. Mittal, and R. Narayanam. Is your data relevant?; Dynamic selection of relevant data for federated learning. In AAAI Conference Artificial Intelligence, 2022.
[39] S. R. Pandey, N. H. Tran, M. Bennis, Y. K. Tun, A. Manzoor, and C. S. Hong. A crowdsourcing framework for on-device federated learning. IEEE Trans. Wirel. Commun., 19(5):3241-32566 2020.
[40] K. Pillutla, K. Malik, A.-R. Mohamed, M. Rabbat, M. Sanjabi, and L. Xiao. Federated learning with partial model personalization. Proceedings of the 39th International Conference on Machine Learning, 2022.
[41] A. Shamsian, A. Navon, E. Fetaya, and G. Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, 2021.
[42] L. S. Shapley. value for n-person games. 1988.
[43] C. Shen, X. Wang, J. Song, L. Sun, and M. Song. Amalgamating knowledge towards compre- hensive classification. In AAAI Conference on Artificial Intelligence, 2019.
[44] Shen, M. Xue, X. Wang, J. Song, L. Sun, and M. Song. Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 3503-3512, 2019.
[45] Z. Shi, L.. Zhang, Z. Yao, L. Lyu, C. Chen, L. Wang. J. Wang, and X.-Y. Li. Fedfaim: A model performance-basee fair incentive mechanism for federated learning. IEEE Transactions on Big Data, 2022.
[46] R. H. L. Sim, Y. Zhang, M. C. Chan, B. Kian, and H. Low. Collaborative machine learning with incentive-aware model rewards. In International Conference on Machine Learning. 2020.
[47] Snell, K. Swersky, and R. S. Zemel. Prototypical networks for few-shot learning. NIPS, 2017.
[48] T. Song, Y. Tong, and S. Wei. Profit allocation for federated learning. 2019 IEEE International Conference on Big Data (Big Data), pages 2577-2586, 2019.
[49] Sun, A. Li, Wang, H. Yang, H. Li, and Y. Chen. Soteria: Provable defense against privacy leakage in federated learning from representation perspective. 202 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9307-9315, 2020.
[50] Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. pages 1199-1208, 2018.
[51] Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI Conference on Artificial Intelligence, 2022.
[52] Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and Jiang. Federated learning from pre-trained models: A contrastive learning approach. In NeurIPS, 2022.
[53] M. Tang, X. Ning, Y. Wang, J. Sun, Y. Wang, H. H. Li, and Y. Chen. Fedcor: Correlation- based active client selection strategy for heterogeneous federated learning. 2022 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10092-10101, 2021.
13

---

[54] Z. Tang. F. Shao, L. Chen, Y. Ye, C. Wu, and J. Xiao. Optimizing federated learning on non- iid data using local shapley value. In L. Fang, Y. Chen, G. Zhai, J. Wang. R. Wang, and W. Dong, editors, Artificial Intelligence First CAAI International Conference, CICAI 2021, Hangzhou, China, June 5-6, 2021, Proceedings, Part II, volume 13070 of Lecture Notes in Computer Science, pages 165. Springer, 2021.
[55] S. S. Tay, X. Xu, C. S. Foo, and B. K. H. Low. Incentivizing collaboration in machine learning via synthetic data rewards. In AAAI Conference on Artificial Intelligence. 2022.
[56] W. Wan, S. Hu, J. Lu, L. Y. Zhang, H. Jin, and Y. He. Shielding federated learning: Ro- bust aggregation with adaptive client selection. In International Joint Conference on Artificial Intelligence, 2022.
[57] Wang, L. Zhang, A. Li, X. You, and H. Cheng. Efficient participant contribution evaluation for horizontal and vertical federated learning. 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 91 911-923 2022.
[58] T. Wang, J. Rausch, C. Zhang, R. Jía, and D. X. Song. A principled approach data valuation for federated learning. In Federated Learning, 2020.
[59] S. Wei, Y. Tong, Z. Zhou, and T. Song. Efficient and fair data valuation for horizontal federated learning. In Federated Learning, 2020.
[60] W. Wei, L. Liu, Y. Wu, G. Su, and A. Iyengar. Gradient-leakage resilient federated learning. 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pages 797-807, 2021.
[61] D. Xie, R. Yu, G. Fang, J. Song. Z. Feng, X. Wang, L. Sun, and M. Song. Federated selective aggregation for knowledge amalgamation. ArXiv, abs/2207. 13309, 2022.
[62] C. Xu, Z. Hong, M. Huang, and T. Jiang. Acceleration of federated learning with alleviated for- getting in local training. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[63] Xu, Z. Chen, Q. Quek, and K. F. E. Chong. Fedcorr: Multi-stage federated learn- ing for label noise correction. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10174-10183, 2022.
[64] Y. Xue, C. Niu. Zheng. S. Tang, C. Lyu, Wu, and G. Chen. Toward understanding the influence of individual clients in federated learning. In AAAI Conference on Artificial Intelligence, 2021.
[65] C.-S. Yang, Liu, H. Sun. T. Li, and Z. Li. Wtdp-shapley: Efficient and effective incentive mechanism in federated learning for intelligent safety inspection. IEEE Transactions on Big Data, 2022.
[66] Ye, Y. Ji, X. Wang, K. Ou, D. Tao, and M. Song. Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2824-2833. 2019.
[67] Ye, X. Wang, Y. Ji, K. Ou, and M. Song. Amalgamating filtered knowledge: Learning lask-customized student from multi-task teachers. pages 4128-4134, 2019.
[68] R. Ye, Z. Ni, C. Xu, J. Wang, S. Chen, and Y. C. Eldar. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. ArXiv, abs/2210.07615, 2022.
[69] H. Yu, Z. Liu, Y. Liu, T. Chen, M. Cong, X. Weng, D. T. Niyato, and Q. Yang. A sustainable incentive scheme for federated learning. IEEE Intelligent Systems, 35:58 69, 2020.
[70] H. Yuan and T. Ma. Federated accelerated stochastic gradient descent. In Neural Information Processing Systems, 2020.
[71] R. Zeng, S. Zhang, J. Wang, and X. Chu. Fmore: An incentive scheme of multi-dimensionn auction for federated learning in mec. 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pages 278-288, 2020.
14

---

[72] J. Zhang, Y. Wu, and R. Pan. Incentive mechanism for horizontal federated learning based on reputation and reverse auction. Proceedings of the Web Conference 2021, 2021.
[73] Q. Zhang, J. Lin, and Q. Zhang. multi-agent reinforcement learning approach for efficient client selection in federated learning. In AAAI Conference Artificial Intelligence, 2022.
[74] L. Zhu, Z. Liu, and S. Han. Deep leakage from gradients. In Neural Information Processing Systems, 2019.
[75] Z. Zhu, J. Hong, and J. Zhou. Data-free knowledge distillation for heterogeneous federated learning. pages 12878-12888 2021.
15

---

Misibell
0.0
025
04
0.5
Sharkyy
0.1
0.30
0.2
00'
(a) Impact of different '
(b) Impact of different &
Figure 4: The impact of modification using the logistic function with different parameters.
A
Additional Experimental Results
A.1 Sensitivity test of utility function
To begin with, we conduct a sensitivity analysis on the utility function by varying two hyperparam- eters: & and T. The hyperparameter determines the logistic growth rate, thereby impacting the steepness of the curve, while ' represents the threshold at which half of the population is consid- ered satisfied. This adjustment is aimed at capturing user satisfaction, which serves as practical incentive for participants to actively participate in federated learning (FL) collaborations.
Figure 4 depicts the calculated actual Shapley values for the MNIST dataset in the presence of mislaheled data, considering the existence of 8 corrupted clients. Various combinations of the hy- perparameters k and T are explored in this analysis. Specifically, Figure
investigating the impact of different values of 7'' from the set 1.0, 0.9669, 0.95. The selection of 1.0 as the thresh- old follows greedy approach, while 0.9669 represents the performance achieved when considering all participants. Furthermore, 0.95 corresponds to an empirically derived threshold based on a com- prehensive dataset understanding.
The findings reveal that selecting a greedy selection for T leads to the disparity between clients with high and low contributions, due to the convex nature of the left-sided logistic function. Conversely, empirically setting an appropriate value for reduces the marginal gain for coalitions that surpasses the threshold, owing to the concave property of the right-sided logistic function. Consequently, this approach minimizes the discrepancy between clients with high and low contributions.
Figure 4b examines the influence of different values of k. A larger value of k indicates steeper logistic function, thereby strongly favoring coalitions that surpass the threshold T and consequently widening the gap between clients with high and low contributions. However, as i approaches in- finity, the logistic function transforms into a ReLU function, disregarding all coalitions that fail to surpass the threshold T and assigning equal value those that do. Consequently, in this scenario, the relative performance, as depicted by the difference between client and client in Figure 4b would be disregarded due hard threshold.
The selection of hyperparameters i and T should be determined by the desired characteristics the applications to enhance the influence of coalitions with superior performance or ensure individual rationality. Thus, we maintain them as carefully selected hyperparameters, allowing for flexibility and customization based specific application requirements.
A.2 Ablation Study for smaller G
The federated knowledge amalgamation process utilizes FPMs comprising only two convolution layers, each of which introduces minimal additional GPU memory usage. As a result, in the reported experiments, we are able to simultaneously distill the knowledge from all client models on a single V100 GPU. However, the number of clients increases, the limited memory capacity of the GPU
16

---

Table 3: Federated knowledge amalgamation with different G.
MNIST
CIFARIO
G
Non-IID
Mislabel
runtime
Non-IID
Mislabel
runtime
n
0,9448
0,9612
160
0.9290
0.9641
295
2
0.9427
0.9726
633
0.9333
0,9645
299
Table 4: Results of participant contribution evaluation on Aaay-ImaeeNet dataset.
Dataset
Scenario
GT
TMC
GTG
DIG-FL
SPACE(Avg)
SPACE
Non-IID
0.9082
0.9610
0.7425
0.8944
0.9175
0.9092
Mislabel
0.7833
0,9233
0.7998
0.8405
0.9293
0.9256
Imag-em
Time(s)
84994
69212
98855
266
252
453
Comm(MB)
2639200
2097600
8000
8000
8000
400
may pose a challenge. To address this issue, we propose a hierarchical amalgamation framework that mitigates the high GPU memory requirement.
To evaluate the effectiveness of hierarchical amalgamation, conducted experiments by amalga- mating groups of G = 2 teacher models instead of all n teacher models in a hierarchical manner. The results, shown in Table demonstrate that knowledge transfer from the local datasets can still be achieved through hierarchical amalgamation, yielding performance comparable to amalgamating all clients simultaneously. Although hierarchical amalgamation alleviates the high GPU memory requirement, it introduces computational complexity increase in the knowledge amalgamation process from (n) to (n log n). as evident from Table 3
A.3 Comparison of estimated Shapley Value
Figure iillustrates the scatter plots depicting the estimated Shapley values compared to the actual Shapley values. Regarding the centralized method, TMC-Shapley outperforms GTT-hhalley indi- cating that Monte Carlo sampling is more efficient for approximating the Shapley value than group testing. To accelerate the approximation process. GTG-Shapley employs gradient-based model re- construction. While it performs well on the MNIST dataset. its performance deteriorates on the CIFARIO dataset due to the increased complexity of both the model and the dataset. The results indicate that DIG-FL tends to overestimate clients' contributions with incomplete class data, lead- ing to suboptimal performance in non-IID scenarios. Conversely, the proposed SPACE approach exhibits strong performance across all scenarios.
A.4 Results on Tiny-Imagenl
To assess the efficacy of the proposed SPACE framework on more complicated datasets, an addi- tional experiment was conducted, incorporating a portion of the Tiny-ImageNet dataset 223 This subset encompasses 50 categories and involves 10 clients. The results are provided in Table The SPACE method demonstrates satisfying performance in terms of Pearson's 's Correlation Coefficient (PCC). It's noteworthy that achieving feature aligmment in a complex dataset like Tiny-ImageNeet more challenging, Consequently, the knowledge amalgamation process requires an extended number of epochs to converge, particularly in scenarios where the data distribution across clients non-IlI, and clients possess data from disjoint classes. On the other hand, the SPACE(Avg) approach, which employs FedAvg for obtaining the embedding space, may offer superior time efficiency in cases with a limited number of communication rounds. Nonetheless, the advantages in communication efficiency stemming from single-round amalgamation continue to position SPACE as the preferred choice when mitigating communication costs is key consideration.
A.5 Scenario without labeled validation set
Given the inherent challenges associated with acquiring sufficiently well-labeled validation dataset, there arises an important question regarding the adaptability of the proposed approach in cases where label information unavailable. To investigate this, we conducted supplementary experiments em- ploying unlabeled data for knowledge amalgamation. In this context, we drew inspiration from the work of (44) and modified the loss function to incorporate weights based on entropy purity rather than the confidence associated with accurate result predictions. Specifically, clients with lower en-
17

---

MMST Non
NidSs Madabel
CIFR10 Non ID
CIA610 Milebel
041
0:00
1
02-
CO.
600 665
630 615 t36 6Us aio
0.6
0.
01
0.1
64
65
66
Cd-
16
0.00
c20-
C0-
00-
01
u.a Das 322 Mubs 1.0)
07
044
CO-
26
0:00
c30-
03
CD-
0.2
07
04
04
0.1
6.6
0.
6.6
1.5
C3-
2.
032
0.00
455
Casmated Chepkey
Eatimited Shapley
Eximated Shuplyy niez
Catimtted
Figure 5: Estimated Shapley and the actual Shapley value all approaches under all scenarios.
Table 5: Results of SPACE with and without labels.
MNIST
CIFARTO
Label
Non-IID
Mislabel
Non-IID
Mislabel
Non-IID
Mislabel
0.9448
0.9612
0,9290
0.9641
0.9092
0.9256
0.8772
0.9611
0.8653
0.9641
0.9091
0.9231
tropy values were assigned greater weight in the loss function. The experimental results are shown in Table[5] While some marginal reductions performance are evident, particularly in scenarios char- acterized by Non-IID data distributions, it is important to note that these performance decreases primarily attributed to the absence of label information. Nevertheless, the results clearly demonstrate the viability of leveraging unlabeled data as a practical alternative for knowledge amalgamation, of- fering a feasible solution cases where the acquisition of labeled data challenging.
B
Pseudocode
The algorithmic representation of the proposed SPACE approach is presented Algorithm I Ini- tially, the server distributes the initial model 0 to all participating clients. Each client then performs empirical risk minimization to obtain locally optimized model 0, and transmits the server. Using the locally optimized models as teacher models, the federated knowledge amalgamation tech-
18

---

Algorithm SPACE
Require: Initial FL model 8 client set N, local datasets (DI...., Dnd validation dataset Dy, fea-
ture encoder f((-) and utility function V(.)
1: # Knowledge Amalgamation
2: for In....n
3: 0, -LocaUUpdate((. D.)
4: 0* = =KA00, ......., On). Dv)
5: # Build server prototypes and clients' prototypes
6: for j I.....IC do
7: PV P En f(0 Vx E Dvs
8: for I.....n do
9: for ===1... 1= do
p P... E. f(0", 2). Vx Da.j
11: # Evaluate contribution with Prototype-basee Model Evaluation
12: r=
13: (cI....,c.) )a) 0
14: while Convergence criteria not met do
15: r=r+1
H" J'Nll II
17:
-V(W)
for k In... do
19:
if Pu--1
then
S= [ + 1I....,... ['(n]] [n]}
21:
for j I.....//Cl do
p(1) Eies Eies [D.j]
p p())
23:
Uk V(S)
24:
else
25:
Uk 0
26:
CxT(M] = r-1
(xr(H] (Ux-1 - 0k)
27:
return {Ci....,
nique is employed to distill knowledge into the server model, resulting in enhanced server model denoted 6*. Subsequently, prototypes are constructed using 6 for both the validation and local datasets. Following prototype construction, SPACE proceeds to calculate participant contributions using the Shapley value, leveraging the prototype-based model evaluation. To ensure convergence, SPACE samples multiple permutations of participants. As the modified utility function may com- press the utility values of small coalitions, SPACE performs pruning operations at the sequence level. If the utility value VSS) of coalition S falls below the pruning threshold, subsets of S are pruned as their marginal gains become negligible.
C Potential Malicious Attack
Malicious attack by data poisoning. The manipulation of data labels, frequently employed ma- licious strategy by clients engaged in data poisoning attacks, is denoted as "label-fippiigg" in ref- erence (6). In our work, we adopted the term "mislabeled" refer to unintentional and intentional mislabeling instances. In this case, malicious clients attempt to disrupt the training process by submitting gradients derived from mislabeled data. Explicitly, as depicted in Table 2 the results highlight the effectiveness of SPACE in pinpointing those malevolent clients harboring inaccurately labeled data. With the server's 's privately owned validation set, our framework efficiently identifies such malicious clients, even if they form the majority in the federated process. In addition, the ex- perimental results in [egunstddemosstrate the robustness our framework in the federated training. It is worth noting that the SPACE proposed employs contributions as weighting factors, thereby re- ducing the negative impact of malicious participants, i.e., the malicious clients usually contribute less due to their distributions differing from that of the server.
19

---

Maliciously inflating contribution. Clients may aim to inflate their contributions maliciously to obtain more significant rewards. If the prototypes from the validation set become exposed, clients might exploit this knowledge to boost their contributions in an ad-hoc manner. In such a case, model evaluations, where all clients have access solely to the
the SPACE utilizes prototype-base feature extraction layers of the amalgamated model while excluding access to the fully-connected layers, i.c., the final score. This strategic approach facilitates the creation of prototypes without com- promising the confidentiality of the distribution that underlies the validation set. Note that another potential threat clients employing the zeroth-order optimization technique 4 to derive prototypes with disproportionately high contributions. In this case, the client's s access to the server in that the SPACE does not require each client to recalculate their contributions frequently.
simply detect this attack by limiting
In addition the aforementioned attacks directly associated with contribution evaluation, federated learning is known for its vulnerability to spectrum of attacks (6.. including but not limited to re- construction attacks 74. It noted that to counter these threats, researchers have proposed defense mechanisms, such gradient pruning 774,, data representation perturbation [49], and differential privacy (DP) [60]. These works are essential. However, they are, in essence, orthogonal to the main theme of this paper, which focuses contribution evaluation. Nevertheless, these privacy- preserving strategies can be integrated with our method, working collaboratively to enhance data privacy protection.
20


Reviews:
[{"review_id": "YqThKNnJDE", "reviewer": "Reviewer_vCp6", "summary": "This paper studies the client contribution evaluation problem under federated learning settings. The goal is to achieve more computational and communicational efficient contribution evaluation. The paper proposes Federated Knowledge Amalgamation and Prototype-based Model Evaluation technique for the goal. Federated Knowledge Amalgamation treats client models as teacher models that together train the server model which serves as the student model. This training approach avoids the typical multi-round training of federated learning frameworks, thus reducing the communication costs. Prototype-based Model Evaluation derives the contribution of a client by computing and comparing the prototypes of the client models and the server model, thus reducing both the computation and communication costs of contribution evaluation. Experimental results on MNIST and CIFAR10 datasets confirm the effectiveness of the proposal. ", "questions": "\"For the non-IID scenario, we selected a subset of clients, ranging from 0% to 80%, and assigned them local datasets containing incomplete categories.\" Which categories are incomplete for each of the clients? ", "limitations": "As discussed in the paper, the proposed technique relies on \"a proper and reliable validation set with accurately labeled data, which can be challenging to achieve in real-world scenarios\".", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies participant contribution evaluation in federated learning which is an important and trending topic. \n\n2. The proposed techniques are intuitive and are effective as shown by experiments on two benchmark datasets.\n\n3. The paper is well written and easy to follow overall.", "weaknesses": "While the paper proposes an interesting and empirically effective technique for improving the efficiency of participant contribution evaluation in federated learning, there is no theoretical guarantee on the effectiveness of the proposed technique in terms of measuring the true participant contribution.\n\nMinor presentation issues: \nMissing whitespace: \"Pandey et al.[34]\", \"Zhang et al.[64]\"\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n> While the paper proposes an interesting and empirically effective technique for improving the efficiency of participant contribution evaluation in federated learning, there is no theoretical guarantee on the effectiveness of the proposed technique in terms of measuring the true participant contribution.\n\nPlease see \"Author Rebuttal by Authors\" above.\n\n> Minor presentation issues: Missing whitespace: \"Pandey et al.[34]\", \"Zhang et al.[64]\"\n\nThanks for pointing this out. We would revise it in our manuscript.\n\n> \"For the non-IID scenario, we selected a subset of clients, ranging from 0% to 80%, and assigned them local datasets containing incomplete categories.\" Which categories are incomplete for each of the clients?\n\nSpecifically, the non-IID clients only own datasets with partial categories. For instance, a client might be assigned a dataset containing only 3 of the ten classes available in the MNIST dataset. This allocation strategy allows us to create clients with significant heterogeneity in their datasets, resulting in channeling situations in federated learning because the data distribution differs between each client."}, {"review_id": "yCNCwxkzPi", "reviewer": "Reviewer_a6mr", "summary": "The paper introduces a novel approach named Single-round Participants Amalgamation for Contribution Evaluation (SPACE) for efficiently evaluating the contribution of participants in Federated Learning (FL). Accurately evaluating participant contribution has been a challenge in current FL, especially considering cost and scalability. Current methods mostly use the Shapley value, a measure from cooperative game theory, but calculating it is computationally expensive. SPACE, on the other hand, combines two new components - Federated Knowledge Amalgamation and Prototype-based Model Evaluation - to reduce evaluation effort. Federated Knowledge Amalgamation distills information from all local models into the server model in just one round of communication, saving time. Prototype-based Model Evaluation compares server and client prototype similarities, effectively eliminating the dependency on the size of the validation set.\n\nAdditionally, SPACE modifies the utility function using a logistic function to better reflect user satisfaction, enhancing the utility function's rationality in real-world settings. Experimental results show that SPACE outperforms current methods in terms of both running time and Pearson's Correlation Coefficient, a measure of correlation strength. The efficacy of SPACE has been tested in various applications, including client reweighting and selection, and consistently demonstrates exceptional performance.", "questions": "- Are there specific domains or types of applications where SPACE proves particularly effective or ineffective?\n- You mention that SPACE enables participant evaluation within a single communication round. What impact does this have on the overall quality and accuracy of the federated learning model?\n- How generalizable is the SPACE approach? Could it be adapted to various federated learning architectures and different types of machine learning models?\n- Is there a possibility that SPACE could be used in a malicious way, such as inflating a participant's contribution unfairly?\n- How does SPACE handle privacy concerns? Does the process of distilling information from local models to a server model pose any risks to data privacy?", "limitations": "See weaknesses above", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- SPACE, appears to be a novel and innovative approach to evaluating the contribution of participants in Federated Learning. The introduction of components like Federated Knowledge Amalgamation and meta-learning using prototypes could potentially provide new avenues for research.\n- The method aims to reduce the time and computational resources required for evaluating participant contributions by eliminating dependence on the size of the validation set and enabling evaluation within a single communication round. This efficiency is particularly valuable in Federated Learning, where minimizing communication is crucial due to privacy and resource concerns.", "weaknesses": "- Although not specific to this one paper, a recurring trend in federated learning conducts experiments that may overfit to specific use cases. In particular, most papers demonstrate the efficacy of their methods using datasets limited to Federated MNIST and non-iid versions of CIFAR (in this case with only 10 classes). Although these experiments might demonstrate proof of concept, adaptability and effectiveness of the method across a broader range of datasets and scenarios remains unclear. As such, I believe additional experiments with more extensive datasets, or an additional discussion on how this method can be applied to real-world scenarios might help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and summary of our paper. We have addressed all your questions in the following.\n>  I believe additional experiments with more extensive datasets, or an additional discussion on how this method can be applied to real-world scenarios might help.\n\nPlease see \"Author Rebuttal by Authors\" above.\n\n> Are there specific domains or types of applications where SPACE proves particularly effective or ineffective?\n\nThe experimental results indicate that the proposed SPACE approach is effective, especially in a mislabeled data scenario. This is because prototypes obtained from clients with mislabeled data exhibit substantial deviation from the prototypes of the validation set. Consequently, such deviations are efficiently identified by the prototype-based model evaluation approach employed in the proposed method.\n\n> You mention that SPACE enables participant evaluation within a single communication round. What impact does this have on the overall quality and accuracy of the federated learning model?\n\nOur SPACE assesses participants' contributions to the federated learning process rather than directly engaging in model training. Consequently, its implementation does not directly influence the overall quality and accuracy of the federated model. However, when utilized as the weighting mechanism for clients, it contributes positively to the robustness of the federated learning model, as evidenced by the results shown in Figure 2.\n\n> How generalizable is the SPACE approach? Could it be adapted to various federated learning architectures and different types of machine learning models?\n\nOur proposed SPACE is designed to assess participant contributions within the context of horizontal federated learning settings. In such scenarios, participants collectively engage in a federated learning process while sharing a common feature space yet possessing distinct data samples.  Moreover, our SPACE can effortlessly integrate into other deep learning classification models because the two major components, i.e., the information aggregation (Federated Knowledge Amalgamation) and the evaluation protocol (Prototype-based model evaluation), are both model agnostic.\n\n> Is there a possibility that SPACE could be used in a malicious way, such as inflating a participant's contribution unfairly?\n\nIf the prototypes of the validation set are exposed, clients could exploit this information to enhance their contributions artificially. A precautionary measure is implemented to counteract the risk of information leakage during the knowledge amalgamation. This involves granting clients access solely to the feature extraction layers of the amalgamated model while excluding the fully-connected layers. This approach facilitates the creation of prototypes without compromising the confidentiality of the distribution underlying the validation set. Another conceivable threat involves clients employing the zeroth-order optimization technique [C] to derive prototypes of disproportionately high contribution. However, this attack can be readily countered by limiting the client's access to the server. Such a limitation is sensible since clients typically do not need to re-calculate their contributions frequently.\n\n> How does SPACE handle privacy concerns? Does the process of distilling information from local models to a server model pose any risks to data privacy?\n\nIndeed, federated learning is susceptible to various attacks [D], mainly when uploading model weights or gradients, as it introduces the potential risk of reconstruction attacks [E]. Such attacks allow adversaries to reconstruct client training data by exploiting predicted confidence values, model parameters, and gradients. To counter these threats, researchers have proposed defense mechanisms, such as gradient pruning [E], data representation perturbation [F], and differential privacy (DP) [G]. These privacy-preserving strategies can be effectively integrated with our method, working collaboratively to enhance data privacy protection. However, it is crucial to emphasize that our current work centers on contribution evaluation, and the topic of attack-defense mechanisms lies beyond the scope of our discussion.\n\n[C] Chen, P. Y., Zhang, H., Sharma, Y., Yi, J., & Hsieh, C. J. (2017, November). Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security (pp. 15-26).\n\n[D] Chen, Y., Gui, Y., Lin, H., Gan, W., & Wu, Y. (2022, December). Federated learning attacks and defenses: A survey. In 2022 IEEE International Conference on Big Data (Big Data) (pp. 4256-4265). IEEE.\n\n[E] Zhu, L., Liu, Z., & Han, S. (2019). Deep leakage from gradients. Advances in neural information processing systems, 32.\n\n[F] Sun, J., Li, A., Wang, B., Yang, H., Li, H., & Chen, Y. (2021). Soteria: Provable defense against privacy leakage in federated learning from representation perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9311-9319).\n\n[G] Wei, W., Liu, L., Wut, Y., Su, G., & Iyengar, A. (2021, July). Gradient-leakage resilient federated learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) (pp. 797-807). IEEE.\n"}, {"review_id": "QAn6Nqb4hW", "reviewer": "Reviewer_AsTD", "summary": "This paper proposes a single-round participant contribution evaluation method for FL. The novel and interesting part is using the sample embedding similarity between client data and (server) validation data to indicate contribution, thus avoiding the time-consuming model retraining step.", "questions": "See weakness.", "limitations": "Not enough. More discussions on when the proposed model can work well should be given.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Very useful problem\n2. Interesting idea with using embedding instead of retraining models for contribution evaluation.", "weaknesses": "1. No theoretical analysis of why the embedding similarity can lead to an excellent approximation to the original Shapley-based contribution evaluation with re-training. Since embedding similarity-based contribution evaluation is quite different from original retraining-based contribution evaluation, such a theoretical analysis is a must; only empirical analysis cannot verify that the proposed contribution evaluation can always be good.\n\n2. Experiment is too limited. More datasets, more parties, and malicious behaviors of parties should all be counted to ensure that the contribution obtained by the proposed method is robust.\n\n3. The method seems to only work for classification. How to extend it to regression tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback. We have answered all your concerns. In the following, we respond point by point.\n\n> No theoretical analysis of why the embedding similarity can lead to an excellent approximation to the original Shapley-based contribution evaluation with re-training. \n\nPlease see \"Author Rebuttal by Authors\" above.\n\n> Experiment is too limited. More datasets, more parties, and malicious behaviors of parties should all be counted to ensure that the contribution obtained by the proposed method is robust.\n\nPlease see \"Author Rebuttal by Authors\" above.\n\n> The method seems to only work for classification. How to extend it to regression tasks?\n\nThanks. Our methodology is primarily designed for classification tasks, drawing from the works cited in references [29, 43, 51, 52, 53]. It is important to note that federated learning also encompasses the significant domain of regression tasks[H, I]. However, it is evident from our literature review that the discourse surrounding the evaluation of contributions in the context of regression tasks could be more extensive. As such, a notable gap exists in the literature regarding the evaluation of contributions within regression tasks, thus signifying a crucial avenue for future exploration.\n\n> More discussions on when the proposed model can work well should be given.\n\nThe experimental results indicate that the proposed SPACE approach is effective, especially in a mislabeled data scenario. This is because prototypes obtained from clients with mislabeled data exhibit substantial deviation from the prototypes of the validation set. Consequently, such deviations are efficiently identified by the prototype-based model evaluation approach employed in the proposed method.\n\n\n[H] Lee, H., Bertozzi, A. L., Kovačević, J., & Chi, Y. (2022, May). Privacy-Preserving Federated Multi-Task Linear Regression: A One-Shot Linear Mixing Approach Inspired By Graph Regularization. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5947-5951). IEEE.\n\n[I] Su, L., Xu, J., & Yang, P. (2022). Global convergence of federated learning for mixed regression. Advances in Neural Information Processing Systems, 35, 29889-29902.\n\n\n"}, {"review_id": "18CpQzNRer", "reviewer": "Reviewer_AA9d", "summary": "This paper studies to evaluate the contribution of each client during federated training efficiently. It proposes a framework named SPACE, which trains a student model in the server with one communication round to measure the similarity between local datasets and the validation dataset in server. Finally, extensive experiments are conducted to prove the efficiency of SPACE. ", "questions": "1. One important assumption in this paper is the server holds validation dataset, which is rare in reality. What if the validation dataset is hold by distributed clients rather than the central server? \n2. In section 1, the paper states \"if the server and client distributions are similar, the client dataset proves helpful in correctly classifying the data in the validation set of the server.\" \n    - First, just like Equation 1 the target of FL is to cooperatively train a model that performs well on the test datasets in all clients rather than the validation set of the server. \n    - Secondly, it seems like the authors assume that the validation dataset in the server shares the same distribution with the distributed test datasets. In my opinion, the assumption is too strong and unpractical. If the server already knows the distribution of the test datasets, there is no need to federated-train a model. \n3. The contribution estimation in SPACE is decoupled from federated training. I' curious how does SPACE deal with client over-sampling, where the number of participation for different clients is unbalanced and some clients may be not selected during training. \n4. In the experiment, the dataset and the number of clients are too small (5 for CIFAR10 and 10 for MNIST). ", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and organized.\n- The proposed solution is feasible and efficient. \n- Extensive experiments are conducted to evaluate the proposed framework. ", "weaknesses": "- The assumption of SPACE is too strong. (See Question 1 and 2)\n- The proposed contribution estimation framework is decoupled from federated training. (See question 3)\n- The dataset and client number in the experiment are too small. (See Question 4)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback. We have answered all your concerns. In the following, we respond point by point.\n> One important assumption in this paper is the server holds validation dataset, which is rare in reality. What if the validation dataset is hold by distributed clients rather than the central server?\n\nIf distributed clients solely retain their validation dataset, SPACE remains operational, albeit with the need for an extra unlabeled server dataset for knowledge amalgamation. Furthermore, a modification of the distillation loss for unlabeled data, as illustrated in [39], becomes necessary. Alternatively, SPACE(Avg) could serve as a simpler alternative for building feature embedding space if access to an additional unlabeled server dataset is unfeasible. After constructing the embedding space, clients must upload the prototypes of their respective validation sets. The global validation prototypes are then obtained by performing a weighted sum of the local validation prototypes, and the contribution can be calculated using the prototype-based evaluation. However, a potential drawback of this setup is that if mislabeled data exists in any of the local validation sets, the global validation set would also be adversely affected.\n\nWe provide additional experimental results for  SPACE with unlabeled data for knowledge integration. The assignment of weights to clients in the loss function is guided by their prediction entropy, with greater weighting for those with lower entropy. Although slight decreases in performance are observed, particularly in Non-IID scenarios, these can be attributed to the absence of label information. Nonetheless, the outcomes manifest the feasibility of leveraging unlabeled data as a viable substitute for knowledge integration, especially in practical situations where obtaining labeled data poses difficulties.\n\n\n* Result of SPACE(unlabel)\n\n|  Scenario       | MNIST   | CIFAR  | Tiny-ImageNet  |\n|---        |---    |---|---|\n| Non-IID   | 0.8772    |  0.8653     | 0.9091  |\n| Mislabel  | 0.9611    |  0.9641     | 0.9231  |\n\n> The authors assume that the validation dataset in the server shares the same distribution with the distributed test datasets. In my opinion, the assumption is too strong and unpractical. If the server already knows the distribution of the test datasets, there is no need to federated-train a model.\n\nThanks for pointing out this issue. Following previous works [29, 43, 51, 52, 53], we assume that the server initially possesses a validation set. This validation set is pivotal for ensuring equitable and reliable assessment of contributions within the context of federated learning. As deliberated in the \"Broader Impact\" section, we are fully cognizant of the challenges of procuring a trustworthy validation in real-world settings. Nonetheless, specific scenarios exist wherein servers may be motivated to assemble such a validation set in exchange for substantial commercial benefits. Notably, instances like those discussed in [J, K] underscore this idea of servers having access to validation sets for contribution evaluation in real-world applications. In [J], multiple city-gas companies collaborate to train a hazard identification model. Similarly, [K] presents experiments in the healthcare domain involving the participation of eight esteemed medical institutions in China to construct healthcare decision-support models. Both of these applications necessitate a reliable server-based validation to ensure AI models' accuracy objectives before their deployment in real-world scenarios. Though the size of the validation set might not be large enough for centralized training under the setting, it provides a reliable evaluation of the model performance, especially when the clients hold mislabeled data. In light of such examples, we contend that our underlying assumption of the availability of a validation set on the server holds practical in specific scenarios.\n\n> The contribution estimation in SPACE is decoupled from federated training. I' curious how does SPACE deal with client over-sampling, where the number of participation for different clients is unbalanced and some clients may be not selected during training.\n\nThe clustered sampling (CS) method [11] is introduced to tackle over-sampling and reduce performance variance in federated learning. Instead of employing multinomial distribution (MD) for sampling with replacement [25], which has exhibited substantial performance variance, the clustered sampling approach first categorizes clients into distinct clusters. Subsequently, clients are sampled from these clusters, ensuring that clients with unique distributions are more likely to be sampled. As mentioned in Section 5.2 within the context of client selection, we first utilize the SPACE framework to derive the prototypes and contributions of individual clients. The prototypes are then utilized for clustering purposes, while clients' contributions determine the clients' probability. This combined approach effectively addresses the over-sampling issue while enhancing the training outcomes.\n\n> In the experiment, the dataset and the number of clients are too small (5 for CIFAR10 and 10 for MNIST).\n\nPlease see \"Author Rebuttal by Authors\" above.\n\n[J] Yang, C., Liu, J., Sun, H., Li, T., & Li, Z. (2022). WTDP-Shapley: Efficient and Effective Incentive Mechanism in Federated Learning for Intelligent Safety Inspection. IEEE Transactions on Big Data.\n\n[K] Liu, Z., Chen, Y., Zhao, Y., Yu, H., Liu, Y., Bao, R., ... & Yang, Q. (2022, June). Contribution-aware federated learning for smart healthcare. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 11, pp. 12396-12404).\n\n"}, {"review_id": "IDcvNqRfUB", "reviewer": "Reviewer_HEwC", "summary": "This paper introduced a novel method to evaluate the participant contribution under the setting of federated learning. In the beginning, the author raised two challenges of recent works related to the participant contribution of FL: Multi-round training and dataset dependency, which are actually about the communication and computational costs in FL. The author derived the SPACE model to address these two challenges. Specifically, to address the high communication cost issue, the author introduced a module named Federated Knowledge Amalgamation, which enables a single round of communication. Then the author introduced Prototype-based Model Evaluation to reduce the evaluation complexity since this method only needs similarities comparison without iterating through a validation set. In addition, in the contribution part, the author rectified the utility function with a logistic function to address the rationality violation issue. Last, the author evaluated participant contributions to the SPACE model on two datasets compared with four baselines. Furthermore, the author experimented with SPACE on the client reweighting and selection problem. ", "questions": "Like the suggestion in weakness: \n1. The author should provide more detail about the theoretical analysis on communication and computational cost. For example, how the design has the advantage to reduce complexity and appropriate for GPU?\n2. In section 4.3, the author pointed out a problem that applying model performance as utility function may violate rationality. And the author add a activation as solution. Could you explain why you choose this activation and why this design can avoid violations?\n3. The focus in this paper is to reduce complexity, so the author should provide results of the computational cost and communication cost in experiment part.\n", "limitations": "The author can provide more details of the theoretical complexity analysis. In addition, a more comprehensive experiments should be conducted, such as including model complexity, ablation study and case study.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author raised two challenges of participant contribution evaluation in FL, and introduces corresponding techniques to address them. \n2. The author explained the reason and function of each module in detail. \n3. The author provides a theoretical value of communication costs and computational costs of baselines and derived model.\n", "weaknesses": "1. The author should pay attention to the writing style. For example, more focus on key points, like how to reduce computational complexity. Otherwise, it is hard to follow.  \n2. The author should provide more detail about the theoretical analysis on communication and computational cost. For example, how the design has the advantage to reduce complexity and appropriate for GPU?\n3. In section 4.3, the author pointed out a problem that applying model performance as utility function may violate rationality. And the author add a activation as solution. Could you explain why you choose this activation and why this design can avoid violations?\n4. The focus in this paper is to reduce complexity, so the author should provide results of the computational cost and communication cost in experiment part.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n> The author should pay attention to the writing style. For example, more focus on key points, like how to reduce computational complexity. Otherwise, it is hard to follow.\n\nThanks for pointing out this issue. We have already provided the detailed complexity analysis in the supplementary materials and will move to the main manuscript in the camera-ready version if space allows.\n\n> The author should provide more detail about the theoretical analysis on communication and computational cost. For example, how the design has the advantage to reduce complexity and appropriate for GPU?\n\nThanks. We have implemented the hierarchy amalgamation technique to infer SPACE to minimize overall GPU memory usage. Specifically, as the number of clients 'n' increases, the number of Feature Projector Modules (FPMs) that necessitate training also escalates. Rather than conducting the amalgamation of all clients simultaneously, leading to a substantial demand for peak GPU memory, our proposed hierarchy amalgamation approach facilitates the amalgamation of fewer clients at a time, effectively reducing the peak GPU memory requirements. \n\n> In section 4.3, the author pointed out a problem that applying model performance as utility function may violate rationality. And the author add a activation as solution. Could you explain why you choose this activation and why this design can avoid violations?\n\nWe employ the sigmoid function because of its capacity to represent the characteristics of user satisfaction aptly. As model performance surpasses a certain threshold, the number of satisfied users increases significantly, but eventually, the satisfaction tends to saturate for exceedingly high-performance levels. For instance, a model with a performance of 97% may not exhibit substantially different user satisfaction compared to one with a performance of 98%. The sigmoid function effectively captures these two aspects of user satisfaction. To ensure adherence to individual rationality, we can select a sigmoid function with large values for the parameters $k$ and set $T$ between the greatest individual performance and the full coalition performance. This configuration substantially diminishes the individual value while amplifying the value of coalitions, thus mitigating the risk of violating individual rationality.\n\n> The focus in this paper is to reduce complexity, so the author should provide results of the computational cost and communication cost in experiment part.\n\nThank you for the valuable suggestion. Indeed, we have already included the computational cost in terms of runtime in Table 2, indicating the computational efficiency of our proposed method. To further illustrate the communicational efficiency, we will supplement the communication cost in terms of the amount of data transmitted, measured in megabytes (MB), as follows. Note that the communication cost shown in the table is calculated as $Comm = 2 \\cdot n \\cdot R_g \\cdot |\\theta|$, where $R_g$ denotes the number of communication rounds, $|\\theta|$ represents the number of model parameters(MB) and constant of 2 stands for the upload and download of the models. For approaches that require retraining, the number of clients $n$ involves repeated clients during retraining.\n\nThe findings concerning communication costs underscore the notable communication efficiency inherent in the proposed SPACE framework. The adoption of a single-round amalgamation strategy results in a substantial decrease in communication costs, yielding a scale advantage proportional to a factor of $R_g$, especially in comparison to approaches that demand the complete training of the federated model.\n\n* Result on MNIST\n\n|           |  GT       | TMC   | GTG  | DIG-FL  | SPACE(Avg)  | SPACE  |\n|---        |---        |---    |---|---|---|---|\n| Non-IID   | 0.6877    |  **0.9824**    | 0.9287   | 0.8715 | 0.9713   | 0.9448    |\n| Mislabel  | 0.4230    |  0.9507    | 0.9608   | 0.9580 | 0.9529   | **0.9612**    |\n| Times(s)  | 97137     |  84796     | 62473    | 315    | 294      | **160**   |\n| Comm(MB)  | 11813.12  |  10292.48  | 35.2     | 35.2   | 35.2     | **1.76**   |\n\n\n* Result on CIFAR\n\n|           |  GT       | TMC   | GTG  | DIG-FL  | SPACE(Avg)  | SPACE  |\n|---        |---        |---    |---|---|---|---|\n| Non-IID   | 0.6089    |  0.8877     | 0.8208  | 0.7546   | **0.9540**   | 0.9290    |\n| Mislabel  | 0.5192    |  0.9595     | 0.4148  | 0.9598   | 0.9565   | **0.9641**    |\n| Times(s)  | 7468     |  4950      | 835   | 536      | 315      | **295**   |\n| Comm(MB)  | 307800   |  202920    | 11400    | 11400     | 11400     | **570**   |"}]

Author rebuttal:
We would like to thank all the reviewers for their constructive comments. Common questions asked by multiple reviewers would be replied here in a unified manner.

>  Theoretical support of why SPACE leads to an excellent approximation to the original Shapley-based contribution evaluation with re-training

Sorry for the misunderstanding. Actually, our SPACE still applies the Shapley value to calculate each participant's contribution as in previous works. The major contribution of SPACE is that we employ the similarity between prototypes of the server validation set and prototypes formed by clients in a coalition as a utility function for Shapley value calculation instead of the model performance, leading to a lower computation cost. It is worth noting that the theoretical foundation of why we can adopt the similarity function as our utility (for model performance) is that the differences in empirical risk between two distributions are bounded by the divergence between the two distributions when the labeling functions are the same (Theorem 1 in [A]). This finding implies that if the divergence between the client coalition dataset and the validation set is small, then the model trained on the client coalition dataset may obtain high performance on the validation set. In that case, it is reasonable using the divergence to approximate the model performance on the validation set. However, the variation divergence is difficult to compute in practice. Therefore, we intuitively adopt the prototypes' similarity as an alternative to a distribution divergence. The rationale for selecting prototype similarity rests on two crucial considerations. Firstly, it can be easily computed, significantly expediting the evaluation process without necessitating a complete iteration across all samples. Secondly, its privacy-preserving nature renders it suitable for transmission within the framework of federated learning, an aspect that several preceding studies have embraced.

> The experiment is limited in both client numbers and datasets.

In the experiment, we conducted contribution evaluation experiments on the MNIST and CIFAR10 datasets, following the established protocols of previous studies. However, it is important to highlight that the number of clients used in our experiments was limited due to the necessity of calculating the actual Shapley value as the ground truth for evaluation. Specifically, doubling the number of clients from 10 to 20 would introduce an exponential increase in effort by a factor of 1000. Consequently, we performed experiments with a maximum of 10 clients until a more efficient evaluation metric could be proposed. Notably, our proposed SPACE method demonstrated the ability to simultaneously amalgamate information from 100 clients, as demonstrated in the experiment of client selection. During the rebuttal phase, we conduct an additional experiment to include the partial Tiny-ImageNet dataset [B], including 50 categories with ten clients. The results are as follows: The proposed SPACE method achieved satisfactory performance in terms of PCC. Notably, feature alignment is more challenging in such a complex dataset. Thus knowledge amalgamation requires more epochs for convergence, particularly in non-IID scenarios where clients hold data from disjoint classes. The SPACE(Avg) approach, which utilizes FedAvg to obtain the embedding space, may exhibit superior time efficiency for cases with only a few communication rounds. However, the communication benefits from single-round amalgamation still make SPACE the preferred solution when communication cost is a concern. Note that the communication cost shown in the table is calculated as $Comm = 2 \cdot n \cdot R_g \cdot |\theta|$, where $R_g$ denotes the number of communication rounds, $|\theta|$ represents the number of model parameters(MB) and constant of 2 stands for the upload and download of the models. For approaches that require retraining, the number of clients $n$ involves repeated clients during retraining.
* Result on Tiny-ImageNet

|           |  GT       | TMC   | GTG  | DIG-FL  | SPACE(Avg)  | SPACE  |
|---        |---        |---    |---|---|---|---|
| Non-IID   | 0.9082    |  **0.9610**     | 0.7425  | 0.8944   | 0.9175   | 0.9092    |
| Mislabel  | 0.7833    |  0.9233     | 0.7998  | 0.8405   | **0.9293**   | 0.9256    |
| Times(s)  | 84994     |  69212      | 98855   | 266      | **252**      | 453   |
| Comm(MB)  | 2639200   |  2097600    | 8000    | 8000     | 8000     | **400**   |

[A] Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010). A theory of learning from different domains. Machine learning, 79, 151-175.

[B] Le, Y., & Yang, X. (2015). Tiny imagenet visual recognition challenge. CS 231N, 7(7), 3.


Meta-review: The majority of reviewers lean towards accepting the paper. vCp6 and a6mr raise the issue of generality, which they deem adequately addressed in the rebuttal. a6mr also questions on privacy concerns, but accepts the understanding that the key focus of this paper is on contribution evaluation, which is complementary. AsTD raises the issue of malicious users, which is sufficiently addressed in the rebuttal and authors are encouraged to include the discussion in the revised version. AA9d questions the assumptions in SPACE and is satisfied by the responses. HEwC requests more information on communication efficiency, where the results in the response are important and may be included in the revised version. Overall, the meta-reviewer agrees with the reviewers that the paper may be accepted. Authors are encouraged to include the above revisions.
Decision: Accept (poster)