SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our study, we adopt a similar perspective to Snell et al. [47] by considering the clients' local datasets as the support set. From the embedding space derived through the Federated Knowledge Amalgamation described earlier, we construct prototypes for each client, denoted as  $\mathcal{P}_i^{(j)}$ , representing the prototype for class  $j$  of the  $i$ -th client. To address the challenge of evaluation complexity dependence on the size of the validation set, we introduce a modification to the evaluation step. Rather than iterating through all validation samples, we devise a strategy where prototypes are built for the server's validation set. These prototypes, denoted as  $\mathcal{P}_{\mathcal{V}}^{(j)}$ , represent the prototypes for class  $j$  of the validation set. Model evaluation is then performed by comparing the similarities of the prototypes between the clients and the server. This modification effectively eliminates the computational burden associated with iterating through all validation samples. The prototypes are built as follows:
$$
\mathcal {P} _ {i} ^ {(j)} = \frac {1}{| \mathcal {D} _ {i , j} |} \sum_ {(x, y) \in \mathcal {D} _ {i, j}} f \left(\theta^ {*}, x\right) \tag {8}
$$
where  $\mathcal{D}_{i,j}$  implies the data that is class  $j$  in the client  $i$ -th dataset and function  $f(\cdot)$  is used to extract feature from data  $x$  with model weight  $\theta^{*}$ . We set the prototype of clients as a zero vector if the client does not possess any data of the corresponding class, such that  $\mathcal{P}_i^{(j)} = 0$  if  $|\mathcal{D}_{i,j}| = 0$ . When clients cooperate in coalitions, the prototypes for class  $j$  of coalition  $\mathcal{S}$  is denoted as  $\mathcal{P}_S^{(j)}$ , which can simply be achieved by the weighted sum of prototypes in the coalition.
$$
\mathcal {P} _ {\mathcal {S}} ^ {(j)} = \sum_ {i \in \mathcal {S}} \frac {\left| \mathcal {D} _ {i , j} \right|}{\sum_ {i \in \mathcal {S}} \left| \mathcal {D} _ {i , j} \right|} \mathcal {P} _ {i} ^ {(j)} \tag {9}
$$
We then define the performance of the coalition by the confidence of correctly classifying the given prototypes of the validation set. The confidence is calculated by the relative value between the
similarity of the same class and the sum of the similarities of all classes, while a softmax function is applied to ensure the value lies within the range [0, 1]. To be more specific, the value is computed as follows:
$$
V ^ {\prime} (\mathcal {S}) = \frac {1}{| \mathcal {C} |} \sum_ {j} \frac {e ^ {S i m \left(\mathcal {P} _ {\mathcal {V}} ^ {(j)} , \mathcal {P} _ {\mathcal {S}} ^ {(j)}\right)}}{\sum_ {k} e ^ {S i m \left(\mathcal {P} _ {\mathcal {V}} ^ {(j)} , \mathcal {P} _ {\mathcal {S}} ^ {(k)}\right)}}, \tag {10}
$$
where  $|\mathcal{C}|$  represents the number of classes in the classification task. We adopt cosine similarity as our similarity function.
4.3 Contribution Evaluation
Shapley value is applied to evaluate the participant contribution. However, adopting the model performance  $V'$  as a utility function may usually lead to a violation of rationality. This violation occurs when the marginal performance gain from additional clients decreases as the coalition size increases, as observed in [32]. Consequently, the expected value of the marginal gain becomes smaller than the individual value, thereby violating individual rationality [3], which states that  $c_i \geq u(\{i\})$ , where  $u(\cdot)$  denotes a utility function. To illustrate this issue, we provide an example below:
Example 4.1. Given two clients, each client's local dataset allows them to achieve model accuracy of 0.8 and 0.6, respectively. However, when collaborating in a federated learning setting, they can attain a model performance of 0.9. The Shapley value of each participant can be determined by using the model performance as the utility function, resulting in Shapley values of them becoming 0.55 and 0.35. Thus, their payoffs are lower with cooperation.
Despite the seemingly irrational nature of the decision, data owners in real-world scenarios demonstrate a willingness to participate in the federation in the above example due to the higher level of user satisfaction achieved by a model with an accuracy of 0.9. Relying solely on model performance as the utility function fails to adequately capture the underlying user satisfaction. Based on this observation, we rectify the conventional utility function with a logistic function [9], representing the percentage of users satisfied with the model performance.
$$
V (\mathcal {S}) = \frac {1}{1 + e ^ {- k \left(V ^ {\prime} (\mathcal {S}) - T\right)}} \tag {11}
$$
where  $k$  denotes the logistic growth rate, which determines the steepness of the curve and  $T$  represents the threshold, where half of the population is satisfied. The choice of  $T$  should be greater than  $\max_{i} V'(\{i\})$  to encourage the cooperation of participants because the convexity of the left-sided logistic function would up-weight the utility value of stronger coalitions. When  $k$  is set to infinity, the individual rationality would be satisfied since the  $V(\{i\})$  is compressed to zero and participant contribution is lower-bounded by zero.
Finally, the contribution of  $i$ -th participant can be calculated by Shapley value defined in Equation 4. The proposed rectification of the utility function with the logistic function makes the utility value more sparse because the utility of the coalition whose performance does not surpass the threshold  $T$  would be highly compressed. Inspired by [32], we propose a pruning technique that leverages the sparsity to further accelerate the computation of the Shapley value. We set a pruning threshold  $\tau$ , such that given a coalition  $\mathcal{S}$  with  $V(\mathcal{S}) \leq \tau$ , we prune the computation of subsets of  $\mathcal{S}$ .<sup>2</sup>
Table 1 presents a comparative analysis of SPACE's computational complexity in contrast to established methods. The sampling-based approximation of the Shapley value, applied in TMC-Shapley, GTG-Shapley, and SPACE, necessitates sampling complexity of  $n \log n$ , as studied in [35]. It is worth noting that SPACE achieves a notably reduced computational complexity in calculating the Shapley value at merely  $O(n \log n \cdot |\mathcal{C}|)$ , which represents a substantial reduction compared to the  $O(n \log n \cdot R_g \cdot |\mathcal{D}_{\mathcal{V}}|)$  required by other Shapley value-based approaches. This efficiency gain is attributed to the use of the single-round amalgamation and the prototype-based model evaluation. The prototype-based model evaluation eliminates the necessity for repeated iteration over the validation set. Consequently, the computation of the Shapley value is facilitated by direct comparisons of
Further discussion is provided in the appendices.
2The overall algorithm of SPACE is shown in appendices.
Table 1: Comparison of complexity between previous works.  $R_{l}$ ,  $R_{g}$ , and  $R_{a}$  denote the number of iterations for local training, global communication rounds, and knowledge amalgamation.  $|\theta|$  represents the number of model parameters,  $|\mathcal{P}|$  denotes the dimension of prototypes, and  $|\mathcal{C}|$  indicates the number of classes.  $|\mathcal{D}_i|$  and  $|\mathcal{D}_{\mathcal{V}}|$  represent the sizes of the local dataset and the validation set.
Method	Communication Cost	Client Computation Cost	Server Computation Cost
Real Shapley	2n·Rg·|θ|	2n·Rg·Rl·|Di|	2n·|Dv|
GT-Shapley	n(log n)2·Rg·|θ|	n(log n)2·Rg·Rl·|Di|	n(log n)2·|Dv|
TMC-Shapley	n log n·Rg·|θ|	n log n·Rg·Rl·|Di|	n log n·|Dv|
GTG-Shapley	Rg·|θ|	Rg·Rl·|Di|	n log n·Rg·|Dv|
DIG-FL	Rg·|θ|	Rg·Rl·|Di|	Rg·|Dv|
SPACE(Ours)	|θ| + |P| · |C|	Rl·|Di|	n log n·Ra·|Dv|
distances between the weighted sum of prototypes. Moreover, by employing the knowledge amalgamation technique, SPACE is capable of evaluating participant contributions with a single communication round, thereby significantly alleviating the communication and computational load on clients with limited resources by a factor of  $R_{g}$ . This acceleration does introduce an additional computation load for knowledge amalgamation on the server, with a complexity of  $O(n\log n\cdot R_a\cdot |\mathcal{D}_{\mathcal{V}}|)$ . However, this trade-off is generally more suitable for generic federated learning frameworks, given that the server typically possesses superior computational resources compared to the clients.
In this section, we demonstrate the effectiveness of the proposed SPACE method for evaluating participant contributions. We compare it with previous approaches and show that SPACE achieves superior computational efficiency without compromising performance. We also evaluate the SPACE on two federated-learning tasks, client reweighting, and client selection.
Datasets and Non-IID Setting. Following [57], we conduct experiments on the widely adopted image dataset MNIST [24] and CIFAR10 [22]. For a fair comparison, we adopted the experimental settings used in the HFL framework [57] with two different scenarios: unlabeled data and non-IID data. In the unlabeled data scenario, we deliberately introduced label corruption in a variable proportion of the total clients. Specifically, we corrupted different percentages of clients, ranging from  $0\%$  to  $80\%$ . Within each client, we intentionally corrupted  $50\%$  of the local training labels to simulate the presence of unlabeled data. For the non-IID scenario, we selected a subset of clients, ranging from  $0\%$  to  $80\%$ , and assigned them local datasets containing incomplete categories. We set the total number of clients to 10 for the MNIST dataset and 5 for the CIFAR10 dataset.
Baselines and Implementation. We compare SPACE with four state-of-the-are methods, including i) GT-Shapley [19],ii) TMC-Shapley [16], iii) GTG-Shapley [32] and iv) DIG-FL [57]. 3 We also compare a variant of SPACE, v)  $SPACE(Avg)$ , which applies FedAvg instead of knowledge amalgamation for aggregation. For adjusting the utility function, we empirically set  $k$  as 100 while  $T$  as 0.95 and 0.5 for evaluation on MNIST and CIFAR10.
Evaluation Metric. The performance of all approaches is assessed from two perspectives: accuracy and efficiency. To evaluate the accuracy, we employ Pearson's Correlation Coefficient to measure the correlation between the estimated and actual Shapley values. The actual Shapley values are obtained by performing retraining for a total of  $2^{n}$  times. In terms of efficiency, we consider two vital metrics. Execution time (in seconds) is measured on a single V100 GPU without parallel training to assess the time efficiency of the approaches. Communication cost (in megabytes) is calculated according to the formula  $\text{Comm} = 2 \cdot n \cdot R_{g} \cdot |\theta|$ , where  $R_{g}$  denotes the number of communication rounds,  $|\theta|$  represents the number of model parameters(MB) and constant of 2 stands for the upload and download of the models.
<sup>3</sup>Since GT-Shapley and TMC-Shapley are centralized learning schemes, we set the retraining rounds for TMC-Shapley to  $n \log n$ , and for GT-Shapley to  $n (\log n)^2$ , where  $n$  is the number of participants for the federated learning scenario.
Table 2: Results of participant contribution evaluation.
Dataset	Scenario	GT	TMC	GTG	DIG-FL	SPACE(Avg)	SPACE
MNIST	Non-IID	0.6877	0.9824	0.9287	0.8715	0.9713	0.9448
Mislabel	0.4230	0.9507	0.9608	0.9580	0.9529	0.9612
Time(s)	97137	84796	62473	315	294	160
Comm(MB)	11813.12	10292.48	35.2	35.2	35.2	1.76
CIFAR	Non-IID	0.6089	0.8877	0.8208	0.7546	0.9540	0.9290
Mislabel	0.5192	0.9595	0.4148	0.9598	0.9565	0.9641
Time(s)	7468	4950	835	536	315	295
Comm(MB)	307800	202920	11400	11400	11400	570
Figure 2: The effect of client reweighting on model performance.
5.2 Quantitative Results
Participant contribution. Table 2 presents the outcomes of participant contribution evaluation. The proposed SPACE approach demonstrates outstanding efficiency and achieves the highest Pearson correlation coefficient (PCC) in scenarios involving unlabeled data. In terms of PCC, SPACE performs similarly to SPACE(Avg), but with superior efficiency resulting from the single-round knowledge amalgamation. The communication and computational costs associated with GT-Shapley and TMC-Shapley are excessively high due to the requirement for model retraining. However, TMC-Shapley shows improved efficiency by attaining high PCC results with fewer sampled permutations. GTG-Shapley adopts a reconstruction technique to expedite the computation process, which effectively reduces the required communication cost, yet the testing time in GTG-Shapley remains considerably long. The resource-saving version of DIG-FL achieves comparable computational efficiency to SPACE. Nevertheless, it has a tendency to overestimate the contribution of clients with incomplete categories and exhibits inferior PCC results in non-IID scenarios.
Client Reweighting. In non-IID scenarios or when some clients have erroneous data, aggregating local models based on data ratio may lead to suboptimal performance. Leveraging participant contribution as a weighting factor can enhance the robustness of federated learning [57]. We propose two reweighting mechanisms, namely static and dynamic, with the former employing participant contribution calculated using the knowledge amalgamated model and the latter recalculating participant contribution for each communication round using the current model. Figure 2 shows the reweighting results. Despite the dynamic approach introducing additional computational overhead, it yields improved performance. The static and dynamic approaches achieve accuracy improvements of  $2.53\%$  and  $2.69\%$ , respectively, under the non-IID scenario, and  $3.21\%$  and  $3.52\%$ , respectively, under the mislabel scenario, for the CIFAR10 dataset. The DIG-FL approach measures client contribution in federated learning by the similarities between the gradients of the local datasets and the gradients of the validation set. However, this approach disregards clients in communication rounds where the two gradients diverge, leading to significant variances in the federated training process.
Client Selection. Client selection is a critical issue when dealing with a large number of clients. Li et al. [28] proposed a client sampling approach based on the multinomial distribution (MD) that ensures zero bias but has significant variances. To tackle this issue, Fraboni et al. [13] suggested clustered sampling (CS), which has been theoretically shown to reduce variance. They proposed two CS algorithms: Algorithm 1 clusters clients based on data ratio, and Algorithm 2 incorporates the similarities between local and global models. Algorithm 2 outperforms both MD and Algorithm 1. However, Algorithm 2's time complexity is  $O(R_{g} \cdot (n^{2}|\theta| + X))$ , where  $|\theta|$  is the number of model parameters and  $O(X)$  is the complexity of the clustering algorithm. This computational cost
Figure 3: The effect of client selection on model performance.
results from updating a similarity matrix and re-clustering at each communication round. We propose a method that combines SPACE with CS. We utilize the prototypes obtained from SPACE to construct the similarity matrix, eliminating the need for repeated updates. Additionally, we enhance the weighting by incorporating the estimated contribution, resulting in a new weighting scheme for client  $p_i$  given by  $(1 - \beta)p_i + \beta c_i$ , where  $\beta$  is a hyper-parameter that balances the two terms. As in [13], the CIFAR10 dataset is distributed among 100 clients using a Dirichlet distribution with  $\alpha \in \{10, 0.001\}$ , where smaller  $\alpha$  implies larger heterogeneity. We set  $\beta = 0$  and  $\beta = 0.5$  when  $\alpha = 10$  and  $\alpha = 0.001$ , respectively. Figure 3 presents the results of client selection. Our approach, which combines prototypes and participant contribution, outperforms prior methods, indicating the superiority of our proposed method. This improvement can be attributed to the fact that prototypes enhance the accuracy of the clustering process. Furthermore, as discussed in Section 5.2, the incorporation of participant contribution has proven to be useful in enhancing the performance of federated learning, especially in non-IID settings.
We propose Single-round Participant Amalgamation for Contribution Evaluation (SPACE), an efficient approach for evaluating participant contribution in federated learning. The novelty of SPACE lies in two key components: Federated Knowledge Amalgamation and Prototype-based Model Evaluation. By leveraging a single communication round, Federated Knowledge Amalgamation constructs a robust embedding space, while Prototype-based Model Evaluation reduces the complexity associated with the validation set size. Our experimental results demonstrate that SPACE outperforms existing methods in contribution evaluation and shows versatility in client reweighting and selection. However, efficient client selection for amalgamation and the impact of the validation set quality are areas that require further exploration in future research.
In our study, we highlight the importance of participant contribution through three applications: incentive mechanisms, client reweighting, and client selection. Furthermore, the potential applications extend to client debugging and attack detection. Since the computational complexity associated with participant contribution hampers its widespread implementation, we propose a novel FL framework, SPACE, to alleviate this issue and pave the way for broader adoption of participant contribution. An efficient algorithm for evaluating participants provides decision-makers with valuable insights before initiating extensive federated learning processes, which can be both computationally and temporally intensive. Furthermore, rapid evaluation offers enhanced flexibility when dealing with dynamic cooperative entities, facilitating quick adaptation to new collaborations—a capability that previous methods lack. Our proposed approach, SPACE, significantly contributes to these applications. However, our framework relies on the assumption of having a proper and reliable validation set with accurately labeled data. This prerequisite can be particularly challenging to fulfill in real-world settings and thus limits the application to specific scenarios characterized by significant commercial incentives [33, 65]. Exploring methods to decrease dependence on the validation set while maintaining functionality, even when erroneous data impact many clients, presents an intriguing direction for future exploration. We defer this critical aspect to be addressed in future research efforts.
This work was supported in part by the National Science and Technology Council, Taiwan, under grant NSTC-112-2223-E-002-015, and by the Ministry of Education, Taiwan, under grant MOE 112L9009.
[72] J. Zhang, Y. Wu, and R. Pan. Incentive mechanism for horizontal federated learning based on reputation and reverse auction. Proceedings of the Web Conference 2021, 2021.
[73] S. Q. Zhang, J. Lin, and Q. Zhang. A multi-agent reinforcement learning approach for efficient client selection in federated learning. In AAAI Conference on Artificial Intelligence, 2022.
[74] L. Zhu, Z. Liu, and S. Han. Deep leakage from gradients. In Neural Information Processing Systems, 2019.
[75] Z. Zhu, J. Hong, and J. Zhou. Data-free knowledge distillation for heterogeneous federated learning. pages 12878-12889, 2021.
(a) Impact of different  $T$
(b) Impact of different  $k$
Figure 4: The impact of modification using the logistic function with different parameters.
A.1 Sensitivity test of utility function
To begin with, we conduct a sensitivity analysis on the utility function by varying two hyperparameters:  $k$  and  $T$ . The hyperparameter  $k$  determines the logistic growth rate, thereby impacting the steepness of the curve, while  $T$  represents the threshold at which half of the population is considered satisfied. This adjustment is aimed at capturing user satisfaction, which serves as a practical incentive for participants to actively participate in federated learning (FL) collaborations.
Figure 4 depicts the calculated actual Shapley values for the MNIST dataset in the presence of mislabeled data, considering the existence of 8 corrupted clients. Various combinations of the hyperparameters  $k$  and  $T$  are explored in this analysis. Specifically, Figure 4a focuses on investigating the impact of different values of  $T$  from the set 1.0, 0.9669, 0.95. The selection of 1.0 as the threshold follows a greedy approach, while 0.9669 represents the performance achieved when considering all participants. Furthermore, 0.95 corresponds to an empirically derived threshold based on a comprehensive dataset understanding.
The findings reveal that selecting a greedy selection for  $T$  leads to the disparity between clients with high and low contributions, due to the convex nature of the left-sided logistic function. Conversely, empirically setting an appropriate value for  $T$  reduces the marginal gain for coalitions that surpasses the threshold, owing to the concave property of the right-sided logistic function. Consequently, this approach minimizes the discrepancy between clients with high and low contributions.
Figure 4b examines the influence of different values of  $k$ . A larger value of  $k$  indicates a steeper logistic function, thereby strongly favoring coalitions that surpass the threshold  $T$  and consequently widening the gap between clients with high and low contributions. However, as  $k$  approaches infinity, the logistic function transforms into a ReLU function, disregarding all coalitions that fail to surpass the threshold  $T$  and assigning equal value to those that do. Consequently, in this scenario, the relative performance, as depicted by the difference between client 0 and client 1 in Figure 4b, would be disregarded due to a hard threshold.
The selection of hyperparameters  $k$  and  $T$  should be determined by the desired characteristics of the applications to enhance the influence of coalitions with superior performance or to ensure individual rationality. Thus, we maintain them as carefully selected hyperparameters, allowing for flexibility and customization based on specific application requirements.
A.2 Ablation Study for smaller  $G$
The federated knowledge amalgamation process utilizes FPMs comprising only two convolution layers, each of which introduces minimal additional GPU memory usage. As a result, in the reported experiments, we are able to simultaneously distill the knowledge from all client models on a single V100 GPU. However, as the number of clients  $n$  increases, the limited memory capacity of the GPU
Table 3: Federated knowledge amalgamation with different  $G$  .
MNIST	CIFAR10
G	Non-IID	Mislabel	runtime	Non-IID	Mislabel	runtime
n	0.9448	0.9612	160	0.9290	0.9641	295
2	0.9427	0.9726	633	0.9333	0.9645	299
Table 4: Results of participant contribution evaluation on Tiny-ImageNet dataset.
Dataset	Scenario	GT	TMC	GTG	DIG-FL	SPACE(Avg)	SPACE
Tiny-ImageNet	Non-IID	0.9082	0.9610	0.7425	0.8944	0.9175	0.9092
Mislabel	0.7833	0.9233	0.7998	0.8405	0.9293	0.9256
Time(s)	84994	69212	98855	266	252	453
Comm(MB)	2639200	2097600	8000	8000	8000	400
may pose a challenge. To address this issue, we propose a hierarchical amalgamation framework that mitigates the high GPU memory requirement.
To evaluate the effectiveness of hierarchical amalgamation, we conducted experiments by amalgamating groups of  $G = 2$  teacher models instead of all  $n$  teacher models in a hierarchical manner. The results, shown in Table 3, demonstrate that knowledge transfer from the local datasets can still be achieved through hierarchical amalgamation, yielding performance comparable to amalgamating all clients simultaneously. Although hierarchical amalgamation alleviates the high GPU memory requirement, it introduces a computational complexity increase in the knowledge amalgamation process from  $O(n)$  to  $O(n\log n)$ , as evident from Table 3.
A.3 Comparison of estimated Shapley Value