SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5 illustrates the scatter plots depicting the estimated Shapley values compared to the actual Shapley values. Regarding the centralized method, TMC-Shapley outperforms GT-Shapley, indicating that Monte Carlo sampling is more efficient for approximating the Shapley value than group testing. To accelerate the approximation process, GTG-Shapley employs gradient-based model reconstruction. While it performs well on the MNIST dataset, its performance deteriorates on the CIFAR10 dataset due to the increased complexity of both the model and the dataset. The results indicate that DIG-FL tends to overestimate clients' contributions with incomplete class data, leading to suboptimal performance in non-IID scenarios. Conversely, the proposed SPACE approach exhibits strong performance across all scenarios.
A.4 Results on Tiny-ImageNet
To assess the efficacy of the proposed SPACE framework on more complicated datasets, an additional experiment was conducted, incorporating a portion of the Tiny-ImageNet dataset [23]. This subset encompasses 50 categories and involves 10 clients. The results are provided in Table 4. The SPACE method demonstrates satisfying performance in terms of Pearson's Correlation Coefficient (PCC). It's noteworthy that achieving feature alignment in a complex dataset like Tiny-ImageNet is more challenging. Consequently, the knowledge amalgamation process requires an extended number of epochs to converge, particularly in scenarios where the data distribution across clients is non-IID, and clients possess data from disjoint classes. On the other hand, the SPACE(Avg) approach, which employs FedAvg for obtaining the embedding space, may offer superior time efficiency in cases with a limited number of communication rounds. Nonetheless, the advantages in communication efficiency stemming from single-round amalgamation continue to position SPACE as the preferred choice when mitigating communication costs is a key consideration.
A.5 Scenario without labeled validation set
Given the inherent challenges associated with acquiring a sufficiently well-labeled validation dataset, there arises an important question regarding the adaptability of the proposed approach in cases where label information is unavailable. To investigate this, we conducted supplementary experiments employing unlabeled data for knowledge amalgamation. In this context, we drew inspiration from the work of [44] and modified the loss function to incorporate weights based on entropy purity rather than the confidence associated with accurate result predictions. Specifically, clients with lower en
Figure 5: Estimated Shapley and the actual Shapley value of all approaches under all scenarios.
Table 5: Results of SPACE with and without labels.
MNIST	CIFAR10	Tiny-ImageNet
Label	Non-IID	Mislabel	Non-IID	Mislabel	Non-IID	Mislabel
✓	0.9448	0.9612	0.9290	0.9641	0.9092	0.9256
0.8772	0.9611	0.8653	0.9641	0.9091	0.9231
tropy values were assigned greater weight in the loss function. The experimental results are shown in Table 5. While some marginal reductions in performance are evident, particularly in scenarios characterized by Non-IID data distributions, it is important to note that these performance decreases are primarily attributed to the absence of label information. Nevertheless, the results clearly demonstrate the viability of leveraging unlabeled data as a practical alternative for knowledge amalgamation, offering a feasible solution in cases where the acquisition of labeled data is challenging.
The algorithmic representation of the proposed SPACE approach is presented in Algorithm 1. Initially, the server distributes the initial model  $\theta$  to all participating clients. Each client then performs empirical risk minimization to obtain a locally optimized model  $\theta_{i}$  and transmits it to the server. Using the locally optimized models as teacher models, the federated knowledge amalgamation tech
Algorithm 1 SPACE
Require: Initial FL model  $\theta$  , client set  $\mathcal{N}$  , local datasets  $\{\mathcal{D}_1,\dots,\mathcal{D}_n\}$  , validation dataset  $\mathcal{D}_{\mathcal{V}}$  , fea  
ture encoder  $f(\cdot)$  and utility function  $V(\cdot)$    
1: # Knowledge Amalgamation   
2: for  $i = 1,\ldots ,n$  do   
3:  $\theta_{i} = \mathrm{LocalUpdate}(\theta ,\mathcal{D}_{i})$    
4:  $\theta^{*} = \mathrm{KA}(\theta ,\{\theta_{1},\dots,\theta_{n}\} ,\mathcal{D}_{\mathcal{V}})$    
5: # Build server prototypes and clients' prototypes   
6: for  $j = 1,\ldots ,|\mathcal{C}|$  do   
7:  $\mathcal{P}_{\mathcal{V}}^{(j)} = \frac{1}{|\mathcal{D}_{\mathcal{V},j}|}\sum_{x}f(\theta^{*},x),\forall x\in \mathcal{D}_{\mathcal{V},j}$    
8: for  $i = 1,\ldots ,n$  do   
9: for  $j = 1,\ldots ,|\mathcal{C}|$  do   
10:  $\mathcal{P}_i^{(j)} = \frac{1}{|\mathcal{D}_{i,j}|}\sum_x f(\theta^*,x),\forall x\in \mathcal{D}_{i,j}$    
11: # Evaluate contribution with Prototype-based Model Evaluation   
12:  $\mathbf{r} = 0$    
13:  $\{c_1,\dots,c_n\} = 0$    
14: while Convergence criteria not met do   
15:  $\mathrm{r} = \mathrm{r} + 1$    
16:  $\pi^r\sim \Pi$    
17:  $v_{0} = V(\mathcal{N})$    
18: for  $k = 1,\dots,n - 1$  do   
19: if  $v_{k - 1} > \tau$  then   
20:  $\mathcal{S} = \{\pi^r [k + 1],\dots,\pi^r [n]\}$    
21: for  $j = 1,\dots,|\mathcal{C}|$  do   
22:  $\mathcal{P}_{\mathcal{S}}^{(j)} = \sum_{i\in \mathcal{S}}\frac{|D_{i,j}|}{\sum_{i\in\mathcal{S}}|D_{i,j}|}\mathcal{P}_{i}^{(j)}$    
23:  $v_{k} = V(S)$    
24: else   
25:  $v_{k} = 0$    
26:  $c_{\pi^r [k]} = \frac{r - 1}{r} c_{\pi^r [k]} + \frac{1}{r} (v_{k - 1} - v_{k})$    
27: return  $\{c_i,\dots,c_n\}$
nique is employed to distill knowledge into the server model, resulting in an enhanced server model denoted as  $\theta^{*}$ . Subsequently, prototypes are constructed using  $\theta^{*}$  for both the validation set and local datasets. Following prototype construction, SPACE proceeds to calculate participant contributions using the Shapley value, leveraging the prototype-based model evaluation. To ensure convergence, SPACE samples multiple permutations of participants. As the modified utility function may compress the utility values of small coalitions, SPACE performs pruning operations at the sequence level. If the utility value  $V(S)$  of a coalition  $S$  falls below the pruning threshold, subsets of  $S$  are pruned as their marginal gains become negligible.
C Potential Malicious Attack
Malicious attack by data poisoning. The manipulation of data labels, a frequently employed malicious strategy by clients engaged in data poisoning attacks, is denoted as "label-flipping" in reference [6]. In our work, we adopted the term "mislabeled" to refer to unintentional and intentional mislabeling instances. In this case, malicious clients attempt to disrupt the training process by submitting gradients derived from mislabeled data. Explicitly, as depicted in Table 2, the results highlight the effectiveness of SPACE in pinpointing those malevolent clients harboring inaccurately labeled data. With the server's privately owned validation set, our framework efficiently identifies such malicious clients, even if they form the majority in the federated process. In addition, the experimental results in Figure 2 demonstrate the robustness of our framework in the federated training. It is worth noting that the SPACE proposed employs contributions as weighting factors, thereby reducing the negative impact of malicious participants, i.e., the malicious clients usually contribute less due to their distributions differing from that of the server.
Maliciously inflating contribution. Clients may aim to inflate their contributions maliciously to obtain more significant rewards. If the prototypes from the validation set become exposed, clients might exploit this knowledge to boost their contributions in an ad-hoc manner. In such a case, the SPACE utilizes prototype-based model evaluations, where all clients have access solely to the feature extraction layers of the amalgamated model while excluding access to the fully-connected layers, i.e., the final score. This strategic approach facilitates the creation of prototypes without compromising the confidentiality of the distribution that underlies the validation set. Note that another potential threat is clients employing the zeroth-order optimization technique [4] to derive prototypes with disproportionately high contributions. In this case, we can simply detect this attack by limiting the client's access to the server in that the SPACE does not require each client to recalculate their contributions frequently.
In addition to the aforementioned attacks directly associated with contribution evaluation, federated learning is known for its vulnerability to a spectrum of attacks [6], including but not limited to reconstruction attacks [74]. It is noted that to counter these threats, researchers have proposed defense mechanisms, such as gradient pruning [74], data representation perturbation [49], and differential privacy (DP) [60]. These works are essential. However, they are, in essence, orthogonal to the main theme of this paper, which focuses on contribution evaluation. Nevertheless, these privacy-preserving strategies can be integrated with our method, working collaboratively to enhance data privacy protection.