SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yi-Chung Chen
r10942081@ntu.edu.tw
Hsi-Wen Chen
hwchen@arbor.ee.ntu.edu.tw
Shun-Gui Wang
r11921099@ntu.edu.tw
Ming-Syan Chen
mschen@ntu.edu.tw
The evaluation of participant contribution in federated learning (FL) has recently gained significant attention due to its applicability in various domains, such as incentive mechanisms, robustness enhancement, and client selection. Previous approaches have predominantly relied on the widely adopted Shapley value for participant evaluation. However, the computation of the Shapley value is expensive, despite using techniques like gradient-based model reconstruction and truncating unnecessary evaluations. Therefore, we present an efficient approach called Single-round Participants Amalgamation for Contribution Evaluation (SPACE). SPACE incorporates two novel components, namely Federated Knowledge Amalgamation and Prototype-based Model Evaluation to reduce the evaluation effort by eliminating the dependence on the size of the validation set and enabling participant evaluation within a single communication round. Experimental results demonstrate that SPACE outperforms state-of-the-art methods in terms of both running time and Pearson's Correlation Coefficient (PCC). Furthermore, extensive experiments conducted on applications, client reweighting, and client selection highlight the effectiveness of SPACE. The code is available at https://github.com/culiver/SPACE.
With the rapid growth of mobile services, the combination of rich user interactions and robust sensors means they are able to access an unprecedented amount of data. While conventional machine learning approaches require centralizing the training data in a single machine or a data center, which may limit the scalability of machine learning applications, Federated Learning (FL) [36] has been proposed to leave the training data distributed on the client side and then aggregate locally computed updates, e.g., the gradient of the neural network, to a central coordinating server, without accessing the raw data. A line of subsequent efforts has significantly improved the efficiency [1, 15, 30, 62, 70, 73], and robustness [17, 38, 63] of FL. Moreover, such distributed training paradigm has been adapted to client personalization [8, 27, 40, 41], and heterogeneous model architecture [10, 75], making it even more practical.
Recently, there has been increasing interest in evaluating the participant's contribution to FL for a fair incentive mechanism [12, 29, 33, 45, 46, 55, 65, 69, 71, 72] based on the participant contribution in attracting data owners. Additionally, deciding the participant weights [54, 57] during the training process based on their contributions can accelerate the convergence of the training process but also enhance the model's robustness by alleviating the negative effect from the confused or even incorrect
data. Moreover, if the communication budget is limited, participant contributions can be utilized as a reference to select important participants [7, 11, 13, 53, 56, 73], reducing communication costs while maintaining the model's performance.
One straightforward method to evaluate participant contribution is to assess their individual performance independently. For instance, the accuracy of the local model has been employed as a measure [39], as well as the consistency between the local and global models, indicating their similarity to the consensus of the coalition [45, 72]. However, these approaches do not fully consider the impact of client cooperation. To consider cooperation, some studies have adopted the influence function [21] from centralized data evaluation to assess participant contribution [57, 64]. However, calculating the influence function involves clients sending their locally computed Hessian matrices to the server, leading to unacceptable client computation and communication costs. On the other hand, researchers have explored using the Shapley value [42] in cooperative game theory to measure participant contributions. While the exact computation of the Shapley value requires enumerating all possible client combinations, Song et al. [48] approximate the target model with gradient updates to avoid model retraining. To further speed up the computation, Wei et al. [59] and Liu et al. [32] propose truncation techniques, eliminating unnecessary model reconstruction and evaluation.
Despite the aforementioned efforts, the approximation cost of the Shapley value can still be prohibitively high for the following two reasons. i) Multi-Round Training: FL algorithms, such as FedAvg, involve multiple communication rounds wherein models are transmitted between the server and clients, and local training takes place on the client side before achieving model convergence. Previous studies [32, 48, 57-59] compute participant contributions for each communication round and aggregate them to obtain the overall participant contribution. Consequently, the communication and computation efforts escalate with an increase in the number of communication rounds. ii) Dataset-Dependent Evaluation: As demonstrated in [32], the evaluation of models constitutes the primary computational time component required for calculating the Shapley value. Even with the adoption of permutation sampling approximation, the complexity of evaluating per-round contributions depends on the size of the validation dataset on the server. As the validation set continually grows, the evaluation process becomes computationally unacceptable.
In this paper, we propose Single-round Participants Amalgamation for Contribution Evaluation (SPACE) to efficiently calculate the participant contribution in FL. Specifically, we analyze the data distribution between the server and the client dataset. If the server and client distributions are similar, the client dataset proves helpful in correctly classifying the data in the validation set of the server. While previous works [37, 51, 52, 68] have shown that prototypes, i.e., the embedding vectors representing each class, can be employed as a privacy-preserving approach to compare data distribution, we propose Prototype-based Model Evaluation, which evaluates the model performance by measuring the similarities between the local and server prototypes. Therefore, the distribution of the validation set can be constructed and thus eliminates the dependency of evaluation complexity on the size of the validation set.
To derive meaningful prototype embeddings, we propose a novel approach called Federated Knowledge Amalgamation, wherein client models can be viewed as multiple distributed teacher models, and their knowledge can be distilled to educate the server model simultaneously. Thus, federated knowledge amalgamation enables us to distill knowledge from all local models in just one communication round to reduce the time for prototype construction. The educated model is distributed to each participant, who then constructs prototypes based on their local data. We then evaluate the model performance by measuring the similarities between server and client prototypes. Besides, unlike previous works that adopt the model performance as the utility function, we modify the utility function by incorporating a logistic function. This adjustment aims to reflect users' satisfaction better, thereby enhancing the rationality of the utility function in real-world scenarios. Extensive results demonstrate the advantage of SPACE in multiple applications, including contribution evaluation, client re-weighting, and client selection.
- We propose SPACE, an efficient and effective approach for client contribution evaluation in an FL setting, which significantly reduces the required communication and computational cost and exhibits outstanding performance in terms of Pearson's correlation coefficient (PCC).
- We introduce Prototype-based Model Evaluation, which effectively removes the computation dependency on the size of the validation set. To efficiently establish a robust embedding space for
building prototypes, we introduce Federated Knowledge Amalgamation that enables the aggregation of embedding space information from local models in an FL setting, all within a single round of communication.
- Extensive experiments have been conducted to show that SPACE outperforms previous state-of-the-art methods in terms of both running time and PCC. SPACE consistently exhibits exceptional performance in applications including client reweighting and selection.
Contribution evaluation in FL. Client contribution evaluation in federated learning (FL) has gained significant attention for its potential applications, including attack detection [14, 31], debugging [25, 26], incentive mechanisms [33, 45, 46, 48, 55, 65, 72], and convergence acceleration [54, 57]. This concept draws inspiration from centralized data evaluation tasks [16, 19]. Recent works have proposed methods tailored for FL. Pandey et al. [39] consider the relative accuracy of the local model as individual contribution. Zhang et al. [72] calculate the cosine similarity between local gradients and a unit vector pointing towards the optimal model as contributions. Shi et al. [45] measure contribution based on the cosine similarity between local and global gradients. However, these methods do not fully capture the impact of cooperation. To address this limitation, the Shapley value is widely adopted for evaluating client contribution in FL due to its desirable properties, including efficiency, symmetry, linearity, and null player property. However, directly computing the Shapley value incurs exponential computational costs involving model retraining and inference. To alleviate this burden, Song et al. [48] reconstruct the target model using gradient updates to avoid model retraining. Wang et al. [58] enhance the evaluation using permutation sampling and a group testing approach. Additionally, Wei et al. [59] and Liu et al. [32] incorporate truncation techniques to reduce computation costs. Despite these efforts, approximating the Shapley value still requires significant computation.
Knowledge Amalgamation. Knowledge amalgamation (KA), a variant of knowledge distillation [18], involves a setup with one student model and multiple teacher models. KA has been widely adopted in prior works [5, 20, 34, 43, 44, 61, 66, 67] due to its desirable properties. Shen et al. [43] introduced KA, where compact feature representations are learned from teachers and used to update the student model. Shen et al. [44] proposed a strategy that extracts task-specific knowledge from diverse teachers to create component networks, which are then combined to construct the student network. Luo et al. [34] introduced a common feature learning scheme in heterogeneous network settings, transforming the features of all teachers into a shared space. Xie et al. [61] proposed a related approach applying KA to transfer prior knowledge from multiple pre-trained models. However, their focus is on knowledge transfer from given pre-trained models, while our approach is the first to employ KA within a conventional FL framework. In our setting, client models can be viewed as distributed teacher models, and their knowledge can be distilled to educate the server model.
Since federated learning aims to construct a unified, global machine-learning model to collect data information located on numerous remote devices (client), the objective is to develop this model while ensuring that the data generated by each device is processed and stored locally, with only intermediate updates being sent to a central server at regular intervals. The primary objective is usually to minimize the objective function as follows. Let  $\mathcal{N}$  denote a set consisting of  $n$  clients,
$$
\theta^ {*} = \arg \min  _ {\theta} \sum_ {i} ^ {n} p _ {i} L _ {i} (\theta), \tag {1}
$$
where  $p_i \geq 0$  and  $\sum_{i} p_i = 1$ .  $L_i$  is the objective function for the  $i$ -th client. Generally, each client may utilize the same objective function, such as the empirical risk, which can be defined as follows.
$$
L _ {i} = \sum_ {\left(x _ {i, j}, y _ {i, j}\right) \in \mathcal {D} _ {i}} l \left(\theta , x _ {i, j}, y _ {i, j}\right), \tag {2}
$$
where  $\mathcal{D}_i$  is the dataset in  $i$ -th client.  $p_i$  specifies the relative impact of each device, such as the uniform weighting  $\frac{1}{n}$  or depending on the number of samples in each dataset  $\frac{|\mathcal{D}_i|}{\sum_i^n |\mathcal{D}_i|}$ .
Figure 1: Overall architecture of SPACE. Initially, the clients train their respective local models using their local datasets. Subsequently, the locally optimized models are frozen and serve as teacher models to educate the student model on the server by Federated Knowledge Amalgamation. Then, the educated model  $\theta^{*}$  is distributed to all clients, who then utilize it to construct their corresponding prototypes  $\mathcal{P}_i$ . These prototypes are then uploaded to the server. The server employs the Prototype-based Model Evaluation approach to assess the model's performance and determine the contribution  $\mathbf{c}$  of each participant.
Despite early efforts to enhance the efficacy of federated learning, many data owners hesitate when it comes to partaking in data federation owing to apprehensions surrounding the possibility of inequitable remuneration. Furthermore, in federated learning, privacy concerns preclude the server from directly accessing unprocessed data stored on client devices. Consequently, there is a pressing need for a privacy-preserving federated contribution evaluation method that ensures equitable distribution of rewards.
Specifically, the evaluation of participant contributions can be abstracted into cooperative game scenarios. Given a cooperative game  $(\mathcal{N}, V)$ , where  $\mathcal{N}$  denotes the set of  $n$  participants and  $V(\cdot)$  is a utility function defined as  $V: 2^n \to \mathbb{R}$ , which assigns a value to a coalition  $S \in \mathcal{N}$ , the objective of payoff allocation is to determine a payoff allocation vector, denoted by  $\mathbf{c} = (c_1, \dots, c_n)$ , where  $c_i$  represents the payoff for the  $i$ -th participant in  $\mathcal{N}$ .
The idea of cooperative games has been extensively researched [2, 3], with the Shapley value being the most commonly used metric due to its existence and uniqueness in a cooperative game. Additionally, the Shapley value is characterized by a set of desirable axiomatic properties, including (i) efficiency, (ii) symmetry, (iii) null player, and (iv) linearity, which make it a desirable approach for evaluating contributions. The Shapley value is defined as follows,
$$
\phi_ {i} (\mathcal {N}, V) = \sum_ {S \in \mathcal {N} \backslash \{i \}} \frac {| S | ! (n - | S | - 1) !}{n !} (V (S \cup \{i \}) - V (S)). \tag {3}
$$
$S$  denotes the subset of participants from  $\mathcal{N}$ . The Shapley value can be seen as the average marginal gain that a participant joins the coalitions. To accelerate the approximation, the Monte Carlo estimation method for approximating Shapley values from [16] is adopted,
$$
\phi_ {i} = \underset {\pi \sim \Pi} {\mathbb {E}} \left[ V \left(S _ {\pi} ^ {i} \cup \{i \}\right) - V \left(S _ {\pi} ^ {i}\right) \right], \tag {4}
$$
where  $S_{\pi}^{i}$  is the set of data from FL participants joining before  $i$  in the  $\pi$ -th permutation of the sequence of FL participants. The calculated Shapley values are clipped to be non-negative to follow the non-negative assumption as in [3].
Shapley value is a widely used approach for evaluating participant contributions in federated learning due to its desirable properties. However, despite the proposal of various acceleration techniques
such as Monte Carlo sampling, gradient-based model reconstruction, and truncation, the computation of the Shapley value remains prohibitively slow due to the following two key challenges Multi-Round Training: multiple communication rounds are required for model convergence, and Dataset-dependent Evaluation: the evaluation of model performance depends on the size of the validation set on the server. Thus, we propose the Single-round Participant Amalgamation for Contribution Evaluation (SPACE) approach, which analyzes participant contributions using only one communication round. The key idea behind SPACE is to compare the distribution between the local dataset and the validation dataset directly via prototype embeddings of the local dataset and the validation dataset [37, 51, 52, 68]. To establish robust prototype embeddings, we introduce Federated Knowledge Amalgamation to distill feature information from all locally optimized models to the server model simultaneously, which enables the server model to learn information in a single communication round. Once the prototypes are obtained, we evaluate the performance of coalition  $S$  by comparing the similarities between the prototypes formed by clients in coalition  $\mathcal{P}_S$  and prototypes formed by validation set  $\mathcal{P}_{\mathcal{V}}$ . Using prototypes eliminates the need to iterate over all samples in the validation set, resulting in faster performance evaluation. Figure 1 shows the architecture of SPACE.
4.1 Federated Knowledge Amalgamation
As previously mentioned, the SPACE framework utilizes prototypes to assess participant contributions. Therefore, employing a semantically meaningful deep model is essential to ensure the extraction of expressive features from data samples. One straightforward approach is to utilize the converged model achieved through federated training, which encompasses training with all the local data. However, the convergence of federated learning algorithms like FedAvg necessitates numerous rounds of local-global communication and local training. Consequently, the evaluation of contributions can only be conducted after extensive training, resulting in excessively high costs and impeding the utilization of participant contributions in applications like client selection.
In contrast to conventional knowledge distillation tasks, which typically involve a single student model and a single teacher model, knowledge amalgamation techniques are employed to entail one student model and multiple teacher models. Furthermore, in federated learning, the client models can be considered teacher models that acquire knowledge from their local datasets. Consequently, the application of knowledge amalgamation facilitates the construction of an expressive embedding space for federated learning.
SPACE first distributes the global model  $\theta$  to each client and each client implements empirical risk minimization to obtain a locally optimized model  $\theta_{i}$ . The knowledge amalgamation is then applied to transfer the knowledge to the initial global model  $\theta$  with the locally optimized models serving as teacher models and the server validation dataset for distillation. To simultaneously distill knowledge from all the teacher models, the features of both the student and teacher models must be projected to common feature space for the feature alignment.
Therefore, we use feature projector modules (FPM) to project the intermediate feature of models to the common feature space. Since the data distributions of clients are different, the projections from the original feature space to the common feature space are also different. Therefore, each model, including the student model and teacher models, would be assigned an individual learnable FPM for projection. After projection, the student model then starts to learn information from teacher models by minimizing the feature distillation loss  $L_{feat}$ , which is the L1 distance between the projected student features  $\tilde{\mathbf{f}}_s$  and the projected teacher features  $\tilde{\mathbf{f}}_{t_i}$ . However, due to the differences in local data distribution, the teacher models are not equally important, and the loss should be weighted differently depending on the sample used for knowledge amalgamation. We define the weighting of each teacher model  $w_i$  by its confidence in predicting the class of data sample. Specifically, given a data sample  $d$  with its class as  $j$  for distillation, the weighting of each teacher model is the softmax of the  $j$ -th logit of its prediction  $\mathcal{F}(\theta_i,d)$ , which can be written as follows:
$$
L _ {f e a t} = \sum_ {i} w _ {i} \left| \tilde {\mathbf {f}} _ {s} - \tilde {\mathbf {f}} _ {t _ {i}} \right|, w _ {i} = \frac {e ^ {\mathcal {F} \left(\theta_ {i} , d\right) _ {j}}}{\sum_ {i} e ^ {\mathcal {F} \left(\theta_ {i} , d\right) _ {j}}} \tag {5}
$$
To prevent the FPM from projecting all the features to a constant vector, which is a trivial optimal solution for the feature distillation loss, reconstruction loss is proposed to ensure the projected feature contains the knowledge in the original feature space. For each teacher model, we apply an inverse feature projector module (iFPM) to reconstruct features in the original feature space, which
projects  $\tilde{\mathbf{f}}_{t_i}$  to the reconstructed feature  $\hat{\mathbf{f}}_{t_i}$ . The reconstruction loss is defined as follows:
$$
L _ {r e c} = \sum_ {i} \left| \mathbf {f} _ {t _ {i}} - \hat {\mathbf {f}} _ {t _ {i}} \right| \tag {6}
$$
where  $\mathbf{f}_{t_i}$  is the original feature of  $i$ -th client before projection. Aside from feature distillation, we also use prediction distillation to enhance the overall amalgamation performance. For prediction distillation, the student model only learns from the teacher that predicts the highest confidence in the correct prediction. The prediction distillation loss  $L_{KL}$  is defined as follow:
$$
L _ {K L} = K L \left(\mathcal {F} \left(\theta_ {S}, d\right), \mathcal {F} \left(\theta_ {\text {b e s t}}, d\right)\right), \theta_ {\text {b e s t}} = \arg \max  _ {\theta_ {i}} \left(\mathcal {F} \left(\theta_ {i}, d\right) _ {j}\right) \tag {7}
$$
where  $KL(\cdot)$  denotes the KL divergence loss. The overall amalgamation loss  $L_{amlg}$  becomes,  $L_{amlg} = \lambda_{feat}L_{feat} + \lambda_{rec}L_{rec} + \lambda_{KL}L_{KL}$ , where  $\lambda$  are used for balancing the loss terms. After knowledge amalgamation, the server model  $\theta^{*}$  is then distributed to each client for extracting features and building prototypes of each category.
While the theoretical complexity of knowledge amalgamation is  $O(n)$ , practical considerations arise as the number of clients  $n$  increases, leading to an increase in the number of FPMs that require training. This poses a challenge due to the limited memory capacity of GPUs, as training all FPMs simultaneously may not be feasible. To address this issue, we suggest a hierarchical amalgamation framework that alleviates the high GPU memory requirement. By allocating a GPU memory budget, we can amalgamate groups of  $G$  teacher models instead of all  $n$  teacher models at once. Although this modification increases the complexity of knowledge amalgamation to  $O(n\log n)$ , it enables the implementation of knowledge amalgamation on any GPU device capable of simultaneously amalgamating at least two teacher models.
4.2 Prototype-based Model Evaluation
The concept of utilizing prototypes in classification models has its roots in the Prototypical Networks introduced by Snell et al. [47], which were initially proposed for addressing few-shot classification tasks. In this framework, the classification of an embedded query point is achieved by identifying the nearest class prototype. The underlying assumption is that an embedding space exists where data points tend to cluster around a single prototype representation for each class. By leveraging prototypes, it becomes unnecessary to compute the distances between all samples in the query set and the support set, as required in Sung et al. [50]. The prototypes capture the distribution of samples in the support set, simplifying the classification process.