CoAst: Validation-Free Contribution Assessment for Federated Learning based on Cross-Round Valuation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 06:29:16

Abstract: In the federated learning (FL) process, since the data held by each participant is different, 
it is necessary to figure out which participant has a higher contribution to the model performance.
Effective contribution assessment can help motivate data owners to participate in the FL training.
The research work in this field can be divided into two directions based on whether a validation dataset is required.
Validation-based methods need to use representative validation data to measure the model accuracy, which is difficult to obtain in practical FL scenarios.
Existing validation-free methods assess the contribution based on the parameters and gradients of local models and the global model in a single training round, which is easily compromised by the stochasticity of DL training. 
In this work, we propose CoAst, a practical method to assess the FL participants' contribution without access to any validation data.
The core idea of CoAst involves two aspects: one is to only count the most important part of model parameters through a weights quantization, and the other is a cross-round valuation based on the similarity between the current local parameters and the global parameter updates in several subsequent communication rounds.
Extensive experiments show that the assessment reliability of CoAst is comparable to existing validation-based methods and outperforms existing validation-free methods.
We believe that CoAst will inspire the community to study a new FL paradigm with an inherent contribution assessment.

Main paper: 
 introduction: 
 With the development of deep learning (DL), the concept that "Data is the new oil" has gained more and more consensus among people [5] . The emerging remarkable capabilities demonstrated by large language models [18] further bring attention to the enormous value of collaborating on a large amount of data. To train DL models on data owned by different parties, collaborative learning techniques, represented by federated learning (FL) [7, 23] , have been extensively studied. However, due to disparities in the data held by different participants, including variations in data quality and quantity, each participant's contribution to the performance of the FL model differs a lot. How to accurately evaluate the contribution of each participant is crucial for the fair distribution of rewards. This process benefits the promotion of data quality and creates incentives for data sharing [17] .There have been a line of research in the community on contribution assessment of participants in FL, which can be mainly divided into two categories, i.e., validation-based methods [2, 6, 19] and validation-free methods [10, 14, 21, 22] . The effectiveness of validation-based methods heavily relies on a representative validation dataset, which is used to evaluate the model performance. However, in real-world FL scenarios, obtaining the representative validation dataset that covers the distribution of all clients' data can be challenging. To overcome the limitations imposed by the validation dataset, validation-free methods are proposed to assess the contribution based on the statistical characteristics of model parameters. They estimate the parameters' correlation, information gain, and mutual information among local models (produced by clients) and the global model (aggregated by the server) to answer whose contribution is higher. However, the existing validation-free efforts, detailed in the next section, only consider the models' parameters (or gradient) in a single training round. We refer to a training round as the process of the client updating and uploading the local model and getting the global model from the server after aggregation. Due to the difficulties, such as the stochastic nature of gradient descent and the uninterpretable nature of the DL model, the accuracy of these assessment methods can sometimes be seriously compromised. Performing validation-free assessment faces two challenges: 1) the training process of DL models does not update parameters in a linear way, so only comparing the parameters of the local model in each round with the parameters of the final model will ignore the clients' contribution reflected in the iterative process of parameter updating. 2) due to the parameter redundancy of DL models, not all parameters in the local model reflect the client's contribution to the performance improvement of the global model. Unimportant local parameters may comprise the assessment's effectiveness.In this work, we propose CoAst, a validation-free FL contribution assessment method, performing client valuation in a cross-round way. The overview of CoAst is depicted in Figure 1 . The core idea of the cross-round valuation is to evaluate a local model's contribution in a certain round (e.g., 𝑡) leveraging the parameter updates of global model in several subsequent rounds (e.g., rounds {𝑡 + 1, . . . , 𝑡 + 𝑘 }).To cope with the interference of unimportant parameters, CoAst shares a similar idea with the ternary weights quantization [9, 20] . CoAst spotlights the most important parameters by neglecting the parameters with small updates and keeping only the sign of the parameters with a large update. It is worth noting that our assessment goal is to accurately and effectively measure each client's contribution to the FL model, rather than improve the FL model performace. As a result, CoAst neither affects the local training procedure nor changes the global models. Experimental results show that the assessment accuracy of CoAst is comparable to the validation-based methods and outperforms the existing validation-free methods. We have submitted the source code in supplementary materials and will open-source the CoAst once the paper is accepted. We summarize the contribution as follows.• We design a new validation-free contribution assessment method, CoAst, which can answer whose contribution is greater in a practical FL scenario without a validation dataset. • We propose a cross-round assessment mechanism to consider the effect of the intermediate local models on the final trained model, and we utilize the ternary weight quantization to capture the parameters that contribute the most. • Experimental results show that CoAst outperforms the SOTA validation-free methods in assessment effectiveness and achieves comparable -performance to validation-based methods.The rest of the paper is organized as follows. In Section 2, we review the related works. In Section 3, we formulize the targeted scenario and problem. Section 4 presents the detailed design of our proposed CoAst. Then, we describe the experimental settings in Section 5 and provide the evaluation results in Section 6. The limitation of this work is discussed in Section 7. Finally, we conclude this paper in Section 8. 
related works: 
 Validation-based methods. The validation-based methods use a validation dataset to assess a client's contribution by evaluating its impact on the performance of the aggregated model. The leaveone-out is the most natural way to assess the value. It assesses the data value of one contributor by calculating the model performance change when the contributor is removed from the set of contributors. However, the leave-one-out is unfair to multiple similar and mutually substitutable contributors. Ruoxi et al. [6] use the Shapley value to assess data value in the FL scenario. They compute the marginal increase of the average accuracy of the model due to the addition of one data contributor. Guan et al. [15] extend the application scenarios of Shapley value-based solutions to FL scenarios. Zhenan et al. [2] apply the Shapley value-based solutions to vertical federated learning and improve the efficiency through approximation. Zhaoxuan et al. [19] allow for efficient data valuation without long-term model training. They assess the contribution through a domain-aware generalization bound, which is derived from the neural tangent kernel (NTK) theory. There are also research efforts to improve the system performance [11, 13, 16] and to analyze the fairness [25] . When the server uses this line of work to assess the clients' contribution, it requires a representative dataset which covers the distribution among all clients' data. However, in real-world FL scenarios, obtaining such a representative validation dataset is infeasible.Validation-free methods. The validation-free methods use statistics of training data or the correlation among local and global parameters to value the clients. These works usually have some specific assumptions on the distribution of gradients, model parameters, or local training data. Therefore, they may face performance degradation in the real world when their assumptions are not satisfied. Xinyi et al. [22] propose a volume measure on the replication robustness, which assesses the contribution based on the diversity of training data. However, work [19] shows that this idea not only suffers from exploding volumes in high-dimensional inputs but also entirely ignores the useful information in the validation dataset. Rachael et al. [14] measure the data value based on the information gains of the model parameters. They hold that the contributors with the highest value can reduce the uncertainty of the model parameters. However, when the distribution of data is complex, the accuracy of the information gains is biased. Xinyi et al. [21] use the gradient similarity to measure the data value of the contributors' combination by comparing the data of one combination of the contributors with the gradient similarity of the global FL model trained by all contributors. However, due to the randomness of the stochastic gradient descent and gradient pruning, the value assessed in some rounds may not accurately reflect the true value of the data, or even the value of high-value data is negative. Hongtao et al. [10] propose a test data-free data value evaluation based on the pairwise correlation among the models based on the statistical characteristics of the models. They assume that the parameters trained by different contributors share the same distribution, which may not be satisfied when the data is imbalanced and non-independent, and identically distributed. 
problem overview 3.1 targeted scenario: 
 We assume all participants, including clients and the server, are honest and follow the agreed-on training protocol of FL. Due to differences in training data quality and quantity among clients, each client's contribution to overall model performance varies. The server can access the model parameters uploaded by each client, but it lacks a representative validation dataset. Each client delegates the server to evaluate the contribution of each client without offering validation data. Without loss of generality, we assume that each client is involved in all training rounds. The training data of each client is prepared before the FL training, and they will not add new data during the training process.Consider an FL training procedure with one server and 𝑁 clients.The training procedure consists of 𝑀 training rounds. Recall that, in one training round, a client uploads its local parameters to the server once and receives the corresponding aggregated parameters (a.k.a. global parameters). After 𝑀 training rounds, the contribution of all clients is determined. We denote the ground-truth contribution of all clients as 𝑃 = {𝑝 1 , 𝑝 2 , . . . , 𝑝 𝑁 }, where 𝑝 𝑖 is the contribution of client 𝑖. The ranking of all clients' contribution is denoted as 𝑅 = {𝑟 1 , 𝑟 2 , . . . , 𝑟 𝑁 }, andEQUATIONwhere | • | returns the number of elements in a set. Our goal is to design a function 𝐿, which can measure how much each client improves the performance of the global model. That is,EQUATIONwhere Θwhere 𝑑 (•, •) is the distance measurement function.Due to the multi-round property of FL algorithm, the contribution score of each client in round 𝑡 (denoted as P𝑡 ) can be naturally represented as:EQUATIONwhere Θ 𝑡 is the parameters of the global model aggregated in round 𝑡. So, we can reformulate the Equation 2toEQUATION 
coast design 4.1 design overview: 
 We propose two key techniques to address the challenges introduced in the introduction section. First, we design a cross-round valuation mechanism. Due to stochastic gradient descent, gradient pruning, and parameter regularization, the parameter updates in a certain round (e.g., round 𝑡) may not accurately reflect the true value of the client, and even high-value clients may be assigned negative contributions. Fortunately, the FL training process minimizes the optimization objective and improves the model's accuracy, which means that the global parameter updates have a positive contribution in most training rounds. It allows us to value the client 𝑖 in round 𝑡 with global parameters of subsequent several rounds. Second, we borrow ideas from efforts on model compression and quantization to filter out those unimportant parameters. Binary weight quantization [4] is proposed for the efficiency of computation and storage and demonstrates that retaining only the sign of parameters during model updates can still reach acceptable accuracy. Then, the ternary weight quantization [9] sets unimportant parameter updates to zero on top of binary weight quantization and reaches comparable accuracy to full precision training. It shows that by setting a threshold, parameter updates that contribute minimally to the model performance can be effectively filtered out. Thus, we apply the idea of ternary weight quantization to the global model aggregation and then value the client's contribution with the remaining important parameters.We demonstrate the workflow of CoAst in Figure 2 . The contribution assessment is transparent to clients, and there is nothing to // The zero_init function returns multiple zero tensors with the same shapes as inputs After aggregation, we use cross-round valuation to measure the contributions of local models. In the next part, we will detail these two key designs.3 Δ 1 , Δ 2 , . . . , Δ 𝑁 = Θ 𝑡 -1 -𝜃 𝑡 1 , Θ 𝑡 -1 -𝜃 𝑡 2 , . . . , Θ 𝑡 -1 -θ𝑡 𝑖 [ 𝑗] [ℎ] = Θ 𝑡 -1 [ 𝑗] [ℎ] + 𝛼 • sgn(Δ 𝑖 [ 𝑗] [ℎ]) //Without loss of generality, we use the training process of round 𝑡 as an example to detail the algorithm design. 𝑁 clients first locally train the local models of round 𝑡 based on the global parameters of round 𝑡 -1, denoted as Θ 𝑡 -1 . Then, all clients upload local parameters of round 𝑡 to the server, which are denoted as {𝜃 𝑡 𝑖 } 𝑖 ∈ {1,...,𝑁 } . In the following procedure, clients do nothing but wait for the aggregated global parameters of round 𝑡, denoted as Θ 𝑡 , to continue their local training in the next round.Once all local parameters {𝜃 𝑡 𝑖 } 𝑖 ∈ {1,...,𝑁 } are received, CoAst calculates the local updates of round 𝑡, denoted as {Δ 𝑡 𝑖 } 𝑖 ∈ {1,...,𝑁 } , whereEQUATIONThen CoAst performs parameter pruning according to their importance, similar to the idea of model quantization such as binarization and ternarization. Considering the structural and functional differences between layers of DL models, CoAst quantifies the parameter importance in a layer-wise manner. Here we denote the parameter updates in each layer as Δ 𝑖 := [𝛿 1 , 𝛿 2 , . . . , 𝛿 𝑙 ] where 𝑙 refers to the number of layers. Take the 𝑗-th layer as an example. CoAst first calculates the 𝑟 -th percentile of |𝛿 𝑗 |, denoted as 𝛿 𝑟 𝑗 , where 𝑟 is a hyperparameter controlling the pruning rate. Then, CoAst prunes the parameters by clipping the parameter updates. For each element of 𝛿 𝑗 , denoted as 𝑢, it is clipped as:EQUATION)It means we regard the elements of 𝛿 𝑗 whose absolute values are greater than 𝛿 𝑟 𝑗 as important and only keep their signs, while the remaining elements are pruned to 0. Given the clipped parameter updates Δ𝑡 𝑖 , the pruned local parameters θ𝑡 𝑖 can be calculated as θ𝑡EQUATION)where 𝛼 is a hyperparameter for normalization. We report the whole parameter pruning procedure in Algorithm 1.In each round, the contribution of the model is valued by the global model aggregated of the next 𝑘 rounds, and it also values the local models of the last 𝑘 rounds. Recall that the parameters of the global model are the average of the pruned local parameters (Equation 8), which are calculated by adding the sign of the parameter update. Therefore, the global parameters Θ 𝑡 is calculated byEQUATIONTherefore, in round 𝑡, the parameter update of the global model denoted as 𝑈 𝑡 := Θ 𝑡 -Θ 𝑡 -1 , is proportional to the sum of the pruned local updates, i.e.,EQUATIONRecall that the value of element ũ ∈ Δ ′ 𝑖 belongs to {-1, 0, 1}. That is, the 𝑈 𝑡 (Equation 10) is the normalized voting result, which indicates, for each element 𝑏 ∈ Θ 𝑡 -1 , how many clients believe that its value should be increased by 𝛼 𝑁 , and how many clients believe that it should be decreased by 𝛼 𝑁 . After 𝑘 rounds following the 𝑡-th round, we obtain the parameters Θ 𝑡 +𝑘 , which can be calculated byEQUATIONIn the following part, we denote 𝑡 +𝑘 𝑒=𝑡 +1 𝑈 𝑒 as 𝑈 (𝑡,𝑘 ) for simplification. Since the global parameters are obtained by averaging all local parameters, we can regard the parameters of any client as a correction to the sum of parameters of the other 𝑁 -1 clients. Without loss of generality, for any client 𝑖, we reformulate Equation 11toEQUATION)We assume that the model's performance is improved after the 𝑘 rounds. That is, among all clients, if a client's local model of round 𝑡, i.e., 𝜃 𝑡 𝑖 , during the parameter aggregation process is more similar to the subsequent global parameter updates, i.e., 𝑈 (𝑡,𝑘 ) , then the contribution of this client in this training round is greater. We demonstrate this idea in Figure 3 . Clients will recalculate the local models based on the last global model, so the choice of 𝑘 should not be too large. Otherwise, the update direction represented by 𝑈 (𝑡,𝑘 ) may not capture the details of stochastic gradient updates and thus cannot indicate the contribution in one training round.Therefore, the contribution of client 𝑖 can be measured by the similarity between 𝜃 𝑡 𝑖 and 𝑈 (𝑡,𝑘 ) . Here, we use Signed Cosine similarity to measure the similarity because the 𝜃 𝑡 𝑖 and 𝑈 (𝑡,𝑘 ) are local parameters and global updates, respectively. Note that Signed Cosine similarity is sensitive to the sign information of vectors and can better reflect the directional relationship between vectors. Due to the parameter pruning, the model updates can be considered as the sign of each parameter's update (Equation 10). That is, in our design, the sign of these updates is more important than their magnitude.Note that 𝜃 and 𝑈 (𝑡,𝑘 ) share the same shape, and we assume that they can be indexed through ℎ. CoAst calculates the client's contribution in round 𝑡 byEQUATIONwhere sgn(•) is the function to indicate the sign of the value. 
implementation 5.1 dataset settings: 
 We evaluate the CoAst's performance on three datasets, i.e., CIFAR-10 [8], CIFAR-100 [8] , and STL-10 [1] . We randomly partition the training dataset among each participant. We assume that by randomly and evenly partitioning these three datasets according to the number of clients, several datasets with the same data valuation can be obtained. Therefore, if a group of clients uses these partitioned datasets for training, the contribution of these clients is the same. We have set up four scenarios to mimic the contribution differences caused by data quality and quantity differences. 𝑁 in the following settings denotes the number of clients.Assuming that randomly partitioned datasets share a similar distribution, the more samples in the training dataset, the higher the contribution to the model accuracy. In this scenario, we prepare datasets with different contributions by randomly assigning different numbers of samples to each client. Let the number of clients be 𝑁 and the size of the In this scenario, we mimic the data of different quality by adjusting the resolution of the training data through Gaussian blur. Different degrees of Gaussian blur can be achieved by setting kernels of different sizes and variances. We first randomly and evenly partition the dataset according to the number of clients. Then, we use different degrees of Gaussian blur to preprocess the training data. Let the sequence of kernel sizes and standard deviation be 𝑠 𝑖 and 𝜎 𝑖 , where 𝑖 ∈ {1, . . . , 𝑁 }. We select a linearly decreasing sequence of kernel sizes and standard deviation, i.e., 𝑠 𝑖 = 2 • 𝑖 + 1, 𝜎 𝑖 = 0.4 • 𝑖 + 1. We report some data samples in Figure 4b .Masking. In this scenario, we prepare the dataset with different quality by adding a mask on training data. The content covered by the mask is set to 0. We first randomly and evenly partition the dataset according to the number of clients. Then, for each client, we randomly mask a part of the image. The area of the mask covers 𝑟 𝑖 % of the image for client 𝑖, and its position is randomly generated. The 𝑟 𝑖 is a random number between 𝑙 𝑖 and 𝑢 𝑖 , where 𝑙 𝑖 = 0.5 • 𝑖 𝑁 , 𝑢 𝑖 = 0.75 • 𝑖 𝑁 . We report some data samples in Figure 4c .We perform experiments with Pytorch on a server with two A100 (80G) GPU cards. We use three model architectures, i.e., TinyRes-Net, ConvNet [24] , and ResNet-4. The TinyResNet consists of a convolution layer, whose weight shape is 3 × 7 × 7 × 64, and a ResBlock [3] , whose kernel size is 3 × 3 and output channel is 64. The ResNet-4 consists of 4 ResBlocks, whose kernel size is 3 × 3 and output channel is 64, 128, 128, and 128. In our setting, we use 1 central server and 5 clients, (i.e., 𝑁 = 5). We initialize the learning rate to 0.01 and gradually decrease it as the training progresses. We use the FedAvg method to aggregate the local models.We use three baseline methods. The validation-based method [15] , denoted as baseline 1, measures the accuracy of the aggregated models w/ or w/o one local model with the validation dataset to calculate the contribution. Although baseline 1 is computationally time-consuming, it has the highest accuracy among Shapley valuebased solutions by exhaustively considering all possible cases. We implement two SOTA validation-free methods, i.e., Fed-PCA [10] and CGSV [21] , as the baseline 2 and baseline 3.To measure the accuracy of the contribution assessment, we use the Spearman correlation coefficient [12] as the distance measurement function 𝑑 in Equation 3. The Spearman correlation coefficient is good at measuring the degree to which the ranks of the two variables are associated with each other. And it also is used in related works [17, 21] . Formally, 𝑅 and R are two sequences of length 𝑛, which denote the ground-truth order and predicted order of the clients' contribution, i.e., 𝑅 = [𝑜 1 , 𝑜 2 , . . . , 𝑜 𝑛 ], R = [ ô1 , ô2 , . . . , ô𝑛 ]. We calculate the Spearman correlation coefficient (𝜌) through the:EQUATIONThe 𝜌 ranges from -1 to 1, where -1 indicates a perfectly negative correlation, i.e., sequences 𝑅 and R are in reverse order, 0 indicates no correlation, i.e., random guessing, and 1 indicates a perfectly positive correlation. 
evaluation: 
 We measure the performance of our CoAst in four configurations with three datasets and three model architectures. Configuration 1 is CIFAR-10 and TinyResNet; Configuration 2 is CIFAR-100 and TinyResNet; Configuration 3 is STL-10 and ResNet-4; Configuration 4 is CIFAR-100 and ConvNet.In the CoAst's experiments, for hyperparameters in Algorithm 1, we set 𝑟 to 10, 𝛼 to 0.02, and 𝑁 to 5. We set the 𝑘 in Equation 13to 2. We report the overall performance in Table 1 . SV, PCA, and CGSV represent baseline 1, baseline 2, and baseline 3, respectively. The average accuracy of baseline 1 and CoAst are 0.855 and 0.9, which means that our CoAst is comparable to that of baseline 1, 1.00 1.00 0.30 0.30 1.00 1.00 0.60 0.30 1.00 1.00 0.00 -0.70 1.00 2 0.90 -0.10 1.00 1.00 1.00 0.10 0.60 1.00 1.00 0.10 -0.30 0.90 1.00 0.10 -1.00 1.00 3 0.70 0.00 1.00 1.00 0.90 0.30 0.30 1.00 1.00 0.70 0.40 0.20 0.10 -0.30 0.40 0.60 4 0.90 -0.10 1.00 1.00 1.00 -1.00 -0.10 1.00 1.00 -0.40 0.70 0.70 1.00 0.10 -0.30 1.00 A. 0.78 -0.30 1.00 1.00 0.98 -0.08 0.28 1.00 1.00 0.25 0.28 0.70 0.78 -0.03 -0.40 0.90 Table 1 : Overall performance of CoAst. SV [15] represents baseline 1, a validation-based method. Note that some efforts, in these years, optimize the system performance by estimating the Shapley value rather than improving the assessment accuracy (Section 2). Therefore, we use the classical SV [15] to compare accuracy. PCA [10] and CGSV [21] represent baseline 2 and baseline 3. Our CoAst (denoted as O). (C. is short for configuration, and A. is short for Average.) which is a validation-based method. CoAst even outperforms baseline 1 by 0.22, 0.02, and 0.12 in setting 1, setting 2, and setting 4. Our CoAst's performance outperforms the SOTA validation-free methods, i.e., Fed-PCA (baseline 2) and CGSV (baseline 3), in almost all cases. On average, our CoAst outperforms Fed-PCA by 0.94 and CGSV by 0.52 in all cases. The poor performance of Fed-PCA is because the model architecture used in our experiment is too complex to perform precise probability analysis. Experimental results demonstrate the CoAst's effectiveness and robustness in different cases.6.2.1 𝑘 in cross-round valuation. We explore how 𝑘 affects the performance of CoAst. We perform the experiment with different 𝑘 values, which means that different numbers of global updates are used to assess the local models' contribution. We report the results in Table 2 . In different experimental settings, CoAst with 𝑘 = 5 reaches the best performance, and the performance of CoAst with 𝑘 = 2 is comparable to that of CoAst with 𝑘 = 5. The small and large 𝑘 values, i.e., 𝑘 = 1 and 𝑘 = 10, have relatively poor performance. This is because a small 𝑘 value may still let the contribution assessment be affected by stochasticity, while a large value of 𝑘 may make CoAst lack attention to the stochastic gradient descent process. We recommend setting 𝑘 to 2 or 5 when using our CoAst in practical use.Recall that the parameter pruning consists of a parameter update clipping procedure (Line 9 in Algorithm 1) and a top r% update selection procedure (Line 7 in Algorithm 1). To measure the effect of parameter pruning, we perform the following contrast experiments with different settings, which are as follows. We report the results of these contrast experiments in Table 3 . By comparing the results of ours and Exp1, we can conclude that increasing the proportion of parameter selection leads to a slight decrease in performance, which is likely due to the noise introduced by the selected parameters. Our CoAst's average performance is 0.40625 and 0.25625 higher than Exp2 and Exp3, respectively, indicating that the design of parameter pruning effectively ensures the accuracy of contribution assessment.In the CoAst, we use a hyperparameter 𝛼 (Equation 8) to normalize the local parameter update in each round. Here we explore how the hyperparameter 𝛼 value affects the contribution assessment effectiveness. We perform three experiments with CIFAR-10 and TinyResNet. We set 𝑀 to 100, 𝑁 to 5, and 𝑟 to 10. In the first two experiments, we set the value of 𝛼 to 0.01 and 0.02, respectively. In the third experiment, we use an adaptive clipping strategy, where we set 𝛼 as the average value of the selected r% parameters (i.e., 10) of each layer. We report the results of these three experiments in Table 4 . By comparing the experimental results of the first two experiments, it can be seen that the choice of hyperparameters has little impact on the assessment performance. However, in the third experiment, the assessment performance decreases. This is because clipping the parameter update to different values interferes with the assessment process. In our design, we aim to quantize all parameter updates and assess the contribution based on the direction of the local parameters and global parameter updates.Here we explore how the number of clients, i.e., 𝑁 , affects the stability of CoAst. We experiment with CIFAR-100 and TinyResNet and set 𝑘 to 10 and 𝑟 to 10. We report the experimental results in Table 5 . When the number of clients increased from 5 to 10, the performance change of Fed-PCA averaged 0.4775, the performance change of CGSV averaged 0.38, and the accuracy change of our method averaged 0.07. When the number of clients is 5, our CoAst outperforms Fed-PCA and CGSV by an average of 1.025 and 0.5225, respectively. When the number of clients is 10, our CoAst outperforms Fed-PCA and CGSV by an average of 0.775 and 0.795, respectively. The experimental results fully demonstrate the stability of our method with respect to changes in the number of clients. 
discussion and limitation: 
 Recall that in the procedure of parameter aggregation, CoAst prunes local parameters according to their importance. Although weight quantization methods are often used in practical FL scenarios to reduce network bandwidth, they tend to incurring performance degradation of the model as well as slower convergence speed. Thus the convergence time of our CoAst is slightly longer than that of C.Setting 1: Quantity Setting 2: Noise Setting 3: Resolution Setting 4: Mask k=1 k=2 k=5 k=10 k=1 k=2 k=5 k=10 k=1 k=2 k=5 k=10 k=1 k=2 k=5 k=10 1 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.90 1.00 1.00 1.00 2 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.90 0.90 0.90 1.00 1.00 1.00 1.00 3 0.80 1.00 1.00 0.90 1.00 1.00 1.00 1.00 0.10 0.20 0.60 0.50 0.60 0.60 0.60 -0.60 4 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.70 0.70 0.70 0.70 0.90 1.00 1.00 1.00 A. 0.95 1.00 1.00 0.98 1.00 1.00 1.00 1.00 0.70 0.70 0.80 0.78 0.85 0.90 0.90 0.60 the typical method, i.e., without applying any quantization. We report the model performance trained through the typical method and ours in Table 6 . The model trained through the typical method converges after 100 rounds (i.e., 𝑀 = 100), while the model trained in our method requires 200 rounds (i.e., 𝑀 = 200) to converge. As can be seen, the performance of the model trained through CoAst outperforms that of the model trained through the typical method at the 100th round only in one case. When the model converges, the performance of the model trained through CoAst exceeds the model trained in the typical scenario in two cases. However, experiments in Section 6. the accuracy of the assessment since quantization mitigates the negative impact of redundant model parameters on the assessment. As future work, we will draw on research ideas in the direction of model quantization to improve the model performance in accuracy and convergence. 
conclusion: 
 In this work, we propose a validation-free contribution assessment, CoAst, for the FL scenario. Compared with existing efforts, it greatly improves the contribution assessment performance under different dataset settings by introducing two key designs: parameter pruning and cross-round valuation. Comprehensive evaluations showed that our CoAst outperforms existing methods on different dataset settings and different models. We believe that CoAst will inspire the data valuation design in other scenarios in the future. 
 
 Appendix:  


Decision: The reviewers think this paper interesting, intuitive, well organised, but still have some limitations. In the first round, this paper got 3BA anf 1BR. After the rebuttal, the BR is changed into BA, so this paper got 4BA. This score is OK for a poster.

Reviews: review: paper_summary: paper_summary: This paper explores contribution assessment problem in federated setting. The authors proposed a new validation-free contribution assessment that can measure contributions of each client without a validation dataset. Experiments show that the proposed method is effective compared to existing works.
strength_weakness: strength_weakness: 1. The paper studies the validation-free contribution assessment problem, which is an interesting issue in federated learning.

2. The paper is well organized.  The implementation details and experiments are clear and easy to read.
limitations: limitations: 1. The experiment was implemented with only five clients and the same level of heterogeneity. How does the model perform in real-world situations that are often encountered in federated learning, such as heterogeneity (through Dirichlet distribution [1]) and more clients?

2. It seems to me that the baseline SV outperforms the proposed method in experiment setting 3. It's better for authors to explain this result.

3. The number of clients in the experiment was increased to 10. It's very common for contemporary federated learning to have hundreds of clients or more. Can the proposed algorithm effectively scale up to handle such a scenario?

4. Why is baseline SV not compared in ablation experiments about client numbers?

[1] Ganghua Wang, Ali Payani, Myungjin Lee, and Ramana Kompella. Mitigating group bias in federated learning: Beyond local fairness. arXiv preprint arXiv:2305.09931, 2023.
paper_summary: paper_summary: This paper propose a novel contribution assess method in FL without the need of validation dataset.
strength_weakness: strength_weakness: 1. The pruning parameter methods are intuitive.
limitations: limitations: 1. The result tables are a little bit confusing. Why the metrics are close to 1? Does this metric make sense on the assessment of contribution?
2. Can the idea works on the Non-IID scenarios? 
3. The overall assessment algorithm framework is encouraged to be provided.
paper_summary: paper_summary: This paper studies the problem of validation-free contribution evaluation in federated learning. To achieve this, the authors utilize weight quantization and cross-round parameter comparison. Experiments are conducted on real-world datasets.
strength_weakness: strength_weakness: 1. The target problem is interesting, and has a significant impact on the field of federated learning.
2. The paper is well-organized and easy to follow.
limitations: limitations: 1. How to determine k in practical application?
2. How to eliminate the differences between the pruned model and the original model during evaluation?
paper_summary: paper_summary: This paper proposes a method to evaluate the user contribution in federated learning. The method requires no validation data. It measures the contribution by the similarity of model updates. Experiment results demonstrate the performance of the proposed method in 4 scenarios (data quantity, noise, resolution, masking), using the Shapley ratio as the ground truth.
strength_weakness: strength_weakness: 1.	The paper is well-written and easy to understand.
2.	The proposed method is reasonable, as the similarity with the global model update certainly reflects the contribution of the single client’s update.
3.	The claim is supported by the experiment results.
limitations: limitations: 1.	The proposed method is somewhat trivial, since it is only a similarity computation along with sparsification. 
2.	The ‘low-quality’ data used in the experiments are man-made according to some simple rules. It is obvious that those ‘tampered’ data will make the model update deviate the average. So I think it would be better if the authors could conduct some experiments on natural data, e.g., partitioning the dataset into high-quality data and low-quality data.
3.	It is also good to test the method on noisy-labeled data, i.e., some of the clients get a portion of mislabeled samples.
4.	It seems currently, the proposed method is just to compute the contribution scores, however, the model performance is not affected. It would be great to have a discussion about how to use those contribution scores.