A Real-time Contribution Measurement Method for Participants in Federated Learning
Authors: Bingjie Yan
Yize Zhou
Boyi Liu
Jun Wang
Yuhan Zhang
Li Liu
Xiaolan Nie
Zhiwei Fan
Zhixuan Liang
Venue:  (2021)
Keywords: Federated Learning
Contribution Evaluation
Multi-party Participation

Abstract: Federated learning is a framework for protecting distributed data privacy and has participated in commercial activities. However, there is a lack of a sufficiently reasonable contribution measurement mechanism to distribute the reward for each agent. In the commercial union, if there is no mechanism like this, every agent will get the same reward. This is unfair to agents that provide better data, so such a mechanism is needed. To address this issue, this work proposes a real-time contribution measurement method. Firstly, the method defines the impact of each agent. Furthermore, we comprehensively consider the current round and the previous round to obtain the contribution rate of each agent. To verify effectiveness of the proposed method, the work conducts pseudo-distributed training and an experiment on the Penn Treebank dataset. Comparing the Shapley Value in game theory, the comparative experiment result shows that the proposed method is more sensitive to both data quantity and data quality under the premise of maintaining real-time.

Under review as a conference paper at ICLR 2021
A REAL-TIME CONTRIBUTION MEASUREMENT
METHOD FOR PARTICIPANTS IN FEDERATED LEARN- ING
Anonymous authors Paper under double-bling brind review
ABSTRACT
Federated learning is a framework for protecting distributed data privacy and has participated in commercial activities. However, there is lack of sufficiently reasonable contribution measurement mechanism to distribute the reward for each agent. In the commercial union, if there is no mechanism like this, every agent will get the same reward. This unfair to agents that provide better data, so such mechanism is needed. To address this issue, this work proposes real-time contribution measurement method. Firstly, the method defines the impact of each agent. Furthermore, we comprehensively consider the current round and the pre- vious round to obtain the contribution rate of each agent. To verify effectiveness of the proposed method, the work conducts pseudo-distributed training and an experiment on the Penn Treebank dataset. Comparing the Shapley Value in game theory, the comparative experiment result shows that the proposed method more sensitive to both data quantity and data quality under the premise of maintaining real-time.
INTRODUCTION
There are lots of data generate, collect, and access every day by smart terminals. But cause of the privacy of these data, it is usually difficult to use them. Such as the language model to predict the next word even entire reply(Ion al. 2016).
The emergence federated learning breaks this data barrier. It can use agent computing power to conduct model training while maintaining data localization and privacy protection, and obtain an excellent global model.
But in commercial federation, each agent should get corresponding rewards based on its contri- bution to the model, not the same rewards. There are many methods for contribution measurement. Such WWan et al.] (2019) measured the contribution of each group features in vertical federated learning, and Zhhn et al. (2020) proposed an incentive mechanism to make each agent willing to contribute better data. But most of them need consume large computing resources and many methods are calculated offline.
In order address this problem, this paper proposes method for obtaining the contribution of each agent in real time with a small amount of calculation in horizontal federated learning.
Our contributions in this paper are follows:
We propose method to measure agents' contributions and compare this method with Shap- Icy Value.
The method we propose sensitive to data volume and data quality, and can be used for mutual comparison between agents.
In the training process, the contribution to each agent can be obtained real time, with low computational complexity.

---

Under review as a conference paper at ICLR 2021
Felersee Leaning (Comtsbuton Maaummeent
ponneers
O
Apenh
Apert
Figure 1: Experimental results with randomize the word sequence.
2
RELATED WORK
In this section, we will introduce the current research situation and application of federated learning, as well as the existing problems of reward distribution in the commercialization of federated learning framework, and introduce the necessity and importance of our work.
2.1 FEDERATED LEARNING
Distribute the training data on each mobile device to maintain the localization of the data, instead of transmitting the data to the central server, updating the model locally, and uploading the update results to the server. While maintaining data localization and privacy. it can aggregate the data of each agent. The central server collects agent data and uses FedSGD, FedAVG(Brredan McMahan iet al. (0166) and other algorithms to maintain the central model in combination with the different optimizer(Felbab al. 22199) and sends the updated model to each agent. During the transmission process, methods such as homomorphic encryption are used to protect the security of data transmis- sion and maintain the continuous iterative update of the model. This method federated learning.
At present, for different datasets, federated learning framework can be classified into horizontal federated learning, vertical federated learning, and federated transfer learning( Yang et al., 2019). Horizontal federated learning is suitable for situations where the data provided by the agents has more of the same characteristics. (McMahan et al. 2017). Google proposed solution to update the horizontal federated learning model in Android phones. vertical federated learning is suitable for situations where there less feature overlap but more user id overlap. Hardy et al. (2017) proposed vertical federated learning method for training logistic model. [Cenn et al. (2020) proposed the FedHealth method, which uses federated learning to aggregate data and uses transfer learning to obtain a personalized model.
Under the framework of federated learning, there are many different directions of research. Aono et al. 2017) the data transmitted by the homomorphic encryption agents and the central for model training, which further strengthens the privacy of the agents' data. In (Lu et al.) 2019 conjunction with the blockchain gradient information from being maliciously tampered with. Konečný et al. (2016) research on reducing the consumption of communication resources in federated learning.
Kim al. 2019), data verification carried
prevent the
2.2 APPLICATION AND COMMERCIALIZATTON OF FEDERATED LEARNING
Since federated learning was proposed, federated learning has been successfully applied to more and more scenarios. When considering data privacy issues, many companies will choose to use federated learning protect data privacy to achieve cooperation. Such WeBank has successfully used federated learning in bank federations for credit evaluation and other financial aspect. In (Ren et al. 2019), federated learning is applied to dynamic IoT system Lu et al. (2019): Kim et al. (019
2

---

Under review as a conference paper at ICLR 2021
blockchain for data verification alternative to the commercialization of federated learn- ing. [aoo et al. (2019) achieved model commercialization by providing a healthy marketplace for collaborative-tramiin models.
In the commercialization process of federated learning, the method of using this decentralized train- ing model provides privacy, security, supervision and economic benefits, but also brings many fair- ness and systemic challenges.
2.3 CONTRIBUTION MEASUREMENT AND INCENTIVE MECHANISM IN FEDERATED LEARNING
Wang al. (2019) proposed grouped instance influence measure contribution of agents horizon- tal federated learning and grouped feature importance in vertical federated learning which equitably and accurately realized the measurement of contributions of all agents. But this method cannot take into account the amount of data.
Geyer et al. 2017) proposed federated learning optimization algorithm based on differential pri- vacy protection of the agents that can hide the contribution of the agents. In order to balance the contribution of the soft training model and ensure the collaborative convergence, Xu al. (2019) proposed the corresponding parameter aggregation scheme. Kang et al. (2019) used the weighted subjective logic model to design a reputation- basedd worker selection plan to achieve reliable feder- ated learning. In (Zhan et al. (2020), designed incentive mechanism based on deep reinforcement learning to determine the optimal pricing strategy for parameter servers and the optimal training strategy for agents.
Our work focus on real-time contribution measurement mechanism for each agent in vertical feder- ated learning. Current research has problems such as poor real-time performance and high resource consumption by contribution measurement methods. The method proposed in this paper can obtain real-time contribution measurement each agent in the process of federated learning. At the same time, it verifies that the method proposed in this paper has high sensitivity on data quantity and data quality.
METHODOLOGY
In order solve the above issues, we propose a corresponding method which does need obtain agents data and data scale, but can measure agents contribution. This method of measuring agents contributions is suitable for evaluating agents real time.
3.1 FEDERATED LEARNING
In this section, we will introduce the basic framework of federated learning and how parameters are updated.
Federated learning distributed learning method in which the server maintains an central model and distributes it to individual agents. To solve privacy problems, the server does not need to obtain data the agents, so the computing power of the agents used to learn at the agents local environment.
The server will set up a fraction C, select the agents proportionally for the server's central model update. Then the selected agents' updated model parameters are uploaded to the server for parameter updating of the server model. is then distributed to the each agent to improve the model of the agents.
In this way, it continue to improve the central model of the server and the local model the agents. Under the premise of ensuring the correctness and privacy of the agents, using this method make full use of the computing power of the agents and large amount of private data for learning, and at the same time maintain excellent global model.

---

Under review as a conference paper at ICLR 2021
3.2 ATTENTION AGGREGATION
In this section, we will introduce a FedAtt algorithm proposed by Shaoxiong Ji, Shirui Pan, Guodong Long Ji et al. (019), which is outperformed to the common FedAvg and FedSGD algorithm. The proposed attention mechanism in Federated Learning is used after the agents transmitted the parameters the server, the server uses Formula equation to calculate the value of attention a of each layer parameter that should be allocated to the agent, and multiplies each layer parameter by the corresponding attention update the central model. The server integrates the update parameters passed by the agents according to theirs own parameter information, and finally completes an update of the central model through the communication between the server and the agents.
0% = softmax($1)
Σ c
The s4 is the norm difference from the central model. We use w' represent layer parameters of the server and w for (th layer parameters of the agent model k. So the definition of s follows
$ = I| wxllp
So for m agents that are selected to update the central model, the method of updating becomes
W++1 We (w W, Σ
an(w: w
k=
To protect the agents' data privacy, you can add the randomized mechanism before the agent passes parameters to the server. Randomly generate random vector that obeys the standard distribution NO0, d²), multiply the corresponding weight B, the results of the final update parameters as shown in Formula equation
m
W++1 <- Wt
Σ an (w. - w +00(0,
(4)
The implementation process of the whole algorithm is shown as Algorithm
I
Algorithm Attentive Federated Optimization
1: is the ordinal of neural network layers; c is the stepsize of server optimization
2: Input: server parameters Wi at f, agents parameters W/+1 w² We41:"" "" w at
3: Output: server parameters W/+1 at +-t++1
4: Initialize attention a {a1,02, 02, (0m)
5: for each layer / in model do
6: for each agents E S; from 1 to m do
7: s = =lw w/llp
8: end for
et Σ, of
9: a/ softmarrss)
end for
11: ag {a af .ake
12: Wr+1 +- Wi R= (wl(wr w/ - +3N((,00))
>
3.3 AGENTS CONTRIBUTIONS
In this section, the method for measuring agents contribution proposed in this paper will be intro- duced in detail.
Before calculating each agent contribution, make the following assumptions: think each terminal does not tamper with the updated gradient itself during the previous process of passing and updating parameters between the server and the agents.
4

---

Under review as a conference paper at ICLR 2021
The agents contribution to the server should be calculated after each layer of agents' parameter attention. When the server has calculated the attention of agent K, it can calculate the impact of this agent K the server model parameters in the T update. We define imp, follows:
imp; cora(1 w B0 o²)) + +y -mmmpp-11
(5)
where E (0, 1) is forgetting coefficient, because large variations in the early stage of the training model, the model is not stable, and the impact of the previous endpoint should be reduced after multiple iterations. If the agent k is not in the selected M agents this time, think that the impact of the agent & round is imp, = impp_1 that is, only the number of rounds that the agent participates in the update calculated. Note that the server's attention to the agents the attention of each layer, so when calculating the impact it also calculates the impact of each layer by reweighting and averaging.
We will be each agents attention MinMaxScaler normalized limited range, then Softmax, to obtain the contribution of each agent. We use con to represent the contribution of agent time. The whole method of measuring agents contribution proposed in this paper is shown in Algorithm 2
Algorithm 2 Measure Agents Contributions
Input: / is the ordinal neural network layers; e the stepsize of server optimization, server parameters at time 1++1, 1, agents parameters w/41, wf+1,"" w at time 1,1, is forgetting coefficient, St is selected agents set.
2: Output: Agents contributions con, at t
3: for each agents from 1 to K do
4: if k E S, then
5: imp, = ca.(wi 1 + (,N((0,0)) + + imp.-1
6: else
7: impp, = impp_1
8: end if
9: end for
con, = Sottnaa(MinMaaSSSlee(mmm))))))))
4 EXPERIMENT
In this section, we will introduce the verification experiments performed for our proposed agents' contribution measurement method.
4.1 EXPERIMENTAL ENVIRONMENT AND CONFIGURATION
The system used in our verification experiment is Ubuntu 18.04 LTS, the backend used is the GPU version of Pytorch, with the NVIDIA GTX1660TI GPU acceleration for model calculation. The GRU-based agents model used the language processing model of RNN. For a detailed model de- scription, see the subsection under this Section.
4.2 DATASET
We perform experimental verification on public language datasets of Penn Treebank to verify the effectiveness of our proposed algorithm and its sensitivity to special variables.
4.3
MODEL
In natural language processing, the LSTM model is often used for processing, We used a smaller GRU-based agent language model. First take texts input, and convert words into word vectors according to a pre-built dictionary. The converted word vector is then used as an input to the LSTM model, and the prediction result is finally output.
Penn Treebank is available
https: / tht.t com/ wo jaa remba/ httpaaassssttteeeeatteteeo

---

Under review as a conference paper at ICLR 2021
4.4 PARAMETERS
This section enumerates the meanings and values of the various parameters mentioned. At the same time, we believe that the data each agent is independent and identically distributed. The parameters and their descriptions are listed in Tab.
Table 1: Table of parameters mentioned
Name
Represent letter
Value
Number of agents
K
20
Server training rounds
round epoch
10
The number of local epoch
The fraction of agents Learning rate of agents Learning rate of server
C
0.3
"
0.01
r/
0.001
Batch size Step size Differential privacy Embedding dimension
B
128
1.2
8
0.001
none
300
Forgetting coefficient
0.9
4.5 EXPERIMENTAL RESULTS
We use testing perplexity as an indicator of the evaluation model. In information theory, perplexity a measurement of how well probability distribution probability model predicts a sample. The perplexity of discrete probability distribution is defined as:
ppl(2) Hp) 2- E. p(or)logsp(x)
(6)
In the above formula, H (p) is the expected probability distribution. If the prediction result of model is m(x), then the perplexity of the language model defined as:
ppl(x) (p.m) 2 2. p(x))oggsm(x)
Obviously, (p) (p, m). Therefore, the smaller the perplexity, the more representative the probability distribution can be to better predict the sample distribution.
And we compared with the results of Shapley Value evaluation. Shapley Value is originated from coalitional game theory and has proven theoretical properties. It provides way to measure the impact and contribution of various agents. The definition of Shapley Value is:
QQ!! ([S) - 1Q1 1)! 1) (AQu()(()) - AQ(x)) |S|
(((x)
(8)
QCSV()
S is the set of all agents, Q C S = .... is subset of the agent set S, is the index of the agents, represents the size of the set, (Q(x) = impQ denotes the impact of agent set Q.
Because the complexity of directly calculating Shapley value is too high, we use the following estimation method get the (x) of each Agents
1 M
(x(x)=
Σ (Aq AQm (x))
(9)
where M is the number of iterations. Ag (x) denotes the impact random set Q". Finally, all the data obtained by Shapley Value is subjected to the same regularization processing ours.

---

Under review as a conference paper at ICLR 2021
To ensure the fairness of the measurement contribution in this experiment, ensure that each agent drawn the same number of times.
We randomly divided the data in the dataset, and distributed them evenly to all agents. Then, the corresponding special processing is performed on the data of the last few agents: reduce the amount of data by 30% and 70%, randomly generate the word sequence. Comparing the measurement results after special processing with the results when unprocessed data, concluded that the sensitivity of variables such data size and data quality is evaluated.
4.5.1 EXPERIMENTAL RESULTS OF NORMAL MEASUREMENT
We did not do any special treatment to any agents, and randomly divided the data set into each agent. The result shown in Fig. 2 In the figure, the abscissa is the index of agents, and the ordinate is the agents' contribution ratios. It can be that the deviation of each agent is not large, only individual agents are too high or too low. And the measurement result is similar to Shapley Value.
Sieply
0.00
0,02
10
Aggertss
Appntiindo
0.34
Specil Aguza
033
0.00
0.00
Agucti lndio
Agembitndt
Figure Experimental results of normal mea- Figure 3: Experimental results with randomize the surement.
word sequence.
4.5.2 EXPERIMENTAL RESULTS WITH RANDOMIZE THE WORD SEQUENCE
In this experiment, modified the datasets of the next four agents into random word sequences. These should be regarded as dirty data by the model, so as to get a smaller return. The results are shown in Fig. Both the method proposed in this paper and the Shpaley Value method identify these bad agents and give them a small contribution.
4.5.3 EXPERIMENTAL RESULTS WITH REDUCE THE AMOUNT DATA
To demonstrate the sensitivity of our measurement method to the amount of data, we performed the following experiments.
In this experiment, we reduced the data amount of the last 4 agents by 70%, and the data amount of other agents remained unchanged. As you in Fig. the contribution of the specially treated agents are significantly reduced. But it relatively not obvious in the measurement results of Shapley Value.

---

Under review as a conference paper at ICLR 2021
040
DCC
lndio
0.5
0384
032
0330
Speciil sgonts
000
Siencall Algges
Figure 4:
Experimental results with reduce the Figure 5: Experimental results with reduce the amount of data.
amount of data.
In order reflect the relative relationship between the amount of data, we processed the data of the last agents: agents with index 16 to 17 reduced the amount of data by 30%, agents with index 18 reduced the amount of data by 70%.
It can be seen in Fig. that the method proposed in this paper can reflect the reduced amount of data, while Shapley Value cannot show well. Note that the method proposed in this paper can better reflect the difference of smaller data volume.
CONCLUSION
A reasonable and fair completion of the measurement of the contribution of each participant in fed- crated learning and the establishment of an incentive mechanism are essential for the development "FedAtt" method to train the models Penn Treebank
of federated learning. In this paper, dataset. At the same time, we perform several special processing on the agents data, and compare the measurement results of the special processing with the measurement results the unprocessed data. Experimental results show that we use this method to reasonably establish a measurement contribution mechanism to evaluate the sensitivity of indicators such data size and data quality. The data is evaluated and trained by this method, and the calculated results are real-time and fast, and the contribution rate reflects accurately.
For future work, It is hoped that a better algorithm can be found that can more accurately measure the contribution to malicious agents, and the same time prevent the contribution confusion caused by more than half of malicious agents. It also should consider that the negative contribution of agents measurement and the comprehensiveness of the measurement. Preliminary classification (i.e., positive and negative) is carried out at the end to avoid attacks under the federated learning framework BaechanssmBaadaassee introduced, such the establishment of PVCG mechanism the supply side(Cong al. 2020). Improvements in these directions will promote the implementation and application the federated learning incentive mechanism.
et al., 2018); at the same time, the concept of game theory is
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yoshinori Aono, Takuya Hayashi, Lihua Wang, Shiho Moriai, et al. Privacy-preserve deep learn- ing via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 1335)1133, 2017.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning, 2018.
Xianglin Bao, Cheng Su, Yan Xiong, Wenchao Huang, and Yifei Hu. Flchain: A blockchain for auditable federated learning with trust and incentive. In 2019 5th International Conference on Big Data Computing and Communications (BIGCOM), pp. 151-159. IEEE, 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Ar- cas. communication-effiicen learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05622, 2016.
Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 2020.
Mingshu Cong, Xi Weng, Han Yu, Jiabao Qu, and Siu Ming Yiu. Optimal procurement auction for cooperative production of virtual products: Viekrey-clarke- groves meet gremer-melean 2020.
Vukasin Felbab, Péter Kiss, and Tomás Horváth. Optimization in federated learning. In ITAT, pp- 58-65, 2019.
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv: 172.:7777 2017.
Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thore. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv preprint 171.v:1711.10661 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Alexandra Ion, Johannes Frohnhofen, and Ludwig Wall. Proceedings of the 29th annual symposium on user interface software and technology. 2016.
Shaoxiong Ji, Shirui Pan, Guodong Long, Xue Li, Jing Jiang, and Zi Huang. Learning private neural language modeling with attentive aggregation. In 20/9 International Joint Conference on Neural Networks (IJCN)), pp. 1-8. IEEE, 2019.
Jiawen Kang, Zehui Xiong, Dusit Niyato, Shengli Xie, and Junshan Zhang. Incentive mechanism for reliable federated learning: A joint optimization approach to combining reputation and contract theory. IEEE Internet of Things Journal, 1700001141 2019.
Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Blockchained on-device federated learning. IEEE Communications Letters, 24(6):127--1288 2019.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.0549) 2016.
Yunlong Lu, Xiaohong Huang, Yueyue Dai, Sabita Maharjan, and Yan Zhang. Blockchain and federated learning for privacy-preserved data sharing industrial iot. IEEE Transactions on Industrial Informaics,, 16(6):41774188. 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-cfficii learning deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273- 1282. PMLR, 2017.
Jianji Ren, Haichao Wang. Tingting Hou, Shuai Zheng, and Chaosheng Tang. Federated learning- based computation offloading optimization in edge computing-supporte internet of things. IEEE Access, 7:69194 69201, 2019.

---

Under review as a conference paper at ICLR 2021
Guan Wang. Charlie Xiaoqian Dang, and Ziye Zhou. Measure contribution of participants in feder- ated learning. In 2019 IEEE International Conference on Big Data (Big Data), pp. 2597-2604. IEEE, 2019.
Zirui Xu, Zhao Yang. Jinjun Xiong, Jianlei Yang, and Xiang Chen. Elfish: Resource-aware federated learning on heterogeneous edge devices. arXiv preprint 1rXXiv:912.01000 2019.
Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept
and applications. ACM Trans. Intell. Syst. Technol, 10(2), January 2019. ISSN 2157-6904. doi: 10.1 10.1145/3298981
Yufeng Zhan, Peng Li, Zhihao Qu, Deze Zeng, and Song Guo. A learning-based incentive mecha- nism for federated learning. IEEE Internet of Things Journal, 2020.
10


Reviews:
[{"review_id": "w8_7d-o5qH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a low computational complexity method for weighting contributions of clients in a federated learning setting. The main contributions are to compare the weighting method with Shapley values and their sensitivity to low data volume and quality. The paper is based on the FedAtt paper that calculates weights based on the Euclidean distance between the server model and each client and for each layer.\n\nThe experimental setup is well described, including details about the hardware, software, datasets, model, and evaluation criteria. However, the model only specifies a \"smaller GRU-based model\" without giving any details of what that model is. They do not clearly describe some parameters of the approximation of the Shapley value calculation, reducing the value of the comparison between FedAtt and Shapley values. They could also have taken additional steps to improve the claims' confidence, e.g., only one dataset was used, which is relatively weak compared to the original FedAtt paper. The graphs in the results section could be described with more detail to explain what, e.g., the colors of the \"special agents\" mean. Also, there are no confidence measures specified, making it hard to evaluate the claims' validity.\n\nThe references include essential papers but are missing some core references, such as Federated Learning and Shapley values themselves. Also, related papers such as \"Active Federated Learning\" by Goetz et al. talk about very similar ideas but lack any mention in the paper.  The language and grammar could be improved, and some of the formulations make it hard to read. The comparison to Shapley values is also not motived in any detail, thus further reducing the paper contributions' value.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_x8-Dx8deQk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a new contribution measurement approach for federated learning. The basic idea is that the agent with a larger model update has a larger contribution. Specifically, based on FedAtt [1], the impact of a client is computed as the local updates plus the impact of the previous round times a decay rate. The experiments on a dataset show that the proposed approach can have a similar contribution measurement compared with Shapley Value.\n\n[1] Learning private neural language modeling with attentive aggregation.    IJCNN 2019\n \nStrengths: \n\n(1) The motivation of the paper is clear.\n\n(2) The studied area is important. Effective incentive mechanisms in federated learning are still an open challenge.\n\nWeakness:\n\n(1) The proposed idea lacks novelty and may not be applicable in general federated learning algorithms. The contribution of each client is simply evaluated by its local update in FedAtt. FedAtt is not a widely used federated learning algorithm currently. It is not clear whether the proposed approach is applicable to other standard federated learning algorithms such as FedAvg. Also, I do not understand why the paper focuses on FedAtt instead of FedAvg.\n\n(2) The paper lacks reasonable explanations for the proposed approach. A client may have arbitrary bad data and the local updated model may be far from the global optimal model. In such a case, since the distance between the local model and the global model is large, the contribution is also large according to the proposed approach, which is not reasonable. It is not clear how the proposed approach can handle such cases.\n\n(3) The experiments are weak and not clear. \n\n  a) It is not explained how the agent contribution rate is computed. \n\n  b) The experiments are conducted on a single dataset. More datasets are needed. \n\n  c) From Figure 2, it is hard to say that the proposed approach has a similar measurement with SV. \n\n  d) Since the motivation is to reduce the computation overhead, the authors should show compare the computation complexity or the computation time of the proposed approach and SV.\n\nMinor issues:\n\n(1) The writing can be improved, e.g., “Such” -> “For example,”\n\n(2) Figure 1 is not referred to in the text.\n\n(3) Figure2-5: orange and blue colors are not explained.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "80mIhJhBkT-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper designs an equation, i.e., equation (5) in the paper, to measure the impact or contribution of each participant/agent in federated learning. The designed measurement method is applied to attention aggregation algorithm of federated learning. Few experiments using Penn Treebank are conducted to support its claims.\n\nThis paper should be rejected because (1) the paper is unpolished and thus is hard to read, (2) the novelty appears quite weak, and (3) the experiments are difficult to understand and generally do not support its contributions\n\nConcerns:\n\nThe paper is difficult to read due to the poor use of English. Many sentences are incomprehensible. Thus, it was often impossible for me to determine exactly what the authors would like to say or describe. Please have your submission proof-read for English writing style and grammar issues. Moreover, please treat the equations as the parts of sentences and make sure that the caption formats of Figures obey the ICLR format.\n\nI also have a serious concern about the novelty of this paper. If my understanding is correct (due to the aforementioned reason), Subsection 3.3 is the only new material proposed by the authors. However, the proposed equation, i.e., equation (5), seems like a design choice without any theoretical justification or providing any intuitive reason, which significantly degrades the novelty of this paper.\n\nFinally, the experiments should be refined to support its main claims. As claimed in Section 1, the proposed measurement method is real-time and has low computational complexity. However, no experiment nor quantitative comparison addressing the running time and complexity between the proposed method and Shapley Value. Actually, the authors compared their method with the method of approximating Shapley Value instead of exact Shapley Value. Furthermore, please cite for Shapley Value papers.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3NFR5hFguhG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is to measure each client’s contribution to training the federated learning model. In particular, the contribution is measured by the distance between the local model and the global model in each iteration. The targeting problem is interesting, and the use of attention-based model divergence is also an interesting idea to measure the contribution. However, the paper lacks strict theoretical discussion to prove the proposed solution is a reasonable one rather than a heuristic method. Moreover, the experiment is too weak to support the claims. The paper’s technique contribution and originality are also limited. \n\nBelow are some detailed concerns.\n\n1) The authors need to make a clear definition of the assumed application scenario so that the below problems can be avoided or solved. \n\nIf the client’s contribution is linked to rewards, it is unavoidable that some clients will produce fake data to gain more contribution to the commercial federation system. Therefore, the paper should discuss the prevention of “attacking by fake data”. \n\nFor example, if the client randomly shuffles the index of neurons in the trained local model w_k, then the client’s local model will get a bigger s_k^l calculated by equation 2. Thus, this client is likely to gain a big reward at every iteration.\n\nAccording to equation 5, the contribution at the early stage will be discounted. It is unfair for the clients to be selected at an early stage. Therefore, from a systematic perspective, some clients may refuse to contribute to the training process at an early stage. \n\n\n2) Contribution is not enough\n\nThe core method comes from the FedAtt algorithm – an attention-based federated aggregation method. The paper’s primary contribution relies on section 3.3 to measure the contribution according to the gradients.  \n\n\n3) The experiments are too weak to support their claim. \n\nMore datasets and baseline methods are required, for example, the FEMNIST, FeCeleba.\n\nIt is unclear how to define an objective metric to measure the quality of the proposed method. The contribution is a subjective feeling that various to different tasks and assessor.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although this paper tackles an important problem, all reviewers agree that it requires further work before it can be published. First, the paper would need to be polished in order to be easier to read. Stronger experiments would also be needed in order to support the claims of the paper, e.g. by considering additional datasets and proper baselines. Finally, an important concern about this paper is novelty and originality. It is not clear at this point that the contribution is substantial enough for a conference like ICLR. Addressing these points would significantly improve the paper.
Decision: Reject