Gradient Driven Rewards to Guarantee Fairness in Collaborative Machine Learning

1 Introduction . In collaborative machine learning ( CML ) , multiple agents ( e.g. , researchers , organizations , companies ) pool their resources ( e.g. , data ) together for a common learning task . It spans a wide variety of real-world applications such as digital healthcare [ 49 ] , clinical trial research [ 13 , 23 ] , wake word detection for smart voice assistants [ 27 ] , and next word prediction on mobile devices [ 15 ] . Federated learning ( FL ) provides a natural paradigm of CML [ 18 , 29 , 41 , 43 , 57 , 62 ] . In FL , the agents perform local model training ( e.g. , using stochastic gradient descent ) and share their resulting model parameter updates/gradients via a trusted server [ 40 , 56 , 59 ] . An important distinction of our work here from the standard FL literature is that the agents are self-interested and hence not necessarily cooperative like the worker nodes in distributed learning . The implication is that to achieve competitive predictive performance for the learning task , it is imperative to incentivize/reward the agents for contributing/sharing high-quality information in the form of model parameter updates/gradients [ 47 , 48 , 52 ] . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our work here adopts FL as the CML paradigm for designing a fair reward mechanism such that the ( self-interested ) agents who contribute more would not feel exploited but be rewarded commensurately . This is often regarded as fairness in cooperative game theory [ 42 ] , mechanism design [ 4 ] , and computational social choice [ 11 ] . To design such a fair reward mechanism , we need to address three main questions : Firstly , what is a suitable notion of fairness ? The Shapley value ( SV ) [ 50 ] from cooperative game theory is an appealing choice and has been used in ML [ 14 ] and FL [ 54 , 56 ] . However , existing SV-based works [ 19 , 37 , 54 , 56 ] typically require the availability of ( and all agents to agree on ) an auxiliary validation dataset and significant time overhead from evaluating the agents ’ contributions in the form of SVs and the resulting model training . To overcome these difficulties , we propose to instead exploit the alignment ( specifically , cosine similarity ) of an agent ’ s uploaded/contributed model parameter update/gradient vector ( or that aggregated over some agents ) to that aggregated over all agents ( hence measuring its quality/value and circumventing the need for a validation dataset [ 12 , 52 ] ) for devising our proposed cosine gradient Shapley value ( CGSV ) ( Sec . 3.2 ) which can be efficiently approximated with a bounded error ( Sec . 3.3 ) . Secondly , what is the choice of reward ? Various choices such as monetary rewards from a preallocated budget [ 65 , 66 ] or the total revenue generated from the collaboration through FL [ 9 , 10 ] have been proposed . Though it may seem natural to consider monetary rewards , it is not obvious how a common denomination between money and data/gradients [ 1 , 46 ] can be readily established , which makes it challenging to apply these works in practice . Instead , we propose to consider the aggregated parameter updates/gradients downloaded from the server as rewards to the agents . Finally , how can the gradient reward mechanism ensure fairness ? Our proposed mechanism exploits a sparsifying gradient trick ( Sec . 3.4 ) for controlling the quality of the aggregated parameter update/gradient downloaded from the server as reward to each agent at training time ( rather than post hoc [ 48 , 52 , 65 ] ) such that its quality is commensurate to that of the agent ’ s uploaded/contributed parameter update/gradient [ 2 , 7 ] . Consequently , an agent who uploads/contributes higher-quality parameter updates/gradients over the entire training process should eventually be rewarded with converged model parameters whose resulting training loss ( and hence predictive performance ) is closer to that of the server , as demonstrated in our fairness guarantee ( Sec . 3.5 ) [ 52 ] . In summary , the contributions of our work here to CML and FL include the following : • We propose a novel cosine gradient Shapley value ( CGSV ) ( Sec . 3.2 ) to fairly evaluate the expected marginal contribution of each agent ’ s uploaded model parameter update/gradient without needing an auxiliary validation dataset and present an efficient approximation of CGSV with a bounded error ( Sec . 3.3 ) . • Based on the approximate CGSV , we design a novel training-time gradient reward mechanism ( Sec . 3.4 ) with a fairness guarantee ( Sec . 3.5 ) by exploiting the trick of sparsifying the aggregated parameter update/gradient downloaded from the server as reward to each agent such that its resulting quality is commensurate to that of the agent ’ s uploaded/contributed parameter update/gradient . • We empirically demonstrate the effectiveness of our fair gradient reward mechanism on multiple benchmark datasets in terms of fairness , predictive performance , and time overhead ( Sec . 4 ) . 2 Related Work . Reward design and choice in CML . In related topics such as FL [ 30 , 36 , 38 , 47 , 59 , 63 , 66 ] , Bayesian CML [ 52 ] , collaborative generative modeling [ 55 ] , and data sharing [ 13 , 23 , 48 ] , designing appropriate rewards to encourage collaboration ( e.g. , sharing real or synthetic data , gradients , or other information ) is a non-trivial problem . A useful solution concept should provide a formal notion of fairness , a suitable form/denomination of reward , and a principled way to guarantee fairness via a carefully designed reward mechanism . Previous works have considered monetary rewards from a pre-allocated budget [ 65 , 66 ] or the total revenue generated from the collaboration [ 9 , 10 ] , or simply an abstract yet quantifiable form of reward [ 47 , 48 ] . Though it may seem natural to consider monetary rewards , it is not obvious how a common denomination between money and data/gradients [ 1 , 46 ] can be readily established , which makes it challenging to apply these works in practice . The work of [ 66 ] has explored a different avenue of using a reverse auction to guarantee truthfulness in its mechanism instead of fairness . Fairness notions . The Shapley value ( SV ) [ 50 ] from cooperative game theory is widely regarded as a principled notion of fairness [ 4 , 11 , 42 ] due to its several desirable properties such as symmetry and null player . Existing SV-based works have considered fairness in the sense of rewarding agents according to their contributions [ 19 , 54 , 56 ] . However , they typically require the availability of ( and all agents to agree on ) an auxiliary validation dataset [ 37 , 52 ] and significant time overhead from evaluating the agents ’ contributions in the form of SVs and the resulting model training [ 14 , 19 , 56 ] . In contrast , the work of [ 31 ] has adopted an egalitarian notion of fairness by aiming to equalize the final individual performance among agents , which is fundamentally different from SV . Different from the fairness definition in [ 31 ] , we adopt a fairness notion formalized by SV [ 14 , 19 , 52 , 54 , 56 ] . Our proposed work is novel in the application of SV : While previous works use the validation accuracy [ 14 , 19 , 54 , 56 ] , we exploit the cosine similarity between model parameter updates/gradient vectors [ 12 ] for devising our proposed cosine gradient Shapley value ( CGSV ) ( Sec . 3.2 ) to fairly evaluate the expected marginal contribution of each agent ’ s uploaded model parameter update/gradient . Based on the CGSV , we design a novel training-time gradient reward mechanism ( Sec . 3.4 ) with a fairness guarantee ( Sec . 3.5 ) and empirically show that it outperforms several existing FL baselines in terms of predictive performance , fairness , and time overhead ( Sec . 4.2 ) . 3 Fair Gradient Reward Mechanism . 3.1 Vanilla Federated Learning ( FL ) Problem Setting and Notations . The vanilla FL problem [ 56 , 59 ] involves a set N : = { i } i=1 , ... , N of N honest agents learning a D-dimensional vector w ∈ RD of model parameters to minimize a loss function F ( w ) that can be additively decomposed into N local differentiable loss functions Fi ( w ) defined using the local dataset Di of agent i ∈ N and weighted by its importance pi ≥ 0 ( e.g. , proportional to |Di| ) . That is , F ( w ) : = ∑ i∈N pi Fi ( w ) where ∑ i∈N pi = 1 . We call N the grand coalition ; a coalition S ⊆ N is then a subset of the grand coalition N of N agents . In iteration t = 0 , every agent i ∈ N starts with the same initialized parameter vector wi,0 : = w0 as the server . In iteration t > 0 , every agent i ∈ N calculates a parameter update ∆wi , t : = −ηt∇Fi ( wi , t−1 ) with step size ηt and gradient ∇Fi ( wi , t−1 ) w.r.t . parameter vector wi , t−1 and uploads it to a trusted server who normalizes and aggregates all agents ’ parameter updates as follows : ui , t : = Γ ∆wi , t/‖∆wi , t‖ , uN , t : = ∑ i∈N ri , t−1 ui , t ( 1 ) where Γ is a normalization coefficient used to prevent gradient explosion [ 33 , 45 ] and the importance coefficient ri , t−1 will be described later in Sec 3.4 . So , we call ( 1 ) the gradient aggregation step . The gradient download step then follows where every agent i ∈ N downloads the aggregated parameter update/gradient uN , t ( 1 ) from the server ( as reward ) for updating its model parameters wi , t : = wi , t−1 + uN , t to the same wt : = wt−1 + uN , t as the server . That is , wi , t = wt for all i ∈ N and t ∈ Z+ ∪ { 0 } . We define uS , t for any coalition S ⊆ N in a similar way as uN , t ( 1 ) . For brevity , we omit t from our notations in Secs . 3.2 and 3.3 since we only refer to iteration t . 3.2 Cosine Gradient Shapley Value ( CGSV ) for Fairness . In the gradient aggregation step ( 1 ) , the quality/value of coalition S ’ s ( normalized ) aggregated parameter update/gradient uS can be measured by its cosine similarity cos ( uS , uN ) : = 〈uS , uN 〉/ ( ‖uS‖‖uN ‖ ) to the grand coalition N ’ s aggregated parameter update/gradient uN [ 12 , 28 , 35 ] . We use this cosine similarity measure as our gradient valuation function ν ( S ) : = cos ( uS , uN ) . Intuitively , if the direction of uS aligns more closely with that of uN , then its quality/value ν ( S ) is higher . Using ν , the contribution φi of agent i ∈ N is defined based on the notion of Shapley value ( SV ) [ 50 ] which measures its expected marginal contribution when joining the other agents preceding it in any permutation and satisfies certain desirable fairness properties [ 5 ] , such as null player ( i.e. , an agent with no marginal contribution has zero SV ) , symmetry ( i.e. , agents with identical marginal contributions have equal SVs ) , among others , as formally discussed in Appendix A.1 : Definition 1 ( Cosine gradient Shapley value ( CGSV ) ) . Let ΠN be a set of all possible permutations of N and Sπ , i be the coalition of agents preceding agent i in permutation π ∈ ΠN . The CGSV of agent i ∈ N is defined as φi : = ( 1/N ! ) ∑ π∈ΠN [ ν ( Sπ , i ∪ { i } ) − ν ( Sπ , i ) ] . ( 2 ) If φi is negative , then it follows from the weighted sum of parameter updates/gradients in ( 1 ) that ui points in an opposite direction to some other parameter updates/gradients and hence has negative cosine similarities to them . In practice , due to the noisy training arising from the use of stochastic gradient descent ( SGD ) and/or a highly non-convex loss function , φi may at times be negative even for an honest agent i . When the number of such cases is limited , training via SGD can still converge to yield a competitive predictive performance , as empirically validated in [ 12 ] .

Summary: This paper explores the problem of unequal contribution in a gradient-based collaborative learning. The authors propose Cosine Gradient Shapley (CGS) measure together with gradient compatibility function to evaluate shared gradient qualities to assign a fair reward to contributing agents. Extensive set of experiments were conducted to support the proposed ideas and show effectiveness of the approach.