Gradient Driven Rewards to Guarantee Fairness in Collaborative Machine Learning

1 Introduction . In collaborative machine learning ( CML ) , multiple agents ( e.g. , researchers , organizations , companies ) pool their resources ( e.g. , data ) together for a common learning task . It spans a wide variety of real-world applications such as digital healthcare [ 49 ] , clinical trial research [ 13 , 23 ] , wake word detection for smart voice assistants [ 27 ] , and next word prediction on mobile devices [ 15 ] . Federated learning ( FL ) provides a natural paradigm of CML [ 18 , 29 , 41 , 43 , 57 , 62 ] . In FL , the agents perform local model training ( e.g. , using stochastic gradient descent ) and share their resulting model parameter updates/gradients via a trusted server [ 40 , 56 , 59 ] . An important distinction of our work here from the standard FL literature is that the agents are self-interested and hence not necessarily cooperative like the worker nodes in distributed learning . The implication is that to achieve competitive predictive performance for the learning task , it is imperative to incentivize/reward the agents for contributing/sharing high-quality information in the form of model parameter updates/gradients [ 47 , 48 , 52 ] . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our work here adopts FL as the CML paradigm for designing a fair reward mechanism such that the ( self-interested ) agents who contribute more would not feel exploited but be rewarded commensurately . This is often regarded as fairness in cooperative game theory [ 42 ] , mechanism design [ 4 ] , and computational social choice [ 11 ] . To design such a fair reward mechanism , we need to address three main questions : Firstly , what is a suitable notion of fairness ? The Shapley value ( SV ) [ 50 ] from cooperative game theory is an appealing choice and has been used in ML [ 14 ] and FL [ 54 , 56 ] . However , existing SV-based works [ 19 , 37 , 54 , 56 ] typically require the availability of ( and all agents to agree on ) an auxiliary validation dataset and significant time overhead from evaluating the agents ’ contributions in the form of SVs and the resulting model training . To overcome these difficulties , we propose to instead exploit the alignment ( specifically , cosine similarity ) of an agent ’ s uploaded/contributed model parameter update/gradient vector ( or that aggregated over some agents ) to that aggregated over all agents ( hence measuring its quality/value and circumventing the need for a validation dataset [ 12 , 52 ] ) for devising our proposed cosine gradient Shapley value ( CGSV ) ( Sec . 3.2 ) which can be efficiently approximated with a bounded error ( Sec . 3.3 ) . Secondly , what is the choice of reward ? Various choices such as monetary rewards from a preallocated budget [ 65 , 66 ] or the total revenue generated from the collaboration through FL [ 9 , 10 ] have been proposed . Though it may seem natural to consider monetary rewards , it is not obvious how a common denomination between money and data/gradients [ 1 , 46 ] can be readily established , which makes it challenging to apply these works in practice . Instead , we propose to consider the aggregated parameter updates/gradients downloaded from the server as rewards to the agents . Finally , how can the gradient reward mechanism ensure fairness ? Our proposed mechanism exploits a sparsifying gradient trick ( Sec . 3.4 ) for controlling the quality of the aggregated parameter update/gradient downloaded from the server as reward to each agent at training time ( rather than post hoc [ 48 , 52 , 65 ] ) such that its quality is commensurate to that of the agent ’ s uploaded/contributed parameter update/gradient [ 2 , 7 ] . Consequently , an agent who uploads/contributes higher-quality parameter updates/gradients over the entire training process should eventually be rewarded with converged model parameters whose resulting training loss ( and hence predictive performance ) is closer to that of the server , as demonstrated in our fairness guarantee ( Sec . 3.5 ) [ 52 ] . In summary , the contributions of our work here to CML and FL include the following : • We propose a novel cosine gradient Shapley value ( CGSV ) ( Sec . 3.2 ) to fairly evaluate the expected marginal contribution of each agent ’ s uploaded model parameter update/gradient without needing an auxiliary validation dataset and present an efficient approximation of CGSV with a bounded error ( Sec . 3.3 ) . • Based on the approximate CGSV , we design a novel training-time gradient reward mechanism ( Sec . 3.4 ) with a fairness guarantee ( Sec . 3.5 ) by exploiting the trick of sparsifying the aggregated parameter update/gradient downloaded from the server as reward to each agent such that its resulting quality is commensurate to that of the agent ’ s uploaded/contributed parameter update/gradient . • We empirically demonstrate the effectiveness of our fair gradient reward mechanism on multiple benchmark datasets in terms of fairness , predictive performance , and time overhead ( Sec . 4 ) . 2 Related Work . Reward design and choice in CML . In related topics such as FL [ 30 , 36 , 38 , 47 , 59 , 63 , 66 ] , Bayesian CML [ 52 ] , collaborative generative modeling [ 55 ] , and data sharing [ 13 , 23 , 48 ] , designing appropriate rewards to encourage collaboration ( e.g. , sharing real or synthetic data , gradients , or other information ) is a non-trivial problem . A useful solution concept should provide a formal notion of fairness , a suitable form/denomination of reward , and a principled way to guarantee fairness via a carefully designed reward mechanism . Previous works have considered monetary rewards from a pre-allocated budget [ 65 , 66 ] or the total revenue generated from the collaboration [ 9 , 10 ] , or simply an abstract yet quantifiable form of reward [ 47 , 48 ] . Though it may seem natural to consider monetary rewards , it is not obvious how a common denomination between money and data/gradients [ 1 , 46 ] can be readily established , which makes it challenging to apply these works in practice . The work of [ 66 ] has explored a different avenue of using a reverse auction to guarantee truthfulness in its mechanism instead of fairness . Fairness notions . The Shapley value ( SV ) [ 50 ] from cooperative game theory is widely regarded as a principled notion of fairness [ 4 , 11 , 42 ] due to its several desirable properties such as symmetry and null player . Existing SV-based works have considered fairness in the sense of rewarding agents according to their contributions [ 19 , 54 , 56 ] . However , they typically require the availability of ( and all agents to agree on ) an auxiliary validation dataset [ 37 , 52 ] and significant time overhead from evaluating the agents ’ contributions in the form of SVs and the resulting model training [ 14 , 19 , 56 ] . In contrast , the work of [ 31 ] has adopted an egalitarian notion of fairness by aiming to equalize the final individual performance among agents , which is fundamentally different from SV . Different from the fairness definition in [ 31 ] , we adopt a fairness notion formalized by SV [ 14 , 19 , 52 , 54 , 56 ] . Our proposed work is novel in the application of SV : While previous works use the validation accuracy [ 14 , 19 , 54 , 56 ] , we exploit the cosine similarity between model parameter updates/gradient vectors [ 12 ] for devising our proposed cosine gradient Shapley value ( CGSV ) ( Sec . 3.2 ) to fairly evaluate the expected marginal contribution of each agent ’ s uploaded model parameter update/gradient . Based on the CGSV , we design a novel training-time gradient reward mechanism ( Sec . 3.4 ) with a fairness guarantee ( Sec . 3.5 ) and empirically show that it outperforms several existing FL baselines in terms of predictive performance , fairness , and time overhead ( Sec . 4.2 ) . 3 Fair Gradient Reward Mechanism . 3.1 Vanilla Federated Learning ( FL ) Problem Setting and Notations . The vanilla FL problem [ 56 , 59 ] involves a set N : = { i } i=1 , ... , N of N honest agents learning a D-dimensional vector w ∈ RD of model parameters to minimize a loss function F ( w ) that can be additively decomposed into N local differentiable loss functions Fi ( w ) defined using the local dataset Di of agent i ∈ N and weighted by its importance pi ≥ 0 ( e.g. , proportional to |Di| ) . That is , F ( w ) : = ∑ i∈N pi Fi ( w ) where ∑ i∈N pi = 1 . We call N the grand coalition ; a coalition S ⊆ N is then a subset of the grand coalition N of N agents . In iteration t = 0 , every agent i ∈ N starts with the same initialized parameter vector wi,0 : = w0 as the server . In iteration t > 0 , every agent i ∈ N calculates a parameter update ∆wi , t : = −ηt∇Fi ( wi , t−1 ) with step size ηt and gradient ∇Fi ( wi , t−1 ) w.r.t . parameter vector wi , t−1 and uploads it to a trusted server who normalizes and aggregates all agents ’ parameter updates as follows : ui , t : = Γ ∆wi , t/‖∆wi , t‖ , uN , t : = ∑ i∈N ri , t−1 ui , t ( 1 ) where Γ is a normalization coefficient used to prevent gradient explosion [ 33 , 45 ] and the importance coefficient ri , t−1 will be described later in Sec 3.4 . So , we call ( 1 ) the gradient aggregation step . The gradient download step then follows where every agent i ∈ N downloads the aggregated parameter update/gradient uN , t ( 1 ) from the server ( as reward ) for updating its model parameters wi , t : = wi , t−1 + uN , t to the same wt : = wt−1 + uN , t as the server . That is , wi , t = wt for all i ∈ N and t ∈ Z+ ∪ { 0 } . We define uS , t for any coalition S ⊆ N in a similar way as uN , t ( 1 ) . For brevity , we omit t from our notations in Secs . 3.2 and 3.3 since we only refer to iteration t . 3.2 Cosine Gradient Shapley Value ( CGSV ) for Fairness . In the gradient aggregation step ( 1 ) , the quality/value of coalition S ’ s ( normalized ) aggregated parameter update/gradient uS can be measured by its cosine similarity cos ( uS , uN ) : = 〈uS , uN 〉/ ( ‖uS‖‖uN ‖ ) to the grand coalition N ’ s aggregated parameter update/gradient uN [ 12 , 28 , 35 ] . We use this cosine similarity measure as our gradient valuation function ν ( S ) : = cos ( uS , uN ) . Intuitively , if the direction of uS aligns more closely with that of uN , then its quality/value ν ( S ) is higher . Using ν , the contribution φi of agent i ∈ N is defined based on the notion of Shapley value ( SV ) [ 50 ] which measures its expected marginal contribution when joining the other agents preceding it in any permutation and satisfies certain desirable fairness properties [ 5 ] , such as null player ( i.e. , an agent with no marginal contribution has zero SV ) , symmetry ( i.e. , agents with identical marginal contributions have equal SVs ) , among others , as formally discussed in Appendix A.1 : Definition 1 ( Cosine gradient Shapley value ( CGSV ) ) . Let ΠN be a set of all possible permutations of N and Sπ , i be the coalition of agents preceding agent i in permutation π ∈ ΠN . The CGSV of agent i ∈ N is defined as φi : = ( 1/N ! ) ∑ π∈ΠN [ ν ( Sπ , i ∪ { i } ) − ν ( Sπ , i ) ] . ( 2 ) If φi is negative , then it follows from the weighted sum of parameter updates/gradients in ( 1 ) that ui points in an opposite direction to some other parameter updates/gradients and hence has negative cosine similarities to them . In practice , due to the noisy training arising from the use of stochastic gradient descent ( SGD ) and/or a highly non-convex loss function , φi may at times be negative even for an honest agent i . When the number of such cases is limited , training via SGD can still converge to yield a competitive predictive performance , as empirically validated in [ 12 ] .

Summary: The paper studies a collaborative setting where platform aims to facilitate self-interested agents pooling their resources towards a common learning task. Each agent calculates the gradient on their local objective that is based on their local dataset, and the common goal is to optimize a weighted sum of these objectives. The author propose that agents be given sparse approximations of the average gradient, where the approximation quality matches the ranked order given by the Shapley value, where the value of a subset is the angle between the average gradient reported by agents on that subset and the average gradient across all agents. They prove that agents who upload better agents have better predictive performance guarantees and evaluate their approach on several datasets.