Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning

1 INTRODUCTION . Valuation problems are becoming increasingly more significant in various machine learning applications , ranging from feature interpretation ( Lundberg and Lee , 2017 ) , data valuation ( Ghorbani and Zou , 2019 ; Sim et al. , 2020 ) and model valuation for ensembles ( Rozemberczki and Sarkar , 2021 ) . They are often formulated as an importance assignment problem in cooperative games . A cooperative game ( N , F ( S ) ) consists of a grand coalition N = { 1 , ... , n } of n players and a value function ( a.k.a . characteristic function ) F ( S ) : 2N → R describing the collective payoff of a coalition/cooperation S. A fundamental problem in cooperative game theory is to assign an importance vector ( solution concept ) φ ( F ) ∈ Rn to n players . In this paper , we explore a probabilistic treatment of cooperative games ( N , F ( S ) ) . Such a treatment makes it possible to conduct learning and inference in a unified manner , and will yield connections with classical valuation criteria . Concretely , we seek a probability distribution over coalitions p ( S = S ) 1 , measuring the odds that a specific coalition S happens . Generally , we consider distributions where the probability of a coalition p ( S ) grows monotonically with the payoff F ( S ) . Among all the possible probability mass functions ( pmfs ) , how should we construct the proper p ( S ) ? We advocate to choose the pmf with the maximum entropy H ( p ) . This principle makes sense since maximizing the entropy minimizes the amount of prior information built into the distribution . In other words , it amounts to assume nothing about what is unknown , i.e. , choosing the most “ uniform ” distribution . Now finding a proper p ( S ) becomes the following constrained optimization problem : suppose each coalition S is associated with a payoff F ( S ) with probability p ( S ) . We would like to maximize the entropy H ( p ) = − ∑ S⊆N p ( S ) log p ( S ) , subject to the constraints that 1Note that distributions over subsets of N are equivalent to distributions of |N | = n binary random variables X1 , ... , Xn ∈ { 0 , 1 } : We use Xi as the indicator function of the event i ∈ S , or Xi = [ i ∈ S ] . With slight abuse of notation , we use S as a random variable represented as sets and often abbreviate p ( S = S ) as p ( S ) . ∑ S p ( S ) = 1 , p ( S ) ≥ 0 and ∑ S p ( S ) F ( S ) = µ ( i.e. , the average payoff is known as µ ) . Solving this optimization problem ( derivation in Appendix A ) , we reach the maximum entropy distribution : p ( S ) = exp ( F ( S ) /T ) Z , Z : = ∑ S′⊆N exp ( F ( S′ ) /T ) , ( 1 ) where T > 0 is the temperature . This is an energy-based model ( EBM ( LeCun et al. , 2006 ) ) with −F ( S ) as the energy function . The above energy-based treatment admits two benefits : i ) Where supervision is available , it enables learnable value functions F ( S ) through efficient training techniques of energy-based learning , such as noise contrastive estimation ( Gutmann and Hyvärinen , 2010 ) and score matching ( Hyvärinen , 2005 ) . ii ) Approximate inference techniques such as variational inference or sampling can be adopted to solve the valuation problem . Specifically , it enables to perform mean-field variational inference where parameters of the inferred surrogate distribution can be used as principled player valuations . Below , we explore mean-field variational inference for the energy-based formulation . Perhaps surprisingly , by conducting only one-step fixed point iteration for maximizing the mean-field ( ELBO ) objective , we recover classical valuation criteria , such as the Shapley value ( Shapley , 1953 ) and the Banzhaf value ( Penrose , 1946 ; Banzhaf III , 1964 ) . This observation also verifies the rationality of existing criteria , as they are all trying to decouple the correlations among players through the mean-field approach . By running the fixed point iteration for multiple steps , we achieve a trajectory of valuations , among which we define the valuation with the best conceivable decoupling error as the Variational Index . Our major contributions can be summarized as below : i ) We present a theoretically justified energy-based treatment for cooperative games . Through mean field inference , we provide a unified perspective on popular game-theoretic criteria . This provides an alternative motivation of existing criteria via a decoupling perspective , i.e. , decoupling correlations among n players through the mean-field approach . ii ) In pursuit of better decoupling performance , we propose to run fixed point iteration for multiple steps , which generates a trajectory of valuations . Under uniform initializations , they all satisfy a set of game-theoretic axioms , which are required for being suitable valuation criteria . We define the valuation with the best conceivable decoupling error as the Variational Index . iii ) Synthetic and real-world experiments demonstrate intriguing properties of the proposed Variational Index , including lower decoupling error and better valuation performance . 2 PRELIMINARIES AND BACKGROUND . Notation . We assume ei ∈ Rn being the standard ith basis vector and use boldface letters x ∈ RN and x ∈ Rn interchangebly to indicate an n-dimensional vector , where xi is the ith entry of x . By default , f ( · ) is used to denote a continuous function , and F ( · ) to represent a set function . For a differentiable function f ( · ) , ∇f ( · ) denotes its gradient . x|xi←k is the operation of setting the ith element of x to k , while keeping all other elements unchanged , i.e. , x|xi←k = x− xiei + kei . For two sets S and T , S + T and S − T represent set union and set difference , respectively . |S| is the cardinality of S. i is used to denote the singleton { i } with a bit abuse of notation . Existing valuation criteria . Various valuation criteria have been proposed from the area of cooperative games , amongst them the most famous ones are the Shapley value ( Shapley , 1953 ) and the Banzhaf value , which is extended from the Banzhaf power index ( Penrose , 1946 ; Banzhaf III , 1964 ) . For the Shapley value , the importance assigned to player i is : Shi = ∑ S⊆N−i [ F ( S + i ) − F ( S ) ] |S| ! ( n− |S| − 1 ) ! n ! . ( 2 ) One can see that it gives less weight to n/2-sized coalitions . The Banzhaf value assigns the following importance to player i : Bai = ∑ S⊆N−i [ F ( S + i ) − F ( S ) ] 1 2n−1 , ( 3 ) which uses uniform weights for all the coalitions . See Greco et al . ( 2015 ) for a comparison of them . Valuation problems in machine learning . Currently most classes of valuation problems ( Lundberg and Lee , 2017 ; Ghorbani and Zou , 2019 ; Sim et al. , 2020 ; Rozemberczki and Sarkar , 2021 ) use Shapley value as the valuation criterion . Along with the rapid progress of model interpretation in the past decades ( Zeiler and Fergus , 2014 ; Ribeiro et al. , 2016 ; Lundberg and Lee , 2017 ; Sundararajan et al. , 2017 ; Petsiuk et al. , 2018 ; Wang et al. , 2021a ) , attribution-based interpretation aims to assign importance to the features for a specific data instance ( x ∈ RN , y ) given a black-box modelM . Here each feature maps to a player in the game ( N , F ( S ) ) , and the value function F ( S ) is usually the model response , such as the predicted probability for classification problems , when feeding a subset S of features toM . The data valuation problem ( Ghorbani and Zou , 2019 ) tries to assign values to the samples in the training dataset N = { ( xi , yi ) } n1 for general supervised machine learning : one training sample corresponds to one player , and the value function F ( S ) indicates the predictor performance on some test dataset given access to only a subset of the training samples in S. Model valuation in ensembles ( Rozemberczki and Sarkar , 2021 ) measures importance of individual models in an ensemble in order to correctly label data points from a dataset , where each pre-trained model maps to a player and the value function measures the predictive performance of subsets of models . 3 RELATED WORK . Energy-based modeling . Energy based learning ( LeCun et al. , 2006 ) is a classical learning framework that uses an energy functionE ( x ) to measure the quality of a data point x . Energy based models have been applied to different domains , such as data generation ( Deng et al. , 2020 ) , out-of-distribution detection ( Liu et al. , 2020 ) , reinforcement learning ( Haarnoja et al. , 2017 ) , memory modeling ( Bartunov et al. , 2019 ) , discriminative learning ( Grathwohl et al. , 2019 ; Gustafsson et al. , 2020 ) and biologically-plausible training ( Scellier and Bengio , 2017 ) . Energy based learning admits principled training methods , such as contrastive divergence ( Hinton , 2002 ) , noise contrastive estimation ( Gutmann and Hyvärinen , 2010 ) and score matching ( Hyvärinen , 2005 ) . For approximate inference , sampling based approaches are mainly MCMC-style algorithms , such as stochastic gradient Langevin dynamics ( Welling and Teh , 2011 ) . For a wide class of EBMs with submodular or supermodular energies ( Djolonga and Krause , 2014 ) , there exist provable mean field inference algorithms with constant factor approximation guarantees ( Bian et al. , 2019 ) . Shapley values in machine learning . Shapley values have been extensively used for valuation problems in machine learning , including attribution-based interpretation ( Lipovetsky and Conklin , 2001 ; Cohen et al. , 2007 ; Strumbelj and Kononenko , 2010 ; Owen , 2014 ; Datta et al. , 2016 ; Lundberg and Lee , 2017 ; Chen et al. , 2018 ; Lundberg et al. , 2018 ; Kumar et al. , 2020 ; Williamson and Feng , 2020 ; Covert et al. , 2020b ; Wang et al. , 2021b ) , data valuation ( Ghorbani and Zou , 2019 ; Jia et al. , 2019b ; a ; Wang et al. , 2020 ; Fan et al. , 2021 ) , collaborative machine learning ( Sim et al. , 2020 ) and recently , model valuation in ensembles ( Rozemberczki and Sarkar , 2021 ) . For a detailed overview of papers using Shapley values for feature interpretation , please see Covert et al . ( 2020a ) and the references therein . To alleviate the exponential computational cost of exact evaluation , various methods have been proposed to approximate Shapley values in polynomial time ( Ancona et al. , 2017 ; 2019 ) . Owen ( 1972 ) proposes the multilinear extension purely as a representation of cooperative games and Okhrati and Lipani ( 2021 ) use it to develop sampling algorithms for Shapley values .

Summary: This paper proposes an energy-based perspective on cooperative games that permits a gradient-based calculation of Shapley/Banzhaf values, as well as the definition of a new alternative value - the variational index. A quick summary of the paper's key ideas is:  - For a given cooperative game $F$, we can seek an entropy maximizing distribution over coalitions $p(S)$ that satisfies a constraint on the mean coalition value $\mu$ - Solving the entropy maximization problem via its Lagrangian yields the Boltzmann distribution $p(S) \propto \exp(F(S)/T)$, where the temperature $T$ has a one-to-one correspondence with the mean coalition value $\mu$ (this result is in the appendix). This distribution gives more probability mass to coalitions that achieve higher values - We can seek a simpler alternative to $p(S)$ by doing mean-field variational inference, i.e., finding a factorized surrogate $q(S)$ where each player's participation is determined by independent Bernoulli RVs. The result will intuitively assign higher probabilities to players that belong to high-value coalitions, so these probabilities can serve a function similar to Shapley/Banzhaf values - The VI approach suggests a KL divergence minimization (or ELBO maximization) objective for learning $q(S)$, which is parameterized by $x \in [0, 1]^n$. Doing gradient descent on this objective yields a relatively simple update rule, where we repeatedly set $x_i^+ = \sigma(\nabla_i f_{mt}(x) / T)$ for $i = 1, \ldots, n$ - The authors define the "variational index" as a function of the solution to the KL divergence minimization problem: $s^* = T\sigma^{-1}(x^*)$ - The authors find that the Banzhaf value can be found using a single-step update to a particular initialization of the KL divergence minimization problem (luckily the temperature $T$ is not important for single-step updates). Similarly, they find that the Shapley value is the average of the single-step update applied to different initializations (again, the temperature doesn't matter). Finally, the authors point out that any single-step update applied to a symmetric initialization will be a probabilistic value (a class of solution concepts in cooperative game theory, of which Shapley/Banzhaf values are special cases) - Lastly, the authors suggest a practical sampling-based approach to calculating the necessary gradients, which are just as difficult to calculate as the Shapley/Banzhaf values because they require calculating the value for every coalition $S \subseteq N$  The experiments compare the variational index to Shapley and Banzhaf values in data and feature removal tasks, finding that it performs quite favorably in the settings examined.