Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning

1 INTRODUCTION . Valuation problems are becoming increasingly more significant in various machine learning applications , ranging from feature interpretation ( Lundberg and Lee , 2017 ) , data valuation ( Ghorbani and Zou , 2019 ; Sim et al. , 2020 ) and model valuation for ensembles ( Rozemberczki and Sarkar , 2021 ) . They are often formulated as an importance assignment problem in cooperative games . A cooperative game ( N , F ( S ) ) consists of a grand coalition N = { 1 , ... , n } of n players and a value function ( a.k.a . characteristic function ) F ( S ) : 2N → R describing the collective payoff of a coalition/cooperation S. A fundamental problem in cooperative game theory is to assign an importance vector ( solution concept ) φ ( F ) ∈ Rn to n players . In this paper , we explore a probabilistic treatment of cooperative games ( N , F ( S ) ) . Such a treatment makes it possible to conduct learning and inference in a unified manner , and will yield connections with classical valuation criteria . Concretely , we seek a probability distribution over coalitions p ( S = S ) 1 , measuring the odds that a specific coalition S happens . Generally , we consider distributions where the probability of a coalition p ( S ) grows monotonically with the payoff F ( S ) . Among all the possible probability mass functions ( pmfs ) , how should we construct the proper p ( S ) ? We advocate to choose the pmf with the maximum entropy H ( p ) . This principle makes sense since maximizing the entropy minimizes the amount of prior information built into the distribution . In other words , it amounts to assume nothing about what is unknown , i.e. , choosing the most “ uniform ” distribution . Now finding a proper p ( S ) becomes the following constrained optimization problem : suppose each coalition S is associated with a payoff F ( S ) with probability p ( S ) . We would like to maximize the entropy H ( p ) = − ∑ S⊆N p ( S ) log p ( S ) , subject to the constraints that 1Note that distributions over subsets of N are equivalent to distributions of |N | = n binary random variables X1 , ... , Xn ∈ { 0 , 1 } : We use Xi as the indicator function of the event i ∈ S , or Xi = [ i ∈ S ] . With slight abuse of notation , we use S as a random variable represented as sets and often abbreviate p ( S = S ) as p ( S ) . ∑ S p ( S ) = 1 , p ( S ) ≥ 0 and ∑ S p ( S ) F ( S ) = µ ( i.e. , the average payoff is known as µ ) . Solving this optimization problem ( derivation in Appendix A ) , we reach the maximum entropy distribution : p ( S ) = exp ( F ( S ) /T ) Z , Z : = ∑ S′⊆N exp ( F ( S′ ) /T ) , ( 1 ) where T > 0 is the temperature . This is an energy-based model ( EBM ( LeCun et al. , 2006 ) ) with −F ( S ) as the energy function . The above energy-based treatment admits two benefits : i ) Where supervision is available , it enables learnable value functions F ( S ) through efficient training techniques of energy-based learning , such as noise contrastive estimation ( Gutmann and Hyvärinen , 2010 ) and score matching ( Hyvärinen , 2005 ) . ii ) Approximate inference techniques such as variational inference or sampling can be adopted to solve the valuation problem . Specifically , it enables to perform mean-field variational inference where parameters of the inferred surrogate distribution can be used as principled player valuations . Below , we explore mean-field variational inference for the energy-based formulation . Perhaps surprisingly , by conducting only one-step fixed point iteration for maximizing the mean-field ( ELBO ) objective , we recover classical valuation criteria , such as the Shapley value ( Shapley , 1953 ) and the Banzhaf value ( Penrose , 1946 ; Banzhaf III , 1964 ) . This observation also verifies the rationality of existing criteria , as they are all trying to decouple the correlations among players through the mean-field approach . By running the fixed point iteration for multiple steps , we achieve a trajectory of valuations , among which we define the valuation with the best conceivable decoupling error as the Variational Index . Our major contributions can be summarized as below : i ) We present a theoretically justified energy-based treatment for cooperative games . Through mean field inference , we provide a unified perspective on popular game-theoretic criteria . This provides an alternative motivation of existing criteria via a decoupling perspective , i.e. , decoupling correlations among n players through the mean-field approach . ii ) In pursuit of better decoupling performance , we propose to run fixed point iteration for multiple steps , which generates a trajectory of valuations . Under uniform initializations , they all satisfy a set of game-theoretic axioms , which are required for being suitable valuation criteria . We define the valuation with the best conceivable decoupling error as the Variational Index . iii ) Synthetic and real-world experiments demonstrate intriguing properties of the proposed Variational Index , including lower decoupling error and better valuation performance . 2 PRELIMINARIES AND BACKGROUND . Notation . We assume ei ∈ Rn being the standard ith basis vector and use boldface letters x ∈ RN and x ∈ Rn interchangebly to indicate an n-dimensional vector , where xi is the ith entry of x . By default , f ( · ) is used to denote a continuous function , and F ( · ) to represent a set function . For a differentiable function f ( · ) , ∇f ( · ) denotes its gradient . x|xi←k is the operation of setting the ith element of x to k , while keeping all other elements unchanged , i.e. , x|xi←k = x− xiei + kei . For two sets S and T , S + T and S − T represent set union and set difference , respectively . |S| is the cardinality of S. i is used to denote the singleton { i } with a bit abuse of notation . Existing valuation criteria . Various valuation criteria have been proposed from the area of cooperative games , amongst them the most famous ones are the Shapley value ( Shapley , 1953 ) and the Banzhaf value , which is extended from the Banzhaf power index ( Penrose , 1946 ; Banzhaf III , 1964 ) . For the Shapley value , the importance assigned to player i is : Shi = ∑ S⊆N−i [ F ( S + i ) − F ( S ) ] |S| ! ( n− |S| − 1 ) ! n ! . ( 2 ) One can see that it gives less weight to n/2-sized coalitions . The Banzhaf value assigns the following importance to player i : Bai = ∑ S⊆N−i [ F ( S + i ) − F ( S ) ] 1 2n−1 , ( 3 ) which uses uniform weights for all the coalitions . See Greco et al . ( 2015 ) for a comparison of them . Valuation problems in machine learning . Currently most classes of valuation problems ( Lundberg and Lee , 2017 ; Ghorbani and Zou , 2019 ; Sim et al. , 2020 ; Rozemberczki and Sarkar , 2021 ) use Shapley value as the valuation criterion . Along with the rapid progress of model interpretation in the past decades ( Zeiler and Fergus , 2014 ; Ribeiro et al. , 2016 ; Lundberg and Lee , 2017 ; Sundararajan et al. , 2017 ; Petsiuk et al. , 2018 ; Wang et al. , 2021a ) , attribution-based interpretation aims to assign importance to the features for a specific data instance ( x ∈ RN , y ) given a black-box modelM . Here each feature maps to a player in the game ( N , F ( S ) ) , and the value function F ( S ) is usually the model response , such as the predicted probability for classification problems , when feeding a subset S of features toM . The data valuation problem ( Ghorbani and Zou , 2019 ) tries to assign values to the samples in the training dataset N = { ( xi , yi ) } n1 for general supervised machine learning : one training sample corresponds to one player , and the value function F ( S ) indicates the predictor performance on some test dataset given access to only a subset of the training samples in S. Model valuation in ensembles ( Rozemberczki and Sarkar , 2021 ) measures importance of individual models in an ensemble in order to correctly label data points from a dataset , where each pre-trained model maps to a player and the value function measures the predictive performance of subsets of models . 3 RELATED WORK . Energy-based modeling . Energy based learning ( LeCun et al. , 2006 ) is a classical learning framework that uses an energy functionE ( x ) to measure the quality of a data point x . Energy based models have been applied to different domains , such as data generation ( Deng et al. , 2020 ) , out-of-distribution detection ( Liu et al. , 2020 ) , reinforcement learning ( Haarnoja et al. , 2017 ) , memory modeling ( Bartunov et al. , 2019 ) , discriminative learning ( Grathwohl et al. , 2019 ; Gustafsson et al. , 2020 ) and biologically-plausible training ( Scellier and Bengio , 2017 ) . Energy based learning admits principled training methods , such as contrastive divergence ( Hinton , 2002 ) , noise contrastive estimation ( Gutmann and Hyvärinen , 2010 ) and score matching ( Hyvärinen , 2005 ) . For approximate inference , sampling based approaches are mainly MCMC-style algorithms , such as stochastic gradient Langevin dynamics ( Welling and Teh , 2011 ) . For a wide class of EBMs with submodular or supermodular energies ( Djolonga and Krause , 2014 ) , there exist provable mean field inference algorithms with constant factor approximation guarantees ( Bian et al. , 2019 ) . Shapley values in machine learning . Shapley values have been extensively used for valuation problems in machine learning , including attribution-based interpretation ( Lipovetsky and Conklin , 2001 ; Cohen et al. , 2007 ; Strumbelj and Kononenko , 2010 ; Owen , 2014 ; Datta et al. , 2016 ; Lundberg and Lee , 2017 ; Chen et al. , 2018 ; Lundberg et al. , 2018 ; Kumar et al. , 2020 ; Williamson and Feng , 2020 ; Covert et al. , 2020b ; Wang et al. , 2021b ) , data valuation ( Ghorbani and Zou , 2019 ; Jia et al. , 2019b ; a ; Wang et al. , 2020 ; Fan et al. , 2021 ) , collaborative machine learning ( Sim et al. , 2020 ) and recently , model valuation in ensembles ( Rozemberczki and Sarkar , 2021 ) . For a detailed overview of papers using Shapley values for feature interpretation , please see Covert et al . ( 2020a ) and the references therein . To alleviate the exponential computational cost of exact evaluation , various methods have been proposed to approximate Shapley values in polynomial time ( Ancona et al. , 2017 ; 2019 ) . Owen ( 1972 ) proposes the multilinear extension purely as a representation of cooperative games and Okhrati and Lipani ( 2021 ) use it to develop sampling algorithms for Shapley values .

Summary: This paper studies valuation problems from cooperative game theory. There are $n$ agents and a valuation function $F: [n] \to R$ where $F(S)$ is the collective payoff of the coalition $S \subseteq [n]$. The goal is to use this function $F$ to define an importance vector $\phi(F) \in R^n$. Examples include the Shapley value and Banzhaf index.  The authors introduce a probabilistic treatment of this problem, where they use $F$ to define a probability distribution $p$ where $p(S)$ is the probability that coalition $S$ forms. They then phrase the problem of defining an importance vector $\phi(F)$ as a decoupling problem. Under $p$, the $n$ agents may be correlated in a complicated way, but to assign each of them an individual importance value, one must decouple their interactions, or simplify their correlations. The goal is then to find a product distribution $q$ that is as close to $p$ as possible under the KL divergence. Specifically, the authors define $q$ to be an $n$ independent Bernoulli distribution, where the probability that agent $i$ participates in the coalition is denoted $x_i$. The authors show how to optimize the probabilities $x_1, \dots, x_n$ using coordinate ascent. Finally, they define the importance score of player $i$ as $\log(x_i/(1-x_i))$ (ignoring a temperature $T$ term for simplicity). The authors show that the resulting importance vector satisfies many of the game-theoretic axioms that the Shapley value and Banzhaf index satisfy, like the null player, marginalism, and symmetric axioms.  In the experiments, the authors look at small instances with $n = 25$ where it is actually possible to compute the gradients exactly (as opposed to an approximate sampling method). The applications they look at are for data valuation and feature attribution in the context of machine learning. For these tasks, they show that their proposed approach performs about the same as the Shapley value and Banzhaf index, and sometimes a bit better.