# Ensembles of Random SHAPs
## 3 Shapley Values and the Explanation Model  
One of the most powerful approaches to explaining predictions of the black-box machine learning models is the approach based on using the Shapley values [12] as a key concept in coalitional games. According to the concept, the total gain of a game isdistributed to players such that desirable properties, including efficiency, symmetry, and linearity, are fulfilled. In the framework of the machine learning, the gain can be viewed as the machine learning model prediction or the model output, and a player is a feature of input data. Hence, contributions of features to the model prediction can be estimated by Shapley values. The \(i\)-th feature importance is defined by the Shapley value  
\[\phi_{i}(f)=\phi_{i}=\sum_{S\subseteq N\setminus\{i\}}B(S,N)[f(S\cup\{i\})-f(S )], \tag{1}\]  
where \(f(S)\) is the characteristic function in terms of coalitional games or the black-box model prediction under condition that a subset \(S\) of features are used as the corresponding input in terms of machine learning: \(N\) is the set of all features; \(B(S,N)\) is defined as  
\[B(S,N)=\frac{|S|!(|N|-|S|-1)!}{|N|!}. \tag{2}\]  
It can be seen from the above expression that the Shapley value \(\phi_{i}\) can be regarded as the average contribution of the \(i\)-th feature across all possible permutations of the feature set. The Shapley value has the following important properties:  
**Efficiency**. The total gain is distributed as \(\sum_{k=0}^{m}\phi_{k}=f(\mathbf{x})\). **Symmetry**. If two players with numbers \(i\) and \(j\) make equal contributions, i.e., \(f(S\cup\{i\})=f(S\cup\{j\})\) for all subsets \(S\) which contain neither \(i\) nor \(j\), then \(\phi_{i}=\phi_{j}\). **Dummy**. If a player makes zero contributions, i.e., \(f(S\cup\{j\})=f(S)\) for a player \(j\) and all \(S\subseteq N\setminus\{j\}\), then \(\phi_{j}=0\). **Linearity**. A linear combination of multiple games \(f_{1},\ldots,f_{n}\), represented as \(f(S)=\sum_{k=1}^{n}c_{k}f_{k}(S)\), has gains derived from \(f\): \(\phi_{i}(f)=\sum_{k=1}^{m}c_{k}\phi_{i}(f_{k})\) for every \(i\). Let us consider a machine learning problem. Suppose that there is a dataset \(\{(\mathbf{x}_{1},y_{1}),\)\(\ldots,(\mathbf{x}_{n},y_{n})\}\) of \(n\) points \((\mathbf{x}_{i},y_{i})\), where \(\mathbf{x}_{i}\in\mathcal{X}\subset\mathbb{R}^{m}\) is a feature vector consisting of \(m\) features, \(y_{i}\) is the observed output for the feature vector \(\mathbf{x}_{i}\) such that \(y_{i}\in\mathbb{R}\) in the regression problem and \(y_{i}\in\{1,2,\ldots,T\}\) in the classification problem with \(T\) classes. If a task is to interpret or to explain the prediction from the model \(f(\mathbf{x}^{*})\) at a local feature vector \(\mathbf{x}^{*}\), then the prediction \(f(\mathbf{x}^{*})\) can be represented by using Shapley values as follows [10; 11]:  
\[f(\mathbf{x}^{*})=\phi_{0}+\sum_{j=0}^{m}\phi_{j}^{*}, \tag{3}\]  
where \(\phi_{0}=\mathbb{E}[f(\mathbf{x})]\), \(\phi_{j}^{*}\) is the value \(\phi_{j}\) for the prediction \(\mathbf{x}=\mathbf{x}^{*}\). The above implies that the Shapley values explain the difference between the prediction \(f(\mathbf{x}^{*})\) and the global average prediction. One of the crucial questions for implementing the SHAP method is how to remove features from subset \(N\setminus S\), i.e., how to fill input features from subset \(N\setminus S\) in order to obtain predictions \(f(S)\) of the black-box model. A detailed description of the various ways for removing features is presented by [13]. One of the ways is simply by setting the removed features to zero [56; 57] or by setting them to user-defined default values [9]. According to this way, features are often replaced with their mean values. Another way removes a feature by replacing them with a sample from a conditional generative model [58]. In the LIME method for tabular data, features are replaced with independent draws from specific distributions [13] such that each distribution depends on the original feature values.