# Shapley Values as a Strategy for Ensemble Weights Estimation
## 3 Methods
### Weighting Strategies  
We have used five ensemble weight estimation strategies: Equal, Random, Perf, posShape, and maxShape. The purpose of the Equal weight strategy was to determine how well the model performs in an ensemble without any effort to estimate the weights. All ensemble members receive the same values under the Equal weighting approach. In the Random weighting strategy, each model gets its value randomly generated in a range from 0 to 1, which constitutes a baseline to compare against other weighting heuristics. Performance weighting (Perf) was calculated using Function (6), where \(BCE\) measure corresponds to Formula (5) and was calculated on validation data (Figure 4). This corresponds to the reciprocal of \(BCE\):  
\[w_{n}^{Perf}=\frac{1}{BCE} \tag{6}\]  
Figure 3: Decision tree visualization for Bank Marketing dataset, where the darker the color of the node the more target class cases (clients subscribed to the service) of the training data exist there. Shapley value was first introduced as a way to evaluate coalition participants' contribution to an overall coalition in a game theory paper [18]. Since then, Shapley values have been adopted in many research fields such as economics [40] and machine learning [19; 29; 30]. In our proposed weighting strategy, the Shapley transformation of performance-based weight for model n was calculated as follows:  
\[\mathrm{w_{n}^{Shap}(v)=\sum_{G\subseteq N\setminus\{n\}}\frac{|S|!(N-|S|-1)!}{ N!}(v(S\cup\{n\})-v(S))-\emptyset \tag{7}\]  
with reciprocal \(BCE\) corresponding to coalition members' contribution \(v\). \(N\) is the set of classification models and sub-ensemble \(S\) is a subset \(S\subseteq N\). \((S\cup\{n\})-v(S)\) represents a marginal contribution of a single coalition member. The empty coalition \(\emptyset\) value was calculated using Formula (7) with all predictions fixed at 0.5 and tested against ground-truth values of validation data, which in effect provides a performance baseline derived with respect to existing class imbalance. It is well known that a baseline threshold of \(BCE\) in the case of balanced classes is exactly 0.693 and \(BCE\) exceeding this value indicates that a corresponding detector is performing worse than chance [41]. Moreover, this threshold value decreases in case of a stronger class imbalance. Therefore, a need to account for that when using Shapley estimation with a possibility to assign value to an empty coalition arises. The posShape weighting strategy calculated weights based on Formula (7) and models that resulted in negative Shapley values were not included in the ensemble by setting their weight to 0:  
\[f(w)=\left\{ \begin{array}{ll}w,&w\geq 0\\ 0,&w<0\end{array} \right. \tag{8}\]  
where \(w\) is obtained using Formula (8) and posShape transformation of \(w\) is identical to rectified linear unit activation function ReLU [42]. The maxShap weights were calculated in the case of negative Shapley values, which meant the net loss with respect to the overall coalition. For such models, the assumption was made that the classes were switched when training a model or that some labeling noise was present in the training data. To allow such models to be used in a model coalition, we have inverted the model predictions as follows:  
\[f(p)=\left\{ \begin{array}{ll}p,&p\geq 0\\ 1-p,&p<0\end{array} \right. \tag{9}\]  
Figure 4: Double hold-out validation scheme with Zipf-derived data splitting for training ensemble members. where \(p\) is the output from the detection model. This was supposed to enable a model that was trained on mislabeled data to still be useful in the overall coalition to the extent of its contribution if it was due to label noise. The voting-based Shapley weights Roz were calculated using the method described in [19]. This method uses marginal contribution approximation to speed up the calculations and calculates Shapley values at individual data point levels.