# Shapley Values as a Strategy for Ensemble Weights Estimation
## 3 Methods
### Base Learners  
We have used two types of classifiers as base learners in the experiments: CART (classification and regression tree) and binary logistic regression. These two classifier types were chosen because they are generally widely known and are commonly applied for machine learning classification tasks, such as, for example, logistic regression in the banking industry [34]. Other classifiers [35; 36] would be also suitable, as long as they can predict class probabilities. In our research, these learners are used to train either a single model on a full training dataset meaning that the training dataset was not divided into smaller batches (such model will be called a monolith) or multiple models that are trained on a part of training data and are further combined into ensemble.  
The CART classifier represents a traditional decision tree [37]. The leaf nodes contain predicted values of the class of the input vector and the intermediate nodes contain binary decisions, each rejecting a subset of potential sequential points and classification choices. The tree is built by applying a greedy split algorithm over the learning data by minimizing the Gini impurity (1); more specifically, the best split gives a minimal weighted average of class impurity in child nodes. The Gini impurity is calculated as:  
\[\text{Gini}(k)=\sum_{i\subseteq C}\rho_{i,k}(1-\rho_{i,k})=1-\sum_{i\subseteq C }\rho_{i,k}^{2} \tag{1}\]  
where \(C\) is a list of all classes, \(k\) corresponds to a specific category (child node) after splitting and \(\rho_{i,k}\) is a probability of category \(k\) having class \(i\). The main advantage of decision trees over other classifiers is that the \"reasoning\" behind the classification output can always be described in human-friendly terms. An illustrative example of the Bank Marketing Dataset used in our work is provided in Figure 3. Our approach differs from the random forest ensemble learning method by not using bootstrap aggregation and combining classifiers trained on datasets with varying sizes respecting data size distribution patterns defined in Section 3.3.  
Binary logistic regression is also one of the traditional approaches [38] to the problem of binary classification. It models the linear expectation of vector \(X=\{x_{1},x_{2},\ldots,x_{n}\}\) belonging to class \(A\) as \(E(A|X)\), where:  
\[E(A|X)=\frac{e^{y}}{1+e^{y}} \tag{2}\]  
\[y=\beta_{0}+\beta_{1}*x_{1}+\beta_{2}*x_{2}+\cdots+\beta_{n}*x_{n} \tag{3}\]  
Figure 1: Workflow of the proposed ensemble weight calculation process.  
Figure 2: Overview of the proposed ensemble weight calculation strategy.  
The logistic regression model fitting weighting coefficients \(\{\beta_{0},\beta_{1},\beta_{2},\cdots,\beta_{n}\}\) in a manner that best satisfies the logistic loss (log-loss) function:  
\[cost(P(A|X),X)=\left\{\begin{array}{l}-log(P(A|X)),\textit{if }X\textit{ is in class }A\\ -log(1-P(A|X)),\textit{if }X\textit{ is not in class }A\end{array}\right. \tag{4}\]  
The feature selection step and regularization were omitted from the logistic regression model, as the goal of this research was to compare existing weighting strategies to two novel approaches.  
The performance of models for detection tasks in our experiments was evaluated using binary cross-entropy metric [39]. Cross-entropy is one of the more popular loss functions used for classification problems. It quantifies the difference between two probability distributions--the predicted output distribution and the real output distribution. Binary cross-entropy (\(BCE\)) can be seen as a generalization of log-loss for multi-class cases and is defined as:  
\[BCE\Big{(}y,\hat{y}\Big{)}=-(y\cdot log(\hat{y})+(1-y)\cdot log(1-\hat{y})) \tag{5}\]  
The \(y\) represents the ground-truth class label and \(\hat{y}\) corresponds to the model's prediction of class probability.