# Intelligent Decision Forest Models for Customer Churn Prediction
## 1 Introduction
Consequently, CCP exhibits the class imbalance problem since there are more instances of non-churners (majority) than churners (minority). It is imperative to develop effective ML-based CCP models to accommodate the class imbalance problem [15; 17]. This study pays close attention to class imbalance while developing ML-based CCP models with high prediction performance. Intelligent decision forest (DF) models such as Logistic Model Tree (LMT), Random Forest (RF), Functional Tree (FT), and enhanced variations of LMT, RF, and FT based on weighted soft voting and stacking ensembles are utilized for CCP. DF models generate extremely efficient decision trees (DTs) utilizing the prowess of all attributes in a dataset based on the diversity of tree models and predictive performance. This characteristic of DF is contrary to conventional DTs that utilize only a portion of the attributes [22]. LMT as a DF method combines logistic regression (LR), and DT induction approaches into a distinct model component. The crux of LMT is introducing an LR function at the leaf nodes by gradually improving superior leaf nodes on the tree. Similarly, FT results from the functional induction of multivariate DTs and discriminant functions. That is, FT uses positive induction to hybridize a DT with a linear function, creating a DT with multivariate decision nodes and leaf nodes that employ discriminant functions to make predictions. Also, RF as a DF model is the collection of unrelated trees working together as a group (forest). RF creates subsets of data attributes that are used to create trees and then subsequently merged. Furthermore, enhanced DF models based on weighted soft voting and stacking ensemble techniques are proposed. In this context, the weighted soft voting ensemble considers the probability value of each DF model in predicting the appropriate class label, while the stacking ensemble method takes advantage of the efficacy of multiple DF models. Our choice of weighted soft voting and stacking ensemble methods over other ensemble methods (hard voting, multischeme, dagging, bagging, and boosting) is based on their ability to handle uncertainty in the generated probability point and final decision process. These ensemble methods are suggested to augment the prediction performances of DF models to generate robust and generalizable CCP models. In addition, the synthetic minority over-sampling technique (SMOTE) is deployed as a viable solution to resolve the latent class imbalance problem in customer churn datasets. The primary goal of this study is to investigate the effectiveness of DF models (LMT, RF, FT, and their enhanced ensemble variants) for CCP with the occurrence of the class imbalance problem. The following is a summary of the primary accomplishment of this study:  
1. To empirically examine the effectiveness of DF models (LMT, RF, FT) on both balanced and imbalanced CCP datasets. 2. To develop enhanced ensemble variants of DF models (LMT, RF, FT) based on weighted soft voting and stacking ensemble methods. 3. To empirically evaluate and compare DF models (LMT, RF, FT) and their enhanced ensemble variants with existing CCP models. Additionally, the following research questions (RQs) are being addressed in this study:  
1. How efficient are the investigated DF models (LMT, FT, and RF) in CCP compared with prominent ML classifiers? 2. How efficient are the DF models' enhanced ensemble variations in CCP? 3. How do the suggested DF models and their ensemble variations compare to existing state-of-the-art CCP solutions? The rest of the paper is structured as follows. Section 2 provides an in-depth examination of current CCP solutions. Section 3 describes the experimental framework and focuses on the proposed solutions. Section 4 discusses the research observations in depth, and Section 5 ends the study.