Prediction on Customer Churn with Mobile App Behavior Data | by Luke Sun | Towards Data Science

In the previous article, we created a logistic regression model to predict user enrollment using app behavior data. Hopefully, you had good learning there. This post aims to improve your model building skills with new techniques and tricks based on a larger mobile app behavior data. It is split into 7 parts.
1. Business challenge
2. Data processing
3. Model building
4. Model validation
5. Feature analysis
6. Feature selection
7. Conclusion
Now let’s begin the journey 🏃‍♀️🏃‍♂️.
Business challenge
Business challenge
We are tasked by a Fintech firm to analyze mobile app behavior data to identify potential churn customers. The goal is to predict which users are likely to churn, so the firm can focus on re-engaging these users with better products.
2. Data processing
EDA should be performed before data processing. Detailed steps are introduced in this article. The video below shows the final data after EDA.
2.1 One-hot encoding
One-hot encoding is a technique to convert categorical variables into numerical variables. It is needed as the model we are to build cannot read categorical data. One-hot encoding simply creates additional features based on the number of unique categories. Here, specifically,
dataset = pd.get_dummies(dataset)
The above automatically convert all categorical variables to numerical variables. But one drawback of one-hot encoding is the dummy variable trap. It is a scenario in which variables are highly correlated to each other. To avoid the trap, one of the dummy variables has to be dropped. Specifically,
dataset = dataset.drop(columns = [‘housing_na’, ‘zodiac_sign_na’, ‘payment_type_na’])
2.2 Data split
This is to split the data into train and test sets. Specifically,
X_train, X_test, y_train, y_test = train_test_split(dataset.drop(columns = ‘churn’), dataset[‘churn’], test_size = 0.2,random_state = 0)
2.3 Data balancing
There are many ways to combat imbalanced classes, such as changing performance metrics, collecting more data, over-sampling or down-sampling data, etc. Here we use the down-sampling method.
First, let’s investigate the imbalance level of the dependent variable in y_train.
As shown in Fig.1, the dependent variable is slightly imbalanced. To down-sample the data, we take the index of each class, and randomly choose the index of the majority class at a number of minority class in y_train. Then concatenate the index of both classes and down-sample x_train and y_train.
pos_index = y_train[y_train.values == 1].indexneg_index = y_train[y_train.values == 0].indexif len(pos_index) > len(neg_index):    higher = pos_index    lower = neg_indexelse:    higher = neg_index    lower = pos_indexrandom.seed(0)higher = np.random.choice(higher, size=len(lower))lower = np.asarray(lower)new_indexes = np.concatenate((lower, higher))X_train = X_train.loc[new_indexes,]y_train = y_train[new_indexes]
2.4. Feature scaling
Fundamentally, feature scaling is to normalize the range of the variables. This is to avoid any variable having a dominant impact on the model. For a neural network, feature scaling helps gradient descent converge faster than without it.
Here we use standardization to normalize the variables. Specifically,
from sklearn.preprocessing import StandardScalersc_X = StandardScaler()X_train2 = pd.DataFrame(sc_X.fit_transform(X_train))X_test2 = pd.DataFrame(sc_X.transform(X_test))
3. Model building
Specifically,
from sklearn.linear_model import LogisticRegressionclassifier = LogisticRegression(random_state = 0)classifier.fit(X_train, y_train)
Now, let’s test and evaluate the model. Specifically,
y_pred = classifier.predict(X_test)from sklearn.metrics import confusion_matrix, accuracy_score, f1_scorecm = confusion_matrix(y_test, y_pred)accuracy_score(y_test, y_pred)f1_score(y_test, y_pred)
Finally, we got an accuracy of 0.61 and F1 of 0. 61. Not too bad performance.
4. Model validation
With the model trained and tested, one question is how good the model is to generalize to an unknown dataset. We use cross-validation to measure the size of the performance difference between known datasets and unknown datasets. Specifically,
from sklearn.model_selection import cross_val_scoreaccuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10)
With the above, we found 10-fold cross-validation produces an average accuracy of 0.64.5 with a standard deviation of 0.023. This indicates the model can generalize well on an unknown dataset ✨✨.
5. Feature analysis
With 41 features, we built a logistic regression model. But how to know which feature is more important in predicting the dependent variable? Specifically,
pd.concat([pd.DataFrame(X_train.columns, columns = [“features”]), pd.DataFrame(np.transpose(classifier.coef_), columns = [“coef”])],axis = 1)
As shown in Figure 2, we found two features that are very important: purchase_partners and purchase. This indicates a user’s purchase history plays a great role when deciding churn or not. Meanwhile, this indicates that not all variables are important for prediction.
6. Feature selection
Feature selection is a technique to select a subset of the most relevant features for modeling training.
In this application, x_train contains 41 features, but as seen in Figure 2, not all features play important roles. Using feature selection helps to reduce the number of unimportant features and achieve similar performance with less training data. A more detailed explanation of feature selection can be found here.
Here, we use the Recursive Feature Elimination (RFE). It works by fitting the given algorithm, ranking the feature by importance, discarding the least important features, and refitting until a specified number of features is achieved. Specifically,
from sklearn.feature_selection import RFEfrom sklearn.linear_model import LogisticRegressionclassifier = LogisticRegression()rfe = RFE(classifier, 20)rfe = rfe.fit(X_train, y_train)
Note above, we set to select 20 features. Figure 3 shows all selected features.
Great. with the RFE selected features, let’s retrain and test the model.
classifier.fit(X_train[X_train.columns[rfe.support_]], y_train)y_pred = classifier.predict(X_test[X_train.columns[rfe.support_]])
In the end, we got an accuracy of 0.61 and F1 of 0.61. The same performance as the model trained on 41 features 😇😇!
If we apply cross-validation again, we got an average accuracy of 0.647 with a standard deviation of 0.014. Again, very much the same as the previous model.
7. Conclusion
Initially, we trained a logistic regression model with 41 features, achieving an accuracy of 0.645. But using feature selection, we created a light version of the model with only 20 features, with an accuracy of 0.647. Half of the features are of no relevance in deciding the customer’s churn. Well done!
Great! That’s all of this journey! If you need the source code, feel free to visit my Github page 🤞🤞 (FYI, the repos is actively maintained).