Churn Prediction using Neural Networks and ML models | by Devarsh Raval | Towards Data Science

This story is a walk-through of a notebook I uploaded on Kaggle. Originally, it only used machine learning models and since then I have added a couple of basic neural network models. The churn prediction topic has been extensively covered by many blogs on Medium and notebooks on Kaggle, however, there are very few using neural networks. The application of neural networks to structured data in itself is seldom covered in the literature. I learned neural networks through the deeplearning.ai specialization on Coursera and the documentation of Tensorflow with Keras.
Customer attrition or customer churn occurs when customers or subscribers stop doing business with a company or service. Customer churn is a critical metric because it is much more cost effective to retain existing customers than it is to acquire new customers as it saves cost of sales and marketing. Customer retention is more cost-effective as you’ve already earned the trust and loyalty of existing customers.
There are various ways to calculate this metric as churn rate may represent the total number of customers lost, the percentage of customers lost compared to the company’s total customer count, the value of recurring business lost, or the percent of recurring value lost. However, in this dataset, it is defined as a binary variable for each customer and calculating the rate is not the objective. Thus the objective here is to identify and quantify the factors which influence churn rate.
This is a fairly easy and beginner level project with fewer variables. It is not a useful application for neural networks as number of training examples are comparatively less but it is easy to understand neural networks using this.
The data cleaning steps are skipped here. Missing values were only minute and found in Total Charges column and thus dropped. No features were dropped owing to multi-collinearity as only few features are present.
The first step in data analysis is familiarizing yourself with the data variables, features and the target. This dataset contains 20 features and one target variable. The customer ID feature is a string identification, thus not useful for prediction.
In the categorical features, some features are binary and some have exactly 3 unique values. On examining, it is noted that only Contract and Internet service have a different number of unique values in the categorical features. The ‘No internet service’ class could be assigned to ‘No’ as in some notebooks on Kaggle. However, dummy variables seem better encoding option instead as in former case there will be loss of data that the customer has chosen not to opt for a service despite having internet service. In case the number of features were larger, label encoding or mapping would be considered as then the one hot encoding would become a large sparse matrix.
It is important to check the distribution of these features provided in our data to check for biases if the feature values are impartially distributed. A function such as below is used to plot the distributions.
def srt_dist(df=df,cols=cat_feats):    fig, axes = plt.subplots(8, 2,squeeze=True)    axes = axes.flatten()    for i, j in zip(cols, axes):        (df[i].value_counts()*100.0 /len(df)).plot.pie(autopct='%.1f%%',figsize =(10,37), fontsize =15,ax=j )                                                                              j.yaxis.label.set_size(15)srt_dist()
It is observed that very few are senior citizens, only 30% have dependents, and only 10% have no phone service. Thus, the correlations drawn from these variables can be doubted.
Naturally, month to month contract customers tenure lower than customer with two year contracts.
Thus, the target variable has 73 % instances of ‘no-churn’. The machine learning models will be skewed because of this and will not perform as good on unseen data with more instances of ‘churn’.
One method to counteract this imbalance of classes is using stratified cross validation, which makes folds with uniform proportions of the instances.
sns.pairplot(df,vars = ['tenure','MonthlyCharges','TotalCharges'], hue="Churn")
From the top row middle plot, it is observed that customers having lower tenure and higher monthly charges are tend to churn, with almost a linear relationship. It can be noted that as shown in the total charges distribution, the neural network model also indicates that customers with lower total charges churn. However this might be due to customers with higher tenure having more total charges over time.
Also as you can see below; having month-to-month contract and fiber obtic internet have a really huge effect on churn probability.
cat_feats=['gender', 'SeniorCitizen', 'Partner', 'Dependents','PhoneService','MultipleLines', 'InternetService', 'OnlineSecurity', 'OnlineBackup','DeviceProtection', 'TechSupport', 'StreamingTV', 'StreamingMovies','Contract', 'PaperlessBilling', 'PaymentMethod'] # As formed in notebook in upper blocksfig,axes = plt.subplots(16)axes = axes.flatten()for i, j in zip(cat_feats, axes):sortd = df.groupby([i])['Churn'].median().sort_values(ascending=False)    j=sns.catplot(x=i,                y='Churn',                data=df,                kind='bar')    j.set_ylabels("Churn Probability")
It is observed probability of churn is high for highly correlated features such as No partner or dependent, no tech support, month to month contract, etc.
The dataset is scaled according to MinMax scaler with range of 0 to 1 and the training set is the first 3993 observations according to the assignment.
The below function was used for stratified cross validation.
def stratified_cv(X, y, clf_class, shuffle=True,  **kwargs):    stratified_k_fold = StratifiedKFold().split(X,y)    y_pred = y.copy()    for ii, jj in stratified_k_fold:         Xtrain, Xtest = X.iloc[ii], X.iloc[jj]        ytrain = y.iloc[ii]        clf = clf_class(**kwargs)        clf.fit(X_train,y_train)        y_pred.iloc[jj] = clf.predict(Xtest)    return y_predprint('Gradient Boosting Classifier:\n {}\n'.format(    metrics.classification_report(y, stratified_cv(X, y,                                                  ensemble.GradientBoostingClassifier))))                                            print('Support vector machine(SVM):\n {}\n'.format(    metrics.classification_report(y, stratified_cv(X, y, svm.SVC))))print('Random Forest Classifier:\n {}\n'.format(       metrics.classification_report(y, stratified_cv(X, y,                                                  ensemble.RandomForestClassifier))))print('K Nearest Neighbor Classifier:\n {}\n'.format(       metrics.classification_report(y, stratified_cv(X, y,                                                  neighbors.KNeighborsClassifier,n_neighbors=11))))print('Logistic Regression:\n {}\n'.format(       metrics.classification_report(y, stratified_cv(X, y,                                            linear_model.LogisticRegression))))print('XGBoost Classifier:\n {}\n'.format(     metrics.classification_report(y, stratified_cv(X, y,        XGBClassifier))))Code used for classification reports of ML model
Hyper parameter tuning of ML models
For Random Forest, the number of estimators and max features per tree is tuned:
# Tuning Random Forestfrom sklearn.ensemble import RandomForestClassifier# Create param grid.   param_rf=[{'n_estimators' : list(range(10,150,15)),            'max_features' : list(range(6,32,5))}]# Create grid search objectclf = RandomizedSearchCV(RandomForestClassifier(), param_distributions = param_rf, n_iter=50, cv = 5, refit=True,verbose=1, n_jobs=-1,)# Fit on databest_clf = clf.fit(X, y)print(best_clf.best_params_)best_clf.best_score_
Out[]:
{'n_estimators': 130, 'max_features': 6}0.78967497404260
For logistic regression, the inverse regularization parameter C is tuned:
# Tuning Logistic Regressionfrom sklearn.linear_model import LogisticRegressionparam_grid = [    {'penalty' : ['l1', 'l2'],    'C' : np.logspace(-5, 5, 20),    'solver' : ['liblinear'] }]clf = RandomizedSearchCV(LogisticRegression(), param_distributions = param_grid, n_iter=20, cv = 5, refit=True,verbose=1, n_jobs=-1,)# Fit on databest_clf = clf.fit(X, y)print(best_clf.best_params_)best_clf.best_score_
Out[]:
{'solver': 'liblinear', 'penalty': 'l2', 'C': 8858.667904100832}0.8043221203472578
For neural networks, both types of modelling, the pre-made estimators and Keras Sequential models are used. Additionally, most references I came across are on Convolutional Neural Networks and Image Classification for hypertuning pre-made estimator. The Keras models are hypertuned for learning rate and number of layers. The hyper parameter tuned model shows similar performance as the dataset is smaller than usual neural network applications.
Here only the Keras models are shown to make the blog succinct.
A 64–8–1 dense layered model with a decaying learning rate of batch size 32 is used. L2 regularization and dropout for each layer is also used.
# Model 1nn_model = Sequential()nn_model.add(Dense(64,kernel_regularizer=tf.keras.regularizers.l2(0.001), input_dim=46, activation='relu' ))nn_model.add(Dropout(rate=0.2))nn_model.add(Dense(8,kernel_regularizer=tf.keras.regularizers.l2(0.001),activation='relu'))nn_model.add(Dropout(rate=0.1))nn_model.add(Dense(1, activation='sigmoid'))lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay( 0.001,      decay_steps=(X_train.shape[0]/32)*50,      decay_rate=1,      staircase=False)#This time decay means for every 50 epochs the learning rate will be half of 0.001 valuedef get_optimizer():    return tf.keras.optimizers.Adam(lr_schedule)def get_callbacks():    return [tf.keras.callbacks.EarlyStopping(monitor='val_accuracy',patience=70,restore_best_weights=True)]nn_model.compile(loss = "binary_crossentropy",                   optimizer = get_optimizer(),                   metrics=['accuracy'])    history = nn_model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=150, batch_size=32,   callbacks=get_callbacks(),verbose=0)plt.plot(history.history['accuracy']) plt.plot(history.history['val_accuracy']) plt.title('model accuracy') plt.ylabel('accuracy')plt.xlabel('epoch') plt.legend(['train', 'test'], loc='upper left') plt.show()
Thus, as the batch size is small, the model converges sooner around 20 epochs. The model accuracy on test set is 80.72%.
Evaluation
yprednn=nn_model.predict(X_test)yprednn=yprednn.round()print('Neural Network:\n {}\n'.format(    metrics.classification_report(yprednn, y_test)))nn_conf_matrix=metrics.confusion_matrix(yprednn,y_test)conf_mat_nn = pd.DataFrame(nn_conf_matrix,     columns=["Predicted NO", "Predicted YES"],     index=["Actual NO", "Actual YES"])print(conf_mat_nn)
Out[]:
Neural Network:               precision    recall  f1-score   support         0.0       0.92      0.84      0.87      2443         1.0       0.51      0.69      0.58       596    accuracy                           0.81      3039   macro avg       0.71      0.76      0.73      3039weighted avg       0.84      0.81      0.82      3039Confusion Matrix :            Predicted NO  Predicted YESActual NO           2042            401Actual YES           185            411
Hyper parameter tuning with Keras
The documentation on Keras tuner explains this very well. Here the number of hidden units, number of neurons in the hidden layers, learning rate and drop out rates are hypertuned.
According to Andrew Ng’s course, learning rate is by far most important followed by momentum beta, mini batch size and number of hidden units.
from tensorflow import kerasfrom tensorflow.keras import layersfrom kerastuner.tuners import RandomSearchimport IPythonimport kerastuner as ktdef build_model(hp):    inputs = tf.keras.Input(46,)    x = inputs    for i in range(hp.Int('num_layers', 1,3)):        x =  tf.keras.layers.Dense(units=hp.Int('units_' + str(i),32,256, step=32, default=64),             kernel_regularizer=tf.keras.regularizers.l2(0.001))(x)        x = tf.keras.layers.BatchNormalization()(x)        x = tf.keras.layers.ReLU()(x)    x = tf.keras.layers.Dense(      hp.Int('hidden_size', 4,64, step=4, default=8),             kernel_regularizer=tf.keras.regularizers.l2(0.001),             activation='relu')(x)    x = tf.keras.layers.Dropout(      hp.Float('dropout', 0, 0.5, step=0.1, default=0.5))(x)    outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)model = tf.keras.Model(inputs, outputs)    model.compile(    optimizer=tf.keras.optimizers.Adam(      hp.Float('learning_rate', 1e-3,1e-1, sampling='log')),    loss="binary_crossentropy",     metrics=['accuracy'])    return modeltuner = RandomSearch(    build_model,    objective='val_accuracy',    max_trials=10,    executions_per_trial=1)batch_size=32tuner.search(X_train, y_train,                     epochs=100,batch_size=batch_size,                     validation_data=(X_test,y_test),                     callbacks= [tf.keras.callbacks.EarlyStopping(monitor='val_accuracy',                                                               patience=40,restore_best_weights=True)],verbose=False)best_hp = tuner.get_best_hyperparameters()[0] best_hp.values
Out[]:
{'num_layers': 3, 'units_0': 96, 'hidden_size': 52, 'dropout': 0.5, 'learning_rate': 0.0075386035028952945, 'units_1': 64, 'units_2': 96}
The hypertuning suggests a 5 layer model with units of first three as shown in output.
The plot shows how a complex model can have large variance. The performance of the model was similar but slightly lower owing to the complexity.
Hypertuning may not be useful for small datasets to the extent as larger ones. However, in the Keras documentation under section of overfit and underfit, it is concluded that accuracy improves as capacity of the neural network is ‘small’. Thus, as this is a similarly sized dataset, we can also use a network of 3 hidden layers with 64 to 128 hidden units at most.
Testing different batch sizes: It results in slightly lower performances with both lesser and greater sizes than 32. Thus this code block may be skipped but can be found in the notebook. The batch sizes normally used for small and medium sized datasets with 10e3 observations are 8, 32, 64, 128. The multiple of 8 is so that the batch size fits in the memory and runs faster.
The confusion matrix and ROC curve give the sense of true positive negative accuracy, however it is precision-recall curve that gives sense of accuracy in imbalanced dataset. In this dataset there are more negative instances than positive and thus Precision-Recall curve shows real performance.
The ROC can be overly optimistic as it will be more if model predicts negative instances properly but fails on positive instances whereas the Precision-Recall curve is based on positive instances.
The confusion matrix values are shown as percentages as neural network model uses one set validation than the CV. The XGBoost performance was very similar to Random Forest and thus not shown here.
1. Random Forest performance
rf_conf_matrix  = metrics.confusion_matrix(y, stratified_cv(X, y, ensemble.RandomForestClassifier,n_estimators=113))conf_mat_rf = pd.DataFrame(rf_conf_matrix,     columns=["Predicted NO", "Predicted YES"],     index=["Actual NO", "Actual YES"])print((conf_mat_rf/7032)*100)cv=StratifiedKFold(n_splits=6)classifier=RandomForestClassifier(n_estimators=113)from sklearn.metrics import aucfrom sklearn.metrics import plot_roc_curvefrom sklearn.model_selection import StratifiedKFoldtprs=[]aucs=[]mean_fpr=np.linspace(0,1,100)fig,ax=plt.subplots()for i,(train,test) in enumerate(cv.split(X,y)):    classifier.fit(X.iloc[train],y.iloc[train])viz=plot_roc_curve(classifier,X.iloc[test],y.iloc[test],name='ROC fold {}'.format(i),alpha=0.3,lw=1,ax=ax)    interp_tpr = np.interp(mean_fpr, viz.fpr, viz.tpr)    interp_tpr[0] = 0.0    tprs.append(interp_tpr)    aucs.append(viz.roc_auc)ax.plot([0, 1], [0, 1], linestyle='--', lw=2, color='r',        label='Chance', alpha=.8)mean_tpr = np.mean(tprs, axis=0)mean_tpr[-1] = 1.0mean_auc = auc(mean_fpr, mean_tpr)std_auc = np.std(aucs)ax.plot(mean_fpr, mean_tpr, color='b',label=r'Mean ROC (AUC = %0.2f $\pm$ %0.2f)' % (mean_auc, std_auc),lw=2, alpha=.8)std_tpr = np.std(tprs, axis=0)tprs_upper = np.minimum(mean_tpr + std_tpr, 1)tprs_lower = np.maximum(mean_tpr - std_tpr, 0)ax.fill_between(mean_fpr, tprs_lower, tprs_upper, color='grey', alpha=.2,label=r'$\pm$ 1 std. dev.')ax.set(xlim=[-0.05, 1.05], ylim=[-0.05, 1.05],       title="Receiver operating characteristic example")ax.legend(loc="lower right")plt.show()# Precision Recall # break code blockrfmodel=RandomForestClassifier(n_estimators= 130, max_features= 6,n_jobs=-1)rfmodel.fit(X_train,y_train)lg_probs = rfmodel.predict_proba(X_test)lg_probs=lg_probs[:,1]yhat = rfmodel.predict(X_test)lr_precision, lr_recall, _ = precision_recall_curve(y_test,lg_probs)lr_f1, lr_auc = f1_score(y_test, yhat), auc(lr_recall, lr_precision)# summarize scoresprint('RF: f1=%.3f auc=%.3f' % (lr_f1, lr_auc))# plot the precision-recall curvesno_skill = len(y_test[y_test==1]) / len(y_test)pyplot.plot([0, 1], [no_skill, no_skill], linestyle='--', label='No Skill')pyplot.plot(lr_recall, lr_precision, marker='.', label='RF')# axis labelspyplot.xlabel('Recall')pyplot.ylabel('Precision')# show the legendpyplot.legend()# show the plotpyplot.show()Out[]:Confusion Matrix: Predicted NO  Predicted YESActual NO      70.036974       3.384528Actual YES      6.143345      20.435154Precision- Recall:RF: f1=0.543 auc=0.603
2.Logistic Regression
The code is similar as above.
Confusion matrix:Predicted NO  Predicted YESActual NO      65.799204       7.622298Actual YES     12.044937      14.533561Precision-Recall AUC:Logistic: f1=0.543 auc=0.640
3. Neural Network model performance
Confusion Matrix:Predicted NO  Predicted YESActual NO      67.193156      13.195130Actual YES      6.087529      13.524186ROC:No Skill: ROC AUC=0.500Neural Network: ROC AUC=0.832Precision-Recall:Neural Network: f1=0.584 auc=0.628
We can see that Random Forest and XGBoost are most accurate models, the Logistic Regression generalizes best and predicts both classes, churn and no churn, equally accurately. Thus, Logistic Regression has the best performance according to the Precision Recall curve. The neural network also performs better on precision recall than RF and XGBoost.
Thus it is Logistic Regression that will predict better if more positive instances, churn labels, are present in unseen data.
1)Feature importance according to Logistic Regression.
weights = pd.Series(lgmodel.coef_[0],index=X.columns.values)print (weights.sort_values(ascending = False)[:20].plot(kind='bar'))
2) Feature importance according to Random Forest
rf =  ensemble.RandomForestClassifier(n_estimators=130,max_features=6, n_jobs=-1)rf.fit(X, y)feature_importance = rf.feature_importances_feat_importances = pd.Series(rf.feature_importances_, index=X.columns)feat_importances = feat_importances.nlargest(19)feat_importances.plot(kind='barh' , figsize=(10,10))
3) Neural Network Feature Importance
As Keras does not provide a feature importance feature in the documentation i have demonstrated two ways of doing it. Reference is a Stack Over Flow answer.
from keras.wrappers.scikit_learn import KerasClassifier, KerasRegressorimport eli5from eli5.sklearn import PermutationImportancedef base_model():    nn_model = Sequential()    nn_model.add(Dense(64,kernel_regularizer=tf.keras.regularizers.l2(0.001),                input_dim=46, activation='relu' ))    nn_model.add(Dropout(rate=0.2))nn_model.add(Dense(8,kernel_regularizer=tf.keras.regularizers.l2(0.001),                    activation='relu'))    nn_model.add(Dropout(rate=0.1))    nn_model.add(Dense(1, activation='sigmoid'))    lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay(                  0.001,                  decay_steps=(X_train.shape[0]/32)*50,                  decay_rate=1,                  staircase=False)    def get_optimizer():        return tf.keras.optimizers.Adam(lr_schedule)    def get_callbacks():        return [            tf.keras.callbacks.EarlyStopping(monitor='val_accuracy',patience=70,restore_best_weights=True)]    nn_model.compile(loss = "binary_crossentropy",                   optimizer = get_optimizer(),                   metrics=['accuracy'])    return nn_modelmy_model = KerasRegressor(build_fn=base_model)    my_model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=150, batch_size=32,                    callbacks= get_callbacks(),verbose=0)perm = PermutationImportance(my_model, random_state=1).fit(X[:500].values,y[:500].values,verbose=False)eli5.show_weights(perm, feature_names = X.columns.tolist())
import shapfrom tensorflow.keras import Sequential# load JS visualization code to notebookshap.initjs()# explain the model's predictions using SHAP# (same syntax works for LightGBM, CatBoost, scikit-learn and spark models)explainer = shap.DeepExplainer(nn_model,data=X[:500].values)shap_values = explainer.shap_values(X.values)# visualize the first prediction's explanation (use matplotlib=True to avoid Javascript)#shap.force_plot(explainer.expected_value, shap_values[0,:], X.iloc[0,:])shap.summary_plot(shap_values, X, plot_type="bar")
It can be seen Total charges is the most important feature and rightfully so. The number one reason customers will ‘churn’ if they find the service to be expensive or unaffordable.Tenure is also important, the customers who have been using the sevice for a long time or have long time contracts, which are cheaper in general, are less likely to churn. It is interesting to observe that Tenure is listed as more important for neural network model.As observed in the EDA, most customers who are on month to month contracts are more likely to churn. Itcan be hypothesized that the reason is to be attributed to personal reasons of customers to have reservations about long term contracts or higher costs per unit time resulting from monthly contracts.As seen in EDA, other important features are online security, electronic payment method, fiber optic internet service, tech support.The features which are not important are gender, dependents, partner, streaming TV, backup and device protection.
It can be seen Total charges is the most important feature and rightfully so. The number one reason customers will ‘churn’ if they find the service to be expensive or unaffordable.
Tenure is also important, the customers who have been using the sevice for a long time or have long time contracts, which are cheaper in general, are less likely to churn. It is interesting to observe that Tenure is listed as more important for neural network model.
As observed in the EDA, most customers who are on month to month contracts are more likely to churn. Itcan be hypothesized that the reason is to be attributed to personal reasons of customers to have reservations about long term contracts or higher costs per unit time resulting from monthly contracts.
As seen in EDA, other important features are online security, electronic payment method, fiber optic internet service, tech support.
The features which are not important are gender, dependents, partner, streaming TV, backup and device protection.
Offers and improving churn rate:
Discounts: As the most important feature is total charges, followed by monthly charges, potential churners identified through the modelling should be offered huge discount on next month or months contract. This covers 80 % of the reasons identified for churning. For this modelling, the False Negative Rate should be minimized or Recall should be maximized so that the discounts are sent to maximum of the potential churners.New contract: A six month or four month contract should be implemented. This will encourage the reserved customers who want shorter contracts and will increase their tenure on the service thus making them less likely to churn.Online Security: This service should be promoted more and offered complimentary/free for trial periods depending on cost to company. The customers who do not have online security are more likely to churn and thus this offer could be combined with the first one mentioned and discount could only be offered on this.Fiber optic: The fiber optic internet is costly and thus should either be promoted to appropriate target audience or better technology can be implemented to cut cost on this service. Ultimately the market research team has to decide the break even point for this service, whether it is profiting as much as the loss of customers it is causing.
Discounts: As the most important feature is total charges, followed by monthly charges, potential churners identified through the modelling should be offered huge discount on next month or months contract. This covers 80 % of the reasons identified for churning. For this modelling, the False Negative Rate should be minimized or Recall should be maximized so that the discounts are sent to maximum of the potential churners.
New contract: A six month or four month contract should be implemented. This will encourage the reserved customers who want shorter contracts and will increase their tenure on the service thus making them less likely to churn.
Online Security: This service should be promoted more and offered complimentary/free for trial periods depending on cost to company. The customers who do not have online security are more likely to churn and thus this offer could be combined with the first one mentioned and discount could only be offered on this.
Fiber optic: The fiber optic internet is costly and thus should either be promoted to appropriate target audience or better technology can be implemented to cut cost on this service. Ultimately the market research team has to decide the break even point for this service, whether it is profiting as much as the loss of customers it is causing.
Another method to quantify the offers to be made is using manually generated features and their effect on the model.
Random Forest vs Neural Networks for Predicting Customer Churn, Abhinav Sagar, Medium, https://towardsdatascience.com/random-forest-vs-neural-networks-for-predicting-customer-churn-691666c7431eTelecom Customer Churn Prediction, Pavan Raj, https://www.kaggle.com/pavanraj159/telecom-customer-churn-predictionChurn Prediction of Telco Customers, Melda Dede, https://www.kaggle.com/meldadede/churn-prediction-of-telco-customersUsing basic neural networks to predict churn, Laurier Mantel, https://www.kaggle.com/lauriermantel/using-basic-neural-networks-to-predict-churnHow to Use ROC Curves and Precision-Recall Curves for Classification in Python, Jason Brownlee, https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/Tensorflow, Keras,sklearn documentationFeature importance of Neural Network,StackOverFlow, https://stackoverflow.com/questions/45361559/feature-importance-chart-in-neural-network-using-keras-in-python#:~:text=It%20most%20easily%20works%20with,using%20it%20is%20very%20straightforward.&text=At%20the%2
Random Forest vs Neural Networks for Predicting Customer Churn, Abhinav Sagar, Medium, https://towardsdatascience.com/random-forest-vs-neural-networks-for-predicting-customer-churn-691666c7431e
Telecom Customer Churn Prediction, Pavan Raj, https://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction
Churn Prediction of Telco Customers, Melda Dede, https://www.kaggle.com/meldadede/churn-prediction-of-telco-customers
Using basic neural networks to predict churn, Laurier Mantel, https://www.kaggle.com/lauriermantel/using-basic-neural-networks-to-predict-churn
How to Use ROC Curves and Precision-Recall Curves for Classification in Python, Jason Brownlee, https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/
Tensorflow, Keras,sklearn documentation
Feature importance of Neural Network,StackOverFlow, https://stackoverflow.com/questions/45361559/feature-importance-chart-in-neural-network-using-keras-in-python#:~:text=It%20most%20easily%20works%20with,using%20it%20is%20very%20straightforward.&text=At%20the%2