ANN Binary Classification | Towards Data Science

This article aims to explain how to create an artificial neural network (ANN) to predict if a banker customer is leaving or not using raw banking customers' data. The article is split into 6 parts as below.
Problem statementData processingModel buildingModel compilingModel fittingModel prediction
Problem statement
Data processing
Model building
Model compiling
Model fitting
Model prediction
Now let’s begin the journey 🏃‍♂️🏃‍♀️!
1. Problem statement
We are tasked by a bank to predict if a customer is likely to leave the bank. 6-month customer banking data is given. The bank plans to use your findings to re-engage with customers who are leaving.
2. Data processing
First, import the data with Pandas using read_csv() as below.
dataset = pd.read_csv(‘Churn_Modelling.csv’)
Figure 1 shows a snippet of the data. The first 13 columns are independent variables about customer ID, name, Credit Score, Geography, Gender, Age, etc. The last column is a dependent variable if the customer left or stayed.
Now, let’s start data processing.
1) Feature selection
The first question: do we need all independent variables to fit the model 🤔? The answer is No. For instance, Row Number, Customer ID, or Surname has no impact on the customer leaving or staying. But customer’s age may have, as young customers possibly tend to leave the bank. So with this logic, we tell which independent variables would impact the outcome. But remember only the neural network can tell which features have a high impact on the outcome.
So independent variable X is:
X = dataset.iloc[:, 3: 13].values
Dependent variable y is:
y = dataset.iloc[:, 13].values
2) Categorical encoding
Neural networks only take numerical values for learning.
Therefore, categorical variables, such as Geography and Gender need to be encoded into numerical ones. Here we use fit_transform() method of LabelEncoder from sklearn. We create two label encoders for each column.
from sklearn.preprocessing import LabelEncoderlabelencoder_X_1 = LabelEncoder()X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])
Notice above, we input index 1, as the index of Geography column in X is 1. After encoding, country German becomes 1, France is 0, Spain is 2. With the same method, encode Gender column as below.
labelencoder_X_2 = LabelEncoder()X[:, 2] = labelencoder_X_2.fit_transform(X[:, 2])
Now, the male is 1 and female become 0 (hopefully Dear Lady won’t feel offended as this is purely random 😊).
3) One-hot encoding
Notice above after encoding, German becomes 1, France is 0, Spain is 2. However, there is no relational order between countries. Namely, Spain is not higher than German, and France is not lower than Spain. So we need to create dummy variables for the categorical variables to remove the numerical relationship introduced by categorical encoding. Here only need to do it for Geography column, since there is only 2 categories for the Gender column.
from sklearn.preprocessing import OneHotEncoderonehotencoder = OneHotEncoder(categorical_features = [1])X = onehotencoder.fit_transform(X).toarray()
Figure 2 is the encoded data. Notice the first 3 columns are one-hot encoded dummy variables for Country German, France, and Spain.
To avoid the dummy data trap, we remove the 1st column because two columns with 0 and 1 are enough to encode 3 countries.
X = X[:, 1:]
4) Data split
Next, split the data into training and test set with the test set taking 20%. We use random_state to make sure splitting remains the same each time.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y,test_size = 0.2, random_state = 0)
5) Feature scaling
Scaling the features is to avoid intensive computation and also avoid one variable dominating the others. For a binary classification problem, no need to scale the dependent variable. But for regression, we need to scale the dependent variables.
Normal methods include Standardization and Normalization as shown in Figure 3. Here we take Standardization.
from sklearn.preprocessing import StandardScalersc = StandardScaler()X_train = sc.fit_transform(X_train)X_test = sc.transform(X_test)
Note, we use the scale set from the training set to transform the test set. With this step, data processing is done. Figure 4 is the test data you should get.
3. Model building
Here we build a Sequential model using Keras. The first step is to initialize the sequential model.
classifier = Sequential()
For this problem, the model is built with 2 dense layers.
#add input layer and first hidden layerclassifier.add(Dense(output_dim = 6, init = ‘uniform’, activation = ‘relu’, input_dim = 11))#add 2nd hidden layerclassifier.add(Dense(output_dim = 6, init = ‘uniform’, activation = ‘relu’))
Note that Output_dim for the hidden layer is a choice of art, depending on your experience. As a tip, based on experimentation, choosing it as the average between the number of nodes in the input layer and the number of nodes in the output layer. A more sensible method is to use parameter tuning which tests different models with different parameters using a technique like k-fold cross-validation. Here, the input dimension is 11 features and the output dimension is 1, thus the output dimension is 6. Here we use uniform distributions to randomize weights between 0 and 1. Use ReLU function for hidden layer activation, based on my experimentation, this is the best. Do feel free to try yours.
Below add the output layer.
classifier.add(Dense(output_dim = 1, init = ‘uniform’, activation = ‘sigmoid’))
For the output layer, we use Sigmoid function to get the probability of the customer leave or stay at the bank. If dealing with multi-classification, use Softmax function.
4. Model compiling
Model compiling is to apply Stochastic Gradient Descent (SGD) on the network.
classifier.compile(optimizer = ‘Adam’, loss =’binary_crossentropy’, metrics = [‘accuracy’])
Here we use Adam (one type of SGD) as an optimizer to find the optimized weights that make neural network most powerful. The loss function that the optimizer is based on is binary cross-entropy. The metrics we use to evaluate the model performance are accuracy.
5. Model fit
Since we use SGD, the batch size is set to 10, indicating neural network updates its weight after 10 observations. Epoch is a round of whole data flow through the network. Here we choose 100.
classifier.fit(X_train, y_train, batch_size = 10, epochs = 100)
Now time to fit the model. After about 35 epochs, model accuracy converges to 0.837. Not bad, right ✨✨?
6. Model prediction
With the model fitted, we test the model on test data. Use a threshold of 0.5, to turn data into True(leaving) and False(stay) data.
y_pred = classifier.predict(X_test)y_pred = (y_pred > 0.5)
Then we use confusion_matrix to investigate the model performance on the test set.
from sklearn.metrics import confusion_matrixcm = confusion_matrix(y_test, y_pred
The accuracy is 0.859, higher than training accuracy, implying over-fitting.
With the above model, the bank can test new customers and get the probability of leaving or stay. Then, the bank can take 10% of customers with the highest possibility and perform deep mining on the customer data to understand why they are likely to leave. That is the purpose of an artificial neural network.
Now one last question: how to make a prediction on new customer data 🤔? For instance, customer data in Figure 5:
As you imagine, we need to follow the same data processing. First, encode the variables. For instance, Geography France is encoded into (0,0) in the dummy variables, Gender male is 1. Following the method, we produce the below array.
new_customer = [[0, 0, 600, 1, 40, 3, 60000, 2, 1, 1, 50000]]
Alright, one homework question: why do we use [[]] to create the array 🤔?
The next step is to scale the variables into the same scale as the training set.
new_customer = sc.transform(sc.transform(new_customer))
With the data ready, the prediction is done by:
new_prediction = classifier.predict(new_customer)new_prediction = (new_prediction > 0.5)
The prediction result is False with a possibility of 0.418%, indicating this customer is not likely to leave the bank 👍👍.
Great! That’s all. If you need some extra, visit my Github repos (FYI, the repos is actively maintained) 💕💕.