Model Evaluation and Parameter Tuning | Towards Data Science

In the previous article, we built an ANN to solve a binary classification problem. If you remember, one question left for homework is why we used [[]] for new customer data. The answer is we need to put a customer data in a horizontal vector, not a vertical vector because all observations in the input data are in lines not in columns. Hope you got it right😎.
This article focuses on cross-validation and grid search for model optimization, split into 2 parts:
Model evaluationParameter tuningSummary
Model evaluation
Parameter tuning
Summary
Now let’s begin the journey 🏃‍♀️🏃‍♂️.
Model evaluation
Model evaluation
You may wonder why we have to spend efforts on model evaluation 🤔? The problem is if we re-run the ANN, every time the model produces different accuracy not only on the train set and the test set. So evaluating the model performance on one single test is not the most relevant way.
A typical method is to use K-fold cross-validation. Figure 1 shows how it works.
We split the training set into K folds (e.g. K=10). Then train the model on 9 folds and test it on the last remaining fold. With 10 folds, we make 10 different combinations with 9 training sets and 1 test set, and train/test the model 10 times. After that, we take an average of 10 evaluations and compute the standard deviation. With that, we can determine which category of the model belongs to as in Figure 2.
To implement K-fold cross-validation, we use a scikit_learn wrapper in Keras: KerasClassifier. Specifically, we use Keras to build the model and use scikit_learn for cross-validation. First thing is to build a function for the model architecture as the function is a required argument for the Keras wrapper. As you notice below, this is the same ANN architecture as we built before.
from keras.wrappers.scikit_learn import KerasClassifierfrom sklearn.model_selection import cross_val_scorefrom keras.models import Sequentialfrom keras.layers import Densedef build_classifier(): classifier = Sequential() classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’, input_dim = 11)) classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’)) classifier.add(Dense(units = 1, kernel_initializer = ‘uniform’, activation = ‘sigmoid’)) classifier.compile(optimizer = ‘adam’, loss = ‘binary_crossentropy’, metrics = [‘accuracy’]) return classifier
With the classifier built using the above function, we create a KerasClassifier object. Below we specify the batch size of 10 and the number of epochs of 100 which model needs to be trained on.
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 100)
Now, let’s apply K-fold cross-validation on the classifier by using cross_val_score() method. This function returns a list of training accuracy. Parameter cv is the fold number we use for cross-validation. Here the classifier will be trained on 10 different training sets, split from the initial training set.
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10)mean = accuracies.mean()std = accuracies.std()
Here, cross-validation may take a while. In the end, we get an accuracy of 10 evaluation as shown in Figure 2. The average accuracy is 0.843 and the standard deviation is 1.60% 🤪.
2. Parameter tuning
With reliable model accuracy, let’s try to improve it using two techniques.
2.1 Dropout regularization
One technique we have not mentioned is dropout regularization. This is the solution for over-fitting which is related to high variance. How does dropout work? At each training iteration, some neurons are randomly disabled to prevent them from depending on each other. By overwriting these neurons, neural networks retain a different configuration of neurons each time, helping the neural networks learn independent correlations of the data. This prevents the neurons from over-learn.
Let’s implement it using Keras. Basically, we add a drop out layer after each hidden layer. Note p=0.1 means 10% of neurons will be disabled each iteration.
from keras.layers import Dropoutclassifier.add(Dense(output_dim = 6, init = ‘uniform’, activation = ‘relu’, input_dim = 11))#add dropout layerclassifier.add(Dropout(p =0.1))classifier.add(Dense(output_dim = 6, init = ‘uniform’, activation = ‘relu’))#add dropout layerclassifier.add(Dropout(p =0.1))#add output layerclassifier.add(Dense(output_dim = 1, init = ‘uniform’, activation = ‘sigmoid’))
2.2 Parameter tuning
Neural networks have a few hyper-parameters, such as the number of epochs, batch size, and learning rate. Parameter tuning is about finding the best parameter for the model. Here we use grid search to test different combinations of the parameters.
To implement it, we use a scikit_learn wrapper in Keras: KerasClassifier to wrap the neural network. Then create a grid search object and apply parameter tuning on the wrapped classifier. First, build the classifier function as below.
def build_classifier(optimizer): classifier = Sequential() classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’, input_dim = 11)) classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’)) classifier.add(Dense(units = 1, kernel_initializer = ‘uniform’, activation = ‘sigmoid’)) classifier.compile(optimizer = optimizer, loss = ‘binary_crossentropy’, metrics = [‘accuracy’]) return classifierclassifier = KerasClassifier(build_fn = build_classifier)
Note above, the number of epoch and batch size is not specified, because they are the parameters we plan to tune, which will be specified in the grid search object below.
Let’s create a dictionary for parameters that contain values we want the model to try on.
parameters = {‘batch_size’: [25, 32], ‘nb_epoch’: [100, 500], ‘optimizer’: [‘adam’, ‘rmsprop’]}
If you notice above, we have an argument optimizer for build_classifier() function. This argument gives a way to tune the optimizer. To implement grid search, we first create an object of GridSearchCV class with the classifier and parameters.
grid_search = GridSearchCV(estimator=classifier, param_grid =parameters, scoring = ‘accuracy, cv = 10’)
Finally, let’s fit ANN on the training set while running grid search to find optimal parameters.
grid_search = grid_search.fit(X_train, y_train)
What we are most interested in is the best parameter which produces the highest accuracy. So with below:
best_parameters = grid_search.best_params_best_accuracy = grid_search.best_score_
Finally, after a few hours, we got an improved accuracy of 0.849 ✨✨. The optimal parameter is batch size 25, epoch number 500, and optimizer Adam as shown in Figure 3.
3. Summary
In a summary, through cross-validation, we found the model accuracy is about 0.843. By using random dropout regularization and grid search, we improved the model accuracy to 0.849. Generally, Grid Search can be performed in 2 steps with a step first to find the rough range and another small step to refine the best range.
Great! That’s all! ✨✨If you need some extra, visit my Github page. (FYI, the repos is actively maintained 💕💕)