Training, Validating and Testing — Successfully Comparing Model Performances | Towards Data Science

If you have already used machine learning algorithms, you likely have encountered functions named like train_test_split() or similar sounding functions that are available in libraries such as scikit-learn, TensorFlow or others. Training a model is the first step in making good predictions, however identifying how well the predictive power is, is a different question. Splitting data is necessary to build a solid basis to train, compare and test your models.
I will give this to you, model selection is allegedly not the most interesting or exciting task, however it is essential for everyone working with data.
Throwing around models and machine learning algorithms is easy and can be done by almost anyone — the challenge lies in doing your analysis right.
Let’s start with the reason why we are doing this: Every dataset contains patterns called real as well as random effects. Let’s assume we are looking at some random dataset. If we created a model that allowed us to predict values, the model is exposed to real and random effects. What we do not know is, which of the data points are random and therefore show a completely unique pattern in this particular dataset, and which data points are “typical” (real effects), hence would look similar in other, new data.
Let’s put it the other way around, if we had a dataset with only perfectly “real” data our model would fit well, and also perform equally well on new data, given the real effects are present in any other dataset too.
But to be real, this is just hypothetical. Data contains random patterns and our model will fit to both, random and real data — and this is exactly why we need to measure accuracy for all models.
Real world datasets will contain both, random and real effects, hence it is unlikely to have a model that is 100% accurate — and even if so, it is very likely to overfit the data the model was trained on. Further, new data points will also contain (entirely new) random effects and is simply unlikely that our model is able to capture these well.
The questions we want to answer are simple ones, “ is my model accurate” and “how accurate is my model?”. Basically there are two entry points to answer this question, do I have a
single model or are thereseveral models
single model or are there
several models
which have to be trained, evaluated and compared among each other. Why is it important to know whether we look at a single or different models? Because depending on this assumption, we need to split our data slightly differently.
In general, cross validation is the preferred way to evaluate the performance of a model and/or tweak hyperparameters, however when dealing with a variety of models, it is often simply convenient to apply different (sub) datasets and see how the models perform.
If you are more interested in cross validation (or Monte Carlo Cross Validation), follow this link:
towardsdatascience.com
Single model case: In order to test our model with regard to its predictive accuracy it seems quite intuitive to split data into a training portion and a test portion, so that the model can be trained on one dataset, but tested on a different, new data portion.
The reason for this test is simple, imagine we used the full dataset to train the model and then use the same data to predict the model’s accuracy. Naturally, we expect the model to perform well, given that all data points are already “known”. This is exactly what the term overfit refers to. What we are interested in is, if the model is also capable of handling data that has just been introduced — the test data.
Training a single model is quite straightforward. We split the data into two datasets:
Training data for the model fittingTesting data for estimating the model’s accuracy
Training data for the model fitting
Testing data for estimating the model’s accuracy
A brief look at the R documentation reveals an example code to split data into train and test — which is the way to go, if we only tested one model. If we had several models to test, the data should be split into three portions — but I will get to this later.
# 0.8 is the size of the training datatrain_index <- sample(1:nrow(adult), 0.8 * nrow(adult))test_index <- setdiff(1:nrow(adult), train_index)# Build X_train, y_train, X_test, y_testX_train <- adult[train_index, -15]  # income = column 15y_train <- adult[train_index, "income"] # income = column 15X_test <- adult[test_index, -15] # income = column 15y_test <- adult[test_index, "income"] # income = column 15
In order to conclude if our trained model has good predictive power, we simply use the trained model and predict the response for the test data. These predictions can then be used to compare with the true response variable. The following demo code illustrates how we measure accuracy in the simple one model case (R like pseudo code):
model <- classifier( y_train ~ X_train ) # y explained by Xpredictions <- predict( model, X_test )RMSE( pred=predictions, obs=y_test ) # predictions vs. Obs
Multiple model case. Now it gets a bit less intuitive. If we created several models — let’s assume we changed a specific hyperparameter a couple of times to see the impact on the models or we used different classification models like SVM and KNN — each of these examples will be considered a different model.
Let’s assume more specifically that there are 10 models, each one is characterized by having different hyperparameters compared to all other models. As an example, models may differ with regard to a larger XYZ value, some a smaller value lambda, etc. — you see my point.
At this point, we trained 10 models on the same training data. Each model will provide its very own accuracy, some will do better some worse than others. The question now is, how can we choose the best model and how can we estimate the “true” accuracy of the model?
Step 1: Choose the right model among each model type
As we currently look at 10 models that have only been introduced to training data, for this reason we need a data portion our models have not seen before — the validation data.
We can use the validation data to calculate accuracy metrics for every single model (each having different hyper/parameters). This allows us to rank the models by type and their best scores and select the best k models we would like to test the overall performance later on. I usually define k=2 or 3.
As an example, if we had 10 models, that have 3 different underlying types (for example 3 KNN models, 4 CART models and 3 Support Vector Machine models), we simply select the best model out of each category.
This concludes training and validation.
Step 2: Measure the true accuracy of your model
Having three models at hand, we must not yet assume that performance on the validation data is the “real” performance of each model. At this point we introduce the last step, accompanied by the test dataset. For each of the k best (top 3) models we use the model to classify the test data and calculate the test error metrics.
This sounds like we’re doing the same thing all over again, just for three models and using the test set — but why? Recall my short monologue about random and real effects. The best model (in terms of highest accuracy) after validation may be good, but there it is also likely that one model may have benefited from being more suitable for the random pattern in the validation data — the random effects — and this, only by chance.
This is not what we want to measure our models on, we do not want to choose a model, assuming it is better, but in reality the model only outperformed simply due to luck. For this reason, the test set introduces another layer that minimizes chances that a model benefits from pure randomness.
This concludes the third step, the final evaluation. We simply pick the best out of the three models. This model was fitted (training data), performed among the top 3 on the validation data and turned out to have the best accuracy on the test data.
What do we do if the observed accuracy is now considerably lower than the accuracy on the validation set?
The simple answer is, nothing. Stick with the model. We have worked through legitimate steps to evaluate and compare all our the models. If the results do eventually show a lower score, then this is what the model performance is on new and unseen data. Do not run this procedure again, e.g. using different data splits, until your final model does better — why? Re-applying this procedure to find a better model under the same setting simply means, that you probably found a model that overfits the data, but may again work less well on new data in the future.
To make this process of selecting the best among several models a bit more tangible, the steps are outlined with a bit more structure and terminology:
Training dataset: the training dataset is used to train the model. This dataset should be the largest portion of the dataset. Model generally perform well when tested against the training dataset, simply because the error is underestimated (“downward bias”).
Validation dataset: Required if several models are compared. The validation dataset is used to validate the models and pick the best among them.
Test dataset: Eventually determines the true performance of the selected models. The test error is likely to be higher than what was observed in the previous steps. This step let’s is select the one final model among all.
As outlined earlier, we do not know which data points are random effects and which are not. There are various ways to split the data, but in general choosing random data points is a good idea to start with — R’s sample or Python’s random.sample functions may be very handy.
This post provided a short guidance through the challenges model selection under a larger number of models. Although, this topic can certainly not be found among the most exciting machine learning lectures it is crucial to follow established procedures to pick and validate data models.
{See you next time}
[1] Photo by invisiblepower on Unsplash— Thanks!