How to get a fuller picture of a model’s accuracy | by Allison Stafford | Towards Data Science

Using a single train-test split gives you a single snapshot of the performance of a machine learning model or algorithm. It’s like evaluating a football team (American or otherwise) based on a single game. If you really want to know how well a team performs in general, you’re going to want more than just this snapshot. Likewise, we shouldn’t evaluate our algorithms on a single random split. Luckily, there’s a better way. I bring you k-fold cross validation.
Today, I’m using Gareth James’s classic Advertising data set where we have 200 records of TV, radio, and newspaper advertising spend and the resulting sales figure.
I created a linear regression model to predict sales based on TV*Radio, and validated it on the test data set:
from sklearn.linear_model import LinearRegressionfrom sklearn.model_selection import train_test_splitimport pandas as pdimport numpy as npdf = pd.read_csv(‘data/advertising.csv’) # import datadf[‘TVxradio’] = df[‘TV’] * df[‘radio’] # create featurex = np.array(df[‘TVxradio’]).reshape(-1, 1)y = np.array(df[‘sales’])x_train, x_test, y_train, y_test = train_test_split(     x1, y1, test_size=0.3, random_state=54546)model = LinearRegression()model.fit(x_train, y_train)print(f’model accuracy on test set: {model.score(x_test, y_test)}’)
Results:
model accuracy on test set: 0.9721866564067523
AWESOME!! The model is killing it!
Or is it?
This one score fails to show you the true predictive power of the model. In this situation, you’re judging the football team based on a single game. You don’t know if this is a ‘lucky’ train_test_split. In this case, yes. I cheated. Choosing a new random_state (71464) brings our model accuracy down to 0.80.
There is a better way to optimize a model without being at the whim of your random state. A way that provides insight into the distribution of scores and reduces the risk of overfitting to your testing data set.
On smaller data sets. If you have a huge data set, you can afford to slice it into three representative parts — train, test, final evaluation. Unfortunately, you don’t always have this luxury; your data set might not be large enough to slice into three representative parts. This is where cross validation comes in. K-fold cross validation prevents overfitting to your test data without further reducing the size of your training data set.
To get a less biased (read: optimistic) evaluation of your model. K-fold cross validation reports on the performance of a model on several (k) samples from your training set. This gives you insight into the distribution of accuracy you are likely to see for a model.
When you have enough computing resources/time. K-fold cross validation is more computationally expensive than slicing your data into three parts. It re-fits the model and tests it k-times, for each iteration, as opposed to one time. Thus, It is much more valuable with small data sets where this runtime/computational cost is not significant, and the extra partition would reduce the training data set significantly.
When you run cross_val_score with a cross validation splitting strategy (cv) set to an integer, it does the following:
Randomly slice data set into k “folds” (aka. smaller data sets)Set aside one slice, and train the model on the other slices.Validate the model on the saved set and record the score.Repeat steps two and three withholding each fold one time.
Randomly slice data set into k “folds” (aka. smaller data sets)
Set aside one slice, and train the model on the other slices.
Validate the model on the saved set and record the score.
Repeat steps two and three withholding each fold one time.
You can then use the scores to get an estimate of the mean, standard deviation, and 95% confidence interval for your model’s accuracy.
How to use k-fold cross validation
The basic use of scikit-learn’s cross_val_score includes the following parameters:
Estimator — the model you are using to fit the data
X — the data to fit
y — the target variable (if using supervised learning) [optional]
cv — the number of folds you want [defaults to three]
Note: There is no one ‘right’ number of folds. You want to consider the size of the data set in each fold. Ideally, the folds would be equal-size, especially important in very small data sets — so consider choosing a k-value that evenly divides your data set. Stanford’s Hastie and Tibshirani point to k=5 or k=10 as typical k-values.
Also note: cross_val_score does not automatically shuffle. If you want to run it multiple times (a la bootstrapping), make sure to include a shuffle.
As always, check out the docs and user guide for more information.
Going back to the Advertising data set, with the same single factor linear regression model (tv spend * radio spend), there are only a few changes needed to implement k-fold cross_val_score.
# import necessary python modules and classesfrom sklearn.model_selection import cross_val_scorefrom sklearn.linear_model import LinearRegression # your model herefrom sklearn.model_selection import train_test_splitimport pandas as pdimport numpy as np# import datadf = pd.read_csv('data/advertising.csv')# engineer feature of interestdf['TVxradio'] = df['TV'] * df['radio']# slice datax = np.array(df['TVxradio']).reshape(-1, 1)y = np.array(df['sales'])# train-test splitx_train, x_test, y_train, y_test = train_test_split(     x, y, random_state=0)# select modelmodel = LinearRegression()# k-fold cross validationscores = cross_val_score(model, x_train, y_train, cv=10)print(scores)print(     f"95% CI Accuracy: "     f"{round(scores.mean(), 2)} "     f"(+/- {round(scores.std() * 2, 2)})")# test model on test setmodel.fit(x_train, y_train)print(f'model accuracy on test set: {model.score(x_test, y_test)}')
Results:
[0.87595678, 0.97943611, 0.8740311, 0.93150634, 0.94679878, 0.89219105, 0.91211392, 0.97882041, 0.8539725, 0.95655506]95% CI Accuracy: 0.92 (+/- 0.09)model accuracy on test set: 0.9041657637532045
Here, you get a lot more than the single snapshot: the mean and standard deviation of the accuracy over repeated trials. In this case I printed the approximate 95% confidence interval. Now that’s worth getting excited about.
When using preprocessing, cross validation takes a little more additional code, but still completely doable and worth it for this better understanding of a model’s accuracy.
As usual, when using preprocessing, you should not include your validation data, so that this data is ‘new’ to the model in the testing phase as opposed to leaking in via preprocessing. In order to avoid this while using k-fold cross validation, you’ll want to use a Pipeline to pass these data transformations to cross_val_score in the first parameter. Although this adds some complexity to the code, it is fairly straightforward and is essential to producing a valid assessment of model performance.
In this example, I am using the same Advertising data set with multiple regression. The features include TV spend, radio spend, newspaper spend, and TV spend * radio spend, and the model is predicting the product sales. With the pipeline, fed into the cross_val_score, the function will first split the data into train and validate, and then transform the training data set using the transformers in the order indicated by make_pipeline, in this case using StandardScaler() before fitting to LinearRegression().
# import necessary python modules and classesfrom sklearn.model_selection import cross_val_scorefrom sklearn.linear_model import LinearRegression # your model herefrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import make_pipelinefrom sklearn import preprocessingimport pandas as pdimport numpy as np# import datadf = pd.read_csv('data/advertising.csv')# engineer additional feature of interestdf['TVxradio'] = df['TV'] * df['radio']# slice dataX = np.array(df[['TV', 'radio', 'newspaper', 'TVxradio']])y = np.array(df['sales'])# train-test splitX_train, X_test, y_train, y_test = train_test_split(     X, y, random_state=5)# setup pipeline with preprocessing transformerslr = make_pipeline(preprocessing.StandardScaler(),                   LinearRegression())# k-fold cross validationscores = cross_val_score(lr, X_train, y_train, cv=10)print(scores)print(     f"95% CI Accuracy: "     f"{round(scores.mean(), 2)} "     f"(+/- {round(scores.std() * 2, 2)})")# test model on test setmodel = LinearRegression().fit(X_train, y_train)print(f'model accuracy on test set: {model.score(X_test, y_test)}')
Results:
[0.9859427, 0.97482388, 0.97914505, 0.82607816, 0.97520017, 0.98505002, 0.96018605, 0.973842, 0.984283, 0.8727062]95% CI Accuracy: 0.95 (+/- 0.11)model accuracy on test set: 0.9715483746381636
You see here that the mean model accuracy increases with the addition of these features, and that the 95% accuracy confidence interval also widens. This model’s accuracy varies more with changes in the training dataset as compared to my more simple model above.
K-fold cross validation is an excellent way to provide insight into the true quality of your team, accuracy of your machine learning model, by providing a broader evaluation, as opposed to a single, highly-tuned trial. It allows you to get the most out of your training set, while still saving bias from slipping in via overfitting to the testing set.
Feel free to check out the GitHub repository for this blog.