Model Validation in Python. Applying Train/Test Split, KFolds, and... | by Sadrach Pierre, Ph.D. | Towards Data Science

Scikit-learn is an open source machine learning library that provides tools for building, training and testing models. The model selection module has many functions that are useful for model testing and validation. In this post, we will discuss some of the important model selection functions in scikit-learn.
Let’s get started!
For our purposes, we will be working with The Wines Reviews data set, which can be found here.
To start, let’s read our data into a Pandas data frame:
import pandas as pddf = pd.read_csv("winemag-data-130k-v2.csv")
Next, let’s print the first five rows of data:
print(df.head())
Let’s consider the task of predicting whether wine price is more expensive than $50 based on the variety, winery, country and review points. We can build a random forest classifier to perform this task. First, let’sconvert the categorical features into categorical codes that can be handled by random forests:
df['country_cat'.format(i)] = df['country'].astype('category').copy()df['country_cat'.format(i)] = df['country_cat'.format(i)].cat.codesdf['winery_cat'.format(i)] = df['winery'].astype('category').copy()df['winery_cat'.format(i)] = df['winery_cat'.format(i)].cat.codesdf['variety_cat'.format(i)] = df['variety'].astype('category').copy()df['variety_cat'.format(i)] = df['variety_cat'.format(i)].cat.codes
Let’s also impute missing values. We won’t do any fancy imputing here but check out Predicting Missing Values with Python for a more reliable method of imputation. Here, let’s replace missing values with 0:
df.fillna(0, inplace=True)
Next let’s assign binary labels to price values based on whether they are greater than or equal to $50. Wines more than $50 have a label of ‘1’, otherwise ‘0':
df['price_class']=np.where(df['price']>=50,1,0)
Now, let’s define our features and targets:
X = np.array(df[['country_cat', 'winery_cat', 'variety_cat', 'points']])y = np.array(df['price_class'])
The first function we’ll discuss is the ‘train_test_split()’ method. This function split arrays or matrices into random train and test subsets. Let’s import this function from scikit-learn:
from sklearn.model_selection import train_test_split
To split our function for training and testing we do the following:
X_train, X_test, y_train, y_test = train_test_split(X,y)
The default test size is 0.25. You can also change the size of the train/test split. For example, to change the test size to 0.3 we do the following:
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3)
Next, we can train and test our random forest model:
from sklearn.ensemble import RandomForestClassifiermodel = RandomForestClassifier()model.fit(X_train, y_train)y_pred = model.predict_proba(X_test)
Let’ evaluate the area under the receiver operating characteristic curve (AUROC):
from sklearn.metrics import roc_auc_scorefrom sklearn.metrics import roc_curve, aucfpr, tpr, _ = roc_curve(y_test, y_pred)roc_auc = auc(fpr, tpr)
And plot the ROC:
import matplotlib.pyplot as pltimport seaborn as snssns.set()plt.figure()lw = 2plt.plot(fpr, tpr, color='red',         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.title('Receiver operating characteristic for Wine Price Classification')plt.legend(loc="lower right")plt.show()
We can also look at the precision/recall curve and the average precision:
precision, recall, _ = precision_recall_curve(y_test,y_pred)average_precision = average_precision_score(y_test, y_pred)sns.set()plt.figure()plt.step(recall, precision, where='post')plt.xlabel('Recall')plt.ylabel('Precision')plt.ylim([0.0, 1.05])plt.xlim([0.0, 1.0])plt.title(    'Average precision score Wine Price Classification, micro-averaged over all classes: AP={0:0.2f}'    .format(average_precision)
Next let’s discuss the ‘KFold’ method. This function provides train and test indices to split data into train and test sets. It splits the data into K folds either randomly or consecutively (default), and each fold is used once for validation while the remaining k-1 folds make up the training set. We’ll use the default value for the number of folds, which is 5. Let’s initialize a ‘KFold’ object, get the data splits, and initialize arrays for the predicted target values and true target values:
from sklearn.model_selection import KFoldfolds = KFold()folds.get_n_splits(df)y_true  = []y_pred = []
If we want to change the number of folds from 5 to 10 we do:
folds = KFold(n_splits=10)
Next, let’s train and test our model in a for-loop and append the results:
for train_index, test_index in folds.split(df):    df_test = df.iloc[test_index]    df_train = df.iloc[train_index]         X_train = np.array(df_train[['country_cat', 'winery_cat', 'variety_cat', 'points']])    y_train = np.array(df_train['price_class'])    X_test = np.array(df_test[['country_cat', 'winery_cat', 'variety_cat', 'points']])    y_test = np.array(df_test['price_class'])        y_true.append(y_test)    model = RandomForestClassifier()    model.fit(X_train, y_train)        y_pred.append(model.predict_proba(X_test)[:,1])y_pred = [item for sublist in y_pred for item in sublist]   y_true = [item for sublist in y_true for item in sublist]
Next we plot the ROC curve:
fpr, tpr, _ = roc_curve(y_true, y_pred)roc_auc = auc(fpr, tpr)sns.set()plt.figure()lw = 2plt.plot(fpr, tpr, color='red',         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.title('Receiver operating characteristic for Wine Price Classification (KFolds)')plt.legend(loc="lower right")plt.show()
And the precision/recall curve:
precision, recall, _ = precision_recall_curve(y_true,y_pred)average_precision = average_precision_score(y_true, y_pred)sns.set()plt.figure()plt.step(recall, precision, where='post')plt.xlabel('Recall')plt.ylabel('Precision')plt.ylim([0.0, 1.05])plt.xlim([0.0, 1.0])plt.title(    'Average precision score Wine Price Classification (KFolds), micro-averaged over all classes: AP={0:0.2f}'    .format(average_precision))
Finally, let’s discuss it the ‘LeaveOneOut’ method. This function provides train/test indices to split data for training and testing. Each sample is used once as a singleton test while the remaining samples are used for training. Here we initialize the ‘LeaveOneOut’ object and get the data splits as before:
from sklearn.model_selection import LeaveOneOutloo = LeaveOneOut()loo.get_n_splits(df)y_true  = []y_pred = []
And we loop over the indices and append the results. Code is truncated for clarity:
for train_index, test_index in loo.split(df):    df_test = df.iloc[test_index]    df_train = df.iloc[train_index]    ...
Next we plot the results:
I’ll stop here but I encourage you to play around with the functions yourself.
To summarize, in this post we discussed how to use functions in the model selection module to validate models. First we discussed the train/test split function which randomly selects data for training and testing. We then discussed the ‘K-Fold’ function, which splits data into either consecutive or random folds for training and testing. Finally, we went over the ‘LeaveOneOut’ function which uses a singleton sample for testing and the remaining data for training for each sample in the data. I hope you found this post useful/interesting. The code in this post is available on GitHub. Thank you for reading!