Train and Test Data Split
By: Hitesh Mandav
Tags: Numpy, Data Science, Sklearn, Machine Learning, Scikit Learn

Now we have understood the need to split the data let's see how can we do this.

Using Python

Here we are going to use python to implement a function that will do this split for us.

First, we need a data set. We will create a sample data frame using pandas.

import pandas as pd df = pd.DataFrame({‘Temprature’: [98.3, 99.8, 97.3,96.4, 92.5 ], ‘Humidity’: [0.23, 0.67, 1.7, 0.8, 1.3], ‘Rained’ : [0, 0, 1, 1, 1]})

print(df)

This sample data tells us the temperature, humidity and if it has rained on that particular day

Now we can write a function that will take data and split ratio as parameters and return two data sets one for testing and one for training. To do this we will use NumPy.

import numpy as np # mannual basic train test spliting funtion

def split_train_test(data, test_ratio):

shuffled_indicies = np.random.permutation(len(data))

test_data_size = int(len(data) * test_ratio)

test_indicies = shuffled_indicies[:test_data_size]

train_indicies = shuffled_indicies[test_data_size:]

return data.iloc[train_indicies], data.iloc[test_indicies] train_data, test_data = split_train_test(df, 0.2)

print(f’Train data set count {len(train_data)}

Test data set count {len(test_data)}’)

In this method, we are creating an array of shuffled indices for the length of the data set. This is done so that the data doesn't represent a pattern if it is sorted over a particular feature. Then we get the length of the test data size.

We use these variables to get indices of train and test data and return data sets for test and train.

But in this approach, we are going to run into a problem. If we call this function multiple times we will always get a different data set for test and trains we are using np.random to generate the shuffled indices.

This will lead to the very problem that we are trying to eliminate in the long run our entire data set will be exposed to the model and we will not have any data that is not seen by the model.

So to eliminate this and get the same shuffled indices every time we can set a seed for np.random. and this will create the same shuffled indices every time. The seed will take an integer value and the will generate same shuffled indices as long as the seed value is the same.

So we will add a parameter called random_seed and pass that to np.random.seed() in the split_train_test method.

import numpy as np # mannual basic train test spliting funtion

def split_train_test(data, test_ratio, random_seed): # setting random seed value to genrate same shuffled indicies

np.random.seed(random_seed) shuffled_indicies = np.random.permutation(len(data))

test_data_size = int(len(data) * test_ratio)

test_indicies = shuffled_indicies[:test_data_size]

train_indicies = shuffled_indicies[test_data_size:]

return data.iloc[train_indicies], data.iloc[test_indicies] train_data, test_data = split_train_test(df, 0.2, 42)

print(f’Train data set count {len(train_data)}

Test data set count {len(test_data)}’)

This method will now always return the same data sets even if it is called multiple times as long as the seed value is same.

Now, this data splitting is needed for every ML model that we will create so to make our task easier scikit-learn has some inbuilt methods that will take care of this splitting for us.

Split data using scikit-learn.

In sklearn.model_selection we have a train_test_split method that we can use to split data into training and testing sets.

Below is the implementation

from sklearn.model_selection import train_test_split #basic train test split using sklearn

train_data, test_data = train_test_split(df, test_size = 0.2, random_state = 42)

print(f’Train data set count {len(train_data)}

Test data set count {len(test_data)}’)

Here we are passing all the same values that we used in the above function that we created.

Now sometimes we have a feature that we want to split evenly into training and testing data.

For example, I have taken a dataset ( Bostan housing Dataset ) where I have a feature called “CHAS” which contains two values 0 and 1.

I have copied the data in a housing data frame and now I can print the value counts for the “CHAS” feature.

print(f’Data set value counts for CHAS column

{housing[“CHAS”].value_counts()}’ )

Here we have 367 data points with a value of 0 and 27 data points with a value of 1. This is a ratio of ~13.6 for the 0 and 1 distribution in the dataset.

Now after analyzing this data we decide that this feature needs to be split evenly in training and test data.

We can do this using scikit-learn’s StratifiedShuffleSplit

from sklearn.model_selection import StratifiedShuffleSplit split = StratifiedShuffleSplit(n_splits=1, test_size = 0.2, random_state = 42)

for train_i, test_i in split.split( housing, housing[‘CHAS’]):

strat_train_data = housing.iloc[train_i]

strat_test_data = housing.iloc[test_i]



print(f’Train data set count {len(strat_train_data)}

Test data set count {len(strat_test_data)}’))

print(f”Train value count for CHAS column

{strat_train_data[‘CHAS’].value_counts()}”)

print(f’Test value count for CHAS column

{strat_test_data[“CHAS”].value_counts()}’)

Here we can see the train set has a ratio of ~13.3 and the test set has a ratio of ~14.8 for the 0 and 1 distribution. so the data have split almost equally considering the ‘CHAS’ feature.

This is a very short introduction to how you can split training and test data for your ML model.