Assigning Panel Data to Training, Testing and Validation Groups for Machine Learning Models | by Shad Griffin | Towards Data Science

Make sure your groups are independent and avoid deployment disasters
github.com
Cross-Sectional data includes individual entities measured in one time period. For example, 10,000 people measured once is cross-sectional data.
Time series includes one entity measured over multiple time periods. For example, a single machine measured every day for ten years is a time-series.
Panel data includes multiple entities measured over multiple time periods. For example, 1,000 consumers measured monthly over ten months is panel data. Or, 100 machines measured daily for 100 days, is panel data.
Panel data is quite common in data science. Sometimes, it is called cross-sectional time-series data. I have also heard it referred to as pooled time series data. Whatever you want to call it, as a practicing data scientist, you’ll more than likely have to deal with it.
It is standard procedure when building machine learning models to assign records in your data to modeling groups. Typically, we randomly sub-set the data into Training, Testing and Validation groups. Random, in this case, means that each record in the data set has an equal chance of being assigned to one of the three groups.
When you are working with Panel Data, however, it makes sense to alter the normal process a little.
In this notebook, I walk through a simple example of how to do this.
Import all of the relevant Python Libraries
import numpy as npimport numpy.dual as dualimport pandas as pd
Import the data from GitHub
#Remove the data if you run this notebook more than once!rm equipment_failure_data_1.csv#import first half from github!wget https://raw.githubusercontent.com/shadgriffin/machine_failure/master/equipment_failure_data_1.csv# Convert csv to pandas dataframepd_data_1 = pd.read_csv("equipment_failure_data_1.csv", sep=",", header=0)#Remove the data if you run this notebook more than once!rm equipment_failure_data_2.csv#Import the second half from github!wget https://raw.githubusercontent.com/shadgriffin/machine_failure/master/equipment_failure_data_2.csv# convert to pandas dataframepd_data_2 = pd.read_csv("equipment_failure_data_2.csv", sep=",", header=0)#concatenate the two data files into one dataframepd_data=pd.concat([pd_data_1, pd_data_2])
pd_data.head()
Here is a description of the fields in the data set.
ID — ID field that represents a specific machine.
DATE — The date of the observation.
REGION_CLUSTER — a field that represents the region in which the machine is located.
MAINTENANCE_VENDOR — a field that represents the company that provides maintenance and service to the machine.
MANUFACTURER — the company that manufactured the equipment in question.
WELL_GROUP — a field representing the type of Machine.
EQUIPMENT_AGE — Age of the machine, in days.
S15 — A Sensor Value.
S17 — A Sensor Value.
S13 — A Sensor Value.
S16 — A Sensor Value.
S19 — A Sensor Value.
S18 — A Sensor Value.
S8 — A Sensor Value.
EQUIPMENT_FAILURE — A ‘1’ means that the equipment failed. A ‘0’ means the equipment did not fail.
This data represents a panel data set. We have multiple machines measured over mulitple time periods. ID represents the machine and DATE represents the date. Now, let’s examine how many machines and how many dates we have.
Examine the number of rows and columns. The data has 307,751 rows and 16 columns
pd_data.shape
There are 421 machines in the data set.
xxxx = pd.DataFrame(pd_data.groupby(['ID']).agg(['count']))xxxx.shape
There are 731 unique dates in the data set.
xxxx = pd.DataFrame(pd_data.groupby(['DATE']).agg(['count']))xxxx.shape
We could just randomly assign each record to either the training, testing or validation group. While that could work, I wouldn’t recommend it. I would recommend assigning records to groups at an entity level (machine in this case).
Why?
Well, I could use some multi-syllabic words (like auto-correlation, boomerang or mayonnaise) to describe why, but let’s just think about it.
Why do we separate the data into training, testing and validation groups?
We want to ensure that our model is not over-fit. In other words, we want to make sure that our model applies to new data as it comes available.
For example, let’s pretend that we built a model that predicts what happened last year with 100% accuracy. Good job, right? Well, it really doesn’t matter how well the model predicts last year. We need it to predict today, tomorrow and the day after that. So, if a model predicts last year with 100% accuracy but fails to predict tomorrow, it kind of sucks.
Building a model on the training data and verifying the accuracy on the testing and validation data set reduces the chances of over-fitting (Note, it is still possible but I will save that discussion for another day).
In order to prevent over-fitting, our training, testing and validation groups must be independent. That is, the data in the training group must be different than the testing and validation groups. Or, at least as different as possible.
So what happens if we just randomly assign each record in a panel data set to each of the groups?
We end up with records from each entity in each group.
For example with a simple random selection method, if we are dealing with 100 machines measured daily over 100 days, it is probable that the machine physically sitting beside the break room will appear in the training, testing and validation groups. Likewise, if you are dealing with individuals (one of whom is named Steve Wakahookie), it is probable that Steve Wakahookie will appear in all three groups.
In other words, your training, testing and validation groups ARE NOT independent because good ol’ Steve and the machine beside the break room are present in all three groups.
Steve is in the testing. Steve is in the training. Steve is in the validation. If he is lurking in all three groups, the three groups are not independent.
Now, if we assign group membership based on entity, all of the Steve’s records will be in either the training, testing or validation group. Likewise, all of the records associated with the machine by the break room will be in only one of the three groups.
If Steve is in the training, but not the testing or validation, the groups ARE independent.
If the machine by the break room is in the testing, but not in the training or validation, the groups ARE independent.
Here is a few lines of python code to that ensure that your training, testing and validation groups are independent.
Get a Unique List of All IDs (machines).
aa=pd_datapd_id=aa.drop_duplicates(subset='ID')pd_id=pd_id[['ID']]pd_id.shape
Create a new variable with a random number between 0 and .
np.random.seed(42)pd_id['wookie'] = (np.random.randint(0, 10000, pd_id.shape[0]))/10000pd_id=pd_id[['ID', 'wookie']]
Give each machine a 30% chance of being in the validation, a 35% chance of being in the testing and a 35% chance of being in the training data set.
pd_id['MODELING_GROUP'] = np.where(((pd_id.wookie <= 0.35)), 'TRAINING', np.where(((pd_id.wookie <= 0.65)), 'VALIDATION', 'TESTING'))
This is how many machines fall in each group.
tips_summed = pd_id.groupby([‘MODELING_GROUP’])[‘wookie’].count()tips_summed
Append the Group of each id to each individual record.
pd_data=pd_data.sort_values(by=['ID'], ascending=[True])pd_id=pd_id.sort_values(by=['ID'], ascending=[True])pd_data =pd_data.merge(pd_id, on=['ID'], how='inner')pd_data.head()
This is how many records are in each group.
tips_summed = pd_data.groupby(['MODELING_GROUP'])['wookie'].count()tips_summed
So, there you go. Now, we are ready to build a machine learning model. By placing entities and not records into your training, testing and validation groups you can ensure independence between the groups and build models that work yesterday, today and tomorrow.
Shad Griffin is a Data Scientist at IBM