Data Processing Example using Python | by Kamil Mysiak | Towards Data Science

Forbes’s survey found that the least enjoyable part of a data scientist’s job encompasses 80% of their time. 20% is spent collecting data and another 60% is spent cleaning and organizing of data sets. Personally, I disagree with the notion that 80% is the least enjoyable part of our jobs. I often see the task of data cleansing as an open-ended problem. Typically, each data set can be processed in hundreds of different ways depending on the problem at hand but we can very rarely apply the same set of analyses and transformations from one dataset to another. I find that building different processing pipelines and examining how their differences affect model performance is an enjoyable part of my job.
With that said, I want to take the time and walk you through the code and the thought process of preparing a dataset for analysis which in this case will be a regression (ie. multiple regression).
Those of you who follow me know that I’m particular to human resources datasets as I have been working in the industry for most of my career.
If you have a rare HR dataset please share with us :)
We will be working with a dataset of 310 active and terminated employees along with information much as marital status, gender, department, pay rate, state, position, etc. Since we are prepping the data for regression analysis, our target feature is EngagementSurvey.
The code book for our dataset can be found here.
import numpy as npimport pandas as pdimport datetimeimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.pipeline import make_pipelinefrom feature_engine import missing_data_imputers as mdifrom feature_engine import categorical_encoders as cefrom sklearn.model_selection import train_test_split%matplotlib inlinewith open('HRDataset.csv') as f:    df = pd.read_csv(f)f.close()df.head()df.info()
Upon loading our data we can see a number of unique feature types. We have categorical features such as “Employee_Name” and “Position”. We have binary features such as “MarriedID”. We have continuous features such as “PayRate” and “EmpSatisfaction”. We have discrete features such as “DaysLateLast30” and finally we have date features such as “LastPerformanceReview_Date”.
The first step I typically take is reviewing the unique count of values per feature to determine if any features can be quickly deleted due to very high or very low variability. In other words, do we have any features which have as many unique values as the length of the dataset or features which have just one unique value?
for col in df.columns:    print(col, df[col].nunique(), len(df))
df.drop(['Employee_Name'], axis=1, inplace=True)df.drop(['EmpID'], axis=1, inplace=True)df.drop(['DOB'], axis=1, inplace=True)df.drop(['DaysLateLast30'], axis=1, inplace=True)
We can safely remove “Employee_Name”, “Emp_ID”, “DOB” since most if not all, values are unique for each feature. Also, we can remove “DaysLateLast30” as this feature only contains one unique value.
Next, by examining the codebook, which contains the definitions for each feature, we can see that we have many duplicate features. For example, “MarriedStatusID” is a numerical feature that produces the code that matches the married statues in “MaritalDesc” feature. We can drop these features.
df.drop(['MaritalStatusID', 'EmpStatusID', 'DeptID'], axis=1, inplace=True)df.drop(['GenderID'], axis=1, inplace=True)df.drop(['PerformanceScore'], axis=1, inplace=True)df.drop(['MarriedID'], axis=1, inplace=True)
You might be asking yourself “What about ‘PositionID’, ‘Position’, ‘ManagerID’ and ManagerName’?”. As you can see from the output above, the unique value counts for these feature pairs do not match. “PositionID” has 32 unique values whereas “Position” has 30.
df[['PositionID', 'Position']].sort_values('PositionID')[50:70]df[['ManagerName', 'ManagerID']].sort_values(by='ManagerID').tail(50)df.drop('PositionID', axis=1, inplace=True)df.drop('ManagerID', axis=1, inplace=True)
We are going to drop “PositionID” as it does not maintain all available positions and we are going to drop “ManagerID” as “ManagerName” does not contain any missing values.
Next, let’s examine the individual unique values for each feature. This will help us see any odds values and mistakes which will need to be fixed.
for col in df.columns:    print(col, df[col].unique(), len(df))
From the codebook, we know that features such as “FromDiversityJobFairID”, and “Termd” are binary codings for “Yes” and “No”. In order to simplify our analysis and help with formatting, we need to convert the binary to string. We also see trailing spaces for the position of “Data Analyst” and Department of “Production” which need to be removed. Finally, we see a coding mistake for “HispanicLatino” which needs to be corrected.
diversity_map = {1: 'yes', 0: 'no'}termd_map = {1: 'yes', 0: 'no'}hispanic_latino_map = {'No': 'no', 'Yes': 'yes', 'no': 'no', 'yes': 'yes'}df['FromDiversityJobFairID'].replace(diversity_map, inplace=True)df['Termd'].replace(termd_map, inplace=True)df['HispanicLatino'].replace(hispanic_latino_map, inplace=True)df['Position'] = df['Position'].str.strip()df['Department'] = df['Department'].str.strip()
You might be asking yourself “How come some zip codes are 5 digits and some are only 4?”. In the US all zip codes are 5 digits long. After a little bit of googling, many Massachusetts zip codes actually begin with zero, and by default, python stripped the zeros which resulted in 4 digit zip codes. Since we will be treating zip codes as a categorical feature the length wouldn’t matter.
Believe it or not but datetime features very often contain a plethora of info just waiting to be unleashed. This is especially evident when one is familiar with the industry from which the data originates from.
df['DateofHire'] = pd.to_datetime(df['DateofHire'])df['DateofTermination'] = pd.to_datetime(df['DateofTermination'])df['LastPerformanceReview_Date'] = pd.to_datetime(df['LastPerformanceReview_Date'])df['DateofHire_month'] = df['DateofHire'].dt.monthdf['DateofHire_day'] = df['DateofHire'].dt.day df['DateofHire_year'] = df['DateofHire'].dt.yeardf['DateofHire_quarter'] = df['DateofHire'].dt.quarterdf['DateofHire_day_week'] = df['DateofHire'].dt.day_name()df['DateofHire_weekday'] = np.where(df['DateofHire_day_week'].isin(['Sunday','Saturday']),'yes','no')df['DateofTerm_month'] = df['DateofTermination'].dt.monthdf['DateofTerm_day'] = df['DateofTermination'].dt.daydf['DateofTerm_year'] = df['DateofTermination'].dt.yeardf['DateofTerm_quarter'] = df['DateofTermination'].dt.quarterdf['DateofTerm_day_week'] = df['DateofTermination'].dt.day_name()df['DateofTerm_weekday'] = np.where(df['DateofTerm_day_week'].isin(['Sunday','Saturday']),'yes','no')df['LastPerform_month'] = df['LastPerformanceReview_Date'].dt.monthdf['LastPerform_day'] = df['LastPerformanceReview_Date'].dt.day df['LastPerform_year'] = df['LastPerformanceReview_Date'].dt.year df['LastPerform_quarter'] = df['LastPerformanceReview_Date'].dt.quarterdf['LastPerform_day_week'] = df['LastPerformanceReview_Date'].dt.day_name()df['LastPerform_weekday'] = np.where(df['LastPerform_day_week'].isin(['Sunday','Saturday']),'yes','no')df['tenure_termed'] = df['DateofTermination'] - df['DateofHire']df['tenure'] = datetime.datetime.today() - df['DateofHire']df['days_since_review'] = datetime.datetime.today() - df['LastPerformanceReview_Date']df.drop(['DateofHire', 'DateofTermination', 'LastPerformanceReview_Date'], axis=1, inplace=True)df.head()
First, we need to convert our features to datetime format. Next, using the “datetime” library we can extract new features from our original datetime features with information such as a month, day, year, quarter, weekday string, and even whether or not the day falls on a weekend. Finally, we can subtract individual dates from each other to calculate things like tenure_termed (terminated date — hire date) and tenure (today’s date — hire date). Once we have extracted the necessary information we can drop the original features.
df['days_since_review'] = df['days_since_review'].astype(str)df['days_since_review'] = [i[0:3] for i in df['days_since_review']]df['days_since_review'] = df['days_since_review'].astype(str)df['days_since_review'] = [i[0:3] for i in df['days_since_review']]df['tenure_termed'] = df['tenure_termed'].astype(str)df['tenure_termed'] = [i[0:2] for i in df['tenure_termed']]for var in df.columns:    df[var].replace(to_replace=['NaT','Na'], value=np.nan, inplace=True)df.head()
Perhaps I’m being a little obsessive-compulsive but I like tidy datasets, therefore, let’s remove the irrelevant information such as “days” and the timestamp from these new features. Finally, we convert the ‘NaT” and “Na” to true numpy “NaN”.
for var in df.columns:    print(var, '\n', df[var].value_counts()/len(df))df.drop(['CitizenDesc', 'DateofHire_weekday', 'DateofTerm_weekday',          'LastPerform_quarter', 'LastPerform_weekday', 'LastPerform_year'], axis=1, inplace=True)
Cardinality refers to the number of unique values/categories for each feature. Numeric, especially continuous, features will have very high cardinality but we mainly need to concern ourselves from categorical features. First, we need to identify features that contain values/categories which suck up all the variance. In other words, 90%+ of all the observations fall under one or two values. For example, “CitizenDesc” has three unique values but we see that “US Citizen” contains 95% of all the observations. Other features which exhibit this pattern, unfortunately, are our newly engineered features such as “DateofHire_weekday”, “DateofTerm_weekday”, “LastPerform_quarter”, “LastPerform_weekday”, and “LastPerform_year”. We can safely drop these features as they do not provide enough variability to be meaningful.
Using the same code as above, we once again turn our attention onto categorical features but this time we are looking for values which we consider “rare”. How you define “rare” is really up to you but I have found that this decision has to be made a feature by feature. Some values might be rare if they appear less than 1% of the time. In other features, the threshold might be 2% or even 5%. Our ultimate goal will be to group these values together into a new value/category called “rare”. This procedure reduces the overall cardinality of the feature and if you choose to one-hot encode your categories features this method will drastically reduce the number of newly created “dummy” features.
State: Anything less than 1% will be considered ‘rare’
Position: Anything less than 2% will be considered ‘rare’
Zip: Anything less than 2% will be considered ‘rare’
RaceDesc: Anything less than 2% will be considered ‘rare’
RecruitmentSource: Anything less than 2% will be considered ‘rare’
DateofHire_day: Anything less than 2% will be considered ‘rare’
DateofTerm_month: Anything less than 2% will be considered ‘rare’
DateofTerm_day: Anything less than 2% will be considered ‘rare’
LastPerform_day: Anything less than 2% will be considered ‘rare’
LastPerform_day_week: Anything less than 2% will be considered ‘rare’
DateofHire_year: Anything less than 2% will be considered ‘rare’
DateofTerm_year: Anything less than 2% will be considered ‘rare’
ManagerName: Anything less than 5% will be considered ‘rare’
Deciding how to process missing values is one of the most important and contentious decisions a data scientist will make.
for var in df.columns:    if df[var].isnull().sum()/len(df) > 0:        print(var, df[var].isnull().mean().round(3))df.drop('tenure_termed', axis=1, inplace=True)
TermReason is a categorical feature with only a few missing data points. We can impute this data using the mode as this wouldn’t change the distribution of the feature. Furthermore, we can safely assume that a missing TermReason simply means the employee is still active. The remaining features with missing data are what we call “Missing Not At Random” (MNAR). In other words, there is an underlying reason these features are missing. First, the percentages of missing values seem to repeat which gives us a clue that there is a discernible pattern to these missing values. Secondly, we know from the data that roughly 67% of all employees are active and would not have a Termination Date. Lastly, oftentimes employees hired after a recent performance review cycle will not have a date associated with their last performance review date. If you wish to read more about missing values please consider this resource.
Some would argue 67% of missing values effectively renders the feature useless and I’m going to agree with this notion for our “tenure_termed” feature. Imputing this numerical feature would potentially introduce too much error variance/bias into our data. However, features such as “DateofTerm_month”, and “LastPerform_month” are categorical in nature with a definitive pattern underlying their missing data. I want to capture the importance of the missing values by imputing all missing values with the string “missing”. This way we are introducing another value/category to each feature that appropriately captures the pattern behind the missing values.
On the other hand, “days_since_review” is a numeric feature which is MNAR. In other to capture the significance of these missing values we are going to impute an arbitrary number (ie. -9999) and create a new feature that will indicate whether or not an observation was missing for this feature.
TermReason: impute with mode
DateofTerm_month: impute with ‘missing’ to create a new category
DateofTerm_day: impute with ‘missing’ to create a new category
DateofTerm_year: impute with ‘missing’ to create a new category
DateofTerm_quarter: impute with ‘missing’ to create a new category
DateofTerm_day_week: impute with ‘missing’ to create a new category
LastPerform_month: impute with ‘missing’ to create a new category
LastPerform_day: impute with ‘missing’ to create a new category
LastPerform_day_week: impute with ‘missing’ to create a new category
tenure_termed: drop due to large number of missing data
days_since_review: arbitrary imputation along with a missing indicator feature
Outliers are another contentious topic which requires some thought. There are a number of ways of dealing with outliers. If you have a very large dataset and a relatively small number of outliers you can simply delete them. I’m usually wary of this method as it changes the distribution of said feature(s) which might cause new values to become outliers. That said, it is an option often utilized. Other methods include adding an indicator feature, rescaling the entire feature using np.log(), and transforming a continuous feature into discrete by applying discretization which will encompass the outliers into one bin.
First, we need to identify if we have any outliers. The most well-known method for identifying outliers is the z-score method which standardizes the feature values to a mean of zero, a standard deviation of one, and any value which falls 3 standard deviations (plus or minus) is considered an outlier. Personally, I believe this method is flaw as the z-score relies on the mean and standard deviation of the feature. Both the mean and standard deviation are highly influenced by existing outliers. Any outlier included in the calculation of the mean and standard deviation will expand the range of the z-scores and potentially omitting existing outliers. This problem can be overcome by utilizing the median instead of the mean.
Let’s utilize a more robust method that relies on the inter-quartile range and the median. You can adjust this method and use (3 * IQR) to identify only the extreme outliers.
def outlier_treatment(feature):    sorted(feature)    q1,q3 = np.percentile(feature , [25,75])    IQR = q3 - q1    lower_range = q1 - (1.5 * IQR)    upper_range = q3 + (1.5 * IQR)    return lower_range,upper_rangeoutlier_treatment(df['PayRate'])lower_range, upper_range = outlier_treatment(df['PayRate'])df[(df['PayRate'] < lower_range) | (df['PayRate'] > upper_range)]
X = df.drop('EngagementSurvey', axis=1)y = df['EngagementSurvey']X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# impute categorical features with more than 5% missing values w/ a new category 'missing'process_pipe = make_pipeline(    mdi.CategoricalVariableImputer(variables=['DateofTerm_month', 'DateofTerm_day','DateofTerm_quarter', 'DateofTerm_day_week','LastPerform_month', 'LastPerform_day','LastPerform_day_week', 'DateofTerm_year'], imputation_method='missing'),    # Imputing categorical features with less than 5% missing values w/the mode    mdi.CategoricalVariableImputer(variables=['TermReason'], imputation_method='frequent'),    # Imputing missing values for numerical feature 'days_since_review' with an arbitrary digit    mdi.ArbitraryNumberImputer(arbitrary_number = -99999, variables='days_since_review'),   # We are adding a feature to indicate (binary indicator) which records were missing    mdi.AddMissingIndicator(variables=['days_since_review']),    # Encoding rare categories (less than 1% & the feature must have at least 5 categories)    ce.RareLabelCategoricalEncoder(tol=0.01, n_categories=5,                                   variables=['State']),    # Encoding rare categories (less than 2% & the feature must have at least 5 categories)    ce.RareLabelCategoricalEncoder(tol=0.02, n_categories=5,    variables=['Position', 'Zip', 'DateofTerm_day', 'LastPerform_day_week', 'DateofTerm_year', 'RaceDesc', 'TermReason', 'RecruitmentSource','DateofHire_day', 'DateofTerm_month', 'LastPerform_day', 'DateofHire_year']),    # Encoding rare categories (less than 5% & the feature must have at least 5 categories)    ce.RareLabelCategoricalEncoder(tol=0.05, n_categories=5,                                   variables=['ManagerName']),    # Target or Mean encoding for categorical features    ce.OrdinalCategoricalEncoder(encoding_method='ordered',    variables=['FromDiversityJobFairID', 'Termd','Position', 'State','Zip','Sex', 'MaritalDesc','HispanicLatino', 'RaceDesc', 'TermReason','EmploymentStatus', 'Department', 'ManagerName', 'RecruitmentSource', 'DateofHire_month','DateofHire_day', 'DateofHire_day','DateofHire_quarter', 'DateofHire_day_week', 'DateofTerm_month', 'DateofTerm_day','DateofTerm_year', 'DateofTerm_quarter','DateofTerm_day_week', 'LastPerform_month', 'LastPerform_day', 'LastPerform_day_week'] ))
One topic we haven’t discussed is categorical feature encoding. I typically try and avoid using one-hot encoding due to the fact it has a tendency to greatly expand the feature space. The encoding of “rare” values/categories certainly helps with this issue if we were to use one-hot encoding. That said, I opted to use Target or Mean encoding as it does not expand the feature set. This method replaces the categories with digits from 0 to k-1. We first calculate the mean for the target variable for each category for each categorical feature and then the means are replaced with the aforementioned digits based on the mean size. For example, we have a binary target and the first categorical feature is gender and it has three categories (male, female, and undisclosed). Let’s assume the mean for male is 0.8, female is 0.5, and undisclosed is 0.2. The encoded values will be male=2, female=1 and undisclosed=0.
process_pipe.fit(X_train, y_train)X_train_clean = process_pipe.transform(X_train)X_test_clean = process_pipe.transform(X_test)X_train_clean.head()
Feel free to provide feedback if you believe I might have missed an important step. Thanks for reading!