Demystify Employee Leaving with EDA
By: Luke Sun
Tags: Data Science, Exploratory Data Analysis, Binary Classification, Correlation, Data Visualization

Demystify Employee Leaving with EDA

Img from pixabay via link

In previous posts, I tried to predict if a bank customer is likely to leave, OR if an app user is likely to churn or subscribe. Now I will share recent work in the human resource domain to bring some predictive power to any firm struggling to retain their employees.

In this first post, I will focus on exploring datasets for any interesting patterns. As always, it is split into below:

1. Problem Statement

2. Data Review

3. Distribution Analysis

4. Independent Variable Correlation Analysis

5. Response Variable Correlation Analysis

6. Takeaways

Let’s begin the journey 🏃‍♂️🏃‍♀️.

1. Problem Statement

If you are from the Human Resources department, you may agree that hiring and retaining employees are complex tasks that require capital, time, and skills.

According to toggle hire, companies spend 15%-20% of the employee’s salary to recruit a new candidate. Hiring an employee in a company in a company with 0–500 people costs an average of $7,645 🙀🙀.

As a data scientist, you are tasked to develop a model that could predict which employees are more likely to quit. Your findings will support more effective and efficient employee retainment.

2. Data Review

The dataset used here is IBM HR Analytics dataset from Kaggle. There are 1470 records with 35 features. Check the video below to have a real touch on the raw dataset.

To better understand the meaning of each feature, I made an easy-to-view figure below.

Fig.1 Data variable overview

To understand the distribution of numerical variables, use below.

employee_df.describe()

For data quality check (if there is any NULL or missing data in each column) and view the data type, use below:

employee_df.info()

Note that you can also use below to check if there are any NA values in a dataframe:

employee_df.isna().any()

employee_df.isnull().any()

With all the above, we found 26 numerical variables and 9 categorical variables with no missing values.

3. Distribution Analysis

We use the histogram as a simple visualization for numerical variable distribution analysis.

First, let’s convert binary categorical data into numerical. Use a one-line function with lambda to apply to the whole column. Then, pandas built-in plot function to create a histogram. Specifically,

employee_df[‘Attrition’] = employee_df[‘Attrition’].apply(lambda x: 1 if x == ‘Yes’ else 0)

employee_df[‘OverTime’] = employee_df[‘OverTime’].apply(lambda x: 1 if x == ‘Yes’ else 0)

employee_df[‘Over18’] = employee_df[‘Over18’].apply(lambda x: 1 if x == ‘Y’ else 0)

employee_df.hist(bins = 30, figsize= (15, 15), color = 'r')

Fig.2 Distribution analysis for numerical variables

Reviewing Fig.2, you may find column ‘EmployeeCount’, ‘Over18’, and ‘StandardHours’ is of no value because these remain the same for all employees. So let’s remove them:

employee_df.drop([‘EmployeeCount’, ‘EmployeeNumber’, ‘Over18’, ‘StandardHours’], axis = 1, inplace = True)

4. Independent Variable Correlation Analysis

Variables correlation is an important issue that is worth every data scientist’s nerve.

In general, correlated features don’t improve model performance. It is wise to remove correlated features considering the curse of dimensionality and the interpretability of the model. To perform correlation analysis, use:

correlations = employee_df.corr()

plt.subplots(figsize = (20, 20))

sns.heatmap(correlations, annot = True)

Fig.3 shows some sensible correlations. For instance, ‘TotalWorkingYears’ is strongly correlated with ‘JobLevel’ and ‘MonthlyIncome’. In general, we can see quite a lot of variables are poorly correlated. Great, because it is desirable to train a predictive model with features that are not highly correlated with each other.

Fig.3 Correlation analysis for predictors

5. Response Variable Correlation Analysis

To analyze how each feature impacts employee leaves, we can inspect each variable’s distribution with respect to the response variable.

First, for employee ‘Age’, let’s use countplot,

plt.figure(figsize = (25, 10), dpi = 300)

sns.countplot(x = ‘Age’, hue = ‘Attrition’, data = employee_df )

Fig.4 shows employees between 25–35, show a high leave ratio compared to the rest ages. Below 21, a very high leave percentage can be spotted, indicating young professionals are more likely to leave the firm 🤭.

Fig.4 Employee Leave Countplot Vs Age

Second, for ‘JobRole’, count plot is:

sns.countplot(x = ‘JobRole’, hue = ‘Attrition’, data = employee_df)

Fig.5 shows ‘sales executive’, ‘sales representative’, and ‘lab technician’ are more likely to leave compared to other roles.

Fig.5 Employee Leave Countplot Vs JobRole

Third, let’s use KDE (Kernel Density Estimate) to analyze the probability density function of the numerical variables.

KDE takes bandwidth and a kernel function to approximate the true distribution of the population by weighing the distances near each sample. The more data points nearby, the estimate is high, indicating the probability of seeing a point. Bandwidth determines how ‘smooth’ the resulting curve is. A lower bandwidth means only points close to the current position are given weight. Enough of the explanation, if you want more introduction, try this link. Let’s see how to achieve it.

For the variable ‘DistanceFromHome’,

sns.kdeplot(left_df[‘DistanceFromHome’], label = ‘Employee left’, shade = True, color = ‘r’, )

sns.kdeplot(stayed_df[‘DistanceFromHome’], label = ‘Employee stay’, shade = True, color = ‘b’)

As shown in Fig.6, more employees tend to leave when the distance from home is more than 10 km 🆒🆒.

Fig.6 KDE plot for Distance

For the variable ‘YearsWithCurrManager’, more employees tend to leave with less than 2 years with the current managers and tend to stay if having more than 5 years experience 🧨🧨, as shown in Fig.7.

Fig.7 KDE plot for YearsWithCurrManager

For the variable ‘ TotalWorkingYears’, more employees tend to leave with less than 7 years of experience ️📣, as shown in Fig.8.

Fig.8 KDE plot for TotalWorkingYears

Now, let’s use boxplot to visualize and compare the distribution of ‘MonthlyIncome’ between ‘Gender’. As shown in Fig.9, female and male monthly income shows a similar variability with similar median, 75%, and 25% quartile values 👏👏, as shown in Fig.9.

Fig.9 Boxplot for MonthlyIncome btw Gender

At last, let’s see different ‘JobRole’ income distribution. Fig.10 shows the monthly income distribution and variability of different roles. Interestingly, we found ‘Sales Representative’ shows a compact distribution, with income broadly lower than other roles 🙀. This could explain why ‘Sales Representative’ tends to leave more as shown in Fig.5.

Fig.10 Boxplot for MonthlyIncome btw JobRoles

6. Takeaway

Data quality check tells us that our dataset is quite clean with no missing values.

Independent variable correlation analysis indicates quite a lot of features seem to be poorly correlated with one another.

Interesting patterns between independent and response variables show some hints on why employees may leave.

Hopefully, this sheds some light on how to better understand your employee or colleague's decision when it comes to leaving.

Great! End of this journey. In the next one, we will focus on the real fun part: modeling. See you there💕💕