Solving Staff Attrition with Data | by Andrew Olton | Towards Data Science

Every now and then I enjoy hopping over to Kaggle to see if there are any interesting data sets that I may want to play with. Earlier this week I came across a fictional dataset on staff attrition. You can find the dataset using the link below:
www.kaggle.com
The dataset was shared by a Mr. Ludovic Benistant who asks the question: “Why are our best and most experienced employees leaving prematurely?”He also posed a challenge to “Have fun with this database and try to predict which valuable employees will leave next” which I was certainly up for.
I must admit however, that as interesting as his question was, I didn’t have much of an interest in answering it. I had other questions to ask:
Can I predict which staff members will leave next?Can I create a system which may assist in staff retention?
Can I predict which staff members will leave next?
Can I create a system which may assist in staff retention?
I’ll admit, that Question 1 isn’t very far from what Ludovic wanted to know, but my real aim in this experiment was to come up with a system for solving Question 2.
The dataset is simulated and contained the following fields:
Employee satisfaction level
Last evaluation
Number of projects
Average monthly hours
Time spent at the company
Whether they have had a work accident
Whether they have had a promotion in the last 5 years
Department
Salary
Whether the employee has left
To solve this problem, I decided to use a combination of three type of analytics:
Descriptive Analytics: What are the observations that help us shape our various hypotheses about staff attrition?
Predictive Analytics: Which members of staff are about to leave?
Prescriptive Analytics: What insights or suggestions can be made concerning those members of staff who are likely to leave?
Just for the fun of it, I thought I’d get the easy stuff out of the way first: building a machine learning model to predict which staff would leave or not.
I think I should point out when exactly this sort of thing would be used in the real world. If I were to apply this algorithm in the real world, I would probably run this algorithm after a performance appraisal or at the time when the “Last evaluation” would be populated. I think doing so would fit the model seamlessly into the workflow of the organization.
For those who may be wondering about my setup, I’m using a Python environment called Anaconda which I use to manage all of my packages, and I’m using the Keras api with a Theano backend for developing my models. I intend to learn Tensor Flow very soon however.
For this task I created a model with 1 input layer, 4 hidden layers and an output layer. The accuracy score is (I’m pretty sure there’s some overfitting happening here). Here’s the code:
This model yielded an accuracy score of 97%.
With the model created, I can predict who will leave the organization and the next thing I needed to find out was why. For this, I decided to use correlation analysis to determine which factors contribute the most to staff attrition.
I found that variable most correlated to attrition was job satisfaction which had a score of -0.39. A negative correlation means that the likelihood of departure increases as job satisfaction decreases (makes sense right?). I also observed that the employees who left were 34% less satisfied with their jobs that the employees who stayed.
I was off to a good start, but I wasn’t quite done yet.
After learning that employees were leaving because they weren’t satisfied with their jobs, I needed to know why this was the case and what could be done to increase job satisfaction. To find ways to address job satisfaction among employees, I did deeper investigation into the factors that contributed to job satisfaction.
I did another correlation analysis on job satisfaction and I discovered that the time spent at the company and the number of projects worked on had the greatest impact (this was true for staff who left and staff who stayed). Other factors such as the employee’s average monthly hours and their last evaluation also had an impact.
Now that I had a machine learning model that can predict whether or not a member of staff may leave and I had an idea of what contributes to job satisfaction, I had what I needed to build a system that creates recommendations for improving job satisfaction for employees who may be leaving soon. I thought it useful to create a “retention profile” which summarizes the traits of the members of staff who did not leave the organization. Doing so would allow me to compare the profile of a staff member who is likely to leave with this “retention profile” and easily generate insights that are actionable.
I should add that if this system were to be used in the wild, we would also take the value of the employee to the organization into consideration (which would have made answering Ludovic’s question necessary).
Here’s the code I wrote for creating the retention profile:
Now that I have a retention profile, the next step is to somehow create a system that would incorporate the model that I created earlier to provide users with recommendations on treating with the members of staff who are likely to leave the company.
The logic I came up with is fairly simple:
Use the Machine Learning Model to find staff who are likely to leave the organizationCompare this person’s scores to the retention profile created previouslyBased on the scores and the correlation of the attributes to job satisfaction...Create actionable suggestions for improving the likelihood of retention
Use the Machine Learning Model to find staff who are likely to leave the organization
Compare this person’s scores to the retention profile created previously
Based on the scores and the correlation of the attributes to job satisfaction...
Create actionable suggestions for improving the likelihood of retention
The output I got looked something like this:
This member of staff will leave.I’ve noticed that this member of staff’s ‘number_project’ is below the norm. You may want to consider increasing his/her ‘number_project’ by 2.8155514094640495
I’ll admit it’s not the best thing to look at, but I think it’s a great start. The final product is an actionable insight.
Prescriptive Analytics can be useful for bolstering the decision making power of staff who may be new to managerial positions and may be acting or being trained for such positions. I suppose that in some cases it can even be used to for evaluating potential candidates for such positions.
I personally feel that for this type of scenario, Prescriptive Analytics can be useful for getting to the root cause of employee dissatisfaction and finding ways to strengthen the relationship between staff and management. The best way to see if this system is truly capable of delivering value, would be to try it out in real life.
One of the things that we were not able to ascertain from the data is the quality of projects at the organization and I do believe that correlations alone may not be enough to determine the best way of improving job satisfaction. Perhaps with more data we could get a better understanding of the factors that contribute to job satisfaction.
*Update: A friend of mine mentioned that this type of project would be useful for setting strategic goals for departments within an organization. I didn’t think of it before, but I definitely agree with her (Thanks Shades).
I hope this experiment was useful and I hope you learned something. I’m very new at this and I would love your feedback. Please be as honest as possible, constructive criticism is always welcome.
The code for this experiment can be found here.
If you’d like to see what others have done with the data, follow this link:
www.kaggle.com
I was very impressed by what other people came up with.
Now that I’ve conquered this challenge, I’m going to test this approach with a similar dataset: The IBM Employee Attrition Dataset. I think this time around I’ll actually answer the question. :)
www.kaggle.com
See you soon.