Chasing Goodbyes
Tags: Logistic Regression, Human Resources, Machine Learning

By Arindam Roy | Project: Mechanic of Machine Learning

Employees leaving a firm and the turnover process is perhaps the costliest loss to a lot of industries. An employee can choose to exits a firm due to a multitude of factors, ranging form retirement, poaching, dissatisfaction to the good old getting fired. In this article, I discuss how machine learning can help us predict employee attrition and change the way we leave our jobs.

The Challenge

Predicting why an employee would leave is close to impossible. Often times it is not due to a single factor, but a cumulation of them. Thus, predicting the people who would leave has been anybody's guess until recently. But now, with improving focus on employees and machine learning, we can find trends in the data to search for groups who are more likely to leave. Further, attention to the issues faced by these people can significantly bring down the employee turnover rates. So, the hypothetical business problem I have taken up is: to develop a risk model that forecasts the likelihood of employee attrition in a business based on historic data. In this exercise, I have used data created by IBM (available on Kaggle) as an example. I have done an in depth analysis and solution development of this dataset using logistic regression. You can find the analysis on my GitHub and my notes on logistic regression in the same repository.

List of all features

Hypothesis

The functional idea behind this activity is that an employee’s chance of attrition can be predicted by the feedback and metrics maintained. Further insights might lie in the data of whereabouts of the employee, like the distance from office, married or not, age, rate of travel, et cetera. This is proven when we compare the mean p-value of the group that stayed vs the group that left the firm.

About Logistic Regression

Logistic regression is a way of predicting a binary value feature, by calculating the probability of the feature based on all other feature values and fitting it to a probability curve, trained on historical data (The other features may not be binary in nature and can have any nature). Though this method in theory, can also be used for multi discrete value features as well, it is not usually recommended, as there are better models to do so. (Please see notes on repository for further understanding.)

Roadmap to implement

Now that we have an understanding of logistic regression, allow me to walk you through the implementation process. Firstly we check for any unclean data (Null value, blanks, etcetera | There is none for this case). Then, we do visualizations and draw observations and assumptions. Post this, we split the data into our train and test sets. We fit the data to our train model and predict for our test model values of people going out.

My Observations and Assumptions (Please find in repository)

My fit results (Please find in repository)

Business Solution

By implementing the above model, a firm can predict the employees who are at risk of leaving at an accuracy of 84%. The company can work with these employees on the pain-points and reduce the attrition rate, retaining people, culture, value and lastly, cost. For some, it can be as simple as the distance of commute, for others, it can be a bunch of issues, but striving to keep each employee will surely produce results in the long run. This model gives great opportunity and insight into the workforce for any employer.

An Ugly Truth

I realize that that up till this point, this article is written as if there were no evil in the world, so I would like to discuss the downsides of the model. Any employer can flip the use of this solution and can try to find & fire those at risk, generating revenue and keeping the sure-shot long term employees (This is one of many adverse uses ). But I would like to remind that such cynicism often leads to catastrophe, thus I am optimistic that this model cannot be used in the upside down fashion for sustainable purposes.

Conclusion

Machine learning has and will change the way we do a lot of things. Leaving our jobs seems to be just another one in the bucket. The solution developed looks to be robust for a rather small dataset and can be brought to great use. Implementing it correctly will generate huge amount of value. Implementing it for the wrong reasons will surely be perilous for everyone involved.

About project Mechanic of Machine Learning:

I am a mechanical engineer by education. Now, I want to deep dive in the world of Machine Learning, hence the name, mechanic of ML :D. I have taken up this project to understand the in-depth mathematics involved in regularly used ML algorithms. Under this project, I will be sharing useful material and links as I explore this domain. The objective is to learn and spread the same. Stay tuned to my GitHub for updates!

References: