Attrition Prediction Using Machine Learning in WEKA
By: Akshay Gautam
Tags: Machine Learning, Employee Attrition, Weka, Towards Data Science, Data Science

Data Processing

Data processing is a technique in which we make the data useful by data cleaning, data transformation, and data reduction. The data used for this analysis is already cleaned, i.e., it does not have any missing values, but it is very noisy contains attributes like ‘EmployeeCount,’ ‘Over18’, and ‘StandardHours.’ These attributes have the same value for all the instances and do not contain any relevant information, so I decided to drop them for the analysis. The class variable in this dataset is highly imbalanced (84% majority class (‘no’) and 16% minority class (‘yes’)), so to oversample the minority class, I used SMOTE balancer in WEKA.

For this analysis, I will be using a k-fold cross-validation technique to build the model. After all, it is one of the most common techniques to build a predictive model, and it also helps to overcome over fitting in the model.

Feature Selection

For feature selection and analysis, I used the filtering method Information Gain attribute evaluation with the full training set. Information gain correlates all the attributes with the class variable and assign a value depending on their weight. It also helps us to evaluate the essential attributes, and with the help of this feature selection method, we can build our model with high accuracy and precision.

There are 35 attributes in this dataset from them; I dropped ‘Age,’ ‘Over18’, ‘EmployeeCount,’ and ‘StandardHours.’ Because they contain the same values for all the instance and does not contain any relevant information, we evaluated the top 20 attributes ( Below Figure) and applied our machine learning algorithms.

InfoGain Ranked Attributes

Classification Techniques

1. Bayes Net

Bayesian Network is the classification algorithm based on the Bayes’ theorem. We can call it an ‘independent feature model’ because it assumes that there is no relationship between the features (Datalab, 2018).

2. J48 — Decision Tree

The decision tree is a supervised machine learning algorithm that is used for classification as well as regression. It is the tree-like structure, which includes nodes (attributes), branch (rule), and leaf (outcome). The decision tree evaluates the best feature and then split the data into small samples and repeats this process until there is no attribute remaining and no more instances remaining.

3. K- Nearest Neighbor (K-NN)

K-NN is the machine learning algorithm that is widely used for classification and regression. It is a supervised machine learning algorithm. It is one of the most basic and essential algorithms used in different organizations. K-NN is lazy and non — parametric. By lazy, it means that it uses all the given data for the training purpose, and by non-parametric, it means that it does not make any assumption about the given data. This algorithm identifies the k-nearest neighbor using the distance proximity function to evaluate the unidentified data point. We are going to use it in our analysis because it builds the most accurate model and predictions. For this analysis, I will be using k=1 for our model.

4. Random Forest

Random forest is the machine learning algorithm that is used for classification and regression. It is a supervised machine learning algorithm. It creates many decision trees on a data sample and create a prediction for each of them and build the best solution through voting. It is one of the best algorithms to overcome overfitting in the model. Random forest is better than the decision tree because it can also work on the large dataset, and it also creates a model with high accuracy. Its total processing is based on ‘wisdom of crowds’ (Yiu, 2019).

Result Analysis

To evaluate the model, I will try to understand the confusion matrix of that particular algorithms. With the help of the confusion matrix, we can evaluate the performance of any model. It is a matrix which includes the following values:

True Positives — where the actual class is ‘yes’ and predicted as ‘yes’ by the model, that is, the model has correctly predicted these instances.

True Negative — where the actual class is ‘no’ and also predicted as ‘no’ by the model; that is, the model correctly predicted these instances.

False Positives — where the actual class is ‘no’ and predicted as ‘yes’ by the model; that is, the model wrongly predicted these instances.

False Negatives — where the actual is ‘yes’ and predicted as ‘no’ by the model; that is, the model wrongly predicted these instances. (Rawat, 2019)

Sensitivity — It is the proportion (probability) of the attired employees who were predicted as attired.

Specificity — It is the proportion (probability) of non-attired employees who were predicted as not attired.

I will be using these values for the evaluation of our model. The section below is the evaluation after feature selection.

1. Bayes Net

In this algorithm, the correctly classified instances are 1954, and the accuracy rate is 89.6%. This algorithm took 0.12 sec to build. Below figure shows the confusion matrix of this algorithm.

True positives — 721 instances

True negatives — 1233 instances

False positives — 0 instances

False Negative — 227 instances

Sensitivity — 76.1%

Specificity — 100%

Bayes Net — Confusion Matrix

2. J48 — Decision Tree

1. In this algorithm, the correctly classified instances are 1836, and the accuracy rate is 84.2%. This algorithm took 0.19 sec to build and created the pruned tree. The size of the tree is 213. While the number of leaves is 124. Below figure shows the confusion matrix generated by this algorithm.

True positives — 734 instances

True negatives — 1102 instances

False positives — 131 instances

False Negative — 214 instances

Sensitivity — 77.4%

Specificity — 89%

J48 — Confusion Matrix

3. K-Nearest Neighbor

In this algorithm, the correctly classified instances are 1776, and the accuracy rate is 81.4%. In this model, we selected ‘1’ nearest neighbor that is k=1. This model took 0 sec to build. Below figure shows the confusion matrix generated by this algorithm.

True positives — 787 instances

True negatives — 989 instances

False positives — 244 instances

False Negative — 161 instances

Sensitivity — 83%

Specificity — 80.2%

K-NN — Confusion Matrix

4. Random Forest

In this algorithm, the correctly classifies instances are 1966, and the accuracy rate is 90.14%. This model took 0.83 sec to build, and it performed 100 iterations. Below figure shows the confusion matrix generated by the random forest classifier.

True positives — 786 instances

True negatives — 1180 instances

False positives — 53 instances

False Negative — 162 instances

Sensitivity — 82.9%

Specificity — 95%

Random Forest — Confusion Matrix

Comparison — Graphical Representation

Comparison between different measures

Above figure represents the comparison between the different measures of four different machine learning algorithms. As we can see from the graph, the Random forest shows the highest accuracy rate than the others. On the other hand, K-nearest neighbor shows the lowest accuracy rate. Bayes net shows the highest precision value ‘1’ while the k-nearest neighbor shows the lowest precision value. If we compare in terms of accuracy, the random forest is performing better in predicting the class variable that is attrition of the employee. However, in terms of precision, the Bayes net is performing better with the value ‘1’.

Comparison of accuracy after feature selection

As seen in above figure, after applying feature selection, there is a significant change in the accuracy of our models. There is 8.2%, 1.7%, 1.6%, and 4.4% increment in the accuracy of the models Bayes Net, J48, K-NN, and Random Forest, respectively. The highest increment in the accuracy is in the Bayes Net, while the lowest increment is in the K-NN.

Conclusion

In this analysis, we tried to predict employee attrition using machine learning. To come up with the best model is quite challenging because these data and databases are entirely redundant. We found out that the random forest is performing better than the other algorithms with 90.1% accuracy. So random forest is recommended to all the companies who are trying to predict their employee attrition.

Thank you for reading.

References

Datalab, A. (2018, December 4). What is Bayesian Network Classifier? Retrieved from Medium: https://medium.com/@analyttica/what-is-bayesiannetwork-classifier-4d2771f91f63

Rawat, S. (2019, October 3). Is accuracy EVERYTHING? — Various metrics to evaluate the classification model. Retrieved from Medium: https://towardsdatascience.com/is-accuracyeverything-96da9afd540d

Yiu, T. (2019, june 13). Understanding Random Forest — How the Algorithm Works and Why it Is So Effective. Retrieved from Towards data science website: https://towardsdatascience.com/understandingrandom-forest-58381e0602d2