Predicting HR Attrition using Support Vector Machines | by Arpan Mishra | Towards Data Science

Following my previous post, I will be covering how we can apply Support Vector Machines using python and Scikit-Learn on a real world dataset. This data was taken from a recent InClass hackathon hosted by IIT, Guwahati on Kaggle as part of their Summer Analytics 2020 capstone project, you can download it from here.
In this post, you will see how easy it is to train and tune SVM and make them work for your problem statement. Using this I ranked 29th in the hackathon (top 4%) with very minimal pre-processing and feature engineering.
Before we proceed, you must have an intuitive understanding of how SVM works. I would suggest you go through my previous post for an in-depth explanation of this algorithm.
towardsdatascience.com
Now assuming that you understand the theory behind support vector machines ...
As the COVID-19 keeps unleashing its havoc, the world continues to get pushed into the crisis of the great economic recession, more and more companies start to cut down their underperforming employees. Companies firing hundreds and thousands of Employees is a typical headline today. Cutting down employees or reducing an employee salary is a tough decision to take. It needs to be taken with utmost care as imprecision in the identification of employees whose performance is attriting may lead to sabotaging of both employees’ career and the company’s reputation in the market.
Aim: To predict Employee Attrition by the given data about his/her past history.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.model_selection import StratifiedKFoldfrom sklearn.compose importmake_column_transformerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import GridSearchCV,RandomizedSearchCVfrom sklearn.svm import  SVCfrom sklearn.decomposition import PCAfrom xgboost import XGBClassifierfrom sklearn.model_selection import cross_val_scorefrom sklearn.preprocessing import StandardScaler,RobustScalerfrom sklearn.preprocessing import OneHotEncoder,LabelEncoderfrom sklearn.pipeline import make_pipelinepd.set_option('display.max_columns', 500)pd.set_option('display.max_rows', 1000)
To preserve the original training data I have made a copy of it.
training = pd.read_csv('train.csv')train_data = training.copy()test_data = pd.read_csv('test.csv')
train_data.info()
We have 1628 observations and 29 features, out of which 22 are integers and 7 are objects. Some of the integer data type features might also be categorical. We have to predict Attrition which can either be 0 or 1 (1 if the employee left the company).
train_data.head()
Let's print out some statistics of our variables
The weird thing to note here is that Behaviour has 0 standard deviation and mean=min=max = 1. This means that this columns has a value of 1 for all the observations, so we will drop it. We will also drop Id because it will have unique values for all the employees.
train_id = train_data.Idtrain_data = train_data.drop(['Behaviour','Id'],axis = 1)test_id = test_data.Idtest_data = test_data.drop(['Behaviour','Id'],axis = 1)
On exploring a little further I found that the feature PerformanceRating has only two values, 3 or 4 so I have mapped them to 0 and 1 respectively.
train_data['PerformanceRating'] = train_data['PerformanceRating'].apply(lambda x: 0 if x == 3 else 1)test_data['PerformanceRating'] = test_data['PerformanceRating'].apply(lambda x: 0 if x == 3 else 1)
Let's check the distribution of our target variable Attrition.
train_data[‘Attrition’].value_counts().plot(kind = ‘bar’)
We have more or less a balanced distribution.
Checking for duplicates
print('Number of duplicates: ',train_data.duplicated().sum())
train_data[train_data.duplicated()]['Attrition'].value_counts().plot(kind = 'bar')
Alright, so the data has 628 duplicates and all the duplicates correspond to Attrition 1. So that means that data was oversampled to make it balanced. We will drop the duplicates now and check the distribution again.
train_unq = train_data.drop_duplicates()print('New train set: ',train_unq.shape)X = train_unq.drop('Attrition',axis = 1)y = train_unq['Attrition']y.value_counts().plot(kind = 'bar')plt.show()
Now our training data has 1000 data points and the target variable is imbalanced. There are many ways to tackle imbalanced data sets like upsampling or downsampling using SMOTE.
We can also use a cross validation strategy such as stratified k-fold which keeps the distribution of our target variable (here Attrition) similar across the folds. The training and validation data is split using stratified sampling instead of random sampling. The stratas here are the two values of our target variable. If you don’t understand what this means then don’t worry about it just remember that it is an effective method to tackle imbalanced datasets while we train our model. You can learn more about it in the scikit-learn user guide for cross validation here.
We will follow the following steps now:
Drop performance rating (on exploration I found that 85% values were of a single class this might lead to overfitting)
One Hot Encode all the ‘object’ data type features
Use standard scaling on all the integer data type features.
Use the pre-processed data and split it using Stratified K-Fold.
Fit and validate with 3 candidate models: Random Forest, XGBoost and Support Vector Classifier
These are a lot of steps. Well, scikit-learn allows us to do all of this in a few simple lines of codes using pipelines and column transformers.
So, first, we make a pipeline for pre-processing
categorical = [f for f in training.columns if training[f].dtype == object]numeric = [f for f in X.columns if f not in categorical+['Id',','Behaviour','PerformanceRating']]pre_pipe = make_column_transformer((OneHotEncoder(),categorical),(StandardScaler(),numeric))
The variable categorical stores all column names of object data type features and numeric stores all the integer data type columns.
Now we define the pipeline for the models and print out the cross validation scores of the different models. Using cross_val_score we can pass “skf” into “cv” parameter and it will take care of splitting iterating and calculating the scores for us. I have used ‘roc_auc’ as the scoring metric as the data is imbalanced. For a better understanding of the parameters for cross_val_score check out its documentation.
# random forest pipelinepipe_rf = make_pipeline(pre_pipe,RandomForestClassifier())#xgboost pipelinepipe_xgb = make_pipeline(pre_pipe,XGBClassifier()) # SVM pipelinepipe_svc = make_pipeline(pre_pipe,SVC(probability=True))print(‘RF:‘,np.mean(cross_val_score(X=X,y=y,cv=skf,estimator=pipe_rf,scoring=’roc_auc’))) print(‘XGB:‘,np.mean(cross_val_score(X=X,y=y,cv=skf,estimator=pipe_xgb,scoring=’roc_auc’))) print(‘SVC:’,np.mean(cross_val_score(X=X,y=y,cv=skf,estimator=pipe_svc,scoring=’roc_auc’)))
Clearly, support vector classifier performs better than our tree based models. It is possible that Random Forest and XGBoost may perform better after removing more features and tuning but in this post we will use SVM.
After One hot encoding we had 46 features in total in the dataset, as the dimensionality increases there is more chance of overfitting and there might be irrelevant features as well. So I decided to go with PCA to reduce the dimensionality.
So, first I took 46 components and included PCA in our pipeline. I have visualised the cumulative explained variance.
n = 46pipe_svc = make_pipeline(pre_pipe,PCA(n_components=n),SVC(probability=True,C = 1,kernel='rbf'))plt.figure(figsize=(10,8))pipe_svc.fit(X,y)plt.plot(range(1,n+1),pipe_svc.named_steps['pca'].explained_variance_ratio_.cumsum())plt.xticks(range(1,n+1,2))plt.title('Explained Variance')
We can see that about 34 components explain 100% variance. So we take 34 components and fit our model.
n = 34pre_pipe = make_column_transformer((OneHotEncoder(),categorical),(StandardScaler(),numeric),remainder = 'passthrough')pipe_svc = make_pipeline(pre_pipe,PCA(n_components=n),SVC(probability=True,C = 1,kernel='rbf'))print('SVC: ',np.mean(cross_val_score(X=X,y=y,cv=skf,estimator=pipe_svc,scoring='roc_auc')))
Our score has reduced by 0.01 but we have considerably reduced the dimensions and there is now higher chance that this model will perform better on unseen data.
To squeeze out every bit of performance from the model, it is now time to tune our SVM model. we will use GridSearchCV for this if you`re not familiar with it take a look at the documentation.
The parameters that can we tune are:
C: This is the regularization parameter
Kernel: Linear, Polynomial or RBF (Gaussian)
Gamma: Equivalent to the variance term in the Gaussian kernel
Class Weight: Balances the weights of the classes in target variable (weight is inversely proportional to class frequency)
There are more parameters but these are the important ones that affect performance, take a look at the scikit-learn documentation before going further.
Tuning is also a very easy process with pipelines, we just define out parameter grid and pass the entire pipeline into grid search. I have not tuned kernel here. Since ‘rbf’ is the most used kernel and works for most of the tasks, I will just use it to save on some compute time. You can go ahead and try tuning kernel as well.
param_grid = {        'svc__C':[0.001,0.01,0.1,1,10,100,1000],    'svc__gamma': ['auto','scale'],    'svc__class_weight': ['balanced',None]}grid_search = GridSearchCV(pipe_svc,param_grid=param_grid,cv = skf, verbose=2, n_jobs = -1,scoring='roc_auc')grid_search.fit(X,y)print('Best score ',grid_search.best_score_)print('Best parameters ',grid_search.best_params_)best_svc = grid_search.best_estimator_
You may have noticed that I have prepended ‘svc__’ in front of all the parameters in the parameter grid, that is done just to specify which step of our pipeline are we tuning in the grid search as we are not just passing an estimator into GridSearchCV but an entire pipeline. I`m sure there might be better ways to do this, let me know in the comments if you know of better techniques.
We can try tuning our model further to find the best C by just tweaking the range of C and fixing the values of other hyperparameters like so
pipe_svc = make_pipeline(pre_pipe,PCA(n_components=n),SVC(probability=True,C = 1,kernel='rbf',class_weight=None,gamma='auto'))param_grid={    'svc__C':[0.01,0.03,0.05,0.07,0.1,0.3,0.5,0.7,1]  }grid_search = GridSearchCV(pipe_svc,param_grid=param_grid,cv = skf, verbose=2, n_jobs = -1,scoring = 'roc_auc')grid_search.fit(X,y)print('Best score ',grid_search.best_score_)print('Best parameters ',grid_search.best_params_)best_svc = grid_search.best_estimator_
The final mean validation score now is 0.8319. Time to make our submission on Kaggle.
best_svc.predict_proba(test_data)[:,1]submission = pd.DataFrame(prediction,columns=['Attrition'])submission['Id'] = test['Id']submission = submission[['Id','Attrition']]submission.to_csv('submissionfile_SVC.csv',index = None)
And voila !
As you can see that my rank dropped 4 places once the private leader board was revealed. This happened because my model overfit on the training data, I should have probably not gone too crazy tuning “C”. Better feature selection might also help in improving the score. What do you think? Feel free to go through my Kaggle notebook and execute the code.
Thank you if you stuck around till the end. I hope you found this post useful and learnt something new from this. Please provide valuable feedback in the comments, I would love to know your ideas on improving the performance.