How I improved a Class Imbalance problem using sklearn’s LinearSVC
Tags: Linearsvc, Python, Artificial Intelligence, Class Imbalance, Data Science

How I improved a Class Imbalance problem using sklearn’s LinearSVC Tracyrenee Follow Dec 20 · 6 min read

Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.

The advantages of support vector machines are:

Effective in high dimensional spaces.

Effective in cases where number of dimensions, or features, is greater than the number of samples.

Uses a subset of training points in the decision function, called support vectors, so it is also memory efficient.

Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.

SVMs can be used on datasets where there is a class imbalance by implementing the class_weight parameter in the fit method. Below is a diagram of a dataset that has weighted and unweighted boundaries:-

For large datasets, LinearSVC is the most appropriate model to use, and is the model that was implemented in this post for training, fitting, and predicting on the large dataset used, which was was an Analytics Vidhya Hackathon question concerning the likelihood of having a stroke, the link being found here:- McKinsey Analytics Online Hackathon — Healthcare Analytics (analyticsvidhya.com)

The problem statement for this competition question reads as follows:-

“Your Client, a chain of hospitals aiming to create the next generation of healthcare for its patients, has retained McKinsey to help achieve its vision. The company brings the best doctors and enables them to provide proactive health care for its patients. One such investment is a Center of Data Science Excellence.

In this case, your client wants to have study around one of the critical disease “Stroke”. Stroke is a disease that affects the arteries leading to and within the brain. A stroke occurs when a blood vessel that carries oxygen and nutrients to the brain is either blocked by a clot or bursts (or ruptures). When that happens, part of the brain cannot get the blood (and oxygen) it needs, so it and brain cells die.

Over the last few years, the Client has captured several health, demographic and lifestyle details about its patients. This includes details such as age and gender, along with several health parameters (e.g. hypertension, body mass index) and lifestyle related variables (e.g. smoking status, occupation type).

The Client wants you to predict the probability of stroke happening to their patients. This will help doctors take proactive health measures for these patients.”

In order to solve this problem I created an .ipynb file using Google Colab, a great, free online Jupyter Notebook that I can use from any computer that has internet access. Many libraries are already installed on Google Colab, so I only needed to import them into the program, being pandas, numpy and seaborn.

I downloaded the train and test files from Analytics Vidhya’s competition and saved them into my personal GitHub account. Once saved, I loaded them and read them into the .ipynb file I had created:-

I checked for any null values and found that the columns “bmi” and “smoking_status” had many missing values that needed to be imputed:-

I imputed the missing values by replacing any missing values in the “bmi” column with the median value. I replaced any missing values in the “smoling_status” column with the most commonly used value, called mode().

I then put the target, or “stroke” column on a graph using seaborn’s distplot() function. When the target, being a dependant variable, is put on a graph it is clearly evident that there is a large class imbalance in this competition question:-

I counted the number of examples and found only 1.8% of the values were class 1, revealing special measures need to be taken to address this class imbalance:-

I used a preprocessing measure by checking to see which columns were categorical and put them on an ordinal encoder, thereby transforming them into numeric values:-

I defined the dependant and independent variables, which will be used to make predictions. The target variable, train.stroke, is defined as y. The independent variables are X and X_test. The independent variables are composed of the train and test files with “id” and “stroke” dropped from the train file and “id” dropped from the test file:-

I put the values in X and X_test dataframes on a scaler, using sklearn’s StandardScaler() function. Scaling is necessary to put the data in the same range to optimise the predictions that will be made on it:-

It is always a good idea to to graphically view the data as to how it appears in the computer’s memory, so I created a three dimensional graph of this information. As can be seen, the data represents a huge mass of zeros and the small fraction of ones are scattered within the mass. As can be seen, there are no clear lines of demarcation separating these two classes, which makes it difficult to make accurate predictions:-

After I created the three dimensional graph of the target variable, I split the X dataframe into training and validating sets using sklearn’s train_test_split() function. I set the parameters up to signify there is a class imbalance by setting stratify=True.

I put y_val, being the validation set target variable, on a graph and it illustrated the class imbalance that will be present when the validation set is predicted on:-

I defined the model and chose LinearSVC because it is suitable for large datasets. I set the parameters up to attain maximum efficiency: class_weight=’balanced’, dual=True,max_iter=680, and C=10. I achieved an accuracy of 96.59% using this parameter tuning:-

I predicted on the validation dataset and also achieved an accuracy of 96.59%!

I put the prediction, y_pred, on a graph and was pleased to see that the class imbalance had been addressed. The model had picked up 80 ones as opposed to y_val actually having 78:-

I decided to plot a graph of the correct examples versus the incorrect ones. The correct examples are portrayed as the purple dots while the incorrect ones are yellow:-

I predicted on the test X_test dataset and was pleased that the class imbalance had been addressed when I put the prediction on a graph:-

When I checked the predictions on Analytics Vidhya’s solution checker I found that I scored 54.99, which is an improvement on the previous model I had used, CatBoost. If I had adjusted the parameters on the LinearSVC() model I could have achieved a higher accuracy, but the class imbalance would not have been accurately addressed:-

The code for this program can be found in its entirety in my personal GitHub account, the link being found here:- AV-Stroke/AV_Stroke_LinearSV.ipynb at main · TracyRenee61/AV-Stroke (github.com)