Canada CRS Draw Prediction using Machine Learning (regression models)
Tags: Case Study, Python, Exploratory Data Analysis, Machine Learning, Regression

Can AI predict the next express entry score?

Yes you heard it right! In this blog we’ll try to solve a unique problem of predicting the Canada immigration CRS score that has never been tried before, because of one simple but a big problem that is, no data availability!

I would like to take this opportunity to take you to the whole journey of fetching data, exploring data, feature engineering and finally experimenting with various ML models before arriving at our final solution.

The blog is structured in the following manner for ease of access:-

1.Introduction

2 Business Problem

3 Data

4 Mapping to Machine Learning problem

5. First cut solution

6. Data Loading and Preprocessing

7. Exploratory Data Analysis

8. Feature Engineering

9. Modeling Preparation and Modeling

10. Results

11. Future Work

12. Profiles for future connect

13. References

So, let’s begin.

1. Introduction

Immigration across various countries is on a rise since the last few years. Each country has its own immigration point-based system and this case study will focus on Canada Immigration. Canada is among the top-most choices for immigration because of its ease and transparency in the whole process and this case study will focus on the same.

2. Business Problem

2.1 Description

The most popular Canada Immigration program is the Express Entry (Federal Skilled Worker) program which is completely a point based system. Candidates create their profiles in FSW (Federal skilled worker path) under the express entry program and each candidate is given a score by Canada’s govt. which is known as Comprehensive Ranking Score (CRS) which depends on candidates’ factors like age, education, work experience, IELTS score and more. All the candidates’ profiles are there in the express entry pool, with few profiles expiring and many new profiles adding.

2.2 Context

Canada ‘s IRCC (immigration and refugee) wing holds a draw in almost every 15 days and releases a CRS Cut-off score along with the number of candidates that this particular draw can influx. The problem is that the candidates do not know what the next CRS cut-off score would be and they wait for weeks or at times for months to get selected for the express entry immigration program.

In this case study, I will try to create a (Time Series) Regression Predictive ML model that can help candidates with the next cut-off score prediction. This will help thousands of candidates across the globe waiting in the pool to get prepared in advance. For example, if a candidate gets to know that he will not be at-par with the next draw then instead of wasting time, he can either try to improve his CRS-score by re- writing IELTS exam or he can plan moving to other Canada Immigration programs which supports lower CRS scores. Also, an indication whether the score would go up or down from the existing score will be quite a useful insight for candidates waiting for the draw.

2.3 Problem Statement:

Predict the next CRS Cut-Off score by researching and collecting the relevant data in the express entry pool.

2.4 Business objectives and constraints

· Good accuracy requirement.

· No low-latency requirement as such.

3. Data:

3.1 Problem for Data collection

Despite being one of the most searched topics under immigration, I could not find any existing relevant dataset for CRS score prediction. The problems:-

Data not available only on single website but multiple websites such as:-

https://www.canada.ca/en/immigration-refugees-citizenship/services/immigrate-canada/express-entry/submit-profile/rounds-invitations/results-previous.html

Aforementioned is Canada’s official immigration website that gives out the information on current and past CRS- cut off scores.

Another 5,000 ITAs in December 9 Express Entry draw | Moving2Canada

COVID-19 is a global pandemic and the Government of Canada has enacted exceptional measures to mitigate the spread of…moving2canada.com

This is another famous immigration website link that shows relevant information in a simple manner

https://open.canada.ca/data/dataset/593e9165-c6ce-4f9b-b519-03d315f92cd4?activity_id=5a1a7a1f-e0c1-4420-9bb9-2813806fe00e#wb-auto-6

I have tried to fetch data from all the aforementioned Canada’s official websites.

Also from websites such as www.facebook.com , www.images.google.com and www.youtube.com

The data is not present in a single format. For example:- some data is present in an image format, some as html doc, some as text on a website and some information is presented in a video.

Different attributes had to be collected from different data sources.

3.2 Data Collection Snapshots

This is the kind of data which is mentioned on most of the sites. The first column represents Draw Number, Second Column represents the date of draw, third the number of ITAs issued, and fourth column represents the cut-off score.

We need more attributes such as the number of candidates waiting in the pool with the score range. Now, this part of data is really hard to pull off because no website maintains such historic data, so, I had to do a detailed online research for this on facebook.com, google.com and also on youtube.com to fetch data like:-

The first column in the above pic represents CRS score range of candidates and the second column represents the number candidates with the scores inside this range.

Now, this kind of data is not available on a single platform with same format, so I also had to play YouTube videos from immigration channels to fetch the data such as :-

www.youtube.com

3.3 Basic Data Preparation

After facing the data collection challenges, It was obvious that this process could not be automated for aforementioned reasons and a lot of manual hard work was required. Also, since the number of rows would be small so I decided to go for it and prepared the following dataset in an excel sheet after spending almost 5 days.

I gathered all the data from different sources for only FSW from 2018 till date and calculated the scores within 450 and above because as per the business problem understanding the score of 450 is like a separator. Score of 450 or above reflects higher probability of selection when compared to that of a lower score than 450.

3.4 Data overview

There are 60 rows(25 draws in 2018, 24 draws in 2019, 11 draws in 2020 due to Covid-19, approximately one draw per 15 days except in 2020) and 6 columns as of now, namely:-

S.No. — The draw number Date- The date on which draw took place Below_450- Total number of candidates in the pool that has scores within 450 Above_450- Total number of candidates in the pool that has scores above 450 Total- Total number of candidates in the pool before the cut-off Cut_off — the cut-off score for a particular draw.

4. Mapping the real-world problem to an ML problem:

4.1Type of Machine Learning Problem

There are 60 rows of data present with the class label as cut-off, we have to predict what the next cut-off score would be. So, this problem can be posed as a regression problem.

4.2 Performance Metric

The most suitable KPIs for our regression problem are:-

MSE (mean squared error) MAE (mean absolute error)

4.3 Machine Learning Objectives

Objective: Predict the score (integer value) for each data-point and minimize the MSE and MAE

5. First cut solution

Despite being one of the most searched topics under immigration, I could not find any existing solution/ research paper for CRS score prediction. As the objective of the model is to predict the CRS score, this problem is a straight forward regression problem. So, I would first build a random model, then a baseline line model and then would try my best to reduce the metrics by experimenting with various complex models such as RF, GBDT and more. However, to apply any machine learning algorithm on the dataset, basically there are 5 general steps:

Load the dataset EDA and Feature Engineering Instantiate and Fit the model to the training dataset Prediction on the test set Evaluate — Calculate MSE and MAE for our models and compare them

So, let’s roll up the sleeves and jump to coding

6. Data Loading and Preprocessing

First, let us import the necessary libraries.

import pandas as pd

import matplotlib.pyplot as plt

import numpy as np

import scipy as sp

import seaborn as sns

from sklearn.linear_model import LinearRegression

from sklearn.metrics import mean_squared_error

from sklearn.metrics import mean_absolute_error

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

from sklearn.preprocessing import MinMaxScaler

from sklearn.model_selection import GridSearchCV

Next, we load the data from excel file into a pandas dataframe

data = pd.read_excel(‘Dataset_cs1.xlsx’)

data.head()

This is the final data after removing null valuesand rectifying date column as it was not formatted properly while data collection and we had to convert it into string before making any changes. Following are the code snippets for the same:-

date_1 = date_1.apply(lambda x: x.strftime(‘%Y-%m-%d’))

date_2 = date_2.apply(lambda x: x.strftime(‘%Y-%m-%d’))

date_3 = date_3.apply(lambda x: x.strftime(‘%Y-%m-%d’))

print(date_3)

Finally, concatenating rectified date series to data and this is how data looks now:-

data = pd.concat([data, date], axis=1)

data.head()

7. Exploratory Data Analysis

After data loading and preprocessing in which I rectified the date column, let’s begin EDA. It is always a good idea to get the data overview by using .corr() method.

Next, I am Visualizing the correlation of features with target variable using a scatter plot. Code snippet:-

#scatter plot for below_450 and CRS

plt.scatter(data[‘below_450’], data[‘CRS’])

plt.xlabel(‘number of applicants below the score 450’)

plt.ylabel(‘Cut-off score’)

plt.title(‘relation b/w # candidates below 450 and cut-off score’)

plt.show()

Similarly, I have done it for other features and following are my initial observations:-

We can see from first plot that the CRS inclined only after crossing 100000 applicants in the pool and then it is a positive correlation In the second plot, we observe initially that even if there is a slight increase in the above 450 scoring candidates then there is a very strong inclination but gradually it maintains a steady increasing path Conclusion:- there is strong correlation between the scores of candidates and cut-off

Let’s check the relationship between total number of candidates and CRS

# total number of candidates and CRS

#scatter plot for above_450 and CRS

plt.scatter(data[‘total’], data[‘CRS’])

plt.xlabel(‘total number of applicants in pool’)

plt.ylabel(‘Cut-off score’)

plt.title(‘relation b/w total # candidates and cut-off score’)

plt.show()

Observation:-

As stated by the numbers above in corr(), there is a strong correlation between total candidates waiting in the pool and the cut-off score.

Let’s plot graphs to understand what happens to our features as the date increases. To plot this, I have used the concept of subplot:-

plt.figure(figsize=(16,12)) plt.subplot(221)

plt.scatter(data[‘Date’], data[‘CRS’], c= ‘b’, label = ‘CRS with date’)

plt.xlabel(‘date’)

plt.ylabel(‘CRS’)

plt.legend()

plt.title(‘relation b/w cut-off and date’) plt.subplot(222)

plt.scatter(data[‘Date’], data[‘total’],c= ‘r’, label = ‘# ppl before draw’)

plt.xlabel(‘date’)

plt.ylabel(‘ total candidates in pool’)

plt.legend()

plt.title(‘relation b/w total ppl in pool vs date’) plt.subplot(223)

plt.scatter(data[‘Date’], data[‘below_450’],c= ‘g’, label = ‘# ppl who scored below_450 before draw’)

plt.xlabel(‘date’)

plt.ylabel(‘ total candidates in pool’)

plt.legend()

plt.title(‘relation b/w total ppl in pool vs date’) plt.subplot(224)

plt.scatter(data[‘Date’], data[‘above_450’],c= ‘y’, label = ‘# ppl who scored above_450 before draw’)

plt.xlabel(‘date’)

plt.ylabel(‘ total candidates in pool’)

plt.legend()

plt.title(‘relation b/w total ppl in pool vs date’) plt.show()

Observation:-

From the above two graph rows we realized that as the time increased, more and more people started entering the pool and scored better too resulting in higher cut-offs

For overall observation, plotting pair-plot.

Following is the code snippet:-

sns.set_style(“whitegrid”) sns.pairplot(data, hue=”CRS”, size=3) plt.show()

There are some very interesting observations from Pair-wise plot :-

CRS score goes approx. above 470 when number of ppl below_450 in pool are approx 120000 CRS score lies aprx. between 457 and 468 when ppl below_450 are between 105000 and 120000 CRS score lies below 460 if ppl below_450 are less than 100000 CRS score is above 470 if #ppl with 450_above score are more than aprx 10000 CRS score is between 457 and 468 when ppl above_450 are between 5000 and 10000 CRS score lies below 460 if ppl above_450 are less than 5000 CRS score is above 470 if total #ppl in pool are more than aprx 125000 CRS score is between 470 and 457 if total #ppl in pool are aprx between 105000 and 125000 CRS score is lower than 457 if total #ppl in pool are less than 105000

Next, Let’s go for univariate analysis

below_450

Plotting Distribution plot:-

sns.FacetGrid(data, size=5).map(sns.distplot, “below_450”) plt.title(‘below_450 distribution plot’) plt.show()

Observation:-

Most of the candidates who scored more than 450 were below 6000 in numbers

Total

Distribution plot:-

sns.FacetGrid(data, size=5).map(sns.distplot, “total”) plt.title(‘total distribution plot’) plt.show()

Observation:-

Though the range of candidates waiting in pool is not very distinctive probably because the number of candidates were less in the beginning and gradually were increasing. However, candidates with scores in the range 70k to 90k has been the highest range so far.

Want to check the year wise distribution in the number of candidates in the pool, so plotting bar plot

Let’s checkout Bar Plot:-

With the help of bar plot lets check the total number of candidates in each year:-

plt.bar(2018 + 0.00, data[‘below_450’][data[‘Date’].dt.year == 2018].sum(), color = ‘b’, width = 0.25, label= ‘below 450’) plt.bar(2018 + 0.25, data[‘above_450’][data[‘Date’].dt.year == 2018].sum(), color = ‘g’, width = 0.25, label= ‘above 450’) plt.bar(2018 + 0.50, data[‘total’][data[‘Date’].dt.year == 2018].sum(), color = ‘r’, width = 0.25, label= ‘total’) plt.bar(2019 + .00, data[‘below_450’][data[‘Date’].dt.year == 2019].sum(), color = ‘b’, width = 0.25) plt.bar(2019 + .25, data[‘above_450’][data[‘Date’].dt.year == 2019].sum(), color = ‘g’, width = 0.25) plt.bar(2019 + .50, data[‘total’][data[‘Date’].dt.year == 2019].sum(), color = ‘r’, width = 0.25) plt.bar(2020 + .00, data[‘below_450’][data[‘Date’].dt.year == 2020].sum(), color = ‘b’, width = 0.25) plt.bar(2020 + .25, data[‘above_450’][data[‘Date’].dt.year == 2020].sum(), color = ‘g’, width = 0.25) plt.bar(2020 + .50, data[‘total’][data[‘Date’].dt.year == 2020].sum(), color = ‘r’, width = 0.25) plt.title(‘year wise distribution of number of candidates in each group’) plt.ylabel(‘number of candidates’) plt.xlabel(‘candidates in 2018, candidates in 2019, candidates in 2020’) plt.legend() plt.show()

Observations:-

we can see the rise in candidates applying for immigration each year. However, there’s a decline in the number of candidates in 2020 because of restrictions due to covid-19. Intrestingly, there’s a significant rise in the applicants who are scoring above 450 and hence this probably is the reason for high cut-offs by every passing year. Let’s dig deep in cut-off scores’ statistics

PDF and CDF:-

Plotting PDF and CDF for in depth understanding of our score features

let’s plot PDF and CDF of our features below_450, above_450 and Total

Following is the code snippet for one and similarly we plot for other features:-

counts, bins= np.histogram(data[‘below_450’])

counts,bins

pdf= counts/sum(counts)

cdf= np.cumsum(pdf)

plt.plot(bins[1:], pdf, label= ‘pdf’)

plt.plot(bins[1:], cdf, label= ‘cdf’)

plt.title(‘below 450 pdf’)

plt.xlabel(‘number of candidates who scored less than 450’)

plt.legend()

plt.show()

Observations:-

1.90% of the people who scored less than 450 are around 120000 in numbers

2. More than 50% of the candidates of all who have scored more than 450 are less than 5000 in numbers and approx. 90% of the people are less than 22500

3. 90% of the total candidates are less than 150000 in the pool

Now, Let’s calculate the average and median scores :-

#lets check statistics of above graphs

print(‘mean:’)

print(‘below_450:’,np.mean(data[“below_450”]))

print(‘below_450:’,np.mean(data[“above_450”]))

print(‘total:’,np.mean(data[“total”]))

print(‘*******************************’)

print(‘Standard deviation:’)

print(‘below_450:’,np.std(data[“below_450”]))

print(‘above_450:’,np.std(data[“above_450”]))

print(‘toal:’,np.std(data[“total”]))

And below is the code snippet to calculate quantiles and percentiles:-

print(“Quantiles:”) print(‘below_450:’,np.percentile(data[“below_450”],np.arange(0, 100, 25)))

print(‘above_450:’,np.percentile(data[“above_450”],np.arange(0, 100, 25)))

print(‘total:’,np.percentile(data[“total”], np.arange(0, 100, 25))) print(“90th Percentiles:”) print(‘below_450:’,np.percentile(data[“below_450”],90))

print(‘above_450:’,np.percentile(data[“above_450”],90))

print(‘total:’,np.percentile(data[“total”], 90))

Conclusion of EDA

1.Number of draws have declined in 2020, most probably because of covid-19 scare.

2.We can approximate the average frequecy of draws to 15 days.

3.Our assumption for score separator was correct for 450 as per business case study because the avg and median scores are both around 450.

4.Although the number of people have reduced in the pool but people who have scored more than 450 have increased at good pace.

5.This implies that this feature of above_450 is also one of the most crucial features because avg and median scores each year have increased just as above_450 feature.

6.Though statistically by calculating Corr() it is valued less than total and below_450 that is probably because of lesser numbers of draws in 2020, but we’ll keep this feature as important as others.

7.Almost all the values in all the features have had similar mean and median values, this implies that the data is well set and there are almost no outliers except a couple of values in CRS.

8.There’s a strong correlation between features and CRS score, also among the features themselves.

9.Few Statistics:-

1.CRS score goes approx. above 470 when number of ppl below_450 in pool are approx 120000

2.CRS score lies aprx. between 457 and 468 when ppl below_450 are between 105000 and 120000

3.CRS score lies below 460 if ppl below_450 are less than 100000

4.CRS score is above 470 if #ppl with 450_above score are more than aprx 10000

5.CRS score is between 457 and 468 when ppl above_450 are between 5000 and 10000

6.CRS score lies below 460 if ppl above_450 are less than 5000

7.CRS score is above 470 if total #ppl in pool are more than aprx 125000

8.CRS score is between 470 and 457 if total #ppl in pool are aprx between 105000 and 125000

9.CRS score is lower than 457 if total #ppl in pool are less than 105000

mean: below_450: 99598.17 below_450: 7769.09 total: 107367.25

Standard deviation: below_450: 15632.69 above_450: 8513.50 toal: 23264.110

10. Next, I’d like to see if by doing feature engineering such as the number of days between each draw and others can be of some value.

8. Feature Engineering

Leveraging time series, we are creating window features wherein we take the last 3 feature values average by calculating the window features i.e. last 3 average values for below_450, above_450, total and CRS

Following is the code snippet that I had used to create these 4 additional window features:-

# calculating the window features i.e. last 3 average values for below_450, above_450, total and CRS lists = [[], [], [], []]

cnt=-1 for col in range(1, data.shape[1]-1):

cnt= cnt+1



for row in range(3, data.shape[0]):

temp_list = []



for z in range(row-3, row):

temp_list.append(data[data.columns.values[col]][z])



lists[cnt].append(mean(temp_list))



print(len(lists[0]))

I calculated the average of last 3 values only because our dataset is small and this is how our data looks after adding 4 window features:-

Let’s see how our added features are statistically described:-

Observation:-

The values in all three above features do not deviate much from original features as these are calculated from their means

Adding one more feature representing the gap between two consecutive draws as this will help in understanding the draws frequency and might be a very helpful feature:-

#calculating the gap between each draw as days_gap days_gap =[] for i in range(1, data.shape[0]):

days_gap.append((data[‘Date’][i]- data[‘Date’][i-1]).days)



days_gap.insert(0, median(days_gap))

print(len(days_gap))

Let us analize days_gap feature:-

#plotting days_gap feature to understand the outlier behaviour

plt.plot(range(data.shape[0]), data[‘days_gap’])

plt.xlabel(‘each draw no’)

plt.ylabel(‘number of days gap’)

plt.title(‘curve to see the trend of days gap’)

plt.show()

Observation: -

As we can observe the outlier in the days_gap because of Covid-19 outbreak, let’s replace this by second highest value from the feature

Following is the code for doing the same:-

print(‘the second highest value for days gap is:’, data[‘days_gap’][data[‘days_gap’]< max(data[‘days_gap’])].max()) output-> the second highest value for days gap is: 28

Let/s replace 182 days by 28:-

data[‘days_gap’].replace(182 , 28, inplace=True)

Now, this is our final data looks like before we move to scaling:-

Now at this point of time it is important to save our preprocessed data into a pickle file so that we do not have to re-run program every time.

# Data Checkpoint DATA = data.copy() import pickle data.to_pickle(“./pro_data.pkl”)

9. Modeling Preparation and Modeling

Time based Splitting

Splitting data based on time as we need to predict new draws and testing should be done on the latest data. We are Splitting the data based on time rather than randomness, keeping only 10% as test data because our dataset is very small and we need few values from 2020 to make a good model. Below is the code snippet for train data and similarly we do it for X test data, y train and test data:-

x_train_data=data[[‘below_450’, ‘above_450’, ‘total’,

‘below_450_w’, ‘above_450_w’, ‘total_w’, ‘crs_w’, ‘days_gap’]][data[‘Date’]<= ‘2020–03–04’] print(x_train_data.shape)

x_train_data.head()

Standard scaling

It is paramount for us to standardize our dataset as most of the regression models use Euclidian distances for their algorithms and we not want to jeopardize our values because of unscaled data. So first we create the scaler object and then we just fit on the train data and not on the test data to avoid data leakage problem and finally we save our trained model into the pickle file for future use.

First we scaler object:-

scaler = StandardScaler()

Following is the code snippet for fitting, transforming and saving the trained model in the pickle file and similarly we do it for the test dataset and for the Y train and test dataset. It is important to scale the target variable also as it helps in improving the accuracy.

Standardize features by removing the mean and scaling to unit variance

The standard score of a sample x is calculated as:

z = (x — u) / s

code snippet:-

scaler_x_train= scaler.fit(x_train_data)

x_train = pd.DataFrame(scaler_x_train.transform(x_train_data), columns= x_columns)

print(x_train.shape)

x_train.head()

Minmax scaling

First we minmax object:-

minmax = MinMaxScaler()

Just like scaling we do mixmax to get the values in the range between 0 and 1.

The transformation is given by:

X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0)) X_scaled = X_std * (max - min) + min code snippet:- minmax_x_train= minmax.fit(x_train)

X_train = pd.DataFrame(minmax_x_train.transform(x_train), columns= x_columns)

print(X_train.shape)

X_train.head()

Similarly we do it for X_test, Y_train and Y_test datasets. Also, we save the trained model in the pickle file.

Modelling

After scaling the dataset, let’s jump to one of the most important sections of the case study i.e modeling.

The process that I have used here for modeling is that:-

1. We create a random model for setting our worst performance bars

2. Then, we create our baseline model, in this case I have built a Linear Regression model as the loss function in this case is MSE which is also our KPI

3. Then we try and create multiple complex models such as Ensemble and Time series model(s) to minimize MSE and MAE.

4. Since our dataset is small we have an added advantage to experiments with multiple models to minimize the KPIs and at last comparing all the models’ performances.

Let’s begin:-

9.1. Creating a random model which just outputs the average value of the target feature i.e CRS:-

random_y = [] for row in range(X_train.shape[0]): random_y.append(mean(Y_train[‘CRS’]))

Random Model test MSE:-

Deviation of approx 0.132 in test data is not so good. However, this is just a random model which simply calculated the average values to get the prediction and hence we know the maximum deviation value

9.2. Creating our baseline model- Linear Regression Model(Ridge):-

We choose linear regression because

· Our dataset is small, Linear regression tends to work well with similar datasets

· Also, the loss function for Linear Regression is MSE which is also one of our KPI’s

But, first we need to hypertune the model with the help of GridSearchCV, Code:-

from sklearn.linear_model import Ridge from sklearn.model_selection import GridSearchCV

lr = Ridge()

parameters= {‘alpha’:[0.001,0.01,0.1,0.5,1,10,100]} reg = GridSearchCV(lr, param_grid= parameters, cv=5, scoring= ‘neg_mean_squared_error’, return_train_score=True ) reg.fit(X_train, Y_train)

Let’s plot and check the cv vs train scores to finalize hyperparameter value:-

cv_mse= results[‘mean_test_score’] train_mse= results[‘mean_train_score’] alpha_values= results[‘param_alpha’] plt.figure(figsize=(15,5)) plt.plot(alpha_values, cv_mse, label= ‘CV curve’) plt.scatter(alpha_values, cv_mse, label= ‘CV mse points’) plt.plot(alpha_values, train_mse, label= ‘train curve’) plt.scatter(alpha_values, train_mse, label= ‘train mse points’) plt.title(‘CV curve vs Train curve’) plt.xlabel(‘Alphas’) plt.ylabel(‘mse scores’) plt.legend() plt.show()

Once we get the hypermater for L2 norm, we instantiate the model object and fit the train set. As seen from the above curve alpha=1 has best cv and test region, so we select 1 as the alpha value from our Ridge Model:-

final_lr = Ridge(alpha=1, random_state=1) final_lr.fit(X_train, Y_train) y_hat_tr = final_lr.predict(X_train) y_hat_te = final_lr.predict(X_test)

MSE Calculation for Linear Regression Model (Ridge)

# calculating mse for train and test data print(‘Linear Regression Train MSE:’,mean_squared_error(Y_train, y_hat_tr)) output-> Linear Regression Train MSE: 0.009160673524770603 print(‘Linear Regression Test MSE:’, mean_squared_error(Y_test, y_hat_te)) output-> Linear Regression Train MSE: 0.009160673524770603

MAE Calculation for Ridge

print(‘Linear Regression Train MAE:’,mean_absolute_error(Y_train, y_hat_tr)) output-> Linear Regression Train MAE: 0.07946065494731351 # calculating MAE for test final_lr_mae = mean_absolute_error(Y_test, y_hat_te) print('MAE for Test Linear Regression:', final_lr_mae) output-> MAE for Test Linear Regression: 0.10158228942325226

To keep our blog a little short, I am going to skip few experiments with models that were good but not the best. To check out the full case study you can check out my GitHub profile which is mentioned at the end of the blog.

9.3. Creating GBDT Model (boosting)

Before trying this, I also experimented with Decision Tree and Random Forest. The performances were good but GBDT just seems to do a little better.

Following are the code snippets for the same:-

from sklearn.ensemble import GradientBoostingRegressor score_lst= [] for m in range(32,257): gbdt = GradientBoostingRegressor(n_estimators= m, max_depth=2) gbdt.fit(X_train,Y_train) score_lst.append(gbdt.train_score_) # fetching the final training scores train_score=[] for i in score_lst: train_score.append(i[-1]) Calculatng MSE and MAE for GBDT print(‘GBDT Train MSE:’,mean_squared_error(Y_train, gbdt_hat_tr)) print(‘GBDT Test MSE:’,mean_squared_error(Y_test, gbdt_hat_te)) print(‘*****************************************’) print(‘GBDT Train MAE:’,mean_absolute_error(Y_train, gbdt_hat_tr)) print(‘GBDT Test MAE:’,mean_absolute_error(Y_test, gbdt_hat_te))

MSE and MAE for GBDT:-

GBDT Train MSE: 1.4228967196440351e-05

GBDT Test MSE: 0.003160014966977961

*****************************************

GBDT Train MAE: 0.0031675632474617876

GBDT Test MAE: 0.03454233727873762

9.4. Creating a StackingCv Regressor Model

Out of all the models, StackingCV Regressor has given the best performance in terms of minimizing losses. I chose StackingCV Regssor over other models because it uses the concept of out-of-fold predictions: the dataset is split into k folds, and in k successive rounds, k-1 folds are used to fit the first level regressor. In each round, the first-level regressors are then applied to the remaining 1 subset that was not used for model fitting in each iteration. The resulting predictions are then stacked and provided — as input data — to the second-level regressor. After the training of the StackingCVRegressor , the first-level regressors are fit to the entire dataset for optimal predicitons.

Following is the code snippet in which I am going to use my already trained models in StackingCV Regressor:- reg1= Ridge() reg2= KNeighborsRegressor() reg3= DecisionTreeRegressor(max_depth=2) reg4= RandomForestRegressor(n_estimators= 41) reg5= GradientBoostingRegressor(n_estimators= 256, max_depth=2) #keeping linear regressor i.e Ridge as our meta regressor as linear models tend to do well as meta models stack = StackingCVRegressor(regressors=(reg1, reg2, reg3, reg4, reg5), meta_regressor=reg1) stack_fit = stack.fit(X_train.values.reshape(-1,8), Y_train.values.reshape(-1)) stack_hat_te= stack_fit.predict(X_test) stack_hat_tr= stack_fit.predict(X_train)

Checking MSE and MAE for StackingCV Regressor:-

print(‘Stack Train MSE:’,mean_squared_error(Y_train, stack_hat_tr_2))

print(‘Stack Test MSE:’,mean_squared_error(Y_test, stack_hat_te_2))

print(‘*****************************************’)

print(‘Stack Train MAE:’,mean_absolute_error(Y_train, stack_hat_tr_2))

print(‘Stack Test MAE:’,mean_absolute_error(Y_test, stack_hat_te_2))

STackingCV Regressor MSE and MAE:-

Stack Train MSE: 0.0019723588373711577

Stack Test MSE: 0.0019978481041382943

*****************************************

Stack Train MAE: 0.03534644559393333

Stack Test MAE: 0.027902297298640244

Observation:-

As we can observe above that train and test MSE have dropped to almost 0.001 which is a very good value for our model. Throughout this case study, I have tried various models which I have not mentioned here to keep blog a little shorter. I have tried time series models such as ARIMA and some custom models based on stacking give quite an interesting results. However, I would recommend you to check out the complete case study on my GitHub profile, which is explained in details and is kept quite simple to understand.

Following is the code snippet and results of all the models:-

from prettytable import PrettyTable

x = PrettyTable() x.field_names= [‘Model’, ‘Test MSE’, ‘Test MAE’]

x.add_row([‘Linear Regression’, ‘0.012’, ‘0.101’])

x.add_row([‘K-NN’, ‘0.008’, ‘0.065’])

x.add_row([‘Decision Tree’, ‘0.006’, ‘0.053’])

x.add_row([‘Random Forest’, ‘0.006’, ‘0.060’])

x.add_row([‘GBDT’, ‘0.003’, ‘0.034’])

x.add_row([‘ARIMA’, ‘0.65’, ‘0.77’])

x.add_row([‘StackingCv-Reg’, ‘0.0019’, ‘0.027’])

x.add_row([‘StackingCv-GBDT’, ‘0.003’, ‘0.041’])

x.add_row([‘CustomCv-Reg’, ‘0.007’, ‘0.060’])

print(x)

10. Result

Final Comparisons of models output:-

+-------------------+----------+----------+

| Model | Test MSE | Test MAE |

+-------------------+----------+----------+

| Linear Regression | 0.012 | 0.101 |

| K-NN | 0.008 | 0.065 |

| Decision Tree | 0.006 | 0.053 |

| Random Forest | 0.006 | 0.060 |

| GBDT | 0.003 | 0.034 |

| ARIMA | 0.65 | 0.77 |

| StackingCv-Reg | 0.0019 | 0.027 |

| StackingCv-GBDT | 0.003 | 0.041 |

| CustomCv-Reg | 0.007 | 0.060 |

+-------------------+----------+----------+

Finally I am saving STacking RegressorCV model in pickle:-

model_file= ‘final_model.sav’ pickle.dump(stack_fit, open(model_file, ‘wb’))

Observation:-

Intrestingly, our window features such as days_gap, crs_w are the top features, followed by above_450 feature

Conclusion:-

As per our observation from summary above StackingCV with linear regression as Meta model and GBDT model tend to out-perform others. In our case, StackingCv-Reg has better Test MSE and MAE values. It uses the best of all models. I hope this case study serves your purpose in someway. You can always refer the full code by visiting to my Github profile. Thank you:)

11. Future Work

As the data grows over the years we can try and build better ARIMA or FB-Prophet models. However, it would take really long to apply any deep learning model because of limited number of data points. Nonetheless, our ensemble models do a good job.

12. Profiles :-

GitHub link:- https://github.com/TarunLohchab/CRS/blob/main/sc1_crs.ipynb

To connect with me :-

LinkedIn link:- https://www.linkedin.com/in/tarun-lohchab-3080b41bb/

13. References:-

http://rasbt.github.io/mlxtend/user_guide/regressor/StackingCVRegressor/

https://www.canada.ca/en/immigration-refugees-citizenship/services/immigrate-canada/express-entry/submit-profile/rounds-invitations.html

https://scikit-learn.org/stable/

https://www.appliedaicourse.com/

https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/