Permanent Residency — a Data Science Exploration
By: Lauren Howard
Tags: Decision Tree, Xgboost, Data Science, Supervised Learning, Permanent Residency

Immigration is a hot topic in the United States, but how can data science evaluate the immigration process and potentially help you out?

1 in 7 of the US population are immigrants and for many of these people, permanent residency is a high priority in the hope of staying in the United States. As a UK citizen currently undergoing the permanent residency process*, this was a topic that I was very passionate about.

The Immigrant’s Statue, New York City, NY

For many people currently employed in the United States, employment-based permanent residency is a potential option to explore if they want to remain to be in the country. However, with the number of applications doubling since 2008 and higher scrutiny being placed on immigration-related cases, I wanted to predict whether a case is likely to be denied and investigate its potential determinants.

This may help in deciding whether this lengthy (between 6 months to several years) and costly (could be 1000’s of dollars) process is feasible for the employer and/or the applicant.

Number of Employment based Permanent Residency applications since 2008.

The Data

I started with data from the Department of Labour regarding individual applications from 2008–2019. I also incorporated economic data from the applicant’s country of citizenship and the year of application to analyze whether this had an influence on applications overall. The big challenge faced with the application dataset was that there was a lot of criteria which may not be related to every applicant. Therefore, the dataset contained multiple null values and features to interpret as to whether they would be good contributors to application outcome and useful in the modeling process.

The final features utilized in the models can be categorized into personal information, employer and employment status, application filing date and economic influences of applicant’s citizenship. In total, after dummying the categorical variables, there were over 500 features contributing to application outcome!

In the original dataset, the potential outcomes were “Certified”, “Denied” and “Withdrawn”. I removed the applications that were “Withdrawn” as the potential outcome was unknown. This left me with nearly 850,000 applications.

Modeling Process

I treated this project as a classification problem and aimed to predict whether the application is likely to be “Certified” or “Denied”. The error metrics I wanted to optimize during the modeling process were Accuracy (To correctly predict as many outcomes as possible) and Precision (To reduce the number of applications being incorrectly predicted as “Certified”). I determined this as the “Worst Case Scenario” — predicting the application to “Certified” and then it actually being denied.

The best performing model was using the XGBoost algorithm, which performed with 73% Accuracy and 69% Precision. I chose to utilize XGBoost as it was designed to optimize model performance by stacking models and iteratively correcting errors made by the previous model. I also used GridSearch to find the best parameter combination to tune the XGBoost algorithm to optimize its accuracy and precision. XGBoost also has the capabilities to assess feature importance. This was particularly significant as I wanted to investigate which features are significant in the outcome of applications, to potentially suggest improvements to the application.

XGBoost Model Performance

I was still aware of the “Worst Case Scenario” and this was something I considered whilst testing different models. For example, I implemented a Naive Bayes model as I had a large number of features in my dataset in an attempt to reduce this case. Although this was successful in reducing the incorrectly predicted “Denied” cases to just 6%, the overall accuracy and precision were severely compromised. These were reduced to 54% and 52% respectively. As shown below, although 94% of the Denied cases were correctly predicted, only 14% of Certified cases were. This demonstrates well how different models respond differently to the data and how outcomes can differ significantly.

Naive Bayes Model Performance

Comparing Model Performances

In order to compare model performances against each other, I used a ROC Curve and an AUC score. The ROC Curve shows how well a model optimizes the number of positive predictions whilst reducing the number of predictions in the “Worst Case Scenario”. From this, the area of the curve (AUC) score between 0.5 and 1 is calculated (The dashed line in the graph below represents a score of 0.5). A score closer to 1 demonstrates a model’s increased ability to distinguish between each class. As shown below, the ROC Curve for XGBoost shows that it performs better than the Naive Bayes model both increasing the model’s True Positive Rate whilst reducing the worst case scenario likelihood. This is reflected in the AUC scores of 0.65 for Naive Bayes and 0.82 for XGBoost.

ROC Curve for Naive Bayes and XGBoost Models

With this mind and the ability to assess feature importance, the XGBoost was determined to be the final and most interpretable model.

Feature Importance

As mentioned, my original dataset contained over 500 features, therefore I wanted to isolate the most important features to reduce model complexity and likelihood to overfitting. However, the feature importance generated by the XGBoost algorithm can sometimes be misleading, depending on which importance metric is used (Such a great article explaining why). With this in mind, I also used a Decision Tree model to compare feature importances and compared similarities between the two. This allowed me to be more confident in which features were most and least influential to application outcome. By comparing the feature importance outcome I was able to rank each feature and weight the importance of each.

Most and Least Important features determined by XGBoost & Decision Trees

Initially, I wanted to use these feature importance to suggest potential improvements to applications. However, it seems that a lot of these features are pretty unique to the applicant and cannot easily be altered. The most important features found from both models were mostly related to the job and the employer themselves — especially the prevailing wage for the job and the size of the company. Through further research, I found that part of the application process is that the employer has to prove that they can pay the prevailing wage to the applicant — therefore, a lot of applications with high wage requests were denied!

On the other hand, this project demonstrated some positive outcomes in the Permanent Residency application process. Personal features related to the applicant themselves, such as their country of citizenship, the state in which they reside and the visa they originally the United States on, had very little influence on the model. This suggests that the process does not discriminate against these features, something we would really hope for!

Applications over time

A slight aside to this project, driven by the outcome of the feature importance was to look at the application data over time. My first revelation was that although the number of applications has increased, the percentage of application rejections has reduced from 18% in 2008 to 6% in 2018! This isn’t a steady change, so it’s difficult to decipher whether this is likely to remain the case in the future (especially noting the increase after 2018).

Percentage of Application Rejections

With changes in the United States Administration over this time period, I was interested to see whether this had any particular influences. I color coordinated each graph to show when the change in administration occurred. It seems that although there is a growth in application numbers, the raw total of rejections remains fairly constant from 2010 onwards, which is reflected in the percentage of rejections graph above.

Number of applications 2008–2019

Number of application rejections 2008–2019

The most significant points in these graphs are the drop in applications just before October 2018 and the peak in rejections just after October 2010. Unfortunately, there is little transparency between the applicants and the Immigration Service as to what causes these anomalies.

A full Jupyter Notebook of code for the project analysis can be located here.

*I am personally undergoing a different pathway to US Permanent Residency and not reflected in this project.