H-1B Visa Analysis — R Shiny with Plotly Visuals | by Suraj Malpani | Towards Data Science

About a month ago, I had three telephone interviews scheduled in a particular week. All of them ended up “not moving further” with my application because of my U.S. visa status. One of the conversations lasted less than 60 seconds since the first question they asked was if I would need visa sponsorship now or in the future, and that was it. The interview ended there.
Yes, the infamous H-1B visa, which provides American companies the ability to hire foreign talent temporarily for up to 6 years. Employers have to file an application with USCIS on behalf of the employee, and approval from USCIS makes employees eligible to work in the United States. Understandably, employers are becoming hesitant to hire someone who would need visa sponsorship in the future due to frequent legal changes and uncertainty.
Many international students like me are finding it difficult to find a job because of this reason. Being an aspiring Data Scientist, I decided to analyze the publicly available data on the USCIS data hub. A detailed description of the data is available here.
I used the ‘plotly’ package in R to create interactive graphs and ended up developing a web app using ‘R Shiny’. The detailed description of how to use the app and explore this data interactively is at the end of this article. The aim was to make the job-hunt more focused, and the app will be useful for all the international students who are in the same boat as me.
Note: I wanted to keep the focus of this article as a descriptive report, thus I have omitted some of the code from the post. However, all of it is available on Github. The plots/graphs in this post are static, but one can interact with them in the web app.
What follows are some of the insights that I unearthed:
Yes, they have. When we look at the total number of approvals and denials in the following graph, we realize that since 2015, visa approvals are dwindling while denials are surging. We will delve deeper into the number of approvals and denials in the further sections, and we’ll analyze them through various angles such as geographical, industrial, organizational, etc.
## Code for plotly chart of total approvals and denialsrequire(tidyverse)require(plotly)output$plot1 <- renderPlotly({ Data %>% group_by(Year) %>% summarize(Approvals = sum(Initial_Approvals), Denials = sum(Initial_Denials), C_Approvals = sum(Continuing_Approvals), C_Denials = sum(Continuing_Denials)) %>% plot_ly(x = ~Year, y = ~Approvals, type = “scatter”, mode = “lines”, color = I(‘dark green’), name = “Approvals”) %>% add_trace(x = ~Year, y = ~Denials, type = “scatter”, mode = “lines”, color = I(‘red’), name = “Denials”) %>% layout(title = “H-1B Visas by Year”, xaxis = list(title = “Year”), yaxis = list(title = “Count”)) })
What cities/states are you most likely to find professional opportunities as an international student?
I wanted to inspect which areas of the country are observing the highest visa approvals, which also means more opportunities for international students. Moreover, since a lot of companies prefer “local” candidates, I am pondering whether I should be moving to any of these cities. So, I investigated for the states and cities with maximum visa approvals for the last two years; the last two years specifically since we are witnessing a sharp decline during this period. You can select whichever year/s you wish to look at in the app and perform deeper analysis.
The state of California leads with the highest number of visa approvals. No surprises there as California is one of the largest states in the continental U.S. and has Silicon Valley and several cities with major tech hubs that are often hiring. With less than half the approvals of California, New York, New Jersey, and Texas take the second, third, and fourth spot on the list.
City wise, I found out that New York City (NYC) is the hotspot with the highest number of visa approvals in the last two years. NYC is followed by Chicago, San Francisco, Boston respectively.
# Plot of top cities for shiny serverrequire(plotly)require(ggmap)#geo styling for plotg <- list( scope = ‘usa’, showland = TRUE, landcolor = toRGB(‘light gray’), projection = list(type = ‘albers usa’), showlakes = TRUE, lakecolor = toRGB(‘white’) )  output$cities <- renderPlotly({ Data %>% filter(Year %in% input$Year) %>% group_by(City) %>% summarize(Approvals = sum(Initial_Approvals), Denials = sum(Initial_Denials), C_Approvals = sum(Continuing_Approvals), C_Denials = sum(Continuing_Denials)) %>% arrange(desc(Approvals)) %>% top_n(50, Approvals) %>% left_join(coords_cities, by=”City”) %>% plot_geo(lat = ~lat, lon = ~lon, color = ~Approvals, size=~(Approvals)) %>% add_markers(hovertext = ~(paste(“City:”, City, “\nNo. of Approvals:”, Approvals))) %>% layout(title = ‘Top cities with H-1B Visa approvals in the selected Years’, geo=g) })
When we look at the denial rates in the following graph, we notice that visa denials across the board increased in 2018. The denial rate is calculated as the percentage of denials out of total applications.
‘Educational Services’ industry has been least impacted and has the lowest overall denial rate. The exemption for the education industry from the 85,000 annual number of H-1B approvals could be an explanation.
Institutions of higher education, nonprofit organizations that are related or affiliated to institutions of higher education, nonprofit research organizations, and governmental-research organizations are exempt from H-1B annual quotas.
# Code for denial rate plot output$dept_denial <- renderPlotly({ plot_ly(Dept_Data, x = ~Year, y=~Denial_Rate, color =~Dept_Name, type=’scatter’, mode = ‘line’) %>% layout(title = “H-1B Visas Denial Rate by Department”, xaxis = list(title = “Year”), yaxis = list(range = c(0,50), title = “% Denials”)) })
When considering the total number of approvals, the ‘Professional and Technical Services’ industry dominates, with more than 50% of the total approvals. Although, since 2015 the approvals in this industry have dropped and steadily declined.
This is the most interesting part. On average, I personally spend about 5–10 minutes for each job application. I would not want to apply to a job with an employer only to hear back from them letting me know that they are not considering international students or applicants who “require sponsorship now or in the future”.
You can look up any employer in the app to see details about how many candidates they sponsored over the years. For the analysis part, I decided to find top employers, as in the employers with the highest approvals in the last two years.
#Code for employer wordcloudrequire(wordcloud2)employer_wc <- Data %>% filter(Year > 2017) %>% group_by(Employer) %>% summarize(Approvals = sum(Initial_Approvals), Denials = sum(Initial_Denials), C_Approvals = sum(Continuing_Approvals), C_Denials = sum(Continuing_Denials)) %>% mutate(Denial_Rate = Denials/(Approvals+Denials)*100) %>%  select(Employer, Approvals) %>% arrange(desc(Approvals)) %>% top_n(15, Approvals)wordcloud2(employer_wc, size = .25, minRotation = 0, maxRotation = 0, color=rep_len(c(“blueviolet”,”blue”,”darkgreen”,”purple”,”deeppink”,”darkorchid”,”darkviolet”,”brown”,”darkred”), nrow(demoFreq) ), backgroundColor = “white”)
As we can see all the tech companies are leading in visa approvals. Apple tops the list with about 1700 approvals in the last two years. Deloitte, Intel, IBM, Microsoft, Facebook, Google, and others are also top contenders.
This is an analysis I completed with some of the insights that I discovered. I understand there’s much more one can do with this data. If you have any changes you’d like to see in the app, feel free to suggest in the comments or collaborate directly on Github.
Excited to use and interact with this data? You can do so using either of the following options:
1) Run locally using R: If you have R and RStudio/any other IDE installed on your system, run the following command in your R Console. The app will be loaded directly from the GitHub repo. It might take some time to load the app for the first time.
runGitHub("Shiny_H1b","SurajMalpani")
2) Web hosting: https://surajmalpani.shinyapps.io/Visas/
You can access the web app by visiting the above address. However, I would personally recommend using the first method (runGitHub) as some of the plots are not supported by Shinyapps, hence I had to remove them.
There are 4 tabs as you can see in the following video. Their descriptions are mentioned below the video:
i) All Data: This tab contains all the data from USCIS. You can look up for any or multiple employers, years, states, cities of the interest. After filtering the data based on your input, you can download the filtered file in a .csv format for further analysis.
ii) Approvals vs Denials: You can look at the total number of approvals and denials over the years, since 2009.
iii) Geographic: You can look at the top 10 states and the top 50 cities with the highest number of approvals. You can select year/s you wish to look at and the plots will be updated for the selected years.
iv) Departments: You can inspect the number of approvals w.r.t each department over time. Additionally, you can look at the denial rates for these departments.
Please feel free to ask questions/give suggestions in the comments section or feel free to connect with me on my Website or LinkedIn.
Thank you for reading. I hope this analysis and the web app are helpful to you!
References:
Plotly WebsiteShiny RStudioPlotly-RUSCIS