H-1B Visa Analysis — R Shiny with Plotly Visuals
By: Suraj Malpani
Tags: Data Visualization, Data Science, Plotly, Programming, R Shiny

H-1B Visa Analysis — R Shiny with Plotly Visuals

Photo by Andrew Neel on Unsplash

About a month ago, I had three telephone interviews scheduled in a particular week. All of them ended up “not moving further” with my application because of my U.S. visa status. One of the conversations lasted less than 60 seconds since the first question they asked was if I would need visa sponsorship now or in the future, and that was it. The interview ended there.

Yes, the infamous H-1B visa, which provides American companies the ability to hire foreign talent temporarily for up to 6 years. Employers have to file an application with USCIS on behalf of the employee, and approval from USCIS makes employees eligible to work in the United States. Understandably, employers are becoming hesitant to hire someone who would need visa sponsorship in the future due to frequent legal changes and uncertainty.

Many international students like me are finding it difficult to find a job because of this reason. Being an aspiring Data Scientist, I decided to analyze the publicly available data on the USCIS data hub. A detailed description of the data is available here.

I used the ‘plotly’ package in R to create interactive graphs and ended up developing a web app using ‘R Shiny’. The detailed description of how to use the app and explore this data interactively is at the end of this article. The aim was to make the job-hunt more focused, and the app will be useful for all the international students who are in the same boat as me.

Note: I wanted to keep the focus of this article as a descriptive report, thus I have omitted some of the code from the post. However, all of it is available on Github. The plots/graphs in this post are static, but one can interact with them in the web app.

What follows are some of the insights that I unearthed:

Have the denials for H-1B visas risen?

Yes, they have. When we look at the total number of approvals and denials in the following graph, we realize that since 2015, visa approvals are dwindling while denials are surging. We will delve deeper into the number of approvals and denials in the further sections, and we’ll analyze them through various angles such as geographical, industrial, organizational, etc.

## Code for plotly chart of total approvals and denials require(tidyverse)

require(plotly)

output$plot1 <- renderPlotly({

Data %>%

group_by(Year) %>%

summarize(Approvals = sum(Initial_Approvals), Denials = sum(Initial_Denials), C_Approvals = sum(Continuing_Approvals), C_Denials = sum(Continuing_Denials)) %>%

plot_ly(x = ~Year, y = ~Approvals, type = “scatter”, mode = “lines”, color = I(‘dark green’), name = “Approvals”) %>%

add_trace(x = ~Year, y = ~Denials, type = “scatter”, mode = “lines”, color = I(‘red’), name = “Denials”) %>%

layout(title = “H-1B Visas by Year”,

xaxis = list(title = “Year”),

yaxis = list(title = “Count”))

})

Total Visa Approvals And Denials Since 2009

Geographic Analysis

What cities/states are you most likely to find professional opportunities as an international student?

I wanted to inspect which areas of the country are observing the highest visa approvals, which also means more opportunities for international students. Moreover, since a lot of companies prefer “local” candidates, I am pondering whether I should be moving to any of these cities. So, I investigated for the states and cities with maximum visa approvals for the last two years; the last two years specifically since we are witnessing a sharp decline during this period. You can select whichever year/s you wish to look at in the app and perform deeper analysis.

The state of California leads with the highest number of visa approvals. No surprises there as California is one of the largest states in the continental U.S. and has Silicon Valley and several cities with major tech hubs that are often hiring. With less than half the approvals of California, New York, New Jersey, and Texas take the second, third, and fourth spot on the list.

Top states with highest H-1B visa approvals in the last two years

City wise, I found out that New York City (NYC) is the hotspot with the highest number of visa approvals in the last two years. NYC is followed by Chicago, San Francisco, Boston respectively.

# Plot of top cities for shiny server require(plotly)

require(ggmap) #geo styling for plot

g <- list(

scope = ‘usa’,

showland = TRUE,

landcolor = toRGB(‘light gray’),

projection = list(type = ‘albers usa’),

showlakes = TRUE,

lakecolor = toRGB(‘white’)

)



output$cities <- renderPlotly({

Data %>%

filter(Year %in% input$Year) %>%

group_by(City) %>%

summarize(Approvals = sum(Initial_Approvals), Denials = sum(Initial_Denials),

C_Approvals = sum(Continuing_Approvals), C_Denials = sum(Continuing_Denials)) %>%

arrange(desc(Approvals)) %>%

top_n(50, Approvals) %>%

left_join(coords_cities, by=”City”) %>%

plot_geo(lat = ~lat, lon = ~lon, color = ~Approvals, size=~(Approvals)) %>%

add_markers(hovertext = ~(paste(“City:”, City, “

No. of Approvals:”, Approvals))) %>%

layout(title = ‘Top cities with H-1B Visa approvals in the selected Years’, geo=g)

})

Top cities with highest H-1B approvals in the last two years

Which industries are impacted the most?

When we look at the denial rates in the following graph, we notice that visa denials across the board increased in 2018. The denial rate is calculated as the percentage of denials out of total applications.

‘Educational Services’ industry has been least impacted and has the lowest overall denial rate. The exemption for the education industry from the 85,000 annual number of H-1B approvals could be an explanation.

Institutions of higher education, nonprofit organizations that are related or affiliated to institutions of higher education, nonprofit research organizations, and governmental-research organizations are exempt from H-1B annual quotas.

# Code for denial rate plot

output$dept_denial <- renderPlotly({

plot_ly(Dept_Data, x = ~Year, y=~Denial_Rate, color =~Dept_Name, type=’scatter’, mode = ‘line’) %>%

layout(title = “H-1B Visas Denial Rate by Department”,

xaxis = list(title = “Year”),

yaxis = list(range = c(0,50), title = “% Denials”))

})

Denial Rates For The Departments

When considering the total number of approvals, the ‘Professional and Technical Services’ industry dominates, with more than 50% of the total approvals. Although, since 2015 the approvals in this industry have dropped and steadily declined.

Number of approvals per department

Which employers to target?

This is the most interesting part. On average, I personally spend about 5–10 minutes for each job application. I would not want to apply to a job with an employer only to hear back from them letting me know that they are not considering international students or applicants who “require sponsorship now or in the future”.

You can look up any employer in the app to see details about how many candidates they sponsored over the years. For the analysis part, I decided to find top employers, as in the employers with the highest approvals in the last two years.

#Code for employer wordcloud require(wordcloud2)

employer_wc <- Data %>%

filter(Year > 2017) %>%

group_by(Employer) %>%

summarize(Approvals = sum(Initial_Approvals), Denials = sum(Initial_Denials), C_Approvals = sum(Continuing_Approvals), C_Denials = sum(Continuing_Denials)) %>%

mutate(Denial_Rate = Denials/(Approvals+Denials)*100) %>%

select(Employer, Approvals) %>%

arrange(desc(Approvals)) %>%

top_n(15, Approvals) wordcloud2(employer_wc, size = .25, minRotation = 0, maxRotation = 0, color=rep_len(c(“blueviolet”,”blue”,”darkgreen”,”purple”,”deeppink”,”darkorchid”,”darkviolet”,”brown”,”darkred”), nrow(demoFreq) ), backgroundColor = “white”)

Top employers with H-1B approvals in the last two years

As we can see all the tech companies are leading in visa approvals. Apple tops the list with about 1700 approvals in the last two years. Deloitte, Intel, IBM, Microsoft, Facebook, Google, and others are also top contenders.

This is an analysis I completed with some of the insights that I discovered. I understand there’s much more one can do with this data. If you have any changes you’d like to see in the app, feel free to suggest in the comments or collaborate directly on Github.