Predict Data Science Salaries with Data Science
By: Junting, Jt
Tags: Python, Regression, Salary, Data Science, Machine Learning

Predict Data Science Salaries with Data Science

Photo by Carlos Muza on Unsplash

This post summarizes key insights in my Kaggle notebook analyzing US Data Science job market and salaries with Exploratory Analysis and Machine Learning algorithms (mainly Backward-stepwise Regression) in Python.

The Market Needs a Lot More Analysts, but is More Generous to Data Scientists

Among those 12,360 data science job hires in this study (data source: picklesueat and Glassdoor), Data Analyst hires top the chart by 27%. Data Engineers are the second largest group, occupying 23% of all positions. Only 17% are Business Analysts and 10% are Data Scientists. Other analysts (System Analysts, Quality Analysts, Reporting Analysts, …etc.) account for 22.7%. All analyst-type hires combined dominate the total hires by 67%.

However, from the graph below, we see the overall salary distribution of Data Scientists locates much more to the right comparing with those of Data Engineers and Analysts, showing Data Scientists’ salaries are obviously higher.

In general, the average salary of all data science jobs is $89,000 a year, but the variance is quite high. The middle 50% centered by the median ($80,000) can range from $60K to $110K.

Job Location Matters More Than Anything Else

Although many jobs are temporarily remote, most companies still plan to resume to their physical offices after the economy reopens and specify the job location when hiring. However, as many companies pay their employees differently based on the candidate’s current residential location, the final deal with each candidate during the pandemic is unknown in this dataset.

(The point plot at the right side: point estimates with 95% confidence interval)

Not surprisingly, a California-based job is paid more ($110K+), but such a big gap, up to $50K a year, between California and others like Florida is still astonishing. Moreover, regardless of living costs, data science professionals in Texas are definitely undervalued, given such a high demand.

(The point plot at the right side: point estimates with 95% confidence interval)

Some would attribute the salary variation to living costs of the location. If that were the case, New York and San Francisco would have been with the highest salary level for data science professionals. However, cities in Silicon Valley with mid-high living costs such as Santa Clara, Sunnyvale and San Jose actually perform much better. Therefore, some other factors that made Silicon Valley prosperous might be the real cause to those high salary pays.

Here is another view to how job location has a great impact on salaries. Earlier we saw Data Scientists’ salaries (avg. $123K) are significantly higher than Data Engineers and Analysts, but notice the second peak on the left of the distribution below (salaries from lower than $90K)? They look as if a bunch of Analysts’ salaries being misplaced with the title “Data Scientists”.

It turns out that, as job descriptions are confirmed to be similar, the presence of California as the job location in the high-paid group (≥$90K) is a lot higher. On the other hand, the underpaid group (<$90K) locates more at lower-salary states such as Texas and Illinois.

Also, from the regression analysis, nearly half of the variables in the final model are related to job locations, which outweigh other factors such as seniority, industries and sectors.

SMBs Often Offer Better Packages in Competition with Big Corps

Giant companies are not always the top choice in pursuit of higher salaries. Although in general giant companies (with 10,000+ employees and revenues above $5 billion) tend to pay higher, financially-unpublic small-and-medium businesses (SMBs) would also offer competitive packages.

In the heatmap below, the left-hand side shows the number of companies hiring data science professionals with their size and scale, whereas the right-hand side shows the average salaries they offer. Some buckets with extremely high salaries but low sample sizes are considered extreme values (e.g. only three companies are with 10,000+ employees and revenue $1–2 billion. That they offer $152K would be regarded as a rare case).

From the next heatmap for Data Scientists’ salaries and the upcoming ones for Business Analysts, Data Analysts, and Data Engineers, we would gradually realize that giant companies only pay much higher salaries to Data Scientists, which require higher qualifications as well, but for other types of positions, high-paying clusters are small to medium businesses.

“Machine Learning” is Hot, and Data Scientists are Swiss Knives

Some keywords on job titles also have an impact on the salaries. The most obvious one is “Machine Learning” (sometimes “ML”). Data shows “Machine Learning” written on the job title would increase this job’s salary by more than 10%. However, the often pairing phrase “Artificial Intelligence” does not have comparable effects, and “Machine Learning” seldom shows up on analysts’ titles.

Likewise, “Machine Learning” appears more frequent in the job descriptions for Data Scientists and Data Engineers than in those for analysts. On the other hand, Data Scientists are required or preferred to have a much extended breadth of qualifications, including PHD degrees, whereas other positions focus more on fewer skillsets and require/prefer only Master degrees or lower.

For Data Engineers, in addition to common skills such as Python and SQL, data infrastructure words and phrases such as Hadoop, Spark, AWS and ETL appear much more frequent than other positions.