A Beginner’s Guide to Grabbing and Analyzing Salary Data in Python | by Matt Grierson | Towards Data Science

When I first started learning Python, one of the resources I wish I had were really simple, practical examples of how to grab and analyze data. For years I clung to Excel simply because I could do everything relatively fast and it seemed like Python was a lot of learning for a questionable benefit at the time.
Below, I want to discuss three extremely useful libraries for Exploratory Data Analysis (EDA), highlighting the value of each to those who may wonder about the cost-benefit of learning Python. The full code shown below can be found here as a Jupyter Notebook.
If you work in one of the FAANG (Facebook/Apple/Amazon/Netflix/Google), you’ve likely heard of site levels.fyi. For those who haven’t, Levels makes it easy to compare and contrast different career levels across different companies, and is generally considered more accurate in terms of actual tech salaries relative to other compensation sites like glassdoor.com. Levels.fyi has a great interface and its visualizations make it easy to compare salary bands across companies, however, I found myself wanting to analyze the data in a way that pre-aggregated views on the site doesn’t allow. Below is a step-by-step process for how I’ve done it. To follow along, you will need something to write Python in (would recommend Jupyter) and know how to install the various libraries mentioned (would recommend downloading Anaconda for beginners).
Retrieving the data is the most confusing part starting out, since the site structure determines how accessible the data is. Luckily, after examining network calls using developers tools, I found level.fyi delivers its entire salary payload from a single endpoint. Grabbing JSON data with Python is ridiculously easy, and I personally use the requests library. Combine this with the pandas library and we have stored the JSON into a tidy dataframe with two lines of code:
import pandas as pdimport requestsdata = requests.get('https://www.levels.fyi/js/salaryData.json').json()df = pd.DataFrame(data)
Using the requests.get allows us to make a proper http request and returns a Response object. Adding the json() at the end transforms the response into JSON. Now that we have the data in a JSON format, all we need to do is construct a DataFrame which yields the following:
You can see that there’s a few useless columns that we pulled in, such as rowNumber. We’ll go ahead and get rid of them so that we don’t clutter our analysis in the next steps. We’ll also want to replace blank values with the numpy ‘NaN’ so that some of the exploratory analysis in the next steps function properly:
import numpy as npdf = df.drop(['cityid', 'dmaid','rowNumber'], axis=1)df = df.replace("", np.nan)
Now that we have our DataFrame cleaned up, let’s start to explore the dataset!
Pandas has a lot of really useful methods for quickly understanding your dataset such as describe(), info(), shape(), etc. While they are certainly useful in their own right, I find that these can be cumbersome to remember early on. They’re also more focused on descriptive statistics that don’t do a whole lot to help you understand larger issues like data completeness or correlations. Luckily this is Python land, and there are three libraries in particular that do a great job in doing the heavy lifting for us.
For this dataset I first wanted to understand the completeness of each feature and whether I have enough to conduct meaningful analysis (ex/ examining wage difference by gender, or variance in stock/bonus by tenure). To do this I prefer to use the missingno library, which can return a variety of visualizations to help better understand the missing data in the dataset. All I need to do is import the library and use the following one-liner:
import missingno as msnomsno.matrix(df, figsize=(12, 8))
Voila! Beyond seeing how complete the data is, we also learn certain features like ‘tag’, ‘basesalary’, and ‘bonus’ weren’t always required form fills, but became required later. It also shows that outside of ‘otherdetails’ (and ‘gender’ for a stretch), optional data like ‘stockgrantvalue’ was filled out fairly frequently and can be included in my initial analysis.
Now that I have a quick idea on the completeness of my data, I can run a more in-depth exploratory analysis. To do this, I’m going to use a library called Pandas Profiling. Think of it like dataframe.describe(), but on steroids. Pandas Profiling takes every feature of the dataset, looks at the datatype, and then creates a huge page describing each feature in detail using distribution graphs, categorical bins, extreme values, etc. It also provides an overview of the dataset, correlation matrices, and a section displaying all the missing data using the MissingNo library mentioned above! Lukas Frei has written an extensive article about the library here which I highly recommend for a more in-depth overview of Pandas Profiling.
The first thing we need to do is make sure our datatypes are the right types so that Pandas Profiling can interpret them correctly. Using df.info() shows that all of our columns are currently object types, so let’s change that with the following code:
df['timestamp'] = pd.to_datetime(df['timestamp'])df[['yearsofexperience','basesalary','bonus','stockgrantvalue','totalyearlycompensation','yearsatcompany']] = df[['yearsofexperience','basesalary','bonus','stockgrantvalue','totalyearlycompensation','yearsatcompany']].apply(pd.to_numeric)
If you run df.info() again you’ll now see ‘timestamp’ is a datetime and the others have changed to floats. Now that our data is in the correct format, let’s use the 1-line wonder to generate the pandas profile!
import pandas_profiling as ppdf.profile_report(style={'full_width':True})
Once it runs you should see something similar to the graphic below with an overview of the dataset!
Pretty cool, right? In addition to summarizing the data, it also highlights relevant information like highly cardinal data, percentage of missing data within variables, and zero-values. For this article I’m most interested in totalyearlycompensation, so let’s examine that variable and see what we can learn...
Already we can see there is something causing a major skew in the data, as the distribution is highly concentrated on the left. Expanding the details and going to the extreme values tab shows the following:
It seems odd that the highest value is 5000, so we need to go to https://www.levels.fyi/addcomp.html and understand what the form to enter salary data looks like:
Looking at the form brings up several possibilities for the skew. Some people may be making $5 mil+ a year (possible!), there could be people just entering in junk data, and/or maybe there was a time on the site when users entered the data differently (the whole number instead of thousands). Since we have no control over data quality, let’s see if the ‘statistics’ tab can help us understand more about this:
When looking at the quantile data on the left, we can see that the 5th/95th percent marks seem to match the data on the whole (a range between $96,000 and $500,000). Instead of trying to create static filters or comb through data to remove these outliers, we can instead use the another mind-blowing method in Python called quantile to easily filter out the top and bottom 5%:
df = df[df['totalyearlycompensation'].between(df['totalyearlycompensation'].quantile(.05),df['totalyearlycompensation'].quantile(.95))]
Ta-da! We’ve instantly and dynamically removed the outliers from our dataset. If we re-run through pandas-profiling our totalyearlycompensation variable now looks like this:
You can even extrapolate further and use the pandas query method to analyze a specific subset of the data. For example:
df.query("company in ('Facebook','Amazon','Apple','Netflix','Google') and title in ('Product Manager','Software Engineer') and yearsofexperience>4 and yearsofexperience<16").profile_report(style={'full_width':True})
gives us a report that analyzes the subset of FAANG product managers and software engineers with 5–15 years of experience.
You can go further with the use of some additional libraries to plot a comparison out in a violin plot!
Originally, my article was only going to address the two libraries above, but there has been a new one releaseI want to cover as well called AutoViz. It is a newer library that essentially does the work of finding ‘relevant’ details of a dataset, grouping them together in logical ways, then providing an output to the user in various graph forms. As described in this article:
An XGBoost model is repeatedly used to determine the most consistent set of features determined to be important by using a random set of features each time; the most prominent selected features can then serve to guide future plotting and visualization
To apply and try this out on our dataset, all we need to do is add a few additional lines of code, and name the dependent variable we want to analyze (totalyearlycompensation in this example). Let’s also filter the dataset to software engineers, so we can analyze a single job type when looking at our output:
from autoviz.AutoViz_Class import AutoViz_ClassAV = AutoViz_Class()df = df[df['title'] == 'Software Engineer']df_av = AV.AutoViz(filename="", sep=",", depVar="totalyearlycompensation", dfte=df, header=0, verbose=2, lowess=False, chart_format="svg", max_rows_analyzed=15000, max_cols_analyzed=30)
Once you run the above you should have something like this, followed by graphs:
So we can see that out of the 13 features in the dataset, AutoViz decided to drop 6 due to low-information relevance, surprisingly enough company and location are among them. However, if we look at the location counts using df.value_counts(normalize=True) we see that 51% of the salaries come from just 5 locations, so there may not be enough data to make location a strong indicator.
Let’s now look at the graphs it generated and see what we can learn. Under ‘continuous variables mapped against target variable’, we see the following:
From this we can see that:
Base salaries tend to top out around $250,000, regardless of total yearly compensationStock grants and bonuses are highly variable and can comprise a large portion of a Software Engineer’s total yearly compensationBonuses tend to more evenly dispersed, while stock grants appear to have three clusters of points (90/50/20 degree angles). This could be a data quality issue, but this could also suggest considerable stock value appreciation
Base salaries tend to top out around $250,000, regardless of total yearly compensation
Stock grants and bonuses are highly variable and can comprise a large portion of a Software Engineer’s total yearly compensation
Bonuses tend to more evenly dispersed, while stock grants appear to have three clusters of points (90/50/20 degree angles). This could be a data quality issue, but this could also suggest considerable stock value appreciation
The histogram KDE portion is very important when considering the output, so let’s discuss that section next. First, we can see that males account for roughly 88% of the dataset, so extracting meaningful insights between genders will be difficult given the substantial imbalance. Second, the data skews heavily toward people who are early in their careers — 63% of the reported salaries are from people with 0–6 years of experience. This tells us that those with more experience may want to consider the limited sample size before jumping to too many conclusions.
The next set of variables in the KDE plot section also show a strong right skew. Bonus and stockgrantvalue in particular also have several spikes in them. That’s the KDE way of saying that these are the highest probability picks in the dataset.
Why is this important? If we had continued to just look through the pair-wise scatter plot section we would have encountered this:
If we had not seen the skews in the KDE I would have no way of telling that, for bonus, 40% of the values are actually under the number ‘40’.
AutoViz is an excellent new library that actually has the potential to save you time when you take an initial look at data. As shown above though, the program will not save you from misshapen or funky data, so understanding your dataset is key to generating useful insights with this!
Hopefully this post has given you a few new tools to work with when starting out with a dataset. There are many more in-depth and hardcore packages out there that I may discuss in a future post, but I think these hit a nice mix between high-value output and low-intensity input. Personally, I start with missingno, run pandas_profile after I’m confident the data is relatively complete, then filter/massage into a more detailed dataset to run against AutoViz for more granular auto-generated insights. Thanks for reading!