Do Female Employees make less money? | by Jonathan Cabreira | Towards Data Science

Survey Data Analysis using pandas, Seaborn, Matplotlib, and Scipy.
This is my first post on Medium and because of that, I decided to practice some statistics with survey data. In order to do that, I'm going to use the Brent Ozar salary survey, which is released every year since 2017, as far as I am aware. The main idea here is to answer the following question: Do Females employees make less money than male employees?. Many people would say "Yes! They definitely do!", others might say "Of course female make less money!". Well, it seems to me that when we make this kind of statement, we must be concerned about something called confirmation bias. In other words, if you believe that women are underpaid in comparison to men, there is a chance that you will assume this statement without deep analysis into the quality of the data you are working on.
We’ll be using pandas, Matplotlib, seaborn and Scipy to make sense of our data. Moreover, we’ll also be using The 2019 Data Professional Salary Survey Results and you can find it at Bent Ozar’s webpage.
Take this post as an approach that can help you to analyze some surveys and any other data you want to statistically examine.
Data Exploration
Let’s first import some libraries that we will need and then explore the data.
#Import librariesimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlinesns.set_style(‘ticks’)#Ignore woarningsimport warningswarnings.filterwarnings(‘ignore’)# Import dataraw_data = pd.read_excel(‘2019_Data_Professional_Salary_Survey_Responses.xlsx’,header = 2)raw_data.head()
Let’s take a look at the data type and check if there are nan variables in the data set.
raw_data.info()
The first thing that catches my eye is the SalaryUSD column. This feature must be numerical, and in order to do some math with this data, we have to convert them to float. In addition, after a few observations, I’ve noticed that there are some characters in the Salary column that should be deleted. Also, due to the fact that some rows have been identified as integer and others as float, I’ve created a function to treat all this data convert them to string and then, convert the entire column to float. Finally, all the spaces (‘ ‘) on the column labels of the DataFrame have been replaced with an underscore (‘_’).
#Convert all of the salary rows into stringdef str_salary(salary):    sep = '.'    if type(salary) == int:        salary = str(salary)    elif type(salary) == str:        # Replace characteres and take the cents out of our data        salary = salary.replace(" ","").replace("$","").replace(",","").split(sep)[0]    elif type(salary) == float:        salary = str(salary)            return salary# Replace spaces(“ “) in columns name with underscore (“_”)raw_data.columns = raw_data.columns.str.replace(“ “,”_”)# Apply str_salary functionraw_data['SalaryUSD'] = raw_data.SalaryUSD.apply(str_salary)raw_data['SalaryUSD'].head()
In order to analyze only 2019 data, we will filter the DataFrame by year and by the columns that are relevant to our analysis.
df = raw_data.loc[raw_data.Survey_Year == 2019, ['Survey_Year','Country','SalaryUSD','JobTitle','YearsWithThisTypeOfJob','Education','Certifications','PopulationOfLargestCityWithin20Miles', 'Gender' ]]df.head()
Since we have defined our DataFrame, we are now able to get some information. First, let’s count the numbers of employees by country and work on the one on which we have more data. Second, we’ll count the number of employees by job title and finally, count the total of men and women.
# Answers by Countrydf.Country.value_counts()
As we can see, we have way more employees from the US in this survey and, because of that, we’ll filter our DataFrame in order to work only with the United States data.
#Filter dataframe by countryUS_2019 = df.loc[df.Country == ‘United States’,:]US_2019.head(3)
So, let’s take a look at the number of employees by their genders and see if we can take some information out of it. Are the number of female and male employees equal or, at least, is it close to equal? Let's check this out!
US_2019.Gender.value_counts()
It seems that we have an issue here. Unfortunately, the total number of males is way greater than the total of females. Also, we are not able to get any statistical information from the non-binary gender due to the fact that only one non-binary person answered the survey in 2019. So, let’s drop this specific row and focus only on two genders.
# Delet the row - Non-binary/third genderUS_2019 = US_2019[US_2019.Gender != 'Non-binary/third gender']US_2019.Gender.value_counts()
OK, we already know that (unfortunately) the number of females is way lower than the number of males. Lets now answer the following question: how are the genders distributed in terms of job title? Let’s first count the number of employees in each position.# Job Titles US_2019.JobTitle.value_counts()
# Job Titles US_2019.JobTitle.value_counts()
To my surprise, we have only 4 Data Scientists and 21 Engineers... Ok, let’s not consider this! So, now we have an overview of job titles in this survey. How about the distribution of gender by job title? Let’s check this out!
# number of each gender by job titleGender_df = US_2019.loc[ : , [‘JobTitle’, ‘Gender’, ‘SalaryUSD’ ]]Count_gender = Gender_df.pivot_table(Gender_df, index = [‘JobTitle’, ‘Gender’], aggfunc= ‘count’)Count_gender
I don’t know about you guys, but I am the visual kind of learner and that’s the reason why I really like to build graphs and tables. We can use the seaborn library to build a bar chart with the information we have just checked.
import seaborn as sns# use Seaborn stylessns.set()pd.pivot_table(Gender_df, index= 'JobTitle', columns= 'Gender', aggfunc= 'count').plot(kind= 'barh', figsize= (9,9))plt.xlabel("Salary USD")
The chart above clearly shows a gap between genders, not only regarding the employees who have answered this survey in general but also concerning job titles. Have you noticed this in your company?
Great. Now we know that there are 541 males and 60 female in our survey and we also know that there are way more male employees in all of the positions. It’s important to notice that we cannot guarantee that these facts are absolutely true because we don’t know how exactly the survey was conducted. Moreover, we have to be aware of the representativeness of our online samples so that we can trust in the validity of conclusions derived from our internet survey data. In other words, the data might be biased.
For instance, when conducting a survey, we are prone to over-sample some kinds of people ( employees in this case) and under-sample others. Not all individuals know Brent Ozar’s website — the one responsible for the survey, and others are more unlikely to participate in this survey if this kind of subject matter does not interest them.
We could try to deal with this issue using statistic techniques such as weighting, to compensate for this type of ‘sampling bias’. This is not the case of our analysis, we will just draw some straightforward insights from our data and practice some basic statistics since we are not really worried about that topic.
It’s time to look at the salary data so we can check if there is a gap in it. It’s important to bear in mind that we won’t take into account some features such as years of experience, education, and size of the company because we just want to answer the simple question: Is there a gap between female and male salaries? I would use those features, for example, to answer why there is or there isn’t a gap between the genders. That being said, let’s work with the salary column for our analysis.
A histogram shows frequency for intervals of values of a metric variable by grouping data into bins. Let's take a look at the distribution of the salary. How do you expect it to be? well, I would suppose that it looks like a Gaussian distribution positively skewed with very few employees making more than a specific threshold ( maybe $ 150k).
#Import sub-package stats from scipy libraryfrom scipy import statssns.distplot(US_2019.SalaryUSD, fit = stats.norm)
Good guess! The histogram above is similar(not exactly the same) to what I’ve expected. The data looks close to a normal distribution or, at least, it seems to be bell-shaped. Is it normal? Let’s check it.
We will test the normality of our data through one qualitative analysis ( Q-Q plot) and two quantitative tests (Shapiro-Wilk and D’Agosino’s k2 tests).
The Q-Q plot is a popular plot for checking the data distribution. The Quantile-Quantile plot or Q-Q plot generate its own sample of the distribution we are comparing it with ( Normal distribution in this case). The generated sample is divided into quantiles and each data point in the sample is combined with a similar member from the distribution we are comparing with at the same cumulative distribution. As a result, we have a scatter plot with the theoretical values on the x-axis and the data sample quantiles on the y-axis.
#Import qqplotfrom statsmodels.graphics.gofplots import qqplotqqplot(US_2019.SalaryUSD, line = ‘s’)
So, how do we interpret this plot? Well, deviations of the blue dots from the red line show a deviation from the distribution we are comparing with (Gaussian). Based on this, we can say that our feature does not seem to be a good match for normal distribution. If the data were normally distributed, most of the blue points would be on the red line, which is not the case. Even though there are many blue points on the line, there are many more not on it. This indicates that the data would assume a different distribution, as we have observed in the histogram.
In addition to the previous analysis, we will use two quantitative tests to check the normality of our data. Let’s start with the Shapiro-Wilk Test which evaluates a specific sample and quantifies how likely it is that the data will follow a normal distribution. As far as I know, Shapiro-Wilk test is not only a reliable test but also a widely used one and we can apply it using the shapiro() function from SciPy library. This function returns both the W-statistics calculated by the test and the p-value.
The Second test we’ll use is the D’Agosino’s k2 Test. This normality test calculates the kurtosis and the skewness of the data to determine if they follow a Gaussian distribution. Basically what the test does is the combination of skew and kurtosis of the sample to produce an omnibus test of normality. For those who are not familiar with these terms, kurtosis is the quantification of how much of the distribution is in the tails, and skew is the quantification of how much the distribution is pushed right or left ( it measures the asymmetry in the distribution). In order to calculate de statistic and the p-value, we’ll use the normaltest() function from SciPy library.
Shapiro-Wilk Test
from scipy.stats import shapirostatistic,p = shapiro(US_2019.SalaryUSD)print(f'P-value: {p :<.3f}')#For a threshold of 5% (alpha)if p > 0.05:    print(f'Sample looks Gaussian (fail to regect H0)')else:    print(f'Sample does not look gaussian ( regect H0)')
D’Agosino’s k2 Test
from scipy.stats import normalteststatistic,p = normaltest(US_2019.SalaryUSD)print(f'P-value:{p:<.3f}')#For a threshould of 5% (alpha)if p > 0.05:    print(f'Sample looks Gaussian (fail to regect H0)')else:    print(f'Sample does not look gaussian (regect H0)')
Since we’ve analyzed our tests, both in qualitative and quantitative manners, we can make an assumption about the data. All of the tests and visualization techniques we’ve used so far do not indicate a normal distribution.
To make sure our data represents our problem properly, we have to be aware of the possibility of having outliers in our data set. Outliers are entries that are outside the range of what is expected and unlike the other observations. They affect the skewness of our data distribution and, consequently, some statistical terms such as standard deviation and mean. In particular, these measures are highly sensitive to outliers due to the way they are calculated.
Let’s split our data by gender and plot a boxplot so that we will be able to illustrate our data through their quartiles. This can also help us to identify some potential outliers, such as those data located outside the whiskers in the plot.
fig = plt.figure(figsize = (10,8))sns.boxplot(x = ‘Gender’, y = ‘SalaryUSD’, data = US_2019)
Let’s calculate some terms so that we can have some clue about the data.
def statistics(salary): #Import variation from scipy.stats import variation #Calculate terms mean = salary.mean() median = salary.median() mode = salary.mode() kurtosis = salary.kurtosis() skew = salary.skew() CV = variation(salary) #dict stats = [(‘Country’, [‘United States’]), (‘Mean’,[round(mean,2)]), (‘Median’,[round(median,2)]), (‘Mode’, [int(mode)]), (‘Kurtosis’,[round(kurtosis,2)]), (‘Skew’,[round(skew,2)]), (‘Coefficient of variation’,[round(CV,2)])  ] #dataframe stats_df = pd.DataFrame.from_items(stats)  return stats_df
Apply statistics function
Salary_statistics = statistics(US_2019['SalaryUSD'])Salary_statistics
So, once we have calculated these statistics we are able to identify some characteristics of our data. First, the median and mean are greater than the mode. This means that the right tail of the distribution is longer, or in other words, the data is positively skewed. Second, the skewness of our data is greater than 1, confirming the high skewness of it. Third, the high kurtosis in our data set indicates that its distribution has heavy tail or outliers — we have to investigate it. Finally, the coefficient of variation (CV) which is a statistic measure of the dispersion of data points in a data set around its mean, indicates that our sample has a significant variation of 43%, relative to its mean.
Up to this point, all analysis we have made give us evidence that our **data does not follow a normal distribution. Besides, high kurtosis can be an indicator of outliers. Let’s investigate it.
We need a reasonable standard deviation for our analysis. What do I mean by that? Well, the standard deviation represents how spread out the data are from its mean and, as I’ve said before, this measure is highly affected by outliers. In addition, we will see later that the standard error is sensible to the standard deviation of the population and because of that, we want to filter out outliers.
Right. We know that we have to filter out the outliers. But how can we do that? The answer is simple: we will use the interquartile range for that.
Interquartile Range is basically the distance between the third quartile and the first quartile. In our case, the table below shows that Q3 = $ 123.600,00 and Q1 = 83.000,00. Let’s calculate the difference between them and determine our Interquartile Range (IQR).
#Statisticsround(US_2019.SalaryUSD.describe().to_frame(),3)
#Calculate the percentiles — 25 and 75 %q75, q25 = np.percentile(US_2019.SalaryUSD , [75 ,25])print(f’Q3:{q75}’)print(f’Q1:{q25}’)# Calculate the percentile Rangeiqr = q75 — q25print(f’Interquartile Range: {iqr}’)
Now that we have our IQR we are able to filter the outliers out of our data set. We will do this considering that any data outside the median plus(+) and minus(-) 1.5 * IQR is an outlier.
# IQR +/- 40600min_salary = US_2019.SalaryUSD.median()- 1.5*iqrmax_salary = US_2019.SalaryUSD.median() + 1.5*iqrprint(f'Minimun salary (threshould) : {min_salary}')print(f'Maximun salary (threshould) : {max_salary}')
#data set within the rangeUS_2019_No_Outliers = US_2019[(US_2019.SalaryUSD>=min_salary) & (US_2019.SalaryUSD <= max_salary)]US_2019_No_Outliers.head()
After the removal of outliers, we can see how our data set has changed. First, the boxplot shows no outliers for both genders. Second, the data distribution roughly represents a Gaussian. Moreover, the mean is now closer to the median the Coefficient of variation is only 25% which means that our data is evenly distributed within the range we have defined.
# Boxplot with data within the rangefig = plt.figure(figsize = (10,8))sns.boxplot(x = ‘Gender’, y = ‘SalaryUSD’, data = US_2019_No_Outliers)
sns.distplot(US_2019_No_Outliers.SalaryUSD, fit = stats.norm)
# statistical terms for data within the rangeSalary_statistics_No_Outliers = statistics(US_2019_No_Outliers[‘SalaryUSD’])Salary_statistics_No_Outliers
#Statistical terms before and after the removal of outliersSalary_statistics.append(Salary_statistics_No_Outliers)
Notice that we have removed 34 entries that were above the maximum threshold and 8 that were below the minimum threshold. This indicates that 7% of the observations are responsible for causing a significant variability in our data set.
below_threshould = US_2019[US_2019.SalaryUSD <= min_salary]above_threshould = US_2019[US_2019.SalaryUSD >= max_salary]print(f'Total of salaries above the max threshould: {above_threshould.SalaryUSD.count()}')print(f'Total of salaries below the min threshould:{below_threshould.SalaryUSD.count()}')
print(f'Entries outside the range: { round((below_threshould.SalaryUSD.count() + above_threshould.SalaryUSD.count()) / US_2019.SalaryUSD.count()*100)} %' )
Another important aspect of the outliers is that 33 of them are male and only one is female. This suggests that 97% of the employees that make more money are male. Isn’t that intriguing?!
above_threshould.Gender.value_counts()
Now that we have a fairly balanced data set, we are able to do a Standard Error Analysis. Why would I do that? Well, the standard Error shows us how good our data is. In other words, it measures the accuracy with which our sample represents the entire population. The formula below shows how this statistical term is calculated.
Two important things to notice. First, the larger the sample, the closer the sample mean will match the mean of the population. Besides, we can see through the formula that if we have a larger number of samples, the standard error will be decreased. Second, the smaller the standard deviation of a population, the smaller the standard error. As we know, the standard deviation is a measure of spread and it is highly affected by outliers. This means that the standard deviation of a sample will be much larger in case of the existence of an outlier in our data set.
def mean_std_count(salary, data): mean = salary.mean() std = salary.std() count = salary.count()  std_error = [(‘Gender’,[data]), (‘Salary Mean’,[round(mean,2)]), (‘Salary std’,[round(std,2)]), (‘Sample’,[count]) ]  std_erro_df = pd.DataFrame.from_items(std_error) return std_erro_dfFemale_stats = mean_std_count(US_2019_No_Outliers[US_2019_No_Outliers.Gender == ‘Female’].SalaryUSD, ‘Female’)Male_stats = mean_std_count(US_2019_No_Outliers[US_2019_No_Outliers.Gender == ‘Male’].SalaryUSD, ‘Male’)Salary_stats = mean_std_count(US_2019_No_Outliers.SalaryUSD, ‘Total’)Female_stats.append(Male_stats).append(Salary_stats)
The table above shows us that we’ve got a gap in an average salary of about $3700. Don’t you think it is significant? In my opinion, it seems to be quite noteworthy.
Even though our data does not fit a normal distribution, after the removal of the outliers we’ve got something more concise in terms of mean and standard deviation. In addition, whether or not our data set follows a specific distribution shouldn’t matter too much for our analysis since we don’t need a Gaussian distribution for standard error to work.
That being said, if we consider a normal distribution of the sample, we can say that 95% of the salary should be within the range of $51.000 — $152.350 and I don’t see any problem with it. Actually, it sounds right to me.
def standard_error(salary, data):    #calculate terms    mean = salary.mean()    std = salary.std()    count = salary.count()    std_error = std/((count)**(1/2))    min68 = mean- std_error    max68 = mean+ std_error    min95 = mean- 2*std_error    max95 = mean+ 2*std_error        #create df    std_error = [('Gender',[data]),                 ('Standard Error',[round(std_error,2)]),                 ('Min  [68% range] ',[round(min68,2)]),                 ('Max [ 68% range] ',[round(max68,2)]),                 ('Min [95% range]',[round(min95,2)]),                 ('Max [95% range]',[round(max95,2)]),                  ]        std_error_df = pd.DataFrame.from_items(std_error)    return std_error_df
Female and Male salary DataFrame — without outliers.
Female_std_error = standard_error(US_2019_No_Outliers[US_2019_No_Outliers.Gender == ‘Female’].SalaryUSD, ‘Female’)Male_std_error = standard_error(US_2019_No_Outliers[US_2019_No_Outliers.Gender == ‘Male’].SalaryUSD, ‘Male’)#Append male and female dfa = Female_std_error.append(Male_std_error)a
In order to interpret the data and try to set a conclusion, we have to analyze the standard error looking at the overlapping nature of the bars. In our case, the error bars overlap and the overlapping region does not contain the mean of the lower sample. Due to this fact, we can assume that there is no strong evidence that the population is either similar or different. The uncertainty is too high in both the 68% and 95% range. We can clearly see this in the bar chart below.
def mean_error_plot(salary1, salary2, gender1, gender2):    #Calculate terms    mean1 = salary1.mean()    mean2 = salary2.mean()    std1 = salary1.std()    std2 = salary2.std()    count1 = salary1.count()    count2 = salary2.count()    std_error1 = std1/((count1)**(1/2))    std_error2 = std2/((count2)**(1/2))        #Create lists for plot    Genders = [gender1,gender2,gender1,gender2]    x_pos = np.arange(len(Genders))    Gd = [mean1,mean2,mean1,mean2]    error = [std_error1,std_error2,2*std_error1,2*std_error2]        #Create the plot        fig, ax = plt.subplots(figsize = (10,10))    ax.bar(x_pos, Gd, yerr=error, align='center', alpha=0.5, ecolor='black', capsize=10,color=['red', 'red', 'black', 'black'])    ax.set_ylabel('Salary [$]')    ax.set_xticks(x_pos)    ax.set_xticklabels(Genders)    ax.set_title('Genders - Range')    ax.yaxis.grid(True)         #Create horizontal lines    ax.axhline(y = 100936.60, color = 'r', linewidth = 1,xmin=0.143, xmax=0.38 , linestyle='dashed', alpha=0.5  )    ax.axhline(y = 101766.23, color = 'r', linewidth = 1,xmin=0.143, xmax=0.38, linestyle='dashed', alpha=0.5 )        ax.axhline(y = 99809.28, color = 'r', linewidth = 1,xmin=0.62, xmax=0.85 , linestyle='dashed', alpha=0.5  )    ax.axhline(y = 104318.53, color = 'r', linewidth = 1,xmin=0.62, xmax=0.85, linestyle='dashed', alpha=0.5 )        #Create legend    import matplotlib.patches as mpatches    red_patch = mpatches.Patch(color='red', label='68% range')     black_patch = mpatches.Patch(color='black', label='95% range')    plt.legend(handles=[black_patch,red_patch], loc = 'upper left')
Female and male DataFrame — without outliers
Female_no_out = US_2019_No_Outliers[US_2019_No_Outliers.Gender == ‘Female’].SalaryUSDMale_no_out = US_2019_No_Outliers[US_2019_No_Outliers.Gender == ‘Male’].SalaryUSD#apply mean_error_plo functionmean_error_plot(Female_no_out,Male_no_out,’Female’,’Male’)
So, what can we do now? How about using another approach to analyze it? That is exactly what we will do! Let’s compare the two means by hypothesis testing.
The reason for which we use the hypothesis test is to determine whether the null hypothesis is likely to be true given sample data. We accept the null hypothesis if there is little evidence against it and we might reject the null hypothesis in favor of the alternative hypothesis if the null is unlikely.
That being said, once we determine the null and alternative hypothesis, we are able to choose the significance level. This statistic term is the probability threshold that determines when we reject the null hypothesis. Moreover, After performing the test, the p-value helps us to interpret the results. This value is the level of marginal significance within a test representing the probability of the occurrence of a given event.
Back to our case study, let's start by setting the first hypothesis as stating that both salary means( male and female) are equal. Consequently, we will assume that the alternative hypothesis is that both means are not equal. In order to do that, we will carry out the two-sample T-test with the stats.ttest_ind() function.
Assumptions: We are assuming that both samples are normally distributed and have the same variance.
Let’s run the two sample t-test on both data with outliers and without it so that we can have some insight and consequently draw a conclusion.
# Import t-test functionfrom scipy.stats import ttest_ind# T-test - data without outliersttest_no_outliers = stats.ttest_ind(a = Female_no_out, b = Male_no_out, equal_var= False)ttest_no_outliers
pvalue_no_outliers = round(ttest_no_outliers[1],2)print(f’P-value of data without outliers: {pvalue_no_outliers}’)
The test gives us a p-value of 0.31 which is greater than the significance value ( 0.05). In this case, we cannot reject the null hypothesis. Thus we would expect to find the same gender sample means in 31% of our samples.
Now let’s carry out the test on the salary data which includes outliers on them.
# Filter salary by gender considering outliersFemale_salary_with_outliers = US_2019[US_2019.Gender == 'Female'].SalaryUSDMale_salary_with_outilers = US_2019[US_2019.Gender == 'Male'].SalaryUSD#T-test on data with outliersttest_outliers = stats.ttest_ind(a = Female_salary_with_outliers, b = Male_salary_with_outilers, equal_var= False)ttest_outliers
pvalue_outliers = round(ttest_outliers[1],2)print(f’P-value of data without outliers: {pvalue_outliers}’)
As we can see above, if we keep the outliers we can say that men and women are not paid equally due tue the p-value of 0.02 which is lower than the level of significance (0.05). So, this suggests that there is indeed a difference between male and female salary when we include the outliers in the data set.
After the analysis and assumptions we’ve made, I would say that it seems that both genders are getting paid equally in normal employment. However, when we include the number of people that are much better paid in our test, we can see a gap in the numbers of outliers — 97% of them are men
I know that it was a broad analysis. If we looked deep into all features, we would probably have found out reasons for that conclusion. In case you want to analyze it better, feel free to find the data on Brent’s original blog and apply all your knowledge. Please, let me know your conclusion!
Note : I also did an analysis using a different criteria for determining outliers. Check that out on my GitHub page.