Exploratory Data Analysis of Stack Overflow Developer Survey-2020
By: Mohiuddin Amanulla Chishty
Tags: Python, Data Science, Data Analysis, Data Visualization, Pandas

Shape reveals the number of columns and rows in the data set

By performing the above operation we get the information on the size of the data set it contains about 64461 rows and 61 columns that’s a huge amount of data.

Data Preparation and Cleaning:

The data we currently have contains 61 columns and that’s a large amount of info but for this exploration of data, we will restrict our analysis to the following questions.

The demographic of the respondents and the spread of programming community across different geographical location

What kind of programming skills, experience, and preferences is distributed across the globe

Employment-related trends, information, opinions, and preference.

hence I’ll be considering a subset of the actual data columns which are relevant to answering the above points

List of columns selected for this analysis

Now that we have the columns finalized let’s have a quick insight into the data by doing an info() call on the data.

Most columns have the data type, Object, either because they contain values of different types, or they contain empty values, which are represented using NaN. It appears that every column contains some empty values since the Non-Null count for every column is lower than the total number of rows (64461). We’ll need to deal with empty values and manually adjust the data type for each column on a case-by-case basis.

Only two of the columns were detected as numeric columns ( Age and WorkWeekHrs ), even though there are a few other columns that have mostly numeric values. To make our analysis easier, let's convert some other columns into numeric data types while ignoring any non-numeric value (they will get converted to NaNs)

Converting to numeric values and replacing null values with NaN.

Now the columns Age1stCode , YearsCode , YearsCodePro have been converted to numeric and if the value is not present it's replaced with Nan .let's look at the Basic statics of the numeric columns.

Describe method reveals basic statics on numeric columns

There seems to be a problem with the age column, as the minimum value is 1 and the max value is 279. This is a common issue with surveys: responses may contain invalid values due to accidental or intentional errors while responding. A simple fix would be to ignore the rows where the value in the age column is higher than 100 years or lower than 10 years as invalid survey responses.

Using the drop method to eliminate entries which are irrelevant

The gender column has multiple options, but to simplify our analysis, we’ll remove values containing more than one option.

dropping genders with multiple options

We’ve now cleaned up and prepared the dataset for analysis

Exploratory Analysis and Visualization:

Before we can ask interesting questions about the survey responses, it would help to understand what the demographics i.e. country, age, gender, education level, employment level, etc. of the respondents look like. It’s important to explore these variables in order to understand how representative the survey is of the worldwide programming community, as a survey of this scale generally tends to have some selection bias.

Country

Let’s look at the number of countries from which there are responses in the survey, and plot the 15 countries with the highest number of responses