How to Explore Datasets for Beginners
By: Mars Escobin
Tags: Data Science, Data Analysis, Jupyter Notebook, Exploratory Data Analysis, Data

Data exploration can be overwhelming for anyone who has little to no background in data analysis. There is technically no limits to what you can explore and there is no guidelines to what you should be looking for. But as the saying goes, all journeys begin with a single step. This post attempts to lay down not one but five easy steps you can follow when you’re exploring a dataset. I call it LAUGH:

Step 1: L-oad the data Step 2: A-sk for definitions Step 3: U-se questions Step 4: G-et a feel Step 5: H-ave next actions

By the end of this post, you would have learned what you should care about when exploring a dataset and how to do basic data exploration in a Jupyter Notebook.

Jupyter Notebook

First we discuss the software we will be using to explore data- Jupyter notebook. Jupyter stands for Julia, Python and R which are three common programming languages in data science. Born in 2014, a Jupyter notebook is an interactive computing environment that many scientists have hailed to be the de facto choice for data exploration.

Imagine a Google Doc that can crunch numbers as well as create tables and graphs for you. This is what a Jupyter notebook can do, and much more. I thought Random Nerds gave a beginner friendly tutorial on how to install it in case you haven’t yet.

STEP 1: Load the data

We start LAUGH-ing by loading our data into a Jupyter notebook. The easiest way to do so is by using a powerful python library called Pandas (for Python Data Analysis Library). But first, we create our notebook:

Once inside the new notebook, import the pandas library and give it a shorter name (optional). I will call it ‘pd’ which is the conventional alias for pandas.

Pandas

Pandas provide straightforward commands to read almost any type of data. The most common filetypes are csv, json and excel:

pd.read_excel("filename") | For an Excel file

pd.read_csv("filename") | For a CSV file

pd.read_json("filename") | For a JSON file

You can load a dataset straight from a URL. For example, I used Pandas to read a CSV dataset from a website:

You can also download the dataset first and load them from your computer. For example, I stored a CSV file in my Downloads folder before loading it into my notebook:

STEP 2: Ask for definitions

Perhaps the most underrated step of any dataset exploration is the one that deals with understanding the variables inside it. Having a good grasp of what each column header means will help form a story in your head as to what variables are likely to be important and which ones are likely okay to ignore. Familiarizing yourself with how the columns were sourced and measured will make your exploration easier. It will enable you to anticipate risks such as missing data, data duplicates and biases in the data.

Often you will find these definitions from a data dictionary which is typically available for public datasets. For example:

However datasets from private organizations are rarely documented well, in which case you have to ask for the definitions from someone who is more familiar with it. Prepare to sit down with this person as you go over what each column means and how the data inside them were measured and sourced.

Step 3: Use questions

Using questions is the smartest thing you can do during an EDA (a fancier term for data exploration, stands for Exploratory Data Analysis). If you skip this step, you will likely find yourself fatigued and neck deep into the rabbit hole because what makes EDAs overwhelming is the exponential number of paths you can explore. As the cheshire cat says “.. if you don’t know where you are going, any road will take you there.” To avoid being sucked into the black hole that is EDA, pause and list out your questions first before writing any code. This will allow you to specify which variables you care about, what to do with them and when to stop the exploration Remember, the data is only as good as the questions it’s trying to answer.

Step 4: Get a feel

This step is the heart of the exploration. This is where we actually look at the data with the goal of developing an intuition by spotting patterns and anomalies. As mentioned, questions are the guiding light of EDAs and we can start getting a feel of the data by listing out common questions such as:

“How big of a dataset am I working with?”

“How does the dataset look like?”

“What kind of variables are in the dataset?”

All these can be answered pretty quickly with straightforward Pandas commands. As a walkthrough, I will use a Covid-19 dataset I downloaded from WHO’s website.

Notice I stored the dataset in a variable called “covid”. This will help me call the dataset without typing “pd.read_csv(‘filename’)” every time. Now, let’s start answering some questions.