Introduction to Pandas DataFrames | by Rebecca Vickery | Towards Data Science

Pandas is a python package designed for fast and flexible data processing, manipulation and analysis. Pandas has a number of fundamental data structures (a data management and storage format). If you are working with two-dimensional labelled data, which is data that has both columns and rows with row headers — similar to a spreadsheet table, then the DataFrame is the data structure that you will use with Pandas.
In the following article, I will be giving an introduction to some of the most common data processing and manipulation methods available in this package.
Throughout this tutorial, I will be using a dataset from drivendata.org, a machine learning competition website. The dataset can be downloaded here.
The dataset comprises a set of training data for fitting a machine learning model and a test data file for making predictions. For the purposes of this article, as I am only performing preprocessing and analysis I will only be using the training data.
The first step when working with data using Pandas is to read the data into your editor, throughout this article I will be using Jupyter Notebooks.
Pandas provides methods for most types of data imports.
To obtain data from a CSV file you can use the pandas.read_csv() function. The features (values) and labels in the dataset we are using are in separate files so we will read both in and will learn how to merge them together a bit later.
import pandas as pdtrain_values = pd.read_csv('train_values.csv')train_labels = pd.read_csv('train_labels.csv')
Pandas also has functions to read data from a variety of other sources including:
Excel — pandas.read_excel().
JSON — pandas.read_json().
SQL — pandas.read_sql().
Google BigQuery — pandas.read_gbq().
When working on a data science project the data that you are working with will often be in multiple files and/or from multiple sources. There will often, therefore, be times when you need to combine datasets. There are a number of ways you might want to do this and pandas has a method for each.
Merge
If you have had previous experience working with SQL queries then this method will be quite familiar. Merging is the process of joining two datasets together via a common identifier, similar to a join in SQL.
In the case of our datasets, we have a train_values dataset and a train_labels dataset each with the same building_id per row which is referred to as the key.
The merge function takes a number of arguments but the three we need to pay attention to for combining our datasets is how and on. A full list of arguments and their explanations can be found here.
The how argument refers to the method that determines which keys should appear in the resulting data set. For example, you may have rows in the values dataset which do not appear in the labels dataset. If this was the case we would need to determine which rows to keep. There are four variables for the how argument.
left — this argument would keep only the keys from the left data frame.
right — this would only keep key from the right data frame.
outer — will keep all keys from both data frames including non-matching rows.
inner — will keep all keys from both data frames but will not include non-matching rows.
The below code joins the two data frames with an inner join using building_id in the on argument.
train_data = pd.merge(train_values, train_labels, how='inner', on=['building_id', 'building_id'])
Concatenate
The concat() function in pandas is another method for combining data frames. However, rather than joining on a common key concat() effectively glues the data frames together either by adding columns to columns or rows to rows referred to as the axis.
As our two datasets have the exact same indexes we can use the concat function to combine them as a direct replacement for merge. In the below, we pass the two data frames as a list and specify axis=1 which tells the function that we want to concatenate on columns. The downside of concat in our case is that we now have two occurrences of the building_id column.
train_data_concat = pd.concat([train_values, train_labels], axis=1)
One of the most common problems in data sets is missing data. Pandas has a number of methods for handling missing values in data frames.
For dealing with missing data we have a number of options.
Drop all the rows containing missing data in any column.
Drop the column or columns containing missing data.
Impute the missing value with a sensible replacement this could be a 0 or perhaps the median value for the remaining data in the column.
Pandas has methods to perform each of these.
Firstly to detect if there are missing values pandas has the isna() function. This returns True if the value is missing and False if not. We can use this to determine the volume of missing values per column as shown below.
train_data.apply(lambda x: sum(x.isna()) / len(train_data))
Our data set doesn’t actually have any missing values but let's work through examples of how we might handle them if we did.
The Pandas function drop_na() drops rows or columns (depending on the parameter you choose) that contain missing values. This function takes the axis parameter which you set as 0 to drop rows, and 1 to drop columns.
The alternative function is fillna() . This function will replace missing values with the value of your choice. You can replace with a fixed value such as 0, or you can use a calculation such as the mean. You can also apply different values to different columns by passing a dictionary of values per column.
The below code fills any missing values with the mode for that column.
train_data_numeric = train_data.apply(lambda x: x.fillna(x.mode()),axis=0)
During both data analysis and preprocessing, there will be many occasions when you will want to select a subset of your data. You may want to select specific columns, a subset of rows or create subsets of your data based on specific conditions. Pandas provides a variety of methods for performing these tasks.
To select subsets and rows of data from your data frame based on either labels or positions pandas has two methods loc and iloc.
The loc method selects rows based on the index label. Let’s walk through a quick example.
This is our data frame. The index is the column of numbers to the left of the building_id column.
In pandas, we always count from 0 so if we want to select the first three rows using loc we run the following.
subset_loc = train_data.loc[[0, 1, 2]]subset_loc.head()
This gives us the first three rows.
The iloc method select rows by the index position. This might be used, for example, if the user does not know the index or if the index is not numeric.
Selecting the first three rows looks very similar to the loc method.
subset_iloc = train_data.iloc[[0, 1, 2]]subset_iloc.head()
To select rows based on specific conditions we can use the where()function. This is very similar to the where clause in a SQL query. A few examples are shown below.
To select data based on a boolean condition. The below code selects only the rows where the value in the columnhas_superstructure_cement_mortar_brick is 1.
has_cement_mortar_brick = train_data[train_data['has_superstructure_cement_mortar_brick'] == 1]has_cement_mortar_brick.head()
To select data based on a numerical condition. The below code selects only the rows where the value of the age column is greater than 50.
has_age_greater_than_50 = train_data[train_data['age'] > 50]has_age_greater_than_50.head()
You can combine multiple conditions as shown below.
two_conditions = train_data_concat[train_data['has_superstructure_cement_mortar_brick'] == 1 & (train_data['age'] > 50)]two_conditions.head()
This article has given a brief introduction to pandas data frames and some of the most common tasks you may perform on them. The Pandas user guide is an excellent reference point to find out more about these and other methods available in the library. If you are interested in finding out more about how to analyse data with the Pandas package I have written another article which can be found here.
Thanks for reading!
I send out a monthly newsletter if you would like to join please sign up via this link. Looking forward to being part of your learning journey!