Python for Product Managers — Part 2: Using pandas for data analysis
By: Joshua K
Tags: Metrics, Data Analysis, Python, Product Management, Data Science

In Part 1 of this series, we looked at installing Python, setting up Jupyter Notebooks and a glimpse of how you can use Python in your data analytics workflow.

As we saw last time, Pandas is a really powerful module for data analysis in Python. In this instalment, we’ll look at the tools available to us to load and explore some data.

Importing pandas

Importing the pandas module is the first step, but it’s luckily very easy. A common convention seems to be to import the pandas module and give it the alias “pd” — I guess this just saves a few keystrokes.

To import pandas you simply need to include the following line at the top of your script or notebook:

import pandas as pd

If you’re using Jupyter notebooks and want visualisations to appear in line then the following snippet will import pandas, enable inline visualisation and set some default styles:

# Show graphs and charts inline in the

notebook %matplotlib inline # Import some libraries

import pandas as pd

import matplotlib

import matplotlib.pyplot as plt # Set some default plot styles to make things look nice matplotlib.rcParams['figure.figsize'] = (20.0, 10.0) plt.style.use('bmh')

I like the bmh style, but there are many others available, so take a look and choose one that you like.

Understanding Pandas dataframes

The main data structure in Pandas is the dataframe. You can think of this a bit like an Excel spreadsheet or a database table. It’s a collection of data with columns and rows.

Each row has an index, by default an integer, but you can set other columns of unique data to be the index.

Loading data from text files

Loading a CSV file is easy with the from_csv method:

df = pd.read_csv('my_csv_file.csv')

There are a few useful parameters you can set to help with different situations:

sep — define the column separator that is used in the file (e.g. \t for tab, |) — the default is a comma (‘,’)

— define the column separator that is used in the file (e.g. \t for tab, |) — the default is a comma (‘,’) parse_dates — specify a column or an array of columns that contain dates and pandas will try to ensure that the datatype of the data in the column will be a datetime — this makes it easier to do date based grouping (such as group by Year or Month)

— specify a column or an array of columns that contain dates and pandas will try to ensure that the datatype of the data in the column will be a datetime — this makes it easier to do date based grouping (such as group by Year or Month) header=None — use this if the data file has no headers for data. the names parameter is useful alongside this to set the name of each column, for example names=["col1", "col2", "col3"]

And so, a more complex example of loading a tab delimited file containing data like the example above would look like:

df = pd.read_csv('my_csv_file.txt', sep='\t' header=None, names=['id', 'end_datetime', 'start_datetime', 'user_id'], parse_dates=['end_datetime','start_datetime'])

Loading data from a database

Reading data from a database is also relatively easy using the from_sql method.Obviously you’ll need to be able to access the database and to be able to write SQL to define the data that you want out. Here’s an example using the mysql.connector module to read the sessions data directly from the database.

import mysql.connector # Create a connection to the database

conn = mysql.connector.connect(host='hostname',database='db_name',user='user_name',password='xxxxxx') # Define the SQL query that will extract the data

sql = "SELECT * FROM sessions" # Create a dataframe containing the results of the query

df = pd.read_sql(sql, conn, parse_dates=['start_datetime', 'end_datetime']) df.head()

I just used MySQL as an example there, assuming you have the right database modules available and loaded, you can easily get data from SQLite, MS SQL Server or PostgreSQL using the same syntax.

Obviously, the advantage of reading directly from the database are twofold:

If you know SQL, you might be able to do some filtering and data manipulation before you even load the data into Python Assuming your database gets updated frequently, its easy to refresh and repeat your analysis to get up to date insights into your data

Whilst point 1 above is true, one of the advantages of Python is that a lot of the data manipulation happens in memory on your local machine and so may be faster than some database operations on large or slow databases.

Queries with parameters

Depending on what you’re doing, you might want to pass in a parameter to the query. There’s a nice way to do this without messing about with the SQL as a string.

# Define the sql with parameter placeholders

sql = "SELECT * FROM sessions WHERE YEAR(start_datetime) = %s" # Extract the data with the parameters supplied

df = pd.read_sql(sql, conn, parse_dates=['start_datetime', 'end_datetime'], params=[2018])

df.head()

And of course, you can have several parameters in the query, just pass more in via the params array.

Loading data from existing Python variables

Finally, you might have some existing Python data structures kicking around that you’d like as a dataframe. Pandas provides an easy way to load a Python dictionary into a dataframe via from_dict:

my_dict = [

{'id': 0, 'name':'Bob', 'age':64},

{'id': 1, 'name':'Sally', 'age':61},

{'id': 2, 'name':'June', 'age':32},

{'id': 3, 'name':'Nish', 'age':41},

{'id': 4, 'name':'Lexi', 'age':25} ] df = pd.DataFrame.from_dict(my_dict) df.head()

You might of noticed above the use of df.head() to display a summary of the data in the dataframe. Well, you’ve probably guessed it, but df.head() shows a summary of the first 5 rows of the dataframe. If you want to see the last 5 rows, then there’s df.tail() . You can also set how many rows you want to see: df.head(n=10)

Sorting a pandas dataframe

Of course, first 5 and last 5 is pretty meaningless unless that data is in some kind of order!

df.sort_values('start_datetime').head(n=3)

By default it will sort ascending. To flip that, we can do the following:

df.sort_values('start_datetime', ascending=False).head(n=3)

And finally, you can sort based on several columns by passing in an array:

df.sort_values(['start_datetime', 'user_id']).head(n=3)

Filtering data in a pandas dataframe

Another common operation might be to filter data based on a column value:

df[df['user_id']=='8ec6e7e7-a5d7-4a48-bb8a-d4d899e1f5be'].head()

Remember when we loaded the data we made extra sure that the dates in our data were parsed as datetimes? Well, that means we can find all the sessions in a particular month and year as follows:

df[(df['start_datetime'].dt.month==10) & (df['start_datetime'].dt.year >= 2017)].head()

So, there you go, some powerful support for loading and exploring data. Next time we’ll look at grouping, transforming and joining data!

If you’ve got any comments, suggestions or requests then please let me know in the comments.

Read the rest of the series

Follow the full series of posts to master Python!