Python Tricks for Data Science
Tags: Tricks, Pandas, Data Science, Python

This image reminds me of df.explode(), probably my favourite pandas method. Credit: Robert Zunikoff

Do you often find yourself scratching your head, trying to remember that really neat bit of pandas code you found two days ago on a stack overflow post? Well, that’s what this article is for — keeping all the syntactic sugar in one place.

This article is full of code I use on a regular basis; it’s a way to keep all my favourite snippets in one place and share it with others. I’ll come back and update this article with new ones and if you’ve found it useful, please feel free to send in other useful suggestions.

How to find the exact location of null values in a pandas dataframe and see the whole row?

# Syntax df[df.isna().any(axis=1)] # Example import pandas as pd

titanic = pd.read_csv(“train.csv”)

titanic[titanic.isna().any(axis=1)]

Comes in handy if you’re trying to see nan values in context with other values in the same row.

How to check for percentages of null values in each column?

This is a pretty fast way to see how bad the damage is in each column of your whole dataframe.

How to check if a column of strings contains any numbers that have been encoded as strings?

# Syntax df[df.column.str.contains("\d+", na=False)] # Example titanic[titanic.Ticket.str.contains("\d+", na=False)]

This is a good one for data cleaning or feature engineering — say you’re trying to get numbers out of a column or replace them.

How to reliably install python packages into a Jupyter notebook?

This is a pretty basic one but I find worth remembering! It’s much handier to fire up your notebook and then install any new dependencies you need in one place.

# Install a pip package in the current Jupyter kernel

import sys

!{sys.executable} -m pip install your-desired-python-package

This installs your desired package into the same python environment you’re using in that notebook.

How to make a correlation matrix much more readable?

I find the default output of seaborn heatmap representations of correlation matrices pretty confusing to read.

Half the output is not necessary and the color scheme isn’t that easy to read on first glance; this is my preferred solution.

It’s also good to round the values and specify the font too!

How to get all possible combinations of a dataframe’s columns?

This one comes in handy if you’re testing a model accuracy and trying to find out how different features affect the model.

It’s pretty processor intensive to do that and generally too automation heavy, but can be useful if you’re interested in honing in on a few specific feature combinations.

How to read in lots of csvs and join them together to create one dataframe?

import os

import glob # Creates a list containing absolute file paths for all csvs in # current working directory sheets = glob.glob(os.path.join(os.getcwd(), "*.csv")) # Use a generator to read in all the csvs as dataframes

# Then join them all together

# This only works if the columns are the same in each csv big_dataframe = pd.concat(pd.read_csv((s)for s in sheets))

This is one I use quite a lot after scraping a website or making an API call that’s rate limited, meaning I’ll write out data to file per API call and at the end will want to join it all together into one dataframe.

How to deal with a column of lists?

The explode method creates a new row with duplicate information for each item in the list, which is pretty nifty.

While this might create some additional new data that you don’t want, it allows you to hit some new data points. In the example above, you could now find out which actors typically get the highest star_rating.