My Python Pandas Cheat Sheet. The pandas functions I use everyday as... | by GreekDataGuy | Towards Data Science

A mentor once told me that software engineers are like indexes not textbooks; we don’t memorize everything but we know how to look it up quickly.
Being able to look up and use functions fast allows us to achieve a certain flow when writing code. So I’ve created this cheatsheet of functions I use everyday building web apps and machine learning models.
This is not a comprehensive list but contains the functions I use most, an example, and my insights as to when it’s most useful.
Contents:1) Setup2) Importing3) Exporting4) Viewing and Inspecting5) Selecting6) Adding / Dropping7) Combining8) Filtering9) Sorting10) Aggregating11) Cleaning12) Other13) Conclusion
If you want to run these examples yourself, download the Anime recommendation dataset from Kaggle, unzip and drop it in the same folder as your jupyter notebook.
Next Run these commands and you should be able to replicate my results for any of the below functions.
import pandas as pdimport numpy as npanime = pd.read_csv('anime-recommendations-database/anime.csv')rating = pd.read_csv('anime-recommendations-database/rating.csv')anime_modified = anime.set_index('name')
Convert a CSV directly into a data frame. Sometimes loading data from a CSV also requires specifying an encoding (ie:encoding='ISO-8859–1'). It’s the first thing you should try if your data frame contains unreadable characters.
Another similar function also exists called pd.read_excel for excel files.
anime = pd.read_csv('anime-recommendations-database/anime.csv')
Useful when you want to manually instantiate simple data so that you can see how it changes as it flows through a pipeline.
df = pd.DataFrame([[1,'Bob', 'Builder'],                  [2,'Sally', 'Baker'],                  [3,'Scott', 'Candle Stick Maker']], columns=['id','name', 'occupation'])
Useful when you want to make changes to a data frame while maintaining a copy of the original. It’s good practise to copy all data frames immediately after loading them.
anime_copy = anime.copy(deep=True)
This dumps to the same directory as the notebook. I’m only saving the 1st 10 rows below but you don’t need to do that. Again, df.to_excel() also exists and functions basically the same for excel files.
rating[:10].to_csv('saved_ratings.csv', index=False)
Display the first n records from a data frame. I often print the top record of a data frame somewhere in my notebook so I can refer back to it if I forget what’s inside.
anime.head(3)rating.tail(1)
This is not a pandas function per se but len() counts rows and can be saved to a variable and used elsewhere.
len(df)#=> 3
Count unique values in a column.
len(ratings['user_id'].unique())
Useful for getting some general information like header, number of values and datatype by column. A similar but less useful function is df.dtypes which just gives column data types.
anime.info()
Really useful if the data frame has a lot of numeric values. Knowing the mean, min and max of the rating column give us a sense of how the data frame looks overall.
anime.describe()
Get counts of values for a particular column.
anime.type.value_counts()
This works if you need to pull the values in columns into X and y variables so you can fit a machine learning model.
anime['genre'].tolist()anime['genre']
Create a list of values from index. Note I’ve used anime_modified data frame here as the index values are more interesting.
anime_modified.index.tolist()
anime.columns.tolist()
I do this on occasion when I have test and train sets in 2 separate data frames and want to mark which rows are related to what set before combining them.
anime['train set'] = True
Useful when you only want to keep a few columns from a giant data frame and don’t want to specify each that you want to drop.
anime[['name','episodes']]
Useful when you only need to drop a few columns. Otherwise, it can be tedious to write them all out and I prefer the previous option.
anime.drop(['anime_id', 'genre', 'members'], axis=1).head()
We’ll manually create a small data frame here because it’s easier to look at. The interesting part here is df.sum(axis=0) which adds the values across rows. Alternatively df.sum(axis=1) adds values across columns.
The same logic applies when calculating counts or means, ie: df.mean(axis=0).
df = pd.DataFrame([[1,'Bob', 8000],                  [2,'Sally', 9000],                  [3,'Scott', 20]], columns=['id','name', 'power level'])df.append(df.sum(axis=0), ignore_index=True)
Use this if you have 2 data frames with the same columns and want to combine them.
Here we split a data frame in 2 them add them back together.
df1 = anime[0:2]df2 = anime[2:4]pd.concat([df1, df2], ignore_index=True)
This functions like a SQL left join, when you have 2 data frames and want to join on a column.
rating.merge(anime, left_on=’anime_id’, right_on=’anime_id’, suffixes=(‘_left’, ‘_right’))
The index values in anime_modified are the names of the anime. Notice how we’ve used those names to grab specific columns.
anime_modified.loc[['Haikyuu!! Second Season','Gintama']]
This differs from the previous function. Using iloc, the 1st row has an index of 0, the 2nd row has an index of 1, and so on... even if you’ve modified the data frame and are now using string values in the index column.
Use this is you want the first 3 rows in a data frame.
anime_modified.iloc[0:3]
Retrieve rows where a column’s value is in a given list. anime[anime[‘type’] == 'TV'] also works when matching on a single value.
anime[anime['type'].isin(['TV', 'Movie'])]
This is just like slicing a list. Slice a data frame to get all rows before/between/after specified indices.
anime[1:3]
Filter data frame for rows that meet a condition. Note this maintains existing index values.
anime[anime['rating'] > 8]
Sort data frame by values in a column.
anime.sort_values('rating', ascending=False)
Count number of records for each distinct value in a column.
anime.groupby('type').count()
Note I added reset_index() otherwise the type column becomes the index column — I recommend doing the same in most cases.
anime.groupby(["type"]).agg({  "rating": "sum",  "episodes": "count",  "name": "last"}).reset_index()
Nothing better than a pivot table for pulling a subset of data from a data frame.
Note I’ve heavily filtered the data frame so it’s quicker to build the pivot table.
tmp_df = rating.copy()tmp_df.sort_values('user_id', ascending=True, inplace=True)tmp_df = tmp_df[tmp_df.user_id < 10] tmp_df = tmp_df[tmp_df.anime_id < 30]tmp_df = tmp_df[tmp_df.rating != -1]pd.pivot_table(tmp_df, values='rating', index=['user_id'], columns=['anime_id'], aggfunc=np.sum, fill_value=0)
Set cells with NaN value to 0 . In the example we create the same pivot table as before but without fill_value=0 then use fillna(0) to fill them in afterwards.
pivot = pd.pivot_table(tmp_df, values='rating', index=['user_id'], columns=['anime_id'], aggfunc=np.sum)pivot.fillna(0)
I use this all the time taking a small sample from a larger data frame. It allows randomly rearranging rows while maintaining indices if frac=1.
anime.sample(frac=0.25)
Iterate over index and rows in data frame.
for idx,row in anime[:2].iterrows():    print(idx, row)
Start notebook with a very high data rate limit.
jupyter notebook — NotebookApp.iopub_data_rate_limit=1.0e10
I hope this can be a reference guide for you as well. I’ll try to continuously update this as I find more useful pandas functions.
If there are any functions you can’t live without please post them in the comments below!