A Complete Pandas Glossary for Data Science | by Terence Shin | Towards Data Science

Like most others, I tried to learn Pandas through boot camps — unfortunately, the problem with boot camps is that you forget everything pretty quickly if you don’t practice what you learn!
At the same time, I found that there was a need for a central Pandas resource that I could refer to when working on personal data science projects. That’s how this came into fruition. Use this as a resource to learn Pandas and also to refer to!
SetupCreating and Reading DataManipulating DataFramesSummary FunctionsMapping FunctionsGrouping and Sorting VariablesHandling Missing Data
Setup
Creating and Reading Data
Manipulating DataFrames
Summary Functions
Mapping Functions
Grouping and Sorting Variables
Handling Missing Data
import pandas as pd
A DataFrame is simply a table made up of multiple arrays. In the example below, the code would create a table with two columns, ABC and DEF.
pd.DataFrame({'ABC':[1,2,3],'DEF':[4,5,6]},index=[1,2,3])
A Series is a sequence of values, also known as a list. From a visual perspective, imagine it being one column of a table.
pd.Series([1,2,3],index=[], name ='ABC')
The most common way of getting our data. This converts a CSV file into a DataFrame.
# exampledf = pd.read_csv("filename.csv", index_col=0)
Vice versa, if you want to convert a DataFrame into a CSV, you can use the code below:
# exampledf.to_csv("filename.csv", index_col=0)
This tells you how large a DataFrame is and is in the format (rows, columns).
df.shape()
If you want to get a visual idea of what a DataFrame looks like, .head() returns the first 5 rows of the given DataFrame.
df.head()
# For one columndf.variable.dtype# For all columnsdf.dtypes
This is useful if you want to convert integers to floats (or vice versa).
df.variable.astype()
# a) Method 1df.property_name# b) Method 2df['property_name']
# if you want to get the first value in a seriesdf['property_name'][0]
Index-based selection retrieves data based on its numerical position in the DataFrame. It follows a rows-first, columns-second format. Iloc’s indexing scheme is such that the first number is inclusive and the last number is exclusive.
df.iloc[]
Label-based selection is another way to index a DataFrame, but it retrieves data based on the actual data values rather than the numerical position. Loc’s indexing scheme is such that the both the first and last values are inclusive.
df.loc[]
Because label-based selection relies on the index of the DataFrame, you can use .set_index() to assign a column to the index.
df.set_index("variable")
We can filter out a DataFrame using Label-based selection too.
# a) Single Condition df.loc[df.property_name == 'ABC']# b) Multiple conditions using ANDdf.loc[df.property_name == 'ABC' & df.property_name == 'DEF']# c) Multiple conditions using ORdf.loc[df.property_name == 'ABC' | df.property_name == 'DEF']
We can use isin() to filter a DataFrame as well. If you know SQL, it’s similar to the WHERE ___ IN() statement.
df.loc[df.property_name isin(['ABC','DEF'])
The first line of code will filter a DataFrame to only show rows where the property name is null. Vice versa, the second line of code with the filter it so that the property name is not null.
df.loc[df.property_name.isnull()]df.loc[df.property_name.notnull()]
df['new_column'] = 'ABC'
You’ll often want to rename a column to something easier to refer to. Using the code below, the column ABC would be renamed to DEF.
df.rename(columns={'ABC': 'DEF'})
This gives a high-level summary of a DataFrame or a variable. It is type-sensitive, meaning that its output will be different for numerical variables compared to string variables.
df.describe()df.variable.describe()
This returns the average of a variable.
df.variable.mean()
This returns all of the unique values of a variable.
df.variable.unique()
This shows a list of unique values and also the frequency of occurrence in the DataFrame.
df.variable.value_counts()
Mapping is used to transform an initial set of values to another set of values through a function. For example, we could use mapping to convert the values of a column from meters to centimeters or we could normalize the values.
.map() is used to transform a Series.
df.numerical_variable.map()
.apply() is similar to .map(), except that it transforms the entire DataFrame.
df.numerical_variable.apply()
Get the count for each value of a variable (same as value_counts)
df.groupby('variable').variable.count()
Get the min value for each value of a variable
df.groupby('variable').variable.min()
Get a summary (length, min, max) for each value of a variable
df.groupby(['variable']).variable.agg([len, min, max])
Multi-indexing
df.groupby(['variable_one', 'variable_two'])
Sorting by one variable
df.sort_values(by='variable', ascending=False)
Sorting by multiple variables
df.sort_values(by=['variable_one', 'variable_two'])
Sorting by index
df.sort_index()
Handling missing data is one of the most important steps in EDA. Below are a number of ways to handle missing data.
df.isna().sum()
If you have a DataFrame with a large number of rows and you can afford to remove rows with null values completely, then .dropna() is a useful tool.
df.dropna()
This is similar to above except that it drops any columns with null values rather than the rows.
df.dropna(axis=1)
If you would rather fill missing values instead of removing the row or column completely, you can use the code below:
df.variable.fillna("n/a")
Let’s say there’s a DataFrame where someone already filled missing values with “n/a”, but you want the missing values to be filled with “unknown”. Then you can use the following code below:
df.variable.replace("n/a", "unknown")
This is useful when you want to combine two DataFrames that have the same columns. For example, if we wanted to combine January sales and February sales together to analyze longer-term trends, you could use the following code:
Jan_sales = pd.read_csv("jan_sales.csv")Feb_sales = pd.read_csv("feb_sales.csv")pd.concat([Jan_sales, Feb_sales])
If you want to combine two columns that have a common index (e.g. customer_id), then you can use .join().
Use the parameter, on, to determine what column to join on.
To determine if it’s a left, right, inner, or outer join, you use the parameter, how.
# exampletable_1.join(table_2, on='customer_id', how='left')
If you don’t know about SQL joins, read here. It’s essentially the same idea.
I hope this is helpful! Please provide comments if you feel that I missed something or if something isn’t clear. Thanks!
If you like my work and want to support me...
The BEST way to support me is by following me on Medium here.Be one of the FIRST to follow me on Twitter here. I’ll be posting lots of updates and interesting stuff here!Also, be one of the FIRST to subscribe to my new YouTube channel here!Follow me on LinkedIn here.Sign up on my email list here.Check out my website, terenceshin.com.
The BEST way to support me is by following me on Medium here.
Be one of the FIRST to follow me on Twitter here. I’ll be posting lots of updates and interesting stuff here!
Also, be one of the FIRST to subscribe to my new YouTube channel here!
Follow me on LinkedIn here.
Sign up on my email list here.
Check out my website, terenceshin.com.