A Practical Summary of Pandas in 30 Python Snippets | by Lucas Soares | Towards Data Science

Pandas is one of the most important Python libraries for Data Science and Machine Learning. I used it practically every day but I always feel like I am missing out on useful functionalities, so I decided to write a compressed summary to help guide my progress with my pandas' skills.
In this article, I will share a practical summary of the main functionalities of the pandas library in 30 Python snippets.
The functionalities we will cover here are:
Working with dataframes
Reading and writing tabular data
Filtering rows in dataframes
Selecting specific rows and columns
Creating plots
Creating new columns from existing columns
Calculating summary statistics
Sorting values from a table
Combining data from multiple tables
Handling Time Series Data
Manipulating Textual Data
Let’s get started!
Let’s start with creating a basic dataframe from a dictionary:
import pandas as pddata = {"Country": ["Portugal", "USA", "France"],             "Population": [10.31*10**6, 329.5*10**6, 67.39*10**6]               }df = pd.DataFrame(data)df
When you use a Python dictionary to create a pandas dataframe, the keys from that dictionary will be the column headers of the dataframe and the corresponding values will be the columns. You can store multiple data types in the same dataframe.
To select a column from a pandas dataframe we do:
df["Country"]# Output0    Portugal1         USA2      FranceName: Country, dtype: object
Using brackets “[]”, which will return a Series object.
As stated in the documentation:
A pandas Series has no column labels, as it is just a single column of a DataFrame. A Series does have row labels.
We can also check the data type for each column according to pandas interpretation:
df.dtypesCountry        objectPopulation    float64dtype: object
In this case, the data types are floats (float64) and strings (object)
Pandas provides the read_csv()function to read data stored as a csv file into a pandas DataFrame. Pandas supports many different file formats or data sources out of the box (csv, excel, sql, json, parquet).
df = pd.read_csv("Advertising.csv")df
If you want to see a controlled subset of the data you can do:
df.head(8)
Here we are showing only the first 8 rows of the dataframe.
If we want to visualize the last 8 rows of the dataframe we can do:
df.tail(8)
We can also read excel files easily with pandas:
df = pd.read_excel("Advertising.xlsx")df
You can get a technical summary from a dataframe:
df.info()# Output<class 'pandas.core.frame.DataFrame'>RangeIndex: 200 entries, 0 to 199Data columns (total 4 columns): #   Column     Non-Null Count  Dtype  ---  ------     --------------  -----   0   TV         200 non-null    float64 1   radio      200 non-null    float64 2   newspaper  200 non-null    float64 3   sales      200 non-null    float64dtypes: float64(4)memory usage: 6.4 KB
This will output technical information about the dataframe like the number of entries, the data types, the number of columns, the number of null-values, etc.
Let’s say I am interested in TV ratings bigger than 100.0, how could I filter my dataframe to get that specific subset?
df[df["TV"]>100]
We can also do conditional expressions to help our filtering. For example, say we have the titanic dataset (same used in Pandas documentation):
df = pd.read_csv("titanic.csv")df
And we wanted the info from people under 35 years that survived - we would then use:
survived_under_35 = df[(df["Survived"]==1) & (df["Age"]<35)]survived_under_35
Here we use a conditional expression inside parenthesis combined with the and operator here as & to combine the two filtering statements.
If we want to work only with the values from a specific column that are not null, we can use the conditional function notna() that returns True for each row the values are not a Null value.
df[df["Age"].notna()]
For more on conditional expressions and filtering, see this section of the pandas documentation.
Say I wanted the names of the passengers older than 40 years old. Since I want a subset of both rows and columns, using just selection brackets [] is not going to be sufficient anymore, so I use the loc/iloc operators:
df.loc[df["Age"]>40, "Name"]6                                McCarthy, Mr. Timothy J11                              Bonnell, Miss. Elizabeth15                      Hewlett, Mrs. (Mary D Kingcome) 33                                 Wheadon, Mr. Edward H35                        Holverson, Mr. Alexander Oskar                             ...                        862    Swift, Mrs. Frederick Joel (Margaret Welles Ba...865                             Bystrom, Mrs. (Karolina)871     Beckwith, Mrs. Richard Leonard (Sallie Monypeny)873                          Vander Cruyssen, Mr. Victor879        Potter, Mrs. Thomas Jr (Lily Alexenia Wilson)Name: Name, Length: 150, dtype: object
As said in the documentation:
When using loc/iloc, the part before the comma is the rows you want, and the part after the comma is the columns you want to select.
When using the column names, row labels, or a condition expression, use the loc operator in front of the selection brackets [].
For both the part before and after the comma, you can use a single label, a list of labels, a slice of labels, a conditional expression, or a colon. If you use a colon that means you want to select all rows or columns.
When your selection is based on the position on the table, use iloc. For example, say you wanted the rows from 23 to 42 and the columns 1 to 3:
df.iloc[22:42, 0:3]
When selecting specific rows and/or columns with loc|iloc, new values can be assigned to the selected data. For example, to assign the name anonymous to the first 3 elements of the third column. For more on selection see here.
df = pd.read_csv("Advertising.csv")
If you want to quickly visualize the data you can simply call the .plot() method:
df.plot();
Here pandas creates a one-line plot for each column with numeric data. If you want to get specific, you can select the desired column, for example, say you wanted to plot only the sales:
df["sales"].plot();
If you wanted to assess the effect of TV/radio/newspaper ratings on sales, you could plot them against each other one by one:
df.plot.scatter(x="TV", y="sales", alpha=0.6);
df.plot.scatter(x="radio", y="sales", alpha=0.6);
df.plot.scatter(x="newspaper", y="sales", alpha=0.6);
There are many plotting options in pandas, and we can check them using the built-in dir() function from Python:
for method_name in dir(df.plot):    if not method_name.startswith("_"):        print(method_name)areabarbarhboxdensityhexbinhistkdelinepiescatter
For more plotting with pandas, check this section of the pandas documentation.
Creating a new column in pandas is simple. Just use brackets [] with the new column name at the left side assignment and the data for that column on the right side assigment:
df["Date"] = pd.date_range("1965-01-01", periods=len(df))df
Note: here I also used another cool pandas method called date_range which generates a list of date values from a starting point with a given periodicity indicated by the period parameter.
But, what if we want to create a new column from an existing column? For example, say we had the air quality dataset used by the pandas documentation:
df = pd.read_csv("air_quality_no2.csv")df
Say you want to check the ratio of the values in Antwerp versus London and save that result as a new column:
df["ratio_antwerp_london"] = (df["station_antwerp"] / df["station_london"])df
If you wanted to apply a more complex function to a column and have the result saved as a new column of the dataframe, you could use the .apply() method.
For example, let's apply a conditional to the NO2 values from the London station, where, if they are larger than a threshold, the output will be “High” and otherwise “Low”::
def highLow(temp):    if temp > 24.77:        return "High"    else:        return "Low"df["station_london_high_low"] = df["station_london"].apply(highLow)df
To rename data columns use pandas .rename():
data = {"Name": ["John", "Mary", "Paul"],             "Age": [20,25,30],             "Occupation": ["Software Engineer", "Artist", "Product Manager"]               }df = pd.DataFrame(data)df_renamed = df.rename(columns={"Name":"First Name",                                      "Occupation": "Job"})df_renamed
Notice that I don’t have to rename all of the columns, the Age column remained the same.
Calculating statistics in pandas is straight forward:
df = pd.read_csv("titanic.csv")print(df["Age"].mean())print(df["Age"].max())print(df["Age"].min())print(df["Age"].median())29.6991176470588280.00.4228.0df.describe()
You can also calculate specific combinations of aggregating statistics for given columns:
df.agg(    {        "Age": ["min", "max", "mean"],        "Parch": ["min", "max", "mean"]    })
The .describe() method comes in handy to speed things up:
df.describe()
It provides a quick overview of the numerical data in the table. You can also count the number of values/records by category with just one line of code:
df["Pclass"].value_counts()3    4911    2162    184Name: Pclass, dtype: int64
You can see more details in this section.
Say you had a dataset of names and ages:
data = {                "Name": ["Lucas", "Beatriz", "Clara", "Sara"],                "Age": [30, 26, 20, 22]               }df = pd.DataFrame(data)df
and you wanted to sort them by age:
df.sort_values(by="Age")
You can also do it in descending order:
df.sort_values(by="Age", ascending=False)
df1 = pd.DataFrame(    {        "Name": ["Lucas", "Beatriz", "Clara", "Sara"],         "Age": [30, 26, 20, 22]    })df2 = pd.DataFrame(     {         "Name": ["John", "Mary"],         "Age": [45, 38]     })df_names_ages = pd.concat([df1, df2], axis=0)df_names_ages
One can also merge two dataframes using a shared column.
df1 = pd.DataFrame(    {        "Name": ["Lucas", "Beatriz", "Clara", "Sara"],         "Age": [30, 26, 20, 22],        "City": ["Lisbon", "Barcelona", "London", "Paris"],    })df2 = pd.DataFrame(     {         "City": ["Lisbon", "Barcelona", "London", "Paris"],         "Occupation": ["Machine Learning Enginner", "Scientist", "Writer","Teacher"]     })df_merged = pd.merge(df1,df2,how="left", on="City")df_merged
You can learn more about concatenating and merging in pandas in this section.
import pandas as pdimport matplotlib.pyplot as plt
Say we have a dataset like this:
df = pd.read_csv("typingPerformance.csv")df
This is actually a dataset I created to track my touch typing performance to showing how fast I am typing on my keyboard.
One issue here is that the data type for the “date” column is a string. I would like it to be a datetime object so I could perform standard operations for dates on it. Pandas offers a simple way to convert columns to datetime objects:
df["date"] = pd.to_datetime(df["date"])df
Now, we can check the data type to make sure it is datetime:
df.dtypesdate        datetime64[ns]wpm                 objectaccuracy           float64dtype: object
Perfect! Now we can answer questions with simple expressions. For example, when did I start tracking my typing performance?
df["date"].min()Timestamp('2021-01-14 00:00:00')
When did I last practice?
df["date"].max()Timestamp('2021-10-01 00:00:00')
For how long have I been tracking?
df["date"].max() - df['date'].min()Timedelta('260 days 00:00:00')
I can now create a new column specifying something about the date like what month was it when that data point was created:
df["month"] = df["date"].dt.monthdf
Now, I can plot for instance the average wpm (words per minute) performance at each month:
df_monthly_performance = df.groupby("month")["wpm"].mean()df_monthly_performance.plot()plt.ylabel("wpm");
There is a ton of date-related functionalities in pandas and I recommend checking out this section of the documentation to learn more about it!
data = {"Full Name": ["Mark Harrison Junior", "Mary Elisabeth"],             "Age": [40, 50]}df = pd.DataFrame(data)df
we can use the .str() method to work with string columns. For example, we can make all the strings in a given column lower case:
df["Full Name"].str.lower()0    mark harrison junior1          mary elisabethName: Full Name, dtype: object
Or create another column based on text manipulation of a given column:
df["Last Name"] = df["Full Name"].str.split(" ").str.get(-1)df
We can check if string columns contain a certain string:
df["Full Name"].str.contains("Junior")0     True1    FalseName: Full Name, dtype: bool
You can learn more about text manipulation in this section.
Pandas is one of the most ubiquitous Python libraries for Data Scientists and Machine Learning engineers. It has amazing functionalities that if learned properly can be of major help when working with data and speed up your workflow.
Put in the time to learn pandas the right way, read the documentation, practice the individual functionalities that tend to reoccur in every project, and I guarantee the results will speak for themselves.
If you want to learn more about pandas, check out these two courses from Udemy:
The Ultimate Pandas Bootcamp
Data Analysis with Pandas
(These are affiliate links, if you use them I get a small commission, cheers! :))
If you liked this post, follow me on Medium, subscribe to my newsletter, or join Medium (if you haven’t already :)). Connect with me on Twitter, LinkedIn, and Instagram! Thanks and see you next time! :)
pandas documentation
pandas user guide