11 Pandas Built-in Functions You Should Know | by Roman Orac | Towards Data Science

I’ve been using pandas for a few years and each time I feel I am typing too much, I google the operation and I usually find a shorter way of doing it — a new pandas trick!
I learned about these functions recently and I deem them essential because of ease of use.
By reading this article, you’ll learn:
How to retrieve a column value from a DataframeHow to change a column value in a DataframeThe proper way of adding a new column to a DataFrameHow to retrieve a Series or a DataFrameHow to create a DataFrame from a pandas SeriesFilter with between functionFix the order of the rows with reindex functionAdvanced usages of Describe functionText search with regexBigger than memory datasets with pandasHow to save the disk space
How to retrieve a column value from a Dataframe
How to change a column value in a Dataframe
The proper way of adding a new column to a DataFrame
How to retrieve a Series or a DataFrame
How to create a DataFrame from a pandas Series
Filter with between function
Fix the order of the rows with reindex function
Advanced usages of Describe function
Text search with regex
Bigger than memory datasets with pandas
How to save the disk space
Let’s create a simple DataFrame to show these tips in action:
import pandas as pddf = pd.DataFrame({"col1": ["A0", "A1", "A0", "A2", "A0", "A1"]})
There are two ways of retrieving column values from a DataFrame:
based on integer-location indexing (iloc),
based on the label indexing (loc).
To retrieve the first column value for the col1 column (integer-location indexing):
df.iloc[0].col1# output: 'A0'
To retrieve the last column value:
df.iloc[-1].col1# output: ‘A1’
You can also retrieve a column value with the label. By default, pandas DataFrame has indices from 0 to n, where n is the number of rows in a DataFrame.
df.loc[1, “col1”]# output: ‘A1’
See pandas documentation to learn more about loc and iloc functions.
Similar to the example above, you can change the value of a column with loc and iloc functions. Which one you use depends if you have the integer position of the row (iloc) or label of the row (loc).
To change the first column value based on integer position:
df.iloc[0].col1 = “First”
To change the last column value based on integer position:
df.iloc[-1].col1 = “Last”
To change the column value by label:
df.loc[2, “col1”] = “Index”
By applying these changes, we get the following Dataframe:
Let’s say we would like to add a list of integers as a column to our DataFrame.
new_column = list(range(len(df))new_column# output: [0, 1, 2, 3, 4, 5]
How NOT to add a column (it doesn’t work):
df.new_column1 = new_column
Also NOT a proper way of adding a column, but it works. You might get a copy warning.
df["new_column2"] = new_column
A good way to add a new column:
df.loc[:, ‘new_column3’] = new_column
The proper way:
df = df.assign(new_column4=new_column)
assign function also takes an index as an argument with which we can define the order of rows in the column.
You might be tempted to add the inplace=True argument to the assign function to add a new column in place, but it won’t work. The assign function will also add the in-place column.
# This will add two columnsdf.assign(new_column4=new_column, inplace=True)
This is actually another benefit of the assign function. You can add multiple columns at the same time.
See pandas documentation, to learn more about assign function.
When using square brackets with pandas DataFrame, it returns:
a Series if you use single square brackets
a Dataframe if you use double square brackets.
Let’s look at an example.
How to retrieve a Series from a Dataframe?
df["col1"]
type(df["col1"])# pandas.core.series.Series
How to retrieve a DataFrame (with a subset of columns) from a Dataframe?
df[["col1"]]
type(df[["col1"]])# pandas.core.frame.DataFrame
When performing a grouping with aggregation, pandas returns a Series. What is the simplest way to convert a Series back to a Dataframe?
Let’s look at an example:
df = pd.DataFrame(    {        "col1": ["A0", "A1", "A0", "A2", "A0", "A1"],        "col2": [1, 2, 3, 4, 5, 6],    })df_group = df.groupby("col1").col2.sum()df_group
df_group is a Series.
Series has a to_frame function, which can covert a Series back to DataFrame:
df_group.to_frame()
See pandas documentation, to learn more about to_frame function.
I’ve been using “between” function in SQL for years, but I only discovered it recently in pandas.
Let’s say we have a DataFrame with prices and we would like to filter prices between 2 and 4.
df = pd.DataFrame({'price': [1.99, 3, 5, 0.5, 3.5, 5.5, 3.9]})
With between function, you can reduce this filter:
df[(df.price >= 2) & (df.price <= 4)]
To this:
df[df.price.between(2, 4)]
It might not seem much, but those parentheses are annoying when writing many filters. The filter with between function is also more readable.
between function sets interval left <= series <= right.
See pandas, documentation to learn more about between function.
The reindex function conforms a Series or a DataFrame to a new index. I resort to the reindex function when making reports with columns that have a predefined order.
Let’s add sizes of T-shirts to our Dataframe. The goal of the analysis is to calculate the mean price for each size:
df = pd.DataFrame({'price': [1.99, 3, 5], 'size': ['medium', 'large', 'small']})df_avg = df.groupby('size').price.mean()df_avg
Sizes have a random order in the table above. It should be ordered: small, medium, large. As sizes are strings we cannot use the sort_values function. Here comes reindex function to the rescue:
df_avg.reindex(['small', 'medium', 'large'])
By specifying the order of sizes in the reindex function makes a result table easier to interpret.
See pandas, documentation to learn more about reindex function.
Describe function is an essential tool when working on Exploratory Data Analysis. It shows basic summary statistics for all columns in a DataFrame.
df.price.describe()
What if we would like to calculate 10 quantiles instead of 3?
df.price.describe(percentiles=np.arange(0, 1, 0.1))
Describe function takes percentiles argument. We can specify the number of percentiles with NumPy’s arange function to avoid typing each percentile by hand.
This feature becomes really useful when combined with the group by function:
df.groupby('size').describe(percentiles=np.arange(0, 1, 0.1))
See pandas, documentation to learn more about describe function.
Our T-shirt dataset has 3 sizes. Let’s say we would like to filter small and medium sizes. A cumbersome way of filtering is:
df[(df['size'] == 'small') | (df['size'] == 'medium')]
This is bad because we usually combine it with other filters, which makes the expression unreadable. Is there a better way?
pandas string columns have an “str” accessor, which implements many functions that simplify manipulating string. One of them is “contains” function, which supports search with regular expressions.
df[df['size'].str.contains('small|medium')]
The filter with “contains” function is more readable, easier to extend and combine with other filters.
See pandas, documentation to learn more about contains function.
pandas cannot even read bigger than the main memory datasets. It throws a MemoryError or Jupyter Kernel crashes. But to process a big dataset you don’t need Dask or Vaex. You just need some ingenuity. Sounds too good to be true?
In case you’ve missed my article about Dask and Vaex with bigger than main memory datasets:
towardsdatascience.com
When doing an analysis you usually don’t need all rows or all columns in the dataset.
In a case, you don’t need all rows, you can read the dataset in chunks and filter unnecessary rows to reduce the memory usage:
iter_csv = pd.read_csv('dataset.csv', iterator=True, chunksize=1000)df = pd.concat([chunk[chunk['field'] > constant] for chunk in iter_csv])
Reading a dataset in chunks is slower than reading it all once. I would recommend using this approach only with bigger than memory datasets.
In a case, you don’t need all columns, you can specify required columns with “usecols” argument when reading a dataset:
df = pd.read_csv('file.csv', usecols=['col1', 'col2'])
The great thing about these two approaches is that you can combine them.
When working on multiple Data Science projects, you usually end up with many preprocessed datasets from different experiments. The small SSD on a laptop can get cluttered quickly.
Pandas enables us to compress the dataset when saving it and then reading back in compressed format.
Let’s create a big pandas DataFrame with random numbers.
import numpy as npdf = pd.DataFrame(np.random.randn(50000,300))
When we save this file as CSV, it takes almost 300 MB on the hard drive.
df.to_csv('random_data.csv', index=False)
We can reduce the file size to 136 MB by using gz extension instead of csv.
df.to_csv('random_data.gz', index=False)
We don’t lose any functionality with this as it is also easy to read the gzipped data to the DataFrame.
df = pd.read_csv('random_data.gz')
In this article, I’ve listed 11 pandas functions that I use daily to wrangle data and perform the analysis.
I’m sure you’re already met a few of them, but maybe you were using those functions in the wrong way. A good example is setting a new column into a DataFrame.
Pandas has multiple ways of achieving the same goal — some are more correct than others. Using the pandas-preferred approach will bring you fewer problems in the long run.
It’s always good to be on the safe side
- Correlation Analysis 101 in Python [Article]- 50% Off All AI Nanodegree Programs! [Course]- Data Science for Business Leaders [Course]- Free skill tests for Data Scientists & Machine Learning Engineers
Some of the links above are affiliate links and if you go through them to make a purchase I’ll earn a commission. Keep in mind that I link courses because of their quality and not because of the commission I receive from your purchases.
Follow me on Twitter, where I regularly tweet about Data Science and Machine Learning.