Bypassing Pandas Memory Limitations | by Michael Beale | Towards Data Science

When you encounter a new set of data, you need to spend some time getting to know it. The process is not unlike meeting a new person for the first time. Instead of asking questions like “What’s your name?”, “Where are you from?”, and “Katy Perry or Taylor Swift?” you ask “What does your distribution look like?”, “Do you contain any missing or unexpected data?”, and “How do you correlate to other pieces of data?”.
A typical Data Scientist spends 80% of their day on preparing data for processing1. Tools like Pandas have made the process more efficient by adding a powerful set of features to explore and munge data. Pandas can turn a vanilla CSV file into insightful aggregations and charts. Plus Pandas’ number one feature is that it keeps me out of Excel.
Pandas is not all roses and sunshine however. Since the DataFrames (the foundation of Pandas) are kept in memory, there are limits to how much data can be processed at a time. Analyzing datasets the size of the New York Taxi data (1+ Billion rows and 10 years of information) can cause out of memory exceptions while trying to pack those rows into Pandas. Most Pandas related tutorials only work with 6 months of data to avoid that scenario.
Processing large amounts of data (too big to fit in memory) in Pandas requires one of the below approaches:
Break up the data into manageable pieces (Chunking).Use services outside of Pandas to handle filtering and aggregating of data.A combination of the above 2 methods.
Break up the data into manageable pieces (Chunking).
Use services outside of Pandas to handle filtering and aggregating of data.
A combination of the above 2 methods.
Apache projects Spark and Arrow were built to offer solutions to work with big data but I find installing and using databases to be a painless alternative to the complexity that Arrow and Spark come with. Recently I came across the open-source project QuestDB which claims high performance in the time-series database realm so I decided to give it a try.
QuestDB’s REST API accepts requests using a query language I’m already familiar with: SQL. The results can be paginated through if needed, but they can also be filtered to a more Pandas memory friendly size. Although performance analysis is not the primary goal of this post, with QuestDB’s SIMD instructions and high-performance design, there is also a boost in processing speed.
Incorporating a database like QuestDB in the data pipeline provides a few advantages over the common ETL approach (outlined in the image below):
A dynamic way to query the dataset.
A reduction in the complexity of filtering and aggregating.
Shortening the path from raw data to Pandas.
For all the below examples, I’m using a simple function to query QuestDB and convert those results into a Pandas DataFrame.
import pandas as pdimport requests defread_questdb(query, host, per_page = 10000, max_records = None):"""    Sends a SQL query to a specified host.        Parameters    ----------------    query : string      A SQL query    host : string      A host location in the form of a URL.  http://host:port    per_page : int      The limit of results to return per page.    max_records : int      The maximum amount of records to return (soft limit).    """data = []    offset = 1    page = 1    get_more = True    columns = None    params = {"query": query, "limit": f"{offset},{per_page * page}","count": "true"}    while get_more:        r = requests.get(f"{host}exec", params=params)        jsond = r.json()        if r.status_code != 200:            raise ValueError(jsond['error'])        if not columns:            columns = [x['name'] for x in jsond['columns']]        if jsond['dataset'] != []:            data = data + jsond['dataset']        if jsond["count"] < (per_page * page) or (max_records != None and (per_page*page) >= max_records):            get_more = False        else:            offset = (per_page * page) + 1            page += 1            params['limit'] = f"{offset},{per_page * page}"            params['nm'] = 'true'    return pd.DataFrame(data=data, columns=columns)
For the data, I have preloaded the New York Taxi dataset which has 1,636,055,518 rows of data from 2009 through 2019. The column definitions are available on the nyc.gov website.
There is more information about installing QuestDB as well as how to load data in the documentation.
Let’s start by analyzing fare_amount. Ingesting 1.6 billion rows into Pandas is not possible on my Macbook Pro so I need to use QuestDB to help me shrink the data in a more manageable size.
What is the average fare_amount?
query = "SELECT avg(fare_amount) FROM trips"avg_df = read_questdb(query, hostname)avg_df['avg'][0]>>>11.771664413356518
I’m also curious about how the fare_amount relates to passenger_count. Do larger groups tend to travel long distances? Below is the plot of the average fare by passenger count.
query = "SELECT passenger_count, avg(fare_amount) FROM trips"df = read_questdb(query, hostname)ax = df.plot.bar(x='passenger_count', y='avg', title='Average Fare by Passenger Count', label='Passenger Count')ax.legend(["Avg Fare Amount ($)"])#add a horizontal line for the averageax.axhline(y=avg_df['avg'][0], color='red')
The passenger_count of 0 looks out of place to me and although there may be a good reason for this, I’m going to remove that data from the DataFrame for now as well as sort and reindex the data.
#drop zero passenger count data and set index and sort valuesdf = df.drop([6]).set_index('passenger_count').sort_values('passenger_count')#change to integersdf.index = df.index.astype('int64')#don't need to specify x axisax = df.plot.bar(y='avg', title='Average Fare by Passenger Count', label='Passenger Count')ax.legend(["Avg Fare Amount ($)"])ax.axhline(y=avg_df['avg'][0], color='red')>>>avg1 11.5484582 12.1195143 11.8922984 12.0913685 11.4661986 12.2769637 31.622068
Now that the passenger counts are in ascending order and the 0 passenger count column has been dropped, I can see that the averages for the different passenger counts are in line with the dataset average with the exception of a passenger count of 7.
Since I’ve narrowed down my target dataset to the rows with a passenger_count of 7, I can bring the target rows into a Pandas DataFrame and use the `describe` function to get an overview of the fare_amount distribution.
query = "select * from trips where passenger_count = 7"df7 = read_questdb(query, hostname)df7['fare_amount'].describe()>>>count    1494.000000mean     31.622068std      31.708050min      0.01000025%      7.00000050%      12.00000075%      70.000000max      373.000000Name: fare_amount, dtype: float64
Three initial observations from that output:
The standard deviation is larger than the mean. Since there can’t be negative fare amounts, that must mean the data is heavily skewed.The min value is 0.01. I’m not a frequent user of Taxi Cabs but I’m pretty sure once the meter is turned on, it starts above $0.01.The max value is 373. Outliers could be applying pressure to the mean.
The standard deviation is larger than the mean. Since there can’t be negative fare amounts, that must mean the data is heavily skewed.
The min value is 0.01. I’m not a frequent user of Taxi Cabs but I’m pretty sure once the meter is turned on, it starts above $0.01.
The max value is 373. Outliers could be applying pressure to the mean.
I can plot the distribution of the data.
ax = df7['fare_amount'].plot.hist()
As expected the distribution is heavily skewed with a few larger outliers dragging the mean higher. But what does this tell us? Are all the other passenger_counts skewed similarly? Instead of mean, should I switch to using the median value? What other questions can be asked of this data?
Switching gears here, I’m interested in knowing how the average fare per hour driven has changed over time. I can use sampling in QuestDB to help organize fare_amount averages in the monthly buckets.
query = """SELECT pickup_datetime, avg(fare_amount) / (avg(cast((dropoff_datetime-pickup_datetime) as long)) / 1000000 / 60 / 60) fare_hr FROM trips SAMPLE BY 1M;""" fare_change_df = read_questdb(query,hostname)fare_change_df['pickup_datetime'] = pd.to_datetime(fare_change_df['pickup_datetime'])ax = fare_change_df[:125].plot(x='pickup_datetime',y='fare_hr')
This chart creates new questions. Why did the fare_amount spike in 2012? What caused a dip in 2015? The average hourly rate is on a steady decline; what factors would cause that?
There is still a long way to go before I can feel comfortable with the taxi data. I only touched the surface on a single column and there are infinite amounts of additional questions about the other columns and what relationships the columns have with each other. Each question answered leads to more questions.
Pandas is one of the more important libraries in the Python data science stack but as the data size grows, working completely within Pandas requires help from additional tools. I like to look to QuestDB to perform high volume filtering, aggregation, and grouping operations to extend the utility of Pandas.
[1] Gil Press, Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says (2016), Forbes