How to Reduce the Size of a Pandas Dataframe in Python | by Nicklas Ankarstad | Towards Data Science

Whether you are building a model for a hobby project or for work purposes, chances are that your first attempt will include opening up a jupyter notebook and reading in some data. Eventually you will undoubtly run into the memory issues with your notebook. Before you start dropping rows or try complex sampling techniques to reduce the size of your data, you should check the structure of the data.
To explore how we can reduce the size of a dataset, we need some sample data. For this tutorial, I am using an old dataset I created for the Future Sales Prediction contest on Kaggle (here is my write-up on pre-processing the data). Once fully joined and feature engineered, the dataset has 58 columns and 11,128,050 records. That’s a lot of data to fit into a small laptop. We need a solution to reduce the size of the data.
Before we begin, we should check learn a bit more about the data. One function that is very helpful to use is df.info() from the pandas library.
df.info(memory_usage = "deep")
This code snippit returns the below output:
<class 'pandas.core.frame.DataFrame'>Int64Index: 11128050 entries, 0 to 11128049Data columns (total 58 columns): #   Column                                            Dtype         ---  ------                                            -----          0   date_block_num                                    int64          1   item_id                                           int64          2   shop_id                                           int64          3   target                                            float64        4   item_name                                         object         5   item_category_id                                  int64          6   Category_type                                     object         7   City                                              object         8   Month_End_Date                                    datetime64[ns] 9   Number_of_Mondays                                 int64          10  Number_of_Tuesdays                                int64          11  Number_of_Wednesdays                              int64          12  Number_of_Thursdays                               int64          13  Number_of_Fridays                                 int64          14  Number_of_Saturdays                               int64          15  Number_of_Sundays                                 int64          16  Year                                              int64          17  Month                                             int64          18  Days_in_Month                                     int64          19  min_item_sale_date_block_num                      int64          20  Months_Since_Item_First_Sold                      int64          21  avg_first_months_sales_by_item_category_id        float64        22  avg_first_months_sales_by_item_category_and_shop  float64        23  target_lag_1                                      float64        24  target_lag_2                                      float64        25  target_lag_3                                      float64        26  target_lag_4                                      float64        27  target_lag_5                                      float64        28  target_lag_6                                      float64        29  target_lag_12                                     float64        30  avg_monthly_by_item_lag_1                         float64        31  avg_monthly_by_item_lag_2                         float64        32  avg_monthly_by_item_lag_3                         float64        33  avg_monthly_by_item_lag_6                         float64        34  avg_monthly_by_item_lag_12                        float64        35  avg_monthly_by_shop_lag_1                         float64        36  avg_monthly_by_shop_lag_2                         float64        37  avg_monthly_by_shop_lag_3                         float64        38  avg_monthly_by_shop_lag_6                         float64        39  avg_monthly_by_shop_lag_12                        float64        40  avg_monthly_by_category_lag_1                     float64        41  avg_monthly_by_category_lag_2                     float64        42  avg_monthly_by_category_lag_3                     float64        43  avg_monthly_by_category_lag_6                     float64        44  avg_monthly_by_category_lag_12                    float64        45  avg_monthly_by_city_lag_1                         float64        46  avg_monthly_by_city_lag_2                         float64        47  avg_monthly_by_city_lag_3                         float64        48  avg_monthly_by_city_lag_6                         float64        49  avg_monthly_by_city_lag_12                        float64        50  item_price_lag_1                                  float64        51  item_price_lag_2                                  float64        52  item_price_lag_3                                  float64        53  item_price_lag_4                                  float64        54  item_price_lag_5                                  float64        55  item_price_lag_6                                  float64        56  target_3_month_avg                                float64        57  target_6_month_avg                                float64       dtypes: datetime64[ns](1), float64(38), int64(16), object(3)memory usage: 7.6 GB
With the df.info we get the following information:
The number of rows or entries
The number of columns
The index and name of each column
The data type of each column
Total memory usage of the data frame
How many columns belong to each data type
Looking closely at this table, we see numbers behind each data type illustrating the bits they use.
Since a lot of these are listed as either int64 or float64, we can probably reduce them down to smaller space datatypes like int16 or float8. Downcasting means we reduce the datatypes of each feature to its lowest possible type.
## downcasting loopfor column in df: if df[column].dtype == ‘float64’: df[column]=pd.to_numeric(df[column], downcast=’float’) if df[column].dtype == ‘int64’: df[column]=pd.to_numeric(all_data[column], downcast=’integer’)## dropping an unused columndf = df.drop('item_name',axis =1)
We can check the size of the dataframe again.
<class 'pandas.core.frame.DataFrame'>Int64Index: 11128050 entries, 0 to 11128049Data columns (total 57 columns): #   Column                                            Dtype         ---  ------                                            -----          0   date_block_num                                    int8           1   item_id                                           int16          2   shop_id                                           int8           3   target                                            float32        4   item_category_id                                  int8           5   Category_type                                     object         6   City                                              object         7   Month_End_Date                                    datetime64[ns] 8   Number_of_Mondays                                 int8           9   Number_of_Tuesdays                                int8           10  Number_of_Wednesdays                              int8           11  Number_of_Thursdays                               int8           12  Number_of_Fridays                                 int8           13  Number_of_Saturdays                               int8           14  Number_of_Sundays                                 int8           15  Year                                              int16          16  Month                                             int8           17  Days_in_Month                                     int8           18  min_item_sale_date_block_num                      int8           19  Months_Since_Item_First_Sold                      int8           20  avg_first_months_sales_by_item_category_id        float32        21  avg_first_months_sales_by_item_category_and_shop  float32        22  target_lag_1                                      float32        23  target_lag_2                                      float32        24  target_lag_3                                      float32        25  target_lag_4                                      float32        26  target_lag_5                                      float32        27  target_lag_6                                      float32        28  target_lag_12                                     float32        29  avg_monthly_by_item_lag_1                         float32        30  avg_monthly_by_item_lag_2                         float32        31  avg_monthly_by_item_lag_3                         float32        32  avg_monthly_by_item_lag_6                         float32        33  avg_monthly_by_item_lag_12                        float32        34  avg_monthly_by_shop_lag_1                         float32        35  avg_monthly_by_shop_lag_2                         float32        36  avg_monthly_by_shop_lag_3                         float32        37  avg_monthly_by_shop_lag_6                         float32        38  avg_monthly_by_shop_lag_12                        float32        39  avg_monthly_by_category_lag_1                     float32        40  avg_monthly_by_category_lag_2                     float32        41  avg_monthly_by_category_lag_3                     float32        42  avg_monthly_by_category_lag_6                     float32        43  avg_monthly_by_category_lag_12                    float32        44  avg_monthly_by_city_lag_1                         float32        45  avg_monthly_by_city_lag_2                         float32        46  avg_monthly_by_city_lag_3                         float32        47  avg_monthly_by_city_lag_6                         float32        48  avg_monthly_by_city_lag_12                        float32        49  item_price_lag_1                                  float32        50  item_price_lag_2                                  float32        51  item_price_lag_3                                  float32        52  item_price_lag_4                                  float32        53  item_price_lag_5                                  float32        54  item_price_lag_6                                  float32        55  target_3_month_avg                                float32        56  target_6_month_avg                                float32       dtypes: datetime64[ns](1), float32(38), int16(2), int8(14), object(2)memory usage: 3.2 GB
We see that we now have several 8, 16 and 32 bit columns and we have effectively cut the size of our dataframe in half by simply changing the data types.
What’s even more interesting is that the memory usage of the dataframe is down to 3.2gb. This is roughly half of the 7.6gb we had when we first started. With this type of memory reduction, we can load more data into smaller machines, which ultimately reduces costs.
In this tutorial, we loaded 11 million record dataset into a pandas dataframe. We learned how to check the size and structure of the data by using the .info() function within pandas. This gave us useful information like the number of rows and columns, the size memory usage of the dataframe and the data type of each column.
We learned about downcasting, which we used to cut the size of our dataframe in half allowing us to use less memory and do more with our data.