How to group data by time intervals in Python Pandas? | by Ankit Goel | Towards Data Science

If you have ever dealt with Time-Series data analysis, you would have come across these problems for sure —
Combining data into certain intervals like based on each day, a week, or a month.Aggregating data in the time interval like if you are dealing with price data then problems like total amount added in an hour, or a day.Finding patterns for other features in the dataset based on a time interval.
Combining data into certain intervals like based on each day, a week, or a month.
Aggregating data in the time interval like if you are dealing with price data then problems like total amount added in an hour, or a day.
Finding patterns for other features in the dataset based on a time interval.
In this article, you will learn about how you can solve these problems with just one-line of code using only 2 different Pandas API’s i.e. resample() and Grouper().
As we know, the best way to learn something is to start applying it. So, I am going to use a sample time-series dataset provided by World Bank Open data and is related to the crowd-sourced price data collected from 15 countries. For more details about the data, refer Crowdsourced Price Data Collection Pilot. For this exercise, we are going to use data collected for Argentina.
📚 Resources: Google Colab Implementation | Github Repository | Dataset 📚
This data is collected by different contributors who participated in the survey conducted by the World Bank in the year 2015. The basic idea of the survey was to collect prices for different goods and services in different countries. We are going to use only a few columns from the dataset for the demo purposes —
Pandas provides an API named as resample() which can be used to resample the data into different intervals. Let’s see a few examples of how we can use this —
Let’s say we need to find how much amount was added by a contributor in an hour, we can simply do so using —
# data re-sampled based on an hourdata.resample('H', on='created_at').price.sum()# outputcreated_at2015-12-14 18:00:00     5449.902015-12-14 19:00:00       15.982015-12-14 20:00:00       66.982015-12-14 21:00:00        0.002015-12-14 22:00:00        0.00
Here is what we are doing here —
First, we resampled the data into an hour ‘H’ frequency for our date column i.e. created_at. We can use different frequencies, I will go through a few of them in this article. Check out Pandas Time Frequencies for a complete list of frequencies. You can even go up to nanoseconds.After this, we selected the ‘price’ from the resampled data. Later we will see how we can aggregate on multiple fields i.e. total amount, quantity, and the unique number of items in a single command.Computed the sum for all the prices. This will give us the total amount added in that hour.
First, we resampled the data into an hour ‘H’ frequency for our date column i.e. created_at. We can use different frequencies, I will go through a few of them in this article. Check out Pandas Time Frequencies for a complete list of frequencies. You can even go up to nanoseconds.
After this, we selected the ‘price’ from the resampled data. Later we will see how we can aggregate on multiple fields i.e. total amount, quantity, and the unique number of items in a single command.
Computed the sum for all the prices. This will give us the total amount added in that hour.
By default, the time interval starts from the starting of the hour i.e. the 0th minute like 18:00, 19:00, and so on. We can change that to start from different minutes of the hour using offset attribute like —
# Starting at 15 minutes 10 seconds for each hourdata.resample('H', on='created_at', offset='15Min10s').price.sum()# Outputcreated_at2015-12-14 17:15:10     5370.002015-12-14 18:15:10       79.902015-12-14 19:15:10       64.562015-12-14 20:15:10       18.402015-12-14 21:15:10        0.00
Please note, you need to have Pandas version > 1.10 for the above command to work.
In this example, we will see how we can resample the data based on each week.
# data re-sampled based on an each week, just change the frequencydata.resample('W', on='created_at').price.sum()# outputcreated_at2015-12-20    4.305638e+042015-12-27    6.733851e+042016-01-03    4.443459e+042016-01-10    1.822236e+042016-01-17    1.908385e+05
By default, the week starts from Sunday, we can change that to start from different days i.e. let’s say if we would like to combine based on the week starting on Monday, we can do so using —
# data re-sampled based on an each week, week starting Mondaydata.resample('W-MON', on='created_at').price.sum()# outputcreated_at2015-12-14    5.532860e+032015-12-21    3.850762e+042015-12-28    6.686329e+042016-01-04    5.392410e+042016-01-11    1.260869e+04
This is similar to what we have done in the examples before.
# data re-sampled based on each monthdata.resample('M', on='created_at').price.sum()# Outputcreated_at2015-12-31    1.538769e+052016-01-31    4.297143e+052016-02-29    9.352684e+052016-03-31    7.425185e+062016-04-30    1.384351e+07
One observation to note here is that the output labels for each month are based on the last day of the month, we can use the ‘MS’ frequency to start it from 1st day of the month i.e. instead of 2015–12–31 it would be 2015–12–01 —
# month frequency from start of the monthdata.resample('MS', on='created_at').price.sum()created_at2015-12-01    1.538769e+052016-01-01    4.297143e+052016-02-01    9.352684e+052016-03-01    7.425185e+062016-04-01    1.384351e+07
Often we need to apply different aggregations on different columns like in our example we might need to find —
Unique items that were added in each hour.The total quantity that was added in each hour.The total amount that was added in each hour.
Unique items that were added in each hour.
The total quantity that was added in each hour.
The total amount that was added in each hour.
We can do so in a one-line by using agg() on the resampled data. Let’s see how we can do it —
# aggregating multiple fields for each hourdata.resample('H', on='created_at').agg({'price':'sum', 'quantity':'sum','item_code':'nunique'})
In the above examples, we re-sampled the data and applied aggregations on it. What if we would like to group data by other fields in addition to time-interval? Pandas provide an API known as grouper() which can help us to do that.
In this section, we will see how we can group data on different fields and analyze them for different intervals.
Let’s say we need to analyze data based on store type for each month, we can do so using —
# Grouping data based on month and store typedata.groupby([pd.Grouper(key='created_at', freq='M'), 'store_type']).price.sum().head(15)# Outputcreated_at  store_type                2015-12-31  other                          34300.00            public_semi_public_service       833.90            small_medium_shop               2484.23            specialized_shop              107086.002016-01-31  market                           473.75            other                         314741.00            private_service_provider         325.00            public_semi_public_service       276.79            small_medium_shop              31042.79            specialized_shop               29648.442016-02-29  market                          1974.04            other                         527950.00            private_service_provider        1620.00            public_semi_public_service      1028.52            small_medium_shop             224653.83
Let’s understand how I did it —
First, we passed the Grouper object as part of the groupby statement which groups the data based on month i.e. ‘M’ frequency. This is similar to resample(), so whatever we discussed above applies here as well.We added store_type to the groupby so that for each month we can see different store types.For each group, we selected the price, calculated the sum, and selected the top 15 rows.
First, we passed the Grouper object as part of the groupby statement which groups the data based on month i.e. ‘M’ frequency. This is similar to resample(), so whatever we discussed above applies here as well.
We added store_type to the groupby so that for each month we can see different store types.
For each group, we selected the price, calculated the sum, and selected the top 15 rows.
As we did in the last example, we can do a similar thing for item_name as well.
# Grouping data based on each month and item_namedata.groupby([pd.Grouper(key='created_at', freq='M'), 'item_name']).price.sum()# Outputcreated_at  item_name                                   2015-12-31  Bar soap, solid, SB                                33.17            Beer, domestic brand, single bottle, WKB           29.79            Black tea, BL                                      12.00            Black tea, in bags, WKB                            60.99            Bread, white, sliced, WKB                          85.45                                                              ...   2016-08-31  Wheat flour, not self-rising, BL                  150.38            White sugar, WKB                                  266.47            Women's haircut, basic hairdresser               7730.00            Wrist-watch, men's, CITIZEN Eco-Drive BM6060    52205.00            Yoghurt, plain, WKB                               150.96
We can apply aggregation on multiple fields similarly the way we did using resample(). The only thing which is different here is that the data would be grouped by store_type as well and also, we can do NamedAggregation (assign a name to each aggregation) on groupby object which doesn’t work for re-sample.
# grouping data and named aggregation on item_code, quantity, and pricedata.groupby([pd.Grouper(key='created_at', freq='M'), 'store_type']).agg(unique_items=('item_code', 'nunique'),         total_quantity=('quantity','sum'),         total_amount=('price','sum'))
I hope this article will help you to save time in analyzing time-series data. I recommend you to check out the documentation for the resample() and grouper() API to know about other things you can do with them.
If you would like to learn about other Pandas API’s which can help you with data analysis tasks then do checkout the article Pandas: Put Away Novice Data Analyst Status where I explained different things that you can do with Pandas.
Let me know in the comments or ping me on LinkedIn if you are facing any problems with using Pandas or Data Analysis in general. We can try to solve them together. That’s all for now, see you in the next article.
Cheers!!! Stay Safe!!! Keep Learning!!!