Pandas DataFrame Group by Consecutive Certain Values | by Christopher Tao | Towards Data Science

It is very common that we want to segment a Pandas DataFrame by consecutive values. However, dealing with consecutive values is almost always not easy in any circumstances such as SQL, so does Pandas.
Standard SQL provides a bunch of window functions to facilitate this kind of manipulation, but there are not too many window functions handy in Pandas. Fortunately, there are many workarounds in Python and sometimes make it even easier than classic window functions.
In this article, I’ll demonstrate how to group Pandas DataFrame by certain values that appear arbitrary times.
If the title is sort of confused to you, the following sample data will make it clear. Let’s create the sample DataFrame first.
df = pd.DataFrame({  'item':['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M'],   'value':[12, 3, 5, 0, 0, 0, 23, 4, 2, 0, 0, 12, 0]})
In this example, we want to segment the DataFrame for whatever number of consecutive values 0. The red square indicates that three segments are expected to be separated from this original DataFrame.
What we are trying to do here is generally two:
Filter the DataFrame with rows having value == 0.Separate these rows if they are not consecutive. In other words, there is at least one row with value != 0.
Filter the DataFrame with rows having value == 0.
Separate these rows if they are not consecutive. In other words, there is at least one row with value != 0.
The first step is very easy, but apparently not the second. Let’s have the intuitive steps before coding the solution.
Create a “mask” series with all boolean values. True if the value == 0, otherwise False.Filter the DataFrame using the mask series. So, we have all the rows with value == 0.Inverse the mask series, so True become False and vice versa.Perform a cumulative sum on the inversed mask series.The cumulative sum series can be used to group by and achieve what we want.
Create a “mask” series with all boolean values. True if the value == 0, otherwise False.
Filter the DataFrame using the mask series. So, we have all the rows with value == 0.
Inverse the mask series, so True become False and vice versa.
Perform a cumulative sum on the inversed mask series.
The cumulative sum series can be used to group by and achieve what we want.
It is important to clarify that if we cum boolean values in Python, True will be treated as 1, whereas False will be treated as 0.
I know, it might still be confusing. Let’s have a look at the following DataFrame which is created for this intuition only, not as part of the solution though.
# This is not part of the solution, just for demo purposesdf_tmp = dfdf_tmp['mask'] = maskdf_tmp['inversed_mask'] = ~maskdf_tmp['inversed_mask_cumsum'] = (~mask).cumsum()
The green rectangles show the expected groups of rows that we are expected to obtain.
So, why do we need to perform a cumsum() on the inversed mask column here?
The inversed mask will have True (1) for any non-zero values, and False (0) for any zeros.When we apply cumsum() on this inversed mask column, the result for the consecutive 0 will remain the same, while if it is not 0, the cumulative sum will make the value increased.
The inversed mask will have True (1) for any non-zero values, and False (0) for any zeros.
When we apply cumsum() on this inversed mask column, the result for the consecutive 0 will remain the same, while if it is not 0, the cumulative sum will make the value increased.
After the intuition, you might think this is not easy. However, when you see the solution, you may laugh at how easy it is.
df[df['value'] == 0].groupby((df['value'] != 0).cumsum())
Yes, that’s it.
We can verify whether the groups are correct or not.
for k, v in df[df['value'] == 0].groupby((df['value'] != 0).cumsum()):    print(f'[group {k}]')    print(v)    print('\n')
That’s exactly what we expected :)
Indeed, I wouldn’t say that Pandas provides many useful window functions. However, Pandas in Python is a typical “programming language” not like SQL. So, it provides a lot more possibilities and sometimes can solve problems in even simpler ways.
medium.com
If you feel my articles are helpful, please consider joining Medium Membership to support me and thousands of other writers! (Click the link above)