Pandas DataFrame Group by Consecutive Same Values | by Christopher Tao | Towards Data Science

It is very common that we want to segment a Pandas DataFrame by consecutive values. However, dealing with consecutive values is almost always not easy in any circumstances such as SQL, so does Pandas. Also, standard SQL provides a bunch of window functions to facilitate this kind of manipulation, but there are not too many window functions handy in Pandas. Fortunately, there are many workarounds in Python and sometimes make it even easier than classic window functions.
In this article, I’ll demonstrate how to group Pandas DataFrame by consecutive same values that repeat one or multiple times.
If you are still not quite sure what is the problem we’re trying to solve, don’t worry, you will understand by the sample data that is generated as follows:
df = pd.DataFrame({    'item':['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M'],     'v':[1, 2, 3, 3, 3, 4, 5, 5, 6, 7, 8, 8, 9]})
In the screenshot, the green lines split the groups that we expected.
In this example, it is actually very easy to group these values. Just simple groupby('v').
However, it is actually assuming that the value v must be monotonically increasing.
What if the value column is not monotonically increasing?
df = pd.DataFrame({    'item':['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N'],     'v':[1, 2, 3, 3, 3, 4, 5, 5, 6, 7, 8, 8, 9, 3]})
In this new example, we added the 13th row which has its value v == 3 again. If we simply groupby('v'), the 13th row will be put in the same group as the 2nd, 3rd and 4th rows, which is not what we want.
In other words, the 13th row should be in a separated group, because it is not consecutive.
The basic idea is to create such a column that can be grouped by. It must have the same values for the consecutive original values, but different values when the original value changes. We can use cumsum(). Here are the intuitive steps.
Create a new column shift down the original values by 1 rowCompare the shifted values with the original values. True if they are equal, otherwise FalseUsing cumsum() on the boolean column, the resulted column can be used for grouping by
Create a new column shift down the original values by 1 row
Compare the shifted values with the original values. True if they are equal, otherwise False
Using cumsum() on the boolean column, the resulted column can be used for grouping by
Here is a sample DataFrame created for intuition purposes only.
Why does it work? If you focus on each group:
The first boolean value of shifted_not_equal_to_original is True and the rest of them are False (if there are multiple rows in this group). So, the cumsum() function will make a difference between the first row of each group from the last row of its previous group.
The boolean value is True because the shifted value is different from the original value if the previous row value is different from the current. On the other hand, from the second row of this consecutive streak, it will be False because the value is equal to its precedent row.
I know the intuition looks complicated, but once you understand those, it is very easy to use this approach as follows.
df.groupby((df['v'].shift() != df['v']).cumsum())
Let’s verify the groups:
for k, v in df.groupby((df['v'].shift() != df['v']).cumsum()):    print(f'[group {k}]')    print(v)
Very importantly, the 13th row is grouped separately with the 2nd, 3rd and 4th row, which is exactly what we expected.
Therefore, this approach will be able to group the consecutive values regardless of whether the values are repeated later on or not, as long as it is not consecutive, another group will be created.
Indeed, I wouldn’t say that Pandas provides many useful window functions. However, Pandas in Python is a typical “programming language” not like SQL. So, it provides a lot more possibilities and sometimes can solve problems in even simpler ways.
medium.com
If you feel my articles are helpful, please consider joining Medium Membership to support me and thousands of other writers! (Click the link above)