Question: I have a pandas dataframe, for example df_dupl = pd.DataFrame({ 'EVENT_TIME': ['00:01', '00:01', '00:01', '00:03', '00:03', '00:03', '00:06', '00:06', '00:06', '00:08', '00:08', '00:10', '00:10', '00:11', '00:11', '00:13', '00:13', '00:13'], 'UNIQUE_ID': [123, 123, 123, 125, 125, 125, 123, 123, 123, 127, 127, 123, 123, 123, 123, 123, 123, 123], 'Value1': ['A', 'B', 'A', 'A', 'B', 'A', 'A', 'B', 'A', 'A', 'B', 'A', 'B', 'C', 'B', 'A', 'B', 'A'], 'Value2': [0.3, 0.2, 0.2, 0.1, 1.3, 0.2, 0.3, 0.2, 0.2, 0.1, 1.3, 0.3, 0.2, 0.3, 0.2, 0.3, 0.2, 0.2] })  I want to remove the sequences of rows that have the same values as the previous (by EVENT_TIME) rows with the same UNIQUE_ID. For the example the result should look like this: df = pd.DataFrame({ 'EVENT_TIME': ['00:01', '00:01', '00:01', '00:03', '00:03', '00:03', '00:08', '00:08', '00:10', '00:10', '00:11', '00:11', '00:13', '00:13', '00:13'], 'UNIQUE_ID': [123, 123, 123, 125, 125, 125, 127, 127, 123, 123, 123, 123, 123, 123, 123], 'Value1': ['A', 'B', 'A', 'A', 'B', 'A', 'A', 'B', 'A', 'B', 'C', 'B', 'A', 'B', 'A'], 'Value2': [0.3, 0.2, 0.2, 0.1, 1.3, 0.2, 0.1, 1.3, 0.3, 0.2, 0.3, 0.2, 0.3, 0.2, 0.2] }).  The rows with time 00:06 should be removed, because the previous sub-dataframe with UNIQUE_ID 123 (time 00:01) is identical. On the other hand the rows with time 00:13 should remain - they are also identical to the rows with Time 00:01, but there are other rows with UNIQUE_ID 123 in between. The key thinq is that I want to compare the whole sub-dataframes, not single rows. I can achieve the desired result by using the folllowing function, but it is quite slow. def del_dupl_gr(df): out = [] for x in df['UNIQUE_ID'].unique(): prev_df = pd.DataFrame() for y in df[df['UNIQUE_ID'] == x]['EVENT_TIME'].unique(): test_df = df[(df['UNIQUE_ID'] == x) & (df['EVENT_TIME'] == y)] if not test_df.iloc[:, 2:].reset_index(drop=True).equals(prev_df.iloc[:, 2:].reset_index(drop=True)): out.append(test_df) prev_df = test_df return pd.concat(out).sort_index().reset_index(drop=True)  The real dataframe is quite large (over million rows) and this looping takes a lot of time. I am sure there must be proper (or at least faster) way to do this. Results Thanks for all the submitted answers. I compared their speed. In some cases i slightly edited the methods to produce exactly the same results. So in all sort_values methods I added kind='stable' to ensure that the order is preserved, and at the end I added .reset_index(drop=True).    Method 1000 rows 10 000 rows 100 000 rows     original 556 ms 5.41 s Not tested   mozway 1.24 s 10.1 s Not tested   Andrej Kesely 696 ms 4.56 s Not tested   Quang Hoang 11.3 ms 34.1 ms 318 ms    EDIT The accepted answer works fine if there are no NaN values in the dataframe. In my case (though not mentioned in the original question), there are NaN values which I need to be treated as equal. So the comparison part of the code needs to be modified as follows (the other parts remain the same) dup = ((df_dupl.groupby(['UNIQUE_ID',enums])[value_cols].shift().eq(df_dupl[value_cols]) | (df_dupl.groupby(['UNIQUE_ID',enums])[value_cols].shift().isna() & df_dupl[value_cols].isna())).all(axis=1).groupby(groups).transform('all') & sizes.groupby([df_dupl['UNIQUE_ID'],enums]).diff().eq(0))  This makes the method about 50% slower, but it remains by far the fastest of all the solutions.

Answer: Another approach is to shift the rows by enumeration, then compare: # the value columns value_cols = df.columns[2:] # groups are identified as `EVENT_TIME` and `UNIQUE_ID` groupby = df_dupl.groupby(['EVENT_TIME','UNIQUE_ID'])['Value1'] # these are the groups groups = groupby.ngroup() # enumeration within the groups enums = groupby.cumcount() # sizes of the groups - populated across the rows sizes = groupby.transform('size') dup = (df_dupl.groupby(['UNIQUE_ID',enums])[value_cols].shift(). # shift by enumeration within `UNIQUE_ID` .eq(df_dupl[value_cols]).all(axis=1) # equal the current rows .groupby(groups).transform('all') # identical across the groups & sizes.groupby([df_dupl['UNIQUE_ID'],enums]).diff().eq(0). # and the group size are equal too ) # output df_dupl.loc[~dup]  Output:  EVENT_TIME UNIQUE_ID Value1 Value2 0 00:01 123 A 0.3 1 00:01 123 B 0.2 2 00:01 123 A 0.2 3 00:03 125 A 0.1 4 00:03 125 B 1.3 5 00:03 125 A 0.2 9 00:08 127 A 0.1 10 00:08 127 B 1.3 11 00:10 123 A 0.3 12 00:10 123 B 0.2 13 00:11 123 C 0.3 14 00:11 123 B 0.2 15 00:13 123 A 0.3 16 00:13 123 B 0.2 17 00:13 123 A 0.2