Question: I have one polars dataframe and I am trying to find the differences (fields that have their values changed) on multiple columns between groups on one key. There can be multiple groups in the dataframe and more than one column. The groups is essentially a datetime in int format (YYYYMMDD) How can I find the rows where there is a new value of (any) column on a date? Sample Data: raw_df = pl.DataFrame([ {'id': 'AAPL','update_time': 20241112,'status':'trading', 'underlying': 'y'}, {'id': 'MSFT','update_time': 20241113,'status': 'trading', 'underlying': 'x'}, {'id': 'NVDA','update_time': 20241112,'status': 'trading', 'underlying': 'z'}, {'id': 'MSFT','update_time': 20241112,'status': 'pending','underlying': 'x'}, {'id': 'AAPL','update_time': 20241113,'status': 'trading', 'underlying': 'y'}, {'id': 'NVDA','update_time': 20241113,'status': 'trading', 'underlying': 'z'}, {'id': 'TSLA','update_time': 20241112,'status': 'closed', 'underlying': 'v'}, ] ) expected_df = pl.DataFrame([ {'id': 'MSFT','update_time': 20241112,'status':'pending', 'underlying': 'x'}, {'id': 'MSFT','update_time': 20241113,'status': 'trading', 'underlying': 'x'}, ] )  Below is what the data input looks like. shape: (7, 4) ┌──────┬─────────────┬─────────┬────────────┐ │ id ┆ update_time ┆ status ┆ underlying │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str ┆ str │ ╞══════╪═════════════╪═════════╪════════════╡ │ AAPL ┆ 20241112 ┆ trading ┆ y │ │ MSFT ┆ 20241113 ┆ trading ┆ x │ │ NVDA ┆ 20241112 ┆ trading ┆ z │ │ MSFT ┆ 20241112 ┆ pending ┆ x │ │ AAPL ┆ 20241113 ┆ trading ┆ y │ │ NVDA ┆ 20241113 ┆ trading ┆ z │ │ TSLA ┆ 20241112 ┆ closed ┆ v │ └──────┴─────────────┴─────────┴────────────┘  And the expected result below, showing the id the update time and the field that has changed. If there are more than 1 field changed/updated, ideally it should go into a new row. I am trying to find the fields that have changed, grouped by 'update_time' on the key 'id'. One caveat is that there can be ids' that are present in a group but not in another group for example 'TSLA'. Hence, these ids that are not common or the intersection between groups can be ignored. Since only MSFT has the status changed, it should be filtered to those two rows where it has been updated. The field changes should only be done on all other columns except update_time which is what we are using to group by. shape: (2, 3) ┌──────┬─────────────┬─────────┐ │ id ┆ update_time ┆ status │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞══════╪═════════════╪═════════╡ │ MSFT ┆ 20241112 ┆ pending │ │ MSFT ┆ 20241113 ┆ trading │ └──────┴─────────────┴─────────┴  Could not figure out how to do this, but this is the closest i have, which could not work on the caveat mentioned earlier. def find_updated_field_differences(df): columns_to_check = [col for col in df.columns if col != 'id' and col != 'update_time'] sorted_df = df.sort('update_time') grouped_df = sorted_df.groupby(["update_time"]) result_data = [] for group_key, group_df in grouped_df: print(group_df) for col in columns_to_check: group_df = group_df.with_columns( (pl.col(col) != pl.col(col).shift()).alias(f"{col}_changed") ) differing_rows = group_df.filter( pl.any([pl.col(f"{col}_changed") for col in columns_to_check]) ) result_data.append(differing_rows) differing_df = pl.concat(result_data) differing_df = differing_df.sort("id") return differing_df 

Answer: Your approach is slightly over-complicating things. What I suggest is that you first sort the data by id and update_time, then shift the data to prepare for comaparison. After that, you can identify the rows where id is the same but where there is a difference: import polars as pl def find_updated_field_differences(df): sorted_df = df.sort(['id', 'update_time']) shifted_df = sorted_df.shift(-1) mask = ( (sorted_df['id'] == shifted_df['id']) & ((sorted_df['status'] != shifted_df['status']) | (sorted_df['underlying'] != shifted_df['underlying'])) ) start_changes = sorted_df.filter(mask) end_changes = sorted_df.shift(-1).filter(mask) differing_df = pl.concat([start_changes, end_changes]).unique() return differing_df.sort(['id', 'update_time']) raw_df = pl.DataFrame([ {'id': 'AAPL', 'update_time': 20241112, 'status': 'trading', 'underlying': 'y'}, {'id': 'MSFT', 'update_time': 20241113, 'status': 'trading', 'underlying': 'x'}, {'id': 'NVDA', 'update_time': 20241112, 'status': 'trading', 'underlying': 'z'}, {'id': 'MSFT', 'update_time': 20241112, 'status': 'pending', 'underlying': 'x'}, {'id': 'AAPL', 'update_time': 20241113, 'status': 'trading', 'underlying': 'y'}, {'id': 'NVDA', 'update_time': 20241113, 'status': 'trading', 'underlying': 'z'}, {'id': 'TSLA', 'update_time': 20241112, 'status': 'closed', 'underlying': 'v'}, ]) result_df = find_updated_field_differences(raw_df) print(result_df)  which gives you shape: (2, 4) ┌──────┬─────────────┬─────────┬────────────┐ │ id ┆ update_time ┆ status ┆ underlying │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str ┆ str │ ╞══════╪═════════════╪═════════╪════════════╡ │ MSFT ┆ 20241112 ┆ pending ┆ x │ │ MSFT ┆ 20241113 ┆ trading ┆ x │