Question: I have a dataframe where I want to find the difference in the unique_users for the latest day(2023-09-06) and previous day(2023-09-07) for a specific hour 2 and 13 separately for each key 'bsnl' and 'other' and for a specific Exception. I need to consider the Exception and Hour and Date to calculate the difference.  DateTime Date Hour Exception IMSI_Operator 0 2023-09-06 02:00:00 2023-09-06 00:00:00 2 s2ap {'bsnl': {'total_sessions': 50007, 'unique_users': 38880}, 'Other': {'total_sessions': 50, 'unique_users': 32}} 1 2023-09-06 13:00:00 2023-09-06 00:00:00 13 s2ap {'bsnl': {'total_sessions': 60004, 'unique_users': 49816}, 'Other': {'total_sessions': 34, 'unique_users': 22}} 2 2023-09-07 13:00:00 2023-09-07 00:00:00 13 s2ap {'bsnl': {'total_sessions': 45224, 'unique_users': 37525}, 'Other': {'total_sessions': 32, 'unique_users': 27}} 3 2023-09-07 02:00:00 2023-09-07 00:00:00 2 s2ap {'bsnl': {'total_sessions': 47713, 'unique_users': 37284}, 'Other': {'total_sessions': 43, 'unique_users': 27}}  What I have tried: import pandas as pd import json # Sample DataFrame data = { 'Date': ['2023-09-06 00:00:00', '2023-09-06 00:00:00', '2023-09-07 00:00:00', '2023-09-07 00:00:00'], 'Hour': [2, 13, 13, 2], 'Exception': ['s2ap', 's2ap', 's2ap', 's2ap'], 'IMSI_Operator': [ {'bsnl': {'total_sessions': 50007, 'unique_users': 38880}, 'Other': {'total_sessions': 50, 'unique_users': 32}}, {'bsnl': {'total_sessions': 60004, 'unique_users': 49816}, 'Other': {'total_sessions': 34, 'unique_users': 22}}, {'bsnl': {'total_sessions': 45224, 'unique_users': 37525}, 'Other': {'total_sessions': 32, 'unique_users': 27}}, {'bsnl': {'total_sessions': 47713, 'unique_users': 37284}, 'Other': {'total_sessions': 43, 'unique_users': 27}} ] } result_df = pd.DataFrame(data) # Convert 'Date' to datetime result_df['Date'] = pd.to_datetime(result_df['Date']) # Filter for the latest day latest_day = result_df[result_df['Date'] == result_df['Date'].max()] # Filter for the previous day previous_day = result_df[result_df['Date'] == (result_df['Date'].max() - pd.DateOffset(days=1))] result = {} for key in latest_day['IMSI_Operator'].values[0].keys(): latest_unique_users = latest_day['IMSI_Operator'].values[0][key]['unique_users'] previous_unique_users = previous_day['IMSI_Operator'].values[0][key]['unique_users'] result[key] = { 'unique_users_diff': latest_unique_users - previous_unique_users, } print(result)  But I would like to get the difference in the original dataframe itself. The previous date 2023-09-06 rows are not neeed in the final dataframe. my current code is not considering the specific hour and specific exception for the calcualtion. Expected output: Hour Exception Date IMSI_Operator bsnl_unique_users_diff Other_unique_users_diff 2 s2ap 2023-09-07 {'bsnl': {'total_sessions': 60004, 'unique_users': 49816}, 'Other': {'total_sessions': 34, 'unique_users': 22}} -1596 5 13 s2ap 2023-09-07 {'bsnl': {'total_sessions': 50007, 'unique_users': 38880}, 'Other': {'total_sessions': 50, 'unique_users': 32}} -12291 -5 

Answer: You can slice and merge with help of json_normalize to rework the dictionaries to columns: date1 = pd.Timestamp('2023-09-07') date2 = pd.Timestamp('2023-09-06') hours = [2, 13] tmp = (df.join(pd.json_normalize(df['IMSI_Operator'])) .query('Hour in @hours') .set_index(['Date', 'Hour']) ) cols = list(tmp.filter(like='unique_users')) out = (tmp.reset_index()[list(df)] .merge(tmp.loc[date1, cols].sub(tmp.loc[date2, cols]) .add_suffix('_diff').assign(Date=date1), on=['Date', 'Hour'], how='right') .reset_index() )  Output:  index Date Hour Exception IMSI_Operator bsnl.unique_users_diff Other.unique_users_diff 0 0 2023-09-07 2 s2ap {'bsnl': {'total_sessions': 47713, 'unique_use... -1596 -5 1 1 2023-09-07 13 s2ap {'bsnl': {'total_sessions': 45224, 'unique_use... -12291 5