Package: a-pandas-ex-plode-tool
Library to handle any nested iterable (list, tuple, dict, json, etc.) in Pandas - no matter how deeply it is nested!Update:2022/09/30:DataFrame is now created directly from iter2022/09/30:No more warning (PerformanceWarning: DataFrame is highly fragmented), when DataFrame is created from a huge nested dict (depth: 1486)  Try it:https://raw.githubusercontent.com/hansalemaos/a_pandas_ex_plode_tool/main/recursion%20_hardcore_test.pypipinstalla-pandas-ex-plode-toolfroma_pandas_ex_plode_toolimportpd_add_explode_toolspd_add_explode_tools()importpandasaspddf=pd.read_csv("https://github.com/pandas-dev/pandas/raw/main/doc/data/air_quality_long.csv")HANDLE NESTED ITERABLESThe code above will add some methods topd. / pd.DataFrame / pd.Series, you can use pandas like you did before, but you will have a couple of methods more:pd.Q_AnyNestedIterable_2df()pd.Q_CorruptJsonFile_2dict()pd.Q_ReadFileWithAllEncodings_2df()df.d_filter_dtypes()df.d_multiple_columns_to_one()df.d_df_to_nested_dict()df.d_add_value_to_existing_columns_with_loc()df.d_set_values_with_df_loc()df.d_drop_rows_with_df_loc()df.d_dfloc()df.d_stack()df.d_unstack()df.d_sort_columns_with_sorted()df.d_merge_multiple_dfs_and_series_on_one_column()df.d_merge_multiple_dfs_and_series_on_index()df.d_update_original_iter()df.ds_all_nans_to_pdNA()df.ds_explode_dicts_in_column()df.ds_isna()df.ds_normalize_lists()df.s_delete_duplicates_from_iters_in_cells()df.s_flatten_all_iters_in_cells()df.s_as_flattened_list()df.s_explode_lists_and_tuples()All methods added to pandas have one of these prefixes:ds_(for DataFrames and Series)s_(only for Series)d_(only for DataFrames)Q_(added to pd.)pd.Q_AnyNestedIterable_2df() / df.d_filter_dtypes() / df.d_update_original_iter()pd.Q_AnyNestedIterable_2df()transforms any nasty iterable into a beautiful Pandas DataFrame with aMultiIndexdf.d_filter_dtypes()avoids TypeError Exceptionsdf.loc[df.aa_value >30,'aa_value'] = 90000000Traceback (most recent call last):....TypeError: '>' not supported between instances of 'str' and 'int'df.loc[df.d_filter_dtypes(allowed_dtypes=(int,float),fillvalue=pd.NA,column='aa_value') > 30] <------- No more exception!df.d_update_original_iter()After you have updated the DataFrame, you can update the original nasty iterable and keep its ugly structure.I have tested these methods a lot with examples from Stack Overflow. Until now, everything has been working like a charm. Here are about 15 examples!Nestediterablefrom:'https://stackoverflow.com/questions/61984148/how-to-handle-nested-lists-and-dictionaries-in-pandas-dataframe'{'critic_reviews':[{'review_critic':'XYZ','review_score':90},{'review_critic':'ABC','review_score':90},{'review_critic':'123','review_score':90}],'genres':['Sports','Golf'],'score':85,'title':'Golf Simulator','url':'http://example.com/golf-simulator'}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)# create DF stacked or unstacked, it doesn't matteraa_all_keysaa_valuecritic_reviews0review_critic(critic_reviews,0,review_critic)XYZreview_score(critic_reviews,0,review_score)901review_critic(critic_reviews,1,review_critic)ABCreview_score(critic_reviews,1,review_score)902review_critic(critic_reviews,2,review_critic)123review_score(critic_reviews,2,review_score)90genres0NaN(genres,0)Sports1NaN(genres,1)GolfscoreNaNNaN(score,)85titleNaNNaN(title,)GolfSimulatorurlNaNNaN(url,)http://example.com/golf-simulator#Avoid exceptions with df.d_filter_dtypes()df.loc[df.aa_value.str.contains('[Gg]',na=False),'aa_value']='UPDATE1111'#df.loc to update the dataframe (VERY IMPORTANT: To update the original iterable you have to pass 'aa_value')aa_all_keysaa_valuecritic_reviews0review_critic(critic_reviews,0,review_critic)XYZreview_score(critic_reviews,0,review_score)901review_critic(critic_reviews,1,review_critic)ABCreview_score(critic_reviews,1,review_score)902review_critic(critic_reviews,2,review_critic)123review_score(critic_reviews,2,review_score)90genres0NaN(genres,0)Sports1NaN(genres,1)UPDATE1111scoreNaNNaN(score,)85titleNaNNaN(title,)UPDATE1111urlNaNNaN(url,)UPDATE1111mod_iter=df.d_update_original_iter(data,verbose=True)#updating the nested iterable, the new values have to be in the column 'aa_value', if you have added new columns to the dataframe, drop them before updating the original iterable[genres][1]Oldvalue:Golf[genres][1]Updatedvalue:UPDATE1111[title]Oldvalue:GolfSimulator[title]Updatedvalue:UPDATE1111[url]Oldvalue:http://example.com/golf-simulator[url]Updatedvalue:UPDATE1111{'critic_reviews':[{'review_critic':'XYZ','review_score':90},{'review_critic':'ABC','review_score':90},{'review_critic':'123','review_score':90}],'genres':['Sports','UPDATE1111'],'score':85,'title':'UPDATE1111','url':'UPDATE1111'}#Nested iterable from:https://stackoverflow.com/questions/73430585/how-to-convert-a-list-of-nested-dictionaries-includes-tuples-as-a-dataframedata=[{'cb':({'ID':1,'Name':'A','num':50},{'ID':2,'Name':'A','num':68}),'final_value':118},{'cb':({'ID':1,'Name':'A','num':50},{'ID':4,'Name':'A','num':67}),'final_value':117},{'cb':({'ID':1,'Name':'A','num':50},{'ID':6,'Name':'A','num':67}),'final_value':117}]df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value0cb0ID(0,cb,0,ID)1Name(0,cb,0,Name)Anum(0,cb,0,num)501ID(0,cb,1,ID)2Name(0,cb,1,Name)Anum(0,cb,1,num)68final_valueNaNNaN(0,final_value)1181cb0ID(1,cb,0,ID)1Name(1,cb,0,Name)Anum(1,cb,0,num)501ID(1,cb,1,ID)4Name(1,cb,1,Name)Anum(1,cb,1,num)67final_valueNaNNaN(1,final_value)1172cb0ID(2,cb,0,ID)1Name(2,cb,0,Name)Anum(2,cb,0,num)501ID(2,cb,1,ID)6Name(2,cb,1,Name)Anum(2,cb,1,num)67final_valueNaNNaN(2,final_value)117df.loc[df.d_filter_dtypes(allowed_dtypes=(int,float),fillvalue=pd.NA,column='aa_value')>30,'aa_value']=900000aa_all_keysaa_value0cb0ID(0,cb,0,ID)1Name(0,cb,0,Name)Anum(0,cb,0,num)9000001ID(0,cb,1,ID)2Name(0,cb,1,Name)Anum(0,cb,1,num)900000final_valueNaNNaN(0,final_value)9000001cb0ID(1,cb,0,ID)1Name(1,cb,0,Name)Anum(1,cb,0,num)9000001ID(1,cb,1,ID)4Name(1,cb,1,Name)Anum(1,cb,1,num)900000final_valueNaNNaN(1,final_value)9000002cb0ID(2,cb,0,ID)1Name(2,cb,0,Name)Anum(2,cb,0,num)9000001ID(2,cb,1,ID)6Name(2,cb,1,Name)Anum(2,cb,1,num)900000final_valueNaNNaN(2,final_value)900000mod_iter=df.d_update_original_iter(data,verbose=True)[0][cb][0][num]Oldvalue:50[0][cb][0][num]Updatedvalue:900000[0][cb][1][num]Oldvalue:68[0][cb][1][num]Updatedvalue:900000[0][final_value]Oldvalue:118[0][final_value]Updatedvalue:900000[1][cb][0][num]Oldvalue:50[1][cb][0][num]Updatedvalue:900000[1][cb][1][num]Oldvalue:67[1][cb][1][num]Updatedvalue:900000[1][final_value]Oldvalue:117[1][final_value]Updatedvalue:900000[2][cb][0][num]Oldvalue:50[2][cb][0][num]Updatedvalue:900000[2][cb][1][num]Oldvalue:67[2][cb][1][num]Updatedvalue:900000[2][final_value]Oldvalue:117[2][final_value]Updatedvalue:900000[{'cb':({'ID':1,'Name':'A','num':900000},{'ID':2,'Name':'A','num':900000}),'final_value':900000},{'cb':({'ID':1,'Name':'A','num':900000},{'ID':4,'Name':'A','num':900000}),'final_value':900000},{'cb':({'ID':1,'Name':'A','num':900000},{'ID':6,'Name':'A','num':900000}),'final_value':900000}]#Nested iterable from:https://stackoverflow.com/questions/69943509/problems-when-flatten-a-dictdata=[{'application_contacts':[{'adress':'X','email':'test@test.com'}],'application_details':{'email':None,'phone':None},'employer':{'Name':'Nom','email':None},'id':'1'},{'application_contacts':[{'adress':'Z','email':None}],'application_details':{'email':'testy@test_a.com','phone':None},'employer':{'Name':'Nom','email':None},'id':'2'},{'application_contacts':[{'adress':'Y','email':None}],'application_details':{'email':'testy@test_a.com','phone':None},'employer':{'Name':'Nom','email':None},'id':'3'}]df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value0application_contacts0adress(0,application_contacts,0,adress)Xemail(0,application_contacts,0,email)test@test.comapplication_detailsemailNaN(0,application_details,email)NonephoneNaN(0,application_details,phone)NoneemployerNameNaN(0,employer,Name)NomemailNaN(0,employer,email)NoneidNaNNaN(0,id)11application_contacts0adress(1,application_contacts,0,adress)Zemail(1,application_contacts,0,email)Noneapplication_detailsemailNaN(1,application_details,email)testy@test_a.comphoneNaN(1,application_details,phone)NoneemployerNameNaN(1,employer,Name)NomemailNaN(1,employer,email)NoneidNaNNaN(1,id)22application_contacts0adress(2,application_contacts,0,adress)Yemail(2,application_contacts,0,email)Noneapplication_detailsemailNaN(2,application_details,email)testy@test_a.comphoneNaN(2,application_details,phone)NoneemployerNameNaN(2,employer,Name)NomemailNaN(2,employer,email)NoneidNaNNaN(2,id)3df.loc[df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'test_a\.\w+\b',na=False),'aa_value']='UPPPPPPPPPPPPPPPDATE.COM'aa_all_keysaa_value0application_contacts0adress(0,application_contacts,0,adress)Xemail(0,application_contacts,0,email)test@test.comapplication_detailsemailNaN(0,application_details,email)NonephoneNaN(0,application_details,phone)NoneemployerNameNaN(0,employer,Name)NomemailNaN(0,employer,email)NoneidNaNNaN(0,id)11application_contacts0adress(1,application_contacts,0,adress)Zemail(1,application_contacts,0,email)Noneapplication_detailsemailNaN(1,application_details,email)UPPPPPPPPPPPPPPPDATE.COMphoneNaN(1,application_details,phone)NoneemployerNameNaN(1,employer,Name)NomemailNaN(1,employer,email)NoneidNaNNaN(1,id)22application_contacts0adress(2,application_contacts,0,adress)Yemail(2,application_contacts,0,email)Noneapplication_detailsemailNaN(2,application_details,email)UPPPPPPPPPPPPPPPDATE.COMphoneNaN(2,application_details,phone)NoneemployerNameNaN(2,employer,Name)NomemailNaN(2,employer,email)NoneidNaNNaN(2,id)3mod_iter=df.d_update_original_iter(data,verbose=True)[1][application_details][email]Oldvalue:testy@test_a.com[1][application_details][email]Updatedvalue:UPPPPPPPPPPPPPPPDATE.COM[2][application_details][email]Oldvalue:testy@test_a.com[2][application_details][email]Updatedvalue:UPPPPPPPPPPPPPPPDATE.COM[{'application_contacts':[{'adress':'X','email':'test@test.com'}],'application_details':{'email':None,'phone':None},'employer':{'Name':'Nom','email':None},'id':'1'},{'application_contacts':[{'adress':'Z','email':None}],'application_details':{'email':'UPPPPPPPPPPPPPPPDATE.COM','phone':None},'employer':{'Name':'Nom','email':None},'id':'2'},{'application_contacts':[{'adress':'Y','email':None}],'application_details':{'email':'UPPPPPPPPPPPPPPPDATE.COM','phone':None},'employer':{'Name':'Nom','email':None},'id':'3'}]#Nested iterable from:https://stackoverflow.com/questions/62765371/convert-nested-dataframe-to-a-simple-dataframeframedata={'A':[1,2,3],'B':[4,5,6],'departure':[{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'}]}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valueA0NaN(A,0)11NaN(A,1)22NaN(A,2)3B0NaN(B,0)41NaN(B,1)52NaN(B,2)6departure0actual(departure,0,actual)Noneactual_runway(departure,0,actual_runway)Noneairport(departure,0,airport)Findeldelay(departure,0,delay)Noneestimated(departure,0,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,0,estimated_runway)Nonegate(departure,0,gate)Noneiata(departure,0,iata)LUXicao(departure,0,icao)ELLXscheduled(departure,0,scheduled)2020-07-07T06:30:00+00:00terminal(departure,0,terminal)Nonetimezone(departure,0,timezone)Europe/Luxembourg1actual(departure,1,actual)Noneactual_runway(departure,1,actual_runway)Noneairport(departure,1,airport)Findeldelay(departure,1,delay)Noneestimated(departure,1,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,1,estimated_runway)Nonegate(departure,1,gate)Noneiata(departure,1,iata)LUXicao(departure,1,icao)ELLXscheduled(departure,1,scheduled)2020-07-07T06:30:00+00:00terminal(departure,1,terminal)Nonetimezone(departure,1,timezone)Europe/Luxembourg2actual(departure,2,actual)Noneactual_runway(departure,2,actual_runway)Noneairport(departure,2,airport)Findeldelay(departure,2,delay)Noneestimated(departure,2,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,2,estimated_runway)Nonegate(departure,2,gate)Noneiata(departure,2,iata)LUXicao(departure,2,icao)ELLXscheduled(departure,2,scheduled)2020-07-07T06:30:00+00:00terminal(departure,2,terminal)Nonetimezone(departure,2,timezone)Europe/Luxembourgdf.loc[df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value')=='ELLX','aa_value']='ELLX-UPDATED'aa_all_keysaa_valueA0NaN(A,0)11NaN(A,1)22NaN(A,2)3B0NaN(B,0)41NaN(B,1)52NaN(B,2)6departure0actual(departure,0,actual)Noneactual_runway(departure,0,actual_runway)Noneairport(departure,0,airport)Findeldelay(departure,0,delay)Noneestimated(departure,0,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,0,estimated_runway)Nonegate(departure,0,gate)Noneiata(departure,0,iata)LUXicao(departure,0,icao)ELLX-UPDATEDscheduled(departure,0,scheduled)2020-07-07T06:30:00+00:00terminal(departure,0,terminal)Nonetimezone(departure,0,timezone)Europe/Luxembourg1actual(departure,1,actual)Noneactual_runway(departure,1,actual_runway)Noneairport(departure,1,airport)Findeldelay(departure,1,delay)Noneestimated(departure,1,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,1,estimated_runway)Nonegate(departure,1,gate)Noneiata(departure,1,iata)LUXicao(departure,1,icao)ELLX-UPDATEDscheduled(departure,1,scheduled)2020-07-07T06:30:00+00:00terminal(departure,1,terminal)Nonetimezone(departure,1,timezone)Europe/Luxembourg2actual(departure,2,actual)Noneactual_runway(departure,2,actual_runway)Noneairport(departure,2,airport)Findeldelay(departure,2,delay)Noneestimated(departure,2,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,2,estimated_runway)Nonegate(departure,2,gate)Noneiata(departure,2,iata)LUXicao(departure,2,icao)ELLX-UPDATEDscheduled(departure,2,scheduled)2020-07-07T06:30:00+00:00terminal(departure,2,terminal)Nonetimezone(departure,2,timezone)Europe/Luxembourgmod_iter=df.d_update_original_iter(data,verbose=True)[departure][0][icao]Oldvalue:ELLX[departure][0][icao]Updatedvalue:ELLX-UPDATED[departure][1][icao]Oldvalue:ELLX[departure][1][icao]Updatedvalue:ELLX-UPDATED[departure][2][icao]Oldvalue:ELLX[departure][2][icao]Updatedvalue:ELLX-UPDATED{'A':[1,2,3],'B':[4,5,6],'departure':[{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX-UPDATED','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX-UPDATED','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX-UPDATED','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'}]}#Nested iterable from:https://stackoverflow.com/questions/64359762/constructing-a-pandas-dataframe-with-columns-and-sub-columns-from-nested-dictiondata={'level1':{'t1':{'s1':{'col1':5,'col2':4,'col3':4,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':8},'s3':{'col1':11,'col2':8,'col3':2,'col4':9},'s4':{'col1':5,'col2':4,'col3':4,'col4':9}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':8},'s3':{'col1':11,'col2':8,'col3':2,'col4':9},'s4':{'col1':5,'col2':4,'col3':4,'col4':9}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':6,'col3':7,'col4':8},'s3':{'col1':9,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}},'level2':{'t1':{'s1':{'col1':5,'col2':4,'col3':9,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':5},'s3':{'col1':11,'col2':8,'col3':2,'col4':13},'s4':{'col1':5,'col2':4,'col3':4,'col4':20}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':8},'s3':{'col1':11,'col2':8,'col3':2,'col4':9},'s4':{'col1':5,'col2':4,'col3':4,'col4':9}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':6,'col3':7,'col4':8},'s3':{'col1':9,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}}}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valuelevel1t1s1col1(level1,t1,s1,col1)5col2(level1,t1,s1,col2)4col3(level1,t1,s1,col3)4col4(level1,t1,s1,col4)9s2col1(level1,t1,s2,col1)1......level2t3s3col4(level2,t3,s3,col4)12s4col1(level2,t3,s4,col1)13col2(level2,t3,s4,col2)14col3(level2,t3,s4,col3)15col4(level2,t3,s4,col4)16[96rowsx2columns]df.loc[(df.d_filter_dtypes(allowed_dtypes=(int),fillvalue=pd.NA,column='aa_value')>5)&(df.d_filter_dtypes(allowed_dtypes=(int),fillvalue=pd.NA,column='aa_value')<10),'aa_value']=1000000aa_all_keysaa_valuelevel1t1s1col1(level1,t1,s1,col1)5col2(level1,t1,s1,col2)4col3(level1,t1,s1,col3)4col4(level1,t1,s1,col4)1000000s2col1(level1,t1,s2,col1)1......level2t3s3col4(level2,t3,s3,col4)12s4col1(level2,t3,s4,col1)13col2(level2,t3,s4,col2)14col3(level2,t3,s4,col3)15col4(level2,t3,s4,col4)16[96rowsx2columns]mod_iter=df.d_update_original_iter(data,verbose=True)[level1][t1][s1][col4]Oldvalue:9[level1][t1][s1][col4]Updatedvalue:1000000[level1][t1][s2][col4]Oldvalue:8[level1][t1][s2][col4]Updatedvalue:1000000[level1][t1][s3][col2]Oldvalue:8[level1][t1][s3][col2]Updatedvalue:1000000[level1][t1][s3][col4]Oldvalue:9[level1][t1][s3][col4]Updatedvalue:1000000[level1][t1][s4][col4]Oldvalue:9[level1][t1][s4][col4]Updatedvalue:1000000[level1][t2][s1][col4]Oldvalue:9[level1][t2][s1][col4]Updatedvalue:1000000[level1][t2][s2][col4]Oldvalue:8[level1][t2][s2][col4]Updatedvalue:1000000[level1][t2][s3][col2]Oldvalue:8[level1][t2][s3][col2]Updatedvalue:1000000[level1][t2][s3][col4]Oldvalue:9[level1][t2][s3][col4]Updatedvalue:1000000[level1][t2][s4][col4]Oldvalue:9[level1][t2][s4][col4]Updatedvalue:1000000[level1][t3][s2][col2]Oldvalue:6[level1][t3][s2][col2]Updatedvalue:1000000[level1][t3][s2][col3]Oldvalue:7[level1][t3][s2][col3]Updatedvalue:1000000[level1][t3][s2][col4]Oldvalue:8[level1][t3][s2][col4]Updatedvalue:1000000[level1][t3][s3][col1]Oldvalue:9[level1][t3][s3][col1]Updatedvalue:1000000[level2][t1][s1][col3]Oldvalue:9[level2][t1][s1][col3]Updatedvalue:1000000[level2][t1][s1][col4]Oldvalue:9[level2][t1][s1][col4]Updatedvalue:1000000[level2][t1][s3][col2]Oldvalue:8[level2][t1][s3][col2]Updatedvalue:1000000[level2][t2][s1][col4]Oldvalue:9[level2][t2][s1][col4]Updatedvalue:1000000[level2][t2][s2][col4]Oldvalue:8[level2][t2][s2][col4]Updatedvalue:1000000[level2][t2][s3][col2]Oldvalue:8[level2][t2][s3][col2]Updatedvalue:1000000[level2][t2][s3][col4]Oldvalue:9[level2][t2][s3][col4]Updatedvalue:1000000[level2][t2][s4][col4]Oldvalue:9[level2][t2][s4][col4]Updatedvalue:1000000[level2][t3][s2][col2]Oldvalue:6[level2][t3][s2][col2]Updatedvalue:1000000[level2][t3][s2][col3]Oldvalue:7[level2][t3][s2][col3]Updatedvalue:1000000[level2][t3][s2][col4]Oldvalue:8[level2][t3][s2][col4]Updatedvalue:1000000[level2][t3][s3][col1]Oldvalue:9[level2][t3][s3][col1]Updatedvalue:1000000{'level1':{'t1':{'s1':{'col1':5,'col2':4,'col3':4,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':1000000},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':1000000},'s4':{'col1':5,'col2':4,'col3':4,'col4':1000000}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':1000000},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':1000000},'s4':{'col1':5,'col2':4,'col3':4,'col4':1000000}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':1000000,'col3':1000000,'col4':1000000},'s3':{'col1':1000000,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}},'level2':{'t1':{'s1':{'col1':5,'col2':4,'col3':1000000,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':5},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':13},'s4':{'col1':5,'col2':4,'col3':4,'col4':20}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':1000000},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':1000000},'s4':{'col1':5,'col2':4,'col3':4,'col4':1000000}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':1000000,'col3':1000000,'col4':1000000},'s3':{'col1':1000000,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}}}#Nested iterable from:https://stackoverflow.com/questions/72146094/problems-matching-values-from-nested-dictionarydata={'_links':{'next':None,'prev':None},'limit':250,'offset':0,'runs':[{'assignedto_id':None,'blocked_count':0,'completed_on':None,'config':None,'config_ids':[],'created_by':1,'created_on':1651790693,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':1,'id':13,'include_all':False,'is_completed':False,'milestone_id':None,'name':'2022-05-05-testrun','passed_count':2,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':0,'updated_on':1651790693,'url':'https://xxxxxxxxxx.testrail.io/index.php?/runs/view/13'},{'assignedto_id':None,'blocked_count':0,'completed_on':1650989972,'config':None,'config_ids':[],'created_by':5,'created_on':1650966329,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':0,'id':9,'include_all':False,'is_completed':True,'milestone_id':None,'name':'This is a new test run','passed_count':0,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':3,'updated_on':1650966329,'url':'https://xxxxxxxxxx.testrail.io/index.php?/runs/view/9'}],'size':2}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value_linksnextNaN(_links,next)NoneprevNaN(_links,prev)NonelimitNaNNaN(limit,)250offsetNaNNaN(offset,)0runs0assignedto_id(runs,0,assignedto_id)None......1suite_id(runs,1,suite_id)1untested_count(runs,1,untested_count)3updated_on(runs,1,updated_on)1650966329url(runs,1,url)https://xxxxxxxxxx.testrail.io/index.php?/runs...sizeNaNNaN(size,)2[63rowsx2columns]df.loc[(df.d_filter_dtypes(allowed_dtypes=(bool),fillvalue=pd.NA,column='aa_value')==False),'aa_value']=Truedf.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'https?://.*',na=False)),'aa_value']='WWW.PYTHON.ORG'aa_all_keysaa_value_linksnextNaN(_links,next)NoneprevNaN(_links,prev)NonelimitNaNNaN(limit,)250offsetNaNNaN(offset,)0runs0assignedto_id(runs,0,assignedto_id)None......1suite_id(runs,1,suite_id)1untested_count(runs,1,untested_count)3updated_on(runs,1,updated_on)1650966329url(runs,1,url)WWW.PYTHON.ORGsizeNaNNaN(size,)2[63rowsx2columns]mod_iter=df.d_update_original_iter(data,verbose=True)[runs][0][include_all]Oldvalue:False[runs][0][include_all]Updatedvalue:True[runs][0][is_completed]Oldvalue:False[runs][0][is_completed]Updatedvalue:True[runs][0][url]Oldvalue:https://xxxxxxxxxx.testrail.io/index.php?/runs/view/13[runs][0][url]Updatedvalue:WWW.PYTHON.ORG[runs][1][include_all]Oldvalue:False[runs][1][include_all]Updatedvalue:True[runs][1][url]Oldvalue:https://xxxxxxxxxx.testrail.io/index.php?/runs/view/9[runs][1][url]Updatedvalue:WWW.PYTHON.ORG{'_links':{'next':None,'prev':None},'limit':250,'offset':0,'runs':[{'assignedto_id':None,'blocked_count':0,'completed_on':None,'config':None,'config_ids':[],'created_by':1,'created_on':1651790693,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':1,'id':13,'include_all':True,'is_completed':True,'milestone_id':None,'name':'2022-05-05-testrun','passed_count':2,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':0,'updated_on':1651790693,'url':'WWW.PYTHON.ORG'},{'assignedto_id':None,'blocked_count':0,'completed_on':1650989972,'config':None,'config_ids':[],'created_by':5,'created_on':1650966329,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':0,'id':9,'include_all':True,'is_completed':True,'milestone_id':None,'name':'This is a new test run','passed_count':0,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':3,'updated_on':1650966329,'url':'WWW.PYTHON.ORG'}],'size':2}#Nested iterable from:https://stackoverflow.com/questions/73708706/how-to-get-values-from-list-of-nested-dictionaries/73839430#73839430data={'results':[{'end_time':'2021-01-21','key':'q1','result_type':'multipleChoice','start_time':'2021-01-21','value':['1']},{'end_time':'2021-01-21','key':'q2','result_type':'multipleChoice','start_time':'2021-01-21','value':['False']},{'end_time':'2021-01-21','key':'q3','result_type':'multipleChoice','start_time':'2021-01-21','value':['3']},{'end_time':'2021-01-21','key':'q4','result_type':'multipleChoice','start_time':'2021-01-21','value':['3']}]}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valueresults0end_timeNaN(results,0,end_time)2021-01-21keyNaN(results,0,key)q1result_typeNaN(results,0,result_type)multipleChoicestart_timeNaN(results,0,start_time)2021-01-21value0(results,0,value,0)11end_timeNaN(results,1,end_time)2021-01-21keyNaN(results,1,key)q2result_typeNaN(results,1,result_type)multipleChoicestart_timeNaN(results,1,start_time)2021-01-21value0(results,1,value,0)False2end_timeNaN(results,2,end_time)2021-01-21keyNaN(results,2,key)q3result_typeNaN(results,2,result_type)multipleChoicestart_timeNaN(results,2,start_time)2021-01-21value0(results,2,value,0)33end_timeNaN(results,3,end_time)2021-01-21keyNaN(results,3,key)q4result_typeNaN(results,3,result_type)multipleChoicestart_timeNaN(results,3,start_time)2021-01-21value0(results,3,value,0)3df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'^2021.*',na=False)),'aa_value']=10000000000aa_all_keysaa_valueresults0end_timeNaN(results,0,end_time)10000000000keyNaN(results,0,key)q1result_typeNaN(results,0,result_type)multipleChoicestart_timeNaN(results,0,start_time)10000000000value0(results,0,value,0)11end_timeNaN(results,1,end_time)10000000000keyNaN(results,1,key)q2result_typeNaN(results,1,result_type)multipleChoicestart_timeNaN(results,1,start_time)10000000000value0(results,1,value,0)False2end_timeNaN(results,2,end_time)10000000000keyNaN(results,2,key)q3result_typeNaN(results,2,result_type)multipleChoicestart_timeNaN(results,2,start_time)10000000000value0(results,2,value,0)33end_timeNaN(results,3,end_time)10000000000keyNaN(results,3,key)q4result_typeNaN(results,3,result_type)multipleChoicestart_timeNaN(results,3,start_time)10000000000value0(results,3,value,0)3mod_iter=df.d_update_original_iter(data,verbose=True)[results][0][end_time]Oldvalue:2021-01-21[results][0][end_time]Updatedvalue:10000000000[results][0][start_time]Oldvalue:2021-01-21[results][0][start_time]Updatedvalue:10000000000[results][1][end_time]Oldvalue:2021-01-21[results][1][end_time]Updatedvalue:10000000000[results][1][start_time]Oldvalue:2021-01-21[results][1][start_time]Updatedvalue:10000000000[results][2][end_time]Oldvalue:2021-01-21[results][2][end_time]Updatedvalue:10000000000[results][2][start_time]Oldvalue:2021-01-21[results][2][start_time]Updatedvalue:10000000000[results][3][end_time]Oldvalue:2021-01-21[results][3][end_time]Updatedvalue:10000000000[results][3][start_time]Oldvalue:2021-01-21[results][3][start_time]Updatedvalue:10000000000{'results':[{'end_time':10000000000,'key':'q1','result_type':'multipleChoice','start_time':10000000000,'value':['1']},{'end_time':10000000000,'key':'q2','result_type':'multipleChoice','start_time':10000000000,'value':['False']},{'end_time':10000000000,'key':'q3','result_type':'multipleChoice','start_time':10000000000,'value':['3']},{'end_time':10000000000,'key':'q4','result_type':'multipleChoice','start_time':10000000000,'value':['3']}]}#Nested iterable from:https://stackoverflow.com/questions/66461902/flattening-nested-dictionary-into-dataframe-pythondata={1:{2:{'IDs':{'BookID':['543533254353','4324232342'],'SalesID':['543267765345','4353543'],'StoreID':['111111','1121111']},'Name':'boring Tales of Dragon Slayers'},'IDs':{'BookID':['543533254353'],'SalesID':['543267765345'],'StoreID':['123445452543']},'Name':'Thrilling Tales of Dragon Slayers'}}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value1IDsBookID0NaN(1,IDs,BookID,0)543533254353SalesID0NaN(1,IDs,SalesID,0)543267765345StoreID0NaN(1,IDs,StoreID,0)123445452543NameNaNNaNNaN(1,Name)ThrillingTalesofDragonSlayers2IDsBookID0(1,2,IDs,BookID,0)5435332543531(1,2,IDs,BookID,1)4324232342SalesID0(1,2,IDs,SalesID,0)5432677653451(1,2,IDs,SalesID,1)4353543StoreID0(1,2,IDs,StoreID,0)1111111(1,2,IDs,StoreID,1)1121111NameNaNNaN(1,2,Name)boringTalesofDragonSlayersdf.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'^\d+$',na=False)),'aa_value']=df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'^\d+$',na=False)),'aa_value'].astype(float)aa_all_keysaa_value1IDsBookID0NaN(1,IDs,BookID,0)543533254353.0SalesID0NaN(1,IDs,SalesID,0)543267765345.0StoreID0NaN(1,IDs,StoreID,0)123445452543.0NameNaNNaNNaN(1,Name)ThrillingTalesofDragonSlayers2IDsBookID0(1,2,IDs,BookID,0)543533254353.01(1,2,IDs,BookID,1)4324232342.0SalesID0(1,2,IDs,SalesID,0)543267765345.01(1,2,IDs,SalesID,1)4353543.0StoreID0(1,2,IDs,StoreID,0)111111.01(1,2,IDs,StoreID,1)1121111.0NameNaNNaN(1,2,Name)boringTalesofDragonSlayersmod_iter=df.d_update_original_iter(data,verbose=True)[1][2][IDs][BookID][0]Oldvalue:543533254353[1][2][IDs][BookID][0]Updatedvalue:543533254353.0[1][2][IDs][BookID][1]Oldvalue:4324232342[1][2][IDs][BookID][1]Updatedvalue:4324232342.0[1][2][IDs][SalesID][0]Oldvalue:543267765345[1][2][IDs][SalesID][0]Updatedvalue:543267765345.0[1][2][IDs][SalesID][1]Oldvalue:4353543[1][2][IDs][SalesID][1]Updatedvalue:4353543.0[1][2][IDs][StoreID][0]Oldvalue:111111[1][2][IDs][StoreID][0]Updatedvalue:111111.0[1][2][IDs][StoreID][1]Oldvalue:1121111[1][2][IDs][StoreID][1]Updatedvalue:1121111.0[1][IDs][BookID][0]Oldvalue:543533254353[1][IDs][BookID][0]Updatedvalue:543533254353.0[1][IDs][SalesID][0]Oldvalue:543267765345[1][IDs][SalesID][0]Updatedvalue:543267765345.0[1][IDs][StoreID][0]Oldvalue:123445452543[1][IDs][StoreID][0]Updatedvalue:123445452543.0{1:{2:{'IDs':{'BookID':[543533254353.0,4324232342.0],'SalesID':[543267765345.0,4353543.0],'StoreID':[111111.0,1121111.0]},'Name':'boring Tales of Dragon Slayers'},'IDs':{'BookID':[543533254353.0],'SalesID':[543267765345.0],'StoreID':[123445452543.0]},'Name':'Thrilling Tales of Dragon Slayers'}}Nestediterablefrom:'https://stackoverflow.com/questions/61984148/how-to-handle-nested-lists-and-dictionaries-in-pandas-dataframe'{'critic_reviews':[{'review_critic':'XYZ','review_score':90},{'review_critic':'ABC','review_score':90},{'review_critic':'123','review_score':90}],'genres':['Sports','Golf'],'score':85,'title':'Golf Simulator','url':'http://example.com/golf-simulator'}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)# create DF stacked or unstacked, it doesn't matteraa_all_keysaa_valuecritic_reviews0review_critic(critic_reviews,0,review_critic)XYZreview_score(critic_reviews,0,review_score)901review_critic(critic_reviews,1,review_critic)ABCreview_score(critic_reviews,1,review_score)902review_critic(critic_reviews,2,review_critic)123review_score(critic_reviews,2,review_score)90genres0NaN(genres,0)Sports1NaN(genres,1)GolfscoreNaNNaN(score,)85titleNaNNaN(title,)GolfSimulatorurlNaNNaN(url,)http://example.com/golf-simulatordf.loc[df.aa_value.str.contains('[Gg]',na=False),'aa_value']='UPDATE1111'#df.loc to update the dataframe (VERY IMPORTANT: To update the original iterable you have to pass 'aa_value')aa_all_keysaa_valuecritic_reviews0review_critic(critic_reviews,0,review_critic)XYZreview_score(critic_reviews,0,review_score)901review_critic(critic_reviews,1,review_critic)ABCreview_score(critic_reviews,1,review_score)902review_critic(critic_reviews,2,review_critic)123review_score(critic_reviews,2,review_score)90genres0NaN(genres,0)Sports1NaN(genres,1)UPDATE1111scoreNaNNaN(score,)85titleNaNNaN(title,)UPDATE1111urlNaNNaN(url,)UPDATE1111mod_iter=df.d_update_original_iter(data,verbose=True)#updating the nested iterable, the new values have to be in the column 'aa_value', if you have added new columns to the dataframe, drop them before updating the original iterable[genres][1]Oldvalue:Golf[genres][1]Updatedvalue:UPDATE1111[title]Oldvalue:GolfSimulator[title]Updatedvalue:UPDATE1111[url]Oldvalue:http://example.com/golf-simulator[url]Updatedvalue:UPDATE1111{'critic_reviews':[{'review_critic':'XYZ','review_score':90},{'review_critic':'ABC','review_score':90},{'review_critic':'123','review_score':90}],'genres':['Sports','UPDATE1111'],'score':85,'title':'UPDATE1111','url':'UPDATE1111'}#Nested iterable from:https://stackoverflow.com/questions/72990265/convert-nested-list-in-dictionary-to-dataframe/72990346data={'a':'test','b':1657,'c':'asset','d':[['2089','0.0'],['2088','0.0']],'e':[['2088','0.0'],['2088','0.0'],['2088','0.00']],'f':[['2088','0.0','x','foo'],['2088','0.0','bar','i'],['2088','0.00','z','0.2']],'x':['test1','test2']}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valueaNaNNaN(a,)testbNaNNaN(b,)1657cNaNNaN(c,)assetd00(d,0,0)20891(d,0,1)0.010(d,1,0)20881(d,1,1)0.0e00(e,0,0)20881(e,0,1)0.010(e,1,0)20881(e,1,1)0.020(e,2,0)20881(e,2,1)0.00f00(f,0,0)20881(f,0,1)0.02(f,0,2)x3(f,0,3)foo10(f,1,0)20881(f,1,1)0.02(f,1,2)bar3(f,1,3)i20(f,2,0)20881(f,2,1)0.002(f,2,2)z3(f,2,3)0.2x0NaN(x,0)test11NaN(x,1)test2df.loc[df.aa_value==1657,'aa_value']=1657*30aa_all_keysaa_valueaNaNNaN(a,)testbNaNNaN(b,)49710cNaNNaN(c,)assetd00(d,0,0)20891(d,0,1)0.010(d,1,0)20881(d,1,1)0.0e00(e,0,0)20881(e,0,1)0.010(e,1,0)20881(e,1,1)0.020(e,2,0)20881(e,2,1)0.00f00(f,0,0)20881(f,0,1)0.02(f,0,2)x3(f,0,3)foo10(f,1,0)20881(f,1,1)0.02(f,1,2)bar3(f,1,3)i20(f,2,0)20881(f,2,1)0.002(f,2,2)z3(f,2,3)0.2x0NaN(x,0)test11NaN(x,1)test2mod_iter=df.d_update_original_iter(data,verbose=True)[b]Oldvalue:1657[b]Updatedvalue:49710{'a':'test','b':49710,'c':'asset','d':[['2089','0.0'],['2088','0.0']],'e':[['2088','0.0'],['2088','0.0'],['2088','0.00']],'f':[['2088','0.0','x','foo'],['2088','0.0','bar','i'],['2088','0.00','z','0.2']],'x':['test1','test2']}#Nested iterable from:https://stackoverflow.com/questions/73430585/how-to-convert-a-list-of-nested-dictionaries-includes-tuples-as-a-dataframedata=[{'cb':({'ID':1,'Name':'A','num':50},{'ID':2,'Name':'A','num':68}),'final_value':118},{'cb':({'ID':1,'Name':'A','num':50},{'ID':4,'Name':'A','num':67}),'final_value':117},{'cb':({'ID':1,'Name':'A','num':50},{'ID':6,'Name':'A','num':67}),'final_value':117}]df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value0cb0ID(0,cb,0,ID)1Name(0,cb,0,Name)Anum(0,cb,0,num)501ID(0,cb,1,ID)2Name(0,cb,1,Name)Anum(0,cb,1,num)68final_valueNaNNaN(0,final_value)1181cb0ID(1,cb,0,ID)1Name(1,cb,0,Name)Anum(1,cb,0,num)501ID(1,cb,1,ID)4Name(1,cb,1,Name)Anum(1,cb,1,num)67final_valueNaNNaN(1,final_value)1172cb0ID(2,cb,0,ID)1Name(2,cb,0,Name)Anum(2,cb,0,num)501ID(2,cb,1,ID)6Name(2,cb,1,Name)Anum(2,cb,1,num)67final_valueNaNNaN(2,final_value)117df.d_filter_dtypes(allowed_dtypes=(int,float),fillvalue=pd.NA,column='aa_value')>30,'aa_value']=900000aa_all_keysaa_value0cb0ID(0,cb,0,ID)1Name(0,cb,0,Name)Anum(0,cb,0,num)9000001ID(0,cb,1,ID)2Name(0,cb,1,Name)Anum(0,cb,1,num)900000final_valueNaNNaN(0,final_value)9000001cb0ID(1,cb,0,ID)1Name(1,cb,0,Name)Anum(1,cb,0,num)9000001ID(1,cb,1,ID)4Name(1,cb,1,Name)Anum(1,cb,1,num)900000final_valueNaNNaN(1,final_value)9000002cb0ID(2,cb,0,ID)1Name(2,cb,0,Name)Anum(2,cb,0,num)9000001ID(2,cb,1,ID)6Name(2,cb,1,Name)Anum(2,cb,1,num)900000final_valueNaNNaN(2,final_value)900000mod_iter=df.d_update_original_iter(data,verbose=True)[0][cb][0][num]Oldvalue:50[0][cb][0][num]Updatedvalue:900000[0][cb][1][num]Oldvalue:68[0][cb][1][num]Updatedvalue:900000[0][final_value]Oldvalue:118[0][final_value]Updatedvalue:900000[1][cb][0][num]Oldvalue:50[1][cb][0][num]Updatedvalue:900000[1][cb][1][num]Oldvalue:67[1][cb][1][num]Updatedvalue:900000[1][final_value]Oldvalue:117[1][final_value]Updatedvalue:900000[2][cb][0][num]Oldvalue:50[2][cb][0][num]Updatedvalue:900000[2][cb][1][num]Oldvalue:67[2][cb][1][num]Updatedvalue:900000[2][final_value]Oldvalue:117[2][final_value]Updatedvalue:900000[{'cb':({'ID':1,'Name':'A','num':900000},{'ID':2,'Name':'A','num':900000}),'final_value':900000},{'cb':({'ID':1,'Name':'A','num':900000},{'ID':4,'Name':'A','num':900000}),'final_value':900000},{'cb':({'ID':1,'Name':'A','num':900000},{'ID':6,'Name':'A','num':900000}),'final_value':900000}]#Nested iterable from:https://stackoverflow.com/questions/69943509/problems-when-flatten-a-dictdata=[{'application_contacts':[{'adress':'X','email':'test@test.com'}],'application_details':{'email':None,'phone':None},'employer':{'Name':'Nom','email':None},'id':'1'},{'application_contacts':[{'adress':'Z','email':None}],'application_details':{'email':'testy@test_a.com','phone':None},'employer':{'Name':'Nom','email':None},'id':'2'},{'application_contacts':[{'adress':'Y','email':None}],'application_details':{'email':'testy@test_a.com','phone':None},'employer':{'Name':'Nom','email':None},'id':'3'}]df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value0application_contacts0adress(0,application_contacts,0,adress)Xemail(0,application_contacts,0,email)test@test.comapplication_detailsemailNaN(0,application_details,email)NonephoneNaN(0,application_details,phone)NoneemployerNameNaN(0,employer,Name)NomemailNaN(0,employer,email)NoneidNaNNaN(0,id)11application_contacts0adress(1,application_contacts,0,adress)Zemail(1,application_contacts,0,email)Noneapplication_detailsemailNaN(1,application_details,email)testy@test_a.comphoneNaN(1,application_details,phone)NoneemployerNameNaN(1,employer,Name)NomemailNaN(1,employer,email)NoneidNaNNaN(1,id)22application_contacts0adress(2,application_contacts,0,adress)Yemail(2,application_contacts,0,email)Noneapplication_detailsemailNaN(2,application_details,email)testy@test_a.comphoneNaN(2,application_details,phone)NoneemployerNameNaN(2,employer,Name)NomemailNaN(2,employer,email)NoneidNaNNaN(2,id)3df.loc[df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'test_a\.\w+\b',na=False),'aa_value']='UPPPPPPPPPPPPPPPDATE.COM'aa_all_keysaa_value0application_contacts0adress(0,application_contacts,0,adress)Xemail(0,application_contacts,0,email)test@test.comapplication_detailsemailNaN(0,application_details,email)NonephoneNaN(0,application_details,phone)NoneemployerNameNaN(0,employer,Name)NomemailNaN(0,employer,email)NoneidNaNNaN(0,id)11application_contacts0adress(1,application_contacts,0,adress)Zemail(1,application_contacts,0,email)Noneapplication_detailsemailNaN(1,application_details,email)UPPPPPPPPPPPPPPPDATE.COMphoneNaN(1,application_details,phone)NoneemployerNameNaN(1,employer,Name)NomemailNaN(1,employer,email)NoneidNaNNaN(1,id)22application_contacts0adress(2,application_contacts,0,adress)Yemail(2,application_contacts,0,email)Noneapplication_detailsemailNaN(2,application_details,email)UPPPPPPPPPPPPPPPDATE.COMphoneNaN(2,application_details,phone)NoneemployerNameNaN(2,employer,Name)NomemailNaN(2,employer,email)NoneidNaNNaN(2,id)3mod_iter=df.d_update_original_iter(data,verbose=True)[1][application_details][email]Oldvalue:testy@test_a.com[1][application_details][email]Updatedvalue:UPPPPPPPPPPPPPPPDATE.COM[2][application_details][email]Oldvalue:testy@test_a.com[2][application_details][email]Updatedvalue:UPPPPPPPPPPPPPPPDATE.COM[{'application_contacts':[{'adress':'X','email':'test@test.com'}],'application_details':{'email':None,'phone':None},'employer':{'Name':'Nom','email':None},'id':'1'},{'application_contacts':[{'adress':'Z','email':None}],'application_details':{'email':'UPPPPPPPPPPPPPPPDATE.COM','phone':None},'employer':{'Name':'Nom','email':None},'id':'2'},{'application_contacts':[{'adress':'Y','email':None}],'application_details':{'email':'UPPPPPPPPPPPPPPPDATE.COM','phone':None},'employer':{'Name':'Nom','email':None},'id':'3'}]#Nested iterable from:https://stackoverflow.com/questions/62765371/convert-nested-dataframe-to-a-simple-dataframeframedata={'A':[1,2,3],'B':[4,5,6],'departure':[{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'}]}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valueA0NaN(A,0)11NaN(A,1)22NaN(A,2)3B0NaN(B,0)41NaN(B,1)52NaN(B,2)6departure0actual(departure,0,actual)Noneactual_runway(departure,0,actual_runway)Noneairport(departure,0,airport)Findeldelay(departure,0,delay)Noneestimated(departure,0,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,0,estimated_runway)Nonegate(departure,0,gate)Noneiata(departure,0,iata)LUXicao(departure,0,icao)ELLXscheduled(departure,0,scheduled)2020-07-07T06:30:00+00:00terminal(departure,0,terminal)Nonetimezone(departure,0,timezone)Europe/Luxembourg1actual(departure,1,actual)Noneactual_runway(departure,1,actual_runway)Noneairport(departure,1,airport)Findeldelay(departure,1,delay)Noneestimated(departure,1,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,1,estimated_runway)Nonegate(departure,1,gate)Noneiata(departure,1,iata)LUXicao(departure,1,icao)ELLXscheduled(departure,1,scheduled)2020-07-07T06:30:00+00:00terminal(departure,1,terminal)Nonetimezone(departure,1,timezone)Europe/Luxembourg2actual(departure,2,actual)Noneactual_runway(departure,2,actual_runway)Noneairport(departure,2,airport)Findeldelay(departure,2,delay)Noneestimated(departure,2,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,2,estimated_runway)Nonegate(departure,2,gate)Noneiata(departure,2,iata)LUXicao(departure,2,icao)ELLXscheduled(departure,2,scheduled)2020-07-07T06:30:00+00:00terminal(departure,2,terminal)Nonetimezone(departure,2,timezone)Europe/Luxembourgdf.loc[df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value')=='ELLX','aa_value']='ELLX-UPDATED'aa_all_keysaa_valueA0NaN(A,0)11NaN(A,1)22NaN(A,2)3B0NaN(B,0)41NaN(B,1)52NaN(B,2)6departure0actual(departure,0,actual)Noneactual_runway(departure,0,actual_runway)Noneairport(departure,0,airport)Findeldelay(departure,0,delay)Noneestimated(departure,0,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,0,estimated_runway)Nonegate(departure,0,gate)Noneiata(departure,0,iata)LUXicao(departure,0,icao)ELLX-UPDATEDscheduled(departure,0,scheduled)2020-07-07T06:30:00+00:00terminal(departure,0,terminal)Nonetimezone(departure,0,timezone)Europe/Luxembourg1actual(departure,1,actual)Noneactual_runway(departure,1,actual_runway)Noneairport(departure,1,airport)Findeldelay(departure,1,delay)Noneestimated(departure,1,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,1,estimated_runway)Nonegate(departure,1,gate)Noneiata(departure,1,iata)LUXicao(departure,1,icao)ELLX-UPDATEDscheduled(departure,1,scheduled)2020-07-07T06:30:00+00:00terminal(departure,1,terminal)Nonetimezone(departure,1,timezone)Europe/Luxembourg2actual(departure,2,actual)Noneactual_runway(departure,2,actual_runway)Noneairport(departure,2,airport)Findeldelay(departure,2,delay)Noneestimated(departure,2,estimated)2020-07-07T06:30:00+00:00estimated_runway(departure,2,estimated_runway)Nonegate(departure,2,gate)Noneiata(departure,2,iata)LUXicao(departure,2,icao)ELLX-UPDATEDscheduled(departure,2,scheduled)2020-07-07T06:30:00+00:00terminal(departure,2,terminal)Nonetimezone(departure,2,timezone)Europe/Luxembourgmod_iter=df.d_update_original_iter(data,verbose=True)[departure][0][icao]Oldvalue:ELLX[departure][0][icao]Updatedvalue:ELLX-UPDATED[departure][1][icao]Oldvalue:ELLX[departure][1][icao]Updatedvalue:ELLX-UPDATED[departure][2][icao]Oldvalue:ELLX[departure][2][icao]Updatedvalue:ELLX-UPDATED{'A':[1,2,3],'B':[4,5,6],'departure':[{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX-UPDATED','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX-UPDATED','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'},{'actual':None,'actual_runway':None,'airport':'Findel','delay':None,'estimated':'2020-07-07T06:30:00+00:00','estimated_runway':None,'gate':None,'iata':'LUX','icao':'ELLX-UPDATED','scheduled':'2020-07-07T06:30:00+00:00','terminal':None,'timezone':'Europe/Luxembourg'}]}#Nested iterable from:https://stackoverflow.com/questions/64359762/constructing-a-pandas-dataframe-with-columns-and-sub-columns-from-nested-dictiondata={'level1':{'t1':{'s1':{'col1':5,'col2':4,'col3':4,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':8},'s3':{'col1':11,'col2':8,'col3':2,'col4':9},'s4':{'col1':5,'col2':4,'col3':4,'col4':9}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':8},'s3':{'col1':11,'col2':8,'col3':2,'col4':9},'s4':{'col1':5,'col2':4,'col3':4,'col4':9}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':6,'col3':7,'col4':8},'s3':{'col1':9,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}},'level2':{'t1':{'s1':{'col1':5,'col2':4,'col3':9,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':5},'s3':{'col1':11,'col2':8,'col3':2,'col4':13},'s4':{'col1':5,'col2':4,'col3':4,'col4':20}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':9},'s2':{'col1':1,'col2':5,'col3':4,'col4':8},'s3':{'col1':11,'col2':8,'col3':2,'col4':9},'s4':{'col1':5,'col2':4,'col3':4,'col4':9}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':6,'col3':7,'col4':8},'s3':{'col1':9,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}}}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valuelevel1t1s1col1(level1,t1,s1,col1)5col2(level1,t1,s1,col2)4col3(level1,t1,s1,col3)4col4(level1,t1,s1,col4)9s2col1(level1,t1,s2,col1)1......level2t3s3col4(level2,t3,s3,col4)12s4col1(level2,t3,s4,col1)13col2(level2,t3,s4,col2)14col3(level2,t3,s4,col3)15col4(level2,t3,s4,col4)16[96rowsx2columns]df.loc[(df.d_filter_dtypes(allowed_dtypes=(int),fillvalue=pd.NA,column='aa_value')>5)&(df.d_filter_dtypes(allowed_dtypes=(int),fillvalue=pd.NA,column='aa_value')<10),'aa_value']=1000000aa_all_keysaa_valuelevel1t1s1col1(level1,t1,s1,col1)5col2(level1,t1,s1,col2)4col3(level1,t1,s1,col3)4col4(level1,t1,s1,col4)1000000s2col1(level1,t1,s2,col1)1......level2t3s3col4(level2,t3,s3,col4)12s4col1(level2,t3,s4,col1)13col2(level2,t3,s4,col2)14col3(level2,t3,s4,col3)15col4(level2,t3,s4,col4)16[96rowsx2columns]mod_iter=df.d_update_original_iter(data,verbose=True)[level1][t1][s1][col4]Oldvalue:9[level1][t1][s1][col4]Updatedvalue:1000000[level1][t1][s2][col4]Oldvalue:8[level1][t1][s2][col4]Updatedvalue:1000000[level1][t1][s3][col2]Oldvalue:8[level1][t1][s3][col2]Updatedvalue:1000000[level1][t1][s3][col4]Oldvalue:9[level1][t1][s3][col4]Updatedvalue:1000000[level1][t1][s4][col4]Oldvalue:9[level1][t1][s4][col4]Updatedvalue:1000000[level1][t2][s1][col4]Oldvalue:9[level1][t2][s1][col4]Updatedvalue:1000000[level1][t2][s2][col4]Oldvalue:8[level1][t2][s2][col4]Updatedvalue:1000000[level1][t2][s3][col2]Oldvalue:8[level1][t2][s3][col2]Updatedvalue:1000000[level1][t2][s3][col4]Oldvalue:9[level1][t2][s3][col4]Updatedvalue:1000000[level1][t2][s4][col4]Oldvalue:9[level1][t2][s4][col4]Updatedvalue:1000000[level1][t3][s2][col2]Oldvalue:6[level1][t3][s2][col2]Updatedvalue:1000000[level1][t3][s2][col3]Oldvalue:7[level1][t3][s2][col3]Updatedvalue:1000000[level1][t3][s2][col4]Oldvalue:8[level1][t3][s2][col4]Updatedvalue:1000000[level1][t3][s3][col1]Oldvalue:9[level1][t3][s3][col1]Updatedvalue:1000000[level2][t1][s1][col3]Oldvalue:9[level2][t1][s1][col3]Updatedvalue:1000000[level2][t1][s1][col4]Oldvalue:9[level2][t1][s1][col4]Updatedvalue:1000000[level2][t1][s3][col2]Oldvalue:8[level2][t1][s3][col2]Updatedvalue:1000000[level2][t2][s1][col4]Oldvalue:9[level2][t2][s1][col4]Updatedvalue:1000000[level2][t2][s2][col4]Oldvalue:8[level2][t2][s2][col4]Updatedvalue:1000000[level2][t2][s3][col2]Oldvalue:8[level2][t2][s3][col2]Updatedvalue:1000000[level2][t2][s3][col4]Oldvalue:9[level2][t2][s3][col4]Updatedvalue:1000000[level2][t2][s4][col4]Oldvalue:9[level2][t2][s4][col4]Updatedvalue:1000000[level2][t3][s2][col2]Oldvalue:6[level2][t3][s2][col2]Updatedvalue:1000000[level2][t3][s2][col3]Oldvalue:7[level2][t3][s2][col3]Updatedvalue:1000000[level2][t3][s2][col4]Oldvalue:8[level2][t3][s2][col4]Updatedvalue:1000000[level2][t3][s3][col1]Oldvalue:9[level2][t3][s3][col1]Updatedvalue:1000000{'level1':{'t1':{'s1':{'col1':5,'col2':4,'col3':4,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':1000000},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':1000000},'s4':{'col1':5,'col2':4,'col3':4,'col4':1000000}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':1000000},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':1000000},'s4':{'col1':5,'col2':4,'col3':4,'col4':1000000}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':1000000,'col3':1000000,'col4':1000000},'s3':{'col1':1000000,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}},'level2':{'t1':{'s1':{'col1':5,'col2':4,'col3':1000000,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':5},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':13},'s4':{'col1':5,'col2':4,'col3':4,'col4':20}},'t2':{'s1':{'col1':5,'col2':4,'col3':4,'col4':1000000},'s2':{'col1':1,'col2':5,'col3':4,'col4':1000000},'s3':{'col1':11,'col2':1000000,'col3':2,'col4':1000000},'s4':{'col1':5,'col2':4,'col3':4,'col4':1000000}},'t3':{'s1':{'col1':1,'col2':2,'col3':3,'col4':4},'s2':{'col1':5,'col2':1000000,'col3':1000000,'col4':1000000},'s3':{'col1':1000000,'col2':10,'col3':11,'col4':12},'s4':{'col1':13,'col2':14,'col3':15,'col4':16}}}}#Nested iterable from:https://stackoverflow.com/questions/72146094/problems-matching-values-from-nested-dictionarydata={'_links':{'next':None,'prev':None},'limit':250,'offset':0,'runs':[{'assignedto_id':None,'blocked_count':0,'completed_on':None,'config':None,'config_ids':[],'created_by':1,'created_on':1651790693,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':1,'id':13,'include_all':False,'is_completed':False,'milestone_id':None,'name':'2022-05-05-testrun','passed_count':2,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':0,'updated_on':1651790693,'url':'https://xxxxxxxxxx.testrail.io/index.php?/runs/view/13'},{'assignedto_id':None,'blocked_count':0,'completed_on':1650989972,'config':None,'config_ids':[],'created_by':5,'created_on':1650966329,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':0,'id':9,'include_all':False,'is_completed':True,'milestone_id':None,'name':'This is a new test run','passed_count':0,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':3,'updated_on':1650966329,'url':'https://xxxxxxxxxx.testrail.io/index.php?/runs/view/9'}],'size':2}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value_linksnextNaN(_links,next)NoneprevNaN(_links,prev)NonelimitNaNNaN(limit,)250offsetNaNNaN(offset,)0runs0assignedto_id(runs,0,assignedto_id)None......1suite_id(runs,1,suite_id)1untested_count(runs,1,untested_count)3updated_on(runs,1,updated_on)1650966329url(runs,1,url)https://xxxxxxxxxx.testrail.io/index.php?/runs...sizeNaNNaN(size,)2[63rowsx2columns]df.loc[(df.d_filter_dtypes(allowed_dtypes=(bool),fillvalue=pd.NA,column='aa_value')==False),'aa_value']=Truedf.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'https?://.*',na=False)),'aa_value']='WWW.PYTHON.ORG'aa_all_keysaa_value_linksnextNaN(_links,next)NoneprevNaN(_links,prev)NonelimitNaNNaN(limit,)250offsetNaNNaN(offset,)0runs0assignedto_id(runs,0,assignedto_id)None......1suite_id(runs,1,suite_id)1untested_count(runs,1,untested_count)3updated_on(runs,1,updated_on)1650966329url(runs,1,url)WWW.PYTHON.ORGsizeNaNNaN(size,)2[63rowsx2columns]mod_iter=df.d_update_original_iter(data,verbose=True)[runs][0][include_all]Oldvalue:False[runs][0][include_all]Updatedvalue:True[runs][0][is_completed]Oldvalue:False[runs][0][is_completed]Updatedvalue:True[runs][0][url]Oldvalue:https://xxxxxxxxxx.testrail.io/index.php?/runs/view/13[runs][0][url]Updatedvalue:WWW.PYTHON.ORG[runs][1][include_all]Oldvalue:False[runs][1][include_all]Updatedvalue:True[runs][1][url]Oldvalue:https://xxxxxxxxxx.testrail.io/index.php?/runs/view/9[runs][1][url]Updatedvalue:WWW.PYTHON.ORG{'_links':{'next':None,'prev':None},'limit':250,'offset':0,'runs':[{'assignedto_id':None,'blocked_count':0,'completed_on':None,'config':None,'config_ids':[],'created_by':1,'created_on':1651790693,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':1,'id':13,'include_all':True,'is_completed':True,'milestone_id':None,'name':'2022-05-05-testrun','passed_count':2,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':0,'updated_on':1651790693,'url':'WWW.PYTHON.ORG'},{'assignedto_id':None,'blocked_count':0,'completed_on':1650989972,'config':None,'config_ids':[],'created_by':5,'created_on':1650966329,'custom_status1_count':0,'custom_status2_count':0,'custom_status3_count':0,'custom_status4_count':0,'custom_status5_count':0,'custom_status6_count':0,'custom_status7_count':0,'description':None,'failed_count':0,'id':9,'include_all':True,'is_completed':True,'milestone_id':None,'name':'This is a new test run','passed_count':0,'plan_id':None,'project_id':1,'refs':None,'retest_count':0,'suite_id':1,'untested_count':3,'updated_on':1650966329,'url':'WWW.PYTHON.ORG'}],'size':2}#Nested iterable from:https://stackoverflow.com/questions/73708706/how-to-get-values-from-list-of-nested-dictionaries/73839430#73839430data={'results':[{'end_time':'2021-01-21','key':'q1','result_type':'multipleChoice','start_time':'2021-01-21','value':['1']},{'end_time':'2021-01-21','key':'q2','result_type':'multipleChoice','start_time':'2021-01-21','value':['False']},{'end_time':'2021-01-21','key':'q3','result_type':'multipleChoice','start_time':'2021-01-21','value':['3']},{'end_time':'2021-01-21','key':'q4','result_type':'multipleChoice','start_time':'2021-01-21','value':['3']}]}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valueresults0end_timeNaN(results,0,end_time)2021-01-21keyNaN(results,0,key)q1result_typeNaN(results,0,result_type)multipleChoicestart_timeNaN(results,0,start_time)2021-01-21value0(results,0,value,0)11end_timeNaN(results,1,end_time)2021-01-21keyNaN(results,1,key)q2result_typeNaN(results,1,result_type)multipleChoicestart_timeNaN(results,1,start_time)2021-01-21value0(results,1,value,0)False2end_timeNaN(results,2,end_time)2021-01-21keyNaN(results,2,key)q3result_typeNaN(results,2,result_type)multipleChoicestart_timeNaN(results,2,start_time)2021-01-21value0(results,2,value,0)33end_timeNaN(results,3,end_time)2021-01-21keyNaN(results,3,key)q4result_typeNaN(results,3,result_type)multipleChoicestart_timeNaN(results,3,start_time)2021-01-21value0(results,3,value,0)3df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'^2021.*',na=False)),'aa_value']=10000000000aa_all_keysaa_valueresults0end_timeNaN(results,0,end_time)10000000000keyNaN(results,0,key)q1result_typeNaN(results,0,result_type)multipleChoicestart_timeNaN(results,0,start_time)10000000000value0(results,0,value,0)11end_timeNaN(results,1,end_time)10000000000keyNaN(results,1,key)q2result_typeNaN(results,1,result_type)multipleChoicestart_timeNaN(results,1,start_time)10000000000value0(results,1,value,0)False2end_timeNaN(results,2,end_time)10000000000keyNaN(results,2,key)q3result_typeNaN(results,2,result_type)multipleChoicestart_timeNaN(results,2,start_time)10000000000value0(results,2,value,0)33end_timeNaN(results,3,end_time)10000000000keyNaN(results,3,key)q4result_typeNaN(results,3,result_type)multipleChoicestart_timeNaN(results,3,start_time)10000000000value0(results,3,value,0)3mod_iter=df.d_update_original_iter(data,verbose=True)[results][0][end_time]Oldvalue:2021-01-21[results][0][end_time]Updatedvalue:10000000000[results][0][start_time]Oldvalue:2021-01-21[results][0][start_time]Updatedvalue:10000000000[results][1][end_time]Oldvalue:2021-01-21[results][1][end_time]Updatedvalue:10000000000[results][1][start_time]Oldvalue:2021-01-21[results][1][start_time]Updatedvalue:10000000000[results][2][end_time]Oldvalue:2021-01-21[results][2][end_time]Updatedvalue:10000000000[results][2][start_time]Oldvalue:2021-01-21[results][2][start_time]Updatedvalue:10000000000[results][3][end_time]Oldvalue:2021-01-21[results][3][end_time]Updatedvalue:10000000000[results][3][start_time]Oldvalue:2021-01-21[results][3][start_time]Updatedvalue:10000000000{'results':[{'end_time':10000000000,'key':'q1','result_type':'multipleChoice','start_time':10000000000,'value':['1']},{'end_time':10000000000,'key':'q2','result_type':'multipleChoice','start_time':10000000000,'value':['False']},{'end_time':10000000000,'key':'q3','result_type':'multipleChoice','start_time':10000000000,'value':['3']},{'end_time':10000000000,'key':'q4','result_type':'multipleChoice','start_time':10000000000,'value':['3']}]}#Nested iterable from:https://stackoverflow.com/questions/66461902/flattening-nested-dictionary-into-dataframe-pythondata={1:{2:{'IDs':{'BookID':['543533254353','4324232342'],'SalesID':['543267765345','4353543'],'StoreID':['111111','1121111']},'Name':'boring Tales of Dragon Slayers'},'IDs':{'BookID':['543533254353'],'SalesID':['543267765345'],'StoreID':['123445452543']},'Name':'Thrilling Tales of Dragon Slayers'}}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value1IDsBookID0NaN(1,IDs,BookID,0)543533254353SalesID0NaN(1,IDs,SalesID,0)543267765345StoreID0NaN(1,IDs,StoreID,0)123445452543NameNaNNaNNaN(1,Name)ThrillingTalesofDragonSlayers2IDsBookID0(1,2,IDs,BookID,0)5435332543531(1,2,IDs,BookID,1)4324232342SalesID0(1,2,IDs,SalesID,0)5432677653451(1,2,IDs,SalesID,1)4353543StoreID0(1,2,IDs,StoreID,0)1111111(1,2,IDs,StoreID,1)1121111NameNaNNaN(1,2,Name)boringTalesofDragonSlayersdf.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'^\d+$',na=False)),'aa_value']=df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='aa_value').str.contains(r'^\d+$',na=False)),'aa_value'].astype(float)aa_all_keysaa_value1IDsBookID0NaN(1,IDs,BookID,0)543533254353.0SalesID0NaN(1,IDs,SalesID,0)543267765345.0StoreID0NaN(1,IDs,StoreID,0)123445452543.0NameNaNNaNNaN(1,Name)ThrillingTalesofDragonSlayers2IDsBookID0(1,2,IDs,BookID,0)543533254353.01(1,2,IDs,BookID,1)4324232342.0SalesID0(1,2,IDs,SalesID,0)543267765345.01(1,2,IDs,SalesID,1)4353543.0StoreID0(1,2,IDs,StoreID,0)111111.01(1,2,IDs,StoreID,1)1121111.0NameNaNNaN(1,2,Name)boringTalesofDragonSlayersmod_iter=df.d_update_original_iter(data,verbose=True)[1][2][IDs][BookID][0]Oldvalue:543533254353[1][2][IDs][BookID][0]Updatedvalue:543533254353.0[1][2][IDs][BookID][1]Oldvalue:4324232342[1][2][IDs][BookID][1]Updatedvalue:4324232342.0[1][2][IDs][SalesID][0]Oldvalue:543267765345[1][2][IDs][SalesID][0]Updatedvalue:543267765345.0[1][2][IDs][SalesID][1]Oldvalue:4353543[1][2][IDs][SalesID][1]Updatedvalue:4353543.0[1][2][IDs][StoreID][0]Oldvalue:111111[1][2][IDs][StoreID][0]Updatedvalue:111111.0[1][2][IDs][StoreID][1]Oldvalue:1121111[1][2][IDs][StoreID][1]Updatedvalue:1121111.0[1][IDs][BookID][0]Oldvalue:543533254353[1][IDs][BookID][0]Updatedvalue:543533254353.0[1][IDs][SalesID][0]Oldvalue:543267765345[1][IDs][SalesID][0]Updatedvalue:543267765345.0[1][IDs][StoreID][0]Oldvalue:123445452543[1][IDs][StoreID][0]Updatedvalue:123445452543.0{1:{2:{'IDs':{'BookID':[543533254353.0,4324232342.0],'SalesID':[543267765345.0,4353543.0],'StoreID':[111111.0,1121111.0]},'Name':'boring Tales of Dragon Slayers'},'IDs':{'BookID':[543533254353.0],'SalesID':[543267765345.0],'StoreID':[123445452543.0]},'Name':'Thrilling Tales of Dragon Slayers'}}#Nested iterable from:https://stackoverflow.com/questions/72017771/key-error-when-accessing-a-nested-dictionarydata=[{'blocks':[{'block_id':'BJNTn','text':{'text':'You have a new message.','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'WPn/l','text':{'text':'*Heard By*\nFriend','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'5yp','text':{'text':'*Which Direction? *\nNorth','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'fKEpF','text':{'text':'*Which Destination*\nNew York','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'qjAH','text':{'text':'*New Customer:*\\Yes','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'yt4','elements':[{'action_id':'+bc','text':{'bar':'View results','emoji':True,'type':'plain_text'},'type':'button','url':'www.example.com/results'}],'type':'actions'},{'block_id':'IBr','text':{'text':' ','type':'mrkdwn','verbatim':False},'type':'section'}],'bot_id':'BPD4K3SJW','subtype':'bot_message','text':"This content can't be displayed.",'timestamp':'1650905606.755969','type':'message','username':'admin'},{'blocks':[{'block_id':'Smd','text':{'text':'You have a new message.','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'6YaLt','text':{'text':'*Heard By*\nOnline Search','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'w3o','text':{'text':'*Which Direction: *\nNorth','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'PTQ','text':{'text':'*Which Destination? *\nMiami','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'JCfSP','text':{'text':'*New Customer? *\nNo','type':'mrkdwn','verbatim':False},'type':'section'},{'block_id':'yt4','elements':[{'action_id':'+bc','text':{'bar':'View results','emoji':True,'type':'plain_text'},'type':'button','url':'www.example.com/results'}],'type':'actions'},{'block_id':'RJOA','text':{'text':' ','type':'mrkdwn','verbatim':False},'type':'section'}],'bot_id':'BPD4K3SJW','subtype':'bot_message','text':"This content can't be displayed.",'timestamp':'1650899428.077709','type':'message','username':'admin'}]df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value0blocks0.0block_idNaNNaNNaN(0,blocks,0,block_id)BJNTntexttextNaNNaN(0,blocks,0,text,text)Youhaveanewmessage.typeNaNNaN(0,blocks,0,text,type)mrkdwnverbatimNaNNaN(0,blocks,0,text,verbatim)FalsetypeNaNNaNNaN(0,blocks,0,type)section......1subtypeNaNNaNNaNNaNNaN(1,subtype)bot_messagetextNaNNaNNaNNaNNaN(1,text)Thiscontentcan't be displayed.timestampNaNNaNNaNNaNNaN(1,timestamp)1650899428.077709typeNaNNaNNaNNaNNaN(1,type)messageusernameNaNNaNNaNNaNNaN(1,username)admin[88rowsx2columns]df.loc[(df.d_filter_dtypes(allowed_dtypes=(bool),fillvalue=pd.NA,column='aa_value')==False),'aa_value']='NOOOOOOOOOOOOOOOOOOO MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'aa_all_keysaa_value0blocks0.0block_idNaNNaNNaN(0,blocks,0,block_id)BJNTntexttextNaNNaN(0,blocks,0,text,text)Youhaveanewmessage.typeNaNNaN(0,blocks,0,text,type)mrkdwnverbatimNaNNaN(0,blocks,0,text,verbatim)NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREB...typeNaNNaNNaN(0,blocks,0,type)section......1subtypeNaNNaNNaNNaNNaN(1,subtype)bot_messagetextNaNNaNNaNNaNNaN(1,text)Thiscontentcan't be displayed.timestampNaNNaNNaNNaNNaN(1,timestamp)1650899428.077709typeNaNNaNNaNNaNNaN(1,type)messageusernameNaNNaNNaNNaNNaN(1,username)admin[88rowsx2columns]mod_iter=df.d_update_original_iter(data,verbose=True)[0][blocks][0][text][verbatim]Oldvalue:False[0][blocks][0][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[0][blocks][1][text][verbatim]Oldvalue:False[0][blocks][1][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[0][blocks][2][text][verbatim]Oldvalue:False[0][blocks][2][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[0][blocks][3][text][verbatim]Oldvalue:False[0][blocks][3][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[0][blocks][4][text][verbatim]Oldvalue:False[0][blocks][4][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[0][blocks][6][text][verbatim]Oldvalue:False[0][blocks][6][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[1][blocks][0][text][verbatim]Oldvalue:False[1][blocks][0][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[1][blocks][1][text][verbatim]Oldvalue:False[1][blocks][1][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[1][blocks][2][text][verbatim]Oldvalue:False[1][blocks][2][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[1][blocks][3][text][verbatim]Oldvalue:False[1][blocks][3][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[1][blocks][4][text][verbatim]Oldvalue:False[1][blocks][4][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[1][blocks][6][text][verbatim]Oldvalue:False[1][blocks][6][text][verbatim]Updatedvalue:NOOOOOOOOOOOOOOOOOOOMOOOOOOOOOOOOOOOOOOOOREBOOOOOOOOOOOOOOOL[{'blocks':[{'block_id':'BJNTn','text':{'text':'You have a new message.','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'WPn/l','text':{'text':'*Heard By*\nFriend','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'5yp','text':{'text':'*Which Direction? *\nNorth','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'fKEpF','text':{'text':'*Which Destination*\nNew York','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'qjAH','text':{'text':'*New Customer:*\\Yes','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'yt4','elements':[{'action_id':'+bc','text':{'bar':'View results','emoji':True,'type':'plain_text'},'type':'button','url':'www.example.com/results'}],'type':'actions'},{'block_id':'IBr','text':{'text':' ','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'}],'bot_id':'BPD4K3SJW','subtype':'bot_message','text':"This content can't be displayed.",'timestamp':'1650905606.755969','type':'message','username':'admin'},{'blocks':[{'block_id':'Smd','text':{'text':'You have a new message.','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'6YaLt','text':{'text':'*Heard By*\nOnline Search','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'w3o','text':{'text':'*Which Direction: *\nNorth','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'PTQ','text':{'text':'*Which Destination? *\nMiami','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'JCfSP','text':{'text':'*New Customer? *\nNo','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'},{'block_id':'yt4','elements':[{'action_id':'+bc','text':{'bar':'View results','emoji':True,'type':'plain_text'},'type':'button','url':'www.example.com/results'}],'type':'actions'},{'block_id':'RJOA','text':{'text':' ','type':'mrkdwn','verbatim':'NOOOOOOOOOOOOOOOOOOO ''MOOOOOOOOOOOOOOOOOOOORE BOOOOOOOOOOOOOOOL'},'type':'section'}],'bot_id':'BPD4K3SJW','subtype':'bot_message','text':"This content can't be displayed.",'timestamp':'1650899428.077709','type':'message','username':'admin'}]#Nested iterable from:https://stackoverflow.com/questions/73643077/how-to-transform-a-list-of-nested-dictionaries-into-a-data-frame-pd-json-normaldata=[{'apple':{'price':4,'units':3}},{'banana':{'price':2,'units':20}},{'orange':{'price':5,'units':15}}]df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_value0appleprice(0,apple,price)4units(0,apple,units)31bananaprice(1,banana,price)2units(1,banana,units)202orangeprice(2,orange,price)5units(2,orange,units)15df.loc[(df.d_filter_dtypes(allowed_dtypes=(int),fillvalue=pd.NA,column='aa_value')>3)&(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_1').str.contains("banana")),'aa_value']=50000level_0level_1level_2aa_all_keysaa_value00appleprice(0,apple,price)410appleunits(0,apple,units)321bananaprice(1,banana,price)231bananaunits(1,banana,units)5000042orangeprice(2,orange,price)552orangeunits(2,orange,units)15mod_iter=df.d_update_original_iter(data,verbose=True)[1][banana][units]Oldvalue:20[1][banana][units]Updatedvalue:50000[{'apple':{'price':4,'units':3}},{'banana':{'price':2,'units':50000}},{'orange':{'price':5,'units':15}}]#Nested iterable from:https://stackoverflow.com/questions/58110440/opening-nested-dict-in-a-single-column-to-multiple-columns-in-pandasdata={'simple25b':{'hands':{'0':{'currency':'rm','handId':'xyz','time':'2019-09-23 11:00:01'},'1':{'currency':'rm','handId':'abc','time':'2019-09-23 11:01:18'}}},'simple5af':{'hands':{'0':{'currency':'rm','handId':'akg','time':'2019-09-23 10:53:22'},'1':{'currency':'rm','handId':'mzc','time':'2019-09-23 10:54:15'},'2':{'currency':'rm','handId':'swk','time':'2019-09-23 10:56:03'},'3':{'currency':'rm','handId':'pQc','time':'2019-09-23 10:57:15'},'4':{'currency':'rm','handId':'ywh','time':'2019-09-23 10:58:53'}}}}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valuesimple25bhands0currency(simple25b,hands,0,currency)rmhandId(simple25b,hands,0,handId)xyztime(simple25b,hands,0,time)2019-09-2311:00:011currency(simple25b,hands,1,currency)rmhandId(simple25b,hands,1,handId)abctime(simple25b,hands,1,time)2019-09-2311:01:18simple5afhands0currency(simple5af,hands,0,currency)rmhandId(simple5af,hands,0,handId)akgtime(simple5af,hands,0,time)2019-09-2310:53:221currency(simple5af,hands,1,currency)rmhandId(simple5af,hands,1,handId)mzctime(simple5af,hands,1,time)2019-09-2310:54:152currency(simple5af,hands,2,currency)rmhandId(simple5af,hands,2,handId)swktime(simple5af,hands,2,time)2019-09-2310:56:033currency(simple5af,hands,3,currency)rmhandId(simple5af,hands,3,handId)pQctime(simple5af,hands,3,time)2019-09-2310:57:154currency(simple5af,hands,4,currency)rmhandId(simple5af,hands,4,handId)ywhtime(simple5af,hands,4,time)2019-09-2310:58:53df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_3').str.contains("time")),'aa_value']=pd.to_datetime(df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_3').str.contains("time")),'aa_value'])level_0level_1...aa_all_keysaa_value0simple25bhands...(simple25b,hands,0,currency)rm1simple25bhands...(simple25b,hands,0,handId)xyz2simple25bhands...(simple25b,hands,0,time)2019-09-2311:00:013simple25bhands...(simple25b,hands,1,currency)rm4simple25bhands...(simple25b,hands,1,handId)abc5simple25bhands...(simple25b,hands,1,time)2019-09-2311:01:186simple5afhands...(simple5af,hands,0,currency)rm7simple5afhands...(simple5af,hands,0,handId)akg8simple5afhands...(simple5af,hands,0,time)2019-09-2310:53:229simple5afhands...(simple5af,hands,1,currency)rm10simple5afhands...(simple5af,hands,1,handId)mzc11simple5afhands...(simple5af,hands,1,time)2019-09-2310:54:1512simple5afhands...(simple5af,hands,2,currency)rm13simple5afhands...(simple5af,hands,2,handId)swk14simple5afhands...(simple5af,hands,2,time)2019-09-2310:56:0315simple5afhands...(simple5af,hands,3,currency)rm16simple5afhands...(simple5af,hands,3,handId)pQc17simple5afhands...(simple5af,hands,3,time)2019-09-2310:57:1518simple5afhands...(simple5af,hands,4,currency)rm19simple5afhands...(simple5af,hands,4,handId)ywh20simple5afhands...(simple5af,hands,4,time)2019-09-2310:58:53[21rowsx6columns]mod_iter=df.d_update_original_iter(data,verbose=True){'simple25b':{'hands':{'0':{'currency':'rm','handId':'xyz','time':Timestamp('2019-09-23 11:00:01')},'1':{'currency':'rm','handId':'abc','time':Timestamp('2019-09-23 11:01:18')}}},'simple5af':{'hands':{'0':{'currency':'rm','handId':'akg','time':Timestamp('2019-09-23 10:53:22')},'1':{'currency':'rm','handId':'mzc','time':Timestamp('2019-09-23 10:54:15')},'2':{'currency':'rm','handId':'swk','time':Timestamp('2019-09-23 10:56:03')},'3':{'currency':'rm','handId':'pQc','time':Timestamp('2019-09-23 10:57:15')},'4':{'currency':'rm','handId':'ywh','time':Timestamp('2019-09-23 10:58:53')}}}}#Nested iterable from:https://stackoverflow.com/questions/62059970/how-can-i-convert-nested-dictionary-to-pd-dataframe-fasterdata={'file':'name','main':[{'answer':[{'comment':'It is defined as','user':'John','value':[{'my_value':5,'value_2':10},{'my_value':24,'value_2':30}]},{'comment':'as John said above it simply means','user':'Sam','value':[{'my_value':9,'value_2':10},{'my_value':54,'value_2':19}]}],'closed':'no','question':'what is ?','question_no':'Q.1'}]}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valuefileNaNNaNNaNNaNNaNNaN(file,)namemain0answer0commentNaNNaN(main,0,answer,0,comment)ItisdefinedasuserNaNNaN(main,0,answer,0,user)Johnvalue0my_value(main,0,answer,0,value,0,my_value)5value_2(main,0,answer,0,value,0,value_2)101my_value(main,0,answer,0,value,1,my_value)24value_2(main,0,answer,0,value,1,value_2)301commentNaNNaN(main,0,answer,1,comment)asJohnsaidaboveitsimplymeansuserNaNNaN(main,0,answer,1,user)Samvalue0my_value(main,0,answer,1,value,0,my_value)9value_2(main,0,answer,1,value,0,value_2)101my_value(main,0,answer,1,value,1,my_value)54value_2(main,0,answer,1,value,1,value_2)19closedNaNNaNNaNNaN(main,0,closed)noquestionNaNNaNNaNNaN(main,0,question)whatis?question_noNaNNaNNaNNaN(main,0,question_no)Q.1df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_6').str.contains("value_2",na=False)),'aa_value']=df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_6').str.contains("value_2",na=False)),'aa_value']*1000level_0...aa_value0file...name1main...Itisdefinedas2main...John3main...54main...100005main...246main...300007main...asJohnsaidaboveitsimplymeans8main...Sam9main...910main...1000011main...5412main...1900013main...no14main...whatis?15main...Q.1[16rowsx9columns]mod_iter=df.d_update_original_iter(data,verbose=True)[main][0][answer][0][value][0][value_2]Oldvalue:10[main][0][answer][0][value][0][value_2]Updatedvalue:10000[main][0][answer][0][value][1][value_2]Oldvalue:30[main][0][answer][0][value][1][value_2]Updatedvalue:30000[main][0][answer][1][value][0][value_2]Oldvalue:10[main][0][answer][1][value][0][value_2]Updatedvalue:10000[main][0][answer][1][value][1][value_2]Oldvalue:19[main][0][answer][1][value][1][value_2]Updatedvalue:19000{'file':'name','main':[{'answer':[{'comment':'It is defined as','user':'John','value':[{'my_value':5,'value_2':10000},{'my_value':24,'value_2':30000}]},{'comment':'as John said above it simply means','user':'Sam','value':[{'my_value':9,'value_2':10000},{'my_value':54,'value_2':19000}]}],'closed':'no','question':'what is ?','question_no':'Q.1'}]}#Nested iterable from:https://stackoverflow.com/questions/39634369/4-dimensional-nested-dictionary-to-pandas-data-framedata={'orders':[{'created_at':'2016-09-20T22:04:49+02:00','email':'test@aol.com','id':4314127108,'line_items':[{'destination_location':{'address1':'Teststreet ''12','address2':'','city':'Berlin','country_code':'DE','id':2383331012,'name':'Test Test','zip':'10117'},'gift_card':False,'name':'Blueberry Cup'},{'destination_location':{'address1':'Teststreet ''12','address2':'','city':'Berlin','country_code':'DE','id':2383331012,'name':'Test Test','zip':'10117'},'gift_card':False,'name':'Strawberry Cup'}]}]}df=pd.Q_AnyNestedIterable_2df(data,unstack=False)aa_all_keysaa_valueorders0created_atNaNNaNNaN(orders,0,created_at)2016-09-20T22:04:49+02:00emailNaNNaNNaN(orders,0,email)test@aol.comidNaNNaNNaN(orders,0,id)4314127108line_items0destination_locationaddress1(orders,0,line_items,0,destination_locatio...Teststreet12address2(orders,0,line_items,0,destination_locatio...city(orders,0,line_items,0,destination_locatio...Berlincountry_code(orders,0,line_items,0,destination_locatio...DEid(orders,0,line_items,0,destination_locatio...2383331012name(orders,0,line_items,0,destination_locatio...TestTestzip(orders,0,line_items,0,destination_locatio...10117gift_cardNaN(orders,0,line_items,0,gift_card)FalsenameNaN(orders,0,line_items,0,name)BlueberryCup1destination_locationaddress1(orders,0,line_items,1,destination_locatio...Teststreet12address2(orders,0,line_items,1,destination_locatio...city(orders,0,line_items,1,destination_locatio...Berlincountry_code(orders,0,line_items,1,destination_locatio...DEid(orders,0,line_items,1,destination_locatio...2383331012name(orders,0,line_items,1,destination_locatio...TestTestzip(orders,0,line_items,1,destination_locatio...10117gift_cardNaN(orders,0,line_items,1,gift_card)FalsenameNaN(orders,0,line_items,1,name)StrawberryCupdf.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_6').str.contains("value_2",na=False)),'aa_value']=df.loc[(df.d_filter_dtypes(allowed_dtypes=(str),fillvalue=pd.NA,column='level_6').str.contains("value_2",na=False)),'aa_value']*1000level_0...aa_value0orders...2016-09-20T22:04:49+02:001orders...test@aol.com2orders...43141271083orders...Teststreet124orders...5orders...FRANKFURTISTVIELBESSERALSBERLIN6orders...DE7orders...23833310128orders...TestTest9orders...1011710orders...False11orders...BlueberryCup12orders...Teststreet1213orders...14orders...FRANKFURTISTVIELBESSERALSBERLIN15orders...DE16orders...238333101217orders...TestTest18orders...1011719orders...False20orders...StrawberryCup[21rowsx8columns]mod_iter=df.d_update_original_iter(data,verbose=True)[orders][0][line_items][0][destination_location][city]Oldvalue:Berlin[orders][0][line_items][0][destination_location][city]Updatedvalue:FRANKFURTISTVIELBESSERALSBERLIN[orders][0][line_items][1][destination_location][city]Oldvalue:Berlin[orders][0][line_items][1][destination_location][city]Updatedvalue:FRANKFURTISTVIELBESSERALSBERLIN{'orders':[{'created_at':'2016-09-20T22:04:49+02:00','email':'test@aol.com','id':4314127108,'line_items':[{'destination_location':{'address1':'Teststreet ''12','address2':'','city':'FRANKFURT IST ''VIEL BESSER ALS ''BERLIN','country_code':'DE','id':2383331012,'name':'Test Test','zip':'10117'},'gift_card':False,'name':'Blueberry Cup'},{'destination_location':{'address1':'Teststreet ''12','address2':'','city':'FRANKFURT IST ''VIEL BESSER ALS ''BERLIN','country_code':'DE','id':2383331012,'name':'Test Test','zip':'10117'},'gift_card':False,'name':'Strawberry Cup'}]}]}df.s_delete_duplicates_from_iters_in_cellsdelete_duplicates_in_column_full_of_iters(df:pandas.core.series.Series)->pandas.core.series.Seriesdf=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})getdi=lambdax:[(randrange(1,4),randrange(1,4))forvinrange(20)]#create some random tuplesdf["dicttest"]=df.lkey.apply(lambdax:getdi(x))print(df)df["dicttest"]=df.dicttest.s_delete_duplicates_from_iters_in_cells()print(df)lkeyvaluedicttest0foo1[(2,1),(3,3),(3,3),(2,1),(1,2),(1,2...1bar2[(3,2),(1,1),(1,1),(1,2),(3,2),(1,2...2baz3[(1,2),(3,1),(2,1),(2,1),(1,1),(2,3...3foo5[(2,3),(2,3),(3,3),(2,2),(1,2),(1,2...lkeyvaluedicttest0foo1[(2,1),(3,3),(1,2),(1,3),(3,2),(2,3...1bar2[(3,2),(1,1),(1,2),(2,1),(3,1),(3,3...2baz3[(1,2),(3,1),(2,1),(1,1),(2,3),(3,3...3foo5[(2,3),(3,3),(2,2),(1,2),(1,1),(1,3...Parameters:df:pd.SeriesColumnwithduplicatesthataredifficulttohandleReturns:pd.Seriesdf.ds_explode_dicts_in_column()explode_dicts_in_column(df:pandas.core.frame.DataFrame,column_to_explode:str,drop_exploded_column:bool=True)->pandas.core.frame.DataFramedf=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})getdi=lambdax:{v:{v*randrange(1,10):v*randrange(1,10)}forvinrange((randrange(1,10)))}#create random nested dictsdf["dicttest"]=df.lkey.apply(getdi)print(df)print(df.ds_explode_dicts_in_column('dicttest'))lkeyvaluedicttest0foo1{0:{0:0},1:{1:7},2:{2:8},3:{3:18},...1bar2{0:{0:0},1:{9:4},2:{10:6},3:{3:21},...2baz3{0:{0:0},1:{9:7},2:{2:10},3:{21:27}...3foo5{0:{0:0}}lkeyvaluelevel_0level_1aa_all_keysaa_value0foo100(0,0)01foo111(1,1)72foo122(2,2)83foo133(3,3)184foo1432(4,32)165foo1535(5,35)156bar200(0,0)07bar219(1,9)48bar2210(2,10)69bar233(3,3)2110bar2424(4,24)3611baz300(0,0)012baz319(1,9)713baz322(2,2)1014baz3321(3,21)2715baz3428(4,28)2016baz3515(5,15)3017baz366(6,6)618baz3721(7,21)719baz3824(8,24)4820foo500(0,0)0Parameters:df:pd.DataFramepd.DataFramecolumn_to_explode:strcolumnwithdictincellsdrop_exploded_column:boolDropcolumnafterexploding(default=True)Returns:pd.DataFramedf.d_df_to_nested_dict()Parameters----------df:pd.DataFrameDataFrametoconvertgroupby:strcolumnwhosevalueswillbethetoplevelkeyssubkeys:listcolumnswholsevalueswillbethenestedkeysdf=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")NesteddictfromDataFrame:df[:5].d_df_to_nested_dict(groupby='Survived',subkeys=['PassengerId','Age','Pclass','Name','Sex']){0:{'PassengerId':{0:1,4:5},'Age':{0:22.0,4:35.0},'Pclass':{0:3,4:3},'Name':{0:'Braund, Mr. Owen Harris',4:'Allen, Mr. William Henry'},'Sex':{0:'male',4:'male'}},1:{'PassengerId':{1:2,2:3,3:4},'Age':{1:38.0,2:26.0,3:35.0},'Pclass':{1:1,2:3,3:1},'Name':{1:'Cumings, Mrs. John Bradley (Florence Briggs Thayer)',2:'Heikkinen, Miss. Laina',3:'Futrelle, Mrs. Jacques Heath (Lily May Peel)'},'Sex':{1:'female',2:'female',3:'female'}}}df[:5].d_df_to_nested_dict(groupby='Sex',subkeys=['PassengerId','Name'])Out[39]:{'male':{'PassengerId':{0:1,4:5},'Name':{0:'Braund, Mr. Owen Harris',4:'Allen, Mr. William Henry'}},'female':{'PassengerId':{1:2,2:3,3:4},'Name':{1:'Cumings, Mrs. John Bradley (Florence Briggs Thayer)',2:'Heikkinen, Miss. Laina',3:'Futrelle, Mrs. Jacques Heath (Lily May Peel)'}}}df.s_explode_lists_and_tuples()explode_lists_and_tuples_in_column(df:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],column:Optional[str]=None,concat_with_df:bool=False)->pandas.core.frame.DataFramedf=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})getdi=lambdax:[(randrange(1,4),randrange(1,4))forvinrange(randrange(1,5))]#create some random tuplesdf["dicttest"]=df.lkey.apply(lambdax:getdi(x))print(df)df1=df.s_explode_lists_and_tuples(column='dicttest',concat_with_df=True)print(df1)df2=df.s_explode_lists_and_tuples(column='dicttest',concat_with_df=False)print(df2)df3=df.dicttest.s_explode_lists_and_tuples(column=None)print(df3)lkeyvaluedicttest0foo1[(3,3)]1bar2[(2,3),(2,1),(2,2)]2baz3[(2,3),(2,3)]3foo5[(1,2)]lkeyvaluedicttestdicttest_0dicttest_1dicttest_20foo1[(3,3)](3,3)<NA><NA>1bar2[(2,3),(2,1),(2,2)](2,3)(2,1)(2,2)2baz3[(2,3),(2,3)](2,3)(2,3)<NA>3foo5[(1,2)](1,2)<NA><NA>dicttest_0dicttest_1dicttest_20(3,3)<NA><NA>1(2,3)(2,1)(2,2)2(2,3)(2,3)<NA>3(1,2)<NA><NA>dicttest_0dicttest_1dicttest_20(3,3)<NA><NA>1(2,3)(2,1)(2,2)2(2,3)(2,3)<NA>3(1,2)<NA><NA>Parameters:df:Union[pd.Series,pd.DataFrame]pd.Series,pd.DataFramewithlists/tuplesincellscolumn:Union[str,None]Nonecanonlybeusedifapd.Seriesispassed.IfaDataFrameispassed,acolumnneedstobepassedtoo.concat_with_df:boolifTrue->returnsdf+explodedSeriesasDataFrameifFalse->returnsexplodedSeriesasDataFrameReturns:pd.DataFrameMissingvaluesarefilledwithpd.NAdf.s_flatten_all_iters_in_cells()flatten_all_iters_in_cells(df:pandas.core.series.Series)->pandas.core.series.Seriesdf=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})getdi=lambdax:[(randrange(1,4),randrange(1,4))forvinrange(20)]#create some random tuplesdf["dicttest"]=df.lkey.apply(lambdax:getdi(x))print(df)df["dicttest"]=df.dicttest.s_flatten_all_iters_in_cells()print(df)lkeyvaluedicttest0foo1[(2,2),(3,3),(3,2),(1,3),(1,2),(2,2...1bar2[(1,1),(3,1),(1,3),(3,2),(3,1),(2,2...2baz3[(3,1),(1,1),(3,3),(1,3),(3,2),(3,3...3foo5[(3,3),(3,3),(3,2),(2,3),(3,3),(2,3...lkeyvaluedicttest0foo1[2,2,3,3,3,2,1,3,1,2,2,2,1,3,1,...1bar2[1,1,3,1,1,3,3,2,3,1,2,2,3,1,3,...2baz3[3,1,1,1,3,3,1,3,3,2,3,3,1,3,1,...3foo5[3,3,3,3,3,2,2,3,3,3,2,3,3,3,3,...Parameters:df:pd.SeriesColumnwithduplicatesthataredifficulttohandleReturns:pd.Seriesdf.d_multiple_columns_to_one()make_several_columns_fit_in_one(df:pandas.core.frame.DataFrame,columns:list)->listdf=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")[:20]print(df)df['Ticket_Fare_Embarked']=df.d_multiple_columns_to_one(columns=['Ticket','Fare','Embarked'])PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS5603Moran,Mr.Jamesmale...03308778.4583NaNQ6701McCarthy,Mr.TimothyJmale...01746351.8625E46S7803Palsson,Master.GostaLeonardmale...134990921.0750NaNS8913Johnson,Mrs.OscarW(ElisabethVilhelminaBerg)female...234774211.1333NaNS91012Nasser,Mrs.Nicholas(AdeleAchem)female...023773630.0708NaNC101113Sandstrom,Miss.MargueriteRutfemale...1PP954916.7000G6S111211Bonnell,Miss.Elizabethfemale...011378326.5500C103S121303Saundercock,Mr.WilliamHenrymale...0A/5.21518.0500NaNS131403Andersson,Mr.AndersJohanmale...534708231.2750NaNS141503Vestrom,Miss.HuldaAmandaAdolfinafemale...03504067.8542NaNS151612Hewlett,Mrs.(MaryDKingcome)female...024870616.0000NaNS161703Rice,Master.Eugenemale...138265229.1250NaNQ171812Williams,Mr.CharlesEugenemale...024437313.0000NaNS181903VanderPlanke,Mrs.Julius(EmeliaMariaVande...female...034576318.0000NaNS192013Masselmani,Mrs.Fatimafemale...026497.2250NaNC[20rowsx12columns]dfOut[30]:PassengerIdSurvivedPclassNameSex...TicketFareCabinEmbarkedTicket_Fare_Embarked0103Braund,Mr.OwenHarrismale...A/5211717.2500NaNS[A/521171,7.25,S]1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...PC1759971.2833C85C[PC17599,71.2833,C]2313Heikkinen,Miss.Lainafemale...STON/O2.31012827.9250NaNS[STON/O2.3101282,7.925,S]3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...11380353.1000C123S[113803,53.1,S]4503Allen,Mr.WilliamHenrymale...3734508.0500NaNS[373450,8.05,S]5603Moran,Mr.Jamesmale...3308778.4583NaNQ[330877,8.4583,Q]6701McCarthy,Mr.TimothyJmale...1746351.8625E46S[17463,51.8625,S]7803Palsson,Master.GostaLeonardmale...34990921.0750NaNS[349909,21.075,S]8913Johnson,Mrs.OscarW(ElisabethVilhelminaBerg)female...34774211.1333NaNS[347742,11.1333,S]91012Nasser,Mrs.Nicholas(AdeleAchem)female...23773630.0708NaNC[237736,30.0708,C]101113Sandstrom,Miss.MargueriteRutfemale...PP954916.7000G6S[PP9549,16.7,S]111211Bonnell,Miss.Elizabethfemale...11378326.5500C103S[113783,26.55,S]121303Saundercock,Mr.WilliamHenrymale...A/5.21518.0500NaNS[A/5.2151,8.05,S]131403Andersson,Mr.AndersJohanmale...34708231.2750NaNS[347082,31.275,S]141503Vestrom,Miss.HuldaAmandaAdolfinafemale...3504067.8542NaNS[350406,7.8542,S]151612Hewlett,Mrs.(MaryDKingcome)female...24870616.0000NaNS[248706,16.0,S]161703Rice,Master.Eugenemale...38265229.1250NaNQ[382652,29.125,Q]171812Williams,Mr.CharlesEugenemale...24437313.0000NaNS[244373,13.0,S]181903VanderPlanke,Mrs.Julius(EmeliaMariaVande...female...34576318.0000NaNS[345763,18.0,S]192013Masselmani,Mrs.Fatimafemale...26497.2250NaNC[2649,7.225,C][20rowsx13columns]Parameters:df:pd.DataFrameDataFramecolumns:listcolumnstosqueezeReturns:listdf.ds_normalize_lists()normalize_lists_in_column_end_user(df:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],column:Optional[str]=None)->pandas.core.series.Seriesdf=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})getdi=lambdax:[(randrange(1,4),randrange(1,4))forvinrange(randrange(1,5))]#create some random tuplesdf["dicttest"]=df.lkey.apply(lambdax:getdi(x))print(df)df1=df.ds_normalize_lists(column='dicttest')print(df1)df2=df.dicttest.ds_normalize_lists(column='dicttest')print(df2)lkeyvaluedicttest0foo1[(3,2)]1bar2[(3,1)]2baz3[(3,2),(3,3)]3foo5[(2,3),(2,1)]0[(3,2),<NA>]1[(3,1),<NA>]2[(3,2),(3,3)]3[(2,3),(2,1)]Name:dicttest,dtype:object0[(3,2),<NA>]1[(3,1),<NA>]2[(3,2),(3,3)]3[(2,3),(2,1)]Name:dicttest,dtype:objectParameters:df:Union[pd.Series,pd.DataFrame]pd.Series,pd.DataFramewithlists/tuplesincellscolumn:Union[str,None]Nonecanonlybeusedifapd.Seriesispassed.IfaDataFrameispassed,acolumnneedstobepassedtoo.Returns:pd.DataFrameMissingvaluesarefilledwithpd.NAdf.d_merge_multiple_dfs_and_series_on_index() / df.d_merge_multiple_dfs_and_series_on_one_column()qq_ds_merge_multiple_dfs_and_series_on_index(df:pandas.core.frame.DataFrame,list_with_ds:list[typing.Union[pandas.core.series.Series,pandas.core.frame.DataFrame]],how='inner',on=None,sort=False,suffixes=('_x','_y'),indicator=False,validate=None)->pandas.core.frame.DataFramedf1=pd.DataFrame({'lkeyaaaaaaaaaaaaaaaaaa':['foo','bar','baz','foo'],'value':[1,2,3,5]})df2=pd.DataFrame({'lkeybbbbbbbbbb':['foo','bar','baz','foo'],'value':[5,6,7,8]})df3=pd.DataFrame({'lkeyccccccccccccccc':['foo','bar','baz','foo'],'value':[15,16,17,18]})df4=pd.DataFrame({'lkeyddddddddddddd':['foo','bar','baz','foo'],'value':[115,116,117,118]})df5=pd.DataFrame({'lkeyeeeee':['foo','bar','baz','foo'],'value':[1115,1116,1117,1118]})df1.d_merge_multiple_dfs_and_series_on_index(list_with_ds=[df2,df3,df4,df5],how="outer")Out[85]:lkeyaaaaaaaaaaaaaaaaaavalue_x_000lkeybbbbbbbbbbvalue_y_000lkeycccccccccccccccvalue_x_002lkeydddddddddddddvalue_y_002lkeyeeeeevalue0foo1foo5foo15foo115foo11151bar2bar6bar16bar116bar11162baz3baz7baz17baz117baz11173foo5foo8foo18foo118foo1118Parameters:df:pd.DataFrameDataFramelist_with_ds:list[Union[pd.Series,pd.DataFrame]]AlistofDataFramesandSeriesyouwanttomergehow:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default="inner")on:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=None)sort:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=False)suffixes:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=("_x","_y"))indicator:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=False)validate:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=None)Returns:pd.DataFrameqq_ds_merge_multiple_dfs_and_series_on_column(df:pandas.core.frame.DataFrame,list_with_ds:list[typing.Union[pandas.core.series.Series,pandas.core.frame.DataFrame]],column:str,how='inner',sort=False,suffixes=('_x','_y'),indicator=False,validate=None)->pandas.core.frame.DataFramedf1=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})df2=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[5,6,7,8]})df3=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[15,16,17,18]})df4=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[115,116,117,118]})df5=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1115,1116,1117,1118]})df1.d_merge_multiple_dfs_and_series_on_one_column(list_with_ds=[df2,df3,df4,df5],column='lkey',how="outer",sort=False,suffixes=("_x","_y"),indicator=False,validate=None,)lkeyvalue_x_000value_y_000value_x_002value_y_002value0foo151511511151foo151511511182foo151511811153foo151511811184foo151811511155foo151811511186foo151811811157foo151811811188foo181511511159foo1815115111810foo1815118111511foo1815118111812foo1818115111513foo1818115111814foo1818118111515foo1818118111816foo5515115111517foo5515115111818foo5515118111519foo5515118111820foo5518115111521foo5518115111822foo5518118111523foo5518118111824foo5815115111525foo5815115111826foo5815118111527foo5815118111828foo5818115111529foo5818115111830foo5818118111531foo5818118111832bar2616116111633baz37171171117Parameters:df:pd.DataFrame:DataFramelist_with_ds:list[Union[pd.Series,pd.DataFrame]]AlistofDataFramesandSeriesyouwanttomergecolumn:strColumntomergeon-hastobepresentineverydfhow:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default="inner")sort:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=False)suffixes:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=("_x","_y"))indicator:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=False)validate:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html(default=None)Returns:pd.DataFramedf.dicttest.s_as_flattened_list()series_as_flattened_list(df)->listdf=pd.DataFrame({'lkey':['foo','bar','baz','foo'],'value':[1,2,3,5]})getdi=lambdax:[(randrange(1,4),randrange(1,4))forvinrange(20)]#create some random tuplesdf["dicttest"]=df.lkey.apply(lambdax:getdi(x))print(df)lkeyvaluedicttest0foo1[(3,2),(3,3),(3,1),(1,2),(2,1),(3,2...1bar2[(1,3),(3,3),(1,2),(3,3),(2,3),(1,3...2baz3[(1,1),(1,1),(3,3),(1,2),(1,1),(2,2...3foo5[(2,1),(2,1),(1,3),(1,3),(3,2),(2,1...list_=df.dicttest.s_as_flattened_list()print(list_[:20])[3,2,3,3,3,1,1,2,2,1,3,2,2,1,1,2,3,3,2,2]Parameters:df:pd.SeriesSeriestoflatten(removesallkeysindicts,onlykeepsthevalues)Returns:listdf1.d_stack() / df1.d_unstack()unstacked_df_back_to_multiindex(dataframe:pandas.core.frame.DataFrame)->pandas.core.frame.DataFrameDon't use df.stack()!!!!nested={"Moli":{"Buy":75,"Sell":53,"Quantity":300,"TF":True},"Anna":{"Buy":55,"Sell":83,"Quantity":154,"TF":False},"Bob":{"Buy":25,"Sell":33,"Quantity":100,"TF":False},"Annie":{"Buy":74,"Sell":83,"Quantity":96,"TF":True}}df1=pd.Q_AnyNestedIterable_2df(nested,unstack=True)print(df1)df1.d_stack()level_0level_1aa_all_keysaa_value0AnnaBuy(Anna,Buy)551AnnaQuantity(Anna,Quantity)1542AnnaSell(Anna,Sell)833AnnaTF(Anna,TF)False4AnnieBuy(Annie,Buy)745AnnieQuantity(Annie,Quantity)966AnnieSell(Annie,Sell)837AnnieTF(Annie,TF)True8BobBuy(Bob,Buy)259BobQuantity(Bob,Quantity)10010BobSell(Bob,Sell)3311BobTF(Bob,TF)False12MoliBuy(Moli,Buy)7513MoliQuantity(Moli,Quantity)30014MoliSell(Moli,Sell)5315MoliTF(Moli,TF)TrueOut[64]:aa_all_keysaa_valuelevel_0level_1AnnaBuy(Anna,Buy)55Quantity(Anna,Quantity)154Sell(Anna,Sell)83TF(Anna,TF)FalseAnnieBuy(Annie,Buy)74Quantity(Annie,Quantity)96Sell(Annie,Sell)83TF(Annie,TF)TrueBobBuy(Bob,Buy)25Quantity(Bob,Quantity)100Sell(Bob,Sell)33TF(Bob,TF)FalseMoliBuy(Moli,Buy)75Quantity(Moli,Quantity)300Sell(Moli,Sell)53TF(Moli,TF)TrueParameters:dataframe:pd.DataFramepd.DataFrameReturns:pd.DataFramepd.Q_ReadFileWithAllEncodings_2df()There are plenty of good libraries out there that help you with finding the right encoding for your file,

    but sometimes they don't work like expected, and you have to choose the best encoding manually. This method

    opens any file in all encodings available in your env and returns all results in a DataFrame.pd.Q_ReadFileWithAllEncodings_2df(r"C:\Users\Gamer\Documents\Downloads\corruptjson1.json")codecstrict_encoded\0ascii['ascii'codeccan't decode byte 0xef in posit...1base64_codec[Incorrectpadding]2big5[({\r\n"doc_id":"some_number",\r\n"url":"www...3big5hkscs[({\r\n"doc_id":"some_number",\r\n"url":"www...4bz2_codec[Invaliddatastream]........115utf_7['utf7'codeccan't decode byte 0xef in positi...116utf_8[({\r\n"doc_id":"some_number",\r\n"url":"www...117utf_8_sig[({\r\n"doc_id":"some_number",\r\n"url":"www...118uu_codec[Missing"begin"lineininputdata]119zlib_codec[Error-3whiledecompressingdata:incorrect...strict_badignore_encoded\0True[({\r\n"doc_id":"some_number",\r\n"url":"www...1True[]Parameters:filepath(str):filepathReturns:pd.DataFramepd.Q_CorruptJsonFile_2dict()read_corrupt_json(filepath:str)->dictUsage:pdQ_CorruptJsonFile_2dictf(r'C:\corruptjson1.json')IfyouneedtoreadacorruptedJSONfile,youcantrythismethod.Itwillfirsttrytoreadthefileusingujson.Secondstep:Thefilewillbereadusingallencodersfoundinyourenv.Eachresultwillbepassedtoast.literal_eval,json.loadsandujson.loadsThirdstep:KeysandvaluesareextractedusingregexAllpositiveresultsarereturnedasadict,youhavetocheckwhichonefitsbesttoyourneedsfinaldict={"ujson_file_reading_result":ujson_file_reading_result,"literal_eval_after_newline_removed":literal_eval_after_newline_removed,"json_after_head_tail_removed":json_after_head_tail_removed,"ujson_after_head_tail_removed":ujson_after_head_tail_removed,"regex_get_single_item_keys":allgoodresultsdict,}Ifthekeysarenotdouble-quoted,itwon't work.ItworkswellwithspacesandnotcorrectlyescapedcharactersExamplefromhttps://stackoverflow.com/questions/59927549/how-to-fix-a-possibly-corrupted-json-file-problems-with-a-curly-bracket-charact{"doc_id":"some_number","url":"www.seedurl1.com","scrape_date":"2019-10-22 16:17:22","publish_date":"unknown","author":"unknown","urls_out":["https://www.something.com","https://www.sometingelse.com/smth"],"text":"lots of text here"}﻿{"doc_id":"some_other_number","url":"www.seedurl2.com/smth","scrape_date":"2019-10-22 17:44:40","publish_date":"unknown","author":"unknown","urls_out":["www.anotherurl.com/smth","http://urlx.com/smth.htm"],"text":"lots more text over here."}Result:{'ujson_file_reading_result':None,'literal_eval_after_newline_removed':EmptyDataFrameColumns:[level_0,level_1,level_2,aa_all_keys,aa_value,ast_results]Index:[],'json_after_head_tail_removed':level_0...json_loads862punycode...{'doc_id':'some_number','url':'www.seedurl1...865punycode...{'doc_id':'some_number','url':'www.seedurl1...[2rowsx8columns],'ujson_after_head_tail_removed':level_0...ujson_loads862punycode...{'doc_id':'some_number','url':'www.seedurl1...865punycode...{'doc_id':'some_number','url':'www.seedurl1...[2rowsx9columns],'regex_get_single_item_keys':[{'aa_value':{0:'some_number',1:'www.seedurl1.com',2:'2019-10-22 16:17:22',3:'unknown',4:'unknown',5:['https://www.something.com','https://www.sometingelse.com/smth'],6:'lots of text here'},'aa_key':{0:'doc_id',1:'url',2:'scrape_date',3:'publish_date',4:'author',5:'urls_out',6:'text'}},{'aa_value':{7:'some_other_number',8:'www.seedurl2.com/smth',9:'2019-10-22 17:44:40',10:'unknown',........Parameters:filepath(str):filepathReturns:dictdf.d_sort_columns_with_sorted()qq_d_sort_columns_alphabetically(df:pandas.core.frame.DataFrame,reverse:bool=False)->pandas.core.frame.DataFrameSortscolumnsalphabeticallywithsorted()!Notwithnatsort()!df=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")print(df)df.d_sort_columns_with_sorted()PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS...................................88688702Montvila,Rev.Juozasmale...021153613.0000NaNS88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500NaNQ[891rowsx12columns]Out[66]:AgeCabinEmbarkedFareName...PclassSexSibSpSurvivedTicket022.0NaNS7.2500Braund,Mr.OwenHarris...3male10A/521171138.0C85C71.2833Cumings,Mrs.JohnBradley(FlorenceBriggsTh......1female11PC17599226.0NaNS7.9250Heikkinen,Miss.Laina...3female01STON/O2.3101282335.0C123S53.1000Futrelle,Mrs.JacquesHeath(LilyMayPeel)...1female11113803435.0NaNS8.0500Allen,Mr.WilliamHenry...3male00373450...................................88627.0NaNS13.0000Montvila,Rev.Juozas...2male0021153688719.0B42S30.0000Graham,Miss.MargaretEdith...1female01112053888NaNNaNS23.4500Johnston,Miss.CatherineHelen"Carrie"...3female10W./C.660788926.0C148C30.0000Behr,Mr.KarlHowell...1male0111136989032.0NaNQ7.7500Dooley,Mr.Patrick...3male00370376[891rowsx12columns]Parameters:df:pd.DataFramereverse:boolZ-AinsteadofA-Z(default=False)Returns:pd.DataFramedf.ds_isna()is_nan_true_false_check(df:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],include_na_strings:bool=True,include_empty_iters:bool=False,include_0_len_string:bool=False)->Union[pandas.core.series.Series,pandas.core.frame.DataFrame]df=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")df.ds_isna()Out[107]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0FalseFalseFalseFalseFalse...FalseFalseFalseTrueFalse1FalseFalseFalseFalseFalse...FalseFalseFalseFalseFalse2FalseFalseFalseFalseFalse...FalseFalseFalseTrueFalse3FalseFalseFalseFalseFalse...FalseFalseFalseFalseFalse4FalseFalseFalseFalseFalse...FalseFalseFalseTrueFalse...................................886FalseFalseFalseFalseFalse...FalseFalseFalseTrueFalse887FalseFalseFalseFalseFalse...FalseFalseFalseFalseFalse888FalseFalseFalseFalseFalse...FalseFalseFalseTrueFalse889FalseFalseFalseFalseFalse...FalseFalseFalseFalseFalse890FalseFalseFalseFalseFalse...FalseFalseFalseTrueFalse[891rowsx12columns]df.Cabin.ds_isna()Out[108]:0True1False2True3False4True...886True887False888True889False890TrueName:Cabin,Length:891,dtype:boolParameters:df:Union[pd.Series,pd.DataFrame]pd.Series,pd.DataFrameinclude_na_strings:boolWhenTrue->treatedasnan:["<NA>","<NAN>","<nan>","np.nan","NoneType","None","-1.#IND","1.#QNAN","1.#IND","-1.#QNAN","#N/A N/A","#N/A","N/A","n/a","NA","#NA","NULL","null","NaN","-NaN","nan","-nan",](default=True)include_empty_iters:boolWhenTrue->[],{}aretreatedasnan(default=False)include_0_len_string:boolWhenTrue->''istreatedasnan(default=False)Returns:dictReturns:Union[pd.Series,pd.DataFrame]df.d_add_value_to_existing_columns_with_loc()df_loc_add(df:pandas.core.frame.DataFrame,condition:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],add_to_colum:Any,column:str,throw_towel_early:bool=False,as_last_chance_convert_to_string:bool=False)->pandas.core.frame.DataFramedf=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")print(df[:6])df[:6].d_add_value_to_existing_columns_with_loc(condition=(df.Pclass==3),add_to_colum=100000,column="Fare")PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS5603Moran,Mr.Jamesmale...03308778.4583NaNQ[6rowsx12columns]Out[37]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/521171100007.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.3101282100007.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...0373450100008.0500NaNS5603Moran,Mr.Jamesmale...0330877100008.4583NaNQ[6rowsx12columns]Parameters:df:pd.DataFrameDataFrametoupdatecondition:Union[pd.Series,pd.DataFrame]Passaconditionwithdf.loc:df.loc[df['shield']>6]add_to_colum::AnyValuethatyouwanttoaddtooldvaluescolumn:strColumnwhichshouldbeupdatedthrow_towel_early:boolIfFalse:Ifthereisanexception,willbeiteratinglinebylinechangingeachvalue.Ifitfails,itwillkeeptheoldvalue.(default=False)as_last_chance_convert_to_string:boolIfyouwanttochangethevalueatanycost,youcanchangebothvaluestostringsandaddthemup,whichwillresultin:1+1="11""Big"+"Brother"="BigBrother"Returns:pd.DataFramedf.d_drop_rows_with_df_loc()df_loc_drop(df:pandas.core.frame.DataFrame,condition:Union[pandas.core.series.Series,pandas.core.frame.DataFrame])->pandas.core.frame.DataFramedf.d_drop_rows_with_df_loc(df.level_1.str.contains("aa_k"))isthesameasdf.loc[~df.level_1.str.contains('aa_k')].copy()df=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")dfOut[54]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS...................................88688702Montvila,Rev.Juozasmale...021153613.0000NaNS88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500NaNQ[891rowsx12columns]df.d_drop_rows_with_df_loc(df.Sex.str.contains(r"male$",regex=True,na=False))Out[55]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S8913Johnson,Mrs.OscarW(ElisabethVilhelminaBerg)female...234774211.1333NaNS91012Nasser,Mrs.Nicholas(AdeleAchem)female...023773630.0708NaNC...................................88088112Shelley,Mrs.William(ImanitaParrishHall)female...123043326.0000NaNS88288303Dahlberg,Miss.GerdaUlrikafemale...0755210.5167NaNS88588603Rice,Mrs.William(MargaretNorton)female...538265229.1250NaNQ88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS[314rowsx12columns]Parameters:df:pd.DataFrameDataFramecondition:Union[pd.Series,pd.DataFrame]Conditionwithdf.loc:df.loc[df['shield']>6]Returns:pd.DataFramedf.d_set_values_with_df_locdf_loc_set(df:pandas.core.frame.DataFrame,condition:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],new_data:Any,column:str)->pandas.core.frame.DataFramedf=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")dfOut[51]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS...................................88688702Montvila,Rev.Juozasmale...021153613.0000NaNS88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500NaNQ[891rowsx12columns]df.d_set_values_with_df_loc(condition=df.Sex.str.contains(r"male$",regex=True,na=False),column='Fare',new_data=100000)Out[52]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/521171100000.0000NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...0373450100000.0000NaNS...................................88688702Montvila,Rev.Juozasmale...0211536100000.0000NaNS88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS88989011Behr,Mr.KarlHowellmale...0111369100000.0000C148C89089103Dooley,Mr.Patrickmale...0370376100000.0000NaNQ[891rowsx12columns]Parameters:df:pd.DataframeDataFramecondition:Union[pd.Series,pd.DataFrame]Passaconditionwithdf.loc:df.loc[df['shield']>6]new_data:AnyNewvaluesforupdatecolumn:strColumnwhichshouldbeupdatedReturns:pd.DataFramedf.d_dfloc()df_loc(df:pandas.core.frame.DataFrame,condition:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],column:Optional[str]=None)->Union[pandas.core.series.Series,pandas.core.frame.DataFrame]df.d_dfloc(df.aa_value.str.contains("author"))isthesameasdf.loc[df.aa_value.str.contains('author')].copy()df=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")print(df)print(df.d_dfloc(df.Sex.str.contains(r"male$",regex=True,na=False)))df.d_dfloc(df.Sex.str.contains(r"male$",regex=True,na=False),column='Name')PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS...................................88688702Montvila,Rev.Juozasmale...021153613.0000NaNS88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500NaNQ[891rowsx12columns]PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS5603Moran,Mr.Jamesmale...03308778.4583NaNQ6701McCarthy,Mr.TimothyJmale...01746351.8625E46S7803Palsson,Master.GostaLeonardmale...134990921.0750NaNS...................................88388402Banfield,Mr.FrederickJamesmale...0C.A./SOTON3406810.5000NaNS88488503Sutehall,Mr.HenryJrmale...0SOTON/OQ3920767.0500NaNS88688702Montvila,Rev.Juozasmale...021153613.0000NaNS88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500NaNQ[577rowsx12columns]Out[60]:0Braund,Mr.OwenHarris4Allen,Mr.WilliamHenry5Moran,Mr.James6McCarthy,Mr.TimothyJ7Palsson,Master.GostaLeonard...883Banfield,Mr.FrederickJames884Sutehall,Mr.HenryJr886Montvila,Rev.Juozas889Behr,Mr.KarlHowell890Dooley,Mr.PatrickName:Name,Length:577,dtype:objectParameters:df:pd.DataFrameDataFramecondition:Union[pd.Series,pd.DataFrame]Passaconditionwithdf.loc:df.loc[df['shield']>6]column:Union[None,str]ifastringispassed,themethodwillreturnpd.SeriesNonewillreturnthewholeDataFrame(default=None)Returns:Union[pd.Series,pd.DataFrame]df.df.ds_all_nans_to_pdNA()all_nans_in_df_to_pdNA(df:Union[pandas.core.series.Series,pandas.core.frame.DataFrame],include_na_strings:bool=True,include_empty_iters:bool=False,include_0_len_string:bool=False)->Union[pandas.core.series.Series,pandas.core.frame.DataFrame]df=pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv")dfOut[86]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500NaNS1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250NaNS3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500NaNS...................................88688702Montvila,Rev.Juozasmale...021153613.0000NaNS88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500NaNS88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500NaNQ[891rowsx12columns]df.ds_all_nans_to_pdNA()Out[87]:PassengerIdSurvivedPclassNameSex...ParchTicketFareCabinEmbarked0103Braund,Mr.OwenHarrismale...0A/5211717.2500<NA>S1211Cumings,Mrs.JohnBradley(FlorenceBriggsTh...female...0PC1759971.2833C85C2313Heikkinen,Miss.Lainafemale...0STON/O2.31012827.9250<NA>S3411Futrelle,Mrs.JacquesHeath(LilyMayPeel)female...011380353.1000C123S4503Allen,Mr.WilliamHenrymale...03734508.0500<NA>S...................................88688702Montvila,Rev.Juozasmale...021153613.0000<NA>S88788811Graham,Miss.MargaretEdithfemale...011205330.0000B42S88888903Johnston,Miss.CatherineHelen"Carrie"female...2W./C.660723.4500<NA>S88989011Behr,Mr.KarlHowellmale...011136930.0000C148C89089103Dooley,Mr.Patrickmale...03703767.7500<NA>Q[891rowsx12columns]df.Cabin.ds_all_nans_to_pdNA()Out[88]:0<NA>1C852<NA>3C1234<NA>...886<NA>887B42888<NA>889C148890<NA>Name:Cabin,Length:891,dtype:objectParameters:df:Union[pd.Series,pd.DataFrame]pd.Series,pd.DataFrameinclude_na_strings:boolWhenTrue->treatedasnan:["<NA>","<NAN>","<nan>","np.nan","NoneType","None","-1.#IND","1.#QNAN","1.#IND","-1.#QNAN","#N/A N/A","#N/A","N/A","n/a","NA","#NA","NULL","null","NaN","-NaN","nan","-nan",](default=True)include_empty_iters:boolWhenTrue->[],{}aretreatedasnan(default=False)include_0_len_string:boolWhenTrue->''istreatedasnan(default=False)Returns:dictReturns:Union[pd.Series,pd.DataFrame]