Package: a-pandas-ex-numexpr
Pandas DataFrame Operations 8 times faster (or even more)DataFrame.query has never worked for me. On my PC, it has been extremely slow when using small DataFrames, and only a little bit, if at all, faster when using huge DataFrames.DataFrame.query uses pd.eval, and pd.eval uses numexpr. The weird thing is that numexpr is insanely fast when it is used against a DataFrame, but nor pd.eval neither DataFrame.query aren’t. First I thought there was a problem with my Pandas/environment configuration, but then I read on thePandas page:You will only see the performance benefits of using the numexpr engine with DataFrame.query() if your frame has more than approximately 200,000 rows.Well,a_pandas_ex_numexpradds different methods to the DataFrame/Series classes, and will get tremendous speed-ups(up to 8 times faster in my tests)even for small DataFrames. All tests were done using:https://github.com/pandas-dev/pandas/raw/main/doc/data/titanic.csvLet the numbers speak for themselvesHow to import / use a_pandas_ex_numexprfroma_pandas_ex_numexprimportpd_add_numexprpd_add_numexpr()importpandasaspddafra="https://github.com/pandas-dev/pandas/raw/main/doc/data/titanic.csv"df=pd.read_csv(dafra)dfOut[3]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[891rowsx12columns]Speed test - a_pandas_ex_numexpr# Code explanation at the end of the pagewholedict={'c':df.Pclass}%timeitdf['Survived'].ne_query('b * 99.5 / 000.1 + 42123.323211 / 1335523.42232 * c',return_np=True,local_dict=wholedict)30.8µs±229nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived'].ne_query('b * 99.5 / 000.1 + 42123.323211 / 1335523.42232 * c',return_np=False,local_dict=wholedict)70.1µs±2.44µsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived']*99.5/000.1+42123.323211/1335523.42232*df['Pclass']262µs±4.25µsperloop(mean±std.dev.of7runs,1,000loopseach)%timeitpd.eval("df.Survived * 99.5 / 000.1 + 42123.323211 / 1335523.42232 * df.Pclass")#used by df.query1.37ms±45.4µsperloop(mean±std.dev.of7runs,1,000loopseach)df['Survived'].ne_query('b * 99.5 / 000.1 + 42123.323211 / 1335523.42232 * c',return_np=False,local_dict=wholedict)Out[33]:00.0946221995.0315412995.0946223995.03154140.094622...8860.063081887995.0315418880.094622889995.0315418900.094622Length:891,dtype:float64df['Survived']*99.5/000.1+42123.323211/1335523.42232*df['Pclass']Out[34]:00.0946221995.0315412995.0946223995.03154140.094622...8860.063081887995.0315418880.094622889995.0315418900.094622Length:891,dtype:float64wholedict={'c':df.Pclass}%timeitdf['Survived'].ne_query('b * 99.5 * c',return_np=True,local_dict=wholedict)27µs±245nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived'].ne_query('b * 99.5 * c',return_np=False,local_dict=wholedict)65.7µs±1.65µsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived']*99.5*df['Pclass']140µs±5.46µsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitpd.eval("df.Survived * 99.5 * df.Pclass")916µs±7.1µsperloop(mean±std.dev.of7runs,1,000loopseach)wholedict={'c':df.Pclass}%timeitdf['Survived'].ne_query('b / c',return_np=True,local_dict=wholedict)26.5µs±200nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived'].ne_query('b / c',return_np=False,local_dict=wholedict)# returns a Series60.3µs±336nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived']/df['Pclass']68.2µs±599nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitpd.eval("df.Survived / df.Pclass")929µs±31.7µsperloop(mean±std.dev.of7runs,1,000loopseach)All functions/methodsSpeed of some “ready to use methods” for Series%timeitdf.loc[df.PassengerId.ne_less_than(100)]142µs±412nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId<100]212µs±897nsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################%timeitdf.loc[df.Survived.ne_not_equal(0)]157µs±390nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.Survived!=0]229µs±1.46µsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################%timeitdf.loc[df.PassengerId.ne_greater_than(100)]174µs±375nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId>100]248µs±2.26µsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################%timeitdf.loc[df.PassengerId.ne_equal(1)]138µs±626nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId==1]209µs±1.04µsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################%timeitdf.loc[df.Cabin.ne_search_for_string_contains('C1')]329µs±1.18µsperloop(mean±std.dev.of7runs,1,000loopseach)%timeitdf.loc[df.Cabin.str.contains('C1',na=False)]403µs±924nsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################%timeitdf.loc[df.PassengerId.ne_greater_than_or_equal_to(100)]175µs±832nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId>=100]251µs±2.77µsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################%timeitdf.loc[df.PassengerId.ne_less_than_or_equal_to(100)]145µs±1.82µsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId<=100]212µs±1.63µsperloop(mean±std.dev.of7runs,1,000loopseach)##############################################################Overview - all methods for DataFrames/Series# Always use 'b' as the variable for the Series/DataFramedf.ne_search_in_all_columns('b == 1')array([0,1,2,3,8,9,10,11,15,17,19,21,22,23,25,28,31,32,36,39,43,44,47,52,53,55,56,58,61,65,66,68,74,78,79,81,82,84,85,88,97,98,106,107,109,123,125,127,128,133,136,141,142,146,151,156,161,165,166,172,183,184,186,187,190,192,193,194,195,198...]# Returns duplicated index if the value is found in# several columns. Exceptions will be ignored# the dtype argument is useful when searching for# strings -> dtype='S' (ascii only)df.ne_search_in_all_columns('b == "1"',dtype='S')array([0,1,2,3,8,9,10,11,15,17,19,21,22,23,25,28,31,32,36,39,43,44,47,52,53,55,56,58,61,65,66,68,74,78,79,81,82,84,85,88,97,98,106,107,109,...]# Converts all columns to  dtype='S' before searching# Might not work with special characters# UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 0:df.ne_search_string_allhits_contains('C1')Out[6]:PassengerIdSurvivedPclass...FareCabinEmbarked3411...53.1000C123S111211...26.5500C103S11011101...52.0000C110S13713801...53.1000C123S26826911...153.4625C125S27327401...29.7000C118C29829911...30.5000C106S33133201...28.5000C124S35135201...35.0000C128S44945011...30.5000C104S45245301...27.7500C111C57157211...51.4792C101S60961011...153.4625C125S66967011...52.0000C126S71171201...26.5500C124S71271311...52.0000C126S88989011...30.0000C148C[17rowsx12columns]# Series doesn't return duplicated resultsdf.Cabin.ne_search_string_allhits_contains('C1')Out[9]:3C12311C103110C110137C123268C125273C118298C106331C124351C128449C104452C111571C101609C125669C126711C124712C126889C148Name:Cabin,dtype:object%timeitdf.Cabin.ne_search_string_allhits_contains('C1')274µs±2.74µsperloop(mean±std.dev.of7runs,1,000loopseach)%timeitdf.Cabin.loc[df.Cabin.str.contains('C1',na=False)]351µs±1.16µsperloop(mean±std.dev.of7runs,1,000loopseach)# All rows where the string/substring C1 is found.# Numbers are converted to string (ascii)df.ne_search_string_dataframe_contains('C1')Out[13]:PassengerIdSurvivedPclass...FareCabinEmbarked3411...53.1000C123S111211...26.5500C103S11011101...52.0000C110S13713801...53.1000C123S26826911...153.4625C125S27327401...29.7000C118C29829911...30.5000C106S33133201...28.5000C124S35135201...35.0000C128S44945011...30.5000C104S45245301...27.7500C111C57157211...51.4792C101S60961011...153.4625C125S66967011...52.0000C126S71171201...26.5500C124S71271311...52.0000C126S88989011...30.0000C148C[17rowsx12columns]df.ne_search_string_dataframe_contains('610')Out[14]:PassengerIdSurvivedPclass...FareCabinEmbarked19419511...27.7208B4C60961011...153.4625C125S[2rowsx12columns]# Converts all columns to ascii and searches in each column# For each presence in a column, you  get a duplicate of the indexdf.ne_search_string_dataframe_allhits_equal('1')df.ne_search_string_dataframe_allhits_equal('1')Out[15]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS0103...7.2500NaNS1211...71.2833C85C1211...71.2833C85C1211...71.2833C85C.......................88788811...30.0000B42S88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C88989011...30.0000C148C[886rowsx12columns]# All equal strings in a Seriesdf.Embarked.ne_search_string_dataframe_allhits_equal('S')Out[16]:0S2S3S4S6S..883S884S886S887S888SName:Embarked,Length:644,dtype:object%timeitdf.Embarked.ne_search_string_dataframe_allhits_equal('S')160µs±2.14µsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.Embarked.loc[df.Embarked=='S']178µs±3.04µsperloop(mean±std.dev.of7runs,10,000loopseach)# Converts the whole df to ascii and checks where the# the value is present. Exceptions are ignoreddf.ne_search_string_dataframe_equal('C123')Out[20]:PassengerIdSurvivedPclass...FareCabinEmbarked3411...53.1C123S13713801...53.1C123S[2rowsx12columns]# Might not be efficient (The only method that was slower during testing)!%timeitdf.Cabin.loc[df.Cabin.ne_search_for_string_series_equal('C123')]252µs±1.02µsperloop(mean±std.dev.of7runs,1,000loopseach)%timeitdf.Cabin.loc[df.Cabin=='C123']158µs±1.28µsperloop(mean±std.dev.of7runs,10,000loopseach)Out[21]:PassengerIdSurvivedPclass...FareCabinEmbarked3411...53.1C123S13713801...53.1C123S[2rowsx12columns]# Returns bool valuesdf.loc[df.ne_search_for_string_contains('C1')]Out[7]:array([[False,False,False,...,False,False,False],[False,False,False,...,False,False,False],[False,False,False,...,False,False,False],...,[False,False,False,...,False,False,False],[False,False,False,...,False,True,False],[False,False,False,...,False,False,False]])# returns Booldf.loc[df.Cabin.ne_search_for_string_contains('C1')]Out[14]:PassengerIdSurvivedPclass...FareCabinEmbarked3411...53.1000C123S111211...26.5500C103S11011101...52.0000C110S13713801...53.1000C123S26826911...153.4625C125S27327401...29.7000C118C29829911...30.5000C106S33133201...28.5000C124S35135201...35.0000C128S44945011...30.5000C104S45245301...27.7500C111C57157211...51.4792C101S60961011...153.4625C125S66967011...52.0000C126S71171201...26.5500C124S71271311...52.0000C126S88989011...30.0000C148C[17rowsx12columns]%timeitdf.loc[df.Cabin.ne_search_for_string_contains('C1')]329µs±1.18µsperloop(mean±std.dev.of7runs,1,000loopseach)%timeitdf.loc[df.Cabin.str.contains('C1',na=False)]403µs±924nsperloop(mean±std.dev.of7runs,1,000loopseach)# Returns the index of all rows where the value was found.# Exceptions (e.g. wrong datatype etc.) are ignored.# duplicates (more positive results in one row) are not deleteddf.ne_equal_df_ind(1)array([0,1,2,3,8,9,10,11,15,17,19,21,22,23,25,28,31,32,36,39,43,44,47,52,53,55,56,58,61,65,66,68,74,78,79,81,82,84,85,88,97,98,106,107,109,123...]# You can pass dtype='S' to convert the values to string# (or other formats) before performing the search.# df.ne_equal_df_ind(b'1', 'S')# If you use 'S', you have to pass a binary valuedf.ne_equal_df_ind(b'1','S')Out[16]:array([0,1,2,3,8,9,10,11,15,17,19,21,22,23,25,28,31,32,36,39,43,44,47,52,53,55,56,58,61,65,66,68,74,...]# same as DataFrame.ne_equal_df_ind# but deletes all duplicatesdf.ne_equal_df_ind_no_dup(b'1','S')array([0,1,2,3,6,7,8,9,10,11,13,15,16,17,18,19,21,22,23,24,25,27,28,30,31,32,34,35,36,39,40,41,43,44# Same as DataFrame.ne_equal_df_ind,# but returns the DataFrame (df.loc[])df.ne_equal_df_dup(b'1','S')Out[18]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S8913...11.1333NaNS.......................85685711...164.8667NaNS86987013...11.1333NaNS87187211...52.5542D35S87988011...83.1583C50C88088112...26.0000NaNS[886rowsx12columns]# Same as DataFrame.ne_equal_df_ind_no_dup# but returns the DataFrame (df.loc)df.ne_equal_df_no_dup(b'1','S')Out[19]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S6701...51.8625E46S.......................87988011...83.1583C50C88088112...26.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C[524rowsx12columns]# Returns boolarray([True,False,False,False,False,False,False,False,False,False,False,False,...]df.loc[df.PassengerId.ne_equal(1)]%timeitdf.loc[df.PassengerId.ne_equal(1)]138µs±626nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId==1]209µs±1.04µsperloop(mean±std.dev.of7runs,1,000loopseach)# Every time the condition is False, the index is# added to the return value.# Example:# A row has 6 columns. 2 of them have the value 1.# That means the index of the row will be added 4 times# to the final resultdf.loc[df.ne_not_equal_df_ind(1)]array([1,2,3,...,888,889,890],dtype=int64)df.loc[df.ne_not_equal_df_ind(1)]PassengerIdSurvivedPclass...FareCabinEmbarked1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS5603...8.4583NaNQ.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[5344rowsx12columns]# Same as DataFrame.ne_not_equal_df_ind# but drops all duplicatesdf.ne_not_equal_df_ind_no_dup(0)array([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,...]df.loc[df.ne_not_equal_df_ind_no_dup(0)]Out[26]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[891rowsx12columns]# same as DataFrame.ne_not_equal_df_ind# but returns the DataFrame (df.loc)df.ne_not_equal_df_dup(0)Out[28]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[4387rowsx12columns]# same as DataFrame.ne_not_equal_df_no_dup# but returns the DataFrame (df.loc)df.ne_not_equal_df_no_dup(0)Out[29]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[891rowsx12columns]returnsBooldf.Survived.ne_not_equal(0)array([False,True,True,True,False,False,False,False,True,True,True,True,False,False,False,True,False,True,False,True...]df.loc[df.Survived.ne_not_equal(0)]PassengerIdSurvivedPclass...FareCabinEmbarked1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S8913...11.1333NaNS91012...30.0708NaNC.......................87587613...7.2250NaNC87988011...83.1583C50C88088112...26.0000NaNS88788811...30.0000B42S88989011...30.0000C148C[342rowsx12columns]%timeitdf.loc[df.Survived.ne_not_equal(0)]157µs±390nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.Survived!=0]229µs±1.46µsperloop(mean±std.dev.of7runs,1,000loopseach)# returns index, duplicates are possible# if the condition is valid for more than one# column. Exceptions (e.g. wrong dtype) are ignoreddf.ne_greater_than_df_ind(100)array([100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115...]# Same as DataFrame.ne_greater_than_df_ind# but gets rid off all duplicatesdf.ne_greater_than_df_ind_no_dup(0)array([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16...]# Same as DataFrame.ne_greater_than_df_ind# but returns the DataFrame (df.loc)df.ne_greater_than_df_dup(0)Out[22]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[4210rowsx12columns]# same as DataFrame.ne_greater_than_df_ind_no_dup# but returns the DataFrame (df.loc)df.ne_greater_than_df_no_dup(600)Out[24]:PassengerIdSurvivedPclass...FareCabinEmbarked60060112...27.0000NaNS60160203...7.8958NaNS60260301...42.4000NaNS60360403...8.0500NaNS60460511...26.5500NaNC.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[291rowsx12columns]# Returns booldf.PassengerId.ne_greater_than(5)Out[26]:array([False,False,False,False,False,True,True,True,True,True,True,True,True,True,True...]df.loc[df.PassengerId.ne_greater_than(100)]%timeitdf.loc[df.PassengerId.ne_greater_than(100)]174µs±375nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId>100]248µs±2.26µsperloop(mean±std.dev.of7runs,1,000loopseach)# returns index, duplicates are possible# if the condition is valid for more than one# column. Exceptions (e.g. wrong dtype) are ignoreddf.ne_less_than_df_ind(10)array([0,1,2,...,881,884,890],dtype=int64)# Same as DataFrame.ne_less_than_df_ind# but without duplicatesdf.ne_less_than_df_ind_no_dup(100)Out[28]:array([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,...]# Same as DataFrame.ne_less_than_df_ind,# but returns DataFrame (df.loc)df.ne_less_than_df_dup(1)Out[29]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS4503...8.0500NaNS5603...8.4583NaNQ6701...51.8625E46S7803...21.0750NaNS.......................67467502...0.0000NaNS73273302...0.0000NaNS80680701...0.0000A36S81581601...0.0000B102S82282301...0.0000NaNS[1857rowsx12columns]# Same as DataFrame.ne_less_than_df_ind_no_dup# but returns DataFrame (df.loc)df.ne_less_than_df_no_dup(1)Out[30]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[834rowsx12columns]# returns bool# df.PassengerId.ne_less_than(100)df.PassengerId.ne_less_than(100)Out[31]:array([True,True,True,True,True,True...]%timeitdf.loc[df.PassengerId.ne_less_than(100)]142µs±412nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId<100]212µs±897nsperloop(mean±std.dev.of7runs,1,000loopseach)# returns index, duplicates are possible# if the condition is valid for more than one# column. Exceptions (e.g. wrong dtype) are ignoreddf.ne_greater_than_or_equal_to_df_ind(100)Out[35]:array([99,100,101,102,103,104,...]# Same as DataFrame.ne_greater_than_or_equal_to_df_ind ,# but without duplicates# df.ne_greater_than_or_equal_to_df_ind_no_dup(100)df.ne_greater_than_or_equal_to_df_ind_no_dup(100)Out[36]:array([27,31,88,99,100,101,102,...]# Same as DataFrame.ne_greater_than_or_equal_to_df_ind,# but returns DataFrame (df.loc)df.ne_greater_than_or_equal_to_df_dup(100)Out[37]:PassengerIdSurvivedPclass...FareCabinEmbarked9910002...26.0000NaNS10010103...7.8958NaNS10110203...7.8958NaNS10210301...77.2875D26S10310403...8.6542NaNS.......................74274311...262.3750B57B59B63B66C76376411...120.0000B96B98S77978011...211.3375B3S80280311...120.0000B96B98S85685711...164.8667NaNS[845rowsx12columns]# Same as DataFrame.ne_greater_than_or_equal_to_df_ind,# but returns DataFrame (df.loc)df.ne_greater_than_or_equal_to_df_no_dup(100)Out[38]:PassengerIdSurvivedPclass...FareCabinEmbarked272801...263.0000C23C25C27S313211...146.5208B78C888911...263.0000C23C25C27S9910002...26.0000NaNS10010103...7.8958NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[795rowsx12columns]# returns booldf.PassengerId.ne_greater_than_or_equal_to(100)Out[39]:array([False,False,False,False,False...])df.PassengerId.ne_greater_than_or_equal_to(100)%timeitdf.loc[df.PassengerId.ne_greater_than_or_equal_to(100)]175µs±832nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId>=100]251µs±2.77µsperloop(mean±std.dev.of7runs,1,000loopseach)# returns index, duplicates are possible# if the condition is valid for more than one# column. Exceptions (e.g. wrong dtype) are ignoreddf.ne_less_than_or_equal_to_df_ind(100)Out[40]:array([0,1,2,...,888,889,890],dtype=int64)# Same as DataFrame.ne_less_than_or_equal_to_df_ind ,# but without duplicatesdf.ne_less_than_or_equal_to_df_ind_no_dup(100)Out[41]:array([0,1,2,3,4,5,6,7,8,...])# Same as DataFrame.ne_less_than_or_equal_to_df_ind,# but returns DataFrame (df.loc)df.ne_less_than_or_equal_to_df_dup(100)Out[42]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[5216rowsx12columns]# Same as DataFrame.ne_less_than_or_equal_to_df_ind,# but returns DataFrame (df.loc)df.ne_less_than_or_equal_to_df_no_dup(0)Out[53]:PassengerIdSurvivedPclass...FareCabinEmbarked0103...7.2500NaNS1211...71.2833C85C2313...7.9250NaNS3411...53.1000C123S4503...8.0500NaNS.......................88688702...13.0000NaNS88788811...30.0000B42S88888903...23.4500NaNS88989011...30.0000C148C89089103...7.7500NaNQ[829rowsx12columns]# returns booldf.PassengerId.ne_less_than_or_equal_to(100)Out[55]:array([True,True,True,True,....]%timeitdf.loc[df.PassengerId.ne_less_than_or_equal_to(100)]145µs±1.82µsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df.PassengerId<=100]212µs±1.63µsperloop(mean±std.dev.of7runs,1,000loopseach)# Combining conditions%timeitdf.loc[df.PassengerId.ne_greater_than(100)&df.Cabin.ne_search_for_string_series_contains('C1')]360µs±2.56µsperloop(mean±std.dev.of7runs,1,000loopseach)%timeitdf.loc[(df.PassengerId>100)&df.Cabin.str.contains('C1',na=False)]552µs±3.49µsperloop(mean±std.dev.of7runs,1,000loopseach)# you can pass your own queries# If you want to compare the DataFrame/Series to another array# the variable 'b' represents the DataFrame/Series# That means: don't use it for something elsewholedict={'c':np.array([1])}df[['Survived','Pclass']].ne_query('b == c',local_dict=wholedict)Out[14]:array([[False,False],[True,True],[True,False],...,[False,False],[True,True],[False,False]])# You can use any NumExpr operator/function# https://numexpr.readthedocs.io/projects/NumExpr3/en/latest/user_guide.html# And get a tremendous speedup (even with small DataFrames)%timeitdf['Survived']+df.Pclass68.6µs±167nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived']*df.Pclass69µs±260nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived']==df.Pclass72.3µs±817nsperloop(mean±std.dev.of7runs,10,000loopseach)# You have to pass the Series/Arrays that you are using in the expression as a dict (local_dict)wholedict={'c':df.Pclass}%timeitdf['Survived'].ne_query('b + c',local_dict=wholedict)25.2µs±130nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived'].ne_query('b * c',local_dict=wholedict)25.3µs±177nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf['Survived'].ne_query('b == c',local_dict=wholedict)25.2µs±197nsperloop(mean±std.dev.of7runs,10,000loopseach)# Exceptions are not ignored# If you want to compare the DataFrame with a scalar:df[['Survived','Pclass']].ne_query('b == 1')# works also for Serieswholedict={'c':np.array([1])}df['Survived'].ne_query('b == c',local_dict=wholedict)# scalardf['Pclass'].ne_query('b == 1')%timeitdf.loc[df['Pclass'].ne_query('b == 1')]155µs±530nsperloop(mean±std.dev.of7runs,10,000loopseach)%timeitdf.loc[df['Pclass']==1]220µs±3.96µsperloop(mean±std.dev.of7runs,1,000loopseach)