Pandas query method saves double handling of variables | by Alexis Lucattini | Towards Data Science

Standard methods to retrieve rows with certain conditions in a pandas DataFrame object requires ‘double handling’; it’s not particularly elegant.
For example, we want to retrieve rows where column A is greater than 1, this is the standard way to do it using the .loc attribute.
## Setup ### Import pandasimport pandas as pd# Create dataframe from dict# Specify the type in the suffix of each column namemy_df = pd.DataFrame({"A_int": [1, 2, -3],                      "B_float": [7.5, 1.9, 8.4],                      "C_str": ['eight', 'nine', 'Ten'],                      "D_str": ['Mar 2017', 'May 2018', 'Jun 2016']}                    )# Convert to datetime columnmy_df['D_date'] = pd.to_datetime(my_df['D_str'])""">>> my_df   A_int  B_float  C_str     D_str     D_date0      1      7.5  eight  Mar 2017 2017-03-011      2      1.9   nine  May 2018 2018-05-012     -3      8.4    Ten  Jun 2016 2016-06-01"""## Show loc example 1 ##ex_1_loc = my_df.loc[my_df['A_int'] > 0, :]""">>> ex_1_loc   A_int  B_float  C_str     D_str     D_date0      1      7.5  eight  Mar 2017 2017-03-011      2      1.9   nine  May 2018 2018-05-01"""
Here we obtain the Boolean series of rows where column A is greater than 0 for that given row. The .loc is used to pull out rows set to True. Note that the variable my_df is used twice here in a single line of code.It is fortunate that this variable name is not very long. I find this ‘double handling’ of the variable syntactically similar to base methods in R, particularly intimidating when nested and overall quite unnecessary.
Some other languages can perform this filtering in a much more readable format:
## SQL EXAMPLE ##SELECT *FROM my_dfWHERE A > 0## R-TIDYVERSE EXAMPLE ##library(tidyverse)my_df %>%  dplyr::filter(A > 0)
For python, we can do this simpler (and similar to the SQL / R-Tidyverse structure above) with pandas ‘query’ method. query returns only the rows that match the given condition, all columns remain present in the dataframe.
#ex_1_loc = my_df.loc[my_df['A_int'] > 0, :]ex_1_query = my_df.query("A > 0")# Or use @ when referencing a variablepivot_val = 0ex_1a_query = my_df.query("A > @pivot_val")
Let's see six more examples where query is an appropriate replacement to the .loc approach
Given an external list, obtain rows of the dataframe where a certain column matches an element in that list
# Create a list of legitimate entrieslegit_entries = ['eight', 'nine', 'ten']# Filter column 'C_str' by the arrayex_2_loc = my_df.loc[my_df['C_str'].isin(legit_entries), :]ex_2_query = my_df.query("C_str in @legit_entries")"""   A_int  B_float  C_str     D_str     D_date0      1      7.5  eight  Mar 2017 2017-03-011      2      1.9   nine  May 2018 2018-05-01"""
Compare two columns — this requires triple handling in the .loc example.
# Return rows where 'A_int' is greater than 'B_float'ex_3_loc = my_df.loc[my_df['A_int'] > my_df['B_float'], :]ex_3_query = my_df.query("A_int > B_float")"""   A_int  B_float C_str     D_str     D_date1      2      1.9  nine  May 2018 2018-05-01"""
Filter using multiple conditions over potentially multiple columns. Both & and | bitwise operators are allowed.
# Return rows where 'A_int' is greater than zero# And where C_str is in the legit_entries arrayex_4_loc = my_df.loc[(my_df["A_int"] > 0) &                     (my_df['C_str'].isin(legit_entries)), :]ex_4_query = my_df.query("A_int > 0 & C_str in @legit_entries")"""   A_int  B_float  C_str     D_str     D_date0      1      7.5  eight  Mar 2017 2017-03-011      2      1.9   nine  May 2018 2018-05-01"""
Query recognises dates and can compare them to strings. All strings inside a query quote must be quoted.
# Return rows where D_date is after Jan 2018ex_5_loc = my_df.loc[my_df['D_date'] > 'Jan 2018', :]ex_5_query = my_df.query("D_date > 'Jan 2018'")"""   A_int  B_float C_str     D_str     D_date1      2      1.9  nine  May 2018 2018-05-01"""
You can perform a function on a given column prior to comparison.Note that the output of the dataframe still contains the C_str column in its original format?
# First convert C_str to lowercase, than compare entriesex_6_loc = my_df.loc[my_df['C_str'].str.lower().isin(legit_entries),                     :]ex_6_query = my_df.query("C_str.str.lower() in @legit_entries")"""   A_int  B_float  C_str     D_str     D_date0      1      7.5  eight  Mar 2017 2017-03-011      2      1.9   nine  May 2018 2018-05-012     -3      8.4    Ten  Jun 2016 2016-06-01"""
7. Column names containing spacesInitially forbidden with pandas query, this has been resolved as of 0.25.2.Quote the column with backticks. Other special characters in the column may not work.
# Return rows where 'E rogue column str' column contains the word 'this'my_df['E rogue_column_str'] = ['this', 'and', 'that']""">>>my_df   A_int  B_float  C_str     D_str     D_date E rogue_column_str0      1      7.5  eight  Mar 2017 2017-03-01               this1      2      1.9   nine  May 2018 2018-05-01                and2     -3      8.4    Ten  Jun 2016 2016-06-01               that"""ex_7_loc = my_df.loc[my_df["E rogue_column_str"] == 'this', :]ex_7_query = my_df.query("`E rogue_column_str` == 'this'"))"""   A_int  B_float  C_str     D_str     D_date E rogue_column_str0      1      7.5  eight  Mar 2017 2017-03-01               this"""
In the latest release of pandas (0.25.3 at the time of writing), the .item() method was deprecated. Combined with the query function, item was a highly useful tool to obtain a single value from a given column when it was appropriate to assume that only one row would be returned. This would convert the column from a type pd.Series to that of the series’ datatype i.e int, str, float etc. If more than one row was returned, a ValueError is raised by the method.
We can use the squeeze method as an alternative with some small changes in the code. Like item, squeeze will convert a pandas series from series type to the datatype when the filtered series is only one row long. However, a ValueError is not raised if multiple values exist in the series, instead just the same series is returned. We can exploit this by checking if the returned value is still of a series type.
# Previous codetry:    single_value = ex_3_query['B_float'].item()    print(single_value)except ValueError:    print("Error, expected to return only one value, got %d" %          len(ex_3_query['B_float']))# New codesingle_value = ex_3_query['B_float'].squeeze()if isinstance(single_value, pd.Series):    print("Error, expected to return only one value, got %d" %          len(ex_3_query['B_float']))else:    print(single_value)
References:Pandas Query API