Swiss army knife of pandas filtering | by Adiamaan Keerthi | Towards Data Science

Pandas eval and query are some of the most powerful, lesser-known, intuitive functions that are exposed by pandas API. It lets you query the data frame in a readable and intuitive way, different from the traditional way of masking and loc-ing the data. The resulting code is efficient, readable, and flexible.
Filtering a data frame is an important process in any data pipeline and in pandas, it is usually a two-step process. Creating a mask to select the rows and then subsetting the data frame using that mask.
Let’s load in the iris dataset,
>> import pandas as pd>> df = pd.read_csv("./data/iris.csv")>> df.columnsIndex(['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species'], dtype='object')>> df.shape(150, 5)
Let’s say you want to look at rows that are of species Setosa and sepal-length > 5. Such multi-column/condition filtering occurs more often in real-world EDA and data pipelines than you can think of. It can be generally written as,
Without query,
df.loc[(df.sepal_length > 5) & (df.species == "Iris-setosa"), :]
The problem with the above code is, df is repeated three times and there are a pair of square brackets and parenthesis. This can be easily rewritten using a query.
With query,
df.query("sepal_length > 5 and species == 'Iris-setosa'")
Pandas query provides a readable, intuitive interface to dataframe filtering that is easier on the eyes and quick to write.
Now that we have seen how to use query, we will look at practical tips of unlocking the power of query and eval.
pd.eval, pd.query ,df.evaland df.queryuses pd.eval in the background. pd.evalis the function that parses the string expression and substitutes variables and recognizes dataframe columns in the query.
Filtering using NaN is mostly done using pd.isna and then using it to mask the dataframe.
Without query,
# Set the top 5 rows sepal length asdf.iloc[:5, 0] = None# Filtering rows that are nulldf.loc[~pd.isna(df.sepal_length), :]# Filtering rows that are not nulldf.loc[pd.isna(df.sepal_length), :]
With query,
# Filtering rows that are nulldf.query("sepal_length == sepal_length")# Filtering rows that are not nulldf.query("sepal_length != sepal_length")
This works because matching NaN with itself returns false.
Whenever the comparison value is dynamic in a mask, you can use @ to dynamically insert the value into the comparison string.
length_thresh = 5species = 'Iris-setosa' df.query("sepal_length >= @length_thresh and species == @species")
length_thresh and species values are resolved dynamically.
In case the column names contain space in between you can use backticks to escape it,
df.query("`Sepal Length` >= @length_thresh and species == @species")
Flexibility can be added to the query statement by using f-strings. @ lets you parameterize the right side of an inequality, whereas f-strings will let you parameterized both sides of an inequality.
ltype = "sepal"length = 5.0df.query(f"{ltype}_length >= {length}")
By changing the ltype the variable you can control whether sepal_length or petal_length is used for filtering.
You can also create a complicated query substring and then combine them, making it readable instead of a single complicated combination of filtering.
# Define setosa filterspecies, length, ltype = "setosa", 3.0, "sepal"setosa_mask = f"species == 'Iris_{species}' and {ltype}_length >={length}"# Define virginica filterspecies, width, wtype = "virginica", 3.0, "petal"virginica_mask = f"species == 'Iris_{species}' and {wtype}_width >={width}"# combine filter>> df.query(f"{setosa_mask} or {virginica_mask}")-- Returns a dataframe of (96, 5)
Apart from the expressionand inplaceargument, pandas.DataFrame.query accepts **kwargswhich can take in any argument that you can pass to pandas.eval. Query internally calls Eval to execute the expression.
pandas.eval is akin to python’s eval which executes python code in a string.
assert eval("2 + 3 * len('hello')") == 17
Looking at this example, it is dangerously clear that you can execute any python statement inside it.
Eval when not used right is Evil!
With that said, pandas.evalis not as dangerous as it sounds. Unlike python’s eval pandas.evalcannot execute arbitrary functions.
>> eval("print('danger!')")danger>> pd.eval("print('me')")ValueError: "print" is not a supported function
It is clear that pandas.eval is similar to python’s eval but for a restricted subset of functions and operands.
There are other arguments of eval that can be used for more control.
Parser: The eval function accepts an input called parser which can be either python or pandas. Depending on the parser the expression is parsed slightly differently.
Engine: The engine which takes care of executing the expression. By default it is set to numexpr which is efficient and faster than the other option which is python
Local and Global dictionaries: Sometimes it is useful to pass in variables that are not defined in the current scope inside the expression.
df.query("setosa_length > thresh", local_dict={'thresh': 5.0})
Pandas eval can also execute statements and assign them to dataframes. For example, you can do this,
df.eval("length_ratio = sepal_length + petal_length", inplace=True)
The above statement will create a column length_ratio in the dataframe. This is the opposite of why we would want to use query statement. Unlike query expression which improves readability, this type of usage is ambiguous and hinders readability.
Query https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.htmlEvalhttps://pandas.pydata.org/docs/reference/api/pandas.eval.html#pandas.eval
Query https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html
Evalhttps://pandas.pydata.org/docs/reference/api/pandas.eval.html#pandas.eval
Author’s LinkedIn Profile