Get More Efficient in your Data Analysis with Pandas `query()` and `eval()` Methods | by Olusegun Folarin | Towards Data Science

One of the common tasks in Exploratory data Analysis is asking questions of the data. This is usually made possible by filtering or slicing on columns, aggregation function and so on. Because Pandas is built on numpy, it allows you to carry out vectorized operations which are computationally efficient than using for loops.
As a data analyst/scientist, you may want to filter on your data for some insight. This involves writing expression by creating a mask which is then passed to the DataFrame as shown below in an example:
# import the necessary librariesimport pandas as pdimport numpy as np# create sample dataframedf = pd.DataFrame(np.random.randint(20, 100, size=(100,4)),                  columns=['A', 'B', 'C', 'D'])# create a maskmask = (df['A'] > 30) & (df['B'] < 30)
Even though this may look efficient, NumPy evaluates each subexpression of the above code separately. This means that the above expression is split into three subexpressions as seen below.
temp1 = df[‘A’] > 30temp2 = df['B'] < 30mask = temp1 & temp2
This where pd.eval() and pd.query() come in handy to increase the computational efficiency. Let us take a look at each of them and how to use them.
pd.eval() allows you to compute compound expression more efficiently. Let’s have an example of an operation you would want to compute using the DataFrame created above. We can compute the same filtering from the initial example using pd.eval() as:
mask = df.eval("A > 30 & B < 30")
We created the same mask by writing an expression in a string. This creates a mask with low memory usage. We can also complete arithmetic operations using eval() . Instead of the below operation:
(df['A'] + df['B']) / (df['C'] - df['D'])
Using eval() , we could have it written as:
df.eval("(A + B) / (C + D)")
Both results will be the same. You should notice here that the column names are just passed into the string as if they were variables.
We can also create a new column from arithmetic operations on other columns. Let’s say we would like to store the example above to a new column, we would do something like this:
df['E'] = (df['A'] + df['B']) / (df['C'] - df['D'])
The same expression can be written using eval() as:
df.eval("E = (A + B) / (C + D)", inplace=True)
This expression creates the column ‘E’ specified in the string expression. By passing inplace=True as an argument, you can alter the DataFrame permanently.
Perhaps, one of my favourite use of eval() is using an external variable in the DataFramemanipulation. Let’s say we would like to create a new column which is adding the mean of column ‘A’ to every row in A. We could easily do that using the following line of code:
col_mean = df['A'].mean()df.eval("F = A + @col_mean", inplace=True)
As you would have noticed, there is an ‘@’ character preceding the col_mean variable. This is because we use the ‘@’ character to separate a global/local python variable from the column names.
Alternatively, we can include the col_mean computation inside the eval statement.
df.eval("G = A + A.mean()", inplace=True)
One advantage of using query() is that it allows you to filter or compute on DataFrame by writing queries in simple natural language strings. Cool right?! Let take a look at query() in code. Let’s have a look at our previous example where we wanted to filter on column A > 30 and column B < 30 and return the filtered DataFrame, we could do that using query() :
# instead ofnew_df = df[(df['A'] > 30 & df['B'] < 30)]#ormask = (df['A'] > 30 & df['B'] < 30)new_df = df[mask]# use querynew_df = df.query("A > 30 and B < 30")
I hear you ask, what if my column name has space in it? Easy! you just wrap the column name in backticks. An example below:
df.query("D == `B D`")
The expression above is similar to:
df[df[‘D’ == df[‘B D’]]
Apart from being computationally better, using query makes your code easier to read and understand.
Of course like eval , you can also use an external variable on a DataFrameusing query as long as you precede the variable with the ‘@’ sign to specify it is a variable and not a column name in the DataFrame
So the next time you are exploring your data using pandas, make your code memory efficient by using eval and query that not only saves you memory usage (especially if you are working on a large dataset) but also makes your code easier to read.