Pandas Query for SQL-like Querying | by Matt Przybyla | Towards Data Science

DatasetPandasQueryTutorial CodeSummaryReferences
Dataset
Pandas
Query
Tutorial Code
Summary
References
The dataset used in this analysis and tutorial for pandas query is a dummy dataset created to mimic a dataframe with both text and numeric features. Feel free to use your own .csv file with either or both text and numeric columns to follow the tutorial.
Pandas [2] is one of the most common libraries used by data scientists and machine learning engineers. It is mainly used in the exploratory data analysis step of building a model, as well as the ad-hoc analysis of model results. It also contains several functions, including the query function.
The query function in pandas is a useful function that acts similarly to the ‘where’ clause in SQL. The benefit of it, however, is that you do not need to keep switching from pandas, Jupyter Notebook, and the SQL platform you are currently using. Some other benefits are listed below:
compiles several conditions
simple function that works off your pandas dataframe
references descriptive statistics instead of having to subquery
I have compiled several examples of how the pandas query function can be used. The screenshot below includes a summary of useful examples with a query. Below, you can see how there are both text and numeric conditions that can be satisfied. Find rows where a certain column contains a certain text value, or create a descriptive statistic using the ‘@’ symbol. For example, ‘@mean_of_column’ is referencing a value you have established from using the ‘.mean()’ function. In line 65, rows returned had a ‘Blue’ value and a ‘Confirmed_Recovery’ value that is greater than the mean of ‘Confirmed_Recovery’ itself, while also including the ‘Label_3’ value.
The tutorial code referenced in the grey box is all of the code used to demonstrate the query function. First, I imported pandas and read my dataframe. Then, I add new rows that I knew would satisfy certain conditions for displaying how the query function could be used. The ‘append’ function was useful here as well because it quickly added new rows to the dataframe without having to explicitly call out each column, once in a series format. I returned the ‘df.tail()’ to see if the rows returned were what I was expecting.
I then returned rows where values from one column equalled that of another. You can also use this same logic but reverse it to see which values for the columns you specify to not equal one another. Next, you can compare if the value of one column is bigger than one value of another column, or vice versa.
The most useful feature, I believe, of query is the ‘@’ method. Just like SQL where a subquery can be used to reference for selecting rows where certain conditions are satisfied, so can this method. The ‘@’ method saves the value that you want to compare against. For the example below, I looked at the mean of one column to compare.
Lastly, you can execute several conditions in one line of code using the query function by inputting ‘&’ similar to that of ‘and’ in SQL.
# All the python code below for use:# import libraryimport pandas as pd# read in your dataframedf = pd.read_csv('/Users/example.csv')# write out new rowsrows = [pd.Series([100, 100, 20,'Blue','Label_1'], index=df.columns),pd.Series([100, 80, 60,'Blue','Label_1'], index=df.columns),pd.Series([80, 60, 100,'Blue','Label_1'], index=df.columns)]# append the multiple rowsnew_df = df.append(rows , ignore_index=True)# check the newest 3 rows you madenew_df.tail(3)# return rows where values from one column equal that of another# they do not for this comparisonnew_df.query('Confirmed_Test == Confirmed_New')# return rows where values from one column equal that of another# they do for this comparisonnew_df.query('Confirmed_Test == Confirmed_Recovery')# return rows where values from one column do not equal that of another# they do for this comparisonnew_df.query('Confirmed_New != Confirmed_Recovery').head()# return rows where values from one column are bigger than that of anothernew_df.query('Confirmed_New > Confirmed_Recovery').head()# see which rows where the 'Confirmed_New' values# are greater than the mean of the total column# use the '@' to reference 'cn_mean'cn_mean = new_df['Confirmed_New'].mean()new_df.query('Confirmed_New > @cn_mean').head()# multiple conditions examplecn_min = new_df['Confirmed_New'].min()cn_max = new_df['Confirmed_New'].max()new_df.query('Confirmed_New > @cn_min & Confirmed_New < @cn_max').head()# text conditions# use double quotes for matching on the string value# all the rows returned have 'Blue' for the 'Text_Feature' columnnew_df.query('Text_Feature == "Blue"').head()# return rows which have a 'Blue' value, and a 'Confirmed_Recovery' that is greater# than the mean of 'Confirmed_Recovery' itself, while also satisfying the 'Label_3' value# only 3 return here (did not print out head())cr_mean = new_df['Confirmed_Recovery'].mean()new_df.query('Text_Feature == "Blue" & Confirmed_Recovery > @cr_mean & Text_Predictor == "Label_3"')
To see the code in python-format rather than written out, here is the embedded gist [4]:
The query function from pandas is an easy and quick way to manipulate your dataframe. You can use SQL-like clauses that return certain rows from satisfying the conditions that you determine. It is beneficial if you are already in your Jupyter Notebook .ipynb file or .py file rather than having to re-upload or execute SQL commands on a SQL platform. Pandas query is also intuitive and does not take long to learn. I hope you found the examples in this article to be useful. Thank you for reading!
[1] Mélody P, Unsplash (2017)
[2] pandas, pandas (2020)
[3] M.Przybyla, screenshot (2020)
[4] M.Przybyla, gist (2020)