Using Logical Comparisons With Pandas DataFrames | by Byron Dolon | Towards Data Science

Logical comparisons are used everywhere.
The Pandas library gives you a lot of different ways that you can compare a DataFrame or Series to other Pandas objects, lists, scalar values, and more. The traditional comparison operators (<, >, <=, >=, ==, !=) can be used to compare a DataFrame to another set of values.
However, you can also use wrappers for more flexibility in your logical comparison operations. These wrappers allow you to specify the axis for comparison, so you can choose to perform the comparison at the row or column level. Also, if you are working with a MultiIndex, you may specify which index you want to work with.
In this piece, we’ll first take a quick look at logical comparisons with the standard operators. After that, we’ll go through five different examples of how you can use these logical comparison wrappers to process and better understand your data.
The data used in this piece is sourced from Yahoo Finance. We’ll be using a subset of Tesla stock price data. Run the code below if you want to follow along. (And if you’re curious as to the function I used to get the data scroll to the very bottom and click on the first link.)
import pandas as pd# fixed data so sample data will stay the samedf = pd.read_html("https://finance.yahoo.com/quote/TSLA/history?period1=1277942400&period2=1594857600&interval=1d&filter=history&frequency=1d")[0]df = df.head(10) # only work with the first 10 points
The wrappers available for use are:
eq (equivalent to ==) — equals to
ne (equivalent to !=) — not equals to
le (equivalent to <=) — less than or equals to
lt (equivalent to <) — less than
ge (equivalent to >=) — greater than or equals to
gt (equivalent to >) — greater than
Before we dive into the wrappers, let’s quickly review how to perform a logical comparison in Pandas.
With the regular comparison operators, a basic example of comparing a DataFrame column to an integer would look like this:
old = df['Open'] >= 270
Here, we’re looking to see whether each value in the “Open” column is greater than or equal to the fixed integer “270”. However, if you try to run this, at first it won’t work.
You’ll most likely see this:
TypeError: '>=' not supported between instances of 'str' and 'int'
This is important to take care of now because when you use both the regular comparison operators and the wrappers, you’ll need to make sure that you are actually able to compare the two elements. Remember to do something like the following in your pre-processing, not just for these exercises, but in general when you’re analyzing data:
df = df.astype({"Open":'float',                "High":'float',                "Low":'float',                "Close*":'float',                "Adj Close**":'float',                "Volume":'float'})
Now, if you run the original comparison again, you’ll get this series back:
You can see that the operation returns a series of Boolean values. If you check the original DataFrame, you’ll see that there should be a corresponding “True” or “False” for each row where the value was greater than or equal to (>=) 270 or not.
Now, let’s dive into how you can do the same and more with the wrappers.
In the data set, you’ll see that there is a “Close*” column and an “Adj Close**” column. The Adjusted Close price is altered to reflect potential dividends and splits, whereas the Close price is only adjusted for splits. To see if these events may have happened, we can do a basic test to see if values in the two columns are not equal.
To do so, we run the following:
# is the adj close different from the close?df['Close Comparison'] = df['Adj Close**'].ne(df['Close*'])
Here, all we did is call the .ne() function on the “Adj Close**” column and pass “Close*”, the column we want to compare, as an argument to the function.
If we take a look at the resulting DataFrame, you’ll see that we‘ve created a new column “Close Comparison” that will show “True” if the two original Close columns are different and “False” if they are the same. In this case, you can see that the values for “Close*” and “Adj Close**” on every row are the same, so the “Close Comparison” only has “False” values. Technically, this would mean that we could remove the “Adj Close**” column, at least for this subset of data, since it only contains duplicate values to the “Close*” column.
We’d often like to see whether a stock’s price increased by the end of the day. One way to do this would be to see a “True” value if the “Close*” price was greater than the “Open” price or “False” otherwise.
To implement this, we run the following:
# is the close greater than the open?df['Bool Price Increase'] = df['Close*'].gt(df['Open'])
Here, we see that the “Close*” price at the end of the day was higher than the “Open” price at the beginning of the day 4/10 times in the first two weeks of July 2020. This might not be that informative because it’s such a small sample, but if you were to extend this to months or even years of data, it could indicate the overall trend of the stock (up or down).
So far, we’ve just been comparing columns to one another. You can also use the logical operators to compare values in a column to a scalar value like an integer. For example, let’s say that if the volume traded per day is greater than or equal to 100 million, we’ll call it a “High Volume” day.
To do so, we run the following:
# was the volume greater than 100m?df['High Volume'] = df['Volume'].ge(100000000)
Instead of passing a column to the logical comparison function, this time we simply have to pass our scalar value “100000000”.
Now, we can see that on 5/10 days the volume was greater than or equal to 100 million.
Earlier, we compared if the “Open” and “Close*” value in each row were different. It would be cool if instead, we compared the value of a column to the preceding value, to track an increase or decrease over time. Doing this means we can check if the “Close*” value for July 15 was greater than the value for July 14.
To do so, we run the following:
# was the close greater than yesterday's close?df['Close (t-1)'] = df['Close*'].shift(-1)df['Bool Over Time Increase'] = df['Close*'].gt(df['Close*'].shift(-1))
For illustration purposes, I included the “Close (t-1)” column so you can compare each row directly. In practice, you don’t need to add an entirely new column, as all we’re doing is passing the “Close*” column again into the logical operator, but we’re also calling shift(-1) on it to move all the values “up by one”.
What’s going on here is basically subtracting one from the index, so the value for July 14 moves “up”, which lets us compare it to the real value on July 15. As a result, you can see that on 7/10 days the “Close*” value was greater than the “Close*” value on the day before.
As a final exercise, let’s say that we developed a model to predict the stock prices for 10 days. We’ll store those predictions in a list, then compare the both the “Open” and “Close*” values of each day to the list values.
To do so, we run the following:
# did the open and close price match the predictions?predictions = [309.2, 303.36, 300, 489, 391, 445, 402.84, 274.32, 410, 223.93]df2 = df[['Open','Close*']].eq(predictions, axis='index')
Here, we’ve compared our generated list of predictions for the daily stock prices and compared it to the “Close*” column. To do so, we pass “predictions” into the eq() function and set axis='index'. By default, the comparison wrappers have axis='columns', but in this case, we actually want to work with each row in each column.
What this means is Pandas will compare “309.2”, which is the first element in the list, to the first values of “Open” and “Close*”. Then it will move on to the second value in the list and the second values of the DataFrame and so on. Remember that the index of a list and a DataFrame both start at 0, so you would look at “308.6” and “309.2” respectively for the first DataFrame column values (scroll back up if you want to double-check the results).
Based on these arbitrary predictions, you can see that there were no matches between the “Open” column values and the list of predictions. There were 4/10 matches between the “Close*” column values and the list of predictions.
I hope you found this very basic introduction to logical comparisons in Pandas using the wrappers useful. Remember to only compare data that can be compared (i.e. don’t try to compare a string to a float) and manually double-check the results to make sure your calculations are producing the intended results.
Go forth and compare!
More by me:- 2 Easy Ways to Get Tables From a Website- Top 4 Repositories on GitHub to Learn Pandas- An Introduction to the Cohort Analysis With Tableau- How to Quickly Create and Unpack Lists with Pandas- Learning to Forecast With Tableau in 5 Minutes Or Less