How To Select Rows From PySpark DataFrames Based on Column Values | Towards Data Science

Filtering rows of DataFrames is among the most commonly performed operations in PySpark. In today’s short guide we will discuss how to select a range of rows based on certain conditions in a few different ways.
Specifically, we will explore how to perform row selection using
the filter() function
the where() function
Spark SQL
First, let’s create an example DataFrame that we will reference throughout this article in order to demonstrate a couple of concepts.
from pyspark.sql import SparkSession# Create an instance of spark sessionspark_session = SparkSession.builder \    .master('local[1]') \    .appName('Example') \    .getOrCreate()df = spark_session.createDataFrame(    [        (1, True, 1.0, 100),        (2, False, 2.0, 200),        (3, False, 3.0, 300),        (4, True, 4.0, 400),        (5, True, 5.0, 500),    ],    ['colA', 'colB', 'colC', 'colD'])df.show()+----+-----+----+----+|colA| colB|colC|colD|+----+-----+----+----+|   1| true| 1.0| 100||   2|false| 2.0| 200||   3|false| 3.0| 300||   4| true| 4.0| 400||   5| true| 5.0| 500|+----+-----+----+----+
The first option you have when it comes to filtering DataFrame rows is pyspark.sql.DataFrame.filter() function that performs filtering based on the specified conditions.
For example, say we want to keep only the rows whose values in colC are greater or equal to 3.0. The following expression will do the trick:
df = df.filter(df.colC >= 3.0)df.show()+----+-----+----+----+|colA| colB|colC|colD|+----+-----+----+----+|   3|false| 3.0| 300||   4| true| 4.0| 400||   5| true| 5.0| 500|+----+-----+----+----+
You can even specify Column functions such as pyspark.sql.Column.between in order to keep only rows between the specified lower and upper bounds, as shown below.
df = df.filter(df.colD.between(200, 400))df.show()+----+-----+----+----+|colA| colB|colC|colD|+----+-----+----+----+|   2|false| 2.0| 200||   3|false| 3.0| 300||   4| true| 4.0| 400|+----+-----+----+----+
pyspark.sql.DataFrame.where() is an alias to filter() we discussed in the previous section. It can be used in the same way in order to filter the rows of the DataFrame based on the conditions provided.
df = df.where(~df.colB)df.show()+----+-----+----+----+|colA| colB|colC|colD|+----+-----+----+----+|   2|false| 2.0| 200||   3|false| 3.0| 300|+----+-----+----+----+
Alternatively, you can even use Spark SQL in order to query the DataFrame using an SQL expression. For example,
# Create a view for the dataframedf.createOrReplaceTempView("df_view")df = spark_session.sql(    """    SELECT *     FROM df_view    WHERE colC >= 2.0    """)df.show()+----+-----+----+----+|colA| colB|colC|colD|+----+-----+----+----+|   2|false| 2.0| 200||   3|false| 3.0| 300||   4| true| 4.0| 400||   5| true| 5.0| 500|+----+-----+----+----+
In today’s short guide we discussed how to perform row selection from PySpark DataFrames based on specific conditions. Specifically, we showcased how to do so using filter() and where() methods as well as Spark SQL.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read. You’ll also get full access to every story on Medium.
You may also like