Delete rows in PySpark dataframe based on multiple conditions - GeeksforGeeks

29 Jun, 2021
In this article, we are going to see how to delete rows in PySpark dataframe based on multiple conditions.
Here we are going to use the logical expression to filter the row. Filter() function is used to filter the rows from RDD/DataFrame based on the given condition or SQL expression.
Syntax: filter( condition)
Parameters: 
Condition: Logical condition or SQL expression
Example 1: 
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql# modulefrom pyspark.sql import SparkSession  # spark library importimport pyspark.sql.functions  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "Amit", " DU"],        ["2", "Mohit", "DU"],        ["3", "rohith", "BHU"],        ["4", "sridevi", "LPU"],        ["1", "sravan", "KLMP"],        ["5", "gnanesh", "IIT"]]  # specify column namescolumns = ['student_ID', 'student_NAME', 'college']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  dataframe = dataframe.filter(dataframe.college != "IIT")  dataframe.show()
Output:
Example 2:
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql# modulefrom pyspark.sql import SparkSession  # spark library importimport pyspark.sql.functions  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "Amit", " DU"],        ["2", "Mohit", "DU"],        ["3", "rohith", "BHU"],        ["4", "sridevi", "LPU"],        ["1", "sravan", "KLMP"],        ["5", "gnanesh", "IIT"]]  # specify column namescolumns = ['student_ID', 'student_NAME', 'college']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  dataframe = dataframe.filter(    ((dataframe.college != "DU")     & (dataframe.student_ID != "3")))  dataframe.show()
Output:
It evaluates a list of conditions and returns a single value. Thus passing the condition and its required values will get the job done.
Syntax: When( Condition, Value)
Parameters:
Condition: Boolean or columns expression.
Value: Literal Value
Example: 
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql # modulefrom pyspark.sql import SparkSession  # spark library importimport pyspark.sql.functions  # spark library importfrom pyspark.sql.functions import when  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "Amit", " DU"],        ["2", "Mohit", "DU"],        ["3", "rohith", "BHU"],        ["4", "sridevi", "LPU"],        ["1", "sravan", "KLMP"],        ["5", "gnanesh", "IIT"]]  # specify column namescolumns = ['student_ID', 'student_NAME', 'college']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  dataframe.withColumn('New_col',                     when(dataframe.student_ID != '5', "True")                     .when(dataframe.student_NAME != 'gnanesh', "True")                     ).filter("New_col == True").drop("New_col").show()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
Enumerate() in Python
How to Install PIP on Windows ?
Different ways to create Pandas Dataframe
Python String | replace()
Create a Pandas DataFrame from Lists
Reading and Writing to text files in Python
*args and **kwargs in Python
How To Convert Python Dictionary To JSON?
sum() function in Python