Filter PySpark DataFrame Columns with None or Null Values - GeeksforGeeks

09 May, 2021
Many times while working on PySpark SQL dataframe, the dataframes contains many NULL/None values in columns, in many of the cases before performing any of the operations of the dataframe firstly we have to handle the NULL/None values in order to get the desired result or output, we have to filter those NULL values from the dataframe.
In this article are going to learn how to filter the PySpark dataframe column with NULL/None values.
For filtering the NULL/None values we have the function in PySpark API know as a filter() and with this function, we are using isNotNull() function.
Syntax: 
df.filter(condition) : This function returns the new dataframe with the values which satisfies the given condition.
df.column_name.isNotNull() : This function is used to filter the rows that are not NULL/None in the dataframe column.
Example 1: Filtering PySpark dataframe column with None value
In the below code we have created the Spark Session, and then we have created the Dataframe which contains some None values in every column. Now, we have filtered the None values present in the Name column using filter() in which we have passed the condition df.Name.isNotNull() to filter the None values of Name column.
Python
# importing necessary librariesfrom pyspark.sql import SparkSession  # function to create SparkSessiondef create_session():    spk = SparkSession.builder \        .master("local") \        .appName("Filter_values.com") \        .getOrCreate()    return spk  # function to create dataframedef create_df(spark, data, schema):    df1 = spark.createDataFrame(data, schema)    return df1    if __name__ == "__main__":      # calling function to create SparkSession    spark = create_session()      input_data = [("Shivansh", "Data Scientist", "Noida"),                  (None, "Software Developer", None),                  ("Swati", "Data Analyst", "Hyderabad"),                  (None, None, "Noida"),                  ("Arpit", "Android Developer", "Banglore"),                  (None, None, None)]          schema = ["Name", "Job Profile", "City"]      # calling function to create dataframe    df = create_df(spark, input_data, schema)      # filtering the columns with None values    df = df.filter(df.Name.isNotNull())      # visulizing the dataframe    df.show()
Output:
Original Dataframe
Dataframe after filtering NULL/None values
Example 2: Filtering PySpark dataframe column with NULL/None values using filter() function
In the below code we have created the Spark Session, and then we have created the Dataframe which contains some None values in every column. Now, we have filtered the None values present in the City column using filter() in which we have passed the condition in English language form i.e, “City is Not Null” This is the condition to filter the None values of the City column.
Note: The condition must be in double-quotes.
Python
# importing necessary librariesfrom pyspark.sql import SparkSession  # function to create new SparkSessiondef create_session():    spk = SparkSession.builder \        .master("local") \        .appName("Filter_values.com") \        .getOrCreate()    return spk    def create_df(spark, data, schema):    df1 = spark.createDataFrame(data, schema)    return df1    if __name__ == "__main__":      # calling function to create SparkSession    spark = create_session()      input_data = [("Shivansh", "Data Scientist", "Noida"),                  (None, "Software Developer", None),                  ("Swati", "Data Analyst", "Hyderabad"),                  (None, None, "Noida"),                  ("Arpit", "Android Developer", "Banglore"),                  (None, None, None)]          schema = ["Name", "Job Profile", "City"]      # calling function to create dataframe    df = create_df(spark, input_data, schema)      # filtering the columns with None values    df = df.filter("City is Not NULL")      # visulizing the dataframe    df.show()
Output:
Original Dataframe
After filtering NULL/None values from the city column
Example 3: Filter columns with None values using filter() when column name has space
In the below code, we have created the Spark Session, and then we have created the Dataframe which contains some None values in every column. We have filtered the None values present in the ‘Job Profile’ column using filter() function in which we have passed the condition df[“Job Profile”].isNotNull() to filter the None values of the Job Profile column.
Note: For accessing the column name which has space between the words, is accessed by using square brackets [] means with reference to the dataframe we have to give the name using square brackets.
Python
# importing necessary librariesfrom pyspark.sql import SparkSession  # function to create SparkSessiondef create_session():    spk = SparkSession.builder \        .master("local") \        .appName("Filter_values.com") \        .getOrCreate()    return spk    def create_df(spark, data, schema):    df1 = spark.createDataFrame(data, schema)    return df1    if __name__ == "__main__":      # calling function to create SparkSession    spark = create_session()      input_data = [("Shivansh", "Data Scientist", "Noida"),                  (None, "Software Developer", None),                  ("Swati", "Data Analyst", "Hyderabad"),                  (None, None, "Noida"),                  ("Arpit", "Android Developer", "Banglore"),                  (None, None, None)]          schema = ["Name", "Job Profile", "City"]      # calling function to create dataframe    df = create_df(spark, input_data, schema)      # filtering the Job Profile with None values    df = df.filter(df["Job Profile"].isNotNull())      # visulizing the dataframe    df.show()
Output:
Original Dataframe
After filtering NULL/None values from the Job Profile column
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
How to Install PIP on Windows ?
Enumerate() in Python
Different ways to create Pandas Dataframe
*args and **kwargs in Python
Reading and Writing to text files in Python
Create a Pandas DataFrame from Lists
Convert integer to string in Python
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?