How to Replace Null Values in Spark DataFrames | Towards Data Science

The replacement of null values in PySpark DataFrames is one of the most common operations undertaken. This can be achieved by using either DataFrame.fillna() or DataFrameNaFunctions.fill() methods. In today’s article we are going to discuss the main difference between these two functions.
While working with Spark DataFrames, many operations that we typically perform over them may return null values in some of the records. From that point onwards, some other operations may result in error if null/empty values are observed and thus we have to somehow replace these values in order to keep processing a DataFrame.
Additionally, when reporting tables (e.g. when outputting them into csv files) it is quite common to avoid the inclusion of empty values. For instance if an operation that was executed to create counts returns null values, it is more elegant to replace these values with 0.
Before start discussing how to replace null values in PySpark and exploring the difference between fill() and fillNa(), let’s create a sample DataFrame that will use as a reference throughout the article.
from pyspark.sql import SparkSessionspark_session = SparkSession.builder \    .master('local[1]') \    .appName('Example') \    .getOrCreate()df = spark_session.createDataFrame(    [       (1, 'UK', 'London', None),       (2, 'US', 'New York', 8000000),       (3, 'US', 'Washington DC', None),       (4, 'UK', 'Manchester', 550000),    ],    ['id', 'country', 'city', 'population'])df.show()+---+---------+--------------+-----------+| id|  country|         city | population|+---+---------+--------------+-----------+|  1|       UK|        London|       null||  2|       US|      New York|    8000000||  3|       US| Washington DC|       null||  4|       UK|    Manchester|     550000|+---+---------+--------------+-----------+
pyspark.sql.DataFrame.fillna() function was introduced in Spark version 1.3.1 and is used to replace null values with another specified value. It accepts two parameters namely value and subset.
value corresponds to the desired value you want to replace nulls with. If the value is a dict object then it should be a mapping where keys correspond to column names and values to replacement values that must be int, float, bool or str.
subset corresponds to a list of column names that will be considered when replacing null values. If value parameter is a dict then this parameter will be ignored.
Now if we want to replace all null values in a DataFrame we can do so by simply providing only the value parameter: df.na.fill(value=0).show()#Replace Replace 0 for null on only population column df.na.fill(value=0,subset=["population"]).show()
df.fillna(value=0).show()+---+---------+--------------+-----------+| id|  country|         city | population|+---+---------+--------------+-----------+|  1|       UK|        London|          0||  2|       US|      New York|    8000000||  3|       US| Washington DC|          0||  4|       UK|    Manchester|     550000|+---+---------+--------------+-----------+
The above operation will replace all null values in integer columns with the value of 0.
We can even specify the column name explicitly using the subset parameter:
df.fillna(value=0, subset=['population']).show()+---+---------+--------------+-----------+| id|  country|         city | population|+---+---------+--------------+-----------+|  1|       UK|        London|          0||  2|       US|      New York|    8000000||  3|       US| Washington DC|          0||  4|       UK|    Manchester|     550000|+---+---------+--------------+-----------+
Now pyspark.sql.DataFrameNaFunctions.fill() (which again was introduced back in version 1.3.1) is an alias to pyspark.sql.DataFrame.fillna() and both of the methods will lead to the exact same result.
As we can see below the results with na.fill() are identical to those observed when pyspark.sql.DataFrame.fillna() was applied to the DataFrames.
df.na.fill(value=0).show()+---+---------+--------------+-----------+| id|  country|         city | population|+---+---------+--------------+-----------+|  1|       UK|        London|          0||  2|       US|      New York|    8000000||  3|       US| Washington DC|          0||  4|       UK|    Manchester|     550000|+---+---------+--------------+-----------+
The above operation will replace all null values in integer columns with the value of 0. Similarly, we can explicitly specify the column name using the subset parameter:
df.na.fill(value=0, subset=['population']).show()+---+---------+--------------+-----------+| id|  country|         city | population|+---+---------+--------------+-----------+|  1|       UK|        London|          0||  2|       US|      New York|    8000000||  3|       US| Washington DC|          0||  4|       UK|    Manchester|     550000|+---+---------+--------------+-----------+
In today’s article we discussed why it is sometimes important to replace null values in a Spark DataFrame. Additionally, we discussed how to use fillna() and fill() in order to do so which are essentially alias to each other.
You can find more Spark related articles below.