distinct() vs dropDuplicates() in Apache Spark | by Giorgos Myrianthous | Towards Data Science

The Spark DataFrame API comes with two functions that can be used in order to remove duplicates from a given DataFrame. These are distinct() and dropDuplicates() . Even though both methods pretty much do the same job, they actually come with one difference which is quite important in some use cases.
In this article, we are going to explore how both of these functions work and what their main difference is. Additionally, we will discuss when to use one over the other.
Note that the examples that we’ll use to explore these methods have been constructed using the Python API. However, they are fairly simple and thus can be used using the Scala API too (even though some links provided will refer to the former API).
distinct()
Returns a new DataFrame containing the distinct rows in this DataFrame.
distinct() will return the distinct rows of the DataFrame. As an example consider the following DataFrame
>>> df.show()+---+------+---+                                                                | id|  name|age|+---+------+---+|  1|Andrew| 25||  1|Andrew| 25||  1|Andrew| 26||  2| Maria| 30|+---+------+---+
The method take no arguments and thus all columns are taken into account when dropping the duplicates:
>>> df.distinct().show()+---+------+---+| id|  name|age|+---+------+---+|  1|Andrew| 26||  2| Maria| 30||  1|Andrew| 25|+---+------+---+
Now if you need to consider only a subset of the columns when dropping duplicates, then you first have to make a column selection before calling distinct() as shown below.
>>> df.select(['id', 'name']).distinct().show()+---+------+| id|  name|+---+------+|  2| Maria||  1|Andrew|+---+------+
This means that the returned DataFrame will contain only the subset of the columns that was used to eliminate the duplicates. If that’s the case, then probably distinct() won’t do the trick.
dropDuplicates(subset=None)
Return a new DataFrame with duplicate rows removed, optionally only considering certain columns.
For a static batch DataFrame, it just drops duplicate rows. For a streaming DataFrame, it will keep all data across triggers as intermediate state to drop duplicates rows. You can use withWatermark() to limit how late the duplicate data can be and system will accordingly limit the state. In addition, too late data older than watermark will be dropped to avoid any possibility of duplicates.
drop_duplicates() is an alias for dropDuplicates().
Now dropDuplicates() will drop the duplicates detected over a specified set of columns (if provided) but in contrast to distinct() , it will return all the columns of the original dataframe. For instance, if you want to drop duplicates by considering all the columns you could run the following command
>>> df.dropDuplicates().show()+---+------+---+| id|  name|age|+---+------+---+|  1|Andrew| 26||  2| Maria| 30||  1|Andrew| 25|+---+------+---+
Therefore, dropDuplicates() is the way to go if you want to drop duplicates over a subset of columns, but at the same time you want to keep all the columns of the original structure.
df.dropDuplicates(['id', 'name']).show()+---+------+---+| id|  name|age|+---+------+---+|  2| Maria| 30||  1|Andrew| 25|+---+------+---+
In this article we explored two useful functions of the Spark DataFrame API, namely the distinct() and dropDuplicates() methods. Both can be used to eliminate duplicated rows of a Spark DataFrame however, their difference is that distinct() takes no arguments at all, while dropDuplicates() can be given a subset of columns to consider when dropping duplicated records.
This means that dropDuplicates() is a more suitable option when one wants to drop duplicates by considering only a subset of the columns but at the same time all the columns of the original DataFrame should be returned.