Show distinct column values in PySpark dataframe

06 Jun, 2021
In this article, we are going to display the distinct column values from dataframe using pyspark in Python. For this, we are using distinct() and dropDuplicates() functions along with select() function.
Let’s create a sample dataframe.
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee data with 5 row valuesdata =[["1", "sravan", "company 1"],       ["3", "bobby", "company 3"],       ["2", "ojaswi", "company 2"],       ["1", "sravan", "company 1"],       ["3", "bobby", "company 3"],       ["4", "rohith", "company 2"],       ["5", "gnanesh", "company 1"]]  # specify column namescolumns = ['Employee ID','Employee NAME','Company Name']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data,columns)  dataframe.show()
Output:
This function returns distinct values from column using distinct() function.
Syntax: dataframe.select(“column_name”).distinct().show()
Example1: For a single column.
Python3
# select first column to get# unique data using distinct function()dataframe.select("Employee ID").distinct().show()
Output:
+-----------+
|Employee ID|
+-----------+
|          3|
|          5|
|          1|
|          4|
|          2|
+-----------+
Example 2: For multiple columns.
Python code to display unique data from 2 columns using distinct() function.
Syntax: dataframe.select(“column_name 1, column_name 2 “).distinct().show()
Code:
Python3
# select first and second column# to get unique data using distinct function()dataframe.select(["Employee ID",                  "Employee NAME"]).distinct().show()
Output:
+-----------+-------------+
|Employee ID|Employee NAME|
+-----------+-------------+
|          5|      gnanesh|
|          4|       rohith|
|          1|       sravan|
|          2|       ojaswi|
|          3|        bobby|
+-----------+-------------+
This function displays unique data in one column from dataframe using dropDuplicates() function.
Syntax: dataframe.select(“column_name”).dropDuplicates().show()
Example 1: For single columns.
Python3
#select first  column to get# unique data using dropDuplicates function()dataframe.select("Employee ID").dropDuplicates().show()
Output:
+-----------+
|Employee ID|
+-----------+
|          3|
|          5|
|          1|
|          4|
|          2|
+-----------+
Example 2: For multiple columns
Python code to display unique data from 2 columns using dropDuplicates() function
Python3
#select first and second column# to get unique data using dropDuplicates function()dataframe.select(["Employee ID",                  "Employee NAME"]).dropDuplicates().show()
Output:
+-----------+-------------+
|Employee ID|Employee NAME|
+-----------+-------------+
|          5|      gnanesh|
|          4|       rohith|
|          1|       sravan|
|          2|       ojaswi|
|          3|        bobby|
+-----------+-------------+
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
Python | os.path.join() method
How to drop one or multiple columns in Pandas Dataframe
How To Convert Python Dictionary To JSON?
Check if element exists in list in Python
Python | Get unique values from a list
Python | datetime.timedelta() function