How to join on multiple columns in Pyspark?

19 Dec, 2021
In this article, we will discuss how to join multiple columns in PySpark Dataframe using Python.
Let’s create the first dataframe:
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [(1, "sravan"), (2, "ojsawi"), (3, "bobby")]  # specify column namescolumns = ['ID1', 'NAME1']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  dataframe.show()
Output:
Let’s create the second dataframe:
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [(1, "sravan"), (2, "ojsawi"),        (3, "bobby"),        (4, "rohith"), (5, "gnanesh")]  # specify column namescolumns = ['ID2', 'NAME2']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns)  dataframe1.show()
Output:
we can join the multiple columns by using join() function using conditional operator
Syntax: dataframe.join(dataframe1, (dataframe.column1== dataframe1.column1) & (dataframe.column2== dataframe1.column2))
where, 
dataframe is the first dataframe
dataframe1 is the second dataframe
column1 is the first matching column in both the dataframes
column2 is the second matching column in both the dataframes
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [(1, "sravan"), (2, "ojsawi"), (3, "bobby")]  # specify column namescolumns = ['ID1', 'NAME1']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # list  of employee datadata = [(1, "sravan"), (2, "ojsawi"), (3, "bobby"),        (4, "rohith"), (5, "gnanesh")]  # specify column namescolumns = ['ID2', 'NAME2']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns)  # join based on ID and name columndataframe.join(dataframe1, (dataframe.ID1 == dataframe1.ID2)               & (dataframe.NAME1 == dataframe1.NAME2)).show()
Output:
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [(1, "sravan"), (2, "ojsawi"), (3, "bobby")]  # specify column namescolumns = ['ID1', 'NAME1']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # list  of employee datadata = [(1, "sravan"), (2, "ojsawi"), (3, "bobby"),        (4, "rohith"), (5, "gnanesh")]  # specify column namescolumns = ['ID2', 'NAME2']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns)  # join based on ID and name columndataframe.join(dataframe1, (dataframe.ID1 == dataframe1.ID2)               | (dataframe.NAME1 == dataframe1.NAME2)).show()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
How to drop one or multiple columns in Pandas Dataframe
Python | os.path.join() method
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Python | Get unique values from a list
Python | datetime.timedelta() function