Full outer join in PySpark dataframe - GeeksforGeeks

19 Dec, 2021
In this article, we are going to see how to perform Full Outer Join in PySpark DataFrames in Python.
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  dataframe.show()
Output:
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata1 = [["1", "45000", "IT"],          ["2", "145000", "Manager"],          ["6", "45000", "HR"],         ["5", "34000", "Sales"]]  # specify column namescolumns = ['ID', 'salary', 'department']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data1, columns)  dataframe1.show()
Output:
This is used to join the two PySpark dataframes with all rows and columns using full keyword
Syntax: dataframe1.join(dataframe2,dataframe1.column_name ==  dataframe2.column_name,”full”).show()
where
dataframe1 is the first PySpark dataframe
dataframe2 is the second PySpark dataframe
column_name is the column with respect to dataframe
Example: Python program to join two dataframes based on the ID column.
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],         ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # list  of employee datadata1 = [["1", "45000", "IT"],          ["2", "145000", "Manager"],         ["6", "45000", "HR"],         ["5", "34000", "Sales"]]  # specify column namescolumns = ['ID', 'salary', 'department']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data1, columns)    # join two dataframes based on# ID column using full keyworddataframe.join(dataframe1,               dataframe.ID == dataframe1.ID,                "full").show()
Output:
This is used to join the two PySpark dataframes with all rows and columns using fullouter keyword
Syntax: dataframe1.join(dataframe2,dataframe1.column_name ==  dataframe2.column_name,”fullouter”).show()
where
dataframe1 is the first PySpark dataframe
dataframe2 is the second PySpark dataframe
column_name is the column with respect to dataframe
Example: Python program to join two dataframes based on the ID column.
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],         ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # list  of employee datadata1 = [["1", "45000", "IT"],          ["2", "145000", "Manager"],         ["6", "45000", "HR"],         ["5", "34000", "Sales"]]  # specify column namescolumns = ['ID', 'salary', 'department']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data1, columns)    # join two dataframes based on ID# column using full outer keyworddataframe.join(dataframe1,               dataframe.ID == dataframe1.ID,                "fullouter").show()
Output:
This is used to join the two PySpark dataframes with all rows and columns using the outer keyword.
Syntax: dataframe1.join(dataframe2,dataframe1.column_name ==  dataframe2.column_name,”outer”).show()
where,
dataframe1 is the first PySpark dataframe
dataframe2 is the second PySpark dataframe
column_name is the column with respect to dataframe
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],         ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # list  of employee datadata1 = [["1", "45000", "IT"],         ["2", "145000", "Manager"],         ["6", "45000", "HR"],         ["5", "34000", "Sales"]]  # specify column namescolumns = ['ID', 'salary', 'department']  # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data1, columns)    # join two dataframes based on# ID column using outer keyworddataframe.join(dataframe1,               dataframe.ID == dataframe1.ID,                "outer").show()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
How to drop one or multiple columns in Pandas Dataframe
How To Convert Python Dictionary To JSON?
Selecting rows in pandas DataFrame based on conditions
Check if element exists in list in Python
Defaultdict in Python
Python | Get unique values from a list
Python | os.path.join() method
Create a directory in Python
Python | Pandas dataframe.groupby()