Merge two DataFrames with different amounts of columns in PySpark

21 Dec, 2021
In this article, we will discuss how to perform union on two dataframes with different amounts of columns in PySpark in Python.
Here we are having 3 columns named id, name, and address.
Python3
# importing moduleimport pyspark # import when and lit functionfrom pyspark.sql.functions import when, lit # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate() # list  of employee datadata = [["1", "sravan", "kakumanu"],        ["2", "ojaswi", "hyd"],        ["3", "rohith", "delhi"],        ["4", "sridevi", "kakumanu"],        ["5", "bobby", "guntur"]] # specify column namescolumns = ['ID', 'NAME', 'Address'] # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns) # displaydataframe1.show()
Output:
Here we are going to create dataframe with 2 columns
Python3
# importing moduleimport pyspark # import when and lit functionfrom pyspark.sql.functions import when, lit # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate() # list  of employee datadata = [["1", 23],        ["2", 21],        ["3", 32],        ] # specify column namescolumns = ['ID', 'Age'] # creating a dataframe from the lists of datadataframe2 = spark.createDataFrame(data, columns) # displaydataframe2.show()
Output:
We can not perform union operations because the columns are different, so we have to add the missing columns. Here In first dataframe (dataframe1) , the columns  [‘ID’, ‘NAME’, ‘Address’] and second dataframe (dataframe2 ) columns are  [‘ID’,’Age’].
Now we have to add the Age column to the first dataframe and NAME and Address in the second dataframe, we can do this by using lit() function. This function is available in pyspark.sql.functions which is used to add a column with a value. Here we are going to add a value with None.
Syntax:
for column in [column for column in dataframe1.columns if column not in dataframe2.columns]:
    dataframe2 = dataframe2.withColumn(column, lit(None))
where, 
dataframe1 is the firstdata frame
dataframe2 is the second dataframe
Python3
# importing moduleimport pyspark # import lit functionfrom pyspark.sql.functions import lit # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "kakumanu"],        ["2", "ojaswi", "hyd"],        ["3", "rohith", "delhi"],        ["4", "sridevi", "kakumanu"],        ["5", "bobby", "guntur"]] # specify column namescolumns = ['ID', 'NAME', 'Address'] # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns) # list  of employee datadata = [["1", 23],        ["2", 21],        ["3", 32],        ] # specify column namescolumns = ['ID', 'Age'] # creating a dataframe from the lists of datadataframe2 = spark.createDataFrame(data, columns) # add columns in dataframe1 that are missing from dataframe2for column in [column for column in dataframe2.columns               if column not in dataframe1.columns]:    dataframe1 = dataframe1.withColumn(column, lit(None)) # add columns in dataframe2 that are missing from dataframe1for column in [column for column in dataframe1.columns               if column not in dataframe2.columns]:    dataframe2 = dataframe2.withColumn(column, lit(None)) # now see the columns of dataframe1print(dataframe1.columns) # now see the columns of dataframe2print(dataframe2.columns)
Output:
['ID', 'NAME', 'Address', 'Age']
['ID', 'Age', 'NAME', 'Address']
Now we can perform union by using union() function. This function will join two dataframes.
Syntax: dataframe1.union(dataframe2)
Example:
Python3
# importing moduleimport pyspark # import lit functionfrom pyspark.sql.functions import lit # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "kakumanu"],        ["2", "ojaswi", "hyd"],        ["3", "rohith", "delhi"],        ["4", "sridevi", "kakumanu"],        ["5", "bobby", "guntur"]] # specify column namescolumns = ['ID', 'NAME', 'Address'] # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns) # list  of employee datadata = [["1", 23],        ["2", 21],        ["3", 32],        ] # specify column namescolumns = ['ID', 'Age'] # creating a dataframe from the lists of datadataframe2 = spark.createDataFrame(data, columns) # add columns in dataframe1 that are missing from dataframe2for column in [column for column in dataframe2.columns               if column not in dataframe1.columns]:    dataframe1 = dataframe1.withColumn(column, lit(None)) # add columns in dataframe2 that are missing from dataframe1for column in [column for column in dataframe1.columns               if column not in dataframe2.columns]:    dataframe2 = dataframe2.withColumn(column, lit(None)) # perform uniondataframe1.union(dataframe2).show()
Output:
Syntax: dataframe1.unionAll(dataframe2)
Python3
# importing moduleimport pyspark # import lit functionfrom pyspark.sql.functions import lit # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "kakumanu"],        ["2", "ojaswi", "hyd"],        ["3", "rohith", "delhi"],        ["4", "sridevi", "kakumanu"],        ["5", "bobby", "guntur"]] # specify column namescolumns = ['ID', 'NAME', 'Address'] # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns) # list  of employee datadata = [["1", 23],        ["2", 21],        ["3", 32],        ] # specify column namescolumns = ['ID', 'Age'] # creating a dataframe from the lists of datadataframe2 = spark.createDataFrame(data, columns) # add columns in dataframe1 that are missing# from dataframe2for column in [column for column in dataframe2.columns\               if column not in dataframe1.columns]:    dataframe1 = dataframe1.withColumn(column, lit(None)) # add columns in dataframe2 that are missing# from dataframe1for column in [column for column in dataframe1.columns \               if column not in dataframe2.columns]:    dataframe2 = dataframe2.withColumn(column, lit(None)) # perform unionAlldataframe1.unionAll(dataframe2).show()
Output:
We can also perform unionByName, This will join dataframes by name.
Syntax: dataframe1.unionByName(dataframe2)
Example:
Python3
# importing moduleimport pyspark # import lit functionfrom pyspark.sql.functions import lit # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "kakumanu"],        ["2", "ojaswi", "hyd"],        ["3", "rohith", "delhi"],        ["4", "sridevi", "kakumanu"],        ["5", "bobby", "guntur"]] # specify column namescolumns = ['ID', 'NAME', 'Address'] # creating a dataframe from the lists of datadataframe1 = spark.createDataFrame(data, columns) # list  of employee datadata = [["1", 23],        ["2", 21],        ["3", 32],        ] # specify column namescolumns = ['ID', 'Age'] # creating a dataframe from the lists of datadataframe2 = spark.createDataFrame(data, columns) # add columns in dataframe1 that are missing from dataframe2for column in [column for column in dataframe2.columns \               if column not in dataframe1.columns]:    dataframe1 = dataframe1.withColumn(column, lit(None)) # add columns in dataframe2 that are missing from dataframe1for column in [column for column in dataframe1.columns \               if column not in dataframe2.columns]:    dataframe2 = dataframe2.withColumn(column, lit(None)) # perform unionByNamedataframe1.unionByName(dataframe2).show()
Output:
sagartomar9927
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
Different ways to create Pandas Dataframe
Enumerate() in Python
Read a file line by line in Python
Python String | replace()
How to Install PIP on Windows ?
*args and **kwargs in Python
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON