How to create an empty PySpark DataFrame ?

11 Aug, 2021
In this article, we are going to see how to create an empty PySpark dataframe. Empty Pysaprk dataframe is a dataframe containing no data and may or may not specify the schema of the dataframe.
We’ll first create an empty RDD by specifying an empty schema.
emptyRDD() method creates an RDD without any data.
createDataFrame() method creates a pyspark dataframe with the specified data and schema of the dataframe.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import * # Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate() # Create an empty RDDemp_RDD = spark.sparkContext.emptyRDD() # Create empty schemacolumns = StructType([]) # Create an empty RDD with empty schemadata = spark.createDataFrame(data = emp_RDD,                             schema = columns) # Print the dataframeprint('Dataframe :')data.show() # Print the schemaprint('Schema :')data.printSchema()
Output: 
Dataframe :
++
||
++
++

Schema :
root
It is possible that we will not get a file for processing. However, we must still manually create a DataFrame with the appropriate schema.
Specify the schema of the dataframe as columns = [‘Name’, ‘Age’, ‘Gender’].
Create an empty RDD with an expecting schema.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import * # Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate() # Create an empty RDDemp_RDD = spark.sparkContext.emptyRDD() # Create an expected schemacolumns = StructType([StructField('Name',                                  StringType(), True),                    StructField('Age',                                StringType(), True),                    StructField('Gender',                                StringType(), True)]) # Create an empty RDD with expected schemadf = spark.createDataFrame(data = emp_RDD,                           schema = columns) # Print the dataframeprint('Dataframe :')df.show() # Print the schemaprint('Schema :')df.printSchema()
Output :
Dataframe :
+----+---+------+
|Name|Age|Gender|
+----+---+------+
+----+---+------+

Schema :
root
 |-- Name: string (nullable = true)
 |-- Age: string (nullable = true)
 |-- Gender: string (nullable = true)
Create an empty schema as columns.
Specify data as empty([]) and schema as columns in CreateDataFrame() method.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import * # Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate() # Create an empty schemacolumns = StructType([]) # Create an empty dataframe with empty schemadf = spark.createDataFrame(data = [],                           schema = columns) # Print the dataframeprint('Dataframe :')df.show() # Print the schemaprint('Schema :')df.printSchema()
Output:
Dataframe :
++
||
++
++

Schema :
root
Specify the schema of the dataframe as columns = [‘Name’, ‘Age’, ‘Gender’].
Specify data as empty([]) and schema as columns in CreateDataFrame() method.
Code:
Python3
# Import necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql.types import * # Create a spark sessionspark = SparkSession.builder.appName('Empty_Dataframe').getOrCreate() # Create an expected schemacolumns = StructType([StructField('Name',                                  StringType(), True),                    StructField('Age',                                StringType(), True),                    StructField('Gender',                                StringType(), True)]) # Create a dataframe with expected schemadf = spark.createDataFrame(data = [],                           schema = columns) # Print the dataframeprint('Dataframe :')df.show() # Print the schemaprint('Schema :')df.printSchema()
Output :
Dataframe :
+----+---+------+
|Name|Age|Gender|
+----+---+------+
+----+---+------+

Schema :
root
 |-- Name: string (nullable = true)
 |-- Age: string (nullable = true)
 |-- Gender: string (nullable = true)
surindertarika1234
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.