How to Convert Pandas to PySpark DataFrame ?

23 May, 2021
In this article, we will learn How to Convert Pandas to PySpark DataFrame. Sometimes we will get csv, xlsx, etc. format data, and we have to store it in PySpark DataFrame and that can be done by loading data in Pandas then converted PySpark DataFrame. For conversion, we pass the Pandas dataframe into the CreateDataFrame() method.
Syntax: spark.createDataframe(data, schema)
Parameter:
data – list of values on which dataframe is created.
schema – It’s the structure of dataset or list of column names.
where spark is the SparkSession object.
Example 1: Create a DataFrame and then Convert using spark.createDataFrame() method
Python3
# import the pandasimport pandas as pd  # from  pyspark library import # SparkSessionfrom pyspark.sql import SparkSession  # Building the SparkSession and name# it :'pandas to spark'spark = SparkSession.builder.appName(  "pandas to spark").getOrCreate()  # Create the DataFrame with the help # of pd.DataFrame()data = pd.DataFrame({'State': ['Alaska', 'California',                               'Florida', 'Washington'],                                            'city': ["Anchorage", "Los Angeles",                               "Miami", "Bellevue"]})  # create DataFramedf_spark = spark.createDataFrame(data)  df_spark.show()
Output:
Example 2: Create a DataFrame and then Convert using spark.createDataFrame() method
In this method, we are using Apache Arrow to convert Pandas to Pyspark DataFrame.
Python3
import the pandasimport pandas as pd  # from  pyspark library import # SparkSessionfrom pyspark.sql import SparkSession  # Building the SparkSession and name # it :'pandas to spark'spark = SparkSession.builder.appName(  "pandas to spark").getOrCreate()  # Create the DataFrame with the help # of pd.DataFrame()data = pd.DataFrame({'State': ['Alaska', 'California',                               'Florida', 'Washington'],                                            'city': ["Anchorage", "Los Angeles",                              "Miami", "Bellevue"]})    # enableing the Apache Arrow for converting# Pandas to pySpark DF(DataFrame)spark.conf.set("spark.sql.execution.arrow.enabled", "true")  # Creating the DataFramesprak_arrow = spark.createDataFrame(data)  # Show the DataFramesprak_arrow.show()
Output:
Example 3: Load a DataFrame from CSV and then Convert 
In this method, we can easily read the CSV file in Pandas Dataframe as well as in Pyspark Dataframe. The dataset used here is heart.csv.
Python3
# import the pandas libraryimport pandas as pd  # Read the Dataset in Pandas Dataframedf_pd = pd.read_csv('heart.csv')  # Show the dataset here head() # will return top 5 rowsdf_pd.head()
Output:
Python3
# Reading the csv file in # Pyspark DataFramedf_spark2 = spark.read.option(  'header', 'true').csv("heart.csv")  # Showing the data in the from of # table and showing only top 5 rowsdf_spark2.show(5)
Output:
We can also convert pyspark Dataframe to pandas Dataframe. For this, we will use DataFrame.toPandas() method.
Syntax: DataFrame.toPandas()
Returns the contents of this DataFrame as Pandas pandas.DataFrame.
Python3
# Convert Pyspark DataFrame to # Pandas DataFrame by toPandas() # Function head() will show only# top 5 rows of the datasetdf_spark2.toPandas().head()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Different ways to create Pandas Dataframe
Enumerate() in Python
How to Install PIP on Windows ?
*args and **kwargs in Python
Python Classes and Objects
Python OOPs Concepts
Convert integer to string in Python
Introduction To PYTHON
How to drop one or multiple columns in Pandas Dataframe
Python | os.path.join() method