How to display a PySpark DataFrame in table format ? - GeeksforGeeks

27 Jul, 2021
In this article, we are going to display the data of the PySpark dataframe in table format. We are going to use show() function and toPandas function to display the dataframe in the required format.
show(): Used to display the dataframe.
Syntax: dataframe.show( n, vertical = True, truncate = n)
where,
dataframe is the input dataframeN is the number of rows to be displayed from the top ,if n is not specified it will print entire rows in the dataframevertical parameter specifies the data in the dataframe displayed in vertical format if it is true, otherwise it will display in horizontal format like a dataframetruncate is a parameter us used to trim the values in the dataframe given as a number to trim
dataframe is the input dataframe
N is the number of rows to be displayed from the top ,if n is not specified it will print entire rows in the dataframe
vertical parameter specifies the data in the dataframe displayed in vertical format if it is true, otherwise it will display in horizontal format like a dataframe
truncate is a parameter us used to trim the values in the dataframe given as a number to trim
toPanads(): Pandas stand for a panel data structure which is used to represent data in a two-dimensional format like a table.
Syntax: dataframe.toPandas()
where, dataframe is the input dataframe
Let’s create a sample dataframe.
Python3
# importing moduleimport pyspark # importing sparksession from# pyspark.sql modulefrom pyspark.sql import SparkSession # creating sparksession and giving# an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate() # list  of employee data with 5 row valuesdata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 2"],        ["3", "bobby", "company 3"],        ["4", "rohith", "company 2"],        ["5", "gnanesh", "company 1"]] # specify column namescolumns = ['Employee ID', 'Employee NAME', 'Company Name'] # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns) print(dataframe)
Output:
DataFrame[Employee ID: string, Employee NAME: string, Company Name: string]
Example 1: Using show() function without parameters. It will result in the entire dataframe as we have.
Python3
# Display df using show()dataframe.show()
Output:
Example 2: Using show() function with n as a parameter, which displays top n rows.
Syntax: DataFrame.show(n)
Where, n is a row
Code:
Python3
# show() function to get 2 rowsdataframe.show(2)
Output:
Example 3:
Using show() function with vertical = True as parameter. Display the records in the dataframe vertically.
Syntax: DataFrame.show(vertical)
vertical can be either true and false.
Code:
Python3
# display dataframe evrticallydataframe.show(vertical = True)
Output:
Example 4: Using show() function with truncate as a parameter. Display first one letter in each value of all the columns
Python3
# display dataframe with truncatedataframe.show(truncate = 1)
Output:
Example 5: Using show() with all parameters.
Python3
# display dataframe with all parametersdataframe.show(n=3,vertical=True,truncate=2)
Output:
Example 6: Using toPandas() method, which converts it to Pandas Dataframe which perfectly looks like a table.
Python3
# display dataframe by using topandas() functiondataframe.toPandas()
Output:
gulshankumarar231
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
Read a file line by line in Python
How to Install PIP on Windows ?
Enumerate() in Python
Different ways to create Pandas Dataframe
Iterate over a list in Python
Python String | replace()
*args and **kwargs in Python
Reading and Writing to text files in Python
Create a Pandas DataFrame from Lists