Extract First and last N rows from PySpark DataFrame - GeeksforGeeks

06 Jun, 2021
In this article, we are going to get the extract first N rows and Last N rows from the dataframe using PySpark in Python. To do our task first we will create a sample dataframe.
We have to create a spark object with the help of the spark session and give the app name by using getorcreate() method.
spark = SparkSession.builder.appName('sparkdf').getOrCreate()
Finally, after creating the data with the list and column list to the method:
dataframe = spark.createDataFrame(data, columns)
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee data with 5 row valuesdata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 2"],        ["3", "bobby", "company 3"],        ["4", "rohith", "company 2"],        ["5", "gnanesh", "company 1"]]  # specify column namescolumns = ['Employee ID', 'Employee NAME', 'Company Name']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  print('Actual data in dataframe')dataframe.show()
Output:
We can extract the first N rows by using several methods which are discussed below with the help of some examples:
Method 1: Using head()
This function is used to extract top N rows in the given dataframe
Syntax: dataframe.head(n)
where, 
n specifies the number of rows to be extracted from first
dataframe is the dataframe name created from the nested lists using pyspark.
Python3
print("Top 2 rows ")  # extract top 2 rowsa = dataframe.head(2)print(a)  print("Top 1 row ")  # extract top 1 rowa = dataframe.head(1)print(a)
Output:
Top 2 rows  
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′), 
Row(Employee ID=’2′, Employee NAME=’ojaswi’, Company Name=’company 2′)]
Top 1 row  
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)]
Method 2: Using first()
This function is used to extract only one row in the dataframe.
Syntax: dataframe.first()
It doesn’t take any parameter
dataframe is the dataframe name created from the nested lists using pyspark
Python3
print("Top row ")  # extract top  rowa = dataframe.first()print(a)
Output:
Top row  
Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)
Method 3: Using show() 
Used to display the dataframe from top to bottom by default.
Syntax: dataframe.show(n)
where,
dataframe is the input dataframe
n is the number of rows to be displayed from the top ,if n is not specified it will print entire rows in the dataframe
Python3
# show() function to get # 2 rowsdataframe.show(2)
Output:
Extracting the last rows means getting the last N rows from the given dataframe. For this, we are using tail() function and can get the last N rows
Syntax: dataframe.tail(n)
where,
n is the number to get last n rows
data frame is the input dataframe
Example:
Python3
print("Last 2 rows ")  # extract last 2 rowsa = dataframe.tail(2)print(a)  print("Last 1 row ")  # extract last 1 rowa = dataframe.tail(1)print(a)
Output:
Last 2 rows  
[Row(Employee ID=’4′, Employee NAME=’rohith’, Company Name=’company 2′), 
Row(Employee ID=’5′, Employee NAME=’gnanesh’, Company Name=’company 1′)]
Last 1 row  
[Row(Employee ID=’5′, Employee NAME=’gnanesh’, Company Name=’company 1′)]
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
How To Convert Python Dictionary To JSON?
Check if element exists in list in Python
How to drop one or multiple columns in Pandas Dataframe
Python Classes and Objects
Python | os.path.join() method
Create a directory in Python
Defaultdict in Python
Python | Get unique values from a list
Python | Pandas dataframe.groupby()