Get specific row from PySpark dataframe - GeeksforGeeks

18 Jul, 2021
In this article, we will discuss how to get the specific row from the PySpark dataframe.
Creating Dataframe for demonstration:
Python3
# importing moduleimport pyspark  # importing sparksession# from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession# and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee data with 5 row valuesdata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 2"],        ["3", "bobby", "company 3"],        ["4", "rohith", "company 2"],        ["5", "gnanesh", "company 1"]]  # specify column namescolumns = ['Employee ID', 'Employee NAME',           'Company Name']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # display dataframedataframe.show()
Output:
This is used to get the all row’s data from the dataframe in list format.
Syntax: dataframe.collect()[index_position]
Where,
dataframe is the pyspark dataframe
index_position is the index row in dataframe
Example: Python code to access rows
Python3
# get first rowprint(dataframe.collect()[0])  # get second rowprint(dataframe.collect()[1])  # get last rowprint(dataframe.collect()[-1])  # get third rowprint(dataframe.collect()[2])
Output:
Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)
Row(Employee ID=’2′, Employee NAME=’ojaswi’, Company Name=’company 2′)
Row(Employee ID=’5′, Employee NAME=’gnanesh’, Company Name=’company 1′)
Row(Employee ID=’3′, Employee NAME=’bobby’, Company Name=’company 3′)
This function is used to get the top n rows from the pyspark dataframe.
Syntax: dataframe.show(no_of_rows)
where, no_of_rows is the row number to get the data
Example: Python code to get the data using show() function
Python3
# display dataframe only top 2 rowsprint(dataframe.show(2))  # display dataframe only top 1 rowprint(dataframe.show(1))  # display dataframe print(dataframe.show())
Output:
This function is used to return only the first row in the dataframe.
Syntax: dataframe.first()
Example: Python code to select the first row in the dataframe.
Python3
# display first row of the dataframeprint(dataframe.first())
Output:
Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)
This method is used to display top n rows in the dataframe.
Syntax: dataframe.head(n)
where, n is the number of rows to be displayed
Example: Python code to display the number of rows to be displayed.
Python3
# display only 1 rowprint(dataframe.head(1))  # display only top 3  rowsprint(dataframe.head(3))  # display only top 2 rowsprint(dataframe.head(2))
Output:
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)]
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′), 
Row(Employee ID=’2′, Employee NAME=’ojaswi’, Company Name=’company 2′), 
Row(Employee ID=’3′, Employee NAME=’bobby’, Company Name=’company 3′)]
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′), 
Row(Employee ID=’2′, Employee NAME=’ojaswi’, Company Name=’company 2′)]
Used to return last n rows in the dataframe
Syntax: dataframe.tail(n)
where n is the no of rows to be returned from last in the dataframe.
Example: Python code to get last n rows
Python3
# display only 1 row from lastprint(dataframe.tail(1))  # display only top 3  rows from lastprint(dataframe.tail(3))  # display only top 2 rows from lastprint(dataframe.tail(2))
Output:
[Row(Employee ID=’5′, Employee NAME=’gnanesh’, Company Name=’company 1′)]
[Row(Employee ID=’3′, Employee NAME=’bobby’, Company Name=’company 3′),
 Row(Employee ID=’4′, Employee NAME=’rohith’, Company Name=’company 2′),
  Row(Employee ID=’5′, Employee NAME=’gnanesh’, Company Name=’company 1′)]
[Row(Employee ID=’4′, Employee NAME=’rohith’, Company Name=’company 2′),
 Row(Employee ID=’5′, Employee NAME=’gnanesh’, Company Name=’company 1′)]
This method is used to select a particular row from the dataframe, It can be used with collect() function.
Syntax: dataframe.select([columns]).collect()[index]
where, 
dataframe is the pyspark dataframe
Columns is the list of columns to be displayed in each row
Index is the index number of row to be displayed.
Example: Python code to select the particular row.
Python3
# select first rowprint(dataframe.select(['Employee ID',                        'Employee NAME',                        'Company Name']).collect()[0])  # select third rowprint(dataframe.select(['Employee ID',                        'Employee NAME',                        'Company Name']).collect()[2])  # select forth rowprint(dataframe.select(['Employee ID',                        'Employee NAME',                        'Company Name']).collect()[3])
Output:
Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)
Row(Employee ID=’3′, Employee NAME=’bobby’, Company Name=’company 3′)
Row(Employee ID=’4′, Employee NAME=’rohith’, Company Name=’company 2′)
This method is also used to select top n rows
Syntax: dataframe.take(n)
where n is the number of rows to be selected
Python3
# select top 2 rowsprint(dataframe.take(2))  # select top 4 rowsprint(dataframe.take(4))  # select top 1 rowprint(dataframe.take(1))
Output:
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′), 
Row(Employee ID=’2′, Employee NAME=’ojaswi’, Company Name=’company 2′)]
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′),
Row(Employee ID=’2′, Employee NAME=’ojaswi’, Company Name=’company 2′),
 Row(Employee ID=’3′, Employee NAME=’bobby’, Company Name=’company 3′),
  Row(Employee ID=’4′, Employee NAME=’rohith’, Company Name=’company 2′)]
[Row(Employee ID=’1′, Employee NAME=’sravan’, Company Name=’company 1′)]
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
Python Dictionary
Read a file line by line in Python
Enumerate() in Python
How to Install PIP on Windows ?
Iterate over a list in Python
Different ways to create Pandas Dataframe
Python program to convert a list to string
Python String | replace()
Reading and Writing to text files in Python
sum() function in Python