How to get a value from the Row object in PySpark Dataframe?

04 Jan, 2022
In this article, we are going to learn how to get a value from the Row object in PySpark DataFrame.
We will create a Spark DataFrame with at least one row using createDataFrame(). We then get a Row object from a list of row objects returned by DataFrame.collect(). We then use the __getitem()__ magic method to get an item of a particular column name. Given below is the syntax.
Syntax : DataFrame.__getitem__(‘Column_Name’)
Returns : value corresponding to the column name in the Row object
Python
# library importimport pysparkfrom pyspark.sql import SparkSessionfrom pyspark.sql import Row  # Session Creationrandom_value_session = SparkSession.builder.appName(    'Random_Value_Session').getOrCreate()  # Data filled in our DataFrame# 5 rows belowrows = [['All England Open', 'March', 'Super 1000'],        ['Malaysia Open', 'January', 'Super 750'],        ['Korea Open', 'April', 'Super 500'],        ['Hylo Open', 'November', 'Super 100'],        ['Spain Masters', 'March', 'Super 300']]  # Columns of our DataFramecolumns = ['Tournament', 'Month', 'Level']  #DataFrame is createddataframe = random_value_session.createDataFrame(rows,                                                 columns)  # Showing the DataFramedataframe.show()  # getting list of rows using collect()row_list = dataframe.collect()  # Printing the first Row object# from which data is extractedprint(row_list[0])  # Using __getitem__() magic method# To get value corresponding to a particular# columnprint(row_list[0].__getitem__('Level'))print(row_list[0].__getitem__('Tournament'))print(row_list[0].__getitem__('Level'))print(row_list[0].__getitem__('Month'))
Output: 
+----------------+--------+----------+
|      Tournament|   Month|     Level|
+----------------+--------+----------+
|All England Open|   March|Super 1000|
|   Malaysia Open| January| Super 750|
|      Korea Open|   April| Super 500|
|       Hylo Open|November| Super 100|
|   Spain Masters|   March| Super 300|
+----------------+--------+----------+

Row(Tournament='All England Open', Month='March', Level='Super 1000')
Super 1000
All England Open
Super 1000
March
We will create a Spark DataFrame with atleast one row using createDataFrame(). We then get a Row object from a list of row objects returned by DataFrame.collect(). We then use the asDict() method to get a dictionary where column names are keys and their row values are dictionary values. Given below is the syntax:
Syntax : DataFrame.asDict(recursive)
Parameters : 
recursive: bool : returns nested rows as dict. The default value is False.
We then get easily get the value from the dictionary using DictionaryName[‘key_name’].
Python
# library imports are done hereimport pysparkfrom pyspark.sql import SparkSession  # Session Creationrandom_value_session = SparkSession.builder.appName(    'Random_Value_Session').getOrCreate()  # Data filled in our DataFrame# Rows below will be filledrows = [['French Open', 'October', 'Super 750'],        ['Macau Open', 'November', 'Super 300'],        ['India Open', 'January', 'Super 500'],        ['Odisha Open', 'January', 'Super 100'],        ['China Open', 'November', 'Super 1000']]  # DataFrame Columnscolumns = ['Tournament', 'Month', 'Level']  # DataFrame creationdataframe = random_value_session.createDataFrame(rows,                                                 columns)  # DataFrame printdataframe.show()  # list of rows using collect()row_list = dataframe.collect()  # Printing the second Row object# from which we will read dataprint(row_list[1])print()  # Printing dictionary to make# things more clearprint(row_list[1].asDict())print()  # Using asDict() method to convert row object# into a dictionary where the column names are keys# Using column names as keys to get respective valuesprint(row_list[1].asDict()['Tournament'])print(row_list[1].asDict()['Month'])print(row_list[1].asDict()['Level'])
Output : 
+-----------+--------+----------+
| Tournament|   Month|     Level|
+-----------+--------+----------+
|French Open| October| Super 750|
| Macau Open|November| Super 300|
| India Open| January| Super 500|
|Odisha Open| January| Super 100|
| China Open|November|Super 1000|
+-----------+--------+----------+

Row(Tournament='Macau Open', Month='November', Level='Super 300')

{'Tournament': 'Macau Open', 'Month': 'November', 'Level': 'Super 300'}

Macau Open
November
Super 300
Here we will imagine a Row object like a Python List and perform operations. We will create a Spark DataFrame with at least one row using createDataFrame(). We then get a Row object from a list of row objects returned by DataFrame.collect(). Since we are imagining the Row object like a List, we just use : 
Syntax : RowObject[‘Column_name’]
Returns : Value corresponding to the column name in the row object.
Python
# library imports are done hereimport pysparkfrom pyspark.sql import SparkSession  # Session Creationrandom_value_session = SparkSession.builder.appName(    'Random_Value_Session').getOrCreate()  # Data filled in our DataFrame# Rows below will be filledrows = [['Denmark Open', 'October', 'Super 1000'],        ['Indonesia Open', 'June', 'Super 1000'],        ['Korea Open', 'April', 'Super 500'],        ['Japan Open', 'August', 'Super 750'],        ['Akita Masters', 'July', 'Super 100']]  # DataFrame Columnscolumns = ['Tournament', 'Month', 'Level']  # DataFrame creationdataframe = random_value_session.createDataFrame(rows,                                                 columns)  # DataFrame printdataframe.show()  # list of rows using collect()row_list = dataframe.collect()  # Lets take the third Row objectrow_object = row_list[2]  # If we imagine it as a Python List,# We can get the first value of the list,# index 0, let's try itprint(row_object[0])  # We got the value of column at index 0# which is - 'Tournament'  # A few more examplesprint(row_list[4][0])print(row_list[3][1])print(row_list[4][2])
Output: 
+--------------+-------+----------+
|    Tournament|  Month|     Level|
+--------------+-------+----------+
|  Denmark Open|October|Super 1000|
|Indonesia Open|   June|Super 1000|
|    Korea Open|  April| Super 500|
|    Japan Open| August| Super 750|
| Akita Masters|   July| Super 100|
+--------------+-------+----------+

Korea Open
Akita Masters
August
Super 100
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
Different ways to create Pandas Dataframe
Enumerate() in Python
Read a file line by line in Python
Python String | replace()
How to Install PIP on Windows ?
*args and **kwargs in Python
Python Classes and Objects
Iterate over a list in Python
Introduction To PYTHON