PySpark – Sort dataframe by multiple columns

30 Jun, 2021
In this article, we are going to see how to sort the PySpark dataframe by multiple columns.
It can be done in these ways:
Using sort()
Using orderBy()
Creating Dataframe for demonstration:
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "sravan", "vignan"],        ["2", "ojaswi", "vvit"],        ["3", "rohith", "vvit"],        ["4", "sridevi", "vignan"],        ["1", "sravan", "vignan"],        ["5", "gnanesh", "iit"]]  # specify column namescolumns = ['student ID', 'student NAME', 'college']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  print("Actual data in dataframe")  # show dataframedataframe.show()
Output:
This function is used to sort the column.
Syntax: dataframe.sort([‘column1′,’column2′,’column n’],ascending=True)
Where,
dataframe is the dataframe name created from the nested lists using pyspark
where columns are the llst of columns
ascending = True specifies order the dataframe in increasing order, ascending=False specifies order the dataframe in decreasing order
Example 1: Python code to sort dataframe by passing a list of multiple columns(2 columns) in ascending order.
Python3
# show dataframe by sorting the dataframe# based on two columns in ascending orderdataframe.sort(['college','student ID'],               ascending = True).show()
Output:
Example 2: Python program to sort the data frame by passing a list of columns in descending order
Python3
# show dataframe by sorting the dataframe# based on two columns in descending orderdataframe.sort(['college','student NAME'],               ascending = False).show()
Output:
orderBy() function that sorts one or more columns. By default, it orders by ascending.
Syntax: orderBy(*cols, ascending=True)
Parameters:
cols: Columns by which sorting is needed to be performed.
ascending: Boolean value to say that sorting is to be done in ascending order
Example 1: Python program to show dataframe by sorting the dataframe based on two columns in descending order using orderby() function
Python3
# show dataframe by sorting the dataframe# based on two columns in descending# order using orderby() functiondataframe.orderBy(['student ID','student NAME'],                  ascending = False).show()
Output:
Example 2: Python program to show dataframe by sorting the dataframe based on two columns in ascending order using orderby() function
Python3
# show dataframe by sorting the dataframe# based on two columns in ascending# order using orderby() functiondataframe.orderBy(['student ID','student NAME'],                  ascending = True).show()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python | os.path.join() method
Python OOPs Concepts
How to drop one or multiple columns in Pandas Dataframe
Introduction To PYTHON
How To Convert Python Dictionary To JSON?
Check if element exists in list in Python
Python | datetime.timedelta() function
Python | Get unique values from a list