PySpark - Order by multiple columns - GeeksforGeeks

19 Dec, 2021
In this article, we are going to see how to orderby multiple columns in  PySpark DataFrames through Python.
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  dataframe.show()
Output:
orderby means we are going to sort the dataframe by multiple columns in ascending or descending order. we can do this by using the following methods.
This function will return the dataframe after ordering the multiple columns. It will sort first based on the column name given.
Syntax:
Ascending order: dataframe.orderBy([‘column1′,’column2′,......,’column n’], ascending=True).show()
Descending Order: dataframe.orderBy([‘column1′,’column2′,......,’column n’], ascending=False).show()
where:
dataframe is the Pyspark Input dataframe
ascending=True specifies to sort the dataframe in ascending order
ascending=False specifies to sort the dataframe in descending order
Example 1: Sort the PySpark dataframe in ascending order with orderBy().
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # orderBy dataframe in asc orderdataframe.orderBy(['Name', 'ID', 'Company'],                  ascending=True).show()
Output:
Example 2: Sort the PySpark dataframe in descending order with orderBy().
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # orderBy dataframe in desc orderdataframe.orderBy(['Name', 'ID', 'Company'],                   ascending=False).show()
Output:
This function will return the dataframe after ordering the multiple columns. It will sort first based on the column name given.
Syntax:
Ascending order: dataframe.sort([‘column1′,’column2′,......,’column n’], ascending=True).show()
Descending Order: dataframe.sort([‘column1′,’column2′,......,’column n’], ascending=False).show()
where,
dataframe is the Pyspark Input dataframeascending=True specifies to sort the dataframe in ascending orderascending=False specifies to sort the dataframe in descending order
dataframe is the Pyspark Input dataframe
ascending=True specifies to sort the dataframe in ascending order
ascending=False specifies to sort the dataframe in descending order
Example 1: Sort PySpark dataframe in ascending order
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # orderBy dataframe in asc orderdataframe.sort(['Name', 'ID', 'Company'],               ascending=True).show()
Output:
Example 2: Sort the PySpark dataframe in descending order
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["5", "bobby", "company 1"]]  # specify column namescolumns = ['ID', 'NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # orderBy dataframe in desc orderdataframe.sort(['Name', 'ID', 'Company'],               ascending=False).show()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Python Classes and Objects
How to drop one or multiple columns in Pandas Dataframe
Defaultdict in Python
Python | Get unique values from a list
Python | os.path.join() method
Create a directory in Python
Python | Pandas dataframe.groupby()