Sort the PySpark DataFrame columns by Ascending or Descending order

06 Jun, 2021
In this article, we are going to sort the dataframe columns in the pyspark. For this, we are using sort() and orderBy() functions in ascending order and descending order sorting.
Let’s create a sample dataframe.
Python3
# importing moduleimport pyspark  # importing sparksession from # pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of employee datadata = [["1", "sravan", "company 1"],        ["2", "ojaswi", "company 1"],        ["3", "rohith", "company 2"],        ["4", "sridevi", "company 1"],        ["1", "sravan", "company 1"],        ["4", "sridevi", "company 1"]]  # specify column namescolumns = ['Employee_ID', 'Employee NAME', 'Company']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # display data in the dataframedataframe.show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          1|       sravan|company 1|
|          2|       ojaswi|company 1|
|          3|       rohith|company 2|
|          4|      sridevi|company 1|
|          1|       sravan|company 1|
|          4|      sridevi|company 1|
+-----------+-------------+---------+
The sort function is used to sort the data frame column.
Syntax: dataframe.sort([‘column name’], ascending=True).show()
Example 1: Arrange in ascending Using Sort() with one column
Sort the data based on Employee Name in increasing order
Python3
# sort the dataframe based on # employee name column in ascending orderdataframe.sort(['Employee NAME'],               ascending = True).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          1|       sravan|company 1|
|          1|       sravan|company 1|
|          2|       ojaswi|company 1|
|          3|       rohith|company 2|
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
+-----------+-------------+---------+
Sort the data based on Employee name in decreasing order:
Syntax: dataframe.sort([‘column name’], ascending = False).show()
Code:
Python3
# sort the dataframe based on # employee name column in descending orderdataframe.sort(['Employee NAME'],               ascending = False).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
|          1|       sravan|company 1|
|          1|       sravan|company 1|
|          3|       rohith|company 2|
|          2|       ojaswi|company 1|
+-----------+-------------+---------+
Example 2: Using Sort() with multiple columns
We are going to sort the dataframe based on employee id and employee name in ascending order.
Python3
# sort the dataframe based on employee ID# and employee Name columns in ascending orderdataframe.sort(['Employee_ID','Employee NAME'],               ascending = True).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          1|       sravan|company 1|
|          1|       sravan|company 1|
|          2|       ojaswi|company 1|
|          3|       rohith|company 2|
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
+-----------+-------------+---------+
We are going to sort the dataframe based on employee ID, company, and employee name in descending order
Python3
# sort the dataframe based on employee ID ,# company and employee Name columns in descending orderdataframe.sort(['Employee_ID','Employee NAME',                'Company'], ascending = False).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
|          3|       rohith|company 2|
|          2|       ojaswi|company 1|
|          1|       sravan|company 1|
|          1|       sravan|company 1|
+-----------+-------------+---------+
Example 3: Sort by ASC methods.
ASC method of the Column function, it returns a sort expression based on the ascending order of the given column name.
Python3
dataframe.sort(dataframe.Employee_ID.asc()).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          1|       sravan|company 1|
|          1|       sravan|company 1|
|          2|       ojaswi|company 1|
|          3|       rohith|company 2|
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
+-----------+-------------+---------+
Example 4: Sort by DESC methods.
DESC method of the Column function, it returns a sort expression based on the descending order of the given column name.
Python3
dataframe.sort(dataframe.Employee_ID.desc()).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
|          3|       rohith|company 2|
|          2|       ojaswi|company 1|
|          1|       sravan|company 1|
|          1|       sravan|company 1|
+-----------+-------------+---------+
The orderBy() function sorts by one or more columns. By default, it sorts by ascending order.
Syntax: orderBy(*cols, ascending=True)
Parameters:
cols→ Columns by which sorting is needed to be performed.
ascending→ Boolean value to say that sorting is to be done in ascending order
Example 1: ascending for one column
Python program to sort the dataframe based on Employee ID in ascending order
Python3
# sort the dataframe based on employee I# columns in descending orderdataframe.orderBy(['Employee_ID'],                  ascending=False).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
|          3|       rohith|company 2|
|          2|       ojaswi|company 1|
|          1|       sravan|company 1|
|          1|       sravan|company 1|
+-----------+-------------+---------+
Python program to sort the dataframe based on  Employee ID in descending order
Python3
# sort the dataframe based on# Employee ID in descending orderdataframe.orderBy(['Employee_ID'],                  ascending = False).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
|          3|       rohith|company 2|
|          2|       ojaswi|company 1|
|          1|       sravan|company 1|
|          1|       sravan|company 1|
+-----------+-------------+---------+
Example 2: Ascending multiple columns
Sort the dataframe based on employee ID and employee Name columns in descending order using orderBy.
Python3
# sort the dataframe based on employee ID # and employee Name columns in descending orderdataframe.orderBy(['Employee ID','Employee NAME'],                  ascending = False).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
|          3|       rohith|company 2|
|          2|       ojaswi|company 1|
|          1|       sravan|company 1|
|          1|       sravan|company 1|
+-----------+-------------+---------+
Sort the dataframe based on employee ID and employee Name columns in ascending order
Python3
# sort the dataframe based on employee ID # and employee Name columns in ascending orderdataframe.orderBy(['Employee_ID','Employee NAME'],                  ascending =True).show()
Output:
+-----------+-------------+---------+
|Employee_ID|Employee NAME|  Company|
+-----------+-------------+---------+
|          1|       sravan|company 1|
|          1|       sravan|company 1|
|          2|       ojaswi|company 1|
|          3|       rohith|company 2|
|          4|      sridevi|company 1|
|          4|      sridevi|company 1|
+-----------+-------------+---------+
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.