PySpark - orderBy() and sort() - GeeksforGeeks

06 Jun, 2021
In this article, we will see how to sort the data frame by specified columns in PySpark. We can make use of orderBy() and sort()  to sort the data frame in PySpark
OrderBy() function is used to sort an object by its index value.
Syntax: DataFrame.orderBy(cols, args)
Parameters :
cols: List of columns to be ordered
args: Specifies the sorting order i.e (ascending or descending) of columns listed in cols
Return type: Returns a new DataFrame sorted by the specified columns.
Dataframe Creation: Create a new SparkSession object named spark then create a data frame with the custom data.
Python3
# Importing necessary librariesfrom pyspark.sql import SparkSessionfrom pyspark.sql import functions as f  # Create a spark sessionspark = SparkSession.builder.appName(  'pyspark - example join').getOrCreate()  # Define data in  a dataframedataframe = [    ("Sam", "Software Engineer", "IND", 10000),    ("Raj", "Data Scientist", "US", 41000),    ("Jonas", "Sales Person", "UK", 230000),    ("Peter", "CTO", "Ireland", 50000),    ("Hola", "Data Analyst", "Australia", 111000),    ("Ram", "CEO", "Iran", 300000),    ("Lekhana", "Advertising", "UK", 250000),    ("Thanos", "Marketing", "UIND", 114000),    ("Nick", "Data Engineer", "Ireland", 680000),    ("Wade", "Data Engineer", "IND", 70000)]  # Column names of dataframecolumns = ["Name", "Job", "Country", "salary"]  # Create the spark dataframedf = spark.createDataFrame(data=dataframe, schema=columns)  # Printing the dataframedf.show()
Output :
Example 1: Sorting the data frame by a single column
Sort the data frame by the ascending order of ‘Salary’ of employees in the data frame.
Python3
# Order the data by ascending order # of Salarydf.orderBy(['Salary'], ascending = [True]).show()  # or# df.orderBy(f.col("Salary").asc()).show()  # or# df.orderBy(['Salary']).show()
Output :
Example 2: Sorting the data frame in decreasing order.
Python3
# Order the data by dec order # of Salarydf.orderBy(['Salary'], ascending = [False]).show()
Output:
Example 3: Sorting the data frame by more than one column
Sort the data frame by the descending order of ‘Job’ and ascending order of ‘Salary’ of employees in the data frame. When there is a conflict between two rows having the same ‘Job’, then it’ll be resolved by listing rows in the ascending order of ‘Salary’.
Python3
# Sort the dataframe by descending order# of 'Job' and whenever there is conflict# in 'Job', it'll be resolved by ordering # based on ascending order of 'Salary'df.orderBy(f.col("Job").desc(),f.col("Salary").asc()).show()  # or# df.orderBy(["Job", "Salary"],ascending = [False, True]).show()
Output :
It takes the Boolean value as an argument to sort in ascending or descending order.
Syntax:sort(x, decreasing, na.last)
Parameters:x: list of Column or column names to sort bydecreasing: Boolean value to sort in descending orderna.last: Boolean value to put NA at the end
Example 1: Sort the data frame by the ascending order of the “Name” of the employee.
Python3
# Sort the dataframe by ascending # order of 'Name'df.sort(["Name"],ascending = [True]).show()
Output :
Example 2: Sort the column in decreasing order.
Python3
# Sort the dataframe by scendding order of 'Name'df.sort(["Name"],ascending = [False]).show()
Output:
Example 3: Sort multiple columns in ascending order.
Python3
# Sort the dataframe by acendding order of 'Name'df.sort(["Name","salary"],ascending = [True]).show()
Output:
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
Box Plot in Python using Matplotlib
Python | Get dictionary keys as a list
Bar Plot in Matplotlib
Multithreading in Python | Set 2 (Synchronization)
Python Dictionary keys() method
loops in python
Python - Call function from another file
Ways to filter Pandas DataFrame by column values
Python | Convert set into a list
Python program to find number of days between two given dates