sort() vs orderBy() in Spark | Towards Data Science

Sorting a Spark DataFrame is probably one of the most commonly used operations. You can use either sort() or orderBy() built-in functions to sort a particular DataFrame in ascending or descending order over at least one column. Even though both functions are supposed to order the data in a Spark DataFrame, they have one significant difference.
In today’s article we are going to discuss the difference between sort() and orderBy() and understand when to use one over the other.
First let’s create an example DataFrame that will use throughout the article to demonstrate how both sort() and orderBy() work. Note that I’ll be using the Python API but what we discuss today is also applicable for Scala as well.
df = spark.createDataFrame(   [      ('Andrew', 'Johnson', 'Engineering', 'UK', 34),      ('Maria', 'Brown', 'Finance', 'US', 41),      ('Michael', 'Stevenson', 'Sales', 'US', 31),      ('Mark', 'Anderson', 'Engineering', 'Ireland', 28),      ('Jen', 'White', 'Engineering', 'UK', 29)  ],  ['first_name', 'last_name', 'department', 'country', 'age'])df.show(truncate=False)+----------+---------+-----------+--------+---+|first_name|last_name| department||country|age|+----------+---------+-----------+--------+---+|    Andrew|  Johnson|Engineering|      UK| 34||     Maria|    Brown|    Finance|      US| 41||   Michael|Stevenson|      Sales|      US| 31||      Mark| Anderson|Engineering| Ireland| 28||       Jen|    White|Engineering|      UK| 29|+----------+---------+-----------+--------+---+
Now we can use sort() method do sort the DataFrame df based on the country and age of the employees in ascending order.
df.sort('country', 'age').show(truncate=False)+----------+---------+-----------+--------+---+|first_name|last_name| department||country|age|+----------+---------+-----------+--------+---+|      Mark| Anderson|Engineering| Ireland| 28||       Jen|    White|Engineering|      UK| 29||    Andrew|  Johnson|Engineering|      UK| 34||   Michael|Stevenson|      Sales|      US| 31||     Maria|    Brown|    Finance|      US| 41|+----------+---------+-----------+--------+---+
Note however that sort() method will sort the records in each partition and then return the final output which means that the order of the output data is not guaranteed because the data is ordered on partition-level but your DataFrame may have thousands of partitions distributed across the cluster. Since the data is not collected into a single executor the sort() method is efficient thus more suitable when sorting is not critical for your use-case.
For example, if employees Jen and Andrew are stored on a different partition the final order may have been wrong, for instance:
+----------+---------+-----------+--------+---+|first_name|last_name| department||country|age|+----------+---------+-----------+--------+---+|      Mark| Anderson|Engineering| Ireland| 28||    Andrew|  Johnson|Engineering|      UK| 34||       Jen|    White|Engineering|      UK| 29||   Michael|Stevenson|      Sales|      US| 31||     Maria|    Brown|    Finance|      US| 41|+----------+---------+-----------+--------+---+
In the same way we can use orderBy() to order our DataFrame based on the country and age of the employees in ascending order:
df.orderBy('country', 'age').show(truncate=False)+----------+---------+-----------+--------+---+|first_name|last_name| department||country|age|+----------+---------+-----------+--------+---+|      Mark| Anderson|Engineering| Ireland| 28||       Jen|    White|Engineering|      UK| 29||    Andrew|  Johnson|Engineering|      UK| 34||   Michael|Stevenson|      Sales|      US| 31||     Maria|    Brown|    Finance|      US| 41|+----------+---------+-----------+--------+---+
Unlike sort(), the orderBy() function guarantees a total order in the output. This happens because the data will be collected into a single executor in order to be sorted. This means that orderBy() is more inefficient compared to sort().
Both sort() and orderBy() functions can be used to sort Spark DataFrames on at least one column and any desired order, namely ascending or descending.
sort() is more efficient compared to orderBy() because the data is sorted on each partition individually and this is why the order in the output data is not guaranteed.
On the other hand, orderBy() collects all the data into a single executor and then sorts them. This means that the order of the output data is guaranteed but this is probably a very costly operation.
Therefore, if you need to sort a DataFrame where the order is not so critical and at the same time you want it to be as fast as possible then sort() is more suitable. In case the order is critical, make sure to use orderBy() that guarantees that the output data is ordered based on the user parameters.