The Definitive Way To Sort Arrays In Spark 3.0 | by German Schiavon Matteo | Towards Data Science

Earlier last year(2020) I had the need to sort an array, and I found that there were two functions, very similar in name, but different in functionality.
These are array_sort and sort_array.
Which one to use? At first, I felt confused, Why would there be two functions to do the same?
Well, the difference is that array_sort :
def array_sort(e: Column):Sorts the input array in ascending order and null elements will be placed at the end of the returned array.
While sort_array :
def sort_array(e: Column, asc: Boolean)Sorts the input array for the given column in ascending or descending order elements.Null elements will be placed at the beginning of the returned array in ascending order or at the end of the returned array in descending order.
After seeing this I decided to open a pull request to unify this behaviour in only array_sort , but after some discussion with the committers, they decided that it would be a good idea to add a way to sort arrays given a comparator function, to match Presto array functions.
Obviously, this was a lot harder than the initial idea of unifying the behaviour of the two functions, but with the help of the committers/reviewers got merged.
Enough history, let’s see how the new array_sort works in Spark 3.0
It receives a comparator function, in this function you will define the logic used to compare the elements of the array.
So what?
The comparator is really powerful when you want to order an array with custom logic or to compare arrays of structs choosing the field that you want to use in the sorting.
Okay, but how?
Let’s define our DataFrame with an array of structs
case class Person(name: String, age: Int)val df = Seq(Array(Person(“andrew”, 23), Person(“juan”,28), Person(“peter”, 22))).toDF
If we print the schema and show the content:
> df.printSchemaroot |-- value: array (nullable = true) |    |-- element: struct (containsNull = true) |    |    |-- name: string (nullable = true) |    |    |-- age: integer (nullable = false)> df.show(false)+---------------------------------------+|value                                  |+---------------------------------------+|[[andrew, 23], [juan, 28], [peter, 22]]|+---------------------------------------+
Now let’s order the DataFrame using the name column:
df.withColumn("arr_sorted", expr("array_sort(value,(l, r) -> case when l.name > r.name then -1 when l.name < r.name then 1 else 0 end)"))
Now let’s see the content
+---------------------------------------+|arr_sorted                             |+---------------------------------------+|[[peter, 22], [juan, 28], [andrew, 23]]|+---------------------------------------+
Nice! we got the array sorted by name!
Is there another way to do it?
Yes! You can do it more programatically, by registering a UDF with the comparator logic.
What?
Okay, imagine that now you want to order the array by the name length, then you would do something like this:
spark.udf.register("fStringLength", (x: Person, y: Person) => {  if (x.name.length < y.name.length) -1  else if (x.name.length == y.name.length) 0  else 1})
And then, call the UDF
df.selectExpr(“array_sort(value, (x, y) -> fStringLength(x, y)) as arr_sorted”).show(false)
There you go! Array sorted by name length
+---------------------------------------+|arr_sorted                             |+---------------------------------------+|[[juan, 28], [peter, 22], [andrew, 23]]|+---------------------------------------+
I hope the newarray_sort is more clear after reading the post, it is definitely a powerful function of Spark 3.0
There are multiple ways to sort arrays in Spark, the new function brings a new set to possibilities sorting complex arrays.