Bug: array_sort: Improve documentation for default comparator's behavior for different types
Status: Open | Priority: Trivial | Resolution: 
Created: 14/May/24 14:41
Affects Version/s: 3.5.0

h1. tl;dr

It would be helpful for the documentation for array_sort() to include the default comparator behavior for different array element types, especially structs. It would also be helpful for the {{{\{DATATYPE_MISMATCH.INVALID_ORDERING_TYPE }}}} error to recommend using a custom comparator instead of the default comparator, especially when sorting on a complex type (e.g., a struct containing an unorderable field, like a map).

 
----
h1. Background

The default comparator for {{array_sort()}} for struct elements is to sort by every field in the struct in schema order (i.e., ORDER BY field1, field2, ..., fieldN). This requires every field to be orderable: if they aren't an error occurs.

 

Here's a small example:
{code:java}
import pyspark.sql.functions as F
import pyspark.sql.types as T

schema = T.StructType([
    T.StructField(
        'value',
        T.ArrayType(
            T.StructType([
                T.StructField('orderable', T.IntegerType(), True),
                T.StructField('unorderable', T.MapType(T.StringType(), T.StringType(), True), True), # remove this field and both commands below succeed
            ]),
            False
        ),
        False
    )
])
df = spark.createDataFrame([], schema=schema)

df.select(F.array_sort(df['value'])).collect(){code}
Output:
{code:java}
[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] Cannot resolve "(namedlambdavariable() < namedlambdavariable())" due to data type mismatch: The `<` does not support ordering on type "STRUCT<orderable: INT, unorderable: MAP<STRING, STRING>>". SQLSTATE: 42K09 {code}
 

If the default comparator doesn't work for a user (e.g., they have an unorderable field like a map in their struct), array_sort() accepts a custom comparator, where users can order array elements however they like.

 

Building on the previous example:

 
{code:java}
import pyspark.sql as psql


def comparator(l: psql.Column, r: psql.Column) -> psql.Column:
    """Order structs l and r by order field.
    Rules:
        * Nulls are last
        * In ascending order
    """
    return (
        F.when(l['order'].isNull() & r['order'].isNull(), 0)
        .when(l['order'].isNull(), 1)
        .when(r['order'].isNull(), -1)
        .when(l['order'] < r['order'], -1)
        .when(l['order'] == r['order'], 0)
        .otherwise(1)
    )

df.select(F.array_sort(df['value'], comparator)).collect(){code}
This works as intended.

 
----
h1. Ask

The documentation for {{array_sort()}} should include information on the behavior of the default comparator for various datatypes. For the array-of-unorderable-structs example, it would be helpful to know that the default comparator for structs compares all fields in schema order (i.e., {{{}ORDER BY field1, field2, ..., fieldN{}}}).

 

Additionally, when users passes an unorderable type to array_sort() and uses the default comparator, the returned error should recommend the user use a custom comparator instead.