Spark 3.0 SQL Feature Update| ANSI SQL Compliance, Store Assignment policy, Upgraded query semantics, Function Upgrades
By: Prabhakaran Vijayanagulu
Tags: Big Data Analytics, Big Data, Spark Sql, Apache Spark, Data Engineering

Spark has added a lot of notable features with Spark SQL. Some will have a huge impact on checks like data quality and data validations. Though there is a lot of upgraded features, I’m listing out a few of those as these would be used in most common cases.

ANSI SQL Complaint Features

For Spark developers, who are at the initial stage of learning SQL commands, query identifier validation might be helpful. They might use keywords as identifiers that are not meant to be used. Even it will work completely fine with spark, This will confuse others who work with the code in the future.

Spark will allow certain unusual cases like using some reserved keywords as identifiers. Something like this:

select * from table_1 create where create.column_1= 1

This query will run without any issues in spark. create is the most common reserved keyword, used for creating tables with SQL, but it can be used as an identifier in spark without any issues.

To overcome this, and involve query validation in runtime, Spark is now compliant with ANSI SQL standard. Validation has been added to the catalyst parser level. To enable ANSI mode query validation, switch property spark.sql.ansi.enabled to true. For the same query shown above, Spark will now throw the exception.

Error in SQL statement: ParseException: no viable alternative at input 'create'(line 1, pos 38) == SQL ==

select * from table_1 create where create.column_1= 1

----------------------^^^ com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.catalyst.parser.ParseException:

Following are the keywords standardized in Spark 3.0:

To disable the ANSI standard validation, Falsify spark.sql.ansi.enabled.

Store Assignment policy

This feature is introduced for stern data quality checks during migration from SQL kind of environment. Below Insert table statement(Ingesting string into an integer column) would execute without any runtime exception in Spark 2.4 and below.

This will work fine in Spark 3.0 as well. But by setting up the property “spark.sql.storeAssignmentPolicy” as ‘ANSI’, casting exception is thrown.

This definitely improves data quality checks during the migration.

Optional mode in “from_json” (FAIL_FAST/PERMISSIVE)

from_json method is used in parsing JSON type value within a column. In Spark 3.0, it supports PERMISSIVE/FAIL_FAST mode similar to spark.read.json. So, if the JSON value cannot be parsed or malformed, it will throw an exception as follows:

Exponential notation

In Spark 3.0, numbers written in scientific notation(for example, 1E11) would be parsed as Double. In Spark version 2.4 and below, they’re parsed as Decimal. To restore the behavior before Spark 3.0, you can set spark.sql.legacy.exponentLiteralAsDecimal.enabled to false.

In Spark 2.4,

In Spark 3.0, and setting ‘spark.sql.legacy.exponentLiteralAsDecimal.enabled’ as true.

Negative Decimal zeroes

In Spark version 2.4 and below, float/double -0.0 is semantically equal to 0.0, but -0.0 and 0.0 are considered as different values when used in aggregate grouping keys, window partition keys, and join keys. It’s a Bug and it is fixed in Spark 3.0. Now, Distinct of (-0.0,0.0) will give (0.0).

Sample Dataset

With Spark 2.4, you will get as follows,

Whereas with Spark 3.0, with no additional property set, consideration will be the same irrespective of negative or positive.

Keyword Strings to ‘Date’ and ’Timestamp’

In Spark 3.0, Certain keywords are supported for converting a string to date.

To Generate a ‘date’ from a string, Following are the keywords.

Possible keywords:

select date 'x' 'x' can have following values: > epoch -> 1970-01-01 (minimum date) > today -> the current date in the time zone specified by spark.sql.session.timeZone > yesterday -> the current date - 1 > tomorrow -> the current date + 1 > now -> the date of running the current query. It has the same notion as today

Similarly, We have an option for getting timestamp from string.

Possible keywords:

select timestamp 'x' 'x' can have following values > epoch -> 1970-01-01 00:00:00+00 (Unix system time zero) > today -> midnight today > yesterday -> midnight yesterday > tomorrow -> midnight tomorrow > now -> current query start time

In 3.0, Some of the SQL functions also have been updated/added, which has benefits in data aggregation and getting insights out of it in a simpler way. Find some of them below.

Dataset used for following section

Max_By(), Min_By()

These functions are one of the significant features that I have noted. These will get you the value of a column, with respect to maximum/minimum some other column.

Usage: max_by(x,y) / min_by(x,y)

→ x = column from which value to be fetched.

→ y = column to get the aggregated maximum/minimum.

From the above data, the Overall Maximum Price is 2.598 and the Overall Minimum price is 2.514.

But with Max_by/Min_by, Maximum = 2.572, which is for the maximum order number 6 and Minimum = 2.548, which is for the minimum order number 1.

WITH clause inside subqueries

CTE using with clause can now be used within subqueries. This will improve query readability and would be significant in using multiple CTE’s.Something like this:

select * from

(with inner_cte as (select * from sql_functions_test where order_num = 5)

select order_num from inner_cte);

Filter (Where…)

This feature would be helpful in aggregation only for a set of rows, filtered based on a condition.