Spark 3.0 SQL Feature Update| ANSI SQL Compliance, Store Assignment policy, Upgraded query semantics, Function Upgrades | by Prabhakaran Vijayanagulu | Towards Data Science

Spark has added a lot of notable features with Spark SQL. Some will have a huge impact on checks like data quality and data validations. Though there is a lot of upgraded features, I’m listing out a few of those as these would be used in most common cases.
For Spark developers, who are at the initial stage of learning SQL commands, query identifier validation might be helpful. They might use keywords as identifiers that are not meant to be used. Even it will work completely fine with spark, This will confuse others who work with the code in the future.
Spark will allow certain unusual cases like using some reserved keywords as identifiers. Something like this:
select * from table_1 create where create.column_1= 1
This query will run without any issues in spark. create is the most common reserved keyword, used for creating tables with SQL, but it can be used as an identifier in spark without any issues.
To overcome this, and involve query validation in runtime, Spark is now compliant with ANSI SQL standard. Validation has been added to the catalyst parser level. To enable ANSI mode query validation, switch property spark.sql.ansi.enabled to true. For the same query shown above, Spark will now throw the exception.
Error in SQL statement: ParseException:  no viable alternative at input 'create'(line 1, pos 38)== SQL == select * from table_1 create  where create.column_1= 1 ----------------------^^^com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.catalyst.parser.ParseException:
Following are the keywords standardized in Spark 3.0:
To disable the ANSI standard validation, Falsify spark.sql.ansi.enabled.
This feature is introduced for stern data quality checks during migration from SQL kind of environment. Below Insert table statement(Ingesting string into an integer column) would execute without any runtime exception in Spark 2.4 and below.
This will work fine in Spark 3.0 as well. But by setting up the property “spark.sql.storeAssignmentPolicy” as ‘ANSI’, casting exception is thrown.
This definitely improves data quality checks during the migration.
from_json method is used in parsing JSON type value within a column. In Spark 3.0, it supports PERMISSIVE/FAIL_FAST mode similar to spark.read.json. So, if the JSON value cannot be parsed or malformed, it will throw an exception as follows:
In Spark 3.0, numbers written in scientific notation(for example, 1E11) would be parsed as Double. In Spark version 2.4 and below, they’re parsed as Decimal. To restore the behavior before Spark 3.0, you can set spark.sql.legacy.exponentLiteralAsDecimal.enabled to false.
In Spark 2.4,
In Spark 3.0, and setting ‘spark.sql.legacy.exponentLiteralAsDecimal.enabled’ as true.
In Spark version 2.4 and below, float/double -0.0 is semantically equal to 0.0, but -0.0 and 0.0 are considered as different values when used in aggregate grouping keys, window partition keys, and join keys. It’s a Bug and it is fixed in Spark 3.0. Now, Distinct of (-0.0,0.0) will give (0.0).
Sample Dataset
With Spark 2.4, you will get as follows,
Whereas with Spark 3.0, with no additional property set, consideration will be the same irrespective of negative or positive.
In Spark 3.0, Certain keywords are supported for converting a string to date.
To Generate a ‘date’ from a string, Following are the keywords.
Possible keywords:
select date 'x''x' can have following values:> epoch -> 1970-01-01 (minimum date)> today -> the current date in the time zone specified by spark.sql.session.timeZone> yesterday -> the current date - 1> tomorrow -> the current date + 1> now -> the date of running the current query. It has the same notion as today
Similarly, We have an option for getting timestamp from string.
Possible keywords:
select timestamp 'x''x' can have following values> epoch -> 1970-01-01 00:00:00+00 (Unix system time zero)> today -> midnight today> yesterday -> midnight yesterday> tomorrow -> midnight tomorrow> now -> current query start time
In 3.0, Some of the SQL functions also have been updated/added, which has benefits in data aggregation and getting insights out of it in a simpler way. Find some of them below.
These functions are one of the significant features that I have noted. These will get you the value of a column, with respect to maximum/minimum some other column.
Usage: max_by(x,y) / min_by(x,y)
→ x = column from which value to be fetched.
→ y = column to get the aggregated maximum/minimum.
From the above data, the Overall Maximum Price is 2.598 and the Overall Minimum price is 2.514.
But with Max_by/Min_by, Maximum = 2.572, which is for the maximum order number 6 and Minimum = 2.548, which is for the minimum order number 1.
CTE using with clause can now be used within subqueries. This will improve query readability and would be significant in using multiple CTE’s.Something like this:
select * from (with inner_cte as (select * from sql_functions_test where order_num = 5)select order_num from inner_cte);
This feature would be helpful in aggregation only for a set of rows, filtered based on a condition.
This is an alternative to ‘replace’ function. But, It does have some advantages over replace.
Usage : overlay(input_string placing replace_string from start_position for number_of_positions_to_be_replaced)
Note: The input column should be a string.
This is to validate a column based on the condition expression provided.
Usage: Any(expr) , Every(expr), Some(expr)
EVERY: will return true, only if all column values return true.
ANY/SOME: will return true, even if one value returns true.
This is similar to filter(where..), which will give you the non-distinct count of column records, that satisfy the condition provided.
Usage: count_if(expr)
This is to validate boolean columns, mimicking AND /OR operation.
Usage: bool_and(column_value) , bool_or(column_value)
Considering the sample table, having a boolean column.
Applying BOOL_AND: returns true only if all the values are true.
Applying BOOL_OR: returns true even if one value is true.
Spark has not only added new features but fixed a few bugs with the earlier versions. Validation and quality checks have been easier than before. Functions like count with filter, max_by and min_by will reduce the complexity of executing multiple sub-queries.