av9ash/GitBugs huggingface dataset:
Bug: Aggregate in not causes 
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 13:58
Affects Version/s: 3.4.1

 

The below snippet is confirmed working with Spark 3.2.1 and broken Spark 3.4.1. i believe this is a bug. 
{code:java}
       Dataset<Row> ds = dummyDataset
                .withColumn("flag", functions.not(functions.coalesce(functions.col("bool1"), functions.lit(false)).equalTo(true)))
                .groupBy("code")
                .agg(functions.max(functions.col("flag")).alias("flag"));
        ds.show(); {code}
It fails with:
{code:java}
Caused by: java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:208)
	at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpression$7(V2ExpressionBuilder.scala:185)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:184)
	at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33)
	at org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803)
	at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateAggregateFunc(V2ExpressionBuilder.scala:293)
	at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98)
	at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33)
	at org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803)
	at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.translate$1(DataSourceStrategy.scala:700){code}
Bug: Map normalization upon creation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 10:14
Affects Version/s: 4.0.0

Add handling of map normalization upon creation in ArrayBasedMapBuilder. Currently a map with keys 0.0 and -0.0 will behave as if they are separate values. This will cause issues when doing GROUP BY on map types.

Refer to this conversion [https://github.com/apache/spark/pull/45549#discussion_r1537803505]
Bug: Action performed on Dataset is triggering an error as "The Spark SQL phase planning failed with an internal error."
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/24 16:47
Affects Version/s: 3.4.2

Hi team, 

We are upgrading to spark 3.4.2 from 3.3.2 and facing issue in the spark driver as below.

"org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace."

 

Complete stack trace is as attached.

After analysing further , found that when we try to perform any actions on the dataset, spark is unable to plan its actions, results in this exception. 

Attaching a Exception stack trace & code snippet where exactly we are trigging an action to convert it as java RDD.

 

Attaching the spark submit command as well for your reference.

Can you please check this issue let us know if any fix available for this issue.
Bug: Issue with Spark Connect on Python 3.12
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/24 07:19
Affects Version/s: 3.4.1

When trying to create a remote Spark session with PySpark on Python 3.12 a {{ModuleNotFoundError: No module named 'distutils'}} excpetion is thrown. In Python 3.12 {{distutils}} was removed from the stdlib. As a workaround we can {{import setuptools}} before creating the session. See also [this question on SOF|https://stackoverflow.com/questions/78207291] and the [answer|https://stackoverflow.com/a/78212125/11474852] by Anderson Bravalheri.
Bug: Improve picking the side of partially clustered distribution accroding to partition size
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/24 05:54
Affects Version/s: 4.0.0

Now we pick up the side of partially clustered distribution:

SPJ currently relies on a simple heuristic and always pick the side with less data size based on table statistics as the side fully clustered, even though it could also contain skewed partitions. 


We can potentially do fine-grained comparison based on partition values, since we have the information now.
Bug: date_format() java.lang.ArithmeticException: long overflow for large dates
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/24 18:12
Affects Version/s: 3.4.0

The following works for CATS, but not for DATE_FORMAT():

select  cast(cast('5881580' AS DATE) AS STRING);
+5881580-01-01

spark-sql (default)> select date_format(cast('5881580' AS DATE), 'yyyyyyy-mm-dd');

24/03/26 11:08:23 ERROR SparkSQLDriver: Failed in [select date_format(cast('5881580' AS DATE), 'yyyyyyy-mm-dd')]

java.lang.ArithmeticException: long overflow

 at java.base/java.lang.Math.multiplyExact(Math.java:1004)

 at org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.instantToMicros(SparkDateTimeUtils.scala:122)

 at org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.instantToMicros$(SparkDateTimeUtils.scala:116)

 at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:41)

 at org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.daysToMicros(SparkDateTimeUtils.scala:174)

 at org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.daysToMicros$(SparkDateTimeUtils.scala:172)

 at org.apache.spark.sql.catalyst.util.DateTimeUtils$.daysToMicros(DateTimeUtils.scala:41)

 at org.apache.spark.sql.catalyst.expressions.Cast.$anonfun$castToTimestamp$14(Cast.scala:642)

 at scala.runtime.java8.JFunction1$mcJI$sp.apply(JFunction1$mcJI$sp.scala:17)

 at org.apache.spark.sql.catalyst.expressions.Cast.buildCast(Cast.scala:557)

 at org.apache.spark.sql.catalyst.expressions.Cast.$anonfun$castToTimestamp$13(Cast.scala:642)

 at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:1170)

 at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:558)
Bug: StringRepeat (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 12:44
Affects Version/s: 4.0.0

Enable collation support for the *StringRepeat* built-in string function in Spark. First confirm what is the expected behaviour for these functions when given collated strings, then move on to the implementation that would enable handling strings of all collation types. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringRepeat* function so it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Spark history sever fails to display query for cached JDBC relation named in quotes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/24 12:12
Affects Version/s: 3.2.1

Spark history sever fails to display query for cached JDBC relation (or calculation derived from it)  named in quotes

(Screenshot and generated history in attachments)

How to reproduce:
{code:java}
val ticketsDf = spark.read.jdbc("jdbc:postgresql://localhost:5432/demo", """ "test-schema".tickets """.trim, properties)
val bookingDf = spark.read.parquet("path/bookings")

ticketsDf.cache().count()

val resultDf = bookingDf.join(ticketsDf, Seq("book_ref"))

resultDf.write.mode(SaveMode.Overwrite).parquet("path/result") {code}
 

So the problem is in SparkPlanGraphNode class which creates a dot node. When there is no metrics to display it simply returns tagged name and in this case name contains quotes which corrupts dot file.
Suggested solution is to escape name string
Bug: Cache miss for queries using With expressions
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Mar/24 18:47
Affects Version/s: 4.0.0

For example:
{noformat}
create or replace temp view v1 as
select id from range(10);

create or replace temp view q1 as
select * from v1
where id between 2 and 4;

cache table q1;

explain select * from q1;

== Physical Plan ==
*(1) Filter ((id#51L >= 2) AND (id#51L <= 4))
+- *(1) Range (0, 10, step=1, splits=8)
{noformat}
Similarly:
{noformat}
create or replace temp view q2 as
select count_if(id > 3) as cnt
from v1;

cache table q2;

explain select * from q2;

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[], functions=[count(if (NOT _common_expr_0#88) null else _common_expr_0#88)])
   +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=182]
      +- HashAggregate(keys=[], functions=[partial_count(if (NOT _common_expr_0#88) null else _common_expr_0#88)])
         +- Project [(id#86L > 3) AS _common_expr_0#88]
            +- Range (0, 10, step=1, splits=8)

{noformat}
In the output of the above explain commands, neither include an {{InMemoryRelation}} node.

The culprit seems to be the common expression ids in the {{With}} expressions used in runtime replacements for {{between}} and {{{}count_if{}}}, e.g. [this code|https://github.com/apache/spark/blob/39500a315166d8e342b678ef3038995a03ce84d6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Between.scala#L43].
Bug: Move `o.a.s.variant` to `o.a.s.types.variant`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/24 23:56
Affects Version/s: 4.0.0

{code}
-package org.apache.spark.variant;
+package org.apache.spark.types.variant;
{code}
 
{code:java}
-package org.apache.spark.sql.catalyst.expressions
+package org.apache.spark.sql.catalyst.expressions.variant
{code}
Bug: Add UserDefinedType support to DataTypeUtils.canWrite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/24 22:43
Affects Version/s: 4.0.0

Our customer hits an issue recently when they tries to save a DataFrame containing some UDTs as table (`saveAsTable`). The error looks like:

```
- Cannot write 'xxx': struct<...> is incompatible with struct<...>
```

The catalog strings between two sides are actually same which makes the customer confused.

It is because `DataTypeUtils.canWrite` doesn't handle `UserDefinedType`. If the `UserDefinedType`'s underlying sql type is same as read side, `canWrite` should return true for two sides.
Bug: Replace Deprecated `JsonParser#getCurrentName` with `JsonParser#currentName`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/24 14:06
Affects Version/s: 4.0.0

[https://github.com/FasterXML/jackson-core/blob/8fba680579885bf9cdae72e93f16de557056d6e3/src/main/java/com/fasterxml/jackson/core/JsonParser.java#L1521-L1551]

 
{code:java}
    /**
     * Deprecated alias of {@link #currentName()}.
     *
     * @return Name of the current field in the parsing context
     *
     * @throws IOException for low-level read issues, or
     *   {@link JsonParseException} for decoding problems
     *
     * @deprecated Since 2.17 use {@link #currentName} instead.
     */
    @Deprecated
    public abstract String getCurrentName() throws IOException;    /**
     * Method that can be called to get the name associated with
     * the current token: for {@link JsonToken#FIELD_NAME}s it will
     * be the same as what {@link #getText} returns;
     * for field values it will be preceding field name;
     * and for others (array values, root-level values) null.
     *
     * @return Name of the current field in the parsing context
     *
     * @throws IOException for low-level read issues, or
     *   {@link JsonParseException} for decoding problems
     *
     * @since 2.10
     */
    public String currentName() throws IOException {
        // !!! TODO: switch direction in 2.18 or later
        return getCurrentName();
    } {code}
Bug: Support json_length function
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/24 07:53
Affects Version/s: 4.0.0

At the moment, we don't support json_length built-in function in apache spark.

This function is supported by
 # presto: [https://prestodb.io/docs/current/functions/json.html#json_size]
 # clickhouse: [https://clickhouse.com/docs/en/sql-reference/functions/json-functions#jsonlengthjson-indices_or_keys]
 # mysql: [https://dev.mysql.com/doc/refman/8.0/en/json-attribute-functions.html#function_json-length]

 

*Definition*

json_length(json_txt, path) - Return the length of a JSON array or a JSON object.

If the value does not exist or has a wrong type, {{0}} will be returned.

Examples:

 
{code:java}
SELECT json_length('{"x": {"a": 1, "b": 2}}', '$.x'); -- 2
SELECT json_length('{"x": [1, 2, 3]}', '$.x'); -- 3
SELECT json_length('{"x": {"a": 1, "b": 2}}', '$.x.a'); -- 1{code}
 

*The advantages:*
 # skip parse phase, so its performance is better than _size(get_json_object(json_txt, path)_
 # it has more functions than _json_array_length_ and can be implemented in a unified manner
 # it allows naive users to directly get json length with a built-in json function instead of UDF
Bug: Add support for aggregation and join operations on arrays of collated strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 08:33
Affects Version/s: 4.0.0

Example of aggregation sequence:
{code:java}
create table t(a array<string collate utf8_binary_lcase>) using parquet;


insert into t(a) values(array('a' collate utf8_binary_lcase));
insert into t(a) values(array('A' collate utf8_binary_lcase));


select distinct a from t; {code}
Example of join sequence:
{code:java}
create table l(a array<string collate utf8_binary_lcase>) using parquet;
create table r(a array<string collate utf8_binary_lcase>) using parquet;


insert into l(a) values(array('a' collate utf8_binary_lcase));
insert into r(a) values(array('A' collate utf8_binary_lcase));


select * from l join r where l.a = r.a; {code}
Both runs should yield one row since the arrays are considered equal.
Bug: AQE doesn't allow for extension of InMemoryTableScanExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/24 15:21
Affects Version/s: 3.5.0

As part of SPARK-42101, we added support to AQE for handling InMemoryTableScanExec. 

This change directly references `InMemoryTableScanExec` which limits users from extending the caching functionality that was added as part of SPARK-32274
Bug: Disable spark.sql.parquet.inferTimestampNTZ.enabled by default
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/24 21:38
Affects Version/s: 4.0.0

Since Spark 4.0, the SQL config `spark.sql.parquet.inferTimestampNTZ.enabled` is turned off by default. Consequently, when reading Parquet files that were not produced by Spark, the Parquet reader will no longer automatically recognize data as the TIMESTAMP_NTZ data type. This change ensures backward compatibility with releases of Spark version 3.2 and earlier. It also aligns the behavior of schema inference for Parquet files with that of other data sources such as CSV, JSON, ORC, and JDBC, enhancing consistency across the data sources. To revert to the previous behavior where TIMESTAMP_NTZ types were inferred, set `spark.sql.parquet.inferTimestampNTZ.enabled` to true.
Bug: Validate alter table statements and check the numeric table stats
Status: Open | Priority: Major | Resolution: 
Created: 18/Mar/24 16:04
Affects Version/s: 2.4.8

SPARK-30262 resolved/avoided the NumberFormatException in Spark when the "totalSize", "numRows", "rawDataSize" table properties are empty, however the table stats (intentionally or by mistake) can be still set to an invalid/empty value through SparkSQL with an ALTER TABLE statement:
{code}
scala> spark.sql("alter table t1p set tblproperties('numRows'='', 'STATS_GENERATED_VIA_STATS_TASK'='true')").show()
{code}
 
Spark should validate the sparkSQL "alter table" statements and not allow non-numeric values in the "totalSize", "numRows", "rawDataSize" table properties.
Though the NumberFormatException will not occur anymore in Spark 3.x, these table stats should have numeric values and may cause problems in other applications if those are not numbers.

Note: beeline/Hive validates alter table statements.
Bug: startswith, endswith (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/24 06:42
Affects Version/s: 4.0.0

Implement *startsWith* and *endsWith* built-in string Spark functions using {_}StringSearch{_}, an efficient ICU service for string matching. Refer to the latest unit tests in CollationSuite to understand how these functions are used in SparkSQL, and feel free to use your chosen Spark SQL Editor to play around with the existing functions to learn more about how they work.

 

Currently, these 2 functions support all collation types:
 # binary collations (UCS_BASIC, UNICODE) *special cases - these collation types work using the existing string comparison functions - i.e. contains(), startsWith(), endsWith()
 # special lowercase non-binary collations (UCS_BASIC) *special case - these collation types work by using lower() to convert both strings to lowercase, and then use above functions
 # other non-binary collations (UNICODE_CI; special collations for various languages with case and accent sensitivity) - these collation types usually require special handling, which can sometimes be complex

 

To understand what changes were introduced in order to enable collation support for these functions, take a look at the Spark PRs and Jira tickets below:
 * [https://github.com/apache/spark/pull/45216] this PR enables:
 ** partial collation support for *contains* (skipping the 3rd type of collations shown above)
 ** complete collation support for {*}startsWith{*}, *endsWith* (using a special _matchAt_ implementation directly in {_}UTF8String{_})

 * [https://github.com/apache/spark/pull/45382] this PR enables:
 ** complete collation support for *contains* (using {_}StringSearch{_}) _-> now we should also use this approach for startsWith & endsWith_

 

Focusing on the 3rd type of collations as shown above, the goal for this Jira ticket is to re-implement the *startsWith* and *endsWith* functions so that they use _StringSearch_ instead (following the general logic in the second PR). As for the current test cases in CollationSuite, they should already mostly cover the expected behaviour of *startsWith* and *endsWith* for the 3rd type of collations.

 

Read more about _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: contains (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/24 11:46
Affects Version/s: 4.0.0

Implemented efficient collation-aware in-place substring comparison to enable collation support for: {_}contains{_}. Spark SQL users should now be able to use COLLATE within arguments for built-in string function: CONTAINS in Spark SQL queries (for non-binary collations).
Bug: Allow reading Parquet TimestampLTZ as TimestampNTZ
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/24 18:54
Affects Version/s: 4.0.0

Currently, Parquet TimestampNTZ type columns can be read as TimestampLTZ, while reading TimestampLTZ as TimestampNTZ will cause errors. This makes it impossible to read parquet files containing both TimestampLTZ and TimestampNTZ as TimestampNTZ.

To make the data type system on Parquet simpler, we should allow reading TimestampLTZ as TimestampNTZ in the Parquet data source.
Bug: Exclude `logback` dependency from SBT like Maven
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/24 22:36
Affects Version/s: 4.0.0

{code}
$ build/mvn dependency:tree --pl core | grep logback
Using `mvn` from path: /opt/homebrew/bin/mvn
Using SPARK_LOCAL_IP=localhost
{code}

{code}
$ build/sbt "core/test:dependencyTree" | grep logback
Using SPARK_LOCAL_IP=localhost
[info]   |       +-ch.qos.logback:logback-classic:1.2.13
[info]   |       | +-ch.qos.logback:logback-core:1.2.13
[info]   |       +-ch.qos.logback:logback-core:1.2.13
[info]   | | +-ch.qos.logback:logback-classic:1.2.13
[info]   | | | +-ch.qos.logback:logback-core:1.2.13
[info]   | | +-ch.qos.logback:logback-core:1.2.13
[info]   | +-ch.qos.logback:logback-classic:1.2.13
[info]   | | +-ch.qos.logback:logback-core:1.2.13
[info]   | +-ch.qos.logback:logback-core:1.2.13
{code}
Bug: Refactor and split list/timer unit tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/24 22:24
Affects Version/s: 4.0.0

Refactor ListState and timer related unit tests.
As planned in test plan for state-v2, list/timer should be tested in both integration and unit tests. Currently timer related tests could be refactored to use base suite class in {{{}ValueStateSuite{}}}, and list state unit tests are needed in addition to {{{}TransformWithListStateSuite{}}}.
Bug: Extend CustomTaskMetric to allow metric values from multiple sources
Status: Closed | Priority: Major | Resolution: Invalid
Created: 05/Mar/24 20:23
Affects Version/s: 4.0.0

Custom task metrics allow a DSV2 to add metrics that can be displayed in the UI. However, for DSV2 file sources, the FilePartitionReader may have multiple file readers, and each of these may report their metrics which need to be aggregated and bubbled up to the Scan. There is no way currently to update a metric value.
A new interface that extends CustomTaskMetric and defines a way to allow updates would allow a DSV2 file scan implementation to over come this.
Bug: Cancel running stage if the result is empty relation
Status: Open | Priority: Major | Resolution: 
Created: 19/Mar/24 06:44
Affects Version/s: 3.5.1

How to reproduce:
bin/spark-sql --master yarn --conf spark.driver.host=10.211.174.53
{code:sql}
set spark.sql.adaptive.enabled=true;
select a from (select id as a, id as b, id as z from range(100000000)) t1
join (select id as c, id as d from range(200000000)) t2 on t1.a = t2.c
join (select id as e, id as f from range(300000000)) t3 on t2.d = t3.e
where z % 10 < 0
group by 1;
{code}
Bug: Support collated strings in array operations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 15:46
Affects Version/s: 4.0.0

Collations need to be properly supported in following array operations but currently yield unexpected results: ArraysOverlap, ArrayDistinct, ArrayUnion, ArrayIntersect, ArrayExcept. Example query:

{code:java}
select array_contains(array('aaa' collate utf8_binary_lcase), 'AAA' collate utf8_binary_lcase){code}
We would expect the result of query to be true.