av9ash/GitBugs huggingface dataset:
Bug: Fix collation support for the StringToMap expression (binary & lowercase collation only)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/24 12:39
Affects Version/s: 4.0.0

Enable collation support for *StringToMap* built-in string function in Spark ({*}str_to_map{*}). First confirm what is the expected behaviour for this function when given collated strings, and then move on to implementation and testing. You will find this expression in the *complexTypeCreator.scala* file. However, this experssion is currently implemented as pass-through function, which is wrong because it doesn't provide appropriate collation awareness for non-default delimiters.

 

Example 1.
{code:java}
SELECT str_to_map('a:1,b:2,c:3', ',', ':');{code}
This query will give the correct result, regardless of the collation.
{code:java}
{"a":"1","b":"2","c":"3"}{code}
 

Example 2.
{code:java}
SELECT str_to_map('ay1xby2xcy3', 'X', 'Y');{code}
This query will give the *incorrect* result, under UTF8_LCASE collation. The correct result should be:
{code:java}
{"a":"1","b":"2","c":"3"}{code}
 

Update the corresponding E2E SQL tests (CollationSQLExpressionsSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringToMap* expression so that it supports UTF8_BINARY and UTF8_LCASE collations (i.e. StringTypeBinaryLcase). To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the related Spark PRs and Jira tickets for completed tasks in this parent (for example: https://issues.apache.org/jira/browse/SPARK-47414).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for string [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Create a function lookup registry for non-sql functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/24 15:23
Affects Version/s: 4.0.0

Create a dedicated function registry for Column API functions that are not supposed to be exposed through the regular SQL API.

 

A list of names and expressions:
 * years -> org.apache.spark.sql.catalyst.expressions.Years
 * months -> org.apache.spark.sql.catalyst.expressions.Months
 * days -> org.apache.spark.sql.catalyst.expressions.Days
 * hours -> org.apache.spark.sql.catalyst.expressions.Hours
 * bucket -> org.apache.spark.sql.catalyst.expressions.Bucket
 * unwrap_udt -> org.apache.spark.sql.catalyst.expressions.UnwrapUDT
 * timestampadd -> org.apache.spark.sql.catalyst.expressions.TimestampAdd
 * timestampdiff -> org.apache.spark.sql.catalyst.expressions.TimestampDiff
 * collect_top_k -> org.apache.spark.sql.catalyst.expressions.aggregate.CollectTopK
 * product -> org.apache.spark.sql.catalyst.expressions.aggregate.Product
Bug: [M0] Improve exceptions thrown from parser/interpreter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/24 07:54
Affects Version/s: 4.0.0

With SPARK-48350 we have introduced exception condition and states for exceptions thrown internally from SQL Scripting parser or interpreter.

However, these errors are throwing SparkException, which isn't completely suitable for scripting because with it we need to parametrize each exception to output the line where the error occurred. Example in: [https://github.com/apache/spark/pull/47404.]

We want to introduce a class for exceptions that will have Origin as a parameter, and then each message can automatically have prepended info about the line on which the exception occurred.

 

Introduction of Error class for SQL Scripting: [https://github.com/apache/spark/pull/47147]
Bug: Fix approx_count_distinct not working correctly with collation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/24 13:12
Affects Version/s: 4.0.0

When running in spark-shell:
{code:java}
create table t(col string collate utf8_lcase)
insert into t values 'a', 'a', 'A'
select approx_count_distinct(col) from t {code}
we get 2 as an answer, but it should be 1.
Bug: Fix Between with ScalarSubqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/24 21:18
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-46366 introduced a regression where types of queries like:
{code:java}
SELECT *
FROM t1
WHERE (select max(t2.t2c)
       from t2 where t1.t1b = t2.t2b
      ) between 1 and 20; {code}
would fail in optimizer rule.
Bug: TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/24 06:09
Affects Version/s: 4.0.0

SPARK-47363 added the support for users to provide initial state for streaming query. Such query operators like `TransformWithStateExec` might have `hasInitialState` as false which means the initial state related parameters are not used. But when query planner applies rules on the query, it will still apply on the initial state query plan. When `hasInitialState` is false, some related parameters  like `initialStateGroupingAttrs` are invalid and some rules will use these invalid parameters to transform the initial state query plan.

For example, `EnsureRequirements` may apply invalid Sort and Exchange on the initial query plan. We encountered these invalid query plan in our extension rules.
Bug: Rebasing in legacy formatters/parsers must support non JVM default time zones
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/24 01:19
Affects Version/s: 3.5.0

Currently, rebasing timestamp defaults to JVM timezone and so it produces incorrect results when the explicitly over-riden timezone in the TimestampFormatter library is not the same as JVM timezone.



To fix it, explicitly pass the overridden timezone parameter to {{rebaseJulianToGregorianMicros}} and {{rebaseGregorianToJulianMicros}}
Bug: Fix cached Variant with column size greater than 128KB or individual variant larger than 2kb
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/24 19:40
Affects Version/s: 4.0.0

Currently, the `actualSize` method of the `VARIANT` `columnType` isn't overridden, so we use the default size of 2kb for the `actualSize`. We should define `actualSize` so the cached variant column can correctly be written to the byte buffer
Bug: Fix the discrepancy between implementation, comment and documentation of option recursive.fields.max.depth in ProtoBuf connector
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/24 17:25
Affects Version/s: 3.5.0

After the three PRs ([https://github.com/apache/spark/pull/38922,] [https://github.com/apache/spark/pull/40011,] [https://github.com/apache/spark/pull/40141]) working on the same option, there are some legacy comments and documentation that has not been updated to the latest implementation. This task should consolidate them. Below is the correct description of the behavior.

The `recursive.fields.max.depth` parameter can be specified in the from_protobuf options to control the maximum allowed recursion depth for a field. Setting `recursive.fields.max.depth` to 1 drops all-recursive fields, setting it to 2 allows it to be recursed once, and setting it to 3 allows it to be recursed twice. Attempting to set the `recursive.fields.max.depth` to a value greater than 10 is not allowed. If the `recursive.fields.max.depth` is specified to a value smaller than 1, recursive fields are not permitted. The default value of the option is -1. if a protobuf record has more depth for recursive fields than the allowed value, it will be truncated and some fields may be discarded. This check is based on the fully qualified field type. SQL Schema for the protobuf message
{code:java}
message Person { string name = 1; Person bff = 2 }{code}
will vary based on the value of `recursive.fields.max.depth`.
{code:java}
1: struct<name: string>
2: struct<name string, bff: <name: string>>
3: struct<name string, bff: <name: string, bff: struct<name: string>>> ...
{code}
Bug: Emit Kubernetes event when driver fails to request executor
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/24 13:50
Affects Version/s: 3.5.3

In Kubernetes, when a driver pod fails to request executor pods (i.e due to being out of resource quota), the only visibility around this issue is inside the driver logs. 

We would like to expose this issue as a Kubernetes driver event to enhance debugging. A possible solution is to add event emission logic in ExecutorPodsAllocator.scala when we fail to request executors:
[https://bbgithub.dev.bloomberg.com/dnaspark/apache-spark-internal/blob/develop-3.4/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala#L439-L463]
Bug: Investigate OrcEncryptionSuite UnsatisfiedLinkError on AppleSilicon MacOS environment
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 30/Jul/24 04:56
Affects Version/s: 4.0.0

{code}
git reset --hard 49b4c3bc9c09325de941dfaf41e4fd3a4a4c345f // [SPARK-45393][BUILD] Upgrade Hadoop to 3.4.0
build/sbt clean "sql/testOnly org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite"
...
[info] OrcEncryptionSuite:
12:42:55.441 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12:42:57.950 WARN org.apache.hadoop.crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: 'boolean org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl()'
	at org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl(Native Method)
	at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:86)
	at org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.<init>(OpensslAesCtrCryptoCodec.java:36)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[info] - Write and read an encrypted file (2 seconds, 486 milliseconds)
[info] - Write and read an encrypted table (402 milliseconds)
[info] - SPARK-35325: Write and read encrypted nested columns (299 milliseconds)
[info] - SPARK-35992: Write and read fully-encrypted columns with default masking (623 milliseconds)
12:42:59.856 WARN org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite: 

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.datasources.orc.OrcEncryptionSuite, threads: rpc-boss-3-1 (daemon=true), Thread-17 (daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true), shuffle-boss-6-1 (daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true), Thread-18 (daemon=true), ForkJoinPool.commonPool-worker-3 (daemon=true) =====
[info] Run completed in 5 seconds, 291 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
{code}
Bug: Add `reason` string to all job / stage / job group cancellation calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/24 22:56
Affects Version/s: 4.0.0

Today it is difficult to determine _why_ a job, stage, or job group was canceled. We should leverage existing Spark functionality to provide a {{reason}} string explaining the cancellation cause, and should add new APIs to let us provide this reason when canceling job groups.

{*}Details{*}:
 * Since SPARK-19549 Allow providing reasons for stage/job cancelling - ASF JIRA (Spark 2.20), Spark’s {{cancelJob}} and {{cancelStage}} methods accept an optional {{reason: String}} that is added to logging output and user-facing error messages when jobs or stages are canceled. In our internal calls to these methods, we should always supply a reason. For example, we should set an appropriate reason when the “kill” links are clicked in the Spark UI (see [code|https://github.com/apache/spark/blob/b14c1f036f8f394ad1903998128c05d04dd584a9/core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala#L54C1-L55]).

 * Other APIs currently lack a {{reason}} field. For example, {{cancelJobGroup}} and {{cancelJobsWithTag}} don’t provide any way to specify a reason, so we only see generic logs like “asked to cancel job group <group name>”. We should add an ability to pass in a group cancellation reason and thread that through into the scheduler’s logging and job failure reasons.

This feature can be implemented in two PRs:
 # Modify the current {{SparkContext}} and its downstream APIs to add the {{reason}} string, such as {{cancelJobGroup}} and {{cancelJobsWithTag}}

      2. Add reasons for all internal calls to these methods
Bug: Remote (most) hard coded expressions from SparkConnectPlanner/functions.scala
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/24 01:16
Affects Version/s: 4.0.0

There are a number of hard coded expressions in the SparkConnectPlanner. Most of these expressions are hardcoded because they are missing a proper constructor, or because they are not registered in the FunctionRegistry. functions.scala has a similar problem. We should try to remove these exceptions.
Bug: Avoid `raw` type usage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/24 17:50
Affects Version/s: kubernetes-operator-0.1.0

{code}
-          Class listenerClass = Class.forName(name);
+          Class<?> listenerClass = Class.forName(name);
{code}

{code}
-  public static Object resolveValueToPrimitiveType(Class clazz, String value) {
+  public static Object resolveValueToPrimitiveType(Class<?> clazz, String value) {
{code}
Bug: Column.endswith(None) occasionally causes NPE
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/24 21:53
Affects Version/s: 4.0.0

This one is pretty hard to repro, since it only seems to happen occasionally.

Invoking `Column.endswith()` seems to result in an NPE, with Spark 4.0:

{code:python}
from pyspark.sql.types import *
import pyspark.sql.functions as f

schema = StructType([StructField("s", StringType(), True)])

strings = [Row("abc"), Row("bcd"), Row(None)]

df = sc.parallelize(strings).toDF(schema)

df.select( f.col('s').endswith(None) ).collect()
{code}

Here is the resulting stack trace:
{code}
py4j.protocol.Py4JJavaError: An error occurred while calling o205.endsWith.
: java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.Column.expr()" because "x$1" is null
  at org.apache.spark.sql.Column$.$anonfun$fn$2(Column.scala:77)
  at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:75)
  at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35)
  at org.apache.spark.sql.Column$.$anonfun$fn$1(Column.scala:77)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:84)
  at org.apache.spark.sql.package$.withOrigin(package.scala:111)
  at org.apache.spark.sql.Column$.fn(Column.scala:76)
  at org.apache.spark.sql.Column$.fn(Column.scala:64)
  at org.apache.spark.sql.Column.fn(Column.scala:169)
  at org.apache.spark.sql.Column.endsWith(Column.scala:1078)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
  at py4j.Gateway.invoke(Gateway.java:282)
  at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  at py4j.commands.CallCommand.execute(CallCommand.java:79)
  at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
  at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
  at java.base/java.lang.Thread.run(Thread.java:840)
{code}

This seems to point to {{Column::fn}}, which looks new to Spark 4.0.
Bug: CONV function behaves inconsistently
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/24 11:37
Affects Version/s: 3.5.1

I'm currently working on CONV function, and I found something confused about the implementation in Spark.
All codes below is from NumberConverter.scala.
h3.  negative and signed situation
{code:sql}
spark-sql (default)> select conv('FFFFFFFFFFFFFFFE', 16, -16);
-2
spark-sql (default)> select conv('-FFFFFFFFFFFFFFFE', 16, -16);
-2
{code}
Ideally, these two queries should yield different results, but they both return -2.
{code:java}
    if (toBase < 0 && v < 0) {
      v = -v
      negative = true
    }
{code}
According to code above, when toBase < 0 and v < 0, negative sign is set to true regardless of the original value. This will lead to incorrect result as the examples above, because negative sign is ignored in the second case. A potential adjustment is negative = !negative, which would correctly interpret the double negation and yield 2.
h3. ansi mode
{code:java}
if (negative && toBase > 0) {
      if (v < 0) {
        v = -1
      } else {
        v = -v
      }
    }
{code}
Here, -1 is used to indicate an overflow condition but does not throw an exception when ANSI mode is enabled, unlike the overflow handling in the encode method.
h3. overflow check
{code:java}
val bound = java.lang.Long.divideUnsigned(-1 - radix, radix)
if (v >= bound) {...}
{code}
The inclusion of the equality in the overflow check seems unnecessary.

 
----
I am still learning function in Spark. Please feel free to point out any mistakes I might have. And some of these questions are also mentioned in [SPARK-44943|https://issues.apache.org/jira/browse/SPARK-44943].
Bug: Unexpected behavior using spark mask function handle string contains invalid UTF-8 or wide character
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/24 03:47
Affects Version/s: 3.2.4

In the spark the mask function when apply with a string contains invalid character or wide character would cause unexpected behavior.

Example to use `*` mask a string contains wide-character {{🙂}}
{code:sql}
select mask("🙂", "Y", "y", "n", "*");
{code}
could cause result is ** instead of *. Looks spark mask treat {{🙂}} as 2 characters.

Example to use wide-character {{🙂}} do mask would cause wrong garbled code problem
{code:sql}
select mask("ABC", "🙂");
{code}
result is `???`.

Example to mask a string contains a invalid UTF-8 character
{code:java}
select mask("\xED");
{code}
result is `xXX` instead of `\xED`, looks spark treat it as four character `\`, `x`, `E`, `D`.

Looks spark mask can only handle BMP character (that is 16 bits) and can't guarantee result for invalid UTC-8 character and wide-character when doing mask.

My question here is *does that the limitation / issue of spark mask function or spark mask by design only handle for BMP character ?*

If it is a limitation of mask function, could spark address this part in mask function document or comments ?
Bug: Add OFFSET operator as an option in the subquery generator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/24 19:10
Affects Version/s: 4.0.0

{{GeneratedSubquerySuite}} is a test suite that generates SQL with variations of subqueries. Currently, the operators supported are Joins, Set Operations, Aggregate (with/without group by) and Limit. Implementing OFFSET will increase coverage by 1 additional axis, and should be simple.
Bug: View fails with internal error after upgrade causes expected syntax error.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/24 18:02
Affects Version/s: 4.0.0

On older Spark:
CREATE VIEW v AS SELECT 1 ! IN (2);
SEELCT * FROM v;
=> true
Upgrade to Spark 4
SELECT * FROM v;
Internal error 

This makes it hard to debug the problem.
Rather than assuming that failure to parse a view text is an internal error we should assume something like upgrade broke it and expose the actual error
Bug: NPE in DataSourceV2ScanExecBase redact method when session is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/24 08:35
Affects Version/s: 4.0.0

If we call DataSourceV2ScanExecBase redact method from a thread that don't have a session in thread local we get an NPE.

[https://github.com/apache/spark/blob/3f6e2d6ef354c034ce0b39a092d23dd99431a16c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala#L70C3-L75C4]

 

Changing 
{code:java}
session.sessionState.conf.stringRedactionPattern {code}
to 
{code:java}
conf.stringRedactionPattern  {code}
could prevent this problem as conf checks if session is null or not.

[https://github.com/apache/spark/blob/af5eb085b0ac01eec2bb5db206b7d499d462abc4/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L69]

In case of DataSourceScanExec we already use conf.stringRedactionPattern:
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L93-L95]
Bug: Rename unreleased try_remainder() function to try_mod()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/24 00:43
Affects Version/s: 4.0.0

the try_remainder() function is the try_* version of `%` and `mod`.
As such, given that there is NO `remainder()` function and no other product seems to have try_remainder() we want to rename try_remainder to try_mod()
Bug: ScalaUDF in subquery should run through analyzer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/24 02:28
Affects Version/s: 3.4.3

We got a customer issue that a `MergeInto` query on Iceberg table works earlier but cannot work after upgrading to Spark 3.4.

The error looks like

```
Caused by: org.apache.spark.SparkRuntimeException: Error while decoding: org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to nullable on unresolved object
upcast(getcolumnbyordinal(0, StringType), StringType, - root class: java.lang.String).toString.
```

The source table of `MergeInto` uses `ScalaUDF`. The error happens when Spark invokes the deserializer of input encoder of the `ScalaUDF` and the deserializer is not resolved yet.

The encoders of ScalaUDF are resolved by the rule `ResolveEncodersInUDF` which will be applied at the end of analysis phase.

During rewriting `MergeInto` to `ReplaceData` query, Spark creates an `Exists` subquery and `ScalaUDF` is part of the plan of the subquery. Note that the `ScalaUDF` is already resolved by the analyzer.

Then, in `ResolveSubquery` rule which resolves the subquery, it will resolve the subquery plan if it is not resolved yet. Because the subquery containing `ScalaUDF` is resolved, the rule skips it so `ResolveEncodersInUDF` won't be applied on it. So the analyzed `ReplaceData` query contains a `ScalaUDF` with encoders unresolved that cause the error.
Bug: Add inequality (!=, <, <=, >, >=) predicates for correlation in GeneratedSubquerySuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/24 19:10
Affects Version/s: 4.0.0

{{GeneratedSubquerySuite}} is a test suite that generates SQL with variations of subqueries. Currently, the operators supported are Joins, Set Operations, Aggregate (with/without group by) and Limit. Implementing inequality (!=, <, <=, >, >=) predicates will increase coverage by 1 additional axis, and should be simple.