av9ash/GitBugs huggingface dataset:
Bug: Support GROUP BY calendar_interval_type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/23 09:50
Affects Version/s: 4.0.0

Currently, Spark GROUP BY only allows orderable data types, otherwise the plan analysis fails: [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala#L197-L203]

However, this is too strict as GROUP BY only cares about equality, not ordering. The CalendarInterval type is not orderable (1 month and 30 days, we don't know which one is larger), but has well-defined equality. In fact, we already support `SELECT DISTINCT calendar_interval_type` in some cases (when hash aggregate is picked by the planner).

The proposal here is to officially support calendar interval type in GROUP BY. We should relax the check inside `CheckAnalysis`, and make `CalendarInterval` implements `Comparable` using natural ordering (compare months first, then days, then seconds), and test with both hash aggregate and sort aggregate.
Bug: Fix the ambiguous column reference in self join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/23 06:14
Affects Version/s: 4.0.0

{code:java}
In [20]: df1 = spark.range(10).withColumn("a", sf.lit(0))

In [21]: df2 = spark.range(10).withColumn("a", sf.lit(0)).withColumnRenamed("a", "b")

In [22]: df1.join(df2, df1["a"] == df2["b"])
Out[22]: DataFrame[id: bigint, a: int, id: bigint, b: int]

In [23]: df1.join(df2, df1["a"] == df2["b"]).drop("b")
Out[23]: DataFrame[id: bigint, a: int, id: bigint]

In [24]: df1 = spark.range(10).withColumn("a", sf.lit(0))

In [25]: df2 = df1.withColumnRenamed("a", "b")

In [26]: df1.join(df2, df1["a"] == df2["b"])
Out[26]: 23/12/22 09:33:28 ERROR ErrorUtils: Spark Connect RPC error during: analyze. UserId: ruifeng.zheng. SessionId: eaa2161f-4b64-4dbf-9809-af6b696d3005.
org.apache.spark.sql.AnalysisException: [AMBIGUOUS_COLUMN_REFERENCE] Column a is ambiguous. It's because you joined several DataFrame together, and some of these DataFrames are the same.
This column points to one of the DataFrame but Spark is unable to figure out which one.
Please alias the DataFrames with different names via DataFrame.alias before joining them,
and specify the column using qualified name, e.g. df.alias("a").join(df.alias("b"), col("a.id") > col("b.id")). SQLSTATE: 42702
	at org.apache.spark.sql.catalyst.analysis.ColumnResolutionHelper.findPlanById(ColumnResolutionHelper.scala:555)
	at org.apache.spark.sql.catalyst.analysis.ColumnResolutionHelper.$anonfun$resolveUnresolvedAttributeByPlanId$2(ColumnResolutionHelper.scala:511)
	at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:469)
	at org.apache.spark.sql.catalyst.analysis.ColumnResolutionHelper.resolveUnresolvedAttributeByPlanId(ColumnResolutionHelper.scala:511)
	at org.apache.spark.sql.catalyst.analysis.ColumnResolutionHelper.tryResolveColumnByPlanId(ColumnResolutionHelper.scala:494)
	at org.apache.spark.sql.catalyst.analysis.ColumnResolutionHelper.$anonfun$tryResolveColumnByPlanId$2(ColumnResolutionHelper.scala:497)
	at scala.collection.immutable.List.map(List.scala:246){code}
Bug: Add spark dataset chunk read API (python only)
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 11/Dec/23 09:52
Affects Version/s: 4.0.0

*Design doc:*
h1. [https://docs.google.com/document/d/1LHzwCjm2SluHkta_08cM3jxFSgfF-niaCZbtIThG-H8/edit#heading=h.cxcvohcybvo2]

 

*Proposed API:*
{code:java}
def persist_dataframe_as_chunks(dataframe: DataFrame) -> list[str]:
    """
    Persist and materialize the spark dataframe as chunks, each chunk is an arrow batch.
    It tries to persist data to spark worker memory firstly, if memory is not sufficient,
    then it fallbacks to persist spilled data to spark worker local disk.
    Return the list of chunk ids.
    This function is only available when it is called from spark driver process.
    """

def read_chunk(chunk_id):
    """
    Read chunk by id, return this chunk as an arrow table.
    You can call this function from spark driver, spark python UDF python,
    descendant process of spark driver, or descendant process of spark python UDF worker.
    """

def unpersist_chunks(chunk_ids: list[str]) -> None:
    """
    Remove chunks by chunk ids.
    This function is only available when it is called from spark driver process.
    """{code}
*Motivation:*

(1)
In Ray on spark, we want to support loading Ray data from arbitrary spark Dataframe with in-memory conversion,

for Ray on spark, Ray datasource read-task runs as child process of Ray worker node, and in Ray on spark, we launch Ray worker node as child process of pyspark UDF worker.

So that the above proposed API allows descendent python process of pyspark UDF worker to read a chunk data of given spark dataframe, based on this, we can achieve efficient "spark DataFrame" to "Ray dataset" conversion.

(2)
For spark torch distributor, we want to implement an efficient {{Torch DataLoader}} that loads data from spark dataframe without saving spark dataframe to cloud storage. This API makes it feasible. This issue has a similar pattern to use-case (1)

(3)
For petastorm spark dataset converter (see [https://www.databricks.com/blog/2020/06/16/simplify-data-conversion-from-apache-spark-to-tensorflow-and-pytorch.html]) , using the added API, we can achieve better performance
Bug: SELECT * EXCEPT(all fields from a struct) results in an assertion failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/23 14:45
Affects Version/s: 3.0.0

This example
{code:sql}
-- Removing all fields results in an empty struct
> SELECT * EXCEPT(c1.a) FROM VALUES(named_struct('a', 1)) AS t(c1);
{code}
throws an AssertionError during serialization:
{code:java}
AssertionError: assertion failed: each serializer expression should contain at least one `BoundReference`
{code}

instead of just return an empty struct
Bug: Update resource-managers/kubernetes/integration-tests/README.md to java 17 and 21
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/23 20:43
Affects Version/s: 4.0.0

In the file resource-managers/kubernetes/integration-tests/README.md 
change java 8 to 17 and 11 to 21 
-Dspark.kubernetes.test.sparkTgz=spark-3.0.0-SNAPSHOT-bin-example.tgz \
to 4 
OpenJDK to azul/zulu-openjdk
Bug: Upgrade `datasketches-java` to 5.0.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Dec/23 14:01
Affects Version/s: 4.0.0

* [https://github.com/apache/datasketches-java/releases/tag/4.0.0]
 * [https://github.com/apache/datasketches-java/releases/tag/4.1.0]
 * [https://github.com/apache/datasketches-java/releases/tag/4.2.0]
 * [https://github.com/apache/datasketches-java/releases/tag/5.0.0]
 * [https://github.com/apache/datasketches-java/releases/tag/5.0.1|https://github.com/apache/datasketches-java/releases/tag/5.0.0]
Bug: Pin the bundler version in CI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/23 23:05
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/7226413850/job/19691970695

{code}
Requirement already satisfied: docutils<0.18.0 in /usr/local/lib/python3.9/dist-packages (0.17.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
ERROR:  Error installing bundler:
	The last version of bundler (>= 0) to support your Ruby & RubyGems was 2.4.22. Try installing it with `gem install bundler -v 2.4.22`
	bundler requires Ruby version >= 3.0.0. The current ruby version is 2.7.0.0.
{code}
Bug: critical CVE vulnerability with a fix in Derby
Status: Open | Priority: Major | Resolution: 
Created: 05/Dec/23 11:06
Affects Version/s: 3.4.1

 

It would be necessary to upgrade Derby dependency in order to solve a critical vulnerability that was fixed in the latest release of Derby in November:

[https://db.apache.org/derby/releases/release-10_17_1_0.cgi]

https://issues.apache.org/jira/browse/DERBY-7147?focusedCommentId=17799544&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17799544

 

 

The vuln:

```

│                   Library                    │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │                            Title                             │

│ org.apache.derby:derby (derby-10.14.2.0.jar) │ CVE-2022-46337 │ CRITICAL │ fixed  │ 10.14.2.0         │ 10.17.1.0     │ A cleverly devised username might bypass LDAP authentication │

```
Bug: autoBroadcastJoinThreshold compared to project of a plan not a relation size
Status: Open | Priority: Major | Resolution: 
Created: 26/Dec/23 17:07
Affects Version/s: 3.1.1

From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum size in bytes for a table that will be broadcasted to all worker nodes when performing a join.

[https://spark.apache.org/docs/3.5.0/configuration.html#runtime-sql-configuration]

In fact Spark compares plan.statistics.sizeInBytes  of a project (columns selected in join), not a relation size.

[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala#L368]

Join can select only a few columns and sizeInBytes will be lesser than autoBroadcastJoinThreshold, but broadcasted table can be huge and it is loaded entirely into drivers memory which can lead to OOM.

spark.sql.autoBroadcastJoinThreshold parameter compared to projection instead of broadcasted table size seems quite risky feature: there will be more broadcasted relations but more chances to get OOM on the driver too.

The solution is to disable spark.sql.autoBroadcastJoinThreshold and set hints on really small relations, but in that case autoBroadcastJoinThreshold seems useless.  It would be more usefull to have autoBroadcastJoinThreshold which campres to relations size and have predicted memory usage on the driver.

 

Original task and test when autobroadcast compared to relation totalSize:

https://issues.apache.org/jira/browse/SPARK-2393

[https://github.com/apache/spark/pull/1238/files#diff-00485e6cae519f81adca5ceee66227c6eae35db709619d505468f8765175ac18R39]

 

Task and PR where autoBroadcastJoinThreshold started to be compared to project of a plan instead of relations:

https://issues.apache.org/jira/browse/SPARK-13329

[https://github.com/apache/spark/pull/11210]

 

Related topic on SO: [https://stackoverflow.com/questions/74435020/how-dataframe-count-selects-broadcasthashjoin-while-dataframe-show-selects-s]
Bug: sha2(df.a, 1024) throws a different exception in Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/23 22:23
Affects Version/s: 4.0.0

{code}
from pyspark.sql import functions as sf
spark.range(1).select(sf.sha2(sf.col("id"), 1024)).collect()
{code}

Non-connect:

{code}
...
pyspark.errors.exceptions.captured.IllegalArgumentException: requirement failed: numBits 1024 is not in the permitted values (0, 224, 256, 384, 512)
{code}

Connect:

{code}
...
pyspark.errors.exceptions.connect.AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "sha2(id, 1024)" due to data type mismatch: Parameter 1 requires the "BINARY" type, however "id" has the type "BIGINT". SQLSTATE: 42K09;
'Project [unresolvedalias(sha2(id#1L, 1024))]
+- Range (0, 1, step=1, splits=Some(1))
{code}
Bug: Support timestamp types in UnwrapCastInBinaryComparison
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Dec/23 09:11
Affects Version/s: 4.0.0

We have an optimization rule `UnwrapCastInBinaryComparison` that handles similar cases but it doesn't cover timestamp types.

For a query plan like

```
== Analyzed Logical Plan ==
batch: timestamp
Project [batch#26466]
+- Filter (batch#26466 >= cast(2023-12-21 10:00:00 as timestamp))
   +- SubqueryAlias spark_catalog.default.timestamp_view
      +- View (`spark_catalog`.`default`.`timestamp_view`, [batch#26466])
         +- Project [cast(batch#26467 as timestamp) AS batch#26466]
            +- Project [cast(batch#26463 as timestamp) AS batch#26467]
               +- SubqueryAlias spark_catalog.default.table_timestamp
                  +- Relation spark_catalog.default.table_timestamp[batch#26463] parquet

== Optimized Logical Plan ==
Project [cast(batch#26463 as timestamp) AS batch#26466]
+- Filter (isnotnull(batch#26463) AND (cast(batch#26463 as timestamp) >= 2023-12-21 10:00:00))
   +- Relation spark_catalog.default.table_timestamp[batch#26463] parquet
```

The predicate compares a timestamp_ntz column with a literal value. As the column is wrapped in a cast expression to timestamp type, the literal (string) is wrapped with a cast to timestamp type. The literal with cast is foldable so it is evaluated to literal of timestamp earlier. So the predicate becomes `cast(batch#26463 as timestamp) >= 2023-12-21 10:00:00`. As the cast is in column side, it cannot be pushed down to data source/table.
Bug: Single Hive/Datasource table partitioning to multiple storage system- (e.g, S3 and HDFS)
Status: Open | Priority: Major | Resolution: 
Created: 27/Dec/23 08:47
Affects Version/s: 3.3.2

Single Hive/Datasource table partitioning to multiple storage system- (e.g, S3 and HDFS)

For Hive table:

 
{code:java}
CREATE  TABLE htable a string, b string)  PARTITIONED BY ( p string ) location "hdfs://{cluster}}/user/hadoop/htable/";

alter table htable  add partition(p='p1')  location 's3a://{bucketname}/usr/hive/warehouse/htable/p=p1';{code}
 

When inserting into htable,  or insert overwrite htable.  New data of “p=p1” will insert table location storage. This does not meet the requirements.

Is there any best practise?  Or is there a plan to support this feature?

Thanks
Bug: Support for copy from write compatible postgresql databases (pg, redshift, snowflake, gauss)
Status: Open | Priority: Major | Resolution: 
Created: 27/Dec/23 01:43
Affects Version/s: 4.0.0

Now many databases are compatible with pg syntax and support copy from syntax. The copy form import performance is 10 times higher than that of jdbc batch.

[https://github.com/melin/datatunnel/blob/master/connectors/jdbc/src/main/scala/com/superior/datatunnel/plugin/jdbc/support/CopyHelper.scala]

Supports upsert data import： [https://github.com/melin/datatunnel/blob/master/connectors/jdbc/src/main/scala/com/superior/datatunnel/plugin/jdbc/support/DataTunnelJdbcRelationProvider.scala]

!image-2023-12-27-09-44-19-292.png!

 

[~yao]
Bug: Optimize spark jdbc write speed with Multi-Row Inserts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Dec/23 07:16
Affects Version/s: 4.0.0

INSERT INTO table_name (column1, column2, column3)
VALUES (value1, value2, value3),
(value4, value5, value6),
(value7, value8, value9);
Bug: Change Utils.isJavaVersionAtLeast21 to use the utility method from commons-lang3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/23 11:33
Affects Version/s: 4.0.0

commons-lang3 added support for checking `JAVA_21` after version 3.13.0, so we can directly use the utility methods provided by commons-lang3.
Bug: literal function is giving wrong value for minimum datetime
Status: Open | Priority: Major | Resolution: 
Created: 22/Dec/23 18:56
Affects Version/s: 3.3.0

While using PySpark functions f.lit to read datetime.min value, it is giving wrong value. 

Below is the screenshot for the same,

!image-2023-12-22-10-56-30-473.png!

Expectation is to get same value as datetime.min in case of f.lit(datetime.min).
Bug: Replacing current time prior to inline table eval
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/23 13:52
Affects Version/s: 4.0.0

When current time/date specific functions are being used as part of inline tables each invocation will be evaluated. Instead, proper behaviour would be to replace time expressions with current time/date and always return single value. Example


SELECT COUNT(DISTINCT ct) FROM VALUES
(CURRENT_TIMESTAMP()),
(CURRENT_TIMESTAMP()),
(CURRENT_TIMESTAMP()) as data(ct)
 
Is supposed to return 1, while currently it returns 3.
Bug: COUNT bug in lateral/exists subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/23 01:32
Affects Version/s: 3.0.0

Some further instances of a COUNT bug.

 

One example is this test from join-lateral.sql

[https://github.com/apache/spark/blame/master/sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out#L757]

 

According to PostgreSQL, the query should return 2 rows:

c1 | c2 | sum

---{-}++{-}--{-}{-}----

  0 |  1 |   2

  1 |  2 |    NULL

 

whereas Spark SQL only returns the first one.

 

Similar instance is the following query, which should return 1 row from t1 but has an empty result now:

{{create temporary view t1(c1, c2) as values (0, 1), (1, 2);}}
{{create temporary view t2(c1, c2) as values (0, 2), (0, 3);}}

{{SELECT tt1.c2}}
{{FROM t1 as tt1}}
{{WHERE tt1.c1 in (}}
select max(tt2.c1)
from t2 as tt2
 where tt1.c2 is null);
Bug: Revert SPARK-43049 due to performance regression of using CLOB
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Dec/23 21:20
Affects Version/s: 3.5.0

SPARK-43049 causes performance regression when writing string fields to an Oracle database due to strings written as CLOB instead of VARCHAR2. CLOB is known to have bad performance in Oracle so when creating a table and writing to it using Spark, the internal SQL statement that writes 20+ string fields would take at least 5x performance hit from the original patch (2 min vs 10+ min).

I confirmed internally that running a job with the commit reverted brings back the original performance numbers.
Bug: Set the rebase configs to the CORRECTED mode by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/23 07:26
Affects Version/s: 4.0.0

Set all rebase related SQL configs to the `CORRECTED` mode by default. Here are the affected configs:
- spark.sql.parquet.int96RebaseModeInWrite
- spark.sql.parquet.datetimeRebaseModeInWrite
- spark.sql.parquet.int96RebaseModeInRead
- spark.sql.parquet.datetimeRebaseModeInRead
- spark.sql.avro.datetimeRebaseModeInWrite
- spark.sql.avro.datetimeRebaseModeInRead

The configs were set to the `EXCEPTION` mode to give users a choice to select proper mode for compatibility with old Spark versions <= 2.4.5. Those versions are not able to detect the rebase mode from meta information in parquet and avro files. Since the versions are out of broad usage, Spark starting from the version 4.0.0 will write/ read ancient datatime without rebasing and any exceptions. This should be more convenient for users.
Bug: Support EXECUTE IMMEDIATE sytax in Spark SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/23 15:51
Affects Version/s: 3.5.0

Introducing new EXECUTE IMMEDIATE syntax to support parameterized queries from within SQL.

This API executes query passed as string with arguments.

Other DBs that support this:
 * [Oracle|https://docs.oracle.com/cd/B13789_01/appdev.101/b10807/13_elems017.htm]
 * [Snowflake|https://docs.snowflake.com/en/sql-reference/sql/execute-immediate]
 * [PgSql|https://www.postgresql.org/docs/current/ecpg-sql-execute-immediate.html#:~:text=Description,statement%2C%20without%20retrieving%20result%20rows.]
Bug: The stage that retries due to FetchFailed will cause the downstream skewJoin AQE become ineffective
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/23 10:00
Affects Version/s: 3.5.0

In our production environment, a SQL statement containing a Union all was executed. Due to a FetchFailed issue in Stage 24.0 of the first Union subquery, after rerunning some stages, Stage 46.0, which was supposed to follow the Skew Join rule, is now unable to take effect.
Bug: Change Python Data Source's description to be pretty in explain
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/23 02:06
Affects Version/s: 4.0.0

Now it's as below:

{code}

== Physical Plan ==
*(1) Project [x#0, y#1]
+- BatchScan test[x#0, y#1] class org.apache.spark.sql.execution.python.PythonTableProvider$$anon$1$$anon$2 RuntimeFilters: []
{code}
Bug: Decode parquet binary with getBytesUnsafe method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/23 08:28
Affects Version/s: 4.0.0

Now spark will get a parquet binary object with getBytes() method.

The *Binary.getBytes()* method will always make a new copy of the internal bytes.

We can use *Binary.getBytesUnsafe()* method to get the cached bytes if it has already been called getBytes() and has the cached bytes.

!image-2023-12-14-16-30-39-104.png!
Bug: Spark Connect Repl does not work with ClosureCleaner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/23 14:59
Affects Version/s: 4.0.0

SPARK-45136 added ClosureCleaner support to SparkConnect client. Unfortunately, this change breaks ConnectRepl launched by `./connector/connect/bin/spark-connect-scala-client`. To reproduce the issue:
 # Run `./connector/connect/bin/spark-connect-shell`
 # Run  `./connector/connect/bin/spark-connect-scala-client`
 # In the REPL, execute this code:
```
@ def plus1(x: Int): Int = x + 1
@ val plus1_udf = udf(plus1 _)
```

This will fail with the following error:

```

java.lang.reflect.InaccessibleObjectException: Unable to make private native java.lang.reflect.Field[] java.lang.Class.getDeclaredFields0(boolean) accessible: module java.base does not "opens java.lang" to unnamed module @45099dd3
  java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
  java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
  java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
  java.lang.reflect.Method.setAccessible(Method.java:193)
  org.apache.spark.util.ClosureCleaner$.getFinalModifiersFieldForJava17(ClosureCleaner.scala:577)
  org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:560)
  org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18(ClosureCleaner.scala:533)
  org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18$adapted(ClosureCleaner.scala:525)
  scala.collection.ArrayOps$WithFilter.foreach(ArrayOps.scala:73)
  org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16(ClosureCleaner.scala:525)
  org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16$adapted(ClosureCleaner.scala:522)
  scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
  scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
  scala.collection.AbstractIterable.foreach(Iterable.scala:933)
  scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:903)
  org.apache.spark.util.ClosureCleaner$.cleanupAmmoniteReplClosure(ClosureCleaner.scala:522)
  org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:251)
  org.apache.spark.sql.expressions.SparkConnectClosureCleaner$.clean(UserDefinedFunction.scala:210)
  org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:187)
  org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:180)
  org.apache.spark.sql.functions$.udf(functions.scala:7956)
  ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1)
  ammonite.$sess.cmd1$.<clinit>(cmd1.sc:7)

```

 

This is because ClosureCleaner is heavily reliant on using reflection API and is not compatible with Java 17. The rest of Spark bypasses this by adding `--add-opens` JVM flags, see https://issues.apache.org/jira/browse/SPARK-36796. We need to add these options to Spark Connect Client launch script as well