av9ash/GitBugs huggingface dataset:
Bug: The ArrayInsert function should make explicit casting when element type not equals derived component type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/23 11:24
Affects Version/s: 3.4.1

Generally speaking, array_insert has same insert semantic with  array_prepend/array_append. however, if we run sql use element cast like below, array_prepend/array_append can get right result. but array_insert failed.
{code:java}
spark-sql (default)> select array_prepend(array(1), cast(2 as tinyint));
[2,1]
Time taken: 0.123 seconds, Fetched 1 row(s) {code}
{code:java}
spark-sql (default)> select array_append(array(1), cast(2 as tinyint)); 
[1,2] 
Time taken: 0.206 seconds, Fetched 1 row(s)
{code}
{code:java}
spark-sql (default)> select array_insert(array(1), 2, cast(2 as tinyint));
[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] Cannot resolve "array_insert(array(1), 2, CAST(2 AS TINYINT))" due to data type mismatch: Input to `array_insert` should have been "ARRAY" followed by a value with same element type, but it's ["ARRAY<INT>", "TINYINT"].; line 1 pos 7;
'Project [unresolvedalias(array_insert(array(1), 2, cast(2 as tinyint)), None)]
+- OneRowRelation {code}
The reported error is clear, however, we may should do explicit casting here. because multiset type such as array or map allow the operands of same type family  to coexist.
Bug: Shutting down SparkConnectClient can have outstanding ReleaseExecute
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/23 17:38
Affects Version/s: 4.0.0

In Spark Connect scala client, there can be outstanding asynchronous ReleaseExecute calls when the client is shutdown. 

In ExecutePlanResponseReattachableIterator we (ab)use a grpc thread in createRetryingReleaseExecuteResponseObserver to run it.

 
When we do {{{}SparkConnectClient.shutdown{}}}, which does {{channel.shutdownNow()}}
it kills it before it finishes the ReleaseExecute. Maybe a more graceful shutdown would work, or we should move to a more explicit threadpool that we manage, like in python?

See discussion in https://github.com/apache/spark/pull/42929/files#r1329071826
Bug: Unsupported map and array constructors by `sql()` in connect clients
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/23 16:25
Affects Version/s: 4.0.0

The code below demonstrates the issue:

{code:scala}
spark.sql("select element_at(?, 1)", Array(array(lit(1)))).collect()
{code}

It fails with the error:

{code:java}
[info]   java.lang.UnsupportedOperationException: literal unresolved_function {
[info]   function_name: "array"
[info]   arguments {
[info]     literal {
[info]       integer: 1
[info]     }
[info]   }
[info] }
[info]  not supported (yet).
{code}
Bug: Avoid Spark connect ML model to change input pandas dataframe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/23 08:31
Affects Version/s: 4.0.0

Currently, to avoid data copy, Spark connect ML model directly changes input pandas dataframe for appending prediction columns. But we can use `pandas_df.copy(deep=False)` to shallow copy it and then append prediction columns in copied dataframe. This is easier for user to use it.
Bug: GenerateExec fails to initialize non-deterministic expressions before use
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/23 16:37
Affects Version/s: 3.5.0

The following query fails:
{noformat}
select *
from explode(
  transform(sequence(0, cast(rand()*1000 as int) + 1), x -> x * 22)
);
{noformat}
The error is:
{noformat}
23/09/14 09:27:25 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.IllegalArgumentException: requirement failed: Nondeterministic expression org.apache.spark.sql.catalyst.expressions.Rand should be initialized before eval.
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.sql.catalyst.expressions.Nondeterministic.eval(Expression.scala:497)
	at org.apache.spark.sql.catalyst.expressions.Nondeterministic.eval$(Expression.scala:495)
	at org.apache.spark.sql.catalyst.expressions.RDG.eval(randomExpressions.scala:35)
	at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.eval(arithmetic.scala:384)
	at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:543)
	at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.eval(arithmetic.scala:384)
	at org.apache.spark.sql.catalyst.expressions.Sequence.eval(collectionOperations.scala:3062)
	at org.apache.spark.sql.catalyst.expressions.SimpleHigherOrderFunction.eval(higherOrderFunctions.scala:275)
	at org.apache.spark.sql.catalyst.expressions.SimpleHigherOrderFunction.eval$(higherOrderFunctions.scala:274)
	at org.apache.spark.sql.catalyst.expressions.ArrayTransform.eval(higherOrderFunctions.scala:308)
	at org.apache.spark.sql.catalyst.expressions.ExplodeBase.eval(generators.scala:375)
	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$8(GenerateExec.scala:108)
...        
{noformat}
However, this query succeeds:
{noformat}
select *
from explode(
  sequence(0, cast(rand()*1000 as int) + 1)
);
{noformat}
The difference is that {{transform}} turns off whole-stage codegen, which exposes a bug in {{GenerateExec}} where the non-deterministic expression passed to the generator function is not initialized before being used.

An even simpler reprod case is:
{noformat}
set spark.sql.codegen.wholeStage=false;

select explode(array(rand()));
{noformat}
Bug: Unsupported map/array constructors via `call_function` in parameterized `sql()`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/23 06:04
Affects Version/s: 4.0.0

The example below demonstrates the issue:

{code:scala}
scala> sql("SELECT element_at(:mapParam, 'a')", Map("mapParam" -> call_function("map", lit("a"), lit(1))))
org.apache.spark.sql.catalyst.ExtendedAnalysisException: [UNBOUND_SQL_PARAMETER] Found the unbound parameter: mapParam. Please, fix `args` and provide a mapping of the parameter to a SQL literal.; line 1 pos 18;
'NameParameterizedQuery [mapParam='map(a, 1)]
+- 'Project [unresolvedalias('element_at(namedparameter(mapParam), a), None)]
   +- OneRowRelation
{code}
Bug: There are ProblemFilters in the wrong place in MimaExcludes.scala.
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/23 16:13
Affects Version/s: 3.5.1

The targets of SPARK-43265, SPARK-44255, SPARK-44475, SPARK-44496, SPARK-43997 are all Apache 3.5.0, and ProblemFilters related to them should be in v35excludes, not in defaultExcludes, in order to re-participate in the Mima check in the Spark 4.0.0 cycle.
Bug: Downgrade scala-maven-plugin to 4.7.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/23 05:39
Affects Version/s: 4.0.0

The `scala-maven-plugin` versions 4.7.2 and later will try to automatically append the `-release` option as a Scala compilation argument when it is not specified by the user:

1. 4.7.2 and 4.8.0: Try to add the `-release` option for Scala versions 2.13.9 and higher.
2. 4.8.1: Try to append the `-release` option for Scala versions 2.12.x/2.13.x/3.1.1, and append `-java-output-version` for Scala 3.1.2.

The addition of the `-release` option has caused issues mentioned in SPARK-44376 | https://github.com/apache/spark/pull/41943 and https://github.com/apache/spark/pull/40442#issuecomment-1474161688. This is because the `-release` option has stronger compilation restrictions than `-target`, ensuring not only bytecode format, but also that the API used in the code is compatible with the specified version of Java. However, many APIs in the `sun.*` package are not `exports` in Java 11, 17, and 21, such as `sun.nio.ch.DirectBuffer`, `sun.util.calendar.ZoneInfo`, and `sun.nio.cs.StreamDecoder`, making them invisible when compiling across different versions.

For discussions within the Scala community, see https://github.com/scala/bug/issues/12643, https://github.com/scala/bug/issues/12824, https://github.com/scala/bug/issues/12866.

I have also submitted an issue to the `scala-maven-plugin` community to discuss the possibility of adding additional settings to control the addition of the `-release` option: https://github.com/davidB/scala-maven-plugin/issues/722.

For Apache Spark 4.0, in the short term, I suggest downgrading `scala-maven-plugin` to version 4.7.1 to avoid the automatic adding the `-release` option as a Scala compilation argument. In the long term, we should reduce our use of APIs that are not `exports` for compatibility with the `-release` compilation option.
Bug: Fix eas_decrypt and ln in connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/23 08:25
Affects Version/s: 3.5.0

The current {{eas_descrypt}} reference to {{aes_encrypt}} is clearly a bug. The {{ln}} reference to {{log}} is more like a cosmetic issue, but because {{ln}} and {{log}} function implementations are different in Spark SQL we should use the same implementation in Spark Connect too.
Bug: Multi-tenant history server
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/23 04:48
Affects Version/s: 3.4.1

Spark history server makes use of the configuration "spark.history.fs.logDirectory" to locate the log events. This works well for a single tenant. When it is used for a multi-tenant deployment, the log events of multiple tenants are stored in a single directory which does not provide a logical separation of events for each tenant.

The proposal/wish is to have a support for Multi-tenant history server, where-in the configuration "spark.history.fs.logDirectory" can be a base directory. The sub-directories can contain the log events for each tenant. The sub-directories can be named after each tenant, for e.g. "tenant1", "tenant2" etc.

When it is combined to work with Spark Driver/Executor which makes use of the property "spark.eventLog.dir", the value of this property can be appropriately set for each tenant.
Bug:  percentile_cont gets internal error when user input fails runtime replacement's input type check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/23 14:19
Affects Version/s: 3.3.2

This query throws an internal error rather than producing a useful error message:
{noformat}
select percentile_cont(b) WITHIN GROUP (ORDER BY a DESC) as x 
from (values (12, 0.25), (13, 0.25), (22, 0.25)) as (a, b);

[INTERNAL_ERROR] Cannot resolve the runtime replaceable expression "percentile_cont(a, b)". The replacement is unresolved: "percentile(a, b, 1)".
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot resolve the runtime replaceable expression "percentile_cont(a, b)". The replacement is unresolved: "percentile(a, b, 1)".
	at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
	at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$6(CheckAnalysis.scala:313)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$6$adapted(CheckAnalysis.scala:277)
...
{noformat}
It should instead inform the user that the input expression must be foldable.

{{PercentileCont}} does not check the user's input. If the runtime replacement (an instance of {{Percentile}}) rejects the user's input, the runtime replacement ends up unresolved.
Bug: reflect() fails with an internal error on NULL class and method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/23 13:22
Affects Version/s: 3.3.2

The example below demonstrates the issue:

{code:sql}
spark-sql (default)> select reflect('java.util.UUID', CAST(NULL AS STRING));
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
{code}
Bug: percentile_approx() fails with an internal error on NULL accuracy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/23 11:46
Affects Version/s: 3.3.2

The example below demonstrates the issue:

{code:sql}
spark-sql (default)> SELECT percentile_approx(col, array(0.5, 0.4, 0.1), NULL) FROM VALUES (0), (1), (2), (10) AS tab(col);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
{code}
Bug: Kafka DSv2 streaming source implementation calls planInputPartitions 4 times per microbatch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/23 13:31
Affects Version/s: 4.0.0

I was tracking through method calls for DSv2 streaming source, and figured out planInputPartitions is called 4 times per microbatch.

It turned out that multiple calls of planInputPartitions is due to `DataSourceV2ScanExecBase.supportsColumnar`, though it is called through `MicroBatchScanExec.inputPartitions` which is defined as lazy, hence shouldn't happen.

The behavior seems to be coupled with catalyst and very hard to figure out why, but with SPARK-44505, we can at least fix this per each data source.
Bug: spark delete can use parallel commit?
Status: Open | Priority: Major | Resolution: 
Created: 07/Sep/23 03:19
Affects Version/s: 3.4.0

In spark procudure, we have partial-progress.enabled which enable committing groups of files in partial way, so in delete or merge sql, should we run in a similar way? because there are multiple writes for a iceberg table,  in this situation, we cloud have some commit conflicts, then delete tasks will be failed. 
In the code, we can see a final commit after running each DataWritingSparkTask , in this siutation we can move commit into each task?
    try {
      sparkContext.runJob(
        rdd,
        (context: TaskContext, iter: Iterator[InternalRow]) =>
          DataWritingSparkTask.run(writerFactory, context, iter, useCommitCoordinator),
        rdd.partitions.indices,
        (index, result: DataWritingSparkTaskResult) => {
          val commitMessage = result.writerCommitMessage
          messages(index) = commitMessage
          totalNumRowsAccumulator.add(result.numRows)
          batchWrite.onDataWriterCommit(commitMessage)
        }
      )

      logInfo(s"Data source write support $batchWrite is committing.")
      batchWrite.commit(messages)
      logInfo(s"Data source write support $batchWrite committed.")
      commitProgress = Some(StreamWriterCommitProgress(totalNumRowsAccumulator.value))
Bug: Describe the binary and datetime formats of `to_char`/`to_varchar`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Sep/23 13:19
Affects Version/s: 4.0.0

In the PR, I propose to document the recent changes related to the `format` of the `to_char`/`to_varchar` functions:
1. binary formats added by https://github.com/apache/spark/pull/42632
2. datetime formats introduced by https://github.com/apache/spark/pull/42534
Bug: Connect: Use UUIDv7 for operation IDs to make operations chronologically sortable
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 01/Sep/23 12:26
Affects Version/s: 3.5.1

Spark Connect currently uses UUIDv4 for operation IDs. Using UUIDv7 instead allows us to sort operations by ID to receive a chronological order while keeping the collision-free properties we require from this ID.
Bug: to_char() fails with an internal error on NULL format
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/23 14:44
Affects Version/s: 4.0.0

The example below demonstrates the issue:

{code:sql}
spark-sql (default)> SELECT to_char(x'537061726b2053514c', CAST(NULL AS STRING));
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
{code}
Bug: HiveExternalCatalog.listPartitions should restore Spark SQL stats
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/23 18:14
Affects Version/s: 3.2.4

If partitions are stored in HMS with Spark populated stats such as {{spark.sql.statistics.totalSize}}, currently {{HiveExternalCatalog.listPartitions}} doesn't call {{restorePartitionMetadata}} to restore those stats.
Bug: ToPrettyString will treat java encoder or kryo encoder as BinaryType
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/23 12:32
Affects Version/s: 3.5.0

{code:java}
// spark sql
// java encoder or kryo encoder
implicit val im = Encoders.kryo[java.util.Map[Int, Int]].asInstanceOf[ExpressionEncoder[util.Map[Int, Int]]]

val ds = spark.createDataset(Seq(Collections.singletonMap(1,1)))

ds.show()
// +--------------------+
// |               value|
// +--------------------+
// |[01 00 6A 61 76 6...|
// +--------------------+

ds.collect().foreach(println(_))
// {1=1}
 {code}
ds.collect().foreach(println) is more readable than ds.show(), not sure is the designed intentionally.
Bug: Scala-specific improvements in Dataset[T] API 
Status: Open | Priority: Minor | Resolution: 
Created: 14/Sep/23 15:13
Affects Version/s: 3.4.1

*Q1.* What are you trying to do? 

The main idea is to use the power of scala's macrosses to give developers more convenient and typesafe API to use in join conditions. 

 

*Q2.* What problem is this proposal NOT designed to solve?

R/Java/Python/DataFrame API is out of scope. The solution is not affecting plan generation too. 

 

*Q3.* How is it done today, and what are the limits of current practice?

Currently the join condition is specified via strings, which might lead to silly mistakes (typos, incompatible column types etc) and sometimes hard to read (in case when several joins are made and the final type is tuple of tuple of tuples...)

 

*Q4.* What is new in your approach and why do you think it will be successful?

Scala macroses can be used to extract the column name directly from lambda (extractor). As a side effect its possible to check the column type and prohibit to build inconsistent join expression (like boolean-timestamp comparison)

 

*Q5.* Who cares? If you are successful, what difference will it make?

Mainly scala developers who prefers typesafe code - they would have a more clean and nice API that will make the codebase a bit clearer, especially in case when several chained joins is used

 

*Q6.* What are the risks?

The overusage of macrosses may slow down the compilation speed. In additional macrosses are hard to maintain

 

*Q7.* How long will it take?

Currently the approach is already implemented as a separate [lib|https://github.com/Salamahin/joinwiz] that makes a bit more than just gives alternative API (for example abstracts Dataset[T] to F[T] which allows to run some spark-specific code without spark session for testing purposes)

Adaptation of it won't be a hard job, matter of several weeks

 

*Q8.* What are the mid-term and final “exams” to check for success?

API convenience is very hard to estimate as its more or less a question of taste

 

*Appendix A*

You may find the examples of such 'cleaner' API [here|https://github.com/Salamahin/joinwiz/blob/master/joinwiz_core/src/test/scala/joinwiz/ComputationEngineTest.scala]

Note that backward and forward compatibility is achieved by introducing a brand-new API without modifying an old one
Bug: Update the Oracle docker image version used for test and integration to use Oracle Database 23c Free
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Sep/23 09:12
Affects Version/s: 3.5.0

This proposes to update the Docker image used for integration tests and builds to Oracle Database 23c Free.

The Docker image used for integration tests and builds currently uses Oracle XE version 21.3.0. Oracle 21 support ends in April 2024. The latest Oracle release is 23c, it is a long-term release supported till 2032. With Oracle 23c, Oracle has changed the name of the free version of its database, from Oracle XE (Express Edition) to Oracle Database Free.
Bug: Fix one unit mistake related to spark.eventLog.buffer.kb
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Sep/23 05:51
Affects Version/s: 3.5.0

The configuration "spark.eventLog.buffer.kb" has a definition with KiB unit.
{code:java}
private[spark] val EVENT_LOG_OUTPUT_BUFFER_SIZE = ConfigBuilder("spark.eventLog.buffer.kb")
  .doc("Buffer size to use when writing to output streams, in KiB unless otherwise specified.")
  .version("1.0.0")
  .bytesConf(ByteUnit.KiB)
  .createWithDefaultString("100k"){code}
It is used in a context acting as the size of bytes.
{code:java}
protected val outputBufferSize = sparkConf.get(EVENT_LOG_OUTPUT_BUFFER_SIZE).toInt {code}
This may be a mistake and needs to be fixed.

The default buffer size for the event-log output buffer will be 100k as configured.
Bug: Fix one unit mistake related to spark.eventLog.buffer.kb
Status: Open | Priority: Minor | Resolution: 
Created: 26/Sep/23 05:50
Affects Version/s: 3.5.0

The configuration "spark.eventLog.buffer.kb" has a definition with KiB unit.
{code:java}
private[spark] val EVENT_LOG_OUTPUT_BUFFER_SIZE = ConfigBuilder("spark.eventLog.buffer.kb")
  .doc("Buffer size to use when writing to output streams, in KiB unless otherwise specified.")
  .version("1.0.0")
  .bytesConf(ByteUnit.KiB)
  .createWithDefaultString("100k"){code}
It is used in a context acting as the size of bytes.
{code:java}
protected val outputBufferSize = sparkConf.get(EVENT_LOG_OUTPUT_BUFFER_SIZE).toInt {code}
This may be a mistake and needs to be fixed.

The default buffer size for the event-log output buffer will be 100k as configured.
Bug: Ignore task completion from old stage after retrying indeterminate stages
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Sep/23 18:39
Affects Version/s: 3.3.2

SPARK-25342 Added a support for rolling back shuffle map stage so that all tasks of the stage can be retried when the stage output is indeterminate. This is done by clearing all map outputs at the time of stage submission. This approach workouts well except for this case:

Assume both Shuffle 1 and 2 are indeterminate

ShuffleMapStage1 –{-}–{-}> Shuffle 1 ---–> ShuffleMapStage2 ----> Shuffle 2 ----> ResultStage
 * ShuffleMapStage1 is complete
 * A task from ShuffleMapStage2 fails with FetchFailed. Other tasks are still running
 * Both ShuffleMapStage1 and ShuffleMapStage2 are retried
 * ShuffleMapStage1 is retried and completes
 * ShuffleMapStage2 reattempt is scheduled for execution
 * Before all tasks of ShuffleMapStage2 reattempt could finish, one/more laggard tasks from the original attempt of ShuffleMapStage2 finish and ShuffleMapStage2 also gets marked as complete
 * Result Stage gets scheduled and finishes

Internally within Uber, we have been using the stage rollback functionality even for deterministic stages from Spark 2.4.3 to add fault tolerance from server going down in [remote shuffle service |https://github.com/uber/RemoteShuffleService]and have faced this scenario quite often

Ideally, such laggard tasks should not be considered towards the partition completion.
Bug: Remove org.scala-lang scala-library added for JDK 11 workaround 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Sep/23 03:29
Affects Version/s: 4.0.0

https://github.com/apache/spark/pull/25800 added 

{code}
    <!-- SPARK-28932 This is required in JDK11 -->
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
    </dependency>
{code}

Now with JDK 17 it works without them
Bug: Remove JDK 8/11 workaround in KryoSerializerBenchmark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Sep/23 03:46
Affects Version/s: 4.0.0

https://github.com/apache/spark/pull/25966 added a bit of extra flags for JDK 8/11 consistency. We don't need them anymore because we dropped JDK 8/11