av9ash/GitBugs huggingface dataset:
Bug: Add Streaming related fields to log4j ThreadContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/24 22:51
Affects Version/s: 4.0.0

There are some special informations needed for structured streaming queries. Specifically, each query has a query_id and run_id. Also if using MicrobatchExecution (default), there is a batch_id.

 

A (query_id, run_id, batch_id) identifies the microbatch the streaming query runs. Adding these field to a threadContext would help especially when there are multiple queries running.
Bug: In spark ML, replace RDD read / write API invocation with Dataframe read / write API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/24 12:39
Affects Version/s: 4.0.0

In spark ML, replace RDD read / write API invocation with Dataframe read / write API.

 

Motivation:

In databricks runtime, RDD read / write API has some issue for certain storage types that requires the account key, but Dataframe read / write API works.
Bug: Update doc's version field to align with fixedVersion field of JIRA ticket
Status: Open | Priority: Major | Resolution: 
Created: 16/Jul/24 02:30
Affects Version/s: 4.0.0

When a PR merged into more than 2 branches and if it contains version information, for example:
- since tag for scala/java API
- versionadded directive for python APIs
- .version(..) method for configurations
- Since Versions in documentation
- Since annotation in expression information
- e.t.c

we currently add the smallest version to the code. Take 3.4.4, 3.5.2, 4.0.0 for an example, it logs as fixed/since 3.4.4, it's easy for users who stay in 3.4.x and upgrade to 4.x but hard for those who use 3.5.x to know whether it gets fixed or not.

So, I suggest we make these fields more informative which means logging '3.4.4, 3.5.2, 4.0.0' together, which also aligns with the JIRA ticket "Fix Version/s:" field.

See:
https://github.com/apache/spark/commit/d8820a07eb82acd35a1c9d4ff6ee4f65fc6aac29
Bug: GenericUDF Can not CatchException From Child UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/24 11:39
Affects Version/s: 3.5.0

During the upgrade from Spark 3.3.1 to 3.5.1, we encountered syntax issues with this pr. The problem arose from DeferredObject currently passing a value instead of a function, which prevented users from catching exceptions in GenericUDF, resulting in semantic differences.

Here is an example case we encountered. Originally, the semantics were that {{str_to_map_udf}} would throw an exception due to issues with the input string, while {{merge_map_udf}} could catch the exception and return a null value. However, currently, any exception encountered by {{str_to_map_udf}} will cause the program to fail.
{code:java}
select merge_map_udf(str_to_map_udf(col1), parse_map_udf(col2), map("key", "value")) from table {code}
Bug: String trim function returning incorrect results in Spark connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/24 08:47
Affects Version/s: 3.5.0

With spark connect, the snippet: 


 
{code:java}
import org.apache.spark.sql.functions.trim 
import spark.implicits._ 

val df = Seq((" john"), ("jane "), (" ann ")).toDF("name") 
val trimmedDF = df.withColumn("name", trim($"name", " ")) 
trimmedDF.collect(){code}
returns an incorrect result
{code:java}
Array[Row] = Array([], [], []){code}
We get the same result when using `{{{}rtrim{}}}` and `{{{}ltrim{}}}` as well.
Bug: Add documentation for new options added to State Data Source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/24 18:48
Affects Version/s: 4.0.0

In [https://github.com/apache/spark/pull/46944] and [https://github.com/apache/spark/pull/47188], we introduced some new options to the State Data Source. This task aims to explain these new features in the documentation.
Bug: spark.excludeOnFailure.killExcludedExecutors does not take effect in Spark on YARN mode.
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/24 09:39
Affects Version/s: 3.4.3

In one of my Spark jobs, I enabled *spark.excludeOnFailure.killExcludedExecutors=true* and *spark.excludeOnFailure=true*, but I found that the excluded executor was not killed. The logs contain the following error message:
{code:java}
Not attempting to kill executors on excluded host cdh-001 since allocation client is not defined.
{code}
I checked the relevant code and found that the reason is that the *healthTracker* is initialized before the *sc.schedulerBackend*, causing *sc.schedulerBackend* to be null when *healthTracker* is initialized.
{code:scala}
private def maybeCreateHealthTracker(sc: SparkContext): Option[HealthTracker] = {
  if (HealthTracker.isExcludeOnFailureEnabled(sc.conf)) {
    val executorAllocClient: Option[ExecutorAllocationClient] = sc.schedulerBackend match {
      case b: ExecutorAllocationClient => Some(b)
      case _ => None
    }
    Some(new HealthTracker(sc, executorAllocClient))
  } else {
    None
  }
} {code}
Bug: Incorrect AnalysisException thrown using when() and mixed data types
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/24 06:50
Affects Version/s: 3.3.0

Observe the following sample code, where I'm using when() based on typeof():
{code:java}
from pyspark.sql.types import (
    ArrayType,
    IntegerType,
    StringType,
    StructField,
    StructType,
)

schema = StructType(
    [
        StructField("ID", IntegerType(), nullable=False),
        StructField("name", StringType(), nullable=False),
        StructField("colors", ArrayType(StringType()), nullable=False),
    ]
)
data = [
    (1, "John", ["red", "blue", "green"]),
    (2, "Jane", ["yellow", "orange", "purple"]),
    (3, "Bob", ["black", "white"]),
    (4, "Alice", ["pink"]),
    (5, "Tom", ["brown", "gray"]),
]
df = spark.createDataFrame(data, schema)
col = "name"
df = df.withColumn(
    col,
    F.when(F.expr(f"typeof({col}) == 'string'"), F.trim(col))
    .when(
        F.expr(f"typeof({col}) LIKE 'array%'"),
        F.array_join(col, ","),
    )
    .otherwise(F.lit(None)),
)
{code}
 
Here's the exception I'm seeing:
{noformat}
pyspark.sql.utils.AnalysisException: cannot resolve 'array_join(name, ',')' due to data type mismatch: argument 1 requires array<string> type, however, 'name' is of string type.;
'Project [ID#0, CASE WHEN (typeof(name#1) = string) THEN trim(name#1, None) WHEN typeof(name#1) LIKE array% THEN array_join(name#1, ,, None) ELSE null END AS name#6, colors#2]
+- LogicalRDD [ID#0, name#1, colors#2], false
{noformat}
 

If I change col to "colors", I get this similar exception:
{noformat}
pyspark.sql.utils.AnalysisException: cannot resolve 'trim(colors)' due to data type mismatch: argument 1 requires string type, however, 'colors' is of array<string> type.;
'Project [ID#0, name#1, CASE WHEN (typeof(colors#2) = string) THEN trim(colors#2, None) WHEN typeof(colors#2) LIKE array% THEN array_join(colors#2, ,, None) ELSE null END AS colors#6]
+- LogicalRDD [ID#0, name#1, colors#2], false
{noformat}
 

It seems to try to evaluate all possible paths of code for type checking, even if that code path won't be hit for the current query. I was able to repro this on 3.3.0 and 3.4.0.
Bug: ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class org.apache.spark.sql.catalyst.util.ArrayData when parsing JSON with "spark.sql.json.enablePartialResults" enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/24 02:37
Affects Version/s: 3.5.1

When enabling "spark.sql.json.enablePartialResults" and running the following query: 
{code:java}
select from_json('{"a":"b","c":"d"}', 'array<struct<a:string, c:int>>'){code}
the code fails with: 
{code:java}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4) (ip-10-110-51-101.us-west-2.compute.internal executor driver): java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class org.apache.spark.sql.catalyst.util.ArrayData (org.apache.spark.unsafe.types.UTF8String and org.apache.spark.sql.catalyst.util.ArrayData are in unnamed module of loader 'app')
    at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray(rows.scala:53)
    at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray$(rows.scala:53)
    at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:172)
    at org.apache.spark.sql.catalyst.expressions.JsonToStructs.$anonfun$converter$2(jsonExpressions.scala:831)
    at org.apache.spark.sql.catalyst.expressions.JsonToStructs.nullSafeEval(jsonExpressions.scala:893)
    at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:687)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) {code}
Bug: MultiInsert is split to multiple sql executions, resulting in no exchange reuse
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/24 06:58
Affects Version/s: 3.5.1

MultiInsert is split to multiple sql executions, resulting in no exchange reuse.

 

Reproduce sql:
{code:java}
create table wangzhen_t1(c1 int);
create table wangzhen_t2(c1 int);
create table wangzhen_t3(c1 int);

insert into wangzhen_t1 values (1), (2), (3);

from (select /*+ REPARTITION(3) */ c1 from wangzhen_t1)
insert overwrite table wangzhen_t2 select c1
insert overwrite table wangzhen_t3 select c1; {code}
 

In Spark 3.1, there is only one SQL execution and there is a reuse exchange.

!image-2024-07-05-14-59-35-340.png!

 

However, in Spark 3.5, it was split to multiple executions and there was no ReuseExchange.

!image-2024-07-05-16-42-01-973.png!

!image-2024-07-05-16-42-17-817.png!!image-2024-07-05-16-42-34-738.png!!image-2024-07-05-16-42-46-500.png!
Bug: State Data Source Read Change Feed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/24 23:14
Affects Version/s: 4.0.0

The current state reader can only return the entire state at a specific version. If an error occurs related to state, knowing the change of state across versions to find out at which version state starts to go wrong is important for debugging purposes. This PR adds ability of showing the evolution of state as Change Data Capture (CDC) format to state data source.

An example usage:
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required 
.option("changeEndBatchId", 10)  #not required, default: latest batch Id available
{code}
Bug: [CONNECT] session.stop() should be best effort
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/24 14:27
Affects Version/s: 3.5.1

The stop() on a connect session calls ReleaseSession RPC.

However, if the session is already dead, this call throws an error. Generate warnings instead since calling stop on a dead session is a no-op.

Further stop() API should be idempotent.
Bug: Resubmitting a stage without verifying the stage attempt number may result in an infinite loop
Status: Open | Priority: Major | Resolution: 
Created: 09/Jul/24 12:59
Affects Version/s: 3.2.2

In org.apache.spark.scheduler.DAGScheduler#processShuffleMapStageCompletion
{code:java}
private def processShuffleMapStageCompletion(shuffleStage: ShuffleMapStage): Unit = {
    // some code ... 
    
    if (!shuffleStage.isAvailable) {
      // Some tasks had failed; let's resubmit this shuffleStage.
      // TODO: Lower-level scheduler should also deal with this
      logInfo(log"Resubmitting ${MDC(STAGE, shuffleStage)} " +
        log"(${MDC(STAGE_NAME, shuffleStage.name)}) " +
        log"because some of its tasks had failed: " +
        log"${MDC(PARTITION_IDS, shuffleStage.findMissingPartitions().mkString(", "))}")
      submitStage(shuffleStage) // resubmit without check
    } else {
      markMapStageJobsAsFinished(shuffleStage)
      submitWaitingChildStages(shuffleStage)
    } 
}{code}
The code above shows that the DAGScheduler will resubmit the stage directly without checking if the stage attempt number is greater than maxConsecutiveStageAttempts.  However, resubmitting the stage may still lead to failure or the stage may continually fail, causing an infinite loop.
Bug: Remove the check for the existence of ./dev/free_disk_space_container
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/24 06:31
Affects Version/s: 4.0.0

`./dev/free_disk_space_container` has already been backported to branch-3.4 and branch-3.5 through https://github.com/apache/spark/pull/45624 and https://github.com/apache/spark/pull/43381, so there is no need to check its existence before execution.
Bug: Speed up LogicalPlanIntegrity.validateExprIdUniqueness for large query plans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/24 21:08
Affects Version/s: 3.5.1

When enabling plan validation (spark.sql.planChangeValidation) for large queries, the function LogicalPlanIntegrity.validateExprIdUniqueness is especially slow.
Bug: In CountVectorizer, only read binary parameter once per transform, not once per row
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/24 20:33
Affects Version/s: 2.0.0

SPARK-13629 added a binary parameter to CountVectorizer, but due to the way the code is structured the configuration parameter is read once-per-row in a UDF. Instead, we should read it once-per-transform call (similar to how other parameters are read).
Bug: XML, JSON and CSV default timestamp formatting fails for seconds offset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/24 12:08
Affects Version/s: 3.5.0

Older dates for certain timezones (e.g. America/Los_Angeles) have wierd offsets - such as: -07:52:58.

JSON, XML and CSV default ISO timestamp formatter is using `XXX` for format which omits seconds information, so we end up with wrong result.

example:

sql("SET spark.sql.session.timeZone=America/Los_Angeles")
sql("SELECT to_json(struct(CAST('1800-01-01T00:00:00+00:00' AS TIMESTAMP) AS ts))").show(false)



will display:

{"ts":"1799-12-31T16:07:02.000-07:52"} and hence we will lose 58 seconds
Bug: Deprecate peakExecutionMemory metrics
Status: Open | Priority: Major | Resolution: 
Created: 03/Jul/24 00:13
Affects Version/s: 4.0.0

Currently we have a `peakExecutionMemory` in `TaskMetrics`([code|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala#L109-L114]), as described in the code comments:
{code:java}
/**
 * Peak memory used by internal data structures created during shuffles, aggregations and
 * joins. The value of this accumulator should be approximately the sum of the peak sizes
 * across all such data structures created in this task. For SQL jobs, this only tracks all
 * unsafe operators and ExternalSort.
 */ {code}
The naming is confusing since this does not really reflect the whole execution memory, thus it's barely useful.

 

As a followup for https://issues.apache.org/jira/browse/SPARK-48628 and https://issues.apache.org/jira/browse/SPARK-48788

once we land both, we should have a good replacement about task execution memory metrics, then let's consider deprecate this one, hopefully in Spark 4.0
Bug: INSERT with partial column list to table with char/varchar crashes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jul/24 08:47
Affects Version/s: 3.5.1

```
24/07/03 16:29:01 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: [INTERNAL_ERROR] Unsupported data type VarcharType(64). SQLSTATE: XX000
	at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
	at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport.makeWriter(ParquetWriteSupport.scala:266)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport.$anonfun$init$2(ParquetWriteSupport.scala:111)
	at scala.collection.immutable.List.map(List.scala:247)
	at scala.collection.immutable.List.map(List.scala:79)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport.init(ParquetWriteSupport.scala:111)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:478)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:422)
	at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:411)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:36)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetUtils$$anon$1.newInstance(ParquetUtils.scala:500)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:180)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:165)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:391)
	at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:107)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:896)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:896)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171)
	at org.apache.spark.scheduler.Task.run(Task.scala:146)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:640)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:643)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
```
Bug: Fix SchemaOfJson Expression to work with Collations
Status: Open | Priority: Major | Resolution: 
Created: 05/Jul/24 07:40
Affects Version/s: 4.0.0

It was noticed that SchemaOfJson uses:
{code:java}
private lazy val jsonInferSchema = new JsonInferSchema(jsonOptions){code}
which could potentially pose problem with collations, as JsonInferSchema only uses StringType object at all the places in the code.
Bug: Replace calls to bridged APIs based on SparkSession#sqlContext with SparkSession API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/24 07:10
Affects Version/s: 4.0.0

In the internal code of Spark, there are instances where, despite having a SparkSession instance, the bridged APIs based on SparkSession#sqlContext are still used. So we can makes some simplifications:


1. `SparkSession#sqlContext#read` -> `SparkSession#read`


```scala
/**
   * Returns a [[DataFrameReader]] that can be used to read non-streaming data in as a
   * `DataFrame`.
   * {{{
   *   sqlContext.read.parquet("/path/to/file.parquet")
   *   sqlContext.read.schema(schema).json("/path/to/file.json")
   * }}}
   *
   * @group genericdata
   * @since 1.4.0
   */
  def read: DataFrameReader = sparkSession.read
```

2. `SparkSession#sqlContext#setConf` -> `SparkSession#conf#set`


```scala
  /**
   * Set the given Spark SQL configuration property.
   *
   * @group config
   * @since 1.0.0
   */
  def setConf(key: String, value: String): Unit = {
    sparkSession.conf.set(key, value)
  }
```


3. `SparkSession#sqlContext#getConf` -> `SparkSession#conf#get`

```scala
/**
   * Return the value of Spark SQL configuration property for the given key.
   *
   * @group config
   * @since 1.0.0
   */
  def getConf(key: String): String = {
    sparkSession.conf.get(key)
  }
```

4. `SparkSession#sqlContext#createDataFrame` -> `SparkSession#createDataFrame`

```scala
/**
   * Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).
   *
   * @group dataframes
   * @since 1.3.0
   */
  def createDataFrame[A <: Product : TypeTag](rdd: RDD[A]): DataFrame = {
    sparkSession.createDataFrame(rdd)
  }
```

5. `SparkSession#sqlContext#sessionState` -> `SparkSession#sessionState`

```scala
private[sql] def sessionState: SessionState = sparkSession.sessionState
```

6. `SparkSession#sqlContext#sharedState` -> `SparkSession#sharedState`

```scala
private[sql] def sharedState: SharedState = sparkSession.sharedState
```

7. `SparkSession#sqlContext#streams` -> `SparkSession#streams`


```
/**
   * Returns a `StreamingQueryManager` that allows managing all the
   * [[org.apache.spark.sql.streaming.StreamingQuery StreamingQueries]] active on `this` context.
   *
   * @since 2.0.0
   */
  def streams: StreamingQueryManager = sparkSession.streams
```

8. `SparkSession#sqlContext#uncacheTable` -> ``SparkSession#catalog#uncacheTable`

```
/**
   * Removes the specified table from the in-memory cache.
   * @group cachemgmt
   * @since 1.3.0
   */
  def uncacheTable(tableName: String): Unit = {
    sparkSession.catalog.uncacheTable(tableName)
  }
```
Bug: [SUPPORT] Support setting the maximum number of partitions for a table
Status: Open | Priority: Major | Resolution: 
Created: 04/Jul/24 16:08
Affects Version/s: 4.0.0

Many companies have Hive tables with tens or even hundreds of thousands of partitions. This is because data warehouse developers do not understand the use of partitions. If the number of partitions can be limited, irregular use can be better restricted.

Similar to hive, Alibaba Cloud odps supports some constraints to better ensure cluster stability:
[https://www.alibabacloud.com/help/en/maxcompute/product-overview/limits-4#section-7mi-eex-be0]
Bug: Make errors in NamedParametersSupport generic to handle functions and procedures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/24 17:21
Affects Version/s: 4.0.0

Functions and procedures are routines that support position and named-based arguments. We need to make the logic in {{NamedParametersSupport}} generic (including its error messages) so that it can be reused for procedures.
Bug: Propagate function signature in `DataFrame(...).transform(...)`
Status: Open | Priority: Minor | Resolution: 
Created: 25/Jul/24 21:01
Affects Version/s: 3.5.3

Propagating the function signature for `func` in `DataFrame(...).transform(...)` would enable type checkers like `mypy` to enforce that `func` is being correctly called through `DataFrame(...).transform(...)`.

 

`ParamSpec` seems like a suitable approach for this:
{code:python}
P = ParamSpec("P")

...

def transform(
    self,
    func: Callable[Concatenate["DataFrame", P], "DataFrame"],
    *args: P.args,
    **kwargs: P.kwargs,
) -> "DataFrame":
    ...
{code}
Bug: Insert statement fails when multiple parameters are being used
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jul/24 12:50
Affects Version/s: 3.5.3

{code:java}
// Create table
spark.sql("create table testtab (id int, name string)").show()

// Insert into table using single param - WORKS
spark.sql("insert into identifier(:tab) values(1, 'test1')",
  Map("tab" -> "testtab")).show()

// Select from table using param - WORKS
spark.sql("select * from identifier(:tab)",
  Map("tab" -> "testtab")).show()

// Insert into table using multiple params - FAILS
spark.sql("insert into identifier(:tab) values(2, :name)",
  Map("tab" -> "testtab", "name" -> "test2")).show() {code}