av9ash/GitBugs huggingface dataset:
Bug: Add applyInArrow to pyspark.sql.GroupedData
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Sep/22 07:22
Affects Version/s: 3.4.0

PySpark allows to transform a {{DataFrame}} via Pandas and Arrow API:

{code:python}
def map_arrow(iter: Iterator[pyarrow.RecordBatch]) -> Iterator[pyarrow.RecordBatch]:
    return iter

def map_pandas(iter: Iterator[pandas.DataFrame]) -> Iterator[pandas.DataFrame]:
    return iter

df.mapInArrow(map_arrow, schema="...")
df.mapInPandas(map_pandas, schema="...")
{code}

A grouped {{DataFrame}} currently supports only the Pandas API:

{code:python}
def apply_pandas(df: pandas.DataFrame) -> pandas.DataFrame:
    return df

df.groupBy("id").applyInPandas(apply_pandas, schema="...")
{code}

A similar method for the Arrow API would useful, especially given that Arrow is used by many other libraries.

An Arrow-based method allows to process the {{DataFrame}} with *any* Arrow-based API, e.g. Polars:

{code:python}
def apply_polars(df: polars.DataFrame) -> polars.DataFrame:
  return df

def apply_arrow(iter: Iterator[pyarrow.RecordBatch]) -> Iterator[pyarrow.RecordBatch]:
  for batch in iter:
    df = polars.from_arrow(pyarrow.Table.from_batches([batch]))
    for b in apply_polars(df).to_arrow().to_batches():
      yield b

df.groupBy("id").applyInArrow(apply_arrow, schema="...")
{code}

https://stackoverflow.com/questions/71606278/is-there-an-apache-arrow-equivalent-of-the-spark-pandas-udf
https://stackoverflow.com/questions/73203318/how-to-transform-spark-dataframe-to-polars-dataframe
Bug: Upgrade Scala to 2.13.11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/22 07:45
Affects Version/s: 4.0.0

We tested and decided to skip the following releases. This issue aims to use 2.13.11.
- 2022-09-21: v2.13.9 released [https://github.com/scala/scala/releases/tag/v2.13.9]
- 2022-10-13: 2.13.10 released [https://github.com/scala/scala/releases/tag/v2.13.10]
 

Scala 2.13.11 Milestone
- https://github.com/scala/scala/milestone/100
Bug: Decouple plan transformation and validation on server side 
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/22 19:53
Affects Version/s: 3.4.0

Project connect, from some perspectives, can be thought as replacing the SQL parser to generate a parsed (but the difference that is unresolved) plan, then the plan is passed to the analyzer. This means that connect should also do validation on the proto as there are many in-validate parser cases that analyzer does not expect to see, which potentially could cause problems if connect only pass through the proto (of course have it translated) to analyzer.


Meanwhile I think this is a good idea to decouple the validation and transformation so that we have two stages:
stage 1: proto validation. For example validate if necessary fields are populated or not.
stage 2: transformation, which convert the proto to a plan with assumption that the plan is valid parsed version of the plan.
Bug: Introduce Arrow Python UDFs
Status: Resolved | Priority: Major | Resolution: Done
Created: 01/Sep/22 21:32
Affects Version/s: 3.5.0

Python user-defined function (UDF) enables users to run arbitrary code against PySpark columns. It uses Pickle for (de)serialization and executes row by row.

One major performance bottleneck of Python UDFs is (de)serialization, that is, the data interchanging between the worker JVM and the spawned Python subprocess which actually executes the UDF. We should seek an alternative to handle the (de)serialization: Arrow, which is used in the (de)serialization of Pandas UDF already.


There should be two ways to enable/disable the Arrow optimization for Python UDFs:

- the Spark configuration `spark.sql.execution.pythonUDF.arrow.enabled`, disabled by default.
- the `useArrow` parameter of the `udf` function, None by default.

The Spark configuration takes effect only when `useArrow` is None. Otherwise, `useArrow` decides whether a specific user-defined function is optimized by Arrow or not.

The reason why we introduce these two ways is to provide both a convenient, per-Spark-session control and a finer-grained, per-UDF control of the Arrow optimization for Python UDFs.
Bug: Sorting issue with partitioned-writing and AQE turned on
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/22 22:19
Affects Version/s: 3.1.3

We are attempting to partition data by a few columns, sort by a particular _sortCol_ and write out one file per partition. 
{code:java}
df
    .repartition(col("day"), col("month"), col("year"))
    .withColumn("partitionId",spark_partition_id)
    .withColumn("monotonicallyIncreasingIdUnsorted",monotonicallyIncreasingId)
    .sortWithinPartitions("year", "month", "day", "sortCol")
    .withColumn("monotonicallyIncreasingIdSorted",monotonicallyIncreasingId)
    .write
    .partitionBy("year", "month", "day")
    .parquet(path){code}
When inspecting the results, we observe one file per partition, however we see an _alternating_ pattern of unsorted rows in some files.
{code:java}
{"sortCol":100000,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287832121344,"monotonicallyIncreasingIdSorted":6287832121344}
{"sortCol":1303413,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287877022389,"monotonicallyIncreasingIdSorted":6287876860586}
{"sortCol":100000,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287877567881,"monotonicallyIncreasingIdSorted":6287832121345}
{"sortCol":1303413,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287835105553,"monotonicallyIncreasingIdSorted":6287876860587}
{"sortCol":100000,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287832570127,"monotonicallyIncreasingIdSorted":6287832121346}
{"sortCol":1303413,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287879965760,"monotonicallyIncreasingIdSorted":6287876860588}
{"sortCol":100000,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287878762347,"monotonicallyIncreasingIdSorted":6287832121347}
{"sortCol":1303413,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287837165012,"monotonicallyIncreasingIdSorted":6287876860589}
{"sortCol":100000,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287832910545,"monotonicallyIncreasingIdSorted":6287832121348}
{"sortCol":1303413,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287881244758,"monotonicallyIncreasingIdSorted":6287876860590}
{"sortCol":100000,"partitionId":732,"monotonicallyIncreasingIdUnsorted":6287880041345,"monotonicallyIncreasingIdSorted":6287832121349}{code}
Here is a [gist|https://gist.github.com/Swebask/543030748a768be92d3c0ae343d2ae89] to reproduce the issue. 

Turning off AQE with spark.conf.set("spark.sql.adaptive.enabled", false) fixes the issue.

I'm working on identifying why AQE affects the sort order. Any leads or thoughts would be appreciated!
Bug: Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/22 19:42
Affects Version/s: 3.2.0

Spurious failures because of the assert:

{noformat}
22/09/29 09:46:24 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 3063.0 in stage 1997.0 (TID 677249),5,main]
java.lang.AssertionError: assertion failed: task count shouldn't below 0
	at scala.Predef$.assert(Predef.scala:223)
	at org.apache.spark.executor.ExecutorMetricsPoller.decrementCount$1(ExecutorMetricsPoller.scala:130)
	at org.apache.spark.executor.ExecutorMetricsPoller.$anonfun$onTaskCompletion$3(ExecutorMetricsPoller.scala:135)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1822)
	at org.apache.spark.executor.ExecutorMetricsPoller.onTaskCompletion(ExecutorMetricsPoller.scala:135)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:737)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
22/09/29 09:46:24 INFO MemoryStore: MemoryStore cleared
22/09/29 09:46:24 INFO BlockManager: BlockManager stopped
22/09/29 09:46:24 INFO ShutdownHookManager: Shutdown hook called
22/09/29 09:46:24 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1664443624160_0001/spark-93efc2d4-84de-494b-a3b7-2cb1c3a45426
{noformat}
Bug: Spark fall back to use getPartitions instead of getPartitionsByFilter when date_add functions used in where clause 
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 29/Sep/22 12:18
Affects Version/s: 3.2.1

When I run a insert overwrite statement, I got error saying:

 
{code:java}
MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. listPartitions {code}
 

It's weird as I only selected for about 3 partitions, so I rerun the sql and checked the metastore, then I found it's fetching all columns in all partitions:

 
{code:java}
select "CD_ID", "COMMENT", "COLUMN_NAME", "TYPE_NAME" from "COLUMNS_V2" where "CD_ID" 
in (675384,675393,675385,675394,675396,675397,675395,675398,675399,675401,675402,675400,675406……){code}
 

 

After testing, I found the problem is with the date_add function in where clause, if remove it ,sql works fine, else metastore would fetch all columns in all partitions.

 

 
{code:java}
insert overwrite table test.tmp_test_metastore_usage
SELECT userid
    ,SUBSTR(sendtime,1,10) AS creation_date
    ,cast(json_bh_esdate_deltadays_max as DECIMAL(38,2)) AS bh_esdate_deltadays_max
    ,json_bh_qiye_industryphyname AS bh_qiye_industryphyname
    ,cast(json_bh_esdate_deltadays_min as DECIMAL(38,2)) AS bh_esdate_deltadays_min
    ,cast(json_bh_subconam_min as DECIMAL(38,2)) AS bh_subconam_min
    ,cast(json_bh_qiye_regcap_min as DECIMAL(38,2)) AS bh_qiye_regcap_min
    ,json_bh_industryphyname AS bh_industryphyname
    ,cast(json_bh_subconam_mean as DECIMAL(38,2)) AS bh_subconam_mean
    ,cast(json_bh_industryphyname_nunique as DECIMAL(38,2)) AS bh_industryphyname_nunique
    ,cast(current_timestamp() as string) as dw_cre_date
    ,cast(current_timestamp() as string) as dw_upd_date
FROM (
    SELECT userid
        ,sendtime
        ,json_bh_esdate_deltadays_max
        ,json_bh_qiye_industryphyname
        ,json_bh_esdate_deltadays_min
        ,json_bh_subconam_min
        ,json_bh_qiye_regcap_min
        ,json_bh_industryphyname
        ,json_bh_subconam_mean
        ,json_bh_industryphyname_nunique
        ,row_number() OVER (
            PARTITION BY userid,dt ORDER BY sendtime DESC
            ) rn
    FROM edw.tmp_test_metastore_usage_source
    WHERE dt >= date_add('2022-09-22',-3 )
        AND json_bizid IN ('6101')
        AND json_dingid IN ('611')
    ) t
WHERE rn = 1 {code}
 

 By the way 2.4.7 works good.
Bug: Implement `GroupBy.quantile`.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/22 04:48
Affects Version/s: 3.4.0

We should implement `GroupBy.quantile` for increasing pandas API coverage.

pandas docs: https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.quantile.html
Bug: Implement `Expanding.quantile`.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/22 04:56
Affects Version/s: 3.4.0

We should implement `Expanding.quantile` for increasing pandas API coverage.

pandas docs: https://pandas.pydata.org/docs/reference/api/pandas.core.window.expanding.Expanding.quantile.html
Bug: Floating-point value with an INT/BYTE/SHORT/LONG type errors out in DataFrame but evaluates to a rounded value in SparkSQL
Status: Open | Priority: Major | Resolution: 
Created: 22/Sep/22 01:40
Affects Version/s: 3.2.1

h3. Describe the bug

Storing an invalid INT value {{1.1}} using DataFrames via {{spark-shell}} expectedly errors out. However, it is evaluated to a rounded value {{1}} if the value is inserted into the table via {{{}spark-sql{}}}.
h3. Steps to reproduce:

On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-sql{}}}:
{code:java}
$SPARK_HOME/bin/spark-sql {code}
Execute the following:
{code:java}
spark-sql> create table int_floating_point_vals(c1 INT) stored as ORC;
22/09/19 16:49:11 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
Time taken: 0.216 seconds
spark-sql> insert into int_floating_point_vals select 1.1;
Time taken: 1.747 seconds
spark-sql> select * from int_floating_point_vals;
1
Time taken: 0.518 seconds, Fetched 1 row(s){code}
h3. Expected behavior

We expect the two Spark interfaces ({{{}spark-sql{}}} & {{{}spark-shell{}}}) to behave consistently for the same data type & input combination ({{{}INT{}}} and {{{}1.1{}}}).
h4. Here is a simplified example in {{{}spark-shell{}}}, where insertion of the aforementioned value correctly raises an exception:

On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-shell{}}}:
{code:java}
$SPARK_HOME/bin/spark-shell{code}
Execute the following:
{code:java}
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.types._
val rdd = sc.parallelize(Seq(Row(1.1)))
val schema = new StructType().add(StructField("c1", IntegerType, true))
val df = spark.createDataFrame(rdd, schema)
df.write.mode("overwrite").format("orc").saveAsTable("int_floating_point_vals") {code}
The following exception is raised:
{code:java}
java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.lang.Double is not a valid external type for schema of int{code}
Bug: Remove push-based shuffle data after query finished
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/22 14:20
Affects Version/s: 3.3.0

Now spark will only cleanup shuffle data files except push-based shuffle files.
In our production cluster, push-based shuffle service will create too many shuffle merge data files as there are several spark thrift server.
Could we cleanup the merged data files after the query finished?
Bug: The performance will be worse after codegen
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/22 09:04
Affects Version/s: 3.4.0


{code:scala}
import org.apache.spark.benchmark.Benchmark

val dir = "/tmp/spark/benchmark"
val N = 2000000
val columns = Range(0, 100).map(i => s"id % $i AS id$i")

spark.range(N).selectExpr(columns: _*).write.mode("Overwrite").parquet(dir)

// Seq(1, 2, 5, 10, 15, 25, 40, 60, 100)
Seq(60).foreach{ cnt =>
  val selectExps = columns.take(cnt).map(_.split(" ").last).map(c => s"count(distinct $c)")

  val benchmark = new Benchmark("Benchmark count distinct", N, minNumIters = 1)
  benchmark.addCase(s"$cnt count distinct with codegen") { _ =>
    withSQLConf(
      "spark.sql.codegen.wholeStage" -> "true",
      "spark.sql.codegen.factoryMode" -> "FALLBACK") {
      spark.read.parquet(dir).selectExpr(selectExps: _*).write.format("noop").mode("Overwrite").save()
    }
  }

  benchmark.addCase(s"$cnt count distinct without codegen") { _ =>
    withSQLConf(
      "spark.sql.codegen.wholeStage" -> "false",
      "spark.sql.codegen.factoryMode" -> "NO_CODEGEN") {
      spark.read.parquet(dir).selectExpr(selectExps: _*).write.format("noop").mode("Overwrite").save()
    }
  }
  benchmark.run()
}
{code}


{noformat}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark count distinct:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
60 count distinct with codegen                   628146         628146           0          0.0      314072.8       1.0X
60 count distinct without codegen                147635         147635           0          0.0       73817.5       4.3X
{noformat}
Bug: Populate ExecutorDecommission with more informative messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/22 10:45
Affects Version/s: 3.4.0

Currently the message in {{ExecutorDecommission}} is a fixed value {{{}"Executor decommission."{}}}, and it is the same for all cases, including spot instance interruptions and auto-scaling down. We should put a detailed message in {{ExecutorDecommission}} to better differentiate those cases.
Bug: recoverDiskStore should not stop by existing recomputed files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 00:20
Affects Version/s: 3.2.3

{code:java}
org.apache.commons.io.FileExistsException: File element in parameter 'null' already exists: '...'
	at org.apache.commons.io.FileUtils.requireAbsent(FileUtils.java:2587)
	at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2305)
	at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2283)
	at org.apache.spark.storage.DiskStore.moveFileToBlock(DiskStore.scala:150)
	at org.apache.spark.storage.BlockManager$TempFileBasedBlockStoreUpdater.saveToDiskStore(BlockManager.scala:487)
	at org.apache.spark.storage.BlockManager$BlockStoreUpdater.$anonfun$save$1(BlockManager.scala:407)
	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1445)
	at org.apache.spark.storage.BlockManager$BlockStoreUpdater.save(BlockManager.scala:380)
	at org.apache.spark.storage.BlockManager$TempFileBasedBlockStoreUpdater.save(BlockManager.scala:490)
	at org.apache.spark.shuffle.KubernetesLocalDiskShuffleExecutorComponents$.$anonfun$recoverDiskStore$14(KubernetesLocalDiskShuffleExecutorComponents.scala:95)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at org.apache.spark.shuffle.KubernetesLocalDiskShuffleExecutorComponents$.recoverDiskStore(KubernetesLocalDiskShuffleExecutorComponents.scala:91) {code}
Bug: A DECIMAL value with division by 0 errors in DataFrame but evaluates to NULL in SparkSQL
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Sep/22 23:03
Affects Version/s: 3.2.1

h3. Describe the bug

Storing an invalid value (e.g. {{{}BigDecimal("1.0/0"){}}}) via {{spark-shell}} errors out during RDD creation. However, {{1.0/0}} evaluates to {{NULL}} if the value is inserted into a {{DECIMAL(20,10)}} column of a table via {{{}spark-sql{}}}.
h3. To Reproduce

On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-sql{}}}:
{code:java}
$SPARK_HOME/bin/spark-sql{code}
Execute the following: (evaluated to {{{}NULL{}}})
{code:java}
spark-sql> create table decimal_vals(c1 DECIMAL(20,10)) stored as ORC;
spark-sql> insert into decimal_vals select 1.0/0;
spark-sql> select * from decimal_vals;
NULL{code}
Using {{{}spark-shell{}}}:
{code:java}
$SPARK_HOME/bin/spark-shell{code}
Execute the following: (errors out during RDD creation)
{code:java}
scala> import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.{Row, SparkSession}
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val rdd = sc.parallelize(Seq(Row(BigDecimal("1.0/0"))))
java.lang.NumberFormatException
  at java.math.BigDecimal.<init>(BigDecimal.java:497)
  at java.math.BigDecimal.<init>(BigDecimal.java:383)
  at java.math.BigDecimal.<init>(BigDecimal.java:809)
  at scala.math.BigDecimal$.exact(BigDecimal.scala:126)
  at scala.math.BigDecimal$.apply(BigDecimal.scala:284)
  ... 49 elided{code}
h3. Expected behavior

We expect the two Spark interfaces ({{{}spark-sql{}}} & {{{}spark-shell{}}}) to behave consistently for the same data type & input combination ({{{}BigDecimal{}}}/{{{}DECIMAL(20,10){}}} and {{{}1.0/0{}}}).
Bug: Update the document for DataFrame.to_orc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/22 07:47
Affects Version/s: 3.4.0

From pandas 1.5.0. `pandas.DataFrame.to_orc` is supported.

Pandas API on Spark already support this feature, but the behavior is a bit different from pandas, so we should update the documents.
Bug: Incorrect Count when reading CSV file
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/22 17:07
Affects Version/s: 3.1.2

I'm trying to read the below data from a CSV file and end up with a wrong count, although the dataframe contains all the records below. df_inputfile.count() prints 3 although it should have been 4.


{code:java}
B1123451020-502,"","{""m"": {""difference"": 60}}","","","",2022-02-12T15:40:00.783Z
B1456741975-266,"","{""m"": {""difference"": 60}}","","","",2022-02-04T17:03:59.566Z
B1789753479-460,"","",",","","",2022-02-18T14:46:57.332Z
B1456741977-123,"","{""m"": {""difference"": 60}}","","","",2022-02-04T17:03:59.566Z {code}
Here's the code:
{code:java}
df_inputfile = (spark.read.format("com.databricks.spark.csv")
                                     .option("inferSchema", "true")
                                     .option("header","false")                
                                     .option("quotedstring",'\"')
                                     .option("escape",'\"')
                                     .option("multiline","true")
                                     .option("delimiter",",")
                                     .load('<path to csv>'))

print(df_inputfile.count()) # Prints 3
print(df_inputfile.distinct().count()) # Prints 4 {code}
Adding a cache() statement before the count results in correct output. Removing the option 'escape' also results in a correct count. 
{noformat}
option("escape",'\"'){noformat}
It looks like this is happening because of the single comma in the 4th column of the 3rd row. Can someone please explain what's going on?
Bug: Error at where clause, when sql case executes by else branch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Sep/22 09:55
Affects Version/s: 3.3.0

Hello!

The Spark SQL phase optimization failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
 - Spark verison 3.3.0
 - Scala version 2.12
 - DatasourceV2
 - Postgres
 - Postrgres JDBC Driver: 42+
 - Java8

Case:

select
    case
        when (t_name = 'foo') then 'foo'
        else 'default'
    end as case_when
from
    t
where
    case
        when (t_name = 'foo') then 'foo'
        else 'default'
    end *= 'foo';  -> works as expected*

*----------------------------------------------------------*

select
    case
        when (t_name = 'foo') then 'foo'
        else 'default'
    end as case_when
from
    t
where
    case
        when (t_name = 'foo') then 'foo'
        else 'default'
    end *= 'default'; -> query throw ex;*

In where clause when we try find rows by else branch, spark thrown exception:
The Spark SQL phase optimization failed with an internal error. Please, fill a bug report in, and provide the full stack trace.

Caused by: java.lang.AssertionError: assertion failed
    at scala.Predef$.assert(Predef.scala:208)

 org.apache.spark.sql.execution.datasources.v2.PushablePredicate.$anonfun$unapply$1(DataSourceV2Strategy.scala:589)

At debugger def unapply in PushablePredicate.class
when sql case return 'foo' -> function unapply accept: (t_name = 'foo'), as instance of Predicate
when sql case return 'default' -> function unapply accept: COALESCE(t_name = 'foo', FALSE) as instance of GeneralScalarExpression and assertation failed with error
Bug: Initial prototype implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/22 09:25
Affects Version/s: 3.2.2

In [https://github.com/apache/spark/pull/37710] we created a prototype that shows the end to end integration of Spark Connect with the rest of the system.

 

Since the PR is quite large, we will track follow up items as children of SPARK-39375