av9ash/GitBugs huggingface dataset:
Bug: Describe Table As JSON
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/24 01:08
Affects Version/s: 4.0.0

Support DESCRIBE TABLE ...  [AS JSON] option to display table metadata in JSON format. 

 

*Context:*

The Spark SQL command DESCRIBE TABLE  displays table metadata in a DataFrame format geared toward human consumption. This format causes parsing challenges, e.g. if fields contain special characters or the format changes as new features are added.

The new AS JSON  option would return the table metadata as a JSON string that supports parsing via machine, while being extensible with a minimized risk of breaking changes. It is not meant to be human-readable.

 

*SQL Ref Spec:*

{ DESC | DESCRIBE } [ TABLE ] [ EXTENDED | FORMATTED ] table_name \{ [ PARTITION clause ] | [ column_name ] } [ AS JSON ] 

 

*JSON Schema:*

```
{
"table_name": "<table_name>",
"catalog_name": [...],
"database_name": [...],
"qualified_name": "<qualified_name>"
"type": "<table_type>",
"provider": "<provider>",
"columns": [

{ "id": 1, "name": "<name>", "type": <type_json>, "comment": "<comment>", "default": "<default_val>" }

],
"partition_values":

{ "<col_name>": "<val>" }

,
"location": "<path>",
"view_definition": "<view_defn>",
"owner": "<owner>",
"comment": "<comment>",
"table_properties":

{ "property1": "<property1>", "property2": "<property2>" }

,
"storage_properties":

{ "property1": "<property1>", "property2": "<property2>" }

,
"serde_library": "<serde_library>",
"inputformat": "<input_format>",
"outputformt": "<output_format>",
"bucket_columns": [<col_name>],
"sort_columns": [<col_name>],
"created_time": "<timestamp>",
"last_access": "<timestamp>",
"partition_provider": "<partition_provider>"
}
```
Bug: DataFrame.toPandas() and DataFrame.mapInPandas() have different data types
Status: Open | Priority: Major | Resolution: 
Created: 04/Dec/24 14:17
Affects Version/s: 3.5.3

When using .toPandas() the ints are list([0, 1]), but when using .mapInPandas() they are numpy.ndarray([0, 1]):
{code:java}
from pandas import DataFrame
from pyspark.sql import SparkSession
from pyspark.sql.types import Row, StructType, StringType
def stringify(data_frame: DataFrame) -> DataFrame:
    return data_frame.map(lambda record: repr((type(record), record)))
spark = SparkSession.Builder().getOrCreate()
spark_data_frame = spark.createDataFrame([Row(ints=[row, row + 1]) for row in range(1)])
print(stringify(spark_data_frame.toPandas()), spark_data_frame.mapInPandas(
    lambda data_frames: (stringify(data_frame) for data_frame in data_frames),
    schema=StructType().add('ints', StringType()),
).toPandas()) {code}
{code:java}
                       ints
0  (<class 'list'>, [0, 1])
                       ints
0  (<class 'numpy.ndarray'>, array([0, 1]))
{code}
I have noticed this while trying to sort the dataframe by this column. Pandas fails to sort such dataframe when using .mapInPandas(), but succeeds when using .toPandas():
{code:java}
from pandas import DataFrame
from pyspark.sql import SparkSession
from pyspark.sql.types import Row, StructType, StringType
def stringify(data_frame: DataFrame) -> DataFrame:
    return data_frame.sort_values('ints').map(lambda record: repr((type(record), record)))
spark = SparkSession.Builder().getOrCreate()
spark_data_frame = spark.createDataFrame([Row(ints=[row, row + 1]) for row in range(10000)])
print(stringify(spark_data_frame.toPandas()), spark_data_frame.mapInPandas(
    lambda data_frames: (stringify(data_frame) for data_frame in data_frames),
    schema=StructType().add('ints', StringType()),
).toPandas()) {code}
{code:java}
  File "/usr/local/lib/python3.12/site-packages/pandas/core/frame.py", line 7200, in sort_values
    indexer = nargsort(
              ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pandas/core/sorting.py", line 439, in nargsort
    indexer = non_nan_idx[non_nans.argsort(kind=kind)]
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() {code}
Bug: [M1] Create new exception for EXECUTE IMMEDIATE
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Dec/24 14:15
Affects Version/s: 4.0.0

Currently, if the SQL Script code is passed to EXECUTE IMMEDIATE, the exception would be something like:
`[UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): CompoundBody are not supported in Unity Catalog. SQLSTATE: 0AKUC`.

Until we make EXECUTE IMMEDIATE working with SQL Scripting, we should create a new, more descriptive, exception for this case.
Bug: Support hint for reading in JDBC data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Dec/24 01:37
Affects Version/s: 4.0.0

We're using DataFrameReader to dump data from an Oracle source, which has a huge table, and want to add hints to target generated SQL.

Many RDBMS supports SQL hint, and it's useful for performance tuning in a data dump case.

Can you help to improve this method?
{code:java}
def jdbc(
    url: String,
    table: String,
    predicates: Array[String],
    connectionProperties: Properties): DataFrame  {code}
Bug: Support Nano Second Timestamp 
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/24 12:38
Affects Version/s: 4.1.0

Iceberg Table V3 will release support for nano second timestamp，spark core/sql only support to micro second for now. Ideally should keep aline with iceberg`s timestamp percision, because  nano second make big difference in quants trading scenorio.
Bug: Improve Spark Shading/Relocation for JVM users
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/24 21:13
Affects Version/s: 3.5.3

Improve the shading of Spark to reduce the class path conflicts & (optionally): setup tests to alert when we accidentally add new classes to the classpath.

 

While long term Spark Connect may reduce the need for these improvements, it is not yet ready for all use cases – especially more complex ones which are more likely to encounter issues with shading and relocation. Even in a post spark connect world, proper shading will make writing complex UDFs depending on external libraries easier.
Bug: Improve auto-generated column aliases for aggregate expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/24 23:49
Affects Version/s: 4.0.0

{{SELECT 1 AS a, 2 AS b}}
{{|> AGGREGATE SUM(a)}}


This prints a table with default title pipeexpression(sum(a)). I would have hoped for just sum(a)  which is what I get with SELECT SUM(a) FROM (SELECT 1 AS a, 2 AS b).
Bug: Refactor Data[Frame|Stream]Reader to leverage the analyzer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/24 00:11
Affects Version/s: 4.0.0

The DataFrameReader and DataStreamReader typically creates analyzed relations as part of their respective `.load()` methods.

This creates inconsistencies for what rules get applied to the query plan as part of Catalyst depending on your API of choice, such as SQL vs Python or SQL vs Scala.

The goal of this Jira is to refactor the logic in DataFrameReader and DataStreamReader classes to create unresolved plans that get analyzed as part of Catalyst.
Bug: support arrow large_list conversion
Status: Open | Priority: Major | Resolution: 
Created: 19/Dec/24 20:34
Affects Version/s: 3.5.3

currently spark will throw UNSUPPORTED_ARROWTYPE exception when reading arrow format file which contains large offset lists

ex:

```

PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] large_list<item: int64> is not supported in conversion to Arrow.

```
Bug: uniform() causes internal error with floating point seed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/24 01:16
Affects Version/s: 4.0.0

SELECT uniform(10::float, -10::double, 1.0::float) FROM VALUES(1), (2), (3), (4), (5), (6), (7);
[[INTERNAL_ERROR|https://docs.databricks.com/error-messages/error-classes.html#internal_error]] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000

Presumable were missing a check to disallow float?

generally it is a question whether we should be more aggressive in casting.
E.g. we do support substr('hello', 1.0, 1.0). 
thsi applies to allowing DECIMAL arguments for the range as well as simply casting the seed to an integral number.
Bug: randstr(-1) causes Java stacktrace and argument types are too restrictive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/24 23:39
Affects Version/s: 4.0.0

randstr(-1) result in a runtime stacktrace:

SparkException: Job aborted due to stage failure: Task 0 in stage 18.0 failed 4 times, most recent failure: Lost task 0.3 in stage 18.0 (TID 111) (10.68.143.147 executor 0): java.lang.NegativeArraySizeException: -1 at org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.randStr(ExpressionImplUtils.java:327) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)

Further, it is unusual not to allow static numeric types to down cast or upcast to the supported  type.
In this case both length and see must be SMALLINT or INT.
TINYINT shouls work for sure 
And BIGINT should work as long as teh value itself is not too large.
In fact compare to substr('hello', 1.0, 1.0) which happily works!
Bug: Set analyzed on analysis failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/24 22:38
Affects Version/s: 4.0.0

As part of SPARK-44145, a callback was added to track completion of analysis and optimization phase of a query. While the analyzed plan is sent when analysis completes successfully it does not when it fail. In that case, we should fallback to the ParsedPlan.

The purpose of the analyze event is to track when analysis completes, as such it should also be sent on both success & failure.
Bug: V1Writes should respect the output ordering
Status: Open | Priority: Major | Resolution: 
Created: 02/Dec/24 10:54
Affects Version/s: 3.5.3

In V1Writes rule, it would add a new local sort if the outputOrdering is not matched with requiredOrdering. However it only add the requiredOrdering in the new local sort, and the `EliminateSorts` rule would remove the underlying outputOrdering, which cause the final written dada miss the user's query order.
Bug: Fix discrepancy in DESCRIBE TABLE columns list quoting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/24 14:15
Affects Version/s: 4.0.0

Previously, the `DESCRIBE TABLE` SQL command displayed column lists such as bucket and sort columns as so:

Bucket Columns          [`a`]                                       
Sort Columns            [`b`] 

 

However view query output columns were not quoted:



View Query Output Columns      [c1] 

 

This issue fixes the discrepancy.
Bug: Delete Artifact resources on GC of instance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Dec/24 11:55
Affects Version/s: 4.0.0

Currently, resources of an instance of ArtifactManager aren't cleaned up automatically during it's GC. Previously, artifact manager was only used in Spark Connect where it's lifetime was controlled by the Spark Connect Session.

With the recent changes allowing the artifact manager to be used in 'Classic Spark', we should GC related resources when the SparkSession gets GCed (and thus, the ArtifactManager instance is no longer reachable)
Bug: JDBC Reader Fails to Handle Complex Types (Array, Map) from Trino
Status: Open | Priority: Major | Resolution: 
Created: 24/Dec/24 07:57
Affects Version/s: 3.5.4

{*}Description{*}:

I am encountering an issue when using Spark to read data from a Trino instance via JDBC. Specifically, when querying complex types such as {{ARRAY}} or {{MAP}} from Trino, Spark throws an error indicating that it cannot recognize these SQL types. Below is the context:

{*}Code Example{*}:

 
{code:java}
val sourceDF = spark.read
  .format("jdbc")
  .option("driver", "io.trino.jdbc.TrinoDriver")
  .option("url", "jdbc:trino://localhost:8181")
  .option("query", "select address from minio.qa.nbcheck1")
  .load(){code}
 

 

*Error Message:*

 
{code:java}
2/04 03:49:59 INFO SparkContext: SparkContext already stopped.
Exception in thread "main" org.apache.spark.SparkSQLException: [UNRECOGNIZED_SQL_TYPE] Unrecognized SQL type - name: array (row(city varchar, state varchar)), id: ARRAY.
at org.apache.spark.sql.errors.QueryExecutionErrors$.unrecognizedSqlTypeError(QueryExecutionErrors.scala:992){code}
 

 

{*}Root Cause{*}:

The error seems to be occurring because Spark's JDBC data source does not recognize complex SQL types like {{ARRAY}} or {{MAP}} from Trino by default. This is confirmed by the following relevant section of Spark's code:

[https://github.com/apache/spark/blob/v3.5.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala]
{code:java}
private def getCatalystType(
  sqlType: Int,
  typeName: String,
  precision: Int,
  scale: Int,
  signed: Boolean,
  isTimestampNTZ: Boolean): DataType = sqlType match {
    ...
    case _ =>
      // For unmatched types:
      // including java.sql.Types.ARRAY, DATALINK, DISTINCT, JAVA_OBJECT, NULL, OTHER, REF_CURSOR,
      // TIME_WITH_TIMEZONE, TIMESTAMP_WITH_TIMEZONE, and others.
      val jdbcType = classOf[JDBCType].getEnumConstants()
        .find(_.getVendorTypeNumber == sqlType)
        .map(_.getName)
        .getOrElse(sqlType.toString)
      throw QueryExecutionErrors.unrecognizedSqlTypeError(jdbcType, typeName){code}
As you can see, the method for translating JDBC types to Spark Catalyst types doesn't currently handle ARRAY or MAP, among other types, leading to the error. The JDBC schema translation fails when complex types such as ARRAY or MAP are present.

{*}Expected Behavior{*}:

Spark should not fail when encountering complex types like {{ARRAY}} or {{MAP}} from a Trino JDBC source. Instead, it should either:
 # Convert these complex types into a serialized string format (e.g., JSON) over the wire.
 # Provide an option for users to manually handle such complex types after loading them into a DataFrame.
Bug: Refactor CreateUserDefinedFunction command to extend from UnaryRunnableCommand
Status: Open | Priority: Major | Resolution: 
Created: 30/Dec/24 06:23
Affects Version/s: 4.0.0

Refactor CreateUserDefinedFunction command to extend from UnaryRunnableCommand, similar to UnsetNamespacePropertiesCommand so that we only need one logical plan for the command.

See comment: [https://github.com/apache/spark/pull/49126/files#r1890214852]

Another improvement: We don't hold the expr string in the command, but the parsed expression ({{{}ColumnDefinition#defaultValue{}}}). Then we validate the resolved expression in {{{}ResolveDefaultColumns.validateDefaultValueExpr ({}}}similar to how we validate column default values for CREATE TABLE. ){{{}{}}}

See comment: https://github.com/apache/spark/pull/49126#discussion_r1900525129
Bug: Got UNSUPPORTED_INSERT.RDD_BASED error when insert into a jdbc datasource
Status: Open | Priority: Major | Resolution: 
Created: 25/Dec/24 09:49
Affects Version/s: 3.5.4

I have some SQL scripts for pushing the calculation results of the data warehouse to the JDBC DATASOURCE.

When I attempted to upgrade from Spark 3.3.3 to Spark 3.5.4, I found that one of them failed to run properly: it pushed data to the RDS database of Alibaba Cloud, but other similar scripts worked without any issues. 

This script can run normally on both Spark 3.3.3 and 3.4.4.
 
SCRIPT:
{code:java}
CREATE TEMPORARY VIEW sg_apdata_v_prd
USING org.apache.spark.sql.jdbc
OPTIONS (
url "jdbc:mysql://rm-abc.mysql.rds.aliyuncs.com:3306/db_01?useUnicode=true&characterEncoding=UTF-8&useSSL=false",
dbtable "`apdata`",
user 'sgroot',
password '####'
);{code}
 
{code:java}
desc sg_apdata_v_prd;   

user_name string 
region string 
ap_name string 
dtime string 
ddate string{code}
 
{code:java}
explain
INSERT INTO sg_apdata_v_prd
select 'name' as user_name, 'region' as region, 'ap_name' as ap_name, '2024-12-24 23:45:00' as dtime, '2024-12-24' as ddate;{code}
 
{code:java}
== Physical Plan ==                                                                                                                                                                                                                        org.apache.spark.sql.AnalysisException: [UNSUPPORTED_INSERT.RDD_BASED] Can't insert into the target. An RDD-based table is not allowed.;                                                                                                   'InsertIntoStatement Project [user_name#0, region#1, ap_name#2, dtime#3, staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, ddate#4, 10, true, false, true) AS ddate#5], false, false, false                                                                                                                                                                                                                                 +- Project [name AS user_name#67, region AS region#68, ap_name AS ap_name#69, 2024-12-24 23:45:00 AS dtime#70, 2024-12-24 AS ddate#71]                                                                                                        +- OneRowRelation   {code}
 

Spark 3.4.4 Physical Plan is : 
{code:java}
== Physical Plan == Execute InsertIntoDataSourceCommand +- InsertIntoDataSourceCommand Relation [user_name#0,region#1,ap_name#2,dtime#3,ddate#4] JDBCRelation(`sg-apdata`) [numPartitions=1], false +- Project [cast(user_name#10 as string) AS user_name#21, cast(region#11 as string) AS region#22, cast(ap_name#12 as string) AS ap_name#23, cast(dtime#13 as string) AS dtime#24, cast(ddate#14 as string) AS ddate#25] +- Project [name AS user_name#10, region AS region#11, ap_name AS ap_name#12, 2024-12-24 23:45:00 AS dtime#13, 2024-12-24 AS ddate#14] +- OneRowRelation{code}
Bug: Extend CTESubstitution.scala to make it aware of recursion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/24 10:56
Affects Version/s: 4.0.0

Self-contained changes to CTESubstitution.scala that make CTE substitutions and resolutions aware of the recursion. [https://docs.google.com/document/d/1qcEJxqoXcr5cSt6HgIQjWQSqhfkSaVYkoDHsg5oxXp4/edit]
Bug: Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/24 04:07
Affects Version/s: 3.5.0

In spark connect, when there is no initial state, we derived the state schema from the input:
 * create the initialDs from the original input: [https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L679-L689]
 * derived the state expression encoder from this `initialDs` which is incorrect: [https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L729]

 

Our unit tests fail to cover this case because it doesn't do the state update: [https://github.com/apache/spark/blob/master/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateStreamingSuite.scala#L55-L59]

after changing the `stateFunc` to the following
{code:java}
val stateFunc =
  (key: String, values: Iterator[ClickEvent], state: GroupState[ClickState]) => {
    if (state.exists) throw new IllegalArgumentException("state.exists should be false")
    val newState = ClickState(key, values.size)
    state.update(newState)
    Iterator(newState)
  } {code}
the test is actually failing with
{code:java}
Cause: org.apache.spark.SparkException: Job aborted due to stage failure: Task 122 in stage 2.0 failed 1 times, most recent failure: Lost task 122.0 in stage 2.0 (TID 12) (192.168.68.84 executor driver): java.lang.ClassCastException: class org.apache.spark.sql.streaming.ClickState cannot be cast to class org.apache.spark.sql.streaming.ClickEvent (org.apache.spark.sql.streaming.ClickState and org.apache.spark.sql.streaming.ClickEvent are in unnamed module of loader 'app')
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.CreateNamedStruct_0$(Unknown Source)
 {code}
Bug: when the job is cancelled during shuffle retry in parent stage, might leave behind zombie tasks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/24 10:56
Affects Version/s: 2.1.0

Assume a job is stage1-> stage2, when FetchFailed occurs during the stage 2, the stage1 and stage2  will resubmit (stage2 may still have some tasks running even if stage2 is resubmitted , this is as expected, the reason see the comment from irashid https://issues.apache.org/jira/browse/SPARK-2666 )

But during the execution of the stage1-retry , if the SQL is canceled, the tasks in stage1 and stage1-retry can all be killed, but the tasks previously running in stage2 are still running and can't be killed. These tasks can greatly affect cluster stability and occupy resources.
Bug: Use UnsafeProjection for LocalRelation rows instead of ComparableLocalRelation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/24 15:18
Affects Version/s: 4.0.0

`UnsafeRow.equals` compares the whole byte sequence under it, so it's a convenient way to compare all kind of row values, including `ArrayBasedMapData`.
Bug: Scala Streaming foreachBatch doesn't work for input type Dataset[T] due to missing encoder for T.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Dec/24 13:13
Affects Version/s: 3.5.4

Currently, spark connect implementation of Scala streaming does not take the encoder for T when the input is Dataset[T]. This will lead to a runtime exception when the function passed is `(Dataset[T], Long) => Unit`, while it is executed as `(Dataframe, Long) => Unit`.

 

We can reproduce the failure with the following test:

 

```

import org.apache.spark.sql._
import org.apache.spark.sql.streaming._

val inputPath = "/tmp/input"
case class C(id: Int, name: String, age: Int)
val data = Seq((1, "Alice", 29), (2, "Bob", 35), (3, "Cathy", 26), (4, "David", 40), (5, "Eve", 30)).toDF("id", "name", "age")
data.write.format("csv").mode("overwrite").save(inputPath)
val df = spark.readStream.format("csv").schema("id INT, name String, age INT").load("/tmp/input")
val query = df.writeStream.foreachBatch \{ (batchDF: DataFrame, batchId: Long) => { print(batchDF.collectAsList()) } }.trigger(Trigger.ProcessingTime("1 seconds")).start()
Thread.sleep(2000)
query.stop()

```

 

The fix should address this by passing the encoder for T together with the function definition (as other Scala UDFs), and then restore the encoder in SparkConnectPlanner.
Bug: Memory footprint reduction in the fallback from hash aggregation to the sort
Status: Open | Priority: Major | Resolution: 
Created: 27/Dec/24 07:30
Affects Version/s: 3.5.4

Currently, under the memory pressure, spark uses fallback from a hash-based aggregation to the sort-based one. The logic reuses internal hash map components for sorting to avoid unnecessary allocation. Yet, there is still [a spot|https://github.com/apache/spark/blob/b309db092cbb67fedb7cec9ab0ebcd07d8d95b04/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java#L118] when a new allocation is initiated. Basically, every time we have duplicates in key aggregates we can't reuse the internal array and have to allocate a new one. The natural approach here would be invalidation of the redundant old array before a new allocation. That would reduce the overall memory footprint and decrease the chance of the oom error and subsequent task failure.