av9ash/GitBugs huggingface dataset:
Bug: Add sorted column in BucketTransform
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Dec/21 07:44
Affects Version/s: 3.3.0

In V1, we can create table with sorted bucket like the following:

{code:java}
      sql("CREATE TABLE tbl(a INT, b INT) USING parquet " +
        "CLUSTERED BY (a) SORTED BY (b) INTO 5 BUCKETS")
{code}

However, creating table with sorted bucket in V2 failed with Exception

{code:java}
org.apache.spark.sql.AnalysisException: Cannot convert bucketing with sort columns to a transform.
{code}

We should be able to create table with sorted bucket in V2.
Bug: Unwrap cast should skip if downcast failed with ansi enabled
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Dec/21 00:41
Affects Version/s: 3.1.2

Currently, unwrap cast throws ArithmeticException if down cast failed with ansi enabled. Since UnwrapCastInBinaryComparison is an optimizer rule, we should always skip on failure regardless of ansi config.
Bug: The sampling job will lead to the wrong statistics
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Dec/21 09:18
Affects Version/s: 3.2.0

code for reproduce
{code:java}
    spark.range(0, 10)
      .repartitionByRange(10, col("id"))
      .show() {code}
!截屏2021-12-07 下午5.17.12.png!
Bug: Bump dev.ludovic.netlib to 2.2.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Dec/21 09:19
Affects Version/s: 3.2.0

Bump the version of dev.ludovic.netlib from 2.2.0 to 2.2.1. This fixes a computation bug in sgemm. See [https://github.com/luhenry/netlib/issues/7|the issue[]|http://example.com]the] and [https://github.com/luhenry/netlib/compare/v2.2.0...v2.2.1|the diff]
Bug: Support the configuration of the initial number of scan partitions when executing a take on a query
Status: Resolved | Priority: Trivial | Resolution: Duplicate
Created: 10/Dec/21 04:54
Affects Version/s: 3.2.0

Now the initial number of scanned partitions is 1 by default when executing a take on a query.
This number does not support configuration.
Sometimes the first task runs slower. If we have this configuration, we can increase the initial parallelism.
Bug: sql-migration-guide wrong description
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Dec/21 08:47
Affects Version/s: 3.2.0

Description in the documentation:
{code:java}
In Spark 3.2, the unit list interval literals can not mix year-month fields (YEAR and MONTH) and day-time fields (WEEK, DAY, …, MICROSECOND). For example, INTERVAL 1 day 1 hour is invalid in Spark 3.2. In Spark 3.1 and earlier, there is no such limitation and the literal returns value of CalendarIntervalType. To restore the behavior before Spark 3.2, you can set spark.sql.legacy.interval.enabled to true. {code}
”INTERVAL 1 day 1 hour is invalid in Spark 3.2.“ 

Is this example correct? According to the description of DayTimeIntervalType, INTERVAL 1 day 1 hour is valid in Spark 3.2
Bug: Add spill size metrics for sort merge join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Dec/21 09:57
Affects Version/s: 3.3.0

Sort merge join allows buffered side to spill if the size is too large to hold in memory. It would be good to add a "spill size" SQL metrics in sort merge join, to track how often the spill happens, and how much of spill size would be in case when it spills.
Bug: Fix black version in dev/reformat-python
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Dec/21 19:41
Affects Version/s: 3.3.0

SPARK-37737 updated the black from 21.5b2 to 21.12b0 and applies dev/reformat-python. However, reformat-python still recommend to install black:21.5b2
Bug: Remove ojdbc6 dependency and update docker-integration test docs
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Dec/21 14:52
Affects Version/s: 3.3.0

These are a couple of changes I found worthwhile while running docker integration tests for [#34745|https://github.com/apache/spark/pull/34745].

The doc changes are minor fixes to add the missing repository to the suggested command to run the docker integration tests.

The library change relates to this comment: [#34745 (comment)|https://github.com/apache/spark/pull/34745#discussion_r773084417]; I don't know if my testing was thorough enough, but I found that the referenced dependency was not needed and could be removed.
Bug: Code clean up on promoting strings in math functions
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Dec/21 15:27
Affects Version/s: 3.3.0

There is some similar logic in promoting string to double type in TypeCoercion and AnsiTypeCoercion.

We can change the function Abs/UnaryMinus/UnaryPositive to extend 

ImplicitCastInputTypes so that string will implicitly cast to Double type in the rule `ImplicitTypeCasts`.  We don't have to repeat the logic in rule `PromoteStrings` or `PromoteStringLiterals`
Bug: Add config property to set default Spark listeners
Status: In Progress | Priority: Trivial | Resolution: 
Created: 10/Dec/21 00:53
Affects Version/s: 3.2.0

{{spark.extraListeners}} allows users to custom Spark Listeners. As Spark platform administrators, we want to set our own set of "default" listeners for all jobs on our platforms, however using {{spark.extraListeners}} makes it easy for our clients to unknowingly override our default listeners.

I would like to propose adding {{spark.defaultListeners}} which is intended to be set by administrators and will be combined with {{spark.extraListeners}} at runtime. This is similar in spirit to {{spark.driver.defaultJavaOptions}} and {{spark.plugins.defaultList}}.
Bug: Avoid loading all functions when obtaining hive's DelegationToken
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 06/Dec/21 14:57
Affects Version/s: 3.2.0

At present, when obtaining the delegationToken of hive, all functions will be loaded.
This is unnecessary, it takes time to load the function, and it also increases the burden on the hive meta store.

 

!getDelegationToken_load_functions.png!
Bug: NPE when describe extended a column without col stats
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/23 09:13
Affects Version/s: 3.5.0

Currently executing DESCRIBE TABLE EXTENDED a column without col stats with v2 table will throw a null pointer exception.
{code:java}
Cannot invoke "org.apache.spark.sql.connector.read.colstats.ColumnStatistics.min()" because the return value of "scala.Option.get()" is null
java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.connector.read.colstats.ColumnStatistics.min()" because the return value of "scala.Option.get()" is null
at org.apache.spark.sql.execution.datasources.v2.DescribeColumnExec.run(DescribeColumnExec.scala:63)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:118)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:150)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:241)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:116)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918){code}
Bug: Directly writing the parquet table location will cause the query to be null
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Dec/23 07:08
Affects Version/s: 3.5.0

reproduce code
{code:java}
spark.sql("create table data_table (column string) using parquet")
val identifier = TableIdentifier("data_table")
val location = spark.sessionState.catalog.getTableMetadata(identifier).location
val df = spark.range(2000).map(f => {
  BigInt(100, scala.util.Random).toString(36)
}).toDF()

df.repartition(1).write.mode("overwrite").parquet(location.getPath)
spark.sql("select * from data_table").show() {code}
it will get the null result
{code:java}
+------+
|column|
+------+
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
|  NULL|
+------+
only showing top 20 rows {code}
Bug: Correctness bug in correlated subquery with OFFSET
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/23 14:33
Affects Version/s: 4.0.0

Subqueries with correlation under LIMIT with OFFSET have a correctness bug, introduced recently when support for correlation under OFFSET was enabled but were not handled correctly. (So we went from unsupported, query throws error -> wrong results.)

It’s a bug in all types of correlated subqueries: scalar, lateral, IN, EXISTS

 

It's easy to repro with a query like
{code:java}
create table x(x1 int, x2 int);
insert into x values (1, 1), (2, 2);
create table y(y1 int, y2 int);
insert into y values (1, 1), (1, 2), (2, 4);


select * from x where exists (select * from y where x1 = y1 limit 1 offset 2){code}
Correct result: empty set, see postgres: [https://www.db-fiddle.com/f/dtXNn7hwDnemiCTUhvwgYM/0] 

Spark result: Array([2,2])

 

The [PR|https://github.com/apache/spark/pull/43111/files/324a106611e6d62c31535cfc43863fdaa16e5dda#diff-583171e935b2dc349378063a5841c5b98b30a2d57ac3743a9eccfe7bffcb8f2aR1403] where it was introduced added a test for it, but the golden file results for the test actually were incorrect and we didn't notice. (The bug was initially found by https://github.com/apache/spark/pull/44084)

I'll work on both:
 * Adding support for offset in DecorrelateInnerQuery (the transformation is into a filter on row_number window function, similar to limit).

 * Adding a feature flag to enable/disable offset in subquery support
Bug: Array Indexing is 1-based via ANSI SQL Standard
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/23 03:30
Affects Version/s: 3.4.3

SPARK-28402 was closed by the reporter, because he might think it's a postgres specific feature.

 
{code:none}
<element reference> ::=
    <array value expression>
       <left bracket or trigraph> <numeric value expression> <right bracket or trigraph>
  
1) If the value of  or  is the null value, then
the result of  is the null value.
2) Let the value of  be i.
Case:
a) If i is greater than zero and less than or equal to the cardinality of ,
then the result of  is the value of the i-th element of the value of .

 {code}

The numeric value expression in the range of *greater than zero and less than or equal to the cardinality* a.k.a., [1, size(array)]

Who supports?(Incomplete)
- postgres
- Oracle
- trino/presto
- MySQL

Who doesn't support?(Incomplete)

- hive
- spark sql (ANSI off/on)


For ANSI compliance, we shall make array indexing 1-based when spark.sql.ansi.enabled=true
Bug: Push calculation from Aggregate through Expand
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/23 13:54
Affects Version/s: 4.0.0

Push down calculations in Aggregate through Expand, so the expressions will only be evaluated once.

For example:
 
Agg(grouping, sum($a + $b)) <= Expand <= Project, if $a and $b comes from the Project,
 
a + b will be pushed down to the Project:

Agg(grouping, sum("a + b")) <= Expand <= Project(old projections ++ ($a + $b).as("a + b"))
Bug: Support get partitions by names in partition filter
Status: Open | Priority: Major | Resolution: 
Created: 01/Dec/23 16:23
Affects Version/s: 3.5.0

{{Hive#getPartitionsByFilter}} api is low-performance api and would cause Hive MetaStore backend DBS suffer heavy load if there are many calls to tables containing many partitions.

In this path, we introduce a new switch that enable filtering partitions in Spark side, and then get target partitions by the high performance API {{{}Hive#getPartitionsByNames}}.
Bug: Spark 3.3.3 tuple encoders built using Encoders.tuple do not correctly cast null into None for Option values
Status: Open | Priority: Major | Resolution: 
Created: 04/Dec/23 22:28
Affects Version/s: 3.3.3

In Spark {{3.3.2}} encoders created using {{Encoders.tuple(encoder1, encoder2, ..)}} correctly handle casting {{null}} into {{None}} when the target type is an Option. 

In Spark {{{}3.3.3{}}}, this behaviour has changed and the Option value comes through as {{null}} which is likely to cause a {{NullPointerException}} for most Scala code that operates on the Option. The change seems to be related to the following commit:

[https://github.com/apache/spark/commit/9110c05d54c392e55693eba4509be37c571d610a]

I have made a reproduction with a couple of examples in a public Github repo here:

[https://github.com/q-willboulter/spark-tuple-encoders-bug] 

The common use case where this is likely to be encountered is while doing any joins that can return null, e.g. left or outer joins. When casting the result of a left join it is sensible to wrap the right-hand side in an Option to handle the case where there is no match. Since 3.3.3 this would fail if the encoder is derived manually using {{Encoders.tuple(leftEncoder, rightEncoder).}}

If the entire tuple encoder {{Encoder[(Left, Option[Right]])}} is derived at once using reflection, the encoder works as expected. The bug appears to be in the following function inside {{ExpressionEncoder.scala}}
{code:java}
def tuple(encoders: Seq[ExpressionEncoder[_]]): ExpressionEncoder[_] = ...{code}
Bug: Upgrade Derby to 10.16.1.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/23 03:18
Affects Version/s: 4.0.0

[https://db.apache.org/derby/releases/release-10_16_1_1.cgi]

1. Drop Java Security Manager.
{quote}Derby no longer supports the Java SecurityManager. This is because the Open JDK team deprecated the SecurityManager and marked it for removal.
{quote}
2. Compile on Java 17
{quote}Compile 10.16 into Java 17 byte code
{quote}
Bug: Limit over certain correlated subqueries results in Nosuchelement exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/23 17:05
Affects Version/s: 3.4.0

The types of queries that result in errors are:
 * have LIMIT in the subquery
 * predicate with correlated references does not depend on the inner query (references exclusively outer table).

For example:
{code:java}
SELECT COUNT(DISTINCT(t1a))
FROM t1
WHERE t1d IN (SELECT t2d
              FROM   t2
              WHERE t1a IS NOT NULL
              LIMIT 10);
 {code}
Here, WHERE t1a IS NOT NULL can be conceptually lifted to the join that connects inner and outer query. 

Currently, this query results in an error ("no such element exception").
Bug: Push down part of filter through aggregate with nondeterministic field
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Dec/23 09:06
Affects Version/s: 3.5.0

{code:java}
testRelation
    .groupBy($"a")($"a", Rand(10).as("rand"))
    .where($"a" > 5 && $"rand" > 5)
{code}
after optimize
{code:java}
testRelation
    .where($"a" > 5)
    .groupBy($"a")($"a", Rand(10).as("rand"))
    .where($"rand" > 5)
{code}
Bug: failed to insert the table using the default value of union
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/23 03:01
Affects Version/s: 3.4.0

 

Obtain the following tables and data
{code:java}
create table test_spark(k string default null,v int default null) stored as orc;
create table test_spark_1(k string default null,v int default null) stored as orc;
insert into table test_spark_1 values('k1',1),('k2',2),('k3',3);
create table test_spark_2(k string default null,v int default null) stored as orc; 
insert into table test_spark_2 values('k3',3),('k4',4),('k5',5);

{code}
Execute the following SQL
{code:java}
insert into table test_spark (k) 
select k from test_spark_1
union
select k from test_spark_2 

{code}
exception:
{code:java}
23/12/01 10:44:25 INFO HiveSessionStateBuilder$$anon$1: here is CatalogAndIdentifier
23/12/01 10:44:25 INFO HiveSessionStateBuilder$$anon$1: here is CatalogAndIdentifier
23/12/01 10:44:25 INFO HiveSessionStateBuilder$$anon$1: here is CatalogAndIdentifier
23/12/01 10:44:26 INFO Analyzer$ResolveUserSpecifiedColumns: i.userSpecifiedCols.size is 1
23/12/01 10:44:26 INFO Analyzer$ResolveUserSpecifiedColumns: i.userSpecifiedCols.size is 1
23/12/01 10:44:26 INFO Analyzer$ResolveUserSpecifiedColumns: i.table.output 2 ,resolved :1 , i.query 1
23/12/01 10:44:26 INFO Analyzer$ResolveUserSpecifiedColumns: here is ResolveUserSpecifiedColumns tableOutoyt: 2---nameToQueryExpr : 1Error in query: `default`.`test_spark` requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s). {code}
Bug: Create DataFrame Bug
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/23 03:16
Affects Version/s: 3.5.0

Scala version is 3.3.1

Spark version is 3.5.0


I am using spark-core 3.5.1. I am trying to create a DataFrame through the reflection api, but "No TypeTag available for Person" will appear. I have tried for a long time, but I still don't quite understand why TypeTag cannot recognize my Person case class. 
{code:java}
    import sparkSession.implicits._
    import scala.reflect.runtime.universe._
    case class Person(name: String)
    val a = List(Person("A"), Person("B"), Person("C"))
    val df = sparkSession.createDataFrame(a)
    df.show(){code}
!https://media.discordapp.net/attachments/839723072239566878/1183747749204725821/image.png?ex=65897600&is=65770100&hm=4eeba8d8499499439590a34260f8b441c6594c572c545f5f61f8dc65beeb6a4b&=&format=webp&quality=lossless&width=1178&height=142!
I tested it and it is indeed a problem unique to Scala3
There is no problem on Scala2.13
Bug: Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol
Status: Open | Priority: Major | Resolution: 
Created: 04/Dec/23 16:00
Affects Version/s: 3.5.0

I am trying to create an external partioned bucketed table using this code:
{code:java}
spark.read.parquet("s3://faucct/input")
  .repartition(128, col("product_id"))
  .write.partitionBy("features_date").bucketBy(128, "product_id")
  .option("path", "s3://faucct/tmp/output")
  .option("compression", "uncompressed")
  .saveAsTable("tmp.output"){code}
At first it took more time than expected because it had to rename a lot of files in the end, which requires copying in S3. But I have used the configuration from the documentation – [https://spark.apache.org/docs/3.0.0-preview/cloud-integration.html#committing-work-into-cloud-storage-safely-and-fast]:
{code:java}
spark.hadoop.fs.s3a.committer.name directory
spark.sql.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter {code}
It is properly partitioned: every partition_date has exactly 128 files named like [part-00117-43293810-d0e9-4eee-9be8-e9e50a3e10fd_00117-5eb66a54-2fbb-4775-8f3b-3040b2966a71.c000.parquet|https://s3.console.aws.amazon.com/s3/object/joom-analytics-recom?region=eu-central-1&prefix=recom/dataset/best/best-to-cart-rt/user-product-v4/to_cart-faucct/fnw/ipw/msv2/2023-09-15/14d/tmp_3/features_date%3D2023-09-01/part-00117-43293810-d0e9-4eee-9be8-e9e50a3e10fd_00117-5eb66a54-2fbb-4775-8f3b-3040b2966a71.c000.parquet].
Then I am trying to join this table with another one, for example like this:
{code:java}
spark.table("tmp.output").repartition(128, $"product_id")
  .join(spark.table("tmp.output").repartition(128, $"product_id"), Seq("product_id")).count(){code}
Because of the configuration I get the following errors:
{code:java}
org.apache.spark.SparkException: [INVALID_BUCKET_FILE] Invalid bucket file: s3://faucct/tmp/output/features_date=2023-09-01/part-00000-43293810-d0e9-4eee-9be8-e9e50a3e10fd_00000-5eb66a54-2fbb-4775-8f3b-3040b2966a71.c000.parquet.
  at org.apache.spark.sql.errors.QueryExecutionErrors$.invalidBucketFile(QueryExecutionErrors.scala:2731)
  at org.apache.spark.sql.execution.FileSourceScanExec.$anonfun$createBucketedReadRDD$5(DataSourceScanExec.scala:636) {code}
Bug: Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/23 21:11
Affects Version/s: 4.0.0

`AccumulableInfo` is one of the top heap consumers in driver's heap dumps for stages with many tasks. For a stage with a large number of tasks ({_}O(100k){_}), we saw {*}{{*}}30%{{*}}{*} of the heap usage stemming from `TaskInfo.accumulables()`.

!screenshot-1.png|width=641,height=98!  

The `TaskSetManager` today keeps around the TaskInfo objects ([ref1|https://github.com/apache/spark/blob/c1ba963e64a22dea28e17b1ed954e6d03d38da1e/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L134], [ref2|https://github.com/apache/spark/blob/c1ba963e64a22dea28e17b1ed954e6d03d38da1e/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L192])) and in turn the task metrics (`AccumulableInfo`) for every task attempt until the stage is completed. This means that for stages with a large number of tasks, we keep metrics for all the tasks (`AccumulableInfo`) around even when the task has completed and its metrics have been aggregated. Given a task has a large number of metrics, stages with many tasks end up with a large heap usage in the form of task metrics.
Ideally, we should clear up a task's TaskInfo upon the task's completion, thereby reducing the driver's heap usage.
Bug: Support `readyz` in REST Submission API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/23 20:07
Affects Version/s: 4.0.0

Like https://kubernetes.io/docs/reference/using-api/health-checks/, we need to provide `/readyz` API.

As a workaround, we can use the following.
{code}
readinessProbe:
  exec:
    command: ["sh", "-c", "! (curl -s http://localhost:6066/v1/submissions/status/none | grep -q STANDBY)"]
{code}
Bug: XML: Capture values interspersed between elements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/23 19:03
Affects Version/s: 4.0.0

In XML, elements typically consist of a name and a value, with the value enclosed between the opening and closing tags. But XML also allows to include arbitrary values interspersed between these elements. To address this, we provide an option named `valueTags`, which is enabled by default, to capture these values. Consider the following example:

```

<ROW>
    <a>1</a>
  value1
  <b>
    value2
    <c>2</c>
    value3
  </b>
</ROW>

```
In this example, `<a>`,`<b>`, and `<c>` are named elements with their respective values enclosed within tags. There are arbitrary values value1 value2 value3 interspersed between the elements. Please note that there can be multiple occurrences of values in a single element (i.e. there are value2, value3 in the element <b>)

 

We should parse the values between tags into the valueTags field. If there are multiple occurrences of value tags, the value tag field will be converted to an array type.