av9ash/GitBugs huggingface dataset:
Bug: ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout
Status: Open | Priority: Minor | Resolution: 
Created: 14/Oct/21 10:44
Affects Version/s: 2.4.1

The ExecutorAllocationManager removes idle executors after the configured spark.dynamicAllocation.executorIdleTimeout but in the schedule() it does not use the same configuration.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L249

The value for intervalMillis is 100 and timeunit is ms. Hence you see this log approximately 10 times in a second. 
 | executor.scheduleWithFixedDelay(scheduleTask, 0, intervalMillis, TimeUnit.MILLISECONDS)

In older versions it was at info level
[https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L454]
In the latest versions (and master) it has been changed to debug
[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L540]
The change request for the above - 
[https://github.com/apache/spark/commit/3584d849438ad48ff54af3c982c124a8443dc590]

However, this check for executors to be removed should be using spark.dynamicAllocation.executorIdleTimeout instead of https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L153
Bug: Disable two level of map for final hash aggregation by default
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Oct/21 23:16
Affects Version/s: 3.2.0

This JIRA is to disable two level of maps for final hash aggregation by default. The feature was introduced in [#32242|https://github.com/apache/spark/pull/32242] and we found it can lead to query performance regression when the final aggregation gets rows with a lot of distinct keys. The 1st level hash map is full so a lot of rows will waste the 1st hash map lookup and inserted into 2nd hash map. This feature still benefits query with not so many distinct keys though, so introducing a config to allow query to enable the feature when seeing benefit.
Bug: Fast fail with explicit err msg when calling withWatermark on non-streaming dataset
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 08/Oct/21 07:11
Affects Version/s: 3.1.2

[Dataset.withWatermark|https://github.com/apache/spark/blob/v3.2.0-rc7/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L740] is a function specific for SS.
Now it could be triggered on a batch dataset, and add a specific rule to eliminate in analyze phase. User can call this API and nothing happens, it may be a little bit confused.
If the usage is not as expected, maybe we can just fast fail it with explicit message, and also we do not have to keep on extra rule to do this.
Bug: Upgrade joda-time to 2.10.12
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Oct/21 05:22
Affects Version/s: 3.3.0

joda-time 2.10.12 seems to support an updated TZDB.
https://github.com/JodaOrg/joda-time/compare/v2.10.10...v2.10.12
https://github.com/JodaOrg/joda-time/issues/566#issuecomment-930207547
Bug: pyspark.sql.catalog stubs are missing annotation for createExternalTable
Status: Open | Priority: Minor | Resolution: 
Created: 01/Oct/21 18:32
Affects Version/s: 3.1.0

While method is deprecated, we might still want to have annotations for it, especially when it is unlikely to go away (SPARK-31088) and annotations will be present in 3.3, after SPARK-36881 is merged.
Bug: Misleading Spark Streaming source documentation
Status: Open | Priority: Trivial | Resolution: 
Created: 12/Oct/21 08:16
Affects Version/s: 3.0.1

The documentation at [https://spark.apache.org/docs/latest/streaming-programming-guide.html#advanced-sources] clearly states that *Kafka* (and Kinesis) are available in the Python API v 3.1.2 in *Spark Streaming (DStreams)*. (see attachments for highlight)

However, there is no way to create DStream from Kafka in PySpark >= 3.0.0, as the `kafka.py` file is missing in [https://github.com/apache/spark/tree/master/python/pyspark/streaming]. I'm coming from PySpark 2.4.4 where this was possible. _Should Kafka be excluded as advanced source for spark streaming in Python API in the docs?_

 

Note that I'm aware of this Kafka integration guide [https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html] but I'm not interested in Structured Streaming as it doesn't support arbitrary stateful operations in Python. DStreams support this functionality with `updateStateByKey`.
Bug: Support Java 17 in SparkR SystemRequirements
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Oct/21 20:55
Affects Version/s: 3.3.0

Please bump Java version to <= 17 in [DESCRIPTION|https://github.com/apache/spark/blob/f9f95686cb397271f55aaff29ec4352b4ef9aade/R/pkg/DESCRIPTION]

Currently it is set to be:
{code:java}
SystemRequirements: Java (>= 8, < 12){code}
 [PR|https://github.com/apache/spark/pull/34371] has been created for this issue already.
Bug: Add KMeans distanceMeasure param to PythonMLLibAPI
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/Oct/21 11:24
Affects Version/s: 3.2.0

SPARK-22119 added KMeans {{distanceMeasure}} to the Python API.

We should include this parameter too in the {{PythonMLLibAPI.t}}{{rainKMeansModel}} method.
Bug: Incorrect Spark 3.2.0 package names with included Hadoop binaries
Status: Resolved | Priority: Trivial | Resolution: Duplicate
Created: 27/Oct/21 14:53
Affects Version/s: 3.2.0

*Spark 3.2.0+Hadoop* packages contains Hadoop 3.3 binaries, however file names still refer to Hadoop 3.2, i.e. _spark-3.2.0-bin-*hadoop3.2*.tgz_

[https://dlcdn.apache.org/spark/spark-3.2.0/]

[https://dlcdn.apache.org/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.2.tgz]

[https://dlcdn.apache.org/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.2-scala2.13.tgz]
Bug: Fix behavior inconsistent in Hive table when ‘path’ is provided in SERDEPROPERTIES
Status: Open | Priority: Trivial | Resolution: 
Created: 17/Oct/21 03:34
Affects Version/s: 2.4.5

If a Hive table is created with both {{WITH SERDEPROPERTIES ('path'='<tableLocation>')}} and {{LOCATION <tableLocation>}}, Spark can return doubled rows when reading the table. This issue seems to be an extension of SPARK-30507.


 Reproduce steps:
 # Create table and insert records via Hive (Spark doesn't allow to insert into table like this)
{code:sql}
CREATE TABLE `test_table`(
  `c1` LONG,
  `c2` STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES ('path'='<tableLocationPath>'" )
STORED AS
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION '<tableLocationPath>';

INSERT INTO TABLE `test_table`
VALUES (0, '0');

SELECT * FROM `test_table`;
-- will return
-- 0 0
{code}

 # Read above table from Spark
{code:sql}
SELECT * FROM `test_table`;
-- will return
-- 0 0
-- 0 0
{code}

But if we set {{spark.sql.hive.convertMetastoreParquet=false}}, Spark will return same result as Hive (i.e. single row)

A similar case is that, if a Hive table is created with both {{WITH SERDEPROPERTIES ('path'='<anotherPath>')}} and {{LOCATION <tableLocation>}}, Spark will read both rows under {{anotherPath}} and rows under {{tableLocation}}, regardless of {{spark.sql.hive.convertMetastoreParquet}} ‘s value. However, actually Hive seems to return only rows under {{tableLocation}}

Another similar case is that, if {{path}} is provided in {{TBLPROPERTIES}}, Spark won’t double the rows when {{'path'='<tableLocation>'}}. If {{'path'='<anotherPath>'}}, Spark will read both rows under {{anotherPath}} and rows under {{tableLocation}}, Hive seems to keep ignoring the {{path}} in {{TBLPROPERTIES}}

Code examples for the above cases (diff patch wrote in {{HiveParquetMetastoreSuite.scala}}) can be found in Attachments
Bug: Impl mapValues for KVGDS#mapValues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/23 23:05
Affects Version/s: 3.5.0

Use an resolved func to pass the mapValues together with all aggExprs. Then on the server side unfold it to apply mapValues first before running aggregate.

e.g. https://github.com/apache/spark/commit/a234a9b0851ebce87c0ef831b24866f94f0c0d36
Bug: Mounts the hadoop config map on the executor pod
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/23 07:10
Affects Version/s: 3.4.0

Since SPARK-25815 [,|https://github.com/apache/spark/pull/22911,] the hadoop config map will not be mounted on the executor pod.

Per the  [https://github.com/apache/spark/pull/22911] description:
{code:java}
The main two things that don't need to happen in executors anymore are:
1. adding the Hadoop config to the executor pods: this is not needed
since the Spark driver will serialize the Hadoop config and send
it to executors when running tasks. {code}
But in fact, the executor still need the hadoop configuration.

 

!https://user-images.githubusercontent.com/6757692/238268640-8ff41144-5812-4232-b572-2de2408348ed.png!

 

As shown in above picture, the driver can resolve `hdfs://zeus`, but the executor can not.

so we still need to mount the hadoop config map in executor side.
Bug: Support  modify the SESSION_CATALOG_NAME value
Status: Open | Priority: Major | Resolution: 
Created: 02/May/23 07:49
Affects Version/s: 3.5.0

{code:java}
private[sql] object CatalogManager {
val SESSION_CATALOG_NAME: String = "spark_catalog"
}{code}
 
The SESSION_CATALOG_NAME value cannot be modified。

If the platform supports hive and spark sql, the metadata catalog name is hive_metastore. It's more appropriate. The user directly copies the table name and brings the hive_metastore catalog. In this case, the default spark catalog name needs to be changed。

 

!image-2023-12-27-09-55-55-693.png!

[~fanjia]
Bug: Subquery on single table with having clause can't be optimized
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/23 05:28
Affects Version/s: 3.4.0

Eg:
{code:java}
sql("create view t(c1, c2) as values (0, 1), (0, 2), (1, 2)")

sql("select c1, c2, (select count(*) cnt from t t2 where t1.c1 = t2.c1 " +
"having cnt = 0) from t t1").show() {code}
The error will throw:
{code:java}
[PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery in batch Operator Optimization before Inferring Filters generated an invalid plan: The plan becomes unresolved: 'Project [toprettystring(c1#224, Some(America/Los_Angeles)) AS toprettystring(c1)#238, toprettystring(c2#225, Some(America/Los_Angeles)) AS toprettystring(c2)#239, toprettystring(cnt#246L, Some(America/Los_Angeles)) AS toprettystring(scalarsubquery(c1))#240]
+- 'Project [c1#224, c2#225, CASE WHEN isnull(alwaysTrue#245) THEN 0 WHEN NOT (cnt#222L = 0) THEN null ELSE cnt#222L END AS cnt#246L]
   +- 'Join LeftOuter, (c1#224 = c1#224#244)
      :- Project [col1#226 AS c1#224, col2#227 AS c2#225]
      :  +- LocalRelation [col1#226, col2#227]
      +- Project [cnt#222L, c1#224#244, cnt#222L, c1#224, true AS alwaysTrue#245]
         +- Project [cnt#222L, c1#224 AS c1#224#244, cnt#222L, c1#224]
            +- Aggregate [c1#224], [count(1) AS cnt#222L, c1#224]
               +- Project [col1#228 AS c1#224]
                  +- LocalRelation [col1#228, col2#229]The previous plan: Project [toprettystring(c1#224, Some(America/Los_Angeles)) AS toprettystring(c1)#238, toprettystring(c2#225, Some(America/Los_Angeles)) AS toprettystring(c2)#239, toprettystring(scalar-subquery#223 [c1#224 && (c1#224 = c1#224#244)], Some(America/Los_Angeles)) AS toprettystring(scalarsubquery(c1))#240]
:  +- Project [cnt#222L, c1#224 AS c1#224#244]
:     +- Filter (cnt#222L = 0)
:        +- Aggregate [c1#224], [count(1) AS cnt#222L, c1#224]
:           +- Project [col1#228 AS c1#224]
:              +- LocalRelation [col1#228, col2#229]
+- Project [col1#226 AS c1#224, col2#227 AS c2#225]
   +- LocalRelation [col1#226, col2#227] {code}
 

The reason are when execute subquery decorrelation, the fields in the subquery but not in having clause are wrongly pull up. This problem only occurs when there contain having clause.
Bug: Add Primary Key syntax support
Status: Open | Priority: Major | Resolution: 
Created: 08/May/23 01:31
Affects Version/s: 3.5.0

apache paimon and hudi support primary key definitions. It is necessary to support the primary key definition syntax

https://docs.snowflake.com/en/sql-reference/sql/create-table-constraint#constraint-properties

[~gurwls223]
Bug: Add to_varchar alias for to_char SQL function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/23 15:44
Affects Version/s: 3.4.1

We want to add the alias to_varchar for the function to_char. 

For users who are migrating to Spark SQL such that the SQL engine they formerly used supported to_varchar instead of to_char, this change would minimize the number of changes to their application to ensure it is compatible with Spark SQL syntax and support.
Bug: Build and Run Spark on Java 21
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/23 00:22
Affects Version/s: 3.5.0

- [https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html]
||JDK version||Minimum Scala versions||
|21 (ea)|3.3.1 (soon), 2.13.11, 2.12.18|
|20|3.3.0, 2.13.11, 2.12.18|
|19|3.2.0, 2.13.9, 2.12.16|
|18|3.1.3, 2.13.7, 2.12.15|
|17 (LTS)|3.0.0, 2.13.6, 2.12.15|
|11 (LTS)|3.0.0, 2.13.0, 2.12.4, 2.11.12|
|8 (LTS)|3.0.0, 2.13.0, 2.12.0, 2.11.0|
Bug: Have a separate config for Memory limits for kubernetes pods
Status: Open | Priority: Major | Resolution: 
Created: 15/May/23 01:20
Affects Version/s: 3.4.4

Whole allocated memory to JVM is set into pod resources as both request and limits.

This means there's not a way to use more memory for burst-like jobs in a shared environment.

For example, if spark job uses external process (outside of JVM) to access data, a bit of extra memory required for that, and having configured higher limits for mem could be of use.

Another thought here - have a way to configure different JVM/ pod memory request also could be a valid use case.

 

Github PR: [https://github.com/apache/spark/pull/41067]
Bug: Improve SparkConnectPlanner by reuse Dataset and avoid construct new Dataset
Status: Open | Priority: Major | Resolution: 
Created: 27/May/23 10:08
Affects Version/s: 3.5.0

Currently, SparkConnectPlanner.transformRelation always return the LogicalPlan of Dataset.
The SparkConnectStreamHandler.handlePlan constructs a new Dataset for it.
Sometimes, SparkConnectStreamHandler.handlePlan could reuse the Dataset created by SparkConnectPlanner.transformRelation.
Bug: enable spark sql to drop multiple partitions in one call
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/May/23 08:05
Affects Version/s: 3.2.1

Now spark sql cannot drop multiple partitions in one call, so I fix it

With this patch we can drop multiple partitions like this : 

alter table test.table_partition drop partition(dt<='2023-04-02', dt>='2023-03-31')
Bug: CTAS Command Nodes Prevent Some Optimizer Rules From Running
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/May/23 16:49
Affects Version/s: 3.4.0

The changes introduced to resolve SPARK-41713 in [https://github.com/apache/spark/pull/39220] modified the CTAS commands from having a `DataWritingCommand` trait to a `LeafRunnableCommand` trait. The `DataWritingCommand` trait extends `UnaryCommand`, and has children set to the value of query in the CTAS command. This means that when `transform` is called to traverse the tree with the CTAS command at the root, the entire query is traversed. `LeafRunnableCommand` has a `LeafLike` trait which explicitly sets the value of children to `Nil`. This means that when `transform` is called on the command, no children are found and the query is unaffected by the rule.

In practice, this means that optimizer rules that rely on `transform` (such as `BooleanSimplification`) to traverse the tree do not work with a CTAS. 

This can be demonstrated with a simple query in spark-shell. Without the CTAS we can run a command with an easily simplified boolean expression (`id == 9 && id == 9`) and see it gets optimized out:

!Working - No Create Table.png|width=883,height=342!

With a CTAS, the optimisation does not get applied (as we can see from the `AND` still present in the optimized and physical plans):

!Not Working - Create Table.png|width=885,height=524!

This works in 3.2.0 which had the old CTAS implementation:

!Working - 3.2.0.png|width=885,height=345!
Bug: Support SELECT * EXCEPT AND  SELECT * REPLACE
Status: Open | Priority: Major | Resolution: 
Created: 26/May/23 01:32
Affects Version/s: 3.5.0

ref: [https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#select_except]

https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#select_replace

[~fanjia]
Bug: Add config to control max ratio of decommissioning executors
Status: Open | Priority: Major | Resolution: 
Created: 07/May/23 01:58
Affects Version/s: 3.4.0

Decommission too many executors at the same time with shuffle or rdd migration could severely hurt performance of shuffle fetch. Block manager decommissioner try to migrate shuffle or rdd as soon as possible, this will compete network and disk IO with shuffle fetch in the target executor.
Bug: StructType.toDDL does not pick up on non-nullability of column in nested struct
Status: Open | Priority: Major | Resolution: 
Created: 02/May/23 14:56
Affects Version/s: 3.3.0

h2. The problem

When converting a StructType instance containing a nested StructType column which in turn contains a column for which {{nullable = false}} to a DDL string using {{{}.toDDL{}}}, the resulting DDL string does not include this non-nullability. For example:
{code:java}
val testschema = StructType(List(
  StructField("key", IntegerType, false),
  StructField("value", StringType, true),
  StructField("nestedCols", StructType(List(
    StructField("nestedKey", IntegerType, false),
    StructField("nestedValue", StringType, true)
  )), false)
))

println(testschema.toDDL)
println(StructType.fromDDL(testschema.toDDL)){code}
gives:
{code:java}
key INT NOT NULL,value STRING,nestedCols STRUCT<nestedKey: INT, nestedValue: STRING> NOT NULL

StructType(
  StructField(key,IntegerType,false),
  StructField(value,StringType,true),
  StructField(nestedCols,StructType(
    StructField(nestedKey,IntegerType,true),
    StructField(nestedValue,StringType,true)
  ),false)
){code}
 

This is due to the fact that {{StructType.toDDL}} calls {{StructField.toDDL}} for its fields, which in turn calls {{.sql}} for its {{{}dataType{}}}. If {{dataType}} is a {{{}StructType{}}}, the call to {{.sql}} in turn calls {{.sql}} for all the nested fields, and this last method does not include the nullability of the field in its output.
h2. Proposed solution

{{StructField.toDDL}} should call {{dataType.toDDL}} for a {{{}StructType{}}}, since this will include information about nullability of nested columns.
Bug: Spark AWS Glue date partition push down broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/23 14:03
Affects Version/s: 3.1.0

When using the following project: [https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore]
To have glue supported as as a hive metastore for spark there is an issue when reading a date-partitioned data set. Writing is fine.
You get the following error: 


{quote}org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported expression '2023 - 05 - 03' (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: beed68c6-b228-442e-8783-52c25b9d2243; Proxy: null)
{quote}
 

A fix for this is making sure the date passed to glue is quoted
Bug: Support LCA in grouping expressions
Status: Reopened | Priority: Major | Resolution: 
Created: 29/May/23 04:59
Affects Version/s: 3.5.0

Teradata supports it:
{code:sql}
create table t1(a int) using  parquet;
select a + 1 as a1, a1 + 1 as a2 from t1 group by a1, a2;
{code}


{noformat}
[UNSUPPORTED_FEATURE.LATERAL_COLUMN_ALIAS_IN_GROUP_BY] The feature is not supported: Referencing a lateral column alias via GROUP BY alias/ALL is not supported yet.
{noformat}
Bug: Add support to create DataFrame Reference in Spark connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/23 00:32
Affects Version/s: 3.5.0

Add support in Spark Connect to cache a DataFrame on server side. From client side, it can create a reference to that DataFrame given the cache key.

 

This function will be used in streaming foreachBatch(). Server needs to call user function for every batch which takes a DataFrame as argument. With the new function, we can just cache the DataFrame on the server. Pass the id back to client which can creates the DataFrame reference. The server will replace the reference when transforming.
Bug: Unsigned integer types are deserialized as signed numeric equivalents
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/23 12:37
Affects Version/s: 3.4.0

I'm not sure if "bug" is the correct tag for this jira, but i've tagged it like that for now since the behavior seems odd, happy to update to "improvement" or something else based on the conversation!

h2. Issue

Protobuf supports unsigned integer types, including `uint32` and `uint64`. When deserializing protobuf values with fields of these types, uint32 is converted to `IntegerType` and uint64 is converted to `LongType` in the resulting spark struct. `IntegerType` and `LongType` are [signed|https://spark.apache.org/docs/latest/sql-ref-datatypes.html] integer types, so this can lead to confusing results.

Namely, if a uint32 value in a stored proto is above 2^31 or a uint64 value is above 2^63, their representation in binary will contain a 1 in the highest bit, which when interpreted as a signed integer will come out as negative (I.e. overflow).

I propose that we deserialize unsigned integer types into a type that can contain them correctly, e.g.
uint32 => `LongType`
uint64 => `Decimal(20, 0)`

h2. Backwards Compatibility / Default Behavior
Should we maintain backwards compatibility and we add an option that allows deserializing these types differently? Or should we change change the default behavior (with an option to go back to the old way)? 

I think by default it makes more sense to deserialize them as the larger types so that it's semantically more correct. However, there may be existing users of this library that would be affected by this behavior change. Though, maybe we can justify the change since the function is tagged as `Experimental` (and spark 3.4.0 was only released very recently).

h2. Precedent
I believe that unsigned integer types in parquet are deserialized in a similar manner, i.e. put into a larger type so that the unsigned representation natively fits. https://issues.apache.org/jira/browse/SPARK-34817 and https://github.com/apache/spark/pull/31921
Bug: Statsd sink reporter reports incorrect counter metrics.
Status: Open | Priority: Major | Resolution: 
Created: 17/May/23 09:30
Affects Version/s: 3.1.3

There is a mismatch between the definition of counter metrics between dropwizard (which  is used by spark) and statsD. While Dropwizard interprets counters as cumulative metrics, statsD interprets them as delta metrics. This causes double aggregation in statsd causing inconsistent metrics.