av9ash/GitBugs huggingface dataset:
Bug: Support Auto Partition Statistics Collection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/22 19:34
Affects Version/s: 3.4.0

Currently https://issues.apache.org/jira/browse/SPARK-21127 supports storing the aggregated stats at table level for partitioned tables with config spark.sql.statistics.size.autoUpdate.enabled.

Supporting partition level stats are useful to know which partitions are outliers (skewed partition) and query optimizer works better with partition level stats in case of partition pruning.
Bug: Support Left Semi join in row level runtime filters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/22 10:49
Affects Version/s: 3.3.0

Support Left Semi join in the runtime filtering as well.

This is a follow up to https://issues.apache.org/jira/browse/SPARK-32268 once [https://github.com/apache/spark/pull/35789] is merged.
Bug: If `Sum`, `Count`, `Any` accompany distinct, cannot do partial agg push down.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/22 03:22
Affects Version/s: 3.3.0

Spark could partial push down sum(distinct col), count(distinct col) if data source have multiple partitions, and Spark will sum the value again.

So the result may not correctly.
Bug: Add factory method getConnection into JDBCDialect.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 10:01
Affects Version/s: 3.3.0

At present, the parameter of the factory method for obtaining JDBC connection is empty because the JDBC URL of some databases is fixed and unique.
However, for databases such as Clickhouse, connection is related to the shard node.

So I think the parameter form of getConnection: Partition = > Connection is more general.
Bug: Avoid possible errors due to incorrect file size or type supplied in hadoop conf.
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Mar/22 13:37
Affects Version/s: 3.2.1

This would avoid failures, in case the files are a bit large or a user places a binary file inside the HADOOP_CONF_DIR.

Both of which are not supported at the moment.

The reason is, underlying etcd store does limit the size of each entry to only 1.5 MiB.

[https://etcd.io/docs/v3.4.0/dev-guide/limit/] 

We can process this like SPARK-32221 https://issues.apache.org/jira/browse/SPARK-32221.
Bug: Improve error messages of ParseException from ANTLR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 05:25
Affects Version/s: 3.3.0

This task is intended to improve the error messages of ParseException directly coming from ANTLR.
h2. Bad Error Messages

Many error messages defined in ANTLR are not user-friendly. For example,
{code:java}
spark.sql("sel 1")
 
ParseException: 
mismatched input 'sel' expecting {'(', 'APPLY', 'CONVERT', 'COPY', 'OPTIMIZE', 'RESTORE', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'SYNC', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)
 
== SQL ==
sel 1
^^^ {code}
Following the [Spark Error Message Guidelines|https://spark.apache.org/error-message-guidelines.html], the words in this message are vague and hard to follow. It states ‘What’, but is unclear on the ‘Why’ and ‘How’.

Or,
{code:java}
spark.sql("") // empty query

ParseException: 
mismatched input '<EOF>' expecting {'(', 'CONVERT', 'COPY', 'OPTIMIZE', 'RESTORE', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==

^^^ {code}
Instead of simply telling users it’s an empty line, it outputs a long message, even giving the jargon '<EOF>'.
h2. Where do these error messages come from?

There has been much work on improving ParseException in general (see [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala] for example). But lots of the above error messages are defined in ANTLR and stay unmodified in Spark.

When such an error is encountered in ANTLR, ANTLR notified the exception listener with a message like ‘mismatched input {} expecting {}’. The Spark exception listener _appends_ the line and position to the message, as well as the problematic SQL and several ‘^^^’ marking the error position. Then it throws a ParseException with the appended error message. Spark doesn’t modify the error message given from ANTLR. 

This task focuses on those error messages from ANTLR.
h2. Goals
 # Improve the error messages of ParseException that are from ANTLR; Modify all affected test cases accordingly.
 # Make sure the new error message framework is applied in this change.

h2. Proposed Error Messages Change

It should be in each sub-task and includes concrete before & after cases. See the description of each sub-task for more details.
Bug: It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/22 09:21
Affects Version/s: 3.1.0

 It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated, to reduce I/O operations and improve performance for the External Shuffle Service.
Bug: Add migration guide for spark.sql.hive.convertMetastoreInsertDir and spark.sql.hive.convertMetastoreCtas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 07:05
Affects Version/s: 3.0.3

After we migration to spark3, many job throw exception since in data source API, we can't support overwrite into partition table while reading from this table.
Bug: Expose the number partitions in a stage to TaskContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/22 23:58
Affects Version/s: 3.2.1

Add a new api to expose total partition count in the stage belonging to the task in TaskContext, so that the task knows what fraction of the computation is doing.

With this extra information, users can also generate 32bit unique int ids as below rather than using `monotonically_increasing_id` which generates 64bit long ids.

 
{code:java}
   rdd.mapPartitions { rowsIter =>
        val partitionId = TaskContext.get().partitionId()
        val numPartitions = TaskContext.get().numPartitions()
        var i = 0
        rowsIter.map { row =>
          val rowId = partitionId + i * numPartitions
          i += 1
          (rowId, row)
       }
  }{code}
Bug: Would be nice to be able to configure a client ID pattern in Kafka integration
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Mar/22 13:48
Affects Version/s: 3.0.0

By default Kafka client automatically generated a unique client ID.
Client ID is used by many data lineage tool to gather consumer/producer (for consumer the consumer group is also used, but only client ID can be used for producer).

Setting the [client.id](https://kafka.apache.org/documentation/#producerconfigs_client.id) is options passed to Spark Kafka read or write is not possible, as it would force the same client.id on at east both the driver and the executor.

What could be done is to be able to passed Spark specific option, maybe named `clientIdPrefix`.

e.g.

```scala
val df = spark
  .read
  .format("kafka")
  .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
  .option("subscribePattern", "topic.*")
  .option("startingOffsets", "earliest")
  .option("endingOffsets", "latest")
  .option("clientIdPrefix", "my-workflow-")
  .load()
```

Possible implement would be to update [InternalKafkaProducerPool](https://github.com/apache/spark/blob/master/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/InternalKafkaProducerPool.scala#L75), or maybe in Spark `KafkaConfigUpdater` ?
Bug: Is there a spark (spark-sql_2.12) using log4j 2.17.X ? 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Mar/22 15:56
Affects Version/s: 3.1.0

Hello - 

our system alerting that our spark dependency brings log4j 1.2.17 and i could not find any updated release having log4j 2.17 and above.

the dependency i'm using is :   

<groupId>org.apache.spark</groupId>  

<artifactId>spark-sql_2.12</artifactId>

<version>3.1.0</version>

Is there a plan to fix this ?  anyway to avoid it ? 

<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.1.0</version>

<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.1.0</version>

<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.1.0</version>
Bug: Change spark.sessionstate.conf.getConf/setConf operation to spark.conf.get/set
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/22 10:57
Affects Version/s: 3.4.0

In the sql module, we provide {{SparkSession.conf}} as a unified entry for {{{}SQLConf.set/get{}}}, which can prevent users or logic from modifying StaticSQLConf and Spark configs. However, I found {{SparkSession.sessionstate.conf}} is used in some code to getConf or setConf, which can skip the check of {{{}RuntimeConfig{}}}.
In this PR, we want to unify the behavior of {{SQLConf.getConf/setConf}} to {{{}SparkSession.conf{}}}.
Bug: Better ParseException message for char without length
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/22 23:27
Affects Version/s: 3.3.0

We support char and varchar types. But when users input the type without length, the message is confusing and not helpful at all:
{code:sql}
> SELECT cast('a' as CHAR)

DataType char is not supported.(line 1, pos 19)

== SQL ==
SELECT cast('a' AS CHAR)
-------------------^^^{code}
This ticket would like to improve the error message for these special cases to:
{code:java}
Datatype char requires a length parameter, for example char(10). Please specify the length.{code}
Bug: Retrying on file manager operation in HDFSMetadataLog
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 19/Mar/22 20:13
Affects Version/s: 3.4.0

Currently HDFSMetadataLog uses CheckpointFileManager to do some file operation like opening metadata file. It is very easy to be affected by network blips and causes the streaming query failed. Although we can restart the streaming query, but it takes more time to recover.

Such file operations should be resilient with such situation by retrying.
Bug: Add offset commit time to streaming query listener
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/22 02:00
Affects Version/s: 3.2.1

A major portion of the batch duration is committing offsets at the end of the micro-batch.  The timing for this operation is missing from the durationMs metrics.  Lets add this metric to have a more complete picture of where the time is going during the processing of a micro-batch
Bug: Kafka, spark data frame read fails when a message does not exist for the timestamp in all partitions
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/Mar/22 05:38
Affects Version/s: 3.2.1

*Issue:*
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.spark.util.UninterruptibleThreadRunner$$anon$1.run(UninterruptibleThreadRunner.scala:33)
Caused by: java.lang.AssertionError: *No offset matched from request of topic-partition source_JSON_sgattu_primittive-5 and timestamp 1648013731709.*
at org.apache.spark.sql.kafka010.KafkaOffsetReaderConsumer.$anonfun$readTimestampOffsets$1(KafkaOffsetReaderConsumer.scala:280)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:400)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)

To repro the issue, produce few messages to kafka and then try to fetch using below program (Used scala shell)

bin/kafka-console-producer.sh --bootstrap-server pkc-epwny.eastus.azure.confluent.cloud:9092 --topic source_JSON_sgattu_primittive --producer.config kafkabroker.properties
{quote}{"ordertime":1497014222380,"orderid":18,"itemid":"Item_184","BinaryField":true}
{"ordertime":1497014222399,"orderid":19,"itemid":"Item_185","BinaryField":false}
{quote}
import org.apache.spark.sql.functions._
import spark.implicits._

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val kafkaServer: String = "XXX:9092"
val topicSampleName: String = "t2"
val columns=Array("id", "value", "last", "year", "binary", "intbinary")
val df=sc.parallelize(Seq(
(1, "John", "Doe", 1986, true, 1),
(2, "Ive", "Fish", 1990, false, 0),
(4, "John", "Wayne", 1995, false, 1)
)).toDF(columns: _*)

spark.read.format("kafka").option("kafka.bootstrap.servers", kafkaServer).option("kafka.sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';")
.option("kafka.sasl.mechanism", "PLAIN")
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.bootstrap.servers", "XXX:9092")
.option("subscribe", "kafkaTopic")
.option("startingOffsetsByTimestamp", """\{"source_JSON_sgattu_primittive":{"0":1648013731709,"5":1648013731709,"4":1648013731709,"1":1648013731709,"2":1648013731709,"3":1648013731709}}""")
.load()
Bug: Keep track of SQL query text in Catalyst TreeNode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/22 16:02
Affects Version/s: 3.3.0

Spark SQL uses the class Origin for tracking the position of each TreeNode in the SQL query text. When there is a parser error, we can show the position info in the error message:
{code:java}
> sql("create tabe foo(i int)")
org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'create tabe'(line 1, pos 7)


== SQL ==
create tabe foo(i int)
-------^^^ {code}
It contains two fields: line and startPosition. This is enough for the parser since the SQL query text is known.

However, the SQL query text is unknown in the execution phase. One solution is to include the query text in Origin and show it in the runtime error message.
Bug: Repartition by Column that is Int not working properly only on particular numbers. (11, 33)
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/22 03:12
Affects Version/s: 3.1.2

My understanding is when you call .repartition(a column). For each unique key in that field. The data will go to that partition. There should never be two keys repartitioned to the same part. That behavior is true with a String column. That behavior is also true with an Int column except on certain numbers. In my use case. The magic numbers 11 and 33.
{code:java}
//Int based column repartition
spark.sparkContext.parallelize(Seq(1, 11, 33)).toDF("collectionIndex").
repartition($"collectionIndex").write.mode("overwrite").parquet("path")
//Produces two part files
//String based column repartition
spark.sparkContext.parallelize(Seq("1", "11", "33")).toDF("collectionIndex").
repartition($"collectionIndex").write.mode("overwrite").parquet("path1")
//Produces three part files {code}
 
{code:java}
//Not working as expected
spark.read.parquet("path/part-00000...").distinct.show
spark.read.parquet("path/part-00001...").distinct.show

//Working as expected
spark.read.parquet("path1/part-00000...").distinct.show
spark.read.parquet("path1/part-00001...").distinct.show
spark.read.parquet("path1/part-00002...").distinct.show {code}
!image-2022-03-24-22-16-44-560.png!

This problem really manifested itself when doing something like
{code:java}
spark.sparkContext.parallelize(Seq(1, 11, 33)).toDF("collectionIndex"). repartition($"collectionIndex").write.mode("overwrite").partitionBy("collectionIndex").parquet("path") {code}
Because you end up with incorrect partitions where the data is commingled.
Bug: Make condition in DeleteFromTable required
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/22 18:26
Affects Version/s: 3.3.0

Right now, the condition in {{DeleteFromTable}} is optional. When it is not set, it is equivalent to a true literal. As a consequence, rules that handle such statements have to catch these two different yet equivalent representations, which makes those rules more complex. Instead, we can simply default the condition while parsing and make it required.
Bug: Strengthen the contract on iterator method in StateStore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 07:48
Affects Version/s: 3.3.0

The root cause of SPARK-38320 was that the logic initialized the iterator first, and performed some updates against state store, and iterated through iterator expecting that all updates in between should be visible in iterator.

That is not guaranteed in RocksDB state store, and the contract of Java ConcurrentHashMap which is used in HDFSBackedStateStore does not also guarantee it.

It would be clearer if we update the contract to draw a line on behavioral guarantee to callers so that callers don't get such expectation.
Bug: Show timing for Spark rules and phases in SQL UI and Rest API
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Mar/22 22:20
Affects Version/s: 3.2.1

Currently information regarding spark phases and rule timing is not available on the SQL UI nor Rest API. The Jira aims to add clickable field on SQL UI to show timing of spark phases and rule timing information along with a separate endpoint to show the timing statistics.
Bug: Requesting Restful API can cause NullPointerException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/22 02:37
Affects Version/s: 3.1.0

When requesting Restful API  \{baseURL}/api/v1/applications/$appId/sql/$executionId which is introduced by this PR [https://github.com/apache/spark/pull/28208,] it can cause NullPointerException. The root cause is, when calling method doUpdate() of `LiveExecutionData`, `metricsValues` can be null. Then, when statement `printableMetrics(graph.allNodes, exec.metricValues)` is executed, it will throw  NullPointerException.
Bug: Avoid unnecessary sort in FileFormatWriter if user has specified sort in AQE
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Mar/22 02:08
Affects Version/s: 3.3.0

FileFormatWriter will check and add an implicit sort for dynamic partition columns or bucket columns according to the input physical plan. The check became always failure since AQE AdaptiveSparkPlanExec has no outputOrdering.

That casues a redundant sort if user has specified a sort which satisfies the required ordering (dynamic partition and bucket columns).
Bug: Include unit into the sql string of TIMESTAMPADD/DIFF 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/22 12:01
Affects Version/s: 3.3.0

After https://github.com/apache/spark/pull/35805, the sql method doesn't include unit anymore. The ticket aims to override the sql method in the TIMESTAMPADD and TIMESTAMPDIFF expressions, and prepend unit.
Bug: Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/22 18:44
Affects Version/s: 3.3.0

1. Unregister the functions `timestampadd()` and `timestampdiff()` in `FunctionRegistry.expressions`.
2. Remove the aliases `date_add` for `timestampadd()` and `date_diff` for `timestampdiff()`.
3. Align tests (regenerate golden files) to the syntax rules

where the first parameter `unit` can have one of the identifiers:
   - YEAR
   - QUARTER
   - MONTH
   - WEEK
   - DAY, DAYOFYEAR (valid for timestampadd)
   - HOUR
   - MINUTE
   - SECOND
   - MILLISECOND
   - MICROSECOND

h4. Why are the changes needed?
1. The `timestampadd()`/`timestampdiff()` functions (and their aliases) with arbitrary string column as the first parameter is not require by any standard.
2. Remove the functions and aliases should reduce maintenance cost.
Bug: support load json file in case-insensitive way
Status: Open | Priority: Major | Resolution: 
Created: 18/Mar/22 09:47
Affects Version/s: 3.1.1

The task is to load json files into dataFrame.

 

Currently we use this method:

// textfile is rdd[string], read from json files

val table = spark.table(hiveTableName)
val hiveSchema = table.schema
var df = spark.read.option("mode", "DROPMALFORMED").schema(hiveSchema).json(textfile)

 

The problem is that the field in hiveSchema is all in lower-case,  however the field of json string have upper case. 

For example:

hive schema:

(id  bigint,  name string)

 

json string

{"Id":123, "Name":"Tom"}

 

in this case,  the json string will not be loaded into dataFrame

I have to use the schema of hive table, due to business requirement, that's the pre-condition.

currently I have to transform the key in json string to lower case, like \{"id":123, "name":"Tom"}

 

but I was wondering if there's any better solution for this issue?
Bug: Incorporate numRowsDroppedByWatermark metric from SessionWindowStateStoreRestoreExec into StateOperatorProgress
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/22 23:36
Affects Version/s: 3.3.0

Although we added `numRowsDroppedByWatermark` to `SessionWindowStateStoreRestoreExec`, but currently only `StateStoreWriter` will be collected metrics for `StateOperatorProgress`. So if we need `numRowsDroppedByWatermark` from `SessionWindowStateStoreRestoreExec` to be used in streaming listener, we need to incorporate `SessionWindowStateStoreRestoreExec` into `StateOperatorProgress`.
Bug: Throw Exception if overwriting hive partition table with dynamic and staticPartitionOverwriteMode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 07:37
Affects Version/s: 3.4.0

The `spark.sql.sources.partitionOverwriteMode` allows us to overwrite the existing data of the table through staticmode, but for hive table, it is disastrous. It may deleting all data in hive partitioned table while writing with dynamic overwrite and `partitionOverwriteMode=STATIC`.

Here we add a check for this and throw Exception if this happends.