av9ash/GitBugs huggingface dataset:
Bug: Creating partitions through jdbc connection to beeline is slow
Status: Open | Priority: Major | Resolution: 
Created: 15/Dec/23 03:01
Affects Version/s: 3.0.0

Use jdbc to connect to spark beeline through the connection pool to perform partition creation operations. When the number of connections exceeds 4, the speed of creating partitions will be very slow, and the execution time of each SQL is 4s-10s. Spark 2.x does not have this problem, and the execution time of each SQL is within 1 second.
Bug: Structured Streaming UI doesn't display graph correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/23 23:10
Affects Version/s: 4.0.0

The Streaming UI is broken currently at spark master. Running a simple query:

```

q = spark.readStream.format("rate").load().writeStream.format("memory").queryName("test_wei").start()

```

Would make the spark UI shows empty graph for "operation duration":

!https://private-user-images.githubusercontent.com/10248890/289990561-fdb78c92-2d6f-41a9-ba23-3068d128caa8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDI0MjI2NjksIm5iZiI6MTcwMjQyMjM2OSwicGF0aCI6Ii8xMDI0ODg5MC8yODk5OTA1NjEtZmRiNzhjOTItMmQ2Zi00MWE5LWJhMjMtMzA2OGQxMjhjYWE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjEyVDIzMDYwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI4N2FkZDYyZGQwZGZmNWJhN2IzMTM3ZmI1MzNhOGExZGY2MThjZjMwZDU5MzZiOTI4ZGVkMjc3MjBhMTNhZjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.KXhI0NnwIpfTRjVcsXuA82AnaURHgtkLOYVzifI-mp8!

Here is the error:

!https://private-user-images.githubusercontent.com/10248890/289990953-cd477d48-a45e-4ee9-b45e-06dc1dbeb9d9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDI0MjI2NjksIm5iZiI6MTcwMjQyMjM2OSwicGF0aCI6Ii8xMDI0ODg5MC8yODk5OTA5NTMtY2Q0NzdkNDgtYTQ1ZS00ZWU5LWI0NWUtMDZkYzFkYmViOWQ5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjEyVDIzMDYwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI2MzIyYjQ5OWQ3YWZlMGYzYjFmYTljZjIwZjBmZDBiNzQyZmE3OTI2ZjkxNzVhNWU0ZDAwYTA4NDRkMTRjOTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JAqEG_4NCEiRvHO6yv59hZdkH_5_tSUuaOkpEbH-I20!

 

I verified the same query runs fine on spark 3.5, as in the following graph.

!https://private-user-images.githubusercontent.com/10248890/289990563-642aa6c3-7728-43c7-8a11-cbf79c4362c5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDI0MjI2NjksIm5iZiI6MTcwMjQyMjM2OSwicGF0aCI6Ii8xMDI0ODg5MC8yODk5OTA1NjMtNjQyYWE2YzMtNzcyOC00M2M3LThhMTEtY2JmNzljNDM2MmM1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjEyVDIzMDYwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM0YmZiYTkyNGZkNzBkOGMzMmUyYTYzZTMyYzQ1ZTZkNDU3MDk2M2ZlOGNlZmQxNGYzNTFjZWRiNTQ2ZmQzZWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.MJ8-78Qv5KkoLNGBrLXS-8gcC7LZepFsOD4r7pcnzSI!

 

This should be a problem from the library updates, this could be a potential source of error: [https://github.com/apache/spark/pull/42879]
Bug:  Issue with CSV schema inference and malformed records
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/23 10:02
Affects Version/s: 3.5.0

There appears to be a discrepancy in the behavior of schema inference in the CSV reader compared to JSON. When processing CSV files without a predefined schema, the mechanism to handle malformed records seems to be inconsistent. Unlike the JSON format, where a `_corrupt_record` column is automatically added in the presence of malformed records, the CSV format does not exhibit this behavior. This inconsistency can lead to unexpected results and data loss during processing.

*Steps to Reproduce:*
 # Create a CSV file with malformed records without providing a schema.
 # Observe that the `_corrupt_record` column is not automatically added to the final dataframe.

*Expected Result:* The `_corrupt_record` column should be automatically added to the final dataframe when processing a CSV file with malformed records, similar to the behavior observed with JSON files.
Bug: LegacyFastTimestampFormatter.parseOptional should not throw exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/23 22:02
Affects Version/s: 3.5.0

When setting spark.sql.legacy.timeParserPolicy=LEGACY, Spark will use the LegacyFastTimestampFormatter to infer potential timestamp columns. The inference shouldn't throw exception.

However, when the input is 23012150952, there is exception:

```

For input string: "23012150952"

java.lang.NumberFormatException: For input string: "23012150952"

at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)

at java.base/java.lang.Integer.parseInt(Integer.java:668)

at java.base/java.lang.Integer.parseInt(Integer.java:786)

at org.apache.commons.lang3.time.FastDateParser$NumberStrategy.parse(FastDateParser.java:304)

at org.apache.commons.lang3.time.FastDateParser.parse(FastDateParser.java:1045)

at org.apache.commons.lang3.time.FastDateFormat.parse(FastDateFormat.java:651)

at org.apache.spark.sql.catalyst.util.LegacyFastTimestampFormatter.parseOptional(TimestampFormatter.scala:418)

```
Bug: spark.catalog.listDatabases() fails if containing schemas with special characters when spark.sql.legacy.keepCommandOutputSchema set to true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/23 19:14
Affects Version/s: 4.0.0

When the SQL conf {{spark.sql.legacy.keepCommandOutputSchema}} is set to true:
Before:
// support there is a xyyu-db-with-hyphen schema in the catalog
spark.catalog.listDatabases()

[INVALID_IDENTIFIER] The identifier xyyu-db-with-hyphen is invalid. Please, consider quoting it with back-quotes
Bug: Migrate sub-classes of AnalysisException to error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/23 18:15
Affects Version/s: 4.0.0

Currently, there are a few sub-classes of AnalysisException haven't ported on error classes yet:
- NonEmptyNamespaceException
- UnresolvedException
- ExtendedAnalysisException

This ticket aims to migrate two of them.
Bug: Expose new ArtifactManager API to support in-memory artifacts and sub-directory structure
Status: Open | Priority: Major | Resolution: 
Created: 01/Dec/23 13:42
Affects Version/s: 4.0.0

Currently, without the use of a REPL/Class finder, there is no API to support adding in-memory artifacts to the remote Spark Connect session. 

Further, there is currently no API to preserve/impose a sub-directory structure on the files we send over the wire.
Bug: calculation error 
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/23 10:47
Affects Version/s: 3.4.1

Hi,

I had wrong values while using the POW function in SPARK SQL, please see this example (in this example I multiply (10^-2)*25051 and the expected result is 250.51

 

from pyspark.sql.functions import lit
df = spark.range(1)
df.createOrReplaceTempView("TEST")
spark.sql("SELECT POWER(10, -2)*25051 FROM TEST").show()

 

but I got 250.51000000000002
Bug: Add ExecutedPlanPrepRules to SparkSessionExtensions
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 04/Dec/23 09:44
Affects Version/s: 3.2.0

Some rules (Rule[SparkPlan]) are applied when preparing for the executedPlan. However, users do not have the ability to add rules in this context.
{code:java}
// org.apache.spark.sql.execution.QueryExecution#preparations  
private[execution] def preparations(
    sparkSession: SparkSession,
    adaptiveExecutionRule: Option[InsertAdaptiveSparkPlan] = None,
    subquery: Boolean): Seq[Rule[SparkPlan]] = {
  // `AdaptiveSparkPlanExec` is a leaf node. If inserted, all the following rules will be no-op
  // as the original plan is hidden behind `AdaptiveSparkPlanExec`.
  adaptiveExecutionRule.toSeq ++
  Seq(
    CoalesceBucketsInJoin,
    PlanDynamicPruningFilters(sparkSession),
    PlanSubqueries(sparkSession),
    RemoveRedundantProjects,
    EnsureRequirements(),
    // `ReplaceHashWithSortAgg` needs to be added after `EnsureRequirements` to guarantee the
    // sort order of each node is checked to be valid.
    ReplaceHashWithSortAgg,
    // `RemoveRedundantSorts` needs to be added after `EnsureRequirements` to guarantee the same
    // number of partitions when instantiating PartitioningCollection.
    RemoveRedundantSorts,
    DisableUnnecessaryBucketedScan,
    ApplyColumnarRulesAndInsertTransitions(
      sparkSession.sessionState.columnarRules, outputsColumnar = false),
    CollapseCodegenStages()) ++
    (if (subquery) {
      Nil
    } else {
      Seq(ReuseExchangeAndSubquery)
    })
}{code}
We need to add some "Rule[SparkPlan]"s at this position because currently, all such rules are present in AQE, which requires users to use AQE and meet the requirements to enter AdaptiveSparkPlanExec. This makes it difficult to implement certain extensions for simple SQLs.

For example, adding some new datasource filters for external data sources is challenging. Modifying DataSourceStrategy directly is not conducive to staying in sync with future advancements in the community. Additionally, customizing the Strategy makes it difficult to append new functionalities in an incremental manner. If we define AQE rules, they would not be effective for the simplest 'SELECT * FROM ... WHERE ...' statements. Therefore, it is necessary to introduce a customizable Rule[SparkPlan] between sparkPlan and executedPlan.

We could add an extension called "ExecutedPlanPrepRule" to SparkSessionExtensions,  which would allow users to add their own rules.
Bug: add optional batch id _spark_metadata sink file status 
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/23 06:38
Affects Version/s: 3.5.0

Currently FileStreamSinkLog use SinkFileStatus class for serialization purpose but it does preserve batch id which can be used effectively for snapshots

Change the definition to 

{code}

case class SinkFileStatus (path : String, size : Long, isDir : Boolean, modificationTime : Long, blockReplication: Int, blockSize: Long, action : String, bactchId : Option[Long])

{code}
Bug: Improve `FileSystemPersistenceEngine` to allow non-exist parents
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/23 22:24
Affects Version/s: 4.0.0

To prevent the following
{code}
info]   java.io.IOException: No such file or directory
[info]   at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
[info]   at java.base/java.io.File.createNewFile(File.java:1043)
[info]   at org.apache.spark.deploy.master.FileSystemPersistenceEngine.serializeIntoFile(FileSystemPersistenceEngine.scala:62)
[info]   at org.apache.spark.deploy.master.FileSystemPersistenceEngine.persist(FileSystemPersistenceEngine.scala:45)
{code}
Bug: Directory with number name should not be treated as metadata log
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/23 05:54
Affects Version/s: 3.3.3

HDFSMetadataLog takes a metadata path as parameter. When it goes to retrieves all batches metadata, it calls `CheckpointFileManager.list` to get all files under the metadata path. However, currently all implementations of `CheckpointFileManager.list` returns all files/directories under the given path. So if there is a dictionary with name of batch number (a long value), the directory will be returned too and cause trouble when HDFSMetadataLog goes to read it.

Actually, `CheckpointFileManager.list` method clearly defines that it lists the "files" in a path. That's being said, current implementations don't follow the doc. We should fix it.
Bug: Spark cannot support Docker bridge network in YARN
Status: Open | Priority: Major | Resolution: 
Created: 09/Dec/23 19:35
Affects Version/s: 3.5.1

Hello Spark team,

I recently found a possible bug in Spark YarnAllocator.

Basically when I try to run Spark applications on YARN with Docker bridge network, the job failed with binding address error at Executor side.

I believe it is caused by the YarnAllocator implementation in Spark, the executor is trying to bind the hostname of the NodeManager instead of the hostname of the container. In host network it's fine but bridge network will break.

For more details please checkout [RCA - Spark + YARN Docker Bridge Network|https://github.com/EC528-Fall-2023/Kata-Containers-for-SPARK/blob/main/docs/troubleshoot/rca-docker-bridge-net.md].

It looks like YARN Container API does not return the container hostname related information, which mean to solve this issue, we may also need to make changes at Hadoop YARN side?

 

Please let me know if you have any questions, many thanks!

—

Best Regards,

Jingwei Zhang
Bug: Cancel zombie tasks when a ShuffleMapStage is finished
Status: Open | Priority: Major | Resolution: 
Created: 09/Dec/23 00:26
Affects Version/s: 3.5.0

Spark cancels zombie or speculative tasks when a job is finished. We should also cancel the running tasks when a ShuffleMapStage is finished, because we won’t be using the result of those tasks any more.
Bug: Protobuf: Permissive mode should return null rather than struct with null fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/23 22:54
Affects Version/s: 3.4.2

Consider a protobuf with two fields {{message Person { string name = 1; int id = 2; }}
 * The struct returned by {{from_protobuf("Person")}} like this:
 ** STRUCT<name STRING, id INT>

 * If the underlying binary record fails to deserialize, it results in a exception and query fails.

 * Buf if the option {{mode}} is set to {{PERMISSIVE}} , malformed records are tolerated {{null}} is returned.
 ** {*}BUT{*}: The retuned struct looks like this \{"name: null, id: "null"}

 * 
 ** 
 *** This is not convenient to the user.
 *** *Ideally,* {{from_protobuf()}} *should return* {{null}} *.*
 ** {{from_protobuf()}} borrowed the current behavior from {{from_avro()}} implementation. It is not clear what the motivation was.

I think we should update the implementation to return {{null}} rather than a struct with null-fields inside.
Bug: Remove unnecessary override functions when constructing `WrappedCloseableIterator` in `ResponseValidator#wrapIterator`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/23 07:14
Affects Version/s: 4.0.0

Should reuse functions defined in {{WrappedCloseableIterator}} instead of overriding them

 
*ResponseValidator#wrapIterator*
 
{code:java}
def wrapIterator[T <: GeneratedMessageV3, V <: CloseableIterator[T]](
    inner: V): WrappedCloseableIterator[T] = {
  new WrappedCloseableIterator[T] {

    override def innerIterator: Iterator[T] = inner

    override def hasNext: Boolean = {
      innerIterator.hasNext
    }

    override def next(): T = {
      verifyResponse {
        innerIterator.next()
      }
    }

    override def close(): Unit = {
      innerIterator match {
        case it: CloseableIterator[T] => it.close()
        case _ => // nothing
      }
    }
  }
} {code}
*WrappedCloseableIterator*
 
{code:java}
private[sql] abstract class WrappedCloseableIterator[E] extends CloseableIterator[E] {

  def innerIterator: Iterator[E]

  override def next(): E = innerIterator.next()

  override def hasNext: Boolean = innerIterator.hasNext

  override def close(): Unit = innerIterator match {
    case it: CloseableIterator[E] => it.close()
    case _ => // nothing
  }
} {code}
Bug: TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join
Status: Open | Priority: Major | Resolution: 
Created: 07/Dec/23 00:32
Affects Version/s: 3.4.2

{{TPCDSQueryTestSuite}} fails for q39a and a39b with {{broadcastHashJoinConf}} and {{shuffledHashJoinConf}}. It works fine with {{sortMergeJoinConf}}

{code}SPARK_TPCDS_DATA=<path of TPCDS SF=1 data> build/sbt "~sql/testOnly *TPCDSQueryTestSuite -- -z q39a"{code}
{code}
[info] - q39a *** FAILED *** (19 seconds, 139 milliseconds)
[info]   java.lang.Exception: Expected "...25	1.022382911080458[8 ..." but got "...25	1.022382911080458[5 ..."
{code}

{code}SPARK_TPCDS_DATA=<path of TPCDS SF=1 data> build/sbt "~sql/testOnly *TPCDSQueryTestSuite -- -z q39b"{code}

{code}
[info] - q39b *** FAILED *** (19 seconds, 351 milliseconds)
[info]   java.lang.Exception: Expected "...34	1.563403519178623[3	3	10427	2	381.25	1.0623056061004696
[info] 3	3315	1	271.75	1.555976998814345	3	3315	2	393.75	1.0196319345405949
[info] 3	3393	1	260.0	1.5009563026568116	3	3393	2	470.25	1.129275872154205
[info] 4	16211	1	257.6666666666667	1.6381074811154002]	4	16211	2	352.25	1....", but got "...34	1.563403519178623[5	3	10427	2	381.25	1.0623056061004696
[info] 3	3315	1	271.75	1.555976998814345	3	3315	2	393.75	1.0196319345405949
[info] 3	3393	1	260.0	1.5009563026568118	3	3393	2	470.25	1.129275872154205
[info] 4	16211	1	257.6666666666667	1.6381074811154]	4	16211	2	352.25	1...." Result did not match
{code}
Bug: Test missing test coverage for Column (pyspark.sql.column)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/23 03:49
Affects Version/s: 4.0.0

https://app.codecov.io/gh/apache/spark/commit/1a651753f4e760643d719add3b16acd311454c76/blob/python/pyspark/sql/column.py
Bug: Test catalog error classes (pyspark.sql.catalog)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/23 02:57
Affects Version/s: 4.0.0

See https://app.codecov.io/gh/apache/spark/commit/1a651753f4e760643d719add3b16acd311454c76/blob/python/pyspark/sql/catalog.py
Bug: Test captured errors (pyspark.errors.exceptions.captured)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/23 02:27
Affects Version/s: 4.0.0

https://app.codecov.io/gh/apache/spark/commit/1a651753f4e760643d719add3b16acd311454c76/blob/python/pyspark/errors/exceptions/captured.py
Bug: Exclude generated files from the code coverage report
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/23 02:47
Affects Version/s: 4.0.0

We should exclude https://app.codecov.io/gh/apache/spark/commit/1a651753f4e760643d719add3b16acd311454c76/tree/python/pyspark/sql/connect/proto
Bug: Fix error handling routine so it wouldn't fall into infinite recursion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/23 10:07
Affects Version/s: 4.0.0

Currently, we can fall into infinite recursion as follows:

{quote}[Some error happens] -> _handle_error -> _handle_rpc_error -> _display_server_stack_trace -> RuntimeConf.get -> SparkConnectClient.config -> [An error happens] -> _handle_error.{quote}
Bug: New assertions in AddArtifact RPC make the connect client incompatible with older clusters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/23 09:36
Affects Version/s: 4.0.0

We added new assertions to the AddArtifact RPC - [https://github.com/apache/spark/commit/d9c5f9d6#diff-d4744c7abd099c57d04746140aba3c20b93f1ac011f5915f963e0a3e0758690e]

As part of this change, we have also updated the RPC implementation to return session id as part of its response.

 

Since the assertion depends on the session id to be present, it makes the protocol incompatible such that newer Connect clients that apply this assertion are incompatible with older clusters that don't have the corresponding service side changes.
Bug: Enable all Kinesis tests in Github Actions by default
Status: Open | Priority: Major | Resolution: 
Created: 04/Dec/23 10:21
Affects Version/s: 3.5.0

This ticket is created as per the discussion in this PR: [https://github.com/apache/spark/pull/43736#issuecomment-1833368339.]  Some Kinesis requires interaction with Amazon Kinesis service which would incur billing costs to users, thus they are not enabled by default. Further investigations are needed to figure out a way to run all Kinesis tests in Github Actions.
Bug: spark-submit does not provide a method to setApplicationTimeouts
Status: Open | Priority: Major | Resolution: 
Created: 04/Dec/23 08:35
Affects Version/s: 3.5.0

 
{{ApplicationSubmissionContext#setApplicationTimeouts(Map<ApplicationTimeoutType, Long> applicationTimeouts)}}

This can be used to enforce application maximum lifetime{{{}. {}}}

{{}}

{{Javadoc at}}

[https://hadoop.apache.org/docs/r3.0.2/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setApplicationTimeouts-java.util.Map-]{{{}{}}}

{{https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html}}

It would be {{preferable to be able to set this to enforce timeout of an application.}}

{{}}

It is possible to set these cluster wide from yarn scheduler queues:

{{[https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html]}}

{{}}

{{}}

{{yarn.scheduler.capacity.<queue-path>.maximum-application-lifetime}}

{{yarn.scheduler.capacity.root.<queue-path>.default-application-lifetime}} 

{{}}

{{however this is suboptimal in many cases as a submitted might have different requirements for application lifetime than the queue defaults.}}

{{}}
Bug: Restrict charsets in decode()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/23 09:42
Affects Version/s: 4.0.0

Currently the list of supported charsets in decode() is not stable and fully depends on the used JDK version. So, sometimes user code might not work because a devop changed Java version in Spark cluster. The ticket aims to restrict the list of supported charsets by:

{code}
'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'
{code}
Bug: Supports parse multiple sql statements
Status: Open | Priority: Major | Resolution: 
Created: 01/Dec/23 05:58
Affects Version/s: 4.0.0

 
In the SqlBaseParser.g4 file, add the following code to support the parsing of multiple sql. select * from (select * from test), which resolves into two statements. Need to add alias
{code:java}
sqlStatements
: singleStatement* EOF
;

singleStatement
: statement SEMICOLON?
; {code}
Bug: The filter of partition including cast function may lead the partition pruning to disable
Status: Open | Priority: Minor | Resolution: 
Created: 20/Dec/23 02:42
Affects Version/s: 3.2.0

SQL：select * from test_db.test_table where day between date_sub('2023-12-01',1) and  '2023-12-03'

The Physical Plan of sql above will implement _cast_ function on partition col 'day',  like this, {_}cast(day as date) > 2023-11-30{_}. In this situation, spark just pass the filter condition _day < "2023-12-03"_ to HiveMetastore, not including filter condition {_}cast(day as date) > 2023-11-30{_}, which may lead performance of HMS degarde if the HiveTable has huge number of partitions.

 

In this regard, a new rule may solve this problem. This rule can convert binary comparison _cast(day as date) > 2023-11-30_ to {_}day > cast(2023-11-30 as string){_}. The right node is foldable, so the result is {_}day > "2023-11-30"{_}, and the filter condition passed to HMS will be _day > "2023-11-30" and_ _day < "2023-12-03"._