av9ash/GitBugs huggingface dataset:
Bug: Generated column expression validation fails if there is a char/varchar column anywhere in the schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/23 22:37
Affects Version/s: 3.4.0

When validating generated column expressions, this call https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GeneratedColumn.scala#L123 to checkAnalysis fails when there are char or varchar columns anywhere in the schema.

 

For example, this query will fail
{code:java}
CREATE TABLE default.example (
    name VARCHAR(64),
    tstamp TIMESTAMP,
    tstamp_date DATE GENERATED ALWAYS AS (CAST(tstamp as DATE))
){code}
Bug: [CONNECT][SCALA] range query returns incorrect schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/23 05:58
Affects Version/s: 3.4.1

The following code on Spark Connect produces the following output

Code:

 
{code:java}
val df = spark.range(3)

df.show()
df.printSchema(){code}
 

Output:
{code:java}
+---+
| id|
+---+
|  0|
|  1|
|  2|
+---+

root
 |-- value: long (nullable = true) {code}
The mismatch is that one shows the column as "id" while the other shows this as "value".
Bug: SparkConnectArtifactManager#cleanUpResources deletes all artifacts instead of session-specific artifacts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/23 13:01
Affects Version/s: 3.5.0

_SparkConnectArtifactManager#cleanUpResources_ deletes all resources instead of session-specific resources. 

This method is triggered through the _userSessionMapping_ cache when an entry is removed ([code|https://github.com/apache/spark/blob/b02ea4cd370ce6a066561dfde9d517ea70805a2b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala#L304]). Once triggered, further artifact transfers and existing artifact usage would fail.
Bug: Task failures during custom JAR fetch in executors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/23 06:46
Affects Version/s: 3.5.0

When attempting to use a custom JAR in a Spark Connect session, the tasks fail due to the following error:
{code:java}
23/07/03 17:00:15 INFO Executor: Fetching spark://ip-10-110-22-170.us-west-2.compute.internal:43743/artifacts/d9548b02-ff3b-4278-ab52-aef5d1fc724e//home/venkata.gudesa/spark/artifacts/spark-d6141194-c487-40fd-ba40-444d922808ea/d9548b02-ff3b-4278-ab52-aef5d1fc724e/jars/TestHelloV2.jar with timestamp 0 23/07/03 17:00:15 ERROR Executor: Exception in task 6.0 in stage 4.0 (TID 55) java.lang.RuntimeException: Stream '/artifacts/d9548b02-ff3b-4278-ab52-aef5d1fc724e//home/venkata.gudesa/spark/artifacts/spark-d6141194-c487-40fd-ba40-444d922808ea/d9548b02-ff3b-4278-ab52-aef5d1fc724e/jars/TestHelloV2.jar' was not found.     at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:260)     at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:142)     at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)     at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
 {code}
 

*Root Cause: The URI for the JAR file is invalid.* (Instead of the URI being in the form of {_}/artifacts/<sessionUUID>/jars/<JAR>{_}, its instead \{_}/artifacts/<sessionUUID>/<absolutePath>{_})
Bug: Define the computing logic through PartitionEvaluator API and use it in SQL operators
Status: Open | Priority: Major | Resolution: 
Created: 04/Jul/23 00:39
Affects Version/s: 3.5.0

Define the computing logic through PartitionEvaluator API and use it in SQL operators. This will avoid lambda-based distributed execution 
Ref : SPARK-43061.

 

Note: this is an umbrella Jira to apply PartitionEvaluator  based approach in SQL operators
Bug: Broadcast operation is not required when no parameters are specified
Status: Open | Priority: Minor | Resolution: 
Created: 05/Jul/23 03:48
Affects Version/s: 3.2.0

The ability introduced by SPARK-14912, we can broadcast the parameters of the data source to the read and write operations, but if the user does not specify a specific parameter, the propagation operation will also be performed, which affects the performance has a greater impact, so we need to avoid broadcasting the full Hadoop parameters when the user does not specify a specific parameter

 

!image-2023-07-05-11-51-41-708.png!
Bug: Broadcast operation is not required when no parameters are specified
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 05/Jul/23 03:48
Affects Version/s: 3.2.0

The ability introduced by SPARK-14912, we can broadcast the parameters of the data source to the read and write operations, but if the user does not specify a specific parameter, the propagation operation will also be performed, which affects the performance has a greater impact, so we need to avoid broadcasting the full Hadoop parameters when the user does not specify a specific parameter
Bug: Clean up the compilation warnings related to `it will become a keyword in Scala 3`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jul/23 05:27
Affects Version/s: 3.5.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/JavaTypeInferenceSuite.scala:74:21: [deprecation @  | origin= | version=2.13.7] Wrap `enum` in backticks to use it as an identifier, it will become a keyword in Scala 3.
[warn]   @BeanProperty var enum: java.time.Month = _ {code}
enum will become a keyword in Scala 3, this also includes {{export}} and {{{}given{}}}.

 

Scala 2.13
{code:java}
Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.8).
Type in expressions for evaluation. Or try :help.


scala> val enum: Int = 1
           ^
       warning: Wrap `enum` in backticks to use it as an identifier, it will become a keyword in Scala 3. [quickfixable]
val enum: Int = 1


scala> val export: Int = 1
           ^
       warning: Wrap `export` in backticks to use it as an identifier, it will become a keyword in Scala 3. [quickfixable]
val export: Int = 1


scala> val given: Int = 1
           ^
       warning: Wrap `given` in backticks to use it as an identifier, it will become a keyword in Scala 3. [quickfixable]
val given: Int = 1 {code}
 
Scala 3
 
{code:java}
Welcome to Scala 3.3.1 (17.0.8, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                                                                                     
scala> val enum: Int = 1
-- [E032] Syntax Error: --------------------------------------------------------
1 |val enum: Int = 1
  |    ^^^^
  |    pattern expected
  |
  | longer explanation available when compiling with `-explain`
                                                                                                                                                                                                                                                     
scala> val export: Int = 1
-- [E032] Syntax Error: --------------------------------------------------------
1 |val export: Int = 1
  |    ^^^^^^
  |    pattern expected
  |
  | longer explanation available when compiling with `-explain`
                                                                                                                                                                                                                                                     
scala> val given: Int = 1
-- [E040] Syntax Error: --------------------------------------------------------
1 |val given: Int = 1
  |         ^
  |         an identifier expected, but ':' found
  |
  | longer explanation available when compiling with `-explain` {code}
Bug: Lower unknown shuffle map output log level to WARN if shuffle migration is enabled
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jul/23 04:47
Affects Version/s: 4.0.0

When decommission and shuffle migration is enabled, there're lots of error message like `Asked to update map output for unknown shuffle` . 

As shuffle clean and unregister is done by ContextCleaner in an async way, when target block manager received the shuffle block from decommissioned block manager, then update shuffle location to map output tracker, but at that time, shuffle might have been unregistered. This should not consider as error.
Bug: Remove TaskPagedTable in StagePage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jul/23 04:21
Affects Version/s: 3.4.1

In [SPARK-21809|https://issues.apache.org/jira/browse/SPARK-21809], we introduced stagespage-template.html to show the running status of Stage. TaskPagedTable is no longer effective, but there are still many PRs updating related codes.
Bug: Fix warning condition in MLLib RankingMetrics ndcgAk
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jul/23 13:23
Affects Version/s: 3.4.1

The implementation of nDCG evaluation in MLLib with relevance score (added in 3.4.0, see https://issues.apache.org/jira/browse/SPARK-39446 and [pull request|https://github.com/apache/spark/pull/36843]) implements the following warning when the input data isn't correct: "# of ground truth set and # of relevance value set should be equal, check input data"

 

The logic for raising warnings is faulty at the moment: it raises a warning when the following conditions are both true:
 # {{rel}} is empty
 # {{lab.size}} and {{rel.size}} are not equal.

 

With the current logic, RankingMetrics will:
 * raise incorrect warning when a user is using it in the "binary" mode (i.e. no relevance values in the input)
 * not raise warning (that could be necessary) when the user is using it in the "non-binary" model (i.e. with relevance values in the input)

 

The logic should be to raise a warning should be:
 # {{rel}} is *not empty*
 # {{lab.size}} and {{rel.size}} are not equal.
Bug: Remove useless function `hasRangeExprAgainstEventTimeCol` from `UnsupportedOperationChecker`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jul/23 05:50
Affects Version/s: 4.0.0

funciton `hasRangeExprAgainstEventTimeCol` was introduced by SPARK-40940 and no longer be used after SPARK-42376
Bug: Add a flag to resolve docker tags to hashes at launch time
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jul/23 21:28
Affects Version/s: 3.5.0

If you have an Spark docker tag (like say 3.3) you might want to update the container but only for newly launched jobs not existing jobs. To allow this we can resolve the tag to hash at launch time.

 

In some environments this may also offer a small performance improvement as it saves K8s from having to re-resolve the tag with additional executor launches.
Bug: `SparkConnectServiceSuite` has directory residue after testing
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jul/23 06:01
Affects Version/s: 4.0.0

run 

 

 
{code:java}
build/sbt "connect/testOnly org.apache.spark.sql.connect.planner.SparkConnectServiceSuite"
git status {code}
 

There are residual directories as follows

 
{code:java}
	connector/connect/server/282ce745-440f-44ac-9f43-4fad70d89a44/
	connector/connect/server/my/ {code}
Bug: CheckAnalysis uses error subclass as an error class
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jul/23 17:41
Affects Version/s: 3.5.0

{{CheckAnalysis}} treats {{TYPE_CHECK_FAILURE_WITH_HINT}} as an error class, but it is instead an error subclass of {{{}DATATYPE_MISMATCH{}}}.
{noformat}
spark-sql (default)> select bitmap_count(12);
[INTERNAL_ERROR] Cannot find main error class 'TYPE_CHECK_FAILURE_WITH_HINT'
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot find main error class 'TYPE_CHECK_FAILURE_WITH_HINT'
at org.apache.spark.SparkException$.internalError(SparkException.scala:83)
at org.apache.spark.SparkException$.internalError(SparkException.scala:87)
at org.apache.spark.ErrorClassesJsonReader.$anonfun$getMessageTemplate$1(ErrorClassesJSONReader.scala:68)
at scala.collection.immutable.HashMap$HashMap1.getOrElse0(HashMap.scala:361)
at scala.collection.immutable.HashMap$HashTrieMap.getOrElse0(HashMap.scala:594)
at scala.collection.immutable.HashMap$HashTrieMap.getOrElse0(HashMap.scala:589)
at scala.collection.immutable.HashMap.getOrElse(HashMap.scala:73)
{noformat}
This issue only occurs when an expression uses {{TypeCheckResult.TypeCheckFailure}} to indicate input type check failure. {{TypeCheckResult.TypeCheckFailure}} appears to be deprecated in favor of {{{}TypeCheckResult.DataTypeMismatch{}}}, but recently two expressions were added that use {{{}TypeCheckResult.TypeCheckFailure{}}}: {{BitmapCount}} and {{{}BitmapOrAgg{}}}.

{{BitmapCount}} and {{BitmapOrAgg}} should probably be fixed to use {{{}TypeCheckResult.DataTypeMismatch{}}}. Regardless, the code in {{CheckAnalysis}} that handles {{TypeCheckResult.TypeCheckFailure}} should be corrected (or removed).
Bug: Utils.getOrCreateLocalRootDirs will never take effect after the first call fails, even if the exception is recovered
Status: Open | Priority: Minor | Resolution: 
Created: 18/Jul/23 03:22
Affects Version/s: 3.4.1

{code:java}
private[spark] def getOrCreateLocalRootDirs(conf: SparkConf): Array[String] = {
  if (localRootDirs == null || localRootDirs.isEmpty) {
    this.synchronized {
      if (localRootDirs == null) {
        localRootDirs = getOrCreateLocalRootDirsImpl(conf)
      }
    }
  }
  localRootDirs
}{code}
localRootDirs is only initialized once in the Executor/Driver life cycle. If it fails due to a FileSystem exception (such as a full disk) during the first initialization, localRootDirs will be assigned a value of None instead of null.

Even if the FileSystem exception recovered, the localRootDirs won't re-apply from FileSystem, causing the task on the Executor to continue to fail (tasks relay on fetchFiles locally)
Bug: SparkSession.builder.create doesn't set active session
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jul/23 12:37
Affects Version/s: 3.5.0

ML and other uses rely on {color:#e01e5a}_active_spark_session{color} to find spark session.

Sessions created using getOrCreate method set this variable, but sessions created with create don't.
Bug: Upgrade protobuf-java to 3.23.4
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jul/23 07:43
Affects Version/s: 3.5.0

* [https://github.com/protocolbuffers/protobuf/releases/tag/v23.3]
 * https://github.com/protocolbuffers/protobuf/releases/tag/v23.4
Bug: CheckOverflowInTableInsert does not handle cases where the cast was moved by the optimizer
Status: Open | Priority: Minor | Resolution: 
Created: 14/Jul/23 07:10
Affects Version/s: 3.4.0

CheckOverflowInTableInsert is supposed to rewrite error messages that come from overflow exceptions on casts that are implicitly inserted for INSERTs to match the output schema. This does not work in all cases, for example when the cast is moved by the optimizer
 
{code:java}
CREATE TABLE source(value BIGINT) USING parquet
INSERT INTO source values (${Long.MaxValue.toString})
CREATE TABLE target(value INT) USING parquet
INSERT INTO target SELECT CASE WHEN value < 0 THEN 0 ELSE value END FROM source{code}
In this case, the expression will no nothing because it only becomes effective if the direct child is a cast ([matching code|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L2186])
Bug: The execution time of "Test encryption" is too long after SPARK-44275
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jul/23 13:08
Affects Version/s: 3.5.0

After SPARK-44275, this make SparkConnectClient init with a RetryPolicy, this causes `Test encryption` in `SparkConnectClientSuite` to run longer.

Before

https://github.com/apache/spark/actions/runs/5482075870/jobs/9987058551

- Test encryption (323 milliseconds)

After

https://github.com/apache/spark/actions/runs/5482264019/jobs/9987435847

- Test encryption (10 minutes, 8 seconds)
Bug: Fix the trim logic did't handle ASCII control characters correctly 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jul/23 09:31
Affects Version/s: 3.1.1

The trim logic in Cast expression introduced in [#29375|https://github.com/apache/spark/pull/29375] trim ASCII control characters unexpectly.[!https://user-images.githubusercontent.com/25627922/244651442-ca6a2fb1-2143-4264-84d1-70b6bb755ec7.png!|https://user-images.githubusercontent.com/25627922/244651442-ca6a2fb1-2143-4264-84d1-70b6bb755ec7.png]
And hive
[!https://user-images.githubusercontent.com/25627922/244651942-017aaa4a-133e-4396-9694-79f03f027bbe.png!|https://user-images.githubusercontent.com/25627922/244651942-017aaa4a-133e-4396-9694-79f03f027bbe.png]
Bug: How to specify parameters in spark-sumbit to make HiveDelegationTokenProvider refresh token regularly
Status: Open | Priority: Minor | Resolution: 
Created: 12/Jul/23 03:35
Affects Version/s: 3.1.1

export KRB5CCNAME=FILE:/tmp/krb5cc_1001
./bin/spark-submit -{-}master yarn --deploy-mode client --proxy-user xxxx --conf spark.app.name=spark-hive-test --conf spark.security.credentials.renewalRatio=0.000058 --conf spark.kerberos.renewal.credentials=ccache{-}  -class org.apache.spark.examples.sql.hive.SparkHiveExample /examples/jars/spark-examples_2.12-3.1.1.jar

spark version 3.1.1，I configured it to refresh every 5 seconds。

--deploy-mode client/cluster wtih/without --proxy-user have all been tried, but none of them will work

Missing any configuration parameters？
Bug: Make some syntactic simplification
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jul/23 06:28
Affects Version/s: 3.5.0

- Use `exists` instead of `find` and `emptiness check`
- Use `orNull` instead of `etOrElse(null)`
- Use `getOrElse(key, value)` instead of `get(key).getOrElse(value)` on map
- Use `find` instead of `filter` + `headOption`
Bug: Add Python inbuilt functions to DataFrame for ease of use for Python developers
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jul/23 19:50
Affects Version/s: 3.4.1

Python developers are used to common inbuilt functions when developing but PySpark doesn't support any of the most used inbuilt functionality for DataFrames. PySpark already has this functionality for columns but not for the DataFrame itself. Adding this support for DataFrames would simplify some parts of development. For example:


{code:java}
if df == df1:       # DataFrame Equality 
if df != df2:       # DataFrame Inequality

df_large = df * 100 # Quickly make a larger dataframe through union of copies
                    # Very useful for performance testing

df_sub = df1 - df2  # Simple DataFrame subtraction
                    # Equivalent to df1.subtract(df2)

df4 = df + df1      # Equivalent to df.union(df1)

len(df)             # Equivalent to df.count()

for row in df:      # Equivalent to `for row in df.collect():`
    some_work(row)

if "company_name" in df: # Check if item is in the DataFrame

{code}
 

There is an ongoing DataFrame equality function effort in PR: 41833, I've also built my own.


These are suggestions, any other functions to be added or removed from this list can be discussed.
Bug: Add a new checkstyle rule to prohibit the use of `@Test(expected = SomeException.class)`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jul/23 02:33
Affects Version/s: 3.5.0

[https://github.com/junit-team/junit4/wiki/Exception-testing]

 
{code:java}
The expected parameter should be used with care. The above test will pass if any code in the method throws IndexOutOfBoundsException. Using the method you also cannot test the value of the message in the exception, or the state of a domain object after the exception has been thrown.For these reasons, the previous approaches are recommended. {code}
Bug: Upgrade scala-parser-combinators to 2.3
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jul/23 08:06
Affects Version/s: 3.5.0

[https://github.com/scala/scala-parser-combinators/releases/tag/v2.3.0]

 

new version:
 * Drop support for Scala 2.11.x
 * Fix {{Parsers.Parser.|||}}
Bug: Add Support for TPCH Micro Benchmark
Status: Open | Priority: Trivial | Resolution: 
Created: 04/Jul/23 16:54
Affects Version/s: 3.4.1

I am proposing this Jira to add support for benchmark for TPCH. I see that there is currently support for TPCDS, but couldn't find one for TPCH. I will add support for it.
Bug: Remove redundant method parameter in kafka connector
Status: Open | Priority: Trivial | Resolution: 
Created: 29/Jul/23 00:54
Affects Version/s: 4.0.0

There are have redundant parameters in org.apache.spark.sql.kafka010.KafkaWriter#validateQuery and org.apache.spark.sql.kafka010.KafkaWriter#write, can remove them. They are not used, remove them to make the code more concise.
Bug: eagerly load SparkExitCode class in SparkUncaughtExceptionHandler
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/Jul/23 08:44
Affects Version/s: 3.1.3

There are two background for this improvement proposal:

1. When running spark on yarn, the disk might be corrupted during application running. The corrupted disk might contain the spark jars(cache archive from spark.yarn.archive). In that case , the executor JVM cannot load any spark related classes any more.

2. Spark leverages the OutputCommitCoordinator to avoid data race between speculate tasks so that no tasks could commit the same partition in the same time. In other words, once a task's commit request is allowed, other commit requests would be denied until the committing task is failed.

 

We encountered a corner case combined the above two cases, which makes the spark hangs.  A short timeline could be described as below:
 # task 5372(tid: 21662) starts running in 21:55
 # the disk contains the spark archive for that task/executor is corrupted, thus making the archive inaccessible from executor's JVM perspective, it happened around 22:00
 # the task continues running, at 22:05, it requests commit from coordinator and performs the commit. 
 # however due the corrupted disk, some exception raised in the executor JVM.
 # The SparkUncaughtExceptionHandler kicks in, however as the jar/disk is corrupted, the handler itself throws an exception, and the halt process throws an exception too.
 # The executor is hanging there, no more tasks are running. However the authorized commit request is still valid in the driver side
 # Speculate tasks start to click in, due to no commit permission, all speculate tasks are killed/denied.
 # The job is hanging until our SRE killed the container from outside.

Some screenshot are provided below.

!image-2023-07-25-16-46-03-989.png!

!image-2023-07-25-16-46-28-158.png!

!image-2023-07-25-16-46-42-522.png!

For this specific case: I'd like to the propose to eagerly load SparkExitCode class in the 
SparkUncaughtExceptionHandler, so that the halt process could be executed rather than throws an exception as SparkExitCode is not loadable during the previous scenario.
Bug: AQE and InMemoryTableScanExec correctness bug
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 18/Oct/23 13:34
Affects Version/s: 3.4.1

The following query should return 1000000
{code:java}
import org.apache.spark.storage.StorageLevel

val df = spark.range(0, 1000000, 1, 5).map(l => (l, l))
val ee = df.select($"_1".as("src"), $"_2".as("dst"))
  .persist(StorageLevel.MEMORY_AND_DISK)

ee.count()
val minNbrs1 = ee
  .groupBy("src").agg(min(col("dst")).as("min_number"))
  .persist(StorageLevel.MEMORY_AND_DISK)
val join = ee.join(minNbrs1, "src")
join.count(){code}
but on spark 3.5.0 there is a correctness bug causing it to return `104800` or some other smaller value.