av9ash/GitBugs huggingface dataset:
Bug: Fail on uploading test files, not when downloading them
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Apr/22 15:42
Affects Version/s: 3.4.0

The CI workflow "Report test results" fails when there are no artifacts to be downloaded from the triggering workflow. In some situations, the triggering workflow is not skipped, but all test jobs are skipped in case no code changes are detected.

In that situation, no test files are uploaded, which makes the triggered workflow fail.

Downloading no test files can have two reasons:
1. No tests have been executed or no test files have been generated.
2. No code has been built and tested deliberately.

You want to be notified in the first situation to fix the CI. Therefore, CI should fail when code is built and tests are run but no test result files are been found.
Bug: smaller pyspark install if not using streaming?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 13/Apr/22 02:15
Affects Version/s: 3.2.1

h3. Describe the feature

i am trying to include pyspark in my docker image, but the size is around 300MB

the largest jar is rocksdbjni-6.20.3.jar at 35MB

is it safe to remove this jar if i have no need for SparkStreaming?

is there any advice on getting the install smaller? perhaps a map of which jars are needed for batch vs sql vs streaming?
h3. Use Case

smaller python package means i can pack more concurrent pods on to my eks workers
Bug: Creating a struct with a float inside 
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 08/Apr/22 20:58
Affects Version/s: 3.2.1

When creating a dataframe using createDataFrame that contains a float inside a struct, the float is set to null. This only happens if using a list of dictionaries as data type, if I use a list of Rows it works fine:
{code:java}
data = [{"MyStruct": {"MyInt": 10, "MyFloat": 10.1}, "MyFloat": 10.1}]

spark.createDataFrame(data).show()
# +-------+------------------------------+
# |MyFloat|MyStruct                      |
# +-------+------------------------------+
# |10.1   |{MyInt -> 10, MyFloat -> null}|
# +-------+------------------------------+ 


data = [Row(MyStruct=Row(MyInt=10, MyFloat=10.1), MyFloat=10.1)]

spark.createDataFrame(data).show()
# +-------+------------------------------+
# |MyFloat|MyStruct                      |
# +-------+------------------------------+
# |10.1   |{MyInt -> 10, MyFloat -> 10.1}|
# +-------+------------------------------+ {code}

Note MyFloat inside MyStruct is set to null in the first example. Interestingly enough, when I do the same with Row, or if I specify the schema, then this does not happen (second example).
Bug: Improve the test coverage for pyspark/find_spark_home.py
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Apr/22 03:09
Affects Version/s: 3.4.0

We should test when the environment variables are not set (https://app.codecov.io/gh/apache/spark/blob/master/python/pyspark/find_spark_home.py)
Bug: [SQL] behavior of schema_of_json not same with 2.4.0
Status: Open | Priority: Minor | Resolution: 
Created: 02/Apr/22 03:32
Affects Version/s: 3.1.1

When I switch to spark 3.1.1 from spark 2.4.0, I found a built-in function throw errors:
|== Physical Plan == org.apache.spark.sql.AnalysisException: cannot resolve 'schema_of_json(get_json_object(`adtnl_info_txt`, '$.all_model_scores'))' due to data type mismatch: The input json should be a foldable string expression and not null; however, got get_json_object(`adtnl_info_txt`, '$.all_model_scores').; line 3 pos 2; |

But schema_of_json worked well in 2.4.0, So, is it a bug, or a new feature, which doesn't support non-Literal expressions?
Bug: Support Char in Literal.create
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Apr/22 01:47
Affects Version/s: 3.4.0

https://github.com/apache/spark/commit/54fcaafb094e299f21c18370fddb4a727c88d875 added the support of Char at the literal. Liternal.create should work with this too.
Bug: Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Apr/22 02:49
Affects Version/s: 3.4.0

{{ExternalShuffleBlockResolver}} and {{YarnShuffleService}}  use {{{}LevelDB directly{}}}, this is not conducive to extending the use of {{RocksDB}} in this scenario. This pr is encapsulated for expansibility. It will be the pre-work of SPARK-38888
Bug: Clean sparkStaging dir should before unregister()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Apr/22 03:32
Affects Version/s: 3.2.1

{code:java}
      ShutdownHookManager.addShutdownHook(priority) { () =>
        try {
          val maxAppAttempts = client.getMaxRegAttempts(sparkConf, yarnConf)
          val isLastAttempt = appAttemptId.getAttemptId() >= maxAppAttempts

          if (!finished) {
            // The default state of ApplicationMaster is failed if it is invoked by shut down hook.
            // This behavior is different compared to 1.x version.
            // If user application is exited ahead of time by calling System.exit(N), here mark
            // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
            // System.exit(0) to terminate the application.
            finish(finalStatus,
              ApplicationMaster.EXIT_EARLY,
              "Shutdown hook called before final status was reported.")
          }

          if (!unregistered) {
            // we only want to unregister if we don't want the RM to retry
            if (finalStatus == FinalApplicationStatus.SUCCEEDED || isLastAttempt) {
              unregister(finalStatus, finalMsg)
              cleanupStagingDir(new Path(System.getenv("SPARK_YARN_STAGING_DIR")))
            }
          }
        } catch {
          case e: Throwable =>
            logWarning("Ignoring Exception while stopping ApplicationMaster from shutdown hook", e)
        }
      }{code}

unregister may throw exception, clean staging dir should before unregister.
Bug: Tasks not killed caused by race conditions between killTask() and launchTask()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Apr/22 16:33
Affects Version/s: 3.2.1

Sometimes when the scheduler tries to cancel a task right after it launches that task on the executor, the KillTask and LaunchTask events can come in a reversed order, causing the task to escape the kill-task signal and finish "secretly". And those tasks even show as an un-launched task in Spark UI.
Bug: Spark should fail fast if initial memory too large(set by "spark.executor.extraJavaOptions") for executor to start
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 20/Apr/22 01:43
Affects Version/s: 3.4.0

if you set initial memory(set by "spark.executor.extraJavaOptions=-Xms\{XXX}G" ) larger than maximum memory(set by "spark.executor.memory")

Eg.

     *spark.executor.memory=1G*

     *spark.executor.extraJavaOptions=-Xms2G*

 

from the driver process you just see executor failures with no warning, since the more meaningful errors are buried in the executor logs. 

Eg., on Yarn, you see:
{noformat}
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size{noformat}
Instead we should just fail fast with a clear error message in the driver logs.
Bug: Introduce 4 new functions to KVUtils
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 24/Apr/22 03:03
Affects Version/s: 3.4.0

Introduce 4 new functions
 * count：Counts the number of elements in the KVStoreView which satisfy a predicate.
 * foreach：Applies a function f to all values produced by KVStoreView.
 * mapToSeq：Maps all values of KVStoreView to new Seq using a transformation function.
 * size: The size of KVStoreView.

And use the above functions to simplify the code related to `KVStoreView`, and the above functions will release `LevelDB/RocksDBIterator` earlier.
Bug: [CVE-2020-9480] Transitive dependency  "unused"  from spark-sql_2.12  highlight as vulnerable in dependency tracker
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 26/Apr/22 07:04
Affects Version/s: 3.2.1

I am using spark-sql_2.12 dependency version 3.2.1 in my project. My dependency tracker highlights  the transitive dependency  "unused"  from  spark-sql_2.12 as vulnerable. I check there is no update for this artifacts since 2014. Is the artifact used anywhere in spark ?

To resolve this vulnerability,  can I exclude this "unused" artifact from spark-sql_2.12 ?  Will it cause any issues in my project ?
Bug: Test Bug in StatisticsSuite "change stats after add/drop partition command"
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Apr/22 00:36
Affects Version/s: 3.4.0

[https://github.com/apache/spark/blob/cbffc12f90e45d33e651e38cf886d7ab4bcf96da/sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala#L979]

It should be `partDir2` instead of `partDir1`. Looks like it is a copy paste bug.
Bug: Increase the performance of ExpressionSet
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Apr/22 17:49
Affects Version/s: 3.2.1

Since the constructor of `ExpressionSet` is protected, callers can only modify the state of an `ExpressionSet` by calling `add` and `remove`, so it maintains the invariants of:
 # Every expr `e` in `baseSet` satisfies `e.deterministic && e.canonicalized == e`
 # Every deterministic expr `e` in `originals` satisfies that `e.canonicalized` is already accessed.

The current implementation ignores the invariant and unnecessarily calls `.canonicalized` in lots of places which can cause performance issues as `canonicalized` is a pretty expensive operation.
Bug: Fix potential risk of JVM crash and RocksDBIterator resource leak
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Apr/22 05:31
Affects Version/s: 3.4.0

There is lock contention between `RocksDBIterator.finalize()` and `RockDB.close()` method. If `RocksDB.close()` method is executed first, the `RocksDBIterator`s hold by `Finalizer` will not close before the `RockDB` close.

If user use a `rocksdbjni` compile with `DEBUG_LEVEL=1` to enables all assertions of `rocksdbjni`(official `rocksdbjni` compile with `DEBUG_LEVEL=0` and diabled all assertions), above issue will lead to JVM crash.
Bug: Fast Fail the remaining push blocks if shuffle stage finalized
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Apr/22 13:09
Affects Version/s: 3.3.0

Map task will try to push all map outputs to external shuffle service now.

After the shuffle stage is finalized, the reduce fetch blocks RPC will be blocked if there are still many map output blocks in flight.

We could stop pushing the remaining blocks if the shuffle stage is finalized.
Bug:   Use `Collections.singletonList` instead of `Arrays.asList` when there is only one argument
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 28/Apr/22 10:07
Affects Version/s: 3.4.0

Use `Collections.singletonList` instead of `Arrays.asList` when there is only one argument.

 

before
{code:java}
List<String> one = Arrays.asList("one"); {code}
after
{code:java}
List<String> one = Collections.singletonList("one"); {code}
Bug: Allow user to insert specified columns into insertable view
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Apr/22 12:31
Affects Version/s: 3.2.1

The option `spark.sql.defaultColumn.useNullsForMissingDefautValues` allows us to insert specified columns into table (SPARK-38795), but currently this option does not work for insertable view, 

Below INSERT INTO will result in AnalysisException even when the useNullsForMissingDefautValues option is true,
{code:java}
spark.sql("CREATE TEMPORARY VIEW v1 (c1 int, c2 string) USING org.apache.spark.sql.json.DefaultSource OPTIONS ( path 'json_dir')");
spark.sql("INSERT INTO v1(c1) VALUES(100)");

org.apache.spark.sql.AnalysisException: unknown requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s).
{code}
 

I can provide a fix for this issue.
Bug: Use `withTempPath` to clean up temporary data directory after `SPARK-37463: read/write Timestamp ntz to Orc with different time zone`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/22 06:55
Affects Version/s: 3.3.0

"SPARK-37463: read/write Timestamp ntz to Orc with different time zone" create a dir named `ts_ntz_orc` under `sql/core/` directory and didn't clean it up after test.
Bug: Fix compilation warnings related to "`enum` will become a keyword in Scala 3"
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/22 02:40
Affects Version/s: 3.4.0

[WARNING] spark-source/core/src/test/scala/org/apache/spark/internal/config/ConfigEntrySuite.scala:172: [deprecation @  | origin= | version=2.13.7] Wrap `enum` in backticks to use it as an identifier, it will become a keyword in Scala 3.
[WARNING] spark-source/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:553: [deprecation @  | origin= | version=2.13.7] Wrap `enum` in backticks to use it as an identifier, it will become a keyword in Scala 3.
Bug: Close `NioBufferedFileInputStream` opened by `ExternalAppendOnlyUnsafeRowArraySuite`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Apr/22 08:57
Affects Version/s: 3.4.0

UT "test iterator invalidation (with spill)"  in `ExternalAppendOnlyUnsafeRowArraySuite`  is use to test `ConcurrentModificationException` scenarios,  this case will open 3 files wrapped by `NioBufferedFileInputStream` and the open files are not closed after this test.
Bug: Optimize RemoteBlockPushResolver with a memory pool
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Apr/22 08:24
Affects Version/s: 3.3.0

For push-based shuffle service, there are many {{BLOCK_APPEND_COLLISION_DETECTED}} when there are many small map tasks outputs. In {{{}RemoteBlockPushResolver{}}}, if one map task pushed blocks is writing, the others map tasks pushed blocks will failed in {{onComplete()}} method.
And {{RemoteBlockPushResolver}} has no memory limit , so many executors will OOM when there are many small pushed blocks waiting to be written to the final data file.
Bug: Return Array of Predicate for SupportsPushDownCatalystFilters.pushedFilters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Apr/22 16:07
Affects Version/s: 3.3.0

in SupportsPushDownCatalystFilters, change


{code:java}
def pushedFilters: Array[Filter]
{code}


to


{code:java}
def pushedFilters: Array[Predicate]
{code}
Bug: [SQL] Support <paramName> style for error message parameters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Apr/22 20:01
Affects Version/s: 3.3.0

In preparation of scraping error messages and turn them into docs we want to name the parameters used in error messages.
E.g
{quote}{{"CANNOT_CAST_DATATYPE" : {}}
{{"message" : [ "Cannot cast <sourceType> to <targetType>." ],}}
{{"sqlState" : "22005"}}{quote}
Bug: Turn spark.sql.ansi.strictIndexOperator into internal config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Apr/22 10:03
Affects Version/s: 3.3.0

Currently, all the ANSI error message shows the hint "If necessary set spark.sql.ansi.enabled to false to bypass this error." 

"Map key not exist" and "array index out of bound" errors are special. It shows the config spark.sql.ansi.strictIndexOperator instead.

This one special case can confuse users. To make it simple:
 * Show the configuration spark.sql.ansi.enabled instead
 * If it is "map key not exist" error, show the hint for using "try_element_at". Otherwise, we don't show it. For array, `[]` operator is using 0-based index while `try_element_at` is using 1-based index.
Bug: Upgrade Python 3.9 version in CI
Status: Open | Priority: Minor | Resolution: 
Created: 20/Apr/22 08:09
Affects Version/s: 3.2.1

Currently, the CI uses Python 3.9.5.

This caused issues while solving SPARK-38870.
We should probably upgrade the CI to a more recent version of Python 3.9 (latest version is 3.9.12)
Bug: Upgrade Jetty to 9.4.46
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Apr/22 09:21
Affects Version/s: 3.2.0

Upgrade Jetty version to 9.4.45.v20220203 in current Spark master in order to fix the below vulnerabilities
h4. 

CVE-2021-28169

CVE-2021-34428

CVE-2021-34429
Bug: PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Apr/22 18:44
Affects Version/s: 3.2.1

This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was merged as part of Spark 3.0.0

This change is desirable as it further exposes the metricsLevel config parameter that was added for the Scala/Java Spark APIs when working with the Kinesis Streaming integration, and makes it available to the PySpark API as well.

This change passes all tests, and local testing was done with a development Kinesis stream in AWS, in order to confirm that metrics were no longer being reported to CloudWatch after specifying MetricsLevel.NONE in the PySpark Kinesis streaming context creation, and also worked as it does today when leaving the MetricsLevel parameter out, which would result in a default of DETAILED, with CloudWatch metrics appearing again.

https://github.com/apache/spark/pull/36201
Bug: GroupState.getCurrentWatermarkMs() on static Dataframe throws exception but should return -1
Status: Open | Priority: Minor | Resolution: 
Created: 11/Apr/22 14:17
Affects Version/s: 3.1.2

As per documentation GroupState.getCurrentWatermarkMs() returns -1 when operating on a static Dataframe, as found here: [https://spark.apache.org/docs/3.1.3/api/scala/org/apache/spark/sql/streaming/GroupState.html#getCurrentWatermarkMs():Long]

 

In reality this function throws UnsupportedOperationException as documented when operating on a DataFrame without a watermark set.  This happens even if the static DataFrame had a (useless) watermark configured.
Bug: Log executed SQL queries
Status: Open | Priority: Minor | Resolution: 
Created: 15/Apr/22 15:18
Affects Version/s: 3.2.1

When Spark issues an SQL query to the database through JDBC, log (at debug or info level) this query in order to be able to grab it. This is useful to check the performance of this query, running a SQL explain plan with the appropriate tool.
As SQL queries are sometimes generated, it is sometimes hard to predict what query Spark will sent to the database.

Example: I would like to watch this query [https://github.com/apache/spark/blob/v3.2.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala#L352]
Bug: 'test 1 resource profile' throws exception when running it in IDEA separately
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Apr/22 03:58
Affects Version/s: 3.2.1

The test `test 1 resource profile` of DAGSchedulerSuite will fail if I run it in IDEA separately.
    
The root cause is the ResourceProfile is initialized before SparkContext, and it will take `DEFAULT_RESOURCE_PROFILE_ID` as the resource profile id. But the test asserts that the id is not equal  to DEFAULT_RESOURCE_PROFILE_ID.
    
{code:java}
assert(expectedid.get != ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID){code}

    
The exception is like below,
    
{code:java}
    0 equaled 0
    ScalaTestFailureLocation: org.apache.spark.scheduler.DAGSchedulerSuite at (DAGSchedulerSuite.scala:3269)
    org.scalatest.exceptions.TestFailedException: 0 equaled 0
            at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
            at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
            at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
            at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
            at org.apache.spark.scheduler.DAGSchedulerSuite.$anonfun$new$191(DAGSchedulerSuite.scala:3269)
            at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85){code}
This issue does not exist when running all DAGSchedulerSuite one by one, since the SparkContext will be initialized at the very beginning.

 

I will submit a patch to fix it.
Bug: Avoid OOM when bufferedPlan has a lot of duplicate keys in SortMergeJoin codegen
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Apr/22 03:54
Affects Version/s: 3.2.1

WholeStageCodegenExec is wrapped in BufferedRowIterator.

BufferedRowIterator uses a LinkedList to hold the output of WholeStageCodegenExec.

When the parent of SortMergeJoin cannot codegen, SortMergeJoin needs to append the output to this LinkedList.

SortMergeJoin processes a record in streamedPlan each time. If all records in bufferedPlan can match this record, all records in bufferedPlan will be saved in LinkedList, resulting in OOM.

The above situation is very common in our internal use, so it is best to add a configuration to the codegen code. If there are enough pieces in the LinkedList, stop SortMergeJoin and let the parent consume it first.
Bug: CLONE - Improve the test coverage for pyspark/find_spark_home.py
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 12/Apr/22 13:04
Affects Version/s: 3.4.0

We should test when the environment variables are not set (https://app.codecov.io/gh/apache/spark/blob/master/python/pyspark/find_spark_home.py)
Bug: Introduce a `viewToSeq` function for `KVUtils`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Apr/22 13:47
Affects Version/s: 3.4.0

There are many codes in spark that convert KVStoreView into `List`, and these codes will not close `KVStoreIterator`, these resources are mainly recycled by `finalize()` method implemented in `LevelDB` and `RockSB`, this makes `KVStoreIterator` resource recycling unpredictable.
Bug: Fix the docs of try_multiply/try_subtract/ANSI cast
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Apr/22 12:52
Affects Version/s: 3.3.0

* Fix the valid combinations of ANSI CAST. 
 * Fix the usage of try_multiply/try_subtract, from "expr1 _FUNC_ expr2" to "_FUNC_(expr1<  expr2)"
Bug: Migrate Junit4 to Junit5
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Apr/22 03:05
Affects Version/s: 3.4.0

JUnit5 is a powerful and flexible update to the JUnit framework, and it provides a variety of improvements and new features to organize and
describe test cases, as well as help in understanding test results：
 # JUnit 5 leverages features from Java 8 or later, such as lambda functions, making tests more powerful and easier to maintain, but Junit 4 still a Java 7 compatible version
 # JUnit 5 has added some useful new features for describing, organizing, and executing tests. For examples: [Parameterized Tests|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests] and [Conditional Test Execution|https://junit.org/junit5/docs/current/user-guide/#extensions-conditions] may make our test code look simpler, [Parallel Execution|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution] may make our test faster
Bug: Fix scala license declaration
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Apr/22 03:58
Affects Version/s: 3.1.2

ASF License v2

[https://www.scala-lang.org/download/2.13.8.html]

[https://www.scala-lang.org/download/2.12.15.html] 

[3-clause BSD license|https://opensource.org/licenses/BSD-3-Clause]

[https://www.scala-lang.org/download/2.11.12.html]