av9ash/GitBugs huggingface dataset:
Bug: Add style rules to prohibit to use a Guava's API which is incompatible with newer versions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jun/21 07:05
Affects Version/s: 3.2.0

SPARK-30272 replaced Objects.toStringHelper which is an APIs Guava 14 provides with commons.lang3's API because Objects.toStringHelper is no longer present in newer Guava.
But toStringHelper was introduced into Spark again and replaced them in SPARK-35420.
I think it's better to have a style rule to avoid such repetition.
Bug: Fix condition check for FULL OUTER sort merge join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jun/21 01:17
Affects Version/s: 3.2.0

The condition check for FULL OUTER sort merge join ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L1368] ) has unnecessary trip when `leftIndex == leftMatches.size` or `rightIndex == rightMatches.size`. Though this does not affect correctness (`scanNextInBuffered()` returns false anyway).
Bug: Set a default value for spark.kubernetes.test.sparkTgz in pom.xml for Kubernetes integration tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/21 04:37
Affects Version/s: 3.2.0

In kubernetes/integration-tests/pom.xml, there is no default value for spark.kubernetes.test.sparkTgz so running tests with the following command will fail.

{code}
build/mvn -Dspark.kubernetes.test.namespace=default -Pkubernetes -Pkubernetes-integration-tests -Psparkr  -pl resource-managers/kubernetes/integration-tests integration-test
{code}
+ mkdir -p /home/kou/work/oss/spark/resource-managers/kubernetes/integration-tests/target/spark-dist-unpacked
+ tar -xzvf --test-exclude-tags --strip-components=1 -C /home/kou/work/oss/spark/resource-managers/kubernetes/integration-tests/target/spark-dist-unpacked
tar (child): --test-exclude-tags: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
[ERROR] Command execution failed.
{code}
Bug: Spark Package Python Import does not work for namespace packages
Status: Resolved | Priority: Trivial | Resolution: Not A Problem
Created: 16/Jun/21 17:31
Affects Version/s: 3.0.0

If one includes python files within several jars that comprise a python "namespace package"
[https://www.python.org/dev/peps/pep-0420/]

Then only one of packages is imported
Bug: Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Jun/21 15:37
Affects Version/s: 3.1.1

When trying to follow steps in documentation about building Spark with a specific Hadoop version, this seems to fail with the following compiler error.
{code:java}
 [INFO] Compiling 560 Scala sources and 99 Java sources to /Users/puigcalvachef/Documents/os/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] /Users/puigcalvachef/Documents/os/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] /Users/puigcalvachef/Documents/os/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] /Users/puigcalvachef/Documents/os/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] /Users/puigcalvachef/Documents/os/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] four errors found
{code}
The minimal build to reproduce it would be just by building spark core module using:
{code:bash}
./build/mvn -Dhadoop.version=2.7.4 -pl :spark-core_2.12  -DskipTests clean install {code}
After some testing, it seems that something changed between 3.1.1-rc1 and 3.1.1-rc2 that made the feature start failing.

I tried a few versions of  Hadoop that fail: 2.7.4, 2.8.1, 2.8.5 
But it was successful when using Hadoop 3.0.0.
Bug: Add plan check for stream-stream join unit test
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 09/Jun/21 07:15
Affects Version/s: 3.2.0

The unit test of [https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala#L566] was introduce in [https://github.com/apache/spark/pull/21587,] to fix the planner side of thing for stream-stream join. Ideally check the query result should capture the bug, but it would be better to add plan check to make the purpose of unit test more clearly and capture future bug from planner change.
Bug: Instead of File#mkdirs, Files#createDirectories is expected.
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/Jun/21 15:04
Affects Version/s: 3.1.2

I noticed a little problem when I was studying spark source code.

The following is an excerpt from the comments of the class: org.apache.spark.util.Utils and method: createDirectory
{code:java}
//This sporadically fails - not sure why ... 
{code}
Not sure why? I'm sure it's improper.

Although File#mkdirs can create files, they are very unfriendly in exception handling.

Files#createDirectories in JDK7 is better.

Most importantly, in order to ensure correctness, I also added several targeted test cases.
Bug: Create hive permanent function with owner name
Status: In Progress | Priority: Trivial | Resolution: 
Created: 28/Jun/21 04:54
Affects Version/s: 3.1.2

Now create a hive permanent function, no owner name, null value

 
{code:java}
private def toHiveFunction(f: CatalogFunction, db: String): HiveFunction = {
  val resourceUris = f.resources.map { resource =>
    new ResourceUri(ResourceType.valueOf(
      resource.resourceType.resourceType.toUpperCase(Locale.ROOT)), resource.uri)
  }
  new HiveFunction(
    f.identifier.funcName,
    db,
    f.className,
    null,
    PrincipalType.USER,
    (System.currentTimeMillis / 1000).toInt,
    FunctionType.JAVA,
    resourceUris.asJava)
}
{code}
Bug: Fix documentation error in Spark SQL Guide - Getting Started
Status: Closed | Priority: Trivial | Resolution: Fixed
Created: 27/Jun/21 12:19
Affects Version/s: 3.1.2

The hyperlinks in Python code blocks in [https://spark.apache.org/docs/latest/sql-getting-started.html|https://spark.apache.org/docs/latest/sql-getting-started.html)] currently point to invalid addresses and return 404.
Bug: Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Jun/21 06:02
Affects Version/s: 3.1.0

The test suite for ShuffleBlockPusherSuite was added with SPARK-32917 and in this suite, the configuration for push-based shuffle is incorrectly referenced as {{spark.shuffle.push.based.enabled}}. We need to remove this config from here.

{{ShuffleBlockPusher}} is created only when push based shuffle is enabled and this suite is for {{ShuffleBlockPusher}}, so no other change is required.
Bug: Add bitwise AND operation to BitArray and add intersect AND operation for bloom filters
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Jun/21 14:47
Affects Version/s: 3.1.2

This issue will add
 # a bitwise AND operation to BitArray (similar to existing `putAll` method)
 # an intersect operation for combining bloom filters using bitwise AND operation (similar to existing `mergeInPlace` method).
Bug: Minor refactor of StateStoreCustomMetric
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Jun/21 06:04
Affects Version/s: 3.1.2

Code in {{[SymmetricHashJoinStateManager|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala#L321]}} relies on the subclass implementations of [StateStoreCustomMetric|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala#L187].

If a new subclass of {{StateStoreCustomMetric}} is added, it requires code changes to {{SymmetricHashJoinStateManager}} and we may miss the update if there is no existing test coverage.

To fix this, add new method to trait {{withNewDesc(desc : String)}} to {{StateStoreCustomMetric}} and use it in {{SymmetricHashJoinStateManager}}
Bug: Remove links to Nomad integration in the Documentation
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Jun/21 08:28
Affects Version/s: 3.1.2

In Project documentation `docs/cluster-overview.md` there is still active reference to the [Nomad integration project by Hashicorp|https://github.com/hashicorp/nomad-spark]. 

As per the repository project is no longer active. The task is to remove the reference from the Documentation.
Bug: Remove duplicate installations in build_and_test.yml
Status: Resolved | Priority: Trivial | Resolution: Not A Problem
Created: 01/Jun/21 12:58
Affects Version/s: 3.1.2

There are many duplicate installations of the python package in [https://github.com/apache/spark/blob/a59063d5446a78d713c19a0d43f86c9f72ffd77d/.github/workflows/build_and_test.yml#L360-L386,] so we could remove unnecessary installations.
Bug: Spark Cluster Crashing
Status: Open | Priority: Critical | Resolution: 
Created: 12/Jan/24 15:38
Affects Version/s: 3.4.0

h3. Description:
 * We have a spark cluster installed over a k8s cluster with one driver and multiple executors (120).
 * We configure our batch duration to 30 seconds.
 * The Spark Cluster is reading from a 120 partition topic at Kafka and writing to an hourly index at ElasticSearch.
 * ES has 30 DataNodes, 1 shard per DataNode for each index.
 * Configuration of Driver STS is in Appendix.
 * The driver is observed periodically restarting every 10 mins, although the restart do not necessarily occur each 10mins, but when it happens it happens each 10 mins.
 * The restarts frequency increase with the increase of the throughput.
 * When the restarts are happening, we see OptionalDataException, attached “logs_cveshv-events-streaming-core-cp-type2-filter-driver-0_051023_1500_prev.log” is the log resulting in a restart of the driver.

h3. Analysis:
 # We’ve done a test with 250 K Records/second, and the processing was good between 15 and 20 seconds.
 # We were able to avoid all the restarts by simply disabling liveness checks.
 # This resulted in NO RESTARTS to Streaming Core, we tried the above with two scenarios:

 * Speculation Disabled --> After 10 to 20 minutes the batch duration increased to minutes and eventually processing was very slow, during which, main error logs observed are about {*}The executor with id 7 exited with exit code 50(Uncaught exception).{*}, logs at WARN level and TRACE level were collected:

 * {*}WARN{*}: Logs attached “cveshv-events-streaming-core-cp-type2-filter-driver-0_liveness_300000_failed_120124_0336_2.log”
 * {*}TRACE{*}: Logs attached “cveshv-events-streaming-TRACE (2).zip”

 * Speculation Enabled -->  the batch duration increased to minutes (big lag) only after around 2 hours, logs related are “cveshv-events-streaming-core-cp-type2-filter-driver-0_zrdm71bnrt201_1201240818.log”.

h3. Conclusion:
 * The liveness check is failing and thus causing the restarts.
 * The logs indicates that there are some unhandled exceptions to executors.
 * Issue can be somewhere else as well, below is the liveness check that was disabled and that was causing the restarts initially every 10 mins after 3 occurrences.

 
h3. !image-2024-01-12-10-44-45-717.png!
h3. Next Action:
 * Please help us identify the RC of the issue, we’ve tried too many configurations and with 2 different spark versions 3.4 and 3.5 and we’re not able to avoid the issue.

 

Appendix:

 

!image-2024-01-12-10-45-18-905.png!

!image-2024-01-12-10-45-30-398.png!

!image-2024-01-12-10-45-40-397.png!

!image-2024-01-12-10-45-50-427.png!
Bug: Data is silently lost in Tab separated CSV with empty (whitespace) rows
Status: Open | Priority: Critical | Resolution: 
Created: 26/Jan/24 10:31
Affects Version/s: 3.4.1

When reading a tab separated file that contains lines that only contain tabs (i.e. empty strings as values of the columns for that row), then these rows will silently be skipped (as empty lines) and the resulting dataframe will have less rows than expected.

This behavior is inconsistent with the behavior for e.g. semicolon separated files, where the resulting dataframe will have a row with only empty string values.

A minimal reproducible example would look like:

A minimal reproducible example: A file containing this
{code:java}
a\tb\tc\r\n
\t\t\r\n
1\t2\t3{code}
will create a dataframe with one row (a=1,b=2,c=3)
whereas this
{code:java}
a;b;c\r\n
;;\r\n
1;2;3{code}
will read as two rows (first row contains empty strings)

I used the following pyspark command to read the dataframes
{code:java}
 spark.read.option("header","true").option("sep","\t").csv("<tabseparated file>").collect()
spark.read.option("header","true").option("sep",";").csv("<semicolon file>").collect()
{code}
I ran into this particularly on databricks (I assume they use the same reader), but [this stack overflow post|https://stackoverflow.com/questions/47823858/replacing-empty-lines-with-characters-when-reading-csv-using-spark#comment137288546_47823858] indicates, that this is an old issue that may have been taken over from databricks when their csv reader was adopted in SPARK-12420

I recommend to at least add a respective test case to the CSV reader.

 

Why is this behaviour a problem:
 * It violates some of the core assumptions
 ** a properly configured roundtrip via csv write/read should result in the same set of rows
 ** changing the csv separator (when everything is properly esacped) should have no effect

Potential resolutions:
 * When the configured delimiter consists of only whitespace
 ** deactivate the "skip empty line feature"
 ** or skip only lines that are completely empty (only a (carriage return) newline)
 * Change the skip empty line feature to only skip if the line is completely empty (only contains a newlin)
 ** this may break some user code that relies on the current behaviour
Bug: Spark Cluster Crashing 
Status: Open | Priority: Critical | Resolution: 
Created: 12/Jan/24 15:09
Affects Version/s: 3.4.0

I am getting these errors in the spark executors:


2024-01-12 03:26:17.887 WARN     [task-result-getter-2]:org.apache.spark.internal.Logging - Lost task 65.0 in stage 79.0 (TID 10250) (10.1.208.60 executor 119): TaskKilled (Stage cancelled: Job aborted due to stage failure: Exception while getting task result: java.io.OptionalDataException)
2024-01-12 03:26:17.891 WARN     [task-result-getter-3]:org.apache.spark.internal.Logging - Lost task 69.0 in stage 79.0 (TID 10263) (10.1.99.211 executor 72): TaskKilled (Stage cancelled: Job aborted due to stage failure: Exception while getting task result: java.io.OptionalDataException)
2024-01-12 03:26:17.893 WARN     [task-result-getter-0]:org.apache.spark.internal.Logging - Lost task 115.0 in stage 79.0 (TID 10202) (10.1.236.96 executor 27): TaskKilled (Stage cancelled: Job aborted due to stage failure: Exception while getting task result: java.io.OptionalDataException)
2024-01-12 03:26:17.895 WARN     [task-result-getter-1]:org.apache.spark.internal.Logging - Lost task 4.0 in stage 79.0 (TID 10231) (10.1.165.84 executor 80): TaskKilled (Stage cancelled: Job aborted due to stage failure: Exception while getting task result: java.io.OptionalDataException)
2024-01-12 03:26:17.897 WARN     [task-result-getter-2]:org.apache.spark.internal.Logging - Lost task 75.0 in stage 79.0 (TID 10228) (10.1.6.211 executor 18): TaskKilled (Stage cancelled: Job aborted due to stage failure: Exception while getting task result: java.io.OptionalDataException)
2024-01-12 03:26:17.902 WARN     [task-result-getter-3]:org.apache.spark.internal.Logging - Lost task 102.0 in stage 79.0 (TID 10285) (10.1.160.108 executor 53): TaskKilled (Stage cancelled: Job aborted due to stage failure: Exception while getting task result: java.io.OptionalDataException)
2024-01-12 03:27:13.092 ERROR    [dispatcher-CoarseGrainedScheduler]:org.apache.spark.internal.Logging - Lost executor 117 on 10.1.197.197: 
The executor with id 117 exited with exit code 50(Uncaught exception).
 
 
 
The API gave the following container statuses:
 
 
container name: spark-kubernetes-executor
container image: ngxp-registry.service.lab.ngxp.cci.att.com:5000/nova/midlayer/midlayer-streaming-core:1.7.1
container state: terminated
container started at: 2024-01-12T03:03:46Z
container finished at: 2024-01-12T03:27:12Z
exit code: 50
termination reason: Error
      
2024-01-12 03:27:13.095 WARN     [dispatcher-CoarseGrainedScheduler]:org.apache.spark.internal.Logging - Lost task 79.0 in stage 79.0 (TID 10305) (10.1.197.197 executor 117): ExecutorLostFailure (executor 117 exited caused by one of the running tasks) Reason: 
The executor with id 117 exited with exit code 50(Uncaught exception).
Bug: Structured Streaming - Arbitrary State API v2
Status: Open | Priority: Major | Resolution: 
Created: 23/Jan/24 18:34
Affects Version/s: 4.0.0

We are planning to introduce a new operator for [Spark Structured Streaming|https://spark.apache.org/streaming/] around [arbitrary stateful operations|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#arbitrary-stateful-operations]. The operator(s) we have today ([mapGroupsWithState/flatMapGroupsWithState|https://spark.apache.org/docs/3.1.3/api/java/org/apache/spark/sql/streaming/GroupState.html]) have a bunch of limitations (as noted in Q3 below) that prevent the use of the APIs for complex/advanced use-cases for stateful stream processing. We plan to remove these limitations with the introduction of a new API to handle state manipulation for streaming queries.

 

As part of this effort, we want to address:
 * composite types

 * multiple state variables (flexible data modeling)

 * state TTL

 * callback/timer improvements

 * schema evolution

 * initial state

 * output modes

 * co-process functions

 * chaining operators after new operator etc
Bug: Introducing collation concept into Spark
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/24 17:10
Affects Version/s: 4.0.0

This feature will introduce collation support to the Spark engine. This means that:

 
 # Every StringType will have an associated collation. Default remains UTF8 Binary, which will behave under the same rules as current UTF8 String comparison.
 # Collation will be respected in all collation sensitive operations - comparisons, hashing, string operations (contains, startWith, endsWith etc.)
 # Collation can be set through following ways:
 ## COLLATE expression. e.g. strExpr COLLATE collation_name
 ## In CREATE TABLE column definition
 ## By setting session collation.
 # All the Spark operators need to respect collation settings (filters, joins, shuffles, aggs etc.)

 

This is a high level description of the feature. You can find detailed design under [this|https://docs.google.com/document/d/1A9RQiwq-n3R3vuh571yjOLaaIuIYRTyCx7UFr0Qg-eY/edit?usp=sharing] link (doc is in attachment as well).
Bug: Fix error message regression by restoring new_msg
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/24 00:45
Affects Version/s: 4.0.0

>>> from pyspark.sql.types import StructType, StructField, StringType, IntegerType
>>> schema = StructType([
...     StructField("name", StringType(), nullable=True),
...     StructField("age", IntegerType(), nullable=False)
... ])
>>> df = spark.createDataFrame([("asd", None])], schema)
pyspark.errors.exceptions.base.PySparkValueError: [CANNOT_BE_NONE] Argument `obj` cannot be None.
Bug: Avoid Deadlock in DAGScheduler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/24 13:34
Affects Version/s: 3.4.3

* The DAGScheduler could currently run into a deadlock with another thread if both access the partitions of the same RDD at the same time.
 * To make progress in getCacheLocs, we require both exclusive access to the RDD partitions and the location cache. We first lock on the location cache, and then on the RDD.
 * When accessing partitions of an RDD, the RDD first acquires exclusive access on the partitions, and then might acquire exclusive access on the location cache.
 * If thread 1 is able to acquire access on the RDD, while thread 2 holds the access to the location cache, we can run into a deadlock situation.
Bug: Kafka custom partition location assignment in Spark Structured Streaming (rack awareness)
Status: Open | Priority: Major | Resolution: 
Created: 22/Jan/24 23:53
Affects Version/s: 3.1.0

I'd like to propose, and implement if approved, support for custom partition location assignment. [Please find the design doc for SPARK-46798 describing the change here.|https://docs.google.com/document/d/1RoEk_mt8AUh9sTQZ1NfzIuuYKf1zx6BP1K3IlJ2b8iM/edit#heading=h.pbt6pdb2jt5c]

SPARK-15406 Added Kafka consumer support to Spark Structured Streaming, but it did not add custom partition location assignment as a feature. The Structured Streaming Kafka consumer as it exists today evenly allocates Kafka topic partitions to executors without regard to Kafka broker rack information or executor location. This behavior can drive large cross-AZ networking costs in large deployments.

[The design doc for SPARK-15406|https://docs.google.com/document/d/19t2rWe51x7tq2e5AOfrsM9qb8_m7BRuv9fel9i0PqR8/edit#heading=h.k36c6oyz89xw] described the ability to assign Kafka partitions to particular executors (a feature which would enable rack awareness), but it seems that feature was never implemented.

For DStreams users, there does seem to be a way to assign Kafka partitions to Spark executors in a custom fashion with [LocationStrategies.PreferFixed|https://github.com/apache/spark/blob/master/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/LocationStrategy.scala#L69], so this sort of functionality has a precedent.
Bug: Write a subquery generator that generates subqueries of different variations to increase testing coverage in this area
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/24 22:02
Affects Version/s: 3.5.1

There are a lot of subquery correctness issues, ranging from very old bugs to new ones that are being introduced due to work being done on subquery correlation optimization. This is especially in the areas of COUNT bugs and null behaviors.

To increase test coverage and robustness in this area, we want to write a subquery generator that writes variations of subqueries, producing SQL tests that also run against Postgres (from my work in SPARK-46179).
Bug: Spark-submit in cluster mode with standalone cluster uses wrong JAVA_HOME path
Status: Open | Priority: Major | Resolution: 
Created: 30/Jan/24 02:44
Affects Version/s: 3.5.0

When run spark submit to a standalone cluster using cluster mode, the worker machine will use the JAVA_HOME value from remote machine instead of from worker machine.

To reproduce:
 * Create a standalone cluster using docker compose, set JAVA_HOME in each worker different from local machine.
 * Run spark-submit, deploy-mode cluster
 * Monitor the log from worker, the driver will print out: DriverRunner: Launch Command: "<value from local JAVA_HOME>" "-cp" ...

Reason:

When Master create a new driver in receiveAndReply method, it uses the environment variables from submitter to build the driver description command. After that, when launch the driver, a new local (of worker) is built but it still use environment variable from driver description (which came from submitter). The result is the building java command will use the submitter java home path instead of worker path.

Suggestion:

Replace JAVA_HOME and SPARK_HOME in buildLocalCommand method of 
org.apache.spark.deploy.worker.CommandUtils by worker value