av9ash/GitBugs huggingface dataset:
Bug: Move SessionHolder.forTesting(...) into the test package
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jul/24 11:49
Affects Version/s: 4.0.0

The `SessionHolder.forTesting(...)` method is widely used in several Spark Connect testing suites. However, this test-specific code is located in the source code, and incorrect use may lead to the creation of a partial/buggy Spark Connect session during production runtime.

 

This ticket proposes to isolate this piece of code within the test package to avoid accidental usage during runtime.
Bug: Provide modifiedAfter and modifiedBefore options when filtering from a stream source
Status: Open | Priority: Minor | Resolution: 
Created: 29/Jul/24 21:43
Affects Version/s: 3.5.1

In the following Jira issue
https://issues.apache.org/jira/browse/SPARK-31962



Two new options, *modifiiedBefore* and *modifiedAfter* for batch reads (for example, CSV) where introduced, and eventually merged into version 3.1.1 via PR:

https://issues.apache.org/jira/browse/SPARK-31962

 

This was introduced in a way that batch reads allow these two options, however a stream is explicitly not allowed.



When loading files from a data source as a stream, there too can be times where thousands of files are within a respective file path. This applies to both batch and stream use cases.  Note:  The Databricks "cloudFiles" AutoLoader supports these options in a stream.  

[https://docs.databricks.com/en/ingestion/auto-loader/options.html#id20]

 

{{*Suggested Example Usages*}}
{{_Start stream with all CSV files modified after date:_}}
{{spark.readStream.option("modifiedAfter","2020-06-15T05:00:00").option("quote", '"').option("escape", '"').csv(source_path)}}

{{_Start Stream with all CSV files modified before date:_}}
{{spark.readStream.option("modifiedAfter","2020-06-15T05:00:00").option("quote", '"').option("escape", '"').csv(source_path)}}

_Start stream with all CSV files modified between two dates:_

{{spark.readStream.option("modifiedAfter","2019-06-15T05:00:00").{{{}option("modifiedBefore","2020-06-15T05:00:00"){}}}option("quote", '"').option("escape", '"').csv(source_path)}}
Bug: Add a config to add ability to only enable speculative execution on non-data-writing tasks.
Status: Open | Priority: Minor | Resolution: 
Created: 25/Jul/24 09:24
Affects Version/s: 3.3.2

Speculative Execution is a very good tool to avoid straggler tasks, but if we use Speculative Execution with some customized BatchWrite, if there are bugs in user implemented BatchWrite, it will produce wrong results.

 

So this ticket propose to add a config to add the ability to only enable speculative execution on non-data-writing tasks, which might be more safe.
Bug: applyInPandas does not respect streaming watermark
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jul/24 15:02
Affects Version/s: 3.5.0

When I use GroupedData.applyInPandas to implement aggregation in a streaming query, it fails to respect a watermark specified using DataFrame.withWatermark.

This query reproduces the behaviour I'm seeing:
 
{code:python}
from pyspark.sql.functions import window
from typing import Tuple
import pandas as pd

df_source_stream = (
    spark.readStream
    .format("rate")
    .option("rowsPerSecond", 3)
    .load()
    .withColumn("bucket", window("timestamp", "10 seconds").end)
)

def my_function(
    key: Tuple[str], df: pd.DataFrame
) -> pd.DataFrame:
    return pd.DataFrame({"bucket": [key[0]], "count": [df.shape[0]]})

df = (
    df_source_stream
    .withWatermark("bucket", "10 seconds")
    .groupBy("bucket")
    .applyInPandas(my_function, "bucket TIMESTAMP, count INT")
)
display(df)
{code}
I expect the output of the query to contain one row per {{bucket}} value, but a new row is emitted for each incoming microbatch.

In contrast, an out of the box aggregate behaves as expected. For example:
{code:python}
df = (
    df_source_stream
    .withWatermark("bucket", "10 seconds")
    .groupBy("bucket")
    .count()  # standard aggregate in place of applyInPandas
)
{code}
The output of this query contains *one* row per {{bucket}} value.
Bug: Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jul/24 17:38
Affects Version/s: 4.0.0

*Summary:* 
This change proposes to log a warning message when the {{.unpersist()}} method is called on RDDs that have been locally checkpointed in Apache Spark. This aims to inform users about the potential risks of unpersisting such RDDs without altering the existing behavior of the method.

*Background:*
 Local checkpointing in Spark truncates the lineage of an RDD, meaning that the RDD cannot be recomputed from its source. If an RDD that has been locally checkpointed is unpersisted, it loses its data and cannot be regenerated. This can lead to job failures if subsequent actions or transformations are attempted on the unpersisted RDD.

*Proposed Change:* 
To mitigate this issue, a warning message will be logged whenever {{.unpersist()}} is called on a locally checkpointed RDD. This approach maintains the current functionality while alerting users to the potential consequences of their actions. This change is intended to be non-disruptive and is a step towards better user awareness and debugging.
Bug: Introduce interface to ensure extra strategy do planning of scan plan (with additional filters and projections)
Status: Open | Priority: Minor | Resolution: 
Created: 17/Jul/24 13:33
Affects Version/s: 4.0.0

If we have some plan that contains scan with filter (or project) as parent, it can happen we want to do planning of scan using extra strategies instead of DataSourceV2Strategy.

One of use-cases:
Snowflake and BigQuery connectors have their own strategies and we want to prohibit DataSourceV2Strategy to do planning of scan node.

Despite the fact extra strategies have priority, it can happen that strategy fails to plan 
Filter->Relation, but it can plan only Relation without its parent. In that case, DataSourceV2Strategy will jump in and it will be able to plan Filter->Relation in just one pass.

So we want to have ability to prevent that for certain Scan classes
Bug: Fix `spark-operator` module to define test framework explicitly
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jul/24 23:18
Affects Version/s: kubernetes-operator-0.1.0

[https://docs.gradle.org/8.9/userguide/upgrading_version_8.html#test_framework_implementation_dependencies]

 
{code:java}
$ ./gradlew clean test --warning-mode=fail

...

> Task :spark-operator:test
The automatic loading of test framework implementation dependencies has been deprecated. This is scheduled to be removed in Gradle 9.0. Declare the desired test framework directly on the test suite or explicitly declare the test framework implementation dependencies on the test's runtime classpath. Consult the upgrading guide for further information: https://docs.gradle.org/8.9/userguide/upgrading_version_8.html#test_framework_implementation_dependencies
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appendedFAILURE: Build failed with an exception.* What went wrong:
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.BUILD FAILED in 6s
16 actionable tasks: 16 executed {code}
Bug: Use `BasePluginExtension` in `spark-operator/build.gradle`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jul/24 23:01
Affects Version/s: kubernetes-operator-0.1.0

{code}
$ ./gradlew test --warning-mode all

> Configure project :spark-operator
Build file '/Users/dongjoon/APACHE/spark-kubernetes-operator/spark-operator/build.gradle': line 58
The org.gradle.api.plugins.BasePluginConvention type has been deprecated. This is scheduled to be removed in Gradle 9.0. Consult the upgrading guide for further information: https://docs.gradle.org/8.7/userguide/upgrading_version_8.html#base_convention_deprecation
        at build_1pl46tz9v8uzxd6mxgrewtcjs.run(/Users/dongjoon/APACHE/spark-kubernetes-operator/spark-operator/build.gradle:58)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)
Build file '/Users/dongjoon/APACHE/spark-kubernetes-operator/spark-operator/build.gradle': line 58
The BasePluginExtension.archivesBaseName property has been deprecated. This is scheduled to be removed in Gradle 9.0. Please use the archivesName property instead. For more information, please refer to https://docs.gradle.org/8.7/dsl/org.gradle.api.plugins.BasePluginExtension.html#org.gradle.api.plugins.BasePluginExtension:archivesName in the Gradle documentation.
        at build_1pl46tz9v8uzxd6mxgrewtcjs.run(/Users/dongjoon/APACHE/spark-kubernetes-operator/spark-operator/build.gradle:58)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)
{code}
Bug: Pivoting watermarked streaming dataframe fails with correctness error
Status: Open | Priority: Minor | Resolution: 
Created: 11/Jul/24 08:55
Affects Version/s: 3.5.0

When I run this query:
{code:python}
from pyspark.sql.functions import col, first

df_src = (
    spark.readStream
    .format("rate")
    .option("rowsPerSecond", 3)
    .load()
    .withColumn("text_value", col("value").cast("string"))
    .withColumn("label", col("text_value"))
)

df_pivot = (
    df_src
    .withWatermark("timestamp", "10 minutes")
    .groupBy("label")
    .pivot("value", ["x", "y"])
    .agg(first("value"))
)
display(df_pivot)
{code}
 
it fails with error {_}AnalysisException: Detected pattern of possible 'correctness' issue due to global watermark{_}.

This happens because the analyzer [resolves the pivot into two aggregate operations|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L817]. Multiple aggregations in a watermarked streaming pipeline [cause this error to be thrown|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala#L123] to prevent correctness issues caused by stateful operations using a single global watermark.

I've raised this as a bug because of the pivot failure, but please advise if that's inappropriate! Some observations:
 * We could say that pivot is not supported in a watermarked streaming pipeline. That would be disappointing, and it's not always true (depending on the datatype of the column being pivoted - for example, pivoting a string column causes the pivot to be implemented as a one-step aggregate. You can see this in my example by replacing {{first("value")}} with {{first("text_value")}}).
 * I don't understand the internals well enough to know whether the two-step aggregate is vulnerable to global watermark correctness issues at all. If not, it would seem to me that the unsupported operation check is too aggressive - it would need to be able to distinguish between the general multiple stateful operation problem and the adjacent aggregates implemented by the analyzer for pivot.
 * If the two-step aggregate for pivot is safe, one can disable the correctness check with {{{}spark.conf.set("spark.sql.streaming.statefulOperator.checkCorrectness.enabled", "false"){}}}, but that affects all operations - in principle a query could contain pivot operations *and* multiple operations vulnerable to global watermark correctness.

Many thanks
Bug: FileStreamSource: maxCachedFiles set to 0 causes empty batches
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jul/24 01:35
Affects Version/s: 4.0.0

While testing changes from https://issues.apache.org/jira/browse/SPARK-44924 it was discovered that setting {{maxCachedFiles}} to 0 to try to force each batch to perform a full listing was inadvertently causing every other batch to trigger with no files being processed.  

In this case, the list of {{unselectedFiles}} from a file listing gets 0 files taken from it and assigned to {{unreadFiles}} as an empty list, which is re-used in the next batch and only checked if {{unreadFiles}} is null.
Bug: Pass actual exception when url_decode fails
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jul/24 09:18
Affects Version/s: 3.5.1

Currently url_decode function ignores actual exception, which contains information that is useful for quickly locating the problem.

 

Like executing this sql:
{code:java}
select url_decode('https%3A%2F%2spark.apache.org'); {code}
We only get the error message:
{code:java}
org.apache.spark.SparkIllegalArgumentException: [CANNOT_DECODE_URL] The provided URL cannot be decoded: https%3A%2F%2spark.apache.org. Please ensure that the URL is properly formatted and try again.
    at org.apache.spark.sql.errors.QueryExecutionErrors$.illegalUrlError(QueryExecutionErrors.scala:376)
    at org.apache.spark.sql.catalyst.expressions.UrlCodec$.decode(urlExpressions.scala:118)
    at org.apache.spark.sql.catalyst.expressions.UrlCodec.decode(urlExpressions.scala) {code}
However, the actual useful exception information is ignored:
{code:java}
java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "2s" {code}
Bug: Improve error message in ValidateExternalType
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Jul/24 16:24
Affects Version/s: 4.0.0

When we have mixed schema rows error message "\{actual} is not a valid external type for schema of \{expected}" that don't help to understand column with problem. I suggest to add information about source column.
h2. How to reproduce
{code:java}
class ErrorMsgSuite extends AnyFunSuite with SharedSparkContext {
  test("shouldThrowSchemaError") {
    val seq: Seq[Row] = Seq(
      Row(
        toBytes("0"),
        toBytes(""),
        1L,
      ),
      Row(
        toBytes("0"),
        toBytes(""),
        1L,
      ),
    )    val schema: StructType = new StructType()
      .add("f1", BinaryType)
      .add("f3", StringType)
      .add("f2", LongType)    val df = sqlContext.createDataFrame(sqlContext.sparkContext.parallelize(seq), schema)    val exception = intercept[RuntimeException] {
      df.show()
    }    assert(
      exception.getCause.getMessage
        .contains("[B is not a valid external type for schema of string")
    )
    assertResult(
      "[B is not a valid external type for schema of string"
    )(exception.getCause.getMessage)
  }  def toBytes(x: String): Array[Byte] = x.toCharArray.map(_.toByte)
} {code}
After fix error message may contain extra info
{code:java}
[B is not a valid external type for schema of string at getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, f3) {code}
Code: [https://github.com/mrk-andreev/example-spark-schema/blob/main/spark_4.0.0/src/test/scala/ErrorMsgSuite.scala]
Bug: Improve/recover scheduled jobs in GitHub Actions
Status: Resolved | Priority: Blocker | Resolution: Done
Created: 20/Jun/22 00:52
Affects Version/s: 3.4.0

There are five problems to address.

*First*, the scheduled jobs are broken as below:

https://github.com/apache/spark/actions/runs/2513261706
https://github.com/apache/spark/actions/runs/2512750310
https://github.com/apache/spark/actions/runs/2509238648
https://github.com/apache/spark/actions/runs/2508246903
https://github.com/apache/spark/actions/runs/2507327914
https://github.com/apache/spark/actions/runs/2506654808
https://github.com/apache/spark/actions/runs/2506143939
https://github.com/apache/spark/actions/runs/2502449498
https://github.com/apache/spark/actions/runs/2501400490
https://github.com/apache/spark/actions/runs/2500407628
https://github.com/apache/spark/actions/runs/2499722093
https://github.com/apache/spark/actions/runs/2499196539
https://github.com/apache/spark/actions/runs/2496544415
https://github.com/apache/spark/actions/runs/2495444227
https://github.com/apache/spark/actions/runs/2493402272
https://github.com/apache/spark/actions/runs/2492759618
https://github.com/apache/spark/actions/runs/2492227816

See also https://github.com/apache/spark/pull/36899 or https://github.com/apache/spark/pull/36890
In the master branch, seems like at least Hadoop 2 build is broken currently.

*Second*, it is very difficult to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.

Since GitHub added the feature to import other workflow, we should leverage this feature, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test_ansi.yml and https://docs.github.com/en/actions/using-workflows/reusing-workflows. Once we can separate them, it will be defined as a separate workflow.

Namely, each scheduled job should be classified under "All workflows" at https://github.com/apache/spark/actions so other developers can easily track them.

*Third*, we should set the scheduled jobs for branch-3.3, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L78-L83 for branch-3.2 job.

*Forth*, we should improve duplicated test skipping logic. See also https://github.com/apache/spark/pull/36413#issuecomment-1157205469 and https://github.com/apache/spark/pull/36888

*Fifth*, we should probably replace the base image (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain ubunto image w/ Docker image cache. See also https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md
Bug: IllegalStateException from connector does not work well with error class framework
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 08/Jun/22 06:41
Affects Version/s: 3.3.0

With SPARK-39346, Spark SQL binds several exceptions to the internal error, and produces different guidance on dealing with the exception. This assumes these exceptions are only used for noticing internal bugs.

This applies to "connectors" as well, and introduces side-effect on the error log. For Kafka data source, it is a breaking and unacceptable change, because there is an important use case Kafka data source determines a case of "dataloss", and throws IllegalStateException with instruction message on workaround.

I mentioned this as "important" use case, because it can even happen with some valid scenarios - streaming query has some maintenance period and Kafka's retention on topic removes some records in the meanwhile.

Two problems arise:

1) This does not mean Spark has a bug and end users have to report, hence the guidance message on internal error is misleading.

2) Most importantly, instruction message is shown after a long stack trace. With the modification of existing test suite, I see the message being appeared in "line 90" of the error log.

We should roll the right error message back, at least for Kafka's case.
Bug: Release candidates do not have the correct version for PySpark
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Jun/22 06:14
Affects Version/s: 3.3.1

https://github.com/apache/spark/blob/v3.3.0-rc5/dev/create-release/release-tag.sh#L88 fails to replace the version in https://github.com/apache/spark/blob/v3.3.0-rc5/python/pyspark/version.py#L19 because now we have {code}: str ={code} hint ...
Bug: Inline type hints in PySpark
Status: Resolved | Priority: Critical | Resolution: Done
Created: 03/Jun/22 05:01
Affects Version/s: 3.2.1

* SPARK-36845 Inline type hint files for files in python/pyspark/sql
* SPARK-37093 Inline type hints python/pyspark/streaming
* SPARK-37094 Inline type hints for files in python/pyspark
* SPARK-37395 Inline type hint files for files in python/pyspark/ml
* SPARK-37396 Inline type hint files for files in python/pyspark/mllib
Bug: DECODE docs refer to Oracle instead of Spark
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 08/Jun/22 21:20
Affects Version/s: 3.2.0

[https://spark.apache.org/docs/latest/api/sql/index.html#decode]

If no match is found, then {color:#de350b}Oracle{color} returns default. If default is omitted, returns null.
Bug: Support ANALYZE TABLE on v2 tables
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/22 00:50
Affects Version/s: 3.3.4

According to https://github.com/delta-io/delta/pull/840 to implement ANALYZE TABLE in Delta, we need to add the missing APIs in Spark to allow a data source to report the file set to calculate the stats.
Bug: Improve EventLoggingListener and ReplayListener performance by replacing Json4S ASTs with Jackson trees
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/22 01:53
Affects Version/s: 3.0.0

Spark's event log JsonProtocol currently uses Json4s ASTs to generate and parse JSON. Performance overheads from Json4s account for a significant proportion of all time spent in JsonProtocol. If we replace Json4s usage with direct usage of Jackson APIs then we can significantly improve performance (~2x improvement for writing and reading in my own local microbenchmarks).

This performance improvement translates to faster history server load times and reduced load on the Spark driver (and reduced likelihood of dropping events because the listener cannot keep up, therefore reducing the likelihood of inconsistent Spark UIs).

Reducing our usage of Json4s is also a step towards being able to eventually remove our dependency on Json4s: Spark's current use of Json4s creates library conflicts for end users who want to adopt Json4s 4 (see discussion on PRs for SPARK-36408). If Spark can eventually remove its Json4s dependency then we will completely eliminate such conflicts.