av9ash/GitBugs huggingface dataset:
Bug: Fix SECURITY.md
Status: Resolved | Priority: Trivial | Resolution: Not A Problem
Created: 24/Mar/22 19:19
Affects Version/s: 3.4.0

At [Github Security -> Security policy|https://github.com/apache/spark/security/policy] 
The info there does not tell users what to do, if they have found a security issue.

The default text for this page is 

 
"
# Security Policy

## Supported Versions

Use this section to tell people about which versions of your project are
currently being supported with security updates.

| Version | Supported          |
| ------- | ------------------ |
| 5.1.x   | :white_check_mark: |
| 5.0.x   | :x:                |
| 4.0.x   | :white_check_mark: |
| < 4.0   | :x:                |

## Reporting a Vulnerability

Use this section to tell people how to report a vulnerability.

Tell them where to go, how often they can expect to get an update on a
reported vulnerability, what to expect if the vulnerability is accepted or
declined, etc.
"

We should change this to something like:

"
Reporting security issues
Apache Spark uses the standard process outlined by the Apache Security Team for reporting vulnerabilities. Note that vulnerabilities should not be publicly disclosed until the project has responded.

To report a possible security vulnerability, please email security@spark.apache.org. This is a non-public list that will reach the Apache Security team, as well as the Spark PMC.

For more info https://spark.apache.org/security.html 
"
Bug: Replace `value.formatted(formatString)` with `formatString.format(value)` to clean up compilation warning
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Mar/22 03:07
Affects Version/s: 3.4.0

There are compile warnings as follows:
{code:java}
[WARNING] /spark-source/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala:67: [deprecation @ org.apache.spark.streaming.ui.RecordRateUIData.formattedAvg.$anonfun | origin=scala.Predef.StringFormat.formatted | version=2.12.16] method formatted in class StringFormat is deprecated (since 2.12.16): Use `formatString.format(value)` instead of `value.formatted(formatString)`,
or use the `f""` string interpolator. In Java 15 and later, `formatted` resolves to the new method in String which has reversed parameters.
[WARNING] /spark-source/sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala:201: [deprecation @ org.apache.spark.sql.streaming.ui.StreamingQueryPagedTable.row | origin=scala.Predef.StringFormat.formatted | version=2.12.16] method formatted in class StringFormat is deprecated (since 2.12.16): Use `formatString.format(value)` instead of `value.formatted(formatString)`,
or use the `f""` string interpolator. In Java 15 and later, `formatted` resolves to the new method in String which has reversed parameters.
[WARNING] /spark-source/sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala:202: [deprecation @ org.apache.spark.sql.streaming.ui.StreamingQueryPagedTable.row | origin=scala.Predef.StringFormat.formatted | version=2.12.16] method formatted in class StringFormat is deprecated (since 2.12.16): Use `formatString.format(value)` instead of `value.formatted(formatString)`,
or use the `f""` string interpolator. In Java 15 and later, `formatted` resolves to the new method in String which has reversed parameters. {code}
Bug: Support `spark.sql.codegen.cleanedSourcePrint` flag to print Codegen cleanedSource
Status: Open | Priority: Trivial | Resolution: 
Created: 24/Mar/22 13:09
Affects Version/s: 3.2.1

When we use spark-sql,  encountering problems in codegen source, we often have to change the log level to DEBUG, but there are too many logs in this mode (DEBUG) .

 

Then `spark.sql.codegen.cleanedSourcePrint` can ensure that just printing codegen source.
Bug: Get rid of invalid configuration elements in mvn_scalafmt in main pom.xml
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Mar/22 02:35
Affects Version/s: 3.2.1

After loading latest spark code into IntelliJ IDEA, it complains that configuration 'parameters' and 'skip' under mvn_scalafmt plugin are not allowed, see screenshot attached for details. 

 

I've contacted the author of mvn_scalafmt, Ciaran Kearney, to confirm if these 2 configuration items are no longer there since v 1.0.0, here's his return, 

 
{quote}That's correct.  The command line parameters were removed by scalafmt itself a few versions ago and skip was replaced by validateOnly (which checks formatting without changing files.
{quote}
 

I think we should get rid of the 'parameters' since it's invalid , and replace 'skip' with 'validateOnly' as Ciaran said.

 

I can make a quick fix for this.
Bug: Update document to make a good guide of multiple versions of the Spark Shuffle Service 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Mar/22 10:29
Affects Version/s: 3.2.0

We have tried to run multiple versions of the Spark Shuffle Service  according [https://github.com/apache/spark/blob/master/docs/running-on-yarn.md]

but, it wont work. 

Then we solved it by using colon when writing %s.classpath instead of commas.

 

Related discussing is in 

https://issues.apache.org/jira/browse/YARN-4577?focusedCommentId=17493624&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17493624
Bug: Add test case for invalid gapDuration of sessionwindow
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Mar/22 05:45
Affects Version/s: 3.2.1

Since the dynamic gapduration has been added in the session window[[33691|https://github.com/apache/spark/pull/33691]|[https://github.com/apache/spark/pull/33691]], users are allowed to enter invalid gapduration . However, for now, test cases are only added for zero and negative gapduration. I think it is necessary to add test cases for invalid gapduration.
Bug: Issues with optimization on adding input to state store in stream-stream join 
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Sep/24 02:29
Affects Version/s: 3.4.0

NOTE: The ticket is describing the actual issue after RCA of tests the reporter reported, but I want to still assign the reporter to [~azera], to give the proper credit. Tests are directly pointing to the edge cases and I could easily spot on the root cause based on the simple reproducers. Thanks [~azera] !

Report link (user@): [https://lists.apache.org/thread/kmq9vlgdbsoytwlcch147n3mlxy56llf]

[https://github.com/apache/spark/blob/039fd13eacb1cef835045e3a60cebf958589e1a2/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L671-L677]
{code:java}
        val isLeftSemiWithMatch =
          joinType == LeftSemi && joinSide == LeftSide && iteratorNotEmpty
        // Add to state store only if both removal predicates do not match,
        // and the row is not matched for left side of left semi join.
        val shouldAddToState =
          !stateKeyWatermarkPredicateFunc(key) && !stateValueWatermarkPredicateFunc(thisRow) &&
          !isLeftSemiWithMatch {code}
The above condition is to determine whether the stream-stream join add the input into state store. The criteria of `both removal predicates do not match` means the input is going to be evicted in this batch - before Spark introduced multiple stateful operators, watermark for late record and watermark for eviction were same, hence the input won't be matched with the condition after filtering out late records (Not sure about the edge case this condition was dealing with.)

After multiple stateful operators (https://issues.apache.org/jira/browse/SPARK-40925), watermark for late record and watermark for eviction are no longer the same. That said, input can be determined as not late, and can be evicted at the same batch. The above condition has to reflect this change but it was missed, hence having correctness issues on the report.

SPARK-45637 is a one of the case, which I'll mark as duplicated.
Bug: The outer query is broken when the subquery uses window function which receives time window as parameter
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Sep/24 12:58
Affects Version/s: 3.4.0

NOTE: The ticket is describing the actual issue after RCA of tests the reporter reported, but I want to still assign the reporter to [~azera], to give the proper credit. Tests are directly pointing to the edge cases and I could easily spot on the root cause based on the simple reproducers. Thanks [~azera] !

This issue is from Spark 3.4.0+. Culprit issue link: https://issues.apache.org/jira/browse/SPARK-40821

This is a silly bug and I'm not sure how return works in Scala after figuring out what happened, but here's my observation.

[https://github.com/apache/spark/blob/c54c017e93090a5fb2edf1b5ef029561b6387a3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala#L90-L94]

This lambda is a partial function - I can't find the exact behavior definition of calling "return" in the partial function. Looks like calling return does not only complete the execution of current context (here it was Aggregate), but somehow complete the execution in broader context and the above operator is  "lost".

Thanks [~azera] for the report of the broken test!

EDIT: It does not look like happening for every case - we have relevant tests in MultiStatefulOperatorsSuite and they have been passing for a while.

Looks like the issue is coupled with subquery. I don't understand how the breakage is happening selectively, but that's what I observed.
Bug: When dynamic allocation enabled and remaining executors laying on a same host, specluative tasks will not be triggered
Status: Open | Priority: Critical | Resolution: 
Created: 01/Sep/24 13:14
Affects Version/s: 3.5.2

We find some cases that dynamic allocation enabled, sometimes tasks launched on remaining executors on a same slow host will execute for a long time, and it makes a spark job keep in running state for a long time.Specluative executes cannot fix the problem,cause if there is one slow task and three executors(set spark.dynamicAllocation.minExecutors = 3),maxNeed will be 2 and ExecutorAllocationManager will think the count of executors is sufficient and will not require new executors.Or a new executor allocated but lay on the same host.Both the case result to long-time-not-finish of a job.We should detect this case, exclude the old host and require for new executors
Bug: Fix Expected only partition pruning predicates exception
Status: Open | Priority: Major | Resolution: 
Created: 04/Sep/24 03:46
Affects Version/s: 3.5.0

How to reproduce:
{code:scala}
sql("CREATE TABLE t (ID BIGINT, DT STRING) USING parquet PARTITIONED BY (DT)")
sql("set spark.sql.hive.metastorePartitionPruningFastFallback=true")
sql("select * from t where dt=20240820").show
{code}



{noformat}
org.apache.spark.sql.AnalysisException: Expected only partition pruning predicates: List(isnotnull(DT#21), (cast(DT#21 as bigint) = 20240820)).
  at org.apache.spark.sql.errors.QueryCompilationErrors$.nonPartitionPruningPredicatesNotExpectedError(QueryCompilationErrors.scala:2414)
  at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.generatePartitionPredicateByFilter(ExternalCatalo
{noformat}
Bug: 'TTransportException: MaxMessageSize reached' occurs if get a table with a large number of partitions 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/24 07:24
Affects Version/s: 4.0.0

{noformat}
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) 
at org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) 
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) 
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) 
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) 
at org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63) 
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) 
at org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329) 
at org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273) 
at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461) 
at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454) 
at org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388) 
at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269) 
at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248) 
at org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110) 
at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270) 
at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205) 
at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) 
...
{noformat}
Bug: org.apache.spark.SparkUnsupportedOperationException: [ENCODER_NOT_FOUND] Not found an encoder of the type T
Status: Open | Priority: Major | Resolution: 
Created: 25/Sep/24 20:49
Affects Version/s: 3.5.2

If a Bean class is having a generic type T as a  bean property and T has upper bound of Serializable,  i.e T <: Serializable , spark should allow creation of an Encoder, treating T as a binary field.

Currently it throws following exception

 

org.apache.spark.SparkUnsupportedOperationException: [ENCODER_NOT_FOUND] Not found an encoder of the type T to Spark SQL internal representation. Consider to change the input type to one of supported at '[https://spark.apache.org/docs/latest/sql-ref-datatypes.html&#39;|https://spark.apache.org/docs/latest/sql-ref-datatypes.html'].
  at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFindEncoderForTypeError(QueryExecutionErrors.scala:1483)
  at org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:142)
  at org.apache.spark.sql.catalyst.JavaTypeInference$.$anonfun$encoderFor$1(JavaTypeInference.scala:128)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
  at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
  at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
  at scala.collection.TraversableLike.map(TraversableLike.scala:286)
  at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
  at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
  at org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:126)
  at org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:58)
  at org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:51)
  at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:62)
  at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179)
Bug: Union ( & UnionExec) nodes equality not take into account unaligned positions of branches causing NO ( reuse of exchange and cached plans)
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/24 19:22
Affects Version/s: 3.3.4

Ideally a Union( plan1, plan2) and Union (plan2, plan1) are logically equal, so long as the output attributes of plan1 and plan2 are matching in terms of name , data type, metdadata etc ( though differing in terms of exprId).

But because current equality and hashCode is dependent on the order of the children, the canonicalizations do not match.

This causes reuse of exchange not happening  in following situations:

 

Exchange 1. =  Union( plan1, plan2)

Exchange 2 =  Union( plan2, plan1)

 

similarly the cached lookup also misses picking the InMemoryRelation.

 

Will be submitting a PR and bug tests for the above scenarios.
Bug: Disable PruneFilters for streaming workloads
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/24 17:32
Affects Version/s: 4.0.0

PruneFilters replaces the {{null}} / {{false}} filter with an empty relation, which means the subtree of the filter is also lost. The optimization does not care about whichever operator is in the subtree, hence some important operators like stateful operator, watermark node, observe node could be lost.

The filter could be evaluated to {{null}} / {{false}} selectively among microbatches in various reasons (one simple example is the modification of the query during restart), which means stateful operator might not be available for batch N and be available for batch N + 1. For this case, streaming query will fail as batch N + 1 cannot load the state from batch N, and it's not recoverable in most cases.

We have to disable the rule for streaming workloads, with the consideration of backward compatibility - we should avoid breaking existing query.
Bug: Support returning RecordBatches from applyInArrow
Status: Open | Priority: Major | Resolution: 
Created: 09/Sep/24 12:11
Affects Version/s: 4.0.0

Currently the new `applyInArrow` method in PySpark uses a function that takes a `pyarrow.Table` and returns a `pyarrow.Table`. This limits the ability for this function to scale, as the entire result set must fit in memory at once in a `Table`. However, we have use cases that can result in a large amount of data that needs to be returned from the function on certain edge cases. The result is immediately turned into a series of batches from the Table, so there's no reason to not just allow an iterator of batches to be returned instead.
Bug: Support line plot with plotly backend
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/24 06:12
Affects Version/s: 4.0.0

While Pandas on Spark supports plotting, PySpark currently lacks this feature. The proposed API will enable users to generate visualizations, such as line plots, by leveraging libraries like Plotly. This will provide users with an intuitive, interactive way to explore and understand large datasets directly from PySpark DataFrames, streamlining the data analysis workflow in distributed environments.

 

See more at [https://docs.google.com/document/d/1IjOEzC8zcetG86WDvqkereQPj_NGLNW7Bdu910g30Dg/edit?usp=sharing] for PySpark Plotting API Specification.
Bug: Make sure internal/dev API is properly annotated in SparkSession
Status: Resolved | Priority: Major | Resolution: Done
Created: 19/Sep/24 02:55
Affects Version/s: 4.0.0

We need to take care of the following fields:
 * sharedState
 * sessionState
 * listenerManager
 * experimentalMethods
 * dataSource -> needs to private[sql]
 * artifactManager
 * baseRelationToDataFrame

...
Bug: Figure out what to do with ExecutionListenerManager
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/24 16:32
Affects Version/s: 4.0.0

ExecutionListenerManager is a user facing API. It allows a user to track queries executed by the current session. A user can register a listener that implements QueryExecutionListener.

The problem here is that QueryExecutionListener uses QueryExecution. QueryExecution is very much an internal class that exposes a lot of the gory parts of spark sql. This not something we can expose in sql/api.

There are roughly two solutions here:
 * Make this a classic only feature. We add a shim for QueryExecution, and throw an exception when a Connect user tries to use this.
 * Create a more lightweight version of query execution, and expose this for both clients.

We'll probably end up doing the shims first, and doing the nice solution later.
Bug: See if regular MiMa can deal with multiple jars
Status: Resolved | Priority: Major | Resolution: Done
Created: 10/Sep/24 00:36
Affects Version/s: 4.0.0

The current MiMa checks do checks on a per module basis. This is a bit problematic when you move classes to different modules. This ticket is here to investigate if we can make MiMa work for multiple modules at the same time.
Bug: Remove the ANSI config suggestion in DATETIME_FIELD_OUT_OF_BOUNDS
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/24 11:46
Affects Version/s: 4.0.0

DATETIME_FIELD_OUT_OF_BOUNDS contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: Remove the ANSI config suggestion in INVALID_INTERVAL_WITH_MICROSECONDS_ADDITION
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 10:34
Affects Version/s: 4.0.0

INVALID_INTERVAL_WITH_MICROSECONDS_ADDITION contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: SQL Pipe Syntax
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/24 20:36
Affects Version/s: 4.0.0

This umbrella Jira ticket tracks implementing new support for issuing SQL queries using pipe syntax.

The objective is to make it easy to compose queries by specifying a sequence of SQL clauses separated by the pipe token |> wherein each operator represents a fully-defined transformation of the preceding relation. Each pipe operator may refer to the names and rows generated by the preceding pipe operator only; otherwise, each step is stateless.
 * Research paper: [https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/]
 * Open-source ZetaSQL implementation: [https://github.com/google/zetasql/blob/master/docs/pipe-syntax.md]
 * Spark prototype: https://github.com/apache/spark/pull/47837

 
For example, here's query 13 from TPC-H:
 
SELECT c_count, COUNT( * ) AS custdist FROM
  (SELECT c_custkey, COUNT(o_orderkey) c_count FROM customer
  LEFT OUTER JOIN orders ON c_custkey = o_custkey
  AND o_comment NOT LIKE '%unusual%packages%' GROUP BY c_custkey) AS c_orders
GROUP BY c_count
ORDER BY custdist DESC, c_count DESC;
 
With the new syntax, it becomes:
 
FROM customer
 |> LEFT OUTER JOIN orders ON c_custkey = o_custkey
    AND o_comment NOT LIKE '%unusual%packages%'
 |> AGGREGATE COUNT(o_orderkey) c_count
    GROUP BY c_custkey
 |> AGGREGATE COUNT( * ) AS custdist
    GROUP BY c_count
 |> ORDER BY custdist DESC, c_count DESC;
Bug: Provide error Conditions for make_date/make_timestamp errors _LEGACY_ERROR_TEMP_2000
Status: Open | Priority: Major | Resolution: 
Created: 24/Sep/24 17:51
Affects Version/s: 4.0.0

invoking make_date() or make_timestamp() with out of domain arguments result in error messages which have not been converted to the new error format.
All of these errors are related to ANSI mode, so we should fix them to facilitate the move to ANSI mode

Example:
{noformat}
SET spark.sql.ansi.enabled = true;
SELECT make_date(2024, 13, 1);
Invalid value for MonthOfYear (valid values 1 - 12): 13. If necessary set "spark.sql.ansi.enabled" to false to bypass this error.
org.apache.spark.SparkDateTimeException: Invalid value for MonthOfYear (valid values 1 - 12): 13. If necessary set "spark.sql.ansi.enabled" to false to bypass this error.
 at org.apache.spark.sql.errors.QueryExecutionErrors$.ansiDateTimeError(QueryExecutionErrors.scala:287)
 at org.apache.spark.sql.catalyst.expressions.MakeDate.nullSafeEval(datetimeExpressions.scala:2483)
{noformat}