av9ash/GitBugs huggingface dataset:
Bug: Can executors recover/reuse shuffle files upon failure?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 08/May/23 08:37
Affects Version/s: 3.3.1

Hello,

We've been in touch with a few spark specialists who suggested us a potential solution to improve the reliability of our jobs that are shuffle heavy

Here is what our setup looks like
 * Spark version: 3.3.1
 * Java version: 1.8
 * We do not use external shuffle service
 * We use spot instances

We run spark jobs on clusters that use Amazon EBS volumes. The spark.local.dir is mounted on this EBS volume. One of the offerings from the service we use is EBS migration which basically means if a host is about to get evicted, a new host is created and the EBS volume is attached to it

When Spark assigns a new executor to the newly created instance, it basically can recover all the shuffle files that are already persisted in the migrated EBS volume

Is this how it works? Do executors recover / re-register the shuffle files that they found?

So far I have not come across any recovery mechanism. I can only see 
{noformat}
KubernetesLocalDiskShuffleDataIO{noformat}
 that has a pre-init step where it tries to register the available shuffle files to itself

A natural follow-up on this,

If what they claim is true, then ideally we should expect that when an executor is killed/OOM'd and a new executor is spawned on the same host, the new executor registers the shuffle files to itself. Is that so?

Thanks
Bug: Introducing a new SparkListenerEvent for jobs running in AQE mode
Status: Open | Priority: Minor | Resolution: 
Created: 09/May/23 12:09
Affects Version/s: 3.4.0

Prior to AQE, we used to rely on SparkListenerJobEnd to figure out when a job has been completed from Spark's scheduler point of view

Unfortunately with AQE a job is split into multiple jobs and there's no way for us to figure out when a job is completed

We have a listener that tracks some spark metrics. In order to establish the right relations among stages, jobs, and tasks we had to keep a mapping around. We used the SparkListenerJobEnd event to clear out these entries, however with AQE we cannot know when is the right time to remove these entries that we track

What suggestion would you have for this?

Would having a SparkListenerJobGroupEnd be a rationale addition?

Because our jobs have 1 single action, we probably can rely on stage metrics i.e if a stage completed that had some output metrics > 0, that's a useful indicator that the next Job End likely was the last job in that Job group
Bug: Spark Streaming is not able to read a .txt file whose name has [] special character
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/May/23 18:28
Affects Version/s: 3.4.0

* For example, If a directory contains a following file:
/path/abc[123]
and users would load spark.readStream.format("text").load("/path") as stream input. It throws an exception, saying no matching path /path/abc[123]. Spark thinks abc[123] is a regex that only matches file named abc1, abc2 and abc3.

 * Upon investigation this is due to how we [getBatch|https://github.com/databricks/runtime/blob/3af402d23620a0952e151d96c3184d2233217c87/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala#L269] in the FileStreamSource. In `FileStreamSource` we already check file pattern matching and find all match file names. However, in DataSource we check for glob characters again and try to expend it [here|https://github.com/databricks/runtime/blob/3af402d23620a0952e151d96c3184d2233217c87/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L274].
Bug: The Generator's statistics should be ratio times greater than the child nodes
Status: Open | Priority: Minor | Resolution: 
Created: 05/May/23 12:31
Affects Version/s: 3.4.0

The Generator's statistics should be ratio times greater than the child nodes.

Generator is an expression that produces zero or more rows given a single input row.

If `UserDefinedGenerator` and `HiveUDTF` were used, the output could be N times that of the child node, resulting in a statistical error.

Because of incorrect statistics, Spark may select an incorrect execution plan. For example, if `BroadcastHashJoinExec` is selected, the Job fails to broadcast buildSide.
Bug: Asc/desc arrow icons for sorting column does not get displayed in the table column
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/May/23 05:35
Affects Version/s: 3.3.0

The sorting icon is not displayed when the column is clicked to sort by asc/desc.

See attached image: (The index column is sort by asc order by the down arrow is not displayed)

!image (4).png!

This broke due to the upgrade from DataTables from 1.10.20 to 1.10.25. I have confirmed that version 1.13.2 in master branch does not have this problem. 
For reference, in 1.13.2, it looks like the following:
!Screen Shot 2023-05-01 at 10.39.00 PM.png!
Bug: Add latest timestamp on no-execution trigger for Idle event in streaming query listener
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/May/23 01:08
Affects Version/s: 3.5.0

From SPARK-43183, we introduced a new callback "onQueryIdle" in streaming query listener to separate the idle event from progress update event. The QueryIdleEvent is introduced, but the timestamp is missing which is useful to track the latest no-execution timestamp. QueryStartedEvent and QueryProgressEvent also provide such information.
Bug: Improve the document of GenTPCDSData, so that developers could easy to generate TPCDS table data.
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/May/23 02:28
Affects Version/s: 3.5.0


{code:java}
build/sbt "sql/Test/runMain <this class> --dsdgenDir <path> --location <path> --scaleFactor 1"
{code}

The command show above easy to cause OOM.

Please refer https://issues.apache.org/jira/browse/SPARK-43573
Bug: Spark caching in the context of a single job
Status: Resolved | Priority: Trivial | Resolution: Invalid
Created: 08/May/23 09:00
Affects Version/s: 3.3.1

Does caching benefit a spark job with only a single action in it? Spark IIRC already optimizes shuffles by persisting them onto the disk

I am unable to find a counter-example where caching would benefit a job with a single action. In every case I can think of, the shuffle checkpoint acts as a good enough caching mechanism in itself

FWIW, I am talking specifically in the context of the Dataframe API. The StorageLevel allowed in my case is DISK_ONLY i.e. I am not looking to speed up by caching data in memory

To rephrase, is DISK_ONLY caching better or same as shuffle checkpointing in the context of a single action
Bug: Reconsider design of ignoreCorruptFiles feature
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/22 21:30
Affects Version/s: 3.0.0

I'm filing this ticket as a followup to the discussion at [https://github.com/apache/spark/pull/36775#issuecomment-1148136217] regarding the `ignoreCorruptFiles` feature: the current implementation is based towards considering a broad range of IOExceptions to be corruption, but this is likely overly-broad and might mis-identify transient errors as corruption (causing non-corrupt data to be erroneously discarded).

SPARK-39389 fixes one instance of that problem, but we are still vulnerable to similar issues because of the overall design of this feature.

I think we should reconsider the design of this feature: maybe we should switch the default behavior so that only an explicit allowlist of known corruption exceptions can cause files to be skipped. This could be done through involvement of other parts of the code, e.g. rewrapping exceptions into a `CorruptFileException` so higher layers can positively identify corruption.

Any changes to behavior here could potentially impact users jobs, so we'd need to think carefully about when we want to change (in a 3.x release? 4.x?) and how we want to provide escape hatches (e.g. configs to revert back to old behavior).
Bug: If spark.default.parallelism is unset, RDD defaultPartitioner may pick a value that is too large to successfully run
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/22 00:28
Affects Version/s: 1.0.0

[According to its docs|https://github.com/apache/spark/blob/899f6c90eb2de5b46a36710a131d7417010ce4b3/core/src/main/scala/org/apache/spark/Partitioner.scala#L45-L65], {{Partitioner.defaultPartitioner}} will use the maximum number of RDD partitions as its partition count when {{spark.default.parallelism}} is not set. If that number of upstream partitions is very large then this can result in shuffles where {{{}numMappers * numReducers = numMappers^2{}}}, which can cause various problems that prevent the job from successfully running.

To help users identify when they have run into this problem, I think we should add warning logs to Spark.

As an example of the problem, let's say that I have an RDD with 100,000 partitions and then do a {{reduceByKey}} on it without specifying an explicit partitioner or partition count. In this case, Spark will plan a reduce stage with 100,000 partitions:
{code:java}
scala>  sc.parallelize(1 to 100000, 100000).map(x => (x, x)).reduceByKey(_ + _).toDebugString
res7: String =
(100000) ShuffledRDD[21] at reduceByKey at <console>:25 []
   +-(100000) MapPartitionsRDD[20] at map at <console>:25 []
        |     ParallelCollectionRDD[19] at parallelize at <console>:25 []
{code}
This results in the creation of 10 billion shuffle blocks, so if this job _does_ run it is likely to be extremely show. However, it's more likely that the driver will crash when serializing map output statuses: if we were able to use one bit per mapper / reducer pair (which is probably overly optimistic in terms of compressibility) then the map statuses would be ~1.25 gigabytes (and the actual size is probably much larger)!

I don't think that users are likely to intentionally wind up in this scenario: it's more likely that either (a) their job depends on {{spark.default.parallelism}} being set but it was run on an environment lacking a value for that config, or (b) their input data significantly grew in size. These scenarios may be rare, but they can be frustrating to debug (especially if a failure occurs midway through a long-running job).

I think we should do something to handle this scenario.

A good starting point might be for {{Partitioner.defaultPartitioner}} to log a warning when the default partition size exceeds some threshold.

In addition, I think it might be a good idea to log a similar warning in {{MapOutputTrackerMaster}} right before we start trying to serialize map statuses: in a real-world situation where this problem cropped up, the map stage ran successfully but the driver crashed when serializing map statuses. Putting a warning about partition counts here makes it more likely that users will spot that error in the logs and be able to identify the source of the problem (compared to a warning that appears much earlier in the job and therefore much farther from the likely site of a crash).
Bug: Support stage level schedule for standalone cluster when dynamic allocation is disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/22 15:23
Affects Version/s: 3.3.0

[SPARK-39062|https://issues.apache.org/jira/browse/SPARK-39062] added stage level schedule support for standalone cluster when dynamic allocation was enabled, spark would request for executors for different resource profiles.
While when dynamic allocation is disabled, we can also leverage stage level schedule to schedule tasks based on resource profile(task resource requests) to executors with default resource profile.
Bug: Reconsider exposure of Json4s symbols in public ResourceInformation.toJson API
Status: Open | Priority: Major | Resolution: 
Created: 02/Jul/22 00:18
Affects Version/s: 3.0.0

Followup to discussion at [https://github.com/apache/spark/pull/36885#discussion_r898644147]

Spark's `ResourceInformation` class exposes Json4s symbols in its public API (via the toJson method). 

In the long run, I don't think that exposing third-party library symbols in our public APIs is a good idea. Perhaps we should make this API private or change it to require that users / implementors return an object which can be turned to JSON via Jackson ObjectMapper.

In principle we still have some freedom to change this because the ResourceInformation class is marked as `@Evolving`

I'm filing this ticket so that I can reference it in TODO code comments.
Bug: Broadcast joins should pushdown join constraints as Filter to the larger relation
Status: Open | Priority: Major | Resolution: 
Created: 12/Jul/22 09:21
Affects Version/s: 3.2.0

SPARK-19609 was bulk-closed a while ago, but not fixed. I've decided to re-open it here for more visibility, since I believe this bug has a major impact and that fixing it could drastically improve the performance of many pipelines.

Allow me to paste the initial description again here:

_For broadcast inner-joins, where the smaller relation is known to be small enough to materialize on a worker, the set of values for all join columns is known and fits in memory. Spark should translate these values into a {{Filter}} pushed down to the datasource. The common join condition of equality, i.e. {{{}lhs.a == rhs.a{}}}, can be written as an {{a in ...}} clause. An example of pushing such filters is already present in the form of {{IsNotNull}} filters via_ [~sameerag]{_}'s work on SPARK-12957 subtasks.{_}

_This optimization could even work when the smaller relation does not fit entirely in memory. This could be done by partitioning the smaller relation into N pieces, applying this predicate pushdown for each piece, and unioning the results._

 

Essentially, when doing a Broadcast join, the smaller side can be used to filter down the bigger side before performing the join. As of today, the join will read all partitions of the bigger side, without pruning partitions
Bug: Upgrade jetty-http from 9.4.46.v20220331 to 9.4.48.v20220622
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/22 12:53
Affects Version/s: 3.4.0

[Release note |https://github.com/eclipse/jetty.project/releases] 

[CVE-2022-2047|https://nvd.nist.gov/vuln/detail/CVE-2022-2047]
Bug: Support 2-chars lineSep in CSV datasource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/22 03:29
Affects Version/s: 3.3.0

Univocity parser allows to set line separator to 1 to 2 characters ([code|https://github.com/uniVocity/univocity-parsers/blob/master/src/main/java/com/univocity/parsers/common/Format.java#L103]), CSV options should not block this usage ([code|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala#L218]).

 

Due to the limitation around the `normalizedNewLine` (https://github.com/uniVocity/univocity-parsers/issues/170), setting 2 chars as a line separator could cause some weird/bad behaviors. Thus, we probably should leave this proposed fix as an undocumented feature and warn users to do this.

 

A more proper fix could be further investigated in the future.
Bug: Disable scheduled builds that do not pass even once
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/22 11:38
Affects Version/s: 3.4.0

We added some more builds that have never beed tested so far. We should probably disable them to check the build status easily for now.

See also SPARK-39681, SPARK-39685, SPARK-39682 and SPARK-39684
Bug: PySpark build broken in branch-3.2 with Scala 2.13
Status: Open | Priority: Major | Resolution: 
Created: 05/Jul/22 11:25
Affects Version/s: 3.2.1

https://github.com/apache/spark/runs/7189972241?check_suite_focus=true

{code}
========================================================================
Running PySpark tests
========================================================================
Traceback (most recent call last):
  File "./python/run-tests.py", line 65, in <module>
    raise RuntimeError("Cannot find assembly build directory, please build Spark first.")
RuntimeError: Cannot find assembly build directory, please build Spark first.
{code}
Bug: RowMatrix(...).computeCovariance() VS Correlation.corr(..., ...)
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 03/Jul/22 15:38
Affects Version/s: 3.2.1

I have a Pyspark DF with one column. This column type is Vector and the values are DenseVectors of size 768.  The DF has 1 million rows.

I want to calculate the Covariance matrix of this set of vectors.

When I try to calculate it with `RowMatrix(df.rdd.map(list)).computeCovariance()`, it takes 1.57 minuts.

When I try to calculate the Correlation matrix with `Correlation.corr(df, '_1')`, it takes 33 seconds.

Covariance and Correlation's formula are pretty much the same, therefore, I don't understand the gap between them
Bug: Use UnsafeProjection.create to respect `spark.sql.codegen.factoryMode` in ExpresssionEncoder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/22 02:47
Affects Version/s: 3.4.0

Related to https://issues.apache.org/jira/browse/SPARK-25374. We should make the codegen configuration work in the opposite way too (unsafe projections)
Bug: Python Lint does not actually check for `black` formatter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/22 15:31
Affects Version/s: 3.3.0

In our current `dev/lint-python` script we execute the following check: {{}}

 
{code:java}
$BLACK_BUILD 2> /dev/null 
if [ $? -ne 0 ]; then
  # ...
fi{code}
However, even if black is installed simply invoking it will actually return a non-zero exit code because no src files have been specified.  To correctly check for the existence of `black` we need a different check.
Bug: Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/22 22:05
Affects Version/s: 3.2.1

One of our data scientists discovered a problem wherein a data frame `.show()` call printed non-empty results, but `.count()` printed 0. I've narrowed the issue to a small, reproducible test case which exhibits this aberrant behavior. In pyspark, run the following code:
{code:python}
from pyspark.sql.types import *
parquet_pushdown_bug_df = spark.createDataFrame([{"COL0": int(0)}], schema=StructType(fields=[StructField("COL0",IntegerType(),True)]))
parquet_pushdown_bug_df.repartition(1).write.mode("overwrite").parquet("parquet_pushdown_bug/col0=0/parquet_pushdown_bug.parquet")
reread_parquet_pushdown_bug_df = spark.read.parquet("parquet_pushdown_bug")
reread_parquet_pushdown_bug_df.filter("col0 = 0").show()
print(reread_parquet_pushdown_bug_df.filter("col0 = 0").count())
{code}
In my usage, this prints a data frame with 1 row and a count of 0. However, disabling `spark.sql.parquet.filterPushdown` produces consistent results:
{code:python}
spark.conf.set("spark.sql.parquet.filterPushdown", False)
reread_parquet_pushdown_bug_df.filter("col0 = 0").show()
reread_parquet_pushdown_bug_df.filter("col0 = 0").count()
{code}
This will print the same data frame, however it will print a count of 1. The key to triggering this bug is not just enabling `spark.sql.parquet.filterPushdown` (which is enabled by default). The case of the column in the data frame (before writing) must differ from the case of the partition column in the file path, i.e. COL0 versus col0 or col0 versus COL0.
Bug: regexp_replace should support column arguments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/22 20:58
Affects Version/s: 3.3.0

{{F.regexp_replace}} in PySpark currently only supports strings for the second and third argument: [https://github.com/apache/spark/blob/1df6006ea977ae3b8c53fe33630e277e8c1bc49c/python/pyspark/sql/functions.py#L3265]

In Scala, columns are also supported: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2836|https://github.com/apache/spark/blob/1df6006ea977ae3b8c53fe33630e277e8c1bc49c/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2836]

The desire to use columns as arguments for the function has been raised previously on StackExchange: [https://stackoverflow.com/questions/64613761/in-pyspark-using-regexp-replace-how-to-replace-a-group-with-value-from-another|https://stackoverflow.com/questions/64613761/in-pyspark-using-regexp-replace-how-to-replace-a-group-with-value-from-another,], where the suggested fix was to use {{{}F.expr{}}}.

It should be relatively straightforward to support in PySpark the two function signatures supported in Scala.
Bug: ExecutionListenerManager's registration of the ExecutionListenerBus should be lazy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/22 23:16
Affects Version/s: 3.4.0

Today, ExecutionListenerManager eagerly registers an ExecutionListenerBus SparkListener when it is created, even if the SparkSession has no query execution listeners registered. In applications with many short-lived SparkSessions, this can cause a buildup of empty listeners on the shared listener bus, increasing Spark listener processing times on the driver.

If we make the registration lazy then we avoid this driver-side listener performance overhead.
Bug: Fix flaky hive - slow tests because of out-of-memory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/22 09:31
Affects Version/s: 3.4.0

e.g., https://github.com/apache/spark/runs/7514294675?check_suite_focus=true and https://github.com/apache/spark/runs/7511747752?check_suite_focus=true
Bug: Factor PythonArrowInput out as a symmetry to PythonArrowOutput
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/22 09:52
Affects Version/s: 3.4.0

In https://issues.apache.org/jira/browse/SPARK-29317, we factored {{PythonArrowOutput}} out. It's better to factor {{PythonArrowInput}} out too to be consistent
Bug: Missing sbin scripts in PySpark packages
Status: Open | Priority: Major | Resolution: 
Created: 19/Jul/22 13:02
Affects Version/s: 3.2.0

In the PySpark setup.py, only a subset of all scripts is included.
I'm in particular missing the `submit-all.sh` script:
{code:python}
        package_data={
            'pyspark.jars': ['*.jar'],
            'pyspark.bin': ['*'],
            'pyspark.sbin': ['spark-config.sh', 'spark-daemon.sh',
                             'start-history-server.sh',
                             'stop-history-server.sh', ],

            [...]
        },
{code}
 

The solution is super simple, just change 'pyspark.sbin' to:
{code:python}
'pyspark.sbin': ['*'],
{code}
 

I would happily submit a PR to github, but I have no clue on the organizational details.

This would be great to get backported for pyspark 3.2.x as well as 3.3.x soon.
Bug: Upgrade sbt from 1.7.0 to 1.7.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/22 05:56
Affects Version/s: 3.4.0

release notes: [https://github.com/sbt/sbt/releases]

https://github.com/sbt/sbt/compare/v1.7.0...v1.7.1

sbt 1.7.1 Bug fix
 * Fixes Java incremental compilation, specifically parsing of annotations in class files of [@SethTisue|https://github.com/SethTisue] in [sbt/zinc#1111|https://github.com/sbt/zinc/pull/1111]
Bug: pandas and pandas on Spark API parameter naming difference 
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/22 22:50
Affects Version/s: 3.3.0

I noticed there are some parameter naming differences between pandas and pandas on Spark. For example, in "read_csv", the path parameter is "filepath_or_buffer" for pandas and "path" for pandas on Spark. I wonder why such a difference exists and may I ask to change it to match exactly the same in pandas.