av9ash/GitBugs huggingface dataset:
Bug: Self-contained examples with parameter descriptions in PySpark documentation
Status: Resolved | Priority: Critical | Resolution: Done
Created: 08/Aug/22 11:39
Affects Version/s: 3.4.0

This JIRA aims to improve PySpark documentation in:
- {{pyspark}}
- {{pyspark.sql}}
- {{pyspark.sql.streaming}}

We should:
- Make the examples self-contained, e.g., https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html
- Document {{Parameters}} https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html#pandas.DataFrame.pivot. There are many API that misses parameters in PySpark, e.g., [DataFrame.union|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.union.html#pyspark.sql.DataFrame.union]

If the size of file is large, e.g., dataframe.py, we should split that down into each subtask, and improve documentation.
Bug: Fix return type for new median(interval) function 
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 19/Aug/22 17:55
Affects Version/s: 3.4.0

median() right now returns an interval of the same type as the input.
We should instead match mean and avg():

The result type is computed as for the arguments:

- year-month interval: The result is an `INTERVAL YEAR TO MONTH`.
- day-time interval: The result is an `INTERVAL DAY TO SECOND`.
- In all other cases the result is a DOUBLE.
Bug: Spark on Kubernetes doesn't filter data by date
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/22 21:05
Affects Version/s: 3.2.2

I'm creating a Dataset with type date and saving it into s3. When I read it and try to use where() clause, I've noticed it doesn't return data even though it's there

Below is the code snippet I'm running

 
{code:java}
from pyspark.sql.types import Row
from pyspark.sql.functions import *
ds = spark.range(10).withColumn("date", lit("2022-01-01")).withColumn("date", col("date").cast("date"))
ds.where("date = '2022-01-01'").show()
ds.write.mode("overwrite").parquet("s3a://bucket/test")
df = spark.read.format("parquet").load("s3a://bucket/test")
df.where("date = '2022-01-01'").show()
{code}
The first show() returns data, while the second one - no.

I've noticed that it's Kubernetes master related, as the same code snipped works ok with master "local"

UPD: if the column is used as a partition and has the type "date" there is no filtering problem.
Bug: url_decode() exposes a Java error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/22 00:21
Affects Version/s: 3.4.0

Given a badly encode string Spark returns a Java error.
It should the return an ERROR_CLASS

spark-sql> SELECT url_decode('http%3A%2F%2spark.apache.org');

22/08/20 17:17:20 ERROR SparkSQLDriver: Failed in [SELECT url_decode('http%3A%2F%2spark.apache.org')]

java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "2s"

 at java.base/java.net.URLDecoder.decode(URLDecoder.java:232)

 at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)

 at org.apache.spark.sql.catalyst.expressions.UrlCodec$.decode(urlExpressions.scala:113)

 at org.apache.spark.sql.catalyst.expressions.UrlCodec.decode(urlExpressions.scala)
Bug: spark.sql.files.maxPartitionBytes does not observe on-disk compression
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/22 22:00
Affects Version/s: 3.2.0

Why does `spark.sql.files.maxPartitionBytes` estimate the number of partitions based on {_}file size on disk instead of the uncompressed file size{_}?

For example I have a dataset that is 213GB on disk. When I read this in to my application I get 2050 partitions based on the default value of 128MB for maxPartitionBytes. My application is a simple broadcast index join that adds 1 column to the dataframe and writes it out. There is no shuffle.

Initially the size of input /output records seem ok, but I still get a large amount of memory "spill" on the executors. I believe this is due to the data being highly compressed and each partition becoming too big when it is deserialized to work on in memory.

!image-2022-08-10-16-59-05-233.png!

(If I try to do a repartition immediately after reading I still see the first stage spilling memory to disk, so that is not the right solution or what I'm interested in.) 

Instead, I attempt to lower maxPartitionBytes by the (average) compression ratio of my files (about 7x, so let's round up to 8). So I set maxPartitionBytes=16MB.  At this point  I see that spark is reading in from the file in 12-28 MB chunks. Now it makes 14316 partitions on the initial file read and completes with no spillage. 

!image-2022-08-10-16-59-59-778.png!
 
Is there something I'm missing here? Is this just intended behavior? How can I tune my partition size correctly for my application when I do not know how much the data will be compressed ahead of time?
Bug: Decimal multiply can produce the wrong answer because it rounds twice
Status: Open | Priority: Major | Resolution: 
Created: 17/Aug/22 21:06
Affects Version/s: 3.2.0

This looks like it has been around for a long time, but I have reproduced it in 3.2.0+

The example here is multiplying Decimal(38, 10) by another Decimal(38, 10), but I think it can be reproduced with other number combinations, and possibly with divide too.
{code:java}
Seq("9173594185998001607642838421.5479932913").toDF.selectExpr("CAST(value as DECIMAL(38,10)) as a").selectExpr("a * CAST(-12 as DECIMAL(38,10))").show(truncate=false)
{code}
This produces an answer in Spark of {{-110083130231976019291714061058.575920}} But if I do the calculation in regular java BigDecimal I get {{-110083130231976019291714061058.575919}}
{code:java}
BigDecimal l = new BigDecimal("9173594185998001607642838421.5479932913");
BigDecimal r = new BigDecimal("-12.0000000000");
BigDecimal prod = l.multiply(r);
BigDecimal rounded_prod = prod.setScale(6, RoundingMode.HALF_UP);
{code}
Spark does essentially all of the same operations, but it used Decimal to do it instead of java's BigDecimal directly. Spark, by way of Decimal, will set a MathContext for the multiply operation that has a max precision of 38 and will do half up rounding. That means that the result of the multiply operation in Spark is {{{}-110083130231976019291714061058.57591950{}}}, but for the java BigDecimal code the result is {{{}-110083130231976019291714061058.57591949560000000000{}}}. Then in CheckOverflow for 3.2.0 and 3.3.0 or in just the regular Multiply expression in 3.4.0 the setScale is called (as a part of Decimal.setPrecision). At that point the already rounded number is rounded yet again resulting in what is arguably a wrong answer by Spark.

I have not fully tested this, but it looks like we could just remove the MathContext entirely in Decimal, or set it to UNLIMITED. All of the decimal operations appear to have their own overflow and rounding anyways. If we want to potentially reduce the total memory usage, we could also set the max precision to 39 and truncate (round down) the result in the math context instead.  That would then let us round the result correctly in setPrecision afterwards.
Bug: Memory Profiler on Executors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/22 23:18
Affects Version/s: 3.4.0

The ticket proposes to implement PySpark memory profiling on executors. See more [design|https://docs.google.com/document/d/e/2PACX-1vR2K4TdrM1eAjNDC1bsflCNRH67UWLoC-lCv6TSUVXD91Ruksm99pYTnCeIm7Ui3RgrrRNcQU_D8-oh/pub].

There are many factors in a PySpark program’s performance. Memory, as one of the key factors of a program’s performance, had been missing in PySpark profiling. A PySpark program on the Spark driver can be profiled with [Memory Profiler|https://www.google.com/url?q=https://pypi.org/project/memory-profiler/&sa=D&source=editors&ust=1668027860192689&usg=AOvVaw1t4LRcObEGuhaTr5oHEUwU] as a normal Python process, but there was not an easy way to profile memory on Spark executors.

PySpark UDFs, one of the most popular Python APIs, enable users to run custom code on top of the Apache Spark™ engine. However, it is difficult to optimize UDFs without understanding memory consumption.

The ticket proposes to introduce the PySpark memory profiler, which profiles memory on executors. It provides information about total memory usage and pinpoints which lines of code in a UDF attribute to the most memory usage. That will help optimize PySpark UDFs and reduce the likelihood of out-of-memory errors.
Bug: Improve performance of applyInPandas for very small groups
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Aug/22 09:24
Affects Version/s: 3.4.0

Calling {{DataFrame.groupby(...).applyInPandas(...)}} for very small groups in PySpark is very slow. The reason is that for each group, PySpark creates a Pandas DataFrame and calls into the Python code. For very small groups, the overhead is huge, for large groups, it is smaller.

Here is a benchmarks (seconds to {{groupBy(...).applyInPandas(...)}} 10m rows):
||groupSize||Scala||pyspark.sql||pyspark.pandas||
|1024|8.9|16.2|7.8|
|512|9.4|26.7|9.8|
|256|9.3|44.5|20.2|
|128|9.5|82.7|48.8|
|64|9.5|158.2|91.9|
|32|9.6|319.8|207.3|
|16|9.6|652.6|261.5|
|8|9.5|1,376|663.0|
|4|9.8|2,656|1,168|
|2|10.4|5,412|2,456|
|1|11.3|9,491|4,642|

*Idea to overcome this* is to call into Python side with a Pandas DataFrame that contains potentially multiple groups, then perform a Pandas {{DataFrame.groupBy(...).apply(...)}} or provide the {{DataFrameGroupBy}} to the Python method. With large groups, that Panadas DataFrame has all rows of a single group, with small groups it contains many groups. This should improve efficiency.
Bug: Rebalance/Repartition Hints Not Working in PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/22 11:50
Affects Version/s: 3.2.0

Partitioning hints in PySpark do not work because the column parameters are not converted to Catalyst `Expression` instances before being passed to the hint resolver.

The behavior of the hints is documented [here|https://spark.apache.org/docs/3.3.0/sql-ref-syntax-qry-select-hints.html#partitioning-hints-types].

Example:

 
{code:java}
>>> df = spark.range(1024)
>>> 
>>> df
DataFrame[id: bigint]
>>> df.hint("rebalance", "id")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maxwellconradt/spark/python/pyspark/sql/dataframe.py", line 980, in hint
    jdf = self._jdf.hint(name, self._jseq(parameters))
  File "/Users/maxwellconradt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/Users/maxwellconradt/spark/python/pyspark/sql/utils.py", line 196, in deco
    raise converted from None
pyspark.sql.utils.AnalysisException: REBALANCE Hint parameter should include columns, but id found
>>> df.hint("repartition", "id")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maxwellconradt/spark/python/pyspark/sql/dataframe.py", line 980, in hint
    jdf = self._jdf.hint(name, self._jseq(parameters))
  File "/Users/maxwellconradt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/Users/maxwellconradt/spark/python/pyspark/sql/utils.py", line 196, in deco
    raise converted from None
pyspark.sql.utils.AnalysisException: REPARTITION Hint parameter should include columns, but id found {code}
Bug: Enhance collapse window optimization to work in case partition or order by keys are expressions
Status: Open | Priority: Major | Resolution: 
Created: 22/Aug/22 08:06
Affects Version/s: 3.2.0

In window operator with multiple window functions, if any expression is present in partition by or sort order columns, windows are not collapsed even if partition and order by expression is same for all those window functions.

E.g. query:

val w = Window.{_}partitionBy{_}("key").orderBy({_}lower{_}({_}col{_}("value")))

df.select({_}lead{_}("key", 1).over(w), {_}lead{_}("value", 1).over(w))

Current Plan:

-Window(lead(value,1), key, _w1) -------------- W1

- Sort (key, _w1)

-Project (lower(“value”) as _w1) --------- P1

-Window(lead(key,1), key, _w0) ---------------- W2

-Sort(key, _w0)

-Exchange(key)

-Project (lower(“value”) as _w0) ---- P2

-Scan

 

W1 and W2 can be merged in single window
Bug: IndexOutOfBoundsException on groupby + apply pandas grouped map udf function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/22 12:05
Affects Version/s: 3.2.1

I'm grouping on relatively small subset of groups with big size groups.

Working with pyarrow version 2.0.0, machines memory is {color:#444444}64 GiB.{color}

I'm getting the following error:
{code:java}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 387 in stage 162.0 failed 4 times, most recent failure: Lost task 387.3 in stage 162.0 (TID 29957) (ip-172-21-129-187.eu-west-1.compute.internal executor 71): java.lang.IndexOutOfBoundsException: index: 2147483628, length: 36 (expected: range(0, 2147483648))
	at org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:699)
	at org.apache.arrow.memory.ArrowBuf.setBytes(ArrowBuf.java:890)
	at org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:1087)
	at org.apache.spark.sql.execution.arrow.StringWriter.setValue(ArrowWriter.scala:251)
	at org.apache.spark.sql.execution.arrow.ArrowFieldWriter.write(ArrowWriter.scala:130)
	at org.apache.spark.sql.execution.arrow.ArrowWriter.write(ArrowWriter.scala:95)
	at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.$anonfun$writeIteratorToStream$1(ArrowPythonRunner.scala:92)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1474)
	at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.writeIteratorToStream(ArrowPythonRunner.scala:103)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:435)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2031)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:270) {code}
Why do I hit this 2 GB limit? according SPARK-34588 this is supported, perhaps related to SPARK-34020.

Please assist.

Note:
Is it related to the usage of BaseVariableWidthVector and not BaseLargeVariableWidthVector?
Bug: Allow executeTake() / collectLimit's number of starting partitions to be customized
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/22 20:25
Affects Version/s: 3.4.0

Today, Spark’s executeTake() code allow for the limitScaleUpFactor to be customized but does not allow for the initial number of partitions to be customized: it’s currently hardcoded to {{{}1{}}}.

We should add a configuration so that the initial partition count can be customized. By setting this new configuration to a high value we could effectively mitigate the “run multiple jobs” overhead in {{take}} behavior. We could also set it to higher-than-1-but-still-small values (like, say, {{{}10{}}}) to achieve a middle-ground trade-off.

 

Essentially, we need to make {{numPartsToTry = 1L}} ([code|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L481]) customizable. We should do this via a new SQL conf, similar to the {{limitScaleUpFactor}} conf.

 

Spark has several near-duplicate versions of this code ([see code search|https://github.com/apache/spark/search?q=numPartsToTry+%3D+1]) in:
 * SparkPlan
 * RDD
 * pyspark rdd

Also, in pyspark  {{limitScaleUpFactor}}  is not supported either. So for now, I will focus on scala side first, leaving python side untouched and meanwhile sync with pyspark members. Depending on the progress we can do them all in one PR or make scala side change first and leave pyspark change as a follow-up.
Bug: DAGScheduler may not schduler new stage in condition of push-based shuffle enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/22 09:24
Affects Version/s: 3.1.1

In condition of push-based shuffle being enabled and speculative tasks existing, a shuffleMapStage will be resubmitting once fetchFailed occurring, then its parent stages will be resubmitting firstly and it will cost some time to compute. Before the shuffleMapStage being resubmitted, its all speculative tasks success and register map output, but speculative task successful events can not trigger shuffleMergeFinalized because this stage has been removed from runningStages.

Then this stage is resubmitted, but speculative tasks have registered map output and there are no missing tasks to compute, resubmitting stages will also not trigger shuffleMergeFinalized. Eventually this stage‘s _shuffleMergedFinalized keeps false.

Then AQE will submit next stages which are dependent on  this shuffleMapStage occurring fetchFailed. And in getMissingParentStages, this stage will be marked as missing and will be resubmitted, but next stages are added to waitingStages after this stage being finished, so next stages will not be submitted even though this stage's resubmitting has been finished.

I have only met some times in my production env and it is difficult to reproduce。
Bug: Better example for Co-grouped Map
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/22 00:44
Affects Version/s: 3.4.0

Currently, Co-grouped Map uses pandas `asof_merge` as an example, which may misguide users since `asof_merge` is supported in Pandas API on Spark.

So unsupported `merge_sorted` is used instead.
Bug: Aggregate should be group only after collapse project to aggregate
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Aug/22 03:26
Affects Version/s: 3.3.0

CollapseProject rule will merge project expressions into AggregateExpressions in aggregate, which will make the *aggregate.groupOnly* to false.

{code}
val df = testData.distinct().select('key + 1, ('key + 1).cast("long"))
df.queryExecution.optimizedPlan.collect {
  case a: Aggregate => a
}.foreach(agg => assert(agg.groupOnly === true)) 
{code}
Bug: support scaleUpFactor and initialNumPartition in pyspark rdd API
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Aug/22 00:37
Affects Version/s: 3.4.0

This is a followup on https://issues.apache.org/jira/browse/SPARK-40211

`scaleUpFactor` and `initialNumPartition` config are not supported yet in pyspark rdd take API

 

(see [https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L2799)]

basically it hardcoded `scaleUpFactor` as 1 and `initialNumPartition` as 4, therefore pyspark rdd take API is inconsistent with scala API.

 

Anyone familiar with pyspark can help support this (referring to [scala implementation|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1448])?
Bug: Merge adjacent CaseWhen branches if their values are the same
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Aug/22 06:48
Affects Version/s: 3.4.0

For example:
{code:sql}
          CASE
                    WHEN f1.buyer_id IS NOT NULL THEN 1
                    WHEN f2.buyer_id IS NOT NULL THEN 1
                    ELSE 0
          END
{code}

The excepted result:
{code:sql}
          CASE
                    WHEN f1.buyer_id IS NOT NULL or f2.buyer_id IS NOT NULL THEN 1
                    ELSE 0
          END
{code}
Bug: The order of filtering predicates is not reasonable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/22 06:18
Affects Version/s: 3.1.2

{code:java}
select id, data FROM testcat.ns1.ns2.table
where id =2
and md5(data) = '8cde774d6f7333752ed72cacddb05126'
and trim(data) = 'a' {code}
Based on the SQL, we currently get the filters in the following order:
{code:java}
// `(md5(cast(data#23 as binary)) = 8cde774d6f7333752ed72cacddb05126)) AND (trim(data#23, None) = a))` comes before `(id#22L = 2)`
== Physical Plan == *(1) Project [id#22L, data#23]
 +- *(1) Filter ((((isnotnull(data#23) AND isnotnull(id#22L)) AND (md5(cast(data#23 as binary)) = 8cde774d6f7333752ed72cacddb05126)) AND (trim(data#23, None) = a)) AND (id#22L = 2))
    +- BatchScan[id#22L, data#23] class org.apache.spark.sql.connector.InMemoryTable$InMemoryBatchScan{code}
In this predicate order, all data needs to participate in the evaluation, even if some data does not meet the later filtering criteria and it may causes spark tasks to execute slowly.

 

So i think that filtering predicates that need to be evaluated should automatically be placed to the far right to avoid data that does not meet the criteria being evaluated.

 

As shown below:
{noformat}
//  `(id#22L = 2)` comes before `(md5(cast(data#23 as binary)) = 8cde774d6f7333752ed72cacddb05126)) AND (trim(data#23, None) = a))`
== Physical Plan == *(1) Project [id#22L, data#23]
 +- *(1) Filter ((((isnotnull(data#23) AND isnotnull(id#22L)) AND (id#22L = 2) AND (md5(cast(data#23 as binary)) = 8cde774d6f7333752ed72cacddb05126)) AND (trim(data#23, None) = a)))
    +- BatchScan[id#22L, data#23] class org.apache.spark.sql.connector.InMemoryTable$InMemoryBatchScan{noformat}
Bug:  ANSI mode: allow explicitly casting fraction strings as Integral types
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 19/Aug/22 03:50
Affects Version/s: 3.4.0

It's part of the ANSI SQL standard, and more consistent with the non-ansi casting. We can have different behavior for implicit casting to avoid `'1.2' = 1` returns true.
Bug: PySpark installation doesn't support Scala 2.13 binaries
Status: Open | Priority: Major | Resolution: 
Created: 06/Aug/22 09:08
Affects Version/s: 3.3.0

[PyPi|https://pypi.org/project/pyspark/] doesn't support Spark binary [installation|https://spark.apache.org/docs/latest/api/python/getting_started/install.html#using-pypi] for Scala 2.13.
Currently, the setup [script|https://github.com/apache/spark/blob/master/python/pyspark/install.py] allows to set versions of Spark, Hadoop (PYSPARK_HADOOP_VERSION), and mirror (PYSPARK_RELEASE_MIRROR) to download needed Spark binaries, but it's always Scala 2.12 compatible binaries. There isn't any parameter to download "spark-3.3.0-bin-hadoop3-scala2.13.tgz".
It's possible to download Spark manually and set the needed SPARK_HOME, but it's hard to use with pip or Poetry.
Also, env vars (e.g. PYSPARK_HADOOP_VERSION) are easy to use with pip and CLI but not possible with package managers like Poetry.