av9ash/GitBugs huggingface dataset:
Bug: Incorect result of filter using isNull condition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 12:02
Affects Version/s: 3.2.0

Simple code that allows to reproduce this issue:
{code:java}
 val frame = Seq((false, 1)).toDF("bool", "number")
frame
  .checkpoint()
  .withColumn("conditions", when(col("bool"), "I am not null"))
  .filter(col("conditions").isNull)
  .show(false){code}
Although "conditions" column is null
{code:java}
 +-----+------+----------+
|bool |number|conditions|
+-----+------+----------+
|false|1     |null      |
+-----+------+----------+{code}
empty result is shown.

Execution plans:
{code:java}
== Parsed Logical Plan ==
'Filter isnull('conditions)
+- Project [bool#124, number#125, CASE WHEN bool#124 THEN I am not null END AS conditions#252]
   +- LogicalRDD [bool#124, number#125], false

== Analyzed Logical Plan ==
bool: boolean, number: int, conditions: string
Filter isnull(conditions#252)
+- Project [bool#124, number#125, CASE WHEN bool#124 THEN I am not null END AS conditions#252]
   +- LogicalRDD [bool#124, number#125], false

== Optimized Logical Plan ==
LocalRelation <empty>, [bool#124, number#125, conditions#252]

== Physical Plan ==
LocalTableScan <empty>, [bool#124, number#125, conditions#252]
 {code}
After removing checkpoint proper result is returned  and execution plans are as follow:
{code:java}
== Parsed Logical Plan ==
'Filter isnull('conditions)
+- Project [bool#124, number#125, CASE WHEN bool#124 THEN I am not null END AS conditions#256]
   +- Project [_1#119 AS bool#124, _2#120 AS number#125]
      +- LocalRelation [_1#119, _2#120]

== Analyzed Logical Plan ==
bool: boolean, number: int, conditions: string
Filter isnull(conditions#256)
+- Project [bool#124, number#125, CASE WHEN bool#124 THEN I am not null END AS conditions#256]
   +- Project [_1#119 AS bool#124, _2#120 AS number#125]
      +- LocalRelation [_1#119, _2#120]

== Optimized Logical Plan ==
LocalRelation [bool#124, number#125, conditions#256]

== Physical Plan ==
LocalTableScan [bool#124, number#125, conditions#256]
 {code}
It seems that the most important difference is LogicalRDD ->  LocalRelation

There are following ways (workarounds) to retrieve correct result:

1) remove checkpoint

2) add explicit .otherwise(null) to when

3) add checkpoint() or cache() just before filter

4) downgrade to Spark 3.1.2
Bug: Hidden File Metadata Support for Spark SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 00:54
Affects Version/s: 3.2.0

Provide a new interface in Spark SQL that allows users to query the metadata of the input files for all file formats, expose them as *built-in hidden columns* meaning *users can only see them when they explicitly reference them* (e.g. file path, file name)
Bug: pyspark.pandas read_csv() and to_csv() should handle local files 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Nov/21 18:06
Affects Version/s: 3.2.0

Pandas programmers who move their code to Spark would like to import and export text files to and from their local disk. I know there are technical hurdles to this (since Spark is usually in a cluster that does not know where your local computer is) but it would really help code migration. 

For read_csv() and to_csv(), the syntax {{*file://c:/Temp/my_file.csv* }}(or something like this) should import and export to the local disk on Windows. Similarly for Mac and Linux.
Bug: try_simplify_traceback should not fail when tb_frame.f_lineno is None
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 08:53
Affects Version/s: 3.1.2

{code}
Traceback (most recent call last):
  File "/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/worker.py", line 630, in main
    tb = try_simplify_traceback(sys.exc_info()[-1])
  File "/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/util.py", line 217, in try_simplify_traceback
    new_tb = types.TracebackType(
TypeError: 'NoneType' object cannot be interpreted as an integer
{code}
Bug: DataFrame.take() only uses one worker
Status: Open | Priority: Major | Resolution: 
Created: 01/Nov/21 20:01
Affects Version/s: 3.1.1

Say you have query:
{code:java}
>>> df = spark.sql("select * from mytable where x = 99"){code}
Now, out of billions of row, there's only ten rows where x is 99.

If I do:
{code:java}
>>> df.limit(10).collect()
[Stage 1:>      (0 + 1) / 1]{code}
It only uses one worker. This takes a really long time since one CPU is reading the billions of row.

However, if I do this:
{code:java}
>>> df.limit(10).rdd.collect()
[Stage 1:>      (0 + 10) / 22]{code}
All the workers are running.

I think there's some optimization issue DataFrame.take(...).

This did not use to be an issue, but I'm not sure if it was working with 3.0 or 2.4.
Bug: pyspark.pandas histogram accepts the range option but does not use it
Status: Open | Priority: Major | Resolution: 
Created: 01/Nov/21 21:18
Affects Version/s: 3.2.0

In pyspark.pandas if you write a line like this
{quote}DF.plot.hist(bins=30, range=[0, 20], title="US Counties -- DeathsPer100k (<20)")
{quote}
it compiles and runs, but the plot does not respect the range. All the values are shown.

The workaround is to create a new DataFrame that pre-selects just the rows you want, but line above should work also.
Bug: pyspark should not pass all options to session states.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/21 09:21
Affects Version/s: 3.2.0

in current session.py
have Such code 
{code}
for key, value in self._options.items():
session._jsparkSession.sessionState().conf().setConfString(key, value)
{code}
Looks like it will pass all options to a existed/new created session 
In scala code, we will check it it is a static conf.
Bug: PySpark tests failing on Pandas 0.23
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/21 17:21
Affects Version/s: 3.2.0

I was running Spark tests with Pandas {{0.23.4}} and got the error below. The minimum Pandas version is currently {{0.23.2}} [(Github)|https://github.com/apache/spark/blob/v3.2.0/python/setup.py#L114]. Upgrading to {{0.24.0}} fixes the error. I think Spark needs [this fix (Github)|https://github.com/pandas-dev/pandas/pull/21160/files#diff-1b7183f5b3970e2a1d39a82d71686e39c765d18a34231b54c857b0c4c9bb8222] in Pandas.
{code:java}
$ python/run-tests --testnames 'pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTest.test_floordiv'

...

======================================================================
ERROR [5.785s]: test_floordiv (pyspark.pandas.tests.data_type_ops.test_boolean_ops.BooleanOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py", line 128, in test_floordiv
    self.assert_eq(b_pser // b_pser.astype(int), b_psser // b_psser.astype(int))
  File "/home/circleci/miniconda/envs/python3/lib/python3.6/site-packages/pandas/core/ops.py", line 1069, in wrapper
    result = safe_na_op(lvalues, rvalues)
  File "/home/circleci/miniconda/envs/python3/lib/python3.6/site-packages/pandas/core/ops.py", line 1033, in safe_na_op
    return na_op(lvalues, rvalues)
  File "/home/circleci/miniconda/envs/python3/lib/python3.6/site-packages/pandas/core/ops.py", line 1027, in na_op
    result = missing.fill_zeros(result, x, y, op_name, fill_zeros)
  File "/home/circleci/miniconda/envs/python3/lib/python3.6/site-packages/pandas/core/missing.py", line 641, in fill_zeros
    signs = np.sign(y if name.startswith(('r', '__r')) else x)
TypeError: ufunc 'sign' did not contain a loop with signature matching types dtype('bool') dtype('bool')
{code}
These are my relevant package versions:
{code:java}
$ conda list | grep -e numpy -e pyarrow -e pandas -e python
# packages in environment at /home/circleci/miniconda/envs/python3:
numpy                     1.16.6           py36h0a8e133_3  
numpy-base                1.16.6           py36h41b4c56_3  
pandas                    0.23.4           py36h04863e7_0  
pyarrow                   1.0.1           py36h6200943_36_cpu    conda-forge
python                    3.6.12               hcff3b4d_2    anaconda
python-dateutil           2.8.1                      py_0    anaconda
python_abi                3.6                     1_cp36m    conda-forg
{code}
Bug: Upgrade commons-cli to 1.5.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/21 04:33
Affects Version/s: 3.3.0

Currently used commons-cli is too old and contains an issue which affects the behavior of bin/spark-sql

{code}
bin/spark-sql -e 'SELECT "Spark"'
...
Error in query: 
no viable alternative at input 'SELECT "'(line 1, pos 7)

== SQL ==
SELECT "Spark
-------^^^
{code}

The root cause of this issue seems to be resolved in CLI-185.
Bug: Uses Python's standard string formatter for SQL API in pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/21 06:33
Affects Version/s: 3.3.0

Currently {{pyspark.pandas.sql}} uses its own hacky parser: https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py

We should ideally switch it to the standard Python formatter https://docs.python.org/3/library/string.html#custom-string-formatting
Bug: DataFrame.mapInArrow
Status: Open | Priority: Major | Resolution: 
Created: 07/Nov/21 05:08
Affects Version/s: 3.3.0

h2. Background

*Usability problem*

In Scala APIs, we have added Arrow integration and developer APIs in Apache Spark, for example, at ArrowConverters but this is again inconvenient to use out of the box.

In PySpark, in order to use Arrow format to connect to other external systems, they should manually convert pandas DataFrame in pandas UDF to the Arrow batch, which is inconvenient.


*Technical problem*

pandas UDFs are used in other use cases too. For example, they convert it back to
Arrow batch, and integrate with other systems, see also https://github.com/apache/spark/pull/26783#issue-534127514.
However, it doesn’t vectorize operations because pandas doesn’t support nested structure
natively, and the performance impact seems non-trivial.

In addition, it requires virtually copying during the conversion between pandas and Arrow format that consumes computation (Spark internal format -> Arrow format -> pandas DataFrame -> Arrow format). See https://github.com/apache/spark/pull/26783#issue-534127514 for performance impact.


h2. Other notes:

See also:
- SPARK-30153
- SPARK-26413

h2. Proposal

I would like to propose an API DataFrame.mapInArrow like {{DataFrame.mapInPandas}}, and {{RDD.mapPartitions}}.

The API shape would look like:

*Scala:*

{code}
def mapInArrow(
    f: Iterator[ArrowRecordBatch] => Iterator[ArrowRecordBatch],
    schema: StructType): DataFrame = {
  // ...
}
{code}

{code}
df.mapInArrow(_.map { case arrowBatch: ArrowRecordBatch =>
  // do something with `ArrowRecordBatch` and create new `ArrowRecordBatch`.
  // ...
  arrowBatch
}, df.schema).show()
{code}


*Python:*

{code}
def mapInArrow(
        self,
        func: Callable[Iterator[pyarrow.RecordBatch], Iterator[pyarrow.RecordBatch]],
        schema: StructType) -> DataFrame:
    # ...
{code}

{code}
def do_something(iterator):
    for arrow_batch in iterator:
        # do something with `pyarrow.RecordBatch` and create new `pyarrow.RecordBatch`.
        # ...
        yield arrow_batch

df.mapInPandas(do_something, df.schema).show()
{code}
Bug: SPJ: Convert V2 Transform expressions into catalyst expressions and load their associated functions from V2 FunctionCatalog
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Nov/21 23:49
Affects Version/s: 3.3.0

We need to add logic to convert a V2 {{Transform}} expression into its catalyst expression counterpart, and also load its function definition from the V2 FunctionCatalog provided.
Bug: SPJ: Introduce a new DataSource V2 interface HasPartitionKey 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/21 23:37
Affects Version/s: 3.3.0

One of the pre-requisite for the feature is to allow V2 input partitions to report their partition values to Spark, which can use them to compare if both sides of join are co-partitioned, and also optionally group input partitions together.
Bug: Catalyst optimizer very time-consuming and memory-intensive with some "explode(array)" 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 19:58
Affects Version/s: 3.1.2

The problem occurs with the simple code below:
{code:java}
import session.implicits._

Seq(
  (1, "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x")
).toDF()
  .checkpoint() // or save and reload to truncate lineage
  .createOrReplaceTempView("sub")

session.sql("""
  SELECT
    *
  FROM
  (
    SELECT
      EXPLODE( ARRAY( * ) ) result
    FROM
    (
      SELECT
        _1 a, _2 b, _3 c, _4 d, _5 e, _6 f, _7 g, _8 h, _9 i, _10 j, _11 k, _12 l, _13 m, _14 n, _15 o, _16 p, _17 q, _18 r, _19 s, _20 t, _21 u
      FROM
        sub
    )
  )
  WHERE
    result != ''
  """).show() {code}
It takes several minutes and a very high Java heap usage, when it should be immediate.

It does not occur when replacing the unique integer value (1) with a string value ({_}"x"{_}).

All the time is spent in the _PruneFilters_ optimization rule.

Not reproduced in Spark 2.4.1.
Bug: In AQE, wrong InMemoryRelation size estimation causes "Cannot broadcast the table that is larger than 8GB: 8 GB" failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/21 19:01
Affects Version/s: 3.1.1

There is a period in time where an InMemoryRelation will have the cached buffers loaded, but the statistics will be inaccurate (anywhere between 0 -> size in bytes reported by accumulators). When AQE is enabled, it is possible that join planning strategies will happen in this window. In this scenario, join children sizes including InMemoryRelation are greatly underestimated and a broadcast join can be planned when it shouldn't be. We have seen scenarios where a broadcast join is planned with the builder size greater than 8GB because at planning time, the optimizer believes the InMemoryRelation is 0 bytes.

Here is an example test case where the broadcast threshold is being ignored. It can mimic the 8GB error by increasing the size of the tables.
{code:java}
withSQLConf(
  SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
  SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "1048584") {
  // Spark estimates a string column as 20 bytes so with 60k rows, these relations should be
  // estimated at ~120m bytes which is greater than the broadcast join threshold
  Seq.fill(60000)("a").toDF("key")
    .createOrReplaceTempView("temp")
  Seq.fill(60000)("b").toDF("key")
    .createOrReplaceTempView("temp2")

  Seq("a").toDF("key").createOrReplaceTempView("smallTemp")
  spark.sql("SELECT key as newKey FROM temp").persist()

  val query =
  s"""
     |SELECT t3.newKey
     |FROM
     |  (SELECT t1.newKey
     |  FROM (SELECT key as newKey FROM temp) as t1
     |        JOIN
     |        (SELECT key FROM smallTemp) as t2
     |        ON t1.newKey = t2.key
     |  ) as t3
     |  JOIN
     |  (SELECT key FROM temp2) as t4
     |  ON t3.newKey = t4.key
     |UNION
     |SELECT t1.newKey
     |FROM
     |    (SELECT key as newKey FROM temp) as t1
     |    JOIN
     |    (SELECT key FROM temp2) as t2
     |    ON t1.newKey = t2.key
     |""".stripMargin
  val df = spark.sql(query)
  df.collect()
  val adaptivePlan = df.queryExecution.executedPlan
  val bhj = findTopLevelBroadcastHashJoin(adaptivePlan)
  assert(bhj.length == 1) {code}
Bug: Add Weight of Evidence and Information value to ml.feature
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Nov/21 07:55
Affects Version/s: 3.2.0

The weight of evidence (WOE) and information value (IV) provide a great framework for exploratory analysis and variable screening for binary classifiers as well as beneficial and help us analyze multiple points as listed below:

1. Helps check the linear relationship of a feature with its dependent feature to be used in the model.

2. Is a good variable transformation method for both continuous and categorical features.

3. Is better than on-hot encoding as this method of variable transformation does not increase the complexity of the model.

4. Detect linear and non-linear relationships.

5. Be useful in feature selection.

6. Is a good measure of the predictive power of a feature and it also helps point out the suspicious feature.
Bug: Support mapreduce.job.send-token-conf when starting containers in YARN
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/21 23:45
Affects Version/s: 3.3.0

{{mapreduce.job.send-token-conf}} is a useful feature in Hadoop (see [YARN-5910|https://issues.apache.org/jira/browse/YARN-5910] with which RM is not required to statically have config for all the secure HDFS clusters. Currently it only works for MRv2 but it'd be nice if Spark can also use this feature. I think we only need to pass the config to {{LaunchContainerContext}} in {{Client.createContainerLaunchContext}}.
Bug: support AS OF syntax
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/21 21:52
Affects Version/s: 3.3.0

https://docs.databricks.com/delta/quick-start.html#query-an-earlier-version-of-the-table-time-travel

Delta Lake time travel allows user to query an older snapshot of a Delta table. To query an older version of a table, user needs to specify a version or timestamp in a SELECT statement using AS OF syntax as the follows

SELECT * FROM default.people10m VERSION AS OF 0;

SELECT * FROM default.people10m TIMESTAMP AS OF '2019-01-29 00:37:58';

This ticket is opened to add AS OF syntax in Spark
Bug: Inline type hint files for files in python/pyspark/mllib
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/21 09:47
Affects Version/s: 3.3.0

Currently there are type hint stub files ({{*.pyi}}) to show the expected types for functions, but we can also take advantage of static type checking within the functions by inlining the type hints.
Bug: Read/Write Timestamp ntz from/to Orc uses int64
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/21 12:43
Affects Version/s: 3.3.0

There are some example code:
import java.util.TimeZone

TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"))
sql("set spark.sql.session.timeZone=America/Los_Angeles")

val df = sql("select timestamp_ntz '2021-06-01 00:00:00' ts_ntz, timestamp '2021-06-01 00:00:00' ts")

df.write.mode("overwrite").orc("ts_ntz_orc")
df.write.mode("overwrite").parquet("ts_ntz_parquet")
df.write.mode("overwrite").format("avro").save("ts_ntz_avro")

val query = """
  select 'orc', *
  from `orc`.`ts_ntz_orc`
  union all
  select 'parquet', *
  from `parquet`.`ts_ntz_parquet`
  union all
  select 'avro', *
  from `avro`.`ts_ntz_avro`
"""

val tzs = Seq("America/Los_Angeles", "UTC", "Europe/Amsterdam")
for (tz <- tzs) {
  TimeZone.setDefault(TimeZone.getTimeZone(tz))
  sql(s"set spark.sql.session.timeZone=$tz")

  println(s"Time zone is ${TimeZone.getDefault.getID}")
  sql(query).show(false)
}

The output show below looks so strange.
Time zone is America/Los_Angeles
+-------+-------------------+-------------------+
|orc    |ts_ntz             |ts                 |
+-------+-------------------+-------------------+
|orc    |2021-06-01 00:00:00|2021-06-01 00:00:00|
|parquet|2021-06-01 00:00:00|2021-06-01 00:00:00|
|avro   |2021-06-01 00:00:00|2021-06-01 00:00:00|
+-------+-------------------+-------------------+

Time zone is UTC
+-------+-------------------+-------------------+
|orc    |ts_ntz             |ts                 |
+-------+-------------------+-------------------+
|orc    |2021-05-31 17:00:00|2021-06-01 00:00:00|
|parquet|2021-06-01 00:00:00|2021-06-01 07:00:00|
|avro   |2021-06-01 00:00:00|2021-06-01 07:00:00|
+-------+-------------------+-------------------+

Time zone is Europe/Amsterdam
+-------+-------------------+-------------------+
|orc    |ts_ntz             |ts                 |
+-------+-------------------+-------------------+
|orc    |2021-05-31 15:00:00|2021-06-01 00:00:00|
|parquet|2021-06-01 00:00:00|2021-06-01 09:00:00|
|avro   |2021-06-01 00:00:00|2021-06-01 09:00:00|
+-------+-------------------+-------------------+
Bug: Build and test on Python 3.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/21 21:01
Affects Version/s: 3.3.0

Python 3.10 introduced breaking changes. We need to update PySpark.
- https://docs.python.org/3/whatsnew/3.10.html

For example, the following.
{code}
>>> from collections import Callable
<stdin>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
{code}

This is targeting Apache Spark 3.3.0 in 2022.
Bug: Disappearance of skipped stages mislead the bug hunting 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/21 07:41
Affects Version/s: 3.1.2

# 
 ## With FetchFailedException and Map Stage Retries

When rerunning spark-sql shell with the original SQL in [https://gist.github.com/yaooqinn/6acb7b74b343a6a6dffe8401f6b7b45c#gistcomment-3977315]

!https://user-images.githubusercontent.com/8326978/143821530-ff498caa-abce-483d-a24b-315aacf7e0a0.png!

1. stage 3 threw FetchFailedException and caused itself and its parent stage(stage 2) to retry
2. stage 2 was skipped before but its attemptId was still 0, so when its retry happened it got removed from `Skipped Stages` 

The DAG of Job 2 doesn't show that stage 2 is skipped anymore.

!https://user-images.githubusercontent.com/8326978/143824666-6390b64a-a45b-4bc8-b05d-c5abbb28cdef.png!

Besides, a retried stage usually has a subset of tasks from the original stage. If we mark it as an original one, the metrics might lead us into pitfalls.
Bug: Optimize write path on RocksDB state store provider
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/21 06:34
Affects Version/s: 3.3.0

We figured out that RocksDB class does additional lookup on the key for write operations (put/delete) to track the number of rows. This is required to fulfill the metric of the state store, but after benchmarking it turns out performance hit is significant.

We can't find a good alternative to retain the number of rows without additional lookup, so we are proposing a new config to flag tracking the number of rows.
 * *config name: spark.sql.streaming.stateStore.rocksdb.trackTotalNumberOfRows*

 * *default value: true* (since we already serve the number and we want to avoid breaking change)

*We will give "0" for the number of keys in the state store metric when the config is turned off.* The ideal value seems to be a negative one, but currently SQL metric doesn't allow negative value and there seems to be some technical/historical issue not to.

*We will also handle the case the config is flipped during restart* - this will enable the way end users enjoy the benefit but also not lose the chance to know the number of state rows. End users can turn off the flag to maximize the performance, and turn on the flag (restart required) when they want to see the actual number of keys (for observability/debugging/etc).
Bug: Inline type hints for python/pyspark/mllib/linalg/distributed.py
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/21 12:48
Affects Version/s: 3.3.0

Inline type hints from python/pyspark/mlib/linalg/distributed.pyi to python/pyspark/mllib/linalg/distributed.py