av9ash/GitBugs huggingface dataset:
Bug: JsonSource's infer should have the same exception handle logic as JacksonParser's parse logic
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Nov/21 08:03
Affects Version/s: 3.3.0

JacksonParser's exception handle logic is different with org.apache.spark.sql.catalyst.json.JsonInferSchema#infer logic, the different can be saw as below:
{code:java}
// code JacksonParser's parse
try {
      Utils.tryWithResource(createParser(factory, record)) { parser =>
        // a null first token is equivalent to testing for input.trim.isEmpty
        // but it works on any token stream and not just strings
        parser.nextToken() match {
          case null => None
          case _ => rootConverter.apply(parser) match {
            case null => throw QueryExecutionErrors.rootConverterReturnNullError()
            case rows => rows.toSeq
          }
        }
      }
    } catch {
      case e: SparkUpgradeException => throw e
      case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
        // JSON parser currently doesn't support partial results for corrupted records.
        // For such records, all fields other than the field configured by
        // `columnNameOfCorruptRecord` are set to `null`.
        throw BadRecordException(() => recordLiteral(record), () => None, e)
      case e: CharConversionException if options.encoding.isEmpty =>
        val msg =
          """JSON parser cannot handle a character in its input.
            |Specifying encoding as an input option explicitly might help to resolve the issue.
            |""".stripMargin + e.getMessage
        val wrappedCharException = new CharConversionException(msg)
        wrappedCharException.initCause(e)
        throw BadRecordException(() => recordLiteral(record), () => None, wrappedCharException)
      case PartialResultException(row, cause) =>
        throw BadRecordException(
          record = () => recordLiteral(record),
          partialResult = () => Some(row),
          cause)
    }
{code}
v.s. 
{code:java}
// JsonInferSchema's infer logic
    val mergedTypesFromPartitions = json.mapPartitions { iter =>
      val factory = options.buildJsonFactory()
      iter.flatMap { row =>
        try {
          Utils.tryWithResource(createParser(factory, row)) { parser =>
            parser.nextToken()
            Some(inferField(parser))
          }
        } catch {
          case  e @ (_: RuntimeException | _: JsonProcessingException) => parseMode match {
            case PermissiveMode =>
              Some(StructType(Seq(StructField(columnNameOfCorruptRecord, StringType))))
            case DropMalformedMode =>
              None
            case FailFastMode =>
              throw QueryExecutionErrors.malformedRecordsDetectedInSchemaInferenceError(e)
          }
        }
      }.reduceOption(typeMerger).toIterator
    }

{code}
They should have the same exception handle logic, otherwise it may confuse user because of the inconsistency.
Bug: expose driver gc time and duration time
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Nov/21 02:19
Affects Version/s: 3.2.1

when we browse executor pages of driver side or history side sparkUI,driver's gc statistics is not acquired obviously ,thus making it hard to deside how to config the resource of driver.

 

we can use the application time as driver's task duration time, and use "TotalGCTime" to the JVMHeapMemory of ExecutorMetricType
Bug: Unified "fetchWaitTime" and "shuffleReadTime" metrics On UI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Nov/21 04:15
Affects Version/s: 3.2.0

Metrics in Executor/Task page shown as "

Shuffle Read Block Time", and the SQL page shown as "fetch wait time" which make us confused  !executor-page.png!

!sql-page.png!
Bug: ALTER (DATABASE|SCHEMA|NAMESPACE) ... SET LOCATION command not documented
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/21 08:32
Affects Version/s: 3.2.0

The instruction of {color:#ff0000}ALTER DATABASE ... SET LOCATION ... {color:#172b4d}command{color}{color} should be documented in a sql-ref-syntax-ddl-create-table page.
Bug: yarn-client mode client's appid value is null
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/21 10:57
Affects Version/s: 3.2.0

In yarn-client mode, *Client.appId* variable is not assigned, it is always {*}null{*}, in cluster mode, this variable will be assigned to the true value. In this patch, we assign true application id to `appId` too.
Bug: TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
Status: In Progress | Priority: Minor | Resolution: 
Created: 21/Nov/21 23:16
Affects Version/s: 3.2.0

TimeZoneAwareExpression like hour, date_format etc. throws 

NoSuchElementException: None.get on expr.eval()

*hour(current_timestamp).expr.eval()*

*date_format(current_timestamp, "dd").expr.eval()*

 
{code:java}
java.util.NoSuchElementException: None.get
  at scala.None$.get(Option.scala:529)
  at scala.None$.get(Option.scala:527)
  at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:53)
  at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:53)
  at org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId$lzycompute(datetimeExpressions.scala:772)
  at org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId(datetimeExpressions.scala:772)
  at org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter(datetimeExpressions.scala:70)
  at org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter$(datetimeExpressions.scala:67)
  at org.apache.spark.sql.catalyst.expressions.DateFormatClass.getFormatter(datetimeExpressions.scala:772)
  at org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.$anonfun$formatterOption$1(datetimeExpressions.scala:64)
{code}
Bug: Buggy method retrival in pyspark.docs.conf.setup
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Nov/21 14:06
Affects Version/s: 3.1.0

[Currently we have this code|https://github.com/apache/spark/blob/04af08e9b6d692ed4b5df5581e39de138067211c/python/docs/source/conf.py#L374-L376]

{code:python}
def setup(app):
    # The app.add_javascript() is deprecated.
    getattr(app, "add_js_file", getattr(app, "add_javascript"))('copybutton.js')
{code}

where nested {{getattr}} should handle compatibility issues between different Sphinx versions.

However, {{getattr(app, "add_javascript"))}} is missing {{default}} and will fail in latest Sphinx version, without ever falling back to {{getattr(app, "add_js_file")}}
Bug: Allow nondeterministic expression in aggregate function
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/21 07:06
Affects Version/s: 3.3.0

Nondeterministic expression in aggregate function is not allow in spark, so we cannot execute query like
{code:java}
SELECT COUNT(RANDOM());
{code}
and raise \{{nondeterministic expression ... should not appear in the arguments of an aggregate function. }}error message.
[related code section|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L298]

 

Hence other DB like PostgreSQL, we can call the SQL.
{code:java}
postgres=# SELECT COUNT(RANDOM());
 count
-------
     1
(1 row) {code}
 

I tried to remove the error message section, then I found spark could execute the query. 
{code:java}
scala> spark.sql("SELECT COUNT(RANDOM())").show()
+-------------+
|count(rand())|
+-------------+
|            1|
+-------------+ {code}
 

It could be useful for spark users to be able to execute those kinds of queries because they can simply call
{code:java}
spark.sql("SELECT COUNT(DISTINCT(INPUT_FILE_NAME())) FROM table WHERE ...") {code}
to find target files, for example.
Bug: Add SQL configs to control newly added join code-gen in 3.3
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Nov/21 07:02
Affects Version/s: 3.3.0

During Spark 3.3, we added code-gen for FULL OUTER shuffled hash join, FULL OUTER sort merge join, and Existence sort merge join. Given the join test coverage is not high, and we would love to avoid any upcoming release regression due to it. So here we introduce three internal configs to allow users and developers to disable code-gen in case we found any bug after release.
Bug: Introduce Z-order for efficient data skipping
Status: Open | Priority: Minor | Resolution: 
Created: 18/Nov/21 02:06
Affects Version/s: 3.3.0

This is the umbrella Jira to track the progress of introducing Z-order in Spark. Z-order enables to sort tuples in a way, to allow efficiently data skipping for columnar file format (Parquet and ORC).

For query with filter on combination of multiple columns, example:
{code:java}
SELECT *
FROM table
WHERE x = 0 OR y = 0
{code}
Parquet/ORC cannot skip file/row-groups efficiently when reading, even though the table is sorted (locally or globally) on any columns. However when table is Z-order sorted on multiple columns, Parquet/ORC can skip file/row-groups efficiently when reading.

We should add the feature in Spark to allow OSS Spark users benefitted in running these queries.

 

Reference for other systems support Z-order:
 * Databricks Delta Lake:
 ** [https://databricks.com/blog/2018/07/31/processing-petabytes-of-data-in-seconds-with-databricks-delta.html]
 ** [https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-optimize.html]
 * Presto:
 ** [https://github.com/prestodb/presto/blob/master/presto-hive-common/src/main/java/com/facebook/presto/hive/zorder/ZOrder.java]
 * Impala:
 ** https://issues.apache.org/jira/browse/IMPALA-8755 
 * AWS:
 ** [https://aws.amazon.com/blogs/database/z-order-indexing-for-multifaceted-queries-in-amazon-dynamodb-part-1/]
 ** [https://aws.amazon.com/blogs/database/z-order-indexing-for-multifaceted-queries-in-amazon-dynamodb-part-2/]
 ** [https://cdn2.hubspot.net/hubfs/392937/Whitepaper/WP/interleaved_keys_v12_1.pdf]
Bug: pandas `convert_dtypes` method support
Status: Open | Priority: Minor | Resolution: 
Created: 15/Nov/21 13:30
Affects Version/s: 3.2.0

Support for the {{convert_dtypes}} method as part of the new pandas API in pyspark?

[https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.convert_dtypes.html]
Bug: Avoid unnecessary buffer and copy in full outer sort merge join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Nov/21 00:02
Affects Version/s: 3.3.0

FULL OUTER sort merge join (non-code-gen path) copies join keys and buffers input rows, even when rows from both sides do have matched keys ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L1637-L1641] ). This is unnecessary, as we can just output the row with smaller join keys, and only buffer when both sides have matched keys. This would save us from unnecessary copy and buffer, when both join sides have a lot of rows not matched with each other.
Bug: Clarify output of FPGrowth
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Nov/21 17:04
Affects Version/s: 3.2.0

The association rules returned by FPGrow include more columns than are documented, like {{{}lift{}}}:

[https://spark.apache.org/docs/latest/ml-frequent-pattern-mining.html]

We should offer a basic description of these columns. An _itemset_ should also be briefly defined.
Bug: Support Decimal RoundingMode.UP, DOWN, HALF_DOWN
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Nov/21 21:54
Affects Version/s: 3.2.0

Currently we support only Decimal RoundingModes : HALF_UP (round) and HALF_EVEN (bround). But we have use cases that needs RoundingMode.UP and RoundingMode.DOWN. In our projects we use UDF, i also see few people do complex operations to do the same with spark native methods.

[https://stackoverflow.com/questions/34888419/round-down-double-in-spark/40476117]

[https://stackoverflow.com/questions/54683066/is-there-a-rounddown-function-in-sql-as-there-is-in-excel]

[https://stackoverflow.com/questions/48279641/oracle-sql-round-half]

 

Opening support for the other rounding modes might interest a lot of use cases. 
*SAP Hana Sql ROUND function does it :* 
{code:java}
ROUND(<number> [, <position> [, <rounding_mode>]]){code}
REF : [https://help.sap.com/viewer/7c78579ce9b14a669c1f3295b0d8ca16/Cloud/en-US/20e6a27575191014bd54a07fd86c585d.html]


*Sql Server does something similar to this* :
{code:java}
ROUND ( numeric_expression , length [ ,function ] ){code}
REF : [https://docs.microsoft.com/en-us/sql/t-sql/functions/round-transact-sql?view=sql-server-ver15]
Bug: Delete py_container_checks.zip after the test in DepsTestsSuite finishes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Nov/21 08:52
Affects Version/s: 3.2.0

When K8s integration tests run, py_container_checks.zip  still remains in resource-managers/kubernetes/integration-tests/tests/.
It's is created in the test "Launcher python client dependencies using a zip file" in DepsTestsSuite.
Bug: Don't obtain JDBC connection for empty partition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Nov/21 14:48
Affects Version/s: 3.2.0

In JdbcUtils.scala:savePartition, the code obtains a database connection even when there is no data in the partition. It works correctly and returns it, but, may be advantageous to simply skip that and other overhead if nothing will be written at all.
Bug: Capture SQL statement executed at source in spark debug log
Status: Open | Priority: Minor | Resolution: 
Created: 12/Nov/21 03:17
Affects Version/s: 3.1.0

Hello,

  It would be great if there is an option to print sql statement in executor log which sent to source system upon enabling debug or some setting. 

Example,

 I do not have access to mssql server system table hence, I do not know what query send to JDBC server and I could not find any option to enable it.
Bug: Not log empty aggregate and group by in JDBCScan
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Nov/21 23:04
Affects Version/s: 3.2.0

Address the comment https://github.com/apache/spark/pull/34451#discussion_r740220800

Current behavior:

{code:java}
Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1@72e75786 [NAME#1,SALARY#2] PushedAggregates: [], PushedFilters: [IsNotNull(SALARY), GreaterThan(SALARY,100.00)], PushedGroupby: [], ReadSchema: struct<NAME:string,SALARY:decimal(20,2)>
{code}


After the fix, it will be

{code:java}
Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1@72e75786 [NAME#1,SALARY#2] PushedFilters: [IsNotNull(SALARY), GreaterThan(SALARY,100.00)], ReadSchema: struct<NAME:string,SALARY:decimal(20,2)>
{code}
Bug: Exclude hadoop-client-api transitive dependency from orc-core
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Nov/21 04:56
Affects Version/s: 3.3.0

Like hadoop-common and hadoop-hdfs, this PR proposes to exclude hadoop-client-api transitive dependency from orc-core.
Why are the changes needed?

Since Apache Hadoop 2.7 doesn't work on Java 17, Apache ORC has a dependency on Hadoop 3.3.1.
This causes test-dependencies.sh failure on Java 17. As a result, run-tests.py also fails.
Bug: The partitionOverwriteMode option is not respected when using insertInto
Status: Open | Priority: Minor | Resolution: 
Created: 10/Nov/21 11:56
Affects Version/s: 3.2.0

From the documentation of the {{spark.sql.sources.partitionOverwriteMode}} configuration option:
{quote}This can also be set as an output option for a data source using key partitionOverwriteMode (which takes precedence over this setting), e.g. dataframe.write.option("partitionOverwriteMode", "dynamic").save(path).
{quote}
This is true when using .save(), however .insertInto() does not respect the output option.
Bug: Ignore test_memory_limit on non-Linux environment
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Nov/21 01:31
Affects Version/s: 3.2.0

{code}
$ build/sbt -Phadoop-cloud -Phadoop-3.2 test:package
$ python/run-tests
...
======================================================================
FAIL: test_memory_limit (pyspark.tests.test_worker.WorkerMemoryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dongjoon/APACHE/spark-merge/python/pyspark/tests/test_worker.py", line 212, in test_memory_limit
    self.assertEqual(soft_limit, 2 * 1024 * 1024 * 1024)
AssertionError: 9223372036854775807 != 2147483648

----------------------------------------------------------------------
{code}
Bug: More descriptions and adding an image to the failure message about enabling GitHub Actions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Nov/21 04:40
Affects Version/s: 3.3.0

I've seen and experienced that the build-and-test workflow of first-time PRs fails and it was caused by developers forgetting to enable Github Actions on their own repositories.

I think developers will be able to notice the cause quicker by adding more descriptions and an image to the test-failure message.
Bug: Update Apache Parent POM version to 24 in the pom.xml
Status: Open | Priority: Minor | Resolution: 
Created: 03/Nov/21 15:16
Affects Version/s: 3.2.0

Some nice things about the version 24:

1. Deploy SHA-512 for source-release to Remote Repository
2. Reproducible builds option

 

Resources:

[1] [https://lists.apache.org/thread/9wk97dwjlcoxlk1onxotfo8k98b2v0sk]
[2] [https://maven.apache.org/guides/mini/guide-reproducible-builds.html]
[3] [https://github.com/apache/maven-apache-parent/compare/apache-18...apache-24diff]
Bug: Replace map with expressions which produce no result with foreach 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Nov/21 11:31
Affects Version/s: 3.3.0

Use foreach instead of  map with expressions which produce no result.

 

Before

 
{code:java}
def functionWithNoReturnValue: Unit = {}  
Seq(1, 2).map(functionWithNoReturnValue) {code}
 

 

After

 
{code:java}
def functionWithNoReturnValue: Unit = {}   
Seq(1, 2).foreach(functionWithNoReturnValue) {code}
Bug: The number of dynamic partitions should early check when writing to external tables
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 05/Nov/21 10:58
Affects Version/s: 3.2.0

[SPARK-29295|https://issues.apache.org/jira/browse/SPARK-29295] introduces a mechanism that writes to external tables is a dynamic partition method, and the data in the target partition will be deleted first.

Assuming that 1001 partitions are written, the data of 10001 partitions will be deleted first, but because hive.exec.max.dynamic.partitions is 1000 by default, loadDynamicPartitions will fail at this time, but the data of 1001 partitions has been deleted.
Bug: Fix the format of the document
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Nov/21 11:31
Affects Version/s: 3.2.0

Currently, the format of `sql-ref-syntax-dml-insert-table.md` is: [currently format|https://spark.apache.org/docs/latest/sql-ref-syntax-dml-insert-into.html]

it should be normal like other parts.
Bug: ignoreCorruptFiles results data loss
Status: In Progress | Priority: Critical | Resolution: 
Created: 28/Sep/22 01:20
Affects Version/s: 3.1.0

Let's take a look at the case below, the left and the right are visiting the same table and its partitions, and both of them are ignoreCorruptFiles=true. The right side shows that a task skips partial of data it reads because of encountering 'corrupt data', while the left read this file correctly. As ignoreCorruptFiles coarsely works with RuntimeException and IOException, it can not always represent data corruption.

!image-2022-09-28-09-20-21-693.png!
Bug: SPIP: Support Docker Official Image for Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/22 06:44
Affects Version/s: 3.5.0

This SPIP is proposed to add [Docker Official Image(DOI)|https://github.com/docker-library/official-images] to ensure the Spark Docker images meet the quality standards for Docker images, to provide these Docker images for users who want to use Apache Spark via Docker image.

There are also several [Apache projects that release the Docker Official Images|https://hub.docker.com/search?q=apache&image_filter=official], such as: [flink|https://hub.docker.com/_/flink], [storm|https://hub.docker.com/_/storm], [solr|https://hub.docker.com/_/solr], [zookeeper|https://hub.docker.com/_/zookeeper], [httpd|https://hub.docker.com/_/httpd] (with 50M+ to 1B+ download for each). From the huge download statistics, we can see the real demands of users, and from the support of other apache projects, we should also be able to do it.

After support:
 * The Dockerfile will still be maintained by the Apache Spark community and reviewed by Docker.

 * The images will be maintained by the Docker community to ensure the quality standards for Docker images of the Docker community.

It will also reduce the extra docker images maintenance effort (such as frequently rebuilding, image security update) of the Apache Spark community.

 

SPIP DOC: [https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o]

DISCUSS: [https://lists.apache.org/thread/l1793y5224n8bqkp3s6ltgkykso4htb3]
Bug: PYSPARK: Observation computes the wrong results when using `corr` function 
Status: Open | Priority: Major | Resolution: 
Created: 25/Sep/22 12:10
Affects Version/s: 3.3.0

Minimalistic description of the odd computation results.

When creating a new `Observation` object and computing a simple correlation function between 2 columns, the results appear to be non-deterministic.
{code:java}
# Init
from pyspark.sql import SparkSession, Observation
import pyspark.sql.functions as F

df = spark.createDataFrame([(float(i), float(i*10),) for i in range(10)], schema="id double, id2 double")

for i in range(10):
    o = Observation(f"test_{i}")
    df_o = df.observe(o, F.corr("id", "id2").eqNullSafe(1.0))
    df_o.count()
    print(o.get)

# Results
{'(corr(id, id2) <=> 1.0)': False}
{'(corr(id, id2) <=> 1.0)': False}
{'(corr(id, id2) <=> 1.0)': False}
{'(corr(id, id2) <=> 1.0)': True}
{'(corr(id, id2) <=> 1.0)': True}
{'(corr(id, id2) <=> 1.0)': True}
{'(corr(id, id2) <=> 1.0)': True}
{'(corr(id, id2) <=> 1.0)': True}
{'(corr(id, id2) <=> 1.0)': True}
{'(corr(id, id2) <=> 1.0)': False}{code}
Bug: Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 20/Sep/22 08:57
Affects Version/s: 3.2.1

spark.sql(
      s"""
         |SELECT
         | Info ,
         | PERCENTILE_APPROX(cost,0.5) cost_p50,
         | PERCENTILE_APPROX(cost,0.9) cost_p90,
         | PERCENTILE_APPROX(cost,0.95) cost_p95,
         | PERCENTILE_APPROX(cost,0.99) cost_p99,
         | PERCENTILE_APPROX(cost,0.999) cost_p999
         |FROM
         | textData
         |""".stripMargin)
 * When we used spark 2.4.0, aggregation adopted objHashAggregator, stage 2 pull shuffle data very quick . but , when we use spark 3.2.1 and use old shuffle , 140M shuffle data cost 3 hours. 

 * If we upgrade the Shuffle, will we get performance regression？

 *