av9ash/GitBugs huggingface dataset:
Bug: Avoid distribute user jar from driver in yarn client mode
Status: In Progress | Priority: Trivial | Resolution: 
Created: 14/Oct/20 11:27
Affects Version/s: 3.0.1

{code:java}
spark-submit --master yarn --deploy-mode client --num-executors 100 big_jar.jar
{code}
When the number of applied executors is large and the jar size is large, the executor pulls the jar from the driver, and the driver network traffic is high, and a timeout may occur. The driver and the executor of the yarn cluster may not be in the same data center.
{code:java}
20/10/04 00:46:02,269 [rpc-server-3-13] ERROR TransportRequestHandler: Error sending result StreamResponse{streamId=/jars/xxx-jar-with-dependencies.jar, byteCount=145417300, body=FileSegmentManagedBuffer{file=xxx-jar-with-dependencies.jar, offset=0, length=145417300}} to /x.x.x.x:33527; closing connection

{code}
 

We can automatically add the user's jar to --jars, which can avoid this problem.
Bug: Only disable auto bucketed scan for cached query
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 06/Oct/20 08:33
Affects Version/s: 3.1.0

As a followup from discussion in [https://github.com/apache/spark/pull/29804#discussion_r500033528,] auto bucketed scan is disabled by default due to regression for cached query. Suggested by [~cloud_fan], we can enable auto bucketed scan globally with special handling for cached query, similar to adaptive execution.
Bug: Check whether the executor and external service connection is available
Status: Open | Priority: Trivial | Resolution: 
Created: 15/Oct/20 11:19
Affects Version/s: 3.0.1

At present, the executor only establishes a connection with the external shuffle service once at initialization and registers.

In yarn, nodemanager may stop working, shuffle service does not work, but the container/executor process is still executing, ShuffleMapTask can still be executed, and the returned result mapstatus is still the address of the external shuffle service
When the next stage reads shuffle data, it will not be connected to the shuffle serivce.
The final job execution failed.

The approach I thought of:
Before ShuffleMapTask starts to write data, check whether the connection is available, or regularly test whether the connection is normal, such as the driver and executor heartbeat check threads.
Bug: Expose inverse hyperbolic functions through sql.functions API
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 03/Oct/20 16:23
Affects Version/s: 2.4.0

Recent changes elsewhere have added inverse hyperbolic functions acosh, asinh, atanh to the SQL interpreter, but have not exposed the same functions through the Scala API provided via org.apache.spark.sql.functions._.

It would be desirable if these functions could be made available via the Scala API.

 

A patch for this is already available - https://github.com/apache/spark/pull/29938
Bug: Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils
Status: In Progress | Priority: Trivial | Resolution: 
Created: 10/Oct/20 11:45
Affects Version/s: 3.0.1

Currently, JdbcUtils calls executeBatch regardless of DatabaseMetaData.supportsBatchUpdates. How about executing "stmt.execute()" when the JDBC driver doesn't support batch updates as [https://github.com/apache/spark/pull/30001]?
Bug: Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 16/Apr/21 21:37
Affects Version/s: 3.0.1

I think this also shows up for all versions of Spark that pickle the data when doing a collect from python.

When you do a collect in python java will do a collect and convert the UnsafeRows into GenericRowWithSchema instances before it sends them to the Pickler. The Pickler, by default, will try to dedupe objects using hashCode and .equals for the object.  But .equals and .hashCode for GenericRowWithSchema only looks at the data, not the schema. But when we pickle the row the keys from the schema are written out.

This can result in data corruption, sort of, in a few cases where a row has the same number of elements as a struct within the row does, or a sub-struct within another struct. 

If the data happens to be the same, the keys for the resulting row or struct can be wrong.

My repro case is a bit convoluted, but it does happen.
Bug: Spark Job web page is extremely slow while there are more than 1500 events in timeline
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 26/Apr/21 07:29
Affects Version/s: 3.1.1

In a spark streaming application, there are 1000+ executors, more than 2000 events (executors events, job events) generated, then the jobs/job web page of spark is not able to show. The browser (chrome, firefox, safari) freezes. I had to open another window to open stages, executors pages by their addresses manually.

The jobs page is the home page, so this is rather annoyed. The problem is that the vis-timeline rending is too slow.

Their are some suggestion:

1) the page should not render timeline when the page is loading unless user clicks the link.

2) the executor group and job group should be separated, and user can choose to show one, both or neither.

3) the executor group should display executor event by time horizontally. Currently it is displayed executors one line by one line. If more than 100 executors, the page is not that good.

4)the vis-timeline library is not maintained anymore since 2017. Should be replaced a new one like [https://github.com/visjs/vis-timeline]

5) it is also good to get recent events, e.g. 500, and load more if user want see more, however the data could all be loaded once.
Bug: Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Apr/21 06:20
Affects Version/s: 2.4.7

SPARK-34988 upgraded Jetty to 9.4.39 for CVE-2021-28165.
But after the upgrade, Jetty 9.4.40 was released to fix the ERR_CONNECTION_RESET issue (https://github.com/eclipse/jetty.project/issues/6152).
This issue seems to affect Jetty 9.4.39 when POST method is used with SSL.
For Spark, job submission using REST and ThriftServer with HTTPS protocol can be affected.
Bug: I am facing this issue regularly, how to fix this issue.
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 20/Apr/21 10:19
Affects Version/s: 2.2.2

21/04/19 21:02:11 WARN hdfs.DataStreamer: Exception for BP-823308525-10.56.47.77-1544458538172:blk_1170699623_96969312
 java.io.EOFException: Unexpected EOF while trying to read response from server
 at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:448)
 at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
 at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)
 21/04/19 21:04:01 WARN hdfs.DataStreamer: Error Recovery for BP-823308525-10.56.47.77-1544458538172:blk_1170699623_96969312 in pipeline [DatanodeInfoWithStorage[10.34.39.42:9866,DS-0ad94d03-fa3f-486b-b204-3e8d2df91f17,DISK], DatanodeInfoWithStorage[10.56.47.67:9866,DS-c28dab54-8fa0-4a49-80ec-345cc0cc52bd,DISK], DatanodeInfoWithStorage[10.56.47.55:9866,DS-79f5dd22-d0bc-4fe0-8e50-8a570779de17,DISK]]: datanode 0(DatanodeInfoWithStorage[10.56.47.36:9866,DS-0ad94d03-fa3f-486b-b204-3e8d2df91f17,DISK]) is bad.
Bug: Leverage the resources in the forked repository in GitHub Actions build
Status: Resolved | Priority: Critical | Resolution: Done
Created: 18/Apr/21 02:58
Affects Version/s: 3.0.2

We're sharing resources allocated to one AS organisation, across all ASF projects.
Now the builds sometimes have to wait up to 6 ~ 8 hours.
This JIRA targets to run the workflows in forked repositories and distributes the overhead.
Bug: Install PyArrow and pandas on Jenkins
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 01/Apr/21 03:54
Affects Version/s: 3.2.0

Looks like Jenkins mahcines don't have pandas and PyArrow (ever since it got upgraded?) which result in skipping related tests in PySpark, see also https://github.com/apache/spark/pull/31470#issuecomment-811618571

It would be great if we can install both in Python 3.6 on Jenkins.
Bug: ANSI SQL compliance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/21 13:26
Affects Version/s: 3.0.0

Build an ANSI compliant dialect in Spark, for better data quality and easier migration from traditional DBMS to Spark. For example, Spark will throw an exception at runtime instead of returning null results when the inputs to a SQL operator/function are invalid. 

The new dialect is controlled by SQL Configuration `spark.sql.ansi.enabled`:
{code:java}
-- `spark.sql.ansi.enabled=true`
SELECT 2147483647 + 1;
java.lang.ArithmeticException: integer overflow

-- `spark.sql.ansi.enabled=false`
SELECT 2147483647 + 1;
+----------------+
|(2147483647 + 1)|
+----------------+
|     -2147483648|
+----------------+
{code}

Full details of this dialect are documented in [https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html|https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html].

Note that some ANSI dialect features maybe not from the ANSI SQL standard directly, but their behaviors align with ANSI SQL's style.
Bug: Memory leak when dataset is being persisted
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/21 20:02
Affects Version/s: 3.1.1

If a Java- or Scala-application with SparkSession runs a long time and persists a lot of datasets, it can crash because of a memory leak.
 I've noticed the following. When we have a dataset and persist it, the SparkSession used to load that dataset is cloned in CacheManager, and this clone is added as a listener to `listenersPlusTimers` in `ListenerBus`. But this clone isn't removed from the list of listeners after that, e.g. unpersisting the dataset. If we persist a lot of datasets, the SparkSession is cloned and added to `ListenerBus` many times. This leads to a memory leak since the `listenersPlusTimers` list become very large.

I've found out that the SparkSession is cloned is CacheManager when the parameters `spark.sql.sources.bucketing.autoBucketedScan.enabled` and `spark.sql.adaptive.enabled` are true. The first one is true by default, and this default behavior leads to the problem. When auto bucketed scan is disabled, the SparkSession isn't cloned, and there are no duplicates in ListenerBus, so the memory leak doesn't occur.

Here is a small Java application to reproduce the memory leak: [https://github.com/iamelin/spark-memory-leak]
Bug: Handle ANSI intervals in WindowExecBase
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/21 09:59
Affects Version/s: 3.2.0

Handle YearMonthIntervalType and DayTimeIntervalType in createBoundOrdering():
https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala#L97-L99
Bug: Add checksum for shuffle blocks
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Apr/21 16:17
Affects Version/s: 3.2.0

Shuffle data corruption is a long-standing issue in Spark. For example, in SPARK-18105, people continually reports corruption issue. However, data corruption is difficult to reproduce in most cases and even harder to tell the root cause. We don't know if it's a Spark issue or not. With the checksum support for the shuffle, Spark itself can at least distinguish the cause between disk and network, which is very important for users.
Bug: Error-handling SQL functions for ANSI SQL mode migration
Status: Resolved | Priority: Major | Resolution: Done
Created: 21/Apr/21 04:47
Affects Version/s: 3.2.0

Create new Error-handling version SQL functions for existing SQL functions/operators, which returns NULL if overflow/error occurs. So that:
1. Users can manage to finish queries without interruptions in ANSI mode.
2. Users can get NULLs instead of unreasonable results if overflow occurs when ANSI mode is off.
For example, the behavior of the following SQL operations is unreasonable:
{code:java}
2147483647 + 2 => -2147483647
CAST(2147483648L AS INT) => -2147483648
{code}
With the new safe version SQL functions:
{code:java}
TRY_ADD(2147483647, 2) => null
TRY_CAST(2147483648L AS INT) => null
{code}
Bug: Port Koalas miscellaneous unit tests into PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/21 16:44
Affects Version/s: 3.2.0

This JIRA aims to port Koalas miscellaneous unit tests to [PySpark tests|https://github.com/apache/spark/tree/master/python/pyspark/tests].
Bug: Revisit pandas-on-Spark test cases that are disabled because of pandas nondeterministic return values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 18:18
Affects Version/s: 3.2.0

Some test cases have been disabled in the places as shown below because of pandas nondeterministic return values:
 * pandas returns `None` or `nan` randomly
python/pyspark/pandas/tests/test_series.py test_astype

 * pandas returns `True` or `False` randomly
python/pyspark/pandas/tests/indexes/test_base.py test_monotonic

We should revisit them later.
Bug: Port Koalas operations on different frames tests into PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/21 16:34
Affects Version/s: 3.2.0

This JIRA aims to port Koalas operations on different frames related unit tests to [PySpark tests|https://github.com/apache/spark/tree/master/python/pyspark/tests].
Bug: Consolidate PySpark testing utils
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/21 00:51
Affects Version/s: 3.2.0

`python/pyspark/pandas/testing` hold test utilites for pandas-on-spark, and `python/pyspark/testing` contain test utilities for pyspark. Consolidating them makes code cleaner and easier to maintain.
Bug: Promote KubernetesUtils to DeveloperApi
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/21 06:12
Affects Version/s: 3.2.0

Since SPARK-22757, `KubernetesUtils` is used as an important utility class for all K8s module. This issue aims to promote it to DeveloperApi to maintain it officially in a backward compatible way.
Bug: Spark from_json(JsonToStruct)  function return wrong value in permissive mode in case best effort
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Apr/21 17:15
Affects Version/s: 3.0.2

I use spark 3.1.1 and 3.0.2.

 Function `from_json` return wrong result with Permissive mode.
 In corner case:
 1. Json message has complex nested structure
 \{sameNameField)damaged, nestedVal:{badSchemaNestedVal, sameNameField_WhichValueWillAppearInwrongPlace}}
 2. Nested -> Nested Field: Schema is satisfy align with value in json.

scala code to reproduce:
{code:java}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.from_json
import org.apache.spark.sql.types.IntegerType
import org.apache.spark.sql.types.StringType
import org.apache.spark.sql.types.StructField
import org.apache.spark.sql.types.StructType

object Main {

  def main(args: Array[String]): Unit = {
    implicit val spark: SparkSession = SparkSession.builder().master("local[*]").getOrCreate()
    import spark.implicits._

    val schemaForFieldWhichWillHaveWrongValue = StructField("problematicName", StringType, nullable = true)
    val nestedFieldWhichNotSatisfyJsonMessage = StructField(
      "badNestedField",
      StructType(Seq(StructField("SomethingWhichNotInJsonMessage", IntegerType, nullable = true)))
    )
    val nestedFieldWithNestedFieldWhichNotSatisfyJsonMessage =
      StructField(
        "nestedField",
        StructType(Seq(nestedFieldWhichNotSatisfyJsonMessage, schemaForFieldWhichWillHaveWrongValue))
      )
    val customSchema = StructType(Seq(
      schemaForFieldWhichWillHaveWrongValue,
      nestedFieldWithNestedFieldWhichNotSatisfyJsonMessage
    ))

    val jsonStringToTest =
      """{"problematicName":"ThisValueWillBeOverwritten","nestedField":{"badNestedField":"14","problematicName":"thisValueInTwoPlaces"}}"""
    val df = List(jsonStringToTest)
      .toDF("json")
      // issue happen only in permissive mode during best effort
      .select(from_json($"json", customSchema).as("toBeFlatten"))
      .select("toBeFlatten.*")
    df.show(truncate = false)

    assert(
      df.select("problematicName").as[String].first() == "ThisValueWillBeOverwritten",
      "wrong value in root schema, parser take value from column with same name but in another nested elvel"
    )
  }

}

{code}
I was not able to debug this issue, to find the exact root cause.
 But what I find in debug, that In `org.apache.spark.sql.catalyst.util.FailureSafeParser` in line 64. code block `e.partialResult()` already have a wrong value.

I hope this will help to fix the issue.

I do a DIRTY HACK to fix the issue.
 I just fork this function and hardcode `None` -> `Iterator(toResultRow(None, e.record))`.
 In my case, it's better to do not have any values in the row, than theoretically have a wrong value in some column.
Bug: [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 05:25
Affects Version/s: 3.0.0

I'm trying to migrate spark 2 to spark 3 in k8s.

 

In my environment, on Spark 3.x, jars listed in spark.jars and spark.jars.packages are not added to sparkContext.

After driver's process is launched, jars are not propagated to Executors. So, NoClassDefException is raised in executors.

 

In spark.properties, the only main application jar is contained in spark.jars. It is different from Spark 2.

 

How to solve this situation? Is it any changed spark options in spark 3 from spark 2?
Bug: Upgrade flake8 to 3.8.0 or above in Jenkins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/21 13:46
Affects Version/s: 3.2.0

In flake8 < 3.8.0, F401 error occurs for imports in *if* statements when TYPE_CHECKING is True. However, TYPE_CHECKING is always False at runtime, so there is no need to treat it as an error in static analysis.

Since this behavior is fixed In flake8 >= 3.8.0, we should upgrade the flake8 installed in Jenkins to 3.8.0 or above. Otherwise, it occurs F401 error for several lines in pandas-on-PySpark that uses TYPE_CHECKING.

And also we might update the {{MINIMUM_FLAKE8}} in the {{lint-python}} from 3.5.0 to 3.8.0.
Bug: EXPLAIN command should handle empty output of an analyzed plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/21 03:10
Affects Version/s: 3.2.0

Currently, EXPLAIN command puts an empty line if there is no output for an analyzed plan. For example,
{code:java}
sql("CREATE VIEW test AS SELECT 1").explain(true)
{code}

produces:

{code:java}
== Parsed Logical Plan ==
'CreateViewStatement [test], SELECT 1, false, false, PersistedView
+- 'Project [unresolvedalias(1, None)]
   +- OneRowRelation

== Analyzed Logical Plan ==

CreateViewCommand `default`.`test`, SELECT 1, false, false, PersistedView, true
   +- Project [1 AS 1#7]
      +- OneRowRelation

== Optimized Logical Plan ==
CreateViewCommand `default`.`test`, SELECT 1, false, false, PersistedView, true
   +- Project [1 AS 1#7]
      +- OneRowRelation

== Physical Plan ==
Execute CreateViewCommand
   +- CreateViewCommand `default`.`test`, SELECT 1, false, false, PersistedView, true
         +- Project [1 AS 1#7]
            +- OneRowRelation
{code}
Bug: Getting Critical Vulnerability CVE-2021-20231 on spark 3.0.0 branch
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Apr/21 03:25
Affects Version/s: 3.0.0

Currently while running Trivy Scan on Spark build we are getting the following critical vulnerability 
CVE-2021-20231   
CVE-2021-20232
How to fix these vulnerabilities in spark 3.0.0 branch ?
Bug: DataFrameWriter.text support zstd compression
Status: Open | Priority: Major | Resolution: 
Created: 22/Apr/21 19:57
Affects Version/s: 3.1.1

[http://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrameWriter.text.html] specifies that only the following compression codecs are supported: `none, bzip2, gzip, lz4, snappy and deflate`

However, RDD API supports compression with zstd if users specify 'org.apache.hadoop.io.compress.ZStandardCodec' compressor in the saveAsTextFile method.

Please also expose zstd in the DataFrameWriter.
Bug: Recover Python linter (Sphinx build) in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/21 02:54
Affects Version/s: 3.0.2

It fails as below:

{code}
Encoding error:
'ascii' codec can't decode byte 0xc2 in position 69: ordinal not in range(128)
The full traceback has been saved in /tmp/sphinx-err-122qgk2d.log, if you want to report the issue to the developers.
make: *** [Makefile:20: html] Error 2

re-running make html to print full warning list:
Running Sphinx v3.0.4

Encoding error:
'ascii' codec can't decode byte 0xc2 in position 69: ordinal not in range(128)
The full traceback has been saved in /tmp/sphinx-err-mqvr_y8v.log, if you want to report the issue to the developers.
make: *** [Makefile:20: html] Error 2
{code}
Bug: Spark shell failed to access external file to read content
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 01/Apr/21 01:39
Affects Version/s: 3.0.0

Spark shell failed to access external file to read content

Spark version 3.0, Hadoop version 3.2

Start spark normally, enter spark shell and enter Val B1= sc.textFile ("S3a: / / spark test / a.txt") is normal. When entering B1. Collect(). Foreach (println) to traverse and read the contents of the file, an error is reported

The error message is: java.lang.NumberFormatException : For input string: "100M"
Bug: read partitioned parquet: my_col=NOW replaced by <current_date> on read()
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Apr/21 06:02
Affects Version/s: 3.1.1

When reading parquet file partitioned with a column containing the value "NOW", The value is interpreted as now() and replaced by the current time at the moment of the read() funct is executed
{code:java}
// step to reproduce
df = spark.createDataFrame(data=[("NOW",1), ("TEST", 2)], schema = ["col1", "id"])
df.write.partitionBy("col1").parquet("test/test.parquet")
>>> /home/test/test.parquet/col1=NOW
df_loaded = spark.read.option(
 "basePath",
 "test/test.parquet",
).parquet("test/test.parquet/col1=*")
>>> 
+---+--------------------------+
|id |col1                      |
+---+--------------------------+
|2  |TEST                      |
|1  |2021-04-18 14:36:46.532273|
+---+--------------------------+{code}
Bug: EXPLAIN CODEGEN does not work with AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/21 06:28
Affects Version/s: 3.2.0

`EXPLAIN CODEGEN <query>` (and Dataset.explain("codegen")) prints out the generated code for each stage of plan. The current implementation is to match `WholeStageCodegenExec` operator in query plan and prints out generated code ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala#L111-L118] ). This does not work with AQE as we wrap the whole query plan inside `AdaptiveSparkPlanExec` and do not run whole stage code-gen physical plan rule eagerly (`CollapseCodegenStages`). This introduces unexpected behavior change for EXPLAIN query (and Dataset.explain), as we enable AQE by default now.