av9ash/GitBugs huggingface dataset:
Bug: Kafka doesn't recover from data loss
Status: Open | Priority: Major | Resolution: 
Created: 28/Jun/21 09:30
Affects Version/s: 3.1.1

I configured a strcutured streaming source for kafka with failOnDataLoss=false, 

Getting this error when checkopint offsets are not found :

 
{code:java}
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 5.0 failed 1 times, most recent failure: Lost task 7.0 in stage 5.0 (TID 113) ( executor driver): java.lang.IllegalStateException: This consumer has already been closed.
  at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2439)
  at org.apache.kafka.clients.consumer.KafkaConsumer.seekToBeginning(KafkaConsumer.java:1656)
  at org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumer.getAvailableOffsetRange(KafkaDataConsumer.scala:108)
  at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.getEarliestAvailableOffsetBetween(KafkaDataConsumer.scala:385)
  at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.$anonfun$get$1(KafkaDataConsumer.scala:332)
  at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77)
  at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.runUninterruptiblyIfPossible(KafkaDataConsumer.scala:604)
  at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.get(KafkaDataConsumer.scala:287)
  at org.apache.spark.sql.kafka010.KafkaBatchPartitionReader.next(KafkaBatchPartitionReader.scala:63)
  at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:79)
  at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:112)
  at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)

{code}
 

The issue seems to me to be related to the OffsetOutOfRange exception in (line (323 in KafkaDataConsumer): 

 
{code:java}
 case e: OffsetOutOfRangeException =>
    // When there is some error thrown, it's better to use a new consumer to drop all cached
    // states in the old consumer. We don't need to worry about the performance because this
    // is not a common path.
    releaseConsumer()
    fetchedData.reset()

    reportDataLoss(topicPartition, groupId, failOnDataLoss,
      s"Cannot fetch offset $toFetchOffset", e)
    toFetchOffset = getEarliestAvailableOffsetBetween(consumer, toFetchOffset, untilOffset)
}
{code}
seems like releaseConsumer will destoy the consumer , which later is used ...
Bug: array_zip unexpected column names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 11:53
Affects Version/s: 3.1.2

{{When I'm using the array_zip function in combination with renamed columns, I get an unexpected schema written to disk.}}
{code:java}
// code placeholder
from pyspark.sql import * 
from pyspark.sql.functions import *

spark = SparkSession.builder.getOrCreate()

data = [
  Row(a1=["a", "a"], b1=["b", "b"]),
]
df = (
  spark.sparkContext.parallelize(data).toDF()
    .withColumnRenamed("a1", "a2")
    .withColumnRenamed("b1", "b2")
    .withColumn("zipped", arrays_zip(col("a2"), col("b2")))
)
df.printSchema()
// root
//  |-- a2: array (nullable = true)
//  |    |-- element: string (containsNull = true)
//  |-- b2: array (nullable = true)
//  |    |-- element: string (containsNull = true)
//  |-- zipped: array (nullable = true)
//  |    |-- element: struct (containsNull = false)
//  |    |    |-- a2: string (nullable = true)
//  |    |    |-- b2: string (nullable = true)

df.write.save("test.parquet")
spark.read.load("test.parquet").printSchema()
// root
//  |-- a2: array (nullable = true)
//  |    |-- element: string (containsNull = true)
//  |-- b2: array (nullable = true)
//  |    |-- element: string (containsNull = true)
//  |-- zipped: array (nullable = true)
//  |    |-- element: struct (containsNull = true)
//  |    |    |-- a1: string (nullable = true)
//  |    |    |-- b1: string (nullable = true){code}
I would expect the schema of the DataFrame written to disk to be the same as that printed out. It seems that instead of using the renamed version of the column names, it uses the old column names.
Bug: Spark fails to launch executors with very large user classpath lists on YARN
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/21 20:10
Affects Version/s: 3.1.2

When running Spark on YARN, the {{user-class-path}} argument to {{CoarseGrainedExecutorBackend}} is used to pass a list of user JAR URIs to executor processes. The argument is specified once for each JAR, and the URIs are fully-qualified, so the paths can be quite long. With large user JAR lists (say 1000+), this can result in system-level argument length limits being exceeded, typically manifesting as the error message:
{code}
/bin/bash: Argument list too long
{code}

A [Google search|https://www.google.com/search?q=spark%20%22%2Fbin%2Fbash%3A%20argument%20list%20too%20long%22&oq=spark%20%22%2Fbin%2Fbash%3A%20argument%20list%20too%20long%22] indicates that this is not a theoretical problem and afflicts real users, including ours. This issue was originally observed on Spark 2.3, but has been confirmed to exist in the master branch as well.
Bug: Move to_pandas_on_spark to the Spark DataFrame.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 01:44
Affects Version/s: 3.2.0

Inspired by https://github.com/apache/spark/pull/32729#discussion_r643591322,

As Koalas is ported into PySpark, we don't need auto patch anymore ([https://github.com/apache/spark/blob/master/python/pyspark/pandas/__init__.py#L136-L150])

Thus, we should to_pandas_on_spark and to_koalas(deprecated) to the PySpark DataFrame and add related tests.
Bug: Cannot run program "python" error when run do-release-docker.sh
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 03:41
Affects Version/s: 3.0.0

Moving back into docs dir.
Moving to SQL directory and building docs.
Generating SQL API Markdown files.
21/06/15 09:45:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.io.IOException: Cannot run program "python": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:97)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 14 more
Bug: Add `index_col` argument for ps.sql.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 04:10
Affects Version/s: 3.2.0

The current behavior of [ps.sql |https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.sql.html]always lost the index, so we should add the `index_col` arguments for this API so that we can preserve the index.
Bug: Merge Binder integration and quickstart notebook
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/21 05:40
Affects Version/s: 3.2.0

We should merge:

https://github.com/apache/spark/blob/master/python/docs/source/getting_started/quickstart.ipynb
https://github.com/databricks/koalas/blob/master/docs/source/getting_started/10min.ipynb
Bug: Throw an error if `version` and `timestamp` are used together in DataFrame.to_delta.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 04:38
Affects Version/s: 3.2.0

[read_delta |https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.read_delta.html#databricks.koalas.read_delta]has an arguments named `version` and `timestamp`, but it cannot be used together.

We should raise the proper error message when they are used together.
Bug: Increase stacksize in Maven for AppVeyor build
Status: In Progress | Priority: Major | Resolution: 
Created: 14/Jun/21 08:23
Affects Version/s: 3.2.0

https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark

{code}
java.lang.StackOverflowError
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
scala.reflect.internal.Trees.$anonfun$itransform$1(Trees.scala:1421)
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:43)
rmer.transform(TypingTransformers.scala:57)
	at scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
	at scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
	at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
	at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
	at scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)

{code}

AppVeyor build is flaky due to stack size limit. We should better increase.
Bug: Improving testability of GroupState in streaming flatMapGroupsWithState
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/21 17:39
Affects Version/s: 3.1.2

GroupStateImpl is the internal implementation of the GroupState interface which mean to be not exposed. Thus, it only has a private constructor. Such access control does benefit encapsulation, however, this introduces difficulties for unit tests and the users are calling the engine to construct such GroupState instances in order to test their customized state transition functions.

The solution is to introduce new interfaces that allow users to create instances of GroupState but also access internal values of what they have set (for example, has to state been updated, or removed). This would allow them to write unit tests of the state transition function with custom GroupState objects and then verifying whether the state was updated in an expected way.
Bug: leveldbjni doesn't work in Apple Silicon on macOS
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/21 04:43
Affects Version/s: 3.1.2

leveldbjni doesn't contain the native library for Apple Silicon on macOS. We will need to build native library for Apple Silicon on macOS, and cut a new release so Spark can use it.

However, it is not maintained for a long time, and the last release was in 2016. Per [discussion|http://apache-spark-developers-list.1001551.n3.nabble.com/leveldbjni-dependency-td30146.html] in spark dev mailing list, other platform also runs into the same support issue. Perhaps, we should we consider racksdb as replacement.

Note, here is the rocksdb task to support Apple Silicon, https://github.com/facebook/rocksdb/issues/7720
Bug: Support auto-assigning labels to less important pods (e.g. decommissioning pods)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/21 18:01
Affects Version/s: 3.2.0

To allow for folks to use pod disruption budgets or replicasets we should indicate which pods Spark cares about "the least", those would be pods that are otherwise exiting soon.

 

With PDBs the user would create a PDB representing the label of decommissioning executors and this could have a higher number of unavailable than the PDB for the "regular" execs. For people using replicasets in 1.21 we could also set a label of "controller.kubernetes.io/pod-deletion-cost" (see [https://github.com/kubernetes/kubernetes/pull/99163] ) to hint to the controller that a pod is less important to us.
Bug: Extracting date field from timestamp should work in ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 17:09
Affects Version/s: 3.2.0

 Add a new ANSI type coercion rule: when getting a date field from a Timestamp column, cast the column as Date type.
 This is Spark's hack to make the implementation simple. In the default type coercion rules, the implicit cast rule does the work. However, The ANSI implicit cast rule doesn't allow converting Timestamp type as Date type, so we need to have this additional rule
 to make sure the date field extraction from Timestamp columns works.
Bug: Option "--files" with local:// prefix is not honoured for Spark on kubernetes
Status: Open | Priority: Major | Resolution: 
Created: 10/Jun/21 16:52
Affects Version/s: 3.0.2

When we provide a local file as a dependency using "--files" option, the file is not getting copied to work directories of executors.
h5. Example 1:

 
{code:java}
$SPARK_HOME/bin/spark-submit --master k8s://https://<ip-address-k8s> \ 
--deploy-mode cluster \ 
--name spark-pi \ 
--class org.apache.spark.examples.SparkPi \ 
--conf spark.executor.instances=1 \ 
--conf spark.kubernetes.container.image=<spark-3.1.2-image> \ 
--conf spark.kubernetes.driver.pod.name=sparkdriverpod \ 
--files local:///etc/xattr.conf \ 
local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar 1000000
{code}
 
h6. Content of Spark Executor work-dir:

 
{code:java}
~$ kubectl exec -n default spark-pi-22de6279f6bec01c-exec-1 ls /opt/spark/work-dir/
spark-examples_2.12-3.1.2.jar
{code}
 

We can notice here that the file _/etc/xattr.conf_ is *NOT* copied to  _/opt/spark/work-dir/ ._

 
----
 

{{Instead of using "–files", if we use "--jars" option the file is getting copied as expected.}}
h5. Example 2:
{code:java}
$SPARK_HOME/bin/spark-submit --master k8s://https://<ip-address-k8s> \ 
--deploy-mode cluster \ 
--name spark-pi \ 
--class org.apache.spark.examples.SparkPi \ 
--conf spark.executor.instances=1 \ 
--conf spark.kubernetes.container.image=<spark-3.1.2-image> \ 
--conf spark.kubernetes.driver.pod.name=sparkdriverpod \ 
--jars local:///etc/xattr.conf \ 
local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar 1000000


{code}
h6. Content of Spark Executor work-dir:

 
{code:java}
~$ kubectl exec -n default spark-pi-22de6279f6bec01c-exec-1 ls /opt/spark/work-dir/
spark-examples_2.12-3.1.2.jar

xattr.conf

{code}
We can notice here that the file _/etc/xattr.conf_ *IS COPIED* to  _/opt/spark/work-dir/ ._

 

I tested this with versions *3.1.2* and *3.0.2*. It is behaving the same way in both cases.
Bug: Migrate Spark Arm Job from Jenkins to GitHub Actions
Status: Open | Priority: Major | Resolution: 
Created: 02/Jun/21 03:37
Affects Version/s: 3.3.0

There were two Arm CI jobs in AMP lab:

[https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-python-arm/]

[https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/]

As note from ML[1], AMPLab jenkins had been shut down at the end of 2021.

We should consider to migrate the Arm job from jenkins to somewhere like github action. Unfortunately, the github action doen't support native arm64 vitualenv yet.

 * Github Action self hosted
 ** Running jobs in self-hosted ARM VM.
 ** Reuse many actions and build script
 ** Easy to migrate when official GA arm support ready

 

[1] [https://www.mail-archive.com/dev@spark.apache.org/msg28480.html]
Bug: Add new adaptive config into sql-performance-tuning docs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 05:27
Affects Version/s: 3.2.0

Describe the new config `spark.sql.adaptive.autoBroadcastJoinThreshold` and `spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold` at sql-performance-tuning docs.
Bug: Update the doc of running-on-kubernetes
Status: Closed | Priority: Major | Resolution: Invalid
Created: 04/Jun/21 07:55
Affects Version/s: 3.1.2

The commands of building PySpark / SparkR docker image  can not run in the lateset version 

 
{code:java}
./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build {code}
{code:java}
./bin/docker-image-tool.sh -r <repo> -t my-tag -R ./kubernetes/dockerfiles/spark/bindings/R/Dockerfile build
{code}
Bug: PartitionFilters and pushFilters not applied to window functions
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 29/Jun/21 16:20
Affects Version/s: 2.4.8

Spark does not apply partition and pushed filters when the partition by column and window function partition columns are not the same. For example, in the code below, the data frame is created with a partition on "id". And I use the partitioned data frame to calculate lag which is partitioned by "name". In this case, the query plan shows the partitionFilters and pushed Filters as empty.
{code:java}
spark
  .createDataFrame(
    Seq(
      Person(
        1,
        "Andy",
        new Timestamp(1499955986039L),
        new Timestamp(1499955982342L)
      ),
      Person(
        2,
        "Jeff",
        new Timestamp(1499955986339L),
        new Timestamp(1499955986666L)
      )
    )
  )
 .write
  .partitionBy("id")
  .mode(SaveMode.Append)
  .parquet("spark-warehouse/people")

val dfPeople =
  spark.read.parquet("spark-warehouse/people")
dfPeople
  .select(
    $"id",
    $"name",
    lag(col("ts2"), 1).over(Window.partitionBy("name").orderBy("ts"))
  )
  .filter($"id" === 1)
  .explain()
{code}
Bug: Support DayTimeIntervalType in width-bucket function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 06:45
Affects Version/s: 3.2.0

At now, width-bucket support the type [DoubleType, DoubleType, DoubleType, LongType],

we hope that support[DayTimeIntervaType, DayTimeIntervaType, DayTimeIntervaType, LongType]
Bug: Support YearMonthIntervalType in width-bucket function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 06:50
Affects Version/s: 3.2.0

At now, width-bucket support the type [DoubleType, DoubleType, DoubleType, LongType],

we hope that support[YearMonthIntervalType, YearMonthIntervalType, YearMonthIntervalType, LongType]
Bug: Refactor Parquet vectorized reader to remove duplicated code paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jun/21 00:58
Affects Version/s: 3.2.0

Currently in Parquet vectorized code path, there are many code duplications such as the following:
{code:java}
  public void readIntegers(
      int total,
      WritableColumnVector c,
      int rowId,
      int level,
      VectorizedValuesReader data) throws IOException {
    int left = total;
    while (left > 0) {
      if (this.currentCount == 0) this.readNextGroup();
      int n = Math.min(left, this.currentCount);
      switch (mode) {
        case RLE:
          if (currentValue == level) {
            data.readIntegers(n, c, rowId);
          } else {
            c.putNulls(rowId, n);
          }
          break;
        case PACKED:
          for (int i = 0; i < n; ++i) {
            if (currentBuffer[currentBufferIdx++] == level) {
              c.putInt(rowId + i, data.readInteger());
            } else {
              c.putNull(rowId + i);
            }
          }
          break;
      }
      rowId += n;
      left -= n;
      currentCount -= n;
    }
  }
{code}

This makes it hard to maintain as any change on this will need to be replicated in 20+ places. The issue becomes more serious when we are going to implement column index and complex type support for the vectorized path.

The original intention is for performance. However now days JIT compilers tend to be smart on this and will inline virtual calls as much as possible.
Bug: Improve error message quality
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Jun/21 21:21
Affects Version/s: 3.2.0

In the SPIP: Standardize Exception Messages in Spark, there are three major improvements proposed:
 # Group error messages in dedicated files: SPARK-33539
 # Establish an error message guideline for developers SPARK-35140
 # Improve error message quality

Based on the guideline, we should start improving the error messages based on the guideline.
Bug: Upgrade kinesis-client to 1.14.4
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Jun/21 11:55
Affects Version/s: 3.2.0

Upgrading to 1.14.1 or newer is recommended by the community for users who use kinesis-client 1.14.0 due to a bug.
https://github.com/awslabs/amazon-kinesis-client/tree/master#recommended-upgrade-for-all-users-of-the-1x-amazon-kinesis-client
https://github.com/awslabs/amazon-kinesis-client/issues/778
Bug: Support reading inputbytes and inputrecords of the stage int spark context interface 
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Jun/21 07:42
Affects Version/s: 3.2.0

One of our projects needs to count the amount of data scanned and the number of scanned data rows during the execution of sparksql statements, but the current version of spark does not provide an interface to view these data, so I want to obtain this type of data through the spark context interface
Bug: Introduce the strategy on mismatched offset for start offset timestamp on Kafka data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 10:47
Affects Version/s: 3.0.2

1. Rationalization

We encountered a real-world case Spark fails the query if some of the partitions don't have matching offset by timestamp.

This is intended behavior to avoid bring unintended output for some cases like:

* timestamp 2 is presented as timestamp-offset, but the some of partitions don't have the record yet
* record with timestamp 1 comes "later" in the following micro-batch

which is possible since Kafka allows to specify the timestamp in record.

Here the unintended output we talked about was the risk of reading record with timestamp 1 in the next micro-batch despite the option specifying timestamp 2.

But for many cases end users just suppose timestamp is increasing monotonically, and current behavior blocks these cases to make progress.

2. Proposal

For the cases the timestamp is supposed to increase monotonically, it's safe to consider the offset to be latest (technically, offset for latest record + 1) if there's no matching record via timestamp.

This would be pretty much helpful for the case where there's a skew between partitions and some partitions have older records.

* AS-IS: Spark simply fails the query and end users have to deal with workarounds requiring manual steps.
* TO-BE: Spark will assign the latest offset for these partitions, so that Spark can read newer records from these partitions in further micro-batches.

To retain the existing behavior and also give some help for the proposed "TO-BE" behavior, we'd like to introduce the strategy on mismatched offset for start offset timestamp.