av9ash/GitBugs huggingface dataset:
Bug: Fix InSet bucket pruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/20 14:51
Affects Version/s: 3.1.0

This pr fix InSet bucket pruning because of it's values should not be Literal:

https://github.com/apache/spark/blob/cbd3fdea62dab73fc4a96702de8fd1f07722da66/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L255
Bug: Spark job can not be killed in BoradcastNestedLoopJoin
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Nov/20 07:55
Affects Version/s: 3.0.1

 

If we kill a spark job in spark web UI, the task content will be marked interrupted, as the code below shows:
{code:java}
/**
 * Kills a task by setting the interrupted flag to true. This relies on the upper level Spark
 * code and user code to properly handle the flag. This function should be idempotent so it can
 * be called multiple times.
 * If interruptThread is true, we will also call Thread.interrupt() on the Task's executor thread.
 */
def kill(interruptThread: Boolean, reason: String): Unit = {
  require(reason != null)
  _reasonIfKilled = reason
  if (context != null) {
    context.markInterrupted(reason)
  }
  if (interruptThread && taskThread != null) {
    taskThread.interrupt()
  }
}{code}
 

And spark will check the interrupt flag during the loop to stop it. Like this:
{code:java}
 /**
 * :: DeveloperApi ::
 * An iterator that wraps around an existing iterator to provide task killing functionality.
 * It works by checking the interrupted flag in [[TaskContext]].
 */
@DeveloperApi
class InterruptibleIterator[+T](val context: TaskContext, val delegate: Iterator[T])
  extends Iterator[T] {

  def hasNext: Boolean = {
    // TODO(aarondav/rxin): Check Thread.interrupted instead of context.interrupted if interrupt
    // is allowed. The assumption is that Thread.interrupted does not have a memory fence in read
    // (just a volatile field in C), while context.interrupted is a volatile in the JVM, which
    // introduces an expensive read fence.
    context.killTaskIfInterrupted()
    delegate.hasNext
  }

  def next(): T = delegate.next()
}{code}
In my case, there is a "not in" in my spark sql,  which leads to the "BoradcastNestedLoopJoin"

The related code as below:
{code:java}
private def leftExistenceJoin(
    relation: Broadcast[Array[InternalRow]],
    exists: Boolean): RDD[InternalRow] = {
  assert(buildSide == BuildRight)
  streamed.execute().mapPartitionsInternal { streamedIter =>
    val buildRows = relation.value
    val joinedRow = new JoinedRow

    if (condition.isDefined) {
      streamedIter.filter(l =>
        buildRows.exists(r => boundCondition(joinedRow(l, r))) == exists
      )
    } else if (buildRows.nonEmpty == exists) {
      streamedIter
    } else {
      Iterator.empty
    }
  }
}{code}
The "streamedIter" and "buildRows" both have millions of records,  the executor get stuck in the join loop, I found something wrong in my sql and try to kill the job, but the executor thread is not interrupted. I have to restart the executor to stop it.

I think we should also do this check: " context.killTaskIfInterrupted() "  in BoradcastNestedLoopJoin to support real cancel.

 

 
{code:java}
 {code}
Bug: Catalyst StringType converter unable to convert enum type
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Nov/20 21:39
Affects Version/s: 2.4.7

Given that:
 # JavaTypeInference maps [Enums to StringType|https://github.com/apache/spark/blob/55105a0784459331d5506eee9f37c2e655a2a6a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala#L135] 
 # The converter for [StringType is StringConverter|https://github.com/apache/spark/blob/55105a0784459331d5506eee9f37c2e655a2a6a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala#L64]
 # StringConverter is unable to convert an [Enums to UTF8String|https://github.com/apache/spark/blob/55105a0784459331d5506eee9f37c2e655a2a6a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala#L294], failing with an InvalidArgumentException

 

It can be argued that CatalystTypeConverters should align with the exprectations set by JavaTypeInference and convert enums to their string representation. 

 

Edit:

Added PRs for 3.0.X and 2.4.X branches
Bug: Merge In and InSet predicate
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Nov/20 05:12
Affects Version/s: 3.1.0

Maybe we should create a base class for {{In}} and {{InSet}}, so that these 2 classes are only different in the expression tree, eval and codegen are the same.

[https://github.com/apache/spark/pull/28269#issuecomment-655365714]
Bug: element_at with CreateArray not respect one based index
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 07:38
Affects Version/s: 3.1.0

var df = spark.sql("select element_at(array(3, 2, 1), 0)")
df.printSchema()

df = spark.sql("select element_at(array(3, 2, 1), 1)")
df.printSchema()

df = spark.sql("select element_at(array(3, 2, 1), 2)")
df.printSchema()

df = spark.sql("select element_at(array(3, 2, 1), 3)")
df.printSchema()

root
 |-- element_at(array(3, 2, 1), 0): integer (nullable = false)

root
 |-- element_at(array(3, 2, 1), 1): integer (nullable = false)

root
 |-- element_at(array(3, 2, 1), 2): integer (nullable = false)

root
 |-- element_at(array(3, 2, 1), 3): integer (nullable = true)

 

In this case, the nullable property in element_at with CreateArray statement is not correct.
Bug: Skip schema inference in DataframeWriter.save() if table provider supports external metadata
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/20 05:16
Affects Version/s: 3.1.0

For all the v2 data sources which are not FileDataSourceV2, Spark always infers the table schema/partitioning on DataframeWriter.save().
The inference of table schema/partitioning can be expensive. However, there is no such trait or flag for indicating a V2 source can use the input DataFrame's schema on DataframeWriter.save(). We can resolve the problem by adding a new expected behavior for the method TableProvider.supportsExternalMetadata():
When TableProvider.supportsExternalMetadata() is true, Spark will use the input Dataframe's schema in DataframeWriter.save() and skip schema/partitioning inference.
Bug: mistakenly generate markdown to html for available-patterns-for-shs-custom-executor-log-ur
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 12:51
Affects Version/s: 3.0.1

http://spark.apache.org/docs/3.0.1/running-on-yarn.html#available-patterns-for-shs-custom-executor-log-url
Bug: k8s pyspark shell doesn't honor pyspark memory setting
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/20 00:43
Affects Version/s: 3.1.0

When running the interactive pyspark shell in client mode on k8s, it doesn't honor thespark.executor.pyspark.memory memory setting when requesting containers from k8s. 

If you run it in cluster mode and specify a python script then it does work.
Bug: DSv2 SHOW TABLES doesn't show `default`
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/20 20:26
Affects Version/s: 3.1.0

DSv1:
{code:scala}
  test("namespace is not specified and the default catalog is set") {
    withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
      withTable("table") {
        spark.sql(s"CREATE TABLE table (id bigint, data string) $defaultUsing")
        sql("SHOW TABLES").show()
      }
    }
  }
{code}
{code}
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
| default|    table|      false|
+--------+---------+-----------+
{code}

DSv2:
{code}
+---------+---------+
|namespace|tableName|
+---------+---------+
|         |    table|
+---------+---------+
{code}
Bug: Clean up useless variables in MutableApplicationInfo
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 16:28
Affects Version/s: 3.1.0

 
{code:java}
private class MutableApplicationInfo {
  var id: String = null
  var name: String = null
  var coresGranted: Option[Int] = None
  var maxCores: Option[Int] = None
  var coresPerExecutor: Option[Int] = None
  var memoryPerExecutorMB: Option[Int] = None

  def toView(): ApplicationInfoWrapper = {
    val apiInfo = ApplicationInfo(id, name, coresGranted, maxCores, coresPerExecutor,
      memoryPerExecutorMB, Nil)
    new ApplicationInfoWrapper(apiInfo, List(attempt.toView()))
  }

}
{code}
 

coresGranted, maxCores, coresPerExecutor and memoryPerExecutorMB always None and never reassign
Bug: simplify CaseWhen with EqualTo
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Nov/20 06:56
Affects Version/s: 3.1.0

{code:sql}
create table t(a int, b int, c int) using parquet;
SELECT * 
FROM   (SELECT CASE 
                 WHEN a = 100 THEN 1 
                 WHEN b > 1000 THEN 2 
                 WHEN c IS NOT NULL THEN 3 
               END AS x 
        FROM   t) tmp 
WHERE  x = 2
{code}


{noformat}
Before this PR:
Filter (CASE WHEN (a#1 = 100) THEN 1 WHEN (b#2 > 1000) THEN 2 WHEN isnotnull(c#3) THEN 3 END = 2)

After this PR:
Filter (b#2 > 1000)
{noformat}
Bug: Dataset.observe() functionality does not work with structured streaming
Status: Open | Priority: Major | Resolution: 
Created: 05/Nov/20 14:28
Affects Version/s: 3.0.0

The dataset observe() functionality does not work as expected with spark in cluster mode.
Related discussion here:
[http://apache-spark-user-list.1001560.n3.nabble.com/Best-way-to-emit-custom-metrics-to-Prometheus-in-spark-structured-streaming-td38826.html]

Using lit() as the aggregation column goes through well. However sum, count etc returns 0 all the time.
Bug: Spark to Handle Nano seconds in Timestamps
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/20 00:10
Affects Version/s: 3.0.1

Spark currently does not support Nano seconds in Timestamps.

With the modern day high-frequency trading and transactions, data will be classified per nano seconds and adding this support which Hive has would improve the Spark adoption.

Creating a Jira to generate traction within the Apache SPARK community. 

Thank you,

Akhil Chalamalasetty
Bug: Support nullable Avro schemas for non-nullable data in Avro writing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/20 08:58
Affects Version/s: 2.4.0

Currently when users try to use nullable Avro schemas for non-nullable data in Avro writing, Spark will throw a IncompatibleSchemaException.

There are some cases when users do not have full control over the nullability of the data, or the nullability of the Avro schemas they have to use. We should support nullable Avro schemas for non-nullable data in Avro writing for better usability.
Bug: Fix the build with sbt to copy hadoop-client-runtime.jar
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 14:39
Affects Version/s: 3.1.0

With the current master, spark-shell doesn't work if it's built with sbt package.
It's due to hadoop-client-runtime.jar isn't copied to assembly/target/scala-2.12/jars.
{code}
$ bin/spark-shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper
	at org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:426)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:877)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1013)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1022)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.com.ctc.wstx.io.InputBootstrapper
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 11 more
{code}
Bug: Support array_contains_array func
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Nov/20 04:00
Affects Version/s: 3.1.0

Postgres SQL support  judge if array1 contains array2, 

 
{code:java}
@>  contains    ARRAY[1,4,3] @> ARRAY[3,1]    t
{code}
 

[https://www.postgresql.org/docs/9.1/functions-array.html]

 

Users in our production environment also meet this requirement too. we cans support this
Bug: DAG Scheduler exhibits exponential runtime with PartitionerAwareUnion
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/Nov/20 09:41
Affects Version/s: 2.4.2

The current implementation of the {{DAGScheduler}} exhibits exponential runtime in DAGs with many {{PartitionerAwareUnions}}. The reason seems to be a mutual recursion between {{PartitionerAwareUnion.getPreferredLocations}} and {{DAGScheduler.getPreferredLocs}}.

A minimal example reproducing the issue:
{code:scala}
object Example extends App {
  val partitioner = new HashPartitioner(2)
  val sc = new SparkContext(new SparkConf().setAppName("").setMaster("local[*]"))
  val rdd1 = sc.emptyRDD[(Int, Int)].partitionBy(partitioner)
  val rdd2 = (1 to 30).map(_ => rdd1)
  val rdd3 = rdd2.reduce(_ union _)
  rdd3.collect()
}
{code}
The whole app should take around one second to complete, as no actual work is done. However, it takes more time to submit the job than I am willing to wait.

The underlying cause appears to be mutual recursion between {{PartitionerAwareUnion.getPreferredLocations}} and {{DAGScheduler.getPreferredLocs}}, which restarts graph traversal at each {{PartitionerAwareUnion}} with no memoization. Each node of the DAG is visited {{O(n!)}} (exponentially many) times.

Note, that it is clear to me that you could use {{sc.union(rdd2)}} instead of {{rdd2.reduce(_ union _)}} to eliminate the problem. I use this just to demonstrate the issue in a sufficiently small example. Given a large DAG and many PartitionerAwareUnions, especially contructed by iterative algorithms, the problem can become relevant even without "abuse" of the union operation.

The exponential recursion in DAG Schedular was largely fixed with SPARK-682, but in the special case of PartitionerAwareUnion, it is still possible. This may actually be an underlying cause of SPARK-29181.
Bug: Incorrectly Parsing CSV file
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/20 22:25
Affects Version/s: 2.4.7

Here is a test case: 

[https://github.com/mores/maven-examples/blob/master/comma/src/test/java/org/test/CommaTest.java]

It shows how I believe apache commons csv and opencsv correctly parses the sample csv file.

spark is not correctly parsing the sample csv file.
Bug: Add "unused-import" compile arg to scalac and remove all unused imports in Scala code 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Nov/20 06:07
Affects Version/s: 3.1.0

* Add new scala compile arg to defense against new unused imports:
 ** "-Ywarn-unused-import" for Scala 2.12
 ** "-Wconf:cat=unused-imports:ws" or "-Wconf:cat=unused-imports:error" for Scala 2.13
 * Remove all unused imports in Scala code
Bug: Add config to control if cancel invoke interrupt task on thriftserver
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 06:53
Affects Version/s: 3.1.0

After [#29933|https://github.com/apache/spark/pull/29933], we support cancel query if timeout, but the default behavior of `SparkContext.cancelJobGroups` won't interrupt task and just let task finish by itself. In some case it's dangerous, e.g., data skew or exists a heavily shuffle. A task will hold in a long time after do cancel and the resource will not release.
Bug: Doubts about the use of the "DiskBlockObjectWriter#revertPartialWritesAndClose" method in Spark Code
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 11/Nov/20 15:01
Affects Version/s: 3.1.0

Although there are some similar discussions in SPARK-17562, but I still have some questions.

I found "DiskBlockObjectWriter#revertPartialWritesAndClose" method is called in 5 places in Spark Code,

Two of the call points are in the "ExternalAppendOnlyMap#spillMemoryIteratorToDisk" method, two similar call points are in the "ExternalSorter#spillMemoryIteratorToDisk" method, and the last  is in the "BypassMergeSortShuffleWriter#stop" method.

Let's take the use of "ExternalAppendOnlyMap#spillMemoryIteratorToDisk" as an example:

 
{code:java}
var success = false
try {
  while (inMemoryIterator.hasNext) {
    ...

    if (objectsWritten == serializerBatchSize) {
      flush()
    }
  }
  if (objectsWritten > 0) {
    flush()
    writer.close()
  } else {
    writer.revertPartialWritesAndClose() // The first call point 
  }
  success = true
} finally {
  if (!success) {
    writer.revertPartialWritesAndClose() // The second call point
    if (file.exists()) {
      if (!file.delete()) {
        logWarning(s"Error deleting ${file}")
      }
    }
  }
}

{code}
 

There are two questions about the above code:

1. Can the first call "writer.revertPartialWritesAndClose() " be replaced by "writer.close()"?

I think there are two possibilities to get into this branch：
 * One possibility is all data has been called flush(), I think we can call "writer.close()" directly because all data has been flushed, "committedPosition"  of DiskBlockObjectWriter should eq file.length.
 * Another is inMemoryIterator is empty, in this scenario whether calling "revertPartialWritesAndClose()" or calling "close()", the file.length is both 0, the test suite "commit() and close() without ever opening or writing" in DiskBlockObjectWriterSuite can prove that

And I try to use "writer.close()"  instead of "writer.revertPartialWritesAndClose() " , all UTs in core module passed, so what is the specific scenario that must call the "revertPartialWritesAndClose() " method?

2. For the 2nd call point, the main goal is to roll back writeMetrics in DiskBlockObjectWriter? 
 If we want to delete this file,  Is the truncate operation in the "revertPartialWritesAndClose() " method really necessary？In this scenario, should we just roll back writeMetrics without truncate file to reduce one disk operation？
Bug: Column.__repr__ shouldn't encode JVM response
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Nov/20 23:04
Affects Version/s: 3.1.0

At the moment PySpark {{Column}} {{encodes}} JVM response in {{__repr__}} method.

As a result, column names using only ASCII characters get {{b}} prefix

{code:python}
>>> from pyspark.sql.functions import col                                                                                                                                                
>>> col("abc")                                                                                                                                                                           
Column<b'abc'>
{code}

and the others ugly byte string

{code:python}
>>> col("wąż")                                                                                                                                                                           
Column<b'w\xc4\x85\xc5\xbc'>
{code}

This behaviour is inconsistent with other parts of the API, for example:

{code:python}
>>> spark.createDataFrame([], "`wąż` long")                                                                                                                                              
DataFrame[wąż: bigint]
{code}

and Scala

{code:scala}
scala> col("wąż")
res0: org.apache.spark.sql.Column = wąż
{code}

and R

{code:r}
> column("wąż")
Column wąż 
{code}

Encoding has been originally introduced with SPARK-5859, but it doesn't seem like it is really required.

Desired behaviour

{code:python}
>>> col("wąż")                                                                                                                                                                           
Column<'wąż'>
{code}

or

{code:python}
>>> col("wąż")                                                                                                                                                                           
Column<wąż>
{code}
Bug: Update PR labeler to add back in the negated file matches
Status: Open | Priority: Minor | Resolution: 
Created: 06/Nov/20 08:02
Affects Version/s: 3.0.1

When we replaced the probot autolabeler in this issue https://issues.apache.org/jira/browse/SPARK-33282, we were not able to port over values that were negated (minimatch globs that used ! operator). In order to properly do this, we need the upstream `actions/labeler` repo provided by github to support the `any` function to as discussed in their README.

 

When they support the `any` function, we need to update the PR labeler config in .github/labeler.yml to so that things are labeled the same as before.

Thie feature is available in their main branch (renamed from master), as discussed in an issue I linked in the original issue https://issues.apache.org/jira/browse/SPARK-33282, but I was unable to get it to work properly when I tried it out. Labels that were previously applying correctly stopped doing so when I switched to main.

I've opened an issue in the labeler repo asking if they'll consider cutting a release or what the timeline for that is. That can be found here: https://github.com/actions/labeler/issues/111
Bug: Support Java Class with circular references
Status: Open | Priority: Minor | Resolution: 
Created: 30/Nov/20 07:48
Affects Version/s: 3.1.2

If the target Java data class has a circular reference, Spark will fail fast from creating the Dataset or running Encoders.

 

For example, with protobuf class, there is a reference with Descriptor, there is no way to build a dataset from the protobuf class.

From this line

{color:#7a869a}Encoders.bean(ProtoBuffOuterClass.ProtoBuff.class);{color}

 

It will throw out immediately

 
{quote}Exception in thread "main" java.lang.UnsupportedOperationException: Cannot have circular references in bean class, but got the circular reference of class class com.google.protobuf.Descriptors$Descriptor
{quote}
 

Can we add  a parameter, for example, 

 
{code:java}
Encoders.bean(Class<T> clas, List<Fields> fieldsToIgnore);{code}
````

or

 
{code:java}
Encoders.bean(Class<T> clas, boolean skipCircularRefField);{code}
 

 which subsequently, instead of throwing an exception @ [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L556], it instead skip the field.

 
{code:java}
if (seenTypeSet.contains(t)) {
if(skipCircularRefField)
  println("field skipped") //just skip this field
else throw new UnsupportedOperationException( s"cannot have circular references in class, but got the circular reference of class $t")
}
{code}