av9ash/GitBugs huggingface dataset:
Bug: CTE With clause not working when using JDBC connection
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/21 15:35
Affects Version/s: 3.1.2

When using a JDBC data source, the "With" CTE function generates invalid SQL syntax when called

In the below example you can assume that SOURCE_CONNECTION, SQL_DRIVER and TABLE are all correctly defined.

 
SQLServerException: Incorrect syntax near the keyword 'WITH'.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262) at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1632) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:602) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:524) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7418) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3272) at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:247) at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:222) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:446) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:61) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:385) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:424) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:391) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:391) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:264) at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:439)
Bug: Support multiple loggers in testing method withLogAppender
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/21 13:30
Affects Version/s: 3.2.0

A test case of AdaptiveQueryExecSuite becomes flaky since there are too many debug logs in RootLogger:
https://github.com/Yikun/spark/runs/2715222392?check_suite_focus=true
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139125/testReport/

To fix it,  I suggest supporting multiple loggers in the testing method withLogAppender. So that the LogAppender gets clean target log outputs.
Bug: Performance degradation in avro SpecificRecordBuilders
Status: Open | Priority: Minor | Resolution: 
Created: 11/Jun/21 18:28
Affects Version/s: 3.2.0

Creating this bug to let you know that when we tested out spark 3.2.0 we saw a significant performance degradation where our code was handling Avro Specific Record objects.  This slowed down some of our jobs with a factor 4.

Spark 3.2.0 upsteps the avro version from 1.8.2 to 1.10.2.

The degradation was caused by a change introduced in avro 1.9.0.  This change degrades performance when creating avro specific records in certain classloader topologies, like the ones used in spark.

We notified and [proposed|https://github.com/apache/avro/pull/1253] a simple fix upstream in the avro project.  (Links contain more details)

It is unclear for us how many other projects are using avro specific records in a spark context and will be impacted by this degradation.
 Feel free to close this issue if you think this issue is too much of a corner case.
Bug: Mismatched types when creating a new column when using Arrow
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 18/Jun/21 10:17
Affects Version/s: 3.1.2

I was looking into a question on StackOverflow where a user had an error due to a mismatch of field types  (https://stackoverflow.com/questions/68032084/how-to-fix-error-in-readbin-using-arrow-package-with-sparkr/).  I've made a reproducible example below.

 
{code:java}
library(SparkR)

library(arrow)

SparkR::sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = "true"))

readr::write_csv(iris, "iris.csv")dfSchema <- structType(
  structField("Sepal.Length", "int"),
  structField("Sepal.Width", "double"),
  structField("Petal.Length", "double"),
  structField("Petal.Width", "double"),
  structField("Species", "string")
)

spark_df <- SparkR::read.df(path="iris.csv", source="csv", schema=dfSchema)

# Apply an R native function to each partition.
returnSchema <-  structType(structField("Sep_add_one", "int"))

df <- SparkR::dapply(spark_df, function(rdf){
  data.frame(rdf$Sepal.Length + 1)
  }, returnSchema
)

collect(df)
{code}
It results in this error:
{code:java}
Error in readBin(con, raw(), as.integer(dataLen), endian = "big") : 
  invalid 'n' argument
{code}
I've never used SparkR before, so could be wrong, but I *think* it may be due to the use of this line in DataFrame.R:
{code:java}
arrowTable <- arrow::read_ipc_stream(readRaw(conn))
{code}
readRaw assumes that the the value is an integer; however when returnSchema is being created, the code creates a double (the code runs without error if it's updated to `rdf$Sepal.Length + 1L`)

I wasn't sure if perhaps readTypedObject needs to be used or if maybe some type checking might be useful?

Apologies for the partial explanation - not entirely sure of the cause.
Bug: Spark 3.1.1 Internet Explorer 11 compatibility issues
Status: Open | Priority: Minor | Resolution: 
Created: 19/Jun/21 08:20
Affects Version/s: 3.0.3

Spark UI-Executor tab is empty in IE11

Spark UI-Stages DAG visualization is empty in IE11

other tabs looks Ok.

Spark job history shows completed and incomplete applications list .But when we go inside each application same issue may be there.

Attaching some screenshots
Bug: [Spark Sql] Add Java-comptable Dataset.join overloads
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jun/21 14:50
Affects Version/s: 2.0.0

h2. Problem

When using Spark SQL with Java, the required syntax to utilize the following two overloads are unnatural and not obvious to developers that haven't had to interoperate with Scala before:
{code:java}
def join(right: Dataset[_], usingColumns: Seq[String]): DataFrame
def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame
{code}
Examples:

Java 11 
{code:java}
Dataset<Row> dataset1 = ...;
Dataset<Row> dataset2 = ...;

// Overload with multiple usingColumns, no join type
dataset1
  .join(dataset2, JavaConverters.asScalaBuffer(List.of("column", "column2))
  .show();

// Overload with multiple usingColumns and a join type
dataset1
  .join(
    dataset2,
    JavaConverters.asScalaBuffer(List.of("column", "column2")),
    "left")
  .show();
{code}
 
 Additionally there is no overload that takes a single usingColumnn and a joinType, forcing the developer to use the Seq[String] overload regardless of language.

Examples:

Scala
{code:java}
val dataset1 :DataFrame = ...;
val dataset2 :DataFrame = ...;

dataset1
  .join(dataset2, Seq("column"), "left")
  .show();
{code}
 
 Java 11
{code:java}
Dataset<Row> dataset1 = ...;
Dataset<Row> dataset2 = ...;

dataset1
 .join(dataset2, JavaConverters.asScalaBuffer(List.of("column")), "left")
 .show();
{code}
h2. Proposed Improvement

Add 3 additional overloads to Dataset:
  
{code:java}
def join(right: Dataset[_], usingColumn: List[String]): DataFrame
def join(right: Dataset[_], usingColumn: String, joinType: String): DataFrame
def join(right: Dataset[_], usingColumn: List[String], joinType: String): DataFrame
{code}
Bug: pyspark.sql.functions GroupBy agg CountDistinct() return bad value
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jun/21 10:07
Affects Version/s: 3.0.1

from pyspark.sql import functions as F

gr_month = list.groupBy(F.year('date').alias('year'), F.month('date').alias('month')).agg(F.countDistinct('id').alias("n_id")).orderBy(F.col("year").desc(), F.col("month").desc()).persist()
 gr_month.show()
 
|year|month|n_id|
|2021|6|58801|
|2021|5|93699|

list.filter(F.year('date') == "2021").filter(F.month('date') == "6").select('id').distinct().count()

98916

 

98916 != 58801
Bug: PythonUDFSuite move assume into its methods
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/21 04:47
Affects Version/s: 3.2.0

When we run Spark test  with such command:
`./build/mvn -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn -Pkubernetes clean test`

get this exception:
{code:java}
 PythonUDFSuite:
 org.apache.spark.sql.execution.python.PythonUDFSuite *** ABORTED ***
   java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: org.apache.spark.sql.execution.python.PythonUDFSuite
   at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)
   at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
   at scala.collection.Iterator.foreach(Iterator.scala:941)
   at scala.collection.Iterator.foreach$(Iterator.scala:941)
   at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
   at scala.collection.IterableLike.foreach(IterableLike.scala:74)
   at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
   at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
   at scala.collection.TraversableLike.map(TraversableLike.scala:238)
{code}

The test env has not PYSpark module so it failed.
Bug: Ambiguous variable reference in functions.py column()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jun/21 02:50
Affects Version/s: 3.1.2

In functions.py, there is a function added {{def column(col)}}. There is also another method in the same file {{def col(col)}}. This leads to some ambiguity on whether the parameter is being referred to or the function. In pyspark 3.1.2, this leads to {{TypeError: 'str' object is not callable}} when the function {{column(col)}} is called - the highest preference is given to the string variable in scope as opposed to the function {{col }}in the file as intended.

This PR fixes that ambiguity by changing the variable name to {{col_like}}.

 

I have already raised a PR to fix the issue - https://github.com/apache/spark/pull/32771
Bug: [SQL] JSON read behavior is different depending on the cache setting when nullable is false.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jun/21 02:58
Affects Version/s: 3.1.1

Below is the reproduced code.

 
{code:java}
import org.apache.spark.sql.Encoders
 
case class TestSchema(x: Int, y: Int)
case class BaseSchema(value: TestSchema)
 
val schema = Encoders.product[BaseSchema].schema
val testDS = Seq("""{"value":{"x":1}}""", """{"value":{"x":2}}""").toDS
val jsonDS = spark.read.schema(schema).json(testDS)

jsonDS.show
+---------+
|    value|
+---------+
|{1, null}|
|{2, null}|
+---------+

jsonDS.cache.show
+------+
| value|
+------+
|{1, 0}|
|{2, 0}|
+------+

{code}
 

The above result occurs when a schema is created with a nested StructType and nullable of StructField is false.
Bug: Increase the timeout in FallbackStorageSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/21 02:46
Affects Version/s: 3.1.2

The aarch64 case failed due to:

 

{{Discovery starting. Discovery completed in 2 seconds, 396 milliseconds. Run starting. Expected test count is: 9 FallbackStorageSuite: - fallback storage APIs - copy/exists - SPARK-34142: fallback storage API - cleanUp - migrate shuffle data to fallback storage - Upload from all decommissioned executors}}
 {{- Upload multi stages *** FAILED ***}}
 \{{ The code passed to eventually never returned normally. Attempted 20 times over 10.011176743 seconds. Last failure message: fallbackStorage.exists(0, file) was false. (FallbackStorageSuite.scala:243)}}
 {{- lz4 - Newly added executors should access old data from remote storage *** FAILED ***}}
 \{{ The code passed to eventually never returned normally. Attempted 20 times over 10.010694845 seconds. Last failure message: fallbackStorage.exists(0, file) was false. (FallbackStorageSuite.scala:268)}}
 {{- lzf - Newly added executors should access old data from remote storage *** FAILED ***}}
 \{{ The code passed to eventually never returned normally. Attempted 20 times over 10.009722811000001 seconds. Last failure message: fallbackStorage.exists(0, file) was false. (FallbackStorageSuite.scala:268)}}
 {{- snappy - Newly added executors should access old data from remote storage *** FAILED ***}}
 \{{ The code passed to eventually never returned normally. Attempted 20 times over 10.009750581 seconds. Last failure message: fallbackStorage.exists(0, file) was false. (FallbackStorageSuite.scala:268)}}
 {{- zstd - Newly added executors should access old data from remote storage *** FAILED ***}}
 \{{ The code passed to eventually never returned normally. Attempted 20 times over 10.00968885 seconds. Last failure message: fallbackStorage.exists(0, file) was false. (FallbackStorageSuite.scala:268)}}
 {{Run completed in 1 minute, 37 seconds.}}
 {{Total number of tests run: 9}}
 {{Suites: completed 2, aborted 0}}
 {{Tests: succeeded 4, failed 5, canceled 0, ignored 0, pending 0}}
 {{*** 5 TESTS FAILED ***}}
Bug: Bug fix for deadlock during the executor shutdown
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jun/21 14:33
Affects Version/s: 3.1.2

When a executor received a TERM signal, it (the second TERM signal) will lock java.lang.Shutdown class and then call Shutdown.exit() method to exit the JVM.
 Shutdown will call SparkShutdownHook to shutdown the executor.
 During the executor shutdown phase, RemoteProcessDisconnected event will be send to the RPC inbox, and then WorkerWatcher will try to call System.exit(-1) again.
 Because java.lang.Shutdown has already locked, a deadlock has occurred.
Bug: Enable vectorized read for VectorizedPlainValuesReader.readBooleans
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jun/21 21:23
Affects Version/s: 3.2.0

Currently we decode PLAIN encoded booleans as follow:
{code:java}
  public final void readBooleans(int total, WritableColumnVector c, int rowId) {
    // TODO: properly vectorize this
    for (int i = 0; i < total; i++) {
      c.putBoolean(rowId + i, readBoolean());
    }
  }
{code}

Ideally we should vectorize this.
Bug: Watermark timestamp only can be format in UTC timeZone, unfriendly to users in other time zones
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 23/Jun/21 10:39
Affects Version/s: 3.1.2

Timestamp is formatted in `ProgressReporter` by `formatTimestamp` for watermark and eventTime stats. the timestampFormat is hardcoded in UTC time zone.

`

private val timestampFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'") // ISO8601
 timestampFormat.setTimeZone(DateTimeUtils.getTimeZone("UTC"))

`

When users set the different timezone by java options `-Duser.timezone` , they may be confused by the information mixed with different timezone.

eg

`

 {color:#FF0000}*2021-06-23 16:12:07*{color} [stream execution thread for [id = 92f4f363-df85-48e9-aef9-5ea6f2b70316, runId = 5733ef8e-11d1-46c4-95cc-219bde6e7a20]] INFO [MicroBatchExecution:54]: Streaming query made progress: {
 "id" : "92f4f363-df85-48e9-aef9-5ea6f2b70316",
 "runId" : "5733ef8e-11d1-46c4-95cc-219bde6e7a20",
 "name" : null,
 "timestamp" : "2021-06-23T08:11:56.790Z",
 "batchId" : 91740,
 "numInputRows" : 2577,
 "inputRowsPerSecond" : 155.33453887884266,
 "processedRowsPerSecond" : 242.29033471229786,
 "durationMs" :

{ "addBatch" : 8671, "getBatch" : 3, "getOffset" : 1139, "queryPlanning" : 79, "triggerExecution" : 10636, "walCommit" : 162 }

,
 "eventTime" :

{color:#FF0000}*{ "avg" : "2021-06-23T08:11:46.307Z", "max" : "2021-06-23T08:11:55.000Z", "min" : "2021-06-23T08:11:37.000Z", "watermark" : "2021-06-23T07:41:39.000Z" }*{color}

,

`

maybe we need to unified the timezone for time format
Bug: Bug fix for thread leak in JobCancellationSuite
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 10/Jun/21 13:56
Affects Version/s: 3.1.2

When we call Thread.interrupt() method, that thread's interrupt status will be set but it may not really interrupt.

So when spark task runs in an infinite loop, spark context may fail to interrupt the task thread, and resulting in thread leak.
Bug: Volcano resource manager for Spark on Kubernetes
Status: Open | Priority: Minor | Resolution: 
Created: 03/Jun/21 07:15
Affects Version/s: 3.1.1

Dear Spark Developers, 
  
 Hello from the Netherlands! Posting this here as I still haven't gotten accepted to post in the spark dev mailing list.
  
 My team is planning to use spark with Kubernetes support on our shared (multi-tenant) on premise Kubernetes cluster. However we would like to have certain scheduling features like fair-share and preemption which as we understand are not built into the current spark-kubernetes resource manager yet. We have been working on and are close to a first successful prototype integration with Volcano ([https://volcano.sh/en/docs/]). Briefly this means a new resource manager component with lots in common with existing spark-kubernetes resource manager, but instead of pods it launches Volcano jobs which delegate the driver and executor pod creation and lifecycle management to Volcano. We are interested in contributing this to open source, either directly in spark or as a separate project.
  
 So, two questions: 
  
 1. Do the spark maintainers see this as a valuable contribution to the mainline spark codebase? If so, can we have some guidance on how to publish the changes? 
  
 2. Are any other developers / organizations interested to contribute to this effort? If so, please get in touch.
  
 Best,
 Dipanjan
Bug: Upgrade Jetty to 9.4.42
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/21 06:38
Affects Version/s: 3.2.0

Recently, CVE-2021-28169 was reported and 9.4.40 which Spark uses in the current master affects.
https://nvd.nist.gov/vuln/detail/CVE-2021-28169.
Bug: Remove type collection AllTimestampTypes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jun/21 06:58
Affects Version/s: 3.3.0

As discussed in https://github.com/apache/spark/pull/33115#discussion_r659866760, it is more convenient to have a new data type "AnyTimestampType" instead of using type collection `AllTimestampTypes`:
1. simplify the pattern match 
2. In the default type coercion rules, when implicit casting a type to a TypeCollection type, Spark chooses the first convertible data type as the result. If we are going to make the default timestamp type configurable, having AnyTimestampType is better
Bug: Document Parquet encryption feature in Spark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jun/21 12:14
Affects Version/s: 3.2.0

Spark 3.2.0 will use parquet-mr.1.12.0 version (or higher), that contains the encryption feature which can be called from Spark SQL. The aim of this Jira is to document the use of Parquet encryption in Spark.
Bug: Allow custom plugin for AQE cost evaluator
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jun/21 08:09
Affects Version/s: 3.2.0

Current AQE has cost evaluator to decide whether to use new plan after replanning. The current used evaluator is `SimpleCostEvaluator` to make decision based on number of shuffle in the query plan. This is not perfect cost evaluator, and different production environments might want to use different custom evaluators. E.g., sometimes we might want to still do skew join even though it might introduce extra shuffle (trade off resource for better latency), sometimes we might want to take sort into consideration for cost as well. So We want to make the cost evaluator to be plugable, and developers can implement their own `CostEvaluator` subclass and plug in dynamically based on configuration.
Bug: Unify reuse map data structures in non-AQE and AQE rules
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jun/21 10:58
Affects Version/s: 3.2.0

We can unify reuse map data structures in non-AQE and AQE rules (`ReuseExchangeAndSubquery`, `ReuseAdaptiveSubquery`) to a simple `Map[<canonicalized plan>, <plan>]`.

Please find discussion here: [https://github.com/apache/spark/pull/28885#discussion_r655073897]
Bug: ChromeUIHistoryServerSuite and ChromeUISeleniumSuite failed on Mac 10.15
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 11/Jun/21 16:25
Affects Version/s: 3.1.1

import org.openqa.selenium.chrome.\{ChromeDriver, ChromeOptions} 

ChromeDriver is not getting resolved.

 

Also - Class org.openqa.selenium.remote.RemoteWebDriver not found - continuing with a stub.
Bug: Add since versions for Avro options in Documentation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jun/21 08:54
Affects Version/s: 3.2.0

There are two new Avro options `datetimeRebaseMode` and `positionalFieldMatching` after Spark 3.2.
We should document the since version so that users can know whether the option works in their Spark version.
Bug: ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jun/21 02:29
Affects Version/s: 3.2.0

yarn sub-module contains config.properties.
{code}
spark.yarn.isHadoopProvided = ${spark.yarn.isHadoopProvided}
{code}

The ${spark.yarn.isHadoopProvided} part is replaced with true or false in build depending on whether Hadoop is provided or not (specified by -Phadoop-provided).
The edited config.properties will be loaded at runtime to control how to populate Hadoop-related classpath.

If we build with Maven, these process works but doesn't with SBT.
Bug: Support pathlib.PurePath-like objects in DataFrameReader / DataFrameWriter
Status: Open | Priority: Minor | Resolution: 
Created: 28/Jun/21 22:51
Affects Version/s: 2.4.8

It would be nice to support Path objects in `spark.\{read,write}.\{parquet,orc,csv,...etc}` methods.

Without pyspark source code changes it currently seems possible only by the ugly monkeypatching hacks - https://stackoverflow.com/q/68170685/2649222.
Bug: Upgrade Ivy to 2.5.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jun/21 12:51
Affects Version/s: 3.1.2

Apache Ivy 2.5.0 was released nearly two years ago. The new bug fixes and features can be found here: [https://ant.apache.org/ivy/history/latest-milestone/release-notes.html]

Most notably, the adding of ivy.maven.lookup.sources and ivy.maven.lookup.javadoc configs can significantly speed up module resolution time if these are turned off, especially behind a proxy. These could arguably be turned off by default, because when submitting jobs you probably don't care about the sources or javadoc jars.