av9ash/GitBugs huggingface dataset:
Bug: Add Jar with Azure SAS token fails with URL encoded characters
Status: Open | Priority: Major | Resolution: 
Created: 23/Oct/20 13:04
Affects Version/s: 2.4.3

I am running spark-submit using an Azure SAS token to access the jar file. When the sig of the SAS token contains URL encoded characters before the end, I get a 403 error trying to download the jar. It appears to be related to the URL encoding change that occurs within DependencyUtils: [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala#L137.]

Error message:

+ exec /usr/local/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.0.0.44 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class MyClass 'https://storageaccount.blob.core.windows.net/blob/my-jar.jar?sv=2019-12-12&ss=b&srt=sco&sp=r&se=*****&st=*******&spr=https&sig=sigwith%2Band%2Fending%3D'

ava.io.IOException: Server returned HTTP response code: 403 for URL: https://storageaccount.blob.core.windows.net/blob/ivm-0.2.40-Spark-2.2.jar?sv=2019-12-12&ss=b&srt=sco&sp=r&se=**********&st=*********&spr=https&sig=sigwith+and/ending= at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1900) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:713) at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:137) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367) at scala.Option.map(Option.scala:146) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:366) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


It may not be clear in the example above, but when I submit the sas token url, it looks like:

sig=sigwith%2Band%2Fending%3D

The 403 error from the stacktrace gives

sig=sigwith+and/ending=

Is there something I can do to ensure that these characters do not get URL decoded in this way?
Bug: UDF in when operation applied in all the rows regardless the condition
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 22/Oct/20 10:59
Affects Version/s: 2.4.5

Hi all,

I think there is a bug or, at least, an undocumented behaviour of pyspark UDFs. The code here is trying to apply the UDF just for a subset of rows (convenient for long dataframes).

 
{code:java}
@udf(returnType=BooleanType())
def test_udf(x):
  if x is None:
    raise Exception(x)
  return True
  
data = [(1,11,1),(1,22,2),(1,33,3),(2,44,1),(3,55,1),(4,66,1)]
dataColumns = ["uid","price","day"]
test = spark.createDataFrame(data=data, schema = dataColumns)w = Window.partitionBy('uid').orderBy('uid','day')
test = test.withColumn('lag_price', F.lead(F.col('price')).over(w))
print(test.dtypes)
test = test.withColumn('condition', F.col('lag_price').isNotNull())
test.withColumn('appliedUDF', F.when(F.col('condition'), test_udf(F.col('lag_price'))).otherwise(False)).show()
{code}
It throws this error:
{code:java}
File "<command-3513778682084612>", line 4, in test_udf
Exception: None
{code}
Is it normal? Am I missing something?
Bug: Mesos dispatcher service is not working due to empty --pyFiles conf in cluster mode which is the default
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Oct/20 04:18
Affects Version/s: 3.0.0

In MesosCluster mode, all of the spark jobs fail to run because by default "--py-files" is set to an empty string which causes spark-submit to use wrong jar name. This issue is caused by [SPARK-26466 | https://github.com/apache/spark/commit/38f030725c561979ca98b2a6cc7ca6c02a1f80ed]

 For ex:
{quote}--total-executor-cores
 2
 --py-files
 --conf
 spark.driver.maxResultSize=15g
{quote}
 so the very next --conf is used as the value for --py-files so `spark.driver.maxResultSize=15g` is used as jar name 

which causes error as below:
{quote}20/10/19 20:19:18 WARN DependencyUtils: Local jar \{dir}/slaves/d9971b08-4929-4b00-9677-b14088c38603-S13/frameworks/86ff1f10-79fd-44f0-b807-71741091b457-0000/executors/driver-20201019200043-0014-retry-9/runs/3697919f-c3dc-4a13-8928-2fb3a20ac98d/spark.driver.maxResultSize=15g does not exist, skipping.
 20/10/19 20:19:18 WARN SparkSubmit$$anon$2: Failed to load org.apache.spark.examples.SparkPi.
 java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
{quote}
In order to reproduce the bug run any spark provided examples on mesos in cluster mode:
{code:java}
 ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://<server>:7077 --deploy-mode cluster --conf spark.master.rest.enabled=true --executor-memory 5G --total-executor-cores 2 <home>/examples/jars/spark-examples_2.12-3.0.1.jar 100
{code}
Bug: Validate that all shuffle files are migrated during E2E testing
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Oct/20 00:22
Affects Version/s: 3.1.0

Currently the end to end testing only migrates a small number of shuffle files. While we can't migrate large amounts of data in the time of the integration suite we can extend the decommissioning test to migrate multiple shuffle files of different sizes.
Bug: Spark 3.0 streaming metrics are not in Prometheus
Status: Open | Priority: Major | Resolution: 
Created: 20/Oct/20 15:25
Affects Version/s: 3.0.0

I'm running a Spark 3.0 application (Spark Structured Streaming) on Kubernetes and I'm trying to use the new native Prometheus metric sink. I'm able to make it work and get all the metrics described [here|https://spark.apache.org/docs/latest/monitoring.html#executor-metrics].

However, the metrics I really need are the ones provided upon enabling the following config: spark.sql.streaming.metricsEnabled, as proposed in [this Spark Summit presentation|https://speakerdeck.com/dongjoon/native-support-of-prometheus-monitoring-in-apache-spark-3?slide=32]. Now, even with that config set to "true", I can't see any streaming metrics under {{/metrics/executors/prometheus}} as advertised. One thing to note is that I can see them under {{metrics/json}}, therefore, we know that the configuration was properly applied.

Why aren't streaming metrics sent to the Prometheus sink? Do I need to add some additional configuration? Is that not supported yet?
Bug: on k8s dynamic resource allocation failed
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Oct/20 09:07
Affects Version/s: 3.0.1

when spark running with following parameters :spark.dynamicAllocation.enabled=true,spark.shuffle.service.enabled=false,spark.dynamicAllocation.shuffleTracking.enabled=true,spark.dynamicAllocation.minExecutors=0,spark.dynamicAllocation.maxExecutor=3,spark.dynamicAllocation.executorIdleTimeout=60,
it means using dynamic resource allocation without shuffle service. if  submitting a job with shuffle operation, then executor will not be deleted forever even if the timeout is arrived.
Bug: Encountering an invalid rolling event log folder prevents loading other applications in SHS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/20 11:15
Affects Version/s: 3.0.1

A follow-on issue from https://issues.apache.org/jira/browse/SPARK-33133

If an invalid rolling event log folder is encountered by the Spark History Server upon startup, it crashes the whole loading process and prevents any valid applications from loading. We should simply catch the error, log it, and continue loading other applications.
Bug: Detect duplicated mountPath and fail at Spark side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/20 05:13
Affects Version/s: 2.4.7

If there is a mountPath conflict, the pod is created and repeats the following error messages and keep running. This should not keep running and we had better fail at Spark side.
{code}
$ k get pod -l 'spark-role in (driver,executor)'
NAME    READY   STATUS    RESTARTS   AGE
tpcds   1/1     Running   0          14m
{code}

{code}
20/10/18 05:09:26 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: ...
Message: Pod "tpcds-exec-1" is invalid: spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/data1": must be unique.
{code}
Bug: Use 11-jre-slim as default in K8s Dockerfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/20 06:30
Affects Version/s: 3.0.1

Apache Spark supports both Java8/Java11. However, there is a difference.
1. Java8-built distribution can run both Java8/Java11
2. Java11-built distribution can run on Java11, but not Java8.

In short, we had better use Java11 in Dockerfile to embrace both cases without any issues.
Bug: Fix grouping sets with having clause can not resolve qualified col name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 11:18
Affects Version/s: 2.4.0

Grouping sets construct new aggregate lost the qualified name of grouping expression. Here is a example:
{code:java}
-- Works resolved by ResolveReferences
select c1 from values (1) as t1(c1) group by grouping sets(t1.c1) having c1 = 1

-- Works because of the extra expression c1
select c1 as c2 from values (1) as t1(c1) group by grouping sets(t1.c1) having t1.c1 = 1

-- Failed
select c1 from values (1) as t1(c1) group by grouping sets(t1.c1) having t1.c1 = 1{code}
Bug: Add SQL config to control fast-fail behavior in FileFormatWriter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/20 17:03
Affects Version/s: 3.1.0

In SPARK-29649, we catch `FileAlreadyExistsException` in `FileFormatWriter` and fail fast for the task set to prevent task retry.

After latest discussion, it is important to be able to keep original behavior that is to retry tasks even `FileAlreadyExistsException` is thrown, because `FileAlreadyExistsException` could be recoverable in some cases.

We are going to add a config we can control this behavior and set it false for fast-fail by default.
Bug: Throw Spark specific exception for commit protocol errors
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 16/Oct/20 03:35
Affects Version/s: 3.1.0

In SPARK-29649, we catch `FileAlreadyExistsException` in `FileFormatWriter` and fail fast for the task set to prevent task retry.

One concern for this approach is we might break some cases where users rely on task retry and their tasks throw `FileAlreadyExistsException` in user code, instead of Spark internally.

To take care both cases, better approach is to throw a Spark specific exception for commit protocol errors.
Bug: Use hive-service-rpc dep instead of inline the generated code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/20 11:55
Affects Version/s: 3.1.0

Hive's hive-service-rpc module started since hive-2.1.0 and it contains only the thrift IDL file and the code generated by it.

Removing the inlined code will help maintain and upgrade builtin  hive versions
Bug: Simplify offset window function(Remove direction field)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 08:06
Affects Version/s: 3.1.0

The current Lead/Lag extends OffsetWindowFunction. OffsetWindowFunction contains field direction and use direction to calculate the boundary.
We can use single literal expression unify the two properties.
For example:
 3 means direction is Asc and boundary is 3.
 -3 means direction is Desc and boundary is -3.
Bug: Support ALTER TABLE in JDBC v2 Table Catalog: update type and nullability of columns (Teradata dialect)
Status: Open | Priority: Major | Resolution: 
Created: 15/Oct/20 08:47
Affects Version/s: 3.1.0

Override the default SQL strings for:
ALTER TABLE ADD COLUMN
ALTER TABLE UPDATE COLUMN TYPE
ALTER TABLE RENAME COLUMN
ALTER TABLE UPDATE COLUMN NULLABILITY
in the following PostgreSQL JDBC dialect according to official documentation.
Write PostgreSQL integration tests for JDBC.
Bug: HiveExternalCatalogVersionsSuite fails on Ubuntu 20.04
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/20 02:49
Affects Version/s: 3.0.1

As we see SPARK-29536, PySpark 2.4.x doesn't work on Python 3.8. And, this breaks HiveExternalCatalogVersionsSuite on Ubuntu 20.04 because it's default Python3 is 3.8.
Bug: Jenkins PRB is not responding
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/20 18:37
Affects Version/s: 3.1.0

I noticed there has not been a successful build recently and all the workers seem empty. When I ask jenkins to test a PR I'm not getting the normal response (  [https://github.com/apache/spark/pull/29924#issuecomment-707990107] ).
Bug: SQLConf inconsistency between SharedState and SessionState
Status: Open | Priority: Major | Resolution: 
Created: 14/Oct/20 11:48
Affects Version/s: 3.0.1


{code:java}
package com.netease.mammut.spark.training.sql

import java.lang.reflect.Field

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession

object StaticConfigTest extends App {

  println(org.apache.spark.SPARK_VERSION)

  private val wh = "spark.sql.warehouse.dir"

  private val td = "spark.sql.globalTempDatabase"

  System.setProperty(wh, "./data1")

  val conf = new SparkConf()
  println(s"=====> SparkConf: $wh=${conf.get(wh)}")

  conf.set(wh, "./data2")
  private val spark: SparkSession =
    SparkSession.builder()
      .config(conf)
      .enableHiveSupport()
      .config(wh, "./data3")
      .config(td, "bob")
      .getOrCreate()

  val confField: Field = spark.sharedState.getClass.getDeclaredField("conf")
  confField.setAccessible(true)
  private val shared: SparkConf = confField.get(spark.sharedState).asInstanceOf[SparkConf]
  println()
  println(s"=====> SparkSession SharedState: $wh=${shared.get(wh)}")
  println(s"=====> SparkSession SharedState: $td=${shared.get(td)}")

  spark.sql(s"set $wh").show()
  println(s"=====> SparkSession 1 SessionState: $wh=${spark.conf.get(wh)}")
  println(s"=====> SparkSession 1 SessionState: $td=${spark.conf.get(td)}")

  SparkSession.clearActiveSession()
  SparkSession.clearDefaultSession()

  val spark2: SparkSession =
    SparkSession.builder()
      .config(conf)
      .enableHiveSupport()
      .config(wh, "./data4")
      .config(td, "alice")
      .getOrCreate()

  spark2.sql(s"set $wh").show()
  println(s"=====> SparkSession 2 SessionState: $wh=${spark2.conf.get(wh)}")
  println(s"=====> SparkSession 2 SessionState: $td=${spark2.conf.get(td)}")
}


//bin/spark-submit --class com.netease.mammut.spark.training.sql.StaticConfigTest /Users/kentyao/sugar/target/mammut-spark-training-1.0-SNAPSHOT.jar
//3.0.0
//=====> SparkConf: spark.sql.warehouse.dir=./data1
//
//=====> SparkSession SharedState: spark.sql.warehouse.dir=./data3
//=====> SparkSession SharedState: spark.sql.globalTempDatabase=bob
//+--------------------+-------+
//|                 key|  value|
//+--------------------+-------+
//|spark.sql.warehou...|./data3|
//+--------------------+-------+
//
//=====> SparkSession 1 SessionState: spark.sql.warehouse.dir=./data3
//=====> SparkSession 1 SessionState: spark.sql.globalTempDatabase=bob
//+--------------------+-------+
//|                 key|  value|
//+--------------------+-------+
//|spark.sql.warehou...|./data4|
//+--------------------+-------+
// Ooppps~
//=====> SparkSession 2 SessionState: spark.sql.warehouse.dir=./data4
//=====> SparkSession 2 SessionState: spark.sql.globalTempDatabase=alice

{code}
Bug: Use listLocatedStatus from FileSystem implementations
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Oct/20 16:20
Affects Version/s: 3.0.1

{{HadoopFsUtils.parallelListLeafFiles}} currently only calls {{listLocatedStatus}} when the {{FileSystem}} impl is {{DistributedFileSystem}} or {{ViewFileSystem}}. For other types of {{FileSystem}}, it calls {{listStatus}} and then subsequently calls {{getFileBlockLocations}} on all the result {{FileStatus}}es.

In Hadoop client, {{listLocatedStatus}} is a well-defined API and in fact it is often overridden by specific file system implementations, such as S3A. The default {{listLocatedStatus}} also has similar behavior as it's done in Spark.

Therefore, instead of re-implement the logic in Spark itself, it's better to rely on the {{FileSystem}}-specific implementation for {{listLocatedStatus}}, which could include its own optimizations in the code path.
Bug: Improve the error when Lead and Lag are not allowed to specify window frame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 02:59
Affects Version/s: 3.1.0

Except for Postgresql, other data sources (for example: vertica, oracle, redshift, mysql, presto) are not allowed to specify window frame for the Lead and Lag functions.
But the current error message is not clear enough.

{code:java}
Window Frame $f must match the required frame
{code}

The following error message is better.
{code:java}
Cannot specify window frame for lead function
{code}
Bug: Support ALTER TABLE in JDBC v2 Table Catalog: update type and nullability of columns (DB2 dialect)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/20 22:40
Affects Version/s: 3.1.0

Override the default SQL strings for:

* ALTER TABLE UPDATE COLUMN TYPE
* ALTER TABLE UPDATE COLUMN NULLABILITY

in the following DB2 JDBC dialect according to official documentation.

Write DB2 integration tests for JDBC.
Bug: Upgrade Google Guava
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Oct/20 02:39
Affects Version/s: 3.0.1

Hadoop versions newer than 3.2.0 (such as 3.2.1 and 3.3.0) have started using features from newer versions of Google Guava.

This leads to MethodNotFound exceptions, etc in Spark builds that specify newer versions of Hadoop. I believe this is due to the use of new methods in com.google.common.base.Preconditions.

The above versions of Hadoop use guava-27.0-jre, whereas Spark is currently glued to guava-14.0.1.

I have been running a Spark cluster with the version bumped to guava-29.0-jre without issue.

Partly due to the way Spark is built, this change is a little more complicated that just changing the version, because newer versions of guava have a new dependency on com.google.guava:failureaccess:1.0.
Bug: CREATE TEMPORARY TABLE fails with location
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/20 08:50
Affects Version/s: 3.0.0

The problem is produced when you use CREATE TEMPORARY TABLE with LOCATION

 
{code:java}
spark.range(3).write.parquet("/data/tmp/testspark1")

spark.sql("CREATE TEMPORARY TABLE t USING parquet OPTIONS (path '/data/tmp/testspark1')")
spark.sql("CREATE TEMPORARY TABLE t USING parquet LOCATION '/data/tmp/testspark1'")
{code}
The error message in both cases is 
{code:java}
org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet. It must be specified manually.;
  at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$12(DataSource.scala:200)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:200)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:408)
  at org.apache.spark.sql.execution.datasources.CreateTempViewUsing.run(ddl.scala:94)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
  at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
  at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
{code}