av9ash/GitBugs huggingface dataset:
Bug: Regression in "operating on canonicalized plan" check in CustomShuffleReaderExec
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Mar/21 00:47
Affects Version/s: 3.1.1

In Spark 3.0.2 if I attempt to execute on a canonicalized version of CustomShuffleReaderExec I get an error "operating on canonicalized plan", as expected.

There is a regression in Spark 3.1.1 where this check can never be reached because of a new call to sendDriverMetrics that was added prior to the check. This method will fail if operating on a canonicalized plan because it assumes the existence of metrics that do not exist if this is a canonicalized plan.
{code:java}
 private lazy val shuffleRDD: RDD[_] = {
  sendDriverMetrics()

  shuffleStage.map { stage =>
    stage.shuffle.getShuffleRDD(partitionSpecs.toArray)
  }.getOrElse {
    throw new IllegalStateException("operating on canonicalized plan")
  }
}{code}
The specific error looks like this:


{code:java}
java.util.NoSuchElementException: key not found: numPartitions
        at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:101)
        at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:99)
        at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.sendDriverMetrics(CustomShuffleReaderExec.scala:122)
        at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.shuffleRDD$lzycompute(CustomShuffleReaderExec.scala:182)
        at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.shuffleRDD(CustomShuffleReaderExec.scala:181)
        at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.doExecuteColumnar(CustomShuffleReaderExec.scala:196)
 {code}
I think the fix is simply to avoid calling sendDriverMetrics if the plan is canonicalized and I am planning on creating a PR to fix this.
Bug: Select queries fail  with Error: java.lang.IllegalArgumentException: Error: name expected at the position 10 of 'decimal(2,-2)' but '-' is found. (state=,code=0)
Status: Open | Priority: Minor | Resolution: 
Created: 09/Mar/21 10:31
Affects Version/s: 2.4.5

Temporary views are created

Select filter queries are executed on the Temporary views.

 

[Actual Issue] : - Select queries fail with Error: java.lang.IllegalArgumentException: Error: name expected at the position 10 of 'decimal(2,-2)' but '-' is found. (state=,code=0)

 

[Expected Result] :- Select queries should be success on Temporary views.
Bug: Update the documents to explain the usage of LIST FILE and LIST JAR in case they take multiple file names
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Mar/21 07:11
Affects Version/s: 3.0.2

LIST FILE and LIST JAR command can take multiple file name but the usage doesn't explain about that.
For the current master branch, SPARK-34603 contains the fix of this issue and branch-2.4 doesn't have document about those commands even though they are supported so we need to fix this issue for branch-3.0 and branch-3.1.
Bug: Can we create 'SessionState' with a username in 'HiveClientImpl'
Status: Open | Priority: Minor | Resolution: 
Created: 05/Mar/21 03:22
Affects Version/s: 3.0.0

[https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L165]

Like this:

val state = new SessionState(hiveConf, userName)

We can then easily use the Hive Authorization through the user information in the 'SessionState'.
Bug: Self-join with script transformation failed to resolve attribute correctly
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Mar/21 04:41
Affects Version/s: 2.4.0

To reproduce,
{code:java}
// code placeholder
create temporary view t as select * from values 0, 1, 2 as t(a);
WITH temp AS (
SELECT TRANSFORM(a) USING 'cat' AS (b string) FROM t
)
SELECT t1.b FROM temp t1 JOIN temp t2 ON t1.b = t2.b

{code}
 

Spark will throw AnalysisException
Bug: New PySpark documentation has different URLs
Status: In Progress | Priority: Minor | Resolution: 
Created: 03/Mar/21 09:21
Affects Version/s: 3.1.1

The new documentation site moved some subsites to different URLs, notably the PySpark API reference ([see here|https://spark.apache.org/docs/latest/api/python/pyspark.sql.html]). (Note the new `/reference/` bit in the new URL.)

It's the first hit when you google "pyspark sql functions", you'll also get there if you search for individual functions or modules (e.g. "pyspark streaming").

I looked through various JIRA tickets and pull requests, but couldn't find a mention of this. Even the pull request introducing the new documentation site mentions the only visible change to users is the design, not its location.

Possible resolution:
* let the links be refreshed by search engines and live with dead links in various places (stack overflow, emails, bookmarks, ...)
* identify the missing pages and provide a 301 redirects for these (could be found in logs, google analytics, or maybe we can list all assets generated before/now and diff them)
* change sphinx configuration to result in identical links as before

Links to potentially relevant tickets and PRs:
* https://issues.apache.org/jira/browse/SPARK-31851
* https://github.com/apache/spark/pull/29188
* https://issues.apache.org/jira/browse/SPARK-32188
Bug: typed udf fails when it refers to type member in abstract class
Status: Open | Priority: Minor | Resolution: 
Created: 01/Mar/21 12:16
Affects Version/s: 3.0.1

 Please consider a following scenario: 
{code:java}
scala> abstract class SomeAbstractClass {
     |   type SomeTypeMember
     | }
defined class SomeAbstractClassscala> class SomeSpecificClass extends SomeAbstractClass {
     |   override type SomeTypeMember = Int
     | }
defined class SomeSpecificClassscala> def someFunction(someInstance: SomeAbstractClass): Any = {
     |   udf { _: someInstance.SomeTypeMember => 42 }
     | }
someFunction: (someInstance: SomeAbstractClass)Any
scala> someFunction(new SomeSpecificClass)
java.lang.NoClassDefFoundError: no Java class corresponding to someInstance.SomeTypeMember found
  at scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1354)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:227)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:68)
  at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:56)
  at org.apache.spark.sql.functions$.$anonfun$udf$1(functions.scala:4509)
  at scala.util.Try$.apply(Try.scala:213)
  at org.apache.spark.sql.functions$.udf(functions.scala:4509)
  at someFunction(<console>:25)
  ... 47 elided

{code}
On *spark-2.4.7* it works. I guess that it is related to:
{code:java}
In Spark 3.0, using org.apache.spark.sql.functions.udf(AnyRef, DataType) is not allowed by default. Remove the return type parameter to automatically switch to typed Scala udf is recommended, or set spark.sql.legacy.allowUntypedScalaUDF to true to keep using it. In Spark version 2.4 and below, if org.apache.spark.sql.functions.udf(AnyRef, DataType) gets a Scala closure with primitive-type argument, the returned UDF returns null if the input values is null. However, in Spark 3.0, the UDF returns the default value of the Java type if the input value is null. For example, val f = udf((x: Int) => x, IntegerType), f($"x") returns null in Spark 2.4 and below if column x is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
{code}
[https://spark.apache.org/docs/latest/sql-migration-guide.html#udfs-and-built-in-functions]

Does spark try to do some type inferation? When it refers to `SomeAbstractClass.SomeTypeMember` it really does not exist.

Some workaround could be runtime type casting, something like: 
{code:java}
udf { param: Any => {
     ...
     param.asInstanceOf[someInstance.SomeTypeMember]
     ...  
} {code}
----
 I classified it as bug since on previous versions of spark it worked. However, I believe that it can work as designed.
Bug: SparkContext - avoid using local lazy val
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Mar/21 09:44
Affects Version/s: 2.4.7

`org.apache.spark.SparkContext#getCallSite` uses local lazy val for `callsite`. But in scala 2.11, local lazy val need synchronization on the containing object `this` (see [https://docs.scala-lang.org/sips/improved-lazy-val-initialization.html#version-6---no-synchronization-on-this-and-concurrent-initialization-of-fields] and [https://github.com/scala/scala-dev/issues/133] )

`getCallSite` is called at the job submission, and thus will be a bottle neck if we are submitting a large amount of jobs on a single spark session. We observed thread blocked due to this in our load test.

!image-2021-03-24-17-42-50-412.png!
Bug: AtLeastNNonNulls does not show up correctly in explain
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Mar/21 17:44
Affects Version/s: 3.1.2

I see this in an explain plan

{noformat}
(12) Filter
Input [3]: [c1#2410L, c2#2419, c3#2422]
Condition : AtLeastNNulls(n, c1#2410L)

I expect it to be AtLeastNNonNulls and n to have the actual value.
{noformat}

Proposed fix is to change https://github.com/databricks/runtime/blob/09ad623c458c40cbaa0dbb4b9f600d96531d35f5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala#L384 to
{noformat}
override def toString: String = s"AtLeastNNonNulls(${n}, ${children.mkString(",")})"
{noformat}
Or maybe it's OK to remove and use a default implementation?
Bug: ExternalAppendOnlyUnsafeRowArrayBenchmark can't run with spark-submit
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Mar/21 05:44
Affects Version/s: 3.0.2

The following exception will appear when run ExternalAppendOnlyUnsafeRowArrayBenchmark with
{code:java}
bin/spark-submit --class org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArrayBenchmark --jars spark-core_2.12-3.2.0-SNAPSHOT-tests.jar spark-sql_2.12-3.2.0-SNAPSHOT-tests.jar 
{code}
command :
{code:java}
Exception in thread "main" java.lang.IllegalStateException: SparkContext should only be created and accessed on the driver.Exception in thread "main" java.lang.IllegalStateException: SparkContext should only be created and accessed on the driver. at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$assertOnDriver(SparkContext.scala:2679) at org.apache.spark.SparkContext.<init>(SparkContext.scala:89) at org.apache.spark.SparkContext.<init>(SparkContext.scala:137) at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArrayBenchmark$.withFakeTaskContext(ExternalAppendOnlyUnsafeRowArrayBenchmark.scala:53) at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArrayBenchmark$.testAgainstRawArrayBuffer(ExternalAppendOnlyUnsafeRowArrayBenchmark.scala:120) at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArrayBenchmark$.$anonfun$runBenchmarkSuite$1(ExternalAppendOnlyUnsafeRowArrayBenchmark.scala:190) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.benchmark.BenchmarkBase.runBenchmark(BenchmarkBase.scala:40) at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArrayBenchmark$.runBenchmarkSuite(ExternalAppendOnlyUnsafeRowArrayBenchmark.scala:187) at org.apache.spark.benchmark.BenchmarkBase.main(BenchmarkBase.scala:58) at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArrayBenchmark.main(ExternalAppendOnlyUnsafeRowArrayBenchmark.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
Bug: Spark submit should ignore cache for SNAPSHOT dependencies
Status: In Progress | Priority: Minor | Resolution: 
Created: 16/Mar/21 09:20
Affects Version/s: 3.1.1

When spark-submit is executed with --packages, it will not download the dependency jars when they are available in cache (e.g. ivy cache), even when the dependencies are SNAPSHOTs. 

This might block developers who work on external modules in Spark (e.g. spark-avro), since they need to remove the cache manually every time when they update the code during developments (which generates SNAPSHOT jars). Without knowing this, they could be blocked wondering why their code changes are not reflected in spark-submit executions.
Bug: Add virtual operators to the built-in function document.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/21 17:05
Affects Version/s: 2.4.7

After SPARK-34697, DESCRIBE FUNCTION and SHOW FUNCTIONS can describe/show built-in operators including the following virtual operators.

* !=
* <>
* between
* case
* ||

But they are still absent from the built-in functions document.
Bug: Cache Maven, SBT and Scala in all jobs that use them
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/21 04:51
Affects Version/s: 3.0.2

We should cache SBT, Maven and Scala for all jobs that use them. This is currently missing in some jobs such as https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L411-L430
Bug: [UI] StagePage input size/records not show when records greater than zero
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/21 16:55
Affects Version/s: 3.1.1

!No input size records.png|width=547,height=212!

The `Input Size / Records` should show in summary metrics table and task columns, as input records greater than zero and bytes is zero. One example is spark streaming job read from kafka
Bug: Local scheduler backend support update token
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Mar/21 02:44
Affects Version/s: 3.2.0

`LocalSchedulerBackend` doesn't extend `CoarseGrainedSchedulerBackend` so in local mode, we don't support update token.

In proxy use case with follow cmd, we will get exception
{code:java}
 ./bin/spark-shell --master local --proxy-user user_name

> spark.sql("show tables")
{code}


{code:java}
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:477)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
{code}
Bug: Helper class for batch Dataset.observe()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Mar/21 20:11
Affects Version/s: 3.2.0

The {{observe}} method has been added to the {{Dataset}} API in 3.0.0. It allows to collect aggregate metrics over data of a Dataset while they are being processed during an action.

These metrics are collected in a separate thread after registering {{QueryExecutionListener}} for batch datasets and {{StreamingQueryListener}} for stream datasets, respectively. While in streaming context it makes perfectly sense to process incremental metrics in an event-based fashion, for simple batch datatset processing, a single result should be retrievable without the need to register listeners or handling threading.

Introducing an {{Observation}} helper class can hide that complexity for simple use-cases in batch processing.

Similar to {{AccumulatorV2}} provided by {{SparkContext}} (e.g. {{SparkContext.LongAccumulator}}), the {{SparkSession}} can provide a method to create a new {{Observation}} instance and register it with the session.

Alternatively, an {{Observation}} instance could be instantiated on its own which on calling {{Observation.on(Dataset)}} registers with {{Dataset.sparkSession}}. This "registration" registers a listener with the session that retrieves the metrics.

The {{Observation}} class provides methods to retrieve the metrics. This retrieval has to wait for the listener to be called in a separate thread. So all methods will wait for this, optionally with a timeout:
 - {{Observation.get}} waits without timeout and returns the metric.
 - {{Observation.option(time, unit)}} waits at most {{time}}, returns the metric as an {{Option}}, or {{None}} when the timeout occurs.
 - {{Observation.waitCompleted(time, unit)}} waits for the metrics and indicates timeout by returning {{false}}.

Obviously, an action has to be called on the observed dataset before any of these methods are called, otherwise a timeout will occur.

With {{Observation.reset}}, another action can be observed. Finally, {{Observation.close}} unregisters the listener from the session.
Bug: Support SchemaRegistry in from_avro method
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Mar/21 18:23
Affects Version/s: 3.3.0

Confluent Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving your Avro®, JSON Schema, and Protobuf schemas. 
It would be nice to implement a new method that uses SchemaRegistry instead of the raw schema.
In addition, the DataBricks has already implemented this function. just check this link out:
https://docs.databricks.com/spark/latest/structured-streaming/avro-dataframe.html#example-with-schema-registry



Maybe it would be a simple and short method but really usefull
Bug: Add modified configs for SQL execution in UI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Mar/21 05:05
Affects Version/s: 3.2.0

For SQL user, it's very common to add some config to optimize sql. Within a script, it would look like this
{code:java}
set k1=v1;
set k2=v2;
set ...

INSERT INTO TABLE t1
SELECT ...

{code}
 
 It's hard to find the configs used by sql without the raw sql string. Current UI provide a `Environment` tab that we can only get some global initial config, however it's not enough.

Some use case:
 * Jar based job, we might set config many times due to many sql execution.
 * SQL server e.g. (SparkThriftServer), we might execute thousands scripts every day with different session.

We expect a feature that can list the modified configs which could affect the sql execution.
Bug: Support nested column in Spark vectorized readers
Status: In Progress | Priority: Minor | Resolution: 
Created: 25/Mar/21 08:53
Affects Version/s: 3.2.0

This is the umbrella task to track the overall progress. The task is to support nested column type in Spark vectorized reader, namely Parquet and ORC. Currently both Parquet and ORC vectorized readers do not support nested column type (struct, array and map). We implemented nested column vectorized reader for FB-ORC in our internal fork of Spark. We are seeing performance improvement compared to non-vectorized reader when reading nested columns. In addition, this can also help improve the non-nested column performance when reading non-nested and nested columns together in one query.

 

Parquet: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L173]

 

ORC:

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala#L138]
Bug: Use origin plan if can not coalesce shuffle partition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Mar/21 10:25
Affects Version/s: 3.2.0

The `CoalesceShufflePartitions` can not coalesce such case if the total shuffle partitions size of mappers are big enough. Then it's confused to use `CustomShuffleReaderExec` which marked as `coalesced` but has no affect with partition number.
Bug: Support Not(Int) and Not(InSet) propagate null
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Mar/21 12:07
Affects Version/s: 3.2.0

The semantics of `Not(In)` could be seen like `And(a != b, a != c)` that match the `NullIntolerant`.

As we already simplify the `NullIntolerant` expression to null if it's children have null. E.g. `a != null` => `null`. It's safe to do this with `Not(In)`/`Not(InSet)`.
Bug: Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Mar/21 13:27
Affects Version/s: 3.2.0

Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX):
{code:html}
21/02/25 10:10:45 INFO ApplicationCache: Failed to load application attempt application_1613641231234_0421/Some(1) 21/02/25 10:10:52 INFO FsHistoryProvider: Parsing hdfs://graph-product-001:8020/system/spark2-history/application_1613641231234_0421 for listing data...
{code}