av9ash/GitBugs huggingface dataset:
Bug: Spark SQL CodeGenerator does not check for UserDefined type
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 16/Oct/20 22:56
Affects Version/s: 2.4.7

The CodeGenerator takes the DataType given to   {{getValueFromVector()}} as is, and generates code based on its type. The generated code is not aware of the actual type, and therefore cannot be compiled. For example, using a DataFrame with a Spark ML Vector (VectorUDT) the generated code is:

{{InternalRow datasourcev2scan_value_2 = datasourcev2scan_isNull_2 ? null : (datasourcev2scan_mutableStateArray_2[2].getStruct(datasourcev2scan_rowIdx_0, 4));}}

{{ Which leads to a runtime error of}}

{{20/10/14 13:20:51 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 153, Column 126: No applicable constructor/method found for actual parameters "int, int"; candidates are: "public org.apache.spark.sql.vectorized.ColumnarRow org.apache.spark.sql.vectorized.ColumnVector.getStruct(int)"}}
{{ org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 153, Column 126: No applicable constructor/method found for actual parameters "int, int"; candidates are: "public org.apache.spark.sql.vectorized.ColumnarRow org.apache.spark.sql.vectorized.ColumnVector.getStruct(int)"}}
{{ at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124)}}
{{...}}


{{ which then throws Spark to an infinite loop of this error.}}

The solution is quite simple, {{getValueFromVector()}} should match nad handle UserDefinedType the same as {{CodeGenerator.javaType()}} is doing.
Bug: Support parse the sql statements with c-style comments
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Oct/20 06:22
Affects Version/s: 3.0.1

Now the spark-sql does not support parse the sql statements with C-style comments.
For the sql statements:
{code:java}
/* SELECT 'test'; */
SELECT 'test';
{code}
Would be split to two statements:
The first: "/* SELECT 'test'"
The second: "*/ SELECT 'test'"

Then it would throw an exception because the first one is illegal.
Bug: Fix sbt resolvers clash
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Oct/20 00:06
Affects Version/s: 3.1.0

During sbt upgrade from 0.13 to 1.x, exact resolvers list was used as is.

That leads to local resolvers name clashing, which is observed as warning from SBT:


{code:java}
[warn] Multiple resolvers having different access mechanism configured with same name 'local'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolve
r (`publishTo`).
{code}
This needs to be fixed to avoid potential errors and reduce log noise.
Bug: Optimizer rules for HigherOrderFunctions
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/Oct/20 16:11
Affects Version/s: 3.2.0

SimpleHigherOrderFunction like ArrayTransform, ArrayFilter, etc, can be combined and reordered to achieve more optimal plan.

Possible rules:
* Combine 2 consecutive array transforms
* Combine 2 consecutive array filters
* Push array filter through array sort
* Remove array sort before array exists and array forall.
* Combine 2 consecutive map filters
Bug: Too many "Auto-application to `()` is deprecated."  related compilation warnings
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Oct/20 09:34
Affects Version/s: 3.1.0

There are too many  "Auto-application to `()` is deprecated." related compilation warnings when compile with Scala 2.13 like
{code:java}
[WARNING] [Warn] /spark-src/core/src/test/scala/org/apache/spark/PartitioningSuite.scala:246: Auto-application to `()` is deprecated. Supply the empty argument list `()` explicitly to invoke method stdev,
or remove the empty argument list from its definition (Java-defined methods are exempt).
In Scala 3, an unapplied method like this will be eta-expanded into a function.
{code}
A lot of them, but it's easy to fix.

If there is a definition as follows:
{code:java}
Class Foo {
   def bar(): Unit = {}
}

val foo = new Foo{code}
Should be
{code:java}
foo.bar()
{code}
not
{code:java}
foo.bar {code}
Bug: PipelineModel load resulting in error
Status: Open | Priority: Minor | Resolution: 
Created: 20/Oct/20 04:24
Affects Version/s: 3.0.1

Steps to reproduce:
 # Define pipeline : Pipeline(stages=[discretizer, one_hot_encoder, cv1, cv2, assembler])
 # Save PipelineModel : PipelineModel.write().save('/path')
 # Load PipelineModel : PipelineModel.load('/path')

Getting following error:

TypeError: array() argument 1 must be a unicode character, not bytes

Error is in PickleSerializer class in park-3.0.1-bin-hadoop2.7/python/pyspark/serializers.py

Changing pickle.loads(obj, encoding=encoding) to pickle.loads(obj) resolves it.
Bug: Inconsistent NULL conversions to strings redux
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Oct/20 21:31
Affects Version/s: 3.0.2

The changes in SPARK-32501 introduced some behavior that I'd like to clean up a bit.

Here's sample code to illustrate the behavior I'd like to clean up:
{noformat}
val rows = Seq[String](null)
  .toDF("value")
  .withColumn("struct1", struct('value as "value1"))
  .withColumn("struct2", struct('value as "value1", 'value as "value2"))
  .withColumn("array1", array('value))
  .withColumn("array2", array('value, 'value))

// Show the DataFrame using the "first" codepath.        
rows.show(truncate=false)
+-----+-------+-------------+------+--------+
|value|struct1|struct2      |array1|array2  |
+-----+-------+-------------+------+--------+
|null |{ null}|{ null, null}|[]    |[, null]|
+-----+-------+-------------+------+--------+

// Write the DataFrame to disk, then read it back and show it to trigger the "codegen" code path:
rows.write.parquet("rows")
spark.read.parquet("rows").show(truncate=false)

+-----+-------+-------------+-------+-------------+
|value|struct1|struct2      |array1 |array2       |
+-----+-------+-------------+-------+-------------+
|null |{ null}|{ null, null}|[ null]|[ null, null]|
+-----+-------+-------------+-------+-------------+
{noformat}
Notice:
# If the first element of a struct is null, it is printed with a leading space (e.g. "\{ null}"). I think it's preferable to print it without the leading space (e.g. "\{null}"). This is consistent with how non-null values are printed inside a struct.
# If the first element of an array is null, it is not printed at all in the first code path, and the "codegen" code path prints it with a leading space. I think both code paths should be consistent and print it without a leading space (e.g. "[null]").

The desired result of this ticket is to product the following output via both code paths:
{noformat}
+-----+-------+------------+------+------------+
|value|struct1|struct2     |array1|array2      |
+-----+-------+------------+------+------------+
|null |{null} |{null, null}|[null]|[null, null]|
+-----+-------+------------+------+------------+
{noformat}
Bug: Rule SimplifyCasts will not work for nested columns
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 30/Oct/20 09:13
Affects Version/s: 3.0.0

We use SparkSQL and Catalyst to optimize the Spark job. We have read the source code and test the rule of SimplifyCasts which will work for simple SQL without nested cast.

The SQL "select cast(string_date as string) from t1" will be optimized.

{code:java}
== Analyzed Logical Plan ==
string_date: string
Project [cast(string_date#12 as string) AS string_date#24]
+- SubqueryAlias t1
 +- LogicalRDD [name#8, c1#9, c2#10, c5#11L, string_date#12, string_timestamp#13, timestamp_field#14, bool_field#15], false

== Optimized Logical Plan ==
Project [string_date#12]
+- LogicalRDD [name#8, c1#9, c2#10, c5#11L, string_date#12, string_timestamp#13, timestamp_field#14, bool_field#15], false
{code}

However, it fail to optimize with the nested cast like this "select cast(cast(string_date as string) as string) from t1".

{code:java}
== Analyzed Logical Plan ==
CAST(CAST(string_date AS STRING) AS STRING): string
Project [cast(cast(string_date#12 as string) as string) AS CAST(CAST(string_date AS STRING) AS STRING)#24]
+- SubqueryAlias t1
 +- LogicalRDD [name#8, c1#9, c2#10, c5#11L, string_date#12, string_timestamp#13, timestamp_field#14, bool_field#15], false

== Optimized Logical Plan ==
Project [string_date#12 AS CAST(CAST(string_date AS STRING) AS STRING)#24]
+- LogicalRDD [name#8, c1#9, c2#10, c5#11L, string_date#12, string_timestamp#13, timestamp_field#14, bool_field#15], false
{code}
Bug: Remove useless externalBlockStoreSize from RDDInfo 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Oct/20 07:18
Affects Version/s: 3.1.0

"external block store" API was removed after SPARK-12667, {{externalBlockStoreSize}} in {{RDDInfo}} looks like always 0 and useless. So this pr just to remove this useless variable.
Bug: Migrate DROP TABLE to new resolution framework
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/20 03:20
Affects Version/s: 3.1.0

Migrate "DROP TABLE t" to new resolution framework so that it has a consistent resolution rule for v1/v2 commands.

Currently, the following resolves to a table instead of a temp view:

{code:java}
sql("CREATE TABLE testcat.ns.t (id bigint) USING foo")
sql("CREATE TEMPORARY VIEW t AS SELECT 2 as id")
sql("USE testcat.ns")
sql("DROP TABLE t") // 't' is resolved to testcat.ns.t
{code}
Bug: Add status plugin for live application
Status: In Progress | Priority: Minor | Resolution: 
Created: 27/Oct/20 02:32
Affects Version/s: 2.4.7

There are cases that developer may want to extend the current REST API of Web UI. In most cases, adding external module is a better option than directly editing the original Spark code.

For an external module, to extend the REST API of the Web UI, 2 things may need to be done:
 * Add extra API to provide extra status info. This can be simply done by implementing another ApiRequestContext which will be automatically loaded.
 * If the info can not be calculated from the original data in the store, add extra listeners to generate them.

For history server, there is an interface called AppHistoryServerPlugin, which is loaded based on SPI, providing a method to create listeners. In live application, the only way is spark.extraListeners based on Utils.loadExtensions. But this is not enough for the cases.

To let the API get the status info, the data need to be written to the AppStatusStore, which is the only store that an API can get by accessing "ui.store" or "ui.sc.statusStore". But listeners created by Utils.loadExtensions only get a SparkConf in construction, and are unable to write the AppStatusStore.

So I think we still need plugin like AppHistorySever for live UI. For concerns like SPARK-22786, the plugin for live app can be separated from the history server one, and also loaded using Utils.loadExtensions with an extra configurations. So by default, nothing will be loaded.
Bug: In Execution web page, when `Succeeded Job` has many child url elements,they will extend over the edge of the page. 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Oct/20 10:35
Affects Version/s: 3.0.0

Spark Version:3.0.1

Problem: In Execution web page, when *{color:#de350b}Succeeded Jobs(or failed Jobs){color}* has many child url elements,they will extend over the edge of the page, as the attachment shows.
Bug: Disable GetPrimaryKeys and GetCrossReference APIs explicitly for Spark ThriftServer
Status: Open | Priority: Minor | Resolution: 
Created: 22/Oct/20 07:30
Affects Version/s: 3.1.0

We do not have those features supported yet, users may get confused by the results which generated directly by Hive implementation

e.g.
https://github.com/yaooqinn/kyuubi/blob/master/kyuubi-common/src/main/scala/org/apache/kyuubi/service/FrontendService.scala#L369-L383
Bug: Pyspark ml tests failing with rounding errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Oct/20 07:45
Affects Version/s: 3.0.1

The tests _{{pyspark.ml.recommendation}}_ and _{{pyspark.ml.tests.test_algorithms}}_ occasionally fail (depends on environment) with
{code:java}
File "/home/jenkins/python/pyspark/ml/tests/test_algorithms.py", line 96, in test_raw_and_probability_prediction
 self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1))
AssertionError: False is not true{code}
{code:java}
File "/home/jenkins/python/pyspark/ml/recommendation.py", line 256, in _main_.ALS
Failed example:
 predictions[0]
Expected:
 Row(user=0, item=2, newPrediction=0.6929101347923279)
Got:
 Row(user=0, item=2, newPrediction=0.6929104924201965)
...{code}
Bug: Early state store eviction for left semi stream-stream join
Status: Open | Priority: Minor | Resolution: 
Created: 21/Oct/20 20:17
Affects Version/s: 3.1.0

As a followup from discussion [https://github.com/apache/spark/pull/30076/files/3918727a08c8d0d4c65ccc8ea902f77051b78b1d#r508926034] and [https://github.com/apache/spark/pull/30076#discussion_r509222802] , for left semi stream-stream join, the matched left side rows can be evicted from left state store immediately, without waiting for to be below watermark. However it needs more thought for how to implement efficiently to not iterate all values if watermark predicate is on key.
Bug: Expose filtered aggregation API
Status: Resolved | Priority: Minor | Resolution: Won't Do
Created: 20/Oct/20 17:14
Affects Version/s: 3.0.0

Spark currently supports filtered aggregation but does not expose API allowing to use them when using the `spark.sql.functions` package.

It is possible to use them when writing directly SQL:
{code:scala}
scala> val df = spark.range(100)
scala> df.registerTempTable("df")
scala> spark.sql("select count(1) as classic_cnt, count(1) FILTER (WHERE id < 50) from df").show()
+-----------+-------------------------------------------------+ 
|classic_cnt|count(1) FILTER (WHERE (id < CAST(50 AS BIGINT)))|
+-----------+-------------------------------------------------+
|        100|                                               50|
+-----------+-------------------------------------------------+{code}
These aggregations are especially useful when filtering on overlapping datasets (where a pivot would not work):
{code:sql}
SELECT 
 AVG(revenue) FILTER (WHERE age < 25),
 AVG(revenue) FILTER (WHERE age < 35),
 AVG(revenue) FILTER (WHERE age < 45)
FROM people;{code}

I did not find an issue tracking this, hence I am creating this one and I will join a PR to illustrate a possible implementation.
Bug: spark doesn't read data source column if it is used as an index to an array under a struct
Status: Open | Priority: Minor | Resolution: 
Created: 19/Oct/20 18:06
Affects Version/s: 3.0.0

{code:python}
df = spark.createDataFrame([[1, [[1, 2]]]], schema='x:int,y:struct<a:array<int>>')
df.write.mode('overwrite').parquet('test')
{code}

{code:python}
# This causes an error "Caused by: java.lang.RuntimeException: Couldn't find x#720 in [y#721]"
spark.read.parquet('test').select(F.expr('y.a[x]')).show()

# Explain works fine, note it doesn't read x in ReadSchema
spark.read.parquet('test').select(F.expr('y.a[x]')).explain()

== Physical Plan ==
*(1) !Project [y#713.a[x#712] AS y.a AS `a`[x]#717]
+- FileScan parquet [y#713] Batched: false, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<y:struct<a:array<int>>>
{code}


The code works well if I 

{code:python}
# manually select the column it misses
spark.read.parquet('test').select(F.expr('y.a[x]'), F.col('x')).show()

# use element_at function
spark.read.parquet('test').select(F.element_at('y.a', F.col('x') + 1)).show()
{code}
Bug: Replace compiler reporter with more robust and maintainable solution
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Oct/20 22:24
Affects Version/s: 3.1.0

Currently existing solution to have build to fail on any warning except deprecation ones ([https://github.com/apache/spark/blob/v3.0.1/project/SparkBuild.scala#L285)] is not very maintainable in scope of build upgrade to latest sbt. 

At upgrade to sbt 1.4.0 this snippet would fail build import at all.

Implement new solution, using switch over compiler versions, silencer compiler plugin for Scala prior 2.13.2 and built-in warning configuration since Scala 2.13.2

Depends on changes for SPARK-21708
Bug: The 'Shuffle Read Size / Records' field in Stage Summary metrics was shown as 'NaN Undefined'
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Oct/20 11:36
Affects Version/s: 3.0.1

Spark Version: 3.0.1

Description: The 'Shuffle Read Size / Records' field in Stage Summary metrics was shown as '*NaN Undefind*'  when the *readBytes* value is negative,  as the attachment shows.

*curl http:/hadoop001:18081/api/v1/applications/application_1601774913550_0225/stages/2/0/taskSummary*

{
 "quantiles" : [ 0.05, 0.25, 0.5, 0.75, 0.95 ],
 "executorDeserializeTime" : [ 7.0, 357.0, 390.0, 484.0, 492.0 ],
  ...
 "shuffleReadMetrics" :

{ {color:#de350b}"*readBytes*"{color} : [ {color:#de350b}*-2.0*{color}, 1775984.0, 1779759.0, 1781727.0, 1788426.0 ], "readRecords" : [ 2001.0, 2002.0, 2002.0, 2002.0, 2002.0 ],  ... }
Bug: Change Executor  filesystem.* metrics to type Counter
Status: Open | Priority: Minor | Resolution: 
Created: 13/Oct/20 08:51
Affects Version/s: 2.4.5

i recently integrated the spark executor metrics documented in [Spark Monitoring|https://spark.apache.org/docs/3.0.0/monitoring.html#component-instance--executor] into our prometheus based monitoring system.

In a discussion here [https://github.com/prometheus/jmx_exporter/pull/543] we noticed that the filesystem metrics are wrongly declared as type Gauge, although they are counter metrics.

 

I guess every spark version is affected which contains this metrics (i checked at least 2.4.5 and 3.0.0)

It is just a minor bug, but could be fixed at some point.
Bug: Optionally skip remote/local dependency resolution at submission client in k8s cluster mode
Status: Open | Priority: Minor | Resolution: 
Created: 07/Oct/20 07:43
Affects Version/s: 2.4.7

Usecase:
 # Users use Apache Livy to submit spark jobs.  Livy pod's spark-submit command will be invoked to submit jobs to k8e cluster with *--deploy-mode cluster*
 # The Livy pod (submission client) does *NOT* have permission (secure token) to access remote dependencies specified by *--files, --jars, --py-files* from S3, gcs, hdfs:// . Only future driver pod and executor pod will have permissions with dynamic secure tokens to download those remote files.

Is it possible to support options to *disable* remote resource resolution/download at Livy pod or submission client host? 

Of course, users/developers will make sure that
 # Remote files are accessible by driver and executor pods with secure tokens.
 # files with local paths are placed into driver and executor in advance.
Bug: RowMatrix is incompatible with spark.driver.maxResultSize=0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Oct/20 00:31
Affects Version/s: 3.0.0

RowMatrix does not work if spark.driver.maxResultSize=0, as this requirement breaks:

 
{code:java}
require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,      s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "        + s"as it's bigger than maxResultSize ($maxDriverResultSizeInBytes Bytes)")
{code}
 

[https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala#L795.]

 

This check should likely only happen if maxDriverResultSizeInBytes > 0.
Bug: Improve error message for insufficient K8s volume confs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Oct/20 08:56
Affects Version/s: 3.1.0

Providing error handling when creating k8s volumes and clearer error messages.

For example, when creating a *hostPath* volume, if you don't specify 


{code:java}
hostPath.volumeName.options.path
{code}
 it fails with a 
{code:java}
 key not found error
{code}
 which is clear that you are missing a key, but I couldn't find anywhere in the docs that says you need to specify it.


To reproduce the issue, you have to do the spark-submit command like this for example:

{code:java}

./bin/spark-submit \
--master k8s://https://127.0.0.1:32768 \
--deploy-mode cluster \
--name spark-app\
--class class \
--conf spark.kubernetes.driver.volumes.hostPath.spark.mount.path=/tmp/jars/ \
--conf spark.kubernetes.executor.volumes.hostPath.spark.mount.path=/tmp/jars/ \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=spark:latest \
 local:///opt/spark/examples/jars/app.jar

{code}
Bug: Expand the stack size of a thread in a test in LocalityPlacementStrategySuite for Java 11
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Oct/20 21:41
Affects Version/s: 3.1.0

`LocalityPlacementStrategySuite` fails with Java 11 due to `StackOverflowError`.

{code}
[info] - handle large number of containers and tasks (SPARK-18750) *** FAILED *** (170 milliseconds)
[info]   StackOverflowError should not be thrown; however, got:
[info]   
[info]   java.lang.StackOverflowError
[info]          at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
[info]          at java.base/java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1541)
[info]          at java.base/java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:668)
[info]          at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:591)
[info]          at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
[info]          at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
[info]          at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
{code}
Bug: Spark SQL null semantics documentation is incorrect
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/Oct/20 17:03
Affects Version/s: 3.0.2

The documentation of Spark SQL's null semantics is (I believe) incorrect.

The documentation states that "NULL AND False" yields NULL, when in fact it yields False.

{noformat}
Seq[(java.lang.Boolean, java.lang.Boolean)](
  (true, null),
  (false, null),
  (null, true),
  (null, false),
  (null, null)
)
  .toDF("left_operand", "right_operand")
  .withColumn("OR", 'left_operand || 'right_operand)
  .withColumn("AND", 'left_operand && 'right_operand)
  .show(truncate = false)

+------------+-------------+----+-----+
|left_operand|right_operand|OR  |AND  |
+------------+-------------+----+-----+
|true        |null         |true|null |
|false       |null         |null|false|
|null        |true         |true|null |
|null        |false        |null|false|  <---- this line is incorrect in the docs
|null        |null         |null|null |
+------------+-------------+----+-----+
{noformat}
Bug: Clean up unit test file UnsupportedOperationsSuite.scala
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Oct/20 19:50
Affects Version/s: 3.1.0

As a follow up from [https://github.com/apache/spark/pull/30076,] there are many copy-paste in the unit test file UnsupportedOperationsSuite.scala to check different join types (inner, outer, semi) with similar code structure. It would be helpful to clean them up and refactor to reuse code.