av9ash/GitBugs huggingface dataset:
Bug: HiveSessionCatalog call super.makeFunctionExpression should show error message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/20 12:59
Affects Version/s: 3.0.0

When we create a UDAF function use class extended UserDefinedAggregetFunction,  when we call the function,  in support hive mode, in HiveSessionCatalog, it will call super.makeFunctionExpression, 

but it will catch error  such as the function need 2 parameter and we only give 1, throw exception only show 
{code:java}
No handler for UDF/UDAF/UDTF xxxxxxxx
{code}
This is confused , we should show error thrown by super method too.
Bug: Remove Python2 test from K8s IT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/20 17:59
Affects Version/s: 3.1.0

Since we removed Python 2 support, the test case fails.
{code}
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example *** FAILED ***
  The code passed to eventually never returned normally. Attempted 113 times over 2.0014854648999996 minutes. Last failure message: false was not true. (KubernetesSuite.scala:370)
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
- Run SparkR on simple dataframe.R example
Run completed in 11 minutes, 15 seconds.
Total number of tests run: 20
Suites: completed 2, aborted 0
Tests: succeeded 19, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
{code}
Bug: `approx_percentile` in Spark SQL gives incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/20 12:35
Affects Version/s: 2.4.4

The `approx_percentile` function in Spark SQL does not give the correct result. I'm not sure how incorrect it is; it may just be a boundary issue. From the docs:
{quote}The accuracy parameter (default: 10000) is a positive numeric literal which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation.
{quote}
This is not true. Here is a minimum example in `pyspark` where, essentially, the median of 5 and 8 is being calculated as 5:
{code:python}
import pyspark.sql.functions as psf

df = spark.createDataFrame(
    [('bar', 5), ('bar', 8)], ['name', 'val']
)
median = psf.expr('percentile_approx(val, 0.5, 2147483647)')

df.groupBy('name').agg(median.alias('median'))    # gives the median as 5
{code}
I've tested this with Spark v2.4.4, pyspark v2.4.5- although I suspect this is an issue with the underlying algorithm.
Bug: Fix HistoryServerSuite in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 27/Jul/20 05:38
Affects Version/s: 3.1.0

{code}
- app environment *** FAILED ***
  java.lang.AssertionError: assertion failed: Expected:
...
- download all logs for app with multiple attempts *** FAILED ***
  404 was not equal to 200 (HistoryServerSuite.scala:249)
...
- response codes on bad paths *** FAILED ***
...
- ui and api authorization checks *** FAILED ***
  404 did not equal 200 Unexpected status code 404 for http://localhost:18080/api/v1/applications/local-1430917381535/logs (user = irashid) (HistoryServerSuite.scala:580)
{code}
Bug: tmpfs=true, not pointing to SPARK_LOCAL_DIRS in k8s
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 09/Jul/20 21:44
Affects Version/s: 3.0.0

In Spark-Submit, I have these config "{color:#4c9aff}*spark.kubernetes.local.dirs.tmpfs=true*{color}", still spark is not pointing its spill data to SPARK_LOCAL_DIRS path.

K8s is evicting the pod due to error "{color:#de350b}*Pod ephemeral local storage usage exceeds the total limit of containers.*{color}"

 

We use Spark launcher to do spark submit in k8s. Since it is evicted, the pod logs for stack trace is not available. we have only pod events given in attachment
Bug: Use CaseInsensitiveMap for DataFrameReader/Writer options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/20 05:57
Affects Version/s: 2.0.2

When a user have multiple options like path, paTH, and PATH for the same key path, option/options is non-deterministic because extraOptions is HashMap. This issue aims to use *CaseInsensitiveMap* instead of *HashMap* to fix this bug fundamentally.

{code}
spark.read
  .option("paTh", "1")
  .option("PATH", "2")
  .option("Path", "3")
  .option("patH", "4")
  .load("5")
...
org.apache.spark.sql.AnalysisException:
Path does not exist: file:/.../1;
{code}
Bug: Update jackson-databind versions to fix various vulnerabilities.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Jul/20 10:39
Affects Version/s: 2.4.6

As a vulnerability for Fasterxml Jackson version 2.6.7.3 is affected by CVE-2017-15095 and CVE-2018-5968 CVEs [https://nvd.nist.gov/vuln/detail/CVE-2018-5968], Would it be possible to upgrade the jackson version for spark-2.4.6 and so on(2.4.x).
Bug: Infer filters from DPP
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/20 10:23
Affects Version/s: 3.1.0

For [this test suite|https://github.com/apache/spark/blob/69be17bfc3845668fd8f5aed8026c04da4496677/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala#L1285-L1297].
Table {{fact_stats}} is partitioned by {{store_id}} and table {{code_stats}} is partitioned by {{store_id}}.
DPP add a new predicate: {{fact_stats.store_id IN dynamicpruning#2723}}.
We can infer {{code_stats.store_id IN dynamicpruning#2723}} base on [{{t1.store_id = t2.store_id}}|https://github.com/apache/spark/blob/69be17bfc3845668fd8f5aed8026c04da4496677/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala#L1294].
Bug: Add support to catch SparkPlan regression base on TPC-DS queries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/20 11:41
Affects Version/s: 3.1.0

Nowadays, Spark is getting more and more complex. Any changes might cause regression unintentionally. Spark already has some benchmark to catch the performance regression. But, yet, it doesn't have a way to detect the regression inside SparkPlan. It would be good if we could find some possible regression early during the compile phase before the runtime phase.
Bug: Partially push down support data filter if it mixed in partition filters 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/20 16:53
Affects Version/s: 3.1.0

We support partially push partition filters since SPARK-28169. We can also support partially push down data filters if it mixed in partition filters and data filters. For example:
{code:scala}
spark.sql(
  s"""
     |CREATE TABLE t(i INT, p STRING)
     |USING parquet
     |PARTITIONED BY (p)""".stripMargin)

spark.range(0, 1000).selectExpr("id as col").createOrReplaceTempView("temp")
for (part <- Seq(1, 2, 3, 4)) {
  sql(s"""
         |INSERT OVERWRITE TABLE t PARTITION (p='$part')
         |SELECT col FROM temp""".stripMargin)
}

spark.sql("SELECT * FROM t WHERE  WHERE (p = '1' AND i = 1) OR (p = '2' and i = 2)").explain()
{code}
We can also push down {{ i = 1 or i = 2 }}.
Bug: Chinese characters are garbled when opening csv files with Excel
Status: Resolved | Priority: Major | Resolution: Workaround
Created: 13/Jul/20 03:52
Affects Version/s: 3.1.0

How to reproduce this issue:

{code:scala}
spark.sql("SELECT '我爱中文' AS chinese").write.option("header", "true").csv("/tmp/spark/csv")
{code}

 !garbled.png!
Bug: Writing timestamp in mysql gets fails 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Jul/20 05:57
Affects Version/s: 2.3.4

When we are writing to mysql with TIMESTAMP column it supports only range '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07'. Mysql has DATETIME datatype which has 1000-01-01 00:00:00' to '9999-12-31 23:59:59' range.

How to map spark timestamp datatype to mysql datetime datatype in order to use higher supporting range ?

[https://dev.mysql.com/doc/refman/5.7/en/datetime.html]
Bug: Please fix CVE related to Guava 14.0.1
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Jul/20 19:40
Affects Version/s: 3.0.0

Please fix the following CVE related to Guava 14.0.1
|cve|severity|cvss|
|CVE-2018-10237|medium|5.9|

 

Our security team is trying to block us from using spark because of this issue

 

One thing that's very weird is I see from this [pom file|[https://github.com/apache/spark/blob/v3.0.0/common/network-common/pom.xml]] you reference guava but it's not clear what version.

 

But if I look on [maven|[https://mvnrepository.com/artifact/org.apache.spark/spark-network-common_2.12/3.0.0]] the guava reference is not showing up

 

Is this reference somehow being shaded into the network common jar?  It's not clear to me.

 

Also, I've noticed code like [this file|[https://github.com/apache/spark/blob/v3.0.0/common/network-common/src/main/java/org/apache/spark/network/util/LimitedInputStream.java]] which is a copy-paste of some guava source code.

 

The CVE scanner we use Twistlock/Palo Alto Networks - Prisma Cloud Compute Edition is very thorough and will find CVEs in copy-pasted code and shaded jars.

 

Please fix this CVE so we can use spark
Bug: 11 Critical & 4 High severity issues in Apcahe Spark 3.0.0 - dependency libraries
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Jul/20 22:08
Affects Version/s: 3.0.0

||*[CVE-2018-1337|https://nvd.nist.gov/vuln/detail/CVE-2018-1337]*|In Apache Directory LDAP API before 1.0.2,   - upgrade dependency to 1.0.2|

||*[CVE-2018-17190|https://nvd.nist.gov/vuln/detail/CVE-2018-17190]*|In all versions of Apache Spark,|

||*[CVE-2017-15718|https://nvd.nist.gov/vuln/detail/CVE-2017-15718]*|The YARN NodeManager in Apache Hadoop 2.7.3 and 2.7.4 - upgrade lib|

||*[CVE-2018-21234|https://nvd.nist.gov/vuln/detail/CVE-2018-21234]*|Jodd before 5.0.4 performs Deserialization of Untrusted JSON Data when setClassMetadataName is set.|

||*[CVE-2019-17571|https://nvd.nist.gov/vuln/detail/CVE-2019-17571]*|Included in Log4j 1.2 is a SocketServer class that is vulnerable to deserialization of untrusted data which can be exploited to remotely execute arbitrary code when combined with a deserialization gadget when listening to untrusted network traffic for log data. This affects Log4j versions up to 1.2 up to 1.2.17.|

||*[CVE-2018-17190|https://nvd.nist.gov/vuln/detail/CVE-2018-17190]*|In all versions of Apache Spark, its standalone resource manager accepts code to execute on a 'master' host, that then runs that code on 'worker|

||*[CVE-2020-9480|https://nvd.nist.gov/vuln/detail/CVE-2020-9480]*|In Apache Spark 2.4.5 and earlier, a standalone resource manager's master may be configured to require authentication (spark.authenticate) via a shared secret.|
Bug: Remove redundant ProjectExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/20 23:33
Affects Version/s: 3.0.0

Currently Spark executed plan can have redundant `ProjectExec` node. For example: 

After Filter: 
{code:java}
== Physical Plan ==
*(1) Project [a#14L, b#15L, c#16, key#17] 
+- *(1) Filter (isnotnull(a#14L) AND (a#14L > 5))
   +- *(1) ColumnarToRow
      +- FileScan parquet [a#14L,b#15L,c#16,key#17] {code}
The `Project [a#14L, b#15L, c#16, key#17]` is redundant because its output is exactly the same as filter's output.

Before Aggregate:
{code:java}
== Physical Plan ==
*(2) HashAggregate(keys=[key#17], functions=[sum(a#14L), last(b#15L, false)], output=[sum_a#39L, key#17, last_b#41L])
+- Exchange hashpartitioning(key#17, 5), true, [id=#77]
   +- *(1) HashAggregate(keys=[key#17], functions=[partial_sum(a#14L), partial_last(b#15L, false)], output=[key#17, sum#49L, last#50L, valueSet#51])
      +- *(1) Project [key#17, a#14L, b#15L]
         +- *(1) Filter (isnotnull(a#14L) AND (a#14L > 100))
            +- *(1) ColumnarToRow
               +- FileScan parquet [a#14L,b#15L,key#17] {code}
The `Project [key#17, a#14L, b#15L]` is redundant because hash aggregate doesn't require child plan's output to be in a specific order.

 

In general, a project is redundant when
 # It has the same output attributes and order as its child's output when ordering of these attributes is required.
 # It has the same output attributes as its child's output when attribute output ordering is not required.
Bug: ApplyColumnarRulesAndInsertTransitions should be idempotent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/20 17:10
Affects Version/s: 3.1.0

It's good hygiene to keep rules idempotent, even if we know the rule is going to be run only once. This is for future-proof.

ApplyColumnarRulesAndInsertTransitions can add columnar-to-row and row-to-columnar operators repeatedly, we should fix it.
Bug: NotInSubquery SingleColumn Optimize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/20 07:38
Affects Version/s: 3.0.0

Normally,
A NotInSubquery will plan into BroadcastNestedLoopJoinExec, which is very very time consuming. For example, I've done TPCH benchmark lately, Query 16 almost took half of the entire TPCH 22Query execution Time. So i proposed that to do the following optimize.

Inside BroadcastNestedLoopJoinExec, we can identify not in subquery with only single column in following pattern.

{code:java}
case _@Or(
            _@EqualTo(leftAttr: AttributeReference, rightAttr: AttributeReference),
            _@IsNull(
              _@EqualTo(_: AttributeReference, _: AttributeReference)
            )
          )
{code}

if buildSide rows is small enough, we can change build side data into a HashMap.
so the M*N calculation can be optimized into M*log(N)

I've done a benchmark job in 1TB TPCH, before apply the optimize
Query 16 take around 18 mins to finish, after apply the M*log(N) optimize, it takes only 30s to finish. 

But this optimize only works on single column not in subquery, so i am here to seek advise whether the community need this update or not. I will do the pull request first, if the community member thought it's hack, it's fine to just ignore this request.
Bug: flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 08:40
Affects Version/s: 3.1.0

{code}
FAIL: test_train_prediction (pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests)
Test that error on test data improves as model is trained.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 466, in test_train_prediction
    eventually(condition, timeout=180.0)
  File "/home/runner/work/spark/spark/python/pyspark/testing/utils.py", line 81, in eventually
    lastValue = condition()
  File "/home/runner/work/spark/spark/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 461, in condition
    self.assertGreater(errors[1] - errors[-1], 2)
AssertionError: 1.672640157855923 not greater than 2
{code}
Bug: [SQL Parser] Report Error for invalid usage of SET command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/20 17:06
Affects Version/s: 3.0.0

{code:java}
SET spark.sql.ansi.enabled true{code}

 The above SQL command does not change the conf value and it just tries to display the value of conf "spark.sql.ansi.enabled true".

We can disallow using the space in the conf name and issue a user friendly error instead. In the error message, we should tell users a workaround to use the quote if they still needs to specify a conf with a space.
Bug: Inconsistent default unit between Spark memory configs and JVM option
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Jul/20 18:11
Affects Version/s: 2.2.1

Spark's maximum memory can be configured in several ways:
- via Spark config
- command line argument
- environment variables 

Both for executors and for the driver the memory can be configured separately. All of these are following the format of JVM memory configurations in a way they are using the very same size unit suffixes ("k", "m", "g" or "t") but there is an inconsistency regarding the default unit. When no suffix is given then the given amount is passed as it is to the JVM (to the -Xmx and -Xms options) where this memory options are using bytes as a default unit, for this please see the example [here|https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html]:

{noformat}
The following examples show how to set the maximum allowed size of allocated memory to 80 MB using various units:

-Xmx83886080 
-Xmx81920k 
-Xmx80m
{noformat}

Although the Spark memory config default suffix unit is "m".
Bug: Rename classes/files with the Jdbc prefix to JDBC
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Jul/20 18:48
Affects Version/s: 3.1.0

I have found 7 files with Jdbc:
{code}
➜ apache-spark git:(master) find . -name "Jdbc*.scala" -type f
./core/src/test/scala/org/apache/spark/rdd/JdbcRDDSuite.scala
./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtilsSuite.scala
./sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
./sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/JdbcConnectionUriSuite.scala
{code}
and 8 starts from JDBC:
{code}
➜ apache-spark git:(master) find . -name "JDBC*.scala" -type f
./sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
./sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalogSuite.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala
./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTable.scala
{code}
Bug: Issue with deserialization and persist api in latest spark java versions
Status: Reopened | Priority: Major | Resolution: 
Created: 29/Jul/20 14:58
Affects Version/s: 2.4.4

Hey Team, We have class level object instantiations in one of our Classes. When we want to persist that data into the Dataset of this class Type it's not persisting the null values instead it's taking class level precedence. i.e. It's showing as new object.

Eg: 

_Test.class has below class level attributes:_

_private Test1 testNumber = new Test1();_

_private Test2 testNumber2;_

 

String inputLocation = "src/test/resources/pipeline/test.parquet";

Dataset<Row> ds = this.session.read().parquet(inputLocation);
ds.printSchema();
ds.foreach(input->{
 System.out.println(input); // When we verified it's showing testNumber, testNumber2 as null
});
Dataset<Test> inputDataSet = ds.as(Encoders.bean(Test.class));

inputDataSet.foreach(input->{
 System.out.println(input); // When we verified it's showing testNumber as new Test1(), testNumber2 as null
});

 

 

This is the same issue with dataset.persist() call aswell. It is happening with all 2.4.4 and higher versions. Can you please fix it?
Bug: Unused DPP Filter causes issue in canonicalization and prevents reuse exchange
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 12:05
Affects Version/s: 3.0.0

As part of PlanDynamicPruningFilter rule, the unused DPP Filter are simply replaced by `DynamicPruningExpression(TrueLiteral)` so that they can be avoided. But these unnecessary`DynamicPruningExpression(TrueLiteral)` partition filter inside the FileSourceScanExec affects the canonicalization of the node and so in many cases, this can prevent ReuseExchange from happening.
Bug: JDBC doesn't check duplicate column names in nested structures 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 12:12
Affects Version/s: 3.1.0

JdbcUtils.getCustomSchema calls checkColumnNameDuplication() which checks duplicates on top-level but not in nested structures as other built-in datasources do, see

[https://github.com/apache/spark/blob/8bc799f92005c903868ef209f5aec8deb6ccce5a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L822-L823]
Bug: Add in the ability for a user to replace the serialization format of the cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/20 18:12
Affects Version/s: 3.1.0

Caching a dataset or dataframe can be a very expensive operation, but has a huge benefit for later queries that use it.  There are many use cases that could benefit from caching the data but not enough to justify the current scheme.  I would like to propose that we make the serialization of the caching plugable.  That way users can explore other formats and compression code.

 

As an example I took the line item table from TPCH at a scale factor of 10 and converted it to parquet.  This resulted in 2.1 GB of data on disk. With the current caching it can take nearly 8 GB to store that same data in memory, and about 5 GB to store in on disk.

 

If I want to read all of that data and and write it out again.
{code:java}
scala> val a = spark.read.parquet("../data/tpch/SF10_parquet/lineitem.tbl/")
a: org.apache.spark.sql.DataFrame = [l_orderkey: bigint, l_partkey: bigint ... 14 more fields]

scala> spark.time(a.write.mode("overwrite").parquet("./target/tmp"))
Time taken: 25832 ms {code}
But a query that reads that data directly from the cache after it is built only takes 21531 ms. For some queries having much more data that can be stored in the cache might be worth the extra query time.

 

It also takes about a lot less time to do the parquet compression than it does to do the cache compression.
Bug: AQE doesn't adequately allow for Columnar Processing extension 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/20 14:50
Affects Version/s: 3.0.0

In SPARK-27396 we added support to extended Columnar Processing. We did the initial work as to what we thought was sufficient but adaptive query execution was being developed at the same time.

We have discovered that the changes made to AQE are not sufficient for users to properly extend it for columnar processing because AQE hardcodes to look for specific classes/execs.