av9ash/GitBugs huggingface dataset:
Bug: Eliminate deprecation warnings in ORC datasource
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 12/Jan/20 20:40
Affects Version/s: 2.4.4

While compiling the sql/hive/src/main/java/org/apache/hadoop/hive/ql/io/orc/SparkOrcNewRecordReader.java, the compiler outputs the following warnings:
{code}
Warning:(44, 13) java: getTypes() in org.apache.orc.Reader has been deprecated
Warning:(47, 24) java: getTypes() in org.apache.orc.Reader has been deprecated
{code}
The ticket aims to eliminate the warnings.
Bug: Remove SQL configs deprecated before v2.4
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jan/20 10:18
Affects Version/s: 3.0.0

Remove the following SQL configs:
 * spark.sql.variable.substitute.depth
 * spark.sql.execution.pandas.respectSessionTimeZone
 * spark.sql.parquet.int64AsTimestampMillis

Recently all deprecated SQL configs were gathered to the deprecatedSQLConfigs map:
 [https://github.com/apache/spark/blob/1ffa627ffb93dc1027cb4b72f36ec9b7319f48e4/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2160-L2189]
Bug: Use PeriodicRDDCheckpointer instead of NodeIdCache
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/20 09:27
Affects Version/s: 3.0.0

Both PeriodicRDDCheckpointer and NodeIdCache are used in tree models:

PeriodicRDDCheckpointer is used in GBT to maintain prediction and errors,

NodeIdCache is used to maintain NodeIds.

 

The functionality of them are highly similar, and the update process of NodeIdCache is simply. I guess we can delete NodeIdCache and use PeriodicRDDCheckpointer instead.
Bug: The Executor Computing Time in Time Line of Stage Page is Wrong
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/20 11:24
Affects Version/s: 2.0.2

The Executor Computing Time in Time Line of Stage Page is Wrong. It includes the Scheduler Delay Time, while the Proportion excludes the Scheduler Delay

val executorOverhead = serializationTime + deserializationTime
val executorRunTime = if (taskInfo.duration.isDefined) {
 totalExecutionTime - executorOverhead - gettingResultTime
} else {
 metricsOpt.map(_.executorRunTime).getOrElse(
 totalExecutionTime - executorOverhead - gettingResultTime)
}
val executorComputingTime = executorRunTime - shuffleReadTime - shuffleWriteTime
val executorComputingTimeProportion =
 math.max(100 - schedulerDelayProportion - shuffleReadTimeProportion -
 shuffleWriteTimeProportion - serializationTimeProportion -
 deserializationTimeProportion - gettingResultTimeProportion, 0)
Bug: Use multiple lines to display data columns for show create table command
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jan/20 08:43
Affects Version/s: 3.0.0

Currently data columns are displayed in one line for show create table command, when the table has many columns (to make things even worse, columns may have long names or comments), the displayed result is really hard to read.

To improve readability, we could print each column in a separate line. Note that other systems like Hive/MySQL also display in this way.

Also, for data columns, table properties and options, we'd better put the right parenthesis to the end of the last column/property/option, instead of occupying a separate line.

As a result, before the change:
{noformat}
spark-sql> show create table test_table;
CREATE TABLE `test_table` (`col1` INT COMMENT 'This is comment for column 1', `col2` STRING COMMENT 'This is comment for column 2', `col3` DOUBLE COMMENT 'This is comment for column 3')
USING parquet
OPTIONS (
  `bar` '2',
  `foo` '1'
)
TBLPROPERTIES (
  'a' = 'x',
  'b' = 'y'
)
{noformat}
after the change:
{noformat}
spark-sql> show create table test_table;
CREATE TABLE `test_table` (
  `col1` INT COMMENT 'This is comment for column 1',
  `col2` STRING COMMENT 'This is comment for column 2',
  `col3` DOUBLE COMMENT 'This is comment for column 3')
USING parquet
OPTIONS (
  `bar` '2',
  `foo` '1')
TBLPROPERTIES (
  'a' = 'x',
  'b' = 'y')
{noformat}
Bug: Uneven spaces for some fields in EXPLAIN FORMATTED
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 06/Jan/20 13:06
Affects Version/s: 3.0.0

Output of EXPLAIN EXTENDED have uneven spaces. Eg,

{code:java}
(4) Project [codegen id : 1]
Output    : [key#x, val#x]
Input     : [key#x, val#x]
     
(5) HashAggregate [codegen id : 1]
Input: [key#x, val#x]
{code}

Unlike input field for HashAggregate, Output and Input fields of Project have more spaces.
Bug: Update SqlBase.g4 to create commentSpec pattern as same as locationSpec
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jan/20 08:23
Affects Version/s: 3.0.0

In `SqlBase.g4`, the comment clause is used as `COMMENT comment=STRING` and `COMMENT STRING` in many places.

While the location clause often appears along with the comment clause with a pattern defined as 

{code:sql}
locationSpec
    : LOCATION STRING
    ;
{code}

Then, we have to visit locationSpec as a List but comment as a single token
Bug: OneForOneStreamManager use AtomicLong
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jan/20 07:10
Affects Version/s: 3.0.0

Using compound operations as well as increments and decrements on primitive fields are not atomic operations. Here when volatile primitive field is incremented or decremented,  we run into data loss if threads interleave in steps of update. 

 

Refer: [https://wiki.sei.cmu.edu/confluence/display/java/VNA02-J.+Ensure+that+compound+operations+on+shared+variables+are+atomic]
Bug: Call requireNonStaticConf() only once
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 01/Jan/20 13:00
Affects Version/s: 2.4.4

The RuntimeConfig.requireNonStaticConf() method can be called 2 times for the same input:
1. Inside of set(<config>, true)
2. set() converts the second argument to a string and calls set(<config>, "true") where requireNonStaticConf() is invoked one more time
Bug: Delta Merge and Arbitrary Stateful Processing in Structured streaming  (foreachBatch)
Status: Closed | Priority: Trivial | Resolution: Invalid
Created: 24/Jan/20 13:30
Affects Version/s: 2.4.3

Hi ,

I have an application that makes Arbitrary Stateful Processing in Structured Streaming and used delta.merge to update delta table and faced strange behaviour:

1. I've noticed that logs inside implementation of {{MapGroupsWithStateFunction}}/ {{FlatMapGroupsWithStateFunction}} in my application outputted twice.

2. While finding a root cause I've also found that number State rows reported by Spark is also doubles.

 

I thought that may be there's a bug in my code, so I back to {{JavaStructuredSessionization}} from Apache Spark examples and changed it a bit. Still got same result.

The problem happens only if I do not perform datch.DF.persist inside foreachBatch.
{code:java}
StreamingQuery query = sessionUpdates
        .writeStream()
        .outputMode("update")
        .foreachBatch((VoidFunction2<Dataset<SessionUpdate>, Long>) (batchDf, v2) -> {
            // following doubles number of spark state rows and causes MapGroupsWithStateFunction to log twice withport persisting
            deltaTable.as("sessions").merge(batchDf.toDF().as("updates"), mergeExpr)
                    .whenNotMatched().insertAll()
                    .whenMatched()
                    .updateAll()
                    .execute();
        })
        .trigger(Trigger.ProcessingTime(10000))
        .queryName("ACME")
        .start(); 
{code}
According to [https://docs.databricks.com/_static/notebooks/merge-in-streaming.html] and [Apache spark docs|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch] there's seems to be no need to persist dataset/dataframe inside {{foreachBatch.}}

Sample code from Apache Spark examples with delta: [JavaStructuredSessionization with Delta merge|https://github.com/yurkao/delta-merge-sss/blob/master/src/main/java/JavaStructuredSessionization.java]

 

 

Appreciate your clarification.
Bug: Make build delete pyspark.zip file properly
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Jan/20 23:01
Affects Version/s: 2.0.2

The build uses Ant tasks to delete, then recreate, the {{pyspark.zip}} file within {{python/lib}}.  The only problem is the Ant task definition for the delete operation is incorrect (it uses {{dir}} instead of {{file}}), so it doesn't actually get deleted by this task.
Bug: Calculating size of table having large number of partitions causes flooding logs
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 02/Jan/20 15:32
Affects Version/s: 2.4.5

For a partitioned table, if the number of partitions are very large, e.g. tens of thousands or even larger, calculating its total size causes flooding logs.

The flooding happens in:
 # `calculateLocationSize` prints the starting and ending for calculating the location size, and it is called per partition;
 # `bulkListLeafFiles` prints all partition paths.
Bug: `ThriftServerWithSparkContextSuite` fails always on spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 07/Feb/20 23:34
Affects Version/s: 3.0.0

This is a release blocker for 3.0.0.

The same test profile Hadoop-2.7/Hive-2.3 fails on `branch-3.0` while it succeeds in `master`.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3/

The failure comes from `ThriftServerWithSparkContextSuite`.
{code}
info] org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite *** ABORTED *** (40 seconds, 707 milliseconds)
[info]   org.apache.hive.service.ServiceException: Failed to Start HiveServer2
{code}
Bug: Spark MLlib, GraphX 3.0 QA umbrella
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 22/Feb/20 04:03
Affects Version/s: 3.0.0

Description
 This JIRA lists tasks for the next Spark release's QA period for MLlib and GraphX. *SparkR is separate.

The list below gives an overview of what is involved, and the corresponding JIRA issues are linked below that.
h2. API
 * Check binary API compatibility for Scala/Java
 * Audit new public APIs (from the generated html doc)
 ** Scala
 ** Java compatibility
 ** Python coverage
 * Check Experimental, DeveloperApi tags

h2. Algorithms and performance
 * Performance tests

h2. Documentation and example code
 * For new algorithms, create JIRAs for updating the user guide sections & examples
 * Update Programming Guide
 * Update website
Bug: ML, GraphX 3.0 QA: API: New Scala APIs, docs
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Feb/20 03:49
Affects Version/s: 3.0.0

Audit new public Scala APIs added to MLlib & GraphX. Take note of:
 * Protected/public classes or methods. If access can be more private, then it should be.
 * Also look for non-sealed traits.
 * Documentation: Missing? Bad links or formatting?

*Make sure to check the object doc!*

As you find issues, please create JIRAs and link them to this issue. 

For *user guide issues* link the new JIRAs to the relevant user guide QA issue
Bug: ML, GraphX 3.0 QA: API: Binary incompatible changes
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Feb/20 03:48
Affects Version/s: 3.0.0

Generate a list of binary incompatible changes using MiMa and create new JIRAs for issues found. Filter out false positives as needed.

If you want to take this task, look at the analogous task from the previous release QA, and ping the Assignee for advice.
Bug: IntegralDivide's dataType should not depend on SQLConf.get
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 17/Feb/20 18:07
Affects Version/s: 3.0.0

{{IntegralDivide}}'s dataType depends on the value of {{SQLConf.get.integralDivideReturnLong}}. This is a problem because the configuration can change between different phases of planning, and this can silently break a query plan which can lead to crashes or data corruption.
Bug: Hive 2.3 profile should exclude hive-service-rpc
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 10/Feb/20 22:17
Affects Version/s: 3.0.0

hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark can pick up classes defined in hive instead of its thrift server module, which can cause hard to debug runtime errors due to class loading order and compilation errors for applications depend on spark.

 

If you compare hive-service-rpc 2.3.6's jar ([https://search.maven.org/remotecontent?filepath=org/apache/hive/hive-service-rpc/2.3.6/hive-service-rpc-2.3.6.jar]) and spark thrift server's jar (e.g. [https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hive-thriftserver_2.12/3.0.0-SNAPSHOT/spark-hive-thriftserver_2.12-3.0.0-20200207.021914-364.jar),] you will see that all of classes provided by hive-service-rpc-2.3.6.jar are covered by spark thrift server's jar. I am attaching the list of jar contents for your reference.
Bug: Use jekyll-redirect-from 0.15.0 instead of the latest
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Feb/20 07:48
Affects Version/s: 2.4.5

We use Ruby 2.3 in our release docker image.

The latest version of `jekyll-redirect-from 0.16.0` causes a failure on Ruby 2.3.
- https://github.com/jekyll/jekyll-redirect-from/releases/tag/v0.16.0

{code}
root@dc0bc546e377:/# gem install jekyll-redirect-from
ERROR:  Error installing jekyll-redirect-from:
	jekyll-redirect-from requires Ruby version >= 2.4.0.
{code}
Bug: Achieve LAST_ACCESS_TIME column update in TBLS table of hive metastore on hive table access through pyspark
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 27/Feb/20 06:45
Affects Version/s: 2.4.5

I have a requirement where i am looking to update LAST_ACCESS_TIME in TBLS of Hive metastore whenever any table is accessed through spark. I set this below property in hive-site.xml and hive honors it and updates the LAST_ACCESS_TIME everytime it is accessed. 

<property>

    <name>hive.exec.pre.hooks</name>

    <value>org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec</value>

 </property>

However, the same thing i want to achieve using pyspark/spark-shell but its not honoring this property of hive hooks. Is there an alternate approach of achieving this - 'Update of LAST_ACCESS_TIME in hive metastore on access using spark'. 

I passed the property like this - 

spark-sql -e 'set spark.hadoop.hive.exec.post.hooks=org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;select * from db.table;'

as well as i put the same property in /etc/spark/conf/hive-site.xml location.
Bug: Take productPrefix into account in MurmurHash3.productHash
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 17/Feb/20 04:25
Affects Version/s: 2.4.5

See the issue in https://github.com/scala/bug/issues/10495 and fix https://github.com/scala/scala/pull/7648.

Currently, different expressions with same children can produce the same hash as below:

{code}
scala> spark.range(1).selectExpr("id - 1").queryExecution.analyzed.semanticHash()
res0: Int = -565572825

scala> spark.range(1).selectExpr("id + 1").queryExecution.analyzed.semanticHash()
res1: Int = -565572825
{code}

The reason seems to be it doesn't take the product's class itself
Bug: ML, GraphX 3.0 QA: API: Experimental, DeveloperApi, final, sealed audit
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Feb/20 03:50
Affects Version/s: 3.0.0

We should make a pass through the items marked as Experimental or DeveloperApi and see if any are stable enough to be unmarked.

We should also check for items marked final or sealed to see if they are stable enough to be opened up as APIs.
Bug: Hive 2.3 profile should still use orc-nohive
Status: Closed | Priority: Critical | Resolution: Not A Bug
Created: 10/Feb/20 22:25
Affects Version/s: 3.0.0

Originally reported at [https://github.com/apache/spark/pull/26619#issuecomment-583802901]

 

Right now, Hive 2.3 profile pulls in regular orc, which depends on hive-storage-api. However, hive-storage-api and hive-common have the following common class files

 

org/apache/hadoop/hive/common/ValidReadTxnList.class
 org/apache/hadoop/hive/common/ValidTxnList.class
 org/apache/hadoop/hive/common/ValidTxnList$RangeResponse.class

For example, [https://github.com/apache/hive/blob/rel/storage-release-2.6.0/storage-api/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java] (pulled in by orc 1.5.8) and [https://github.com/apache/hive/blob/rel/release-2.3.6/common/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java] (from hive-common 2.3.6) both are in the classpath and they are different. Having both versions in the classpath can cause unexpected behavior due to classloading order. We should still use orc-nohive, which has hive-storage-api shaded.
Bug: Remove resource coordination support from Standalone
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 27/Feb/20 09:29
Affects Version/s: 3.0.0

Resource coordination is used for the case where multiple workers running on the same host. However, it should be a rarely or event impossible use case in current Standalone(which already allow multiple executor in a single worker). We should remove support for it to simply the implementation and reduce the potential maintain cost in the future.
Bug: Respect mapOutputSize in memory in adaptive execution
Status: Open | Priority: Major | Resolution: 
Created: 03/Feb/20 13:12
Affects Version/s: 3.1.0

Currently, Spark adaptive execution use the MapOutputStatistics information to adjust the plan dynamically, but this MapOutputSize does not respect the compression factor. So there are cases that the original SparkPlan is `SortMergeJoin`, but the Plan after adaptive adjustment was changed to `BroadcastHashJoin`, but this `BroadcastHashJoin` might causing OOMs due to inaccurate estimation.

 

Also, if the shuffle implementation is local shuffle(intel Spark-Adaptive execution impl), then in some cases, it will cause `Too large Frame` exception.

 

So I propose to respect the compression factor in adaptive execution, or use `dataSize` metrics in `ShuffleExchangeExec` in adaptive execution.
Bug: Add note for recommended versions of Pandas and PyArrow for 2.4.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/20 20:20
Affects Version/s: 2.4.5

CI testing for branch 2.4 has been with the versions below. These are recommened and any newer versions can not be guaranteed correct.

 

for 2.4, python 3.6.8:

-bash-4.1$ python -c "import pandas; import pyarrow; print('pandas: %s' % pandas.__version__); print('pyarrow: %s' % pyarrow.__version__)"
 pandas: 0.19.2
 pyarrow: 0.8.0

for master/3.0, python 3.6.8:

-bash-4.1$ python -c "import pandas; import pyarrow; print('pandas: %s' % pandas.__version__); print('pyarrow: %s' % pyarrow.__version__)"
 pandas: 0.24.2
 pyarrow: 0.15.1
Bug: insertInto() with existing column as partition key cause weird partition result
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 10/Feb/20 07:08
Affects Version/s: 2.4.4

{code:java}
val tableName = s"${config.service}_$saveDatabase.${config.table}_partitioned"
val writer = TableWriter.getWriter(tableDF.withColumn(config.dateColumn, typedLit[String](date.toString))) 

if (xsc.tableExistIn(config.service, saveDatabase, s"${config.table}_partitioned")) writer.insertInto(tableName)
else writer.partitionBy(config.dateColumn).saveAsTable(tableName){code}
This code checks whether table exists in desired path. (somewhere in S3 in this case) If table already exists in path then insert a new partition with insertInto() function.

If config.dateColumn not exists in table schema, no problem occurred. (just new column will be added) but if it is already exists in schema, Spark does not use given column as a partition key, instead it will create a hundred of partitions. Below is a part of Spark logs:
(Note that the name of partition column is date_ymd, which is already exists in source table. original value is a date string like '2020-01-01')


20/02/10 05:33:01 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=174 s3://\{my_path_at_s3}_partitioned_test/date_ymd=174
20/02/10 05:33:02 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=62 s3://\{my_path_at_s3}_partitioned_test/date_ymd=62
20/02/10 05:33:02 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=83 s3://\{my_path_at_s3}_partitioned_test/date_ymd=83
20/02/10 05:33:03 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=231 s3://\{my_path_at_s3}_partitioned_test/date_ymd=231
20/02/10 05:33:03 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=268 s3://\{my_path_at_s3}_partitioned_test/date_ymd=268
20/02/10 05:33:04 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=33 s3://\{my_path_at_s3}_partitioned_test/date_ymd=33
20/02/10 05:33:05 INFO S3NativeFileSystem2: rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=40 s3://\{my_path_at_s3}_partitioned_test/date_ymd=40
rename s3://\{my_path_at_s3}_partitioned_test/.spark-staging-e3c1c1fc-6bbe-4e77-8b7f-201cfd60d061/date_ymd=__HIVE_DEFAULT_PARTITION__ s3://\{my_path_at_s3}_partitioned_test/date_ymd=__HIVE_DEFAULT_PARTITION__

When I use different partition key which not in table schema such as 'stamp_date', everything goes fine. I'm not sure that it is a Spark bugs, I just wrote the report. (I think it is related with Hive...)



Thanks for reading!
Bug: Missing SortedMap type in pyspark
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 10/Feb/20 21:32
Affects Version/s: 2.4.5

Currently there is only MapType in pyspark API, which does not keep order of key-value map.

 

*SortedMapType* would fill this gap: a map with sorted key-value pairs (like TreeMap in Java).

 

For example:

*SortedMapType* would be very useful when user wants to persist row from DataFrame into Mongo-db.

row which consist a column with values of an ordered map type:

col : [1 -> 22, 2 -> 16, 3 -> 25]

will be persisted as following json:

{   "1": 22,   "2": 16,   "3": 25 }

 

instead of MapType which currently results in:

{   "2": 16,   "1": 22,   "3": 25 }
Bug: SparkR ML algorithms parity 
Status: Resolved | Priority: Major | Resolution: Done
Created: 13/Feb/20 19:14
Affects Version/s: 3.0.0

As of 3.0 the following algorithms are missing form SparkR

* {{LinearRegression}} 
* {{FMRegressor}} (Added to ML in 3.0)
* {{FMClassifier}} (Added to ML in 3.0)