av9ash/GitBugs huggingface dataset:
Bug: Fix perf regression of date/timestamp formatting in toHiveString
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/20 19:48
Affects Version/s: 3.0.0

HiveResult.toHiveString has to convert incoming Java date/timestamps types to days/microseconds because existing API of DateFormatter/TimestampFormatter don't accept java.sql.Timestamp/java.util.Date and java.time.Instant/java.time.LocalDate. Internally, the formatters perform conversions to Java types again. This badly impacts on the performance. The ticket aims to add new APIs to DateFormatter and TimestampFormatter that should accept Java types.
Bug: Spark History Server UI does not come up when hosted on a custom path
Status: Open | Priority: Major | Resolution: 
Created: 26/May/20 09:57
Affects Version/s: 2.4.5

I tried to bringup spark-history server using the start-history-server.sh script. The UI works perfectly fine when there is no path specified.

i.e. http://<IP Address of the Machine>:18080

But If I bringup history server using custom path, I do not see the UI working properly.

Following is my configuration

spark.history.fs.logDirectory=<Spark Log Directory Location>
spark.ui.proxyBase=/test

When I hit the url  http://<IP Address of the Machine>:18080/test, I do not see the History Server UI working properly. Attaching the screenshot of the faulty UI.

Wanted to know if I'm missing any configuration

 

!image-2020-05-26-15-26-21-616.png!
Bug: Failure on pushing down filters with java.time.Instant values in ORC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/20 12:23
Affects Version/s: 3.0.0

When spark.sql.datetime.java8API.enabled is set to true, filters pushed down with java.time.Instant values to ORC datasource fails with the exception:
{code}
[info]   java.lang.IllegalArgumentException: Wrong value class java.time.Instant for TIMESTAMP.EQUALS leaf
[info]   at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.checkLiteralType(SearchArgumentImpl.java:192)
[info]   at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.<init>(SearchArgumentImpl.java:75)
[info]   at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.equals(SearchArgumentImpl.java:352)
[info]   at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:234)
[info]   at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.convertibleFiltersHelper$1(OrcFilters.scala:134)
[info]   at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.$anonfun$convertibleFilters$4(OrcFilters.scala:137)
[info]   at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
{code}
Bug: Pushdown IsNotNull to file scan if possible
Status: Open | Priority: Major | Resolution: 
Created: 25/May/20 08:02
Affects Version/s: 3.1.0

We should Pushdown {{IsNotNull}} to file scan if possible. For example:
{code:sql}
CREATE TABLE t1(c1 string, c2 string) USING parquet;
EXPLAIN SELECT t1.* FROM t1 WHERE coalesce(t1.c1, t1.c2) IS NOT NULL;
{code}

{noformat}
== Physical Plan ==
*(1) Filter isnotnull(coalesce(c1#43, c2#44))
+- *(1) ColumnarToRow
   +- FileScan parquet default.t1[c1#43,c2#44] Batched: true, DataFilters: [isnotnull(coalesce(c1#43, c2#44))], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop2.7/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:string,c2:string>
{noformat}

{code:sql}
EXPLAIN SELECT t1.* FROM t1 WHERE t1.c1 IS NOT NULL OR t1.c2 IS NOT NULL;
{code}
{noformat}
== Physical Plan ==
*(1) Filter (isnotnull(c1#43) OR isnotnull(c2#44))
+- *(1) ColumnarToRow
   +- FileScan parquet default.t1[c1#43,c2#44] Batched: true, DataFilters: [(isnotnull(c1#43) OR isnotnull(c2#44))], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop2.7/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [Or(IsNotNull(c1),IsNotNull(c2))], ReadSchema: struct<c1:string,c2:string>
{noformat}


Real performance test case:
 !default.png!  !pushdown.png!
Bug: Support for multiple null values when reading CSV data
Status: Open | Priority: Major | Resolution: 
Created: 18/May/20 05:07
Affects Version/s: 2.3.2

There are CSV files out there with multiple values that are supposed to be interpreted as null. Currently the `nullValue` option of spark.read is accepting only one value.

Can we come with something like below:
{code:java}
spark.read.option("nullValue1", "abc").option("nullValue2", "def").csv("1.csv")
{code}
Or
{code:java}
spark.read.option("nullValue", Seq("abc","def")).csv("1.csv") {code}
Where the values "abc" and "def" are to be interpreted as NULLs.
Bug: Add support of MDC in spark driver logs.
Status: In Progress | Priority: Major | Resolution: 
Created: 20/May/20 08:56
Affects Version/s: 3.0.0

In order to align driver logs with spark executors logs, add support to log MDC state for properties that are shared between the driver the server executors (e.g. application name. task id) 
 The use case is applicable in 2 cases:
 # streaming
 # running spark under job server (in this case you have a long-running spark context that handles many tasks from different sources).

See also [SPARK-8981|https://issues.apache.org/jira/browse/SPARK-8981] that handles the MDC in executor context.
Bug: Spark Structured Streaming with "assign" KafkaConsumer mode still uses group.id
Status: In Progress | Priority: Major | Resolution: 
Created: 23/May/20 21:24
Affects Version/s: 2.4.5

[Spark Structured Streaming - Kafka integration|https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries] provides the [assign|https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assign-java.util.Collection-] strategy to consume data from Kafka. This strategy assumes manual assignment of offsets in topic partitions. According to KafkaConsumer specification, the consumer group in the "assign" strategy is not used.

When creating a consumer to read data, Spark provides an internally-generated group id like this:

[KafkaRelation|https://github.com/apache/spark/blob/bcadd5c3096109878fe26fb0d57a9b7d6fdaa257/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala#L60]:
{code:scala}
val uniqueGroupId = s"spark-kafka-relation-${UUID.randomUUID}"
{code}

This is done for any consumer strategy, even for "assign". The problem is that with a secured Kafka cluster a client cannot use an arbitrary consumer group id. That's why a Structured Streaming application fails with an exception like:

{code:none}
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-kafka-relation-ecab045d-4ee6-425e-88a0-495d4100a013-driver-0
{code}

With Spark 2.4.5, the only way is to reconfigure the broker - add the needed entries in ACL (for example, [this discussion|https://stackoverflow.com/a/59300360] on StackOverflow).
 
With Spark 3.0.0, this problem could be avoided by two workarounds:
- SPARK-26121: Specify a custom prefix for the consumer group id generated by Spark (allowed by the broker)
- SPARK-26350: Specify a custom group id

However, with the "assign" strategy the user does not need to worry about consumer group - the consumer group should be disregarded. Therefore a better fix could be to not set the consumer property "group.id" when using the "assign" strategy.
Bug: Fix test BarrierTaskContextSuite."share messages with allGather() call"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/20 08:07
Affects Version/s: 3.0.0

{code:java}
test("share messages with allGather() call") {  
  val conf = new SparkConf()
      .setMaster("local-cluster[4, 1, 1024]")
      .setAppName("test-cluster")
  sc = new SparkContext(conf)
  val rdd = sc.makeRDD(1 to 10, 4)
  val rdd2 = rdd.barrier().mapPartitions { it =>  
    val context = BarrierTaskContext.get()      
    // Sleep for a random time before global sync.
    Thread.sleep(Random.nextInt(1000))  
    // Pass partitionId message in
    val message: String = context.partitionId().toString
    val messages: Array[String] = context.allGather(message)
    messages.toList.iterator 
   } 
   // Take a sorted list of all the partitionId messages
   val messages = rdd2.collect().head
   // All the task partitionIds are shared
   for((x, i) <- messages.view.zipWithIndex) assert(x.toString == i.toString)
 }
{code}
In this test, the desired `messages`(a.k.a rdd2.collect().head) should be ["0", "1", "2", "3"], but is "0" in reality.
Bug: Support tensor type (TensorType) in Spark SQL/DataFrame
Status: Open | Priority: Major | Resolution: 
Created: 20/May/20 17:43
Affects Version/s: 3.1.0

More and more DS/ML workloads are dealing with tensors. For example, a decoded color image can be represented by a 3D tensor. It would be nice to natively support tensor type. A local tensor is essentially an array with shape info, stored together. Native support is better than user-defined type (UDT) because PyArrow does not support UDTs but it already supports tensors.
Bug: Overwrite partition failed with 'WRONG FS' when the target partition is not belong to the filesystem as same as the table 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/20 06:15
Affects Version/s: 2.1.3

With https://issues.apache.org/jira/browse/SPARK-18107, we will disable the underlying replace(overwrite) and instead do delete in spark side and only do copy in hive side to bypass the performance issue - https://issues.apache.org/jira/browse/HIVE-11940
 
Conditionally, if the table location and partition location do not belong to the same [[FileSystem]], We should not disable hive overwrite. Otherwise, hive will use the [[FileSystem]] instance belong to the table location to copy files, which will fail [[FileSystem#checkPath]]
see https://github.com/apache/hive/blob/rel/release-2.3.7/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1648-L1659
Bug: Assert optimized plan is initialized before tracking the execution of planning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 12:10
Affects Version/s: 3.0.0

The {{QueryPlanningTracker}} in {{QueryExeuction}} reports the planning time that also includes the optimization time. This happens because the {{optimizedPlan}} in {{QueryExecution}} is lazy and only will initialize when first called. When {{df.queryExecution.executedPlan}} is called, the the tracker starts recording the planning time, and then calls the optimized plan. This causes the planning time to start before optimization and also include the planning time.

This PR fixes this behavior by introducing a method {{assertOptimized}}, similar to {{assertAnalyzed}} that explicitly initializes the optimized plan. This method is called before measuring the time for {{sparkPlan}} and {{executedPlan}}. We call it before {{sparkPlan}} because that also counts as planning time.
Bug: Improve handling the case where different barrier sync types in a single sync
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/20 09:34
Affects Version/s: 3.0.0

Currently, we use cleanupBarrierStage when detecting different barrier sync types in a single sync. This cause a problem that a new `ContextBarrierState` can be created again if there's following requesters on the way. And those corresponding tasks will fail because of killing instead of different barrier sync types detected.
Bug: Inconsistent error messages of casting timestamp to int
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 20:47
Affects Version/s: 3.0.0

Casting 1900-05-05 18:34:56.1 to int produces different error messages for in the non-codegen and codegen mode:
# codegen: "Casting -2198208303900000 to int causes overflow"
# non-codegen: "Casting -2198208304 to int causes overflow"

The error message in the non-codegen contains intermediate results.
Bug: SparkSQL can't recognize the modified length of Hive varchar
Status: In Progress | Priority: Major | Resolution: 
Created: 17/May/20 14:01
Affects Version/s: 3.0.0

{code:java}
//Create a hive table in sparksql
spark-sql> create table test_table(id varchar(5)) stored as textfile;

// Create a file under linux, the content is 1234567890

// Then load this file into this hive table
spark-sql> load data local inpath '/home/test_table' overwrite into table test_table;

// Query this table in sparksql
spark-sql> select * from test_table;
12345

// Modify the length of this field in hive
hive> alter table test_table change column id id varchar(10);

// Query this table in hive
hive> select * from test_table;
OK
1234567890

// Query this table in sparksql again
spark-sql> select * from test_table;
// The obtained id is still 12345, so the length modification does not take effect in SparkSQL
12345{code}
Bug: QuantileDiscretizer raise error parameter splits given invalid value (splits array includes -0.0 and 0.0)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/20 09:22
Affects Version/s: 2.4.5

Reproduce code

{code}

import scala.util.Random
val rng = new Random(3)

val a1 = Array.tabulate(200)(_=>rng.nextDouble * 2.0 - 1.0) ++ Array.fill(20)(0.0) ++ Array.fill(20)(-0.0)

import spark.implicits._
val df1 = sc.parallelize(a1, 2).toDF("id")

import org.apache.spark.ml.feature.QuantileDiscretizer
val qd = new QuantileDiscretizer().setInputCol("id").setOutputCol("out").setNumBuckets(200).setRelativeError(0.0)

val model = qd.fit(df1)

{code}

Raise error like:

  at org.apache.spark.ml.param.Param.validate(params.scala:76)
  at org.apache.spark.ml.param.ParamPair.<init>(params.scala:634)
  at org.apache.spark.ml.param.Param.$minus$greater(params.scala:85)
  at org.apache.spark.ml.param.Params.set(params.scala:713)
  at org.apache.spark.ml.param.Params.set$(params.scala:712)
  at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:41)
  at org.apache.spark.ml.feature.Bucketizer.setSplits(Bucketizer.scala:77)
  at org.apache.spark.ml.feature.QuantileDiscretizer.fit(QuantileDiscretizer.scala:231)
  ... 49 elided
java.lang.IllegalArgumentException: quantileDiscretizer_479bb5a3ca99 parameter splits given invalid value [-Infinity,-0.9986765732730827,..., -0.0, 0.0, ..., 0.9907184077958491,Infinity]

0.0 > -0.0 is False, which break the paremater validation check.
Bug: Check casting timestamps to byte/short/int/long before 1970-01-01
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/20 19:59
Affects Version/s: 3.0.0

There are tests for casting timestamps to byte/short/int/long after the epoch 1970-01-01 00:00:00Z. Actually, the tests checks casting positive values but there are no test for "negative" timestamps before the epoch. The ticket aims to add such tests.
Bug: TaskMemoryManager allocate failed
Status: In Progress | Priority: Major | Resolution: 
Created: 15/May/20 11:06
Affects Version/s: 3.0.0

{code:java}
Stage Id = 291, Task Id = 289765 , Error Message = 
org.apache.spark.util.TaskCompletionListenerException: null

Previous exception in task: Unable to acquire 65536 bytes of memory, got 0
    org.apache.spark.memory.MemoryConsumer.throwOom(MemoryConsumer.java:157)
    org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:98)
    org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.reset(UnsafeInMemorySorter.java:186)
    org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:229)
    org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:206)
    org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:285)
    org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:96)
    org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.growPointerArrayIfNecessary(UnsafeExternalSorter.java:348)
    org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:403)
    org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:135)
    org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.sort_addToSorter_0$(Unknown Source)
    org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
	org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:621)
	org.apache.spark.sql.execution.window.WindowExec$$anonfun$11$$anon$1.fetchNextRow(WindowExec.scala:320)
    org.apache.spark.sql.execution.window.WindowExec$$anonfun$11$$anon$1.<init>(WindowExec.scala:329)
	org
{code}
Bug:  Turn on spark.sql.legacy.createHiveTableByDefault.enabled by default
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 12/May/20 02:20
Affects Version/s: 3.0.0

According to the latest status of [[DISCUSS] Resolve ambiguous parser rule between two "create table"s|http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Resolve-ambiguous-parser-rule-between-two-quot-create-table-quot-s-td29051i20.html#a29355], there're might be a choice to turn this conf on by default to unblock Spark 3.0 release.
Bug: Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 03:44
Affects Version/s: 3.0.0


{code:java}
Caused by: sbt.ForkMain$ForkError: org.apache.derby.iapi.error.StandardException: Another instance of Derby may have already booted the database /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/metastore_db.
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
	at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
	at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
	at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
	at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
	... 138 more
{code}
Bug: Make test-dependencies.sh detect version string correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/20 21:44
Affects Version/s: 2.4.6

Currently, SBT jobs are broken like the following.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/476/console
{code}
[error] running /home/jenkins/workspace/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/dev/test-dependencies.sh ; received return code 1
Build step 'Execute shell' marked build as failure
{code}
Bug: NPE on big dataset plans
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 13/May/20 07:55
Affects Version/s: 2.4.4

We have big dataset containing 275 SQL operations more than 275 joins.

On the terminal operation to write data, it fails with NullPointerException.

 

I understand that such big number of operations might not be what spark is designed for, but NullPointerException is not an ideal way to fail in this case.

 

For more details, please see the stacktrace.
Bug: Support Java 8 datetime types by Random data generator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/20 20:21
Affects Version/s: 3.1.0

Currently, RandomDataGenerator.forType can generate:
* java.sql.Date values for DateType
* java.sql.Timestamp values for TimestampType
The ticket aims to support java.time.Instant for TimestampType and java.time.LocalDate for DateType.
Bug: Test parquet dictionary encoding of random dates/timestamps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/20 18:43
Affects Version/s: 3.1.0

Currently, dictionary encoding is not tested in ParquetHadoopFsRelationSuite test "test all data types" because dates and timestamps are uniformly distributed, and dictionary encoding is not applied for the types in fact.