av9ash/GitBugs huggingface dataset:
Bug: ORC writer doesn't support colon in column names
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Aug/20 15:10
Affects Version/s: 2.3.0

Hi,

I'm getting an {{IllegalArgumentException: Can't parse category at 'struct<a:b^:int>'}} when exporting to ORC a dataframe whose column names contain colon ({{:}}). Reproducible as hereunder. Same problem also occurs if the name with colon appears nested as member of a struct.

Seems related with SPARK-21791(which was solved in 2.3.0).

In my real-life case, the column was actually {{xsi:type}}, coming from some parsed xml. Thus other users may be affected too.

Has it been fixed after Spark 2.3.0? (sorry, can't test easily)

Any workaround? Would be acceptable for me to find and replace all colons with underscore in column names, but not easy to do in a big set of nested struct columns...

Thanks

 

 
{code:java}
 spark.conf.set("spark.sql.orc.impl", "native")

 val dfColon = Seq(1).toDF("a:b")
 dfColon.printSchema()
 dfColon.show()
 dfColon.write.orc("test_colon")
 // Fails with IllegalArgumentException: Can't parse category at 'struct<a:b^:int>'
 
 import org.apache.spark.sql.functions.struct
 val dfColonStruct = dfColon.withColumn("x", struct($"a:b")).drop("a:b")
 dfColonStruct.printSchema()
 dfColonStruct.show()
 dfColon.write.orc("test_colon_struct")
 // Fails with IllegalArgumentException: Can't parse category at 'struct<x:struct<a:b^:int>>'
{code}
Bug: Querying ORC table in Spark3 using spark.sql.orc.impl=hive produces incorrect when timestamp is present in predicate
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 13/Aug/20 18:57
Affects Version/s: 3.0.0

*How to reproduce this behavior?*
 * TZ="America/Los_Angeles" ./bin/spark-shell
 * sql("set spark.sql.hive.convertMetastoreOrc=true")
 * sql("set spark.sql.orc.impl=hive")
 * sql("create table t_spark(col timestamp) stored as orc;")
 * sql("insert into t_spark values (cast('2100-01-01 01:33:33.123America/Los_Angeles' as timestamp));")
 * sql("select col, date_format(col, 'DD') from t_spark where col = cast('2100-01-01 01:33:33.123America/Los_Angeles' as timestamp);").show(false)
 *This will return empty results, which is incorrect.*
 * sql("set spark.sql.orc.impl=native")
 * sql("select col, date_format(col, 'DD') from t_spark where col = cast('2100-01-01 01:33:33.123America/Los_Angeles' as timestamp);").show(false)
 *This will return 1 row, which is the expected output.*

 

The above query using (True, hive) returns *correct results if pushdown filters are turned off*. 
 * sql("set spark.sql.orc.filterPushdown=false")
 * sql("select col, date_format(col, 'DD') from t_spark where col = cast('2100-01-01 01:33:33.123America/Los_Angeles' as timestamp);").show(false)
 *This will return 1 row, which is the expected output.*
Bug: Handle Null error message in hive ThriftServer UI
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Aug/20 01:52
Affects Version/s: 3.0.0

This fix is trying to prevent NullPointerException caused by Null error message by wrapping the message with Option and using getOrElse to handle Null value.

The possible reason why the error message would be null is that java Throwable allows the detailMessage to be null. However, in the render code we assume that the error message would be be null and try to do indexOf() on the string. Therefore, if some exception doesn't set the error message, it would trigger NullPointerException in the hive ThriftServer UI.
Bug: Reduce user confusion and subtle bugs by optionally preventing date & timestamp comparison
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/20 21:12
Affects Version/s: 3.0.0

https://issues.apache.org/jira/browse/SPARK-23549 made Spark's handling of date vs. timestamp comparison consistent with SQL, which, unfortunately, isn't consistent with common sense.

When dates are compared with timestamps, they are promoted to timestamps at midnight of the date, in the server timezone, which is almost always UTC. This only works well if all timestamps in the data are logically time instants as opposed to dates + times, which only become instants with a known timezone.

The fundamental issue is that dates are a human time concept and instant are a machine time concept. While we can technically promote one to the other, logically, it only works 100% if midnight for all dates in the system is in the server timezone. 

Every major modern platform offers a clear distinction between machine time (instants) and human time (an instant with a timezone, UTC offset, etc.), because we have learned the hard way that date & time handling is a never-ending source of confusion and bugs. SQL, being an ancient language (40+ years old), is well behind software engineering best practices; using it as a guiding light is necessary for Spark to win market share, but unfortunate in every other way.

For example, Java has:
 * java.time.LocalDate
 * java.time.Instant
 * java.time.ZonedDateTime
 * java.time.OffsetDateTime

I am not suggesting we add new data types to Spark. I am suggesting we go to the heart of the matter, which is that most date vs. time handling issues are the result of confusion or carelessness.

What about introducing a new setting that makes comparisons between dates and timestamps illegal, preferably with a helpful exception message?

If it existed, I would certainly make it the default for all our clusters. The minor coding convenience that comes from being able to compare dates & timestamps with an automatic type promotion pales in comparison with the risk of subtle bugs that remain undetected for a long time.
Bug: WithFields Expression should not be foldable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/20 14:58
Affects Version/s: 3.1.0

The following query currently fails on master brach: 
{code:scala}
sql("SELECT named_struct('a', 1, 'b', 2) a")
.select($"a".withField("c", lit(3)).as("a"))
.show(false)

// java.lang.UnsupportedOperationException: Cannot evaluate expression: with_fields(named_struct(a, 1, b, 2), c, 3)
{code}
This happens because the Catalyst optimizer tries to statically evaluate the {{WithFields Expression}} (via the {{ConstantFolding}} rule), however it cannot do so because {{WithFields Expression}} is {{Unevaluable}}. 

The likely solution here is to change {{WithFields Expression}} so that it is not {{foldable}} under any circumstance. This should solve the issue because then Catalyst will no longer attempt to statically evaluate {{WithFields Expression}} anymore.
Bug: Fix release script to uri encode the user provided passwords.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/20 05:33
Affects Version/s: 2.4.6

As I was trying to do the release using the docker
{code:java}
 dev/create-release/do-release-docker.sh{code}
script, there were some failures.

 
 # If the release manager password contains a char, that is not allowed in URL, then it fails the build at the clone spark step.
 # If the .gitignore file is missing, it fails the build at rm .gitignore step.
Bug: Data with date type are saved into hive table with wrong value '1970-01-01'
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Aug/20 15:58
Affects Version/s: 3.0.0

 

How to reproduce:
{code:java}
scala> spark.sql("create table t1(d date)")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("insert into table t1 values(cast('2020-08-09' as date))")
res3: org.apache.spark.sql.DataFrame = []
scala> spark.sql("select d from t1").show
+----------+
|         d|
+----------+
|1970-01-01|
+----------+  
{code}
 

Spark 3.0 introduced DaysWritable which extends DateWrite from hive to handle date type. DaysWritable.toString() is called to write its value into hive table. DateWrite.toString() is defined as:

 
{code:java}
@Override
public String toString() {
 // For toString, the time does not matter
 return get(false).toString();
}

public Date get(boolean doesTimeMatter) {
  return new Date(daysToMillis(daysSinceEpoch, doesTimeMatter));
}
{code}
 

DaysWritable didn't override toString(), neither get(boolean doesTimeMatter)。It did override get():
{code:java}
override def get(): Date = new Date(DateWritable.daysToMillis(julianDays))
{code}
but this didn't help with toString(), so with daysSinceEpoch in DateWritable always as 0, calls to DaysWritable.toString() will always return '1970-01-01', and as a result date value stored into hive table will always have value '1970-01-01'。
Bug: yarnClient.killApplication(appId) is never called
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/20 23:04
Affects Version/s: 2.4.0

*Problem Statement:* 

When an application is submitted using spark-submit in cluster mode using yarn, the spark application continues to run on the cluster, even if spark-submit itself has been requested to shutdown (Ctrl-C/SIGTERM/etc.)

While there is code inside org.apache.spark.deploy.yarn.Client.scala that would lead you to believe the spark application on the cluster will shut down, this code is not currently reachable.

Example of behavior:

spark-submit ...

<Ctrl-C> or kill -15 <pid>

spark-submit itself dies

job can still be found running on the cluster

 

*Expectation:*

When spark-submit is in monitoring a yarn app and spark-submit itself is requested to shutdown (SIGTERM, HUP,etc.), it should call yarnClient.killApplication(appId) so that the actual spark application running on the cluster is killed.

 

 

*Proposal*

There is already a shutdown hook registered which cleans up temp files.  Could this be extended to call yarnClient.killApplication? 

I believe the default behavior should be to request yarn to kill the application, however I can imagine use cases where you may still want it to run.  So facilitate these use cases, an option should be provided to skip this hook.
Bug: kafka consumer cache capacity is unclear
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 07/Aug/20 11:04
Affects Version/s: 3.0.0

The [docs|https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html] mention
{noformat}
The cache for consumers has a default maximum size of 64.  If you expect
 to be handling more than (64 * number of executors) Kafka partitions, 
you can change this setting via spark.streaming.kafka.consumer.cache.maxCapacity{noformat}
However, for structured streaming, the code seems to expect
{code:java}
spark.kafka.consumer.cache.capacity/spark.sql.kafkaConsumerCache.capacity{code}
Would be nice to clear the ambiguity in the documentation or even merge these configurations in the code
Bug: BlockManagerDecommissionIntegrationSuite is still flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/20 01:05
Affects Version/s: 3.1.0

As reported by [~gurwls223], BlockManagerDecommissionIntegrationSuite test is apparently still flaky (even after https://github.com/apache/spark/pull/29226): https://github.com/apache/spark/pull/29226#issuecomment-670286829.

The new flakyness is because the executors are not launching in the 6 seconds time out I had given them when run under github checks.
Bug: Race condition in FsHistoryProvider listing iteration
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Aug/20 12:02
Affects Version/s: 3.0.0

It seems to be a race condition in FSHistoryProvider that iterate the listing with high concurrency:
{code:java}
val stale = listing.view(classOf[LogInfo])
 .index("lastProcessed")
 .last(newLastScanTime - 1)
 .asScala
 .toList{code}

`toList` will iterate items in the listing, if one of listing entries was deleted by some clean thread, checkForLogs will throw out a `java.util.NoSuchElementException` and abort current execution.
Bug: Parquet encryption policy interface
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/20 10:53
Affects Version/s: 3.1.0

Provide an interface for Parquet column encryption policy engine clients.

*PARQUET ENCRYPTION AT parquet-mr-1.12*
- https://github.com/apache/parquet-format/blob/encryption/Encryption.md
Bug: Using pyspark with a MultiLayerPerceptron model given inconsistent outputs if a large amount of data is fed into it and at least one of the model outputs is fed to a Python UDF.
Status: Open | Priority: Major | Resolution: 
Created: 03/Aug/20 16:34
Affects Version/s: 2.4.3

Using pyspark with a MultiLayerPerceptron model given inconsistent outputs if a large amount of data is fed into it and at least one of the model outputs is fed to a Python UDF.

This data correctness issue impacts both the Spark 2.4 releases and the latest Master branch.

I do not understand the root cause and cannot recreate 100% of the time. But I have a simplified code sample (attached) that triggers the bug regularly. I raised an inquiry on the mailing list as a Spark 2.4 issue but nobody had a suggested root cause and I have since recreated the problem on master so I am now raising a bug here.

During debugging I have narrowed the problem down somewhat and some observations I have made while doing this are:
 * I can recreate the problem with a very simple MultilayerPerceptron with no hidden layers (just 14 features and 2 outputs), I also see it with a more complex MultilayerPerceptron model so I don't think the model details are important.
 * I cannot recreate the problem unless the model output is fed to a python UDF, removing this leads to good outputs for the model and having it means that model outputs are inconsistent (note that not just the Python UDF outputs are inconsistent)
 * I cannot recreate the problem on minuscule amounts of data or when my data is partitioned heavily. 100,000 rows of input with 2 partitions sees the issue happen most of the time.
 * Some of the bad outputs I get could be explained if certain features were zero when they came into the model (when they are not in my actual feature data)
 * I can recreate the problem on several different servers

My environment is CentOS 7.6 with Python 3.6.3 and Spark 2.4.3, I can also recreate the issue from the code on the Spark master branch but strangely I cannot recreate the issue with Spark 2.4.3 and Python 2.7. I'm not sure why the version of python would matter.

The attached code sample triggers the problem for me the vast majority of the time when pasted into a pyspark shell. This code generates a dataframe containing 100,000 identical rows, transforms it with a MultiLayerPerceptron model and feeds one of the model output columns to a simple Python UDF to generate an additional column. The resulting dataframe has the distinct rows selected and since all the inputs are identical I would expect to get 1 row back, instead I get many unique rows with the number returned varying each time I run the code. To run the code you will need the model files locally. I have attached the model as a zip archive and unzipping this to /tmp should be all you need to do to get the code to run.
Bug: Allow DataSourceReadBenchmark to run for select formats
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Aug/20 19:42
Affects Version/s: 3.0.0

Currently DataSourceReadBenchmark runs benchmarks for Parquet, ORC, CSV, and Json file formats and there is no way to specify at runtime a single format or a subset of formats, like there is for BuiltInDataSourceWriteBenchmark.
Bug: SHOW VIEWS fails with MetaException ... ClassNotFoundException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 16:16
Affects Version/s: 3.1.0

SHOW VIEWS can fail with the error:
{code}
java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class com.ibm.spss.hive.serde2.xml.XmlSerDe not found)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:290)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:486)
at org.apache.spark.sql.hive.client.HiveClientImpl.convertHiveTableToCatalogTable(HiveClientImpl.scala:485)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$2(HiveClientImpl.scala:472)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:472)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:349)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$retryLocked$1(HiveClientImpl.scala:252)
at org.apache.spark.sql.hive.client.HiveClientImpl.synchronizeOnObject(HiveClientImpl.scala:288)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:244)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:331)
at org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:472)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$listTablesByType$1(HiveClientImpl.scala:873)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:349)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$retryLocked$1(HiveClientImpl.scala:252)
at org.apache.spark.sql.hive.client.HiveClientImpl.synchronizeOnObject(HiveClientImpl.scala:288)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:244)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:331)
at org.apache.spark.sql.hive.client.HiveClientImpl.listTablesByType(HiveClientImpl.scala:866)
at org.apache.spark.sql.hive.client.PoolingHiveClient.$anonfun$listTablesByType$1(PoolingHiveClient.scala:266)
at org.apache.spark.sql.hive.client.PoolingHiveClient.withHiveClient(PoolingHiveClient.scala:112)
at org.apache.spark.sql.hive.client.PoolingHiveClient.listTablesByType(PoolingHiveClient.scala:266)
at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$listViews$1(HiveExternalCatalog.scala:940)
{code}
Bug: Disallow FileSystem.get(Configuration conf) by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 07:50
Affects Version/s: 3.1.0

The method `FileSystem.get(Configuration conf)` will return a default FileSystem instance if the conf is not set properly. This can cause regression if the target file system is not the default implementation. 
If we disallow it in Scala style check by default and suggest developers use FileSystem.get(URI uri, Configuration conf) or Path.getFileSystem(), we can reduce potential regression and PR review effort.
Bug: test of org.apache.spark.sql.kafka010.KafkaSourceStressSuite failed for aarch64
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/20 07:53
Affects Version/s: 3.0.0

The aarch64 maven job failed after the commit https://github.com/apache/spark/commit/813532d10310027fee9e12680792cee2e1c2b7c7   merged, see the log https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/353/testReport/junit/org.apache.spark.sql.kafka010/KafkaSourceStressSuite/stress_test_with_multiple_topics_and_partitions/

I took test in my aarch64 instance, if I reset the commit 813532d10310027fee9e12680792cee2e1c2b7c7 the test is ok.
Bug: Add a hint-specific suite for CTE for test coverage
Status: In Progress | Priority: Major | Resolution: 
Created: 05/Aug/20 07:23
Affects Version/s: 3.1.0

This ticket is to address the below comments to help us understand the test coverage of SQL HINT for CTE.
https://github.com/apache/spark/pull/29062#discussion_r463247491
https://github.com/apache/spark/pull/29062#discussion_r463248167
Bug: MSCK REPAIR TABLE should drop the missing partitions from metastore
Status: In Progress | Priority: Major | Resolution: 
Created: 05/Aug/20 08:45
Affects Version/s: 3.0.0

{{msck repair table <tablename>}} alter table <tablename> recover partitions is often used in environments where the new partitions are loaded as directories on HDFS and users want to create the missing partitions in bulk. However, currently it only supports addition of missing partitions. If there are any partitions which are present in metastore but not on the FileSystem, it should also delete them so that it truly repairs the table metadata.

In Hive, it already support the grammar `msck repair table drop partitions`.

 

https://issues.apache.org/jira/browse/HIVE-17824
Bug: Query with broadcast hints fail when query has a WITH clause
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 05/Aug/20 00:37
Affects Version/s: 3.0.0

If a query has a WITH clause and a query hint (like `BROADCAST`), the query fails

In the below code sample, executing `sql2` fails, but `sql1` passes.
{code:java}
import spark.implicits._
val df = List(
  ("1", "B", "C"),
  ("A", "2", "C"),
  ("A", "B", "3")
).toDF("COL_A", "COL_B", "COL_C")
df.createOrReplaceTempView("table1")

val df1 = List(
  ("A", "2", "3"),
  ("1", "B", "3"),
  ("1", "2", "C")
).toDF("COL_A", "COL_B", "COL_C")
df1.createOrReplaceTempView("table2")

val sql1 = "select /*+ BROADCAST(a) */ a.COL_A from table1 a inner join table2 b on a.COL_A = b.COL_A"
val sql2 = "with X as (select /*+ BROADCAST(a) */ a.COL_A from table1 a inner join table2 b on a.COL_A = b.COL_A) select X.COL_A from X"
val df2 = spark.sql(sql2)

println(s"Row Count ${df2.count()}")
println("Rows... ")
df2.show(false)
{code}
 

I tried executing this sample program with spark2.4.0, and both sql statements work
Bug: Add Hive Murmur3Hash expression
Status: Open | Priority: Minor | Resolution: 
Created: 27/Aug/20 06:07
Affects Version/s: 3.1.0

To allow Spark to write Hive 3 compatible bucketed table, we need to follow the same hash function with Hive/Presto. Hive murmur3hash is quite some different with Spark murmur3hash (with different default seed, different logic for NULL, array, map, struct, detail in [https://github.com/apache/hive/blob/ece58fff1b53ea451bfc524c4c15f63ee12eca00/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L813]).  So here introduce a Hive murmur3hash expression.
Bug: request executor cores with decimal when spark on k8s
Status: Open | Priority: Minor | Resolution: 
Created: 31/Aug/20 02:09
Affects Version/s: 3.0.0

  In current spark version which does not support to request executor cores with decimal when spark on k8s . because the cores is Int type in CoarseGrainedExecutorBackend class.

 !screenshot-1.png!
Bug: Fill up missing since version for math expressions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/20 14:54
Affects Version/s: 3.1.0

the mark of since version is absent for more than 40 math expressions,  which were added in different versions, v1.1.1, v1.4.0, v1.5.0, v2.0.0, v2.3.0, etc

we should fill them up properly and may visit other groups of functions later for the same.