av9ash/GitBugs huggingface dataset:
Bug: When the comparator of ArraySort returns null, it should fail.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/22 23:08
Affects Version/s: 3.3.0

When the comparator of {{ArraySort}} returns {{null}}, currently it handles it as {{0}} (equal).

According to the doc, 

{quote}
It returns -1, 0, or 1 as the first element is less than, equal to, or greater than the second element. If the comparator function returns other values (including null), the function will fail and raise an error.
{quote}

It's fine to return non -1, 0, 1 integers to follow the Java convention (still need to update the doc, though), but it should throw an exception for {{null}} result.
Bug: Refine ANSI error messages and remove 'To return NULL instead'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/22 16:58
Affects Version/s: 3.4.0

Proposal: remove ‘To return NULL instead’ as this should not be the purpose and the user may not even aware it was returning NULLs.

Example:

Old:
{{1[MAP_KEY_DOES_NOT_EXIST] Key 3 does not exist. To return NULL instead, use `try_element_at`. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.2== SQL(line 1, position 8) ==3SELECT m[3] FROM VALUES(map(1, 'Hello', 2, ...4       ^^^^}}
Proposed:
{{1[MAP_KEY_DOES_NOT_EXIST] Key 3 does not exist in the map. Use `try_element_at` to tolerate a non-existent key and return NULL. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.2== SQL(line 1, position 8) ==3SELECT m[3] FROM VALUES(map(1, 'Hello', 2, ...4       ^^^^}}
Bug:  when doing iteration compute in graphx, checkpoint need support storagelevel
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jun/22 07:14
Affects Version/s: 2.4.4

this issue related to spark-30502, in that issue ,it olny fix partial ml algorithm.

in graphx pregel, inside iteration compute, the message checkpointer also should support setting storage level, other than the default memory only level
Bug: Handle Null partition values in PartitioningUtils
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/22 16:34
Affects Version/s: 3.3.0

partitions with null values we get a NPE on partition discovery, earlier we use to get `DEFAULT_PARTITION_NAME`

 

{quote} [info]   java.lang.NullPointerException:
[info]   at org.apache.spark.sql.execution.datasources.PartitioningUtils$.removeLeadingZerosFromNumberTypePartition(PartitioningUtils.scala:362)
[info]   at org.apache.spark.sql.execution.datasources.PartitioningUtils$.$anonfun$getPathFragment$1(PartitioningUtils.scala:355)
[info]   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[info]   at scala.collection.Iterator.foreach(Iterator.scala:943)
[info]   at scala.collection.Iterator.foreach$(Iterator.scala:943){quote}
Bug: spark-sql remain hive resource download dir after exit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/22 10:23
Affects Version/s: 3.3.0


{code:java}
drwxrwxr-x  2 yi.zhu           yi.zhu                4096 Jun  7 18:06 da92eec4-2db1-4941-9e53-b28c38e25e31_resources
drwxrwxr-x  2 yi.zhu           yi.zhu                4096 Jun  7 18:14 dad364e8-ed1d-4ced-a6df-4897361c69b1_resources
drwxrwxr-x  2 yi.zhu           yi.zhu                4096 Jun  7 18:13 ee0a2ee7-ff3e-4346-9181-e8e491b1ca15_resources
drwxr-xr-x  2 yi.zhu           yi.zhu                4096 Jun  7 18:16 hsperfdata_yi.zhu

{code}
Bug: When raising an exception, pass parameters as a map instead of an array
Status: Open | Priority: Major | Resolution: 
Created: 08/Jun/22 16:03
Affects Version/s: 3.3.1

We have moved away from c-style parameters in error message texts towards symbolic parameters. E.g.

 
{code:java}
"CANNOT_CAST_DATATYPE" : {
  "message" : [
    "Cannot cast <sourceType> to <targetType>."
  ],
  "sqlState" : "22005"
},{code}
{{However when we raise an exception we merely pass a simple array and assume positional assignment. }}
{code:java}
def cannotCastFromNullTypeError(to: DataType): Throwable = {
  new SparkException(errorClass = "CANNOT_CAST_DATATYPE",
     messageParameters = Array(NullType.typeName, to.typeName), null)
}{code}
 

This has multiple downsides:
 # It's not possible to mention the same parameter twice in an error message.
 # When reworking an error message we cannon shuffle parameters without changing the code
 # There is a risk that the error message and the exception go out of synch unnoticed given we do not want to check for the message text in the code.

So in this PR we propose the following new usage:
{code:java}
def cannotCastFromNullTypeError(to: DataType): Throwable = {
  new SparkException(errorClass = "CANNOT_CAST_DATATYPE",
    messageParameters = Map("sourceType" -> NullType.typeName, "targetType" ->to.typeName),
    context = null)
}{code}
getMessage will then substitute the parameters in the message appropriately.

Moving forward this should be the preferred way to raise exceptions.
Bug: Improve error message for user specified column list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/22 11:31
Affects Version/s: 3.4.0

{noformat}
scala> spark.sql("create table t1(id int, name string) using parquet")
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql("insert into t1(id, nn) values(1, 'name')")
org.apache.spark.sql.AnalysisException: Cannot resolve column name nn; line 1 pos 0
  at org.apache.spark.sql.errors.QueryCompilationErrors$.cannotResolveUserSpecifiedColumnsError(QueryCompilationErrors.scala:406)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUserSpecifiedColumns$.$anonfun$resolveUserSpecifiedColumns$2(Analyzer.scala:3406)
  at scala.Option.getOrElse(Option.scala:189)
{noformat}
Bug: Ignore comment syntax in dfs command
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Jun/22 18:32
Affects Version/s: 3.2.1

In version 3.2, Spark SQL ignores semicolons in comment syntax when splitting input lines.

This behavior throws an unexpected dfs command error:

(Related PR: SPARK-33100 SPARK-30049)


{code:bash}
mkdir testDfsCommand
echo 'abc' > testDfsCommand/a---.txt

./spark-2.4.8-bin-hadoop2.7/bin/spark-sql -e 'dfs -cat testDfsCommand/a---.txt;'
# abc
./spark-2.4.8-bin-hadoop2.7/bin/spark-sql -e 'dfs -cat testDfsCommand/*;'
# abc

./spark-3.2.1-bin-hadoop2.7/bin/spark-sql -e 'dfs -cat testDfsCommand/a---.txt;'
# cat: `testDfsCommand/a---.txt;': No such file or directory
# Command -cat testDfsCommand/a---.txt; failed with exit code = 1
./spark-3.2.1-bin-hadoop2.7/bin/spark-sql -e 'dfs -cat testDfsCommand/*;'
# cat: `testDfsCommand/*;': No such file or directory
# Command -cat testDfsCommand/*; failed with exit code = 1
{code}
Bug: FileAlreadyExistsException while insertInto() DF to hive table or directly write().parquet()
Status: Open | Priority: Major | Resolution: 
Created: 04/Jun/22 16:16
Affects Version/s: 2.4.8

 

I have such structure of table where I want to write DF:
{code:java}
CREATE EXTERNAL TABLE `usl_rdm_idl_spark_stg.okogu_h`(
  `ctl_loading` bigint,
  `ctl_validfrom` timestamp,
  `end_dt` date,
  `okogu_accept_dt` date)
PARTITIONED BY (
  `p1day` string)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  'hdfs://FESS-DEV/data/usl/rdm_idl_spark/stg/okogu_h'
TBLPROPERTIES (
  'bucketing_version'='2',
  'spark.sql.partitionProvider'='catalog',
  'transient_lastDdlTime'='1654082666')
{code}
Final DF has the same structure as mentioned table structure. The issue happens when attr "p1day" (table is partitioned by this attr) has *null* value only. So when I try to write it with any option 
{code:java}
finalDF.write().mode(SaveMode.Append).partitionBy("p1day").parquet("somepath);{code}
 or
{code:java}
finalDF.write().mode(SaveMode.Append).insertInto(String.format("%s.%s", tgtSchema, tgtTable));{code}
I see such error:
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.fs.FileAlreadyExistsException: /data/usl/rdm_idl_spark/stg/okogu_h/.hive-staging_hive_2022-06-01_16-59-37_442_6329951430234699240-1/-ext-10000/_temporary/0/_temporary/attempt_20220601165937_0116_m_000001_586/p1day=__HIVE_DEFAULT_PARTITION__/part-00001-05999af9-8a25-406e-a307-f97781547db2.c000 for client 10.106.105.11 already exists{code}
 

For me it works correctly only when I replace null value in "p1day" column with any value( for ex. "1"):
{code:java}
finalDF.withColumn("p1day",lit("1"));{code}
 

Is it a bug in spark-sql code? I use org.apache.spark:spark-sql_2.11:2.4.8
Bug: fix spark.kubernetes.memoryOverheadFactor deprecation warning
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Jun/22 13:45
Affects Version/s: 3.4.0

see [https://github.com/apache/spark/pull/36744] for details.

 

It looks like we deprecated {{spark.kubernetes.memoryOverheadFactor, but it has a default value which leads it to printing warnings all the time.}}

{{}}
{code:java}
22/06/01 23:53:49 WARN SparkConf: The configuration key 'spark.kubernetes.memoryOverheadFactor' has been deprecated as of Spark 3.3.0 and may be removed in the future. Please use spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor{code}
{{}}

{{We should remove the default value if possible. It should only be used as fallback but we should be able to use the default from }}spark.driver.memoryOverheadFactor.
Bug: pmCache memory leak caused by IsolatedClassLoader
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Jun/22 15:40
Affects Version/s: 2.4.4

I found this bug in Spark 2.4.4, because the related code has not changed, so this bug still exists on master, the following is a brief description of this bug:

In May 2015, [SPARK-6907|https://github.com/apache/spark/commit/daa70bf135f23381f5f410aa95a1c0e5a2888568] introduced isolated classloader for HiveMetastore to support Hive multi-version loading, but this PR resulted in [RawStore cleanup mechanism|https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/ThreadFactoryWithGarbageCleanup.java#L27-L42] is broken because the `ThreadWithGarbageCleanup` class used by `HiveServer2-Handler-Pool` and `HiveServer2-Background-Pool` and `HiveServer2-HttpHandler-Pool` is loaded by AppClassLoader, in the source code of `ThreadWithGarbageCleanup` class: `RawStore threadLocalRawStore = HiveMetaStore.HMSHandler.getRawStore();` This line of code will use the `threadLocalMS` instance in `HiveMetaStore.HMSHandler` (loaded by AppClassLoader), and in the process of thread execution, the `client` actually created by isolatedClassLoader, in the process of obtaining `RawStore` instance through `HiveMetaStore.HMSHandler#getMSForConf`, the `ms` instance is set to `threadLocalMS`, but the static `threadLocalMS` instance belongs to `HMSHandler`(loaded by IsolatedClassLoader$$anon$1), that is, the set and get methods do not operate on the same `threadLocalMS` instance, so in `ThreadWithGarbageCleanup#cacheThreadLocalRawStore` method, the obtained `RawStore` instance is null, so the subsequent `RawStore` cleaning logic does not take effect, because the `shutdown` method of `RawStore` instance is not called, resulting in `pmCache` of `JDOPersistenceManagerFactory` memory leak.

Long-running Spark ThriftServer end up with frequent GCs, resulting in poor performance.

I analyzed the heap dump using MAT and executed the following OQL: `SELECT * FROM INSTANCEOF java.lang.Class c WHERE c.@displayName.contains("class org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler ")`, two instances of the `HMSHandler` *Class* can be found in the heap. Also know that they each hold a static `threadLocalMS` instance.

We execute the following OQL: `select * from org.datanucleus.api.jdo.JDOPersistenceManagerFactory`, we can see that `pmCache` of the `JDOPersistenceManagerFactory` instance has over 200,000 elements and consumes 1.3GB of memory.

We execute the following OQL: `SELECT * FROM INSTANCEOF java.lang.Class c WHERE c.@displayName.contains("class org.apache.hive.service.server.ThreadFactoryWithGarbageCleanup")`, we can see that there is no element in the static instance `threadRawStoreMap` of `ThreadFactoryWithGarbageCleanup`, which confirms the above statement, because `HMSHandler.getRawStore()` in `ThreadWithGarbageCleanup#cacheThreadLocalRawStore` is called on the `threadLocalMS` instance in `HMSHandler`(loaded by AppClassLoader) instead of `threadLocalMS` instance in `HMSHandler`(loaded by IsolatedClassLoader$$anon$1).
Bug: BlockInfoManager should not release write locks on task end
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Jun/22 14:50
Affects Version/s: 3.2.1

The BlockInfoManager releases all locks held by a task when the task is done. It also release write locks, the problem with that is that a thread (other than the main task thread) might still be modifying the block. By releasing it the block now seems readable, and a reader might observe a block in a partial or non-existent state.

This is fortunately not the massive problem as it appears to be because the BlockManager (only place where we write blocks) is well behaved and always puts the block in a consistent state. This means the errors caused by this are only transient.

Given the fact that the write code is well behaved we don't need to release the write locks on task end. We should remove that.
Bug: Stop using Log4J2's extended throwable logging pattern in default logging configurations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/22 02:54
Affects Version/s: 3.3.0

This PR addresses a performance problem in Log4J 2 related to exception logging: in certain scenarios I observed that Log4J2's exception stacktrace logging can be ~10x slower than Log4J 1.

The problem stems from a new log pattern format in Log4J2 called ["extended exception"|https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternExtendedException], which enriches the regular stacktrace string with information on the name of the JAR files that contained the classes in each stack frame.

Log4J queries the classloader to determine the source JAR for each class. This isn't cheap, but this information is cached and reused in future exception logging calls. In certain scenarios involving runtime-generated classes, this lookup will fail and the failed lookup result will _not_ be cached. As a result, expensive classloading operations will be performed every time such an exception is logged. In addition to being very slow, these operations take out a lock on the classloader and thus can cause severe lock contention if multiple threads are logging errors. This issue is described in more detail in a comment on a Log4J2 JIRA and in a linked blogpost: https://issues.apache.org/jira/browse/LOG4J2-2391?focusedCommentId=16667140&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16667140 . Spark frequently uses generated classes and lambdas and thus Spark executor logs will almost always trigger this edge-case and suffer from poor performance.

By default, if you do not specify an explicit exception format in your logging pattern then Log4J2 will add this "extended exception" pattern (see PatternLayout's {{alwaysWriteExceptions}} flag in Log4J's documentation, plus [the code implementing that flag|https://github.com/apache/logging-log4j2/blob/d6c8ab0863c551cdf0f8a5b1966ab45e3cddf572/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/PatternParser.java#L206-L209] in Log4J2).

In this PR, I have updated Spark's default Log4J2 configurations so that each pattern layout includes an explicit {{%ex}} so that it uses the normal (non-extended) exception logging format.

Although it's true that any program logging exceptions at a high rate should probably just fix the source of the exceptions, I think it's still a good idea for us to try to fix this out-of-the-box performance difference so that users' existing workloads do not regress when upgrading to 3.3.0.
Bug: Logs are still "in_progress" when we read HDFS files
Status: Open | Priority: Major | Resolution: 
Created: 02/Jun/22 14:12
Affects Version/s: 3.0.2

When I read file from HDFS using hadoop FileSystem after SparkSession initializing, logs are still "in_progress" in event log. In Spark History Server logs from this application are empty (no jobs, no stages etc.).

To be clear: job is finished successfully, but in Spark History Server, we have this strange situation.
Bug: There are problems in canUpCast function
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/22 08:14
Affects Version/s: 3.2.1

Spark's canUpCast means one value can be cast to another type without the loss of precision. But actually long type can not be cast to float/double
{code:java}
def canUpCast(from: DataType, to: DataType): Boolean = (from, to) match {
  case _ if from == to => true
  case (from: NumericType, to: DecimalType) if to.isWiderThan(from) => true
  case (from: DecimalType, to: NumericType) if from.isTighterThan(to) => true
  case (f, t) if legalNumericPrecedence(f, t) => true
  case (DateType, TimestampType) => true
  case (_: AtomicType, StringType) => true
  case (_: CalendarIntervalType, StringType) => true
  case (NullType, _) => true

... {code}
{code:java}
private def legalNumericPrecedence(from: DataType, to: DataType): Boolean = { 
val fromPrecedence = TypeCoercion.numericPrecedence.indexOf(from)
val toPrecedence = TypeCoercion.numericPrecedence.indexOf(to) fromPrecedence >= 0 && fromPrecedence < toPrecedence
}
{code}
{code:java}
val numericPrecedence =
  IndexedSeq(
    ByteType,
    ShortType,
    IntegerType,
    LongType,
    FloatType,
    DoubleType) {code}
Bug: Support coalesce partition through cartesianProduct
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 28/Jun/22 03:42
Affects Version/s: 3.2.0

`CoalesceShufflePartitions` can not optimize CartesianProductExec and the result partition would be `left partition * right partition` which can be quite lagre.

 

It's better to support partial optimize with `CartesianProduct`.
Bug: Deprecate scoreLabelsWeight in BinaryClassificationMetrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jun/22 12:21
Affects Version/s: 3.4.0

scoreLabelsWeight in BinaryClassificationMetrics is a public variable,

but it should be private, moveover, it is only used once, so deprecate it now and remove it in 4.0.0
Bug: The analysis exception is incorrect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/22 13:03
Affects Version/s: 3.3.0

{noformat}
scala> spark.sql("create table t1(user_id int, auct_end_dt date) using parquet;")
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql("select * from t1 join t2 on t1.user_id = t2.user_id where t1.auct_end_dt >= Date_sub('2020-12-27', 90)").show
org.apache.spark.sql.AnalysisException: cannot resolve 'date_sub('2020-12-27', 90)' due to data type mismatch: argument 1 requires date type, however, ''2020-12-27'' is of string type.; line 1 pos 76
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.RemoveTempResolvedColumn$.$anonfun$apply$82(Analyzer.scala:4334)
  at org.apache.spark.sql.catalyst.analysis.RemoveTempResolvedColumn$.$anonfun$apply$82$adapted(Analyzer.scala:4327)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:365)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:364)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:364)
{noformat}

The analysis exception should be:
{noformat}
org.apache.spark.sql.AnalysisException: Table or view not found: t2
{noformat}
Bug: "Since <version>" docs on array_agg are incorrect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/22 19:31
Affects Version/s: 3.3.0

[https://spark.apache.org/docs/latest/api/sql/#array_agg]

The docs currently say "Since: 2.0.0", but `array_agg` was added in Spark 3.3.0.
Bug: Streamline the options for the `.write` method of a Spark DataFrame
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 15/Jun/22 02:34
Affects Version/s: 3.2.1

Hi Team!

I'd like to set up a much easier way to optimize my {{delta}} tables. Specifically, I am referring to the {{sql}} command {{{}OPTIMIZE <table>{}}}.

Let me show you the differences:

*Current:*

First, run:
{code:java}
import pandas as pd
df = spark.createDataFrame(
    pd.DataFrame(
        {'a': [1,2,3,4],
         'b': ['a','b','c','d']}
    )
)
df.write.mode('overwrite').format('delta').save('./folder'){code}
 Then, once it's saved, run:
{code:java}
CREATE TABLE df USING DELTA LOCATION './folder' {code}
 Then, once the table is loaded, run:
{code:java}
OPTIMIZE df
--or
OPTIMIZE df ZORDER BY (b) {code}
As you can see, there are many steps needed.

*Future:*
I'd like to be able to do something like this:
{code:java}
import pandas as pd
df = spark.createDataFrame(
    pd.DataFrame(
        {'a':[1,2,3,4],
         'b':['a','b','c','d']}
    )
)
df.write.mode('overwrite').format('delta').options(optimize=True).save('./folder')
#or
df.write.mode('overwrite').format('delta').options(optimize=True, zorder_by=('b')).save('./folder') {code}
As you can see, it's much more streamlined, and keeps the code to a higher-level.

Thank you.

 
References:
 * [https://docs.azuredatabricks.net/_static/notebooks/delta/optimize-python.html]
 * [https://medium.com/@debusinha2009/cheatsheet-on-understanding-zorder-and-optimize-for-your-delta-tables-1556282221d3]
 * [https://www.cloudiqtech.com/partition-optimize-and-zorder-delta-tables-in-azure-databricks/]
 * [https://docs.databricks.com/delta/optimizations/file-mgmt.html]
 * [https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-optimize.html]
 * [https://stackoverflow.com/questions/65320949/parquet-vs-delta-format-in-azure-data-lake-gen-2-store?_sm_au_=iVV4WjsV0q7WQktrJfsTkK7RqJB10]
 * [https://www.i-programmer.info/news/197-data-mining/12582-databricks-delta-adds-faster-parquet-import.html#:~:text=Databricks%20says%20Delta%20is%2010,data%20management%2C%20and%20query%20serving].