av9ash/GitBugs huggingface dataset:
Bug: OneVsRestModel.transform should not call setter of submodels
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/21 03:12
Affects Version/s: 3.2.0

featuresCol of submodels maybe changed in transform:
{code:java}
 scala> val df = spark.read.format("libsvm").load("/d0/Dev/Opensource/spark/data/mllib/sample_multiclass_classification_data.txt")
21/01/08 09:52:01 WARN LibSVMFileFormat: 'numFeatures' option not specified, determining the number of features by going though the input. If you know the number in advance, please specify it via 'numFeatures' option to avoid the extra scan.
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]

scala> val lr = new LogisticRegression().setMaxIter(1).setTol(1E-6).setFitIntercept(true)
lr: org.apache.spark.ml.classification.LogisticRegression = logreg_3003cb3321a1

scala> val ovr = new OneVsRest().setClassifier(lr)
ovr: org.apache.spark.ml.classification.OneVsRest = oneVsRest_b2ec3ec45dbf

scala> val ovrm = ovr.fit(df)
21/01/08 09:52:05 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
21/01/08 09:52:05 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
ovrm: org.apache.spark.ml.classification.OneVsRestModel = OneVsRestModel: uid=oneVsRest_b2ec3ec45dbf, classifier=logreg_3003cb3321a1, numClasses=3, numFeatures=4

scala> val df2 = df.withColumnRenamed("features", "features2")
df2: org.apache.spark.sql.DataFrame = [label: double, features2: vector]

scala> ovrm.setFeaturesCol("features2")
res0: ovrm.type = OneVsRestModel: uid=oneVsRest_b2ec3ec45dbf, classifier=logreg_3003cb3321a1, numClasses=3, numFeatures=4


scala> ovrm.models.map(_.getFeaturesCol)
res1: Array[String] = Array(features, features, features)

scala> ovrm.transform(df2)
res2: org.apache.spark.sql.DataFrame = [label: double, features2: vector ... 2 more fields]

scala> ovrm.models.map(_.getFeaturesCol)
res3: Array[String] = Array(features2, features2, features2)
{code}
Bug: Repair enumeration conversion error for AllJobsPage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jan/21 12:57
Affects Version/s: 3.0.0

For AllJobsPage class, AllJobsPage gets the schedulingMode of enumerated type by loading the spark.scheduler.mode configuration from Sparkconf, but an enumeration conversion error occurs when I set the value of this configuration to lowercase.

The reason for this problem is that the value of the SchedulingMode enumeration class is uppercase, which occurs when I configure spark. scheduler.mode to be lowercase.
Bug: Add UI elements to collapse/expand all stages detail in stages tab
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Jan/21 03:08
Affects Version/s: 3.0.1

Currently there is no way to expand all stages details all in once. It is hard if you want to navigate to search for specific stages details.

Proposed to add a UI elements to collapse/expand all stages detail in stages tab. 

!image-2021-01-10-10-53-23-392.png!
Bug: Use join hint in test cases for Join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/21 05:55
Affects Version/s: 3.0.0

There are some existing test cases that constructing various joins by tuning the SQL configuration AUTO_BROADCASTJOIN_THRESHOLD, PREFER_SORTMERGEJOIN,SHUFFLE_PARTITIONS, etc. 
This can be tricky. Constructing a specific join by using join hint can be simpler and more robust.
Bug: function runCliWithin of CliSuite can not cover some test cases
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jan/21 06:34
Affects Version/s: 3.0.1

Here is a  test case(query contains two statements splitted by '\n') which can not be covered by runCliWithin:

{code:java}
    runCliWithin(1.minute)("select 'test1';\n select 'test2';" -> "test2")

{code}



{code:java}
11:35:00.383 pool-1-thread-1-ScalaTest-running-CliSuite INFO CliSuite: Cli driver is booted. Waiting for expected answers.
11:35:01.104 Thread-6 INFO CliSuite: 2021-01-06 19:35:01.104 - stdout> spark-sql> select 'test1';
11:35:01.104 Thread-6 INFO CliSuite: stdout> found expected output line 0: 'spark-sql> select 'test1';'
11:35:10.120 Thread-6 INFO CliSuite: 2021-01-06 19:35:10.12 - stdout> test1
11:35:10.121 Thread-7 INFO CliSuite: 2021-01-06 19:35:10.121 - stderr> Time taken: 8.987 seconds, Fetched 1 row(s)
11:35:10.151 Thread-6 INFO CliSuite: 2021-01-06 19:35:10.151 - stdout> spark-sql>  select 'test2';
11:35:10.220 Thread-6 INFO CliSuite: 2021-01-06 19:35:10.22 - stdout> test2
11:35:10.220 Thread-7 INFO CliSuite: 2021-01-06 19:35:10.22 - stderr> Time taken: 0.068 seconds, Fetched 1 row(s)
11:35:10.443 Thread-6 INFO CliSuite: 2021-01-06 19:35:10.443 - stdout> spark-sql> 
11:36:00.390 pool-1-thread-1-ScalaTest-running-CliSuite ERROR CliSuite: 
=======================
CliSuite failure output
=======================
Spark SQL CLI command line: ../../bin/spark-sql --master local --driver-java-options -Dderby.system.durability=test --conf spark.ui.enabled=false --hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/Users/fwang12/ebay/apache-spark/target/tmp/spark-7c275c0c-fc8e-49c7-b643-18f20fe8ba51;create=true --hiveconf hive.exec.scratchdir=/Users/fwang12/ebay/apache-spark/target/tmp/spark-bb0fded2-1a25-45d2-9f78-16898f32aefc --hiveconf conf1=conftest --hiveconf conf2=1 --hiveconf hive.metastore.warehouse.dir=/Users/fwang12/ebay/apache-spark/target/tmp/spark-4901396f-9a7a-4299-b7fc-9cb3b24c46f4
Exception: java.util.concurrent.TimeoutException: Futures timed out after [1 minute]
Failed to capture next expected output "         >  select 'test2';" within 1 minute.
{code}

It seems that it is not recommended to transfer multiple queries one time, but there is some UT like this:
https://github.com/apache/spark/blob/f9daf035f473fea12a2ee67428db8d78f29973d5/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala#L542-L544
Bug: Refactor v2CommandExec to provide an API to create an InternalRow
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jan/21 00:55
Affects Version/s: 3.2.0

There are many v2 commands such as "SHOW TABLES", "DESCRIBE TABLE", etc. that require creating InternalRow. Creating InternalRow can be refactored into v2CommandExec to remove duplicate code to create serializer, etc.
Bug: spark sql DoubleType(0 * (-1))  return "-0.0"
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 02/Jan/21 09:05
Affects Version/s: 2.3.2

spark version: 2.3.2
{code:java}
create table test_zjg(a double);
insert into test_zjg values(-1.0);
select a*0 from test_zjg
{code}
 After select operation, *{color:#de350b}we will get -0.0 which expected as 0.0:{color}*

\+------------------------\+

\|(a * CAST(0 AS DOUBLE))\|

\+------------------------\+

\|-0.0                               \|

\+------------------------\+
Bug: add_hours SQL function
Status: In Progress | Priority: Minor | Resolution: 
Created: 04/Jan/21 01:20
Affects Version/s: 3.1.0

This will add an add_hours function that'll make it easier to add and subtract hours from datetime values.  Users currently need to manipulate strings and write code like `expr("first_datetime + INTERVAL 2 hours")`.  It'll be a lot easier to let them write code like `add_hours(col("first_datetime"), lit(2))`.  There's [a Stackoverflow question on this topic with a lot of views|https://stackoverflow.com/questions/40883084/adding-12-hours-to-datetime-column-in-spark], which indicates that this feature will help a lot of users.

See the [GitHub pull request description|https://github.com/apache/spark/pull/31000] for additional details.
Bug: Fix incorrect min partition condition in getRanges
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jan/21 18:06
Affects Version/s: 3.2.0

When calculating offset ranges, we consider minPartitions configuration. If minPartitions is not set or is less than or equal the size of given ranges, it means there are enough partitions at Kafka so we don't need to split offsets to satisfy min partition requirement. But the current condition is offsetRanges.size > minPartitions.get and is not correct. Currently getRanges will split offsets in unnecessary case.
Bug: Fix of typos in documentation of pyspark.sql.functions and output of lint-python
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 30/Jan/21 23:24
Affects Version/s: 3.0.1

Minor documentation and standard output issues:

* {{dev/lint-python}} contains a typo when printing a warning regarding bad Sphinx version ("lower then 3.1" rather than "lower than 3.1")
* The documentations of the functions {{lag}} and {{lead}} of {{pyspark.sql.functions}} refer to a parameter {{defaultValue}}, which in reality is named {{default}}.
* The documentation strings of functions in {{pyspark.sql.functions}} make reference to the {{Column}} class, which is not resolved by Sphinx unless fully qualified as {{pyspark.sql.Column}}
Bug: Allow multiple namespaces with session catalog
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Jan/21 01:01
Affects Version/s: 3.1.0

SPARK-30885 removed the ability for tables in session catalogs being queried with SQL to have multiple namespaces. This seems to have been added as a follow up, not as part of the core change. We should explore if this restriction can be relaxed.
Bug: VectorSlicer refine numFeatures checking and toString method
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Jan/21 06:52
Affects Version/s: 3.2.0

Current checking of numFeatures needs updates:

1, should use inputAttr.size instead of inputAttr.numAttributes to get numFeatures;

2, this checking is necessary only if $(indices).nonEmpty, otherwise, $(indices).max will throw exception java.lang.UnsupportedOperationException: empty.max

3, in toString, should add length of names;
Bug: Object hash aggregate should not fallback if no more input rows
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/Jan/21 23:39
Affects Version/s: 3.2.0

Object hash aggregate will fallback to sort-based aggregation based on number of keys seen so far [0]. The default config threshold is 128 (spark.sql.objectHashAggregate.sortBased.fallbackThreshold in [1]). There's an edge case we can do better, where we do not fallback if there's no more input rows. Suppose the task only has 128 group-by keys in hash ma, we don't need to fallback in this case, and we can save the extra sort.

 

[0]: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala#L161] 

[1]: [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1615]
Bug: Add more metrics (fallback, spill) for object hash aggregate
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/Jan/21 06:38
Affects Version/s: 3.2.0

As object hash aggregate fallback mechanism is special - it will fallback to sort-based aggregation based on number of keys seen so far [0]. This fallback logic sometimes is sub-optimal and leads to unnecessary sort, and performance degradation in run-time. The first step to help user/developer debug is to add more related metrics on UI, e.g. spill size, and number of fallback to sort-based aggregation. (spill size metrics was already added for hash aggregate [1])

 

[0]: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala#L161] 

[1]: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala#L68]
Bug: Rename `isTemporaryTable` to `isTempView` in `SessionCatalog`
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Jan/21 20:47
Affects Version/s: 3.2.0

Currently, there are two methods that do the same but have different names:

{code:java}
def isTempView(nameParts: Seq[String]): Boolean
{code}

and

{code:java}
def isTemporaryTable(name: TableIdentifier): Boolean
{code}
 It would be nice to rename `SessionCatalog.isTemporaryTable()` to `SessionCatalog.isTempView()`.
Bug: Make EventLoggingListener.codecMap thread-safe
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Jan/21 08:14
Affects Version/s: 2.4.7

2.x version of history server
 EventLoggingListener.codecMap is of type mutable.HashMap, which is not thread safe
 This will cause the history server to suddenly get stuck and not work.

The 3.x version was changed to EventLogFileReader.codecMap to ConcurrentHashMap type, so there is no such problem.(-SPARK-28869-)

PID 117049 0x1c939

!top.png!

 

!jstack.png!
Bug: Replaces find and emptiness check with exists.
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 11/Jan/21 11:05
Affects Version/s: 3.2.0

Semantic consistency and code Simpilefications

before:
{code:java}
seq.find(p).isDefined
or
seq.find(p).isEmpty
{code}
after:
{code:java}
seq.exists(p)
or
!seq.exists(p)
{code}
Bug: Automatically active aarch64 profile when build on Arm platform
Status: Closed | Priority: Trivial | Resolution: Duplicate
Created: 07/Jan/21 03:36
Affects Version/s: 3.0.1

The support of build on aarch64 platform was added in [https://github.com/apache/spark/pull/26636] by adding a new profile to switch to use arm version of leveldb jni when build on Arm, users have to specify the profile when building, and this message is currently missing in the build guide.

In fact, we can automatically active aarch64 profile when build on Arm platform to make it simpler for users. This can be done as:

https://github.com/apache/hadoop/blob/b612c310c26394aa406c99d8598c9cb7621df052/pom.xml#L688
Bug: API Auditing
Status: Resolved | Priority: Blocker | Resolution: Done
Created: 07/Mar/23 07:47
Affects Version/s: 3.4.0

Audit user-facing API of Spark 3.4. The main goal is to ensure public API docs to be ready for release, for example, no private classes/methods is leaking and marked public.

There are 3 common ways to audit API:
 * build docs (into a local website) against branch-3.4 to check
 * 'git diff' to check the source code differences between v3.3.2 and branch-3.4
 * [https://github.com/apache/spark-website/pull/443] shows most of the API doc differences between v3.3.2 and the 3.4.0 RC4(the latest RC); commits are categorized by components
Bug: Spark 3.4 history server's SQL tab incorrectly groups SQL executions when replaying event logs from Spark 3.3 and earlier
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 11/Mar/23 03:04
Affects Version/s: 3.4.0

In Spark 3.4.0 RC4, the Spark History Server's SQL tab incorrectly groups SQL executions when replaying event logs generated by older Spark versions.

 

{*}Reproduction{*}:

{{In ./bin/spark-shell --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=eventlogs, run three non-nested SQL queries:}}
{code:java}
sql("select * from range(10)").collect()
sql("select * from range(20)").collect()
sql("select * from range(30)").collect(){code}
Exit the shell and use the Spark History Server to replay this application's UI.

In the SQL tab I expect to see three separate queries, but Spark 3.4's history server incorrectly groups the second and third queries as nested queries of the first (see attached screenshot).

 

{*}Root cause{*}: 

[https://github.com/apache/spark/pull/39268] / SPARK-41752 added a new *non-optional* {{rootExecutionId: Long}} field to the SparkListenerSQLExecutionStart case class.

When JsonProtocol deserializes this event it uses the "ignore missing properties" Jackson deserialization option, causing the {{rootExecutionField}} to be initialized with a default value of {{{}0{}}}.

The value {{0}} is a legitimate execution ID, so in the deserialized event we have no ability to distinguish between the absence of a value and a case where all queries have the first query as the root.

*Proposed* {*}fix{*}:

I think we should change this field to be of type {{Option[Long]}} . I believe this is a release blocker for Spark 3.4.0 because we cannot change the type of this new field in a future release without breaking binary compatibility.
Bug: Data duplication and loss occur after executing 'insert overwrite...' in Spark 3.1.1
Status: Open | Priority: Critical | Resolution: 
Created: 07/Mar/23 07:51
Affects Version/s: 3.1.1

We are currently using Spark version 3.1.1 in our production environment. We have noticed that occasionally, after executing 'insert overwrite ... select', the resulting data is inconsistent, with some data being duplicated or lost. This issue does not occur all the time and seems to be more prevalent on large tables with tens of millions of records.

We compared the execution plans for two runs of the same SQL and found that they were identical. In the case where the SQL was executed successfully, the amount of data being written and read during the shuffle stage was the same. However, in the case where the problem occurred, the amount of data being written and read during the shuffle stage was different. Please see the attached screenshots for the write/read data during shuffle stage.
 
Normal SQL:
!image-2023-03-07-15-59-08-818.png!


SQL with issues:
!image-2023-03-07-15-59-27-665.png!
 
Is this problem caused by a bug in version 3.1.1, specifically (SPARK-34534): 'New protocol FetchShuffleBlocks in OneForOneBlockFetcher lead to data loss or correctness'? Or is it caused by something else? What could be the root cause of this problem?
Bug: pyspark.ml.stat.Correlation - Spearman Correlation method giving incorrect and inconsistent results for the same DataFrame if it has huge amount of Ties.
Status: Open | Priority: Critical | Resolution: 
Created: 23/Mar/23 05:20
Affects Version/s: 3.3.0

pyspark.ml.stat.Correlation

Following is the Scenario where the Correlation function fails for giving correct Spearman Coefficient Results.

Tested E.g -> Spark DataFrame has 2 columns A and B.

!image-2023-03-23-10-55-26-879.png|width=562,height=162!

Column A has 3 Distinct Values and total of 108Million rows

Column B has 4 Distinct Values and total of 108Million rows

If I Calculate the correlation for this DataFrame in Python Pandas DF.corr, it gives the correct answer even if i run the same code multiple times the same answer is produced. (Each column has only 3-4 distinct values)

!image-2023-03-23-10-53-37-461.png|width=468,height=287!

 

Coming to Spark and using Spearman Correlation produces a *different results* for the *same dataframe* on multiple runs. (see below) (each column in this df has only 3-4 distinct values)

!image-2023-03-23-10-52-49-392.png|width=516,height=322!

 

Basically in python Pandas Df.corr it gives same results on same dataframe on multiple runs which is expected behaviour. However, in Spark using the same data it gives different result, moreover running the same cell with same data multiple times produces different results meaning the output is inconsistent.

Coming to data the only observation I could conclude is Ties in data. (Only 3-4 Distinct values over 108M Rows.) This scenario is not handled in Spark Correlation method as the same data when used in python using df.corr produces consistent results.

The only Workaround we could find to get consistent and the same output as from python in Spark is by using Pandas UDF as shown below:

!image-2023-03-23-10-52-11-481.png|width=518,height=111!

!image-2023-03-23-10-51-28-420.png|width=509,height=270!

 

We also tried pyspark.pandas.DataFrame .corr method and it produces incorrect and inconsistent results for this case too.

Only PandasUDF seems to provide consistent results.

 

Another point to note is : If i add some random noise to the data, which will inturn increase the distinct values in the data. It again gives consistent results for any runs. Which makes me believe that the Python version handles ties correctly and gives consistent results no matter how many ties exist. However, pyspark method is somehow not able to handle many ties in data.
Bug: Inconsistent output from label propagation algorithm in Graph X due to tie-breaking logic in vertexProgram
Status: Open | Priority: Major | Resolution: 
Created: 19/Mar/23 23:34
Affects Version/s: 1.1.0

We are experiencing inconsistent output from the label propagation algorithm in Graph X. When we run the algorithm on the same input, we observe different outputs each time. This behavior is unexpected since the algorithm is not designed to be random, and we should be getting the same output for the same input.

We suspect that the issue lies in the tie-breaking logic used by the [vertexProgram|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/LabelPropagation.scala#L65] when picking labels. Currently, the vertexProgram chooses the label with the maximum frequency, and in case of a tie, it selects the label that appears first. This logic does not handle tie cases correctly, resulting in different outputs for the same input.

{code:scala}
def vertexProgram(vid: VertexId, attr: Long, message: Map[VertexId, Long]): VertexId = {  
    if (message.isEmpty) attr else message.maxBy(_._2)._1
}
{code}

To solve this issue, we propose changing the tie-breaking logic to something like vertex ID. This change will ensure that the same label is always selected in case of a tie, resulting in consistent output from the algorithm for the same input.

{code:scala}
def vertexProgram(vid: VertexId, attr: Long, message: Map[VertexId, Long]): VertexId = {
    if (message.isEmpty) attr else message.maxBy{ case (key, value) => (value, key) }._1
}
{code}

Thanks!
Bug: Add the LISTAGG() aggregate function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/23 11:07
Affects Version/s: 3.4.0

{{listagg()}} is a common and useful aggregation function to concatenate string values in a column, optionally by a certain order. The systems below have supported such function already:
 * Oracle: [https://docs.oracle.com/cd/E11882_01/server.112/e41084/functions089.htm#SQLRF30030]
 * Snowflake: [https://docs.snowflake.com/en/sql-reference/functions/listagg]
 * Amazon Redshift: [https://docs.aws.amazon.com/redshift/latest/dg/r_LISTAGG.html]
 * Google BigQuery: [https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#string_agg]

Need to introduce this new aggregate in Spark, both as a regular aggregate and as a window function.

Proposed syntax:
{code:sql}
LISTAGG( [ DISTINCT ] <expr> [, <delimiter> ] ) [ WITHIN GROUP ( <orderby_clause> ) ]
{code}
Bug: Spark SQL reads unnecessary nested fields
Status: Open | Priority: Major | Resolution: 
Created: 21/Mar/23 07:00
Affects Version/s: 3.3.2

When we use more than one field from structure after explode, all fields will be read.

Example:
1) Loading data
{code:scala}
val jsonStr = """{
 "items": [
   {"itemId": 1, "itemData1": "a", "itemData2": 11},
   {"itemId": 2, "itemData1": "b", "itemData2": 22}
 ]
}"""
val df = spark.read.json(Seq(jsonStr).toDS)
df.write.format("parquet").mode("overwrite").saveAsTable("persisted")
{code}
2) read query with explain
{code:scala}
val read = spark.table("persisted")
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", true)

read
    .select(explode('items).as('item))
    .select($"item.itemId", $"item.itemData1")
    .explain
// ReadSchema: struct<items:array<struct<itemData1:string,itemData2:bigint,itemId:bigint>>>
{code}
We use only *itemId* and *itemData1* fields from structure in array, but read schema contains *itemData2* field as well.