av9ash/GitBugs huggingface dataset:
Bug: MLlib GBTClassifier has wrong impurity method 'variance' instead of 'gini' or 'entropy'. 
Status: Open | Priority: Major | Resolution: 
Created: 17/Aug/23 12:04
Affects Version/s: 3.4.1

Impurity method 'variance' should only be used for regressors, *not* classifiers. For classifiers gini and entropy should be available as it is already the case for the RandomForestClassifier [https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.ml.classification.RandomForestClassifier.html] .

Because of this bug 'minInfoGain' hyperparameter cannot be tuned to combat overfitting.
Bug: Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/23 06:38
Affects Version/s: 3.4.0

Follow-up on https://issues.apache.org/jira/browse/SPARK-40646.

I found that JSON parsing is significantly slower due to exception creation in control flow. Also, some fields are not parsed correctly and the exception is thrown in certain cases: 
{code:java}
Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.util.GenericArrayData cannot be cast to org.apache.spark.sql.catalyst.InternalRow
	at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getStruct(rows.scala:51)
	at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getStruct$(rows.scala:51)
	at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getStruct(rows.scala:195)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:590)
	... 39 more {code}
Bug: Adding extension for Spark SQL authorization with Ranger-Hive policies
Status: Open | Priority: Major | Resolution: 
Created: 29/Aug/23 08:25
Affects Version/s: 3.4.2

Apache Ranger have not plugin for Spark SQL authorization and due to this limitation it's required some extra work and using extensions to manage Spark authorizations.

 

Spark HWC have some performance issues and Cloudera (CDP) solution is complex and required extra copy of data (big data!).

 

Seems natively adding plugins like "Submarine Spark Security Plugin" in Spark will help to implementation of standard and high performance solution. 

Reference:  [https://submarine.apache.org/zh-cn/docs/0.6.0/userDocs/submarine-security/spark-security/] 

 

This plugin works fine with table/column/row level and masking options.
Bug: Switch languages consistently across docs for all code snippets
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/23 18:57
Affects Version/s: 3.4.1

When a user chooses a different language for a code snippet, all code snippets on that page should switch to the chosen language. This was the behavior for, for example, Spark 2.0 doc: [https://spark.apache.org/docs/2.0.0/structured-streaming-programming-guide.html]

But it was broken for later docs, for example the Spark 3.4.1 doc: [https://spark.apache.org/docs/latest/quick-start.html]

We should fix this behavior change and possibly add test cases to prevent future regressions.
Bug: Adding metric tracking number of jobs running on a cluster
Status: Open | Priority: Major | Resolution: 
Created: 18/Aug/23 01:17
Affects Version/s: 4.0.0

We frequently come across issues where in there are considerable number of jobs/notebooks running on one interactive cluster. This leads to either degraded performance due to cluster being overloaded or leads to driver OOM. It would be helpful to have a metric identifying number of notebooks/jobs running on a cluster as this would be important datapoint for customers to balance out their workloads.

We propose to add a metric which keeps track of number of jobs running on a cluster.
Bug: Add Spark version drop down to the PySpark doc site
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/23 16:38
Affects Version/s: 4.0.0

Currently, PySpark documentation does not have a version dropdown. While by default we want people to land on the latest version, it will be helpful and easier for people to find docs if we have this version dropdown. 

Other libraries such as numpy have such version dropdown.  !image-2023-08-09-09-38-00-805.png|width=214,height=189!
Bug: Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/23 14:46
Affects Version/s: 3.3.1

When union-ing two DataFrames read from parquet containing nested structures (2 fields of array types where one is double and second is integer) data from the second field seems to be lost (zeros are set instead). 

This seems to be the case only if nested vectorised reader is used (spark.sql.parquet.enableNestedColumnVectorizedReader=true). 

The following Python code reproduces the problem: 
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql.types import *

# PREPARING DATA
data1 = []
data2 = []

for i in range(2): 
    data1.append( (([1,2,3],[1,1,2]),i))
    data2.append( (([1.0,2.0,3.0],[1,1]),i+10))

schema1 = StructType([
        StructField('value', StructType([
             StructField('f1', ArrayType(IntegerType()), True),
             StructField('f2', ArrayType(IntegerType()), True)             
             ])),
         StructField('id', IntegerType(), True)
])

schema2 = StructType([
        StructField('value', StructType([
             StructField('f1', ArrayType(DoubleType()), True),
             StructField('f2', ArrayType(IntegerType()), True)             
             ])),
         StructField('id', IntegerType(), True)
])

spark = SparkSession.builder.getOrCreate()
data_dir = "/user/<user>/"

df1 = spark.createDataFrame(data1, schema1)
df1.write.mode('overwrite').parquet(data_dir + "data1") 
df2 = spark.createDataFrame(data2, schema2)
df2.write.mode('overwrite').parquet(data_dir + "data2") 


# READING DATA
parquet1 = spark.read.parquet(data_dir + "data1")
parquet2 = spark.read.parquet(data_dir + "data2")


# UNION
out = parquet1.union(parquet2)


parquet1.select("value.f2").distinct().show()
out.select("value.f2").distinct().show()
print(parquet1.collect())
print(out.collect()) {code}
Output: 
{code:java}
+---------+
|       f2|
+---------+
|[1, 1, 2]|
+---------+

+---------+
|       f2|
+---------+
|[0, 0, 0]|
|   [1, 1]|
+---------+


[
Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=0), 
Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=1)
]

[
Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=0), 
Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=1), 
Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=10), 
Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=11)
] {code}
Please notice that values for the field f2 are lost after the union is done. This only happens when this data is read from parquet files. 

Could you please look into this? 

Best regards,

Jakub
Bug: Use Jira notification options to sync with Github
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/23 07:25
Affects Version/s: 4.0.0

h3. Jira notification options

You can use the file to enable Jira notifications that will fire when a GitHub issue or pull request has a ticket in its title, such as {{{}"[TICKET-1234] Improve foo bar"{}}}.
You can set one or more of these options:
 * {{{}comment{}}}: Add the PR/issue event as a comment in the referenced Jira ticket.
 * {{{}worklog{}}}: Add the event as a worklog entry instead of a comment in the Jira ticket you reference.
 * {{{}label{}}}: Add a 'pull-request-available' label to referenced tickets.
 * {{{}link{}}}: When you create a GitHub PR/issue, embed a link to the PR or issue in the Jira ticket you reference.
Bug: Spark 3.4 multi-column sum slows with many columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/23 14:16
Affects Version/s: 3.4.0

The code below is a minimal reproducible example of an issue I discovered with Pyspark 3.4.x. I want to sum the values of multiple columns and put the sum of those columns (per row) into a new column. This code works and returns in a reasonable amount of time in Pyspark 3.3.x, but is extremely slow in Pyspark 3.4.x when the number of columns grows. See below for execution timing summary as N varies.
{code:java}
import pyspark.sql.functions as F
import random
import string
from functools import reduce
from operator import add
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# generate a dataframe N columns by M rows with random 8 digit column 
# names and random integers in [-5,10]
N = 30
M = 100
columns = [''.join(random.choices(string.ascii_uppercase +
                                  string.digits, k=8))
           for _ in range(N)]
data = [tuple([random.randint(-5,10) for _ in range(N)])
        for _ in range(M)]

df = spark.sparkContext.parallelize(data).toDF(columns)
# 3 ways to add a sum column, all of them slow for high N in spark 3.4
df = df.withColumn("col_sum1", sum(df[col] for col in columns))
df = df.withColumn("col_sum2", reduce(add, [F.col(col) for col in columns]))
df = df.withColumn("col_sum3", F.expr("+".join(columns))) {code}
Timing results for Spark 3.3:
||N||Exe Time (s)||
|5|0.514|
|10|0.248|
|15|0.327|
|20|0.403|
|25|0.279|
|30|0.322|
|50|0.430|

Timing results for Spark 3.4:
||N||Exe Time (s)||
|5|0.379|
|10|0.318|
|15|0.405|
|20|1.32|
|25|28.8|
|30|448|
|50|>10000 (did not finish)|
Bug: Update SparkConnectClientParser usage() method to match implementation
Status: Open | Priority: Major | Resolution: 
Created: 17/Aug/23 13:32
Affects Version/s: 3.5.0

Several missing options as well as inconsistent ones (`enable-ssl` vs `use_ssl`)  

https://github.com/apache/spark/blob/7af4e358f3f4902cc9601e56c2662b8921a925d6/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClientParser.scala#L31-L42
Bug: Make Python the first language in all Spark code snippet
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 15/Aug/23 18:54
Affects Version/s: 3.5.0

Currently, the first and default language for all code snippets is Sacla. For instance: https://spark.apache.org/docs/latest/quick-start.html

We should make Python the first language for all the code snippets.
Bug: spark job copies jars repeatedly if fs.defaultFS and application jar are same url
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 07:22
Affects Version/s: 3.4.1

In the org.apache.spark.deploy.yarn.Client#compareUri method, hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is that in the last pr, different URIs of user information are also regarded as different file systems. Uri.getauthority is used to determine the user information, but authority contains the host so the URI above must be different from authority. To determine whether the user authentication information is different, you only need to determine URI.getUserInfo.

 

the last pr and issue link:
https://issues.apache.org/jira/browse/SPARK-22587

https://github.com/apache/spark/pull/19885
Bug: SparkConnect ReattachExecute could raise before ExecutePlan even attaches.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/23 16:41
Affects Version/s: 3.5.0

If a ReattachExecute is sent very quickly after ExecutePlan, the following could happen:
 * ExecutePlan didn't reach *executeHolder.runGrpcResponseSender(responseSender)* in SparkConnectExecutePlanHandler yet.
 * ReattachExecute races around and reaches *executeHolder.runGrpcResponseSender(responseSender)* in SparkConnectReattachExecuteHandler first.
 * When ExecutePlan reaches {*}executeHolder.runGrpcResponseSender(responseSender){*}, and executionObserver.attachConsumer(this) is called in ExecuteGrpcResponseSender of ExecutePlan, it will kick out the ExecuteGrpcResponseSender or ReattachExecute.

So even though ReattachExecute came later, it will get interrupted by the earlier ExecutePlan and finish with a *SparkSQLException(errorClass = "INVALID_CURSOR.DISCONNECTED", Map.empty)* (which was assumed to be a situation where a stale hanging RPC is replaced by a reconnection.

 

That would be very unlikely to happen in practice, because ExecutePlan shouldn't be abandoned so fast, but because of  https://issues.apache.org/jira/browse/SPARK-44833 it is slightly more likely (though there there is also a 50ms sleep before retry, which again make it unlikely)
Bug: Improve error messages for Python UDTF returning non iterable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/23 17:59
Affects Version/s: 3.5.0

When the return type of a UDTF is not an iterable, the error message can be confusing to users. For example for this UDTF:
{code:java}
@udtf(returnType="x: int")
class TestUDTF:
    def eval(self, a):
        return a {code}
Currently it fails with this error for regular UDTFs:

    return tuple(map(verify_and_convert_result, res))
TypeError: 'int' object is not iterable

And this error for arrow-optimized UDTFs:

    raise ValueError("DataFrame constructor not properly called!")
ValueError: DataFrame constructor not properly called!
Bug: Spark Connect reattach when initial ExecutePlan didn't reach server doing too eager Reattach
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/23 16:34
Affects Version/s: 3.5.0

In
{code:java}
case ex: StatusRuntimeException
    if Option(StatusProto.fromThrowable(ex))
      .exists(_.getMessage.contains("INVALID_HANDLE.OPERATION_NOT_FOUND")) =>
  if (lastReturnedResponseId.isDefined) {
    throw new IllegalStateException(
      "OPERATION_NOT_FOUND on the server but responses were already received from it.",
      ex)
  }
  // Try a new ExecutePlan, and throw upstream for retry.
->  iter = rawBlockingStub.executePlan(initialRequest)
->  throw new GrpcRetryHandler.RetryException {code}
we call executePlan, and throw RetryException to have an exception handled upstream.

Then it goes to
{code:java}
retry {
  if (firstTry) {
    // on first try, we use the existing iter.
    firstTry = false
  } else {
    // on retry, the iter is borked, so we need a new one
->    iter = rawBlockingStub.reattachExecute(createReattachExecuteRequest())
  } {code}
and because it's not firstTry, immediately does reattach.

This causes no failure - the reattach will work and attach to the query, the original executePlan will get detached. But it could be improved.

Same issue is also present in python reattach.py.
Bug: Fix compilation warnings related to `Top-level wildcard is not allowed and will error under -Xsource:3`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/23 12:49
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/ExecuteHolder.scala:175:32: [deprecation @  | origin= | version=2.13.7] Top-level wildcard is not allowed and will error under -Xsource:3
[warn]   def removeGrpcResponseSender[_](sender: ExecuteGrpcResponseSender[_]): Unit = synchronized { {code}
Bug: CSV conversion performance severely degraded for null fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/23 18:40
Affects Version/s: 3.3.0

 
[https://github.com/apache/spark/pull/36110/files]
 introduced a SQLConf access in a critical section for every field processed in a record that is null.

This causes severe degradation of performance causing one workload that was completing in a couple of seconds to now take around 8 minutes.

This conf needs to be moved out of the critical path, there's no need for it to be in this location.

The version of Spark prior to this commit didn't exhibit the slowdown. I also generated a patch on an affected version with the suspected line removed and the problem went away.
Bug: Reflect function behavior different from Hive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/23 17:54
Affects Version/s: 3.4.1

Spark reflect function will fail if underlying method call throws exception. This causes the whole job to fail.

In Hive however the exception is caught and null is returned. Simple test to reproduce the behavior
{code:java}
select reflect('java.net.URLDecoder', 'decode', '%') {code}
The workaround would be to wrap this call in a try
[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala#L136]


We can support this by adding a new UDF `try_reflect` which mimics the Hive's behavior. Please share your thoughts on this.
Bug: Ignore `from_protobuf messageClassName` and `from_protobuf messageClassName options` in `PlanGenerationTestSuite`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/23 07:43
Affects Version/s: 3.5.0

The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module when testing the connect module. This allows `mvn clean install` and `mvn package test` to successfully pass tests. But if `mvn clean test` is executed directly, an error in `package org.sparkproject.spark_protobuf.protobuf does not exist` will occur. This is because `mvn clean test` directly uses the classes file of the `spark-protobuf` module for testing, without the 'package', hence it does not `shade` and `relocate` protobuf. 

On the other hand, SPARK-43646 breaks the usability of importing Spark as a Maven project into IDEA. 

Therefore, the changes of SPARK-43646 be reverted in [https://github.com/apache/spark/pull/42746]. 

In summary, when using Maven to test the connect module:
 * sometimes the shaded spark-protobuf jar will be used, such as 'mvn package test',
 * sometimes the unshaded classes directory will be used, such as 'mvn clean test'.

Therefore, there is no perfect solution to pass both tests, so 'from_protobuf messageClassName' and 'from_protobuf messageClassName' in 'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when we find a better solution.
Bug: Spark json schema inference and fromJson api having inconsistent behavior
Status: Open | Priority: Major | Resolution: 
Created: 28/Aug/23 19:12
Affects Version/s: 3.3.2

Spark json reader can infer datatype of a fields. I am ingesting millions of datapoints and  generating a `DataFrameA`. what i notice that Schema inference mark datatype of a field with tons of Integers and Empty Strings as a Long. That is an okay behavior as I don't set `primitivesAsString` as I do want  primitive type inference. I store `DataFrameA` into `TableA` 

Now, this inference behavior is not respected by `fromJson` of `from_json` api when I am trying to write new data on `TableA`. Means, if I read a chunk of new input data into using `spark.read.schema(fromJson(getStruct(TableA)).json('/path/to/more/data')` reader complains that EmptyString cannot be cast to Long .

ps - `getStruct(TableA)` is psuedo method that returns `struct` of TableA schema somehow. and `/path/to/more/data` is new dataset which has some records with value for this fields as an empty string.

 

I think if reader doesn't complain about Empty string during schema inference it shouldn't complain either on reading without inference. May be treat Empty as Null just like during schema inference. Empty string is a legal value for String type field but not Number types fields so I don't see any reason not to treat it as a Null. Another option is to give additional reader option - treatEmptyAsNull so it's more explicit? 

ps - I marked it as bug but could be more suited as improvements.
Bug: The precision of LongDecimal is inconsistent with Hive.
Status: Open | Priority: Major | Resolution: 
Created: 22/Aug/23 02:08
Affects Version/s: 3.4.0

The precision of LongDecimal in Hive is 19 but it is 20 in Spark. This leads to type conversion errors in some cases.

 

Relevant code:

[https://github.com/apache/spark/blob/4646991abd7f4a47a1b8712e2017a2fae98f7c5a/sql/api/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L129|https://github.com/apache/spark/blob/4646991abd7f4a47a1b8712e2017a2fae98f7c5a/sql/api/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L129C51-L129C51]

[https://github.com/apache/hive/blob/3d3acc7a19399d749a39818573a76a0dbbaf2598/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/HiveDecimalUtils.java#L76]

 

Reproduce:

create table and view in hive:
{code:java}
create table t (value bigint);
create view v as select value * 0.1 from t; {code}
read in spark:
{code:java}
select * from v; {code}
error occurred:
{code:java}
org.apache.spark.sql.AnalysisException: [CANNOT_UP_CAST_DATATYPE] Cannot up cast `(value * 0.1)` from "DECIMAL(22,1)" to "DECIMAL(21,1)".The type path of the target object is:
You can either add an explicit cast to the input data or choose a higher precision type of the field in the target object	at org.apache.spark.sql.errors.QueryCompilationErrors$.upCastFailureError(QueryCompilationErrors.scala:285)	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveUpCast$$fail(Analyzer.scala:3627)	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$$anonfun$apply$57$$anonfun$applyOrElse$235.applyOrElse(Analyzer.scala:3658)	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$$anonfun$apply$57$$anonfun$applyOrElse$235.applyOrElse(Analyzer.scala:3635) {code}
Bug: Implement SESSION_USER function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 23:15
Affects Version/s: 4.0.0

According to SQL standard SESSION_USER and CURRENT_USER behavior differs for routines:
- CURRENT_USER inside a routine should return security definer of a routine, e.g. owner identity
- SESSION_USER inside a routine should return connected user.
Bug: Convert binary to string by to_char for the formats: hex, base64, utf-8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/23 07:07
Affects Version/s: 4.0.0

Map the to_char() function with a binary input to one of hex(), base64(), decode() to achieve feature parity with:
- Snowflake: https://docs.snowflake.com/en/sql-reference/functions/to_char
- SAP SQL Anywhere: https://help.sap.com/docs/SAP_SQL_Anywhere/93079d4ba8e44920ae63ffb4def91f5b/81fe51196ce21014b9c6cf43b2666698.html
- Oracle: https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/TO_CHAR-number.html#GUID-00DA076D-2468-41AB-A3AC-CC78DBA0D9CB
- Vertica: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Formatting/TO_CHAR.htm
Bug: Fix inherited namedtuples to work in createDataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/23 02:06
Affects Version/s: 3.5.0

{code}
from collections import namedtuple
MyTuple = namedtuple("MyTuple", ["zz", "b", "a"])

class MyInheritedTuple(MyTuple):
    pass

df = spark.createDataFrame([MyInheritedTuple(1, 2, 3), MyInheritedTuple(11, 22, 33)])
df.collect()
{code}

{code}
[Row(zz=None, b=None, a=None), Row(zz=None, b=None, a=None)]
{code}

should be

{code}
[Row(zz=1, b=2, a=3), Row(zz=11, b=22, a=33)]
{code}