av9ash/GitBugs huggingface dataset:
Bug: Isotonic regression result differs from sklearn implementation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Nov/22 15:51
Affects Version/s: 3.3.1

 
{code:python}
import pandas as pd
from pyspark.sql.types import DoubleType
from sklearn.isotonic import IsotonicRegression as IsotonicRegression_sklearn
from pyspark.ml.regression import IsotonicRegression as IsotonicRegression_pyspark

# The P(positives | model_score):
# 0.6 -> 0.5 (1 out of the 2 labels is positive)
# 0.333 -> 0.333 (1 out of the 3 labels is positive)
# 0.20 -> 0.25 (1 out of the 4 labels is positive)
tc_pd = pd.DataFrame({
    "model_score": [0.6, 0.6, 0.333, 0.333, 0.333, 0.20, 0.20, 0.20, 0.20],         
    "label": [1, 0, 0, 1, 0, 1, 0, 0, 0],         
    "weight": 1,     }
)

# The fraction of positives for each of the distinct model_scores would be the best fit.
# Resulting in the following expected calibrated model_scores:
# "calibrated_model_score": [0.5, 0.5, 0.333, 0.333, 0.333, 0.25, 0.25, 0.25, 0.25]

# The sklearn implementation of Isotonic Regression. 
from sklearn.isotonic import IsotonicRegression as IsotonicRegression_sklearn
tc_regressor_sklearn = IsotonicRegression_sklearn().fit(X=tc_pd['model_score'], y=tc_pd['label'], sample_weight=tc_pd['weight'])
print("sklearn:", tc_regressor_sklearn.predict(tc_pd['model_score']))
# >> sklearn: [0.5 0.5 0.33333333 0.33333333 0.33333333 0.25 0.25 0.25 0.25 ]

# The pyspark implementation of Isotonic Regression. 
tc_df = spark.createDataFrame(tc_pd)
tc_df = tc_df.withColumn('model_score', F.col('model_score').cast(DoubleType()))

isotonic_regressor_pyspark = IsotonicRegression_pyspark(featuresCol='model_score', labelCol='label', weightCol='weight')
tc_model = isotonic_regressor_pyspark.fit(tc_df)
tc_pd = tc_model.transform(tc_df).toPandas()
print("pyspark:", tc_pd['prediction'].values)
# >> pyspark: [0.5 0.5 0.33333333 0.33333333 0.33333333 0. 0. 0. 0. ]

# The result from the pyspark implementation seems unclear. Similar small toy examples lead to similar non-expected results for the pyspark implementation. 

# Strangely enough, for 'large' datasets, the difference between calibrated model_scores generated by both implementations dissapears. 
{code}
Bug: AM shutdown hook fails with IllegalStateException if AM crashes on startup (recurrence of SPARK-3900)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Nov/22 05:04
Affects Version/s: 3.4.0

SPARK-3900 fixed the {{IllegalStateException}} in cleanupStagingDir in ApplicationMaster's shutdownhook. However, SPARK-21138 accidentally reverted/undid that change when fixing the "Wrong FS" bug. Now, we are seeing SPARK-3900 reported by our users at Linkedin. We need to bring back the fix for SPARK-3900.

The illegalStateException when creating a new filesystem object is due to the limitation in Hadoop that we can not register a shutdownhook during shutdown. So, when a spark job fails during pre-launch, as part of shutdown, cleanupStagingDir would be called. Then, if we attempt to create a new filesystem object for the first time, HDFS would try to register a hook to shutdown KeyProviderCache when creating a ClientContext for DFSClient. As a result, we hit the {{IllegalStateException}}. We should avoid the creation of a new filesystem object in cleanupStagingDir() when it is called in a shutdown hook. This was introduced in SPARK-3900. However, SPARK-21138 accidentally reverted/undid that change. We need to bring back that fix to Spark to avoid the {{IllegalStateException}}.
Bug: Test case for insert partition should verify value 
Status: In Progress | Priority: Minor | Resolution: 
Created: 01/Nov/22 14:48
Affects Version/s: 3.0.0

Spark3 has not validated the Partition Column type while inserting the data but on the Hive side exception is thrown while inserting different type values.

*Spark Code:*

 
{code:java}
scala> val tableName="test_partition_table"
tableName: String = test_partition_table

scala>scala> spark.sql(s"DROP TABLE IF EXISTS $tableName")
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql(s"CREATE EXTERNAL TABLE $tableName ( id INT, name STRING ) PARTITIONED BY (age INT) LOCATION 'file:/tmp/spark-warehouse/$tableName'")
res1: org.apache.spark.sql.DataFrame = []

scala> spark.sql("SHOW tables").show(truncate=false)
+---------+---------------------+-----------+
|namespace|tableName            |isTemporary|
+---------+---------------------+-----------+
|default  |test_partition_table |false      |
+---------+---------------------+-----------+

scala> spark.sql("SET spark.sql.sources.validatePartitionColumns").show(50, false)
+------------------------------------------+-----+
|key                                       |value|
+------------------------------------------+-----+
|spark.sql.sources.validatePartitionColumns|true |
+------------------------------------------+-----+

scala> spark.sql(s"""INSERT INTO $tableName partition (age=25) VALUES (1, 'Ranga')""")
res4: org.apache.spark.sql.DataFrame = []scala> spark.sql(s"show partitions $tableName").show(50, false)
+---------+
|partition|
+---------+
|age=25   |
+---------+

scala> spark.sql(s"select * from $tableName").show(50, false)
+---+-----+---+
|id |name |age|
+---+-----+---+
|1  |Ranga|25 |
+---+-----+---+

scala> spark.sql(s"""INSERT INTO $tableName partition (age=\"test_age\") VALUES (2, 'Nishanth')""")
res7: org.apache.spark.sql.DataFrame = []scala> spark.sql(s"show partitions $tableName").show(50, false)
+------------+
|partition   |
+------------+
|age=25      |
|age=test_age|
+------------+

scala> spark.sql(s"select * from $tableName").show(50, false)
+---+--------+----+
|id |name    |age |
+---+--------+----+
|1  |Ranga   |25  |
|2  |Nishanth|null|
+---+--------+----+ {code}
*Hive Code:*

 

 
{code:java}
> INSERT INTO test_partition_table partition (age="test_age2") VALUES (3, 'Nishanth');
Error: Error while compiling statement: FAILED: SemanticException [Error 10248]: Cannot add partition column age of type string as it cannot be converted to type int (state=42000,code=10248){code}
 

*Expected Result:*

When *spark.sql.sources.validatePartitionColumns=true* it needs to be validated the datatype value and exception needs to be thrown if we provide wrong data type value.

*Reference:*

[https://spark.apache.org/docs/3.3.1/sql-migration-guide.html#data-sources]
Bug: Optimize constructor use of `StructType`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Nov/22 06:06
Affects Version/s: 3.4.0

There are two main ways to construct `StructType`:

- Primary constructor

```scala
case class StructType(fields: Array[StructField])
```

- Use `Seq` as input constructor

```scala
def apply(fields: Seq[StructField]): StructType = StructType(fields.toArray)
```

These two construction methods are widely used in Spark, but the latter requires an additional collection conversion.

This pr changes the following 3 scenarios to use primary constructor to reduce one collection conversion:

1. For manually create `Seq` input scenes, change to use manually create `Array` input instead, for examaple:

https://github.com/apache/spark/blob/bcf03fe3f86a7230fd977c059b73a58554370d5d/mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala#L55-L63

2. For the scenario where 'toSeq' is added to create input for compatibility with Scala 2.13, directly call 'toArray' to instead, for example:

https://github.com/apache/spark/blob/bcf03fe3f86a7230fd977c059b73a58554370d5d/connector/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala#L108-L113

3. For scenes whose input is originally `Array`, remove the redundant `toSeq`, for example:

https://github.com/apache/spark/blob/bcf03fe3f86a7230fd977c059b73a58554370d5d/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L587-L592
Bug: Optimized query plan cost/statistics overview
Status: Open | Priority: Minor | Resolution: 
Created: 29/Nov/22 15:10
Affects Version/s: 3.3.0

*Motivation*

Spark SQL supports running the `EXPLAIN COST` statement on a query to show the optimized logical plan and its data costs per stage (i.e. statistics) [https://spark.apache.org/docs/latest/sql-ref-syntax-qry-explain.html]. However, it can currently be difficult to determine what the total data read cost will be for a complex query with many stages. Other query engines such as Trino/Presto attempt to provide a general estimate of resource costs of a query when running the `EXPLAIN` statement, which includes CPU, memory, row count, and data size [https://trino.io/docs/current/optimizer/cost-in-explain.html.]

*Proposal*

We suggested adding an overview/estimation of the total resources that will be used within the optimized logical plan of a Spark query, or maybe as an alternative, provide this overview/estimation when the `EXPLAIN COST` statement is called on a query. As a first version, it would already be beneficial if this general cost estimation would include anything that is available within the statistics of the optimized query plan, such as:
 * The amount of data the will be read in bytes
 * The total amount of rows 
 * etc.

Given that the optimized logical plan is divided in stages it would already be sufficient to show these parameters per stage so they can be aggregated for the entire job later on if needed.
Bug: Support configuring the ip dscp to the sink.
Status: Open | Priority: Minor | Resolution: 
Created: 29/Nov/22 11:13
Affects Version/s: 3.3.1

Usage:

In my case I have 2 zones one for computation the other for result storage and the bond width between the 2 zones is limited.

I have 2 task with different priorities and I want the hight priority task can transfer data without the effecting of the low priority task.

 

Possible solution:

Support configuring the ip dscp to the sink.
Bug: YarnAllocator.rpIdToYarnResource map is not properly updated
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/22 13:11
Affects Version/s: 3.1.0

Log messages "INFO YarnAllocator: Resource profile 0 doesn't exist, adding it" repeats multiple times in yarn stderr.

The log should be outputted only once because it happens in _YarnAllocator.createYarnResourceForResourceProfile_ on a default ResourceProfile

After digging into the code, I found a bug caused by misleading usage of _ConcurrentHashMap.contains_
Bug: Code cleanup for assertXXX methods in ExpressionTypeCheckingSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Nov/22 09:28
Affects Version/s: 3.4.0

https://github.com/apache/spark/blob/d979736a9eb754725d33fd5baca88a1c1a8c23ce/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ExpressionTypeCheckingSuite.scala#L61-L108
Bug: Bad error message when try_to_binary called with non-foldable format
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 19/Nov/22 22:31
Affects Version/s: 3.3.1

For example:
{noformat}
spark-sql> SELECT try_to_binary(col1, col2) from values ('abc', 'utf-8') as data(col1, col2);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
...
Caused by: java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:208)
	at org.apache.spark.sql.catalyst.expressions.ToBinary.$anonfun$replacement$1(stringExpressions.scala:2597)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.sql.catalyst.expressions.ToBinary.replacement$lzycompute(stringExpressions.scala:2596)
	at org.apache.spark.sql.catalyst.expressions.ToBinary.replacement(stringExpressions.scala:2596)
	at org.apache.spark.sql.catalyst.expressions.RuntimeReplaceable.dataType(Expression.scala:354)
{noformat}
Compare to {{to_binary}}:
{noformat}
spark-sql> SELECT to_binary(col1, col2) from values ('abc', 'utf-8') as data(col1, col2);
The 'format' parameter of function 'to_binary' needs to be a string literal.; line 1 pos 7
spark-sql> 
{noformat}
Bug: Regression in IntegralDivide
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Nov/22 23:00
Affects Version/s: 3.4.0

There has been a regression in Integral Divide after the removal of PromotePrecision from Spark 3.4.0.
{code:java}
scala> val data = Seq(Row(BigDecimal("-7.70892"), BigDecimal("4.27138661282262736522411173299611831E+40")))
scala> val simpleSchema = StructType(Array(      
| StructField("a", DecimalType(6,5),true),      
| StructField("b", DecimalType(36,-5), true)))
scala> val df = spark.createDataFrame(spark.sparkContext.parallelize(data), simpleSchema)
{code}
 The above statements result in an AnalysisException thrown

 
{code:java}
org.apache.spark.sql.AnalysisException: Decimal scale (0) cannot be greater than precision (-4).
  at org.apache.spark.sql.errors.QueryCompilationErrors$.decimalCannotGreaterThanPrecisionError(QueryCompilationErrors.scala:2237)
  at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:49)
  at org.apache.spark.sql.types.DecimalType$.bounded(DecimalType.scala:164)
  at org.apache.spark.sql.catalyst.expressions.IntegralDivide.resultDecimalType(arithmetic.scala:868)
  at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.dataType(arithmetic.scala:238)
  at org.apache.spark.sql.catalyst.expressions.IntegralDivide.org$apache$spark$sql$catalyst$expressions$DivModLike$$super$dataType(arithmetic.scala:842)

{code}
I believe this is happening because we aren't promoting the precision like we were before this [PR|https://github.com/apache/spark/commit/301a13963808d1ad44be5cacf0a20f65b853d5a2] went in. Without promoting precision the resultDecimalType in the example above tries to return a Decimal with precision of -4 and scale of 0 which is invalid
Bug: avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Nov/22 23:52
Affects Version/s: 3.3.1

Currently the  analyzer phase rules on subquery referencing the aggregate expression in outer query, avoids introducing a new aggregate only for a single level aggregate function. It introduces new aggregate expression for nested aggregate functions.

It is possible to avoid  adding this extra aggregate expression  easily, atleast if the outer projection involving aggregate function is exactly same as the one that is used in subquery, or if the outer query's projection involving aggregate function is a subtree of the subquery's expression.

 

Thus consider the following 2 cases:

1) select  cos (sum(a)) , b from t1  group by b having exists (select x from t2 where y = cos(sum(a)) )

2) select  sum(a) , b from t1  group by b having exists (select x from t2 where y = cos(sum(a)) )

 

In both the above cases, there is no need for adding extra aggregate expression.

 

I am also investigating if its possible to avoid if the case is 

 

3) select  Cos(sum(a)) , b from t1  group by b having exists (select x from t2 where y = sum(a) )

 

This Jira also is needed for another issue where subquery datasource v2  is projecting columns which are not needed. ( no Jira filed yet for that, will do that..)

 

Will be opening a PR for this soon..
Bug: Cache Table is not working while nested caches exist
Status: In Progress | Priority: Minor | Resolution: 
Created: 18/Nov/22 04:06
Affects Version/s: 3.3.1

For example the following statement:
{code:java}
//代码占位符
cache table t1 as select a from testData3 group by a;
cache table t2 as select a,b from testData2 where a in (select a from t1);
select key,value,b from testData t3 join t2 on t3.key=t2.a;{code}
The cached t2 is not used in the third statement
Bug: Add error message to SchemaColumnConvertNotSupportedException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Nov/22 04:55
Affects Version/s: 3.4.0

Just found the fact that SchemaColumnConvertNotSupportedException doesn't have any error message is annoying for debugging.

In stack trace, we only see SchemaColumnConvertNotSupportedException but don't know what column is wrong.
Bug: Support Bit manipulation function COUNTSET
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Nov/22 00:26
Affects Version/s: 3.1.4

Support Bit manipulation function COUNTSET
The function shall return the number of 1 bits in the specified integer value. If the optional second argument is set to zero, it shall return the number of 0 bits instead.

COUNTSET(integer_type a [, INT zero_or_one])
Bug: Support Bit manipulation function SETBIT
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Nov/22 00:10
Affects Version/s: 3.1.4

Support function to change a bit at a specified position. It shall change a bit at a specified position to a 1,  If the optional third argument is set to zero, the specified bit shall be set to 0 instead.

SETBIT(integer_type a, INT position [, INT zero_or_one])
Bug: Support Bit manipulation function SHIFTRIGHT
Status: Open | Priority: Minor | Resolution: 
Created: 10/Nov/22 00:24
Affects Version/s: 3.1.4

Support Bit manipulation function SHIFTRIGHT

The function shall shift an integer value right by a specified number of bits. As the least significant bit will be taken out of the original value, it shall be discarded and the most significant bit becomes 0.
*SHIFTRIGHT(integer_type a, INT positions)*
Bug: Replace `FRAME_LESS_OFFSET_WITHOUT_FOLDABLE` with `NON_FOLDABLE_INPUT`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Nov/22 08:54
Affects Version/s: 3.4.0

`FRAME_LESS_OFFSET_WITHOUT_FOLDABLE` and `NON_FOLDABLE_INPUT` look similar, but `NON_FOLDABLE_INPUT` is more general
Bug: Upgrade `org.tukaani:xz` to 1.9
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Nov/22 17:50
Affects Version/s: 3.4.0

https://github.com/apache/avro/blob/3a9e5a789b5165e0c8c4da799c387fdf84bfb75e/lang/java/pom.xml#L59
Bug: BigInteger Serialization doesn't work with JavaBean Encoder
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Nov/22 15:07
Affects Version/s: 3.3.1

When creating a dataset using the [Java Bean Encoder|https://spark.apache.org/docs/3.2.0/api/java/org/apache/spark/sql/Encoders.html#bean-java.lang.Class-] with a bean that contains a field which is a {{java.math.BigInteger}} the dataset will fail to serialize correctly. When trying to serialize the dataset, Spark throws the following error:

 
{code:java}
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot up cast `bigInteger` from struct<> to decimal(38,18).
 {code}
 

Reproduction steps:

Using the Java Dataset API:
 # Create a Bean with a  {{java.math.BigInteger}} field
 # Pass said Bean into the Java SparkSession {{createDataset}} function
Bug: Test some subclasses of  error class DATATYPE_MISMATCH
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Nov/22 01:28
Affects Version/s: 3.4.0

include:
 # DATATYPE_MISMATCH.BINARY_ARRAY_DIFF_TYPES
 # DATATYPE_MISMATCH.CANNOT_CONVERT_TO_JSON
 # DATATYPE_MISMATCH.FRAME_LESS_OFFSET_WITHOUT_FOLDABLE
 # DATATYPE_MISMATCH.MAP_FROM_ENTRIES_WRONG_TYPE
 # DATATYPE_MISMATCH.NON_STRING_TYPE
 # DATATYPE_MISMATCH.NULL_TYPE
 # DATATYPE_MISMATCH.SPECIFIED_WINDOW_FRAME_DIFF_TYPES
 # DATATYPE_MISMATCH.SPECIFIED_WINDOW_FRAME_UNACCEPTED_TYPE (already exists)

[spark/sql/core/src/test/resources/sql-tests/results/window.sql.out |https://github.com/apache/spark/blob/7009ef0510dae444c72e7513357e681b08379603/sql/core/src/test/resources/sql-tests/results/window.sql.out#L106]Line 106 in [7009ef0|https://github.com/apache/spark/commit/7009ef0510dae444c72e7513357e681b08379603]
||"errorClass" : "DATATYPE_MISMATCH.SPECIFIED_WINDOW_FRAME_UNACCEPTED_TYPE",||

      9. DATATYPE_MISMATCH.SPECIFIED_WINDOW_FRAME_WITHOUT_FOLDABLE

     10. DATATYPE_MISMATCH.UNSPECIFIED_FRAME
Bug: When the value of quote or escape exists in the content of csv file, the character in the csv file will be misidentified
Status: In Progress | Priority: Minor | Resolution: 
Created: 01/Nov/22 03:41
Affects Version/s: 3.2.1

When the value of quote or escape exists in the content of csv file, the character in the csv file will be misidentified
We found that when the value of quote or escape exists in the content of csv file, the character in the csv file will be misidentified.
When this content is being read by Azure Data Factory copy activity and written to CSV, the content is
"test\\" =
test"
we read csv as below:
df = spark.read.csv(path='test.csv'
, sep=','
, header=True
, quote='"'
, escape='\'
, multiLine=True
, lineSep='\n'
)
resulting in the following being written to the CSV: *test\" =* and *test* in the next line ,but what we want {*}test\\" = test{*}.
Now when the above is being read by Spark:
 # The first \ is being interpreted as being an escaping of the second \ (so the content looks like a single literal )
 # The " now appears to be an unescaped quote character, so we're back in the situation where Spark tries to handle this using STOP_AT_DELIMITER.
As before, the rest of the CSV after this point is being parsed incorrectly.

We could change the "quote,escape..." to avoid it for the scenario above, but the content of their csv file is very large and it may occur any character. the data sources that we have which are affected by this issue are systems outside of our control, so we have no means of controlling what content/characters will be there.When we change the "quote,escape...", it may conflict with the content again, and it still have issues in the following content.
As far as designing the content to avoid certain characters - the data sources that we have which are affected by this issue are systems outside of our control, so we have no means of controlling what content/characters will be there.
Bug: Remove legacy example of round function with negative scale 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Nov/22 09:58
Affects Version/s: 3.2.2

Run this sql in spark-sql:    select round(1.233, -1); 
Error: org.apache.spark.sql.AnalysisException: Negative scale is not allowed: -1. You can use spark.sql.legacy.allowNegativeScaleOfDecimal=true to enable legacy mode to allow it.; 

But the document of round function implies that negative scale argument is allowed.
Bug: Improve documentation and typing of applyInPandas for groupby and cogroup
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Nov/22 10:23
Affects Version/s: 3.4.0

Documentation of method `applyInPandas` for grouby and cogroup does not mention in the main description that there are two allowed signatures for the provided function. The Examples state that this is possible, and the parameters sections states that for cogroup.

That should be made cleaner.

Finally, type information for `PandasCogroupedMapFunction` does not mention the three-argument callable alternative.
Bug: Upgrade `scala-parallel-collections` to 1.0.4 for Scala 2.13
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Nov/22 05:24
Affects Version/s: 3.4.0

scala-parallel-collections started from version 1.0.4 to verify the compatibility of Java 17 through CI
 * [https://github.com/scala/scala-parallel-collections/compare/v1.0.3...v1.0.4]
 * https://github.com/scala/scala-parallel-collections/releases/tag/v1.0.4